# Example of a foliation for Ueda type one

Paulo Sad

(04/08/2023)

###### Abstract

We give an example of a projective 2-dimensional foliation which is regular
along a curve of genus 3 and has Ueda type one with normal bundle of order
two. 111 MSC Number: 32S65

Introduction

In this Introduction we present a short list of results of the so called Ueda
Theory, developed in [5] with focus in local properties. Later on this matter
was studied in [4] from a global point of view, and recently appered again in
the work of [3].

Let $C$ be a smooth, compact, connected holomorphic curve contained in some
holomorphic surface $M$ satisfying $C.C=0$ ($C.C$ stands for the self
intersection number of $C$ inside $M$). We consider a type $\nu$ coordinate
system $\\{\psi_{i}:\mathbb{D}\times\mathbb{D}\rightarrow M\\}$ for a
neighborhood of $C$:

$z_{k}=\psi_{k}^{-1}\circ\psi_{i}(z_{i},y_{i})$

$y_{k}={\lambda}_{ik}(z_{i})y_{i}+f_{ik}(z_{i})y_{i}^{\nu+1}+\dots$

for some $\nu\geq 1$.

The normal bundle $N_{C}$ to $C$ is given by

$z_{k}=\psi_{k}^{-1}\circ\psi_{i}(z_{i},0)$

$y_{k}={\lambda}_{ik}(z_{i})y_{i}$

Since $C.C=0$, we may assume that
$\lambda_{ik}(z_{i})=\lambda_{ik}\in\mathbb{C}^{*}$.

The collection $\\{f_{ik}\\}$ is a cocycle of the sheaf of holomorphic
functions of $N_{C}^{-1}$; when it is a coboundary, we may replace the type
$\nu$ coordinate system by a $\nu+1$ coordinate system. The curve $C$ has
finite type $n\in\mathbb{N}$ when there exists a system of coordinates of type
$n$ whose associated cocycle is not a coboundary; we say that $n$ is the Ueda
type of $C$. The Ueda type of $C$ is infinite when the cocycle associated to
any coordinate system is a coboundary.

The finite type is characterized by the following result:

Theorem 1 If $C$ has finite type then it has a fundamental system of pseudo-
concave neighborhoods.

In particular, there are no compact curves in a neighborhood of $C$ disjoint
of $C$.

When the Ueda type is infinite, the extra condition of $N_{C}$ being a torsion
bundle leads to the following

Theorem 2 If $C$ has infinite type and its normal bundle $N_{C}$ is a torsion
bundle of order $k\in\mathbb{N}$ then there exists a holomorphic foliation by
compact leaves in a neighborhood of $C$ such that $C$ is a leaf (in fact the
foiation can be seen as a fibration which has $C$ as a multiple fiber of
multiplicity $k$).

Both results appeared in [5]. We state now the results of [4] and [3]
respectively in the case $M$ is a projective surface.

Theorem 3 Assume that $N_{C}$ is a torsion bundle of order $k\in\mathbb{N}$
and let $n\in\mathbb{N}\cup\\{\infty\\}$ be the Ueda type of $C$. If $n>k$
then in fact $n=\infty$ (and Theorem 2 applies).

Theorem 4 Assume $N_{C}$ is a torsion bundle of order $k\in\mathbb{N}$ and let
$n\in\mathbb{N}$ be the Ueda type of $C$. If $n=k$ then there exists a
foliation in $M$ which has $C$ as a leaf.

We remind that $C$ being a leaf means in particular that it contains no
singularity of the foliation.

Theorem 4 leads to the following question: suppose $N_{C}$ is a torsion bundle
of order $k\in\mathbb{N}$ and let $n\in\mathbb{N}$ be the Ueda type of $C$;
what can be said about the existence of a foliation having $C$ as a leaf if
$n<k$? We present here an example in the positive direction when $n=1$ and
$k=2$ (correcting a statement of [6]).

We have in mind the following picture: we start with the projective plane
$\mathbb{P}^{2}$ with some foliation which has a smooth algebraic invaariant
curve $S$ of degree $d$; this curve $S$ contains singularities
$p_{1}\dots,p_{l}$ with tangent multiplicities (of the foliation along $S$)
$m_{1},\dots,m_{l}$. According to [2], the degree $0$ divisor
$\sum_{j=1}^{l}m_{j}p_{j}-LD_{\infty}$ is a principal divisor
($d.L=\sum_{j=1}^{l}m_{j}$ and $D_{\infty}$ is the divisor obtained
intersecting $S$ with some line, which we fix at infinity). Since a generic
choice of points in $S$ produces no ”resonances” there is no foliation having
such points as singularities in the invariant curve $S$. In particular, when
we blow-up $\mathbb{P}^{2}$ at $d^{2}$ points in $S$ chosen in a generic way
the strict transform $\widehat{S}$ of $S$ can not be a leaf of a foliation in
the transformed surface $\widehat{\mathbb{P}^{2}}$.

In this paper we will work with a particular Neeman’s example. This family of
examples (see [4]) is constructed in the following way: we start with a fixed
quartic $S$ and blow it up at 16 points which lie at two different bitangent
lines (4 points $P_{1},\dots,P_{4}$) and some cubic (12 points
$P_{5},\dots,P_{16}$). The Ueda type of $\widehat{S}$ is 1, and the normal
bundle $N_{\widehat{S}}$ is associated to the divisor
$\sum_{j=1}^{16}P_{j}-4D_{\infty}$. We see that
$2(\sum_{j=1}^{16}P_{j}-4D_{\infty}$) is a principal divisor, so that
$N_{\widehat{S}}$ is a torsion bundle of order 2. Such a resonance is realized
by a foliation in $\mathbb{P}^{2}$ of degree 10; it has 16 singularities of
tangent multiplicities equal to 2 along $S$. The group of holonomy of
$S\setminus\\{P_{1},\dots,P_{16}\\}$ (which is the same of $\widehat{S}$) is
infinite, although the local holonomy maps at the singularities are all igual
to the identity map.

Example

Let $F(X,Y)$ be a degree four polynomial which defines a smooth projective
quartic $S$ transverse to the line at infinity; we assume also that
$\\{F=0\\}\cap\\{F_{X}=0\\}$ has 12 different points and $l_{1}(X,Y)=Y+X=0$
and $l_{2}(X,Y)=Y+X+b=0$ are bitangent lines. We put

$G=l_{1}l_{2}F_{X}^{2}$

$h=\dfrac{(l_{1}l_{2})_{X}F_{Y}}{2}+\dfrac{(l_{1}l_{2})_{Y}F_{X}}{2}+l_{1}l_{2}F_{XY}=\dfrac{(l_{1}l_{2})_{X}(F_{X}+F_{Y})}{2}+l_{1}l_{2}F_{XY}$

$k=-F_{X}-\dfrac{(l_{1}l_{2})_{X}F_{XY}}{2}$

$h^{\prime}=-(l_{1}l_{2})_{X}F_{X}-l_{1}l_{2}F_{XX}$

$k^{\prime}=F_{X}+\dfrac{(l_{1}l_{2})_{X}F_{XX}}{2}$

$A=hF_{X}+kF$

$B=h^{\prime}F_{X}+k^{\prime}F$

The foliation $\cal F$ we will study is given by

$GdF+F(BdX-AdY)=0$

Let $\widehat{\cal F}$ be the transformed foliation from $\cal F$ and
$\widehat{S}$ be the transformed curve from $S$ after blowing-up at the points
of $\\{G=0\\}\cap S$.

Theorem. $\widehat{S}$ is a leaf of $\widehat{\cal F}$.

First of all we need to compute the degree of the foliation $\cal F$.

Lemma. $\cal F$ has degree 10.

Proof: it ammounts to showing that the homogeneous polynomial of degree 12 in
the development of

$X(GF_{X}+FB)+Y(GF_{Y}-FA)=$

$XGF_{X}+YGF_{Y}+F[X(h^{\prime}F_{X}+k^{\prime}F)-Y(hF_{X}+kF)]$

is in fact equal to 0. In what follows, all the equalities refer to
homogeneous polynomials of top degree. For example, $XGF_{X}+YGF_{Y}=4GF$.

Let us compute $X(h^{\prime}F_{X}+k^{\prime}F)-Y(hF_{X}+kF)$. We have

$X(h^{\prime}F_{X}+k^{\prime}F)-Y(hF_{X}+kF)=$

$=-X(l_{1}l_{2}F_{X})_{X}F_{X}-Y\left[\dfrac{(l_{1}l_{2})_{X}F_{Y}}{2}+\dfrac{(l_{1}l_{2})_{Y}F_{X}}{2}+l_{1}l_{2}F_{XY}\right]F_{X}$

$+X\left[\dfrac{(l_{1}l_{2})_{XX}F_{X}}{2}+\dfrac{(l_{1}l_{2})_{X}F_{XX}}{2}\right]F-Y\left[-F_{X}-\dfrac{(l_{1}l_{2})_{X}F_{XY}}{2}\right]F$

$=-X(l_{1}l_{2}F_{X})_{X}F_{X}-Y\left[\dfrac{(l_{1}l_{2})_{X}F_{Y}}{2}+\dfrac{(l_{1}l_{2})_{Y}F_{X}}{2}+l_{1}l_{2}F_{XY}\right]F_{X}$

$+(X+Y)FF_{X}+\left[\dfrac{X(l_{1}l_{2})_{X}F_{XX}}{2}+\dfrac{Y(l_{1}l_{2})_{X}F_{XY}}{2}\right]F$

$=-X(l_{1}l_{2}F_{X})_{X}F_{X}-Y\left[\dfrac{(l_{1}l_{2})_{X}F_{Y}}{2}+\dfrac{(l_{1}l_{2})_{Y}F_{X}}{2}+l_{1}l_{2}F_{XY}\right]F_{X}$

$+(X+Y)FF_{X}+\dfrac{3(l_{1}l_{2})_{X}F_{X}F}{2}$

$=-\left[\dfrac{X(l_{1}l_{2})_{X}F_{X}^{2}}{2}+\dfrac{Y(l_{1}l_{2})_{X}F_{X}F_{Y}}{2}\right]-\dfrac{X(l_{1}l_{2})_{X}F_{X}^{2}}{2}$

$-\left[Xl_{1}l_{2}F_{XX}F_{X}+Yl_{1}l_{2}F_{XY}F_{X}\right]-\dfrac{Y(l_{1}l_{2})_{Y}F_{X}^{2}}{2}$

$+(X+Y)FF_{X}+\dfrac{3(l_{1}l_{2})_{X}F_{X}F}{2}$

$=-2(l_{1}l_{2})_{X}FF_{X}-3l_{1}l_{2}F_{X}^{2}-\dfrac{\left[X(l_{1}l_{2})_{X}F_{X}^{2}+Y(l_{1}l_{2})_{Y}F_{X}^{2}\right]}{2}$

$+(X+Y)FF_{X}+\dfrac{3(l_{1}l_{2})_{X}F_{X}F}{2}$

$-2(l_{1}l_{2})_{X}FF_{X}-3l_{1}l_{2}F_{X}^{2}--l_{1}l_{2}F_{X}^{2}+(X+Y)FF_{X}+\dfrac{3(l_{1}l_{2})_{X}FF_{X}}{2}$

$=-l_{1}l_{2}F_{X}^{2}-3l_{1}l_{2}F_{X}^{2}-2(l_{1}l_{2})_{X}FF_{X}+(X+Y)FF_{X}+\dfrac{3(l_{1}l_{2})_{X}FF_{X}}{2}$

$=-4l_{1}l_{2}F_{X}^{2}$.

It follows finally that

$XGF_{X}+YGF_{Y}+F[X(h^{\prime}F_{X}+k^{\prime}F)-Y(hF_{X}+kF)]$

$=4GF-4l_{1}l_{2}F_{X}^{2}F=0$ ∎

In particular, the line at infinite is not invariant for $\cal F$. Let us
observe that the 16 points of $\\{G=0\\}\cap S$ are singularities of $\cal F$
having tangent multiplicity equal to 2. These are all the singularities of
$\cal F$ along $S$. In fact, from [1] we have that $(d({\cal
F})+2).d(S)=tang(\cal F,S)+S.S$ (d stands for degree), so that $tang(\cal
F,S)=$ 32\. But this is already achieved at the singularities of
$\\{G=0\\}\cap S$.

In order to prove that $\widehat{S}$ is a leaf of $\widehat{\cal F}$ we have
to check at each of the points $P_{1}\dots,P_{16}$ the following conditions
(see [6]):

1\. $A=\dfrac{\partial G}{\partial Y}$

2\. $AF_{X}+BF_{Y}=0$

3\. $A_{Y}=\dfrac{G_{YY}}{2}+\dfrac{F_{YY}G_{Y}}{2F_{Y}}$

4\. $B_{Y}-A_{X}=-\dfrac{F_{XY}G_{Y}}{F_{Y}}-G_{XY}$

5\. $B_{X}=-\dfrac{G_{XX}}{2}-\dfrac{F_{XX}G_{Y}}{2F_{Y}}$

Let us notice that at the points $P_{1},\dots,P_{4}$ we have
$(l_{1}l_{2})_{X}=(l_{1}l_{2})_{Y}$ and $F_{X}=F_{Y}$.

Condition 1. We have $A=0$ and $\dfrac{\partial G}{\partial
Y}=(l_{1}l_{2})_{Y}F_{X}^{2}+2l_{1}l_{2}F_{X}F_{XX}=0$ at
$P_{5},\dots,P_{16}$. At $P_{1},\dots,P_{4}$, we see that
$A=hF_{X}=(l_{1}l_{2})_{Y}F_{X}^{2}=\dfrac{\partial G}{\partial Y}$

Condition 2. $A=B=0$ at $P_{5},\dots,P_{16}$ implies $AF_{X}+BF_{Y}=0$ at
these points. As for $P_{1},\dots,P_{4}$:

$AF_{X}+BF_{Y}=hF_{X}^{2}-(l_{1}l_{2}F_{X})_{X}F_{X}F_{Y}=(l_{1}l_{2})_{X}F_{Y}F_{X}^{2}-(l_{1}l_{2})_{X}F_{X}^{2}F_{Y}=0$

Condition 3. We have at $P_{5},\dots,P_{16}$:
$\dfrac{G_{YY}}{2}=l_{1}l_{2}F_{XY}^{2}$, $\dfrac{F_{YY}G_{Y}}{2F_{Y}}=0$ and
$A_{Y}=l_{1}l_{2}F_{XY}^{2}$. At $P_{1},\dots,P_{4}$:

$A_{Y}=\dfrac{(l_{1}l_{2})_{XY}F_{Y}F_{X}}{2}+\dfrac{(l_{1}l_{2})_{X}F_{YY}F_{X}}{2}+\dfrac{(l_{1}l_{2})_{YY}F_{X}^{2}}{2}+\dfrac{(l_{1}l_{2})_{X}F_{XY}F_{X}}{2}+$

$(l_{1}l_{2})_{Y}F_{XY}F_{X}+\dfrac{(l_{1}l_{2})_{Y}F_{Y}F_{XY}}{2}+\dfrac{(l_{1}l_{2})_{Y}F_{X}F_{XY}}{2}-F_{X}F_{Y}-\dfrac{(l_{1}l_{2})_{X}F_{XY}F_{Y}}{2}=$

$F_{X}^{2}+\dfrac{(l_{1}l_{2})_{X}F_{YY}F_{X}}{2}+2(l_{1}l_{2})_{Y}F_{X}F_{XY}$
and

$\dfrac{G_{YY}}{2}+\dfrac{F_{YY}G_{Y}}{2F_{Y}}=\dfrac{(l_{1}l_{2})_{YY}F_{X}^{2}}{2}+2(l_{1}l_{2})_{Y}F_{X}F_{XY}+\dfrac{(l_{1}l_{2})_{Y}F_{YY}F_{X}^{2}}{2F_{Y}}=$

$F_{X}^{2}+2(l_{1}l_{2})_{Y}F_{X}F_{XY}+\dfrac{(l_{1}l_{2})_{X}F_{X}F_{YY}}{2}$

Condition 4. At $P_{5},\dots,P_{16}$ we have
$B_{Y}-A_{X}=-2l_{1}l_{2}F_{XY}F_{XX}$ and
$-\dfrac{F_{XY}G_{Y}}{F_{Y}}-G_{XY}=-G_{XY}=-2l_{1}l_{2}F_{XY}F_{XX}$. As for
$P_{1},\dots,P_{4}:$

$B_{Y}-A_{X}=-(l_{1}l_{2})_{XY}F_{X}^{2}-(l_{1}l_{2})_{X}F_{XY}F_{X}-(l_{1}l_{2})_{Y}F_{X}F_{XX}-(l_{1}l_{2})_{X}F_{XY}F_{X}+$

$\dfrac{(l_{1}l_{2})_{XX}F_{Y}F_{X}}{2}+\dfrac{(l_{1}l_{2})_{X}F_{Y}F_{XX}}{2}-\dfrac{(l_{1}l_{2})_{XX}F_{Y}F_{X}}{2}-\dfrac{(l_{1}l_{2})_{X}F_{XY}F_{X}}{2}-\dfrac{(l_{1}l_{2})_{XY}F_{X}^{2}}{2}-\dfrac{(l_{1}l_{2})_{Y}F_{XX}F_{X}}{2}-(l_{1}l_{2})_{X}F_{X}F_{XY}-\dfrac{(l_{1}l_{2})_{X}F_{Y}F_{XX}}{2}-\dfrac{(l_{1}l_{2})_{Y}F_{X}F_{XX}}{2}+F_{X}^{2}+\dfrac{(l_{1}l_{2})_{X}F_{XY}F_{X}}{2}=-2F_{X}^{2}-3(l_{1}l_{2})_{X}F_{X}F_{XY}-2(l_{1}l_{2})_{Y}F_{X}F_{XX}$;
this is exactly $\dfrac{F_{XY}G_{Y}}{F_{Y}}-G_{XY}$

Condition 5. At $P_{5},\dots,P_{16}$ we have
$B_{X}=h^{\prime}F_{XX}=-l_{1}l_{2}F_{XX}^{2}=-\dfrac{G_{XX}}{2}=-\dfrac{G_{XX}}{2}-\dfrac{F_{XX}G_{Y}}{2F_{Y}}$

As for
$P_{1},\dots,P_{4}$:$B_{X}=-2F_{X}^{2}-2(l_{1}l_{2})_{X}F_{X}F_{XX}-(l_{1}l_{2})_{X}F_{X}F_{XX}+F_{X}^{2}+\dfrac{(l_{1}l_{2})_{X}F_{X}F_{XX}}{2}=-F_{X}^{2}-\dfrac{5(l_{1}l_{2})_{X}F_{X}F_{XX}}{2}$.

On the other hand,
$-\dfrac{G_{XX}}{2}-\dfrac{F_{XX}G_{Y}}{2F_{Y}}=-F_{X}^{2}-2(l_{1}l_{2})_{X}F_{X}F_{XX}-\dfrac{(l_{1}l_{2})_{Y}F_{X}F_{XX}}{2}=-F_{X}^{2}-\dfrac{5(l_{1}l_{2})_{X}F_{X}F_{XX}}{2}$.

Other examples

The example above fits into to a family where the points $P_{5},\dots,P_{16}$
are the zeros along $\\{F=0\\}$ of a cubic $C$ which satisfies
$C(P_{j})=F_{X}(P_{j}))$ for $j=1,2,3,4$. We put

$G=l_{1}l_{2}C^{2}$

$h=\dfrac{(l_{1}l_{2})_{Y}C}{2}+\dfrac{(l_{1}l_{2})_{X}F_{Y}}{2}+l_{1}l_{2}C_{Y}=\dfrac{(l_{1}l_{2})_{X}(C+F_{Y})}{2}+l_{1}l_{2}C_{Y}$

$k=-C-\dfrac{(l_{1}l_{2})_{X}C_{Y}}{2}$

$h^{\prime}=-\dfrac{(l_{1}l_{2})_{Y}C}{2}-\dfrac{(l_{1}l_{2})_{X}F_{X}}{2}-l_{1}l_{2}C_{X}=-\dfrac{(l_{1}l_{2})_{X}(C+F_{X})}{2}-l_{1}l_{2}C_{X}$

$k^{\prime}=C+\dfrac{(l_{1}l_{2})_{X}C_{X}}{2}$

$A=hF_{X}+kF$

$B=h^{\prime}F_{X}+k^{\prime}F$

We have the same Theorem as before; the computations for the proof are the
same. It is clear that if $C(P_{j})=\lambda F(P_{j})$ for $j=1,2,3,4$ and some
$\lambda\neq 0$ we may replace $C$ by $\dfrac{C}{\lambda}$.

We do not know if there are cubics (whose zeroes along $\\{F=0\\}$ are
different from $P_{1}\dots P_{4}$) for which the Theorem is untrue.

## References

  * [1] M. Brunella. _Birational Geometry of Foliations_ , Impa Monographs, Springer Verlag 2015
  * [2] P. Sad. _Regular Foliations along Curves_ , Annales de la Faculte des Sciences de Toulouse : Mathematiques, Serie 6, Tome 8 (1999) no. 4, pp. 661-675.
  * [3] B. Claudon, F. Loray,J.V. Pereira, F. Touzet. _Compact Leaves of Codimension one Holomorphic Foliations on Projective Manifolds_ , Annales Scientifiques de l’Ecole Normale Superieure (4) 51 (2018), 1457-1506.
  * [4] A. Neeman. Memoirs of the American Mathematical Society, 1989.
  * [5] T. Ueda. _On the neighborhood of a compact complex curve with topologically trivial normal bundle_ , J. Math. Kyoto Univ. 22 (1982/83), no. 4, 583-607.
  * [6] P.Sad. _Regular Foliations and Trace Divisors_ , Bulletin of the Brazilian Mathematical Society, New Series volume 53, 1033-1041 (2022).


The adjusted residualized outcomes can be re-written as a function of the
residualized outcomes and the fitted values from the regression. First, for
the treatment outcomes:

$\displaystyle{\rm
Var}_{w}(\hat{e}_{i}(1)-\tilde{\mathbf{X}}_{i}^{\top}\gamma_{\ast}^{res})$
$\displaystyle={\rm
Var}_{w}(Y_{i}(1)-\hat{g}(\mathbf{X}_{i})-\tilde{\mathbf{X}}_{i}^{\top}\gamma_{\ast}^{res})$
$\displaystyle={\rm Var}_{w}(Y_{i}(1)-\hat{g}(\mathbf{X}_{i}))+{\rm
Var}_{w}(\tilde{\mathbf{X}}_{i}^{\top}\gamma_{\ast}^{res})-2{\rm
Cov}_{w}(Y_{i}(1)-\hat{g}(\mathbf{X}_{i}),\tilde{\mathbf{X}}_{i}^{\top}\gamma_{\ast}^{res})$

Similarly,

$\displaystyle{\rm
Var}_{w}(\hat{e}_{i}(0)-\tilde{\mathbf{X}}_{i}^{\top}\gamma_{\ast}^{res})$
$\displaystyle={\rm
Var}_{w}(Y_{i}(0)-\hat{g}(\mathbf{X}_{i})-\tilde{\mathbf{X}}_{i}^{\top}\gamma_{\ast}^{res})$
$\displaystyle={\rm Var}_{w}(Y_{i}(0)-\hat{g}(\mathbf{X}_{i}))+{\rm
Var}_{w}(\tilde{\mathbf{X}}_{i}^{\top}\gamma_{\ast}^{res})-2{\rm
Cov}_{w}(Y_{i}(0)-\hat{g}(\mathbf{X}_{i}),\tilde{\mathbf{X}}_{i}^{\top}\gamma_{\ast}^{res})$

Plugging into Equation (A10):

$\displaystyle{\rm AVar}_{\tilde{F}}(\hat{\tau}_{wLS})-{\rm
AVar}_{\tilde{F}}(\hat{\tau}^{res}_{wLS})$ $\displaystyle\ =\ $
$\displaystyle\frac{1}{p}\left\\{{\rm
Var}_{w}(Y_{i}(1)-\tilde{\mathbf{X}}_{i}^{\top}\gamma_{\ast})-{\rm
Var}_{w}(Y_{i}(1)-\hat{g}(\mathbf{X}_{i}))\right\\}$ $\displaystyle\ +\
\frac{1}{1-p}\left\\{{\rm
Var}_{w}(Y_{i}(0)-\tilde{\mathbf{X}}_{i}^{\top}\gamma_{\ast})-{\rm
Var}_{w}(Y_{i}(0)-\hat{g}(\mathbf{X}_{i}))\right\\}$ $\displaystyle\ -\
\left\\{\frac{1}{p(1-p)}{\rm
Var}_{w}(\tilde{\mathbf{X}}_{i}^{\top}\gamma_{\ast}^{res})-\frac{2}{p}{\rm
Cov}_{w}(Y_{i}(1)-\hat{g}(\mathbf{X}_{i}),\tilde{\mathbf{X}}_{i}^{\top}\gamma_{\ast}^{res})-\frac{2}{1-p}{\rm
Cov}_{w}(Y_{i}(0)-\hat{g}(\mathbf{X}_{i}),\tilde{\mathbf{X}}_{i}^{\top}\gamma_{\ast}^{res})\right\\}$
$\displaystyle\ =\ $ $\displaystyle\frac{1}{p}\left\\{{\rm
Var}_{w}(Y_{i}(1)-\tilde{\mathbf{X}}_{i}^{\top}\gamma_{\ast})-{\rm
Var}_{w}(Y_{i}(1)-\hat{g}(\mathbf{X}_{i}))\right\\}+\frac{1}{1-p}\left\\{{\rm
Var}_{w}(Y_{i}(0)-\tilde{\mathbf{X}}_{i}^{\top}\gamma_{\ast})-{\rm
Var}_{w}(Y_{i}(0)-\hat{g}(\mathbf{X}_{i}))\right\\}$ $\displaystyle\ +\
\left\\{-\frac{1}{p(1-p)}{\rm
Var}_{w}(\tilde{\mathbf{X}}_{i}^{\top}\gamma_{\ast}^{res})+\frac{2}{p}{\rm
Cov}_{w}(\hat{e}_{i}(1),\tilde{\mathbf{X}}_{i}^{\top}\gamma_{\ast}^{res})+\frac{2}{1-p}{\rm
Cov}_{w}(\hat{e}_{i}(0),\tilde{\mathbf{X}}_{i}^{\top}\gamma_{\ast}^{res})\right\\}$

∎

##  Appendix B Diagnostic Measure

We detail how to estimate the diagnostic measures in this section. To estimate
the diagnostic for the post-residualized weighted estimator, we compute the
estimated weighted variance of both the residuals and the outcomes for the
units assigned to control:

$\displaystyle\hat{R}^{2}_{0}$ $\displaystyle=1-\frac{\widehat{{\rm
Var}}_{w,0}(\hat{e}_{i})}{\widehat{{\rm Var}}_{w,0}(Y_{i})}$
$\displaystyle=1-\frac{\sum_{i\in\mathcal{S}}w_{i}^{2}(1-T_{i})(\hat{e}_{i}-\hat{\mu}^{res}_{0})^{2}}{\sum_{i\in\mathcal{S}}w_{i}^{2}(1-T_{i})(Y_{i}-\hat{\mu}_{0})^{2}}$
(A11)

where $\hat{\mu}_{0}$ and $\hat{\mu}_{0}^{res}$ are defined as:

$\hat{\mu}_{0}=\frac{\sum_{i\in\mathcal{S}}w_{i}(1-T_{i})Y_{i}}{\sum_{i\in\mathcal{S}}w_{i}(1-T_{i})},\
\ \ \ \ \ \
\hat{\mu}^{res}_{0}=\frac{\sum_{i\in\mathcal{S}}w_{i}(1-T_{i})\hat{e}_{i}}{\sum_{i\in\mathcal{S}}w_{i}(1-T_{i})}$
(A12)

For the post-residualized weighted least squares estimator, estimating the
diagnostic follows similarly, but we now have to account for the covariate
adjustment taking place:

$\displaystyle\hat{R}^{2}_{0,wLS}$ $\displaystyle=1-\frac{\widehat{{\rm
Var}}_{w,0}(\hat{e}_{i}-\tilde{\mathbf{X}}_{i}^{\top}\hat{\gamma}_{0}^{res})}{\widehat{{\rm
Var}}_{w,0}(Y_{i}-\tilde{\mathbf{X}}_{i}^{\top}\hat{\gamma}_{0})}$
$\displaystyle=1-\frac{\sum_{i\in\mathcal{S}}w_{i}^{2}(1-T_{i})(\hat{\epsilon}^{res}_{i}-\hat{\epsilon}^{res}_{0})^{2}}{\sum_{i\in\mathcal{S}}w_{i}^{2}(1-T_{i})(\hat{\epsilon}_{i}-\hat{\epsilon}_{0})^{2}},$
(A13)

where $\hat{\epsilon}_{i}$ represents the residuals estimated from regressing
the outcomes $Y_{i}$ on the pre-treatment covariates $\tilde{\mathbf{X}}_{i}$,
across the subset of units assigned to control (i.e.,
$Y_{i}-\tilde{\mathbf{X}}_{i}^{\top}\hat{\gamma}_{0}$, where
$\hat{\gamma}_{0}$ is estimated by running the regression
$Y_{i}\sim\tilde{\mathbf{X}}_{i}$ across units assigned to control).
$\hat{\epsilon}_{i}^{res}$ is analogously defined for the residualized
outcomes $\hat{e}_{i}$. $\hat{\epsilon}_{0}$ and $\hat{\epsilon}_{0}^{res}$
are the weighted average of both $\hat{\epsilon}_{i}$ and
$\hat{\epsilon}_{i}^{res}$, respectively.

When treating $\hat{Y}_{i}$ as a covariate, the diagnostic can be estimated in
an analogous way, but by first performing sample splitting. More specifically,
the procedure for including $\hat{Y}_{i}$ as a covariate for the weighted
estimator is as follows:

  1. 1.

Across the subset of units assigned to control, randomly partition the units
into two subsets: $S_{1}$ and $S_{2}$. Without loss of generality, we will use
$S_{1}$ as our training sample, and $S_{2}$ as our testing sample.

  2. 2.

Regress $\hat{Y}_{i}$ on the outcomes across $S_{1}$ to obtain a $\hat{\beta}$
value.

  3. 3.

Using $\hat{\beta}$, estimate the out-of-sample residuals $\hat{e}_{i}^{oos}$
across $S_{2}$, where $\hat{e}_{i}^{oos}:=Y_{i}-\hat{\beta}\hat{Y}_{i}$.

  4. 4.

Estimate the diagnostic using $\hat{e}_{i}^{oos}$ and the outcomes $Y_{i}$
across $S_{2}$ using Equation (A11).

  5. 5.

Cross-fit: repeat steps 1-3, but flipping $S_{1}$ and $S_{2}$ (i.e., regress
$\hat{Y}_{i}$ on the outcomes across $S_{2}$ to obtain a $\hat{\beta}$ value,
and estimate the diagnostic across $S_{1}$).

  6. 6.

Average the two diagnostic values together.

When including $\hat{Y}_{i}$ as a covariate for the weighted least squares
estimator, researchers can repeat the procedure above; however, when
estimating the diagnostic using $\hat{e}_{i}^{oos}$, researchers must account
for $\tilde{\mathbf{X}}_{i}$. More specifically:

  1. 1.

Follow Steps 1-3 above to obtain $\hat{e}_{i}^{oos}$ across $S_{1}$.

  2. 2.

Regress $\hat{e}_{i}^{oos}$ on $\tilde{\mathbf{X}}_{i}$, and regress $Y_{i}$
on $\tilde{\mathbf{X}}_{i}$ across $S_{2}$. Use Equation (A13) to estimate the
diagnostic value.

  3. 3.

Cross fit, and average the two diagnostic values together.

When researchers have relatively small sample sizes, it can be advantageous to
perform repeated sample splitting, and take the average of the diagnostic
across all the repeated splits (see [50] for more details).

##  Appendix C Simulations

This section provides details associated with the simulations described in
Section 6 of the main manuscript.

###  C.1 Simulation Set-Up

To begin, we randomly generate a set of covariates
$\begin{bmatrix}X_{1}&X_{2}&X_{S}&X_{\tau}\end{bmatrix}\sim
MVN(\mathbf{0},\Sigma)$ with the following covariance structure:

$\Sigma=\begin{bmatrix}1&0&0.45&0.5\\\ 0&1&0&0\\\ 0.45&0&1&0.9\\\
0.5&0&0.9&1\end{bmatrix}$

where, recall, $(X_{1i},X_{2i})$ are observed pre-treatment covariates,
$X_{Si}$ controls the probability of inclusion in the experimental sample, and
$X_{\tau i}$ determines the treatment effect.

Unit $i$’s propensity for being included in the experimental sample (recorded
as $S_{i}=1$) is governed by a logit model on the covariate $X_{Si}$:

$P(S_{i}=1)\propto\frac{\exp(X_{Si})}{1+\exp(X_{Si})}.$

At each iteration of the simulation, an experimental sample is drawn using the
propensity score, as well as a random sample of the population. The sampled
population is used to estimate the residualizing model and sampling weights.

Each specific data generating process for the potential outcome under control
is determined by the values of the $\beta$s and $\gamma$s and $\alpha$. Below,
we provide the parameter values and simplified DGP for $Y_{i}(0)$.

  * •

Scenario 1: Linear Data Generating Process, identical population/sample DGP

$\beta_{1}=2,\beta_{2}=1,\beta_{3}=0,\beta_{S}=0,\gamma_{1}=0,\gamma_{2}=0,\gamma_{3}=0,\gamma_{4}=0,\alpha=0$,
yielding:

$Y_{i}(0)=2X_{1i}+X_{2i}+\varepsilon_{i}$

  * •

Scenario 2: Nonlinear Data Generating Process, identical population/sample DGP

$\beta_{1}=2,\beta_{2}=1,\beta_{3}=0,\beta_{S}=2.5,\gamma_{1}=0.5,\gamma_{2}=3,\gamma_{3}=2.5,\gamma_{4}=0,\alpha=0$,
yielding:

$Y_{i}(0)=2X_{1i}+X_{2i}+0.5X_{1i}^{2}+3\sqrt{|X_{2i}|}+2.5\big{(}X_{1i}\cdot
X_{2i}\big{)}+\varepsilon_{i}$

  * •

Scenario 3: Linear Data Generating Process, different population/sample DGP

$\beta_{1}=2,\beta_{2}=1,\beta_{3}=-1,\beta_{S}=\beta_{S},\gamma_{1}=0,\gamma_{2}=0,\gamma_{3}=0,\gamma_{4}=0,\alpha=0.5$,
yielding:

$\displaystyle Y_{i}(0)=$ $\displaystyle
2X_{1i}+X_{2i}+\beta_{S}\cdot(1-S_{i})\cdot(0.5-X_{1i})+\varepsilon_{i},$

  * •

Scenario 4: Nonlinear Data Generating Process, different population/sample DGP

$\beta_{1}=2,\beta_{2}=1,\beta_{3}=-1,\beta_{S}=\beta_{S},\gamma_{1}=0.5,\gamma_{2}=3,\gamma_{3}=2.5,\gamma_{4}=1.5,\alpha=0.5$,
yielding:

$\displaystyle Y_{i}(0)=$ $\displaystyle
2X_{1i}+X_{2i}+0.5X_{1i}^{2}+3\sqrt{|X_{2i}|}+2.5\big{(}X_{1i}\cdot X_{2i})$
$\displaystyle\beta_{S}\cdot(1-S_{i})\cdot(0.5-X_{1i}+1.5X_{1i}\cdot
X_{2i})+\varepsilon_{i},$

For Scenarios 3 and 4, $\beta_{S}$ takes on values $\\{-5,-2,-1,0,1,2,5\\}$.

###  C.2 Supplementary Tables

Table A1 presents summary results for estimator performance under Scenarios 1
and 2, including MSE, Bias, and SE. Column 1 presents the baseline results for
the difference-in-means (DiM). Columns 2-4 present the results for the
weighted estimators and columns 5-7 present results for the weighted least
squares estimator. For the weighted and weighted least squares estimators we
present the standard estimator without residualizing, the directly
residualized estimator and inclusion of $\hat{Y}$ as a covariate.

Summary of Estimator Performance (N=10,000)

|  |  | Weighted | Weighted Least Squares
---|---|---|---|---
|  | DiM | $\widehat{\tau}_{W}$ | $\widehat{\tau}^{res}_{W}$ | $\widehat{\tau}^{cov}_{W}$ | $\widehat{\tau}_{wLS}$ | $\widehat{\tau}^{res}_{wLS}$ | $\widehat{\tau}^{cov}_{wLS}$
Scenario 1: Linear Outcome Model |  |  |
n=100 | MSE | 36.44 | 30.05 | 1.48 | 1.34 | 1.34 | 1.34 | 1.30
| Bias | 3.60 | -0.13 | 0.05 | 0.12 | 0.19 | 0.19 | 0.27
| SE | 4.85 | 5.48 | 1.22 | 1.15 | 1.14 | 1.14 | 1.11
n=1000 | MSE | 16.41 | 2.98 | 0.17 | 0.15 | 0.14 | 0.14 | 0.13
| Bias | 3.74 | 0.00 | -0.01 | 0.00 | 0.00 | 0.00 | 0.01
| SE | 1.56 | 1.73 | 0.41 | 0.38 | 0.38 | 0.38 | 0.36
n=5000 | MSE | 14.39 | 0.64 | 0.04 | 0.03 | 0.03 | 0.03 | 0.03
| Bias | 3.72 | 0.01 | 0.00 | 0.00 | 0.01 | 0.01 | 0.01
| SE | 0.72 | 0.80 | 0.19 | 0.19 | 0.18 | 0.18 | 0.18
Scenario 2: Nonlinear Outcome Model |  |
n=100 | MSE | 70.71 | 58.80 | 8.25 | 8.20 | 36.59 | 8.16 | 8.04
| Bias | 3.44 | -0.30 | 0.09 | 0.14 | 0.04 | 0.23 | 0.26
| SE | 7.68 | 7.67 | 2.87 | 2.86 | 6.05 | 2.85 | 2.83
n=1000 | MSE | 20.37 | 5.58 | 0.82 | 0.80 | 3.53 | 0.79 | 0.78
| Bias | 3.78 | 0.05 | -0.00 | -0.00 | 0.05 | 0.00 | 0.01
| SE | 2.46 | 2.36 | 0.91 | 0.90 | 1.88 | 0.89 | 0.89
n=5000 | MSE | 14.80 | 1.17 | 0.18 | 0.18 | 0.83 | 0.17 | 0.17
| Bias | 3.68 | -0.02 | -0.01 | -0.01 | -0.03 | -0.01 | -0.00
| SE | 1.12 | 1.08 | 0.42 | 0.42 | 0.91 | 0.42 | 0.42

Table A1: Summary of estimator performance for Scenarios 1 and 2. The
population is fixed at $N=10,000$, and $1,000$ iterations were run for each
sample size. MSE is scaled by 100, and the bias and standard error are scaled
by 10.

Table A2 presents summary results for estimator performance under Scenarios 3
and 4, including MSE and Bias. In these scenarios we vary the value of
$\beta_{S}$, presented in column 1, which controls the degree of alignment
between the experimental sample outcomes and the population outcomes. We fix
the experimental sample size at $n=1,000$. Columns 2-3 presents the baseline
results for the difference-in-means (DiM). Columns 4-9 present the results for
the weighted estimators and columns 10-15 present results for the weighted
least squares estimator. For the weighted and weighted least squares
estimators we present the standard estimator without residualizing, the
directly residualized estimator and inclusion of $\hat{Y}$ as a covariate.

Summary of Estimator Performance - Scenario 3 and 4 (N = 10,000)

|  |  | Weighted | Weighted Least Squares
---|---|---|---|---
| DiM | $\widehat{\tau}_{W}$ | $\widehat{\tau}^{res}_{W}$ | $\widehat{\tau}^{cov}_{W}$ | $\widehat{\tau}_{wLS}$ | $\widehat{\tau}^{res}_{wLS}$ | $\widehat{\tau}^{cov}_{wLS}$
$\beta_{S}$ | MSE | Bias | MSE | Bias | MSE | Bias | MSE | Bias | MSE | Bias | MSE | Bias | MSE | Bias
Scenario 3: Linear Outcome |  |  |  |  |  |  |  |
-5 | 16.41 | 3.74 | 2.98 | 0.00 | 10.69 | -0.11 | 0.36 | -0.03 | 0.14 | 0.00 | 0.14 | 0.00 | 0.13 | 0.01
-2.5 | 15.83 | 3.67 | 3.07 | -0.06 | 2.55 | 0.06 | 0.25 | 0.01 | 0.14 | 0.02 | 0.14 | 0.02 | 0.13 | 0.04
-2 | 16.05 | 3.72 | 2.99 | 0.01 | 1.54 | 0.02 | 0.22 | 0.02 | 0.14 | 0.02 | 0.14 | 0.02 | 0.14 | 0.04
-1 | 16.11 | 3.73 | 2.88 | 0.05 | 0.39 | -0.02 | 0.16 | 0.00 | 0.14 | -0.00 | 0.14 | -0.00 | 0.13 | 0.02
-0.5 | 16.37 | 3.75 | 2.89 | 0.07 | 0.17 | -0.02 | 0.14 | -0.00 | 0.13 | 0.00 | 0.13 | 0.00 | 0.13 | 0.02
0 | 16.50 | 3.75 | 3.04 | 0.06 | 0.16 | 0.00 | 0.14 | 0.00 | 0.13 | 0.00 | 0.13 | 0.00 | 0.13 | 0.02
0.5 | 16.38 | 3.74 | 3.19 | 0.04 | 0.41 | 0.01 | 0.21 | 0.01 | 0.13 | 0.01 | 0.13 | 0.01 | 0.12 | 0.02
1 | 16.11 | 3.72 | 3.03 | 0.00 | 0.92 | 0.01 | 0.54 | 0.02 | 0.13 | -0.01 | 0.13 | -0.01 | 0.12 | 0.01
2 | 16.23 | 3.74 | 3.03 | 0.01 | 2.68 | 0.04 | 2.68 | 0.05 | 0.14 | -0.00 | 0.14 | -0.00 | 0.13 | 0.01
2.5 | 16.09 | 3.71 | 3.15 | -0.01 | 3.92 | 0.01 | 3.15 | -0.00 | 0.14 | -0.01 | 0.14 | -0.01 | 0.13 | 0.01
5 | 16.33 | 3.71 | 3.23 | 0.00 | 14.32 | -0.01 | 1.54 | 0.02 | 0.14 | -0.00 | 0.14 | -0.00 | 0.13 | 0.01
Scenario 4: Nonlinear Outcome |  |  |  |  |  |  |  |
-5 | 20.31 | 3.74 | 5.77 | 0.04 | 37.03 | -0.01 | 5.66 | 0.04 | 3.72 | 0.05 | 26.19 | 0.10 | 1.02 | 0.03
-2.5 | 20.31 | 3.74 | 6.17 | -0.01 | 9.55 | 0.10 | 5.10 | 0.04 | 3.96 | 0.05 | 7.57 | 0.06 | 1.67 | 0.04
-2 | 19.50 | 3.65 | 5.92 | -0.08 | 6.22 | -0.00 | 4.27 | -0.04 | 3.86 | -0.04 | 5.05 | -0.05 | 2.89 | -0.00
-1 | 19.77 | 3.73 | 5.71 | -0.02 | 2.18 | -0.08 | 2.14 | -0.07 | 3.91 | -0.09 | 1.92 | -0.05 | 1.08 | 0.02
-0.5 | 19.75 | 3.68 | 5.70 | -0.06 | 1.10 | -0.03 | 1.09 | -0.03 | 3.96 | -0.12 | 1.06 | -0.01 | 0.83 | 0.05
0 | 19.74 | 3.69 | 5.81 | -0.04 | 0.81 | 0.01 | 0.80 | 0.01 | 3.71 | -0.05 | 0.77 | 0.02 | 0.77 | 0.02
0.5 | 20.49 | 3.83 | 5.40 | 0.09 | 1.42 | 0.03 | 1.30 | 0.03 | 3.65 | 0.04 | 1.09 | 0.01 | 0.75 | 0.02
1 | 20.24 | 3.80 | 5.52 | 0.08 | 2.84 | -0.05 | 2.04 | -0.01 | 3.95 | 0.06 | 1.91 | -0.07 | 0.80 | -0.01
2 | 20.03 | 3.72 | 5.83 | 0.05 | 7.99 | -0.02 | 3.04 | 0.02 | 4.24 | 0.03 | 5.27 | -0.06 | 0.84 | -0.00
2.5 | 20.45 | 3.74 | 6.04 | 0.06 | 12.15 | -0.11 | 3.51 | -0.01 | 4.28 | 0.05 | 8.32 | -0.09 | 0.85 | -0.00
5 | 20.80 | 3.75 | 6.29 | 0.08 | 45.95 | -0.25 | 5.05 | 0.02 | 4.09 | 0.06 | 29.97 | -0.27 | 0.92 | -0.02

Table A2: Summary of estimator performance for Scenarios 3 and 4, where
$n=1,000$ and $N=10,000$. 1,000 iterations were run for each $\beta_{S}$
value. The bias is scaled by 10, and the MSE is scaled by 100.

In Table A3 we summarize the true positive and true negative rates for the
diagnostic measures for the post-residualized estimators.888True positive
rates were calculated by taking the total number of true positives (i.e.,
cases where the diagnostic correctly indicated there would be efficiency gain
from residualizing) and dividing by the total number of cases in which
residualizing led to efficiency gain. True negatives are similarly defined.
Column 1 presents the value of $\beta_{S}$. Columns 2-9 present the post-
residualized weighted, post-residualized weighted least squares, the post-
residualized weighted estimator with $\hat{Y}$ as a covariate, and the post-
residualized weighted least squares estimator with $\hat{Y}$ as a covariate,
respectively. We see that in general, the diagnostic measures are able to
adequately capture when residualizing results in precision gain. We see that
using sample splitting to estimate the pseudo-$R^{2}$ measure for the case in
which we include $\hat{Y}_{i}$ as a covariate can sometimes be conservative,
which results in a low true positive rate in cases when the divergence between
the experimental sample and population are rather large. In cases where
residualizing always leads to losses or gains in precision, the total number
of true positive or true negative rates is zero (respectively).

Diagnostic Performance across Simulations

| $\widehat{\tau}^{res}_{W}$ | $\widehat{\tau}^{cov}_{W}$ | $\widehat{\tau}^{res}_{wLS}$ | $\widehat{\tau}^{cov}_{wLS}$
---|---|---|---|---
$\beta_{S}$ | TPR | TNR | TPR | TNR | TPR | TNR | TPR | TNR
Scenario 3: Linear Outcomes |  |  |
-5 | 0/0 | 1000/1000 | 1000/1000 | 0/0 | 207/472 | 329/528 | 338/705 | 166/295
-2.5 | 1/942 | 58/58 | 1000/1000 | 0/0 | 203/499 | 304/501 | 308/694 | 177/306
-2 | 999/1000 | 0/0 | 1000/1000 | 0/0 | 216/514 | 288/486 | 310/689 | 175/311
-1 | 1000/1000 | 0/0 | 1000/1000 | 0/0 | 219/525 | 287/475 | 293/689 | 188/311
-0.5 | 1000/1000 | 0/0 | 1000/1000 | 0/0 | 214/519 | 282/481 | 293/689 | 183/311
0 | 1000/1000 | 0/0 | 1000/1000 | 0/0 | 223/523 | 275/477 | 283/683 | 177/317
0.5 | 1000/1000 | 0/0 | 1000/1000 | 0/0 | 222/536 | 268/464 | 260/666 | 194/334
1 | 1000/1000 | 0/0 | 1000/1000 | 0/0 | 233/519 | 283/481 | 254/669 | 199/331
2 | 999/1000 | 0/0 | 998/1000 | 0/0 | 228/490 | 321/510 | 297/695 | 175/305
2.5 | 0/0 | 999/1000 | 188/490 | 346/510 | 209/466 | 336/534 | 341/705 | 149/295
5 | 0/0 | 1000/1000 | 1000/1000 | 0/0 | 214/486 | 303/514 | 322/699 | 155/301
Scenario 4: Nonlinear Outcomes |  |  |
-5 | 0/0 | 1000/1000 | 360/718 | 224/282 | 0/0 | 1000/1000 | 58/1000 | 0/0
-2.5 | 0/0 | 998/1000 | 881/985 | 10/15 | 0/0 | 1000/1000 | 0/1000 | 0/0
-2 | 87/217 | 738/783 | 950/996 | 2/4 | 0/0 | 998/1000 | 0/994 | 5/6
-1 | 1000/1000 | 0/0 | 1000/1000 | 0/0 | 1000/1000 | 0/0 | 1000/1000 | 0/0
-0.5 | 1000/1000 | 0/0 | 1000/1000 | 0/0 | 1000/1000 | 0/0 | 1000/1000 | 0/0
0 | 1000/1000 | 0/0 | 1000/1000 | 0/0 | 1000/1000 | 0/0 | 1000/1000 | 0/0
0.5 | 1000/1000 | 0/0 | 1000/1000 | 0/0 | 1000/1000 | 0/0 | 1000/1000 | 0/0
1 | 1000/1000 | 0/0 | 1000/1000 | 0/0 | 999/1000 | 0/0 | 1000/1000 | 0/0
2 | 13/28 | 907/972 | 1000/1000 | 0/0 | 22/28 | 906/972 | 1000/1000 | 0/0
2.5 | 0/0 | 1000/1000 | 1000/1000 | 0/0 | 0/0 | 1000/1000 | 1000/1000 | 0/0
5 | 0/0 | 1000/1000 | 999/1000 | 0/0 | 0/0 | 1000/1000 | 1000/1000 | 0/0

Table A3: True positive rates (TPR) and true negative rates (TNR) for the
diagnostic measures.

Finally, in Table A4 we evaluate the 95% coverage rates for the proposed post-
residualized estimators. We see that in all scenarios, we achieve at least
nominal coverage. When the population and sample data generating processes
diverge significantly, we showed in the previous sections that there could be
a loss in efficiency from using post residualized weighting. However, coverage
rates are not affected by residualizing.

Coverage Rates

| Weighted | Weighted Least Squares
---|---|---
$\beta_{S}$ | $\widehat{\tau}_{W}$ | $\widehat{\tau}^{res}_{W}$ | $\widehat{\tau}^{cov}_{W}$ | $\widehat{\tau}_{wLS}$ | $\widehat{\tau}^{res}_{wLS}$ | $\widehat{\tau}^{cov}_{wLS}$
Scenario 3: Linear Outcome |
-5 | 0.95 | 0.95 | 0.97 | 0.99 | 0.99 | 0.99
-2.5 | 0.95 | 0.96 | 0.97 | 0.98 | 0.98 | 0.98
-2 | 0.95 | 0.97 | 0.98 | 0.97 | 0.97 | 0.98
-1 | 0.95 | 0.97 | 0.98 | 0.98 | 0.97 | 0.98
-0.5 | 0.95 | 0.98 | 0.99 | 0.98 | 0.98 | 0.98
0 | 0.95 | 0.99 | 0.98 | 0.99 | 0.99 | 0.99
0.5 | 0.95 | 0.97 | 0.98 | 0.99 | 0.99 | 0.99
1 | 0.96 | 0.95 | 0.95 | 0.98 | 0.98 | 0.98
2 | 0.95 | 0.94 | 0.94 | 0.98 | 0.98 | 0.98
2.5 | 0.94 | 0.94 | 0.94 | 0.98 | 0.98 | 0.98
5 | 0.94 | 0.94 | 0.95 | 0.98 | 0.98 | 0.99
Scenario 4: Nonlinear Outcome |
-5 | 0.95 | 0.96 | 0.95 | 0.96 | 0.96 | 0.96
-2.5 | 0.94 | 0.96 | 0.95 | 0.96 | 0.96 | 0.95
-2 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96
-1 | 0.96 | 0.97 | 0.97 | 0.95 | 0.96 | 0.95
-0.5 | 0.95 | 0.95 | 0.96 | 0.95 | 0.95 | 0.96
0 | 0.94 | 0.96 | 0.96 | 0.95 | 0.96 | 0.96
0.5 | 0.95 | 0.96 | 0.96 | 0.96 | 0.96 | 0.97
1 | 0.95 | 0.94 | 0.96 | 0.95 | 0.96 | 0.96
2 | 0.95 | 0.95 | 0.96 | 0.94 | 0.95 | 0.96
2.5 | 0.96 | 0.95 | 0.96 | 0.94 | 0.95 | 0.96
5 | 0.96 | 0.94 | 0.94 | 0.94 | 0.95 | 0.96

Table A4: 95% coverage rates of Normal approximation confidence intervals
across 1000 simulations.

##  Appendix D Additional Information for Empirical Application

As discussed in Section 7, we construct our target population using a leave-
one-out procedure. Table A5 provides a summary of the site specific and target
population average treatment effects. More specifically, the difference-in-
means (DiM) columns denote the experimental estimate in the specific site. The
target PATE is defined as the average difference-in-means estimate across the
other 15 sites. Standard errors are presented in parentheses. Certain sites,
such as MT (Butte, MT) contain only 38 experimental units, and the point
estimate of the experimental site DiM is vastly different from the target
PATE. Thus, we expect the task of generalizing to be more difficult for these
sites.

Summary of Experimental Sites and Target Population

|  |  |  |  | Earnings | Employment
---|---|---|---|---|---|---
|  | Expt. | Target Pop | Prob. of | (in $1000) | (Percentage)
Site | Location | Size ($n$) | Size ($N$) | Treatment | DiM | Target PATE | DiM | Target PATE
CC | Corpus Christi, TX | 524 | 5578 | 0.65 | -0.21 (1.16) | 1.37 (1.16) | -0.28 (3.2) | 1.8 (3.2)
CI | Cedar Rapids, IA | 190 | 5912 | 0.63 | 1.35 (1.89) | 1.24 (1.89) | -0.77 (5.07) | 1.71 (5.07)
CV | Coosa Valley, GA | 788 | 5314 | 0.66 | 1.63 (0.95) | 1.18 (0.95) | 5.95 (2.63) | 0.98 (2.63)
HF | Heartland, FL | 234 | 5868 | 0.73 | 0.95 (1.38) | 1.28 (1.38) | 6.8 (5.07) | 1.42 (5.07)
IN | Fort Wayne, IN | 1392 | 4710 | 0.67 | 1.73 (0.83) | 1.1 (0.83) | -0.4 (1.58) | 2.23 (1.58)
JC | Jersey City, NJ | 81 | 6021 | 0.64 | -0.53 (3.01) | 1.27 (3.01) | -2.39 (9.66) | 1.67 (9.66)
JK | Jackson, MO | 353 | 5749 | 0.67 | 2.16 (1.22) | 1.19 (1.22) | 5.66 (4.16) | 1.38 (4.16)
LC | Larimer County, CO | 485 | 5617 | 0.69 | 1.61 (1.32) | 1.21 (1.32) | -1.97 (3.24) | 1.93 (3.24)
MD | Decatur, IL | 177 | 5925 | 0.70 | 1.24 (2.5) | 1.23 (2.5) | 0.03 (5.24) | 1.67 (5.24)
MN | Northwest MN | 179 | 5923 | 0.67 | -1.43 (2.3) | 1.32 (2.3) | -0.52 (6.26) | 1.69 (6.26)
MT | Butte, MT | 38 | 6064 | 0.71 | -5.21 (4.1) | 1.27 (4.1) | -7.41 (5.14) | 1.67 (5.14)
NE | Omaha, NE | 636 | 5466 | 0.66 | 1.11 (0.98) | 1.25 (0.98) | -1.15 (2.56) | 1.98 (2.56)
OH | Marion, OH | 74 | 6028 | 0.70 | -2.99 (2.71) | 1.3 (2.71) | -6.82 (10.37) | 1.74 (10.37)
OK | Oakland, CA | 87 | 6015 | 0.64 | 1.83 (3.48) | 1.24 (3.48) | 3.34 (10.77) | 1.57 (10.77)
PR | Providence, RI | 463 | 5639 | 0.69 | 3.03 (1.34) | 1.12 (1.34) | 6.78 (4.58) | 1.34 (4.58)
SM | Springfield, MO | 401 | 5701 | 0.67 | 0.6 (1.31) | 1.29 (1.31) | 5.44 (3.34) | 1.36 (3.34)

Table A5: Summary of the JTPA study.

###  D.1 Estimating the Residualizing Model

Pre-treatment covariates were taken from the baseline survey conducted at the
beginning of the original JTPA experiment, to assess whether or not
individuals were eligible for JTPA services. A full list of the covariates
included in the residualizing model is provided in Table A6. In addition to
the pre-treatment covariates, we also include normalized measures of previous
earnings. Specifically, we include the $z$-score of an individual’s previous
earnings, relative to the experimental site, as well as the $z$-score of an
individual’s previous earnings, relative to the entire population.

Baseline Covariates included in Residualizing Models

Ethnicity |  | Weeks Worked† |  | Public Assistance History |  | Family Income† |
---|---|---|---|---|---|---|---
White |  | Zero |  | Food Stamps |  | Less than $3,000 |
Black |  | 1-26 weeks |  | Cash Welfare, other than AFDC |  | $3,000-$6,000 |
Hispanic |  | 27-52 weeks |  | Unemployment Benefits |  | More than $6,000 |
AAPI |  |  |  |  |  |  |
|  | Earnings |  | AFDC Histories |  | Accessibility |
Education |  | Previous Earnings‡ | * | Ever AFDC case head |  | Driver’s License |
ABE/ESL |  | Weekly Pay | * | Case head anytime† |  | Car available for regular use |
High school diploma |  | Quantile within Site | * | Received AFDC† |  | Telephone at home |
GED certificate |  |  $<$ 25% | * | Years as AFDC case head: |  |  |
Some college |  |  $>$ 50% | * | Less than 2 years |  | Household Composition |
Occupational Training |  |  $>$ 90% | * | 2-5 years |  | Marital Status |
Technical certificate |  | Quantile across Experiment | * | More than 5 years |  | Spouse present |
Job search assistance |  |  $<$ 25% | * |  |  | Household Size |
Years of Education‡ | * |  $>$ 50% | * | Age |  | Number of children present |
|  |  $>$ 90% | * | Age‡ | * | Child under 6 present |
Work History |  | Non-Zero Previous Earnings | * | Age Buckets |  |  |
Ever employed |  | UI Reported Earnings |  | 20-21 |  | Geographic Region |
Employed upon application |  |  |  | 22-29 |  | West | *
Total earnings† |  | Living in Public Housing |  | 30-44 |  | Midwest | *
Hourly earnings |  | Yes |  | 45-54 |  | South | *
Hours worked | * |  |  | 55 or older |  | North | *

Table A6: We provide a list of all of the covariates included in the Super
Learner. Many of these variables were included in the original JTPA study’s
regression model. Any variable denoted with an asterisk $(*)$ was not included
in the original JTPA study’s regression model. † indicates that the measure is
from the past 12 months prior to the baseline survey, ‡ indicates higher order
terms included of that variable.

###  D.2 Numerical Results for Empirical Application

Table A7 provides numerical results for the mean absolute error across all 16
experimental sites for the six different estimators. We note that the mean
absolute error of the point estimates do not vary substantially from using
post-residualized weighting. This supports the results in Section 7.2.1.

| Weighted Estimator |  | Weighted Least Squares
---|---|---|---
| $\widehat{\tau}_{W}$ | $\widehat{\tau}^{res}_{W}$ | $\widehat{\tau}^{cov}_{W}$ |  | $\widehat{\tau}_{wLS}$ | $\widehat{\tau}^{res}_{wLS}$ | $\widehat{\tau}^{cov}_{wLS}$
Earnings | 2.37 | 2.07 | 2.19 |  | 2.46 | 2.28 | 2.24
Employment ($\times$ 100) | 8.53 | 8.06 | 8.21 |  | 7.95 | 7.65 | 8.01

Table A7: Mean absolute error across sites.

Table A8 reports the estimated standard errors (columns 3-5 for weighted
estimators and columns 8-10 for weighted least squares estimators) for each
site, along with the estimated diagnostics (columns 6-7 for weighted
estimators and columns 11-12 for weighted least squares estimators). In
general, the diagnostics are able to adequately determine whether or not we
expect there to be improvements in standard error for accounting for the
population outcome information, as discussed in Section 7.2.2.

Standard Errors and Diagnostics for Residualizing Models for Residualizing
Models

|  | Weighted | Weighted Least Squares
---|---|---|---
Site | $n$ | $\widehat{\tau}_{W}$ | $\widehat{\tau}^{res}_{W}$ | $\widehat{\tau}^{cov}_{W}$ | $\hat{R}^{2}_{0}$ | $\hat{R}^{2}_{0,cov}$ | $\widehat{\tau}_{wLS}$ | $\widehat{\tau}^{res}_{wLS}$ | $\widehat{\tau}^{cov}_{wLS}$ | $\hat{R}_{0,wLS}^{2}$ | $\hat{R}^{2}_{0,wLS,cov}$
Outcome: Earnings
NE | 636 | 1.70 | 1.53 | 1.53 | 0.23 | 0.22 | 1.58 | 1.53 | 1.53 | 0.08 | 0.06
LC | 485 | 2.46 | 2.02 | 2.11 | 0.42 | 0.32 | 2.40 | 2.08 | 2.14 | 0.38 | 0.26
HF | 234 | 1.88 | 1.63 | 1.66 | 0.36 | 0.19 | 1.87 | 1.66 | 1.69 | 0.42 | 0.18
IN | 1392 | 1.03 | 0.93 | 0.92 | 0.25 | 0.26 | 1.00 | 0.91 | 0.91 | 0.22 | 0.21
CV | 788 | 1.40 | 1.25 | 1.22 | 0.04 | 0.01 | 1.36 | 1.23 | 1.20 | 0.08 | 0.07
CC | 524 | 2.51 | 2.52 | 2.48 | -0.06 | -0.18 | 2.42 | 2.42 | 2.39 | -0.13 | -0.23
JK | 353 | 2.29 | 2.28 | 2.25 | 0.19 | -0.25 | 2.19 | 2.18 | 2.16 | 0.30 | 0.10
MT | 38 | 6.44 | 7.04 | 8.40 | -0.36 | -9.31 | 4.64 | 4.83 | 6.09 | 0.40 | -3.90
PR | 463 | 2.69 | 2.61 | 2.60 | 0.08 | -0.16 | 2.82 | 2.75 | 2.71 | 0.03 | -0.17
MN | 179 | 4.79 | 4.70 | 4.80 | -0.03 | -0.35 | 3.72 | 4.26 | 4.20 | -0.31 | -0.56
MD | 177 | 2.87 | 2.46 | 2.48 | 0.33 | 0.24 | 2.67 | 2.30 | 2.32 | 0.30 | 0.13
SM | 401 | 2.07 | 2.28 | 2.12 | -0.30 | -0.13 | 2.13 | 2.23 | 2.11 | -0.14 | -0.09
OH | 74 | 3.97 | 3.27 | 3.42 | 0.33 | -0.22 | 3.94 | 3.75 | 3.77 | 0.29 | -0.37
CI | 190 | 3.84 | 3.33 | 3.07 | 0.41 | 0.31 | 3.47 | 3.15 | 2.94 | 0.28 | -0.18
OK | 87 | 4.69 | 5.07 | 4.64 | -0.05 | -0.43 | 4.61 | 4.39 | 4.22 | 0.14 | -0.19
JC | 81 | 7.24 | 8.81 | 8.50 | -0.75 | -1.15 | 6.14 | 7.51 | 6.56 | -0.19 | -0.49
Outcome: Employment
NE | 636 | 0.04 | 0.04 | 0.04 | 0.03 | -0.01 | 0.04 | 0.04 | 0.04 | 0.02 | -0.00
LC | 485 | 0.06 | 0.06 | 0.05 | 0.19 | 0.20 | 0.06 | 0.06 | 0.05 | 0.11 | 0.09
HF | 234 | 0.06 | 0.06 | 0.06 | 0.04 | -0.03 | 0.06 | 0.06 | 0.06 | 0.02 | -0.03
IN | 1392 | 0.02 | 0.02 | 0.02 | -0.15 | -0.04 | 0.02 | 0.02 | 0.02 | -0.21 | -0.08
CV | 788 | 0.03 | 0.03 | 0.03 | -0.01 | -0.01 | 0.03 | 0.03 | 0.03 | -0.03 | -0.01
CC | 524 | 0.06 | 0.06 | 0.06 | -0.09 | -0.10 | 0.06 | 0.06 | 0.06 | -0.10 | -0.08
JK | 353 | 0.10 | 0.09 | 0.09 | 0.13 | -1.53 | 0.09 | 0.09 | 0.09 | 0.08 | -0.85
MT | 38 | 0.13 | 0.13 | 0.13 | — | — | 0.13 | 0.15 | 0.14 | — | —
PR | 463 | 0.06 | 0.06 | 0.06 | 0.03 | -0.05 | 0.07 | 0.06 | 0.07 | 0.03 | -0.03
MN | 179 | 0.13 | 0.12 | 0.11 | 0.23 | -3.09 | 0.12 | 0.11 | 0.11 | 0.21 | -0.89
MD | 177 | 0.09 | 0.08 | 0.08 | 0.19 | -0.06 | 0.08 | 0.08 | 0.08 | 0.20 | -4.0e28
SM | 401 | 0.06 | 0.06 | 0.06 | 0.07 | -0.15 | 0.06 | 0.06 | 0.06 | 0.05 | -0.03
OH | 74 | 0.07 | 0.06 | 0.07 | 0.09 | -1.78 | 0.08 | 0.07 | 0.08 | 0.08 | -1.8e28
CI | 190 | 0.04 | 0.04 | 0.05 | 0.19 | -0.07 | 0.05 | 0.05 | 0.05 | 0.27 | -4.4e28
OK | 87 | 0.21 | 0.19 | 0.19 | 0.12 | -2.03 | 0.17 | 0.17 | 0.18 | 0.21 | -1.35
JC | 81 | 0.16 | 0.14 | 0.14 | -0.88 | -3.88 | 0.12 | 0.13 | 0.13 | -0.92 | -0.58

Table A8: Standard error and diagnostic values for post-residualized weighting
across the 16 experimental sites for two primary outcomes–earnings and
employment. The diagnostic values for the site of Butte, Montana (MT) are null
when outcome is employment, because all units in the control group were
unemployed.

Finally, Table A9 presents the true positive rate and false positive rate for
our diagnostics across the sites where the diagnostic indicated residualizing
would increase precision (or not). We present these counts for both outcomes,
separately.

| Weighted Estimator |  | Weighted Least Squares
---|---|---|---
| $\widehat{\tau}^{res}_{W}$ | $\widehat{\tau}^{cov}_{W}$ |  | $\widehat{\tau}^{res}_{wLS}$ | $\widehat{\tau}^{cov}_{wLS}$
Earnings |  |  |  |  |
True Positive Rate | 10/11 | 7/12 |  | 11/11 | 7/13
True Negative Rate | 5/5 | 4/4 |  | 4/5 | 3/3
Employment |  |  |  |  |
True Positive Rate | 11/13 | 1/12 |  | 10/10 | 1/7
True Negative Rate | 3/3 | 4/4 |  | 5/6 | 8/9
Table A9: Performance of proposed diagnostic measures, as measured through the
true positive rate and false positive rate.

###  D.3 Using Proxy Outcomes

To illustrate use of a proxy outcome, we run the same analysis as in Section
7, except we use employment as a proxy for earnings, and vice versa when
building the residualizing model. This mimics a situation in which we have
access to a related, but different outcome measure in our target population.
Because employment is binary while earnings are continuous, we expect that
direct residualizing may not result in substantial efficiency gains, and thus
that our diagnostic measures would indicate not to residualize. However,
treating $\hat{Y}_{i}$ as a covariate should still result in efficiency gain,
as earnings and employment are correlated and the model can adjust for the
scaling differences.

####  D.3.1 Bias

Table A10 presents the mean absolute error of the different estimation
methods. When earnings is the outcome, both directly residualizing and using
$\hat{Y}_{i}$ as a covariate result in relatively stable performance. However,
when employment is the outcome, the scaling differences between earnings (in
$\$1000$) and the binary employment measure lead to large residuals. We see a
loss to precision from direct residualizing, and exacerbated finite sample
bias. However, when including $\hat{Y}_{i}$ as a covariate, we are able to
account for the scaling differences, and the mean absolute error is lower.

Estimator Performance Summary with Proxy Outcomes
Weighted Weighted Least Squares $\widehat{\tau}_{W}$
$\widehat{\tau}^{res}_{W}$ $\widehat{\tau}^{cov}_{W}$ $\widehat{\tau}_{wLS}$
$\widehat{\tau}^{res}_{wLS}$ $\widehat{\tau}^{cov}_{wLS}$ Earnings 2.37 2.35
2.14 2.46 2.44 2.21 Employment ($\times$ 100) 8.53 66.15 7.85 7.95 65.33 7.45

Table A10: Mean absolute errors for each estimator, across all experimental
sites when using proxy outcomes.

####  D.3.2 Diagnostics

We estimate the same diagnostics as in Section 7.2.1 to determine when to
expect precision gains from performing post-residualized weighting. We
summarize the true positive and true negative rates of the diagnostic in Table
A11. We see that the performance of the diagnostic is good for direct
residualization. However, we see that the diagnostic for including
$\hat{Y}_{i}$ as a covariate is relatively conservative, and fails to identify
all the cases in which it is beneficial to residualize. However, the true
negative rate of the diagnostic for including $\hat{Y}_{i}$ as a covariate is
very high (almost 100%), which indicates that the diagnostic is very effective
at identifying when residualizing fails to lead to precision gain.

| Weighted Estimator |  | Weighted Least Squares
---|---|---|---
| $\widehat{\tau}^{res}_{W}$ | $\widehat{\tau}^{cov}_{W}$ |  | $\widehat{\tau}^{res}_{wLS}$ | $\widehat{\tau}^{cov}_{wLS}$
Earnings |  |  |  |  |
True Positive Rate | 13/14 | 7/12 |  | 12/13 | 6/13
True Negative Rate | 2/2 | 4/4 |  | 2/3 | 3/3
Employment |  |  |  |  |
True Positive Rate | – | 2/12 |  | – | 3/10
True Negative Rate | 16/16 | 4/4 |  | 16/16 | 5/6
Table A11: Performance of proposed diagnostic measures using proxy outcomes,
as measured through the true positive rate and false positive rate.

Table A12 provides the standard errors and diagnostic measures for each site
and estimator. Within the “Weighted” and “Weighted Least Squares” sections,
the left three columns present the standard error for the corresponding
estimator for each site, and the right two columns present the diagnostic
measure. One key takeaway is that, when employment is the outcome, using
earnings as a proxy outcome results in large scaling differences between our
residualizing model, captured by $\hat{Y}_{i}$, and the true outcome measure.
This is unsurprising since earnings is continuous and employment is binary. As
a result, the $\hat{R}^{2}_{0}$ measures for the estimators that use direct
residualizing (i.e., $\hat{\tau}_{W}^{res}$ and $\hat{\tau}_{wLS}^{res}$) are
all negative, indicating that we should not use direct residualizing in that
setting. However, even in this scenario, the diagnostic for using
$\hat{Y}_{i}$ as a covariate does not indicate significant gains. When using
employment as a proxy for earnings, the diagnostics indicate small gains to
direct residualizing across most sites, and gains from including $\hat{Y}_{i}$
as a covariate across about half of sites.

Standard Errors and Diagnostics for Residualizing Models with Proxy Outcomes

|  | Weighted | Weighted Least Squares
---|---|---|---
Site | $n$ | $\widehat{\tau}_{W}$ | $\widehat{\tau}^{res}_{W}$ | $\widehat{\tau}^{cov}_{W}$ | $\hat{R}^{2}_{0}$ | $\hat{R}^{2}_{0,cov}$ | $\widehat{\tau}_{wLS}$ | $\widehat{\tau}^{res}_{wLS}$ | $\widehat{\tau}^{cov}_{wLS}$ | $\hat{R}_{0,wLS}^{2}$ | $\hat{R}^{2}_{0,wLS,cov}$
Outcome: Earnings
NE | 636 | 1.70 | 1.70 | 1.62 | 0.00 | 0.09 | 1.58 | 1.58 | 1.53 | 0.00 | 0.03
LC | 485 | 2.46 | 2.45 | 2.39 | 0.00 | 0.15 | 2.40 | 2.40 | 2.37 | 0.00 | 0.05
HF | 234 | 1.88 | 1.88 | 1.76 | 0.01 | 0.15 | 1.87 | 1.86 | 1.78 | 0.01 | 0.08
IN | 1392 | 1.03 | 1.03 | 0.96 | 0.01 | 0.27 | 1.00 | 0.99 | 0.95 | 0.01 | 0.21
CV | 788 | 1.40 | 1.40 | 1.39 | 0.00 | -0.06 | 1.36 | 1.36 | 1.37 | 0.00 | -0.03
CC | 524 | 2.51 | 2.51 | 2.46 | 0.01 | -0.03 | 2.42 | 2.41 | 2.40 | 0.00 | -0.10
JK | 353 | 2.29 | 2.28 | 2.10 | 0.01 | 0.07 | 2.19 | 2.18 | 2.07 | 0.01 | 0.04
MT | 38 | 6.44 | 6.44 | 9.60 | -0.00 | -9.23 | 4.64 | 4.65 | 7.32 | 0.01 | -5.86
PR | 463 | 2.69 | 2.69 | 2.70 | 0.00 | -0.16 | 2.82 | 2.82 | 2.82 | -0.00 | -0.15
MN | 179 | 4.79 | 4.78 | 4.13 | 0.00 | 0.13 | 3.72 | 3.71 | 3.71 | 0.00 | -0.14
MD | 177 | 2.87 | 2.87 | 2.61 | 0.01 | 0.14 | 2.67 | 2.66 | 2.43 | 0.01 | 0.16
SM | 401 | 2.07 | 2.07 | 2.04 | 0.00 | -0.07 | 2.13 | 2.12 | 2.06 | 0.00 | -0.02
OH | 74 | 3.97 | 3.97 | 4.00 | 0.00 | -0.44 | 3.94 | 3.93 | 3.75 | 0.00 | -0.50
CI | 190 | 3.84 | 3.84 | 3.40 | 0.00 | -0.03 | 3.47 | 3.47 | 3.07 | 0.00 | -0.22
OK | 87 | 4.69 | 4.71 | 4.51 | -0.01 | -0.88 | 4.61 | 4.61 | 4.06 | -0.01 | -0.67
JC | 81 | 7.24 | 7.26 | 8.52 | -0.01 | -0.82 | 6.14 | 6.17 | 6.75 | -0.01 | -0.83
Outcome: Employment
NE | 636 | 0.04 | 0.56 | 0.04 | -352.40 | -0.00 | 0.04 | 0.49 | 0.04 | -248.43 | -0.01
LC | 485 | 0.06 | 0.70 | 0.05 | -220.79 | 0.13 | 0.06 | 0.56 | 0.05 | -193.43 | 0.02
HF | 234 | 0.06 | 0.94 | 0.06 | -260.76 | -0.02 | 0.06 | 0.90 | 0.06 | -282.80 | 0.06
IN | 1392 | 0.02 | 0.34 | 0.02 | -391.67 | -0.04 | 0.02 | 0.32 | 0.02 | -354.59 | -0.05
CV | 788 | 0.03 | 0.42 | 0.03 | -151.95 | 0.02 | 0.03 | 0.38 | 0.03 | -129.68 | 0.03
CC | 524 | 0.06 | 1.00 | 0.06 | -284.99 | -0.21 | 0.06 | 0.89 | 0.06 | -236.05 | -0.18
JK | 353 | 0.10 | 1.10 | 0.08 | -104.99 | -3.17 | 0.09 | 0.92 | 0.08 | -88.67 | -2.13
MT | 38 | 0.13 | 2.42 | 0.12 | — | — | 0.13 | 2.49 | 0.13 | — | —
PR | 463 | 0.06 | 0.93 | 0.06 | -228.05 | -0.05 | 0.07 | 0.80 | 0.07 | -200.67 | -0.06
MN | 179 | 0.13 | 1.57 | 0.13 | -207.13 | -14.37 | 0.12 | 1.61 | 0.12 | -189.16 | -3.35
MD | 177 | 0.09 | 0.93 | 0.08 | -66.00 | -0.14 | 0.08 | 0.81 | 0.08 | -77.66 | -4.7e28
SM | 401 | 0.06 | 0.76 | 0.06 | -95.56 | -0.12 | 0.06 | 0.68 | 0.06 | -84.67 | -0.10
OH | 74 | 0.07 | 1.77 | 0.07 | -1202.77 | -0.56 | 0.08 | 1.60 | 0.08 | -985.02 | -2.4e28
CI | 190 | 0.04 | 1.40 | 0.05 | -1312.75 | -0.47 | 0.05 | 1.41 | 0.05 | -1241.70 | -1.1e28
OK | 87 | 0.21 | 3.24 | 0.16 | -249.20 | -1.29 | 0.17 | 2.44 | 0.16 | -65.65 | -0.28
JC | 81 | 0.16 | 3.20 | 0.17 | -6487.60 | -4.51 | 0.12 | 1.70 | 0.14 | -300.75 | -0.24

Table A12: Standard error and diagnostic values for post-residualized
weighting using proxy outcomes across the 16 experimental sites for two
primary outcomes–earnings and employment. Once again, the diagnostics for MT
are null when employment is the outcome, because all the units in the control
group are unemployed.

####  D.3.3 Efficiency Gain

Table A12 presents the standard errors of each weighting method, with and
without post-residualizing, for each site. Table A13 presents the average
standard error across sites for post-residualized weighting using proxy
outcomes, where we restrict our attention to the sites identified by the
diagnostic measures for when we expect precision gains. When using employment
as a proxy for earnings, direct residualizing indicates small gains in 13/16
sites, and including $\hat{Y}_{i}$ as a covariate indicates gains in just
under half of sites. The relative improvement in variance is small due to the
differences in magnitude between $\hat{Y}_{i}$ and $Y_{i}$. In particular, we
see around a 0.3-0.4% reduction in variance from performing direct
residualizing. However, when including $\hat{Y}_{i}$ as a covariate, which
accounts for the scaling difference, the improvements are more substantial. In
particular, when using $\hat{Y}_{i}$ as a covariate in the weighted estimator,
there is a 14% reduction in variance. Using weighted least squares, there is a
9% reduction in variance from including $\hat{Y}_{i}$ as a covariate. The
primary takeaway to highlight is that using $\hat{Y}_{i}$ as a covariate to
perform post-residualized weighting can allow us to leverage proxy outcomes
that exist on different scales than the outcome of interest, where we expect
greater gains the more closely related the outcome and proxy outcome are.

For employment, we do not consider direct residualizing because the diagnostic
measure did not identify any experimental sites in which directly
residualizing would lead to precision gains. When including $\hat{Y}_{i}$ as a
covariate the diagnostic indicated 5 sites that indicate gains from post-
residualized weighting; among these we see a 5% reduction in variance when
using $\hat{Y}_{i}$ as a covariate in the weighted estimator, and a 1%
reduction in variance in the weighted least squares estimator. Finally, we
emphasize that estimating the PATE results in variance inflation relative to
the within-sample difference-in-means, as expected. However, we see that post-
residualized weighting can offset some of this loss in precision.

This exercise shows how a proxy outcome can be used for building the
residualizing model. When the two variables are on very different scales, we
expect that direct residualizing would not be beneficial, as evidenced here
and captured by our diagnostic measures. Including $\hat{Y}_{i}$ as a
covariate addresses scaling concerns, although as we see when using earnings
as a proxy for employment, does not always allow for gains. We see that even
using proxy outcomes, our diagnostic measures can accurately capture when
there is potential for precision gains, and our post-residualized weighting
method can lead to precision gains in estimation of the target PATE.

Summary of Standard Errors across Experimental Sites Subset by Diagnostic,
using Proxy Outcomes
Earnings Employment Number of Sites DiM Standard Post Resid. Weighting Number
of Sites DiM Standard Post Resid. Weighting Weighted Direct Residualizing 13
1.53 2.58 2.57 0 – – – $\hat{Y}_{i}$ as Covariate 7 1.50 2.43 2.23 2 2.93 4.01
4.00 Weighted Least Squares Direct Residualizing 13 1.74 2.57 2.57 0 – – –
$\hat{Y}_{i}$ as Covariate 6 1.37 1.95 1.85 3 3.65 4.68 4.63

Table A13: Summary of standard errors across the 16 experimental sites
identified by the diagnostic measures.


# The Bid Picture: Auction-Inspired Multi-player Generative Adversarial
Networks Training

Joo Yong Shim1111Contact Author Jean Seong Bjorn Choe1&Jong-Kook Kim1
1Korea University, Seoul, 02841, Korea
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

This article proposes auction-inspired multi-player generative adversarial
networks training, which mitigates the mode collapse problem of GANs. Mode
collapse occurs when an over-fitted generator generates a limited range of
samples, often concentrating on a small subset of the data distribution.
Despite the restricted diversity of generated samples, the discriminator can
still be deceived into distinguishing these samples as real samples from the
actual distribution. In the absence of external standards, a model cannot
recognize its failure during the training phase. We extend the two-player game
of generative adversarial networks to the multi-player game. During the
training, the values of each model are determined by the bids submitted by
other players in an auction-like process.

##  1 Introduction

The explosive popularity of generative tasks and applications has led to
remarkable progress in generative model research DALLE ; diffwave ; GALIP ;
alqahtani2021applications . However, there still exists a problem called the
generative learning trilemma trilemma . The generative learning trilemma is an
issue in which the generative learning framework cannot meet the three
essential requirements of a successful generative model at once: (i) High-
Quality Samples, (ii) Fast Sampling, (iii) Mode Coverage / Diversity.
Generative Adversarial Networks (GANs) goodfellow  has mode collapse problem,
producing only a small subset of data or the same samples repeatedly
salimans2016improved ; arjovsky2017towards . This research addresses the
generative learning trilemma by focusing on the problem of GANs, specifically
the mode coverage issue. Specifically, it extends the two-player min-max game
of GANs to the multi-player games of multiple GANs and introduces an auction-
inspired auxiliary training algorithm. Note that a preliminary version of the
proposed method in this paper was published in our previous paper ASK2023 .

GANs are usually defined as a two-player min-max game of two neural networks:
a discriminator and a generator goodfellow . These two networks are trained in
an adversarial way, where the objective function is defined as:

$\begin{split}\min_{G}\max_{D}V(D,G)&=\mathbb{E}_{x\sim p_{data}}\left[\log
D\left(x\right)\right]\\\ &+\mathbb{E}_{z\sim
p_{z}}\left[\log(1-D\left(G(z\right))\right]\end{split}$ (1)

, where $x\sim p_{data}$ is an image sample from the real data distribution,
and $z\sim p_{z}$ is a prior input noise, which is generated randomly. Here,
the generator $G$ tries to minimize the objective function (1), generating
realistic samples which are indistinguishable from the discriminator $D$.
Whereas the discriminator $D$ tries to maximize the objective function,
identifying the fake samples generated by the generator $G$ from the real
data. In the optimal scenario, it reaches the equilibrium point at the global
optimal state, where $p_{g}=p_{data}$. In this state, the generator $G$
generates perfect samples and the discriminator $D$ is unable to distinguish
the real and generated distribution, i.e. $D(x)=\frac{1}{2}$.

The mode collapse problem usually occurs when the generator $G$ captures only
a few modes of data distribution that can easily deceive the discriminator
$D$. In this state, the generator $G$ captures only a portion of the actual
data distribution and produces a deceptive image, which the discriminator $D$
cannot distinguish from the real data, resulting in the illusion that the
optimal situation has been reached. In this state, neither the generator $G$
nor the discriminator $D$ can be improved.

The idea starts from the question: Is it possible to detect mode collapse
during training, particularly whether the generator is misleading the
discriminator by generating only a limited set of modes from the dataset? In
the classical training process of GANs, it is difficult for a single model to
realize mode collapse during the training phase. A single discriminator is the
only way in which the samples of the generator can be evaluated. Likewise, the
fake samples that the discriminator can source are solely those produced by
its associated generator. This prompts the following question: Wouldn’t it be
helpful to use appropriate external references to provide hints about the
performance of the generator and the discriminator? How about extending the
two-player game to the multi-player game and giving them some relative values
to each other?

After all these questions, this work introduces the auction-inspired multi-
player generative adversarial networks training process. This method
simultaneously trains multiple GANs by using an algorithm that includes an
auction-like evaluation process and auxiliary training update. In the
traditional GANs training process, the discriminator tries to approximate the
distance between the real distribution of the data and the sample distribution
to evaluate the generator’s performance. The usual distance metric is Jensen-
Shannon divergence, which is minimized when the real samples and the fake
samples are completely indistinguishable. However, this distance cannot be
applied when evaluating generators from the other GANs. Thus, a new score
metric is defined to evaluate external GANs. The concept of an auction is
adopted in this scoring process, to give a proper valuation of multiple GANs
since the auctions are usually used when there is uncertainty between sellers
and buyers in the valuation of itemsauction . The best GANs are then selected
in each step using this new score metric. An additional training process is
followed, where each discriminator tries to fit its output values to the best
discriminator’s output. It is assumed that the measurement of the best
discriminator is more accurate than the others and it will give the proper
reference value during the training phase, guiding them to be on the right
track.

The main contributions of the proposed method can be summarized as follows:

  * •

This work provides a novel idea for training GANs to solve the mode collapse
problem by extending a two-player game to a multi-player game. Multiple GANs
are trained effectively by adopting an auction-like evaluation approach.

  * •

The additional auxiliary training process is proposed to properly guide the
training process of multiple GANs and prevent them from falling into failure
modes.

  * •

The impacts of training multi-GANs by using the proposed algorithm are
examined qualitatively and quantitatively.

##  2 Related Works

There have been a lot of efforts to solve the mode collapse problem of GANs.
Some of the works tried to mitigate the mode collapse in GANs by improving the
learning process of GANs unrolled ; wgan ; wgan_gp ; lsgan  or by enforcing
GANs to cover diverse modes infogan ; modegan . Unrolled GANsunrolled
proposed an enhanced training process for GANs to reduce mode collapse. It
examines how the discriminator will be affected in the next $k$ steps to the
current update of the generator and lessens the chance of the generator
overfitting to a specific discriminator, which as a result, allows generators
to cover diverse modes of data distribution. WGAN wgan  and LSGAN lsgan  are
other notable models which reduced the mode collapse problem. They introduced
different divergence metrics to stabilize the training of GANs and generate
better samples. InfoGAN infogan  tried to allay the diversity issue by
enforcing GANs to cover diverse modes. Unlike the classical GANs model which
uses single input latent variable z, it adds latent code $c$ to the input as
$(z,c)$. Then, it extends information theoretic regulation, which tries to
increase the Mutual Information (MI) between the code $c$ added in the latent
space and the generated sample. This helps to learn disentangled
presentations, preventing mode collapse. ModeGAN modegan , another trial to
enforce diverse modes of GANs, trains generators in collaboration using
encoders. It supports sample diversity by using the fact that the fake sample
generated by the generator is likely to belong to the same mode when the real
sample passes via the encoder-decoder. This work belongs to both approaches,
in which it tries to improve the learning process of GANs by introducing an
additional auxiliary training process and enforcing diverse modes by
referencing multiple generators and discriminators.

This work is not the first attempt to extend the two-player game to the multi-
player game mohebbi2023games . There were previous researches that used
multiple generators or discriminators ghosh2018multi ; hoang2018mgan ;
nguyen2017dual ; mclgan ; albuquerque19a . MAD-GAN ghosh2018multi  uses
multiple generators that capture different modes and the discriminator is
trained to identify whether the samples are real or fake and find which
generator has generated the fake sample. D2GAN nguyen2017dual  extended to a
three-player game of two discriminators and one generator. D2GAN uses one more
discriminator that is trained by using reverse KL distance, learning to judge
fake data as real. It allows parameters to adjust the quality and diversity of
the generated images by reducing the KL distance and reverse KL distance by
minimizing the loss function. MCL-GAN mclgan  also uses multiple
discriminators where each sample is assigned to best-suited discriminators.
This makes each discriminator to be the expert model for assigned samples and
helps to mitigate the mode collapse problem. This work uses multiple GANs to
improve mode coverage and diversity performance, but it is different in that
the proposed algorithm can be easily applied without any special structural
changes, and multiple models evaluate values of each other.

Algorithm 1 Overall Algorithm

0: Pairs of $N$ GANs $\mathcal{G}=\\{(G_{1},D_{1}),\dots,(G_{N},D_{N})\\}$, a
dataset $\mathcal{D}$ and a hyperparameter $\lambda$.

1: for number of training iteration epochs do

2: for all $G,D\in\mathcal{G}$ do

3: individual training

4: end for

5: do Auction and select the $BestGAN(G^{*},D^{*})$

6: for all $G,D\in\mathcal{G}$ do

7: for Minibatch of data $x$ from $\mathcal{D}$ do

8: Draw a batch of samples $x^{\prime}$ from $G$.

9: Calculate the discriminator loss $L_{D}$ from $x$ and $x^{\prime}$.

10: Calculate the auxiliary loss $L_{\text{aux}}$ from $L_{D}$, $x$,
$x^{\prime}$ and $D^{*}$.

11: Update $D$ to minimize $\mathcal{L}_{D}=L_{D}+\lambda L_{D^{*}}$.

12: Update $G$ using $D$.

13: end for

14: end for

15: end for

##  3 Methods

A set of $N$ pairs of GANs consists of a generator (G) and a discriminator
(D), $\\{(G_{1},D_{1}),(G_{2},D_{2}),\dots,(G_{N},D_{N})\\}$ is considered.
Each pair of GANs is trained to maximize performance by an auction-based two-
stage update algorithm. The overall algorithm is described in Algorithm 1.

The first-step update is conducted independently for each model, a process
termed “individual training”. This update process is applied across all pairs
of GANs in the same way as the standard GANs training. For instance, when
using vanilla GANs, each GANs is updated based on the objective function Eq.
(1). Note that various types of GANs can be applied by simply employing the
training method and objective function for selected GANs models or algorithms.

The second-step update is “auxiliary training”, which uses the external value
acquired from the best GANs model. The best GANs pair is selected using the
auction-inspired valuation process. It is expected the performance of the GANs
can be calibrated during auxiliary training, by calculating the relative loss
that refers to the outputs of the best GANs.

###  3.1 Auction-Inspired Valuation Process

In the auction, all generators become auctioneers and present a set of
generated images as a lot for the auction, and all discriminators become
participants and submit bids for images.

Auctions are held for each generator (auctioneer), thus a total of $N$
auctions will be held. In each auction phase, all discriminators
(participants), except their paired discriminator, produce a numerical value
determining the authenticity of the generated images, and these values are
employed as bidding values. The winner of each auction could be determined by
using winner selection methods; nevertheless, the winner selection step is
excluded as the main objective of this auction process is to attain accurate
valuation for the generated images. An example of this procedure in the case
of three pairs of GANs is illustrated in Figure.1 and Figure.2.

Figure 1: Example auction framework Figure 2: Example single auction phase

The auction phase can be described as follows:

  1. 1.

The auctioneer $i$ presents a lot $L^{i}$, which is the group of $K$ items
(samples) generated by the generator $G_{i}$

$\displaystyle L^{i}$ $\displaystyle=\\{I_{1}^{i},I_{2}^{i},…,I_{k}^{i}\\}$
$\displaystyle I_{k}^{i}$ $\displaystyle\leftarrow G_{i}(z_{k})$

Figure 3: Auction process 1

  2. 2.

Each bidder $j$ ($j=1,….,N$, $j\neq i$) is asked to evaluate $K$ items from
auctioneer $i$. The value of the item is the output of the discriminator
$D_{j}$. The final bid $B^{ij}$ is the mean of values for all items.

$\displaystyle B^{ij}$
$\displaystyle=\frac{1}{K}(B_{1}^{ij},B_{2}^{ij},…,B_{k}^{ij})$ $\displaystyle
B_{k}^{ij}$ $\displaystyle\leftarrow D_{j}(I_{k}^{i})$

Figure 4: Auction process 2

###  3.2 Best GANs Selection

Based on the auction result, the score of the GANs $i=(G_{i},D_{i})$ is
determined by using the score function $S(i)$. This scoring involves
calculating the mean difference of the bid prices received from all other
participants for images generated by $G_{i}$, and the bid price submitted by
$D_{i}$ for images presented by all other auctioneers. The score $S(i)$ is
formulated as follows:

$S(i)=\frac{1}{N-1}\sum_{\begin{subarray}{c}j=1\\\ j\neq
i\end{subarray}}^{N}B^{ij}-\frac{1}{N-1}\sum_{\begin{subarray}{c}j=1\\\ j\neq
i\end{subarray}}^{N}B^{ji}$ (2)

Here, $\sum B^{ij}$ is the sum of the values evaluated by all other
discriminators $D_{j}(j\in{1,….,N},j\neq i)$ for the images generated by
$G_{i}$. This represents the sum of the probability values that discriminators
of other GANs thought that the images generated by $G_{i}$ are from the real.
Images generated by relatively poor GANs are more likely to be determined as
fake images, that is, the value of $D_{j}(I_{k}^{i})$ is more likely to be
low, thus resulting in lower $B^{ij}$ values. $\sum B^{ji}$ is the sum of the
values that discriminator $D_{i}$ evaluated for the images generated by all
other generators $G_{j}(j\in{1,….,N},j\neq i)$. Similarly, this means the sum
of probability values that $D_{i}$ thought as the real for the images
generated by other generators. If the GANs $i$ has relatively poor
performance, it is highly likely that images generated by other GANs
$I_{k}^{i}\leftarrow G_{i}(z_{k})$ will be judged as relatively good images,
that is, $D_{i}(I_{k}^{j})$ is more likely to be high. Therefore, the worse
the GANs, the higher the value of $B^{ji}$ will be measured. Taken together,
the score is designed as the above formula $S(i)$, and the better the
performance, the higher the score.

###  3.3 Auxiliary Loss

The auxiliary loss is calculated using the determined best GANs
$(G^{*},D^{*})$. The key idea is to match the loss output of a discriminator
to the loss of the best discriminator $D^{*}$. Therefore, the auxiliary loss
$L_{D^{*}}$ is defined as follows:

$\displaystyle
L_{\text{aux}}=\text{MSE}(L_{D}(X,\bar{X}),L_{D^{*}}(X,\bar{X})),$ (3)

where $L_{D}$ is the loss function of discriminator $D$.

Finally, optimize the objective augmented using the auxiliary loss:

$\mathcal{L}_{D}=L_{D}+\lambda L_{\text{aux}},$ (4)

where $\lambda$ is a hyperparameter to be determined.

##  4 Evaluations

This section presents the analysis of the proposed algorithm applied to two
classical GANs models: vanilla GANs goodfellow  and WGAN wgan . The algorithm
is demonstrated using a synthetic mixture of Gaussian distributions comprised
of eight modes. A simple 2D-Gaussian dataset is used for the experiment
because mode collapse is more evident in such a simple dataset. Specifically,
eight GANs were trained using and without using the proposed algorithm in the
same environment. For all experiments, ReLU networks containing two hidden
layers each of width 256, are applied for all generators and discriminators.
Additionally, 2-dimensional latent variables are used in all the experiments.

###  4.1 Qualitative Evaluations

GANs trained using the proposed method cover various points without falling
into mode collapse, while GANs simply trained in the standard way are often
biased to one side or miss data points in some areas (see Figure
LABEL:fig:Bid_result_1). In the classical training, some of the GANs showed
biased results or holes in the data distribution, whereas all modes were
covered where the proposed training method was applied. Although not as
distinctly different as the results in GANs model, experiments on WGANs also
show progressive results. WGANs trained using the proposed method covered
various modes of data distribution in a balanced and sparse manner, while some
samples were leaning on particular sides in WGANs where the proposed method
was not applied (see Figure LABEL:fig:Bid_result_2).

###  4.2 Quantitative Evaluations

Quantitative metrics are implemented to assess both the quality and the mode
coverage of the generated samples. To evaluate quality, the mean likelihood
estimation for modes is used. For the mode coverage assessment, the
Wasserstein distance between the sample mode distribution and a uniform
distribution is measured. In Figure LABEL:fig:result_sparsity_gan. and Figure
LABEL:fig:result_sparsity_wgan., the mode coverage is compared quantitatively.
For GANs models, it can be seen that the application of the proposed method is
better at mode coverage. The higher the value, the higher the degree of spread
of samples, which means there is less mode overfitting occurring. The
differences between WGANs are not as clear as those of GANs, but there was
slight progress in mode coverage.

For the performance evaluations, the likelihood metric was used and there was
an even slight improvement. All values in Table 1 and 2 are the mean values
obtained from 10 experiments. The mean refers to the mean likelihood of eight
GANs, and the min refers to the worst performance result among eight GANs. The
performance of the worst has improved particularly, which is expected to be
due to the auxiliary training that refers to the best GANs, resulting in
preventing it from going to failure mode.

Table 1: Likelihood Evaluations in GANs
| classic | proposed
---|---|---
mean likelihood | $-3.530\pm 0.556$ | $-3.235\pm 0.577$
min likelihood | $-6.414\pm 1.528$ | $-4.995\pm 1.120$
Table 2: Likelihood Evaluations in WGAN
| classic | proposed
---|---|---
mean likelihood | $-0.651\pm 0.831$ | $-0.478\pm 0.571$
min likelihood | $-4.756\pm 5.525$ | $-2.691\pm 1.503$

##  5 Conclusion

In conclusion, this work introduces a novel approach to tackle the mode
collapse problem in GANs by extending the traditional two-player game into a
multi-player game. This framework effectively trains multiple GANs, offering a
solution to the mode collapse problem. One notable contribution of this work
is the introduction of an auction-like evaluation approach, which assists in
selecting and valuing the best-performing GANs. This approach guides the
training process effectively and helps prevent failure modes. Additionally, an
auxiliary training process is proposed, providing proper guidance for training
multiple GANs. While the proposed method has successfully mitigated the mode
collapse and instability problem, it does require additional computational
efforts. Future work will focus on improving performance and implementing the
approach using other GANs models. Exploring better bid functions or
alternative auction mechanisms may prove beneficial. Further investigation
into updating using relative loss is also needed. All these limitations are
left for future research.

## References

  * (1) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever, “Zero-shot text-to-image generation,” in _Proceedings of the Advances in Neural Information Processing Systems_ , 2021, pp. 8821–8831.
  * (2) Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro, “Diffwave: A versatile diffusion model for audio synthesis,” in _Proceedings of the International Conference on Learning Representations_ , 2021.
  * (3) M. Tao, B.-K. Bao, H. Tang, and C. Xu, “Galip: Generative adversarial clips for text-to-image synthesis,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023.
  * (4) H. Alqahtani, M. Kavakli-Thorne, and G. Kumar, “Applications of generative adversarial networks (gans): An updated review,” _Archives of Computational Methods in Engineering_ , vol. 28, pp. 525–552, 2021.
  * (5) Z. Xiao, K. Kreis, and A. Vahdat, “Tackling the generative learning trilemma with denoising diffusion gans,” in _Proceedings of the International Conference on Learning Representations_ , 2022.
  * (6) I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Proceedings of the Advances in Neural Information Processing Systems_ , vol. 27, 2014, pp. 2672–2680.
  * (7) T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, “Improved techniques for training gans,” in _Proceedings of the Advances in Neural Information Processing Systems_ , vol. 29, 2016.
  * (8) M. Arjovsky and L. Bottou, “Towards principled methods for training generative adversarial networks,” in _Proceedings of the International Conference on Learning Representations_ , 2017.
  * (9) J. Y. Shim, J. S. B. Choe, and J.-K. Kim, “A study on auction-inspired multi-gan training,” in _Annual Spring Conference of KIPS_ , vol. 30, no. 1, 2023, pp. 527–529.
  * (10) V. Krishna, _Auction Theory (Second Edition)_ , second edition ed. San Diego: Academic Press, 2010.
  * (11) L. Metz, B. Poole, D. Pfau, and J. Sohl-Dickstein, “Unrolled generative adversarial networks,” _arXiv preprint arXiv:1611.02163_ , 2016.
  * (12) M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein generative adversarial networks,” in _Proceedings of the International Conference on Machine Learning_ , 2017, pp. 214–223.
  * (13) I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville, “Improved training of wasserstein gans,” in _Proceedings of the Advances in Neural Information Processing Systems_ , vol. 30, 2017, pp. 5767––5777.
  * (14) X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley, “Least squares generative adversarial networks,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 2794–2802.
  * (15) X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel, “Infogan: Interpretable representation learning by information maximizing generative adversarial nets,” in _Proceedings of the Advances in Neural Information Processing Systems_ , vol. 29, 2016.
  * (16) T. Che, Y. Li, A. Jacob, Y. Bengio, and W. Li, “Mode regularized generative adversarial networks,” in _Proceedings of the International Conference on Learning Representations_ , 2017. [Online]. Available: https://openreview.net/forum?id=HJKkY35le
  * (17) M. Mohebbi Moghaddam, B. Boroomand, M. Jalali, A. Zareian, A. Daeijavad, M. H. Manshaei, and M. Krunz, “Games of gans: game-theoretical models for generative adversarial networks,” _Artificial Intelligence Review_ , pp. 1–37, 2023.
  * (18) A. Ghosh, V. Kulharia, V. P. Namboodiri, P. H. Torr, and P. K. Dokania, “Multi-agent diverse generative adversarial networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 8513–8521.
  * (19) Q. Hoang, T. D. Nguyen, T. Le, and D. Phung, “MGAN: Training generative adversarial nets with multiple generators,” in _International Conference on Learning Representations_ , 2018.
  * (20) T. Nguyen, T. Le, H. Vu, and D. Phung, “Dual discriminator generative adversarial nets,” in _Proceedings of the Advances in Neural Information Processing Systems_ , vol. 30, 2017.
  * (21) J. Choi and B. Han, “Mcl-gan: Generative adversarial networks with multiple specialized discriminators,” in _Proceedings of the Advances in Neural Information Processing Systems_ , vol. 35, 2022, pp. 29 597–29 609.
  * (22) I. Albuquerque, J. Monteiro, T. Doan, B. Considine, T. Falk, and I. Mitliagkas, “Multi-objective training of generative adversarial networks with multiple discriminators,” in _Proceedings of the Advances in Neural Information Processing Systems_ , ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15 Jun 2019, pp. 202–211.