# Nash equilibria for total expected reward absorbing Markov games: the
constrained and unconstrained cases111Supported by grant PID2021-122442NB-I00
from the Spanish Ministerio de Ciencia e Innovación.

François Dufour222 Institut Polytechnique de Bordeaux; INRIA Bordeaux Sud
Ouest, Team: ASTRAL; IMB, Institut de Mathématiques de Bordeaux, Université de
Bordeaux, France<EMAIL_ADDRESS>Tomás Prieto-
Rumeau333Statistics Department, UNED, Madrid, Spain. e-mail:
<EMAIL_ADDRESS>(Author for correspondence)

###### Abstract

We consider a nonzero-sum $N$-player Markov game on an abstract measurable
state space with compact metric action spaces. The payoff functions are
bounded Carathéodory functions and the transitions of the system are assumed
to have a density function satisfying some continuity conditions. The
optimality criterion of the players is given by a total expected payoff on an
infinite discrete-time horizon. Under the condition that the game model is
absorbing, we establish the existence of Markov strategies that are a
noncooperative equilibrium in the family of all history-dependent strategies
of the players for both the constrained and the unconstrained problems, We
obtain, as a particular case of results, the existence of Nash equilibria for
discounted constrained and unconstrained game models.

Keywords: Nonzero-sum Markov games; Nash equilibrium; Constrained and
unconstrained games; Total expected payoff criterion; Absorbing game model.

AMS 2020 Subject Classification: 91A10, 91A15.

##  1 Introduction

The topic of noncooperative games has been extensively studied in the last
decades and research on this subject has spread in many directions. Here, we
are interested in nonzero-sum Markov games; namely, we deal with a stochastic
dynamic game on an infinite time horizon: the state of the system evolves
according to a stochastic Markov-like kernel, players take their actions after
each transition of the system, and their goal is to maximize a given
optimality criterion in a noncooperative way. The primary goal is to establish
the existence of a Nash equilibrium: that is, a strategy for each of the
players in such a way that none of the players can improve his payoff by using
another strategy. The interested reader can consult, for instance, the survey
[19] to have an overview of this topic.

When dealing with the discounted payoff optimality criterion, the usual
technique is the dynamic programming approach. This consists in considering a
one-shot game (with a single decision epoch) and then establish the existence
of a fixed point for some suitably defined selectors, which will yield the
Nash equilibrium. Such results have been obtained under various hypotheses in,
e.g., [25, 26], and the most refined and elegant conditions have been proposed
in [18] by introducing so-called “decomposable coarser transition kernels”.

A natural generalization of the game models described so far is to consider
games with constraints. In this case, the players try to maximize their payoff
function subject to the condition that some constraints (related to some other
payoff functions) must be satisfied. A noncooperative equilibrium is then
defined as a set of strategies of the players which satisfy simultaneously all
the constraints and for which, in addition, no player can improve his payoff
when unilaterally modifying his strategy while still satisfying his own
constraints. Such constrained game models were studied first in [2] for a
model with finite state and actions spaces, and then generalized in [3] for
game models with countable state space and compact action spaces, under some
conditions which make the countable state space model nearly finite in each
transition of the system. It is not possible, in general, to solve these
constrained game models by using the dynamic programming approach. The linear
programming approach (which consists in considering the spaces of occupation
measures associated to the strategies of the players) appears to be well
suited to study constrained problems.

Extending this linear programming approach, in the context of obtaining Nash
equilibria for game models, from the finite or countable state space cases to
a more general state space entails, however, serious technical difficulties.
This was achieved in [11] and, based on this reference, there has been a
recent growing interest in the study of constrained games. In [11], a game
model on a measurable state space with ARAT (additive reward, additive
transition) structure under the discounted optimality criterion was studied.
The approach consists in defining a correspondence on a space of measures,
endowed with the weak-strong topology, which is shown to have a fixed point.
From this fixed point, optimal constrained strategies of the players are
obtained. The approach in [11] combines also the use of Young measures to
identify Markov strategies of the players. With this approach, in the
reference [21], a countable state space discounted game model is studied,
dropping the ARAT condition. Also, in [22], the authors study a discounted
constrained game on a general state space and consider the weaker notion of an
approximate equilibria.

The above cited references are, generally, concerned with games under the
discounted payoff optimality criterion. In this paper, we shall consider games
under the total expected payoff optimality criterion, for constrained and
unconstrained games, both of which have received much smaller attention than
the discounted payoff counterpart. Indeed, as far as we know, only a few
references deal with that topic: in [17], the existence of an
$\epsilon$-equilibrium for a countable state space game is established, while
in the references [9, 10], a stopping zero-sum game (in which one player is
allowed to stop the evolution of the system) with countable state space under
the total expected payoff criterion is studied, and in [23] for a finite state
game. Summarizing, there do not exist general existence results for Nash
equilibria for the total expected payoff criterion, even in the case of a
countable state space.

The extension from the discounted optimality criterion to the total expected
payoff optimality criterion is far from being straightforward. First of all,
the dynamic programming approach using the one-shot game does not work in our
context: indeed, the usual technique —see, e.g., [26]— consists in showing
that the Bellman operator maps the final payoff function, which typically
belongs to a compact subset of the space of functions $L^{\infty}$, into
itself when computing the one-shot optimal payoff. To use this technique,
however, the discount factor plays a crucial role and, for the total expected
payoff criterion, the dynamic programming approach does not yield an operator
mapping a compact set of functions into itself. Secondly, while, under the
discounted optimality criterion, the occupation measures are probability or
finite measures, in the context of the total expected payoff criterion, the
occupation measures may be infinite. Ensuring finiteness of these occupation
measures and establishing compactness properties of these spaces of measures
becomes more technically demanding and some additional conditions must be
imposed. More precisely, we will need to assume that the game model is
uniformly absorbing, which means that the dynamic system enters into some
subset $\Delta$ of the state space (in which no further reward is earned) in a
uniformly bounded expected time, with the queues of the hitting time of
$\Delta$ converging uniformly to zero as well (a precise definition will be
given in the text). As already mentioned, our techniques herein will allow to
deal, at the same time, with constrained and unconstrained game models.

In this paper we will consider an $N$-player Markov Markov game under the
total expected reward optimality criterion. The state space is an abstract
measurable space and the action spaces are compact metric spaces. The payoff
functions are assumed to be bounded Carathéodory functions and we impose that
the transition probabilities have a density function with respect to some
reference probability measure, and that they satisfy some suitable
$L^{1}$-continuity properties. Some uniformly absorbing requirement must be
imposed to ensure finiteness of the payoff criteria and compactness of the
occupation measures, plus the usual Slater condition. It is worth stressing
that we do not need the ARAT separation property and, instead, we impose some
minimal sufficient conditions ensuring the continuity of the transition and
cost functionals.

The rest of the paper is organized as follows. In the remaining of this
section we introduce some notation and recall some standard results that will
be useful in the sequel. Section 2 is devoted to define the constrained and
unconstrained game models, and to propose some basic assumptions. Our main
results in the paper are stated in Section 3. In Section 4 we study the
occupation measures and introduce the spaces of Young measures, which shall be
identified with Markov stationary strategies of the players. Some useful
continuity results relating narrow convergence of Young measures and weak-
strong convergence of measures are established. Our main results on the
existence of constrained and unconstrained equilibria are proved in Section 5.

#### Notation and terminology.

A metric space $\SS$ will be always endowed with its Borel $\sigma$-algebra
$\boldsymbol{\mathfrak{B}}(\SS)$. On the product of a finite number of metric
spaces $\SS=\SS^{1}\times\ldots\times\SS^{N}$, we will consider the product
topology which makes the product again a metric space. If the metric spaces
$\SS^{i}$ are separable, then we have
$\boldsymbol{\mathfrak{B}}(\SS)=\boldsymbol{\mathfrak{B}}(\SS^{1})\otimes\ldots\otimes\boldsymbol{\mathfrak{B}}(\SS^{N})$.

On a measurable space $(\mathbf{\Omega},\mathcal{F})$ we will consider the set
of finite signed measures $\boldsymbol{\mathcal{M}}(\mathbf{\Omega})$, the set
of finite nonnegative measures
$\boldsymbol{\mathcal{M}}^{+}(\mathbf{\Omega})$, and the set of probability
measures $\boldsymbol{\mathcal{P}}(\mathbf{\Omega})$. For a set
$\Gamma\in\mathcal{F}$, we denote by
$\mathbf{I}_{\Gamma}:\Omega\rightarrow\\{0,1\\}$ the indicator function of the
set $\Gamma$, that is, $\mathbf{I}_{\Gamma}(\omega)=1$ if and only if
$\omega\in\Gamma$. For $\omega\in\mathbf{\Omega}$, we write
$\delta_{\\{\omega\\}}$ for the Dirac probability measure at $\omega$ defined
on $(\mathbf{\Omega},\mathcal{F})$ by
$\delta_{\\{\omega\\}}(B)=\mathbf{I}_{B}(\omega)$ for any $B\in\mathcal{F}$.
If $\mu\in\boldsymbol{\mathcal{M}}(\mathbf{\Omega})$ and
$\Gamma\in\mathcal{F}$, we denote by $\mu_{\Gamma}$ the measure on
$(\mathbf{\Omega},\mathcal{F})$ defined by $\mu_{\Gamma}(B)=\mu(\Gamma\cap B)$
for $B\in\mathcal{F}$. The trace $\sigma$-algebra of a set
$\Gamma\subseteq\mathbf{\Omega}$ is denoted by $\mathcal{F}_{\Gamma}$. On
$\boldsymbol{\mathcal{P}}(\mathbf{\Omega})$, the $s$-topology is the coarsest
topology that makes $\mu\mapsto\mu(D)$ continuous for every $D\in\mathcal{F}$.

Given a measurable space $(\mathbf{\Omega},\mathcal{F})$ and
$\lambda\in\boldsymbol{\mathcal{P}}(\mathbf{\Omega})$, we will denote by
$L^{1}(\mathbf{\Omega},\mathcal{F},\lambda)$ the family of measurable
functions (identifying those which are $\lambda$-a.s. equal)
$f:\mathbf{\Omega}\rightarrow\mathbb{R}$ which are $\lambda$-integrable, i.e.,
$\|f\|_{1}=\int_{\mathbf{X}}|f(x)|\lambda(dx)<\infty$. Also, let
$L^{\infty}(\mathbf{\Omega},\mathcal{F},\lambda)$ be the set of
$\lambda$-essentially bounded measurable functions
$f:\mathbf{\Omega}\rightarrow\mathbb{R}$ (again, we identify functions that
coincide $\lambda$-a.s.). We will denote by $\|f\|_{\infty}$ the corresponding
essential supremum. On $L^{\infty}(\mathbf{\Omega},\mathcal{F},\lambda)$ we
will consider the weak∗ topology, that is, we have $f_{n}\
{\stackrel{{\scriptstyle*}}{{\rightharpoonup}}\ }f$ whenever

$\int_{\mathbf{\Omega}}f_{n}h\,d\lambda\rightarrow\int_{\mathbf{\Omega}}fh\,d\lambda\quad\hbox{for
every $h\in L^{1}(\mathbf{\Omega},\mathcal{F},\lambda)$}.$

Let $(\mathbf{\Omega},\mathcal{F})$ and
$(\widetilde{\mathbf{\Omega}},\widetilde{\mathcal{F}})$ be two measurable
spaces. A kernel on $\widetilde{\mathbf{\Omega}}$ given $\mathbf{\Omega}$ is a
mapping
$Q:\mathbf{\Omega}\times\widetilde{\mathcal{F}}\rightarrow\mathbb{R}^{+}$ such
that $\omega\mapsto Q(B|\omega)$ is measurable on
$(\mathbf{\Omega},\mathcal{F})$ for every $B\in\widetilde{\mathcal{F}}$, and
$B\mapsto Q(B|\omega)$ is in
$\boldsymbol{\mathcal{M}}^{+}(\widetilde{\mathbf{\Omega}})$ for every
$\omega\in\mathbf{\Omega}$. If $Q(\widetilde{\mathbf{\Omega}}|\omega)=1$ for
all $\omega\in\mathbf{\Omega}$ then we say that $Q$ is a stochastic kernel. We
write $\mathbb{I}_{\Gamma}$ for the kernel on $\mathbf{\Omega}$ given
$\mathbf{\Omega}$ defined by
$\mathbb{I}_{\Gamma}(B|\omega)=\mathbf{I}_{\Gamma}(\omega)\delta_{\\{\omega\\}}(B)$
for $\omega\in\mathbf{\Omega}$ and $B\in\mathcal{F}$. Let $Q$ be a stochastic
kernel on $\widetilde{\mathbf{\Omega}}$ given $\mathbf{\Omega}$. For a bounded
measurable function $f:\widetilde{\mathbf{\Omega}}\rightarrow\mathbb{R}$, we
will denote by $Qf:\mathbf{\Omega}\rightarrow\mathbb{R}$ the measurable
function

$Qf(\omega)=\int_{\mathbf{\Omega^{\prime}}}f(\widetilde{\omega})Q(d\widetilde{\omega}|\omega)\quad\hbox{for
$\omega\in\mathbf{\Omega}$}.$

For a measure $\mu\in\boldsymbol{\mathcal{M}}^{+}(\mathbf{\Omega})$, we denote
by $\mu Q$ the finite measure on
$(\widetilde{\mathbf{\Omega}},\widetilde{\mathcal{F}})$ given by

$B\mapsto\mu
Q\,(B)=\int_{\mathbf{\Omega}}Q(B|\omega)\mu(d\omega)\quad\hbox{for
$B\in\widetilde{\mathcal{F}}$}.$

The product of the $\sigma$-algebras $\mathcal{F}$ and
$\widetilde{\mathcal{F}}$ is denoted by
$\mathcal{F}\otimes\widetilde{\mathcal{F}}$ and consists of the
$\sigma$-algebra generated by the measurable rectangles, that is, the sets of
the form $\Gamma\times\widetilde{\Gamma}$ for $\Gamma\in\mathcal{F}$ and
$\widetilde{\Gamma}\in\widetilde{\mathcal{F}}$. We denote by $\mu\otimes Q$
the unique probability measure (or finite measure) on the product space
$(\mathbf{\Omega}\times\widetilde{\mathbf{\Omega}},\mathcal{F}\otimes\widetilde{\mathcal{F}})$
satisfying

$(\mu\otimes
Q)(\Gamma\times\widetilde{\Gamma})=\int_{\Gamma}Q(\widetilde{\Gamma}|\omega)\mu(d\omega)\quad\hbox{for
$\Gamma\in\mathcal{F}$ and $\widetilde{\Gamma}\in\widetilde{\mathcal{F}}$,}$

see Proposition III-2-1 in [24] for a proof of existence and uniqueness of
such measure. Let $(\overline{\mathbf{\Omega}},\overline{\mathcal{F}})$ be a
third measurable space and $R$ a stochastic kernel on
$\overline{\mathbf{\Omega}}$ given $\widetilde{\mathbf{\Omega}}$. Then we will
denote by $QR$ the stochastic kernel on $\overline{\mathbf{\Omega}}$ given
$\mathbf{\Omega}$ given by

$QR(\Gamma|\omega)=\int_{\widetilde{\mathbf{\Omega}}}R(\Gamma|\tilde{\omega})Q(d\tilde{\omega}|\omega)\quad\hbox{for
$\Gamma\in\overline{\mathcal{F}}$ and $\omega\in\mathcal{F}$}.$

Given
$\mu\in\boldsymbol{\mathcal{M}}(\mathbf{\Omega}\times\widetilde{\mathbf{\Omega}})$,
the marginal measures are
$\mu^{\mathbf{\Omega}}\in\boldsymbol{\mathcal{M}}(\mathbf{\Omega})$ and
$\mu^{\widetilde{\mathbf{\Omega}}}\in\boldsymbol{\mathcal{M}}(\widetilde{\mathbf{\Omega}})$
defined by
$\mu^{\mathbf{\Omega}}(\cdot)=\mu(\cdot\times\widetilde{\mathbf{\Omega}})$ and
$\mu^{\widetilde{\mathbf{\Omega}}}(\cdot)=\mu(\mathbf{\Omega}\times\cdot)$. If
$\pi$ is a kernel on
$\widetilde{\mathbf{\Omega}},\times\overline{\mathbf{\Omega}}$ given
$\mathbf{\Omega}$ the marginal kernels are $\pi^{\widetilde{\mathbf{\Omega}}}$
and $\pi^{\overline{\mathbf{\Omega}}}$, respectively defined by
$\pi^{\widetilde{\mathbf{\Omega}}}=\pi(\cdot\times\overline{\mathbf{\Omega}}|\omega)$
and
$\pi^{\overline{\mathbf{\Omega}}}=\pi(\widetilde{\mathbf{\Omega}}\times\cdot|\omega)$
for $\omega\in\mathbf{\Omega}$.

We say that $f:\mathbf{\Omega}\times\SS\rightarrow\SS^{\prime}$, where
$\SS^{\prime}$ is a metric space, is a Carathéodory function if $f(\cdot,s)$
is measurable on $\mathbf{\Omega}$ for every $s\in\SS$ and $f(\omega,\cdot)$
is continuous on $\SS$ for every $\omega\in\mathbf{\Omega}$. The family of the
so-defined Carathéodory functions is denoted by
${\mathcal{C}ar}(\mathbf{\Omega}\times\SS,\SS^{\prime})$. The family of
Carathéodory functions which, in addition, are bounded is denoted by
${\mathcal{C}ar}_{b}(\mathbf{\Omega}\times\SS,\SS^{\prime})$. When the metric
space $\SS$ is separable then any
$f\in{\mathcal{C}ar}(\mathbf{\Omega}\times\SS,\SS^{\prime})$ is a jointly
measurable function on
$(\mathbf{\Omega}\times\mathbf{S},\mathcal{F}\otimes\boldsymbol{\mathfrak{B}}(\mathbf{S}))$;
see [1, Lemma 4.51].

Given $\lambda\in\boldsymbol{\mathcal{P}}(\mathbf{\Omega})$, let
$\boldsymbol{\mathcal{P}}_{\lambda}(\mathbf{\Omega})=\\{\eta\in\boldsymbol{\mathcal{P}}(\mathbf{\Omega}):\eta\ll\lambda\\}$
be the family of probability of probability measures which are absolutely
continuous with respect to $\lambda$.

If $\SS$ is a Polish space (a complete and separable metric space), on
$\boldsymbol{\mathcal{M}}(\mathbf{\Omega}\times\SS)$ we will consider the
$ws$-topology (weak-strong topology) which is the coarsest topology for which
the mappings

$\mu\mapsto\int_{\mathbf{\Omega}\times\SS}f(\omega,s)\mu(d\omega,ds)$

for $f\in{\mathcal{C}ar}_{b}(\mathbf{\Omega}\times\SS,\mathbb{R})$ are
continuous. There are other equivalent definitions of this topology as
discussed, for instance, in [15, Section 3.3].

Inequality $\geq$ in $\mathbb{R}^{p}$ means a componentwise inequality $\geq$,
while the inequality $>$ in $\mathbb{R}^{p}$ is a componentwise strict
inequality $>$. Let $\mathbf{1}\in\mathbb{R}^{p}$ be the vector with all
components equal to one.

The next disintegration lemma will be useful in the forthcoming (see Theorem 1
in [27]).

######  Lemma 1.1 (Disintegration lemma)

Let $(\mathbf{\Omega},\mathcal{F})$ be a measurable space and let $\SS$ be a
Polish space. Let $\varphi:\mathbf{\Omega}\twoheadrightarrow\SS$ be a weakly
measurable correspondence with nonempty closed values, and let $\mathbf{K}$ be
the graph of the correspondence. For every
$\mu\in\boldsymbol{\mathcal{M}}^{+}(\mathbf{\Omega}\times\SS)$ such that
$\mu(\mathbf{K}^{c})=0$ there exists a stochastic kernel $Q$ on $\SS$ given
$\mathbf{\Omega}$ such that

$\mu=\mu^{\mathbf{\Omega}}\otimes Q$ (1.1)

and such that $Q(\varphi(\omega)|\omega)=1$ for each
$\omega\in\mathbf{\Omega}$. Moreover, $Q$ is unique
$\mu^{\mathbf{\Omega}}$-almost surely, meaning that if $Q$ and $Q^{\prime}$
are two stochastic kernels that satisfy (1.1) then for all $\omega$ in a set
of $\mu^{\mathbf{\Omega}}$-probability one, the probability measures
$Q(\cdot|\omega)$ and $Q^{\prime}(\cdot|\omega)$ coincide.

##  2 Definition of the game model

###  2.1 Elements of the noncooperative game model

Next we give the primitive data of our $N$-person game model.

  1. (a).

The state space is a measurable space $\mathbf{X}$ endowed with a
$\sigma$-algebra $\boldsymbol{\mathfrak{X}}$.

  2. (b).

The separable metric space $\mathbf{A}^{i}$, with $i\in\\{1,\ldots,N\\}$,
stands for the action space of player $i$. Given any $x\in\mathbf{X}$, the
nonempty measurable set $\mathbf{A}^{i}(x)\subseteq\mathbf{A}^{i}$ is the set
of actions available to player $i$ at state $x$. We will use the notations

$\mathbf{A}=\mathbf{A}^{1}\times\ldots\times\mathbf{A}^{N}\quad\hbox{and}\quad\mathbf{A}(x)=\mathbf{A}^{1}(x)\times\ldots\times\mathbf{A}^{N}(x)\subseteq\mathbf{A}.$

A typical element of $\mathbf{A}$ will be written $a=(a^{1},\ldots,a^{N})$.

  3. (c).

Given $i\in\\{1,\ldots,N\\}$, the bounded measurable functions
$r^{i}:\mathbf{X}\times\mathbf{A}\rightarrow\mathbb{R}$ and
$c^{i}:\mathbf{X}\times\mathbf{A}\rightarrow\mathbb{R}^{p}$ stand for the
reward and constraint functions for player $i$. The components of $c^{i}$ will
be denoted by $c^{i,j}$ for $1\leq j\leq p$. The corresponding constraint
constant is $\rho^{i}\in\mathbb{R}^{p}$. Here, $p\geq 1$ is a fixed integer
assumed to be the same for all the players. We write
$\rho=(\rho^{1},\ldots,\rho^{N})\in\mathbb{R}^{pN}$.

  4. (d).

The transitions of the system are given by a stochastic kernel $Q$ on
$\mathbf{X}$ given $\mathbf{X}\times\mathbf{A}$.

  5. (e).

The initial distribution of the system is the probability measure
$\eta\in\boldsymbol{\mathcal{P}}(\mathbf{X})$.

We will consider the sets

$\mathbf{H}_{0}=\mathbf{X}\quad\hbox{and}\quad\mathbf{H}_{t}=(\mathbf{X}\times\mathbf{A})^{t}\times\mathbf{X}\
\ \hbox{for $t\geq 1$},$

which are the sets of histories of the state-action process up to time $t\geq
0$. An element of $\mathbf{H}_{t}$ is denoted by
$h_{t}=(x_{0},a_{0},\ldots,x_{t-1},a_{t-1},x_{t})$. We note that, throughout
this paper, sub-indices will usually refer to the time component $t\geq 0$,
while super-indices will typically denote the players $i\in\\{1,\ldots,N\\}$.

For this model, it is assumed that, at time $t\geq 0$, the players choose
their actions independently of each other conditionally on the history $h_{t}$
of the system; hence, its noncooperative nature. This game will be denoted by
$\mathcal{G}(\eta,\rho)$. We use this notation because, in the sequel, we will
need to vary both the initial distribution and the constraint constants.

###### Definition 2.1

Fix a player $i\in\\{1,\ldots,N\\}$.

  * (i).

A policy for player $i$ is a sequence $\\{\pi_{t}^{i}\\}_{t\geq 0}$ of
stochastic kernels on $\mathbf{A}^{i}$ given $\mathbf{H}_{t}$ that verify
$\pi^{i}_{t}(\mathbf{A}^{i}(x_{t})|x_{0},a_{0},\ldots,x_{t})=1$ for every
$t\geq 0$ and $h_{t}=(x_{0},a_{0},\ldots,x_{t})\in\mathbf{H}_{t}$. Let
$\mathbf{\Pi}^{i}$ be the family of all policies of player $i$.

  * (ii).

Let $\mathbf{M}^{i}$ be the family of stochastic kernels $\pi^{i}$ on
$\mathbf{A}^{i}$ given $\mathbf{X}$ for which $\pi^{i}(\mathbf{A}^{i}(x)|x)=1$
for every $x\in\mathbf{X}$. The policy $\\{\pi^{i}_{t}\\}_{t\geq
0}\in\mathbf{\Pi}_{i}$ is said to be a stationary Markov policy for player $i$
if, for some $\pi^{i}\in\mathbf{M}^{i}$, we have

$\pi^{i}_{t}(\cdot|x_{0},a_{0},\ldots,x_{t})=\pi^{i}(\cdot|x_{t})\quad\hbox{for
every $t\geq 0$ and $h_{t}=(x_{0},a_{0},\ldots,x_{t})\in\mathbf{H}_{t}$}.$

Our Assumption $(\mathrm{A}_{4})$ below will ensure that these sets of
policies are nonempty. We will usually refer to $\mathbf{\Pi}^{i}$ as to the
class of history-dependent policies of player $i$. The family of history-
dependent policies for the players is
$\mathbf{\Pi}=\mathbf{\Pi}^{1}\times\ldots\times\mathbf{\Pi}^{N}$. We will say
that $\pi\in\mathbf{\Pi}$ is a strategy profile. We can identify the class of
stationary Markov policies for player $i$ with $\mathbf{M}^{i}$ itself, and so
we will write $\mathbf{M}^{i}\subseteq\mathbf{\Pi}^{i}$. Similarly, we
introduce the notation
$\mathbf{M}=\mathbf{M}^{1}\times\ldots\times\mathbf{M}^{N}$ for the class of
stationary Markov profiles of the players.

#### The $-i$ notation.

Given $\pi=(\pi^{1},\ldots,\pi^{N})$ and some $i\in\\{1,\ldots,N\\}$, let

$\pi^{-i}=(\pi^{1},\ldots,\pi^{i-1},\pi^{i+1},\ldots,\pi^{N})\in\mathbf{\Pi}^{1}\times\ldots\times\mathbf{\Pi}^{i-1}\times\mathbf{\Pi}^{i+1}\times\ldots\times\mathbf{\Pi}^{N}.$

In addition, given $\sigma\in\mathbf{\Pi}^{i}$, we will use the notation
$(\pi^{-i},\sigma)$ to denote the strategy profile in $\mathbf{\Pi}$ for which
player $i$ uses the policy $\sigma$ and the remaining players use the policies
$\pi^{j}$ for $j\neq i$. Similarly, we will use notations such as
$\mathbf{\Pi}^{-i}$ and $\mathbf{M}^{-i}$ to consider the product spaces of
all the $\mathbf{\Pi}^{j}$ and $\mathbf{M}^{j}$ except $\mathbf{\Pi}^{i}$ and
$\mathbf{M}^{i}$, respectively.

#### Construction of the state-action process.

The canonical space
$\mathbf{H}_{\infty}=(\mathbf{X}\times\mathbf{A})^{\mathbb{N}}$ is endowed
with the product $\sigma$-algebra
$(\boldsymbol{\mathfrak{X}}\otimes\boldsymbol{\mathfrak{B}}(\mathbf{A}))^{\mathbb{N}}$.
Let $(X_{t},A_{t})_{t\geq 0}$ be the corresponding coordinates mappings with
$A_{t}=(A_{t}^{1},\ldots,A_{t}^{N})$. We shall use the notation
$H_{t}=(X_{0},A_{0},\ldots,X_{t})$ for $t\geq 1$ and $H_{0}=X_{0}$. Let us
consider an initial probability measure
$\eta\in\boldsymbol{\mathcal{P}}(\mathbf{X})$ and a strategy profile
$\pi\in\mathbf{\Pi}$. There exists a unique probability measure
$\mathbb{P}_{\eta,\pi}$ on $\mathbf{H}_{\infty}$ such that for every
$B\in\boldsymbol{\mathfrak{X}}$,
$C^{i}\in\boldsymbol{\mathfrak{B}}(\mathbf{A}^{i})$ for $i=1,\ldots,N$, and
$t\geq 0$ we have: (i): $\mathbb{P}_{\eta,\pi}\\{X_{0}\in B\\}=\eta(B)$;

$\text{(ii): }\mathbb{P}_{\eta,\pi}(A^{1}_{t}\in C^{1},\ldots,A^{N}_{t}\in
C^{N}\mid H_{t})=\pi_{t}^{1}(C^{1}|H_{t})\cdots\pi_{t}^{N}(C^{N}|H_{t}),$
(2.2)

and (iii): $\mathbb{P}_{\eta,\pi}(X_{t+1}\in B|H_{t},A_{t})=Q(B|X_{t},A_{t})$.
We denote by $\mathbb{E}_{\eta,\pi}$ the expectation operator associated to
$\mathbb{P}_{\eta,\pi}$. If the initial distribution is the Dirac measure
$\delta_{x}$ concentrated at a given state $x\in\mathbf{X}$ we will simply
write $\mathbb{P}_{x,\pi}$ and $\mathbb{E}_{x,\pi}$.

###  2.2 Correlated strategies, absorbing models, and Nash equilibria

In Definition 2.1 it is assumed that the players choose their actions
independently of each other. It will be technically useful, however, to
introduce correlated strategies for which the players can take dependent
actions. The notion of correlated strategies plays a very important role in
the analysis of this type of game. In particular, it will allow to introduce
the set of possible answers for each player (see Definition 4.7) which are
defined from the occupation measures of the process generated precisely from
these correlated strategies (see Definition 4.2). To show our main results, we
will have to show that this set of occupation measures is bounded and compact,
which leads to a notion of absorbing model also defined on the basis of
correlated strategies.

###### Definition 2.2

  1. (i).

Let $\tilde{\mathbf{\Pi}}$ be the set of correlated strategies defined as
follows: we say that $\pi=\\{\pi_{t}\\}_{t\geq 0}$ is in
$\tilde{\mathbf{\Pi}}$ if, for every $t\geq 0$ and
$h_{t}=(x_{0},a_{0},\ldots,x_{t})\in\mathbf{H}_{t}$, we have that $\pi_{t}$ is
a stochastic kernel on $\mathbf{A}$ given $\mathbf{H}_{t}$ that verifies
$\pi_{t}(\mathbf{A}(x_{t})|x_{0},a_{0},\ldots,x_{t})=1$.

  2. (ii).

The class of correlated Markov strategies of the players is
$\tilde{\mathbf{M}}$ and it is defined as the set of stochastic kernels $\pi$
on $\mathbf{A}$ given $\mathbf{X}$ such that $\pi(\mathbf{A}(x)|x)=1$ for
every $x\in\mathbf{X}$. As usual, we will assume that
$\tilde{\mathbf{M}}\subseteq\tilde{\mathbf{\Pi}}$.

  3. (iii).

Given an initial distribution $\eta\in\boldsymbol{\mathcal{P}}(\mathbf{X})$
and a correlated strategy $\pi\in\tilde{\mathbf{\Pi}}$, we can construct the
state-action process as we did before, except that (2.2) is replaced with
$\mathbb{P}_{\eta,\pi}(A_{t}\in C\mid H_{t})=\pi_{t}(C|H_{t})$ for any
$C\in\boldsymbol{\mathfrak{B}}(\mathbf{A})$.

We have the obvious inclusion $\mathbf{\Pi}\subseteq\tilde{\mathbf{\Pi}}$.
Moreover, we can associate to each $\pi=(\pi^{1},\ldots,\pi^{N})\in\mathbf{M}$
the transition kernel (denoted again by $\pi$) on $\mathbf{A}$ given
$\mathbf{X}$ defined as

$\pi(da|x)=\pi^{1}(da^{1}|x)\times\cdots\times\pi^{N}(da^{N}|x)\quad\hbox{for
$x\in\mathbf{X}$},$ (2.3)

so that we also have $\mathbf{M}\subseteq\tilde{\mathbf{M}}$. For each
$\pi\in\tilde{\mathbf{M}}$, we denote by $Q_{\pi}$ the stochastic kernel on
$\mathbf{X}$ given $\mathbf{X}$ defined by

$Q_{\pi}(D|x)=\int_{\mathbf{A}}Q(D|x,a)\pi(da|x)\quad\hbox{for
$x\in\mathbf{X}$ and $D\in\boldsymbol{\mathfrak{X}}$}.$

The compositions of $Q_{\pi}$ with itself are denoted by $Q^{t}_{\pi}$ for any
$t\geq 0$, with the convention that $Q^{0}_{\pi}(\cdot|x)$ is the Dirac
probability measure concentrated at $x$.

#### Absorbing games.

Given a subset of the state space $\Delta\in\boldsymbol{\mathfrak{X}}$, we
define the hitting time $T_{\Delta}$ as the measurable function
$T_{\Delta}:\mathbf{H}_{\infty}\rightarrow\mathbb{N}\cup\\{\infty\\}$ given by

$T_{\Delta}(x_{0},a_{0},x_{1},a_{1},\ldots)=\min\\{n\geq 0:x_{n}\in\Delta\\},$

where the $\min$ over the empty set is defined as $+\infty$. Next we propose
the some definitions related to the notion of an absorbing game.

###### Definition 2.3

Fix $\Delta\in\boldsymbol{\mathfrak{X}}$ and an initial distribution
$\eta\in\boldsymbol{\mathcal{P}}(\mathbf{X})$. We say that the game model
$\mathcal{G}(\eta,\rho)$ is absorbing to $\Delta$ if the conditions (a) and
(b) below are satisfied, and we say that $\mathcal{G}(\eta,\rho)$ is uniformly
absorbing to $\Delta$ if it is absorbing and, in addition, it satisfies
condition (c).

  1. (a).

For every $(x,a)\in\Delta\times\mathbf{A}$ we have $Q(\Delta|x,a)=1$ and,
besides, for every $1\leq i\leq N$ and $1\leq j\leq p$ it is $r^{i}(x,a)=0$
and $c^{i,j}(x,a)=0$;

  2. (b).

For any $\pi\in\tilde{\mathbf{\Pi}}$ the expected hitting time
$\mathbb{E}_{\eta,\pi}[T_{\Delta}]$ is finite.

  3. (c).

We have the following limit:

$\lim_{n\rightarrow\infty}\sup_{\pi\in\tilde{\mathbf{M}}}\sum_{t=n}^{\infty}\mathbb{P}_{\eta,\pi}\\{T_{\Delta}>t\\}=0.$

The condition (a) means that, once the state process enters in $\Delta$, it
remains in $\Delta$ thereafter at no further reward or cost (related to the
functions $r^{i}$ and $c^{i}$). The condition (c) can be written in several
equivalent ways, for instance:

$\lim_{n\rightarrow\infty}\sup_{\pi\in\tilde{\mathbf{M}}}\mathbb{E}_{\eta,\pi}[(T_{\Delta}-n)^{+}]=0\quad\hbox{or}\quad\sum_{t=0}^{n}\mathbb{P}_{\eta,\pi}\\{T_{\Delta}>t\\}\uparrow\mathbb{E}_{\eta,\pi}[T_{\Delta}]\
\hbox{uniformly in $\pi\in\tilde{\mathbf{M}}$}.$

Our next result summarizes some important properties. In particular, it is
shown that the expected hitting time $\mathbb{E}_{\eta,\pi}[T_{\Delta}]$ is
uniformly bounded over all correlated strategies, which will imply that the
set of occupation measures is bounded (see Remark 4.3(a)), a key element to
show the compactness of this set.

###### Proposition 2.4

Consider a set $\Delta\in\boldsymbol{\mathfrak{X}}$ and an initial
distribution $\eta\in\boldsymbol{\mathcal{P}}(\mathbf{X})$.

  * (i).

If the game model $\mathcal{G}(\eta,\rho)$ is absorbing to $\Delta$ then
$\sup_{\pi\in\tilde{\mathbf{\Pi}}}\mathbb{E}_{\eta,\pi}[T_{\Delta}]<\infty$.

  * (ii).

The family of initial distributions
$\eta\in\boldsymbol{\mathcal{P}}(\mathbf{X})$ for which the game model
$\mathcal{G}(\eta,\rho)$ is absorbing (respectively, uniformly absorbing) to
$\Delta$ is a convex subset of $\boldsymbol{\mathcal{P}}(\mathbf{X})$.

Proof. (i). The proof of this item is partially based on some arguments used
in [13, Sections 4.4 and 5.5] for the special case of a Borel state space. We
will use the following characterization of the set
$\boldsymbol{\mathcal{S}}_{\eta}=\\{\mathbb{P}_{\eta,\pi}\in\boldsymbol{\mathcal{P}}(\mathbf{H}_{\infty}):\pi\in\tilde{\mathbf{\Pi}}\\}$
of strategic probability measures. A probability measure
$\mathbb{P}\in\boldsymbol{\mathcal{P}}(\mathbf{H}_{\infty})$ is in
$\boldsymbol{\mathcal{S}}_{\eta}$ if and only if

$\displaystyle\mathbb{P}(dx_{0})=\eta(dx_{0})\quad\text{and}\quad\mathbb{P}(dx_{0},\ldots,dx_{t},da_{t},dx_{t+1})={\mathbb{P}}(dx_{0},\ldots,dx_{t},da_{t})Q(dx_{t+1}|x_{t},a_{t})$
(2.4)

for $t\in\mathbb{N}$, where the above differential notation refers to the
marginal of $\mathbb{P}$ on the corresponding variables. Let us show the claim
by contradiction. So, assume that there exists a sequence
$\\{\pi_{k}\\}_{k\in\mathbb{N}}$ in $\tilde{\mathbf{\Pi}}$ satisfying
$\mathbb{E}_{\eta,\pi_{k}}[T_{\Delta}]\geq 2^{k}$ for any $k\in\mathbb{N}$.
Consider $\mathbb{P}\in\boldsymbol{\mathcal{P}}(\mathbf{H}_{\infty})$ defined
as

$\mathbb{P}=\sum_{k\in\mathbb{N}}\frac{1}{2^{k+1}}\mathbb{P}_{\eta,\pi_{k}}.$

It is easily seen that $\mathbb{P}$ satisfies (2.4). Therefore
$\mathbb{P}\in\boldsymbol{\mathcal{S}}_{\eta}$, so that there exists
$\pi\in\tilde{\mathbf{\Pi}}$ with $\mathbb{P}=\mathbb{P}_{\eta,\pi}$. We have,
however,

$\mathbb{E}_{\eta,\pi}[T_{\Delta}]=\int_{\mathbf{H}_{\infty}}T_{\Delta}d\mathbb{P}=\sum_{k\in\mathbb{N}}\frac{1}{2^{k+1}}\int_{\mathbf{H}_{\infty}}T_{\Delta}d\mathbb{P}_{\eta,\pi_{k}}=\sum_{k\in\mathbb{N}}\frac{1}{2^{k+1}}\mathbb{E}_{\eta,\pi_{k}}[T_{\Delta}]=\infty,$

leading to a contradiction with the condition (b) in Definition 2.3.

(ii). This result, for both the absorbing and the uniformly absorbing cases,
is a direct consequence of the fact that
$\alpha\mathbb{P}_{\eta,\pi}+(1-\alpha)\mathbb{P}_{\eta^{\prime},\pi}=\mathbb{P}_{\alpha\eta+(1-\alpha)\eta^{\prime},\pi}$
for any $\eta,\eta^{\prime}$ in $\boldsymbol{\mathcal{P}}(\mathbf{X})$ and
$\pi\in\tilde{\mathbf{\Pi}}$. $\Box$

#### Equilibria of the game model.

Given a strategy profile $\pi\in\mathbf{\Pi}$, the total expected payoff of
player $i\in\\{1,\ldots,N\\}$ is

$\displaystyle
R^{i}(\eta,\pi)=\mathbb{E}_{\eta,\pi}\Big{[}\sum_{t=0}^{\infty}r^{i}(X_{t},A_{t})\Big{]}=\mathbb{E}_{\eta,\pi}\Big{[}\sum_{0\leq
t<T_{\Delta}}r^{i}(X_{t},A_{t})\Big{]}\in\mathbb{R},$ (2.5)

and the corresponding total expected cost (for the constraints) is

$\displaystyle
C^{i}(\eta,\pi)=\mathbb{E}_{\eta,\pi}\Big{[}\sum_{t=0}^{\infty}c^{i}(X_{t},A_{t})\Big{]}=\mathbb{E}_{\eta,\pi}\Big{[}\sum_{0\leq
t<T_{\Delta}}c^{i}(X_{t},A_{t})\Big{]}\in\mathbb{R}^{p}.$ (2.6)

In Section 2.3, we will make assumptions ensuring that $R^{i}(\eta,\pi)$ and
$C^{i}(\eta,\pi)$ are finite —see Remark 3.5(a)— for any $\pi\in\mathbf{\Pi}$.
For the remainder of this section, we will assume that this is the case. We
say that strategy profile $\pi\in\mathbf{\Pi}$ satisfies the constraint of
player $i$ when $C^{i}(\eta,\pi)\geq\rho_{i}$.

We propose now the definitions of constrained and unconstrained Nash
equilibria.

###### Definition 2.5

We say that the strategy profile $\pi_{*}\in\mathbf{\Pi}$ is:

  * (i).

an unconstrained Nash equilibrium in the class of all strategy profiles if for
every $1\leq i\leq N$ and $\sigma\in\mathbf{\Pi}^{i}$ we have

$R^{i}(\eta,(\pi_{*}^{-i},\sigma))\leq R^{i}(\eta,\pi_{*});$

  * (ii).

a constrained Nash equilibrium in the class of all strategy profiles if for
every $1\leq i\leq N$ we have $C^{i}(\eta,\pi_{*})\geq\rho^{i}$ and, in
addition,

$\forall\sigma\in\mathbf{\Pi}^{i},\
C^{i}(\eta,(\pi_{*}^{-i},\sigma))\geq\rho^{i}\ \Rightarrow\
R^{i}(\eta,(\pi_{*}^{-i},\sigma))\leq R^{i}(\eta,\pi_{*}).$

Next we introduce the standard Slater condition. It states that whatever
policies the other players use, player $i$ can find a policy so as to satisfy
his own constraints.

###### Definition 2.6

We say that the game model $\mathcal{G}(\eta,\rho)$ satisfies the Slater
condition when, for each strategy profile $\pi\in\mathbf{\Pi}$ and any player
$1\leq i\leq N$, there exists $\sigma^{i}\in\mathbf{\Pi}^{i}$ such that
$C^{i}(\eta,(\pi^{-i},\sigma^{i}))>\rho^{i}$.

###  2.3 Assumptions and Young measures

In this section we will formulate the assumptions we will need in the sequel.
We will also introduce the notion of Young measure. We present three sets of
basic assumptions that slightly differ from each other. It is easy to notice
that assumption 2.3 is weaker than 2.3, which is itself weaker than 2.3. In
what follows, many results will be proved for an arbitrary initial
distribution in $\boldsymbol{\mathcal{P}}_{\lambda}(\mathbf{X})$ which will be
noted generically by $\eta$.

  * Assumption $\boldsymbol{\mathrm{A}}$

Consider the game model $\mathcal{G}(\eta,\rho)$ with initial distribution
$\eta\in\boldsymbol{\mathcal{P}}(\mathbf{X})$ and constraint constants
$\rho\in\mathbb{R}^{pN}$. We say that $\mathcal{G}(\eta,\rho)$ satisfies
Assumption 2.3 when there exist a probability measure
$\lambda\in\boldsymbol{\mathcal{P}}(\mathbf{X})$ and a set
$\Delta\in\boldsymbol{\mathfrak{X}}$ for which the following conditions hold:

    * $(\mathrm{A}_{1})$

The game model $\mathcal{G}(\eta,\rho)$ is absorbing to $\Delta$.

    * $(\mathrm{A}_{2})$

The game model $\mathcal{G}(\eta,\rho)$ satisfies the Slater condition.

    * $(\mathrm{A}_{3})$

The $\sigma$-algebra $\boldsymbol{\mathfrak{X}}$ is countably generated.

    * $(\mathrm{A}_{4})$

For each player $i\in\\{1,\ldots,N\\}$, the action set $\mathbf{A}^{i}$ is
compact and the correspondence from $\mathbf{X}$ to $\mathbf{A}^{i}$ given by
$x\mapsto\mathbf{A}^{i}(x)$ is weakly measurable with nonempty compact values.

    * $(\mathrm{A}_{5})$

For each player $i\in\\{1,\ldots,N\\}$, we have that $r^{i}$ and $c^{i}$ are
bounded Carathéodory functions, that is,
$r^{i}\in{\mathcal{C}ar}_{b}(\mathbf{X}\times\mathbf{A},\mathbb{R})$ and
$c^{i}\in{\mathcal{C}ar}_{b}(\mathbf{X}\times\mathbf{A},\mathbb{R}^{p})$. Let
$\mathbf{r}>0$ be a componentwise bound for all the $r^{i}$ and $c^{i}$.

    * $(\mathrm{A}_{6})$

There exists a measurable density function
$q:\mathbf{X}\times\mathbf{X}\times\mathbf{A}\rightarrow\mathbb{R}^{+}$ such
that for each $B\in\boldsymbol{\mathfrak{X}}$ and
$(x,a)\in\mathbf{X}\times\mathbf{A}$ we have

$Q(B|x,a)=\int_{B}q(y,x,a)\lambda(dy)\quad\hbox{and}\quad\lim_{n\rightarrow\infty}\int_{\mathbf{X}}|q(y,x,a_{n})-q(y,x,a)|\lambda(dy)=0$

whenever $a_{n}\rightarrow a$ in $\mathbf{A}$.

    * $(\mathrm{A}_{7})$

The game model $\mathcal{G}(\lambda,\rho)$ is absorbing to $\Delta$.

    * $(\mathrm{A}_{8})$

The initial distribution $\eta$ satisfies $\eta\ll\lambda$.

  * Assumption $\boldsymbol{\mathrm{A}^{\prime}}$

Consider the game model $\mathcal{G}(\eta,\rho)$ with initial distribution
$\eta\in\boldsymbol{\mathcal{P}}(\mathbf{X})$ and constraint constants
$\rho\in\mathbb{R}^{pN}$. We say that $\mathcal{G}(\eta,\rho)$ satisfies
Assumptions 2.3 when it satisfies Assumption 2.3 except for $(\mathrm{A}_{1})$
which is replaced by the following stronger condition:

    * $(\mathrm{A}^{\prime}_{1})$

The game model $\mathcal{G}(\eta,\rho)$ is uniformly absorbing to $\Delta$.

  * Assumption $\boldsymbol{\mathcal{A}}$

Consider the game model $\mathcal{G}(\eta,\rho)$ with initial distribution
$\eta\in\boldsymbol{\mathcal{P}}(\mathbf{X})$ and constraint constants
$\rho\in\mathbb{R}^{pN}$. We say that $\mathcal{G}(\eta,\rho)$ satisfies
Assumptions 2.3 when it satisfies Assumption 2.3 except for $(\mathrm{A}_{7})$
which is replaced by the following stronger condition:

    * $(\mathcal{A}_{7})$

The game model $\mathcal{G}(\lambda,\rho)$ is uniformly absorbing to $\Delta$.

The Assumptions 2.3, 2.3 and 2.3 will be discussed in the next section.

#### The space of Young measures $\boldsymbol{\mathcal{Y}}$.

Now we introduce the notion of Young measure in order to endow the spaces
$\mathbf{M}$ and $\tilde{\mathbf{M}}$ of stationary Markov strategies with a
suitable metric. Note also that the last assumption we will need in this paper
will be formulated at the end of this section, in terms of continuity
properties of functions defined on these Young measure spaces. To do this we
will rely on the reference probability measure $\lambda$ on the state space
$\mathbf{X}$ introduced in Assumption $(\mathrm{A}_{6})$.

Recall that, for the noncooperative game model, the class of stationary Markov
profiles of the players is $\mathbf{M}^{1}\times\ldots\times\mathbf{M}^{N}$.
Given a player $1\leq i\leq N$, we consider in $\mathbf{M}^{i}$ the following
equivalence relation: for $\phi,\varphi\in\mathbf{M}^{i}$ we say that

$\phi\sim\varphi\quad\hbox{when}\quad\phi(\cdot|x)=\varphi(\cdot|x)\quad\hbox{for
$\lambda$-almost every $x\in\mathbf{X}$}.$

We will denote by $\boldsymbol{\mathcal{Y}}^{i}$ the corresponding family of
equivalence classes, which will be referred to as Young measures. The set
$\boldsymbol{\mathcal{Y}}^{i}$ of Young measures is equipped with the narrow
(stable) topology: it is the coarsest topology that makes the mappings

$\pi^{i}\mapsto\int_{\mathbf{X}}\int_{\mathbf{A}^{i}}f(x,a^{i})\pi^{i}(da^{i}|x)\lambda(dx)$

continuous for any $f$ which is a Carathéodory function on
$\mathbf{X}\times\mathbf{A}^{i}$ bounded in $L^{1}$; more precisely, this
means that $f\in{\mathcal{C}ar}(\mathbf{X}\times\mathbf{A}^{i},\mathbb{R})$ is
such that for some $F\in L^{1}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$
we have $|f(x,a^{i})|\leq F(x)$ for every
$(x,a^{i})\in\mathbf{X}\times\mathbf{A}^{i}$; see, e.g., [4, Theorem 2.2].
Using [5, Lemma 1], it follows that $\boldsymbol{\mathcal{Y}}^{i}$ is a
compact metric space for the narrow topology. We also define

$\boldsymbol{\mathcal{Y}}=\boldsymbol{\mathcal{Y}}^{1}\times\ldots\times\boldsymbol{\mathcal{Y}}^{N},$

which is endowed with the product topology, and it is therefore a compact
metric space as well.

#### Young measures $\boldsymbol{\mathcal{Y}}$ and Markov strategies
$\mathbf{M}$.

We will say that two Markov strategies of the noncooperative game model
$\pi=(\phi^{1},\ldots,\phi^{N})$ and
$\pi^{\prime}=(\varphi^{1},\ldots,\varphi^{N})$ in $\mathbf{M}$ are in the
same equivalence class of Young measures whenever $\phi^{i}\sim\varphi^{i}$
for every $1\leq i\leq N$, and we will write $\pi\sim\pi^{\prime}$ as well. In
this case, since the initial distribution $\eta$ is absolutely continuous with
respect to $\lambda$, and since the transition of the system has a density
with respect to $\lambda$, it is easily seen [12, Lemma 2.2] that both
strategies yield the same strategic probability measure, that is,
$\mathbb{P}_{\eta,\pi}=\mathbb{P}_{\eta,\pi^{\prime}}$. Therefore, $\pi$ and
$\pi^{\prime}$ are indistinguishable since they are driven by the same
strategic probability measures and, besides, they also have the same costs and
rewards since those are defined from the corresponding strategic measures.
Hence, in the sequel we shall identify the set of Young measures
$\boldsymbol{\mathcal{Y}}$ with the family of Markov profiles $\mathbf{M}$ of
the players.

#### Consistence of the notation.

Given $\pi\in\mathbf{M}$ and a function
$f\in{\mathcal{C}ar}_{b}(\mathbf{X}\times\mathbf{A})$, define the measurable
function $f_{\pi}$ on $\mathbf{X}$ as

$f_{\pi}(x)=\int_{\mathbf{A}}f(x,a)\pi(da|x)\quad\hbox{for $x\in\mathbf{X}$}.$

If $\pi^{\prime}\in\mathbf{M}$ is such that $\pi^{\prime}\sim\pi$ then
$f_{\pi^{\prime}}=f_{\pi}$ with $\lambda$-probability one, and so they both
belong to the same equivalence class in
$L^{\infty}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$. Therefore, it is
consistent to define the function $f_{\pi}\in
L^{\infty}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$ for
$\pi\in\boldsymbol{\mathcal{Y}}$.

Suppose that $v\in L^{\infty}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$
and $\pi\in\mathbf{M}$. We have

$Q_{\pi}v(x)=\int_{\mathbf{A}}\int_{\mathbf{X}}v(y)q(y,x,a)\lambda(dy)\pi(da|x)\quad\hbox{for
$x\in\mathbf{X}$}.$

Hence, the above integral does not depend on the representative $v$ chosen in
$L^{\infty}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$; and, in addition,
if $\pi^{\prime}\in\mathbf{M}$ is in the same equivalence class of Young
measures as $\pi$, then $Q_{\pi}v=Q_{\pi^{\prime}}v$ with
$\lambda$-probability one, and so both $\pi$ and $\pi^{\prime}$ yield the same
element in $L^{\infty}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$.
Consequently, the notation $Q_{\pi}v\in
L^{\infty}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$ for $v\in
L^{\infty}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$ and
$\pi\in\boldsymbol{\mathcal{Y}}$ is consistent. The same applies for the
successive compositions $Q^{t}_{\pi}v$ of the stochastic kernels for $t\geq
0$. Note also that Assumption $(\mathrm{A}_{6})$ implies, in particular, that
$Qv$ is well defined, meaning that $Qv=Qv^{\prime}$ whenever $v$ and
$v^{\prime}$ belong to the same equivalence class in
$L^{\infty}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$.

#### Young measures $\tilde{\boldsymbol{\mathcal{Y}}}$ and Markov strategies
$\tilde{\mathbf{M}}$.

We recall that a stationary correlated Markov strategy
$\pi\in\tilde{\mathbf{M}}$ is given by a stochastic kernel $\pi$ on
$\mathbf{A}$ given $\mathbf{X}$ with $\pi(\mathbf{A}(x)|x)=1$ for every
$x\in\mathbf{X}$. As before, we can identify kernels
$\pi,\pi^{\prime}\in\tilde{\mathbf{M}}$ such that
$\pi(\cdot|x)=\pi^{\prime}(\cdot|x)$ $\lambda$-a.s. on $\mathbf{X}$ (written
$\pi\sim\pi^{\prime}$) and then define the set
$\tilde{\boldsymbol{\mathcal{Y}}}$ of Young measures as the corresponding
equivalence classes. The associated narrow topology is the coarsest one that
makes the mappings
$\pi\mapsto\int_{\mathbf{X}}\int_{\mathbf{A}}f(x,a)\pi(da|x)\lambda(dx)$
continuous for every
$f\in{\mathcal{C}ar}(\mathbf{X}\times\mathbf{A},\mathbb{R})$ bounded by a
$\lambda$-integrable function. Again, $\tilde{\boldsymbol{\mathcal{Y}}}$ is a
compact metric space with its narrow topology.

Observe that the equivalence relation of $\boldsymbol{\mathcal{Y}}$ is
compatible with that of $\tilde{\boldsymbol{\mathcal{Y}}}$ meaning that, given
$\pi,\pi^{\prime}\in\mathbf{M}\subseteq\tilde{\mathbf{M}}$, we have
$\pi\sim\pi^{\prime}$ in $\mathbf{M}$ if and only if $\pi\sim\pi^{\prime}$ in
$\tilde{\mathbf{M}}$. Notations such as $f_{\pi}$ and $Q_{\pi}^{t}v$ for
$\pi\in\tilde{\boldsymbol{\mathcal{Y}}}$,
$f\in{\mathcal{C}ar}_{b}(\mathbf{X}\times\mathbf{A})$, $v\in
L^{\infty}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$, and $t\geq 0$ are
consistent as well. Invoking the same previous arguments, we shall hereafter
identify the space $\tilde{\boldsymbol{\mathcal{Y}}}$ of Young measures and
the class $\tilde{\mathbf{M}}$ of correlated Markov strategies of the players.

We make the following very important remark.

###### Remark 2.7

Although it is true that

$\boldsymbol{\mathcal{Y}}^{1}\times\ldots\times\boldsymbol{\mathcal{Y}}^{N}=\boldsymbol{\mathcal{Y}}\subseteq\tilde{\boldsymbol{\mathcal{Y}}},$

it turns out that the trace of the narrow topology of
$\tilde{\boldsymbol{\mathcal{Y}}}$ on $\boldsymbol{\mathcal{Y}}$ does not
coincide, in general, with the product topology of the
$\boldsymbol{\mathcal{Y}}^{i}$. Namely, suppose that $\\{\pi_{n}\\}$ and $\pi$
are in $\boldsymbol{\mathcal{Y}}$. If $\pi_{n}\rightarrow\pi$ in
$\tilde{\boldsymbol{\mathcal{Y}}}$ then it is easy to verify that
$\pi_{n}\rightarrow\pi$ in $\boldsymbol{\mathcal{Y}}$. The converse:
$\pi_{n}\rightarrow\pi$ in ${\boldsymbol{\mathcal{Y}}}$ implies
$\pi_{n}\rightarrow\pi$ in $\tilde{\boldsymbol{\mathcal{Y}}}$, however, is not
necessarily true. Therefore, to fix the terminology, by convergence in
$\boldsymbol{\mathcal{Y}}$ we shall refer to convergence in the product
topology of the $\boldsymbol{\mathcal{Y}}^{i}$, whereas convergence in
$\tilde{\boldsymbol{\mathcal{Y}}}$ will mean convergence in the narrow
topology of $\tilde{\boldsymbol{\mathcal{Y}}}$.

We now introduce an additional condition that will allow us to establish
continuity results for the game model. These continuity conditions will be
expressed in terms of functions defined on the set of Markov strategies
$\boldsymbol{\mathcal{Y}}$. In the next section, we will propose sufficient
conditions for Assumption 2.3 below.

  * Assumption $\boldsymbol{\mathrm{B}}$

We say that the game model $\mathcal{G}(\eta,\rho)$ satisfies Assumption 2.3
when the following mappings, defined on $\boldsymbol{\mathcal{Y}}$ and taking
values in $L^{\infty}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$,

$\pi\mapsto r^{i}_{\pi}\,,\quad\pi\mapsto
c^{i,j}_{\pi}\,,\quad\hbox{and}\quad\pi\mapsto Q_{\pi}v$

are continuous for any $1\leq i\leq N$, $1\leq j\leq p$, and $v\in
L^{\infty}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$.

##  3 Main results

In this section we will present our two main results. The first one shows the
existence of a Markovian equilibrium in the case where $\lambda$ is absolutely
continuous with respect to the initial distribution $\eta$ and under
assumptions 2.3 and 2.3. The second result relaxes the condition
$\lambda\ll\eta$ but requires strengthening Assumption 2.3 to 2.3.

###### Proposition 3.1

Suppose that we are given an initial distribution
$\eta\in\boldsymbol{\mathcal{P}}(\mathbf{X})$ and constraint constants
$\rho\in\mathbb{R}^{Np}$ such that $\lambda\ll\eta$, and that the game
$\mathcal{G}(\eta,\rho)$ satisfies Assumptions 2.3 and 2.3. Both the
constrained and the unconstrained game have a stationary Markov profile which
is a Nash equilibrium in the class of all strategy profiles.

Proof. See Section 5.1 $\Box$

###### Theorem 3.2

Suppose that we are given an initial distribution
$\eta\in\boldsymbol{\mathcal{P}}(\mathbf{X})$ and constraint constants
$\rho\in\mathbb{R}^{Np}$ such that the game $\mathcal{G}(\eta,\rho)$ satisfies
Assumptions 2.3 and 2.3. Both the constrained and the unconstrained game have
a stationary Markov profile which is a Nash equilibrium in the class of all
strategy profiles.

Proof. See Section 5.2 $\Box$

###### Remark 3.3

The proof of Theorem 3.2 will proceed in several steps. First, we need to
consider the case where the reference probability measure $\lambda$ is
absolutely continuous with respect to the initial distribution $\eta$ and show
the existence of a Markovian equilibrium under assumptions 2.3 and 2.3, which
is precisely Proposition 3.1. Then, in a second step, we will drop the
hypothesis $\lambda\ll\eta$ by considering a sequence of game models
$\mathcal{G}(\eta_{n},\rho_{n})$ that satisfies Assumption 2.3 and 2.3 for
$\eta_{n}$ given by a convex combinations of $\eta$ and $\lambda$:

$\eta_{n}=\frac{n}{n+1}\eta+\frac{1}{n+1}\lambda$ (3.7)

and for suitably defined constraint constants $\rho_{n}$. On the basis of
Proposition 3.1, this yields the existence of a constrained Nash equilibrium
$\hat{\pi}_{n}\in\boldsymbol{\mathcal{Y}}$ for the game model
$\mathcal{G}(\eta_{n},\rho_{n})$. Finally, we will prove Theorem 3.2 by
showing that a limit point of the sequence
$\\{\hat{\pi}_{n}\\}\in\boldsymbol{\mathcal{Y}}$ provides a Markovian
equilibrium for the game model $\mathcal{G}(\eta,\rho)$.

Some comments regarding the assumptions are in order now.

###### Remark 3.4

  1. (a).

In the context of an absorbing game model $\mathcal{G}(\eta,\rho)$, the
assumptions $(\mathrm{A}_{1})$-$(\mathrm{A}_{6})$ and $(\mathrm{A}_{8})$ are
conditions classically met in the literature, see for example [11] for the
special case of a discounted model. Assumption $(\mathrm{A}_{7})$ is a key
technical condition which will allow to show very important properties of the
absorbing model by showing in particular that a measure in
$\boldsymbol{\mathcal{M}}^{+}(\mathbf{X}\times\mathbf{A})$ is an occupation
measure if and only if it satisfies the characteristic equations (see items
(i) and (iv) of Proposition 4.6). From this point of view, all the conditions
of Assumption 2.3 are very natural.

  2. (b).

The proof of the existence of a Markovian noncooperative equilibrium relies on
the fact that the set of the marginals on $\mathbf{X}\times\mathbf{A}^{i}$ of
the occupation measure is a compact space in order to use the Kakutani-Fan-
Glicksberg fixed point Theorem. It will be shown in Proposition 4.10 that,
under Assumption 2.3, this set is compact if and only if the model is
uniformly absorbing. This is why it is necessary to replace Assumption 2.3 by
Assumption 2.3 to prove Proposition 3.1. Now regarding the proof of Theorem
3.2, a key step is to show $\mathcal{G}(\eta_{n},\rho_{n})$ is uniformly
absorbing as explained in Remark 3.3. It is, therefore, necessary to reinforce
the absorbing condition of $\mathcal{G}(\lambda,\rho)$ by assuming that
$\mathcal{G}(\lambda,\rho)$ is uniformly absorbing, which leads to replace
Assumption 2.3 by Assumption 2.3 in the statement of the theorem 3.2.

Let us now describe some consequences of these assumptions.

###### Remark 3.5

  1. (a).

The functions $r^{i}$ and $c^{i}$ being bounded under $(\mathrm{A}_{5})$ and
the game model $\mathcal{G}(\eta,\rho)$ being absorbing to $\Delta$ under
$(\mathrm{A}_{1})$ (and also a fortiori under $(\mathrm{A}^{\prime}_{1})$) we
have that
$R^{i}(\eta,\pi)\leq\mathbf{r}\sup_{\pi\in\Pi}\mathbb{E}_{\eta,\pi}[T_{\Delta}]$
and
$C^{i}(\eta,\pi)\leq\mathbf{r}\sup_{\pi\in\Pi}\mathbb{E}_{\eta,\pi}[T_{\Delta}]\mathbf{1}$
(see equations (2.5) and (2.6)), which are therefore finite for any
$\pi\in\Pi$ and each player $i$ by using Proposition 2.4.

  2. (b).

The Slater condition $(\mathrm{A}_{2})$ implies that $\lambda(\Delta^{c})>0$.
Indeed, otherwise we would have $\eta(\Delta)=1$ and so the process would
always remain with probability one in $\Delta$ and the corresponding reward
functions would be all zero. In this very particular case where
$\lambda(\Delta^{c})=0$, the problem is degenerate and all Markov strategies
are noncooperative equilibria.

  3. (c).

By Assumption $(\mathrm{A}_{4})$, the correspondences
$x\mapsto\mathbf{A}^{i}(x)$ are measurable [1, Lemma 18.2] and they have
measurable graph [1, Theorem 18.6]. Therefore, the following sets are
measurable:

$\displaystyle\mathbf{K}^{i}=\\{(x,a^{i})\in\mathbf{X}\times\mathbf{A}^{i}:a^{i}\in\mathbf{A}^{i}(x)\\}\in\boldsymbol{\mathfrak{X}}\otimes\boldsymbol{\mathfrak{B}}(\mathbf{A}^{i})\quad\hbox{for
$1\leq i\leq N$, and}$
$\displaystyle\mathbf{K}=\\{(x,a)\in\mathbf{X}\times\mathbf{A}:a\in\mathbf{A}(x)\\}\in\boldsymbol{\mathfrak{X}}\otimes\boldsymbol{\mathfrak{B}}(\mathbf{A}).$

The Kuratowski-Ryll-Nardzewski selection theorem [1, Theorem 18.13] yields the
existence of measurable selectors for $x\mapsto\mathbf{A}_{i}(x)$ for each
$1\leq i\leq N$. In particular, $\mathbf{M}^{i}$ is nonempty, and so are all
the sets of strategies defined so far: $\mathbf{\Pi}^{i}$, $\mathbf{\Pi}$,
$\tilde{\mathbf{\Pi}}$, $\mathbf{M}$, and $\tilde{\mathbf{M}}$.

We conclude this section by proposing sufficient conditions yielding the
continuity properties stated in Assumption 2.3. We show that some game models
classically studied in the literature satisfy our hypotheses, such as
countable state space models or ARAT-type models. It is also important to note
that the expected discounted models are a special case of the total expected
absorbing models. In this way, our results here generalize those in [11, 21].

###### Corollary 3.6

Suppose that the game model $\mathcal{G}(\eta,\rho)$ satisfies Assumption 2.3.
Under any of the conditions (i) and (ii) below, Assumption 2.3 is satisfied.

  * (i).

The state space $\mathbf{X}$ is countable.

  * (ii).

The game model has ARAT structure (additive reward, additive transition)
meaning that:

    * (a).

(Additive reward.) For any $1\leq i\leq N$, $1\leq j\leq p$, and $1\leq l\leq
N$, there exist functions $r^{i}_{l}$ and $c^{i,j}_{l}$ in
${\mathcal{C}ar}_{b}(\mathbf{X}\times\mathbf{A}^{l})$ such that

$r^{i}(x,a^{1},\ldots,a^{N})=\sum_{l=1}^{N}r^{i}_{l}(x,a^{l})\quad\hbox{and}\quad
c^{i,j}(x,a^{1},\ldots,a^{N})=\sum_{l=1}^{N}c^{i,j}_{l}(x,a^{l})$

for any $(x,a)\in\mathbf{X}\times\mathbf{A}$.

    * (b).

(Additive transition.) There exist nonnegative measurable functions
$q^{l}:\mathbf{X}\times\mathbf{X}\times\mathbf{A}^{l}\rightarrow\mathbb{R}$
such that

$Q(B|x,a^{1},\ldots,a^{N})=\sum_{l=1}^{N}\int_{B}q^{l}(y,x,a^{l})\lambda(dy)\quad\hbox{for
$B\in\boldsymbol{\mathfrak{X}}$ and
$(x,a^{1},\ldots,a^{N})\in\mathbf{X}\times\mathbf{A}$}$

with, in addition,
$\lim_{n\rightarrow\infty}\int_{\mathbf{X}}|q^{l}(y,x,a^{l}_{n})-q^{l}(y,x,a)|\lambda(dy)=0$
for any $x\in\mathbf{X}$ whenever $a_{n}^{l}\rightarrow a^{l}$ as
$n\rightarrow\infty$ in $\mathbf{A}^{l}$.

Proof. See Section 5.3. $\Box$

###### Remark 3.7

The absolute continuity condition in Assumption $(\mathrm{A}_{8})$ is not
restrictive with respect to Assumptions 2.3, 2.3 and 2.3. Indeed, if it were
not true that $\eta\ll\lambda$, then we would consider the reference
probability measure $\bar{\lambda}=(\eta+\lambda)/2$. Clearly we have
$\eta\ll\bar{\lambda}$, while the function
$\bar{q}(y,x,a)=q(y,x,a)(d\lambda/d\bar{\lambda})(y)$ would satisfy Assumption
$(\mathrm{A}_{6})$. Finally, regarding Assumptions $(\mathrm{A}_{7})$ or
$(\mathcal{A}_{7})$, the convexity property in Proposition 2.4(ii) ensures
that the game model $\mathcal{G}(\bar{\lambda},\rho)$ is (respectively,
uniformly) absorbing to $\Delta$ if $\mathcal{G}(\lambda,\rho)$ and
$\mathcal{G}(\eta,\rho)$ are (respectively, uniformly) absorbing. However,
changing the measure $\lambda$ to $\bar{\lambda}$ may affect Assumption 2.3.
Nevertheless, it is important to emphasize that conditions (i) and (ii) of
Corollary 3.6 implying Assumption 2.3 are not affected by a change of the
reference probability measure $\lambda$.

#### A note on discounted games.

As mentioned in [14, p. 132], a $\beta$-discounted model can be transformed
into an equivalent absorbing model just by adding an isolated absorbing
cemetery state $x_{\Delta}$ with a single available action $a_{\Delta}$ at no
reward or cost. In this way, the new state space is
$\mathbf{X}^{\prime}=\mathbf{X}\cup\\{x_{\Delta}\\}$ and the transitions of
the system are given by

$Q^{\prime}(B|x,a)=\begin{cases}\beta Q(B|x,a)\ \hbox{when
$B\subseteq\mathbf{X}$}\\\ 1-\beta\ \hbox{if $B=\\{x_{\Delta}\\}$}\end{cases}$

for $(x,a)\in\mathbf{X}\times\mathbf{A}$ and
$Q^{\prime}(\\{x_{\Delta}\\}|x_{\Delta},a_{\Delta})=1$. The reference
probability measure would be

$\lambda^{\prime}(B)=\beta\lambda(B\cap\mathbf{X})+(1-\beta)\delta_{\\{x_{\Delta}\\}}(B)\quad\hbox{for
measurable $B\subseteq\mathbf{X}^{\prime}$}.$

It is then easily seen that this game model is uniformly absorbing to
$\\{x_{\Delta}\\}$ for any initial distribution. As a consequence, Assumptions
$(\mathrm{A}_{1})$, $(\mathrm{A}_{7})$, $(\mathrm{A}^{\prime}_{1})$ and
$(\mathcal{A}_{7})$ can be dropped, which makes Assumptions 2.3, 2.3 and 2.3
equivalent. This shows that under Assumptions 2.3 and 2.3, we obtain the
existence of constrained and unconstrained equilibria for discounted games. As
a consequence, using Corollary 3.6 we would obtain the results in [11] and
[21].

##  4 Occupation measures

###  4.1 Occupation measures and their topological properties

Throughout this subsection we shall assume that we are given a game model
$\mathcal{G}(\eta,\rho)$ with initial distribution
$\eta\in\boldsymbol{\mathcal{P}}_{\lambda}(\mathbf{X})$ and constraint
constant $\rho\in\mathbb{R}^{pN}$ that satisfies Assumption 2.3.

First of all we state some useful properties of the kernel
$\mathbb{I}_{\Delta^{c}}$ on $\mathbf{X}$ given $\mathbf{X}$ which was defined
in Section 1.

###### Lemma 4.1

The kernel $\mathbb{I}_{\Delta^{c}}$ on $\mathbf{X}$ given $\mathbf{X}$
satisfies the following properties. Given any
$\pi\in\tilde{\boldsymbol{\mathcal{Y}}}$, $f\in
L^{\infty}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$, and
$\mu\in\boldsymbol{\mathcal{M}}^{+}(\mathbf{X})$:

  * (i).

$Q_{\pi}\mathbb{I}_{\Delta^{c}}(B|x)=Q_{\pi}(B\cap\Delta^{c}|x)$ for
$B\in\boldsymbol{\mathfrak{X}}$ and $x\in\mathbf{X}$.

  * (ii).

$\mathbb{I}_{\Delta^{c}}Q_{\pi}(B|x)=Q_{\pi}(B|x)\mathbf{I}_{\Delta^{c}}(x)$
for $B\in\boldsymbol{\mathfrak{X}}$ and $x\in\mathbf{X}$.

  * (iii).

$\mathbb{I}_{\Delta^{c}}Q_{\pi}\mathbb{I}_{\Delta^{c}}=Q_{\pi}\mathbb{I}_{\Delta^{c}}$
and, as a consequence,
$\mathbb{I}_{\Delta^{c}}(Q_{\pi}\mathbb{I}_{\Delta^{c}})^{t}=Q_{\pi}^{t}\mathbb{I}_{\Delta^{c}}=(Q_{\pi}\mathbb{I}_{\Delta^{c}})^{t}$
for any $t\geq 1$.

  * (iv).

$(\mathbb{I}_{\Delta^{c}}f)(x)=f(x)\mathbf{I}_{\Delta^{c}}(x)$ for
$x\in\mathbf{X}$.

  * (v).

$\mu\mathbb{I}_{\Delta^{c}}(B)=\mu(B\cap\Delta^{c})$, which can be also
written $\mu\mathbb{I}_{\Delta^{c}}=\mu_{\Delta^{c}}$.

Next we propose the definition of the occupation measure induced by a
correlated strategy of the players. This definition can be specialized to
noncooperative strategy profiles. We recall that we are making the convention
that the sum over an empty set is zero.

###### Definition 4.2

Given any strategy profile $\pi\in\tilde{\mathbf{\Pi}}$, the occupation
measure
$\mu_{\eta,\pi}\in\boldsymbol{\mathcal{M}}^{+}(\mathbf{X}\times\mathbf{A})$
for the initial distribution
$\eta\in\boldsymbol{\mathcal{P}}_{\lambda}(\mathbf{X})$ is defined, for
measurable sets $B\in\boldsymbol{\mathfrak{X}}$ and
$D^{i}\in\mathfrak{B}(\mathbf{A}^{i})$ for $1\leq i\leq N$, as

$\displaystyle\mu_{\eta,\pi}(B\times D^{1}\times\ldots\times D^{N})$
$\displaystyle=$
$\displaystyle\mathbb{E}_{\eta,\pi}\Big{[}\sum_{t=0}^{\infty}\mathbf{I}_{\\{T_{\Delta}>t\\}}\cdot\mathbf{I}_{\\{X_{t}\in
B,A^{1}_{t}\in D^{1},\ldots,A^{N}_{t}\in D^{N}\\}}\Big{]}.$

We introduce the notations
$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}=\\{\mu_{\eta,\pi}:\pi\in\tilde{\mathbf{\Pi}}\\}$
and $\boldsymbol{\mathcal{O}}_{\eta}=\\{\mu_{\eta,\pi}:\pi\in\mathbf{\Pi}\\}$,
with
$\boldsymbol{\mathcal{O}}_{\eta}\subseteq\tilde{\boldsymbol{\mathcal{O}}}_{\eta}$.

Some important comments concerning this definition are in order.

###### Remark 4.3

  1. (a).

Note that
$\mu_{\eta,\pi}(\mathbf{X}\times\mathbf{A})=\mathbb{E}_{\eta,\pi}[T_{\Delta}]$
for $\pi\in\tilde{\mathbf{\Pi}}$, which is finite because
$\mathcal{G}(\eta,\rho)$ is absorbing to $\Delta$. Moreover, by Proposition
2.4, we have

$\sup_{\pi\in\tilde{\mathbf{\Pi}}}\mu_{\eta,\pi}(\mathbf{X}\times\mathbf{A})<\infty,$

which is usually referred to as $\tilde{\boldsymbol{\mathcal{O}}}_{\eta}$
being bounded. Also observe that, by construction of the process,
$\mu_{\eta,\pi}(\mathbf{K}^{c})=0$. Clearly, the set
$\boldsymbol{\mathcal{O}}_{\eta}$ of occupation measures of the noncooperative
game inherits the same properties.

  2. (b).

Observe that, although the process will eventually visit the set $\Delta$ —it
might even be $\eta(\Delta)>0$— we have
$\mu_{\eta,\pi}^{\mathbf{X}}(\Delta)=0$. This is because, by its definition,
the occupation measure “does not count” visits to $\Delta$. In fact, the
$\mathbf{X}$-marginal of the occupation measure is given by

$\mu_{\eta,\pi}^{\mathbf{X}}(B)=\mathbb{E}_{\eta,\pi}\big{[}\sum_{t=0}^{\infty}\mathbf{I}_{\\{T_{\Delta}>t\\}}\cdot\mathbf{I}_{\\{X_{t}\in
B\\}}\big{]}=\sum_{t=0}^{\infty}\mathbb{P}_{\eta,\pi}\\{X_{t}\in
B-\Delta\\}\quad\hbox{for $B\in\boldsymbol{\mathfrak{X}}$}$ (4.8)

because we have $\\{X_{t}\in B,T_{\Delta}>t\\}=\\{X_{t}\in B-\Delta\\}$.

  3. (c).

It follows directly from Definitions 2.3 and 4.2, and Assumption
$(\mathrm{A}_{5})$ that the total expected payoffs of the strategy profile
$\pi\in\mathbf{\Pi}$ for the initial distribution
$\eta\in\boldsymbol{\mathcal{P}}_{\lambda}(\mathbf{X})$ equal

$R^{i}(\eta,\pi)=\int_{\mathbf{X}\times\mathbf{A}}r^{i}d\mu_{\eta,\pi}\quad\hbox{and}\quad
C^{i}(\eta,\pi)=\int_{\mathbf{X}\times\mathbf{A}}c^{i}d\mu_{\eta,\pi}\quad\hbox{for
$1\leq i\leq N$}.$ (4.9)

  4. (d).

Regarding Markov strategies in $\mathbf{M}$ or $\tilde{\mathbf{M}}$, since
their occupation measure is defined based on the corresponding strategic
probability measures, if follows that two Markov strategies in the same
equivalence class of $\boldsymbol{\mathcal{Y}}$ or
$\tilde{\boldsymbol{\mathcal{Y}}}$ yield the same occupation measure. So, the
notation $\mu_{\eta,\pi}$ for $\pi\in\boldsymbol{\mathcal{Y}}$ or
$\pi\in\tilde{\boldsymbol{\mathcal{Y}}}$ is consistent.

Let us first show the following technical results before deriving properties
on occupation measures.

###### Lemma 4.4

Let $\boldsymbol{\Gamma}$ be an arbitrary subset of $\tilde{\mathbf{M}}$ and
let $\\{h_{\pi}\\}_{\pi\in\boldsymbol{\Gamma}}$ be a family of non-negative
functions in $L^{1}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$ which are
uniformly $\lambda$-integrable. Under these conditions,

$\lim_{t\rightarrow\infty}\sup_{\pi\in\boldsymbol{\Gamma}}\int_{\mathbf{X}}Q^{t}_{\pi}(\Delta^{c}|x)h_{\pi}(x)\lambda(dx)=0.$

Proof. Consider a fixed arbitrary $\epsilon>0$. By the uniform integrability
hypothesis, there exists $c_{\epsilon}>0$ such that

$\sup_{\pi\in\boldsymbol{\Gamma}}\int_{\\{x\in\mathbf{X}:h_{\pi}(x)>c_{\epsilon}\\}}h_{\pi}(x)\lambda(dx)\leq\epsilon.$

Therefore, for any $\pi\in\mathbf{\Gamma}$ and $t\geq 1$

$\int_{\mathbf{X}}Q^{t}_{\pi}(\Delta^{c}|x)h_{\pi}(x)\lambda(dx)\leq\epsilon+c_{\epsilon}\mathbb{P}_{\lambda,\pi}\\{T_{\Delta}>t\\}\leq\epsilon+\frac{c_{\epsilon}}{t}\cdot\mathbb{E}_{\lambda,\pi}[T_{\Delta}].$

From Assumption $(\mathrm{A}_{7})$ and applying Proposition 2.4(i) we have
that
$\sup_{\pi\in\boldsymbol{\Gamma}}\mathbb{E}_{\lambda,\pi}[T_{\Delta}]<\infty$.
Hence, we choose $t$ large enough so as to obtain that
$\sup_{\pi\in\boldsymbol{\Gamma}}\int_{\mathbf{X}}Q^{t}_{\pi}(\Delta^{c}|x)h_{\pi}(x)\lambda(dx)<2\epsilon$,
and the result follows. $\Box$

###### Lemma 4.5

Given any $\pi\in\tilde{\mathbf{M}}$, the measure
$\gamma\in\boldsymbol{\mathcal{M}}^{+}(\mathbf{X})$ defined as

$\displaystyle\gamma=\eta\sum_{k=0}^{\infty}Q_{\pi}^{k}\mathbb{I}_{\Delta^{c}}$
(4.10)

satisfies $\gamma\ll\lambda$ and it is the unique solution of the equation

$\displaystyle\xi=(\eta+\xi Q_{\pi})\mathbb{I}_{\Delta^{c}}\quad\hbox{for
$\xi\in\boldsymbol{\mathcal{M}}^{+}(\mathbf{X})$}.$ (4.11)

Moreover, $\gamma=\mu_{\eta,\pi}^{\mathbf{X}}$.

Proof. First of all, observe that $\gamma$ defined in (4.10) is indeed in
$\boldsymbol{\mathcal{M}}^{+}(\mathbf{X})$ because

$\gamma(\mathbf{X})=\sum_{k=0}^{\infty}(\eta
Q^{k}_{\pi}\mathbb{I}_{\Delta^{c}})(\mathbf{X})=\sum_{k=0}^{\infty}(\eta
Q^{k}_{\pi})(\Delta^{c})=\sum_{k=0}^{\infty}\mathbb{P}_{\eta,\pi}\\{T_{\Delta}>k\\}=\mathbb{E}_{\eta,\pi}[T_{\Delta}]<\infty.$

Recalling that $\eta\ll\lambda$ and using Assumption $(\mathrm{A}_{6})$, it
easily follows that $\gamma\ll\lambda$. Suppose now that
$\xi\in\boldsymbol{\mathcal{M}}^{+}(\mathbf{X})$ is a solution of (4.11). A
first direct consequence is that $\xi\ll\lambda$. Iterating this equation we
obtain that

$\displaystyle\xi=\eta\mathbb{I}_{\Delta^{c}}\sum_{k=0}^{t}(Q_{\pi}\mathbb{I}_{\Delta^{c}})^{k}+\xi(Q_{\pi}\mathbb{I}_{\Delta^{c}})^{t+1}\quad\hbox{for
any $t\in\mathbb{N}$}.$ (4.12)

Now, by Lemma 4.1(iii) we have
$\mathbb{I}_{\Delta^{c}}(Q_{\pi}\mathbb{I}_{\Delta^{c}})^{k}=Q^{k}_{\pi}\mathbb{I}_{\Delta^{c}}$
and
$(Q_{\pi}\mathbb{I}_{\Delta^{c}})^{t+1}=Q^{t+1}_{\pi}\mathbb{I}_{\Delta^{c}}$.
Therefore, the equation (4.12) becomes

$\xi=\eta\sum_{k=0}^{t}Q_{\pi}^{k}\mathbb{I}_{\Delta^{c}}+\xi
Q_{\pi}^{t+1}\mathbb{I}_{\Delta^{c}}\quad\hbox{for any $t\in\mathbb{N}$}.$
(4.13)

Using Lemma 4.4 (here we make use the fact that $\xi\ll\lambda$) it follows
that $\xi Q_{\pi}^{t+1}\mathbb{I}_{\Delta^{c}}(\mathbf{X})=\xi
Q_{\pi}^{t+1}(\Delta^{c})$ converges to $0$ as $t\rightarrow\infty$.
Therefore, taking the limit as $t\rightarrow\infty$ in equation (4.12) we get
that indeed $\xi=\gamma$, which completes the proof of the uniqueness. For the
last statement observe that, by (4.8), it follows that $\gamma$ is precisely
the $\mathbf{X}$-marginal measure of the occupation measure $\mu_{\eta,\pi}$,
that is, $\gamma=\mu^{\mathbf{X}}_{\eta,\pi}$. $\Box$

###### Proposition 4.6

The occupation measures satisfy the following properties.

  1. (i).

Given $\pi\in\tilde{\mathbf{\Pi}}$, the occupation measure $\mu_{\eta,\pi}$
satisfies the so-called characteristic equations (written in the variable
$\mu\in\boldsymbol{\mathcal{M}}^{+}(\mathbf{X}\times\mathbf{A})$):

$\mu(\mathbf{K}^{c})=0\quad\hbox{and}\quad\mu^{\mathbf{X}}=(\eta+\mu
Q)\mathbb{I}_{\Delta^{c}}.$ (4.14)

  2. (ii).

If $\pi\in\tilde{\mathbf{M}}$ is a correlated Markov strategy then
$\mu_{\eta,\pi}=\mu^{\mathbf{X}}_{\eta,\pi}\otimes\pi$. Moreover, if
$\pi\in\mathbf{M}$ then
$\mu_{\eta,\pi}=\mu_{\eta,\pi}^{\mathbf{X}\times\mathbf{A}^{i}}\otimes\pi^{-i}$
for any $i\in\\{1,\ldots,N\\}$.

  3. (iii).

If $\pi\in\mathbf{\Pi}$ is such that $\pi^{-i}\in\mathbf{M}^{-i}$ then there
exists $\sigma\in\mathbf{M}^{i}$ with
$\mu_{\eta,\pi}=\mu_{\eta,(\pi^{-i},\sigma)}$.

  4. (iv).

If $\mu\in\boldsymbol{\mathcal{M}}^{+}(\mathbf{X}\times\mathbf{A})$ is a
solution of (4.14) then there exists $\pi\in\tilde{\mathbf{M}}$ such that
$\mu=\mu_{\eta,\pi}$ and so,

$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}=\big{\\{}\mu\in\boldsymbol{\mathcal{M}}^{+}(\mathbf{X}\times\mathbf{A}):\mu(\mathbf{K}^{c})=0\text{
and }\mu^{\mathbf{X}}=(\eta+\mu Q)\mathbb{I}_{\Delta^{c}}\big{\\}}.$

Moreover, we have $\mu^{\mathbf{X}}\ll\lambda$ and if $\lambda\ll\eta$ then
$\mu^{\mathbf{X}}\sim\lambda_{\Delta^{c}}$.

Proof. (i). To prove the stated result, note that for any
$B\in\boldsymbol{\mathfrak{X}}$ we have

$\displaystyle\mu^{\mathbf{X}}(B)=\sum_{t=0}^{\infty}\mathbb{P}_{\eta,\pi}\\{T_{\Delta}>t,X_{t}\in
B\\}=\eta(B-\Delta)+\sum_{t=1}^{\infty}\mathbb{E}_{\eta,\pi}\big{[}\mathbb{P}_{\eta,\pi}\\{T_{\Delta}>t,X_{t}\in
B\mid H_{t-1},A_{t-1}\\}\big{]}.$

Observe now that for each $t\geq 1$, on the set $\\{T_{\Delta}\leq t-1\\}$,
the conditional probability within brackets vanishes, and so

$\displaystyle\mu^{\mathbf{X}}(B)$ $\displaystyle=$
$\displaystyle\eta(B-\Delta)+\sum_{t=1}^{\infty}\mathbb{E}_{\eta,\pi}\big{[}Q(B-\Delta\mid
X_{t-1},A_{t-1})\cdot\mathbf{I}_{\\{T_{\Delta}>t-1\\}}\big{]}$
$\displaystyle=$
$\displaystyle\eta(B-\Delta)+\int_{\mathbf{X}\times\mathbf{A}}Q(B-\Delta|x,a)\mu(dx,da),$

which can be equivalently written precisely as $\mu^{\mathbf{X}}=(\eta+\mu
Q)\mathbb{I}_{\Delta^{c}}$. By construction of the state-action process, it is
clear that $\mu(\mathbf{K}^{c})=0$.
(ii). Given $B\in\boldsymbol{\mathfrak{X}}$ and
$D^{i}\in\boldsymbol{\mathfrak{B}}(\mathbf{A}^{i})$ we can write

$\displaystyle\mu_{\eta,\pi}(B\times D^{1}\times\ldots\times D^{N})$
$\displaystyle=$
$\displaystyle\sum_{t=0}^{\infty}\mathbb{E}_{\eta,\pi}\Big{[}\mathbf{I}_{\\{T_{\Delta}>t\\}}\mathbf{I}_{\\{X_{t}\in
B\\}}\pi(D^{1}\times\ldots\times D^{N}|X_{t})\Big{]}$ $\displaystyle=$
$\displaystyle\int_{B}\pi(D^{1}\times\ldots\times
D^{N}|x)\mu^{\mathbf{X}}_{\eta,\pi}(dx)$

because, precisely, $\mu_{\eta,\pi}^{\mathbf{X}}(\Gamma)=\sum_{t\geq
0}\mathbb{P}_{\eta,\pi}\\{T_{\Delta}>t,X_{t}\in\Gamma\\}$ for
$\Gamma\in\boldsymbol{\mathfrak{X}}$, and the stated result follows. The
second part of the statement is an easy consequence of the first part and the
fact that, this time, $\pi\in\mathbf{M}$ is a noncooperative Markov profile.
(iii). The occupation measure of the strategy profile $\pi$ satisfies, for
$B\in\boldsymbol{\mathfrak{X}}$ and
$D^{j}\in\boldsymbol{\mathfrak{B}}(\mathbf{A}^{j})$ for $1\leq j\leq N$

$\displaystyle\mu_{\eta,\pi}(B\times D^{1}\times\ldots\times D^{N})$
$\displaystyle=$
$\displaystyle\sum_{t=0}^{\infty}\mathbb{E}_{\eta,\pi}\Big{[}\mathbf{I}_{\\{T_{\Delta}>t\\}}\mathbf{I}_{\\{X_{t}\in
B\\}}\pi^{-i}(D^{-i}|X_{t})\pi^{i}(D^{i}|H_{t})\Big{]}$ (4.15)
$\displaystyle=$
$\displaystyle\int_{B}\pi^{-i}(D^{-i}|x)\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,\pi}(dx\times
D^{i}),$

where $D^{-i}$ denotes the product of all the sets $D^{j}$ except $D^{i}$, and
where $\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,\pi}(dx\times D^{i})$
denotes integration with respect to the measure
$B\mapsto\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,\pi}(B\times D^{i})$. By
the disintegration result in Lemma 1.1, there exists some
$\sigma\in\mathbf{M}^{i}$ such that
$\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,\pi}=\mu^{\mathbf{X}}_{\eta,\pi}\otimes\sigma$.
It then follows from (4.15) that
$\mu_{\eta,\pi}=\mu^{\mathbf{X}}_{\eta,\pi}\otimes(\pi^{-i},\sigma)$ and so
applying statement (i) in this proposition,

$\displaystyle\mu_{\eta,\pi}^{\mathbf{X}}$ $\displaystyle=$
$\displaystyle(\eta+\mu_{\eta,\pi}Q)\mathbb{I}_{\Delta^{c}}$ $\displaystyle=$
$\displaystyle(\eta+\mu^{\mathbf{X}}_{\eta,\pi}Q_{(\pi^{-i},\sigma)})\mathbb{I}_{\Delta^{c}}.$

By Lemma 4.5 we derive that
$\mu_{\eta,\pi}^{\mathbf{X}}=\mu^{\mathbf{X}}_{\eta,(\pi^{-i},\sigma)}$, and
by item (ii) that $\mu_{\eta,\pi}=\mu_{\eta,(\pi^{-i},\sigma)}$.
(iv). By the disintegration in Lemma 1.1, we obtain that
$\mu=\mu^{\mathbf{X}}\otimes\pi$ for some $\pi\in\tilde{\mathbf{M}}$.
Therefore, $\mu^{\mathbf{X}}$ satisfies equation (4.11) and so
$\mu^{\mathbf{X}}=\mu^{\mathbf{X}}_{\eta,\pi}\ll\lambda$, while item (ii)
yields $\mu=\mu^{\mathbf{X}}_{\eta,\pi}\otimes\pi=\mu_{\eta,\pi}$. By using
item (i), we get the characterization of
$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}$. Now, let
$B\in\boldsymbol{\mathfrak{X}}$ be such that $B\subseteq\Delta^{c}$ and
$\mu^{\mathbf{X}}(B)=0$. Since (4.14) implies that
$\mu^{\mathbf{X}}(B)\geq\eta_{\Delta^{c}}(B)$ then necessarily
$\lambda_{\Delta^{c}}(B)=0$. We conclude that
$\mu^{\mathbf{X}}\sim\lambda_{\Delta^{c}}$. $\Box$

Now, we introduce $\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{i}$ as the set of
possible responses for each player $1\leq i\leq N$.

###### Definition 4.7

Given an initial distribution
$\eta\in\boldsymbol{\mathcal{P}}_{\lambda}(\mathbf{X})$ we define

$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{i}=\\{\mu^{\mathbf{X}\times\mathbf{A}^{i}}:\mu\in\tilde{\boldsymbol{\mathcal{O}}}_{\eta}\\}\subseteq\boldsymbol{\mathcal{M}}^{+}(\mathbf{X}\times\mathbf{A}^{i}).$

In our next result, we use the notion of a uniformly absorbing game model (see
Definition 2.3). Recall that, by Assumption 2.3, we are considering an initial
distribution $\eta\in\boldsymbol{\mathcal{P}}_{\lambda}(\mathbf{X})$ such that
the game model $\mathcal{G}(\eta,\rho)$ is absorbing to $\Delta$. To obtain
this important result, we need two preliminary technical Lemmas. A direct
consequence of Assumption 2.3 is the following result.

###### Lemma 4.8

If $v\in L^{\infty}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$ then
$Qv\in{\mathcal{C}ar}_{b}(\mathbf{X}\times\mathbf{A},\mathbb{R})$.

In our next lemma, recall that $\tilde{\boldsymbol{\mathcal{Y}}}$ is endowed
with the narrow topology and that in
$L^{\infty}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$ we consider the
weak∗ convergence.

###### Lemma 4.9

The following continuity results hold.

  * (i).

Given any $f\in{\mathcal{C}ar}_{b}(\mathbf{X}\times\mathbf{A})$ and $v\in
L^{\infty}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$, the mappings
$\pi\mapsto f_{\pi}$ and $\pi\mapsto Q_{\pi}v$ from
$\tilde{\boldsymbol{\mathcal{Y}}}$ to
$L^{\infty}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$ are continuous.

  * (ii).

If $v_{n}\ {\stackrel{{\scriptstyle*}}{{\rightharpoonup}}\ }v$ in
$L^{\infty}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$ and
$\pi_{n}\rightarrow\pi$ in $\tilde{\boldsymbol{\mathcal{Y}}}$ then, for any
$t\geq 0$, we have $Q^{t}_{\pi_{n}}v_{n}\
{\stackrel{{\scriptstyle*}}{{\rightharpoonup}}\ }Q_{\pi}v$.

Proof. Part (i) is a direct consequence of the definition of the narrow
topology and Lemma 4.8. For item (ii), the reader is referred to Lemma 4.1 in
[20] or Lemmas 3.6 and 3.7 in [11]. $\Box$
The above result just concerns convergence in
$\tilde{\boldsymbol{\mathcal{Y}}}$ and it is not necessarily true for
convergence in $\boldsymbol{\mathcal{Y}}$. Indeed, the fact that
$\boldsymbol{\mathcal{Y}}\subseteq\tilde{\boldsymbol{\mathcal{Y}}}$ should not
be misleading since, in $\boldsymbol{\mathcal{Y}}$, we are considering the
product topology of the $\boldsymbol{\mathcal{Y}}^{i}$ which, as mentioned in
Remark 2.7, does not coincide with the trace topology of
$\tilde{\boldsymbol{\mathcal{Y}}}$.

The next proposition shows that it is necessary to reinforce the hypothesis of
an absorbing model by assuming that the model is uniformly absorbing: we will
need this to show that the set $\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{i}$
of possible responses of each player is compact in order to use the Kakutani-
Fan-Glicksberg fixed point theorem leading to the existence of a Markovian
noncooperative equilibrium.

###### Proposition 4.10

The sets $\tilde{\boldsymbol{\mathcal{O}}}_{\eta}$ and
$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{i}$ are convex and the following
statements are equivalent.

  * $(a)$

The game model $\mathcal{G}(\eta,\rho)$ is uniformly absorbing to $\Delta$.

  * $(b)$

The set $\tilde{\boldsymbol{\mathcal{O}}}_{\eta}$ is a compact metric space
for the $ws$-topology.

  * $(c)$

The set $\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{i}$ is a compact metric
space for the $ws$-topology with $i\in\\{1,\ldots,N\\}$.

Proof. Regarding the first claim, observe that the convexity of
$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}$ is a direct consequence of
Proposition 4.6(iv). Convexity of
$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{i}$ is a straightforward consequence
of convexity of $\tilde{\boldsymbol{\mathcal{O}}}_{\eta}$.

$(a)\Rightarrow(b)$ Let us first show that
$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}$ is relatively compact for the
$ws$-topology. Applying Theorem 5.2.(ii) in [6], this is equivalent to show
that the set of $\mathbf{X}$-marginal measures of
$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}$, which we denote by
$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{\mathbf{X}}$, is relatively
$s$-compact and that the set of $\mathbf{A}$-marginal measures of
$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}$, denoted by
$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{\mathbf{A}}$, is relatively
$w$-compact. Recalling Remark 4.3(a), we have that
$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}$ is a bounded subset of
$\boldsymbol{\mathcal{M}}^{+}(\mathbf{X}\times\mathbf{A})$. Since $\mathbf{A}$
is compact, it is clear that
$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{\mathbf{A}}$ is relatively
$w$-compact by using [8, Theorem 8.6.7]. To prove that
$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{\mathbf{X}}$ is relatively
$s$-compact, let us show that

$\lim_{n\rightarrow\infty}\sup_{\mu\in\tilde{\boldsymbol{\mathcal{O}}}_{\eta}}\mu^{\mathbf{X}}(\Gamma_{n})=0$
(4.16)

for any decreasing sequence of sets $\Gamma_{n}\in\boldsymbol{\mathfrak{X}}$
with $\Gamma_{n}\downarrow\emptyset$. Indeed, from [8, Lemma 4.6.5] this
implies that $\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{\mathbf{X}}$ is
uniformly countably additive and so, relatively compact for the $s$-topology;
see [8, Theorem 4.7.25]. Since $\mu^{\mathbf{X}}(\Delta)=0$, there is no loss
of generality in assuming that the $\Gamma_{n}$ are subsets of $\Delta^{c}$.
By Proposition 4.6(iv), for every
$\mu\in\tilde{\boldsymbol{\mathcal{O}}}_{\eta}$ there exists a correlated
Markov strategy $\pi_{\mu}\in\tilde{\boldsymbol{\mathcal{Y}}}$ such that
$\mu=\mu_{\eta,\pi_{\mu}}$ and so, for any fixed $k\geq 0$,

$\displaystyle\mu^{\mathbf{X}}(\Gamma_{n})$ $\displaystyle\leq$
$\displaystyle\sum_{t=0}^{k}\mathbb{P}_{\eta,\pi_{\mu}}\\{X_{t}\in\Gamma_{n}\\}+\sum_{t>k}\mathbb{P}_{\eta,\pi_{\mu}}\\{X_{t}\in\Delta^{c}\\}$
$\displaystyle=$
$\displaystyle\sum_{t=0}^{k}\mathbb{P}_{\eta,\pi_{\mu}}\\{X_{t}\in\Gamma_{n}\\}+\sum_{t>k}\mathbb{P}_{\eta,\pi_{\mu}}\\{T_{\Delta}>t\\}$

Therefore,

$\sup_{\mu\in\tilde{\boldsymbol{\mathcal{O}}}_{\eta}}\mu^{\mathbf{X}}(\Gamma_{n})\leq\sum_{t=0}^{k}\sup_{\mu\in\tilde{\boldsymbol{\mathcal{O}}}_{\eta}}\mathbb{P}_{\eta,\pi_{\mu}}\\{X_{t}\in\Gamma_{n}\\}+\sup_{\mu\in\tilde{\boldsymbol{\mathcal{O}}}_{\eta}}\sum_{t>k}\mathbb{P}_{\eta,\pi_{\mu}}\\{T_{\Delta}>t\\}.$
(4.17)

Let us now pay attention to first term in righthand of (4.17). Suppose first
that $0\leq t\leq k$ and $n\in\mathbb{N}$ remain fixed. By Lemma 4.9 we have
that the mapping $\pi\mapsto Q^{t}_{\pi}(\Gamma_{n}|\cdot)$ from
$\tilde{\boldsymbol{\mathcal{Y}}}$ to
$L^{\infty}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$ is continuous.
Since $\eta\ll\lambda$, this implies that the mapping
$\pi\mapsto\mathbb{P}_{\eta,\pi_{\mu}}\\{X_{t}\in\Gamma_{n}\\}$ is continuous
on $\tilde{\boldsymbol{\mathcal{Y}}}$. Observe now that, by hypothesis, we
have $\mathbf{I}_{\Gamma_{n}}\ {\stackrel{{\scriptstyle*}}{{\rightharpoonup}}\
}0$ as $n\rightarrow\infty$, and so by Lemma 4.9 again, for every
$\pi\in\tilde{\boldsymbol{\mathcal{Y}}}$ we have
$Q^{t}_{\pi}(\Gamma_{n}|\cdot)\
{\stackrel{{\scriptstyle*}}{{\rightharpoonup}}\ }0$ and, therefore,
$\mathbb{P}_{\eta,\pi}\\{X_{t}\in\Gamma_{n}\\}\rightarrow 0$. Summarizing, the
sequence (in $n\in\mathbb{N}$) of continuous mappings
$\pi\mapsto\mathbb{P}_{\eta,\pi}\\{X_{t}\in\Gamma_{n}\\}$ decreases to $0$ and
hence, by Dini’s theorem, the convergence is uniform since
$\tilde{\boldsymbol{\mathcal{Y}}}$ is a compact metric space. So, for each
fixed $0\leq t\leq k$ we have

$\lim_{n\rightarrow\infty}\sup_{\mu\in\tilde{\boldsymbol{\mathcal{O}}}_{\eta}}\mathbb{P}_{\eta,\pi_{\mu}}\\{X_{t}\in\Gamma_{n}\\}=0.$

Regarding the rightmost expression in (4.17), it converges to $0$ as
$n\rightarrow\infty$ as a direct consequence of the fact that
$\mathcal{G}(\eta,\rho)$ is uniformly absorbing to $\Delta$. This completes
the proof of (4.16). Therefore, once we know that
$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}$ is relatively compact for the
$ws$-topology, it follows that it is also metrizable by Proposition 2.3 in
[6].

To prove the compactness of $\tilde{\boldsymbol{\mathcal{O}}}_{\eta}$, the
last step consists in showing that it is closed. To see this, let
$\\{\mu_{n}\\}_{n\geq 0}$ be a sequence in
$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}$ converging in the $ws$-topology to
some $\mu\in\boldsymbol{\mathcal{M}}^{+}(\mathbf{X}\times\mathbf{A})$. First
of all, let us show that $\mu(\mathbf{K}^{c})=0$. The measurable function
$(x,a)\mapsto\mathbf{I}_{\mathbf{K}^{c}}(x,a)$ is such that
$a\mapsto\mathbf{I}_{\mathbf{K}^{c}}(x,a)=\mathbf{I}_{\mathbf{A}^{c}(x)}(a)$
is lower semicontinuous on $\mathbf{A}$ for any fixed $x\in\mathbf{X}$ because
$\mathbf{A}(x)$ is compact. Thus, $\mathbf{I}_{\mathbf{K}^{c}}$ is a
nonnegative normal integrand and [6, Theorem 3.1.(c)] yields
$\mathop{\underline{\lim}}_{n}\mu_{n}(\mathbf{K}^{c})\geq\mu(\mathbf{K}^{c})$
and so $\mu(\mathbf{K}^{c})=0$. On the other hand, it is clear that
$\mu_{n}^{\mathbf{X}}(\Delta)=0$ for all $n\geq 0$ implies that
$\mu^{\mathbf{X}}(\Delta)=0$. To conclude the proof, choose an arbitrary
measurable subset $B$ of $\Delta^{c}$. For every $n\geq 0$ we have

$\mu_{n}^{\mathbf{X}}(B)=\eta(B)+\int_{\mathbf{X}\times\mathbf{A}}Q\mathbf{I}_{B}(x,a)\mu_{n}(dx,da)$

By Lemma 4.8, the function $Q\mathbf{I}_{B}(x,a)$ is in
${\mathcal{C}ar}_{b}(\mathbf{X}\times\mathbf{A},\mathbb{R})$ so that we can
take limits as $n\rightarrow\infty$ to obtain that
$\mu^{\mathbf{X}}(B)=\eta(B)+\mu Q(B)$, thus completing the proof that
$\mu\in\tilde{\boldsymbol{\mathcal{O}}}_{\eta}$.

$(b)\Rightarrow(c)$ Since the mapping from
$\boldsymbol{\mathcal{M}}^{+}(\mathbf{X}\times\mathbf{A})$ to
$\boldsymbol{\mathcal{M}}^{+}(\mathbf{X}\times\mathbf{A}^{i})$ which
associates to $\mu\in\boldsymbol{\mathcal{M}}^{+}(\mathbf{X}\times\mathbf{A})$
its marginal measure $\mu^{\mathbf{X}\times\mathbf{A}^{i}}$ is continuous for
the respective $ws$-topologies, it follows that
$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{i}$ is compact. Again from [6,
Proposition 2.3], noting that the set of $\mathbf{X}$-marginal measures of
$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{i}$ is precisely
$\tilde{\boldsymbol{\mathcal{O}}}^{\mathbf{X}}_{\eta}$, which has been shown
to be relatively $s$-compact, we conclude that
$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{i}$ is metrizable.

$(b)\Rightarrow(a)$ Since $\tilde{\boldsymbol{\mathcal{O}}}_{\eta}$ is compact
for the $ws$-topology, it follows from Theorem 5.2 in [6] that the set of
$\mathbf{X}$-marginal measures of $\tilde{\boldsymbol{\mathcal{O}}}_{\eta}$
(denoted by $\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{\mathbf{X}}$) is
relatively s-compact. By Proposition 4.6(iv),
$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{\mathbf{X}}=\\{\mu^{\mathbf{X}}_{\pi}:\pi\in\tilde{\boldsymbol{\mathcal{Y}}}\\}$.
Combining Proposition 2.2 in [6] and Corollary 2.7 in [16] we get that the
family $\\{h_{\pi}\\}_{\pi\in\tilde{\boldsymbol{\mathcal{Y}}}}$ of density
functions $h_{\pi}={d\mu^{\mathbf{X}}_{\pi}}/{d\lambda}$ is uniformly
$\lambda$-integrable. Now, observe that for
$\pi\in\tilde{\boldsymbol{\mathcal{Y}}}$,

$\sum_{k=t}^{\infty}\mathbb{P}_{\eta,\pi}\\{T_{\Delta}>k\\}=\mu^{\mathbf{X}}_{\pi}Q_{\pi}^{t}(\Delta^{c})=\int_{\mathbf{X}}Q_{\pi}^{t}(\Delta^{c}|x)h_{\pi}(x)\lambda(dx)$

and by using Lemma 4.4 we can conclude that the rightmost term in the previous
equation converges to zero uniformly in
$\pi\in\tilde{\boldsymbol{\mathcal{Y}}}$ as $t\rightarrow\infty$. This
establishes that $\mathcal{G}(\eta,\rho)$ is indeed uniformly absorbing to
$\Delta$.

$(c)\Rightarrow(a)$ Observe that for $i\in\\{1,\ldots,N\\}$, the sets of
$\mathbf{X}$-marginal measures of $\tilde{\boldsymbol{\mathcal{O}}}_{\eta}$
and $\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{i}$ are the same. Consequently,
by Theorem 5.2 in [6] the set of $\mathbf{X}$-marginal measures of
$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}$ is relatively s-compact and the rest
of the proof is identical to that of $(b)\Rightarrow(a)$. $\Box$

###  4.2 Continuity properties of the occupation measures

In this subsection we shall assume that the game model
$\mathcal{G}(\eta,\rho)$ satisfies Assumptions 2.3 and 2.3. In particular
under condition $(\mathrm{A}^{\prime}_{1})$, the set
$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}$ of occupation measures is compact by
Proposition 4.10. Under condition 2.3, we can obtain the following result
similar to Lemma 4.9(ii), whose proof is omitted.

###### Lemma 4.11

If $v_{n}\ {\stackrel{{\scriptstyle*}}{{\rightharpoonup}}\ }v$ in
$L^{\infty}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$ and
$\pi_{n}\rightarrow\pi$ in $\boldsymbol{\mathcal{Y}}$ then
$Q^{t}_{\pi_{n}}v_{n}\ {\stackrel{{\scriptstyle*}}{{\rightharpoonup}}\
}Q^{t}_{\pi}v$ for any $t\in\mathbb{N}$.

At this point, recall the notation $\eta_{n}$ (see (3.7)) for the initial
distributions which are a combination of $\eta$ and $\lambda$.

###### Proposition 4.12

Under any of the conditions (i) and (ii) below:

  * (i).

$\pi_{n}\rightarrow\pi$ in $\boldsymbol{\mathcal{Y}}$ and
$f\in{\mathcal{C}ar}_{b}(\mathbf{X}\times\mathbf{A},\mathbb{R})$ is such that
$\mathbb{I}_{\Delta^{c}}f_{\pi_{n}}(\cdot)\
{\stackrel{{\scriptstyle*}}{{\rightharpoonup}}\
}\mathbb{I}_{\Delta^{c}}f_{\pi}(\cdot)$ in
$L^{\infty}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$,

  * (ii).

$\pi_{n}\rightarrow\pi$ in $\tilde{\boldsymbol{\mathcal{Y}}}$ and
$f\in{\mathcal{C}ar}_{b}(\mathbf{X}\times\mathbf{A},\mathbb{R})$,

we have the following limits:

$\lim_{n\rightarrow\infty}\int_{\mathbf{X}\times\mathbf{A}}fd\mu_{\eta_{n},\pi_{n}}=\int_{\mathbf{X}\times\mathbf{A}}fd\mu_{\eta,\pi}\quad\hbox{and}\quad\lim_{n\rightarrow\infty}\int_{\mathbf{X}\times\mathbf{A}}fd\mu_{\eta,\pi_{n}}=\int_{\mathbf{X}\times\mathbf{A}}fd\mu_{\eta,\pi}$

Proof: We will only prove the first limit in case (i), the remaining cases
being obtained by using similar arguments. Recalling (3.7), observe that

$\mu_{\eta_{n},\pi_{n}}=\frac{n}{n+1}\mu_{\eta,\pi_{n}}+\frac{1}{n+1}\mu_{\lambda,\pi_{n}}\quad\hbox{and}\quad\int_{\mathbf{X}\times\mathbf{A}}f(x,a)d\mu_{\lambda,\pi_{n}}\leq\mathbf{f}\sup_{\pi\in\tilde{\mathbf{\Pi}}}\mu_{\lambda,\pi}(\mathbf{X}\times\mathbf{A})$

for some constant $\mathbf{f}$. From Remark 4.3(a), we only have to show that

$\lim_{n\rightarrow\infty}\int_{\mathbf{X}\times\mathbf{A}}f(x,a)d\mu_{\eta,\pi_{n}}=\int_{\mathbf{X}\times\mathbf{A}}f(x,a)d\mu_{\eta,\pi}.$
(4.18)

Equivalently, the above sequence being bounded, we will prove that any
convergent subsequence has the desired limit. To simplify the notation, and
without loss of generality, we will suppose that the whole sequence is
converging and also that $\\{\mu_{\eta,\pi_{n}}\\}_{n\in\mathbb{N}}$ is a
convergent sequence in $\tilde{\boldsymbol{\mathcal{O}}}_{\eta}$ (recall
Assumption $(\mathrm{A}^{\prime}_{1})$ and Proposition 4.10). We have
$\mu_{\eta,\pi_{n}}=\mu^{\mathbf{X}}_{\eta,\pi_{n}}\otimes\pi_{n}$ with (by
Lemma 4.5 and, in particular, (4.13))

$\mu^{\mathbf{X}}_{\eta,\pi_{n}}=\sum_{k=0}^{t-1}\eta
Q^{k}_{\pi_{n}}\mathbb{I}_{\Delta^{c}}+\mu^{\mathbf{X}}_{\eta,\pi_{n}}Q^{t}_{\pi_{n}}\mathbb{I}_{\Delta^{c}}$

for any $t\in\mathbb{N}^{*}$. Consequently, integrating the function
$f_{\pi_{n}}$ with respect to the above measures, we can write

$\displaystyle\int_{\mathbf{X}\times\mathbf{A}}fd\mu_{\eta,\pi_{n}}$
$\displaystyle=$
$\displaystyle\sum_{k=0}^{t-1}\int_{\mathbf{X}}f_{\pi_{n}}\,d\eta
Q^{k}_{\pi_{n}}\mathbb{I}_{\Delta^{c}}+\int_{\mathbf{X}}f_{\pi_{n}}\,d\mu^{\mathbf{X}}_{\eta,\pi_{n}}Q^{t}_{\pi_{n}}\mathbb{I}_{\Delta^{c}}$
(4.19) $\displaystyle=$
$\displaystyle\sum_{k=0}^{t-1}\int_{\mathbf{X}}Q^{k}_{\pi_{n}}\mathbb{I}_{\Delta^{c}}f_{\pi_{n}}d\eta+\int_{\mathbf{X}}\mathbb{I}_{\Delta^{c}}f_{\pi_{n}}d\mu^{\mathbf{X}}_{\eta,\pi_{n}}Q^{t}_{\pi_{n}},$

for any $t\in\mathbb{N}^{*}$. Observe also that

$\displaystyle\Big{|}\int_{\mathbf{X}}\mathbb{I}_{\Delta^{c}}f_{\pi_{n}}\,d\mu^{\mathbf{X}}_{\eta,\pi_{n}}Q^{t}_{\pi_{n}}\Big{|}\leq\mathbf{f}\cdot\mu^{\mathbf{X}}_{\eta,\pi_{n}}Q^{t}_{\pi_{n}}(\Delta^{c})=\mathbf{f}\sum_{k=t}^{\infty}\mathbb{P}_{\eta,\pi_{n}}\\{T_{\Delta}>k\\}$
(4.20)

for any $t\in\mathbb{N}^{*}$ and $n\in\mathbb{N}$. By hypothesis we have
$\mathbb{I}_{\Delta^{c}}f_{\pi_{n}}(\cdot)\
{\stackrel{{\scriptstyle*}}{{\rightharpoonup}}\
}\mathbb{I}_{\Delta^{c}}f_{\pi}(\cdot)$ in
$L^{\infty}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$ and since
$d\eta/d\lambda$ is in $L^{1}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$,
we have by Lemma 4.11 that

$\displaystyle\int_{\mathbf{X}}Q^{k}_{\pi_{n}}\mathbb{I}_{\Delta^{c}}f_{\pi_{n}}\,\frac{d\eta}{d\lambda}\,d\lambda\rightarrow\int_{\mathbf{X}}Q^{k}_{\pi}\mathbb{I}_{\Delta^{c}}f_{\pi}\,\frac{d\eta}{d\lambda}\,d\lambda.$
(4.21)

Combining equations (4.19)–(4.21) we get that for any $t\in\mathbb{N}^{*}$,

$\displaystyle\Big{|}\lim_{n\rightarrow\infty}\int_{\mathbf{X}\times\mathbf{A}}fd\mu_{\eta,\pi_{n}}-\sum_{k=0}^{t-1}\int_{\mathbf{X}}f_{\pi}\,d\eta
Q^{k}_{\pi}\mathbb{I}_{\Delta^{c}}\Big{|}\leq\mathbf{f}\sup_{n\in\mathbb{N}}\sum_{k=t}^{\infty}\mathbb{P}_{\eta,\pi_{n}}\\{T_{\Delta}>k\\}.$

Finally, by Assumption 2.3 we get

$\lim_{n\rightarrow\infty}\int_{\mathbf{X}\times\mathbf{A}}fd\mu_{\eta,\pi_{n}}=\sum_{k=0}^{\infty}\int_{\mathbf{X}}f_{\pi}\,d\eta
Q^{k}_{\pi}\mathbb{I}_{\Delta^{c}}=\int_{\mathbf{X}}f_{\pi}d\mu^{\mathbf{X}}_{\eta,\pi}=\int_{\mathbf{X}\times\mathbf{A}}fd\mu_{\eta,\pi},$

this establishes the limit in (4.18). $\Box$
The next result will be useful in the forthcoming.

###### Corollary 4.13

The following convergence results hold.

  1. (i).

If $\\{\pi_{n}\\}$ in $\tilde{\boldsymbol{\mathcal{Y}}}$ converges to
$\pi\in\tilde{\boldsymbol{\mathcal{Y}}}$ then
$\mu_{\eta,\pi_{n}}\rightarrow\mu_{\eta,\pi}$ and
$\mu_{\eta_{n},\pi_{n}}\rightarrow\mu_{\eta,\pi}$.

  2. (ii).

Consider $\pi^{-i}\in\boldsymbol{\mathcal{Y}}^{-i}$ fixed for
$i\in\\{1,\ldots,N\\}$. If $\sigma_{n}\rightarrow\sigma$ in
$\boldsymbol{\mathcal{Y}}^{i}$ then
$\mu_{\eta,(\pi^{-i},\sigma_{n})}\rightarrow\mu_{\eta,(\pi^{-i},\sigma)}$ and
$\mu_{\eta_{n},(\pi^{-i},\sigma_{n})}\rightarrow\mu_{\eta,(\pi^{-i},\sigma)}$
in the $ws$-topology.

  3. (iii).

Fix a player $1\leq i\leq N$ and suppose that $\\{\pi_{n}\\}$ in
$\boldsymbol{\mathcal{Y}}$ converges to $\pi\in\boldsymbol{\mathcal{Y}}$.
Then,

$\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,\pi_{n}}\rightarrow\mu_{\eta,\pi}^{\mathbf{X}\times\mathbf{A}^{i}}\quad\hbox{and}\quad\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta_{n},\pi_{n}}\rightarrow\mu_{\eta,\pi}^{\mathbf{X}\times\mathbf{A}^{i}}.$

  4. (iv).

Consider the initial distributions $\eta_{n}$ defined in (3.7) and an
arbitrary sequence $\pi_{n}\rightarrow\pi$ in $\boldsymbol{\mathcal{Y}}$.
Then,

$R^{i}(\eta,\pi_{n})\rightarrow R^{i}(\eta,\pi)\quad\hbox{and}\quad
C^{i}(\eta,\pi_{n})\rightarrow C^{i}(\eta,\pi).$

and also

$R^{i}(\eta_{n},\pi_{n})\rightarrow R^{i}(\eta,\pi)\quad\hbox{and}\quad
C^{i}(\eta_{n},\pi_{n})\rightarrow C^{i}(\eta,\pi).$

Proof: Part (i) follows directly from the second condition in Proposition
4.12. For the second part, observe that if
$g\in{\mathcal{C}ar}(\mathbf{X}\times\mathbf{A})$ then

$g^{i}(x,a^{i})=\int_{\mathbf{A}^{-i}}g(x,(a^{i},a^{-i}))\pi^{-i}(da^{-i}|x)$

is in ${\mathcal{C}ar}(\mathbf{X}\times\mathbf{A}^{i})$. With this in mind, it
is easy to see that, letting $\pi_{n}=(\pi^{-i},\sigma_{n})$ and
$\pi=(\pi^{-i},\sigma)$, we have that $\pi_{n}\rightarrow\pi$ in
$\tilde{\boldsymbol{\mathcal{Y}}}$. Apply now Proposition 4.12(ii). For item
(iii), consider a fixed integer $i\in\\{1,\ldots,N\\}$ and an arbitrary
$g\in{\mathcal{C}ar}_{b}(\mathbf{X}\times\mathbf{A}^{i},\mathbb{R})$. Define
then the function $f$ on $\mathbf{X}\times\mathbf{A}$ by
$f(x,a_{1},\ldots,a_{i},\ldots,a_{N})=g(x,a_{i})$ with
$f\in{\mathcal{C}ar}_{b}(\mathbf{X}\times\mathbf{A},\mathbb{R})$, which
satisfies $\mathbb{I}_{\Delta^{c}}f_{\pi_{n}}(\cdot)\
{\stackrel{{\scriptstyle*}}{{\rightharpoonup}}\
}\mathbb{I}_{\Delta^{c}}f_{\pi}(\cdot)$ in
$L^{\infty}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$. Applying
Proposition 4.12 we conclude that
$\int_{\mathbf{X}\times\mathbf{A}^{i}}gd\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,\pi_{n}}\rightarrow\int_{\mathbf{X}\times\mathbf{A}^{i}}gd\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,\pi}$,
and the result follows. The proof of item (iv) is a direct consequence of
Assumption 2.3, Remark 4.3(c), and Proposition 4.12. $\Box$
By Corollary 4.13(i), when considering the narrow convergence in
$\tilde{\boldsymbol{\mathcal{Y}}}$ we have that the mapping which associates
to each $\pi\in\tilde{\boldsymbol{\mathcal{Y}}}$ its occupation measure
$\mu_{\eta,\pi}$ is continuous. For the product narrow topology on
$\boldsymbol{\mathcal{Y}}$, however, such a result is not true in general and,
instead, we get weaker results as in Corollary 4.13(ii) and (iii) above.
Notice that there is some kind of duality in these results: indeed, item (ii)
shows that if the convergence $\pi_{n}\rightarrow\pi$ takes place in just one
variable then we get convergence of the whole occupation measures
$\mu_{\eta,\pi_{n}}\rightarrow\mu_{\eta,\pi}$, while in item (iii) if the
whole sequence $\pi_{n}$ converges to $\pi$ then we get convergence of,
individually, each component of the occupation measures
$\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,\pi_{n}}\rightarrow\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,\pi}$.

##  5 Proofs of the main results

We will show Proposition 3.1 and Theorem 3.2 in the constrained case. The
unconstrained case is easily deduced from the constrained one. Indeed, by
considering constraint constants satisfying
$\rho<-\mathbf{r}\sup_{\pi\in\Pi}\mathbb{E}_{\eta,\pi}[T_{\Delta}]\mathbf{1}$
we get that $C^{i}(\eta,\pi)>\rho$ for any $\pi\in\Pi$ and any player $i$ (see
item (a) of Remark 3.5) yielding that the constraints and and the Slater
condition $(\mathrm{A}_{2})$ are trivially satisfied.

###  5.1 Proof of Proposition 3.1

We will suppose in this subsection that we are given an initial distribution
$\eta\in\boldsymbol{\mathcal{P}}(\mathbf{X})$ satisfying $\lambda\ll\eta$ and
constraint constants $\rho\in\mathbb{R}^{Np}$ such that the game
$\mathcal{G}(\eta,\rho)$ satisfies Assumptions 2.3 and 2.3.  By recalling
Assumption $(\mathrm{A}_{8})$ and Proposition 4.6(iv), this yields to an
important property, namely,
$\mu^{\mathbf{X}}_{\eta,\pi}\sim\lambda_{\Delta^{c}}$ for any
$\pi\in\tilde{\mathbf{M}}$. The objective of this subsection is to introduce a
correspondence defined as the composition of a function
$\mathcal{J}_{\eta}:\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{1}\times\ldots\times\tilde{\boldsymbol{\mathcal{O}}}^{N}_{\eta}\rightarrow\boldsymbol{\mathcal{Y}}^{1}\times\ldots\times\boldsymbol{\mathcal{Y}}^{N}$
and a correspondence
$\mathcal{H}_{\eta,\rho}:\boldsymbol{\mathcal{Y}}^{1}\times\ldots\times\boldsymbol{\mathcal{Y}}^{N}\twoheadrightarrow\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{1}\times\ldots\times\tilde{\boldsymbol{\mathcal{O}}}^{N}_{\eta}$
and to show that it has a fixed point, from which we will derive equilibrium
stationary Markov policies in the special case where $\lambda\ll\eta$.

#### The function $\mathcal{J}_{\eta}$.

Consider a fixed $\vartheta\in{\mathbf{M}}$. For any
$\pi\in\tilde{\mathbf{M}}$, let us define $\gamma_{\pi}\in\tilde{\mathbf{M}}$
as

$\displaystyle\gamma_{\pi}(B|x)=\pi(B|x)\mathbf{I}_{\Delta^{c}}(x)+\vartheta(B|x)\mathbf{I}_{\Delta}(x)\quad\hbox{for
$B\in\boldsymbol{\mathfrak{B}}(\mathbf{A})$ and $x\in\mathbf{X}$.}$ (5.1)

This definition ensures that $\gamma_{\pi}\in\mathbf{M}$ if
$\pi\in\mathbf{M}$.

###### Lemma 5.1

  1. (i).

Given any $\pi\in\tilde{\mathbf{M}}$ we have equality of the occupation
measures $\mu_{\eta,\pi}=\mu_{\eta,\gamma_{\pi}}$.

  2. (ii).

Let $i\in\\{1,\ldots,N\\}$ be fixed. For any
$m\in\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{i}$ consider
$\tilde{\mathbf{M}}_{m}^{i}=\\{\pi\in\tilde{\mathbf{M}}:\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,\pi}=m\\}$.
Then the set

$\\{\gamma_{\pi}^{\mathbf{A}^{i}}:\pi\in\tilde{\mathbf{M}}_{m}^{i}\\}\subseteq\mathbf{M}^{i}$

is contained in a unique class of equivalence of
$\boldsymbol{\mathcal{Y}}^{i}$, that will be denoted by
$\mathcal{J}^{i}_{\eta}(m)$.

  3. (iii).

Given $\pi\in{\mathbf{M}}$ and $1\leq i\leq N$, let
$\sigma=\gamma_{\pi}^{\mathbf{A}^{i}}$ and consider
$\pi^{\prime}=(\pi^{-i},\sigma)$. Then
$\mu_{\eta,\pi}=\mu_{\eta,\pi^{\prime}}$.

Proof. (i). By its definition, it is clear that
$Q_{\pi}\mathbb{I}_{\Delta^{c}}=Q_{\gamma_{\pi}}\mathbb{I}_{\Delta^{c}}$.
Combining Lemma 4.1(iii) and Lemma 4.5 we conclude that the
$\mathbf{X}$-marginals of the occupation measures of $\pi$ and $\gamma_{\pi}$
coincide: $\mu_{\eta,\pi}^{\mathbf{X}}=\mu_{\eta,\gamma_{\pi}}^{\mathbf{X}}$.
Since $\mu^{\mathbf{X}}_{\eta,\gamma_{\pi}}(\Delta)=0$ and
$\pi(\cdot|x)=\gamma_{\pi}(\cdot|x)$ when $x\in\Delta^{c}$, we conclude that
$\mu_{\eta,\gamma_{\pi}}\otimes\pi=\mu_{\eta,\gamma_{\pi}}\otimes\gamma_{\pi}$.
Summarizing, we have

$\mu_{\eta,\pi}=\mu_{\eta,\pi}^{\mathbf{X}}\otimes\pi=\mu_{\eta,\gamma_{\pi}}^{\mathbf{X}}\otimes\pi=\mu_{\eta,\gamma_{\pi}}^{\mathbf{X}}\otimes\gamma_{\pi}=\mu_{\eta,\gamma_{\pi}}$

by using Proposition 4.6(ii).

(ii). We must show that if $\pi,\bar{\pi}\in\tilde{\mathbf{M}}$ are such that
$\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,{\pi}}=\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,\bar{\pi}}$
then $\gamma_{\pi}^{\mathbf{A}^{i}}$ and
$\gamma_{{\bar{\pi}}}^{\mathbf{A}^{i}}$ belong to the same class of
equivalence in $\boldsymbol{\mathcal{Y}}^{i}$. Clearly, we have
$\mu^{\mathbf{X}}_{\eta,{\pi}}=\mu^{\mathbf{X}}_{\eta,\bar{\pi}}$. By
Proposition 4.6(ii), it follows that
$\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,{\pi}}=\mu^{\mathbf{X}}_{\eta,{\pi}}\otimes\pi^{\mathbf{A}^{i}}$
and
$\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,{\bar{\pi}}}=\mu^{\mathbf{X}}_{\eta,{\bar{\pi}}}\otimes\bar{\pi}^{\mathbf{A}^{i}}$
implying that
$\mu^{\mathbf{X}}_{\eta,{\pi}}\otimes\pi^{\mathbf{A}^{i}}=\mu^{\mathbf{X}}_{\eta,{\pi}}\otimes\bar{\pi}^{\mathbf{A}^{i}}$.
Now, recalling that $\mu^{\mathbf{X}}_{\eta,\pi}\sim\lambda_{\Delta^{c}}$ we
obtain by the disintegration Lemma that
$\pi^{\mathbf{A}^{i}}(\cdot|x)=\bar{\pi}^{\mathbf{A}^{i}}(\cdot|x)$ for
$\lambda_{\Delta^{c}}$-almost all $x\in\mathbf{X}$ which shows that
$\gamma^{\mathbf{A}^{i}}_{\pi}(\cdot|x)=\gamma^{\mathbf{A}^{i}}_{\bar{\pi}}(\cdot|x)$
with $\lambda$-almost all $x\in\mathbf{X}$.

(iii). Proceeding as in part (i), we can show that
$\mu_{\eta,\pi}^{\mathbf{X}}=\mu_{\eta,\pi^{\prime}}^{\mathbf{X}}$, which is a
measure equivalent to $\lambda_{\Delta^{c}}$. Since
$\mu_{\eta,\pi}^{\mathbf{X}}(\Delta)=\mu_{\eta,\pi^{\prime}}^{\mathbf{X}}(\Delta)=0$
and $\pi^{\mathbf{A}^{i}}=\sigma$ on $\Delta^{c}$ we conclude by Proposition
4.6(ii) that
$\mu_{\eta,\pi}^{\mathbf{X}\times\mathbf{A}^{i}}=\mu_{\eta,\pi^{\prime}}^{\mathbf{X}\times\mathbf{A}^{i}}$and
so,

$\mu_{\eta,\pi}=\mu_{\eta,\pi}^{\mathbf{X}\times\mathbf{A}^{i}}\otimes\pi^{-i}\quad\hbox{and}\quad\mu_{\eta,\pi^{\prime}}=\mu_{\eta,\pi^{\prime}}^{\mathbf{X}\times\mathbf{A}^{i}}\otimes\pi^{-i},$

and the result follows. $\Box$
Using the result in Lemma 5.1(ii), we indeed have defined a function
$\mathcal{J}^{i}_{\eta}$ from $\tilde{\boldsymbol{\mathcal{O}}}^{i}_{\eta}$ to
$\boldsymbol{\mathcal{Y}}^{i}$. We can therefore consider the function

$\mathcal{J}_{\eta}:\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{1}\times\ldots\times\tilde{\boldsymbol{\mathcal{O}}}^{N}_{\eta}\rightarrow\boldsymbol{\mathcal{Y}}^{1}\times\ldots\times\boldsymbol{\mathcal{Y}}^{N}=\boldsymbol{\mathcal{Y}}$

whose components are the $\mathcal{J}_{\eta}^{i}$. Based on this lemma and
using Remark 4.3(d), without risk of confusion we can assume that
$\gamma_{\pi}\in\tilde{\boldsymbol{\mathcal{Y}}}$ and
$\gamma_{\pi}^{\mathbf{A}^{i}}\in\boldsymbol{\mathcal{Y}}^{i}$ for
$\pi\in\tilde{\boldsymbol{\mathcal{Y}}}$. Indeed, two Markov correlated
strategies $\pi,\pi^{\prime}$ in the same class of equivalence of
$\boldsymbol{\mathcal{Y}}$ have the same occupation measure and they yield the
same class of equivalence for $\gamma_{\pi}^{\mathbf{A}^{i}}$ and
$\gamma_{\pi^{\prime}}^{\mathbf{A}^{i}}$

###### Proposition 5.2

The function $\mathcal{J}_{\eta}$ is continuous.

Proof: We make the proof of the continuity for the function
$\mathcal{J}^{i}_{\eta}$ for any fixed $1\leq i\leq N$. Suppose that
$\\{m_{n}\\}_{n\geq 0}$ is a sequence in
$\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{i}$ converging in the $ws$-topology
to some $m\in\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{i}$. There exist
$\pi_{n}$ and $\pi$ in $\tilde{\boldsymbol{\mathcal{Y}}}$ such that
$m_{n}=\mu^{\mathbf{X}\times\mathbf{A}_{i}}_{\eta,\pi_{n}}$ for any
$n\in\mathbb{N}$ and $m=\mu^{\mathbf{X}\times\mathbf{A}_{i}}_{\eta,\pi}$. Our
goal is to prove that
$\gamma_{\pi_{n}}^{\mathbf{A}_{i}}\rightarrow\gamma_{\pi}^{\mathbf{A}_{i}}$ in
$\boldsymbol{\mathcal{Y}}^{i}$. Since $\boldsymbol{\mathcal{Y}}^{i}$ is
compact, it suffices to show that this limit holds for any convergent
subsequence of $\\{\gamma_{\pi_{n}}^{\mathbf{A}_{i}}\\}$ (still denoted by
$\\{\gamma_{\pi_{n}}^{\mathbf{A}_{i}}\\}$). There is no loss of generality in
assuming that $\pi_{n}\rightarrow\pi^{*}$ for some
$\pi^{*}\in\tilde{\boldsymbol{\mathcal{Y}}}$ and, therefore, as a direct
consequence of the definition in (5.1) we also have
$\gamma_{\pi_{n}}\rightarrow\gamma_{\pi^{*}}$ and
$\gamma^{\mathbf{A}^{i}}_{\pi_{n}}\rightarrow\gamma^{\mathbf{A}^{i}}_{\pi^{*}}$.
By Corollary 4.13(i) we obtain that

$\mu_{\eta,\pi_{n}}\rightarrow\mu_{\eta,\pi^{*}}\quad\hbox{and so}\quad
m_{n}=\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,\pi_{n}}\rightarrow\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,\pi^{*}}.$

We deduce that
$\mu^{\mathbf{X}\times\mathbf{A}_{i}}_{\eta,\pi^{*}}=\mu^{\mathbf{X}\times\mathbf{A}_{i}}_{\eta,\pi}=m$
and from Lemma 5.1(ii) we conclude that
$\gamma^{\mathbf{A}^{i}}_{\pi^{*}}=\gamma^{\mathbf{A}^{i}}_{\pi}$, which
completes the proof. $\Box$

#### The correspondence $\mathcal{H}_{\eta,\rho}$.

Fix a player $i$ and an arbitrary $\pi^{-i}\in\boldsymbol{\mathcal{Y}}^{-i}$.
Define

$\mathcal{L}_{\eta}^{i}(\pi^{-i})=\big{\\{}\mu_{\eta,(\pi^{-i},\sigma)}^{\mathbf{X}\times\mathbf{A}^{i}}:\sigma\in\boldsymbol{\mathcal{Y}}^{i}\big{\\}}\subseteq\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{i}\subseteq\boldsymbol{\mathcal{M}}^{+}(\mathbf{X}\times\mathbf{A}^{i}),$

which is the set of $(\mathbf{X}\times\mathbf{A}^{i})$-marginals of the
occupation measures for the initial distribution $\eta$ and the strategy
profiles $(\pi^{-i},\sigma)$ as the policy $\pi^{-i}$ of all the players (but
$i$) remain fixed and the policy of player $i$ varies in
$\boldsymbol{\mathcal{Y}}^{i}$.

###### Proposition 5.3

Given any $1\leq i\leq N$ and $\pi^{-i}\in\boldsymbol{\mathcal{Y}}^{-i}$, the
set $\mathcal{L}_{\eta}(\pi^{-i})$ is convex and compact for the
$ws$-topology.

Proof. Let $\gamma,\gamma^{\prime}$ in $\mathcal{L}_{\eta}^{i}(\pi^{-i})$ and
fix some $0\leq\alpha\leq 1$. We want to prove that
$\alpha\gamma+(1-\alpha)\gamma^{\prime}\in\mathcal{L}_{\eta}^{i}(\pi^{-i})$.
There exist $\sigma,\sigma^{\prime}\in\boldsymbol{\mathcal{Y}}_{i}$ satisfying
$\gamma=\mu_{\eta,(\pi^{-i},\sigma)}^{\mathbf{X}\times\mathbf{A}^{i}}$ and
$\gamma^{\prime}=\mu_{\eta,(\pi^{-i},\sigma^{\prime})}^{\mathbf{X}\times\mathbf{A}^{i}}$.
Convexity of $\tilde{\boldsymbol{\mathcal{O}}}_{\eta}$ implies that

$\mu_{\eta,\hat{\pi}}=\alpha\mu_{\eta,(\pi^{-i},\sigma)}+(1-\alpha)\mu_{\eta,(\pi^{-i},\sigma^{\prime})}$
(5.2)

for some $\hat{\pi}\in\tilde{\boldsymbol{\mathcal{Y}}}$. To get the result,
let us show that
$\mu^{{\mathbf{X}\times\mathbf{A}^{i}}}_{\eta,\hat{\pi}}=\mu^{{\mathbf{X}\times\mathbf{A}^{i}}}_{\eta,(\pi^{-i},\tilde{\sigma})}$
for some $\tilde{\sigma}\in\boldsymbol{\mathcal{Y}}^{i}$. Observe that, by
(5.2) and Proposition 4.6(ii),

$\displaystyle\mu_{\eta,\hat{\pi}}$
$\displaystyle=\big{[}\alpha\mu_{\eta,(\pi^{-i},\sigma)}^{\mathbf{X}\times\mathbf{A}^{i}}+(1-\alpha)\mu_{\eta,(\pi^{-i},\sigma^{\prime})}^{\mathbf{X}\times\mathbf{A}^{i}}\big{]}\otimes\pi^{-i}=\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,\hat{\pi}}\otimes\pi^{-i}.$

Moreover,
$\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,\hat{\pi}}=\mu^{\mathbf{X}}_{\eta,\hat{\pi}}\otimes\hat{\pi}^{\mathbf{A}_{i}}$
and so, letting
$\tilde{\sigma}=\hat{\pi}^{\mathbf{A}^{i}}\in\boldsymbol{\mathcal{Y}}^{i}$ we
obtain
$\mu_{\eta,\hat{\pi}}=\mu_{\eta,\hat{\pi}}^{\mathbf{X}}\otimes(\pi^{-i},\tilde{\sigma})$.
Since $\mu_{\eta,\hat{\pi}}^{\mathbf{X}}$ is equivalent to $\lambda$ on
$\Delta^{c}$ (recall Proposition 4.6(iv)), it follows that $\hat{\pi}$ and
$(\pi^{-i},\tilde{\sigma})$ coincide $\lambda$-a.s. on $\Delta^{c}$. In
particular $\gamma_{\hat{\pi}}=\gamma_{(\pi^{-i},\tilde{\sigma})}$ and thus,
using Lemma 5.1(i),

$\mu_{\eta,\hat{\pi}}=\mu_{\eta,\gamma_{\hat{\pi}}}=\mu_{\eta,\gamma_{(\pi^{-i},\tilde{\sigma})}}=\mu_{\eta,(\pi^{-i},\tilde{\sigma})},$

as we wanted to prove. This establishes convexity of
$\mathcal{L}_{\eta}^{i}(\pi^{-i})$. To prove compactness we will show that
$\mathcal{L}_{\eta}^{i}(\pi^{-i})$ is closed. Suppose that
$\gamma_{n}\rightarrow\gamma$ where
$\gamma_{n}\in\mathcal{L}_{\eta}^{i}(\pi^{-i})$ and
$\gamma\in\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{i}$. For each $n$ there is
some $\sigma^{i}_{n}\in\boldsymbol{\mathcal{Y}}^{i}$ such that
$\gamma_{n}=\mu_{\eta,(\pi^{-i},\sigma^{i}_{n})}^{\mathbf{X}\times\mathbf{A}^{i}}$.
For some subsequence of $\\{\sigma^{i}_{n}\\}$, still denoted by
$\\{\sigma^{i}_{n}\\}$, we have $\sigma^{i}_{n}\rightarrow\sigma_{*}^{i}$ for
some $\sigma_{*}^{i}\in\boldsymbol{\mathcal{Y}}^{i}$. By Corollary 4.13(iii),
this shows that
$\gamma_{n}\rightarrow\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,(\pi^{-i},\sigma^{i}_{*})}$,
which indeed belongs to $\mathcal{L}_{\eta}^{i}(\pi^{-i})$. $\Box$
Given a player $1\leq i\leq N$ and a strategy profile
$\pi^{-i}\in\boldsymbol{\mathcal{Y}}^{-i}$ for the remaining players, let

$\mathcal{A}^{i}_{\eta,\rho^{i}}(\pi^{-i})=\Big{\\{}\mu_{\eta,(\pi^{-i},\sigma)}^{\mathbf{X}\times\mathbf{A}^{i}}:\sigma\in\boldsymbol{\mathcal{Y}}^{i}\hbox{
is such that
}C^{i}(\eta,(\pi^{-i},\sigma))\geq\rho^{i}\Big{\\}}\subseteq\mathcal{L}^{i}(\pi^{-i}).$

Thus, $\mathcal{A}^{i}_{\eta,\rho^{i}}(\pi^{-i})$ is the set of
$(\mathbf{X}\times\mathbf{A}^{i})$-marginals of the occupation measures of the
Markov policies $\sigma\in\boldsymbol{\mathcal{Y}}^{i}$ of player $i$ such
that the Markov profile $(\pi^{-i},\sigma)$ satisfies player $i$’s constraint.

###### Proposition 5.4

Consider a player $1\leq i\leq N$ and a sequence
$\\{\pi^{-i}_{n}\\}\subseteq\boldsymbol{\mathcal{Y}}^{-i}$ such that
$\pi_{n}^{-i}\rightarrow\pi^{-i}$ in the product topology of
$\boldsymbol{\mathcal{Y}}^{-i}$ for some $\pi\in\boldsymbol{\mathcal{Y}}$.

  1. (i).

If $C^{i}(\eta,\pi)\geq\rho^{i}$ then there exists a sequence
$\\{\gamma_{n}\\}$ in
$\boldsymbol{\mathcal{M}}^{+}(\mathbf{X}\times\mathbf{A}^{i})$ such that
$\gamma_{n}\rightarrow\mu_{\eta,\pi}^{\mathbf{X}\times\mathbf{A}^{i}}$ and for
some $K\in\mathbb{N}$, we have
$\gamma_{n}\in\boldsymbol{\mathcal{A}}^{i}_{\eta,\rho^{i}}(\pi_{n}^{-i})$ for
$n\geq K$.

  2. (ii).

If $\mathcal{G}(\eta,\rho)$ satisfies the Slater condition and
$C^{i}(\eta,\pi)\geq\rho^{i}$ then there exists a sequence $\\{\gamma_{n}\\}$
in $\boldsymbol{\mathcal{M}}^{+}(\mathbf{X}\times\mathbf{A}^{i})$ such that
$\gamma_{n}\rightarrow\mu_{\eta,\pi}^{\mathbf{X}\times\mathbf{A}^{i}}$ and
such that, for some $K\in\mathbb{N}$, we have
$\gamma_{n}\in\boldsymbol{\mathcal{A}}^{i}_{\eta_{n},\rho^{i}}(\pi_{n}^{-i})$
for $n\geq K$.

Proof: We will prove only item (ii) since item (i) can be easily obtained by
using the same arguments. From Corollary 4.13(iv) we have
$\lim_{n\rightarrow\infty}C^{i}(\eta_{n},(\pi_{n}^{-i},\pi^{i}))=C^{i}(\eta,\pi)\geq\rho^{i}$.
So, there exist some sequence $\\{\epsilon_{n}\\}_{n\geq 1}$ contained in
$[0,1)$ with $\epsilon_{n}\rightarrow 0$ and some index $n_{0}$ for which

$C^{i}(\eta_{n},(\pi_{n}^{-i},\pi^{i}))\geq\rho^{i}-\epsilon_{n}\mathbf{1}\quad\hbox{for
all $n\geq n_{0}$}.$

By the Slater condition, we can find some
$\bar{\pi}^{i}\in\boldsymbol{\mathcal{Y}}^{i}$ and $\delta>0$ such that
$C^{i}(\eta,(\pi^{-i},\bar{\pi}^{i}))>\rho^{i}+\delta\mathbf{1}$. Again from
Corollary 4.13(iv), there is some $n_{1}\geq n_{0}$ such that

$C^{i}(\eta_{n},(\pi_{n}^{-i},\bar{\pi}^{i}))>\rho^{i}+\delta\mathbf{1}\quad\hbox{for
all $n\geq n_{1}$}.$

Observe that for any $n\geq 1$ both
$\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta_{n},(\pi_{n}^{-i},\pi^{i})}$ and
$\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta_{n},(\pi_{n}^{-i},\bar{\pi}^{i})}$
belong to $\mathcal{L}^{i}_{\eta_{n}}(\pi_{n}^{-i})$ which is a convex set by
Proposition 5.3. Hence, there exists some
$\gamma_{n}\in\mathcal{L}^{i}_{\eta_{n}}(\pi^{-i}_{n})$ and
$\sigma^{i}_{n}\in\boldsymbol{\mathcal{Y}}^{i}$ such that

$\displaystyle\gamma_{n}=\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta_{n},(\pi^{-i}_{n},\sigma^{i}_{n})}=(1-\sqrt{\epsilon_{n}})\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta_{n},(\pi^{-i}_{n},\pi^{i})}+\sqrt{\epsilon_{n}}\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta_{n},(\pi^{-i}_{n},\bar{\pi}^{i})}$
(5.3)

with, as a consequence of Proposition 4.6(ii),

$\mu_{\eta_{n},(\pi^{-i}_{n},\sigma^{i}_{n})}=\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta_{n},(\pi^{-i}_{n},\sigma^{i}_{n})}\otimes\pi^{-i}_{n}=(1-\sqrt{\epsilon_{n}})\mu_{\eta_{n},(\pi^{-i}_{n},\pi^{i})}+\sqrt{\epsilon_{n}}\mu_{\eta_{n},(\pi^{-i}_{n},\bar{\pi}^{i})}$

and so for any $n\geq n_{1}$

$\displaystyle C^{i}(\eta_{n},(\pi^{-i}_{n},\sigma^{i}_{n}))$ $\displaystyle=$
$\displaystyle(1-\sqrt{\epsilon_{n}})C^{i}(\eta_{n},(\pi^{-i}_{n},\pi^{i}))+\sqrt{\epsilon_{n}}C^{i}(\eta_{n},(\pi^{-i}_{n},\bar{\pi}^{i}))$
$\displaystyle\geq$
$\displaystyle\rho^{i}+\sqrt{\epsilon_{n}}\big{[}\delta-(1-\sqrt{\epsilon_{n}})\sqrt{\epsilon_{n}}\big{]}\mathbf{1}.$

Therefore, there exists some $K\geq n_{1}$ such that $n\geq K$ implies
$C^{i}(\eta_{n},(\pi^{-i}_{n},\sigma^{i}_{n}))\geq\rho^{i}$. Since
$\gamma_{n}=\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta_{n},(\pi^{-i}_{n},\sigma^{i}_{n})}\in\mathcal{L}^{i}_{\eta_{n}}(\pi^{-i}_{n})$,
this establishes precisely that
$\gamma_{n}\in\mathcal{A}^{i}_{\eta_{n},\rho^{i}}(\pi^{-i}_{n})$ for all
$n\geq K$. Finally, from (5.3) and Corollary 4.13(iii) we have that
$\lim_{n\rightarrow\infty}\gamma_{n}=\lim_{n\rightarrow\infty}\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta_{n},(\pi_{n}^{-i},\pi^{i})}=\mu_{\eta,\pi}^{\mathbf{X}\times\mathbf{A}^{i}}$.
This completes the proof. $\Box$

###### Proposition 5.5

Given any $1\leq i\leq N$, the correspondence
$\mathcal{A}^{i}_{\eta,\rho^{i}}:\boldsymbol{\mathcal{Y}}^{-i}\twoheadrightarrow\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{i}$
defined by $\pi^{-i}\mapsto\mathcal{A}^{i}_{\eta,\rho^{i}}(\pi^{-i})$ is
continuous with nonempty convex and compact values.

Proof. The Slater condition implies that
$\mathcal{A}^{i}_{\eta,\rho^{i}}(\pi^{-i})$ is nonempty for any
$\pi^{-i}\in\boldsymbol{\mathcal{Y}}^{-i}$. To prove convexity, let
$\gamma,\gamma^{\prime}\in\mathcal{A}^{i}_{\eta,\rho^{i}}(\pi^{-i})$ and
$0\leq\alpha\leq 1$. Then, there exist
$\sigma,\sigma^{\prime}\in\boldsymbol{\mathcal{Y}}^{i}$ with such that
$\gamma=\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,(\pi^{-i},\sigma)}$ and
$\gamma^{\prime}=\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,(\pi^{-i},\sigma^{\prime})}$
and satisfying $C^{i}(\eta,(\pi^{-i},\sigma))\geq\rho^{i}$ and
$C^{i}(\eta,(\pi^{-i},\sigma^{\prime}))\geq\rho^{i}$. By convexity of
$\mathcal{L}^{i}_{\eta}(\pi^{-i})$ in Proposition 5.3, there exists some
$\sigma^{*}\in\boldsymbol{\mathcal{Y}}^{i}$ such that
$\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,(\pi^{-i},\sigma^{*})}=\alpha\gamma+(1-\alpha)\gamma^{\prime}$.
But then

$\displaystyle\mu_{\eta,(\pi^{-i},\sigma^{*})}$ $\displaystyle=$
$\displaystyle\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,(\pi^{-i},\sigma^{*})}\otimes\pi^{-i}=\alpha(\gamma\otimes\pi^{-i})+(1-\alpha)(\gamma^{\prime}\otimes\pi^{-i})$
$\displaystyle=$
$\displaystyle\alpha\mu_{\eta,(\pi^{-i},\sigma)}+(1-\alpha)\mu_{\eta,(\pi^{-i},\sigma^{\prime})},$

and so by integration of the function $c^{i}(x,a)$ with respect to these
occupation measures,

$C^{i}(\eta,(\pi^{-i},\sigma^{*}))=\alpha
C^{i}(\eta,(\pi^{-i},\sigma))+(1-\alpha)C^{i}(\eta,(\pi^{-i},\sigma^{\prime}))\geq\rho^{i},$

which establishes that $\alpha\gamma+(1-\alpha)\gamma^{\prime}$ is in
$\mathcal{A}^{i}_{\eta,\rho^{i}}(\pi^{-i})$.

The correspondence $\mathcal{A}^{i}_{\eta,\rho^{i}}$ takes values in the
compact metric space $\tilde{\boldsymbol{\mathcal{O}}}^{i}_{\eta}$ and thus,
by the Closed Graph Theorem in [1, Theorem 17.11], it is upper semicontinuous
and compact-valued if and only if its graph is closed. Suppose that we have a
convergent sequence $(\pi_{n}^{-i},\gamma_{n})$ in the graph of
$\mathcal{A}^{i}_{\eta,\rho^{i}}$ converging to some
$(\pi^{-i},\gamma)\in\boldsymbol{\mathcal{Y}}^{-i}\times\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{i}$.
We must show that $\gamma\in\mathcal{A}^{i}_{\eta,\rho^{i}}(\pi^{-i})$. For
each $n\geq 1$ there exists $\sigma_{n}\in\boldsymbol{\mathcal{Y}}^{i}$ such
that

$\displaystyle\gamma_{n}=\mu_{\eta,(\pi^{-i}_{n},\sigma_{n})}^{\mathbf{X}\times\mathbf{A}^{i}}\quad\hbox{and}\quad
C^{i}(\eta,(\pi^{-i}_{n},\sigma_{n}))\geq\rho^{i}.$ (5.4)

For some subsequence $\\{\sigma_{n^{\prime}}\\}$ of $\\{\sigma_{n}\\}$ we have
$\sigma_{n^{\prime}}\rightarrow\sigma$ for some
$\sigma\in\boldsymbol{\mathcal{Y}}^{i}$ and so using Corollary 4.13(iii) and
(iv)

$\gamma=\mu_{\eta,(\pi^{-i},\sigma)}^{\mathbf{X}\times\mathbf{A}^{i}}\quad\hbox{and}\quad
C^{i}(\eta,(\pi^{-i},\sigma))\geq\rho^{i},$

as we wanted to prove. Lower semicontinuity of the correspondence follows from
Proposition 5.4(i) and the sequential characterization of lower semicontinuity
in [1, Theorem 17.21]. $\Box$
Given some player $1\leq i\leq N$ and a Markov profile
$\pi^{-i}\in\boldsymbol{\mathcal{Y}}^{-i}$ for the remaining players, if
player $i$ chooses the Markov policy $\sigma\in\boldsymbol{\mathcal{Y}}^{i}$
then his payoff is (recall Proposition 4.6(ii))

$R^{i}(\eta,(\pi^{-i},\sigma))=\int_{\mathbf{X}\times\mathbf{A}}r^{i}d\mu_{\eta,(\pi^{-i},\sigma)}=\int_{\mathbf{X}\times\mathbf{A}}r^{i}d(\mu_{\eta,(\pi^{-i},\sigma)}^{\mathbf{X}\times\mathbf{A}^{i}}\otimes\pi^{-i}),$

and his goal is to maximize this payoff over all
$\sigma\in\boldsymbol{\mathcal{Y}}^{i}$ such that
$C^{i}(\eta,(\pi^{-i},\sigma))\geq\rho^{i}$ or, which is the same, maximize

$\int_{\mathbf{X}\times\mathbf{A}}r^{i}d(\gamma\otimes\pi^{-i})$

over all $\gamma$ belonging to $\mathcal{A}_{\eta,\rho^{i}}^{i}(\pi^{-i})$.
Based on this, we define the correspondence
$\mathcal{H}^{i}_{\eta,\rho^{i}}:\boldsymbol{\mathcal{Y}}^{-i}\twoheadrightarrow\tilde{\boldsymbol{\mathcal{O}}}^{i}_{\eta}$
given by

$\mathcal{H}^{i}_{\eta,\rho^{i}}(\pi^{-i})=\mathop{\hbox{\rm arg
max}}\limits_{\gamma\in\mathcal{A}^{i}_{\eta,\rho^{i}}(\pi^{-i})}\Big{\\{}\int_{\mathbf{X}\times\mathbf{A}}r^{i}d(\gamma\otimes\pi^{-i})\Big{\\}}.$
(5.5)

###### Proposition 5.6

For any $i\in\\{1,\ldots,N\\}$, the correspondence
$\mathcal{H}^{i}_{\eta,\rho^{i}}:\boldsymbol{\mathcal{Y}}^{-i}\twoheadrightarrow\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{i}$
is upper semicontinuous with nonempty compact and convex values.

Proof. On the graph of the correspondence $\mathcal{A}^{i}_{\eta,\rho^{i}}$
consider the function

$f_{\eta}(\pi^{-i},\gamma)=\int_{\mathbf{X}\times\mathbf{A}}r^{i}d(\gamma\otimes\pi^{-i})$

and let us prove that it is continuous. To this end, let
$\pi^{-i}_{n}\rightarrow\pi^{-i}$ in $\boldsymbol{\mathcal{Y}}^{-i}$ and
$\gamma_{n}\rightarrow\gamma$ in $\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{i}$
with $\gamma_{n}\in\mathcal{A}^{i}_{\eta,\rho^{i}}(\pi_{n}^{-i})$ and
$\gamma\in\mathcal{A}^{i}_{\eta,\rho^{i}}(\pi^{-i})$. We must show that
$f_{\eta}(\pi^{-i}_{n},\gamma_{n})\rightarrow f_{\eta}(\pi,\gamma)$, and we
will prove that this limit holds through any convergence subsequence, which
will be denoted by $\\{n\\}$ without loss of generality. There exist
$\sigma_{n},\sigma\in\boldsymbol{\mathcal{Y}}^{i}$ such that

$\gamma_{n}=\mu_{\eta,(\pi_{n}^{-i},\sigma_{n})}^{\mathbf{X}\times\mathbf{A}^{i}}\quad\hbox{and}\quad\gamma=\mu_{\eta,(\pi^{-i},\sigma)}^{\mathbf{X}\times\mathbf{A}^{i}}$

and we can also assume that $\sigma_{n}\rightarrow\sigma^{*}$ for some
$\sigma^{*}\in\boldsymbol{\mathcal{Y}}^{i}$. Using Corollary 4.13(iv) we
obtain that

$R^{i}(\eta,(\pi^{-i}_{n},\sigma_{n}))\rightarrow
R^{i}(\eta,(\pi^{-i},\sigma^{*})).$

On the other hand, by Corollary 4.13(iii),

$\gamma_{n}\rightarrow\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,(\pi^{-i},\sigma^{*})}\quad\hbox{and
so}\quad\gamma=\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,(\pi^{-i},\sigma^{*})}=\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,(\pi^{-i},\sigma)}.$

This shows that

$\mu_{\eta,(\pi^{-i},\sigma)}=\mu_{\eta,(\pi^{-i},\sigma)}^{\mathbf{X}\times\mathbf{A}^{i}}\otimes\pi^{-i}=\mu_{\eta,(\pi^{-i},\sigma^{*})}^{\mathbf{X}\times\mathbf{A}^{i}}\otimes\pi^{-i}=\mu_{\eta,(\pi^{-i},\sigma^{*})}$

and so $R^{i}(\eta,(\pi^{-i}_{n},\sigma_{n}))\rightarrow
R^{i}(\eta,(\pi^{-i},\sigma))$, which can be also written as
$f_{\eta}(\gamma_{n},\pi^{-i}_{n})\rightarrow f_{\eta}(\gamma,\pi^{-i})$. Once
we know that $f_{\eta}$ is continuous on the graph of
$\mathcal{A}^{i}_{\eta,\rho^{i}}$, we can apply the Berge Maximum Theorem [1,
Theorem 17.31] and conclude that the $\mathop{\hbox{\rm arg max}}\limits$
correspondence $\mathcal{H}^{i}_{\eta,\rho^{i}}$ is upper semicontinuous with
nonempty compact values.

Finally, observe that the function $f_{\eta}$ is linear in $\gamma$ for fixed
$\pi^{-i}\in\boldsymbol{\mathcal{Y}}^{-i}$ and so the set of maximizers
$\mathcal{H}^{i}_{\eta,\rho^{i}}(\pi^{-i})$ is convex. $\Box$
By considering the product of the correspondences
$\mathcal{H}^{i}_{\eta,\rho^{i}}$ we obtain the following result, which easily
follows from [1, Theorem 17.28].

###### Corollary 5.7

The correspondence
$\mathcal{H}_{\eta,\rho}:\boldsymbol{\mathcal{Y}}\twoheadrightarrow\tilde{\boldsymbol{\mathcal{O}}}^{1}_{\eta}\times\ldots\times\tilde{\boldsymbol{\mathcal{O}}}^{N}_{\eta}$
defined by

$\pi\mapsto\prod_{i=1}^{N}\mathcal{H}_{\eta,\rho}^{i}(\pi^{-i}).$

is upper semicontinuous with nonempty compact and convex values.

#### Proof of Proposition 3.1:

To get the result, let us show that the following results hold.

  * (i)

The correspondence

$\mathcal{H}_{\eta,\rho}\circ\mathcal{J}_{\eta}:\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{1}\times\ldots\times\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{N}\twoheadrightarrow\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{1}\times\ldots\times\tilde{\boldsymbol{\mathcal{O}}}_{\eta}^{N}$

has a fixed point $(\gamma_{*}^{1},\ldots,\gamma^{N}_{*})$.

  * (ii)

The Markov profile $\pi_{*}\in\boldsymbol{\mathcal{Y}}$ given by
$\pi_{*}^{i}=\mathcal{J}^{i}_{\eta}(\gamma_{*}^{i})$ for $1\leq i\leq N$ is a
constrained equilibrium in the class of all strategy profiles $\mathbf{\Pi}$
of the players for the game model $\mathcal{G}(\eta,\rho)$.

Let us first proceed to the proof of item (i). By [1, Theorem 17.23], the
composition $\mathcal{H}_{\eta,\rho}\circ\mathcal{J}_{\eta}$ is an upper
semicontinuous correspondence as it is the composition of a continuous
function $\mathcal{J}_{\eta}$ and an upper semicontinuous correspondence
$\mathcal{H}_{\eta,\rho}$. Besides, it has nonempty compact and convex values.
By the Closed Graph Theorem [1, Theorem 17.11], it is also a closed
correspondence.

Since
$\tilde{\boldsymbol{\mathcal{O}}}^{1}_{\eta}\times\ldots\times\tilde{\boldsymbol{\mathcal{O}}}^{N}_{\eta}$
is a nonempty compact convex subset of the locally convex Hausdorff space
$\boldsymbol{\mathcal{M}}(\mathbf{X}\times\mathbf{A}^{1})\times\ldots\times\boldsymbol{\mathcal{M}}(\mathbf{X}\times\mathbf{A}^{N})$
—Proposition 2.2 in [11]— we can use the Kakutani-Fan-Glicksberg fixed point
theorem [1, Corollary 17.55] to get the existence of a fixed point for the
correspondence $\mathcal{H}_{\eta,\rho}\circ\mathcal{J}_{\eta}$.
Let us now proceed to the proof of item (ii). If
$(\gamma_{*}^{1},\ldots,\gamma_{*}^{N})$ is a fixed point of
$\mathcal{H}_{\eta,\rho}\circ\mathcal{J}_{\eta}$, consider the Markov policies
$\pi_{*}^{i}=\mathcal{J}^{i}_{\eta}(\gamma^{i}_{*})\in\boldsymbol{\mathcal{Y}}^{i}$
for $1\leq i\leq N$ and let
$\pi_{*}=(\pi^{1}_{*},\ldots,\pi^{N}_{*})\in\boldsymbol{\mathcal{Y}}$. Since
$\gamma^{i}_{*}\in\mathcal{L}^{i}_{\eta}(\pi_{*}^{-i})$ we have that for some
$\sigma\in\boldsymbol{\mathcal{Y}}^{i}$ it is
$\gamma_{*}^{i}=\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,(\pi_{*}^{-i},\sigma)}$.
But now using Lemma 5.1(iii) we also have

$\gamma_{*}^{i}=\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,(\pi_{*}^{-i},\sigma)}=\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,(\pi_{*}^{-i},\mathcal{J}_{\eta}^{i}(\gamma^{i}_{*}))}=\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,\pi^{*}}.$

In particular, for each $1\leq i\leq N$ we have
$\gamma^{i}_{*}\otimes\pi_{*}^{-i}=\mu_{\eta,\pi_{*}}$. Moreover, since
$\gamma_{*}^{i}\in\mathcal{A}^{i}_{\eta,\rho^{i}}(\pi^{-i}_{*})$ it follows
that $C^{i}(\eta,\pi_{*})\geq\rho^{i}$. We conclude that the Markov profile
$\pi_{*}\in\boldsymbol{\mathcal{Y}}$ satisfies the constraints of all the
players.

If player $i$ varies his policy from
$\pi_{*}^{i}\in\boldsymbol{\mathcal{Y}}^{i}$ to some
$\pi^{i}\in\mathbf{\Pi}^{i}$ which satisfies his own constraint (i.e.
$C^{i}(\eta,(\pi_{*}^{-i},\pi^{i}))\geq\rho^{i}$) we can use the result in
Proposition 4.6(iii) to derive the existence of some
$\sigma\in\boldsymbol{\mathcal{Y}}^{i}$ such that

$\mu_{\eta,(\pi_{*}^{-i},\pi^{i})}=\mu_{\eta,(\pi_{*}^{-i},\sigma)}$

with, again, $C^{i}(\eta,(\pi_{*}^{-i},\sigma))\geq\rho^{i}$. This implies
that
$\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,(\pi_{*}^{-i},\sigma)}\in\mathcal{A}^{i}_{\eta,\rho^{i}}(\pi_{*}^{-i})$
and thus

$\int_{\mathbf{X}\times\mathbf{A}}r^{i}d(\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,(\pi_{*}^{-i},\sigma)}\otimes\pi_{*}^{-i})\leq\int_{\mathbf{X}\times\mathbf{A}}r^{i}d(\gamma_{*}^{i}\otimes\pi_{*}^{-i})$

or, equivalently,

$R^{i}(\eta,(\pi^{-i}_{*},\pi))=R^{i}(\eta,(\pi_{*}^{-i},\sigma))\leq
R^{i}(\eta,\pi_{*}).$

This completes the proof. $\Box$

###  5.2 Proof of Theorem 3.2

Clearly, for all $n\geq 1$ we have $\lambda\ll\eta_{n}$. We also have that
$\\{\eta_{n}\\}_{n\in\mathbb{N}}$ converges to $\eta$ in total variation and
that the corresponding density functions with respect to $\lambda$ converge
strongly (or, in norm) in
$L^{1}(\mathbf{X},\boldsymbol{\mathfrak{X}},\lambda)$:
$\displaystyle\left\|{d\eta_{n}}/{d\lambda}-{d\eta}/{d\lambda}\right\|_{1}\rightarrow
0$. Since the constraint function $c_{i}$ is bounded by $\mathbf{r}$, we have
$|C^{i}(\eta_{n},\pi)-C^{i}(\eta,\pi)|\leq\mathbf{r}/(n+1)$. Also, for the
constraint constants $\rho_{n}^{i}=\rho^{i}-\frac{\mathbf{r}}{n+1}\mathbf{1}$
with $1\leq i\leq N$, the game model $\mathcal{G}(\eta_{n},\rho_{n})$
satisfies the Slater condition in Definition 2.6. Under assumptions
$(\mathrm{A}^{\prime}_{1})$ and $(\mathcal{A}_{7})$, we obtain that the game
model $\mathcal{G}(\eta_{n},\rho_{n})$ is uniformly absorbing to $\Delta$ by
using item (ii) of Proposition 2.4. We can conclude that the game model
$\mathcal{G}(\eta_{n},\rho_{n})$ satisfies Assumptions 2.3 and 2.3 and so,
Proposition 3.1 yields the existence of a constrained Nash equilibrium
$\hat{\pi}_{n}\in\boldsymbol{\mathcal{Y}}$ for the game model
$\mathcal{G}(\eta_{n},\rho_{n})$ with $n\geq 1$. This means that

$C^{i}(\eta_{n},\hat{\pi}_{n})\geq\rho^{i}_{n}\quad\hbox{for $1\leq i\leq N$}$
(5.6)

and that, for any $1\leq i\leq N$ and $\pi^{i}\in\mathbf{\Pi}^{i}$,

$C^{i}(\eta_{n},(\hat{\pi}_{n}^{-i},\pi^{i}))\geq\rho^{i}_{n}\ \Rightarrow\
R^{i}(\eta_{n},(\hat{\pi}_{n}^{-i},\pi^{i}))\leq
R^{i}(\eta_{n},\hat{\pi}_{n}).$

Without loss of generality, we assume that the sequence of so-defined
equilibria converges to some $\hat{\pi}\in\boldsymbol{\mathcal{Y}}$, that is,
for each $1\leq i\leq N$ we have $\hat{\pi}_{n}^{i}\rightarrow\hat{\pi}^{i}$
in $\boldsymbol{\mathcal{Y}}^{i}$ as $n\rightarrow\infty$. We want to show
that $\hat{\pi}$ is a constrained equilibrium for the game model
$\mathcal{G}(\eta,\rho)$. To see this, note first that we can take the limit
in (5.6) to obtain that $C^{i}(\eta,\hat{\pi})\geq\rho^{i}$ for every $1\leq
i\leq N$ by using Corollary 4.13(iv). Secondly, fix $i\in\\{1,\ldots,N\\}$ and
choose any $\pi^{i}\in\mathbf{\Pi}^{i}$ such that
$C^{i}(\eta,(\hat{\pi}^{-i},\pi^{i}))\geq\rho^{i}$. By Proposition 4.6(iii) it
follows that there is some $\sigma\in\boldsymbol{\mathcal{Y}}^{i}$ such that
$(\hat{\pi}^{-i},\pi^{i})$ and
$(\hat{\pi}^{-i},\sigma)\in\boldsymbol{\mathcal{Y}}$ yield the same payoffs
$C^{i}$ and $R^{i}$. Hence we have
$C^{i}(\eta,(\hat{\pi}^{-i},\sigma))\geq\rho^{i}$ and we must show that

$R^{i}(\eta,(\hat{\pi}^{-i},\sigma))\leq R^{i}(\eta,\hat{\pi}).$

We will use Proposition 5.4(ii) for the Markov profile
$(\pi_{*}^{-i},\sigma)\in\boldsymbol{\mathcal{Y}}$ and the sequence
$\\{\hat{\pi}_{n}^{-i}\\}$ to derive the existence of a sequence
$\gamma_{n}\rightarrow\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,(\hat{\pi}^{-i},\sigma)}$
such that
$\gamma_{n}\in\mathcal{A}^{i}_{\eta_{n},\rho^{i}}(\hat{\pi}_{n}^{-i})$ for
large enough $n\geq K$. So, for such $n\geq K$, let
$\sigma_{n}\in\boldsymbol{\mathcal{Y}}^{i}$ be such that
$\gamma_{n}=\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta_{n},(\hat{\pi}_{n}^{-i},\sigma_{n})}$
which satisfies
$C^{i}(\eta_{n},(\hat{\pi}_{n}^{-i},\sigma_{n}))\geq\rho^{i}\geq\rho^{i}_{n}$.
This implies that for any $n\geq K$ we have

$R^{i}(\eta_{n},(\hat{\pi}_{n}^{-i},\sigma_{n}))\leq
R^{i}(\eta_{n},\hat{\pi}_{n}).$

There exists some $\bar{\sigma}\in\boldsymbol{\mathcal{Y}}^{i}$ and a
subsequence of $\\{\sigma_{n}\\}$ (still denoted by $\\{\sigma_{n}\\}$)
satisfying $\sigma_{n}\rightarrow\bar{\sigma}$ in
$\boldsymbol{\mathcal{Y}}^{i}$ and then taking the limit we have

$R^{i}(\eta,(\hat{\pi}^{-i},\bar{\sigma}))\leq R^{i}(\eta,\hat{\pi})$

by using Corollary 4.13(iv). But then item (iii) of Corollary 4.13 implies
that

$\gamma_{n}=\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta_{n},(\hat{\pi}_{n}^{-i},\sigma_{n})}\rightarrow\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,(\hat{\pi}^{-i},\bar{\sigma})}=\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,(\hat{\pi}^{-i},\sigma)}$

so that

$\mu_{\eta,(\hat{\pi}^{-i},\bar{\sigma})}=\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,(\hat{\pi}^{-i},\bar{\sigma})}\otimes\hat{\pi}^{-i}=\mu^{\mathbf{X}\times\mathbf{A}^{i}}_{\eta,(\hat{\pi}^{-i},\sigma)}\otimes\hat{\pi}^{-i}=\mu_{\eta,(\hat{\pi}^{-i},\sigma)}$

and, hence,
$R^{i}(\eta,(\hat{\pi}^{-i},\bar{\sigma}))=R^{i}(\eta,(\hat{\pi}^{-i},\sigma))$
and $R^{i}(\eta,(\hat{\pi}^{-i},\sigma))\leq R^{i}(\eta,\hat{\pi})$ follows.

###  5.3 Proof of Corollary 3.6

(i). To check this result, we must show that the convergences
$\pi^{i}_{n}\rightarrow\pi^{i}$ in $\boldsymbol{\mathcal{Y}}^{i}$ for each
$1\leq i\leq N$ imply that

$\pi_{n}(da|x)=\pi^{1}_{n}(da^{1}|x)\times\cdots\times\pi^{N}_{n}(da^{N}|x)\rightarrow\pi^{1}(da^{1}|x)\times\cdots\times\pi^{N}(da^{N}|x)=\pi(da|x)\quad\hbox{in
$\tilde{\boldsymbol{\mathcal{Y}}}$}.$