# Developing, Evaluating and Scaling Learning Agents in Multi-Agent
Environments

I.Ian<EMAIL_ADDRESS>[ D.Thomas
<EMAIL_ADDRESS>[ Y.Yoram
<EMAIL_ADDRESS>[ A.Avishkar
<EMAIL_ADDRESS>[ J.Kalesha
<EMAIL_ADDRESS>[ J.Jerome
<EMAIL_ADDRESS>[ V.Vibhavari
<EMAIL_ADDRESS>[ B.Bart De
<EMAIL_ADDRESS>[ E.Edgar A. Duéñez-
<EMAIL_ADDRESS>[ R.Romuald<EMAIL_ADDRESS>[ R.Richard<EMAIL_ADDRESS>[ D.Daniel
<EMAIL_ADDRESS>[ E.Edward
<EMAIL_ADDRESS>[ M.Mina
<EMAIL_ADDRESS>[ M.Marc
<EMAIL_ADDRESS>[ K.Kate
<EMAIL_ADDRESS>[ G.Guy
<EMAIL_ADDRESS>[ S.Siqi<EMAIL_ADDRESS>[ L.Luke<EMAIL_ADDRESS>[ K.R.Kevin R.
<EMAIL_ADDRESS>[ P.Paul
<EMAIL_ADDRESS>[ J.Julien
<EMAIL_ADDRESS>[ F.Florian
<EMAIL_ADDRESS>[ A.Andrea
<EMAIL_ADDRESS>[ E.Eugene
<EMAIL_ADDRESS>[ Z.Zhe<EMAIL_ADDRESS>[
K.Karl<EMAIL_ADDRESS>Corresponding author. .[ Game
Theory & Multi-Agent Team, DeepMind, London, UKpresep=
]e1

(0000)

###### Abstract

The Game Theory & Multi-Agent team at DeepMind studies several aspects of
multi-agent learning ranging from computing approximations to fundamental
concepts in game theory to simulating social dilemmas in rich spatial
environments and training 3-d humanoids in difficult team coordination tasks.
A signature aim of our group is to use the resources and expertise made
available to us at DeepMind in deep reinforcement learning to explore multi-
agent systems in complex environments and use these benchmarks to advance our
understanding. Here, we summarise the recent work of our team and present a
taxonomy that we feel highlights many important open challenges in multi-agent
research.

Game Theory,

Multi-Agent,

Reinforcement Learning,

Equilibrium,

Mechanism Design,

###### keywords:

††volume: 0

##  1 Introduction

While multi-agent research pervades much of the work at DeepMind, largely due
to work on solving grand challenges in games [(1), (2)], the Game Theory &
Multi-Agent team, in particular, focuses on key aspects of multi-agent
research. From our perspective, multi-agent research extends along several
axes including

  * •

Number of Players: 2, >2, many, infinite

  * •

Players: agents, agents + humans

  * •

Payoffs / Incentives: competitive, mixed-motive, cooperative

  * •

States: Discrete, Continuous

  * •

Time: One-shot, Repeated / Iterated, Extensive-form, Infinite

  * •

Observability: Perfect Information, Imperfect Information

  * •

Inspiration: Game Theory, Evolution, Sociology.

Individual projects often work in settings or domains that vary along several
of these dimensions at once. In the following sections, we will describe the
areas of research prioritised within the group. These include the computation
of fundamental solution concepts from game theory such as equilibria in §2,
the development of key multi-agent skills like negotiation in §3, the ability
to shape the outcomes of multi-agent systems via mechanism design in §4,
novel, game-theoretic approaches to evaluating performance of multi-agent
systems in §5, approaches inspired by nature and the success of human cultural
evolution in §6, the formulation of fundamental problems in machine learning
as games in §7, the development and pursuit of solving challenging benchmarks
to further push multi-agent research in §8, and multi-agent research aimed at
solving current real world problems in cooperation in §9.

##  2 Computing Equilibria / Solution Concepts

This line of work focuses primarily on selecting strategies based on outcomes
in a game as opposed to strategies selected according to behavioural criteria
(discussed next in §3). This includes the famous equilibrium concepts of Nash
equilibrium (NE), correlated equilibrium (CE), and coarse-correlated
equilibrium (CCE) but also others of our own design. We also differentiate
between two-player, zero-sum games where these equilibrium concepts are well
understood and $n$-player, general-sum games where they serve as a guide
toward strategic play but do not necessarily provide a full-answer. Our group
is particularly interested in exploring these equilibrium concepts and others
in the $n$-player, general-sum setting. By studying these concepts empirically
in a variety of games, we expect to generate insights that will extend our
understanding of equilibrium concepts and performant strategies in $n$-player,
general-sum games.

###  2.1 Two Player, Zero-Sum

In two-player, zero-sum games, Nash equilibrium is widely accepted as a
suitable solution concept. While several algorithms offer convergence
guarantees to a Nash equilibrium in simple game classes such as normal-form
games, there are still opportunities for research in more complex classes such
as imperfect information games.

One approach we take is to design an algorithm to maximise reward against a
worst-case opponent [(3)] or an improved opponent [(4)] with final iterate
convergence to Nash equilibria. Other methods draw inspiration from
counterfactual regret minimization [(5), (6)] and replicator dynamics [(7)],
whose time average converges to Nash equilibria.

In [(8)], we show that the popular naive learning approach of Follow the
Regularized Leader (FoReL) cycles and fails to converge to the Nash
equilibrium in sequential, imperfect information games. However, if we modify
the players’ payoffs by adding a regularization term that we anneal over
iterations, we can recover convergence guarantees of the last iterate. We call
this approach _friction_ -FoReL or _F-FoReL_ and demonstrate that this
approach can be used to train state-of-the-art model-free reinforcement
learning agents for two-player, zero-sum, sequential, imperfect information
games.

###  2.2 N-Player, General-Sum

In games with more than two players and in general-sum games, it is less clear
what optimal play entails. There are several solution concepts that are still
well defined in this setting and have been studied in various contexts. Our
group works on computing these solution concepts and their variants as well as
defining new ones.

####  2.2.1 Nash Equilibria

ADIDAS [(9)] stochastically approximates the Nash equilibrium of a normal-form
game with many players, each with many actions. It does this by tracing a
homotopy of quantal response equilibria defined with decaying levels of noise
disturbance (equivalently, entropy bonus). For each entropy bonus level, the
Nash equilibrium of the perturbed game is approximated by minimising a
suitable energy function (a measure of $\epsilon$-Nash) with gradient descent.
Critically, individual entries of the payoff tensor are only sampled as needed
obviating the need to store the entire tensor in memory. This approach was
used to approximate a unique Nash equilibrium (the limiting logit equilibrium)
of a game with 7-players and 21-actions amounting to a payoff tensor with over
12 billion entries.

####  2.2.2 (Coarse) Correlated Equilibria

Maximum Gini (Coarse) Correlated Equilibrium (MG(C)CE) [(10)] is a Tsallis-
entropy-maximising equilibrium selection criterion. MG(C)CE uniquely selects
an equilibrium from the convex polytope of valid (C)CE solutions and is
invariant to positive affine transformations of the payoffs. It is easy to
solve via existing off-the-shelf quadratic programming optimisers. MG(C)CE has
a particularly compact dual representation if it selects a full-support
solution.

####  2.2.3 Stationary Distributions of Fundamental Dynamics

_Replicator dynamics_ is a key concept from evolutionary game theory that
describes the dynamics of a population of strategies over time. In essence,
they have been shown to describe a learning process and various reinforcement
learning algorithms have been shown to converge in the limit to some version
of a replicator dynamic [(11), (12), (13)]. As such, replicator dynamics serve
as a formal basis to understand the behaviour of various multi-agent learning
algorithms and as a basis to develop new multi-agent learning algorithms
[(14), (7)]. We study the properties of the stationary distribution of these
dynamics and how their equilibria might serve as a tractable solution concept
in large games [(15)].

The main advantages of this new concept concerns its uniqueness and efficient
computation in many-player and general-sum games, making it a promising
foundation for new multi-agent learning algorithms. The $\alpha$-regularised
replicator dynamics define an irreducible Markov chain over a strategy set,
which is called the _response graph_ of the game [(16)], by indicating when a
player has an incentive to make a unilateral deviation from their current
strategy. The ordered masses of this Markov chain’s unique stationary
distribution yields the solution concept. The Markov transition matrix is
directly linked to a solution concept called Markov-Conley chains (MCC) [(15),
(17)]. The correspondence of the stationary distribution to the MCC solution
concept occurs in the limit of infinite $\alpha$.

###  2.3 Infinite Number of Players

As solving games with a high number of players becomes quickly intractable, an
innovative approach for anonymous symmetric games considers instead the
asymptotic limit where the number of players becomes infinite. This gives rise
to the so-called Mean Field Games introduced in [(18), (19)] with numerous
applications in crowd dynamics [(20)], finance [(21)], epidemiology [(22)] or
energy management [(23)] among others. Under structural conditions, a Nash
policy of the Mean Field Game provides a good approximation for the solution
of the corresponding $n$-player game, with an error vanishing as $n$ goes to
infinity. For this reason, learning an equilibrium in Mean Field Games shall
pave the way for scalable algorithms in multi-agent systems with a large
population of players.

We introduced and studied the convergence to a Nash equilibrium of several
scalable learning algorithms for Mean Field Games, together with more specific
applications to flocking [(24)] or vehicles traffic routing management [(25)].
Our algorithms rely on Fictitious Play [(26), (27)] or Online Mirror Descent
algorithms [(28)] and can be efficiently combined with Deep Reinforcement
Learning [(29)]. Using a well chosen enlarged class of policies, pinned as
_master policies_ , allows us to generalise efficiently to several initial
population distributions [(30)]. All known Nash-converging algorithms require
some conditions on the game in order to work—monotonicity and contractivity
being the most popular. Mean-Field PSRO [(31)] provides a method to converge
to a Mean Field Nash equilibrium, correlated and coarse-correlated equilibria
in _all games_ through the use of accelerated adversarial regret minimization.

###  2.4 Inspired Approaches for Very Large (Infinite) Strategy Spaces

Much of our work into training agents to play games is inspired by iterative
self-play approaches like Fictitious Play [(32)] and Double-Oracle (DO)
[(33)]. We have extended these approaches into settings where strategies are
represented by neural networks. In this case, the strategy space of each agent
is the space of learnable parameters which is typically $\mathbb{R}^{d}$, and
therefore infinite. Nevertheless, we have found that self-play approaches with
approximate best responses perform well despite their lack of guarantees.

Early approaches that adapted to the multi-agent reinforcement learning
setting were based on Fictious Play [(34), (35)]. A subsequent approach,
policy-space response oracles (PSRO) [(36)] extends Double-Oracle by
approximately computing best responses with reinforcement learning. In more
detail, PSRO constructs _meta-games_ where each action in the meta-game
corresponds to a best response (“oracle”) policy, typically represented by a
neural network. At each step, a Nash equilibrium of this meta-game is
computed, after which a best response is approximated with (single-agent) deep
reinforcement learning (RL) and then added to the meta-game strategy set.

$\alpha$-PSRO [(37)] alters the core algorithm of PSRO by replacing Nash
equilibirum as the relevant meta-game solution concept with the stationary
distribution of replicator dynamics developed in $\alpha$-Rank [(15)]. We show
this change allows $\alpha$-PSRO to converge towards this particular
stationary distribution, and demonstrate this empirically.

Similarly, (JPSRO) Joint PSRO [(10)] replaces the Nash equilibrium with a
(C)CE in an $n$-player, general-sum normal-form game. It benefits from the
scaling properties of PSRO while converging to a (C)CE, a solution concept
more suitable for cooperating agents.

One drawback of standard DO / PSRO approaches are that they scale poorly in
real-world games that call for approximate best-response operators such as
deep RL. Neural Population Learning (NeuPL, [(38)]) makes progress towards
bringing game-theoretic population learning algorithms to large games,
leveraging the expressiveness of neural networks. Specifically, a single
conditional network is used to explore, represent and optimise a “live"
population of strategies that are concurrently updated to best respond to
mixed subsets of the population. By virtue of this shared representation,
NeuPL enables positive transfer across strategies yet retains convergence
guarantees to solution concepts such as NE. At convergence, NeuPL only retains
a sequence of exact best-responses, yielding a compact representation of the
strategically relevant policy space of the game.

Best-response policy iteration (BRPI) is a family of policy iteration methods
[(39), (1), (40)] that use an approximate best response calculator for policy
improvement. Members of this family include different approximations to
fictitious play. We applied BRPI to the board game Diplomacy [(41)]. This
domain has a combinatorial action space, meaning that a player has to choose
its own action from many possibilities (median number of possibilities in
movement phases is $10^{45.8}$), and evaluate its performance against many
more. We introduced sampled best response, an approximate best response
calculation technique, to scale BRPI to this game.

##  3 Developing Multi-Agent Skills

While solution concepts provide an axiomatic approach to selecting strategies
in games, they currently do not provide a full satisfactory answer, as we
mentioned, in the $n$-player, general-sum setting and in some cases, are
intractable to compute. Given this shortcoming, several general strategies or
skills have been identified as critical for navigating the complex
interactions of multi-agent systems, for example, negotiation and team-
formation. Developing these multi-agent skills provides agents with abilities
to help them perform well in a variety of settings including interactions with
humans.

###  3.1 Team Formation and Alliances

Cooperation between intelligent agents requires reasoning about what other
agents might do, finding potential partners, forming teams and a joint plan of
actions and negotiating how to share gains resulting from the collaboration.
The game of Diplomacy [(42)] is a prominent AI challenge where identifying a
mutually beneficial joint plan is crucial to winning the game. We have shown
how reinforcement learning methods can deal with the large action spaces that
occur in this game by applying a custom improvement operator that considers
the actions that others might take [(41)].

Beyond modeling the actions of other agents, organising into teams or
coalitions provides structure that can facilitate collective action, even for
a population of self-interested agents. Towards this end, we have examined
direct mechanisms for negotiating team formation and showed that reinforcement
learning can allow agents to construct coalitions and share the gains in a
fair manner, similar to power indices from cooperative game theory [(43)].

The process of bilateral alliance formation induces a social dilemma between
the parties. Teaming up with peers requires an element of trust when peers
might be incentivised to defect from a joint plan so as to increase their
reward at the expense of others. We have found that independent multi-agent
reinforcement learning algorithms fail to discover beneficial alliances in
simple models of economic competition. When the agents are augmented with a
peer-to-peer contract mechanism they learn to discover and enforce alliances
[(44)].

###  3.2 Evolution of Cooperation

In order for multi-agent systems to survive in diverse and unknown
environments, they must learn when and how to cooperate. In [(45)], we draw on
interdependence theory from social psychology and imbue reinforcement learning
agents with Social Value Orientation (SVO), a flexible formalization of
preferences over group outcome distributions. We demonstrate that
heterogeneity in SVO generates meaningful and complex behavioural variation
among agents, matching predictions made by interdependence theory. Empirical
results in mixed-motive dilemmas suggest agents trained in heterogeneous
populations develop particularly generalised, high-performing policies
relative to those trained in homogeneous populations. Given the high fitness
of diverse populations, it is possible evolutionary pressure selects for such
heterogeneity.

In other work [(46)], we aim for agents to learn how to share rewards through
interaction with their multi-agent environment. To derive such a learning
rule, we first formulate a tractable measure of “price of anarchy”, a
technical, game-theoretic definition that quantifies group-level
inefficiency—it compares the welfare that can be achieved through perfect
coordination against that achieved by self-interested agents at a Nash
equilibrium. In the process of constructing a method to minimise an upper
bound on this measure, we recover a variant of the celebrated Win-Stay, Lose-
Shift strategy from behavioural game theory, thereby establishing a connection
between the global goal of maximum welfare and an established agent-centric
learning rule. We demonstrate that this method improves outcomes for each
agent and the group as a whole in several social dilemmas.

###  3.3 Coordination

A desirable property of our agents is that they are able to coordinate and
collaborate well with novel partners. This is particularly relevant to
robotics and AI assistants which are meant to work with, or augment, humans in
real-world tasks. Zero-shot coordination (ZSC) examines the problem of
enabling agents to generalise their coordination strategy to held-out
partners, for example other independently trained agents or humans.

The game of Overcooked simulates a collaborative cooking environment and has
recently been proposed as a coordination challenge for AI [(47)]. For this
domain, we introduced Fictitious Co-Play (FCP) [(48)], a state-of-the-art
method which trains an agent to collaborate with a diverse set of training
partners. FCP was able to show strong performance when paired with both novel
AI agents and human partners. Notably, our method was able to achieve this
performance without the expensive process of collecting human data for the
training process. This is a promising result towards efficiently producing
agents which are capable of assisting and cooperating with humans in novel
contexts.

###  3.4 Agents that Model Social Opponents and Norms

In order to design agents capable of interacting with humans in society, it is
imperative that they be able to learn how to interpret and adapt to human
behaviour.

In zero-sum games, there are often intransitivities in strategy space [(49)]
(e.g., rock beats scissors beats paper beats rock). Modelling and
understanding other players’ incentives becomes critical to appropriately and
quickly adapt to unknown opponents. We have developed techniques inspired by
hierarchical reinforcement learning to train agents that can infer other
agents’ policies and best respond to them as zero-shot generalisation [(50),
(51), (52)]. These techniques are a form of centralised training and
decentralised execution, where privileged information is only available in
hindsight to aid the training of representations.

Beyond modeling of individual motivations, agents embedded in a social context
need to understand and appropriately respond to society-wide phenomena like
norms. Our research has developed new algorithms to imbue our agents with the
ability to adapt to norms [(53)], as well as using these agents as models of
human behaviour to understand the emergence of norms and institutions
(including those that appear arbitrary or maladaptive) from first principles
[(54)].

###  3.5 Imitating other Agents

Human intelligence is especially dependent on our ability to acquire knowledge
efficiently from other humans. This knowledge is collectively known as
culture, and the transfer of knowledge from one individual to another is
cultural transmission. Cultural transmission leads to a feedback loop
bootstrapping individual and collective abilities, which in 12 millennia has
taken us from hunting and gathering to international video calling. This
suggests we endow agents with the ability to transmit their knowledge across
the multi-agent system, so that subsequent “generations" may grow in
intelligence.

Inspired by this, we have used deep reinforcement learning to generate
artificial agents capable of test-time cultural transmission [(55)],
particularly focusing on real-time third-person imitation of other agents. Our
trained agent can infer and recall navigational knowledge demonstrated by
experts, purely via a real-time experience stream from a multi-agent RL
environment. This knowledge transfer generalises across a vast space of
previously unseen tasks. For example, our agents quickly learn new behaviours
by observing a single human demonstration, without ever training on human
data. This paves the way for cultural evolution as an algorithm for developing
more generally intelligent artificial agents.

###  3.6 Language and Communication

Language has been mostly explored through a data-driven paradigm over the last
decade (rae2021scaling, ). However, language, and more broadly communication,
is intrinsically a multi-agent problem: communication requires at least two
interacting agents per se. In this spirit, our team focused on two facets of
language in multi-agent settings: language emergence and improving language
processing methods.

Language emergence explores how a communication protocol may emerge and evolve
when multiple agents must cooperate to solve a task. From a scientific
perspective, such computational approaches aim to simulate the prerequisite or
processes that could trigger the emergence of a structured language and shed
light on human language evolution. From a machine learning perspective, it
provides a novel benchmark to analyze agent dynamics and human-machine
interactions. In this spirit, we explore how a large population may entail
different language properties, either emphasising the importance of population
heterogeneity rita2021role  or population dynamics in large-scale experiments
chaabouni2021emergent . We also considered the impact of embodied
communication, i.e., when communication occurs within small interactive
worlds. There, we highlight how simple human biases help promote communication
in cooperative tasks eccles2019biases ; kalinowska2022situated .

On the other hand, game-theoretic solutions can also be used to build better
machine learning models for natural language processing systems. For instance,
we show that vocabulary selection can be cast as a team formation game between
the possible words. Power indices from, e.g., the Shapley value and Banzhaf
index cooperative game, allow finding small vocabularies, significantly
reducing memory and compute resources while retaining high performance.
Concurrently, we show that using strategies to dynamically prune vocabulary
allow training a language model from scratch by solely relying on interaction
donati2021learning .

##  4 Shaping Outcomes of Multi-Agent Systems

In some games, there may be a strong asymmetry between player payoffs /
strategy spaces where some players have privileged action over others. This
includes settings such as auction design where a player called the _designer_
specifies the rules of an auction in order to elicit certain behaviour from
the remaining players called _participants_. Seminal results in auction design
typically study settings with finite numbers of discrete outcomes. In our
group, given DeepMind’s work on deep reinforcement learning, we studying
mechanism design in the multi-agent reinforcement learning setting where
agents might have preferences over the weight space of neural networks (i.e.,
items are points in $\mathbb{R}^{d}$). The amount of asymmetry between players
varies over domains we study.

###  4.1 Static-Mechanisms

In some settings, we learn a static mechanism once and deploy it. For
instance, we have shown how neural networks can be used as a language for
expressing agent preferences where these preferences are given over continuous
spaces bachrachneural . This enables applying frameworks from the field of
mechanism design, such as the VCG auction milgrom2004putting  to reach
decisions that maximise the welfare of all the agents.

Auctions are the protocol of choice to allocate goods to strategic buyers that
have preferences over them. While researchers have proposed reliable auction
rules that work in extremely general settings, these protocols may require
extracting high payments from participants so as to ensure incentive
constraints are maintained. This may not always be desirable, particularly in
situations where the "auctioneer" is only interested in the efficient
allocation of resources and does not care about revenue. By casting auction
rule design as a statistical learning problem, we trade generality for
participant welfare effectively and automatically, using a novel deep learning
network architecture and auction representation tacchetti22learning . Our
analysis shows that the resulting auction rules outperform state-of-the art
approaches in terms of participants’ welfare, applicability, robustness.

We have also analysed how reinforcement learning agents can be employed as the
“inner loop” in an empirical approach to mechanism design. Many real-world
tasks have spatial and temporal intricacies that cannot readily be reduced to
matrix games. In bakker21 , we examined how interventions on geometry, cost,
specialization and topology affected the emergence of reciprocal care in a
spatialised, temporally-extended network game. The effects of mechanism choice
on agent learning were subtle and sometimes counter-intuitive, pointing at a
need for more detailed modelling of mechanistic interventions, especially in
an increasingly socio-technical world.

###  4.2 Dynamic-Mechanisms

In other work, we deploy a mechanism online which must learn from
participants’ behaviour and adjust itself on the fly in order to elicit the
desired outcomes. This setting brings several additional challenges including
complex effects of hysteresis where the rules of the mechanism early in the
lifetime of agents affects their behaviour later in their lifetime.

Evaluation in multi-agent settings focuses primarily on the interaction among
fixed, non-learning co-players. While this evaluation methodology has merit,
it fails to capture the dynamics faced by agents interacting with continually
learning co-players. The Good Shepherd balaguer22shepherd  addresses this
limitation, and constructs agents (“mechanisms”) that perform well when
evaluated over the learning trajectory of their adaptive co-players
(“participants”). The algorithm consists of two nested learning loops: an
inner loop where participants learn to best respond to fixed mechanisms; and
an outer loop where the mechanism agent updates its policy based on
experience.

HCMD-zero balaguer22hcmd  is a general-purpose method to construct mechanism
agents that are preferred over baseline alternatives by human participants
engaged in economic interactions. HCMD-zero learns by mediating interactions
among participants, while remaining engaged in an electoral contest with
copies of itself, thereby accessing direct feedback from participants. Results
on the Public Investment Game, a stylised resource allocation game that
highlights the tension between productivity, equality and the temptation to
free-ride, show that HCMD-zero produces competitive mechanism agents that are
consistently preferred by human participants over baseline alternatives, and
does so automatically, without requiring human knowledge, and by using human
data sparingly and effectively.

We have also considered the setting where a single agent might learn to reward
others, thus eliciting desirable cooperative behaviours. In the Learning to
Incentivize Others (LIO) algorithm yang2020learning , the agent has a
“gifting” policy represented as a neural network. The parameters of this
network are adjusted to maximise the original environment reward (without
gifts) and regularised to approximately maintain budget-balance.

##  5 Evaluating / Analysing Players & Strategies

Evaluating the performance of players or strategies in a game is not always
straightforward. In a sport like bobsledding, teams’ performances are largely
independent of each other and so teams can be ranked by finishing time.
However, in a game like football where two teams play head-to-head in various
stadium environments, performances are intertwined. This is made evident in
the classic rock-paper-scissors game where each strategy wins, ties, and loses
to exactly one other strategy. Given a game with arbitrary payoffs, how can we
score or rank the strategies available to us in that game? We are especially
interested in game-theoretic approaches that satisfy certain desirable
properties. Along similar lines, we aim to use principles from game theory to
statistically evaluate the performance and compatibility of players in real
world sports settings.

###  5.1 Rating

Rating strategies in normal-form games is an important aspect of evaluating
agents in multiplayer games and tasks in multitask scenarios. Normal-form
games can be constructed empirically from measured performance between
matchups of agent policies, where each policy corresponds to a player’s
strategy in the game. Such empirical games are often called _meta-games_ , and
are utilised in empirical game-theoretic analysis (EGTA) and PSRO.
Traditionally, only transitive dependencies between strategies were considered
when defining a rating (e.g. Elo), while cyclic (“rock-paper-scissors”)
dynamics were ignored. Recent work balduzzi2018re  provided a mechanism to
effectively rate strategies in two-player zero-sum games. The approach taken
there was to compute the Nash equilibrium of this metagame and then rank
strategies either according to their mass under the equilibrium or their _Nash
average_. Our work in marris2021_jpsro_icml  generalises Nash averages to
player payoff gradients at the solution (e.g., Nash equilibrium) of the game.
Each of the solution concepts described in §2.2 can be used to define a new
rating or ranking procedure in a similar manner. For example, $\alpha$-Rank
uses the ordered masses of replicator dynamic’s unique stationary distribution
to provide strategy profile rankings.

###  5.2 Sports Analytics

The availability of team sports data (e.g., in football or basketball) has
been steadily increasing over recent years. Football, in particular, is a
great test bed to study novel AI techniques with the potential to offer
decision-makers advice in the form of an automated video-assistant coach
(AVAC) (tuyls2021game, ). Techniques from computer vision, statistical
learning, and game theory all contribute to the understanding of the game.
Players can be tracked in broadcast video using computer vision; event and
pose detection techniques can provide further context. Statistical learning
allows constructing player representations that capture individual playing
style and intra-team chemistry, and also allows estimating the contribution of
individual player actions. Game theory provides the framework and tools to
study the sequential decision-making problems players face in the presence of
other players (cooperative and adversarial).

The AVAC system sits at the intersection of these three research fields. In
(tuyls2021game, ), we identify three frontiers: video predictive models,
strategic video generation models, and interactive decision-making, and lay
out a roadmap of specific research and engineering problems. The development
of AI-powered sports analytics techniques offer great potential for decision-
makers, broadcasters, players, and fans.

##  6 Inspired by Nature, Biology, and Sociology

Many of the most outstanding forms of intelligence that we see in the real
world have arisen from multi-agent systems. For example, cultural evolution is
thought to have been crucial to social intelligence and our ability to
cooperate in large numbers and solve complex problems (e.g., via market
economies). For this reason, some of our research looks to replicate the
forces in nature that are believed to have led to humanity’s and nature’s
exponential development. This is particularly evident in the work we described
in §3.5.

Living systems show a striking ability to adapt to novel situations and to
produce ever more complex and sophisticated forms of life. It is only through
(social) interactions that individuals have acquired these abilities, be they
implicitly through a process of natural selection, or explicitly in the form
of learning (szathmary1995major, ; ginsburg2021evolutionary, ). Social
interactions can lead to autocurricula (that is, a curriculum that
automatically matches the skill level of the individual) because the task to
learn or adapt to is precisely the interaction with others
leibo2019autocurricula . We use nature as inspiration for designing learning
systems that are able to leverage social interactions in their learning path
either implicitly through pure interactions LeiboZLMG17 ; perolat2017multi ,
or explicitly via an evolutionary or ecological level above reinforcement
learning wang2018evolving ; leibo2019malthusian ; sunehag2019reinforcement .
We also model known or putative human inductive biases to achieve more human-
like behaviours hughes2018inequity ; jaques2019social ; mckee2020social ,
which result in agents that can escape defecting equilibria in social
dilemmas.

##  7 Gamification

The bulk of modern AI and machine learning is built on the foundation of
learning as optimization. Given how pervasive multi-agent intelligence is in
the natural world, we suspect a form of learning intrinsically tied to game
theory and multi-agent research is warranted. Our team explores several lines
of work that attempt to reformulate fundamental aspects of learning as games.

We aim to reformulate fundamental problems in machine learning as solving
games. For example, in gemp2020eigengame ; gemp2021eigengame , we reformulate
the problem of singular value decomposition (SVD) as finding the Nash
equilibrium of an _EigenGame_ , bearing similarity to a congestion game. In
this case, players control eigenvectors in order to maximise the variance
captured in a dataset minus a cost for their eigenvector aligning with other
players’. SVD underlies several machine learning problems including principal
component analysis, linear discriminant analysis, latent semantic indexing,
spectral clustering, proto-value functions, and others. Formulating SVD as a
game makes it obvious how to design an algorithm that distributes over
players.

We also revisit reinforcement learning problems where the agent’s utility is
concave in the occupancy measure induced by its policy (CURL) and formulate
them as Mean Field Games geist2021curl . CURL encompasses frameworks such as
(pure) exploration, imitation learning, marginal matching or constrained MDPs,
that are efficiently solved using algorithms recently introduced for solving
Mean Field Games elie2020convergence ; perolat2021scaling ; muller2022learning
. This also provides new convergence rates by drawing connections between
Fictitious Play and Frank-Wolfe.

Lastly, in §3.6 we mentioned how we used Shapley values from cooperative game
theory to optimally select features, in this case vocabularies, in language
tasks patel2021game .

##  8 Test Suites and Benchmarks

Testing benchmarks are key to the success of artificially intelligent systems.
Rapid progress tends to occur whenever there is a public, objective way to
measure the quality of a proposed solution. A prime example is the development
of ImageNet deng2009imagenet  which together with AlexNet
krizhevsky2012imagenet  launched the Deep Learning revolution. Reinforcement
learning has comparatively fewer examples of this, and even fewer so in multi-
agent reinforcement learning. Each of the projects below explores various
dimensions of the multi-agent space as outlined in the introduction and we
have open-sourced several of these in order to aid researchers around the
world.

Melting Pot leibo2021scalable  is a test suite for evaluating zero-shot
generalisation to novel social circumstances. We will continue to expand and
improve the suite, intending to increase coverage of the breadth of social
situations like: zero-sum games, general-sum games, games where the incentives
of players are mostly aligned, and those in which they are mostly opposed,
games of coordination, games of team competition, and many more.

Diplomacy is a decades old AI challenge designed to test game playing agents
in a simultaneous move, extensive-form game setting with an incredibly large
action space. The game is designed to emphasise the interactions between
agents, meaning there is no universally effective strategy, and instead agents
must choose a strategy compatible with those of their co-players. We
specifically look at the no-press version of this game (no communication
between agents). This game continues to serve as a benchmark for AI approaches
today gray2020human ; bakhtin2021no ; paquette2019no ; anthony2020learning .

Soccer is a long-standing grand challenge for AI because it captures many key
open problems that must be solved on the route to useful physically embodied
AI systems. Those challenges include: high-frequency, high-dimensional,
continuous full body control; agile locomotion; reusing multiple skills in
different contexts (simultaneously or seamlessly chained together); long-term
planning and strategy; cooperating with others; being robust to a range of
other agents’ behaviours and adversarial situations. In the MuJoCo soccer
project, we used soccer to demonstrate that a learning system which controls a
simulated humanoid embodiment using 56-dimensional instantaneous joint
torques, can learn mid-level skills (locomotion, getting up, kicking,
dribbling) and ultimately some teamwork (division of labor, passing) in the
service of long-term goals. Teams of agents were trained to play soccer via
reinforcement learning and imitation of basic human motor control. They
exhibit human-like motor control and complex behaviour at different scales,
quantified by a range of analysis and statistics, including those used in
real-world sports analytics to quantify teamwork liu2021motor ; liu2019boxhead
.

The reformulation of machine learning problems as games represents an
opportunity for new domains of study. For example, EigenGame gemp2020eigengame
; gemp2021eigengame  is an $n$-player, general-sum game where each player’s
strategy space is constrained to the unit sphere, a Riemannian manifold.
Because the strategy spaces are not finite as in a finite normal-form game,
EigengGame does not enjoy their traditional properties. For instance, there is
no symmetric Nash equilibrium in an EigenGame despite the fact that EigenGame
is a symmetric game. The sorts of games that capture machine learning and
other foundational mathematical problems may present new areas of study for
game theorists and machine learning practitioners alike.

Language emergence simulation only recently started to scale up to medium-size
populations with non-artificial inputs. This scaling-up paves the way to new
agent dynamics for communication and raises novel multi-agent questions. What
makes a language stable, evolve, or degenerate? Can we link language structure
to game equilibria? We hence release a large-scale framework on language
emergence to help the community towards this goal chaabouni2021emergent .

OpenSpiel LanctotEtAl2019OpenSpiel  is an open-source framework we have
released for reinforcement learning and search in extensive-form games. It
supports $n$-player games, simultaneous move games, imperfect and perfect
information, and mean-field games. It contains many basic implementations of
algorithms listed in this document, including some visualizations and
evaluation tools such as $\alpha$-Rank omidshafiei2019alpha .

##  9 MA for Good

Much of our research into multi-agent systems can be tied to providing
solutions to real-world problems, however, some have more direct applications.
This line of work aims to concretely impact humanity in a positive way.

A key part of our motivation from multi-agent research relates to aspiring to
solve problems of cooperation between AI agents and their peers (both other AI
agents or humans). As AI technology becomes more ubiquitous, it has the
potential to improve collective behaviour in many domains, such as transport,
consumer and financial markets, communication and social media, or security
domains dafoe2021cooperative . Better understanding of multi-agent cooperation
could yield benefits in both small-scale problems such as better traffic
flows, and large-scale problems such as pandemic preparedness or global
trading. We have surveyed key problems and challenges to unlock stronger
“Cooperative AI” dafoe2020open , including aspects of multi-agent
understanding (such as predicting the goals and behaviours of others),
communication and sharing of information, intents and preferences, commitment
and making credible promises regarding agent behaviour and norms and
institutions, such as shared beliefs or rules, which serve as social
infrastructure that reinforces understanding, communication and commitment.
The Cooperative AI Foundation (CAIF) has been set up as an independent body
aimed at advancing this research agenda across disciplines.

Multi-agent reinforcement learning is a powerful tool for modelling human
societies. This is particularly critical as many of the biggest challenges of
our time are social in nature, like the emergence of discrimination.
Statistical discrimination refers to the behaviour of estimating the quality
of a social partner based purely on the statistical association between some
readily perceptible characteristics and quality, without taking into account
the specific quality of the individual in question arrow1972some . In recent
work duenez2021statistical , we studied statistical discrimination in a
temporally and spatially extended context where individuals have the chance to
progressively refine their estimate of the quality of partners, and where the
limitations of learning agents are explicitly modelled. We discovered that an
agent’s information processing ability mediates whether they will learn to
individuate their partners and choose them based on their intrinsic quality,
or fail to do so and engage in statistical discrimination.

##  10 Conclusion

In this summary, we have briefly outlined the research pursued by the Game
Theory & Multi-Agent team at DeepMind. Our agenda explores several axes of the
multi-agent problem space as described in §1. Each axis brings key technical
challenges which we encounter in a variety of externally and internally
defined domains. Given the expertise of DeepMind, much of our research takes a
deep (neural network) multi-agent reinforcement learning approach, however, we
also work on fundamental aspects of game theory such as computation of
equilibria. Each of the sections in this summary represents important open
challenges that our group has prioritised in order to efficiently push multi-
agent research in a way that maximises impact on our understanding of multi-
agent problems and their real world challenges.

## References

  * (1) D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T.P. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel and D. Hassabis, Mastering the game of Go without human knowledge, Nat. 550(7676) (2017), 354–359.
  * (2) O. Vinyals, I. Babuschkin, W.M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D.H. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai, J.P. Agapiou, M. Jaderberg, A.S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky, J. Molloy, T.L. Paine, Ç. Gülçehre, Z. Wang, T. Pfaff, Y. Wu, R. Ring, D. Yogatama, D. Wünsch, K. McKinney, O. Smith, T. Schaul, T.P. Lillicrap, K. Kavukcuoglu, D. Hassabis, C. Apps and D. Silver, Grandmaster level in StarCraft II using multi-agent reinforcement learning, Nat. 575(7782) (2019), 350–354.
  * (3) E. Lockhart, M. Lanctot, J. Pérolat, J.-B. Lespiau, D. Morrill, F. Timbers and K. Tuyls, Computing Approximate Equilibria in Sequential Adversarial Games by Exploitability Descent, in: Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI), 2019\.
  * (4) R. Munos, J. Perolat, J.-B. Lespiau, M. Rowland, B.D. Vylder, M. Lanctot, F. Timbers, D. Hennes, S. Omidshafiei, A. Gruslys, M.G. Azar, E. Lockhart and K. Tuyls, Fast Computation of Nash Equilibria in Imperfect Information Games, in: Proceedings of the International Conference on Machine Learning (ICML), 2020\.
  * (5) S. Srinivasan, M. Lanctot, V. Zambaldi, J. Pérolat, K. Tuyls, R. Munos and M. Bowling, Actor-Critic Policy Optimization in Partially Observable Multiagent Environments, in: Advances in Neural Information Processing Systems (NeurIPS), 2018\.
  * (6) A. Gruslys, M. Lanctot, R. Munos, F. Timbers, M. Schmid, J. Perolat, D. Morrill, V. Zambaldi, J.-B. Lespiau, J. Schultz, M.G. Azar, M. Bowling and K. Tuyls, The Advantage Regret-Matching Actor-Critic, 2020\.
  * (7) D. Hennes, D. Morrill, S. Omidshafiei, R. Munos, J. Perolat, M. Lanctot, A. Gruslys, J.-B. Lespiau, P. Parmas, E. Duenez-Guzman and K. Tuyls, Neural Replicator Dynamics, in: Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2020\.
  * (8) J. Perolat, R. Munos, J.-B. Lespiau, S. Omidshafiei, M. Rowland, P. Ortega, N. Burch, T. Anthony, D. Balduzzi, B.D. Vylder, G. Piliouras, M. Lanctot and K. Tuyls, From Poincaré Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularization, in: Proceedings of the The Thirty-eighth International Conference on Machine Learning (ICML), 2021\.
  * (9) I. Gemp, R. Savani, M. Lanctot, Y. Bachrach, T. Anthony, R. Everett, A. Tacchetti, T. Eccles and J. Kramár, Sample-based Approximation of Nash in Large Many-Player Games via Gradient Descent, in: Proceedings of the 21st International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’22, International Foundation for Autonomous Agents and Multiagent Systems, 2022\.
  * (10) L. Marris, P. Muller, M. Lanctot, K. Tuyls and T. Graepel, Multi-Agent Training beyond Zero-Sum with Correlated Equilibrium Meta-Solvers, in: Proceedings of the 38th International Conference on Machine Learning, M. Meila and T. Zhang, eds, Proceedings of Machine Learning Research, Vol. 139, PMLR, 2021, pp. 7480–7491. http://proceedings.mlr.press/v139/marris21a.html.
  * (11) T. Börgers and R. Sarin, Learning Through Reinforcement and Replicator Dynamics, Journal of Economic Theory 77(1) (1997), 1–14.
  * (12) K. Tuyls, K. Verbeeck and T. Lenaerts, A selection-mutation model for q-learning in multi-agent systems, in: The Second International Joint Conference on Autonomous Agents & Multiagent Systems, AAMAS 2003, July 14-18, 2003, Melbourne, Victoria, Australia, Proceedings, ACM, 2003, pp. 693–700.
  * (13) D. Bloembergen, K. Tuyls, D. Hennes and M. Kaisers, Evolutionary Dynamics of Multi-Agent Learning: A Survey, J. Artif. Intell. Res. 53 (2015), 659–697.
  * (14) K. Tuyls, D. Heytens, A. Nowé and B. Manderick, Extended Replicator Dynamics as a Key to Reinforcement Learning in Multi-agent Systems, in: Machine Learning: ECML 2003, 14th European Conference on Machine Learning, Cavtat-Dubrovnik, Croatia, September 22-26, 2003, Proceedings, N. Lavrac, D. Gamberger, L. Todorovski and H. Blockeel, eds, Lecture Notes in Computer Science, Vol. 2837, Springer, 2003, pp. 421–431.
  * (15) S. Omidshafiei, C. Papadimitriou, G. Piliouras, K. Tuyls, M. Rowland, J.-B. Lespiau, W.M. Czarnecki, M. Lanctot, J. Perolat and R. Munos, $\alpha$-Rank: Multi-Agent Evaluation by Evolution, Scientific Reports 9(1) (2019), 9937\. https://doi.org/10.1038/s41598-019-45619-9.
  * (16) M. Lanctot, V. Zambaldi, A. Gruslys, A. Lazaridou, K. Tuyls, J. Perolat, D. Silver and T. Graepel, A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning, in: Neural Information Processing Systems (NIPS), 2017\.
  * (17) C.H. Papadimitriou and G. Piliouras, From Nash Equilibria to Chain Recurrent Sets: An Algorithmic Solution Concept for Game Theory, Entropy 20(10) (2018), 782\.
  * (18) J.-M. Lasry and P.-L. Lions, Mean field games, Jpn. J. Math. (2007). doi:10.1007/s11537-007-0657-8.
  * (19) M. Huang, R.P. Malhamé and P.E. Caines, Large population stochastic dynamic games: closed-loop McKean-Vlasov systems and the Nash certainty equivalence principle, Communications in Information & Systems 6 (2006).
  * (20) C. Dogbé, Modeling crowd dynamics by the mean-field limit approach, Mathematical and Computer Modelling 52(9–10) (2010), 1506–1520.
  * (21) R. Carmona, J.-P. Fouque and L.-H. Sun, Mean Field Games and systemic risk, Communications in Mathematical Sciences 13(4) (2015), 911–933.
  * (22) R. Elie, E. Hubert and G. Turinici, Contact rate epidemic control of COVID-19: an equilibrium view, Mathematical Modelling of Natural Phenomena 15 (2020), 35\.
  * (23) R. Élie, E. Hubert, T. Mastrolia and D. Possamaï, Mean–field moral hazard for optimal energy demand response management, Mathematical Finance 31(1) (2021), 399–473.
  * (24) S. Perrin, M. Laurière, J. Pérolat, M. Geist, R. Élie and O. Pietquin, Mean field games flock! the reinforcement learning way, arXiv preprint arXiv:2105.07933 (2021).
  * (25) T. Cabannes, M. Lauriere, J. Perolat, R. Marinier, S. Girgin, S. Perrin, O. Pietquin, A.M. Bayen, E. Goubault and R. Elie, Solving N-player dynamic routing games with congestion: a mean field approach, arXiv preprint arXiv:2110.11943 (2021).
  * (26) R. Elie, J. Perolat, M. Laurière, M. Geist and O. Pietquin, On the Convergence of Model Free Learning in Mean Field Games, in: proc. of AAAI, 2020\.
  * (27) S. Perrin, J. Pérolat, M. Laurière, M. Geist, R. Elie and O. Pietquin, Fictitious Play for Mean Field Games: Continuous Time Analysis and Applications, in: proc. of NeurIPS, 2020\.
  * (28) J. Perolat, S. Perrin, R. Elie, M. Laurière, G. Piliouras, M. Geist, K. Tuyls and O. Pietquin, Scaling up Mean Field Games with Online Mirror Descent, arXiv preprint arXiv:2103.00623 (2021).
  * (29) M. Laurière, S. Perrin, S. Girgin, P. Muller, A. Jain, T. Cabannes, G. Piliouras, J. Pérolat, R. Élie, O. Pietquin et al., Scalable Deep Reinforcement Learning Algorithms for Mean Field Games, arXiv preprint arXiv:2203.11973 (2022).
  * (30) S. Perrin, M. Laurière, J. Pérolat, R. Élie, M. Geist and O. Pietquin, Generalization in mean field games by learning master policies, arXiv preprint arXiv:2109.09717 (2021).
  * (31) P. Muller, M. Rowland, R. Elie, G. Piliouras, J. Perolat, M. Lauriere, R. Marinier, O. Pietquin and K. Tuyls, Learning Equilibria in Mean-Field Games: Introducing Mean-Field PSRO, arXiv, 2021\. doi:10.48550/ARXIV.2111.08350. https://arxiv.org/abs/2111.08350.
  * (32) G.W. Brown, Iterative solution of games by fictitious play, Activity analysis of production and allocation 13(1) (1951), 374–376.
  * (33) H.B. McMahan, G.J. Gordon and A. Blum, Planning in the presence of cost functions controlled by an adversary, in: Proceedings of the 20th International Conference on Machine Learning (ICML-03), 2003, pp. 536–543.
  * (34) J. Heinrich, M. Lanctot and D. Silver, Fictitious Self-Play in Extensive-Form Games, in: Proceedings of the 32nd International Conference on Machine Learning (ICML 2015), 2015\.
  * (35) J. Heinrich and D. Silver, Deep Reinforcement Learning from Self-Play in Imperfect-Information Games, CoRR abs/1603.01121 (2016).
  * (36) M. Lanctot, V. Zambaldi, A. Gruslys, A. Lazaridou, K. Tuyls, J. Perolat, D. Silver and T. Graepel, A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning., in: Advances in Neural Information Processing Systems, 2017\.
  * (37) P. Muller, S. Omidshafiei, M. Rowland, K. Tuyls, J. Perolat, S. Liu, D. Hennes, L. Marris, M. Lanctot, E. Hughes, Z. Wang, G. Lever, N. Heess, T. Graepel and R. Munos, A Generalized Training Approach for Multiagent Learning, in: Proceedings of the Eighth International Conference on Learning Representations (ICLR), 2020\.
  * (38) S. Liu, L. Marris, D. Hennes, J. Merel, N. Heess and T. Graepel, NeuPL: Neural Population Learning, in: International Conference on Learning Representations, 2022\. https://openreview.net/forum?id=MIX3fJkl_1.
  * (39) M.G. Lagoudakis and R. Parr, Least-squares policy iteration, The Journal of Machine Learning Research 4 (2003), 1107–1149.
  * (40) T.W. Anthony, Expert iteration, PhD thesis, UCL (University College London), 2021\.
  * (41) T. Anthony, T. Eccles, A. Tacchetti, J. Kramár, I. Gemp, T. Hudson, N. Porcel, M. Lanctot, J. Pérolat, R. Everett et al., Learning to play no-press diplomacy with best response policy iteration, Advances in Neural Information Processing Systems 33 (2020), 17987–18003.
  * (42) A. Calhamer, Diplomacy, Board Game. Avalon Hill (1959).
  * (43) Y. Bachrach, R. Everett, E. Hughes, A. Lazaridou, J.Z. Leibo, M. Lanctot, M. Johanson, W.M. Czarnecki and T. Graepel, Negotiating team formation using deep reinforcement learning, Artificial Intelligence 288 (2020), 103356\.
  * (44) E. Hughes, T.W. Anthony, T. Eccles, J.Z. Leibo, D. Balduzzi and Y. Bachrach, Learning to resolve alliance dilemmas in many-player zero-sum games, arXiv preprint arXiv:2003.00799 (2020).
  * (45) K.R. McKee, I. Gemp, B. McWilliams, E.A. Duèñez-Guzmán, E. Hughes and J.Z. Leibo, Social Diversity and Social Preferences in Mixed-Motive Reinforcement Learning, in: Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, 2020, pp. 869–877.
  * (46) I. Gemp, K.R. McKee, R. Everett, E. Duéñez-Guzmán, Y. Bachrach, D. Balduzzi and A. Tacchetti, D3C: Reducing the Price of Anarchy in Multi-Agent Learning, in: Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, 2022, pp. 498–506.
  * (47) M. Carroll, R. Shah, M.K. Ho, T. Griffiths, S. Seshia, P. Abbeel and A. Dragan, On the utility of learning about humans for human-ai coordination, Advances in neural information processing systems 32 (2019).
  * (48) D. Strouse, K. McKee, M. Botvinick, E. Hughes and R. Everett, Collaborating with humans without human data, Advances in Neural Information Processing Systems 34 (2021).
  * (49) W.M. Czarnecki, G. Gidel, B. Tracey, K. Tuyls, S. Omidshafiei, D. Balduzzi and M. Jaderberg, Real world games look like spinning tops, Advances in Neural Information Processing Systems 33 (2020), 17443–17454.
  * (50) A. Vezhnevets, Y. Wu, M. Eckstein, R. Leblond and J.Z. Leibo, Options as responses: Grounding behavioural hierarchies in multi-agent reinforcement learning, in: International Conference on Machine Learning, PMLR, 2020, pp. 9733–9742.
  * (51) K. Kopparapu, E.A. Duéñez-Guzmán, J. Matyas, A.S. Vezhnevets, J.P. Agapiou, K.R. McKee, R. Everett, J. Marecki, J.Z. Leibo and T. Graepel, Hidden Agenda: a Social Deduction Game with Diverse Learned Equilibria, arXiv preprint arXiv:2201.01816 (2022).
  * (52) P. Moreno, E. Hughes, K.R. McKee, B.A. Pires and T. Weber, Neural Recursive Belief States in Multi-Agent Reinforcement Learning, arXiv, 2021\. doi:10.48550/ARXIV.2102.02274. https://arxiv.org/abs/2102.02274.
  * (53) E. Vinitsky, R. Köster, J.P. Agapiou, E. Duéñez-Guzmán, A.S. Vezhnevets and J.Z. Leibo, A learning agent that acquires social norms from public sanctions in decentralized multi-agent settings, arXiv preprint arXiv:2106.09012 (2021).
  * (54) R. Köster, D. Hadfield-Menell, R. Everett, L. Weidinger, G.K. Hadfield and J.Z. Leibo, Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents, Proceedings of the National Academy of Sciences 119(3) (2022).
  * (55) C.G.I. Team, A. Bhoopchand, B. Brownfield, A. Collister, A.D. Lago, A. Edwards, R. Everett, A. Frechette, Y.G. Oliveira, E. Hughes et al., Learning Robust Real-Time Cultural Transmission without Human Data, arXiv preprint arXiv:2203.00715 (2022).
  * (56) J.W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young et al., Scaling language models: Methods, analysis & insights from training gopher, arXiv preprint arXiv:2112.11446 (2021).
  * (57) M. Rita, F. Strub, J.-B. Grill, O. Pietquin and E. Dupoux, On the role of population heterogeneity in emergent communication, International Conference on Learning Representations (2021).
  * (58) R. Chaabouni, F. Strub, F. Altché, E. Tarassov, C. Tallec, E. Davoodi, K.W. Mathewson, O. Tieleman, A. Lazaridou and B. Piot, Emergent communication at scale, International Conference on Learning Representations (2021).
  * (59) T. Eccles, Y. Bachrach, G. Lever, A. Lazaridou and T. Graepel, Biases for emergent communication in multi-agent reinforcement learning, Advances in neural information processing systems 32 (2019).
  * (60) A. Kalinowska, E. Davoodi, F. Strub, K. Mathewson, T. Murphey and P. Pilarski, Situated Communication: A Solution to Over-communication between Artificial Agents, Emergent Communication Workshop at ICLR 2022 (2022).
  * (61) A.M. Donati, G. Quispe, C. Ollion, S.L. Corff, F. Strub and O. Pietquin, Learning Natural Language Generation from Scratch, Conference of the North American Chapter of the Association for Computational Linguistics (2022).
  * (62) Y. Bachrach, I. Gemp, M. Garnelo, J. Kramar, T. Eccles, D. Rosenbaum and T. Graepel, A Neural Network Auction For Group Decision Making Over a Continuous Space, in: Proceedings of the 30th International Joint Conference on Artificial Intelligence (IJCAI) Demonstrations Track, 2021\.
  * (63) P. Milgrom and P.R. Milgrom, Putting auction theory to work, Cambridge University Press, 2004\.
  * (64) A. Tacchetti, D. Strouse, M. Garnelo, T. Graepel and Y. Bachrach, Learning Truthful, Efficient, and Welfare Maximizing Auction Rules, in: ICLR Workshop on Gamification and Multiagent Solutions, 2022\.
  * (65) M.A. Bakker, R. Everett, L. Weidinger, I. Gabriel, W.S. Isaac, J.Z. Leibo and E. Hughes, Modelling Cooperation in Network Games with Spatio-Temporal Complexity, CoRR abs/2102.06911 (2021). https://arxiv.org/abs/2102.06911.
  * (66) J. Balaguer, R. Köster, C. Summerfield and A. Tacchetti, The Good Shepherd: An Oracle Agent for Mechanism Design, in: ICLR Workshop on Gamification and Multiagent Solutions, 2022\.
  * (67) J. Balaguer, R. Köster, A. Weinstein, L. Campbell-Gillingham, C. Summerfield, M. Botvinick and A. Tacchetti, HCMD-zero: Learning Value Aligned Mechanisms from Data, in: ICLR Workshop on Gamification and Multiagent Solutions, 2022\.
  * (68) J. Yang, A. Li, M. Farajtabar, P. Sunehag, E. Hughes and H. Zha, Learning to incentivize other learning agents, Advances in Neural Information Processing Systems 33 (2020), 15208–15219.
  * (69) D. Balduzzi, K. Tuyls, J. Perolat and T. Graepel, Re-evaluating evaluation, Advances in Neural Information Processing Systems 31 (2018).
  * (70) K. Tuyls, S. Omidshafiei, P. Muller, Z. Wang, J. Connor, D. Hennes, I. Graham, W. Spearman, T. Waskett, D. Steel et al., Game Plan: What AI can do for Football, and What Football can do for AI, Journal of Artificial Intelligence Research 71 (2021), 41–88.
  * (71) E. Szathmáry and J.M. Smith, The major transitions in evolution, WH Freeman Spektrum Oxford, UK:, 1995\.
  * (72) S. Ginsburg and E. Jablonka, Evolutionary transitions in learning and cognition, Philosophical Transactions of the Royal Society B 376(1821) (2021), 20190766\.
  * (73) J.Z. Leibo, E. Hughes, M. Lanctot and T. Graepel, Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research, arXiv preprint arXiv:1903.00742 (2019).
  * (74) J.Z. Leibo, V.F. Zambaldi, M. Lanctot, J. Marecki and T. Graepel, Multi-agent Reinforcement Learning in Sequential Social Dilemmas, CoRR abs/1702.03037 (2017).
  * (75) J. Perolat, J.Z. Leibo, V. Zambaldi, C. Beattie, K. Tuyls and T. Graepel, A multi-agent reinforcement learning model of common-pool resource appropriation, Advances in Neural Information Processing Systems 30 (2017).
  * (76) J.X. Wang, E. Hughes, C. Fernando, W.M. Czarnecki, E.A. Duéñez-Guzmán and J.Z. Leibo, Evolving intrinsic motivations for altruistic behavior, arXiv preprint arXiv:1811.05931 (2018).
  * (77) J.Z. Leibo, J. Perolat, E. Hughes, S. Wheelwright, A.H. Marblestone, E. Duéñez-Guzmán, P. Sunehag, I. Dunning and T. Graepel, Malthusian Reinforcement Learning, in: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, 2019, pp. 1099–1107.
  * (78) P. Sunehag, G. Lever, S. Liu, J. Merel, N. Heess, J.Z. Leibo, E. Hughes, T. Eccles and T. Graepel, Reinforcement learning agents acquire flocking and symbiotic behaviour in simulated ecosystems, in: ALIFE 2019: The 2019 Conference on Artificial Life, MIT Press, 2019, pp. 103–110.
  * (79) E. Hughes, J.Z. Leibo, M. Phillips, K. Tuyls, E. Dueñez-Guzman, A. García Castañeda, I. Dunning, T. Zhu, K. McKee, R. Koster et al., Inequity aversion improves cooperation in intertemporal social dilemmas, Advances in neural information processing systems 31 (2018).
  * (80) N. Jaques, A. Lazaridou, E. Hughes, C. Gulcehre, P. Ortega, D. Strouse, J.Z. Leibo and N. De Freitas, Social influence as intrinsic motivation for multi-agent deep reinforcement learning, in: International Conference on Machine Learning, PMLR, 2019, pp. 3040–3049.
  * (81) I. Gemp, B. McWilliams, C. Vernade and T. Graepel, Eigengame: PCA as a Nash equilibrium, in: International Conference on Learning Representations, 2020\. https://openreview.net/forum?id=NzTU59SYbNq.
  * (82) I. Gemp, B. McWilliams, C. Vernade and T. Graepel, EigenGame Unloaded: When playing games is better than optimizing (2021). https://openreview.net/forum?id=So6YAqnqgMj.
  * (83) M. Geist, J. Pérolat, M. Laurière, R. Elie, S. Perrin, O. Bachem, R. Munos and O. Pietquin, Concave Utility Reinforcement Learning: the Mean-field Game viewpoint, in: proc. of AAMAS, 2022\.
  * (84) R. Patel, M. Garnelo, I. Gemp, C. Dyer and Y. Bachrach, Game-theoretic Vocabulary Selection via the Shapley Value and Banzhaf Index, in: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 2789–2798.
  * (85) J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in: 2009 IEEE conference on computer vision and pattern recognition, Ieee, 2009, pp. 248–255.
  * (86) A. Krizhevsky, I. Sutskever and G.E. Hinton, Imagenet classification with deep convolutional neural networks, Advances in neural information processing systems 25 (2012).
  * (87) J.Z. Leibo, E.A. Dueñez-Guzman, A. Vezhnevets, J.P. Agapiou, P. Sunehag, R. Koster, J. Matyas, C. Beattie, I. Mordatch and T. Graepel, Scalable evaluation of multi-agent reinforcement learning with melting pot, in: International Conference on Machine Learning, PMLR, 2021, pp. 6187–6199.
  * (88) J. Gray, A. Lerer, A. Bakhtin and N. Brown, Human-level performance in no-press diplomacy via equilibrium search, arXiv preprint arXiv:2010.02923 (2020).
  * (89) A. Bakhtin, D. Wu, A. Lerer and N. Brown, No-Press Diplomacy from Scratch, Advances in Neural Information Processing Systems 34 (2021).
  * (90) P. Paquette, Y. Lu, S.S. Bocco, M. Smith, S. O-G, J.K. Kummerfeld, J. Pineau, S. Singh and A.C. Courville, No-press diplomacy: Modeling multi-agent gameplay, Advances in Neural Information Processing Systems 32 (2019).
  * (91) S. Liu, G. Lever, Z. Wang, J. Merel, S. Eslami, D. Hennes, W.M. Czarnecki, Y. Tassa, S. Omidshafiei, A. Abdolmaleki et al., From motor control to team play in simulated humanoid football, arXiv preprint arXiv:2105.12196 (2021).
  * (92) G. Lever, J. Merel, N. Heess, S. Tunyasuvunakool, S. Liu and T. Graepel, Emergent Coordination through Competition, 2019\.
  * (93) M. Lanctot, E. Lockhart, J.-B. Lespiau, V. Zambaldi, S. Upadhyay, J. Pérolat, S. Srinivasan, F. Timbers, K. Tuyls, S. Omidshafiei, D. Hennes, D. Morrill, P. Muller, T. Ewalds, R. Faulkner, J. Kramár, B.D. Vylder, B. Saeta, J. Bradbury, D. Ding, S. Borgeaud, M. Lai, J. Schrittwieser, T. Anthony, E. Hughes, I. Danihelka and J. Ryan-Davis, OpenSpiel: A Framework for Reinforcement Learning in Games, CoRR abs/1908.09453 (2019). http://arxiv.org/abs/1908.09453.
  * (94) A. Dafoe, Y. Bachrach, G. Hadfield, E. Horvitz, K. Larson and T. Graepel, Cooperative AI: machines must learn to find common ground, Nature Publishing Group, 2021\.
  * (95) A. Dafoe, E. Hughes, Y. Bachrach, T. Collins, K.R. McKee, J.Z. Leibo, K. Larson and T. Graepel, Open problems in cooperative ai, arXiv preprint arXiv:2012.08630 (2020).
  * (96) K.J. Arrow, Some mathematical models of race discrimination in the labor market, Racial discrimination in economic life (1972), 187–204.
  * (97) E.A. Duéñez-Guzmán, K.R. McKee, Y. Mao, B. Coppin, S. Chiappa, A.S. Vezhnevets, M.A. Bakker, Y. Bachrach, S. Sadedin, W. Isaac et al., Statistical discrimination in learning agents, arXiv preprint arXiv:2110.11404 (2021).