# Evolution of reciprocity with limited payoff memory

Nikoleta E. Glynatsi1, Alex McAvoy2,3,†, Christian Hilbe1,† 1Max Planck
Research Group on the Dynamics of Social Behavior, Max Planck Institute for
Evolutionary Biology, Plön, Germany 2School of Data Science and Society,
University of North Carolina at Chapel Hill, Chapel Hill, NC 27599 3Department
of Mathematics, University of North Carolina at Chapel Hill, Chapel Hill, NC
27599 † A.M. and C.H. contributed equally to this work.

###### Abstract

Direct reciprocity is a mechanism for the evolution of cooperation in repeated
social interactions. According to this literature, individuals naturally learn
to adopt conditionally cooperative strategies if they have multiple encounters
with their partner. Corresponding models have greatly facilitated our
understanding of cooperation, yet they often make strong assumptions on how
individuals remember and process payoff information. For example, when
strategies are updated through social learning, it is commonly assumed that
individuals compare their average payoffs. This would require them to compute
(or remember) their payoffs against everyone else in the population. To
understand how more realistic constraints influence direct reciprocity, we
consider the evolution of conditional behaviors when individuals learn based
on more recent experiences. Even in the most extreme case that they only take
into account their very last interaction, we find that cooperation can still
evolve. However, such individuals adopt less generous strategies, and they
tend to cooperate less often than in the classical setup with average payoffs.
Interestingly, once individuals remember the payoffs of two or three recent
interactions, cooperation rates quickly approach the classical limit. These
findings contribute to a literature that explores which kind of cognitive
capabilities are required for reciprocal cooperation. While our results
suggest that some rudimentary form of payoff memory is necessary, it already
suffices to remember a few interactions.

Keywords: Evolution of cooperation; direct reciprocity; repeated prisoner’s
dilemma; social learning; evolutionary dynamics

## 1 Introduction

Evolutionary game theory describes the dynamics of populations when an
individual’s fitness depends on the traits or strategies of other population
members (1, 2, 3, 4). This theory can be used to describe the dynamics of
animal conflict (5), cancer cells (6), and of cooperation (7). Respective
models translate strategic interactions into games (8). These games specify
how individuals (players) interact, which strategies individuals can choose,
and what fitness consequences (or payoffs) the different strategies have. In
addition, these models also specify the mode by which successful strategies
spread over time. In models of biological evolution, individuals with a high
fitness produce more offspring; in models of cultural evolution, such
individuals are imitated more often. Although biological and cultural
evolution are sometimes treated as equivalent, there can be important
differences (9, 10, 11). For example, models of biological evolution do not
require individuals to have any particular cognitive abilities. Here, it is
the evolutionary process itself that biases the population towards strategies
with higher fitness. In contrast, in models of cultural evolution, individuals
need to be aware of the different strategies present in the population, and
they need to identify those strategies with a higher payoff. As a consequence,
evolutionary outcomes may depend on how easily different behaviors can be
learned (12), and on how easy payoff comparisons are.

These difficulties to learn strategies by social imitation are particularly
pronounced in models of direct reciprocity. This literature follows Trivers’
insight that individuals have more of an incentive to cooperate in social
dilemmas when they interact repeatedly (13). In repeated interactions,
individuals can condition their behavior on their past experiences with their
interaction partner. They may use strategies such as Tit-for-Tat (14, 15) or
Generous Tit-for-Tat (16, 17) to preferentially cooperate with other
cooperators. Such conditional strategies approximate human behavior fairly
well (18, 19, 20, 21, 22) and they have also been documented in several other
species (23, 24, 25) – although direct reciprocity is generally more difficult
to demonstrate in animals (26, 27, 28). However, at the outset, it is not
clear how easy it is to learn reciprocal strategies by social imitation. As
one obstacle, even if others’ strategies are perfectly observable, individuals
might find it difficult to identify which ones have the highest payoff. After
all, the payoff of a strategy of direct reciprocity is not determined by the
outcome of any single round. Rather, it is determined by how well this
strategy fares over an entire sequence of rounds, against many different
population members. In practice, such information might be both difficult to
obtain and to process.

Most models of direct reciprocity abstract from these difficulties (29, 30,
31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46). They just
assume individuals can easily copy the strategies of others. Similarly, they
just assume that updating decisions are based on the strategies’ average (or
expected) payoffs, which are based on all rounds and all interactions. These
assumptions create a curious inconsistency in how models represent an
individual’s cognitive abilities. On the one hand, when playing the game,
individuals are often assumed to have restricted memory. Respective studies
typically assume that individuals make their decisions each round based on the
outcome of the last round only (with only a few exceptions, see Refs. 47, 48,
49, 50, 51). Yet when learning new strategies, individuals are assumed to
remember (or compute) each others’ precise average payoff across many rounds
and many interaction partners. Herein, we wish to explore whether this latter
assumption is actually necessary for the evolution of reciprocity through
social imitation. We ask whether individuals can learn to adopt reciprocal
strategies even when learning is based on payoff information from a limited
number of rounds.

To explore that question, we theoretically study imitation dynamics in the
repeated prisoner’s dilemma, using two extreme scenarios. The first scenario
is the usual modeling approach. Here, individuals update their strategies
based on their expected payoffs. We contrast this model with an alternative
scenario where individuals update their strategies based on the very last
(one-shot) payoff they obtained. We find that individuals with limited payoff
memory tend to adopt less generous strategies. Yet moderate levels of
cooperation can still evolve. Moreover, as we increase the individuals’ payoff
memory to include the last two or three one-shot payoffs, cooperation rates
quickly approach the rates observed in the classical baseline case.

Overall, these findings suggest that while memory is important, already
minimal payoff information may suffice for the evolution of direct reciprocity
based on social learning. They also suggest that the classical model of
reciprocity (based on expected payoffs) can often be interpreted as a useful
approximation to more realistic models that include cognitive constraints.

## 2 Model and Methods

To explore the impact of limited payoff memory, we adapt existing models of
the evolution of direct reciprocity. These models involve two different time
scales. The short time scale describes the game dynamics. Here, individuals
with fixed strategies are randomly matched to interact with each other in
repeated social dilemmas. The long time scale describes the evolutionary
dynamics. Here, individuals can update their repeated-game strategies based on
the payoffs they yield. In the following, we introduce the basic setup of our
model; all details and derivations are described in the electronic
supplementary material.

Description of the game dynamics. We consider a well-mixed population
consisting of $N$ players. Players are randomly matched in pairs to
participate in a repeated donation game (52) with their respective co-player.
Each round, they can either cooperate ($C$) or defect ($D$). A cooperating
player provides a benefit $b$ to the other player at their own cost $c$, with
$0\\!<\\!c\\!<\\!b$. A defecting player provides no benefit and pays no cost.
Thus, the players’ payoffs in a single round are given by the matrix

$\displaystyle\bordermatrix{&C&D\cr C&\ b-c&\ -c\ \cr D&\ b&\ 0\ \cr}.$ (1)

In particular, payoffs take the form of a prisoner’s dilemma: Mutual
cooperation yields a better payoff than mutual defection
($b\\!-\\!c\\!>\\!0$), but each player individually prefers to defect
independent of the co-player’s action ($b\\!>\\!b\\!-\\!c$ and $0\\!>\\!-c$).
To incorporate that individuals interact repeatedly, we assume that after each
round, there is a constant continuation probability $\delta$ of interacting
for another round. For $\delta\\!=\\!0$, we recover the case of a conventional
(one-shot) prisoner’s dilemma. Here, mutual defection is the only equilibrium.
As $\delta$ increases, the game turns into a repeated game. Here, additional
equilibria emerge, with some of them allowing for full cooperation (53, 54,
55, 56).

In a one-shot donation game, players can only choose among two pure strategies
(they can either cooperate or defect). In the repeated game, strategies become
arbitrarily complex. Here, strategies are contingent rules, telling players
what to do depending on the outcome of all previous rounds. For simplicity, in
the following we assume individuals use reactive strategies (17). A reactive
strategy only depends on the other player’s action in the last round. Such
strategies can be written as a three-dimensional tuple $s=(y,p,q)$. The first
entry $y$ is the probability that the player opens with cooperation in the
first round. The two other entries are the probabilities that the player
cooperates in all subsequent rounds, depending on whether the co-player
cooperated ($p$) or defected ($q$) in the previous round. The set of reactive
strategies is simple enough to facilitate an explicit mathematical analysis
(1). Yet it is rich enough to capture several important strategies of repeated
games. For example, it contains ALLD $=\\!(0,0,0)$, the strategy that always
defects. Similarly, it contains Tit-for-Tat, TFT $=\\!(1,1,0)$, the strategy
that copies the co-player’s previous action (and that cooperates in the first
round). Finally, it contains Generous Tit-for-Tat, GTFT$=\\!(1,1,q)$, where
$q\\!>\\!0$ reflects a player’s generosity in response to a co-player’s
defection (16, 17).

In the short run, the players’ strategies are taken to be fixed. Players use
their strategies to decide whether to cooperate in a series of repeated games
against all other population members. In the long run, however, the players’
strategies may change depending on the payoffs they yield, as we describe in
the following.

Description of the evolutionary dynamics. Herein, we assume population members
update their strategies based on social learning. To model these strategy
updates, we consider a pairwise comparison process (57). This process assumes
that at regular time intervals, one population member is randomly selected,
and given the chance to revise its strategy. We refer to this player as the
‘learner’. With probability $\mu$ (reflecting a mutation rate), the learner
simply adopts a random strategy (all reactive strategies have the same
probability to be chosen). With the converse probability $1\\!-\\!\mu$, the
learner randomly picks a ‘role model’ from the population. The learner then
compares its own payoff $\pi_{\text{L}}$ from the repeated game to the role
model’s payoff $\pi_{\text{RM}}$. The learner adopts the role model’s strategy
with a probability $\varphi$ described by a Fermi function (58, 59),

$\varphi\left(\pi_{\text{L}},\pi_{\text{RM}}\right)=\frac{1}{1\\!+\\!e^{-\\!\beta\left(\pi_{\text{RM}}-\pi_{\text{L}}\right)}}.$
(2)

The selection strength parameter $\beta\\!\geq\\!0$ indicates how sensitive
players are to payoff differences. For $\beta\\!=\\!0$, payoff differences are
irrelevant, and the learner simply adopts the role model’s strategy with
probability one half. As the selection strength $\beta$ increases, players are
increasingly biased to imitate the role model only if it has the higher
payoff.

We deviate from previous models in how we interpret the payoffs
$\pi_{\text{L}}$ and $\pi_{\text{RM}}$, which form the basis of the pairwise
comparisons in Eq. (2). In previous work, these payoffs are taken to be the
respective players’ expected payoffs. We interpret that setup as a model with
perfect payoff memory. There, the payoffs $\pi_{\text{L}}$ and
$\pi_{\text{RM}}$ represent an average over all possible repeated games the
two individuals have played with all population members (Figure 1, upper left
panel). The use of expected payoffs is mathematically convenient, because
explicit formulas for these payoffs are available (1). Herein, we compare this
model of perfect payoff memory to a model with limited payoff memory. In that
model, the players’ payoffs $\pi_{\text{L}}$ and $\pi_{\text{RM}}$ are taken
to be the payoffs that each player received in their very last round prior to
making social comparisons. That is, players only consider the very last
repeated game they participated in, and there they only take into account the
outcome of the very last round (Figure 1, lower left panel). This assumption
could reflect, for example, a strong recency bias in how individuals evaluate
payoffs. In addition to this extreme case of limited payoff memory, later on
we also explore cases in which players take into account the outcome of two,
three, four, or more recent rounds.

Both in the case of perfect and limited memory, we iterate the elementary
strategy updating step described above for many time steps. This gives rise to
a stochastic process that describes which strategies players adopt over time.
We explore the dynamics of this process mathematically and with computer
simulations. For the results presented in the following, we assume that
mutations are rare ($\mu\\!\rightarrow\\!0$). This assumption is fairly common
in evolutionary game theory, because it makes some computations more efficient
(60, 61, 62), and because the results can be interpreted more easily. However,
in Section 3 of the electronic supplementary material we show that our main
results continue to hold for strictly positive mutation rates.

## 3 Results

Stability of cooperative populations. To get some intuition for the
differences between perfect and limited payoff memory, we first analyze when
cooperation is stable in either scenario. To this end, we consider a resident
population in which all players but one adopt a strategy of Generous Tit-for-
Tat, GTFT $=\\!(1,1,q)$. The remaining mutant player adopts ALLD. We say
cooperation is stochastically stable if the single mutant is more likely to
imitate the residents than vice versa. For simplicity, we consider a large
population ($N\\!\rightarrow\\!\infty$) and strong selection
($\beta\\!\rightarrow\\!\infty$). More general results are derived in the
electronic supplementary material.

In the case of perfect payoff memory, it is straightforward to characterize
when cooperation is stochastically stable. Here, we simply need to compute the
players’ expected payoffs. Because the population mostly consists of
residents, and because residents mutually cooperate with each other, their
expected payoff is $\pi_{\texttt{GTFT}}=b\\!-\\!c$. On the other hand, the
defecting mutant only interacts with residents. Given the residents’ strategy,
the mutant receives a benefit in the first round, and in every subsequent
round with probability $q$. As a result, the mutant’s expected payoff is
$\pi_{\texttt{ALLD}}\\!=\\!(1\\!-\\!\delta\\!+\\!\delta q)b$. For perfect
payoff memory, the requirement for cooperation to be stochastically stable
reduces to the condition $\pi_{\texttt{GTFT}}>\pi_{\texttt{ALLD}}$. This
yields

$q<1\\!-\\!\frac{c}{\delta b}.$ (3)

In particular, we recover the previous observation that $q\\!=\\!1-c/(\delta
b)$ is the maximum generosity that cooperators should have (16, 17, 63).
Because $q\\!\geq\\!0$, we also conclude that cooperation can only be stable
if $\delta\\!>\\!c/b$. Again, this condition for the feasibility of direct
reciprocity is the condition found in the literature (7).

The logic of the case with limited payoff memory is somewhat different. Here
we need to compute how likely each player obtains one of the four possible
payoffs $\\{b\\!-\\!c,-c,b,0\\}$ in the very last round of a game, before they
make social comparisons. Because residents almost always interact with other
residents, their last one-shot payoff is $\pi_{\texttt{GTFT}}=b\\!-\\!c$
almost surely. For the defecting mutant, there are two possibilities. (i) If
the mutant’s co-player happens to cooperate in the last round, the mutant
receives $\pi_{\texttt{ALLD}}\\!=\\!b$. This case occurs with probability
$1\\!-\\!\delta\\!+\\!\delta q$. (ii) If the co-player defects in the last
round, the mutant receives $\pi_{\texttt{ALLD}}=\\!0$. This occurs with the
converse probability $\delta(1\\!-\\!q)$. Because $b\\!-\\!c\\!<\\!b$,
residents tend to imitate the mutant in the first case. Because
$b\\!-\\!c\\!>\\!0$, mutants tend to imitate the resident in the second case.
Cooperation is stochastically stable if the first case is less likely than the
second. This yields the condition

$q<1\\!-\\!\frac{1}{2\delta}.$ (4)

Interestingly, this condition no longer depends on the exact payoff values $c$
and $b$. This independence arises because of our assumption of strong
selection, in which case only the payoff ordering $b\\!>\\!c\\!>\\!0$ matters.
Because $q$ is non-negative, condition (4) can only be satisfied if
$\delta\\!>\\!1/2$. That is, players need to interact in more than two rounds
in expectation.

By comparing the two cases, we find that payoff memory affects whether a
conditionally cooperative strategy ($1,1,q$) is viable. With perfect memory,
the maximum generosity $q$ needs to satisfy Eq. (3). In particular, this
generosity can become arbitrarily large, provided the game’s benefit-to-cost
ratio $b/c$ and the continuation probability $\delta$ are sufficiently large.
In contrast, with limited payoff memory, the maximum generosity is bounded by
one half, and it is independent of the benefit-to-cost ratio.

Evolutionary dynamics of reciprocity. To explore whether the previous static
observations describe the dynamics of evolving populations, we turn to
simulations. We have run separate simulations for perfect and limited payoff
memory. In each case, we consider both a low and a high benefit of cooperation
($b/c\\!=\\!3$ and $b/c\\!=\\!10$, respectively). For each simulation, we
record which strategies ($y,p,q$) the players adopt over time. Figure 1
depicts the conditional cooperation probabilities $p$ and $q$ (we omit the
opening move $y$ because we use a discount factor $\delta$ close to one, such
that first-round behavior is largely irrelevant). In all simulations, we find
that the players’ strategies cluster in two regions of the strategy space. The
first region corresponds to a neighborhood of ALLD with
$(p,q)\\!\approx\\!(0,0)$. The second region corresponds to a thin strip of
cooperative strategies with $(p,q)\\!\approx\\!(1,q)$. Within this strip, we
observe that most strategies satisfy the constraints on $q$ imposed by the
inequalities (3) and (4). That is, with perfect memory, most evolving
strategies have $q\\!<\\!1\\!-\\!c/b$, whereas with limited payoff memory,
most strategies have $q\\!<\\!1/2$. In particular, for limited payoff memory,
changes in the benefit parameter have no effect on the qualitative
distribution of strategies.

In each case, the evolutionary dynamics follow a similar cyclic pattern (as
described in Refs. 33, 17): Resident populations of defectors are most likely
invaded by strategies close to TFT. Once the population adopts conditionally
cooperative strategies $(1,1,q)$, neutral drift may introduce larger values of
generosity $q$. If the resident’s generosity $q$ violates the conditions (3)
and (4), defectors can re-invade, and the cycle starts again. The relative
time spent near ALLD and near the strip of conditionally cooperative
strategies depends on the considered memory setting (Supplementary Figure 3,
depicting the case of high benefits). For perfect memory, we find that ALLD is
replaced relatively quickly by more cooperative strategies. Here, it takes on
average 159 invading mutants until ALLD is successfully replaced. In contrast,
for limited memory, ALLD is more robust, resisting on average 798 mutant
strategies. This picture reverses when we consider an initial population that
adopts GTFT. Such populations are much more robust under perfect memory than
they are under limited memory. Overall, we find that the impact of memory on
the population’s average cooperation rate is substantial. For perfect memory,
this rate is 52% for low benefits, and 98% for high benefits. For limited
payoff memory, the evolving cooperation rates are smaller but still strictly
positive, with 37% cooperation for low benefits and 51% cooperation for high
benefits (Figure 1).

To further investigate the influence of different parameters, we have
systematically varied the benefit $b$ and the selection strength $\beta$ in
Figure 2. According to Figure 2a, perfect memory consistently results in a
higher cooperation rate, and this relative advantage further increases with an
increasing benefit $b$. Interestingly, for limited payoff memory, the
cooperation rate remains stable at approximately 50% once $b\\!\geq\\!5$. This
again reflects our earlier observation that the feasibility of cooperation in
this scenario is largely independent of the exact values of $b$ and $c$, as
described by Eq. (4). With respect to the effect of different selection
strengths, Figure 2b suggests that both perfect and limited payoff memory
yields similar cooperation rates for weak selection $\beta\\!<\\!1$. Beyond
weak selection, increasing selection has a positive effect under perfect
payoff memory, but a negative effect under limited payoff memory.

Beyond reactive strategies. While the results presented in the main text focus
on reactive strategies, the patterns we observe do not seem to depend on the
considered strategy space. To illustrate this point in more detail, in the
electronic supplementary material we consider the dynamics among memory-1
strategies. Here, players take into account both their co-player’s and their
own last move, see Refs. (64, 65). Also in that case, we observe that perfect
memory leads to systematically higher cooperation rates (Supplementary Figures
5,6). Again, this advantage of perfect memory is particularly pronounced for
strong selection, or when there is a high benefit of cooperation
(Supplementary Figure 7).

The effect of increasing individual payoff memory. So far, we have taken a
rather extreme interpretation of limited payoff memory. In the respective
scenario, we assumed that individuals update their strategies based on their
experience in a single round of the prisoner’s dilemma, against a single co-
player. The limited payoff memory framework can be expanded in various ways.
In particular, individuals may recall a larger number of rounds, they may
recall their interactions with several co-players, or both. To gain further
insights on the impact of payoff memory, we explore four additional scenarios.
In the first scenario, players recall the payoffs they obtained in the last
two rounds against a single co-player. In the second scenario, players recall
their last-round payoffs against two co-players. In the third scenario, they
recall the two last rounds against two co-players. Finally, in the last
scenario, players update based on the average payoff they receive over all
rounds with a single co-player (further extensions are possible, but we do not
explore them here).

For most scenarios, we can again derive an analytical condition for when
cooperation is stochastically stable. As before, we assume populations are
large and that selection is strong. For simplicity, we also assume that the
game continues almost certainly after each round (i.e., $\delta$ approaches
one). The details of this analysis can be found in the electronic
supplementary material. In the first two scenarios, we interestingly find that
for $b\\!>\\!2c$, cooperation is stochastically stable when
$q\\!<\\!\frac{\sqrt{2}}{2}\approx 0.707$. Comparing this condition with the
more stringent condition in Eq. (4) suggests that there are now more
conditionally cooperative strategies that can sustain cooperation. Hence,
cooperation should evolve more easily. In the last scenario, we find that
cooperation is stochastically stable when $q<1-\frac{c}{b}$, which is the same
condition as in Eq. (3), even though only a single co-player is considered
instead of the whole population.

We complement these analytical results with additional simulations, see Figure
3. We observe that a minimal increase in the players’ payoff memory (compared
to the baseline case with a single round recalled) can promote cooperation
considerably. Specifically, in all four scenarios with extended memory, we see
similar cooperation rates, and they approach the rates observed under perfect
memory. These results suggest that while it takes some payoff memory to
sustain substantial cooperation rates, the requirements on memory seem to be
rather modest. Already remembering a few interactions, either with the same
co-player or across different co-players, may provide players with enough
information to adopt reciprocal strategies.

## 4 Discussion

In economics, if payoff is measured in terms of money and a decision is to be
made at time $t$ in the future, then currency accumulated early on is weighted
more in that decision because it has more time to accumulate interest. Such a
model discounts the future relative to the past. It also does not necessarily
require “memory” because rewards are accumulated into a factor used in
decision-making; the specific time stamps of rewards do not themselves provide
better information beyond their effects on total payoff. In a similar fashion,
the probabilistic interpretation of discounting as a continuation probability
(15) also effectively discounts the future relative to the past. A foraging
animal deciding between two behaviors might tend to choose the one that yields
a moderate reward sooner relative to a larger reward later, since earlier
rewards (e.g., food) contribute to immediate survival, and there is no
guarantee that later rewards will happen at all (66).

Within the context of a single repeated game, the model we consider here is,
in some ways, dual to the classical model of temporally-discounted rewards in
repeated games. Instead of making decisions based on expected rewards in the
future, we consider individuals who make decisions based on actual rewards in
the past. Thus, the ability to estimate the future payoff of a strategy is
replaced by the memory of how this strategy previously fared against others.
This involves two time scales: interaction partners and rounds within those
interactions. As a result, we are dealing with a model that discounts the past
rather than the future. Intriguingly, treating payoffs in this manner is
reminiscent of the reward-smoothing technique of “eligibility traces” in
reinforcement learning (67), which uses past rewards (discounted
appropriately) to shape present perceived payoff. There is a sound basis for
this method in neuroscience, where rewards and (temporal-difference) learning
are associated to dopaminergic neurons (68) and spike-timing-dependent
plasticity (69). This suggests a more biologically-encoded interpretation of
memory, which is equally applicable to models of direct reciprocity where
rewards have a neurological basis.

Of course, the precise nature of “memory” also depends on what payoffs in a
game represent, which should be taken into account when applying game-
theoretic models. For example, a payoff stream of monetary currency might
truly accumulate and not require memory on the parts of agents. Even in the
context of money, however, not all of what was obtained in the past is
necessarily available at the time a decision is made, which brings memory into
play. The serial position effect in human psychology shows that in an ordered
list of items (e.g., words), humans tend to have difficulty remembering the
entirety of sequences, demonstrating moderate recall for those items coming
earlier (primacy effect), substantial recall for those coming later (recency
effect), and lower recall for those in between (70). It is therefore
reasonable that when presented with a stream of payoffs, whether on the
timescale of pairing for repeated interactions or in a stream of one-shot
games, players might be able to effectively incorporate only the most recent
payoffs.

In fact, even beyond specific psychological considerations, a curious
interpretation of payoffs arises from the formula commonly used for expected
payoffs in repeated games. If $\delta\in\left[0,1\right)$ is the probability
of continuing to another round in the game, then the expected payoff to an
agent is $\left(1-\delta\right)\sum_{t=0}^{\infty}\delta^{t}u_{t}$, where
$u_{t}$ is the reward the agent receives in the stage game at time $t$. Here,
additional stochasticity arises due to uncertainty in the game length, and an
agent might not be able to compute his or her expected payoff for use in
decision-making. The probability that the game terminates after the
interaction at time $T$ is $\delta^{T}\left(1-\delta\right)$, in which case
$\left(1-\delta\right)\sum_{t=0}^{\infty}\delta^{t}u_{t}$ is exactly the
expected payoff the agent receives at time $T$, i.e. _in the last round_. As
an unbiased estimator of this expectation, the agent might thus use $u_{T}$ as
a proxy for “success” when evaluating his or her behavior. This gives a purely
model-driven justification for why considering payoff in the last round of the
game can result in more realistic extensions of traditional models.

We note that the expected _total_ payoff in the game,
$u_{0}+u_{1}+\cdots+u_{T}$, is given by $\sum_{t=0}^{\infty}\delta^{t}u_{t}$.
This version of “expected payoff” appears less common in the literature on
direct reciprocity than its normalization,
$\left(1-\delta\right)\sum_{t=0}^{\infty}\delta^{t}u_{t}$, likely owing to the
fact that payoffs can grow arbitrarily large with sufficiently long time
horizons ($\delta\rightarrow 1^{-}$). Non-normalized payoffs interfere with
selection intensity ($\beta$) in models of social imitation, which is
(presumably) why they appear less frequently in the literature. On that point,
we note that many of the differences between realized and expected payoffs
disappear in the limit of weak selection (71). Non-weak selection can
introduce substantial differences between models with realized and expected
payoffs (72), which is especially important to understand in models of social
systems with cultural transmission (73).

Our main contribution is an application of these ideas to direct reciprocity,
which is one of the key mechanisms to explain why unrelated individuals might
cooperate (7). According to this mechanism, cooperation pays if it makes the
interaction partner more cooperative in future. To describe which strategies
are most effective, the previous theoretical literature assumes that the
evolutionary dynamics are driven by the players’ expected payoffs (29, 30, 31,
32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44). To the extent that
strategies are learned (not inherited), this assumption seems to impose rather
stringent requirements on the individuals’ cognitive abilities. In the most
extreme case, this would require individuals to remember (or compute) their
payoffs against all population members, for all possible ways in which their
repeated games may unfold. This assumption introduces a curious inconsistency
in how these models represent an individual’s cognitive abilities. For playing
their games, individuals are often assumed to only recall the outcome of the
very last round. Yet to update their strategies, individuals are implicitly
assumed to have a record of the outcome of all rounds, across all interaction
partners.

It is natural to ask, then, to what extent perfect payoff memory is in fact
required for the evolution of reciprocity. To this end, we consider a model in
which individuals only remember the payoff of their very last interaction, or
the payoffs of the last few interactions. By only considering an individual’s
most recent experiences, the evolutionary process is subject to additional
stochasticity. Strategies that perform well on average (across an entire
repeated game and across many interaction partners) may still get replaced if
the respective player happened to yield an inferior payoff in the very last
round. A similar element of stochasticity has been previously explored in the
context of one-shot (non-repeated) games (74, 75, 76, 77, 78). This literature
studies which strategies are selected for when individuals only interact with
a finite sample of population members. In the respective models, individuals
can only choose among two strategies. They can either cooperate or defect, and
stochastic sampling affects which of these two strategies is favored. Instead,
in repeated games, players have access to a large set of strategies (in our
case, all reactive strategies; 79). Here, stochastic sampling does not only
affect whether cooperative or non-cooperative strategies are favored; it also
affects which conditionally cooperative strategies are favored.

To address these questions, we combine analytical methods and computer
simulations. In the most extreme case, we consider individuals who update
their strategies based on only one piece of information: the last round of a
single repeated game. For that case, we find that individuals are less
generous, and they tend to be less cooperative overall (Figure 1). However,
once individuals update their strategies based on two or more recent
experiences, overall cooperation rates quickly approach the levels observed
under perfect payoff memory (Figure 3). These findings suggest that models
based on expected payoffs can serve as a useful approximation to more
realistic models with limited payoff memory. Our findings also contribute to a
wider literature that explores which kinds of cognitive capacities are
required for reciprocal altruism to be feasible (e.g., 80, 81). While more
payoff memory is always favorable, reciprocal cooperation can already be
sustained if individuals have a record of two or three past outcomes. We
believe that this kind of result, derived entirely within a theoretical model,
is crucial for making model-informed deductions about reciprocity in natural
systems.

Data accessibility. All data and code used in this manuscript are openly
available at
https://zenodo.org/records/10066227.

## References

  * (1) Hofbauer, J., Sigmund, K. _et al._ _Evolutionary games and population dynamics_ (Cambridge university press, 1998).
  * (2) Nowak, M. A., Sasaki, A., Taylor, C. & Fudenberg, D.  Emergence of cooperation and evolutionary stability in finite populations.  _Nature_ 428, 646–650 (2004).
  * (3) Hauert, C. & Szabó, G.  Game theory and physics.  _American Journal of Physics_ 73, 405–414 (2005).
  * (4) Traulsen, A. & Glynatsi, N. E.  The future of theoretical evolutionary game theory.  _Philosophical Transactions B_ (2022).
  * (5) Maynard Smith, J. & Price, G. R.  The logic of animal conflict.  _Nature_ 246, 15–18 (1973).
  * (6) Stein, A. _et al._ Stackelberg evolutionary game theory: how to manage evolving systems.  _Philosophical Transactions of the Royal Society B_ 378, 20210495 (2023).
  * (7) Nowak, M. A.  Five rules for the evolution of cooperation.  _Science_ 314, 1560–1563 (2006).
  * (8) Smith, J. M.  _Evolution and the Theory of Games_ (Cambridge university press, 1982).
  * (9) Wu, B., Bauer, B., Galla, T. & Traulsen, A.  Fitness-based models and pairwise comparison models of evolutionary games are typically different—even in unstructured populations.  _New Journal of Physics_ 17, 023043 (2015).
  * (10) Smolla, M. _et al._ Underappreciated features of cultural evolution.  _Philosophical Transactions of the Royal Society B_ 376, 20200259 (2021).
  * (11) Denton, K. K., Ram, Y. & Feldman, M. W.  Conformity and content-biased cultural transmission in the evolution of altruism.  _Theoretical Population Biology_ 143, 52–61 (2022).
  * (12) Chatterjee, K., Zufferey, D. & Nowak, M. A.  Evolutionary game dynamics in populations with different learners.  _Journal of Theoretical Biology_ 301, 161–173 (2012).
  * (13) Trivers, R. L.  The evolution of reciprocal altruism.  _The Quarterly review of biology_ 46, 35–57 (1971).
  * (14) Rapoport, A. & Chammah, A. M.  _Prisoner’s Dilemma_ (University of Michigan Press, Ann Arbor, 1965).
  * (15) Axelrod, R. & Hamilton, W. D.  The evolution of cooperation.  _Science_ 211, 1390–1396 (1981).
  * (16) Molander, P.  The optimal level of generosity in a selfish, uncertain environment.  _Journal of Conflict Resolution_ 29, 611–618 (1985).
  * (17) Nowak, M. A. & Sigmund, K.  Tit for tat in heterogeneous populations.  _Nature_ 355, 250–253 (1992).
  * (18) Fischbacher, U., Gächter, S. & Fehr, E.  Are people conditionally cooperative? Evidence from a public goods experiment.  _Economic Letters_ 71, 397–404 (2001).
  * (19) Rand, D. G. & Nowak, M. A.  Human cooperation.  _Trends in Cogn. Sciences_ 117, 413–425 (2012).
  * (20) Dal Bó, P. & Fréchette, G. R.  Strategy choice in the infinitely repeated prisoner’s dilemma.  _American Economic Review_ 109, 3929–3952 (2019).
  * (21) Montero-Porras, E., Grujić, J., Fernández Domingos, E. & Lenaerts, T.  Inferring strategies from observations in long iterated prisoner’s dilemma experiments.  _Scientific Reports_ 12, 7589 (2022).
  * (22) Rossetti, C. & Hilbe, C.  Direct reciprocity among humans.  _Ethology_ https://doi.org/10.1111/eth.13407 (2023).
  * (23) Carter, G. G. & Wilkinson, G. S.  Food sharing in vampire bats, reciprocal help predicts donations more than relatedness or harassment.  _Proceedings of the Royal Society B: Biological Sciences_ 280, 20122573 (2013).
  * (24) Schweinfurth, M. K., Aeschbacher, J., Santi, M. & Taborsky, M.  Male norway rats cooperate according to direct but not generalized reciprocity rules.  _Animal Behaviour_ 152, 93–101 (2019).
  * (25) Voelkl, B. _et al._ Matching times of leading and following suggest cooperation through direct reciprocity during V-formation flight in ibis.  _Proceedings of the National Academy of Sciences USA_ 112, 2115–2120 (2015).
  * (26) Clutton-Brock, T.  Cooperation between non-kin in animal societies.  _Nature_ 462, 51–57 (2009).
  * (27) Silk, J. B.  Reciprocal altruism.  _Current Biology_ 23, 827–828 (2013).
  * (28) Taborsky, M.  Social evolution: Reciprocity there is.  _Current Biology_ 23, 486–488 (2013).
  * (29) Brauchli, K., Killingback, T. & Doebeli, M.  Evolution of cooperation in spatially structured populations.  _Journal of Theoretical Biology_ 200, 405–417 (1999).
  * (30) Brandt, H. & Sigmund, K.  The good, the bad and the discriminator - errors in direct and indirect reciprocity.  _Journal of Theoretical Biology_ 239, 183–194 (2006).
  * (31) Ohtsuki, H. & Nowak, M. A.  Direct reciprocity on graphs.  _Journal of Theoretical Biology_ 247, 462–470 (2007).
  * (32) Szolnoki, A., Perc, M. & Szabó, G.  Phase diagrams for three-strategy evolutionary prisoner’s dilemma games on regular graphs.  _Physical Review E_ 80, 056104 (2009).
  * (33) Imhof, L. A. & Nowak, M. A.  Stochastic evolutionary dynamics of direct reciprocity.  _Proceedings of the Royal Society B: Biological Sciences_ 277, 463–468 (2010).
  * (34) van Segbroeck, S., Pacheco, J. M., Lenaerts, T. & Santos, F. C.  Emergence of fairness in repeated group interactions.  _Physical Review Letters_ 108, 158104 (2012).
  * (35) Grujic, J., Cuesta, J. A. & Sanchez, A.  On the coexistence of cooperators, defectors and conditional cooperators in the multiplayer iterated prisoner’s dilemma.  _Journal of Theoretical Biology_ 300, 299–308 (2012).
  * (36) Martinez-Vaquero, L. A., Cuesta, J. A. & Sanchez, A.  Generosity pays in the presence of direct reciprocity: A comprehensive study of 2$\times$ 2 repeated games.  _PLoS One_ 7, e35135 (2012).
  * (37) Stewart, A. J. & Plotkin, J. B.  From extortion to generosity, evolution in the iterated prisoner’s dilemma.  _Proceedings of the National Academy of Sciences USA_ 110, 15348–15353 (2013).
  * (38) Pinheiro, F. L., Vasconcelos, V. V., Santos, F. C. & Pacheco, J. M.  Evolution of all-or-none strategies in repeated public goods dilemmas.  _PLoS Comput Biol_ 10, e1003945 (2014).
  * (39) Stewart, A. J. & Plotkin, J. B.  The evolvability of cooperation under local and non-local mutations.  _Games_ 6, 231–250 (2015).
  * (40) Baek, S. K., Jeong, H.-C., Hilbe, C. & Nowak, M. A.  Comparing reactive and memory-one strategies of direct reciprocity.  _Scientific reports_ 6, 1–13 (2016).
  * (41) McAvoy, A. & Nowak, M. A.  Reactive learning strategies for iterated games.  _Proceedings of the Royal Society A_ 475, 20180819 (2019).
  * (42) Glynatsi, N. E. & Knight, V. A.  Using a theory of mind to find best responses to memory-one strategies.  _Scientific Reports_ 10, 17287 (2020).
  * (43) Schmid, L., Hilbe, C., Chatterjee, K. & Nowak, M. A.  Direct reciprocity between individuals that use different strategy spaces.  _PLoS Computational Biology_ 18, e1010149 (2022).
  * (44) Murase, Y., Hilbe, C. & Baek, S. K.  Evolution of direct reciprocity in group-structured populations.  _Scientific Reports_ 12, 18645 (2022).
  * (45) Cooney, D. B.  Assortment and reciprocity mechanisms for promotion of cooperation in a model of multilevel selection.  _Bulletin of Mathematical Biology_ 84, 126 (2022).
  * (46) Chen, X. & Fu, F.  Outlearning extortioners: Unbending strategies can foster reciprocal fairness and cooperation.  _PNAS Nexus_ 2, pgad176 (2023).
  * (47) Hauert, C. & Schuster, H. G.  Effects of increasing the number of players and memory size in the iterated prisoner’s dilemma: a numerical approach.  _Proceedings of the Royal Society of London. Series B: Biological Sciences_ 264, 513–519 (1997).
  * (48) van Veelen, M., García, J., Rand, D. G. & Nowak, M. A.  Direct reciprocity in structured populations.  _Proceedings of the National Academy of Sciences USA_ 109, 9929–9934 (2012).
  * (49) Stewart, A. J. & Plotkin, J. B.  Small groups and long memories promote cooperation.  _Scientific reports_ 6, 1–11 (2016).
  * (50) Li, J. _et al._ Evolution of cooperation through cumulative reciprocity.  _Nature Computational Science_ 2, 677–686 (2022).
  * (51) Murase, Y. & Baek, S. K.  Grouping promotes both partnership and rivalry with long memory in direct reciprocity.  _PLoS Computational Biology_ 19, e1011228 (2023).
  * (52) Sigmund, K.  _The calculus of selfishness_ (Princeton University Press, 2010).
  * (53) Friedman, J.  A non-cooperative equilibrium for supergames.  _Review of Economic Studies_ 38, 1–12 (1971).
  * (54) Akin, E.  The iterated prisoner’s dilemma: Good strategies and their dynamics.  In Assani, I. (ed.) _Ergodic Theory, Advances in Dynamics_ , 77–107 (de Gruyter, Berlin, 2016).
  * (55) Hilbe, C., Traulsen, A. & Sigmund, K.  Partners or rivals? Strategies for the iterated prisoner’s dilemma.  _Games and Economic Behavior_ 92, 41–52 (2015).
  * (56) Stewart, A. J. & Plotkin, J. B.  Collapse of cooperation in evolving games.  _Proceedings of the National Academy of Sciences USA_ 111, 17558 – 17563 (2014).
  * (57) Traulsen, A., Pacheco, J. M. & Nowak, M. A.  Pairwise comparison and selection temperature in evolutionary game dynamics.  _Journal of theoretical biology_ 246, 522–529 (2007).
  * (58) Blume, L. E.  The statistical mechanics of best-response strategy revision.  _Games and Economic Behavior_ 11, 111–145 (1995).
  * (59) Szabó, G. & Tőke, C.  Evolutionary Prisoner’s Dilemma game on a square lattice.  _Physical Review E_ 58, 69–73 (1998).
  * (60) Fudenberg, D. & Imhof, L. A.  Imitation processes with small mutations.  _Journal of Economic Theory_ 131, 251–262 (2006).
  * (61) Wu, B., Gokhale, C. S., Wang, L. & Traulsen, A.  How small are small mutation rates?  _Journal of Mathematical Biology_ 64, 803–827 (2012).
  * (62) McAvoy, A.  Comment on “Imitation processes with small mutations”.  _J. Econ. Theory_ 159, 66–69 (2015).
  * (63) Schmid, L., Chatterjee, K., Hilbe, C. & Nowak, M.  A unified framework of direct and indirect reciprocity.  _Nature Human Behaviour_ 5, 1292–1302 (2021).
  * (64) Nowak, M. A. & Sigmund, K.  A strategy of win-stay, lose-shift that outperforms tit-for-tat in the Prisoner’s Dilemma game.  _Nature_ 364, 56–58 (1993).
  * (65) Imhof, L. A., Fudenberg, D. & Nowak, M. A.  Tit-for-tat or win-stay, lose-shift?  _Journal of Theoretical Biology_ 247, 574–580 (2007).
  * (66) Stephens, D. W. & Krebs, J. R.  _Foraging Theory_.  Monographs in Behavior and Ecology (Princeton University Press, 1986).
  * (67) Sutton, R. S. & Barto, A. G.  _Reinforcement Learning: An Introduction_ (The MIT Press, 2018), second edn.
  * (68) Schultz, W.  Predictive Reward Signal of Dopamine Neurons.  _Journal of Neurophysiology_ 80, 1–27 (1998).
  * (69) Dan, Y. & Poo, M.-M.  Spike Timing-Dependent Plasticity of Neural Circuits.  _Neuron_ 44, 23–30 (2004).
  * (70) Murdock, B. B.  The serial position effect of free recall.  _Journal of Experimental Psychology_ 64, 482–488 (1962).
  * (71) McAvoy, A., Allen, B. & Nowak, M. A.  Social goods dilemmas in heterogeneous societies.  _Nature Human Behaviour_ 4, 819–831 (2020).
  * (72) McAvoy, A., Rao, A. & Hauert, C.  Intriguing effects of selection intensity on the evolution of prosocial behaviors.  _PLOS Computational Biology_ 17, e1009611 (2021).
  * (73) Cavalli-Sforza, L. L. & Feldman, M. W.  _Cultural Transmission and Evolution: A Quantitative Approach_.  Cultural Transmission and Evolution: A Quantitative Approach (Princeton University Press, 1981).
  * (74) Sánchez, A. & Cuesta, J. A.  Altruism may arise from individual selection.  _Journal of theoretical biology_ 235, 233–240 (2005).
  * (75) Roca, C. P., Cuesta, J. A. & Sánchez, A.  Time scales in evolutionary dynamics.  _Physical review letters_ 97, 158701 (2006).
  * (76) Traulsen, A., Nowak, M. A. & Pacheco, J. M.  Stochastic payoff evaluation increases the temperature of selection.  _Journal of theoretical biology_ 244, 349–356 (2007).
  * (77) Woelfing, B. & Traulsen, A.  Stochastic sampling of interaction partners versus deterministic payoff assignment.  _Journal of Theoretical Biology_ 257, 689–695 (2009).
  * (78) Hauert, C. & Miekisz, J.  Effects of sampling interaction partners and competitors in evolutionary games.  _Physical Review E_ 98, 052301 (2018).
  * (79) Nowak, M. & Sigmund, K.  Game-dynamical aspects of the prisoner’s dilemma.  _Applied Mathematics and Computation_ 30, 191–213 (1989).
  * (80) Stevens, J. R., Volstorf, J., Schooler, L. J. & Rieskamp, J.  Forgetting constrains the emergence of cooperative decision strategies.  _Frontiers in Psychology_ 1, 235 (2011).
  * (81) Volstorf, J., Rieskamp, J. & Stevens, J. R.  The good, the bad, and the rare: Memory for partners in social interactions.  _PloS one_ 6, e18945 (2011).

Figure 1: Evolutionary dynamics under perfect and limited payoff memory. The
leftmost panels give a schematic overview of the two main scenarios we
compare. The two scenarios differ in how many past interactions individuals
take into account when updating their strategy. In the scenario with perfect
payoff memory, individuals consider all their past interactions (against all
population members, and taking every turn of each repeated game into account).
In the scenario with limited payoff memory, individuals only consider their
very last interaction (against one specific population member, taking into
account only one round of the repeated game). The four panels on the right
side depict the outcome of evolutionary simulations for repeated games with
either a low or a high benefit of cooperation. Colors represent how often the
respective region of the strategy space is visited over time. In all four
panels, two regions are visited particularly often. One region corresponds to
a neighborhood of ALLD with $p\approx q\\!\approx\\!0$ (lower left corner).
The other region corresponds to a strip of conditionally cooperative
strategies with $p\\!\approx\\!1$ and $q$ satisfying the constraints (3) and
(4), respectively (lower right corner). The resulting average cooperation rate
depends on which of these two neighborhoods is visited more often. Simulations
are run for $T\\!=\\!10^{7}$ time steps, using a cost $c\\!=\\!1$, a
continuation probability of $\delta\\!=\\!0.999$ and a selection strength of
$\beta\\!=\\!1$, in a population of size $N\\!=\\!100$.

Figure 2: Evolution of direct reciprocity for different parameter values. To
explore the robustness of our results, we have run simulations for different
benefit values (left panels, a), and for different selection strengths (right
panels, b). In each case, we record the resulting average cooperation rate
over the entire simulation (upper panels). In addition we record the
individuals’ average generosity. Here, we only take into account those
residents with $p\\!\approx\\!1$ and we compute the average of their
cooperation probability $q$. These simulations suggest that perfect payoff
memory consistently leads to more cooperation and more generosity. Unless
explicitly varied, the parameters of the simulation are $N\\!=\\!100$,
$b\\!=\\!3$, $c\\!=\\!1$, $\beta\\!=\\!1$, $\delta\\!=\\!0.99$. Simulations
are run for $T\\!=\\!5\times 10^{7}$ time steps for each parameter
combination.

Figure 3: Average cooperation rates for different payoff memories. We vary how
much information individuals take into account when updating their strategies.
From left to right, we consider the following cases. (i) Updating occurs based
on expected payoffs (perfect memory), (ii) it occurs based on the last round
of one interaction (limited memory), (iii) based on the last round of two
interactions, (iv) based on the last two rounds of one interaction, (v) based
on the last two rounds of two interactions, and (vi) based on the average
payoff of one interaction. Again, simulations are run either for a comparably
low benefit of cooperation ($b/c\\!=\\!3$), or for a high benefit
($b/c\\!=\\!10$). We observe that perfect memory always yields the highest
cooperation rate. However, when individuals take into account at least two
past interactions – cases (iii) to (vi) – evolving cooperation rates are close
to this optimum. Baseline parameters are the same as in Figure 2.

See pages - of si.pdf


Sklodowska-Curie grant agreement No 721463 to the SUNDIAL ITN, and from the
Spanish Ministry of Economy and Competitiveness (MINECO) under grant number
AYA2016-76219-P. SGB acknowledges support from the research project
PID2019-106027GA-C44 of the Spanish Ministerio de Ciencia e Innovación, and GV
acknowledges support from ANID program FONDECYT Postdoctorado 3200802.

## References

  * Accurso et al. (2017) Accurso, G., Saintonge, A., Catinella, B., et al. 2017, MNRAS, 470, 4750
  * Akhlaghi (2018) Akhlaghi, M. 2018, Gnuastro: GNU Astronomy Utilities, Astrophysics Source Code Library, record ascl:1801.009
  * Altintas et al. (2006) Altintas, I., Barney, O., Cheng, Z., et al. 2006, in Journal of Physics Conference Series, Vol. 46, Journal of Physics Conference Series, 468–478
  * Aniano et al. (2011) Aniano, G., Draine, B. T., Gordon, K. D., & Sandstrom, K. 2011, PASP, 123, 1218
  * Annibali et al. (2009) Annibali, F., Tosi, M., Monelli, M., et al. 2009, AJ, 138, 169
  * Annibali et al. (2015) Annibali, F., Tosi, M., Pasquali, A., et al. 2015, AJ, 150, 143
  * Bell et al. (2006) Bell, T. A., Roueff, E., Viti, S., & Williams, D. A. 2006, MNRAS, 371, 1865
  * Berg et al. (2012) Berg, D. A., Skillman, E. D., Marble, A. R., et al. 2012, ApJ, 754, 98
  * Bigiel et al. (2008) Bigiel, F., Leroy, A., Walter, F., et al. 2008, AJ, 136, 2846
  * Billett et al. (2002) Billett, O. H., Hunter, D. A., & Elmegreen, B. G. 2002, AJ, 123, 1454
  * Bisbas et al. (2021) Bisbas, T. G., Tan, J. C., & Tanaka, K. E. I. 2021, MNRAS, 502, 2701
  * Bohlin et al. (1978) Bohlin, R. C., Savage, B. D., & Drake, J. F. 1978, ApJ, 224, 132
  * Bolatto et al. (1999) Bolatto, A. D., Jackson, J. M., & Ingalls, J. G. 1999, ApJ, 513, 275
  * Bolatto et al. (2011) Bolatto, A. D., Leroy, A. K., Jameson, K., et al. 2011, ApJ, 741, 12
  * Bolatto et al. (2008) Bolatto, A. D., Leroy, A. K., Rosolowsky, E., Walter, F., & Blitz, L. 2008, ApJ, 686, 948
  * Bolatto et al. (2013) Bolatto, A. D., Wolfire, M., & Leroy, A. K. 2013, ARA&A, 51, 207
  * Boquien et al. (2013) Boquien, M., Boselli, A., Buat, V., et al. 2013, A&A, 554, A14
  * Calzetti et al. (2015) Calzetti, D., Johnson, K. E., Adamo, A., et al. 2015, ApJ, 811, 75
  * Calzetti et al. (1997) Calzetti, D., Meurer, G. R., Bohlin, R. C., et al. 1997, AJ, 114, 1834
  * Cannon et al. (2003) Cannon, J. M., Dohm-Palmer, R. C., Skillman, E. D., et al. 2003, AJ, 126, 2806
  * Cannon et al. (2004a) Cannon, J. M., McClure-Griffiths, N. M., Skillman, E. D., & Côté, S. 2004a, ApJ, 607, 274
  * Cannon et al. (2005) Cannon, J. M., Skillman, E. D., Sembach, K. R., & Bomans, D. J. 2005, ApJ, 618, 247
  * Cannon et al. (2006) Cannon, J. M., Smith, J.-D. T., Walter, F., et al. 2006, ApJ, 647, 293
  * Cardelli et al. (1989) Cardelli, J. A., Clayton, G. C., & Mathis, J. S. 1989, ApJ, 345, 245
  * Casasola et al. (2022) Casasola, V., Bianchi, S., Magrini, L., et al. 2022, A&A, 668, A130
  * Cignoni et al. (2018) Cignoni, M., Sacchi, E., Aloisi, A., et al. 2018, ApJ, 856, 62
  * Cignoni et al. (2019) Cignoni, M., Sacchi, E., Tosi, M., et al. 2019, ApJ, 887, 112
  * Cormier et al. (2014) Cormier, D., Madden, S. C., Lebouteiller, V., et al. 2014, A&A, 564, A121
  * Côté et al. (2000) Côté, S., Carignan, C., & Freeman, K. C. 2000, AJ, 120, 3027
  * Cresci et al. (2005) Cresci, G., Vanzi, L., & Sauvage, M. 2005, A&A, 433, 447
  * Croxall et al. (2009) Croxall, K. V., van Zee, L., Lee, H., et al. 2009, ApJ, 705, 723
  * Di Teodoro & Fraternali (2015) Di Teodoro, E. M. & Fraternali, F. 2015, MNRAS, 451, 3021
  * Disney et al. (1989) Disney, M., Davies, J., & Phillipps, S. 1989, MNRAS, 239, 939
  * Draine (1978) Draine, B. T. 1978, ApJS, 36, 595
  * Draine (2003b) Draine, B. T. 2003b, ApJ, 598, 1017
  * Elson et al. (2013) Elson, E. C., de Blok, W. J. G., & Kraan-Korteweg, R. C. 2013, MNRAS, 429, 2550
  * Emsellem et al. (2022) Emsellem, E., Schinnerer, E., Santoro, F., et al. 2022, A&A, 659, A191
  * Feldmann et al. (2012) Feldmann, R., Gnedin, N. Y., & Kravtsov, A. V. 2012, ApJ, 747, 124
  * Freudling et al. (2013) Freudling, W., Romaniello, M., Bramich, D. M., et al. 2013, A&A, 559, A96
  * Galametz et al. (2012) Galametz, M., Kennicutt, R. C., Albrecht, M., et al. 2012, MNRAS, 425, 763
  * Glover & Clark (2016) Glover, S. C. O. & Clark, P. C. 2016, MNRAS, 456, 3596
  * Glover & Mac Low (2011) Glover, S. C. O. & Mac Low, M. M. 2011, MNRAS, 412, 337
  * Gong et al. (2018) Gong, M., Ostriker, E. C., & Kim, C.-G. 2018, ApJ, 858, 16
  * Gong et al. (2020) Gong, M., Ostriker, E. C., Kim, C.-G., & Kim, J.-G. 2020, ApJ, 903, 142
  * Gong et al. (2017) Gong, M., Ostriker, E. C., & Wolfire, M. G. 2017, ApJ, 843, 38
  * Heckman et al. (2001) Heckman, T. M., Sembach, K. R., Meurer, G. R., et al. 2001, ApJ, 554, 1021
  * Hensley & Draine (2021) Hensley, B. S. & Draine, B. T. 2021, ApJ, 906, 73
  * Hollenbach & Tielens (1997) Hollenbach, D. J. & Tielens, A. G. G. M. 1997, ARA&A, 35, 179
  * Hu et al. (2022) Hu, C.-Y., Schruba, A., Sternberg, A., & van Dishoeck, E. F. 2022, ApJ, 931, 28
  * Hughes et al. (2014) Hughes, T. M., Baes, M., Fritz, J., et al. 2014, A&A, 565, A4
  * Hunt et al. (2015a) Hunt, L. K., Draine, B. T., Bianchi, S., et al. 2015a, A&A, 576, A33
  * Hunt et al. (2020) Hunt, L. K., Tortora, C., Ginolfi, M., & Schneider, R. 2020, A&A, 643, A180
  * Hunt et al. (2017) Hunt, L. K., Weiß, A., Henkel, C., et al. 2017, A&A, 606, A99
  * Imara et al. (2020) Imara, N., De Looze, I., Faesi, C. M., & Cormier, D. 2020, ApJ, 895, 21
  * Indebetouw et al. (2013) Indebetouw, R., Brogan, C., Chen, C. H. R., et al. 2013, ApJ, 774, 73
  * Karachentsev et al. (2013) Karachentsev, I. D., Makarov, D. I., & Kaisina, E. I. 2013, AJ, 145, 101
  * Kirkpatrick et al. (2014) Kirkpatrick, A., Calzetti, D., Kennicutt, R., et al. 2014, ApJ, 789, 130
  * Kobulnicky & Skillman (2008) Kobulnicky, H. A. & Skillman, E. D. 2008, AJ, 135, 527
  * Kobulnicky et al. (1997) Kobulnicky, H. A., Skillman, E. D., Roy, J.-R., Walsh, J. R., & Rosa, M. R. 1997, ApJ, 477, 679
  * Krumholz et al. (2009) Krumholz, M. R., McKee, C. F., & Tumlinson, J. 2009, ApJ, 693, 216
  * Lee et al. (2018) Lee, C., Leroy, A. K., Bolatto, A. D., et al. 2018, MNRAS, 474, 4672
  * Lee et al. (2015) Lee, C., Leroy, A. K., Schnee, S., et al. 2015, MNRAS, 450, 2708
  * Lelli et al. (2014a) Lelli, F., Verheijen, M., & Fraternali, F. 2014a, A&A, 566, A71
  * Lelli et al. (2014b) Lelli, F., Verheijen, M., & Fraternali, F. 2014b, MNRAS, 445, 1694
  * Leroy et al. (2009) Leroy, A. K., Bolatto, A., Bot, C., et al. 2009, ApJ, 702, 352
  * Leroy et al. (2013) Leroy, A. K., Lee, C., Schruba, A., et al. 2013, ApJ, 769, L12
  * Lombardi et al. (2006) Lombardi, M., Alves, J., & Lada, C. J. 2006, A&A, 454, 781
  * López-Sánchez et al. (2012) López-Sánchez, Á. R., Koribalski, B. S., van Eymeren, J., et al. 2012, MNRAS, 419, 1051
  * Madden et al. (2013) Madden, S. C., Rémy-Ruyer, A., Galametz, M., et al. 2013, PASP, 125, 600
  * Marasco et al. (2023) Marasco, A., Belfiore, F., Cresci, G., et al. 2023, A&A, 670, A92
  * Martins et al. (2012) Martins, F., Förster Schreiber, N. M., Eisenhauer, F., & Lutz, D. 2012, A&A, 547, A17
  * McMullin et al. (2007) McMullin, J. P., Waters, B., Schiebel, D., Young, W., & Golap, K. 2007, in Astronomical Society of the Pacific Conference Series, Vol. 376, Astronomical Data Analysis Software and Systems XVI, ed. R. A. Shaw, F. Hill, & D. J. Bell, 127
  * McQuinn et al. (2010) McQuinn, K. B. W., Skillman, E. D., Cannon, J. M., et al. 2010, ApJ, 721, 297
  * Misselt et al. (2001) Misselt, K. A., Gordon, K. D., Clayton, G. C., & Wolff, M. J. 2001, ApJ, 551, 277
  * Miura et al. (2018) Miura, R. E., Espada, D., Hirota, A., et al. 2018, ApJ, 864, 120
  * Monreal-Ibero et al. (2017) Monreal-Ibero, A., Walsh, J. R., Iglesias-Páramo, J., et al. 2017, A&A, 603, A130
  * Narayanan et al. (2011) Narayanan, D., Krumholz, M., Ostriker, E. C., & Hernquist, L. 2011, MNRAS, 418, 664
  * Narayanan et al. (2012) Narayanan, D., Krumholz, M. R., Ostriker, E. C., & Hernquist, L. 2012, MNRAS, 421, 3127
  * Natta & Panagia (1984) Natta, A. & Panagia, N. 1984, ApJ, 287, 228
  * O’Connell et al. (1994) O’Connell, R. W., Gallagher, John S., I., & Hunter, D. A. 1994, ApJ, 433, 65
  * O’Donnell (1994) O’Donnell, J. E. 1994, ApJ, 422, 158
  * Ott (2010) Ott, S. 2010, in Astronomical Society of the Pacific Conference Series, Vol. 434, Astronomical Data Analysis Software and Systems XIX, ed. Y. Mizumoto, K. I. Morita, & M. Ohishi, 139
  * Pak et al. (1998) Pak, S., Jaffe, D. T., van Dishoeck, E. F., Johansson, L. E. B., & Booth, R. S. 1998, ApJ, 498, 735
  * Papadopoulos et al. (2004) Papadopoulos, P. P., Thi, W. F., & Viti, S. 2004, MNRAS, 351, 147
  * Pilyugin et al. (2015) Pilyugin, L. S., Grebel, E. K., & Zinchenko, I. A. 2015, MNRAS, 450, 3254
  * Pineda et al. (2008) Pineda, J. E., Caselli, P., & Goodman, A. A. 2008, ApJ, 679, 481
  * Planck Collaboration et al. (2014) Planck Collaboration, Abergel, A., Ade, P. A. R., et al. 2014, A&A, 571, A11
  * Poglitsch et al. (2010) Poglitsch, A., Waelkens, C., Geis, N., et al. 2010, A&A, 518, L2
  * Remy et al. (2017) Remy, Q., Grenier, I. A., Marshall, D. J., & Casandjian, J. M. 2017, A&A, 601, A78
  * Rémy-Ruyer et al. (2014) Rémy-Ruyer, A., Madden, S. C., Galliano, F., et al. 2014, A&A, 563, A31
  * Rubio et al. (2015) Rubio, M., Elmegreen, B. G., Hunter, D. A., et al. 2015, Nature, 525, 218
  * Rubio et al. (1993) Rubio, M., Lequeux, J., & Boulanger, F. 1993, A&A, 271, 9
  * Sabbi et al. (2018) Sabbi, E., Calzetti, D., Ubeda, L., et al. 2018, ApJS, 235, 23
  * Sanders et al. (1984) Sanders, D. B., Solomon, P. M., & Scoville, N. Z. 1984, ApJ, 276, 182
  * Sandstrom et al. (2013) Sandstrom, K. M., Leroy, A. K., Walter, F., et al. 2013, ApJ, 777, 5
  * Schaerer et al. (1997) Schaerer, D., Contini, T., Kunth, D., & Meynet, G. 1997, ApJ, 481, L75
  * Schruba et al. (2017) Schruba, A., Leroy, A. K., Kruijssen, J. M. D., et al. 2017, ApJ, 835, 278
  * Shetty et al. (2011) Shetty, R., Glover, S. C., Dullemond, C. P., & Klessen, R. S. 2011, MNRAS, 412, 1686
  * Shi et al. (2020) Shi, Y., Wang, J., Zhang, Z.-Y., et al. 2020, ApJ, 892, 147
  * Skillman et al. (2003a) Skillman, E. D., Côté, S., & Miller, B. W. 2003a, AJ, 125, 593
  * Skillman et al. (2003b) Skillman, E. D., Côté, S., & Miller, B. W. 2003b, AJ, 125, 610
  * Sternberg et al. (2014) Sternberg, A., Le Petit, F., Roueff, E., & Le Bourlot, J. 2014, ApJ, 790, 10
  * Tokuda et al. (2021) Tokuda, K., Kondo, H., Ohno, T., et al. 2021, ApJ, 922, 171
  * Tosi et al. (2001) Tosi, M., Sabbi, E., Bellazzini, M., et al. 2001, AJ, 122, 1271
  * Turner et al. (2015) Turner, J. L., Beck, S. C., Benford, D. J., et al. 2015, Nature, 519, 331
  * Turner et al. (2000) Turner, J. L., Beck, S. C., & Ho, P. T. P. 2000, ApJ, 532, L109
  * Turner et al. (1997) Turner, J. L., Beck, S. C., & Hurt, R. L. 1997, ApJ, 474, L11
  * Turner et al. (2017) Turner, J. L., Consiglio, S. M., Beck, S. C., et al. 2017, ApJ, 846, 73
  * van Dishoeck & Black (1988) van Dishoeck, E. F. & Black, J. H. 1988, ApJ, 334, 771
  * Vazdekis et al. (2016) Vazdekis, A., Koleva, M., Ricciardelli, E., Röck, B., & Falcón-Barroso, J. 2016, MNRAS, 463, 3409
  * Vázquez et al. (2004) Vázquez, G. A., Leitherer, C., Heckman, T. M., et al. 2004, ApJ, 600, 162
  * Visser et al. (2009) Visser, R., van Dishoeck, E. F., & Black, J. H. 2009, A&A, 503, 323
  * Weilbacher et al. (2020) Weilbacher, P. M., Palsa, R., Streicher, O., et al. 2020, A&A, 641, A28
  * Westmoquette et al. (2013) Westmoquette, M. S., James, B., Monreal-Ibero, A., & Walsh, J. R. 2013, A&A, 550, A88
  * Whelan et al. (2011) Whelan, D. G., Johnson, K. E., Whitney, B. A., Indebetouw, R., & Wood, K. 2011, ApJ, 729, 111
  * Wiklind & Henkel (1989) Wiklind, T. & Henkel, C. 1989, A&A, 225, 1
  * Wilson et al. (2014) Wilson, C. D., Rangwala, N., Glenn, J., et al. 2014, ApJ, 789, L36
  * Witt & Gordon (1996) Witt, A. N. & Gordon, K. D. 1996, ApJ, 463, 681
  * Witt & Gordon (2000) Witt, A. N. & Gordon, K. D. 2000, ApJ, 528, 799
  * Wolfire et al. (2010) Wolfire, M. G., Hollenbach, D., & McKee, C. F. 2010, ApJ, 716, 1191
  * Wolfire et al. (1993) Wolfire, M. G., Hollenbach, D., & Tielens, A. G. G. M. 1993, ApJ, 402, 195

##  Appendix A Details of the fits to the PACS data to derive
$\tau_{\mathrm{160}}$

The 160 $\mu$m optical depths $\tau_{\mathrm{160}}$ of the two approaches
(namely modified single-temperature blackbodies, MBB, and modified two-
temperature blackbodies, MBB-2T) are compared in Fig. 14. The scatter of the
comparison (with imposed $\gamma\,\geq\,0$) is $\sim$0.13 dex, with
$\tau_{\mathrm{160}}$(MBB-T2) larger than $\tau_{\mathrm{160}}$(MBB), as shown
by the non-unit power-law slope and positive intercept of the best (robust)
fit.

Figure 14: $\tau_{\mathrm{160}}$ from MBB-T2 fits plotted against MBB
$\tau_{\mathrm{160}}$.
$\log\,\tau_{160}(\mathrm{MBB-T2})\,=\,(1.073\,\pm\,0.010)\
\log\,\tau_{160}(\mathrm{MBB})+(0.46\,\pm\,0.05)$ The scatter of the
comparison is $\sim$0.13 dex; the shaded region denotes $\pm\,1\sigma$
relative to the best fit. In general, $\tau_{\mathrm{160}}$(MBB-T2) is larger
than $\tau_{\mathrm{160}}$(MBB), as shown by the non-unit power-law slope and
positive intercept.

The distributions of the rms (calculated as the root-mean-square difference in
$\log_{10}$ space between the model and the data) of the two fits are reported
in Fig. 15. Given that the expected uncertainty in the individual PACS
photometry is $\sim$10%, the rms values of both MBB and MBB-T2 fits are quite
good, and, except for NGC 1705, well within the expected error. However, the
MBB-T2 fits are overall superior to MBB, possibly due to the extra fitted
parameter and the consequent zero degrees of freedom. Three regions in both
NGC 625 and NGC 5253 are not included in Fig. 15, because of rms $>\,0.5$ due
to the very low 70 $\mu$m fluxes and the consequent poor fits.

Figure 15: Distributions of rms values from MBB (left) and MBB-T2 (right)
fits. Median values for each galaxy are shown by vertical dashed lines. The
rms values of the MBB-T2 fits is generally excellent, possibly due to the
extra fitted parameter and the consequent zero degrees of freedom. There are 3
regions each in NGC 625 and NGC 5253 not included here because their rms
values exceed 0.5 dex.

The overall $\gamma$ distribution (restricted to $0\,\leq\,\gamma\,\leq\,1$)
is shown in Fig. 16. There are (45, 0, 33) fits for NGC (625, 1705, 5253),
respectively, that would have preferred either $\gamma\,<\,0$ (58%, 0, 37%).
The median $\gamma\,=\,0.0$ for NGC 625 is associated with the relatively high
fraction (45/77 = 58%) of regions that would have have been best fit with
these anomalous $\gamma$ values.

Figure 16: Distribution of fitted $\gamma$ from MBB-T2 fits. The vertical
dashed lines show the median fraction of IR emission due to the warmer
component: (0.00, 0.14, 0.08) for NGC (625, 1705, 5253).