# Adaptive action supervision in reinforcement learning
from real-world multi-agent demonstrations

Keisuke Fujii1,2,3 , Kazushi Tsutsui1, Atom Scott1,
Hiroshi Nakahara1, Naoya Takeishi4,2, Yoshinobu Kawahara5,2
1Graduate School of Informatics, Nagoya University, Nagoya, Aichi, Japan.
2Center for Advanced Intelligence Project, RIKEN, Osaka, Osaka, Japan.
3PRESTO, Japan Science and Technology Agency, Tokyo, Japan
4The Graduate School of Engineering, The University of Tokyo, Tokyo, Japan
5Graduate School of Information Science and Technology, Osaka University,
Osaka, Japan
a<EMAIL_ADDRESS>
https://orcid.org/0000-0001-5487-4297

###### Abstract

Modeling of real-world biological multi-agents is a fundamental problem in
various scientific and engineering fields. Reinforcement learning (RL) is a
powerful framework to generate flexible and diverse behaviors in cyberspace;
however, when modeling real-world biological multi-agents, there is a domain
gap between behaviors in the source (i.e., real-world data) and the target
(i.e., cyberspace for RL), and the source environment parameters are usually
unknown. In this paper, we propose a method for adaptive action supervision in
RL from real-world demonstrations in multi-agent scenarios. We adopt an
approach that combines RL and supervised learning by selecting actions of
demonstrations in RL based on the minimum distance of dynamic time warping for
utilizing the information of the unknown source dynamics. This approach can be
easily applied to many existing neural network architectures and provide us
with an RL model balanced between reproducibility as imitation and
generalization ability to obtain rewards in cyberspace. In the experiments,
using chase-and-escape and football tasks with the different dynamics between
the unknown source and target environments, we show that our approach achieved
a balance between the reproducibility and the generalization ability compared
with the baselines. In particular, we used the tracking data of professional
football players as expert demonstrations in football and show successful
performances despite the larger gap between behaviors in the source and target
environments than the chase-and-escape task.

##  1 INTRODUCTION

Modeling real-world biological multi-agents is a fundamental problem in
various scientific and engineering fields. For example, animals, vehicles,
pedestrians, and athletes observe others’ states and execute their own actions
in complex situations. Pioneering works have proposed rule-based modeling
approaches such as in human pedestrians [Helbing95] and animal groups
[Couzin02] for each domain using hand-crafted functions (e.g., social forces).
Recent advances in reinforcement learning (RL) with neural network approaches
have enabled flexible and diverse modeling of such behaviors often in
cyberspace [ross2010efficient, ho2016generative].

Figure 1: Our problem setting and solution. We consider a Real-to-Sim domain
adaptation problem in which the source and target are real-world data in a
physical space and simulated data in cyberspace, respectively. We first
perform inverse RL modeling from real-world data, but we do not consider
unknown transition model $\mathcal{T}^{E}$ because we can obtain the next
state from demonstration data. We then perform forward RL with a temporal
transition model $\mathcal{T}$, but there should be a discrepancy between
$\mathcal{T}^{E}$ and $\mathcal{T}$. Since we cannot access the information of
$\mathcal{T}^{E}$, our solution comprises the supervised learning using the
observed joint action $\vec{a}$ and adapted $\vec{a}^{E}$ in cyberspace for
the learning of the Q function (see also Section 3).

However, when modeling real-world biological multi-agents, domain gaps may
occur between behaviors in the sources (real-world data) with unknown dynamics
and targets (cyberspace in RL) as shown in Fig. 1. The opposite configuration
of the source and target has been actively studied and known as Sim-to-Real
[rusu2017sim], which transfers the knowledge from cyberspace or human
demonstrations to almost known source dynamics (simulation in Sim-to-Real)
such as real-world robotics [schaal1996learning, kolter2007hierarchical]. In
contrast, domain adaptation in real-world situations where the parameters of
the source environment are often unknown cannot utilize explicit dynamics
regarding source environments (e.g. transition model). In other words, we
consider a Real-to-Sim domain adaptation problem in which the unknown source
and the target are real-world data in a physical space and simulated data in
cyberspace, respectively (Fig. 1). Team sports (e.g., football) and biological
multi-agent motions (e.g., chase-and-escape) are examples that can be
addressed with the above approach. Agents observe others’ states and execute
planned actions [fujii2021data], and particularly in team sports, most of the
governing equations are unknown. In such complex real-world behaviors, studies
have separately investigated RL in cyberspace for flexible adaptation to
complex environments [kurach2020google, li2021celebrating] and data-driven
modeling for reproducing real-world behaviors [Zheng16, Le17,
fujii2021learning]. However, given the gap between these forward and backward
approaches (see Fig. 1) in multi-agent RL (MARL) scenarios, an integrated
approach to combine both strengths will be required.

In this paper, we propose a method for adaptive action supervision in RL from
real-world multi-agent demonstrations. To utilize the information of unknown
source dynamics, we adopt an approach that combines RL and supervised learning
by selecting the action of demonstrations based on the minimum distance
between trajectories in source and target environments. Our goal is to balance
reproducibility as imitations and generalization ability to obtain rewards
(e.g., when different initial values are given in the same environment). These
are mutually independent in general, and our approach that combines RL and
supervised learning will help us achieve our goal. Compared with the case of
Sim-to-Real in robotics, there have been no simulators for real-world agents
with explicit dynamics in biological multi-agent scenarios. In our
experiments, we use simple chase-and-escape and football tasks with different
dynamics between unknown source and target environments.

In summary, our main contributions are as follows. (1) We propose a novel
method for adaptive action supervision in RL from multi-agent demonstration,
which will bridge the gap between RL in cyberspace and real-world data-driven
modeling. (2) We adopt an approach that combines RL and supervised learning by
selecting actions of demonstrations in RL based on the minimum distance of
dynamic time warping (DTW) [vintsyuk1968speech] to utilize the information of
the unknown source dynamics. This approach can be easily applied to existing
neural network architectures and provide an RL model balanced between
reproducibility as imitation and generalization ability. (3) In the
experiments, using a chase-and-escape and football tasks with the different
dynamics between the unknown source and target environments, our approach
struck a balance between the reproducibility and generalization compared with
the baselines. In particular, we used the tracking data of professional
football players as expert demonstrations in a football RL environment and
show successful performances. Our framework can estimate values for real agent
behaviors and decision making if the model can imitate behaviors of players,
which may be difficult for either data-driven and RL approaches.

In the remainder, we describe the background of our problem and our method in
Sections 2 and 3, overview related works in Section 4, and present results and
conclusions in Sections 5 and 6.

##  2 BACKGROUND

Here, we consider a sequential decision-making setting of multiple agents
interacting with a fully observable environment. We consider a forward RL
model in Fig. 1 right, defined as a tuple $(K,S,A,\mathcal{T},R,\gamma)$,
where $K$ is the fixed number of agents; $S$ is the set of states $s$;
$A=[A_{1},...,A_{K}]$ represents the set of joint action $\vec{a}\in A$ (for a
variable number of agents), and $A_{k}$ is the set of local action $a_{k}$
that agent $k$ can take; $\mathcal{T}(s^{\prime}|s,\vec{a}):S\times A\times
S\rightarrow[0,1]$ is the transition model for all agents;
$R=[R_{1},...,R_{K}]:S\times A\rightarrow\mathbb{R}^{K}$ is the joint reward
function; and $\gamma\in(0,1]$ is the discount factor. In on-policy RL, the
agent learns a policy $\pi_{k}:S_{k}\times A\rightarrow[0,1]$, where $S_{k}$
is a set of states for $k$.

The objective of agent $k$ is to discover the policy $\pi_{k}$ that maps
states to actions, thereby maximizing the expected total reward over the
agent’s lifespan, i.e., $G_{k}=\sum_{t}^{T}\gamma^{t}R_{k,t}$, where $R_{k,t}$
is the reward of agent $k$ at time $t$ and $T$ is the time horizon. The value
$Q^{\pi}_{k}(s_{k},a_{k})$ related to a specific state-action pair
$(s_{k},a_{k})$ serves as a expected future reward that can be acquired from
$(s_{k},a_{k})$ when adhering to policy $\pi_{k}$. The optimal value function
$Q^{*}(s,a)$, offering the maximal values across all states, is determined by
the Bellman equation:

$Q_{k}^{*}(s_{k},a_{k})=\mathbb{E}\left[{R_{k}(s_{k},a_{k})+\gamma\sum_{s_{k}^{\prime}}\mathcal{T}_{k}(s_{k}^{\prime}|s_{k},a_{k})\max_{a_{k}^{\prime}}Q_{k}^{*}(s_{k}^{\prime},a_{k}^{\prime})}\right],$
(1)

where $\mathcal{T}_{k}$ is the transition model of agent $k$. The optimal
policy $\pi_{k}$ is then $\pi_{k}(s_{k})=\operatorname*{arg\,max}_{a_{k}\in
A}Q_{k}^{*}(s_{k},a_{k}).$ Since our approach can be easily applied to
existing neural network in model-free RL, we consider both independent policy
for each agent and the joint policy $\vec{\pi}$ inducing the joint action-
value function
$Q_{tot}^{\vec{\pi}}(s,\vec{a})=\mathbb{E}_{s_{0:\infty},\vec{a}_{0:\infty}}\left[\sum_{t=0}^{\infty}\gamma^{t}R_{t}\mid
s_{0}=s,\vec{a}_{0}=\vec{a},\vec{\pi}\right]$, where $R_{t}$ is the value of
the joint reward at time $t$.

In a multi-agent system in complex real-world environments (e.g., team
sports), (i) transition functions are difficult to design explicitly. Instead,
(ii) if we can utilize the demonstrations of expert behaviors (e.g.,
trajectories of professional sports players), we can formulate and solve it as
a machine learning problem (e.g., learning from demonstration). In other
words, if the problem falls in the case that satisfies the two conditions, (i)
and (ii), learning from demonstration is a better option than a pure RL
approach by constructing the environment without demonstrations. As shown in
Fig. 1 right, we perform forward RL with a temporal transition model
$\mathcal{T}(s^{\prime}|s,\vec{a})$, but there should be a discrepancy between
$\mathcal{T}$ and $\mathcal{T}^{E}(s^{\prime E}|s^{E},\vec{a}^{E})$ in Fig.1
left. Since we cannot access the information of $\mathcal{T}^{E}$, our
solution is the supervised learning using the observed joint action $\vec{a}$
and adapted $\vec{a}^{E}$ in cyberspace for the learning of the Q function
(see also Section 3). Next, we introduce DQN framework according to the
previous work [hester2018deep]. For simplicity, we describe the following
explanation using a single-agent RL framework and omit the agent index $k$.

DQN leverages a deep neural network to approximate the value function $Q(s,a)$
[mnih2015human]. The network is designed to generate a set action values
$Q(s,\cdot;\theta)$ for a given state input $s$, where $\theta$ represents the
network’s parameters. DQN employs a separate target network, which is
duplicated from the main network after every $\tau$ steps to ensure more
consistent target Q-values. The agent records all of its experiences in a
replay buffer $\mathcal{D}^{replay}$, which is subsequently uniformly sampled
for network updates.

The double Q-learning updates the current network by computing the argmax over
the subsequent state values and uses the target network for action value
[van2016deep]. The loss for Double DQN (DDQN) is defined as:

$J_{DQ}(Q)=\sum^{T-1}_{t}\left(R_{t}+\gamma
Q(s_{t+1},a^{\max}_{t+1};\theta^{\prime})-Q(s_{t},a_{t};\theta)\right)^{2},$
(2)

where $\theta^{\prime}$ refers to the parameters of the target network, and
$a^{\max}_{t+1}=\operatorname*{arg\,max}_{a_{t}}Q(s_{t+1},a_{t};\theta)$. The
upward bias typically associated with regular Q-learning updates is reduced by
separating the value functions employed for these two variables. For more
efficient learning, e.g., to sample more significant transitions more
frequently from its replay buffer, prioritized experience replay
[schaul2016prioritized] have been used.

##  3 ADAPTIVE ACTION SUPERVISION IN RL

In many real-world settings of RL, we can access observation data of the
multi-agent system, but we cannot access an accurate model of the system. To
construct an alternative simulator, we want the agent to learn as much as
possible from the demonstration data. In particular, we aim to decrease the
domain gap between behaviors in the source data and the target environments.
Here, we describe our adaptive action supervision approach for RL from
demonstrations. We adopt the following three steps according to the deep
Q-learning from demonstrations (DQfD) [hester2018deep]. The first is pre-
training, which learns to imitate the demonstrator. The second is sampling
actions from the pre-trained RL model in the target RL environment. The third
is training in the RL environment. During the pre-training and training
phases, the network is updated with mainly two losses: the 1-step double
Q-learning loss in Eq. (2) and a dynamic time-warping supervised
classification loss. As mentioned above, the Q-learning loss ensures that the
network satisfies the Bellman equation and can be used as a starting point for
TD learning. For the second loss, we propose a simple supervised loss for
actions and a dynamic time-warping action assignment for efficient pre-
training and RL.

The supervised loss is crucial for pre-training because the demonstration data
usually covers a narrow part of the state space and does not take all possible
actions. Here we consider a single agent case for simplicity (i.e., we removed
the notation of agent $k$, but we can easily extend it to multi-agent cases).
The previous DQfD [hester2018deep] introduces a large margin classification
loss [piot2014boosted]:

$J_{MS}(Q)=\sum^{T}_{t}\max_{a_{t}\in
A}[Q(s_{t},a_{t})+l(a^{E}_{t},a_{t})]-Q(s_{t},a^{E}_{t}),$ (3)

where $a^{E}_{t}$ is the action the expert demonstrator takes in state $s_{t}$
and $l(a^{E}_{t},a_{t})$ is a margin function that is $0$ when
$a_{t}=a^{E}_{t}$ and positive otherwise. This loss makes the value of the
expert’s action higher than the other actions’ values, at least by the margin
$l$. This approach would be effective for learning maximum Q-function values;
however, when limited data are available, the direct approach to maximize the
Q-function values for the action of the demonstration may be efficient.
Therefore, we propose a simple supervised loss for actions represented by the
cross-entropy of softmax values of the Q-function such that

$J_{AS}(Q_{t})=-\sum^{T}_{t}\mathbf{a}^{E}_{t}\cdot\log\left(\mathrm{softmax}(\mathbf{q}_{s_{t}})\right),$
(4)

where $\mathbf{a}_{t}^{E}\in\\{0,1\\}^{|A|}$ (i.e., one-hot vector of expert
actions), $\mathbf{q}_{s_{t}}=[Q(s_{t},a_{t}=1),...,Q(s_{t},a=|A|)]$, and the
$\log$ applies element-wise. Ideally, Q-functions in the source and target
domains should be compared, but when using limited data, it would be better
that more reliable action data is used as supervised data (rather than using
approximated Q-function from data). This loss aims to achieve both
reproducibility and generalization by maximizing the Q-function values for the
action of the demonstration. A similar idea has been used [hester2018deep,
Lakshminarayanan2016], which used only similar supervised losses in pre-
training or RL, respectively, but we explicitly define and use this loss for
both pre-training and RL to balance reproducibility and generalization.

Eq. (4) and the large margin classification loss in DQfD [piot2014boosted] in
Eq. (3) assume that the timestamp of expert action $a^{E}_{t}$ should be the
same as that of the RL model $a_{t}$. However, when there is a discrepancy
between the source and target environments, the appropriate timestamp of
expert actions can vary from that of the RL model actions. Thus, we propose a
dynamic time-warping supervised loss for actions, which does not require prior
knowledge, utilizing DTW framework [vintsyuk1968speech], a well-known
algorithm in many domains [sakoe1978dynamic, myers1980performance,
tappert1990state].

Here, we first consider two state sequences in RL and demonstration:
$s=s_{1},\ldots,s_{t},\ldots,s_{n}$ and
$s^{E}=s^{E}_{1},\ldots,s^{E}_{j},\ldots,s^{E}_{m}$, where $n$ and $m$ are the
lengths of the sequences. We select $a^{E}_{t^{\prime}}$ at $t^{\prime}$
(which is not necessarily equal to $t$) for demonstration defined as:

$t^{\prime}=\operatorname*{arg\,min}_{j}W(s,s^{E})_{t,j},$ (5)

where $W(s,s^{E})\in\mathbb{R}^{n\times m}$ is a warping path matrix based on
a local distance matrix $d(s,s^{E})\in\mathbb{R}^{n\times m}$ (e.g., Euclidean
distance) and some constraints such as monotonicity, continuity, and boundary
[sakoe1978dynamic]. $W(s,s^{E})_{t,j}\in\mathbb{R}$ is the $(t,j)$ component
of $W(s,s^{E})$. Then we obtain the supervised action loss with adaptive
action supervision by modifying Eq. (4) such that

$J_{AS+DA}(Q_{t})=-\sum^{T}_{t}\mathbf{a}^{E}_{t^{\prime}}\cdot\log\left(\mathrm{softmax}(\mathbf{q}_{s_{t}})\right).$
(6)

Additionally, we introduce an $\mathcal{L}_{2}$ regularization loss that
targets the weights and biases of the network, aiming to avoid overfitting
given the small size of the demonstration dataset. The total loss used to
update the network is as follows:

$J(Q)=J_{DQ}(Q)+{\lambda_{1}}J_{AS+DA}(Q)+{\lambda_{2}}J_{\mathcal{L}_{2}}(Q).$
(7)

The $\lambda$ parameters control the weight of these losses. As an ablation
study, we examine removing some of these losses in Section 5. The behavior
policy is $\epsilon$-greedy based on the Q-values. Note that, similarly to
DQfD [hester2018deep], after the pre-training phase is finished, the agent
starts interacting with the environment, collecting its own data, and adding
it to its replay buffer $\mathcal{D}^{replay}$. The agent overwrites the
buffer when the buffer is full, but does not overwrite the demonstration data.

##  4 RELATED WORK

In RL from demonstration [schaal1996learning], the direct approach recovers
experts’ policies from demonstrations by supervised learning
[pomerleau1991efficient, ross2010efficient, Ross11] or generative adversarial
learning [ho2016generative, song2018multi], which make the learned policies
similar to the expert policies (reviewed e.g., by [ramirez2022model] and
[da2019survey, zhu2020transfer] as transfer learning). However, it is
sometimes challenging to collect high-quality (e.g., optimal) demonstrations
in many tasks. To obtain better policies from demonstrations, several methods
combine imitation learning and RL such as [silver2016mastering,
hu2018knowledge, Lakshminarayanan2016]. Some approaches
[vecerik2017leveraging, hester2018deep] have been proposed to explore the
sparse-reward environment by learning from demonstrations. Recently, cross
domain adaptation problems have been considered to achieve the desired
movements such as when changing morphologies [raychaudhuri2021cross,
fickinger2022cross]. In this case, imitation in terms of reproducibility would
be difficult in principle because the problem (e.g., morphology) is changed.
Our problem setting is different in terms of achieving both the ability to
maximize a reward and reproducibility (imitation ability) rather than only the
former.

These methods are often designed for single-agent tasks and attempt to find
better policies by exploring demonstration actions. Many MARL algorithms have
been proposed by modifying single-agent RL algorithms for a multi-agent
environment. One of the early approaches is independent learning where an
agent learns its own policy independently of the other agents
[omidshafiei2017deep, tampuu2017multiagent]. Recently, in learning from
demonstrations, for example, researchers have proposed MARL as a rehearsal for
decentralized planning [kraemer2016multi], MARL augmented by mixing
demonstrations from a centralized policy [lee2019improved] with sub-optimal
demonstrations, and centralized learning and decentralized execution
[peng2021hybrid]. Other researchers have proposed imitation learning from
observations under transition model disparity [gangwani2021imitation] between
the dynamics of the expert and the learner by changing different configuration
parameters in cyberspace. However, domain adaptation in RL to cyberspace from
real-world multi-agent demonstrations has been rarely investigated.

In RL applications, grid-world, robot Soccer, video games, and robotics have
been intensively investigated. Among these domains, robotics and robot soccer
are specifically related to real-world problems. In robotics, noise in sensors
and actuators, limited computational resources, and the harmfulness of random
exploration to people or animals are some of the many challenges
[hua2021learning]. There have been successful applications of transfer
learning in robotics [schaal1996learning, kolter2007hierarchical,
sakato2014learning] (reviewed by e.g., [zhu2021survey]).These are mostly
transferred from cyberspace or human demonstrations to real-world robotics
(sometimes called Sim-to-Real [rusu2017sim]), which utilize almost known
dynamics about the (at least) target dynamics. In contrast, our Real-to-Sim
problem cannot utilize explicit dynamics about both source and target
environments, and thus such domain adaptation in RL is challenging.

Robot soccer is similar to our task, in particular, RoboCup (the Robot World
Cup Initiative) involves attempts by robot teams to actually play a soccer
game [Kitano97]. Some researchers have adopted imitation learning approaches
[hussein2018deep, nguyen2020structure], but the source and target environments
are basically the same. In terms of simulators based on real-world data for
data analysis, to our knowledge, there have been no domain adaptation methods
in RL from real-world data to simulation environments.

In the tactical behaviors of team sports, agents select an action that follows
a policy (or strategy) in a state, receives a reward from the environment and
others, and updates the state [fujii2021data]. Due to the difficulty in
modeling the entire framework from data for various reasons [van2021learning],
we can adopt two approaches: to estimate the related variables and functions
from data (i.e., inverse approach) as a sub-problem, and to build a model
(e.g., RL) to generate data in cyberspace (i.e., forward approach, e.g.,
[kurach2020google, li2021celebrating]).

For the former, there have been many studies on inverse approaches. There have
been many studies on estimating reward functions by inverse RL [Luo2020,
Rahimian2020] and the state-action value function (Q-function) [Liu2018,
Liu2020, ding2022deep, nakahara2023action]. Researchers have performed
trajectory prediction in terms of the policy function estimation, as imitation
learning [Le17, Teranishi2020, fujii2020policy] and behavioral modeling
[Zheng16, Zhan19, Yeh2019, fujii2022estimating, teranishi2022evaluation] to
mimic (not optimize) the policy using neural network approaches. This approach
did not consider the reward in RL (and simulation) and usually performed a
trajectory prediction.

For the latter approach, researchers have proposed new MARL algorithms with
efficient learning, computation, and communication [roy2020promoting,
espeholt2019seed, liu2021semantic, li2021celebrating]. Recently, the ball-
passing behaviors in artificial agents of Google Research Football (GFootball)
[kurach2020google] and professional football players were compared
[scott2022does], but a gap still exists between these forward and backward
approaches. In other research fields, e.g., for animal behavioral analysis,
forward [banino2018vector, ishiwaka2022deepfoids] and backward approaches
[ashwood2022dynamic, fujii2021learning] have also been used separately. Our
approach integrates both approaches to combine the reproducibility as
imitation and generalization to obtain rewards.

Figure 2: Our RL problem setting. (a) the predators and prey are represented
as gray and black disks, respectively. (b) and (c) are football 2vs2 and 4vs8
tasks, respectively. Yellow circle, yellow line, gray and black circles are
ball, goal line, attackers, and defenders, respectively. The play areas are
represented by a black square/rectangle surrounding them.

##  5 EXPERIMENTS

The purpose of our experiments is to validate the proposed methods for
application to real-world multi-agent modeling, which usually has no explicit
equations in a source environment. Hence, for verification of our methods, we
first examined a simple but different simulation environment from the
demonstration: a predator-prey cooperative and competitive interaction, namely
a chase-and-escape task. Next, we investigated a football environment with the
demonstrations of real-world football players. We considered a 2vs2 task for a
simple extension of the 2vs1 chase-and-escape task and then a 4vs8 task (4
attackers) for more realistic situations. We basically considered
decentralized multi-agent models, which do not communicate with each other
(i.e., without central control) for simplicity, but in the football 4vs8 task,
we also examined centralized models.

Here, we commonly compared our full model DQAAS (deep q-learning with adaptive
action supervision) to four baseline methods: a simple DQN with DDQN and
prioritized experience replay introduced in Section 2 (without demonstration),
DQfD [hester2018deep], DQAS (deep q-learning with action supervision), and
DQfAD (DQfD with adaptive demonstration using DTW). Using these baselines with
the same network architectures for fair comparisons, we investigated the
effect of adaptive action supervision. Note that our approach was also used in
the pre-trained phase in all tasks. We hypothesized that our approach would
find a balance between imitation reproducibility and generalization compared
to the baselines. In addition, only for the football 4vs8 task requiring more
agent interaction, we examined CDS [li2021celebrating], which is a recent
centralized MARL method in GFootball, as a base model. That is, by replacing
it with the above DQN, we also investigated the effectiveness of our approach
as a centralized MARL method. Our evaluation metrics in the test phase were
twofold: one is the DTW distance between the RL model and demonstration
trajectories representing imitation reproducibility, and another is the
obtained reward by RL agents. We used well-known DTW distance here because it
would be easy to verify whether the learning of our model is successful or
not. During the test phase, $\epsilon$ in $\epsilon$-greedy exploration was
set to 0 and each agent was made to take greedy actions. With 5 different
random seeds, we evaluated the mean and standard error of the performances. We
used different initial settings for the test. In the source environment, we
did not use the RL environment and just pre-trained the model from
demonstration data. It should be noted that our purpose is not to develop a
state-of-the-art MARL algorithms and the strengths of our approach are to
enable us to apply it to many existing methods and to provide us with an RL
model striking a balance between reproducibility as imitation and
generalization.

Figure 3: Example RL results of the baseline (DQfD [hester2018deep], center)
and our approach (DQAAS, right), and the demonstration in the source domain
(left) in the 2vs1 chase-and-escape task. Histograms are the Q-function values
for each action. There are 13 actions including acceleration in 12 directions
every 30 degrees in the relative coordinate system (action 0 means moving
towards the prey) and doing nothing (action 12: round point).

###  5.1 Performance on Chase-and-escape

First, we verified our method using a chase-and-escape task, in which the
predators and prey interacted in a two-dimensional world with continuous space
and discrete time. The numbers of predators and prey were 2 and 1,
respectively. We first describe the common setting between the source
(demonstration) and target RL tasks. The environment was constructed by
modifying an environment called the predator-prey in Multi-Agent Particle
Environment (MAPE) [lowe2017multi, tsutsui2022emergence,
tsutsui2022collaborative]. Following [tsutsui2022collaborative], the play area
size was constrained to the range of -1 to 1 on the $x$ and $y$ axes, all
agent (predator/prey) disk diameters were set to 0.1, obstacles were
eliminated, and predator-to-predator contact was ignored for simplicity. The
predators were rewarded for capturing the prey ($+1$), namely contacting the
disks, and punished for moving out of the area ($-1$), and the prey was
punished for being captured by the predator or for moving out of the area
($-1$).

Fig. 2a shows an example of the chase-and-escape task. The time step was 0.1 s
and the time limit in each episode was set to 30 s. The initial position of
each episode was randomly selected from a range of -0.5 to 0.5 on the $x$ and
$y$ axes. If the predator captured the prey within the time limit, the
predator was successful; otherwise, the prey was successful. If one side
(predators/prey) moved out of the area, the other side (prey/predators) was
deemed successful. There are 13 actions including acceleration in 12
directions at every 30 degrees in relative coordinate system and doing
nothing. For the relative mobility of predators and prey in both environments,
to examine the effect of domain adaptation, we set the same mobility of the
prey in the source and target RL environments, but for the predators, we set
120% and 110% of the prey mobility in the source and target RL environments,
respectively. The predators did not share the rewards for simplicity.

Before using RL algorithm, we first created the demonstration dataset using
the DQN (without data). After learning 10 million episodes based on the
previous setting [tsutsui2022emergence], we obtained 500 episodes for
demonstration with randomized initial conditions (locations). We split the
datasets into 400, 50, and 50 for training, validation, and testing during
pre-training, respectively. Then we pre-trained and trained the models
according to Section 3. To examine the learning performance of the predator
movements with the fixed prey movements, we performed RL of all predators with
the learned (and fixed) prey. For the target RL, we used 50 train and 10 test
episodes from the above 100 episodes for pre-train validation and testing
(i.e., we did not use the test condition in the target RL during the pre-
training and training phases).

The model performance was evaluated by computational simulation of the 10 test
episodes as the test phase using the trained models. The termination
conditions in each episode were the same as in training. We calculated and
analyzed the proportion of successful predation and DTW distance between the
source and target trajectories in the test phase.

Figure 4: Example RL results of the baseline (DQN, center) and our approach
(DQAAS, right), and the demonstration (left) in the 2vs2 football task. In the
demonstration and DQAAS, the agents obtained the goal, but the DQN failed.
Configurations are the same as Fig. 3. There are 12 actions including the
movement in 8 directions (with constant velocity) every 45 degrees in the
relative coordinate system (actions 0-7 and action 0 means moving toward the
center of the goal), doing nothing (action 8: round point), and high pass
(action 9: $p_{h}$), short pass (action 10: $p_{s}$), and shot (action 11:
$s$), which are partially based on GFootball [kurach2020google].

| Reward | DTW distance
---|---|---
| pre-trained | 0.5M steps | pre-trained | 0.5M steps
DQN | 0.04 $\pm$ 0.06 | 0.11 $\pm$ 0.03 |  4.12 $\pm$ 0.73 | 5.70 $\pm$ 0.62
DQfD | 0.00 $\pm$ 0.00 | 0.04 $\pm$ 0.03 | 5.02 $\pm$ 0.34 | 4.94 $\pm$ 0.23
DQfAD | 0.00 $\pm$ 0.00 | 0.06 $\pm$ 0.03 | 5.02 $\pm$ 0.34 | 4.80 $\pm$ 0.57
DQAS |  0.25 $\pm$ 0.08 |  0.26 $\pm$ 0.08 | 5.37 $\pm$ 0.40 | 4.97 $\pm$ 1.16
DQAAS |  0.25 $\pm$ 0.08 |  0.29 $\pm$ 0.09 | 5.37 $\pm$ 0.40 |  4.73 $\pm$ 1.07
|  |  |  |

Table 1:  Performance on 2vs1 chase-and-escape task.

We then show the proportion of successful predation and DTW distance between
the source and target trajectories for each model in Table 1. The results show
that our approaches (DQAAS) achieved better performances for both indicators
than baselines. The obtained rewards and DTW distance had a trade-off
relationship. In general, how to strike a balance is not obvious. In this
task, with increased training steps, the DQAAS first learned the ability to
maximize a reward and then learned the reproducibility at the expense of the
reward.

Here we show example results of the baseline (DQfD [hester2018deep]) and our
approach (DQAAS) in Fig 3. The demonstration (left) shows that two predators
chased the prey almost linearly and caught the prey in this scenario. In the
source domain (demonstration), the predators were much faster than the prey
(120 %), but in the target domain (RL), the predators were only slightly
faster than the prey (110 %). Then, the task becomes more challenging than in
the source domain and learning the Q-function correctly becomes more important
to catch prey. Compared with the baseline, our approach correctly learned
Q-function values in which the distribution concentrated near action 0 (here
action 0 means moving toward the prey). These results imply the effectiveness
of our approach quantitatively and qualitatively.

###  5.2 Performance on Football Tasks

Next, we used real-world demonstrations of football players and verified our
method. We created an original football environment (called NFootball) in our
provided code because in a recent popular environment (GFootball
[kurach2020google]) the transition algorithms are difficult to customize and
some commands (e.g., pass) did not work well within our intended timings.
NFootball has a simple football environment and all algorithms are written in
Python and then transparent. Similarly to GFootball and MAPE environments,
players interacted in a two-dimensional world with continuous space and
discrete time. The play pitch size was the same as GFootball
[kurach2020google]: a range of -1 to 1 and -0.42 to 0.42 on the $x$ and $y$
axes, respectively, and the goal on the $y$ axis was in the range of -0.044 to
0.044.

Figure 5: Example RL results of the baseline (DQN, center) and our approach
(DQAAS, right), and the demonstration (left) in the 4vs8 football task.
Configurations are similar to Fig. 4 (in this task, 4 attackers are learned).
In the demonstration and DQAAS, the agents obtained the goal, but the DQN
failed. The action space is the same as the 2vs2 football task.

Figs. 2b and c show examples of two football tasks: 2vs2 and 4vs8,
respectively. The initial position of each episode was selected as the last
passer’s possession in the goal scenes based on real-world data as explained
below. The time step was 0.1 s and the time limit in each episode was set to
8.5 s (based on the maximum time length of the real-world data with a margin).
All attackers and defenders were rewarded and punished for goal ($+10$) and
concede ($-10$), respectively. Also, all defenders and attackers were rewarded
and punished for ball gain ($+1$) and lost ($-1$), respectively. To complete
matches, each player is punished for moving out of the pitch ($-5$). If any
reward or punishment is obtained, the episode is finished. We consider the
2vs2 task for a simple extension of the 2vs1 chase-and-escape task and the
4vs8 task (4 attackers) for more realistic situations. Note that currently,
the learning of 11vs11 is difficult and time-consuming, and thus we limited
the scenarios. To examine the learning performances of the attacking movements
with the fixed defenders’ movements, we first performed the RL of all players
with the DQAAS algorithm, and then we performed RL of all attackers with the
learned (and fixed) defenders. The mobilities of the attackers and defenders
are the same. There are 12 actions including the movement in 8 directions
(with constant velocity) every 45 degrees in the relative coordinate system,
doing nothing, and high pass, short pass, and shot, which are partially based
on GFootball [kurach2020google].

Before using RL algorithm, we first created the demonstration dataset using
real-world player location data in professional soccer league games. We used
the data of 54 games in the Meiji J1 League 2019 season held in Japan. The
dataset includes event data (i.e., labels of actions, e.g., passing and
shooting, recorded at 30 Hz and the xy coordinates of the ball) and tracking
data (i.e., xy coordinates of all players recorded at 25 Hz) provided by Data
Stadium Inc. We extracted 198 last-pass-and-goal sequences and 1,385 last-pass
sequences (including a ball lost) for training and pre-training of the RL
model from the dataset. In pre-training, we split the dataset into 1,121
training, 125 validation, and 139 test sequences (or episodes). We set shot
rewards ($+1$) for the attacker in addition to the above rewards and
punishments (but we did not use the out-of-pitch punishment) because the goal
reward was sparse and limited. In the target RL, we used 16 train and 5 test
episodes from the above 198 episodes (we did not use the test condition in the
target RL during the pre-training and training phases). We calculated and
analyzed the obtained reward and DTW distance between the source and target
trajectories in the test phase.

| Reward | DTW distance
---|---|---
| pre-trained | 0.5M steps | pre-trained | 0.5M steps
DQN | 0.00 $\pm$ 0.00 | 1.40 $\pm$ 1.16 | 3.15 $\pm$ 0.47 | 2.69 $\pm$ 0.53
DQfD | 0.00 $\pm$ 0.00 | 0.00 $\pm$ 0.00 | 4.58 $\pm$ 0.00 | 5.18 $\pm$ 0.00
DQfAD | 0.00 $\pm$ 0.00 | 0.00 $\pm$ 0.00 | 4.58 $\pm$ 0.00 | 5.29 $\pm$ 0.01
DQAS |  8.00 $\pm$ 0.00 |  8.00 $\pm$ 0.00 |  2.25 $\pm$ 0.00 |  2.25 $\pm$ 0.00
DQAAS |  8.00 $\pm$ 0.00 |  8.00 $\pm$ 0.00 |  2.25 $\pm$ 0.00 |  2.25 $\pm$ 0.00
|  |  |  |

Table 2:  Performance on 2vs2 football task.

Next, we show the quantitative and qualitative performances in the 2vs2 and
4vs8 tasks in this order. First, we show the average return and the DTW
distance between the source and target trajectories for each model of the 2vs2
task in Table 2. The results show that our approaches (DQAS and DQAAS)
achieved better performances for both indicators than baselines with
demonstrations (DQfD and DQfAD). However, the two performance indicators in
the learning models from demonstrations (except for DQN) did not change
according to the learning steps. It suggests that the pre-trained models from
demonstrations obtained other local solutions and may be struggle to obtain
better solutions (in particular, in terms of reproducibility). For example, in
Fig 4, we show the demonstration, example results of the baseline without
demonstration (DQN), and our approach (DQAAS). The demonstration (left) shows
that the attacker #1 passed the ball to the attacker #0 during moving toward
the goal. However, in our approach (right), agents learned movements simply to
pass the ball and shoot without moving toward the goal. In contrast, the model
without demonstration (center) learned moving toward the goal without passing
and shooting the ball. Ideally, combining both generalization and
reproducibility will be expected but the domain-specific modeling and reality
of the simulator is left for future work in this task. In terms of the
Q-learning, as shown in Fig. 4, the agents obtained the goal in the
demonstration (left) and DQAAS (right), but the DQN (center) failed. Although
our approach did not reproduce the demonstration movements toward the goal,
compared with DQN, our approach correctly learned Q-function values in which
the higher values were observed in actions 10 and 11 for the passer (agent 1)
and shooter (agent 0), respectively.

| Reward | DTW distance
---|---|---
| pre-trained | 0.5M steps | pre-trained | 0.5M steps
DQN | 0.00 $\pm$ 0.00 | 0.16 $\pm$ 0.13 |  3.22 $\pm$ 0.22 |  3.24 $\pm$ 0.22
CDS | 0.12 $\pm$ 0.24 | 0.12 $\pm$ 0.24 |  3.10 $\pm$ 0.07 |  3.25 $\pm$ 0.12
DQfD | 0.00 $\pm$ 0.00 | 0.00 $\pm$ 0.00 | 3.35 $\pm$ 0.00 | 3.78 $\pm$ 0.00
DQfAD | 0.00 $\pm$ 0.00 | 0.00 $\pm$ 0.00 | 3.35 $\pm$ 0.00 | 4.28 $\pm$ 0.00
CDS+fD | 0.00 $\pm$ 0.00 | 0.00 $\pm$ 0.00 | 3.71 $\pm$ 0.00 | 3.76 $\pm$ 0.00
CDS+fAD | 0.00 $\pm$ 0.00 | 0.00 $\pm$ 0.00 | 3.71 $\pm$ 0.00 | 3.76 $\pm$ 0.00
DQAS | 0.00 $\pm$ 0.00 |  6.00 $\pm$ 0.00 | 4.54 $\pm$ 0.00 |  3.27 $\pm$ 0.00
DQAAS | 0.00 $\pm$ 0.00 |  6.00 $\pm$ 0.00 | 4.54 $\pm$ 0.00 |  3.30 $\pm$ 0.00
CDS+AS |  6.00 $\pm$ 0.00 |  6.00 $\pm$ 0.00 |  3.25 $\pm$ 0.00 |  3.30 $\pm$ 0.00
CDS+AAS |  6.00 $\pm$ 0.00 |  6.00 $\pm$ 0.00 |  3.25 $\pm$ 0.00 |  3.30 $\pm$ 0.00
|  |  |  |

Table 3:  Performance on 4vs8 football task.

Next, we show the results of the 4vs8 football task. The quantitative results
in Tables 3 in DQN-based RL models show that our approaches (DQAS and DQAAS)
achieved better performances for both indicators than baselines with
demonstrations (DQfD and DQAAS). These results and discussions were similar to
those in the 2vs2 task shown in Table 2. In addition, we examined the
centralized learning approach using CDS [li2021celebrating]. These results
shown in Table 3 in CDS-based RL models were very similar to those in Table 3
in DQN-based RL models. We confirmed that the cause of the reproducibility
issue may not be the centralized/decentralized or classic/recent deep RL. More
task-specific modeling using domain knowledge [zare2021improving,
nguyen2020structure] can be a possible solution, which is left for future
work. In terms of Q-learning (Fig. 5), compared with DQN (center), our
approach (rights) correctly learned Q-function values for actions 10 and 11
for the passer and shooter, which were similar results to those in Fig. 4. If
the model can imitate behaviors of players in the real-world football, we can
estimate values for their behaviors and decision making using estimated
Q-function values, which may be difficult for either data-driven and RL
approaches.

##  6 CONCLUSION

We proposed a novel method for domain adaptation in RL from real-world multi-
agent demonstration, which will bridge the gap between RL in cyberspace and
data-driven modeling. In the experiments, using chase-and-escape and football
tasks with the different dynamics between the unknown source and target
environments, we showed that our approach balanced between the reproducibility
and generalization more effectively than the baselines. In particular, we used
the tracking data of professional football players as expert demonstrations in
a football RL environment and demonstrated successful performances in both
despite the larger gap between behaviors in the source and target environments
than in the chase-and-escape task. Possible future research directions are to
create a better multi-agent simulator and RL model utilizing domain knowledge
for reproducing not only actions but also movements such as used by
[tsutsui2023synergizing]. In another direction, although modeling football
movements would be currently challenging, for example, application to multi-
animal behaviors will provide more scientifically valuable insights.

## ACKNOWLEDGEMENTS

This work was supported by JSPS KAKENHI (Grant Numbers 21H04892, 21H05300 and
23H03282) and JST PRESTO (JPMJPR20CA).

## REFERENCES

  * Ashwood et al., 2022 Ashwood, Z., Jha, A., and Pillow, J. W. (2022).  Dynamic inverse reinforcement learning for characterizing animal behavior.  Advances in Neural Information Processing Systems, 35.
  * Banino et al., 2018 Banino, A., Barry, C., Uria, B., Blundell, C., Lillicrap, T., Mirowski, P., Pritzel, A., Chadwick, M. J., Degris, T., Modayil, J., et al. (2018).  Vector-based navigation using grid-like representations in artificial agents.  Nature, 557(7705):429–433.
  * Couzin et al., 2002 Couzin, I. D., Krause, J., James, R., Ruxton, G. D., and Franks, N. R. (2002).  Collective memory and spatial sorting in animal groups.  Journal of Theoretical Biology, 218(1):1–11.
  * Da Silva and Costa, 2019 Da Silva, F. L. and Costa, A. H. R. (2019).  A survey on transfer learning for multiagent reinforcement learning systems.  Journal of Artificial Intelligence Research, 64:645–703.
  * Ding et al., 2022 Ding, N., Takeda, K., and Fujii, K. (2022).  Deep reinforcement learning in a racket sport for player evaluation with technical and tactical contexts.  IEEE Access, 10:54764–54772.
  * Espeholt et al., 2019 Espeholt, L., Marinier, R., Stanczyk, P., Wang, K., and Michalski, M. (2019).  Seed rl: Scalable and efficient deep-rl with accelerated central inference.  In International Conference on Learning Representations.
  * Fickinger et al., 2022 Fickinger, A., Cohen, S., Russell, S., and Amos, B. (2022).  Cross-domain imitation learning via optimal transport.  In International Conference on Learning Representations.
  * Fujii, 2021 Fujii, K. (2021).  Data-driven analysis for understanding team sports behaviors.  Journal of Robotics and Mechatronics, 33(3):505–514.
  * Fujii et al., 2020 Fujii, K., Takeishi, N., Kawahara, Y., and Takeda, K. (2020).  Policy learning with partial observation and mechanical constraints for multi-person modeling.  arXiv preprint arXiv:2007.03155.
  * Fujii et al., 2021 Fujii, K., Takeishi, N., Tsutsui, K., Fujioka, E., Nishiumi, N., Tanaka, R., Fukushiro, M., Ide, K., Kohno, H., Yoda, K., Takahashi, S., Hiryu, S., and Kawahara, Y. (2021).  Learning interaction rules from multi-animal trajectories via augmented behavioral models.  In Advances in Neural Information Processing Systems 34, pages 11108–11122.
  * Fujii et al., 2022 Fujii, K., Takeuchi, K., Kuribayashi, A., Takeishi, N., Kawahara, Y., and Takeda, K. (2022).  Estimating counterfactual treatment outcomes over time in complex multi-agent scenarios.  arXiv preprint arXiv:2206.01900.
  * Gangwani et al., 2022 Gangwani, T., Zhou, Y., and Peng, J. (2022).  Imitation learning from observations under transition model disparity.  In International Conference on Learning Representations.
  * Helbing and Molnar, 1995 Helbing, D. and Molnar, P. (1995).  Social force model for pedestrian dynamics.  Physical Review E, 51(5):4282.
  * Hester et al., 2018 Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Osband, I., et al. (2018).  Deep q-learning from demonstrations.  In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference, pages 3223–3230.
  * Ho and Ermon, 2016 Ho, J. and Ermon, S. (2016).  Generative adversarial imitation learning.  In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 4572–4580.
  * Hu et al., 2018 Hu, Y., Li, J., Li, X., Pan, G., and Xu, M. (2018).  Knowledge-guided agent-tactic-aware learning for starcraft micromanagement.  In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 1471–1477.
  * Hua et al., 2021 Hua, J., Zeng, L., Li, G., and Ju, Z. (2021).  Learning for a robot: Deep reinforcement learning, imitation learning, transfer learning.  Sensors, 21(4):1278.
  * Hussein et al., 2018 Hussein, A., Elyan, E., and Jayne, C. (2018).  Deep imitation learning with memory for robocup soccer simulation.  In International Conference on Engineering Applications of Neural Networks, pages 31–43. Springer.
  * Ishiwaka et al., 2022 Ishiwaka, Y., Zeng, X. S., Ogawa, S., Westwater, D. M., Tone, T., and Nakada, M. (2022).  Deepfoids: Adaptive bio-inspired fish simulation with deep reinforcement learning.  Advances in Neural Information Processing Systems, 35.
  * Kitano et al., 1997 Kitano, H., Asada, M., Kuniyoshi, Y., Noda, I., and Osawa, E. (1997).  Robocup: The robot world cup initiative.  In Proceedings of the First International Conference on Autonomous Agents, pages 340–347.
  * Kolter et al., 2007 Kolter, J., Abbeel, P., and Ng, A. (2007).  Hierarchical apprenticeship learning with application to quadruped locomotion.  Advances in Neural Information Processing Systems, 20.
  * Kraemer and Banerjee, 2016 Kraemer, L. and Banerjee, B. (2016).  Multi-agent reinforcement learning as a rehearsal for decentralized planning.  Neurocomputing, 190:82–94.
  * Kurach et al., 2020 Kurach, K., Raichuk, A., Stańczyk, P., Zając, M., Bachem, O., Espeholt, L., Riquelme, C., Vincent, D., Michalski, M., Bousquet, O., et al. (2020).  Google research football: A novel reinforcement learning environment.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4501–4510.
  * Lakshminarayanan et al., 2016 Lakshminarayanan, A. S., Ozair, S., and Bengio, Y. (2016).  Reinforcement learning with few expert demonstrations.  In NIPS Workshop on Deep Learning for Action and Interaction.
  * Le et al., 2017 Le, H. M., Yue, Y., Carr, P., and Lucey, P. (2017).  Coordinated multi-agent imitation learning.  In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1995–2003. JMLR. org.
  * Lee and Lee, 2019 Lee, H.-R. and Lee, T. (2019).  Improved cooperative multi-agent reinforcement learning algorithm augmented by mixing demonstrations from centralized policy.  In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 1089–1098.
  * Li et al., 2021 Li, C., Wang, T., Wu, C., Zhao, Q., Yang, J., and Zhang, C. (2021).  Celebrating diversity in shared multi-agent reinforcement learning.  Advances in Neural Information Processing Systems, 34:3991–4002.
  * Liu et al., 2020 Liu, G., Luo, Y., Schulte, O., and Kharrat, T. (2020).  Deep soccer analytics: learning an action-value function for evaluating soccer players.  Data Mining and Knowledge Discovery, 34(5):1531–1559.
  * Liu and Schulte, 2018 Liu, G. and Schulte, O. (2018).  Deep reinforcement learning in ice hockey for context-aware player evaluation.  arXiv preprint arXiv:1805.11088.
  * Liu et al., 2021 Liu, I.-J., Ren, Z., Yeh, R. A., and Schwing, A. G. (2021).  Semantic tracklets: An object-centric representation for visual multi-agent reinforcement learning.  In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5603–5610. IEEE.
  * Lowe et al., 2017 Lowe, R., Wu, Y. I., Tamar, A., Harb, J., Pieter Abbeel, O., and Mordatch, I. (2017).  Multi-agent actor-critic for mixed cooperative-competitive environments.  Advances in Neural Information Processing Systems, 30:6382–6393.
  * Luo et al., 2020 Luo, Y., Schulte, O., and Poupart, P. (2020).  Inverse reinforcement learning for team sports: Valuing actions and players.  In Bessiere, C., editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 3356–3363. International Joint Conferences on Artificial Intelligence Organization.
  * Mnih et al., 2015 Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015).  Human-level control through deep reinforcement learning.  Nature, 518(7540):529–533.
  * Myers et al., 1980 Myers, C., Rabiner, L., and Rosenberg, A. (1980).  Performance tradeoffs in dynamic time warping algorithms for isolated word recognition.  IEEE Transactions on Acoustics, Speech, and Signal Processing, 28(6):623–635.
  * Nakahara et al., 2023 Nakahara, H., Tsutsui, K., Takeda, K., and Fujii, K. (2023).  Action valuation of on-and off-ball soccer players based on multi-agent deep reinforcement learning.  IEEE Access, 11:131237–131244.
  * Nguyen and Prokopenko, 2020 Nguyen, Q. D. and Prokopenko, M. (2020).  Structure-preserving imitation learning with delayed reward: An evaluation within the robocup soccer 2d simulation environment.  Frontiers in Robotics and AI, 7:123.
  * Omidshafiei et al., 2017 Omidshafiei, S., Pazis, J., Amato, C., How, J. P., and Vian, J. (2017).  Deep decentralized multi-task multi-agent reinforcement learning under partial observability.  In International Conference on Machine Learning, pages 2681–2690. PMLR.
  * Peng et al., 2021 Peng, P., Xing, J., and Cao, L. (2021).  Hybrid learning for multi-agent cooperation with sub-optimal demonstrations.  In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 3037–3043.
  * Piot et al., 2014 Piot, B., Geist, M., and Pietquin, O. (2014).  Boosted bellman residual minimization handling expert demonstrations.  In Joint European Conference on machine learning and knowledge discovery in databases, pages 549–564. Springer.
  * Pomerleau, 1991 Pomerleau, D. A. (1991).  Efficient training of artificial neural networks for autonomous navigation.  Neural Computation, 3(1):88–97.
  * Rahimian and Toka, 2020 Rahimian, P. and Toka, L. (2020).  Inferring the strategy of offensive and defensive play in soccer with inverse reinforcement learning.  In Machine Learning and Data Mining for Sports Analytics (MLSA 2018) in ECML-PKDD Workshop.
  * Ramírez et al., 2022 Ramírez, J., Yu, W., and Perrusquía, A. (2022).  Model-free reinforcement learning from expert demonstrations: a survey.  Artificial Intelligence Review, 55(4):3213–3241.
  * Raychaudhuri et al., 2021 Raychaudhuri, D. S., Paul, S., Vanbaar, J., and Roy-Chowdhury, A. K. (2021).  Cross-domain imitation from observations.  In International Conference on Machine Learning, pages 8902–8912. PMLR.
  * Ross and Bagnell, 2010 Ross, S. and Bagnell, D. (2010).  Efficient reductions for imitation learning.  In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 661–668. JMLR Workshop and Conference Proceedings.
  * Ross et al., 2011 Ross, S., Gordon, G., and Bagnell, D. (2011).  A reduction of imitation learning and structured prediction to no-regret online learning.  In Proceedings of the fourteenth International Conference on Artificial Intelligence and Statistics, pages 627–635.
  * Roy et al., 2020 Roy, J., Barde, P., Harvey, F., Nowrouzezahrai, D., and Pal, C. (2020).  Promoting coordination through policy regularization in multi-agent deep reinforcement learning.  Advances in Neural Information Processing Systems, 33:15774–15785.
  * Rusu et al., 2017 Rusu, A. A., Večerík, M., Rothörl, T., Heess, N., Pascanu, R., and Hadsell, R. (2017).  Sim-to-real robot learning from pixels with progressive nets.  In Conference on Robot Learning, pages 262–270. PMLR.
  * Sakato et al., 2014 Sakato, T., Ozeki, M., and Oka, N. (2014).  Learning through imitation and reinforcement learning: Toward the acquisition of painting motions.  In 2014 IIAI 3rd International Conference on Advanced Applied Informatics, pages 873–880. IEEE.
  * Sakoe and Chiba, 1978 Sakoe, H. and Chiba, S. (1978).  Dynamic programming algorithm optimization for spoken word recognition.  IEEE transactions on acoustics, speech, and signal processing, 26(1):43–49.
  * Schaal, 1996 Schaal, S. (1996).  Learning from demonstration.  Advances in Neural Information Processing Systems, 9:1040–1046.
  * Schaul et al., 2016 Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2016).  Prioritized experience replay.  In International Conference on Learning Representations.
  * Scott et al., 2022 Scott, A., Fujii, K., and Onishi, M. (2022).  How does AI play football? An analysis of RL and real-world football strategies.  In 14th International Conference on Agents and Artificial Intelligence (ICAART’ 22), volume 1, pages 42–52.
  * Silver et al., 2016 Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016).  Mastering the game of go with deep neural networks and tree search.  nature, 529(7587):484–489.
  * Song et al., 2018 Song, J., Ren, H., Sadigh, D., and Ermon, S. (2018).  Multi-agent generative adversarial imitation learning.  In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 7472–7483.
  * Tampuu et al., 2017 Tampuu, A., Matiisen, T., Kodelja, D., Kuzovkin, I., Korjus, K., Aru, J., Aru, J., and Vicente, R. (2017).  Multiagent cooperation and competition with deep reinforcement learning.  PloS one, 12(4):e0172395.
  * Tappert et al., 1990 Tappert, C. C., Suen, C. Y., and Wakahara, T. (1990).  The state of the art in online handwriting recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(8):787–808.
  * Teranishi et al., 2020 Teranishi, M., Fujii, K., and Takeda, K. (2020).  Trajectory prediction with imitation learning reflecting defensive evaluation in team sports.  In 2020 IEEE 9th Global Conference on Consumer Electronics (GCCE), pages 124–125. IEEE.
  * Teranishi et al., 2022 Teranishi, M., Tsutsui, K., Takeda, K., and Fujii, K. (2022).  Evaluation of creating scoring opportunities for teammates in soccer via trajectory prediction.  In International Workshop on Machine Learning and Data Mining for Sports Analytics. Springer.
  * Tsutsui et al., 2023 Tsutsui, K., Takeda, K., and Fujii, K. (2023).  Synergizing deep reinforcement learning and biological pursuit behavioral rule for robust and interpretable navigation.  In 1st Workshop on the Synergy of Scientific and Machine Learning Modeling in International Conference on Machine Learning.
  * Tsutsui et al., 2022a Tsutsui, K., Tanaka, R., Takeda, K., and Fujii, K. (2022a).  Collaborative hunting in artificial agents with deep reinforcement learning.  bioRxiv.
  * Tsutsui et al., 2022b Tsutsui, K., Tanaka, R., Takeda, K., and Fujii, K. (2022b).  Emergence of collaborative hunting via multi-agent deep reinforcement learning.  In ICPR Workshop on Human Behavior Understanding. Springer.
  * Van Hasselt et al., 2016 Van Hasselt, H., Guez, A., and Silver, D. (2016).  Deep reinforcement learning with double q-learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30.
  * Van Roy et al., 2021 Van Roy, M., Robberechts, P., Yang, W.-C., De Raedt, L., and Davis, J. (2021).  Learning a markov model for evaluating soccer decision making.  In Reinforcement Learning for Real Life (RL4RealLife) Workshop at ICML 2021.
  * Vecerik et al., 2017 Vecerik, M., Hester, T., Scholz, J., Wang, F., Pietquin, O., Piot, B., Heess, N., Rothörl, T., Lampe, T., and Riedmiller, M. (2017).  Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards.  arXiv preprint arXiv:1707.08817.
  * Vintsyuk, 1968 Vintsyuk, T. K. (1968).  Speech discrimination by dynamic programming.  Cybernetics, 4(1):52–57.
  * Yeh et al., 2019 Yeh, R. A., Schwing, A. G., Huang, J., and Murphy, K. (2019).  Diverse generation for multi-agent sports games.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  * Zare et al., 2021 Zare, N., Amini, O., Sayareh, A., Sarvmaili, M., Firouzkouhi, A., Matwin, S., and Soares, A. (2021).  Improving dribbling, passing, and marking actions in soccer simulation 2d games using machine learning.  In Robot World Cup, pages 340–351. Springer.
  * Zhan et al., 2019 Zhan, E., Zheng, S., Yue, Y., Sha, L., and Lucey, P. (2019).  Generating multi-agent trajectories using programmatic weak supervision.  In International Conference on Learning Representations.
  * Zheng et al., 2016 Zheng, S., Yue, Y., and Hobbs, J. (2016).  Generating long-term trajectories using deep hierarchical networks.  In Advances in Neural Information Processing Systems 29, pages 1543–1551.
  * Zhu et al., 2020 Zhu, Z., Lin, K., and Zhou, J. (2020).  Transfer learning in deep reinforcement learning: A survey.  arXiv preprint arXiv:2009.07888.
  * Zhu and Zhao, 2021 Zhu, Z. and Zhao, H. (2021).  A survey of deep rl and il for autonomous driving policy learning.  IEEE Transactions on Intelligent Transportation Systems.


# Shell-model study of octupole collectivity near 208Pb

P. Van Isacker  Grand Accélérateur National d’Ions Lourds, CEA/DRF-CNRS/IN2P3,
Boulevard Henri Becquerel, F-14076 Caen, France  M. Rejmund  Grand
Accélérateur National d’Ions Lourds, CEA/DRF-CNRS/IN2P3, Boulevard Henri
Becquerel, F-14076 Caen, France

###### Abstract

We show that the collectivity of the particle-hole wave function of low-lying
octupole $3^{-}$ states in doubly magic nuclei is mainly due to the neutron-
proton interaction. Both the enhanced reduced transition probability to the
ground state, $B({\rm E3};3^{-}_{1}\rightarrow 0^{+}_{1})$, and the coupling
of the octupole excitation to a nucleon result from the coherent action of all
the components of the collective state. The results obtained with a realistic
shell-model interaction both for 208Pb and 209Pb agree with the geometric
collective model of Bohr and Mottelson, where octupole excitations are
associated with phonons corresponding to collective shape oscillations of the
surface of the nucleus.

###### pacs:

21.60.Fw, 21.60.Cs, 21.60.Ev

Introduction. Collective excitations are ubiquitous in quantum many-body
systems. Their occurrence has given rise to models that capture essential
properties in terms of a few degrees of freedom, usually of bosonic nature.
Ultimately, however, their description invokes excitations of the constituent
particles of the system, which may vary from fully collective to purely
single-particle. Nuclei are prime examples of this dichotomy. Depending on its
location in the nuclear chart (i.e., its number of neutrons $N$ and protons
$Z$), a nucleus may exhibit either single-particle or collective types of
excitation. To complicate matters even further, in many nuclei both types of
excitation coexist. This dual nature is also reflected in the history of
nuclear physics: the 1950s saw the development of the nuclear shell model
Mayer49 ; Haxel49 , which stresses the importance of single-particle behavior,
in parallel with the elaboration of the geometric collective model Rainwater50
; Bohr53 . It is by now understood that both descriptions of the nucleus are
not necessarily incompatible; for example, nuclear rotational motion arises as
a result of an SU(3) symmetry of the shell model Elliott58a ; Elliott58b .
Nevertheless, the interplay between single-particle and collective nuclear
excitations to this day remains to be fully understood microscopically.

In many doubly magic nuclei the lowest-energy excitation has spin-parity
$J^{\pi}=3^{-}$. In the geometric collective model this state is described as
a surface vibration in the octupole degree of freedom. In the shell model it
corresponds to a coherent superposition of particle-hole (ph) excitations of
nucleons across the shell closures and we show here that this superposition
obeys universal symmetry properties. Nuclei with one nucleon more or one
nucleon less than the double shell closure are then expected to exhibit
excitations associated with the single particle or hole as well as those where
the single nucleon is coupled to the octupole vibration BM75 . Such odd-mass
nuclei are therefore ideal testing grounds of the interplay between single-
particle and collective nuclear excitations.

Octupole states in doubly magic nuclei. In a one-particle one-hole (1p1h)
approximation an octupole excitation in a doubly magic nucleus corresponds to
the linear combination of 1p1h excitations across the shell closure

$|3^{-}_{\rm
c}\rangle=\sum_{\rho}\sum_{k^{\prime}k}c^{\rho}_{k^{\prime}k}|j_{\rho
k^{\prime}}j_{\rho k}^{-1};3^{-}\rangle$ (1)

where particle orbitals above the shell closure are denoted with primed
indices $j_{\rho k^{\prime}}$ [short-hand notation for the complete set of
single-particle labels $(nlj)_{\rho k^{\prime}}$] and hole orbitals below the
shell closure with unprimed indices $j_{\rho k}$. Both occur for neutrons
($\rho=\nu$) as well as protons ($\rho=\pi$). The coefficients
$c^{\rho}_{k^{\prime}k}$ in Eq. (1) are the amplitudes of the wave function,
with $(c^{\rho}_{k^{\prime}k})^{2}$ being the probability to find the
$|3^{-}_{\rm c}\rangle$ state in the $|j_{\rho k^{\prime}}j_{\rho
k}^{-1};3^{-}\rangle$ configuration. The values of $c^{\rho}_{k^{\prime}k}$
result from the diagonalization of the nuclear Hamiltonian

$\hat{H}=\sum_{\rho}\biggl{(}\sum_{k}\epsilon_{\rho k}\hat{n}_{\rho
k}+\sum_{k^{\prime}}\epsilon_{\rho k^{\prime}}\hat{n}_{\rho
k^{\prime}}+\hat{V}_{\rho\rho}\biggr{)}+\hat{V}_{\nu\pi},$ (2)

where $\hat{n}_{\rho k}$ is the number operator for $\rho$ nucleons in orbital
$j_{\rho k}$, $\epsilon_{\rho k}$ is its single-nucleon energy, and
$\hat{V}_{\nu\nu}$, $\hat{V}_{\pi\pi}$, and $\hat{V}_{\nu\pi}$ are the
neutron-neutron ($\nu\nu$), proton-proton ($\pi\pi$), and neutron-proton
($\nu\pi$) interactions, respectively. The orbitals appropriate for the 208Pb
region span two major shells for neutrons and protons, from $N=82$ to $184$
for neutrons and from $Z=50$ to $126$ for protons. The effective single-
nucleon energies have been deduced from the experimental data by Warburton and
Brown Warburton1991 . There are $\sim$35000 two-body matrix elements in this
space, obtained in a variety of ways, as described by Brown Brown00 . The set
used here is taken from Ref. Wrzesinski01 .

Figure 1: (a) Components in the wave function of the collective $3^{-}_{\rm
c}$ state calculated with a realistic shell-model Hamiltonian
($c^{\rho}_{k^{\prime}k}$, black) and compared with the analytic expression
(3) ($\bar{c}^{\rho}_{k^{\prime}k}$ with $\alpha_{\nu}=0.734$ and
$\alpha_{\pi}=0.679$, blue). (b) The reduced transition matrix elements
$\langle{\rm E3}\rangle\equiv\langle 0^{+}_{1}\|{\rm E3}\|j_{\rho
k^{\prime}}j_{\rho k}^{-1};3^{-}\rangle$, in units $e$fm3, multiplied with the
coefficients $c^{\rho}_{k^{\prime}k}$ (black) and
$\bar{c}^{\rho}_{k^{\prime}k}$ (blue). (c) Components calculated with a
realistic shell-model Hamiltonian with zero $\nu\pi$ interaction
($\tilde{c}^{\rho}_{k^{\prime}k}$, red). On the $y$ axis are indicated all
possible 1p1h configurations in a simplified notation, e.g. $f_{5}\rightarrow
s_{1}$ stands for $|s_{1/2}f_{5/2}^{-1};3^{-}\rangle$.

With this realistic shell-model Hamiltonian the first-excited $3^{-}$ level in
208Pb is calculated at an excitation energy $E_{\rm x}=2460$ keV, to be
compared with the experimental value of 2615 keV NNDC . The state has a highly
fragmented 1p1h structure as illustrated in Fig. 1(a)(black), which shows the
coefficients $c^{\rho}_{k^{\prime}k}$ for the different 1p1h excitations. The
components of the electric octupole (or E3) decay of the $3^{-}_{1}$ level to
the $0^{+}$ ground state can be obtained from the product of the
$c^{\rho}_{k^{\prime}k}$ and the corresponding reduced transition matrix
elements $\langle 0^{+}_{1}\|{\rm E3}\|j_{\rho k^{\prime}}j_{\rho
k}^{-1};3^{-}\rangle$; they are shown in Fig. 1(b)(black). Although the
coefficients $c^{\rho}_{k^{\prime}k}$ themselves have alternating signs, see
Fig. 1(a)(black), all components of the E3 transition act coherently. The
collective character of the octupole state $3^{-}_{\rm c}$ follows from above-
mentioned coherence with respect to E3 decay. As a result the total E3 reduced
transition matrix element, obtained by multiplying with the effective charges
$e_{\nu}=0.35$ and $e_{\pi}=1.35$ and summing all contributions, leads to an
enhanced transition probability, $B({\rm E3};3^{-}_{\rm c}\rightarrow
0^{+}_{1})=36$ W.u., to be compared with the experimental value of 34.0(5)
W.u. Spear83 .

The fragmentation and coherence in the E3 decay of the $3^{-}_{\rm c}$ state
can be understood in the context of a schematic model that assumes degenerate
single-nucleon energies below and above the shell closures (with gaps
$\Delta\epsilon_{\rho}$) and a surface delta interaction (SDI) between the
nucleons Isacker20 . For the SDI it is assumed that Brussaard77  (i) the
interaction takes place at the surface only, (ii) the two-body force is of
extreme short range, and (iii) the probability of finding a nucleon at the
nuclear surface is independent of the shell-model orbital in which the nucleon
moves. The SDI is a crude approximation to a realistic shell-model interaction
in terms of the isovector strengths $a_{1\rho}$ of the $\nu\nu$ and $\pi\pi$
interactions, and the isoscalar and isovector strengths $a_{0}$ and $a_{1}$ of
the $\nu\pi$ interaction, all with an estimated value of $(25/A)$ MeV
Brussaard77 . With the assumptions of the schematic model the coefficients in
Eq. (1) can be derived analytically note1 ,

$\bar{c}^{\rho}_{k^{\prime}k}\approx\alpha_{\rho}(S_{\rho})^{-1/2}g^{\rho}_{k^{\prime}k},$
(3)

with $S_{\rho}\equiv\sum_{kk^{\prime}}(g^{\rho}_{k^{\prime}k})^{2}$ and

$g^{\rho}_{k^{\prime}k}=(-)^{j_{\rho k}-1/2}[j_{\rho k}][j_{\rho
k^{\prime}}]\Bigl{(}\begin{array}[]{ccc}j_{\rho k^{\prime}}&j_{\rho k}&3\\\
\nicefrac{{1}}{{2}}&-\nicefrac{{1}}{{2}}&0\end{array}\Bigr{)},$ (4)

where the symbol between brackets is a Wigner $3j$ coefficient Talmi93  and
$[j]\equiv\sqrt{2j+1}$. For 208Pb, $S_{\nu}=25.39$ and $S_{\pi}=16.29$. The
coefficients $\alpha_{\nu}$ and $\alpha_{\pi}$ in Eq. (3) are obtained from
the diagonalization of the $2\times 2$ matrix

$\left[\begin{array}[]{cc}\displaystyle\Delta\epsilon_{\nu}-\frac{a_{1\nu}}{2}S_{\nu}&\displaystyle-\frac{3a_{0}+a_{1}}{4}\sqrt{S_{\nu}S_{\pi}}\\\
\displaystyle-\frac{3a_{0}+a_{1}}{4}\sqrt{S_{\nu}S_{\pi}}&\displaystyle\Delta\epsilon_{\pi}-\frac{a_{1\pi}}{2}S_{\pi}\end{array}\right],$
(5)

which decouples approximately from the rest of the 1p1h space. This defines
the neutron and proton octupole excitations $3^{-}_{{\rm c}\nu}$ and
$3^{-}_{{\rm c}\pi}$, which are strongly coupled by the off-diagonal element
in the matrix (5) due to the $\nu\pi$ interaction, giving rise to a
combination

$|3^{-}_{\rm c}\rangle=\alpha_{\nu}|3^{-}_{{\rm
c}\nu}\rangle+\alpha_{\pi}|3^{-}_{{\rm c}\pi}\rangle$ (6)

at low energy, which is symmetric (i.e., the $\alpha_{\nu}$ and $\alpha_{\pi}$
have the same sign), and an anti-symmetric (sometimes called isovector)
combination at higher energy.

The coefficients $\bar{c}^{\rho}_{k^{\prime}k}$ of Eq. (3) are shown in Fig.
1(a)(blue). The $3^{-}_{\rm c}$ states in the realistic and schematic
calculation have a similar structure with an overlap $\sum_{\rho
kk^{\prime}}c^{\rho}_{k^{\prime}k}\bar{c}^{\rho}_{k^{\prime}k}\approx 0.87$.
The strong mixing between $3^{-}_{{\rm c}\nu}$ and $3^{-}_{{\rm c}\pi}$,
obtained with SDI, indicates that the $\nu\pi$ interaction is a key component
of octupole collectivity (see also below). The coherence of the E3 decay can
also be proven analytically for the schematic model, leading to Isacker20

$B({\rm E3};3^{-}_{\rm c}\rightarrow
0^{+}_{1})=\biggr{(}\sum_{\rho}e_{\rho}\alpha_{\rho}\frac{S^{(3)}_{\rho}}{\sqrt{S_{\rho}}}\biggr{)}^{2}e^{2}b^{6},$
(7)

where $b$ is the length parameter of the harmonic oscillator and
$S^{(\lambda)}_{\rho}\equiv\sum_{kk^{\prime}}(g^{\rho}_{k^{\prime}k})^{2}I_{n_{\rho
k}l_{\rho k}n_{\rho k^{\prime}}l_{\rho k^{\prime}}}^{(\lambda)}$, in terms of
radial integrals that are all positive. The corresponding expression in the
geometric collective model reads (see Sect. 6.3 of Ref. BM75 )

$B({\rm E3};3^{-}_{\rm c}\rightarrow
0^{+}_{1})=\biggl{(}\frac{3}{4\pi}ZeR^{3}\sqrt{\frac{\hbar\omega_{3}}{2C_{3}}}\biggr{)}^{2},$
(8)

where $\sqrt{\hbar\omega_{3}/2C_{3}}$ is the amplitude of the octupole
vibration.

A further remark concerns the origin of octupole collectivity. A different set
of coefficients $\tilde{c}^{\rho}_{k^{\prime}k}$ in Eq. (1) is obtained if the
$\nu\pi$ component of the realistic interaction is put to zero, see Fig.
1(c)(red). This leads to a strongly reduced fragmentation of the $3^{-}_{{\rm
c}\rho}$ states: Without $\nu\pi$ interaction octupole collectivity is greatly
diminished. It is also diminished if the $\nu\nu$ and $\pi\pi$ interactions
are omitted but the loss of collectivity is less important in that case.
Therefore, the collectivity of the octupole states exists predominantly by
virtue of the $\nu\pi$ interaction, which couples the neutron and proton
$3^{-}_{{\rm c}\rho}$ states and generates their collective structure.

Octupole states in odd-mass nuclei. We consider an odd-mass nucleus with one
neutron more than a doubly magic nucleus. The case of a proton and/or a hole
coupled to a doubly magic nucleus can be treated in a similar fashion. The
shell-model calculation is carried out in a basis with 1p0h and 2p1h states,

$|j_{\nu r^{\prime}}\rangle\quad{\rm and}\quad|(j_{\rho k^{\prime}}j_{\rho
k}^{-1})^{(J_{k})}j_{\nu s^{\prime}};J\rangle.$ (9)

The first state represents a neutron in the orbital $j_{\nu r^{\prime}}$ and
in the second state the 1p1h configuration is coupled to angular momentum
$J_{k}$, which is subsequently coupled with $j_{\nu s^{\prime}}$ to total $J$.
All possible combinations of $j_{\rho k^{\prime}}$, $j_{\rho k}^{-1}$,
$J_{k}$, and $j_{\nu s^{\prime}}$ lead to an overcomplete, non-orthogonal
basis, which can be reduced to a complete, orthogonal one by diagonalizing the
overlap matrix. This basis may include spurious components with
$J^{\pi}_{k}=1^{-}$, which are removed by adding a center-of-mass term
Gloeckner74  to the physical Hamiltonian (2). The calculation in the basis (9)
requires the evaluation of different types of matrix element, which can be
done by standard angular-momentum recoupling methods.

States of the odd-mass nucleus can be obtained by coupling a particle to the
octupole phonon of the core. If such configurations occur at low excitation
energy and if they are not fragmented over many states by the nuclear
interaction, a simple ‘octupole’ description of the odd-mass nucleus can be
envisaged. In this octupole model, besides single-particle states, only states
obtained from the coupling of a particle to the octupole phonon are
considered. A case of particular interest occurs if two orbitals $j_{\nu
r^{\prime}}$ and $j_{\nu s^{\prime}}$ in the basis (9) have opposite parity
and satisfy $|j_{\nu s^{\prime}}-j_{\nu r^{\prime}}|=|l_{\nu
s^{\prime}}-l_{\nu r^{\prime}}|=3$. In 209Pb this happens for $j_{\nu
r^{\prime}}=g_{9/2}$ and $j_{\nu s^{\prime}}=j_{15/2}$. In that case the
single-particle state $|j_{\nu r^{\prime}}\rangle$ and the octupole-particle
state $|3^{-}_{\rm c}\times j_{\nu s^{\prime}};J\rangle$ may mix if $J=j_{\nu
r^{\prime}}$, giving rise to the eigenstates

$\displaystyle|J_{{\rm o}1}\rangle={}$ $\displaystyle\alpha|j_{\nu
r^{\prime}}\rangle+\beta|3^{-}_{\rm c}\times j_{\nu s^{\prime}};J\rangle,$
$\displaystyle|J_{{\rm o}2}\rangle={}$ $\displaystyle\beta|j_{\nu
r^{\prime}}\rangle-\alpha|3^{-}_{\rm c}\times j_{\nu s^{\prime}};J\rangle,$
(10)

where the subscript ‘o’ indicates that these are states in the octupole model.
Since they are obtained from the diagonalization of a $2\times 2$ Hamiltonian
matrix in the basis $|j_{\nu r^{\prime}}\rangle$ and $|3^{-}_{\rm c}\times
j_{\nu s^{\prime}};J\rangle$, they carry an additional index 1 or 2.

In the particle-vibration coupling approach of the geometric collective model
BM75  the diagonal elements of the $2\times 2$ matrix are

$\displaystyle\langle j_{\nu r^{\prime}}|\hat{H}|j_{\nu r^{\prime}}\rangle={}$
$\displaystyle E_{0}+\epsilon_{\nu r^{\prime}},$ (11) $\displaystyle\langle
3^{-}_{\rm c}\times j_{\nu s^{\prime}};J|\hat{H}|3^{-}_{\rm c}\times j_{\nu
s^{\prime}};J\rangle={}$ $\displaystyle E_{0}+\epsilon_{\nu s^{\prime}}+E_{\rm
x}(3^{-}_{\rm c}),$

where $E_{0}$ is the ground-state energy of the doubly magic nucleus and
$E_{\rm x}(3^{-}_{\rm c})$ the excitation energy of its $3^{-}_{\rm c}$ state.
The mixing between the single-particle and octupole-particle states is derived
from the phonon character of the octupole excitation, leading to (see Sect.
6.5 of Ref. BM75 )

$\displaystyle h(j_{\nu r^{\prime}},3^{-}_{\rm c}j_{\nu
s^{\prime}})\equiv\langle j_{\nu r^{\prime}}|\hat{H}|3^{-}_{\rm c}\times
j_{\nu s^{\prime}};J=j_{\nu r^{\prime}}\rangle$ (12)
$\displaystyle=\varphi\sqrt{\frac{7}{4\pi}}[j_{\nu
s^{\prime}}]\Bigl{(}\begin{array}[]{ccc}j_{\nu s^{\prime}}&j_{\nu
r^{\prime}}&3\\\
\nicefrac{{1}}{{2}}&-\nicefrac{{1}}{{2}}&0\end{array}\Bigr{)}\sqrt{\frac{\hbar\omega_{3}}{2C_{3}}}\langle
j_{\nu s^{\prime}}|k_{3}(r)|j_{\nu r^{\prime}}\rangle,$ (15)

where $k_{3}(r)=R_{0}\partial V(r)/\partial r$ is the form factor of the
octupole oscillation and $\varphi$ is a phase.

Application to 209Pb. We test the validity of the particle-vibration coupling
approach of the geometric collective model by comparing it to a shell-model
calculation in a 1p0h+2p1h basis. In a first step the diagonalization of the
shell-model Hamiltonian (2) for the nucleus 208Pb determines the coefficients
$c^{\rho}_{k^{\prime}k}$. Next, we evaluate the diagonal matrix elements
$\langle j_{\nu r^{\prime}}|\hat{H}|j_{\nu r^{\prime}}\rangle$ and $\langle
3^{-}_{\rm c}\times j_{\nu s^{\prime}};J|\hat{H}|3^{-}_{\rm c}\times j_{\nu
s^{\prime}};J\rangle$. While the first is just the expression (11), the second
is calculated microscopically with use of the coefficients
$c^{\rho}_{k^{\prime}k}$ derived for 208Pb. The off-diagonal matrix element
$h(j_{\nu r^{\prime}},3^{-}_{\rm c}j_{\nu s^{\prime}})$ in Eq. (12) is also
calculated microscopically with the expression

$\displaystyle-\sqrt{\frac{7}{2J+1}}\biggl{(}\sum_{k^{\prime}k}c^{\nu}_{k^{\prime}k}\langle
j_{\nu r^{\prime}}j_{\nu s^{\prime}}^{-1};3|\hat{V}_{\nu\nu}|j_{\nu
k^{\prime}}j_{\nu k}^{-1};3\rangle$
$\displaystyle+\sum_{k^{\prime}k}c^{\pi}_{k^{\prime}k}\langle j_{\nu
r^{\prime}}j_{\nu s^{\prime}}^{-1};3|\hat{V}_{\nu\pi}|j_{\pi k^{\prime}}j_{\pi
k}^{-1};3\rangle\biggr{)}.$ (16)

Although the individual terms in the sums in Eq. (16) are small, they all act
coherently to yield a large off-diagonal matrix element Ralet19 .

The coefficients $\alpha$ and $\beta$ in Eq. (10) are obtained by
diagonalizing a $2\times 2$ Hamiltonian matrix with elements calculated in the
shell model. In a final step a diagonalization of the Hamiltonian (2) is
carried out in the full 1p0h+2p1h basis, followed by the calculation of the
overlaps $|\langle J_{k}^{\pi}|J_{{\rm o}i}^{\pi}\rangle|^{2}$, where
$|J_{k}^{\pi}\rangle$ is the $k^{\rm th}$ eigenstate in the full space and
$|J_{{\rm o}i}^{\pi}\rangle$, $i=1,2$, an eigenstate in the two-dimensional
space. The states (10) constitute a good approximation to the lowest shell-
model states in the 1p0h+2p1h space if $|\langle J_{k}^{\pi}|J_{{\rm
o}i}^{\pi}\rangle|^{2}\lesssim 1$ for $k=i=1,2$, and $|\langle
J_{k}^{\pi}|J_{{\rm o}i}^{\pi}\rangle|^{2}\approx 0$ otherwise.

Figure 2:  Overlaps $|\langle J_{k}^{\pi}|J_{{\rm o}i}^{\pi}\rangle|^{2}$ (see
text) for the $J^{\pi}=9/2^{+}$ and $15/2^{-}$ states in 209Pb. The two
eigenstates $|J_{{\rm o}i}^{\pi}\rangle$, $i=1,2$, are obtained in the
octupole model with the single-particle orbitals $g_{9/2}$ and $j_{15/2}$, and
their coupling to $3^{-}_{\rm c}$. The eigenstates $|J_{k}^{\pi}\rangle$ are
obtained with the realistic shell-model Hamiltonian in the 1p0h+2p1h space.
For display purposes the squared overlaps are multiplied by 1, 10, 100, or
1000, indicated in black, red, blue, or green, respectively.

Figure 2 provides an illustration of this test. The two-component wave
functions originate from the single-particle orbitals $g_{9/2}$ and
$j_{15/2}$, and their coupling to $3^{-}_{\rm c}$; the wave functions in the
1p0h+2p1h space have 467 and 530 components for $J^{\pi}=9/2^{+}$ and
$15/2^{-}$, respectively. The lower-energy eigenstate of the $2\times 2$
matrix is predominantly of single-particle character, either $g_{9/2}$ or
$j_{15/2}$, and this finding is confirmed by the calculation in the 1p0h+2p1h
space. The upper-energy eigenstate is predominantly of octupole-particle
character, and the square of its overlap with the full-space wave function is
0.97 (0.91) for $J_{2}^{\pi}=9/2_{2}^{+}$ ($15/2_{2}^{-}$).

Figure 3:  Distribution of a neutron in the $j_{15/2}$ orbital coupled to the
octupole excitation $3^{-}_{\rm c}$ of 208Pb. The bars (in the color code of
Fig. 2) indicate the overlaps $|\langle J^{\pi}_{k}|3^{-}_{\rm c}\times
j_{15/2};J\rangle|^{2}$ for $J^{\pi}=13/2^{+},21/2^{+}$ in 209Pb, where
$|J^{\pi}_{k}\rangle$ are eigenstates of the shell-model Hamiltonian in the
1p0h+2p1h space.

Many spin-parities are not among those available to a single neutron in the
126-184 orbitals but can be obtained by coupling the neutron to the octupole
phonon. Are such particle-octupole states found in the low-energy
eigenspectrum of the realistic shell-model Hamiltonian? Figure 3 shows two
representative cases, namely the states $|3^{-}_{\rm c}\times
j_{15/2};J^{\pi}\rangle$ with $J^{\pi}=13/2^{+}$ and $21/2^{+}$, and their
distribution over the eigenstates in 209Pb. The $J^{\pi}=21/2^{+}_{1}$ state
is overwhelmingly (the squared overlap is 0.996) that of a $j_{15/2}$ neutron
coupled to $3^{-}_{\rm c}$. In contrast, the $J^{\pi}=13/2^{+}_{1}$ state,
while still dominantly $3^{-}_{\rm c}\times j_{15/2}$ (the squared overlap is
0.74), interacts with $|(g_{9/2}g_{9/2})^{(0)}i_{13/2}^{-1};13/2\rangle$,
which happens to be very close in energy.

The conclusion of a systematic study of all possible cases involving single-
particle as well as octupole-particle excitations in 209Pb is that no member
of the $3^{-}_{\rm c}\times g_{9/2}$ multiplet shows any fragmentation in the
shell-model calculation, that the $3^{-}_{\rm c}\times j_{15/2}$ multiplet is
pure except for $J^{\pi}=13/2^{+}$, and that the high-spin members
($J^{\pi}\gtrsim 11/2^{-}$) of the $3^{-}_{\rm c}\times i_{11/2}$ multiplet
are also unfragmented. Other multiplets occur at higher energies in the region
of higher level density and display considerable fragmentation. However, close
to the full strength of particle-octupole coupled states can be found within
$\sim 400$ keV around the unperturbed particle-octupole energy. Our results
demonstrate that the $3^{-}_{\rm c}$ state preserves its bosonic identity
while coupling with a single particle or hole.

In the schematic model the following expression is obtained for the off-
diagonal matrix element (16):

$\displaystyle h(j_{\nu r^{\prime}},3^{-}_{\rm c}j_{\nu s^{\prime}})$ (17)
$\displaystyle\quad\approx(-)^{j_{\nu r^{\prime}}+1/2}\sqrt{7}[j_{\nu
s^{\prime}}]\Bigl{(}\begin{array}[]{ccc}j_{\nu s^{\prime}}&j_{\nu
r^{\prime}}&3\\\
\nicefrac{{1}}{{2}}&-\nicefrac{{1}}{{2}}&0\end{array}\Bigr{)}f(a_{1\nu},a_{0},a_{1}),$
(20)

with

$f(a_{1\nu},a_{0},a_{1})=\biggl{(}\frac{a_{1\nu}}{2}\alpha_{\nu}\sqrt{S_{\nu}}+\frac{3a_{0}+a_{1}}{4}\alpha_{\pi}\sqrt{S_{\pi}}\biggr{)}.$
(21)

The analytic expressions (7) and (17) together with the corresponding
equations (8) and (12) in the geometric collective model, lead to the
following result:

$|\langle j_{\nu s^{\prime}}|k_{3}(r)|j_{\nu
r^{\prime}}\rangle|\approx\frac{15}{16}\sqrt{\frac{15}{2\pi}}Z\sqrt{A}\frac{f(a_{1\nu},a_{0},a_{1})}{S},$
(22)

where $S=\sum_{\rho}e_{\rho}\alpha_{\rho}S^{(3)}_{\rho}S_{\rho}^{-1/2}$ and
use is made of Eq. (2.36) of Ref. Brussaard77  for the oscillator length $b$.
Equation (22) gives a simple expression for the matrix element of
$R_{0}\partial V(r)/\partial r$ in terms of the orbitals included in the
shell-model space and of the strength parameters of the SDI.

In summary, octupole excitations in doubly magic nuclei exhibit universal
symmetry properties that explain their collective structure and phonon-like
behavior. In 208Pb the collectivity of the particle-hole wave function of the
$3_{1}^{-}$ state is mainly due to the neutron-proton interaction. The
enhanced reduced transition probability to the ground state, $B({\rm
E3};3^{-}_{1}\rightarrow 0^{+}_{1})$, results from the fully coherent action
of all participating particle-hole excitations and is a consequence of the
attractive nature of the residual interaction. When coupled to a low-energy
single particle or hole excitation, the collective nature and bosonic identity
of the octupole state is preserved: E3 collectivity in odd-mass nuclei remains
concentrated in a single state and is not fragmented over many shell-model
eigenstates. In the geometric collective model the coupling of octupole
degrees of freedom to a nucleon arises from the variation in the average
nuclear potential due to the collective vibration. In the shell model this
coupling results from the coherent action of all the components of the
collective state. The findings obtained here with a realistic shell-model
interaction both for 208Pb and 209Pb agree with the geometric collective model
and confirm the phononic behavior of octupole excitations in nuclei.

## References

  * (1) M. G. Mayer, On closed shells in nuclei. II, Phys. Rev. 75, 1969 (1949).
  * (2) O. Haxel, J. H. D. Jensen, and H. Suess, Zur Interpretation der ausgezeichneten Nukleonenzahlen im Bau der Atomkerns, Die Naturwissenschaften 35, 376 (1949).
  * (3) J. Rainwater, Nuclear energy level argument for a spheroidal nuclear model, Phys. Rev. 79, 432 (1950).
  * (4) A. Bohr and B. R. Mottelson, Collective and individual-particle aspects of nuclear structure, Mat. Fys. Medd. Dan. Vid. Selsk. 27, no 16 (1953).
  * (5) J. P. Elliott, Collective motion in the nuclear shell model I. Classification scheme for states of mixed configurations, Proc. Roy. Soc. (London) A 245, 128 (1958).
  * (6) J. P. Elliott, Collective motion in the nuclear shell model I. The introduction of intrinsic wave-functions, Proc. Roy. Soc. (London) A 245, 562 (1958).
  * (7) A. Bohr and B. R. Mottelson, Nuclear Structure II. Nuclear Deformations (Benjamin, New York, 1975).
  * (8) E. K. Warburton and B. A. Brown, Appraisal of the Kuo-Herling shell-model interaction and application to $A$=210–212 nuclei, Phys. Rev. C 43, 602 (1991).
  * (9) B. A. Brown, Double-octupole states in 208Pb, Phys. Rev. Lett. 85, 5300 (2000).
  * (10) J. Wrzesiński et al., High-spin states in 208Pb, Eur. Phys. J. A 10, 259 (2001).
  * (11) ENSDF, National Nuclear Data Center, https://www.nndc.bnl.gov/
  * (12) R. Spear, W. Vermeer, M. Esat, J. Kuehner, A. Baxter, and S. Hinds, An improved determination of the quadrupole moment of the first excited state of 208Pb, Phys. Lett. B 128, 29 (1983).
  * (13) P. Van Isacker, A solvable model for octupole phonons, Eur. Phys. J. Special Topics 229, 2443 (2020).
  * (14) P. J. Brussaard and P. W. M. Glaudemans, Shell-Model Applications in Nuclear Spectroscopy (North-Holland, Amsterdam, 1977).
  * (15) This expression is derived in the phase convention of radial oscillator wave functions that are positive for $r\rightarrow\infty$.
  * (16) I. Talmi, Simple Models of Complex Nuclei. The Shell Model and the Interacting Boson Model (Harwood, Chur, 1993).
  * (17) D. H. Gloeckner and R. D. Lawson, Spurious center-of-mass motion, Phys. Lett. B 53, 313 (1974).
  * (18) D. Ralet et al., Evidence of octupole-phonons at high spin in 207Pb, Phys. Lett. B 797, 134797 (2019).