# Learning Adaptive Multi-Objective Robot Navigation
with Demonstrations

Jorge de Heuvel  Tharun Sethuraman  Maren Bennewitz  All authors are with the
Humanoid Robots Lab at the University of Bonn, Germany. M. Bennewitz and J. de
Heuvel are additionally with the Lamarr Institute for Machine Learning and
Artificial Intelligence, Bonn, Germany. This work has partially been funded by
the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under
the grant number BE 4420/2-2 (FOR 2535 Anticipating Human Behavior) and by the
Federal Ministry of Education and Research (BMBF) under the grant number
16KIS1949.

###### Abstract

Preference-aligned robot navigation in human environments is typically
achieved through learning-based approaches, utilizing demonstrations and user
feedback for personalization. However, personal preferences are subject to
change and might even be context-dependent. Yet traditional reinforcement
learning (RL) approaches with a static reward function often fall short in
adapting to these varying user preferences. This paper introduces a framework
that combines multi-objective reinforcement learning (MORL) with
demonstration-based learning. Our approach allows for dynamic adaptation to
changing user preferences without retraining. Through rigorous evaluations,
including sim-to-real and robot-to-robot transfers, we demonstrate our
framework’s capability to reflect user preferences accurately while achieving
high navigational performance in terms of collision avoidance and goal
pursuance.

##  I Introduction

In recent years, the field of robotics has witnessed remarkable advancements,
particularly in the domain of autonomous mobile robot navigation. Central to
these advancements is the application of deep reinforcement learning (RL),
offering approaches to end-to-end learning of nuanced navigation policies.
Whenever robots coexist and navigate in human environments, a central aspect
for the best human-robot interaction is the alignment of embodied AI systems
to user preferences and values [1, 2].

Despite the progress, traditional RL approaches in robot navigation have
primarily focused on optimizing baseline objectives such as minimizing travel
time or avoiding obstacles. Although these fixed-objective RL frameworks have
laid a strong foundation, they fall short in addressing the complexities of
social navigation, where multiple, often conflicting objectives must be
balanced. For instance, navigating in human-populated environments requires
not just navigational efficiency and safety but also adherence to social
norms, personal space, and user preferences regarding proxemics, approaching
behavior and navigational efficiency [3]. Furthermore, existing research on
navigation often treats user preferences as static [4]. The pre-configured
reward functions of traditional RL policies cannot be adapted to changing
preferences or complex social dynamics without retraining, highlighting a
significant gap in the current methodology.

As a solution, multi-objective reinforcement learning (MORL) emerges [5],
enabling instant post-training adaptation of behavior to accommodate changing
preferences without the need for retraining. This flexibility offers on-the-
fly adaption of various criteria such as efficiency and social appropriateness
but also ensures the maintenance of baseline objectives, including goal
pursuance and collision avoidance.

Figure 1:  Our multi-objective reinforcement learning (MORL) navigation agent
learns to adapt its behavior to varying preferences without re-training. a)
The navigation style can fluently shift between demonstration-induced,
distance keeping, and efficiency objectives. b) We modulate the MORL reward
vector $\boldsymbol{r}_{t}$ with a c) varying preference
$\boldsymbol{\lambda}$, while providing $\boldsymbol{\lambda}$ as input to the
agent. d) The resulting human-centered policy balances the baseline objectives
with preference-reflection.

De Heuvel _et al._[2, 6] have employed an additional behavior cloning loss to
shape RL-based navigation behavior around the human. Yet, the application of
personalized navigation strategies raises questions regarding the balance of
demonstration-based preferences with respect to baseline objectives and
contextual adaptability. It becomes essential to devise mechanisms that can
modulate the influence of demonstrations and preferences [7], ensuring
flexibility and contextually appropriateness.

This paper introduces a novel framework that unites the flexibility of multi-
objective reinforcement learning with the personalization capabilities of
demonstration-based learning, tailored to the unique challenges of social
robot navigation, see Fig. 1. With changing preferences of the user, the
navigation behavior can be adjusted on-the-fly without retraining. Our reward
design achieves distinct preference-adaption while simultaneously meeting
essential navigational baseline objectives such as goal pursuance or collision
avoidance. In our extensive evaluation, we demonstrate the navigational
performance with regards to the quality of preference-reflection. The study
concludes with both a sim-to-real and real-world robot-to-robot transfer of
the learned policy.

In summary, the main contributions of our work are:

  * •

A multi-objective reinforcement learning social robot navigation framework
that enables policy adaption post training.

  * •

The incorporation of demonstration data as a tuneable objective.

  * •

Extensive qualitative and quantitative analyses including a real-world and
robot-to-robot transfer.

##  II Related Work

In the context of learning-based mobile robot navigation, this section spans
multi-objective reinforcement learning and learning from demonstrations.

###  II-A Multi-Objective Reinforcement Learning in Robotics

Multi-objective reinforcement learning (MORL) extends traditional RL
frameworks to optimize several objectives simultaneously, offering a more
versatile approach to decision-making in complex scenarios. While MORL
frameworks for discrete [8] and continuous action spaces [9, 10] have been
presented, the research field around unified benchmarking libraries [11] is
just gaining traction. Yet, MORL has been applied to the fields of autonomous
driving [12] and robotic tasks such as manipulation [13], or navigation [7].

In the scope of mobile robot navigation, the work of Marta _et al._[7]
demonstrates the fluent interpolation between navigational preferences
obtained from a human feedback reward model and baseline objectives.
Specifically, a social force model is modulated that guides the robot to the
goal among surrounding pedestrians. While their reward model is based on
pairwise trajectory queries, we modulate the influence of demonstration data
on a policy that directly drives the robot. Picking up the idea of tuneable
preference reflection, our approach allows diversity beyond the pairwise query
space by integrating demonstration data.

To adjust navigation behavior among humans, Ballou  _et al._ [14] have
employed a meta reinforcement learning setup. The policy can be finetuned to
changes in the reward function efficiently by leveraging the learned meta
knowledge, e.g., for goal pursuance or distance keeping. However, the
adaptation to shifting objectives is not instantaneous. In contrast, our MORL
policy will adapt to changes in preference weights immediately, without
requiring any retraining.

The closest to our study, yet not specifically MORL by definition, is the work
by Choi _et al._[15], who use multi-agent training with parameterized rewards
[16] for an adaptable navigation policy. Each agent is initialized with random
reward weights at the beginning of each episode. Finally, the authors estimate
preferences in a human feedback loop using Bayesian inference. Our goal is the
analysis of navigation behavior rather than preference inference, and the MORL
agent estimates different objective’s Q-values separately.

A similar single-agent approach has been presented by Lee _et al._[16], where
a high-level skill agent dials in the navigation behavior of a low-level
tuneable navigation agent based on observations. Similar to [15], a
parameterized reward function is used. The approach follows a two-step
training process, separating high- and low level agent training. Their skill
vector as the output of the high-level agent is optimized for ideal navigation
in different scenarios, and thus provides explainable insights into the
learned context-dependent reward priorities. In our work, the skill vector
will serve for preference tuning.

###  II-B Personalized Navigation from Preferences

The concept of personalized navigation has gained traction, with researchers
exploring how robots can adapt their navigation strategies based on individual
preferences. Users can express navigational preferences either by ranking
pairwise trajectory queries [15, 17, 7] or providing demonstrations [2, 6].
From both feedback modalities, a preference-aligned navigation policy can be
distilled. In this work, we lay the groundwork for a demonstration-data
infused policy that can be further aligned on-thy-fly without retraining,
e.g., via Bayesian preference inference. We do not focus on the preference
inference process from users, which has extensively been studied [15, 7], but
on the policy’s behavior under changing preferences.

##  III Our Approach

###  III-A Problem Statement

We consider a wheeled robot navigating in the vicinity of a single human and
unknown obstacles towards a local goal. The human has certain preferences
about the navigation style of the robot that may change depending on
navigational context. These navigation preferences can be expressed in the
form of demonstrations, e.g., through a VR interface [2]. The robot should
pursue a local goal location in the environment while performing collision
avoidance, rendering this work into the scope of point navigation. We assume
the position of the agent and human in the environment to be known. The robot
observes its environment through a 2D lidar sensor. The sensor and goal
information is processed by the navigation policy together with a preference
vector, based on which the policy should generate style-adapted navigation
behavior through velocity commands for the robot.

###  III-B Multi-Objective Reinforcement Learning

Multi-objective reinforcement learning (MORL) broadens the scope of
traditional RL by integrating multiple, often conflicting, objectives [5]. In
MORL, the agent is trained to learn policies that strike a balance among these
diverse objectives, as opposed to a singular reward function in classical RL.

The MORL problem is formulated within the framework of a Markov Decision
Process (MDP), defined by the tuple
$(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma)$. Here,
$\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space,
$\mathcal{P}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1]$ is
the state transition probability, and $\gamma$ is the discount factor. A
distinctive feature of MORL is the multi-dimensional reward function
$\mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}^{n}$, which
outputs a vector of rewards $\boldsymbol{r}_{t}$ for $n$ different objectives.

A single policy optimally adheres to a given set of preferences, represented
by the convex preference weight vector
$\boldsymbol{\lambda}\in\mathbb{R}^{n}$. The policy
$\pi_{\boldsymbol{\lambda}}(s)$ optimizes a scalarized reward function
$R_{\boldsymbol{\lambda}}(s,a)=\boldsymbol{\lambda}^{\top}\boldsymbol{r}(s,a)$,
itemizing the different objectives.

We employed the MORL-TD3 implementation of Basaklar _et al._[9], which
introduces several modifications with respect to the policy loss and
preference-space exploration. Together, they enhance policy convergence in
terms of training efficiency, robustness and preference-reflection.

An integral component of this framework is the preference interpolator
$I(\boldsymbol{\lambda})=\boldsymbol{\lambda}_{p}$, which projects the
original preference vectors $\boldsymbol{\lambda}$ into a normalized solution
space, thereby aligning preferences with multi-objective value solutions
$\boldsymbol{Q}$. First, solutions for the maximum expectable values of
$\boldsymbol{Q}(s,a,\boldsymbol{\lambda})$ are obtained for key preferences
evenly distributed on the preference space, initializing a radial basis
function interpolation with a linear kernel. The interpolator is subject to
continuous updates throughout the training process.

Another feature is the angle term
$g(\boldsymbol{\lambda}_{p},\boldsymbol{Q})$, a component of the loss function
designed to minimize the directional angle between the interpolated preference
vectors $\boldsymbol{\lambda}_{p}$ and the multi-objective vector
$\boldsymbol{Q}$:

$g(\boldsymbol{\lambda}_{p},\boldsymbol{Q}(s,a,\boldsymbol{\lambda};\theta))=\cos^{-1}\left(\frac{\boldsymbol{\lambda}_{p}^{T}\boldsymbol{Q}(s,a,\boldsymbol{\lambda};\theta)}{\|\boldsymbol{\lambda}_{p}\|\|\boldsymbol{Q}(s,a,\boldsymbol{\lambda};\theta)\|}\right)$
(1)

This mechanism ensures that the resulting policies align with the specified
preferences, regardless of the varying scales of individual objectives’
rewards. The actor network is updated by maximizing the term
$\boldsymbol{\lambda}\boldsymbol{Q}$, where $\boldsymbol{\lambda}$ is the
original convex preference vector and $\boldsymbol{Q}$ is the critic network’s
output, while simultaneously minimizing the directional angle term.

The loss functions for the critic is defined as:

$\displaystyle
L_{\text{critic}}(\theta_{i})=\mathbb{E}_{(s,a,r,s^{\prime},\boldsymbol{\lambda})\sim
D}\left[\boldsymbol{y}-\boldsymbol{Q}(s,a,\boldsymbol{\lambda};\theta_{i})\right]$
$\displaystyle+\mathbb{E}_{(s,a,\boldsymbol{\lambda})\sim
D}\left[g(\boldsymbol{\lambda}_{p},\boldsymbol{Q}(s,a,\boldsymbol{\lambda};\theta_{i}))\right]$

where
$\boldsymbol{y}=\boldsymbol{r}+\gamma\arg_{\boldsymbol{Q}}\min_{i=1,2}\boldsymbol{\lambda}\boldsymbol{Q}(s^{\prime},\tilde{a},\boldsymbol{\lambda};\theta_{i}^{\prime})$
is the target value obtained from the target networks. The actor loss is

$\displaystyle\nabla_{\phi}L_{\text{actor}}(\phi)=\mathbb{E}_{(s,a,r,s^{\prime},\boldsymbol{\lambda})\sim
D}\left[\nabla_{a}\boldsymbol{\lambda}^{T}\boldsymbol{Q}(\cdot)|_{a=\pi(\cdot)}\nabla_{\phi}\pi(\cdot)\right]$
$\displaystyle+\alpha\mathbb{E}_{(s,a,\boldsymbol{\lambda})\sim
D}\left[\nabla_{a}g(\boldsymbol{\lambda}_{p},\boldsymbol{Q}(\cdot))|_{a=\pi(\cdot)}\nabla_{\phi}\pi(\cdot)\right]$

with
$\boldsymbol{Q}(\cdot)=\boldsymbol{Q}(s,a,\boldsymbol{\lambda};\theta_{1})$
and $\pi(s,\boldsymbol{\lambda};\phi)$ and $\alpha=10$ and the critic’s and
actor’s parameters $\theta_{i}$ and $\phi$.

To efficiently learn across the entire preference space in MORL-TD3, a
hindsight experience replay mechanism [18] retrospectively generates
alternative preference vectors for past experiences, effectively augmenting
the training data diversity in the preference domain and enabling the policy
to adapt to various objectives more effectively.

Moreover, the training process involves running a number of $C_{p}$
environments in parallel for $N$ time steps, resulting in a collection of
$N\times C_{p}$ transitions. Each environment is tailored to explore a
distinct segment of the preference vector space. This parallelization strategy
increases data collection efficiency, which is crucial for training robust and
effective multi-objective reinforcement learning models.

####  III-B1 State and Action Space

The state space contains information about the local goal location, the human,
and surrounding obstacles observed through the lidar sensor. More
specifically, we provide the robot-centric relative 2D goal location
$\boldsymbol{p}_{g}$ and human position $\boldsymbol{p}_{h}$ in polar
coordinates. The $360^{\circ}$ lidar scan is limited to a scanning range of
$4$\mathrm{m}$$ and is min-pooled from its original resolution of $720$ rays
down to $N_{\text{lidar}}=30$ rays. We merge the poses of goal and human with
the lidar distance values $\mathcal{L}_{t}=\\{{d}_{i}^{t}|0\leq
i<N_{\text{lidar}}\\}$ in the state space vector as
$s_{t}=\left(\boldsymbol{p}_{g},\boldsymbol{p}_{h},\mathcal{L}_{t}\right)$.

The robot is controlled with action commands of linear and angular velocity as
$a_{t}=(v,\omega)$ that lie within a range of
$v\in[0,0.5]$\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$$ and
$\omega\in[-\pi,+\pi]$\mathrm{rad}\text{\,}{\mathrm{s}}^{-1}$$. The inference
loop runs at $5$\mathrm{Hz}$$.

####  III-B2 Networks

The networks of actor, critic, behavior cloning policy, and reward model (see
below) are fully-connected multi-layer perceptron (MLP) networks that share
the same architecture of 4 layers with 256 neurons each.

###  III-C Incorporating Demonstrations

Figure 2:  Exploration of D-REX-related demonstration parameters averages over
20 trajectory rollouts. a) The execution of the $\epsilon$-greedy noise-
injected behavior cloning (BC) policy trained with a demonstration
augmentation factor of $N_{D}=$ 1,000 reveals a degradation of goal-
approaching performance measured by the normalized baseline reward with
growing strength of the injected noise. b) The demonstration augmentation
factor $N_{D}$ indicates how many times the human-centric oracle demonstration
trajectory (see Sec.3) was rolled out with randomized obstacle placement.
Better generalization comes with higher randomization efforts. Both plots
compare against the optimal demonstration behavior’s reward.

In this work, we distill preferences from demonstration trajectories $\tau$
into a reward model that natively integrates into MORL as one of the
objectives and guides the learning agent to demonstration-like behavior. In a
typical RL from human feedback setting, a reward model is derived from
pairwise A$\succ$B preference queries in a human feedback process via a
ranking loss. Specifically, we employ the disturbance-based reward
extrapolation (D-REX) approach by Brown _et al._[19]. D-REX solves the problem
of non-existent ranking for demonstration data, by ranking over noise-injected
demonstration trajectories. First, a behavior cloning (BC) policy $\pi_{BC}$
is trained from $N_{D}$ demonstration trajectories. Subsequently, the BC
policy $\pi_{BC}(\cdot|\epsilon)$ is executed in the environment with
increasing level of $\epsilon$-greedy policy noise
$\epsilon\in\mathcal{E}=(\epsilon_{1},\epsilon_{2},\ldots,\epsilon_{d})$ with
$\epsilon_{1}>\epsilon_{2}>\ldots>\epsilon_{d}$. Trajectory rollouts generated
with lower noise are automatically ranked superior compared to their higher-
noise counterparts. Finally, a rich preference-ranking dataset

$D_{\text{rank}}=\\{\tau_{i}\prec\tau_{j}|\tau_{i}\sim\pi_{BC}(\cdot|\epsilon_{i}),\tau_{j}\sim\pi_{BC}(\cdot|\epsilon_{j}),\epsilon_{i}>\epsilon_{j}\\}$

is obtained. From $D_{\text{rank}}$, we train a reward model
$\hat{R}(s,a)\in[0,1]$ using the Bradley-Terry model [20] implemented as a
binary cross entropy loss such that
$\sum_{s\in\tau_{i}}\hat{R}_{\theta}(s,a)<\sum_{s\in\tau_{j}}\hat{R}_{\theta}(s,a)\text{
when }\tau_{i}\prec\tau_{j}$.

For our demonstration data and reward model, we choose down a noise range
$\mathcal{E}=($0$,\ldots,$0.2$)$ for the ranking dataset $D_{\text{rank}}$ and
obtain $N_{D}=$ 1,000 demonstration augmentations from a single human-centric
demonstration trajectory pattern, based in initial testing, compare Fig. 2 and
Sec. 3.

###  III-D Reward

The scope reward functions covers navigational baseline objectives, and three
distinct style objectives based on quantifiable metrics and demonstrations. In
our MORL setup, the baseline objectives are summed and occupy the first entry
in the reward vector $\boldsymbol{r}_{t}$ with a static preference weight of
one. Note that this is neglected in further notations of
$\boldsymbol{\lambda}$ to focus on the tuneable objectives. For the other
objectives occupying entries in the reward vector, the preference weights are
dynamic. For the final reward sum
$R_{\boldsymbol{\lambda}}(s,a)=\boldsymbol{\lambda}^{\top}\boldsymbol{r}(s,a)$
under convex preference weights, it does not make a difference whether the
baseline reward is added to all dynamic objectives individually or occupies a
single separate entry with $\lambda_{\text{baseline}}=1$. However, we have
found it beneficial for the optimization of the angle term $g$ to keep the
baseline reward as a separate vector entry, conversely coming at the cost of
decreased preference adaption upon summation of the baseline reward to all
dynamic objectives.

The reward vector for our MORL framework consists of

$\displaystyle\boldsymbol{r}_{t}=(\underbrace{r_{\text{baseline}}^{t}}_{\text{static}},\underbrace{r^{t}_{\text{demo}},r_{\text{distance}}^{t},r^{t}_{\text{efficiency}}}_{\text{dynamic
objectives}})\text{,}$ (2)

as explained below.

####  III-D1 Baseline

The basal navigation behavior the agent needs to exhibit in any case is goal
pursuance and collision avoidance. Goal-oriented navigation can be achieved
with a continuous potential reward
$r_{\text{goal}}^{t}=125\cdot(d_{g}^{t}-d_{g}^{t-1})$, using the difference in
distance between robot and goal $d_{g}=|\boldsymbol{p}_{g}|$ between two
subsequent time steps. Note that the total non-discounted cumulative goal
reward $R=\sum_{t=0}^{T}r_{\text{goal}}^{t}$ is independent of the number of
steps taken to reach the goal. This way, the baseline reward is not biased
towards shortest path driving behavior. For collision avoidance, a sparse
collision penalty of $r^{t}_{\text{collision}}=-$1,000 upon contact between
robot and any obstacle is typically used. Our baseline reward function becomes
$r_{\text{baseline}}^{t}=r_{\text{goal}}^{t}+r^{t}_{\text{collision}}$.

####  III-D2 Tuneable Style Objectives

Our three style objectives cover proxemics, efficiency, and demonstration-
reflection: An essential comfort factor in a human-robot navigation scenario
are proxemics. We therefore include human distance keeping as one of our
tuneable objectives. To do so, we define a quadratic penalty for positional
closeness $d_{h}=|\boldsymbol{p}_{h}|$ to the human within a range
$d_{\text{thresh}}=2$\mathrm{m}$$ as

$r_{\text{distance}}=\frac{-10}{(d_{\text{thresh}}-d_{\text{min}})^{2}}(d_{h}-d_{\text{thresh}})^{2}\text{
if }d_{h}\leq d_{\text{thresh}}\text{,}$ (3)

else zero, with $d_{\text{min}}=0.3$\mathrm{m}$$. A quadratic term was chosen
over a linear one, as the steps required to circumnavigate the human are
directly proportional to the radius. Where circumnavigating the human at a
larger distance at the cost of more steps should give a smaller cumulative
penalty, only a quadratically decaying term works. This was confirmed in
preliminary experiments.

The second style objective is navigational efficiency, or shortest path
navigation, implemented with a constant time penalty
$r^{t}_{\text{efficiency}}=-10$. This can be in conflict with distance-keeping
to path-obstructing obstacles, and usually means passing by closely for
minimal detours.

The third and last objective is demonstration-like behavior
$r^{t}_{\text{demo}}$, as elaborated below. Note that all rewards of the
tuneable objectives are defined as penalties with identical ranges of
$[-10,0]$.

####  III-D3 Demonstration Acquisition and Reward

Figure 3:  Trajectory rollouts for different preference vectors (columns) for
different scene setups (rows). As can be seen, the navigation policy shifts
its behavior according to the set preference. The colorbars on the right
indicate the interpolated preference space $\Lambda_{i}$ for each plot row.
While shifting Rrow 1) from shortest driving behavior under the maximum
efficiency preference (yellow) to distance-keeping (blue), the minimum
distance to the human increases At the same time, a tendency to navigate
alongside obstacles - if present close to the path - has developed. Shifting
towards the maximum demonstration preference (Row 2), the trajectory shapes
increasingly resemble the demonstration pattern (black). On the shift back to
maximum efficiency (Row 3), the demonstration pattern disappears in favor of
shortest trajectories.

The integration of the demonstration-based reward model
$\hat{R}_{\theta}(s_{t},a_{t})$ in $r^{t}_{\text{demo}}$ makes the amount of
demonstration reflection tuneable. The demonstrations may capture nuanced
navigation styles which are difficult to express using an analytical reward
function, such as a characteristically-shaped trajectories when approaching
the user. In this work, we rely on a demonstration oracle that generates an
optimal demonstration pattern, see Fig. 3. Specifically, the robot
circumnavigates the human in a distinct circular manner. Initially, the robot
advances directly towards the human. Upon reaching a proximity of
$d_{h}=1\unit{}$, it executes a $90^{\circ}$ left turn and proceeds to orbit
the human clockwise at a radius $d_{h}$. Once the goal is abeam to the robot,
it rotates $90^{\circ}$ away from the human and proceeds directly towards the
target. While these are not user-demonstrations, the distinct oracle pattern
enables a clearer performance analysis, as its behavior is by design
contradictory to the other two objectives. It produces longer trajectories in
contrast to the time-efficiency penalty $r^{t}_{\text{efficiency}}$. Also the
distance to the human is quite close at $d_{h}=1$\mathrm{m}$$, contradicting
the distance penalty $r_{\text{distance}}^{t}$ with an impact radius of
$2$\mathrm{m}$$. As the demonstration is anchored solely around the human and
goal position, we can easily augment the single demonstration trajectory by
rolling it out $N_{D}$ times in randomized obstacle configurations, while
recording only collision-free rollouts. The resulting dataset is handed to the
D-REX pipeline, as elaborated in Sec. 2. The final reward term is
$r^{t}_{\text{demo}}=-10\cdot(\hat{R}_{\theta}(s_{t},a_{t})-1)$, aligning it
to the per-step magnitude of the other tuneable objectives’ reward.

##  IV Experimental Evaluation

With our experimental evaluation, we aim to find proof for the following
research claims:

  * •

C1: Our framework produces a preference-adaptable yet reliable navigation
policy.

  * •

C2: The D-REX-based reward model successfully captures and teaches the
demonstration patterns to the agent.

  * •

C3: Our policy generalizes from simulation to the real world, even on a robot
not used for training.

We will perform a qualitative analysis that discusses the behavior diversity
on selected navigation scenarios, followed by a quantitative analysis that
averages navigation task-relevant metrics over a larger number of
trajectories. Our evaluation concludes with a sim-to-real and robot-to-robot
experiment.

###  IV-A Training and Environment

For training, we employ the iGibson simulator [21] with a simulated Kobuki
Turtlebot 2. To generate our human-centered training environments, we randomly
sample the robot start and goal position to have a distance in the range of 6
and $12$\mathrm{m}$$ in an open space. Subsequently, the human is placed at
the center between start and goal. We sample a total of three static obstacles
in form of rectangular-shaped boxes around the scene, while avoiding the
start, goal, and human positions. On its way to the goal, the robot has to
avoid the human, while keeping clear of the obstacles may conflict with the
human distance-keeping objective.

In total, we train our agent with $\gamma=1.0$ on $T\times C_{p}=$ 600k
environment steps jointly collected by $C_{p}=3$ environments and use the
final model for evaluation.

###  IV-B Qualitative Navigation Analysis

In the depicted navigation scenarios of Fig. 3, the navigation strategies of
our MORL agent are illustrated under varying preference weights and obstacle
configurations. Three distinct subplot rows linearly interpolate convex
preferences between pairwise combination for two of the three navigation
objectives, while keeping the third at zero, respectively. For Row 1)
interpolating between distance and efficiency, the preference vector
parameterized by $\mu\in[0,1]$ would read
$\boldsymbol{\lambda}_{1}(\mu)=(0,\mu,1-\mu)$. The other rows follow
analogously by their respective pairwise vector index combinations. The
resulting set of $\boldsymbol{\lambda}_{i}(\mu)$ is
$\Lambda_{i}=\left\\{\left(\frac{i}{N},1-\frac{i}{N},0\right)\mid\frac{i}{N}=\mu,i=0,1,\ldots,N\right\\}$
with $N=10$, jointly forming the test set
$\Lambda=\Lambda_{1}\cup\Lambda_{3}\cup\Lambda_{3}$, see Sec. IV-C.

The plots show the resulting set of trajectories representing the robot’s path
from an initial location (black dot) to a designated target (blue star), in
the context of static obstacles and the presence of a human (red circle).
Also, the original oracle demonstration trajectory is included (black line).

For the shift from efficiency to distance keeping (Fig. 3.1), the trajectory
set indicates an increasing distance to the human during traversal. Vice
versa, the robot traverses with increasing closeness to the human, until
finally it barely passes without collision. A resulting reduction in path
length can be observed, as expected from the efficiency penalty
$r^{t}_{\text{efficiency}}$. Note that the agent learned to occasionally stay
close to nearby obstacles under the maximum human distance objective, followed
by a distinct goal-ward turn when the obstacle was passed.

For the shift from distance keeping to demonstration-like behavior (Fig. 3.2),
a decrease of minimum distance to the human $d_{h}$ can be observed. In
contrast to the shift into the efficiency objective above, the trajectories
now shape into the characteristic demonstration pattern of straight
approaching and circular circumnavigation of the human, followed by a bending
in the line between human an goal. While the sharp corners in front and behind
the human are not as pronounced as in the demonstration trajectory, the
navigation behavior clearly approaches demonstration-like patterns in support
of C2.

The overall picture completes, as we shift preferences from demonstration back
into efficiency, see Fig. 3.3. While in column A) the demonstration driven
trajectories do a nuanced bend around the human, the efficiency driven
trajectories pilot right towards the goal after barely passing the human.

Whenever obstacles are close to the human, the agent avoids collisions at the
cost of reduced distance to the human. Under maximum distance preference,
before and after obstacles, the distance to the human is maintained. Notably,
all trajectories traverse with the human on the right hand site, as defined by
the demo.

We find evidence in C1 and C2, as the results demonstrate the robot’s
capability to modulate its path planning strategy from human-distant
navigation over demonstration-obedient to efficiency-focused direct paths.

###  IV-C Quantitative Analysis

Figure 4:  Quantitative metrics of OUR agent for different preference
configurations (e), tested for statistical significance for dissimilar mean
between the maximum preferences, with *** for $p\leq 0.001$, and ns for no
significance. a) The navigation time is smallest for maximized efficiency
preference, as expected. b) The Fréchet distance to the demonstration
trajectory decreases as the demonstration preference increases. c) The minimum
distance to any obstacle is taken directly from the lidar sensor. d) The
minimum distance to the human grows with its preference weight. A non-MORL
baseline agent (red dotted line) that only obeys the baseline reward term is
included in each plot.

####  IV-C1 Preference Reflection

We performed a quantitative evaluation of the preference-reflecting agent with
respect to multiple performance- and navigation-based metrics, see Fig. 4. The
agent was evaluated for 100 episodes in random environment setups for
different preference weights $\boldsymbol{\lambda}\in\Lambda$. The
interpolated convex preference vectors as described in Sec. IV-B are indicated
as colored fractions in Fig. 4e. Statistical significance for the difference
in the respective mean values between the maximum preferences is indicated, as
measures through a student’s t-test. Regarding the navigational performance,
we observe success rate of $100$$\mathrm{\char 37\relax}$, without timeouts or
collisions, compare Table I, first entry. As the preference for distance-
keeping to the human increases, both the measured minimum human distance and
navigation time increase, see Fig. 4a+d. In other words, the robot
successfully trades both longer demonstration-like and efficiency-driven
trajectories for keeping a higher distance as desired. To measure how well the
original demonstration trajectory is reflected with respect to C2, we compute
the Fréchet distance [22] to the executed trajectory, see Fig. 4b. The Fréchet
distance finds its minimum at $0.41$$\mathrm{m}$ as the demonstration
preference is maximized. Compared to the maximum distance keeping objective,
also the efficiency objective reduces the demonstration Fréchet distance,
which is expected with the demonstration trajectory closely passing by the
human. Comparing the diverging trends of the minimum lidar-recorded distance
to any obstacle $\min(d_{\text{obst}})$ in Fig. 4c against the minimum human
distance $\min(d_{h})$ Fig. 4d, the policy has clearly learned to distinguish
between the human and other static obstacles for distance-keeping. While the
minimum human distance increases together with its preference weight, the
minimum distance to surrounding obstacles does not. The agent has learned to
pursue obstacle-closer trajectories with a higher collision risk in order to
make room for the proxemic preference of the human. The quantitative analysis
complements the picture of successful preference reflection obtained from the
qualitative, finding measurable evidence our research claims C1 and C2.

| $\boldsymbol{\lambda}$ | OUR | -NH | -RM | -RM-NH | -$\gamma$
---|---|---|---|---|---|---
SR$\uparrow$ [%] | $\boldsymbol{\lambda}_{i}\in\Lambda$ | 100 | 96.8 | 100 | 79.6 | 96.0
CR$\downarrow$ [%] | $\boldsymbol{\lambda}_{i}\in\Lambda$ | 0 | 2.7 | 0 | 11.4 | 4.0
TR$\downarrow$ [%] | $\boldsymbol{\lambda}_{i}\in\Lambda$ | 0 | 0.5 | 0 | 9.0 | 0.0
$\min(d_{h})$$\uparrow$ [m] | $\boldsymbol{\lambda}_{\text{dist}}$ | 1.18 | 0.52 | 1.16 | 0.48 | 1.22
Fréchet$\downarrow$ [m] | $\boldsymbol{\lambda}_{\text{demo}}$ | 0.41 | 0.57 | 0.46 | 0.49 | 0.50
Nav. time$\downarrow$ [s] | $\boldsymbol{\lambda}_{\text{eff}}$ | 17.3 | 16.9 | 17.4 | 19.2 | 18.3
TABLE I:  Ablation study with respect to the state space and reward model. For
the ablation identifiers and preference vectors
$\\{\boldsymbol{\lambda}_{\text{dist}},\boldsymbol{\lambda}_{\text{demo}},\boldsymbol{\lambda}_{\text{eff}}\\}$,
please refer to Sec. IV-C3. For brevity, the identifiers are shortent after
OUR, so that, e.g., -NH corresponds to OUR-NH. The results were averaged over
100 trajectories for single $\boldsymbol{\lambda}$, and for the success rate
(SR), collision rate (CR), and timeout rate (TR) additionally over all
$\boldsymbol{\lambda}_{i}\in\Lambda$.

####  IV-C2 Baseline

A non-multi-objective policy that only obeys the navigational baseline rewards
$r_{\text{baseline}}$ of goal pursuance and collision avoidance was trained.
The quantitative results of the baseline agent are included in Fig. 4 as a red
dotted line. Here, two metrics stand out: While the MORL agent trades human
for obstacle distance, the baseline agent lacking the human-distance reward
treats both the human and static obstacles similarly. For this reason, a
similar minimum value for $d_{h}=$0.64$$\mathrm{m}$$ and
$d_{\text{obst}}=$0.72$$\mathrm{m}$$ is measured for the baseline agent, in
clear contrast to the MORL agent. Also, a higher demonstration Fréchet
distance confirms the lack of demonstration knowledge. It is also notable that
the baseline agent has a lower success rate of $97$$\mathrm{\char 37\relax}$.

####  IV-C3 Ablation Study

We ablated the architecture with respect to the state space and demonstration
reward model, compare Table I. The state space changes apply to all involved
models: D-REX BC policy, D-REX reward model, actor, and critic. Firstly, we
exclude the human position from the state space in OUR-NH. Secondly, the
reward model reasons only about the state without the corresponding action as
$r^{t}_{\text{demo}}=\hat{R}_{\theta}(s_{t})$ in OUR-RM, to test the
dependency of demonstration reflection on the action data. Thirdly, the above
two ablations are combined in OUR-RM-NH. Fourthly, we lower the discount value
from $\gamma=1$ to $\gamma=0.99$ in OUR-$\gamma$ to test preference reflection
under shorter-horizon reward optimization. Note that the the maximum
preference vectors in Table I are
$\boldsymbol{\lambda}_{\text{demo}}=(1,0,0)$,
$\boldsymbol{\lambda}_{\text{dist}}=(0,1,0)$,
$\boldsymbol{\lambda}_{\text{eff}}=(0,0,1)$, respectively.

Compared to OUR, we observe a decrease in the distance-reflection capabilities
upon removal of the human position from the state space in OUR-NH and OUR-RM-
NH. Given the intrinsic correlation between demonstration and distance
preferences relative to the human position, this result appears to be a
natural consequence. It furthermore underlines that the MLP networks cannot
differentiate the human from obstacles in the low-resolution lidar. With a
similar collision-free performance, the state-only reward model in OUR-RM is a
contender to our flagship approach, but at slightly weaker preference-
reflection. A slightly better human distance-keeping ability is revealed by
OUR-$\gamma$, but at the cost of more collisions and weaker demonstration and
efficiency adaption. Note that we have also experimented with lower values of
discount value $\gamma$, e.g., at $\gamma=0.95$ without convergence of
training. We attribute this result with the agent’s willingness to defer goal
rewards into the future under higher $\gamma$ for better compliance with
preference objectives.

###  IV-D Real-World Transfer

Figure 5: Real-world experiment setup (top) and results (bottom) with the
policy OUR in a A) sim-to-real transfer with the Kobuki Turtlebot 2 and a B)
robot-to-robot transfer to the Toyota HSR. The navigation behavior in real
world successfully reflects varying the preferences on both robots. While the
Turtlebot exhibits better demonstration reflection, the HSR keeps more
distance to the human under the maximum distance preference.

We evaluate our tuneable policy on a real Kobuki Turtlebot 2 using the Robot
Operating System (ROS) [23], and furthermore perform a robot-to-robot transfer
of the same Turtlebot-trained policy to the Toyota Human Support Robot (HSR).
For localization, we rely on Adaptive Monte Carlo Localization (AMCL) [24] in
an environment pre-mapped with Gmapping [25]. We provide the ground truth
human and goal positions to the agent. As the angular lidar coverage of the
HSR at $270^{\circ}$ differs from the $360^{\circ}$ lidar on the Turtlebot, we
pad the missing rear values with the max imum range values of $4$\mathrm{m}$$.
Because the robot is only allowed to drive forward, the assumption of no
obstacles at the rear should not significantly influence the behavior
performance. Note that the HSR’s lidar sensor is not mounted above but to the
font of the z-axis center of rotation, a possible source for a state space
robot-to-robot discrepancy. Subsequently, we performed navigation runs on both
robots for the preference vectors $\boldsymbol{\lambda}$ in $\Lambda$ with
$N=5$, see Sec. IV-B.

The recorded Turtlebot trajectories are depicted in Fig. 5A, and the HSR
trajectories in Fig. 5B, with an overlay of the built map used solely for
localization. We can successfully observe an adaption of navigation behavior
according to preferences on both robots. For the maximum distance preference
(Fig. 5A1), an oscillatory pattern can be observed on the Turtlebot that we
attribute to a slight oversteering of the angular velocity. For this
preference, the HSR drives closer to the obstacle, but finalizes its
trajectory with a wiggly motion just before the goal, see. Fig. 5B1. For
maximum demonstration reflection, the Turtlebot resembles a trajectory closer
to the demonstration pattern than the HSR, compare Fig. 52\. The used
environment featured significantly more obstacles in the lidar range of the
robot as compared to simulation, highlighting the sim-to-real generalization
capabilities of the system. The reader is invited to refer to the supplemental
video of the real world experiments. In conclusion, the policy smoothly
transfers to the real robot, supporting research claim C3.

##  V Conclusion

In summary, we introduced an innovative framework combining multi-objective
reinforcement learning (MORL) with demonstration-based learning for adaptive,
personalized robot navigation in human environments. Our approach successfully
modulates the conflicting objectives of distance keeping, navigational
efficiency, and demonstration reflection without retraining. To achieve this,
we distill demonstration data into a reward model that shapes the agent’s
trajectories during navigation with variable strength. In various qualitative
and quantitative experiments, we demonstrated the adaption capability to
varying preferences. Finally, we successfully deployed the agent on two real
robots, showing both a sim-to-real and a robot-to-robot transfer.

While the amount of demonstration-reflection can be modulated, a limitation of
our approach is the inability to alter the demonstration data itself without
retraining. This poses an interesting avenue for future research.

We believe that designing robot policies in a multi-objective, therefore
tuneable manner is a step forward in developing robots capable of seamlessly
integrating into human-centric spaces.

## References

  * [1] P. Vamplew, R. Dazeley, C. Foale, S. Firmin, and J. Mummery, “Human-aligned artificial intelligence is a multiobjective problem,” _Ethics and Information Technology_ , vol. 20, no. 1, Mar. 2018.
  * [2] J. de Heuvel, N. Corral, L. Bruckschen, and M. Bennewitz, “Learning Personalized Human-Aware Robot Navigation Using Virtual Reality Demonstrations from a User Study,” in _2022 31th IEEE International Conference on Robot Human Interactive Communication (RO-MAN)_ , 2022.
  * [3] C. Mavrogiannis, F. Baldini, A. Wang, D. Zhao, P. Trautman, A. Steinfeld, and J. Oh, “Core Challenges of Social Robot Navigation: A Survey,” _ACM Transactions on Human-Robot Interaction_ , vol. 12, Feb. 2023.
  * [4] C. Pérez-D’Arpino, C. Liu, P. Goebel, R. Martín-Martín, and S. Savarese, “Robot Navigation in Constrained Pedestrian Environments using Reinforcement Learning,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , May 2021.
  * [5] C. F. Hayes, R. R adulescu, E. Bargiacchi, J. Källström, M. Macfarlane, M. Reymond, T. Verstraeten, L. M. Zintgraf, R. Dazeley, F. Heintz, E. Howley, A. A. Irissappane, P. Mannion, A. Nowé, G. Ramos, M. Restelli, P. Vamplew, and D. M. Roijers, “A practical guide to multi-objective reinforcement learning and planning,” _Autonomous Agents and Multi-Agent Systems_ , vol. 36, no. 1, Apr. 2022.
  * [6] J. de Heuvel, N. Corral, B. Kreis, J. Conradi, A. Driemel, and M. Bennewitz, “Learning Depth Vision-Based Personalized Robot Navigation From Dynamic Demonstrations in Virtual Reality,” in _2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , Oct. 2023.
  * [7] D. Marta, S. Holk, C. Pek, J. Tumova, and I. Leite, “Aligning Human Preferences with Baseline Objectives in Reinforcement Learning,” in _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , May 2023.
  * [8] R. Yang, X. Sun, and K. Narasimhan, “A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation,” in _Advances in Neural Information Processing Systems_ , vol. 32. Curran Associates, Inc., 2019.
  * [9] T. Basaklar, S. Gumussoy, and U. Y. Ogras, “PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithm,” 2023.
  * [10] J. Xu, Y. Tian, P. Ma, D. Rus, S. Sueda, and W. Matusik, “Prediction-Guided Multi-Objective Reinforcement Learning for Continuous Robot Control,” in _Proceedings of the 37th International Conference on Machine Learning_. PMLR, Nov. 2020.
  * [11] F. Felten, L. N. Alegre, A. Nowe, A. Bazzan, E. G. Talbi, G. Danoy, and B. C. da Silva, “A Toolkit for Reliable Benchmarking and Research in Multi-Objective Reinforcement Learning,” _Advances in Neural Information Processing Systems_ , vol. 36, 2023.
  * [12] X. He and C. Lv, “Toward personalized decision making for autonomous vehicles: A constrained multi-objective reinforcement learning technique,” _Transportation Research Part C: Emerging Technologies_ , vol. 156, Nov. 2023\.
  * [13] S. Huang, A. Abdolmaleki, G. Vezzani, P. Brakel, D. J. Mankowitz, M. Neunert, S. Bohez, Y. Tassa, N. Heess, M. Riedmiller, and R. Hadsell, “A Constrained Multi-Objective Reinforcement Learning Framework,” in _Proceedings of the 5th Conference on Robot Learning_. PMLR, Jan. 2022.
  * [14] A. Ballou, X. Alameda-Pineda, and C. Reinke, “Variational meta reinforcement learning for social robotics,” _Applied Intelligence_ , vol. 53, no. 22, Nov. 2023.
  * [15] J. Choi, C. Dance, J.-e. Kim, K.-s. Park, J. Han, J. Seo, and M. Kim, “Fast Adaptation of Deep Reinforcement Learning-Based Navigation Skills to Human Preference,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, May 2020.
  * [16] K. Lee, S. Kim, and J. Choi, “Adaptive and Explainable Deployment of Navigation Skills via Hierarchical Deep Reinforcement Learning,” May 2023\.
  * [17] L. Keselman, K. Shih, M. Hebert, and A. Steinfeld, “Optimizing Algorithms from Pairwise User Preferences,” in _2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , Oct. 2023.
  * [18] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba, “Hindsight Experience Replay,” in _Advances in Neural Information Processing Systems_ , vol. 30. Curran Associates, Inc., 2017.
  * [19] D. S. Brown, W. Goo, and S. Niekum, “Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations,” in _Proceedings of the Conference on Robot Learning_. PMLR, May 2020.
  * [20] R. A. Bradley and M. E. Terry, “Rank analysis of incomplete block designs: I. The method of paired comparisons,” _Biometrika_ , vol. 39, no. 3/4, 1952.
  * [21] C. Li, F. Xia, R. Martín-Martín, M. Lingelbach, S. Srivastava, B. Shen, K. Vainio, C. Gokmen, G. Dharan, T. Jain, A. Kurenkov, C. K. Liu, H. Gweon, J. Wu, L. Fei-Fei, and S. Savarese, “iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks,” _arXiv:2108.03272 [cs]_ , Nov. 2021.
  * [22] H. Alt and M. Godau, “Computing the fréchet distance between two polygonal curves,” _International Journal of Computational Geometry & Applications_, vol. 05, no. 01n02, Mar. 1995.
  * [23] M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, and A. Y. Ng, “ROS: An open-source Robot Operating System,” in _ICRA Workshop on Open Source Software_ , vol. 3. Kobe, Japan, 2009.
  * [24] D. Fox, W. Burgard, F. Dellaert, and S. Thrun, “Monte carlo localization: Efficient position estimation for mobile robots,” _Aaai/iaai_ , vol. 1999, no. 343-349, 1999.
  * [25] G. Grisetti, C. Stachniss, and W. Burgard, “Improving Grid-based SLAM with Rao-Blackwellized Particle Filters by Adaptive Proposals and Selective Resampling,” in _Proceedings of the 2005 IEEE International Conference on Robotics and Automation_. IEEE, 2005.


$\displaystyle(2D_{24}^{2}+D_{2}^{2}+D_{4}^{2})D_{\Delta_{1},\Delta_{2},\Delta_{3},\Delta_{4}}=-(D_{2}+D_{4})_{AB}(D_{1}+D_{3})^{AB}D_{\Delta_{1},\Delta_{2},\Delta_{3},\Delta_{4}}\,.$
(B.10)

Since $(D_{1}+D_{3})$ commutes with $(D_{2}+D_{4})$ and
$D_{i}^{2}=-2m_{i}^{2}=-2\Delta_{i}(\Delta_{i}-d)$ when acting on a conformal
partial wave, we get

$\displaystyle(D_{23}^{2}-m_{2}^{2}-m_{3}^{2})D_{\Delta_{1},\Delta_{2},\Delta_{3},\Delta_{4}}=(D_{14}^{2}-m_{1}^{2}-m_{4}^{2})D_{\Delta_{1},\Delta_{2},\Delta_{3},\Delta_{4}}\,.$
(B.11)

In particular, when $\Delta_{i}=\Delta$, we have

$\displaystyle
D_{13}^{2}D_{\Delta,\Delta,\Delta,\Delta}=D_{24}^{2}D_{\Delta,\Delta,\Delta,\Delta}\,.$
(B.12)

The above relation can be easily generalized to $n$-points,

$\displaystyle\Big{(}D_{I}^{2}-\sum_{i\in
I}m_{i}^{2}\Big{)}D_{\Delta_{1},\Delta_{2},\ldots,\Delta_{n}}=\Big{(}D_{\bar{I}}^{2}-\sum_{i\in\bar{I}}m_{i}^{2}\Big{)}D_{\Delta_{1},\Delta_{2},\ldots,\Delta_{n}}\,,$
(B.13)

where $I\cup\bar{I}=\\{1,2,\ldots,n\\}$ and $I\cap\bar{I}=\emptyset$.

We can also carry out the derivatives in eq. (B.11) explicitly to obtain
relations involving boundary positions. For example, we can use

$\displaystyle-\frac{1}{2}D_{13}^{2}D_{\Delta_{1},\Delta_{2},\Delta_{3},\Delta_{4}}=4\Delta_{1}\Delta_{3}(P_{1}\cdot
P_{3})D_{\Delta_{1}+1,\Delta_{2},\Delta_{3}+1,\Delta_{4}}+\Delta_{1}\Delta_{3}D_{\Delta_{1},\Delta_{2},\Delta_{3},\Delta_{4}}\,,$
(B.14)

and a similar equation for $D_{24}^{2}$ to show that

$\displaystyle(P_{1}\cdot
P_{3})D_{\Delta+1,\Delta,\Delta+1,\Delta}=(P_{2}\cdot
P_{4})D_{\Delta,\Delta+1,\Delta,\Delta+1}\quad\text{ for
}\Delta_{i}=\Delta\,.$ (B.15)

One can also show that acting $D_{12}^{2}$ and $D_{13}^{2}$ consecutively
gives

$\displaystyle\frac{1}{4}D_{12}^{2}D_{13}^{2}D_{\Delta_{1},\Delta_{2},\Delta_{3},\Delta_{4}}$
$\displaystyle=16\Delta_{1}(\Delta_{1}+1)\Delta_{2}\Delta_{3}(P_{1}\\!\cdot\\!P_{2})(P_{1}\\!\cdot\\!P_{3})D_{\Delta_{1}+2,\Delta_{2}+1,\Delta_{3}+1,\Delta_{4}}$
(B.16)
$\displaystyle\quad+4\Delta_{1}(\Delta_{1}+1)\Delta_{2}\Delta_{3}\Big{[}(P_{1}\\!\cdot\\!P_{2})D_{\Delta_{1}+1,\Delta_{2}+1,\Delta_{3},\Delta_{4}}+(2\leftrightarrow
3)\Big{]}$
$\displaystyle\quad-4\Delta_{1}\Delta_{2}\Delta_{3}(P_{2}\\!\cdot\\!P_{3})D_{\Delta_{1},\Delta_{2}+1,\Delta_{3}+1,\Delta_{4}}+\Delta_{1}^{2}\Delta_{2}\Delta_{3}D_{\Delta_{1},\Delta_{2},\Delta_{3},\Delta_{4}}\,,$

which is symmetric under the $2\leftrightarrow 3$ exchange. This also means
that $D_{\Delta_{1},\Delta_{2},\Delta_{3},\Delta_{4}}$ lives in the kernel of
$[D_{12}^{2},D_{13}^{2}]$, namely,
$[D_{12}^{2},D_{13}^{2}]D_{\Delta_{1},\Delta_{2},\Delta_{3},\Delta_{4}}=0$.

Next, we act $D_{1}^{AB}$ onto eq. (B.9) and then use eq. (B.11) to eliminate
$D_{14}^{2}$. This gives

$\displaystyle(D_{12}^{2}+D_{13}^{2}+D_{23}^{2})D_{\Delta_{1},\Delta_{2},\Delta_{3},\Delta_{4}}+(m_{4}^{2}-m_{1}^{2}-m_{2}^{2}-m_{3}^{2})D_{\Delta_{1},\Delta_{2},\Delta_{3},\Delta_{4}}=0\,.$
(B.17)

When $\Delta_{i}=\Delta$, we get

$\displaystyle(D_{12}^{2}+D_{13}^{2}+D_{23}^{2})D_{\Delta,\Delta,\Delta,\Delta}=2\Delta(\Delta-d)D_{\Delta,\Delta,\Delta,\Delta}\,.$
(B.18)

This equation also implies the identity

$\displaystyle(P_{1}\cdot P_{2})D_{\Delta+1,\Delta+1,\Delta,\Delta}$
$\displaystyle+(P_{1}\cdot P_{3})D_{\Delta+1,\Delta,\Delta+1,\Delta}$
$\displaystyle+(P_{2}\cdot
P_{3})D_{\Delta,\Delta+1,\Delta+1,\Delta}=-\frac{4\Delta-d}{4\Delta}D_{\Delta,\Delta,\Delta,\Delta}\,.$
(B.19)

In the on-shell limit $\Delta=d$, we thus derive the relation

$\displaystyle(D_{12}^{2}+D_{13}^{2}+D_{23}^{2})D_{d,d,d,d,}=0\,,$ (B.20)

which resembles the relation $s+t+u=0$ for flat space Mandelstam variables.

The above identities can also be derived using integration-by-parts relations.
Below we show the derivation of eq. (B.11). Our calculation will be given in
the embedding space, which is in parallel with the one in the physical space
DHoker:1999kzh . To start with, we define

$\displaystyle
D_{\partial\Delta_{1},\Delta_{2},\partial\Delta_{3},\Delta_{4}}$
$\displaystyle\equiv\int_{\textrm{AdS}}dXG^{AB}\Bigg{[}\frac{\partial}{\partial
X^{A}}\frac{1}{(-2P_{1}\cdot X)^{\Delta_{1}}}\Bigg{]}\frac{1}{(-2P_{2}\cdot
X)^{\Delta_{2}}}$
$\displaystyle\qquad\qquad\quad\quad\\!\times\Bigg{[}\frac{\partial}{\partial
X^{B}}\frac{1}{(-2P_{3}\cdot X)^{\Delta_{3}}}\Bigg{]}\frac{1}{(-2P_{4}\cdot
X)^{\Delta_{4}}}$ $\displaystyle=4\Delta_{1}\Delta_{3}(P_{1}\cdot
P_{3})D_{\Delta_{1}+1,\Delta_{2},\Delta_{3}+1,\Delta_{4}}+\Delta_{1}\Delta_{3}D_{\Delta_{1},\Delta_{2},\Delta_{3},\Delta_{4}}\,.$
(B.21)

We then use the identity

$\displaystyle\partial_{A}G^{AB}\partial_{B}\left[\frac{1}{(-2P_{1}\cdot
X)^{\Delta_{1}}}\frac{1}{(-2P_{3}\cdot X)^{\Delta_{3}}}\right]$
$\displaystyle=2G^{AB}\frac{\partial}{\partial X^{A}}\frac{1}{(-2P_{1}\cdot
X)^{\Delta_{1}}}\frac{\partial}{\partial X^{B}}\frac{1}{(-2P_{3}\cdot
X)^{\Delta_{3}}}$ $\displaystyle\quad+\frac{m_{1}^{2}+m_{3}^{2}}{(-2P_{1}\cdot
X)^{\Delta_{1}}(-2P_{3}\cdot X)^{\Delta_{3}}}$ (B.22)

to write eq. (B.3) as

$\displaystyle
D_{\partial\Delta_{1},\Delta_{2},\partial\Delta_{3},\Delta_{4}}+\frac{1}{2}(m_{1}^{2}+m_{3}^{2})D_{\Delta_{1},\Delta_{2},\Delta_{3},\Delta_{4}}$
$\displaystyle=\frac{1}{2}\int_{\textrm{AdS}}dX(\partial_{A}G^{AB}\partial_{B})\left[\frac{1}{(-2P_{1}\cdot
X)^{\Delta_{1}}}\frac{1}{(-2P_{3}\cdot
X)^{\Delta_{3}}}\right]\frac{1}{(-2P_{2}\cdot
X)^{\Delta_{2}}}\frac{1}{(-2P_{4}\cdot X)^{\Delta_{4}}}$
$\displaystyle=\frac{1}{2}\int_{\textrm{AdS}}dX(\partial_{A}G^{AB}\partial_{B})\left[\frac{1}{(-2P_{2}\cdot
X)^{\Delta_{2}}}\frac{1}{(-2P_{4}\cdot
X)^{\Delta_{4}}}\right]\frac{1}{(-2P_{1}\cdot
X)^{\Delta_{1}}}\frac{1}{(-2P_{3}\cdot X)^{\Delta_{3}}}$
$\displaystyle=D_{\Delta_{1},\partial\Delta_{2},\Delta_{3},\partial\Delta_{4}}+\frac{1}{2}(m_{2}^{2}+m_{4}^{2})D_{\Delta_{1},\Delta_{2},\Delta_{3},\Delta_{4}}\,,$
(B.23)

where integration-by-parts has been used to obtain the second equation. This
leads to the identity

$\displaystyle\quad 4\Delta_{1}\Delta_{3}(P_{1}\cdot
P_{3})D_{\Delta_{1}+1,\Delta_{2},\Delta_{3}+1,\Delta_{4}}+\frac{1}{2}(2\Delta_{1}\Delta_{3}+m_{1}^{2}+m_{3}^{2})D_{\Delta_{1},\Delta_{2},\Delta_{3},\Delta_{4}}$
$\displaystyle=4\Delta_{2}\Delta_{4}(P_{2}\cdot
P_{4})D_{\Delta_{1},\Delta_{2}+1,\Delta_{3},\Delta_{4}+1}+\frac{1}{2}(2\Delta_{2}\Delta_{4}+m_{2}^{2}+m_{4}^{2})D_{\Delta_{1},\Delta_{2},\Delta_{3},\Delta_{4}}\,.$
(B.24)

When $\Delta_{i}=\Delta$, we can recover $(P_{1}\cdot
P_{3})D_{\Delta+1,\Delta,\Delta+1,\Delta}=(P_{2}\cdot
P_{4})D_{\Delta,\Delta+1,\Delta,\Delta+1}$. Now using eq. (B.14) to replace
$P_{i}\cdot P_{j}$ by $D_{13}^{2}$ immediately leads to

$\displaystyle(D_{23}^{2}-m_{2}^{2}-m_{3}^{2})D_{\Delta_{1},\Delta_{2},\Delta_{3},\Delta_{4}}=(D_{14}^{2}-m_{1}^{2}-m_{4}^{2})D_{\Delta_{1},\Delta_{2},\Delta_{3},\Delta_{4}}\,.$
(B.25)

##  Appendix C Derivation of eq. (6.37)

In this appendix, we give a derivation of eq. (6.37), copied here as

$\displaystyle\textbf{M}^{-1}\Bigg{[}\prod_{1\leqslant
i<j}^{4}\frac{\Gamma(\delta_{ij}+l_{ij})}{\Gamma(\delta_{ij})}\Bigg{]}=\frac{2}{\pi^{d/2}}\frac{\prod_{i=1}^{4}\Gamma(\tilde{\Delta}_{i})}{\Gamma\Big{(}\frac{\tilde{\Sigma}-d}{2}\Big{)}}\Bigg{[}\prod_{1\leqslant
i<j}^{4}P_{ij}^{l_{ij}}\Bigg{]}D_{\tilde{\Delta}_{1}\tilde{\Delta}_{2}\tilde{\Delta}_{3}\tilde{\Delta}_{4}}\
.$ (C.1)

It suffices to consider the inverse Mellin transform of

$\frac{\Gamma(\delta_{12}+l)}{\Gamma(\delta_{12})}\ .$ (C.2)

We first parameterize the delta functions in the inverse Mellin transform
(6.38)

$\delta(\Delta_{i}-\sum_{j}\delta_{ij})=\int_{0}^{\infty}\frac{dt_{i}}{t_{i}}t_{i}^{\Delta_{i}-\sum_{j}\delta_{ij}}$
(C.3)

and then evaluate the $\delta_{ij}$ contour integral using

$\int_{-i\infty}^{i\infty}\frac{d\delta_{ij}}{2\pi
i}\Gamma(\delta_{ij}+l)(t_{i}t_{j}P_{ij})^{-\delta_{ij}}=(t_{i}t_{j}P_{ij})^{l}e^{-t_{i}t_{j}P_{ij}}\
.$ (C.4)

This leads to

$\textbf{M}^{-1}\Bigg{[}\frac{\Gamma(\delta_{12}+l)}{\Gamma(\delta_{12})}\Bigg{]}=(P_{12})^{l}\int_{0}^{\infty}\Bigg{[}\prod_{i=1}^{4}\frac{dt_{i}}{t_{i}}\Bigg{]}t_{1}^{\Delta_{1}+l}t_{2}^{\Delta_{2}+l}t_{3}^{\Delta_{3}}t_{4}^{\Delta_{4}}\prod_{i<j}e^{-t_{i}t_{j}P_{ij}}\
.$ (C.5)

To convert this expression to a $D$-function, we first insert the identity
element

$\begin{split}1&=\frac{1}{\Gamma[(\Delta_{1}+l)/2+(\Delta_{2}+l)/2+\Delta_{3}/2+\Delta_{4}/2-d/2]}\\\
&\times\int_{0}^{\infty}\frac{dz}{z}z^{-d/2+(\Delta_{1}+l)/2+(\Delta_{2}+l)/2+\Delta_{3}/2+\Delta_{4}/2}e^{-z}\end{split}$
(C.6)

and then rescale $t_{i}\rightarrow t_{i}^{\prime}/\sqrt{z}$. Now we can carry
out the $z$ integral as

$\displaystyle\frac{\pi^{d/2}}{2}\int_{0}^{\infty}\frac{dz}{z}z^{-d/2}\exp\Big{[}-z+\frac{1}{z}\Big{(}\sum_{i}t^{\prime}_{i}P_{i}\Big{)}^{2}\Big{]}=\int_{\textrm{AdS}}dX\prod_{i}e^{2t_{i}^{\prime}X\cdot
P_{i}}\,.$ (C.7)

Finally, we perform the $t^{\prime}_{i}$ integral as

$\int_{0}^{\infty}\frac{dt^{\prime}_{i}}{t^{\prime}_{i}}(t^{\prime}_{i})^{\Delta^{\prime}_{i}}e^{t^{\prime}(2X\cdot
P_{i})}=\frac{\Gamma(\Delta^{\prime}_{i})}{(-2X\cdot
P_{i})^{\Delta^{\prime}_{i}}}$ (C.8)

to reach our desired result

$\textbf{M}^{-1}\Bigg{[}\frac{\Gamma(\delta_{12}+l)}{\Gamma(\delta_{12})}\Bigg{]}=\frac{2}{\pi^{d/2}}P_{1,2}^{l}\frac{\Gamma(\Delta_{1}+l)\Gamma(\Delta_{2}+l)\Gamma(\Delta_{3})\Gamma(\Delta_{4})D_{\Delta_{1}+l_{1},\Delta_{2}+l_{2},\Delta_{3},\Delta_{4}}}{\Gamma[(\Delta_{1}+l)/2+(\Delta_{2}+l)/2+\Delta_{3}/2+\Delta_{4}/2-d/2]}\
.$ (C.9)

It is trivial to see that eq. (C.9) generalizes to eq. (6.37).

## References

  * (1) J. M. Maldacena, _The Large N limit of superconformal field theories and supergravity_ , _Adv. Theor. Math. Phys._ 2 (1998) 231 [hep-th/9711200].
  * (2) S. S. Gubser, I. R. Klebanov and A. M. Polyakov, _Gauge theory correlators from noncritical string theory_ , _Phys. Lett. B_ 428 (1998) 105 [hep-th/9802109].
  * (3) E. Witten, _Anti-de Sitter space and holography_ , _Adv. Theor. Math. Phys._ 2 (1998) 253 [hep-th/9802150].
  * (4) J. Penedones, _Writing CFT correlation functions as AdS scattering amplitudes_ , _JHEP_ 03 (2011) 025 [1011.1485].
  * (5) L. Susskind, _The World as a hologram_ , _J. Math. Phys._ 36 (1995) 6377 [hep-th/9409089].
  * (6) G. ’t Hooft, _Dimensional reduction in quantum gravity_ , _Conf. Proc. C_ 930308 (1993) 284 [gr-qc/9310026].
  * (7) S. Caron-Huot, _Analyticity in Spin in Conformal Theories_ , _JHEP_ 09 (2017) 078 [1703.00278].
  * (8) D. Simmons-Duffin, D. Stanford and E. Witten, _A spacetime derivation of the Lorentzian OPE inversion formula_ , _JHEP_ 07 (2018) 085 [1711.03816].
  * (9) D. Harlow and H. Ooguri, _Symmetries in quantum field theory and quantum gravity_ , _Commun. Math. Phys._ 383 (2021) 1669 [1810.05338].
  * (10) L. F. Alday, C. Behan, P. Ferrero and X. Zhou, _Gluon Scattering in AdS from CFT_ , 2103.15830.
  * (11) A. Fayyazuddin and M. Spalinski, _Large N superconformal gauge theories and supergravity orientifolds_ , _Nucl. Phys. B_ 535 (1998) 219 [hep-th/9805096].
  * (12) O. Aharony, A. Fayyazuddin and J. M. Maldacena, _The Large N limit of N=2, N=1 field theories from three-branes in F theory_ , _JHEP_ 07 (1998) 013 [hep-th/9806159].
  * (13) N. Seiberg, _Five-dimensional SUSY field theories, nontrivial fixed points and string dynamics_ , _Phys. Lett. B_ 388 (1996) 753 [hep-th/9608111].
  * (14) O. J. Ganor and A. Hanany, _Small E(8) instantons and tensionless noncritical strings_ , _Nucl. Phys. B_ 474 (1996) 122 [hep-th/9602120].
  * (15) N. Seiberg and E. Witten, _Comments on string dynamics in six-dimensions_ , _Nucl. Phys. B_ 471 (1996) 121 [hep-th/9603003].
  * (16) A. Karch and E. Katz, _Adding flavor to AdS / CFT_ , _JHEP_ 06 (2002) 043 [hep-th/0205236].
  * (17) S. Hohenegger and I. Kirsch, _A Note on the holography of Chern-Simons matter theories with flavour_ , _JHEP_ 04 (2009) 129 [0903.1730].
  * (18) Y. Hikida, W. Li and T. Takayanagi, _ABJM with Flavors and FQHE_ , _JHEP_ 07 (2009) 065 [0903.2194].
  * (19) D. Gaiotto and D. L. Jafferis, _Notes on adding D6 branes wrapping RP**3 in AdS(4) x CP**3_ , _JHEP_ 11 (2012) 015 [0903.2175].
  * (20) R. H. Boels, B. A. Kniehl, O. V. Tarasov and G. Yang, _Color-kinematic Duality for Form Factors_ , _JHEP_ 02 (2013) 063 [1211.7028].
  * (21) O. T. Engelund and R. Roiban, _Correlation functions of local composite operators from generalized unitarity_ , _JHEP_ 03 (2013) 172 [1209.0227].
  * (22) D. Carmi, L. Di Pietro and S. Komatsu, _A Study of Quantum Field Theories in AdS at Finite Coupling_ , _JHEP_ 01 (2019) 200 [1810.04185].
  * (23) M. Mariño and T. Reis, _A new renormalon in two dimensions_ , _JHEP_ 07 (2020) 216 [1912.06228].
  * (24) S. Giombi and H. Khanchandani, _CFT in AdS and boundary RG flows_ , _JHEP_ 11 (2020) 118 [2007.04955].
  * (25) Z. Bern, J. Carrasco and H. Johansson, _New Relations for Gauge-Theory Amplitudes_ , _Phys. Rev. D_ 78 (2008) 085011 [0805.3993].
  * (26) Z. Bern, J. J. M. Carrasco and H. Johansson, _Perturbative Quantum Gravity as a Double Copy of Gauge Theory_ , _Phys. Rev. Lett._ 105 (2010) 061602 [1004.0476].
  * (27) Z. Bern, J. J. Carrasco, M. Chiodaroli, H. Johansson and R. Roiban, _The Duality Between Color and Kinematics and its Applications_ , 1909.01358.
  * (28) Z. Bern, T. Dennen, Y.-t. Huang and M. Kiermaier, _Gravity as the Square of Gauge Theory_ , _Phys. Rev. D_ 82 (2010) 065003 [1004.0693].
  * (29) Z. Bern, J. J. M. Carrasco, L. J. Dixon, H. Johansson and R. Roiban, _The Complete Four-Loop Four-Point Amplitude in N=4 Super-Yang-Mills Theory_ , _Phys. Rev. D_ 82 (2010) 125040 [1008.3327].
  * (30) J. J. M. Carrasco and H. Johansson, _Five-Point Amplitudes in N=4 Super-Yang-Mills Theory and N=8 Supergravity_ , _Phys. Rev. D_ 85 (2012) 025006 [1106.4711].
  * (31) Z. Bern, C. Boucher-Veronneau and H. Johansson, _N $>$= 4 Supergravity Amplitudes from Gauge Theory at One Loop_, _Phys. Rev. D_ 84 (2011) 105035 [1107.1935].
  * (32) C. Boucher-Veronneau and L. J. Dixon, _N $>$\- 4 Supergravity Amplitudes from Gauge Theory at Two Loops_, _JHEP_ 12 (2011) 046 [1110.1132].
  * (33) Z. Bern, J. J. M. Carrasco, L. J. Dixon, H. Johansson and R. Roiban, _Simplifying Multiloop Integrands and Ultraviolet Divergences of Gauge Theory and Gravity Amplitudes_ , _Phys. Rev. D_ 85 (2012) 105014 [1201.5366].
  * (34) Y.-J. Du and H. Luo, _On General BCJ Relation at One-loop Level in Yang-Mills Theory_ , _JHEP_ 01 (2013) 129 [1207.4549].
  * (35) S. Oxburgh and C. D. White, _BCJ duality and the double copy in the soft limit_ , _JHEP_ 02 (2013) 127 [1210.1110].
  * (36) E. Y. Yuan, _Virtual Color-Kinematics Duality: 6-pt 1-Loop MHV Amplitudes_ , _JHEP_ 05 (2013) 070 [1210.1816].
  * (37) R. H. Boels, R. S. Isermann, R. Monteiro and D. O’Connell, _Colour-Kinematics Duality for One-Loop Rational Amplitudes_ , _JHEP_ 04 (2013) 107 [1301.4165].
  * (38) N. E. J. Bjerrum-Bohr, T. Dennen, R. Monteiro and D. O’Connell, _Integrand Oxidation and One-Loop Colour-Dual Numerators in N=4 Gauge Theory_ , _JHEP_ 07 (2013) 092 [1303.2913].
  * (39) Z. Bern, S. Davies, T. Dennen, Y.-t. Huang and J. Nohle, _Color-Kinematics Duality for Pure Yang-Mills and Gravity at One and Two Loops_ , _Phys. Rev. D_ 92 (2015) 045041 [1303.6605].
  * (40) A. Ochirov and P. Tourkine, _BCJ duality and double copy in the closed string sector_ , _JHEP_ 05 (2014) 136 [1312.1326].
  * (41) C. R. Mafra and O. Schlotterer, _Two-loop five-point amplitudes of super Yang-Mills and supergravity in pure spinor superspace_ , _JHEP_ 10 (2015) 124 [1505.02746].
  * (42) G. Mogull and D. O’Connell, _Overcoming Obstacles to Colour-Kinematics Duality at Two Loops_ , _JHEP_ 12 (2015) 135 [1511.06652].
  * (43) S. He, R. Monteiro and O. Schlotterer, _String-inspired BCJ numerators for one-loop MHV amplitudes_ , _JHEP_ 01 (2016) 171 [1507.06288].
  * (44) G. Yang, _Color-kinematics duality and Sudakov form factor at five loops for N=4 supersymmetric Yang-Mills theory_ , _Phys. Rev. Lett._ 117 (2016) 271602 [1610.02394].
  * (45) Z. Bern, J. J. Carrasco, W.-M. Chen, H. Johansson and R. Roiban, _Gravity Amplitudes as Generalized Double Copies of Gauge-Theory Amplitudes_ , _Phys. Rev. Lett._ 118 (2017) 181602 [1701.02519].
  * (46) Z. Bern, J. J. M. Carrasco, W.-M. Chen, H. Johansson, R. Roiban and M. Zeng, _Five-loop four-point integrand of $N=8$ supergravity as a generalized double copy_, _Phys. Rev. D_ 96 (2017) 126012 [1708.06807].
  * (47) R. H. Boels, T. Huber and G. Yang, _Four-Loop Nonplanar Cusp Anomalous Dimension in N=4 Supersymmetric Yang-Mills Theory_ , _Phys. Rev. Lett._ 119 (2017) 201601 [1705.03444].
  * (48) S. He, O. Schlotterer and Y. Zhang, _New BCJ representations for one-loop amplitudes in gauge theories and gravity_ , _Nucl. Phys. B_ 930 (2018) 328 [1706.00640].
  * (49) H. Johansson and A. Ochirov, _Pure Gravities via Color-Kinematics Duality for Fundamental Matter_ , _JHEP_ 11 (2015) 046 [1407.4772].
  * (50) M. Chiodaroli, Q. Jin and R. Roiban, _Color/kinematics duality for general abelian orbifolds of N=4 super Yang-Mills theory_ , _JHEP_ 01 (2014) 152 [1311.3600].
  * (51) H. Johansson and A. Ochirov, _Color-Kinematics Duality for QCD Amplitudes_ , _JHEP_ 01 (2016) 170 [1507.00332].
  * (52) T. Bargheer, S. He and T. McLoughlin, _New Relations for Three-Dimensional Supersymmetric Scattering Amplitudes_ , _Phys. Rev. Lett._ 108 (2012) 231601 [1203.0562].
  * (53) A. Anastasiou, L. Borsten, M. J. Duff, M. J. Hughes, A. Marrani, S. Nagy et al., _Twin supergravities from Yang-Mills theory squared_ , _Phys. Rev. D_ 96 (2017) 026013 [1610.07192].
  * (54) Y.-t. Huang, H. Johansson and S. Lee, _On Three-Algebra and Bi-Fundamental Matter Amplitudes and Integrability of Supergravity_ , _JHEP_ 11 (2013) 050 [1307.2222].
  * (55) M. Chiodaroli, M. Gunaydin, H. Johansson and R. Roiban, _Complete construction of magical, symmetric and homogeneous N=2 supergravities as double copies of gauge theories_ , _Phys. Rev. Lett._ 117 (2016) 011603 [1512.09130].
  * (56) A. Anastasiou, L. Borsten, M. J. Duff, A. Marrani, S. Nagy and M. Zoccali, _Are all supergravity theories Yang–Mills squared?_ , _Nucl. Phys. B_ 934 (2018) 606 [1707.03234].
  * (57) M. Chiodaroli, M. Günaydin, H. Johansson and R. Roiban, _Non-Abelian gauged supergravities as double copies_ , _JHEP_ 06 (2019) 099 [1812.10434].
  * (58) C. Cheung, C.-H. Shen and C. Wen, _Unifying Relations for Scattering Amplitudes_ , _JHEP_ 02 (2018) 095 [1705.03025].
  * (59) F. Cachazo, S. He and E. Y. Yuan, _Scattering Equations and Matrices: From Einstein To Yang-Mills, DBI and NLSM_ , _JHEP_ 07 (2015) 149 [1412.3479].
  * (60) C. Cheung, G. N. Remmen, C.-H. Shen and C. Wen, _Pions as Gluons in Higher Dimensions_ , _JHEP_ 04 (2018) 129 [1709.04932].
  * (61) H. Elvang, M. Hadjiantonis, C. R. Jones and S. Paranjape, _Electromagnetic Duality and D3-Brane Scattering Amplitudes Beyond Leading Order_ , 2006.08928.
  * (62) G. Chen and Y.-J. Du, _Amplitude Relations in Non-linear Sigma Model_ , _JHEP_ 01 (2014) 061 [1311.1133].
  * (63) J. J. M. Carrasco, C. R. Mafra and O. Schlotterer, _Semi-abelian Z-theory: NLSM $+\phi^{3}$ from the open string_, _JHEP_ 08 (2017) 135 [1612.06446].
  * (64) S. Mizera and B. Skrzypek, _Perturbiner Methods for Effective Field Theories and the Double Copy_ , _JHEP_ 10 (2018) 018 [1809.02096].
  * (65) L. de la Cruz, A. Kniss and S. Weinzierl, _Proof of the fundamental BCJ relations for QCD amplitudes_ , _JHEP_ 09 (2015) 197 [1508.01432].
  * (66) R. W. Brown and S. G. Naculich, _KLT-type relations for QCD and bicolor amplitudes from color-factor symmetry_ , _JHEP_ 03 (2018) 057 [1802.01620].
  * (67) R. W. Brown and S. G. Naculich, _Color-factor symmetry and BCJ relations for QCD amplitudes_ , _JHEP_ 11 (2016) 060 [1608.05291].
  * (68) S. G. Naculich, _Amplitudes for massive vector and scalar bosons in spontaneously-broken gauge theory from the CHY representation_ , _JHEP_ 09 (2015) 122 [1506.06134].
  * (69) L. A. Johnson, C. R. Jones and S. Paranjape, _Constraints on a Massive Double-Copy and Applications to Massive Gravity_ , 2004.12948.
  * (70) J. Plefka, C. Shi and T. Wang, _Double copy of massive scalar QCD_ , _Phys. Rev. D_ 101 (2020) 066004 [1911.06785].
  * (71) H. Johansson and A. Ochirov, _Double copy for massive quantum particles with spin_ , _JHEP_ 09 (2019) 040 [1906.12292].
  * (72) S. G. Naculich, _Scattering equations and BCJ relations for gauge and gravitational amplitudes with massive scalar particles_ , _JHEP_ 09 (2014) 029 [1407.7836].
  * (73) J. J. M. Carrasco and I. A. Vazquez-Holm, _Loop-Level Double-Copy for Massive Quantum Particles_ , 2010.13435.
  * (74) A. Momeni, J. Rumbutis and A. J. Tolley, _Massive Gravity from Double Copy_ , _JHEP_ 12 (2020) 030 [2004.07853].
  * (75) K. Haddad and A. Helset, _The double copy for heavy particles_ , _Phys. Rev. Lett._ 125 (2020) 181603 [2005.13897].
  * (76) A. Momeni, J. Rumbutis and A. J. Tolley, _Kaluza-Klein from Colour-Kinematics Duality for Massive Fields_ , 2012.09711.
  * (77) J. A. Farrow, A. E. Lipstein and P. McFadden, _Double copy structure of CFT correlators_ , _JHEP_ 02 (2019) 130 [1812.11129].
  * (78) S. Albayrak and S. Kharel, _Towards the higher point holographic momentum space amplitudes_ , _JHEP_ 02 (2019) 040 [1810.12459].
  * (79) A. Bzowski, P. McFadden and K. Skenderis, _Conformal $n$-point functions in momentum space_, _Phys. Rev. Lett._ 124 (2020) 131602 [1910.10162].
  * (80) S. Albayrak and S. Kharel, _Towards the higher point holographic momentum space amplitudes. Part II. Gravitons_ , _JHEP_ 12 (2019) 135 [1908.01835].
  * (81) A. E. Lipstein and P. McFadden, _Double copy structure and the flat space limit of conformal correlators in even dimensions_ , _Phys. Rev. D_ 101 (2020) 125006 [1912.10046].
  * (82) S. Albayrak, C. Chowdhury and S. Kharel, _Study of momentum space scalar amplitudes in AdS spacetime_ , _Phys. Rev. D_ 101 (2020) 124043 [2001.06777].
  * (83) G. Mack, _D-dimensional Conformal Field Theories with anomalous dimensions as Dual Resonance Models_ , _Bulg. J. Phys._ 36 (2009) 214 [0909.1024].
  * (84) G. Mack, _D-independent representation of Conformal Field Theories in D dimensions via transformation to auxiliary Dual Resonance Models. Scalar amplitudes_ , 0907.2407.
  * (85) M. F. Paulos, _Towards Feynman rules for Mellin amplitudes_ , _JHEP_ 10 (2011) 074 [1107.1504].
  * (86) A. L. Fitzpatrick, J. Kaplan, J. Penedones, S. Raju and B. C. van Rees, _A Natural Language for AdS/CFT Correlators_ , _JHEP_ 11 (2011) 095 [1107.1499].
  * (87) S. Kharel and G. Siopsis, _Tree-level Correlators of scalar and vector fields in AdS/CFT_ , _JHEP_ 11 (2013) 159 [1308.2515].
  * (88) L. Rastelli and X. Zhou, _Mellin amplitudes for $AdS_{5}\times S^{5}$_, _Phys. Rev. Lett._ 118 (2017) 091602 [1608.06624].
  * (89) J. Penedones, J. A. Silva and A. Zhiboedov, _Nonperturbative Mellin Amplitudes: Existence, Properties, Applications_ , _JHEP_ 08 (2020) 031 [1912.11100].
  * (90) C. Sleight and M. Taronna, _On the consistency of (partially-)massless matter couplings in de Sitter space_ , 2106.00366.
  * (91) C. Sleight and M. Taronna, _Bootstrapping Inflationary Correlators in Mellin Space_ , _JHEP_ 02 (2020) 098 [1907.01143].
  * (92) S. Albayrak, S. Kharel and D. Meltzer, _On duality of color and kinematics in (A)dS momentum space_ , _JHEP_ 03 (2021) 249 [2012.10460].
  * (93) C. Armstrong, A. E. Lipstein and J. Mei, _Color/kinematics duality in AdS 4_, _JHEP_ 02 (2021) 194 [2012.02059].
  * (94) X. Zhou, _Double Copy Relation for AdS_ , 2106.07651.
  * (95) C. Sleight and M. Taronna, _Spinning Mellin Bootstrap: Conformal Partial Waves, Crossing Kernels and Applications_ , _Fortsch. Phys._ 66 (2018) 1800038 [1804.09334].
  * (96) X. Zhou, _On Superconformal Four-Point Mellin Amplitudes in Dimension $d>2$_, _JHEP_ 08 (2018) 187 [1712.02800].
  * (97) L. Rastelli and X. Zhou, _How to Succeed at Holographic Correlators Without Really Trying_ , _JHEP_ 04 (2018) 014 [1710.05923].
  * (98) L. F. Alday and X. Zhou, _All Holographic Four-Point Functions in All Maximally Supersymmetric CFTs_ , _Phys. Rev. X_ 11 (2021) 011056 [2006.12505].
  * (99) L. F. Alday and X. Zhou, _All Tree-Level Correlators for M-theory on $AdS_{7}\times S^{4}$_, _Phys. Rev. Lett._ 125 (2020) 131604 [2006.06653].
  * (100) P. A. M. Dirac, _Wave equations in conformal space_ , _Annals of Mathematics_ 37 (1936) 429.
  * (101) S. Ferrara, A. F. Grillo and R. Gatto, _Tensor representations of conformal algebra and conformally covariant operator product expansion_ , _Annals Phys._ 76 (1973) 161.
  * (102) V. de Alfaro, S. Fubini and G. Furlan, _Conformal Invariance in Quantum Mechanics_ , _Nuovo Cim. A_ 34 (1976) 569.
  * (103) S. Weinberg, _Six-dimensional Methods for Four-dimensional Conformal Field Theories_ , _Phys. Rev. D_ 82 (2010) 045031 [1006.3480].
  * (104) M. S. Costa, J. Penedones, D. Poland and S. Rychkov, _Spinning Conformal Correlators_ , _JHEP_ 11 (2011) 071 [1107.3554].
  * (105) E. Casali and A. Puhm, _Double Copy for Celestial Amplitudes_ , _Phys. Rev. Lett._ 126 (2021) 101602 [2007.15027].
  * (106) E. Casali and A. Sharma, _Celestial double copy from the worldsheet_ , _JHEP_ 05 (2021) 157 [2011.10052].
  * (107) N. Kalyanapuram, _Gauge and Gravity Amplitudes on the Celestial Sphere_ , _Phys. Rev. D_ 103 (2021) 085015 [2012.04579].
  * (108) K. Roehrig and D. Skinner, _Ambitwistor Strings and the Scattering Equations on AdS ${}_{3}\times$S3_, 2007.07234.
  * (109) L. Mason and D. Skinner, _Ambitwistor strings and the scattering equations_ , _JHEP_ 07 (2014) 048 [1311.2564].
  * (110) N. Berkovits, _Infinite Tension Limit of the Pure Spinor Superstring_ , _JHEP_ 03 (2014) 017 [1311.4156].
  * (111) T. Adamo, E. Casali and D. Skinner, _Ambitwistor strings and the scattering equations at one loop_ , _JHEP_ 04 (2014) 104 [1312.3828].
  * (112) E. Casali, Y. Geyer, L. Mason, R. Monteiro and K. A. Roehrig, _New Ambitwistor String Theories_ , _JHEP_ 11 (2015) 038 [1506.08771].
  * (113) L. Eberhardt, S. Komatsu and S. Mizera, _Scattering equations in AdS: scalar correlators in arbitrary dimensions_ , _JHEP_ 11 (2020) 158 [2007.06574].
  * (114) L. Eberhardt, S. Komatsu and S. Mizera, Work in Progress; Talk at QCD meets Gravity VI.
  * (115) F. A. Dolan and H. Osborn, _Conformal partial waves and the operator product expansion_ , _Nucl. Phys. B_ 678 (2004) 491 [hep-th/0309180].
  * (116) I. Buric, S. Lacroix, J. Mann, L. Quintavalle and V. Schomerus, _Gaudin Models and Multipoint Conformal Blocks: General Theory_ , 2105.00021.
  * (117) I. Buric, S. Lacroix, J. A. Mann, L. Quintavalle and V. Schomerus, _From Gaudin Integrable Models to $d$-dimensional Multipoint Conformal Blocks_, _Phys. Rev. Lett._ 126 (2021) 021602 [2009.11882].
  * (118) V. Del Duca, L. J. Dixon and F. Maltoni, _New color decompositions for gauge amplitudes at tree and loop level_ , _Nucl. Phys. B_ 571 (2000) 51 [hep-ph/9910563].
  * (119) F. Cachazo, S. He and E. Y. Yuan, _Scattering of Massless Particles: Scalars, Gluons and Gravitons_ , _JHEP_ 07 (2014) 033 [1309.0885].
  * (120) R. Kleiss and H. Kuijf, _Multi - Gluon Cross-sections and Five Jet Production at Hadron Colliders_ , _Nucl. Phys. B_ 312 (1989) 616.
  * (121) M. S. Costa, T. Hansen, J. a. Penedones and E. Trevisani, _Projectors and seed conformal blocks for traceless mixed-symmetry tensors_ , _JHEP_ 07 (2016) 018 [1603.05551].
  * (122) J. Penedones, _TASI lectures on AdS/CFT_ , in _Theoretical Advanced Study Institute in Elementary Particle Physics: New Frontiers in Fields and Strings_ , 8, 2016, DOI [1608.04948].
  * (123) S. Giombi, C. Sleight and M. Taronna, _Spinning AdS Loop Diagrams: Two Point Functions_ , _JHEP_ 06 (2018) 030 [1708.08404].
  * (124) J. J. M. Carrasco, C. R. Mafra and O. Schlotterer, _Abelian Z-theory: NLSM amplitudes and $\alpha$’-corrections from the open string_, _JHEP_ 06 (2017) 093 [1608.02569].
  * (125) D. Z. Freedman, S. D. Mathur, A. Matusis and L. Rastelli, _Correlation functions in the CFT(d) / AdS(d+1) correspondence_ , _Nucl. Phys. B_ 546 (1999) 96 [hep-th/9804058].
  * (126) I. R. Klebanov and E. Witten, _AdS / CFT correspondence and symmetry breaking_ , _Nucl. Phys. B_ 556 (1999) 89 [hep-th/9905104].
  * (127) M. Gunaydin and N. Marcus, _The Unitary Supermultiplet of $N=8$ Conformal Superalgebra Involving Fields of Spin $\leq 2$_, _Class. Quant. Grav._ 2 (1985) L19.
  * (128) M. Gunaydin, D. Minic and M. Zagermann, _4D doubleton conformal theories, CPT and IIB string on AdS ${}_{5}\times$ S5_, _Nucl. Phys. B_ 534 (1998) 96 [hep-th/9806042].
  * (129) M. Grinberg and J. Maldacena, _Proper time to the black hole singularity from thermal one-point functions_ , _JHEP_ 03 (2021) 131 [2011.01004].
  * (130) Y.-J. Du and C.-H. Fu, _Explicit BCJ numerators of nonlinear simga model_ , _JHEP_ 09 (2016) 174 [1606.05846].
  * (131) Y.-J. Du and F. Teng, _BCJ numerators from reduced Pfaffian_ , _JHEP_ 04 (2017) 033 [1703.05717].
  * (132) G. Chalmers, H. Nastase, K. Schalm and R. Siebelink, _R current correlators in N=4 superYang-Mills theory from anti-de Sitter supergravity_ , _Nucl. Phys. B_ 540 (1999) 247 [hep-th/9805105].
  * (133) M. S. Costa, V. Goncalves and J. Penedones, _Spinning AdS Propagators_ , _JHEP_ 09 (2014) 064 [1404.5625].
  * (134) E. D’Hoker, D. Z. Freedman, S. D. Mathur, A. Matusis and L. Rastelli, _Graviton exchange and complete four point functions in the AdS / CFT correspondence_ , _Nucl. Phys. B_ 562 (1999) 353 [hep-th/9903196].
  * (135) A. Zhiboedov, _A note on three-point functions of conserved currents_ , 1206.6370.
  * (136) E. D’Hoker, D. Z. Freedman and L. Rastelli, _AdS / CFT four point functions: How to succeed at z integrals without really trying_ , _Nucl. Phys. B_ 562 (1999) 395 [hep-th/9905049].
  * (137) S. Caron-Huot and Y.-Z. Li, _Helicity basis for three-dimensional conformal field theory_ , 2102.08160.
  * (138) H.-H. Chi, H. Elvang, A. Herderschee, C. R. T. Jones and S. Paranjape, _Generalizations of the Double-Copy: the KLT Bootstrap_ , 2106.12600.
  * (139) J. Broedel and L. J. Dixon, _Color-kinematics duality and double-copy construction for amplitudes from higher-dimension operators_ , _JHEP_ 10 (2012) 091 [1208.0876].
  * (140) M. R. Gaberdiel and R. Gopakumar, _The Worldsheet Dual of Free Super Yang-Mills in 4D_ , 2105.10496.
  * (141) M. R. Gaberdiel and R. Gopakumar, _The String Dual to Free ${\cal N}=4$ Super Yang-Mills_, 2104.08263.
  * (142) M. F. Paulos, J. Penedones, J. Toledo, B. C. van Rees and P. Vieira, _The S-matrix bootstrap. Part I: QFT in AdS_ , _JHEP_ 11 (2017) 133 [1607.06109].
  * (143) Y.-Z. Li, _Notes on flat-space limit of AdS/CFT_ , 2106.04606.
  * (144) A. Bzowski, P. McFadden and K. Skenderis, _Conformal correlators as simplex integrals in momentum space_ , _JHEP_ 01 (2021) 192 [2008.07543].
  * (145) S. Jain, R. Rajan, A. Mehta, A. A. Nizami and A. Suresh, _Double copy structure of parity-violating CFT correlators_ , 2104.12803.
  * (146) J. Polchinski and M. J. Strassler, _Hard scattering and gauge / string duality_ , _Phys. Rev. Lett._ 88 (2002) 031601 [hep-th/0109174].
  * (147) R. C. Brower, J. Polchinski, M. J. Strassler and C.-I. Tan, _The Pomeron and gauge/string duality_ , _JHEP_ 12 (2007) 005 [hep-th/0603115].
  * (148) C. G. Callan and F. Wilczek, _Infrared behavior at negative curvature_ , _Nuclear Physics B_ 340 (1990) 366.
  * (149) O. Aharony, M. Berkooz, D. Tong and S. Yankielowicz, _Confinement in Anti-de Sitter Space_ , _JHEP_ 02 (2013) 076 [1210.5195].
  * (150) O. Aharony, M. Berkooz, A. Karasik and T. Vaknin, _Supersymmetric field theories on AdS ${}_{p}\times$ Sq_, _JHEP_ 04 (2016) 066 [1512.04698].
  * (151) V. P. Nair, _A Current Algebra for Some Gauge Theory Amplitudes_ , _Phys. Lett. B_ 214 (1988) 215.
  * (152) A. Herderschee, S. Koren and T. Trott, _Massive On-Shell Supersymmetric Scattering Amplitudes_ , _JHEP_ 10 (2019) 092 [1902.07204].
  * (153) A. Herderschee, S. Koren and T. Trott, _Constructing $\mathcal{N}$ = 4 Coulomb branch superamplitudes_, _JHEP_ 08 (2019) 107 [1902.07205].
  * (154) T. Dennen, Y.-t. Huang and W. Siegel, _Supertwistor space for 6D maximal super Yang-Mills_ , _JHEP_ 04 (2010) 127 [0910.2688].
  * (155) R. H. Boels and D. O’Connell, _Simple superamplitudes in higher dimensions_ , _JHEP_ 06 (2012) 163 [1201.2653].
  * (156) N. Craig, H. Elvang, M. Kiermaier and T. Slatyer, _Massive amplitudes on the Coulomb branch of N=4 SYM_ , _JHEP_ 12 (2011) 097 [1104.2050].
  * (157) N. Arkani-Hamed, D. Baumann, H. Lee and G. L. Pimentel, _The Cosmological Bootstrap: Inflationary Correlators from Symmetries and Singularities_ , _JHEP_ 04 (2020) 105 [1811.00024].
  * (158) D. Harlow and D. Stanford, _Operator Dictionaries and Wave Functions in AdS/CFT and dS/CFT_ , 1104.2621.
  * (159) C. Sleight and M. Taronna, _From AdS to dS Exchanges: Spectral Representation, Mellin Amplitudes and Crossing_ , 2007.09993.
  * (160) C. Sleight, _A Mellin Space Approach to Cosmological Correlators_ , _JHEP_ 01 (2020) 090 [1906.12302].
  * (161) A. Strominger, _Lectures on the Infrared Structure of Gravity and Gauge Theory_ , 1703.05448.
  * (162) D. Kapec, V. Lysov, S. Pasterski and A. Strominger, _Semiclassical Virasoro symmetry of the quantum gravity $\mathcal{S}$-matrix_, _JHEP_ 08 (2014) 058 [1406.3312].
  * (163) S. Pasterski, A. Strominger and A. Zhiboedov, _New Gravitational Memories_ , _JHEP_ 12 (2016) 053 [1502.06120].
  * (164) N. Arkani-Hamed, M. Pate, A.-M. Raclariu and A. Strominger, _Celestial Amplitudes from UV to IR_ , 2012.04208.
  * (165) S. He and Y. Zhang, _New Formulas for Amplitudes from Higher-Dimensional Operators_ , _JHEP_ 02 (2017) 019 [1608.08448].
  * (166) J. J. M. Carrasco, L. Rodina and S. Zekioglu, _Composing Effective Prediction at Five Points_ , 2104.08370.
  * (167) J. J. M. Carrasco, L. Rodina, Z. Yin and S. Zekioglu, _Simple encoding of higher derivative gauge and gravity counterterms_ , _Phys. Rev. Lett._ 125 (2020) 251602 [1910.12850].
  * (168) A. Denner, U. Nierste and R. Scharf, _A Compact expression for the scalar one loop four point function_ , _Nucl. Phys. B_ 367 (1991) 637.
  * (169) Z. Bern, L. J. Dixon and D. A. Kosower, _Dimensionally regulated pentagon integrals_ , _Nucl. Phys. B_ 412 (1994) 751 [hep-ph/9306240].
  * (170) M. Bianchi, M. B. Green, S. Kovacs and G. Rossi, _Instantons in supersymmetric Yang-Mills and D instantons in IIB superstring theory_ , _JHEP_ 08 (1998) 013 [hep-th/9807033].
  * (171) M. Borinsky, _Tropical Monte Carlo quadrature for Feynman integrals_ , 2008.12310.