# Adversarial Robust Decision Transformer: Enhancing Robustness of RvS via
Minimax Returns-to-go

Xiaohang Tang
University College London
<EMAIL_ADDRESS>&Afonso Marques∗
University College London
<EMAIL_ADDRESS>&Parameswaran Kamalaruban∗
Featurespace
<EMAIL_ADDRESS>&Ilija Bogunovic
University College London
<EMAIL_ADDRESS>Equal contribution. Codebase:
https://github.com/xiaohangt/ardt/tree/master

###### Abstract

Decision Transformer (DT), as one of the representative Reinforcement Learning
via Supervised Learning (RvS) methods, has achieved strong performance in
offline learning tasks by leveraging the powerful Transformer architecture for
sequential decision-making. However, in adversarial environments, these
methods can be non-robust, since the return is dependent on the strategies of
both the decision-maker and adversary. Training a probabilistic model
conditioned on observed return to predict action can fail to generalize, as
the trajectories that achieve a return in the dataset might have done so due
to a weak and suboptimal behavior adversary. To address this, we propose a
worst-case-aware RvS algorithm, the Adversarial Robust Decision Transformer
(ARDT), which learns and conditions the policy on in-sample minimax returns-
to-go. ARDT aligns the target return with the worst-case return learned
through minimax expectile regression, thereby enhancing robustness against
powerful test-time adversaries. In experiments conducted on sequential games
with full data coverage, ARDT can generate a maximin (Nash Equilibrium)
strategy, the solution with the largest adversarial robustness. In large-scale
sequential games and continuous adversarial RL environments with partial data
coverage, ARDT demonstrates significantly superior robustness to powerful
test-time adversaries and attains higher worst-case returns compared to
contemporary DT methods.

##  1 Introduction

Reinforcement Learning via Supervised Learning (RvS) (Emmons et al., 2021),
has garnered attention within the domain of offline Reinforcement Learning
(RL). Framing RL as a problem of outcome-conditioned sequence modeling, these
methodologies have showcased strong performance in offline benchmark tasks (Wu
et al., 2024; Yamagata et al., 2023; Janner et al., 2021; Schmidhuber, 2019;
Kumar et al., 2019). One of the representative RvS methods is Decision
Transformer (DT) (Chen et al., 2021), which simply trains a policy conditioned
on target return via behavior cloning loss. Given the apparent efficacy and
simplicity of RvS, our attention is drawn to exploring its performance in
adversarial settings. In this paper, we aim to use RvS to achieve _adversarial
robustness_ , a critical capability for RL agents to manage environmental
variations and adversarial disturbances (Pinto et al., 2017; Tessler et al.,
2019; Kamalaruban et al., 2020; Vinitsky et al., 2020; Curi et al., 2021;
Rigter et al., 2022). Such challenges are prevalent in real-world scenarios,
e.g., changes in road conditions and decision-making among multiple autonomous
vehicles.

In offline RL with adversarial actions, the naive training of action-
prediction models conditioned on the history and observed returns such as DT,
or expected returns such as ESPER and DoC (Paster et al., 2022; Yang et al.,
2022; Yamagata et al., 2023), may result in a non-robust policy. This issue
arises from the potential distributional shifts in the policy of the adversary
during offline learning. Specifically, if the behavior policies used for data
collection are suboptimal, a well-trained conditional policy overfitted to the
training dataset may fail to achieve the desired target return during testing,
if the adversary’s policy at test time has changed or become more effective.

We illustrate this using a sequential game with data collected by a uniform
random policy with full coverage shown in Figure 1. The standard RvS methods
(DT and ERC-DT) when conditioned on high target return, show poor worst-case
performance. DT’s failure stems from training data containing the sequences
$(a_{0},\bar{a}_{2})$ and $(a_{1},\bar{a}_{3})$, inaccurately indicating both
actions $a_{0}$ and $a_{1}$ as optimal. ERC-DT missteps because the
adversary’s uniform random policy skews the expected return in favor of
$a_{0}$ over $a_{1}$, wrongly favoring $a_{0}$. A robust strategy, however,
exclusively selects $a_{1}$ against an optimal adversary to guarantee a high
reward. Moreover, in more complex multi-decision scenarios, the adversarial
strategy should not only minimize but also counterbalance with maximized
returns by the decision-maker at the subsequent state, aligning with a
_minimax_ return approach for robust policy development.

Main Contributions. Consequently, there is a need for data curation to
indicate the potential worst-case returns associated with each action in the
Decision Transformer (DT), improving the robustness against adversarial
actions. Building on this concept, we introduce the first robust training
algorithm for Decision Transformer, the Adversarial Robust Decision
Transformer (ARDT). To the best of our knowledge, this paper represents the
first exploration of the robustness of RvS methods from an _adversarial_
perspective. In ARDT, we leverage Expectile Regression (Newey and Powell,
1987; Aigner et al., 1976) to transform the original returns-to-go to in-
sample minimax returns-to-go, and train DT with the relabeled returns. In the
experiments, we illustrate the robustness of ARDT in three settings (i) Games
with full data coverage (ii) A long-horizon discrete game with partial data
coverage (iii) Realistic continuous adversarial RL problems. We provide
evidence showcasing the robustness of ARDT to more powerful adversary compared
to the behavior policy and the distributional shifts of the adversary’s policy
in the test time. Furthermore, in the adversarial MuJoCo tasks, namely Noisy
Action Robust MDP (Tessler et al., 2019), ARDT exhibits better robustness to a
population of adversarial perturbations compared to existing DT methods.

Algorithms | Policy ($\hat{R}=6$): (${P}(a_{0})$, ${P}(a_{1})$)
---|---
DT | (0.5400, 0.4600)
ERC-DT | (0.9998, 0.0002)
ARDT (ours) | (0.0006, 0.9994)
Algorithms | Expected Worst-case Return
---|---
DT | 0.4600
ERC-DT | 0.0002
ARDT (ours) | 0.9994

Figure 1:  LHS presents the game where decision-maker $P$ is confronted by
adversary $A$. In the worst-case scenario, if $P$ chooses action $a_{0}$, $A$
will respond with $\bar{a}_{0}$, and if $P$ chooses $a_{1}$, $A$ will counter
with $\bar{a}_{4}$. Consequently, the worst-case returns for actions $a_{0}$
and $a_{1}$ are $0$ and $1$, respectively. Therefore, the robust choice of
action for the decision-maker is $a_{1}$. RHS displays tables of action
probabilities and the worst-case returns for the Decision Transformer (DT),
Expected Return-Conditioned DT (ERC-DT) methods and our algorithm, when
conditioned on the largest return-to-go $6$. After training using uniformly
collected data that covered all possible trajectories, the results reveal that
DT fails to select the robust action $a_{1}$, whereas our algorithm manages to
do so.

Related work. Reinforcement Learning via Supervised Learning is a promising
direction for solving general offline RL problems by converting the RL problem
into the outcome-conditioned prediction problem (Kumar et al., 2019;
Schmidhuber, 2019; Ghosh et al., 2019; Paster et al., 2020; Emmons et al.,
2021). Decision Transformer (DT) is one of the RvS instance (Chen et al.,
2021; Kim et al., 2023), leveraging the strength of the sequence model
Transformer (Vaswani et al., 2017). The variants of DT have been extensively
investigated to address the trajectory stitching (Wu et al., 2024; Yamagata et
al., 2023), stochasticity (Paster et al., 2020; Ortega et al., 2021; Paster et
al., 2022; Yang et al., 2022), goal-conditioned problems (Furuta et al.,
2021), and generalization in different environments (Meng et al., 2021; Xu et
al., 2022; Lee et al., 2022; Xu et al., 2023). For different problems, it is
critical to select an appropriate target as the condition of the policy.
$Q$-Learning DT (Yamagata et al., 2023) relabels the trajectories with the
estimated $Q$-values learned to achieve trajectory stitching (Kumar et al.,
2020). ESPER (Paster et al., 2022), aims at solving the environmental
stochasticity via relabeling the trajectories with expected returns-to-go.
Although our method also transform the returns-to-go to estimated values, we
instead associate worst-case returns-to-go with trajectories, aiming at
improving its robustness against adversarial perturbations.

Adversarial RL can be framed as a sequential game; with online game-solving
explored extensively in literature (Lanctot et al., 2009; Heinrich et al.,
2015; Sunehag et al., 2017; Lowe et al., 2017; Lanctot et al., 2017;
Srinivasan et al., 2018; Schrittwieser et al., 2020; Tang et al., 2023). In
the offline setting, where there is no interaction with the environment,
solving games is challenging due to distributional shifts in the adversary’s
policy, necessitating adequate data coverage. Tabular methods, such as Nash
$Q$-Learning—a decades-old approach for decision-making in competitive games
(Hu and Wellman, 2003)—and other similar methods (Cui and Du, 2022; Zhong et
al., 2022) address these challenges through pessimistic out-of-sample
estimation. While value-based methods are viable for addressing adversarial
problems, our focus is on enhancing the robustness of RvS methods in these
settings. Note that while some offline multi-agent RL methods, including DT-
based approaches, are designed for cooperative games (Meng et al., 2021; Wen
et al., 2022; Tseng et al., 2022; Wang et al., 2024), our research is oriented
towards competitive games.

##  2 Problem Setup

We consider the adversarial reinforcement learning problem involving a
protagonist and an adversary, as described by Pinto et al. (2017). This
setting is rooted in the two-player Zero-Sum Markov Game framework Littman
(1994); Perolat et al. (2015). This game is formally defined by the tuple
$(\mathcal{S},\mathcal{A},\bar{\mathcal{A}},T,R,p_{0})$. Here, $\mathcal{S}$
represents the state space, where $\mathcal{A}$ and $\bar{\mathcal{A}}$ denote
the action spaces for the protagonist and adversary, respectively. The reward
function $R:\mathcal{S}\times\mathcal{A}\times\bar{\mathcal{A}}\to\mathbb{R}$
and the transition kernel
$T:\mathcal{S}\times\mathcal{A}\times\bar{\mathcal{A}}\to\Delta(\mathcal{S})$
depend on the state and the joint actions of both players. The initial state
distribution is denoted by $p_{0}$. At each time-step $t\leq H$, both players
observe the state $s_{t}\in\mathcal{S}$ and take actions $a_{t}\in\mathcal{A}$
and $\bar{a}_{t}\in\bar{\mathcal{A}}$. The adversary is considered adaptive if
its action $\bar{a}_{t}$ depends on $s_{t}$ and $a_{t}$. The protagonist
receives a reward $r_{t}=R(s_{t},a_{t},\bar{a}_{t})$, while the adversary
receives a negative reward $-r_{t}$.

Denote trajectory
$\tau=\left({s_{t},a_{t},\bar{a}_{t},r_{t}}\right)_{t=0}^{H}$ and its sub-
trajectory $\tau_{i:j}=\left({s_{t},a_{t},\bar{a}_{t},r_{t}}\right)_{t=i}^{j}$
for $0\leq i\leq j\leq H$. The return-to-go refers to the sum of observed
rewards from state $s_{t}$ onwards:
$\widehat{R}(\tau_{t:H})=\sum_{t^{\prime}=t}^{H}r_{t^{\prime}}$. The
protagonist policy $\pi$ and the adversary policy $\bar{\pi}$ can be history-
dependent, mapping history $(\tau_{0:t-1},s_{t})$ to the distribution over
protagonist and adversary’s actions, respectively.

In offline RL, direct interaction with the environment is unavailable.
Instead, we rely on an offline dataset $\mathcal{D}$, which consists of
trajectories generated by executing a pair of behavioral policies,
$(\pi_{\mathcal{D}},\bar{\pi}_{\mathcal{D}})$. Our objective in this
adversarial offline RL setting is to utilize this dataset $\mathcal{D}$ to
learn a protagonist policy that seeks to maximize the return, while being
counteracted by an adversary’s policy $\bar{\pi}$:

$\displaystyle\max_{{\pi}}\min_{\bar{\pi}}\mathbb{E}_{\tau\sim\rho^{\pi,\bar{\pi}}}\\!\left[{\sum_{t}r_{t}}\right],$
(1)

where
$\rho^{\pi,\bar{\pi}}(\tau)=p_{0}(s_{0})\cdot\prod_{t}\pi(a_{t}\mid\tau_{0:t-1},s_{t})\cdot\bar{\pi}(\bar{a}_{t}\mid\tau_{0:t-1},s_{t},a_{t})\cdot
T(s_{t+1}\mid s_{t},a_{t},\bar{a}_{t})$. The maximin solution to this problem
and its corresponding optimal adversary are denoted as $\pi^{*}$ and
$\bar{\pi}^{*}$, respectively.

###  2.1 RvS in Adversarial Environments

We extend the RvS framework by incorporating adversarial settings within the
Hindsight Information Matching (HIM) paradigm (Furuta et al., 2021). In
adversarial HIM, the protagonist receives a goal (denoted by $z$) and acts
within an adversarial environment. The protagonist’s performance is evaluated
based on how well it achieves the presented goal. Mathematically, the
protagonist aims to minimize a distance metric $D(\cdot,\cdot)$ between the
target goal $z$ and the information function $I(\cdot)$ applied to a
trajectory $\tau$. This sub-trajectory follows a distribution $\rho$ dependent
on both the protagonist’s policy $\pi_{z}=\pi(a_{t}\mid\tau_{0:t-1},s_{t},z)$
and the test-time adversarial policy $\bar{\pi}_{\textnormal{test}}$.
Formally, the information distance is minimized as follows:

$\min_{\pi}\mathbb{E}_{\tau\sim\rho^{\pi_{z},\bar{\pi}_{\textnormal{test}}}}\\!\left[{D(I(\tau),z)}\right],$
(2)

where $\rho^{\pi_{z},\bar{\pi}_{\textnormal{test}}}$ represents the trajectory
distribution obtained by executing rollouts in a Markov Game with the
protagonist’s policy $\pi_{z}$ and the test-time adversarial policy
$\bar{\pi}_{\textnormal{test}}$. Under HIM framework, Decision Transformer
Chen et al. (2021) uses adopt return-to-go value as the information function,
while ESPER Paster et al. (2022) employs the expected return-to-go. The well-
trained RvS policy, including DT and ESPER, is derived from behavior
trajectories filtered based on the condition $I(\tau)=z$:

$\displaystyle\pi_{z}=\pi(a_{t}\mid\tau_{0:t-1},s_{t},z)=\rho^{\pi_{\mathcal{D}},\bar{\pi}_{\mathcal{D}}}(a_{t}\mid\tau_{0:t-1},s_{t},I(\tau)=z).$
(3)

If the conditional policy ensures that the information distance in Eq. (2) is
minimized to $0$, the generated policy $\pi_{z}$ is robust against
$\bar{\pi}_{\text{test}}$ by simply conditioning on a large target return, or
even the maximin optimal protagonist
$\pi^{*}=\lim_{z\rightarrow+\infty}\pi_{z}$ if the test-time adversary is
optimal. Consequently, our objective shifts from learning a maximin policy in
Eq. (1) to minimizing the distance in Eq. (2).

For a given target goal $z$, the trajectories where the information function
$I(\tau)=z$ can be considered as optimal demonstrations of achieving that
goal, i.e. information distance minimization, $D(I(\tau),z)=0$. However, in
adversarial environments, this information distance is hard to be minimized to
$0$. In adversarial offline RL, a trajectory in the dataset might achieve a
goal due to the presence of a weak behavior adversary, rather than through the
protagonist’s effective actions. Filtering datasets for trajectories with high
return-to-go could select the ones from encounters with suboptimal
adversaries. Consequently, an agent trained through behavioral cloning on
these trajectories is not guaranteed to achieve a high return when facing a
different and powerful adversary at test time. We show this formally through a
theorem:

######  Theorem 1.

Let $\pi_{\mathcal{D}}$ and $\bar{\pi}_{\mathcal{D}}$ be the data collecting
policies used to gather data for training an RvS protagonist policy $\pi$.
Assume $T(s_{t+1}\mid
s_{t},a_{t},{\bar{a}_{t}})=\rho^{\pi_{\mathcal{D}},\bar{\pi}_{\mathcal{D}}}(s_{t+1}\mid\tau_{0:t-1},s_{t},a_{t},{\bar{a}_{t}},I(\tau)=z)$
111This condition can be readily met in environments formulated as a game with
deterministic transitions, which are prevalent in many sequential games (e.g.,
Go and Chess (Schrittwieser et al., 2020)), as well as in frequently
considered MuJoCo environments involving adversarial action noise (Pinto et
al., 2017; Cheng et al., 2022). , for any goal $z$ such that
$\rho^{\pi_{\mathcal{D}},\bar{\pi}_{\mathcal{D}}}(I(\tau)=z\mid s_{0})>0$, the
information distance can be minimized:
$\mathbb{E}_{\tau\sim\rho^{\pi_{z},\bar{\pi}_{\textnormal{test}}}}\\!\left[{D(I(\tau_{t}),z)}\right]=0$
if
$\bar{\pi}_{\textnormal{test}}(\bar{a}_{t}\mid\tau_{0:t-1},s_{t},a_{t})=\rho^{\pi_{\mathcal{D}},\bar{\pi}_{\mathcal{D}}}(\bar{a}_{t}\mid\tau_{0:t-1},s_{t},a_{t},I(\tau)=z)$.

Theorem 1 suggests that the protagonist policy $\pi_{z}$, defined in Eq. (3),
can minimize the information distance when the test-time adversarial policy is
$\bar{\pi}_{\text{test}}=\rho^{\pi_{\mathcal{D}},\bar{\pi}_{\mathcal{D}}}(\bar{a}_{t}\mid\tau_{0:t-1},s_{t},a_{t},I(\tau)=z)$.
In other words, $\pi_{z}$, conditioned on a large target return, is guaranteed
to be robust against
$\rho^{\pi_{\mathcal{D}},\bar{\pi}_{\mathcal{D}}}(\bar{a}_{t}\mid\tau_{0:t-1},s_{t},a_{t},I(\tau)=z)$.
Consequently, the policies of DT and ESPER are only guaranteed to perform well
against behavior adversarial policies and will be vulnerable to powerful test-
time adversaries since their information function is dependent on behavior
adversary. This is a concern because, in practice, offline RL datasets are
often collected in settings with weak or suboptimal adversaries.

In this context, to enhance robustness, we consider using an information
function $I(\tau)$ to train the protagonist policy $\pi_{z}$ by simulating the
worst-case scenarios we might encounter during test time. A natural choice for
$I(\tau)$ is the minimax return-to-go, which we formally introduce in Section
3 and use as a central component of our method. With this $I(\tau)$ and
sufficient data coverage,
$\rho^{\pi_{\mathcal{D}},\bar{\pi}_{\mathcal{D}}}(\bar{a}_{t}\mid\tau_{0:t-1},s_{t},a_{t},I(\tau)=z)$
can be approximated to the optimal adversarial policy, implying that $\pi_{z}$
is robust against the optimal adversary. Furthermore, even without sufficient
data coverage, $\pi_{z}$ remains more robust compared to previous DT methods,
which tend to overfit to the behavior adversarial policy.

Figure 2:  Training of Adversarial Robust Decision Transformer. We adopt
Expectile Regression for estimator $\widetilde{Q}$ to approximate the in-
sample minimax. In the subsequent protagonist DT training, we replace the
original returns-to-go with the learned values $\widetilde{Q}^{*}$ to train
policy.

##  3 Adversarial Robust Decision Transformer

To tackle the problem of suboptimal adversarial behavior in the dataset, we
propose Adversarial Robust Decision Transformer (ARDT), a worst-case-aware RvS
algorithm. In ARDT, the information function is defined as the expected
minimax return-to-go:

$\displaystyle
I(\tau)~{}=~{}\min_{{\bar{a}}_{t:H}}\max_{{a}_{t+1:H}}\mathbb{E}_{\begin{subarray}{c}s_{t^{\prime}+1}\sim
T(\cdot\mid s_{t^{\prime}},a_{t^{\prime}},\bar{a}_{t^{\prime}})\\\
t^{\prime}\in[t,H]\end{subarray}}\\!\left[{\sum_{t^{\prime}=t}^{H}R(s_{t^{\prime}},a_{t^{\prime}},\bar{a}_{t^{\prime}})~{}\bigg{|}~{}\tau_{0:t-1},s_{t},a_{t}}\right].$
(4)

Intuitively, this information function represents the expected return when an
adversary aims to minimize the value, while the protagonist subsequently seeks
to maximize it. To train a protagonist policy as described in Eq. (3), ARDT
first relabels the trajectory with the minimax returns-to-go in Eq. (4).
Secondly, the protagonist policy is then trained similarly to a standard
Decision Transformer (DT). This two-step overview of ARDT is provided in
Figure 2, where the left block includes the minimax return estiation with
expectile regression for trajectory relabeling, and the right block represents
the DT training conditioned on the relabeled returns-to-go.

We relabel the trajectories in our dataset with the worst-case returns-to-go
by considering the optimal in-sample adversarial actions. In practice, we
approximate the proposed information function using an (in-sample) _minimax
returns-to-go_ predictor, denoted by $\widetilde{Q}$. We simplify the notation
of the appearance of actions in the dataset
$a_{t}\in\mathcal{A}:\pi_{\mathcal{D}}(a_{t}|\tau_{0:t-1},s_{t})>0$ and
$\bar{a}_{t}\in\bar{\mathcal{A}}:\bar{\pi}_{\mathcal{D}}(\bar{a}_{t}|\tau_{0:t-1},s_{t},a_{t})>0$
to $a_{t}\in\mathcal{D}$ and $\bar{a}_{t}\in\mathcal{D}$, respectively. The
optimal predictor $\widetilde{Q}^{*}$ is a history-dependent state-action
value function satisfying the following condition that at any time
$t=1,\cdots,T$:

$\displaystyle\widetilde{Q}^{*}(\tau_{0:t-1},s_{t},a_{t})~{}=~{}$
$\displaystyle\min_{\bar{a}_{t}\in\mathcal{D}}r_{t}+\mathbb{E}_{s_{t+1}\sim
T(\cdot\mid
s_{t},a_{t},{\bar{a}_{t}})}\\!\left[{\max_{a_{t+1}\in\mathcal{D}}\widetilde{Q}^{*}(\tau_{0:t},s_{t+1},a_{t+1})}\right],$
(5)

We adopt Expectile Regression (ER) (Newey and Powell, 1987; Aigner et al.,
1976) to approximate $\widetilde{Q}$. This choice is particularly advantageous
for RL applications because it helps avoid out-of-sample estimation or the
need to query actions to approximate the minimum or maximum return (Kostrikov
et al., 2021). Specifically, the loss function for ER is a weighted mean
squared error (MSE):

$\displaystyle
L_{\text{ER}}^{\alpha}(u)=\mathbb{E}_{u}\\!\left[{|\alpha-\mathbf{1}(u>0)|\cdot
u^{2}}\right].$ (6)

Suppose a random variable $y$ follows a conditional distribution
$y\sim\rho(\cdot|x)$, function
$g_{\alpha}(x):=\arg\min_{g(x)}L_{\text{ER}}^{\alpha}(g(x)-h(x,y))$ can serve
as an approximate minimum or maximum operator for the function $h(x,y)$ over
all possible value of $y$, i.e.,

$\displaystyle\lim_{\alpha\rightarrow 0}g_{\alpha}(x)=\min_{y:\rho(y\mid
x)>0}h(x,y),\text{ or }\lim_{\alpha\rightarrow
1}g_{\alpha}(x)=\max_{y:\rho(y\mid x)>0}h(x,y).$ (7)

Algorithm 1 Adversarial Robust Decision Transformer (ARDT)

1:Inputs and hyperparameters: offline dataset $\mathcal{D}$ containing
collection of trajectories; $\alpha=0.01$

2:Initialization: Initialize the parameters $\omega$ and $\nu$ for in-sample
returns-to-go networks $Q_{\omega}$ and $\widetilde{Q}_{\nu}$ with the
original returns-to-go, and parameter $\theta$ of the protagonist policy
$\pi_{\theta}$.

3:// expectile regression updates for in-sample returns-to-go networks

4:for iteration $k=0,1,\cdots$ do

5: for trajectory $\tau\in\mathcal{D}$ do

6: // update $\omega$ while keeping $\nu$ fixed

7: $\omega\leftarrow\omega-\nabla_{\omega}\ell^{1-\alpha}(\omega)$, where
$\ell^{1-\alpha}(\omega)$ is given in (9).

8: // update $\nu$ while keeping $\omega$ fixed

9: $\nu\leftarrow\nu-\nabla_{\nu}\ell^{\alpha}(\nu)$, where
$\ell^{\alpha}(\nu)$ is given in (8).

10:// relabeling dataset $\mathcal{D}$ using in-sample returns-to-go

11:Create
$\mathcal{D}_{\textnormal{worst}}=\left\\{{\tau=(\cdots,\widetilde{R}_{t-1},s_{t-1},a_{t-1},\widetilde{R}_{t},s_{t},a_{t},\cdots)\mid\tau\in\mathcal{D},\widetilde{R}_{t}=\widetilde{Q}_{\nu}(s_{t},{a_{t}})}\right\\}$

12:// protagonist policy update

13:$L(\theta)=-\sum_{\tau\in\mathcal{D}_{\textnormal{worst}}}\sum_{t}\log{\pi}_{\theta}(a_{t}\mid\tau_{0:t-1},{s_{t}},\widetilde{R}_{t})$

14:for iteration $k=0,1,\cdots$ do

15: $\theta\leftarrow\theta-\nabla_{\theta}L(\theta)$

16:Output: Policy $\pi_{\theta}$

Our algorithm uses coupled losses for maximum and minimum operators to
approximate the in-sample minimax returns-to-go. Formally, given an offline
dataset $\mathcal{D}$, we alternately update the minimax and maximin returns-
to-go estimators: $\widetilde{Q}_{\nu}(s,a)$ and $Q_{\omega}(s,a,\bar{a})$. In
each iteration, we fix $\omega$ (or $\nu$), denoted as $\widehat{\omega}$ (or
$\widehat{\nu}$), and then update $\nu$ (or $\omega$) based on the following
loss functions:

$\displaystyle\ell^{\alpha}(\nu)$
$\displaystyle=\mathbb{E}_{\tau\sim\mathcal{D}}\\!\left[{L^{\alpha}_{\text{ER}}(\widetilde{Q}_{\nu}(\tau_{0:t-1},{s_{t}},a_{t})-{Q}_{\widehat{\omega}}(\tau_{0:t-1},{s_{t}},a_{t},\bar{a}_{t}))}\right],$
(8) $\displaystyle\ell^{1-\alpha}(\omega)$
$\displaystyle=\mathbb{E}_{\tau\sim\mathcal{D}}\\!\left[{L^{1-\alpha}_{\text{ER}}({Q}_{\omega}(\tau_{0:t-1},{s_{t}},a_{t},\bar{a}_{t})-\widetilde{Q}_{\widehat{\nu}}(\tau_{0:t},{s_{t+1}},a_{t+1})-r_{t})}\right].$
(9)

The minimizers ($\nu^{*}$ and $\omega^{*}$) of the two losses mentioned above
(with $\alpha\to 0$), combined with Eq. (7), satisfy the following condition
for all ${s_{t}},a_{t},\bar{a}_{t}\in\mathcal{D}$:

$\displaystyle\widetilde{Q}_{\nu^{*}}(\tau_{0:t-1},{s_{t}},a_{t})$
$\displaystyle=\min_{\bar{a}\in\mathcal{D}}{Q}_{\omega^{*}}(\tau_{0:t-1},{s_{t}},a_{t},\bar{a}),$
(10) $\displaystyle Q_{\omega^{*}}(\tau_{0:t-1},{s_{t}},a_{t},\bar{a}_{t})$
$\displaystyle=\mathbb{E}_{s_{t+1}\sim\rho^{\pi_{\mathcal{D}},\bar{\pi}_{\mathcal{D}}}(\tau_{0:t-1},{s_{t}},a_{t},\bar{a}_{t})}\\!\left[{\max_{a^{\prime}\in\mathcal{D}}\widetilde{Q}_{\nu^{*}}(\tau_{0:t},{s_{t+1}},a^{\prime})+r_{t}}\right].$
(11)

By substituting Eq. (11) into Eq. (10), the minimax returns-to-go predictor
$\widetilde{Q}_{\nu^{*}}$ satisfies Eq. (5).

The formal algorithm of ARDT is presented in Algorithm 1, where from line 3-6
is the minimax return estimation diagrammed as the left block in Figure 2, and
line 7-10 is the Transformer training on the right of Figure 2. Initially, we
warm up two returns-to-go networks with the original returns-to-go values from
the dataset. This step ensures the value function converges to the expected
values, facilitating faster maximin value estimation and guaranteeing accurate
value function approximation at terminal states. Subsequently, we estimate the
minimax returns-to-go by alternately updating the parameters of the two
networks based on the expectile regression losses in Eq. (9) and Eq. (8). Once
the minimax expectile regression converges, we replace the original returns-
to-go values in the trajectories with the values predicted by
$\widetilde{Q}_{\nu}$. Finally, we train the Decision Transformer (DT)
conditioned on the states and relabeled returns-to-go to predict the
protagonist’s actions. Then the ARDT protagonist policy is ready to be
deployed for evaluation in the adversarial environment directly.

##  4 Experiments

In this section, we conduct experiments to examine the robustness of our
algorithm, Adversarial Robust Decision Transformer (ARDT), in three settings:
(i) Short-horizon sequential games, where the offline dataset has full
coverage and the test-time adversary is optimal (Section 4.1), (ii) A long-
horizon sequential game, Connect Four, where the offline dataset has only
partial coverage and the distributional-shifted test-time adversary (Section
4.2), and (iii) The standard continuous Mujoco tasks in the adversarial
setting and a population of test-time adversaries (Section 4.3). We compare
our algorithm with baselines including Decision Transformer and ESPER. The
implementation deatils are in Appendix C. Notably, since the rewards and
states tokens encompass sufficient information about the adversary’s actions,
all DT models are implemented not conditioned on the past adversarial tokens
to reduce the computational cost.

###  4.1 Full Data Coverage Setting

The efficacy of our solution is first evaluated on three short-horizon
sequential games with adaptive adversary: (1) a single-stage game, (2) an
adversarial, single-stage bandit environment depicting a Gambling round
(Paster et al., 2022), and (3) a multi-stage game. These are depicted in
Figure 1, Figure 6, and Figure 8, respectively. The collected data consists of
$10^{5}$ trajectories, encompassing all possible trajectories. The online
policies for data collection employed by the protagonist and adversary were
both uniformly random.

Figure 3 illustrates the efficacy of ARDT in comparison to vanilla DT and
ESPER. Across all three environments, ARDT achieves the return of maximin
(Nash Equilibrium) against the optimal adversary, when conditioned on a large
target return. As illustrated in Figure 1, ARDT is aware of the worst-case
return associated with each action it can take due to learning to condition on
the relabeled minimax returns-to-go, and successfully takes robust action
$a_{1}$. In single-stage game (Figure 3), ESPER is vulnerable to the
adversarial perturbations, and DT fails to generate robust policy when
conditioned on large target return. While, it is worth noting that DT has
achieved worst-case return $1$ when conditioning on target returns around $1$.
This imposes the question: Can we learn the robust policy by simply
conditioning on the largest worst-case return in testing with vanilla DT
training?

We show via the Gambling environment in Figure 3 that vanilla DT can still
fail even conditioning on the largest worst-case return. In this environment,
the worst-case returns of three protagonist actions are $-15$, $-6$ and $1$.
At the target return $1$, DT is unable to consistently reach a mean return of
$1$. Similar to DT in the single-stage game, it tends to assign equal
likelihood to $a_{1}$ and $a_{2}$ to reach return $1$, leading to a drop of
performance to less than $1$ under adversarial perturbations. Therefore,
simply conditioning on the robust return-to-go in testing is insufficient to
generate robust policy with vanilla DT. Moreover, as the target return
approaches $5$, DT’s performance drops more since DT tends to choose $a_{0}$
more often to hopefully achieve a return of $5$, but instead get negative
return against the adversarial perturbations.

Conversely, ARDT’s policy is robust since it is trained with worst-case
returns-to-go. ESPER also performs well in Gambling due to that in this game
the robust action is also the one with the largest expected return (Figure 8).
ARDT also attains the highest return in a multi-stage game when conditioned on
large target return, while other two algorithms fail. Consequently, in the
context of full data coverage and an adaptive adversary, ARDT is capable of
approximating the maximin (Nash Equilibrium) robust policy against the optimal
adversarial policy, whereas DT is unable to do so.

Figure 3:  Worst-case return versus target return plot comparing the proposed
ARDT algorithm against vanilla DT, on our Single-stage Game (left), Gambling
(centre) and our Multi-stage Game (right), over $10$ seeds.

###  4.2 Discrete Game with Partial Data Coverage

Figure 4:  Average return of ARDT and vanilla DT on Connect Four when trained on suboptimal datasets collected with different levels of optimality for both the online protagonist’s policy ($30\%$, $40\%$ and $50\%$ optimal) and the adversary’s policy ($10\%$, $30\%$, $50\%$ optimal), over $10$ seeds. We test against a fixed adversary that acts optimally $50\%$ of the time, and randomly otherwise.  Test-time adversary | 30% optimal | 50% optimal | 70% optimal | 100% optimal
---|---|---|---|---
DT | 0.18 (0.09) | -0.28 (0.05) | -0.57 (0.06) | -1.00 (0.00)
ESPER | 0.44 (0.09) | 0.00 (0.10) | -0.42 (0.05) | -0.98 (0.01)
ARDT (ours) | 0.55 (0.11) | 0.11 (0.22) | 0.02 (0.09) | -1.00 (0.00)
Table 1:  Average returns of ARDT, vanilla DT and ESPER on Connect Four when
trained on data mixed from both a near-random (30%, 50%) dataset and a near-
optimal (90%, 90%) dataset. Evaluation is done against different optimality
levels of adversary.

Connect Four is a discrete sequential two-player game with deterministic
dynamics (Paster et al., 2022), which can also be viewed as a decision-making
problem when facing an adaptive adversary. There is no intermediate reward;
rather, there is a terminal reward of either $-1$, $0$, or $1$, meaning lose,
tie or win for the first-moving player, respectively. Therefore we fix the
target return in Connect Four as $1$. We fix the protagonist to be the first-
move player. This game has a maximum horizon of $22$ steps. Since we have a
solver of this game, we manage to collect the data with $\epsilon$-greedy
policy. We define the optimality of an $\epsilon$-greedy policy of protagonist
or adversary as $(1-\epsilon)\cdot 100\%$. Each dataset includes a total of
$10^{6}$ number of steps for variable-length trajectories.

The results of training ARDT and DT on suboptimal datasets are presented in
Figure 4. To demonstrate the stitching ability of the two methods, datasets
collected by suboptimal protagonist and adversarial policies were chosen,
namely at $30\%$, $40\%$ and $50\%$ optimality for the protagonist and $10\%$,
$30\%$, $50\%$ optimality for the adversary. The figures show that ARDT
outperforms DT learning from suboptimal datasets, and more clearly so when
tested against more powerful adversaries than the behavior adversary.

We hypothesise that one important factor accounting for the negative returns
of DT in the setting in 4 may be the lack of long, winning trajectories in the
dataset. To exclude this factor, in Table 1 we mix near-random and near-
optimal datasets for training, and test our algorithms against different
levels of adversary. Compared to ESPER and DT, ARDT still significantly
outperforms DT, which suggests that even in the presence of trajectories with
high returns in the data, training DT with original returns-to-go in this
setting will limit its ability to extract a powerful and robust policy. In
addition, the results show that ESPER outperforms DT, indicating relabeling
trajectories with expected return can help the extracted policy be robust to
distributionaly-shifted and more powerful test-time adversarial policies.
ESPER conditioned on the expected return generates policy overfitted to the
behavioral adversarial policy, which can fail under distributional shift.
However, ARDT as a worst-case-aware algorithm, can achieve greater robustness
at test time.

Therefore, in discrete environment with partial data coverage and adaptive
adversary, ARDT is more robust to distributional shift and a more powerful
test-time adversary.

###  4.3 Continuous Adversarial Environments

Worst-case Return
---
Datasets | DT | ESPER | ARDT (ours)
Hopper-lr | 112.10 (63.40) | 66.80 (5.50) | 477.90 (219.90)
Hopper-mr | 330.20 (286.80) | 103.20 (11.70) | 482.20 (83.70)
Hopper-hr | 298.62 (319.72) | 63.00 (11.50) | 331.69 (187.18)
Walker2D-lr | 429.60 (31.90) | 380.60 (156.90) | 405.60 (39.80)
Walker2D-mr | 391.00 (20.60) | 426.90 (29.70) | 508.40 (61.80)
Walker2D-hr | 413.22 (54.69) | 502.60 (31.30) | 492.63 (109.28)
Halfcheetah-lr | 1416.00 (48.50) | 1691.80 (63.00) | 1509.20 (33.80)
Halfcheetah-mr | 1229.30 (171.10) | 1725.60 (111.80) | 1594.60 (137.50)
Halfcheetah-hr | 1493.03 (32.87) | 1699.80 (32.80) | 1765.71 (150.63)
Table 2:  Results of DT, ESPER, and our algorithm ARDT with and without
conditioned on past adversarial tokens: the Worst-case Return against $8$
different online adversarial policies on MuJoCo Noisy Action Robust MDP tasks
across $5$ seeds. We tested with the same set of target returns and select the
best target for each method. To expand the data coverage, we inject random
trajectories collected by $0.1$-greedy online policies into the pre-collected
online datasets with high robust returns. We use low (suffix -lr), medium
(suffix -mr) and high randomness (suffix -hr) to indicate the proportion of
random trajectories.

Figure 5:  From left to right, (1) the worst-case return under adversarial
perturbations with different weights in Halfcheetah, (2) in Hopper, and (3)
average returns of algorithms in environments with different relative mass.
The initial target returns in the environments Halfcheetah and Hopper are
$2000$ and $500$, respectively.

In this section, we test the algorithms on a specific instance of Adversarial
Reinforcement Learning problems, namely Noisy Action Robust MDP (NR-MDP),
which was introduced by Tessler et al. (2019). This includes three continuous
control problems of OpenAI Gym MuJoCo, Hopper, Walker2D and Halfcheetah
(Todorov et al., 2012) involving the adversarial action noise. In NR-MDP, the
protagonist and adversary act jointly. Specifically, they exact a total force
over one or more parts of the body of the agent in a Mujoco environment,
corresponding to the weighted sum of their individual forces (adversary’s
weight $\delta$). Despite no longer being in the discrete setting, the return
minimax expectile regression in ARDT is implemented with the raw continuous
states and actions input, without any discretization.

To collect the data, we first trained Action Robust Reinforcement Learning
(ARRL) online policies on NR-MDP tasks (Tessler et al., 2019) to create the
online protagonist policy and adversary policy for evaluation. Subsequently,
the pre-collected data consist of the replayed trajectories removing the ones
with low cumulative rewards. We collect the suboptimal trajectories with the
$\epsilon$-greedy of the saved ARRL agents, i.e. online protagonist and
adversary policies. We then mix them with the pre-collected datasets to create
the final training datasets. The proportion of random trajectories to pre-
collected trajectories are $0.01:1$, $0.1:1$ and $1:1$, respectively, for
datasets categorised as being low, medium, and high randomness. At test time,
we evaluate each of our trained ARDT protagonist policies against a population
of $8$ adversarial online policies, each across $5$ seeds. We sweep over the
same set of target returns for all algorithms, and pick the best results.

As shown in Table 4, ARDT has overall better worst-case performance than DT
and ESPER. In Hopper, ARDT has significantly higher worst-case return than
both ESPER and DT. In Halfcheetah, ARDT has significantly higher worst-case
return than DT and competitive results with ESPER. This is despite the much
lower data coverage than in the previous settings due to the continuous
environment. Low data coverage is in theory a greater limitation for ARDT
given that it attempts to learn a single best worst-case outcome, rather than
an expectation over outcomes. Even so, our proposed solution outperforms the
baselines in most settings across all three environments. In addition, ARDT
also has better performance when varying the adversary’s weight $\delta$ and
relative mass (Figure 5), implying the robustness of ARDT to test-time
environmental changes. Therefore, in the context of a continuous state and
action space, and when facing a diverse population of adversaries or
environmental changes, ARDT trained with minimax returns-to-go can still
achieve superior robustness than our baselines.

##  5 Conclusions and Limitations

This paper introduces a worst-case-aware training algorithm designed to
improve the adversarial robustness of the Decision Transformer. By relabeling
trajectories with the estimated in-sample minimax returns-to-go through
expectile regression, our algorithm is demonstrated to be robust against
adversaries more powerful than the behavior ones, which existing DT methods
cannot achieve. In experiments, our method consistently exhibits superior
worst-case performance against adversarial attacks in both gaming environments
and continuous control settings.

Limitations.  In our experiments, our practical algorithm estimates the
minimax returns-to-go without accounting for stochasticity, as the transitions
in all tested environments, as well as many real-world game settings, are
deterministic when conditioned on the actions of both players. Future work
should explore both stochastic and adversarial environments to further
evaluate the performance of our proposed solution.

Broader Impact.  Adversarial RvS holds potential for improving the robustness
and security of autonomous agents in dynamic and potentially hostile
environments. By training agents to withstand adversarial actions and
unpredictable conditions, our work can improve the reliability and safety of
technologies such as autonomous vehicles, cybersecurity defense systems, and
robotic operations.

##  6 Acknowledgments

Ilija Bogunovic was supported by the EPSRC New Investigator Award
EP/X03917X/1; the Engineering and Physical Sciences Research Council
EP/S021566/1; and Google Research Scholar award. Xiaohang Tang was supported
by the Engineering and Physical Sciences Research Council [grant number
EP/T517793/1, EP/W524335/1]. The authors would like to thank the Department of
Statistical Science, in particular Chakkapas Visavakul, for co-ordinating the
computer resources, and the Department of Electronic and Electrical
Engineering at University College London for providing the computer clusters.

## References

  * Emmons et al. [2021] Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine.  Rvs: What is essential for offline rl via supervised learning?  _arXiv preprint arXiv:2112.10751_ , 2021.
  * Wu et al. [2024] Yueh-Hua Wu, Xiaolong Wang, and Masashi Hamaya.  Elastic decision transformer.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Yamagata et al. [2023] Taku Yamagata, Ahmed Khalil, and Raul Santos-Rodriguez.  Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl.  In _International Conference on Machine Learning_ , pages 38989–39007. PMLR, 2023.
  * Janner et al. [2021] Michael Janner, Qiyang Li, and Sergey Levine.  Offline reinforcement learning as one big sequence modeling problem.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Schmidhuber [2019] Juergen Schmidhuber.  Reinforcement learning upside down: Don’t predict rewards–just map them to actions.  _arXiv preprint arXiv:1912.02875_ , 2019.
  * Kumar et al. [2019] Aviral Kumar, Xue Bin Peng, and Sergey Levine.  Reward-conditioned policies.  _arXiv preprint arXiv:1912.13465_ , 2019.
  * Chen et al. [2021] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch.  Decision transformer: Reinforcement learning via sequence modeling.  _Advances in neural information processing systems_ , 34:15084–15097, 2021.
  * Pinto et al. [2017] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta.  Robust adversarial reinforcement learning.  In _International Conference on Machine Learning_ , pages 2817–2826. PMLR, 2017.
  * Tessler et al. [2019] Chen Tessler, Yonathan Efroni, and Shie Mannor.  Action robust reinforcement learning and applications in continuous control.  In _International Conference on Machine Learning_ , pages 6215–6224. PMLR, 2019.
  * Kamalaruban et al. [2020] Parameswaran Kamalaruban, Yu-Ting Huang, Ya-Ping Hsieh, Paul Rolland, Cheng Shi, and Volkan Cevher.  Robust reinforcement learning via adversarial training with langevin dynamics.  _Advances in Neural Information Processing Systems_ , 33:8127–8138, 2020.
  * Vinitsky et al. [2020] Eugene Vinitsky, Yuqing Du, Kanaad Parvate, Kathy Jang, Pieter Abbeel, and Alexandre Bayen.  Robust reinforcement learning using adversarial populations.  _arXiv preprint arXiv:2008.01825_ , 2020.
  * Curi et al. [2021] Sebastian Curi, Ilija Bogunovic, and Andreas Krause.  Combining pessimism with optimism for robust and efficient model-based deep reinforcement learning.  In _International Conference on Machine Learning_ , pages 2254–2264. PMLR, 2021.
  * Rigter et al. [2022] Marc Rigter, Bruno Lacerda, and Nick Hawes.  Rambo-rl: Robust adversarial model-based offline reinforcement learning.  _Advances in neural information processing systems_ , 35:16082–16097, 2022.
  * Paster et al. [2022] Keiran Paster, Sheila McIlraith, and Jimmy Ba.  You can’t count on luck: Why decision transformers and rvs fail in stochastic environments.  _Advances in Neural Information Processing Systems_ , 35:38966–38979, 2022.
  * Yang et al. [2022] Mengjiao Yang, Dale Schuurmans, Pieter Abbeel, and Ofir Nachum.  Dichotomy of control: Separating what you can control from what you cannot.  _arXiv preprint arXiv:2210.13435_ , 2022.
  * Newey and Powell [1987] Whitney K Newey and James L Powell.  Asymmetric least squares estimation and testing.  _Econometrica: Journal of the Econometric Society_ , pages 819–847, 1987.
  * Aigner et al. [1976] Dennis J Aigner, Takeshi Amemiya, and Dale J Poirier.  On the estimation of production frontiers: maximum likelihood estimation of the parameters of a discontinuous density function.  _International economic review_ , pages 377–396, 1976.
  * Ghosh et al. [2019] Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine.  Learning to reach goals via iterated supervised learning.  _arXiv preprint arXiv:1912.06088_ , 2019.
  * Paster et al. [2020] Keiran Paster, Sheila A McIlraith, and Jimmy Ba.  Planning from pixels using inverse dynamics models.  _arXiv preprint arXiv:2012.02419_ , 2020.
  * Kim et al. [2023] Jeonghye Kim, Suyoung Lee, Woojun Kim, and Youngchul Sung.  Decision convformer: Local filtering in metaformer is sufficient for decision making.  _arXiv preprint arXiv:2310.03022_ , 2023.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Ortega et al. [2021] Pedro A Ortega, Markus Kunesch, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Joel Veness, Jonas Buchli, Jonas Degrave, Bilal Piot, Julien Perolat, et al.  Shaking the foundations: delusions in sequence models for interaction and control.  _arXiv preprint arXiv:2110.10819_ , 2021.
  * Furuta et al. [2021] Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu.  Generalized decision transformer for offline hindsight information matching.  _arXiv preprint arXiv:2111.10364_ , 2021.
  * Meng et al. [2021] Linghui Meng, Muning Wen, Yaodong Yang, Chenyang Le, Xiyun Li, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, and Bo Xu.  Offline pre-trained multi-agent decision transformer: One big sequence model tackles all smac tasks.  _arXiv preprint arXiv:2112.02845_ , 2021.
  * Xu et al. [2022] Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan.  Prompting decision transformer for few-shot policy generalization.  In _international conference on machine learning_ , pages 24631–24645. PMLR, 2022.
  * Lee et al. [2022] Kuang-Huei Lee, Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadarrama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, et al.  Multi-game decision transformers.  _Advances in Neural Information Processing Systems_ , 35:27921–27936, 2022.
  * Xu et al. [2023] Mengdi Xu, Yuchen Lu, Yikang Shen, Shun Zhang, Ding Zhao, and Chuang Gan.  Hyper-decision transformer for efficient online policy adaptation.  _arXiv preprint arXiv:2304.08487_ , 2023.
  * Kumar et al. [2020] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine.  Conservative q-learning for offline reinforcement learning.  _Advances in Neural Information Processing Systems_ , 33:1179–1191, 2020.
  * Lanctot et al. [2009] Marc Lanctot, Kevin Waugh, Martin Zinkevich, and Michael Bowling.  Monte carlo sampling for regret minimization in extensive games.  _Advances in neural information processing systems_ , 22, 2009.
  * Heinrich et al. [2015] Johannes Heinrich, Marc Lanctot, and David Silver.  Fictitious self-play in extensive-form games.  In _International conference on machine learning_ , pages 805–813. PMLR, 2015.
  * Sunehag et al. [2017] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al.  Value-decomposition networks for cooperative multi-agent learning.  _arXiv preprint arXiv:1706.05296_ , 2017.
  * Lowe et al. [2017] Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch.  Multi-agent actor-critic for mixed cooperative-competitive environments.  _Advances in neural information processing systems_ , 30, 2017.
  * Lanctot et al. [2017] Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel.  A unified game-theoretic approach to multiagent reinforcement learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Srinivasan et al. [2018] Sriram Srinivasan, Marc Lanctot, Vinicius Zambaldi, Julien Pérolat, Karl Tuyls, Rémi Munos, and Michael Bowling.  Actor-critic policy optimization in partially observable multiagent environments.  _Advances in neural information processing systems_ , 31, 2018.
  * Schrittwieser et al. [2020] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al.  Mastering atari, go, chess and shogi by planning with a learned model.  _Nature_ , 588(7839):604–609, 2020.
  * Tang et al. [2023] Xiaohang Tang, Stephen Marcus McAleer, Yaodong Yang, et al.  Regret-minimizing double oracle for extensive-form games.  In _International Conference on Machine Learning_ , pages 33599–33615. PMLR, 2023.
  * Hu and Wellman [2003] Junling Hu and Michael P Wellman.  Nash q-learning for general-sum stochastic games.  _Journal of machine learning research_ , 4(Nov):1039–1069, 2003.
  * Cui and Du [2022] Qiwen Cui and Simon S Du.  When are offline two-player zero-sum markov games solvable?  _Advances in Neural Information Processing Systems_ , 35:25779–25791, 2022.
  * Zhong et al. [2022] Han Zhong, Wei Xiong, Jiyuan Tan, Liwei Wang, Tong Zhang, Zhaoran Wang, and Zhuoran Yang.  Pessimistic minimax value iteration: Provably efficient equilibrium learning from offline datasets.  In _International Conference on Machine Learning_ , pages 27117–27142. PMLR, 2022.
  * Wen et al. [2022] Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang.  Multi-agent reinforcement learning is a sequence modeling problem.  _Advances in Neural Information Processing Systems_ , 35:16509–16521, 2022.
  * Tseng et al. [2022] Wei-Cheng Tseng, Tsun-Hsuan Johnson Wang, Yen-Chen Lin, and Phillip Isola.  Offline multi-agent reinforcement learning with knowledge distillation.  _Advances in Neural Information Processing Systems_ , 35:226–237, 2022.
  * Wang et al. [2024] Xiangsen Wang, Haoran Xu, Yinan Zheng, and Xianyuan Zhan.  Offline multi-agent reinforcement learning with implicit global-to-local value regularization.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Littman [1994] Michael L Littman.  Markov games as a framework for multi-agent reinforcement learning.  In _Machine learning proceedings 1994_ , pages 157–163. Elsevier, 1994.
  * Perolat et al. [2015] Julien Perolat, Bruno Scherrer, Bilal Piot, and Olivier Pietquin.  Approximate dynamic programming for two-player zero-sum markov games.  In _International Conference on Machine Learning_ , pages 1321–1329. PMLR, 2015.
  * Cheng et al. [2022] Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal.  Adversarially trained actor critic for offline reinforcement learning.  In _International Conference on Machine Learning_ , pages 3852–3878. PMLR, 2022.
  * Kostrikov et al. [2021] Ilya Kostrikov, Ashvin Nair, and Sergey Levine.  Offline reinforcement learning with implicit q-learning.  _arXiv preprint arXiv:2110.06169_ , 2021.
  * Todorov et al. [2012] Emanuel Todorov, Tom Erez, and Yuval Tassa.  Mujoco: A physics engine for model-based control.  In _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 5026–5033. IEEE, 2012.  doi:10.1109/IROS.2012.6386109.
  * Neumann and Peters [2008] Gerhard Neumann and Jan Peters.  Fitted q-iteration by advantage weighted regression.  _Advances in neural information processing systems_ , 21, 2008.

##  Supplementary Material

##  Appendix A Proofs

######  Theorem.

1 Let $\pi_{\mathcal{D}}$ and $\bar{\pi}_{\mathcal{D}}$ be the data collecting
policies used to gather data for training an RvS protagonist policy $\pi$.
Assume $T(s_{t+1}\mid
s_{t},a_{t},{\bar{a}_{t}})=\rho^{\pi_{\mathcal{D}},\bar{\pi}_{\mathcal{D}}}(s_{t+1}\mid\tau_{0:t-1},s_{t},a_{t},{\bar{a}_{t}},I(\tau)=z)$,
for any goal $z$ such that
$\rho^{\pi_{\mathcal{D}},\bar{\pi}_{\mathcal{D}}}(I(\tau)=z\mid s_{0})>0$, the
information distance can be minimized:
$\mathbb{E}_{\tau\sim\rho^{\pi_{z},\bar{\pi}_{\textnormal{test}}}}\\!\left[{D(I(\tau),z)}\right]=0$
if
$\bar{\pi}_{\textnormal{test}}(\bar{a}_{t}\mid\tau_{0:t-1},s_{t},{a_{t}})=\rho^{\pi_{\mathcal{D}},\bar{\pi}_{\mathcal{D}}}(\bar{a}_{t}\mid\tau_{0:t-1},s_{t},{a_{t}},I(\tau)=z)$.

###### Proof.

According to Eq. (3):

$\displaystyle\pi(a_{t}\mid\tau_{0:t-1},{s_{t}},z)=\rho^{\pi_{\mathcal{D}},\bar{\pi}_{\mathcal{D}}}(a_{t}\mid\tau_{0:t-1},{s_{t}},I(\tau)=z).$
(12)

If the sufficient condition is satisfied:
$\bar{\pi}_{\textnormal{test}}(\bar{a}_{t}\mid\tau_{0:t-1},s_{t},{a_{t}})=\rho^{\pi_{\mathcal{D}},\bar{\pi}_{\mathcal{D}}}(\bar{a}_{t}\mid\tau_{0:t-1},s_{t},{a_{t}},I(\tau)=z)$,
combined with our assumption, we have

$\displaystyle\bar{\pi}_{\textnormal{test}}(\bar{a}_{t}\mid\tau_{0:t-1},s_{t},{a_{t}})\cdot
T(s_{t+1}\mid
s_{t},{\bar{a}_{t}})=\rho^{\pi_{\mathcal{D}},\bar{\pi}_{\mathcal{D}}}(s_{t+1}\mid\tau_{0:t-1},s_{t},{a_{t}},I(\tau)=z).$
(13)

Then, we have:

$\displaystyle\mathbb{E}_{\tau\sim\rho^{\pi_{z},\bar{\pi}_{\textnormal{test}}}(\cdot\mid
s_{0})}\\!\left[{D(I(\tau),z)}\right]$ $\displaystyle~{}=~{}$
$\displaystyle\sum_{\tau}\rho^{\pi_{z},\bar{\pi}_{\textnormal{test}}}(\tau\mid
s_{0})\cdot D(I(\tau),z)$ $\displaystyle~{}=~{}$
$\displaystyle\sum_{\tau}\prod_{t}{\pi(a_{t}\mid\tau_{0:t-1},{s_{t}},z)\cdot\bar{\pi}_{\textnormal{test}}(\bar{a}_{t}\mid\tau_{0:t-1},s_{t},{a_{t}})\cdot
T(s_{t+1}\mid s_{t},a_{t},{\bar{a}_{t}})}\cdot D(I(\tau),z)$
$\displaystyle~{}=~{}$
$\displaystyle\sum_{\tau}\prod_{t}{\rho^{\pi_{\mathcal{D}},\bar{\pi}_{\mathcal{D}}}(a_{t}\mid\tau_{0:t-1},{s_{t}},I(\tau)=z)\cdot\rho^{\pi_{\mathcal{D}},\bar{\pi}_{\mathcal{D}}}(s_{t+1}\mid\tau_{0:t-1},s_{t},{a_{t}},I(\tau)=z)}\cdot
D(I(\tau),z)$ $\displaystyle~{}=~{}$
$\displaystyle\sum_{\tau}\rho^{\pi_{\mathcal{D}},\bar{\pi}_{\mathcal{D}}}(\tau\mid
s_{0},I(\tau)=z)\cdot D(I(\tau),z)$ $\displaystyle~{}=~{}$
$\displaystyle\mathbb{E}_{\tau\sim\rho^{\pi_{\mathcal{D}},\bar{\pi}_{\mathcal{D}}}(\cdot\mid
s_{0},I(\tau)=z)}\\!\left[{D(I(\tau),z)}\right]$ $\displaystyle~{}=~{}$
$\displaystyle 0.$

The second equation satisfies by simply decomposing $\rho$ as in section 2.
The third equation satisfies due to Eq. (13) and Eq. (3). The fourth equation
satisfies by simply merging all $\rho$ terms. The final equation satisfies
since the expectation in the last but one equation is over the probability
distribution conditional on $I(\tau)=z$, leading to that for any sampled
trajectory $\tau$, $I(\tau)=z$, and thus $D(I(\tau),z)$. ∎

##  Appendix B Environments

#### Multi-Stage Game.

Apart from using a single-stage game as a motivating example, we also
conducted experiments on a multi-stage game to highlight the importance of
multi-step minimax instead of a single-step minimum by an adversary. This is
because the optimal adversary will also consider the optimal protagonist
strategy in the subsequent states. A multi-step minimax return is more
accurate in describing the value of states and sub-trajectories.

Figure 6: Multi-stage Game. LHS is the tree representation of the problem,
where P and A are decision-maker (protagonist) and adversary, respectively.
RHS is the same game tree where we attach the learned minimax return and the
action probabilities of the well-trained ARDT of a single run when
conditioning on the target return $7$ to the adversarial node and the
branches, respectively. For example, the highlighted node has minimax return-
to-go $4.9$. $0.99$ on the branch leads to the highlited node, which indicates
the action probability of taking $a_{0}$ at the initial state. Thick lines
represent the optimal adversarial actions.

#### Gambling.

Gambling [Paster et al., 2022] is originally a simple stochastic bandit
problem with three arms and horizon one. When taking the first arm, the reward
will be either $5$ or $-15$ with $50\%$ probability each. Similarly, when
taking the second arm, the reward will be $1$ or $-6$ with equal
probabilities. The third arm has deterministic reward $1$. In this paper, we
extend it to an adversarial setting by considering an adversary controlling
the outcomes, shown in Figure 8.

Figure 7: Connect Four game. Two players put pieces to specific columns in
turns. The pieces will drop to the final empty row.

Figure 8: Two-player zero-sum game Gambling: P: protagonist, and A: adversary.
The payoff at the leaves are for protagonist.

#### Connect Four.

Connect Four has a long horizon of $22$ in the worst case. Figure 8 shows an
example of this game’s dynamics. The outcomes only consist of win, tie, and
lose, leading to the reward for the protagonist $1$, $0$, and $-1$. We
leverage the solver in https://github.com/PascalPons/connect4 to generate a
stochastic policy for data collection.

##  Appendix C Implementation Details

Our implementation is based on the implementation of Decision Transformer
[Chen et al., 2021], and ESPER [Paster et al., 2022]. Our data collection and
part of the evaluation against online adversaries are based on the
implementation in Noisy Action Robust Reinforcement Learning [Tessler et al.,
2019].

The hyperparameters of the Minimax Expectile Regression are in Table 3. In the
practical version of ARDT’s minimax regression, we adopt a leaf regularization
to ensure the accuracy of predicting outcomes at the terminal state (leaf
node). We record our training iterations of minimax regression in the
hyperparameters table. The number of training iterations in full coverage
examples and Connect Four are larger than twice the maximum trajectory length,
guaranteeing the minimax return propagated from the leaves to every node. The
predicted values are relatively accurate according to the prediction in Figure
6. While in the MuJoCo environment, the trajectory length is too long for
minimax regression to make full propagation, partial propagate already brings
robustness based on the MuJoCo results in Section 4.3.

Decision Transformer is trained with cross-entropy loss in the discrete
environment and MSE in the continuous environment, as in Decision Transformer
[Chen et al., 2021]. During action inference at test time, all DT-based
methods are prompted with a desired initial target return and state to
generate the strategy $\pi$. Subsequently, the trajectory is generated in an
auto-regressive manner, utilizing both the actions sampled by the
protagonist’s strategy $\pi$ and the adversarial strategy $\bar{\pi}$, along
with the transitions and rewards provided by the online environments, till the
environment terminates or the maximum trajectory length is reached for
evaluation. Notably, the target return will be subtracted from the observed
immediate reward once received from the environment.

Notably, given that the rewards and states tokens encompass sufficient
information about the adversary’s action, we train the Transformer model with
sequences without adversarial tokens, as it ensures that the model capacity
remains consistent across algorithms. As usual, vanilla DT’s are conditioned
on the observed returns-to-go, contaminated by the adversarial actions and
states. Similarly, ESPER policy is conditional on the expected returns-to-go
and states. Since the adversary only influences the transition and rewards,
ESPER treats the adversary implicitly as part of the environment that causes
stochasticity and addresses it with expected return as the information
function.

Process | Hyperparameters | Values (Full coverage game/Connect Four/MuJoCo)
---|---|---
Transformer training | Number of training steps | 5000/100000/100000
Number of testing iterations | 100
Context length | 4/20/22
Learning rate | 0.0001
Weight decay | 0.0001
Warm up steps | 1000
Drop out | 0.1
Batch size | 128/128/512
| Optimizer | AdamW
Minimax expectile regression | Number of training iterations | 6/44/50
Learning rate | 0.001
Weight decay | 0.0001
Model | LSTM or MLP
Batch size | 128/128/512
Leaf weight | 0.9
Expectile level | 0.01
| Optimizer | AdamW

Table 3: Hyperparameters of ARDT.

Datasets | Returns of Behavior Policy
---|---
Hopper-lr | 942.8 (417.4)
Hopper-mr | 870.5 (467.4)
Hopper-hr | 505.4 (532.2)
Walker2D-lr | 1330.2 (654.2)
Walker2D-mr | 1277.4 (653.9)
Walker2D-hr | 1013.1 (581.1)
Halfcheetah-lr | 1516.4 (215.8)
Halfcheetah-mr | 1395.1 (461.2)
Halfcheetah-hr | 795.3 (764.4)

Table 4: Data profile of MuJoCo NR-MDP.

Dataset | Returns of Behavior Policy
---|---
(30, 10) | 0.49 (0.87)
(30, 30) | 0.13 (0.99)
(30, 50) | -0.23 (0.97)
(40, 10) | 0.56 (0.83)
(40, 30) | 0.24 (0.97)
(40, 50) | -0.13 (0.99)
(50, 10) | 0.61 (0.79)
(50, 30) | 0.32 (0.95)
(50, 50) | -0.02 (1)

Table 5: Data profile of Connect Four. The tuples represent the optimal
percentages of protagonist and adversary.

##  Appendix D Comparison to Other Methods

Tabular value-based works have been studied to solve zero-sum games offline by
leveraging pessimism for out-of-sample estimation [Cui and Du, 2022, Zhong et
al., 2022]. In contrast, we did not incorporate pessimism due to the
difficulty of tuning its associated hyperparameter. We employ expectile
regression to approximate the in-sample minimax return, inspired by Implicit Q
Learning [Kostrikov et al., 2021] to approximate the Bellman optimal
$Q$-values leveraging expectile regression (ER). ER is efficient since it
doesn’t rely on querying the actions during minimax return estimation, thus
reducing complexity. Learning value functions this way, and learning strategy
from the data is much less complex. Thus, learning values and extracting
policy from them is reasonable. In addition, training Transformer based on
relabeled returns-to-go is similar to doing policy extraction from the learned
values (returns-to-go) [Neumann and Peters, 2008]. Transformers demonstrating
strong reasoning in sequence modeling should be exploited to improve
reinforcement learning.

##  Appendix E Data and Computing Resources

We publish our datasets of Connect Four and Mujoco data collected by executing
Noisy Action Robust Reinforcement Learning and save the data on a Google
drive:
https://drive.google.com/drive/folders/1fd9RNmntL0p86ywEcRkbIKZbKTgJ6BY1?usp=sharing.
There are no data access restrictions. The Mujoco data profiles are in Table 5
and 4. The MuJoCo data has a $1000$ number of trajectories, each with $1000$
steps of interactions. The Connect Four datasets also have a $10^{6}$ number
of interaction steps, where each trajectory has a length of at most $22$.

We conducted experiments on GPUs: a GeForce RTX 2080 Ti with memory 11GB and
an NVIDIA A100 with memory 80 GB. The memory for the entire computing is
around 100GB when parallelizing the running over random seeds in the Connect
Four game and MuJoCo.

##  Appendix F Additional Experimental Results

| With Adversarial Token | Without Adversarial Token
---|---|---
Datasets | DT | ESPER | ARDT | DT | ESPER | ARDT
Hopper-lr | 175.9 (7.5) | 210.6 (21.5) | 302.0 (221.0) | 112.1 (63.4) | 66.8 (5.5) | 477.9 (219.9)
Hopper-mr | 361.3 (278.0) | 302.8 (159.4) | 454.7 (402.5) | 330.2 (286.8) | 103.2 (11.7) | 482.2 (83.7)
Hopper-hr | 322.8 (225.9) | 48.8 (0.3) | 158.5 (2.8) | 298.62 (319.72) | 63.0 (11.5) | 331.69 (187.18)
Walker2D-lr | 475.4 (84.8) | 406.1 (108.2) | 528.6 (85.7) | 429.6 (31.9) | 380.6 (156.9) | 405.6 (39.8)
Walker2D-mr | 505.1 (39.5) | 455.2 (47.0) | 485.5 (67.2) | 391.0 (20.6) | 426.9 (29.7) | 508.4 (61.8)
Walker2D-hr | 511.9 (94.6) | 618.8 (100.5) | 475.2 (84.2) | 413.22 (54.69) | 502.6 (31.3) | 492.63 (109.28)
Halfcheetah-lr | 1170.6 (94.9) | 1368.6 (58.1) | 1323.2 (76.8) | 1416.0 (48.5) | 1691.8 (63.0) | 1509.2 (33.8)
Halfcheetah-mr | 1599.9 (464.2) | 1433.5 (64.7) | 1299.9 (115.0) | 1229.3 (171.1) | 1725.6 (111.8) | 1594.6 (137.5)
Halfcheetah-hr | 1140.8 (251.4) | 1558.5 (345.5) | 974.3 (52.3) | 1493.03 (32.87) | 1699.8 (32.8) | 1765.71 (150.63)

Table 6: Ablation study on training policy conditioning on adversary tokens
with average Worst-case returns against $8$ adversaries including algorithms
with and without past adversary tokens. ARDT trained without adversarial token
has better performance in general.  Figure 9: Average results in environments
with different test-time weight of adversarial noise. The target return of the
three environments are set to $2000$, $500$ and $800$.