# Bias Resilient Multi-Step Off-Policy
Goal-Conditioned Reinforcement Learning

Lisheng Wu, Ke Chen
Department of Computer Science
University of Manchester
Manchester, M13 9PL, UK
<EMAIL_ADDRESS>

###### Abstract

In goal-conditioned reinforcement learning (GCRL), sparse rewards present
significant challenges, often obstructing efficient learning. Although multi-
step GCRL can boost this efficiency, it can also lead to off-policy biases in
target values. This paper dives deep into these biases, categorizing them into
two distinct categories: “shooting" and “shifting". Recognizing that certain
behavior policies can hasten policy refinement, we present solutions designed
to capitalize on the positive aspects of these biases while minimizing their
drawbacks, enabling the use of larger step sizes to speed up GCRL. An
empirical study demonstrates that our approach ensures a resilient and robust
improvement, even in ten-step learning scenarios, leading to superior learning
efficiency and performance that generally surpass the baseline and several
state-of-the-art multi-step GCRL benchmarks.

##  1 Introduction

In goal-conditioned reinforcement learning (GCRL) (Schaul et al., 2015a; Liu
et al., 2022), agents aim to complete various tasks each characterized by
different goals. A primary challenge arises from the complexity of reward
engineering, often leading to sparse reward settings—where valuable rewards
are only given upon reaching the goals. Such limited feedback inhibits
efficient learning, causing delays in the learning progression. Although
relabeling goals based on actual outcomes for each trajectory augments the
learning experience (Andrychowicz et al., 2017), the slow propagation of these
sparse learning signals remains a bottleneck for the rapid acquisition of
successful policies. To counteract this, multi-step hindsight experience
replay (MHER) (Yang et al., 2021) accelerates the relay of relabeled rewards
using multi-step targets. However, in the off-policy setting, multi-step
learning can introduce off-policy biases originating from the divergence
between target and behavior policies. As the step-size magnifies, these
challenges are further amplified. While methods like MHER($\lambda$) and
MMHER, proposed by Yang et al. (2021), attempt to mitigate these biases, we
observe that they lack the requisite resilience, rendering their methods
effective only for smaller step sizes.

Though off-policy bias in multi-step RL has been well-studied (Sutton & Barto,
2018; Munos et al., 2016; De Asis et al., 2018), its nuances within GCRL
present novel challenges. In GCRL, the mere act of reaching a goal does not
guarantee its sustained achievement. The agent must learn the optimal strategy
to consistently stay on the goal, which necessitates continuous bootstrapping
of value functions on goal states. As off-policy biases accumulate at these
goal states, they can retroactively distort the value estimates of preceding
states. In light of this, we categorize off-policy bias into the shooting bias
that accumulates up to the goal states and introduce the novel concept of
shifting bias that builds within the goal states. While off-policy biases are
typically perceived as detrimental, by examining the multi-step target values,
we propose that certain behavior policies offer superior alternatives over the
multi-step reward accumulation, guiding the agent to refine its policy more
swiftly. Building on this understanding, we present tailored solutions that
harness the positive effects of both biases while curbing their adverse
effects.

In the broader context of examining bias problems in multi-step GCRL, it is
imperative to recognize that the overall bias is not confined to off-policy
bias. It also includes over-optimistic bias (Hasselt, 2010; Van Hasselt et
al., 2016; Fujimoto et al., 2018) and hindsight biases (Blier & Ollivier,
2021; Schramm et al., 2023). Therefore, to isolate the effects and better
understand the intricate nature of off-policy bias, we utilize existing
techniques (Fujimoto et al., 2018) to counteract the over-optimistic bias and
completely bypass hindsight biases stemming from stochastic dynamics (Blier &
Ollivier, 2021; Schramm et al., 2023) by centering our study on deterministic
environments.

Our main contributions are summarized as follows: i) By analyzing off-policy
bias in multi-step GCRL, we dissect off-policy bias in multi-step GCRL,
categorizing it into two distinct types based on their respective roles, and
introduce metrics for evaluating each type of bias. ii) We probe the root
causes of two types of biases in multi-step GCRL, elucidating their beneficial
and detrimental effects on learning. iii) We propose a novel resilient
strategy for each type of off-policy bias in multi-step GCRL, culminating in
the robust algorithm BR-MHER. iv) Empirical evaluation demonstrates that our
approach robustly outperforms both the HER baseline and several state-of-the-
art multi-step GCRL methods in efficiency, bias mitigation, and performance
across varied GCRL benchmarks.

##  2 Related Work

#### Multi-Step Off-Policy RL

In off-policy multi-step RL, addressing off-policy bias is paramount. There
are several identified biases: overestimation bias stemming from function
approximation errors (Hasselt, 2010; Van Hasselt et al., 2016; Fujimoto et
al., 2018), distributional shift bias due to imbalanced learning distributions
(Schaul et al., 2015b), and hindsight bias which manifests in stochastic
environments (Blier & Ollivier, 2021; Schramm et al., 2023). Both
TD($\lambda$) and Q($\lambda$) methods utilize eligibility traces as a means
to mitigate such biases (Sutton & Barto, 2018; Harutyunyan et al., 2016).
Additionally, Q($\lambda$) (Harutyunyan et al., 2016) brings in off-policy
corrections using the current action-value functions. Importance Sampling (IS)
offers an additional layer for bias correction and aids in convergence (Precup
et al., 2001; Munos et al., 2016), but it does come with a trade-off of
heightened variance. Despite these corrective measures, a majority of these
methods tend to cap their step size, often limiting it to three steps (Barth-
Maron et al., 2018; Hessel et al., 2018).

#### Multi-Step Off-Policy GCRL

Multi-step GCRL has inherent biases akin to traditional multi-step RL,
significantly affecting learning. The prominence of off-policy bias
intensifies in multi-step GCRL, especially as the reward accumulation steps
increase, magnifying both bias and variance (Yang et al., 2021). Yang et al.
(2021) presented MHER, MHER($\lambda$), and MMHER, which merge multi-step
learning with hindsight relabeling (Andrychowicz et al., 2017). While vanilla
MHER lacks mechanisms to handle off-policy bias, MHER($\lambda$) extends the
MHER base by leveraging techniques from TD($\lambda$) (Sutton & Barto, 2018).
MMHER, meanwhile, uses an on-policy target from a learned world model. The
robustness of MHER($\lambda$) and MMHER hinges on MHER’s inherent stability
and the precision of the learned model, respectively. In contrast, our study
introduces two novel subtypes of multi-step off-policy bias in GCRL:
_shooting_ and _shifting_ biases. The latter, unique to GCRL, emerges from
accumulating off-policy biases at goal states, influencing prior state value
estimates. We found pronounced shifting biases in multi-step GCRL for
intricate tasks. Our truncation technique effectively curtails this bias. Our
insights suggest specific behavior policies might offer benefits. With this,
we deploy quantile regression to address off-policy bias comprehensively.

#### Quantile-Based Regression

Quantile regression was pioneered by Koenker & Hallock (2001) and later
adapted for distributional RL to model state-action value functions’ quantile
function (Dabney et al., 2018a; b; Kuznetsov et al., 2020). Expanding on this,
Tang et al. (2022) integrated multi-step RL within distributional RL
frameworks utilizing quantile regression. Motivated by these studies, our
exploration specifically targets the upper quantiles of the value
distribution. While rooted in deterministic settings, we account for
stochasticity introduced by multi-step behavior policies, which sets our work
apart from existing studies. Unlike prior works learning a complete stochastic
value distribution, we prioritize the upper quantile, representing optimal
outcomes under select effective behavior policies tied to goals. This quantile
guides agents toward actions likely surpassing current estimates and
counteracts suboptimal behavior policies’ drawbacks.

##  3 Preliminary

GCRL is framed as a goal-augmented MDP characterized by parameters
$(\mathcal{S},\mathcal{A},\mathcal{T},r,\mathcal{G},p_{d\\!g},\phi,\gamma,T)$.
Here, $\mathcal{S}$, $\mathcal{A}$, $\gamma$, and $T$ are state space, action
space, discount factor, and horizon, respectively. The transition function
$\mathcal{T}(s^{\prime}|s,a)$ gives the probability of transitioning from
state $s$ to $s^{\prime}$ via action $a$. The goal space is $\mathcal{G}$,
with $p_{d\\!g}$ as the desired goal distribution. The function $\phi(s)$ maps
state $s$ to its achieved goal. In sparse reward settings, the reward
$r(s^{\prime},g)$ is binary, being zero if the distance
$d(\phi(s^{\prime}),g)$ between achieved and target goals is less than
$\epsilon$ and $-1$ otherwise. The success of an episode $\tau$ with goal $g$
is defined by $r(s_{T}^{\tau},g)=0$. It is worth noting that merely reaching
the goal does not constitute the termination of an episode. This distinction
is crucial as an agent might reach the goal accidentally and lack the ability
to sustain its presence there. For instance, if an object slides through the
goal position at a non-zero speed, this is not considered a successful
outcome. The agent must reach the goal and maintain its position there for the
episode to be successfully completed. GCRL’s objective is to learn a goal-
conditioned policy $\pi(s,g)$ maximizing:

$J(\pi)=\mathbb{E}_{g\sim
p_{d\\!g},a_{t}\sim\pi(s_{t},g),s_{t+1}\sim\mathcal{T}(\cdot|s_{t},a_{t})}\left[\sum_{t=0}^{T-1}\gamma^{t}r(s_{t+1},g)\right].$

With an actor-critic framework, the policy (actor) $\pi_{\psi}$ and the
action-value function (critic) $Q_{\theta}$ are parameterized by $\psi$ and
$\theta$, respectively. The policy $\pi_{\psi}$ is updated by minimizing the
following loss function:

$L(\psi)=-\mathbb{E}_{\tau\sim\mathcal{B},\,t\sim[0,T),\,g\sim
h(t,\tau)}\left[Q_{\theta}(s^{\tau}_{t},\pi_{\psi}(s^{\tau}_{t},g),g)\right],$
(1)

where $\tau$ represents a trajectory sampled from the replay buffer
$\mathcal{B}$, $t$ is a timestep sampled within the horizon $T$,
$s_{t}^{\tau}$ denotes the state at timestep $t$ for trajectory $\tau$, and
$h(t,\tau)$ is the distribution of goals according to the hindsight relabeling
technique. The action-value function, $Q_{\theta}$, is updated by minimizing
the following loss function:

$L(\theta)=\mathbb{E}_{\tau\sim\mathcal{B},\,t\sim[0,T),\,g\sim
h(t,\tau)}\left[\left(Q_{\theta}(s^{\tau}_{t},a^{\tau}_{t},g)-y^{(n)}(t,\tau,g)\right)^{2}\right],$
(2)

where $a^{\tau}_{t}$ is the action at the $t$th step of trajectory $\tau$, and
$y^{(n)}(t,\tau,g)$ calculates the $n$-step target value, defined as:

$y^{(n)}(t,\tau,g)=\sum_{i=1}^{n-1}\gamma^{i}r(s^{\tau}_{t+i},g)+\gamma^{n}Q_{\bar{\theta}}(s^{\tau}_{t+n},\pi_{\bar{\psi}}(s^{\tau}_{t+n},g),g),$
(3)

where $\bar{\psi}$ and $\bar{\theta}$ are the parameters of the target actor
and critic for learning stabilization. Although $n$ can truncate if $t+n>T$,
this is simplified for clarity. Typically, $\pi$ represents $\pi_{\psi}$, with
emphasis on learning the action-value function. Further context is provided in
Appendix A.

##  4 Bias Decomposition

This section delves into the off-policy bias in multi-step GCRL. Through
comprehensive analysis, we decompose this bias into two distinct types and
examine their implications on action-value estimation.

The temporal difference (TD) error between two consecutive states at the step
$t$ of trajectory $\tau$ is

$\delta_{\Theta}(s^{\tau}_{t},a^{\tau}_{t},g)=\gamma
Q_{\bar{\theta}}(s^{\tau}_{t+1},\pi_{\bar{\psi}}(s^{\tau}_{t+1},g),g)+r(s^{\tau}_{t+1},g)-Q_{\Theta}(s^{\tau}_{t},a^{\tau}_{t},g).$

The variable $\Theta$ in the action-value function $Q_{\Theta}$ can take the
values $\theta$ or $\bar{\theta}$, determining whether the temporal difference
(TD) error is computed with respect to the critic or the target critic,
respectively. In a deterministic environment,
$\delta_{\theta}(s_{t}^{\tau},a_{t}^{\tau},g)$ is equivalent to
$\delta_{\theta}(s,a,g)$ whenever $(s_{t}^{\tau},a_{t}^{\tau},g)$ equals
$(s,a,g)$. The $n$-step TD error between the $n$-step targets and the
estimated action-value is characterized as

$y^{(n)}(t,\tau,g)-Q_{\theta}(s^{\tau}_{t},a^{\tau}_{t},g)=\delta_{\theta}(s^{\tau}_{t},a^{\tau}_{t},g)+\sum_{i=1}^{n-1}\gamma^{i}[A_{\bar{\theta}}(s^{\tau}_{t+i},a^{\tau}_{t+i},g)+\delta_{\bar{\theta}}(s^{\tau}_{t+i},a^{\tau}_{t+i},g)],$
(4)

where
$A_{\bar{\theta}}(s^{\tau}_{t+i},a^{\tau}_{t+i},g)=Q_{\bar{\theta}}(s^{\tau}_{t+i},a^{\tau}_{t+i},g)-Q_{\bar{\theta}}(s^{\tau}_{t+i},\pi(s^{\tau}_{t+i},g),g)$
is the estimated advantage of action $a^{\tau}_{t+i}$ in state
$s^{\tau}_{t+i}$, given goal $g$. The details of the derivation of Eq. (4) can
be found in Appendix A. Through the application of Eq. (4), the critic loss in
Eq. (2) is reformulated as follows:

$L(\theta)=\mathbb{E}_{\tau,\,t,\,g}\left[\left(\delta_{\theta}(s_{t}^{\tau},a_{t}^{\tau},g)-\sum_{i=1}^{n-1}\gamma^{i}\left(A_{\bar{\theta}}(s^{\tau}_{t+i},a^{\tau}_{t+i},g)+\delta_{\bar{\theta}}(s^{\tau}_{t+i},a^{\tau}_{t+i},g)\right)\right)^{2}\right],$
(5)

where the summation term $\sum_{i=1}^{n-1}$ becomes zero when $n=1$. In
single-step learning ($n=1$), $\delta_{\theta}(s_{t}^{\tau},a_{t}^{\tau},g)=0$
for all $\tau\in\mathcal{B}$, $t\in[0,T)$, and $g$ within the support of goal
distribution $h(\tau,t)$ constitutes the optimal solution for $L(\theta)$.
However, this condition does not necessarily hold as $n$ increases, owing to
the additional summation term. In a deterministic environment,
$\delta_{\theta}(s_{t}^{\tau},a_{t}^{\tau},g)$ is equivalent to
$\delta_{\theta}(s,a,g)$ whenever $(s_{t}^{\tau},a_{t}^{\tau},g)$ equals
$(s,a,g)$. Nonetheless, the summation
$\sum_{i=1}^{n-1}\gamma^{i}\left(A_{\bar{\theta}}(s^{\tau}_{t+i},a^{\tau}_{t+i},g)+\delta_{\bar{\theta}}(s^{\tau}_{t+i},a^{\tau}_{t+i},g)\right)$
varies with the trajectory $\tau$ and the timestep $t$, thereby deviating the
optimal TD error $\delta_{\theta}(s_{t}^{\tau},a_{t}^{\tau},g)$ for
$L(\theta)$ from zero. $\delta_{\Theta}(s,a,g)$ itself is no longer attributed
to the network approximation error any more as it is not optimized towards
zero. Consequently, in multi-step learning scenarios, $\delta_{\theta}(s,a,g)$
encompasses biases originating from both the off-policy advantage and the TD
error components, as reflected in the summation term. Given that the parameter
$\bar{\theta}$ of the target critic is updated from the parameter $\theta$ of
the critic, it follows that $\delta_{\bar{\theta}}(s,a,g)$ is similarly
affected by these biases. Subsequently, we will explore how this localized
bias $\delta_{\theta}(s_{t}^{\tau},a_{t}^{\tau},g)$ aggregates to form a
broader off-policy bias in the action-value function.

For on-policy evaluation, let $\tau^{g}$ denote the trajectory resultant from
policy $\pi(\cdot,g)$. Recursive decomposition of the learned action-value
$Q_{\bar{\theta}}(s^{\tau^{g}}_{t},a^{\tau^{g}}_{t},g)$ and its unbiased
counterpart $Q_{\pi}(s^{\tau^{g}}_{t},a^{\tau^{g}}_{t},g)$ along $\tau^{g}$
until episode termination yields:

$\displaystyle Q_{\bar{\theta}}(s^{\tau^{g}}_{t},a^{\tau^{g}}_{t},g)$
$\displaystyle=\sum_{i=t}^{T-1}\gamma^{i-t}[r(s^{\tau^{g}}_{i+1},g)+\delta_{\bar{\theta}}(s^{\tau^{g}}_{i},a^{\tau^{g}}_{i},g)]+\gamma^{T-t}Q_{\bar{\theta}}(s^{\tau^{g}}_{T},\pi(s^{\tau^{g}}_{T},g),g),$
(6) $\displaystyle Q_{\pi}(s^{\tau^{g}}_{t},a^{\tau^{g}}_{t},g)$
$\displaystyle=\sum_{i=t}^{T-1}\gamma^{i-t}r(s^{\tau^{g}}_{i+1},g)+\gamma^{T-t}Q_{\pi}(s^{\tau^{g}}_{T},\pi(s^{\tau^{g}}_{T},g),g).$
(7)

In the context of multi-step GCRL, our aim is to devise a strategy that
minimizes the detrimental effects of off-policy bias, thereby augmenting the
agent’s proficiency in achieving goals. Given this focus, off-policy bias in
failing policies becomes less of interest given our target. Consequently, our
analysis intentionally focuses on successful evaluation trajectories, denoted
as $\tau^{g+}$. Analyzing off-policy bias in successful trajectories, together
with overall success rates, provides a multidimensional perspective. This
approach not only gives a thorough assessment of the algorithm’s performance
but also reveals how bias adversely affects learning efficiency and success
rates.

In the context of a successful trajectory $\tau^{g+}$, both the terminal
reward $r(s^{\tau^{g+}}_{T},g)$ and the terminal action-value
$Q_{\pi}(s^{\tau^{g+}}_{T},\pi(s^{\tau^{g+}}_{T},g),g)$ are zero, in
accordance with the success criteria outlined in Section 3. Given these
conditions and Eqs. (6) and (7), we discern the off-policy bias as:

$\displaystyle B(s^{{\tau}^{g+}}_{t},a^{{\tau}^{g+}}_{t},g)$
$\displaystyle=\underbrace{\sum_{i=t}^{T-1}\gamma^{i-t}\delta_{\bar{\theta}}(s^{{\tau}^{g+}}_{i},a^{{\tau}^{g+}}_{i},g)}_{\text{Shooting
Bias}}\phantom{=}+\gamma^{T-t}\underbrace{\vphantom{\sum_{i=t}^{T-1}}Q_{\bar{\theta}}(s^{{\tau}^{g+}}_{T},\pi(s^{{\tau}^{g+}}_{T},g),g)}_{\text{Shifting
Bias}}.$ (8)

Thus, we dissect the off-policy bias in Eq. (8) into two distinct segments:
the shooting bias and the shifting bias. The shooting bias, accumulating
within the finite time horizon $T$, can distort the action-value function and
affect decision-making. The shifting bias manifests mainly within goal states
which accumulate over paths of potentially infinite lengths, , detailed
further in Section 5.2. Both biases can be fundamentally decomposed into a
discounted sum of TD errors, which tend to deviate from zero due to the
influence of off-policy data. This decomposition becomes particularly evident
when learning with the loss function described in Eq. (5), especially for
cases where $n>1$. While the shooting bias in multi-step GCRL shows
similarities to off-policy biases in traditional multi-step RL, the shifting
bias is unique to multi-step GCRL.

##  5 Bias Management and Measurement

In this section, we analyze the shooting and shift biases, propose strategies
to manage them and introduce metrics to quantify them.

###  5.1 Shooting Bias Management

A direct approach to mitigate the shooting bias is importance sampling (IS)
(Precup et al., 2001), aiming to correct the off-policy bias. However, IS may
suffer from the variance problem. Moreover, IS may not effectively harness
multi-step targets to enhance learning efficiency. For deriving unbiased
targets via importance sampling, the effects of the data in the multi-step
transitions reduce as the step increases, particularly evident in policies
exhibiting low entropy. In extreme scenarios, multi-step learning for a
deterministic policy could revert to single-step learning. Though it appears
reasonable to disdain the off-policy data less irrelevant to the current
policy, we argue that it is worth rethinking a pivotal question: Is bias
always detrimental, or can specific biases positively guide and improve the
target policy?

To explore this, we examine the benefits of multi-step learning using off-
policy trajectories compared to on-policy rollouts. For clarity in multi-step
indexing, we denote the state-action pair $(s,a)$ with a subscript $t$, as
$(s_{t},a_{t})$. In a deterministic environment, the subsequent state
$s_{t+1}$ is deterministically derived from $(s_{t},a_{t})$. The on-policy
rollout originating from the state-action pair $(s_{t},a_{t})$ is denoted as
$\xi^{g}$. For the set of off-policy trajectories that include
$(s_{t},a_{t})$, each trajectory is represented as $\tau_{k}$, where $k$
serves as the index. The timestep in trajectory $\tau_{k}$ corresponding to
$(s_{t},a_{t})$ is denoted as $t_{k}$. As depicted in Fig. 1(a), the on-policy
rollout $\xi^{g}$ can entirely avoid bias relative to the current policy. In
contrast, as illustrated in Fig. 1(b), off-policy trajectories introduce an
off-policy bias by computing target values along these trajectories.

When an agent derives a higher target value $y^{(n)}(t_{k},\tau_{k},g)$ from
an off-policy trajectory $\tau_{k}$ compared to $y^{(n)}(t,\xi^{g},g)$
obtained from the corresponding on-policy rollout $\xi^{g}$, this suggests
that the actions in $\tau_{k}$ starting from timestep $t_{k}$ might offer
better strategies over the next $n$ steps. For instance, as illustrated in
Fig. 1(b), trajectory $\tau_{1}$ achieves the goal in fewer steps from state
$s_{t}$ compared to the on-policy rollout depicted in Fig. 1(a). This
scenario, particularly in two-step learning for updating
$Q_{\theta}(s_{t},a_{t},g)$, implies that
$y^{(2)}(t_{k},\tau_{k},g)>y^{(2)}(t,\xi^{g},g)$ may indicate the superiority
of the action $a^{\tau_{1}}_{t_{1}+1}$ over $\pi(s_{t+1},g)$. Consequently,
$y^{(2)}(t_{1},\tau_{1},g)$ provides a more accurate representation of the
potential benefits of taking action $a_{t}$ in contrast to
$y^{(2)}(t,\xi^{g},g)$.

Relying exclusively on on-policy rollouts for updating
$Q_{\theta}(s_{t},a_{t},g)$ can lead to inefficiencies, particularly in mini-
batch learning scenarios where the agent must wait to recognize and
incorporate the advantages of actions like $a_{t_{1}+1}^{\tau_{1}}$ in the
subsequent steps. This delayed policy adjustment process may hinder the
agent’s ability to promptly leverage more beneficial actions such as $a_{t}$.
To effectively utilize advantageous off-policy strategies, we introduce the
concept of an $n$-step off-policy transition that exhibits beneficial off-
policy bias.

######  Definition 1.

In deterministic environments, for a given starting state-action pair
$(s_{t},a_{t})$, an $n$-step transition sequence
$s^{\tau_{k}}_{t_{k}},a^{\tau_{k}}_{t_{k}},\cdots,s^{\tau_{k}}_{t_{k}+n}$ in
trajectory $\tau_{k}$, where
$(s_{t},a_{t})=(s^{\tau_{k}}_{t_{k}},a^{\tau_{k}}_{t_{k}})$, is said to
exhibit beneficial off-policy bias when the $n$-step target values conditioned
on the goal $g$, derived from this off-policy transition sequence, are greater
than those obtained from the corresponding $n$-step on-policy rollout
$\xi^{g}$. This property is mathematically expressed as follows.

$y^{(n)}(t_{k},\tau_{k},g)>y^{(n)}(t,\xi^{g},g).$ (9)

Figure 1: Illustration of $Q_{\theta}(s_{t},a_{t},g)$ updates in a
deterministic environment using two approaches. (a) On-policy rollout
$\xi^{g}$, where the agent learns from transitions following the sequence
$(s_{t},a_{t},s_{t+1})$. (b) Off-policy trajectories $\\{\tau_{k}\\}$, with
$k=1,2,\cdots$, where the agent learns from multi-step transitions in each
trajectory $\tau_{k}$ at timestep $t_{k}$, characterized by
$(s_{t},a_{t},s_{t+1})=(s^{\tau_{k}}_{t_{k}},a^{\tau_{k}}_{t_{k}},s^{\tau_{k}}_{t_{k}+1})$

However, detrimental off-policy bias co-exists with beneficial bias. Some
targets even approach the worst target value to be taken
$\frac{-1}{1-\gamma}$, making it extremely hard to compare the potential of
each action. Thus, there arises another problem: Can we make use of the
beneficial bias to improve the learning efficiency while mitigating the
detrimental effects of off-policy bias to the learning process?

In the model-free setting, where we often lack access to on-policy rollouts
$\xi_{g}$, accurately identifying beneficial bias presents a significant
challenge. Recognizing that higher target values in multi-step transitions
typically stem from more effective strategies or decisions, we infer that
these transitions are likely to exhibit higher beneficial bias or lower
detrimental bias. This inference guides our focus on such transitions during
the learning process. Accordingly, we define the maximum target value
$y_{\text{max}}(s,a,g)$ and the corresponding learning loss
$L_{\text{max}}(\theta)$ as:

$\displaystyle y_{\text{max}}(s,a,g)$
$\displaystyle=\max_{t_{k},\tau_{k}}\left\\{y^{(n)}(t_{k},\tau_{k},g)\mid
s_{t}^{\tau_{k}}=s,a_{t}^{\tau_{k}}=a\right\\},$ $\displaystyle
L_{\text{max}}(\theta)$
$\displaystyle=\mathbb{E}_{\tau,t,g}\left[\left(Q_{\theta}(s^{\tau}_{t},a^{\tau}_{t},g)-y^{(n)}(t,\tau,g)\right)^{2}\big{|}y^{(n)}(t,\tau,g)=y_{\text{max}}(s^{\tau}_{t},a^{\tau}_{t},g)\right].$

Though obtaining $y_{\text{max}}(s,a,g)$ is feasible in a tabular setting due
to its limited state-action space, this task becomes significantly more
challenging in scenarios with high-dimensional state-action space. An
alternative method involves learning only towards target values larger than
the current action-value estimates, guiding the agent iteratively towards
higher values until the maximum is reached. This is formalized in the loss
function $L_{\text{larger}}(\theta)$:

$L_{\text{larger}}(\theta)=\mathbb{E}_{\tau,t,g}\left[\left(Q_{\theta}(s^{\tau}_{t},a^{\tau}_{t},g)-y^{(n)}(t,\tau,g)\right)^{2}\big{|}y^{(n)}(t,\tau,g)>Q_{\theta}(s^{\tau}_{t},a^{\tau}_{t},g)\right].$

However, the approach of consistently learning towards larger target values
can inadvertently foster over-optimization bias in the action-value function,
as it relentlessly drives the learning process to seek higher values. This
unidirectional focus might impede the action-value function’s adaptability to
fluctuations in target value distributions. Notably, action values often start
near zero following neural network initialization and can decrease
significantly during learning, potentially skewing the identification of the
true maximum value.

#### Bias Management with Quantile Regression

Rather than pursuing the maximum, we reorient our focus towards learning the
action-value function to capture an upper quantile $\rho$ ($0.5<\rho<1$) of
the target values. To learn the upper quantile, we employ quantile regression
(Koenker & Hallock, 2001; Dabney et al., 2018b). In our methodology, we adopt
the loss function $l_{QR}$ from the quantile regression framework to quantify
the discrepancy between each action-value $Q$ and the corresponding target
action-value $y$. Incorporating both the quantile level $\rho$ and the Huber
loss threshold $\kappa$ (Huber, 1992), the formulation of $l_{QR}$ is:

$l_{QR}(y,Q;\rho,\kappa)=\left|\rho-\mathbb{I}(y,Q)\right|\cdot\begin{cases}(y-Q)^{2}&\text{if
}\left|y-Q\right|\leq\kappa,\\\
\kappa\left(2\cdot\left|y-Q\right|-\kappa\right)&\text{otherwise}.\end{cases}$
(10)

where the indicator function $\mathbb{I}(y,Q)$ equals 1 if $y<Q$ and 0
otherwise. Instead of simply taking the average, $l_{QR}$ weight target values
higher and lower than the current action-value by $|\rho|$ than $|1-\rho|$,
respectively. By applying quantile regression with the $l_{QR}$ loss function,
we can reformulate the general loss in Eq. (2) into this expression:

$L_{QR}(\theta)=\mathbb{E}_{\tau,t,g}\left[l_{QR}(y^{(n)}(t,\tau,g),Q_{\theta}(s^{\tau}_{t},a^{\tau}_{t},g);\rho,\kappa)\right].$
(11)

This method empowers the agent to more effectively exploit beneficial biases
while simultaneously mitigating the impact of detrimental biases. Furthermore,
it fosters adaptability to variations in the target action-value distribution,
thereby maintaining consistent responsiveness to the dynamics of action-value
changes.

###  5.2 Shifting Bias Reduction

Contrasting with shooting bias, which is confined within a finite temporal
horizon $T$ as illustrated in Eq. (8), shifting bias distinctively accumulates
in goal states over an infinite horizon. This bias escalates as learning
advances, leading to a consequential shift in the action-value of the goal
state. Such a shift tends toward a biased estimation, diverging from the
actual action-value and thus significantly altering the learning trajectory.
In GCRL, the update mechanism for state-action pairs, preceding the
achievement of the goal, is fundamentally reliant on the action-value at the
goal state. Therefore, a shift in the goal state’s action-value propagates
backwards, affecting the entire chain of state-action pairs. While quantile
regression, as discussed previously, offers a viable approach to mitigate
these biases and guide the agent towards a more robust off-policy strategy,
the continual shift of target action-values poses a challenge to learning
stability. It underscores the necessity of developing strategies that can
adaptively respond to and compensate for these biases.

For the shifting bias, as detailed in Eq. (7), extending the episode’s
unrolling to the infinite future from time $T$ allows for its decomposition
into:

$Q_{\bar{\theta}}(s^{{\tau}^{g+}}_{T},\pi(s^{{\tau}^{g+}}_{T},g),g)=\sum_{i=T}^{\infty}\gamma^{i-T}[r(s^{\tau^{g+}}_{i+1},g)+\delta_{\bar{\theta}}(s^{\tau^{g+}}_{i},a^{\tau^{g+}}_{i},g)]$
(12)

Given that $\tau^{g+}$ is a successful trajectory, we assume that for all
subsequent states following the policy, the rewards satisfy
$r(s^{\tau^{g+}}_{i+1},g)=0$ for all $i\geq T$. This assumption allows us to
simplify Eq. (12) as follows:

$Q_{\bar{\theta}}(s^{{\tau}^{g+}}_{T},\pi(s^{{\tau}^{g+}}_{T},g),g)=\sum_{i=T}^{\infty}\gamma^{i-T}\delta_{\bar{\theta}}(s^{\tau^{g+}}_{i},a^{\tau^{g+}}_{i},g)$
(13)

Unlike shooting bias, which is inherently constrained by the episode horizon
and generally controllable, shifting bias accumulates over an infinite
horizon. As outlined in Sect. 4, while TD errors $\delta_{\bar{\theta}}$ are
typically minimized towards zero in single-step learning, they often deviate
from this baseline due to the influence of off-policy bias in multi-step
scenarios. Such deviations can be exacerbated by self-loops during iterated
learning, potentially escalating even minor TD errors into significant
shifting biases over the infinite horizon. Consequently, for transitions among
goal states, we prefer reverting to single-step learning to minimize TD errors
in these specific transitions. Simultaneously, we aim to retain the benefits
of multi-step learning for states preceding the goal states.

#### Truncated Multi-Step Targets

To counteract the shifting bias, we introduce the strategy of truncating
multi-step transitions at the first goal state. This approach reverts
transitions among goal states to single-step learning, effectively minimizing
TD errors in these segments. Meanwhile, for transitions that do not reach the
goal state, we maintain the benefits of multi-step learning. The truncated
multi-step target, denoted by $\hat{y}^{(n)}$, is computed as follows:

$\hat{y}^{(n)}(t,\tau,g)=r(s^{\tau}_{t+1},g)+\begin{cases}\gamma\hat{y}^{(n-1)}(t+1,\tau,g)&\text{if
}r(s^{\tau}_{t+1},g)\neq 0\text{ and }n>1,\\\ \gamma
Q_{\bar{\theta}}(s_{t+1}^{\tau},\pi(s_{t+1}^{\tau},g),g)&\text{otherwise}.\end{cases}$
(14)

This method effectively reduces shifting bias, thereby ensuring stable
learning targets for preceding state-action pairs. As will be demonstrated in
our subsequent experimental results, this approach contributes significantly
to the stability and efficacy of the learning process.

###  5.3 Bias Resilient Algorithm

Building upon the technical solutions detailed in previous sections, we have
developed the Bias-Resilient MHER (BR-MHER) algorithm, which is specifically
designed to effectively mitigate both shooting and shifting biases in multi-
step GCRL. To enhance the bias mitigation capabilities of BR-MHER, we have
integrated the MHER($\lambda$) technique (Yang et al., 2021). This addition
allows for the interpolation of multi-step targets across various time steps,
further refining the target value derivation process.

With the integration of MHER($\lambda$), BR-MHER employs the truncated multi-
step target $\hat{y}^{(n)}$ as described in Eq. (14), and calculates the new
target value as follows:

$\hat{y}^{(n)}_{\lambda}(t,\tau,g)=\frac{\sum_{i=1}^{n}\lambda^{i}\hat{y}^{(i)}(t,\tau,g)}{\sum_{i=1}^{n}\lambda^{i}}.$
(15)

A comprehensive outline of the BR-MHER algorithm, incorporating these
strategies, is thoughtfully provided in Algorithm 1 for a clear understanding
and ease of implementation.

Algorithm 1 Bias Resilient MHER (BR-MHER)

1:Environment $E$, Replay buffer $\mathcal{B}$, mini-batch size $b$, reward
function $r$, policy $\pi_{\psi}$, quantile $\rho$, Huber loss threshold
$\kappa$, action-value functions $Q_{\theta}^{\rho}$, hindsight relabeling
function $h$.

2:for $episode=1,\dots,N$ do

3: Initialize environment $E$ to obtain initial state $s_{0}$ and goal $g$.

4: for $t=0,\dots,T-1$ do

5: Execute action $a_{t}\sim\pi(s_{t},g)$ and observe the resultant state
$s_{t+1}$.

6: Store transition $(s_{t},a_{t},s_{t+1},g)$ in $\mathcal{B}$.

7: Update state to $s_{t}\leftarrow s_{t+1}$.

8: end for

9: Sample a batch
$B=\\{(s_{t_{i}}^{\tau_{i}},a_{t_{i}}^{\tau_{i}},s_{t_{i}+n}^{\tau_{i}},g_{\tau_{i}})\\}_{i=1}^{b}$
from $\mathcal{B}$.

10: Apply hindsight relabeling with $\hat{g}_{\tau_{i}}\sim h(t_{i},\tau_{i})$
and set $g_{\tau_{i}}\leftarrow\hat{g}_{\tau_{i}}$.

11: Compute the target $\hat{y}_{\lambda}^{(n)}(t_{i},\tau_{i},g_{\tau_{i}})$
using Eq. (15).

12: Optimize the expected critic loss $L_{QR}(\theta)$ according to Eq. (11)
using batch $B$ to update $\theta$.

13: Optimize the expected actor loss $L(\psi)$ as per Eq. (1) using batch $B$
to update $\psi$.

14:end for

###  5.4 Bias Measurement

In analyzing off-policy bias, we aim to quantify shooting and shifting biases
as in Eq. (8), introducing metrics Terminal Shifting Bias (TSB) and Initial
Shooting Bias (ISB) defined as follows:

$\displaystyle TSB$
$\displaystyle=\mathbb{E}_{g,\tau^{g+}}\left[Q_{\theta}(s^{\tau^{g+}}_{T},a^{\tau^{g+}}_{T},g)\right],$
(16) $\displaystyle ISB$
$\displaystyle=\mathbb{E}_{g,\tau^{g+}}\left[Q_{\theta}(s^{\tau^{g+}}_{0},a^{\tau^{g+}}_{0},g)-\sum_{i=0}^{T-1}\gamma^{i}r(s_{i+1}^{\tau^{g+}},g)\right]-\gamma^{T}\cdot
TSB.$ (17)

In the metrics, TSB is calculated as the average shifting bias of the sampled
on-policy evaluation trajectories. Furthermore, ISB is designed to capture the
shooting bias
$\sum_{i=0}^{T-1}\delta_{\bar{\theta}}(s_{i}^{\tau^{g+}},a_{i}^{\tau^{g+}},g)$
which is accumulated over the entire episode horizon. According to Eq. (6), we
calculate ISB by subtracting the return and the discounted shifting bias from
the action-value of the initial state-action pair, leading to Eq. (17).
Contrasting with the approach of averaging shooting biases across all state-
action pairs, ISB effectively mitigates the diluting effects of such averaging
and offers a more precise reflection of the overall severity of shooting bias
in the action-value function.

##  6 Experiments

In this section, we evaluate the performance of our BR-MHER algorithm. The
evaluation metrics for the algorithm include sampling efficiency, success
rates and off-policy biases. The performance of our approaches is further
exemplified through a comparative investigation. In our experiments, we use
six robotic tasks involving three distinct robotic agents: i) Fetch robotic
arm (Plappert et al., 2018): FetchPickAndPlace and FetchSlide; ii)
Anthropomorphic robotic hand (Plappert et al., 2018): HandReach and
HandManipulateBlockRotateXYZ; iii) Sawyer robotic arm (Nair et al., 2018):
SawyerPushAndReachEnvEasy and SawyerReachXYZEnv. We also apply our methods in
the grid-world tasks Simple-MiniGrid-Empty (Chevalier-Boisvert et al., 2018)
of scales 25$\times$25 and 50$\times$50\. For more details about these tasks,
see Appendix B

###  6.1 Baselines

For a comparative study, we include six experimental baselines: i) HER
(Andrychowicz et al., 2017): A straightforward GCRL technique using one-step
targets; ii) MHER (Yang et al., 2021): An expanded version of HER learning
through multi-step targets; iii) MHER($\lambda$) (Yang et al., 2021): A
balanced method similar to TD($\lambda$) (Sutton & Barto, 2018), harmonizing
multi-step targets at various steps via the $\lambda$ parameter; iv) MMHER
(Yang et al., 2021): A model-based approach, calculating on-policy multi-step
targets with a world model; v) IS-MHER: A integration of MHER and importance
sampling method Retrace($\lambda$) (Munos et al., 2016); vi) WGCSL (Yang et
al., 2022): An advanced goal-conditioned supervised learning algorithm with
meticulously designed weighting mechanisms.

###  6.2 Experimental Settings and Implementation

#### Experimental Settings

Our experiments aim to answer the following research questions: Q1) How
effectively does our BR-MHER manage off-policy bias in multi-step GCRL? Q2)
How do our bias mitigation techniques impact learning efficiency? Q3) Does BR-
MHER maintain competitive performance with state-of-the-art methods such as
IS-MHER and WGCSL in terms of success rate and other relevant metrics? Q4) How
do quantile regression and truncated multi-step targets individually and in
combination contribute to mitigating off-policy bias in BR-MHER?

We conduct comprehensive experiments, benchmarking our BR-MHER against the
baselines across a variety of robotic tasks with $n=3,5,7$, and grid-world
tasks with $n=3,5,10$. To ensure the reliability of our results, we perform
five trials with distinct seeds for each task. Training is stopped after 50
epochs, and the performance is assessed based on the success rate along with
the bias and variance metrics, as elaborated in Section 5.4.

#### Implementation

We utilize a modified OpenAI Baselines codebase (Dhariwal et al., 2017),
following the recommendations made by Yang et al. (Yang et al., 2021). To
mitigate over-estimation bias, we adopt the Clipped Double Q-learning (CDQ)
and delayed policy updates from TD3 (Fujimoto et al., 2018) for all GCRL
methods. Our implementation utilizes Multilayer Perceptrons (MLPs) for both
actor and critic networks. For a balanced comparison with quantile regression,
we used Huber loss for all GCRL methods. The upper quantile ($\rho$) is fixed
at 0.75 to balance capturing the upper quantile and adjusting target value
distribution shifts. The Huber loss threshold ($\kappa$) is set at 10.
Detailed specifications of these hyper-parameters and distinct network
configurations for various tasks are in Appendix B.

(a)

(b)

(c)

Figure 2: Comparative study of seven-step ($n$=7) GCRL methods on robotic
tasks and ten-step ($n$=10) GCRL methods on grid-world tasks. (a) Success
rate. (b) ISB for measuring shooting bias. (c) TSB for measuring shifting
bias.

###  6.3 Experiment Results

Due to the limited space, we report only the statistical results (mean and
standard deviation) across five random seeds for the multi-step GCRL methods
with the largest $n$, specifically $7$ for robotic tasks and $10$ for grid-
world tasks. Additional results are in Appendix C.

#### Comparison with HER

As illustrated in Fig. 2(a), BR-MHER achieves higher success rates than HER in
four tasks and equivalent performance in the remaining tasks. Additionally,
BR-MHER outperforms HER in learning efficiency on six tasks and matches
efficiency in both FetchSlide-v1 and HandManipulateBlockRotateXYZ-v0 tasks.
Our method attains a shooting bias comparable to HER, as denoted by ISB in
Fig. 2(b). Regarding shifting bias, highlighted by TSB in Fig. 2(c), BR-MHER
records smaller values in six tasks and analogous values in the other two,
signifying that even unbiased single-step learning may accumulate errors,
detectable by TSB, over an extensive horizon. It also demonstrates that our
method effectively mitigates such intrinsic bias in GCRL, alongside off-policy
shifting bias.

#### Comparison with MHER

MHER, a baseline multi-step GCRL method, exhibits high sensitivity to off-
policy bias. Fig. 2(a) demonstrates the superior performance of BR-MHER in
success rates and learning efficiency across six tasks. However, in cases
where both methods achieve similar success rates, MHER holds a slight edge in
learning efficiency, exclusively in the Sawyer tasks. A detailed bias analysis
further highlights consistently larger off-policy bias magnitudes in all tasks
by MHER, corroborated by the ISB and TSB metrics in Fig. 2(b) and 2(c). This
bias issue, however, is least severe in the Sawyer tasks, allowing MHER to
maintain decent performance.

#### Comparison with MMHER

MMHER is model-based, in contrast to the model-free approach of BR-MHER. Fig.
2(a) shows that BR-MHER generally outperforms or matches MMHER across various
tasks, except in FetchSlide-v1. This task’s unique nature, where actions have
a limited impact once an object is slid, allows future states to predominantly
dictate the return. The world model can swiftly learn these types of
transitions, granting MMHER a notable learning advantage. Despite this, BR-
MHER demonstrates reduced magnitudes of TSB and ISB across all tasks,
indicative of lower off-policy biases as seen in Fig. 2(b) and 2(c). While
MMHER employs on-policy rollouts to avoid these biases, it contends with
accumulating model-prediction errors, evidenced in the Fetch tasks in Fig.
2(b), where an initial increase in ISB possibly reflects heightened model-
prediction errors before decreasing as the model better adapts to the
environments.

#### Comparison with MHER($\lambda$)

BR-MHER achieves superior success rates and learning efficiency across five
diverse tasks and maintains comparable performance on the Sawyer tasks, as
demonstrated in Fig. 2(a). Despite a slight disadvantage in the FetchSlide-v1
task, the ablation study in Sect. 6.4 reveals that the degradation occurs with
the introduction of quantile regression. Despite significantly mitigating TSB
bias, it poses a challenge in this specific task. In FetchSlide-v1, a state
within the target object’s sliding process directly determines future rewards
since actions can no longer influence the target object, leading to a largely
deterministic distribution of target values. This scenario causes the quantile
regression to prefer positive network approximation errors in the target
value, culminating in a slightly optimistic shooting bias, as evident by the
ISB shown in Fig. 2(b), and likely hindering learning. Despite this, Figs.
2(b) and 2(c) confirm that BR-MHER significantly lowers both ISB and TSB,
highlighting its ability to mitigate off-policy biases.

#### Comparison with IS-MHER

BR-MHER outperforms IS-MHER in learning efficiency and success rates across
six tasks, showing similar performance in HandManipulateBlockRotateXYZ-v0, but
slightly inferior in FetchSlide-v1, as shown in Fig. 2(a). IS-MHER exhibits
larger TSB and ISB magnitudes, outlined in Figs.2(b) and 2(c). This trend is
partly due to the evaluation of a deterministic policy, derived from the
stochastic policy used in importance sampling, focused on optimizing success
rate evaluations. The disparity is further exacerbated by the use of truncated
importance ratios in Retrace($\lambda$) (Munos et al., 2016).

#### Comparison with WGCSL

Our comparative assessment highlights a notable inconsistency in WGCSL’s
performance across various tasks. Despite exhibiting enhanced learning
efficiency in grid-world tasks, as depicted in Fig. 2(a), it falls behind BR-
MHER in success rates for five robotic tasks. As observed in Fig. 2(c), BR-
MHER holds smaller magnitudes of ISB, indicating less severe shooting bias
compared to IS-MHER. Although WGCSL employs an action-value function learned
via single-step learning, it still exhibits significant shifting biases in
certain tasks. This is evidenced by the elevated TSB in FetchPickAndPlace-v1
and HandManipulateBlockRotateXYZ-v0 (Fig. 2(c)), correlating with its subpar
performance. This discrepancy stems from the supervised policy’s insufficient
synchronization with the action-value function’s directives, inducing
substantial policy and target value fluctuations during the learning process.
In contrast, our GCRL methods mitigate this issue, demonstrating reduced TSB
in robotic tasks. In grid-world tasks, WGCSL displays markedly lower TSB,
indicating diminished fluctuations. In Simple-MiniGrid-Empty-50x50, BR-MHER
encounters notable TSB variation among different seeds, leading to a stark
success rate oscillation between over 95% and approximately 30%. This
variation might be attributed to the imbalance between learning action-values
in goal states with low variance of target values and states preceding the
goals with higher target value variance. This imbalance occasionally prevents
the effective minimization of TD errors in goal states, leading to a high
magnitude of shifting bias. This observed variation in success rates and the
shifting bias underscores BR-MHER’s limitations in handling shifting bias
effectively in this specific scenario.

In response to the first three research questions: A1) BR-MHER substantially
minimizes both shooting and shifting bias compared to MHER($\lambda$) and
other GCRL methods, as evidenced by the ISB and TSB metrics. A2) BR-MHER
generally excels over MHER($\lambda$), achieving enhanced success rates and
learning efficiency across multiple tasks, albeit with a slight setback in the
FetchSlide-v1 task, potentially attributed to over-optimistic shooting bias
induced by quantile regression. A3) While IS-MHER and WGCSL exhibit superior
performance in specific tasks (FetchSlide-v1 and two grid-world tasks
respectively), BR-MHER consistently delivers solid and reliable performance
across all other tasks and experiments, highlighting its robust and adaptable
nature in various contexts.

###  6.4 Ablation Study

This ablation study is systematically designed to unravel the unique
contributions of each component embedded within the BR-MHER algorithm. It
chiefly emphasizes their roles in the reduction of bias and enhancement of
learning efficiency.

As delineated in the primary text, quantile regression and truncated multi-
step targets emerge as pivotal techniques, seamlessly integrated within the
BR-MHER algorithm. To discern the solitary impact of each, the study explores
two algorithmic variants: QR-MHER and TMHER($\lambda$). QR-MHER singularly
integrates quantile regression, while TMHER($\lambda$) exclusively embodies
truncated multi-step targets, each within the foundational MHER($\lambda$)
framework. This methodical investigation aims to shed light on the nuanced
influences each component imparts on the overarching performance of the BR-
MHER algorithm.

Advancing into the ablation study, six methods, namely HER, MHER,
MHER($\lambda$), TMHER($\lambda$), QR-MHER, and BR-MHER, are meticulously
analyzed. The discerned outcomes are pictorially represented in Fig. 3.

(a)

(b)

(c)

Figure 3: Ablation study of seven-step ($n$=7) GCRL methods on robotic tasks
and ten-step (n=10) GCRL methods on grid-world tasks. (a) Success rate. (b)
ISB for measuring shooting bias. (c) TSB for measuring shifting bias.

From Fig. 3(a), it is evident that MHER($\lambda$) consistently enhances the
stability of MHER across diverse tasks and step sizes, thereby providing a
more stable performance trajectory.

Our variant, TMHER($\lambda$), stands as a further consistent enhancement of
MHER($\lambda$), universally exhibiting positive advancements in success
performance, learning efficiency, and off-policy bias mitigation over
MHER($\lambda$) across all cases. This underscores the favorable impact of
employing truncated multi-step targets. As delineated in Figs. 3(a),
significant improvements with TMHER($\lambda$) are notably observed in three
tasks: HandReach-v0 and two grid-world tasks. In these scenarios, actions can
inadvertently deflect the agent from goal coordinates, culminating in negative
rewards in the bootstrap path and an amplified shifting bias. In such
scenarios, maintaining position at goal states necessitates a self-loop in
state transitions, which can exacerbate shifting bias significantly as TD
errors increase due to off-policy data. Addressing this issue by switching to
single-step learning for transitions among goal states, as facilitated by
truncated multi-step targets, results in a marked improvement in task
performance. It is imperative to note that TMHER($\lambda$) outperforms
MHER($\lambda$) in FetchSlide-v1, a task where the comprehensive integration
of BR-MHER falls short. This observation highlights the potential detrimental
impact of quantile regression on the performance within the FetchSlide-v1
context. Concurrently, despite the improvements in off-policy biases with
TMHER($\lambda$) over MHER($\lambda$), Figs. 3(b,c) reveal that off-policy
biases, as indicated by ISB and TSB, maintain high levels in tasks such as
FetchSlide-v1 and HandManipulateBlockRotateXYZ-v0. This highlights the
circumscribed capacity of TMHER to diminish both shooting and shifting biases.

In the examination of QR-MHER’s performance as depicted in Fig. 3(a), QR-MHER
is observed to markedly excel over TMHER($\lambda$) in both
FetchPickAndPlace-v1 and HandManipulateBlockRotateXYZ-v0. This superiority
highlights the advantageous management of shooting bias. Observations from
Fig. 3(b,c) denote that QR-MHER markedly diminishes the off-policy biases of
MHER($\lambda$) across all tasks, with the exception of HandReach-v0 and grid-
world tasks. In these exceptional tasks, TMHER($\lambda$) demonstrates
superior bias mitigation capabilities. This observed pattern points to a
potential limitation in QR-MHER’s capability to adequately address the shift
in action-values toward biased values within goal states, which can result in
unstable learning and reduced efficiency.

Consequently, our BR-MHER amalgamates the advantages of both QR-MHER and
TMHER($\lambda$) seamlessly. This integration leads to a superior performance
that transcends the peak efficiencies of both QR-MHER and TMHER($\lambda$). In
HandReach-v0 and grid-world tasks, as depicted in Fig. 3(a), BR-MHER
demonstrates an augmented performance compared to the two variants above.
Additionally, the magnitudes of both ISB and TSB are notably reduced,
indicative of enhanced off-policy bias mitigation, as evident in Fig. 3(b,c).
Notably, in these tasks where TMHER($\lambda$) surpasses QR-MHER, BR-MHER
further diminishes both off-policy biases. This highlights the effective
management of both shifting and shooting biases by quantile regression when
paired with truncated multi-step targets. Despite a noted performance decline
in FetchSlide-v1 due to quantile regression, BR-MHER maintains robust
performance across a variety of tasks.

To address the fourth research question: A4) Our findings reveal that quantile
regression effectively reduces shooting bias and is crucial for enhancing
performance in both FetchPickAndPlace-v1 and HandManipulateBlockRotateXYZ-v0.
Truncated multi-step targets prove to be vital for success in HandReach-v0 and
grid-world tasks, particularly in scenarios with pronounced shifting bias.
Furthermore, the combination of truncated multi-step targets with quantile
regression demonstrates an augmented ability to mitigate off-policy bias
issues, indicating their synergistic rather than independent functionality.

##  7 Conclusion

This paper provides a robust exploration of the off-policy bias challenges
inherent in multi-step GCRL. By uncovering and thoroughly analyzing two types
of off-policy biases present in multi-step GCRL, we have shed light on
innovative techniques for regulating these biases. This led to the inception
of our resilient multi-step GCRL algorithm, BR-MHER. Combined metrics,
encapsulating aspects of performance and off-policy bias, robustly showcases
the resilience and robustness of our BR-MHER against bias issues, especially
as the $n$ increases in the $n$-step targets. Notably, our experimental
evaluations robustly demonstrate the superiority of BR-MHER over the state-of-
the-art MHER($\lambda$) and other advanced multi-step baselines for a majority
of the GCRL benchmarks in most tasks, both in terms of success rate and
learning efficiency. Although our analysis primarily pertains to a
deterministic environment, it underscores the potential of harnessing
beneficial off-policy bias instead of avoiding all off-policy bias altogether.

The presented research underpins the critical need for discerning the nature
of biases in multi-step GCRL and postulates that not all biases are
detrimental. Recognizing and leveraging these biases can be transformative,
carving a path for superior, adaptive, and efficient algorithms. It redefines
the approach towards bias in multi-step GCRL, potentially unlocking expansive
applications across a myriad of complex environments. Moreover, while our
approach demonstrates robust performance across a variety of tasks, our
observations in certain specific scenarios highlight opportunities for further
refinement and enhancement of our strategies. Our ongoing work aims to explore
a more robust strategy for off-policy bias mitigation and extend our approach
to stochastic environments.

## Appendix

##  Appendix A Background

This appendix offers an in-depth exploration of the technical aspects of TD3,
HER, MHER, MHER($\lambda$), MMHER, Retrace($\lambda$), and WGCSL.
Additionally, we furnish a detailed derivation of the learning objective of TD
error in multi-step GCRL, as delineated in Eq. (3) from the main manuscript.

###  A.1 Twin Delayed Deep Deterministic Policy Gradient (TD3)

TD3 (Fujimoto et al., 2018) is an off-policy reinforcement learning method
with an actor-critic architecture designed for continuous control tasks. The
actor $\pi_{\psi}$ is parameterized by $\psi$, and there are two critics
$Q_{\theta_{1}}$ and $Q_{\theta_{2}}$ parameterized by $\theta_{1}$ and
$\theta_{2}$, respectively. Given a data tuple $(s,a,s^{\prime},r)$, the
target value is calculated as

$y(s,a,s^{\prime},r)=r+\gamma\min_{i=1,2}Q_{\bar{\theta}_{i}}(s^{\prime},\pi_{\bar{\psi}}(s^{\prime}))$

This approach is also known as the Clipped Double Q-learning (CDQ) algorithm
used to mitigate over-optimistic bias in the learning process. Unlike
traditional actor-critic algorithms that update the actor and critics
simultaneously, TD3 proposes to delay the actor updates until the action-value
errors are small. This delay is typically implemented by setting the actor
update period $k$ to be larger than one. The training loss for the critic is
represented by the expected $L2$ loss between the $Q$-value predictions and
the target values $y$ as

$L_{\theta_{i}}=\mathbb{E}_{(s,a,s^{\prime},r)\sim\mathcal{B}}[(y(s,a,s^{\prime},r)-Q_{\theta_{i}}(s^{\prime},a))^{2}],~{}i\in\\{1,2\\}$

Every $k$ training steps, the actor is optimized by maximizing the $Q$-value
of the action output $\pi_{\psi}(s)$ for the given states, and the
corresponding loss is

$L_{\psi}=-\mathbb{E}_{s\sim\mathcal{B}}[Q_{\theta_{1}}(s,\pi_{\psi}(s))]$

In the following discussion of algorithms, we omit parameters when they are
not necessary, prioritizing clarity and simplicity.

###  A.2 Hindsight Experience Replay (HER)

HER (Andrychowicz et al., 2017) proposes to tackle sparse learning signals in
sparse-reward settings via relabeling the goal as the actually achieved goals
in the future. Consider a goal-conditioned data tuple
$(s^{\tau}_{t},a^{\tau}_{t},s^{\tau}_{t+1},g^{\tau})$, extracted from the
trajectory $\tau$. The goals actually achieved in the future constitute the
set $\\{\phi(s^{\tau}_{t^{\prime}})\\}_{t^{\prime}=t+1}^{T}$. Under HER, the
original goal $g^{\tau}$ is relabeled as $\hat{g}^{\tau}$, selected from the
distribution $p(\\{\phi(s^{\tau}_{t^{\prime}})\\}_{t^{\prime}=t+1}^{T})$. This
distribution is dependent on the hindsight relabeling strategy and is
typically a uniform distribution over
$\\{\phi(s^{\tau}_{t^{\prime}})\\}_{t^{\prime}=t+1}^{T}$.

Consequently, the agent learns from the relabeled data tuple
$(s^{\tau}_{t},a^{\tau}_{t},s^{\tau}_{t+1},\hat{g}^{\tau})$ with a
recalculated reward $r(s^{\tau}_{t+1},\hat{g}^{\tau})$. This process ensures
that the agent, at some point, receives valuable rewards propagated from the
relabeled goal $\hat{g}^{\tau}$. The target value based on the relabeled goal
is computed as:

$y(s^{\tau}_{t},a^{\tau}_{t},s^{\tau}_{t+1},\hat{g}^{\tau})=r(s^{\tau}_{t+1},\hat{g}^{\tau})+\gamma
Q_{\pi}(s^{\tau}_{t+1},\pi(s^{\tau}_{t+1},\hat{g}^{\tau}),\hat{g}^{\tau}),$
(A1)

where the subscript $\pi$ of $Q_{\pi}$ denotes that the target values for the
action-value function are bootstrapped with the action decision from policy
$\pi$. With the hindsight relabeling strategy for trajectory $\tau$ at time
$t$ represented by a stochastic function $h(t,\tau)$, where the relabeled goal
$\hat{g}^{\tau}$ is sampled from the distribution defined by $h(t,\tau)$, the
loss for the goal-conditioned action-value function can be expressed as:

$L_{Q_{\pi}}=\mathbb{E}_{(s^{\tau}_{t},a^{\tau}_{t},s^{\tau}_{t+1},g^{\tau}_{t})\sim\mathcal{B},\hat{g}^{\tau}\sim
h(t,\tau)}[(y(s^{\tau}_{t},a^{\tau}_{t},s^{\tau}_{t+1},\hat{g}^{\tau})-Q_{\pi}(s^{\tau}_{t},a^{\tau}_{t},\hat{g}^{\tau}))^{2}].$
(A2)

Then the goal-conditioned policy learns to optimize the expected $Q$-value
function of $\pi(s^{\tau}_{t},\hat{g}^{\tau})$ for given states $s^{\tau}_{t}$
and corresponding goals $\hat{g}^{\tau}$ as

$L_{\pi}=\mathbb{E}_{(s^{\tau}_{t},g^{\tau}_{t})\sim\mathcal{B},\hat{g}^{\tau}\sim
h(t,\tau)}[-Q_{\pi}(s^{\tau}_{t},\pi(s^{\tau}_{t},\hat{g}^{\tau}),\hat{g}^{\tau}_{t})].$
(A3)

###  A.3 Multi-Step Hindsight Experience Replay (MHER)

In comparison to HER, MHER (Yang et al., 2021) integrates HER into multi-step
GCRL with multi-step relabeling. Unlike Eq. (A1) that calculates the target
value on a single-step transition, MHER calculates the target values on
$n$-step transitions with the goal $g$ as follows

$y^{(n)}(t,\tau,g)=\sum_{i=0}^{n-1}\gamma^{i}r(s^{\tau}_{t+i+1},g)+\gamma^{n}Q_{\pi}(s^{\tau}_{t+n},\pi(s^{\tau}_{t+n},g),g).$
(A4)

MHER optimizes the actor and critic in similar ways as Eq. (A2) and Eq. (A3)
except the target values are calculated via Eq. (A4). MHER enables the agent
to learn faster from $n$-step targets.

###  A.4 MHER($\lambda$)

Inspired by TD($\lambda$) (Sutton & Barto, 2018), MHER($\lambda$) (Yang et
al., 2021) provides a novel approach to balance the trade-off between fewer-
step targets that come with lower off-policy bias and multi-step targets,
which carry more reward information. This balancing act is achieved through
the introduction of an exponential decay weight parameter $\lambda$, where
$\lambda\in[0,1]$. The multi-step targets, balanced by the $\lambda$
parameter, are calculated using the following formula:

$y^{(n)}_{\lambda}(t,\tau,g)=\frac{\sum_{i=1}^{n}\lambda^{i}y^{(i)}(t,\tau,g)}{\sum_{i=1}^{n}\lambda^{i}}.$
(A5)

In this formulation, $y^{(n)}_{\lambda}$ leans towards the one-step target as
$\lambda\rightarrow 0$, while higher weights are assigned to the $n$-step
target as $\lambda$ increases.

###  A.5 Model-Based MHER (MMHER)

MMHER (Yang et al., 2021) calculates on-policy multi-step targets from multi-
step transitions generated via a world-model $M$ of the environment. The
world-model $M$ is trained by past experiences. Given a one-step transition
$(s^{\tau}_{t},a^{\tau}_{t},s^{\tau}_{t+1})$, the agent simulates $(n-1)$-step
on-policy transitions starting from $s^{\tau}_{t+1}$, following policy $\pi$,
in order to pursue the goal $g$ through the world-model $M$. Within these
$(n-1)$-step on-policy transitions, the $i$th ($i=1\cdots n-1$) state
generated from $M$ is denoted as $M^{\pi_{g}}_{i}(s^{\tau}_{t+1})$,
corresponding to the original $s^{\tau}_{t+1+i}$ in the off-policy data. Then,
the multi-step target in MMHER is expressed as

$\displaystyle
y^{(n)}_{M}(s^{\tau}_{t},a^{\tau}_{t},s^{\tau}_{t+1},g)=r(s^{\tau}_{t+1},g)+$
$\displaystyle\sum_{i=1}^{n-1}\gamma^{i}r(M^{\pi_{g}}_{i}(s^{\tau}_{t+1}),g)$
$\displaystyle+$
$\displaystyle\gamma^{n}Q_{\pi}(M^{\pi_{g}}_{n-1}(s^{\tau}_{t+1}),\pi(M^{\pi_{g}}_{n-1}(s^{\tau}_{t+1}),g),g).$
(A6)

However, $y^{(n)}_{M}$ actually suffers from model bias caused by the
prediction errors of $M$, which can be severe when the problems are complex.
To balance the model bias and learning information, Yang et al. (2021)
proposes to integrate $n$-step model-based target and one-step target via a
hyper-parameter $\alpha$ as

$\bar{y}^{(n)}_{M}(s^{\tau}_{t},a^{\tau}_{t},s^{\tau}_{t+1},g)=\frac{\alpha
y^{(n)}_{M}(s^{\tau}_{t},a^{\tau}_{t},s^{\tau}_{t+1},g)+r(s^{\tau}_{t+1},g)+\gamma
Q_{\pi}(s^{\tau}_{t+1},\pi(s^{\tau}_{t+1},g),g)}{1+\alpha}.$ (A7)

The $\bar{y}^{(n)}_{M}(s^{\tau}_{t},a^{\tau}_{t},s^{\tau}_{t+1},g)$ will serve
as the final $n$-step targets for the action-value function to learn.

###  A.6 Weighted Goal-Conditioned Supervised Learning (WGCSL)

The Weighted Goal-Conditioned Supervised Learning (WGCSL) algorithm, as
delineated by Yang et al. (2022), stands as a supervised learning methodology
designed for the execution of goal-conditioned tasks. This innovative
algorithm employs goal relabeling techniques analogous to those used in
Hindsight Experience Replay (HER) (Andrychowicz et al., 2017).

The formal supervised learning objective, $L_{\psi}^{WGCSL}$, is expressed as:

$L_{\psi}^{WGCSL}=\mathbb{E}_{(s^{\tau}_{t},a^{\tau}_{t},g^{\tau}_{t})\sim\mathcal{B},\phi(s^{\tau}_{i})\sim
h(t,\tau)}[w^{\tau}_{t,i}\log~{}\pi(a^{\tau}_{t}|s^{\tau}_{t},\phi(s^{\tau}_{i}))].$

In this equation, $s^{\tau}_{i}$ denotes the prospective state of
$s_{t}^{\tau}$ where $i>t$, with the relabeled goal represented as
$\phi(s^{\tau}_{i})$, and the weight $w^{\tau}_{t,i}$ accentuating the
prioritization of actions possessing a larger potential.

The weight $w^{\tau}_{t,i}$ is further defined as:

$w^{\tau}_{t,i}=\gamma^{i-t}\cdot\text{clip}(\exp(A(s^{\tau}_{t},a^{\tau}_{t},\phi(s^{\tau}_{i}))),0,K)\cdot\epsilon(A(s^{\tau}_{t},a^{\tau}_{t},\phi(s^{\tau}_{i}))).$
(A8)

Here, $\gamma^{i-t}$,
$\text{clip}(\exp(A(s^{\tau}_{t},a^{\tau}_{t},\phi(s^{\tau}_{i}))),0,K)$, and
$\epsilon(A(s^{\tau}_{t},a^{\tau}_{t},\phi(s^{\tau}_{i})))$ symbolize the
discounted relabeling weight (DRW), goal-conditioned exponential advantage
weight (GEAW), and best-advantage weight (BAW) respectively. The DRW
prioritizes learning based on the actions of states temporally closer to the
relabeled goal. The GEAW emphasizes learning on actions with a higher
advantage, employing $K$ to limit values in extreme cases. The BAW addresses
the challenge of multi-modality in goal-conditioned Reinforcement Learning
(RL), guiding the policy towards the modal with the maximal return,
circumventing convergence to a weighted average of multiple modals. BAW is
specifically formulated as:

$\epsilon(A(s^{\tau}_{t},a^{\tau}_{t},\phi(s^{\tau}_{i})))=\begin{cases}1,&\text{if
}A(s^{\tau}_{t},a^{\tau}_{t},\phi(s^{\tau}_{i}))>\hat{A}\\\
\epsilon_{\text{min}},&\text{otherwise}\end{cases}$

Within this context, $\hat{A}$ denotes a threshold calculated from recent
advantage statistics, while $\epsilon_{\text{min}}$ represents a minimal
positive value.

###  A.7 Retrace($\lambda$)

Retrace($\lambda$) is a reinforcement learning algorithm specifically designed
for off-policy correction of the value estimates (Munos et al., 2016). The
algorithm employs importance sampling (IS) for correction, incorporating a
safety mechanism to ensure the stability and robustness of learning. This
mechanism involves truncating the importance sampling ratios to mitigate
excessive variance in the estimates.

The $n$-step goal-conditioned target value based on Retrace($\lambda$) is
calculated as

$y^{(n)}_{\text{ret}}(t,\tau,g)=\sum_{i=1}^{n}\gamma^{i-1}\left(\prod_{s=1}^{i-1}c_{s}\right)\left(r(s^{\tau}_{t+i},g)+\gamma\mathbb{E}_{\pi}[Q_{\pi}(s^{\tau}_{t+i},\cdot)]-\gamma
c_{i}Q(s^{\tau}_{t+i},a^{\tau}_{t+i})\right),$

where $c_{i}$ is the truncated importance sampling ratio. It is defined that
$\prod_{s=1}^{0}c_{s}=1$ and $c_{n}=0$ as the experience beyond the $n$ steps
is truncated. Specifically, the importance sampling ratio $c_{t}$ is:

$c_{i}=\lambda\min\left(1,\frac{\pi(a^{\tau}_{t+i}|s^{\tau}_{t+i})}{\mu(a^{\tau}_{t+i}|s^{\tau}_{t+i})}\right),$
(A9)

where $\mu(a_{t}|s_{t})$ is the behavior policy that generated the action
$a_{t}$ at state $s_{t}$. The truncated importance sampling enables the agent
to safely use off-policy data to correct the estimates.

###  A.8 TD Error Learning Objective in Multi-Step GCRL

This section meticulously delineates the derivation of the objective, depicted
as Eq. (3) in the main text.

$\displaystyle Q_{\theta}(s^{\tau}_{t},a^{\tau}_{t},g)$ $\displaystyle=$
$\displaystyle~{}r_{t}+\gamma
Q_{\bar{\theta}}(s^{\tau}_{t+1},\pi_{\bar{\psi}}(s^{\tau}_{t+1},g),g)-\delta_{\theta}(s^{\tau}_{t},a^{\tau}_{t},g)$
$\displaystyle=$ $\displaystyle~{}r_{t}+{\color[rgb]{0,0,1}\gamma
Q_{\bar{\theta}}(s^{\tau}_{t+1},a^{\tau}_{t+1},g)}-\gamma
A_{\bar{\theta}}(s^{\tau}_{t+1},a^{\tau}_{t+1},g)-\delta_{\theta}(s,a,g)$
(A10) $\displaystyle=$
$\displaystyle~{}r_{t}+{\color[rgb]{0,0,1}\gamma\left[r_{t+1}+\gamma
Q_{\bar{\theta}}(s^{\tau}_{t+2},\pi_{\bar{\psi}}(s^{\tau}_{t+2},g),g)-\delta_{\bar{\theta}}(s_{t+1}^{\tau},a_{t+1}^{\tau},g)\right]}-\gamma
A_{\bar{\theta}}(s^{\tau}_{t+1},a^{\tau}_{t+1},g)$
$\displaystyle-\delta_{\theta}(s^{\tau}_{t},a^{\tau}_{t},g)$ (A11)
$\displaystyle=$ $\displaystyle~{}\cdots$ $\displaystyle=$
$\displaystyle~{}\sum_{i=0}^{n-1}\gamma^{i}r_{t+i}+\gamma^{n}Q_{\bar{\theta}}(s^{\tau}_{t+n},\pi_{\bar{\psi}}(s^{\tau}_{t+n},g),g)-\sum_{i=1}^{n-1}\gamma^{i}[A_{\bar{\theta}}(s^{\tau}_{t+i},a^{\tau}_{t+i},g)+\delta_{\bar{\theta}}(s^{\tau}_{t+i},a^{\tau}_{t+i},g)]$
$\displaystyle-\delta_{\theta}(s^{\tau}_{t},a^{\tau}_{t},g)$ $\displaystyle=$
$\displaystyle~{}y^{(n)}(t,\tau,g)-\sum_{i=1}^{n-1}\gamma^{i}\left[A_{\bar{\theta}}(s^{\tau}_{t+i},a^{\tau}_{t+i},g)+\delta_{\bar{\theta}}(s^{\tau}_{t+i},a^{\tau}_{t+i},g)\right]-\delta_{\theta}(s^{\tau}_{t},a^{\tau}_{t},g).$
(A12)

In Eq. (A10), the highlighted term, $\gamma
Q_{\bar{\theta}}(s^{\tau}_{t+1},a^{\tau}_{t+1},g)$, is purposefully added and
then subtracted by
$Q_{\bar{\theta}}(s^{\tau}_{t+1},\pi_{\bar{\psi}}(s^{\tau}_{t+1},g),g)$,
resulting in $\gamma A_{\bar{\theta}}(s^{\tau}_{t+1},a^{\tau}_{t+1},g)$,
thereby initiating the unrolling process. Following this, Eq. (A11) showcases
the expanded form of the previous term, elucidating the unwinding of the
action-value function’s recursive definition. This expansion includes the
subsequent reward $r_{t+1}$, the ensuing action-value function
$Q_{\bar{\theta}}(s^{\tau}_{t+2},\pi_{\bar{\psi}}(s^{\tau}_{t+2},g),g)$, and
the TD error $\delta_{\bar{\theta}}(s_{t+1}^{\tau},a_{t+1}^{\tau},g)$.
Consequently, Eq.(3) from the main text is directly extrapolated from Eq.
(A12).

##  Appendix B Experimental and Implementation Details

In this appendix, we furnish extensive details regarding our experimental
tasks, implementation specifics, hardware requisites of our experiments, and
information pertinent to our released code.

###  B.1 Environments and Tasks

In this section, we provide comprehensive elaboration on the four unique task
types incorporated into our experimental procedures.

#### Fetch Tasks

The Fetch robotic arm Plappert et al. (2018) represents an advanced, seven-
degree-of-freedom (7-DoF) robotic arm outfitted with a two-pronged parallel
gripper. The goals are denoted by a 3-dimensional vector that specifies the
Cartesian coordinates of the target endpoint. A goal is deemed successfully
achieved if the achieved goal resides within a 5cm radius of the intended
coordinates. Notably, the arm features a four-dimensional continuous action,
which encompasses three Cartesian dimensions to regulate the end effector’s
movement, along with an additional dimension dedicated to controlling the
gripper. For the experimental tasks utilized for our study: i)
FetchPickAndPlace: The agent is required to grasp the target object, relocate
it to the goal position and sustain its position until the termination of the
task; ii) FetchSlide: The task requires the agent to slide a small puck across
the table to a goal position that is outside of the reach of the robot, and
ensure the object remains within the goal location at the end of the task.

#### Hand Tasks

The anthropomorphic robotic Plappert et al. (2018) hand is a 24-DoF robot hand
with a 20-dimensional action space. For the tasks used in our experiments: i)
HandReach: The goal of the task is a 15-dimensional vector corresponding to
the desired Cartesian positions of all five fingertips. The agent is required
to manipulate the robot to reach the states where the mean distance between
five fingertips and the goal positions is less than 1cm. Its state is a
63-dimensional vector; ii) HandManipulateBlockRotateXYZ: The agent is required
to manipulate a block at the palm of the robotic hand to achieve a target
pose. The goal is a 7-dimensional vector including 3-dimensional target
positions and 4-dimensional target rotations. The agent succeeds in this task
if the block differs from the target rotation within 0.1 rad. Its state space
is 61-dimensional.

#### Sawyer Tasks

The Sawyer robotic arm Nair et al. (2018) is a 7-DoF robotic arm. Its end-
effector (EE) is constrained to a 2-dimensional rectangle parallel to a table.
The movements and positioning of this EE are dictated by actions relayed
through a motion capture (mocap) system. For the tasks used in our
experimental study: i) SawyerPushAndReachEnvEasy: This task necessitates the
agent to push a small puck on the table to a target XY position specified by
the 2-dimensional goal. The task operates within a 2-dimensional action space
corresponding to the X and Y axes. The state space for this task, encapsulated
in five dimensions, encompasses both the 3-dimensional coordinates of the EE
and the 2-dimensional positioning of the puck; ii) SawyerReachXYZEnv: This
task demands precise manipulation of the robotic arm’s EE towards a designated
position indicated by Cartesian coordinates. Both the state and the goal
spaces are 3-dimensional, encompassing the current and target coordinates of
the EE. The action associated with this task is a 3-dimensional continuous
variable, each representing the velocity along an axis.

#### Grid-World Task

The grid-world tasks employed in this study have been adapted from Chevalier-
Boisvert et al. (2018) to suit GCRL contexts. In these tasks, the current
state is defined by the agent’s current 2-dimensional coordinates, while the
goal corresponds to the desired coordinates the agent aims to reach. For these
grid-world tasks, we set the agent’s action space to consist of five discrete
actions: upward movement, downward movement, leftward movement, rightward
movement, and maintaining the current position. The tasks Simple-MiniGrid-
Empty-25$\times$25 and Simple-MiniGrid-Empty-50$\times$50 are differentiated
solely by scale, containing 25 and 50 grids along each edge respectively.

The horizon $T$ varies based on the specific task at hand. For instance, the
horizon for tasks such as FetchPickAndPlace, FetchSlide, and HandReach is set
at 50. In contrast, the tasks HandManipulateBlockRotateXYZ,
SawyerPushAndReachEnvEasy, and SawyerReachXYZEnv have a larger horizon of 100.
In the context of grid-world tasks, the horizon is determined as three times
the scale of their respective scales. Consequently, this results in a horizon
of 75 for the task Simple-MiniGrid-Empty-25$\times$25, and extends to 150 for
the task Simple-MiniGrid-Empty-50$\times$50.

###  B.2 Implementation Details

In this section, we detail the implementation of our TD3, GCRL, and WGCSL
methods, outlining the network configurations and hyperparameters used.

#### TD3

TD3 serves as the primary framework for each GCRL method in our experiments,
utilizing CDQ, target networks and delayed actor updates as decribed in Sect.
A.1 to ensure the stability of the learning process. The hyper-parameters
employed in the experiments are shown in Table 1. Notably, in experiments on
grid-world tasks, which operate within a discrete action space, we employ the
GumbelSoftMax activation (Jang et al., 2016) to the policy network outputs.
This approach ensures the action-value function’s differentiability relative
to the actor outputs within these contexts. Observations and goals, serving as
inputs to the actor and critic, are normalized based on historical statistics.
Throughout the training phase, the target networks undergo soft updates with
the parameters of the present actor and critic networks. Each epoch contains
varying numbers of cycles depending on the task: Fetch and Hand tasks comprise
50 cycles per epoch, while Sawyer and grid-world tasks include 10. For
exploration, we apply $\epsilon$-greedy strategies and additionally integrate
Gaussian action noises into the training rollouts of robotic tasks.

#### GCRL Methods

The primary distinctions among GCRL methods stem from the calculation of
target values for the critic, as depicted in Section A. All GCRL methods share
the same relabeling probability of $p_{h}=0.8$, implying an average relabeling
of four out of five data entries. For MMHER, we set $\alpha=0.5$ in Eq. (A6)
and utilize the configurations are outlined in Table 2, following the
suggestions from Yang et al. (2021). For both IS-MHER based on
Retrace($\lambda$) and MHER($\lambda$), we assign a value of 0.7 to $\lambda$
in both Eq. (A9) and Eq.(A5). In IS-MHER where the importance sampling
necessiates a stochastic policy, we establish the stochstic policy via adding
a clipped Gaussian distibuted noises $\epsilon_{a}\sim
clip(\mathcal{N}(0,0.2),-0.5,0.5)$ to the actions. Each experiment is
conducted with five distinct random seeds—$\\{111,222,333,444,555\\}$—and the
statistics of the results are reported.

#### WGCSL

Given the focus of WGCSL on supervised learning, we opt not to integrate TD3
into its framework. Our experimental setup is based on the source code
provided by the author (Yang et al., 2022). The $K$ used for the weight DRW in
Eq. (A8) is set to 10. Apart from the non-application of TD3-specific
parameters and the substitution of Softmax for GumbelSoftmax in grid-world
tasks, all other hyperparameters are consistent with those utilized in other
GCRL methods, see Table 1.

In general, we primarily adhere to the hyper-parameters proposed in the
codebase (Yang et al., 2021; 2022) to maintain consistency and avoid parameter
tuning. This decision stems from our central aim: to determine if our
methodologies can surpass both baseline and contemporary state-of-the-art
methods when subject to identical conditions. Importantly, our strategies are
tailored to tackle the root causes of bias issues, with the intention to
elevate performance through an intrinsic comprehension of these challenges.

Table 1: The hyper-parameters used for TD3 Hyper-parameter | Value
---|---
Actor learning rate | 0.001
Critic learning rate | 0.001
Optimizer | Adam (Kingma & Ba, 2014)
Buffer size | $10^{6}$
Quadratic penalty on actions | 1.0
Observation clip range | [-200, 200]
Normalized observation clip range | [-5, 5]
Random init episodes | 100 (500 for MMHER)
Batch size | 1024
Test episodes per epoch | 120
Discount factor $\gamma$ | $1/T$
$\epsilon$ for $\epsilon$-greedy exploration | 0.3
Gaussian action noise std | 0.2 (robotics)
Relabeling probability | 0.8
Episodes per cycle | 12
Training batches per cycle | 40
Target network update proportion | 0.005
Delayed actor update interval $k$ | 2
Number of hidden layers | 3
Hidden layer size | 256 (robotics), 512 (grid)
Hidden layer activation | ReLU
Actor output activation | Tanh (robotics)
| Softmax + GumbelSoftmax (grid-world)
Training cycles per epoch | 50 (Hand and Fetch tasks)
| 10 (Sawyer and grid-world tasks)
Table 2: The hyper-parameters of the world-model used for MMHER Hyper-parameter | Value
---|---
Learning rate | 0.001
Optimizer | Adam
Number of hidden layers | 8
Hidden layer size | 256
Batch size | 512
Warmup training times | 500

###  B.3 Hardware

In conducting each experiment, we utilized the processing power of a V100 GPU
coupled with eight CPU cores, facilitating efficient model training and
comprehensive data collection from environments.

###  B.4 Code

We have adopted a modified OpenAI Baselines codebase (Dhariwal et al., 2017),
adhering to the guidelines put forth by Yang et al. (2021; 2022). Tensorflow
(Abadi et al., 2016) serves as our chosen deep learning framework for
training. Recognizing the pivotal role of reproducibility in scientific
endeavors, we have rendered our code publicly accessible. Detailed
instructions to aid in the effective usage of the code are provided. This
information can be accessed at https://github.com/BR-MHER/BR-MHER.

(a)

(b)

(c)

Figure I: Comparative study of three-step ($n$=3) GCRL methods on both robotic
tasks and grid-world tasks. (a) Success rate. (b) ISB for measuring shooting
bias. (c) TSB for measuring shifting bias.

(a)

(b)

(c)

Figure II: Comparative study of five-step ($n$=5) GCRL methods on both robotic
tasks and grid-world tasks. (a) Success rate. (b) ISB for measuring shooting
bias. (c) TSB for measuring shifting bias.

##  Appendix C Extended Experiments

In this appendix, we first enrich the comparative analysis of the BR-MHER
application, initially presented in the main text, by conducting a deeper
examination of both the three-step ($n$=3) and five-step ($n$=5) GCRL methods
applied to the same set of tasks. Following this, we conduct an ablation study
to elucidate the components within BR-MHER.

###  C.1 More Results of BR-MHER

This section delineates the remaining comparative analysis of BR-MHER for
$n$=3 and $n$=5, as illustrated in Figs. I and II.

The robust advantages delineated in the main text persist for smaller step
sizes such as 3 or 5. It is observed that BR-MHER consistently mitigates off-
policy bias, as evidenced by the ISB and TSB levels in Figs. I(b, c) and
II(b,c). Here, we can investigate a new research question following the three
in the main text: Q5) How does the performance of BR-MHER change as the step-
size of multi-step GCRL changes? How is it compared to other GCRL methods?
Instead of performing the one-to-one comparison as the main text, we answer in
a general perspective.

As depicted in Figs. I(a) and II(a), BR-MHER overtakes the learning efficiency
of HER in HandManipulateBlockRotateXYZ-v0, although it aligns with HER’s
efficiency in a seven-step learning context. This trend highlights that BR-
MHER is similarly impacted by the detrimental effects of off-policy biases
with increasing step-size. Concurrently, BR-MHER reaps greater advantages than
other GCRL methods in grid-world tasks with step-size augmentation. No
significant performance shifts are noted for other tasks. An analysis of the
performance differential among various multi-step GCRL methods reveals
expanding gaps in FetchPickAndPlace-v1, Hand tasks, and grid-world tasks. This
observation corroborates the enhanced robustness and resilience of BR-MHER to
alterations in step-size and escalating off-policy biases.

Moreover, we find the performance does not decrease monotonically as the step-
size increases. Each multi-step GCRL method seems to have the best step-size
for learning. Take the Simple-MiniGrid-Empty-25x25-v0 for example,
MHER($\lambda$) reaches around 75% success rates with three-step learning,
arising to 85% with five-step while dropping to 40% with ten-step learning. In
contrast, BR-MHER can stably reach 100%. In this case, BR-MHER offers an
robust option to the parameter of step-size. We also notice that there is a
special case in HandManipulateBlockRotateXYZ-v0 that IS-MHER keeps the same
performance as HER when $n=$3, 5 and 7. It probably reduce the multi-step
learning to single-step learning via importance sampling due to the relatively
low entropy of the policy.

Furthermore, the analysis discerns a non-monotonic performance degradation
with step-size amplification. Each multi-step GCRL method appears to have an
optimal step-size for learning. For instance, in the Simple-MiniGrid-
Empty-25x25-v0 task, MHER$(\lambda)$ records approximately 75% success with
three-step learning, which escalates to 85% with five-step learning, but falls
to 40% with ten-step learning. Contrarily, BR-MHER consistently achieves a
100% success rate, underscoring its robustness relative to step-size
parameterization. An anomalous observation in HandManipulateBlockRotateXYZ-v0
shows that IS-MHER mirrors HER’s performance for $n=$3, 5, and 7, potentially
signifying a transition from multi-step to single-step learning via importance
sampling, attributed to the relatively low entropy of the policy.

###  C.2 Ablation Study

In this section, we extend the ablation study analysis of our methods for
$n=3$ and $n=5$, as showcased in Figs. III and IV. The findings discussed in
the main text are largely consistent with these smaller step sizes.

However, a notable exception is observed in Fig. III(a), where QR-MHER shows
enhanced learning efficiency compared to TMHER($\lambda$) in grid-world tasks
with $n=3$. Yet, as $n$ increases, this efficiency advantage diminishes, with
TMHER($\lambda$) outperforming QR-MHER. This shift highlights the increasing
severity of shifting bias with larger step sizes and underscores the
limitations of QR-MHER in addressing this bias alone. It emphasizes the
necessity of integrating truncated multi-step targets to effectively manage
shifting bias.

(a)

(b)

(c)

Figure III: Ablation study of three-step ($n$=3) GCRL methods on both robotic
tasks and grid-world tasks. (a) Success rate. (b) ISB for measuring shooting
bias. (c) TSB for measuring shifting bias.

(a)

(b)

(c)

Figure IV: Ablation study of five-step ($n$=5) GCRL methods on both robotic
tasks and grid-world tasks. (a) Success rate. (b) ISB for measuring shooting
bias. (c) TSB for measuring shifting bias.

## References

  * Abadi et al. (2016) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al.  Tensorflow: a system for large-scale machine learning.  In _Osdi_ , pp. 265–283. Savannah, GA, USA, 2016.
  * Andrychowicz et al. (2017) Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba.  Hindsight experience replay.  _Advances in neural information processing systems_ , 30, 2017.
  * Barth-Maron et al. (2018) Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap.  Distributed distributional deterministic policy gradients.  _arXiv preprint arXiv:1804.08617_ , 2018.
  * Blier & Ollivier (2021) Léonard Blier and Yann Ollivier.  Unbiased methods for multi-goal reinforcement learning.  _arXiv preprint arXiv:2106.08863_ , 2021.
  * Chevalier-Boisvert et al. (2018) Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal.  Minimalistic gridworld environment for openai gym.  https://github.com/maximecb/gym-minigrid, 2018.
  * Dabney et al. (2018a) Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos.  Implicit quantile networks for distributional reinforcement learning.  In _International conference on machine learning_ , pp. 1096–1105. PMLR, 2018a.
  * Dabney et al. (2018b) Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos.  Distributional reinforcement learning with quantile regression.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2018b.
  * De Asis et al. (2018) Kristopher De Asis, J Hernandez-Garcia, G Holland, and Richard Sutton.  Multi-step reinforcement learning: A unifying algorithm.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2018.
  * Dhariwal et al. (2017) Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov.  Openai baselines.  https://github.com/openai/baselines, 2017.
  * Fujimoto et al. (2018) Scott Fujimoto, Herke Hoof, and David Meger.  Addressing function approximation error in actor-critic methods.  In _International conference on machine learning_ , pp. 1587–1596. PMLR, 2018.
  * Harutyunyan et al. (2016) Anna Harutyunyan, Marc G Bellemare, Tom Stepleton, and Rémi Munos.  Q ($\lambda$) with off-policy corrections.  In _International Conference on Algorithmic Learning Theory_ , pp. 305–320. Springer, 2016.
  * Hasselt (2010) Hado Hasselt.  Double q-learning.  _Advances in neural information processing systems_ , 23, 2010.
  * Hessel et al. (2018) Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver.  Rainbow: Combining improvements in deep reinforcement learning.  In _Thirty-second AAAI conference on artificial intelligence_ , 2018\.
  * Huber (1992) Peter J Huber.  Robust estimation of a location parameter.  In _Breakthroughs in statistics: Methodology and distribution_ , pp. 492–518. Springer, 1992.
  * Jang et al. (2016) Eric Jang, Shixiang Gu, and Ben Poole.  Categorical reparameterization with gumbel-softmax.  _arXiv preprint arXiv:1611.01144_ , 2016.
  * Kingma & Ba (2014) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Koenker & Hallock (2001) Roger Koenker and Kevin F Hallock.  Quantile regression.  _Journal of economic perspectives_ , 15(4):143–156, 2001.
  * Kuznetsov et al. (2020) Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov.  Controlling overestimation bias with truncated mixture of continuous distributional quantile critics.  In _International Conference on Machine Learning_ , pp. 5556–5566. PMLR, 2020.
  * Liu et al. (2022) Minghuan Liu, Menghui Zhu, and Weinan Zhang.  Goal-conditioned reinforcement learning: Problems and solutions.  _arXiv preprint arXiv:2201.08299_ , 2022.
  * Munos et al. (2016) Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc Bellemare.  Safe and efficient off-policy reinforcement learning.  _Advances in neural information processing systems_ , 29, 2016.
  * Nair et al. (2018) Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine.  Visual reinforcement learning with imagined goals.  _Advances in neural information processing systems_ , 31, 2018.
  * Plappert et al. (2018) Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al.  Multi-goal reinforcement learning: Challenging robotics environments and request for research.  _arXiv preprint arXiv:1802.09464_ , 2018.
  * Precup et al. (2001) Doina Precup, Richard S Sutton, and Sanjoy Dasgupta.  Off-policy temporal-difference learning with function approximation.  In _ICML_ , pp. 417–424, 2001.
  * Schaul et al. (2015a) Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver.  Universal value function approximators.  In _International Conference on Machine Learning_ , pp. 1312–1320. PMLR, 2015a.
  * Schaul et al. (2015b) Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver.  Prioritized experience replay.  _arXiv preprint arXiv:1511.05952_ , 2015b.
  * Schramm et al. (2023) Liam Schramm, Yunfu Deng, Edgar Granados, and Abdeslam Boularias.  Usher: Unbiased sampling for hindsight experience replay.  In _Conference on Robot Learning_ , pp. 2073–2082. PMLR, 2023.
  * Sutton & Barto (2018) Richard S Sutton and Andrew G Barto.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Tang et al. (2022) Yunhao Tang, Remi Munos, Mark Rowland, Bernardo Avila Pires, Will Dabney, and Marc Bellemare.  The nature of temporal difference errors in multi-step distributional reinforcement learning.  _Advances in Neural Information Processing Systems_ , 35:30265–30276, 2022.
  * Van Hasselt et al. (2016) Hado Van Hasselt, Arthur Guez, and David Silver.  Deep reinforcement learning with double q-learning.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 30, 2016.
  * Yang et al. (2021) Rui Yang, Jiafei Lyu, Yu Yang, Jiangpeng Yan, Feng Luo, Dijun Luo, Lanqing Li, and Xiu Li.  Bias-reduced multi-step hindsight experience replay for efficient multi-goal reinforcement learning.  _arXiv preprint arXiv:2102.12962_ , 2021.
  * Yang et al. (2022) Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang.  Rethinking goal-conditioned supervised learning and its connection to offline rl.  _arXiv preprint arXiv:2202.04478_ , 2022.