#  FastRLAP: A System for Learning High-Speed Driving via
Deep RL and Autonomous Practicing

Kyle Stachowicz†, Arjun Bhorkar†, Dhruv Shah†, Ilya Kostrikov, Sergey Levine
UC Berkeley

###### Abstract

We present a system that enables an autonomous small-scale RC car to drive
aggressively from visual observations using reinforcement learning (RL). Our
system, FastRLAP (_faster lap_), trains autonomously in the real world,
without human interventions, and without requiring any simulation or expert
demonstrations. Our system integrates a number of important components to make
this possible: we initialize the representations for the RL policy and value
function from a large prior dataset of _other_ robots navigating in other
environments (at low speed), which provides a navigation-relevant
representation. From here, a sample-efficient online RL method uses a single
low-speed user-provided demonstration to determine the desired driving course,
extracts a set of navigational checkpoints, and autonomously practices driving
through these checkpoints, resetting automatically on collision or failure.
Perhaps surprisingly, we find that with appropriate initialization and choice
of algorithm, our system can learn to drive over a variety of racing courses
with less than 20 minutes of online training. The resulting policies exhibit
emergent aggressive driving skills, such as timing braking and acceleration
around turns and avoiding areas which impede the robot’s motion, approaching
the performance of a human driver using a similar first-person interface over
the course of training. ††† Equal Contribution. Code and videos at
sites.google.com/view/fastrlap.

##  I Introduction

High-speed vision-based navigation presents a range of challenges: aside from
the usual difficulties associated with collision-free navigation, it requires
controllers that can account for both the vehicle’s dynamics and the perceived
obstacles (Fig. 3). Learning-based methods offer a particularly appealing way
to approach such challenges, as they can directly learn the relationship
between perception and vehicle dynamics and in principle capture high-
performance driving behaviors. One way that prior work has approached such
domains is via imitation learning, acquiring end-to-end skills from expert
demonstrations [1, 2]. However, if our aim is to maximize performance, we
might instead prefer to directly adapt the navigational strategy to the
vehicle _autonomously_. By learning from _autonomous_ experience in an
environment, reinforcement learning (RL) can enable this, analogously to
progress in other domains such as games and robotic manipulation, where
policies trained via RL have even exceeded human-level performance [3, 4, 5,
6].

But the autonomous setting presents major challenges for RL: unlike other
domains, it is impossible to reset the system to a random state, and so the
learning process is highly dependent on the system’s ability to make continual
progress without getting _stuck_. Starting the training process from a
randomly-initialized policy and learning solely via trial-and-error would
likely result in catastrophic failure. Instead, the RL-based system should
train automatically, without supervision, not just improving its policy
performance but also smoothly recovering from failures or collisions. The goal
of this paper is to address these challenges and understand how RL can be
applied to autonomously learn high-speed driving from vision.

Figure 2: FastRLAP learns fast driving policies for a
1/10${}^{\text{th}}$-scale vehicle operating in diverse indoor and outdoor
environments with challenging terrains in tens of minutes, using a combination
of offline pre-training and online reinforcement learning.

Directly applying RL methods in the real world with high-dimensional
observations is very difficult: online RL algorithms require large numbers of
samples, and trial-and-error learning can drive to robot into unsafe or
irrecoverable situations, necessitating frequent human interventions. As an
alternative, we could learn entirely from offline data [7, 8], but this might
yield suboptimal policies, as the desired behavior (e.g., driving aggressively
in tight spaces) might not be well-supported under the prior data. Therefore,
our approach combines online RL training with a set of pre-training and
initialization steps that aim to maximally transfer prior knowledge to
_bootstrap_ real-world RL. Specifically, we aim to use prior data to learn a
useful representation of visual observations that captures driving-related
features, such as free space and obstacles, while also adapting online to the
target domain.

To this end, we design a high-performance system for Fast Reinforcement
Learning via Autonomous Practicing. FastRLAP (* B  _faster-lap_), which uses
an automatic goal curriculum to guide a goal-conditioned RL policy to quickly
adapt to the target environment and improve its performance over multiple
laps, without requiring human interventions (see Fig. LABEL:fig:teaser). We
then leverage data-efficient online RL, bootstrapped with just a single low-
speed demonstration of the track, to rapidly learn a policy that can drive
aggressively in challenging environments (see Fig. 2). This stage takes under
20 minutes, accelerated by pre-trained representations and enabled by a
sample-efficient online RL procedure that effectively uses them.

The primary contribution of this work is FastRLAP, a system for autonomous
learning of vision-based navigation that leverages diverse prior data and
improves by practicing autonomously. We demonstrate our FastRLAP in
challenging environments on a custom 1/10${}^{\text{th}}$-scale RC car
modified for real-world online RL. FastRLAP can autonomously practice and
learn aggressive maneuvers using a novel autonomous practicing framework,
improving by up to 40% over the demonstration lap and achieving performance
close to a human expert. Notably, the online training phase typically takes
less than 20 minutes (as little as 5 minutes!), depending on the size of the
environment. During this time, the robot learns aggressive maneuvers,
drifting, and maintaining a racing line, without any expert demonstrations.
The training requires no human interventions and is fully autonomous. To the
best of our knowledge, FastRLAP is the first instantiation of a vision-based
mobile robotic system that uses model-free RL to autonomously practice high-
speed driving maneuvers and improve online in the real world.

##  II Related Work

Leveraging prior data to bootstrap online learning has been widely studied in
the context of supervised learning [9], representation learning [10],
continual learning [11, 12, 13, 14, 15], and RL [8]. Offline RL has proven
particularly powerful due to its ability to learn actionable representations
directly from existing large datasets, and some works have studied how it can
be combined with fine-tuning through online interaction [16, 17, 18, 19]. This
has enabled a variety of robotic systems that can leverage a combination of
offline data and online interaction to perform real-world tasks [20, 21, 22].
However, most such experiments focus on robotic manipulation or other
scenarios that can evaluated in controlled workspaces. Instead, FastRLAP aims
to perform fully autonomous practicing in unstructured environments, spanning
over 120 meters. To address such real-world domains, FastRLAP incorporates
pre-training data from a variety of environments and different robots, and
employs a combination of diverse but less relevant demonstrations to enable
effective bootstrapping. Additionally, FastRLAP integrates RL with a high-
level practicing pipeline to enable greater autonomy during training.

Learning high-speed navigation has been approached in various ways. Typically,
these systems rely either on highly accurate position information to define
states [23, 24, 25, 26], localize visual observations relative to a high-
fidelity _global map_ [27, 28], or operate via behavior cloning against some
privileged expert which itself can access ground-truth state and mapping [29].
This can be prohibitive in unstructured environments, where (i) onboard state
estimates can be highly inaccurate due to poor localization via noisy odometry
or GPS measurements, and (ii) generating a high-fidelity map can be difficult
or impossible. In contrast, FastRLAP learns aggressive driving behavior
directly from vision, using only a coarse sequence of checkpoints, and can
improve its behavior by self-practice without using privileged state
information.

Prior successes in learning visual navigation for ground and aerial robots
often involves either learning from large-scale simulated data [30, 31, 32,
33], passive data [34], human interventions [35, 36] or leveraging real-world
data from other robots [37]. However, simulating off-road environments can be
extremely challenging due to complex relationships between perception,
vehicular dynamics, and terrain (Fig. 3), human interventions are time-
consuming and expensive, and aggressive driving maneuvers tend to be closely
adapted to the specific environments, calling for learning directly from _on-
task_ data. The closest prior work [38] uses offline RL for off-road driving,
but does not support mechanisms to practice autonomously or adapt its behavior
on-the-fly. We present the first navigation system that combines offline pre-
training with fast, online RL training that can improve with experience and
adapt to individual real-world environments autonomously.

Figure 3: High-speed visual navigation faces challenges due to sensing,
perception, and dynamics: (a) noisy odometry and localization errors across
multiple laps, (b) overexposure and motion blur in the visual observations,
and (c) over-/under-steer due to complex dynamics.

A number of prior papers have studied the problem of autonomous real-world RL,
often through the lens of safety or reset-free training, where the need for
human interventions during training is minimized [39, 40, 41, 42, 43, 44], and
has been applied to robotic manipulation [45, 20], quadruped locomotion [46],
and mobile manipulation [47]. We draw inspiration from these ideas to build an
aggressive navigation system that uses a finite state machine to _practice_
driving around a circuit. FastRLAP scales to novel environments, driving
around 100+ meter courses, and can continually improve its performance.

##  III Problem Formulation

The objective of our high-speed visual navigation task is to drive through a
race course, defined as a sequence of position _checkpoints_ $\\{c_{i}\\}$, in
the minimum possible time. We frame this task as a Markov decision process
(MDP) ${\mathcal{M}}({\mathcal{S}},{\mathcal{A}},p,r)$, where ${\mathcal{S}}$
denotes the state space comprising of states
$s=(V,v,\omega,\alpha,g,a_{\text{prev}})$. Here, $V\in\mathbb{R}^{128\times
128\times 3\times 3}$ is a stacked sequence of the last 3 RGB images;
$v,\omega,\alpha\in\mathbb{R}^{3}$ denote the linear velocity, angular
velocity, and linear acceleration; the goal $g$ is provided as a relative
vector to the next checkpoint, written as a 2D unit vector and a distance;
$a_{\text{prev}}$ is the previous action. All measurements are specified
relative to the robot’s internal reference frame: the policy does not require
information in any fixed external frame of reference. The action space
${\mathcal{A}}$ is specified by motor velocity targets corresponding to
throttle and steering actions. Note that we do not allow the throttle command
to be negative, the policy can only drive the robot forwards. $p$ denotes the
unknown transition dynamics, and $r$ is a reward function corresponding to
reaching $c_{i}$ quickly (described in Sec. IV-B).

To make training in the real world practical and automated, we formulate this
problem in the context of autonomous RL [44], where the robot is not provided
with periodic resets or interventions when it collides with an obstacle or
gets stuck, and needs to automatically recover on its own. The robot must
_practice_ driving around the lap to fully master the task and improve its
performance over time _without_ any human interventions. We measure the
robot’s performance along three metrics: (i) time to complete the first lap,
(ii) final and median lap times over the course of training, and (iii) the
mean number of collisions per lap over the course of training.

##  IV Autonomous Practicing with RL

Our system for learning high-speed driving, FastRLAP, aims to enable
autonomous practicing and sample-efficient end-to-end RL in the real world.
FastRLAP has three components: a simple high-level finite state machine (FSM)
for autonomous practicing, a pre-trained representation of visual
observations, and a sample-efficient RL algorithm for online learning (see
Fig. LABEL:fig:teaser). The FSM (shown in blue) serves the dual purpose of
selecting the next checkpoint for the online RL policy and automatically
recovering from collisions, enabling autonomous practicing in the real world.
The online RL policy (shown in orange) is trained _online_ in the real world
to reach goals commanded by the FSM, and continually improves to learn
aggressive driving maneuvers. To provide for compute- and sample-efficient
training of the online RL policy, we bootstrap it with an offline
representation of navigation-specific visual features trained from prior data
(shown in purple).

###  IV-A Autonomous Practicing and Goal Checkpoint Selection

In the autonomous learning setting, the robot is expected to learn in the
environment without any episodic “resets” or human interventions. In early
stages of training, the RL policy may make mistakes that lead to irrecoverable
states, such as collisions. Without the help of a reset, the robot would get
stuck forever and the learning algorithm may become degenerate due to collapse
in the state distribution [42]. To overcome this, we use a simple FSM that
switches between collision recovery and commanding a relevant goal checkpoint
to the RL agent.

This FSM serves a dual purpose. When the RL policy reaches a goal checkpoint,
as measured by a low-fidelity localization estimate (e.g., from visual-
inertial odometry or GPS), the FSM commands a new goal corresponding to the
next checkpoint in the course sequence $\\{c_{i}\\}$. This forces the learner
to practice reaching all of the checkpoint goals on the race course
sequentially. If the RL policy lands itself into an irrecoverable state (e.g.,
experiences a collision or becomes stuck, see Sec. V for implementation
details), the FSM commands an automatic recovery policy to rescue the robot
and provides a “pseudo-reset”. We use a very simple scripted recovery policy
to perturb the robot’s state, which selects a random steering angle and drives
backward for a short distance. Although other approaches such as using
exploratory policies [42] or learning a recovery policy [41] have also been
studied, we found this very simple strategy to be sufficient to allow the
online RL procedure to learn directly in the real world without human
interventions.

Data: Prior navigation dataset ${\mathcal{D}}$, slow demo lap
${\mathcal{B}}_{\text{slow}}$

1 Keys: Pre-Training, Practicing, Online RL

2 while  _Encoder is not converged_ do

3 $s,a,s^{\prime},\textrm{idx}\leftarrow$ LoadData(${\mathcal{D}}$)

4 $g\leftarrow$ LoadFutureData(${\mathcal{D}}$, idx + RandomOffset())

5 $r\leftarrow$ ComputeReward($s$, $a$, $g$)

6 Train${}_{\text{IQL}}$($(s,g),a,r,(s^{\prime},g)$)

7

8while  _True_ do

9 On Robot

10 $s\leftarrow$Observe()

11 if  _$s$ near $g$_ then

12 $g\leftarrow\text{NextCheckpoint}(g)$

13

14 $r\leftarrow$ ComputeReward($s_{\textrm{prev}}$, $a_{\textrm{prev}}$, $g$)

15 SendToWorkstation($s_{\textrm{prev}}$, $a_{\textrm{prev}}$, $r$, $s$, $g$)

16 $a\sim\pi(\phi(s_{\textrm{image}}),s_{\textrm{proprio}},g)$

17 Actuate($a$)

18 if  _Collision or Stuck_ then

19 Execute recovery policy

20

21

22 On Workstation

23 ReceiveFromRobot(${\mathcal{B}}$)

24 $b$ $\leftarrow$ Sample(${\mathcal{B}}$), $b_{\text{slow}}\leftarrow$
Sample(${\mathcal{B}}_{\text{slow}}$)

25 $\pi,Q\leftarrow$Train${}_{\text{RLPD}}$($\pi,Q,b,b_{\text{slow}}$)

26

27

Algorithm 1 FastRLAP for Autonomous Practicing

###  IV-B Online RL Training

The objective of the low-level policy $\pi$ is to reach the goal checkpoints
commanded by the FSM in the minimum possible time without colliding or getting
stuck. Given the visual observations and a goal vector $\vec{g}$ from the FSM,
$\pi$ must parse the high-dimensional observations to understand its contents
and plan a high-speed trajectory through it without colliding with obstacles
or getting stuck. It is important to note that the goal checkpoints $c_{i}$
are typically beyond line-of-sight (e.g., Fig. LABEL:fig:teaser, blue), up to
40 meters away, and navigating between them requires the robot to learn a
representation of the environment layout, correlating visual observations with
possible _racing lines_ that the robot could take to maintain high speed.

We design a simple reward function $r$ that prioritizes maintaining maximum
instantaneous velocity towards the next goal checkpoint, while also avoiding
irrecoverable states that require the FSM to trigger the recovery policy,
slowing down the robot’s total lap time. We primarily define the reward as
_speed-made-good_ : the component of the instantaneous velocity in the
direction facing the next checkpoint. We additionally add a penalty for
becoming stuck (defined as failing to move despite commanding non-zero
throttle) and colliding:

$r(s,a)=\vec{v}\cdot\frac{\vec{g}}{\lVert\vec{g}\rVert}-C_{\textrm{stuck}}\mathbb{1}_{\textrm{stuck}}-C_{\textrm{collide}}\mathbb{1}_{{\lVert
a\rVert}>A}\lVert a\rVert.$ (1)

Here, $\vec{v}$ and $\vec{g}$ are the observed velocity and relative goal
coordinates (included in the state observation $s$), $a$ is lateral
acceleration, $\mathbb{1}$ is the indicator function to detect irrecoverable
states, and $C_{\text{stuck}},C_{\text{collide}},A>0$ are constants.

To maximize the above reward and continually improve the robot’s lap times,
the system must learn from interactions with the environment in practice laps,
using a _batch_ of new interactions to update its learned behavior using off-
policy RL [3, 48]. Such approaches benefit greatly by performing multiple
training steps for each environment step, known as the update-to-data (UTD)
ratio: a large UTD leads to efficient learning, but often suffers from
overfitting. To overcome this, we train our policies with RLPD [49], a data-
efficient off-policy RL algorithm that trains an ensemble of critics to avoid
catastrophic overestimation [50] and can learn quickly using a combination of
online interactions and a small amount of suboptimal, _on-task_ data.

We obtain this _on-task_ data by collecting a single _slow_ lap in the target
environment, similar to how a racing driver might perform a reconnaissance lap
at low speed to familiarize themselves with the course before attempting laps
at high speeds. While this data is very limited (under a minute in most
environments) and does not contain fast-driving behaviors, prior work has
shown that it can significantly accelerate online learning by avoiding critic
collapse in early stages of training [49]. During the online training, we
sample 50% of each training batch from this low-speed data, interleaved with
50% of data collected online. We found this to be critical to the efficiency
of our system in our evaluations (Sec. VI-B).

However, our system aims to learn effective high-speed navigation skills in as
little as 10-20 minutes. At such low training times, the process is
constrained not only by the robot’s ability to collect data, but also by the
_computational constraints_ of training the neural network, so improvements in
computationally efficiency actually translate directly into faster learning
with higher UTD ratios. Therefore we combine the strengths of data-efficient
online learning with a powerful pre-trained representation of visual
observations to enable computation-efficient training.

###  IV-C Representation Learning with Offline RL

When training image-to-action policies, end-to-end RL allows gradients from
the control objective to optimize the encoder. This results in a task-specific
encoder that produces features that are most relevant to the agent’s task,
rather than general features (e.g., features necessary for classification or
video prediction). Unfortunately, training directly on full images is very
computationally expensive and unacceptably reduces the UTD ratio. Ideally, we
would prefer to pre-train some encoder to produce task-relevant features
_without_ requiring new environment interactions, and then freeze the encoder
during online training.

We address this by training the encoder with offline RL on an existing large-
scale dataset with a similar (but not identical) objective. In particular, we
use RECON [51], a large-scale navigation dataset collected by manually driving
a Clearpath Jackal UGV outdoors at low-speeds. This dataset contains
navigation trajectories from many environments and an entirely different
robot, which importantly does _not_ include aggressive high-speed driving.
Thus, the role of pre-training is not to teach the robot how to drive quickly
and efficiently, but only to extract a navigation-relevant representation to
simplify the online learning problem. The aggressive, high-speed driving
behaviors necessary to solve the desired task must be learned through practice
in the real world, building on this pre-trained foundation.

We apply goal-conditioned offline RL by selecting a 1:1 mixture of random
goals and goals from the robot’s future trajectory in this dataset, and use
Implicit Q-Learning [52] to train a critic network (illustrated in Fig.
LABEL:fig:teaser, purple). We then take the learned encoder, which now encodes
features relevant to the navigation task, and freeze it for training the
aggressive driving policy and critic (orange) as illustrated in Sec. IV-B.

###  IV-D Summary

Algorithm 1 summarizes the FastRLAP autonomous training framework. We use a
diverse offline navigation dataset ${\mathcal{D}}$ to pre-train a navigation-
relevant representation of visual observations using a goal-conditioned RL
objective optimized with IQL (L1). We freeze the encoder trained with this
process and and use it to encode visual observations for the actor and critic
for the online learning phase (see Fig. LABEL:fig:teaser, orange). The reward
function for both the pre-training and online RL tasks is described by Eqn. 1.

During deployment in a previously unseen environment, the practicing FSM
(blue) serves the dual purpose of commanding the next goal checkpoint to the
low-level policy $\pi$ (L1), and automatic collision recovery if the robot is
stuck or in collision (L1). To enable fast training, the inference is split
between the robot and a remote workstation with low-latency communication
between them (see Sec. V for details).

The robot runs fast inference of the trained policy (L1) at 10Hz, and sends
batches of online experience to the workstation (L1) to asynchronously update
the actor and critic networks using RLPD (L1) as quickly as possible. This
process enables FastRLAP to learn aggressive driving behavior from as little
as 10-20 minutes of online experience.

##  V System Design for Online Learning

Figure 4: For fast training, we featurize the visual observations using a pre-
trained encoder onboard the robot (right) and run RL training on these
embeddings on a wirelessly connected workstation (left).

We instantiate our autonomous practicing system on a bespoke
1/10${}^{\text{th}}$-scale autonomous rally car for high-speed navigation.
While the base chassis was inspired by similar platforms [53, 54], the task of
autonomous practicing with online learning adds a number of challenges that
necessitate hardware and networking modifications. We describe these
differences on top of a commercially available Traxxas Slash 4$\times$4
Ultimate RC car (Fig. 4).

Sensing: Since our high-speed system operates directly on visual observations,
we use a forward-facing PCB camera with a fisheye lens to obtain a low-latency
stream of 128$\times$128 RGB images with minimal motion blur. Since the
learned policy requires coarse relative position estimates to intermediate
checkpoints/subgoals, our system also requires a low-fidelity state estimator.

Indoor state estimation: Relying on wheel speeds and onboard IMU for local
odometry is impractical due to wheel slippage. To estimate state in indoor
environments, we use a RealSense T265 tracking camera to provide local visual-
inertial odometry estimates for the positions of the robot and intermediate
checkpoints. We orient the T265 to face upwards to minimize localization
issues faced when colliding with an obstacle.

Outdoor state estimation: The VIO-based T265 system gives poor performance in
outdoor environments. Instead, we use a state estimate from a ublox ZED-F9P
GPS receiver mounted onboard the robot. Due to the difficulty of estimating
ground-truth heading from only local sensor information, and the poor
performance of magnetometers in close proximity to motors, we used an Extended
Kalman Filter to estimate the heading $\theta$. We fuse local wheel odometry
(forward velocity $v_{w}$) with absolute GPS velocity $\vec{v}$ and IMU
angular velocity $\omega$ using the following dynamics and measurement models:

$\theta_{t+1}=\theta_{t}+\omega\Delta t\hskip
30.0pt\vec{v}=(v_{w}\cos\theta,v_{w}\sin\theta)$

Compute: Following F1TENTH [54], we use an NVIDIA Jetson Xavier NX for onboard
compute. We process visual observations onboard using a pre-trained encoder
(Sec. IV-C), and offload the RL training to a desktop workstation over a WiFi
network or LTE (see Fig. 4). This has the dual benefits of low communication
latency (since we only communicate low-dimensional features, under 100kB/s)
and fast training (since only a subset of the layers need to be updated). To
achieve a high UTD ratio, we implement our training algorithm in JAX [55].
Using the just-in-time compiler to combine many update steps into a single
optimized XLA function, we are able to increase our update rates to $\sim$800
actor updates per second. When combined with the on-robot featurization, this
represents a $\sim$10x increase in throughput compared to training without
these optimizations.

Actuation: The Markov assumption requires that the transition dynamics $p$
only depend on the current state and action. However, typical “sensorless”
motors used in RC cars exhibit _cogging_ , a stuttering behavior that depends
heavily on unobservable variables such as rotor positions. This occurs
frequently when steering sharply or starting from a standstill, actions that
the RL agent takes often during exploration. We upgrade the system to use a
“sensored” motor to provide closed-loop startup sequencing without cogging.
While the mechanical top speed of our system is nearly 30m/s, we cap the
operational speed in indoor environments to 3.5m/s and in outdoor experiments
to 4.5m/s for safety.

Figure 5: Schematics for each test environment in the real-world. FastRLAP is
able to learn to drive real-world courses of varying size in both indoor and
outdoor settings. _Note that these schematics are not available to our system
and are shown shown only for illustration._

Action continuity: The closed loop PID and servo actuators tend to act as a
low-pass filter on their targets, smoothing out high-frequency changes. When
sampling uncorrelated actions from an output distribution, this will result in
a low signal-to-noise ratio between the commanded target actions and measured
velocity observations, making critic learning via temporal difference learning
difficult. To overcome this, we enforce _continuity_ in the policy’s outputs
by constraining them to be near the previous action by modifying the action
space: (i) instead of the standard $\tanh$ activation to limit the action
space in $[-1,1]$ (used by actor), we use a _shifted_ $\tanh$ that limits it
to the range $[a_{\text{prev}}-\delta,a_{\text{prev}}+\delta]$, i.e., near
previous action, bounded by $\delta>0$, and (ii) we append the previous action
to the observed state. Please see the supplemental material for further
details.

Detecting blocked states: Our autonomous practicing system uses a recovery
policy when the robot fails to make forward progress (e.g., collision or
stuck). We detect collisions via a simple heuristic condition: at each
timestep, we compare the lateral acceleration of the robot to a threshold
value; if that threshold is violated, the robot is in collision and a penalty
is applied according to Eqn. 1. The “stuck” condition is detected using local
odometry: if the robot has not moved at least 0.5m in the past three seconds,
it is considered to be stuck and a pseudo-reset is performed.

Figure 6: Sample trajectories of FastRLAP practicing in a real-world indoor environment. Starting from a pre-trained representation, FastRLAP often collides into obstacles (green) and learns from it’s mistakes to learn collision-free navigation (orange). By directly minimizing lap times, FastRLAP discovers a smooth racing line (purple), matching human driving performance. _Note that the 3D scan is not availably to our system and is shown only for illustration._ Indoor-A |  |  |  |  |  Outdoor-D
---|---|---|---|---|---
Indoor-B |  |  |  |  |  Outdoor-E
Indoor-C |  |  |  |  |  Sim-F
Figure 7: Examples of high-speed driving in diverse, previously unseen
environments (outer) by autonomous practicing. FastRLAP improves lap times
(inner; best-so-far shown in black) given a slow _demo lap_ (green) and
achieves near-expert lap times (red) in under 40 minutes.

##  VI Faster Lap Times with FastRLAP

In this section we present an experimental evaluation of FastRLAP in a variety
of real-world and simulated environments. We consider several metrics to
analyze the peak performance, as well as cumulative metrics during practice.
The time-to-first-lap (T2F) represents the time taken to complete the first
lap, starting from scratch. We track the best lap time achieved during
training as well as the median time of last five laps completed to capture the
converged behavior. Additionally, we list the median collisions in the last
five laps to capture safety. To contextualize our results, we provide timing
for laps driven in each environment by human drivers watching the robot from a
third-person view (“Human Expert”), as well as the duration of the “slow demo”
lap.

Note that we used the _same hyperparameters_ (i.e., network architecture,
learning rate, reward coefficients) for all experiments discussed here, both
in the real-world and simulation. See the appendix for a full list of
hyperparameter values.

###  VI-A Real-World Deployment

We deploy FastRLAP in three _previously unseen_ environments to demonstrate
autonomous practicing in tightly constrained spaces. Before the start of
training, we manually drive the robot around the course for a slow lap to
define the rough layout of the track. This lap is used in two ways: (i) to
generate a sequence of sparse checkpoints $\\{c_{i}\\}_{i=1}^{n_{c}}$ for the
practicing FSM described in Sec. IV-A, and (ii) to provide a slow-speed
demonstration for off-policy online actor-critic updates as described in Sec.
IV-B. The environments described below are shown in Fig. 7, with additional
details and visualizations in the appendix.

Indoor-A represents a large loop (70 meters in length) through the interior of
a carpeted building with glass walls and many open corridors. The course is
defined by a sequence of $n_{c}=4$ checkpoints spaced roughly 15-20 meters
apart.

Indoor-B is a significantly larger course ($\sim$120 meters in length) with
multiple obstacles, defined by $n_{c}=4$. The floor of this environment is a
tiled and has very low friction, frequently causing dynamical effects such as
over/understeer during cornering.

Indoor-C is a small but challenging indoor race course with two tight
“hairpin” turns, taken at nearly the maximum steering angle and a tight
“chicane” (a right-left sequence). Mastering this environment requires the
robot to discover fast “racing lines” that minimize unnecessary steering, and
carrying a high speed through the turns. This course is designated by
$n_{c}=3$ checkpoints. We extensively compare FastRLAP to alternative
baselines in this environment.

Outdoor-D is a medium-scale (60 meter loop) outdoor course around a building.
In addition to straightforward obstacle avoidance with the building, a tree,
and a nearby table, there are several patches of tall grass which tend to slow
down the robot’s motion. A successful policy should avoid the tall grass to
the maximum extent possible, staying near paths where the grass is shorter and
keeping to the left of the tree when it passes to avoid the grass on the
right.

Outdoor-E is a large-scale (120 meter loop) outdoor course between a dense
grove of trees on one side and a tree and several fallen logs on the other
side. The ground near the trees is covered in leaves, sticks, and other loose
material, causing highly speed-dependent steering characteristics.

Figure 8: Emergent behaviors with FastRLAP. We find that directly minimizing lap times with RL results in the emergent discovery of a “racing line”, braking as late as possible to maintain speed in and out of tight corners (a) and drifting on slick surfaces (c). In a real outdoor environment Outdoor-D, the robot learns to drive on and near paths to avoid being slowed by tall grass (b). In a challenging simulated outdoor environment Sim-F, the robot infers that the bridge is faster than driving through mud via visual correlation, learning a fast driving policy (d). _All the above graphics are rendered using telemetry from our real-world (b, c) and simulated (a, d) experiments._ Please see supplemental material for videos of these emergent behaviors. | T2F | Lap Times (s) | # Collisions
---|---|---|---
Env. | (s) | Best | Median | Demo | Expert† | Median
Indoor-A | 114 | 32.7 | 39.0 | 54 | 25 | 0
Indoor-B | 225 | 44.2 | 65.7 | 70 | 43 | 3
Indoor-C | 117 | 10.9 | 11.7 | 17 | 7 | 0
Outdoor-D | 196 | 17.1 | 22.7 | 43 | 18 | 0
Outdoor-E | 282 | 62.1 | 94.0 | 160 | 40 | 3
Sim-F | 925 | 104.1 | 107.0 | 286 | 112 | 0
Sim-G | 174 | 18.0 | 18.1 | 36 | 19 | 0
TABLE I: Summary of experiments: FastRLAP can consistently learn aggressive
driving policies in environments of varying difficulty levels, improving over
the _demo lap_ by over 40% and achieving lap times within 5% of the expert,
using only egocentric observations. _† Note that the expert has access to
privileged third-person observations._

Table I and Fig. 7 summarize the performance of our system in these
environments. FastRLAP is able to consistently improve over the low-speed
demonstration lap in a handful of laps, and nearly match human performance in
Indoor-B and Outdoor-D in 30 minutes of real-world practice, without any human
interventions. As training progresses, the achieved lap times continue to
decrease, with the path taken by the robot becoming more optimized as a
secondary effect of optimizing speed Fig. 6.

Emergent behaviors: We find that directly maximizing the reward for reaching
the next checkpoint as quickly as possible (Eqn. 1) leads to emergent
behaviors in our system, visualized in Fig. 8. The system learns the concept
of a “racing line”, finding a smooth path through the lap and maximizing its
speed through tight corners and chicanes (a). This can be seen in the speed
profile through a tight corner in Fig. 8(a), where the robot learns to carry
its speed into the _apex_ , then brakes sharply to turn and accelerates out of
the corner, to minimize the driving duration. In Fig. 8(c) with a low-friction
surface, the policy learns to over-steer slightly when turning, drifting into
the corner to achieve fast rotation without braking during the turn (c). In
outdoor environments, the learned policy is also able to distinguish ground
characteristics, preferring smooth, high-traction areas on and around concrete
paths in Outdoor-D over areas with tall grass that impedes the robot’s motion.
Please see our website for videos of our system practicing and driving
aggressively.

###  VI-B Simulated Environments

We also test our approach in two simulated environments — a rally racing
track, and a large, visually challenging off-road environment — to evaluate
FastRLAP in more diverse settings. For these experiments we use a simulated
Clearpath Jackal robot with identical observation space to our robot. Unlike
our RC car-based platform, the Jackal uses a differential drive rather than an
Ackermann steering setup; due to this difference the action space consists of
linear and angular velocity targets rather than linear velocity and steering.

Sim-F is a large, complex world derived from Clearpath’s simulation
environments [56]. In this environment the robot must navigate around a large
pool of mud, which shows non-binary traversability — greatly limiting the
robot’s dynamics and maximum speed — and should be avoided when possible. A
bridge allows the robot to bypass the mud, but is narrow and can be identified
visually. Our method successfully learns a high-performance policy in this
environment, successfully correlating the mud’s appearance with a low rewards,
and selecting the optimal path after the first few laps of trial and error
(see Fig. 8), achieving super-human lap times in under 10 laps (see Tab. I).
This environment is particularly challenging since the mud can cause the robot
to get irrecoverably stuck in a slow-speed zone; all alternative baselines and
ablations failed to solve this task.

Sim-G is a smaller-scale dirt track with sharp turns and chicanes, much like
Indoor-C, making it a particularly interesting environment to study the
emergence of _racing lines_ and agile maneuvers. All baselines were able to
solve this task to some extent, allowing direct comparison between several
alternative approaches to the problem.

###  VI-C Comparative Analysis

We compare the performance of FastRLAP against the baselines and ablations
mentioned in two environments: Indoor-C and Sim-G. we compare our approach to
several baselines and ablations, and demonstrate the importance of each of the
components of our method: pre-trained visual representations, online RL
starting from a slow demo lap, and autonomous recovery behaviors to handle the
reset-free environment. Specifically, we consider the following variations and
ablations:

Offline RL: Ablating the online learning aspect, this baseline uses a policy
trained purely offline, with access to 15 minutes of _expert_ data. Note that
this is _more_ on-task offline data than is available to FastRLAP.

No Demo Lap: Ablating the online learning aspect, this baseline deploys
FastRLAP _without_ sampling any data from the low-speed demo lap
${\mathcal{B}}_{\text{slow}}$ (Alg. 1, L1).

No Pre-Training: Ablating offline pre-training, this baseline uses networks
initialized from scratch and trained online using DrQ [57], a state-of-the-art
pixel-based RL algorithm.

ImageNet Pre-Training: Ablating task-specific pre-training, this baseline uses
the same encoder structure, but trained instead for image classification on
ImageNet [58, 59] for extracting visual features.

No Pseudo-Resets: Ablating the FSM, this baseline deploys FastRLAP _without_
the benefit of scripted pseudo-resets when it is stuck, requiring the robot to
learn recovery behavior.

State-Based: This variation replaces visual observations with _privileged_
state estimates in the form of an approximate 2D pose (measured by onboard
visual-inertial odometry).

|  | Lap Times (s) | # Collisions
---|---|---|---
| T2F (s) | Best | Median | Median
Real-World (Indoor-C)
State-Based | 274 | 12.7 | 18.8 | 3
Offline RL [38] | $\infty$ | – | – | –
No Pre-Training | 233 | 12.7 | 20.0 | 1
ImageNet Pre-Training | 49.8 | 19.7 | 29.7 | 1
No Demo Lap | 239 | 16.0 | 62.6 | 12
FastRLAP (Ours) | 117 | 10.9 | 13.3 | 0
Human FPV | – | 11.1 | 14.4 | 2
Human Oracle† | – | 7.3 | 8.8 | 0
Simulation (Sim-G)
State-Based | 222 | 18.9 | 26.2 | 0
No Demo Lap | 665 | 19.6 | 22.2 | 0
No Pre-Training | 375 | 17.8 | 18.4 | 1
No Pseudo-Resets | 405 | 21.7 | 25.1 | 0
FastRLAP (Ours) | 173 | 18.0 | 18.1 | 0
Human Oracle† | – | 18.6 | 18.9 | 0
TABLE II: Comparing to baselines in real-world and simulated environments,
FastRLAP achieves consistently lower time-to-first lap (T2F), best and median
lap times, and minimum collisions. Notably, FastRLAP can outperform a system
with access to privileged state estimates, and a human driving from first-
person view (FPV). In both environments, FastRLAP achieves close-to-oracle†
driving performance, i.e., an expert human driving with _privileged_ third-
person observations of the robot, shown in gray. Offline RL failed to complete
a single lap in Indoor-C, likely due to its inability to adapt the learned
policies on-the-fly in novel environments. Figure 9: Progression of running
minimum lap times across different baselines in Tab. II in Sim-G. FastRLAP
achieves expert-level performance in 20 minutes, learning efficiently from
pixels as quickly as an agent with privileged state information, while
achieving better overall performance by the end of training.

In Indoor-C, our experiments (Tab. II) show that FastRLAP far outperforms the
ablation without a demo lap in both time-to-first lap (T2F) and best lap time,
while also encountering fewer collisions. Early in the training process, the
demonstration helps the system make progress around the course, enabling broad
state coverage, which yields more useful exploration early on and eventually
leads to better performance. The offline RL baseline completely failed to
produce a usable policy, as it is unable to adapt online to new rich
observations.

Several additional baselines were considered in the simulated Sim-G
environment, summarized in Tab. II. The same trends hold, showing that the
demo lap is very important to fast learning and achieving a low T2F.
Similarly, removing pseudo-resets causes the system to get stuck for extended
durations, resulting in a very slow first lap (even with a demonstration!).
Fig. 9 shows the progression of the running minimum lap times for each method.

Analyzing the role of pre-training with offline RL, we find that FastRLAP
initialized with a generic ImageNet encoder completes its first lap relatively
quickly, achieving a T2F comparable to FastRLAP. However, its asymptotic
performance is comparably poor: its best lap time across training in Indoor-C
is only slightly better than the slow-lap demonstration. This suggests that
while extracting general-purpose visual features (e.g., edges and gradients)
may be sufficient for low-speed navigation, high-speed navigation requires
consideration of task-specific features, such as depth or obstacle detection,
that are better learned with task-specific pre-training.

Most surprisingly, learning directly from visual observations outperformed the
variation with access to privileged state information in both simulated and
real environments. This suggests that the features learned by the pre-trained
encoder are _more informative_ than simple localization estimates, because
they generalize better: an obstacle has similar representation across
different positions and environments, while the state-based agent must learn a
free-space representation of the entire environment inside its critic function
via trial and error, bumping into each object before it can record its
existence.

###  VI-D Qualitative Analysis

To better understand if the learned policy is genuinely learning from the
visual cues in the environment or simply memorizing a sequence of actions, we
consider the value of $Q(s,a)$ for a set of sample images by artificially
injecting a range of steering inputs (actions). This enables us to see how the
critic network assesses the desirability of different actions based on the
visual information in the environment.

We find that the visual observations qualitatively explain the critic outputs,
with obstacles corresponding to low assigned values to actions steering
towards them and relatively higher values assigned to keeping to less-
restrictive areas like paths rather than tall grass. This suggests that there
appears to be a correlation between the visual cues in the environment and the
actions chosen by the learned policy, suggesting that it is not merely
memorizing a sequence of actions but rather learning to make decisions based
on the environment’s visual features.

(a) The critic assigns very low value to actions that would cause contact with
an obstacle.

(b) The critic suggests turning towards the next checkpoint before the current
checkpoint is reached.

(c) In wide-open areas the critic reflects a beeline policy directly towards
the next goal.

(d) The critic prefers turning left rather than the geometric shortest-path on
the right to avoid tall grass.

Figure 10: Qualitative critic evaluations on sample images from environment
Outdoor-D against diverse steering actions, represented as (approximate)
geometric paths overlaid on the image. Hot (cold) colors represent high
(low)-value actions. The green arrow points in the direction of the next
checkpoint.

##  VII Discussion

We presented a system for learning high-speed driving with reinforcement
learning from rich observations, practicing autonomously in the real world.
Our approach uses representations from prior data to initialize the policy,
followed by sample-efficient online RL paired with a checkpoint-based
navigation strategy to recover autonomously from collisions and continue
practicing. Although deep RL is often believed to be inefficient and difficult
to use in the real world, we demonstrate that with appropriate pre-training
and several important design decisions, our system can actually learn
effective driving strategies in under 20 minutes of real-world training. This
result may seem quite surprising when viewed in contrast to prior work that
uses simulated data [30], or hundreds of hours of training [60], and we
believe it provides strong validation that deep RL can indeed be a viable tool
for learning real-world policies even from raw images, when combined with
appropriate pre-training and implemented in the context of an autonomous
training framework.

A qualitative investigation of the policies learned by our system also reveals
interesting emergent behavior. Although we bootstrap training with prior data
(in other domains and from other robots) and a single slow demonstration lap,
the learned policies exhibit skills like drifting, selecting for high-speed
terrain, and maintaining a racing line which deviate significantly from the
behaviors seen in the prior data. Thus, the online RL process not only serves
to robustify previously seen behavior, as observed in prior work incorporating
offline data into real-world RL [21], but also acquires new emergent behaviors
building on the foundation established by the prior data. At the same time,
our ablations establish the importance of task-relevant pre-training,
supporting the notion that representations learned from diverse robot
navigation data serve as an effective foundation for downstream skill
learning, resembling the importance of self-supervised pre-training objectives
in vision and NLP [61, 62].

While our system enables highly effective image-based driving, it does have a
number of limitations. First, the current implementation still relies on a
state estimator to provide a vector to the next checkpoint. This limitation
could be addressed in future work by using image goals rather than position
goals, as has been demonstrated in prior work on long-horizon navigation [63].
Second, although our system provides for autonomous recovery from collisions,
it does not actually account for _safety_ explicitly: that is, the system aims
to avoid collisions because it leads to task failure, but does not perform any
special safety-aware handling of unfamiliar inputs to, for example, slow down
or proceed cautiously in unfamiliar situations. This kind of safety handling
would likely be essential in more complex open-world settings or with larger
vehicles that can seriously damage themselves in a collision. We hope that
addressing these limitations will enable RL-based systems to learn complex and
highly performant navigation skills in a wide range of domains, and we believe
that our work can provide a stepping stone toward this.

## Acknowledgments

This research was partially supported by DARPA RACER, ARL DCIST CRA
W911NF-17-2-0181, the National Science Foundation through IIS-2150826, and the
Office of Naval Research. The authors would like to thank Alejandro
Escontrela, Noriaki Hirose, and Philippe Hansen-Estruch, for their help with
running experiments and providing baseline implementations.

## References

  * [1] M. Bojarski _et al._ , “End to end learning for self-driving cars,” 2016.
  * [2] M. Bansal, A. Krizhevsky, and A. Ogale, “ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst,” in _Robotics: Science and Systems_ , 2019.
  * [3] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” _arxiv_ , 2013.
  * [4] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel _et al._ , “Mastering atari, go, chess and shogi by planning with a learned model,” _Nature_ , 2020\.
  * [5] S. Gu, E. Holly, T. Lillicrap, and S. Levine, “Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2017.
  * [6] O. Kroemer, S. Niekum, and G. Konidaris, “A review of robot learning for manipulation: Challenges, representations, and algorithms,” _JMLR_ , 2021\.
  * [7] S. Lange, T. Gabel, and M. Riedmiller, “Batch reinforcement learning,” _Reinforcement learning: State-of-the-Art_ , 2012.
  * [8] S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems,” 2020.
  * [9] S. Ross, G. Gordon, and D. Bagnell, “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning,” in _International Conference on Artificial Intelligence and Statistics (AISTATS)_ , 2011.
  * [10] D. Yarats, A. Zhang, I. Kostrikov, B. Amos, J. Pineau, and R. Fergus, “Improving sample efficiency in model-free reinforcement learning from images,” in _AAAI Conference on Artificial Intelligence_ , 2021.
  * [11] S. Thrun and T. M. Mitchell, “Lifelong robot learning,” _Robotics and Autonomous Systems_ , 1995.
  * [12] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in _26th Annual International Conference on Machine Learning_ , 2009.
  * [13] C. Florensa, D. Held, X. Geng, and P. Abbeel, “Automatic goal generation for reinforcement learning agents,” in _International Conference on Machine Learning_ , 2018.
  * [14] T. Matiisen, A. Oliver, T. Cohen, and J. Schulman, “Teacher–student curriculum learning,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 31, no. 9, pp. 3732–3740, 2020.
  * [15] S. Sukhbaatar, Z. Lin, I. Kostrikov, G. Synnaeve, A. Szlam, and R. Fergus, “Intrinsic motivation and automatic curricula via asymmetric self-play,” in _Intl. Conf. on Learning Representations (ICLR)_ , 2018.
  * [16] A. Nair, A. Gupta, M. Dalal, and S. Levine, “Awac: Accelerating online reinforcement learning with offline datasets,” _arXiv_ , 2020.
  * [17] A. Villaflor, J. Dolan, and J. Schneider, “Fine-tuning offline reinforcement learning with model-based policy optimization,” 2021.
  * [18] T. Xie, N. Jiang, H. Wang, C. Xiong, and Y. Bai, “Policy finetuning: Bridging sample-efficient offline and online reinforcement learning,” in _Neural Information Processing Systems_ , 2021.
  * [19] S. Lee, Y. Seo, K. Lee, P. Abbeel, and J. Shin, “Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble,” in _Conf. on Robot Learning_ , 2021.
  * [20] H. R. Walke, J. H. Yang, A. Yu, A. Kumar, J. Orbik, A. Singh, and S. Levine, “Don’t start from scratch: Leveraging prior data to automate robotic reinforcement learning,” in _OpenReview_ , 2022.
  * [21] A. Kumar, A. Singh, F. Ebert, Y. Yang, C. Finn, and S. Levine, “Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trials,” _arXiv_ , 2022.
  * [22] N. Gürtler, S. Blaes, P. Kolev, F. Widmaier, M. Wuthrich, S. Bauer, B. Schölkopf, and G. Martius, “Benchmarking offline reinforcement learning on real-robot hardware,” in _Intl. Conf. on Learning Representations (ICLR)_ , 2023.
  * [23] J. Funke, P. Theodosis, R. Hindiyeh, G. Stanek, K. Kritatakirana, C. Gerdes, D. Langer, M. Hernandez, B. Müller-Bessler, and B. Huhnke, “Up to the limits: Autonomous audi tts,” in _IEEE Intelligent Vehicles Symposium_ , 2012\.
  * [24] N. Keivan and G. Sibley, “Realtime simulation-in-the-loop control for agile ground vehicles,” in _Towards Autonomous Robotic Systems_ , 2013.
  * [25] G. Williams, P. Drews, B. Goldfain, J. M. Rehg, and E. A. Theodorou, “Aggressive driving with model predictive path integral control,” _IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 1433–1440, 2016.
  * [26] U. Rosolia and F. Borrelli, “Learning How to Autonomously Race a Car: A Predictive Control Approach,” _IEEE Trans. on Control Systems Technology_ , 2020.
  * [27] P. Drews, G. Williams, B. Goldfain, E. A. Theodorou, and J. M. Rehg, “Aggressive deep driving: Combining convolutional neural networks and model predictive control,” in _Conf. on Robot Learning_ , 2017.
  * [28] ——, “Vision-based high-speed driving with a deep dynamic observer,” _IEEE Robotics and Automation Letters_ , 2019.
  * [29] Y. Pan, C.-A. Cheng, K. Saigol, K. Lee, X. Yan, E. A. Theodorou, and B. Boots, “Imitation learning for agile autonomous driving,” _The International Journal of Robotics Research_ , 2020.
  * [30] A. Loquercio, E. Kaufmann, R. Ranftl, A. Dosovitskiy, V. Koltun, and D. Scaramuzza, “Deep drone racing: From simulation to reality with domain randomization,” _IEEE Transactions on Robotics_ , 2020.
  * [31] A. Loquercio, E. Kaufmann, R. Ranftl, M. Müller, V. Koltun, and D. Scaramuzza, “Learning high-speed flight in the wild,” _Science Robotics_ , 2021.
  * [32] F. Fuchs, Y. Song, E. Kaufmann, D. Scaramuzza, and P. Dürr, “Super-human performance in gran turismo sport using deep reinforcement learning,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 3, pp. 4257–4264, 2021\.
  * [33] T. Gervet, S. Chintala, D. Batra, J. Malik, and D. S. Chaplot, “Navigating to objects in the real world,” _ArXiv_ , vol. abs/2212.00922, 2022.
  * [34] M. Chang, A. Gupta, and S. Gupta, “Semantic visual navigation by watching youtube videos,” in _Advances in Neural Information Processing Systems_ , 2020.
  * [35] A. Kendall, J. Hawke, D. Janz, P. Mazur, D. Reda, J. Allen, V. Lam, A. Bewley, and A. Shah, “Learning to drive in a day,” _CoRR_ , 2018.
  * [36] G. Kahn, P. Abbeel, and S. Levine, “Land: Learning to navigate from disengagements,” _IEEE Robotics and Automation Letters_ , 2021.
  * [37] D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine, “GNM: A General Navigation Model to Drive Any Robot,” in _arXiV_ , 2022.
  * [38] D. Shah, A. Bhorkar, H. Leen, I. Kostrikov, N. Rhinehart, and S. Levine, “Offline reinforcement learning for visual navigation,” in _Conf. on Robot Learning_ , 2022.
  * [39] W. Han, S. Levine, and P. Abbeel, “Learning compound multi-step controllers under unknown dynamics,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2015.
  * [40] C. Richter and N. Roy, “Safe visual navigation via deep learning and novelty detection,” in _Robotics: Science and Systems_ , 2017.
  * [41] B. Eysenbach, S. Gu, J. Ibarz, and S. Levine, “Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning,” in _Intl. Conf. on Learning Representations (ICLR)_ , 2018.
  * [42] H. Zhu, J. Yu, A. Gupta, D. Shah, K. Hartikainen, A. Singh, V. Kumar, and S. Levine, “The Ingredients of Real World Robotic Reinforcement Learning,” in _Intl. Conf. on Learning Representations (ICLR)_ , 2020.
  * [43] K. Lu, A. Grover, P. Abbeel, and I. Mordatch, “Reset-free lifelong learning with skill-space planning,” in _Intl. Conf. on Learning Representations (ICLR)_ , 2021.
  * [44] A. Sharma, K. Xu, N. Sardana, A. Gupta, K. Hausman, S. Levine, and C. Finn, “Autonomous reinforcement learning: Formalism and benchmarking,” in _Intl. Conf. on Learning Representations (ICLR)_ , 2022.
  * [45] A. Gupta, J. Yu, T. Z. Zhao, V. Kumar, A. Rovinsky, K. Xu, T. Devlin, and S. Levine, “Reset-free reinforcement learning via multi-task learning: Learning dexterous manipulation behaviors without human intervention,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2021.
  * [46] S. Ha, P. Xu, Z. Tan, S. Levine, and J. Tan, “Learning to walk in the real world with minimal human effort,” in _Conference on Robot Learning_ , 2020\.
  * [47] C. Sun, J. Orbik, C. M. Devin, B. H. Yang, A. Gupta, G. Berseth, and S. Levine, “Fully autonomous real-world reinforcement learning with applications to mobile manipulation,” in _Conf. on Robot Learning_ , 2022.
  * [48] S. Fujimoto, D. Meger, and D. Precup, “Off-policy deep reinforcement learning without exploration,” in _International Conference on Machine Learning_ , 2019.
  * [49] Anonymous, “Efficient online reinforcement learning with offline data.” OpenReview, 2023.
  * [50] X. Chen, C. Wang, Z. Zhou, and K. Ross, “Randomized Ensembled Double Q-Learning: Learning Fast Without a Model,” Mar. 2021, arXiv:2101.05982 [cs].
  * [51] D. Shah, B. Eysenbach, N. Rhinehart, and S. Levine, “Rapid exploration for open-world navigation with latent goal models,” 2021.
  * [52] I. Kostrikov, A. Nair, and S. Levine, “Offline reinforcement learning with implicit q-learning,” _arxiv_ , 2021.
  * [53] “MIT RACECAR,” 2014. [Online]. Available: https://racecar.mit.edu
  * [54] M. O’Kelly, H. Zheng, D. Karthik, and R. Mangharam, “F1TENTH: An Open-source Evaluation Environment for Continuous Control and Reinforcement Learning,” in _NeurIPS 2019 Competition and Demonstration Track_ , 2020.
  * [55] J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang, “JAX: composable transformations of Python+NumPy programs,” 2018. [Online]. Available: http://github.com/google/jax
  * [56] C. Robotics, “Clearpath additional simulation worlds,” 2022. [Online]. Available: https://github.com/clearpathrobotics/cpr_gazebo
  * [57] I. Kostrikov, D. Yarats, and R. Fergus, “Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels,” Mar. 2021.
  * [58] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2009.
  * [59] S. Parisi, A. Rajeswaran, S. Purushwalkam, and A. Gupta, “The unsurprising effectiveness of pre-trained vision models for control,” in _International Conference on Machine Learning_ , 2022.
  * [60] E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra, “DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames,” in _Intl. Conf. on Learning Representations (ICLR)_ , 2020\.
  * [61] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv_ , 2018.
  * [62] X. Chen, H. Fan, R. Girshick, and K. He, “Improved baselines with momentum contrastive learning,” _arxiv_ , 2020.
  * [63] D. Shah and S. Levine, “ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hints,” in _Robotics: Science and Systems XVIII_ , 2022.

###  -A Hyperparameters

All our experiments use the _same_ set of hyperparameters and there is no
environment-specific tuning used in the results presented in the paper. See
Tab. III for a list of hyperparameters and Fig. 11 for a detailed network
architecture.

Category | Hyperparameter | Value
---|---|---
Actor/Critic | Actor learning rate | 3e-4
| Critic learning rate | 3e-4
| Temperature learning rate | 3e-4
| Actor network architecture | 2x256
| Critic network architecture | 2x256
| Initial target entropy | -3
| Entropy decay rate | 1e-5
| Critic ensemble size | 10
MDP/System | Discount factor | 0.99
| Time step | 0.1s
| Velocity target range (m/s) | $[0.5,3.5]$
| Servo target range (rad) | $[-0.5,0.5]$
|  $C_{\textrm{collide}}$ (real only) | $0.2s^{2}/m$
| $C_{\textrm{stuck}}$ | -10
|  Squashing range $\delta$ | 0.2
Encoder | Layer count | 4
| Convolution size | 3x3
| Stride | 2
| Hidden channels | 32
IQL | Expectile | 0.7
| Value network structure | same as critic
TABLE III: List of hyperparameters used throughout experiments Figure 11:
Network architecture for the critic. Actor (and, for IQL, value) architectures
are identical, but with only one backbone rather than an ensemble.
Prorpioceptive information is concatenated with the action and the goal and
fed through a dense layer, concatenated with the output of a convolutional
encoder applied to a sequence of three camera images, and fed through a
2-layer MLP.

|  | Lap time (s) | # collisions
---|---|---|---
Course name | Method | first | best | median | median (5) | best | total | mean | median | mean (5) | median (5)
Real-A | FastRLAP | 114.31 | 32.74 | 49.51 | 38.99 | 0 | 82 | 2.48 | 2 | 2.00 | 0
Real-B | FastRLAP | 224.70 | 44.21 | 73.71 | 65.72 | 0 | 135 | 7.50 | 7 | 4.20 | 3
Real-C | States | 274.42 | 12.70 | 50.13 | 18.88 | 0 | 190 | 12.67 | 4 | 3.40 | 2
Real-C | Ours | 117.38 | 10.90 | 13.27 | 11.69 | 0 | 126 | 2.74 | 0 | 0.00 | 0
Real-C | No slow lap | 239.21 | 16.01 | 64.51 | 62.62 | 0 | 206 | 22.89 | 12 | 8.80 | 11
Real-C | Human Oracle | 54.40 | 7.21 | 10.08 | 8.79 | 0 | 7 | 1.00 | 0 | 0.00 | 0
Real-C | ImageNet encoder | 49.75 | 19.66 | 30.20 | 21.12 | 0 | 168 | 6.46 | 1 | 0.00 | 0
Real-C | No pretraining | 232.79 | 12.70 | 21.60 | 19.99 | 0 | 174 | 9.67 | 1 | 1.80 | 1
Sim-D | FastRLAP | 925.08 | 104.19 | 107.00 | 107.00 | 0 | 157 | 4.76 | 0 | 0.00 | 0
Sim-E | Blind | 222.17 | 18.89 | 21.70 | 19.50 | 0 | 127 | 1.18 | 0 | 1.20 | 0
Sim-E | States | 222.17 | 19.10 | 23.69 | 26.20 | 0 | 113 | 1.30 | 0 | 1.00 | 1
Sim-E | FastRLAP | 174.30 | 17.99 | 19.10 | 18.10 | 0 | 48 | 0.42 | 0 | 0.00 | 0
Sim-E | No slow lap | 665.04 | 19.70 | 23.33 | 22.20 | 0 | 90 | 0.95 | 0 | 0.00 | 0
Sim-E | No pretraining | 375.26 | 17.80 | 21.90 | 18.39 | 0 | 100 | 1.14 | 0 | 0.00 | 0
Sim-E | No pseudo-resets | 405.02 | 21.70 | 26.31 | 25.10 | 0 | 156 | 1.64 | 0 | 0.20 | 0

TABLE IV: Detailed statistics of all runs

Figure 12: Detailed laptime progression charts for all baselines

###  -B Additional Experimental Details

In addition to the metrics presented in the main paper, please see Tab. IV for
a list of additional metrics per experiment, and Fig. 12 for lap time
progression charts for each experiment. FastRLAP outperforms baselines in
_all_ environments, as measured by any suitable metric — time-to-first lap,
best lap time, mean/median lap times, and minimum number of average and best-
case collisions.

###  -C Implementation Details

The overall system was implemented using ROS 1 Noetic Ninjemys, with the
inference and training code using JAX. We transfered tensors between the
components of our system (new data from the robot to the workstation, and
parameteres from the workstation to the robot) using ROS messages.

###  -D System Details

As mentioned in the paper, we squash the output of the actor dynamically to a
range around the previous action $a_{\textrm{prev}}$ to enforce continuity in
output actions by constraining them to be no more than some positive constant
$\delta$ away from the previous action in each dimension. We choose our
activation function so that the interval
$[a_{\textrm{prev}}-\delta,a_{\textrm{prev}}+\delta]$ is roughly mapped to
itself. More precisely, we ensure that our activation
$f_{a_{\textrm{prev}},\delta}(x)$ is accurate to first-order Taylor expansion
around the previous action. This yields the following activation function
applied to the output of the actor:

$f_{a_{\textrm{prev}},\delta}(x)=\tanh\left(\frac{(x-a_{\textrm{prev}})}{\delta}\right)\delta+a_{\textrm{prev}}$

Figure 13 shows a graphical depiction of this activation function.

Figure 13: Our activation function (actor output only) ensures action
continuity.

###  -E Simulation Environment

The Clearpath Jackal used for simulations differs from the real environments
primarily in its action space, which (as a differential drive robot) allows
turning in place. We limit the linear velocity actions of the robot to
$[-1,2]$ and the angular velocity actions to $[-1.0,1.0]$. Simulated position
measurement is provided in lieu of the RealSense tracker for determining
relative goal locations.

###  -F Code Release

Please see sites.google.com/view/fastrlap for the training and robot-side
inference code as well as modified simulation environments.