# Autonomy 2.0:
Why is self-driving always 5 years away?

Ashesh Jain, Luca Del Pero, Hugo Grimmett, Peter Ondruska
Lyft Level 5 self-driving
{asheshjain, ldelpero, hgrimmett<EMAIL_ADDRESS>

###### Abstract

Despite the numerous successes of machine learning over the past decade (image
recognition, decision-making, NLP, image synthesis), self-driving technology
has not yet followed the same trend. In this paper, we study the history,
composition, and development bottlenecks of the modern self-driving stack. We
argue that the slow progress is caused by approaches that require too much
hand-engineering, an over-reliance on road testing, and high fleet deployment
costs. We observe that the classical stack has several bottlenecks that
preclude the necessary scale needed to capture the long tail of rare events.
To resolve these problems, we outline the principles of _Autonomy 2.0_ , an
ML-first approach to self-driving, as a viable alternative to the currently
adopted state-of-the-art. This approach is based on (i) a fully differentiable
AV stack trainable from human demonstrations, (ii) closed-loop data-driven
reactive simulation, and (iii) large-scale, low-cost data collections as
critical solutions towards scalability issues. We outline the general
architecture, survey promising works in this direction and propose key
challenges to be addressed by the community in the future.

> Keywords: Self-driving, Data, Planning, Simulation

Figure 1: The development process of a typical state-of-the-art autonomy stack
(Autonomy 1.0) vs proposed ML-first stack (Autonomy 2.0). We argue that the
current self-driving industry progress is slow due to an inefficient human-in
the loop development. These issues can be solved by training a differentiable
self-driving stack in a closed-loop simulation constructed out of a large
collection of human driving demonstrations.

##  1 Introduction

Self-Driving Vehicles (SDVs) have been an active research area for decades and
make regular headlines since the DARPA Grand Challenges in 2005-2007. Many
companies are attempting to develop the first level 4+ SDVs [1], some for more
than a decade. We have seen small-scale SDV testing, but despite the numerous
unrealised predictions that ubiquitous SDVs are ‘only 5 years away’ [2, 3, 4],
production-level deployment still seems a distant future. Given the limited
rate of progress, some questions naturally arise: (a) why did the research
community underestimate the problem’s difficulty? (b) are there fundamental
limitations in today’s approaches to SDV development?

Figure 2: Two successes of ML: ImageNet [left] and AlphaGo Zero [right]
achieving super-human performance in perception and decision making. Self-
driving has not followed this trend yet. We argue this is because Autonomy 1.0
self-driving technology adopted across the industry is not fully data enabled
causing scalability bottlenecks.

After the DARPA challenges, most of the industry decomposed the SDV technology
stack into HD mapping, localisation, perception, prediction, and planning [5].
Following breakthroughs enabled by ImageNet [6], the perception and prediction
parts started to become primarily machine-learned. However, behaviour planning
and simulation are still largely _rule-based_ : performance improves using
humans writing increasingly detailed rules that dictate how the SDV should
drive. There has been the belief that, given very accurate perception, rule-
based approaches to planning may suffice for human-level performance. We refer
to this approach as _Autonomy 1.0_.

Production-level performance requires significant _scale_ to discover and
appropriately handle the ’long tail’ of rare events. We argue that Autonomy
1.0 will not achieve this due to three scalability bottlenecks: (i) rule-based
planners and simulators do not effectively model the complexity and diversity
of driving behaviours and need to be re-tuned for different geographical
regions, they fundamentally have not benefited from the breakthroughs in deep-
learning (Fig. 2) (ii) due to the limited efficacy of rule-based simulators,
evaluation is done mainly via road-testing, which lengthens the development
cycle, and (iii) road testing operations are expensive and scale poorly with
SDV performance.

Our proposed solution to these scale bottlenecks is to turn the entire SDV
stack into an ML system that can be trained and validated offline using a
large dataset of diverse, real-world data of human driving. We call this
_Autonomy 2.0_. Autonomy 2.0 is a data-first paradigm: ML turns all parts of
the stack (including planning and simulation) into data problems, and
performance improves with better data sets rather than by designing new
driving rules (Fig. 1). This unlocks the scalability required for mastering
the long tail of rare events and scaling to new geographies. All that is
needed is to collect large enough datasets and retrain the system.

The key challenges to Autonomy 2.0 are (i) formulating the stack as an end-to-
end differentiable network, (ii) validating it offline in a closed-loop with a
machine-learned simulator, and (iii) collecting the large amounts of human
driving data required to train them.

In the next section, we explain and analyse the bottlenecks in Autonomy 1.0 as
typically adopted in the industry, then explain the benefits of Autonomy 2.0,
and end by highlighting the open questions that must be answered for SDVs to
_truly_ be 5 years away.

##  2 Autonomy 1.0

Figure 3: Autonomy 1.0 – the typical technology stack, indicating the amount
of ML used in individual components. While perception and prediction are ML-
based, planning and simulation still rely on non-scalable rule-based systems
(sec. 3)

In this section we survey the state-of-the-art Autonomy 1.0 technology used in
a typical self-driving program. The typical stack (Fig. 3) is composed of
these components: perception, prediction and planning, which subsequently
answer the questions of what is around the car? what is likely to happen next?
and what should the car do?. An essential part of the development cycle of the
stack is testing, which answers ”what is the performance of the system?”

Perception. The typical solution consists of neural networks processing raw
camera [7], LIDAR [8, 9] and radar data [10]. Observations from different
sensors are implicitly or explicitly fused [11, 12], producing the 3D
positions and other attributes (e.g. size, orientation) of the _traffic
participants_ around the SDV (e.g. vehicles, pedestrians and cyclists). These
systems are trained on hand-labelled sensor data [13]. They also typically
rely on HD maps [14, 13] containing information about the static environment
and its traffic rules.

Prediction. Today, most rule-based methods that extrapolate the observed
motion of traffic participants [15] are being replaced with ML motion
forecasting, where neural network are trained to predict a future trajectory
from a few seconds of observations [16, 17, 18, 19]. These predictions are
used to estimate future space occupancy and the likelihood of collisions.

Planning. Despite a lot of progress in the field of reinforcement learning
[20, 21], Autonomy 1.0 still heavily relies on _rule-based_ trajectory
optimisation techniques. These techniques combine global search and local
optimisation to find a trajectory that minimises an expert-designed cost
function [22, 23, 24, 25], which is then fed to a controller converting it
into speed and steering signals. The cost function consists of various comfort
and safety terms hand-engineered by domain experts to achieve a desirable
behaviour, such as average acceleration, distance to other vehicles, adherence
to traffic rules, etc.

Testing estimates “what is the performance of the system?” Typically, the
system performance is road-tested by deploying it under the supervision of a
safety driver, who is ready to disengage the autonomy system at any time, i.e.
take control in the case of incorrect behaviour. Such disengagements are an
important part of the development cycle (Fig. 1). They provide a valuable
source of information on which part of the self-driving stack to improve. For
example, errors caused by planning lead to modifying the expert cost function
by adding new weights.

##  3 Scalability bottlenecks of Autonomy 1.0

The Autonomy 1.0 stack described in the previous section can perform well
under regular conditions [26, 25]. Attaining L4-L5 production-level
performance, however, requires scaling the paradigm to cover the ’long tail’
of rare events such as road closures, road accidents, other agents breaking
the rules-of-the-road etc. At the same time, the solution needs to scale to
multiple cities with diverse agent behaviours. In this section we explore the
scalability bottlenecks that make it challenging for Autonomy 1.0 to handle
this long tail of events.

###  3.1 Trying to capture complex and diverse behaviours with rule-based
systems

Rule-based systems seek to optimise a cost function designed by domain experts
who write new terms and balances their weights. New scenarios require new
terms and weights, and so development requires large engineering teams and is
a major bottleneck. Operating SDVs in a new location can mean person-years of
re-tuning. Designing interfaces between the Autonomy 1.0 components also come
with a high engineering and performance cost. Even though they provide human-
interpretable abstraction between different tasks, they limit the ability to
express the nuance of the encountered driving situations.

###  3.2 Reliance on road-testing and low-realism offline simulation

On-road testing provides the highest possible realism, and safety driver
disengagements can give actionable feedback to improve the system. However,
the time between cost-engineering and road testing makes for a long
development cycle: days or weeks. Moreover, it is non-reproducible: one cannot
repeat scenarios exactly, preventing direct comparison between versions.

One strategy for offline evaluation is to replay sensor logs to evaluate the
performance of a new system version. However, if the SDV actions during replay
are different to those in the log, other traffic participants do not react
accordingly. E.g. an SDV, driving slower than in the original log can cause a
false positive rear-end collision. This hinders fair comparison between
software versions.

A degree of reactivity can be achieved by adding rule-based behaviours to the
other traffic participants or employing a synthetic simulator [27]. However,
rule-based simulation has the same limitations as rule-based planning: hand
encoding diverse and realistic behaviours for pedestrians, cyclists, and other
vehicles scales poorly.

###  3.3 Limited fleet deployment scale

As requirements on system performance, scenario rarity, and the statistical
significance of evaluation increase, so does the amount of required road-
testing. Increasing fleet size can speed up the development cycle. Still,
Autonomy 1.0 SDVs rely on expensive sensors and compute to satisfy the
stringent requirements on perception accuracy imposed by rule-based planning.
This makes SDVs expensive, and so the available budget puts an upper bound on
the development speed and statistical performance guarantees.

##  4 Autonomy 2.0

In this section, we propose Autonomy 2.0, an ML-first approach to self-driving
focused on achieving high scalability. It is based on three key principles: i)
closed-loop simulation, which is learned from the collected real-world driving
logs; ii) decomposing SDV into an end-to-end differentiable neural network;
and iii) the data needed to train the planner and simulator is collected at a
large-scale using commodity sensors.

###  4.1 Data-driven closed-loop reactive simulations

Figure 4: A data-driven reactive simulator [28] enables the synthesis of new,
realistic driving scenarios based on a recorded log. This allows the training
and evaluation self-driving stack offline, without the need for extensive road
testing (sec. 4)

Most of the evaluation in Autonomy 2.0 is done offline in simulation. This is
in contrast to the Autonomy 1.0 reliance on road testing due to the
limitations of rule-based simulation. It does not mean ceasing road-testing
altogether, but its purpose is less prominent in the development cycle and is
mainly used to verify the simulator performance. For simulation to be an
effective replacement of road testing for development, it needs three
properties:

  1. 1.

an appropriate simulation state representation for the task;

  2. 2.

the ability to synthesise diverse and realistic driving scenarios with high
fidelity and reactivity;

  3. 3.

when applied to new scenarios and geographies, performance increases with the
amount of data.

Simulation outcomes must be very realistic since any difference between
simulation and reality would result in inaccurate performance estimates, but
it does not need to be photo-realistic [29] and instead focus only on the
representation of the planner. We reason that that in order to achieve a high
level of realism, the simulation itself must be learned directly from the real
world. Recently, [28] showed how realistic and reactive simulations can be
constructed from previously collected real-world logs using birds-eye-view
representations. As shown in Fig. 4, this simulation can be then deployed to
turn any log into a reactive simulator for testing autonomous driving
policies. The appealing property of such an approach is that simulation
fidelity scales with data. By collecting more logs, the coverage of simulation
proportionally grows as each log can be turned into a simulation case.
However, further work needs to be done to address behavioural multi-modality
since we cannot know the intentions of other traffic participants ahead of
time, and the same action from an SDV can lead to many different outcomes.
There are initial works [30, 18, 31], but more work is required (Sec. 5).

###  4.2 A fully differentiable stack trained from human demonstrations

Autonomy 1.0 has rule-based components that are hand-engineered, as are the
human-interpretable interfaces between perception, prediction, planning, and
simulation (Sec. 2). Unlike Autonomy 1.0, the Autonomy 2.0 stack is fully
trainable from human demonstrations, and so its complexity scales in
proportion to the amount of training data. In order to train such a system,
several conditions need to be met:

  1. 1.

each component, including planning, needs to be trainable and end-to-end
differentiable;

  2. 2.

trainable using human demonstrations;

  3. 3.

performance scales with the amount of training data.

End-to-end differentiability is essential as it allows to back-propagate
gradients across the stack. Perception and prediction are already
differentiable in Autonomy 1.0, but planning is not. Recently, [32] have shown
that a high-capacity driving policy can be represented by a neural network and
trained from a collection of human demonstrations. The appealing property is
this avoids the need for hand-engineering of cost function since the training
signal is implicitly given by the demonstrations.

Figure 5: Architecture of a fully differentiable Autonomy 2.0 stack motivated
by [33] (sec. 4.2) that can be trained end-to-end from data without the need
of engineering individual blocks and interfaces. $d$, $h$, $f$, and $g$ are
learnable neural networks. $d$ and $h$ give a latent representation of the
scene on which planning happens. $f$ represents the policy of the SDV and the
agents in the scene. $g$ is the state transition function function. $I_{0}$ is
the input to the network, while $\\{I_{1},\cdot\cdot,I_{3}\\}$ provides
supervision during training.

Training the system requires proper formulation based on the nature of
sequential decision-making tasks. Imitation learning, while simple, suffers
from covariate shift caused by the accumulation of errors [34]. To implicitly
avoid the problem of covariate shift and causal confusion [35] we argue that
the system needs to be trained in a closed loop with the reactive data-driven
simulation outlined Sec. 4. We can generate new driving episodes under a
current policy without further data collection and then update the policy
based on the imitation loss [36].

Furthermore, inspired by [33] we propose that the networks for each component
(i.e. driving policy, perception, and simulation) can be coupled and trained
together as outlined in Fig. 5. The policy $f$ generates the action that the
SDV should take at a given state $s$, and a (learnable) simulator $g$ (Sec. 4)
transitions the state conditioned on the action and the current state. One way
to enable co-training is for the entire stack to be fully differentiable. The
ML planner is then plugged into a differentiable architecture and is trained
jointly with perception. Importantly, it allows the quality of perception
output to be determined in a way that is most suitable for training driving
policies, rather than setting a more arbitrary perception requirement of
centimetre-level accurate 3D bounding boxes. The interpretability property,
though, is still maintained as the learned perception representation can be
back-projected into interpretable space for examination. This approach to
formulating the autonomy problem has many similarities to MuZero [37].

###  4.3 Large-scale low-cost data collection

The systems discussed so far use human demonstrations as training data, i.e.
sensor data with the corresponding trajectory chosen by a human driver serving
as supervision. To unlock production-level performance, these data need to
have:

  1. 1.

enough size and diversity to include the long tail of rare events (Sec. 3);

  2. 2.

sufficient sensor fidelity, i.e. the sensors used to collect the data need to
be accurate enough to train the planner and simulator effectively and

  3. 3.

cheap enough to collect at this scale and fidelity.

While recently we saw the first public datasets with human demonstrations,
these are limited to a few thousand miles of data [38]. Observing the long
tail will likely require collecting hundreds of millions of miles of data
because most driving is uneventful, e.g. there are roughly 5 crashes per
million miles driven in the US [39].

Collecting these volumes constrains us to rely on crowd-sourcing, as using
cars and drivers dedicated solely for data collection is not economically
viable at this scale. It also constrains us to use only low-cost commodity
sensors like cameras. These, however, produce lower perception accuracy than
high fidelity sensors (like HD LIDAR) used in Autonomy 1.0 to power rule-based
planning. Hence, in choosing sensors for data collection, we face a trade-off
between the scalability (how much data we collect) and the fidelity (the
resulting perception accuracy) we can expect from them.

Sensor type | Perception accuracy on KITTI | Scalability
---|---|---
| (car mAP, moderate) |
Monocular cameras | 22$\%$ [40] | High
Stereo cameras | 52$\%$ [41] | Medium
Stereo + sparse LIDAR | 64$\%$ [42] | Medium
HD LIDAR | 75$\%$ [43] | Low
Table 1: When collecting data, we face a trade-off between sensor scalability
and fidelity, which impacts directly perception accuracy. Scalable sensors
like cameras (top two rows) achieve lower accuracy than the expensive ones
traditionally used in SDVs (e.g. LIDAR, bottom row). Recent progress in
perception algorithms has reduced this gap, and solutions with cameras and
sparse LIDAR (third row) have become really competitive on standard perception
benchmarks like KITTI [44]

Which sensors should we use? Recent progress in perception algorithms showed a
reduced gap in perception accuracy between HD and commodity sensors like
cameras [40, 41] and sparse LIDAR [42] on the KITTI benchmark [44] (Table 1).
Moreover, the actual perception accuracy requirements for training Autonomy
2.0 systems [45] might be relaxed by training perception and planning jointly
(Sec. 4.2). This is corroborated by recent results suggesting that training ML
planners on large amounts of data with lower accuracy perception might be
preferable to smaller amounts with higher accuracy [46]. As the sensor prices
stand today, this points to the camera only (and optionally sparse LIDAR) as
the most promising solution to strike a balance between cost and fidelity, but
this remains an open problem.

##  5 Open questions

In this work, we outlined the Autonomy 2.0 paradigm, which is designed to
solve self-driving using an ML-first approach. By removing the human-in-the-
loop, this paradigm is significantly more scalable, which we argue is the main
limitation for achieving high SDV performance. While promising, this direction
leaves many open questions that are yet to be answered, such as:

  * •

What is the appropriate state representation for simulation and planning? How
much should it be learned vs interpretable?

  * •

How should we measure the probability of scenarios? How should we detect
outlier, never-before-seen cases?

  * •

What are the limits of what can be trained offline from human demonstrations
vs need real-time reasoning using search?

  * •

How much do we need to simulate? How should we measure the performance of the
offline simulation itself?

  * •

How much data do we need to train high-performing planning and simulation
components? What sensors should we use for large-scale data collection?

Answering these questions is critical to self-driving and other real-world
robotic problems and can stimulate the research community to unlock high-
performing SDVs sooner rather than later.

#### Acknowledgments

We would like to thank following contributors who work on Autonomy 2.0 at Lyft
Level 5: Moritz Niendorf, Qiangui Huang, Oliver Scheel, Matt Vitelli, Błażej
Osiński, Ana Ferreira, Luca Bergamini, Ray Gao, Yawei Ye, Sammy Sidhu, Lukas
Platinsky, Christian Pereone, Jasper Friedrichs, Maciej Wołczyk, Long Chen,
and Sacha Arnoud.

## References

  * [1] Levels of autonomy.  https://www.sae.org/binaries/content/assets/cm/content/blog/sae-j3016-visual-chart_5.3.21.pdf.
  * 201 [2012] Sergey Brin predicts self driving vehicles are five years away.  https://www.youtube.com/watch?v=BEjxQd219is, 2012.
  * 201 [2015] Sam Altman predicts self driving vehicles are four years away.  https://techcrunch.com/2015/10/06/elon-musk-sam-altman-say-self-driving-cars-are-going-to-be-on-the-road-in-just-a-few-years/, 2015\.
  * 201 [2016] Ford autonomy team predicts self driving vehicles are five years away.  https://www.zdnet.com/article/ford-self-driving-cars-are-five-years-away-from-changing-the-world/, 2016\.
  * Thrun et al. [2005] S. Thrun, W. Burgard, and D. Fox.  _Probabilistic Robotics (Intelligent Robotics and Autonomous Agents)_.  The MIT Press, 2005.  ISBN 0262201623.
  * Deng et al. [2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Kehl et al. [2017] W. Kehl, F. Manhardt, F. Tombari, S. Ilic, and N. Navab.  Ssd-6d: Making rgb-based 3d detection and 6d pose estimation great again.  2017\.
  * Qi et al. [2016] C. R. Qi, H. Su, K. Mo, and L. J. Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation, 2016.
  * Zhou and Tuzel [2018] Y. Zhou and O. Tuzel.  Voxelnet: End-to-end learning for point cloud based 3d object detection.  In _Int. Conf. on Computer Vision and Pattern Recognition_ , 2018\.
  * Meyer and Kuschk [2019] M. Meyer and G. Kuschk.  Deep learning based 3d object detection for automotive radar and camera.  In _2019 16th European Radar Conference (EuRAD)_ , 2019.
  * Kocić et al. [2018] J. Kocić, N. Jovičić, and V. Drndarević.  Sensors and sensor fusion in autonomous vehicles.  In _2018 26th Telecommunications Forum (TELFOR)_ , 2018.
  * Cho et al. [2014] H. Cho, Y.-W. Seo, B. V. Kumar, and R. R. Rajkumar.  A multi-sensor fusion system for moving object detection and tracking in urban driving environments.  In _Int. Conf. on Robotics and Automation (ICRA)_ , 2014.
  * Kesten et al. [2019] R. Kesten, M. Usman, J. Houston, T. Pandya, K. Nadhamuni, A. Ferreira, M. Yuan, B. Low, A. Jain, P. Ondruska, S. Omari, S. Shah, A. Kulkarni, A. Kazakova, C. Tao, L. Platinsky, W. Jiang, and V. Shet.  Lyft level 5 perception dataset 2020.  https://level5.lyft.com/dataset/, 2019.
  * Caesar et al. [2019] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  _arXiv preprint arXiv:1903.11027_ , 2019.
  * Helbing and Molnar [1998] D. Helbing and P. Molnar.  Social force model for pedestrian dynamics.  _Physical Review E_ , 1998.
  * Lee et al. [2017] N. Lee, W. Choi, P. Vernaza, C. Choy, P. H. S. Torr, and M. Chandraker.  Desire: Distant future prediction in dynamic scenes with interacting agents.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2165–2174, 2017.
  * Sadeghian et al. [2019] A. Sadeghian, V. Kosaraju, A. Sadeghian, N. Hirose, H. Rezatofighi, and S. Savarese.  Sophie: An attentive gan for predicting paths compliant to social and physical constraints.  2019\.
  * Chai et al. [2019] Y. Chai, B. Sapp, M. Bansal, and D. Anguelov.  Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction, 2019.
  * N N et al. [2021] S. N N, R. Moslemi, F. Pittaluga, B. Liu, and M. Chandraker.  Divide-and-conquer for lane-aware diverse trajectory prediction, 2021\.
  * Mnih et al. [2013] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller.  Playing atari with deep reinforcement learning.  12 2013.
  * Silver et al. [2016] D. Silver, A. Huang, C. Maddison, A. Guez, L. Sifre, G. Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis.  Mastering the game of go with deep neural networks and tree search.  _Nature_ , 529:484–489, 01 2016.  doi:10.1038/nature16961.
  * Ziegler et al. [2008] J. Ziegler, M. Werling, and J. Schroder.  Navigating car-like robots in unstructured environments using an obstacle sensitive cost function.  In _Intelligent Vehicles Symposium_ , 2008.
  * Jr. and Lavalle [2000] J. J. K. Jr. and S. M. Lavalle.  Rrt-connect: An efficient approach to single-query path planning.  In _Int. Conf. on Robotics and Automation_ , 2000.
  * Bandyopadhyay et al. [2013] T. Bandyopadhyay, K. S. Won, E. Frazzoli, D. Hsu, W. S. Lee, and D. Rus.  Intention-aware motion planning.  In E. Frazzoli, T. Lozano-Perez, N. Roy, and D. Rus, editors, _Algorithmic Foundations of Robotics X_ , 2013.
  * Montemerlo et al. [2009] M. Montemerlo, J. Becker, S. Bhat, H. Dahlkamp, D. Dolgov, S. Ettinger, D. Haehnel, T. Hilden, G. Hoffmann, B. Huhnke, D. Johnston, S. Klumpp, D. Langer, A. Levandowski, J. Levinson, J. Marcil, D. Orenstein, J. Paefgen, I. Penny, A. Petrovskaya, M. Pflueger, G. Stanek, D. Stavens, A. Vogt, and S. Thrun.  _Junior: The Stanford Entry in the Urban Challenge_.  2009\.
  * Buehler [2009] Buehler.  The darpa urban challenge.  2009\.
  * Dosovitskiy et al. [2017] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun.  CARLA: An open urban driving simulator.  In _1st Annual Conference on Robot Learning_ , 2017.
  * Bergamini et al. [2021] L. Bergamini, Y. Ye, O. Scheel, L. Chen, C. Hu, L. D. Pero, B. Osinski, H. Grimmett, and P. Ondruska.  Simnet: Learning reactive self-driving simulations from real-world observations, 2021.
  * Dosovitskiy et al. [2017] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun.  CARLA: An open urban driving simulator.  In _Proceedings of the 1st Annual Conference on Robot Learning_ , pages 1–16, 2017.
  * Rhinehart et al. [2021] N. Rhinehart, J. He, C. Packer, M. Wright, R. McAllister, J. Gonzalez, and S. Levine.  Contingencies from observations: Tractable contingency planning with learned behavior models, 04 2021.
  * Suo et al. [2021] S. Suo, S. Regalado, S. Casas, and R. Urtasun.  Trafficsim: Learning to simulate realistic multi-agent behaviors.  2021\.
  * Bansal et al. [2019] M. Bansal, A. Krizhevsky, and A. Ogale.  Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst, 2019.
  * Silver et al. [2018] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis.  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  _Science_ , 2018.
  * Ross et al. [2011] S. Ross, G. Gordon, and D. Bagnell.  A reduction of imitation learning and structured prediction to no-regret online learning.  In _Proceedings of the fourteenth international conference on artificial intelligence and statistics_ , pages 627–635. JMLR Workshop and Conference Proceedings, 2011.
  * de Haan et al. [2019] P. de Haan, D. Jayaraman, and S. Levine.  Causal confusion in imitation learning.  _CoRR_ , 2019.
  * Ho and Ermon [2016] J. Ho and S. Ermon.  Generative adversarial imitation learning.  _CoRR_ , 2016.
  * Schrittwieser et al. [2020] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, T. Lillicrap, and D. Silver.  Mastering atari, go, chess and shogi by planning with a learned model.  _Nature_ , 2020.
  * Houston et al. [2020] J. Houston, G. Zuidhof, L. Bergamini, Y. Ye, L. Chen, A. Jain, S. Omari, V. Iglovikov, and P. Ondruska.  One thousand and one hours: Self-driving motion prediction dataset, 2020\.
  * [39] Bureau of transportation statistics motor vehicle safety data.  https://www.bts.gov/content/motor-vehicle-safety-data.
  * Luo et al. [2021] S. Luo, H. Dai, L. Shao, and Y. Ding.  M3dssd: Monocular 3d single stage object detector.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6145–6154, June 2021.
  * Chen et al. [2020] Y. Chen, S. Liu, X. Shen, and J. Jia.  Dsgn: Deep stereo geometry network for 3d object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * Wang et al. [2019] Y. Wang, W.-L. Chao, D. Garg, B. Hariharan, M. Campbell, and K. Q. Weinberger.  Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * Lang et al. [2019] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom.  Pointpillars: Fast encoders for object detection from point clouds.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * Geiger et al. [2013] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun.  Vision meets robotics: The kitti dataset.  _International Journal of Robotics Research (IJRR)_ , 2013.
  * Philion et al. [2020] J. Philion, A. Kar, and S. Fidler.  Learning to evaluate perception models using planner-centric metrics.  2020\.
  * Chen et al. [2021] L. Chen, L. Platinsky, S. Speichert, B. Osinski, O. Scheel, Y. Ye, H. Grimmett, L. del Pero, and P. Ondruska.  What data do we need for training an av motion planner?, 2021.