#  RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-
Context Learning in Multi-Modal Large Language Model

Jianhao Yuan1, Shuyang Sun1, Daniel Omeiza1, Bo Zhao2, Paul Newman1, Lars
Kunze1, Matthew Gadd1
1 University of Oxford 2 Beijing Academy of Artificial Intelligence
<EMAIL_ADDRESS>
###### Abstract

Robots powered by ‘blackbox’ models need to provide human-understandable
explanations which we can trust. Hence, explainability plays a critical role
in trustworthy autonomous decision-making to foster transparency and
acceptance among end users, especially in complex autonomous driving. Recent
advancements in Multi-Modal Large Language models (MLLMs) have shown promising
potential in enhancing the explainability as a driving agent by producing
control predictions along with natural language explanations. However, severe
data scarcity due to expensive annotation costs and significant domain gaps
between different datasets makes the development of a robust and generalisable
system an extremely challenging task. Moreover, the prohibitively expensive
training requirements of MLLM and the unsolved problem of catastrophic
forgetting further limit their generalisability post-deployment. To address
these challenges, we present RAG-Driver, a novel retrieval-augmented multi-
modal large language model that leverages in-context learning for high-
performance, explainable, and generalisable autonomous driving. By grounding
in retrieved expert demonstration, we empirically validate that RAG-Driver
achieves state-of-the-art performance in producing driving action
explanations, justifications, and control signal prediction. More importantly,
it exhibits exceptional zero-shot generalisation capabilities to unseen
environments without further training endeavours111Project Page:
https://yuanjianhao508.github.io/RAG-Driver.

###### Index Terms:

Autonomous driving, multi-modal language model, end-to-end driving, domain
generalisation

##  I Introduction

Driven by the emerging development of deep learning, autonomous driving has
observed a paradigm shift from rules-based decision systems [66, 21] to data-
driven learning-based approaches [28, 6, 36]. However, this comes at the cost
of transparency in decision-making, especially for end-to-end autonomous
driving systems which are considered black-box in nature [13]. Thus, in
addition to precision in action control, explanation provision is key in
ensuring trustworthy decision-making to reconcile the system’s decisions with
end-user expectations to foster confidence and acceptance [79, 8, 57] in
dynamic driving environments.

Traditional approaches have mainly relied on attention visualisation [5, 7,
55] as a proxy to rationalise the decisions of the black-box systems or
auxiliary intermediate tasks such as semantic segmentation [25, 32], object
detection [16, 31], and affordance prediction [68, 45] provide meaningful
intermediate representation for decision-making. However, these methods do not
engage end-users in the dialogue as they are one-directional and not readily
comprehensible by the general users for the purpose of fostering trust and
confidence. An alternative promising approach is the integration of natural
language explanations [38, 33, 54], in particular through Multi-Modal Large
Language Models (MLLMs) [1, 70]. These models, pre-trained on extensive web-
scale datasets, demonstrate remarkable reasoning capacity, enabling the
transformation of complex vehicular decision-making processes into more
understandable narrative formats, thereby offering a new layer of
explainability to conventional systems.

While several early attempts have demonstrated the potential of MLLMs as
general explainable driving agents [78, 76, 51], these methods fall short of
human-level understanding. One of the limitations is their failure to
generalise to unseen environments. A primary obstacle is the lack of high-
quality annotated data [56], coupled with the significant domain shift across
various datasets [23], which hinders the models’ generalisation capacity to
novel environments outside of the training data distribution. Another critical
challenge is the prohibitively expensive training requirement and the unsolved
problem of catastrophic forgetting [39], which make re-training or fine-tuning
impractical solutions due to the immense computational demands and severe
performance degradation. Consequently, this further limits the models’
generalisability after deployment, as they struggle to effectively utilise new
data in constantly evolving environments and driving scenarios.

To address these challenges, we introduce RAG-Driver, a novel retrieval-
augmented multi-modal large language model tailored for generalisable and
explainable end-to-end driving. As illustrated in Fig.1, it outputs natural
language texts corresponding to (1) the driving action and (2) justification
of that driving action along with (3) numerical control signals based on
driving videos. The natural language texts are aligned with control signals
during in-context learning [10] to enable faithful introspective explanation
provision. The novelty of RAG-Driver is the integration of retrieval-augmented
in-context learning (RA-ICL) mechanisms that significantly improve
generalisation performance in unseen driving environments. It allows efficient
recall of similar driving scenarios as contextual information augmenting MLLM
prediction through implicit meta-optimisation (Sec. III-C). Through extensive
experiments, we show that RAG-Driver outperforms existing methods on both in-
domain deployments as well as deployment in unseen environments (without any
fine-tuning). By being grounded in analogical demonstrations, our framework
significantly reduces the need for continuous retraining while enhancing the
generalisability and quality of generated explanatory texts. Our primary
contributions are as follows:

  1. 1.

Proposing a novel retrieval-augmented in-context learning method for Multi-
Modal Large Language Model (MLLM) based generalisable and explainable driving.

  2. 2.

Achieving state-of-the-art introspective driving explanation performance on
the standard benchmark BDD-X [38].

  3. 3.

Demonstrating exceptional zero-shot generalisation capacity to unseen
scenarios without training effort through a customised dataset Spoken-SAX
featuring video sequences narrated by a professional driving instructor.

##  II Related Work

###  II-A Explainable End-to-End Autonomous Driving

End-to-end learned driving [13] maps directly from raw sensor input to vehicle
control signals. This data-driven, joint optimisation of perception,
prediction, and planning can be simple and efficient [28]. In this area,
various learning-based approaches, including behavior cloning [6, 12, 61, 83,
60], inverse optimal control [81, 65, 75], and reinforcement learning [36, 71,
11, 44] are promising. A key area of focus in this area is explainability
[80], which is crucial for improving transparency and building trust towards
wider public acceptance of autonomous systems [54, 22, 57]. One line of works
leverages attention visualisation – either to directly identify salient
regions of input images that are important for driving decision-making [37, 5,
7] or to assist feature aggregation for downstream motion planning tasks [55,
15, 77, 63]. Another line of work uses intermediate auxiliary tasks such as
semantic segmentation [25, 32], object detection [16, 31], and affordance
prediction [68, 45] which help to decoder latent representations to human-
understandable representation. While these methods provide explainable
mechanisms by associating decision-making processes with semantic or visual
representations, they are not readily comprehensible by the general users for
the purpose of fostering trust and confidence.

Alternatively, recent research shows promise in utilising natural language
explanation. Several works develop specialist explainers [38, 41] using
attention alignment in visual input and textual generation for grounded
driving action explanation. ADAPT [33] uses a vision-language transformer with
separate decoder for caption generation alongside control signal prediction.
More recently, several works explore the potential of Multi-modal Large
Language Models (MLLMs). Works such as, DriveGPT4 [78], Lingo [54], and
DrivingMLM [76] have shown promising potential in general question-answering
for driving and action planning. However, a common obstacle in both specialist
and MLLM-based generalist models is the scarcity of data due to expensive
annotation costs and significant domain gaps between different datasets,
making the development of a robust and generalisable model an extremely
challenging task. In our work, we aim to overcome these obstacles by employing
a more robust inference paradigm of retrieval-augmented in-context learning to
bridge domain gaps and circumvent the need for annotations in new domains.

Figure 1: RAG-Driver Overview: Given a query comprising a video of the current
driving scenario and its corresponding control signal, the process starts with
the input being fed into a Retrieval Engine. It searches within a memory
database for driving experiences that are similar to the current scenario,
thereby providing relevant in-context learning samples. Subsequently, a Multi-
Modal Large Language Model (MLLM) processes both the current query and the
retrieved in-context learning samples. Based on high-level task instructions,
the model engages in various prediction tasks: Action Explanation, Action
Justification, and Next Control Signal Prediction.

###  II-B Multi-Modal Large Language Model

Recent advancements in Large Language Models (LLMs) have paved the way for the
emergence of Multi-modal Large Language Models (MLLMs) [1, 70]. Benefiting
from scalable transformer-based architecture and web-scale training data,
these models have demonstrated notable capabilities in general-purpose visual
understanding tasks. One line of work focuses on modality fusion in the latent
space, offering a scalable, end-to-end solution for MLLMs. For instance,
Flamingo [3] and BLIP2 [43] fuse visual tokens to frozen LLM through gated
attention and query transformers, respectively. LLaVA [48] and MiniGPT4 [86]
use simple Multi-Layer Perceptron (MLP) with visual instruction tuning to
align the pre-trained image encoder to LLM. Most relevant to us is the line of
work focusing on video-language models such as Video-LLaVA [46] and Video-
LLaMA [82], which integrate pre-trained video encoders into LLMs using
strategies similar to those in image-based models.

With remarkable perception and reasoning capacity, MLLMs reveal promising
potential in various robotics tasks such as reasoning [20, 2, 30] and planning
[62, 9, 47, 35]. The most similar to us is the idea that engages a generalist
foundation model for end-to-end embodied agent. PaLM-e [20] injects images,
state estimates, and other sensor modalities to LLM and autoregressively
produces natural language commands. RT-2 [9] and RT-X [58] fine-tune on image
and low-level robot control signal pairs to perform end-to-end robot control.
Specifically in driving, numerous approaches leverage language-only LLM for
decision-making [79] and then augment it with external perception module
feedback [53, 24, 14], designed chain-of-thought reasoning template [53, 52]
or downstream planner [67, 69, 14] to form a system-level driving agent.
Another more relevant line of work is end-to-end driving agents. DriveGPT4
[78] leverages a finetuned video-language model Valley [50] on driving
specific visual instruction tuning based on BDD-X [38]. Dolphins [51] further
use the designed Grounded chain-of-thought to enhance reasoning capacity.
DrivingMLM [76] and Reason2Drive [56] scale up the driving visual instruction
tuning dataset through simulator-based data engine and annotation of existing
large-scale datasets, respectively. While these approaches have demonstrated
the potentials of MLLM, the prohibitively expensive training cost and the
unsolved problem of catastrophic forgetting, which makes re-training or fine-
tuning post-deployment challenging, further limit their generalisation
capacity to unseen driving environments. To solve this problem, we leverage a
training-free retrieval-augmented in-context learning mechanism.

###  II-C In-Context Learning and Retrieval-Augmented Generation

While LLMs demonstrate strong generative and reasoning capacity, there are
still several issues associated with their output, such as hallucination [29],
and slow knowledge updates [34]. In-context Learning (ICL) [10, 18] has
emerged as a promising approach in LLM inference, potentially addressing
several of these issues. This paradigm involves providing a test query
alongside a few demonstration examples as contextual information. The LLM then
generates an output for the test instance based on analogies drawn from
context, without any updates to its parameters [64]. While ICL has been
observed to enhance generalisability in various Natural Language Processing
(NLP) tasks, its application in multi-modal contexts remains less explored,
potentially due to the challenges associated with curating structured high-
quality multi-modal ICL datasets. Retrieval-Augmented Generation (RAG) [42] is
another important inference paradigm for LLM. It provides an external
knowledge database to augment the compressed model knowledge within LLM in
inference by dynamically retrieving relevant information pieces as contextual
information. One of its promising applications is leveraging a systematic
approach to curate In-Context Learning (ICL) examples. In this work, we build
upon these inference paradigms and extend their application to Multimodal
Large Language Models (MLLMs). We introduce a retrieval-augmented in-context
learning mechanism through a curated multi-modal driving in-context
instruction tuning dataset and a vector similarity-based retrieval engine
specifically tailored for driving applications.

##  III Method

RAG-Driver is a retrieval-augmented, multi-modal large language model (MLLM)
for generalisable explainable end-to-end driving. Its multi-tasking
capabilities encompass three key areas: (1) Action Explanation, providing a
human-understandable driving action description; (2) Action Justification,
elucidating the rationale behind specific driving actions; and (3) Next
Control Signal Prediction, forecasting upcoming control signals in response to
the driving conditions. As shown in Fig. 1, it is composed of two primary
components: a unified perception and planning unit built upon an MLLM backbone
and a memory unit built upon a hybrid vector and textual database. These
components interact through a retrieval engine, enabling robust multi-modal
in-context learning (ICL) during decision-making.

###  III-A Multi-modal Large Language Model Architecture

Following the successful MLLM paradigm of Video-LLaVA [46], we align visual
and language embedding through visual instruction tuning. We leverage a pre-
trained video encoder and LLM and then inject the video embedding into LLM
through a MLP projector to build a fully differentiable MLLM.

Video Encoder We adopt the pre-trained LanguageBind video encoder [85] as our
frozen visual backbone $f_{v}$, which is based on a ViT-B/32 vision
transformer [19]. As shown in Fig. 2, given an input video frame sequence
$V_{i}=\left\\{v_{i}^{1},v_{i}^{2},\ldots,v_{i}^{k}\right\\}\in\mathbb{R}^{3\times
k\times 224\times 224}$, we first split the video into multiple temporal
sequences, each consisting of patches that share the same spatial location
across different frames. These patches are then transformed through a linear
projection for a vision transformer to output video embedding
$\bm{z}_{vo}\in\mathbb{R}^{2048\times 1024}$. The video encoder is pre-trained
with video-language contrastive learning (i.e. CLIP4clip [49]) without further
fine-tuning.

Figure 2: Video Encoder architecture. Video is first split into $k\times
32\times 32$ patches concatenated in time, where these patches are linear
projected to video embedding. Then, the model is trained with video-language
contrastive learning (CLIP4clip) [49] to obtain language-align video
representation.

Cross-Modality Projector We then leverage a two-layer MLP to project and align
the encoded video embedding $\bm{z}_{vo}$ with language token embeddings
$\bm{z}_{v}\in\mathbb{R}^{2048\times 4096}$.

$f_{p}(\bm{z}_{vo})=\text{GELU}\left(W_{2}\cdot\text{GELU}\left(W_{1}\cdot\bm{z}_{vo}\right)\right)$
(1)

In particular, the projector $f_{p}$ takes form in Eq. 1, where we use GELU
[26] as an activation function. We train the projector with a two-stage
training strategy as detailed in Sec. III-B.

Large Language Model Backbone Finally, the LLM takes in both aligned video
embedding $\bm{z}_{v}$ and language embedding $\bm{z}_{t}$ of textual context
information and task instruction to predict both textual action explanation
and numerical control signal. We adopt Vicuna 1.5 7B [84], which is
instruction-tuned based on LLaMA2 [72] as our LLM backbone. For decoder-only
LLM conditioned on the multi-modal contextual prefix
$\bm{z}_{1:n}=[\bm{z}_{v},\bm{z}_{t}]$ of length $N$, the joint probability of
output $\bm{x}_{n+1:L}$ as in Eq. 2, where $P_{\theta}$ is transformer-based
LLM backbone parameterized by $\theta$.

$P(\bm{x}_{n+1:L}\mid\bm{z}_{1:n})=\prod_{l=n+1}^{L}P_{\theta}(\bm{x}_{l}\mid\bm{x}_{1:l-1},\bm{z}_{1:n})$
(2)

Each output token $\bm{x}_{l}$ is then sampled auto-regressively based on
previous output and context and finally decode to language space through a
text de-tokenizer.

###  III-B Training Strategy

Following the visual instruction tuning paradigm [48, 46], we employ a two-
stage training strategy to progressively enable cross-modality alignment and
multi-task driving capacity. In both stages, we leverage the same next-token
prediction cross-entropy loss as in Eq. 3 aiming to maximize the conditional
answer likelihood in Eq. 2, where $\bm{y}$ is the ground truth token.

$\mathcal{L}_{CE}=-\sum_{i=n+1}^{L}\bm{y}_{l}\log
P(\bm{x}_{l}\mid\bm{z}_{1:n})$ (3)

Pre-training In the first pre-training stage, we only train the cross-modality
projector while freezing the visual encoder and LLM. We use a subset VIDAL-10M
[85] which contains 3 million video-caption pairs. This achieves an alignment
between visual and language features by projecting the pre-trained video
embeddings into language tokens understandable by LLM.

Supervised In-context Instruction Tuning While state-of-the-art LLM exhibits
zero-shot ICL capability, several works [18] and our ablation study Sec. IV-D
have shown further improvement when specifically trained on curated ICL
demonstrations. In particular, we construct a multi-modal instruction tuning
dataset with structured ICL examples based on the BDD-X dataset [38] resulting
in 16K video question and answer pairs. As shown in Fig. 3, for an 8-frame
driving video sequence with associated control signals—speed, course,
acceleration, and curvature as the current query, we use the retrieval
mechanism as in Sec. III-C to retrieve 2 relevant driving experiences, which
are then prefixed to the current query as ICL examples. The dataset is
tailored to support three distinct tasks, each represented through question-
answer pairs in natural language. Note that, with (1) Action Explanation and
(2) Justification naturally represented as natural language, (3) Control
Signal prediction is also formed as language token prediction; this is
feasible due to the distinct mapping of numerical values to specific tokens
within the language model dictionary.

Figure 3: Query Sample. A single query includes three primary components: (1)
Contextual System Information, a constant element across all queries that
offers high-level task instruction; (2) Current Query, comprising all driving
information and specific task instruction in natural language question-answer
format, except driving video (i.e. $<$video$>$) as visual input. Note that the
red sentence is the expected output from MLLM; (3) Multiple In-Context
Learning Examples, retrieved from memory, which serve as an analogical
demonstration of a complete reasoning process—from sensor input in driving
scenarios to action justification and control prediction.

###  III-C Retrival-Augmented In-context Learning

Another critical component of the system is the memory unit, which consists of
a database and a retrieval engine. The database incorporates vectorised video
embedding $\bm{z}_{vo}$, which is extracted with the same video encoder as in
Sec. III-A and control signals $\bm{c}\in\mathbb{R}^{28}$ directly from sensor
recording. Each vector is uniquely associated with the corresponding human
expert textual explanation and justification from train samples as in Sec.
III-B.

Retrieval Mechanism To perform the retrieval, we first leverage a lightweight
MLP projector of the same structure as in Eq. 1 to project the heterogeneous
video and control signal embedding into the same hybrid embedding
$\bm{s}\in\mathbb{R}^{1024}$ through metric learning [27]. In particular, we
adopt a triplet loss with Euclidean distance as shown in Eq. 4:

$\mathcal{L}_{Tri}(\bm{a},\bm{p},\bm{n})=\operatorname*{max}(||\bm{a}-\bm{p}||_{2}-||\bm{a}-\bm{n}||_{2}+\operatorname{margin},0)$
(4)

where the positive $(\bm{a},\bm{p})$ and negative pairs $(\bm{n},\bm{p})$
between hybrid embedding $\bm{s}$ are selected based on text similarity (i.e.,
TF-IDF Score) of driving action and justification in the BDD-X training set as
we aim to form the metric space such that the scenarios lead to similar
driving actions close together and vice versa. This approach addresses the
limitations of relying solely on visual similarity, which we have empirically
found can result in sub-optimal performance (Sec. IV-D), and also solves the
problem of heterogeneous sensor inputs that are hard for similarity
comparison. We then perform retrieval through an efficient vector similarity
search. Given a query vector $\bm{s}^{q}$ the cosine similarity between the
query vector and each vector in the database is computed as follows:

$S_{c}(\bm{s}_{q},\bm{s}^{(i)})=\frac{\bm{s}_{q}\cdot\bm{s}^{(i)}}{\|\bm{s}_{q}\|\|\bm{s}^{(i)}\|}$
(5)

Subsequently, we consistently select the two most relevant driving samples
based on this similarity score. These samples represent the entire reasoning
process, from contextual information to question-answer pairs, as illustrated
in Fig. 3.

Retrieval Augmented In-Context Learning (RA-ICL) To perform the RA-ICL, we
prefix retrieved samples before the current query, facilitating an implicit
gradient descent through the meta-optimiser of LLM, as proved in [17]. This
approach is also applicable to MLLM with architecture specified in Sec. III-A.
For a given transformer-based pre-trained MLLM modeling the prefix-conditioned
output probability as in Eq. 2, consider one head of the multi-head attention
in a single transformer block as follows:

$\displaystyle\mathcal{F}_{\mathrm{ICL}}$
$\displaystyle(\bm{z}_{1:n})=\operatorname{Attention}(Q,K,V)$ (6)
$\displaystyle=$ $\displaystyle
W_{V}\left[\bm{z}_{icl};\bm{z}_{q}\right]\operatorname{Softmax}\left(\frac{\left(W_{K}\bm{z}_{1:n}\right)^{T}(W_{Q}\bm{z}_{1:n})}{\sqrt{d^{i}}}\right)$

where $\bm{z}_{1:n}=\left[\bm{z}_{icl};\bm{z}_{q}\right]$ represented the
conditional prefix consist with ICL example $\bm{z}_{icl}$ and current query
embeddings $\bm{z}_{q}$, respectively. $W_{Q,K,V}\in\mathbb{R}^{d^{i}\times
d^{o}}$ is the linear transformation on query, key, and value in attention
block, respectively. Now, in [17] in-context learning was shown to effectively
achieve a meta-optimisation to implicitly update the model with an estimated
meta-gradient. We provide in our work a new, alternative derivation of this,
supported by more recent work in [40] on the following softmax-free linear
attention expressions

$\displaystyle\mathcal{F}_{\mathrm{ICL}}(\bm{z}_{1:n})=W_{V}\left[\bm{z}_{icl};\bm{z}_{q}\right]\left(W_{K}\left[\bm{z}_{icl};\bm{z}_{q}\right]\right)^{T}W_{Q}\bm{z}_{1:n}$
(7)

This can be simplified, with a more detailed derivation in Sec. -D, as

$\displaystyle\mathcal{F}_{\mathrm{ICL}}(\bm{z}_{1:n})=(\Delta{}W_{ICL}+W_{ZSL})W_{Q}\bm{z}_{1:n}$
(8)

where we separate out the terms $W_{ZSL}$ independent of the ICL examples and
dependent solely on the current query from those $\Delta{}W_{ICL}$ dependent
on the ICL examples, given by:

$\displaystyle\Delta{}W_{ICL}$
$\displaystyle=\sum_{i}~{}W_{V}\bm{z}_{icl,i}(W_{K}\bm{z}_{icl,i})^{T}$ (9)
$\displaystyle W_{ZSL}$ $\displaystyle=W_{V}\bm{z}_{q}(W_{K}\bm{z}_{q})^{T}$

Now, with more detail in Sec. -D, a forward-pass

$\mathcal{F}(\mathbf{x})=(W_{0}+\Delta{}W)\mathbf{x}$ (10)

through a linear layer $\mathcal{F}$ after its weights $W_{0}$ have been
updated by $\Delta{}W$ has the weight updates coupled to the input in the form

$\displaystyle\Delta{}W\mathbf{x}=\sum_{i}\eta\frac{\partial{}L}{\partial\mathbf{y}}|_{\mathbf{y}_{i}}\mathbf{x}_{i}^{T}\mathbf{x}$
(11)

where $\mathbf{x}_{i},\mathbf{y}_{i}$ are the (mini-batch) input and output to
the layer that resulted in the weight update when backpropagating loss $L$ and
with learning rate $\eta$. Therefore we have a weighted sum of dot products,
which is akin to an attention mechanism. Indeed, by inspection of similar dot-
product expressions
$W_{V}\bm{z}_{icl,i}(W_{K}\bm{z}_{icl,i})^{T}\leftrightarrow\eta\frac{\partial{}L}{\partial\mathbf{y}}|_{\mathbf{y}_{i}}\mathbf{x}_{i}^{T}\mathbf{x}$
we note that we match the form for the linear layer above

$(W_{ZSL}+\Delta{}W_{ICL})W_{Q}\bm{z}_{1:n}\leftrightarrow(W+\Delta{}W)\mathbf{x}$

This can therefore be interpreted to say that the output of the attention is
adjusted in a meta-optimal way to conform to the samples provided as input
context, much like gradient descent on the linear layer would adjust that
layer to conform to the mini-batch training data, but crucially in the case of
RAG-Driver, without backpropagation.

RA-ICL serves as an efficient inference method boosting the performance of
MLLM in explainable driving without further training effort, where we
empirically verify it is extremely effective in boosting the model prediction
performance and generalisation capacity.

##  IV Experiments

###  IV-A Settings and Datasets

We empirically evaluate the proposed Retrieval-augmented In-Context Learning
(RA-ICL) framework within the Multi-modal Large Language Model (MLLM),
targeting explainable driving applications. We aim to validate its efficacy in
general driving scenarios with a focus on two main aspects: (1) explainability
in driving action explanation and justification. (2) Control Signal
Prediction. We conduct experiments with the BDD-X [38] dataset, which is a
widely adopted benchmark in explainable driving, comprising 77-hour videos
across the US under different road and weather conditions. We customize the
format as shown in Fig. 3, resulting in 16,803 and 2,123 video question-
answering pairs for training and testing, respectively. More importantly, we
further explore the transfer learning capacity of zero-shot generalisation in
unseen environments. We leverage customised dataset comprising 58 testing
question-answering pairs, recorded in London, UK, presenting a significant
distribution shift from the BDD-X dataset.

Benchmark Settings For all experiments, we train the MLLM using the BDD-X
training split. Subsequent evaluation on general explainability and control
signal prediction capabilities tests are conducted on the BDD-X test split
with the BDD-X training split as the memory database. For the transfer
learning experiments, we employ the same foundational model and test it on the
Spoken-SAX but the memory database is constructed using the BDD-X training
split for zero-shot generalisation.

Implementation Detail For each of the driving videos, we uniformly sample the
video to 8 frames and resize it to $224\times 224$ for all frames. For MLLM,
we train the model for one and two epochs in the pre-training and fine-tuning
stages, respectively. For the embedding projector, we train the model for 300
epochs. Further experiment implementation details are provided in Sec. -B.

Evaluation Metric For the driving action description and justification tasks,
we use the same metrics as [33] including 4-gram BLEU (B4) [59], METEOR (M)
[4], and CIDEr (C) [73]. These metrics aim to evaluate text generation
quality, with BLEU focusing on n-gram precision, METEOR incorporating semantic
and syntactic nuances, and CIDEr emphasizing consensus and relevance in tasks
like image captioning. Moreover, For the control signal evaluation, we again
follow [33] and present Root Mean Square Error (RMSE) in both steering angle
($\text{\,}\mathrm{\SIUnitSymbolDegree}$) and speed
($\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-1}$). We also present “tolerant
accuracy” metrics, $A_{\sigma}$, representing the accuracy of predictions when
binarised as within a tolerance threshold $\sigma$ of the ground truth.

TABLE I:  Driving Action Explanation and Justification Evaluation. B4, C, and
M represent BLEU4, CIDEr, and METEOR, respectively, for measuring the accuracy
and quality of textual driving explanation. The upper part indicates the
evaluation on the in-distribution BDD-X test set. The lower part indicates the
zero-shot generalisation evaluation on the Spoken-SAX dataset.

Method | Is Generalist? | Action |  | Justification
---|---|---|---|---
B4 $\uparrow$ | C $\uparrow$ | M $\uparrow$ |  | B4 $\uparrow$ | C $\uparrow$ | M $\uparrow$
In Distribution Performance (BDD-X) |  |  |  |  |  |  |  |
S2VT [74] | $\times$ | $30.2$ | $179.8$ | $27.5$ |  | $6.3$ | $53.4$ | $11.2$
S2VT++ [74] | $\times$ | $27.1$ | $157.0$ | $26.4$ |  | $5.8$ | $52.7$ | $10.9$
SAA [38] | $\times$ | $31.8$ | $214.8$ | $29.1$ |  | 7.1 | $66.1$ | $12.2$
WAA [38] | $\times$ | $32.3$ | $215.8$ | $29.2$ |  | $7.3$ | $69.5$ | $12.2$
ADAPT [33] | $\times$ | $34.6$ | $247.5$ | $30.6$ |  | $11.4$ | $102.6$ | $15.2$
DriveGPT4 [78] | ✓ | $30.0$ | $214.0$ | $29.8$ |  | $9.4$ | $102.7$ | $14.6$
OURS | ✓ | $34.3$ | $260.8$ | $30.7$ |  | $11.1$ | $109.1$ | $14.8$
$\Delta$ w.r.t generalist SOTA |  | +$14.3\%$ | +$21.9\%$ | +$3.0\%$ |  | +$18.1\%$ | +$6.2\%$ | +$1.4\%$
Zero-shot Generalisation (Spoken-SAX) |  |  |  |  |  |  |  |
ADAPT [33] | $\times$ | $0.0$ | $22.3$ | $26.7$ |  | $3.1$ | $8.6$ | $12.4$
Base w/o ICL | ✓ | $0.0$ | $5.7$ | $21.6$ |  | $2.1$ | $4.7$ | $11.4$
OURS w ICL | ✓ | $9.9$ | $48.9$ | $27.5$ |  | $5.6$ | $17.1$ | $14.6$
$\Delta$ w.r.t SOTA |  | +$-\%$ | +$119.3\%$ | +$3.0\%$ |  | +80.6% | +$98.8\%$ | +$17.7\%$

TABLE II: Control Signals Prediction Accuracy Evaluation on BDD-X dataset.

Method | Course | Speed
---|---|---
RMSE(degree)$\downarrow$ | $A_{0.1}\uparrow$ | $A_{0.5}\uparrow$ | $A_{1.0}\uparrow$ | $A_{5.0}\uparrow$ | $A_{10.0}\uparrow$ | RMSE(m/s)$\downarrow$ | $A_{0.1}\uparrow$ | $A_{0.5}\uparrow$ | $A_{1.0}\uparrow$ | $A_{5.0}\uparrow$ | $A_{10.0}\uparrow$
ADAPT [33] | $5.87$ | $54.49$ | $86.39$ | $91.06$ | $97.36$ | $98.20$ | $2.68$ | $11.77$ | $31.79$ | $47.48$ | $92.75$ | $95.87$
DriveGPT4 [78] | $4.57$ | $69.22$ | $79.14$ | $84.47$ | $95.72$ | $96.74$ | $1.09$ | 56.93 | $77.77$ | $87.97$ | $99.00$ | $99.57$
OURS | 4.48 | 74.32 | 88.69 | 93.12 | 98.30 | 99.10 | 0.69 | $51.12$ | 85.54 | 94.49 | 99.81 | 99.91

TABLE III: Ablation Study on Retrieval Strategy and In-Context Learning Phase.
Visual and Hybrid Search refers to the use of video embeddings or projected
hybrid sensor embeddings, respectively. Training and Inference denote whether
in-context learning examples are provided during the corresponding phase.

Retrieval Strategy | ICL Phase | BDD-X
---|---|---
Visual Search | Hybrid Search | Training | Inference | Act. B4 | Act. C | Jus. B4 | Jus. C | Speed Err. | Course Err.
|  |  |  | $29.8$ | $222.1$ | $5.9$ | $58.9$ | $0.67$ | $5.61$
✓ |  |  | ✓ | $0.0$ | $0.0$ | $0.0$ | $0.0$ | $N/A$ | $N/A$
| ✓ |  | ✓ | $0.0$ | $0.0$ | $0.0$ | $0.0$ | $N/A$ | $N/A$
✓ |  | ✓ | ✓ | $31.2$ | $222.1$ | $7.7$ | $83.1$ | $0.78$ | $5.61$
| ✓ | ✓ | ✓ | $34.3$ | $260.9$ | $11.1$ | $109.1$ | $0.69$ | $4.48$

Figure 4: Qualitative Examples. Comparison between human annotated ground
truth (Left) and prediction from our method (Right) from both BDD-X (Top 4)
and Spoken-London (Bottom 4). Each video is downsampled to four frames for
visualisation. Despite subtle differences in wording, our method provides
driving action explanations and justifications that are comprehensible to
humans.

Baselines We compare with various driving action description and justification
baseline methods such as video-language sequence-to-sequence based recurrent
neural network S2VT [74] and visual-attention based convolutional neural
network WAA [38]. Also, we compare with the state-of-the-art explainable
driving methods, including video transformer-based ADAPT [33] and visual
instruction tuned MLLM DriveGPT4 [78] that is further capable of control
signal prediction.

###  IV-B Explainability in Driving Action and Justification

We begin by evaluating the quality and accuracy of explanations and
justifications for driving actions separately. As shown in upper part of Tab.
I, our method demonstrates comparable performance to the state-of-the-art
specialist method ADAPT [33], a characteristic not observed in previous MLLM-
based methods. In particular, when compared with DriveGPT4 [78] which also
uses MLLM with a similar architecture and number of parameters but
incorporates the extra LLaVA-150K dataset [48] for visual instruction tuning,
our approach, relying solely on the BDD-X dataset, outperforms it in terms of
explainability. This is evidenced by an average performance improvement of
10.8% across all metrics. This underscores the effectiveness of ICL in
enhancing the emergent reasoning capabilities of MLLMs.

###  IV-C Control Signal Prediction

We next evaluate the accuracy of control signal predictions for Course (i.e.,
turning angle) and Speed. As indicated in Tab. II, our method surpasses others
in open-loop control accuracy across various tolerance ranges and in terms of
RMSE, significantly outperforming the baseline approach. In particular, when
compared to the state-of-the-art DriveGPT4, which also uses the same visual
input combined with past control signals for autoregressive prediction, our
method stands out by implementing retrieval-augmented ICL examples. This
indicates the analogy in the overall reasoning process provided by the ICL
examples also contributes to the improvement in numerical control signal
prediction.

###  IV-D Ablation Study on Retrieval Strategy

We perform a more comprehensive ablation study to evaluate the efficacy of our
proposed retrieval-augmented in-context learning. We first aim to investigate
the similarity metric for retrieval. In particular, we compare the use of
visual similarity (i.e., video embeddings only) with hybrid similarity (i.e.,
hybrid video and control signal projected embedding Sec. III-C). Our empirical
findings indicate suboptimal performance when using visual similarity,
possibly because it tends to prioritize ICL examples that are most
perceptually similar, rather than effectively demonstrating the reasoning
process. By fine-tuning the embeddings, we not only harness the potential of
heterogeneous multi-modal sensor input but also enable more effective ICL
example retrieval.

We also investigate whether to apply ICL examples during training or solely in
inference time. As shown in Tab. III, we found that the MLLM is incapable of
making reasonable predictions using ICL examples without prior training,
regardless of the retrieval strategy chosen. This suggests that the pre-
trained MLLM is not equipped to effectively perform zero-shot ICL. We
hypothesize that supervised fine-tuning plays a crucial role in enhancing the
ICL capabilities of MLLM, necessitating the provision of reasoning
demonstrations, which aligns with observations in [18].

###  IV-E Generalisation Capacity

One of the critical capacities of autonomous systems is to generalise to
unseen environments out of its training distribution. However, in the domain
of explainable driving, we observe the existing methods are unable to perform
such generalisation, which poses challenges for their deployment. As shown in
lower part of Tab. I, ADAPT and the base MLLM (i.e., training without ICL)
reveal dramatic performance degradation to the in-distribution situation.
However, our method leverages ICL learning examples to demonstrate a
significant performance boost with a large margin. Note that even though the
memory database is constructed with BDD-X, the RA-ICL can still perform
generalisation in a zero-shot manner. This is potentially due to the
robustness of the hybrid retrieval process, where samples with less
distribution shift can still be selected to serve as effective ICL examples.

###  IV-F Qualitative Demonstration

We also demonstrate a series of quantitative examples comparing the driving
action explanation and justification provided by human ground truth and
prediction from our method. As shown in Fig. 4, we observe RAG-Driver produces
robust intelligible action explanation and justification under different
environments (i.e. night time and adversarial weather) with a control signal
close to the human driver record. More importantly, in the out-of-distribution
setting Spoken-SAX as indicated by the clear visual difference, we observe the
prediction also produces human-understandable answers, qualitatively
validating the exceptional zero-shot generalisation capacity.

##  V Limitations and Future Work

This work aims to develop a generalisable explainable driving commentator
using a Machine Learning Language Model (MLLM), addressing a significant
obstacle that has hindered deployment: the poor generalisation capacity.
However, several problems still need to be addressed. For instance, the
prevalent issue of hallucination in MLLM, while mitigated, is still observed.
We hypothesize that this is due to the limited general video understanding
capacity, as the video encoder only processes 8 frames per video. Also, due to
the limited access to open-source models and computational cost, we employ a
relatively small MLLM with 7 billion parameters, which falls short of some
state-of-the-art models (i.e. GPT4V [1], Gemini [70]). We anticipate that the
development of a more powerful MLLM backbone with lower computational costs
could further enhance the application of MLLM in driving.

##  VI Conclusion

We propose RAG-Driver, a Multi-Modal Large Language Model with Retrieval-
augmented In-context Learning capacity designed for generalisable and
explainable end-to-end driving. It exhibits strong capability in providing
numerical control signals, along with explanations and justifications for
driving actions. More importantly, it shows impressive zero-shot
generalisation to unseen environments without the need for additional
training.

## References

  * Achiam et al. [2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Ahn et al. [2022] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al.  Do as i can, not as i say: Grounding language in robotic affordances.  _arXiv preprint arXiv:2204.01691_ , 2022.
  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Banerjee and Lavie [2005] Satanjeev Banerjee and Alon Lavie.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_ , pages 65–72, 2005.
  * Bojarski et al. [2016a] Mariusz Bojarski, Anna Choromanska, Krzysztof Choromanski, Bernhard Firner, Larry Jackel, Urs Muller, and Karol Zieba.  Visualbackprop: efficient visualization of cnns.  _arXiv preprint arXiv:1611.05418_ , 2016a.
  * Bojarski et al. [2016b] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al.  End to end learning for self-driving cars.  _arXiv preprint arXiv:1604.07316_ , 2016b.
  * Bojarski et al. [2017] Mariusz Bojarski, Philip Yeres, Anna Choromanska, Krzysztof Choromanski, Bernhard Firner, Lawrence Jackel, and Urs Muller.  Explaining how a deep neural network trained with end-to-end learning steers a car.  _arXiv preprint arXiv:1704.07911_ , 2017.
  * Bonnefon et al. [2016] Jean-François Bonnefon, Azim Shariff, and Iyad Rahwan.  The social dilemma of autonomous vehicles.  _Science_ , 352(6293):1573–1576, 2016.
  * Brohan et al. [2023] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al.  Rt-2: Vision-language-action models transfer web knowledge to robotic control.  _arXiv preprint arXiv:2307.15818_ , 2023.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chekroun et al. [2023] Raphael Chekroun, Marin Toromanoff, Sascha Hornauer, and Fabien Moutarde.  Gri: General reinforced imitation and its application to vision-based autonomous driving.  _Robotics_ , 12(5):127, 2023.
  * Chen et al. [2020] Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl.  Learning by cheating.  In _Conference on Robot Learning_ , pages 66–75. PMLR, 2020.
  * Chen et al. [2023a] Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, and Hongyang Li.  End-to-end autonomous driving: Challenges and frontiers.  _arXiv preprint arXiv:2306.16927_ , 2023a.
  * Chen et al. [2023b] Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton.  Driving with llms: Fusing object-level vector modality for explainable autonomous driving.  _arXiv preprint arXiv:2310.01957_ , 2023b.
  * Chitta et al. [2021] Kashyap Chitta, Aditya Prakash, and Andreas Geiger.  Neat: Neural attention fields for end-to-end autonomous driving.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 15793–15803, 2021.
  * Chitta et al. [2022] Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger.  Transfuser: Imitation with transformer-based sensor fusion for autonomous driving.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * Dai et al. [2022] Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei.  Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers.  _arXiv preprint arXiv:2212.10559_ , 2022.
  * Dong et al. [2022] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui.  A survey for in-context learning.  _arXiv preprint arXiv:2301.00234_ , 2022.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Driess et al. [2023] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al.  Palm-e: An embodied multimodal language model.  _arXiv preprint arXiv:2303.03378_ , 2023.
  * Faisal et al. [2019] Asif Faisal, Md Kamruzzaman, Tan Yigitcanlar, and Graham Currie.  Understanding autonomous vehicles.  _Journal of transport and land use_ , 12(1):45–72, 2019.
  * Gadd et al. [2020] Matthew Gadd, Daniele De Martini, Letizia Marchegiani, Paul Newman, and Lars Kunze.  Sense–assess–explain (sax): Building trust in autonomous vehicles in challenging real-world driving scenarios.  In _2020 IEEE Intelligent Vehicles Symposium (IV)_ , pages 150–155. IEEE, 2020.
  * Gulrajani and Lopez-Paz [2020] Ishaan Gulrajani and David Lopez-Paz.  In search of lost domain generalization.  _arXiv preprint arXiv:2007.01434_ , 2020.
  * Han et al. [2024] Wencheng Han, Dongqian Guo, Cheng-Zhong Xu, and Jianbing Shen.  Dme-driver: Integrating human decision logic and 3d scene perception in autonomous driving.  _arXiv preprint arXiv:2401.03641_ , 2024.
  * Hawke et al. [2020] Jeffrey Hawke, Richard Shen, Corina Gurau, Siddharth Sharma, Daniele Reda, Nikolay Nikolov, Przemysław Mazur, Sean Micklethwaite, Nicolas Griffiths, Amar Shah, et al.  Urban driving with conditional imitation learning.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 251–257. IEEE, 2020.
  * Hendrycks and Gimpel [2016] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  _arXiv preprint arXiv:1606.08415_ , 2016.
  * Hoffer and Ailon [2015] Elad Hoffer and Nir Ailon.  Deep metric learning using triplet network.  In _Similarity-Based Pattern Recognition: Third International Workshop, SIMBAD 2015, Copenhagen, Denmark, October 12-14, 2015. Proceedings 3_ , pages 84–92. Springer, 2015.
  * Hu et al. [2023] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li.  Planning-oriented autonomous driving.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 17853–17862, June 2023.
  * Huang et al. [2023a] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al.  A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.  _arXiv preprint arXiv:2311.05232_ , 2023a.
  * Huang et al. [2023b] Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li.  Instruct2act: Mapping multi-modality instructions to robotic actions with large language model.  _arXiv preprint arXiv:2305.11176_ , 2023b.
  * Jaeger et al. [2023] Bernhard Jaeger, Kashyap Chitta, and Andreas Geiger.  Hidden biases of end-to-end driving models.  _arXiv preprint arXiv:2306.07957_ , 2023.
  * Jain et al. [2023] Kanishk Jain, Varun Chhangani, Amogh Tiwari, K Madhava Krishna, and Vineet Gandhi.  Ground then navigate: Language-guided navigation in dynamic scenes.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 4113–4120. IEEE, 2023.
  * Jin et al. [2023] Bu Jin, Xinyu Liu, Yupeng Zheng, Pengfei Li, Hao Zhao, Tong Zhang, Yuhang Zheng, Guyue Zhou, and Jingjing Liu.  Adapt: Action-aware driving caption transformer.  _arXiv preprint arXiv:2302.00673_ , 2023.
  * Kandpal et al. [2023] Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel.  Large language models struggle to learn long-tail knowledge.  In _International Conference on Machine Learning_ , pages 15696–15707. PMLR, 2023.
  * Kassab et al. [2023] Christina Kassab, Matias Mattamala, Lintong Zhang, and Maurice Fallon.  Language-extended indoor slam (lexis): A versatile system for real-time visual scene understanding.  _arXiv preprint arXiv:2309.15065_ , 2023.
  * Kendall et al. [2019] Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah.  Learning to drive in a day.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pages 8248–8254. IEEE, 2019.
  * Kim and Canny [2017] Jinkyu Kim and John Canny.  Interpretable learning for self-driving cars by visualizing causal attention.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2942–2950, 2017.
  * Kim et al. [2018] Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata.  Textual explanations for self-driving vehicles.  _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018.
  * Kirkpatrick et al. [2017] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ , 114(13):3521–3526, 2017.
  * Koohpayegani and Pirsiavash [2024] Soroush Abbasi Koohpayegani and Hamed Pirsiavash.  Sima: Simple softmax-free attention for vision transformers.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 2607–2617, 2024.
  * Kühn et al. [2023] Marc Alexander Kühn, Daniel Omeiza, and Lars Kunze.  Textual explanations for automated commentary driving.  _arXiv preprint arXiv:2304.08178_ , 2023.
  * Lewis et al. [2020] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474, 2020.
  * Li et al. [2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023.
  * Li et al. [2022] Quanyi Li, Zhenghao Peng, and Bolei Zhou.  Efficient learning of safe driving policy via human-ai copilot optimization.  _arXiv preprint arXiv:2202.10341_ , 2022.
  * Li et al. [2018] Zhihao Li, Toshiyuki Motoyoshi, Kazuma Sasaki, Tetsuya Ogata, and Shigeki Sugano.  Rethinking self-driving: Multi-task knowledge for better generalization and accident explanation ability.  _arXiv preprint arXiv:1809.11100_ , 2018.
  * Lin et al. [2023a] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan.  Video-llava: Learning united visual representation by alignment before projection.  _arXiv preprint arXiv:2311.10122_ , 2023a.
  * Lin et al. [2023b] Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg.  Text2motion: From natural language instructions to feasible plans.  _arXiv preprint arXiv:2303.12153_ , 2023b.
  * Liu et al. [2023] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning, 2023.
  * Luo et al. [2021] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li.  CLIP4Clip: An empirical study of clip for end to end video clip retrieval.  _arXiv preprint arXiv:2104.08860_ , 2021.
  * Luo et al. [2023] Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei.  Valley: Video assistant with large language model enhanced ability.  _arXiv preprint arXiv:2306.07207_ , 2023.
  * Ma et al. [2023a] Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao.  Dolphins: Multimodal language model for driving.  _arXiv preprint arXiv:2312.00438_ , 2023a.
  * Ma et al. [2023b] Yunsheng Ma, Can Cui, Xu Cao, Wenqian Ye, Peiran Liu, Juanwu Lu, Amr Abdelraouf, Rohit Gupta, Kyungtae Han, Aniket Bera, et al.  Lampilot: An open benchmark dataset for autonomous driving with language model programs.  _arXiv preprint arXiv:2312.04372_ , 2023b.
  * Mao et al. [2023] Jiageng Mao, Junjie Ye, Yuxi Qian, Marco Pavone, and Yue Wang.  A language agent for autonomous driving.  _arXiv preprint arXiv:2311.10813_ , 2023.
  * Marcu et al. [2023] Ana-Maria Marcu, Long Chen, Jan Hünermann, Alice Karnsund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, et al.  Lingoqa: Video question answering for autonomous driving.  _arXiv preprint arXiv:2312.14115_ , 2023.
  * Mori et al. [2019] Keisuke Mori, Hiroshi Fukui, Takuya Murase, Tsubasa Hirakawa, Takayoshi Yamashita, and Hironobu Fujiyoshi.  Visual explanation by attention branch network for end-to-end learning-based self-driving.  In _2019 IEEE Intelligent Vehicles Symposium (IV)_ , pages 1577–1582, 2019.  doi: 10.1109/IVS.2019.8813900.
  * Nie et al. [2023] Ming Nie, Renyuan Peng, Chunwei Wang, Xinyue Cai, Jianhua Han, Hang Xu, and Li Zhang.  Reason2drive: Towards interpretable and chain-based reasoning for autonomous driving.  _arXiv preprint arXiv:2312.03661_ , 2023.
  * Omeiza et al. [2022] Daniel Omeiza, Helena Webb, Marina Jirotka, and Lars Kunze.  Explanations in autonomous driving: A survey.  _IEEE Transactions on Intelligent Transportation Systems_ , 23(8):10142–10162, 2022.  doi: 10.1109/TITS.2021.3122865.
  * Padalkar et al. [2023] Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et al.  Open x-embodiment: Robotic learning datasets and rt-x models.  _arXiv preprint arXiv:2310.08864_ , 2023.
  * Papineni et al. [2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318, 2002.
  * Peng et al. [2023] Zhenghao Peng, Wenjie Mo, Chenda Duan, Quanyi Li, and Bolei Zhou.  Learning from active human involvement through proxy value propagation.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * Pomerleau [1988] Dean A Pomerleau.  Alvinn: An autonomous land vehicle in a neural network.  _Advances in neural information processing systems_ , 1, 1988.
  * Rana et al. [2023] Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, and Niko Suenderhauf.  Sayplan: Grounding large language models using 3d scene graphs for scalable task planning.  _arXiv preprint arXiv:2307.06135_ , 2023.
  * Renz et al. [2022] Katrin Renz, Kashyap Chitta, Otniel-Bogdan Mercea, A Koepke, Zeynep Akata, and Andreas Geiger.  Plant: Explainable planning transformers via object-level representations.  _arXiv preprint arXiv:2210.14222_ , 2022.
  * Rubin et al. [2021] Ohad Rubin, Jonathan Herzig, and Jonathan Berant.  Learning to retrieve prompts for in-context learning.  _arXiv preprint arXiv:2112.08633_ , 2021.
  * Sadat et al. [2020] Abbas Sadat, Sergio Casas, Mengye Ren, Xinyu Wu, Pranaab Dhawan, and Raquel Urtasun.  Perceive, predict, and plan: Safe motion planning through interpretable semantic representations.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIII 16_ , pages 414–430. Springer, 2020.
  * Schwarting et al. [2018] Wilko Schwarting, Javier Alonso-Mora, and Daniela Rus.  Planning and decision-making for autonomous vehicles.  _Annual Review of Control, Robotics, and Autonomous Systems_ , 2018.
  * Sha et al. [2023] Hao Sha, Yao Mu, Yuxuan Jiang, Li Chen, Chenfeng Xu, Ping Luo, Shengbo Eben Li, Masayoshi Tomizuka, Wei Zhan, and Mingyu Ding.  Languagempc: Large language models as decision makers for autonomous driving.  _arXiv preprint arXiv:2310.03026_ , 2023.
  * Shao et al. [2023] Hao Shao, Letian Wang, Ruobing Chen, Hongsheng Li, and Yu Liu.  Safety-enhanced autonomous driving using interpretable sensor fusion transformer.  In _Conference on Robot Learning_ , pages 726–737. PMLR, 2023.
  * Sharan et al. [2023] SP Sharan, Francesco Pittaluga, Manmohan Chandraker, et al.  Llm-assist: Enhancing closed-loop planning with language-based reasoning.  _arXiv preprint arXiv:2401.00125_ , 2023.
  * Team et al. [2023] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.
  * Toromanoff et al. [2020] Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde.  End-to-end model-free reinforcement learning for urban driving using implicit affordances.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 7153–7162, 2020.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Vedantam et al. [2015] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4566–4575, 2015.
  * Venugopalan et al. [2015] Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko.  Sequence to sequence-video to text.  In _Proceedings of the IEEE international conference on computer vision_ , pages 4534–4542, 2015.
  * Wang et al. [2021] Hengli Wang, Peide Cai, Rui Fan, Yuxiang Sun, and Ming Liu.  End-to-end interactive prediction and planning with optical flow distillation for autonomous driving.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2229–2238, 2021.
  * Wang et al. [2023] Wenhai Wang, Jiangwei Xie, ChuanYang Hu, Haoming Zou, Jianan Fan, Wenwen Tong, Yang Wen, Silei Wu, Hanming Deng, Zhiqi Li, et al.  Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving.  _arXiv preprint arXiv:2312.09245_ , 2023.
  * Xiao et al. [2023] Yi Xiao, Felipe Codevilla, Diego Porres Bustamante, and Antonio M Lopez.  Scaling self-supervised end-to-end driving with multi-view attention learning.  _arXiv preprint arXiv:2302.03198_ , 2023.
  * Xu et al. [2023] Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kenneth KY Wong, Zhenguo Li, and Hengshuang Zhao.  Drivegpt4: Interpretable end-to-end autonomous driving via large language model.  _arXiv preprint arXiv:2310.01412_ , 2023.
  * Yang et al. [2023] Zhenjie Yang, Xiaosong Jia, Hongyang Li, and Junchi Yan.  A survey of large language models for autonomous driving.  _arXiv preprint arXiv:2311.01043_ , 2023.
  * Zablocki et al. [2022] Éloi Zablocki, Hédi Ben-Younes, Patrick Pérez, and Matthieu Cord.  Explainability of deep vision-based autonomous driving systems: Review and challenges.  _International Journal of Computer Vision_ , 130(10):2425–2452, 2022.
  * Zeng et al. [2019] Wenyuan Zeng, Wenjie Luo, Simon Suo, Abbas Sadat, Bin Yang, Sergio Casas, and Raquel Urtasun.  End-to-end interpretable neural motion planner.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8660–8669, 2019.
  * Zhang et al. [2023] Hang Zhang, Xin Li, and Lidong Bing.  Video-llama: An instruction-tuned audio-visual language model for video understanding.  _arXiv preprint arXiv:2306.02858_ , 2023.
  * Zhang et al. [2022] Qihang Zhang, Zhenghao Peng, and Bolei Zhou.  Learning to drive by watching youtube videos: Action-conditioned contrastive policy pretraining.  In _European Conference on Computer Vision_ , pages 111–128. Springer, 2022.
  * Zheng et al. [2023] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _arXiv preprint arXiv:2306.05685_ , 2023.
  * Zhu et al. [2023a] Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al.  Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment.  _arXiv preprint arXiv:2310.01852_ , 2023a.
  * Zhu et al. [2023b] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_ , 2023b.

###  -A Discussion and Limitation

Scale of MLLM While our Multi-modal Large Language Model (MLLM) has exhibited
impressive capabilities in visual reasoning and planning for driving tasks,
it’s worth noting that it comprises only 7 billion parameters. This size is
relatively modest when compared to more well-known models such as GPT4-V [1]
and Gemini [70], which boast a significantly larger parameter count and
demonstrate superior, near-human levels of visual understanding and reasoning.
In various related fields such as visual question answering, problem-solving,
and interactive dialogue, a clear trend has been observed by researchers: the
scale of the model’s parameters and the breadth of training data sources are
pivotal. Performance improvements are typically seen in tandem with model
scaling. Based on this trend, we expect similar advancements in the realm of
driving applications. A larger model could further enhance the capabilities of
MLLM in driving scenarios.

Number of In-Context Learning Examples During training and inference, we
provide 2 ICL examples for each query. This is due to the limitation of the
context window size of the LLM backbone of 2048. With the recent improvement
in LLM context window size, we expect to see a more flexible adoption of ICL
examples.

###  -B Training Details

Embedding Projector We leverage a three-layer MLP as the embedding projector
to fuse the video $1\times 1024$ and control signal $1\times 28$ embedding
into a hybrid embedding. The architecture of the lightweight projector is a
four-layer MLP with GELU activation. It has an input dimension of $1052$ and
an output dimension of $1024$. The margin used in triplet loss is 0.5. We
train the model for 200 epochs with a learning rate of $1e-5$ with Adam
optimiser.

MLLM Backbone We use a learning rate of $2e-5$ with a cosine scheduler. We use
a batch size of 4 with a gradient accumulation step of 2 on 8 A100 GPUs, which
leads to an effective training batch size of 128. We use a warm-up strategy in
the first 5 epochs with a warm-up ratio of 0.03. We train the model for 2
epochs.

###  -C Baseline Details

In our comparison, we evaluate several baseline methods. The first, S2VT [74],
utilises an end-to-end sequence-to-sequence model with Long Short-Term Memory
(LSTM) networks. It is trained on paired video-sentence data, linking video
frame sequences to corresponding word sequences, enabling it to generate
descriptive captions for video events. The second method, WAA [38], employs a
visual attention model that trains a convolutional network from images to
vehicle control commands. This method focuses on identifying influential image
regions through the controller’s attention and uses an attention-based video-
to-text model to produce textual explanations aligned with the controller’s
attention maps, grounding the explanations in relevant scene parts. The third
approach, ADAPT [33], is a transformer-based method that leverages a multi-
task joint training framework. It aligns driving action captioning with
control signal prediction tasks. Finally, DriveGPT4 [78], trained using LLaVA
[48] to generate a visual instruction tuning dataset derived from the BDD-X
dataset [38], processes multimodal input data and is capable of generating
text responses while predicting control signals, fine-tuned with the
assistance of ChatGPT on the new dataset.

###  -D Linear Layer Parameter Update Derivations

Consider a forward-pass through a linear layer $\mathcal{F}$ after its weights
$W_{0}$ have been updated by $\Delta{}W$

$\mathbf{y}=\mathcal{F}(\mathbf{x})=(W_{0}+\Delta{}W)\mathbf{x}=W_{0}\mathbf{x}+\Delta{}W\mathbf{x}$

The weight update itself is expressed as

$\Delta{}W=\eta\frac{\partial{}L}{\partial\mathbf{y}}|_{\mathbf{y}_{i}}\mathbf{x}_{i}^{T}\Rightarrow\Delta{}W\mathbf{x}=\eta\frac{\partial{}L}{\partial\mathbf{y}}|_{\mathbf{y}_{i}}\mathbf{x}_{i}^{T}\mathbf{x}$

where $\mathbf{x}_{i},\mathbf{y}_{i}$ are the input and output to the layer
that resulted in the weight update. Now, if we in fact have optimised over a
mini-batch of input-outputs $\mathbf{x}_{i},\mathbf{y}_{i}$ we have

$\Delta{}W=\sum_{i}\eta\frac{\partial{}L}{\partial\mathbf{y}}|_{\mathbf{y}_{i}}\mathbf{x}_{i}^{T}\Rightarrow\Delta{}W\mathbf{x}=\sum_{i}\eta\frac{\partial{}L}{\partial\mathbf{y}}|_{\mathbf{y}_{i}}\mathbf{x}_{i}^{T}\mathbf{x}$

Therefore we have a weighted sum of dot products, which is akin to an
attention mechanism. Indeed, from Eq. 6 we can apply softmax-free linear
attention expressions such as in [40] for

$\displaystyle
W_{V}\left[\bm{z}_{icl};\bm{z}_{q}\right]\mathrm{softmax}\left(\frac{\left(W_{K}\bm{z}_{1:n}\right)^{T}(W_{Q}\bm{z}_{1:n})}{\sqrt{d^{i}}}\right)$
$\displaystyle\rightarrow
W_{V}\left[\bm{z}_{icl};\bm{z}_{q}\right]\left(W_{K}\left[\bm{z}_{icl};\bm{z}_{q}\right]\right)^{T}W_{Q}\bm{z}_{1:n}$

Multiplying the linear attention matrices through the stacked in-context and
query embeddings we have

$W_{V}\bm{z}_{icl}(W_{K}\bm{z}_{icl})^{T}W_{Q}\bm{z}_{1:n}+W_{V}\bm{z}_{q}(W_{K}\bm{z}_{q})^{T}W_{Q}\bm{z}_{1:n}$

Now take out a common factor of $W_{Q}\bm{z}_{1:n}$ for

$(W_{V}\bm{z}_{icl}(W_{K}\bm{z}_{icl})^{T}+W_{V}\bm{z}_{q}(W_{K}\bm{z}_{q})^{T})W_{Q}\bm{z}_{1:n}$

and put $\Delta{}W_{ICL}=W_{V}\bm{z}_{icl}(W_{K}\bm{z}_{icl})^{T}$ and
$W_{ZSL}=W_{V}\bm{z}_{q}(W_{K}\bm{z}_{q})^{T}$ as the terms both pre-
multiplying $W_{Q}\bm{z}_{1:n}$. Note that $W_{ZSL}$ is independent of the in-
context terms (depending only on the query). Now, in $\Delta{}W_{ICL}$ we in
fact have a set of in-context retrieved samples $\bm{z}_{icl,i}$ such that

$\displaystyle
W_{V}\left[\bm{z}_{icl,0},\bm{z}_{icl,1},\ldots\right](W_{K}\left[\bm{z}_{icl,0},\bm{z}_{icl,1},\ldots\right])^{T}$
$\displaystyle\rightarrow\Delta{}W_{ICL}=\sum_{i}~{}W_{V}\bm{z}_{icl,i}(W_{K}\bm{z}_{icl,i})^{T}$

Finally, by inspection of similar dot-product expressions
$W_{V}\bm{z}_{icl,i}(W_{K}\bm{z}_{icl,i})^{T}\leftrightarrow\eta\frac{\partial{}L}{\partial\mathbf{y}}|_{\mathbf{y}_{i}}\mathbf{x}_{i}^{T}\mathbf{x}$
we note that we match the form for the linear layer above

$(W_{ZSL}+\Delta{}W_{ICL})W_{Q}\bm{z}_{1:n}\leftrightarrow(W+\Delta{}W)\mathbf{x}$

This can therefore be interpreted to say that the output of the attention is
adjusted in a meta-optimal way to conform to the samples provided as input
context, much like gradient descent on the linear layer would adjust that
layer to conform to the mini-batch training data, but crucially in the case of
RAG-Driver, without supervision.