#  Tell Me Where You Are:
Multimodal LLMs Meet Place Recognition
https://ai4ce.github.io/LLM4VPR/

Zonglin Lyu, Juexiao Zhang, Mingxuan Lu, Yiming Li, Chen Feng
New York University
{zl3958, jz4725, ml8465, yimingli<EMAIL_ADDRESS>

###### Abstract

Large language models (LLMs) exhibit a variety of promising capabilities in
robotics, including long-horizon planning and commonsense reasoning. However,
their performance in place recognition is still underexplored. In this work,
we introduce multimodal LLMs (MLLMs) to visual place recognition (VPR), where
a robot must localize itself using visual observations. Our key design is to
use vision-based retrieval to propose several candidates and then leverage
language-based reasoning to carefully inspect each candidate for a final
decision. Specifically, we leverage the robust visual features produced by
off-the-shelf vision foundation models (VFMs) to obtain several candidate
locations. We then prompt an MLLM to describe the differences between the
current observation and each candidate in a pairwise manner, and reason about
the best candidate based on these descriptions. Our results on three datasets
demonstrate that integrating the general-purpose visual features from VFMs
with the reasoning capabilities of MLLMs already provides an effective place
recognition solution, without any VPR-specific supervised training. We believe
our work can inspire new possibilities for applying and designing foundation
models, i.e., VFMs, LLMs, and MLLMs, to enhance the localization and
navigation of mobile robots.

> Keywords: Multimodal LLMs, Vision Foundation Models, Place Recognition

Figure 1: Vision and language meet place recognition. Alice gives Bob verbal
descriptions of her surroundings (a). Bob compares his visual observations
with Alice’s descriptions (b)-(d) and reasons about their accuracy, confirming
(d) as the correct place.

##  1 Introduction

Visual place recognition (VPR), the task of accurately identifying a
previously visited location based on visual input, is a longstanding challenge
in robotics [1]. Prior methods typically approach VPR as a visual
representation learning problem, focusing on improving the robustness of
visual features against irrelevant variations like lighting, weather, and
transient objects. Inspired by the strong reasoning capabilities of large
language models (LLMs), we explore how LLMs can enhance VPR performance.
Intuitively, humans often use verbal descriptions alongside visual input to
recognize their current location. Imagine a Bob and Alice scenario shown in
Fig. 1. Bob is trying to find Alice and needs to verify his location using
Alice’s verbal description: “There is a McDonald’s and a Starbucks on
different sides.” Alice provides no navigation commands, only descriptions of
her surroundings for Bob to recognize the correct place, aligning with the VPR
task definition. Bob compares his visual observations with Alice’s cues and
reasons about the correctness of the location: mismatched store positions
indicate he is in the wrong place, while matching observations confirm he is
in the right place. It seems intuitive to integrate language into VPR based on
the aforementioned example. Yet it remains unexplored how and to what extent
LLMs can improve place recognition.

Figure 2: Comparison between traditional VPR methods and our LLM-VPR. We build
a VPR solution based on off-the-shelf foundation models and do not need VPR-
specific supervised training. Meanwhile, we leverage language-based reasoning
to further refine the localization precision.

Vision and language exhibit distinct properties. Visual observations offer
abundant spatial details, including geometric and photometric cues. In
contrast, language provides abstract information that facilitates spatial
reasoning, such as identifying landmarks and understanding contextual
relationships within an environment. To leverage the strengths of both
modalities, we propose a vision-to-language (coarse-to-fine) framework, in
which vision-based retrieval generates several candidate locations, followed
by language-based reasoning for finer selection. More specifically, we utilize
an off-the-shelf vision foundation model (VFM), DINOv2 [2], to extract robust
visual features from RGB input. These robust features enable coarse retrieval,
proposing several candidate images. Subsequently, we employ an off-the-shelf
multimodal LLM, GPT-4V [3], to perform a finer selection from these
candidates. Nevertheless, fully harnessing the potential of multimodal LLMs
without task-specific training to find the best candidate remains a challenge.

When asked to match two visual observations, humans compare both images by
looking back and forth: they identify potential matching landmarks, examine
each one closely, and seek contextual cues to draw conclusions [4]. This hints
at a comparing-then-reasoning design, i.e., we leverage multimodal LLMs to
describe the delta of each image pair and then use all the textual
descriptions for the final reasoning stage to determine the best candidate.
Moreover, without task-specific fine-tuning, it is challenging for multimodal
LLMs to identify VPR-relevant details, such as buildings, and VPR-irrelevant
details, such as lighting, which can be easily done with human knowledge.
Hence, we inject human knowledge into multimodal LLMs via VPR-specific
prompts. Consequently, we build a VPR system with two off-the-shelf foundation
models, without any additional supervised training used in previous VPR
methods [5, 6, 7, 8, 9], as shown in Fig. 2.

Statement of contributions. We propose LLM-VPR, a framework integrating
language and vision for robotic place recognition. We make the following
contributions: (1) We demonstrate that integrating the general-purpose visual
features from VFMs with the reasoning capabilities of MLLMs already provides
an effective VPR solution, without any VPR-specific supervised training. In
other words, we leverage two foundation models to achieve zero-shot place
recognition. (2) We propose a vision-to-language (coarse-to-fine) framework to
fully exploit the advantages of two modalities in VPR. (3) We propose a
comparing-then-reasoning framework to facilitate spatial reasoning with MLLMs,
achieving fine-grained place recognition. (4) We evaluate LLM-VPR in three
datasets. Quantitative and qualitative results indicate that our method
outperforms vision-only solutions and performs comparably to supervised
methods without training overhead.

##  2 Related Works

Visual Place Recognition (VPR). Most VPR methods consider only the vision
modality and the task is formulated as image retrieval. Traditionally, VPR
involves aggregating hand-crafted features [10, 11] into a global descriptor
[12, 13]. Recent learning-based methods take advantage of deep features [14,
15]. For retrieval methods, learning-based VPR techniques fall into two folds:
global retrieval and local reranking. Global retrieval methods [5, 8, 7, 6]
aggregate latent features extracted by Deep Neural Networks into a global
descriptor. Reranking methods [16, 17, 18] coarsely retrieve the top-K images
with global retrievals and conduct a detailed comparison of local patch
features, sacrificing efficiency to obtain better quality. Some works enhance
VPR with different inputs such as video VPR [19, 20], collaborative VPR [21],
and lidar-based VPR [22, 23, 24]. Recently, language models have been
introduced to conventionally pure vision tasks. However, to the best of our
knowledge, it remains unexplored how and to what extent language can benefit
VPR.

Vision Foundation Models in VPR. VFMs refer to models trained with a wide
range of images and their learned features demonstrate strong generalization
capability. In terms of training methods, VFMs fall into two folds: supervised
and unsupervised. Supervised vision foundation models [15, 14, 25, 26] are
usually trained in a large-scale dataset with labels such as ImageNet1K [27].
Such models can be transferred to different datasets/tasks by changing their
output heads. Most VPR methods [5, 8, 17, 18] initialize model weights
pretrained in such a way, which requires additional fine-tuning. Due to high
annotation costs in supervised methods, self-supervised techniques are
proposed so that models can learn the pattern of images without any
annotations, and thus models can potentially learn from an incredible amount
of unlabelled data [2, 28, 29, 30, 31, 32]. DINO [32, 2] can produce robust
and informative features in VPR task even without fine-tuning [33, 9],
achieving training-free general-purpose solution for VPR. Our method leverages
DINO as a vision-based coarse retriever and further improves its performance
with the help of human language.

Languages in VPR. There are a few works at the intersection of languages and
VPR. TextPlace [34] highlights the importance of text appearing in scenes for
the VPR task, but it specifically requires text to be present in the scene.
FM-Loc [35] employs multimodal LLMs in specific cases of place recognition,
requiring the scene to be an indoor environment that can be divided into
different categories. The solutions presented in these methods are limited to
specific VPR scenarios, whereas our method offers a general-purpose framework
applicable to a wide range of scenarios.

Multimodal LLMs in Robotics. Introducing LLMs to robotics is recently emerged
as an active research area [36, 37, 38, 39, 40, 41]. There are several lines
of works exploring the advantages of language in other relevant vision-based
robotics tasks, such as autonomous driving [42, 43, 44], navigation [45, 46,
47] and robot-human interactions [48]. They generally ask for high-level
actions to control the vehicles or robots which take advantage of MLLMs
strength in semantic understanding and reasoning. In contrast, VPR often
requires highly detailed low-level descriptions of images, which is reportedly
nontrivial for MLLMs [49]. However, the study of multimodal LLMs in visual
place recognition is still underexplored. In this work, we aim to bridge this
gap, potentially leading to more robust and versatile applications of
multimodal LLMs in autonomous robots, including but not limited to topological
localization and spatial reasoning.

##  3 Preliminaries

VPR as Image Retrieval. Given a query image $I_{q}$, the VPR task requires
finding the images capturing the same location as $I_{q}$ from a database of
previously visited images $\\{I_{r}\\}_{r=1}^{N}$. This is generally
considered as retrieving the correct images from the candidate database and
hence is evaluated with recall rates. Specifically, suppose a feature
extractor $F$ maps images to global descriptors, which are considered as the
image descriptors. Then the similarity scores of a query image $I_{q}$ and a
reference image $I_{r}$ are given by the cosine similarities between their
corresponding global descriptors. The top-K candidates are references with the
highest K similarity scores. The R@K metric evaluates the percentage of
correctly retrieved images in the top-K selected candidates.

Feature Aggregation. Modern deep learning models usually do not directly map
images to global descriptors, but rather to convolutional feature maps or
sequences of patch features from ViT. Therefore, various aggregation methods
have been proposed to obtain informative image descriptors for VPR. Notably,
Generalized Mean Pooling (GeM) [50] is a training-free aggregator that
achieves lightweight yet strongest performance in VPR. Our method uses GeM to
aggregate features obtained from pretrained DINOv2. It is computed as:
$X_{GeM}=\frac{1}{N}\left(\sum_{i}^{N}X_{i}^{p}\right)^{\frac{1}{p}}$, where
$X_{i}$ represents the output feature of a patch of an image and $N$
represents the number of patches.

Figure 3: Overview of LLM-VPR. (a) Vision-based Coarse Retriever. (b) Vision-
Language Refiner. We first coarsely retrieve top-10 candidates via [CLS] token
or GeM aggregated descriptor of DINOv2 [2] features. Then we construct ten
query-candidate pairs and feed them one by one to the Vision-Language Refiner
to describe and reason.

##  4 Methodology

###  4.1 Design Rationale of LLM-VPR

Our method introduces MLLMs to VPR based on the following insights:

  * •

Vision and language exhibit distinct yet complementing attributes. Vision
provides abundant details such as geometric and photometric cues, while
language generates conceptual information and relations that aid spatial
reasoning.

  * •

Vision foundation models offer high-quality visual features that allow coarse
retrieval to efficiently filter out apparently irrelevant candidates.

  * •

Multimodal LLMs can extract detailed descriptions of the delta between pairs
of images via comparison and enable more sophisticated reasoning of their
geographical relationships.

Therefore, we propose a vision-to-language (coarse-to-fine) pipeline
consisting of a vision-based coarse retriever and a vision-language refiner.
The retriever selects the top-K candidates for a given query image, and the
refiner generates a text description for each query-candidate pair and then
evaluates all text descriptions to determine the similarity ranks between the
query and each candidate. The entire pipeline is shown in Fig. 3.

###  4.2 Vision-based Coarse Retriever

The vision-based coarse retriever maps images to global descriptors and
selects the top-K candidates based on the cosine similarity between the query
and each candidate. DINOv2 [2] is employed as the vision feature extractor
because its training is self-supervised by a large amount of task-agnostic
vision data and exhibits strong VPR performance in previous work [33].
Besides, DINOv2 is trained with both the global image-level loss and the local
token-level losses, which produce more robust visual features for the VPR
task. There are two approaches to retrieving the top-K candidates: directly
using the [CLS] token or applying GeM pooling to local patch tokens. We
experiment with both options. The VLAD aggregator [13, 12] is not included
because it requires unsupervised training with VPR-specific datasets whereas
we attempt to make our method training-free.

Figure 4: Example of how our method works. The query is selected from
Pittsburgh30K [51], where candidate 1 is the top-1 retrieval of DINOv2 + GeM,
and Candidate 2 is the Top-1 retrieval of our method. Correct retrieval is
with green border, while the incorrect one is with red border.

###  4.3 Vision-Language Refiner

After retrieving the top-K candidates, a vision-language refiner generates
verbal descriptions for the delta of each query-candidate pair and reasons to
finalize the ranking.

Choice of MLLMs. The main challenge in leveraging verbal descriptions is to
generate detailed descriptions relevant to VPR and filter out details
irrelevant to VPR, such as weather and lighting. Unaware of the task-specific
relevance of the visual details, the MLLMs need to be instructed on how to
filter out irrelevant details and pay attention to the relevant ones via
careful prompting. Meanwhile, detailed text descriptions of all retrieved
candidates form a long context, and generating the final ranking based on them
requires the MLLM to have sufficiently good long-text understanding
capabilities. In light of those, we choose GPT4-V [3] for the vision-language
refiner as it fulfills the above considerations. We expect similar performance
from other comparable models.

Design of Prompts. As for the prompting, it is hard to verbally describe a
scene such that humans can imagine all the details. However, telling the delta
between images is achievable and informative. So each query-candidate pair is
sent to GPT-4V with a prompt asking for the similarities and differences
between them, similar to the “Bob and Alice” scenario shown in Fig. 1.
Additionally, we find it is important to clearly state what details should not
be considered in the prompt and prevent the model from inferring imagined
objects due to model hallucination. Moreover, it is informative to instruct
GPT4-V that it needs to recognize the texts in the scene, just as Bob can
recognize McDonald’s because of the texts on its logos in the “Bob and Alice”
example.

Example Illustration. One example from Pittsburgh30K is provided in Fig. 4 as
an illustration of our method, where the shown prompt is a simplified version
due to space limit. Complete prompts are included in our Appendix. To ensure
that the final ranking of retrieved images aligns with human consensus, we
emphasize again in the reranking prompt that we are doing the VPR task and
what is important in this task, to filter out accidentally generated VPR-
irrelevant details. By generating descriptions of delta between image pairs
and ranking similarity based on them, our method simulates the comparing-then-
reasoning behavior of humans.

Table 1: Evaluation Results. $R^{2}$Former indicates the reranking is not
performed, where ∗ and $\dagger$ indicate that Top 10 and Top 100 candidates
are reranked respectively. Ours CLS and Ours GeM indicate that our method is
applied on candidates retrieved with DINO CLS token and GeM pooled DINO
features respectively. The best performances are in bold and the second best
is underlined.

Category | Method | Dataset
---|---|---
Tokyo247 | Baidu Mall | Pittsburgh30K
R @ 1 | R @ 5 | R @ 1 | R @ 5 | R @ 1 | R @ 5
Supervised |  $R^{2}$Former | 65.1 | 79.4 | 53 | 71 | 73.8 | 90.4
$R^{2}$Former∗ | 80.3 | 85.1 | 67.2 | 75.5 | 85.9 | 92.8
$R^{2}$Former$\dagger$ | 88.6 | 91.4 | 75.2 | 84.8 | 87.6 | 94.9
MixVPR | 76.8 | 86.7 | 64.0 | 79.8 | 91.5 | 96.7
Training Free | DINO CLS | 61.3 | 81.9 | 41.5 | 51.2 | 81.7 | 94.9
DINO GeM | 81.9 | 93.7 | 51.5 | 67 | 86.4 | 96.4
LLM-VPR | Ours CLS | 73.0 (11.7$\uparrow$) | 84.1 (2.2$\uparrow$) | 51.5 (10.0$\uparrow$) | 61.8 (10.6$\uparrow$) | 83.0 (1.3$\uparrow$) | 95.3 (0.4$\uparrow$)
Ours GeM |  87.0 (5.1$\uparrow$) |  94.3 (0.6$\uparrow$) | 62.0 (10.5$\uparrow$) | 71.5 (4.5$\uparrow$) | 87.0 (0.6$\uparrow$) |  97.1 (0.7$\uparrow$)
|  |  |  |  |  |  |

##  5 Experiments

Datasets. Due to our budget constraints and the rate limit imposed in GPT-4
usage, we evaluate our method on Tokyo247 [52], Baidu Mall [53], and
Pittsburgh30K [51]. Tokyo247 and Baidu Mall contain non-English signage that
is challenging for multimodal LLMs trained mainly with English data, while
Pittsburg30K contains repetitive structures where the difference is hard to
describe. Tokyo247 contains street view data collected in Tokyo, including
both daytime and nighttime. There are 315 queries and around 76K references in
this dataset. Baidu Mall is an indoor dataset collected at a shopping mall in
China. There are 2293 queries and 689 references in this dataset, and we
randomly subsample 400 queries for evaluation. Pittsburgh30K test split
contains 6816 queries and 10000 references collected from driving vehicles in
downtown Pittsburgh. We randomly subsample 1000 queries from this dataset.
Though we only selected three datasets, they cover the majority of real-world
daily scenarios: indoor, street views, and driving. Random subsampling is
applied to only reduce the costs on GPT-4, where the data distribution is
preserved.

Evaluation Metrics. We use Recall at K as our evaluation metric. If one of the
top-K candidates is correct (geographical location is within $m$ meters of the
query), the query is correctly recognized. The Recall at K is the ratio of the
number of correct retrievals and the total number of queries. The threshold m
is set to 25 in the Tokyo247 and the Pittsburgh30K datasets and 10 in the
Baidu Mall dataset, following [18] and [33] respectively.

Method Setup. Top-10 coarse retrievals are fed to our vision-language refiner.
The output feature of the [CLS] token from DINOv2 [2] and GeM aggregated
descriptors of all patch tokens, denoted DINO CLS and DINO GeM, are selected
as our coarse-retriever. We apply vision-language refiner to the top-10
candidates of these two methods, denoted Ours CLS and Ours GeM.

Baseline Methods. We include $R^{2}$Former [18] and MixVPR [8] as the
supervised baselines because they achieve the SOTA performance among reranking
methods and global retrieval methods. We implement $R^{2}$Former to rerank
both Top-10 candidates (same as our setup) and Top-100 (original setup)
candidates. The performance retrieved by DINO CLS and DINO GeM are also
reported to evaluate whether the vision-language refiner can improve over the
vision-based coarse retriever.

###  5.1  Quantitative Results

Figure 5: Examples of (a) success cases, (b) fail cases, and (c) ‘cannot help’
cases. Correct retrievals (top 1) are in green border, and incorrect ones are
in red border. The top two rows in (a) and the first row in (b) are selected
from Tokyo247. The last row in (c) is selected from Pittsburgh30K. The other
examples are selected from Baidu Mall. Text boxes are summarized GPT4-V
outputs.

The quantitative result is shown in Tab. 1. Our vision-language refiner
achieves comparable results with supervised baselines and steadily outperforms
vision-only training-free baselines.

LLM-VPR largely improves over the coarse retriever. In Baidu Mall, shopfronts
are relatively easy to recognize since there are usually signages, resulting
in satisfactory performance of the vision-language refiner. In Tokyo247, we
can usually find unique landmarks to describe verbally to help localize the
query. However, in Pittsburgh30K, there exists a large number of repetitive
and similar structures, which makes it hard to verbally describe the
similarities and differences. Even in this case, our vision-language refiner
is still capable of improving VPR performance.

LLM-VPR is comparable with supervised methods. In Tokyo247 and Pittsburgh30K,
the performance of our method is comparable to the best supervised baselines
in R@1 and better than them in R@5. In Baidu Mall, our method is comparable
with MixVPR and $R^{2}$Former that reranks top-10 candidates. In daily
scenarios, the observed scenes can usually be described via language, where
the semantic and informative information assists place recognition.

LLM-VPR does not achieve expected results in shopping malls. LLM-VPR is
outperformed by the $R^{2}$Former that reranks top-100 candidates with a big
margin in Baidu Mall, but LLM-VPR is comparable with $R^{2}$Former in the
other two datasets. Intuitively, the shopfronts are easy to describe and
recognize, and we should expect a stronger performance of LLM-VPR. However,
shopping malls can also introduce challenges. There might be stores with the
same brand at different locations, which confuses the vision-language refiner.
The repetitive structures in shopping malls, such as decorations, also
introduce challenges when shopfronts do not clearly appear in the image.
Moreover, multimodal LLMs are usually insensitive to the displacement of a
camera when the camera looks at the same objects (details will be discussed in
section 5.2). Only candidates within 10 meters of the query will be considered
correct in Baidu Mall, and this will amplify the negative impact of
insensitivity to camera displacement.

###  5.2 Qualitative Analysis

When do multimodal LLMs help? It is interesting to investigate when and why
multimodal LLMs help the VPR task. Some examples are shown in Fig. 5. In
general, multimodal LLMs can help when there is (1) enough structural
information that can be described, (2) unique landmarks, and (3) when MLLMs
can avoid the perturbation from VPR-irrelevant items. The camera viewpoint in
the first row and bottom row in Fig. 5 (a) contains enough structural
information, so the structural similarities and differences such as textures,
colors, or arrangements can be verbally described. If there are unique
landmarks such as Fig. 4 and the last two rows in Fig. 5 (a): building or
signage, multimodal LLMs are able to recognize the place, even when the
viewpoint is limited (the second query in Fig. 5 (a)). English texts can help
MLLMs as they can directly compare the contents, but if the texts are not in
English, MLLMs tend to compare the shapes and colors. Due to the self-
supervised nature of DINOv2 [2], the descriptor of an image may be negatively
impacted by VPR-irrelevant items close to the camera. For example, in the top
row in Fig. 5 (a), there are street lamps close to the camera, covering part
of the background. DINOv2 tends to pay attention to the street lamps, which is
less important than other backgrounds in the VPR task. MLLMs can overcome such
a problem by providing text descriptions and spatial reasoning.

When do multimodal LLMs harm? It is rare that MLLMs harm the performance, but
there are some reasons why multimodal LLMs may harm place recognition: (1)
there are a large number of items and (2) MLLMs are insensitive to the
displacement of the camera when looking at the same item. When images contain
a large number of items, such as the first row in Fig. 5 (b), it is too hard
to similarities and differences for all items. Occasionally, MLLMs fail in
some “simple” examples because it is insensitive to the displacement of the
camera when it looks at the same item. In the second row of Fig. 5 (b), both
images look at the same store (in Chinese texts), but we can see that their
camera positions are relatively far in an indoor environment. When converting
these two images into texts, such subtle changes in camera positions are hard
to describe, and therefore MLLMs are not sensitive to the change in distance.
As a result, MLLMs may harm VPR.

When do multimodal LLMs fail to help? When the query-candidate pair contains
repetitive and highly similar structures, then MLLMs may not be able to tell
the differences, such as the first row in Fig. 5 (c). The shopping mall
contains repetitive structures like walls and evaluators. Without a hint of
nearby stores, it is hard to tell the differences between the two images. In
addition, if the viewpoint is highly limited without any cues such as signage,
similar to the last row in Fig. 5 (c), MLLMs struggle due to the limited
descriptive information available.

In addition to these qualitative analyses, we include qualitative ablation
studies in the Appendix.

##  6 Conclusion

We explore the underexamined area of LLMs in visual place recognition (VPR)
inspired by human activities shown in the Bob and Alice example. By
integrating vision-based retrieval to propose several candidate locations and
leveraging language-based reasoning to inspect and decide on the best
candidate, we develop LLM-VPR, a robust VPR solution without any additional
supervised training. Our method achieves comparable results with current
supervised SOTA models, and language-based reasoning is shown to be able to
successfully improve performance over vision-only baselines. We believe our
work opens new possibilities for applying and designing foundation models,
such as VFMs, LLMs, and MLLMs, to enhance the localization and navigation of
mobile robots, thereby paving the way for more advanced and versatile robotic
systems.

#### Acknowledgments

This work is supported by NSF Grant 2238968 and 2345139, and in part through
the NYU IT High-Performance Computing resources, services, and staff
expertise. We thank Gao Zhu for the valuable discussions at the beginning of
this project.

## References

  * Lowry et al. [2015] S. Lowry, N. Sünderhauf, P. Newman, J. J. Leonard, D. Cox, P. Corke, and M. J. Milford.  Visual place recognition: A survey.  _ieee transactions on robotics_ , 32(1):1–19, 2015.
  * Oquab et al. [2023] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, R. Howes, P.-Y. Huang, H. Xu, V. Sharma, S.-W. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski.  Dinov2: Learning robust visual features without supervision, 2023.
  * Achiam et al. [2023] Achiam et al.  Gpt-4 technical report.  2023\.
  * Chun [2000] M. M. Chun.  Contextual cueing of visual attention.  _Trends in cognitive sciences_ , 4(5):170–178, 2000.
  * Arandjelović et al. [2015] R. Arandjelović, P. Gronát, A. Torii, T. Pajdla, and J. Sivic.  Netvlad: Cnn architecture for weakly supervised place recognition.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015.
  * Kim et al. [2017] H. J. Kim, E. Dunn, and J.-M. Frahm.  Learned contextual feature reweighting for image geo-localization.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Zhang et al. [2021] J. Zhang, Y. Cao, and Q. Wu.  Vector of locally and adaptively aggregated descriptors for image feature representation.  _Pattern Recognit._ , 2021.
  * Ali-bey et al. [2023] A. Ali-bey, B. Chaib-draa, and P. Giguère.  Mixvpr: Feature mixing for visual place recognition.  _2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , 2023.
  * Izquierdo and Civera [2023] S. Izquierdo and J. Civera.  Optimal transport aggregation for visual place recognition.  _arXiv preprint arXiv:2311.15937_ , 2023.
  * Hunt [2009] C. Hunt.  Surf: Speeded-up robust features.  2009\.
  * Lowe [1999] D. G. Lowe.  Object recognition from local scale-invariant features.  _Proceedings of the Seventh IEEE International Conference on Computer Vision_ , 1999.
  * Jégou et al. [2010] H. Jégou, M. Douze, C. Schmid, and P. Pérez.  Aggregating local descriptors into a compact image representation.  _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , 2010.
  * Arandjelović and Zisserman [2013] R. Arandjelović and A. Zisserman.  All about vlad.  _2013 IEEE Conference on Computer Vision and Pattern Recognition_ , 2013.
  * He et al. [2015] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015.
  * Dosovitskiy et al. [2021] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ , 2021.
  * Hausler et al. [2021] S. Hausler, S. Garg, M. Xu, M. Milford, and T. Fischer.  Patch-netvlad: Multi-scale fusion of locally-global descriptors for place recognition.  _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * Wang et al. [2022] R. Wang, Y. Shen, W. Zuo, S. Zhou, and N. Zheng.  Transvpr: Transformer-based place recognition with multi-level attention aggregation.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Zhu et al. [2023] S. Zhu, L. Yang, C. Chen, M. Shah, X. Shen, and H. Wang.  ${R^{2}}$ Former: Unified retrieval and reranking transformer for place recognition.  _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Garg and Milford [2021] S. Garg and M. Milford.  Seqnet: Learning descriptors for sequence-based hierarchical place recognition.  _IEEE Robotics and Automation Letters_ , 6:4305–4312, 2021.
  * Garg et al. [2021] S. Garg, M. B. Vankadari, and M. Milford.  Seqmatchnet: Contrastive learning with sequence matching for place recognition & relocalization.  In _Conference on Robot Learning_ , 2021.
  * Li et al. [2023] Y. Li, Z. Lyu, M. Lu, C. Chen, M. Milford, and C. Feng.  Collaborative visual place recognition.  _ArXiv_ , 2023.
  * Uy and Lee [2018] M. A. Uy and G. H. Lee.  Pointnetvlad: Deep point cloud based retrieval for large-scale place recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4470–4479, 2018.
  * Xia et al. [2021] Y. Xia, Y. Xu, S. Li, R. Wang, J. Du, D. Cremers, and U. Stilla.  Soe-net: A self-attention and orientation encoding network for point cloud based place recognition.  In _Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition_ , pages 11348–11357, 2021.
  * Hou et al. [2022] Z. Hou, Y. Yan, C. Xu, and H. Kong.  Hitpr: Hierarchical transformer for place recognition in point cloud.  In _2022 International Conference on Robotics and Automation (ICRA)_ , pages 2612–2618. IEEE, 2022.
  * Liu et al. [2022] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie.  A convnet for the 2020s.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022.
  * Radford et al. [2021] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , 2021.
  * Deng et al. [2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , 2009.
  * He et al. [2020] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020.
  * Chen et al. [2020] X. Chen, H. Fan, R. Girshick, and K. He.  Improved baselines with momentum contrastive learning.  _arXiv preprint arXiv:2003.04297_ , 2020.
  * Caron et al. [2020] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  _Advances in neural information processing systems_ , 2020.
  * Chen et al. [2020] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , 2020.
  * Caron et al. [2021] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021.
  * Keetha et al. [2023] N. Keetha, A. Mishra, J. Karhade, K. M. Jatavallabhula, S. Scherer, M. Krishna, and S. Garg.  Anyloc: Towards universal visual place recognition.  _arXiv_ , 2023.
  * Hong et al. [2019] Z. Hong, Y. Petillot, D. Lane, Y. Miao, and S. Wang.  Textplace: Visual place recognition and topological localization through reading scene texts.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2861–2870, 2019.
  * Mirjalili et al. [2023] R. Mirjalili, M. Krawez, and W. Burgard.  Fm-loc: Using foundation models for improved vision-based localization.  _2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2023.
  * Jin et al. [2024] Y. Jin, D. Li, Y. A, J. Shi, P. Hao, F. Sun, J. Zhang, and B. Fang.  Robotgpt: Robot manipulation learning from chatgpt.  _IEEE Robotics and Automation Letters_ , 9(3):2543–2550, 2024.  doi:10.1109/LRA.2024.3357432.
  * Driess et al. [2023] D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence.  Palm-e: An embodied multimodal language model.  In _arXiv preprint arXiv:2303.03378_ , 2023.
  * Ahn et al. [2022] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, M. Yan, and A. Zeng.  Do as i can and not as i say: Grounding language in robotic affordances.  In _arXiv preprint arXiv:2204.01691_ , 2022.
  * Liang et al. [2022] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng.  Code as policies: Language model programs for embodied control.  In _arXiv preprint arXiv:2209.07753_ , 2022.
  * Huang et al. [2023] W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei.  Voxposer: Composable 3d value maps for robotic manipulation with language models.  _arXiv preprint arXiv:2307.05973_ , 2023.
  * Jiang et al. [2023] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan.  Vima: General robot manipulation with multimodal prompts.  In _Fortieth International Conference on Machine Learning_ , 2023.
  * Chen et al. [2023] L. Chen, O. Sinavski, J. Hünermann, A. Karnsund, A. J. Willmott, D. Birch, D. Maund, and J. Shotton.  Driving with llms: Fusing object-level vector modality for explainable autonomous driving, 2023.
  * Jin et al. [2023] B. Jin, X. Liu, Y. Zheng, P. Li, H. Zhao, T. Zhang, Y. Zheng, G. Zhou, and J. Liu.  Adapt: Action-aware driving caption transformer.  _arXiv preprint arXiv:2302.00673_ , 2023.
  * Xu et al. [2023] Z. Xu, Y. Zhang, E. Xie, Z. Zhao, Y. Guo, K. K. Wong, Z. Li, and H. Zhao.  Drivegpt4: Interpretable end-to-end autonomous driving via large language model.  _ArXiv_ , 2023.
  * Krantz et al. [2023] J. Krantz, S. Banerjee, W. Zhu, J. Corso, P. Anderson, S. Lee, and J. Thomason.  Iterative vision-and-language navigation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023.
  * Yang et al. [2023] Z. Yang, A. Majumdar, and S. Lee.  Behavioral analysis of vision-and-language navigation agents.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023.
  * Dai et al. [2024] Y. Dai, R. Peng, S. Li, and J. Chai.  Think, act, and ask: Open-world interactive personalized robot navigation.  _ICRA_ , 2024.
  * Ren et al. [2023] A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu, L. Takayama, F. Xia, J. Varley, Z. Xu, D. Sadigh, A. Zeng, and A. Majumdar.  Robots that ask for help: Uncertainty alignment for large language model planners.  In _Proceedings of the Conference on Robot Learning (CoRL)_ , 2023.
  * Tong et al. [2024] S. Tong, Z. Liu, Y. Zhai, Y. Ma, Y. LeCun, and S. Xie.  Eyes wide shut? exploring the visual shortcomings of multimodal llms.  _arXiv preprint arXiv:2401.06209_ , 2024.
  * Radenović et al. [2018] F. Radenović, G. Tolias, and O. Chum.  Fine-tuning CNN image retrieval with no human annotation.  _TPAMI_ , 2018.
  * Torii et al. [2013] A. Torii, J. Sivic, T. Pajdla, and M. Okutomi.  Visual place recognition with repetitive structures.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2013.
  * Torii et al. [2015] A. Torii, R. Arandjelovic, J. Sivic, M. Okutomi, and T. Pajdla.  24/7 place recognition by view synthesis.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015.
  * Sun et al. [2017] X. Sun, Y. Xie, P. Luo, and L. Wang.  A dataset for benchmarking image-based localization.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017.

[sections] [sections]l1

##  Appendix A Ablation Studies

Our complete text generation prompt includes three main components: hints for
describing similarities and dissimilarities, hints for object matching, and
constraints on irrelevant details and unnecessary inferences. The description
of similarities and dissimilarities corresponds to the action of comparing in
the “Bob and Alice” example, while the other two are additional constraints.
In Fig. 8 and Fig. 9, we use one example from Tokyo247 [52] to showcase the
influence of ablating the other two constraints. The ablated prompts are shown
in Fig. 7.

Hints on object matching. We first discuss the impact of object matching. The
prompt without object matching is in Fig. 7 (a). The generated texts of the
original prompt and prompt without object matching are shown in Fig. 8. With
object matching, the logical flow of the text description becomes clear,
vivid, and graphic. In contrast, when examining texts generated without object
matching, it is more challenging to grasp the main points and form a clear
mental image. Furthermore, with object matching, the similarities and
differences are discussed in greater detail locally rather than broadly.

Constraints on irrelevant details. If we further remove the constraints on
irrelevant details and unnecessary inference (inferring unseen or unclear
items), for which the prompt is shown in Fig. 7 (b), VPR-irrelevant details
such as lighting, vehicles, activities of a street, and skylines are
mentioned, inducing noises during reasoning, as shown in Fig. 9.

##  Appendix B Limitation and Future Research Direction

Figure 6: (a) Text generation prompt and (b) Reasoning prompt. Indoor or
outdoor are selected based on different datasets.  Figure 7: Prompt ablation
study. The content about object matching is removed in (a), and the content
about all constraints is removed in (b). Kept contents are in bold, the
removed object matching contents are struck out, and removed constraints are
doubly struck out. Figure 8: Generated texts (a) with object matching and (b)
without object matching. Green texts show similarities, and red texts show
differences. Violet texts indcate matched objects. Figure 9: Generated texts
without constraints. The image on the left is the query, and the image on the
right is the candidate. Texts that may introduce irrelevant dissimilarities
are in red.

Although this work shows multimodal LLMs can be beneficial in place
recognition and achieve good performance in a training-free pipeline, there
are several limitations.

  1. 1.

The multimodal LLMs may not strictly follow the prompt. As shown in Fig. 8,
GPT still mentions lighting (Dissimilarities 1) while the prompt in Fig. 7 (a)
explicitly prevents it from doing so.

  2. 2.

The multimodal LLMs can generate non-factual descriptions due to their
hallucination problem. In Fig. 10, it is obvious that the candidate does not
contain any text, but when looking at it along with the query, the model
mistakenly concludes that the “DENSO” text also appears in it.

  3. 3.

Long context issue. Since all descriptions are concatenated and fed into the
refiner, it might get too long for efficient and correct place recognition if
more candidates are involved. So the vision-language refiner relies on the
vision-based coarse retriever to provide coarse filtering.

  4. 4.

The current pipeline requires computing and time budget and an internet
connection to the multimodal LLMs APIs, which is not always ideal when
deployment is considered.

Besides addressing the limitations above, future research may consider
automatic prompting for place recognition tasks. While the current prompts are
human-designed, AI-generated prompts could potentially also be beneficial.
Meanwhile, the proposed pipeline is entirely training-free. Though it is
beyond our current budget, it would be interesting to see whether MLLM
finetuned on VPR datasets can exhibit better spatial understanding and
stronger place recognition performance.

Figure 10: Example of the hallucinated descriptions. Non-factual descriptions
due to hallucination are in red. The candidate does not contain the “DENSO”
text though the building looks similar to the query.