# Object-Guided Day-Night Visual Localization in Urban Scenes

Assia Benbihi  VRG, Faculty of Electrical Engineering
Czech Technical University in Prague  Cédric Pradalier  IRL 2958 GT-CNRS,
Metz, France  Ondřej Chum  VRG, Faculty of Electrical Engineering
Czech Technical University in Prague

###### Abstract

We introduce Object-Guided Localization (OGuL) based on a novel method of
local-feature matching. Direct matching of local features is sensitive to
significant changes in illumination. In contrast, object detection often
survives severe changes in lighting conditions. The proposed method first
detects semantic objects and establishes correspondences of those objects
between images. Object correspondences provide local coarse alignment of the
images in the form of a planar homography. These homographies are consequently
used to guide the matching of local features. Experiments on standard urban
localization datasets (Aachen, Extended-CMU-Season, RobotCar-Season) show that
OGuL significantly improves localization results with as simple local features
as SIFT, and its performance competes with the state-of-the-art CNN-based
methods trained for day-to-night localization.

##  1 Introduction

Visual localization is a key step enabling modern technologies, such as
autonomous driving or augmented reality. Classical approaches to visual
localization exploit the Structure-from-Motion (SfM) pipeline. In this setup,
local features are detected and described by a high-dimensional appearance
descriptor in each image independently. Tentative correspondences of the local
features are established based on the similarity of their descriptors. In the
final step of the SfM pipeline, scene geometry and camera poses are robustly
estimated from the tentative matches.

Figure 1: Matching of local features is guided by the coarse geometry of
matching objects. Left: Bounding boxes of semantic objects detected in the
images. Two sets of corresponding objects (related by a planar homography)
were detected, shown in blue and green. For the objects in red bounding boxes,
no corresponding objects were found in the other image. Right: Each homography
guides the matching of local features from the first image (yellow dot) by
constraining the search space of possible matches in the second image (orange
dots in the blue-circle) - demonstrated on homography relating objects in
blue; and a final set of matches demonstrated on homography relating objects
in green.

This paper targets the task of day-and-night visual localization where severe
changes of illumination break the SfM pipeline in the very beginning - at the
local features detection and matching stages. The negative impact of
illumination changes on local feature matching is twofold. First, local
feature detectors are unlikely to fire at identical locations. Second, the
visual appearance may vary significantly, so that even for features covering
the same surface, the extracted descriptors are dissimilar. There are recent
research directions to alleviate local feature failure by training detectors
and descriptors that are invariant or less sensitive to severe changes in
illumination [12, 13, 32, 49, 36]. However, the detectors and descriptors are
not powerful enough to solve the day-and-night localization problem alone.
Therefore we focus on the matching part of the problem in this paper.

Urban scenes are a distinct type of man-made environment. On one hand, it
introduces challenges that make the matching task harder, such as extreme
illumination variations. The appearance changes are not only caused by
different directions, colors, and/or intensities of the light; there are parts
of the scene that change their appearance completely. This includes elements
that become light sources themselves, for example, street lamps, neon signs,
or see-through windows.

On the other hand, the structure of urban environments exhibits properties
that allow certain geometric reasoning based on mild assumptions. These
assumptions include piece-vice planar structures of the environment, where the
majority of the planes are vertical facades. Many of the man-made structures
on the facade have semantic meaning and it is relatively easy to recognize
them on the category level, which is independent of the view-point or
illumination conditions (e.g. windows). Another assumption is the large number
of facade structures that tend to be aligned with horizontal and vertical
directions. This allows for reliable estimation of horizontal and vertical
vanishing points, which are useful in geometric reasoning.

In this work, we propose the Object-Guided Localization (OGuL) pipeline, that
exploits the urban structures that survive severe changes in illumination. In
particular, semantic objects and edges are used. The correspondences between
semantic objects generate hypotheses of coarse scene alignment, which are used
to guide the matching of local features, see Fig. 1. Edges are used to
estimate vanishing points and to improve the detection of semantic objects.
For semantic objects, we restrict to the most commonly appearing facade
element – windows. However, other types of planar objects can be used in a
straightforwardly.

Figure 2: Left: Illustration of the method. Semantic objects are detected and
their bounding boxes are projectively adjusted using the vanishing points.
Homography hypotheses registering the largest number of object bounding boxes
are estimated. The homographies are used to guide the matching of local
features. Right: Illustration of the box adjustement. An axis-aligned box is
detected in the original image (left) and adjusted to be consistent with the
vanishing points (right). The green / red points are the midpoints of the
horizontal / vertical box edges. The green / red lines join the midpoints of
the box edges with the appropriate vanishing point. Intersections of the red
and green lines (orange dots) define the corners of the projective bounding
box.

OGuL is evaluated on the task of visual localization on three benchmark
datasets, on which it systematically improves day-night localization and
maintains the performance on the day. Our contributions are the following: (i)
We propose a novel feature matching guidance strategy based on semantic object
pairs that are robust to appearance variations. (ii) We augment box detection
to allow for perspective box detection without additional training, using
geometric properties of the image. (iii) Our complete pipeline improves day-
night localization and manages to match pairs that are not handled by State-
of-the-Art (SoA) methods.

##  2 Related Work

This section provides an overview of local feature matching.

Object-Guided Visual Localization.  The most related work to ours is an
object-of-interest-guided visual localization approach for _indoor_
environments [54]. A detector of a fixed set of specific and highly
discriminative planar objects of interest is trained. For a detected object in
a query image, a dense set of point correspondences to its reference image in
the database is established. The final localization is estimated by 2D-to-3D,
exploiting the known 3D position of the object of interest in the environment.
Our method differs in that it detects generic objects and exploits their
coarse correspondences to guide the matching of the surrounding pixels.

Coarse-to-Fine Feature Matching  first aligns the images or their parts
approximately, then the matching is refined to achieve higher accuracy.
Commonly, such a procedure starts at a lower image resolution and proceeds to
a finer resolution [28, 27, 5, 6]. Modern local features based on CNNs
naturally operate at the coarse level as the spatial resolution of the CNN’s
output is lower than the resolution of the input image. Various methods to
obtain finer matches were proposed.

One of such methods is learning-free interpolation, introduced in [12].
Matches obtained at higher levels of the CNN (lower resolution) are refined at
lower levels of the CNN in [55, 48]. In [11], receptive fields of
corresponding deep features are used to guide the matching of standard local
features. Recently, Patch2Pix [60] learns to regress a single pixel
correspondence, while Loftr [46] learns a dense matching of the pixels between
corresponding receptive fields. The coarse matching in these approaches
originates from matching at a lower resolution. In our case, the coarse
matching is a consequence of the imprecise localization of the detected
objects.

Guided Matching.  There are two approaches to expanding a restricted set of
reliable correspondences with either global or local geometric guidance. In
the case of global geometry, the reliable correspondences are used to robustly
estimate global geometry and then restrict the search for further matches to
be consistent with it [20]. Various geometric models can be used, e.g.
epipolar geometry [44], or local optical flow [31, 16]. A recent example of
such an approach is Ransac-Flow [45], where a homography is estimated from the
deep features, used to warp one image to another and coarsely align them
before learning a dense matching. Another approach starts from a local
geometry of the features and gradually extends the neighbourhood [14].

All the guided matching methods rely on the existence of a sufficient number
of correct matches of local features to instantiate the initial geometry. In
contrast, the proposed approach instantiates the initial geometry based on
semantic objects, which are more reliably detected under severe photometric
transformations.

Matching by Global Optimization.  Another type of guidance relies on a voting
scheme that draws matches that agree the most with each other. This requires
the definition of similarity between two matches that usually embeds their
geometric agreement in the form of translation [24], similarity [34, 51],
affine transform [8, 9], or optical flow [33]. Matches are then drawn from
clustering or graph optimization algorithms. This is the case for the SoA
SuperGlue [39] where the similarity is learned using an attention mechanism.

Match Filtering  is a post-processing step that attempts to filter out the
outliers from a given set of matches. Deep learning methods attempt to
classify matches as inliers or outliers [57, 58, 4, 47] while hand-crafted
methods rely on match similarity [10], smoothness constraints [25, 3], or
local affine transform [7]. Any of these methods can be applied as a
postprocessing step to the proposed method.

Semantic Object Alignment.  This line of work [37, 38, 22] focuses on matching
different object instances of the same category, usually only one object per
image, rather than the same instance under various conditions as in our case.
The goal is to establish dense pixel correspondences between the objects for
applications such as style transfer rather than visual localization. Most
related to our work is the use of object region proposals to guide dense
feature matching [17, 18, 19, 56]. However, these proposals represent only a
fragment of the single object to match and are associated through visual
similarity and simple spatial consistency. Instead, OGuL detects multiple
whole objects per image and relies on geometry to match them. This geometric
guidance, in the form of a homography, is more informative than the usual
translation and scale parameters used in these works.

##  3 Object-Guided Feature Matching

We propose to generate coarse geometric guidance of local features based on
the correspondences of semantic objects (Fig. 2). First, objects are detected
and represented by a perspective bounding box (Sec. 3.1). Object
correspondences are established based on their geometric layout in the images
(Sec. 3.2). A pair of objects defines a local geometry in the form of a
homography derived from the corresponding box corners. This geometric model is
used to guide the neighboring local feature matching (Sec. 3.3). The main
advantage of the proposed method is the ability to match local features even
in situations where establishing the feature correspondences based on
appearance only is not possible.

###  3.1 Perspective Object Detection

The first step of the proposed pipeline is the detection of semantic objects.
Object detectors commonly output an axis-aligned bounding box, ignoring the
perspective distortion. Since the detected objects are used to estimate a
homography transformation, more precise localization is needed. Two
complementary methods exploiting estimated vanishing points are proposed. In
the first one (Sec. 3.1.1), boxes are detected and their edges are adjusted so
that they coincide with the vanishing directions of the plane the object lies
on (Fig. 2-right). The second approach (Sec. 3.1.2) rectifies the image before
the boxes are detected so that the planar objects are orthogonal in the
rectified image. The axis-aligned bounding boxes are then projected back to
the original image.

Although more complex, the rectification-based approach allows for the
detection of objects with strong perspective distortion in the original image.
However, this comes with the risk of potential spurious detections when the
rectification creates visual artifacts. Experiments show that each method
separately achieves comparable localization performance, and the fusion of the
two reaches the best results.

Both methods rely on vanishing points derived from line segments. The points
are estimated using sequential RANSAC [15]: a sample of two line segments
defines a vanishing point. The inliers are segments which angle to the
estimated vanishing direction is below a threshold.

####  3.1.1 Vanishing Point Aligned Boxes

When the object detection is performed on the original image, the resulting
boundary boxes are axis-aligned and they need to be adjusted to obtain
perspective boxes. Since the detection is limited to planar objects, there is
always a plane on which the object lies. The orthogonal box is updated so that
its sides lay on a line incident to the vanishing point of that plane. To
associate the box with these vanishing directions, the following voting scheme
is used: the line segments around the box each vote for the vanishing
direction they support. The two orthogonal vanishing directions with the
maximum votes are used to adjust the box. The adjustment is illustrated in
Fig. 2-right: for each box edge, a line between the midpoint and the vanishing
point with the same orientation is derived. The new box corners are the four
intersections of these lines.

####  3.1.2 Image Rectification

The rectifying homography transformation is a composition of two
transformations, projective and affine. The projective transformation sends
vanishing points to infinity in horizontal and vertical directions
respectively. The affine transformation is restricted to anisotropic scaling
and translation so that it leaves horizontal and vertical directions
unchanged. The affine transformation minimizes the sum of squared differences
between the coordinates of line segment endpoints between the rectified and
original images. This choice reduces the amount of re-sampling when the
rectified image is rendered.

In the presence of multiple facades, which is when multiple horizontal
vanishing points are detected, the image is segmented into planes
corresponding to those horizontal vanishing points. We treat the problem as a
1D classification over the image columns where each column is assigned to one
of the horizontal vanishing points. Each horizontal line segment votes for the
horizontal vanishing point it supports. The vote is counted over all the
columns the line segment intersects. The value of the vote is the confidence
of the line segment to be associated with the supported vanishing point. This
confidence is derived from the softmax over the geometric residuals of this
line segment with all the vanishing points. Once the histogram of votes is
produced for all horizontal vanishing points, the columns are classified using
confident majority votes. Given a column, the majority vote of it is confident
when it exceeds a certain threshold and, at the same time, the second best
vote is far behind (first to second best ratio). This produces intervals
associated with a vanishing point. Unlabelled column intervals are split by a
single threshold between the neighboring labels. The threshold is selected by
a maximum likelihood principle.

Each plane is rectified separately before it is processed by the object
detector. The outputted boxes are back-projected to the original image using
the inverse rectification transform.

###  3.2 Object Matching

In the object matching step, a set of homographies maximizing the number of
corresponding box detections between the two images is found. The following
greedy approach is adopted. For each object in the first image, the $K$ most
similar object boxes in the second image are selected. The similarity is
measured by the similarity of deep features computed over the boxes. A
homography hypothesis that maps the object in the first image to the box in
the second image is constructed for all $K$ boxes in the second image. For
each hypothesis, all boxes from one image are projected to the second image.
If the area Intersection-over-Union (IoU) of a projected box and an object box
detected in a second image is greater than a threshold
$\varepsilon_{\lx@glossaries@gls@link{acronym}{iou}{\leavevmode IoU}}=0.5$,
the pair is considered corresponding. No visual similarity is enforced on
corresponding boxes. A homography with the highest number of supporting boxes
is stored and the supporting boxes are excluded. The procedure is repeated
until there is no homography supported by at least two pairs of corresponding
boxes.

###  3.3 Geometry-Guided Feature Matching

The object matching step outputs a set of homographies and pairs of object
boxes consistent with each homography. These boxes provide information about
the spatial support of the homography: a feature in this support complies with
this transformation and is guided with it. A feature in the first image is
projected to the second one and matched to the keypoint with the most similar
visual descriptor that falls within a radius $r_{search}$ from the projection.

###  3.4 Additional Feature Matches

To cover areas with no objects detected or areas off the planes with objects,
additional feature matches are added. This step performs standard feature
matching based on the similarity of the feature descriptors. Local features
already matched in the previous step are not considered in this step.

Method | Aachen v1.0 | Aachen v1.1 | RobotCar Seasons
---|---|---|---
|  | Left | Rear | Right
Night | Night | Day | Night | Day | Night | Day | Night
UprightRootSIFT + NN | 57.1 / 69.4 / 77.6 | 52.9 / 65.4 / 74.9 | 51.3 / 76.8 / 90.7 | 4.7 / 9.6 / 16.3 | 57.4 / 81.8 / 96.2 | 14.1 / 22.8 / 31.3 | 51.2 / 76.5 / 90.0 | 9.5 / 15.4 / 20.2
UprightRootSIFT+OGuL | 74.5 / 84.7 / 98.0 | 63.4 / 80.6 / 94.2 | 51.5 / 76.6 / 90.5 | 5.0 / 10.8 / 17.8 | 57.4 / 81.9 / 96.3 | 16.4 / 27.6 / 36.7 | 51.3 / 76.6 / 90.0 | 10.3 / 19.0 / 24.7
SuperPoint + NN† | 73.5 / 79.6 / 88.8 | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
SuperPoint + NN | 75.5 / 82.7 / 91.8 | 69.1 / 84.8 / 94.8 | 51.5 / 78.1 / 92.7 | 5.9 / 12.6 / 21.1 | 56.7 / 81.8 / 96.2 | 20.4 / 42.1 / 63.2 | 52.4 / 79.1 / 93.3 | 12.0 / 24.5 / 37.7
SuperPoint + OGuL | 77.6 / 84.7 / 95.9 | 69.1 / 85.3 / 96.9 | 51.3 / 78.0 / 92.6 | 5.6 / 13.3 / 23.2 | 56.6 / 81.8 / 96.6 | 21.2 / 42.9 / 64.4 | 52.6 / 79.0 / 93.2 | 13.0 / 24.6 / 39.2
D2-Net + NN† | 74.5 / 86.7 / 100.0 | $-$ | $-$ | $-$ | 54.5 / 80.0 / 95.3 | 20.4 / 40.1 / 55.0 | $-$ | $-$
D2-Net + NN | 79.6 / 89.8 / 100.0 | 68.1 / 84.8 / 96.9 | 52.9 / 80.5 / 95.2 | 15.7 / 37.4 / 56.0 | 54.8 / 81.1 / 96.3 | 32.7 / 66.3 / 89.9 | 53.5 / 80.2 / 95.3 | 19.9 / 42.7 / 64.5
D2-Net + OGuL | 79.6 / 89.8 / 100.0 | 67.5 / 84.8 / 97.9 | 53.3 / 80.2 / 95.0 | 16.5 / 36.2 / 56.0 | 54.8 / 81.1 / 96.3 | 33.9 / 65.0 / 88.8 | 53.3 / 80.1 / 95.1 | 21.0 / 43.2 / 63.4

Table 1: Comparison with NN matching on SoA local features. Evaluation metric
is the percentage of images whose registration error in translation and
rotation falls below (0.5m, 2∘) / (1m, 5∘) / (5m, 10∘) respectively. † denotes
results published in the original papers, other results are obtained by our
execution.

##  4 Implementation Details

Object Detection Training.  Objects are detected using the Faster R-CNN
network [35] with a Resnet-50 [21] backbone with Tensorflow [1]. The network
is trained in three steps adopted from the authors’ guidelines [35]. At test
time, objects are detected at multiple scales of the image ($\times$1 and
$\times$2). For training, we use the box annotations of the window instances
in the Open Images Dataset V4 (OID) [23]. This dataset holds a significant
amount of noisy labels that hinder the precision and the generalization of the
network. So images with such labels are discarded (e.g. incomplete labelling,
occluded windows). This process keeps only 10% of the images to which are
added images from the CMP facades dataset [52]. The final dataset holds 9278
images and 117632 boxes.

Object Pair Homography.  A pair of corresponding objects provides four point-
to-point correspondences, which is enough to instantiate a homography. The
homography used for guided matching of the local features is estimated from
all box correspondences that support the initial homography. A vanishing point
consistency is enforced in this step.

Line Segment Detection.  Line segments are detected with the augmented HT-LCNN
[26] and segments smaller than 20 pixels are discarded.

##  5 Evaluation

This section provides an evaluation of OGuL by comparison to baseline methods,
an ablation study, and a comparison to different SoA approaches.

###  5.1 Experimental Setup

OGuL is evaluated against other feature matching methods: the default Nearest-
Neighbor (NN) approach, the coarse-to-fine Patch2Pix [60] and LOFTR [46], the
graph-based approach SuperGlue [39], and the filtering method AdaLAM [7]. The
evaluation measures the pose estimation performance for outdoor localization
on three benchmark datasets: Aachen v1.0, Aachen V1.1, RobotCar Seasons.

Visual Localization.  We follow the standard evaluation setup from the
localization benchmark 111https://www.visuallocalization.net/benchmark/. The
released code takes feature matches as input and runs 3D structure-based
localization relying on the COLMAP library [42, 43]. The localization is
evaluated with the percentage of estimated poses within an error threshold
with respect to the groundtruth. The same rotation and translation thresholds
as in the benchmark are used here: (0.25m, 2∘) / (0.5m, 5∘) / (5m, 10∘).

Datasets.  Experiments are run on three standard urban localization datasets:
the Aachen datasets (v1.0 and v1.1) [40, 41, 59] and the RobotCarSeasons one
[30, 40]. The last one presents additional challenges because of the lower
image quality and the presence of artefacts such as motion blur, raindrops,
and overexposure. For Aachen, the list of image pairs to match is provided by
the benchmark. For the other dataset, no list is provided so we use the
ground-truth positions to derive reference image pairs and use the global
image descriptor denseVLAD [50] to match query images to the 20 most similar
reference ones, as in [13].

Figure 3: Qualitative results. Top-Down: Box matches, OGuL guided matches, NN
matches. Left-Right: SIFT on Aachen v1.0, SIFT on RobotCarSeasons, D2-Net on
Aachen v1.0. The examples show that the box detection and matching are robust
to strong illumination variations and occlusions. This allows the guided
matching of features in situations where NN fails.

###  5.2 Comparison to baseline methods

In Tab. 1, we compare the method against NN matching for three local features:
the hand-crafted upright-root-SIFT [2, 29] and two SoA deep features
SuperPoint [12] and D2-Net [13]. The SIFT features are provided with the
datasets and the deep features are extracted with the author’s code with the
parameters reported in the benchmark. Except for the input matches, the
localization pipeline stays fixed.

On the Aachen datasets, the OGuL matches consistently improve the performance
over the baseline matches for SIFT by up to 17%. Qualitative results using
SIFT features are shown in Fig. 3. The improvement deteriorates with more
advanced features such as with D2-Net and SuperPoint. Even though the
quantitative results are on par, the set of resolved images is different, see
Fig. 3-right, which shows the potential of the proposed method.

On the RobotCar Seasons, OGuL increases the recall for night queries while
preserving the scores on the day queries. It also allows registering more
night queries, between 50 and 100. This shows that OGuL is a relevant method
for feature matching across day-night images. The bottleneck for the RobotCar
improvement is unreliable box detection. This happens when the image does not
exhibit windows, when the car goes along a road, or when the image is too
deteriorated (blur, overexposure).

###  5.3 Ablation study

We assess the advantage of object-guided feature matching and how each element
of the method contributes to the performances.

| without AF | with AF
---|---|---
O | 44.9 / 60.2 / 76.5 | 71.4 / 82.7 / 94.9
OA | 42.9 / 60.2 / 78.6 | 70.4 / 84.7 / 95.9
R | 41.8 / 59.2 / 76.5 | 72.4 / 82.7 / 93.9
RA | 45.9 / 63.3 / 77.6 | 70.4 / 82.7 / 93.9
O + R | 60.2 / 73.5 / 88.8 | 71.4 / 86.7 / 98.0
(O + R)$\cdot$A | 59.2 / 76.5 / 88.8 | 73.5 / 85.7 / 96.9
Table 2: Comparison of the perspective box detection on Aachen v1.0. O:
Orthogonal boxes. OA: Orthogonal boxes Adjusted. R: orthogonal boxes on
Rectified images. RA: boxes on Rectified images Adjusted. With / without
additional features (AF). Blue: Best without AF. Green: Best with AF.

##### Perspective Box Detection (Tab. 2)

We evaluate the influence of the box derivation on the localization
performance using the SIFT features on Aachen v1.0. Results show that both
derivations achieve comparable results on their own. When no additional
features (Sec. 3.4) are used, merging and adjusting the boxes on the original
and rectified images significantly improves the scores. One explanation is
that these two derivations detect more boxes together.

##### Plane segmentation.

Perspective box detection on the rectified images uses plane segmentation to
rectify only the part of the image relevant to the vanishing point. This task
has been previously addressed in [53], introducing a cost associated with
plane transitions (smoothness cost). In our experiments, segmentation of [53]
produces, for the best value of the cost mixing parameter, 7% worse results
than our method on Aachen v1.0. We conclude, that for challenging images, it
is difficult to set the relative weight of the smoothness cost.

##### Constrained Box Matching (Tab. 3)

The coarse geometry derived from matching objects leads to better results when
constrained by the vanishing directions (+15-20%). This is emphasised when no
additional features are used. Using the vanishing directions to adjust the
boxes also improves the results. This suggests that integrating the vanishing
points in the method is relevant to compensate for the inaccurate box corners.

| without AF | with AF
---|---|---
C + A | 59.2 / 76.5 / 88.8 | 73.5 / 85.7 / 96.9
C + $\bar{\rm A}$ | 60.2 / 73.5 / 88.8 | 71.4 / 86.7 / 98.0
$\overline{\rm C}$ \+ A | 45.9 / 62.2 / 77.6 | 69.4 / 84.7 / 95.9
$\overline{\rm C}$ \+ $\overline{\rm A}$ | 40.8 / 55.1 / 72.4 | 68.4 / 84.7 / 94.9
Table 3: Influence of the Constrained box matching with box Adjustement (A)
and without. Blue: Best without AF. Green: Best with AF.

##### Box Description (Tab. 4)

The aggregated D2-Net descriptors achieve better results than the detector’s
description. One explanation may be that the detector’s features are optimized
for object classification which may not be suited for matching. Note that when
additional features are used, the method is not as sensitive to the box
description as without.

| without AF | with AF
---|---|---
D2-Net + GeM. | 55.1 / 72.4 / 85.7 | 73.5 / 83.7 / 95.9
GeM. | 42.9 / 57.1 / 68.4 | 70.4 / 84.7 / 94.9
Avg_pool | 38.8 / 53.1 / 69.4 | 67.3 / 79.6 / 93.9
Table 4: Comparison of the box description used to prune candidate box matches
on Aachen v1.0. Blue: Best without AF. Green: Best with AF.

##### Parameter Sensitivity.

Additional experiments show that OGuL is not very sensitive to the feature
search radius, the number of candidate match for one box. The method seems to
be robust to noisy box detection as it performs similarly even with low
confidence detections. One reason for this may be the robust estimation of the
box matching.

###  5.4 Comparison with the SoA

Results in Tab. 5 show that our method is competitive with SoA matching
approaches. One advantage of our method (and Patch2Pix) is that it is agnostic
to the local features whereas other existing approaches, such as LOFTR or
SuperGlue, train the guided matching for a specific feature. Although the
numerical performances are comparable, the successful matching pairs of OGuL
and other methods are complementary (Fig. 3-right). This shows that object-
guided matching is a relevant method to push the current limits of feature
matching. Failure cases of the proposed method are shown in Fig. 4: obvious
limitations of the method include images with no objects to detect, incorrect
box detection, and noise in the object matching.

| Aachen v1.0 | Aachen v1.1
---|---|---
SIFT + OGuL | 74.5 / 84.7 / 98.0 | 63.4 / 80.6 / 94.2
SuperPoint + OGuL | 77.6 / 84.7 / 95.9 | 69.1 / 85.3 / 96.9
D2Net + OGuL | 79.6 / 89.8 / 100.0 | 67.5 / 84.8 / 97.9
SIFT + AdaLAM [7] | 69.4 / 81.6 / 92.9 | $-$
Patch2Pix† [60] | 78.6 / 88.8 / 99.0 | $-$
LOFTR† [46] | $-$ | 72.8 / 88.5 / 99.0
SuperPoint + SuperGlue† [39] | 79.6 / 90.8 / 100.0 | 73.3 / 88.0 / 98.4

Table 5: Localization Performance on Aachen. † means we report the results
from the paper. Figure 4: Failure cases for object detection (top) and box
matching (bottom).

##  6 Conclusion

Object-Guided Localization (OGuL) for urban scenes was introduced. The
approach is based on a novel method of local-feature matching. The proposed
method overcomes significant changes in illumination by first detecting
semantic objects and establishing their correspondence between images. Object
correspondences provide hypotheses of planar homography that are used to guide
the matching of local features.

We have experimentally shown the potential of the method. Significant
improvements were achieved with basic SIFT features. With the SoA D2Net
features, the method is on par or slightly better than the standard approach,
resolving and failing in different cases, demonstrating the complementarity of
the two approaches.

##  7 Acknowledgements

The authors thank Tomáš Jeníček and Alan Lukežič for their thoughtful
comments.

## References

  * [1] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al.  Tensorflow: a system for large-scale machine learning.  In OSDI, volume 16, pages 265–283, 2016.
  * [2] Relja Arandjelović and Andrew Zisserman.  Three things everyone should know to improve object retrieval.  In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 2911–2918. IEEE, 2012.
  * [3] JiaWang Bian, Wen-Yan Lin, Yasuyuki Matsushita, Sai-Kit Yeung, Tan-Dat Nguyen, and Ming-Ming Cheng.  Gms: Grid-based motion statistics for fast, ultra-robust feature correspondence.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4181–4190, 2017.
  * [4] Eric Brachmann and Carsten Rother.  Neural-guided ransac: Learning where to sample model hypotheses.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4322–4331, 2019.
  * [5] Thomas Brox, Andrés Bruhn, Nils Papenberg, and Joachim Weickert.  High accuracy optical flow estimation based on a theory for warping.  In European conference on computer vision, pages 25–36. Springer, 2004.
  * [6] Andrés Bruhn, Joachim Weickert, and Christoph Schnörr.  Lucas/kanade meets horn/schunck: Combining local and global optic flow methods.  International journal of computer vision, 61(3):211–231, 2005.
  * [7] Luca Cavalli, Viktor Larsson, Martin Ralf Oswald, Torsten Sattler, and Marc Pollefeys.  Handcrafted outlier detection revisited.  In European Conference on Computer Vision, 2020.
  * [8] Hsin-Yi Chen, Yen-Yu Lin, and Bing-Yu Chen.  Robust feature matching with alternate hough and inverted hough transforms.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2762–2769, 2013.
  * [9] Hsin-Yi Chen, Yen-Yu Lin, and Bing-Yu Chen.  Co-segmentation guided hough transform for robust feature matching.  IEEE transactions on pattern analysis and machine intelligence, 37(12):2388–2401, 2015.
  * [10] Minsu Cho, Jungmin Lee, and Kyoung Mu Lee.  Feature correspondence and deformable object matching via agglomerative correspondence clustering.  In 2009 IEEE 12th International Conference on Computer Vision, pages 1280–1287. IEEE, 2009.
  * [11] François Darmon, Mathieu Aubry, and Pascal Monasse.  Learning to guide local feature matches.  In 3DV, 2020.
  * [12] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superpoint: Self-supervised interest point detection and description.  In CVPR Deep Learning for Visual SLAM Workshop, 2018.
  * [13] Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler.  D2-net: A trainable cnn for joint detection and description of local features.  In CVPR 2019-IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  * [14] Vittorio Ferrari, Tinne Tuytelaars, and Luc Van Gool.  Simultaneous object recognition and segmentation by image exploration.  In European Conference on Computer Vision, pages 40–54. Springer, 2004.
  * [15] Martin A Fischler and Robert C Bolles.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.  Communications of the ACM, 24(6):381–395, 1981.
  * [16] Andreas Geiger, Julius Ziegler, and Christoph Stiller.  Stereoscan: Dense 3d reconstruction in real-time.  In 2011 IEEE intelligent vehicles symposium (IV), pages 963–968. Ieee, 2011.
  * [17] Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce.  Proposal flow.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3475–3484, 2016.
  * [18] Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce.  Proposal flow: Semantic correspondences from object proposals.  IEEE transactions on pattern analysis and machine intelligence, 40(7):1711–1725, 2017.
  * [19] Kai Han, Rafael S Rezende, Bumsub Ham, Kwan-Yee K Wong, Minsu Cho, Cordelia Schmid, and Jean Ponce.  Scnet: Learning semantic correspondence.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1831–1840, 2017.
  * [20] R. I. Hartley and A. Zisserman.  Multiple View Geometry in Computer Vision.  Cambridge University Press, ISBN: 0521540518, second edition, 2004.
  * [21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [22] Seungryong Kim, Dongbo Min, Bumsub Ham, Sangryul Jeon, Stephen Lin, and Kwanghoon Sohn.  Fcss: Fully convolutional self-similarity for dense semantic correspondence.  In Proc. IEEE Conf. Comp. Vision Patt. Recog, volume 1, page 8, 2017\.
  * [23] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al.  The open images dataset v4.  International Journal of Computer Vision, pages 1–26, 2020.
  * [24] Marius Leordeanu and Martial Hebert.  A spectral technique for correspondence problems using pairwise constraints.  In Computer Vision, IEEE International Conference on, volume 2, pages 1482–1489. IEEE Computer Society, 2005.
  * [25] Wen-Yan Daniel Lin, Ming-Ming Cheng, Jiangbo Lu, Hongsheng Yang, Minh N Do, and Philip Torr.  Bilateral functions for global motion modeling.  In European Conference on Computer Vision, pages 341–356. Springer, 2014.
  * [26] Yancong Lin, Silvia L Pintea, and Jan C van Gemert.  Deep hough-transform line priors.  In European Conference on Computer Vision, pages 323–340. Springer, 2020.
  * [27] Ce Liu, Jenny Yuen, and Antonio Torralba.  Sift flow: Dense correspondence across scenes and its applications.  IEEE transactions on pattern analysis and machine intelligence, 33(5):978–994, 2010.
  * [28] Ce Liu, Jenny Yuen, Antonio Torralba, Josef Sivic, and William T Freeman.  Sift flow: Dense correspondence across different scenes.  In European conference on computer vision, pages 28–42. Springer, 2008.
  * [29] David G Lowe.  Distinctive image features from scale-invariant keypoints.  International journal of computer vision, 60(2):91–110, 2004.
  * [30] Will Maddern, Geoffrey Pascoe, Chris Linegar, and Paul Newman.  1 year, 1000 km: The oxford robotcar dataset.  The International Journal of Robotics Research, 36(1):3–15, 2017\.
  * [31] Josef Maier, Martin Humenberger, Markus Murschitz, Oliver Zendel, and Markus Vincze.  Guided matching based on statistical optical flow for fast and robust correspondence analysis.  In European Conference on Computer Vision, pages 101–117. Springer, 2016.
  * [32] A Mishchuk, D Mishkin, F Radenović, and J Matas.  Working hard to know your neighbor’s margins: Local descriptor learning loss.  In Advances in Neural Information Processing Systems, pages 4827–4838, 2017.
  * [33] Kai Ni, Hailin Jin, and Frank Dellaert.  Groupsac: Efficient consensus in the presence of groupings.  In 2009 IEEE 12th International Conference on Computer Vision, pages 2193–2200. IEEE, 2009.
  * [34] Gustavo A Puerto-Souza and Gian Luca Mariottini.  Hierarchical multi-affine (hma) algorithm for fast and accurate feature matching in minimally-invasive surgical images.  In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2007–2012. IEEE, 2012.
  * [35] Shaoqing Ren, Kaiming He, Ross B Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In NIPS, 2015.
  * [36] Jerome Revaud, Philippe Weinzaepfel, César Roberto de Souza, and Martin Humenberger.  R2D2: repeatable and reliable detector and descriptor.  In NeurIPS, 2019.
  * [37] Ignacio Rocco, Relja Arandjelovic, and Josef Sivic.  Convolutional neural network architecture for geometric matching.  In Proc. CVPR, volume 2, 2017.
  * [38] Ignacio Rocco, Relja Arandjelovic, and Josef Sivic.  End-to-end weakly-supervised semantic alignment.  In Proc. CVPR, 2018.
  * [39] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superglue: Learning feature matching with graph neural networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4938–4947, 2020.
  * [40] Torsten Sattler, Will Maddern, Carl Toft, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Marc Pollefeys, Josef Sivic, et al.  Benchmarking 6dof outdoor visual localization in changing conditions.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8601–8610, 2018.
  * [41] Torsten Sattler, Tobias Weyand, Bastian Leibe, and Leif Kobbelt.  Image retrieval for image-based localization revisited.  In BMVC, volume 1, page 4, 2012.
  * [42] Johannes L Schonberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4104–4113, 2016.
  * [43] Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm.  Pixelwise view selection for unstructured multi-view stereo.  In European Conference on Computer Vision (ECCV), 2016.
  * [44] Rajvi Shah, Vanshika Srivastava, and PJ Narayanan.  Geometry-aware feature matching for structure from motion applications.  In 2015 IEEE Winter Conference on Applications of Computer Vision, pages 278–285. IEEE, 2015.
  * [45] Xi Shen, François Darmon, Alexei A Efros, and Mathieu Aubry.  Ransac-flow: generic two-stage image alignment.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16, pages 618–637. Springer, 2020.
  * [46] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou.  Loftr: Detector-free local feature matching with transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8922–8931, 2021.
  * [47] Weiwei Sun, Wei Jiang, Eduard Trulls, Andrea Tagliasacchi, and Kwang Moo Yi.  Acne: Attentive context normalization for robust permutation-equivariant learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11286–11295, 2020.
  * [48] Hajime Taira, Masatoshi Okutomi, Torsten Sattler, Mircea Cimpoi, Marc Pollefeys, Josef Sivic, Tomas Pajdla, and Akihiko Torii.  Inloc: Indoor visual localization with dense matching and view synthesis.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7199–7209, 2018.
  * [49] Yurun Tian, Xin Yu, Bin Fan, Fuchao Wu, Huub Heijnen, and Vassileios Balntas.  Sosnet: Second order similarity regularization for local descriptor learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11016–11025, 2019.
  * [50] Akihiko Torii, Relja Arandjelovic, Josef Sivic, Masatoshi Okutomi, and Tomas Pajdla.  24/7 place recognition by view synthesis.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1808–1817, 2015.
  * [51] Lorenzo Torresani, Vladimir Kolmogorov, and Carsten Rother.  Feature correspondence via graph matching: Models and global optimization.  In European conference on computer vision, pages 596–609. Springer, 2008.
  * [52] Radim Tyleček and Radim Šára.  Spatial pattern templates for recognition of objects with regular structure.  In Proc. GCPR, Saarbrucken, Germany, 2013.
  * [53] Guowei Wan and Sikun Li.  Automatic facades segmentation using detected lines and vanishing points.  In 2011 4th International Congress on Image and Signal Processing, volume 3, pages 1214–1217. IEEE, 2011.
  * [54] Philippe Weinzaepfel, Gabriela Csurka, Yohann Cabon, and Martin Humenberger.  Visual localization by learning objects-of-interest dense match regression.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5634–5643, 2019.
  * [55] Aji Resindra Widya, Akihiko Torii, and Masatoshi Okutomi.  Structure from motion using dense cnn features with keypoint relocalization.  IPSJ Transactions on Computer Vision and Applications, 10(1):1–7, 2018.
  * [56] Fan Yang, Xin Li, Hong Cheng, Jianping Li, and Leiting Chen.  Object-aware dense semantic correspondence.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2777–2785, 2017.
  * [57] Kwang Moo Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, and Pascal Fua.  Learning to find good correspondences.  In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), number CONF, 2018.
  * [58] Jiahui Zhang, Dawei Sun, Zixin Luo, Anbang Yao, Lei Zhou, Tianwei Shen, Yurong Chen, Long Quan, and Hongen Liao.  Learning two-view correspondences and geometry using order-aware network.  In Proceedings of the IEEE International Conference on Computer Vision, pages 5845–5854, 2019.
  * [59] Zichao Zhang, Torsten Sattler, and Davide Scaramuzza.  Reference pose generation for long-term visual localization via learned features and view synthesis.  International Journal of Computer Vision, 129(4):821–844, 2021\.
  * [60] Qunjie Zhou, Torsten Sattler, and Laura Leal-Taixe.  Patch2pix: Epipolar-guided pixel-level correspondences.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4669–4678, 2021.