# POPCat: Propagation of particles for complex annotation tasks

Adam Srebrnjak Yang  University of Waterloo
Waterloo, Canada
<EMAIL_ADDRESS>Dheeraj Khanna  University of Waterloo
Waterloo, Canada
<EMAIL_ADDRESS>John S. Zelek  University of Waterloo
Waterloo, Canada
<EMAIL_ADDRESS>
###### Abstract

Novel dataset creation for all multi-object tracking, crowd-counting, and
industrial-based videos is arduous and time-consuming when faced with a unique
class that densely populates a video sequence. We propose a time efficient
method called POPCat that exploits the multi-target and temporal features of
video data to produce a semi-supervised pipeline for segmentation or box-based
video annotation. The method retains the accuracy level associated with human
level annotation while generating a large volume of semi-supervised
annotations for greater generalization. The method capitalizes on temporal
features through the use of a particle tracker to expand the domain of human-
provided target points. This is done through the use of a particle tracker to
reassociate the initial points to a set of images that follow the labeled
frame. A YOLO model is then trained with this generated data, and then rapidly
infers on the target video. Evaluations are conducted on GMOT-40, AnimalTrack,
and Visdrone-2019 benchmarks. These multi-target video tracking/detection sets
contain multiple similar-looking targets, camera movements, and other features
that would commonly be seen in ”wild” situations. We specifically choose these
difficult datasets to demonstrate the efficacy of the pipeline and for
comparison purposes. The method applied on GMOT-40, AnimalTrack, and Visdrone
shows a margin of improvement on recall/mAP50/mAP over the best results by a
value of 24.5%/9.6%/4.8%, -/43.1%/27.8%, and 7.5%/9.4%/7.5% where metrics were
collected.

###### Index Terms:

Object Detection; Industrial Vision; Dataset Annotation; Semi-supervised;
Annotation Pipeline; Scalable Vision

Figure 1: Detection and Segmentation Outputs on GMOT-40[1] and AnimalTrack[2]

##  I Introduction

Most learning-based computer vision tasks and algorithms utilize a labeled
dataset to complete a desired task. As such, we think it prudent to develop a
faster, scalable, and efficient way to annotate datasets. We focus on multi-
target video dataset creation as it forms the basis for the popular tasks of
crowd counting and multi-object detection. Video datasets such as KITTI[3],
MOT[4], and GMOT-40[1] utilize image-based annotation methodologies to create
the respective datasets. These methods individually annotate all frames of the
video sequence with human-derived methods. We improve the process by utilizing
a trained model to semi-automatically annotate all targets within the video.
This allows for a semi-supervised approach for dataset annotation.

We exploit the multi-target and temporal features of videos to produce a semi-
supervised pipeline for segmentation or box-based video labeling. We enhance
the process by utilizing computer vision methods to train a detector for rapid
inferencing. This, therefore, augments our method with high speed labeling
once the algorithm is trained on a sparse set of human annotations.

We separate our pipeline into four stages: (a) initialization, (b)
propagation, (c) segmentation and box fitting, and (d) model training. We
detail the operation of each stage under Section III and depict it in Fig. 2.

The input to the pipeline is an annotated frame with the output being the
detected per-frame labels for the entire video sequence. We state that this
generic, multi-object, video annotating solution allows for the generation of
per-use-case datasets for computer vision algorithms. We designate the
pipeline as ”POPCat” as it utilizes the propagation of particles to address
complex annotation tasks such as multi-target video datasets.

Modern object detectors adopt an out-of-the-box application with minimal
modification required for implementation (YOLO[5], R-CNN[6], and DETR[7]). We
adopt the YOLO detector series in our pipeline as it has long-term support and
is readily used in industrial computer vision. This familiarity makes the
adoption of our pipeline an easy process, with users already familiar with the
main components.

To be direct, the intended users for POPCat are the smaller-scale computer
vision operations with lesser access to annotators, compute power, or cloud
resources. We keep this in mind throughout development which enforces
restrictions on the computational requirements of the system. For example, a
transformer-based architecture may be advantageous for recall and real-time
for propagation, but it may be too computationally expensive for the target
audience. Most industrial setups are limited to low power compute units which
impose a VRAM limitation on the system. A balance has to be struck between
computational requirements and pipeline performance in order to reach a
greater target audience. We artificially set the graphics requirement for our
system at 12GB of VRAM to broaden the potential user base.

The pipeline excels in situations where multiple targets are present on a
single frame as it can scale up the total labels by the number of propagated
frames. For example, a 40-point target on the initial frame can lead to 1200
labeled instances via label propagation over 30 frames without the use of the
integrated detector. Examples of the segmentation and detection outputs are
shown in Fig. 1.

The contributions of our paper can be separated into four distinct sections:

  1. 1.

The development of a semi-supervised annotation pipeline based on off-the-
shelf components.

  2. 2.

The use of Segment Anything (SAM)[8] for automatic bounding box resizing.

  3. 3.

The training and application of detection modules for rapid labeling of
contextually similar video frames.

  4. 4.

Benchmarking of our detections against ground truths provided in the
GMOT-40[1] dataset, AnimalTrack dataset[2], and VisDrone-2019[9] Video
detection dataset.

##  II Related Work

###  II-A Industrial Applications

Our team reviews numerous industrial implementations of computer vision
systems to identify the common characteristics of industrial settings and
systems. A common focus is kept on the efficiency of computer vision systems
as human-hours are held at a higher value than computational loading[10, 11,
12, 13, 14, 15, 16]. We divide these works into manually annotated training
sets [13, 16, 15, 12, 14, 17], and semi-automated annotation [10, 11].

Multiple authors note that manual annotations may not effectively cover all
situations in production environments [16, 13, 14, 15] and additional training
data may be required. These systems represent the brute force method for
industrial vision systems where the end user is required to manually annotate
a training set. Such datasets may fail to properly generalize when the
background or context of the targets changes. This may require a new dataset
to be generated to retrain the algorithm if the operational environment
changed[16, 13].

The semi-automated training methods utilize color space thresholds [10], and
motion capture [11] to recognize industrial targets. The color space threshold
method is used to identify cardboard boxes from an overhead view[10]. This
method benefits from good contrast between target and background, and sparse
number of targets on the screen[10]. We note that threshold-based methods
would struggle within dense multi-target environments as there is no way to
distinguish overlapping parts from one another. Furthermore, it requires good
contrast between the background and target which cannot be guaranteed in
production. The motion capture system of [11] utilizes a series of RGB cameras
to collect images and fuse the data with the motion capture system. The pose
extracted from the motion capture system is overlayed on different camera
views to generate a significant amount of data. Annotation is required on the
motion capture data, and is then propagated to the different camera views. The
system is capable of annotating tens of thousands of frames with an average
annotation time of 0.2 frames per second for a multi-target setup. However,
the complexity of the system is too high for most industrial setups. We
understood that an effective system must be able to annotate a significant
number of frames with minimal input as shown in [10, 11], however, a
production system must do it with minimal installation requirements.

Our review of current industrial implementations highlights three facts about
datasets. Namely, datasets are an integral part of training the vision system,
must be related to the detection targets, and are expensive to create.

###  II-B Zero-Shot Object Detection

In addition to the methods of industrial implementations, many computer vision
systems rely on zero-shot learning to overcome the requirement for large
custom datasets. These methods have grown rapidly in previous years with the
introduction of large language transformer models such as BERT[18], and
GPT[19]. We limit our discussion here to Z-GMOT[20] as it is relevant to the
later evaluations. Semantic object detection utilize foundational models to
identify targets based on previously seen data[20]. Algorithms such as
Z-GMOT[20] utilize a grounded language inference process (GLIP)[21] to
semantically search for target objects within images. Further textual
descriptors are applied to enhance the accuracy of the detections[22, 20]. Our
team noted that the quality of the output detection may differ depending on
the prompts provided to the semantic detector[20]. Furthermore, novel classes
that are not within the model must be trained or learned by the model[23]. At
this time, semantic models are deemed too complex and potentially
computationally expensive for use within industrial dataset
annotation/creation.

Figure 2: POPCat Pipeline Representation

##  III Methodology

The purpose of this section is to describe each stage of operation of the
pipeline. We cover the target domain, and follow through with the
initialization, point propagation, segmentation and box fitting, model
training, and model application. The pipeline is depicted in Fig. 2

###  III-A Target Domain

The target domain for our system is industrial vision systems as the
environment synergizes with the proposed system. Industrial environments tend
to feature a statically mounted camera or view with a fixed context, that
contains numerous targets moving along a fixed path. This case has been
identified during the literature review, and supports the methodology used in
training. Namely, multiple targets can be annotated on one given frame, and
the path of travel can be learned through the point propagation methodology.
Once learned, a detector can be used to rapidly annotate the remainder of the
video.

###  III-B Initialization

We initialize the POPCat pipeline with a manual frame annotation as seen in
(A) of Fig. 2. The manual annotation is supplied with a sequence of subsequent
frames to create an annotation-sequence pair.

For a 1200-frame video, a single first-frame annotation for a 30-frame portion
of the video would represent an annotation-sequence pair. We reduce the total
number of necessary manual annotations to the total number of annotation-
sequence pairs used. For most video sets, a single annotation-sequence pair is
required.

Frame initiation can take one of two forms: (a) variable box selection and (b)
fixed box selection. Variable box selection requires a fitted bounding box for
each target and is applied when targets take on a variety of sizes over the
provided sequence. These variations may occur due to viewing angle,
foreground-background travel, rapid target movement, and other variables.
Variable box selection requires more annotation time as an individual must
manually mark the extent of target boxes within the first frame image.
Conventional online annotation tools such as CVAT[24] can be used to expedite
the first frame labeling process for variable sized selection. We represent
variable box selection in Algorithm 1 lines 1-2.

The latter labeling method is fixed box selection which is considered the
faster of the two methods. In this method, a user sets a fixed box width and
height, and must then click closest to all of the center points of the targets
to be tracked. This method prioritizes speed as the user only sets the box
sizes once, and then clicks on the center of targets. Based on this setup, a
single annotator can rapidly annotate a given image by selecting the center
point of targets within the image. The fixed box size should fully encompass
the majority of the targets marked for propagation. However, overtly large
boxes are not recommended as overlapped parts may be segmented as one large
part. Therefore, for scenarios where the target takes on a multitude of
shapes, it is recommended that the dimensional average of all boxes be used
for propagation. We represent fixed box selection in Algorithm 1 lines 4-5.

The precision of the labels is important as the point must be present on the
intended targets for tracking. If the points are not on target then the
tracker may track the background movement instead. To alleviate this we
upscale small scale images and allow the user to zoom in on targets to ensure
correct selections. Once labeled, the annotations are passed to the point
propagation stage.

Input: $K,M$: No. of points/bbox in a frame, $M<=K$

$N$: No. of frames propagated

$P_{(j)}$: Set of points in $j^{th}$ frame

$P_{f(j)}$: Updated $P_{(j)}$ points after positional filtering

Output: $Mask_{l}$: Segmentation masks for $N$ frames

$BBox_{l}$: Yolo format compatible BBox annotation

for $N$ frames

/* Annotation initialization */

1 if  _variable BBox_ then

2 $P_{(1)}^{K}=(x_{i},y_{i},w_{i},h_{i})\forall i\in\\{1...K\\}$

3else

4

5 if  _center points_ then

6 $P_{(1)}^{K}=(x_{i},y_{i},W,H)\forall i\in\\{1...K\\}$

7

/* Point tracking and positional filtering */

8 for  _frames j =1 to N-1_ do

9 $P_{(j+1)}^{K}\leftarrow PIPs(P_{(j)}^{K})$

10 $P_{f_{(j+1)}}^{M}\leftarrow PositionalFiltering(P_{(j+1)}^{K})$

/* Segmentation and Bounding Box Resizing */

11

12for  _frames $l$ =1 to N_ do

13 $Mask_{l}\leftarrow SegmentAnything(P_{f_{(j)}}^{M})$

14 $Mask_{l}\leftarrow CannyEdgeDetection(Mask_{l})$

15 $BBox_{l}\leftarrow GetMinBoundingBox(Mask_{l})$

16return $(Mask,BBox)$

Algorithm 1 Label propagation & Segmentation Algorithm (STAGES A-C)

###  III-C Point Propagation

Persistent Independent particles (PIPs)[25] is used for point propagation for
the proposed pipeline [25]. This method effectively tracks the center points
of the marked targets as a particle within the video. We choose not to track
the targets as a bounding box as it would be computationally inefficient and
less robust to changes in background and box sizing. PIPs utilizes a fully
temporal approach to particle tracking which results in highly accurate point
tracks across the propagated frames. The points are tracked in 8 frame
sequences to overcome momentary occlusions in the video [25]. However, due to
a purely temporal focus, false positive tracks can occur when points travel
out of view.

Positional Filtering: Point tracking in PIPs does not define the termination
of any tracked point if the tracked object leaves the field of view. PIPs re-
initialize on the 8th frame, therefore a lost particle would re-initialize to
the last known position of the particle. In the abscence of occlusions, this
manifests as point tracking of the border of the image after a target leaves
the frame of view. Therefore, we apply a positional filter to terminate the
tracks once the target reaches the edge of the field of view. This eliminates
track drift in highly mobile targets.

The input to PIPs are the center point and box dimensions of the labels marked
in the initialization. We propagate the center points through PIPs and keep
the size of the bounding boxes for the tracked point the same to modularize
this stage. We apply the same operations in PIPs regardless of initialization
choice as a way of simplifying the design and configuration of the pipeline
for the end user.

The output of this stage is a set of text files for each frame, where each
file contains the normalized YOLOv8 data representation of the propagated and
filtered targets. These populated text files are fed into the segmentation
stage of the pipeline. We represent the use of PIPs and the positional filter
in Algorithm 1 lines 6-8.

###  III-D Segmentation and Bounding Box Resizing

We apply a segmentation module to enhance the accuracy of box annotations
centered on the propagated points. We utilize Segment anything Model (SAM) as
it is a zero-shot, prompt based segmentation model that is capable of
segmenting most well defined shapes[8]. We utilize the propagated boxes from
PIPs (center point with height and width) as the prompts for automatic
segmentation of targets within each frame. We depict this in Fig. 3. A careful
review of the failure cases of SAM in [26] shows no failure modes that
adversely affect the intended use case. A box is applied around the target to
eliminate much of the over-or-under segmentation issues associated with SAM
[26]. Given that SAM is a foundational model, it requires no further training
to effect accurate segmentation masks.

We utilize SAM to effect a process known as box resizing. Works such as
Grabcut [27] and Autofit [28] demonstrate how precision and recall increased
proportionately to improvements in the bounding box fit. We demonstrate this
process in Fig. 3 where a) shows the originally labeled boxes and c) shows the
ones generated via the segmentation process. To generate these boxes we apply
a canny edge detector on the segmentation mask to extract the contours of the
segmentation. From there, a minimum bounding rectangle is applied to the
contours to effect the minimum box representation of the mask. The contours
are also saved for training of the segmentation detector in subsequent stages.
One should note in Fig. 3 (a) to (c) that the box resizing method can increase
and decrease the size of bounding boxes to capture the full content of the
target.

The application of SAM as shown in Fig 3 is represented in Algorithm 1 lines
9-12. This includes the extraction of segmentation masks and the creation of
fitted bounding boxes. The output of this stage are YOLO formatted text files
that include bounding box dimensions in one set of text file and segmentation
x,y tuples in another set of files.

Figure 3: Application of Segment Anything Model within the POPCat pipeline

###  III-E Training of Yolo Model

The goal of the previous stages was to propagate a first-frame annotation
across a sequence to increase the number of annotated instances in an
automated manner. Our pipeline now diverges from most mainstream methods as we
utilize the generated instances to train a detector or segmentation YOLO-v8
model as depicted in (D) of Fig. 2. The pipeline provides the training data
necessary to effect either a box or segmentation model depending on the
annotation requirements. We note that any detector could be used in place of
the YOLO model as our inputs to this stage is just training data. This
modularity is intentional as it allows for the adaptation of the pipeline for
more specialized use cases. The details of training will be covered under the
implementation section IV-B.

###  III-F Application of Trained Models

By training an object detector based on the instances generated with Algorithm
1, we leverage the speed of the YOLO detection algorithm for high frame
volumes. The numerical advantage is shown as follows. A 10-minute at 30fps
video contains 18000 frames. Assuming a 20 point first frame annotation for
point propagation, Tab. I showcases the breakdown of time taken in seconds to
process each stage of our pipeline. The example demonstrates the effectiveness
of our solution in the areas of human cost (1 labeled frame), and time
efficiency (10 FPS). For comparison purposes, a 1 FPS rate is retrieved from
the Youtube-BB dataset[29], which focuses on the manual annotation of multi-
target videos[29]. This dataset uses a system of expert annotators to quickly
label multi-target videos in the wild. We note that our system has a 10:1
advantage in speed, and only requires the use of a single expert annotator. We
do not evaluate other detectors by training them on our generated annotations
as it is not a comparable training methodology. That is, we would effectively
be comparing a system trained on all frames to one that was trained on a
subset for a challenge.

Stage | Time (sec)
---|---
First frame labeling | 120
PIPs Propagation | 400
SAM Propagation | 300
YOLO Model Training | 360
Yolo-Inference(remaining frames) |  600
Ours Total time to annotate |  1780 $\sim$ 10FPS
Youtube-BB multi-annotator manual method [29] | 1 FPS
TABLE I: Time Breakdown for annotation of a 10 minute video

##  IV Experiments and Results

###  IV-A Datasets

The Visdrone-2019 video detection challenge [9], GMOT-40[1], AnimalTrack[2]
sets are used to benchmark the performance of POPCat. These sets contain high-
quality ground truth bounding box data for video sequences of varying
duration. Set variations in video data included lighting, object sizing,
movement, occlusion, camera position, camera motion, and target volumes.

The Visdrone-2019 challenge [9] test-dev set contains multi-class detection
data, variable-sized classes, and variations in video sizing. Our method
utilizes three annotation-sequence pairs from the start, middle, and end of
each video. This is done to address temporal class imbalances in the video.
Temporal class imbalance refers to the situation where classes at the start of
a sequence are not representative of the end of the sequence. This is simply a
heuristic method to address wide variations in the context of the video. There
is still the possibility that certain classes will be missed through this
method, however, we want to benchmark the effectiveness of the system with
limited data. We train a single detector model for validation on all videos in
the test-dev set to match the challenge protocol. We note that the results
published by the Visdrone team refer to the challenge dataset of the
competition. That dataset differs from the test-dev set which is publicly
available. Subsequent works benchmark results on the test-dev set [30, 31] and
are referenced for comparison.

GMOT-40[1] and AnimalTrack[2] are tracking datasets that benchmark detection
protocols with a variety of trackers. GMOT-40 and AnimalTrack feature numerous
targets per frame in a single-class configuration. We train on each video
separately in our pipeline with a single start frame annotation-sequence pair.
Our training methodology matches the GMOT-40 one-shot protocol to allow for
results comparison. AnimalTrack detectors utilize a subset of dataset for
testing, and our reported results are collected on the same subset. Both
ablation studies are conducted on the full sets of GMOT-40 and AnimalTrack.

###  IV-B Implementation Details

Starting with point propagation, we develop a modified variant of the chain-
demo.py implementation from PIPs [25]. This implementation features a
dictionary which keeps track of the tracking id, box dimensions, and position
of the points being propagated. A positional filtering subroutine is
implemented within this section to eliminate stray points. We set the
resolution at 1280x720 with a 30 frame sequence. This is done to adhere to the
VRAM constraints placed on the system.

Next, SAM is modified to operate in a batched sequence with an image
resolution of 1280x720. The resolution is upscaled to effect accurate
segmentations in operation. We predict in batches of 50 points at a time and
concatenate the results at the end in order to limit the amount of GPU ram
usage. This allows the system to operate on older or more constrained hardware
setups.

Our solution uses a fixed training and validation confidence of 0.2, with an
IOU of 0.5 with agnostic_nms enabled for the yolox.pt weight configuration for
the detector model. Training epochs are fixed at 25 for all YOLO models, with
a batch size of 12, image size of 640, randomized model weight, SGD optimizer,
learning rate of $0.01$, and default parameters as dictated by YOLO for all
processes[5]. Data augmentation methods are disabled during training.
Propagation, segmentation, model training, and inference are performed on a
single NVIDIA Titan X GPU.

With this implementation, the pipeline is able to maintain the low gpu compute
resource constraints while annotating a significant amount of data.

###  IV-C Evaluation Metrics

The results are compared to other methods through mean average precision
(mAP), mean average precision at 50 percent intersection over union (mAP50),
and recall when available. This information details the effectiveness of our
algorithm when compared to the ground truth data. For example, a recall of 75%
is indicative that our system is capable of noting 75% of the total labels
based on the applied pipeline. For data referenced in Table II, recall and
precision was available on a case by case basis. For a more detailed analysis,
we state precision, recall, true positive (TP), false positive (FP), false
negative (FN), mAP50, mAP, and F1 confidence scores for the pipeline. This
should allow future comparison of our solution with one-shot, few-shot, and
semantic-based methods. It is noted that a proper annotation method should
minimize false positives while maximizing true positives to enhance downstream
detector performance[32, 33].

GMOT-40 - Method | Recall% | mAP50% | mAP%
---|---|---|---
GlobalTrack [34]GMOT-40[1] | - | 15.65 | -
Z-GMOT GLIP[20] | - | 66.20 | 36.10
Z-GMOT iGLIP[20] | - | 66.90 | 40.00
Siamese-DETR (COCO[35])[36] | 49.90 | 63.60 | -
Siamese-DETR (Objects365[37])[36] | 55.40 | 69.60 | -
Fixed Box SAM Pos.(Ours) | 74.97 | 74.92 | 40.62
Variable Box SAM Pos.(Ours) | 79.93 | 79.19 | 44.81
AnimalTrack Subset - Method | Recall | mAP50 | mAP
Faster-RCNN [38] AnimalTrack[2] | - | 34.40 | 16.10
Fixed Box SAM Pos.(Ours) | 64.04 | 70.57 | 35.61
Variable Box SAM Pos.(Ours) | 72.55 | 77.57 | 43.90

TABLE II: GMOT-40 and AnimalTrack Subset Detection Comparison. Comparison
results obtained from [1, 2, 20, 36]. (-) means no metric provided by
publication. Fixed/Variable Box indicates provided box format, SAM indicates
presence of SAM module, Pos. indicates presence of positional filter. Best
results are in bold.

Method | Recall% | mAP50% | mAP%
---|---|---|---
Faster R-CNN[38][30] | 13.55 | 26.83 | 10.25
CornerNet[39][30] | 24.03 | 28.37 | 12.29
CenterNet[40][30] | 24.87 | 28.93 | 12.35
FPN[41][30] | 25.59 | 29.88 | 12.93
D&T[42][30] | 25.64 | 32.28 | 14.21
FGFA[43][30] | 27.21 | 33.34 | 14.44
FCOS[44] Baseline WACV22’ [31] | 26.98 | 32.42 | 11.44
Video-Rep. WACV22’ [31] | 41.28 | 49.01 | 21.82
YoloV8 Trained Control[45] | 46.50 | 49.50 | 27.50
Multi-Class Variable SAM Pos.(Ours) | 48.80 | 58.40 | 29.30
Single-Class Variable SAM Pos.(Ours) | 60.30 | 72.00 | 37.70

TABLE III: Visdrone-2019 Task 2 Video Detection Results on test-dev set. Our
trained detector is validated against results provided by [30, 31]. YOLOv8
trained control model is used for comparison purposes[45]. Best results for
the comparion are in bold.

###  IV-D Discussion

Multi-class and single-class performance are explained separately. Starting
with single class datasets, Tab. II shows that the pipeline outperforms all
available works under the reported recall, mAP50, and mAP metrics. The
proposed solution performs the best by a margin of 19.5%, 5.3%, and 0.6% under
the fixed regiment of GMOT-40 for recall, mAP50, and mAP. The variable
regiment outperforms the competitors by a margin of 24.5%, 9.5%, and 4.8% to
the corresponding metric leader.

Our pipeline performance over the test set of AnimalTrack[2] is reported in
Tab.II. POPCat performs 36.1%, and 19.5% better in mAP50 and mAP results when
compared to the faster-RCNN[38] baseline for fixed selection. Results are
averaged to a 43.1%, and 27.8% improvement in mAP and mAP50 for the variable
selection cases.

Benchmarking on the Visdrone-2019 test-dev dataset shows that our pipeline
outperforms all other detectors by a minimum margin of 21.6% in recall, 25.0%
in mAP50, and 14.8% in mAP Tab. II[30, 31]. We acknowledge that our system has
access to 54 frames out of the total 6635 frames of the testing dataset. Our
pipeline is able to capture context specific data about the environment and
targets which leads to significantly better results. We argue that production
applications would allow for a calibration phase for vision systems which
allows for training of the system on the intended operational views. To
validate the operation of yolov8 we utilized a pretrained yolov8-x.pt model
from huggingface [45] and validated it with the same setting on Visdrone
dataset. Our pipeline outperforms the control test by 2.3% recall, 8.9% mAP50,
and 1.8% mAP. This test demonstrates that the labeling methodology has a
significant impact on the performance of the yolov8 detector. Furthermore, it
demonstrates that the YOLOv8 architecture has an excellent capacity to learn
and adapt to difficult datasets.

A third experiment quantifies the effect of class mismatches in the model.
Class mismatch in the multi-class model refers to instances where the
boundaries of an object are marked with the incorrect class. We depict class
mismatch in Fig.4 where the trucks are misclassified as pedestrians (red).
Under the Visdrone challenge, class mismatch occurs as a result of class
imbalance and the sparse labeling methodology applied. To verify this
phenomenon, we perform a single-class experiment on Visdrone and evaluate the
precision and recall of the system. The single class model shows an
improvement of 11.5% recall, 13.6% mAP50, and 8.4% in mAP Tab.III. This means
that the detector is still capable of detecting all listed targets but
struggles with class assignment. We postulate that including a classifier
stage at the end of the system may improve multi-class results. We note that
there is still room for improvement within the propagation module as shown in
this experiment.

Figure 4: Class switching is seen in multi-class inferencing on the
Visdrone-2019 dataset. Video uav0000077_00720_v shows pedestrians as boxed in
red, cars boxed in orange, trucks in green, people sitting in pink, and vans
in yellow.

With this pipeline, a total of 149 ground truth annotations (40 GMOT-40, 58
AnimalTrack, 51 Visdrone) generate 4470 labeled frames for 115 separate videos
(40 GMOT-40, 58 AnimalTrack, 17 Visdrone). We train 99 detection models to
inference a total of 40947 frames (9603 GMOT-40, 24709 AnimalTrack, 6635
Visdrone 2019 Test-set dev) with the data above. The manual frame-to-inference
frames ratio is 1:274, making our pipeline extremely efficient for video
annotation. Furthermore, our work exceeds all previous works under the fixed
box selection with SAM and positional filtering, and has further improvements
with the variable regiment. We believe the best use case for our pipeline are
industrial computer vision settings. These environments tend to feature
multiple targets, a fixed camera, and well-defined backgrounds which synergize
with our pipeline.

GMOT-40[1] \- Method - Total Instances:255235 | Precision$\uparrow$ | Recall%$\uparrow$ | TP$\uparrow$ | FP$\downarrow$ | FP%$\downarrow$ | FN$\downarrow$ | FN%$\downarrow$ | mAP50$\uparrow$ | mAP$\uparrow$ | F1$\uparrow$
---|---|---|---|---|---|---|---|---|---|---
Fixed Box Selection / No SAM / No Positional Filter | 71.54 | 61.33 | 156536 | 62269 | 24.40 | 98699 | 38.67 | 58.63 | 28.04 | 66.04
Fixed Box Selection / No SAM / Positional Filter | 72.81 | 61.70 | 157485 | 58815 | 23.04 | 97750 | 38.30 | 60.33 | 27.78 | 66.80
Variable Box Selection / No SAM / No Positional Filter | 88.43 | 74.38 | 189841 | 24831 | 9.73 | 65394 | 25.62 | 77.35 | 46.02 | 80.80
Variable Box Selection / No SAM / Positional Filter | 89.36 | 74.68 | 190599 | 22687 | 8.89 | 64636 | 25.32 | 77.65 | 45.17 | 81.36
Fixed Box Selection / SAM / No Positional Filter | 86.68 | 74.70 | 190657 | 29312 | 11.48 | 64578 | 25.30 | 74.58 | 40.10 | 80.24
Fixed Box Selection / SAM / Positional Filter | 86.98 | 74.97 | 191336 | 28651 | 11.22 | 63899 | 25.03 | 74.92 | 40.62 | 80.53
Variable Box Selection / SAM / No Positional Filter | 90.21 | 78.87 | 201307 | 21853 | 8.56 | 53928 | 21.13 | 78.91 | 44.67 | 84.16
Variable Box Selection / SAM / Positional Filter | 90.59 | 79.93 | 203998 | 21196 | 8.30 | 51237 | 20.07 | 79.19 | 44.81 | 84.92
AnimalTrack[2] \- Method - Total Instances:428753 | Precision$\uparrow$ | Recall%$\uparrow$ | TP$\uparrow$ | FP$\downarrow$ | FP%$\downarrow$ | FN$\downarrow$ | FN%$\downarrow$ | mAP50$\uparrow$ | mAP$\uparrow$ | F1$\uparrow$
Fixed Box Selection / No SAM / No Positional Filter | 68.68 | 57.58 | 246872 | 112582 | 26.26 | 181881 | 42.42 | 55.65 | 22.52 | 64.13
Fixed Box Selection / No SAM / Positional Filter | 69.38 | 58.85 | 252255 | 111321 | 25.96 | 176498 | 41.17 | 58.19 | 24.19 | 64.79
Variable Box Selection / No SAM / No Positional Filter | 88.75 | 77.57 | 332569 | 42151 | 9.83 | 96184 | 22.43 | 83.88 | 55.17 | 81.11
Variable Box Selection / No SAM / Positional Filter | 89.17 | 76.80 | 329288 | 39993 | 9.33 | 99465 | 23.20 | 83.12 | 53.94 | 80.78
Fixed Box Selection / SAM / No Positional Filter | 83.97 | 68.99 | 295808 | 56461 | 13.17 | 132945 | 31.01 | 75.59 | 39.64 | 75.00
Fixed Box Selection / SAM / Positional Filter | 84.75 | 68.90 | 295401 | 53158 | 12.40 | 133352 | 31.10 | 75.82 | 39.89 | 75.29
Variable Box Selection / SAM / No Positional Filter | 86.83 | 77.88 | 333922 | 50660 | 11.82 | 94831 | 22.12 | 82.98 | 47.70 | 81.07
Variable Box Selection / SAM / Positional Filter | 87.01 | 77.52 | 332377 | 49632 | 11.58 | 96376 | 22.48 | 82.79 | 47.52 | 81.21

TABLE IV: Ablation Study - GMOT-40 and AnimalTrack. Best results for the
experiments are in bold.

###  IV-E Ablation Study

Figure 5: Comparison of pipeline components. Yellow arrows indicate changes in
detection

The goal of the ablation study is to validate the design choices of the POPCat
pipeline. All results are collected with the implementation detailed in
section IV-B for GMOT-40 and AnimalTrack. Visdrone was not included in the
ablation study, as variables such as class distribution, variations in object
sizing, ID switching, and scene entry cannot be ablated in an effective
manner.

####  IV-E1 Application of Positional Filtering

Positional filtering as it relates to the pruning of objects, is used to
control tracks that leave the extent of the image. This feature reduces the
false positive percentage (FP%) by eliminating erroneous annotations due to
loss of track in the propagation stage. The overall FP% observes a reduction
of 1.4% (fixed box selection), 0.8% (Variable box selection), 0.3% (Fixed box
with SAM), and 0.3% (Variable box with SAM) for the GMOT-40 dataset.
Positional filtering minimally impacts the recall%, with increases under 1.0%.
For AnimalTrack, the overall FP% is reduced by 0.3% (fixed box selection),
0.5% (variable box selection), 0.8% (Fixed box with SAM), and 0.2% (Variable
box with SAM). The percentage recall increase by less than 1% for AnimalTrack
cases. Fig.5 depicts the effect of positional filtering on the output results.
By removing lost tracks, we ensure the accuracy of the dataset used to train
the detector. Therefore, the inference frames no longer show FPs as is seen
between a) and b) Fig. 5. The characteristics of the object trajectories
within a video dictate the impact of positional filtering on performance.
Highly mobile targets will benefit more from positional filtering.

####  IV-E2 Application of Segment Anything Masking

The percent recall is increased by 13.3% in GMOT-40, and 10.7% in AnimalTrack
between fixed selections and fixed SAM method. The use of SAM to improve the
precision of bounding boxes halved the FP% with a reduction of 12.4% and 13.3%
for the fixed box annotation of GMOT-40 and AnimalTrack. SAM has a lesser
effect on variable box annotations with the recall increasing by 4.9% for
GMOT-40 and 0.5% increase for AnimalTrack. For the variable box selection, the
effect of SAM on FP% is observed to be mixed, as GMOT-40 has a 0.8% decrease,
and AnimalTrack averages a 2.1% increase. Elevated FP% in AnimalTrack may be
due to differences in the ground truth box size and detected size. A
qualitative review of videos indicates that output box detections are smaller
than ground-truth detections resulting in an incorrect FP assumption. Note the
effectiveness of background exclusion in picture c) compared to d) in Fig.5.
Increases in recall are also observed in the higher number of detections in c)
compared to d) Fig.5. Based on these results, SAM is a critical addition to
the POPCat pipeline.

####  IV-E3 Bounding Box Size Initialization

Depending on the annotation requirements of the dataset, a fixed or variable
box is required. GMOT-40 observes an average increase of 13.0% in recall
between fixed-box and variable-box methods. The SAM-based methods average an
increase of 4.6%. The FP% is halved, an average decrease of 14.4% from 23.7%
for fixed-box to variable-box, and 2.9% from 8.4% for the SAM-based methods is
observed. Under AnimalTrack, the recall is increased by an average of 19.0%
(fixed box and variable box) and 8.7% (fixed SAM and variable SAM). The FP% is
decreased by an average of 16.5% from 26.0% (fixed-box and variable-box) and
1.0% from 12.8% (fixed-SAM and variable-SAM). The performance improvements
result from a better representation of the target data[46]. With a variable
box selection, the target is fully enclosed in the annotation-sequence pair.
This leads to a more accurate box representation of the target which improves
the trained detector performance. This can be observed in Fig. 5 between e)
and f) where the detected area in f) fully encloses the targets when compared
to e). The performance improvements are at the cost of labeling complexity and
annotation time for the first frame.

##  V Conclusion

In this paper, we propose a novel annotation pipeline POPCat that utilizes
point propagation and segmentation to train a detector for annotation of a
given video. The detector results exceed the comparable literature review
results for the GMOT-40, AnimalTrack, and Visdrone-2019 test data. Our
pipeline exploits temporal consistencies in video sequences to propagate
highly accurate labels with an overall manual-to-automatic annotation ratio of
1:274. To accommodate end-user preference, we propose a simple and complex
method of pipeline initiation. We benchmark the results of such choices to
illustrate improvements of up to 50% false positive rates and recall. We
report a variety of detection metrics to provide a complete picture of the
performance for future comparisons. We ensure that the pipeline can operate on
limited GPU hardware while maintaining a highly time-efficient annotation
process. We achieve the goal of minimizing the human cost of annotation by
limiting it to a subset of total frames. The pipeline achieves the goals of
streamlining, simplifying, and improving the speed of custom video dataset
creation.

## References

  * [1] H. Bai, W. Cheng, P. Chu, J. Liu, K. Zhang, and H. Ling, “Gmot-40: A benchmark for generic multiple object tracking,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 6719–6728.
  * [2] L. Zhang, J. Gao, Z. Xiao, and H. Fan, “Animaltrack: A benchmark for multi-animal tracking in the wild,” 2022.
  * [3] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The kitti dataset,” _International Journal of Robotics Research (IJRR)_ , 2013.
  * [4] P. Voigtlaender, M. Krause, A. Osep, J. Luiten, B. B. G. Sekar, A. Geiger, and B. Leibe, “Mots: Multi-object tracking and segmentation,” _arXiv:1902.03604[cs]_ , 2019, arXiv: 1902.03604. [Online]. Available: http://arxiv.org/abs/1902.03604
  * [5] G. Jocher, A. Chaurasia, and J. Qiu, “YOLO by Ultralytics,” Jan. 2023. [Online]. Available: https://github.com/ultralytics/ultralytics
  * [6] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” 2014.
  * [7] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _European conference on computer vision_. Springer, 2020, pp. 213–229.
  * [8] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick, “Segment anything,” _arXiv:2304.02643_ , 2023.
  * [9] P. Zhu, D. Du, L. Wen, X. Bian, H. Ling, Q. Hu, T. Peng, J. Zheng, X. Wang, Y. Zhang, L. Bo, H. Shi, R. Zhu, B. Dong, D. R. Pailla, F. Ni, G. Gao, G. Liu, H. Xiong, J. Ge, J. Zhou, J. Hu, L. Sun, L. Chen, M. Lauer, Q. Liu, S. S. Chennamsetty, T. Sun, T. Wu, V. A. Kollerathu, W. Tian, W. Qin, X. Chen, X. Zhao, Y. Lian, Y. Wu, Y. Li, Y. Li, Y. Wang, Y. Song, Y. Yao, Y. Zhang, Z. Pi, Z. Chen, Z. Xu, Z. Xiao, Z. Luo, and Z. Liu, “Visdrone-vid2019: The vision meets drone object detection in video challenge results,” in _2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)_ , 2019, pp. 227–235.
  * [10] J. Castaño-Amorós, F. Fuentes, and P. Gil, “Mosppa: Monitoring system for palletised packaging recognition and tracking,” _The International Journal of Advanced Manufacturing Technology_ , vol. 125, 02 2023.
  * [11] J. Rutinowski, H. Youssef, S. Franke, I. F. Priyanta, F. Polachowski, M. Roidl, and C. Reining, “Semi-automated computer vision based tracking of multiple industrial entities–a framework and dataset creation approach,” _arXiv preprint arXiv:2304.00950_ , 2023.
  * [12] C. B. Rasmussen, K. Kirk, and T. B. Moeslund, “The challenge of data annotation in deep learning—a case study on whole plant corn silage,” _Sensors_ , vol. 22, no. 4, p. 1596, 2022.
  * [13] L. Dörr, F. Brandt, M. Pouls, and A. Naumann, “Fully-automated packaging structure recognition in logistics environments,” in _2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA)_ , vol. 1. IEEE, 2020, pp. 526–533.
  * [14] R.-J. Hong, Y.-R. Li, M.-H. Hung, J.-W. Chang, and J. C. Hung, “Integrating object detection and semantic segmentation into automated pallet forking and picking system in agv,” in _Frontier Computing_ , J. C. Hung, N. Y. Yen, and J.-W. Chang, Eds. Singapore: Springer Nature Singapore, 2022, pp. 121–129.
  * [15] Y. Li, Y. Niu, Y. Liu, L. Zheng, Z. Wang, and W. Zhe, “Computer vision based conveyor belt congestion recognition in logistics industrial parks,” in _2021 26th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA )_ , 2021, pp. 1–8.
  * [16] H. Pan, Y. Shi, X. Lei, Z. Wang, and F. Xin, “Fast identification model for coal and gangue based on the improved tiny yolo v3,” _Journal of Real-Time Image Processing_ , vol. 19, pp. 1–15, 06 2022.
  * [17] A. Karami, M. Crawford, and E. J. Delp, “Automatic plant counting and location based on a few-shot learning technique,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 13, pp. 5872–5886, 2020.
  * [18] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [19] A. Radford and K. Narasimhan, “Improving language understanding by generative pre-training,” 2018. [Online]. Available: https://api.semanticscholar.org/CorpusID:49313245
  * [20] K. H. Tran, T.-P. Nguyen, A. D. L. Dinh, P. Nguyen, T. Phan, K. Luu, D. Adjeroh, and N. H. Le, “Z-gmot: Zero-shot generic multiple object tracking,” _arXiv preprint arXiv:2305.17648_ , 2023.
  * [21] L. H. Li, P. Zhang, H. Zhang, J. Yang, C. Li, Y. Zhong, L. Wang, L. Yuan, L. Zhang, J.-N. Hwang _et al._ , “Grounded language-image pre-training,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 10 965–10 975.
  * [22] T. Nguyen, C. Pham, K. Nguyen, and M. Hoai, “Few-shot object counting and detection,” in _European Conference on Computer Vision_. Springer, 2022, pp. 348–365.
  * [23] X. Shi, S. Zhang, M. Cheng, L. He, X. Tang, and Z. Cui, “Few-shot semantic segmentation for industrial defect recognition,” _Computers in Industry_ , vol. 148, p. 103901, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0166361523000519
  * [24] CVAT.ai Corporation, “Computer Vision Annotation Tool (CVAT),” Sep. 2022. [Online]. Available: https://github.com/opencv/cvat
  * [25] A. W. Harley, Z. Fang, and K. Fragkiadaki, “Particle video revisited: Tracking through occlusions using point trajectories,” in _European Conference on Computer Vision_. Springer, 2022, pp. 59–75.
  * [26] W. Ji, J. Li, Q. Bi, W. Li, and L. Cheng, “Segment anything is not always perfect: An investigation of sam on different real-world applications,” _arXiv preprint arXiv:2304.05750_ , 2023.
  * [27] C. Rother, V. Kolmogorov, and A. Blake, “”grabcut”: interactive foreground extraction using iterated graph cuts.” _ACM Trans. Graph._ , vol. 23, no. 3, pp. 309–314, 2004. [Online]. Available: http://dblp.uni-trier.de/db/journals/tog/tog23.html#RotherKB04
  * [28] M. Cruz, J. Keh, M. Rivera, N. Velasco, J. A. Jose, E. Sybingco, E. Dadios, W. Madria, and A. Miguel, “Auto-fit: A human-machine collaboration feature for fitting bounding box annotations,” in _2020 IEEE 12th International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment, and Management (HNICEM)_ , 2020, pp. 1–6.
  * [29] E. Real, J. Shlens, S. Mazzocchi, X. Pan, and V. Vanhoucke, “Youtube-boundingboxes: A large high-precision human-annotated data set for object detection in video,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 7464–7473.
  * [30] P. Zhu, L. Wen, D. Du, X. Bian, H. Fan, Q. Hu, and H. Ling, “Detection and tracking meet drones challenge,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 11, pp. 7380–7399, 2022.
  * [31] M. Fujitake and A. Sugimoto, “Video representation learning through prediction for online object detection,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops_ , January 2022, pp. 530–539.
  * [32] E. Ristani, F. Solera, R. Zou, R. Cucchiara, and C. Tomasi, “Performance measures and a data set for multi-target, multi-camera tracking,” in _European conference on computer vision_. Springer, 2016, pp. 17–35.
  * [33] R. Padilla, W. L. Passos, T. L. B. Dias, S. L. Netto, and E. A. B. da Silva, “A comparative analysis of object detection metrics with a companion open-source toolkit,” _Electronics_ , vol. 10, no. 3, 2021. [Online]. Available: https://www.mdpi.com/2079-9292/10/3/279
  * [34] L. Huang, X. Zhao, and K. Huang, “Globaltrack: A simple and strong baseline for long-term tracking,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 07, 2020, pp. 11 037–11 044.
  * [35] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _Computer Vision – ECCV 2014_ , D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, Eds. Cham: Springer International Publishing, 2014, pp. 740–755.
  * [36] Q. Liu, Y. Li, Y. Jiang, and Y. Fu, “Siamese-detr for generic multi-object tracking,” 2023.
  * [37] S. Shao, Z. Li, T. Zhang, C. Peng, G. Yu, X. Zhang, J. Li, and J. Sun, “Objects365: A large-scale, high-quality dataset for object detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2019.
  * [38] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” 2016.
  * [39] H. Law and J. Deng, “Cornernet: Detecting objects as paired keypoints.” _International Journal of Computer Vision_ , vol. 128, no. 3, 2020.
  * [40] K. Duan, S. Bai, L. Xie, H. Qi, Q. Huang, and Q. Tian, “Centernet: Keypoint triplets for object detection,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 6568–6577.
  * [41] T.-Y. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , July 2017.
  * [42] C. Feichtenhofer, A. Pinz, and A. Zisserman, “Detect to track and track to detect,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 3038–3046.
  * [43] X. Zhu, Y. Wang, J. Dai, L. Yuan, and Y. Wei, “Flow-guided feature aggregation for video object detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 408–417.
  * [44] Z. Tian, C. Shen, H. Chen, and T. He, “Fcos: Fully convolutional one-stage object detection,” 2019.
  * [45] “mshamrai/yolov8x-visdrone,” https://huggingface.co/mshamrai/yolov8x-visdrone, accessed: 2023-08-04.
  * [46] C. Agnew, C. Eising, P. Denny, A. Scanlan, P. Van De Ven, and E. M. Grua, “Quantifying the effects of ground truth annotation quality on object detection and instance segmentation performance,” _IEEE Access_ , vol. 11, pp. 25 174–25 188, 2023.


# SiLVR: Scalable Lidar-Visual Reconstruction with Neural Radiance Fields for
Robotic Inspection

Yifu Tao1, Yash Bhalgat2, Lanke Frank Tarimo Fu1, Matias Mattamala1, Nived
Chebrolu1, and Maurice Fallon1

###### Abstract

We present a neural-field-based large-scale reconstruction system that fuses
lidar and vision data to generate high-quality reconstructions that are
geometrically accurate and capture photo-realistic textures. This system
adapts the state-of-the-art neural radiance field (NeRF) representation to
also incorporate lidar data which adds strong geometric constraints on the
depth and surface normals. We exploit the trajectory from a real-time lidar
SLAM system to bootstrap a Structure-from-Motion (SfM) procedure to both
significantly reduce the computation time and to provide metric scale which is
crucial for lidar depth loss. We use submapping to scale the system to large-
scale environments captured over long trajectories. We demonstrate the
reconstruction system with data from a multi-camera, lidar sensor suite
onboard a legged robot, hand-held while scanning building scenes for 600
metres, and onboard an aerial robot surveying a multi-storey mock disaster
site-building. Website: https://ori-drs.github.io/projects/silvr/

Figure 1: Large-scale reconstruction consisting of 8 submaps of Maths
Institute and H B Allen Centre in Oxford. The bottom row shows the novel views
synthesised from the model and surface normals at three different locations.
The trajectory of each submap is visualised in a different colour.

††1Oxford Robotics Inst., Dept. of Eng. Science, Univ. of Oxford, UK. {yifu,
fu, matias, nived<EMAIL_ADDRESS>
2Visual Geometry Group, Dept. of Eng. Science, Univ. of Oxford, UK. yashsb
<EMAIL_ADDRESS>
This project has been partly funded by the Horizon Europe project Digiforest
(101070405). Maurice Fallon is supported by a Royal Society University
Research Fellowship. For the purpose of open access, the authors have applied
a Creative Commons Attribution (CC BY) license to any Accepted Manuscript
version arising.

##  I INTRODUCTION

Dense 3D reconstruction is a task that underpins a range of robotic
applications such as industrial inspection and autonomous navigation. Common
sensors used for reconstruction include cameras and lidar. Camera-based
reconstruction systems use techniques including Structure-from-Motion (SfM)
and Multi-View Stereo (MVS) to produce dense textured reconstructions [1].
However, these systems rely on good lighting conditions as well as having
observed the many view constraints. They also struggle with textureless areas.
Lidar provides accurate geometric information at long range — as it directly
measures distances to surfaces. This makes it desirable for large-scale
outdoor environments, but the sensor measurement is usually much sparser than
a camera image. It also does not capture colour which is important for
applications such as virtual reality and 3D asset generation.

Classical reconstruction systems have used point clouds, occupancy maps, and
sign-distance fields (SDF) as their internal 3D representation. Recently,
neural radiance field (NeRF) [2] has gained popularity for visual
reconstruction. With differentiable rendering, this approach optimises a
continuous 3D representation by minimising the difference between the rendered
image and reference camera images. It can achieve state-of-the-art novel view
synthesis quality.

As with traditional vision-based reconstruction methods, NeRF struggles to
estimate accurate geometry in locations where there is limited multi-view
input and little texture. Autonomous systems commonly encounter these
situations - for example, reconstructing a wall with uniform colour when
traveling directly towards it. This problem could be addressed by using lidar
sensing to give accurate geometric measurements in these textureless objects.
In addition, the use of lidar data can mitigate the need to capture multiple
camera views. It is impractical for an inspection robot (e.g. The Spot
quadruped on an industrial facility) to execute object-centric trajectories
just to improve the visual reconstruction. This motivates the development of a
reconstruction system that fuses these sensors.

In this work, we present a NeRF-based reconstruction system that integrates
both lidar and visual information to generate accurate, textured 3D
reconstruction which also provides photo-realistic novel view synthesis. Our
method builds upon NeRF implementation [3] utilising hash encoding [4] that
takes minutes to achieve photo-realistic rendering. This is extended with
geometric constraints from lidar to improve reconstruction quality. The use of
lidar enables depth measurements even from featureless areas, which cannot be
obtained from SfM [5]. Surface normal can also be computed from a lidar scan,
which is more robust than learning-based priors [6] which can suffer from
input data distribution shift in real-world deployment.

This system is demonstrated using data from a perception sensor suite which
contains three wide field-of-view cameras and a 3D lidar to enable robots to
reconstruct in all directions. We take advantage of a lidar-inertial odometry
and SLAM mapping system [7] as part of the pipeline. Experiments are presented
using a drone, a legged robot, and a handheld device in industrial and urban
environments.

In summary, our main contributions are:

  * •

A dense textured 3D reconstruction system that achieves accurate geometry
that’s on-par with lidar, and photorealistic novel view synthesis

  * •

Integration with a lidar SLAM system, so that the NeRF is trained with both
depth and surface normal obtained from lidar data, and metric-scale trajectory
with a reduction in computation time by 50% compared with up-to-scale offline
Structure-from-Motion method commonly used in literature.

  * •

A sub-mapping system that scales to large outdoor environments — with
trajectories over 600 metres

  * •

Evaluation of the system on real-world large-scale outdoor datasets captured
from multiple robotic platforms.

##  II RELATED WORKS

###  II-A Large-scale 3D Reconstruction

Lidar is the dominant sensor for accurate 3D reconstruction of large-scale
environments [8, 9], thanks to its accurate, long-range measurements.
Volumetric lidar mapping relies on high-frequency odometry estimates from
scan-matching and IMU measurements [10, 11, 12, 7]. Yet, lidar-based systems
may yield partial reconstructions, especially when robots explore scenes and
have limited field-of-view sensors. Visual cues can be used to densify lidar
mapping [13].

Alternatively, SfM systems such as COLMAP [1] can generate large-scale
textured reconstructions from images by first estimating camera poses using
sparse feature points in a joint bundle adjustment process, followed by
refinement with multi-view stereo algorithms [14]. However, they face
challenges in low-texture areas, repetitive patterns, and feature matching
across views can be problematic with changing lighting or non-Lambertian
materials. Urban Radiance Fields [15] proposes using lidar sweeps along with
RGB images to optimise a neural radiance field model that can be used for 3D
surface extraction. Our work shares this approach, fusing both sensor
modalities to generate precise geometry, overcoming limitations of vision-only
approaches in low-texture areas, and being much denser than lidar-only
reconstructions.

A common strategy to extend dense reconstructions to large-scale areas is
through submapping [16, 17, 18, 19, 20]. These approaches partition the scene
into individual local submaps which can incorporate the effect of loop closure
corrections while still producing a consistent global map while achieving
significantly lower runtime. Our work also adopts the submapping approach and
partitions large-scale scenes into local maps (approximately 50x50m) using a
globally-consistent lidar SLAM trajectory. This increases the representation
capability and improves reconstruction, especially for thin objects.

###  II-B Neural Field Representation

Neural Radiance Fields (NeRF) [2] proposed using a multilayer perceptron (MLP)
to represent a continuous radiance field with differentiable volume rendering
to reconstruct novel views. It implicitly induces multi-view consistency with
geometric priors in the learned encodings. NeRF and its many variants used
frequency encoding [21] to encode spatial coordinates, but these suffer from
long training times, typically a few hours per scene. Alternative explicit
representations of radiance fields, including dense voxel-grids with trainable
per-vertex features [22, 4] and more recently 3D Gaussians [23] are shown to
accelerate the training, at the cost of being more memory intensive. Octree or
sparse-grid structures [24, 4] can reduce memory usage by pruning grid-
features in empty space. Our work is built upon Nerfacto [3] which
incorporates the main features from other NeRF works [4, 25, 26] that have
been found to work well with real data.

While NeRFs excel at high-quality view synthesis, obtaining a 3D surface from
these representations remains challenging, mainly due to the flexible
volumetric representation being under-constrained by the limited multi-view
inputs. One approach to improve the reconstruction is to impose depth
regularisation [5, 15] or surface normal regularisation [6]. Another approach
is to impose surface priors on the volumetric field and use representations
such as Signed Distance Field (SDF) [27, 28] to enforce a surface
reconstruction output, although the novel view synthesis quality might be
compromised [29] with this approach. Our method is built upon a volume density
representation which is extended with depth [5] and surface normal [6]
regularisations from lidar measurements instead of using SfM [5] or learnt
priors [6]. In particular, it can significantly improve the reconstruction
quality in texture-less areas with smooth surface.

Neural field representations have been used for lidar-base mapping [30, 31],
showing promise in generating more complete and compact reconstructions than
traditional methods. While these works also build upon implicit map
representation, they do not use visual data for building the map. Our system
uses visual information and multi-view geometry constraints, therefore can
reconstruct regions outside lidar field-of view.

##  III METHOD

We present a system for large-scale 3D reconstruction based on a NeRF
representation tailored for robotic inspection tasks. We use a custom-designed
sensor payload with a 3D lidar sensor, three fisheye cameras, and an IMU,
which is suitable for use on various robot platforms. In Sec. III-B, we
present our approach to fusing both lidar and vision information during the
optimization phase to ensure high-quality reconstruction. Furthermore, we
employ a submapping strategy to scale the approach to large areas described in
Sec. III-D. An overview of the system is presented in Fig. 2.

Figure 2: System Overview: Frontier, our custom perception unit, has three
fisheye colour cameras with an IMU and a 3D lidar. Our online state
estimator’s trajectory is refined with COLMAP and partitioned into submaps.
The camera image, lidar depth, and normal image are used to train a NeRF to
get the final 3D reconstruction.

###  III-A NeRF-based Scene Representation

Our work builds on the differentiable volume rendering framework used for
novel view synthesis [2, 32, 33, 34]. To render a novel view from a NeRF given
a viewpoint, we cast rays from the camera origin along the viewing direction
for each pixel $u$ in the image plane, and render the pixel-colour by
integrating over points sampled along the ray. This volume rendering integral
is approximated using quadrature rule [35, 36] as
$\hat{c}_{u}=\sum_{i=0}^{N}w_{i}c_{i}$, where

$w_{i}=\exp{\left(-\sum_{j=1}^{i-1}\delta_{j}\sigma_{j}\right)}(1-\exp{(-\delta_{i}\sigma_{i})}).$
(1)

Here, $\sigma_{i}$ and $c_{i}$ are the predicted density and color for the
sampled 3D points and $\hat{c}_{u}$ is the rendered pixel color.

Our implementation is built on top of the Nerfacto method from Nerfstudio [3].
Nerfacto’s rendering quality is comparable to state-of-the-art methods such as
MipNeRF-360 [25] while achieving a substantial acceleration in reconstruction
speed since it also incorporates efficient hash encoding from Instant-NGP [4].
We also use scene contraction proposed in [25] to improve memory efficiency
and represent scenes with high-resolution content near the input camera
locations. The contraction function non-linearly maps any point in space into
a cube of side length 2, and represents the scene within this contracted
space. Since there is large variation in exposure and lighting conditions, we
use a per-frame appearance encoding for each image, similar to [15, 26].

###  III-B Geometric Constraints from Lidar Measurements

3D reconstruction with NeRF becomes challenging when the surface has uniform
texture and limited multi-view constraints. Lidar measurements are
complementary as it can provide accurate measurements in such scenarios. In
our work, we incorporate the lidar measurements in the NeRF optimization.
Specifically, we impose a lidar-based depth regularisation by adding a depth
loss [5] defined as the KL-Divergence between a normal distribution around the
lidar depth-measurement $\mathbf{D}$ and the rendered ray distribution $h(t)$
from the NeRF model:

$\mathcal{L}_{\text{Depth
}}=\sum_{\mathbf{r}\in\mathcal{R}}\mathrm{KL}[\mathcal{N}(\mathbf{D},\hat{\sigma})\|h(t)]$
(2)

We also run a semantic segmentation network [37] to obtain a sky mask, and
minimise the weights of these rays similar to [15].

While the depth loss improves 3D reconstruction, we found that the surface
contains wavy artifacts in regions where it is expected to be smooth (see Fig.
5). To mitigate this, we compute the surface normal as the negative gradient
of the NeRF’s density field, and impose a further surface normal
regularisation loss, inspired by [6]:

$\mathcal{L}_{\text{normal
}}=\sum_{\mathbf{r}\in\mathcal{R}}\|\hat{N}(\mathbf{r})-\bar{N}(\mathbf{r})\|_{1}+\left\|1-\hat{N}(\mathbf{r})^{\top}\bar{N}(\mathbf{r})\right\|_{1}$
(3)

Figure 3: Comparison of reconstruction quality of Lidar-SLAM, Nerfacto
(vision-only) and our approach. Reconstructions are coloured with point-to-
point distance to the ground truth with increasing error from blue (0m) to red
(1m). The trajectory is shown in purple and overlaid on the ground truth scan
captured using a Leica BLK360. The zoomed-in views show the difference in
reconstruction quality. Overall, our approach is more complete w.r.t lidar-
only reconstruction, and geometrically more consistent w.r.t vision-only
reconstruction.

###  III-C Bootstrapping Camera Poses from SLAM with scale

Obtaining accurate camera poses is crucial as the pose accuracy directly
impacts the fidelity of the reconstructed model. A popular approach used in
most NeRF works is to obtain camera poses using offline Structure-from-Motion
methods such as COLMAP [1]. However, COLMAP has the following limitations: (1)
long computation time, especially for large image collections collected over a
long trajectory, and (2) inability to register all frames into one global map
when there is limited visual overlap between the images. These issues limit
its application in building a large-scale globally consistent map for robotic
applications.

In our work, we use our lidar-inertial odometry and SLAM system VILENS [7].
While VILENS achieves state-of-the-art results for online motion tracking, we
found that the camera poses obtained are less accurate than those of COLMAP.
This results in blurring artifacts in the renderings of the NeRF model.
Several works [20, 38] use noisy pose inputs and then jointly refine the poses
in parallel with the NeRF optimization to generate better results. In our
experiments with a collection of large number of images, we found that while
this pose-refinement approach sometimes leads to slightly better PSNR (peak
signal-to-noise ratio), the resulting rendering is still less sharp compared
to using the COLMAP estimated poses, and the training process is usually
unstable.

To overcome the above limitations, we use the SLAM poses as a pose prior and
refine the trajectory using COLMAP. Specifically, we replace the COLMAP mapper
with point triangulator which reads prior poses. This method has the advantage
of being faster, as it converts the incremental Structure-from-Motion into a
global bundle adjustment problem, and more importantly, results in COLMAP
being able to merge all available images in a single map. For a mission
spanning over 20 minutes, our COLMAP-with-prior pipeline achieves similar
rendering quality, while only taking half the computation time compared to a
fully offline COLMAP run. The computation time is similar to the robot’s
mission duration, making it more applicable for robotic applications.

After COLMAP computation, we rescale the trajectory using Sim(3) Umeyama
alignment with lidar-slam trajectory, so that the final trajectory is metric.
This is essential to use lidar measurements in III-B, since lidar depth is
also metric.

###  III-D Scaling NeRF with Submapping

Training a NeRF for large-scale scenes is challenging as a single NeRF model
has limited representation power and hardware constraints such as RAM usage
when loading thousands of images. We adopt a sub-mapping approach, and
partition the COLMAP-refined SLAM trajectory into clusters using Spectral
Clustering [39]. Different from [20], our representation is based on Instant-
NGP [4] which is orders of magnitude faster than a Multilayer Perception. The
submaps are trained in their local coordinate frames, and the final
reconstruction are Sim(3) transformed to the world coordinate frame using
their metric pose from III-C.

To generate 3D reconstructions from NeRF, we sample the rays used to train
each model and render the colour and depth to generate a 3D point. When
evaluating the 3D reconstruction, we found that submaps contain artifacts,
especially around their boundary. One cause of the artifacts is that the
boundary regions are observed sparsely with limited view constraints. To
tackle this, we identify regions with low surface density and remove them when
merging submap clouds to get the final reconstruction.

##  IV EXPERIMENTAL RESULTS

###  IV-A Hardware and Datasets

We evaluate our methods on a custom perception unit called Frontier, a multi-
camera lidar inertial device. It includes three 1.6 MP colour111Raw Bayered
images are processed using
https://github.com/leggedrobotics/raw_image_pipeline fisheye Alphasense
cameras on 3 sides of the device, with an IMU from Sevensense Robotics AG and
a Hesai Pandar QT64 lidar. We used the Frontier to collect datasets on
multiple platforms: a legged robot (Boston Dynamics Spot), a drone (DJI M600),
and a hand-held device. Spot and handheld Frontier datasets were taken at the
H B Allen Centre (HBAC) and Mathematical Institute (Maths Inst.) in Oxford.
The DJI M600 drone was operated at the Fire Service College (FSC). In the FSC
dataset, we only use the rectified front camera image as the drone propellers
were visible in the left and right camera images.

We use VILENS [7], a lidar-inertial SLAM system online to provide a globally
consistent trajectory and motion-corrected lidar measurements. The online SLAM
trajectory is further optimised by COLMAP offline, which improves the visual
reconstruction quality, as shown later in Tab. II. The lidar point clouds are
projected as a depth image. The surface normal is computed from the lidar
range image, and also projected as a normal image. We calibrate our cameras-
to-IMU extrinsics with Kalibr [40] and cameras-to-lidar with [41]. When
running COLMAP, we further optimise the intrinsics from Kalibr. For training
the NeRF model, we used an Nvidia RTX 3080 Ti and one iterations takes 4096
rays.

###  IV-B Evaluation Metrics

To evaluate the geometry of the reconstruction, we report Accuracy and
Completeness following the conventions of the DTU dataset [42]. Accuracy is
measured as the distance from the reconstruction to the reference 3D model
(ground truth) and encapsulates the reconstruction quality. Completeness is
the distance from the point-wise reference to the reconstruction and shows how
much of the surface is captured by the reconstruction. For the ground truth,
we use the centimetre-accurate point cloud captured by a survey-grade Leica
BLK360 laser scanner and register the lidar point clouds using transformation
obtained from ICP, which is also used to register NeRF reconstructions.

We also evaluate the visual quality by reporting the Peak Signal-to-Noise
Ratio (PSNR) and Structural Similarity Index (SSIM) [43]. Note that our images
have various exposure times, which lower the test PSNR even if the
reconstructed image is photorealistic.

###  IV-C Evaluation of the 3D Reconstruction

We perform a quantitative evaluation of our method on real-world datasets
captured by different robotic platforms. We compare the point cloud
reconstructions generated with the following configurations:

  1. 1.

Lidar-SLAM: lidar point cloud registered with SLAM poses refined by COLMAP

  2. 2.

Nerfacto [3]: baseline method using only the images

  3. 3.

SiLVR: Our method using photometric loss, depth loss, and surface normal loss

We sample the training rays and generate 10 million points from each submap
for Nerfacto and SiLVR. For Nerfacto, we excluded the points belonging to the
sky by computing a sky segmentation mask with further manual cropping. For
lidar-slam, we only include points within the camera field-of-view. This omits
points pointing backwards that may be occluded by the operator.

TABLE I:  Evaluation of 3D Reconstruction Quality

Method | Accuracy$\downarrow$ | Completeness$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$
---|---|---|---|---
| (m) | (m) | train | test | test
Maths Quad |  |  |  |  |
lidar-SLAM | 0.06 | 0.15 | / | / | /
Nerfacto mono | 1.38 | 0.33 | 31.3 | 17.7 | 0.65
Nerfacto 3-cam | 0.76 | 0.31 | 28.0 | 21.1 | 0.72
Ours mono | 0.08 | 0.12 | 30.1 | 17.5 | 0.61
Ours 3-cam | 0.08 | 0.11 | 27.3 | 20.5 | 0.71
Oxford HBAC |  |  |  |  |
lidar-SLAM | 0.05 | 0.25 | / | / | /
Nerfacto mono | 0.49 | 5.40 | 32.6 | 19.5 | 0.65
Nerfacto 3-cam | 0.28 | 0.40 | 29.8 | 20.6 | 0.74
Ours mono | 0.30 | 4.60 | 31.0 | 21.2 | 0.74
Ours 3-cam | 0.09 | 0.18 | 28.8 | 19.7 | 0.74
FSC |  |  |  |  |
lidar-SLAM | 0.08 | 0.08 | / | / | /
Nerfacto mono | 0.14 | 0.11 | 28.8 | 19.1 | 0.76
Ours mono | 0.11 | 0.09 | 27.7 | 19.1 | 0.75

NeRF models are trained for 10000 iterations for 5 minutes; When computing
accuracy and completeness, we crop regions where there is a change or no
overlap between the ground truth and lidar scans.

We summarise the quantitative results in Tab. I, and show 3D reconstructions
in Fig. 3. Compared to Nerfacto, our method incorporates lidar measurements
and has significantly better geometry which is shown in terms of accuracy and
completeness. Nerfacto struggles when reconstructing the uniformly-coloured
ground in Maths Inst., and the quad area in HBAC where the robot only walked
forwards. Compared to lidar-SLAM, our method generally achieves more complete
reconstruction since it uses dense visual information, while the accuracy
(8-11cm) is nearly on-par with lidar-slam (6-8cm) and much better than
Nerfacto (14-76cm).

Figure 4: Comparison of reconstruction of HBAC building using the front camera
only vs. using all the three cameras. The three-camera setup generates more
complete and accurate reconstructions compared to using only a single front-
facing camera. The multi-camera setting is important in robotic applications
where it would be infeasible to actively scan the entire scene to obtain
strong viewpoint constraints.

The advantage of our multi-camera sensor setup is demonstrated qualitatively
in Fig. 4. Compared to the three-camera setup, using only the front-facing
camera leads not only to an incomplete reconstruction but also worse geometry.
Visual reconstruction with photometric loss is biased towards generating a
good rendering only at the input viewing angle. The reconstruction using the
front-only camera in Fig. 4 is trained with images viewing a shallow angle of
the scene, and results in a poor geometric reconstruction when rendered from
an unseen angle.

###  IV-D Effect of Lidar Surface Normal Loss

While the depth loss Eq. (2) provides geometric constraints, we observe that
the resulting 3D reconstruction is not necessarily smooth for flat surfaces.
The results in Fig. 5 demonstrate how the surface normal loss Eq. (3) further
constrains the reconstruction geometry and improves the reconstruction
quality. Nerfacto fails to estimate the depth of the ground due to its uniform
texture. Using the lidar depth loss ensures ground reconstruction is at the
right height, however, the surface is still not smooth. The surface normal
loss furthers smoothens the surface and results in a higher-quality
reconstruction.

Figure 5: Comparison of surface normal renderings of the Maths Institute.
Incorporating normal constraints in addition to depth from lidar improves the
smoothness of the reconstruction. Right: The smooth reconstruction of the
ground portion highlights this improvement. TABLE II: Ablation: Effect of
Bootstrapping SLAM Poses

Method | Features | Prior |  Traj. Regis | PSNR$\uparrow$ | SSIM$\uparrow$ | Time
---|---|---|---|---|---|---
|  |  |  tered | Train | Test | Test |
|  |  |  (%) |  |  |  | (s)
VILENS | / | / |  100.0 | 23.0 | 17.4 | 0.64 | Online
NeRF refined | / | / |  100.0 | 23.2 | 17.9 | 0.65 | Online
COLMAP Sequential | 1024 |  |  57.6 | 25.9 | 19.1 | 0.71 | 3299.2
1024 | ✓ |  100.0 | 26.2 | 20.6 | 0.74 | 1729.9
8192 |  |  94.0 | 26.1 | 19.8 | 0.72 | 7850.0
8192 | ✓ |  100.0 | 26.2 | 20.4 | 0.73 | 4448.4
COLMAP VocabTree | 1024 |  |  54.7 | 26.2 | 19.0 | 0.71 | 4444.8
1024 | ✓ |  100.0 | 26.3 | 20.4 | 0.73 | 1052.5
8192 |  |  94.8 | 26.6 | 19.9 | 0.72 | 37067.5
8192 | ✓ |  100.0 | 26.3 | 20.4 | 0.74 | 11015.3

Results evaluated on HBAC-Maths dataset with 3254 images and duration of
1270s. Models trained for 4000 iterations. PSNR and SSIM are evaluated after
masking out the sky.

###  IV-E Effect of Bootstrapping SLAM Poses

We compare the performance of different strategies for computing poses: online
SLAM poses, SLAM poses with NeRF pose refinement [3], SLAM poses with COLMAP
[1] in different configurations, and COLMAP without any prior poses. For
COLMAP, we tested different numbers of features extracted per image, and two
different COLMAP feature matching algorithms: sequential matching with loop
closures and Vocabulary Tree Matcher.

The results are summarised in Tab. II. For all COLMAP configurations,
providing the SLAM prior poses not only accelerates pose computation, but also
leads to better test rendering, compared to the offline COLMAP. Our SLAM prior
poses also register all images in the trajectory, while when not provided,
COLMAP only registers 55%-95% images. Extracting more visual features per
image (from 1024 to 8192) leads to higher percentage of image registration and
better visual reconstruction (PSNR and SSIM). This comes at the expense of a
higher computation time, especially with the VocabTree matcher. Using the
COLMAP Sequential Matcher is generally faster than Vocabulary Tree Matcher.

###  IV-F Submapping for Large-Scale Environments

We show a large-scale reconstruction of HBAC-Maths with the handheld Frontier
using submapping in Fig. 1, as well as the trajectory for each submap. To
demonstrate the advantage of submapping, we compare the 3D reconstruction and
rendering quality using one NeRF model for the entire sequence versus NeRF
models built with multiple submaps. We present the qualitative results in Fig.
6. The reconstruction of the bike racks in Maths Inst. is blurred when only
using a single NeRF map due to its limited representation capability for
storing all objects over a large area. While using only a dedicated submap for
that local area, the reconstruction quality improves significantly as seen in
Fig. 6 (right).

Figure 6: Effect of submap size on reconstruction quality. A larger number of
submaps for a given area results in better reconstruction on thin objects such
as the bike racks on the right.

##  V CONCLUSIONS

In summary, we proposed a large-scale 3D reconstruction system fusing both
lidar and vision in a neural field via differentiable rendering. The proposed
sensor fusion overcomes the limitations of individual sensors, namely the
sparsity of the lidar and the fragility of vision in the presence of texture-
less surface, and limited multi-view constraints. We demonstrate large-scale
reconstruction results from real-world datasets collected from multiple robot
platforms in conditions suited for inspection tasks.

## Acknowledgment

The authors would like to thank Ren Komatsu for his help on software
development, Tobit Flatscher for deploying spot robot, Rowan Border for drone
data collection, and Sundara Tejaswi Digumarti for his helpful discussions.

## References

  * [1] J. L. Schönberger and J.-M. Frahm, “Structure-from-motion revisited,” in _IEEE Int. Conf. Computer Vision and Pattern Recognition_ , 2016.
  * [2] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” _Communications of the ACM_ , vol. 65, no. 1, pp. 99–106, 2021\.
  * [3] M. Tancik, E. Weber, E. Ng, R. Li, B. Yi, J. Kerr, T. Wang, A. Kristoffersen, J. Austin, K. Salahi, A. Ahuja, D. McAllister, and A. Kanazawa, “Nerfstudio: A modular framework for neural radiance field development,” in _SIGGRAPH_ , 2023.
  * [4] T. Müller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” _ACM Transactions on Graphics (TOG)_ , vol. 41, no. 4, pp. 102:1–102:15, Jul. 2022.
  * [5] K. Deng, A. Liu, J.-Y. Zhu, and D. Ramanan, “Depth-supervised nerf: Fewer views and faster training for free,” in _IEEE Int. Conf. Computer Vision and Pattern Recognition_ , 2022, pp. 12 882–12 891.
  * [6] Z. Yu, S. Peng, M. Niemeyer, T. Sattler, and A. Geiger, “MonoSDF: Exploring monocular geometric cues for neural implicit surface reconstruction,” _Conf. on Neural Information Processing Systems (NeurIPS)_ , 2022.
  * [7] D. Wisth, M. Camurri, and M. Fallon, “VILENS: Visual, inertial, lidar, and leg odometry for all-terrain legged robots,” _IEEE Trans. Robotics_ , vol. 39, no. 1, pp. 309–326, 2023.
  * [8] J. Behley and C. Stachniss, “Efficient surfel-based slam using 3d laser range data in urban environments,” in _Robotics: Science and Systems (RSS)_ , 2018\.
  * [9] J. Lin and F. Zhang, “R3LIVE: A robust, real-time, rgb-colored, lidar-inertial-visual tightly-coupled state estimation and mapping package,” in _IEEE Intl. Conf. on Robotics and Automation (ICRA)_ , 2022, pp. 10 672–10 678.
  * [10] J. Zhang and S. Singh, “LOAM: Lidar odometry and mapping in real-time.” in _Robotics: Science and Systems (RSS)_ , vol. 2, no. 9. Berkeley, CA, 2014, pp. 1–9.
  * [11] S. Zhao, H. Zhang, P. Wang, L. Nogueira, and S. Scherer, “Super odometry: Imu-centric lidar-visual-inertial estimator for challenging environments,” in _IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_. IEEE, 2021, pp. 8729–8736.
  * [12] W. Xu, Y. Cai, D. He, J. Lin, and F. Zhang, “Fast-lio2: Fast direct lidar-inertial odometry,” _IEEE Trans. Robotics_ , vol. 38, no. 4, pp. 2053–2073, 2022.
  * [13] Y. Tao, M. Popović, Y. Wang, S. T. Digumarti, N. Chebrolu, and M. Fallon, “3d lidar reconstruction with probabilistic depth completion for robotic navigation,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2022, pp. 5339–5346.
  * [14] Y. Furukawa and J. Ponce, “Accurate, dense, and robust multiview stereopsis,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 32, no. 8, pp. 1362–1376, 2010.
  * [15] K. Rematas, A. Liu, P. P. Srinivasan, J. T. Barron, A. Tagliasacchi, T. Funkhouser, and V. Ferrari, “Urban radiance fields,” in _IEEE Int. Conf. Computer Vision and Pattern Recognition_ , 2022, pp. 12 932–12 942.
  * [16] M. Bosse, P. Newman, J. Leonard, M. Soika, W. Feiten, and S. Teller, “An atlas framework for scalable mapping,” in _IEEE Intl. Conf. on Robotics and Automation (ICRA)_ , vol. 2, 2003, pp. 1899–1906 vol.2.
  * [17] B.-J. Ho, P. Sodhi, P. Teixeira, M. Hsiao, T. Kusnur, and M. Kaess, “Virtual occupancy grid map for submap-based pose graph slam and planning in 3d environments,” in _IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_ , 2018, pp. 2175–2182.
  * [18] V. Reijgwart, A. Millane, H. Oleynikova, R. Siegwart, C. Cadena, and J. Nieto, “Voxgraph: Globally consistent, volumetric mapping using signed distance function submaps,” _IEEE Robotics and Automation Letters_ , 2020\.
  * [19] Y. Wang, M. Ramezani, M. Mattamala, S. T. Digumarti, and M. Fallon, “Strategies for large scale elastic and semantic lidar reconstruction,” _Journal of Robotics and Autonomous Systems_ , vol. 155, p. 104185, 2022.
  * [20] M. Tancik, V. Casser, X. Yan, S. Pradhan, B. Mildenhall, P. P. Srinivasan, J. T. Barron, and H. Kretzschmar, “Block-nerf: Scalable large scene neural view synthesis,” in _IEEE Int. Conf. Computer Vision and Pattern Recognition_ , 2022, pp. 8248–8258.
  * [21] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Conf. on Neural Information Processing Systems (NeurIPS)_ , vol. 30, 2017.
  * [22] S. Fridovich-Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa, “Plenoxels: Radiance fields without neural networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5501–5510.
  * [23] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering,” _ACM Transactions on Graphics_ , vol. 42, no. 4, July 2023.
  * [24] A. Yu, R. Li, M. Tancik, H. Li, R. Ng, and A. Kanazawa, “PlenOctrees for real-time rendering of neural radiance fields,” in _Intl. Conf. on Computer Vision (ICCV)_ , 2021.
  * [25] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman, “Mip-NeRF 360: Unbounded anti-aliased neural radiance fields,” _IEEE Int. Conf. Computer Vision and Pattern Recognition_ , 2022.
  * [26] R. Martin-Brualla, N. Radwan, M. S. M. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duckworth, “NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections,” in _IEEE Int. Conf. Computer Vision and Pattern Recognition_ , 2021.
  * [27] Z. Li, T. Müller, A. Evans, R. H. Taylor, M. Unberath, M.-Y. Liu, and C.-H. Lin, “Neuralangelo: High-fidelity neural surface reconstruction,” in _IEEE Int. Conf. Computer Vision and Pattern Recognition_ , 2023.
  * [28] L. Yariv, J. Gu, Y. Kasten, and Y. Lipman, “Volume rendering of neural implicit surfaces,” _Conf. on Neural Information Processing Systems (NeurIPS)_ , vol. 34, pp. 4805–4815, 2021.
  * [29] P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang, “NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction,” _Conf. on Neural Information Processing Systems (NeurIPS)_ , 2021.
  * [30] X. Zhong, Y. Pan, J. Behley, and C. Stachniss, “Shine-mapping: Large-scale 3d mapping using sparse hierarchical implicit neural representations,” in _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , 2023.
  * [31] J. Deng, Q. Wu, X. Chen, S. Xia, Z. Sun, G. Liu, W. Yu, and L. Pei, “Nerf-loam: Neural implicit representation for large-scale incremental lidar odometry and mapping,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023.
  * [32] S. Lombardi, T. Simon, J. Saragih, G. Schwartz, A. Lehrmann, and Y. Sheikh, “Neural volumes: learning dynamic renderable volumes from images,” _ACM Transactions on Graphics (TOG)_ , vol. 38, no. 4, pp. 1–14, 2019.
  * [33] V. Sitzmann, J. Thies, F. Heide, M. Nießner, G. Wetzstein, and M. Zollhofer, “Deepvoxels: Learning persistent 3d feature embeddings,” in _IEEE Int. Conf. Computer Vision and Pattern Recognition_ , 2019, pp. 2437–2446.
  * [34] V. Sitzmann, M. Zollhöfer, and G. Wetzstein, “Scene representation networks: Continuous 3d-structure-aware neural scene representations,” _Conf. on Neural Information Processing Systems (NeurIPS)_ , vol. 32, 2019\.
  * [35] N. Max, “Optical models for direct volume rendering,” _IEEE Trans. on Visualization and Computer Graphics_ , vol. 1, no. 2, pp. 99–108, 1995.
  * [36] J. T. Kajiya and B. P. Von Herzen, “Ray tracing volume densities,” _SIGGRAPH_ , vol. 18, no. 3, pp. 165–174, 1984.
  * [37] Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick, “Detectron2,” https://github.com/facebookresearch/detectron2, 2019.
  * [38] D. Azinović, R. Martin-Brualla, D. B. Goldman, M. Nießner, and J. Thies, “Neural rgb-d surface reconstruction,” in _IEEE Int. Conf. Computer Vision and Pattern Recognition_ , 2022, pp. 6290–6301.
  * [39] U. Von Luxburg, “A tutorial on spectral clustering,” _Statistics and Computing_ , vol. 17, pp. 395–416, 2007.
  * [40] P. Furgale, J. Rehder, and R. Siegwart, “Unified temporal and spatial calibration for multi-sensor systems,” in _IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_. IEEE, 2013, pp. 1280–1286.
  * [41] L. F. T. Fu, N. Chebrolu, and M. Fallon, “Extrinsic calibration of camera to lidar using a differentiable checkerboard model,” in _IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_ , 2023.
  * [42] H. Aanæs, R. R. Jensen, G. Vogiatzis, E. Tola, and A. B. Dahl, “Large-scale data for multiple-view stereopsis,” _International Journal of Computer Vision_ , pp. 1–16, 2016.
  * [43] Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE Transactions on Image Processing_ , vol. 13, no. 4, pp. 600–612, 2004.