# Survey on video anomaly detection in dynamic scenes with moving cameras

Runyu Jiao, Yi Wan, Fabio Poiesi, Yiming Wang  Runyu Jiao and Yi Wan are with
Key Laboratory of High Efficiency and Clean Manufacturing, School of
Mechanical Engineering, Shandong University, Jinan, China. E-mails:
<EMAIL_ADDRESS>Fabio Poiesi and Yiming Wang are with
Fondazione Bruno Kessler, Trento, Italy. E-mails<EMAIL_ADDRESS>This work was supported by the National Natural Science Foundation of China
(No. 51975336), Key Technology R&D Program of Shandong (No. 2020JMRH0202, No.
2022CXGC020701), Major Industrial Research Projects in Shandong Province for
the Transition to New from Old Economic Engines (2021-13). We also acknowledge
the support of the PNRR ICSC National Research Center for High Performance
Computing, Big Data and Quantum Computing (CN00000013), under the NRRP MUR
program funded by the NextGenerationEU. This work was supported by the PNRR
project FAIR - Future AI Research (PE00000013), under the NRRP MUR program
funded by the NextGenerationEU.

###### Abstract

The increasing popularity of compact and inexpensive cameras, e.g. dash
cameras, body cameras, and cameras equipped on robots, has sparked a growing
interest in detecting anomalies within dynamic scenes recorded by moving
cameras. However, existing reviews primarily concentrate on Video Anomaly
Detection (VAD) methods assuming static cameras. The VAD literature with
moving cameras remains fragmented, lacking comprehensive reviews to date. To
address this gap, we endeavor to present the first comprehensive survey on
Moving Camera Video Anomaly Detection (MC-VAD). We delve into the research
papers related to MC-VAD, critically assessing their limitations and
highlighting associated challenges. Our exploration encompasses three
application domains: security, urban transportation, and marine environments,
which in turn cover six specific tasks. We compile an extensive list of 25
publicly-available datasets spanning four distinct environments: underwater,
water surface, ground, and aerial. We summarize the types of anomalies these
datasets correspond to or contain, and present five main categories of
approaches for detecting such anomalies. Lastly, we identify future research
directions and discuss novel contributions that could advance the field of MC-
VAD. With this survey, we aim to offer a valuable reference for researchers
and practitioners striving to develop and advance state-of-the-art MC-VAD
methods.

###### Index Terms:

Video anomaly detection, Moving cameras, Dynamic scenes, Datasets.

##  1 Introduction

Video anomaly detection (VAD) aims to locate unusual activities or behaviors
from normal or expected ones, within a video frame as well as in the sequence
of video frames. Detecting anomalies in the real world is a complex and
challenging task. One challenge is that there exists no clear and unified
definition of anomalies of interest. The definition greatly depends on the
application context, where the normal activities in one context may be deemed
abnormal in another context [1]. For example, the circulation of cars and
bikes on roads is normal [2], while it would be abnormal in pedestrian zones
[3]. Activities like eating, drinking, or exercising are abnormal in an office
setting but may be entirely normal in others [4]. Another challenge is due to
the rare and sporadic occurrences of such anomalies [5]. This makes the
collection of large and well-curated anomaly videos a very demanding task,
thus limiting the learning of anomalous patterns [6]. Existing works often
formulate VAD as an out-of-distribution detection or one-class classification
problem, and address it by modeling the normal patterns using a large amount
of normal videos (i.e. the unsupervised setting) [7, 8, 9, 10, 11, 12, 13,
14], together with a limited amount of abnormal videos with either video-level
supervision (i.e. the semi-supervised setting) [15, 16, 17, 18, 19, 20, 21,
22, 23, 24], or frame-level supervision (i.e. the supervised setting) thanks
to synthetic data generation [25]. Significant progress has been made in Video
Anomaly Detection (VAD) for static cameras (SC-VAD) within the application
domain of video surveillance. Benchmark datasets, including UCSD Anomaly
Detection [3], ShanghaiTech Campus [9], and CUHK Avenue [26], encompass videos
captured in urban environments that present a range of anomalies. These
datasets pose challenges under different supervision settings, as discussed
earlier, while also accounting for minor camera motion resulting from pan-
tilt-zoom operations. There exist large datasets that feature anomalies in
scenes captured in diverse environments, such as UCF-Crime that is composed of
long-hour real-world surveillance videos with 13 anomaly types [6], and
UBnormal that is composed of synthetically-generated videos with 22 anomaly
types [25]. Surveys on SC-VAD have been conducted, categorizing the methods
into two main areas: single-scene anomaly detection, and multi-scene anomaly
detection [1, 27]. Single-scene methods focus on detecting anomalies within
videos where the scene remains constant throughout [3, 28, 26, 29, 30, 31].
Multi-scene methods aim to detect anomalies across different scenes by
utilizing videos captured from multiple static cameras [6, 9].

Figure 1: Difference between (a) static camera applications and (b) moving
camera applications. We review papers that tackle video anomaly detection in
videos captured from moving cameras. Unlike static camera algorithms, moving
camera algorithms have to handle situations where both background and
foreground dynamically change over time.

Unlike previously mentioned surveys on SC-VAD, we focus on presenting the
first comprehensive survey on video anomaly detection in dynamic scenes
captured by _moving cameras_ , which we refer to as Moving Camera Video
Anomaly Detection (MC-VAD) (Fig. 1). By moving cameras, we mean cameras that
are mounted on mobile platforms, such as unmanned aerial vehicles (UAVs) [32,
33, 34, 35], ground robots [36, 37], road vehicles [38, 39, 40, 41], water
surface vehicles [42, 43], or underwater vehicles [44, 45]. As the types of
moving cameras expose a large diversity, MC-VAD naturally spans a much larger
number of application domains as compared to SC-VAD. In this survey, we
identify three broad application domains including _security_ , _urban
transportation_ , and _marine environments_ , covering various anomalies, such
as traffic accidents [2, 46, 47, 48, 49, 50], criminal activities [51, 52], or
hazardous actions/objects [53, 54].

In MC-VAD, the camera is no longer a neutral “observer” (as in SC-AVD), but it
can be a potential anomaly “participator”. Such enrichment in the camera role
introduces a new range of anomalies that do not exist in SC-VAD. For example,
the vehicle that equips the camera can participate in car accidents either
actively (i.e. the responsible) or passively (i.e. the involved) [50].
Moreover, any objects or contexts that might cause abrupt motion deviation of
the equipping platforms, such as drones and unmanned vehicles should also be
considered anomalies, as they pose a potential threat to the mobile platforms
or other traffic participants. Such new anomaly definition leads to the
inclusion of works related to obstacle detection for safe autonomous
navigation, such as early collision detection for mobile robots [55], obstacle
detection for unmanned surface vehicles (USVs) [56, 57], and detection of
hazardous environments for mobile robots [58]. In total, we identify six
specific application tasks relevant to MC-VAD, including _static object
detection_ , _dynamic objects and behaviors analysis_ , _traffic accident
detection_ , _obstacle detection_ , _environmental surveillance_ , and
_underwater exploration_.

In the following, we first discuss surveys related to the topic of VAD in Sec.
1.1, followed by Sec. 1.2 with the statistics of the research papers covered
in this survey in terms of the method for literature searching, and their
distribution over publication venues and publication years. Lastly, Sec. 1.3
motivates the need for this survey.

###  1.1 Related surveys

Because there are no specific surveys about MC-VAD in the literature, we cover
recent surveys on the topic of VAD [59, 60, 1, 61, 27], as well as related
surveys covering topics about abnormal human behavior understanding [62, 63,
31, 64] and hazard detection in autonomous driving [65]. Tab. I summarizes the
surveys covered in this section.

TABLE I: List of recent surveys related to multi-camera video anomaly detection approaches. The categories are arranged based on the scope and relevance within each domain. Within each category, the surveys are sorted chronologically. Survey | Category | Year |  Content
---|---|---|---
[60] | Anomaly detection | 2021 |  Deep learning for anomaly detection, covering key intuitions, objective functions, underlying assumptions, challenges, and future opportunities
[27] | Multimedia anomaly detection | 2021 |  Multimedia datasets for anomaly detection, covering a variety of video, audio, and audio-visual datasets
[59] | Video anomaly detection | 2021 |  Deep learning for video anomaly detection, discussing future opportunities in various domains and challenges
[1] | Video anomaly detection | 2022 |  Single-scene video anomaly detection in surveillance, focusing on problem formulations, datasets, evaluation criteria, and past research
[61] | Video anomaly detection | 2022 |  Anomaly detection in surveillance videos, including a thematic taxonomy of deep models, review and performance analysis
[66] | Video anomaly detection | 2023 |  Generalized video anomaly event detection with systematic taxonomy and comparison of deep models
[67] | Violence detection | 2023 |  Vision-based violence detection in surveillance videos, covering new taxonomy, working flow of the methods, datasets, challenges and future directions
[68] | Violence detection | 2023 |  Violence detection in surveillance videos, including achievements, datasets, performance evaluation, challenges, and future directions
[62] | Human behavior recognition | 2012 |  Video-based scene contextual understanding and abnormal human behavior detection in surveillance applications
[69] | Human behavior analysis | 2014 |  Human behavior analysis in surveillance videos, covering related background modeling, foreground detection, object classification, motion tracking, and behavior analysis methods
[31] | Human interactions recognition | 2015 |  Human interactions prediction and recognition in public spaces, covering taxonomy and extensive survey of methods
[70] | Human activity recognition | 2020 |  Non-intrusive human activity recognition and abnormal behavior detection on elderly people
[64] | Human action recognition | 2021 |  Video-based Human Action Recognition (HAR), including machine learning and deep learning techniques for HAR, datasets, and applications
[71] | Traffic anomaly detection | 2020 |  Anomaly detection in road traffic using visual surveillance, covering problem statements, learning methods, datasets, and applications
[72] | Autonomous driving | 2021 |  Corner cases for visual perception in automated driving, including systematic classification and detection methods
[65] | Autonomous driving | 2023 |  Graph-based hazardous event detection methods for autonomous driving
[73] | Aerial surveillance | 2022 |  Human-centric aerial surveillance tasks such as detection, tracking, identification, and action recognition using drones, UAVs, and other airborne platforms

VAD-related surveys focus on describing applications [59], categorizing
methods [60, 1, 61], or organizing publicly available datasets [27]. The
survey by Ren et al. [59] reviews challenges and application opportunities in
the domains of intelligent transportation, digital education, smart home,
public health, and digital twins. While intelligent transportation is an
important application area for MC-VAD, this survey lacks details in many
aspects including datasets, approaches, and challenges that are specific to
each application. The survey by Pang et al. [60] focuses on reviewing deep
learning-based VAD methods, formulating a comprehensive taxonomy and
discussing key intuitions, objective functions, underlying assumptions,
advantages, and disadvantages. The survey by Chandrakala et al. [61] is
structured similarly to that of [60], with an updated coverage on more recent
deep learning-based VAD methods. The survey by Ramachandra et al. [1] first
assesses the differences between anomalies captured in single-scene and multi-
scene setups, and then focuses on the research progress of single-scene VAD
methods. This survey also includes a comprehensive quantitative experimental
comparison of several VAD algorithms on popular benchmarks. Lastly, the survey
by Kumari et al. [27] focuses on multimedia datasets for the study of VAD.
Datasets are divided into video anomaly datasets, audio anomaly datasets, and
audio-visual anomaly datasets. For video anomaly datasets, the authors listed
38 publicly available video datasets published between 2007 and 2021. While
this list is comprehensive for the SC-VAD problem, only two datasets involve
moving cameras. In general, the above-mentioned surveys are primarily for VAD
with static cameras, there is no dedicated analysis of specific challenges for
VAD with moving cameras.

We also consider the literature about human activity recognition (HAR), or
human abnormal behavior detection, as relevant to VAD, because human
activities are often present and analyzed for anomaly detection. For example,
the survey by Popola and Wang [62] focuses on video-based abnormal human
behavior recognition. This survey starts with an analysis of human motion and
then discusses the contextual characteristics of anomalies, scene modeling,
and behavior abstraction and representation. It also argues that the
definition of anomaly can be often ambiguous, which is similar to the context
of MC-VAD. The survey by Poiesi and Cavallaro [31] reviews methods for the
analysis of human interactions in public places, with particular focus on the
prediction of rendezvous areas where people are likely to interact with each
other or with static objects, e.g. a door or a meeting point. Such methods can
be relevant for analyzing anomalies in crowded scenes. The survey by Pareek
and Thakkar [64] discusses a general framework for action recognition tasks
comprising of feature extraction, feature encoding, dimensionality reduction,
and action classification. None of these HAR surveys specifically address the
VAD problem from moving cameras. The survey by Xiao et al. [65] provides a
comprehensive analysis of hazardous event analysis for autonomous driving
scenarios. In particular, it provides an overview of graph-based methods, and
summarizes the related datasets and evaluation metrics. However, their
discussion does not cover the VAD problem of moving cameras.

To the best of our knowledge, ours is the first survey on the topic of MC-VAD.
We define the concept of MC-VAD, which builds upon the fundamental principles
of other surveys regarding VAD, and discuss the challenges posed by camera
motion. Our survey provides an overview of research progress regarding MC-VAD
ranging from 2001 to 2023, covering six tasks across three broad application
domains. We provide an organized summary of 25 publicly available datasets
collected from moving cameras for the study of MC-VAD. We also organize
related methods addressing MC-VAD into five categories with a discussion of
their respective advantages and limitations.

Figure 2: Overview of MC-VAD publication statistics. (a) Publication sources
can be divided into six categories. Vision conferences: CVPR (Computer Vision
and Pattern Recognition), ECCV (European Conference on Computer Vision), ICCV
(International Conference of Computer Vision), WACV (Winter Conference on
Applications of Computer Vision), ICIP (International Conference of Image
Processing), CVPRW (Computer Vision and Pattern Recognition Workshops). Vision
journals: TPAMI (Transactions on Pattern Analysis and Machine Intelligence),
TIP (Transactions on Image Processing), TCSVT (Transactions on Circuits and
Systems for Video Technology), PR (Pattern Recognition). Robotics conferences
and journals: IROS (international conference on Intelligent Robots and
Systems), ICRA (International Conference on Robotics and Automation), RAL
(Robotics and Automation Letters). Transportation and vehicle conferences and
journals: TITS (Transactions on Intelligent Transportation Systems), IV
(Intelligent Vehicles Symposium), TIV (Transactions on Intelligent Vehicles).
(b) Histogram of discussed papers over the publication year.

###  1.2 Literature statistics

We conducted an extensive search for papers within the context of MC-VAD using
Google Search, Google Scholar, Web of Science, and ScienceDirect platforms. We
used Papers With Code, Google Dataset Search, and Kaggle platforms to collect
relevant datasets. Given the absence of a clear definition of MC-VAD
literature, and considering the plethora of related topics that often use
different terminology, such as hazard detection, violence detection, obstacle
detection, anomaly recognition, among others, it can be challenging for
traditional database search strategies to capture all relevant content.
Therefore, we performed multiple rounds of searches using different keywords,
aiming to filter and select works closely aligned with MC-VAD based on our
definition. We also used citations of each paper as a way to find research
works advancing the underlying literature. Lastly, we counted the academic
papers cited in this survey after an iterative screening process following the
practice in the surveys of [67, 68].

Fig. 2 provides an overview of the MC-VAD publication statistics, reporting
the publication sources and temporal distributions of the research works.
Similarly to [74], these statistics are compiled based on their publication
venues. Fig. 2a shows that most works related to MC-VAD have been published in
computer vision and image processing conferences such as CVPR, ICCV, ECCV,
WACV, ICIP, and their associated workshops. We also found contributions in
image processing and pattern recognition journals, such as TPAMI, TIP, TCSVT,
and PR. Because MC-VAD involves mobile robotic platforms, and finds wide
applications in the fields of robotics and intelligent transportation, a
portion of the research work also comes from robotics conferences and journals
such as IROS, ICRA, and RAL, as well as conferences and journals in the smart
transportation and vehicle research domain such as IEEE TITS, IV, and TIV.

Fig. 2b shows the number of selected papers per year from 2001 to 2023. VAD
research using mobile cameras can be traced back as early as 2001. However,
due to the lack of large-scale datasets and effective data-driven techniques,
only a few studies were conducted in this area prior to 2012. The growing
adoption of moving cameras has boosted the collection of large-scale datasets,
which in turn has sped up the development of MC-VAD over the past decade. Yet,
we also observe a decrease in publications between 2020 and 2023, the reasons
for which are challenging to pinpoint. One possibility could be a shift in
research interest, likely influenced by various factors such as new emerging
technologies or funding priorities due to urging societal need, e.g. the
pandemic. Also, the increasing concern on data privacy and AI ethics may also
slow down the research progress of MC-VAD as the relevant datasets and method
development requires more regulatory and ethical considerations.

Fig. 3 presents the number of publicly available datasets per task across
three application domains. Within the security domain, public datasets for
static object detection are few. This scarcity is likely due to the niche
nature of scenarios typically encountered in this domain, such as indoor
industrial settings, which requires ad-hoc methods for effective resolution.
In contrast, datasets for studying the detection of dynamic objects and
behaviors are more prevalent, likely attributed to the growing interest in
research about to human behavior understanding. Within the urban
transportation domain, there are abundant datasets concentrating on obstacle
detection and traffic accident detection. This trend can be primarily linked
to the surging interest in autonomous driving and mobile robot navigation.
Lastly, within the marine domain, the growing interest in the development of
techniques for obstacle detection, environmental monitoring, and underwater
exploration has led to the creation of relevant datasets.

###  1.3 Motivation

Our analysis of the related literature shows that there is a growing interest
in the problem of VAD with moving cameras. However, no comprehensive
literature organization exists due to the high diversity of the camera-
equipped platforms and the application domains. We are thus motivated to
compile this survey in order to bridge this gap and provide insights to both
interested readers and domain experts. Our survey begins by delineating a
comprehensive taxonomy, reflecting the current research status quo in the
realm of VAD with moving cameras. We proceed by identifying potential
application domains and discussing the unique technical challenges each domain
presents. We then present and categorize an extensive list of datasets based
on the mobile platforms equipped with cameras. These datasets are either
specifically designed for or have the potential to be utilized for MC-VAD
research. We also classify existing methods according to the nature of the
anomalies they address, followed by a detailed discourse on their respective
advantages and limitations. Lastly, we conclude this survey by offering
recommendations for future research directions. In summary, the main
contributions of our survey are:

  * •

Problem definition and new challenges. We define the problem and scope of MC-
VAD and we identify a thorough list of novel challenges introduced by the use
of moving cameras.

  * •

Comprehensive literature review. We review a significant number of research
studies from leading conferences and journals in the fields of computer
vision, artificial intelligence, and robotics. Such comprehensive coverage
presents a complete picture of the research progress regarding MC-VAD in the
relevant application domains.

  * •

Thorough dataset summary. We identify and summarize 25 publicly available VAD
datasets captured from moving cameras that are suitable for the study of MC-
VAD. We provide detailed descriptions for these datasets, covering aspects
such as the application context, number of videos/anomalies, typical
anomalies, participants, and their advantages and disadvantages. We also
provide their respective download links to directly access them.

  * •

Categorized anomaly and methods. We categorize the anomalies present in MC-VAD
into five distinct categories and present a taxonomy to classify methods
addressing MC-VAD accordingly.

  * •

Insightful outlook. We outline the limitations of existing work and discuss
promising future directions to tackle MC-VAD-related challenges.

Fig. 4 shows the overall paper organization. Sec. 2 discusses the new
challenges that camera motion poses for VAD. Sec. 3 describes the primary
applications of MC-VAD. Sec. 4 details the publicly available datasets for
studying MC-VAD, with relevance to the applications discussed in Sec. 3. In
Sec. 5, we first classify the types of anomalies present in the dataset and
then present a taxonomy for categorizing various MC-VAD approaches. In Sec. 6,
we discuss the limitations of existing methods and delve into potential future
research directions. Lastly, in Sec. 7, we draw our conclusions.

Figure 3: Distribution of public MC-VAD datasets (25 in total) over six tasks
across three application domains: static object detection, dynamic objects and
behaviors, traffic accident detection, obstacle detection, environment
surveillance and underwater exploration.  Figure 4:  Survey overview. Sec. 1
introduces and defines the Video Anomaly Detection (VAD) problem, and relates
the study of Moving Camera VAD (MC-VAD) with respect to Static Camera VAD (SC-
VAD). We position our survey among related surveys and compile literature
statistics. Sec. 2 lists MC-VAD-related challenges. Sec. 3 presents three
application areas of MC-VAD, i.e. security, urban transportation, and marine
environment, as well as six specific tasks across different domains. Sec. 4
presents 25 related datasets, spanning four environments including underwater,
water surface, ground, and aerial. Sec. 5 classifies the anomalies contained
in the datasets into two levels, i.e. scene level and scenario level, and five
types of anomalies. We further categorize MC-VAD approaches into five classes.
Sec. 6 presents limitations of existing works and future directions. Sec. 7
concludes our survey.

##  2 MC-VAD challenges

Recent deep learning-based methods have shown compelling results on public SC-
VAD datasets, such as ShanghaiTech [9]. However, MC-VAD still remains a
challenging problem due to the presence of camera ego-motion. The broader
range of applications makes generalized solutions more challenging to emerge.
In this section, we discuss MC-VAD challenges in detail.

Ego-motion. The optical flow captured by moving cameras entangles both the
motion of scene objects and the camera ego-motion. This makes the task of
detecting anomalies based on abnormal motion patterns more challenging. To
address this issue, camera motion can be estimated and compensated for [55].
Early works focused on studying MC-VAD consider simple camera motions, such as
a camera steadily moving along a fixed trajectory [75, 54]. Such limited
motion can exploit planar homography to account for the camera motion.
However, it is still challenging to handle scenarios with unconstrained camera
motions, such as high speeds, and sudden direction variations [76, 77, 78].

Figure 5:  Timeline of relevant research works covering six main categories of
tasks in MC-VAD, from 2001 to 2023. Entries in bold highlight the datasets
that were proposed for each application. Early works were predominantly in the
security sector, focusing on both static object detection and dynamic object
and behavior analysis, a trend that also continues today. MC-VAD has later
emerged in the field of urban transportation, specifically in traffic accident
detection and obstacle detection, since 2016, with a noticeable uprise around
2019. Lastly, the applications of MC-VAD in the field of marine environments
started in 2017 and have recently gained more research attention.

Changing background. Numerous studies have shown evidence that the majority of
VAD algorithms heavily rely on background cues [79, 80]. This makes MC-VAD
with changing background a challenging task due to the dynamics in the
background and camera ego-motion. Methods that rely on a static background [4,
81, 82] are highly likely to fail. Thus, how to limit the impact from the
background remains one of the challenges in the study of MC-VAD.

Presence of abrupt motion. Abrupt changes in the camera ego-motion in terms of
speed and direction present a challenge that is unique to MC-VAD. Even in
fixed surveillance datasets such as UCF-Crime [6], the presence of camera
shake or switch in videos can make most existing SC-VAD methods based on frame
prediction or frame reconstruction ineffective. This situation becomes more
prevalent in videos captured by moving cameras, where abrupt motion can occur
in many unpredictable manners, being normal or abnormal. While ad-hoc
solutions exist in the literature to address particular scenarios, it is
challenging to obtain generalizable solutions to handle such abrupt camera
motion.

First-person perspective. The shift in viewpoint from a third-person
perspective, as in SC-VAD, to a first-person perspective as in MC-VAD
introduces new challenges to MC-VAD, in particular for the patterns of
occlusions. In SC-VAD, surveillance cameras are typically installed in
locations with optimal views for better coverage of the scene. Instead, in the
case of first-person view from a moving camera, occlusions occur more
frequently and severely as compared to the third-person view in the case of
SC-VAD, making the video analysis in MC-VAD more challenging.

Dynamic context priors. There can be rich context priors exploited for VAD,
such as jaywalking, riding a bike on a pedestrian sidewalk, or driving a car
in the wrong direction [1]. Such context prior can be easily extracted in SC-
VAD as the scene has a static and a known structure. However, in MC-VAD, the
scene is continuously changing due to camera motion, thus requiring methods
that are capable of understanding scene semantics regardless of the viewpoint.

Varying anomaly definition. As the scene context changes with camera motion,
the definition of normality and anomaly also changes accordingly. For
instance, for a mobile robot that patrols an office area, actions such as
eating food or holding a knife might be recognized as anomalies, while such
actions in a kitchen would be deemed as normal activities. Such varying
anomaly definition requires context-aware modeling in order to account for the
context changes in dynamic scenes.

Low-quality data. It is also noticeable that current datasets for the study of
MC-VAD have limited data quality. A large proportion of MC-VAD datasets are
acquired through crowdsourcing [83, 2, 6, 47, 48, 53, 49, 50], and the quality
is limited by factors such as camera hardware, installation conditions,
compression techniques, camera motion, and external environmental conditions,
leading to low resolution, blurriness, and sometimes significant noises. Such
datasets increase the difficulty of detecting anomaly detection, hence
multimodal techniques might be helpful to address such challenge [42, 39, 84,
35, 85].

##  3 Applications

Figure 6: Mobile camera platform for abnormal object detection in industrial
scenes [75]. It captures both reference videos featuring normal scenes and
target videos featuring abnormal scenes with the presence of unusual objects,
such as shoes, cups, and bags (highlighted in red boxes).

MC-VAD is of importance to several application domains, including security,
urban transportation, and marine environments. Among these domains, we
identify six application tasks. For security, the tasks are static object
detection, and dynamic object and behavior analysis. For urban transportation,
the tasks are traffic anomaly detection and obstacle detection. For marine
environments, the tasks are obstacle detection, environmental surveillance,
and underwater exploration. In Fig. 5, we present a timeline of relevant
research works that were published between 2001 and 2023. In bold we highlight
datasets that were introduced along with their published work. We can observe
that methods published before 2016 are predominantly in the security sector,
focusing on both static object detection, and dynamic object and behavior
analysis. Since 2016, works in the field of urban transportation have emerged
with a noticeable uprise starting from 2019. The application of MC-VAD in the
field of marine environments started in 2017 and has gradually garnered more
attention in recent years. In the following sections, we dive into each
application domain with thorough coverage of each detailed task.

###  3.1 Security

Early research on VAD within the field of security started with the detection
of abnormal events in videos captured by static surveillance cameras. With
advancements in mobile robot technologies, the interest in video monitoring
via patrol or surveillance robots has increased. Anomalies within the security
domain can be about static objects, such as suspiciously abandoned objects in
public spaces, or dynamic behaviors, such as people fighting.

####  3.1.1 Static object detection

Works that focus on the detection of static objects can model anomalies as the
unusual presence of objects, like abandoned objects [86, 87], illegally parked
vehicles [88], road obstructions [54, 89], or unexpected objects in industrial
scenes [75].

Abandoned objects can be detected in urban scenarios and in industrial
environments using the systems proposed in [54] and [75], respectively. Both
systems employ wheeled vehicles equipped with cameras. Because abandoned
object can have different shapes, colors and textures, object detection
algorithms trained on a finite set of classes may fail. Therefore, frame-based
approaches are preferred. A reference video can be recorded when no objects
are present in the scene. Then, given a target video, anomalous objects can be
detected by matching the frames of the reference video with those of the
target video. The work by Silva et al. [75] introduces a dataset composed of
reference and target videos of a cluttered industrial environment. This
dataset was captured by using various cameras under different lighting
conditions, containing different anomalous objects, such as backpacks,
umbrellas, and bottle caps (Fig. 6). This dataset has been used as a benchmark
by several reference video approaches [90, 91, 92]. These types of approaches
are typically designed for cases where the vehicles navigate along a fixed
path with a highly controlled motion. Hence, their application is limited to
environments with minor structural scene variations and they may fail if new
normal objects are added to the scene. Anomalous static objects can also be
detected with a mobile robot in industrial environments by building a database
of normal frames along with their geographic tags, and by comparing these
normal frames with frames captured during the robot navigation [89]. The
system includes a human-in-the-loop module, where if the anomaly detection
model predicts an incoming frame as an anomaly, the operator will assess
whether the alert is a genuine one. If the operator deems the frame as normal,
this will be added to the database of normal frames, thus making the database
incrementally richer over time. However, this system is bound to the human-in-
loop input, which can render the system impractical to apply to large-scale
applications. The works by Lawson et al. [93, 94] focus on the detection of
anomalous objects using autonomous robots within known office environments.
The anomaly detection model is configured to learn normal conditions in an
unsupervised manner. Their proposed solution employs a Generative Adversarial
Network to compare the robot’s current views with the learned model of
normality to detect anomalies. However, if objects are moved from one place to
another, due to their specific affordance, this may trigger false anomaly
alarms.

Figure 7: Guiabot robot (shown on the left) is used to generate action region
proposals in images and detect abnormal behaviors in the campus environment,
including eating, drinking, and doing sports. Figure taken from [4].

####  3.1.2 Dynamic object and behavior analysis

Dynamic abnormal behavior detection has consistently been a primary research
focus in VAD [3, 26, 9, 25], as human is one of the main concerning
contributors to anomalies. Several related applications have gradually emerged
in MC-VAD with the use of Unmanned Aerial Vehicles (UAVs) [95, 96, 51, 97, 98,
99] and wheeled robots [83, 4] for monitoring human behavior. The system
presented by Medioni et al. [95] features road monitoring by analyzing video
sequences captured from UAVs. The system detects and tracks road vehicles, and
models the speed of cars as Gaussian distributions, taking into account both
scene context (such as road and checkpoint areas) and task context (like
avoiding checkpoints and passing checkpoints). An alarm is triggered if the
speed of a vehicle falls outside the distribution. This system is highly
customized to a specific application scenario as it makes use of prior
knowledge in terms of the properties of dynamic objects, the scene context,
and the task context, making its application easily ineffective in the case of
scenario changes. Another application of MC-VAD with UAV-equipped cameras is
for the detection of anomalous activities in parking lots [32], such as
fighting or stealing [96, 97, 98]. These videos involve nearly-continuous
camera motion, except when the drone is in a hovering position. However, the
development of generalized data-driven solutions is challenging because the
dataset contains numerous small targets. Moreover, videos can be noisy, and
the errors they generate are likely to stem from the frame-level feature
extraction approaches used for anomaly detection [79]. The system proposed by
Singh et al. [51] features real-time detection of violent individuals in
public areas using drones. This system uses a Feature Pyramid Network [100] to
detect humans and to estimate their pose. The angles between the limbs are
utilized by a Support Vector Machine to classify individuals engaging in
violent activities, such as punching, kicking, falling, strangling, throwing,
swinging pipe, pushing, and SOS signals [99]. Face detection and
identification techniques can be further integrated into the system to
identify violent individuals [101]. However, dealing with individual
identities requires careful ethical considerations to be compliant with
privacy-related regulations. There also exist mobile robot systems to detect
actions like eating, drinking, and sports activities [4]. The system can
operate by first classifying the scene, then recognizing the action, and
finally calculating the occurrence probability of certain actions in a scene
(Fig. 7). The system can then be tested on both static and moving camera
datasets. When the system has to process unseen actions or actions that
exhibit a high degree of similarity to others, the accuracy of anomaly
detection can significantly decrease.

###  3.2 Urban transportation

In the context of urban transportation, autonomous driving applications are
most prevalent. These involve detecting long-tailed unexpected activities,
such as animals darting across the road or confronting oncoming drivers in the
wrong lane. It is not only crucial to detect these activities reliably, but
also to do so in a timely manner.

####  3.2.1 Traffic accident detection

Training a model to anticipate accidents from dashcam videos can facilitate
the generation of timely warnings, potentially a few seconds prior to the
occurrence of and accident [2]. Dashcams are affordable cameras that can be
mounted inside a vehicle and that can record street-level visual observations
from the driver’s point of view. In recent years, dashcams have become nearly
ubiquitous in cars, making them among the most widely deployed mobile cameras.
A substantial number of dashcam videos, including those involving accidents,
have been shared on websites such as YouTube, and several researchers have
leveraged this wealth of data to build traffic accident datasets for their
studies [47, 50, 2, 53, 49, 48, 102, 103]. Traffic anomalies can be
categorized as collision or loss of control [50]. Each of these can be further
divided into non-ego and ego-involved anomalies. Non-ego anomalies involve
situations where the ego-vehicle is just an “observer” of the anomaly, hence
capturing a collision between two vehicles at a distance. Ego-involved
anomalies are situations where the ego-vehicle is directly involved, such as
in collision with other vehicles, in which the ego-vehicle is a “participant”
in the anomaly. This classification broadens the scope of anomaly detection
tasks, and extends anomaly detection to video action recognition and online
action detection. Other anomalous traffic patterns, such as speeding
motorcycles, halted vehicles, and close merges, are also of interest to be
detected from dashcam videos of vehicles on highways [53]. Compared to the
setting of traffic accident detection, the featured anomalous events in this
work represent more common and general cases where drivers should pay
attention to while driving. However, this work only tackles five types of
anomalies with 25 abnormal traffic videos, which is rather limited compared to
the diversity of traffic anomalies that one can find in the real world. Driver
attention prediction can be integrated with the information extracted from
dashcam accident videos, in order to better understand the relationship
between driver attention and accident prediction [48]. The annotation of the
driver attention can be added to dashcam accident videos to facilitate the
training of the prediction models for driver attention. Such study assists in
the development of more precise and effective driver attention monitoring
devices, e.g. eye-tracking technology and driver fatigue detection systems.
However, as it is difficult to obtain real driver attention data during real
accidents, the attention annotation is often collected from volunteers after
the incident, which might introduce potential biases.

####  3.2.2 Obstacle detection

According to the statistics provided in [104], road debris and obstacles are
the cause of a significant number of accidents. Obstacles can lead to crashes,
hence mission failures. Hence, obstacle detection is critical in both
autonomous driving and robotic navigation. A stereo vision-based method can be
used to detect small but important road hazards for autonomous vehicles, such
as debris and lost cargos [38]. The method incorporates photometric cues,
considering patches that deviate from the expected road appearance as
potential hazards. As only photometric features are sensitive to lighting
conditions, the method may fail in detecting small obstacles in badly lit
areas or at night. A multi-modal framework that combines sparse LiDAR and
monocular vision can help mitigate the previously mentioned shortcoming in
systems based only on photometric information [40]. The accurate alignment
between LiDAR and camera is key for precise detection and can be achieved by
using an external parameter refinement method based on the Hausdorff distance.
This allows the segmentation of obstacles that are smaller than 15 centimeters
at a distance of 50 meters. In robotic navigation, humans are often considered
as dynamic obstacles. Hence, a stereo camera-based system for multi-person
tracking in crowded spaces can be used [105]. The system jointly estimates
camera position, stereo depth, object detection, and trajectories based only
on visual information, then predicts future motion for dynamic objects, and
fuses this information with a static occupancy map estimated from dense
stereo. Such controlled navigation is particularly important for robotic
platforms that assist people. For example, the work by Wang et al. [106]
features robotic wheelchairs and proposes a pipeline to automatically label
drivable areas and road anomalies with self-supervised learning. Such labels
are then used to train RGB-D semantic segmentation neural networks. Unlike
self-driving cars that primarily focus on pedestrians and cars on roadways
[107, 108], the robotic wheelchair [106] focuses on the detection of small
obstacles, such as stones and tin cans in both indoor and outdoor scenes.

###  3.3 Marine environments

Open water covers over 70% of the Earth, and over 80% of world merchandise
trade by volume is carried out by sea [109]. Traditional monitoring is
performed by human, and thus is limited due to constraints such as monitoring
range, time, and perspective. The use of robotic platforms equipped with
cameras is rapidly expanding in this field, as evidenced by recent surveys and
workshops [52, 110, 111]. Relevant applications to MC-VAD can be divided into
obstacle detection, which can be carried out by Unmanned Surface Vehicles
(USVs), environment surveillance, which can be carried out from Unmanned
Aerial Vehicles (UAVs), and underwater exploration, which can be carried out
by Unmanned Underwater Vehicles (UUVs).

####  3.3.1 Obstacle detection

USVs are frequently employed robotic platforms designed to navigate across
oceanic environments for environmental monitoring or water rescue. USVs
generally function by autonomously navigating along pre-set waypoints or pre-
programmed paths, thus detecting and avoiding obstacles is key to ensure safe
operations. Recently, dynamic camera-based visual methods have been explored
for obstacle detection in marine environment, including semantic segmentation
[43], object detection and tracking [42]. While these techniques facilitate
more accurate and efficient obstacle identification, most of such works have
not considered detection in nighttime scenarios, which is equally important
for long-duration navigation of USVs.

####  3.3.2 Environment surveillance

Maritime surveillance is key for both law enforcement and environmental
protection. The implementation of an intelligent maritime surveillance system
using UAVs equipped with cameras, can enhance the awareness of maritime
incidents, effectively replacing the human effort required for extensive and
long-term ocean monitoring. UAVs can aid in detecting oil spills and hazardous
substances, tracking marine objects like vessels and debris, and recognizing
behavioral patterns, such as vessel rendezvous and high-speed vessels [34].
The versatility and swift operational capability of UAVs can also be leveraged
for search and rescue missions [35]. The primary targets for detection in
these scenarios are swimmers, floating objects, and vessels in open waters.
Currently, most of these works only focus on basic detection, tracking, and
segmentation. However, we believe the utilization of contextual information,
e.g. scene context or temporal context, can also be important for future
research in maritime surveillance.

Figure 8:  MC-VAD dataset taxonomy. We categorize datasets into four types of
environments: underwater, water surface, ground, and aerial. (top) Typical
mobile camera platforms that are used for capturing these datasets, i.e.
unmanned underwater vehicle, unmanned surface vehicle, unmanned ground
vehicle, and unmanned aerial vehicle, respectively. (bottom) Different
environments correspond to different tasks or detection scenes.

####  3.3.3 Underwater exploration

The application of MC-VAD for underwater exploration primarily focuses on the
detection of novel objects. This exploration task is challenging due to the
low visibility and unpredictable nature of underwater environments. We observe
that research in this area is currently limited, with few dedicated datasets
available for the development and evaluation. The work by Zhou et al. [45]
represents a promising attempt in this task. It aims to visually locate
unknown objects, such as underwater sculptures, to facilitate autonomous
exploration in underwater environments. The study utilizes both simulated and
real video sequences where the sandy seafloor, rocks, and oscillating
seagrasses are treated as normal data. In contrast, anomalies or targets are
represented by objects such as shipwrecks, statues, and ship anchors.
Moreover, there are many more specific applications in underwater exploration,
such as underwater garbage cleaning, observation of marine flora and fauna
[112], and underwater robot harvesting in marine farms [44]. However, the
significant variations in water quality and visibility in different marine
regions and depths pose challenges to the utilization and development of
relevant applications.

##  4 Datasets

We review publicly available RGB and RGB-D datasets that can serve for the
development and evaluation of MC-VAD methods. Specifically, we review 25
datasets captured by moving cameras, offering a rich variety of ego-motion and
complex background changes for anomaly detection in different scenarios. As
shown in Fig. 8, we categorize these datasets based on the environment in
which the mobile platform is situated, and the application scenario or tasks
they feature. In addition, we compile various attributes of these datasets in
Tab. II, such as background, number of videos, number of anomalies, types of
sensors, resolution, types of anomalies or participants, occlusion, and
whether the anomalies are acted. Tab. III presents an overview of the
advantages and drawbacks of these datasets.

TABLE II: Summary of MC-VAD datasets and their properties. Key. *: dataset contains images instead of videos, I: Images, T: Type, U: Unspecified, Occ: Occlusion, Act: Acted, EO: Electro-Optics sensor, IR: Infrared Radiation, NV: Night-vision, NI: Near Infrared. The dataset name is clickable and links to the dataset webpage. Dataset | Background | Number | Number of | Modality | Resolution | Typical of anomalies | Occ | Act
---|---|---|---|---|---|---|---|---
|  | of videos | anomalies |  |  | or participants |  |
|  | /images |  |  |  |  |  |
SUIM* [113] | Dynamic | - | - | RGB | Multi | underwater obstacle | $\checkmark$ |
TrashCan* [112] | Static | 7212 I | 7212 I | RGB | Multi | underwater trash and fish |  |
UUD* [44] | Static | - | - | RGB | Multi | seacucumber, seaurchin | $\checkmark$ |
IPATCH [52] | Dynamic | 24 | 17 | RGB, thermal | Multi | pirate surveillance |  | $\checkmark$
SMD[42] | Dynamic | 11 | 11 | RGB, EO | 1920$\times$1080 | vessels | $\checkmark$ |
MODS [43] | Dynamic | 94 | 94 | RGB (stereo) | 1278$\times$958 | marine obstacle | $\checkmark$ |
DAD [2] | Dynamic | 1730 | 620 | RGB | 1280$\times$720 | traffic accident | $\checkmark$ |
DADA-2000 [48] | Dynamic | 2000 | 2000 | RGB | 1456$\times$660 | traffic accident | $\checkmark$ |
HTA [53] | Dynamic | 389 | 25 | RGB | 1280$\times$720 | traffic anomalies | $\checkmark$ |
Car Crash Dataset [49] | Dynamic | 4500 | 1500 | RGB | U | traffic accident | $\checkmark$ |
DoTA [50] | Dynamic | 4677 | 4677 | RGB | 1280$\times$720 | traffic accident | $\checkmark$ |
Lost&Found [38] | Hybrid | 112 | 112 | RGB (stereo) | 2048$\times$1024 | small road hazards |  | $\checkmark$
FieldSAFE [39] | Hybrid | 5 | 5 | RGB (stereo), LiDAR | 1920$\times$1080 | obstacle in agriculture |  | $\checkmark$
|  |  |  | thermal |  |  |  |
Small Obstacle [40] | Static | 15 | 15 | RGB (stereo), LiDAR | 1920$\times$1080 | off-road and small-obstacle |  | $\checkmark$
GMRP* [84] | Static | 3896 I | 3896 I | RGB-D | 1280 x 720 | road anomalies |  |
Hazards&Robots [114] | Static | U | 16 T | RGB | 512$\times$512 | hazard environment |  | $\checkmark$
RoadAnomaly* [41] | Static | 60 I | 60 I | RGB | 1280 $\times$ 720 | road obstacles |  | $\checkmark$
VDAO [75] | Static | 66 | 62 | RGB | 1280$\times$720 | industrial abandoned-object | $\checkmark$ | $\checkmark$
RE-DID [83] | Dynamic | 30 | 30 | RGB | 1280$\times$720 | fighting | $\checkmark$ |
UCF-Crime [6] | Dynamic | 1900 | 1900 | RGB | Multi | abuse, arrest,accident,etc |  |
X-MAS [85] | Dynamic | 2624 | 1220 | RGB-D, IR, LiDAR | U | fallen, convulsions, etc | $\checkmark$ | $\checkmark$
|  |  |  | thermal |  |  |  |
mini-drone video [32] | Dynamic | 38 | 38 | RGB | 960$\times$540 | attacking,stealing,etc |  | $\checkmark$
UAV-Human [33] | Dynamic | 22,476 | U | RGB-D, IR, NV | 1920$\times$1080 | human behaviour |  | $\checkmark$
|  |  |  | skeleton |  |  |  |
Seagull [34] | Dynamic | 19 | 19 | RGB-D, IR, NV | Multi | vessels, hydrocarbon slick |  |
SeaDronesSee [35] | Dynamic | U | U | RGB, RedEdge, NI | Multi | swimmer, floater |  | $\checkmark$

###  4.1 By platforms under water

While existing datasets captured by underwater mobile platforms are primarily
designed for underwater image enhancement [115, 116], there are datasets
introduced for object detection or semantic segmentation, that contain
valuable information about the underwater environment and the types of objects
one might encounter (see Fig. 9a-c). These can be useful for developing and
evaluating MC-VAD methods in the context of underwater exploration [45].

The SUIM dataset [113] is a large-scale dataset that contains over 1,500
images with pixel-level annotations for eight object categories: fish, reefs,
aquatic plants, wrecks/ruins, human divers, robots, and seafloor. These images
are captured with high visibility, allowing for the clear observation of a
variety of underwater objects. Although the primary use of the SUIM dataset is
for underwater image semantic segmentation, it can also be repurposed for MC-
VAD. For example, the SUIM dataset is used in [45] to validate detection of
unknown objects with dynamic backgrounds as they were anomalies in underwater
scenarios. The algorithm employs images without sea fauna as the training set
(normal) and images with marine animals (e.g. fish, turtle) as the test set
(anomalies).

The UUD dataset [44] is an underwater (open-sea farm) object detection dataset
comprising three categories of objects, i.e. sea cucumber, sea urchin, and
scallop, which can be used as anomaly samples. UUD contains 2,227 annotated
images, which are captured by high-definition cameras carried by divers and
robots. The image resolution varies and it is 720p, 1080p, and 4K. UUD
presents typical dynamic backgrounds and motion blur, but its utility for
anomaly detection is somewhat limited by the small number of object
categories.

The TrashCan dataset [112] is an instance-segmentation dataset of underwater
trash. It consists of 7,212 annotated images and includes observations of
trash, Remotely Operated underwater Vehicles (ROVs), and a diversity of
underwater flora and fauna scenarios. Images are extracted from nearly 1K real
ROV-captured underwater videos, with the captured trash items varying
significantly in shape. However, the complex structure of the seabed and low
visibility present challenges for detection.

###  4.2 By platforms on the water surface

TABLE III: Advantages and disadvantages of MC-VAD datasets organized based on
their application domains, anomaly categories and data collection methods.
Paper citations corresponding to the dataset abbreviations are detailed in
Tab. II.

Datasets |  Advantage |  Disadvantage |  Application domain
---|---|---|---
SUIM, TrashCan, UUD |  Natural representation of underwater scenes |  Image dataset without temporal information |  Underwater exploration
IPATCH |  Five complete maritime anomaly scenarios |  Limited number of events |  Abnormal behavior in marine
SMD, MODS |  Large amount of marine scenes and obstacles |  Few types of obstacles |  Obstacle detection in marine
DAD, DADA-2000, CCD, DoTA |  Great amount of diversity in terms of real motion anomalies, a large variation in camera ego-motion, and rich annotations |  Low-resolution and noisy videos |  Traffic accident detection
HTA |  High-definition and smooth ego-motion videos |  Few types of anomalies |  Traffic anomalous behavior
Lost&Found, Small Obstacle, FieldSAFE, GMRP, RoadAnomaly |  Multi-modal data and several small obstacles of different types |  Lack of samples of diverse moving anomalous objects |  Obstacle detection on road
Hazard&Robots |  Various types of hazard events |  Few numbers of scenes |  Hazard environment detection
VDAO |  Target and reference videos with and without anomalies in the same scene |  Camera motion is simple and slow |  Abandoned object detection
RE-DID |  Real-world fights captured from moving cameras |  Small number of samples |  Human abnormal behavior
UCF-Crime |  Great number of diverse types of real-world anomalous events |  Few videos containing PTZ camera motion |  Abnormal behavior
minidrone |  Abnormal events captured from aerial moving cameras |  Few abnormal event categories and samples |  Human behavior surveillance
UAV-Human |  Large number of anomalous actions captured from aerial moving cameras |  Short and indistinct duration of actions |  Human behavior surveillance
Seagull, SeaDroneSee |  Natural marine scenes captured from aerial moving cameras |  Few object categories and scenes |  Marine surveillance

Datasets captured by USVs often encompass data of multiple modalities acquired
from cameras, radars, and Electro-Optical (EO) sensors. Several such datasets
have been proposed to test and validate algorithms designed for maritime
surveillance and robotic navigation [117, 118, 119, 120]. However, these
datasets do not directly cater to the needs of MC-VAD due to factors such as
small camera motion, limited types of detection targets, or simplistic
scenarios.

Figure 9: Examples of anomaly from videos captured (a-c) under water and (d-f)
on the water surface. (a) Divers, fish, and shipwrecks from the SUIM dataset
[113]. (b) Trash can and fish from the TrashCan dataset [112]. (c) Sea
cucumber and sea urchin from the UUD dataset [44]. (d) Large vessels, small
boats, and Ocean Buoy from the Singapore Marine Detection dataset [42]. (e)
Yacht and marine debris from the Marine Obstacle Detection dataset [43]. (f) A
boat that suddenly accelerates across a ship from the IPATCH dataset [52].

We identify a list of datasets that can serve for MC-VAD, especially in the
context of automation of maritime monitoring and navigation security, for
tasks such as detecting anomalous vessel behaviors [52] and surface obstacles
[56, 42, 57, 43] (Fig. 9d-f). These datasets feature complex scenarios and
contain a wide range of targets, making them suitable for training and testing
MC-VAD models.

The Marine Obstacle Detection Dataset (MODD1) [56] is a pioneering small-scale
dataset that consists of 12 video sequences, with 4,454 fully annotated frames
of image resolution 640$\times$480 pixels. The video sequences are recorded
from multiple unmanned platforms (mostly from small USVs) over several months,
under different weather conditions, and at different times of the day. The
dataset contains situations that pose a danger to USV, such as approaching
obstacles and collision threats. Subsequently, [57] upgraded MODD1 to MODD2
that contains 28 video sequences with 11,675 frames at a resolution of
1278$\times$958 pixels. These datasets include various challenging conditions,
such as abrupt motion change of USVs, environmental reflections in water, and
sun glitter.

The IPATCH dataset [52] is designed for maritime anomaly detection, with a
specific focus on preventing pirate activities. IPATCH consists of 24 videos,
captured by multiple cameras stationed on a large tugboat. The dataset can
support three stages of video analysis: low-level analysis for object
detection and tracking, mid-level analysis for simple abnormal event
detection, and high-level analysis for complex threat event detection. The
types of events include simple ones such as boats speeding up, loitering, or
moving around a vessel, as well as more complex events that involve real
threats, such as abnormal approaching to the vessel of a boat that concludes
with the skiff staying at the vessel starboard.

The SMD [42] dataset is collected using Canon 70D cameras and Electro-Optical
sensors for marine object detection. The dataset consists of 70 on-shore
videos and 11 on-board videos, captured by a stationary camera on-shore and a
mobile camera on a vessel, respectively. All the videos in the dataset are of
high-definition (1080$\times$1920 pixels). Moreover, the dataset features
various environmental conditions, such as before sunrise, sunrise, noon,
evening, haze, and rain.

Finally, the MODS dataset [43] serves for marine object detection in order to
avoid obstacles during the autonomous operation of small-sized USVs. It
contains 81K stereo images synchronized with an on-board IMU, with over 60K
objects annotated. MODS covers a broad range of obstacle appearances and
types, including various dynamic obstacles such as boats, buoys, swimmers, and
swans, as well as static obstacles such as shores and wharves. MODS training
set is a combination of the SMD [42], MODD1 [56], and MODD2 [57] datasets. The
dataset presents real-world challenges such as different weather conditions,
water surface ripples, and sunlight reflections on the water.

Figure 10: Examples of anomalous events captured by dashcams mounted on cars.
(a) Image of a dashcam mounted on a car’s windshield. (b) Non-ego traffic
accident (collision) from the DAD dataset [2]. (c) Ego-involved traffic
accident (collision) from the DoTA dataset [50]. (d) Traffic anomalous
behavior (close merge) from the HTA dataset [53]. (e) SMI-RED250 infrared eye-
tracker provides extra driver attention information (white dot) for crowd-
sourced traffic accident videos in the DADA-2000 dataset [48]. (f) Real-life
events-dyadic interactions (fighting) from the RE-DID dataset [83].

###  4.3 By platforms on the ground

In this section, we present datasets that are collected by mobile platforms
that operate on the ground, such as wheeled robots and autonomous cars. Among
all MC-VAD datasets, those with moving cameras on the ground constitute the
largest portion of datasets overall, including indoor scenes, urban streets,
and farmland scenarios. Most of the ground datasets are designed for safe
autonomous navigation where anomalies are typically potential hazards to be
avoided. For example, some of these datasets are designed for detecting
traffic accidents [2, 46, 47, 48, 53, 49, 50], obstacles on urban roads [38,
40, 41], rail transit irregularities [121, 122, 123], hazards in agricultural
settings [39], and detection in hazardous environments [114]. Some of them are
utilized for patrolling or surveillance where anomalies are events of interest
that need to be reported, such as abandoned object detection [75] and violent
behavior detection [83]. Because RGB cameras are nowadays inexpensive, they
are often used to capture data, in particular dashcams are rather popular and
commonly installed in vehicles. Dashcams serve as the main video contributors
for road monitoring and anomalies. The primary challenges in dealing with
dashcam datasets involve unpredictable high-speed camera motion, low
resolution, reflections on car windows, occlusions, and changing weather and
traffic conditions.

The Re-DID dataset [83] is a collection of 30 YouTube videos for the study of
detecting fighting events by identifying the area associated with
interpersonal space (Fig. 10f). Among the videos, 25 of them are recorded
using dashcams, while the remaining ones are taken by mobile phones. Re-DID
includes 73 different fight instances under different lighting (day, night),
weather conditions (sunny, rainy), camera views (wide angle, fish-eye, zoomed
view), and moving and static scenes. Re-DID also contains annotations of
subjects’ bounding boxes, temporal windows where the interaction occurs, and
interpersonal space regions.

The DAD dataset [2] is crowd sourced from dashcam footage (Fig. 10a,b). It
comprises 678 accident videos from six major cities in Taiwan, featuring a
variety of collisions involving motorbikes and cars as primary participants.
Each positive clip in the dataset contains the moment of the accident within
the final ten frames, while a negative clip includes no accident. Compared to
datasets such as KITTI [107] that is captured by high-quality cameras in
normal driving scenarios, the DAD dataset presents more complex road scenes
and diverse accident scenarios.

The A3D dataset [47] is another dashcam dataset containing 1.5K road anomaly
event videos captured from different vehicles in East Asia. These videos are
categorized into 18 types of traffic accidents, and feature various weather
conditions (e.g. sunny, rainy, snowy), driving scenarios (e.g. urban or
rural), and participants (e.g. cars, motorcycles, pedestrians and animals).
The start and end times of each anomaly are annotated: the start time is
defined as the point the accident becomes unavoidable, and the end time is
defined as the point where the participants resume their normal activity.
However, the video quality is generally low, making anomaly detection, object
detection, and ego-motion estimation challenging.

The DADA-2000 dataset [48] mainly serves for the study of predicting driver
attention in accident scenarios. It contains 2K dashcam video clips on 54
kinds of accidents along with the maps of fixations, saccade scan path, and
focusing time to represent driver attention (Fig. 10e).

The HTA dataset [53] targets the challenge of identifying abnormal traffic
patterns on highways using dashcam footage. HTA is derived from the Berkeley
DeepDrive dataset [124], which includes 100K high-resolution (1280$\times$720
pixels, 30 FPS) dashcam videos shot by vehicles in New York and the Bay Area.
HTA features 389 curated videos, where normality is characterized by the
motion of vehicles that do not disrupt the motion of the vehicle with the
dashcam, or where the movement of other vehicles remains relatively self-
similar (Fig. 10d). Anomalies are categorized as speeding vehicles, close
merging events, halted vehicles, vehicle accidents, and speeding motorcycles.
Each of these instances depicts a scenario where a human driver would
typically react with caution.

Figure 11: Examples of anomaly (obstacle) detection performed on videos
captured from unmanned ground vehicles. (first row) RGB image with overlaid
bounding boxes indicating the location of the anomaly. (second row) Ground-
truth segmentation masks containing anomalous objects. (a) Static and dynamic
obstacles from the Lost&Found dataset [38]. The ground-truth mask shows the
free space in purple, a pallet and a ball in blue, and a piece of wood in
yellow. (b) Small hazards (boxes and stones) from the Small Obstacle dataset
[40]. The ground-truth mask shows the road in red, off-road in black, and
obstacles in other colors. (c) Static obstacles (bottles) from the Ground
Mobile Robot Perception dataset [106]. The ground-truth mask shows the road in
dark gray, the rest of the environment in black, and obstacles in light gray.
(d) Dynamic obstacle (animal) from the RoadAnomaly dataset [41]. The ground-
truth mask shows the obstacle in red.

The CCD dataset [49] contains both abnormal and normal dashcam videos. It
includes 1.5K traffic accident video clips and 3K normal dashcam videos from
BDD100K [124]. The CCD dataset is larger than the DAD and A3D datasets, and
contains richer annotations including environmental attributes, details about
whether ego-vehicles are involved, and descriptions of the reasons for each
accident.

The DoTA dataset [50], an enhancement of the A3D dataset, comprises 4,677
anomaly videos that are annotated for temporal, spatial, and categorical
attributes (Fig. 10c). The anomalies in the dataset are categorized into 18
types, including different kinds of collisions, instances of vehicles going
out of control, turning or crossing situations, interactions between vehicles
and pedestrians, vehicles and objects, and so on. Each category is further
divided into ego anomalies and non-ego anomalies. The richness of the
categorical annotations in DoTA has been leveraged to evaluate video action
detection and online action detection methodologies. Other dashcam video
datasets similar to DoTA and designed for MC-VAD include the datasets
introduced by [125, 46, 102, 103].

There are also datasets available for detecting various types of obstacles and
unexpected objects, both in indoor environments with patrolling robots or in
outdoor environments, e.g. roads or fields, with autonomous vehicles.

The VDAO dataset [75] is collected with a camera equipped on a iRobot Roomba
for automatic detection of abandoned objects in cluttered industrial
environments (Fig. 6). The dataset comprises 6 multi-object, 54 single-object,
and 4 no-object (reference) videos, collected from two different cameras.
Various abandoned objects, such as different boxes, bags, and coats, are
considered as anomalies. Factors like cluttered environments, object
occlusions, reflections, and scene shadows in the videos may pose challenges
to effective detection.

The Hazards & Robots dataset [114] is another indoor video dataset, captured
from the front-facing RGB camera equipped on a wheeled robot, Robomaster S1.
The robot patrols in indoor environments (e.g. offices and university
buildings) with natural and artificial illumination. The dataset is composed
of 132,838 RGB frames with a resolution of 512$\times$512 pixels and includes
16 categories of anomalies. These range from subtle objects like screws on the
floor to more noticeable ones like passing humans, as well as hazardous events
(e.g. water puddles to avoid) and harmless objects (e.g. hanging pieces of
cellophane). This dataset provides hazardous situations that are common in
reality and difficult to discern.

Obstacle detection on the road ahead is an essential component for both fully
autonomous vehicles and advanced driver assistance systems. As illustrated in
Fig. 11, obstacles that may appear on the road can be of various types and
sizes. It is also challenging to detect small objects in complex and cluttered
outdoor scenes due to distant objects, varying road surface appearances, and
illumination changes. In addition to monocular RGB cameras, some of these
datasets are collected using stereo cameras [38], radar [40], or multi-sensor
setups [39].

Figure 12: Examples of anomalies on videos captured from aerial platforms. (a)
A drone over the parking lot caught a man in yellow robbing a car from the
minidrone dataset [32]. (b) A drone with multi-modal camera records a person
in the act of hijacking from the UAV-human dataset [33]. (c) A fast-moving
boat on the sea from the Seagull dataset [34]. (d) People in life jackets
floating on the water from the SeadroneSee dataset [35].

The Lost & Found dataset [38] is collected to facilitate the study of
detecting small obstacles or debris on the road for autonomous driving. The
dataset is comprised of 112 stereo video sequences, recorded from 13 different
challenging street scenarios, featuring 37 different obstacle types. The
selected scenarios represent particular challenges including irregular road
profiles, far object distances, different road surface appearance and
illumination changes. The dataset provides broad annotations of free-space
areas and detailed annotations of obstacles on the road. The anomalies include
various small hazards and lost cargo such as pallets, boxes, and different
roadblocks (Fig. 11a).

The Road Anomaly dataset [41] is designed for detecting anomalous objects such
as animals, rocks, lost tires, trash cans, and construction equipment, which
are located on or near the road (Fig. 11d). The dataset contains 60 images,
each with a resolution of 1280$\times$720 pixels, and includes per-pixel
annotations of these unexpected objects. However, annotations are provided
only for objects belonging to the abnormal class in the images, without
labeling anomalies of other categories present in the scene.

RGB-D cameras can measure both the photometric properties of objects and the
physical distance to their surfaces, therefore they can enhance a robot’s
ability to detect anomalous objects on the road, such as bumps and
irregularities. The GMRP dataset [84] comprises 3,896 pairs of RGB-D images
captured using an Intel RealSense D415 camera, specifically designed for road
anomaly detection in scenarios pertaining to autonomous navigation of general
ground robots, such as robotic wheelchairs. Different from most obstacle
detection datasets for autonomous cars, this dataset focuses on a broader
spectrum of ground robot scenarios, encompassing 18 road anomalies across 30
different contexts, including indoor and outdoor environments like corridors
and sidewalks. Anomalies are defined as areas rising more than five
centimeters above the surface of the drivable area, including objects like
traffic cones, plastic bottles, and stones (Fig. 11c).

Because LiDAR has a longer detection range than RGB-D cameras, it can identify
obstacles at greater distances, and its detection capabilities are more robust
to light and color variations. The Small Obstacle dataset, introduced by Singh
et al. [40], includes 2,927 monocular RGB images synchronized with sparse
LiDAR information from a Velodyne Puck 16 and aligned with odometry data. The
dataset annotations are divided into three categories, i.e. road, off-road,
and small obstacles. The external parameters for LiDAR-Camera calibration are
also included.

The FieldSAFE dataset [39] is a multi-modal dataset that is designed for
obstacle detection in agricultural scenarios. The dataset is obtained using
multiple sensors mounted on a tractor during grass-mowing activities. The
sensor pack includes a stereo camera, a thermal camera, a web camera, a 360∘
camera, a LiDAR, and a radar. The dataset includes both static and moving
obstacles such as humans, mannequin dolls, rocks, barrels, buildings, and
vehicles. All obstacles are accompanied by ground-truth labels and geographic
coordinates. As the dataset is collected using the Robot Operating System
(ROS), hence its release is in the ROS bag format.

Lastly, the large-scale, first-person, multimodal X-MAS dataset [85] consists
of more than 500K image pairs, collected from a UGV navigating in outdoor
environments. This dataset incorporates infrared, RGB, thermal, depth, and
LiDAR sensors. X-MAS covers a diverse array of challenging environmental
conditions, such as varying weather patterns, including daytime, nighttime,
rainy, and foggy conditions. The dataset is dedicated to human behavior
monitoring tasks, with a particular focus on anomalous scenarios, such as
falls, convulsions, and drunkenness. The dataset provides extensive
annotations for tasks like object detection, tracking, action classification,
and labeling of normal and abnormal scenes.

###  4.4 By platforms in the sky

In this section, we cover datasets that are obtained from cameras equipped on
UAVs, such as rotorcraft [32, 33, 51] or fixed-wing [34, 35], that feature
various scenes such as cities, farmlands, and marine areas. Similar to
surveillance videos, this type of datasets typically offers a third-person
perspective from the top view, thus with a broader field of view and fewer
occlusions. However, as the distance between the camera and the monitoring
areas is often very large, these datasets typically present challenges in
detecting small objects as shown in Fig. 12. Many of the datasets captured
from flying UAVs are designed for anomalous human behavior analysis.

The MDV dataset [32] comprises 38 videos recorded by a drone flying at low
altitude in a parking lot. The content of this dataset falls into three
categories: normal, suspicious, and illicit behaviors. Normal content captures
people walking, entering cars, and parking vehicles. Whereas suspicious
content portrays individuals acting in questionable ways. Illicit behaviors
primarily include improper parking, theft, and physical altercations.

The AVI dataset [51] includes 2K images captured from a drone at heights
varying from 2 to 8 meters. This dataset is mainly used for human violent
action recognition and pose estimation, and it features five violent action
classes, such as punching and kicking, carried out by 25 individuals aged
between 18 and 25 years. The images are annotated with 14 key points for each
human to aid pose estimation. In general, the task of detecting individual
action anomalies in these aerial images presents significant challenges due to
factors such as lighting variations, shadows, low resolution, and blur
effects.

The UAV-Human dataset [33] is a large-scale multi-modal dataset that is
designed for human behavior analysis. It comprises 22,476 UAV videos, each
with three types of sensors: Azure DK, fisheye camera, and night-vision
camera. Thus, this multi-modal dataset includes RGB, depth, IR, fisheye,
night-vision, and skeleton sequences. The dataset features 119 distinct
subjects and 155 different activity categories, encompassing over 26 anomalous
types, such as vomiting, kicking, punching, stabbing someone with a knife, and
holding someone hostage. The dataset exhibits extensive variations in
environments, time periods, and diverse challenging weather conditions.

UAV-captured datasets also serves for MC-VAD in marine environments. The
Seagull dataset [34] comprises 19 video sequences that feature various types
of objects observed in marine environments, such as cargo ships, small boats,
sailing yachts, and leaked floating oil. These objects are recorded using
visible light, Long Wave Infra-red (LWIR), and hyperspectral sensors. The
flight altitude for surveillance is between 150 and 300 meters above the ocean
surface to ensure a broad coverage. The objects captured are small and the
varying lighting conditions along with sea surface reflections make object
detection challenging.

The Seadronesee dataset [35] encompasses over 54K frames of 20 test subjects
in open water, captured under various lighting conditions across multiple
days. The annotation categories include swimmer (person in water without a
life jacket), floater (person in water with a life jacket), life jacket,
swimmer (person on a boat without a life jacket), floater (person on a boat
with a life jacket), and boats. Seadronesee can be used for the study of
several related tasks, including object detection, single-object tracking, and
multi-object tracking.

TABLE IV: Summary of relevant MC-VAD methods. Each class is organized
according to traditional image processing methods (such as geometry) and deep
learning-based methods.

Methods |  Descriptions |  Traditional methods |  Deep learning-based methods
---|---|---|---
Semantic segmentation |  Based on scene context, and often used for scene-level appearance anomaly detection such as obstacle detection, and abandoned object detection |  [38] [40] [126] |  [127] [128] [129] [41] [130] [131] [132] [84]
Reference frames |  Based on reference videos for each specific scene, and often used in known environments with minor background changes |  [54] [121] [91] [90] [92] |  [93, 94] [89]
Background separation |  Based on foreground extraction or spatial attention for eliminating background noise, and employed for detecting anomalies of pedestrians and vehicles |  [95] [133] |  [4] [51] [134] [135] [136] [137] [138] [139] [140] [99] [141] [123]
Object trajectories |  Based on object detection and tracking, and trajectory prediction and analysis |  |  [2] [142] [143] [144] [46] [145] [146] [147] [50] [148]
Object interactions |  Based on spatio-temporal contextual clues such as object location relationships to detect object-to-object and object-to-scene interactions |  [83] |  [49] [149, 150] [151] [152] [153] [154] [155] [156]

###  4.5 Discussion

We provided a thorough coverage on the datasets that can of use for MC-VAD
research. Note that not all datasets mentioned in this section are
specifically designed for anomaly detection. Some are originally designed for
tasks such as object detection or semantic segmentation. However, they possess
typical MC-VAD characteristics, including complex dynamic backgrounds, camera
motion, and multiple detection categories, that align with the definition of
MC-VAD provided in the introduction. As illustrated in Tabs. II & III,
datasets targeting different application domains exhibit significant
variations in their characteristics.

Underwater datasets primarily focus on object detection and semantic
segmentation tasks [113]. These datasets often present challenges related to
motion blur, low underwater clarity, and limited field of view. On the other
hand, datasets recorded on the water surface primarily focus on object
detection tasks [43]. The typical challenges associated with these datasets
are diverse weather conditions and sea surface reflection. Datasets captured
by cameras moving on the ground generally offer a first-person perspective,
being recorded from dashcams and collected from the internet. One of the main
tasks associated with such datasets is the detection of road accidents [50].
These datasets often present challenges due to complex and dynamic
backgrounds, unpredictable camera motion, and significant occlusions.
Additionally, these datasets may include multiple sensory modalities, which
could enhance the accuracy of anomaly detection. Datasets captured by UAVs
typically offer a high third-person viewpoint. In the task of human behavior
analysis [33], most existing UAV datasets are designed for target detection
and/or tracking, leading to a scarcity of abnormal samples. Datasets that do
feature abnormal behaviors only contain a few instances [32, 51], or the
abnormal behaviors are insufficiently realistic [33], thereby making them
impractical for MC-VAD studies.

Finally, we observe an unbalanced distribution in the available datasets, a
disparity attributable to differing interest levels within their respective
application domains. For example, within the intelligent transport domain,
there exists a significant quantity of real-world datasets, obtained either
from dashcams or cameras mounted on vehicles. Conversely, in other domains,
such as marine environments, the availability of datasets is considerably
less. The infrequency of real anomalous videos in certain application domains
also hampers respective research and development efforts. Despite growing
demand for urban surveillance with moving cameras, real-world anomalous events
such as crime, violence, and fires are challenging to capture with cameras
mounted on mobile robots.

##  5 Approaches

We categorize MC-VAD approaches into scene-level, which focus on detecting
frame-level anomalies, and scenario-level, which focus on detecting temporal-
level anomalies.

Scene-level anomalies are the result of deviations from normal patterns that
can be detected in a single video frame. This category includes appearance-
only anomalies, or collective anomalies that do not require temporal
information. Appearance-only anomalies can be the existence of an unknown
object not seen during training or a known object in an unusual location.
Examples of these include novel objects in underwater exploration, e.g. sunken
ships or statues [45], or a bottle in the middle of the road [84]. Collective
anomalies can be crowds in public places or events with people flowing in
certain directions (e.g. parades) [157, 158, 159]. Crowd counting and density
estimation techniques can be exploited for detecting anomalies in these types
of scenarios [160, 161].

Scenario-level anomalies result from unusual movements or unexpected events
that necessitate temporal information for detection. We further categorize
scenario-level anomalies into short-term motion anomalies, long-term
trajectory anomalies, and contextual anomalies. Short-term motion anomalies
refer to unusual object movements over a short duration, identified by
analyzing optical flow and local information of moving objects. An example
might be a car suddenly stopping at an intersection due to a collision [53].
Long-term trajectory anomalies pertaining to unusual object trajectories over
a longer duration, such as a person suspiciously wandering within a parking
lot [32]. Detecting such anomalies can necessitate long-term object tracking,
which poses a challenge even with static cameras [162], and more so with
moving cameras [163, 164]. Indeed, we note a paucity of literature addressing
long-term trajectory anomalies. Contextual anomalies are not exclusively based
on the motion or trajectory of objects but rather on the overarching context
in which objects interact and behave. Methods to detect contextual anomalies
may involve analyzing object location relationships, object interactions, or
other aspects of semantic scene understanding to identify abnormal patterns.
Moreover, collective anomalies can also form part of scenario-level anomalies
as they might result from crowd intersections. These anomalies can be detected
based on the analysis of individual motions, such as oscillatory features,
trajectory features [165, 166], or the temporal accumulation of intersecting
motions [31].

MC-VAD methods can be further classified into five categories based on the
type of anomaly (Tab. IV). Two out of these five categories belong to scene-
level anomaly detection methods that are semantic segmentation-based and
reference frame-based. The other three belong to scenario-level anomaly
detection methods that are background separation-based, object trajectory-
based, and interaction-based. Among them, the reference frame-based method can
also handle some scenario-level anomalies. Tabs. V-IX provide a summary of
relevant methods within these categories, specifying the datasets each method
employs for evaluation. In order to provide a comprehensive overview of our
analysis, we present a categorization of methods based on anomaly level,
application domains, and tasks in Fig. 13. It is worth noting that both scene-
and scenario-level anomalies are present across all three application domains,
which include security, urban transportation, and marine environments.
However, the proportion of research works significantly varies among different
fields, as indicated by the line width in Fig. 13. In the following sections,
we provide a detailed description of the pertinent methods within each of
these categories.

Figure 13: Summary of the correspondence between MC-VAD approaches, anomaly
levels, field of application and specific tasks. The line width indicates the
quantity of work in each category.

###  5.1 Based on semantic segmentation

Semantic segmentation entails the process of assigning a specific category
label, chosen from a predefined set of possibilities, to each pixel in an
image. VAD approaches that utilize semantic segmentation are predominantly
used for detecting scene-level anomalies, such as obstacle detection, across a
range of application domains.

In a marine environment, the work by Cane and Ferryman [128] evaluates three
deep segmentation networks, namely ENet [167], ESPNet [168], and SegNet [169],
for the detection of typical objects such as large ships, speedboats, sailing
boats, and buoys. These segmentation networks are trained on the ADE20K [170]
ocean dataset in a supervised manner and are tested on four maritime
surveillance datasets, namely MODD1 [56], SMD [42], IPATCH [52], and Seagull
[34]. Although the deep segmentation networks have shown promising results in
terms of marine object detection and segmentation, marine environments present
a unique challenge due to frequent lighting condition variations, leading to
numerous false positives. Recent work in marine obstacle detection
incorporates temporal information to eliminate false positives caused by sun
reflection [132]. Multimodal information can also be leveraged to mitigate
undesired false detections. The method proposed by Arain et al. [129] combines
feature-based stereo matching with learning-based segmentation to produce a
more robust obstacle map. This method enhances the binary obstacle map with
vision-based distance classification (near, mid, and far) and the depth
information from sparse stereo matching, achieving an improved obstacle
detection with fewer false positives.

In urban scenarios, multiple modalities are often exploited alongside visual
inputs to improve anomaly detection accuracy. For example, stereo vision can
be used for detecting small obstacles on the road. The detection task can be
formulated as a statistical hypothesis testing problem on image data applied
to the segmentation map of the road [38]. LiDAR can provide additional context
for monocular segmentation networks in the form of a confidence map, thereby
improving detection performance. The method proposed by Singh et al. [40]
employs the Hausdorff distance for calibration refinement over extrinsic
parameters, achieving accurate calibration between LiDAR and the camera. The
discrepancy between abnormal and normal regions can be enhanced by fusing
multi-modal and multi-scale features [84].

Recent methods also combine semantic segmentation with generative models for
anomaly detection [41, 130, 131]. The main intuition behind this is as
follows: since semantic segmentation models are trained with known classes,
when they are tested with images containing unknown classes, these unknown
regions would be inaccurately segmented (misclassified as other known classes,
either foreground or background or over-segmented) in the predicted semantic
map. If one resynthesizes the image based on the predicted semantic map, the
regions of the unknown classes in the generated image would exhibit a large
discrepancy compared to the original test image. This discrepancy can be
exploited for anomaly detection [41]. Several recent works follow this
framework for anomaly detection. For example, the method presented by Di Biase
et al. [171] introduces a method for integrating different uncertainty
measures, such as softmax entropy, softmax difference, and perceptual
differences, to improve the discrepancy in differentiating between the input
and generated images. Moreover, the work by Vojir et al. [172] further
exploits the property that the anomalous object is not from a class that could
be modeled and is also dissimilar to normal objects in appearance. This method
employs an embedding bottleneck (a small embedding network that takes the
backbone features and transforms them into an embedding vector) to model the
normal class in an explicit embedding space using augmented training data by
synthesized anomalies. It also introduces image-conditioned distance features
that allow known class identification in a nearest-neighbor manner on-the-fly,
and an inpainting module to model the uniqueness of detected anomalies. This
effectively reduces false positives by filtering out regions that are similar
to their neighborhood.

In summary, segmentation-based methods exploit pixel-level labels for the
detection and localization of scene-level anomalies, particularly for tasks
such as obstacle detection and the recognition of anomalous or novel objects.
These methods primarily focus on single image frames, not utilizing temporal
information, which can lead to inaccurate detections in cases where cameras
move under various lighting conditions and complex backgrounds. Additionally,
training these methods may require a substantial amount of annotated datasets,
necessitating efforts associated with dataset collection and annotation. Tab.
V provides a summary of these methods.

TABLE V: Relevant methods based on semantic segmentation. For each method we
indicate the dataset used for evaluation and whether the dataset is publicly
available.

References |  Descriptions |  Domain |  Dataset/availability |  Remarks
---|---|---|---|---
[38] |  Detecting small obstacles in road scenes using stereo vision |  Obstacle detection |  Lost&Found/$\checkmark$ |  Traditional multi-view geometry-based method to operate in real-time. Small obstacles are defined as objects with a height between 5cm and 25cm above the ground.
[128] |  Detecting marine objects based on semantic segmentation |  Maritime surveillance |  ADE20k/$\checkmark$ |  Test sequences are selected in MODD, SMD, IPATCH, and Seagull datasets with different viewpoints (from water surface to high altitude).
[129] |  Detecting underwater obstacles based on the combination of feature-based stereo matching and learning-based segmentation |  Underwater obstacle detection |  John Brewer Reef Dataset/$\times$ |  Obstacles are defined as any object (e.g. seafloor, coral) that hinders robot navigation.
[41] |  Detecting road anomalies based on semantic image synthesis |  Obstacle detection |  Lost&Found/$\checkmark$,
Road Anomaly/$\checkmark$ |  The problem of detecting unknown classes is formulated as identifying poorly-synthesized image regions.
[40] |  Detecting small road obstacles based on multiple modalities (LiDAR and monocular RGB) |  Obstacle detection |  Small Obstacle Dataset/$\checkmark$ |  The method can detect small obstacles (as small as 15cm high) within the range of 50 meters.
[131] |  Detecting anomalies based on the difference between a generated synthetic image from the segmentation map and the input image |  Failure detection and anomaly segmentation |  Cityscapes/$\checkmark$,
StreetHazards Dataset/$\checkmark$,
Pancreatic Tumor Segmentation Dataset/$\checkmark$ |  Evaluation carried out in both autonomous driving and medical imaging applications.
[132] |  Detecting marine obstacles based on temporal context-assisted segmentation |  Marine obstacle detection |  Mastr1478/$\checkmark$, MODS/$\checkmark$ |  The temporal information in consecutive frames is used to reduce the influence of water turbulence and sunlight flicker on obstacle segmentation.
[84] |  Detecting road anomalies based on dynamic data-fusion model |  Obstacle detection |  GMRP/$\checkmark$, KITTI Road/$\checkmark$, and KITTI Semantic Segmentation Dataset/$\checkmark$ |  Includes a benchmark for large-scale mobile robot road anomaly detection.

###  5.2 Based on reference frames

Approaches based on reference frames are typically developed for detecting
anomalies in specific or known environments, where normality is well-defined,
such as abandoned object detection in industrial settings [75, 91, 90, 92],
obstacle detection on railway tracks [121], or robot patrolling [54, 89, 93,
94]. These approaches typically capture and record normal images of the
environment beforehand to detect any anomalies that deviate from the recorded
normal scenes. Fig. 14 illustrates how a typical reference frame-based method
works: it detects abnormal objects by aligning the reference video sequence
with the target video sequence. The reference video is captured by a moving
camera when the scene is in its normal state, i.e. no suspicious object is
present in the scene, while the target video is captured by a camera moving
along the same route that may contain unexpected objects. As the background
and viewpoint vary due to the camera’s ego-motion [173], methods in MC-VAD
require video pre-processing to spatio-temporally align videos in order to
locate the differences among the frames of the target videos and those of the
reference videos.

Figure 14: General pipeline of approaches based on reference frames for MC-
VAD, including video sequence and geometric alignment before frame comparison.
Reference and target videos are input into the pipeline at the same time, and
video alignment is carried out after preprocessing, including temporal
alignment and spatial alignment. The temporal alignment involves matching
between frames in the video sequence. Spatial alignment involves pixel-level
matching in a single frame. Matched images are post-processed to eliminate
false positives, and the abnormal regions are extracted by contrast to achieve
anomaly detection.

Reference frame-based methods can be used for detecting both scene-level and
scenario-level anomalies, depending on the application scenario. At the scene
level, temporal information is primarily used for frame alignment, while the
anomaly itself is independent of time. Conversely, scenario-level anomalies
are dependent on temporal information. For example, abandoned luggage in an
airport may not be considered anomalous upon initial detection, but if it
remains unattended for an extended period, it may raise security concerns and
be classified as an anomaly [87].

The work by Kong et al. [54] follows such an alignment-detection framework for
detecting non-flat abandoned objects. The process begins with a rough
alignment achieved using GPS information to find corresponding frame pairs.
Then, four steps are carried out: (i) Inter-sequence geometric alignment based
on homographies [174], which is computed by a modified random sample consensus
(m-RANSAC) [175] with Scale-Invariant Feature Transform (SIFT) descriptors
[176]. This step aims to find all potential suspicious areas. (ii) Intra-
sequence geometric alignment is then employed to eliminate false alarms caused
by high objects. The parameter selection is determined by the speed of the
camera’s ego motion. (iii) A local appearance comparison is performed between
two aligned intrasequence frames to eliminate false alarms in flat areas. (iv)
Lastly, a temporal filtering step is used to confirm the existence of
anomalous objects.

There also exist methods that do not require temporal alignment between the
test and reference video sequences. They rely solely on video content analysis
without using other modalities, such as GPS. For instance, the method proposed
by Thomaz et al. [90] employs sparse video decomposition, which has been used
in SC-VAD [177, 178], for detecting anomalies in moving-camera videos without
video synchronization. This work represents both the reference and target
videos as a low-rank projection on a union of subspaces with a sparse residual
term that is used for the detection of anomalies. Such methods assume that the
camera trajectories during the target and reference video acquisitions are
similar. However, in real-world scenarios, this condition might not be
satisfied. To tackle this problem, the paper by Jardim et al. [92] presents a
domain-transformable sparse representation for moving camera videos (mcDTSR),
where an additional non-linear domain-transformation term is introduced to
cope with the camera motion difference between reference and target
acquisition [179]. The introduced domain transformation reduces the anomaly
detection error as it allows a better alignment between the reference video
frame and the target video frame.

The work by Lawson et al. [93] involves a patrol robot that autonomously
detects aberrant objects along a fixed path. Both the reference and target
videos are divided into patches using a fixed-size grid for each frame. A
dictionary is then formed to represent the current environment, with features
extracted from patches using AlexNet. As the patrol can cover various scenes
where the anomaly can be defined differently, a deep network model [180] is
used to recognize the robot’s environment to condition the dictionary look-up.
In a subsequent work [94], a generative model, DCGAN [181], is trained using
these patches, and the bottleneck features of the generated patches and actual
patches are compared to detect anomalies.

In summary, reference frame-based methods can serve for anomaly detection
within specific environments. They are particularly effective in cases where
scene changes are minimal, such as unmanned workshops or industrial parks. In
these scenarios, autonomous patrolling can be used for anomaly detection, as
normal conditions can be predefined and recorded in reference frames.
Initially, the accuracy of these methods significantly relied on the alignment
accuracy between reference frames and target frames. However, advancements in
deep learning techniques have gradually reduced this dependency, expanding
their applicability across various scenarios. Tab. VI provides a summary of
these methods.

TABLE VI: Relevant methods based on reference frames. For each method we
indicate the dataset used for evaluation and whether the dataset is publicly
available.

References |  Descriptions |  Domain |  Dataset/availability |  Remarks
---|---|---|---|---
[54] |  Detecting non-flat abandoned object with a moving camera by using the geometric alignment |  Abandoned object detection |  Proprietary dataset/$\times$ |  RANSAC-based homography alignment to detect suspicious areas, an intra-sequence geometric alignment to remove false alarms caused by high objects, a local appearance comparison to remove false alarms in flat areas and a temporal filtering step to confirm suspicious objects.
[93, 94] |  Locating abnormal objects for robot patrol, a scene dictionary is constructed by clustering deep neural network features |  Anomalous object detection |  Proprietary dataset/$\times$ |  Image patches extracted with a fixed-sized grid and processed by a deep network. Deep features of the patch are incorporated into the environmental dictionary through K-means clustering.
[121] |  Detecting obstacles using image subtraction between the input and reference train frontal view |  Railway tracks obstacle detection |  Proprietary dataset/$\times$ |  Temporal alignment to find frame-by-frame correspondences of current and reference sequence, spatial alignment to register pixel-wise image, and subtraction methods are applied to detect obstacles.
[90] |  Detecting anomalies in video sequences obtained from slow-moving cameras based on a family of sparse decomposition algorithms |  Abandoned object detection |  VDAO/$\checkmark$ |  Representation of video data as a low-rank projection on a union of subspaces plus a sparse residue term.
[92] |  Detecting anomalies in video sequences from moving cameras through matrix factorization based on sparse representation |  Abandoned object detection |  VDAO/$\checkmark$ |  Geometric domain-transformations are used to model the effect over the samples caused by the camera movement and rotation, a transformation term in the sparse representation procedure is included for optimization.
[89] |  Detecting anomalies for moving surveillance robots based on a change detection model using a Siamese network. |  Anomalous object detection |  Proprietary dataset/$\times$ |  Geo-tag, camera Pitch and camera Yaw (GPY) information are used for normalcy definition image retrieval. Human feedback enables the system to learn continuously.

###  5.3 Based on background separation

Figure 15: The general framework of background separation-based approaches for
MC-VAD. Dashed lines indicate that in-box steps may require additional
considerations under multi-scenario conditions. In the training phase, the
training video containing only normal samples is input, the moving objects are
extracted through foreground detection, and the features of moving objects are
calculated. Without scene and context analysis, the normal model is
established by directly calculating the features of moving objects in a single
scene. In the testing phase, the testing video containing abnormal samples is
input, processed in the same way as in the training phase, and the features of
moving objects are calculated based on the normal model. The resulting
deviation is considered the anomaly score for anomaly detection.

Approaches based on background separation focus on analyzing the motion of
foreground objects [4], or employ spatial attention mechanisms [182, 2, 80] to
detect abnormal behaviors. Such methods can help mitigate the background-bias
phenomenon [137], i.e. deep neural networks are inclined towards learning the
background information, rather than the patterns of anomalies, for the
recognition of abnormal behaviors. In the context of SC-VAD, the background is
typically static, thus motion-based foreground detection using pixel-level
differences between foreground and background can be employed. However, in MC-
VAD, motion-based foreground detection is no longer applicable as the cameras
are in motion, introducing a dynamically changing background. Therefore,
techniques that use object detection to separate the foreground from the
background are more common [183, 184, 185]. Depending on the application,
these methods can analyze foreground regions independently [4, 99, 186], or
jointly when interactions among objects are being considered [51, 138, 187].
Fig. 15 illustrates the general framework of background separation-based
approaches in MC-VAD.

Typical VAD methods tend to attribute anomalies to the actions of moving
objects [3]. As such, after detecting these objects, they extract key features
from object bounding boxes such as height, width, velocity, direction of
motion, and relative distances [95]. To further enhance the comprehension of
activities, additional features like human posture can be also used [51, 33].

TABLE VII: Relevant methods based on background separation. For each method we
indicate the dataset used for evaluation and whether the dataset is publicly
available.

References |  Descriptions |  Domain |  Dataset/availability |  Remarks
---|---|---|---|---
[95] |  Detecting anomalies through behavior analysis of moving objects captured from airborne moving platforms |  Aerial surveillance |  Proprietary dataset/$\times$ |  Moving regions in the sequence are detected and tracked. Ego-motion estimation is used to compensate for the motion of the observer.
[4] |  Detecting abnormal human behavior based on real-world environments of autonomous moving robots |  Abnormal behavior detection |  UCF101/$\checkmark$, HMDB/$\checkmark$, Non-biased Background Dataset/$\times$, Moving Camera Dataset/$\times$ |  Human action in unconstrained videos is located through generic action region proposals. Motion features are extracted from action proposals by a ConvNet framework
[51] |  Identification of violent individuals using proposed real-time drone surveillance system |  Violence detection |  Aerial Violent Individual/$\times$ |  Feature pyramid network is used to detect the humans. ScatterNet Hybrid Deep Learning network is proposed for human pose estimation. Orientations between limbs are used to identify violent individuals.
[138] |  Detecting potential hazards by identifying human social interactions in continuous RGB-D video sequences obtained by service robots |  Behavior understanding |  Social Activity Dataset/$\checkmark$ |  Extract skeleton and compute features from RGB-D sequences. Individual poses, movements, and spatial relations of features are used to classify and detect social activities such as fights, talk, and handshake.
[99] |  Recognizing human violent action using drones from different heights for an unconstrained environment |  Violence detection |  Proprietary dataset/$\times$ |  Extract key-point and generate 2D skeletons for the person. Support Vector Machine and Random Forest are used for classification to recognize actions.

During the model learning phase, a normal model is constructed to represent a
normal situation by incorporating scene analysis and contextual information
while extracting features. In the case of a single scene, a singular normal
model can be directly established [47]. In the case of multiple scenes, the
relationship between different features and corresponding scenes is typically
required [95, 4]. At inference time, moving object features (such as
appearance, pose and motion trajectories) are extracted, and the deviation
between these features and those extracted from normal model is used for
anomaly detection. These deviations are represented can be the prediction
error and reconstruction error. Additionally, methods such as clustering or
Support Vector Machine (SVM) can also be employed for feature classification
to detect anomalies.

Recognizing and understanding human activity is essential for a wide variety
of MC-VAD applications from security surveillance purposes [51] to having safe
and collaborative interaction between humans and robots in shared workspaces
[188]. The work by [4] leverages the concept of motion boundaries [189] to
select action regions (i.e. interested area only contains human activity) to
eliminate irrelevant regions. Convolutional neural networks are utilized to
extract features within the action regions, followed by action classification
using SVM. During the model learning phase, the model requires identifying the
place as well as the action being performed by the human. Then the prior
probabilities of actions occurring in different scenes are considered to
determine whether an action falls under abnormal behavior.

Due to the articulated structure of the human body, which enables complex
movements across multiple degrees of freedom, obtaining key features solely
from bounding boxes may not effectively capture this advanced movement, thus
pose estimation algorithms are employed to extract key points and skeletal
features. These features are then used for feature computation, offering a
more concise and compact representation compared to pixel-based methods.
Consequently, this approach has garnered considerable attention in the field
of anomaly detection in recent times [136, 140, 190, 187]. The work by Singh
et al. [51] and Srivastava et al. [99, 101] achieves violent individual
identification in an unmanned aerial vehicle surveillance system by utilizing
human detection and pose estimation algorithms to compute key point features.
Coppola et al. [138] extract skeleton and compute features from continuous
RGB-D video sequences captured by a mobile service robot to detect human
interactive behaviors such as handshake, help walking, fighting and talking.

In summary, background-based methods in MC-VAD require object detection in
order to detect foreground objects. In traffic scenes, anomalies are often
associated with accidents, which can be detected by analyzing motion patterns.
In security surveillance scenes, direct analysis of skeletal features in human
bodies, as opposed to frame-level detection, allows for the inclusion of more
compact and valuable information. It is important to note that these methods
inherently involve some level of prior knowledge regarding the specific types
of anomalies present in the application. Therefore, they are better suited for
specific applications or scenarios. These methods heavily rely on the accuracy
of foreground detection. In situations where occlusions or crowds are present,
the accuracy of anomaly detection can be affected. Tab. VII provides a summary
of these methods.

TABLE VIII: Relevant methods based on object trajectories. For each method we
indicate the dataset used for evaluation and whether the dataset is publicly
available.

References |  Descriptions |  Domain |  Dataset/availability |  Remarks
---|---|---|---|---
[2] |  Anticipating accidents in dashcam videos based on a Dynamic-Spatial-Attention (DSA) Recurrent Neural Network (RNN) |  Traffic accident anticipation |  DAD/$\checkmark$ |  Early work that uses crowdsourced dashcam accident datasets for traffic accident anticipation.
[143] |  Accident prevention through joint prediction of on-board camera ego-motion and pedestrian trajectories |  Trajectories prediction |  Cityscapes/$\checkmark$ |  Model consists of two specialized streams, the odometry stream to predict future vehicle odometry sequence, and the bounding box estimation stream to predict odometry distributions conditioned to pedestrian trajectories. A Bayesian RNN encoder-decoder architecture is used to compute model epistemic and aleatoric uncertainties.
[46] |  Localizing future vehicles by simultaneously predicting location and scale from egocentric views |  Trajectory prediction |  HEV-I/$\checkmark$ |  Multi-stream recurrent neural network (RNN) encoder-decoder model to capture both object location and scale, and pixel-level observations to predict vehicle locations.
[50] |  Detecting traffic video anomalies based on future object localization |  Traffic accident detection |  DoTA/$\checkmark$ |  Large scale benchmark for traffic anomaly detection is proposed including a new dataset (DoTA), evaluation metrics (spatial-temporal area under curve), and online video action detection methods.
[147] |  Forecasting egocentric human trajectory with a wearable camera and multi-modal fusion |  Trajectory prediction |  TISS/$\checkmark$ |  Three different modalities (past ego trajectory, past trajectories of nearby people, and the environment such as the scene semantics or the depth of the scene) to forecast the trajectory of the ego-camera.
[148] |  Detecting accidents based on spatio-temporal feature encoding with a multilayer neural network |  Traffic accident detection |  DoTA/$\checkmark$ |  Coarse detection based on temporal coding locates potential accident frames, and fine detection based on spatial coding detects accident frames.

###  5.4 Based on object trajectories

Approaches based on object trajectories focus on constructing a normal model
using trajectories derived from collections of videos that only include normal
behaviors or events [2, 143, 46, 47, 50, 147, 148]. The trajectory of an
object in a test video is then compared with those in the normal model, where
any significant deviation is considered as an anomaly.

Object trajectories are estimated from image sequences captured by the camera,
and thus, motion patterns are determined relative to the camera reference
system. In SC-VAD, where the camera reference system is static, the detected
motion is purely that of the moving object, and clustering techniques can be
employed to detect these deviations [191, 192, 193]. However, in MC-VAD, the
measured trajectories from the camera also include the camera’s ego motion. It
is necessary to differentiate between the motion of the camera and that of the
objects of interest in order to only model the normality of object
trajectories [46]. To account for camera ego-motion, some studies use visual
odometry to predict the ego-motion, and they incorporate a trajectory
prediction module into the framework [95, 143, 47, 50].

Road scenes viewed from aerial moving cameras can be analyzed by using the
method proposed by Medioni et al. [95]. This method initially estimates the
drone ego-motion to register consecutive frames, then it carries out multi-
object detection and tracking to locate moving objects. Given that motion
alone is not a sufficient indicator of anomalous activities, the method also
takes into consideration the interactions between trajectories. These
trajectory patterns are further combined with user-provided information, such
as geospatial context and goal context, to classify and recognize context-
aware anomalous activities.

Traffic accidents can be detected from vehicle dashcams using the method
proposed by Yao et al. [47]. This method leverages the past motion trajectory
of moving objects to predict their current positions. These predictions are
then compared with the observed frames to identify and flag any unexpected
discrepancies. As object motion alone is insufficient for identifying
anomalies, this method employs a combination of optical flow, moving object
bounding box, and camera ego-motion for trajectory prediction. Data for the
bounding box is supplied by object detection [184] and multi-object tracking
[194] techniques, while information about camera motion is obtained via camera
ego-motion estimation [76]. The same authors extended this work in [50], which
presents a method for traffic VAD based on an ego-motion Recurrent Neural
Network (RNN) encoder-decoder to predict future odometry of the ego-vehicle,
and a two-stream RNN encoder-decoder incorporating predicted ego-motion into
future object bounding box predictions. The same work also introduces a new
large-scale traffic VAD dataset named DoTA, with temporal, spatial, and
categorical annotations, a new spatio-temporal evaluation metric for VAD,
called STAUC, and a comprehensive benchmark of state-of-the-art VAD, video
action recognition, and online action detection methods.

Pedestrian motion in traffic scenes can be predicted from vehicle dashcams by
using the method proposed by Bhattacharyya et al. [143]. This method is a two-
stream model for long-term prediction of both person bounding box and the
vehicle odometry. The bounding box prediction stream, implemented as a
Bayesian recurrent neural network encoder-decoder, follows a Bayesian
formulation to predict odometry-conditioned distributions over pedestrian
trajectories. Such Bayesian formulation also enables the modeling of both
epistemic (model) and aleatoric (observation) uncertainties for the bounding
box prediction.

In summary, these methods leverage object trajectories for anomaly detection,
reducing the dependency on pixel-level differences and background changes.
Yet, the performance of the final task, whether it is behavior prediction or
analysis, is closely linked to the performance of other processing modules,
such as object detection, object tracking, and ego-motion/odometry estimation.
Tab. VIII provides a summary of these methods.

TABLE IX: Relevant methods based on object interactions. For each method we
indicate the dataset used for evaluation and whether the dataset is publicly
available.

References |  Descriptions |  Horizon |  Dataset/availability |  Remarks
---|---|---|---|---
[83] |  Detecting violent behaviors, such as fighting, through the detection and localization of dynamic human interactions in real videos |  Violence detection |  RE-DID/$\checkmark$ |  Urban fight scene dataset containing camera motion is presented. Visual information is extracted in the area associated with the interpersonal space to detect fights.
[49] |  Anticipating traffic accidents based on uncertainty spatio-temporal relational learning |  Traffic accident anticipation |  CCD/$\checkmark$ |  Graph convolution and recurrent networks are used for relational feature learning. Bayesian neural networks are leveraged to address the intrinsic variability of latent relational representations.
[150] |  Understanding vehicle behavior based on multi-relational graph convolutional network |  Behavior understanding |  Honda Driving Dataset/$\checkmark$,
Apollo/$\checkmark$, KITTI/$\checkmark$ |  Dynamic scenes are decomposed into multiple relation interaction graphs. Detected lane markings and poles are used as information.
[152] |  Predicting future collisions in real-world scenario setups using scene graphs |  Traffic accident anticipation |  Honda Driving dataset/$\checkmark$,
Scene-graph/$\checkmark$, DoTA/$\checkmark$ |  Spatio-temporal scene-graph embedding methodology with Graph Neural Network and Long Short-Term Memory are used for accident anticipation.
[153] |  Detecting traffic accidents using appearance, motion and context consistency learning in driving scenarios |  Traffic accident detection |  A3D/$\checkmark$, DADA-2000/$\checkmark$ |  Fusion technique is used to integrate appearance, trajectories, interactions, and contextual features of driving scenes, which are then processed by graph convolutional networks.
[154] |  Anticipating accidents based on a Graph and Spatio-temporal Continuity framework |  Traffic accident anticipation |  DAD/$\checkmark$, A3D/$\checkmark$ |  Graph convolution and recurrent networks are used for feature learning. Bayesian neural networks are used to model latent relational representations.

###  5.5 Based on object interactions

Analyzing object interactions and spatial relationships can also provide a
basis for detecting anomalies. Features like relative distance, speed,
direction, and interaction patterns between objects can be analyzed to
identify abnormal events that deviate from normal behavioral patterns. For
instance, in traffic surveillance scenarios, a sudden decrease in the distance
between two vehicles could indicate a potential collision. In the context of
the recent COVID-19 pandemic, the interpersonal distance between individuals
could be used as a measure to flag violations of social distancing rules [195,
196]. Furthermore, interpersonal interactions can also be analyzed to detect
abnormal behaviors such as physical violence or fighting [83]. Because
interactions are the result of individual motions, approaches based on object
interactions often require multi-object tracking as a pre-processing step.

Traffic accident anticipation problems can be addressed through the analysis
of vehicle interactions, employing spatial relationship learning with a Graph
Convolutional Network (GCN), and temporal relationship modeling with a
Recurrent Neural Network [49]. A Bayesian Deep Neural Network can be leveraged
to estimate the epistemic uncertainty associated with these predictions.
Alternatively, the same problem can be conceptualized as identifying irregular
object movements and sudden changes in spatial relations between different
vehicles across sequential frames in dashcam videos [153]. This approach
accomplishes its goal by predicting the visual scene context using a GCN,
taking into account the temporal frame consistency, temporal object location
consistency, and the spatio-temporal relationship consistency among the
participants on the road.

Occlusions in moving camera scenarios can often result in fragmented
trajectories or identity switches [197, 198]. To address this challenge, a
graph-based method can be deployed to reconstruct the spatio-temporal states
of occluded objects by learning their spatio-temporal relationships [154].
Object trajectories can then be aggregated over time to facilitate the
detection of interactions.

As deep learning-based method output predictions are often difficult to
explain, the work by Doshi and Yilmaz [155] introduces an approach for
detecting video anomalies grounded in scene graphs. This approach does not
only monitor objects independently but also monitors the interactions between
objects to detect anomalous events and provide explanations for their root
causes. Furthermore, this explainable methodology facilitates cross-domain
adaptability, allowing for transfer learning in different surveillance
scenarios.

In summary, methods based on object interactions in MC-VAD offer the
distinctive advantage of providing an additional layer of contextual
information through the analysis of relationships between objects. These
methods are capable of detecting anomalies that might be overlooked by other
approaches and often synergize effectively with methods that focus on object
trajectories. When compared with frame-level anomaly detection methods, these
techniques can also enable the explainability of anomaly detection. Moreover,
they face challenges such as the limited availability of contextual anomaly
datasets and the potential for occlusions in moving camera scenarios to
adversely affect the accuracy of anomaly detection. Tab. IX provides a summary
of these methods.

##  6 Outlook

MC-VAD is a recently emerging direction that has gained momentum in recent
years. Although numerous methods have been developed, they are often
scattered. In the previous section, we established an intuitive taxonomy
framework and interconnected previous research efforts. In this section, we
summarize the limitations of these works, discuss promising research trends,
and explore potential directions for the future.

###  6.1 Limitations of existing works

Existing MC-VAD algorithms are not without their limitations. Firstly, they
are often restricted to detecting only a handful of specific anomaly
categories, thus curtailing their versatility across a broad spectrum of
anomaly types. Secondly, they frequently suffer from poor generalizability,
implying the need for retraining on each unique target scene, which
subsequently increases the computational load on the algorithm. They are also
particularly vulnerable to a variety of external factors such as changes in
illumination, obstructions, and complex background scenarios. These factors
often result in frequent false positives or false negatives, consequently
lowering the precision and reliability of the detection process. These
combined limitations likely serve as the principal reasons why MC-VAD
algorithms have proven challenging to implement on a wide scale.

Specifically, most semantic segmentation methods are only capable of detecting
appearance anomalies at the scene level, such as the presence of abnormal
objects. They are limited in their ability to handle other types of scene-
level anomalies that involve motion and span across consecutive frames.
Furthermore, these methods often fail to effectively utilize temporal
information, resulting in the loss of abnormal targets in certain frames due
to occlusion, light reflections, and other interfering factors. The discarded
abnormal information in these frames can be key for tasks that require real-
time performance, such as obstacle detection. Recent works have begun
exploring the improvement of anomaly segmentation by considering temporal
context and reducing false positives.

Methods based on reference frames are limited by the requirement of reference
videos that do not contain any anomalies. They are prone to false positives
when the scene undergoes significant changes or when normal objects that were
not present in the reference frames appear frequently. When the camera
undergoes severe motion, it greatly affects the alignment of frames and
consequently reduces the accuracy of anomaly detection. However, the methods
based on background separation, object trajectory and object interaction are
affected by the effects of low-level algorithms such as detection and
tracking, and have high requirements for data quality.

###  6.2 Recent research trends and future directions

Based on our analysis of existing works, video anomaly detection in dynamic
scenes with moving cameras is a rapidly evolving research field with great
potential for improving traffic safety, environmental surveillance, and
autonomous vehicle technology. However, due to the inherent challenges of MC-
VAD and the limitations of current works, there are still some future research
directions that we deem to be interesting to be explored by the community.

  * •

Datasets and benchmark: Collecting and labeling anomalous events for MC-VAD
datasets remains to be challenging due to the rarity and diversity of
anomalies in the real world, with even reduced chances acquired by moving
cameras. Increasing the deployment of mobile robots for dataset capturing is
an option, yet the data quantity is limited. Thus it is crucial to encourage
dataset acquisition via various ways including simulation environments [25,
199, 45, 152] using game engines and crowd-sourcing [6, 2, 49, 50] via
platforms such as Youtube. In the former, synthetic anomaly data can be
generated using game engines in different scenarios, accompanied by precise
pixel-level annotations. The latter can provide a large number of real data.

Along with the datasets, establishing relevant benchmark evaluation metrics
will also allow researchers to effectively compare different algorithms and
techniques, driving the development of better-performing methods. Although
most VAD methods calculate anomaly scores for each frame and evaluate
performance using area under the curve (AUC) of receiver operating
characteristic (ROC) curves, this metric only measures the effectiveness of
temporal localization of anomalies while overlooking spatial detection
accuracy. This simplistic and incomplete evaluation metric makes it
challenging to assess the true capabilities of algorithms and lacks
interpretability in terms of results. Recent research has recognized this
limitation and proposed more comprehensive evaluation metrics to address these
issues [30, 50].

  * •

Multi-modal data fusion: Combining data from multiple sources like RGB video,
depth maps, LiDAR point clouds, and thermal images [200] can improve the
accuracy and robustness of anomaly detection systems, especially to deal with
the problems of low image quality and many external interference factors are
common in MC-VAD. The XD-Violence [201] dataset introduced audio information
and demonstrated the positive impact of audio-visual fusion on VAD. In
practical applications of MC-VAD, the camera is usually closer to the location
of anomalies, making it more likely to capture valuable audio information.
However, only a few datasets in the field of MC-VAD contain multimodal data,