# Active Reward Learning from Online Preferences ††thanks: The authors would
like to acknowledge the funding provided by the DARPA YFA award; NSF Awards
#2218760, #1953032, and #1941722; and the Office of Naval Research.

Vivek Myers1, Erdem Bıyık2, Dorsa Sadigh1,2 1Computer Science, Stanford
University, 2Electrical Engineering, Stanford University. Emails: vmyers,
ebiyik<EMAIL_ADDRESS>
###### Abstract

Robot policies need to adapt to human preferences and/or new environments.
Human experts may have the domain knowledge required to help robots achieve
this adaptation. However, existing works often require costly offline re-
training on human feedback, and those feedback usually need to be frequent and
too complex for the humans to reliably provide. To avoid placing undue burden
on human experts and allow quick adaptation in critical real-world situations,
we propose designing and sparingly presenting easy-to-answer pairwise action
preference queries in an online fashion. Our approach designs queries and
determines when to present them to maximize the expected value derived from
the queries’ information. We demonstrate our approach with experiments in
simulation, human user studies, and real robot experiments. In these settings,
our approach outperforms baseline techniques while presenting fewer queries to
human experts. Experiment videos, code and appendices are found on our
website:
http://tinyurl.com/online-active

##  I Introduction

Using trained robot policies in a zero-shot manner in real-world scenarios is
challenging for many reasons, including insufficient training data, changing
preferences of human users, or uncertainties present in dynamic environments.
Today’s robot algorithms need to fine-tune and adapt to the specifics of a
given environment or user preferences in an online fashion. For example, the
robot may need to safely explore the new environment and update its policy,
but this is often too costly. An alternative is to require the human
intervention by providing more data (such as demonstrations, physical
feedback, language corrections) [1], or by formally specifying new reward
functions [2, 3], which might be too difficult and expensive. In addition, the
common paradigm of learning from such feedback requires the robot to run its
old policy, gather information during this run, pause to train on this data,
update the policy based on the newly provided data, and to only then initiate
a new run. This is highly impractical in online settings, as we already
observe in interactive imitation learning [4, 5, 6, 7, 8].

Instead, we would like the robot to reduce its uncertainty and update its
policy by effectively asking humans questions and learning from their
responses in an _online fashion_. This requires the robot to generate
informative online questions that i) reduce the robot’s uncertainty, ii) can
easily be answered by humans, and iii) are critically timed, i.e., the robot
should query humans infrequently and at the right time. Prior work has
discovered pairwise comparisons as an effective form of feedback that can
easily be answered by humans to update the robot’s policy [9, 10, 11, 12, 13,
14]. For instance, active preference-based learning techniques are data-
efficient approaches that query humans with the most informative pairwise
comparison—asking the human to compare two different robot trajectories—based
on optimizing an information theoretic objective such as mutual information
[15], volume removal [16, 17], maximum regret [18, 19], dissimilarity [20,
21], or determinantal point processes [22]. However, these works still require
the policy to be updated during an offline training phase making them
impractical for online policy updates. The online setting we are interested in
best captures many real-world scenarios in which a robot is deployed in a
high-value situation where it is worth querying humans during the run to avoid
poor performance. These scenarios include self-driving settings, where
bothering a human is worth it to avoid an accident; high-level task planning
settings where robots need efficient clarification about which task to perform
next; and manipulation settings such as pushing, grasping, door opening, etc.
where the human may have strong preferences about the precise task to perform.
In such real-world settings, we additionally wish to avoid asking too many
questions so that humans can quickly and reliably respond to questions when
they are most needed.

This brings us to our key insight: we would like to model the robot’s
epistemic uncertainty – its uncertainty about the environment – captured by
the robot’s reward function. This uncertainty allows the robot to evaluate
_when_ to ask _the most informative_ pairwise comparisons to the human.
Specifically, we take a multitask learning perspective to this problem and
pretrain a library of robot reward functions on a number of tasks. In test
time, we maintain a posterior over different tasks by directly modeling the
effect of presenting queries to a human on the robot’s belief state. We can
thus compute the expected value of information (EVOI) [23, 24] of any
potential pairwise query corresponding to the expected value the robot gains
by asking that question. Using the EVOI metric, we propose an approach for
selecting when to query a human expert and what queries to make. This is
demonstrated through an example in Figure 1: here, the robot uses our approach
to query a human expert and push an object to a preferred goal location. When
the expected value of a question is high enough, we query the pair of actions
with the highest expected value of information. Our approach asks pairwise
comparison queries over the robot’s actions (or the next short sequence of
actions) that can be conveniently answered by humans. In addition, leveraging
the EVOI metric allows us to ask the most informative questions at the
critical time steps and thus update our policy in an online fashion.

Figure 1:  Overview of our approach to decide if and what to query each time
step. The robot must move the yellow cup to the goal location desired by the
human, denoted by the blue tape square. The robot does not observe this goal
location, and must query the human to determine where to move the cup. (i)
Keep a posterior $\omega$ over the task (location of the dark blue tape
square) conditioned on all past queries made to the human during the
trajectory. (ii) Consider potential preference queries $(a_{1},a_{2})$ between
two actions. (iii) Condition the posterior on each possible response to the
query. (iv) Use human model along with expected return of trajectories
$\xi_{1}$ and $\xi_{2}$ under conditioned $\omega$ posteriors to compute value
of query. Repeat for several query pairs of actions. (v) If the computed value
of any potential query exceeds threshold, send the highest-value query
considered to the human.

In sum, we make the following contributions:

  1. 1.

We propose a formulation of the online pairwise action preference querying
problem using a multitask setting to model reward uncertainty, and show how to
use this formulation to compute the value of asking questions.

  2. 2.

We show that our approach outperforms baselines by attaining a greater score
while asking fewer questions in a tabular GridWorld game, a driving
simulation, and continuous-action-space object-pushing robot environment.

  3. 3.

We conduct a user study with the driving simulation, showing users prefer the
questions and the performance of our approach compared to baselines. We
further demonstrate that our approach transfers to a real robot task allowing
it to move an object to a preferred location.

##  II Related Work

Existing work on incorporating human feedback in robot learning has focused on
interactive imitation learning as well as work on learning from other sources
of data such as physical feedback, language instructions and corrections,
rankings or pairwise comparisons. These works generally require extensive
human feedback and/or learn from experts offline, whereas our approach is the
only, to our knowledge, to both learn from humans using simple pairwise
queries and perform that querying in an online fashion.

Utilizing Human Experts Online. Most prior work on adaptive online querying of
human experts has focused on requesting full demonstrations. DAgger requires
rollouts from a policy that uses expert predictions [4], while ThriftyDAgger
[8] temporarily shifts control to human experts for partial demonstrations in
risky or uncertain situations. Similarly, approaches such as uncertainty-aware
action advising request demonstrations in high-uncertainty situations [25].
Building upon these interactive imitation learning work, HG-DAgger [5] and EIL
[26] require human experts to decide when to stage interventions. These works
require access to expert demonstrations which can be costly for robots with
high degrees of freedom [27, 28, 29, 30, 31], and further require a training
phase based on the collected data.

To avoid excessively burdening humans experts by requiring full
demonstrations, [24] leveraged EVOI and assumed the robot can move to any
state to ask for the optimal action in that state. Approaches such as TAMER,
COACH, and variants request continuous scalar feedback from human experts [32,
33, 34, 35]. Other works tap into other sources of data such as physical
corrections, language instructions or corrections, images, and rankings [14,
1, 36, 37]. While these works provide effective interfaces for collecting
human data, they all require an offline training phase which does not allow
for seamless online integration of new user inputs.

Preference-based Learning. Many works have examined reinforcement learning in
the context of human preferences [9]. In particular, past work tackled reward
learning in the context of preference-based queries [13, 16, 38, 17, 15, 39,
10, 40, 41]. These works adaptively request human rankings or comparisons over
trajectories to infer reward functions. [42] expand these techniques to ask
questions that demonstrate to human experts that the robot is learning from
their instruction, while [15] and [43] propose supplementing preferences with
diverse forms of human feedback. Meanwhile, within the student-teacher
reinforcement learning framework, [44] propose an approach where a teacher
agent gives action preferences to a student agent to help it learn quickly.

Other approaches such as T-REX attempt to learn reward functions that allow
effective policies to be trained from rankings of suboptimal trajectories
[45], while D-REX builds on this approach to learn from suboptimal
trajectories without requiring expert rankings [46].

Outside of reinforcement learning, preference-based active learning has been
used for classification tasks [47, 48] and ranking aggregation [47]. None of
these approaches have tackled our problem of adaptively asking for pairwise
comparisons between actions at deployment to conduct reward learning in an
online setting.

##  III Learning When to Make Informative Queries

We represent the robot’s uncertainty as a distribution over possible reward
functions. This distribution represents variability in human preferences about
how a task should be performed. We assume our reward function in our
environment is parameterized by a hidden latent task vector $\omega$. The
robot does not have access to this hidden task vector during training or
evaluation. However, it may during evaluation at any point ask the human for
their preference between two actions. The human responds with their preference
(up to some noise factor), assuming access to the true task vector.

Formalization. We consider a Markov Decision Process (MDP) of the form
$(S,A,P,R^{\omega})$ where $\omega$ is a task representation parameterizing
the reward function $R^{\omega}$. We assume there is some distribution
$\omega\sim\Omega$ which is known at train time and unknown at evaluation
time. Having access to $\omega$ at train-time represents that we are able to
learn different goal-conditioned policies during training, but must rely on
human feedback to get information about the goal when deployed during
evaluation.

Let us define $Q^{*}(s,a;\omega)$ to be the optimal $Q$-function for the MDP
$(S,A,P,R^{\omega})$ with optimal policy $\pi^{*}(s)=\arg\max_{a}Q^{*}(s,a)$.
At evaluation-time, at a given state $s_{t}\in S$ at time $t$, we allow the
robot to make an instantaneous action query of the form
$(a_{t}^{1},a_{t}^{2})\in A\times A$, asking an expert to choose between the
two possible actions. Denote the state the agent is currently in as $s\in S$.
We assume the human responses follow the Boltzmann-rational response model:

$H^{\pi}(a_{1},a_{2},s,\omega)=\frac{1}{1+e^{\beta(Q^{\pi}(s,a_{2};\omega)-Q^{\pi}(s,a_{1};\omega))}}$
(1)

with a fixed precision constant $\beta$ (higher values indicate a more
accurate human response model). The human returns $a_{1}$ with probability
$H^{\pi^{*}}(a_{1},a_{2},s,\omega)$ and returns $a_{2}$ otherwise with
probability $H^{\pi^{*}}(a_{2},a_{1},s,\omega)$. Assuming this response model
for pairwise comparisons between action queries, our goal is to find out
_when_ to ask _what_ questions to the user. This translates to the objective
of asking informative questions that _maximize the reward $R^{\omega}$_ at
evaluation, while also _minimizing the number of queries to the human expert_.
We next formalize this joint optimization.

###  III-A Querying Action Preferences

Training. During training, we assume we have access to a distribution of
training tasks $\Omega$, or samples from this distribution
$\omega_{1},\ldots,\omega_{n}\sim\Omega$. We train policies across the
distribution $\Omega$ with a method that learns a $Q$-function, such as DQN
[49]. We thus obtain $Q$ functions that are conditioned on the hidden task
vector as $Q^{\pi}(s,a;\omega)$.

Querying. We propose an approach based on the expected value of information
(EVOI) [23] for determining when it is optimal to make queries. Our key
insight is that using a family of task-parameterized $Q$-functions, we can
approximate the expected value of asking a question to an expert. We then use
a threshold $c$ over the computed EVOI to determine when to generate a query.
Intuitively, the threshold $c$ determines the willingness of the robot asking
a question.

At evaluation, we maintain a posterior over the task vector $\omega$. Using
this posterior, we can decide on which actions to take as well as compute the
EVOI of asking a human expert for a preference between two actions of the form
$(a_{1},a_{2})$. We initialize the posterior $\Omega$ using the prior from the
environment if it is known, and otherwise model the prior as a uniform
distribution over the sampled tasks we are given. We update this posterior in
response to each successive query made to the human expert, using the human
response model. Writing the past set of responses, query states, and queries
as
$\mathcal{D}=\Bigl{(}\bigl{(}a^{(1)}_{i_{1}},s^{(1)},(a^{(1)}_{1},a^{(1)}_{2})\bigr{)},\bigl{(}a^{(2)}_{i_{2}},s^{(2)},(a^{(2)}_{1},a^{(2)}_{2})\bigr{)},$
$\ldots,\bigl{(}a^{(n)}_{i_{n}},s^{(n)},(a^{(n)}_{1},a^{(n)}_{2})\bigr{)}\Bigr{)}$
for each $i_{\bullet}\in\left\\{1,2\right\\}$, we obtain the following
posterior update formula

$\displaystyle\Pr\left[\omega\mid\Omega,\mathcal{D}\right]\propto\Pr[\omega\mid\Omega]\prod_{j=1}^{n}H(a^{(j)}_{i_{j}},a^{(j)}_{3-i_{j}},s^{(j)},\omega).$
(2)

We marginalize across this posterior at evaluation to obtain the following
evaluation policy
$\pi(s)\\!=\\!\arg\max_{a}\mathbb{E}_{\omega\mid\Omega,\mathcal{D}}Q^{\pi}(s,a;\omega)$.
To compute the EVOI, we calculate the expected improvement in the expected $Q$
value of the action taken by the robot, marginalizing across the $\omega$
posterior over the human goal belief conditioned on past query responses. The
formulation selects queries based on how much they increase the value function
evaluated on the current state, which we can model using our human response
model and the effect of the queries on our posterior between reward functions.

One key assumption we make here in defining the EVOI is that the optimal
policy given our belief over $\omega$ is well-represented by $Q$-values
$\mathbb{E}_{\omega\mid\Omega,\mathcal{D}}Q^{\pi}(s,a;\omega)$. We make this
assumption to avoid the computational burden of needing to refit policies for
every possible distribution $\Omega\mid\mathcal{D}$ at each time step. This
heuristic of setting the expected $Q$ value to the weighted $Q$ value of
policies parameterized across tasks does result in optimistic myopic errors
when optimal task policies will disagree substantially about which action to
take in future.

A second assumption is made in the pessimistically myopic nature of this EVOI
objective—the objective approximates the expected gain in value function at
the current state when a query is asked assuming no further queries will be
made during the current trajectory.

Intuitively, these two forms of myopic error may actually counteract each
other. In cases where the $Q$ function approximation is overconfident due to
the first assumption (the optimal task policies will disagree in the future),
the EVOI for querying will likely be high anyways at those future points of
disagreement. This results in the $Q$ value heuristic overestimate capturing
the fact that the approach will do better than the myopic optimal policy that
makes no more queries. In other words, the heuristic of using the expected $Q$
across tasks is sometimes overly optimistic, but in such cases the
overconfidence stems from future states with high EVOI where uncertainty will
be resolved anyways. In practice, we indeed find that this heuristic of using
expected $Q$ performs well in our experimental settings.

Denote
$\mathcal{D}_{1}=\mathcal{D}\cup\left\\{(a_{1},s,(a_{1},a_{2}))\right\\}$ and
$\mathcal{D}_{2}=\mathcal{D}\cup\left\\{(a_{2},s,(a_{1},a_{2}))\right\\}$. We
obtain the following formula for the EVOI of the query $(a_{1},a_{2})$. A
complete derivation of this formula can be found in Section -A.

$\displaystyle\operatorname{EVOI}(a_{1},a_{2})=$
$\displaystyle\quad\mathop{\mathbb{E}}_{\omega\mid\Omega,\mathcal{D}}\Bigl{[}H^{\pi}(a_{1},a_{2},s,\omega)\mathop{\mathbb{E}}_{\omega^{\prime}\mid\Omega,\mathcal{D}_{1}}\max_{a}\left[Q^{\pi}(s,a;\omega^{\prime})\right]$
$\displaystyle\qquad+H^{\pi}(a_{2},a_{1},s,\omega)\\!\mathop{\mathbb{E}}_{\omega^{\prime}\mid\Omega,\mathcal{D}_{2}}\\!\max_{a}\left[Q^{\pi}(s,a;\omega^{\prime})\right]$
$\displaystyle\qquad-\max_{a}Q^{\pi}(s,a;\omega)\Bigr{]}$ (3)

When the EVOI of any query at the current state exceeds the query threshold
$c$ we make the query with the highest EVOI and update the posterior
appropriately.

###  III-B Continuous Action Spaces

In settings with a continuous action space, two issues arise: (1) computing
Equation 3 for every pair of actions and (2) computing the $\max_{a}Q$ terms
in Equation 3 for any action becomes intractable.

To solve (1), we randomly sample a fixed number of potential queries from the
action space, and query the highest-EVOI of these queries if it exceeds $c$.
To solve (2), we assume access to task-specific policies
$\pi(s;\omega)\\!=\\!\arg\max_{a}Q^{\pi}(s,a;\omega)$, allowing us to compute
$\max_{a}Q^{\pi}(s,a;\omega)\\!=\\!Q^{\pi}(s,\pi(s;\omega);\omega)$. Actor-
critic approaches can be used to maintain an approximate $\pi(s;\omega)$ value
[50, 51], allowing us to approximate Equation 3. In the continuous setting
when not querying, we approximate the optimal action by using the policy
$\pi(s)\\!=\\!\mathop{\mathbb{E}}_{\omega\mid\Omega,\mathcal{D}}\pi(s;\omega)$.

Selecting Hyperparameters. To select the constant $\beta$, we pick a value
consistent with the scale of the rewards in our environment that models our
belief over the accuracy of the human experts in intuitively assessing the
goal-conditioned optimal $Q$-functions. To select $c$, we find a value that
empirically yields the desired number of queries made by our agent either in
simulation or by examining data from another source such as the replay buffer
during training.

##  IV Experiments

We conduct a number of simulated experiments in a GridWorld111To focus more on
the other experiments where we work with real human users or a real robot, we
present the results of the GridWorld simulations in Section -B, which can be
found on the website. and a driving environment, perform a user study where
humans respond to action preference queries in the driving domain, and
demonstrate our algorithm can be run on a real robot arm for a reaching task.

Baselines. We compare our approach against 2 baselines.

_Random._ We present a random baseline policy that queries the top two actions
in terms of $Q$-value with a fixed probability. In continuous action-space
settings, the second-best action is not well-defined, so we instead select
random actions to query as well.

_Uncertainty._ The uncertainty baseline uses a method similar to the novelty
heuristic in ThriftyDAgger [8] to decide when to query. In particular, we
present a query when the novelty of the current state is high (the variance of
the $Q$-value of the best action integrating across
$\omega\mid\Omega,\mathcal{D}$ exceeds a threshold). Unlike ThriftyDAgger [8],
we cannot request full demonstrations in our setting. We instead query the top
two actions by expected $Q$-value when our approach decides to query. Again,
in continuous action-space settings, the second-best action is not well-
defined, so instead we select actions by the amount they reduce the
uncertainty of the $Q$-value of the action being selected.

To compare approaches in simulation, we simulate an expert policy which
assumes access to the task representation as a proxy for a human expert which
can answer queries made by the robot. We hypothesize that our approach will
outperform baseline approaches in terms of the tradeoff between the task
performance and the number of queries.

###  IV-A Driving Simulation Experiments

We conduct experiments in a simulated driving environment [52] where the goal
is to control a car driving on a crowded highway, and the action space
consists of $5$ discrete actions: $A=\\{\text{shift left},\text{shift
right},\text{slower},\text{faster},\text{idle}\\}$. The tasks (different
$\omega$) in this setting correspond to different preferences over the desired
lane and lane changes, speed range, acceleration, and following distance. We
train our policies across these tasks using DQNs [49, 53].

Figure 2: In the driving environment, our approach needs fewer queries on
average than either baseline, while significantly outperforming both of them
in average score across tasks.

We compared our approach against baselines in this environment across random
different task initializations in Figure 2, using a simulated human expert
modeled by a DQN with access to the task $\omega$. We tuned the $c$ parameter
in simulation to ensure our approach and baselines made similar numbers of
queries for fairness. An example of a decision made by our method is shown in
Figure 3.

Figure 3:  The ego car (green) is in a difficult situation where its desired
speed is higher than the car in the front, but it cannot shift further left. A
human with aggressive preferences may prefer to shift right one lane and try
to overtake the car in front, while a less aggressive human would likely
prefer slowing down. Since these different preferences dramatically affect the
desirability of each approach, not knowing how aggressive the human driver is,
our approach queries for their preference between slowing down and overtaking
by turning right, updating its belief about the human based on their response.

We then vary the querying threshold for each method (e.g., $c$ for ours) to
analyze the tradeoff between number of queries and the performance (score) in
the task. Pareto frontiers showing this tradeoff across different querying
parameters are presented in Figure 6, demonstrating one can tune the $c$
constant of our method to set this tradeoff. These Pareto frontiers suggest
our approach performs robustly and outperforms baselines.

TABLE I: User Study Results | Subjective Driving Metrics | Objective Driving Metrics
---|---|---
Method |  Important Points |  Reasonable Options |  Intelligent Questions |  Adapted Well |  Drove Well |  Speed Non-Adherence |  Lane Adherence |  Number of Queries |  Repetitive Questions
Ours | $\mathbf{5.32\\!\pm\\!0.26}$ | $\mathbf{4.95\\!\pm\\!0.23}$ | $\mathbf{4.86\\!\pm\\!0.18}$ | $\mathbf{5.32\\!\pm\\!0.19}$ | $\mathbf{5.36\\!\pm\\!0.20}$ | $\mathbf{7.94\\!\pm\\!0.68}$ | $\mathbf{0.56\\!\pm\\!0.10}$ | $\mathbf{8.24\\!\pm\\!0.40}$ | $\mathbf{0.00\\!\pm\\!0.00}$
Uncertainty | $4.00\\!\pm\\!0.42$ | $\mathbf{4.82\\!\pm\\!0.36}$ | $\mathbf{4.27\\!\pm\\!0.42}$ | $\mathbf{5.05\\!\pm\\!0.39}$ | $\mathbf{4.73\\!\pm\\!0.41}$ | $14.7\\!\pm\\!2.2$ | $\mathbf{0.51\\!\pm\\!0.08}$ | ${10.33\\!\pm\\!0.31}$ | $8.51\\!\pm\\!0.29$
Random | $4.23\\!\pm\\!0.30$ | $3.41\\!\pm\\!0.31$ | $3.55\\!\pm\\!0.33$ | $4.23\\!\pm\\!0.32$ | $4.32\\!\pm\\!0.36$ | $11.3\\!\pm\\!1.6$ | $0.47\\!\pm\\!0.05$ | $\mathbf{8.81\\!\pm\\!0.33}$ | $1.03\\!\pm\\!0.10$

###  IV-B Driving User Studies

With IRB approval from Stanford University Research Compliance Office, we
repeated the driving experiments on real humans instead of simulated experts.

Experimental Setup. We conducted our experiment using an online web interface.
Subjects completed a pre-experiment survey in which they stated their gender,
preferred lane, and preferred speed (between $20$ m/s and $30$ m/s). We
additionally asked the subjects to state whether they preferred their car to
adhere to their speed preferences or lane preferences more. Subjects were told
to express consistent preferences, and, at the end, were asked to subjectively
rate the effectiveness of each algorithm.

Procedure. We gathered data from 22 subjects (11 female, 10 male, 1 preferred
not to answer), and each subject responded to queries online over the course
of 5 trajectories for each algorithm (our approach or one of the two
baselines, unknown to the subjects) in randomized order.

Independent Variables. The querying method used, either our approach or one of
the two baselines, is the independent variable.

Dependent Measures. Subjective measures are the evaluations performed by the
subjects after each algorithm: a seven-point scale survey of how much they
agree with the statements: “The car asked questions at the important decision
points,” “The questions included choices between reasonable options,”
“Overall, the questions seemed intelligently timed and picked,” “Over time,
the car adapted to my driving preferences,” and “Overall, the car drove in the
way I wanted.”

Objective measures include the speed non-adherence, the percentage relative
difference between the average speed of the car and the subject’s preferred
speed; lane adherence, the proportion of the time the car stayed in the
subject’s preferred lane; as well as the number of queries needed by the
algorithm and the number of repetitive questions asked, defined as consecutive
steps where the algorithm asked the same question. Since adhering to speed
preferences and lane preferences often conflict (when strongly adhering to a
lane, speed is strongly determined by the other cars in that lane), we only
included the subjects who preferred speed or lane adherence in the respective
metric computations.

Hypotheses. (1) Our approach will more efficiently and better capture user
preferences than baselines, indicated by obtaining higher scores on the
subjective dependent measures. (2) Our approach will perform objectively
better than baselines, indicated by greater speed adherence and lane
adherence, and also quantitatively be more query efficient, indicated by using
a lower number of queries and making fewer repetitive queries.

Results. Table I shows the different subjective assessments of the various
approaches given by users with standard errors. The best performing approaches
for each metric (highest or lowest contextually) up to statistical
significance are bolded (Wilcoxon signed-rank with a $p$-value of $0.05$). Our
approach achieves the best performance across all metrics, and is
statistically significantly better for every metric in comparison to random,
and statistically significantly better compared to uncertainty for the
“Important Points” metric, supporting Hypothesis 1. Table I also shows the
different objective assessments with standard errors. The best approaches are
again bolded similarly. Our approach achieves the best performance across all
metrics, supporting Hypothesis 2.

###  IV-C Real Robot Block Pushing Experiments

Figure 4: Our approach makes statistically significantly fewer queries on
average than all the baseline approaches in block-pushing simulation, while
significantly outperforming them in average score across tasks.  Figure 5:
Example trajectories generated by our method on the Fetch robot with a
simulated human expert. The goal location (blue tape square) is observed by
the simulated expert but not our approach.

We first consider a simulated environment where a robot arm must be used to
push a block to a goal location. We view the block location as the task
$\omega$ in this setting, so to succeed in this setting, the robot must query
the expert to gain information about the goal location. Since the action space
is continuous, we use a trained soft actor–critic policy [54, 53] as described
in Section III-A. We conducted experiments in simulation using simulated
expert policies, and found that our approach outperformed baselines in score
while needing fewer queries (see Figure 4). We additionally constructed Pareto
frontiers in Figure 6 modeling the tradeoff between queries made and scores
achieved for all methods, similar to driving simulations. These again show our
method robustly outperforms baselines when asking similar numbers of
questions. See Section -D for the experiment details on the block pushing
setting.

Figure 6: Pareto frontiers comparing approaches for the driving and block
pushing environment. Error bars are one standard error.

Sim2Real Transfer. We then ran the approach on a real Fetch robot arm [55],
using policies trained in the simulation. On initializations where our method
succeeded in simulation, we were also able to transfer to the real robot and
push the yellow cup to an arbitrary goal location, as shown in Figure 5.
Further details about this experiment and the videos are on the website.

##  V Discussion

Summary. In this paper, we proposed an approach that designs easy-to-answer
pairwise action preference queries in an online fashion. Our work maximizes
the expected value derived from the queries’ information and asks questions
from humans at the most informative points of time. We demonstrated our
approach in simulated GridWorld, a driving simulation, and a pushing task with
a real robot. Across all these settings our approach outperformed baselines
while needing fewer questions.

Limitations and Future Work. One limitation of this work is the potential
difficulty of answering preferences between actions. While the human response
model in Equation 1 accounts for uncertainty in distinguishing similarly-
valued actions, there is a challenge in effectively representing queries for
humans in high DoF action spaces. In discrete settings where actions are
easily interpretable, this is not an issue. In continuous settings such as
most robot manipulation tasks, it can be harder for humans to interpret and
compare actions without good representations—a potential solution is to use a
simulator to present visualizations of both actions to human experts. In
future work, we plan to filter queries to ensure they are easily answerable
with a visualization.

An additional limitation is the assumption that queries can be immediately
answered. In settings that require fast responses such as urban driving
environments this assumption may be difficult. A future approach to tackle
this problem would be to train a dynamics model and use this model to predict
EVOI and design queries multiple steps in advance, giving experts more time to
respond.

In the future, we plan to apply this approach to more real-world robot tasks,
such as Meta-World [56]. Additional exploration of complex simulation
environments and baseline approaches would also help to validate our method.

Finally, theoretical analysis should be conducted on optimality, in particular
with regard to (1) the effect of the approximations made to extend to
continuous settings in Section III-B, (2) the selection of $c$ and $\beta$
hyperparameters, and (3) the competing effects of the pessimistic and
optimistic myopia of our approach discussed in Section III-A.

## References

  * [1] Mengxi Li, Alper Canberk, Dylan P Losey and Dorsa Sadigh  “Learning human objectives from sequences of physical corrections”  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , 2021, pp. 2877–2883  IEEE
  * [2] Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell and Anca Dragan  “Inverse reward design”  In _Advances in neural information processing systems_ 30, 2017
  * [3] Ellis Ratner, Dylan Hadfield-Menell and Anca D Dragan  “Simplifying reward design through divide-and-conquer”  In _arXiv preprint arXiv:1806.02501_ , 2018
  * [4] Stéphane Ross and Drew Bagnell  “Efficient reductions for imitation learning”  In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_ , 2010, pp. 661–668  JMLR WorkshopConference Proceedings
  * [5] Michael Kelly, Chelsea Sidrane, Katherine Driggs-Campbell and Mykel J Kochenderfer  “Hg-dagger: Interactive imitation learning with human experts”  In _2019 International Conference on Robotics and Automation (ICRA)_ , 2019, pp. 8077–8083  IEEE
  * [6] Kunal Menda, Katherine Driggs-Campbell and Mykel J Kochenderfer  “Ensembledagger: A bayesian approach to safe imitation learning”  In _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2019, pp. 5041–5048  IEEE
  * [7] Ryan Hoque, Ashwin Balakrishna, Carl Putterman, Michael Luo, Daniel S Brown, Daniel Seita, Brijen Thananjeyan, Ellen Novoseller and Ken Goldberg  “Lazydagger: Reducing context switching in interactive imitation learning”  In _2021 IEEE 17th International Conference on Automation Science and Engineering (CASE)_ , 2021, pp. 502–509  IEEE
  * [8] Ryan Hoque, Ashwin Balakrishna, Ellen Novoseller, Albert Wilcox, Daniel S Brown and Ken Goldberg  “ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning”  In _5th Annual Conference on Robot Learning_ , 2021
  * [9] Christian Wirth, Riad Akrour, Gerhard Neumann and Johannes Fürnkranz  “A survey of preference-based reinforcement learning methods”  In _Journal of Machine Learning Research_ 18.136 Journal of Machine Learning Research/Massachusetts Institute of Technology …, 2017, pp. 1–46
  * [10] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg and Dario Amodei  “Deep reinforcement learning from human preferences”  In _Advances in neural information processing systems_ 30, 2017
  * [11] Maya Cakmak, Siddhartha S Srinivasa, Min Kyung Lee, Jodi Forlizzi and Sara Kiesler  “Human preferences for robot-human hand-over configurations”  In _2011 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2011, pp. 1986–1993
  * [12] Kejun Li, Maegan Tucker, Erdem Biyik, Ellen Novoseller, Joel W. Burdick, Yanan Sui, Dorsa Sadigh, Yisong Yue and Aaron D. Ames  “ROIAL: Region of Interest Active Learning for Characterizing Exoskeleton Gait Preference Landscapes”  In _International Conference on Robotics and Automation (ICRA)_ , 2021
  * [13] Erdem Biyik, Nicolas Huynh, Mykel J. Kochenderfer and Dorsa Sadigh  “Active Preference-Based Gaussian Process Regression for Reward Learning”  In _Proceedings of Robotics: Science and Systems (RSS)_ , 2020
  * [14] Erdem Biyik, Dylan P. Losey, Malayandi Palan, Nicholas C. Landolfi, Gleb Shevchuk and Dorsa Sadigh  “Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences”  In _The International Journal of Robotics Research (IJRR)_ , 2021
  * [15] Erdem Biyik, Malayandi Palan, Nicholas C. Landolfi, Dylan P. Losey and Dorsa Sadigh  “Asking Easy Questions: A User-Friendly Approach to Active Reward Learning”  In _Proceedings of the 3rd Conference on Robot Learning (CoRL)_ , 2019
  * [16] Dorsa Sadigh, Anca D. Dragan, S. Shankar Sastry and Sanjit A. Seshia  “Active Preference-Based Learning of Reward Functions”  In _Proceedings of Robotics: Science and Systems (RSS)_ , 2017
  * [17] Erdem Biyik and Dorsa Sadigh  “Batch active preference-based learning of reward functions”  In _Conference on robot learning_ , 2018, pp. 519–528  PMLR
  * [18] N. Wilde, D. Kulić and S. L. Smith  “Active Preference Learning using Maximum Regret”  In _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2020, pp. 10952–10959
  * [19] Nils Wilde, Erdem Biyik, Dorsa Sadigh and Stephen L. Smith  “Learning Reward Functions from Scale Feedback”  In _Proceedings of the 5th Conference on Robot Learning (CoRL)_ , 2021
  * [20] Sydney M Katz, Anne-Claire Le Bihan and Mykel J Kochenderfer  “Learning an urban air mobility encounter model from expert preferences”  In _2019 IEEE/AIAA 38th Digital Avionics Systems Conference (DASC)_ , 2019, pp. 1–8  IEEE
  * [21] Sydney M Katz, Amir Maleki, Erdem Bıyık and Mykel J Kochenderfer  “Preference-based learning of reward function features”  In _arXiv preprint arXiv:2103.02727_ , 2021
  * [22] Erdem Biyik, Kenneth Wang, Nima Anari and Dorsa Sadigh  “Batch Active Learning Using Determinantal Point Processes”  In _arXiv preprint arXiv:1906.07975_ , 2019
  * [23] Paolo Viappiani and Craig Boutilier  “Optimal Bayesian Recommendation Sets and Myopically Optimal Choice Query Sets.”  In _NeurIPS_ , 2010, pp. 2352–2360
  * [24] Robert Cohn, Edmund Durfee and Satinder Singh  “Comparing action-query strategies in semi-autonomous agents”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 25.1, 2011, pp. 1102–1107
  * [25] Felipe Leno Da Silva, Pablo Hernandez-Leal, Bilal Kartal and Matthew E Taylor  “Uncertainty-aware action advising for deep reinforcement learning agents”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 34.04, 2020, pp. 5792–5799
  * [26] Jonathan Spencer, Sanjiban Choudhury, Matthew Barnes, Matthew Schmittle, Mung Chiang, Peter Ramadge and Sidd Srinivasa  “Expert intervention learning”  In _Autonomous Robots_ 46.1 Springer, 2022, pp. 99–113
  * [27] Brenna D Argall, Sonia Chernova, Manuela Veloso and Brett Browning  “A survey of robot learning from demonstration”  In _Robotics and autonomous systems_ 57.5 Elsevier, 2009, pp. 469–483
  * [28] Baris Akgun, Maya Cakmak, Karl Jiang and Andrea L Thomaz  “Keyframe-based learning from demonstration”  In _International Journal of Social Robotics_ 4.4 Springer, 2012, pp. 343–355
  * [29] Anca D Dragan and Siddhartha S Srinivasa  “Formalizing assistive teleoperation”  MIT Press, July, 2012
  * [30] Shervin Javdani, Siddhartha S Srinivasa and J Andrew Bagnell  “Shared autonomy via hindsight optimization”  In _Robotics science and systems: online proceedings_ 2015 NIH Public Access, 2015
  * [31] Rebecca P Khurshid and Katherine J Kuchenbecker  “Data-driven motion mappings improve transparency in teleoperation”  In _Presence_ 24.2 MIT Press, 2015, pp. 132–154
  * [32] W Bradley Knox and Peter Stone  “Tamer: Training an agent manually via evaluative reinforcement”  In _2008 7th IEEE international conference on development and learning_ , 2008, pp. 292–297  IEEE
  * [33] Carlos Celemin and Javier Solar  “COACH: Learning continuous actions from corrective advice communicated by humans”  In _2015 International Conference on Advanced Robotics (ICAR)_ , 2015, pp. 581–586  IEEE
  * [34] Riku Arakawa, Sosuke Kobayashi, Yuya Unno, Yuta Tsuboi and Shin-ichi Maeda  “Dqn-tamer: Human-in-the-loop reinforcement learning with intractable feedback”  In _arXiv preprint arXiv:1810.11748_ , 2018
  * [35] Garrett Warnell, Nicholas Waytowich, Vernon Lawhern and Peter Stone  “Deep tamer: Interactive agent shaping in high-dimensional state spaces”  In _Proceedings of the AAAI conference on artificial intelligence_ 32.1, 2018
  * [36] Andrea Bajcsy, Dylan P Losey, Marcia K O’Malley and Anca D Dragan  “Learning from physical human corrections, one feature at a time”  In _Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction_ , 2018, pp. 141–149
  * [37] Stewart Jamieson, Jonathan P How and Yogesh Girdhar  “Active Reward Learning for Co-Robotic Vision Based Exploration in Bandwidth Limited Environments”  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 1806–1812  IEEE
  * [38] Vivek Myers, Erdem Biyik, Nima Anari and Dorsa Sadigh  “Learning multimodal rewards from rankings”  In _Conference on Robot Learning_ , 2022, pp. 342–352  PMLR
  * [39] Maegan Tucker, Ellen Novoseller, Claudia Kann, Yanan Sui, Yisong Yue, Joel W Burdick and Aaron D Ames  “Preference-based learning for exoskeleton gait optimization”  In _2020 IEEE international conference on robotics and automation (ICRA)_ , 2020, pp. 2351–2357  IEEE
  * [40] Chandrayee Basu, Erdem Bıyık, Zhixun He, Mukesh Singhal and Dorsa Sadigh  “Active learning of reward dynamics from hierarchical queries”  In _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2019, pp. 120–127  IEEE
  * [41] Donald Joseph Hejna and Dorsa Sadigh  “Few-Shot Preference Learning for Human-in-the-Loop RL”  In _Proceedings of the 6th Conference on Robot Learning (CoRL)_ , 2022
  * [42] Soheil Habibian, Ananth Jonnavittula and Dylan P Losey  “Here’s what I’ve learned: Asking questions that reveal reward learning”  In _ACM Transactions on Human-Robot Interaction_ ACM New York, NY, 2022
  * [43] Hong Jun Jeon, Smitha Milli and Anca Dragan  “Reward-rational (implicit) choice: A unifying formalism for reward learning”  In _Advances in Neural Information Processing Systems_ 33, 2020, pp. 4415–4426
  * [44] Matthieu Zimmer, Paolo Viappiani and Paul Weng  “Teacher-student framework: a reinforcement learning approach”  In _AAMAS Workshop Autonomous Robots and Multirobot Systems_ , 2014
  * [45] Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan and Scott Niekum  “Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations”  In _International conference on machine learning_ , 2019, pp. 783–792  PMLR
  * [46] Daniel S Brown, Wonjoon Goo and Scott Niekum  “Better-than-demonstrator imitation learning via automatically-ranked demonstrations”  In _Conference on robot learning_ , 2020, pp. 330–359  PMLR
  * [47] Xi Chen, Paul N Bennett, Kevyn Collins-Thompson and Eric Horvitz  “Pairwise ranking aggregation in a crowdsourced setting”  In _Proceedings of the sixth ACM international conference on Web search and data mining_ , 2013, pp. 193–202
  * [48] Lin Chen, Hamed Hassani and Amin Karbasi  “Near-optimal active learning of halfspaces via query synthesis in the noisy setting”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 31.1, 2017
  * [49] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra and Martin Riedmiller  “Playing atari with deep reinforcement learning”  In _arXiv preprint arXiv:1312.5602_ , 2013
  * [50] Vijay Konda and John Tsitsiklis  “Actor-critic algorithms”  In _Advances in neural information processing systems_ 12, 1999
  * [51] Ivo Grondman, Lucian Busoniu, Gabriel AD Lopes and Robert Babuska  “A survey of actor-critic reinforcement learning: Standard and natural policy gradients”  In _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ 42.6 IEEE, 2012, pp. 1291–1307
  * [52] Edouard Leurent  “An Environment for Autonomous Driving Decision-Making”  In _GitHub repository_ GitHub, https://github.com/eleurent/highway-env, 2018
  * [53] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus and Noah Dormann  “Stable-Baselines3: Reliable Reinforcement Learning Implementations”  In _Journal of Machine Learning Research_ 22.268, 2021, pp. 1–8
  * [54] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta and Pieter Abbeel  “Soft actor-critic algorithms and applications”  In _arXiv preprint arXiv:1812.05905_ , 2018
  * [55] Melonee Wise, Michael Ferguson, Derek King, Eric Diehr and David Dymesich  “Fetch and freight: Standard platforms for service robot applications”  In _Workshop on autonomous mobile service robots_ , 2016
  * [56] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn and Sergey Levine  “Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning”  In _Conference on robot learning_ , 2020, pp. 1094–1100  PMLR
  * [57] David Yu-Tung Hui, Maxime Chevalier-Boisvert, Dzmitry Bahdanau and Yoshua Bengio  “BabyAI 1.1”  In _arXiv preprint arXiv:2007.12770_ , 2020
  * [58] Mance E Harmon and Stephanie S Harmon  “Reinforcement Learning: A Tutorial.”  Wright Lab Wright-Patterson Afb Oh, 1997
  * [59] Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej and Peter Welinder  “Multi-goal reinforcement learning: Challenging robotics environments and request for research”  In _arXiv preprint arXiv:1802.09464_ , 2018

In the appendix, we present details about our approach and experiments. See
http://tinyurl.com/online-active for a website summarizing our results.

###  -A Derivation

We adopt the setting in Section III. We define the set of past query responses
to be
$\mathcal{D}=\Bigl{(}\bigl{(}a^{(1)}_{i_{1}},s^{(1)},(a^{(1)}_{1},a^{(1)}_{2})\bigr{)},\bigl{(}a^{(2)}_{i_{2}},s^{(2)},(a^{(2)}_{1},a^{(2)}_{2})\bigr{)},\ldots,\bigl{(}a^{(n)}_{i_{n}},s^{(n)},(a^{(n)}_{1},a^{(n)}_{2})\bigr{)}\Bigr{)}$
for each $i_{\bullet}\in\left\\{1,2\right\\}$, and denote
$\mathcal{D}_{1}=\mathcal{D}\cup\left\\{(a_{1},s,(a_{1},a_{2}))\right\\},$
$\mathcal{D}_{2}=\mathcal{D}\cup\left\\{(a_{2},s,(a_{1},a_{2}))\right\\}$ for
fixed actions $(a_{1},a_{2})$ to potentially query in state $s$. The EVOI is
defined as the expected change in the value function
$V(s;\omega)=\max_{a}Q^{*}(s,a;\omega)$ of the current state after asking a
query and updating our belief state about the policy $Q$ function. Assuming
optimality of our learned $\max_{a}Q^{\pi}(s,a;\omega)$, we obtain the
following formula for the EVOI of the query $(a_{1},a_{2})$.

$\displaystyle\operatorname{EVOI}(a_{1},a_{2})$
$\displaystyle=\mathop{\mathbb{E}}_{\text{query
response}}\left[\mathop{\mathbb{E}}_{\omega\mid\text{after
query}}V(s;\omega)\right]-\mathop{\mathbb{E}}_{\omega\mid\text{before
query}}V(s;\omega)$
$\displaystyle=\biggl{[}\mathop{\mathbb{E}}_{\omega\mid\Omega,\mathcal{D}}H^{\pi^{*}}(a_{1},a_{2},s,\omega)\biggr{]}\mathop{\mathbb{E}}_{\omega\mid\Omega,\mathcal{D}_{1}}V(s;\omega)+\biggl{[}\mathop{\mathbb{E}}_{\omega\mid\Omega,\mathcal{D}}H^{\pi^{*}}(a_{2},a_{1},s,\omega)\biggr{]}\mathop{\mathbb{E}}_{\omega\mid\Omega,\mathcal{D}_{2}}V(s;\omega)-\mathop{\mathbb{E}}_{\omega\mid\Omega,\mathcal{D}}V(s.;\omega)$
$\displaystyle=\mathop{\mathbb{E}}_{\omega\mid\Omega,\mathcal{D}}\Bigl{[}H^{\pi}(a_{1},a_{2},s,\omega)\mathop{\mathbb{E}}_{\omega^{\prime}\mid\Omega,\mathcal{D}_{1}}[V(s;\omega^{\prime})]+H^{\pi}(a_{2},a_{1},s,\omega)\mathop{\mathbb{E}}_{\omega^{\prime}\mid\Omega,\mathcal{D}_{2}}[V(s;\omega^{\prime})]-V(s;\omega)\Bigr{]}$
$\displaystyle=\mathop{\mathbb{E}}_{\omega\mid\Omega,\mathcal{D}}\Bigl{[}H^{\pi}(a_{1},a_{2},s,\omega)\mathop{\mathbb{E}}_{\omega^{\prime}\mid\Omega,\mathcal{D}_{1}}\max_{a}Q^{\pi}(s,a;\omega^{\prime})$
$\displaystyle\qquad\qquad+H^{\pi}(a_{2},a_{1},s,\omega)\mathop{\mathbb{E}}_{\omega^{\prime}\mid\Omega,\mathcal{D}_{2}}\max_{a}Q^{\pi}(s,a;\omega^{\prime})-\max_{a}Q^{\pi}(s,a;\omega)\Bigr{]}$

###  -B GridWorld Experiments

Figure 7: GridWorld variants studied. Agent start position and orientation are
shown by the large orange arrow. Empty spaces are gray, walls are black, and
lava is red. The goal location is randomized uniformly across empty spaces at
the start of each episode. In the diagrams above, we show a trajectory of our
approach on each environment for a randomly-selected goal location, as well as
the queries made by our method (with expert responses highlighted in orange).

Figure 8: Pareto-frontiers of methods on the GridWorld environment variants.
Each point in the plots represents the number of queries made and average
score of running a method for a particular initialization of querying
parameters. By varying the querying thresholds for each approach, we see the
trade offs between score and number of questions asked. For our approach, we
additionally label the values of $c$ used to generate some points to compare
robustness to different $c$ values across GridWorld settings.

Our initial results test our algorithm in a GridWorld environment [57]. This
environment involves an agent navigating a grid to reach a goal destination.
In this setting, our task distribution $\Omega$ is a uniform distribution over
possible valid goal locations. We train a policy conditioned on goal locations
$\omega\in\Omega$ to navigate these environments using a tabular model trained
with value iteration [58]. We then compare our querying approach against the
baseline approaches described above, using the learned policy conditioned on
the true goal as our expert policy. We focus on the following three GridWorld
environment variants, shown in Figure 7: Empty Variant: The agent starts in
the upper left corner of a 8x8 GridWorld and navigates to a random goal
location within the grid. Maze Variant: The agent starts in the upper left
corner of a 8x8 GridWorld and navigates to a random goal location within the
grid, avoiding a maze of walls and lava. Rooms Variant: The agent starts in
the upper left corner of a 15x15 GridWorld environment, partitioned into
multiple room sections with walls and lava, and navigates to a random grid
location.

We evaluate our approach against baselines in these environments by comparing
the Pareto frontiers of their average performance in terms of score achieved
and average number of queries made per trajectory. We vary the expected number
of queries made by our approach and baselines by varying the querying
parameters of each method ($c$ for our approach). By plotting the Pareto
frontiers of our approach against baselines, we can compare for a fixed number
of queries made by an approach, the expected score achieved in an environment.
These Pareto frontiers are presented in Figure 8.

Our approach generally outperforms both baseline approaches, achieving higher
score (y-axis value) for given average numbers of questions asked (x-axis
value). Interestingly, the uncertainty baseline performs significantly worse
than the random baseline in this setting. This performance is likely due to to
the fact that in a GridWorld trajectory, even though there is high uncertainty
over the goal location, all policies are able to get roughly similar $Q$
values since the environment is fully solvable, resulting in queries that may
be strongly disconnected from imp ortant decision points. Since the
uncertainty querying is deterministic, unlike random, it can then get in
states where it needs to ask a question but never will, resulting in very poor
performance.

We additionally label some of the points in the Pareto frontier for our
approach with the $c$ values used to generate them. By comparing the location
of the points with the same value of $c$ across GridWorld settings we can see
the value of $c$ transfers across similar settings. For instance, in all three
settings, once $c$ drops below $\approx\\!0.0012$ the agent starts to exceed a
score of $0.5$ and use $5$-$10$ queries. Thus, similar $c$ values tend to
result in similar querying behavior and performance across different GridWorld
settings.

####  -B1 Details

The action space in the GridWorld environment contains three actions:
$A=\\{\text{turn left},\text{turn right},\text{move forward}\\}$. The agent
cannot move through wall spaces, and the episode immediately ends upon moving
into a lava square. To compute parteo-frontiers, we vary the querying
parameters for our three approaches as follows:

Ours:

parameter varies from $10^{-4}$ to $10^{-1}$ with a step size of $\log(1.05)$
in log space.

Uncertainty:

parameter varies from $10^{-4}$ to $10^{1}$ with a step size of $\log(1.05)$
in log space.

Random:

parameter varies from $0.05$ to $0.5$ with a step size of $\log(1.05)$ in log
space.

Across experiments, we use a human response precision parameter of $\beta=10$.
We used value iteration to directly solve for optimal policies across tasks,
using $\gamma=0.99$ and a fixed horizon of 50 steps. These and all experiments
in this paper were run using Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz
processors.

###  -C Driving Details

We use a human response precision parameter of $\beta=10$ across all
experiments. For the the driving environment experiments in Section IV-A, we
use $c=0.05$ for our approach, a querying probability of $0.2$ for the random
baseline, and an uncertainty threshold of $46$ for the uncertainty baseline.
These hyperparameters were tuned so that the approaches ask similar numbers of
questions to better facilitate comparisons of our approach against baselines.

Our DQNs for use by agents and expert policies were trained across a random
distribution of tasks, with 100 tasks with trained DQN policies used for
controlling agents and the remaining 487 tasks and trained DQN policies used
as evaluation tasks and human expert simulators respectively. We trained our
DQNs for 60,000 steps with a batch size of 32, a replay buffer of size 15,000,
an Adam learning rate of $5\cdot 10^{-4}$, $\gamma=0.8$, and a two-hidden-
layer, 256 unit MLP architecture.

####  -C1 Pareto Frontiers

We additionally extend the driving experiments to construct full Pareto
frontiers modelling the full relationship between the number of questions
asked and the score achieved by each method. As in Section -B, we vary the
querying parameter of each approach to generate points on this frontier. The
resulting frontier is presented in Figure 6. We see our approach outperforms
baselines in terms of score across different numbers of queries made.

We varied the querying parameters of the approaches over the following values:

Ours:

0.002, 0.01, 0.05, 0.25, and 1.25.

Uncertainty:

30.67, 46, 69, 103.5, and 155.25.

Random:

0.025, 0.05, 0.1, 0.2, and 0.4.

###  -D Block Pushing Details

We use a human response precision parameter of $\beta=10$, a threshold of
$c=0.03$ for our approach, a querying probability of $0.25$ for the random
baseline, and an uncertainty threshold of $0.01$ for the uncertainty baseline
across the Fetch block pushing experiments.

We trained expert policies using soft actor-critic for the block pushing task.
We trained these policies in a simulated Fetch block pushing environment [59].
To apply our approaches to this setting, we used the generalization to
continuous action spaces described in Section III-B.

We learned a general policy parameterized by the block’s current position as
well as the goal position using soft actor-critic with a 2-hidden-layer 256
unit MLP architecture. These policies required the current location of the
block to be in the environments observation space, so we used the simulator to
keep track to model the block’s location when deploying our approaches.

Using waypoints generated by the simulator, we transferred trajectories from
simulation to the real Fetch robot.

####  -D1 Pareto Frontiers

We allowed querying parameters to vary over the following values for each
approach to generate this plot:

Ours:

0.0075, 0.015, 0.03, 0.06, and 0.12.

Uncertainty:

0.0025, 0.005, 0.01, 0.02, and 0.04.

Random:

0.111, 0.167, 0.25, 0.375, and 0.563.


# Dual Memory Units with Uncertainty Regulation for Weakly Supervised
Video Anomaly Detection

Hang Zhou, Junqing Yu, Wei Yang 11footnotemark: 1 indicates co-corresponding
author.

###### Abstract

Learning discriminative features for effectively separating abnormal events
from normality is crucial for weakly supervised video anomaly detection (WS-
VAD) tasks. Existing approaches, both video and segment-level label oriented,
mainly focus on extracting representations for anomaly data while neglecting
the implication of normal data. We observe that such a scheme is sub-optimal,
i.e., for better distinguishing anomaly one needs to understand what is a
normal state, and may yield a higher false alarm rate. To address this issue,
we propose an Uncertainty Regulated Dual Memory Units (UR-DMU) model to learn
both the representations of normal data and discriminative features of
abnormal data. To be specific, inspired by the traditional global and local
structure on graph convolutional networks, we introduce a Global and Local
Multi-Head Self Attention (GL-MHSA) module for the Transformer network to
obtain more expressive embeddings for capturing associations in videos. Then,
we use two memory banks, one additional abnormal memory for tackling hard
samples, to store and separate abnormal and normal prototypes and maximize the
margins between the two representations. Finally, we propose an uncertainty
learning scheme to learn the normal data latent space, that is robust to noise
from camera switching, object changing, scene transforming, etc. Extensive
experiments on XD-Violence and UCF-Crime datasets demonstrate that our method
outperforms the state-of-the-art methods by a sizable margin.

##  1 Introduction

Video anomaly detection (VAD) aims to identify the time window of abnormal
events in untrimmed videos (Lu, Shi, and Jia 2013; Liu et al. 2018; Sultani,
Chen, and Shah 2018; Wu et al. 2020), and has beneficial applications in the
context of intelligent video surveillance. Yet collecting fine-grained anomaly
videos for training are arduous, weakly supervised approaches, which assign
one label, i.e., normal or abnormal, for each panoptic video, have been
gaining popularity and achieving state-of-the-art performances (Zhong et al.
2019; Zaheer et al. 2020; Tian et al. 2021). Following weakly supervised
setup, multiple instances learning (MIL) is one of the most effective
strategies (Wang et al. 2018b; Ilse, Tomczak, and Welling 2018). MIL-based
anomaly detection methods consider each video as a bag, where abnormal videos
are defined as positive and normal videos as negative. Meanwhile, anomaly
snippets are presented as positive instances in positive bags, and normal
snippets are negative instances. MIL-based methods then learn the instance-
level labels from video bags by selecting top-K instances as abnormal
according to discriminative representations (Zhang, Qing, and Miao 2019; Zhu
and Newsam 2019; Wu et al. 2021; Wu and Liu 2021).

Figure 1: We use two memory banks to store normal and anomaly instances
respectively, both memory banks attract close instances. The normal space is
regulated to be a Gaussian distribution, while no additional regulation
applied for anomaly feature space. Such a strategy can tackle the hard samples
better, the bottom curve shows prediction results of a part of the video
“Black.Hawk.Down”.

Notably, MIST (Feng, Hong, and Zheng 2021) develops a two-stage self-training
procedure by generating pseudo labels using MIL approaches and emphasizing
anomalous regions with a self-guided attention boosted feature encoder. RTFM
(Tian et al. 2021) uses a feature magnitude learning function to recognise the
positive instances, effectively improving the robustness of the MIL approach.
Nevertheless, existing weakly supervised approaches focus on learning
discriminative features for anomalous instances, while neglecting the
implication of normal data, and hence yield high false alarm rate. As shown in
the bottom of Figure 1, RTFM and HL-Net both lead to alarm falsely.

To address the problems and better distinguish anomaly, we propose a dual
memory units module with uncertainty regulation to store normal and abnormal
prototypes respectively and learn their distinctive distributions, as shown in
the top of Figure 1. Previous approaches use only one memory for normality,
while we observe such strategy can not handle hard samples well, e.g., the
action of running exists in both normal exercising videos and criminal
arresting videos, and hence yield a high false alarm rate. Dual memory units
consider more peripheral factors compared to a single memory, and may reduce
false alarms. Specifically, we encourage the anomaly memory bank to intake
more anomaly information from videos annotated as anomaly, while the normal
memory bank to obtain normal data pattern from both normal and abnormal
videos. Following RTFM’s paradigm which indicates that normal features are in
low magnitude while anomaly features are on the contrary, we observe that
normal features exhibit fluctuations due to noises coming from camera
switching, subject changing, etc., and further format low magnitude normal
features with Gaussian distribution. We adopt a normal data uncertainty
learning (NUL) scheme to learn both the mean feature and uncertainty variance,
and consequentially generate a latent normal space. We can segregate the
normality and anomaly according to the latent space and reduce the false alarm
rate with noise perturbations. Moreover, inspired by the traditional global
and local structure on graph convolutional networks, we introduce a Global and
Local Multi-Head Self Attention (GL-MHSA) module for the Transformer network
to obtain more expressive embeddings for capturing associations in videos.

In summary, the main contributions of this paper are as follows:

  * •

We design a Global and Local Multi-Head Self Attention model to capture
different associations of anomaly videos for enhancing the expressive
embeddings.

  * •

We propose the dual memory units, which use both normal and abnormal memory
banks for better distinguishing patterns of normality and anomaly. We also
introduce a dual memory separation loss to learn more robust and
discriminative features.

  * •

We propose a normal data uncertainty learning scheme to enhance the normal
pattern representation ability and better separate the anomaly from latent
normality.

##  2 Related Work

Weakly Supervised Video Anomaly Detection. VAD is an important task in
computer vision (Pang et al. 2021a), and mainstream methods either adopt an
unsupervised learning strategy with only normal data or weakly supervised
learning setup with video-level annotated data. Unsupervised methods with only
normal data can not well handle complex environments, and stimulated by
progress in weakly supervised learning in video action localization (Nguyen et
al. 2018; Lee, Uh, and Byun 2020; Lee et al. 2021; Yang, Qin, and Huang 2022),
WS-VAD gradually gains popularity. (Sultani, Chen, and Shah 2018) introduce a
large real-world surveillance dataset and proposed a vanilla MIL (V-MIL)
framework with ranking loss. (Zhong et al. 2019) consider the normal label to
contain noise and apply pseudo label generation method by graph convolutional
network to tackle noise specifically. Recent work mostly adopt the MIL
training strategy. Most improvements expand the receptive field and enhance
the instance embedding (Wu and Liu 2021), e.g., RTFM proposes Multi-scale
Temporal Network (MTN) to capture the long and short-range temporal
dependencies. Other approaches focus on multi-modal setup, for instance (Wei
et al. 2022; Pang et al. 2021b) use both audio and video information for VAD.

Self-training strategy generate labels from coarse to fine level (Zou et al.
2019; Pang et al. 2020) and has been explored in semi-supervised learning. In
VADs, (Pang et al. 2020) present an end-to-end and self-training method for
anomaly regression without manually labeled training data. (Feng, Hong, and
Zheng 2021) propose a multiple instance pseudo label generator and self-guided
attention boosted encoder to refine discriminative representations. (Li, Liu,
and Jiao 2022) apply multi-sequence learning with Transformer to refining
abnormal scores through self-training method. Both types of methods consider
normal snippets are noisy and lack normal data learning, while learning
anomalous patterns excessively without normal contrast suffers from
overfitting. In contrast, we adopt the MIL setup and use memory banks to store
the normal and abnormal patterns. Moreover, we apply normal data uncertainty
regulation to learn the latent normal space and enlarge the distance between
normality and anomaly.

Memory Networks. Memory network is an efficient neural network to store
different prototypes and has been applied in various tasks (Yang et al. 2021b;
Luo et al. 2021). (Gong et al. 2019) propose a memory augmented deep
autoencoder to suppress the reconstruction of anomaly. (Stephen and Menon
2020) follows the MemAE framework and presents a new readable and updatable
memory scheme, where memory banks also update at test time. (Liu et al. 2021b)
design a multi-level memory module in an autoencoder with skip connections to
memorize normal patterns for optical flow reconstruction. All the above
methods use memory banks to learn from normality. In this work, We introduce
dual memory units to store both normal and abnormal prototypes, and we design
a dual loss to guide memory banks in prototype learning.

Figure 2: The framework of our UR-DMU model consists of three parts: global
and local feature learning (GL-MHSA), dual memory units (DMU), and normal data
uncertainty learning (NUL). GL-MHSA extracts better expressive embeddings, DMU
stores both normal and abnormal patterns for discrimination, and NUL
constrains the normal data as a Gaussian distribution to handle uncertainty
perturbation.

Data Uncertainty Learning. Data uncertainty learning captures the data
variance due to noise and randomness (Chang et al. 2020; Li et al. 2022; Lee
et al. 2021). (Chang et al. 2020) propose the data uncertainty learning scheme
to learn the feature embedding and its uncertainty in face recognition. In
video temporal action localization, (Lee et al. 2021) formulate the background
frames as out-of-distribution (OOD) samples and proposed an uncertainty model
to learn the uncertainty. Different from (Lee et al. 2021), we presume that
the anomaly data is OOD, and the normal data follows a Gaussian distribution.
The latent space of the normal uncertainty pattern is effective for unknown
normal samples. Meanwhile, it also helps the model to separate normality and
anomaly.

##  3 Method

Here we introduce the Uncertainty Regulated Dual Memory Units (UR-DMU) method
to store both normal and abnormal prototypes and enforce additional prior
distributions in details. As illustrated in Figure 2, UR-DMU consists of three
modules: the Global and Local Multi-Head Self Attention (GL-MHSA) for feature
learning, the Dual Memory Units(DMU) for storing prototypes, and the Normal
Data Uncertainty Learning (NUL) for normality latent embedding learning. In
the following sections, we briefly introduce the formations of our problem
first, and then describe the above three modules respectively, and finally
show the training and testing process.

###  3.1 Problem Formation

Following the MIL setup for WS-VAD tasks, we define normal videos as
$\\{v_{i}^{n}\\}_{i=1}^{N}$ and abnormal videos as
$\\{v_{i}^{a}\\}_{i=1}^{N}$. Each anomaly video is a bag $Y^{a}=1$ with at
least one abnormal clip, similarly a normal video is marked as $Y^{n}=0$ with
all normal clips. Our goal is to score each $v_{i}$ according to
$Y_{i}=f(v_{i})$ with the model $f(\cdot)$. It is a common practice to extract
snippet features using pretrained backbones (Tran et al. 2015; Carreira and
Zisserman 2017; Liu et al. 2021c) and conduct future processes on extracted
features, in this paper we choose I3D model pre-trained on Kinetics dataset as
our backbone, and extract RGB features of consecutive 16-frames (Wu et al.
2020). The extracted video features are denoted as
$\textbf{X}_{i3d}\in\mathbb{R}^{N\times F}$ , where $N$ represent the number
of snippets, and $F$ is the dimension of extracted features.

###  3.2 Global and Local Feature Learning

Learning long and short temporal dependencies of anomalous features is crucial
for VAD (Wang et al. 2018a; Wang and Gupta 2018). Traditional approaches adopt
the graph convolution network for extracting global and local associations (Wu
et al. 2020). Recently, the transformer structure shows its superiority in
handling long-range associations between temporal signals (Vaswani et al.
2017; Dosovitskiy et al. 2021; Liu et al. 2021a; Yang et al. 2021a). For
better capturing long and short temporal dependencies simultaneously, we
borrow the idea of global and local structure in GCN and develop the Global
and Local Multi-Head Self Attention (GL-MHSA) for transformer. Different from
the conventional MHSA, we add an extra encoder layer of
$\textbf{V}_{l}=\textbf{X}_{i3d}\textbf{W}_{l}$ to learn local features with a
temporal mask:

$\textbf{T}_{m}(i,j)=-\frac{|i-j|}{e^{\tau}}.$ (1)

Then we normalize the mask with $\text{softmax}(\textbf{T}_{m})$, where $\tau$
is the sensitivity factor for balancing the local distance. While computing
$Q,K,V$, we obtain the global and local feature
$\textbf{X}\in\mathbb{R}^{N\times D}$ from Multilayer Perceptron (MLP) and
LayerNorm (LN), which can be formulated as follows:

X
$\displaystyle=\text{MH}([\text{softmax}(\frac{\textbf{Q}\textbf{K}^{T}}{\sqrt{D}})\textbf{V}_{g};\text{softmax}(\textbf{T}_{m})\textbf{V}_{L}]),$
(2) X $\displaystyle=\text{LN}(\text{MLP}(\text{LN}(\textbf{X}))+\textbf{X}).$

###  3.3 Dual Memory Units

Previous approaches use memory units to store prototypes of normal data for
better scoring. We observe that a single unit is insufficient for
distinguishing hard samples, and we use learnable normal and abnormal memory
banks to store corresponding templates $\textbf{M}\in\mathbb{R}^{M\times D}$,
where $M$ is the memory bank number, $D$ is the output feature dimension. We
apply query and read operations to encourage correct template learning, and by
querying memory banks M we can assess the similarities between features:

S $\displaystyle=\sigma(\frac{\textbf{X}\textbf{M}^{T}}{\sqrt{D}}),\hskip
10.00002pt\textbf{M}_{aug}=\text{S}\textbf{M},$ (3)

where $\sigma$ is sigmoid activation, and $\textbf{S}\in\mathbb{R}^{N\times
M}$ is the query score. Then the memory augmentation feature generated by a
read operation is represented as $\textbf{M}_{aug}$. S indicates whether
snippets are similar to memory banks. We apply topK selection along the second
dimension to determine which snippets are most similar to memory banks:

$\displaystyle\textbf{S}_{k;ind},\textbf{S}_{k;sc}=\text{topK}(\textbf{S},K),\hskip
5.0pt\textbf{S}_{k;i}$ $\displaystyle=\frac{\sum_{j=1}^{K}S_{k;sc}(i,j)}{K}.$
(4)

In equation (4), $K\ll M$, $\textbf{S}_{k;ind}$ means the indices of proposed
K snippets to be stored, and $\textbf{S}_{k;sc}\in\mathbb{R}^{N\times K}$ are
their query scores. $\textbf{S}_{k}\in\mathbb{R}^{N\times 1}$ indicates the
highest scores of memory matching.

Figure 3: The dual memory units store normal and anomaly instances in
different memories. (a) shows that normal video feature goes through dual
memory units and obtains relevant output scores and augment features. (b) is
the case with abnormal video feature input.

Figure 3 shows different input features across the dual memory units. While
sending normal video embedding $\textbf{X}^{n}\in\mathbb{R}^{N\times D}$ into
the abnormal memory banks, $\textbf{S}_{k;a}^{n}$ and $\textbf{M}_{aug;a}^{n}$
are generated by querying and reading operations. Similarly, we can also
obtain $\textbf{S}_{k;n}^{n}$ and $\textbf{M}_{aug;n}^{n}$ as video embedding
goes through normal memory banks. The anomaly query scores
$\textbf{S}_{k;a}^{n}$ are constrained to $\textbf{0}\in\mathbb{R}^{N}$
because of the normal input. And the normal query score $\textbf{S}_{k;n}^{n}$
should be $\textbf{1}\in\mathbb{R}^{N}$.

Likewise, when the abnormal video embedding input to abnormal memory banks, we
obtain $\textbf{S}_{k;a}^{a}$ and $\textbf{M}_{aug;a}^{a}$. Because of the
sparseness nature of the anomaly, only several snippets will be activated by
anomaly memory banks. We can also obtain $\textbf{S}_{k;n}^{a}$ and
$\textbf{M}_{aug;n}^{a}$ from normal memory banks. Different from normal
input, the anomaly query score $\textbf{S}_{k;a}^{n}$ is not all 1 because of
the weak label of anomaly. Same with MIL, there is at least one element equals
$1$ in $\textbf{S}_{k;a}^{a}$ and $\textbf{S}_{k;n}^{a}$.

To train the dual memory units, we define a dual memory loss consists of four
binary cross-entropy (BCE) losses:

$\displaystyle L_{dm}$
$\displaystyle=\text{BCE}(\textbf{S}_{k;n}^{n},\textbf{y}_{n}^{n})+\text{BCE}(\textbf{S}_{k;a}^{n},\textbf{y}_{a}^{n})$
(5)
$\displaystyle+\text{BCE}(S_{k;n;k}^{a},y_{n}^{a})+\text{BCE}(S_{k;a;k}^{a},y_{a}^{a}),$

where $\textbf{y}_{n}^{n}=\textbf{1}\in\mathbb{R}^{N}$ and
$\textbf{y}_{a}^{n}=\textbf{0}\in\mathbb{R}^{N}$.
$S_{k;n;k}^{a},S_{k;a;k}^{a}\in\mathbb{R}$ are means of
$\textbf{S}_{k;n}^{a},\textbf{S}_{k;a}^{a}$ topK result along the first
dimension. The labels $y_{n}^{a},y_{a}^{a}$ are equal to $1$. The first two
terms of $L_{dm}$ limit the normal memory unit to learn normal patterns, and
the last two terms ensure the abnormal pattern learning for abnormal memory
unit. With $L_{dm}$, memory banks store homologous patterns.

With the query scores $\textbf{S}_{k;n}^{n}$, $\textbf{S}_{k;a}^{n}$ and
$\textbf{S}_{k;a}^{a}$, we separate the abnormal and normal feature embedding
by triplet loss.

$\displaystyle L_{trip}$ $\displaystyle=\text{Triplet}(f_{a},f_{p},f_{n}),$
(6) $\displaystyle f_{a}$
$\displaystyle=\text{topK}(\textbf{S}_{k;n}^{n};\textbf{X}^{n}),$
$\displaystyle f_{p}$
$\displaystyle=\text{topK}(\textbf{S}_{k;n}^{a};\textbf{X}^{a}),$
$\displaystyle f_{n}$
$\displaystyle=\text{topK}(\textbf{S}_{k;a}^{a};\textbf{X}^{a}),$

where $f_{a}$ is the average feature of the interest snippets from
$\textbf{X}^{n}$, which indicates the primary part of normal video. The
snippets are selected by the topK index of $\textbf{S}_{k;n}^{n}$. Similarly,
$f_{p}$ means the normal part in an abnormal video, and $f_{n}$ means the
abnormal part in the abnormal video.

###  3.4 Normal Date Uncertainty Learning

As pointed out by (Tian et al. 2021; Lee et al. 2021), the magnitude of the
normal feature is lower than that of anomalies, i.e.,
$\mathbb{E}[\|x^{a}\|_{2}]\geqslant\mathbb{E}[\|x^{n}\|_{2}]$, $x^{a}$ and
$x^{n}$ are the snippet features of anomaly and normality respectively.
Presumably, magnitude feature learning is also appropriate for the
augmentation features $\textbf{M}_{aug}$ of memory output. Hence, the
magnitude separability of augmentation features between abnormal and normal is
defined as $d(a,n)=\|m^{a}\|_{2}-\|m^{n}\|_{2}$. Moreover, effective
representation learning of normality should be able to capture both the core
and cover variance introduced by noise. We propose to learn the distribution
of normality, and then anomaly is out of this distribution (OOD) (Riveram et
al. 2022). This idea has been extensively explored in unsupervised approaches,
but rarely used in WS-VAD. Similar to unsupervised anomaly detection methods,
we propose to use the Gaussian distribution to constraint the latent normal
representation $z^{n}$, i.e.:

$p(z_{i}^{n}|x_{i}^{n})=\mathcal{N}(z_{i}^{n};\mu_{i};\sigma_{i}^{2}I),$ (7)

where $\mu$ is the normal feature’s mean, and $\sigma$ is the variance
(uncertainty). Both of them can be predicted by an encoder network. To ensure
the normal data sampling operation is differentiable, we follow (Chang et al.
2020) and use the re-parameterization to tackle the latent with noise. The DUL
features $z^{n}$ is generated by random noise
$\varepsilon\sim\mathcal{N}(0,1)$:

$z_{i}^{n}=\mu_{i}^{n}+\sigma_{i}^{n}\varepsilon,\\\ $ (8)

Equation (8) represents the latent space of normal data. However, the sample
$z$ is impressionable to $\sigma$ when $\sigma$ is too large, and it may be
far away from the real features $\mu$. To suppress the instability of
$\sigma$, a Kullback-Leibler diversity(KLD) regularization term is necessary:

$\displaystyle L_{kl}$
$\displaystyle=\text{KL}(\mathcal{N}(z^{n}|\mu^{n},\sigma^{n})||\mathcal{N}(0,1))$
(9)
$\displaystyle=-\frac{1}{2D}\sum_{i=1}^{D}(1+\text{log}(\sigma_{i}^{n})^{2}-(\mu_{i}^{n})^{2}-(\sigma_{i}^{n})^{2}),$

In testing stage, calculation of the variance can be skipped, and we only need
the mean to generate robust prediction. Furthermore, as mentioned, anomalies
are OOD samples, and we need to enlarge the margin between anomalies and
normalities for robustness. The anomaly features are also fed to the mean-
encoder to get the embedding $\mu^{a}$ in the same space. We use the magnitude
distance loss to separate them in the latent normal space:

$L_{dis}=\max(0,d-(\left\lVert\mu_{k}^{a}\right\rVert_{2}^{2}-\left\lVert
z_{k}^{n}\right\rVert_{2}^{2})),$ (10)

where $\mu_{k}^{a}$ are the average features of proposed snippets from
$\textbf{M}_{aug}$. The snippets are selected by topK index operation from
$\textbf{S}_{k;a}^{a}$. Same with the above operation, $z_{k}^{n}$ is obtained
from $\textbf{S}_{k;n}^{n}$ and $\textbf{M}_{aug}$.

Finally, we fuse the DUL features with X and feed it to the classifier.

###  3.5 Network training and Testing

Training: Pairs of videos with the same amount of normal and abnormal videos
are divided into a batch and inputted to our model. Then we generate one score
for each snippet and use the BCE loss to discriminate the normality and
anomaly with five auxiliary losses:

$\displaystyle L$
$\displaystyle=L_{cls}+\lambda_{1}L_{dm}+\lambda_{2}L_{trip}+\lambda_{3}L_{kl}+\lambda_{4}L_{dis},$
(11)

where $L_{cls}$ is BCE loss with the supervision of video-level labels,
$\lambda_{1},\lambda_{2},\lambda_{3},\lambda_{4}$ are hyperparameters.

Testing: We feed testing videos to our UR-DMU network. In the DUL module, we
only use the mean-encoder network to obtain feature embeddings. Finally, the
snippet labels are assigned to frame labels.

##  4 Experiments

###  4.1 Datasets and Evaluation Metrics

Dataset. Similar to previous works, we conduct experiments on two large video
anomaly detection datasets: UCF-Crime (Sultani, Chen, and Shah 2018), and XD-
Violence (Wu et al. 2020).

UCF-Crime is a large-scale surveillance dataset that includes 1900 videos with
13 types of anomalous events, e.g., explosion, arrest, and road accidents. The
training set with video-level labels contains 800 normal and 810 abnormal
videos, while the testing set with frame-level labels has 140 normal and 150
abnormal videos.

XD-Violence is a multi-modal dataset with videos and audios, and it includes
different backgrounds: movies, games, live scenes, and etc. There are 4754
videos in the datasets, which contains 3954 training videos with video-level
labels and 800 testing videos with frame-level labels.

Evaluation Metrics. We evaluate WS-VAD performance with the area under curve
(AUC) of the frame-level receiver operating characteristics (ROC) for UCF-
Crime. Meanwhile, the AUC of the frame-level precision-recall curve (AP) is
used for XD-Violence. False alarm rate (FAR) and anomaly subset with only
abnormal data are also applied in ablation experiments (Wu and Liu 2021; Feng,
Hong, and Zheng 2021).

###  4.2 Implementation Details

We extract the snippet features from pre-trained I3D on Kinetics-400. The same
configurations are set for both datasets. In the training stage, each video is
divided into $N=200$ snippets, and the learning rate is 0.0001. We train the
model with batch size of 64 and iterator time of 3000, and set the $M^{a}=60$
and $M^{n}=60$ for UCF-Crime and XD-Violence. The hyperparameters of
$(\lambda_{1},\lambda_{2},\lambda_{3},\lambda_{4})$ are set as
$(0.1,0.1,0.001,0.0001)$ to balance the total loss. All the operations of topK
are equal to $K=\lfloor\frac{M}{16}\rfloor+1$. In equation (10), $d=100$.

| Method | Feature(RGB) | AUC(%)
---|---|---|---
Un | Hasan et al.(2016) | N/A | 50.60
Lu et al.(2013) | N/A | 65.51
GCL(2022) | ResNext | 74.20
Weakly | Sultani et al.(2018) | C3D | 75.41
Sultani et al.(2018) | I3D | 76.21
MA(2019) | PWC | 79.00
GCN (2019) | TSN | 82.12
IBL (2019) | C3D | 78.66
HL-Net (2020) | I3D | 82.44
CLAWS (2020) | C3D | 83.03
MIST (2021) | I3D | 82.30
RTFM (2021) | I3D | 84.30
CRFD (2021) | I3D | 84.89
MSL (2022) | I3D | 85.30
MSL (2022) | VideoSwin | 85.62
Ours | I3D | 86.97

Table 1:  Comparison of the frame-level performance AUC on UCF-Crime testing
set with existing approaches.

###  4.3 Results on UCF-Crime

On UCF-Crime dataset, we compare the AUC scores with existing state-of-the-art
(SOTA) VAD methods in Table 1. All the methods use RGB features, and our
method is based on 10-crop augmentation I3D features. As the results show,
without the prior information of anomaly, unsupervised methods are generally
lower in rank than weakly supervised methods. When compared with weakly
supervised approaches, our method outperforms all the previous methods and
achieves the SOTA AUC score of 86.97%. There is a 1.35% improvement for our
UR-DMU model when compared to the previous SOTA method MSL (85.62% AUC) (Li,
Liu, and Jiao 2022), which is a Transformer-based method with self-training.
Compared with the MIL-based method RTFM (84.30% AUC) (Tian et al. 2021), we
achieve a considerable improvement of 2.67%.

###  4.4 Results on XD-Violence

The AP scores in XD-Violence are displayed with SOTA VAD methods in Table 2.
We use the 5-crop I3D RGB and features and VGGish audio features provided by
(Wu et al. 2020) for experiments, and achieve 81.66% performance with only RGB
features. Compared with the MIL-based method RTFM (77.81% AP), we achieve a
3.85% improvement. Multi-modal methods are also showed in Table 2. With only
RGB features, our method outperforms the ACF (Wei et al. 2022), which utilizes
the attention module to fuse the video and audio modal. We also obtain the
same performances as (Pang et al. 2021b). To evaluate the robustness in multi-
modal, we utilize feature concatenation to fuse video and audio features, We
can also achieve the best performances with an 81.77% AP score. The results of
both datasets indicate our UR-DMU is effective in detecting abnormal events
with disturbances.

Method | Feature | AP(%)
---|---|---
Hasan et al.(2016) | RGB | 30.77
Sultani et al.(2018) | RGB | 73.20
HL-Net (2020) | RGB | 73.67
HL-Net (2020) | RGB+Audio | 78.64
CRFD (2021) | RGB | 75.90
RTFM(2021) | RGB | 77.81
MSL(2022) | RGB | 78.28
MSL(2022) | RGB | 78.59
Pang et al.(2021) | RGB+Audio | 81.69
ACF(2022) | RGB+Audio | 80.13
Ours | RGB | 81.66
Ours | RGB+Audio | 81.77

Table 2:  Comparison of frame-level performance AP on XD-Violence validation
set.

Memory | Dataset
---|---
UCF(%) | XD(%)
AUC | AP | FAR | AUC | AP | FAR
Single A-Mem | 83.72 | 31.46 | 1.32 | 88.07 | 77.59 | 1.38
Single N-Mem | 85.19 | 35.23 | 0.98 | 93.69 | 80.40 | 0.48
Dual Mems | 86.97 | 35.59 | 1.05 | 94.02 | 81.66 | 0.65

Table 3:  Comparison of the results of different memory units.

| Dataset
---|---
| UCF(%) | XD(%)
($M_{A}$,$M_{N}$) | AUC | AP | FAR | AUC | AP | FAR
20,20 | 86.71 | 31.28 | 0.82 | 93.63 | 81.13 | 0.39
30,30 | 86.44 | 31.29 | 1.18 | 93.67 | 80.46 | 1.18
40,40 | 85.09 | 32.40 | 1.53 | 93.07 | 79.90 | 0.67
50,50 | 86.28 | 33.23 | 1.63 | 93.51 | 80.19 | 0.74
60,60 | 86.97 | 35.59 | 1.05 | 94.02 | 81.66 | 0.65
70,70 | 86.47 | 33.60 | 0.49 | 93.06 | 78.87 | 1.00
80,80 | 85.16 | 34.18 | 0.96 | 93.45 | 79.59 | 0.31
90,90 | 86.18 | 35.14 | 2.01 | 93.47 | 79.87 | 0.44
100,100 | 85.23 | 31.57 | 1.63 | 93.92 | 80.75 | 0.78

Table 4:  Comparison of the results of different memory numbers $M_{A}$, and
$M_{N}$. $M_{A}$ is the number of abnormal memory banks, and $M_{N}$ is the
number of normal memory banks.

Loss term | Dataset
---|---
$L_{dm}$ | $L_{trip}$ | $L_{kl}$ | $L_{dis}$ | UCF(AUC%) | XD(AP%)
✓ | ✓ |  |  | 83.46 | 77.14
✓ |  | ✓ |  | 82.78 | 78.05
✓ |  |  | ✓ | 81.24 | 76.49
| ✓ | ✓ | ✓ | 82.09 | 78.74
✓ |  | ✓ | ✓ | 82.78 | 78.16
✓ | ✓ |  | ✓ | 83.92 | 77.57
✓ | ✓ | ✓ |  | 86.60 | 80.79
✓ | ✓ | ✓ | ✓ | 86.97 | 81.66

Table 5:  Comparison of the results of different loss terms.

###  4.5 Ablation Study

Effectiveness of dual memory units. To verify the effect of the different
memory units, we split dual memory units into two single memory units. As
shown in Table 3, the single abnormal memory unit lacks normal patterns to
discriminate what is normality and receives a higher FAR in normality
detection. It also indicates that abnormal patterns are not easy to learn on
weakly supervised data. The single normal memory unit structure can obtain
better results than abnormal memory with rich normal data. The dual memory
units get the best performances because of storing both memory patterns. To
explore a proper number of the memory banks, we set 9 memory cases to search
for the best configuration of memory bank numbers. The results are shown in
Table 4. $M_{a}$ and $M_{n}$ mean the number of abnormal and normal memories.
When $M_{a}=60,M_{n}=60$, our method achieves the best performances on both
datasets.

Effectiveness of different loss terms. To explore the impact of four loss
terms on our complete model, we report the results of 8 loss combinations in
Table 5. The results of the first three cases show that the $L_{trip}$ is most
effective to separate the distance between normality and anomaly. From the
last five cases, the loss terms of $\\{L_{dm},L_{trip},L_{kl}\\}$ are
essential and indispensable for our model. The $L_{dis}$ has a slight effect
on detection performance, and control the border between normal and abnormal.
The above observations reveal that our loss terms are more suitable for
working together.

Module | Dataset
---|---
UCF(%) | XD(%)
GL-MHSA | DMU | NUL | AUC | AP | AUCsub | APsub | FAR | AUC | AP | AUCsub | APsub | FAR
✓ |  |  | 82.89 | 27.07 | 60.18 | 28.30 | 4.86 | 91.57 | 75.67 | 76.17 | 76.99 | 0.96
✓ | ✓ |  | 85.53 | 31.21 | 70.13 | 33.79 | 1.53 | 93.38 | 79.80 | 81.06 | 81.03 | 0.62
✓ |  | ✓ | 85.19 | 33.61 | 69.74 | 35.79 | 2.37 | 93.34 | 78.71 | 78.69 | 79.44 | 0.85
✓ | ✓ | ✓ | 86.97 | 35.59 | 70.81 | 37.25 | 1.05 | 94.02 | 81.66 | 82.36 | 82.85 | 0.65

Table 6:  Comparison of each module’s performance on UCF-Crime and XD-
Violence. AUCsub and APsub are the scores only using abnormal data. FAR is the
false alarm rate with only normal data.  Figure 4: Qualitative results of
anomaly detection performances on UCF-Crime and XD-Violence. The
visualizations of (a)$\thicksim$(d) are from XD-Violence, and
(e)$\thicksim$(h) are from UCF-Crime.

Effectiveness of proposed modules. To investigate the influences of different
modules, we report the results of four combinations of three modules in Table
6, and adopt three additional indicators to comprehensively evaluate our
method. The AUCsub and APsub are separately the AUC score and AP score of the
abnormal video subset, which only contains abnormal videos. They are used to
confirm the anomaly detection performance for anomaly video. Meanwhile, the
FAR is also applied for normal video detection and ensures the performance of
normality detection. GL-MHSA module is the feature learning backbone and
achieves the lowest performance. When adding the DMU module, all the metrics
exhibit great improvements. The increase of AUC/AP scores and the decrease of
FAR indicate that the DMU module can store anomaly and normality patterns.
When NUL module is added, each metric has large improvements. But it can not
map all the normal patterns into Gaussian space without memory banks
auxiliary. Finally, the full model achieves the best performances on two
datasets.

###  4.6 Qualitative Results

To further evaluate our method, the qualitative results are shown on two
datasets in Figure 4. We compare our method (blue line) with RTFM (orange
line) on eight videos to exhibit the frame-level abnormal predictions. The
pink area is the ground truth area of the anomaly. We also show the keyframes
and mark the abnormal/normal frames to describe what happened in the video,
the green rectangles indicate the normal frame and the red rectangles mean the
abnormal frame. As exhibited in the figure, our method not only produces a
precise detection area but also reduces the FAR on the normal part. RTFM is
based on feature magnitude and is effective for scenes that are smooth and
change slowly. Results of Figure 4 (f) and (g) show that RTFM fails to make
correct detection. Our method achieves a lower FAR for normal video (d) and
(h).

##  5 Conclusion

In this work, we propose a UR-DMU model to efficiently separate normality and
abnormality for WS-VAD. To obtain more expressive embeddings, we introduce the
GL-MHSA module to capture long and short temporal dependencies. Then, we
present DMU to store abnormal and normal prototypes under the video-level
label, and the DMU loss is devised to encourage memory units learning and
separate them. Meanwhile, the DUL scheme is proposed to learn latent normal
space from rich normal data with noise perturbation. The predicted anomaly
windows and scores of our approach are still relatively coarse and unstable
because of using snippet-level features. In the future, we plan to explore
more fine-grained locations in both temporal windows and image space with
contrastive learning.

##  6 Acknowledgment

This work is supported by the National Key Research and Development Program of
China under Grant No.2020YFB1805601, National Natural Science Foundation of
China (NSFC No. 62272184), and CCF-Tencent Open Research Fund (CCF-Tencent
RAGR20220120). The computation is completed in the HPC Platform of Huazhong
University of Science and Technology.

## References

  * Carreira and Zisserman (2017) Carreira, J.; and Zisserman, A. 2017.  Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset.  In _CVPR_ , 4727–4733.
  * Chang et al. (2020) Chang, J.; Lan, Z.; Cheng, C.; and Wei, Y. 2020.  Data Uncertainty Learning in Face Recognition.  In _CVPR_ , 5709–5718.
  * Dosovitskiy et al. (2021) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In _ICLR_.
  * Feng, Hong, and Zheng (2021) Feng, J.; Hong, F.; and Zheng, W. 2021.  MIST: Multiple Instance Self-Training Framework for Video Anomaly Detection.  In _CVPR_ , 14009–14018.
  * Gong et al. (2019) Gong, D.; Liu, L.; Le, V.; Saha, B.; Mansour, M. R.; Venkatesh, S.; and Hengel, A. V. D. 2019.  Memorizing Normality to Detect Anomaly: Memory-Augmented Deep Autoencoder for Unsupervised Anomaly Detection.  In _ICCV_ , 1705–1714.
  * Ilse, Tomczak, and Welling (2018) Ilse, M.; Tomczak, J. M.; and Welling, M. 2018.  Attention-based deep multiple instance learning.  In _ICML_ , 2132–2141.
  * Lee, Uh, and Byun (2020) Lee, P.; Uh, Y.; and Byun, H. 2020.  Background Suppression Network for Weakly-Supervised Temporal Action Localization.  In _AAAI_ , 11320–11327.
  * Lee et al. (2021) Lee, P.; Wang, J.; Lu, Y.; and Byun, H. 2021.  Weakly-supervised Temporal Action Localization by Uncertainty Modeling.  In _AAAI_ , 1854–1862.
  * Li, Liu, and Jiao (2022) Li, S.; Liu, F.; and Jiao, L. 2022.  Self-Training Multi-Sequence Learning with Transformer for Weakly Supervised Video Anomaly Detection.  In _AAAI_ , 1395–1403.
  * Li et al. (2022) Li, X.; Dai, Y.; Ge, Y.; Liu, J.; Shan, Y.; and Duan, L. 2022.  Uncertainty Modeling for Out-of-Distribution Generalization.  _arXiv preprint arXiv:2202.03958_.
  * Liu et al. (2018) Liu, W.; Luo, W.; Lian, D.; and Gao, S. 2018.  Future Frame Prediction for Anomaly Detection - A New Baseline.  In _CVPR_ , 6536–6545.
  * Liu et al. (2021a) Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; and Guo, B. 2021a.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.  In _ICCV_ , 9992–10002.
  * Liu et al. (2021b) Liu, Z.; Nie, Y.; Long, C.; Zhang, Q.; and Li, G. 2021b.  A Hybrid Video Anomaly Detection Framework via Memory-Augmented Flow Reconstruction and Flow-Guided Frame Prediction.  In _ICCV_ , 13568–13577.
  * Liu et al. (2021c) Liu, Z.; Ning, J.; Cao, Y.; Wei, Y.; Zhang, Z.; Lin, S.; and Hu, H. 2021c.  Video Swin Transformer.  _arXiv preprint arXiv:2106.13230_.
  * Lu, Shi, and Jia (2013) Lu, C.; Shi, J.; and Jia, J. 2013.  Abnormal Event Detection at 150 FPS in MATLAB.  In _ICCV_ , 2720–2727.
  * Luo et al. (2021) Luo, W.; Zhang, T.; Yang, W.; Liu, J.; Mei, T.; Wu, F.; and Zhang, Y. 2021.  Action Unit Memory Network for Weakly Supervised Temporal Action Localization.  In _CVPR_ , 9969–9979.
  * Nguyen et al. (2018) Nguyen, P.; Liu, T.; Prasad, G.; and Han, B. 2018.  Weakly Supervised Action Localization by Sparse Temporal Pooling Network.  In _CVPR_ , 6752–6761.
  * Pang et al. (2021a) Pang, G.; Shen, C.; Cao, L.; and Hengel, A. V. D. 2021a.  Deep Learning for Anomaly Detection: A Review.  _ACM Computing Surveys_ , 54(2): 1–38.
  * Pang et al. (2020) Pang, G.; Yan, C.; Shen, C.; Hengel, A. V. D.; and Bai, X. 2020.  Self-Trained Deep Ordinal Regression for End-to-End Video Anomaly Detection.  In _CVPR_ , 12170–12179.
  * Pang et al. (2021b) Pang, W.; He, Q.; Hu, Y.; and Li, Y. 2021b.  Violence Detection in Videos Based on Fusing Visual and Audio Information.  In _ICASSP_ , 2260–2264.
  * Riveram et al. (2022) Riveram, A. R.; Khan, A.; Bekkouch, I. E. I.; and Sheikh, T. S. 2022.  Anomaly Detection Based on Zero-Shot Outlier Synthesis and Hierarchical Feature Distillation.  _IEEE Transactions on Neural Networks and Learning Systems_ , 33(1): 281–291.
  * Stephen and Menon (2020) Stephen, K.; and Menon, V. 2020.  Learning Memory-guided Normality for Anomaly Detection.  In _CVPR_ , 14360–14369.
  * Sultani, Chen, and Shah (2018) Sultani, W.; Chen, C.; and Shah, M. 2018.  Real-World Anomaly Detection in Surveillance Videos.  In _CVPR_ , 6479–6488.
  * Tian et al. (2021) Tian, Y.; Pang, G.; Chen, Y.; Singh, R.; Verjans, J. W.; and Carneiro, G. 2021.  Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Learning.  In _ICCV_ , 4955–4966.
  * Tran et al. (2015) Tran, D.; Bourdev, L. D.; Fergus, R.; Torresani, L.; and Paluri, M. 2015.  Learning Spatiotemporal Features with 3D Convolutional Networks.  In _ICCV_ , 4489–4497.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017.  Attention is All you Need.  In _NIPS_ , 5998–6008.
  * Wang et al. (2018a) Wang, X.; Girshick, R. B.; Gupta, A.; and He, K. 2018a.  Non-Local Neural Networks.  In _CVPR_ , 7794–7803.
  * Wang and Gupta (2018) Wang, X.; and Gupta, A. 2018.  Videos as Space-Time Region Graphs.  In _ECCV_ , 413–431.
  * Wang et al. (2018b) Wang, X.; Yan, Y.; Tang, P.; Bai, X.; and Liu, W. 2018b.  Revisiting multiple instance neural networks.  _Pattern Recognit_ , 74: 15–24.
  * Wei et al. (2022) Wei, D.; Liu, C.; Liu, Y.; Liu, J.; Zhu, X.; and Zeng, X. 2022.  Look, Listen and Pay More Attention: Fusing Multi-Modal Information for Video Violence Detection.  In _ICASSP_ , 1980–1984.
  * Wu et al. (2021) Wu, J.; Zhang, W.; Li, G.; Wu, H.; Tan, X.; Li, Y.; Ding, E.; and Lin, L. 2021.  Weakly-Supervised Spatio-Temporal Anomaly Detection in Surveillance Video.  In _IJCAI_ , 1172–1178.
  * Wu and Liu (2021) Wu, P.; and Liu, J. 2021.  Learning Causal Temporal Relation and Feature Discrimination for Anomaly Detection.  _IEEE Transactions on Image Processing_ , 30: 3512–3527.
  * Wu et al. (2020) Wu, P.; Liu, J.; Shi, Y.; Shao, F.; Wu, Z.; and Yang, Z. 2020.  Not only Look, But Also Listen: Learning Multimodal Violence Detection Under Weak Supervision.  In _ECCV_ , 322–339.
  * Yang et al. (2021a) Yang, J.; Li, C.; Zhang, P.; Dai, X.; Xiao, B.; Yuan, L.; and Gao, J. 2021a.  Focal Self-attention for Local-Global Interactions in Vision Transformers.  _arXiv preprint arXiv:2107.00641_.
  * Yang, Qin, and Huang (2022) Yang, Z.; Qin, J.; and Huang, D. 2022.  ACGNet: Action Complement Graph Network for Weakly-Supervised Temporal Action Localization.  In _AAAI_ , 3090–3098.
  * Yang et al. (2021b) Yang, Z.; Zhang, T.; Bozchalooi, I. S.; and Darve, E. 2021b.  Memory-Augmented Generative Adversarial Networks for Anomaly Detection.  _IEEE Transactions on Neural Networks and Learning Systems_ , 33(6): 2324–2334.
  * Zaheer et al. (2020) Zaheer, M. Z.; Mahmood, A.; Astrid, M.; and Lee, S.-I. 2020.  CLAWS: Clustering Assisted Weakly Supervised Learning with Normalcy Suppression for Anomalous Event Detection.  In _ECCV_ , 358–376.
  * Zhang, Qing, and Miao (2019) Zhang, J.; Qing, L.; and Miao, J. 2019.  Temporal Convolutional Network with Complementary Inner Bag Loss for Weakly Supervised Anomaly Detection.  In _ICIP_ , 4030–4034.
  * Zhong et al. (2019) Zhong, J.; Li, N.; Kong, W.; Liu, S.; Li, T. H.; and Li, G. 2019.  Graph Convolutional Label Noise Cleaner: Train a Plug-And-Play Action Classifier for Anomaly Detection.  In _CVPR_ , 1237–1246.
  * Zhu and Newsam (2019) Zhu, Y.; and Newsam, S. D. 2019.  Motion-Aware Feature for Improved Video Anomaly Detection.  In _BMVC_ , 270.
  * Zou et al. (2019) Zou, Y.; Yu, Z.; Liu, X.; and Kumar, J., B.V.K. Vijaya adn Wang. 2019.  Confidence Regularized Self-Training.  In _ICCV_ , 5981–5990.