∎

11institutetext<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
# SalSum: Saliency-based Video Summarization using Generative Adversarial
Networks

George Pantazis† Department of Computer Science and Biomedical Informatics,
University of Thessaly, Lamia, Greece  George Dimas∗ Department of Computer
Science and Biomedical Informatics, University of Thessaly, Lamia, Greece
Dimitris K. Iakovidis⋆ Department of Computer Science and Biomedical
Informatics, University of Thessaly, Lamia, Greece

(Received: Novermber 20, 2020 / Accepted:
Multimedia Tools and Applications)

###### Abstract

The huge amount of video data produced daily by camera-based systems, such as
surveilance, medical and telecommunication systems, emerges the need for
effective video summarization (VS) methods. These methods should be capable of
creating an overview of the video content. In this paper, we propose a novel
VS method based on a Generative Adversarial Network (GAN) model pre-trained
with human eye fixations. The main contribution of the proposed method is that
it can provide perceptually compatible video summaries by combining both
perceived color and spatiotemporal visual attention cues in a unsupervised
scheme. Several fusion approaches are considered for robustness under
uncertainty, and personalization. The proposed method is evaluated in
comparison to state-of-the-art VS approaches on the benchmark dataset VSUMM.
The experimental results conclude that SalSum outperforms the state-of-the-art
approaches by providing the highest f-measure score on the VSUMM benchmark.

###### Keywords:

Video Summarization Visual Attention Visual Saliency Generative Adversarial
Networks

##  1 Introduction

The structure of a video can be considered as a set of non-overlapping scenes,
which are composed of semantically correlated shots. Each shot is composed of
a sequence of video frames representing a continuous action. For example, a
scene describing an office, may consist of shots describing actions of a
clerk, such as typing and walking. In some cases a scene and a shot may be
identical if the length of the first is small.

Videos are often lengthy in duration with complex content, including duplicate
video contents or unuseful information regarding specific application
contexts. Additionally, considering the huge amount of data that is
transferred on a daily basis throughout various online platforms, the need of
summaries describing their content becomes crucial. In many cases removing
redundant or unuseful content can be beneficial, e.g. to shorten its duration
or focus only on important video segments. To this end, various video
summarization (VS) approaches have been proposed throughout different
application contexts such as in medicine, where the time of diagnosis can be
drastically reduced iakovidis2010reduction ; in entertainment, where video
selection can be based on a thumbnail or a movie trailer respectively
DBLP:journals/mta/ShengCLL18 ; in surveillance for security, in order to avoid
hours of footage SenthilMurugan2018  etc.

VS methods can be categorized, based on their output, as static, where we
obtain keyframes de2011vsumm , or dynamic where summaries are described as
video skims gygli2014creating . In detail, a keyframe is a video frame which
can be regarded as the most useful/representative frames of a video. On the
other hand, as video skims, can be described as short video segments usually
obtained by montaging frames before and after keyframes. The number of
keyframes or the duration of the video skims is usually selected by using a
threshold value. Another categorization can be considered regarding their
execution, i.e. whether is offline de2011vsumm , gygli2014creating , or online
kim2019video , with the majority of them belonging to the former. VS methods
can also be categorized as supervised panda2017weakly  or unsupervised
de2011vsumm , gygli2014creating . Supervised VS methods require a training set
for learning patterns from previously annotated videos, whereas the
unsupervised methods do not require any training, which generally makes them
context independent.

Most of the VS methods incorporate feature extraction, which is a process for
information abstraction enabling discriminative content representation. Such
representations may describe spatial or temporal properties of a video, or
both. Commonly used spatial features include video frame intensity, color,
shape, and texture whereas temporal features include motion and orientation of
consecutive frames. A recent trend to feature extraction suggests the use of
machine learning, e.g. via convolutional neural networks (CNNs) enabling
automatic feature extraction. However, even though such feature extraction
approaches can be very effective, they are usually computationally demanding
and their physical interpretation is not straightforward. Such methods are
usually combined with unsupervised clustering methods including among others
k-means de2011vsumm , lee2017robust , kumar2018eratosthenes ,
lu2017unsupervised , fuzzy c-means (FCM) asadi2012video , k-medoids
singh2019pics , DBLP:journals/mta/LuxMSBL10 , non-negative matrix
factorization (NMF) iakovidis2010reduction , DBLP:journals/mta/LiuSZF19 .

An additional beneficial approach in summarizing videos is that of visual
attention as a result of visual saliency. Visual saliency indicates locations
or proportions of pixels within images or frames that attract human attention.
In other words, visual saliency is how humans would observe these specific
images or frames without any filter in a short period of time. It is often
categorized in bottom-up and top-down visual saliency ishtiaq2017visual . The
bottom-up visual saliency is based on the human stimulus, i.e. a stimulus-
impelled signal which is affected by features such as color, contrast,
intensity, orientation etc. while the top-bottom visual saliency is a user-
impelled signal, i.e. depended on the user and affected by factors such as the
appearance of a scene in a frame.

In order to assess the performance of VS methods, and compare them among other
VS methodologies, we usually test them on specific benchmark datasets
available in literature. Among these datasets include the OV dataset OVP
provided by the Open Video Project, the VSUMM dataset de2011vsumm , VSUMM
consisting on videos collected through the OVP, the SumMe dataset
gygli2014creating , SumMe , the Youtube dataset provided by de2011vsumm , the
TVSum dataset DBLP:conf/cvpr/SongVSJ15 , TVSum . In the scope of this work, we
focus on the VSUMM dataset since it consists of videos of different genres and
it’s been broadly adopted for evaluation purposes by various methods in the
literature.

In this paper we propose SalSum, a novel VS method based on the human eye
fixation saliency information, estimated by a data-driven method. In detail,
this method is being trained by exploiting a GAN model to estimate accurate
saliency maps DBLP:journals/corr/PanCMOTSN17 . To the best of our knowledge
this is the first VS analysis using a trainable saliency generation approach.
By using this model, we generate saliency maps for each video frame. Then, hue
histograms of the video frames are calculated to estimate a static score,
while optical flow calculation between saliency maps, generated from
consecutive video frames, provide a spatiotemporal score. The static and
spatiotemporal score, are fused together forming a final score. The final
keyframes are obtained via a user-based technique in accordance with the
computation of score distance between frames and by using the histogram
intersection dissimilarity measure. SalSum can be considered as a unsupervised
method, since the only training that is required regards the GAN model, which
is trained on estimating visual saliency.

The main contributions of the SalSum method are:

  * •

We propose a novel unsupervised VS approach that exploits the capabilities of
a data-driven saliency estimation model, trained to estimate saliency maps
based on human eye fixations. A saliency property that is suitable for VS
applications is that saliency in general, is preserved with respect to objects
appearing in consecutive video frames.

  * •

We consider spatiotemporal features; a) the perceptual representation of
colors within each video frame and b) the optical flow of the saliency maps.
Any radical change in the optical flow is more likely to be associated with a
change in the video content. Therefore, we obtain more relevant video
summaries with respect to human perception. To the best of our knowledge, this
is the first work using the optical flow via saliency for obtaining video
summaries.

The rest of this paper is organized as follows; In Section 2 we discuss
previous related works on VS via saliency as well as VS approaches used to be
compare with in our evaluation experiments. In Section 3 we discuss the data-
driven saliency estimation model that generates saliency maps used for video
summarization and we analyze our methodology step by step. The results from
the evaluated experiments are presented in Section 4. Finally, we conclude the
main aspects of this paper and discuss future prospects concerning this work.

##  2 Related Work

###  2.1 Saliency-based Video Summarization

There are various saliency-based methods for video summarization proposed over
the last years. Most of them focus on a visual attention approaches computed
by either spatial or temporal features while others consider hybrid features
as well.

In DBLP:journals/jiis/JacobPLP17  a variety of image features are considered
for saliency-based VS. These include color, shape, texture and constrast as
spatial features, motion computed via optical flow and facial recognition to
obtain a spatial, temporal and sematic saliency score respectively. These
scores are fused into a global visual attention model where VS is achieved by
selecting consecutive score peaks. In DBLP:journals/mta/DarabiG17  a
personalized saliency-based VS method was proposed. In that method, a score
was assigned to each video frame based on their audio, visual and textual
characteristics. Spatiotemporal saliency scores were estimated for keyframe
selection based on the standard Itti saliency model
DBLP:journals/pami/IttiKN98  and fused with the aforementioned scores. The
video frames were classified into semantic categories based on Scale-Invariant
Feature Transform (SIFT) visual descriptors. Similarly, in ishtiaq2017visual
a fused saliency model was presented, including both spatial and temporal
saliency computed by color, contrast, luminance and texture, shape, size and
motion features. Another study DBLP:journals/tip/ZhangHJYC17  investigated
event saliency VS where a spatiotemporal saliency model was introduced,
considering motion, vision, concept and speech. These features were extracted
by VGG CNN and then they were grouped into clusters using k-means. The results
presented were promising; however, it was noted that it lacks efficiency when
it comes to noisy videos or complex backgrounds and blurred foregrounds. A
visual attention model for VS was introduced in ubeduddinnovel . It was based
on spatial, temporal and semantic saliency using intensity, color, contrast,
orientation, motion, edges and facial recognition respectively. This approach
has been proved suitable for summarization of sports videos.

In DBLP:journals/isci/MademlisTP18  a global saliency model was presented
where VS was achieved using a salient dictionary where keyframes reconstruct
the whole video and simultaneously represent its most salient parts. For this
approach, were used both low features such as luminance, color (hue), optical
flow, edges and mid features like SIFT and IDT (improved dense trajectories).
By clustering these features using the Jenk’s Natural Breaks Optimization
algorithm the final summaries were obtained. However, dictionary-of-
representatives approaches do not guarantee outlier inclusion. Similarly, an
extended work to DBLP:journals/isci/MademlisTP18  considered an alternative
SVD-based frame saliency method and presented to DBLP:conf/icassp/MademlisTP18
. Specifically, the video frame saliency estimation term was replaced with one
based on Regularized SVD-based Low Rank Approximation, taking advantage of the
well-established correlation between midrange matrix singular values and
salient regions. This method correspondingly, is preferred on activity videos.
In DBLP:journals/mta/HuL18  was proposed a novel VS method by fusing the
global and local importance based on multiple features and image quality.
These features included visual attention (static visual saliency and temporal
saliency), HSV features (hue, saturation, light), rule of thirds, contrast and
directionality and image quality. By fusing the importance an optimization of
the clips achieved an optimal VS. However, if the frames quality is low, the
number of the aforementioned features was redefined. A spatiotemporal
saliency-based model was presented in kwan2018efficient  in order to detect
anomalies and summarize low quality videos. The used features included color,
motion and events in order to compute saliency scores, apply smoothing and
group them into clusters via a temporal clustering based on color. This method
was used in videos that include people and/or vehicle motion, traffic
accidents, abnormal motion, leaving suspicious objects etc. An automatic VS
approach suitable for smart glasses was presented in
DBLP:journals/mta/ChiuLW18  where features such color, texture, motion and
audio (speeches) as well as visual saliency are considered. Through a k-means
clustering on RGB color features the final summaries are selected by an
adjusted rand index (ARI). This method is applied mainly on egocentric videos
both indoors and outdoors. A visual attention unsupervised VS method was
presented in DBLP:journals/mta/SunZZW18  where spatial and temporal features
are extracted using the CNN and optical flow respectively. In addition, the
saliency maps are derived and the final summaries are obtained using the Earth
Mover’s Distance and histogram intersection similarity between them and the
ground truth. In DBLP:journals/ejivp/EjazBMCM18  the considered features
included color, multi-scale contrast extracted by a Gaussian pyramid instead
of usual color contrast to create static visual attention, motion obtained
through optical flow to create a dynamic visual attention and relative motion
orientation. Afterwards, an efficient fusion based on these three attention
values takes place and keyframes are extracted by using the Euclidean
similarity.

A spatial and motion saliency prediction model (SMSP) was introduced in
DBLP:journals/tcsv/PaulS19  based on eye tracker data since human eyes are
able to track moving objects accurately. The amount of salient regions and
object motions are the two important features to measure the viewer’s
attention level for determining the keyframes. This was achieved by extracting
features such as intensity (grayscale) and motion. However, the eye tracking
data concerning each video was obtained by humans watching the video.

Several state-of-the-art methods for saliency map generation based on human
eye-fixation exploit the capabilities of machine learning algorithms, such as
CNNs. Novel architectures using ensemble of deep networks (eDN)
DBLP:conf/cvpr/VigDC14 , dual CNN architectures DBLP:conf/cvpr/PanSNMO16 ,
deep spatial contextual long-term recurrent convolutional network (DSCLRCN)
DBLP:journals/tip/LiuH18  have been deployed for the task of visual saliency
generation. In DeepGaze kummerer2014deep , DeepGaze II kummerer2015information
and SUSiNet Koutras_2019_CVPR_Workshops  selected features from CNN models are
used for the construction of saliency maps based on human eye-fixation while
SalGAN (Saliency Generative Adversarial Network) exploits the GAN properties
for accurate saliency generation DBLP:journals/corr/PanCMOTSN17 .Visual
saliency based on human eye-fixation, has been proven to enable the robustness
of methodologies beyond the context of video summarization dimas2019obstacle .

###  2.2 Video Summarization Methods using the OV dataset

When it comes to the OV dataset there are several methods over the years that
their evaluation is based on this benchmark dataset. Specifically, in
DBLP:journals/jodl/MundurRY06  a VS method was proposed via the Delaunay
Triangulation clustering (DT) where each frame was represented by an HSV color
histogram, each histogram was represented as a row vector and the vectors for
each frame were concatenated into a matrix while principal component analysis
(PCA) was used for dimension reduction. After that the Delaunay diagram was
built and clusters were created based on the edges of this diagram. Finally,
frames nearest to the cluster center were selected as keyframes.

In STIMO (STIll and Moving video storyboard) DBLP:journals/mta/FuriniGMP10  an
on-the-fly VS approach was introduced by clustering the matrix produced by
each frame’s histogram in HSV. It was also used triangular inequality to avoid
redudant distance computations. Finally, the pairwise distance of consecutive
frames was computed and if it were greater than a threshold value, the number
of clusters were obtained where keyframes were extracted from each cluster. In
VSUMM de2011vsumm  a VS method was proposed by extracting color features (in
HSV color space with 16 bins in each channel) and frames were grouped using a
k-means clustering. The Euclidean distance of consecutive frames was measured
and keyframes were selected given a threshold value on this specific distance.
There are two variants of this method, i.e. VSUMM1 and VSUMM2 where in the
first one the keyframes are the centers of the clusters while in the latter
the obtained keyframes are processed by merging similar frames concerning the
distances between the initial keyframes. In SMFR (Sparse Modeling for Finding
Representative objects) Elhamifar_2019_ICCV  is achieved a VS by formulating
this problem as a sparse multiple measurement vector problem. In SD (VS via
Sparse Dictionary) DBLP:journals/tmm/CongYL12  VS was formulated as a novel
dictionary selection problem using sparsity consistency, where a keyframes
dictionary was taken in such a way where the original video could be
reconstructed by this representative dictionary.

In VISON (Vıdeo Summarization for Online applications)
DBLP:journals/prl/AlmeidaLT12  VS was approached on the compressed domain
while the quantity and quality of the final summaries were controlled by
users. In SFKD (Spatio-temporal Feature-based Keyframe Detection)
DBLP:journals/prl/MartinB13  every frame was described as a set of local
features. A framework was proposed in which involved local features comparison
and obtained by consecutive frames while building a graph-based appoach on the
locality of these features. After that, attain trial graph partitions were
attained through spectral clustering and the final keyframes were associated
with these partitions. In KBKS (Keypoint-Based Keyframe Selection)
DBLP:journals/tcsv/GuanWLDF13  was proposed a shot VS in which frames were
described by their local features through a keypoint-based framework.

VSCAN DBLP:conf/iciap/MahmoudIG13  was proposed as a VS method where color and
texture features were employed for each frame while a DBSCAN clustering was
used in order to select final keyframes. The HDPS (High Density Peak Search)
rodriguez2014clustering  was a VS method via clustering in which cluster
centers decribed by a higher density than their neighbors and therefere the
number of clusters is arbitrary. In OffMSR (Offline VS via Minimum Sparse
Reconstruction) DBLP:journals/pr/MeiGWWHF15  the SD method was extended by
reconstructing the original video sequence with the smallest possible number
of keyframes. Both the cases for online and offline VS were examined while two
variants of this method been proposed, i.e. OffMSR¬a and OffMSRm where in the
first the most average frame was obtained as the first keyframe while in the
latter the first keyframe was the one with the maximum magnitude. In VSQUAL
(VS by image QUALity assessment) DBLP:conf/ciarp/CirneP14  VS is achieved by
extracting color features and using a Feature Similarity Index Matrix (FSIM)
which was computed by both the phase congruency and the gradient magnitude
where the first one measures the significance of local features while the
second one keeps the information concerning the contrast of an image. Finally
shots were segmented by finding local minima points between consecutive frames
in a sequence of FSIM and keyframes were selected based on the highest value
of similarity.

The VRHDPS (Video Representation by High Density Peak Searching) approach
Wu2017  is an extention of the HDPS method and a VS method was presented via
clustering where the Euclidean distance was calculated between two candidate
frames, their local density was computed and the minimum distance for each
candidate frame was calculated. In DBLP:conf/cvpr/MahasseniLT17  was presented
SUM-GAN, a VS method using adversarial Long short-term memory (LSTM). Both a
supervised and an unsupervised approach of this method were studied, i.e. SUM-
GANsup and SUM-GANdpp respectively. In SCVS (Sparse Coding VS)
DBLP:journals/ijon/LiYLM17  was proposed a shot boundary detection method by
using sparse coding to learn a dictionary from the given video. This was based
on the idea that different shots cannot be reconstructed using the learned
dictionary. Shot boundary detection was achieved by minimizing the
reconstruction loss, restricting the sparsity of the reconstruction matrix
while preserving video’s structure. Finally, frames with the minimum norm
distance in each shot were selected as keyframes.

In PFCVS (Parameter Free Clustering VS) DBLP:conf/cvip/MishraS16  was
presented a VS method via an optimal k-means clustering using the SD Validity
Index. This index is based on a average scattering of the clusters while is a
useful tool at measuring the homogeneity and compactness of the clusters. In
MusselCirne2018 , the VSQUAL was extended to the VISCOM (Video Summarization
using color co-occurrence matrices) method. The video frames were represented
by the mean values of color co-occurance matrices, a shot boundary detection
was achieved by the normalized sum of squared differences resulting in the
final summaries of the video. In Kumar2018  was presented the ESVS
(Eratosthenes Sieve based VS) method in which color features, spatial
frequency and spectral residual were extracted while an optimal k-means
clustering was used by selecting 5 videos. This method is based on the
Eratosthenes Sieve idea and thus, sets of prime-number frames were used. In
addition, there were two variants of this approach, i.e. AVS and EVS selecting
all frames and equal frames partition respectively.

In martins2018opfsumm  OPFSumm (Optimum-Path Forest Summarization) was
proposed where the optimum path forest clustering was used with k-neighborhood
method on color, spectral and temporal information obtained by each frame. The
MSKVS (Mean Shift-based Keyframe VS) method proposed in
DBLP:journals/jvcir/HannaneEA18  aims to achieve VS by using the mean-shift
clustering algorithm on visual and temporal features of the video frames
obtained by the scale-invariant feature transform (SIFT), the difference of
Gaussian (DoG), the orientation and the entropy. In PICS (Parameter
Independent Clustering Strategy) singh2019pics  a parameter-free VS was
presented via k-medoid clustering on the RGB color features of video frames.
In addition, the Calinski–Harabasz index (CH index) was used enabling users to
select keyframes without any computational cost. In
DBLP:journals/jifs/MohanN19  SAEKMVS (Sparse AutoEncoders and K-Means
clustering VS) was presented where VS was achieved by the Dual Layer Loopy
Belief Propagation Network (DLBPN) and a k-means clustering. In KEGC (Keyframe
Extraction via Graph Clustering) DBLP:journals/sivp/GharbiBZ19  was introduced
a VS approach using graph clustering after local features were extracted from
video frames using the LBP descriptor and points of interest using the SIFT
algorithm. In CENTRIST VS DBLP:journals/mta/BendraouES19  a VS method was
proposed via singular value decomposition (SVD) of centrist features (census
transform histogram), which is a visual descriptor for recognizing places or
scene understanding. Final keyframes were selected by a k-means clustering on
the extracted features.

Although several VS approaches have been proposed, many of which have been
based on visual attention and saliency models, to the best of our knowledge
none of them has considered the optical flow of saliency maps estimated
between consecutive frames. In this paper, unlike previous studies, we
investigate this novel approach using a saliency estimator trained on human
eye fixations. The rationale for this approach relies on the observation that
saliency is generally preserved with respect to the objects presented in
consecutive video frames; therefore, a change in the optical flow of the
saliency is likely to be associated with a change in the video content. Also,
by using a generative model based on human eye fixations along with a
perceptual representation of colors, we expect to obtain more perceptually
relevant video summaries.

##  3 Proposed Methodology

The saliency-based summarization methodology proposed in this work, named
SalSum, is schematically illustrated in Fig. 1. The method considers that a
user selects a prefered number of keyframes, constituting a video summary.
Each video frame is analysed both in terms of color and saliency. Color is
represented by hue histograms, and saliency maps are estimated by the
application of SalGAN DBLP:journals/corr/PanCMOTSN17 . Then, a static score is
estimated from the similarity of the hue histograms, and a temporal score is
computed based on the optical flow between consecutive saliency maps estimated
by SalGAN given the respective RGB input frames. These two saliency scores are
fused together to obtain a final saliency score in which any radical change
may indicate a scene transition. The keyframes are selected by considering the
local minina of the signal formed by this score over time. The details of this
method are provided in the following subsections.

Figure 1: The proposed SalSum methodology.

###  3.1 Saliency Maps Estimation

Several state-of-the-art methods exist for saliency map generation using CNN
models. It is worth mentioning that any of the methods mentioned above provide
us with reliable saliency maps. However, the comparison of these methods
exceeds the main purpose of this study. Hence, for this work, SalGAN has been
selected for the task of visual saliency map generation. SalGAN is an end-to-
end saliency map generation model and among the best performing model in
saliency estimation. SalGAN produces saliency maps with an accuracy comparable
to that of other top-performing models in the saliency generation domain.

SalGAN (Saliency Generative Adversarial Network) is a state-of-the-art data-
driven saliency prediction method DBLP:journals/corr/PanCMOTSN17 .
Specifically, SalGAN is a generative adversarial network (GAN) trained with a
joint loss function consisting of two losses, a recostructional and an
adversarial loss. The binary cross-entropy has been chosen as a
reconstructional loss. The network is trained to produce saliency maps,
predicting the human eye-fixations on images, i.e. where the visual attention
of humans is focused on images. Since SalGAN, is a GAN, two networks are
required during its training, a generator and a discriminator network. The
generator model is a deep convolutional neural network with an encoder-decoder
architecture, while the discriminator is a simple convolutional network with a
fully-connected part. The purpose of the generator is to generate accurate
saliency maps, predicting the human eye-fixation on the input image. The goal
of the discriminator is to distinguish the generated saliency maps from ground
truth data. These two networks are trained in parallel, so the generator
constantly trying to “fool” the discriminator, while the discriminator is
trying to get better into classifying correctly the ground truth and the
generated images. This procedure can be described as a min-max game between
the generator and the discriminator, where the minimization of the generator
loss, maximizes the loss of the discriminator by producing accurate saliency
maps, indistinguishable from the ground truth.

Figure 2: The SalGAN architecture on frame 1200 of video #25 of VSUMM. The
convolutional layers of the encoder are depicted withblue andthe pooling
layers in red color. The convolutional layers of the decoder are in green
andthe up-sampling layers in orange color while the the final layer used for
activation is in gray

The encoder component of the SalGAN network has the same architecture to that
of VGG-16 model simonyan2014very , including both the convolutional and the
max-pooling layers (colored in gray and purple in Fig. 2 respectively).
However, the final pooling layer and the fully connected (FC) part have been
removed for the task of saliency map generation. The maxpooling layers enable
the dimensionality reduction of the feature maps produced by the convolutional
operations on the input data. The encoder was pretrained on ImageNet
deng2009imagenet . The encoder is tasked to produce a latent representation of
the input image which is then used for the saliency map generation. The
decoder part of SalGAN is used to generate a saliency map, given the output of
the encoder. The architecture of the decoder is the same as the encoder, with
the max-pooling layers replaced with up-sampling layers, and the convolutional
layers in reverse order (Fig. 2). The activation function used in all
convolutional layers of the is the Rectified Linear Unit (ReLU), with an
exception to the last convolutional layer of the decoder where the sigmoidal
function (logsig) is employed. Fig. 2 illustrates the SalGAN architecture for
the saliency map generation given an input image. The convolutional layers of
the encoder are illustrated in blue, the maxpooling layers in red while the
decoder’s convolutional layers are in green and the up-sampling layers in
orange. The final convolutional layer is depicted with gray color.

###  3.2 Feature Extraction and Score Estimation

The proposed methodology considers features extracted from both the spatial
and temporal domain. Spatial features include perceptual color, visual
saliency, and optical flow used to assess the temporal changes of saliency
over time; therefore, in that sense, it encodes spatiotemporal saliency.

####  3.2.1 Perceptual color

Throughout this paper we select the color in the spatial domain by computing
the quantized, $n$ bins , hue histograms, where $n={4,8,16,32,64}$. TThis
selection was based on the perceptual nature of the HSV color space, which
resemblance how humans interpret color. In detail, in HSV space, color is
dependent to the lighting conditions of a scene which reflects the how humans
perceive it. Furthermore, the hue channel describes exclusively color and
thus, it is suitable for color feature extraction. In addition, when a
perceptually uniform color space is selected it may be appropriate to use
uniform quantization niranjanan2012performance . Hence, we compute the
consecutive hue histograms, quantized in n bins, and then we measure their
divergence using the normalized histogram intersection dissimilarity distance
which is defined as follows niranjanan2012performance :

$d=1-\sum_{i=1}^{N}\frac{\text{min}\left(\text{hist}_{i},\text{hist}_{i+1}\right)}{\text{max}\left(\text{hist}_{i},\text{hist}_{i+1}\right)}$
(1)

where $N$ is the total number of salient frames used in the analysis and
histi, histi+1 are the $i-$th hue histogram and its consecutive. The smaller
the value of $d$, the more similar the consecutive frames are.

####  3.2.2 Spatiotemporal saliency

In the spatiotemporal domain, we compute the normalized motion intensity
between the saliency maps of the consecutive video frames using optical flow.
This is estimated by the intersection divided by the union of the saliency
maps, i.e. their minimum over their maximum values. The optical flow is
calculated between the saliency maps of consecutive frames of a video $V$, at
$t$ and $t+\Delta t$ respectively. A pixel $p$ of a frame, at time $t$ of a
video $V$, can be described as $p$$=(x,y,t)$ and its intensity value as
$V(\textbf{$p$})$. The corresponding pixel $p$${}^{\prime}=(x+\Delta
x,y+\Delta y,t+\Delta t)$, on the next frame at time $t+\Delta t$ will have an
intensity of $V(\textbf{$p$}^{\prime})$. Assuming that the intensity will be
conserved between the two frame, the following relation is derived:

$V(\textbf{$p$})=V(\textbf{$p$}^{\prime})$ (2)

Considering that the movement between these frames is negligible, the Taylor
expansion of Eq. (2) will give us

$\frac{\partial V}{\partial x}\frac{\Delta x}{\Delta t}+\frac{\partial
V}{\partial y}\frac{\Delta y}{\Delta t}+\frac{\partial V}{\partial t}=0\
\Rightarrow\ V_{x}U_{x}+V_{y}U_{y}+V_{t}=0\ \Rightarrow\
\overrightarrow{\nabla}V\cdot\overrightarrow{U}=-V_{t}$ (3)

where $V_{x}=\partial V/\partial x,V_{y}=\partial V/\partial y$ and
$U_{x}=\Delta x/\Delta t,U_{y}=\Delta y/\Delta t$ are the velocities along the
$x-y$ plane.

However, Eq. (3) has two unknown variables; thus, it cannot be solved. There
are several approaches available in the literature. In this paper, we consider
the Lucas-Kanade optical flow method lucas1981iterative . It should be noted
that KLT uses the spatial intensity information to guide search towards the
direction that secures optimal matching and is faster to several other
aproaches since it checks for significantly fewer potential matches among
images dimas2017intelligent . This method assumes that the flow field is
spatially preservative, i.e. the pixels within a certain window are assumed to
have identical value of velocity. Therefore, if we consider a point $p$ and a
window centered at this point all the N pixels within this window must follow
Eq. (3). Specifically we have

$\displaystyle V_{x}(q_{1})U_{x}+V_{y}(q_{1})U_{y}$
$\displaystyle=-V_{t}(q_{1})$ (4) $\displaystyle
V_{x}(q_{2})U_{x}+V_{y}(q_{2})U_{y}$ $\displaystyle=-V_{t}(q_{2})$
$\displaystyle\vdots$ $\displaystyle\vdots$ $\displaystyle
V_{x}(q_{N})U_{x}+V_{y}(q_{N})U_{y}$ $\displaystyle=-V_{t}(q_{N})$

This problem is of the form $Av=B$, where

$A=\begin{pmatrix}V_{x}(q_{1})&V_{y}(q_{1})\\\ V_{x}(q_{2})&V_{y}(q_{2})\\\
\vdots&\vdots\\\ V_{x}(q_{N})&V_{y}(q_{N})\end{pmatrix},\ \ \
v=\begin{pmatrix}U_{x}\\\ U_{y}\end{pmatrix}\ \ \text{and}\ \
B=-\begin{pmatrix}V_{t}(q_{1})\\\ V_{t}(q_{2})\\\ \vdots\\\
V_{t}(q_{N})\end{pmatrix}$ (5)

This gives an over-determined linear system which can be solved using a least
squares method enabling us to compute the velocity vectors $U_{x}$, $U_{y}$ as
$U=(A^{T}A)^{-1}A^{T}B$, where $A^{T}$ is the transpose matrix of $A$.

Then, the motion intensity (magnitude) is computed as
$\sqrt{U_{x}^{2}+U_{y}^{2}}$ and is used for the temporal saliency score in
our analysis. An example of this score is shown in Fig. 3.

Figure 3: A visual representation of the Optical Flow estimation based on the
saliency information, i points to the keyframe index.

####  3.2.3 Score fusion for keyframe extraction

Both the static and the temporal saliency scores are fused into a final
saliency score as

$\text{FinalScore}=\sum_{i}\frac{S_{i}}{\text{variance}(S_{i})},\ \
i=\\{1,2\\}$ (6)

where $S_{i}$ represents the different scores considered. In this paper
$S_{1}$ and $S_{2}$ are the static and temporal scores respectively. To
maximize the quality of this score we also apply a smoothing filter to Eq.
(6).

Using Eq. (1) we obtain the static saliency score $d$ of a video, i.e.
$S_{1}$. In addition, using Eq. (5) we compute the temporal saliency score,
i.e. $S_{2}$. In Fig. 4, an example of this process is illustrated, where we
show both the static and the temporal saliency scores along with the final
saliency score as a result of the fusion for the video #25 of VSUMM dataset.
Minima around peaks with non-negligible difference are candidates to be
keyframes, as the probabillity to indicate different scenes is high.

Figure 4: Static, Temporal and Final Saliency scores for video #25 of the
VSUMM dataset.

##  4 Experiments – Results

###  4.1 Experimental Setting

The variaty of VS methods compared on specific datasets is quite large
enabling us to test the effectiveness of SalSum. In this section we present
our experimental setting and the performance evaluation process used in our
analysis. Additionally, a comparison among the proposed SalSum and other VS
methodologies is presented and discussed.

####  4.1.1 Dataset

We conduct our experiments on the public benchmark dataset VSUMM de2011vsumm ,
VSUMM . It is composed of 50 generic videos of several genres such as
documentary, educational, ephemeral, historical, lectures etc. , which are
selected from the open video project (OVP) OVP . Each and every one of these
videos are in MPEG-1 format with a 29.97 framerate value and a size of
$352\times 240$ pixels. Their duration lies between 1 to 4 min while the whole
dataset consists of approximately 75 min of videos. Every video is watched and
evaluated by 5 humans who selected ground truth (GT) frames according to their
subjective opinions. Since this dataset is publicly available and is consisted
of videos along with their respective summarization GT, its popularity is
quite large and thus, several VS methods select it to perform their
experiments Kumar2018 , singh2019pics , DBLP:journals/jodl/MundurRY06
-DBLP:journals/mta/BendraouES19 .

####  4.1.2 Evaluation

The precision, recall and f-measure metric scores are employed in this work
for the quantitative evaluation of the proposed methodology. The precision
measure is defined as

$\text{precision}=\frac{N_{\text{match}}}{N_{\text{candidate}}}$ (7)

and determines how many selected frames are relevant. $N_{\text{match}}$ is
the number of the correct matches between the GT and automatic summaries
obtained from any VS method and $N_{\text{candidate}}$ is the total number of
the automatic summaries by a specific VS approach. This measures the optical
comprehensiveness of a video, i.e. the accuracy of the VS.

The recall can be described as a coherence metric of a video and determines
how many relevant frames are selected. It is defined as

$\text{recall}=\frac{N_{\text{match}}}{N_{\text{GT}}}$ (8)

where $N_{\text{GT}}$ is the total number of the GT frames provided by a user.

While these metrics describe well enough the summarization quality, the most
common and used metric is the harmonic mean of both the precision and recall
known as f-measure and defined as

$\text{f-measure}=\frac{2\times\text{precision}\times\text{recall}}{\text{precision}+\text{recall}}$
(9)

In order to conclude which features contribute the most to our methodology,
several approaches were compared. These include spatial features, i.e. the
perceptual hue color space with various histogram bin number, spatiotemporal
features, i.e. optical flow, and texture features individually and/or fused
together by various fusion forms. These features concern either the video
frames or the saliency maps generated by the corre-sponding frames or both.
The intensity of the saliency maps was considered as well. The results of this
comparison are summarized in (Table 1).

This is a measure of the overall quality considering both precision and recall
in its calculation. A high value of the f-measure signifies a higher overall
accuracy of the evaluated methodology. Since we have 5 users per video and
therefore 5 sets of GT, we use the mean f-measure in order to obtain one score
per video.

Table 1: Experiments on various features both individually and variance-based fused. These features concern either the video frames or the corresponding saliency maps. Feature | Score
---|---
Individual Features |
Static (saliency maps) | 0.728
Optical flow (video frames) | 0.776
Optical flow (saliency maps) | 0.806
Hue8 (video frames) | 0.796
Hue16 (video frames) | 0.728
LBP (video frames) | 0.645
LBP (saliency maps) | 0.675
Variance-based fused Features |
Hue16 (video frames) + Optical flow (video frames) | 0.817
Hue32 (video frames) + Optical flow (video frames) | 0.815
Hue16 (video frames) + Static (saliency maps) + Optical flow (video frames) | 0.803
Hue32 (video frames) + Static (saliency maps) + Optical flow (video frames) | 0.799
Hue4 (video frames) + Optical flow (saliency maps) | 0.818
Hue8 (video frames) + Optical flow (saliency maps) | 0.835
Hue16 (video frames) + Optical flow (saliency maps) | 0.827
Hue32 (video frames) + Optical flow (saliency maps) | 0.824
Hue64 (video frames) + Optical flow (saliency maps) | 0.827
Hue16 (video frames) + Static (saliency maps) + Optical flow (saliency maps) | 0.771
Hue32 (video frames) + Static (saliency maps) + Optical flow (saliency maps) | 0.763
HSV8 (video frames) + Optical flow (saliency maps) | 0.823
HSV16 (video frames) + Optical flow (saliency maps) | 0.827
HSV32 (video frames) + Optical flow (saliency maps) | 0.828
HSV64 (video frames) + Optical flow (saliency maps) | 0.810

###  4.2 Results and Comparison with existing methods

Initially, these features were examined individually in order to find the most
efficient contribution for both the video frames and the corresponding
saliency maps. The hue color space quantized to 8 bins gives better accuracy
for video frames, whereas optical flow contributes the most to its saliency
maps. After that, several feature combinations were applied using the
variance-based fusion of Eq. (6). Although, most combinations give similar
results, the most efficient is the one of the individual features that give
the most promising results, fused together.

The final fused score introduced in Section 3.2.3 (Eq. (6)) was obtained after
experimenting and comparing various fusion techniques for both the static and
temporal features. However, we considered many fusion approaches such as the
linear, arbitrary weights, min$/$max etc. evangelopoulos2013multimodal ,
zlatintsi2012saliency  and we concluded that the variance-based fusion gives
us better quality of the final summaries over the others (Table 2).

Table 2: Fusion techniques considered and their corresponding f-measures. The
variance-based is preferable. We use as $S_{1}$ and $S_{2}$ the static and
temporal scores respectively.

Fusion | Form | f-measure
---|---|---
Linear | $\sum_{i}w_{i}S_{i}$ | 0.807
Min | min$(S_{i})$ | 0.798
Max | max$(S_{i})$ | 0.798
Exponential | $\sum_{i}(1-w_{t})S_{i},\ \ w_{t}=de^{1-d},\ \ d=\text{max}(\text{frame}_{1})-\text{min}(\text{frame}_{2}))$ | 0.803
Logarithmin | $\sum_{i}\text{min}(S_{i}-w_{i})+\text{max}(w_{i}),\ \ w_{i}=\log(1/\text{var}(S_{i}))$ | 0.821
Complex | $\sqrt{S^{2}},\ \ S=S_{1}+iS_{2}$ | 0.826
Harmonic Mean | $2\times\prod_{i}S_{i}/\sum_{i}S_{i}$ | 0.810
Variance-based | $\sum_{i}S_{i}/\text{var}(S_{i})$ | 0.835

In order to obtain more robust conclusions about the performance of the
proposed method in comparison to the state-of-the-art methods, we have
selected the VS methods that conduct their experiments on VSUMM. These methods
were published in the timespan of 2006-2019 and were described in Section 2.
In Table 3 we summarize the f-measure scores of these methods along with the
one obtained by our methodology. The best score appears in boldface
typesetting whereas the second best is underlined. It can be observed that the
SalSum method provides an adequate summarization for the VSUMM dataset.

Table 3: A comparison between the f-measure scores obtained by several VS
methods using the VSUMM dataset and the one obtained by the SalSum method.
Best score appears in boldface typesetting whereas second best is underlined.

Method | Year | f-measure | Reference
---|---|---|---
DT | 2006 | 0.711 | DBLP:journals/jodl/MundurRY06
STIMO | 2010 | 0.749 | DBLP:journals/mta/FuriniGMP10
OV | 2011 | 0.736 | OVP
VSUMM1 | 2011 | 0.788 | de2011vsumm
VSUMM2 | 2011 | 0.772 | de2011vsumm
SMFR | 2012 | 0.650 |  DBLP:conf/cvpr/ElhamifarSV12 , DBLP:journals/ijon/LiYLM17
SD | 2012 | 0.483 |  DBLP:journals/tmm/CongYL12 , DBLP:conf/cvip/MishraS16
VISON | 2012 | 0.632 |  DBLP:journals/prl/AlmeidaLT12 , MusselCirne2018
SFKD | 2013 | 0.531 |  DBLP:journals/prl/MartinB13 , DBLP:journals/ijon/LiYLM17
KBKS | 2013 | 0.460 |  DBLP:journals/tcsv/GuanWLDF13 , DBLP:conf/cvip/MishraS16
VSCAN | 2013 | 0.713 |  DBLP:conf/iciap/MahmoudIG13 , MusselCirne2018
HDPS | 2014 | 0.480 |  rodriguez2014clustering , Wu2017
OffMSR${}_{\text{a}}$ | 2015 | 0.580 | DBLP:journals/pr/MeiGWWHF15
OffMSRm | 2015 | 0.585 | DBLP:journals/pr/MeiGWWHF15
OffMSR | 2015 | 0.569 | DBLP:journals/pr/MeiGWWHF15
VSQUAL | 2016 | 0.636 |  DBLP:conf/ciarp/CirneP14 , MusselCirne2018
VRHDPS | 2017 | 0.630 | Wu2017
SUM-GAN${}_{\text{dpp}}$ | 2017 | 0.728 | DBLP:conf/cvpr/MahasseniLT17
SUM-GAN${}_{\text{sup}}$ | 2017 | 0.773 | DBLP:conf/cvpr/MahasseniLT17
SCVS | 2017 | 0.734 | DBLP:journals/ijon/LiYLM17
PFCVS | 2017 | 0.609 | DBLP:conf/cvip/MishraS16
VISCOM | 2017 | 0.721 | MusselCirne2018
AVS | 2018 | 0.648 | Kumar2018
EVS | 2018 | 0.631 | Kumar2018
ESVS | 2018 | 0.654 | Kumar2018
OPFSumm | 2018 | 0.728 | martins2018opfsumm
MSKVS | 2018 | 0.616 | DBLP:journals/jvcir/HannaneEA18
PICS | 2019 | 0.640 | singh2019pics
SAEKMVS | 2019 | 0.720 | DBLP:journals/jifs/MohanN19
KEGC | 2019 | 0.723 | DBLP:journals/sivp/GharbiBZ19
CENTRIST VS | 2019 | 0.600 | DBLP:journals/mta/BendraouES19
Proposed | 2020 | 0.835 |

For completeness we note that the f-measure values obtained via SalSum for
each video in the VSUMM dataset as long as the selected number of keyframes
are shown in the Table 4.

Table 4: F-measure scores for each individual video of the VSUMM dataset
obtained by the SalSum method. Mean score is in bold

Video | F-measure | Video | F-measure | Video | F-measure
---|---|---|---|---|---
v21 | 0.6877 | v38 | 0.8301 | v55 | 0.6367
v22 | 0.9333 | v39 | 0.8686 | v56 | 0.8878
v23 | 0.7529 | v40 | 0.8950 | v57 | 0.8572
v24 | 0.8474 | v41 | 0.8552 | v58 | 0.8679
v25 | 0.8690 | v42 | 0.8984 | v59 | 0.9018
v26 | 0.8369 | v43 | 0.9102 | v60 | 0.8756
v27 | 0.7554 | v44 | 0.9778 | v61 | 0.8214
v28 | 0.9123 | v45 | 0.8659 | v62 | 0.6500
v29 | 0.9217 | v46 | 0.7917 | v63 | 0.9179
v30 | 0.7721 | v47 | 10.000 | v64 | 0.8756
v31 | 0.8637 | v48 | 0.9333 | v65 | 0.8663
v32 | 0.7036 | v49 | 0.7329 | v66 | 0.7333
v33 | 0.8742 | v50 | 0.8072 | v67 | 0.7563
v34 | 0.8742 | v51 | 0.8095 | v68 | 0.7300
v35 | 0.8722 | v52 | 0.8964 | v69 | 0.7989
v36 | 0.7000 | v53 | 0.8167 | v70 | 0.8800
v37 | 0.8000 | v54 | 0.8493 | Mean | 0.8354

A qualitative representation of the SalSUM performance is presented in Fig. 5.
As it can be noticed, the selected keyframes by the proposed methodology are
very closely related to and almost match the user-selected ones.

Figure 5: Representative qualitative results. Keyframes selected by the user
(first row) vs. keyframes selected by SalSum (second row) from video #25 of
the VSUMM dataset. (a) depicts airplane take-off, (b) a postal box and (c) a
small sign, (d) a large sign.

##  5 Conclusions - Future Prospects

In this work we introduce a novel VS method called SalSum. Salsum is based on
both saliency information, deriving from human eye fixations estimates, in the
form of saliency maps, provided by GAN model, and perceptual color cues. The
incorporation of saliency and color information is achieved through the
application of optical flow on saliency maps, capturing the spatial flow of
attention in consecutive frames; while the perceptual color aspect is assessed
by the quantization of the hue channel, of HSV color space, on a frame. To
determine the most effective way for the feature combination, a series of
experiments were performed were SalSum performance was examined regarding both
individual use of features and feature fusion using various techniques. The
results obtained, indicate that a variance-based fusion between quantized hue
of video frames and the optical flow of the corresponding saliency maps,
provides the best video summarization performance. With this approach, the
proposed method is able obtain perceptually relevant and intuitive video
summaries, since any radical change both in the spatial attention flow and
color information, is most likely to indicate a change in the video content.
SalSum, when compared to several state-of-the-art VS approaches using the
popular VSUMM dataset, outperformed them scoring an average f-measure score of
0.835 where the second best performing method achieved a score of 0.788.

As a future work, the incorporation of additional features will be considered,
such as specified contexts within a video that interest individuals the most
and/or human-based criteria like quality, quantity, beatifulness among others.
Since human opinions may differ among different viewers, another interesting
direction would be to study the GT selection in a video and compare humans
themselves in order to achieve not the best VS that a user needs but the one
that actually wants and suits his/her preferences.

## Acknowledgments

This research has been co-financed by the European Union and Greek national
funds through the Operational Program Competitiveness, Entrepreneurship and
Innovation, under the call RESEARCH - CREATE - INNOVATE (project
code:T1EDK-02070)

## References

  * (1) Dimitrios K Iakovidis, Spyros Tsevas, and Andreas Polydorou,  “Reduction of capsule endoscopy reading times by unsupervised image mining,”  Computerized Medical Imaging and Graphics, vol. 34, no. 6, pp. 471–478, 2010.
  * (2) Jiachuan Sheng, Yaqi Chen, Yuzhi Li, and Liang Li,  “Embedded learning for computerized production of movie trailers,”  Multimedia Tools Appl., vol. 77, no. 22, pp. 29347–29365, 2018\.
  * (3) A. Senthil Murugan, K. Suganya Devi, A. Sivaranjani, and P. Srinivasan,  “A study on various methods used for video summarization and moving object detection for video surveillance applications,”  Multimedia Tools and Applications, vol. 77, no. 18, pp. 23273–23290, Sep 2018.
  * (4) Sandra Eliza Fontes De Avila, Ana Paula Brandão Lopes, Antonio da Luz Jr, and Arnaldo de Albuquerque Araújo,  “Vsumm: A mechanism designed to produce static video summaries and a novel evaluation method,”  Pattern Recognition Letters, vol. 32, no. 1, pp. 56–68, 2011.
  * (5) Michael Gygli, Helmut Grabner, Hayko Riemenschneider, and Luc Van Gool,  “Creating summaries from user videos,”  in European conference on computer vision. Springer, 2014, pp. 505–520.
  * (6) Hyun Hee Kim and Yong Ho Kim,  “Video summarization using event-related potential responses to shot boundaries in real-time video watching,”  Journal of the Association for Information Science and Technology, vol. 70, no. 2, pp. 164–175, 2019.
  * (7) Rameswar Panda, Abir Das, Ziyan Wu, Jan Ernst, and Amit K Roy-Chowdhury,  “Weakly supervised summarization of web videos,”  in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 3657–3666.
  * (8) Greg C Lee, Fu-Hao Yeh, Ying-Ju Chen, and Tao-Ku Chang,  “Robust handwriting extraction and lecture video summarization,”  Multimedia Tools and Applications, vol. 76, no. 5, pp. 7067–7085, 2017.
  * (9) Krishan Kumar, Deepti D Shrimankar, and Navjot Singh,  “Eratosthenes sieve based key-frame extraction technique for event summarization in videos,”  Multimedia Tools and Applications, vol. 77, no. 6, pp. 7383–7404, 2018.
  * (10) Guoliang Lu, Yiqi Zhou, Xueyong Li, and Peng Yan,  “Unsupervised, efficient and scalable key-frame selection for automatic summarization of surveillance videos,”  Multimedia Tools and Applications, vol. 76, no. 5, pp. 6309–6331, 2017.
  * (11) Ebrahim Asadi and Nasrolla Moghadam Charkari,  “Video summarization using fuzzy c-means clustering,”  in 20th Iranian Conference on Electrical Engineering (ICEE2012). IEEE, 2012, pp. 690–694.
  * (12) Gagandeep Singh, Navjot Singh, and Krishan Kumar,  “Pics: A novel technique for video summarization,”  in Machine Intelligence and Signal Analysis, pp. 411–421. Springer, 2019.
  * (13) Mathias Lux, Oge Marques, Klaus Schöffmann, László Böszörményi, and Georg Lajtai,  “A novel tool for summarization of arthroscopic videos,”  Multimedia Tools Appl., vol. 46, no. 2-3, pp. 521–544, 2010.
  * (14) Huaping Liu, Fuchun Sun, Xinyu Zhang, and Bin Fang,  “Interactive video summarization with human intentions,”  Multimedia Tools Appl., vol. 78, no. 2, pp. 1737–1755, 2019.
  * (15) Uzair Ishtiaq and Tanzila Saba,  “Visual attention modelling and applications in image and video processing,”  Journal of Engineering Technology (ISSN: 0747-9964), vol. 6, no. 1, pp. 146–163, 2017.
  * (16) OpenVideoProject,  “http://www.open-video.org/,” online source, 2011,  Accesed: December 2019.
  * (17) VSUMM,  “https://sites.google.com/site/vsummsite/,” Online, 2011,  Accessed: December 2019.
  * (18) SumMe Dataset,  “https://gyglim.github.io/me/vsum/index.html,” Online, 2014,  Accessed: December 2019.
  * (19) Yale Song, Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes,  “Tvsum: Summarizing web videos using titles,”  in IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, 2015, pp. 5179–5187.
  * (20) TVSum Dataset,  “https://github.com/yalesong/tvsum,” Online, 2015,  Accessed: December 2019.
  * (21) Junting Pan, Cristian Canton-Ferrer, Kevin McGuinness, Noel E. O’Connor, Jordi Torres, Elisa Sayrol, and Xavier Gir’o i Nieto,  “Salgan: Visual saliency prediction with generative adversarial networks,”  CoRR, vol. abs/1701.01081, 2017.
  * (22) Hugo Jacob, Flávio L. C. Pádua, Anísio Lacerda, and Adriano C. M. Pereira,  “A video summarization approach based on the emulation of bottom-up mechanisms of visual attention,”  J. Intell. Inf. Syst., vol. 49, no. 2, pp. 193–211, 2017.
  * (23) Kaveh Darabi and Gheorghita Ghinea,  “User-centred personalised video abstraction approach adopting SIFT features,”  Multimedia Tools Appl., vol. 76, no. 2, pp. 2353–2378, 2017.
  * (24) Laurent Itti, Christof Koch, and Ernst Niebur,  “A model of saliency-based visual attention for rapid scene analysis,”  IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 11, pp. 1254–1259, 1998.
  * (25) Dingwen Zhang, Junwei Han, Lu Jiang, Senmao Ye, and Xiaojun Chang,  “Revealing event saliency in unconstrained video collection,”  IEEE Trans. Image Processing, vol. 26, no. 4, pp. 1746–1758, 2017\.
  * (26) Mohammed Ubeduddin and MR Banwaskar,  “A novel visual-saliency attention-based key-frame accumulation in videos,”  2017\.
  * (27) Ioannis Mademlis, Anastasios Tefas, and Ioannis Pitas,  “A salient dictionary learning framework for activity video summarization via key-frame extraction,”  Inf. Sci., vol. 432, pp. 319–331, 2018.
  * (28) Ioannis Mademlis, Anastasios Tefas, and Ioannis Pitas,  “Regularized svd-based video frame saliency for unsupervised activity video summarization,”  in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2018, Calgary, AB, Canada, April 15-20, 2018, 2018, pp. 2691–2695.
  * (29) Tongling Hu and Zechao Li,  “Video summarization via exploring the global and local importance,”  Multimedia Tools Appl., vol. 77, no. 17, pp. 22083–22098, 2018\.
  * (30) Chiman Kwan, Jin Zhou, Zheshen Wang, and Baoxin Li,  “Efficient anomaly detection algorithms for summarizing low quality videos,”  in Pattern Recognition and Tracking XXIX. International Society for Optics and Photonics, 2018, vol. 10649, p. 1064906.
  * (31) Yen-Chia Chiu, Li-Yi Liu, and Tsaipei Wang,  “Automatic segmentation and summarization for videos taken with smart glasses,”  Multimedia Tools Appl., vol. 77, no. 10, pp. 12679–12699, 2018\.
  * (32) Meijun Sun, Ziqi Zhou, Dong Zhang, and Zheng Wang,  “Hybrid convolutional neural networks and optical flow for video visual attention prediction,”  Multimedia Tools Appl., vol. 77, no. 22, pp. 29231–29244, 2018\.
  * (33) Naveed Ejaz, Sung Wook Baik, Hammad Majeed, Hangbae Chang, and Irfan Mehmood,  “Multi-scale contrast and relative motion-based key frame extraction,”  EURASIP J. Image and Video Processing, vol. 2018, pp. 40, 2018\.
  * (34) Manoranjan Paul and Md. Musfequs Salehin,  “Spatial and motion saliency prediction method using eye tracker data for video summarization,”  IEEE Trans. Circuits Syst. Video Techn., vol. 29, no. 6, pp. 1856–1867, 2019.
  * (35) Eleonora Vig, Michael Dorr, and David D. Cox,  “Large-scale optimization of hierarchical features for saliency prediction in natural images,”  in 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, 2014, pp. 2798–2805.
  * (36) Junting Pan, Elisa Sayrol, Xavier Gir’o i Nieto, Kevin McGuinness, and Noel E. O’Connor,  “Shallow and deep convolutional networks for saliency prediction,”  in 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, 2016, pp. 598–606.
  * (37) Nian Liu and Junwei Han,  “A deep spatial contextual long-term recurrent convolutional network for saliency detection,”  IEEE Trans. Image Processing, vol. 27, no. 7, pp. 3264–3274, 2018\.
  * (38) Matthias Kümmerer, Lucas Theis, and Matthias Bethge,  “Deep gaze i: Boosting saliency prediction with feature maps trained on imagenet,”  arXiv preprint arXiv:1411.1045, 2014.
  * (39) Matthias Kümmerer, Thomas SA Wallis, and Matthias Bethge,  “Information-theoretic model comparison unifies saliency metrics,”  Proceedings of the National Academy of Sciences, vol. 112, no. 52, pp. 16054–16059, 2015.
  * (40) Petros Koutras and Petros Maragos,  “Susinet: See, understand and summarize it,”  in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.
  * (41) George Dimas, Charis Ntakolia, and Dimitris K Iakovidis,  “Obstacle detection based on generative adversarial networks and fuzzy sets for computer-assisted navigation,”  in International Conference on Engineering Applications of Neural Networks. Springer, 2019, pp. 533–544.
  * (42) Padmavathi Mundur, Yong Rao, and Yelena Yesha,  “Keyframe-based video summarization using delaunay clustering,”  Int. J. on Digital Libraries, vol. 6, no. 2, pp. 219–232, 2006\.
  * (43) Marco Furini, Filippo Geraci, Manuela Montangero, and Marco Pellegrini,  “STIMO: still and moving video storyboard for the web scenario,”  Multimedia Tools Appl., vol. 46, no. 1, pp. 47–69, 2010.
  * (44) Ehsan Elhamifar and Zwe Naing,  “Unsupervised procedure learning via joint dynamic summarization,”  in The IEEE International Conference on Computer Vision (ICCV), October 2019.
  * (45) Yang Cong, Junsong Yuan, and Jiebo Luo,  “Towards scalable summarization of consumer videos via sparse dictionary selection,”  IEEE Trans. Multimedia, vol. 14, no. 1, pp. 66–75, 2012.
  * (46) Jurandy Almeida, Neucimar J. Leite, and Ricardo da Silva Torres,  “VISON: video summarization for online applications,”  Pattern Recognition Letters, vol. 33, no. 4, pp. 397–409, 2012\.
  * (47) Ricardo Vázquez Martín and Antonio Bandera,  “Spatio-temporal feature-based keyframe detection from video shots using spectral clustering,”  Pattern Recognition Letters, vol. 34, no. 7, pp. 770–779, 2013\.
  * (48) Genliang Guan, Zhiyong Wang, Shiyang Lu, Jeremiah Da Deng, and David Dagan Feng,  “Keypoint-based keyframe selection,”  IEEE Trans. Circuits Syst. Video Techn., vol. 23, no. 4, pp. 729–734, 2013.
  * (49) Karim M. Mahmoud, Mohamed A. Ismail, and Nagia M. Ghanem,  “VSCAN: an enhanced video summarization using density-based spatial clustering,”  in Image Analysis and Processing - ICIAP 2013 - 17th International Conference, Naples, Italy, September 9-13, 2013. Proceedings, Part I, 2013, pp. 733–742.
  * (50) Alex Rodriguez and Alessandro Laio,  “Clustering by fast search and find of density peaks,”  Science, vol. 344, no. 6191, pp. 1492–1496, 2014.
  * (51) Shaohui Mei, Genliang Guan, Zhiyong Wang, Shuai Wan, Mingyi He, and David Dagan Feng,  “Video summarization via minimum sparse reconstruction,”  Pattern Recognition, vol. 48, no. 2, pp. 522–533, 2015.
  * (52) Marcos Vinicius Mussel Cirne and Hélio Pedrini,  “Summarization of videos by image quality assessment,”  in Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications - 19th Iberoamerican Congress, CIARP 2014, Puerto Vallarta, Mexico, November 2-5, 2014. Proceedings, 2014, pp. 901–908.
  * (53) Jiaxin Wu, Sheng-hua Zhong, Jianmin Jiang, and Yunyun Yang,  “A novel clustering method for static video summarization,”  Multimedia Tools and Applications, vol. 76, no. 7, pp. 9625–9641, Apr 2017.
  * (54) Behrooz Mahasseni, Michael Lam, and Sinisa Todorovic,  “Unsupervised video summarization with adversarial LSTM networks,”  in 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, 2017, pp. 2982–2991.
  * (55) Jiatong Li, Ting Yao, Qiang Ling, and Tao Mei,  “Detecting shot boundary with sparse coding for video summarization,”  Neurocomputing, vol. 266, pp. 66–78, 2017.
  * (56) Deepak Kumar Mishra and Navjot Singh,  “Parameter free clustering approach for event summarization in videos,”  in Proceedings of International Conference on Computer Vision and Image Processing - CVIP 2016, Indian Institute of Technology Roorkee, February 26-28, 2016, Volume 1, 2016, pp. 389–397.
  * (57) Marcos Vinicius Mussel Cirne and Helio Pedrini,  “Viscom: A robust video summarization approach using color co-occurrence matrices,”  Multimedia Tools and Applications, vol. 77, no. 1, pp. 857–875, Jan 2018.
  * (58) Krishan Kumar, Deepti D. Shrimankar, and Navjot Singh,  “Eratosthenes sieve based key-frame extraction technique for event summarization in videos,”  Multimedia Tools and Applications, vol. 77, no. 6, pp. 7383–7404, Mar 2018.
  * (59) Guilherme B Martins, Danillo R Pereira, Jurandy G Almeida, Victor Hugo C de Albuquerque, and João Paulo Papa,  “Opfsumm: on the video summarization using optimum-path forest,”  Multimedia Tools and Applications, pp. 1–17, 2018.
  * (60) Rachida Hannane, Abdessamad Elboushaki, and Karim Afdel,  “MSKVS: adaptive mean shift-based keyframe extraction for video summarization and a new objective verification approach,”  J. Visual Communication and Image Representation, vol. 55, pp. 179–200, 2018.
  * (61) Jesna Mohan and Madhu S. Nair,  “Domain independent static video summarization using sparse autoencoders and k-means clustering,”  Journal of Intelligent and Fuzzy Systems, vol. 36, no. 3, pp. 1945–1955, 2019.
  * (62) Hana Gharbi, Sahbi Bahroun, and Ezzeddine Zagrouba,  “Key frame extraction for video summarization using local description and repeatability graph clustering,”  Signal, Image and Video Processing, vol. 13, no. 3, pp. 507–515, 2019.
  * (63) Youssef Bendraou, Fedwa Essannouni, and Ahmed Salam,  “From local to global key-frame extraction based on important scenes using SVD of centrist features,”  Multimedia Tools Appl., vol. 78, no. 2, pp. 1441–1456, 2019.
  * (64) Karen Simonyan and Andrew Zisserman,  “Very deep convolutional networks for large-scale image recognition,”  arXiv preprint arXiv:1409.1556, 2014.
  * (65) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei,  “Imagenet: A large-scale hierarchical image database,”  in 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009, pp. 248–255.
  * (66) S Niranjanan and SP Raja Gopalan,  “Performance efficiency of quantization using hsv colour space and intersection distance in cbir,”  International Journal of Computer Applications, vol. 42, no. 21, pp. 48–55, 2012.
  * (67) Bruce D Lucas, Takeo Kanade, et al.,  “An iterative image registration technique with an application to stereo vision,”  1981\.
  * (68) George Dimas, Evaggelos Spyrou, Dimitris K Iakovidis, and Anastasios Koulaouzidis,  “Intelligent visual localization of wireless capsule endoscopes enhanced by color information,”  Computers in Biology and Medicine, vol. 89, pp. 429–440, 2017.
  * (69) Georgios Evangelopoulos, Athanasia Zlatintsi, Alexandros Potamianos, Petros Maragos, Konstantinos Rapantzikos, Georgios Skoumas, and Yannis Avrithis,  “Multimodal saliency and fusion for movie summarization based on aural, visual, and textual attention,”  IEEE Transactions on Multimedia, vol. 15, no. 7, pp. 1553–1568, 2013.
  * (70) Athanasia Zlatintsi, Petros Maragos, Alexandros Potamianos, and Georgios Evangelopoulos,  “A saliency-based approach to audio event detection and summarization,”  in 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO). IEEE, 2012, pp. 1294–1298.
  * (71) Ehsan Elhamifar, Guillermo Sapiro, and René Vidal,  “See all by looking at a few: Sparse modeling for finding representative objects,”  in 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, June 16-21, 2012, 2012, pp. 1600–1607.