# Empirical Study of Mix-based Data Augmentation Methods in Physiological Time
Series Data

Peikun Guo  Rice University
Houston, USA
<EMAIL_ADDRESS>Huiyuan Yang  Rice University
Houston, USA
<EMAIL_ADDRESS>Akane Sano  Rice University
Houston, USA
<EMAIL_ADDRESS>
###### Abstract

Data augmentation is a common practice to help generalization in the procedure
of deep model training. In the context of physiological time series
classification, previous research has primarily focused on label-invariant
data augmentation methods. However, another class of augmentation techniques
(i.e., Mixup) that emerged in the computer vision field has yet to be fully
explored in the time series domain. In this study, we systematically review
the mix-based augmentations, including mixup, cutmix, and manifold mixup, on
six physiological datasets, evaluating their performance across different
sensory data and classification tasks. Our results demonstrate that the three
mix-based augmentations can consistently improve the performance on the six
datasets. More importantly, the improvement does not rely on expert knowledge
or extensive parameter tuning. Lastly, we provide an overview of the unique
properties of the mix-based augmentation methods and highlight the potential
benefits of using the mix-based augmentation in physiological time series
data. Our code and results are available at https://github.com/comp-well-
org/Mix-Augmentation-for-Physiological-Time-Series-Classification.

###### Index Terms:

Data augmentation, mixup, physiological time series

##  I Introduction

Data augmentation is a crucial regularization technique for deep neural
network models, as it serves to inform the network of potential variations in
the input data during the training stage while preserving the integrity of the
labels. This technique has been shown to improve network generalization,[1] by
not only artificially increasing the size of the dataset but also imparting
inductive bias through the encoding of information related to data
invariances.

Traditional data augmentation techniques aim to increase the statistical
support of the training data distribution by utilizing human knowledge and
adding additional virtual samples from the vicinity distribution of training
samples. This approach has been shown to improve generalization, as
demonstrated in previous literature. Such data augmentations have been
employed actively and effectively in computer vision [2, 3, 1, 4] and speech
recognition and synthesis [5, 6, 7]. The selection of specific augmentation
methods remains a challenging task, as it is often based on heuristics and is
highly dependent on the dataset, task, and even model architecture [8].
However, unlike in other domains, time series, particularly physiological
data, do not follow a straightforward rule for label-invariant transformation.
Methods such as jittering, rotation, scaling, permutation, magnitude warping,
time warping, window slicing, and window warping have been shown to have
unstable performance across datasets and tasks, or require human
understandings of the data [9, 10, 11]. Two significant limitations of
traditional transformation-based augmentations on physiological time series
data are that: (1) certain transformations can be detrimental to the integrity
of the physiological signal, and (2) the majority of traditional augmentations
are data-dependent, lacking generalization and consistency across different
datasets and tasks.

An alternative approach is represented by mixup regularization [4], which is
based on the assumption that linear interpolations of feature vectors should
lead to linear interpolations of the associated targets. Despite its
simplicity, mixup has been shown to be effective across different domains
(computer vision [4, 12, 13] and speech [14, 15, 16]) and different tasks. For
time series classification tasks, previous studies also employed mix-based
augmentations to enhance model representation and generalization [17, 18, 19].
However, none of the previous works provide a thorough empirical study of mix-
based augmentations across various types of physiological times series,
regarding both the quantitative gain in the metrics and the benefits of
feature representation.

Figure 1: The overview of mix-based augmentation procedure for physiological
time series classification. From left to right: two sequences $x_{i},x_{j}$
are shown as raw input signals. For mixup or cutmix, in each training epoch,
virtual samples are created from the mini-batches; for manifold mixup, the
linear combination is applied at the feature map level. For any of the three
mix-based augmentations, the labels for the virtual samples are also mixed
with the corresponding weights, shown on the right. NORM, MI, and STTC are
representative classes of the PTB-XL dataset. See details in Figure 3.

This study aims to evaluate the efficacy of mix-based data augmentation in the
context of time series classification. Our evaluation compares mix-based
augmentation against traditional data augmentation techniques, as classified
in a previous survey, focusing on basic label-invariant time-domain
transformations commonly used in time series classification. The baseline
augmentations evaluated include jittering, rotation, scaling, permutation,
magnitude warping, time warping, window slicing, and window warping. The
unique benefits of mix-based augmentation are evaluated, and the contributions
of this paper can be summarized as follows:

  * •

We present an empirical study of three mixup-based data augmentation methods
(i.e., mixup, cutmix, and manifold mixup) in the context of time series
classification. We provide detailed formulations of these methods and evaluate
their performances on six physiological and biobehavioral datasets.

  * •

Our experiments reveal two significant distinctions between mixup-based
augmentations and traditional data transformations. First, mixup-based methods
do not require human expert priors, making them more practical in various
applications. Second, mixup-based methods consistently achieve higher or
comparable performance compared to traditional methods.

##  II Related works

###  II-A Traditional label-invariant time series data augmentation

For physiological time series data, the time domain transforms manipulate the
original time series directly, as compared to more advanced augmentations
using generative approaches. In this paper, we focus the evaluations on the
following traditional augmentations: jittering, rotation, scaling,
permutation, window warping, and window slicing.

Jittering refers to the injection of Gaussian or more sophisticated noise
patterns, such as spikes and slope-like trends, into the raw signals. The
schemes are introduced in [20]. Rotation for time series is achieved by
multiplying the signal by a random rotation matrix. As it is not as suitable
for time series data as in the image domain, one commonly used special case is
flipping (changing the sign of the original time series). Permutation is a
transformation that randomly shuffles segments of a time series. This
operation does not preserve the sequential information of the original data.
Window slicing or cropping, introduced in [21], randomly extracts continuous
slices from the original samples. The window warping method is uniquely
applicable to time series. It randomly selects a time interval, then upsamples
or downsamples the segment, while keeping the rest of the time ranges
unaltered. Window warping changes the total length of the original signal,
therefore it is usually used along with window cropping. For time series
classification tasks, all the above-mentioned augmentations do not change the
labels of the altered training samples.

The effectiveness of these augmentations has been previously investigated in
the literature. An empirical study about biobehavioral time series data
augmentation [9] concludes that, while some augmentations are beneficial for
biobehavioral classification tasks, their effectiveness varies across
different datasets and model architectures. This finding agrees with [11],
which also highlights the inconsistency of transformation-based augmentations
across different non-physiological time series datasets.

Based on the analysis of the literature, we claim that two significant
limitations exist when applying traditional transformation-based augmentations
to physiological time series data:

  * •

Certain transformations can be detrimental to the integrity of the
physiological signals. For instance, rotation, permutation, and warping can be
more harmful to ECG signals, as ECG beats possess relatively fixed patterns,
such as the order, width, and intensity of the wave components.

  * •

The majority of traditional augmentations are data-dependent, meaning that
they require expert prior knowledge or multiple trials to select an
appropriate transformation for a specific problem, due to the diverse
properties of biobehavioral time series.

###  II-B Mixup: Vicinal Risk Minimization

Mixup is a data augmentation technique introduced by [4] to train neural
networks by constructing virtual training examples using convex combinations
of pairs of examples and their labels.

The methodology behind Mixup, as described in this paper, is rooted in Vicinal
Risk Minimization (VRM), which diverges from the conventional Empirical Risk
Minimization (ERM) by drawing examples from a vicinity distribution of the
training examples. This aims to enlarge the support for the training
distribution. As stated in Section II, prior knowledge has been traditionally
required for the identification of the vicinity or neighborhood. However,
Mixup provides a more practical and data-agnostic alternative, as it does not
necessitate domain expertise. Despite its simplicity, Mixup presents the
following distinct advantages:

  1. 1.

Regularization: The linear relationship established by mixup transformations
between data augmentation and the supervision signal results in a strong
regularization of the model’s state, leading to improved performance.

  2. 2.

Generalization: The authors of previous studies have reported improvements in
generalization error for state-of-the-art models trained on ImageNet, CIFAR,
speech, and tabular datasets when using mixup. Theoretical analysis [13]
suggests that the soft targets of mixup virtual samples aid in model
generalization in a manner similar to label smoothing and knowledge
distillation. Interpolation/extrapolation of nearest neighbors in feature
space can also improve generalization [22].

Since mixup was proposed, many incremental works have emerged and demonstrated
improvements with different focuses, such as cutmix [23], manifold mixup [12],
MixMatch [24], and AlignMix [25]. In this project, we choose to evaluate
mixup, cutmix, and manifold mixup, as they have not been investigated in the
context of time series classification in literature.

###  II-C Mixup for physiological time series data

In the domain of physiological time series classification, mixup has been
employed to enhance generalization during training as demonstrated in prior
studies. [17, 18] employ mixup for better generalization in ECG classification
task, in the training batches of CNN models. [19] states that mixup improves
the generalization performance of the ECG classification model regardless of
leads and evaluation metrics. However, these studies lack a thorough
examination of mixup’s mechanism and reasons for performance improvement, and
none of them have conducted ablation studies about mixup. Furthermore,
although the 1D variant of vanilla Mixup [4] has been utilized in previous
studies, the empirical results for cutmix [23] and manifold mixup [12] are
currently lacking. In [26], the performance of mixup and cutmix were evaluated
on the UEAMTSC dataset [27] using InceptionTime [28] as the baseline model.
However, the subsets of UEAMTSC in that study were of small scale and not
strictly comprised of time series data (e.g.image contours). This paper, on
the other hand, aims to investigate the effectiveness of the mix-based methods
on various physiological time series datasets, utilizing a higher capacity
residual network structure.

Figure 2: Illustrations of traditional time series data augmentations. Orange:
original signal. Green: augmented signals.

##  III Augmentation methods

In this section, we provide a formal introduction and implementation of mix-
based augmentations. Figure 1 shows the overall paradigm of how the mix-based
augmentation is applied during the training process. The intrinsic difference
between mix-based and traditional augmentations will be discussed in section
V. The technical details for the implementation of the augmentations can be
found in section IV-B.

###  III-A Cutout

In this study, the Cutout augmentation serves as a benchmark for comparison
against mix-based data augmentation methods. Cutout, originally introduced in
the computer vision field to address occlusion issues, involves the removal of
contiguous sections of data. To evaluate its effectiveness in the context of
time series classification, a single-item transformation, similar to
traditional label-invariant regularizations, is applied. Specifically, a
random contiguous section of a time series is replaced with zeros, which can
be considered a dropout (zero-masking) operation at the input layer. The size
of the random time segment is fixed, while the starting index of the interval
is randomly drawn from a uniform distribution, and applied to all channels of
a single data point.

###  III-B Mixup

The time series mix-based augmentations in this study leverage multiple data
samples from one training minibatch to generate virtual data points. We
examine three commonly utilized configurations of mix-based augmentations from
the computer vision domain for time series classification problems, namely
mixup [4], cutmix [23], and manifold mixup (layer mixup, [12]).

The mixup augmentation blends random pairs of time series from the training
data. Let $(x,y)$ denote a time series data instance, where
$x\in\mathbb{R}^{L\times C}$, with $L$ representing the length of the sequence
and $C$ denoting the number of channels, and $y\in\mathbb{R}^{K}$ being the
class label with $K$ classes. The mixing ratio randomly drawn from a Beta
distribution is denoted as $\lambda$. Given two samples $(x_{i},y_{i})$ and
$(x_{j},y_{j})$, the mixup augmentation generates virtual training examples
through the following formulation:

$\displaystyle\tilde{x}$ $\displaystyle=\lambda x_{i}+(1-\lambda)x_{j}$ (1)
$\displaystyle\tilde{y}$ $\displaystyle=\lambda y_{i}+(1-\lambda)y_{j}$ (2)

Mixup extends the training distribution by incorporating the the assumption
that linear interpolations of feature vectors should lead to linear
interpolations of the associated targets [4]. Based on the formulation, the
label for a mixup virtual sample is also a mixture of two original one-hot
labels weighted by $\lambda$. The potential effect and benefit of the soft
target will also be discussed in section V-B.

The mixing ratio, $\lambda$, in the mix-based data augmentation approach is
sampled from a Beta distribution. The value of $\lambda$ close to 0 or 1
results in the created virtual time series being more similar to one of the
raw data points, whereas a value of $\lambda$ close to 0.5 results in a more
blended representation of the raw data points. For physiological time series
data, it is desirable to have virtual time series that are similar to one of
the raw data points, as these signals contain delicate features, such as the
intensity of R peaks in ECG signals, which could be easily destroyed with
random mixing.

Despite its simplicity, in the computer vision domain, mixup has allowed
consistently superior performance in the CIFAR-10, CIFAR-100, and ImageNet
image classification datasets[4]. As we will show in the results section,
mixup can also improve classification metrics in time series classification
problems, along with other desirable features.

###  III-C Cutmix

The image augmentation technique Cutmix, proposed in [23], shares similarities
with the technique Mixup. The authors claim a key advantage of Cutmix is its
ability to prevent the occurrence of ambiguous components in the generated
samples caused by mixing, such as blurred image regions [23]. For time series
cutmix, we select a random time segment from a pair of multivariate time
series, then the values of the pair of time series within the segment are
exchanged across all channels. The length of the segment is also determined
randomly through the mixing ratio $\lambda$ drawn from a Beta distribution.

###  III-D Manifold Mixup

Manifold Mixup, presented in [12], demonstrates consistently superior
performance across various computer vision tasks when compared to the original
input-data-mixup approach. Unlike the original mixup, manifold mixup trains
neural networks on linear combinations of hidden representations of training
samples. The literature suggests that higher-level representations obtained
from intermediate layers of the neural network feature extractor are low-
dimensional, therefore, linear interpolations of hidden representations should
cover meaningful regions of the feature space. In this paper, we implement
Layer Mixup on layer 4 of the ResNet, prior to pooling and the classification
head. The labels are also mixed in the same way as mixup and cutmix.

##  IV Experiments

###  IV-A Datasets

We conduct experiments on six biomedical time series datasets, encompassing
diverse data types and varying sizes. The datasets include two ECG datasets,
PTB-XL for cardiac condition classification and Apnea-ECG for sleep apnea
detection, two EEG datasets, Sleep-EDF for sleep stage recognition and MMIDB
for sleep movement detection, and two IMU datasets, PAMAP2 and UCI-HAR for
human activity recognition. Table I provides a summary of the datasets. Note
that in column ”Periodic”, the IMU datasets are tagged as ”motion”, because
IMU data may contain periodic patterns when the recorded activity contains
periodic motion (e.g.walking).

TABLE I: Dataset Summary Dataset | Category | # channels | # classes | Sample length | Periodic | # samples
---|---|---|---|---|---|---
PTB-XL | ECG | 12 | 5 | 1000 | Yes | 17962
Apnea-ECG | ECG | 1 | 2 | 6000 | Yes | 34243
Sleep-EDFE | EEG | 1 | 5 | 3000 | No | 42308
MMIDB-EEG | EEG | 64 | 2 | 640 | No | 4635
PAMAP2 | IMU | 52 | 12 | 1000 | (motion) | 5452
UCI-HAR | IMU | 9 | 6 | 128 | (motion) | 10299

####  IV-A1 PTB-XL

The PTB-XL dataset [29] is an ECG database of 12-lead recordings, containing
44 diagnostic statements grouped into 5 superclasses (normal, conduction
disturbance, myocardial infarction, hypertrophy, and ST-T change). In this
study, we formulate a five-class cardiac abnormality classification problem.
The data is divided into 10 balanced folds, with the first 8 used for
training, the 9th for validation, and the 10th for testing. The data we use is
sampled at 100Hz. The same stratification process is applied to the other
datasets if no validation/test set is provided.

Figure 3: PTB-XL dataset overview. Top: The five-superclass distribution of
PTB-XL (NORM, MI, STTC, CD, HYP). In this paper, we only train with single-
label samples. Bottom: Example waveforms of PTB-XL classes (lead IV), the
x-axis denotes time in ms, and the y-axis is the normalized voltage.

The dataset is highly class-imbalanced as shown in Figure 3, with over half of
the samples labeled normal and the least-represented class (HYP, hypertrophy
in left ventricle) only constituting 3.3% of the data. This challenge is
addressed in Section V through in-batch resampling and mix-based
augmentations.

####  IV-A2 Apnea-ECG

The Apnea-ECG dataset examines the connection between sleep apnea symptoms and
heart activity in humans [30], as monitored through ECG. This dataset has 70
records, sampled at 100 Hz, with 35 records designated for training and the
remaining 35 for testing. Each record is 7-10 hours in length and includes a
continuous ECG signal along with per-minute apnea annotations indicating the
presence or absence of sleep apnea. We segmented the ECG recordings into
60-second frames at 100 Hz, resulting in 17233 samples for the training set
and 17010 samples for the test set.

####  IV-A3 Sleep-EDFE

The Sleep-EDFE dataset is sourced from the publicly accessible Sleep European
Data Format (EDF) database [31] on Physionet [32]. This database contains
full-night PSG sleep records that include two-channel EEG (Fpz-Cz and Pz-Oz),
a horizontal EOG, and EMG signal records, along with corresponding hypnograms
(sleep stage annotations). The EEG signals have a sampling frequency of 100
Hz, and they are divided into 30-second epochs and normalized to have zero
mean and unit standard deviation.

####  IV-A4 MMIDB-EEG

The EEGMMIDB (EEG Motor Movement/Imagery Database) from PhysioNet is collected
using the BCI200 EEG system4 [33]. It records 64 channels of brain signals at
a sampling rate of 160 Hz, totaling over 1500 recordings of 1-2 minutes each.
Subjects are instructed to wear the EEG device and sit in front of a computer
screen, performing specific typing tasks in response to on-screen prompts.

####  IV-A5 PAMAP2

The PAMAP2 dataset [34] comprises recordings from 9 participants who were
asked to perform 12 daily activities, including household tasks and various
exercises (e.g., Nordic walking, playing soccer). Data from accelerometers,
gyroscopes, magnetometers, temperature sensors, and heart rate monitors are
recorded from inertial measurement units placed on the hand, chest, and ankle
over 10 hours, resulting in a 52-dimensional dataset.

####  IV-A6 UCI-HAR

The UCI-HAR dataset [35] was collected from 30 volunteers aged 19 to 48 years.
Participants were instructed to engage in six basic activities, which included
three static postures (standing, sitting, lying) and three dynamic activities
(walking, walking downstairs, walking upstairs). 3-axial accelerometer and
gyroscope signals were recorded at a constant rate of 50 Hz. The data was
collected using smartphones carried by the participants.

###  IV-B Experimental Setup

####  IV-B1 Network architecture

In all experiments, we use a 1D-CNN-based ResNet-18 [36] as the backbone. This
model has convolutions with a kernel size of 3, and stride 2. The blocks in
the ResNet architecture have convolutional layers with 32, 64, 128, and 256
channels respectively. The output after the final block is average pooled in
the temporal dimension, and then a linear layer is applied to predict the
probability of the positive class. Details of the ResNet-18 structure are
summarized in Table.II.

TABLE II: Structure of the ResNet-18 backbone. $B$: batch size; $L$: length of
sequence; $C$: number of channels.

Layer Name Input Shape Output Shape Parameter Reshape Conv1d
$[\mathrm{B},\mathrm{L},\mathrm{C}]$ $[\mathrm{B},\mathrm{C},\mathrm{L}]$ $-$
Layer1 $[\mathrm{C},\mathrm{L}]$ $[\mathrm{B},64,\mathrm{~{}L}]$
$1\mathrm{x}3,64,\max$ pool Layer2 $[\mathrm{B},64,\mathrm{~{}L}/2]$
$[\mathrm{B},64,\mathrm{~{}L}/2]$
$[[1\mathrm{x}3,64],[1\mathrm{x}3,64]]\mathrm{x}2$ Layer3
$[\mathrm{B},128,\mathrm{~{}L}/4]$ $[\mathrm{B},128,\mathrm{~{}L}/4]$
$[[1\mathrm{x}3,128],[1\mathrm{x}3,128]]\mathrm{x}2$ Layer4
$[\mathrm{B},256,\mathrm{~{}L}/8]$ $[\mathrm{B},512,\mathrm{~{}L}/16]$
$[[1\mathrm{x}3,512],[1\mathrm{x}3,256]]\mathrm{x}2$ Average pool
$[\mathrm{B},512,\mathrm{~{}L}/16]$ $[\mathrm{B},512,1]$ $-$ FC
$[\mathrm{B},512]$ $[\mathrm{B}$, Classes $]$ $[512$, Classes $]$

For the manifold mixup experiments, we take the output of layer 4 of ResNet18
as the input for mixing. We also perform t-SNE visualization on the features
extracted from Layer 4 as validation for the quality of class representation.

####  IV-B2 Augmentation Implementation

Following the previous empirical studies for mix-based regularization: mixup
[4] and MixMatch [24], we use $\alpha=0.4$ and $\alpha=0.75$ for mix-related
hyperparameters in our experiments. For cutmix, the ratio of the random
segment length to the signal length is set to 0.2, following [23]. For the
baseline augmentations, the hyperparameters, such as the intensity of scaling
and jittering, are manually chosen following [9].

Following [4], the implementation of the training step is based on the mini-
batches sampled by a data loader. For each minibatch, random shuffling is
applied, and the mixing operations can all be performed in a vectorized
manner, incurring minimal computation overhead. In our PTB-XL profiling
experiments, the mean increase in the processing time of each mini-batch of
size 128 is less than 0.001 second. We also report results for baselines
(denoted as vanilla) that does not use any data augmentation.

####  IV-B3 Optimization

We use the AdamW optimizer with learning rate 0.001 for all datasets. For the
profiling experiments on PTB-XL dataset, we also tested with Adam optimizer
and 0.005 learning rate. We use a step decay learning rate scheduler with step
size 5, and a decay rate of 0.9 across all experiments. Training takes 50
epochs, which was observed to be sufficient for convergence in all datasets.

####  IV-B4 Computing resource

: All model training was performed on a single NVIDIA GTX 2080Ti GPU.

##  V Results and discussion

###  V-A Quantitative performance of mix-base augmentations

TABLE III: Classification accuracies for mix-based augmentation methods and three traditional augmentations are reported on six datasets. The warp column refers to window warping. Bold numbers indicate the best performance. Dataset | baseline | Mixup | Cutmix | Layer mixup | Cutout | Jitter | Scale | Permute | Warp
---|---|---|---|---|---|---|---|---|---
PTB-XL | 77.94 | 78.91 | 79.64 | 78.97 | 77.61 | 77.60 | 78.03 | 76.57 | 75.67
Apnea-ECG | 78.10 | 74.73 | 78.30 | 79.69 | 78.27 | 76.44 | 70.64 | 77.26 | 76.34
Sleep-EDFE | 83.40 | 84.27 | 84.40 | 84.48 | 83.84 | 81.55 | 83.40 | 85.01 | 83.98
MMIDB-EEG | 78.64 | 79.83 | 77.99 | 80.58 | 80.04 | 79.07 | 80.15 | 69.58 | 77.99
PAMAP2 | 93.62 | 95.31 | 95.83 | 93.88 | 86.98 | 86.85 | 88.18 | 85.82 | 87.59
UCI-HAR | 92.77 | 93.62 | 94.26 | 93.11 | 93.58 | 87.95 | 88.63 | 91.89 | 92.94

Experiments were conducted on six datasets of diverse categories with a
ResNet18-1D backbone. The performance of vanilla (no augmentation), cutout,
and three mix-based augmentations are presented in Table III. We summarize our
results as follows.

I).The mix-based data augmentations can achieve superior accuracy in
comparison to traditional augmentation methods. All three mix-based
augmentation techniques were found to outperform the baseline (no
augmentation) in 16 out of 18 experimental trials, with only two exceptions
(mixup for Apnea-ECG and cutmix for MMIDB-EEG). Furthermore, among the six
datasets examined, the majority of the highest accuracy results were achieved
through the use of cutmix and layer mixup.

Overall, the mix-based augmentations outperform the baselines, but the
performance of augmentation methods varies across different datasets. This is
likely due to the unique characteristics of each dataset and the strengths of
each augmentation method. For instance, datasets containing complex temporal
patterns or high levels of noise may benefit from the use of certain mix-based
augmentation methods that are particularly effective at enhancing
classification accuracy. The results in Table III also show that more
comprehensive mixup schemes (cutmix, manifold mixup) help yield better
accuracy.

II).The mix-based augmentations deliver robust performance, and the accuracy
gain is steady. In addition to the quantitative performance advantages, these
techniques are notable for their low dependency on expert knowledge and
parameter tuning. Across all 18 mix-based experimental trials (excluding
cutout), no significant reduction in accuracy was observed in comparison to
the baseline. In contrast, traditional data transformation techniques can
often result in a drastic decrease in accuracy if not implemented
appropriately. For example, applying scaling in Apnea-ECG resulted in a 7.5%
reduction in accuracy, permutation in MMIDB-EEG resulted in a 9.0% reduction,
and jittering in UCI-HAR resulted in a 4.8% reduction. This finding implies
that traditional transformations can undermine crucial features in
physiological signals, such as wave intensity in ECG data and temporal
correlations in EEG data. As a result, these augmentations can generate
virtual samples that deviate from the actual data distribution, potentially
compromising the generalization performance of the model.

Note that in the experiments, we compared the mix-based augmentations against
some individual traditional augmentations. However, to fully harness the
potential of data augmentation, it is compatible to apply mix-based
augmentations in conjunction with one or more traditional augmentations.

###  V-B Profiling mix-based augmentations on PTB-XL

The PTB-XL dataset [29] is a well-studied ECG dataset for cardiac condition
classification, with the current SOTA accuracy of recognizing five classes
being less than 80% [9]. To evaluate the effectiveness of mix-based data
augmentation methods, extensive profiling experiments were conducted on the
PTB-XL dataset, using over 80 combinations of settings and hyperparameters
(Section IV-B3). The scatter plot of the best validation accuracy vs the best
F1 score is shown in Figure 4(a). Identical combinations of learning rates,
optimizers, etc. were used for each of the four augmentation setups (vanilla,
input mixup, cutmix, layer mixup). The scatter plot illustrates that mix-based
augmentations produce the best results, with all top-performing results (1st
to 28th) obtained from mix-based augmentations, supporting the conclusions
drawn in Section V-A.

Figure 4: (a): The PTB-XL validation accuracy computed after each training
epoch, with baseline and variations of mixup. (b): The scatter plot of
validation accuracy again F1 score for all 80 profiling experiments on PTB-XL
dataset, different colors annotate different augmentations. (c) and (d): The
confusion matrix of cutmix training, with or without a balanced sampler,
respectively. The y-axis represents the true labels, and the x-axis is the
predicted labels.

The PTB-XL dataset presents an imbalanced class distribution for the single-
label data samples. To mitigate the high false negative rate for the minority
class, a batch class-balanced data sampler was utilized in the data loader
during the training process. The effect of the class-balanced sampler on the
model’s performance is shown in Figure 4 (c) and (d), by comparing the
confusion matrices with and without the balanced sampling. As observed in the
bottom row of Figure 4, which corresponds to the data samples of the minority
class hypertrophy (HYP), the model without the balanced sampler was prone to
producing the most false negative (NORM) predictions for the HYP cases.
However, by incorporating both the class-balanced sampler and cutmix, virtual
samples containing the information of the minority class were generated during
the training, reducing the false negative rate. Additionally, the recall of
the other three cardiac condition classes also received similar improvement as
shown in the plot.

###  V-C Feature representations

The advantages of mix-based augmentations, or vicinal risk minimization,
include the provision of more distinguishable representations for different
classes. We present the results of t-SNE dimensional reduction of two models
trained with cutmix and a baseline, on the training and test sets of the PTB-
XL dataset. The feature vectors were calculated using layer 4 of ResNet18. The
visualizations of the training set (Figure 5(a) and (c)) indicate that,
compared to the baseline, cutmix gives more discriminative representations
between classes. The projections of the test set (Figure 5(b) and (d))
demonstrate that the classes are more distinguishable with mix-based
augmentation and balanced sampling.

Figure 5: The t-SNE visualizations of cutmix and vanilla settings after
training on PTB-XL. In both experiments, a balanced data sampler is used
during training to increase the performance of minority classes.

##  VI Conclusion and Future Work

Inspired by the success of mixup in other domains, we investigate mixup and
its variants, cutmix and manifold mixup for physiological for the time series
classification task. This paper empirically shows that mix-based data
augmentation techniques can achieve superior accuracy in comparison to
traditional augmentation methods in the context of time series classification.
In the experiments, the majority of the highest accuracy results were achieved
through the use of cutmix and layer mixup, and these augmentations were found
to deliver robust performance with a steady accuracy gain across various
physiological and biobehavioral datasets. The low dependency on expert
knowledge and parameter tuning, in addition to the quantitative performance
advantages, makes mix-based augmentations more practical and effective in
various applications. These findings highlight the effectiveness of mix-based,
dataset-agnostic augmentations and the importance of appropriately choosing
traditional data transformations, as they can compromise the generalization
performance of the model.

We plan to explore the combination of mix-based augmentation and traditional
time series augmentations, as the effectiveness of well-composited
transformations has been shown in previous studies [11]. Furthermore, we aim
to extend the applicability of mix-based augmentation to the frequency domain,
following the success of such an approach in acoustic data classification
tasks [37].

## References

  * [1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Communications of the ACM_ , vol. 60, no. 6, pp. 84–90, 2017.
  * [2] P. Y. Simard, Y. A. LeCun, J. S. Denker, and B. Victorri, “Transformation invariance in pattern recognition—tangent distance and tangent propagation,” in _Neural networks: tricks of the trade_. Springer, 1998, pp. 239–274.
  * [3] C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” _Journal of big data_ , vol. 6, no. 1, pp. 1–48, 2019.
  * [4] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” _arXiv preprint arXiv:1710.09412_ , 2017.
  * [5] N. Jaitly and G. E. Hinton, “Vocal tract length perturbation (vtlp) improves speech recognition,” in _Proc. ICML Workshop on Deep Learning for Audio, Speech and Language_ , vol. 117, 2013, p. 21.
  * [6] T. Ko, V. Peddinti, D. Povey, M. L. Seltzer, and S. Khudanpur, “A study on data augmentation of reverberant speech for robust speech recognition,” in _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2017, pp. 5220–5224.
  * [7] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” _arXiv preprint arXiv:1904.08779_ , 2019.
  * [8] D. Ho, E. Liang, X. Chen, I. Stoica, and P. Abbeel, “Population based augmentation: Efficient learning of augmentation policy schedules,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 2731–2741.
  * [9] H. Yang, H. Yu, and A. Sano, “Empirical evaluation of data augmentations for biobehavioral time series data with deep learning,” 2022. [Online]. Available: https://arxiv.org/abs/2210.06701
  * [10] Q. Wen, L. Sun, F. Yang, X. Song, J. Gao, X. Wang, and H. Xu, “Time series data augmentation for deep learning: A survey,” _arXiv preprint arXiv:2002.12478_ , 2020.
  * [11] B. K. Iwana and S. Uchida, “An empirical survey of data augmentation for time series classification with neural networks,” _Plos one_ , vol. 16, no. 7, p. e0254841, 2021.
  * [12] V. Verma, A. Lamb, C. Beckham, A. Najafi, I. Mitliagkas, D. Lopez-Paz, and Y. Bengio, “Manifold mixup: Better representations by interpolating hidden states,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 6438–6447.
  * [13] L. Carratino, M. Cissé, R. Jenatton, and J.-P. Vert, “On mixup regularization,” _arXiv preprint arXiv:2006.06049_ , 2020.
  * [14] L. Meng, J. Xu, X. Tan, J. Wang, T. Qin, and B. Xu, “Mixspeech: Data augmentation for low-resource automatic speech recognition,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 7008–7012.
  * [15] N. Jia and C. Zheng, “Emotion speech synthesis method based on multi-channel time–frequency domain generative adversarial networks (mc-tfd gans) and mixup,” _Arabian Journal for Science and Engineering_ , vol. 47, no. 2, pp. 1749–1762, 2022.
  * [16] K. Xu, D. Feng, H. Mi, B. Zhu, D. Wang, L. Zhang, H. Cai, and S. Liu, “Mixup-based acoustic scene classification using multi-channel convolutional neural network,” in _Advances in Multimedia Information Processing–PCM 2018: 19th Pacific-Rim Conference on Multimedia, Hefei, China, September 21-22, 2018, Proceedings, Part III 19_. Springer, 2018, pp. 14–23.
  * [17] L. Antoni, E. Bruoth, P. Bugata, P. Bugata Jr, D. Gajdoš, Š. Horvát, D. Hudák, V. Kmečová, R. Staňa, M. Staňková _et al._ , “Automatic ecg classification and label quality in training data,” _Physiological Measurement_ , vol. 43, no. 6, p. 064008, 2022.
  * [18] H. Han, S. Park, S. Min, H.-S. Choi, E. Kim, H. Kim, S. Park, J. Kim, J. Park, J. An _et al._ , “Towards high generalization performance on electrocardiogram classification,” in _2021 Computing in Cardiology (CinC)_ , vol. 48. IEEE, 2021, pp. 1–4.
  * [19] H. Han, S. Park, S. Min, E. Kim, H. Kim, S. Park, J. Kim, J. Park, J. An, K. Lee _et al._ , “Improving generalization performance of electrocardiogram classification models,” _Physiological Measurement_ , 2023\.
  * [20] T. Wen and R. Keyes, “Time series anomaly detection using convolutional neural networks and transfer learning,” _arXiv preprint arXiv:1905.13628_ , 2019\.
  * [21] Z. Cui, W. Chen, and Y. Chen, “Multi-scale convolutional neural networks for time series classification,” _arXiv preprint arXiv:1603.06995_ , 2016.
  * [22] T. DeVries and G. W. Taylor, “Dataset augmentation in feature space,” _arXiv preprint arXiv:1702.05538_ , 2017.
  * [23] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, “Cutmix: Regularization strategy to train strong classifiers with localizable features,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 6023–6032.
  * [24] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel, “Mixmatch: A holistic approach to semi-supervised learning,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [25] S. Venkataramanan, E. Kijak, L. Amsaleg, and Y. Avrithis, “Alignmixup: Improving representations by interpolating aligned features,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 19 174–19 183.
  * [26] H. Yang and T. Desell, “Robust augmentation for multivariate time series classification,” _arXiv preprint arXiv:2201.11739_ , 2022.
  * [27] A. Bagnall, H. A. Dau, J. Lines, M. Flynn, J. Large, A. Bostrom, P. Southam, and E. Keogh, “The uea multivariate time series classification archive, 2018,” _arXiv preprint arXiv:1811.00075_ , 2018.
  * [28] H. Ismail Fawaz, B. Lucas, G. Forestier, C. Pelletier, D. F. Schmidt, J. Weber, G. I. Webb, L. Idoumghar, P.-A. Muller, and F. Petitjean, “Inceptiontime: Finding alexnet for time series classification,” _Data Mining and Knowledge Discovery_ , vol. 34, no. 6, pp. 1936–1962, 2020.
  * [29] P. Wagner, N. Strodthoff, R.-D. Bousseljot, D. Kreiseler, F. I. Lunze, W. Samek, and T. Schaeffter, “Ptb-xl, a large publicly available electrocardiography dataset,” _Scientific data_ , vol. 7, no. 1, pp. 1–15, 2020.
  * [30] T. Penzel, G. B. Moody, R. G. Mark, A. L. Goldberger, and J. H. Peter, “The apnea-ecg database,” in _Computers in Cardiology 2000. Vol. 27 (Cat. 00CH37163)_. IEEE, 2000, pp. 255–258.
  * [31] B. Kemp, A. Zwinderman, B. Tuk, H. Kamphuisen, and J. Oberyé, “Sleep-edf database expanded,” _physionet. org_ , 2018.
  * [32] G. B. Moody, R. G. Mark, and A. L. Goldberger, “Physionet: a web-based resource for the study of physiologic signals,” _IEEE Engineering in Medicine and Biology Magazine_ , vol. 20, no. 3, pp. 70–75, 2001.
  * [33] G. Schalk, D. J. McFarland, T. Hinterberger, N. Birbaumer, and J. R. Wolpaw, “Bci2000: a general-purpose brain-computer interface (bci) system,” _IEEE Transactions on biomedical engineering_ , vol. 51, no. 6, pp. 1034–1043, 2004.
  * [34] A. Reiss and D. Stricker, “Introducing a new benchmarked dataset for activity monitoring,” in _2012 16th international symposium on wearable computers_. IEEE, 2012, pp. 108–109.
  * [35] J.-L. Reyes-Ortiz, L. Oneto, A. Samà, X. Parra, and D. Anguita, “Transition-aware human activity recognition using smartphones,” _Neurocomputing_ , vol. 171, pp. 754–767, 2016.
  * [36] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [37] G. Kim, D. K. Han, and H. Ko, “Specmix: A mixed sample data augmentation method for training withtime-frequency domain features,” _arXiv preprint arXiv:2108.03020_ , 2021.


# Reimagining Reality: A Comprehensive Survey of Video Inpainting Techniques

Shreyank N Gowda  Yash Thakre  Shashank Narayana Gowda  Xiaobo Jin

###### Abstract

This paper offers a comprehensive analysis of recent advancements in video
inpainting techniques, a critical subset of computer vision and artificial
intelligence. As a process that restores or fills in missing or corrupted
portions of video sequences with plausible content, video inpainting has
evolved significantly with the advent of deep learning methodologies. Despite
the plethora of existing methods and their swift development, the landscape
remains complex, posing challenges to both novices and established
researchers. Our study deconstructs major techniques, their underpinning
theories, and their effective applications. Moreover, we conduct an exhaustive
comparative study, centering on two often-overlooked dimensions: visual
quality and computational efficiency. We adopt a human-centric approach to
assess visual quality, enlisting a panel of annotators to evaluate the output
of different video inpainting techniques. This provides a nuanced qualitative
understanding that complements traditional quantitative metrics. Concurrently,
we delve into the computational aspects, comparing inference times and memory
demands across a standardized hardware setup. This analysis underscores the
balance between quality and efficiency—a critical consideration for practical
applications where resources may be constrained. By integrating human
validation and computational resource comparison, this survey not only
clarifies the present landscape of video inpainting techniques but also charts
a course for future explorations in this vibrant and evolving field.

###### keywords:

Video Inpainting , Deep Learning , Survey

††journal: Neurocomputing

[inst1]organization=Vidiosyncracy,country=India [inst2]organization=Department
of Intelligent Science, School of Advanced Technology, Xi’an Jiaotong-
Liverpool University,country=China

##  1 Introduction

In the realm of video understanding, technologies like action recognition [1,
2, 3] and video object segmentation [4, 5, 6] have significantly advanced,
enhancing our ability to analyze and interpret dynamic scenes. Building upon
these developments, video inpainting has emerged as a captivating research
field at the intersection of computer vision, image processing, and machine
learning. The exponential growth of multimedia content, coupled with the
increasing demand for seamless video editing, has underscored the need for
automatic methods that can effectively fill in missing or corrupted regions in
videos. Video inpainting techniques aim to address this challenge by
reconstructing these regions in a visually plausible manner, enabling a wide
range of applications such as video restoration [7, 8], object removal [9, 10,
11], and content creation [12, 13, 14].

The fundamental objective of video inpainting is to infer missing or damaged
information based on the available surrounding content, often utilizing
temporal and spatial cues within the video sequence. This process involves
sophisticated algorithms that analyze the context of the video frames and
synthesize new content to seamlessly blend with the existing visual
information. By leveraging the inherent structure and patterns within the
video frames, inpainting algorithms strive to create visually coherent and
continuous video streams where the inpainted regions are virtually
indistinguishable from the original content.

This survey paper aims to provide a comprehensive overview of the state-of-
the-art video inpainting techniques developed to date. By exploring the
advancements in this field, we seek to shed light on the diverse
methodologies, innovations, and challenges encountered by researchers. We
categorize video inpainting methods based on their underlying principles and
highlight their key contributions, enabling readers to grasp the evolution of
techniques. To the best of our knowledge, an existing survey for video
inpainting does not exist and hence believe this contribution is vital to the
field.

We categorize these methods into distinct groups, each characterized by its
unique set of approaches and underlying principles. Firstly, we discuss Patch-
Based Methods, which primarily involve substituting missing areas in a video
frame with patches extracted from other parts of the same video. This category
includes Exemplar-Based Inpainting [15, 16, 17, 18], leveraging best matching
patches for filling holes, and Spatio-Temporal Patch Matching [19, 20, 21],
which extends this concept by considering both spatial and temporal elements
in patch selection.

Next, we explore Motion-Based Methods. These techniques utilize the motion
information inherent in video sequences to guide the inpainting process.
Notable approaches within this category include Optical Flow Estimation [22,
23, 24] and Motion Compensation [25, 26, 10], both striving to maintain
temporal coherence in dynamic scenes.

Finally, we delve into the emerging category of Diffusion-Based Methods for
Video Inpainting. These methods represent a novel approach, simulating the
diffusion of pixel information from adjacent areas to seamlessly restore
damaged or missing regions. This section also touches upon hybrid techniques
that combine diffusion-based strategies with other methods for enhanced
effectiveness [27, 28].

By categorizing these methods in this manner, we not only simplify the
complexity inherent in video inpainting but also shed light on the research
trajectory and potential advancements within each category. This
classification allows for a clearer understanding of the field’s current state
and the various approaches being explored to address the challenges in video
inpainting.

Throughout this survey, we discuss the influence of various video
characteristics, including scene complexity, motion dynamics, and temporal
coherence, on the performance of video inpainting methods. Additionally, we
address the challenges and open problems that need to be tackled in the field.
These challenges encompass improving the handling of large and irregular
occlusions, preserving fine details and textures, adapting to dynamic scenes
with camera motions, and effectively handling diverse video content. By
identifying and understanding these challenges, we hope to inspire researchers
to develop novel solutions and push the boundaries of video inpainting. In
summary, this survey aims to provide researchers and practitioners with a
comprehensive understanding of video inpainting techniques, their
advancements, and the challenges that lie ahead. By consolidating the
knowledge accumulated in this rapidly evolving field, we hope to stimulate
further research and innovation.

In addition to reviewing existing literature, we contribute to the field by
conducting a practical evaluation of video inpainting methods. We reimplement
five prominent papers from different categories of techniques discussed in
this survey. These papers are selected based on their impact, representation
of different approaches, and availability of code and models. We carefully
compare these methods on a selected number of test samples, taking into
consideration both the inference speeds and the quality of inpainted outputs.
To ensure a comprehensive evaluation, we employ human annotators to validate
the quality of the inpainted videos. Human feedback plays a critical role in
assessing the perceptual quality and determining the extent to which the
inpainted regions blend seamlessly with the surrounding content. By
incorporating these human-annotated scores, we aim to provide a more holistic
assessment of the inpainting methods, capturing both objective and subjective
aspects of the results.

##  2 Fundamentals of Video Inpainting

Video inpainting is a process used in the field of computer vision and
graphics to modify a video by filling in missing or undesired parts of the
visual data. The term ”inpainting” originally comes from the art restoration
field, where it refers to the process of reconstructing damaged parts of
paintings or photographs. In the context of video, inpainting serves several
purposes:

  * 1.

Restoration: This is similar to the original meaning in art restoration. In
video, inpainting can be used to restore old or damaged footage, removing
artifacts such as scratches, dust, or other types of noise that degrade the
quality.

  * 2.

Object Removal: One of the most common uses of video inpainting is to remove
unwanted objects or features from a video. This could be anything from an
accidental passerby in a shot, to a piece of modern infrastructure in a
historical movie scene, or even watermarks.

  * 3.

Special Effects: In filmmaking and video production, inpainting can be used to
achieve specific visual effects. For instance, it can be used to create the
illusion of magic tricks, disappearing objects, or to manipulate the video in
ways that would be difficult or impossible to achieve during filming.

  * 4.

Editing: Video inpainting can also be used in post-production editing to alter
the content of a scene without having to reshoot it. This can be more cost-
effective and time-efficient.

  * 5.

Privacy and Security: Inpainting can be used to obscure faces, license plates,
or other sensitive information in videos to protect privacy or for security
reasons.

The challenge in video inpainting, as opposed to still image inpainting, is
maintaining consistency across frames. The inpainted regions must match the
surrounding area in terms of texture, color, and lighting, and must also look
natural as the scene moves and changes over time. Advances in AI and machine
learning have significantly improved the capabilities and results of video
inpainting in recent years.

Video inpainting involves complex techniques that combine the principles of
computer vision, image processing, and machine learning. The underlying goal
is to create a visually plausible and consistent output that fills in missing
or unwanted parts of a video. One of the earlier techniques used for video
inpainting is patch-based inpainting [29, 16], which involves copying and
pasting patches (small blocks of pixels) from other parts of the video to fill
in the missing or undesired areas. The algorithm searches for the most similar
patches in terms of texture and color to ensure a visually coherent result,
although this technique is more effective for static backgrounds and can
struggle with complex, dynamic scenes.

Maintaining temporal consistency is crucial in video inpainting [30, 31]. The
inpainted areas must not only blend seamlessly with the spatial surroundings
in a single frame but also across the temporal dimension. Techniques used for
maintaining temporal consistency often involve tracking the movement and
changes in the scene over time to ensure that the inpainted regions behave in
a physically plausible manner. With advancements in AI, deep learning models,
particularly Generative Adversarial Networks (GANs) [32], have been applied to
video inpainting. These models are trained on large datasets and can generate
new content that fills in the missing parts of a video, and are particularly
effective in handling complex scenes and dynamic objects, offering more
realistic and coherent results.

Optical flow [33] algorithms are used to estimate the motion between video
frames. This information is crucial for ensuring that the inpainted content
aligns [20, 34, 35] correctly with the movement in the video, maintaining the
illusion of a continuous, unedited scene. Exemplar-based [15, 16] inpainting
extends the idea of patch-based inpainting by considering the entire structure
and texture of the area surrounding the hole or undesired object. The
algorithm finds similar regions in the video and uses them as references to
fill in the missing parts, which helps in maintaining both spatial and
temporal coherence. In some cases, especially in professional video editing
and restoration, inpainting involves a level of manual input or guidance.
Users may define the areas to be inpainted, set motion paths, or make
adjustments to ensure the desired outcome.

Advanced inpainting systems incorporate elements of scene understanding [36,
37], where the algorithm interprets the context of the scene to make more
informed decisions about how to fill in gaps. This can include predicting the
trajectory of moving objects or understanding the overall geometry and depth
of the scene. Each of these techniques has its strengths and is chosen based
on the specific requirements of the video inpainting task, such as the
complexity of the scene, the nature of the missing or unwanted content, and
the desired level of automation and control. The field is continuously
evolving, with ongoing research aiming to improve the realism, efficiency, and
versatility of video inpainting methods.

##  3 Taxonomy of Video Inpainting Techniques

Video inpainting is a dynamic field that encompasses various techniques
designed to reconstruct missing or corrupted parts in video frames. This
section elaborates on a more comprehensive taxonomy of these techniques.

###  3.1 Patch-Based Methods

Patch-based methods involve replacing missing regions in a video frame with
patches from other parts of the video.

####  3.1.1 Key Ideas

These methods rely on the assumption that similar patterns or textures exist
within different parts of the video, which can be used to fill in missing
regions.

####  3.1.2 Algorithms and Approaches

  * 1.

Exemplar-Based Inpainting: These methods involve filling the holes by copying
the best matching patches from known regions, often using a priority scheme.
Shih et al. [15] improves video inpainting by extending an image inpainting
algorithm with enhanced patch matching and robust tracking, thus reducing
”ghost shadows” and ensuring temporal continuity in different motion segments.
Koochari et al [16] use large patches to effectively fill missing parts in
video frames, separating moving objects from a stationary background and
creating a mosaic for better representation of occluded objects. Lee et al.
[17] introduce a deep neural network-based ”Copy-and-Paste Networks” for video
inpainting, using content from reference frames to effectively fill missing
regions while maintaining temporal coherency.

  * 2.

Spatio-Temporal Patch Matching: These extend exemplar-based inpainting by
considering spatial and temporal dimensions for patch matching. Newson et al.
[38] enhance PatchMatch [39] for spatio-temporal use, significantly speeding
up high-resolution video inpainting while addressing over-smoothing issues.
Kim et al. [40] use flow sub-networks and mask sub-networks to model
PatchMatch. Le et al. [20] utilize motion matching ensuring that the inpainted
areas align seamlessly with the surrounding video content’s motion patterns.
Meanwhile, Liu et al. [41] propose Temporal Adaptive Alignment Network that
aligns and adapts temporal features using patch matching. More recently, Cai
et al. [21] utilize deformed vision transformers to effectively reconstruct
and inpaint missing or corrupted areas in video content by using Deformed
Patch-based Homography (DePtH), which improves patch-level feature alignments.

###  3.2 Motion-Based Methods

Motion-based methods utilize the movement information in videos to achieve
inpainting, particularly effective in scenes with consistent motion.

####  3.2.1 Key Ideas

These techniques use motion estimation to guide the inpainting process,
ensuring temporal coherence and consistency in dynamic scenes.

####  3.2.2 Algorithms and Approaches

  * 1.

Optical Flow Estimation: Zhang et al. [35] propose repairing or reconstructing
missing or damaged areas in video sequences using internal learning
mechanisms, leveraging the video’s own data and optical flow for inpainting.
Work by Ding et al. [22] incorporates ConvLSTM and optical flow for efficient,
real-time processing of large and arbitrary length videos. Wu et al. [23]
develop an approach for predicting future video frames based on a sequence of
past continuous video frames using optical flow. More recently, Li et al. [42]
use a comprehensive framework for video inpainting, utilizing three key
modules: flow completion, feature propagation, and content hallucination, to
facilitate an efficient and effective inpainting process. Kang et al. [43]
propose a framework that enhances flow-based video inpainting methods by
introducing a newly designed flow completion module and an error compensation
network that leverages an error guidance map, aiming to offset the weaknesses
of traditional flow-based methods. Zhang et al. [44, 45] enhance video
inpainting using optical flow guidance in a transformer-based framework. Gao
et al. [24] propose video completion that combines optical flow and edge
information.

  * 2.

Motion Compensation: These methods involve aligning frames based on estimated
motion to maintain consistency. Kim et al. [46] propose an approach extending
deep learning-based image inpainting methods to the video domain which
involves an additional time dimension. Ebdelli et al. [25] present an
algorithm for video inpainting that estimates unknown pixels as a linear
combination of the closest patches using motion-compensated neighbor
embedding. March et al. [26] delve into the mathematical underpinnings of
motion compensated inpainting in video processing, focusing on the application
of Euler equations and analysis of minimizers in this context. Zhang et al.
[47] introduce PFTA-Net, a novel network that incorporates a progressive
feature alignment module with sub-alignments and a progressive refinement
scheme for more accurate motion compensation in video inpainting. Liu et al.
propose Fuseformer [10], a transformer-based architecture for video inpainting
that enhances the quality of reconstructed video by integrating fine-grained
details across frames.

###  3.3 Diffusion-Based Methods for Video Inpainting

Diffusion-based methods represent a novel approach in the field of video
inpainting, utilizing advanced techniques to enhance the restoration of
missing or corrupted parts in video sequences.

####  3.3.1 Key Ideas

Diffusion-based video inpainting leverages the concept of iteratively refining
the inpainted area by simulating the diffusion of pixel information from the
surrounding areas. This process gradually introduces information into the
damaged or missing regions, ensuring a natural and seamless restoration.

####  3.3.2 Algorithms and Approaches

  * 1.

Extending Image-Based Diffusion to Video: Recent advancements involve deep
learning models that simulate diffusion processes. These models are trained to
predict the natural flow of pixel information, making them highly effective
for complex video sequences with varying textures and motion. The ”Pix2video”
work completed by Ceylan et al. [48] introduces a training-free, text-guided
video editing method using image diffusion models that edit an anchor frame
and then propagate changes to subsequent frames. Hoppe et al. [49] introduce
the Random-Mask Video Diffusion (RaMViD) method, which extends image diffusion
models to video using 3D convolutions and a new conditioning technique,
enabling video prediction, infilling, and upsampling. Cherel et al. [50]
present a technique using internal diffusion processes for video inpainting,
focusing on improving the quality and consistency of inpainted video content.
Voleti et al [9] introduce a versatile video diffusion model that excels in
video prediction, generation, and interpolation by leveraging a masked
conditional approach.

  * 2.

Hybrid Approaches: Combining diffusion-based techniques with other methods,
such as optical flow or patch-based inpainting, allows for more robust
solutions that can handle a variety of inpainting challenges in videos. Gu et
al. [27] introduce the Flow-Guided Diffusion model for Video Inpainting
(FGDVI), which enhances temporal consistency and inpainting quality in videos
by reusing an existing image generation diffusion model. Wang et al. [28]
present a method that combines local and nonlocal flow guidance to improve the
quality and consistency of video inpainting.

Whilst a lot of methods can be classified broadly within these groups, there
are approaches which would fit loosely and hence we believe such a style of
classification of methods would be easier to summarize and understand. This
also gives us an idea on the direction of research among these methods and the
potential to improve them.

##  4 Challenges in Video Inpainting

We identify and discuss key factors such as scene complexity, motion dynamics,
and temporal coherence, which significantly impact the performance and
applicability of inpainting techniques.

###  4.1 Scene Complexity

Scene complexity in videos, characterized by varying textures, colors, and
patterns, poses a substantial challenge for inpainting algorithms. The ability
to accurately reconstruct complex scenes while maintaining visual realism is
critical. As outlined in multiple previous works [36, 38, 19], the complexity
of a scene directly influences the choice and performance of inpainting
methods, necessitating more advanced techniques for detailed and intricate
scenes.

###  4.2 Motion Dynamics and Temporal Coherence

The dynamics of motion within a video significantly affect the inpainting
process. Effective inpainting must account for and replicate the natural
motion patterns to ensure seamless integration of the inpainted region into
the video. This aspect is particularly challenging in videos with rapid or
unpredictable motion. We already speak about methods with a specific focus on
motion compensation and optical flow prediction. This emphasizes the
importance of incorporating motion understanding in video inpainting,
especially in high-motion scenes. Maintaining temporal coherence, the
consistency of visual elements over time, is essential for the believability
and quality of inpainted videos. Temporal incoherencies can lead to noticeable
artifacts in video inpainting, thus undermining the overall quality. Advanced
methods strive to ensure that inpainted regions remain consistent and coherent
throughout the video duration.

###  4.3 Challenges and Open Problems

We also address several pressing challenges and open problems in the field of
video inpainting:

  * 1.

Handling Large and Irregular Occlusions: Dealing with large and irregularly
shaped occlusions [51, 30, 52] remains a significant challenge, requiring
innovative approaches to ensure effective and realistic inpainting.

  * 2.

Preserving Fine Details and Textures: The preservation of fine details and
textures, crucial for realism, is a complex task, especially in highly
textured areas. Methods for enhancing texture fidelity in video inpainting has
been a strong research area [53, 35, 54].

  * 3.

Adapting to Dynamic Scenes with Camera Motions: Dynamic scenes, especially
those with camera motions, pose unique challenges. As explored previously [31,
55, 20], adapting inpainting methods to accommodate camera movements is
essential for realistic reconstructions.

  * 4.

Effectively Handling Diverse Video Content: The diversity of video content,
ranging from simple to complex scenes, requires versatile and robust
inpainting solutions.

By consolidating the knowledge accumulated in this rapidly evolving field, we
hope to stimulate further research and innovation. Understanding these
challenges and the influence of various video characteristics is crucial for
the development of more effective and versatile video inpainting methods.

##  5 Evaluation Metrics

In the realm of video inpainting, a key challenge is to evaluate the
effectiveness of various algorithms in reconstructing video sequences. The
quality of video inpainting is not just about replacing missing or damaged
areas; it’s about doing so in a way that the viewer perceives the video as
uninterrupted and authentic. To quantify this, several evaluation metrics are
employed, each focusing on different aspects of video quality and perceptual
integrity.

Peak Signal-to-Noise Ratio (PSNR): PSNR is a commonly used metric for
measuring the quality of reconstruction in video and image processing. It
calculates the ratio between the maximum possible power of a signal and the
power of distorting noise that affects the fidelity of its representation. In
video inpainting, PSNR is used to compare the inpainted video against a ground
truth or original video to assess the quality of reconstruction.

The formula for PSNR is given by:

$PSNR=10\cdot\log_{10}\left(\frac{{\text{MAX}^{2}}}{{\text{MSE}}}\right)$

where MAX is the maximum possible pixel value, and MSE is the Mean Squared
Error between the original and inpainted videos.

Structural Similarity Index (SSIM): SSIM is a perception-based metric that
measures the similarity between two images or videos. It considers changes in
texture, luminance, and contrast, providing a more comprehensive assessment
than PSNR. In video inpainting, SSIM is utilized to evaluate how well the
inpainted areas blend with the surrounding context in terms of perceived
changes in structural information.

The formula for SSIM is a combination of luminance, contrast, and structure
terms:

$SSIM(x,y)=\frac{{(2\mu_{x}\mu_{y}+C_{1})(2\sigma_{xy}+C_{2})}}{{(\mu_{x}^{2}+\mu_{y}^{2}+C_{1})(\sigma_{x}^{2}+\sigma_{y}^{2}+C_{2})}}$

where $\mu_{x}$, $\mu_{y}$ are the local means, $\sigma_{x}$, $\sigma_{y}$ are
the local standard deviations, $\sigma_{xy}$ is the local cross-covariance,
and $C_{1}$ and $C_{2}$ are constants to stabilize the division.

Frechet Inception Distance (FID): FID is a metric that compares the
distribution of generated images to real images in an embedding space. It
evaluates both the visual quality of individual frames (e.g., clarity, color
accuracy) and the temporal consistency across frames in the video. This is
crucial in video inpainting, where maintaining a seamless and consistent flow
from frame to frame is as important as the quality of individual frames.

The formula for FID is given by:

$FID=\|\mu_{\text{real}}-\mu_{\text{generated}}\|^{2}+\text{Tr}(\Sigma_{\text{real}}+\Sigma_{\text{generated}}-2(\Sigma_{\text{real}}\Sigma_{\text{generated}})^{1/2})$

where $\mu_{\text{real}}$ and $\Sigma_{\text{real}}$ are the mean and
covariance matrix of feature vectors from the real video, and
$\mu_{\text{generated}}$ and $\Sigma_{\text{generated}}$ are the mean and
covariance matrix of feature vectors from the generated video.

Video Frechet Inception Distance (VFID): VFID adapts the concept of FID for
video. It extends the evaluation to assess not only the quality of individual
frames but also the temporal consistency between frames in the generated video
compared to the real video.

The formula for VFID is an extension of FID, considering both the frame-wise
features and the temporal dynamics:

$VFID=FID_{\text{frames}}+\alpha\cdot FID_{\text{temporal}}$

where $FID_{\text{frames}}$ is the FID calculated for individual frames,
$FID_{\text{temporal}}$ is the FID calculated for temporal dynamics, and
$\alpha$ is a weighting factor to balance their contributions.

Learned Perceptual Image Patch Similarity (LPIPS): Learned Perceptual Image
Patch Similarity (LPIPS), initially developed for assessing the perceptual
similarity between images, has been adapted and extended for use in evaluating
videos. This metric is particularly valuable in fields like video generation,
video compression, and video editing, where it’s important to assess how
perceptually similar a generated or altered video is to a reference video.
LPIPS measures the distance in VGGNet feature space as a “perceptual loss” for
image regression problems.

In video inpainting evaluation, each metric offers a distinct perspective on
the quality and effectiveness of video restoration. Peak Signal-to-Noise Ratio
(PSNR) is a basic, technical measure that compares videos pixel-by-pixel,
focusing on the accuracy of signal reconstruction but not on human visual
perception. In contrast, the Structural Similarity Index (SSIM) aligns more
closely with human vision by assessing the luminance, contrast, and structural
similarity between the original and inpainted videos, making it better suited
for evaluating the perceptual integrity of inpainted areas. Video Frechet
Inception Distance (VFID) takes a broader approach by evaluating both the
visual quality of individual frames and the temporal consistency across a
video sequence, thus ensuring a seamless flow in the inpainted video. Lastly,
Learned Perceptual Image Patch Similarity (LPIPS) extends the perceptual
evaluation to a more nuanced level, focusing on the perceptual similarity of
video patches, which is particularly relevant for tasks like video generation
and editing where the subtle nuances of human perception play a significant
role. Each metric, therefore, caters to different aspects of video quality,
from basic signal accuracy to complex perceptual fidelity.

###  5.1 Challenges and Limitations in Evaluating Video Inpainting Techniques

Subjective Nature of Quality Assessment: The assessment of video quality is
often subjective, depending on human perception. Metrics like PSNR and SSIM
may not fully capture the aesthetic and contextual quality perceived by
viewers. This subjectivity can lead to discrepancies between quantitative
metric scores and actual perceived quality.

Complexity of Dynamic Content: Video inpainting involves dealing with dynamic
content and complex backgrounds, which are difficult to quantify with standard
metrics. Metrics may not adequately account for the challenges in handling
diverse and complex video content, leading to an incomplete evaluation.

Over-Reliance on Quantitative Metrics: There is often an over-reliance on
quantitative metrics like PSNR and SSIM, which might not fully encompass the
intricacies of video inpainting. This reliance can overshadow other important
aspects such as user experience, realism, and context-awareness.

We show these limitations with varying user scores for different models in
Section 7 and talk about how subjective video inpainting is and the need for a
different metric to properly evaluate models.

##  6 Datasets

Dataset Name | Focus | Content Description | Resolution
---|---|---|---
DAVIS | VOS | Diverse objects | Full HD
YouTube-VOS | VOS | Diverse objects | Various
DEVIL | Inpainting | Camera Motion | Various
Table 1: Summary of Datasets for Video Inpainting Research

YoutubeVOS [56] and DAVIS [57] are two extensively utilized datasets in the
field of video inpainting, each originally designed for specific tasks
unrelated to video inpainting. YoutubeVOS, primarily developed for Video
Object Segmentation, offers a diverse range of videos featuring various
objects and scenarios, making it a valuable resource for understanding object
motion and consistency in videos. DAVIS, on the other hand, was designed for
video object segmentation and tracking, providing high-quality, full-
resolution video sequences with annotated objects, which are crucial for
training models to understand object movement and boundary information.

Despite their initial purposes, both datasets have been adapted for use in
video inpainting tasks. This adaptation is primarily due to their
comprehensive and varied content, which helps in training inpainting models to
handle different scenarios and object movements. However, the lack of datasets
specifically designed for video inpainting led to the introduction of the
DEVIL [58] dataset. This dataset is tailored specifically for video
inpainting, encompassing a wider range of challenges specifically encountered
in this task, such as dealing with different types of occlusions and complex
object interactions.

The absence of a strong, dedicated benchmark in video inpainting has
historically hindered the field’s progress. This is because generic datasets
like YoutubeVOS and DAVIS, while useful, do not fully address the unique
challenges of video inpainting, such as temporal consistency and complex
dynamic backgrounds. The creation of DEVIL marks a significant step towards
overcoming these challenges, providing a more focused and relevant benchmark
for advancing the state of the art in video inpainting. These datasets are
summarized in Table 1.

##  7 Quality Evaluation and Comparison

We run all experiments using a Tesla T4 GPU. We evaluate the following models:
Flow-Guided Transformer for Video Inpainting (FGT) [45], Flow-edge Guided
Video Completion (FGVC) [24], End-to-End Framework for Flow-Guided Video
Inpainting (E2FGVI) [42], FuseFormer [10] and Copy-Paste networks [17]. These
methods range from 2019 to more recent methods and showcase how the outputs of
these models have evolved over time. First, we compare quantitatively these
methods in Table 2. In particular, we look at the inference details of these
methods with a view to them being deployed. From these selected models, E2FGVI
seems the most realistically deployable model accounting for speed and memory
usage. We also report average scores by asking 20 human annotators to give a
score on 1 to 10 for each model on a set of videos. The average score is
reported under User Rating in the Table.

Model | Input | Time | Memory | User Rating
---|---|---|---|---
FGT [45] [ECCV’22] | 80f | 241s | 2.6 GB | 7.2
FGVC [24] [ECCV’20] | 15f | 50s | 1.4GB | 6.5
E2FGVI [42] [CVPR’22] | 70f | 18s | 8.0GB | 6.8
FuseFormer [10] [ICCV’21] | 50f | 39s | 10.5GB | 6.9
CopyPaste [17] [ICCV’19] | 70f | 44s | 2.6GB | 6.7
Table 2: Comparing inference hardware requirements of selected models and the
number of frames needed as input to the model.

From a qualitative perspective, we evaluate on samples from both the
evaluation set of datasets the model has been trained on and in-the-wild
videos taken from public sources 111https://mixkit.co/free-stock-video/.
First, we look at a video containing flamingoes and methods that remove one of
them. This can be seen in Figure 1.

Figure 1: Comparison of different video inpainting methods on a flamingo
video. Different frames are taken from the output and shown along with the
original frame (ground truth).

Next, we use a video of people hiking and plot the frame wise results of
removal in Figure 2.

Figure 2: Comparison of different video inpainting methods on a hiking video.
Different frames are taken from the output and shown along with the original
frame (ground truth).

Next, we use a video of person playing tennis and plot the frame wise results
of removal in Figure 3.

Figure 3: Comparison of different video inpainting methods on a tennis video.
Different frames are taken from the output and shown along with the original
frame (ground truth).

##  8 Applications and Future Directions

###  8.1 Practical Applications of Video Inpainting

  * 1.

Video Editing: Video inpainting is a pivotal tool in professional video
editing, used for removing unwanted elements like microphones, wires, or
unintentional background objects. This technology has found applications in
various sectors including film production, news broadcasting, and content
creation for digital platforms. It enhances the visual appeal and
professionalism of videos by enabling editors to maintain continuity and
visual coherence in their narratives.

  * 2.

Restoration: Inpainting plays a critical role in the restoration of old or
damaged footage. This includes repairing scratches, dust, and other defects in
archival footage. Notable projects in film restoration have demonstrated how
inpainting helps preserve historical and cultural content, providing clearer
and more accessible versions of historically significant films and
documentaries.

  * 3.

Visual Effects: In the realm of visual effects, inpainting serves as a tool
for creating seamless composites and backgrounds. It facilitates the blending
of various elements in a scene, aiding in storytelling and the creation of
fantastical environments. The use of inpainting in major film productions for
generating realistic and visually stunning effects has become increasingly
prevalent.

###  8.2 Current Trends and Emerging Research Directions

  * 1.

Diffusion Models: Recent trends [48, 9] show the integration of diffusion
models in video inpainting. These models, based on advanced machine learning
algorithms, predict and recreate complex scenes with remarkable accuracy,
surpassing traditional techniques in both quality and efficiency.

  * 2.

Real-Time Inpainting: The pursuit of real-time video inpainting [59, 60, 61]
aims to enable live modifications of video streams. This advancement is
particularly significant for live broadcasting and virtual reality
applications, where the ability to alter or enhance video content
instantaneously is invaluable.

  * 3.

Increasing Automation: The trend [48, 11] towards more automated inpainting
processes is making this technology increasingly accessible. Automation
reduces the need for manual intervention, thereby streamlining the video
editing process and making it more efficient for users with varying levels of
expertise.

###  8.3 Open Challenges and Potential Future Developments

  * 1.

Handling Complex Dynamics: A significant challenge in video inpainting is
dealing with intricate and rapid movements within videos. Future research may
focus on developing more sophisticated algorithms that can understand and
replicate complex dynamics, thus enhancing the realism and fluidity of
inpainted content.

  * 2.

Enhanced Resolution and Quality: As video standards continue to evolve towards
higher resolutions like 4K and 8K, there is a growing need for inpainting
techniques that can maintain detail and clarity at these levels. This
development is crucial for ensuring that inpainting technology remains
relevant and effective in the face of rapidly advancing display technologies.

  * 3.

Interactivity and User Control: Future developments in video inpainting might
include more interactive tools, providing users with greater control over the
inpainting process. This would allow for more personalized and creative
applications, catering to a wider range of use cases and artistic visions.

##  9 Conclusion

In conclusion, this paper has provided a thorough examination of the current
state of video inpainting techniques, a vital niche in the realms of computer
vision and artificial intelligence. Video inpainting, which involves the
restoration or completion of missing or corrupted segments in video sequences,
has significantly advanced through the integration of deep learning
methodologies. Our comprehensive study breaks down key techniques, their
foundational theories, and their real-world applications, illuminating a
landscape that, despite its rapid evolution, remains complex and challenging.
Central to our analysis was an exhaustive comparative study focusing on visual
quality and computational efficiency. By employing a human-centric approach
for assessing visual quality, involving a panel of annotators, we gained
valuable qualitative insights that enhance traditional quantitative
assessments. This human validation aspect brought a unique perspective to our
evaluation of various video inpainting methods. In tandem, we conducted a
rigorous examination of the computational demands of these techniques,
comparing inference times and memory requirements across a uniform hardware
setup. Our findings highlight the crucial balance between quality and
efficiency, especially pertinent in practical scenarios where resource
limitations are a common constraint. This paper not only clarifies the
intricate landscape of video inpainting techniques but also sets a direction
for future research in this dynamic and rapidly evolving field. By merging
human-centric evaluations with computational resource analysis, our study
contributes to a deeper understanding of video inpainting and lays the
groundwork for future innovations that are both high in quality and efficient
in execution.

##  10 Acknowledgements

This work was partially supported by “Qing Lan Project” in Jiangsu
universities, NSFC under No. U1804159 and RDF with No. RDF-22-01-020.

## References

  * [1] H. Wang, C. Schmid, Action recognition with improved trajectories, in: Proceedings of the IEEE international conference on computer vision, 2013, pp. 3551–3558.
  * [2] Z. Sun, Q. Ke, H. Rahmani, M. Bennamoun, G. Wang, J. Liu, Human action recognition from various data modalities: A review, IEEE transactions on pattern analysis and machine intelligence (2022).
  * [3] S. N. Gowda, M. Rohrbach, L. Sevilla-Lara, Smart frame selection for action recognition, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, 2021, pp. 1451–1459.
  * [4] S. N. Gowda, P. Eustratiadis, T. Hospedales, L. Sevilla-Lara, Alba: Reinforcement learning for video object segmentation, arXiv preprint arXiv:2005.13039 (2020).
  * [5] R. Yao, G. Lin, S. Xia, J. Zhao, Y. Zhou, Video object segmentation and tracking: A survey, ACM Transactions on Intelligent Systems and Technology (TIST) 11 (4) (2020) 1–47.
  * [6] Y.-T. Hu, J.-B. Huang, A. G. Schwing, Videomatch: Matching based video object segmentation, in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 54–70.
  * [7] X. Wang, K. C. Chan, K. Yu, C. Dong, C. Change Loy, Edvr: Video restoration with enhanced deformable convolutional networks, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, 2019, pp. 0–0.
  * [8] S. Lee, D. Cho, J. Kim, T. H. Kim, Restore from restored: Video restoration with pseudo clean video, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 3537–3546.
  * [9] V. Voleti, A. Jolicoeur-Martineau, C. Pal, Mcvd-masked conditional video diffusion for prediction, generation, and interpolation, Advances in Neural Information Processing Systems 35 (2022) 23371–23385.
  * [10] R. Liu, H. Deng, Y. Huang, X. Shi, L. Lu, W. Sun, X. Wang, J. Dai, H. Li, Fuseformer: Fusing fine-grained information in transformers for video inpainting, in: Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 14040–14049.
  * [11] R. Zhang, W. Li, P. Wang, C. Guan, J. Fang, Y. Song, J. Yu, B. Chen, W. Xu, R. Yang, Autoremover: Automatic object removal for autonomous driving videos, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, 2020, pp. 12853–12861.
  * [12] S. Yun, S. J. Oh, B. Heo, D. Han, J. Kim, Videomix: Rethinking data augmentation for video classification, arXiv preprint arXiv:2012.03457 (2020).
  * [13] S. N. Gowda, M. Rohrbach, F. Keller, L. Sevilla-Lara, Learn2augment: learning to composite videos for data augmentation in action recognition, in: European conference on computer vision, Springer, 2022, pp. 242–259.
  * [14] S. N. Gowda, D. Pandey, S. N. Gowda, From pixels to portraits: A comprehensive survey of talking head generation techniques and applications, arXiv preprint arXiv:2308.16041 (2023).
  * [15] T. K. Shih, N. C. Tang, J.-N. Hwang, Exemplar-based video inpainting without ghost shadow artifacts by maintaining temporal continuity, IEEE transactions on circuits and systems for video technology 19 (3) (2009) 347–360.
  * [16] A. Koochari, M. Soryani, Exemplar-based video inpainting with large patches, Journal of Zhejiang University Science C 11 (2010) 270–277.
  * [17] S. Lee, S. W. Oh, D. Won, S. J. Kim, Copy-and-paste networks for deep video inpainting, in: Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 4413–4421.
  * [18] S. N. Gowda, C. Yuan, Colornet: Investigating the importance of color spaces for image classification, in: Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part IV 14, Springer, 2019, pp. 581–596.
  * [19] A. Newson, A. Almansa, M. Fradet, Y. Gousseau, P. Pérez, Video inpainting of complex scenes, Siam journal on imaging sciences 7 (4) (2014) 1993–2019.
  * [20] T. T. Le, A. Almansa, Y. Gousseau, S. Masnou, Motion-consistent video inpainting, in: 2017 IEEE international conference on image processing (ICIP), IEEE, 2017, pp. 2094–2098.
  * [21] J. Cai, C. Li, X. Tao, C. Yuan, Y.-W. Tai, Devit: Deformed vision transformers in video inpainting, in: Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 779–789.
  * [22] Y. Ding, C. Wang, H. Huang, J. Liu, J. Wang, L. Wang, Frame-recurrent video inpainting by robust optical flow inference, arXiv preprint arXiv:1905.02882 (2019).
  * [23] Y. Wu, R. Gao, J. Park, Q. Chen, Future video synthesis with object motion prediction, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 5539–5548.
  * [24] C. Gao, A. Saraf, J.-B. Huang, J. Kopf, Flow-edge guided video completion, in: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16, Springer, 2020, pp. 713–729.
  * [25] M. Ebdelli, C. Guillemot, O. Le Meur, Examplar-based video inpainting with motion-compensated neighbor embedding, in: 2012 19th IEEE International Conference on Image Processing, IEEE, 2012, pp. 1737–1740.
  * [26] R. March, G. Riey, Euler equations and trace properties of minimizers of a functional for motion compensated inpainting, Inverse Problems and Imaging 16 (4) (2022) 703–737.
  * [27] B. Gu, Y. Yu, H. Fan, L. Zhang, Flow-guided diffusion for video inpainting, arXiv preprint arXiv:2311.15368 (2023).
  * [28] J. Wang, Z. Yang, Z. Huo, W. Chen, Local and nonlocal flow-guided video inpainting, Multimedia Tools and Applications (2023) 1–20.
  * [29] A. Criminisi, P. Pérez, K. Toyama, Region filling and object removal by exemplar-based image inpainting, IEEE Transactions on image processing 13 (9) (2004) 1200–1212.
  * [30] K. A. Patwardhan, G. Sapiro, M. Bertalmio, Video inpainting of occluding and occluded objects, in: IEEE International Conference on Image Processing 2005, Vol. 2, IEEE, 2005, pp. II–69.
  * [31] K. A. Patwardhan, G. Sapiro, M. Bertalmío, Video inpainting under constrained camera motion, IEEE Transactions on Image Processing 16 (2) (2007) 545–553.
  * [32] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial networks, Communications of the ACM 63 (11) (2020) 139–144.
  * [33] B. K. Horn, B. G. Schunck, Determining optical flow, Artificial intelligence 17 (1-3) (1981) 185–203.
  * [34] R. Xu, X. Li, B. Zhou, C. C. Loy, Deep flow-guided video inpainting, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 3723–3732.
  * [35] H. Zhang, L. Mai, N. Xu, Z. Wang, J. Collomosse, H. Jin, An internal learning approach to video inpainting, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 2720–2729.
  * [36] D. Lao, P. Zhu, P. Wonka, G. Sundaramoorthi, Flow-guided video inpainting with scene templates, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 14599–14608.
  * [37] Y. Zeng, J. Fu, H. Chao, Learning joint spatial-temporal transformations for video inpainting, in: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16, Springer, 2020, pp. 528–543.
  * [38] A. Newson, A. Almansa, M. Fradet, Y. Gousseau, P. Pérez, Towards fast, generic video inpainting, in: Proceedings of the 10th European Conference on Visual Media Production, 2013, pp. 1–8.
  * [39] C. Barnes, E. Shechtman, A. Finkelstein, D. B. Goldman, Patchmatch: A randomized correspondence algorithm for structural image editing, ACM Trans. Graph. 28 (3) (2009) 24.
  * [40] D. Kim, S. Woo, J.-Y. Lee, I. S. Kweon, Deep video inpainting, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 5792–5801.
  * [41] R. Liu, Z. Weng, Y. Zhu, B. Li, Temporal adaptive alignment network for deep video inpainting, in: Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, 2021, pp. 927–933.
  * [42] Z. Li, C.-Z. Lu, J. Qin, C.-L. Guo, M.-M. Cheng, Towards an end-to-end framework for flow-guided video inpainting, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 17562–17571.
  * [43] J. Kang, S. W. Oh, S. J. Kim, Error compensation framework for flow-guided video inpainting, in: European Conference on Computer Vision, Springer, 2022, pp. 375–390.
  * [44] K. Zhang, J. Peng, J. Fu, D. Liu, Exploiting optical flow guidance for transformer-based video inpainting, arXiv preprint arXiv:2301.10048 (2023).
  * [45] K. Zhang, J. Fu, D. Liu, Flow-guided transformer for video inpainting, in: European Conference on Computer Vision, Springer, 2022, pp. 74–90.
  * [46] D. Kim, S. Woo, J.-Y. Lee, I. S. Kweon, Recurrent temporal aggregation framework for deep video inpainting, IEEE transactions on pattern analysis and machine intelligence 42 (5) (2019) 1038–1052.
  * [47] Y. Zhang, Z. Wu, Y. Yan, Pfta-net: Progressive feature alignment and temporal attention fusion networks for video inpainting, in: 2023 IEEE International Conference on Image Processing (ICIP), IEEE, 2023, pp. 191–195.
  * [48] D. Ceylan, C.-H. P. Huang, N. J. Mitra, Pix2video: Video editing using image diffusion, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 23206–23217.
  * [49] T. Höppe, A. Mehrjou, S. Bauer, D. Nielsen, A. Dittadi, Diffusion models for video prediction and infilling, arXiv preprint arXiv:2206.07696 (2022).
  * [50] N. Cherel, A. Almansa, Y. Gousseau, A. Newson, Infusion: Internal diffusion for video inpainting, arXiv preprint arXiv:2311.01090 (2023).
  * [51] L. Ke, Y.-W. Tai, C.-K. Tang, Occlusion-aware video object inpainting, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 14468–14478.
  * [52] H. Wang, H. Li, B. Li, Video inpainting for largely occluded moving human, in: 2007 IEEE International Conference on Multimedia and Expo, IEEE, 2007, pp. 1719–1722.
  * [53] C. Wang, X. Chen, S. Min, J. Wang, Z.-J. Zha, Structure-guided deep video inpainting, IEEE Transactions on Circuits and Systems for Video Technology 31 (8) (2020) 2953–2965.
  * [54] S. Zhou, C. Li, K. C. Chan, C. C. Loy, Propainter: Improving propagation and transformer for video inpainting, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 10477–10486.
  * [55] M. Granados, K. I. Kim, J. Tompkin, J. Kautz, C. Theobalt, Background inpainting for videos with dynamic objects and a free-moving camera, in: Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part I 12, Springer, 2012, pp. 682–695.
  * [56] N. Xu, L. Yang, Y. Fan, J. Yang, D. Yue, Y. Liang, B. Price, S. Cohen, T. Huang, Youtube-vos: Sequence-to-sequence video object segmentation, in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 585–601.
  * [57] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, A. Sorkine-Hornung, A benchmark dataset and evaluation methodology for video object segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 724–732.
  * [58] R. Szeto, J. J. Corso, The devil is in the details: A diagnostic evaluation benchmark for video inpainting, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 21054–21063.
  * [59] B. J. Matthews, C. Reichherzer, B. H. Thomas, R. T. Smith, Maskwarp: Visuo-haptic illusions in mixed reality using real-time video inpainting, in: 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), IEEE, 2023, pp. 767–768.
  * [60] T. Kim, G. J. Kim, Real-time and on-line removal of moving human figures in hand-held mobile augmented reality, The Visual Computer 39 (7) (2023) 2571–2582.
  * [61] L. Wang, T. Tian, X. Yan, F. Ruan, G. J. Aadityaa, H. Choset, L. Li, Real-time video inpainting for rgb-d pipeline reconstruction, in: 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2023, pp. 9543–9550.