Zihang<EMAIL_ADDRESS>Weidi<EMAIL_ADDRESS>Department
of Computer Science
University of Oxford, UK Visual Geometry Group,
Department of Engineering Science
University of Oxford, UK Self-supervised Learning for Video Correspondence
Flow  capbtabboxtable[][]

# Self-supervised Learning for Video Correspondence Flow

###### Abstract

The objective of this paper is _self-supervised learning_ of feature
embeddings that are suitable for matching correspondences along the videos,
which we term correspondence flow. By leveraging the natural spatial-temporal
coherence in videos, we propose to train a “pointer” that reconstructs a
target frame by copying pixels from a reference frame.

We make the following contributions: _First_ , we introduce a simple
information bottleneck that forces the model to learn robust features for
correspondence matching, and prevent it from learning trivial solutions, _e.g_
matching based on low-level colour information. _Second_ , to alleveate
tracker drifting, due to complex object deformations, illumination changes and
occlusions, we propose to train a recursive model over long temporal windows
with scheduled sampling and cycle consistency. _Third_ , we achieve the state-
of-the-art performance on DAVIS 2017 video segmentation and JHMDB keypoint
tracking tasks, outperforming all previous self-supervised learning approaches
by a significant margin. _Fourth_ , in order to shed light on the potential of
self-supervised learning on the task of video correspondence flow, we probe
the upper bound by training on additional data, _i.e_ more diverse videos,
further demonstrating significant improvements on video segmentation. The
source code will be released at https://github.com/zlai0/CorrFlow.

##  1 Introduction

Correspondence matching is a fundamental building block for numerous
applications ranging from depth estimation [Kendall et al.(2017)Kendall,
Martirosyan, Dasgupta, Henry, Kennedy, Bachrach, and Bry] and optical flow
[Brox et al.(2004)Brox, Bruhn, Papenberg, and Weickert, Dosovitskiy et
al.(2015)Dosovitskiy, Fischer, Ilg, Hausser, Hazirbas, Golkov, Smagt, Cremers,
and Brox, Ilg et al.(2017)Ilg, Mayer, Saikia, Keuper, Dosovitskiy, and Brox],
to segmentation and tracking [Smith and Brady(1995)], and 3D reconstruction
[Hartley and Zisserman(2004)]. However, training models for correspondence
matching is not trivial, as obtaining manual annotations can be prohibitively
expensive, and sometimes is even impossible due to occlusions and complex
object deformations. In the recent works [Rocco et al.(2017)Rocco,
Arandjelovic, and Sivic, Rocco et al.(2018)Rocco, Arandjelovic, and Sivic],
Rocco  _et al_ proposed to circumvent this issue by pre-training Convolutional
Neural Networks (CNNs) for predicting artificial transformations, and further
bootstrap the model by finetuning on a small dataset with human annotations.
Alternatively, the necessity for labbelled data can be avoided by using self-
supervised learning, _i.e_ a form of unsupervised learning, where part of the
data is withheld for defining a proxy task, such that the model will be forced
to learn the semantic representation that we really care about.

Videos have shown to be appealing as a data source for self-supervised
learning due to their almost infinite supply (from YouTube etc), and the
availability of numerous proxy losses that can be employed from the intrinsic
spatio-temporal coherence, _i.e_ the signals in video tend to vary smoothly in
time [Wiskott and Sejnowski(2002), Jayaraman and Grauman(2016)]. In this
paper, we propose to tackle the task of correspondence flow in videos with
self-supervised learning. We are certainly not the first to explore this idea,
in the seminal paper by Vondrick  _et al_ [Vondrick et al.(2018)Vondrick,
Shrivastava, Fathi, Guadarrama, and Murphy], they propose to learn embeddings
for grayscale frames and use _colorization_ as a proxy task for training.
Despite the promising efforts, the model capacity has been significantly
constrained due to the loss of colour information, and suffers the problem of
tracker drifting as only pair of frames are used during training.

|
---|---
(a) DAVIS 2017 Video Segmentation | (b) Keypoint Tracking
Figure 1:  We propose self-supervised learning of correspondence flow on
videos. Without any fine-tuning, the acquired representation generalizes to
various tasks: (a) video segmentaiton; (b) keypoint tracking. Note: For both
tasks, the annotation for the first frame is given, the goal is to propagate
the annotations through the videos.

We make the following contributions: _First_ , we introduce an embarrassingly
simple idea to avoid trivial solutions while learning pixelwise matching by
frame reconstruction. During training, channel-wise dropout and colour
jitterings are added intentionally on the input frames, the model is therefore
forced _not_ to rely on low-level colour information, and must be robust to
colour jittering. _Second_ , we propose to train the model recursively on
videos over long temporal windows with scheduled sampling and forward-backward
consistency. Both ideas have shown to improve the model robustness and help to
alleviate the tracker drifting problem. _Third_ , after self-supervised
training, we benchmark the model on two downstream tasks focusing on pixelwise
tracking, _e.g_ DAVIS 2017 video segmentation and JHMDB keypoint tracking,
outperforming all previous self-supervised learning approaches by a
significant margin. _Fourth_ , to further shed light on the potential of self-
supervised learning for video correspondence flow, we probe the upper bound by
training on more diverse video data, and further demonstrating significant
improvements on video segmentation.

##  2 Related Work

Correspondence Matching. Recently, many researchers have studied the task of
correspondence matching using deep Convolutional Neural Networks [Han et
al.(2017)Han, Rezende, Ham, Wong, Cho, Schmid, and Ponce, Kim et al.(2017)Kim,
Min, Ham, Jeon, Lin, and Sohn, Novotny et al.(2017)Novotny, Larlus, and
Vedaldi, Rocco et al.(2017)Rocco, Arandjelovic, and Sivic, Rocco et
al.(2018)Rocco, Arandjelovic, and Sivic]. The works from Rocco  _et al_ [Rocco
et al.(2017)Rocco, Arandjelovic, and Sivic, Rocco et al.(2018)Rocco,
Arandjelovic, and Sivic], propose to train CNNs by learning the artificial
transformations between pairs of images. For robust estimation, they applied a
differentiable soft inlier score for evaluating the quality of alignment
between spatial features and providing a loss for learning semantic
correspondences. However, their work may not be ideal as the model still
relies on synthetic transformations. In contrast, we address the challenge of
learning correspondence matching by exploiting the temporal coherence in
videos.

Optical Flow. In the conventional variational approaches, optical flow
estimation is treated as an energy minimization problem based on brightness
constancy and spatial smoothness [Horn and Schunck(1981)]. In later works,
feature matching is used to firstly establish sparse matchings, and then
interpolated into dense flow maps in a pyramidal coarse-to-fine manner [Brox
et al.(2009)Brox, Bregler, and Malik, Revaud et al.(2015)Revaud, Weinzaepfel,
Harchaoui, and Schmid, Weinzaepfel et al.(2013)Weinzaepfel, Revaud, Harchaoui,
and Schmid]. Recently, convolutional neural networks (CNNs) have been applied
to improve the matching by learning effective feature embeddings [Bailer et
al.(2017)Bailer, Varanasi, and Stricker, Jia et al.(2017)Jia, Ranftl, and
Koltun]. Another line of more relevant research is unsupervised learning for
optical flow. The basic principles are based on brightness constancy and
spatial smoothness [Yu et al.(2016)Yu, Harley, and Derpanis, Wang et
al.(2018)Wang, Yang, Yang, Zhao, Wang, and Xu, Meister et al.(2018)Meister,
Hur, and Roth]. This leads to the popular photometric loss which measures the
difference between the reference image and the warped image. For occluded
regions, a mask is implicitly estimated by checking forward-backward
consistency.

Self-supervised Learning. Of more direct relevance to our training framework
are self-supervised frameworks that use video data [Agrawal et
al.(2015)Agrawal, Carreira, and Malik, Denton and Birodkar(2017), Fernando et
al.(2017)Fernando, Bilen, Gavves, and Gould, Gan et al.(2018)Gan, Gong, Liu,
Su, and Guibas, Jakab et al.(2018)Jakab, Gupta, Bilen, and Vedaldi, Jia et
al.(2016)Jia, De Brabandere, and Tuytelaars, Lee et al.(2017)Lee, Huang,
Singh, and Yang, Misra et al.(2016)Misra, Zitnick, and Hebert, Wang and
Gupta(2015), Wiles et al.(2018)Wiles, Koepke, and Zisserman, Isola et
al.(2015)Isola, Zoran, Krishnan, and Adelson, Jayaraman and Grauman(2016),
Jayaraman and Grauman(2015), Jing and Tian(2018), Kim et al.(2018)Kim, Cho,
and Kweon, Vondrick et al.(2018)Vondrick, Shrivastava, Fathi, Guadarrama, and
Murphy]. In [Misra et al.(2016)Misra, Zitnick, and Hebert, Fernando et
al.(2017)Fernando, Bilen, Gavves, and Gould, Wei et al.(2018)Wei, Lim,
Zisserman, and Freeman], the proxy task is defined to focus on temporal
sequence ordering of the frames. Another approach is to use the temporal
coherence as a proxy loss [Isola et al.(2015)Isola, Zoran, Krishnan, and
Adelson, Jayaraman and Grauman(2016), Wang and Gupta(2015)]. Other approaches
use egomotion [Agrawal et al.(2015)Agrawal, Carreira, and Malik, Jayaraman and
Grauman(2015)] in order to enforce equivariance in feature space [Jayaraman
and Grauman(2015)]. Recently [Vondrick et al.(2018)Vondrick, Shrivastava,
Fathi, Guadarrama, and Murphy], leveraged the natural temporal coherency of
colour in videos, to train a network for tracking and correspondence related
tasks. Our approach builds in particular on those that use frame synthesis
[Denton and Birodkar(2017), Jia et al.(2016)Jia, De Brabandere, and
Tuytelaars, Wiles et al.(2018)Wiles, Koepke, and Zisserman], though for us
synthesis is a proxy task rather than the end goal.

##  3 Approach

The goal of this work is to train an embedding network with self-supervised
learning that enables pixelwise correspondence matching. Our basic idea is to
exploit spatial-temporal coherence in videos, that is, the frame appearances
will not change abruptly, and colours can act as a reliable supervision signal
for learning correspondences.

###  3.1 Background

In this section, we briefly review the work by Vondrick  _et al_ [Vondrick et
al.(2018)Vondrick, Shrivastava, Fathi, Guadarrama, and Murphy]. Formally, let
$c_{i}\in\mathbb{R}^{d}$ be the true colour for pixel $i$ in the reference
frame, and let $c_{j}\in\mathbb{R}^{d}$ be the true colour for a pixel $j$ in
the target frame. $y_{j}\in\mathbb{R}^{d}$ is the model’s prediction for
$c_{j}$, it is a linear combination of colours in the reference frame:

$\displaystyle y_{j}=\sum_{i}A_{ij}c_{i},\text{\hskip 15.0pt where
}A_{ij}=\frac{exp\langle f_{i}^{T}f_{j}\rangle}{\sum_{k}exp\langle
f_{k}^{T}f_{j}\rangle}$ (1)

$A$ is an affinity matrix computed from simple dot products between the
feature embeddings of the _grayscale_ target and reference frame ($f$’s).

Despite the efforts in circumventing trivial solutions, training models with
_grayscale_ inputs has introduces a train-test discrepancy when deploying to
the downstream task, as the model has never been trained to encode the
correlation of RGB channels. Another issule lies in the fact that their model
is only trained with pairs of ground truth video frames, which inevitably
leads to model drifting when evaluated on video tracking tasks. As the
prediction for later steps rely on the prediction from previous steps, the
errors accumulate, and there is no mechanism for the model to recover from
previous error states.

In the following sections, we propose to train a framework that aims to close
the gap between training and testing as much as possible, _i.e_ the model
should ideally be trained on _full-colour_ and _high-resolution_ over _long_
video sequences.

Figure 2:  An overview of the proposed self-supervised learning for
correspondence flow. A recursive model is used to compute the dense
correspondence matching over a long temporal window with forward-backward
cycle consistency.

###  3.2 Feature Embedding with Information Bottleneck

Given a collection of frames $\\{I_{1},I_{2},...,I_{N}\\}$ from a video clip,
we parametrize the feature embedding module with CNNs:

$\displaystyle f_{i}=\Phi(g(I_{i});\theta)$ (2)

where $\Phi$ refers to a ResNet feature encoder (details in Arxiv
paper111https://arxiv.org/abs/1905.00875), and $g(\cdot)$ denotes an
_information bottleneck_ that prevents from information leakage. In Vondrick
_et al_ [Vondrick et al.(2018)Vondrick, Shrivastava, Fathi, Guadarrama, and
Murphy], this is simply defined as a function that converts RGB frame into
_grayscale_ , therefore, the model is forced _not_ to rely on colours for
matching correspondences.

We propose to randomly zero out 0, 1, or 2 channels in each input frame with
some probability (one possible input is shown in Figure 3 (a)), and perturb
the brightness, contrast and saturation of an image by up to $10\%$. Despite
the simplicity, this design poses two benefits: _First_ , the input jitterings
and stochastic dropout are essentially acting as an _information bottleneck_ ,
which prevents the model from co-adaptation of low-level colours. When
deploying for downstream tasks, images of full RGB colours are taken as input
directly (no jittering). _Second_ , it can also be treated as data
augmentation that potentially improves the model robustness under challenging
cases, for instance, illumination changes.

###  3.3 Restricted Attention

In the previous work [Vondrick et al.(2018)Vondrick, Shrivastava, Fathi,
Guadarrama, and Murphy], _full attention_ has been used for computing the
affinity matrix, _i.e_ all pairs of pixels in target and reference frames are
correlated (Figure 3 (b)). However, the memory and computational consumption
tends to grow quadratically with the spatial footprint of the feature maps,
thus limiting the resolution. In fact, videos are full of regularities, _i.e_
the appearances in the video clip tend to change smoothly both in spatial and
temporal axes. To fully exploit this property, we propose to use a restricted
attention mechanism (Figure 3 (c)), which leads to dramatic decrease in
computation and memory consumption, and enables to train on _high-resolution_
frames.

|  |  |  |  |
---|---|---|---|---|---
(a) Colour Dropout |  |  | (b) Full Attention |  | (c) Restricted Attention
Figure 3:  Restricted attention and colour dropout. See text for details.

Specifically, we impose a maximum disparity of $M$, _i.e_ pixels in the
reference frame $t$ are searched for locally in a square patch of size
$(2M+1)\times(2M+1)$, centered at the target pixel. Suppose the feature maps
have a dimension of $H\times W$, the affinity volume ($A$) is therefore a $4D$
tensor of dimension $H\times W\times(2M+1)\times(2M+1)$. The $(i,j,k,l)$ entry
of the tensor denotes the similarity between pixel $(i,j)$ of the target
frame, and pixel $(i+k-M,j+l-M)$ of the reference frame.

$\displaystyle A^{ijkl}$ $\displaystyle=\frac{\exp{\langle
f_{t}^{(i+k-M)(j+l-M)},f_{t+1}^{ij}\rangle}}{\sum_{p}\sum_{q}\exp{\langle
f_{t}^{(i+q)(j+p)},f_{t+1}^{ij}\rangle}}$ (3) $\displaystyle\hat{I}_{t+1}$
$\displaystyle=\psi(A_{(t,t+1)},I_{t})=\sum_{p}\sum_{q}A^{ij(p+M)(q+M)}I_{t}$
(4)

where $p,q\in[-M,M]$, $f_{t}=\Phi(g(I_{t});\theta)$ and
$f_{t+1}=\Phi(g(I_{t+1});\theta)$ refer to the feature embeddings for frame
$t$ and $t+1$ respectively. $\psi()$ denotes a soft-copy operation for
reconstructing $\hat{I}_{t+1}$ by “borrowing” colours from $I_{t}$ frame.

###  3.4 Long-term Correspondence Flow

One of the challenges on self-supervised learning of correspondence flow is
how to sample the training frames; If two frames are sampled closely in the
temporal axis, the objects remain unchanged in both appearance and spatial
position, matching becomes a trivial task and the model will not benefit from
training on them. In the contrary, if the frames are sampled with a large
temporal stride, the assumption of using reconstruction as supervision may
fail, due to complex object deformation, illumination change, motion blurs,
and occlusions.

In this section, we propose two approaches to improve the model’s robustness
to tracker drifting, and gently bridge the gap of training with samples that
are neither easy nor that difficult, _i.e_ scheduled sampling, and cycle
consistency.

####  3.4.1 Scheduled Sampling

Scheduled sampling is a widely used curriculum learning strategy for sequence-
to-sequence models [Bengio et al.(2015)Bengio, Vinyals, Jaitly, and Shazeer],
the main idea is to replace some ground truth tokens by the model’s
prediction, therefore improve the robustness and bridge the gap between train
and inference stage.

In our case, for $n$ frames in a video sequence, a shared embedding network is
used to get feature embeddings ($f_{i}=\Phi(g(I_{i});\theta)$ where
$i=1,...,n$), the reconstruction is therefore formalized as a recursive
process:

$\hat{I}_{n}=\begin{cases}\psi(A_{(n-1,n)},I_{n-1})&\text{\small{(1)}}\\\
\psi(A_{(n-1,n)},\hat{I}_{n-1})&\text{\small{(2)}}\end{cases}$

while reconstructing the $n$th frame ($\hat{I}_{n}$), the model may have
access to the previous frame as either groundtruth ($I_{n-1}$) or model
prediction ($\hat{I}_{n-1}$). During training, the probability of using ground
truth frames starts from a higher value ($0.9$) in early training stage, and
is uniformly annealed to a probability of $0.6$. Note that, as the model is
recursive, the scheduled sampling forces the model to recover from error
states and to be robust to drifting.

####  3.4.2 Cycle Consistency

Following the scheduled sampling, we also explicitly adopt a cycle consistency
for training correspondence flow. Unlike [Wang et al.(2019b)Wang, Jabri, and
Efros], we do not use the cycle consistency as the dominating supervision
signal, instead, it is treated as another regularizer for combating drifting.
During training, we apply our model $n$ frames forward and backward to the
current frame.

###  3.5 Training Objectives

Similar to [Vondrick et al.(2018)Vondrick, Shrivastava, Fathi, Guadarrama, and
Murphy], we pose frame reconstruction as a classification problem, the colour
for each pixel is quantized into $16$ classes with K-means clustering in the
Lab space. The objective function is defined as :

$\displaystyle
L=\alpha_{1}\cdot\sum_{i=1}^{n}\mathcal{L}_{1}(I_{i},\hat{I}_{i})+\alpha_{2}\cdot\sum_{j=n}^{1}\mathcal{L}_{2}(I_{j},\hat{I}_{j})$
(5)

where $\mathcal{L}_{1},\mathcal{L}_{2}$ refer to the pixel-wise cross entropy
between groundtruth and reconstructed frames in the _forward_ and _backward_
paths, the loss weights for both paths are set as $\alpha_{1}=1.0$,
$\alpha_{2}=0.1$ respectively, _i.e_ _forward_ path is weighted more than
_backward_ path.

##  4 Experiments and Analysis

In the following sections, we start with the training details, followed by
ablation studies, _e.g_ colour dropout, restricted attention, scheduled
sampling and cycle consistency.

Training Details In this paper, we train CNNs in a completely self-supervised
manner on Kinetics [Kay et al.(2017)Kay, Carreira, Simonyan, Zhang, Hillier,
Vijayanarasimhan, F., Green, Natsev, Suleyman, and Zisserman], meaning we do
_not_ use any information other than video sequences, and _not_ finetune for
any target task. As pre-processing, we decode the videos with a frame rate of
$6$fps, and resize all frames to $256\times 256\times 3$. In all of our
experiments, we use a variant of ResNet-18 as a feature encoder (we refer the
reader to our Arxiv paper for more
details222https://arxiv.org/abs/1905.00875). which ends up with feature
embeddings with spatial resolution of $1/4$ the original image. The max
disparity $M$ in the restricted attention is set to be $6$ (as described in
Section 3.3). The temporal length $n$ is set to $3$ in our case, so when
considering the forward-backward cycles, the sequence length is actually $5$
frames. We train our model end-to-end using a batch size of $8$ for 1M
iterations with an Adam optimizer. The initial learning rate is set to
$2e^{-4}$, and halved on 0.4, 0.6 and 0.8M iterations.

Evaluation Metrics In this paper, we report results on two public benchmarks:
video segmentation, and pose keypoint propagation. For both tasks, a ground
truth annotation is given for the first frame, and the objective is to
propagate the mask to subsequent frames. In video segmentation, we benchmark
our model on DAVIS-2017 [Pont-Tuset et al.(2017)Pont-Tuset, F., Caelles,
Arbeláez, Sorkine-Hornung, and Van Gool], two standard metrics are used, _i.e_
region overlapping ($\mathcal{J}$) and contour accuracy ($\mathcal{F}$). For
pose keypoint tracking, we evaluate our model on JHMDB dataset and report two
different PCK metrics. The first (PCKinstance) considers a keypoint to be
correct if the Normalized Euclidean Distance between that keypoint and the
ground truth is smaller than a threshold $\alpha$. The second (PCKmax) accepts
a keypoint if it is located within $\alpha\cdot\max(w,h)$ pixels of the ground
truth, where $w$ and $h$ are the width and height of the instance bounding
box.

###  4.1 Video Segmentation on DAVIS-2017

####  4.1.1 Ablation Studies

To examine the effects of different components, we conduct a series of
ablation studies by removing one component at a time. All models are trained
from scratch on Kinetics, and evaluated on the video segmentation task
(DAVIS-2017) without finetuning.

Figure 4: Model comparison on the problem of tracker drifting. The proposed
model with cycle consistency has shown to be most robust as masks propagate.

Method |  $\mathcal{J}$(Mean) |  $\mathcal{F}$(Mean) |
---|---|---|---
Ours (Full Model) | 47.7 | 51.3 |
Ours w/o Colour Dropout | 40.5 | 39.5 |
Ours w/o Restricted Attention | 40.8 | 39.7 |
Ours w/o Scheduled Sampling | 40.2 | 39.2 |
Ours w/o Cycle Consistency | 41.0 | 40.4 |

Figure 5: Ablation Studies on DAVIS-2017. $\mathcal{J}$: region overlapping,
$\mathcal{F}$: contour accuracy respectively.

Colour Dropout Instead of taking full-colour input, we follow Vondrick  _et
al_ [Vondrick et al.(2018)Vondrick, Shrivastava, Fathi, Guadarrama, and
Murphy], and convert all frames into grayscale for inputs. As shown in Table
5, both metrics drop significantly, _i.e_ $7.2\%$ in $\mathcal{J}$ and
$11.8\%$ in $\mathcal{F}$. This demonstrates the importance of bridging the
discrepancy between training and testing on utilizing _full-RGB_ colour
information.

Restricted Attention When computing the affinity matrix with full attention,
the model makes use of about $9.2$G of GPU memory to process a single $480$p
image, it is therefore impossible to train on _high-resolution_ frames with
large batch size on standard GPUs (12-24GB). In comparison, our model with
restricted attention only takes $1.4$G GPU memory for the same image. As Table
5 shows, performance dropped by $6.9\%$ and $11.6\%$ on $\mathcal{J}$ and
$\mathcal{F}$ before or after using restricted attention. This decrease
confirms our assumption about spatial coherence in videos, leading both a
decrease of memory consumption, and an effective regularizer that avoids the
model matching correspondences very far away, for instance matching repeated
patterns along the entire image.

Scheduled Sampling When not using the scheduled sampling, _i.e_ all frames
used for copying are groundtruth during training, the performance dropped
significantly from $47.7$ to $40.2$ in $\mathcal{J}$, $51.3$ to $39.2$ in
$\mathcal{F}$, suggesting that the scheduled sampling indeed improves the
model robustness under challenging scenarios, _e.g_ illumination change.

Cycle Consistency Lastly, we evalute the effectiveness of forward-backward
consistency. As seen in Figure 5, while both models start off with high
accuracy early in a video sequence, the model with cycle-consistency maintains
a higher performance in later stages of video sequences, indicating a less
severe drifting problem. This can also be reflected in the quantitative
analysis, where cycle consistency enables a performance boost by $7.5\%$ in
$\mathcal{J}$ and $12.1\%$ in $\mathcal{F}$.

####  4.1.2 Comparison with State-of-the-art

In Table 1, we show comparisons with previous approaches on the DAVIS-2017
video segmentation benchmark. Three phenomena can be observed: _First_ , Our
model clearly dominates all the self-supervised method, surpassing both video
colourization ($49.5$ vs. $34.0$ on $\mathcal{J}$&$\mathcal{F}$) and CycleTime
($49.5$ vs. $40.7$ on $\mathcal{J}$&$\mathcal{F}$). _Second_ , our model
outperforms the optical flow method significantly, suggesting the colour
dropout and scheduled sampling has improved the model’s robustness under
scenarios that optical flow is deemed to fail, _e.g_ large illunimation
changes. _Third_ , our model trained with self-supervised learning can even
approach the results of some supervised methods, for instance, despite of only
using a ResNet18 as feature encoder, the results are comparable with the
ResNet50 pretrained on ImageNet ($49.5$ vs. $49.7$ on
$\mathcal{J}$&$\mathcal{F}~{}(Mean)$). We conjecture this is due to the fact
that the model pretrained with ImageNet has only encoded high-level semantics,
while not optimized for dense correspondence matching.

Method | Supervised | Dataset |  $\mathcal{J}$&$\mathcal{F}$(Mean) |  $\mathcal{J}$(Mean) |  $\mathcal{J}$(Recall) |  $\mathcal{F}$(Mean) |  $\mathcal{F}$(Recall)
---|---|---|---|---|---|---|---
Identity | ✗ | - | 22.9 | 22.1 | 15.9 | 23.6 | 11.7
Optical Flow (FlowNet2) [Ilg et al.(2017)Ilg, Mayer, Saikia, Keuper, Dosovitskiy, and Brox] | ✗ | - | 26.0 | 26.7 | - | 25.2 | -
SIFT Flow [Liu et al.(2011)Liu, Yuen, and Torralba] | ✗ | - | 34.0 | 33.0 | - | 35.0 | -
Transitive Inv. [Wang et al.(2017)Wang, He, and Gupta] | ✗ | - | 29.4 | 32.0 | - | 26.8 | -
DeepCluster [Xie et al.(2016)Xie, Girshick, and Farhadi] | ✗ | YFCC100M | 35.4 | 37.5 | - | 33.2 | -
Video Colorization [Vondrick et al.(2018)Vondrick, Shrivastava, Fathi, Guadarrama, and Murphy] | ✗ | Kinetics | 34.0 | 34.6 | 34.1 | 32.7 | 26.8
CycleTime (ResNet-50) [Wang et al.(2019b)Wang, Jabri, and Efros] | ✗ | VLOG | 40.7 | 41.9 | 40.9 | 39.4 | 33.6
Ours (Full Model ResNet-18) | ✗ |  Kinetics [Kay et al.(2017)Kay, Carreira, Simonyan, Zhang, Hillier, Vijayanarasimhan, F., Green, Natsev, Suleyman, and Zisserman] | 49.5 | 47.7 | 53.2 | 51.3 | 56.5
Ours (Full Model ResNet-18) | ✗ |  OxUvA [Valmadre et al.(2018)Valmadre, Bertinetto, Henriques, R., Vedaldi, Smeulders, Torr, and Gavves] | 50.3 | 48.4 | 53.2 | 52.2 | 56.0
ImageNet (ResNet-50) [He et al.(2016)He, Zhang, Ren, and Sun] | ✓ | ImageNet | 49.7 | 50.3 | - | 49.0 | -
SiamMask [Wang et al.(2019a)Wang, Zhang, Bertinetto, Hu, and Torr] | ✓ | YouTube-VOS | 53.1 | 51.1 | 60.5 | 55.0 | 64.3
OSVOS[Caelles et al.(2017)Caelles, Maninis, Pont-Tuset, Leal-Taixé, Cremers, and Van Gool] | ✓ | DAVIS | 60.3 | 56.6 | 63.8 | 63.9 | 73.8
Table 1: Video segmentation results on DAVIS-2017 dataset. Higher values are
better.

####  4.1.3 Accuracy by Attributes

In Figure 6, we show DAVIS-2017 testing accuracy grouped into different
categories. The proposed method has shown to outperform the previous methods
in all situations, suggesting that our model has learned better feature
embeddings for robust correspondence flow.

Figure 6: Accuracy by attributes.

####  4.1.4 Qualitative Results

As shown in Figure 1 and Figure 7, we provide the qualitative prediction from
our model. The segmentation mask can be propagated through sequences even when
facing large scale variation from camera motion, and object deformations.

Figure 7: Qualitative results on DAVIS-2017.

####  4.1.5 Probing Upper Bound of Self-supervised Learning

Despite the superior performance on video segmentation, we notice that
training models on Kinetics is not ideal, as it is a human-centric video
dataset. However, most of the classes in DAVIS are not covered by Kinetics,
_e.g_ cars, animals.

To shed light on the potential of self-supervised learning on the task of
correspondence flow, we probe the upper bound by training on more diverse
video data. We randomly pick $8$ DAVIS classes and download $50$ additional
videos from YouTube, and further train the model by varying the amount of
additional data. Note that, we only download videos by the class labels, and
_no_ segmentation annotations are provided during further self-supervised
training.

Figure 8: Results after self-supervised learning on additional data.

As shown in Figure 8 and Table 2, two phenomena can be observed: _First_ , as
the number of additional training videos increases (Figure 8), all sequences
have seen a performance boost on both metrics ($\mathcal{J},\mathcal{F}$).
_Second_ , the segmentation on some of the classes are even comparable or
surpassing the supervised learning, _e.g_ Drift-c, Camel, Paragliding. We
conjecture this is due to the fact that, there are only very limited manual
annotations for these classes for supervised learining, _e.g_ Paragliding.
However, with self-supervised learning, we only require raw video data which
is almost infinite.

Method | Dataset | Dog | Dog-j | Drift-c | Drift-s | Camel | Mbike | Paragliding | Goat
---|---|---|---|---|---|---|---|---|---
Self-supervised ($\mathcal{J}$) | Kinetics | 58.0 | 18.6 | 3.9 | 27.1 | 59.4 | 44.8 | 32.4 | 71.1
Self-supervised ($\mathcal{J}$) | Additional | 76.1 | 21.7 | 27.8 | 37.7 | 70.0 | 57.4 | 42.8 | 73.7
Supervised ($\mathcal{J}$) [Yang et al.(2018)Yang, Wang, Xiong, Yang, and Katsaggelos] | COCO+DAVIS | 87.7 | 38.8 | 4.9 | 66.4 | 88.4 | 72.5 | 38.0 | 80.4
Self-supervised ($\mathcal{F}$) | Kinetics | 49.1 | 34.3 | 6.3 | 26.9 | 54.2 | 56.6 | 41.2 | 58.8
Self-supervised ($\mathcal{F}$) | Additional | 71.8 | 35.2 | 35.3 | 34.0 | 63.9 | 65.8 | 52.0 | 64.0
Supervised ($\mathcal{F}$) [Yang et al.(2018)Yang, Wang, Xiong, Yang, and Katsaggelos] | COCO+DAVIS | 84.6 | 45.2 | 8.4 | 57.7 | 92.2 | 76.7 | 58.1 | 74.7
Table 2: Quantitative comparison before and after training on additional
videos.

###  4.2 Keypoint Tracking on JHMDB

As shown in Table 3, our approach exceeds the previous methods [Vondrick et
al.(2018)Vondrick, Shrivastava, Fathi, Guadarrama, and Murphy] by an average
of $11.3\%$ in PCKinstance. Also, we achieve better performance in the more
strict<EMAIL_ADDRESS>metric when compared to the recent work [Wang et al.(2019b)Wang,
Jabri, and Efros]. Interestingly, when taking the benefit of the almost
infinite amount of video data, self-supervised learning methods (CycleTime and
Ours) achieve comparable or even outperforms model trained with the supervised
learning[He et al.(2016)He, Zhang, Ren, and Sun, Song et al.(2017)Song, Wang,
Van Gool, and Hilliges].

Method | Supervised | Dataset |  PCKinstance |  PCKmax |
---|---|---|---|---|---
@.1 | @.2 | @.1 | @.2 |
SIFT Flow[Liu et al.(2011)Liu, Yuen, and Torralba] | ✗ | - | 49.0 | 68.6 | - | - |
Video Colorization [Vondrick et al.(2018)Vondrick, Shrivastava, Fathi, Guadarrama, and Murphy] | ✗ | Kinetics | 45.2 | 69.6 | - | - |
CycleTime (ResNet-50) [Wang et al.(2019b)Wang, Jabri, and Efros] | ✗ | VLOG | 57.7 | 78.5 | - | - |
Ours (Full Model ResNet-18) | ✗ | Kinetics | 58.5 | 78.8 | 71.9 | 88.3 |
ImageNet (ResNet-50) [He et al.(2016)He, Zhang, Ren, and Sun] | ✓ | ImageNet | 58.4 | 78.4 | - | - |
Fully Supervised [Song et al.(2017)Song, Wang, Van Gool, and Hilliges] | ✓ | JHMDB | - | - | 68.7 | 81.6 |
Table 3: Keypoint tracking on JHMDB dataset (validation split 1). Higher
values are better.

##  5 Conclusion

The paper aims to explore the self-supervised learning for pixel-level
correspondence matching in videos. We proposed a simple _information
bottleneck_ that enables the model to be trained on standard RGB images, and
nicely close the gap between training and testing. To alleviate the challenge
from model drifting, we formulate the model in a recursive manner, trained
with scheduled sampling and forward-backward cycle consistency. We demonstrate
state-of-the-art performance on video segmentation and keypoint tracking. To
further shed light on the potential of self-supervised learning on
correspondence flow, we probe the upper bound by training on additional and
more diverse video datasets, and show that self-supervised learning for
correspondence flow is far from being saturated. As future work, potential
extensions can be: _First_ , explore better approaches for overcoming tracker
drifting, _e.g_ use explicit memory modules for long-term correspondence flow.
_Second_ , define robust loss functions that can better handle complex object
deformation and occlusion, _e.g_ predicting visibility mask or apply losses at
feature level [Oord et al.(2018)Oord, Li, and Vinyals]. _Third_ , instead of
doing quantization with Kmeans clustering, train the quantization process to
be more semantically meaningful.

### Acknowledgment

Financial support for this project is provided by EPSRC Seebibyte Grant
EP/M013774/1.

## References

  * [Agrawal et al.(2015)Agrawal, Carreira, and Malik] P. Agrawal, J. Carreira, and J. Malik.  Learning to see by moving.  In _Proc. ICCV_ , 2015.
  * [Bailer et al.(2017)Bailer, Varanasi, and Stricker] C. Bailer, K. Varanasi, and D. Stricker.  Cnn-based patch matching for optical flow with thresholded hinge embedding loss.  In _Proc. CVPR_ , 2017.
  * [Bengio et al.(2015)Bengio, Vinyals, Jaitly, and Shazeer] S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer.  Scheduled sampling for sequence prediction with recurrent neural networks.  In _NIPS_ , 2015.
  * [Brox et al.(2004)Brox, Bruhn, Papenberg, and Weickert] T. Brox, A. Bruhn, N. Papenberg, and J. Weickert.  High accuracy optical flow estimation based on a theory for warping.  In _Proc. ECCV_ , 2004.
  * [Brox et al.(2009)Brox, Bregler, and Malik] T. Brox, C. Bregler, and J. Malik.  Large displacement optical flow.  In _Proc. CVPR_ , 2009.
  * [Caelles et al.(2017)Caelles, Maninis, Pont-Tuset, Leal-Taixé, Cremers, and Van Gool] S. Caelles, K.K Maninis, J. Pont-Tuset, L. Leal-Taixé, D. Cremers, and L. Van Gool.  One-shot video object segmentation.  In _Proc. CVPR_ , 2017.
  * [Denton and Birodkar(2017)] E.L. Denton and V. Birodkar.  Unsupervised learning of disentangled representations from video.  In _NIPS_ , 2017.
  * [Dosovitskiy et al.(2015)Dosovitskiy, Fischer, Ilg, Hausser, Hazirbas, Golkov, Smagt, Cremers, and Brox] A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov, P. Smagt, D. Cremers, and T. Brox.  Flownet: Learning optical flow with convolutional networks.  In _Proc. ICCV_ , 2015.
  * [Fernando et al.(2017)Fernando, Bilen, Gavves, and Gould] B. Fernando, H. Bilen, E. Gavves, and S. Gould.  Self-supervised video representation learning with odd-one-out networks.  In _CVPR_ , 2017.
  * [Gan et al.(2018)Gan, Gong, Liu, Su, and Guibas] C. Gan, B. Gong, K. Liu, H. Su, and L. J. Guibas.  Geometry guided convolutional neural networks for self-supervised video representation learning.  In _Proc. CVPR_ , 2018.
  * [Han et al.(2017)Han, Rezende, Ham, Wong, Cho, Schmid, and Ponce] K. Han, R.S. Rezende, B. Ham, K.Y.K. Wong, M. Cho, C. Schmid, and J. Ponce.  Scnet: Learning semantic correspondence.  In _Proc. ICCV_ , 2017.
  * [Hartley and Zisserman(2004)] R. I. Hartley and A. Zisserman.  _Multiple View Geometry in Computer Vision_.  Cambridge University Press, second edition, 2004.
  * [He et al.(2016)He, Zhang, Ren, and Sun] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _Proc. CVPR_ , 2016.
  * [Horn and Schunck(1981)] B. K. P. Horn and B. G. Schunck.  Determining optical flow.  _Artificial Intelligence_ , 17:185–203, 1981.
  * [Ilg et al.(2017)Ilg, Mayer, Saikia, Keuper, Dosovitskiy, and Brox] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox.  Flownet 2.0: Evolution of optical flow estimation with deep networks.  In _Proc. CVPR_ , 2017.
  * [Isola et al.(2015)Isola, Zoran, Krishnan, and Adelson] P. Isola, D. Zoran, D. Krishnan, and E.H. Adelson.  Learning visual groups from co-occurrences in space and time.  In _Proc. ICLR_ , 2015.
  * [Jakab et al.(2018)Jakab, Gupta, Bilen, and Vedaldi] T. Jakab, A. Gupta, H. Bilen, and A. Vedaldi.  Conditionalimagegenerationforlearning the structure of visual objects.  In _NIPS_ , 2018.
  * [Jayaraman and Grauman(2015)] D. Jayaraman and K. Grauman.  Learning image representations tied to ego-motion.  In _Proc. ICCV_ , 2015.
  * [Jayaraman and Grauman(2016)] D. Jayaraman and K. Grauman.  Slow and steady feature analysis: higher order temporal coherence in video.  In _Proc. CVPR_ , 2016.
  * [Jia et al.(2016)Jia, De Brabandere, and Tuytelaars] X. Jia, B. De Brabandere, and T. Tuytelaars.  Dynamic filter networks.  In _NIPS_ , 2016.
  * [Jia et al.(2017)Jia, Ranftl, and Koltun] X. Jia, R. Ranftl, and V. Koltun.  Accurate optical flow via direct cost volume processing.  In _Proc. CVPR_ , 2017.
  * [Jing and Tian(2018)] L. Jing and Y. Tian.  Self-supervised spatiotemporal feature learning by video geometric transformations.  _arXiv preprint arXiv:1811.11387_ , 2018.
  * [Kay et al.(2017)Kay, Carreira, Simonyan, Zhang, Hillier, Vijayanarasimhan, F., Green, Natsev, Suleyman, and Zisserman] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, Viola F., T. Green, P. Natsev, M. Suleyman, and A. Zisserman.  The kinetics human action video dataset.  2017\.
  * [Kendall et al.(2017)Kendall, Martirosyan, Dasgupta, Henry, Kennedy, Bachrach, and Bry] A. Kendall, H. Martirosyan, S. Dasgupta, P. Henry, R. Kennedy, A. Bachrach, and A. Bry.  End-to-end learning of geometry and context for deep stereo regression.  In _Proc. CVPR_ , 2017.
  * [Kim et al.(2018)Kim, Cho, and Kweon] D. Kim, D. Cho, and I. S. Kweon.  Self-supervised video representation learning with space-time cubic puzzles.  In _AAAI_ , 2018.
  * [Kim et al.(2017)Kim, Min, Ham, Jeon, Lin, and Sohn] S. Kim, D. Min, B. Ham, S. Jeon, S. Lin, and K. Sohn.  Fcss: Fully convolutional self-similarity for dense semantic correspondence.  In _Proc. CVPR_ , 2017.
  * [Lee et al.(2017)Lee, Huang, Singh, and Yang] H.Y. Lee, J.B. Huang, M. Singh, and M.H. Yang.  Unsupervised representation learning by sorting sequences.  In _Proc. ICCV_ , 2017.
  * [Liu et al.(2011)Liu, Yuen, and Torralba] C. Liu, J. Yuen, and A. Torralba.  Sift flow: Dense correspondence across scenes and its applications.  _IEEE transactions on pattern analysis and machine intelligence_ , 2011.
  * [Meister et al.(2018)Meister, Hur, and Roth] S. Meister, J. Hur, and S. Roth.  UnFlow: Unsupervised learning of optical flow with a bidirectional census loss.  In _AAAI_ , 2018.
  * [Misra et al.(2016)Misra, Zitnick, and Hebert] I. Misra, C. L. Zitnick, and M. Hebert.  Shuffle and learn: Unsupervised learning using temporal order verification.  In _ECCV_ , 2016.
  * [Novotny et al.(2017)Novotny, Larlus, and Vedaldi] D. Novotny, D. Larlus, and A. Vedaldi.  Anchornet: A weakly supervised network to learn geometry-sensitive features for semantic matching.  In _Proc. CVPR_ , 2017.
  * [Oord et al.(2018)Oord, Li, and Vinyals] A.v.d. Oord, Y. Li, and O. Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * [Pont-Tuset et al.(2017)Pont-Tuset, F., Caelles, Arbeláez, Sorkine-Hornung, and Van Gool] J. Pont-Tuset, Perazzi. F., S. Caelles, P. Arbeláez, A. Sorkine-Hornung, and L. Van Gool.  The 2017 davis challenge on video object segmentation.  _arXiv:1704.00675_ , 2017.
  * [Revaud et al.(2015)Revaud, Weinzaepfel, Harchaoui, and Schmid] J. Revaud, P. Weinzaepfel, Z. Harchaoui, and C. Schmid.  Epicflow: Edge-preserving interpolation of correspondences for optical flow.  In _Proc. CVPR_ , 2015.
  * [Rocco et al.(2017)Rocco, Arandjelovic, and Sivic] I. Rocco, R. Arandjelovic, and J. Sivic.  Convolutional neural network architecture for geometric matching.  In _Proc. CVPR_ , 2017.
  * [Rocco et al.(2018)Rocco, Arandjelovic, and Sivic] I. Rocco, R. Arandjelovic, and J. Sivic.  End-to-end weakly-supervised semantic alignment.  In _Proc. CVPR_ , 2018.
  * [Smith and Brady(1995)] S.M. Smith and J.M. Brady.  Asset-2: Real-time motion segmentation and shape tracking.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 17(8):814–820, 1995.
  * [Song et al.(2017)Song, Wang, Van Gool, and Hilliges] J. Song, L. Wang, L. Van Gool, and O. Hilliges.  Thin-slicing network: A deep structured model for pose estimation in videos.  In _Proc. CVPR_ , 2017.
  * [Valmadre et al.(2018)Valmadre, Bertinetto, Henriques, R., Vedaldi, Smeulders, Torr, and Gavves] J. Valmadre, L. Bertinetto, J. F. Henriques, Tao R., A. Vedaldi, A. Smeulders, P. H. S. Torr, and E. Gavves.  Long-term tracking in the wild: A benchmark.  In _ECCV_ , 2018.
  * [Vondrick et al.(2018)Vondrick, Shrivastava, Fathi, Guadarrama, and Murphy] C. Vondrick, A. Shrivastava, A. Fathi, S. Guadarrama, and K. Murphy.  Tracking emerges by colorizing videos.  In _Proc. ECCV_ , 2018.
  * [Wang et al.(2019a)Wang, Zhang, Bertinetto, Hu, and Torr] Q. Wang, L. Zhang, L. Bertinetto, W. Hu, and P.H.S Torr.  Fast online object tracking and segmentation: A unifying approach.  In _Proc. CVPR_ , 2019a.
  * [Wang et al.(2017)Wang, He, and Gupta] X. Wang, K. He, and A. Gupta.  Transitive invariance for self-supervised visual representation learning.  In _Proc. CVPR_ , 2017.
  * [Wang et al.(2019b)Wang, Jabri, and Efros] X. Wang, A. Jabri, and A. Efros.  Learning correspondence from the cycle-consistency of time.  In _Proc. CVPR_ , 2019b.
  * [Wang and Gupta(2015)] X.L Wang and A. Gupta.  Unsupervised learning of visual representations using videos.  In _ICCV_ , 2015.
  * [Wang et al.(2018)Wang, Yang, Yang, Zhao, Wang, and Xu] Y. Wang, Y. Yang, Z. Yang, L. Zhao, P. Wang, and W. Xu.  Occlusion aware unsupervised learning of optical flow.  In _Proc. CVPR_ , 2018.
  * [Wei et al.(2018)Wei, Lim, Zisserman, and Freeman] D.L. Wei, J.J. Lim, A. Zisserman, and W.T. Freeman.  Learning and using the arrow of time.  In _CVPR_ , 2018.
  * [Weinzaepfel et al.(2013)Weinzaepfel, Revaud, Harchaoui, and Schmid] P. Weinzaepfel, J. Revaud, Z. Harchaoui, and C. Schmid.  DeepFlow: Large displacement optical flow with deep matching.  In _Proc. ICCV_ , pages 1385–1392, 2013.
  * [Wiles et al.(2018)Wiles, Koepke, and Zisserman] O. Wiles, A.S. Koepke, and A. Zisserman.  X2face: A network for controlling face generation using images, audio, and pose codes.  In _Proc. ECCV_ , 2018.
  * [Wiskott and Sejnowski(2002)] L. Wiskott and T. Sejnowski.  Slow feature analysis: Unsupervised learning of invariances.  In _Neural Computation_ , 2002.
  * [Xie et al.(2016)Xie, Girshick, and Farhadi] J. Xie, R. Girshick, and A. Farhadi.  Unsupervised deep embedding for clustering analysis.  In _Proc. ICML_ , 2016.
  * [Yang et al.(2018)Yang, Wang, Xiong, Yang, and Katsaggelos] L. Yang, Y. Wang, X. Xiong, J. Yang, and A. K. Katsaggelos.  Efficient video object segmentation via network modulation.  In _Proc. CVPR_ , 2018.
  * [Yu et al.(2016)Yu, Harley, and Derpanis] J.J. Yu, A.W. Harley, and K.G. Derpanis.  Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness.  In _Proc. ECCV_ , 2016.

##  Appendix A Network Architecture

We use a modified ResNet-18[He et al.(2016)He, Zhang, Ren, and Sun]
architecture with enlarged output feature maps size. Details of the network
can be found below.

0 | Input image
---|---
Feature extractor
1 |  $7\\!\times\\!7$ conv with stride 2 and 64 filters
2 |  $3\\!\times\\!3$ Residual Block with stride 1 and 64 filters
3 |  $3\\!\times\\!3$ Residual Block with stride 2 and 128 filters
4 |  $3\\!\times\\!3$ Residual Block with stride 1 and 256 filters
5 |  $3\\!\times\\!3$ Residual Block with stride 1 and 256 filters
Table 4: Network architecture. A Residual Block stands for a residually
connected sequence of operations: convolution, batch normalization, rectified
linear units (ReLU), convolution, batch normalization. See [He et al.(2016)He,
Zhang, Ren, and Sun] for details.

###  A.1 Failure Cases

Figure 9 demonstrates some failure cases during mask propagation. Row 1 shows
our tracker fails due to occlusions. As we only use the mask of the previous
frame to propagate, an object is unlikely to be retrieved after it has been
occluded. Similarly, it is difficult to recover an object once it goes out of
the frame (Row 2). Lastly, if the object is under complex deformation, it is
likely to incur the model drifting.

Figure 9: Common failed cases, including occulusion, out-of-view and complex
transformation.