# STint: Self-supervised Temporal Interpolation for Geospatial Data

Nidhin Harilal
&Bri-Mathias Hodge
&Aneesh Subramanian &Claire Monteleoni 11footnotemark: 1 Department of
Computer Science, University of Colorado Boulder, CO, USADepartment of
Electrical, Computer and Energy Engineering, University of Colorado Boulder &
National Renewable Energy Laboratory (NREL), Boulder, CO, USA Department of
Atmospheric and Oceanic Sciences, University of Colorado Boulder, CO, USAINRIA
Paris, France

###### Abstract

Supervised and unsupervised techniques have demonstrated the potential for
temporal interpolation of video data. Nevertheless, most prevailing temporal
interpolation techniques hinge on optical flow, which encodes the motion of
pixels between video frames. On the other hand, geospatial data exhibits lower
temporal resolution while encompassing a spectrum of movements and
deformations that challenge several assumptions inherent to optical flow. In
this work, we propose an unsupervised temporal interpolation technique, which
does not rely on ground truth data or require any motion information like
optical flow, thus offering a promising alternative for better generalization
across geospatial domains. Specifically, we introduce a self-supervised
technique of dual cycle consistency. Our proposed technique incorporates
multiple cycle consistency losses, which result from interpolating two frames
between consecutive input frames through a series of stages. This dual cycle
consistent constraint causes the model to produce intermediate frames in a
self-supervised manner. To the best of our knowledge, this is the first
attempt at unsupervised temporal interpolation without the explicit use of
optical flow. Our experimental evaluations across diverse geospatial datasets
show that STint significantly outperforms existing state-of-the-art methods
for unsupervised temporal interpolation.

##  1 Introduction

Video interpolation is a technique used to generate intermediate frames
between consecutive frames in a video sequence. The primary goal is to enhance
the temporal resolution, reduce motion blur, and create smoother transitions
between frames. Video interpolation is critical in various fields, including
computer vision, video processing, climate modeling, satellite imagery, radar
data processing, etc. In the field of computer vision, video interpolation has
been widely utilized for frame-rate up-conversion [3, 16], slow-motion video
generation [12], and video stabilization [4]. The applicability of video
interpolation extends to climate modeling, where it aids in generating high-
resolution temporal data from low-resolution climate models [18, 7], also
referred to as temporal interpolation. Interpolation techniques also prove
valuable in analyzing other scientific datasets, like satellite imagery
analysis, to fill in missing temporal data and improve frame quality [35].
Similarly, it is essential for radar data processing [21], where it is
employed to enhance the resolution and accuracy of the retrieved information.

Figure 1: Schematic of cycle consistency technique for temporal interpolation.
Here, $I_{T}$ refers to the $T^{th}$ frame in a temporal sequence. The idea is
to use the intermediate frames from a triplet of input sequence to reconstruct
back the middle frame. This enforces similarity between frames at $T=1$, which
is then used to train a network $f$, imposing a cycle consistency as shown by
the two cyclic arrows.

Most existing methods for video interpolation techniques are supervised and
rely on ground truth data to learn the interpolation task. Although a few
unsupervised interpolation methods [27, 19] have been proposed in recent
years, these leverage optical flow estimation [9] to model the motion between
consecutive frames, thus limiting them with the assumptions made by optical
flow algorithms. Optical flow estimation is sensitive to variations in
texture, lighting conditions, and occlusions [32]. In cases where the motion
is non-rigid or exhibits irregular patterns, such as in geospatial datasets,
traditional optical flow-based methods may not produce satisfactory results.
Therefore, such methods that rely on optical flow can struggle to generalize
to other domains, particularly those with complex or unfamiliar motion
patterns.

###  1.1 Unsupervised Temporal Interpolation

The need for unsupervised interpolation methods stems from the challenges in
addressing the gaps and inconsistencies within datasets, particularly in
domains with vast amounts of data, like historical archives of diverse
geospatial datasets. In situations where supervised methods are hindered by
the lack of labeled data points or where manual interpolation becomes
unfeasible due to the sheer volume of data, unsupervised techniques can step
in to bridge these gaps effectively. In the context of data-rich scenarios,
where historical data might be fragmented or irregularly spaced, unsupervised
temporal interpolation methods offer a means to create more continuous and
coherent representations of the data. These pre-trained networks can then be
fine-tuned in target domains with data-poor situations like inferring higher-
temporal information in scarcely recorded regions. By temporally interpolating
patterns and relationships within the existing data points, these methods can
generate valuable insights into historical variations in geospatial data,
aiding in understanding past trends while also extending these insights and
predictions to data-poor regions.

###  1.2 Optical Flow Estimation for Geospatial Data

Geospatial data is inherently complex and multidimensional, involving various
atmospheric, oceanic, and land variables collected over extensive spatial and
temporal scales. Optical flow, primarily designed to capture motion between
frames with high-temporal granularity (usually in fractions of a second),
struggles to adequately capture the intricate interactions and patterns
present in geospatial data. Another critical factor that renders optical flow
unsuitable for geospatial data analysis is the presence of deformable objects.
Unlike traditional image sequences where objects have consistent pixel-level
motion, geospatial data involves dynamic elements such as clouds, atmospheric
fronts, and ocean currents, which can undergo complex and non-rigid
deformations. Optical flow-based methods assume pixel motion arises from rigid
body movements, making them ill-suited for interpolating geospatial data. As a
result, leveraging optical flow for geospatial data interpolation might lead
to oversights and inaccuracies, emphasizing the need for specialized methods
to effectively address the unique characteristics and challenges presented by
geospatial datasets.

To summarize, temporal interpolation between successive frames in video
sequences is a critical component with diverse applications. While supervised
and unsupervised methods for temporal interpolation have shown some promising
outcomes for video data, the state-of-the-art still relies on an optical flow
formulation encoding the motion field of pixels. The low temporal granularity
paired with dynamic spatial qualities, with a range of movements,
deformations, and temporal alterations of geospatial datasets defy several
assumptions inherent to optical flow. In response to these challenges, we
introduce STint (Self-supervised Temporal Interpolation), a temporal
interpolation approach that avoids the need for optical flow. STint leverages
a unique dual cycle consistency training mechanism that is built upon the idea
of cycle consistency from Figure 1, eliminating the need for labels or dense
motion data while efficiently harnessing spatiotemporal context for
interpolation tasks. To summarize, our contributions are as follows:

  * •

We propose STint- a self-supervised framework that exploits a reformed dual
cycle consistency constraint for temporal interpolation.

  * •

STint learns to temporally interpolate without using any motion information
like optical flow, thus making it agnostic to any motion/occlusion-related
assumptions.

  * •

We present the effectiveness of STint’s self-supervised dual cycle consistent
training paired with its optical flow-agnostic property on various geospatial
datasets.

##  2 Related Work

In this section, we discuss the topics relevant to this work, starting from
methods for video frame interpolation and comparing our proposed method with
them. Next, we discuss a few methods applying the cycle consistency constraint
(shown in Figure 1) that inspired us to develop our proposed dual cycle
consistency technique for STint.

###  2.1 Video Frame Interpolation

In recent years, significant advancements have been made in the field of video
interpolation. The SuperSloMo method by Jiang et al. [13] is a supervised
approach that uses a two-stage process involving optical flow estimation and
adaptive convolution layers for synthesizing intermediate frames. Niklaus et
al. [22] proposed a deep learning framework for video frame interpolation
using asymmetric reverse flow and cyclic fine-tuning, which models
bidirectional motion and refines generated frames iteratively. Despite the
methods being new and offering improved performance, they still rely on
optical flow and may face generalization issues in scenarios discussed in
previous sections. Liu et al. [20] developed DVF, a deep learning-based video
frame interpolation network that exploits flowing pixel values (voxel) from a
given frame to the target frame. While the idea of using an augmented version
of optical flow seems better, it is bound by the assumption that most of the
pixel patches between frames are near copies of each other, which may not be
applicable for data types other than high-frame-rate videos. There exists
other methods [14, 25, 6] that follow similar ideas and incorporate motion
estimation for frame synthesis. However, like other optical flow-based
methods, these will most likely fail for data types with non-rigid motion
and/or lower temporal granularity, both of which are generally the
characteristics of geospatial data. The idea of optical flow agnostic video
interpolation has been investigated recently. Specifically, FLAVR [15] was
proposed as a flow-agnostic method that is end-to-end trainable for multi-
frame video interpolation. However, FLAVR [15] is limited to domains where
direct supervision is available. Our work presents an unsupervised framework
for temporal interpolation without the need for optical flow, thus making it
specifically applicable to geospatial data types.

###  2.2 Unsupervised Cycle Consistency

When we talk about unsupervised cycle consistency, we are addressing a
property that assures when we transition from one domain to another, and then
by reverting to the original, we end up close to our starting point. For
instance, in a task such as translating an image from a source to target style
[37] (e.g., transforming a daytime image to appear as if it was taken at
night) and then translating it back to the original style, the cycle
consistency loss measures the dissimilarity between the original and the
twice-translated image. The concept of unsupervised cycle consistency has
shown some promising potential in video interpolation tasks [19, 27]. In the
context of video interpolation, cycle consistency can be seen as trying to
reconstruct back frames from the original temporal resolution by interpolating
between predicted intermediate frames. Liu et al. [19] was the first to
introduce cycle consistency as a constraint to regularize a fully supervised
video interpolation model. Most related to our wor is [27], which used cycle
consistency for unsupervised pre-training on SuperSloMo [12] architecture for
video interpolation. Our work notably differs in several essential respects.
The most important being that our technique does not require any dense motion
information while being fully unsupervised to perform temporal interpolation,
as compared to [19, 27], both of which rely on estimating optical flow to
model dense pixel movements. Secondly, our technique employs an improved dual
cycle consistency loss which not only attempts to reconstruct the original
sequence but also regularizes the predicted intermediate sequence at
appropriate time stamps for better overall interpolation.

##  3 Method

By introducing STint, we aim to contribute a novel approach that operates
under two key rules - unsupervised learning and motion-agnostic enhancement.
The subsequent subsections delve into the methodology and technical details of
the proposed STint.

Figure 2: A detailed overview of the STint framework. Building upon the cycle
consistency pretext task from Figure 1, we propose to learn our model
$\mathcal{M}$ for temporal interpolation via multiple cycle consistency losses
(indicated in inscribed shapes), obtained by interpolating two frames between
consecutive input frames across multiple stages. Our proposed dual cycle
consistency makes all the transformations predicted by the model to be
invertible.

###  3.1 Dual Cycle Consistent Interpolation

We consider the following learning setting: We are given spatiotemporal data
$D_{K}$ with $K$ as the temporal frequency. We are interested in increasing
this temporal frequency using our proposed STint with the following
properties: (a) It is fully unsupervised, and (b) It does not use motion
information, i.e., optical flow. We consider the task of generating two
additional frames between each pair in the original sequence. Specifically,
consider $[I_{0},I_{1},I_{2}]$ to be a triplet of consecutive input frames
from a spatiotemporal sequence $\\{I_{T_{i}}\\}^{N}_{i=0}$, we aim to learn a
network that takes two consecutive frames as inputs and can predict two
intermediate frames in between these consecutive input frames. Considering
$[I_{0},I_{1}]$ as inputs, we obtain the corresponding intermediate sequence
$[I_{\delta},I_{2\delta}]$ as follows:

$[I_{\delta},I_{2\delta}]=\mathcal{M}(I_{0},I_{1})$ (1)

where $\mathcal{M}$ denotes the network that we aim to train for temporal
interpolation. Our goal is to train $\mathcal{M}$ in an unsupervised manner,
that is, without relying on pre-labeled training data or explicit guidance
during the learning process.

Motivated by the recent strides made in video interpolation [19, 27], as well
as the remarkable progress in unpaired image-to-image translation using
Generative Adversarial Networks (GANs) [37], we propose to use a similar
approach for refining our network $\mathcal{M}$ for temporal interpolation.
The cornerstone of our approach is the preservation of dual cycle consistency
over the duration of the entire temporal sequence. Cycle consistency, in
essence, means that when we execute an operation and its inverse
consecutively, we return to our original starting position. For instance, in
the context of our network $\mathcal{M}$ for temporal interpolation, we aim
for a scenario where a series of transformations applied to the sequence
frames, followed by their reverse transformations, should produce a result
that closely resembles the initial temporal sequence. We achieve this via
multiple cycle consistency losses across different stages:

####  3.1.1 Stage I:

Our unsupervised dual cycle consistency constraint consists of two phases, as
shown in Figure 2. The stage-I begins by considering two pairs of consecutive
input frames $[I_{0},I_{1}]$ and $[I_{1},I_{2}]$; therefore, the individual
corresponding intermediate frames using a network $\mathcal{M}$ can be written
as:

$[I_{\delta}^{[1]},I_{2\delta}^{[1]}]=\mathcal{M}(I_{0},I_{1})$ (2)
$[I_{1+\delta}^{[1]},I_{1+2\delta}^{[1]}]=\mathcal{M}(I_{1},I_{2})$ (3)

####  3.1.2 Stage II:

The idea here is to use the predicted intermediate frames from stage I with
the temporal window that matches the original sequence as input pairs again.
In this particular case, the idea is to use
$I_{\delta}^{[1]},I_{1+\delta}^{[1]}]$ and
$[I_{2\delta}^{[1]},I_{1+2t}^{[1]}]$ as inputs. Mathematically, it can be
written as:

$[I_{2\delta}^{[2]},I_{1}^{[2]}]=\mathcal{M}(I_{\delta}^{[1]},I_{1+\delta}^{[1]})$
(4)
$[I_{1}^{[2]},I_{1+\delta}^{[2]}]=\mathcal{M}(I_{2\delta}^{[1]},I_{1+2\delta}^{[1]})$
(5)

The first trick is that both the predictions at $T=1$ from stage II must match
the original middle input frame $I_{1}$, thus we introduce the first cycle
consistency constraint to optimizing the network $\mathcal{M}$ on the
reconstruction error at $T=1$ as follows:

$\mathcal{L}_{CC_{1}}=arg\,min_{\mathcal{M}_{\theta}}(||\,I_{1}-\mathcal{M}(I^{[1]}_{\delta},I^{[1]}_{1+\delta})(2)\,||_{1}+||\,I_{1}-\mathcal{M}(I^{[1]}_{2\delta},I^{[1]}_{1+2\delta})(1)\,||_{1})$
(6)

where $\mathcal{M}(\cdot)(p),p\in[1,2]$ refers to $p^{th}$ prediction out of
two outputs from $\mathcal{M}$. $\mathcal{L}_{CC_{1}}$ corresponds to the
yellow squares shown in Figure 2.

To make the interpolation more stable, we propose to integrate cycle
consistency constraint on the other predictions from stage II. Specifically,
we enforce similarity between predictions from stage II (${I}^{[2]}_{2\delta}$
& ${I}^{[2]}_{1+\delta}$) and predictions from stage I ($I^{[1]}_{2\delta}$ &
$I^{[1]}_{1+\delta}$), shown as inscribed triangle and diamond in Figure 2.
Mathematically, we can write it as:

$\begin{split}\mathcal{L}_{CC_{2}}&=arg\,min_{\mathcal{M}_{\theta}}\frac{1}{2}\left(||\,I_{2\delta}^{[1]}-{I}_{2\delta}^{[2]}\,||_{1}+||\,I_{1+\delta}^{[1]}-I_{1+\delta}^{[2]}\,||_{1}\right)\\\
&=arg\,min_{\mathcal{M}_{\theta}}\frac{1}{2}(||\mathcal{M}(I_{1},I_{2})(2)-\mathcal{M}(I^{[1]}_{\delta},I^{[1]}_{1+\delta})(1)||_{1}\\\
&\;\;\;\;\;\;\;\;\;\;+||\,\mathcal{M}(I_{1},I_{2})(1)-\mathcal{M}(I_{2\delta},I_{1+2\delta})(2)\,||_{1})\end{split}$
(7)

While the first cycle consistency constraint ($\mathcal{L}_{CC_{1}}$) acts as
the key for learning to interpolate between frames, the second cycle
consistency term ($\mathcal{L}_{CC_{2}}$) would help generate stable
intermediate frames such that it maintains both spatial and temporal
consistency with the entirety of the sequence. This dual cycle consistency
procedure can be applied to all such intermediate frames, making the learning
procedure complete. Optimizing on this dual cycle consistency constraints the
model to learn to produce plausible intermediate climate frames so that these
can then be used to reversely reconstruct the original input frames.

$\mathcal{L}=\lambda_{CC_{1}}\mathcal{L}_{CC_{1}}+\lambda_{CC_{2}}\mathcal{L}_{CC_{2}}$
(8)

Avoiding Trivial Solutions: A straightforward solution to the dual cycle
consistency method as defined above could be to merely duplicate the input
frames for intermediate predictions. However, this does not transpire in our
learning setting because we repeatedly compel the model to create reversible
frames for pairs comprising diverse temporal frames. Moreover, no path exists
for transmitting the temporal information of the input frames between
successive forward propagations, thus making the learning avoid trivial
solutions.

###  3.2 Interpolation Network $\mathcal{M}$

Several existing methods, such as ConvLSTMs [30], 3D ResNet [33], PyConv [8]
and SepConv [23], can serve as the baseline model. In this work, we choose a
modified version of 3D-U-Net [5] due to its notable simplicity, paired with
its impressive learning capabilities. 3D-U-Net [5] is designed by adapting the
original U-Net [28] stem to 3D inputs by replacing the 2D convolutions with
consecutive 3D convolutional layers. We go a step ahead and adopt the
following lightweight architectural changes to improve model performance.

  * •

We remove the max-pooling layers from the original 3D-U-Net while retaining
the 3D convolution and transposed convolution layers.

  * •

We incorporate Squeeze-and-Excite (SE) blocks [11] shown in Figure 3 into
3D-U-Net. We adapt these SE blocks to spatiotemporal inputs by simply using a
3D global average pooling operation for the squeeze function, making them 3D
SE blocks.

Henceforth, with these modifications, we consider our network $\mathcal{M}$
for temporal interpolation as 3D-U-Net-SE. The primary driver for integrating
3D SE blocks into our model was to enhance the quality of representations
generated by the original 3D-U-Net. These 3D SE blocks contribute to better
modeling of inter-dependencies among the convolutional features and carry out
feature re-calibration [11]. Through this re-calibration, the model can
leverage global information to selectively accentuate informative features
while diminishing the less useful ones, thereby improving overall performance.

Figure 3: Squeeze and Excitation Block

###  3.3 Unsupervised Training and Fine-tuning

For any given data, we first obtain triplets of frames from a given sequence
such that two intermediate frames exist between every consecutive frame. Each
consecutive pair from this triplet is then used as an input to the STint
framework with a focus on maintaining dual cycle consistency as described in
the previous section. Unless mentioned otherwise, the loss coefficients
$\lambda_{CC_{1}}$ and $\lambda_{CC_{2}}$ have been set empirically using a
validation set as $\lambda_{CC_{1}}=0.65$, $\lambda_{CC_{2}}=0.35$,

Following the initial unsupervised training, we fine-tune our mode by using
the model’s learned parameters as a starting point and further optimizing
these parameters. This process involves a few additional rounds of training
with a lower learning rate to refine the parameters without dramatically
altering the existing structure of the learned features. This ensures that the
model progressively learns to capture more intricate details and nuances,
thereby boosting the quality of the interpolated frames. For fine-tuning, we
use an optimization criterion ($\mathcal{L}_{fine}$) that is a weighted
combination of the reconstructed errors from ground truth and the dual cycle
consistency losses. Mathematically, it can be written as:

$\mathcal{L}_{fine}=\mathcal{L}_{||I_{t}-\hat{I}_{t}||}+\gamma_{CC_{1}}\mathcal{L}_{CC_{1}}+\gamma_{CC_{2}}\mathcal{L}_{CC_{2}}$
(9)

where $I_{t}$ & $\hat{I}_{t}$ corresponds to the ground truth intermediate
frames and the predictions respectively.

We found that including the dual cycle consistency loss along with the
supervised reconstruction loss leads to faster and better convergence. While
the supervised reconstruction loss $\mathcal{L}_{||I-\hat{I}_{t}||}$ helps the
model get direct feedback on the produced interpolated frames, the dual cycle
consistency loss $\mathcal{L}_{CC_{1}},\mathcal{L}_{CC_{2}}$ acts as a
constraint that measures the difference between an original frame and a frame
that has been interpolated and then reverse-interpolated. Minimizing this type
of loss helps the model learn to produce interpolated frames that, when
reverse-interpolated, closely resemble the original frames, thus making the
model generate temporally consistent frames. Finally, we use experimentally
selected weights for this linear combination of losses: $\gamma_{CC_{1}}=0.5$,
$\gamma_{CC_{2}}=0.3$.

##  4 Experiments

In this section, we report the datasets used in the experiments and
comparisons with the baseline and state-of-the-art. We also discuss the
limitations of the proposed method. In all the following experiments, we
report the test error from the epoch with the lowest validation error.

Following previous works, we use PSNR and SSIM metrics to report the
quantitative results of our method. For some geospatial datasets, we also
report a criterion called Scatter Index (SI), also denoted as
$\frac{MSE}{Capacity}$ which can be considered as a normalized version of Mean
Square Error (MSE), where Capacity stands for the maximum value of the data
variable under consideration [36].

###  4.1 Architecture Details

Built entirely on PyTorch [26], the architecture of STint consists of
approximately 40M parameters. Our backbone is a modified version of 3D-U-Net
[5] by incorporating Squeeze-and-Excite (SE) blocks [11] into 3D-U-Net [5].
Our choice of architecure is based on the simplicity of 3D-U-Net and the
powerful learning capabilities offered by SE. Additionally, we find that
replacing the max-pooling layers with 3D global average pooling in 3D-U-Net
version by [5] performs best for our experiments. We use ReLU between 3D
convolutions and 3D transposed convolutions. Following [5], we also introduce
batch normalization (“BN”) before each ReLU. Unless specified otherwise, we
use Adam for optimization with $\beta_{1}=0.9$, $\beta_{2}=0.999$, and no
weight decay.

###  4.2 Baseline & State-of-the-Art

We establish our baseline as a straightforward approach of ‘trivial copy’. It
refers to duplicating existing frames within the video sequence to generate
interpolated frames. It mimics the temporal progression by copying frames at
regular intervals without any complex computations or algorithms. For example,
each sequence element of geospatial data at 3-hourly intervals can be
replicated twice in a consecutive manner to obtain 1-hourly data.

Furthermore, considering that there are not a lot of unsupervised methods for
the task of temporal interpolation except [27], which uses dense optical flow
for frame interpolation. We choose [27] for comparison. In scenarios where the
optical flow was futile (more in subsequent subsections), some hyperparameters
like learning rate needed to be tweaked in order to make the training
possible; however, mostly all the settings were chosen to match the setup of
[27].

Data source | Data type | Total frames | Spatial resolution | Temporal resolution
---|---|---|---|---
IPSL | Surface wind | 186,880 | $96\times 236$ | 3-hourly
ERA5 | Solar-irradiance | 196,375 | $96\times 236$ | 1-hourly
CARRA | Surface temperature | 190,224 | $200\times 500$ | 1-hourly
SEN12MS | Cloud (Satellite) | 60,117 | $256\times 256$ | 2-hourly
UCF-101 | Human action videos | 120,790 | $256\times 256$ | 25 FPS
Table 1: Statistics of datasets used for experiments.

###  4.3 Climate Geospatial Data

Our experiments on geospatial domains encompass different atmospheric
variables. Specifically, we consider three of the most commonly used sources
for atmospheric geospatial data:

  * •

IPSL: We use the 3-hourly resolution climate model outputs of the wind field
from Institut Pierre-Simon Laplace (IPSL) Phase 6 of the Coupled Model
Intercomparison Project (CM6A). IPSL is considered to have high fidelity in
reproducing 20th-century observed climate over the US [1].

  * •

ERA5: We use the 1-hourly downward solar irradiance fields from ERA5 [10],
which is the fifth generation high-resolution observational data (reanalysis)
from European Centre for Medium-Range Weather Forecasts (ECMWF). The ERA5
global reanalysis is the most used data set from Copernicus Climate Change
Service (C3S) in the Copernicus Climate Data Store (CDS) [10].

  * •

CARRA: We consider 1-hourly regional observational records of surface
temperature for the Arctic region [17]. While the ERA5 global reanalysis
serves as a widely used dataset, its broad scope does not cater to the
intricate nuances of the Arctic’s rapidly changing climate. This makes CARRA
an ideal candidate to study temporal interpolation in this geographically
challenging subcontinent.

For each quadruple of consecutive frames from the spatiotemporal sequence, the
middle two frames serve as the ground truth while the other two are inputs.
Table 1 summarizes the statistics of each dataset. We use $80$-percent of the
total frames for unsupervised training and fine-tuning, while the rest is used
for evaluation.

####  4.3.1 STint Pre-training & Finetuning

We use a combination of four NVIDIA Quadro RTX 5000 and an RTX 3090 to train
the model for $400$ epochs with a learning rate with a batch size of 16. We
initialize the learning rate with $3\times 10^{-4}$ and decay it by a factor
of $2$ for every $400,000$ mini-batch updates or when the performance
stagnates. We use Adam optimizer with coefficients $(0.9,0.999)$ with a weight
decay of $0.05$ to optimize the loss. We find that the cycle-consistency loss
weight values of $\lambda_{CC_{1}}=0.65$, $\lambda_{CC_{2}}=0.35$ work best
for our setting and use it throughout the experiments. We load the pre-trained
weights and fine-tune STint in an end-to-end manner. We train the model for 50
epochs with a learning rate of $2\times 10^{-3}$. We perform optimization on a
linear combination of losses cycle consistency losses with the error from
supervision. We use experimentally selected weights for this linear
combination of losses, where the weights for cycle consistency loss are:
$\gamma_{CC_{1}}=0.5$, $\gamma_{CC_{2}}=0.3$.

ERA5 Solar
---
|  $\frac{MSE}{Capacity}$ ($\downarrow$) | PSNR ($\uparrow$) | SSIM ($\uparrow$)
Baseline | 0.3086 | 25.238 | 0.623
SuperSloMo | 0.0907 | 30.157 | 0.733
Proposed | 0.0561 | 32.731 | 0.792
IPSL Wind
Baseline | 0.6206 | 24.097 | 0.619
SuperSloMo | 0.4150 | 29.713 | 0.681
Proposed | 0.2904 | 31.167 | 0.713
CARRA Temperature
Baseline | 0.5319 | 27.832 | 0.667
SuperSloMo | 0.1604 | 30.276 | 0.724
Proposed | 0.0975 | 31.908 | 0.775
Table 2: Temporal interpolation results on geospatial data with different
atmospheric variables.

Table 2 presents our results which show that our unsupervised dual cycle
consistency-based STint outperforms SuperSloMo [27] uses regular cycle
consistency (described in Figure 1) for self-supervised training. We observe
consistent performance gains across all the climate variables. In contrast to
cycle consistency-based methods, predictions using baseline trivial copy
result in notably inferior interpolation frames. Using IPSL wind fields data,
at a frame count three times larger than both ERA5 and CARRA datasets, we
outperform SuperSloMo [27] with PSNR of 31.17dB vs. 29.71dB. Figure 4 shows a
visual comparison of the predictions on ERA5 solar field data. The highly
dynamic temporal nature, followed by extensive deformations of spatial
features between frames, falters the optical flow estimation, thus generating
artifacts while reconstructing the intermediate frames. STint, on the other
hand, generates predictions that are much closer to the actual ground truth
intermediate frames.

Figure 4: An example of temporal interpolation of ERA5 solar field data.
Optical flow-based SuperSloMo [27] can be seen to struggle with the non-
rigidity of motion inherent to solar data. Our proposed STint, a fully
unsupervised and optical flow agnostic technique, generates intermediate frame
$\hat{I}_{t}$ that is much closer to the ground truth $I_{t}$, making it a
robust interpolation method in scenarios with non-rigid motion.

Since the quantitative measures show only a weak correlation with the human
visual system, Figure 5 visually compares the predicted intermediate frames on
IPSL [1] and CARRA [17] from different methods. Optical flow-based SuperSloMo
struggles at regions with high deformations, complex and large movements [2].
This can be seen in the form of tearing artifacts in the predicted
intermediate frames. The predictions from STint are visually much closer to
the ground truth.

Figure 5: Qualitative comparison of a sample of predicted intermediate frames
from STint vs SuperSloMo [27] on IPSL [1] and CARRA [17] geospatial datasets.
The regions with dynamic motions lead to tearing artifacts in optical flow-
based SuperSloMo [27] but are predicted well with STint.

###  4.4 Domain Transfer to Satellite Imagery

We further study the effectiveness of our proposed dual-cycle consistency for
pre-training and reducing gaps for domain transfer. For this study, we
consider self-supervised pre-training on ERA5 solar fields followed by fine-
tuning on SEN12MS cloud imagery dataset [29]. While solar radiation and cloud
imagery may seem like two distinct domains without a connection, cloud cover
acts as spatial filters impacting solar radiation by altering its spatial
properties [34]. Table 3 shows the results indicating the pre-trained model
fine-tuned on SEN12MS outperforms the one trained only on SEN12MS in a
supervised way with random initialization. This can be attributed to the
vulnerability of supervised methods to performance drops in data-scarce
scenarios. Conversely, unsupervised pre-trained models, having learned from
data-rich scenarios, exhibit adaptability through domain transfer, proving
advantageous in data-poor regions such as with SEN12MS data. While
unsupervised pre-training improves performance for both SuperSloMo [27] and
STint, our proposed STint still outperforms the former, demonstrating that our
method’s optical-flow agnostic property generalizes well to satellite imagery.

####  4.4.1 Domain Adaptation to SEN12MS

In this section, we examine a data-poor scenario of another geospatial data,
that is SEN12MS cloud Imagery (60K frames), where labeled data for
interpolation is very less for training. However, an abundance of data is
available in another domain, that is ERA5 (196K frames). Our model evaluation
takes place on the SEN12MS dataset. For each set of four frames, we utilize
the first and fourth frames as input to predict the second and third frames.
We use the same model as above, but we utilize the model pre-trained on ERA5
to fine-tune it on the SEN12MS dataset for 50 epochs. We use a similar
learning rate and weight decay factor for fine-tuning as described in the
previous experiment. For the model trained only with supervision, we use 3D-U-
Net-SE with random initialization and train it for 200 epochs with a learning
rate of $2\times 10^{-3}$.

| ERA5 $\rightarrow$ SEN12MS | SEN12MS
---|---|---
| PSNR/SSIM | PSNR/SSIM
Baseline | 20.66 /0.513 | 20.66 /0.513
SuperSloMo | 26.71 /0.612 | 24.83 /0.559
Proposed | 28.18 /0.647 | 26.97 /0.594
Table 3: Results of Domain transfer results comparing domain transfer
performance on SEN12MS cloud imagery dataset against model trained in a
supervised manner.

###  4.5 Limitation: Video Data

UCF101
---
| PSNR ($\uparrow$) | SSIM ($\uparrow$)
Baseline | 30.14 | 0.877
SuperSloMo | 34.19 | 0.917
Proposed | 32.52 | 0.882
Table 4: Temporal interpolation results on UCF101 action recognition dataset
with 25 frames-per-second.

We test self-supervised training of our approach against SuperSloMo [27] on
UCF101 dataset [31]. We follow the same procedure from previous experiments of
temporal sub-sampling. For a set of four frames, The first and the fourth
frames are used as input to predict the middle two frames. For each frame, we
use standard normalization. For the purpose of data augmentation, following
[27], we leverage the problem’s symmetry by randomly choosing input sequences
during training and reversing the temporal order of the frames. We pre-train
the model from random initialization for 400 epochs with a batch-size of 16.
We fine-tune the pre-trained model on the train set (75 percent) for 60 epochs
and evaluate the performance on the disjoint test set (25 percent). We report
the average value of the metric over all the predicted frames. We use a
combination of losses from supervision and cycle consistency, where the
weights for cycle consistency are $\gamma_{CC_{1}}=0.4$,
$\gamma_{CC_{2}}=0.25$.

The testing setting remains similar to the previous setting, that is, for
every quadruple, the first and fourth frames are used to reconstruct the
middle two frames. While the spatial dynamics of geospatial datasets might
break the underlying assumptions of optical flow, the same pixel-level motion
dynamics assists in generating visually plausible intermediate frames for
video datasets as can be seen from results in Table 4. Optical flow-based
SuperSloMo [27] achieves a gain in PSNR value of 1.67dB over STint. This
implies that the effectiveness of optical flow-based SuperSloMo [27] in
benefiting temporal interpolation hinges on the validity of optical flow’s
assumptions, which holds for the UCF101 dataset with 25-frames-per-second but
does not hold for geospatial datasets which usually exists at an hour-level-
resolutions.

Figure 6: Performance comparison of STint vs SuperSloMo [27] across datasets.
Optical flow leads to better temporal interpolation for SuperSloMo [27] on the
video dataset of UCF101 [31]. The flow-agnostic property makes STint
outperform SuperSoMo [27] on all the geospatial datasets.

Figure 6 compares the performance of STint and SuperSloMo across the datasets
with different temporal frequencies. STint outperforms in scenarios with lower
temporal granularity, while optical flow-based SuperSloMo benefits from the
information of pixel movements for video data such as UCF101 [31].

###  4.6 Physical Interpretation of cycle consistency

Physical consistency and temporal evolution of geospatial data from our
climate system are vital to understanding how our weather patterns and climate
evolve over time. The climate system evolves over varying timescales, ranging
from hourly to seasonal to centuries or even longer. The nonlinear dynamics of
the system dictate a unique solution for the system’s evolution over time,
given various forcings and prior conditions. The physical consistency of these
unique solutions to the system’s equations ensures that the evolution of the
system is consistent with known physical laws such as mass or energy
conservation. Hence, given the evolution of the system over a chaotic
attractor, the state of the system is over a unique trajectory, and hence,
temporal interpolation of the system over this trajectory gives unique
solutions on the attractor [24]. Ensuring that interpolation models are
temporally coherent and consistent, such as STint is essential in improving
our ability to predict changes in the geospatial systems over various
timescales.

##  5 Discussion

This paper advances the state-of-the-art of temporal interpolation for
geospatial data. The prevalent reliance on optical flow, while effective for
video data, exhibits constraints when applied to geospatial data characterized
by distinct temporal dynamics and movement patterns. In response, we have
introduced a new approach termed Self-supervised Temporal Interpolation
(STint). Unlike conventional techniques, STint does not rely on optical flow
or ground truth data. Its novel dual cycle-consistency framework facilitates
the generation of intermediate frames in a self-supervised manner. Through
extensive experimentation across diverse geospatial datasets, we have
established that STint offers a notable enhancement over existing methods. By
addressing the limitations of traditional approaches, STint contributes
meaningfully to the domain of geospatial temporal interpolation. While our
experiments have been tested by sampling from the original high-frequency
dataset, STint can be simply employed to just temporally interpolate
geospatial data in a self-supervised manner.

Training using cycle consistency has its challenges, notably instability
between batches. While training with a lower learning rate, in addition to
certain regularizers, helps alleviate the instability during training, it
leads to longer training periods. In the future, finding ways to make the
cycle consistency technique more efficient would be useful. Additionally,
since STint does not rely on optical flow, it helps in temporally
interpolating geospatial datasets better, but there are certain scenarios like
video data where motion information is more effective in interpolation.
Moreover, it is also worth exploring the idea of an augmented version of
optical flow that is applicable to geospatial and other scientific data.
Lastly, we believe investigating better architectures for temporal
interpolation will be key to generalizing dual cycle consistency for other
data domains.

## Acknowledgments

This project was partially supported by the Climate Change AI Innovation
Grants program, hosted by Climate Change AI with the support of the Quadrature
Climate Foundation, Schmidt Futures, and the Canada Hub of Future Earth. This
research was also partially funded through a grant provided by the National
Science Foundation, under Award No. 2118285.

## References

  * [1] Olivier Boucher, Jérôme Servonnat, Anna Lea Albright, Olivier Aumont, Yves Balkanski, Vladislav Bastrikov, Slimane Bekki, Rémy Bonnet, Sandrine Bony, Laurent Bopp, et al.  Presentation and evaluation of the ipsl-cm6a-lr climate model.  Journal of Advances in Modeling Earth Systems, 12(7):e2019MS002010, 2020.
  * [2] Thomas Brox, Andrés Bruhn, Nils Papenberg, and Joachim Weickert.  High accuracy optical flow estimation based on a theory for warping.  In Computer Vision-ECCV 2004: 8th European Conference on Computer Vision, Prague, Czech Republic, May 11-14, 2004. Proceedings, Part IV 8, pages 25–36. Springer, 2004.
  * [3] Byung-Tae Choi, Sung-Hee Lee, and Sung-Jea Ko.  New frame rate up-conversion using bi-directional motion estimation.  IEEE Transactions on Consumer Electronics, 46(3):603–609, 2000\.
  * [4] Jinsoo Choi and In So Kweon.  Deep iterative frame interpolation for full-frame video stabilization.  ACM Transactions on Graphics (TOG), 39(1):1–9, 2020.
  * [5] Özgün Çiçek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger.  3d u-net: learning dense volumetric segmentation from sparse annotation.  In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19, pages 424–432. Springer, 2016\.
  * [6] Duolikun Danier, Fan Zhang, and David Bull.  St-mfnet: A spatio-temporal multi-flow network for frame interpolation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3521–3531, 2022.
  * [7] Yonas B Dibike and Paulin Coulibaly.  Temporal neural networks for downscaling climate variability and extremes.  Neural networks, 19(2):135–144, 2006.
  * [8] Ionut Cosmin Duta, Li Liu, Fan Zhu, and Ling Shao.  Pyramidal convolution: Rethinking convolutional neural networks for visual recognition.  arXiv preprint arXiv:2006.11538, 2020.
  * [9] David Fleet and Yair Weiss.  Optical flow estimation.  Handbook of mathematical models in computer vision, pages 237–257, 2006.
  * [10] Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz-Sabater, Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, et al.  The era5 global reanalysis.  Quarterly Journal of the Royal Meteorological Society, 146(730):1999–2049, 2020.
  * [11] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018.
  * [12] Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller, and Jan Kautz.  Super slomo: High quality estimation of multiple intermediate frames for video interpolation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9000–9008, 2018.
  * [13] Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller, and Jan Kautz.  Super slomo: High quality estimation of multiple intermediate frames for video interpolation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9000–9008, 2018.
  * [14] Xin Jin, Longhai Wu, Guotao Shen, Youxin Chen, Jie Chen, Jayoon Koo, and Cheul-hee Hahm.  Enhanced bi-directional motion estimation for video frame interpolation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5049–5057, 2023.
  * [15] Tarun Kalluri, Deepak Pathak, Manmohan Chandraker, and Du Tran.  Flavr: Flow-agnostic video representations for fast frame interpolation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2071–2082, 2023.
  * [16] Hoda Rezaee Kaviani and Shahram Shirani.  Frame rate upconversion using optical flow and patch-based reconstruction.  IEEE Transactions on Circuits and Systems for Video Technology, 26(9):1581–1594, 2015.
  * [17] Morten Køltzow, Harald Schyberg, Eivind Støylen, and Xiaohua Yang.  Value of the copernicus arctic regional reanalysis (carra) in representing near-surface temperature and wind speed in the north-east european arctic.  Polar Research, 41, 2022.
  * [18] Taesam Lee and Changsam Jeong.  Nonparametric statistical temporal downscaling of daily precipitation to hourly precipitation and implications for climate change scenarios.  Journal of Hydrology, 510:182–196, 2014.
  * [19] Yu-Lun Liu, Yi-Tung Liao, Yen-Yu Lin, and Yung-Yu Chuang.  Deep video frame interpolation using cyclic frame generation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8794–8802, 2019.
  * [20] Ziwei Liu, Raymond A Yeh, Xiaoou Tang, Yiming Liu, and Aseem Agarwala.  Video frame synthesis using deep voxel flow.  In Proceedings of the IEEE international conference on computer vision, pages 4463–4471, 2017.
  * [21] Jesper E Nielsen, Søren Thorndahl, and Michael R Rasmussen.  A numerical method to generate high temporal resolution precipitation time series by combining weather radar measurements with a nowcast model.  Atmospheric research, 138:1–12, 2014.
  * [22] Simon Niklaus and Feng Liu.  Softmax splatting for video frame interpolation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5437–5446, 2020.
  * [23] Simon Niklaus, Long Mai, and Feng Liu.  Video frame interpolation via adaptive separable convolution.  In Proceedings of the IEEE international conference on computer vision, pages 261–270, 2017.
  * [24] TN Palmer, A Döring, and G Seregin.  The real butterfly effect.  Nonlinearity, 27(9):R123, 2014.
  * [25] Junheum Park, Chul Lee, and Chang-Su Kim.  Asymmetric bilateral motion estimation for video frame interpolation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14539–14548, 2021.
  * [26] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [27] Fitsum A Reda, Deqing Sun, Aysegul Dundar, Mohammad Shoeybi, Guilin Liu, Kevin J Shih, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  Unsupervised video interpolation using cycle consistency.  In Proceedings of the IEEE/CVF international conference on computer Vision, pages 892–900, 2019.
  * [28] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
  * [29] Michael Schmitt, Lloyd Haydn Hughes, Chunping Qiu, and Xiao Xiang Zhu.  Sen12ms–a curated dataset of georeferenced multi-spectral sentinel-1/2 imagery for deep learning and data fusion.  arXiv preprint arXiv:1906.07789, 2019.
  * [30] Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo.  Convolutional lstm network: A machine learning approach for precipitation nowcasting.  Advances in neural information processing systems, 28, 2015.
  * [31] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  A dataset of 101 human action classes from videos in the wild.  Center for Research in Computer Vision, 2(11), 2012.
  * [32] Deqing Sun, Stefan Roth, and Michael J Black.  Secrets of optical flow estimation and their principles.  In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 2432–2439. IEEE, 2010.
  * [33] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri.  A closer look at spatiotemporal convolutions for action recognition.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6450–6459, 2018.
  * [34] P Tzoumanikas, E Nikitidou, AF Bais, and A Kazantzidis.  The effect of clouds on surface solar irradiance, based on data from an all-sky imaging system.  Renewable energy, 95:314–322, 2016.
  * [35] Thomas J Vandal and Ramakrishna R Nemani.  Temporal interpolation of geostationary satellite imagery with optical flow.  IEEE Transactions on Neural Networks and Learning Systems, 2021\.
  * [36] Cort J Willmott, Steven G Ackleson, Robert E Davis, Johannes J Feddema, Katherine M Klink, David R Legates, James O’donnell, and Clinton M Rowe.  Statistics for the evaluation and comparison of models.  Journal of Geophysical Research: Oceans, 90(C5):8995–9005, 1985\.
  * [37] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.