# Spatial-Temporal Transformer based Video
Compression Framework

Yanbo Gao, Wenjia Huang, Shuai Li*, , Hui Yuan, ,
Mao Ye, Siwei Ma This work was supported in part by the National Key R&D
Program of China under Grant 2018YFE0203900; in part by the National Natural
Science Foundation of China under Grant 62271290 and Grant 62001092; in part
by the Natural Science Foundation of Shandong Province under Grant ZR2022ZD38
and in part by SDU QILU Young Scholars Program. Y. Gao and W. Huang are with
School of Software, Shandong University, and S. Li and H. Yuan are with School
of Control Science and Engineering, Shandong University, Jinan, China. E-mail:
<EMAIL_ADDRESS>M. Ye is with School of Computer Science and
Engineering, University of Electronic Science and Technology of China,
Chengdu, China. E-mail<EMAIL_ADDRESS>Ma is with National
Engineering Research Center of Visual Technology, School of Computer Science,
Peking University, Beijing, China. E-mail<EMAIL_ADDRESS>received
Sep. 21, 2023.

###### Abstract

Learned video compression (LVC) has witnessed remarkable advancements in
recent years. Similar as the traditional video coding, LVC inherits motion
estimation/compensation, residual coding and other modules, all of which are
implemented with neural networks (NNs). However, within the framework of NNs
and its training mechanism using gradient backpropagation, most existing works
often struggle to consistently generate stable motion information, which is in
the form of geometric features, from the input color features. Moreover, the
modules such as the inter-prediction and residual coding are independent from
each other, making it inefficient to fully reduce the spatial-temporal
redundancy. To address the above problems, in this paper, we propose a novel
Spatial-Temporal Transformer based Video Compression (STT-VC) framework. It
contains a Relaxed Deformable Transformer (RDT) with Uformer based offsets
estimation for motion estimation and compensation, a Multi-Granularity
Prediction (MGP) module based on multi-reference frames for prediction
refinement, and a Spatial Feature Distribution prior based Transformer (SFD-T)
for efficient temporal-spatial joint residual compression. Specifically, RDT
is developed to stably estimate the motion information between frames by
thoroughly investigating the relationship between the similarity based
geometric motion feature extraction and self-attention. MGP is designed to
fuse the multi-reference frame information by effectively exploring the
coarse-grained prediction feature generated with the coded motion information.
SFD-T is to compress the residual information by jointly exploring the spatial
feature distributions in both residual and temporal prediction to further
reduce the spatial-temporal redundancy. Experimental results demonstrate that
our method achieves the best result with 13.5% BD-Rate saving over VTM and
68.7% BD-Rate saving over the baseline without the proposed modules. Ablation
study validates the effectiveness of each proposed module.

###### Index Terms:

Transformer, Inter-prediction, Learned video compression

##  I Introduction

Video data has experienced an exponential growth with the proliferation of
video-sharing platforms and increasing high-resolution videos, resulting in
the urgent need of more efficient video compression. Video compression is to
reduce the spatial and temporal redundancy with intra/inter prediction,
transform coding, quantization, and entropy coding, in order to achieve high
compression ratios while maintaining perceptual quality.

Traditional block-based video coding approaches have been widely studied with
predefined intra/inter predictions such as the angular intra prediction and
motion estimation/compensation, predefined transform coding such as the DCT,
and predefined entropy coding such as the CABAC [1]. Such block-based video
coding architecture with predefined modules has enjoyed a great success in the
past decades and widely adopted in the industry. With the rapid development of
deep learning, there has been a growing interest in exploring new video
compression methods based on deep neural networks. These methods aim to
leverage the powerful representation learning capabilities of deep models to
learn adaptive transforms instead of using predefined ones, in order to
achieve higher compression efficiency. Various approaches have been proposed
[2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20] including
deep learning enhanced video coding by replacing some modules in the
traditional video coding method and learned video coding approach with a whole
neural network to compress a video. This paper focuses on the learned video
coding, especially learned inter-frame video coding. Existing learned inter-
frame video coding approach generally takes a similar process as the
traditional video coding, including inter-frame prediction based on motion
estimation/compensation, residual compression and entropy coding. While many
learned methods [21, 3] have been developed and achieve state-of-the-art
performance as traditional video coding method, there still exist two key
problems in the exploration of the inter-frame information.

Firstly, motion information as geometric information, used to align the
reference frame to the current frame in order to perform the inter-frame
prediction, is difficult to be stably transformed from the color space, i.e.,
the image frame and its corresponding feature. Current alignment methods
mainly use optical flow and offsets as motion, and both are dense geometric
transformation representations. Learning from color representations based on
gradient backpropagation to generate such geometric feature is usually not
stable as illustrated in [2]. Moreover, existing methods mostly only employ
the immediate previous frame for prediction without fully exploring the
multiple reference frames.

Secondly, after the inter-frame prediction, the residual is compressed
independently from the inter-frame prediction information, neglecting the
useful spatial information in the prediction. To be specific, other than
subtracting the corresponding point-to-point prediction information as a
temporal prediction, the spatial relationship embedded in the prediction can
also assist the spatial compression of the residual.

To address the above problems, a spatial-temporal Transformer based inter-
frame video coding framework is proposed in this paper. First, a Multi-
Granularity Prediction generation with proposed Relaxed Deformable Transformer
is developed, where the multi-reference frame information is fully explored.
Then a Spatial Feature Distribution prior based Transformer (SFD-T) is
proposed to utilize the spatial feature distribution prior embedded in the
temporal prediction feature to reduce the remaining spatial redundancy in the
residual. The contributions of this paper can be summarized as follows.

  * •

We propose a Relaxed Deformable Transformer (RDT) based motion estimation and
compensation module, where the RDT transforms the color feature with the
spatial position embedding to generate the geometric motion alignment
information and produces a coarse prediction feature. The mechanism of using
RDT for producing motion between two frames is thoroughly investigated, and
the deformable transformer is relaxed to the deformable convolution with their
relationship carefully examined.

  * •

We propose a Multi-Granularity Prediction (MGP) based multi-reference
prediction refinement module. The multi-reference frame information is
explored in the manner of video denoising with the coarse prediction feature
as anchor, since it is obtained with coded motion information and thus
contains most information of the current frame.

  * •

We propose a Spatial Feature Distribution prior based Transformer (SFD-T)
module to compress the residuals by further exploring the spatial feature
distribution information embedded in the temporal prediction.

The above modules are all constructed in the form of transformer and comprises
a complete Transformer based video coding framework. Extensive experiments
demonstrate that our method achieves state-of-the-art results, and ablation
studies have also been conducted to validate the proposed modules.

The rest of this paper is organized as follows. Section II presents the
related works in learned image and video compression. Section III describes
the proposed method in details. Experimental results with ablation studies are
provided in Section IV and conclusions are drawn in Section V.

##  II Related Work

In this section, a brief review of the related work in the field of learned
video compression [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
19, 20] is presented. Considering that image compression usually serves to
code the first frame of a video and the spatial motion and residual used in
the video coding, some related learned image compression methods are reviewed
first.

###  II-A Learned Image Compression

Existing learned image compression methods are mostly derived from Variational
Autoencoder (VAE) [22], which uses an Encoder-Decoder architecture with the
quantization process mimicked as the variational process. Most of the encoders
and decoders adopt the convolutional neural networks (CNNs) [23] with down-
sampling and up-sampling in the encoder and decoder, respectively. The input
is first transformed into the latent representation with the encoder, and
quantized by rounding which is replaced by adding a uniform-distributed noise
for gradient backpropagation at training [24]. The quantized residual is then
encoded into the bitstream with the entropy coding and transmitted to the
decoder [2, 22, 25]. To enhance the performance of entropy coding, hyperprior
and context based entropy coding methods [25, 26, 27] were developed and
widely used in the following learned image and video compression.

With the rapid development of Transformers [28, 29], Transformer based image
compression has also been studied [25, 26, 27]. A CNN-Transformer hybrid
framework of image compression was proposed in [25], where the Swin
Transformer blocks are inserted between convolutional layers without changing
the overall architecture. In [26], a symmetric Transformer framework is used
by replacing all CNN layers in the encoder-decoder architecture with
Transformer layers. In addition to exploring Transformer for the encoder and
decoder, there are also works such as ContextFormer [27] and Entroformer [30]
investigating using Transformer for entropy model.

These researches focus on reducing the spatial redundancy within an image, and
can also be used in video coding to compress the motion vectors (if exist) and
residuals. However, these models are developed without considering the
temporal information and cannot directly using them for video coding.

###  II-B Learned Video Compression

Compared with the image compression, learned video compression focuses more on
the inter-frame predictive coding. Most of the existing methods adopt a
similar procedure as the conventional hybrid video coding, and consist of
motion estimation and compensation [4, 5, 6, 8], residual coding and filtering
[19, 7]. Deep video compression (DVC) [31] first proposed a fully learned
video compression, using optical flow and warping to represent motion and
perform motion compensation to generate prediction frame. Then residual is
obtained by subtracting the current frame with the prediction frame, and
further coded with an autoencoder (similarly as the image compression). Many
methods have been developed with a similar procedure. In [2], FVC proposed to
perform the inter-prediction in the feature domain using deformable
convolution to reduce the artifacts around edges brought by the optical flow.
Deep Contextual Video Compression (DCVC) [20], and its variants, including
DCVC-TCM [18], DCVC-HEM [21] and DCVC-DC [3], were proposed to use context
models for spatial compression. It directly processes the concatenated
prediction and the current frame without explicitly obtaining the residual,
using the property of conditional entropy no larger than the entropy of the
residual.

In addition to the single-reference based prediction methods described above,
there also exist methods using multi-scale or multi-reference frame strategy
[5, 6, 4] to help with prediction generation. In [7], the multi-reference
frames are first gradually warped to the current frame with the coded motion
vectors at each time step, and then the warped multi-reference frames are
fused together to generate the final prediction. 3D convolution is used in [8]
to fuse the initial prediction feature and the multi-reference frames without
temporal warping. Other than explicitly fusing multi-reference frames at each
frame, implicit temporal information aggregation with neural architectures
such as LSTM [32] have also been investigated for video coding [9]. In [10],
Conv-LSTM is used together with the U-net architecture to directly learn the
space-time difference between video frames. LSTM have also been explored to
construct the context models for the hyperprior based entropy coding [7, 11].

Figure 1: Framework of the proposed STT-VC.

The above methods all employ the CNNs based encoding architecture. On the
other hand, while there are some Transformer based spatial-temporal modelling
methods, there only exist few Transformer based video coding methods. VCT [12]
directly used Transformer to model temporal dependencies and predict the
distribution of current frame without using explicit motion representation.
Autoregressive prediction within each frame is also used similarly as the
autoregressive context model [33]. Some methods also adopt the self-attention
module in part of the video coding framework such as in encoder and decoder
[13] and quality enhancement [14, 19].

In the area of Transformer based temporal modelling, several representative
modelling methods are briefly described in the following. ViVit [15] proposed
a Transformer based video processing method by factorizing the model into
spatial and temporal processing. For the temporal processing, patches of the
same spatial position are grouped with a temporal position embedding to
synthesize three-dimensional tokens and thus can be processed in the same way
as the spatial Transformer. Video SwinTransformer [16] extended the Swin-T
[34] to three dimensions for video processing, where a 3D shifted window over
a video is used for the multi-head self-attention calculation. CRAFT [35] was
developed for optical flow estimation based on cross-attention between two
frames. VSRT [17] proposed a spatially integrated convolutional self-attention
layer to enhance the extraction of spatial locality information and use an
optical flow based temporal alignment to enhance temporal information
modelling. Deformable Transformer was first proposed for spatial object
detection [36], where the offsets and the weights are generate by a query
feature and then the sampled features are fused together similarly as the
deformable convolution. Then Deformable Attention Transformer (DAT) [37] was
proposed, where, after the offsets are generated, the sampled features are
fused together using the self-attention model. However, such Transformer based
temporal modelling methods do not concern the special needs of video coding
such as the balance between rate and distortion, and cannot be directly used
for video coding.

##  III Proposed Method

###  III-A Overview

This paper proposes a learned inter-frame video coding method to generate a
high-quality prediction and a compactly coded residual, with a Transformer
based framework. It mainly consists of three components, including Relaxed
Deformable Transformer (RDT) for motion estimation and compensation, Multi-
Granularity Prediction (MGP) for prediction refinement, and Spatial Feature
Distribution prior based Transformer (SFD-T) for residual compression. The
framework of the proposed method is illustrated in Fig. 1.

The current frame ${X_{t}}$ and reference frame ${\hat{X}_{t-1}}$ are first
transformed to features ${F_{t}}$ and ${\hat{F}}_{t-1}$, respectively, and the
following prediction and residual compression are performed in the feature
domain. 1) RDT first uses Uformer to conduct the motion estimation (shown in
the light blue box of Fig. 1), which is then coded with a motion vector codec
and finally produces the motion information ${\widehat{mv}}_{off}$ and
attention scores/confidence mask ${\hat{M}}_{c}$. Then the deformable
Transformer based value feature fusion process is relaxed to the deformable
convolution to generate a coarse-grained prediction feature ${F_{c-pre}}$. 2)
MGP is further used to refine the prediction feature to explore the multi-
reference frame information (shown in the light green box of Fig. 1). MGP
applies RDT to align the multi-reference frame features to the coarse-grained
prediction feature ${F_{c-pre}}$, and fuses them through a spatial-channel
attention to generate enhanced prediction feature ${F_{m-pre}}$. The residuals
${F_{resi}}$ are generated by subtracting the enhanced prediction feature
${F_{m-pre}}$ from the current frame feature ${F_{t}}$. 3) SFD-T compresses
the residual ${F_{resi}}$ (shown in the light gray box of Fig. 1), by
exploring the enhanced prediction feature ${F_{m-pre}}$ in the attention
calculation process as a spatial feature distribution prior. Finally, the
decoded residual feature ${{\hat{F}}_{resi}}$ is added with the enhanced
prediction feature ${F_{m-pre}}$ to reconstruct input feature
${{\hat{F}}_{t}}$, which is then further transformed back to pixel domain as
the reconstructed frame ${{\hat{X}}_{t}}$. The three proposed modules RDT, MGP
and SFD-T are presented in detail in the following subsections.

Figure 2: (a) Structure of the proposed RDT based motion estimation and
compensation. (b) Details of the Lewin transformer layer with W-MSA [34] used
in RDT.

###  III-B Relaxed Deformable Transformer (RDT) for Motion Estimation and
Compensation

To perform inter-frame prediction, the motion information between the current
frame and the reference frame needs to be estimated first, known as motion
estimation. Then the reference frame is aligned to the current frame with the
estimated motion information to generate prediction, known as motion
compensation. Currently, warping with optical flow and using deformable
convolution [38] with offsets are the two main approaches for motion
estimation and compensation. It is known that utilizing optical flow for
feature alignment during motion compensation often results in artifacts around
the edges of objects. Therefore, in this paper, offsets with the deformable
convolution (DConv) is used as the base to represent the motion and perform
motion compensation between the reference frame and the current frame.

The existing offsets and DConv based motion estimation and compensation
methods usually estimate the offsets by gradually convolving the features of
the reference and current frames using a CNN. However, the offsets, in the
form of the geometric information, is difficult to be directly obtained from
the color features via gradient backpropagation. On the other hand, motion
estimation and compensation estimates the motion changes between the reference
frame and the current frame based on their similarity, and then aligns the
reference frame with the motion to ensure its similarity to the current frame.
Essentially this process is conducted based on the similarity between the
reference frame and the current frame. Therefore, to overcome the problem of
stably obtaining similarity based geometric motion information from color
features, a relaxed deformable transformer (RDT) is developed. A Uformer is
first used to estimate the offsets and attention/confidence of each offsets
based on the similarity, and a deformable convolution is then used as a
relaxed deformable Transformer to fuse the value features according to the
corresponding offsets and attention/confidence.

First, the current frame and the reconstructed reference frame are transformed
into the feature domain through a feature extraction module. Taking the
processing of the current frame $X_{t}$ as an example, it can be represented
by

$\displaystyle F_{conv}=ReLU({Conv}_{5\times 5}(X_{t}))$ $\displaystyle
F_{t}=ResBlocks(F_{conv})+F_{conv}$ (1)

where $F_{t}$ represents the final feature of $X_{t}$. $ResBlocks$ represent
three Resblocks, each of which consists two convolution modules and a skip
connection.

To calculate the similarity between the features of the reference frame and
current frame, instead of using cross-attention [35], we find that simply
concatenating the features and processing them together with the self-
attention can better cover the similarity calculation among different
positions of the two features and within each feature. Specifically, the two
frame features $F_{t}$ and ${\hat{F}}_{t-1}$ are concatenated together, and
processed with a ${1\times 1}$ convolution to fuse the information and reduce
the channels as

$\displaystyle F_{c}={Conv}_{1\times 1}\langle F_{t},{\hat{F}_{t-1}}\rangle$
(2)

where $\left\langle.\right\rangle$ is used to represent the concatenation
operation for simplicity. Then the spatial position embedding is incorporated
into the features and processed by the self-attention model as

$\displaystyle F_{s}=W_{MSA}(LN(F_{c}+F_{pos}))$ $\displaystyle
W_{MSA}=softmax({QK}^{T}/\sqrt{d}+B)V$ (3)

where ${W_{MSA}}$ represents the calculation of window-based multi-head self-
attention [34], and ${LN}$ represents the normalization layer. ${F_{pos}}$
represents the spatial position embedding, which is obtained by embedding the
absolute position within a block. ${Q,K,V}$ represent the query, key and value
features in the calculation of multi-head self-attention, obtained by
conducting linear projection to features at each layer (${Q,K,V=\
Linear(LN(F_{c}+F_{pos}))}$), and $B$ represents relative position bias as in
[34].

Since the concatenated feature ${F_{c}}$ contains information from both
${F_{t}}$ and ${F_{t-1}}$, the calculation of attention scores ${(Q\cdot K)}$
before normalization between features of two positions ${\left(x,y\right)}$
and ${(x+\Delta x,y+\Delta y)}$ can be formulated as the following Eq. (4) by
substituting ${F_{c}}$ with Eq. (2) and ignoring the position embedding first.

$\displaystyle F(x,$ $\displaystyle y)\cdot F(x+\Delta x,y+\Delta
y)=[f_{a}(F_{t}(x,y))+f_{b}(F_{t-1}(x,y))]$
$\displaystyle\cdot[f_{a}(F_{t}(x+\Delta x,y+\Delta y))+f_{b}(F_{t-1}(x+\Delta
x,y+\Delta y))]$ (4)

where ${f_{a}}$ and ${f_{b}}$ represent the information mixing of ${F_{t}}$
and ${F_{t-1}}$ in the ${1\times 1}$ convolution in Eq. (2). Note that here we
do not concern the detailed function of ${f_{a}}$ and ${f_{b}}$ and simply use
them to represent the information fusion process. Eq. (4) can be further
turned into

$\displaystyle F\left(x,y\right)\cdot$ $\displaystyle\ F(x+\Delta x,y+\Delta
y)=$ $\displaystyle\quad\ f_{a}(F_{t}(x,y))\cdot f_{a}(F_{t}(x+\Delta
x,y+\Delta y))$ $\displaystyle+f_{a}(F_{t}(x,y))\cdot f_{b}(F_{t-1}(x+\Delta
x,y+\Delta y))$ $\displaystyle+f_{b}(F_{t-1}(x,y))\cdot f_{a}(F_{t}(x+\Delta
x,y+\Delta y))$ $\displaystyle+f_{b}(F_{t-1}(x,y))\cdot f_{b}(F_{t-1}(x+\Delta
x,y+\Delta y))$ (5)

It can be seen that the self-attention score calculation on the fused features
covers the estimation of the similarity not only within each feature but also
between features, which is important for motion estimation.

On the other hand, position embedding introduces the spatial geometric
information into the features. In the above self-attention score calculation,
with the position embedding, it can learn to also consider the geometric
distance among different positions in addition to the feature similarity. More
importantly, it helps directly transfer the color features into geometric
information. After the attention score calculation, the value features ($V$)
are fused together based on the attention score. Since $V$ is obtained with
the position embedding, the final feature can be represented as

$\displaystyle F_{s}=\sum_{i\in B}\alpha_{i}V(F_{c}+F_{pos})$ (6)

where ${i\in B}$ represents the block (${\\{8\times 8\\}}$ in the experiments)
of the self-attention calculation. It can be seen that by incorporating the
similarity information contained in the attention score and the geometric
information contained in the value feature, the output feature directly
contains the desired geometric position information based on the similarity
among features. This agrees with the motion estimation based on the similarity
as discussed in the beginning of this subsection. Thus, the motion offsets
between two features can be stably obtained by the above self-attention.

To obtain the motion information which distributes in a large range from small
motion of a fractional pixel to large motion of a few pixels, the Uformer
architecture [39] is used. It directly calculates the self-attention based on
pixel-wise features within a block instead of patchifying them, thus able to
obtain detailed motion. On the other hand, to achieve the large motion based
on global information, U-structure is used by down-sampling and up-sampling
the features, and also concatenating the encoder feature to the output with a
skip connection. In each layer of the Uformer, in addition to the self-
attention calculation as in Eq. (3) the input feature is also added to the
output feature ${F_{a}=F_{s}+F_{c}}$. Then a few convolutional layers are
further used to increase the local processing of the features, known as
locally-enhanced window (LeWin) Transformer layers [39].

$\displaystyle F_{l}={{Conv}_{1\times 1}(DWConv}_{3\times 3}({Conv}_{1\times
1}(LN(F_{a}))))+F_{a}$ (7)

where $DWConv$ represents depth-wise convolution. By using LeWin Transformer
layers, it effectively models the relationship between pixels in the reference
frame and the current frame within each window. The other layers in the
U-architecture use the same form and produce the final output feature $F_{o}$
as shown in Fig. 2. Finally, a $1\times 1$ convolution can be used in the end
to transform the output feature into motion offsets between the current frame
and the reference frame. Since in video coding the motion information needs to
be compressed into the bitstream and transferred to the decoder, the final
feature of Uformer, instead of the motion offsets, is compressed with an
encoder-decoder codec and then transformed into the motion offsets with the
decoder feature. This motion encoding process can be expressed as

$\displaystyle F_{mv-o}=Enc(F_{o})$ $\displaystyle
b_{mv}=Entropy({f_{Q}(F}_{mv-o}))$
$\displaystyle{\widehat{mv}}_{off},{\hat{M}}_{c}={Conv}_{1\times
1}(Dec({\hat{F}}_{mv-o}))$ (8)

Figure 3: (a) Structure of the proposed MGP. (b) Details of the fusion model
used in MGP.

where $Enc$ and $Dec$ represent the encoder and decoder in the motion codec,
respectively. Any motion codec can be used and here for simplicity, the motion
codec in FVC [2] is used. $f_{Q}$ represents the quantization process
generating the quantized feature ${\hat{F}}_{mv-o}$, and $Entropy$ refers to
the entropy coding model. Finally, the decoded features are processed with a
$1\times 1$ convolution to generate the reconstructed motion offsets
${{\widehat{mv}}_{off}}$. Unlike the conventional deformable transformer that
uses the deformed features indicated by the motion offsets to enhance the
current feature with the self-attention operation [37], here the deformed
features are fused together to generate a prediction of the current feature.
Therefore, an attention mask ${{\hat{M}}_{c}}$ is directly obtained together
with the motion offsets as a relaxed version of self-attention.

$\displaystyle F_{c-pre}=\sum_{i\in\\{3\ast 3\\}}$
$\displaystyle{{{\hat{M}}_{c}(i)Conv}_{1\times
1}({\hat{F}}_{t-1}((x_{i},y_{i})+{\widehat{mv}}_{off}(i)))}$
$\displaystyle{\xrightarrow{relax}}{DConv}_{3\times
3}({\hat{F}}_{t-1},{\widehat{mv}}_{off},{\hat{M}}_{c})$ (9)

where ${{\hat{F}}_{t-1}(\left(x_{i},y_{i}\right)+{\widehat{mv}}_{off}(i))}$
represents the deformed reference frame features (bilinearly interpolated),
and ${Conv}_{1\times 1}$ is $1\times 1$ convolution to generate the value
feature. Summarizing the value features with the attention
${\hat{M}}_{c}\left(i\right)$ over the block produces the final prediction
feature. This process is equivalent to a deformable convolution (DConv) with
shared filter weights over the $3\times 3$ locations, where
${\widehat{mv}}_{off}$ and ${\hat{M}}_{c}$ represent the offsets and
confidence mask, respectively, and the multi-head number in the transformer is
the group number in the DConv. For generality, we further relax the shared
embedding weights to non-shared weights as convolution, and thus turning it to
a deformable convolution. This process, deformable convolution with offsets
and confidence mask obtained with a Uformer, is thus coined to relaxed
deformable transformer (RDT), unifying the motion estimation and compensation
in the realm of Transformer.

###  III-C Multi-granularity prediction (MGP) feature generation based on
Multi-frame Enhancement

After motion estimation and compensation, a coarse- grained prediction feature
is generated. However, using only the immediately previous frame cannot
provide accurate prediction of the current frame, especially for moving areas
and occluded background. Predicting such information requires the long-range
temporal information which can be partly provided by the multi-reference
frames. With the pixel-wise motion representation for prediction generation in
the existing LVC framework, it is difficult to directly generate a prediction
by mixing the motion from multi-reference frames, or explore the multi-
reference frames without significantly increasing the rate on the motion
representation. On the other hand, the motion representation is obtained under
the rate-distortion optimization (RDO), where, instead of obtaining a motion
representation of a high rate to provide the best-quality prediction feature,
a motion representation with the smallest RD cost is used to provide a decent-
quality prediction with a relatively small rate. Therefore, the motion
representation is suboptimal only considering the quality of prediction
feature.

To solve the above problems, a multi-granularity prediction generation method
is proposed to fully explore the multi-reference frames. It is developed based
on the observation that the coarse-grained prediction feature, generated with
the motion offsets encoded to the bitstream, contains much information of the
current frame and can be considered as a noisy version of the current frame.
Unlike the previous multi-reference prediction methods focusing on predicting
the motion vectors of multi-reference frames or directly fusing the past
multiple prediction features based on the past noisy motion information, we
propose to take the coarse-grained prediction feature as an approximate of the
current frame feature. Accordingly, the multi-reference frames are explored in
a manner of video denoising where the motion information used in the process
can be obtained at both encoder and decoder without coding into bitstream.

Figure 4: Structure of the proposed SFD-Transformer based residual
compression.

The features of the multiple reference frames, including the immediately
previous frame feature ${\hat{F}}_{t-1}$, are first temporally aligned to the
prediction feature $F_{c-pre}$ and then fused together with the prediction
feature to improve its quality. Taking ${\hat{F}}_{t-1}$ as an example, the
temporal alignment process is similar to the above coarse-grained prediction
feature generation:

$\displaystyle F_{f-pre1}=RDT(\langle F_{c-pre},\hat{F}_{t-1}\rangle)$ (10)

where $RDT$ represents the above Relaxed Deformable Transformer. Note that
here the motion representation is not coded with quantization and entropy
coding since the coarse-grained prediction feature $F_{c-pre}$ and the
reference frame feature ${\hat{F}}_{t-1}$ are available at both encoder and
decoder. Therefore, the motion representation encoded under RDO in the coarse-
grained prediction feature generation only needs to describe the coarse motion
to save bits, while the detailed motion representation and prediction feature
can be further generated with this bit-free temporal alignment and enhancement
process. The same process is applied to the other reference features to
explore the long-range temporal information. In the experiments, three
reference frames are used with features ${\hat{F}}_{t-1}$, ${\hat{F}}_{t-2}$
and ${\hat{F}}_{t-3}$, and corresponding enhanced features using RDT are
denoted by $F_{f-pre1}$, $F_{f-pre2}$ and $F_{f-pre3}$, respectively. Such
enhanced features can be considered as fine-grained prediction feature, which
is generated without the rate cost of the motion representation.

The final prediction feature is obtained by fusing the coarse-grained
prediction feature and the multi-reference fine-grained prediction feature. It
is known that frames that are closer in the temporal dimension tend to have
higher similarity. As a result, each reference feature contributes differently
to the prediction feature, indicating that the importance of $F_{f-pre(i)}$
varies. Moreover, in the spatial domain, the distribution of image details
differs between flat regions and sharp edges, and also between moving regions
and background, leading to variations in the spatial fusion of the different
features. To address this, a spatial-channel attention similar as CBAM [40] is
used to fuse the features with different channel and spatial weights. This
fusion process is illustrated in Fig. 3 and can be described as

$\displaystyle F_{enh\\_cat}$ $\displaystyle=\langle
F_{c-pre},F_{f-pre1},F_{f-pre2},F_{f-pre3}\rangle$ $\displaystyle
F_{attn\\_ch}$ $\displaystyle=C\\_attn(F_{enh\\_cat})\cdot F_{enh\\_cat}$
$\displaystyle F_{enh\\_conv}$ $\displaystyle=ReLU({Conv}_{1\times
1}(F_{attn\\_ch}))$ $\displaystyle F_{attn\\_sp}$
$\displaystyle=S\\_attn(F_{enh\\_conv})\cdot F_{enh\\_conv}$ $\displaystyle
F_{m-pre}$ $\displaystyle=F_{c-pre}+F_{attn\\_sp}$ (11)

where $C\\_attn$ and $S\\_attn$ represents the channel and spatial attention
in CBAM [40], respectively. The $1\times 1$ convolution reduces the channel
number of $F_{attn\\_ch}$ to the same as $F_{c-pre}$. The final prediction
feature $F_{m-pre}$ is obtained by adding the coarse-grained prediction
feature and the fused feature since the quality of the coarse-grained
prediction feature is rather stable with the estimated and encoded motion
representation. Finally, the residual feature is obtained by subtracting the
current frame feature with the enhanced prediction feature
$F_{resi}{=F}_{t}-F_{m-pre}$.

###  III-D Spatial Feature Distribution (SFD) Prior based Residual Compression

After the residuals generated with the inter-prediction, residual compression
is further performed to remove the spatial redundancy within the residual.
Existing LVC methods preform residual compression completely in the spatial
dimension, neglecting the inherent spatial feature distribution information
that may be contained in the prediction frame. Taking an image with spatial
repetitive patterns as an example, the spatial feature distribution in the
residual, including the feature similarity at different locations, still
resembles the distribution of the prediction features. In other words, the
current inter-prediction only removes the pixel-to-pixel temporal redundancy
with the subtraction operation or locally processed context operation, while
the redundancy of the spatial feature distribution between temporal frames is
ignored and can be further reduced. Therefore, a Spatial Feature Distribution
prior based Transformer (SFD-Transformer) is developed for residual
compression, where the spatial prior presented in the prediction feature is
used to guide the self-attention model in compressing the spatial residual
features.

TABLE I: Result comparison in terms of BD-Rate (%) measured with PSNR. The anchor is VTM. Method | UVG | MCL-JCV | HEVC B | HEVC C | HEVC D | HEVC E | Average
---|---|---|---|---|---|---|---
VTM-17.0 [41] | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
HM-16.25 [42] | 36.4 | 41.5 | 38.8 | 36.0 | 33.7 | 44.0 | 38.4
ECM-5.0 [43] | -10.0 | -12.2 | -11.5 | -13.4 | -13.5 | -10.9 | -11.92
CANF-VC [44] | 73.0 | 70.8 | 64.4 | 76.2 | 63.1 | 118.0 | 77.6
DCVC [20] | 166.1 | 121.6 | 123.2 | 143.2 | 98.0 | 266.1 | 153.0
DCVC-TCM [18] | 44.1 | 51.0 | 40.2 | 66.3 | 37.0 | 82.7 | 53.6
DCVC-HEM [21] | 1.1 | 8.6 | 5.1 | 22.2 | 2.4 | 20.5 | 10.0
FVC [2] | 155.0 | 171.6 | 176.5 | 182.41 | 164.7 | 273.9 | 171.4
PROPOSED | -20.4 | -16.4 | -10.1 | -16.0 | -2.2 | -15.8 | -13.5

The framework of the proposed SFD-Transformer is shown in Fig. 4. An encoder-
decoder Transformer architecture is used for the residual encoding and
decoding. Firstly, at encoder, the residual is divided into patches, and
embedded through a linear layer. Then the SFD prior based self-attention is
calculated. Specifically, in the calculation of self-attention scores, the
relationship among the prediction features is also considered via a feature
distance as

$\displaystyle{\rm Attn}_{st-r}=softmax(Q_{R}K_{R}+{\rm
pos}_{b}+Q_{p}K_{p}+mod)V_{R}$ (12)

where $Q_{R}$ and $K_{R}$ represent the query and key features of the current
frame, respectively, while $Q_{P}$ and $K_{P}$ represent those of the
corresponding prediction features, obtained with the same linear embedding
functions. The similarity, i.e., $Q_{P}$ $K_{P}$, is served as the spatial
feature distribution prior. ${\rm pos}_{b}$ and $mod$ represent the two-
dimensional relative position coding and the learnable adjustment modulator,
respectively. This updated self-attention calculation is used in all the
Transformer layers at both encoder and decoder. $Q_{p}$ and $K_{p}$ at decoder
are the same to the encoder, providing the same spatial feature distribution
information for decoding the residual.

With reference to the existing Transformer based image compression model
configuration [26], the number of self-attention blocks of each layer at the
encoder is set to [x2,x2,x6,x2]. The resolution is reduced by down-sampling
after each stage. It can be expressed as

$\displaystyle{F_{resi(i)}}=f_{PM}({\rm
Attn}_{st-r(i-1)}(F_{resi\left(i-1\right)},F_{m-pre(i-1)}))$ (13)

where ${\rm Attn}_{st-r\left(i-1\right)}()$ refers to the SFD prior based
self-attention calculation operation of the ($i-1$)th layer. $f_{PM}$
represents the patch merging to down-sample the feature in the spatial
resolution [34], by reconstructing the feature between patches to be half in
size and double in channel. Accordingly, $Q_{p}$ and $K_{p}$ are also down-
sampled layer by layer with linear projection to match the size of
corresponding residual feature maps, so that the SFD prior fits in the self-
attention calculation of each layer. Take $Q_{p(i)}$ as an example,

$\displaystyle Q_{p(i)}=LN(Linear(Reshape(Q_{p(i-1)},\frac{1}{2})))$ (14)

where $Reshape(Q_{p(i-1)},\frac{1}{2})$ as in [45] represents reshaping the
input $Q_{p(i-1)}$ to the size $\frac{HW}{4}\times\left(4C\right)$, and
$Linear()$ is a linear projection that reduces channel number. $K_{p(i)}$ is
processed in the same way.

The encoded feature $F_{resi(i)}$ after the encoder then undergoes entropy
encoding to generate the bitstream.

$b_{resi}=Entropy({f_{Q}(F}_{resi-o}))$ (15)

The bitstream is transmitted to the decoder and entropy decoding is performed
to generate the initial residual feature. Then SFD prior based Transformer
layers are used to further decode the initial residual features to reconstruct
the residual features of the original resolution ${\hat{F}}_{resi}$. The
prediction feature $F_{m-pre}$ is then added back to reconstruct the input
feature.

${\widetilde{F}}_{t}=F_{m-pre}+{\hat{F}}_{resi}$ (16)

Similarly as FVC [2], non-local attention mechanism is further used to enhance
the reconstructed feature with the multi-reference reconstructed features.
Finally, the enhanced reconstructed feature is transformed back to the pixel
domain to generate the reconstructed frame ${\hat{X}}$ with a few ResBlocks in
the same way as in FVC.

The loss function of the proposed method is

$L\ =R+\lambda D=R_{mv}+\ R_{resi}+\lambda d(X_{t},{\hat{X}}_{t})$ (17)

where $R_{mv}$ and $R_{resi}$ represent the bits introduced by compressing the
offsets map ${\rm mv}_{off}$ and residual feature $F_{resi}$, respectively.
$d(X_{t},{\hat{X}}_{t})$ is the distortion between original frame $X_{t}$ and
reconstructed frame ${\hat{X}}_{t}$. $\lambda$ is the corresponding Lagrange
multiplier. For the first 15 epochs, the distortion $d(X_{m-pre},X_{t})$
between enhanced prediction frame $X_{m-pre}$ (that is generated from the
enhanced prediction feature $F_{m-pre}$ with a simple convolutional layer) and
original frame $X_{t}$ is also used to accelerate the training of the RDT and
MGP prediction module. After that, Eq. (17) is used to continue training.

TABLE II: Result comparison in terms of BD-Rate (%) measured with MS-SSIM. The anchor is VTM. Method | UVG | MCL-JCV | HEVC B | HEVC C | HEVC D | HEVC E | Average
---|---|---|---|---|---|---|---
VTM-17.0 [41] | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
HM-16.25 [42] | 31.1 | 38.8 | 36.6 | 35.2 | 33.0 | 41.1 | 36.0
ECM-5.0 [43] | -9.1 | -11.1 | -10.2 | -11.7 | -11.0 | -9.9 | -10.5
CANF-VC [44] | 46.5 | 26.0 | 43.5 | 30.9 | 17.9 | 173.0 | 56.3
DCVC [20] | 64.9 | 27.5 | 54.4 | 39.7 | 15.2 | 210.4 | 68.7
DCVC-TCM [18] | 1.0 | –10.8 | –11.7 | –15.2 | –29.0 | 16.7 | 8.85
DCVC-HEM [21] | –25.2 | –36.3 | –38.0 | –38.3 | –48.1 | –25.8 | -35.3
FVC [2] | 144.8 | 151.9 | 150.8 | 119.9 | 116.2 | 244.7 | 154.7
PROPOSED | -36.0 | -46.5 | -43.1 | -47.5 | -34.4 | -36.8 | -40.7
Figure 5: Rate-distortion curve comparisons under PSNR and MS-SSIM,
respectively.

##  IV Experiments

###  IV-A Experimental Settings

####  IV-A1 Dataset

Vimeo-90K dataset [46] is used as training dataset similar to the existing LVC
methods [2, 18, 21]. It contains 89,800 video sequences and each sequence
includes 7 frames. The sequences are randomly cropped to the resolution of 256
× 256 as input for training. The first frame in GOP, namely I frame, is
compressed by Cheng2020 in CompressAI [47]. Video sequences from three
datasets are used for test to evaluate the performance of our model, including
HEVC B-E class sequences [48], UVG [49], and MCL-JCV [50].

####  IV-A2 Training and testing Details

The PyTorch platform is used for implementation. Adam [51] optimizer is used
with the batch size set to 8. The hyperparameter $\lambda$ is set to four
different values corresponding to four models ($\lambda$ = 256, 512, 1024,
2048) at different rates. To construct the multi-reference frame structure in
the proposed MGP, when the reference frame buffer has less than three frame
features, the feature of the latest frame is duplicated until there are enough
reference frames. Peak Signal-to-Noise Ratio (PSNR) and Multi-Scale-Structural
Similarity Index (MS-SSIM) are used to evaluate the distortion between the
reconstructed video and the ground-truth video. BD-Rate savings over both PSNR
and MS-SSIM are adopted for evaluation and all evaluations are performed in
the RGB space where the YUV videos are converted to RGB with FFMPEG. We
evaluate 96 frames for each video in the test sets, with an intra period set
to 32 frames. The other settings including the low-delay configurations are
the same as in [3].

TABLE III: Ablation study results, in terms of BD-Rate (%) comparison measured with PSNR.  RDT |  MGP |  SFD-T |  HEVC B |  HEVC C |  HEVC D |  HEVC E |  Average
---|---|---|---|---|---|---|---
✗ |  ✗ |  ✗ |  0.0 |  0.0 |  0.0 |  0.0 |  0.0
✓ |  ✗ |  ✗ |  -17.8 |  -15.3 |  -18.2 |  -24.2 |  -18.9
✓ |  ✓ |  ✗ |  -41.4 |  -44.0 |  -35.3 |  -53.3 |  -43.5
✓ |  ✓ |  ✓ |  -67.0 |  -71.8 |  -61.1 |  -75.1 |  -68.7

(a) A sample frame

(b) Ground Truth

(c) FVC prediction

(d) Coarse prediction

(e) MGP prediction

Figure 6: Visual results comparison.

###  IV-B Comparison with State-of-the-art Methods

To evaluate the performance, the proposed method is compared with the existing
State-of-the-art methods, including HM-16.25 [42], VTM-17.0 [41], ECM-5.0
[43], FVC [2], DCVC [3], DCVC-HEM [21], and DCVC-TCM [18]. Among these
methods, HM, VTM and ECM are the traditional block-based video compression
methods while the others are LVC methods. The results are shown in Table I and
Table II for PSNR and MS-SSIM based BD-Rate comparison, respectively. The
result of VTM is used as anchor for the BD-Rate calculation. From Table I, it
can be seen that the proposed method achieves better performance than the
existing LVC methods. It obtains an average bitrate saving of 13.5% compared
to the anchor, while the other LVC methods perform worse (or similar) than
VTM. The RD curve comparisons are shown in Fig. 5, where our method performs
best compared to the others.

###  IV-C Ablation Study

Ablation experiments are further conducted to validate the effectiveness of
each proposed module in our method. The baseline without the proposed modules
is the FVC model [2] and the HEVC dataset is used for evaluation.

RDT based motion estimation and compensation. On top of the baseline FVC, RDT
is first used to replace the CNN-based motion estimation and compensation
component. The result is shown in Table III. It can be seen that compared with
the baseline FVC, significant improvement in terms of BD-Rate, 18.9% reduction
on average, is achieved with our RDT module. This indicates that the proposed
RDT based motion estimation method can obtain higher-quality motion
information from color space. As shown in Fig. 6, a comparison between Fig. 6c
and Fig. 6d shows that the ball in our coarse-grained prediction picture
obtained with RDT are more clear and complete than the FVC prediction. This
further demonstrates the advantages of the proposed RDT in capturing motion
details and stably transforming features between color space and geometric
space.

MGP based on multi-frame enhancement. MGP is further added on top of the FVC
and RDT to validate the effectiveness of using multi-reference frame to
enhance the prediction generation. As shown in Table III, the performance is
further improved by 24.6% in terms of BD-rate. This demonstrates the
effectiveness of using MGP to explore the multi-reference information for
prediction frame refinement. With sufficient temporal information, the quality
of prediction frame is further improved. A visual result comparison in Fig.6d
and Fig.6e illustrates that the prediction frame with our MGP module contains
more temporal motion details and sharper edges compared to the FVC prediction
and coarse-grained prediction.

SFD-T based residual compression. The performance of the proposed SFD-T module
can be further observed by comparing the results of the RDT & MGP and the
result of the full module in row 2 and row 3 of Table III, respectively. It
can be observed that the proposed SFD-T provides a significant 25.2% BD-Rate
gain over the CNN based residual compression in FVC. This demonstrates that
the SFD-T module effectively reduces redundancy in the spatial feature
distribution embedded in the temporal prediction, improving the residual
compression efficiency. Notably on HEVC test sets, the proposed method with
all the modules surpasses the baseline FVC by an average of 68.7% BD-Rate
saving, demonstrating the effectiveness of our method.

##  V Conclusion

In this paper, we propose a novel Spatial-Temporal Transformer-based video
compression (STT-VC) framework, containing RDT, MGP and SFD-T, designed for
inter-frame coding. RDT is developed for high-quality motion estimation and
compensation by producing a stable feature transformation from color space to
geometric motion space. MGP further enhances the prediction feature by
exploring the multi-reference frame information. It takes full advantage of
the coarse-grained prediction generated by the RDT and characterized by the
coded motion information. Then SFD-T is designed to improve residual
compression efficiency by leveraging joint temporal-spatial feature
distribution priors to further reduce spatial redundancy in the residuals.
Experimental results demonstrate that the proposed STT-VC framework
outperforms VTM by 13.5% on average in all tests and achieves the best
performance. Ablation studies confirm the effectiveness of each proposed
module in the framework and achieve a coding gain of 68.7% BD-Rate saving
against the baseline.

## References

  * [1] B. Bross, Y.-K. Wang, Y. Ye, S. Liu, J. Chen, G. J. Sullivan, and J.-R. Ohm, “Overview of the versatile video coding (vvc) standard and its applications,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 31, pp. 3736–3764, 2021.
  * [2] Z. Hu, G. Lu, and D. Xu, “Fvc: A new framework towards deep video compression in feature space,” _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 1502–1511, 2021.
  * [3] J. Li, B. Li, and Y. Lu, “Neural video compression with diverse contexts,” _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 22 616–22 626, 2023.
  * [4] H. Liu, M.-T. Lu, Z. Ma, F. Wang, Z. Xie, X. Cao, and Y. Wang, “Neural video coding using multiscale motion compensation and spatiotemporal context model,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 31, pp. 3182–3196, 2020.
  * [5] Z. Hu, G. Lu, J. Guo, S. Liu, W. Jiang, and D. Xu, “Coarse-to-fine deep video coding with hyperprior-guided mode prediction,” _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 5911–5920, 2022.
  * [6] J. Lin, D. Liu, H. Li, and F. Wu, “M-lvc: Multiple frames prediction for learned video compression,” _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 3543–3551, 2020.
  * [7] F.-M. Wang, H. Ruan, F. Xiong, J. Yang, L. Li, and R. Wang, “Butterfly: Multiple reference frames feature propagation mechanism for neural video compression,” _2023 Data Compression Conference (DCC)_ , pp. 198–207, 2023\.
  * [8] X. Yi, H. Wang, S. Kwong, and C.-C. J. Kuo, “Task-driven video compression for humans and machines: Framework design and optimization,” _IEEE Transactions on Multimedia_ , 2022.
  * [9] S. Li, W. Li, C. Cook, C. Zhu, and Y. Gao, “Independently recurrent neural network (indrnn): Building a longer and deeper rnn,” _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 5457–5466, 2018.
  * [10] K. Lin, C. Jia, X. Zhang, S. Wang, S. Ma, and W. Gao, “Dmvc: Decomposed motion modeling for learned video compression,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 33, pp. 3502–3515, 2023.
  * [11] M. Chen, A. Patney, and A. C. Bovik, “Movi-codec: Deep video compression without motion,” _2021 Picture Coding Symposium (PCS)_ , pp. 1–5, 2021.
  * [12] F. Mentzer, G. Toderici, D. C. Minnen, S. J. Hwang, S. Caelles, M. Lucic, and E. Agustsson, “Vct: A video compression transformer,” _ArXiv_ , vol. abs/2206.07307, 2022.
  * [13] N. Zou, H. Zhang, F. Cricri, H. R. Tavakoli, J. Lainema, E. B. Aksu, M. M. Hannuksela, and E. Rahtu, “Learned video compression with intra-guided enhancement and implicit motion information,” _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pp. 1870–1874, 2021.
  * [14] X. Zhang, S. Yang, W. Luo, L. Gao, and W. Zhang, “Video compression artifact reduction by fusing motion compensation and global context in a swin-cnn based parallel architecture,” in _AAAI Conference on Artificial Intelligence_ , 2023.
  * [15] A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lucic, and C. Schmid, “Vivit: A video vision transformer,” _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 6816–6826, 2021.
  * [16] Z. Liu, J. Ning, Y. Cao, Y. Wei, Z. Zhang, S. Lin, and H. Hu, “Video swin transformer,” _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 3192–3201, 2021.
  * [17] J. Cao, Y. Li, K. Zhang, and L. V. Gool, “Video super-resolution transformer,” _ArXiv_ , vol. abs/2106.06847, 2021.
  * [18] X. Sheng, J. Li, B. Li, L. Li, D. Liu, and Y. Lu, “Temporal context mining for learned video compression,” _ArXiv_ , vol. abs/2111.13850, 2021.
  * [19] N. Zou, H. Zhang, F. Cricri, H. R. Tavakoli, J. Lainema, E. B. Aksu, M. M. Hannuksela, and E. Rahtu, “End-to-end learning for video frame compression with self-attention,” _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pp. 580–584, 2020.
  * [20] J. Li, B. Li, and Y. Lu, “Deep contextual video compression,” in _Neural Information Processing Systems_ , 2021.
  * [21] ——, “Hybrid spatial-temporal entropy modelling for neural video compression,” _Proceedings of the 30th ACM International Conference on Multimedia_ , 2022.
  * [22] J. Ballé, V. Laparra, and E. P. Simoncelli, “End-to-end optimized image compression,” _ArXiv_ , vol. abs/1611.01704, 2016.
  * [23] Y. LeCun, B. E. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. E. Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” _Neural Computation_ , vol. 1, pp. 541–551, 1989.
  * [24] J. Ballé, D. C. Minnen, S. Singh, S. J. Hwang, and N. Johnston, “Variational image compression with a scale hyperprior,” _ArXiv_ , vol. abs/1802.01436, 2018.
  * [25] M.-T. Lu, P. Guo, H. Shi, C. Cao, and Z. Ma, “Transformer-based image compression,” _2022 Data Compression Conference (DCC)_ , pp. 469–469, 2021\.
  * [26] R. Zou, C. Song, and Z. Zhang, “The devil is in the details: Window-based attention for image compression,” _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 17 471–17 480, 2022.
  * [27] A. B. Koyuncu, H. Gao, and E. G. Steinbach, “Contextformer: A transformer with spatio-channel attention for context modeling in learned image compression,” _ArXiv_ , vol. abs/2203.02452, 2022.
  * [28] A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _NIPS_ , 2017\.
  * [29] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” _ArXiv_ , vol. abs/2010.11929, 2020.
  * [30] Y. Qian, M. Lin, X. Sun, Z. Tan, and R. Jin, “Entroformer: A transformer-based entropy model for learned image compression,” _ArXiv_ , vol. abs/2202.05492, 2022.
  * [31] G. Lu, W. Ouyang, D. Xu, X. Zhang, C. Cai, and Z. Gao, “Dvc: An end-to-end deep video compression framework,” _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 10 998–11 007, 2018.
  * [32] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural Computation_ , vol. 9, pp. 1735–1780, 1997.
  * [33] D. C. Minnen, J. Ballé, and G. Toderici, “Joint autoregressive and hierarchical priors for learned image compression,” in _Neural Information Processing Systems_ , 2018.
  * [34] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 9992–10 002, 2021.
  * [35] X. Sui, S. Li, X. Geng, Y. Wu, X. Xu, Y. Liu, R. S. M. Goh, and H. Zhu, “Craft: Cross-attentional flow transformer for robust optical flow,” _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 17 581–17 590, 2022.
  * [36] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, “Deformable detr: Deformable transformers for end-to-end object detection,” _ArXiv_ , vol. abs/2010.04159, 2020.
  * [37] Z. Xia, X. Pan, S. Song, L. E. Li, and G. Huang, “Vision transformer with deformable attention,” _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 4784–4793, 2022.
  * [38] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei, “Deformable convolutional networks,” _2017 IEEE International Conference on Computer Vision (ICCV)_ , pp. 764–773, 2017.
  * [39] Z. Wang, X. Cun, J. Bao, and J. Liu, “Uformer: A general u-shaped transformer for image restoration,” _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 17 662–17 672, 2021.
  * [40] S. Woo, J. Park, J.-Y. Lee, and I.-S. Kweon, “Cbam: Convolutional block attention module,” _ArXiv_ , vol. abs/1807.06521, 2018.
  * [41] (2022) Vtm-17.0. [Online]. Available: https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/
  * [42] (2022) Hm-16.25. [Online]. Available: https://vcgit.hhi.fraunhofer.de/jvet/HM/
  * [43] (2022) Ecm-5.0. [Online]. Available: https://vcgit.hhi.fraunhofer.de/ecm/ECM
  * [44] Y.-H. Ho, C.-P. Chang, P. Chen, A. Gnutti, and W.-H. Peng, “Canf-vc: Conditional augmented normalizing flows for video compression,” _European Conference on Computer Vision_ , vol. abs/2207.05315, 2022.
  * [45] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,” _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 548–558, 2021.
  * [46] T. Xue, B. Chen, J. Wu, D. Wei, and W. T. Freeman, “Video enhancement with task-oriented flow,” _International Journal of Computer Vision_ , pp. 1–20, 2017.
  * [47] J. Bégaint, F. Racap’e, S. Feltman, and A. Pushparaja, “Compressai: a pytorch library and evaluation platform for end-to-end compression research,” _ArXiv_ , vol. abs/2011.03029, 2020.
  * [48] G. J. Sullivan, J.-R. Ohm, W. Han, and T. Wiegand, “Overview of the high efficiency video coding (hevc) standard,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 22, pp. 1649–1668, 2012.
  * [49] A. Mercat, M. Viitanen, and J. Vanne, “Uvg dataset: 50/120fps 4k sequences for video codec analysis and development,” _Proceedings of the 11th ACM Multimedia Systems Conference_ , 2020.
  * [50] H. Wang, W. Gan, S. Hu, J. Y. Lin, L. Jin, L. Song, P. Wang, I. Katsavounidis, A. Aaron, and C.-C. J. Kuo, “Mcl-jcv: A jnd-based h.264/avc video quality assessment dataset,” _2016 IEEE International Conference on Image Processing (ICIP)_ , pp. 1509–1513, 2016.
  * [51] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _CoRR_ , vol. abs/1412.6980, 2014.