# APLA: Additional Perturbation for Latent Noise with Adversarial Training
Enables Consistency

Yupu Yao1, Shangqi Deng1, Zihan Cao1, Harry Zhang2, Liang-Jian Deng1

###### Abstract

Diffusion models have exhibited promising progress in video generation.
However, they often struggle to retain consistent details within local regions
across frames. One underlying cause is that traditional diffusion models
approximate Gaussian noise distribution by utilizing predictive noise, without
fully accounting for the impact of inherent information within the input
itself. Additionally, these models emphasize the distinction between
predictions and references, neglecting information intrinsic to the videos. To
address this limitation, inspired by the self-attention mechanism, we propose
a novel text-to-video (T2V) generation network structure based on diffusion
models, dubbed Additional Perturbation for Latent noise with Adversarial
training (APLA). Our approach only necessitates a single video as input and
builds upon pre-trained stable diffusion networks. Notably, we introduce an
additional compact network, known as the Video Generation Transformer (VGT).
This auxiliary component is designed to extract perturbations from the
inherent information contained within the input, thereby refining inconsistent
pixels during temporal predictions. We leverage a hybrid architecture of
transformers and convolutions to compensate for temporal intricacies,
enhancing consistency between different frames within the video. Experiments
demonstrate a noticeable improvement in the consistency of the generated
videos both qualitatively and quantitatively.

## Introduction

Generating video is a challenging task in computer vision, whose aim is to
generate high-fidelity, diverse videos from various inputs like text, images,
audio, or sketches. Recent works in deep learning have spurred considerable
advancements in this domain, most notably through the advent of diffusion
models (Ho, Jain, and Abbeel 2020). These models craft videos by iteratively
introducing noise to an initial input, then undoing this noise through a
series of denoising stages. Their strength is particularly notable in
generating high-definition, extended-duration videos with intricate semantics
and dynamics.

(a) Tune-A-Video (Wu et al. 2022)

(b) APLA (Ours)

Figure 1: The comparison (by the same prompt: “A man is skiing”) between Tune-
A-Video and the proposed APLA. (a) The result of Tune-A-Video is that the
snowboard splits into multiple parts on these frames. (b) The obtained outcome
is by our APLA method which keeps the single snowboard in all frames.

With the advancing capabilities of diffusion models, cross-modality generation
tasks, including text-to-image (T2I) and text-to-video (T2V), have made
substantial progress. Yet, the substantial data size of videos presents
challenges in training video generation models from scratch. A recent
approach, “Tune-A-Video” (Wu et al. 2022), aimed to utilize pre-trained T2I
models for video synthesis. However, its outcomes exhibited inconsistencies
across video frames. Ensuring frame consistency in video generation,
especially within fine-tuned models, remains a hurdle. While some efforts have
achieved acceptable frame consistency using diffusion models (Höppe et al.
2022), intricate video details, especially in complex scenarios, are often
absent. For instance, even with identical prompts and inputs, the “Tune-A-
Vide” approach still manifests inconsistencies in generated videos. Moreover,
extending fine-tuning epochs could potentially compromise the semantic
coherence between successive frames.

(a) An SUV is moving on the road, cartoon style

(b) An SUV is moving on the beach

(c) A puppy is eating an orange

(d) A rabbit is eating an orange

Figure 2: Visual demonstrations of APLA using different prompts.

In light of these challenges, we propose a novel architecture, _i.e._ , APLA.
Using self-attention (Vaswani and Shazeer 2023), APLA is geared to capture
inherent video characteristics and establish connections between frames by
adaptively generating parameters. To facilitate efficient information
extraction from inputs, we devise a much smaller model compared to diffusion
models, as shown in the success of style transfer and multi-task learning
(Zhang and Agrawala 2023). The result can be observed visually in Fig. 2.
Consequently, we engineer a decoder-only structure for our Video Generation
Transformer (VGT). With masking and self-attention mechanisms, VGT exhibits an
improvement in predicting unknown frames, demonstrating its ability to distill
intrinsic input information. We also introduce a novel loss function, hyper-
loss, to encourage the model to focus on nuanced input details. Lastly, to
further enhance the generated quality of video and improve the consistency
between different frames, we introduce adversarial training to improve the
quality of output while strengthening the robustness. The contributions of
this work can be summarized as follows:

  1. 1.

A novel architecture, _i.e._ , VGT, builds on top of pre-trained diffusion
models, which enhances the consistency between video frames by learning the
correlation information between input frames.

  2. 2.

A fusion of the diffusion model and adversarial is employed for video
generation, where adversarial training is directly applied to the discrepancy
in noise distributions, rather than judging the similarity between input and
output images.

  3. 3.

Quantitative and qualitative experiments that demonstrate the effectiveness of
the proposed approach, which achieves SOTA performance in frame consistency of
generated videos.

## Related Work

### Cross-Modal Video Generation

Generating videos using multiple input modalities presents a formidable
challenge within the realm of deep learning. A particularly prominent endeavor
in video generation is the synthesis of videos through text-to-video (T2V)
techniques, which entails creating videos grounded in natural language
descriptions. This innovative synthesis process can be conceptualized as an
evolution beyond the established domain of text-to-image (T2I) synthesis. In a
broader context, models for text-to-image (T2I) synthesis can be
systematically categorized into two distinctive classes: transformer-based
models and diffusion-based models. The former category, as exemplified by
references such as (Ramesh et al. 2021; Sim et al. 2019; Zhang 2016),
leverages the power of extensively trained large-scale language models such as
GPT-3 or T5. These models adeptly transform textual input into latent vectors,
which are subsequently employed for downstream image generation. Conversely,
the latter category represented by models like (Nichol et al. 2022; Zhang et
al. 2020, 2021), also incorporates a text encoder in a similar vein, yet
diverges in their approach by integrating the text-encoded information into
the diffusion process. The prowess of diffusion-based models in crafting
intricate images stands evident. This capability has been the foundation for
their evolution from text-to-image (T2I) synthesis to the more dynamic realm
of text-to-video (T2V) synthesis. However, these initial methods had
shortcomings like detail deficiency, temporal inconsistencies, and limited
control. Recent approaches have emerged to address these issues and enhance
text-to-video (T2V) synthesis.

### Diffusion Model

The inception of the Denoising Diffusion Probabilistic Model (DDPM),
documented in (Ho, Jain, and Abbeel 2020), took inspiration from thermodynamic
frameworks. This model, functioning as a Markov chain, exhibits an auto-
encoder structure. Nevertheless, its inference process faced sluggishness
owing to the intricate denoising stages encompassed. To address this issue,
Denoising Diffusion Implicit Models (DDIM)(Song, Meng, and Ermon 2020)
introduced an innovative strategy: by introducing variable variance for
predicted noise, they achieved swift inference for the diffusion model within
a concise span of steps. Building upon the foundations laid by DDIM,
subsequent endeavors (Watson et al. 2022; Zhang, Eisner, and Held 2023) aimed
to propel DDPM inference to even greater speeds.

Figure 3: The process of training the networks. VGT extracts intrinsic
information from latent variables, considering various time steps for noise
incorporation, and especially including the clean latent variable devoid of
noise, namely the original latent variable $z$. As VGT is not trained ever,
the output of VGT is tiny thus the change of the output is small, which is
helpful to improve the consistency of different frames without changing the
content a lot. The discriminator receives the predicted noise and the noise
residuals for corresponding time steps in the diffusion stage.

Broadly speaking, the Diffusion model’s key strength lies in its remarkable
capacity for handling tasks involving cross-modality and multi-modality
interactions. This strength is vividly demonstrated by the Latent Diffusion
Model (LDM) (Rombach et al. 2022; Pan et al. 2023; Eisner, Zhang, and Held
2022), which not only exhibited the potential of diffusion for high-resolution
image generation through the utilization of latent spaces but also showcased
the model’s ability to excel in cross-modality scenarios. The application of
the Diffusion model to video generation has prompted various approaches (Ho et
al. 2022; Lim et al. 2021, 2022; Avigal, Paradis, and Zhang 2020; Avigal et
al. 2021). Among these, Tune-A-Video (Wu et al. 2022; Devgon et al. 2020; Shen
et al. 2024; Jin et al. 2024) stands out for introducing a novel perspective
on video generation via diffusion modeling. This innovative methodology views
video generation as a process of refining a pre-trained stable diffusion. In
doing so, Tune-A-Video reshapes our understanding of how diffusion models can
be harnessed to address the challenges of video generation with a fresh and
effective approach.

## Methodology

In this section, we begin by presenting the overall structure of APLA.
Subsequently, we delve into the details of VGT, designed to extract intrinsic
information. Notably, we introduce two versions of VGT, each showcasing
distinct advantages in the experiment. Following this, we discuss hyper-loss
and introduce our adversarial training strategy.

### APLA

To enhance inter-frame consistency, alongside optimizing high-quality outputs,
it’s imperative to account for the interconnections among distinct frames.
While prior research assumed that inherent information could be naturally
grasped by the model without supplementary steps, the challenge becomes more
pronounced in complex tasks like video generation, where depicting high-level
temporal features during inference proves more intricate compared to image
generation. In light of this, we introduce a novel architecture (depicted in
Fig. 3) that builds upon the diffusion model and integrates an additional
module. This module is specifically designed to capture intrinsic information
and foster inter-frame connections within the temporal domain. This approach
sets us apart from previous methods and addresses the nuances presented by
video generation. A visual comparison between APLA and Tune-A-Video is
illustrated in Fig. 1. Specifically, the added module incorporates a self-
attention mechanism aimed at extracting information directly from inputs, all
without introducing any additional loss. Our hypothesis is rooted in the
potential of self-attention mechanisms to gather relevant details from the
input itself. This implies the ability to dynamically generate parameters
based on input, empowering the model with strong inductive capabilities.

Algorithm 1 Additional Perturbation for Latent Noise with Adversarial Training
(APLA)

0: A text query $q$, a reference video $v$, a pre-trained T2I diffusion model
with encoder $\varepsilon$ and decoder $\mathcal{D}$, a number of iterations
$T$, VGT $\varphi$, a discriminator $D$

0: A generated video $\hat{x}$ that matches the text query

Initialize $\hat{x}$ as an empty list

Extract the first frame $x_{0}$ from $v$

Encode $x_{0}$ and $q$ into a latent code $z_{0}$ using DDIM inversion

for $t=1$ to $T$ do

Compute $z_{t+1}$ through Eq. 1

Fine-tune the model parameters $\theta$ of $\varepsilon$ using Eq. 20

end for

for $t=T,T-1,...,1$ do

Sampling $\hat{z}_{t-1}$ using DDIM inversion

if $t==1$ then

Obtain $\hat{x}=\mathcal{D}(\hat{z}_{t})$

end if

end for

return $\hat{x}$

Figure 4: An illustration of VGT-Pure and VGT-Hyper. The left side shows the
transformer decoder structure, which adapted mask operation on the self-
attention mechanism especially. The right side shows the two versions of VGT.
The Temporal Transformer Decoder only receives the class (_i.e._ , cls) token
of output sequences of the Spatial Transformer Decoder. The rest of the tokens
of the output of the Spatial Transformer Decoder are used to multiply with the
tokens of the Temporal Transformer Decoder output dislodging cls token in VGT-
Pure, while the whole output of the Temporal Transformer Decoder is
transmitted to Transposed Convolution Block directly in VGT-Hyper.

To ensure content consistency with the introduced module, we propose that the
module’s output should manifest as subtle perturbations, significantly smaller
in magnitude than the output of the pre-trained model (referred to as U-Net in
this paper). The evolving ratio of perturbation to U-Net output over epochs
can be visually observed in Fig. 5(b). Let the input be denoted as
$x\in\mathbb{R}^{B\times H\times W\times F\times C}$, where $B$, $H$, $W$,
$F$, and $C$ represent batch, height, width, frame, and channel dimensions
respectively. We define the encoder as $\mathcal{E}$, decoder as
$\mathcal{D}$, and $\tilde{x}=\mathcal{D}(z)=\mathcal{D}(\mathcal{E}(x))$
signifies the intended outcome. Furthermore, considering the process in the
latent space, let $Z$ denote the latent space with $z\in Z$, where
$z\in\mathbb{R}^{B\times h\times w\times F\times c}$, where $h,w,c$ are the
dimension in the latent space respectively. The diffusion process and U-Net
are represented by $\phi$ and $\pi$ respectively. The term $z_{T}$ in the
denoising stage signifies the latent variable’s evolution over $T$ steps with
added noise. A pivotal addition is our module, termed Video Generation
Transformer (VGT). Denoted as $\varphi$, it encapsulates an abstract function.
The series of denoising autoencoders is represented as
$\epsilon_{\theta}(x_{t},t)$, with $t=1,\ldots,T$, where $t$ corresponds to
the specific step in the sequence. Then, we have

$z_{t}=\phi(z_{t-1},t-1),$ (1) $\hat{z}_{T-1}=\pi(z_{T},T).$ (2)
$\hat{z}_{t-1}=\pi(\hat{z}_{t},t),$ (3)

where $\hat{z}_{t}$ represents the predicted output of denoising U-Net at the
$t$-th step. In fact, we add the perturbation to the U-Net, which aims to
capture intrinsic information:

$\hat{z}_{t-1}^{*}=\pi(\hat{z_{t}},t)+\varphi(z_{t},t),$ (4)

thus the original object function:

$\mathcal{L}_{MSE}:=\mathbb{E}_{\mathcal{E}(x),\epsilon\sim\mathcal{N}(0,1),t}\left[\left\|\epsilon-\epsilon_{\theta}\left(\hat{z}_{t},t\right)\right\|_{2}^{2}\right],$
(5)

can be rewritten as:

$\mathcal{L}_{MSE}:=\mathbb{E}_{\mathcal{E}(x),\epsilon\sim\mathcal{N}(0,1),t}\left[\left\|\epsilon-\epsilon_{\theta}\left(\hat{z}_{t}^{*},t\right)\right\|_{2}^{2}\right].$
(6)

For clarity, the intuitive pseudocode is illustrated in the Algrithm 1. With
the adversarial training that enhances the robustness and quality of the
generator output, a discriminator is set to receive the predicted noise and
noise residuals in the corresponding step. More elaborate discussions are
presented in the following sections.

(a) Comparison of VGT versions

(b) Ratio of VGT output and U-Net

Figure 5: (a) is the comparison of different versions of VGT. “EN” represents
the use of a transformer encoder instead of a decoder, which means the mask
operation was not included. As the picture shows, VGT-Hyper performs the best
while the encoder version of VGT-Hyper performs the worst. For VGT-Pure, the
encoder version performs similarly to the decoder version, while the
performance of the two versions is between VGT-Hyper and VGT-Hyper-EN. (b)
shows the ratio of VGT output and U-Net in the denoising step. The result
shows that the norm of VGT output is very tiny compared with the U-Net output,
which shows that the output of VGT did not change the original output much
while improving the consistency of different frames laterally.

### Video Generation Transformer (VGT)

We introduce the proposed VGT, designed as a decoder-only architecture. With
self-attention mechanisms, this Transformer efficiently focuses on input
features. Compared to pure encoder-based structures like BERT (Devlin et al.
2019), the pure Transformer decoder architecture (Liu et al. 2018)
accommodates more tokens, boosting its processing capacity. The design of the
decoder within the Transformer stands out as a unique case in the realm of
autoregressive models (Dai and Le 2015), showcasing potential in unsupervised
time series prediction (Brown, Mann et al. 2020). Besides, this design
empowers the Transformer decoder to extract temporal information from input
data. For tasks involving time sequences, the self-attention mechanisms of the
Transformer decoder enhance output distribution coherence by extracting
contextual insights. Notably, even with incomplete sequence inputs,
Transformer decoders promote model diversity (Esser, Rombach, and Ommer 2021),
thereby bolstering generalization capabilities.

Nonetheless, in previous video generation models employing Transformer
architectures, like Video Transformer (VIVIT) (Arnab et al. 2021), solely the
Transformer encoder is utilized to derive a latent variable. This variable is
subsequently fed into another network, often a classifier, to fulfill the
downstream task. Consequently, VIVIT lacks the direct capability to generate
videos. It is precisely due to this limitation that we introduce the Video
Generation Transformer (VGT). Our framework aims to reconstruct or generate
videos, leading to the proposal of two distinct VGT variants. The first is a
pure Transformer decoder approach, referred to as VGT-pure, while the second
combines self-attention with 3D convolution, termed VGT-Hyper.

VGT version
---
Version | PSNR | Trainable Parameters
VGT-Pure | 52.746 | 60.362M
VGT-Hyper | 58.552 | 97.136M
VGT-Pure-EN | 54.236 | 60.362M
VGT-Hyper-EN | 42.736 | 97.136M
Table 1: The different version of VGT, while “EN” represents the transformer
encoder to be used instead of the encoder, which means there is no mask
operation. We compare the PSNR of the reconstruction quality, as the input is
the single video generated randomly. Meanwhile, we compare the trainable
parameters of different VGT, while the VGT-Pure and VGT-Pure-EN own the lowest
trainable parameters, for the mask operation did not change the quantities of
trainable parameters.

### VGT-Pure

The initial model variant is a pure Transformer decoder. We denote the input
sequence in the $\ell_{th}$ layer as $\mathbf{z}_{s}^{\ell}$ and
$\mathbf{z}_{t}^{\ell}$, where $s$ and $t$ stand for spatial and temporal
aspects respectively. Furthermore, we define a token as
$\mathbf{z}_{cls,s}^{k,\ell}$, with “cl” signifying the class token, and $k$
representing the $k$-th token in the sequence, excluding the class token. In
this context, we replace Multi-Headed Self Attention (MAS) with MASKed Multi-
Headed Self Attention (MMAS). Likewise, each transformer block encompasses
layer normalization (LN). The spatial decoder block can be succinctly
represented as follows:

$\mathbf{z}_{s}^{\ell+1}=\texttt{MMSA}(\texttt{LN}(\mathbf{z}_{s}^{\ell}))+\mathbf{z}_{s}^{\ell},$
(7)

and similarly the temporal decoder block as:

$\mathbf{z}_{t}^{\ell+1}=\texttt{MMSA}(\texttt{LN}(\mathbf{z}_{t}^{\ell}))+\mathbf{z}_{t}^{\ell}.$
(8)

Frame Consistency
---
Method | CLIP Score | FVD | IS | FCI
CogVideo (Hong et al. 2022) | 90.64 | 626 | 50.46 | 0.2942
Plug-and-Play (Tumanyan et al. 2022) | 88.89 | - | - | 0.3048
VideoGPT (Yan et al. 2021) | 47.12 | - | 24.69 | 0.4567
DVD-GAN (Clark, Donahue, and Simonyan 2019) | 48.36 | - | 27.38 | 0.4012
TGANv2 (Saito et al. 2020) | 51.26 | 1209 | 28.87 | 0.3538
MoCoGAN-HD (Tian, Ren, and Chai 2021) | 77.64 | 838 | 32.36 | 0.3156
DIGAN (Yu and Tack 2022) | 81.26 | 655 | 29.71 | 0.3012
TATS-base (Ge and Hayes 2022) | 90.65 | 332 | 79.28 | 0.2647
Tune-A-Video (Wu et al. 2022) | 92.40 | 864 | 64.12 | 0.2716
SAVE (Karim et al. 2023) | 94.81 | - | - | 0.2628
APLA (Ours) | 96.21 | 512 | 71.26 | 0.2576
Table 2: Quantitative comparison with evaluated baselines by measuring content
consistency with text prompt using CLIP Score ($\uparrow$), FVD
($\downarrow$), IS ($\uparrow$) and FCI ($\downarrow$), which measures how
many users prefer each model. FCI ($\downarrow$) is the metric that computes
the optical flow difference of the adjacent frame. The lower, the better.

We consider $L$ layers of spatial decoder at all, and the $\mathbf{z}_{t}^{L}$
can be represented as:

$\mathbf{z}_{s}^{L}=\left[\mathbf{z}_{cls,s}^{L},\mathbf{z}_{s}^{1,L},\mathbf{z}_{s}^{2,L},\ldots,\mathbf{z}_{s}^{F,L}\right]+\mathbf{p},$
(9)

where $\mathbf{p}$ denotes the positional embedding, and $\mathbf{z}$ is
divided into tokens. Among these tokens, the first one is
$\mathbf{z}_{cls,s}\in\mathbb{R}^{F\times(H_{patch}\times W_{patch})\times
L}$, serving as a compact feature commonly used for categorical embedding.
Here, $H_{patch}$ represents the patch’s height, and $W_{patch}$ is the width
of a patch. When we split $\mathbf{z}_{cls,s}$ along the temporal frame
dimension, we obtain individual tokens
$\mathbf{z}_{cls,s}^{1,L},\mathbf{z}_{cls,s}^{2,L},\ldots,\mathbf{z}_{cls,s}^{F,L}$,
where $\mathbf{z}_{cls,s}^{i,L}\in\mathbb{R}^{1\times(H_{patch}\times
W_{patch})\times L}$ for $i=1,2,\ldots,F$. Simultaneously, we maintain the
collection
$\left[\mathbf{z}_{s}^{1,L},\mathbf{z}_{s}^{2,L},\ldots,\mathbf{z}_{s}^{F,L}\right]$
as $\mathbf{z}_{s}^{C,L}$, facilitating skip connections. This arrangement
leads us to use $\mathbf{z}_{cls,s}^{L}$ as the input for the temporal
decoder, setting it apart from the rest. This ultimately yields the expression
for $\mathbf{z}_{t}^{1}$:

$\mathbf{z}_{t}^{1}=\left[\mathbf{z}_{cls,t}^{1},\mathbf{D}\mathbf{z}_{cls,s}^{1,L},\mathbf{D}\mathbf{z}_{cls,s}^{2,L},\ldots,\mathbf{D}\mathbf{z}_{cls,s}^{F,L}\right]+\mathbf{p},$
(10)

where $\mathbf{D}$ is the decoder block. Suppose we have $M$ layers temporal
decoder in total. Similarly, the output of $M_{th}$ can be written as:

$\mathbf{z}_{t}^{M}=\left[\mathbf{z}_{cls,t}^{M},\mathbf{z}_{cls,t}^{1,M},\mathbf{z}_{cls,t}^{2,M},\ldots,\mathbf{z}_{cls,t}^{F,M}\right]+\mathbf{p},$
(11)

Similarly, we denote
$\left[\mathbf{z}_{cls,t}^{1,M},\mathbf{z}_{cls,t}^{2,M},\ldots,\mathbf{z}_{cls,t}^{F,M}\right]$
as $\mathbf{z}_{t}^{C,M}$, then the output of VGT-Pure can be written as:

$\mathbf{y}=\mathbf{z}_{s}^{C,L}\odot\mathbf{z}_{t}^{C,M},$ (12)
$\hat{\mathbf{z}_{t}}=\texttt{MLP}(\mathbf{y}),$ (13)

where $\odot$ represents the Hadamard product, MLP represents multilayer
perceptron and $t$ is $t$-th step. we get
$\hat{\mathbf{z}_{t}}\in\mathbb{R}^{B\times F\times(H_{patch}\times
W_{patch})\times(C\times P\times P)}$, then we rearrange the
$\hat{\mathbf{z}_{t}}$ into $\varphi(z_{t})$, where $\varphi(z_{t})\in
R^{B\times F\times C\times(H_{patch}\times P)\times(W_{patch}\times P)}$ and
$\varphi(\cdot)$ is the function representation of VGT-Pure while $z_{t}$ is
the input of VGT-Pure, and $(H_{patch}\times P)=H$ and $(W_{patch}\times
P)=H$.

### VGT-Hyper

In this section, we introduce the second variant of VGT, named VGT-Hyper,
which leverages 3D convolution(Tran et al. 2015). Particularly, in Eq. 12,
rather than employing element-wise multiplication, we opt for 3D convolution.
We represent the convolution block with the matrix $\mathbf{M}$, leading to
the following expression:

$\mathbf{y^{*}}=\mathbf{M}\mathbf{z}_{t}^{M},$ (14)
$\hat{\mathbf{z}_{t}}=\texttt{MLP}(\mathbf{y^{*}}).$ (15)

Unlike VGT-pure, VGT-Hyper demonstrates superior performance in the
reconstruction task, as indicated in Tab. 1, all the while maintaining a
higher number of trainable parameters. VGT-Hyper capitalizes on the benefits
inherent in a transformer decoder, underscoring the efficacy of the mask
operation for time series tasks, depicted in Fig. 5(a).

### Hyper-Loss for Latent Noise Fitting

Recognizing the limitations of Mean Squared Error (MSE) for certain generative
tasks (Zhang et al. 2018), we introduce a novel loss function tailored for
video generation. We adopt a perceptual loss approach within the diffusion
model, akin to prior studies (Lugmayr et al. 2022). In detail, we formalize
$\ell_{1}$ loss and perceptual loss separately as follows:

$\mathcal{L}_{L1}:=\mathbb{E}_{\mathcal{E}(x),\epsilon\sim\mathcal{N}(0,1),t}\left[\left\|\epsilon-\epsilon_{\theta}\left(z_{t}^{*},t\right)\right\|_{1}\right],$
(16)

and

$\mathcal{L}_{per}:=\mathbb{E}_{\mathcal{E}(x),\epsilon\sim\mathcal{N}(0,1),t}\left[dist_{per}(\epsilon,\epsilon_{\theta}\left(z_{t}^{*},t\right))\right].$
(17)

Expanding upon this, we incorporate a hyper-loss that encompasses the weighted
combination of Mean Squared Error (MSE), $\ell_{1}$ loss, and perceptual loss.
The $\ell_{1}$ loss serves as a regularization term to promote sparsity in the
solution, while the perceptual loss encourages the model to generate more
photorealistic images. This concept is illustrated as follows:

$\mathcal{L}_{hyper}:=\alpha*\mathcal{L}_{MSE}+\beta*\mathcal{L}_{L1}+\gamma*\mathcal{L}_{per},$
(18)

where $\alpha$, $\beta$ and $\gamma$ represent the weights.

Frame Consistency
---
Method | CLIP Score | FCI | CLIP Score (1500 epochs) | FCI (1500 epochs)
Full Model (ours) | 96.21 | 0.2764 | 96.76 | 0.2470
w\o Discriminator | 94.42 | 0.2714 | 93.70 | 0.2178
w\o VGT&Discriminator | 91.44 | 0.1918 | 96.13 | 0.2655
w\o Hyper-Loss&Discriminator | 93.97 | 0.2476 | 93.06 | 0.2588
w\o VGT&Hyper-Loss | 94.83 | 0.2534 | 96.38 | 0.2172
Table 3: Ablation studies on APLA’s different components. Compared via CLIP
score and FCI on each model variation trained with 750 epochs(default) and
1500 epochs respectively.

### Adversarial Training with 1$\times$1 Convolution

In our approach, we view adversarial training as a valuable form of
regularization. For video generation, the distinction between integrating
Generative Adversarial Networks (GANs) and employing perceptual loss lies in
their treatment of temporal information. Perceptual loss is primarily
concerned with the structural attributes of individual frames, whereas
reconstruction loss focuses on pixel-level closeness. In contrast, GAN loss
centers around maintaining consistency across frames, promoting temporal
coherence. Besides, it is important to recognize that GANs have a propensity
for capturing global information by treating all frames holistically. This
approach leads to an enhancement in video quality through adversarial
training. The discriminator, in the APLA, receives the output of the
generator, namely the predicted noise to compare with the noise residual,
which is obtained in the diffusion process. More concretely, the diffusion
process $T$ steps, while the denoising process aims to inverse this process,
which predicts the noise residual, the difference of the $t$-th step and
$t$-$1$-th step, namely the adding noise. In the denoising process, for
instance, the generator predicts the $t$-th step noise residual, and then the
discriminator receives the corresponding $t$-th step noise in the diffusion
process, aiming to decline the distance of two noise distributions (noise
residual and predicted noise residual).

The proposed discriminator structure is streamlined, comprising only a
1$\times$1 convolutional layer. This kernel comprehensively considers frame
positional data, aiding temporal similarity extraction. Denoting the
discriminator as $D(\cdot)$ and with $x\sim p(x)$, we arrive at a min-max
problem:

$\mathop{\max}_{D}\mathop{\min}_{G}\mathbb{E}_{{x}\sim p(x)}\left[\log
D_{G}^{*}({x})\right]+\mathbb{E}_{{x}\sim
p_{g}}\left[\log\left(1-D_{G}^{*}({x})\right)\right],$ (19)

where $G$ represents the generator, which is the united block of U-Net and VGT
in this paper, while $D$ represents the discriminator. Let the generation loss
of the generator, in this paper, which is the fusion of U-Net and VGT, as
$L_{g}$, where $L_{g}=\mathbb{E}_{{x}\sim p_{(}x)}\left[\log
D_{G}^{*}({x})\right]+\mathbb{E}_{{x}\sim
p_{(}x)}\left[\log\left(1-D_{G}^{*}({x})\right)\right]$. Hence, the final
optimization objective is

$\mathop{\min}_{\theta}\mathop{\max}_{G}\mathcal{L}_{hyper}+\lambda L_{g},$
(20)

where $\lambda$ is a coefficient to adjust the performance and $\theta$ is the
parameters of the network.

## Experiments

### Implementation Details

We build the model based on Tune-A-Video (Wu et al. 2022; Elmquist et al.
2022) and utilize the pre-trained weights of the Stable Diffusion (Rombach et
al. 2022). We uniformly sample 24 frames at a resolution of 512$\times$512
from the input video and train the models for 750 steps and 1500 steps with a
learning rate of 3e-5 and a batch size of 1. For the hyperparameter induced by
loss, we set $\alpha$ of 0.5, $\beta$ of 0.2, $\gamma$ of 0.1, and $\lambda$
of 0.5. Due to the limited CUDA memory, we chose the VGT-pure only for the
following experiment. During inference, we employ the DDIM sampler (Song,
Meng, and Ermon 2020) with classifier-free guidance (Ho and Salimans 2022) in
our experiments. For a single video, it takes about 90 minutes for 1500 steps
and approximately 1 minute (while Tune-A-Video takes 60 minutes) for sampling
on an NVIDIA 3090 GPU.

Dataset: To evaluate our approach, we use representative videos taken from the
DAVIS dataset (Pont-Tuset et al. 2017). During fine-tuning, we just train the
model on a single video. The video descriptions and captions automatically
used an off-the-shelf captioning model (Li et al. 2023), which is regarded as
the default prompt of our video.

Qualitative Results: A visual comparison is presented between our approach and
the baseline in Fig. 1(a), focusing on reconstructing tasks using the same
prompt. We observe that Tune-A-Video does not reconstruct the details in each
frame well. Specifically, the sled visibly splits into two pieces in some
frames. Our model provides more stable and smooth results compared with the
Tune-A-Video. Also, readers can find more results obtained by our method in
Fig. 2.

Quantitative Results: Quantitative assessment is conducted, as depicted in
Tab. 2. Regarding content consistency, we evaluate the CLIP score (Radford et
al. 2021) across all generated frames by employing average cosine
measurements. This metric serves as an indicator of the semantic coherence
within the generated videos. In terms of frame consistency, we employ the flow
consistency index (FCI) for comparison. Unlike(Varghese et al. 2020), we
directly compute the flow field between consecutive frames, independent of the
input video. Specifically, we determine the optical flow field between two
adjacent frames, assess alterations in the local optical flow field concerning
each pixel’s value and its domain, and subsequently average all the computed
changes. The results underscore that, in comparison to our baseline model,
enhancements are observed in both content consistency and frame consistency.

### Ablation Studies

We conduct ablation studies to assess the importance of different components
of ALVA, as shown in Tab. 3. The proposed full APLA model performs the best
considering content consistency and frame consistency together. Without some
components, APLA’s performance degrades but still performs better than Tune-A-
Video. We also discuss the influence of the number of epochs. From observing
Tab. 3, we see that with the epoch increasing, the quantitative score is
increasing. However, from the visual result, we see that too many epochs can
cause overfitting, which destroys the result influenced by the prompt. Without
the Discriminator, as the number of epochs increases, it is easy to fall into
local minima which decreases the CLIP score and FCI. For w\o
VGT&Discriminator, although the final FCI is decent, it cannot retain the
semantic consistency and needs too many epochs to reach a good result. For w\o
Hyper-Loss&Discriminator, the single VGT can just reach a normal level and it
is hard for it to approach a better score because of the limitation of
convergence. As w\o VGT&Hyper-Loss, the model performance is close to w\o
Discriminator and even better. However, it still needs too many epochs to
reach such a good result.

## Conclusion

In this study, we introduce APLA, which includes a compact module for
capturing intrinsic or temporal information, and the novel VGT architecture, a
pure transformer decoder similar to GPT. To fortify the robustness and quality
of our APLA model, we employ adversarial training during its training process.
Through experiments, our model achieves state-of-the-art performance in video
reconstruction and videos from textual prompts (T2V).

## References

  * Arnab et al. (2021) Arnab, A.; Dehghani, M.; Heigold, G.; Sun, C.; Lučić, M.; and Schmid, C. 2021.  Vivit: A video vision transformer.  In _ICCV_ , 6836–6846.
  * Avigal, Paradis, and Zhang (2020) Avigal, Y.; Paradis, S.; and Zhang, H. 2020.  6-DoF grasp planning using fast 3D reconstruction and grasp quality CNN.  _arXiv preprint arXiv:2009.08618_.
  * Avigal et al. (2021) Avigal, Y.; Satish, V.; Tam, Z.; Huang, H.; Zhang, H.; Danielczuk, M.; Ichnowski, J.; and Goldberg, K. 2021.  Avplug: Approach vector planning for unicontact grasping amid clutter.  In _2021 IEEE 17th international conference on automation science and engineering (CASE)_ , 1140–1147. IEEE.
  * Brown, Mann et al. (2020) Brown, T. B.; Mann, B.; et al. 2020.  Language Models are Few-Shot Learners.  arXiv:2005.14165.
  * Clark, Donahue, and Simonyan (2019) Clark, A.; Donahue, J.; and Simonyan, K. 2019.  Adversarial Video Generation on Complex Datasets.  arXiv:1907.06571.
  * Dai and Le (2015) Dai, A. M.; and Le, Q. V. 2015.  Semi-supervised Sequence Learning.  arXiv:1511.01432.
  * Devgon et al. (2020) Devgon, S.; Ichnowski, J.; Balakrishna, A.; Zhang, H.; and Goldberg, K. 2020.  Orienting novel 3D objects using self-supervised learning of rotation transforms.  In _2020 IEEE 16th International Conference on Automation Science and Engineering (CASE)_ , 1453–1460. IEEE.
  * Devlin et al. (2019) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  arXiv:1810.04805.
  * Eisner, Zhang, and Held (2022) Eisner, B.; Zhang, H.; and Held, D. 2022.  Flowbot3d: Learning 3d articulation flow to manipulate articulated objects.  _arXiv preprint arXiv:2205.04382_.
  * Elmquist et al. (2022) Elmquist, A.; Young, A.; Hansen, T.; Ashokkumar, S.; Caldararu, S.; Dashora, A.; Mahajan, I.; Zhang, H.; Fang, L.; Shen, H.; et al. 2022.  ART/ATK: A research platform for assessing and mitigating the sim-to-real gap in robotics and autonomous vehicle engineering.  _arXiv preprint arXiv:2211.04886_.
  * Esser, Rombach, and Ommer (2021) Esser, P.; Rombach, R.; and Ommer, B. 2021.  Taming transformers for high-resolution image synthesis.  In _CVPR_ , 12873–12883.
  * Ge and Hayes (2022) Ge, S.; and Hayes, T. 2022.  Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer.  arXiv:2204.03638.
  * Ho et al. (2022) Ho, J.; Chan, W.; Saharia, C.; Whang, J.; Gao, R.; Gritsenko, A.; Kingma, D. P.; Poole, B.; Norouzi, M.; Fleet, D. J.; et al. 2022.  Imagen video: High definition video generation with diffusion models.  _arXiv_.
  * Ho, Jain, and Abbeel (2020) Ho, J.; Jain, A.; and Abbeel, P. 2020.  Denoising diffusion probabilistic models.  _NeurIPS_ , 33: 6840–6851.
  * Ho and Salimans (2022) Ho, J.; and Salimans, T. 2022.  Classifier-free diffusion guidance.  _arXi_.
  * Hong et al. (2022) Hong, W.; Ding, M.; Zheng, W.; Liu, X.; and Tang, J. 2022.  CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers.  arXiv:2205.15868.
  * Höppe et al. (2022) Höppe, T.; Mehrjou, A.; Bauer, S.; Nielsen, D.; and Dittadi, A. 2022.  Diffusion models for video prediction and infilling.  _arXiv_.
  * Jin et al. (2024) Jin, D.; Karmalkar, S.; Zhang, H.; and Carlone, L. 2024.  Multi-Model 3D Registration: Finding Multiple Moving Objects in Cluttered Point Clouds.  _arXiv preprint arXiv:2402.10865_.
  * Karim et al. (2023) Karim, N.; Khalid, U.; Joneidi, M.; Chen, C.; and Rahnavard, N. 2023.  SAVE: Spectral-Shift-Aware Adaptation of Image Diffusion Models for Text-guided Video Editing.  arXiv:2305.18670.
  * Li et al. (2023) Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv_.
  * Lim et al. (2021) Lim, V.; Huang, H.; Chen, L. Y.; Wang, J.; Ichnowski, J.; Seita, D.; Laskey, M.; and Goldberg, K. 2021.  Planar robot casting with real2sim2real self-supervised learning.  _arXiv preprint arXiv:2111.04814_.
  * Lim et al. (2022) Lim, V.; Huang, H.; Chen, L. Y.; Wang, J.; Ichnowski, J.; Seita, D.; Laskey, M.; and Goldberg, K. 2022.  Real2sim2real: Self-supervised learning of physical single-step dynamic actions for planar robot casting.  In _2022 International Conference on Robotics and Automation (ICRA)_ , 8282–8289. IEEE.
  * Liu et al. (2018) Liu, P. J.; Saleh, M.; Pot, E.; Goodrich, B.; Sepassi, R.; Kaiser, L.; and Shazeer, N. 2018.  Generating Wikipedia by Summarizing Long Sequences.  arXiv:1801.10198.
  * Lugmayr et al. (2022) Lugmayr, A.; Danelljan, M.; Romero, A.; Yu, F.; Timofte, R.; and Van Gool, L. 2022.  RePaint: Inpainting Using Denoising Diffusion Probabilistic Models.  In _CVPR_ , 11461–11471.
  * Nichol et al. (2022) Nichol, A.; Dhariwal, P.; Ramesh, A.; Shyam, P.; Mishkin, P.; McGrew, B.; Sutskever, I.; and Chen, M. 2022.  GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models.  arXiv:2112.10741.
  * Pan et al. (2023) Pan, C.; Okorn, B.; Zhang, H.; Eisner, B.; and Held, D. 2023.  Tax-pose: Task-specific cross-pose estimation for robot manipulation.  In _Conference on Robot Learning_ , 1783–1792. PMLR.
  * Pont-Tuset et al. (2017) Pont-Tuset, J.; Perazzi, F.; Caelles, S.; Arbeláez, P.; Sorkine-Hornung, A.; and Van Gool, L. 2017.  The 2017 davis challenge on video object segmentation.  _arXiv_.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning transferable visual models from natural language supervision.  In _ICML_ , 8748–8763. PMLR.
  * Ramesh et al. (2021) Ramesh, A.; Pavlov, M.; Goh, G.; Gray, S.; Voss, C.; Radford, A.; Chen, M.; and Sutskever, I. 2021.  Zero-Shot Text-to-Image Generation.  arXiv:2102.12092.
  * Rombach et al. (2022) Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022.  High-resolution image synthesis with latent diffusion models.  In _CVPR_ , 10684–10695.
  * Saito et al. (2020) Saito, M.; Saito, S.; Koyama, M.; and Kobayashi, S. 2020.  Train Sparsely, Generate Densely: Memory-Efficient Unsupervised Training of High-Resolution Temporal GAN.  _IJCV_ , 128(10-11): 2586–2606.
  * Shen et al. (2024) Shen, S.; Zhu, Z.; Fan, L.; Zhang, H.; and Wu, X. 2024.  DiffCLIP: Leveraging Stable Diffusion for Language Grounded 3D Classification.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 3596–3605.
  * Sim et al. (2019) Sim, K. C.; Beaufays, F.; Benard, A.; Guliani, D.; Kabel, A.; Khare, N.; Lucassen, T.; Zadrazil, P.; Zhang, H.; Johnson, L.; et al. 2019.  Personalization of end-to-end speech recognition on mobile devices for named entities.  In _2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , 23–30. IEEE.
  * Song, Meng, and Ermon (2020) Song, J.; Meng, C.; and Ermon, S. 2020.  Denoising diffusion implicit models.  _arXiv_.
  * Tian, Ren, and Chai (2021) Tian, Y.; Ren, J.; and Chai, M. 2021.  A Good Image Generator Is What You Need for High-Resolution Video Synthesis.  arXiv:2104.15069.
  * Tran et al. (2015) Tran, D.; Bourdev, L.; Fergus, R.; Torresani, L.; and Paluri, M. 2015.  Learning Spatiotemporal Features with 3D Convolutional Networks.  arXiv:1412.0767.
  * Tumanyan et al. (2022) Tumanyan, N.; Geyer, M.; Bagon, S.; and Dekel, T. 2022.  Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation.  arXiv:2211.12572.
  * Varghese et al. (2020) Varghese, S.; Bayzidi, Y.; Bär, A.; Kapoor, N.; Lahiri, S.; Schneider, J. D.; Schmidt, N.; Schlicht, P.; Hüger, F.; and Fingscheidt, T. 2020.  Unsupervised Temporal Consistency Metric for Video Segmentation in Highly-Automated Driving.  In _CVPRW_ , 1369–1378.
  * Vaswani and Shazeer (2023) Vaswani, A.; and Shazeer, N. 2023.  Attention Is All You Need.  arXiv:1706.03762.
  * Watson et al. (2022) Watson, D.; Chan, W.; Ho, J.; and Norouzi, M. 2022.  Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality.  In _ICLR_.
  * Wu et al. (2022) Wu, J. Z.; Ge, Y.; Wang, X.; Lei, W.; Gu, Y.; Hsu, W.; Shan, Y.; Qie, X.; and Shou, M. Z. 2022.  Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation.  _arXiv_.
  * Yan et al. (2021) Yan, W.; Zhang, Y.; Abbeel, P.; and Srinivas, A. 2021.  VideoGPT: Video Generation using VQ-VAE and Transformers.  arXiv:2104.10157.
  * Yu and Tack (2022) Yu, S.; and Tack, J. 2022.  Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks.  arXiv:2202.10571.
  * Zhang (2016) Zhang, H. 2016.  Health diagnosis based on analysis of data captured by wearable technology devices.  _International Journal of Advanced Science and Technology_ , 95: 89–96.
  * Zhang, Eisner, and Held (2023) Zhang, H.; Eisner, B.; and Held, D. 2023.  Flowbot++: Learning generalized articulated objects manipulation via articulation projection.  _arXiv preprint arXiv:2306.12893_.
  * Zhang et al. (2020) Zhang, H.; Ichnowski, J.; Avigal, Y.; Gonzales, J.; Stoica, I.; and Goldberg, K. 2020.  Dex-Net AR: Distributed deep grasp planning using a commodity cellphone and augmented reality app.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 552–558. IEEE.
  * Zhang et al. (2021) Zhang, H.; Ichnowski, J.; Seita, D.; Wang, J.; Huang, H.; and Goldberg, K. 2021.  Robots of the lost arc: Self-supervised learning to dynamically manipulate fixed-endpoint cables.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , 4560–4567. IEEE.
  * Zhang and Agrawala (2023) Zhang, L.; and Agrawala, M. 2023.  Adding conditional control to text-to-image diffusion models.  _arXiv_.
  * Zhang et al. (2018) Zhang, R.; Isola, P.; Efros, A. A.; Shechtman, E.; and Wang, O. 2018.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric.  arXiv:1801.03924.


# Paraphrases do not explain word analogies

Louis Fournier
Cognitive Machine Learning
CNRS-EHESS-INRIA
PSL Research University
Paris, France
<EMAIL_ADDRESS>
&Ewan Dunbar
Cognitive Machine Learning
CNRS-EHESS-INRIA
PSL Research University
Paris, France
and
University of Toronto
Toronto, Canada
<EMAIL_ADDRESS>

###### Abstract

Many types of distributional word embeddings (weakly) encode linguistic
regularities as directions (the difference between _jump_ and _jumped_ will be
in a similar direction to that of _walk_ and _walked_ , and so on). Several
attempts have been made to explain this fact. We respond to Allen and
Hospedales’ recent (ICML, 2019) theoretical explanation, which claims that
word2vec and GloVe will encode linguistic regularities whenever a specific
relation of _paraphrase_ holds between the four words involved in the
regularity. We demonstrate that the explanation does not go through: the
paraphrase relations needed under this explanation do not hold empirically.

##  1 Introduction

The study of linguistic regularities in distributional word embeddings—that
the difference vector calculated between the vectors _jump_ and _jumped_ shows
a similar direction to that of _walk_ and _walked_ , and so on—has been both
stimulating and controversial. While a number of such regularities appear to
hold, across a number of different kinds of embeddings, the standard 3CosAdd
analogy test used to measure the presence of these regularities has come under
fire for confounding analogical regularities with unrelated properties of
semantic embeddings. It is thus important to note that several papers have
proposed theoretical explanations for why linguistic regularities _should_
hold in distributional word embeddings. Particularly in light of the
controversies over linguistic regularities, it is important to examine the
soundness of these arguments.

Allen and Hospedales (2019) develop such an explanation by linking the
semantic definition of an analogy to _paraphrases._ In the sense of Gittens et
al. (2017), paraphrases are sets of words which are semantically and
distributionally closely equivalent to another word or set of words—for
example, _king_ may be paraphrased by {_man_ , _royal_}. Allen and Hospedales
argue that the standard analogy criterion, that _king_ \- _man_ \+ _woman_ =
_queen_ , is equivalent to a criterion whereby {_king_ , _woman_} paraphrases
{_man_ , _queen_}. With this in mind, it becomes possible to rewrite the
arithmetic analogy criterion in terms of vectors encoding the pointwise mutual
information (PMI) between words and their contexts, and to decompose the error
in the analogy equality into several components, including a _paraphrase
error_ term measuring the degree to which the critical paraphrase holds.
Making use of an assumption that the word2vec embedding is a linear
transformation of the PMI matrix, they argue that results in terms of PMI
apply to word vectors. Thus, under their explanation, a major part of success
on an analogy $a-a^{*}+b^{*}=b$ is due to ${a,b^{*}}$ and ${a^{*},b}$ being
close distributional paraphrases.

We first review the literature on the analogy test itself, underlining known
pitfalls which any explanation of linguistic regularities must navigate. We
then show empirically that the relation between the PMI matrix and word2vec
embeddings is to some degree linear, which may be enough to satisfy the
assumption of Allen and Hospedales (2019). We further examine the proposed
decomposition into error terms. We demonstrate that, empirically, these error
terms tend to be undefined due to data sparseness, undermining their
explanatory force. Most importantly, examining a number of analogies which
pass the standard test, we show that the critical paraphrase error term is,
contrary to the proposed explanation, very large.111Code is available at
www.github.com/bootphon/paraphrases_do_not_explain_analogies.

##  2 Related work

Early works proposing explanations of the analogical properties of word
embeddings include Mikolov et al. (2013b) and Pennington et al. (2014). A
geometrical explanation is proposed by Arora et al. (2016), but this
explanation relies on very strong preconditions, notably, that the word
vectors be distributed uniformly in space. Ethayarajh et al. (2019) also
propose an explanation, providing a link between the PMI and the norm of word
embeddings. However, as pointed out by Allen and Hospedales (2019), this
explanation, too, rests on strong assumptions. Notably, the words involved in
the analogy are required to be coplanar, a property that seems unlikely in
light of the lack of parallelism we discuss in the next section.

##  3 Issues with the test

Issues have arisen with the standard way of measuring linguistic analogies.
Levy and Goldberg (2014a), Vylomova et al. (2016), Rogers et al. (2017), and
Fournier et al. (2020) all demonstrate that the standard 3CosAdd measure
conflates several very different properties of embeddings, simultaneously
measuring not only the directional regularities suggested by typical
illustrations of vectors in a parallelogram, but also the similarity of
individual matched pairs such as _king, man_ , as well as the global
arrangement of vectors in semantic fields, such as _king, queen, prince, …_
versus _man, woman, child, …_ in distinct regions of the space. These issues
undermine the construct validity of the standard analogy test. This conflation
of properties explains certain pathological behaviours of the test Linzen
(2016); Rogers et al. (2017). In spite of these issues, Fournier et al. (2020)
demonstrate, using alternative measures, that linguistic regularities are
nevertheless coded by directional similarities. This parallelism is weak, with
directions tending to be closer, in the absolute, to being orthogonal than to
being parallel, but is present above chance level (unmatched word pairs).

Thus, before turning to Allen and Hospedales (2019), one of a number of
theoretical attempts to explain performance on the 3CosAdd objective, we
underscore that such demonstrations run the risk of explaining properties of
the test which may be of secondary interest, or, conversely, of placing undue
emphasis on the role of directional regularities, which have been shown to
play only a small role in success on 3CosAdd.

##  4 Explaining analogies through paraphrases

For a word $w_{i}$ and a word $c_{j}$ which can appear in the context of
$w_{i}$, the pairwise mutual information $PMI(w_{i},c_{j})$ is defined as
$\log\frac{p(w_{i},c_{j})}{p(w_{i})p(c_{j})}$. As shown by Levy and Goldberg
(2014b), skip-gram word2vec with negative sampling factorizes the PMI: PMI
$\approx W^{\top}\cdot C$, with $W$ and $C$ the word and context embedding
matrices of a word2vec model.

For two pairs of words $(a,a^{*})$ and $(b,b^{*})$ from the same semantic
relation, the standard arithmetic analogy test criterion is that
$a-a^{*}+b^{*}=b$. Writing
$\mathcal{W}=\\{a,b^{*}\\},\mathcal{W_{*}}=\\{a^{*},b\\}$, and
$\textbf{PMI}_{x}$ the PMI vector of $x$, Allen and Hospedales (2019) show
that is possible to rewrite the arithmetic analogy formula with PMI vectors,
and to decompose the error in the equality into five terms as follows:

$\begin{split}\textbf{PMI}_{b^{*}}=&\textbf{PMI}_{b}+\textbf{PMI}_{a^{*}}-\textbf{PMI}_{a}\\\
+&\underbrace{\rho^{\mathcal{W,W_{*}}}}_{\text{Paraphrase
error}}+\underbrace{\sigma^{\mathcal{W}}-\sigma^{\mathcal{W_{*}}}}_{\text{Conditional
dependence error}}\\\
+&\underbrace{(\tau^{\mathcal{W}}-\tau^{\mathcal{W}_{*}})\textbf{1}}_{\text{Mutual
dependence error}}\end{split}$ (1)

The error terms are vectors of length $\left|\mathcal{V}\right|$ (vocabulary
size), with each element $j$ defined as:

$\begin{split}\rho^{\mathcal{W,W_{*}}}&=\log\frac{p(c_{j}|\mathcal{W_{*}})}{p(c_{j}|\mathcal{W})}\\\
\sigma^{\mathcal{W}}&=\log\frac{p(\mathcal{W}|c_{j})}{\prod_{\mathcal{W}}p(w_{i}|c_{j})}\\\
\tau^{\mathcal{W}}&=\log\frac{p(\mathcal{W})}{\prod_{\mathcal{W}}p(w_{i})}\end{split}$
(2)

The authors claim that these terms can be embedded linearly into a word2vec
embedding space by multiplying them by the Moore-Penrose pseudo-inverse
$C^{\dagger}$ of the context matrix $C$. Then with $\textbf{w}_{x}$ the
word2vec embedding of $x$,
$C^{{\dagger}}\cdot\textbf{PMI}_{x}\approx\textbf{w}_{x}$. Thus we get the
final decomposition:

$\begin{split}\textbf{w}_{b^{*}}=&\textbf{w}_{b}+\textbf{w}_{a^{*}}-\textbf{w}_{a}+\\\
&C^{{\dagger}}\left(\rho^{\mathcal{W,W_{*}}}+\sigma^{\mathcal{W}}-\sigma^{\mathcal{W_{*}}}-(\tau^{\mathcal{W}}-\tau^{\mathcal{W_{*}}})\textbf{1}\right)\end{split}$
(3)

The paraphrase error term is claimed to be small for successful analogies.
Elaborating on the notation, $\mathcal{W}$ is taken to paraphrase
$\mathcal{W_{*}}$ if, wherever all $w\in\mathcal{W}$ appear together, we
observe the same distribution of surrounding words as for $\mathcal{W_{*}}$.
The paraphrase error assesses the similarity of the distributions of words in
the context of $\mathcal{W}$ (all words in $\mathcal{W}$ appearing together)
versus $\mathcal{W_{*}}$.

##  5 Linearity of the link between PMI and word2vec

Though it is true that there is a relation between the word2vec matrices
$W^{\top}\\!\cdot\\!C$ and the PMI matrix, in practice the link is more
complicated than simple linear matrix factorization, due in part to the
training tricks described in Mikolov et al. (2013a). The result of Allen and
Hospedales (2019) requires that the embedding from PMI vectors to word2vec
embeddings be “linear enough” for $C^{\dagger}\cdot$ PMI to approximate $W$.

To assess this, we use the _text8_ corpus 222 A text dataset composed of 100
million characters from Wikipedia: Mahoney (2006). both to train word2vec
embeddings 333Skip-gram architecture with negative sampling (1 word), negative
sampling exponent equal to 1, no undersampling of common words, and a high
dimension size of 500. These parameters allow us to be as close as possible to
a direct factorization of the PMI matrix. and to estimate a PMI matrix. We
replace infinite values in the PMI matrix by 0. In Figure 1(a), we show the
distribution of the Pearson correlation coefficient (assessing the presence of
a linear relation) between the word2vec embedding and the corresponding row of
$C^{\dagger}\cdot$ PMI for the top ten thousand words in the corpus. As can be
seen from the figure, the correlation tends to be between 0.5 and 0.8. For
instance in Figure 1(b), the word2vec embedding for _king_ is plotted against
the row of $C^{\dagger}\cdot$ PMI corresponding to _king_.

While the relation is not perfectly linear—many words have a correlation of
around 0.55, far lower than that of _king_ —the empirical relations shown here
leave open the possibility that it may indeed be “sufficiently linear” to be
taken for granted. However, while linearity is necessary for the result of
Allen and Hospedales (2019) to go through, it is not sufficient. In the next
section, we assess the critical question of whether the paraphrase error is
small enough to serve as an explanation for the success of linguistic
analogies.

(a) Figure 1a: Histogram of the Pearson correlations between true and
approximated word2vec embeddings for the top ten thousand words in the _text8_
corpus. The mean value is $0.643$ and the variance is $0.014$.

(b) Figure 1b: Plot of the values of the word2vec embedding for _king_ ,
versus coefficients for the row of $C^{\dagger}\cdot$ PMI corresponding to
_king_ , for word2vec matrices trained on the same corpus (_text8_). The
Pearson correlation is one of the best possible at 0.825.

##  6 Empirical analysis of the error terms

We now seek to examine the proposed explanation by calculating the proposed
error terms empirically. However, in practice, many of the terms are
undefined, since they rely on cooccurrences unattested in practical corpora.
The most extreme situation occurs when the two words of a paraphrase
$\mathcal{W}=\\{w_{1},w_{2}\\}$ are _never_ present in the same context window
in the corpus. We found that only 16% of the paraphrase sets associated with
the BATS analogy set Gladkova et al. (2016)—for example, _king, woman_ —were
present together in the _text8_ corpus in a context window of length five. We
refer to such paraphrase sets as “well-defined” with respect to the corpus.
The problem of zero co-occurrence counts was anticipated by Allen and
Hospedales (2019), who propose to restrict their analysis to the case where
the context window is sufficiently large that all relevant terms are well
defined. We stress that our trained word2vec vectors are also trained with a
context window of five, and yield expected levels of performance on the BATS
analogy test, despite having access to little training data on which to model
co-occurrences such as _king, woman_ , _queen, man_ , and so on.

Category |  I01 |  I02 |  I05 |  I06 |  I07 |  I08 |  I09 |  I10 |  D02 |  D03 |  D05 |  D08 |  D10 |  E01 |  E02
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Paraphrase error norm |  177 |  153 |  111 |  127 |  126 |  124 |  138 |  97 |  102 |  122 |  130 |  110 |  107 |  124 |  176
Dependence errors sum norm |  1006 |  938 |  867 |  903 |  957 |  883 |  952 |  908 |  856 |  893 |  514 |  585 |  699 |  749 |  848
All errors sum norm |  1032 |  957 |  878 |  917 |  970 |  897 |  966 |  916 |  864 |  905 |  539 |  602 |  710 |  765 |  875

Category |  E03 |  E04 |  E05 |  E08 |  E09 |  E10 |  L02 |  L03 |  L04 |  L05 |  L06 |  L07 |  L08 |  L09 |  L10
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Paraphrase error norm |  162 |  176 |  155 |  229 |  179 |  190 |  197 |  189 |  209 |  206 |  133 |  169 |  185 |  175 |  432
Dependence errors sum norm |  866 |  797 |  519 |  739 |  910 |  833 |  642 |  982 |  907 |  1103 |  921 |  995 |  1044 |  1017 |  1302
All errors sum norm |  889 |  822 |  553 |  778 |  933 |  865 |  683 |  1007 |  939 |  1131 |  937 |  1016 |  1066 |  1040 |  1416

Table 1: L2 norms of the error terms in 1, following our implementation.

Category |  I01 |  I02 |  I05 |  I06 |  I07 |  I08 |  I09 |  I10 |  D02 |  D03 |  D05 |  D08 |  D10 |  E01 |  E02
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Average rank |  7762K |  7589K |  7759K |  8744K |  8160K |  6454K |  7028K |  11889K |  31952K |  19558K |  7857K |  1506K |  2556K |  4394K |  9507K
Median rank |  1630K |  2195K |  3055K |  3239K |  2530K |  4090K |  3004K |  4535K |  6754K |  3564K |  3260K |  1506K |  2556K |  2117K |  1622K

Category |  E03 |  E04 |  E05 |  E08 |  E09 |  E10 |  L02 |  L03 |  L04 |  L05 |  L06 |  L07 |  L08 |  L09 |  L10
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Average rank |  1305K |  5611K |  9192K |  727K |  8421K |  11946K |  52183K |  1857K |  12687K |  6747K |  2475K |  7727K |  4502K |  4679K |  16871K
Median rank |  695K |  1703K |  1426K |  854K |  1908K |  169K |  52182K |  1261K |  2460K |  1343K |  2255K |  2136K |  1549K |  1 739K |  785K

Table 2: For an analogy equivalent to two paraphrases $\mathcal{W}$ and
$\mathcal{W_{*}}$, the rank of $\mathcal{W_{*}}$ in the list of the closest
paraphrases to $\mathcal{W}$ with respect to the L2 norm of the paraphrase
error vector. 7762K means a rank of 7762000, rounded to the nearest thousand.

At a minimum, if the proposed explanation holds, the cases for which the error
terms are empirically well-defined should show signs of the paraphrase error
being relatively small. We now detail how we implemented the error terms in
cases for which they were well-defined. We count co-occurrences
$N(w_{i},w_{j},w_{k})$ in _text8_ for all triplets of words
$w_{i},w_{j},w_{k}$, with $w_{k}$ at the center of the context window, and
$W=\\{w_{i},w_{j}\\}$ any paraphrase, both occurring anywhere within a context
window of width five. We restrict analysis to the ten thousand most frequent
word types $w_{i}$ and $w_{j}$, yielding $10^{8}$ possible
paraphrases.444$w_{k}$ is allowed to vary over all of the types included in
the training for word2vec, of which there are 71290. Thus, for each
paraphrase, the error vectors have 71290 elements, one for each vocabulary
word. We use the relative frequencies as estimators of
$p(w_{k}|\\{w_{i},w_{j}\\})$ and $p(\\{w_{i},w_{j}\\}|w_{k})$, and marginalize
to obtain $p(w_{i}|w_{k})$, $p(\\{w_{i},w_{j}\\})$ and $p(w_{k})$. The error
terms follow. Since this can still lead to ill-defined elements, we replace
$log(+\infty)$ and $log(0)$ by $+/-log(\epsilon)$, with $\epsilon=10^{-15}$
(within reason, the value of $\epsilon$ is immaterial). We also replace
$log(0/0)$ with 0.

Table 1 shows the mean and median values of the L2 norms of the paraphrase
error vectors across several categories of the BATS dataset. We compare them
with the sum of the four dependence error terms (the dependence error reflects
statistical dependencies within $\mathcal{W}$ and $\mathcal{W_{*}}$ irrelevant
to the analogy), as well as the sum of all five error terms (equal to the
difference between the PMI of $\mathcal{W}$ and $\mathcal{W_{*}}$).The
paraphrase error is indeed smaller than the other error terms. However, as we
now show, the paraphrase error is not small _enough_ to contribute
substantially to the success of analogies.555We note also that the error
values seem relatively consistent between categories, while success on the
analogy test varies differ greatly between categories.

Take the norm of the paraphrase error vector $\rho$ as a measure of the
divergence in the PMI between two paraphrases. For an analogy with associated
paraphrases $\mathcal{W}$ and $\mathcal{W_{*}}$, we assess how many
paraphrases are closer to $\mathcal{W}$ than to $\mathcal{W_{*}}$ by
calculating the rank of the norm of $\rho^{\mathcal{W,W_{*}}}$ among all
$\rho^{\mathcal{W,X}}$, where $\mathcal{X}$ spans over all pairs of words
constructible from the top ten thousand most frequent words in the corpus. To
do so, we define a Paraphrase Conditional Information matrix (PCI). For
$W_{ij}=\\{w_{i},w_{j}\\}$ and $w_{k}$, we define $PCI(l_{ij},k)$, the value
at column $l_{ij}$ and row $k$ to be $log(p(W_{ij}|w_{k}))$, where with
$l_{ij}$ is a unique index associate with tuple $(i,j)$. We compute only the
positive PCI, to obtain a sparse matrix. The difference between two PCI
columns is a paraphrase error vector, and their Euclidean distance is the norm
of the paraphrase error.

We now compute, for each analogy, the distance between the PCI column of
$\mathcal{W}$ and every other column (paraphrase) of the PCI matrix. We
calculate the rank of the true analogy pair $\mathcal{W_{*}}$. Given that the
analogy test generally succeeds in picking out $b$ as being the most similar
to $a-a^{*}+b^{*}$ out of the entire vocabulary (modulo Linzen 2016), we would
expect that, for successful analogies, the paraphrase error for the true
analogy would be among the highest, if small paraphrase error were the
explanation for success. Table 2 displays the mean of this rank within each
BATS category. The rank is extremely low (in the millions), making the
paraphrase error in true analogies far too high to be the explanation for
their success.666Limiting the search to the paraphrases composed by at least
one of the words of $W_{*}$ still results in a very low rank for $W_{*}$.

##  7 Conclusion

Recent work has shown that, in spite of the standard analogy test’s confound
with simple vector similarity, distributional word vectors genuinely do encode
linguistic regularities as directional regularities above and beyond vector
similarity (Fournier et al., 2020). Further research is warranted into the
mechanisms by which distributional word embeddings come to show these
regularities. However, the analysis of analogies as paraphrases does not hold
up as an explanation of performance on the analogy test—nor would an
explanation of performance on the 3CosAdd analogy test be a satisfying result,
since the test is not a useful measure to begin with.

## Acknowledgments

This work was funded in part by the European Research Council
(ERC-2011-AdG-295810 BOOTPHON), the Agence Nationale pour la Recherche
(ANR-17-EURE-0017 Frontcog, ANR-17-CE28-0009 GEOMPHON, ANR-10-IDEX-0001-02
PSL*, ANR-19-P3IA-0001 PRAIRIE 3IA Institute, ANR-18-IDEX-0001 U de Paris,
ANR-10-LABX-0083 EFL) and grants from CIFAR (Learning in Machines and Brains),
Facebook AI Research (Research Grant), Google (Faculty Research Award),
Microsoft Research (Azure Credits and Grant), and Amazon Web Service (AWS
Research Credits).

## References

  * Allen and Hospedales (2019) Carl Allen and Timothy Hospedales. 2019.  Analogies Explained: Towards Understanding Word Embeddings.  In _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 223–231, Long Beach, California, USA. PMLR.
  * Arora et al. (2016) Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2016.  A Latent Variable Model Approach to PMI-based Word Embeddings.  _Transactions of the Association for Computational Linguistics_ , 4:385–399.
  * Ethayarajh et al. (2019) Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. 2019.  Towards Understanding Linear Word Analogies.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3253–3262, Florence, Italy. Association for Computational Linguistics.
  * Fournier et al. (2020) Louis Fournier, Emmanuel Dupoux, and Ewan Dunbar. 2020.  Analogies minus analogy test: measuring regularities in word embeddings.  In _Proceedings of the 24th Conference on Computational Natural Language Learning_ , pages 365–375, Online. Association for Computational Linguistics.
  * Gittens et al. (2017) Alex Gittens, Dimitris Achlioptas, and Michael W. Mahoney. 2017.  Skip-gram âˆ’ Zipf + uniform = vector additivity.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 69–76, Vancouver, Canada. Association for Computational Linguistics.
  * Gladkova et al. (2016) Anna Gladkova, Aleksandr Drozd, and Satoshi Matsuoka. 2016.  Analogy-Based Detection of Morphological and Semantic Relations with Word Embeddings: What Works and What Doesn’t.  In _Proceedings of the NAACL-HLT SRW_ , pages 47–54, San Diego, California, June 12-17, 2016. ACL.
  * Levy and Goldberg (2014a) Omer Levy and Yoav Goldberg. 2014a.  Linguistic Regularities in Sparse and Explicit Word Representations.  In _Proceedings of the Eighteenth Conference on Computational Natural Language Learning_ , pages 171–180, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Levy and Goldberg (2014b) Omer Levy and Yoav Goldberg. 2014b.  Neural word embedding as implicit matrix factorization.  In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 27_ , pages 2177–2185. Curran Associates, Inc.
  * Linzen (2016) Tal Linzen. 2016.  Issues in Evaluating Semantic Spaces Using Word Analogies. In _Proceedings of the First Workshop on Evaluating Vector Space Representations for NLP_. Association for Computational Linguistics.
  * Mahoney (2006) Matt Mahoney. 2006.  Large text compression benchmark: About the test data (web page).
  * Mikolov et al. (2013a) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a.  Efficient Estimation of Word Representations in Vector Space.  In _Proceedings of International Conference on Learning Representations (ICLR)_.
  * Mikolov et al. (2013b) Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b.  Linguistic Regularities in Continuous Space Word Representations. In _Proceedings of NAACL-HLT 2013_ , pages 746–751, Atlanta, Georgia, 9–14 June 2013.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global Vectors for Word Representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Rogers et al. (2017) Anna Rogers, Aleksandr Drozd, and Bofang Li. 2017.  The (Too Many) Problems of Analogical Reasoning with Word Vectors.  In _Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (* SEM 2017)_ , pages 135–148.
  * Vylomova et al. (2016) Ekaterina Vylomova, Laura Rimell, Trevor Cohn, and Timothy Baldwin. 2016.  Take and Took, Gaggle and Goose, Book and Read: Evaluating the Utility of Vector Differences for Lexical Relation Learning.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1671–1682, Berlin, Germany. Association for Computational Linguistics.