# GenVideo: One-shot target-image and shape aware video editing using T2I
diffusion models

Sai Sree Harsha
Adobe
<EMAIL_ADDRESS>Ambareesh Revanur
Adobe
<EMAIL_ADDRESS>Dhwanit Agarwal
Adobe
<EMAIL_ADDRESS>Shradha Agrawal
Adobe
<EMAIL_ADDRESS>
###### Abstract

Video editing methods based on diffusion models that rely solely on a text
prompt for the edit are hindered by the limited expressive power of text
prompts. Thus, incorporating a reference target image as a visual guide
becomes desirable for precise control over edit. Also, most existing methods
struggle to accurately edit a video when the shape and size of the object in
the target image differ from the source object. To address these challenges,
we propose “GenVideo” for editing videos leveraging target-image aware T2I
models. Our approach handles edits with target objects of varying shapes and
sizes while maintaining the temporal consistency of the edit using our novel
target and shape aware InvEdit masks. Further, we propose a novel target-image
aware latent noise correction strategy during inference to improve the
temporal consistency of the edits. Experimental analyses indicate that
GenVideo can effectively handle edits with objects of varying shapes, where
existing approaches fail.

Figure 1: Editing the source car video based on a target-image and a text
prompt Driving a “white bus”/“red cadillac” down a mountain road with GenVideo
method. GenVideo can accurately replace the object in the source video, even
with a target object of a different shape.

##  1 Introduction

Image and video generation using diffusion models [12, 8, 7, 25, 26, 30, 18]
has gained immense popularity in recent times. Recently, many methods have
been proposed for editing visual content like images [35, 33, 36], and videos
[37, 47, 43, 5] using text prompts. Although several works have showcased
high-quality results for the task of image editing [1, 6, 11, 41, 28], the
utilization of diffusion models for the task of video editing has been
limited. Works such as [24, 8], use text-to-video (T2V) diffusion models for
video editing. However, these methods have limited utility since they demand a
large computational infrastructure for training along with large and diverse
video dataset. Due to these limitations, more recent methods have started
exploring inflated text-to-image (T2I) diffusion models as an alternative to
T2V models for video editing in a one-shot or zero-shot fashion [43, 31, 46,
21, 5, 9, 2, 4]. Nonetheless, with the exception of Make-A-Protagonist [46]
which can be additionally conditioned on a target-image, existing and
concurrent video editing methods that are based on T2I diffusion models focus
exclusively on text-driven video editing. Therefore, these methods are not
suitable for scenarios where the nature of the edit cannot be accurately
expressed using only text. In other words, they are not target-image aware. We
emphasize here that target-image awareness provides a precise visual guide for
the desired edit. This approach enables exact replication of shapes, sizes,
and textures, essential for creating content that serves use cases like
creating content that aligns with a particular product specifics (like a
particular model of a car). Further, previous approaches lack shape awareness
and therefore, fail to make edits where the shape and size of the target-
object is substantially different from the source video.

In this paper, we propose a novel target-image and shape aware InvEdit method
to effectively identify a region-of-interest in the video for a given text
prompt and the target-image. This is done using the underlying T2I diffusion
model similar to [6] but for videos and using additional image conditioning.
The strength of our method lies in InvEdit and its simplicity with which it
can be adapted to any image conditioned diffusion model to get target-image
and shape aware mask for video editing. Localized mask guidance is often
crucial for precise edits and avoiding leakage of edits to the unmasked region
but obtaining target-image and shape aware masks is a challenging problem due
to which previous methods are not able to handle edits of substantially
different shapes and sizes.

We further address the key challenge of maintaining the temporal consistency
in the edited video. Some of the previous works in this research area fail to
effectively address this challenge because of their reliance on source-based
Neural Layer Atlases (NLA) [5] or source-based inter-frame feature propagation
[9]. Hence, the temporal consistency of the video generated by these methods
is primarily dictated by the source video. Naturally, these methods fail when
the target-object is substantially different from the source. To address these
concerns, we introduce a novel target-image aware _latent correction_ strategy
to blend the inter-frame latents of the diffusion model on the fly during
inference to improve the inter-frame temporal consistency of the target-object
in the edited video. Our approach uses _InvEdit_ mask guidance to enable
temporal consistency even when the target-object has a different shape and
size than the object in the source video. We show extensive quantitative and
qualitative comparisons with state-of-the-art video editing methods and
conclude that our approach outperforms previous and concurrent approaches on a
diverse set of videos. Our contributions can be summarized as follows:

  * •

We introduce _GenVideo_ , the first pipeline for target-image and shape aware
editing of videos using image diffusion models that can make temporally
consistent edits for target objects of varied shapes and structures.

  * •

We introduce _InvEdit_ , a novel target-image and shape aware, zero-shot
approach for generating video masks using image diffusion models. This mask
enables us to identify the region of interest for editing objects of varying
shapes and sizes.

  * •

We introduce a target-image and shape aware _latent correction_ strategy to
enforce inter-frame temporal consistency in the edited video, where existing
approaches fail.

##  2 Related Work

Text Driven Image and Video Editing: Before the advent of the diffusion models
[38, 12], generative adversarial networks (GANs) have been used to edit images
[27, 44, 15, 29, 42, 34]. Recently, generative diffusion models [33, 35, 36]
trained on internet scale image and text data have achieved significantly
higher quality and diversity of image generation than GANs. Inspired by the
success of these T2I diffusion models, text-to-video diffusion models [13, 37,
14, 22] have been developed for video generation. Authors in [37] inflate the
pretrained T2I diffusion models with additional temporal attention layers to
generate videos, while the recent works like [13] use a 3D-UNet for video
generation. Along with generation, editing of existing images and videos has
also received significant interest. SDEdit [23] carefully adds noise to an
input image and then denoises them using diffusion models to achieve the
desired edits. Prompt-to-prompt (P2P) [11] allows text guided editing of
images by controlling the cross-attention maps. In the realm of video editing,
Tune-A-Video (TAV) [43] finetunes the inflated T2I model on an individual
source video to generate edited videos with similar motion. Video-P2P [21] and
FateZero [31] achieve controllable edits in videos by altering the cross-
attention maps. Pix2Video [4] achieves video editing by altering the self-
attention features during inference. Text2Live [2] utilizes NLA and then edits
each layer independently using text guidance. Text2Video-Zero [16] uses
inflated self-attention layers along with ControlNet [45] to do text guided
video editing. Nonetheless, these works using T2I diffusion models exclusively
focus on image and video editing via text, and cannot be used for edits which
are indescribable by text alone. Hence, they are not target-image aware which
is often a crucial aspect for accurate video editing [46].

Target-Image Aware Visual Content Editing: To cater to visual content edits
that isn’t easily or accurately described by text alone, models like DALLE–2
[33] and Gen–1 [8] have been developed to make target-image aware edits to
images and videos by leveraging the CLIP [32] image embedding. While DALLE–2
edits images, Gen–1 can edit videos but the drawback here is that the entire
image, including the background, influences the edits due to the the lack of
mask guidance. Subject driven solutions like Dreamix [24], which can edit
videos conditioned on the target-image data, typically require multiple target
images for learning the concept/target object accurately and also require
large video diffusion models as used in [13]. While they can generate and edit
videos based on a single image, the edited videos fail to maintain the
fidelity of the edits to the target-object in the target-image. Make-A-
Protagonist [46] addresses these limitations by using an image guided T2I
diffusion model called SD-unCLIP [33]. Their pipeline can achieve target-image
aware video object editing using a single target-image and employ mask-guided
inference to do localized edits. However, the edits still suffers from lack of
temporal consistency and the approach fails when the target object differs in
shape and size from the source object present in the source video. This is
because the mask used for editing the video is generated only using the source
video and has no awareness of the target-object.

Video Editing with Temporal Consistency: Aforementioned methods suffer from
lack of inter-frame temporal consistency in the edited video. A concurrent
work, _TokenFlow_ [9] improves the temporal consistency by inter-frame
propagation of the self-attention features using a nearest neighbor field in
the feature space of source video. However, both the editing and temporal
consistency in _TokenFlow_ are not target-image and shape aware. Thus, it
cannot make edits where the target-object differs a lot in shape and size to
the source object. Moreover, it does not offer a way to localize the edits due
to the lack of mask guidance and also requires manually choosing key frames
for imposing temporal consistency. _StableVideo_ [5] uses NLA of the source
video to enforce temporal consistency and hence, also lacks target-image and
shape awareness for accurate shape changing edits. Another work [19] addresses
shape-aware editing by formulating a deformation map which maps the original
atlas into edited atlas. While the method achieves favorable results, NLA
mapping takes tens of hours of training and an inaccurate NLA mapping results
in objects with missing parts and substantial undesirable background changes.
Our work _GenVideo_ addresses these limitations by providing the first
“target-image and shape aware” video editing using T2I models with only _30
mins of training and 2 mins of inference_ using localized mask guidance to
avoid background changes.

Figure 2: Overview of GenVideo. Inflated attention layers are finetuned during
source video finetuning. During inference, _InvEdit_ predicts a region to edit
and _latent correction_ uses that mask to improve the inter-frame temporal
consistency. $\mathcal{M}_{\phi}$ \- “no mask”.

##  3 Approach

GenVideo aims to edit the given source video based on a target text prompt and
a target-image containing an object of any shape while maintaining temporal
consistency. More formally, given an input source video
$\mathcal{V}^{src}=[I^{src}_{1},\cdots,I^{src}_{N}]$ composed of $N$ frames
containing a source object, a source text prompt $\mathcal{P}^{src}$
describing the source video, a target-image $I^{trg}$ containing the target-
object and a target text prompt $\mathcal{P}^{trg}$ describing the desired
edit to the source video, GenVideo generates a target video
$\mathcal{V}^{trg}=[I^{trg}_{1},\cdots,I^{trg}_{N}]$ which preserves the
motion of the input source video but replaces the source object with a new
target-object from the target-image. The entire training and inference
pipeline is summarized in Fig. 2. First, we fine-tune an inflated Stable
Diffusion unCLIP (SD-unCLIP) model [33, 46] on the source video using the
reconstruction loss from standard LDMs [35] (Sec. 3.1). Then, we employ our
novel target-image and shape aware mask generation approach, called InvEdit,
where we use the fine-tuned model to infer a region-of-interest where the
edits need to be localized (Sec. 3.2). Finally, we introduce a novel _latent
correction_ approach to improve the inter-frame temporal consistency (Sec.
3.3).

Figure 3: InvEdit approach - the mask is generated by first iteratively
computing noise differences across multiple timesteps for the source denoising
branch and target denoising branch. Then, these differences are averaged and
binarized to obtain the InvEdit mask. Figure 4: _Latent correction_ strategy
corrects the latent noise using UNet features from the previous and successive
video frame. Correspondence Error (CE) map computed using the ground truth
(source video correspondences) shows that the DDIM o/p of the model before
correction has high CE (E.) which our latent correction strategy fixes (F.)
using Up-block-2 feature correspondences which have low CE (D.).

###  3.1 Fine-tuning on the source video

The finetuning procedure of the inflated pretrained T2I diffusion model is in
line with the TAV method [43] (Fig. 2). Unlike [43], we use the SD-unCLIP
model [33] which conditions the generation on both target-image and the text
prompt [46]. This model uses CLIP-vision branch to obtain an image embedding
$\mathcal{J}^{*}$ for an input reference image and uses the CLIP-text branch
to obtain the text embedding $\mathcal{C}^{*}$. As part of the inflation
process, spatial self-attention is inflated into spatio-temporal attention
(ST-attn), and an additional temporal attention (T-attn) is introduced after
the ST-attn and cross-attention blocks. See Suppl. for details.

###  3.2 InvEdit mask generation

In this section, we describe the details of InvEdit – our novel zero shot,
target-image and shape aware mask generation strategy using the finetuned
diffusion model from Sec. 3.1. Existing methods[46, 31] compute masks using
the source video alone and hence have no shape awareness regarding the target
object’s relative shape and size (for example, changing a car to a bus).
InvEdit is an adaptation of DiffEdit [6] for videos and includes target-image
and shape awareness.

_InvEdit_ steps: First, we perform the DDIM inversion [38] to transform the
source video into the corresponding random latent noise
$\mathcal{Z}^{src}_{T}=[z^{src}_{T,1},z^{src}_{T,2},\cdots,z^{src}_{T,N}]$.
Then, we denoise the latent code $\mathcal{Z}^{src}_{T}$ using the fine-tuned
inflated SD-unCLIP model from Sec. 3.1 using deterministic DDIM sampling for
both the source and target branches as shown in Fig. 3. For the source branch,
we denoise $\mathcal{Z}^{src}_{T}$ using the source text prompt
$\mathcal{P}^{src}$ and a randomly chosen frame $I^{src}_{n}$ from
$\mathcal{V}^{src}$ as conditioning inputs for the fine-tuned SD-unCLIP model.
Similarly, in parallel, we repeat DDIM sampling using the target text prompt
$\mathcal{P}^{trg}$ and the foreground object in the target-image $I^{trg}$ as
conditioning inputs. We segment the foreground object from the target-image
using GroundedSAM [17, 20].

We compute the difference of the predicted noise by the denoising UNet model
(denoted by $\varepsilon_{\theta}$) obtained at each denoising timestep in the
source and target branches. More formally, for every $I^{src}_{n}$ in
$\mathcal{V}^{src}$, we compute
$\Delta\varepsilon_{t,n}=\mathop{abs}(\varepsilon^{src}_{t,n}-\varepsilon^{trg}_{t,n})$,
where $\textit{
}\varepsilon^{u}_{t,n}=\varepsilon_{\theta}\left({z}^{u}_{t,n},t,\mathcal{C}^{u},\mathcal{J}^{u}_{n}\right),u\in\\{src,trg\\}$.
At each DDIM denoising step we obtain,

$\displaystyle\mathcal{Z}^{src}_{t}=\mathrm{DDIM}(\mathcal{Z}^{src}_{t+1},\epsilon^{src}_{t+1}),\mathcal{Z}^{trg}_{t}=\mathrm{DDIM}(\mathcal{Z}^{trg}_{t+1},\epsilon^{trg}_{t+1}),$
(1)

where $t\in\\{T-1,\cdots,1\\}$ denotes the time step.

These differences $\Delta\varepsilon_{t,n}$, represented as heat maps in Fig.
3, are averaged over multiple denoising time steps and binarized to get the
target-aware InvEdit mask for each frame of the video. In Fig. 3, the InvEdit
mask is able to determine that edits are to be placed in a region shaped like
a truck rather than the car, as the truck is much bigger than the car. We
denote the masks for $N$ frames by
$\mathcal{M}^{inv}=[M_{1},M_{2},\cdots,M_{N}]$, where
$M_{n}=binarize(\mathop{mean}_{t\in[t_{i},t_{j}]}(\Delta\varepsilon_{t,n}))$.
The InvEdit mask is used to identify the region in which the target-image
embeddings are injected into the features of the ResNet blocks of the UNet
along with the timestep embeddings [33].

_InvEdit_ intuition: Our intuition follows DiffEdit [6] for text driven image
editing. We extend it for our target-image aware video editing use case.
During DDIM denoising, the SD-unCLIP model will yield different noise
estimates given different text and image conditioning. The noise estimates
will be different in regions where the latent code will eventually decode
different shapes, colors and textures depending on the conditioning. For the
background, on the other hand, there is little change in the noise estimates.
The difference between the noise estimates can thus be used to infer a mask
that identifies the parts of each video frame need to be edited.

###  3.3 Latent noise correction via self-consistency

While InvEdit mask is able to accurately identify the regions to edit, it does
not address the temporal consistency of the object within the region across
generated frames. As an example, consider the edit where the “silver car” is
edited to a “red cadillac” shown in Fig. 4. While the shape of the car
generated by InvEdit mask in Frame 6 (see Fig. 4 B) appears similar to Frame 5
(see Fig. 4A), it has a different stylistic appearance in the front and in the
side. One trivial way to address this is by computing the optical flow of the
source object in the video and then imposing this flow on the latent noise
features $\mathcal{Z}$. However, this solution will struggle considerably even
in typical situations where the target object is of a different shape. Thus,
the issue of temporal inconsistency becomes challenging given the model has
not seen the target-image before. We address it by introducing a latent
correction strategy during inference. This correction is a blending strategy
in the latent $\mathcal{Z}$ space to improve the inter-frame temporal
consistency of the edited video. This is a three-step process (also see
Supplementary Material (SM)):

  1. 1.

Inter-frame latent field computation: During each denoising time step $t$ of
inference, we utilize the features of the Up-Block-2 of the UNet denoted as
$[f^{t}_{1},\cdots,f^{t}_{N}]$ for estimating latent correspondence maps/field
using nearest neighbors between these features of the consecutive frames.
First, we compute the nearest neighbors field $\mathcal{N}_{i\pm}(\cdot)$
defined as
$\mathcal{N}^{t}_{i\pm}[p]:=\mathrm{argmax}_{q}{d}(f^{t}_{i}[p],f^{t}_{i\pm
1}[q]).$ This field is the mapping from the spatial locations $p$ in the
features of $i^{th}$ frame to its nearest neighbor $q$ (in terms of cosine
similarity ${d}$) in the features of the $(i\pm 1)^{th}$ frame. This field is
the mapping from the spatial locations $p$ in the features of $i^{th}$ frame
to its nearest neighbor $q$ in the features of the $(i\pm 1)^{th}$ frame.

  2. 2.

Blending using inter-frame latent field: Starting from the DDIM inversion of
the source video $\mathcal{Z}^{src}_{T}$, at each denoising timestep $t$
during inference, we blend the latents
$\mathcal{Z}_{t}=[z_{t,1},z_{t,2},\cdots,z_{t,N}]$ of adjacent frames inside
the InvEdit mask region in the latent space of the decoder $\mathcal{D}$ of
the VAE of SD-unCLIP. The blended latents
$\tilde{\mathcal{Z}}_{t}=[\tilde{z}_{t,1},\cdots,\tilde{z}_{t,N}]$ at timestep
$t$ are given by,

$\displaystyle\tilde{z}_{t,i}[p]$ $\displaystyle=w_{-1}(M_{i}\odot
z_{t,(i-1)}[\mathcal{\hat{N}}^{t}_{i-}(p)])+w_{0}(M_{i}\odot z_{t,i})$
$\displaystyle\quad+w_{1}(M_{i}\odot
z_{t,(i+1)}[\mathcal{\hat{N}}^{t}_{i+}(p)])$
$\displaystyle\quad+(1-M_{i})\odot z_{t,i},$

where $w_{-1},w_{0},w_{+1}$ are non-negative weight hyperparameters which add
up to 1 and $\mathcal{\hat{N}}^{t}_{i\pm}[p]$ is the nearest neighbors field
above upsampled to match the dimension of the $z_{t}$. This blending happens
at each inference timestep $t$ for $t\geq T-5$.

  3. 3.

Background preservation: We also correct the latent noise corresponding to
background regions using the inverse InvEdit mask, _i.e.,_ $(1-M_{i})$ and
denoise only the masked area [39]. This is achieved by retaining the clean
latent corresponding to the source video frame outside the masked region as
given by
$\tilde{z}_{t,i}=(1-M_{i})\odot\mathcal{E}({I}^{src}_{i})+(M_{i})\odot\tilde{z}_{t,i}$,
where $\mathcal{E}$ is the encoder of VAE. We skip this step when the
background of the target video is expected to be different from the source.

Why Up-block-2? We use the features of Up-block-2 since it demonstrates a low
correspondence error (CE) than the CE in the latent noise obtained after DDIM
step. Continuing the example of the car in Fig. 4, we first compute the
feature correspondences in consecutive frames using RAFT optical flow [40] on
the source video frames. This serves as a ground truth correspondence for this
example since the edited object has the same shape as the source object. Then
we compute the correspondences of the features of Up-block-2 in consecutive
frames and find that these features have a low CE rate as shown by the heatmap
in Fig. 4D. On the other hand, the CE rate is higher for latent noise computed
after the DDIM Step (Fig. 4E). With our latent correction strategy, the CE is
lowered since the proposed blending strategy improves the consistency of the
latent noise features across consecutive frames as demonstrated by Fig. 4C and
Fig. 4F. We show in our experiments that this process improves the temporal
consistency of the edited target video.

Figure 5: Source videos, Target images, InvEdit mask and the GenVideo results.
Top: “A person wearing a ~~gray~~ black shoe.”, and “A ~~man~~ panda rides a
kite surfboard in deep waters.” Bottom: “~~man~~ naruto skiing on snow”, and “
~~man~~ Tom Cruise walking down the street”.

##  4 Experiments

###  4.1 Implementation details

For all of our experiments, we use the SD-unCLIP version of Stable Diffusion
v2.1 T2I model [33]. We inflate the model with temporal layers as in [43]. See
SM for more details on model inflation. During source finetuning, we train the
inflated layers using $16$ source video frames with a learning rate of
$1\times 10^{-5}$ for $400$ iterations. We use DDIM sampler during inference
to obtain the InvEdit mask on the DDIM-inverted source video by computing the
mean of $\Delta\epsilon_{t}$ across $t\in[0.8\times T,T]$ where $T=50$ and use
a difference threshold of 0.6 to binarize the heatmap. During the _latent
correction_ , we restrict the nearest neighbour search to a $4\times 4$ window
of pixels and set $w_{-1}=0.1$, $w_{0}=0.8$ and $w_{1}=0.1$ across the
experiments. We also fix classifier-free guidance scale to $12.5$ during
training and inference.

Figure 6: Results of InvEdit on zero-shot image editing showing its capability
for target-objects of varying shape and size.

###  4.2 Applications

Shape-aware video object editing : Our approach can be used to do target-image
and shape aware video object editing. Fig. 5, along with the second series in
Fig. 1, showcase the outcomes of temporally consistent video object editing
using GenVideo. This task is difficult as the mask needs to be target-image
and shape aware to effectively identify the region of interest for localized
edits. For instance, in the _man_ $\longrightarrow$ _naruto_ , _man_
$\longrightarrow$ _panda_ and _silver car_ $\longrightarrow$ _white bus_
examples, InvEdit mask is able to accurately identify the mask for _naruto_ ,
_panda_ , and the _white bus_ respectively. Furthermore, our _latent
correction_ strategy is able to correct the UNet latents on the fly during
inference.

Zero-shot image editing: Moreover, our approach is also capable of zero-shot
image editing for objects of varying shapes and sizes. This can be observed in
Fig. 6, which demonstrates object shape changes (like _boat_ $\longrightarrow$
_sailboat_) in images. Additional examples and results are provided in the SM
we have made available.

Table 1: Quantitative comparison of the state-of-the-art video editing
methods. For the user study, _Text_ , _Image_ , and _Visual_ refer to the
average rank in target text alignment, target-image alignment, and visual
quality of the edited video respectively.

Method | Model scoring metrics $(\uparrow)$ | User Study $(\downarrow)$
---|---|---
$\mathrm{CLIP}$-$\mathrm{T}$ | $\mathrm{DINO}$ | $\mathrm{Temp}$ | Text | Image | Visual
TAV [43] | 0.238 | 0.236 | 0.957 | 3.6 | 3.3 | 4.2
StableVideo [5] | 0.234 | 0.189 | 0.980 | 4.3 | 4.3 | 3.7
TokenFlow [9] | 0.231 | 0.216 | 0.985 | 3.3 | 3.8 | 2.1
FateZero [31] | 0.235 | 0.262 | 0.951 | 3.9 | 3.6 | 3.4
Make-a-Pro [46] | 0.234 | 0.195 | 0.949 | 4.0 | 4.1 | 5.0
Ours _GenVideo_ | 0.241 | 0.374 | 0.967 | 1.7 | 1.8 | 2.3

###  4.3 Comparison with baselines

Figure 7: Qualitative comparison of GenVideo against SOTA. Shape aware methods
demonstrate the ability to edit the shape of object. Target-aware methods
condition the diffusion model on both target text prompt and target-image.
GenVideo combines both target awareness and shape awareness, thus allowing it
to infer shape and appearance based on the target-image and target text. This
offers much superior alignment to the target text and target-image. First two
columns show large shape variation while the the third shows results for same
shape edits.  Figure 8: Ablation study of the GenVideo method. Prompt: Driving
a ~~silver car~~ red bus down a mountain road.

We do qualitative and quantitative comparisons with the following five
existing approaches: 1) Tune-A-Video (TAV) [43], 2) StableVideo [5], 3)
TokenFlow [9], 4) FateZero [31] and 5) Make-A-Protagonist (MAP) [46]. We use
target reference image only with our method and [46].

Qualitative Results: Fig. 7 shows a comparison of our _GenVideo_ approach with
the above baselines. The edits in TAV[43] are not target-image aware and
cannot be localized due to lack of mask. Hence, it doesn’t guarantee fidelity
to the target-image and also ends up modifying the background substantially.
StableVideo is not target-image aware and hence, cannot generate indescribable
features of the target-image. While it provides temporal consistency using
NLA, these atlas are computed only using the source video and have no
awareness of target-object. Hence, they fail when the target-object has a
different shape and size. TokenFlow also does not provide temporal consistency
for target objects of varying shapes and sizes since the nearest neighbor
field they use is based solely on the source video frames. Besides both
TokenFlow and StableVideo are not target-image aware and hence, do not align
well with the target-image. FateZero can change shapes to some extent but
since their mask is again source video-based, they cannot create large
variations in the shape (like car to bus). MAP also offers source video based
mask guidance and hence, cannot substantially change shapes. Our results
perform temporally consistent video editing when the target-object has a
substantially different shape and size while maintaining fidelity to the
target-image.

Quantitative Results: Table 1 shows a quantitative comparison between our
approach and the state-of-the-art (SOTA) methods. We use trained CLIP [32]
model to compute the average similarity of generated video frames to target
text prompt which is denoted as CLIP-T in Table 1. To evaluate temporal
consistency, we use CLIP model to compute average inter-frame similarity score
for pairs of consecutive frames of the edited video. We denote this score as
_Temp_ in Table 1. To measure alignment with target-image, we use trained DINO
[3] to compute the similarity between the video frames and the target-image.
We also conduct a user study to evaluate the edited videos on three metrics:
target text alignment, target-image alignment, and visual quality of the
edited video. We asked 20 subjects to rank the edited videos for the six
approaches ( _i.e.,_ ours and the five baselines mentioned above) on the above
three metrics by asking each subject 9 sets of comparisons. The results are
tabulated in Table 1. These results show our proposed approach outperforms
existing approaches in both target text and target-image alignment. This is
reflected in higher CLIP-T and DINO scores as well as in the user study where
_GenVideo_ achieves the best average rank for text and image alignment.
_TokenFlow_ and _StableVideo_ achieved a marginally better score in inter-
frame temporal consistency (measured by _Temp_) but this is because they use
the source video ground truth for enforcing temporal consistency. This works
well for consistency but the output videos do not align well with the target-
image as reflected in much lower DINO scores. In the user study, _TokenFlow_
achieved the best rank for visual quality outperforming our approach by a
small margin. Again, this is because of the source video’s flow-based feature
propagation which leads to good quality videos even though they rank lower in
the text and image alignment. Our results show that our approach outperforms
the other approaches in text and image alignment while being close to the
state-of-the-art in temporal consistency and visual quality. Note that to be
fair we do not expect to be the best in temporal consistency and quality
because we are creating novel views of the target object from a single target
reference image. Rather, the strength of our method lies in being able to
create target-image and shape aware masks for localized video edits and
enforce temporal consistency in masks in a training-free fashion while
ensuring image and text alignment.

###  4.4 Ablations

We show the ablations for our InvEdit and latent correction strategies in Fig.
8. The first row shows the inputs, _i.e.,_ target-image of a red bus and the
source video of a silver car driving down a mountain road. The second row
shows the edited video frames with a finetuned inflated SD-unCLIP model with
target-image guidance without the _InvEdit_ mask. Note that without mask
localization, the fidelity of the edited bus is of low quality and the
background gets substantially modified. In third row, using InvEdit mask
localization, we notice that the fidelity of the bus to the target-image is
improved and the background is not substantially modified. However, we notice
that in the third column, the wheel of the bus disappears. Our _latent
correction_ strategy is able to correct such inconsistencies by imposing
temporal self-consistency between frames and preserving the background as
shown in the fourth row.

Figure 9: Limitation of GenVideo in fine-grained editing of videos. Prompt:
Driving a ~~silver car~~ yellow truck down the mountain road.

##  5 Conclusion

We introduce GenVideo, a pipeline for target-image and shape aware video
editing using image diffusion models. The proposed pipeline allows localized
edits with target-objects using the proposed InvEdit mask and enforces
temporal consistency between the frames using latent correction strategy.
Results show that GenVideo outperforms existing methods both qualitatively and
quantitatively on the video editing task. Limitations and Future Work: The
underlying SD-unCLIP model may have limitations regarding the quality and
diversity of the generated content, thus affecting the edit quality. The
_latent correction_ approach for inter-frame temporal consistency may not
completely eliminate fine-grained inconsistencies, especially for complex
objects like a truck with several wheels as shown in Fig. 9. Furthermore,
similar to previous other methods [31, 43], our method cannot generate
entirely new motion like changing motion from driving to flying. As part of
further work, we aim to test our approach with image conditioned video
diffusion models like [10].

Supplementary Material

##  6 Architecture details

This section provides an overview of the underlying model and explains how the
features are passed through the pipeline during training and inference.

Base model: We use the SD-unCLIP model, a fine-tuned version of the Stable
Diffusion v2.1 text-to-image model that accepts CLIP image embedding and the
text prompt as conditional input. The network broadly consists of the VAE
autoencoder $\\{\mathcal{E},\mathcal{D}\\}$, the latent denoising UNet
$\varepsilon_{\theta}(\cdot)$, and CLIP conditional models (image branch and
text branch) which extract an image embedding $\mathcal{J}$ and a text
embedding $\mathcal{C}$. As shown in Fig. 10, the UNet network consists of
down-block, mid-block, and up-block. Each of these blocks has 4, 1, and 4
subblocks respectively. Each of these subblocks typically constitutes two
ResNet blocks and two inflated attention network blocks arranged as shown in
Fig. 10. The only trainable components of the network belong to the inflated
attention modules explained in the subsequent subsections.

Feature resolutions: The VAE encoder $\mathcal{E}$ reduces the spatial
dimensions from 768 to 96. The down-blocks further reduce the spatial
dimensions to 24 while increasing the channel dimensions from 4 to 1280. The
mid-blocks maintain the spatial dimensions and channel dimensions. The up-
blocks increase the spatial dimensions to 96, while reducing the channel
dimensions to 4. The VAE decoder then increases the spatial resolution back to
768. The CLIP text embedding and image embedding is a vector of size 768.

UNet forward pass: The inputs to the UNet network
$\varepsilon_{\theta}(\cdot)$ are the latent noise from previous timestep
$z_{t}$, the sinusoidal timestep embedding $t_{emb}$, an optional mask
$\mathcal{M}$, the CLIP image embedding $\mathcal{J}$ and the CLIP text
embedding $\mathcal{C}$. The latent noise $z_{t}$ is forwarded into layers of
UNet network. At the end of a ResNet block, the hidden states are updated with
the timestep embedding and the optional image embedding information based on
the input mask –

  * •

When an input mask is not provided, _i.e._ , $\mathcal{M}=\mathcal{M}_{\phi}$:
In this situation, the hidden states are updated by adding the timestep
embedding $t_{emb}$ and the image embedding $\mathcal{J}$ to all spatial
locations of the hidden states.

  * •

When an input mask is provided, _i.e.,_ $\mathcal{M}\neq\mathcal{M}_{\phi}$:
In this situation, the regions that correspond to the background (_i.e.,_
where $\mathcal{M}=0$) are updated by adding the $t_{emb}$ and source image
embedding $\mathcal{J}^{src}$. Similarly, for the regions corresponding to the
foreground (_i.e.,_ where $\mathcal{M}=1$), the hidden states are updated by
adding the timestep embedding $t_{emb}$ and the target image embedding
$\mathcal{J}^{trg}$ as shown in Fig. 10.

Once the hidden states are updated, they are passed into inflated attention
blocks and the subsequent network layers. At the end of each denoising step of
UNet, a latent fusion step is performed when an input mask is provided. In the
next subsections, we explain the latent fusion method and the inflated model
architecture.

Figure 10: Architectural diagram. Top to bottom: UNet architecture with VAE,
UNet block architecture, Attention layer inflation of ST-attn, Cross-attn, and
T-attn.  Figure 11: Correspondence Error (CE) maps computed using ground truth
(source video correspondences) before correction across all blocks of UNet. We
find that Up-block-2 has the lowest CE. Figure 12: Zero-shot image editing
results on the brown bear using InvEdit mask. Background preservation is not
used here.

Latent fusion: Our latent fusion method follows from Make-A-Protagonist [46].
The latent fusion step helps improve the quality of the rendered object in the
edited video. First, UNet features are obtained using only the target image
embedding $\mathcal{J}^{trg}$ with no mask input to UNet to obtain
$\varepsilon_{\theta}(z,t,\mathcal{C},\mathcal{J}^{trg},\mathcal{M}_{\phi})$.
Next, UNet features are obtained using source image embedding
$\mathcal{J}^{src}$ and target image embedding $\mathcal{J}^{trg}$ along with
a mask $\mathcal{M}$ to obtain
$\varepsilon_{\theta}(z,t,\mathcal{C},\\{\mathcal{J}^{src},\mathcal{J}^{trg}\\},\mathcal{M})$.
Note here $\mathcal{J}^{src}$ is used for $\mathcal{M}=0$ region and
$\mathcal{J}^{trg}$ is used for $\mathcal{M}=1$ region. These outputs are
combined using the mask in the following manner:

$\displaystyle z_{t-1}$
$\displaystyle=\frac{1}{1+\mathcal{M}}\big{(}\mathcal{M}\odot\mathrm{DDIM}(\varepsilon_{\theta}(z_{t},t,\mathcal{C},\mathcal{J}^{trg},\mathcal{M}_{\phi}))$
$\displaystyle\quad~{}~{}~{}+\mathrm{DDIM}(\varepsilon_{\theta}(z,t,\mathcal{C},\\{\mathcal{J}^{src},\mathcal{J}^{trg}\\},\mathcal{M}))\big{)}$

Note that when the background is allowed to be changed (like in Fig. 14 and
Fig. 12), $\mathcal{J}^{src}$ is replaced with the CLIP image embedding of
DALLE-2 prior obtained from the target text $\mathcal{P}^{trg}$. In all the
other cases where the background is to be kept the same as the source, it is
the CLIP image embedding of the source video frame, _i.e.,_
$\mathcal{J}^{src}$. More details can be found in [46].

Inflated attention layers: We follow the inflation strategy laid out by Tune-
A-Video [43]. We expand the self-attention layers into spatio-temporal
attention (ST-attn) layers by inputting the features from the first frames
$g_{t,1}$ along with $g_{t,n-1}$ as shown in Fig. 10 for computation of
attention matrix. Here, $g$ denotes features of hidden states in the UNet.
Cross-attention layers continue to accept the text tokens from prompt
$\mathcal{C}$ along with $g_{t,n}$. We additionally introduce temporal self-
attention (T-attn) layers which are trained after permuting the temporal
dimensions and spatial dimensions of the mini-batch. The only trainable
weights in the entire pipeline are the query weights of ST-attn, query weights
of Cross-attn, and all the weights in the T-attn as shown in Fig. 10.

Additional details of training and inference pipeline: During training, the
source video is mapped into the VAE encoder’s latent space. A random timestep
is sampled and noise is added to the latents according to the forward
diffusion process. The text embeddings of the source prompt and the image
embeddings of a random source video frame are passed (as $\mathcal{C}$ and
$\mathcal{J}$ respectively) into the UNet and the mask is
$\mathcal{M}_{\phi}$. The reconstruction loss is imposed at the given timestep
as shown in Fig. 2 of the main paper. The gradients are backpropagated using
the AdamW optimizer to update the parameters of inflated attention modules
described earlier. The inference pipeline consists of two stages - InvEdit
mask computation and the latent correction. While computing the InvEdit mask,
the mask inputs to UNet are absent, _i.e.,_ $\mathcal{M}=\mathcal{M}_{\phi}$.
After computing the InvEdit mask $\mathcal{M}^{inv}$ it is passed into the
UNet for mask guided inference and latent correction, _i.e._ ,
$\mathcal{M}=\mathcal{M}^{inv}$. See Algorithm 1 for inference pseudo-code.

Figure 13: Additional results of GenVideo. Our approach can do object edits
when target-object has substantially different shape and size.  Figure 14:
Additional results of GenVideo on style editing of videos. Background
preservation is not used in B. and C. since the entire video is being edited.
Algorithm 1 GenVideo Inference

1:$\mathcal{V}^{src}:=[I^{src}_{1:N}],\mathcal{P}^{src},\mathcal{P}^{trg},I^{trg}$

2:$\varepsilon_{\theta}(\cdot)(\text{finetuned inflated
UNet}),\mathcal{E},\mathcal{D},\mathrm{CLIP_{t}},\mathrm{CLIP_{v}}$

3:Set Hyperparameters:

4: $T=50$ $\triangleright$ DDIM timesteps

5: $\alpha=0.8$ $\triangleright$ Mask binarization threshold

6: $w_{-1}=0.1,w_{0}=0.8,w_{-1}=0.1$ $\triangleright$ Inter-frame blending
weights

7:$\mathcal{C}^{src},\mathcal{C}^{trg}=\mathrm{CLIP_{t}}(\mathcal{P}^{src}),\mathrm{CLIP_{t}}(\mathcal{P}^{trg})$

8:$\mathcal{J}_{1:N}^{src},\mathcal{J}^{trg}=\mathrm{CLIP_{v}}(I_{1:N}^{src}),\mathrm{CLIP_{v}}(I^{trg})$

9:$\mathcal{Z}^{src}_{T}:=[z^{src}_{T,1},\cdots,z^{src}_{T,N}]=\mathrm{DDIM}^{-1}(\mathcal{E}(\mathcal{V}^{src}))$

10:${z}^{trg}_{T,1:N}={z}^{src}_{T,1:N}$

11:for $t\in[0.8\times T,T]$ do $\triangleright$ Compute the InvEdit mask

12: $t_{emb}=\mathrm{Emb}(t)$ $\triangleright$ sinusoidal timestep embedding

13: $\textit{
}\varepsilon^{src}_{t,1:N}=\varepsilon_{\theta}\left({z}^{src}_{t,1:N},t_{emb},\mathcal{C}^{src},\mathcal{J}^{src}_{1:N},\mathcal{M}_{\phi}\right)$

14:
$\varepsilon^{trg}_{t,1:N}=\varepsilon_{\theta}\left({z}^{trg}_{t,1:N},t_{emb},\mathcal{C}^{trg},\mathcal{J}^{trg},\mathcal{M}_{\phi}\right)$

15:
$\Delta\varepsilon_{t,1:N}=\mathrm{abs}(\varepsilon^{src}_{t,1:N}-\varepsilon^{trg}_{t,1:N})$

16: $z^{src}_{t-1,1:N}=\mathrm{DDIM}(\varepsilon^{src}_{t,1:N})$

17: $z^{trg}_{t-1,1:N}=\mathrm{DDIM}(\varepsilon^{trg}_{t,1:N})$

18:end for

19:$M_{1:N}=\mathrm{binarize}_{\alpha}(\mathop{\mathrm{mean}}_{t\in[0.8\times
T,T]}(\Delta\varepsilon_{t,1:N}))$

20:$\mathcal{M}^{inv}=M_{1:N}$

21:for $t=T,T-1,\cdots,2$ do $\triangleright$ Infer using InvEdit mask

22:

23: $[f^{t}_{1},\cdots,f^{t}_{N}]\leftarrow\text{get }\texttt{Up-
block-2}\text{ features }$

24: $\mathcal{N}^{t}_{i\pm}[p]=\mathrm{argmax}_{q}{d}(f^{t}_{i}[p],f^{t}_{i\pm
1}[q]),1\leq i\leq N$

25: $\mathcal{\hat{N}}^{t}_{i\pm}=\text{Upsample}(\mathcal{N}^{t}_{i\pm})$
$\triangleright$ upsample to match the dim of $\mathcal{Z}$ space

26:
$o_{t,1:N}=\varepsilon_{\theta}({z}^{src}_{t,1:N},t_{emb},\mathcal{C}^{trg},\mathcal{J}^{trg},\mathcal{M}_{\phi})$
$\triangleright$ UNet forward pass as in Sec.6

27:
$o^{\prime}_{t,1:N}=\varepsilon_{\theta}({z}^{src}_{t,1:N},t_{emb},\mathcal{C}^{trg},\\{\mathcal{J}^{src},\mathcal{J}^{trg}\\},\mathcal{M}^{inv})$
$\triangleright$ UNet forward pass as in Sec.6

28:
$z_{t-1}=\frac{1}{1+\mathcal{M}^{inv}}\big{(}\mathcal{M}^{inv}\odot\mathrm{DDIM}(o_{t,1:N})+\mathrm{DDIM}(o^{\prime}_{t,1:N})\big{)}$
$\triangleright$ latent fusion as in Sec. 6

29: $\tilde{z}_{t-1,i}[p]=w_{-1}(M_{i}\odot
z_{t-1,(i-1)}[\mathcal{\hat{N}}^{t}_{i-}(p)])+w_{0}(M_{i}\odot
z_{t-1,i})+w_{1}(M_{i}\odot
z_{t-1,(i+1)}[\mathcal{\hat{N}}^{t}_{i+}(p)])+(1-M_{i})\odot z_{t-1,i},\text{
if }t\geq T-5$ $\triangleright$ inter-frame latent correction

30: Apply optional background preservation

31:end for

32:Output video frames = $\mathcal{D}(\tilde{z}_{1,1:N})$

##  7 Additional results

Selection of the UNet block for _latent correction_ field: We compute the
correspondence error (CE) map across all blocks of UNet as per Sec. 3.3 and
find that the correspondence errors of Up-block-2 are generally lower than
other blocks as shown in Fig. 11. Across all experiments, we assign the
feature with minimal Euclidean distance to the original feature as the
corresponding feature. The correspondences obtained by computing RAFT optical
flow on the source video serve as the ground truth since the object in the
source video and the expected target object have the same shape. For computing
the CE, we compare the correspondences obtained in the feature space with the
ground truth from RAFT.

Additional results of GenVideo. In Fig. 12, Fig. 13 and Fig. 14, we present
some additional results.

  * •

Video object editing: In Fig. 13, we find that GenVideo is able to accurately
identify the region of interest to be modified. In Fig. 13 A, the InvEdit mask
accurately identified the region of edit and modified the region from the
source _rabbit_ to the target _tiger_ while keeping the _watermelon_ intact.
Similarly, in Fig. 13 C, the _rabbit_ was retained correctly and the region
corresponding to the _watermelon_ was edited to _cake_ which has a different
shape than the _watermelon_. Thus, _InvEdit_ correctly handles the edits for
varying shapes and sizes of objects. Results in Fig. 13 B demonstrate the
editing of a _silver swan_ to a _small wooden boat_. In this result, the
InvEdit mask helps in identifying regions that correspond to both _swan_ and
expected _boat_ in order to edit the source video effectively even when they
are of very different shapes and size here.

  * •

Style editing: In Fig. 14, we present results of GenVideo for stylistic
variation of the foreground object (in Fig. 14 A) and stylistic variations of
the entire frames in the video (in Fig. 14 B and Fig. 14 C). When editing the
entire frames in the video, we skip performing the background preservation.

  * •

Zero-shot image editing: In Fig. 12, we show additional results on zero-shot
image editing capabilities of our approach.

## References

  * Avrahami et al. [2022] Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In _CVPR_ , 2022.
  * Bar-Tal et al. [2022] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel.  Text2live: Text-driven layered image and video editing.  In _ECCV_ , 2022.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _ICCV_ , 2021.
  * Ceylan et al. [2023] Duygu Ceylan, Chun-Hao P. Huang, and Niloy J. Mitra.  Pix2video: Video editing using image diffusion.  In _ICCV_ , 2023.
  * Chai et al. [2023] Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu.  Stablevideo: Text-driven consistency-aware diffusion video editing.  In _ICCV_ , 2023.
  * Couairon et al. [2023] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord.  Diffedit: Diffusion-based semantic image editing with mask guidance.  In _ICLR_ , 2023.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _NeurIPS_ , 2021.
  * Esser et al. [2023] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis.  Structure and content-guided video synthesis with diffusion models.  In _ICCV_ , 2023.
  * Geyer et al. [2023] Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel.  Tokenflow: Consistent diffusion features for consistent video editing, 2023.
  * Girdhar et al. [2023] Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra.  Emu video: Factorizing text-to-video generation by explicit image conditioning.  2023\.
  * Hertz et al. [2022] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control, 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In _NeurIPS_ , 2020.
  * Ho et al. [2022] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al.  Imagen video: High definition video generation with diffusion models, 2022.
  * Hong et al. [2022] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang.  Cogvideo: Large-scale pretraining for text-to-video generation via transformers, 2022.
  * Kang et al. [2023] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park.  Scaling up gans for text-to-image synthesis.  _CVPR_ , 2023.
  * Khachatryan et al. [2023] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi.  Text2video-zero: Text-to-image diffusion models are zero-shot video generators.  _arXiv_ , 2023.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick.  Segment anything.  _arXiv_ , 2023.
  * Kumari et al. [2023] Nupur Kumari, Bingliang Zhang, Sheng-Yu Wang, Eli Shechtman, Richard Zhang, and Jun-Yan Zhu.  Ablating concepts in text-to-image diffusion models.  In _ICCV_ , 2023.
  * Lee et al. [2023] Yao-Chih Lee, Ji-Ze Genevieve Jang Jang, Yi-Ting Chen, Elizabeth Qiu, and Jia-Bin Huang.  Shape-aware text-driven layered video editing demo.  _arXiv preprint arXiv:2301.13173_ , 2023.
  * Liu et al. [2023a] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  _arXiv_ , 2023a.
  * Liu et al. [2023b] Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia.  Video-p2p: Video editing with cross-attention control, 2023b.
  * Luo et al. [2023] Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jinren Zhou, and Tieniu Tan.  Videofusion: Decomposed diffusion models for high-quality video generation.  In _CVPR_ , 2023.
  * Meng et al. [2022] Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Image synthesis and editing with stochastic differential equations.  In _ICLR_ , 2022.
  * Molad et al. [2023] Eyal Molad, Eliahu Horwitz, Dani Valevski, Alex Rav Acha, Yossi Matias, Yael Pritch, Yaniv Leviathan, and Yedid Hoshen.  Dreamix: Video diffusion models are general video editors, 2023.
  * Nichol and Dhariwal [2021] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In _ICML_ , 2021.
  * Nichol et al. [2022] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  In _ICML_ , 2022.
  * Oord et al. [2017] Aaron Van Den Oord et al.  The frobnicatable foo filter, 2017.  Neural discrete representation learning. In NeurIPS.
  * Parmar et al. [2023] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu.  Zero-shot image-to-image translation.  In _SIGGRAPH_ , 2023.
  * Patashnik et al. [2021] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In _ICCV_ , 2021.
  * Podell et al. [2023] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  Sdxl: Improving latent diffusion models for high-resolution image synthesis.  _arXiv_ , 2023.
  * Qi et al. [2023] Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen.  Fatezero: Fusing attentions for zero-shot text-based video editing.  In _ICCV_ , 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents, 2022.
  * Revanur et al. [2023] A. Revanur, D. Basu, S. Agrawal, D. Agarwal, and D. Pai.  Coralstyleclip: Co-optimized region and layer selection for image editing.  In _CVPR_ , 2023.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _CVPR_ , 2022.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  In _NeurIPS_ , 2022.
  * Singer et al. [2022] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al.  Make-a-video: Text-to-video generation without text-video data, 2022.
  * Song et al. [2023a] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In _ICLR_ , 2023a.
  * Song et al. [2023b] Yizhi Song, Zhifei Zhang, Zhe Lin, Scott Cohen, Brian Price, Jianming Zhang, Soo Ye Kim, and Daniel Aliaga.  Objectstitch: Object compositing with diffusion model.  In _CVPR_ , 2023b.
  * Teed and Deng [2020] Zachary Teed and Jia Deng.  Raft: Recurrent all-pairs field transforms for optical flow.  In _ECCV_ , 2020.
  * Tumanyan et al. [2023] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel.  Plug-and-play diffusion features for text-driven image-to-image translation.  In _CVPR_ , 2023.
  * Wang et al. [2018] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In _CVPR_ , 2018.
  * Wu et al. [2023] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  In _ICCV_ , 2023.
  * Yu et al. [2022] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al.  Scaling autoregressive models for content-rich text-to-image generation.  _arXiv_ , 2022.
  * Zhang et al. [2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _ICCV_ , 2023.
  * Zhao et al. [2023] Yuyang Zhao, Enze Xie, Lanqing Hong, Zhenguo Li, and Gim Hee Lee.  Make-a-protagonist: Generic video editing with an ensemble of experts, 2023.
  * Zhou et al. [2022] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng.  Magicvideo: Efficient video generation with latent diffusion models.  _arXiv_ , 2022.