#  Fine-grained Controllable Video Generation via Object Appearance and
Context

Hsin-Ping Huang1,2 Yu-Chuan Su1 Deqing Sun1 Lu Jiang1
Xuhui Jia1 Yukun Zhu1 Ming-Hsuan Yang1,2
1Google Research 2UC Merced

###### Abstract

Text-to-video generation has shown promising results. However, by taking only
natural languages as input, users often face difficulties in providing
detailed information to precisely control the model’s output. In this work, we
propose fine-grained controllable video generation (FACTOR) to achieve
detailed control. Specifically, FACTOR aims to control objects’ appearances
and context, including their location and category, in conjunction with the
text prompt. To achieve detailed control, we propose a unified framework to
jointly inject control signals into the existing text-to-video model. Our
model consists of a joint encoder and adaptive cross-attention layers. By
optimizing the encoder and the inserted layer, we adapt the model to generate
videos that are aligned with both text prompts and fine-grained control.
Compared to existing methods relying on dense control signals such as edge
maps, we provide a more intuitive and user-friendly interface to allow object-
level fine-grained control. Our method achieves controllability of object
appearances without finetuning, which reduces the per-subject optimization
efforts for the users. Extensive experiments on standard benchmark datasets
and user-provided inputs validate that our model obtains a 70% improvement in
controllability metrics over competitive baselines. Project page:
https://hhsinping.github.io/factor

Figure 1: Text-to-video generation [41] has limited controllable ability
through user-provided prompts. Even when the users augment the text prompt
with additional description, the model has difficulty controlling the precise
movement and appearance of the object. Text-to-Video+ControlNet [60] achieves
promising visual quality while requiring dense control signals extracted from
a reference video. FACTOR (Ours) improves the controllability through user-
friendly inputs to control the: 1) precise movement of subjects through hand-
drawing trajectory and 2) visual appearance by providing reference examples.

##  1 Introduction

With numerous text-to-video models [22, 48, 47, 20, 40, 5, 26, 43, 15, 41, 63]
demonstrating impressive visual quality in their generated results, users can
now translate their creative ideas into video content with a simple text
prompt and a tap. However, it often requires careful design and iterative
revision of the text prompt to generate a video that fulfills users’ needs. As
shown in Fig. 1 (top), to generate videos with specific object movements and
appearances, the users can augment the text prompt with additional
descriptions such as “from right to left” and “yellow and black trim.”
However, current text-to-video models might fail to generate videos that
reflect the additional, fine-grained descriptions due to the high flexibility
of text prompts and the limited capability of models, specifically in
controllability.

Recently, a few approaches have been developed to enhance the controllability
of text-to-video models by injecting additional inputs. Among this direction,
structure, and appearance are the two prevalent and intuitive forms of user
control. Several models [49, 26, 60, 7] are proposed to generate videos
conditioned on structural controls such as optical flow, depth, and edge maps.
Similarly, video editing [44, 31, 35, 16] takes a video as input and uses
natural languages to manipulate the content and style. Although these models
have demonstrated promising visual results, they rely on control inputs
typically dense in time and space, _i.e_., provided for each pixel in each
frame. Unlike text-to-image generation, providing such inputs for the whole
video by manual hand drawing is impractical. Thus, the structural inputs are
usually obtained from a reference video (Fig. 1 middle). In addition, a few
attempts have been made to achieve appearance control, _i.e_., subject
customization for video generation. However, these methods either rely on
taking the dense control as input [26, 62, 49] or require finetuning per
subject/domain using a set of images [38, 17, 5]. Moreover, existing
controllable video generation models are limited to taking only a single type
of control, either structural [26] or appearance [5]. To achieve
controllability with multiple control signals, the model has to be finetuned
in a sequential manner [49].

In this work, we tackle fine-grained controllable video generation via object
appearance and context, a user-friendly controllable video generation
framework. In this framework, the users first specify the objects in the video
and then provide fine-grained control to generate each object, including their
appearance and context. The context to generate each entity includes 1) the
description of the entity, 2) the user-drawing trajectory, and 3) the user-
provided reference image to achieve both structural and appearance control.
This setting has the following advantages: First, our model accepts sparse
hand-drawing object trajectories as intuitive inputs that require minimal
effort for users to provide, in contrast to dense structural guidance, such as
edge maps. Second, unlike prior works that utilize per-subject finetuning to
achieve appearance control, our method is finetuning-free, and only an
inference pass is needed for subject customization, reducing the users’
efforts.

We propose a joint encoding and adaptive cross-attention framework to
facilitate fine-grained, controllable video generation. We develop the method
on top of an off-the-shelf text-to-video model [41]. Our controllable module
is a unified framework for both types of control signals, and the module is
learned by a single adapting process in contrast to prior works that finetune
the models for multiple control sequentially [49]. It is generic and can be
extended to other types of control easily. Specifically, all control signals
are encoded into a single control sequence through a joint encoder, and the
adaptive cross-attention layers are inserted into transformer blocks of the
model to take the fine-grained control inputs. During training, the newly
inserted layers are updated while other layers of the pre-trained model are
fixed. This design preserves the text-to-video model’s capability to generate
high-quality videos while introducing adaptive layers that augment the model’s
ability to produce videos that satisfy the object-level fined-grained control.

We conduct quantitative studies on the standard benchmark dataset MSR-VTT [53]
and perform a user study to validate the effectiveness of the proposed
approach. Our method showcases a substantial 67% and 73% enhancement in
absolute quality and trajectory control compared to the base text-to-video
model, as observed through objective metrics. The qualitative results show
that our method achieves fine-grained, controllable video generation for
object appearance and context. We summarize our contributions as follows:

  * •

We target the new form of fine-grained controllable video generation task that
aims to synthesize videos via appearance and context from easy-to-give user
inputs.

  * •

We propose a generic and unified framework for controllable video generation.
It is achieved through adaptive training to augment text-to-video models for
fine-grained control without test time optimization.

  * •

We validate that our method offers better controllability compared to prior
works and show the additional benefit of our model in creating complex
interactions, which is challenging for existing text-to-video models.

Figure 2: Overview. a) Joint encoder and adaptive cross-attention: a joint
encoder is learned to encode the prompt and control to capture the interaction
between them. The adaptive layers are inserted into the transformer blocks of
the text-to-video model to take new control signals. Only the inserted layers
are optimized to adapt the model to generate videos satisfying the fine-
grained control. b) Condition encoding: given T time steps, the embedding of
control at time $t$, $c_{t}$, is formed by the control for N entities,
$e^{n}_{t}$, where padding tokens replace the embedding of the non-existing
entity. c) Entity control: the control for entity $n$ at time $t$,
$e^{n}_{t}$, is formed by the embedding of the context including the
description, location, and reference appearance of the objects.

##  2 Related Work

Text-to-video generation. Text-to-video models have successfully generated
open-domain videos conditioned on text prompts by training on large-scale
caption-video pairs. Token-based models [47, 48, 22] are first proposed for
text-to-video generation, which utilizes an auto-regressive model to predict
videos in the latent token space. Several diffusion-based models [40, 21, 20]
extend the 2D diffusion model to generate videos using 3D U-Net and a series
of super-resolution modules. Recent advances [26, 43, 63, 5, 15] adopt a text-
to-image diffusion model [37] to generate videos by inflating the model and
incorporating the temporal layers. While these models produce promising visual
results, their controllability over generated videos is limited since text
prompts cannot accurately convey precise control signals such as the location
and appearance of objects. In this work, we focus on improving the
controllability of text-to-video models through user-friendly inputs. We adopt
a generative transformer model [41] as our base text-to-video model due to its
fast inference speed and flexibility to take multiple control signals but the
approach can be adapted to common diffusion-based models easily.

Controllable text-to-image generation. Various models have been proposed to
improve the controllable ability over text prompts for text-to-image models,
particularly the structure and appearance. ControlNet [59], T2I-Adapter [33]
and Uni-ControlNet [61] add various spatial control signals, including edges,
depth, segmentation, human pose, to a pre-trained diffusion model. GLIGEN [29]
adds spatial and appearance control to the text-to-image model. Some works
focus on structure-guided generation from semantic layouts [1, 54, 56, 58, 18]
and training-free approaches [3, 57, 50, 27, 11].

To control the appearance of subjects for text-to-image models, Textual
Inversion [13], DreamBooth [38] and CustomDiffusion [28] either directly
optimize a word vector to learn the target appearance, or finetune the text-
to-image model on reference images of the subjects. Several encoder-based
methods [14, 46] are proposed to achieve customization without finetuning by
projecting reference images of the subject into the word embeddings. In this
work, we study finetuning-free subject customization for videos with
trajectory control.

Controllable text-to-video generation. To extend controllable text-to-image
models to generate videos, several works [45, 49, 26, 32, 60, 51, 7, 30, 9,
10, 12] adopt the ControlNet [59] backbone to condition the generation on a
sequence of structural inputs including depth, pose, and edge maps. By
training a temporal attention layer built upon the ControlNet model, these
methods learn to produce temporally consistent videos. However, they require a
dense sequence of structural inputs as the condition, which usually need to be
extracted from reference videos. Similarly, video editing approaches [44, 55,
6, 39, 31, 35, 23, 42, 16, 52, 34, 25, 36] focus on changing the appearance
and style of foreground objects and background through text prompts. These
methods also rely on the dense structure provided in the input video, which is
infeasible to be drawn by the users. In contrast, our work focuses on
controllable text-to-video generation that takes sparse and user-friendly
control signals, which remains less explored in this field.

As for appearance control, a few approaches [49, 26, 19, 62] show video
customization results using finetuning approach while still relying on the
dense structural input to generate videos. Others [17, 5] perform finetuning-
based customization without spatial control. In contrast, we developed a
finetuning-free method to achieve appearance control of multiple subjects, and
the model conducts spatial control using sparse trajectory inputs
simultaneously.

##  3 Method

As our method is developed upon a text-to-video generative transformer, we
first briefly review the base model (Sec. 3.1). Then, we give an overview of
our fine-grained controllable video generation (Sec. 3.2). Next, we introduce
the proposed method, which consists of a joint encoder of learning the
interaction between different controls and an adaptive cross-attention module
to adapt the text-to-video model to achieve fine-grained control (Sec. 3.3).
Finally, we present the encoding process of our entity-level fine-grained
control and the data preparation pipeline (Sec. 3.4).

###  3.1 Preliminaries: Text-to-Video Generation

This section presents our base text-to-video generation model [41]. It
consists of an encoder-decoder model that encodes the video into discrete
video tokens and a bidirectional transformer model that predicts the video
tokens conditioned on the embedding of text prompts. Specifically, the video
is encoded and flattened into a long sequence as the input to the transformer.
At training time, the tokens are replaced with a special token [MASK], and the
transformer model is optimized to predict the tokens at [MASK] location
conditioned on the text embedding. We minimize the negative log-likelihood of
conditional video token prediction for the masked tokens $v_{t},t\in M$, $M$
denotes the subset of video tokens that are masked, $v$ denotes the video
tokens sequence, $v_{M}$ denotes the masked version of $v$. The model is
trained by:

$\mathcal{L}={-}\mathop{\mathbb{E}}\limits_{v\in\mathcal{D}}\sum_{t\in M}\log
p(v_{t}|v_{M},p)$ (1)

At inference time, all tokens are replaced with [MASK], and the transformer
model iteratively predicts the tokens conditioned on $p$.

The video token prediction model contains a series of transformer blocks. The
video embeddings are first input to the self-attention. Then, a cross-
attention layer takes video and text embeddings as input to condition the
video token prediction on the text embeddings. We use a token-based generative
transformer model for its fast inference and inherent flexibility in modeling
various control signals directly within the transformer architecture.

###  3.2 Overview

Given a text prompt $p$ and a fine-grained control $c$ as inputs, our model
aims to generate a video to satisfy both input conditions. Specifically, the
users provide fine-grained control $c$ by 1) describing the desired entities
in the video, 2) drawing their trajectories, and 3) providing a reference
appearance image for each entity. This pipeline helps the users to create
videos intuitively. Specifically, the input control signal is given at $T$
timesteps. Assume there are $N$ entities in the video, we define our control
as $c=\\{c_{t}\\}_{t=1}^{T}$, $c_{t}\in\mathbb{R}^{2}$. At a single timestep
$t$, the control is formed by a sequence of entity control, _i.e_., the
embeddings of the $N$ entities $c_{t}=\\{e_{t}^{n}\\}_{n=1}^{N}$,
$e_{t}^{n}\in\mathbb{R}^{2}$. The embeddings encode the desired condition to
generate the $n^{th}$ entity at time $t$.

###  3.3 Joint Encoder and Adaptive Cross-Attention

We explain the main components of our method in this section, including a
joint encoder and an adaptive cross-attention module. Compared to existing
works (_e.g_., [59]) that use a separate encoding process for each input
condition, we utilize a joint encoder to encode both the text prompt $p$ and
the fine-grained control $c$ simultaneously within the same transformer. This
design facilitates the interaction between the text prompt and fine-grained
control, captured into the contextualized embeddings via the joint encoder.
Specifically, we use simple embedding layers to transform the discrete tokens
of prompt $p$ and control $c$ into embedding sequences. The two sequences are
then concatenated and input to self-attention layers to obtain the
contextualized embeddings. Finally, the embeddings are split back into prompt
and control embeddings.

The next task is to incorporate the fine-grained control into the base text-
to-video transformer. Given a sequence of control embeddings, we extend the
existing text-to-video model by inserting an adaptive cross-attention layer in
each transformer block to take additional control. See Fig. 2. During
training, we freeze the weight of the self-attention and the cross-attention
layers of the pre-trained model. We train the joint encoder and the newly
inserted adaptive cross-attention layer from scratch to adapt the text-to-
video model to generate videos aligned with text and the new fine-grained
condition. By fixing the pre-trained weights, the capability to generate high-
quality videos is preserved, and the model learns to generate videos
satisfying the fine-grained control by updating only a small portion
($\approx$ 23%) of the parameters.

Though we use a transformer-based model, our adaptive cross-attention layers
may be extended to other text-to-video models, _e.g_., diffusion-based models,
as their architecture usually contains a series of similar blocks of
convolution and attention layers [37, 5]. Likewise, this module can be easily
extended to take other control signals. This is achieved by first transforming
the control into tokens and inputting them to our joint encoder and adaptive
cross-attention layers. Instead of using multiple condition-specific methods,
our adaptive process is a unified approach to condition the model on multiple
control signals in a single training pipeline.

Figure 3: Trajectory prompts. To highlight that the prompt is not enough to
achieve the fine control provided by our method, we augment the text prompt to
describe the trajectories specified by the bounding boxes as inputs to the
text-to-video model. Phenaki [41] fails to generate the correct object
movement, while FACTOR successfully controls the movement of generated
entities with our hand-drawing trajectory input. The blue and green boxes show
the location of the object in the first and last frames, respectively.

###  3.4 Condition Encoding and Entity Control

This subsection discusses our entity-level fine-grained control in detail. The
entity embeddings $e_{t}^{n}$ for generating the $n^{th}$ entity at timestep
$t$ are constructed by encoding the context of each entity. First, the
description of the entity $d$ is given by text, _e.g_., a cat, and encoded by
an embedding layer to transform the discrete tokens into embeddings
$\psi_{entity}(d)$. Second, the location of the entity $l$ is given by the
top-left and bottom-right bounding box coordinates of the entity and encoded
by an embedding layer as $\psi_{coord}(l)$. Finally, the reference appearance
$r$ of the entity is given by a single example image and encoded by a CLIP
image encoder with an MLP layer to reduce the feature size as
$\psi_{image}(r)$. The embeddings $e^{n}_{t}$ is the concatenation of
description, location, and appearance embeddings of the entity:

$\displaystyle
e^{n}_{t}=Concat(\psi_{entity}(d^{n}_{t}),\psi_{coord}(l^{n}_{t}),\psi_{image}(r^{n}_{t})).$
(2)

We replace the embeddings of $e^{n}_{t}$ with padding embeddings when the
$n^{th}$ entity is missing at timestep $t$. Though all the conditions can be
thoroughly given at $T$ timesteps, we assume prompts, description, and
appearance are fixed in the whole video, and only the location changes over
time to resemble the user-friendly setup. In our user interface, the location
is given by simply drawing a bounding box in the first frame and dragging it
to move to the location in the last frame.

In practice, very few video datasets contain the annotations of objects’
trajectories and visual examples of the objects’ appearance. To train the
model, we utilize an off-the-shelf object detector [24] and tracking algorithm
[4] to extract $N$ entities in the video clip and their locations at $T$
timesteps. For each entity in the video, we need to collect a reference visual
example as our appearance control $r$. However, unlike image [38, 28], it is
difficult to collect multiple videos of the same subject as training data. We
select the reference visual example of the subject from the same video. To
obtain reference images with more diverse appearances, we sample them from a
longer video clip ($\approx$40 frames) outside our training timespan
($\approx$11 frames). We obtain the reference image of the subject by cropping
the region using detected bounding boxes. Finally, we use the collected
triplet of text prompt $p$, control $c$, and video $v$ to train our model.

Figure 4: Trajectory control. Given the trajectories of the two main entities
in the videos as input, FACTOR brings an additional benefit to generate
complex videos containing subject-object (top, middle) and subject-subject
(bottom) interactions between two entities. The trajectory control inputs are
omitted for simplicity.  Figure 5: Appearance control. FACTOR generates videos
with the desired object appearances. The videos contain interaction for the
customized subject (left), the composition of two customized subjects (top-
right), and reasonable motion of live subjects (bottom-right).

##  4 Experimental Results

Dataset. We train our model on two datasets: WebVid [2] and WebLI [8]. WebVid
[2] is a large-scale dataset of short videos and captions collected from stock
footage sites. We use the 10M video-and-caption pairs in our training. WebLI
[8] is a high-volume image-and-language dataset collected from the public web.
We randomly sample a subset of WebLI with $\sim$500M image-and-text pairs in
our training. We train our base text-to-video model from scratch using WebVid
and WebLI data. Then, we train our controllable video generation model using
the same data with our augmented annotations $c$ (Sec. 3.4). We evaluate our
model on MSR-VTT [53] dataset using zero-shot performance on the test set,
which consists of 2,990 examples with 20 prompts per example. Following [5],
we generate one video per sample by randomly selecting one prompt.

Evaluation metrics. We measure the video quality and the alignment between the
generated video and different input conditions including text, trajectory, and
reference images, using the following metrics: 1) Fréchet Video Distance (FVD)
assesses the absolute video quality. 2) CLIP text similarity (CLIP-T) assesses
the alignment between input prompts and generated video frames using CLIP
embeddings. 3) Average Precision (AP) assesses the alignment between input
object trajectories and the generated videos. We apply a pre-trained object
detector [24] to compare the detected bounding boxes between the generated
frames and ground truth frames and report the AP score. 4) CLIP video
similarity (CLIP-V) assesses the similarity between the reference images and
object appearance in the generated video frames. We measure the CLIP embedding
distance between the generated frames and ground truth frames, following [38].

Implementation details. We implement the base text-to-video generative
transformer model following Phenaki [41]. Our implementation of the base model
has 1.3B parameters, and our FACTOR model contains 1.6B parameters. The base
model is trained for 1M steps, and our model is trained for 500K steps at a
batch size of 256 and 128, respectively. The videos are generated at the
resolution of 192$\times$320\. During training, we use a mixture of 20% images
and 80% videos in each batch.

Compared methods. Because our goal is to add fine-grained control to an
existing text-to-video model, we evaluate whether our method can improve the
generative quality of the base model for the fine-grained control inputs. We
validate that our approach enhances the base model’s controllability without
compromising the quality of the output video. Direct comparison with other
works (_e.g_., [49, 7]), which inject dense structural control into the video
generation model, is difficult due to their incompatibility with the sparse
trajectory inputs our work utilizes.

To further show that our method provides reasonable controllability compared
to existing text-to-video models, we present comparisons with state-of-the-art
models when quantitative/qualitative results are available in their papers,
given most of them are not publicly available. We show the results of Imagen-
Video [20], MagicVideo [63], VideoLDM [5], Make-A-Video [40], and ModelScope
[43]. We evaluate two variants of our methods: 1) FACTOR-traj denoted as our
model with text and trajectory control. 2) FACTOR denoted as our full model
with text, trajectory, and appearance control. Please see the appendix for
details.

Figure 6: Comparison to state-of-the-art. FACTOR generates videos with better
interaction between subject and objects. It shows more semantic meaning,
_e.g_., the astronaut stretches out the hand, and the duck gets close to the
astronaut, and better movement due to our fine-grained control. As the state-
of-the-art models are not publicly available, we copy the results from their
papers. The blue and green boxes show the location of the object in the first
and last frames, respectively.  Figure 7: Comparison of appearance control.
VideoLDM [5] uses a backbone finetuned on multiple images while FACTOR is
finetuning-free and conditions on a single reference image. FACTOR generates
subjects that better align with the reference appearance, showing larger
movements and better compatibility with the new environments.

###  4.1 Qualitative Results of Trajectory Control

First, we assess the model’s ability to control the trajectories of generated
entities. We design several prompts, including the description of trajectories
such as the entity moving “from the left to the right” and the camera moving
“towards” the entity. In Fig. 3, we show that given the trajectory prompts as
input, Phenaki [41] fails to generate the correct movement direction for the
elephant, and it fails to generate the lion with proper size and movement in
the video. In contrast, FACTOR-traj, which takes the hand-drawing trajectories
as control inputs, generates the correct moving direction of the entity and
the camera.

In Fig. 4, we further present the results of trajectory control by drawing the
trajectory of two main entities in the videos, such as (elephant, teddy bear)
in the top-left video and (monkey, motorcyclist) in the bottom-left video. By
taking the trajectories as input, FACTOR-traj generates videos with two
entities having the movement following the trajectories given by the users.
Moreover, FACTOR-traj brings an additional benefit of generating complex
videos, including 1) interaction between subject-object such as “pick up” and
2) interaction between subject-subject such as “high five with”, even though
our method is not specifically training on the examples of
relation/interaction.

###  4.2 Qualitative Results of Appearance Control

Next, we present the results of appearance control, _i.e_., subject
customization. We select the reference images of the subjects from the
datasets collected by DreamBooth [38], CustonDiffusion [28] and VideoLDM [5].
For each subject, we use a single reference image as the appearance condition.
Fig. 5 shows that FACTOR successfully generates the subject aligned with the
reference appearance image. FACTOR handles interaction (milk and vase), the
composition of two customized subjects (teapot and duck toy), and the motion
of live subjects (cat walking, walking in the shoe).

We observe several limitations of our approaches. First, we find that the
generated live subjects have limited motion in FACTOR compared to FACTOR-traj
since our current framework conditions the subject appearance in the whole
video uses a single share image. This could be further improved by applying
data augmentation on the reference image in pixel or feature space [26] to
generate more motion dynamics. On the other hand, FACTOR has sub-optimal
performance when the text prompt is not aligned with the trajectory/appearance
control. In this case, the model might perform inferior to the models that
take only a single condition as input.

###  4.3 Comparison to State-of-the-Art

We further compare our video generation results with recent state-of-the-art
text-to-video models. As their models are not publicly available, we present
the results in their papers. As shown in Fig. 6, by taking the additional
trajectory control as input, FACTOR-traj synthesizes videos showing better
interaction between subject and object, _e.g_., the astronaut stretches out
the hand, the duck gets close to the astronaut, as well as better movement of
the entity such as the movement of the boat. In contrast, VideoLDM [5]
generates videos with less semantic meaning – there is no action of feeding
the duck, and Make-A-Video [40] produces videos with less movement. We note
that our method’s ability is limited by the resolution/capability of the base
model, and the method can be extended to other text-to-video models with
better visual quality (_e.g_. [20, 5]) if available.

To evaluate the ability of appearance control, we further compare FACTOR to
VideoLDM [5], where they insert trained temporal layers into a DreamBooth
backbone of Stable Diffusion model to generate video of customized subjects.
Their appearance control method requires multiple reference images and per-
subject finetuning. On the other hand, FACTOR 1) does not require fine-tuning,
and only an inference pass is needed, 2) requires a single reference image as
input, and 3) achieves subject customization for multiple entities. Fig. 7
shows that our videos present better reconstruction of the customized subject.
Using the additional trajectories as input, we show larger movements and
better text alignment than VideoLDM, as their backbone is finetuned on static
images.

Table 1: Quantitative results on MSR-VTT. Our model achieves a lower FVD compared to baselines. The high AP and CLIP-V scores show our model successfully generates videos aligned with the trajectory and appearance control inputs.  Methods |  FVD $(\downarrow)$ |  CLIP-T $(\uparrow)$ |  AP $(\uparrow)$ |  CLIP-V $(\uparrow)$
---|---|---|---|---
MagicVideo | 1290 | — | — | —
VideoLDM | — | 0.2929 | — | —
Make-A-Video | — | 0.3049 | — | —
ModelScope | 550 | 0.2930 | — | —
Phenaki | 384 | 0.2870 | 0.0990 | 0.6626
FACTOR-traj (Ours) | 317 | 0.2787 | 0.2902 | 0.6825
FACTOR (Ours) | 124 | 0.2723 | 0.3407 | 0.7575

###  4.4 Quantitative Results

Results on MSR-VTT. As shown in Tab. 1, FACTOR achieves better FVD scores than
other text-to-video models. In addition, FACTOR obtains a CLIP-T score
comparable to state-of-the-art approaches. Our model has to generate videos
that align with text and the fine-grained control signal including trajectory
and reference appearance. Thus, it obtains a slightly worse CLIP-T score than
state-of-the-art text-to-video models designed to optimize this metric.

We further compare our implementation of Phenaki, FACTOR-traj, and FACTOR as
these models are built upon the same video backbone and training data while
taking different levels of control as inputs, _i.e_., text only,
text+trajectory, text+trajectory+appearance. We find that with additional
trajectory and reference images as input, the video quality score FVD improves
significantly. We further compare the AP score. Although Phenaki is not
trained to generate videos with desired object trajectories, we interpret its
AP score as a chance performance when the text prompt is sufficient to
generate videos that align with the user input trajectory. FACTOR-traj’s
higher AP score compared to Phenaki proves that the model successfully
generates videos that align with the given trajectory control. On the other
hand, FACTOR achieves better CLIP-V scores upon FACTOR-traj and Phenaki,
showing that the model generates videos that better resemble the ground truth
videos, indicating better alignment with the appearance control.

Table 2: User study. We present the percentage of raters that prefer each method. FACTOR-traj only uses text and trajectory as control. Our model has a similar visual quality as the base model. The users prefer our results on text, trajectory, and appearance alignments to the ground truth videos.  | Quality | Text | Trajectory | Appearance
---|---|---|---|---
Phenaki _vs_. FACTOR-traj | 38/62 | 27/73 | 18/82 | 20/80
Phenaki _vs_. FACTOR | 37/63 | 26/74 | 8/92 | 2/98
FACTOR-traj _vs_. FACTOR | 40/60 | 47/53 | 37/63 | 22/78

User study. To further augment the quantitative evaluation, we conduct a user
preference study. We present two videos generated by different methods using
the same prompt and the ground truth video. We ask the raters to select their
preferred results based on the following criteria: 1) Quality: which video has
the better visual quality, 2) Text alignment: which video is better aligned
with the text prompt, 3) Trajectory alignment: which video has the object
trajectory that is better aligned with the ground truth video, and 4)
Appearance alignment: which video better reproduces the object and scene
appearance of the ground truth video. We conduct the study with 30 pairs of
videos and 5 participants.

Tab. 2 show that FACTOR achieves better visual quality ($>$60% preference) and
text alignment ($>$70% preference) compared to baselines. By taking the
additional trajectory and reference images as input, FACTOR’s results have
better trajectory/appearance alignment with the ground truth video, showing
the effectiveness of our control module. Comparing the two variants of our
model, FACTOR taking the reference images as a condition achieves better
appearance alignment, presenting its capability of subject customization.

##  5 Conclusions

We present FACTOR, an approach to fine-grained controllable video generation.
In addition to the text prompt, users can easily control the video generation
process by naming the entities, drawing their trajectories, and providing
their appearance through visual examples. Our key idea is to adapt the text-
to-video model to generate videos conditioned on fine-grained control by
training an adaptive attention layer, and we model the interaction between
inputs through a joint encoding module. Our method generates videos with
controllability on object appearance and trajectory. It brings additional
benefits of generating complex videos including interaction.

## Appendix

We present the implementation and experiment details in the appendix,
including the details of datasets, training, model architectures, and
evaluation metrics.

Datasets. We train the model on the WebVid-10M dataset. The dataset contains
videos at 4 fps with a resolution of $256\times 454$ and an average length of
40 frames. During training, the videos are simply resized to our output
resolution of $96\times 160$ without data augmentation. We randomly sample
11-frame sub-sequences as training samples. To obtain our fine-grained
annotations, we apply an off-the-shelf object detector [24] to obtain the list
of objects in the 40-frame videos. We run a tracking algorithm [4] to extract
the trajectory of each object. We sort the trajectories by the number of
pixels they cover and remove the smaller trajectories that heavily overlap
with the larger ones. Finally, we select 12 object trajectories for each video
in our annotations. At training time, the order of the objects is randomly
shuffled.

To obtain the reference appearance images, we use the detected trajectories to
crop the video frames. We randomly select one cropped region from the 40-frame
videos for each object as the reference appearance image. In addition to
video, we use image data from the WebLI dataset in our training, which is used
20% of the time in each batch. Similarly, we run an object detector [24] to
obtain the bounding box annotations. The image frames are repeated to the
video length as a training sample.

Model architectures and training details. Our base text-to-video model
contains a C-ViViT encoder-decoder and a bidirectional transformer. The
C-ViViT model is trained to compress the video with dimensions $11\times
96\times 160\times 3$ into discrete tokens with dimensions of $6\times
12\times 20$. The model has a hidden size of 512, an embedding dimension of
32, an MLP size of 2048, 8 heads, and a codebook size of 8192. It contains 4
layers of spatial transformers and 4 layers of temporal transformers. The
model has 100M parameters. The model is trained using the AdamW optimizer with
a learning rate of $1\times 10^{-4}$, a batch size of 128, and 1M iterations.

The bidirectional transformer is trained to produce videos at $11\times
96\times 160\times 3$. The transformer takes a sequence length of 1920. It
contains an embedding dimension of 1728, an MLP size of 4096, 36 heads, and 24
layers. The model has 1.3B parameters. The model is trained using the AdamW
optimizer with a learning rate of $4.5\times 10^{-5}$, a batch size of 512,
and 1M iterations. The videos are generated at a unit of 11 frames and can be
extended to an arbitrary length by applying a sliding window approach.
Specifically, we first generate an 11-frame clip and use the last 5 frames as
a condition for the model to generate another 6 frames. This process can be
applied iteratively. We apply a super-resolution model to increase the video
resolution to $192\times 320\times 3$. In our method, the C-ViViT model and
the super-resolution model are kept as the original models for text-to-video
generation, and only the bidirectional transformer model is trained to take
the fine-grained control inputs.

As the video is compressed into a size of $6\times 12\times 20$, the control
signals are given at 6 timesteps to generate 11 frames. The control signal at
each timestep contains prompt embeddings and control embeddings. The length of
the prompt embedding is 64, and the length of the control embedding is 220. In
FACTOR model, the control embedding is formed by the entity control of four
objects, including the embedding of their description, location, and
appearance with lengths of 1, 4, and 50, respectively.

To obtain the appearance embedding, the reference images are resized to
$224\times 224$ and input to a ViT-B/32 model to extract the appearance
features. We use the grid features, which have a length of 50 and a size of
768. The appearance embeddings are then input to an MLP layer to reduce its
size to 512. We use a learned embedding layer to obtain the entity description
and location sequences with a feature size of 512. The three contexts are then
concatenated to form an embedding sequence of length 220 and input to a joint
transformer encoder, which contains a self-attention layer to obtain the
contextualized embeddings. The embeddings are then split into prompt
embeddings of length 64 and control embeddings of size 220 and input to the
cross-attention layer and adaptive cross-attention layers separately. In
total, the FACTOR model contains 1.6B parameters. At training time, the
embedding layer, the joint encoder, and the adaptive cross-attention layers
are learned while other layers are fixed. The model is trained using the AdamW
optimizer with a learning rate of $4.5\times 10^{-5}$, a batch size of 128,
and 500k iterations. To apply classifier-free guidance, at training time, all
the input conditions are dropped simultaneously 10% of the time. At inference
time, we use a classifier guidance scale of 12, an inference temperature of 4,
and 48 sampling steps.

Evaluation metrics. We present the details of evaluation metrics. 1) Fréchet
Video Distance (FVD) assesses the absolute video quality by measuring whether
the distribution of generated videos is close to that of real videos in the
feature space. We use the I3D model trained on Kinetics-400 for video
features. 2) CLIP text similarity (CLIP-T) assesses the alignment between
input prompts and generated video frames using CLIP embeddings. We use the
ViT-B/32 model to compute the CLIP-T score. The score is reported as the
average of all frames. 3) Average Precision (AP) assesses the alignment
between input object trajectories and the generated videos. We apply a pre-
trained object detector [24] to compare the detected bounding boxes between
the generated frames and ground truth frames and report the AP score. The AP
score is computed at the $6^{th}$ frame. 4) CLIP video similarity (CLIP-V)
assesses the similarity between the reference images and object appearance in
the generated video frames. We measure the CLIP embedding distance between the
generated frames and ground truth frames, following [38]. Specifically, we use
the bounding box coordinates in the trajectory of an object to crop the ground
truth video frames. After obtaining a cropped image per frame containing that
object, we randomly select one as the reference appearance image. Since the
contents in the ground truth videos are used as references, the generated
video should mostly reconstruct the ground truth video. We measure the
similarity of the generated and ground truth video frames and report the
average. We use the ViT-B/32 model to compute the CLIP-V score.

## References

  * [1] Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin.  Spatext: Spatio-textual representation for controllable image generation.  In CVPR, 2023.
  * [2] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In ICCV, 2021.
  * [3] Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel.  Multidiffusion: Fusing diffusion paths for controlled image generation.  arXiv preprint arXiv:2302.08113, 2023.
  * [4] Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft.  Simple online and realtime tracking.  In ICIP, 2016.
  * [5] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis.  Align your latents: High-resolution video synthesis with latent diffusion models.  In CVPR, 2023.
  * [6] Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu.  Stablevideo: Text-driven consistency-aware diffusion video editing.  arXiv preprint arXiv:2308.09592, 2023.
  * [7] Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin.  Control-a-video: Controllable text-to-video generation with diffusion models.  arXiv preprint arXiv:2305.13840, 2023.
  * [8] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut.  Pali: A jointly-scaled multilingual language-image model.  In ICLR, 2023.
  * [9] Yutao Chen, Xingning Dong, Tian Gan, Chunluan Zhou, Ming Yang, and Qingpei Guo.  Eve: Efficient zero-shot text-based video editing with depth map guidance and temporal consistency constraints.  arXiv preprint arXiv:2308.10648, 2023.
  * [10] Ernie Chu, Tzuhsuan Huang, Shuo-Yen Lin, and Jun-Cheng Chen.  Medm: Mediating image diffusion models for video-to-video translation with temporal correspondence guidance.  arXiv preprint arXiv:2308.10079, 2023.
  * [11] Dave Epstein, Allan Jabri, Ben Poole, Alexei A. Efros, and Aleksander Holynski.  Diffusion self-guidance for controllable image generation.  In NeurIPS, 2023.
  * [12] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis.  Structure and content-guided video synthesis with diffusion models.  arXiv preprint arXiv:2302.03011, 2023.
  * [13] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  arXiv preprint arXiv:2208.01618, 2022.
  * [14] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or.  Encoder-based domain tuning for fast personalization of text-to-image models.  arXiv preprint arXiv:2302.12228, 2023.
  * [15] Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming-Yu Liu, and Yogesh Balaji.  Preserve your own correlation: A noise prior for video diffusion models.  In ICCV, 2023.
  * [16] Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel.  Tokenflow: Consistent diffusion features for consistent video editing.  arXiv preprint arxiv:2307.10373, 2023.
  * [17] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai.  Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.  arXiv preprint arXiv:2307.04725, 2023.
  * [18] Cusuh Ham, James Hays, Jingwan Lu, Krishna Kumar Singh, Zhifei Zhang, and Tobias Hinz.  Modulating pretrained diffusion models for multimodal image synthesis.  In SIGGRAPH, 2023.
  * [19] Yingqing He, Menghan Xia, Haoxin Chen, Xiaodong Cun, Yuan Gong, Jinbo Xing, Yong Zhang, Xintao Wang, Chao Weng, Ying Shan, et al.  Animate-a-story: Storytelling with retrieval-augmented video generation.  arXiv preprint arXiv:2307.06940, 2023.
  * [20] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans.  Imagen video: High definition video generation with diffusion models.  arXiv preprint arXiv:2210.02303, 2022.
  * [21] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.  Video diffusion models.  arXiv preprint arXiv:2204.03458, 2022.
  * [22] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang.  Cogvideo: Large-scale pretraining for text-to-video generation via transformers.  In ICLR, 2023.
  * [23] Zhihao Hu and Dong Xu.  Videocontrolnet: A motion-guided video-to-video translation framework by using diffusion model with controlnet.  arXiv preprint arXiv:2307.14073, 2023.
  * [24] Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, and Kevin Murphy.  Speed/accuracy trade-offs for modern convolutional object detectors.  In CVPR, 2017.
  * [25] Nisha Huang, Yuxin Zhang, and Weiming Dong.  Style-a-video: Agile diffusion for arbitrary text-based video style transfer.  arXiv preprint arXiv:2305.05464, 2023.
  * [26] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi.  Text2video-zero: Text-to-image diffusion models are zero-shot video generators.  In ICCV, 2023.
  * [27] Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Ha, and Jun-Yan Zhu.  Dense text-to-image generation with attention modulation.  In ICCV, 2023.
  * [28] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  In CVPR, 2023.
  * [29] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee.  Gligen: Open-set grounded text-to-image generation.  In CVPR, 2023.
  * [30] Jun Hao Liew, Hanshu Yan, Jianfeng Zhang, Zhongcong Xu, and Jiashi Feng.  Magicedit: High-fidelity and temporally coherent video editing.  arXiv preprint arXiv:2308.14749, 2023.
  * [31] Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia.  Video-p2p: Video editing with cross-attention control.  arXiv preprint arXiv:2303.04761, 2023.
  * [32] Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Ying Shan, Xiu Li, and Qifeng Chen.  Follow your pose: Pose-guided text-to-video generation using pose-free videos.  arXiv preprint arXiv:2304.01186, 2023.
  * [33] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  arXiv preprint arXiv:2302.08453, 2023.
  * [34] Hao Ouyang, Qiuyu Wang, Yuxi Xiao, Qingyan Bai, Juntao Zhang, Kecheng Zheng, Xiaowei Zhou, Qifeng Chen, and Yujun Shen.  Codef: Content deformation fields for temporally consistent video processing.  arXiv preprint arXiv:2308.07926, 2023.
  * [35] Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen.  Fatezero: Fusing attentions for zero-shot text-based video editing.  arXiv preprint arXiv:2303.09535, 2023.
  * [36] Bosheng Qin, Juncheng Li, Siliang Tang, Tat-Seng Chua, and Yueting Zhuang.  Instructvid2vid: Controllable video editing with natural language instructions.  arXiv preprint arXiv:2305.12328, 2023.
  * [37] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, 2022.
  * [38] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In CVPR, 2023.
  * [39] Chaehun Shin, Heeseung Kim, Che Hyun Lee, Sang gil Lee, and Sungroh Yoon.  Edit-a-video: Single video editing with object-aware consistency.  arXiv preprint arXiv:2303.07945, 2023.
  * [40] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman.  Make-a-video: Text-to-video generation without text-video data.  In ICLR, 2023.
  * [41] Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan.  Phenaki: Variable length video generation from open domain textual description.  In ICLR, 2023.
  * [42] Fu-Yun Wang, Wenshuo Chen, Guanglu Song, Han-Jia Ye, Yu Liu, and Hongsheng Li.  Gen-l-video: Multi-text to long video generation via temporal co-denoising.  arXiv preprint arXiv:2305.18264, 2023.
  * [43] Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang.  Modelscope text-to-video technical report.  arXiv preprint arXiv:2308.06571, 2023.
  * [44] Wen Wang, kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, and Chunhua Shen.  Zero-shot video editing using off-the-shelf image diffusion models.  arXiv preprint arXiv:2303.17599, 2023.
  * [45] Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou.  Videocomposer: Compositional video synthesis with motion controllability.  In NeurIPS, 2023.
  * [46] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo.  Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation.  In ICCV, 2023.
  * [47] Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan.  Godiva: Generating open-domain videos from natural descriptions.  arXiv preprint arXiv:2104.14806, 2021.
  * [48] Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan.  Nüwa: Visual synthesis pre-training for neural visual world creation.  In ECCV, 2022.
  * [49] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  In ICCV, 2023.
  * [50] Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou.  Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion.  In ICCV, 2023.
  * [51] Jinbo Xing, Menghan Xia, Yuxin Liu, Yuechen Zhang, Yong Zhang, Yingqing He, Hanyuan Liu, Haoxin Chen, Xiaodong Cun, Xintao Wang, Ying Shan, and Tien-Tsin Wong.  Make-your-video: Customized video generation using textual and structural guidance.  arXiv preprint arXiv:2306.00943, 2023.
  * [52] Zhen Xing, Qi Dai, Han Hu, Zuxuan Wu, and Yu-Gang Jiang.  Simda: Simple diffusion adapter for efficient video generation.  arXiv preprint arXiv:2308.09710, 2023.
  * [53] Jun Xu, Tao Mei, Ting Yao, and Yong Rui.  Msr-vtt: A large video description dataset for bridging video and language.  In CVPR, 2016.
  * [54] Han Xue, Zhiwu Huang, Qianru Sun, Li Song, and Wenjun Zhang.  Freestyle layout-to-image synthesis.  In CVPR, 2023.
  * [55] Shuai Yang, Yifan Zhou, Ziwei Liu, , and Chen Change Loy.  Rerender a video: Zero-shot text-guided video-to-video translation.  In SIGGRAPH Asia, 2023.
  * [56] Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang.  Reco: Region-controlled text-to-image generation.  In CVPR, 2023.
  * [57] Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang.  Freedom: Training-free energy-guided conditional diffusion model.  In ICCV, 2023.
  * [58] Yu Zeng, Zhe Lin, Jianming Zhang, Qing Liu, John Collomosse, Jason Kuen, and M. Patel, Vishal.  Scenecomposer: Any-level semantic image synthesis.  In CVPR, 2023.
  * [59] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In ICCV, 2023.
  * [60] Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian.  Controlvideo: Training-free controllable text-to-video generation.  arXiv preprint arXiv:2305.13077, 2023.
  * [61] Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan-Yee K Wong.  Uni-controlnet: All-in-one control to text-to-image diffusion models.  In NeurIPS, 2023.
  * [62] Yuyang Zhao, Enze Xie, Lanqing Hong, Zhenguo Li, and Gim Hee Lee.  Make-a-protagonist: Generic video editing with an ensemble of experts.  arXiv preprint arXiv:2305.08850, 2023.
  * [63] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng.  Magicvideo: Efficient video generation with latent diffusion models.  arXiv preprint arXiv:2211.11018, 2022.