See pages - of manuscript.pdf

#  Supporting Information for
Efficient automatic design of robots.

David Matthews, Andrew Spielberg, Daniela Rus, Sam Kriegman*, Josh Bongard

*To whom correspondence should be addressed.
Email<EMAIL_ADDRESS>

This PDF file includes:

Supporting text
SI References
Figures S1 to S13
Tables S1 to S3
Legend for Movie S1

Other supporting materials for this manuscript include the following:

Movie S1
Software S1

###### Contents

  1. Legend for Movie S1:
  2. S1. Simulation.
  3. S2. Optimization.
  4. S3. Physical experiments.
  5. S4. Supplemental experiments.
    1. 4.16. Number of muscles (Fig. S8D)
  6. Figures:
  7. Tables:

## Legend for Movie S1:

This movie summarizes the paper’s results (a novel robot is designed from
scratch in just 10 design attempts) and methods (using gradient based
optimization).

## S1. Simulation.

Each design attempt was modeled by a 64-by-44 partially-filled uniform grid of
present/absent elastic particles (at most 2816 particles; 20-by-14 cm2) and
simulated using the Material Point Method (MPM; [1]) for $T=1024$ timesteps
with step size $dt=0.001$ sec, for a total evaluation period of 1.024
simulated seconds. One second of simulation time can be computed faster or
slower than realtime, depending on the step size. Smaller step sizes are more
accurate than larger step sizes but require more steps (and thus more
wallclock time) to simulate 1 sec. The number of timesteps is also constrained
by memory and the accumulation of floating point rounding errors.

In the main experiment, which culminated in the fabrication and testing of an
automatically-designed physical robot (Figs. 1 and 2), and was repeated 99
times from different random initial configurations (Fig. 3), a design consists
of at most 64 internal voids, which remove contiguous circular regions of
particles from the body grid, and at most 64 muscles, which expand and
contract in volume by increasing and decreasing the distance between their
constituent particles. The 64 voids are moved and resized by optimization; a
the 64 muscles are moved by optimization but have have fixed size (radius of
actuation, $r_{c}=1.26$ cm). Voids and muscles can be removed based on their
location (off the grid); voids can also be removed by reducing their radius to
0. Unless otherwise noted, we will assume the hyperparameters used by the main
experiment.

The mass and elasticity of each particle was based on its position relative to
nearby voids. For each particle, $\varphi\in\Phi$, and void, $k\in K$, the
euclidean distance, $d_{\varphi,k}$, was calculated between every particle-
void pair, $\\{\varphi,k\\}$, and divided by the void’s current radius,
$r_{k}$, yielding a normalized distance,
$d_{\varphi,k}^{*}=\min(1,d_{\varphi,k}/r_{k})$. The mass of particle
$\varphi$ was then computed as:

$m_{\varphi}=\left[\,\min_{k\in K}(d_{\varphi,k}^{*})\right]^{2}\times\text{1
kg}\,,$ (1)

the square of the minimum normalized distance from the particle to every void
in the body. Particles toward the center of a void, with a mass less than
$\lambda=0.1$ kg, were removed from simulation. The particle’s elasticity,
$E_{\varphi}$, was set to be proportional to its mass:
$E_{\varphi}=m_{\varphi}\times\text{20 Pa}$.

Simulated muscles undergo six actuation cycles during the 1024 timesteps of
simulation. At each time step, viscous damping is applied to each particle,
the actuation state of each particle is calculated, and a vertical expansion
or contraction force is generated. The actuation amplitude of each particle is
based on its position relative to nearby muscles. For every muscle, $c\in C$,
the euclidean distance, $d_{\varphi,c}$, between each particle-muscle pair,
$\\{\varphi,c\\}$, was calculated and normalized as in Eq. 1. The actuation
amplitude of particle $p$ was then computed as:

$A_{\varphi}=\left(1-\left(d_{\varphi,c}^{*}\times\sqrt{0.1}\right)\right)^{2},$
(2)

a quadratic interpolation with a minimum amplitude of 0.45 at the border of
each muscle patch. Particles outside of muscle patches, with
$d_{\varphi,c}^{*}=1$, were assigned zero actuation amplitude.

The actuation state of each particle at simulation time $t$ was:

$\omega_{\varphi,t}=4\times
m_{\varphi}\times\tanh(A_{\varphi}\times\sin(t\times dt\times f)),$ (3)

the hyperbolic tangent of a sinusoidal function with amplitude $A_{\varphi}$
and frequency $f=\text{6 Hz}$, normalized by the particle’s mass and relative
to a global actuation strength of 4 Pa per particle. Normalizing by mass helps
avoid numerical instabilities caused by tearing, which can occur from high
particle acceleration due to low mass and high force.

Other void (Fig. S8A) and muscle (Fig. S8B) radii, other numbers of voids
(Fig. S8C) and muscles (Fig. S8D), other interpolation functions for the
particles inside voids (Fig. S8E) and muscles (Fig. S8F), and other thresholds
for particle removal within voids (Fig. S8G) were explored in a subsequent
hyperparameter sweep (Figs. S7-S9) and detailed in Table S2.

## S2. Optimization.

Initialization. Optimization begins with a randomly generated design. Random
designs comprised 64 internal voids and 64 muscles, randomly sized and
positioned along the robot’s body. The position of each patch was selected
from a uniform distribution across the robot’s body (20-by-14 cm2). The radius
of each muscle was fixed at $r_{c}=1.26$cm. The initial radius of each void
was drawn from a normal distribution with mean $\mu=0.046\times 20$cm and
standard deviation of $0.046^{2}\times 20=0.042$cm. This removes 60% of the
2816 particles, on average, under the assumption that the voids are non-
overlapping. In other words, if the 64 sampled voids are arranged side-by-side
on a grid, their a combined area (their “grid coverage”; $O_{K}$) will be
approximately $O_{K}=60\%$ of the robot’s body area, on average. Several other
distributions of initial radius (Table S2) were explored in the hyperparameter
sweep (Fig. S8A,B).

Fitness. The fitness of a design was taken to be the mean forward velocity of
the robot, which was calculated as the mean position of all of the particles
at the final time step subtracted by the mean position of all of the particles
at the first time step.

Loss. Gradient descent toolkits typically default to minimizing an error
function rather than maximizing a fitness function.111 Gradient based
optimization methods have historically used the metaphor of a ball on a hill:
descending a slope (of error or “loss”) within a landscape defined by some
objective function. Evolutionary biology has instead used the metaphor of hill
climbing: evolution as an active force that ascends slopes along a fitness
landscape. These two metaphors, of maximizing fitness (negative error) and
minimizing error (negative fitness), are functionally isomorphic.  For this
reason, the loss function was defined as -1 $\times$ average forward velocity.
That is, negative fitness (locomotive deficiency).

Backpropagation. Backpropagation was performed [2] through the differentiable
simulator, yielding gradients of loss (negative fitness) with respect to
initial particle parameters. Initial attempts to directly optimize these
initial particle parameters did not yield any changes in topology (Fig. S5).
Internal voids, if they possess non-zero radius, explicitly remove particles,
and thus enable robot topology optimization. Gradients were therefore
backpropagated one step further, from the particle parameters through the
patch-based design functions (Eqs. 1-3), yielding gradients of loss with
respect to the patch parameters (positions and sizes of voids, and positions
of muscles).

Gradient descent. Gradient descent using the Adam optimizer [3] with a
learning rate of 0.01 was used to optimize the design parameters. Gradients
from simulation were used to perform nine gradient descent steps, which,
including the initial randomly generated design, was 10 design attempts.

## S3. Physical experiments.

Simulation-to-reality transfer. The physical robot (5 cm wide) was
manufactured as a 4:1 scale model of its simulated counterpart (20 cm wide),
and driven by pneumatic actuation. To do so, the soft fringe of quadratically
decreasing mass/elasticity inside each simulated void (Eq. 1) was sharpened
with a binary step function describing where silicone will and will not be
present. The particles closest to the fully-removed core of each simulated
void, which have very low stiffness and minimally contributed to robot’s
structure, were removed before building the design. More specifically, the
physical voids fully remove material in a radius that is 1.3 times larger than
the fully removed portion of the simulated voids. Outside this radius,
material is fully present. The physical void radius is also used in the
figures when displaying void area rather than their influence on each
particle.

In simulation, muscles expanded along the y axis and were permitted to “hang
off the body” along the exterior surface of the design. When manufacturing the
robot as a silicone body with hollow bladders, actuated regions within 3 mm of
the edge of the body were filled with passive material (Fig. S4A).
Incorporating this post-processing step as a optimization constraint (Sect.
4.8. Constrained actuation (Fig. S4).) results in the very same locally-
optimal design when optimizing from the same randomly generated initial
design.

Physical muscles were connected together with small cutouts, and horizontal
stiffeners were added to the actuation bladders to limit the horizontal
component of expansion when pressure is applied. Finally, a pneumatic port was
also added to the center of the top of the robot. The robot was split along
the sagittal plane, and a three-part mold was generated for each half (Fig.
S1). The mold (six parts in total) was then printed using a Markforged X7 3D
printer with Onyx filament.

Casting the robot. Each mold part was covered in a mold release. The bottom
and middle parts of the mold were assembled (Fig. S1A-B). Smooth-On Dragon
Skin 10 Fast was mixed and degassed in a vacuum chamber for 7 min. Each mold
was filled with silicone, and the third mold piece (lid) was inserted. A
weight was then placed on top of the lid to hold it in place during the 75 min
curing period. The cured robot halves were removed from the molds, and
silicone flashing was trimmed off with scissors and a scalpel. Silicone rubber
tubing was glued into one of the robot halves with Sil-Poxy and allowed to
cure for 12 min. The two robot halves were then siliconed together with
another batch of degassed silicone and allowed to cure for another 75 min.
Four retroreflective balls were glued onto the top four corners of each robot
using Sil-Poxy.

Evaluation. Cornstarch (Argo) was placed on top of a sheet of plastic and
raked to have a thin grooved surface. Two alignment rails were added to the
surface to simplify the comparison of the simulated robot (which can only move
in a straight line) and the physical robot (which has a slightly curved gait
due to asymmetries along its sagittal seam where its two halves meet). The
robot was placed between the two alignment rails and connected to a Festo
proportional pressure control valve (PPM-6TA-L-1-F-0L2H-S1) that generated a
square wave with an amplitude of 300mBar, a wavelength of 500ms, and a duty
cycle of 50%. The robot was actuated for 1 min and its behavior was captured
using a motion capture system (OptiTrack). After evaluating the robot’s
behavior, the environment was reset. Six behavioral trajectories of the
optimized robot (Attempt 10) were collected and five were collected of the
unoptimized robot (randomly generated; Attempt 1). The optimized design moved
significantly further on average (winning design vs randomly generated robot:
MannWhitneyU test; $p<0.01$).

## S4. Supplemental experiments.

Additional experiments were conducted in silico to promote the efficient use
of building material (Sects. 4.1. Eroding unnecessary body parts (Fig.
4A-B,E-G). and 4.2. Minimizing rotational moment (Fig. 4A-B,F).), round sharp
edges and corners on the body (Sect. 4.3. De-brickifying bodies (Fig. 4C-D).),
re-place deactivated voids/muscles (Sect. 4.4. Replacing deactivated patches
(Fig. 4A-G).), include agnostic and antagonistic musculature (Sect. 4.5.
Agonistic and antagonistic musculature (Fig. 4H-L).), achieve other behaviors
besides locomotion (Sect. 4.6. Object manipulation (Fig. 4E-G).), enable
interactive design from user-defined initial morphologies (Sect. 4.7.
Interactive design (Fig. 4H-L).), incorporate details of the manufacturing
process as optimization constraints instead of post-processing steps (Sect.
4.8. Constrained actuation (Fig. S4).), directly optimize particles instead of
explicit voids and muscle patches (Sect. 4.9. Direct particle optimization
(Fig. S5).), and explore other methods of interpolation within voids (Sect.
4.10. Intra-void interpolation (Figs. S6 and S8E).) and muscles (Sect. 4.11.
Intra-muscle interpolation (Fig. S8F).), other thresholds for particle removal
within voids (Sect. 4.12. Threshold for removal (Fig. S8G)) other initial
distributions of void (Sect. 4.13. Initial void radius distribution (Figs.
S8A)) and muscle (Sect. 4.14. Initial muscle radius distribution (Fig. S8B))
radius, as well as different numbers of voids (Sect. 4.15. Number of voids
(Fig. S8C)) and muscles (Sect. 4.16. Number of muscles (Fig. S8D)). We
repeated each experiment (each condition) across several independent trials;
each repeated trial starts from a unique random initial configuration (a
unique random seed).

### 4.1. Eroding unnecessary body parts (Fig. 4A-B,E-G).

To promote efficient use of building materials we combined the gradients from
simulation with a global material erosion term, which encourages the removal
of unnecessary body parts by applying a pressure for a robot’s total mass to
reach some target amount. More specifically, the gradient of fitness (loss;
$\ell$) with respect to particle mass ($m_{\varphi}$) was computed as a linear
combination of simulation loss ($\ell_{\text{sim}})$ and the erosion term:

$\frac{\partial\ell}{\partial
m_{\varphi}}=\frac{\partial\ell_{\text{sim}}}{\partial
m_{\varphi}}+\alpha\times\left[\,\max_{\varphi\in\Phi}\left(\frac{\partial\ell}{\partial
m_{\varphi}}\right)-\min_{\varphi\in\Phi}\left(\frac{\partial\ell}{\partial
m_{\varphi}}\right)\right]\times\tanh\left(\beta\times\left(\bar{m}_{\varphi}-m^{\star}\right)\right),$
(4)

where $\bar{m}_{\varphi}$ is the average particle mass (including removed
particles with zero mass); coefficients $\alpha=0.01$ and $\beta=10$; and a
target average mass $m^{\star}\in\\{0.25,0.5,0.75\\}$. The hyperbolic tangent
function is an S-shaped curve that caps the material erosion gradients at 1,
and the constant $\beta$ compresses the curve to be steeper and more rapidly
approach 1. Thus, with $\alpha=0.01$, the magnitude of the erosion term is
scaled to 1% of the range of particle gradients from simulation. As a result,
gradient descent will tend only to remove (or add) material in ways that
minimally impact the behavior of the robot. Pressure to add material occurs if
the robot’s mass falls below the target; however, the target mass was chosen
to be low enough that this was not observed.

### 4.2. Minimizing rotational moment (Fig. 4A-B,F).

In addition to augmenting the gradients through a material erosion term (Sect.
4.1. Eroding unnecessary body parts (Fig. 4A-B,E-G).), a second augmentation
term to minimize rotational moment of inertia was also introduced. This term
helps to keep the robots compact, especially when used in combination with
other terms which encourage removal of material writ large (Fig. 4A-B, F). To
do so we first computed the robot’s center of mass, $\Phi_{x,y}$, as well as
the euclidean distances between $\Phi_{x,y}$ and each particle $\varphi$,
which we denote $\nu_{\varphi}$. The rotational moment was then computed as:

$\displaystyle\nu_{\varphi}$ $\displaystyle=D(\varphi,\Phi_{x,y})$ (5)
$\displaystyle h(\nu_{\varphi})$
$\displaystyle=\begin{cases}\nu_{\varphi}&\text{if }\nu_{\varphi}<0.25\\\
2\nu_{\varphi}-0.25&\text{otherwise}\\\ \end{cases}$ (6)
$\displaystyle\ell_{I}$ $\displaystyle=\sum_{\varphi\in\Phi}(m_{\varphi}\times
h(\nu_{\varphi})).$ (7)

This loss was then backpropagated through to the mass of each particle and
linearly combined with the particle gradients from simulation:

$\frac{\partial\ell}{\partial
m_{\varphi}}=\frac{\partial\ell_{\text{sim}}}{\partial
m_{\varphi}}+\gamma\frac{\ell_{I}}{\partial m_{\varphi}},$ (8)

where $\gamma\in\\{0.0001,0.0005,0.001\\}$.

### 4.3. De-brickifying bodies (Fig. 4C-D).

Initializing designs on a rectangular grid left a bias for brick-like bodies
with square dorsal-anterior and ventral-posterior edges. To mitigate this bias
we explored an alternative fitness (loss) function which rounds the edges of
the body:

$\displaystyle\ell_{\circ}$
$\displaystyle=\sum_{\varphi\in\Phi}m_{\varphi}\left(2\left(d_{\varphi,\circ}>r_{\circ}\right)-1\right)$
(9) $\displaystyle\ell$
$\displaystyle=\ell_{sim}+\gamma_{\circ}\times\ell_{\circ},$ (10)

where $\gamma_{\circ}\in[0.00001,0.0001]$. This additional term promotes the
addition of material inside of a circle $\circ$ of radius $r_{\circ}$, and the
removal material outside of it.

### 4.4. Replacing deactivated patches (Fig. 4A-G).

When a patch (void or muscle) is removed (reduced to radius zero, moved off
the body, or overlapping another patch) it no longer impacts particle mass (in
the case of voids) nor actuation (in the case of muscles) and thus no longer
receives any gradients from simulation. It remains to determine if
deactivating patches in this way caused or could cause premature convergence
to suboptimal designs. If it does, a potential solution could be re-placing
each removed patch back onto the body at a new, random location, and restoring
its initial radius. This strategy was explored for the robots presented in
Fig. 4A-G.

### 4.5. Agonistic and antagonistic musculature (Fig. 4H-L).

Agonistic and antagonistic musculature were implemented using two different
sets of muscle patches that actuate in antiphase (sine and cosine) at the same
frequency and amplitude. The actuation state of each particle,
$\omega_{\varphi,t}$, was computed as the harmonic addition of the two
channels of actuation (i.e. sine and cosine waves). The contribution from each
channel of actuation was computed as described in Eq. 2.

### 4.6. Object manipulation (Fig. 4E-G).

A round object of 208 particles (radius 2.5 cm) was added to the simulation,
just above the top of the robot, and centered front to back. The robot was
then optimized for: locomotion (maximize robot velocity; Fig. 4E), object
transportation (maximize average velocity of robot and object; Fig. 4F); and
object ejection (maximize velocity of object, Fig. 4G).

### 4.7. Interactive design (Fig. 4H-L).

The assumption of an initial rectangular grid was relaxed and replaced with
any arbitrary user-supplied 2D shape, such as triangles (Fig. 4H,I), a circle
(Fig. 4J), a heptagram (Fig. 4K) or enneagram (nine-sided star; Fig. 4L).

### 4.8. Constrained actuation (Fig. S4).

Manufacturing the automatically-designed robot required a post-processing step
to be applied to the simulated design after optimization (described above in
Sect. S3. Physical experiments.). An additional experiment was thus conducted
to determine if this post-processing step could instead be incorporated into
the simulation ab initio as a optimization constraint, so that simulated
designs could directly be instantiated with closed-bladder actuators. To
achieve this, particles close to the border were forbidden from actuating even
when they were close to a patch of muscle (Fig. S4A). To resolve the passive
border, the resolution of the simulated robot was increased by a factor of two
(128 particles in width and 88 in height; a total of 11264 particles).
Starting with the same random initial configuration, a nearly identical
morphology emerged after ten design attempts with (Fig. S4C) and without (Fig.
1C-L) constrained musculature. A slight yet significant increase in fitness
was observed in silico when actuation was constrained ($p<0.01$); however, it
is unclear whether or not this is an artifact of the higher particle count
(11264 vs 2816 particles).

### 4.9. Direct particle optimization (Fig. S5).

The necessity of internal voids for optimizing robot shape can be seen when
voids are replaced by a particle-based representation that directly stores and
optimizes the mass and elasticity of each particle using gradient descent.
Holding all other variables constant, including the use of muscle patches, the
direct particle representation yielded robots with significantly lower fitness
than that of robots optimized via explicit voids $(p<0.01$; Fig. S5A). This
held true even when the direct particle representation was allowed 40
additional design evaluations ($p<0.01$). Moreover, without internal voids,
shape change did not occur. In every one of the 25 independent trials of
direct particle optimization, all initial particles remained part of the body
after 50 design evaluations. This is significantly different ($p<0.01$) to the
void-based representation which typically reduces the total present particle
count by 6% over the first 10 design attempts (Fig. S5B).

### 4.10. Intra-void interpolation (Figs. S6 and S8E).

Each internal void contains a “soft fringe” that gently interpolates erasure,
quadratically, from outer edge to mid-center (Figs. 2C and S7), prior to fully
removing the particles below a mass threshold at their inner core (Sect. 4.12.
Threshold for removal (Fig. S8G)). Some kind of interpolation is required
along a void’s fringe as a binary step function (fully present particles
encircling completely removed particles) would not allow gradients to
propagate from simulation to the design parameters (void size and position).
This is because, with a step function, given an $\epsilon>0$, there does not
exist any $\delta>0$ where a change of the design parameters by an amount of
$\delta$ always results in every particle changing its mass by an amount of
less than or equal to $\epsilon$. This property is required for
backpropagation.

To evaluate the utility of the soft fringe, additional experiments were
conducted in which this quadratic interpolation was replaced with linear and
cubic interpolation regimes (Fig. S7C). Under these regimes the mass of each
particle is set to:

$\displaystyle d=$ $\displaystyle\min_{k\in K}(d_{\varphi,k}^{*},1)$ (11)
$\displaystyle m_{\varphi}=$ $\displaystyle\begin{cases}0&\text{if
}d^{q}<\lambda\\\ d^{q}&\text{otherwise,}\end{cases}$ (12)

with $q\in\\{1,2,3\\}$ yielding linear, quadratic, and cubic interpolation,
respectively (Fig. S8E), for particles within voids with mass above the
threshold of particle inclusion $\lambda\in\\{0.1,0.2,0.4\\}$ (Fig. S8G). In
Fig. S6B, $\lambda$ was set to $0.5$ for the linear interpolation function.

The fitness of designs with the soft (quadratic) fringe was significantly
higher than that of the hard (linear) fringe ($p<0.01$). However there was not
an appreciable difference in fitness for designs optimized under quadratic and
cubic interpolation regimes. While a soft fringe facilitated optimization
under the tested conditions, it remains to determine precisely how different
interpolation regimes alter the search landscape and gradients.

### 4.11. Intra-muscle interpolation (Fig. S8F).

To evaluate the importance of the original quadratic interpolation within each
actuator patch, additional experiments were conducted in which the quadratic
interpolation was replaced with other interpolation regimes. Under these
regimes actuation amplitude of each particle was set to:

$A_{\varphi}=(1-(d_{\varphi,c}^{*}\times\sqrt{0.1}))^{q},$ (13)

where $q\in\\{1,2,3\\}$. Gradient descent does not appear to be sensitive to
this (or any other tested) hyperparameter and retains supremacy over previous
approaches (solid versus dotted lines in Fig. S8) in all regimes. Note that
different muscle interpolations can generate very different actuation
signatures across the robot’s body. So, we should be careful about drawing
conclusions about which interpolation regime is the best.

### 4.12. Threshold for removal (Fig. S8G)

Particles along the outer fringe of voids decrease in physicality (mass and
elasticity; Sect. 4.10. Intra-void interpolation (Figs. S6 and S8E).) until
their mass falls below a threshold, $\lambda$, after which they are completely
removed from simulation. The physical robot was optimized in simulation with
$\lambda=0.1$ kg (Fig. 2). Later, two higher ($\lambda=0.2,0.4$ kg) thresholds
were explored (Fig. S8G). Fitness slightly (but significantly) decreased when
the threshold was raised to 0.2 kg and slightly (but significantly) decreased
further when threshold was raised to 0.4 kg. However, different thresholds
have different numbers of particles and different material properties. So,
this comparison is not “apples to apples” and does not suggest smaller
thresholds are better. It does show, however, that gradient descent is not
overly sensitive to changes in this hyperparameter, which is interesting since
it directly affects the gradients.

### 4.13. Initial void radius distribution (Figs. S8A)

The distribution of void radii used to design the physical robot (Sect. S2.
Optimization.) was replaced by one of eight other distributions (Fig. S7A):
$\text{Unif}(0,2a),\;\text{Unif}(a-a^{2},a+a^{2}),\;\text{Unif}(0,2b),\;\text{Unif}(b-b^{2},b+b^{2}),\;\text{Norm}(a,a),\;\text{Norm}(a,a^{2}),\;\text{Norm}(b,b),\;\text{Norm}(b,b^{2})$.
When $|K|$ voids are sampled from one of these distributions and arranged
side-by-side on a grid, their grid coverage,

$O_{K}=\frac{\pi}{W}\sum_{k\in K}r_{k}^{2}\;,$ (14)

will be either $O_{K}=30\%$ (when $\mu=a$) or $O_{K}=60\%$ (when $\mu=b$) of
the robot’s body area, $W$, on average.

### 4.14. Initial muscle radius distribution (Fig. S8B)

The muscle radius used to design the physical robot (Sect. S2. Optimization.)
was replaced by radii sampled from one of the same eight distributions used
for voids in Sect. 4.13. Initial void radius distribution (Figs. S8A). That
is:
$\text{Unif}(0,2a),\;\text{Unif}(a-a^{2},a+a^{2}),\;\text{Unif}(0,2b),\;\text{Unif}(b-b^{2},b+b^{2}),\;\text{Norm}(a,a),\;\text{Norm}(a,a^{2}),\;\text{Norm}(b,b),\;\text{Norm}(b,b^{2})$.
When $|C|$ muscles are sampled from one of these distributions and arranged
side-by-side on a grid, their grid coverage, $O_{C}$, which sums muscle areas
in a similar fashion as Eq. 14, will be either $O_{C}=57.5\%$ (when $\mu=a$)
or $O_{C}=115\%$ (when $\mu=b$) of the robot’s body area, on average. In
practice, however, the muscles always overlap.

### 4.15. Number of voids (Fig. S8C)

The $|K|=64$ voids used to design the physical robot were replaced by 8, 16,
32, 128, and 256 voids (Fig. S7B).

### 4.16. Number of muscles (Fig. S8D)

The $|C|=64$ muscles used to design the physical robot were replaced by 8, 16,
32, 128, and 256 muscles (Fig. S7B).

## References

  * [1] Sulsky, D., et al. A particle method for history-dependent materials. Computer Methods in Applied Mechanics and Engineering, 118, 179-196 (1994)
  * [2] Griewank, A. On automatic differentiation. Mathematical Programming: Recent Developments and Applications, 6, 83-107 (1989)
  * [3] Kingma, D. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).

## Figures:

Figure S1: Manufacture. A set of molds were 3D printed for each of the initial
(A) and final (B) design attempts (mirrored molds of a-b not shown). Silicone
was poured into the molds, and two robot halves (bottom: yellow, blue) were
removed. A pneumatic tube was glued into the blue mold with Sil-Poxy and then
the yellow robot half was siliconed onto the blue robot half and allowed to
cure (C). After curing, the robot was rotated into an upright posture and
connected to a pneumatic pressure control system (D). Figure S2: Attempt 1.
All 100 random initial configurations (first attempts) across 100 independent
trials. Green CoM traces indicate that the initial designs primarily bounce up
and down in place. Figure S3: Attempt 10. Optimized designs from the 100
independent trials starting with the 100 different random initial
configurations (random seeds) in Fig. S2. Green CoM traces indicate all
optimized designs were able to locomote forward with stable gaits.
Morphological convergence to posterior-angled legs with anterior musculature
can be observed across these 100 independent trials. However, this design
principle is embodied in the 100 designs in 100 different ways, each with
their own unique body shape, number of limbs (2 to 4), and overall
musculature. Figure S4: Design with constraints. Morphology was constrained
such that muscles are buildable as closed pneumatic bladders (A). This did not
render the body less differentiable in preliminary experiments in terms of
locomotion ability. In fact there seems to be a slight but significant
improvement in fitness (B). However, this may be an artifact stemming from the
use of four times as many particles used to simulate constrained actuation
(higher resolution is needed to resolve the passive border; Table. S2). (C:)
Ten design attempts starting from the same initial design parameters as in
Fig. 1C. The final constrained actuation design looks visually similar to the
final unconstrained design from Figure 1L.  Figure S5: The necessity of
explicit voids. Without internal voids, fitness was greatly diminished
($p<0.01$; A), and shape change did not occur (B). Design attempts 1 through
10 exhibit show minimal changes in both material properties and fitness (C).
Further design attempts (attempts 20 to 50; D-G) hold on to the same
rectangular body plan and actuation placements, and they exhibit unstable
gaits. Figure S6: The utility of intra-void interpolation. A smooth
(quadratic) interpolation was used between the edge and center of each void
($q=2,\lambda=0.1$; A), which rendered the body more differentiable than a
sharper (linear) interpolation ($q=1,\lambda=0.5$; B) in terms of the fitness
achieved after ten design attempts (C). Morphologies optimized with more
sharply interpolated voids (D) were very similar to those with smoother
interpolation: both have posterior-angled legs with anterior musculature.
Figure S7:  Summary of hyperparameter sweep. We repeated our experiment under
tens of thousands of different conditions: different combinations of
hyperparameter values (detailed in Table S2). These hyperparameters determine
the distribution of initial radius for voids and muscles (A), the number of
voids and muscles (B), the interpolation within voids (C) and muscles, and the
threshold for particle removal within voids (D). The original hyperparameters
used to optimize the physical robot (Figs. 1-3) are bolded. The results of the
hyperparameter sweep can be seen in Figs. S8 and S9.  Figure S8: Robustness to
hyperparameters. We repeated our experiment under 31104 different conditions,
spanning eight different distributions of initial radius (four uniform and
four normal distributions, with different locations and scales, whose means
correspond to $|K|$ sampled voids with areas that sum to 30% of the robot’s
body area, when $\mu=a$, or 60%, when $\mu=b$; A,B), six different numbers of
voids (8, 16, 32, 64, 128, 256; C), six different numbers of muscles (8, 16,
32, 64, 128, 256; D), three different methods of void interpolation (linear,
quadratic, cubic; E), three options of muscle interpolation (linear,
quadratic, cubic; F), and three different thresholds for removal of material
(10%, 20%, 40%; G), comparing gradient descent (solid lines) to previous
methods (dotted lines) with a budget of 10 design attempts (H). Means are
plotted with 99% normal confidence intervals with Bonferroni correction for 38
comparisons (solid lines vs dotted lines). Figure S9: Locally optimal robots.
(A-T:) The most fit robot found by gradient descent in each of the 38
comparisons of Fig. S8 Because these comparisons aggregate results across
overlapping sets of hyperparameter values, the best overall robot (T) was the
best in 8 of the 38 comparisons. Other high performing robots also dominated
multiple comparisons, yielding the 20 unique individuals plotted here.
Gradient descent converged with high probability to local peaks in the non-
convex search landscape: Distinct morphologies—different body shapes, numbers
of limbs (3 to 5), and musculature (red)—capable of legged locomotion (green
curves trace the robot’s CoM during behavior). Designs are sorted from least
(A) to most (T) muscle coverage.  Figure S10: Shape change (runs 5-28). (Runs
1-4 can be found in Fig. 3.) Position and size of the internal voids from the
random initial configurations (cyan circles) to final optimized body plan
(purple circles; outlined by a black dashed line) for independent trials 5
through 28. For each run, the behavior is shown for each of the 10 design
attempts (cyan to purple waves). Figure S11: Shape change (runs 29-52).
Position and size of the internal voids from the random initial configurations
(cyan circles) to final optimized body plan (purple circles; outlined by a
black dashed line) for independent trials 29 through 52. For each run, the
behavior is shown for each of the 10 design attempts (cyan to purple waves).
Figure S12: Shape change (runs 53-76). Position and size of the internal voids
from the random initial configurations (cyan circles) to final optimized body
plan (purple circles; outlined by a black dashed line) for independent trials
53 through 76. For each run, the behavior is shown for each of the 10 design
attempts (cyan to purple waves). Figure S13: Shape change (runs 77-100).
Position and size of the internal voids from the random initial configurations
(cyan circles) to final optimized body plan (purple circles; outlined by a
black dashed line) for independent trials 77 through 100. For each run, the
behavior is shown for each of the 10 design attempts (cyan to purple waves).

## Tables:

Table S1: Symbols. The following symbols are used above in Sects. S1-S4. Symbol | Description
---|---
$t\in\\{1,2,\ldots,T\\}$ | Timestep
$dt$ | Step size
$W$ | Workspace length$\,\times\,$height [cm2]
$k\in K$ | Void
$c\in C$ | Muscle
$r_{k}$ | Void radius
$r_{c}$ | Muscle radius
$O_{K}$ | grid coverage of $K$: summed void areas relative to robot’s area.
$O_{C}$ | grid coverage of $C$: summed muscle areas relative to robot’s area.
$\varphi\in\Phi$ | Particle
$m_{\varphi}$ | Particle mass [kg]
$\bar{m}_{\varphi}$ | Avg. particle mass (includes removed particles with zero mass) [kg]
$E_{\varphi}$ | Particle elastic modulus [Pa]
$\Phi_{x,y}$ | Robot’s center of mass (CoM)
$\lambda$ | Intra-void particle removal threshold [kg]
$m^{\star}$ | Target average mass
$A_{\varphi}$ | Particle actuation amplitude
$\omega_{\varphi,t}$ | Actuation state of particle $\varphi$ at time $t$
$f$ | Actuation frequency [Hz]
$q$ | Intra-void/muscle interpolation power
$D(\mathbf{x}_{1},\mathbf{x}_{2})$ | Euclidean dist. between cartesian coordinates $\mathbf{x}_{1}$ and $\mathbf{x}_{2}$ [cm]
$d_{\varphi,c}=D(\varphi,c)$ | Euclidean dist. from $\varphi$ to $c$ [cm]
$d_{\varphi,k}=D(\varphi,k)$ | Euclidean dist. from $\varphi$ to $o$ [cm]
$d_{\varphi,c}^{*}$ or $d_{\varphi,k}^{*}$ | Normalized distance (dist. divided by $r_{c}$ or $r_{k}$)
$\nu_{\varphi}=D(\varphi,\Phi_{x,y})$ | Dist. of a particle to the robot’s CoM
$\ell$ | Loss (negative fitness)
$\ell_{\text{sim}}$ | Simulator loss: the gradients from the differentiable physics simulation
$\ell_{\text{addtl}}$ | Additional losses that were combined with $\ell_{\text{sim}}$ during optimization.
Table S2: Hyperparameters. The following hyperparameters were used in the experiments detailed above in Sects. S1-S4; those used to design the physical robot (Sect. S4) are bolded. Variable | Value
---|---
Particle count |  $\mathbf{64\times 44=2816}$; and $128\times 88=11264$ (Fig. S4).
Workspace length by height, $W$ | $W=20\times 14=280\text{ cm}^{2}$
Void grid coverage, $O_{K}$ | $O_{K}=\frac{\pi}{W}\sum r_{k}^{2}=30\%\text{ or }\mathbf{60}\textbf{\%}$
Muscle grid coverage, $O_{C}$ | $O_{C}=\frac{\pi}{W}\sum r_{c}^{2}\approx 57.5\%\text{ or }\mathbf{115}\textbf{\%}$
Young’s modulus | 20 Pa
Poisson’s Ratio | 0.25
Actuation strength | 4.0 Pa / particle
Time steps ($T$) | $T=1024$
Step size ($dt$) |  $dt=0.001$ seconds
MPM grid size | $128\times 128$
Gravity | $5.4\text{m}/\text{s}^{2}$
Friction | 0.5
Internal viscous damping | 30.0
Global viscous damping | 2.0
Intra-void/muscle interpolation, $q$ | $q\in\\{1,\textbf{2},3\\}$
Threshold for removal, $\lambda$ | $\lambda\in\\{\textbf{0.1},0.2,0.4\\}\text{ kg}$
Number of voids, $|K|$ | $|K|\in\\{8,16,32,\textbf{64},128,256\\}$
Number of muscles, $|C|$ | $|C|\in\\{8,16,32,\textbf{64},128,256\\}$
Initial mean void radius in silico (includes fringe), $r_{k}$ | $r_{k}=\sqrt{\frac{O_{K}\times W}{\pi\times|K|}}\approx\mathbf{0.91}\;\textbf{cm}$
Physical void radius (no fringe) | $r_{k}\sqrt{0.1}\times 1.3\approx\mathbf{0.37}\;\textbf{cm}$
Muscle radius, $r_{c}$ | $r_{c}=\sqrt{\frac{O_{C}\times W}{\pi\times|C|}}\approx\mathbf{1.26}\;\textbf{cm}$
Actuation frequency, $f$ | $f=\frac{40}{2\pi}\approx 6\text{ Hz}$
Initial void radius dist. |  $\\{\text{Norm}(a,a),\text{Norm}(a,a^{2}),\text{Norm}(b,b),\text{{Norm}}(\mathbf{b,b^{2}})$, $\quad\text{Unif}(a-a,a+a),\text{Unif}(a-a^{2},a+a^{2}),\text{Unif}(b-b,b+b),$ $\quad\text{Unif}(b-b^{2},b+b^{2})\\}$, with $a$ and $b$ corresponding to $O_{K}=30\%$ and $\mathbf{60}\textbf{\%.}$
Initial muscle radius dist. |  $\\{\text{Norm}(a,a),\text{Norm}(a,a^{2}),\text{Norm}(b,b),\text{Norm}(b,b^{2}),\mathbf{b,}$ $\quad\text{Unif}(a-a,a+a),\text{Unif}(a-a^{2},a+a^{2}),\text{Unif}(b-b,b+b),$ $\quad\text{Unif}(b-b^{2},b+b^{2})\\}$, with $a$ and $b$ corresponding to $O_{C}=57.5\%$ and $\mathbf{115}\textbf{\%}$.
Table S3: Mold parameters. The following parameters were used to 3D print the automatically-designed mold in which the robot was cast. Variable | Value
---|---
Body width | 5 cm
Body height | 3.5 cm
Body thickness | 1.4 cm
Wall thickness | 0.15 cm
Actuation port size | 0.20 cm
Dilation amount | 1.3


# ReVideo: Remake a Video with Motion and Content Control

Chong Mou1,2, Mingdeng Cao3,4, Xintao Wang3∗, Zhaoyang Zhang3, Ying Shan3,
Jian Zhang1,2
1School of Electronic and Computer Engineering, Peking University
2Peking University Shenzhen Graduate School-Rabbitpre AIGC Joint Research
Laboratory
3ARC Lab, Tencent PCG 4 University of Tokyo
https://mc-e.github.io/project/ReVideo/ Corresponding author

###### Abstract

Despite significant advancements in video generation and editing using
diffusion models, achieving accurate and localized video editing remains a
substantial challenge. Additionally, most existing video editing methods
primarily focus on altering visual content, with limited research dedicated to
motion editing. In this paper, we present a novel attempt to Remake a Video
(ReVideo) which stands out from existing methods by allowing precise video
editing in specific areas through the specification of both content and
motion. Content editing is facilitated by modifying the first frame, while the
trajectory-based motion control offers an intuitive user interaction
experience. ReVideo addresses a new task involving the coupling and training
imbalance between content and motion control. To tackle this, we develop a
three-stage training strategy that progressively decouples these two aspects
from coarse to fine. Furthermore, we propose a spatiotemporal adaptive fusion
module to integrate content and motion control across various sampling steps
and spatial locations. Extensive experiments demonstrate that our ReVideo has
promising performance on several accurate video editing applications, i.e.,
(1) locally changing video content while keeping the motion constant, (2)
keeping content unchanged and customizing new motion trajectories, (3)
modifying both content and motion trajectories. Our method can also seamlessly
extend these applications to multi-area editing without specific training,
demonstrating its flexibility and robustness.

##  1 Introduction

Thanks to the large-scale training data and huge computing power, there have
been significant advancements in diffusion-based [18] image and video
generation. For personalization purposes, many works add control signals to
the generation process, such as text-guided image [35, 36, 34] and video [17,
16, 13] generation, as well as image-guided video generation [4, 49]. Based on
these base models, extensive works explore how to transfer their generation
capabilities to video editing. Early works based on text-to-image diffusion
models implement video editing through zero-shot strategies (e.g., Fate-Zero
[32], Flatten [11]) or one-shot tuning (e.g., Tune-A-Video [46]). However,
these methods are limited by excessive manual design and a lack of video
generation priors. Moreover, text prompt only provide coarse condition,
limiting the editing accuracy. Compared to text, more recent methods adopt
image conditions which can provide more accurate editing guidance. For
instance, VideoComposer [42] generates style-transformed videos by providing
spatial attributes (e.g., edge, depth) of the target video and a style
reference. DreamVideo [44] and Make-a-protagonist [57] can modify a specific
object in the video by providing a reference object. However, these methods
still struggle with local editing and introducing new elements, such as adding
new objects to a video. Recent work EVE [38] proposes a diffusion distillation
strategy to achieve video editing while keeping unedited content unchanged.
Nevertheless, the editing region and target are controlled by text, which is
challenging in complex scenarios. AnyV2V [25] edit a video by modifying the
first frame, enabling accurate customization of local content. Pika [1] can
regenerate a specific area in the video by selecting an editing region.
Although these methods improve the performance of local video editing, they
only focus on visual content editing and cannot customize the motion of new
content.

Motion is another crucial aspect of video, yet research on video motion
editing remains limited. While some methods explore motion-guided video
generation using trajectory-based motion guidance (e.g., DragNUWA [52],
DragAnything [47], MotionCtrl [43]) and box-based motion guidance (e.g.,
Boximator [40], Peekaboo [19]), they do not support motion editing.
Additionally, other works [51, 29, 56] can transfer motion from one video to
another but cannot modify it as well.

In this paper, our goal is to accurately edit content and motion in specific
areas of a video. We create an easy-to-interact pipeline by setting the
content editing as modifying the first frame, with trajectory lines [52] as
the motion control signal. Other unedited content in all frames should be
maintained in editing results and merged with the editing effect. However, we
find that fusing unedited content with motion-customized new content is
challenging, mainly for two reasons: (1) Training imbalance: Unedited content
is dense and easier to learn, while motion trajectories are sparse and
abstract, making them harder to learn. (2) Condition coupling: Unedited
content provides both visual and inter-frame motion information, leading the
model to rely on it for motion estimation, thereby ignoring the hard-to-learn
trajectory lines.

To address these challenges, we design a three-stage training strategy to
harmonize unedited content and motion-customized new content, enabling
harmonious control of different conditions. Besides, we design a
spatiotemporal adaptive fusion module to fuse these two conditions at
different diffusion sampling steps and spatial locations. Furthermore, our
method can compactly inject motion and content conditions into the diffusion
video generation through a single control module. With these techniques, users
can conveniently edit specific regions in the video by modifying the first
frame and drawing trajectory lines. Notably, ReVideo is not limited to single-
region editing and can customize multiple areas in parallel.

In summary, this work makes the following contributions:

  * •

To the best of our knowledge, this is the first attempt to explore local
editing of both content and motion in videos. Our method can also be easily
extended to multi-area video editing.

  * •

We propose a three-stage training strategy and a spatiotemporal adaptive
fusion module to address the coupling of content and motion control in video
editing, enabling compact control through a single module.

  * •

Extensive experiments demonstrate that ReVideo performs well in several
precise video editing applications, including changing content in a specific
region while keeping motion constant, maintaining content while customizing
new motion trajectories, and modifying both content and motion trajectories.
Some examples are presented in Fig. LABEL:fig:teaser.

##  2 Related Works

###  2.1 Controllable Image and Video Generation

Recent advancements in diffusion models [18, 12] drive the rapid development
of image and video generation. In the community of image generation, some
notable works, such as Stable Diffusion [35], Imagen [36], and DALL-E2 [34],
utilize text as the generation condition. To achieve accurate generation
control, some methods, e.g., ControlNet [53] and T2I-Adapter [31], propose
adding control modules on pre-trained diffusion models. Similarly, initial
efforts in controllable video generation concentrate on the text condition,
such as Video LDM [5], Imagen Video [17], VideoCrafter [8], and AnimateDiff
[16]. Recognizing the limitations of text prompts in capturing complex
scenarios, some recent works [4, 49, 54, 13] leverage image conditions for a
more direct approach. External control modules on pre-trained foundation
models are also popular in controllable video generation. Such as video
ControlNet [9, 55] extends the ControlNet [53] in image generation to video
generation conditioned on a sequence of control signals, like edge maps and
depth maps. In addition to spatial structure control, precise temporal motion
control is also important in controllable video generation. Several recent
works study this topic, such as video generation with trajectory-based motion
guidance (e.g., DragNUWA [52], MotionCtrl [43], Motion-I2V [37], DragAnything
[47]) and generation with box-based motion guidance (e.g., TrailBlazer [28],
Boximator [40], [19]). These methods perform the control by training extra
motion controllers on pre-trained video diffusion models.

###  2.2 Diffusion-based Video Editing

Due to the lack of training data, the common approach in video editing is via
training-free strategies [7, 15, 20, 24, 41, 32] or one-shot tuning [46, 3,
23]. For instance, the prior work Tune-A-Video [46] overfits some diffusion
model parameters to a specific video. Then, it uses the overfitting parameters
to produce the editing result conditioned on the target prompt. To enable a
cohesive global appearance among edited frames, many methods extend the
attention module of Stable Diffusion [35] to encompass multiple frames and
conduct cross-frame attention. For instance, Pix2Video [7] edits the first
frame and performs cross-frame attention of each frame on the first frame to
preserve appearance consistency. TokenFlow [15] and Fairy [45] jointly edit a
few key frames at each denoising step and propagate them throughout the video
based on the nearest-neighbor field extracted from the original video.
Inspired by the initial zero-shot image editing method SDEdit [30], the recent
video foundation model SORA [6] achieves video editing by adding noise to the
input video and then denoising it under the target description. Although these
methods can preserve the general structure of original videos, the information
loss and the lack of consistency constraints on the original video make them
unfit for precise video editing but suitable for global editing like style
transfer.

Another strategy is to train a control module to guide the generation with
some characters that should persist in the editing result, such as depth [14,
26, 48], sketch [42], and optical flow [50]. However, existing methods
primarily focus on preserving spatial structure and are unsuitable for precise
video editing. In the community of precise video editing, some works, such as
InsV2V [10] and the recent EVE [38], edit the video by providing editing
instructions. However, the text-based editing instruction struggles to locate
a target region in some complex scenarios. AnyV2V [25] can edit a video by
editing the first frame. Pika [1] is designed to regenerate a selected area in
a video by text guidance. Unlike these works, we aim to achieve accurate
customization in local areas of a video. The editing target includes locally
modifying content and motion and keeping the unedited content unchanged.

##  3 Method

###  3.1 Preliminaries

Stable Video Diffusion (SVD) [4] is a high-quality and commonly used image-to-
video generation model. To utilize the priors of high-quality video
generation, we employ SVD as the base model and add control modules to achieve
our editing target. Given a reference image $\mathbf{c}_{I}$, SVD will
generate a video frame sequence
$\mathbf{x}=\\{\mathbf{x}^{0},\mathbf{x}^{1},...,\mathbf{x}^{N-1}\\}$ of
length $N$, starting with $\mathbf{c}_{I}$. The sampling of SVD is conducted
on a latent denoising diffusion process [35]. At each denoising step, a
conditional 3D UNet $\Phi_{\theta}$ is used to iteratively denoise this
sequence:

$\hat{\mathbf{z}}_{0}=\Phi_{\theta}(\mathbf{z}_{t},t,\mathbf{c}_{I}),$ (1)

where $\mathbf{z}_{t}$ is the latent representation of $\mathbf{x}_{t}$.
$\hat{\mathbf{z}}_{0}$ is the predication of $\mathbf{z}_{0}$. There are two
conditional injection paths for the reference image $\mathbf{c}_{I}$: (1) It
is embedded into tokens by the CLIP [33] image encoder and injected into the
diffusion model through a cross-attention [35] mechanism; (2) It is encoded
into a latent representation by the VAE encoder of the latent diffusion model,
and concatenated with the latent of each frame in channel dimension. SVD
follows the EDM-preconditioning framework [22], which parameterizes the
learnable denoiser $\Phi_{\theta}$ as:

$\Phi_{\theta}(\mathbf{z}_{t},t,\mathbf{c}_{I};\sigma)=c_{skip}(\sigma)\mathbf{z}_{t}+c_{out}(\sigma)F_{\theta}(c_{in}(\sigma)\mathbf{z}_{t},t,\mathbf{c}_{I};c_{noise}(\sigma)),$
(2)

where $\sigma$ is the noise level, and $F_{\theta}$ is the network to be
trained. $c_{skip}$, $c_{out}$, $c_{in}$, and $c_{noise}$ are preconditioning
hyper-parameters. $\Phi_{\theta}$ is trained via denoising score matching
(DSM):

$\mathbb{E}_{\mathbf{z}_{0},t,\mathbf{n}\sim\mathcal{N}(0,\sigma^{2})}\left[\lambda_{\sigma}||\Phi_{\theta}(\mathbf{z}_{0}+\mathbf{n},t,\mathbf{c}_{I})-\mathbf{z}_{0}||_{2}^{2}\right].$
(3)

Figure 1:  Two potential structures to inject motion and content control.

###  3.2 Task Formulation and Some Insights

Task formulation. The purpose of this paper is to locally edit a video,
including visual information and motion information. In addition, the unedited
content in the video should remain unchanged. Therefore, our conditional video
generation involves three control signals: (1) the edited content, (2) the
content of the unedited area, and (3) the motion condition in the edited area.
We implement content editing by modifying the first frame of the video and
then broadcasting it to subsequent video frames. Here, we denote the edited
first frame as $\mathbf{c}_{ref}\in\mathbb{R}^{3\times W\times H}$. For the
motion condition, we use interaction-friendly trajectory lines [52, 47] as the
control signal. Specifically, the motion condition also contains N maps for a
N-frame video. Each map consists of 2 channels, indicating the movement of the
tracked points in the horizontal and vertical directions relative to the
previous frame. The motion condition in this paper is represented as
$\mathbf{c}_{mot}\in\mathbb{R}^{N\times 2\times W\times H}$. The unedited
content $\mathbf{c}_{con}$ can be conveniently provided by the masked video,
i.e., $\mathbf{c}_{con}=\mathbf{V}\cdot\mathbf{M}$, where
$\mathbf{V}\in\mathbb{R}^{N\times 3\times W\times H}$ and
$\mathbf{M}\in\mathbb{R}^{1\times 1\times W\times H}$ refer to the original
video and the editing region mask, respectively.

Since we adopt SVD as the pre-trained base model, its image-to-video
capability can naturally serve as the import port for the edited first frame.
For unedited content and customized motion trajectories, we train additional
control modules to import them into the generation process.

Figure 2:  The motion control capability of two structures in Fig. 1 with
different training strategies. We visualize trajectory lines in a specific
area (red box) and label the editing area with a black box. Toy experiments
present the coupling issue of customized motion and unedited content.

Trajectory sampling. During training, it is essential to extract trajectories
from videos to provide motion condition $\mathbf{c}_{mot}$. At the beginning
of trajectory sampling, we use a grid [52] to sparsify dense sampling points,
obtaining $N_{init}$ initial points. Among these points, those with larger
motions are beneficial to train trajectory control. To filter out these
points, we first apply motion tracking on each point to obtain their path
lengths, i.e., $\\{l_{0},l_{1},...,l_{N_{init}-1}\\}$. We use the mean of
these lengths as the threshold $l_{Th}$ to extract points whose motion length
is greater than $l_{Th}$. Then, we use the normalized lengths of these points
as sampling probabilities to sample $N$ points randomly. Because the high
sparsity is not conducive for the model to learn from these trajectories, we
apply a Gaussian filter [52] to obtain the smooth trajectory map
$\mathbf{c}_{mot}$. More details are presented in Appendix.

Insights. A naive implementation of our editing target is directly training an
extra control module, like ControlNet [53], to inject motion and content
conditions into the diffusion generation process. We present this design in
structure A of Fig. 1. Specifically, at the input, a content encoder $E_{c}$
and a motion encoder $E_{m}$ embed the content condition $\mathbf{c}_{con}$ of
the unedited area and motion condition $\mathbf{c}_{mot}$ of the editing area.
These two embeddings are merged by direct summing to obtain the fused
condition feature $\mathbf{f}_{c}$. Then, a copy of the UNet encoder extracts
multiscale intermediate features from $\mathbf{f}_{c}$, which are added to the
corresponding layers in the diffusion model. This process is formulated as:

$\mathbf{y}_{c}=\mathcal{F}(\mathbf{z}_{t},t,\mathbf{c}_{ref};\Theta)+\mathcal{Z}(\mathcal{F}(\mathbf{z}_{t}+\mathcal{Z}(\mathbf{f}_{c}),t,\mathbf{c}_{ref};\Theta_{c})),$
(4)

where $\mathbf{y}_{c}$ is the new diffusion features. $\mathcal{Z}$ is the
function of zero-conv [53]. $\Theta$ and $\Theta_{c}$ are the parameters of
the SVD model and extra control module. We conduct several toy experiments
based on this idea, as illustrated in Fig.2. The input video contains a woman
initially moving to the left, followed by a shift to the right. The editing
target is to alter the facial motion towards the right while keeping the other
content unchanged. In the toy experiment  1, we fix SVD and train the control
module with Eq. 3. The result shows that the content condition precisely
controls the unedited area of the generated video. But the motion condition
has no control effect, and the trajectory lines in the editing area (labeled
with a black box) are consistent with the unedited area. A possible reason is
that a single control branch has difficulty handling two control conditions
simultaneously. To verify this hypothesis, we train structure B in Fig. 1 to
handle these two conditions separately. The toy experiment  2 in Fig. 2 shows
that the motion control is still ineffective, suggesting that the problem is
more attributed to the control training rather than the network structure. To
enhance the motion control training, we split the training of structure B into
two stages. In the first stage, we only train the motion control module to
endow it with motion control prior. In the second stage, we train the motion
control and content control together. The result in toy experiment  3 shows
that although the motion prior training produces good motion control
capability, the control accuracy is weakened and affected by the unedited
content after introducing the content control. After these toy experiments, we
have the following insights:

$\diamond$ The condition of unedited content not only contains visual
information but also has rich inter-frame motion information. As a more easily
learned condition, the diffusion model tends to predict the motion of the
editing area through unedited content, ignoring the sparse motion trajectory
control.

$\diamond$ The coupling between motion-customized new content and unedited
content is strong, making it difficult to overcome even using the motion prior
and separate control branches.

$\diamond$ Motion prior training is helpful in decoupling motion-customized
content and unedited content.

###  3.3 Coarse-to-fine Training Strategy

To rectify the ignoring of the motion control, we design a coarse-to-fine
training strategy. In addition, structure B in Fig. 1 has a high computational
cost, and we hope to joint control the unedited content and motion-customized
new content on the concise structure A.

Motion prior training. As discussed above, motion trajectory is a sparse and
difficult-to-learn control signal. Toy experiment  3 in Fig. 2 shows that the
motion prior training can alleviate the coupling between motion-customized
content and unedited content. Hence, in the first stage, we only train the
motion trajectory control, allowing the control module to have good motion
control prior.

Decoupling training. Based on the control module from the first stage, the
training in the second stage aims to add content control of unedited areas.
Toy experiment  3 in Fig. 2 shows that even with good motion control priors,
the precision of motion control still degrades after introducing unedited
content condition. Therefore, we design a training strategy to decouple motion
and content control in this stage. Specifically, we set the editing part and
the unedited part in a training sample $\mathbf{V}$ to be two different
videos, i.e., $\mathbf{V}_{1}$ and $\mathbf{V}_{2}$. As shown in Fig. 3,
$\mathbf{V}_{1}$ and $\mathbf{V}_{2}$ are combined through the editing mask
$\mathbf{M}$, i.e.,
$\mathbf{V}=\mathbf{V}_{1}\cdot\mathbf{M}+\mathbf{V}_{2}\cdot(1-\mathbf{M})$.
Since the editing region and the unedited region come from two different
videos, the motion information of the editing region cannot be predicted
through the unedited content. Therefore, it can decouple content control and
motion control during training.

Figure 3:  The data construction strategy for decoupling training and editing
results from this stage.

Deblocking training. As shown in the right part of Fig. 3, although the
decoupling training achieves joint control of customized motion and unedited
content with high accuracy, it breaks the consistency between the edited and
unedited regions, producing block artifacts in the boundary. To rectify this
issue, we design the third training stage to remove block artifacts. The
training in this stage is initialized with the model from the second stage and
trained on normal video data. To preserve the decoupled motion and content
control prior from the second stage, we only fine-tune the key embedding
$\mathbf{W}_{k}$ and value embedding $\mathbf{W}_{v}$ in temporal self-
attention layers of the control module and SVD model. The toy experiment  4 in
Fig. 2 shows that after the training of this stage, the model removes the
block artifacts and retains joint control of unedited content and motion
customization.

Figure 4:  The architecture of our proposed spatiotemporal adaptive fusion
module (left), and the visualization of fusion weight $\mathbf{\Gamma}$ at
different timesteps (right).

###  3.4 Spatiotemporal Adaptive Fusion Module

Although the coarse-to-fine training strategy achieves decoupling of content
control and motion control, we observe considerable failure cases in some
complex motion trajectories. To further distinguish the control roles of
unedited content and motion trajectories in the generation, we design a
spatiotemporal adaptive fusion module (SAFM) as shown in Fig. 4. Specifically,
SAFM predict a weight map $\mathbf{\Gamma}$ through the editing mask
$\mathbf{M}$ to fuse motion and content control instead of direct summing.
Moreover, because diffusion generation is a multi-step iterative process, the
fusion of control conditions between time steps should have adaptive
adjustment. Therefore, we concatenate timestep $t$ and $\mathbf{M}$ in the
channel dimension to form a spatiotemporal condition to guide the
$\mathbf{\Gamma}$ prediction. Mathematically, the fusion of motion and content
conditions is formulated as follows:

$\mathbf{f}_{c}=E_{c}(\mathbf{c}_{con})\cdot\mathbf{\Gamma}+E_{m}(\mathbf{c}_{mot})\cdot(1-\mathbf{\Gamma}),\
\mathbf{\Gamma}=\mathcal{H}(\mathbf{M},t),$ (5)

where $\mathcal{H}$ is the function of spatiotemporal embedding. $\mathcal{H}$
needs to be jointly trained with $\mathbf{W}_{k}$ and $\mathbf{W}_{v}$ in the
deblocking training stage. We visualize $\mathbf{\Gamma}$ at different time
steps in the right part of Fig. 4. It can be seen that $\mathbf{\Gamma}$
learns the spatial characteristics of the editing area. It assigns a higher
weight to the motion condition in the editing area and a higher weight to the
content condition in the unedited area. In addition, $\mathbf{\Gamma}$ learns
to distinguish different sampling steps $t$ and linearly adjusts with $t$.

##  4 Experiments

###  4.1 Implementation Details

In this work, we choose Stable Video Diffusion (SVD) as the base model. Our
three training stages are completed on the WebVid [2] dataset, which contains
10 million text-video pairs. These three stages are optimized for $40K$,
$30K$, and $20K$ iterations, respectively, with Adam [27] optimizer on 4
NVIDIA A100 GPUs. The batch size for each GPU is set as 4, with the resolution
being $512\times 320$. It takes about 6 days to complete all training stages.
During the training process, we use CoTracker [21] to extract motion
trajectories. In the first training stage, trajectory sampling is performed
throughout the video. In the second and third training stages, a rectangular
editing area is randomly selected in the video with the minimum size being
$64\times 64$, and trajectory sampling is performed within it. The number of
trajectory lines for each training sample is randomly selected between 1 and
10.

Figure 5:  The visual comparison between InsV2V [10], AnyV2V [25], Pika [1], and our ReVideo.  Table 1: Quantitative comparison between our ReVideo and other related works. We employ automatic metrics (i.e., CLIP [33] score, PSNR) and human evaluation to evaluate the performance. Method | Automatic Metrics | Human Evaluation
---|---|---
PSNR $\uparrow$ | Text Alignment $\uparrow$ | Consistency $\uparrow$ | Overall $\uparrow$ | Editing Target $\uparrow$
InsV2V [10] | 29.77 | 0.2022 | 0.9808 | $10.2\%$ | $5.1\%$
AnyV2V [25] | 29.80 | 0.2143 | 0.9836 | $2.8\%$ | $4.0\%$
Pika [1] | 33.07 | 0.2184 | 0.9956 | $27.9\%$ | $23.9\%$
ReVideo | 32.85 | 0.2304 | 0.9864 | $\mathbf{59.1\%}$ | $\mathbf{67.0\%}$

###  4.2 Comparison

Among existing methods, Pika [1] is the most similar to ours. Pika can perform
local video editing by defining an editing area. The difference is that Pika
controls the new content in the editing area by text and has no motion
control. In addition, the recent work AnyV2V [25] proposes editing the first
frame of the video to achieve entire video editing, which has similarities
with our ReVideo. InsV2V [10], using editing instructions to edit the video,
can also maintain unedited content. Therefore, in this paper, we compare our
ReVideo with these three methods. The visual comparison in Fig. 5 shows that
in some fine-grained editing scenarios, such as putting sunglasses on a man,
AnyV2V has a loss of edited content. In addition, the unedited area of InsV2V
and AnyV2V suffers from content distortion. Although Pika can generate smooth
and high-fidelity results, it is difficult to accurately customize new content
by text, especially in adding new objects, e.g., adding a dog on the soccer
field. Adding new objects to the scene is also challenging for InsV2V. Due to
the lack of motion control, AnyV2V and Pika usually produce static motion of
the edited content, such as a car driving on the road. In comparison, our
ReVideo can effectively broadcast the edited content throughout the entire
video while allowing users to customize the motion in editing areas.

In addition to visual comparison, we employ automatic metrics and human
evaluation to measure the performance of different methods. For this task, we
build a test set containing 16 videos, with the resolution being $720\times
1280$. Following previous works [7, 25], automatic metrics employ CLIP score
[33] to measure text alignment and temporal consistency. The text alignment is
obtained by calculating the average CLIP cosine similarity between each frame
and editing description. Temporal consistency is computed by average CLIP
cosine similarity between every pair of consecutive frames. We employ PSNR
[39] to measure the reconstruction quality of unedited content. The human
evaluation considers two aspects, i.e., overall video quality, and whether the
editing target is achieved. We allow 20 volunteers to choose the best method
for each test sample on each aspect. The results in Tab. 1 show that our
ReVideo performs better than InsV2V and AnyV2V in all evaluation terms.
Compared with Pika, our performance is slightly lower in the evaluation of
temporal consistency and the quality of unedited content. Notably, AnyV2V and
Pika usually generate static motion of new content due to the lack of motion
control. Static motion tends to score higher in consistency evaluation,
measured by CLIP similarity of adjacent video frames. Our method has obvious
advantages over Pika in text alignment and human evaluation, reflecting the
significant gap between text-guided local editing and user-specified local
editing. Our ReVideo can precisely specify the appearance and motion of the
editing area, better meeting requirements for accurate customization.

Figure 6:  Ablation study of our ReVideo.

###  4.3 Ablation Study

In our ReVideo, we design the spatiotemporal adaptive fusion module (SAFM) to
help decouple the control of unedited content and motion customization in
diffusion generation. It predicts a fusion weight $\mathbf{\Gamma}$
conditioned on the editing area $\mathbf{M}$ and time step t. Then, the fusion
of content and motion control is achieved through Eq. 5. In this part, we
conduct an ablation study on this fusion mechanism. In addition, we only fine-
tune the key embedding and value embedding of the temporal self-attention
layers in the SVD model and control module in the stage of deblocking
training. In the ablation study, we discuss the impact of tuning parameters in
deblocking training.

The effectiveness of SAFM. To demonstrate the effectiveness of SAFM, we
replace SAFM with direct summing of motion and content control. The results in
Fig. 6 show that the direct summing fusion cannot accurately control the
motion in some complex motion trajectories, e.g., wavy lines. In comparison,
using SAFM can help decouple content and motion control in the editing area,
achieving more accurate trajectory guidance.

The effectiveness of time adaptation in SAFM. We remove the time condition in
the SAFM module, i.e., using the same weight map $\mathbf{\Gamma}$ to fuse
content and motion control in each diffusion sampling step. The results in
Fig. 6 show that not distinguishing $\mathbf{\Gamma}$ in different sampling
steps leads to unsatisfactory artifacts at the boundary of the editing area.

Tuning parameters in deblocking training. Although the training in stages 1
and 2 enables the control module to have good local motion control
capabilities, we find that there is still an ignoring of motion control in the
training of stage 3, i.e., deblocking training. As shown in Fig. 6, the local
motion control capability is degraded after we tune the entire control module
in stage 3. Therefore, we optimize a part of the parameters to maintain the
prior of local motion control. Experiments show that fine-tuning spatial
layers still triggers the ignoring of motion control. In comparison, fine-
tuning key embedding and value embedding of the temporal layer in the control
module and the base model has minimal impact on local motion control
capability. The edited and unedited areas are also harmoniously fused. More
ablations of tuning parameters are presented in Appendix.

##  5 Conclusion

In this paper, we aim to solve the problem of local video editing. The editing
target includes visual content and motion trajectory modifications. To the
best of our knowledge, this is the first attempt at this task. In this new
task, We find a coupling problem between unedited content and motion
customization. Directly training these two control conditions on the video
generation model will cause the ignoring of motion control. To address this
issue, we develop a three-stage training strategy to combine these two
conditions coarse to fine. In addition, we design a spatiotemporal adaptive
fusion module to further decouple unedited content and motion-customized
content in different diffusion sampling steps and spatial locations. Extensive
experiments demonstrate that our ReVideo has promising performance on several
accurate video editing applications, i.e., (1) locally changing video content
while keeping the motion constant, (2) keeping content unchanged and
customizing new motion trajectories, (3) modifying both content and motion
trajectories. Our method can also easily extend these applications to multi-
area editing without specific training.

Limitations. Although our method can regenerate local areas of the video, the
regeneration quality is limited by the base model. In some scenarios where the
generation prior of SVD is not ideal, some unexpected artifacts may occur in
the editing results.

## References

  * [1] https://www.pika.art/.
  * [2] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728–1738, 2021.
  * [3] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel.  Text2live: Text-driven layered image and video editing.  In European conference on computer vision, pages 707–723. Springer, 2022.
  * [4] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al.  Stable video diffusion: Scaling latent video diffusion models to large datasets.  arXiv preprint arXiv:2311.15127, 2023.
  * [5] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis.  Align your latents: High-resolution video synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023.
  * [6] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh.  Video generation models as world simulators.  2024\.
  * [7] Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra.  Pix2video: Video editing using image diffusion.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23206–23217, 2023.
  * [8] Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al.  Videocrafter1: Open diffusion models for high-quality video generation.  arXiv preprint arXiv:2310.19512, 2023.
  * [9] Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin.  Control-a-video: Controllable text-to-video generation with diffusion models.  arXiv preprint arXiv:2305.13840, 2023.
  * [10] Jiaxin Cheng, Tianjun Xiao, and Tong He.  Consistent video-to-video transfer using synthetic dataset.  In The Twelfth International Conference on Learning Representations, 2023.
  * [11] Yuren Cong, Mengmeng Xu, Christian Simon, Shoufa Chen, Jiawei Ren, Yanping Xie, Juan-Manuel Perez-Rua, Bodo Rosenhahn, Tao Xiang, and Sen He.  Flatten: optical flow-guided attention for consistent text-to-video editing.  arXiv preprint arXiv:2310.05922, 2023.
  * [12] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  Advances in neural information processing systems, 34:8780–8794, 2021.
  * [13] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis.  Structure and content-guided video synthesis with diffusion models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7346–7356, 2023.
  * [14] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis.  Structure and content-guided video synthesis with diffusion models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7346–7356, 2023.
  * [15] Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel.  Tokenflow: Consistent diffusion features for consistent video editing.  arXiv preprint arXiv:2307.10373, 2023.
  * [16] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai.  Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.  arXiv preprint arXiv:2307.04725, 2023.
  * [17] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al.  Imagen video: High definition video generation with diffusion models.  arXiv preprint arXiv:2210.02303, 2022.
  * [18] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  * [19] Yash Jain, Anshul Nasery, Vibhav Vineet, and Harkirat Behl.  Peekaboo: Interactive video generation via masked-diffusion.  arXiv preprint arXiv:2312.07509, 2023.
  * [20] Ozgur Kara, Bariscan Kurtkaya, Hidir Yesiltepe, James M Rehg, and Pinar Yanardag.  Rave: Randomized noise shuffling for fast and consistent video editing with diffusion models.  arXiv preprint arXiv:2312.04524, 2023.
  * [21] Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht.  Cotracker: It is better to track together.  arXiv preprint arXiv:2307.07635, 2023.
  * [22] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  Advances in Neural Information Processing Systems, 35:26565–26577, 2022.
  * [23] Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel.  Layered neural atlases for consistent video editing.  ACM Transactions on Graphics (TOG), 40(6):1–12, 2021.
  * [24] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi.  Text2video-zero: Text-to-image diffusion models are zero-shot video generators.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15954–15964, 2023.
  * [25] Max Ku, Cong Wei, Weiming Ren, Huan Yang, and Wenhu Chen.  Anyv2v: A plug-and-play framework for any video-to-video editing tasks.  arXiv preprint arXiv:2403.14468, 2024.
  * [26] Feng Liang, Bichen Wu, Jialiang Wang, Licheng Yu, Kunpeng Li, Yinan Zhao, Ishan Misra, Jia-Bin Huang, Peizhao Zhang, Peter Vajda, et al.  Flowvid: Taming imperfect optical flows for consistent video-to-video synthesis.  arXiv preprint arXiv:2312.17681, 2023.
  * [27] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [28] Wan-Duo Kurt Ma, JP Lewis, and W Bastiaan Kleijn.  Trailblazer: Trajectory control for diffusion-based video generation.  arXiv preprint arXiv:2401.00896, 2023.
  * [29] Joanna Materzynska, Josef Sivic, Eli Shechtman, Antonio Torralba, Richard Zhang, and Bryan Russell.  Customizing motion in text-to-video diffusion models.  arXiv preprint arXiv:2312.04966, 2023.
  * [30] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  arXiv preprint arXiv:2108.01073, 2021.
  * [31] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4296–4304, 2024.
  * [32] Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen.  Fatezero: Fusing attentions for zero-shot text-based video editing.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15932–15942, 2023.
  * [33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763, 2021.
  * [34] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [35] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  * [36] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  Advances in neural information processing systems, 35:36479–36494, 2022.
  * [37] Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al.  Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling.  arXiv preprint arXiv:2401.15977, 2024.
  * [38] Uriel Singer, Amit Zohar, Yuval Kirstain, Shelly Sheynin, Adam Polyak, Devi Parikh, and Yaniv Taigman.  Video editing via factorized diffusion distillation.  arXiv preprint arXiv:2403.09334, 2024.
  * [39] Yule Sun, Ang Lu, and Lu Yu.  Weighted-to-spherically-uniform quality evaluation for omnidirectional video.  IEEE Signal Processing Letters, page 1–1, Jan 2017.
  * [40] Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, and Hang Li.  Boximator: Generating rich and controllable motions for video synthesis.  arXiv preprint arXiv:2402.01566, 2024.
  * [41] Wen Wang, Yan Jiang, Kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, and Chunhua Shen.  Zero-shot video editing using off-the-shelf image diffusion models.  arXiv preprint arXiv:2303.17599, 2023.
  * [42] Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou.  Videocomposer: Compositional video synthesis with motion controllability.  Advances in Neural Information Processing Systems, 36, 2024.
  * [43] Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan.  Motionctrl: A unified and flexible motion controller for video generation.  arXiv preprint arXiv:2312.03641, 2023.
  * [44] Yujie Wei, Shiwei Zhang, Zhiwu Qing, Hangjie Yuan, Zhiheng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, and Hongming Shan.  Dreamvideo: Composing your dream videos with customized subject and motion.  arXiv preprint arXiv:2312.04433, 2023.
  * [45] Bichen Wu, Ching-Yao Chuang, Xiaoyan Wang, Yichen Jia, Kapil Krishnakumar, Tong Xiao, Feng Liang, Licheng Yu, and Peter Vajda.  Fairy: Fast parallelized instruction-guided video-to-video synthesis.  arXiv preprint arXiv:2312.13834, 2023.
  * [46] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7623–7633, 2023.
  * [47] Wejia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang.  Draganything: Motion control for anything using entity representation.  arXiv preprint arXiv:2403.07420, 2024.
  * [48] Jinbo Xing, Menghan Xia, Yuxin Liu, Yuechen Zhang, Y He, H Liu, H Chen, X Cun, X Wang, Y Shan, et al.  Make-your-video: Customized video generation using textual and structural guidance.  IEEE Transactions on Visualization and Computer Graphics, 2024.
  * [49] Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Xintao Wang, Tien-Tsin Wong, and Ying Shan.  Dynamicrafter: Animating open-domain images with video diffusion priors.  arXiv preprint arXiv:2310.12190, 2023.
  * [50] Wilson Yan, Andrew Brown, Pieter Abbeel, Rohit Girdhar, and Samaneh Azadi.  Motion-conditioned image animation for video editing.  arXiv preprint arXiv:2311.18827, 2023.
  * [51] Danah Yatim, Rafail Fridman, Omer Bar Tal, Yoni Kasten, and Tali Dekel.  Space-time diffusion features for zero-shot text-driven motion transfer.  arXiv preprint arXiv:2311.17009, 2023.
  * [52] Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan.  Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory.  arXiv preprint arXiv:2308.08089, 2023.
  * [53] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023.
  * [54] Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou.  I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models.  arXiv preprint arXiv:2311.04145, 2023.
  * [55] Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian.  Controlvideo: Training-free controllable text-to-video generation.  arXiv preprint arXiv:2305.13077, 2023.
  * [56] Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Junhao Zhang, Jiawei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou.  Motiondirector: Motion customization of text-to-video diffusion models.  arXiv preprint arXiv:2310.08465, 2023.
  * [57] Yuyang Zhao, Enze Xie, Lanqing Hong, Zhenguo Li, and Gim Hee Lee.  Make-a-protagonist: Generic video editing with an ensemble of experts.  arXiv preprint arXiv:2305.08850, 2023.

##  Appendix A Appendix

Figure 7:  The trajectory sampling pipeline in ReVideo training.

###  A.1 Details of Trajectory Sampling

As described in our main paper, trajectory sampling in the training process
includes three stages, i.e., sparsifying, threshold filtration, and
probabilistic sampling. We present the visualization of this pipeline in Fig.
7. In sparsifying, we use a grid [52] to sparsify the dense sampling points,
obtaining $N_{init}$ initial points. In threshold filtration, we use the mean
of the tracking length of these $N_{init}$ points as the threshold $l_{Th}$ to
filter out points with large motion. In probabilistic sampling, we use the
normalized lengths of these sampling points as sampling probabilities to
sample $N$ points from them. $N$ is randomly selected from 1 to 10.

Figure 8:  The robustness of our ReVideo for irregular editing areas.

Figure 9:  More editing results of our ReVideo.

###  A.2 Robustness for Irregular Editing Area

In our main paper, we demonstrate the robustness of our method on multi-area
editing without specific training. In Fig. 8, we present another robustness of
our method for irregular editing areas. As can be seen, even though our method
is trained on rectangular editing areas, it has stable content and motion
editing capabilities when facing a hand-drawn irregular editing area.

###  A.3 Details of Content Encoder and Motion Encoder

At the input of our spatiotemporal adaptive fusion module (SAFM), two
encoders, i.e., $E_{c}$ and $E_{m}$, separately encode the content and motion
conditions. $E_{c}$ and $E_{m}$ have the same low-cost structure. This
structure contains three sub-blocks, each consisting of a convolution and a
downsampling operation, mapping the condition map to the same size as the
latent $\mathbf{z}_{t}$.

###  A.4 More Editing Results of ReVideo

In Fig. 9, we present more editing results of our ReVideo, including adding
new objects to the video, modifying the motion trajectory of existing content
in the video, editing existing content while maintaining the motion
trajectory, and multi-region editing. As can be seen, the editing results
achieve the editing goals, and motion control and content control coexist
harmoniously.

Figure 10:  The necessity of fine-tuning key embedding and value embedding in
the base model, i.e., SVD.

###  A.5 Necessity of Fine-tuning the Base Model

In the training process of stage 3, we fine-tune the key embedding
$\mathbf{W}_{k}$ and value embedding $\mathbf{W}_{v}$ of the temporal self-
attention layer in the control module and base model. In Fig. 10, we
demonstrate the necessity of fine-tuning the base model in two scenarios.
Specifically, in some complex scenarios, such as the forest shown in the first
row, not fine-tuning the base model would result in content disjunction, e.g.,
the misaligned tree trunk. The second row shows the case where there is a high
coupling between the unedited content and editing content, such as retaining
the motion of hair and only editing the facial movement. Fine-tuning the base
model can alleviate artifacts brought by the motion conflict between the
highly coupled edited and unedited areas. Therefore, jointly fine-tuning the
base model helps to produce more harmonious editing results.

###  A.6 More Frame Editing

In addition to editing a fixed number of frames based on the base model SVD,
our ReVideo can process more frames. In implementation, we use the sliding
window strategy, where the last frame of the editing result in the previous
window is used as the reference image for the current window. Fig. 11 shows
the editing results of our method on a 9-second video containing 90 frames.
One can see that our ReVideo broadcasts the editing of the first frame into
the 90-frame video while controlling the motion of the new content to be
consistent with the original video. At the same time, we also observe that the
error accumulation affects the editing quality. This is an inherent issue in
long video editing, and a more powerful base model can alleviate this issue.

Figure 11:  The ability of our ReVideo to extend the number of editing frames.
The results demonstrate the performance of our ReVideo in processing a
9-second video containing 90 frames.

Figure 12:  More failure cases of Pika in adding new objects to a video. We
set the text consistency control parameter to the highest level during
testing. The editing target is to add a plane in the sky.

###  A.7 More Discussion of Pika in Adding New Object

In the comparison section of the main paper, we find that Pika [1] has weak
editing capabilities in adding new objects. To eliminate the influence of
randomness, we generate 5 times with random seeds in the case of adding new
objects in Fig. 12, i.e., adding a plane in the sky. We set the strength of
text consistency to the highest level, but all 5 editing results failed. This
indicates the inaccuracy of text as the control signal of local redrawing. In
comparison, editing the first frame and then broadcasting it to the entire
video can accurately specify the content of the editing area.