11institutetext: University of Southern California
22institutetext: Google
33institutetext: Pennsylvania State University
44institutetext: Max Planck Institute for Intelligent Systems

# GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation

Quankai Gao 1122 Qiangeng Xu 22 Zhe Cao 22 Ben Mildenhall 22 Wenchao Ma 33 Le
Chen 44 Danhang Tang 22 Ulrich Neumann 11

###### Abstract

Creating 4D fields of Gaussian Splatting from images or videos is a
challenging task due to its under-constrained nature. While the optimization
can draw photometric reference from the input videos or be regulated by
generative models, directly supervising Gaussian motions remains
underexplored. In this paper, we introduce a novel concept, Gaussian flow,
which connects the dynamics of 3D Gaussians and pixel velocities between
consecutive frames. The Gaussian flow can be efficiently obtained by splatting
Gaussian dynamics into the image space. This differentiable process enables
direct dynamic supervision from optical flow. Our method significantly
benefits 4D dynamic content generation and 4D novel view synthesis with
Gaussian Splatting, especially for contents with rich motions that are hard to
be handled by existing methods. The common color drifting issue that happens
in 4D generation is also resolved with improved Guassian dynamics. Superior
visual quality on extensive experiments demonstrates our method’s
effectiveness. Quantitative and qualitative evaluations show that our method
achieves state-of-the-art results on both tasks of 4D generation and 4D novel
view synthesis.
Project page: https://zerg-overmind.github.io/GaussianFlow.github.io/

###### Keywords:

4D Generation 4D Novel View Synthesis 3D Gaussian Splatting Dynamic Scene
Optical Flow.

†† † Contact for paper details<EMAIL_ADDRESS><EMAIL_ADDRESS>
Figure 1: We propose Gaussian flow, a dense 2D motion flow created by
splatting 3D Gaussian dynamics, which significantly benefits tasks such as 4D
generation and 4D novel view synthesis. (a) Based on monocular videos
generated by Lumiere [3] and Sora [4], our model can generate 4D Gaussian
Splatting fields that represent high-quality appearance, geometry and motions.
(b) For 4D novel view synthesis, the motions in our generated 4D Gaussian
fields are smooth and natural, even in highly dynamic regions where other
existing methods suffer from undesirable artifacts.

##  1 Introduction

4D dynamic content creation from monocular or multi-view videos has garnered
significant attention from academia and industry due to its wide applicability
in virtual reality/augmented reality, digital games, and movie industry.
Studies [19, 39, 36, 37] model 4D scenes by 4D dynamic Neural Radiance Fields
(NeRFs) and optimize them based on input multi-view or monocular videos. Once
optimized, the 4D field can be viewed from novel camera poses at preferred
time steps through volumetric rendering. A more challenging task is generating
360 degree 4D content based on uncalibrated monocular videos or synthetic
videos generated by text-to-video or image-to-video models. Since the
monocular input cannot provide enough multi-view cues and unobserved regions
are not supervised due to occlusions, studies [48, 15, 70] optimizes 4D
dynamic NeRFs by leveraging generative models to create plausible and
temporally consistent 3D structures and appearance. The optimization of 4D
NeRFs requires volumetric rendering which makes the process time-consuming.
And real-time rendering of optimized 4D NeRFs is also hardly achieved without
special designs. A more efficient alternative is to model 4D Radiance Fields
by 4D Gaussian Splatting (GS) [61, 30], which extends 3D Gaussian Splatting
[18] with a temporal dimension. Leveraging the efficient rendering of 3D GS,
the lengthy training time of a 4D Radiance Field can be drastically reduced
[67, 42] and rendering can achieve real-time speed during inference.

The optimization of 4D Gaussian fields takes photometric loss as major
supervision. As a result, the scene dynamics are usually under-constraint.
Similarly to 4D NeRFs [21, 36, 39], the radiance properties and the time-
varying spatial properties (location, scales, and orientations) of Gaussians
are both optimized to reduce the photometric Mean Squared Error (MSE) between
the rendered frames and the input video frames. The ambiguities of appearance,
geometry, and dynamics have been introduced in the process and become
prominent with sparse-view or monocular video input. Per-frame Score
Distillation Sampling (SDS) [53] reduces the appearance-geometry ambiguity to
some extent by involving multi-view supervision in latent domain. However,
both monocular photometric supervision and SDS supervision do not directly
supervise scene dynamics.

To avoid temporal inconsistency brought by fast motions, Consistent4D [15]
leverages a video interpolation block, which imposes a photometric consistency
between the interpolated frame and generated frame, at a cost of involving
more frames as pseudo ground truth for fitting. Similarly, AYG [23] uses text-
to-video diffusion model to balance motion magnitude and temporal consistency
with a pre-set frame rate. 4D NeRF model [21] has proven that optical flows on
reference videos are strong motion cues and can significantly benefit scene
dynamics. However, for 4D GS, connecting 4D Gaussian motions with optical
flows has following two challenges. First, a Gaussian’s motion is in 3D space,
but it is its 2D splat that contributes to rendered pixels. Second, multiple
3D Gaussians might contribute to the same pixel in rendering, and each pixel’s
flow does not equal to any one Gaussian’s motion.

To deal with these challenges, we introduce a novel concept, Gaussian flow,
bridging the dynamics of 3D Gaussians and pixel velocities between consecutive
frames. Specifically, we assume the optical flow of each pixel in image space
is influenced by the Gaussians that cover it. The Gaussian flow of each pixel
is considered to be the weighted sum of these Gaussian motions in 2D. To
obtain the Gaussian flow value on each pixel without losing the speed
advantage of Gaussian Splatting, we splat 3D Gaussian dynamics, including
scaling, rotation, and translation in 3D space, onto the image plane along
with its radiance properties. As the whole process is end-to-end
differentiable, the 3D Gaussian dynamics can be directly supervised by
matching Gaussian flow with optical flow on input video frames. We apply such
flow supervision to both 4D content generation and 4D novel view synthesis to
showcase the benefit of our proposed method, especially for contents with rich
motions that are hard to be handled by existing methods. The flow-guided
Guassian dynamics also resolve the color drifting artifacts that are commonly
observed in 4D Generation. We summarize our contributions as follows:

  * •

We introduce a novel concept, Gaussian flow, that first time bridges the 3D
Gaussian dynamics to resulting pixel velocities. Matching Gaussian flows with
optical flows, 3D Gaussian dynamics can be directly supervised.

  * •

The Gaussian flow can be obtained by splatting Gaussian dynamics into the
image space. Following the tile-based design by original 3D Gaussian
Splatting, we implement the dynamics splatting in CUDA with minimal overhead.
The operation to generate dense Gaussian flow from 3D Gaussian dynamics is
highly efficient and end-to-end differentiable.

  * •

With Gaussian flow to optical flow matching, our model drastically improves
over existing methods, especially on scene sequences of fast motions. Color
drifting is also resolved with our improved Gaussian dynamics.

##  2 Related Works

####  2.0.1 3D Generation.

3D generation has drawn tremendous attention with the progress of various 2D
or 3D-aware diffusion models [26, 43, 47, 27] and large vision models [40, 16,
35]. Thanks to the availability of large-scale multi-view image datasets [8,
68, 9], object-level multi-view cues can be encoded in generative models and
are used for generation purpose. Pioneered by DreamFusion [38] that firstly
proposes Score Distillation Sampling (SDS) loss to lift realistic contents
from 2D to 3D via NeRFs, 3D content creation from text or image input has
flourished. This progress includes approaches based on online optimization
[53, 22, 60, 41] and feedforward methods [13, 24, 25, 62, 59] with different
representations such as NeRFs [32], triplane [6, 7, 12] and 3D Gaussian
Splatting [18]. 3D generation becomes more multi-view consistent by involving
multi-view constraints [47] and 3D-aware diffusion models [26] as SDS
supervision. Not limited to high quality rendering, some works [52, 29] also
explore enhancing the quality of generated 3D geometry by incorporating normal
cues.

####  2.0.2 4D Novel View Synthesis and Reconstruction.

By adding timestamp as an additional variable, recent 4D methods with
different dynamic representations such as dynamic NeRF [36, 37, 20, 57, 19,
54, 11], dynamic triplane [10, 5, 45] and 4D Gaussian Splatting [61, 67] are
proposed to achieve high quality 4D motions and scene contents reconstruction
from either calibrated multi-view or uncalibrated RGB monocular video inputs.
There are also some works [34, 33, 71] reconstruct rigid and non-rigid scene
contents with RGB-D sensors, which help to resolve 3D ambiguities by involving
depth cues. Different from static 3D reconstruction and novel view synthesis,
4D novel view synthesis consisting of both rigid and non-rigid deformations is
notoriously challenging and ill-posed with only RGB monocular inputs. Some
progress [20, 11, 54, 56] involve temporal priors and motion cues (e.g.
optical flow) to better regularize temporal photometric consistency and 4D
motions. One of recent works [57] provides an analytical solution for flow
supervision on deformable NeRF without inverting the backward deformation
function from world coordinate to canonical coordinate. Several works [63, 64,
65, 66] explore object-level mesh recovery from monocular videos with optical
flow.

####  2.0.3 4D Generation.

Similar to 3D generation from text prompts or single images, 4D generation
from text prompts or monocular videos also relies on frame-by-frame multi-view
cues from pre-trained diffusion models. Besides, 4D generation methods yet
always rely on either video diffusion models or video interpolation block to
ensure the temporal consistency. Animate124 [70], 4D-fy [2] and one of the
earliest works [48] use dynamic NeRFs as 4D representations and achieve
temporal consistency with text-to-video diffusion models, which can generate
videos with controlled frame rates. Instead of using dynamic NeRF, Align Your
Gaussians [23] and DreamGaussian4D [42] generate vivid 4D contents with 3D
Gaussian Splatting, but again, relying on text-to-video diffusion model for
free frame rate control. Without the use of text-to-video diffusion models,
Consistent4D [15] achieves coherent 4D generation with an off-the-shelf video
interpolation model [14]. Our method benefits 4D Gaussian representations by
involving flow supervision and without the need of specialized temporal
consistency networks.

##  3 Methodology

Figure 2: Between two consecutive frames, a pixel $x_{t_{1}}$ will be pushed
towards $x_{t_{1}}\rightarrow x_{i,t_{2}}$ by the 2D Gaussian $i$’s motion
$i^{t_{1}}\rightarrow i^{t_{2}}$. We can track $x_{t_{1}}$ in Gaussian $i$ by
normalizing it to canonical Gaussian space as $\hat{x}_{i}$ and unnormalize it
to image space to obtain $x_{i,t_{2}}$. Here, we denote this shift
contribution from Gaussian $i$ as $flow^{G}_{i,t_{1},t_{2}}$. The Gaussian
flow $flow^{G}_{t_{1},t_{2}}(x_{t_{1}})$ on pixel $x_{t_{1}}$ is defined as
the weighted sum of the shift contributions from all Gaussians covering the
pixel ($i$ and $j$ in our example). The weighting factor utilizes alpha
composition weights. The Gaussian flow of the entire image can be obtained
efficiently by splatting 3D Gaussian dynamics and rendering with alpha
composition, which can be implemented similarly to the pipeline of the
original 3D Gaussian Splatting [18].

To better illustrate the relationship between Gaussian motions and
corresponding pixel flow in 2D images, we first recap the rendering process of
3D Gaussian Splatting and then investigate its 4D case.

###  3.1 Preliminary

####  3.1.1 3D Gaussian Splatting.

From a set of initialized 3D Gaussian primitives, 3D Gaussian Splatting aims
to recover the 3D scene by minimizing photometric loss between input $m$
images $\\{I\\}_{m}$ and rendered images $\\{I_{r}\\}_{m}$. For each pixel,
its rendered color $C$ is the weighted sum of multiple Gaussians’ colors
$c_{i}$ in depth order along the ray by point-based $\alpha$-blending as in
Eq. 1,

$C=\sum^{N}_{i=1}T_{i}\alpha_{i}c_{i},$ (1)

with weights specifying as

$\alpha_{i}=o_{i}e^{-\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu}_{i})^{T}\mathbf{\Sigma}_{i}^{-1}(\mathbf{x}-\boldsymbol{\mu}_{i})}\quad\text{and}\quad
T_{i}=\sum^{i-1}_{j=1}(1-\alpha_{i}).$ (2)

where $o_{i}\in[0,1]$, $\boldsymbol{\mu}_{i}\in\mathbb{R}^{3\times 1}$, and
$\mathbf{\Sigma}_{i}\in\mathbb{R}^{3\times 3}$ are the opacity, 3D mean, and
3D covariance matrix of $i$-th Gaussian, respectively. And $\mathbf{x}$ is the
intersection between a pixel ray and $i$-th Gaussian. As shown in Eq. 1, the
relationship between a rendered pixel and 3D Gaussians is not bijective.

####  3.1.2 3D Gaussian Splatting in 4D.

Modeling 4D motions with 3D Gaussian Splatting can be done frame-by-frame via
either directly multi-view fitting [30] or moving 3D Gaussians with a time-
variant deformation field [23, 42] or parameterize 3D Gaussians with time
[67]. While with monocular inputs, Gaussian motions are under-constrained
because different Gaussian motions can lead to the same rendered color, and
thus long-term persistent tracks are lost [30]. Though Local Rigidity Loss
[30, 23] is proposed to reduce global freedom of Gaussian motions, it
sometimes brings severe problems due to poor or challenging initialization and
lack of multi-view supervision. As shown in Fig. 6, 3D Gaussians initialized
with the skull mouth closed are hard to be split when the mouth open with
Local Rigidity Loss.

###  3.2 GaussianFlow

It is challenging to connect 4D Gaussian motions with pixel motions since a
Gaussian’s motion is in 3D space, but its 2D projection (2d Gaussian)
contributes to the rendered pixels. Therefore, to solve this dilemma, we
assume the Gaussians’ rotation factors that are tangential to the image plane
are very small between two consecutive frames. By making this assumption, we
can treat the 2D projections of a 3D Gaussian from two consecutive frames as a
same 2D Gaussian deforming (2D translation, rotation, and scaling) with time.
This assumption makes tracking the Gaussian dynamics on image space much
straight forward and is generally valid for natural scenes.

We consider the full freedom of each Gaussian motion in a 4D field, including
1) scaling, 2) rotation, and 3) translation at each time step. As the time
changes, Gaussians covering the queried pixel at $t=t_{1}$ will move to other
places at $t=t_{2}$, as shown in Fig. 2. To specify new pixel location
$\mathbf{x}_{t_{2}}$ at $t=t_{2}$, we first project all the 3D Gaussians into
2D image plane as 2D Gaussians and calculate their motion’s influence on pixel
shifts.

####  3.2.1 Flow from Single Gaussian.

To track pixel shifts (flow) contributed by Gaussian motions, we let the
relative position of a pixel in a deforming 2D Gaussian stay the same. This
setting makes the probabilities at queried pixel location in Gaussian
coordinate system unchanged at two consecutive time steps. According to Eq. 2,
the unchanged probability will grant the pixel with the same radiance and
opacity contribution from the 2D Gaussian, albeit the 2D Gaussian is deformed.

The pixel shift (flow) is the image space distance of the same pixel at two
time steps. We first calculate the pixel shift influenced by a single 2D
Gaussian that covers the pixel. We can find a pixel $\mathbf{x}$’s location at
$t_{2}$ by normalizing its image location at $t_{1}$ to canonical Gaussian
space and unnormalizing it to image space at $t_{2}$:

1) $normalize$. A pixel $\mathbf{x}_{t_{1}}$ following $i$-th 2D Gaussian
distribution can be written as $\mathbf{x}_{t_{1}}\sim
N(\boldsymbol{\mu}_{i,t_{1}}\mathbf{\Sigma}_{i,t_{1}})$. And in $i$-th
Gaussian coordinate system with 2D mean
$\boldsymbol{\mu}_{i,t_{1}}\in\mathbb{R}^{2\times 1}$ and 2D covariance matrix
$\mathbf{\Sigma}_{i,t_{1}}\in\mathbb{R}^{2\times 2}$. After normalizing the
$i$-th Gaussian into the standard normal distribution, we denote the pixel
location in canonical Gaussian space as

$\hat{\mathbf{x}}_{t_{1}}=\mathbf{\Sigma}^{-1}_{i,t_{1}}(\mathbf{x}_{t_{1}}-\boldsymbol{\mu}_{i,t_{1}}),$
(3)

which follows $\hat{\mathbf{x}}_{t_{1}}\sim N(\mathbf{0},\mathbf{I})$ and
$\mathbf{I}\in\mathbb{R}^{2\times 2}$ is identity matrix.

2) $unnormalize$. When $t=t_{2}$, the new location along with the Gaussian
motion denotes $\mathbf{x}_{i,t_{2}}$ on the image plane.

$\displaystyle\mathbf{x}_{i,t_{2}}$
$\displaystyle=\mathbf{\Sigma}_{i,t_{2}}\hat{\mathbf{x}}_{t_{1}}+\boldsymbol{\mu}_{i,t_{2}},$
(4)

and $\mathbf{x}_{t_{2}}\sim
N(\boldsymbol{\mu}_{i,t_{2}},\mathbf{\Sigma}_{i,t_{2}})$. The pixel shift
(flow) contribution from each Gaussian therefore can be calculated as:

$\displaystyle
flow^{G}_{i,t_{1}t_{2}}=\mathbf{x}_{i,t_{2}}-\mathbf{x}_{t_{1}}$ (5)

####  3.2.2 Flow Composition.

In original 3D Gaussian Splatting, a pixel’s color is the weighted sum of the
2D Gaussians’ radiance contribution. Similarly, we define the Gaussian flow
value at a pixel as the weighted sum of the 2D Gaussians’ contributions to its
pixel shift, following alpha composition. With Eq. 3 and Eq. 4, the Gaussian
flow value at pixel $\mathbf{x}_{t_{1}}$ from $t=t_{t_{1}}$ to $t=t_{t_{2}}$
is

$\displaystyle flow^{G}_{t_{1}t_{2}}$
$\displaystyle=\sum^{K}_{i=1}w_{i}flow^{G}_{i,t_{1}t_{2}}$ (6)
$\displaystyle=\sum^{K}_{i=1}w_{i}(\mathbf{x}_{i,t_{2}}-\mathbf{x}_{t_{1}})$
(7)
$\displaystyle=\sum^{K}_{i=1}w_{i}\left[\mathbf{\Sigma}_{i,t_{2}}\mathbf{\Sigma}^{-1}_{i,t_{1}}(\mathbf{x}_{t_{1}}-\boldsymbol{\mu}_{i,t_{1}})+\boldsymbol{\mu}_{i,t_{2}}-\mathbf{x}_{t_{1}})\right],$
(8)

where $K$ is the number of Gaussians along each camera ray sorted in depth
order and each Gaussian has weight
$w_{i}=\frac{T_{i}\alpha_{i}}{\Sigma_{i}T_{i}\alpha_{i}}$ according to Eq. 1,
but normalized to [0,1] along each pixel ray .

In some cases [23, 17, 69, 31], each Gaussian is assumed to be isotropic, and
its scaling matrix $\mathbf{S}=\sigma\mathbf{I}$, where $\sigma$ is the
scaling factor. And its 3D covariance matrix
$\mathbf{RS}\mathbf{S}^{T}\mathbf{R}^{T}=\sigma^{2}\mathbf{I}$. If the scaling
factor of each Gaussian doesn’t change too much across time,
$\mathbf{\Sigma}_{i,t_{2}}\mathbf{\Sigma}^{-1}_{i,t_{1}}\approx\mathbf{I}$.
Therefore, to pair with this line of work, the formulation of our Gaussian
flow as in Eq. 8 can be simplified as

$\displaystyle flow^{G}_{t_{1}t_{2}}$
$\displaystyle=\sum^{K}_{i=1}w_{i}(\boldsymbol{\mu}_{i,t_{2}}-\boldsymbol{\mu}_{i,t_{1}}).$
(9)

In other words, for isotropic Gaussian fields, Gaussian flow between two
different time steps can be approximated as the weighted sum of individual
translation of 2D Gaussian.

Following either Eq. 8 or Eq. 9, the Gaussian flow can be densely calculated
at each pixel. The flow supervision at pixel $\mathbf{x}_{t_{1}}$ from
$t=t_{1}$ to $t=t_{2}$ can then be specified as

$\displaystyle\mathcal{L}_{flow}=||flow^{o}_{t_{1}t_{2}}(\mathbf{x}_{t_{1}})-flow^{G}_{t_{1}t_{2}}||,$
(10)

where optical flow $flow^{o}_{t_{1}t_{2}}$ can be calculated by off-the-shelf
methods as pseudo ground-truth.

Figure 3: Overview of our 4D content generation pipeline. Our model can take
an uncalibrated monocular video or video generated from an image as the input.
We optimize a 3D Gaussian field by matching the first frame photometrically on
reference view and using a 3D-aware SDS loss [26] to supervise the field on
novel views. Then, we optimize the dynamics of the 3D Gaussians with the same
two losses for each frame. Most importantly, we calculate Gaussian flows on
reference view for each consecutive two time step and match it with pre-
computed optical flow of the input video. The gradients from the flow matching
will propagate back through dynamics splatting and rendering process,
resulting in a 4D Gaussian field with natural and smooth motions.

###  3.3 4D Content Generation

As shown in Fig. 3, 4D content generation with Gaussian representation takes
an uncalibrated monocular video either by real capturing or generating from
text-to-video or image-to-video models as input and output a 4D Gaussian
field. 3D Gaussians are initialized from the first video frame with
photometric supervision between rendered image and input image and a 3D-aware
diffusion model [26] for multi-view SDS supervision. In our method, 3D
Gaussian initialization can be done by One-2-3-45 [25] or DreamGaussian [53].
After initialization, 4D Gaussian field is optimized with per-frame
photometric supervision, per-frame SDS supervision, and our flow supervision
as in Eq. 10. The loss function for 4D Gaussian field optimization can be
written as:

$\displaystyle\mathcal{L}=\mathcal{L}_{photometric}+\lambda_{1}\mathcal{L}_{flow}+\lambda_{2}\mathcal{L}_{sds}+\lambda_{3}\mathcal{L}_{other},$
(11)

where $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ are hyperparameters.
$\mathcal{L}_{other}$ is optional and method-dependent. Though not used in our
method, we leave it for completeness.

###  3.4 4D novel view Synthesis

Unlike 4D content generation that has multi-view object-level prior from
3D-aware diffusion model, 4D novel view synthesis takes only multi-view or
monocular input video frames for photometric supervision without any scene-
level prior. 3D Gaussians are usually initialized by sfm [49, 44] from input
videos. After initialization, 4D Gaussian field is then optimized with per-
frame photometric supervision and our flow supervision. We adopt the 4D
Gaussian Fields from [67]. The loss function for 4D Gaussian field
optimization can be written as:

$\displaystyle\mathcal{L}=\mathcal{L}_{photometric}+\lambda_{1}\mathcal{L}_{flow}+\lambda_{3}\mathcal{L}_{other},$
(12)

##  4 Experiments

In this section, we first provide implementation details of the proposed
method and then valid our method on 4D Gaussian representations with (1) 4D
generation and (2) 4D novel view synthesis. We test on the Consistent4D
Dataset [15] and the Plenoptic Video Datasets [19] for both quantitative and
qualitative evaluation. Our method achieves state-of-the-art results on both
tasks.

###  4.1 Implementation Details

We take $t_{2}$ as the next timestep of $t_{1}$ and calculate optical flow
between every two neighbor frames in all experiments. In our CUDA
implementation of Gaussian dynamics splatting, though the number of Gaussians
$K$ along each pixel ray is usually different, we use $K=20$ to balance speed
and effectiveness. A larger $K$ means more number of Gaussians and their
gradient will be counted through backpropagation. For video frames with size
$H\times W\times 3$, we track the motions of Gaussians between every two
neighbor timesteps $t_{1}$ and $t_{2}$ by maintaining two $H\times W\times K$
tensors to record the indices of top-$K$ Gaussians sorted in depth order,
top-$K$ Gaussians’ rendered weights $w_{i}$ for each pixel and an another
tensor with size $H\times W\times K\times 2$ denotes the distances between
pixel coordinate and 2D Gaussian means
$\mathbf{x}_{t_{1}}-\boldsymbol{\mu}_{i,t_{1}}$, respectively. Besides, 2D
mean $\boldsymbol{\mu}_{i,t_{1}}$ and 2D covariance matrices
$\mathbf{\Sigma}_{i,t_{1}}$ and $\mathbf{\Sigma}_{i,t_{2}}$ of each Gaussian
at different two timesteps are accessible via camera projection [18].

Table 1: Quantitative comparisons between ours and others on Consistent4D
dataset.

Method | Pistol | Guppie | Crocodile | Monster | Skull | Trump | Aurorus | Mean
---|---|---|---|---|---|---|---|---
LPIPS$\downarrow$ | CLIP$\uparrow$ | LPIPS$\downarrow$ | CLIP$\uparrow$ | LPIPS$\downarrow$ | CLIP$\uparrow$ | LPIPS$\downarrow$ | CLIP$\uparrow$ | LPIPS$\downarrow$ | CLIP$\uparrow$ | LPIPS$\downarrow$ | CLIP$\uparrow$ | LPIPS$\downarrow$ | CLIP$\uparrow$ | LPIPS$\downarrow$ | CLIP$\uparrow$
D-NeRF [39] | 0.52 | 0.66 | 0.32 | 0.76 | 0.54 | 0.61 | 0.52 | 0.79 | 0.53 | 0.72 | 0.55 | 0.60 | 0.56 | 0.66 | 0.51 | 0.68
K-planes [10] | 0.40 | 0.74 | 0.29 | 0.75 | 0.19 | 0.75 | 0.47 | 0.73 | 0.41 | 0.72 | 0.51 | 0.66 | 0.37 | 0.67 | 0.38 | 0.72
Consistent4D [15] | 0.10 | 0.90 | 0.12 | 0.90 | 0.12 | 0.82 | 0.18 | 0.90 | 0.17 | 0.88 | 0.23 | 0.85 | 0.17 | 0.85 | 0.16 | 0.87
DG4D [42] | 0.12 | 0.92 | 0.12 | 0.91 | 0.12 | 0.88 | 0.19 | 0.90 | 0.18 | 0.90 | 0.22 | 0.83 | 0.17 | 0.86 | 0.16 | 0.87
Ours | 0.10 | 0.94 | 0.10 | 0.93 | 0.10 | 0.90 | 0.17 | 0.92 | 0.17 | 0.92 | 0.20 | 0.85 | 0.15 | 0.89 | 0.14 | 0.91

Figure 4: Qualitative results on Consistent4D dataset.  Figure 5: Qualitative
comparisons between Consistent4D [15] (Con4D) and ours. As a dynamic NeRF-
based method, Consistent4D shows “bubble like” texture and non-consistent
geometry on novel views.  Figure 6: Qualitative comparisons among
DreamGaussian4D [42], our method without flow loss, our method without flow
loss but with Local Rigidity Loss (Ours-r) and ours.

###  4.2 Dataset

####  4.2.1 Consistent4D Dataset.

This dataset includes 14 synthetic and 12 in-the-wild monocular videos. All
the videos have only one moving object with a white background. 7 of the
synthetic videos are provided with multi-view ground-truth for quantitative
evaluation. Each input monocular video with a static camera is set at an
azimuth angle of 0∘. Ground-truth images include four distinct views at
azimuth angles of -75∘, 15∘, 105∘, and 195∘, respectively, while keeping
elevation, radius, and other camera parameters the same with input camera.

####  4.2.2 Plenoptic Video Dataset.

A high-quality real-world dataset consists of 6 scenes with 30FPS and 2028 ×
2704 resolution. There are 15 to 20 camera views per scene for training and 1
camera view for testing. Though the dataset has multi-view synchronized
cameras, all the viewpoints are mostly limited to the frontal part of scenes.

###  4.3 Results and Analysis

####  4.3.1 4D Generation.

We evaluate and compare DreamGaussian4D [42], which is a recent 4D Gaussian-
based state-of-the-art generative model with open-sourced code, and dynamic
NeRF-based methods in Tab. 1 on Consistent4D dataset with ours. Scores on
individual videos are calculated and averaged over four novel views mentioned
above. Note that flow supervision is effective and helps with 4D generative
Gaussian representation. We showcase our superior qualitative results in Fig.
4. Compared to DreamGaussian4D, our method shows better quality as shown in
Fig. 6 after the same number of training iterations. For the two hard dynamic
scenes shown in Fig. 6, our method benefit from flow supervision and generate
desirable motions, while DG4D shows prominent artifacts on the novel views.
Besides, our method also shows less color drifting compared with dynamic NeRF-
based method Consistent4D in Fig. 5, and our results are more consistent in
terms of texture and geometry.

(a) $Flame$ $Steak$

(b) $Cut$ $Spinach$

Figure 7: Qualitative comparisons on DyNeRF dataset [19]. The left column
shows the novel view rendered images and depth maps of a 4D Gaussian method
[67], which suffers from artifacts in the dynamic regions and can hardly
handle time-variant specular effect on the moving glossy object. The right
column shows the results of the same method while optimized with our flow
supervision during training. We refer to our supplementary material for more
comparisons.

####  4.3.2 4D Novel View Synthesis.

We visualize rendered images and depth maps of a very recent state-of-the-art
4D Gaussian method RT-4DGS [67] with (yellow) and without (red) our flow
supervision in Fig. 7(a) and Fig. 7(b). According to zoom-in comparisons, our
method can consistently model realistic motions and correct structures, even
on glossy objects with specular highlights. These regions are known to be
challenging [55, 28] for most methods, even under adequate multi-view
supervision. Our method can reduce ambiguities in photometric supervision by
involving motion cues and is shown to be consistently effective across frames.
By using an off-the-shelf optical flow algorithm [46], we found that only
1$\%$ to 2$\%$ of image pixels from Plenoptic Video Dataset have optical flow
values larger than one pixel. Since our method benefits 4D Gaussian-based
methods more on the regions with large motions, we report PSNR numbers on both
full scene reconstruction and dynamic regions (optical flow value $>1$) in
Tab. 2. With the proposed flow supervision, our method shows better
performance on all scenes and the gains are more prominent on dynamic regions.
Consequently, our method also achieves state-of-the art results on 4D novel
view synthesis.

Table 2: Quantitative evaluation between ours and other methods on the DyNeRF
dataset [19]. We report PSNR numbers on both full-scene novel view synthesis
and dynamic regions where the ground-truth optical flow value is larger than
one pixel. “Ours” denotes RT-4DGS with the proposed flow supervision.

Method | Coffee Martini | Spinach | Cut Beef | Flame Salmon | Flame Steak | Sear Steak | Mean
---|---|---|---|---|---|---|---
HexPlane [5] | - | 32.04 | 32.55 | 29.47 | 32.08 | 32.39 | 31.70
K-Planes [10] | 29.99 | 32.60 | 31.82 | 30.44 | 32.38 | 32.52 | 31.63
MixVoxels [58] | 29.36 | 31.61 | 31.30 | 29.92 | 31.21 | 31.43 | 30.80
NeRFPlayer [50] | 31.53 | 30.56 | 29.35 | 31.65 | 31.93 | 29.12 | 30.69
HyperReel [1] | 28.37 | 32.30 | 32.92 | 28.26 | 32.20 | 32.57 | 31.10
4DGS [61] | 27.34 | 32.46 | 32.90 | 29.20 | 32.51 | 32.49 | 31.15
RT-4DGS [67] | 28.33 | 32.93 | 33.85 | 29.38 | 34.03 | 33.51 | 32.01
Ours | 28.42 | 33.68 | 34.12 | 29.36 | 34.22 | 34.00 | 32.30
Dynamic Region Only
RT-4DGS [67] | 27.36 | 27.47 | 34.48 | 23.16 | 26.04 | 29.52 | 28.00
Ours | 28.02 | 28.71 | 35.16 | 23.36 | 27.53 | 31.15 | 28.99

Figure 8: Visualization of optical and Gaussian flows on the input view and a
novel view. “Ours (no flow)” denotes our model without flow supervision while
“Ours” is our full model. Note that optical flow values of the background
should be ignored because dense optical flow algorithms calculate
correspondences among background pixels. We calculate optical flow
$flow^{o}_{t_{1}t_{2}}$ on rendered sequences by autoflow [51]. From the
$\\#$1 and the $\\#$4 column, we can see that both rendered sequences on input
view have high-quality optical flow, indicating correct motions and
appearance. Comparing Gaussian flows at the $\\#$2 and the $\\#$5 column, we
can see that the underlining Gaussians will move inconsistently without flow
supervision. It is due to the ambiguity of appearance and motions while only
being optimized by photometric loss on a single input view. Aligning Gaussian
flow to optical flow can drastically improve irregular motions ( $\\#$3
column) and create high-quality dynamic motions ($\\#$6 column) on novel
views.

##  5 Ablation Study

We validate our flow supervision through qualitative comparisons shown in Fig.
6. Compared with Ours (no flow) and Ours, the proposed flow supervision shows
its effectiveness on moving parts. For the skull, 3D Gaussians on the teeth
region initialized at $t=t_{1}$ are very close to each other and are hard to
split apart completely when $t=t_{2}$. Because the gradient of incorrectly
grouped Gaussians is small due to the small photometric MSE on view 0.
Moreover, SDS supervision works on latent domains and cannot provide pixel-
wised supervision. And the problem becomes more severe when involving Local
Rigidity Loss (comparing Ours-r and Ours) because the motions of 3D Gaussians
initialized at $t=t_{1}$ are constrained by their neighbors and the Gaussians
are harder to split apart at $t=t_{1}$. Similarly, for bird, regions
consisting of thin structures such as the bird’s beak cannot be perfectly
maintained across frames without our flow supervision. While originally
utilized in 4D Gaussian fields [30] to maintain the structure consistency
during motion, Local Rigidity Loss as a motion constraint can incorrectly
group Gaussians and is less effective than our flow supervision.

We also visualize optical flow $flow^{o}_{t_{1}t_{2}}$ and Gaussian flow
$flow^{G}_{t_{1}t_{2}}$ with and without our flow supervision in Fig. 8. In
both cases, the optical flow $flow^{o}_{t_{1}t_{2}}$ between rendered images
on the input view are very similar to each other (shown in $\\#$1 and $\\#$ 4
column) and align with ground-truth motion because of direct photometric
supervision on input view. However, comparing optical flows on novel view as
shown in $\\#$3 and $\\#$6, without photometric supervision on novel views,
inconsistent Gaussian motions are witnessed without our flow supervision.
Visualization of Gaussian flow $flow^{G}_{t_{1}t_{2}}$ as in $\\#$2 column
also reveals the inconsistent Gaussian motions. Incorrect Gaussian motion can
still hallucinate correct image frames on input view. However, this motion-
appearance ambiguity can lead to unrealistic motions from novel views (the
non-smooth flow color on moving parts in $\\#$3). While $\\#$5 shows
consistent Gaussian flow, indicating the consistent Gaussian motions with flow
supervision.

##  6 Conclusion and Future Work

We present GaussianFlow, an analytical solution to supervise 3D Gaussian
dynamics including scaling, rotation, and translation with 2D optical flow.
Extensive qualitative and quantitative comparisons demonstrate that our method
is general and beneficial to Gaussian-based representations for both 4D
generation and 4D novel view synthesis with motions. In this paper, we only
consider the short-term flow supervision between every two neighbor frames in
our all experiments. Long-term flow supervision across multiple frames is
expected to be better and smoother, which we leave as future work. Another
promising future direction is to explore view-conditioned flow SDS to
supervise Gaussian flow on novel view in the 4D generation task.

##  7 Acknowledgments

We thank Zhengqi Li and Jianchun Chen for thoughtful and valuable discussions.

## References

  * [1] Attal, B., Huang, J.B., Richardt, C., Zollhoefer, M., Kopf, J., O’Toole, M., Kim, C.: Hyperreel: High-fidelity 6-dof video with ray-conditioned sampling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16610–16620 (2023)
  * [2] Bahmani, S., Skorokhodov, I., Rong, V., Wetzstein, G., Guibas, L., Wonka, P., Tulyakov, S., Park, J.J., Tagliasacchi, A., Lindell, D.B.: 4d-fy: Text-to-4d generation using hybrid score distillation sampling. arXiv preprint arXiv:2311.17984 (2023)
  * [3] Bar-Tal, O., Chefer, H., Tov, O., Herrmann, C., Paiss, R., Zada, S., Ephrat, A., Hur, J., Li, Y., Michaeli, T., et al.: Lumiere: A space-time diffusion model for video generation. arXiv preprint arXiv:2401.12945 (2024)
  * [4] Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y., Jing, L., Schnurr, D., Taylor, J., Luhman, T., Luhman, E., Ng, C., Wang, R., Ramesh, A.: Video generation models as world simulators (2024), https://openai.com/research/video-generation-models-as-world-simulators
  * [5] Cao, A., Johnson, J.: Hexplane: A fast representation for dynamic scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 130–141 (2023)
  * [6] Chan, E.R., Lin, C.Z., Chan, M.A., Nagano, K., Pan, B., De Mello, S., Gallo, O., Guibas, L.J., Tremblay, J., Khamis, S., et al.: Efficient geometry-aware 3d generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16123–16133 (2022)
  * [7] Chen, A., Xu, Z., Geiger, A., Yu, J., Su, H.: Tensorf: Tensorial radiance fields. In: European Conference on Computer Vision. pp. 333–350. Springer (2022)
  * [8] Deitke, M., Schwenk, D., Salvador, J., Weihs, L., Michel, O., VanderBilt, E., Schmidt, L., Ehsani, K., Kembhavi, A., Farhadi, A.: Objaverse: A universe of annotated 3d objects. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13142–13153 (2023)
  * [9] Downs, L., Francis, A., Koenig, N., Kinman, B., Hickman, R., Reymann, K., McHugh, T.B., Vanhoucke, V.: Google scanned objects: A high-quality dataset of 3d scanned household items. In: 2022 International Conference on Robotics and Automation (ICRA). pp. 2553–2560. IEEE (2022)
  * [10] Fridovich-Keil, S., Meanti, G., Warburg, F.R., Recht, B., Kanazawa, A.: K-planes: Explicit radiance fields in space, time, and appearance. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12479–12488 (2023)
  * [11] Gao, C., Saraf, A., Kopf, J., Huang, J.B.: Dynamic view synthesis from dynamic monocular video. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5712–5721 (2021)
  * [12] Gao, Q., Xu, Q., Su, H., Neumann, U., Xu, Z.: Strivec: Sparse tri-vector radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 17569–17579 (2023)
  * [13] Hong, Y., Zhang, K., Gu, J., Bi, S., Zhou, Y., Liu, D., Liu, F., Sunkavalli, K., Bui, T., Tan, H.: Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.04400 (2023)
  * [14] Huang, Z., Zhang, T., Heng, W., Shi, B., Zhou, S.: Real-time intermediate flow estimation for video frame interpolation. In: European Conference on Computer Vision. pp. 624–642. Springer (2022)
  * [15] Jiang, Y., Zhang, L., Gao, J., Hu, W., Yao, Y.: Consistent4d: Consistent 360 $\\{$$\backslash$deg$\\}$ dynamic object generation from monocular video. arXiv preprint arXiv:2311.02848 (2023)
  * [16] Jun, H., Nichol, A.: Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463 (2023)
  * [17] Keetha, N., Karhade, J., Jatavallabhula, K.M., Yang, G., Scherer, S., Ramanan, D., Luiten, J.: Splatam: Splat, track & map 3d gaussians for dense rgb-d slam. arXiv preprint arXiv:2312.02126 (2023)
  * [18] Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics 42(4) (2023)
  * [19] Li, T., Slavcheva, M., Zollhoefer, M., Green, S., Lassner, C., Kim, C., Schmidt, T., Lovegrove, S., Goesele, M., Newcombe, R., et al.: Neural 3d video synthesis from multi-view video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5521–5531 (2022)
  * [20] Li, Z., Niklaus, S., Snavely, N., Wang, O.: Neural scene flow fields for space-time view synthesis of dynamic scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6498–6508 (2021)
  * [21] Li, Z., Wang, Q., Cole, F., Tucker, R., Snavely, N.: Dynibar: Neural dynamic image-based rendering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4273–4284 (2023)
  * [22] Lin, C.H., Gao, J., Tang, L., Takikawa, T., Zeng, X., Huang, X., Kreis, K., Fidler, S., Liu, M.Y., Lin, T.Y.: Magic3d: High-resolution text-to-3d content creation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 300–309 (2023)
  * [23] Ling, H., Kim, S.W., Torralba, A., Fidler, S., Kreis, K.: Align your gaussians: Text-to-4d with dynamic 3d gaussians and composed diffusion models. arXiv preprint arXiv:2312.13763 (2023)
  * [24] Liu, M., Shi, R., Chen, L., Zhang, Z., Xu, C., Wei, X., Chen, H., Zeng, C., Gu, J., Su, H.: One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. arXiv preprint arXiv:2311.07885 (2023)
  * [25] Liu, M., Xu, C., Jin, H., Chen, L., Varma T, M., Xu, Z., Su, H.: One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems 36 (2024)
  * [26] Liu, R., Wu, R., Van Hoorick, B., Tokmakov, P., Zakharov, S., Vondrick, C.: Zero-1-to-3: Zero-shot one image to 3d object. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9298–9309 (2023)
  * [27] Liu, Y., Lin, C., Zeng, Z., Long, X., Liu, L., Komura, T., Wang, W.: Syncdreamer: Generating multiview-consistent images from a single-view image. arXiv preprint arXiv:2309.03453 (2023)
  * [28] Liu, Y., Wang, P., Lin, C., Long, X., Wang, J., Liu, L., Komura, T., Wang, W.: Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images. arXiv preprint arXiv:2305.17398 (2023)
  * [29] Long, X., Guo, Y.C., Lin, C., Liu, Y., Dou, Z., Liu, L., Ma, Y., Zhang, S.H., Habermann, M., Theobalt, C., et al.: Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310.15008 (2023)
  * [30] Luiten, J., Kopanas, G., Leibe, B., Ramanan, D.: Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint arXiv:2308.09713 (2023)
  * [31] Matsuki, H., Murai, R., Kelly, P.H., Davison, A.J.: Gaussian splatting slam. arXiv preprint arXiv:2312.06741 (2023)
  * [32] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM 65(1), 99–106 (2021)
  * [33] Newcombe, R.A., Fox, D., Seitz, S.M.: Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 343–352 (2015)
  * [34] Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J., Kohi, P., Shotton, J., Hodges, S., Fitzgibbon, A.: Kinectfusion: Real-time dense surface mapping and tracking. In: 2011 10th IEEE international symposium on mixed and augmented reality. pp. 127–136. Ieee (2011)
  * [35] Nichol, A., Jun, H., Dhariwal, P., Mishkin, P., Chen, M.: Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751 (2022)
  * [36] Park, K., Sinha, U., Barron, J.T., Bouaziz, S., Goldman, D.B., Seitz, S.M., Martin-Brualla, R.: Nerfies: Deformable neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5865–5874 (2021)
  * [37] Park, K., Sinha, U., Hedman, P., Barron, J.T., Bouaziz, S., Goldman, D.B., Martin-Brualla, R., Seitz, S.M.: Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. arXiv preprint arXiv:2106.13228 (2021)
  * [38] Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 (2022)
  * [39] Pumarola, A., Corona, E., Pons-Moll, G., Moreno-Noguer, F.: D-nerf: Neural radiance fields for dynamic scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10318–10327 (2021)
  * [40] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [41] Raj, A., Kaza, S., Poole, B., Niemeyer, M., Ruiz, N., Mildenhall, B., Zada, S., Aberman, K., Rubinstein, M., Barron, J., et al.: Dreambooth3d: Subject-driven text-to-3d generation. arXiv preprint arXiv:2303.13508 (2023)
  * [42] Ren, J., Pan, L., Tang, J., Zhang, C., Cao, A., Zeng, G., Liu, Z.: Dreamgaussian4d: Generative 4d gaussian splatting. arXiv preprint arXiv:2312.17142 (2023)
  * [43] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
  * [44] Schonberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4104–4113 (2016)
  * [45] Shao, R., Zheng, Z., Tu, H., Liu, B., Zhang, H., Liu, Y.: Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16632–16642 (2023)
  * [46] Shi, X., Huang, Z., Bian, W., Li, D., Zhang, M., Cheung, K.C., See, S., Qin, H., Dai, J., Li, H.: Videoflow: Exploiting temporal cues for multi-frame optical flow estimation. arXiv preprint arXiv:2303.08340 (2023)
  * [47] Shi, Y., Wang, P., Ye, J., Long, M., Li, K., Yang, X.: Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308.16512 (2023)
  * [48] Singer, U., Sheynin, S., Polyak, A., Ashual, O., Makarov, I., Kokkinos, F., Goyal, N., Vedaldi, A., Parikh, D., Johnson, J., et al.: Text-to-4d dynamic scene generation. arXiv preprint arXiv:2301.11280 (2023)
  * [49] Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: exploring photo collections in 3d. In: ACM siggraph 2006 papers, pp. 835–846 (2006)
  * [50] Song, L., Chen, A., Li, Z., Chen, Z., Chen, L., Yuan, J., Xu, Y., Geiger, A.: Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. IEEE Transactions on Visualization and Computer Graphics 29(5), 2732–2742 (2023)
  * [51] Sun, D., Vlasic, D., Herrmann, C., Jampani, V., Krainin, M., Chang, H., Zabih, R., Freeman, W.T., Liu, C.: Autoflow: Learning a better training set for optical flow. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10093–10102 (2021)
  * [52] Sun, J., Zhang, B., Shao, R., Wang, L., Liu, W., Xie, Z., Liu, Y.: Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior. arXiv preprint arXiv:2310.16818 (2023)
  * [53] Tang, J., Ren, J., Zhou, H., Liu, Z., Zeng, G.: Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.16653 (2023)
  * [54] Tretschk, E., Tewari, A., Golyanik, V., Zollhöfer, M., Lassner, C., Theobalt, C.: Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 12959–12970 (2021)
  * [55] Verbin, D., Hedman, P., Mildenhall, B., Zickler, T., Barron, J.T., Srinivasan, P.P.: Ref-nerf: Structured view-dependent appearance for neural radiance fields. in 2022 ieee. In: CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5481–5490 (2022)
  * [56] Wang, C., Eckart, B., Lucey, S., Gallo, O.: Neural trajectory fields for dynamic novel view synthesis. arXiv preprint arXiv:2105.05994 (2021)
  * [57] Wang, C., MacDonald, L.E., Jeni, L.A., Lucey, S.: Flow supervision for deformable nerf. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21128–21137 (2023)
  * [58] Wang, F., Tan, S., Li, X., Tian, Z., Song, Y., Liu, H.: Mixed neural voxels for fast multi-view video synthesis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 19706–19716 (2023)
  * [59] Wang, P., Tan, H., Bi, S., Xu, Y., Luan, F., Sunkavalli, K., Wang, W., Xu, Z., Zhang, K.: Pf-lrm: Pose-free large reconstruction model for joint pose and shape prediction. arXiv preprint arXiv:2311.12024 (2023)
  * [60] Wang, Z., Lu, C., Wang, Y., Bao, F., Li, C., Su, H., Zhu, J.: Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems 36 (2024)
  * [61] Wu, G., Yi, T., Fang, J., Xie, L., Zhang, X., Wei, W., Liu, W., Tian, Q., Wang, X.: 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528 (2023)
  * [62] Xu, Y., Tan, H., Luan, F., Bi, S., Wang, P., Li, J., Shi, Z., Sunkavalli, K., Wetzstein, G., Xu, Z., et al.: Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. arXiv preprint arXiv:2311.09217 (2023)
  * [63] Yang, G., Sun, D., Jampani, V., Vlasic, D., Cole, F., Chang, H., Ramanan, D., Freeman, W.T., Liu, C.: Lasr: Learning articulated shape reconstruction from a monocular video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15980–15989 (2021)
  * [64] Yang, G., Sun, D., Jampani, V., Vlasic, D., Cole, F., Liu, C., Ramanan, D.: Viser: Video-specific surface embeddings for articulated 3d shape reconstruction. Advances in Neural Information Processing Systems 34, 19326–19338 (2021)
  * [65] Yang, G., Wang, C., Reddy, N.D., Ramanan, D.: Reconstructing animatable categories from videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16995–17005 (2023)
  * [66] Yang, G., Yang, S., Zhang, J.Z., Manchester, Z., Ramanan, D.: Ppr: Physically plausible reconstruction from monocular videos. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3914–3924 (2023)
  * [67] Yang, Z., Yang, H., Pan, Z., Zhu, X., Zhang, L.: Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. arXiv preprint arXiv:2310.10642 (2023)
  * [68] Yu, X., Xu, M., Zhang, Y., Liu, H., Ye, C., Wu, Y., Yan, Z., Zhu, C., Xiong, Z., Liang, T., et al.: Mvimgnet: A large-scale dataset of multi-view images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9150–9161 (2023)
  * [69] Yugay, V., Li, Y., Gevers, T., Oswald, M.R.: Gaussian-slam: Photo-realistic dense slam with gaussian splatting. arXiv preprint arXiv:2312.10070 (2023)
  * [70] Zhao, Y., Yan, Z., Xie, E., Hong, L., Li, Z., Lee, G.H.: Animate124: Animating one image to 4d dynamic scene. arXiv preprint arXiv:2311.14603 (2023)
  * [71] Zollhöfer, M., Nießner, M., Izadi, S., Rehmann, C., Zach, C., Fisher, M., Wu, C., Fitzgibbon, A., Loop, C., Theobalt, C., et al.: Real-time non-rigid reconstruction using an rgb-d camera. ACM Transactions on Graphics (ToG) 33(4), 1–12 (2014)

Appendix

##  Appendix 0.A Additional Implementation Details

A detailed pseudo code for our flow supervision can be found at Algorithm
$\color[rgb]{1,0,0}{1}$. We extract the projected Gaussian dynamics and obtain
the final Gaussian flow by rendering these dynamics. Variables including the
weights and top-$K$ indices of Gaussians per pixel (as mentioned in
implementation details of our main paper) are calculated in CUDA by modifying
the original CUDA kernel codes of 3D Gaussian Splatting [18]. And Gaussian
flow $flow^{G}$ is calculated by Eq.8 with PyTorch.

In our 4D generation experiment, we run 500 iterations static optimization to
initialize 3D Gaussian fields with a batch size of 16. The Tmax in SDS is
linearly decayed from 0.98 to 0.02. For dynamic representation, we run 600
iterations with batch size of 4 for both DG4D [42] and ours. The flow loss
weight $\lambda_{1}$ in Eq. 11 of our main paper is $1.0$.

In our 4D novel view synthesis experiment, we follow RT-4DGS[67] except that
we add our proposed flow supervision for all cameras. The flow loss weight
$\lambda_{1}$ in Eq. 11 of our main paper is $0.5$.

Input:

$flow^{o}_{t_{k},t_{k+1}}$ : Pseudo ground-truth optical flow from off-the-
shelf optical flow algorithm;

$I^{gt}_{t_{k}}$: ground-truth images , where $k=0,1,...,T$;

$renderer$: A Gaussian renderer;

$Gaussians_{t_{k}}$, $Gaussians_{t_{k+1}}$ : $n$ Gaussians with learnable
parameters at $t_{k}$ and $t_{k+1}$;

$cam_{t_{k}}$ and $cam_{t_{k+1}}$: Camera parameters at $t_{k}$ and $t_{k+1}$;

# Loss init

$\mathcal{L}$ = 0

for  _timestep $k\leq T-1$_ do

// renderer outputs at $t_{k}$

$renderer_{t_{k}}=renderer(Gaussians_{t_{k}},cam_{t_{k}})$;

$I^{render}_{t_{k}}=renderer_{t_{k}}\left[``image"\right]$; # $H\times W\times
3$

$idx_{t_{k}}=renderer_{t_{k}}\left[``index"\right]$; # $H\times W\times K$,
Gaussian indices that cover each pixels

$w_{t_{k}}=renderer_{t_{k}}\left[``weights"\right]$; # $H\times W\times K$

$w_{t_{k}}=w_{t_{k}}/sum(w_{t_{k}},dim=-1)$; # $H\times W\times K$, weight
normalization

$x\\_\mu_{t_{k}}=renderer_{t_{k}}\left[``x\\_mu"\right]$; # $H\times W\times
K\times 2,denotes\quad x_{t_{k}}-\mu_{t_{k}}$

$\mu_{t_{k}}=renderer_{t_{k}}\left[``2D\\_mean"\right]$; # $n\times 2$

$\Sigma_{t_{k}}=renderer_{t_{k}}\left[``2D\\_cov"\right]$; # $n\times 2\times
2$

# renderer outputs at $t_{k+1}$

$renderer_{t_{k+1}}=renderer(Gaussians_{t_{k+1}},cam_{t_{k+1}})$;

$\mu_{t_{k+1}}=renderer_{t_{k+1}}\left[``2D\\_mean"\right]$; # $n\times 2$

$\Sigma_{t_{k+1}}=renderer_{t_{k+1}}\left[``2D\\_cov"\right]$; # $n\times
2\times 2$

# Eq.8 while ignoring resize operations for simplicity

$flow^{G}_{t_{k},t_{k+1}}=w_{t_{k}}*\left(\Sigma_{t_{k+1}}[idx_{t_{k}}]*inv(\Sigma_{t_{k}})[idx_{t_{k}}]*x\\_\mu_{t_{k}}+(\mu_{t_{k+1}}[idx_{t_{k}}]-\mu_{t_{k}}[idx_{t_{k}}]-x\\_\mu_{t_{k}})\right)$

# Eq.10

$\mathcal{L}_{flow}=norm(flow^{o}_{t_{k},t_{k+1}},sum(flow^{G}_{t_{k},t_{k+1}},dim=0))$

# (1) Loss for 4D novel view synthesis

$\mathcal{L}=\mathcal{L}+\mathcal{L}_{photometric}(I^{render}_{t_{k}},I^{gt}_{t_{k}})+\lambda_{1}\mathcal{L}_{flow}+\lambda_{3}\mathcal{L}_{other}$

# (2) Loss for 4D generation

$\mathcal{L}=\mathcal{L}+\mathcal{L}_{photometric}(I^{render}_{t_{k}},I^{gt}_{t_{k}})+\lambda_{1}\mathcal{L}_{flow}+\lambda_{2}\mathcal{L}_{sds}+\lambda_{3}\mathcal{L}_{other}$

end for

Algorithm 1 Detailed pseudo code for GaussianFlow

##  Appendix 0.B More Results

###  0.B.1 More Gaussian Flow in 4D Generation.

More comparisons between Gaussian flow $flow^{G}$ and optical flow $flow^{o}$
on rendered images are shown in Fig. 9. The first row of each example is the
rgb frames rendered from a optimized 4D Gaussian field. We rotate our cameras
for each time steps so that the object can move as optimized and the camera is
moving at the same time to show the scene from different angles. The second
row of each example shows the visualized Gaussian flows. These Gaussian flows
are calculated by the rendered images of consecutive time steps at each camera
view, therefore, containing no camera motion in the flow values. The third row
is the estimated optical flows between the rendered images of consecutive time
steps at each camera view. We use off-the-shelf AutoFlow [51] for the
estimation. We can see that enhanced by the flow supervision from the single
input view, our 4D generation pipeline can model fast motion such as the
explosive motion of the gun hammer (see the last example in Fig. 9).

###  0.B.2 More Results on the DyNeRF Dataset.

More qualitative results on DyNeRF dataset [19] can be found in Fig. 10 and
our video.

Figure 9: Visualization of Gaussian flow $flow^{G}$ and optical flow
$flow^{o}$ on rendered sequences from different views.

(a) $Sear$ $Steak$

(b) $Cut$ $Beef$

Figure 10: Qualitative comparisons on DyNeRF dataset [19]. The left column
shows the novel view rendered images and depth maps of a 4D Gaussian method
[67]. While The right column shows the results of the same method while
optimized with our flow supervision during training.  Figure 11: $Flame$
$Salmon$ Figure 12: Qualitative comparisons on DyNeRF dataset [19]. Since the
details of depth maps on $Flame$ $Salmon$ are hard to be recognized, we only
compare the rendered images. The left column shows the novel view rendered
images of a 4D Gaussian method [67]. While The right column shows the results
of the same method while optimized with our flow supervision during training.