# Diffusion Priors for Dynamic View Synthesis from Monocular Videos

Chaoyang Wang1 Peiye Zhuang1 Aliaksandr Siarohin1 Junli Cao1
Guocheng Qian1,2 Hsin-Ying Lee1 Sergey Tulyakov1
1Snap Research 2KAUST
{cwang9, pzhuang, asiarohin, jcao2, gqian, hlee5<EMAIL_ADDRESS>
###### Abstract

Dynamic novel view synthesis aims to capture the temporal evolution of visual
content within videos. Existing methods struggle to distinguishing between
motion and structure, particularly in scenarios where camera poses are either
unknown or constrained compared to object motion. Furthermore, with
information solely from reference images, it is extremely challenging to
hallucinate unseen regions that are occluded or partially observed in the
given videos. To address these issues, we first finetune a pretrained RGB-D
diffusion model on the video frames using a customization technique.
Subsequently, we distill the knowledge from the finetuned model to a 4D
representations encompassing both dynamic and static Neural Radiance Fields
(NeRF) components. The proposed pipeline achieves geometric consistency while
preserving the scene identity. We perform thorough experiments to evaluate the
efficacy of the proposed method qualitatively and quantitatively. Our results
demonstrate the robustness and utility of our approach in challenging cases,
further advancing dynamic novel view synthesis. Our project website is at
https://mightychaos.github.io/dpdy_proj/.

Figure 1: Comparison with dynamic novel view synthesis methods from monocular
videos. When dealing with self-occlusions, out-of-view details, and complex
motions previous methods render severe artifacts (columns 2-4). In contrast,
our novel approach based on diffusion prior elegantly handles such cases,
producing high quality visual results and geometry (right two columns).

##  1 Introduction

The novel view synthesis of dynamic scenes from monocular casual videos holds
significant importance in various domains due to its potential impact on
understanding and interacting with the real world. While existing methods
approach this challenge through the utilization of hand-crafted geometric and
physics priors [30, 31] or by leveraging monocular depth estimation [18, 23],
recent analyses [8] underscore the limitations of both paradigms. Hand-crafted
geometric and physics priors prove insufficient in disambiguating motion and
structure, particularly in in-the-wild scenarios where the camera motion is
smaller than the object motion, and methods relying on monocular depth
estimation tends to produce paper-thin foreground objects and do not provide
effective supervision for occluded regions, leading to severe artifacts when
the dynamic object is close to the camera.

A critical challenge in dynamic novel view synthesis is to hallucinate regions
unseen in videos, where existing methods struggle when relying solely on
information from reference views. There common scenarios contribute to this
challenge. First, regions behind visible surfaces in the reference views
cannot be recovered in novel views. Second, some parts of objects are entirely
out of view in reference images. Third, without sufficient information from
enough camera poses, some objects cannot be realistically reconstructed. We
demonstrate these challenges in Fig. 1.

To address the need for information beyond reference images in the given
video, leveraging prior knowledge from by pretrained models emerges as a
potential solution. Recently advancements in 3D reconstruction from a single
image, facing similar challenges, have witnessed a great progress by
distilling knowledge from large-scale 2D text-to-image diffusion models as a
2D prior to help synthesize unseen regions [13, 32, 21, 46, 5, 56, 49]. More
recently, the 3D consistency of the reconstructed objects has been further
improved with the help of multi-view diffusion models [43, 22], finetuned on
3D object data. Despite sharing similar challenges, these techniques are not
directly applicable to dynamic novel view synthesis. First, the multi-view
models are trained on object-centric and static data, and cannot handle scenes
that are complex and dynamic. Second, a domain gap exists between the training
images of these diffusion models and the real-world in-the-wild images,
hindering direct knowledge distillation while maintaining consistency.

In response to these challenges, we propose DpDy, an effective dynamic novel
view synthesis pipeline leveraging geometry priors from pretrained diffusion
models with customization techniques. First, we represent a 4D scene with a
dynamic NeRF for dynamic motions and a rigid NeRF for static regions. To
achieve geometry consistency, we integrate knowledge distillation [32, 56]
from a pretrained RGB-D image diffusion model [41] in addition to the
conventional reconstruction objective. Moreover, to preserve the scene
identity and to mitigate the domain gap, we finetune the RGB-D diffusion model
using video frames with customization techniques [38].

We conduct extensive qualitative and quantitative experiments on the
challenging iPhone dataset [9], featuring diverse and complex motions. We
evaluate the quality of the 4D reconstruction using masked Learned Perceptual
Image Patch Similarity (LPIPS) [55] and masked Structural Similarity Index
(SSIM) scores. DpDy performs favorably against all baseline methods. However,
we found that the standard metrics do not adequately reflect the quality of
the rendered novel views. Hence, we performed a series of user studies against
previous works. The human annotators almost unanimously selected our method in
almost all comparisons, supporting the benefits of using 2D diffusion priors
for dynamic novel view supervision.

##  2 Related Works

Dynamic View Synthesis from Monocular Videos involves learning a 4D
representation from a casual monocular video. Previous works [18, 31, 23, 52,
44, 7, 6, 2, 53] typically employ a dynamic Neural Radiance Field (D-NeRF)
[27] as a 4D representation that encodes spatio-temporal scene contents. These
approaches use hand-crafted geometric and physics priors to learn the 4D
representations. For instance, flow-based methods like NSFF [18, 23] utilize a
scene flow field warping loss to enforce temporal consistency. Canonical-based
methods [30, 31, 33, 45, 15], represent a scene using a deformation field that
maps each local observation to a canonical scene space. Most of these methods
are limited to object-centric scenes with controlled camera poses. More
recently, DyniBaR [20] extends the multi-view conditioned NeRF approach, i.e.,
IBRNet [48], to allow dynamic novel view synthesis with a freely chosen input
camera trajectory. Another practical obstacle for applying dynamic NeRFs to
real-world videos is the robust estimation of camera poses when videos contain
a large portion of moving objects. Recent works, such as RoDynRF [23], propose
a joint optimization of dynamic NeRF and camera poses, demonstrating
practicality in real-world applications.

Despite showing promising results, we note that hand-crafted priors such as
deformation and temporal smoothness, utilized by prior works, are insufficient
for reconstructing complex 4D scenes from monocular videos. This is due to the
complexity of resolving ambiguities between motion and structure, as well as
hallucinating unseen or occluded regions. This challenge is particularly
pronounced with slow-motion cameras or in scenarios involving complex dynamic
motions, such as those found in the DyCheck dataset [8]. To overcome these
limitations, we integrate large-scale image diffusion priors to effectively
hallucinate the unseen regions within a scene and regularize 4D
reconstruction.

Figure 2: Overview of our method. To perform dynamic novel view synthesis
given a video, we adopt a 4D representation consisting of dynamic and static
parts. We use two types of supervision. First, we render the input viewpoints
at input time. Besides, we distill prior knowledge of a pre-trained RGB-D
diffusion model on random novel views using score distillation sampling.
Furthermore, to mitigate the domain gaps between the training distributions
and in-the-wild images, we tune the RGB-D diffusion model using the reference
images with a customization technique prior to distillation.

Text-to-Image Diffusion Priors refer to large-scale text-to-image diffusion-
based generative models [39, 37]. These models provide large-scale 2D image
priors that can benefit 3D and 4D generation tasks which struggle with data
limitation issues. For instance, recent text-to-3D generation works [13, 11,
32, 42, 21, 46, 5, 56, 49, 4] have successfully achieved high-quality 3D asset
generation by using image guidance from 2D image priors to the 3D domain. To
achieve this, a Score Distillation Sampling (SDS) approach [32] is introduced,
where noise is added to an image rendered from the 3D representation and
subsequently denoised by a pre-trained text-to-image generative model. SDS
minimizes the Kullback–Leibler (KL) divergence between a prior Gaussian noise
distribution and the estimated noise distribution. Additionally, image-to-3D
generation works [34, 22, 26, 24] also use text-to-image diffusion priors.
Differently, these works have additional requirements where the image identity
should be kept. For this purpose, Dreambooth [38] is proposed to fine-tune the
UNet and text encoders in a text-to-image diffusion model based on the given
image. Dreambooth fine-tuning enables the diffusion prior to memorize the
given image identity, thus providing meaningful guidance for image-based 3D
reconstruction.

While these 3D generation approaches based on text-to-image diffusion priors
have experienced rapid development, their focus has primarily been on
generating 3D assets rather than real-world scenes or videos. Inspired by
this, we extend their application to the in-the-wild 4D scene reconstruction
task and incorporate a pre-trained RGB-D diffusion model, LDM3D [41], as an
RGB-D prior to hallucinate unseen views of 4D scenes.

##  3 Method

We aim to achieve 4D dynamic novel view synthesis from monocular videos. To
achieve this, we propose our method as illustrated in Fig. 2. Specifically, we
represent a 4D dynamic scene using two separate NeRFs [28]: one for rigid
regions and another for dynamic regions of a scene. The rendering of images
involves blending the output from these two NeRF fields. To optimize the NeRF
representations, we apply reconstruction losses for images and depth maps to
minimize the difference between rendered images and the reference video frames
(Sec. 3.1). Additionally, we use an SDS loss in the joint RGB and depth
(a.k.a. RGB-D) space to provide guidance for novel dynamic view synthesis
(Sec. 3.2). Formally, we define the loss function $\mathcal{L}$ as:

$\mathcal{L}=\lambda_{\text{rgb}}\mathcal{L}_{\text{rgb}}+\lambda_{\text{depth}}\mathcal{L}_{\text{depth}}+\lambda_{\text{reg}}\mathcal{L}_{\text{reg}}+\lambda_{\text{sds}}\mathcal{L}_{\text{sds}}.$
(1)

Here, $\mathcal{L}_{\text{rgb}}$ denotes the image-space reconstruction loss
on seen video frames. $\mathcal{L}_{\text{depth}}$ represents an affine-
invariant depth reconstruction loss using a pre-trained depth prediction model
[35]. Additionally, we incorporate a regularization loss
$\mathcal{L}_{\text{reg}}$ to regularize the 4D representation. Finally,
$\mathcal{L}_{\text{sds}}$ is an SDS loss for novel dynamic views in RGB-D
space. We present our technical details in the following.

###  3.1 4D Representation

We represent a 4D scene with two separate NeRFs: the static NeRF focuses on
representing the static regions and the the dynamic NeRF aims to capture the
dynamic motions of the scene. Formally,
$\bm{c}_{s},\sigma_{s},\bm{c}_{d},\sigma_{d}$ denotes the color and density of
a point on a ray from the static and the dynamic NeRF, respectively. Our
method is not specific to the exact implementation of NeRFs. The details of
our implementation is provided in Sec. 4.1.

Given a camera pose and a timestep, we obtain an image from the NeRFs via
differentiable rendering. The color of an image pixel along a NeRF ray $r$,
denoted as $C(r)$, is volumetrically rendered from the blended radiance field
of the static and dynamic NeRFs. The rendering equation is formally written as

$C(r)=\int_{t_{n}}^{t_{f}}T(t)\left[\sigma_{s}(t)\bm{c}_{s}(t)+\sigma_{d}(t)\bm{c}_{d}(t)\right]dt,$
(2)

where $T(t)=\exp(-\int_{t_{n}}^{t}(\sigma_{s}(s)+\sigma_{d}(s))ds)$ is the
accumulated transmittance, and $t_{n}$ and $t_{f}$ are the near and far bounds
of a ray. We derived the following numerical estimator for the continuous
integral in Eq. 7:

$\displaystyle
C(r)=\sum_{i=1}^{N}T_{i}(1-\exp(-(\sigma_{s_{i}}+\sigma_{d_{i}})\delta_{i})\bm{c}_{i},$
(3) $\displaystyle\text{where}\quad
T_{i}=\exp\left(-\sum_{j=1}^{i-1}(\sigma_{s_{j}}+\sigma_{d_{j}})\delta_{j}\right),$
$\displaystyle\bm{c}_{i}=\frac{\sigma_{s_{i}}\bm{c}_{s_{i}}+\sigma_{d_{i}}\bm{c}_{d_{i}}}{\sigma_{s_{i}}+\sigma_{d_{i}}}\quad\text{and}\quad\delta_{i}=t_{i+1}-t_{i}.$

This discretized rendering equation differs from prior discretization
approaches [25, 18] that need separate accumulations of the static and dynamic
components. Eq. 8 is computationally more efficient as it can be implemented
by calling upon the standard NeRF volumetric rendering just once. Please refer
to the supplementary material for mathematical proof.

Reconstruction Losses. We render images and the corresponding depth map from
the NeRFs. Subsequently, we calculate an image reconstruction loss
$\mathcal{L}_{\text{rgb}}$ using the $L_{1}$ norm and an affine-invariant
depth loss $\mathcal{L}_{\text{depth}}$ by comparing the rendered depth map
with a pre-computed depth map obtained from the off-the-shelf monocular depth
estimator MiDAS [36]. It is worth noting that the depth estimation from MiDAS
is both noisy and non-metric, and lacking temporal consistency. As a result,
we only incorporate it during the initial training process and progressively
reduce the weight of the depth loss $\mathcal{L}_{\text{depth}}$ over
training. We mention that the two reconstruction losses are only applied on
existing views that can be seen from the videos. In Sec. 3.2, we introduce
depth guidance for unseen views.

Regularization. We apply additional regularization to the 4D NeRF
representation. The regularization loss $\mathcal{L}_{\text{reg}}$ consists of
two parts which we present in Eq. 4\- 5, respectively. _First_ , to promote
concentration of radiance near the visible surface, we employ the z-variance
loss [56], penalizing the weighted sum of square distances from points on the
ray to the mean depth, _i.e_.,

$\sum_{i}(z_{i}-\mu_{z})^{2}\frac{w_{i}}{\sum_{j}w_{j}},\quad\text{where}\quad\mu_{z}=\sum_{i}\frac{w_{i}z_{i}}{\sum_{j}w_{j}},$
(4)

where $z_{i}$ is the depth for each sampled points along the ray, $\mu_{z}$ is
rendered depth and $w_{i}$ is the normalized volumetric rendering weight.

_Second_ , to encourage proper decomposition of dynamic foreground and static
background, we penalize the skewed entropy of the foreground-background
density ratio $\frac{\sigma_{d}}{\sigma_{d}+\sigma_{s}}$, as proposed by
D2-NeRF [51]. Specifically, the loss is written as:

$H((\frac{\sigma_{d}}{\sigma_{d}+\sigma_{s}})^{k}),$ (5)

where $H(x)=-(x\log(x)+(1-x)\log(1-x))$ is a binary entropy loss. The skew
parameter $k$ is set to 2, promoting separation biased towards increasing
background regions.

Table 1:  Novel view synthesis results. We compare the mLPIPS and mSSIM scores
with existing methods on the iPhone dataset [9].

mLPIPS $\downarrow$ / mSSIM $\uparrow$ | Apple | Block | Paper-windmill | Space-out | Spin | Teddy | Wheel | Macro-average
---|---|---|---|---|---|---|---|---
T-NeRF + Lidar [9] | 0.508 / 0.728 | 0.346 / 0.669 | 0.258 / 0.367 | 0.377 / 0.591 | 0.443 / 0.567 | 0.429 / 0.570 | 0.292 / 0.548 | 0.379 / 0.577
NSFF + Lidar [19] | 0.478 / 0.750 | 0.389 / 0.639 | 0.211 / 0.378 | 0.303 / 0.622 | 0.309 / 0.585 | 0.372 / 0.557 | 0.310 / 0.458 | 0.339 / 0.569
T-NeRF [9] | 0.581 / 0.712 | 0.441/ 0.629 | 0.444 / 0.302 | 0.408 / 0.593 | 0.491 / 0.508 | 0.472 / 0.555 | 0.441 / 0.629 | 0.468 / 0.561
Nerfies [30] | 0.610 / 0.703 | 0.550 / 0.569 | 0.506 / 0.277 | 0.440 / 0.546 | 0.385 / 0.533 | 0.460 / 0.542 | 0.535 / 0.326 | 0.498 / 0.500
HyperNeRF [31] | 0.601 / 0.696 | 0.517 / 0.586 | 0.501 / 0.272 | 0.437 / 0.554 | 0.547 / 0.444 | 0.397 / 0.556 | 0.547 / 0.322 | 0.507 / 0.490
RoDynRF [23] | 0.552 / 0.722 | 0.513 / 0.634 | 0.482 / 0.321 | 0.413 / 0.594 | 0.570 / 0.484 | 0.613 / 0.536 | 0.478 / 0.449 | 0.517 / 0.534
DpDy (Ours) | 0.596 / 0.735 | 0.478 / 0.630 | 0.447 / 0.387 | 0.457 / 0.622 | 0.571 / 0.500 | 0.562 / 0.531 | 0.504 / 0.511 | 0.516 / 0.559

###  3.2 Diffusion Priors for Novel View Supervision

As aforementioned, using reconstruction losses on existing views is
insufficient. To address this challenge, we employ guidance from RGB-D
diffusion priors for novel views of the 4D scenes. Using RGB-D diffusion
priors offers two advantages. Firstly, comparing to the use of RGB diffusion
priors in previous text-to-3D generation work [32], RGB-D guidance provides
direct geometry supervision. Moreover, unlike depth guidance using off-the-
shelf monocular depth estimation, which produces a single certain depth map
conditioned on a given image, the RGB-D diffusion model learns a joint
distribution of images and depth maps. As a result, the RGB-D diffusion model
provides as output a distribution of image-depth pairs, enabling more robust
supervision for 4D scene reconstruction.

Practically, we use LDM3D [41] as the RGB-D prior. An LDM3D model is a latent
diffusion model that consists of an encoder $\mathcal{E}$, a decoder
$\mathcal{D}$, and a denoising function $\epsilon_{\phi}$. The encoder
$\mathcal{E}$ compresses the input RGB-D image $\bm{x}$ into a low-resolution
latent vector $\bm{z}$, denoted as $\bm{z}\coloneqq\mathcal{E}(\bm{x})$, and
the decoder $\mathcal{D}$ recovers the RGB-D image from the latent vector
$\bm{z}$. The denoising score function $\epsilon_{\phi}$ predicts the given
noise on the latent vector that has been perturbed by noise $\bm{{\epsilon}}$,
the estimated noise denoted as $\bm{\hat{{\epsilon}}}$. Formally, we denote
the gradient of the SDS loss $\mathcal{L}_{\text{sds}}$ as:

$\displaystyle\nabla_{\theta}\mathcal{L}_{\text{sds}}=\mathbb{E}_{t,\bm{{\epsilon}}}~{}\omega(t)(\bm{\hat{{\epsilon}}}-\bm{{\epsilon}})\frac{\partial\bm{z}}{\partial\bm{x}}\frac{\partial\bm{x}}{\partial\theta},$
(6)

where $\theta$ represents for the parameters of the NeRFs and $\omega(t)$ is a
weighting function.

Note that the input depth map of the LDM3D model is up-to-affine. Thus, we
normalize the NeRF-rendered depth maps to $0-1$ range.

Personalization. Similar to recent image-to-3D generation work [34], we apply
the Dreambooth fine-tuning approach [38] to refine the LDM3D model using the
given monocular videos. Specifically, we fine-tune the UNet diffusion model
and the text encoder in the LDM3D model. The text prompt is automatically
generated by using BLIP [16]. To obtain the depth map of the video frames, we
use a pre-trained depth estimation model, MiDaS [36]. Since the output depth
from MiDaS is affine-invairant, we apply a $0-1$ normalization on the
predicted depth maps from MiDaS before the fine-tuning process.

Figure 3: Qualitative comparison on the iPhone dataset. For each sequence, we
sample two frames to show case view synthesis result under different object
motions. The left-most column displays reference image frames used during
training, while the images on the right showcase rendering results for a novel
viewpoint. Our method excels in producing the most realistic view synthesis
for dynamic foregrounds, surpassing the baseline method (NSFF [18]) that
incorporates Lidar depth as extra supervision. It is important to note that
while our background maintains geometric consistency, it appears blurrier
compared to the baselines. This observation explains why our method does not
show an advantage when evaluated using image-based metrics such as SSIM and
LPIPS. However, in a user study (see Table 2) focusing on video quality
inspection, our method significantly outperforms the baselines.

##  4 Experiments

###  4.1 Implementation Details

Dynamic NeRF Representation The static and dynamic component of our 4D NeRF
representation is built on multi-level hash grids _i.e_. instant-NGP [29]. The
static component is a standard hash grid. For the dynamic component, we chose
to decompose 4D space-time into three subspaces. Specifically, we have three
hash grids, each encodes xyz, xyt, yzt, xzt subspaces. The resulting encodings
from these hash grids are concatenated and then passed through small MLPs to
produce output colors and densities. The decomposition of 4D into lower-
dimensional subspaces has been explored in previous works [2, 40]. We observe
that different implementations of such decomposition do not significantly
impact final results. Therefore, we choose the implementation with the lowest
rendering time.

During training, we render $240\times 140$-res image, 1/4th of the original
image size. To improve rendering efficiency, we employ the importance sampling
with a proposal density network as in MipNeRF 360 [1]. The small proposal
density network (modeled using hash grids as described above, but with smaller
resolution and cheaper MLPs) samples 128 points per ray and the final NeRF
model samples 64 points. Additional detailed hyperparameters of our 4D
representation are provided in the supplementary.

Optimization Details. The selection of hyperparameters, specifically
$\lambda_{\text{rgb}}$ and $\lambda_{\text{sds}}$, plays a crucial role in
shaping the training dynamics. A high value for $\lambda_{\text{rgb}}$ tends
to result in slow improvement on novel views. Conversely, a large
$\lambda_{\text{sds}}$ can lead to the loss of video identity during the
initial stages of training. To strike a balance, we choose to initiate
training with a substantial $\lambda_{\text{rgb}}$ value set to 1.0,
emphasizing the accurate fitting of input video frames in the early phase.
Subsequently, we decrease it to 0.1, shifting the focus towards enhancing
novel views in the later stages of training. $\lambda_{\text{sds}}$ is kept
fixed as 1.0 throughout all training iterations.

We empirically set the weighting for z-variance as 0.1 and the skewed entropy
of the foreground-background density ratio as 1e-3. The method is trained for
30,000 iterations with a fixed learning rate of 0.01 using the Adam optimizer.

SDS Details. We adopt the noise scheduling approach introduced in HIFA [56].
Specifically, instead of uniformly sample $t$ as in standard SDS [32], we find
deterministically anneal $t$ from 0.6 to 0.2 leads to finer reconstruction
details. We set a small classifier-free guidance weight as 7.5. Using larger
weights, such as 100, would result in over-saturated result.

Virtual View Sampling. During training iterations, we randomly sample a camera
viewpoint for novel view supervision by perturbing a camera position randomly
picked from input video frames. 360∘ viewpoint sampling is currently not
supported due to the complexity of real-world scenes, making it challenging to
automatically sample cameras that avoid occlusion while keeping the object of
interest centered. A more principled approach is deferred to future work.

###  4.2 Evaluation

Dataset. We conduct experiments on the iPhone dataset from DyCheck [8]. It
contains 14 videos, among which half of them comes with held-out camera views
for quantitative evaluation. This dataset presents a more challenging and
realistic scenario compared to widely-used datasets such as Nvidia Dynamic
View Synthesis [54] and the data proposed by Nerfies [30] and HyperNeRF [31].
The iPhone dataset features natural camera motion, in contrast to other
datasets where cameras teleport between positions. Gao  _et al_. [8]
discovered that methods performing well in teleported video sequences
experience a significant performance drop on the iPhone dataset. Teleported
videos makes the target dynamic scene appears quasi-static, thus makes the
problem simpler, but less practical since everyday video captured by users
usually does not contain rapid camera motions.

Baselines. We compare against well-known methods including NSFF [18], Nerfies
[30], HyperNeRF [31], T-NeRF [9], and more recent approach _i.e_. RoDynRF
[23]. Baselines reported in the DyCheck paper were improved through
supervision with a Lidar depth sensor (denoted with “+ Lidar”). Given our
method’s commitment to practicality without assuming the use of depth sensors,
our primary focus lies in comparing against baselines that _do not_ involve
Lidar depth. We also made a sincere attempt to reproduce DynIBaR [20].
However, due to the complexity of their implementation, which includes
undisclosed third-party modules, we were unable to generate reasonable results
on the iPhone dataset. As a result, it was omitted from our comparison.

Metrics. We adopt the evaluation metrics proposed by Gao  _et al_. [8],
including masked Learned Perceptual Image Patch Similarity (mLPIPS) [55] and
Structural Similarity Index (mSSIM) scores, focusing on regions co-visible
between testing and training views. However, we find that these metrics do not
reflect the perceived quality of novel views. For instance, the baseline
method involving the training of a time-modulated NeRF (T-NeRF) without
advanced regularization attains the best performance according to the metrics.
However, a visual inspection reveals that T-NeRF produces least satisfactory
results in dynamic regions, often resulting in fragmented or blurry
foregrounds. This discrepency between visual quality and testing metrics is
due to methods using only input monocular videos is thereotically not possible
to estimate the correct relative scale between moving foreground and static
background. The scale ambiguity introduces shifts, enlargements, or shrinkages
in the rendered foreground compared to ground-truth images, leading to
decreases in SSIM and LPIPS metrics. However, it does not notably impact the
perceived quality for human. Creating a metric that better aligns with
perceived visual quality is a non-trivial task, and we leave this for future
research.

###  4.3 Comparison to Baseline Methods

We present qualitative comparison results in Fig. 3-4, where we maintain a
fixed camera pose identical to the first video frame and render the subsequent
frames. We observe that our method produces the most visually satisfying
results among all the compared methods. Canonical-based methods, such as
Nerfies and HyperNeRF, exhibit limited flexibility in capturing complex or
rapid object motions, such as finger interactions with an apple or the motion
of a circulating paper windmill. T-NeRF consistently produces noisy results
during rapid object motions, as seen when a person quickly shifts a teddy
bear. The baseline with the closest visual quality to ours is NSFF supervised
with Lidar depth. While its background is relatively more stable, its
foreground is often blurrier and more flickering compared to ours. Please
refer to our supplementary video for more detailed visual comparison.

We quantitatively compare our method with the baseline methods, as shown in
Table 1. The baselines are categorized into two groups based on whether Lidar
depth is used. Row 2-3 showcase results using NSFF [18] and T-NeRF [9] with
Lidar depth incorporated during training. Rows 4-8 present results where Lidar
depth is not employed. We find our method is competitive in terms mLPIPS and
mSSIM. However, as previously discussed in Sec. 4.2, we note the metrics do
not reflect the perceived visual quality. As depicted in Fig.4, RoDyRF [23]
yields similar metrics to ours. However, their rendering under test views
exhibits numerous artifacts and severe flickering when inspecting the rendered
videos. We attribute the reason why our method does not exhibit a significant
advantage over RoDyRF in terms of metrics to our results being slightly over-
smoothed and having color drift due to the current limitations of SDS – which
is shared with other SDS-based text-to-3D generation methods. Additionally, as
discussed in Sec.4.2, dynamic regions are slightly shifted (most noticeable in
the human example in the 3rd row of Fig. 4) compared to the ground truth due
to scale ambiguities, rendering metrics unable to fully capture the visual
quality.

Table 2:  User study results. We report the percentage of annotators choosing
our method against a competing baseline. Two different settings are reported:
bullet-time and stabilized-view rendering.

Experiment | T-NeRF [9] | Nerfies [30] | HyperNeRF [31]
---|---|---|---
Bullet-time rendering | 97% | 100% | 97%
Stabilized-view rendering | 97% | 100% | 83%

###  4.4 User Studies

Commonly adopted metric do not precisely reflect the advantages of our method
as discussed before. To compare methods visually, we performed a user study,
in which the annotators were asked to select a visualization that is most
consistent volumetrically, has least ghost artifacts, and overall looks more
realistic. Table 2 reports the results. We performed two different
experiments. In the first, we rendered the dynamic scene using bullet-time
effect. In the second, we stabilized the view. Our method is preferred by
human annotators in almost all the cases. This clearly shows the advantages of
the proposed approach.

Figure 4: Qualitive comparison against RoDynRF [23]. We visualize results for
all testing sequences in the iPhone dataset following the evaluation protocol.
We masked out regions outside the provided covisibility mask. Our method
demonstrates greater realism and fewer artifacts compared to RoDynRF.

###  4.5 Ablation Studies

Ablation on Static-Dynamic Component Decomposition. We find that decomposing
the static and dynamic components in our 4D representation is crucial for
achieving satisfactory visual results. Without the static component, the
background exhibits flickering and is more susceptible to SDS hallucinating
non-existent objects on the dynamically changing background, as highlighted in
Fig. 5 using red boxes. In contrast, our full method achieves a clean
separation between static and dynamic elements, resulting in a more stable
background with fewer hallucinated contents.

Ablation on RGB-D Diffusion Priors. We present results by using RGB diffusion
priors, instead of the RGB-D prior. Specifically, the RGB prior is obtained
from a pre-trained RealisticVision V5.1
model111https://civitai.com/models/4201/realistic-vision-v20 which has been
shown to leads to more realistic text-3D generation [56] compared to
StableDiffusion 1.5 [37]. We keep the same hyperparameters of DreamBooth for
both RGB and RGB-D model finetuning. Fig. 6 shows the comparison. Compared to
using the RGB prior (2nd row of each example), the novel view results with the
RGB-D prior (1st row of each example) exhibit more detailed texture,
reasonable geometries, and fast convergence over training iterations.

Figure 5: Ablation on static-dynamic component decomposition. Our method
incorporates a dynamic foreground and static background decomposition module,
a critical element for synthesizing sharper backgrounds with reduced
hallucination, as highlighted by the red rectangles.  Figure 6: Ablation on
RGB-D v.s. RGB Diffusion Priors. We visualize the rendering of novel views at
every 5,000 training iterations. Using RGB-D SDS produces superior results
compared to using RGB SDS.

##  5 Conclusion

We propose a novel approach for dynamic 4D scene reconstruction from monocular
videos. Unlike previous works that encounter challenges when employing hand-
crafted priors for generating novel views, to the best of our knowledge, our
method is the first to explore 2D image diffusion priors for dynamic novel
view supervision in generic scenes. This incorporation enhances the robustness
of our method in addressing challenges such as self-occlusion, out-of-view
occlusion, and complex object motions. Our findings suggest that future
research should leverage the advantages brought by large generative models.

Despite encouraging results, our method has limitations summarized as follows:
(1) Due to the necessity of rendering the entire image and running large
diffusion models, our method currently requires high-end GPUs for over 10
hours of training per video with 400 frames. Constrained by computational
cost, the resolution of our view synthesis is limited. Future works could
explore more efficient representations, such as Gaussian splatting [50, 12],
and lighter diffusion models [17, 14]; (2) Temporal smoothness is currently
implicitly regularized by the multi-level design of instant-NGP. It may not be
robust enough for flickering-free reconstruction. Although we have
preliminarily explored utilizing video diffusion priors (e.g., AnimateDiff
[10]) in SDS loss, substantial improvement was not observed. We leave the
exploration of stronger video diffusion models as future work; (3) The current
implementation is confined to a bounded dynamic scene. Extending this work to
an unbounded scene can be achieved through either progressively combining
multiple grids [47] or using image-conditioned rendering, as in DyniBaR [20].
(4) Finetuning on single video losses generalization for diffusion models.
Currently, our method does not support 360∘ reconstruction if the input video
did not already enumerate the surrounding views. Image-conditioned generative
models could potentially eliminate the need for finetuning, but currently
available models are trained on object-centric data with backgrounds removed.

## Supplementary Material

In the supplementary material, additional implementation details are provided.
For more visual results, please refer to the webpage at
https://mightychaos.github.io/dpdy_proj/.

## A: 4D Representation Details

4D representation with 3D grids. Motivated by recent works ([40, 2, 3]) that
decompose high-dimensional voxels into lower-dimensional planes, in this work
we choose to decompose the 4D space-time grid into three 3D grids. Each 3D
grid is represented using an instant-NGP, capturing the $(x,y,t)$, $(x,z,t)$,
and $(y,z,t)$ subspaces. The hyperparameters of the instant-NGPs are detailed
in Table 3. To extract the density and color information of a spacetime point
$(x,y,z,t)$, depicted in Fig. 7, we query each of the three 3D grids,
obtaining three embeddings. Subsequently, these embeddings are concatenated
and input into a small MLP to yield a fused embedding. This fused
representation is then directed through additional MLPs to generate
predictions for density ($\sigma_{d}$) and color ($\mathbf{c}_{d}$).

Table 3: Hyper-parameter of Instant-NGP for 4D representation. | Density Proposal | Radiance Field
---|---|---
n_levels | 8 | 16
n_features_per_level | 2 | 2
log2_hashmap_size | 19 | 19
base_resolution | 16 | 16
per_level_scale | 1.447 | 1.447
Figure 7: Illustration of using 3D grids to represent 4D spacetime radiance
fields.

Blending radiance fields. The color of an image pixel along a ray $r$, denoted
as $C(r)$, is rendered from the blended radiance field of the static and
dynamic NeRFs with densities $\sigma_{s}$, $\sigma_{d}$ and colors
$\mathbf{c}_{s}$, $\mathbf{c}_{d}$.

$C(r)=\int_{t_{n}}^{t_{f}}T(t)\left[\sigma_{s}(t)\bm{c}_{s}(t)+\sigma_{d}(t)\bm{c}_{d}(t)\right]dt,$
(7)

where $T(t)=\exp(-\int_{t_{n}}^{t}(\sigma_{s}(s)+\sigma_{d}(s))ds)$ is the
accumulated transmittance, and $t_{n}$ and $t_{f}$ are the near and far bounds
of a ray.

The discretized equation of Eq. 7 is computed as follow:

$\displaystyle
C(r)=\sum_{i=1}^{N}T_{i}(1-\exp(-(\sigma_{s_{i}}+\sigma_{d_{i}})\delta_{i})\bm{c}_{i},$
(8) $\displaystyle\text{where}\quad
T_{i}=\exp\left(-\sum_{j=1}^{i-1}(\sigma_{s_{j}}+\sigma_{d_{j}})\delta_{j}\right),$
$\displaystyle\bm{c}_{i}=\frac{\sigma_{s_{i}}\bm{c}_{s_{i}}+\sigma_{d_{i}}\bm{c}_{d_{i}}}{\sigma_{s_{i}}+\sigma_{d_{i}}}\quad\text{and}\quad\delta_{i}=t_{i+1}-t_{i}.$

###### Proof.

$\displaystyle C(r)=$
$\displaystyle\int_{t_{n}}^{t_{f}}T(t)\left[\sigma_{s}(t)\bm{c}_{s}(t)+\sigma_{d}(t)\bm{c}_{d}(t)\right]dt$
$\displaystyle=$
$\displaystyle\int_{t_{n}}^{t_{f}}e^{-\int_{t_{n}}^{t}(\sigma_{s}(s)+\sigma_{d}(s))ds}\left[\sigma_{s}(t)\bm{c}_{s}(t)+\sigma_{d}(t)\bm{c}_{d}(t)\right]dt$
$\displaystyle=$
$\displaystyle\int_{t_{n}}^{t_{f}}\frac{d}{dt}e^{-\int_{t_{n}}^{t}(\sigma_{s}(s)+\sigma_{d}(s))ds}\frac{\sigma_{s}(t)\bm{c}_{s}(t)+\sigma_{d}(t)\bm{c}_{d}(t)}{\sigma_{s}(t)+\sigma_{d}(t)}dt$
$\displaystyle\approx$
$\displaystyle\sum_{i=1}^{N}(T_{i}-T_{i+1})\frac{\sigma_{s_{i}}\bm{c}_{s_{i}}+\sigma_{d_{i}}\bm{c}_{d_{i}}}{\sigma_{s_{i}}+\sigma_{d_{i}}}$
$\displaystyle=$
$\displaystyle\sum_{i=1}^{N}T_{i}(1-\exp(-(\sigma_{si}+\sigma_{di})\delta_{i}))\frac{\sigma_{s_{i}}\bm{c}_{s_{i}}+\sigma_{d_{i}}\bm{c}_{d_{i}}}{\sigma_{s_{i}}+\sigma_{d_{i}}}$

∎

In practise, to avoid numerical instability, we add a small value
$\epsilon=1e-6$ to the denominator of $\mathbf{c}_{i}$, _i.e_.,

$\bm{c}_{i}=\frac{\sigma_{s_{i}}\bm{c}_{s_{i}}+\sigma_{d_{i}}\bm{c}_{d_{i}}}{\sigma_{s_{i}}+\sigma_{d_{i}}+\epsilon}.$
(9)

## B: Training Details

Rendering details. For each training iteration, we render one reference view
image and one novel view image. Since diffusion models are finetuned using
$512\times 512$-res images, we randomly crop the novel view image into a
square image patch and resize it to $512\times 512$-res. To match the
distribution of the pretrained RGB-D diffusion model, we convert the rendered
depth map to disparity map by taking the reciprocal of depth values. Then the
disparity map is normalized between 0 and 1 by:

$d=\frac{d-d_{\text{min}}}{d_{\text{max}}-d_{\text{min}}+\epsilon},$ (10)

where $d_{\text{max}}$ and $d_{\text{min}}$ are $95\%$-percentile values,
which are more robust to noise compared to directly taking the maximum and
minimum disparity values.

Hyperparameters. We employ the same set of hyperparameters across all
experiments, with detailed weightings for each loss function provided in Table
4.

param. | value | decay step
---|---|---
$\lambda_{\text{rgb}}$ | 1.0 $\Longrightarrow$ 0.1 | 7k
$\lambda_{\text{depth}}$ | 0.1 $\Longrightarrow$ 0.01 | 2k
$\lambda_{\text{z-variance}}$ | 0.1 | -
$\lambda_{\text{f-g decomp}}$ | 1e-4 | -
$\lambda_{\text{sds}}$ | 1.0 | -
Table 4: Hyperparameter of the loss function. $\Longrightarrow$ denotes the
weighting of the loss exponentially decays.

## References

  * [1] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5470–5479, 2022.
  * [2] Ang Cao and Justin Johnson.  Hexplane: A fast representation for dynamic scenes.  CVPR, 2023.
  * [3] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su.  Tensorf: Tensorial radiance fields.  In European Conference on Computer Vision (ECCV), 2022.
  * [4] Dave Zhenyu Chen, Haoxuan Li, Hsin-Ying Lee, Sergey Tulyakov, and Matthias Nießner.  Scenetex: High-quality texture synthesis for indoor scenes via diffusion priors.  arXiv preprint arXiv:2311.17261, 2023.
  * [5] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia.  Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 22246–22256, October 2023.
  * [6] Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian.  Fast dynamic radiance fields with time-aware neural voxels.  In SIGGRAPH Asia 2022 Conference Papers, 2022.
  * [7] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang.  Dynamic view synthesis from dynamic monocular video.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5712–5721, October 2021.
  * [8] Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa.  Dynamic novel-view synthesis: A reality check.  In NeurIPS, 2022.
  * [9] Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa.  Monocular dynamic view synthesis: A reality check.  In Advances in Neural Information Processing Systems (NeurIPS), 2022.
  * [10] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai.  Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.  arXiv preprint arXiv:2307.04725, 2023.
  * [11] Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, and Ben Poole.  Zero-shot text-guided object generation with dream fields.  CVPR, 2022.
  * [12] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.  3d gaussian splatting for real-time radiance field rendering.  ACM Transactions on Graphics (ToG), 42(4):1–14, 2023.
  * [13] Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Popa Tiberiu.  Clip-mesh: Generating textured meshes from text using pretrained image-text models.  SIGGRAPH Asia 2022 Conference Papers, December 2022.
  * [14] Bo-Kyeong Kim, Hyoung-Kyu Song, Thibault Castells, and Shinkook Choi.  Bk-sdm: Architecturally compressed stable diffusion for efficient text-to-image generation.  In Workshop on Efficient Systems for Foundation Models@ ICML2023, 2023.
  * [15] Jiahui Lei and Kostas Daniilidis.  Cadex: Learning canonical deformation coordinate space for dynamic surface representation via neural homeomorphism.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6624–6634, 2022.
  * [16] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi.  BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models.  In ICML, volume 202 of Proceedings of Machine Learning Research, pages 19730–19742. PMLR, 2023.
  * [17] Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren.  Snapfusion: Text-to-image diffusion model on mobile devices within two seconds.  arXiv preprint arXiv:2306.00980, 2023.
  * [18] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang.  Neural scene flow fields for space-time view synthesis of dynamic scenes.  In CVPR, 2021.
  * [19] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang.  Neural scene flow fields for space-time view synthesis of dynamic scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [20] Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely.  Dynibar: Neural dynamic image-based rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [21] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin.  Magic3d: High-resolution text-to-3d content creation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [22] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object.  arXiv preprint arXiv:2303.11328, 2023.
  * [23] Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung-Yu Tseng, Ayush Saraf, Changil Kim, Yung-Yu Chuang, Johannes Kopf, and Jia-Bin Huang.  Robust dynamic radiance fields.  In CVPR, 2023.
  * [24] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al.  Wonder3d: Single image to 3d using cross-domain diffusion.  arXiv preprint arXiv:2310.15008, 2023.
  * [25] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth.  NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections.  In CVPR, 2021.
  * [26] Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi.  Realfusion: 360$\\{$$\backslash$deg$\\}$ reconstruction of any object from a single image.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [27] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 405–421. Springer, 2020.
  * [28] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  NeRF: Representing scenes as neural radiance fields for view synthesis.  In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  * [29] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  In ACM Transactions on Graphics (SIGGRAPH), 2022.
  * [30] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [31] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz.  Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields.  ACM Transactions on Graphics (TOG), 40, 2021.
  * [32] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  International Conference on Learning Representations (ICLR), 2022.
  * [33] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [34] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et al.  Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors.  arXiv preprint arXiv:2306.17843, 2023.
  * [35] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  IEEE transactions on pattern analysis and machine intelligence (T-PAMI), 44, 2020.
  * [36] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  IEEE transactions on pattern analysis and machine intelligence (T-PAMI), 44(3):1623–1637, 2020.
  * [37] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, 2022.
  * [38] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [39] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  Advances in Neural Information Processing Systems (NeurIPS), 35:36479–36494, 2022.
  * [40] Sara Fridovich-Keil and Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa.  K-planes: Explicit radiance fields in space, time, and appearance.  In CVPR, 2023.
  * [41] Gabriela Ben Melech Stan, Diana Wofk, Scottie Fox, Alex Redden, Will Saxton, Jean Yu, Estelle Aflalo, Shao-Yen Tseng, Fabio Nonato, Matthias Muller, et al.  Ldm3d: Latent diffusion model for 3d.  arXiv preprint arXiv:2305.10853, 2023.
  * [42] Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen.  Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior.  arXiv preprint arXiv:2303.14184, 2023.
  * [43] Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa.  Mvdiffusion: Enabling holistic multi-view image generation with correspondence-aware diffusion.  In Advances in Neural Information Processing Systems (NeurIPS), 2023.
  * [44] Chaoyang Wang, Ben Eckart, Simon Lucey, and Orazio Gallo.  Neural trajectory fields for dynamic novel view synthesis.  arXiv preprint arXiv:2105.05994, 2021.
  * [45] Chaoyang Wang, Lachlan Ewen MacDonald, László A. Jeni, and Simon Lucey.  Flow supervision for deformable nerf.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21128–21137, June 2023.
  * [46] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich.  Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [47] Peng Wang, Yuan Liu, Zhaoxi Chen, Lingjie Liu, Ziwei Liu, Taku Komura, Christian Theobalt, and Wenping Wang.  F2-nerf: Fast neural radiance field training with free camera trajectories.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4150–4159, 2023.
  * [48] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser.  Ibrnet: Learning multi-view image-based rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690–4699, 2021.
  * [49] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu.  Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation.  arXiv preprint arXiv:2305.16213, 2023.
  * [50] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang.  4d gaussian splatting for real-time dynamic scene rendering.  arXiv preprint arXiv:2310.08528, 2023.
  * [51] Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Oztireli.  D^ 2nerf: Self-supervised decoupling of dynamic and static objects from a monocular video.  Advances in Neural Information Processing Systems, 35:32653–32666, 2022.
  * [52] Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim.  Space-time neural irradiance fields for free-viewpoint video.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [53] Gengshan Yang, Chaoyang Wang, N. Dinesh Reddy, and Deva Ramanan.  Reconstructing animatable categories from videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16995–17005, June 2023.
  * [54] Jae Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Soo Park, and Jan Kautz.  Novel view synthesis of dynamic scenes with globally coherent depths from a monocular camera.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [55] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [56] Joseph Zhu and Peiye Zhuang.  Hifa: High-fidelity text-to-3d with advanced diffusion guidance.  arXiv preprint arXiv:2305.18766, 2023.


# Evaluating Twitter’s Algorithmic Amplification of Low-Credibility Content:
An Observational Study

Giulio Corsi
Leverhulme Centre for the Future of Intelligence
University of Cambridge
<EMAIL_ADDRESS>
###### Abstract

Artificial intelligence (AI)-powered recommender systems play a crucial role
in determining the content that users are exposed to on social media
platforms. However, the behavioural patterns of these systems are often
opaque, complicating the evaluation of their impact on the dissemination and
consumption of disinformation and misinformation. To begin addressing this
evidence gap, this study presents a measurement approach that uses observed
digital traces to infer the status of algorithmic amplification of low-
credibility content on Twitter over a 14-day period in January 2023. Using an
original dataset of $\approx$ 2.7 million posts on COVID-19 and climate change
published on the platform, this study identifies tweets sharing information
from low-credibility domains, and uses a bootstrapping model with two
stratifications, a tweet’s engagement level and a user’s followers level, to
compare any differences in impressions generated between low-credibility and
high-credibility samples. Additional stratification variables of toxicity,
political bias, and verified status are also examined. This analysis provides
valuable observational evidence on whether the Twitter algorithm favours the
visibility of low-credibility content, with results indicating that tweets
containing low-credibility URL domains perform significantly better than
tweets that do not across both datasets. Furthermore, high toxicity tweets and
those with right-leaning bias see heightened amplification, as do low-
credibility tweets from verified accounts. This suggests that Twitter’s
recommender system may have facilitated the diffusion of false content, even
when originating from notoriously low-credibility sources.

##  1 Introduction

The emergence of social media platforms has brought about a significant
transformation in global patterns of information dissemination and
consumption, as a large number of internet users now rely on these channels as
their primary sources of information acquisition [1, 2, 3, 4]. The rapid
growth in social media membership, and consequently of digital traces
circulating in these platforms, have been accompanied by a progressive rise in
the importance of artificial intelligence (AI) based recommender systems,
content pre-selection, ranking and suggestion systems used to customise users’
online experiences [5, 6].

The integration of AI-based recommender systems into social media platforms
has led to a fundamental shift in the way users consume and interact with
online information [7], significantly increasing the level of automated
content curation while limiting users’ freedom of independent content
discovery [8]. This paradigm shift towards the machine-learning based hyper-
personalisation of social media content raises concerns regarding potential
impacts on the quality and diversity of information available to users, with
clear implications for the integrity of knowledge acquisition processes.
Several recent studies have analysed these risks, concluding that engagement-
based recommender systems - which form the majority of recommendation engines
currently deployed within social media platforms - may be prone to bias [9,
10], user-manipulating behaviour [11], the creation of echo chambers [5, 12],
and to the amplification of false or misleading content [13, 14].

Despite their status as critical infrastructure of social media platforms -
and arguably of information circulation at a societal level - the internal
architectures and practical functioning of recommender systems remain only
superficially understood [15], and while several platforms have previously
released white papers with information on their functioning [16, 17, 18],
limited evidence exists on the characteristics that guide their deployment.
This is also the case for Twitter, (Now X Corp.) which recently made parts of
its recommender system public, providing a window into the functioning of a
social media content suggestion system [19]. However, while this release does
provide new information on the system’s architecture, perhaps the most central
part of the system - a ‘heavy ranker’- deep neural network used to make
recommendation predictions [20] cannot be replicated with currently available
information, limiting the possibility of testing the behaviour of this
recommender system. This lack of evidence is a clear obstacle towards
evaluating the magnitude of any form of algorithmic bias in content
suggestion, and in particular, towards understanding whether, in their drive
to maximise user engagement with a platform, recommender systems are acting as
significant drivers of the diffusion of online disinformation and
misinformation. This is crucial, as understanding how false and low-
credibility content propagates within social media platforms is key to
improving the safety of societal information commons.

In an effort to address the existing evidence gap, this study introduces a
measurement approach that leverages existing digital traces to empirically
observe - analysing recommendation outcomes through impression counts - the
state of the promotion of low-credibility content on Twitter in a two week
period in January 2023. The motivation for choosing Twitter as the object of
this study is threefold: First, Twitter has a large user base with global
reach, with a monthly user base of approximately 450 million users and over
300 million daily tweets [21], making it one the largest and most influential
social media platforms globally. Second, Twitter has been often criticised for
platforming and amplifying extremist content, disinformation and
misinformation [22, 23], and studying its recommender systems may provide
additional insights into how false information spreads on the platform. Third,
Twitter was, at the time of data collection, the only platform which provided
data on impressions through its API. While Twitter’s recommender system has
undergone significant changes since January 2023, this study aims to offer a
snapshot of the recommender system’s behaviour at a crucial juncture in the
platform’s evolution, before the recent system-wide overhaul. Although the
findings may not directly translate to the current recommender system, they
may provide unique insights into a pivotal transitional moment, revealing
baseline tendencies in how the system influenced the propagation of low-
credibility content.

To map the behaviour of Twitter’s recommender system, this study uses an
original dataset of $\approx$ 2.7 million tweets discussing COVID-19 and
climate change published on the platform in a 14-day period in January 2023,
and extracts tweets containing information from low-credibility domains and
high-credibility domains [24], testing whether low-credibility information
gets amplified visibility on the platform. Data on impressions - defined in
the API documentation as the count of how many times a Tweet has been seen
[25] \- was initially made available through the Twitter API in January 2023
and, as a passive metric of how many users have been exposed to a tweet,
provides a powerful window into the content that the Twitter recommender
system tailors to users. Through this process, it is possible to estimate
whether tweets sharing information from low-credibility domains generate
exceptional impressions, which would point towards a recommendation bias
towards low-credibility content, as well as a general lack of functioning
integrity signals [6]. By analysing the visibility of low-credibility content
- which can be used as a baseline for false or misleading information - this
research seeks to provide insights into the dynamics that drive user exposure
to false or misleading information, with potential implications for social
media platforms and the broader digital information ecosystem.

##  2 Data and Methods

###  2.1 Data

####  2.1.1 Twitter Data

Figure 1: Distribution of tweets with low-credibility and high-credibility
domains as a percentage of the full data in each dataset under analysis

The data utilised in this study was collected from the Twitter API V2 in a
14-day time period between January 15th to January 29th, 2023. Given the
research objective of assessing whether the Twitter recommender system
amplifies the visibility false or misleading information, this work uses data
from discussions on COVID-19 and climate change, two debates that are often
considered publicly divisive and subject to a significant circulation of false
content [26, 27, 28]. Data discussing these topics was collected through a
keyword search using the R package AcademictwitteR [29], which at the time of
collection, was chosen for its exclusive capability of collecting data on a
tweet’s impressions. The data was collected at regular intervals ensuring
similar uptime for each day of publication, and the resulting dataset
comprises a total of $\approx$ 2.1m original tweets on COVID-19, and $\approx$
600k original tweets on climate change. Data collected from the Twitter API is
quite granular, and is made up of 21 variables, encompassing several tweet-
level and user-level variables, such as engagement metrics, verification
statuses, locations and users’ profile images.

####  2.1.2 URL Domains Classification

In order to extract tweets that are likely to contain false or misleading
information, this study relies on the use of URL domain credibility ratings,
which is often used in the literature on the study of disinformation and
misinformation [30, 31, 32]. While several datasets domain credibility exist,
such as NewsGuard and IffyNews, this study identifies information reliability
scores using the aggregate reliability scores from [24], which used principal
component analysis to produce aggregate scores derived from the major
available rating sets. This dataset provides credibility scores for 11,520
domains, where 0 represents the lowest credibility, and 1 represents the
highest credibility. In line with the values used by major credibility ratings
providers such as Newsguard, we then consider tweets with a credibility score
lower or equal to 0.4 to be low-credibility and tweets with a credibility
score equal or higher than 0.6 to classify as high-credibility [33]. This
process results in a total of 87,769 tweets from low-credibility sources, and
187,643 tweets from high-credibility sources, with low-credibility domains
present in 3.77% of tweets on COVID-19 and 1.69% of tweets on climate change.
The most common low-credibility and high-credibility domains for both datasets
are shown in Table 1.

COVID-19 | Climate Change
---|---
Key | Domain | Frequency | Key | Domain | Frequency
High Credibility | theguardian.com | 7414 | High Credibility | theguardian.com | 6409
| reuters.com | 5364 |  | nytimes.com | 1275
| washingtonpost.com | 3056 |  | washingtonpost.com | 1239
| cnn.com | 2868 |  | independent.co.uk | 925
| msn.com | 2499 |  | weforum.org | 812
Low Credibility | rumble.com | 13911 | Low Credibility | breitbart.com | 925
| thegatewaypundit.com | 9614 |  | thegatewaypundit.com | 862
| theepochtimes.com | 5645 |  | rumble.com | 860
| expose-news.com | 5230 |  | dailymail.co.uk | 850
| zerohedge.com | 5047 |  | zerohedge.com | 705
Table 1: Distribution of the five most prevalent high-credibility and low-
credibility domains for both datasets

###  2.2 Measuring Amplification

####  2.2.1 Baseline Amplification Benchmark

Measuring recommender-driven amplification is a notoriously difficult task,
which requires clearly defined objectives and robust benchmarks to identify
potential patterns of amplification. In this study, amplification is defined
as a condition where tweets with similar characteristics drawn from two
different groups - low-credibility and high-credibility - exhibit a
significant difference in the outcome variable, which is the number of
impressions obtained. Amongst the available metrics currently provided by the
Twitter API, data on impressions is the most suited for the study of
recommender-based amplification, as it is the only available passive metric of
exposure, measuring how many people a certain tweet has been shown two
irrespective of voluntary engagement. As users have no control on impressions,
this metric is expected to be effective in characterising the behaviour of the
recommender system [34] . To produce a clear measure of amplification, it is
therefore important to establish a robust benchmarking procedure to compare
the two samples under analysis. For this purpose, this study compares the two
previously described samples of high-credibility and low-credibility tweets
through bias-corrected and accelerated bootstrapping (BCa), where the mean
difference between the two samples is measured across 1000 randomly resampled
iterations while incorporating a bias-correction factor in the resampling
process [35, 36]. The number of iterations used ensures that the mean
difference between the two samples is based on a substantial number of
comparisons, reflecting a real difference in impressions. Further, as a non-
parametric statistical approach, bootstrapping requires fewer assumptions
about the distribution of the data to hold, which makes this approach
particularly suited for the study of social media data.

However, a simple bootstrapping benchmark is not sufficient to reliably
determine whether a sample consistently received more impressions than the
other, as it neglects potential user-level and tweet-level factors that could
influence the number of impressions. To remedy this shortcoming, the baseline
benchmark bootstrap comparison is performed with two stratifications,
resampling the data by a tweet’s engagement level and the user’s number of
followers. Engagement was selected as a baseline stratification variable as
engagement-based recommender systems are known to highly value a tweet’s
engagement performance [37], and high-engagement tweets are likely to be shown
more than low-engagement tweets. Followers count was selected as a baseline
stratification variable as, as a networked recommender system, the number of
followers of a tweet creator is likely to have a non-negligible impact on how
many people are exposed to a tweet. While these two variables alone may not
account for the entirety of tweet-level and user-level factors that are likely
to influence impressions, adding an excessive number of stratification
variables with limited explanatory power is likely to be counterproductive, as
it would significantly reduce the number of matched samples. Considering these
limitations, stratifying the baseline benchmark by levels of engagement and
followers appears the most effective strategy to maximise the accuracy and
validity of the results.

As both engagement and followers count are discrete variables with a large
range of values, the complexity of these variables is reduced by assigning the
data to discrete clusters using quantile based discretization, an approach
that allows for a grouping of the data into similar-sized buckets based on
quantile rankings [38]. This approach was tested alongside more traditional
approaches to clustering such as HBDSCAN and K-means clustering, and
consistently provided a more effective grouping of the data. Following an
exploratory analysis of the distribution of both variables, the arbitrary
numbers of discrete groups to be identified is set to 4, a number that
preserves the original variability of the data without placing undue
restrictions on the bootstrapping process, producing a total of 16
combinations of strata of engagement and followers clusters. To guarantee
consistent results in the bootstrapping stage, quantile-based discretization
is applied to the combined datasets of low and high credibility data for each
distinct dataset under investigation.

####  2.2.2 Additional Stratification Variables

After developing a method to compute the baseline level of amplification
across the two datasets, we can add further individual stratification
variables to test the influence of additional grouping variables across
subgroups. For this purpose, each additional stratum is separately added to
the baseline benchmark, computing any difference between to the baseline
amplification of amplification after the addition of a new stratification
variable. At this stage, we test amplification across three additional
stratification variables: toxicity scores, political bias and verified status.

Toxicity scores are obtained through the Perspective API by Jigsaw [39], which
leverages a machine learning model trained on millions of Wikipedia comments
to predict how likely it is that an input text will be perceived as toxic by a
reader. Like tweets, Wikipedia comments are largely short and informal, making
this model quite suited for the analysis of Twitter data [40, 41]. The
Perspective API model produces a toxicity score ranging from 0 to 1 for each
input tweet, with 0 having a null probability of being found toxic, and 1
having a high probability of a text being perceived as toxic. To avoid
creating an excessive number of categories during the stratification process,
the toxicity scores obtained from the Perspective API are used to create 3
clusters of toxicity levels with k-means clustering, allowing for the
stratification of our data according to the degree of language toxicity.

Further, the political bias of the URL domains under analysis is obtained,
similarly to the credibility scores, by annotating data with the GPT-4 API.
While the use of large language models in data annotation tasks is a new
phenomenon, recent literature has extensively analysed the performance of GPT
3.5 and GPT-4 in data labelling tasks, including political stance
identification, with both models exhibiting high accuracy [42, 43]. To
maximise the usability and interpretability of the data, for this task, the
model is asked to classify the political bias of the input domain into one of
five categories: far-left, left, no bias, right and far-right. The model is
also prompted to return a value of -1 whenever it does not have information on
a domain, or if the domain is non-political. Through this process, we can
identify 5,596 political domains - around 10% of all domains in the data -
which are then used as a stratification variable during bootstrapping to
assess whether political bias has an influence on the amplification of low-
credibility content. Furthermore, it should be noted that far-left sources
were not identified in high-credibility domains, and to maximise the
comparability, the data was grouped in two categories, right-bias and left-
bias.

The third and last stratification variable used in this work is a user’s
verified status, which is used to assess whether the latter is used to amplify
low-credibility content on Twitter. Data on users’ verified status, a binary
variable with values of ’true’ for verified accounts and ’false’ for
unverified ones, was obtained via the Twitter API. However, it’s crucial to
clarify that this data refers to the legacy verified status - a verification
badge Twitter formerly awarded to prominent users as a safeguard against
impersonation. As of November 2022, Twitter began phasing out this legacy
verification in favour of Twitter Blue, a paid subscription service that
enables users to purchase verification. While legacy blue ticks have largely
been removed at the time of writing, information on legacy verified status was
still available from the Twitter API at the time of data collection, and these
accounts mainly include public or institutional profiles with a large
following.

##  3 Results

###  3.1 Baseline Amplification Analysis

Figure 2 illustrates the findings from the baseline amplification analysis,
showing the mean percentage difference obtained from each fold of the bias
corrected and accelerated bootstrapping (BCa) procedure [44]. Here, results
reveal that on average, across 1000 stratified bootstrapping samples
stratified by engagement level and followers count, samples of low-credibility
tweets generate more impressions than high-credibility samples across both
datasets, with low-credibility tweets on COVID-19 receiving a baseline
impressions amplification of +19.2% (median +17.3%) and low-credibility tweets
on climate change generating +95.8% impressions (median = +90.1%). In absolute
values, this amounts to a mean difference of +113.7 impressions (median =
+111.4) for COVID-19 tweets, and +474.6 impressions (median = +447.2) for
climate change tweets. This observation suggests that at an aggregate level
across 1000 balanced samples of high-credibility and low-credibility tweets,
the latter experiences heightened amplification within these two issue
domains, with greater amplification in the context of climate change,
indicating that, in aggregate, Twitter users were more likely to be exposed to
low-credibility information by the platform’s recommender system. Results here
also show that this behaviour is observed quite consistently, as 84.4% of
COVID-19 samples have a positive mean difference, and 97.9% of climate change
samples have a positive mean difference, suggesting that it is very rare that
low-credibility samples will outperform high-credibility samples in
impressions counts.

Figure 2: Raincloud plot illustrating the average percentage difference in
impressions between high-credibility and low-credibility tweets, based on
1,000 resamples from each dataset under study.

However, when dealing with skewed distributions such as those of social media
impressions, looking at the aggregate mean may not be sufficient to fully
explain an amplification effect. Rather, we must also assess inter-stratum
breakdowns of variabilities, which are shown in Figure 3, containing heatmaps
of the mean differences in impressions across all 16 strata combinations as
well as the size of each stratum. This step of the analysis delivers a more
nuanced understanding of the results, showing that within bootstrapped
samples, the observed difference in impressions is primarily generated by a
difference in the highest-engagement and highest-followers strata (3,3). For
COVID-19, this amounts to a mean difference of +3,148 impressions between low-
credibility and high-credibility tweets, while for the climate change dataset,
this amounts to +9,197. While the percentage of amplification within these
strata appears extensive in absolute terms, it is more moderate in relative
terms — showing +30.2% amplification for the COVID-19 stratum 3.3 and +129%
for the climate change data within the same stratum. Qualitatively assessing
tweets from these strata, they appear to largely be conspiratorial tweets from
large right and far-right outlets. For example, the low-credibility tweet with
the highest impressions in such strata cites an article from the Daily Mail,
and states: "A shadowy Army unit secretly spied on British citizens who
criticised govt’s Covid lockdown policies… artificial intelligence deployed to
’scrape’ social media for keywords".

This finding on the distribution of intra-stratum variabilities is crucial for
the interpretation of the bootstrapping results, as it shows that while on
aggregate users on Twitter were more likely to be exposed to low-credibility
content, this effect is largely attributable to a difference in high-
engagement, high-followers tweets, which are very impactful in terms of
impressions generation, and are more likely receive amplified visibility when
containing low-credibility content. This is consequential, as this minority of
tweets are responsible for a large share of impressions generated on Twitter.
This finding also aligns with the broader understanding of Twitter activity
dynamics, where the majority of engagement and impressions are typically
generated by a limited number of highly engaging posts, adding a further layer
of understanding that this subgroup of posts is likely to achieve greater
amplification when containing low-credibility content.

Figure 3: Heatmaps illustrating the percentage difference between low-
credibility and high-credibility tweets in each of the 16 strata under
analysis

###  3.2 Analysis of Additional Stratifications

Building on these observations, Figure 4 illustrates the impact of adding
additional stratification variables - in this case toxicity, political bias
and verified status - to the amplification levels observed in the base model.
Here, results showcase the average raw change of adding an additional
stratification variable to the percentage amplification value observed in the
baseline model. This step provides several insights into how additional
stratification variables impact the amplification of low-credibility content.

The first stratification variable added to the baseline model is a tweet’s
toxicity profile, which provides insights into how the presence of
inflammatory language influences the algorithmic amplification of low-
credibility content. Examples of high-toxicity phrasing include strong
insults, profanity, threats, and intentionally harmful or misleading rhetoric.
These types of inflammatory expressions tend to appear in conspiracy theories
and highly controversial claims, precisely the types of questionable
information examined in this study. Here, results show no consistent
relationship between amplification and toxicity for low and medium toxicity
tweets across both the climate change and COVID-19 datasets. However, high-
toxicity tweets exhibit a heightened algorithmic reach versus the baseline
model for both datasets, at a value of +9.9 and +15.2 respectively. This is
important, as it suggests that tweets containing overtly negative, rude or
disrespectful language see greater visibility on the platform when containing
false or misleading domains. By showing a relationship between high-toxicity
and algorithmic amplification across both datasets, these results support the
understanding that content that is emotionally charged, especially when having
a negative or controversial nature, may benefit from algorithmic amplification
within engagement-based recommender systems.

Furthermore, the addition of political bias as a stratification variable
provides insights on the role of political partisanship in recommender-based
amplification. Here, results show that for both COVID-19 and climate change
datasets, tweets expressing right-leaning political bias see heightened
amplification compared to the baseline model, and compared to left-leaning
tweets. For COVID-19, right-leaning shows an increase of +6 from the baseline
for low-credibility content versus high-credibility content, compared o a
change of +0.03 for left-wing tweets. For climate change, right-leaning tweets
exhibit an additional amplification of +24.3, compared to a value of +6 for
left-leaning tweets. These results indicate that in the moment this analysis
was carried out, tweets containing domains with a right-leaning political bias
were more likely to be amplified in discussions on both COVID-19 and climate
change, particularly if compared with tweets with a left-wing bias. This
finding emerges as a crucial aspect of the current analysis, as it suggests
that the Twitter recommender system, in its pursuit of maximising user
engagement, may inadvertently expose users to content characterised by right-
wing political biases. This observation gains particular significance in the
context of ongoing debates surrounding filter bubbles, polarised information
ecosystems, and the potential consequences of such dynamics on the public
sphere. As the Twitter recommender system warrants increased visibility to
content with pronounced right-wing political leanings, there is a clear risk
that users may be driven towards more radical viewpoints - a trend that has
been previously identified in Youtube’s recommender system [45, 46] \-
exacerbating existing social and political divisions while undermining the
platform’s capacity to serve as a space for diverse and open discourse.

Lastly, to warrant more in-depth insights into user-level factors contributing
to the performance of low-credibility tweets, the third additional layer of
stratification is the verification status of a tweet’s author. Here, results
clearly indicate that low-credibility tweets from users with a legacy verified
status obtained evident amplification, with a change over the baseline model
of +155% for COVID-19 data, and + 138% for climate change data. In contrast,
tweets from unverified authors see minimal change compared to baseline
amplification levels. This effect is very large, particularly if compared with
the previous two stratifications, and indicates that low-credibility
information spread by verified users is far more likely to be amplified on
Twitter compared to low-credibility content shared by non-verified users,
suggesting that the legacy checkmark, which acted as a credibility signal
within the algorithm, may have been weaponised to amplify the reach of false
or misleading content. This finding has important implications regarding the
role of status cues and authority in algorithmic amplification, as it
demonstrates that peripheral credibility signals like the verification status
can override actual content quality in terms of driving engagement and reach.

Figure 4: Raw impact of each value of additional stratifications on the
baseline model of low-credibility amplification. Here, raw values mean that if
in the base model a stratum had an amplification of +10%, and the addition of
a third stratification pushed this to +12%, we would report this as a +2%
difference, not +20%. In this sense, results here are a raw difference between
percentages.

##  4 Discussion and Conclusions

The recent addition of impressions data on the Twitter API offers a unique
opportunity to investigate the role of Twitter’s recommender system in
promoting the circulation of disinformation and misinformation. While this
opportunity has now been limited by restrictions to the Twitter API, this
study presents an initial attempt to use an inferential approach based on
impressions data to assess any differences between low-credibility and high-
credibility tweets on Twitter. The main analysis of this work revealed that
tweets containing URLs from low-credibility domains achieve higher visibility
on the platform, with an average difference after bootstrapping of +19.2% for
COVID-19 data, and +95.8% for climate change data. However, results also show
that within bootstrapped samples, the majority of this effect comes from low-
credibility tweets’ overperformance for high-engagement and high-followers
users, which account for the majority of Twitter’s impressions.

This work also set out to uncover notable features that may explain any
impressions-based amplification, and found that toxicity scores obtained from
Jigsaw’s Perspective API showed a clear pattern where high-toxicity tweets
exhibited heightened amplification when containing low-credibility content,
confirming the existing understanding that toxic content may be more easily
amplified by engagement-based recommender systems. These results support
findings from prior works, particularly those showing clear connections
between negative emotional content and viral misinformation spread [47]. By
confirming the role of engagement-based recommender systems in amplified toxic
content, these findings highlight the need for alternative approaches to
social media recommendation, such as bridging-based recommender systems, which
focus on connecting users with diverse perspectives and high-quality
information rather than maximising engagement alone[48]. While developing
recommender systems that balance engagement, equality and diversity is indeed
challenging, and further research is needed on this topic, this works lends
support to the notion that alternatives to engagement-based recommendations
are worth pursuing to improve online discourse and reduce the amplification of
misinforming, toxic content.

Furthermore, one of the most notable results from the additional
stratification analysis is the substantial amplification observed for
politically biased low-credibility tweets, especially those with right-leaning
partisanship. This finding highlights growing concerns about political
polarisation and the rise of filter bubbles on social platforms. It suggests
that in its pursuit of maximising engagement, the version of Twitter’s
recommender system under analysis may have inadvertently contributed to echo
chambers by disproportionately amplifying tweets that expressed partisan
viewpoints, regardless of their truthfulness. This is particularly concerning
given the vulnerability of high-profile socio-political discussions like
climate change or COVID-19 to manipulation by misinformation campaigns. The
fact that right-leaning tweets see substantially more amplification of false
claims on climate change points to the ability of partisan disinformation to
successfully exploit algorithmic loopholes on Twitter. This highlights
concerns about social media deepening societal divides by siloing users into
hyper-partisan circles, limiting exposure to alternate viewpoints. Lastly,
results showed that low-credibility tweets from legacy verified accounts enjoy
significant amplification on Twitter. While legacy verified accounts are only
a minority of the platform’s population, it is important to note that Twitter
recently changed its verification policy, and the blue tick can now be
purchased as part of a monthly subscription plan. While data on paid-for
verified users is not available via the Twitter API, this finding is
concerning, as it suggests that it is possible that the blue tick, which
warrants amplified visibility within the recommender system, may be weaponised
by malicious actors to spread false and misleading information, particularly
during highly emotionally loaded emergencies such as that of COVID-19. This
point is crucial, and deserves further attention in future research.

Finally, this research offers initial empirical evidence on the promotion of
low-credibility content on Twitter, revealing key factors that may contribute
to a significant impressions overperformance of tweets containing false or
misleading information. Given ongoing concerns about Twitter’s impacts on
information quality and integrity under its new management, this historical
assessment of recommendation patterns can also offer an important benchmark
for future analysis. Moreover, the study’s methodological approach of
leveraging impression counts to empirically analyse recommendation outcomes
remains broadly relevant. Evaluating the visibility afforded to different
types of content on social platforms is key to understanding how algorithmic
curation shapes online information landscapes. By offering a framework to
probe the role of recommender systems in propagating low-credibility
information, this research retains value for inspiring further investigation -
both of Twitter’s evolving dynamics and other influential networks. Capturing
past system behaviours can provide a foundation to monitor emerging risks and
work towards safer, more accountable social information ecosystems.

Recommender systems are arguably the most prevalent application of AI and
machine learning globally, affecting billions of internet users daily.
However, despite their ubiquitous nature, there is an evident absence of
regulation on their large-scale deployment. The results of this study
highlight a need, as a minimum, for increased transparency in the field of
social media recommender systems, as only through system-wide access it will
be possible to produce clearer causal explanations for the effects observed in
this work. However, as the recent release of parts of the Twitter algorithm
eloquently shows, in principle transparency is not enough. Rather, it is
crucial that transparency is accompanied the possibility of full replication
of a recommender system,which would allow for the definition of protocols for
comprehending, auditing, and stress-testing these systems to prevent the
unintentional promotion of disinformation, to curb the perpetuation of harmful
biases, and to safeguard the security of knowledge dissemination and
acquisition processes.

Addressing this challenge is admittedly difficult, as recommender systems are
proprietary, high-value assets at the heart of social media platforms.
Nevertheless, it is vital to stress the significance of transparency and
oversight for systems that mediate access to information and shape public
discourse on a global scale. This study contributes to emphasising these
imperatives, while also offering initial insights into how Twitter’s
recommender system may have specifically amplified misleading and false
information. Future research building on these findings should further
investigate the dynamics of disinformation circulation on social media and
explore potential interventions to curb the unintentional spread of
misinformation by recommender systems. Ultimately, continued research in this
domain is essential to ensure the responsible design and implementation of AI
systems.

## References

  * [1] Iryna Pentina and Monideepa Tarafdar  “From “information” to “knowing”: Exploring the role of social media in contemporary news consumption”  In _Computers in human behavior_ 35, 2014, pp. 211–223
  * [2] Elisa Shearer and Jeffrey Gottfried  “News consumption across social media in 2017” Accessed: 20-Apr-2023, https://www.pewresearch.org/journalism/2017/09/07/news-use-across-social-media-platforms-2017/, 2017
  * [3] Mason Walker and Katerina Eva Matsa  “News consumption across social media in 2021” Accessed: 20-Apr-2023, https://www.pewresearch.org/journalism/2021/09/20/news-consumption-across-social-media-in-2021/, 2021
  * [4] Matteo Cinelli et al.  “The echo chamber effect on social media”  In _Proceedings of the National Academy of Sciences_ 118.9, 2021, pp. e2023301118
  * [5] Anitha Anandhan, Liyana Shuib, Maizatul Akmar Ismail and Ghulam Mujtaba  “Social media recommender systems: review and open research issues”  In _IEEE Access_ 6, 2018, pp. 15608–15628
  * [6] Luke Thorburn  “How platform recommenders work” Accessed: 20-Apr-2023  Understanding Recommenders, https://medium.com/understanding-recommenders/how-platform-recommenders-work-15e260d9a15a, 2022
  * [7] Fernando P Santos, Yphtach Lelkes and Simon A Levin  “Link recommendation algorithms and dynamics of polarization in online social networks”  In _Proceedings of the National Academy of Sciences_ 118.50, 2021, pp. e2102141118
  * [8] Eli Pariser  “The filter bubble: How the new personalized web is changing what we read and how we think”  Penguin, 2011
  * [9] Rashidul Islam, Kamrun Naher Keya, Shimei Pan and James Foulds  “Mitigating demographic biases in social media-based recommender systems”  In _KDD (Social Impact Track)_ , 2019
  * [10] Saumya Bhadani  “Biases in Recommendation System”  In _Proceedings of the 15th ACM Conference on Recommender Systems_ , pp. 855–859
  * [11] Peter Burns  “What TikTok’s Secret Heating Button Reveals About Virality Online”, https://medium.com/feedium/algorithmic-heating-virality-is-a-choice-and-the-game-is-rigged-150307f1032a, 2023
  * [12] Faisal Alatawi et al.  “A survey on echo chambers on social media: Description, detection and mitigation”  In _arXiv preprint arXiv:2112.05084_ , 2021
  * [13] Jonas Kaiser, Adrian Rauchfleisch and Yasodara Córdova  “Comparative approaches to mis/disinformation| fighting zika with honey: An analysis of youtube’s video recommendations on brazilian youtube”  In _International Journal of Communication_ 15, 2021, pp. 19
  * [14] Noah Giansiracusa  “How Algorithms Create and Prevent Fake News: Exploring the Impacts of Social Media, Deepfakes, GPT-3, and More”  Springer, 2021
  * [15] Paddy Leerssen  “The Soap Box as a Black Box: Regulating transparency in social media recommender systems”  In _European Journal of Law and Technology_ 11.2, 2020
  * [16] Zhuoran Liu et al.  “Monolith: Real Time Recommendation System With Collisionless Embedding Table”  In _arXiv preprint arXiv:2209.07663_ , 2022
  * [17] Zhe Zhao et al.  “Recommending what video to watch next: a multitask ranking system”  In _Proceedings of the 13th ACM Conference on Recommender Systems_ , pp. 43–51
  * [18] Akos Lada, Meihong Wang and Tak Yan  “How does news feed predict what you want to see”  In _Personalized ranking with machine learning. Retrieved_ 28, 2021, pp. 2021
  * [19] Twitter  “The Twitter Algorithm: TweepCred”, https://github.com/twitter/the-algorithm/blob/main/src/scala/com/twitter/graph/batch/job/tweepcred/README, 2023
  * [20] Zhiqiang Wang, Qingyun She and Junlin Zhang  “MaskNet: Introducing feature-wise multiplication to CTR ranking models by instance-guided mask”  In _arXiv preprint arXiv:2102.07619_ , 2021
  * [21] Juergen Pfeffer et al.  “Just another day on Twitter: a complete 24 hours of Twitter data”  In _Proceedings of the International AAAI Conference on Web and Social Media_ 17, pp. 1073–1081
  * [22] Alexandre Bovet and Hernán A Makse  “Influence of fake news in Twitter during the 2016 US presidential election”  In _Nature communications_ 10.1, 2019, pp. 7
  * [23] Ramez Kouzy et al.  “Coronavirus goes viral: quantifying the COVID-19 misinformation epidemic on Twitter”  In _Cureus_ 12.3, 2020
  * [24] Hause Lin et al.  “High level of correspondence across different news domain quality rating sets”  In _PNAS Nexus_ , 2023, pp. pgad286
  * [25] Twitter  “Using Deep Learning at Scale in Twitter’s Timelines”, https://blog.twitter.com/engineering/en_us/topics/insights/2017/using-deep-learning-at-scale-in-twitters-timelines, 2017
  * [26] Kathie M d’I Treen, Hywel TP Williams and Saffron J O’Neill  “Online misinformation about climate change”  In _Wiley Interdisciplinary Reviews: Climate Change_ 11.5, 2020, pp. e665
  * [27] J Scott Brennen, Felix M Simon, Philip N Howard and Rasmus Kleis Nielsen  “Types, sources, and claims of COVID-19 misinformation”, 2020
  * [28] Timothy Graham, Axel Bruns, Guangnan Zhu and Rod Campbell  “Like a virus: The coordinated spread of coronavirus disinformation”, 2020
  * [29] Christopher Barrie and Justin Chun-ting Ho  “academictwitteR: an R package to access the Twitter Academic Research Product Track v2 API endpoint”  In _Journal of Open Source Software_ 6.62, 2021, pp. 3272
  * [30] Francesco Pierri et al.  “One year of COVID-19 vaccine misinformation on Twitter”  In _arXiv preprint arXiv:2209.01675_ , 2022
  * [31] Max Falkenberg et al.  “Growing polarization around climate change on social media”  In _Nature Climate Change_ , 2022, pp. 1–8
  * [32] Paul Resnick, Aviv Ovadya and Garlin Gilchrist  “Iffy quotient: A platform health metric for misinformation”  In _Cent Soc Media Responsib_ 17, 2018, pp. 1–20
  * [33] Newsguard  “Score and Rating Levels”, 2023  URL: https://www.newsguardtech.com/ratings/rating-process-criteria/
  * [34] Luke Thorburn, Jonathan Stray and Priyanjana Bengani  “Making Amplification Measurable”, 2023  URL: https://medium.com/understanding-recommenders/making-amplification-measurable-2be548e5986c
  * [35] Bradley Efron and Robert J Tibshirani  “An introduction to the bootstrap”  CRC press, 1994
  * [36] Kwanghee Jung, Jaehoon Lee, Vibhuti Gupta and Gyeongcheol Cho  “Comparison of bootstrap confidence interval methods for GSCA using a Monte Carlo simulation”  In _Frontiers in psychology_ 10, 2019, pp. 2215
  * [37] Arvind Narayanan  “Understanding Social Media Recommendation Algorithms”, 2023
  * [38] James Dougherty, Ron Kohavi and Mehran Sahami  “Supervised and unsupervised discretization of continuous features”  In _Machine learning proceedings 1995_ Elsevier, 1995, pp. 194–202
  * [39] Alyssa Lees et al.  “A new generation of perspective api: Efficient multilingual character-level transformers”  In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , pp. 3197–3207
  * [40] Martin Saveski, Brandon Roy and Deb Roy  “The structure of toxic conversations on Twitter”  In _Proceedings of the Web Conference 2021_ , pp. 1086–1097
  * [41] Lana Cuthbertson et al.  “Women, politics and Twitter: Using machine learning to change the discourse”  In _arXiv preprint arXiv:1911.11025_ , 2019
  * [42] Petter Törnberg  “Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning”  In _arXiv preprint arXiv:2304.06588_ , 2023
  * [43] Fabrizio Gilardi, Meysam Alizadeh and Maël Kubli  “Chatgpt outperforms crowd-workers for text-annotation tasks”  In _arXiv preprint arXiv:2303.15056_ , 2023
  * [44] Thomas J Diciccio and Joseph P Romano  “A review of bootstrap confidence intervals”  In _Journal of the Royal Statistical Society Series B: Statistical Methodology_ 50.3, 1988, pp. 338–354
  * [45] Muhammad Haroon et al.  “YouTube, The Great Radicalizer? Auditing and Mitigating Ideological Biases in YouTube Recommendations”  In _arXiv preprint arXiv:2203.10666_ , 2022
  * [46] Manoel Horta Ribeiro et al.  “Auditing radicalization pathways on YouTube”  In _Proceedings of the 2020 conference on fairness, accountability, and transparency_ , 2020, pp. 131–141
  * [47] William J Brady et al.  “Emotion shapes the diffusion of moralized content in social networks”  In _Proceedings of the National Academy of Sciences_ 114.28, 2017, pp. 7313–7318
  * [48] Aviv Ovadya and Luke Thorburn  “Bridging Systems: Open Problems for Countering Destructive Divisiveness across Ranking, Recommenders, and Governance”  In _arXiv preprint arXiv:2301.09976_ , 2023