# Neuroexplicit Diffusion Models for Inpainting of Optical Flow Fields

Tom Fischer  Pascal Peter  Joachim Weickert  Eddy Ilg

###### Abstract

Deep learning has revolutionized the field of computer vision by introducing
large scale neural networks with millions of parameters. Training these
networks requires massive datasets and leads to intransparent models that can
fail to generalize. At the other extreme, models designed from partial
differential equations (PDEs) embed specialized domain knowledge into
mathematical equations and usually rely on few manually chosen
hyperparameters. This makes them transparent by construction and if designed
and calibrated carefully, they can generalize well to unseen scenarios. In
this paper, we show how to bring model- and data-driven approaches together by
combining the explicit PDE-based approaches with convolutional neural networks
to obtain the best of both worlds. We illustrate a joint architecture for the
task of inpainting optical flow fields and show that the combination of model-
and data-driven modeling leads to an effective architecture. Our model
outperforms both fully explicit and fully data-driven baselines in terms of
reconstruction quality, robustness and amount of required training data.
Averaging the endpoint error across different mask densities, our method
outperforms the explicit baselines by $11-27\%$, the GAN baseline by $47\%$
and the Probabilisitic Diffusion baseline by $42\%$. With that, our method
sets a new state of the art for inpainting of optical flow fields from random
masks.

Deep Learning, Diffusion, Inpainting, Optical Flow, Neuroexplicit

##  1 Introduction

Diffusion is a fundamental process in physics that leads to an equilibrium of
local concentrations. It explains many phenomena and finds applications in
image processing (anisodiffweick) and computer vision tasks
(weickertschnoerr). In particular, it motivates the smoothness term in dense
optical flow estimation in variational techniques. However, recent
advancements in optical flow estimation have been dominated by deep-learning
approaches (flownet; flownet2; raft; gmflow; flowformer). While all these
architectures include a task-specific model-driven operation that represents
the data term, the regularization through the smoothness term is handled in a
fully data-driven manner by the learned parameters of the convolutions
(flownet; flownet2; raft) and more recently by attention (gmflow; flowformer).
Notably, none of these approaches utilize a specialized regularization
operation as it has been studied in traditional computer vision. For this
reason, we investigate whether it is possible to integrate diffusion with its
rich mathematical foundations based on partial differential equations (PDEs)
into neural architectures. In the following, we refer to these model-driven
operations that integrate specialized domain knowledge for a certain task as
_explicit_ , the general data-driven operations as _neural_ , and the
combination of both as _neuroexplicit_.

The role of the regularization in traditional computer vision is to propagate
information from confident correspondences to regions with less or little
information. Variational methods do so using smoothness terms that lead to
diffusion terms in the Euler-Lagrange equations. To isolate this behavior, we
focus on inpainting of sparsely masked optical flow fields and compare our
novel architecture with popular state-of-the-art methods. By imposing the
diffusion behavior explicitly, our goal is to achieve interpretable models
with fewer parameters, improved generalization capabilities, and less
dependence on large-scale datasets.

Figure 1: We present a _neuroexplicit_ architecture for inpainting of optical
flow fields. An _explicit_ inpainting model based on partial differential
equations (PDEs) is enhanced with _neural_ parameter selection. The resulting
model inherits both the exceptional out-of-domain generalization from the
explicit side and the reconstruction quality from the neural side.

###  1.1 Contributions

For the first time, we implement an end-to-end trainable network to predict
the diffusion tensor used in an image driven diffusion inpainting of an
optical flow field. Additionally, it predicts the parameters for the
discretization of WWW, which ensures that the diffusion evolution is stable
and well-posed.

We compare our learned diffusion inpainting network with Edge-Enhancing
Diffusion (eed) inpainting, Absolutely Minimizing Lipschitz (raadOF)
inpainting and Laplace-Beltrami (raadOF) inpainting. Additionally, we consider
the most popular state-of-the-art deep learning methods that use U-Nets
(Unet), Wasserstein GANs (wgain), and Probabilistic Diffusion (imagen;
repaint) and show that our proposed method can reconstruct flow fields with a
high level of detail and generalize exceptionally well. Evaluated with test
data from the same domain as the training data, our method achieves an average
improvement of $48-66\%$ in terms of endpoint error when compared to the
baselines, while when tested on a new domain, our method manages to outperform
by $11-47\%$ and sets a new state of the art. Finally, we evaluate on real
world data from autonomous driving and show that in this practical
application, our method is on-par with other methods or significantly
outperforms them. Beyond the good generalization capabilities, the diffusion
networks have comparatively few learnable parameters and competitive inference
times. Our ablation studies show that they can be trained with much less data
and still outperform baselines trained on the full dataset.

###  1.2 Related Work

Diffusion inpainting. Reconstructing missing information from images, known as
inpainting (GL14), has been a long-standing goal in image processing (MM98a;
BSCB00). For inpainting-based image compression (GWWB08), diffusion processes
offer very good performance. They are theoretically well-founded
(anisodiffweick), and their discretizations are well-understood (WWW).
Moreover, they are inherently explainable and can reconstruct high resolution
images in real time (realTimeHom).

Inpainting with deep learning. In recent years, the advances in deep learning
methods have shifted the attention towards large-scale data-driven models.
Generative Adversarial Networks (GANs) (wgain) or Probabilistic Diffusion (PD)
models (repaint) show impressive inpainting qualities for image restoration
and artistic purposes. They do, however, require large amounts of training
data and can fail to generalize to out-of-distribution scenarios.

Inpainting of flow fields. Inpainting for optical flow fields has rarely been
addressed. JPW20 investigated PDE-based inpainting for compression of general
piecewise smooth vector fields. APMW21 use flow field compression and
inpainting of optical flow fields as part of their video compression codec.
However, both works assume having access to the complete flow to optimize the
inpainting mask accordingly, whereas our method works with random, non-optimal
masks.

Relationships between PDE-models and deep learning. A variety of other
neuroexplicit approaches have been explored. Researchers recently have turned
to investigating connections between discrete models for solving PDEs and deep
learning (DiffBlockConn; ruthotto1; ruthotto2; neuralode). CNN architectures
share a particularly close relationship to discrete PDE models due to the
inherent similarity of convolutions and discrete derivatives in the form of
finite differences (differences).

DiffBlockConn and ruthotto2 connected discrete models for solving PDEs and
residual blocks (resnet). Their diffusion blocks realize one explicit step of
a discrete diffusion evolution in a residual block with symmetric filter
structure. Similar to our approach, they construct architectures that realize
diffusion evolutions. However, their work only involves the formulation as a
neural network for executing the method (DiffBlockConn), or the focus revolved
around learning the finite differences (ruthotto2).

Most closely related to our method are the works of AW21 and chenpock that
focused on parameterizing diffusion processes through learning. Both methods
use learning to estimate contrast parameters for the diffusivity, but
formulate the diffusion tensor as explicit functions of image contrast. AW21
construct a multiscale anisotropic diffusion process for image denoising.
chenpock formulate a general framework for diffusion-reaction systems that
support learnable contrast parameters in an isotropic diffusion process and
learnable weights for the finite difference operators. However, once trained,
these parameters are the same for all pixels and do not adapt themselves to
the presented input image content. In contrast, our model learns to drive the
explicit diffusion process in a fully neural way and does not rely on first
order image derivatives as an edge detector.

##  2 Inpainting with Explicit Diffusion

In this section, we review diffusion (eed) and how it can be used for
inpainting.

###  2.1 Definition of Diffusion Inpainting

A given vector valued image $\bm{f}(\bm{x}):\Omega\rightarrow\mathbb{R}^{c}$
is only known on the subset $\Omega_{C}\subset\Omega$ of the rectangular image
domain $\Omega\subset\mathbb{R}^{2}$. For each channel $i\in\\{1,...,c\\}$,
diffusion results in the steady state approached for $t\rightarrow\infty$ of
the initial boundary value problem:

$\displaystyle\begin{split}\partial_{t}u_{i}(\bm{x},t)&=\text{div}(\bm{D}\bm{\nabla}u_{i}(\bm{x},t)),\\\
&\hskip 63.00012pt\text{for
}\bm{x}\in\Omega\setminus\Omega_{C}\times(0,\infty),\end{split}$ (1)
$\displaystyle u_{i}(\bm{x},t)$ $\displaystyle=f_{i}(\bm{x}),\hskip
22.50003pt\text{for }\bm{x}\in\Omega_{C}\times[0,\infty),$ (2) $\displaystyle
u_{i}(\bm{x},0)$ $\displaystyle=0,\hskip 40.00006pt\text{for
}\Omega\setminus\Omega_{C},$ (3)
$\displaystyle\partial_{\bm{n}}u_{i}(\bm{x},t)$
$\displaystyle=0,\qquad\qquad\text{for
}\bm{x}\in\partial\Omega\times[0,\infty).$ (4)

Here, $u_{i}(\bm{x},\infty$) denotes the final inpainting result in channel
$i$, $\text{div}=\bm{\nabla}^{\top}$ denotes the spatial divergence operator,
and $\partial_{\bm{n}}$ represents the directional derivative along the normal
vector to the image boundary $\partial\Omega$. The diffusion tensor $\bm{D}$
is a $2{\mkern-2.0mu\times\mkern-2.0mu}2$ positive definite symmetric matrix
that describes the propagation behavior.

###  2.2 Edge-Enhancing Diffusion

Edge-Enhancing Diffusion (EED) (eed) provides superior inpainting quality
(eedSuperior), achieved by deriving the diffusion tensor $\bm{D}$ through the
structure tensor (Di86):

$\bm{S}(u):=\sum_{i=1}^{c}\bm{\nabla}u_{i,\rho}\bm{\nabla}u_{i,\rho}^{\top}\,.$
(5)

Here, $u_{i,\rho}$ denotes a convolution of channel $u_{i}$ with a Gaussian of
standard deviation $\rho$. The eigenvalues $\mu_{1}\geq\mu_{2}\geq 0$ of
$\bm{S}$ measure the local contrast along the corresponding eigenvectors
$\bm{v}_{1},\bm{v}_{2}$. EED penalises smoothing across image structures by
transforming the larger eigenvalue with a positive, decreasing diffusivity
function $g$. For the remaining direction along image structures, full
diffusion is allowed by setting the eigenvalue to $1$. This results in

$\bm{D}:=g(\bm{S})=g(\mu_{1})\cdot\bm{v}_{1}\bm{v}_{1}^{\top}+1\cdot\bm{v}_{2}\bm{v}_{2}^{\top}\,.$
(6)

In our setting, $\bm{f}$ is a sparse flow field that should be inpainted. So
far, we illustrated a nonlinear diffusion process where the structure tensor
from Equation 5 and consequently the diffusion tensor are determined from the
evolving signal. This has the disadvantage that the diffusion tensor needs to
be re-estimated for every diffusion step. For this reason, we choose a linear
diffusion process, and determine the diffusion tensor using the image
$\bm{I}(\bm{x}):\Omega\rightarrow\mathbb{R}^{3}$ relative to which the optical
flow field is defined. We will refer to $\bm{I}$ as the reference image.

###  2.3 Discretization

Equation 1 can be discretized by means of a finite difference scheme. To
transform the continuous into discrete signals, we sample $\bm{u,f}$ and
$\bm{I}$ at grid sizes $h_{x},h_{y}$. We discretize the temporal derivative by
a forward difference with time step size $\tau$. The spatial first-order
derivative operator $\bm{\nabla}$ and its adjoint $\bm{\nabla}^{\top}$ are
implemented by a convolution matrix $\bm{K}$ and its negated transpose
$-\bm{K}^{\top}$, respectively.

To discretize computation and multiplication with the diffusion tensor
$\bm{D}$, we introduce the following notation for an activation function:

$\Phi(\bm{I,~{}Ku}^{k})=g\Bigl{(}\underbrace{\sum_{i=0}^{c}(\bm{KI})_{i}(\bm{KI})_{i}^{\top}}_{\bm{S}}\Bigr{)}(\bm{Ku^{k}}).$
(7)

Finally, we can define the discrete diffusion evolution and solve it for the
next time step to obtain an explicit scheme:

$\displaystyle\frac{\bm{u}^{k+1}-\bm{u}^{k}}{\tau}=-\bm{K}^{\top}\Phi(\bm{I},~{}\bm{Ku}^{k})\,\,\,\,\Leftrightarrow$
(8)
$\displaystyle\bm{u}^{k+1}=\bm{u}^{k}-\tau(\bm{K}^{\top}\Phi(\bm{I},~{}\bm{K}\bm{u}^{k})),$

where the time levels are indicated by superscripts.

To achieve an inpainting process with good reconstruction qualities, the
choice of the convolution matrix $\bm{K}$ is crucial. WWW introduced a
nonstandard finite difference discretization that implements the discrete
divergence term $\bm{K}^{\top}\Phi(\bm{I,~{}K}\bm{u}^{k})$ on a
$3{\mkern-2.0mu\times\mkern-2.0mu}3$ stencil. It introduces two free
parameters

$\alpha\in[0,\tfrac{1}{2}],~{}~{}|\beta|\leq 1-2\alpha,$ (9)

that have an impact on sharpness and rotation invariance of the
discretization.

###  2.4 Fast-Semi Iterative Scheme

In practice, enforcing the stability of an explicit scheme requires
restricting the time step $\tau$. However, depending on the data, a stable
scheme may require a substantial amount of iterations to converge. Formulated
as a neural architecture, this results in an excessive amount of operations
and intractable optimization. Fortunately, explicit schemes can be accelerated
by extrapolating the outcome of each time step, e.g. by the Fast Semi-
iterative (FSI) Scheme proposed by FSI. In contrast to a naive discretization,
FSI-schemes implement a cycle of $L$ steps and extrapolate the diffusion
result at a fractional time step $k+\frac{l-1}{L}$:

$\displaystyle\bm{u}^{k+\frac{l+1}{L}}=$
$\displaystyle\gamma_{l}(\bm{u}^{k+\frac{l}{L}}+\tau(-\bm{K}^{\top}(\Phi(\bm{I},~{}\bm{K}\bm{u}^{k+\frac{l}{L}})))$
(10) $\displaystyle+(1-\gamma_{l})\bm{u}^{k+\frac{l-1}{L}}$

with $l=0,...,L-1$ indexing the step and $\gamma_{l}:=(4l+2)/(2l+3)$ denoting
the time-varying extrapolation weights.

Figure 2: Our proposed hybrid inpainting model. The Diffusion Tensor Module
takes the reference image as input, and outputs a specific diffusion tensor
$\bm{D}$ and discretization parameter $\alpha$ for every stage of the coarse-
to-fine inpainting pipeline. The inpainting itself is done using a stable and
well-posed anisotropic diffusion process that solves $t$ steps of the explicit
scheme in Equation 8.

##  3 From Explicit to Neuroexplicit Flow Inpainting

In the following paragraphs, we discuss how to transform an explicit diffusion
inpainting approach into a hybrid neuroexplicit architecture, where we define
explicit as parts of the architecture that are derived from the well known PDE
framework for the diffusion process, and neural as generic data-driven, non-
interpretable deep learning architecture parts.

To solve the inpainting task, our method and the baselines receive a sparse,
initial flow field as well as a binary mask that marks the positions of the
known flow vectors. Additionally, we provide the reference image for an image-
driven inpainting process. Image-driven regularizers in traditional optical
flow methods exploit the correlation of contrast in the reference image and
the discontinuities in the unknown flow field. However, considering contrast
alone is not sufficient and leads to over-segmented flow fields. In practice,
due to the aperture problem, one must decide how to regularize based on the
individual image content. As this is of statistical nature and requires prior
knowledge, leveraging deep learning here seems adequate. We bring in a U-Net
(Unet) as the Diffusion Tensor Module (DTM), that we train end-to-end to
predict the ideal diffusion parameters. Concretely, it replaces the heuristic
choice of the structure tensor in the activation $\Phi$ in Equation 7 and the
discretization parameter $\alpha$ in Equation 9. An overview of the complete
model is shown in Figure 2.

###  3.1 Coarse-to-Fine Diffusion Inpainting

Our architecture implements an explicit image-driven inpainting process. In
contrast to traditional methods, the parameters of the process are obtained
from the reference image using the DTM. To reduce the required time steps of
the inpainting process and make it computationally feasible, we employ a
coarse-to-fine scheme (BD96). We build the pyramid using pooling operations
that downsample by a factor $2$, where we use average pooling for the
reference image and max pooling for the mask. To obtain the coarse version of
the sparse flow field, we use average pooling of known flow values. After
obtaining the coarse versions of all inputs, we start the diffusion inpainting
process from the coarsest sparse flow field. The inpainted flow field is then
upsampled using bilinear interpolation and initializes the inpainting process
at the next finer resolution.

For a multiscale diffusion process that spans across $N$ resolutions, we need
a set of parameters at each scale. The DTM performs $N$ down- and upsampling
convolutions. At each feature map after the bottleneck, we apply a separate
convolution to estimate a feature map $\bm{z}$ with five channels. Below, we
explain how each $\bm{z}$ is transformed to parameterize the diffusion process
along the coarse-to-fine pyramid.

###  3.2 Discretization

To implement our scheme, we use the discretization of WWW that we discussed in
Section 2.3. This formulation introduces the two free parameters $\alpha$ and
$\beta$ shown in Equation 9. The first channel of $\bm{z}$ is used as the
discretization parameter $\alpha=\sigma(z_{0})/2$. Notably, the restriction of
$|\beta|\leq 1-2\alpha$ depends on $\alpha$. To guarantee a stable scheme, we
choose $\beta=(1-2\alpha)\text{sign}(b)$, where $b$ is the off-diagonal
element of the diffusion tensor.

###  3.3 Learning the Diffusion Tensor

The remaining four channels in $\bm{z}$ are used to estimate the diffusion
tensor in the activation $\Phi$. Replacing the structure tensor with a neural
edge detector allows learning a prior that decides which image edges will
likely coincide with flow discontinuities. Due to the anisotropic diffusion
process, these discontinuities can be maintained throughout the inpainting
process even if the mask distribution is sparse and suboptimal.

Concretely, we obtain two eigenvalues $\mu_{1}=g(z_{1})$, $\mu_{2}=g(z_{2})$
and one eigenvector
$\bm{v}_{1}=\frac{(z_{3},z_{4})^{\top}}{\lVert(z_{3},z_{4})^{\top}\rVert_{2}}$.
We explicitly compute
$\bm{v}_{2}=\frac{(-z_{4},z_{3})^{\top}}{\lVert(z_{3},z_{4})^{\top}\rVert_{2}}$
to ensure orthogonality to $\bm{v}_{1}$. The eigenvalues are constrained to
the range $[0,1]$ using the Perona-Malik diffusivity
$g(x)=(1+\frac{x^{2}}{\lambda^{2}})^{-1}$ (peronamalik), where the free
parameter $\lambda$ is learned during training. In contrast to the formulation
in Equation 6, we apply the diffusivity to both eigenvalues. This gives the
DTM additional freedom to either replicate the behavior in Equation 6 or
restrict the diffusive flux in both directions.

##  4 Experiments

Our experiments are divided into three parts. In the first part, we compare
our method with a selection of fully explicit and neural inpainting methods.
In the second part, we show an ablation study to test the effectiveness of
learning different components of the diffusion inpainting process to analyze
and determine the balance between explicit and neural parameter selection. For
both parts, we train on the final subset of the FlyingThings dataset
(flyingthings). To evaluate generalization, we test on the Sintel dataset
(Sintel).

Finally, we demonstrate the usefulness of our method in a real-world
application. The KITTI (kitti) dataset is well known for autonomous driving
and provides sparse ground truth that is acquired from registering LiDAR
scans. Densifying it presents a practically highly relevant use case of our
method.

###  4.1 Network and Training Details

For our diffusion inpainting network, we leverage four resolutions for the
coarse-to-fine pyramid. Going from coarsest to finest, we perform $i$
iterations, where $i\in\\{5,15,30,45\\}$ increases with the resolution. At
each resolution, the feature map $\bm{z}$ is obtained from the DTM and
transformed into the parameters as described in Section 3. Each resolution has
access to a separate contrast parameter $\lambda$ in the Perona-Malik
diffusivity discussed in Section 3.3, which is initialized as $1$ and learned
during training. To further speed up the inpainting process, we use the Fast-
Semi Iterative (FSI) scheme proposed by FSI and perform one cycle per
resolution. Time step size and FSI extrapolation weights are chosen to satisfy
a stable and well-posed diffusion inpainting process. For more information
about the FSI scheme, we refer to the supplementary material.

We choose three fully explicit baselines. First, a linear EED inpainting
method where all parameters are chosen explicitly and the diffusion tensor is
also estimated using the reference image. For a fair comparison, we use the
same coarse-to-fine strategy and optimize its hyperparameters on a subset of
the training data and let the inpainting process converge. The previous state
of the art is held by raadOF, who propsoe two anisotropic optical flow
inpainting algorithms: the first is based on the Absolutely Minimizing
Lipschitz Extension (AMLE) PDE and the second one uses the Laplace-Beltrami
(LB) operator. They propose a set of robust hyperparameters for the Sintel
dataset, which we will use for all evaluations. Note that other methods are
not trained or tuned on Sintel and therefore this setting gives raadOF an
advantage.

As the first neural baseline, we choose a FlowNetS (flownet) as a general
purpose U-Net architecture. Instead of two images, we feed it a concatenation
of image, mask, and sparse flow and let the network learn the inpainting
process. As more recent and advanced deep learning-based methods, we include
Generative Adversarial Networks (GANs) and Probabilistic Diffusion (PD). We
use WGAIN (wgain) as the GAN baseline, as it has been used successfully for
inpainting images from sparse masks. For the PD network, we adapted the
popular efficient U-Net architecture (imagen) and use the inpainting
formulation of RePaint (repaint) during inference. Both GAN and PD network are
conditioned on the reference image to learn the correlation between flow and
image edges. For more details on training and adaptations, we refer to the
supplemental material.

$10^{2.29}$$10^{3.29}$$10^{3.99}$$10^{4.29}$$1$$2$$3$Training SamplesSintel
Endpoint ErrorFlowNetSWGAINPDEEDOurs

$2\%$$4\%$$6\%$$8\%$$10\%$$0$$1$$2$$3$Mask DensitySintel Endpoint
ErrorFlowNetSWGAINPDEEDOurs

Figure 3: Our proposed method is robust to changes in the training and
inference setting. The left plot shows the weaker reliance of our method on
training data. Using $194$ samples, we reach a competitive performance to the
network trained on the full dataset. The right plot shows the favorable
generalization to unseen mask densities of our method and the explicit EED
inpainting. As indicated by the gray vertical line, we evaluated each model
optimized for $5\%$ on previously unseen mask densities.

OursFlowNetSWGAINPD$10^{6}$$10^{7}$$10^{8}$$10^{9}$Models# of
ParametersOursFlowNetSWGAINPD$10^{1}$$10^{2}$$10^{3}$$10^{4}$$10^{5}$Inference
time in ms

Figure 4: Weights and inference time of the models. Compared to the baselines,
our model is very lightweight and has competitive inference times. Notably, we
omitted the explicit baselines, since there is no clear way to compare the
methods.

###  4.2 Reconstruction and Generalization

Table 1: Comparison of our method with the baselines on the Sintel dataset. Surprisingly, the explicit EED diffusion inpainting outperforms the data-driven baselines across both datasets. Learning the proposed parts of the diffusion process with our method further improves the reconstruction qualities with significantly fewer iterations and leads to a new state of the art. ∗indicates that the method was already tuned for Sintel. | Training-Domain Test EPE
---|---
|  EED |  AMLE* |  LB* |  Ours |  FlowNetS |  WGAIN |  PD
1% |  $2.06$ |  $2.03$ |  $2.03$ |  $\mathbf{1.01}$ |  $2.33$ |  $2.26$ |  $3.96$
5% |  $1.00$ |  $1.08$ |  $1.00$ |  $\mathbf{0.55}$ |  $1.68$ |  $1.73$ |  $1.09$
10% |  $0.73$ |  $0.82$ |  $0.75$ |  $\mathbf{0.39}$ |  $1.55$ |  $1.43$ |  $0.72$
| Sintel Test EPE
|  EED |  AMLE* |  LB* |  Ours |  FlowNetS |  WGAIN |  PD
1% |  $0.94$ |  $0.94$ |  $0.86$ |  $\mathbf{0.72}$ |  $0.85$ |  $1.14$ |  $2.39$
5% |  $0.52$ |  $0.51$ |  $0.43$ |  $\mathbf{0.40}$ |  $0.57$ |  $0.80$ |  $0.55$
10% |  $0.43$ |  $0.38$ |  $0.31$ |  $\mathbf{0.28}$ |  $0.51$ |  $0.60$ |  $0.40$

Table 1 shows a comparison of our method to the introduced baselines. Edge-
Enhancing Diffusion (EED) inpainting outperforms the purely neural baselines
and can reconstruct a high level of detail. It does, however, require a
significant number of iterations to converge (anywhere from $3{,}000$ to
$100{,}000$), varying drastically with the content of the images and the given
mask density. The reason for this slow convergence is the reliance on the
structure tensor that we discussed in Section 2.2. The diffusivity is limiting
the diffusive flux wherever there is image contrast, which increases the
number of required time steps. Figure 5 shows that relying on the structure
tensor can also be harmful in low-contrast regions where no edge is identified
and information leaks across edges. Since we let the inpainting process fully
converge, this leaking effect can be detrimental to the final performance.
Furthermore, replacing the structure tensor with a neural edge detector leads
to a more robust inpainting in cases where there is a high variance of
contrast within the images, which happens frequently in the FlyingThings data.
Consequently, the discrepancy between EED and our method is much more severe
on that dataset.

Figure 5: Samples generated with a mask density of $5\%$. Every other row
displays the zoomed in area of the red rectangle in the row above. Our method
manages to retain a much higher level of detail in the reconstructed flow
fields. In the bottom row at the dragons chin, we can observe that the PDE
methods (EED, AMLE, and LB) fail to maintain flow edges in low contrast
regions. Notably, both WGAIN and the PD model have poor out-of-distribution
performance. WGAIN tends to have large outliers and fails in the zero flow in
the background. The PD model fails to reproduce the fuzzy material of the
shamans beard due to a lack of comparable materials in the training data.

As can be seen in Figure 5, the CNN methods FlowNetS (flownet) and WGAIN
(wgain) produce noisy flow fields at the flow edges. They fail to capture the
same level of detail as the anisotropic diffusion methods that adapt their
smoothing behavior to the image content. Compared to the CNN methods, the
Probabilistic Diffusion (PD) model with the RePaint (repaint) inpainting has
well-localized discontinuities. However, PD models are highly affected by the
distribution of the training data. Figure 5 shows a case of overfitting in the
first two rows. The FlyingThings dataset contains mostly rigid objects with
straight edges and no materials comparable to the fuzzy beard of the shaman.
Consequently, PD fails to generalize to the out-of-domain sample and
reconstructs a blocky and unnatural looking beard.

Since our method realizes a well-posed diffusion process by construction, it
is naturally robust to changes in its input. We tested the generalization
capabilities to new mask densities and show the results in Figure 4.
Increasing the mask density compared to observed training density should lead
to an increase in performance since more information is presented. The data-
driven baselines that are trained with a specific density (FlowNetS and WGAIN)
fail to capture this intuition and have decreasing performance. Our proposed
method has an increasingly better reconstruction quality with higher
densities. When evaluated on a density of $10\%$, the network trained on $5\%$
density can even reach a very close EPE on to the network that was optimized
on this density ($0.28$ vs. $0.29$).

Figure 4 shows the number of learned parameters of all models. Since the
inpainting behavior in our method is steered by an explicit anisotropic
diffusion process, the network has significantly fewer parameters than the
compared baselines. A vast majority of these parameters are placed in the DTM
to identify flow discontinuities and drive the diffusion process accordingly.
Having so few parameters provides an inherent regularizing effect and leads to
less reliance on available training data. This is reflected in the left plot
in Figure 4, where we compared the performance of all baselines when trained
on a subset of the available training data. Even with a drastic cut of
training data, our proposed method outperforms all other baselines. The
reconstruction quality barely decreases compared to the networks that were
trained on more data.

Table 2: Ablation study of replacing learned with explicit components. The
values indicate endpoint errors relative to our final network for different
mask densities. Learning the eigenvalues over explicit ones ($-\mu$) plays the
biggest role. Exchanging learned with explicit eigenvectors ($-\bm{v}$) and
the learned, spatially-varying discretization parameter with a constant
explicit one ($-\alpha$) leads to consistent decreases in performance.
Learning the second discretization parameter ($+\beta$) can further push the
performance, but does not guarantee stability. We also test the ResNet
implementation of DiffBlockConn and learn the finite difference operators
($+\bm{W}$). However, this does not give a significant performance improvement
and leads to additional computation cost.

| Training-Domain Test EPE | Sintel Test EPE
---|---|---
|  Full |  $-\mu$ |  $-\bm{v}$ |  $-\alpha$ |  $+\beta$ |  $+\bm{W}$ |  Full |  $-\mu$ |  $-\bm{v}$ |  $-\alpha$ |  $+\beta$ |  $+\bm{W}$
01% |  $1.01$ |  $+0.95$ |  $+0.07$ |  $+0.05$ |  $-0.04$ |  $+0.00$ |  $0.72$ |  $+0.17$ |  $+0.07$ |  $+0.03$ |  $-0.02$ |  $+0.01$
05% |  $0.55$ |  $+0.28$ |  $+0.02$ |  $+0.06$ |  $-0.03$ |  $-0.01$ |  $0.40$ |  $+0.04$ |  $+0.06$ |  $+0.02$ |  $-0.01$ |  $-0.01$
10% |  $0.39$ |  $+0.20$ |  $+0.00$ |  $+0.02$ |  $-0.01$ |  $+0.03$ |  $0.28$ |  $+0.04$ |  $+0.04$ |  $+0.02$ |  $-0.01$ |  $+0.00$

###  4.3 Effects of Learned Components

Table 2 shows quantitative results of trained inpainting networks, where we
illustrate the effect of learning the eigenvalues $\mu_{1},\mu_{2}$,
eigenvectors $\bm{v}_{1},\bm{v}_{2}$, and the discretization parameter
$\alpha$. As one can see from Equation 5, the structure tensor computation
considers only first-order image derivatives. Especially in the presence of
noisy images, the structure tensor relies on Gaussian pre-smoothing that can
lead to worse edge localization in the final flow field. In the case of
explicit eigenvalues, the network has to learn a small contrast parameter
$\lambda$ to avoid smoothing across structures. This leads to a slow
convergence within the limited number of performed diffusion steps and poor
inpainting quality. Supplying the eigenvectors of the diffusion tensor by the
learnable module provides a consistent increase in performance.

The discretization parameters are usually chosen as constant hyperparameters,
whereas our DTM outputs one $\alpha$-value per pixel and outperforms the
global constant for all mask densities. This suggests that adapting the
discretization parameters to the image content is preferable to obtain high-
quality reconstructions. We also performed an additional ablation study to
estimate the second discretization parameter $\beta$ independently of
$\alpha$. The improvements are insignificant, while doing so voids the
restriction 9. Hence, we do not recommend learning the second discretization
parameter $\beta$.

DiffBlockConn showed that for each discrete diffusion evolution with a fixed
stopping time, there is an equivalent ResNet architecture that implements it.
We additionally compare our method to the ResNet formulation of DiffBlockConn
where we learned the discrete derivative operators. This formulation requires
much more arithmetic operations compared to the efficient
$3{\mkern-2.0mu\times\mkern-2.0mu}3$-stencil of WWW, since each derivative
operator has to be realized with its own convolution kernel. On the other
hand, also in this case, the performance difference is insignificant and the
explicit discretization from WWW is preferable. For the derivation of the
equivalent ResNet architecture, we refer the reader to the supplementary
material.

###  4.4 Generalization to Real World Data

We further tested the generalization capabilities of all methods to real world
data using the KITTI2015 (kitti) optical flow dataset. This dataset provides
accurate sparse measurements obtained from registering LiDAR scans. Densifying
these measurements resembles a highly relevant application of our method for
autonomous driving.

The original density of the measurements is between $15-25\%$. To measure the
accuracy of our reconstructed dense flow fields, we subsample according to our
different density settings $1$, $5$ and $10\%$. After reconstructing the dense
flow field, we then measure the accuracy of the previously left out
measurements and report the results in Table 3. Please note that we omit the
Probabilistic Diffusion method, since it is optimized for a specific image
resolution and fails to generalize from the square training images to the
wide-angle setting in KITTI.

When looking at the numbers, the advantage in terms of robustness of all the
PDE-based, neural and neuroexplicit methods becomes apparent. The neural
models fail to generalize to this new real-world setting, as well as non-
uniform mask distributions. This is consistent with the observations from
Figure 4. The results show, that our method which combines neural and explicit
components is on-par with Laplace-Beltrami in terms of EPE, but has
significantly fewer outliers especially in the most challenging low density
setting. Most likely, our method could still be improved by supplying
different non-uniform mask distributions during training to adapt to the
setting in KITTI.

Table 3: Generalization Capabilities to Real World Data. We report the mean EPE on all measured pixels in the original KITTI training dataset. As customary for KITTI, we additionally report flow (Fl) outliers in $\%$ as defined by kitti. A displacement is considered an outlier if its endpoint error is $>3$, or it differs by at least $5\%$ of the ground truth displacement. The results show that our method is on par with Laplace-Beltrami in terms of EPE but has significantly fewer outliers in the most challenging low density setting.  | KITTI EPE
---|---
|  EED |  AMLE |  LB |  Ours |  FlowNetS |  WGAIN
1% |  $1.11$ |  $1.26$ |  $\mathbf{1.07}$ |  $\mathbf{1.07}$ |  $1.43$ |  $3.18$
5% |  $0.46$ |  $0.56$ |  $\mathbf{0.46}$ |  $0.47$ |  $2.53$ |  $7.0$
10% |  $0.23$ |  $0.30$ |  $\mathbf{0.23}$ |  $\mathbf{0.23}$ |  $4.96$ |  $6.82$
| KITTI Fl
|  EED |  AMLE |  LB |  Ours |  FlowNetS |  WGAIN
1% |  $1.14$ |  $1.19$ |  $0.94$ |  $\mathbf{0.87}$ |  $1.2$ |  $3.59$
5% |  $0.27$ |  $0.35$ |  $0.26$ |  $\mathbf{0.25}$ |  $2.26$ |  $4.48$
10% |  $\mathbf{0.11}$ |  $0.16$ |  $\mathbf{0.11}$ |  $\mathbf{0.11}$ |  $3.77$ |  $4.39$

##  5 Conclusion and Future Work

We studied discrete diffusion processes in a deep learning context and
illustrated a novel approach to steer the diffusion process by a deep network.
We showed that our method can outperform both model-based and fully data-
driven baselines, while requiring less training data, having fewer parameters,
generalizing better, being well-posed, being supported by a stability
guarantee, and offering competitive runtimes.

In the current work, we only focus on the regularization aspect of optical
flow. Future work will be on embedding our diffusion regularization into an
end-to-end optical flow algorithm. We expect that this will yield more
interpretable methods with the benefits of stability guarantees, better
generalization and requiring less training data.

## Impact Statement

In this paper, we show how the benefits of a well-studied explicit paradigm
can be combined with data-driven deep learning. We call this combination
neuroexplicit, as it is interpretable by design and shows significantly better
generalization abilities, while requiring much fewer parameters and training
data. Our results demonstrate that for the task of optical flow inpainting,
such a neuroexplicit approach is able to outperform both, purely explicit and
state-of-the-art data-driven methods by a large margin, and thus is really
able to leverage the strengths of both worlds. Since data-driven models today
dominate many areas without being interpretable, and explicit models are well
understood, integrating more explicit components into state-of-the-art models
opens up a path to increase trustworthiness and reliability, while requiring
less training data. Our work motivates to explore further neuroexplicit models
in the future.

## Acknowledgements

We gratefully acknowledge the stimulating research environment of the GRK
2853/1 “Neuroexplicit Models of Language, Vision, and Action”, funded by the
Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under
project number 471607914.

## References

  * Alt & Weickert (2021) Alt, T. and Weickert, J.  Learning integrodifferential models for denoising.  In _Proc. 2021 IEEE International Conference on Acoustics, Speech and Signal Processing_ , pp. 2045–2049, Toronto, Canada, June 2021. IEEE Computer Society Press.
  * Alt et al. (2022) Alt, T., Schrader, K., Augustin, M., Peter, P., and Weickert, J.  Connections between numerical algorithms for pdes and neural networks.  _Journal of Mathematical Imaging and Vision_ , 65(1):185–208, jun 2022.  ISSN 0924-9907.
  * Andris et al. (2021) Andris, S., Peter, P., Mohideen, R., Weickert, J., and Hoffmann, S.  Inpainting-based video compression in FullHD.  In Elmoataz, A., Fadili, J., Quéau, Y., Rabin, J., and Simon, L. (eds.), _Scale Space and Variational Methods in Computer Vision_ , volume 12679 of _Lecture Notes in Computer Science_ , pp. 425–436. Springer, Cham, 2021.
  * Arjovsky et al. (2017) Arjovsky, M., Chintala, S., and Bottou, L.  Wasserstein generative adversarial networks.  In _International Conference on Machine Learning_ , pp. 214–223. PMLR, 2017.
  * Ba et al. (2016) Ba, J. L., Kiros, J. R., and Hinton, G. E.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * Bertalmío et al. (2000) Bertalmío, M., Sapiro, G., Caselles, V., and Ballester, C.  Image inpainting.  In _Proc. SIGGRAPH 2000_ , pp. 417–424, New Orleans, LI, July 2000.
  * Bornemann & Deuflhard (1996) Bornemann, F. and Deuflhard, P.  The cascadic multigrid method for elliptic problems.  _Numerische Mathematik_ , 75:135–152, 1996.
  * Butler et al. (2012) Butler, D. J., Wulff, J., Stanley, G. B., and Black, M. J.  A naturalistic open source movie for optical flow evaluation.  In _Proceedings of the 12th European Conference on Computer Vision - Volume Part VI_ , ECCV’12, pp. 611–625, Berlin, Heidelberg, 2012. Springer-Verlag.  ISBN 9783642337826.
  * Chen et al. (2018) Chen, R. T. Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D.  Neural ordinary differential equations.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , NIPS’18, pp. 6572–6583, Red Hook, NY, USA, 2018. Curran Associates Inc.
  * Chen & Pock (2017) Chen, Y. and Pock, T.  Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 39(6):1256–1272, jun 2017.  ISSN 0162-8828.
  * Di Zenzo (1986) Di Zenzo, S.  A note on the gradient of a multi-image.  _Computer Vision, Graphics and Image Processing_ , 33:116–125, 1986.
  * Dosovitskiy et al. (2015) Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. v. d., Cremers, D., and Brox, T.  Flownet: Learning optical flow with convolutional networks.  In _Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV)_ , ICCV ’15, pp. 2758–2766, USA, 2015. IEEE Computer Society.  ISBN 9781467383912.
  * Galić et al. (2008) Galić, I., Weickert, J., Welk, M., Bruhn, A., Belyaev, A., and Seidel, H.-P.  Image compression with anisotropic diffusion.  _Journal of Mathematical Imaging and Vision_ , 31(2–3):255–269, July 2008.
  * Geiger et al. (2012) Geiger, A., Lenz, P., and Urtasun, R.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In _2012 IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 3354–3361. IEEE, 2012.
  * Guillemot & Le Meur (2014) Guillemot, C. and Le Meur, O.  Image inpainting: Overview and recent advances.  _IEEE Signal Processing Magazine_ , 31(1):127–144, 2014.
  * Gulrajani et al. (2017) Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C.  Improved training of wasserstein gans.  _Advances in Neural Information Processing Systems_ , 30, 2017.
  * Haber & Ruthotto (2017) Haber, E. and Ruthotto, L.  Stable architectures for deep neural networks.  _Inverse Problems_ , 34(1), 2017.
  * Hafner et al. (2016) Hafner, D., Ochs, P., Weickert, J., Reißel, M., and Grewenig, S.  Fsi schemes: Fast semi-iterative solvers for PDEs and optimisation methods.  In _Pattern Recognition: 38th German Conference, GCPR 2016, Hannover, Germany, September 12-15, 2016, Proceedings 38_ , pp. 91–102. Springer, 2016.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * Huang et al. (2022) Huang, Z., Shi, X., Zhang, C., Wang, Q., Cheung, K. C., Qin, H., Dai, J., and Li, H.  Flowformer: A transformer architecture for optical flow.  In _Proceedings of the 17th European Conference on Computer Vision – Volume Part XVII_ , pp. 668–685, Berlin, Heidelberg, October 2022. Springer-Verlag.  ISBN 978-3-031-19789-5.
  * Ilg et al. (2017) Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., and Brox, T.  Flownet 2.0: Evolution of optical flow estimation with deep networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 2462–2470, 2017.
  * Jost et al. (2020) Jost, F., Peter, P., and Weickert, J.  Compressing flow fields with edge-aware homogeneous diffusion inpainting.  In _Proc. 45th International Conference on Acoustics, Speech, and Signal Processing_ , pp. 2198–2202, Barcelona, Spain, May 2020. IEEE Computer Society Press.
  * Kämper & Weickert (2022) Kämper, N. and Weickert, J.  Domain decomposition algorithms for real-time homogeneous diffusion inpainting in 4k.  In _Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 1680–1684. IEEE, 2022.
  * Karras et al. (2022) Karras, T., Aittala, M., Aila, T., and Laine, S.  Elucidating the design space of diffusion-based generative models.  _Advances in Neural Information Processing Systems_ , 35:26565–26577, 2022.
  * Kingma & Ba (2015) Kingma, D. and Ba, J.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations (ICLR)_ , San Diega, CA, USA, 2015.
  * Krizhevsky et al. (2012) Krizhevsky, A., Sutskever, I., and Hinton, G. E.  Imagenet classification with deep convolutional neural networks.  In _Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1_ , NIPS’12, pp. 1097–1105, Red Hook, NY, USA, 2012. Curran Associates Inc.
  * Lugmayr et al. (2022) Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., and Van Gool, L.  Repaint: Inpainting using denoising diffusion probabilistic models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 11461–11471, 2022.
  * Masnou & Morel (1998) Masnou, S. and Morel, J.-M.  Level lines based disocclusion.  In _Proc. 1998 IEEE International Conference on Image Processing_ , volume 3, pp. 259–263, Chicago, IL, October 1998.
  * Mayer et al. (2016) Mayer, N., Ilg, E., Hausser, P., Fischer, P., Cremers, D., Dosovitskiy, A., and Brox, T.  A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 4040–4048, 2016.
  * Morton & Mayers (2005) Morton, K. W. and Mayers, D. F.  _Numerical Solution of Partial Differential Equations: An Introduction_.  Cambridge university press, 2005.
  * Nair & Hinton (2010) Nair, V. and Hinton, G. E.  Rectified linear units improve restricted boltzmann machines.  In _Proceedings of the 27th International Conference on Machine Learning_ , ICML’10, pp. 807–814, Madison, WI, USA, 2010. Omnipress.  ISBN 9781605589077.
  * Perona & Malik (1990) Perona, P. and Malik, J.  Scale-space and edge detection using anisotropic diffusion.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 12(7):629–639, jul 1990.  ISSN 0162-8828.
  * Raad et al. (2020) Raad, L., Oliver, M., Ballester, C., Haro, G., and Meinhardt, E.  On anisotropic optical flow inpainting algorithms.  _Image Processing On Line_ , 10:78–104, 2020.
  * Ronneberger et al. (2015) Ronneberger, O., Fischer, P., and Brox, T.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pp. 234–241. Springer, 2015.
  * Ruthotto & Haber (2020) Ruthotto, L. and Haber, E.  Deep neural networks motivated by partial differential equations.  _Journal of Mathematical Imaging and Vision_ , 62:352–364, 2020.
  * Saharia et al. (2022) Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Schmaltz et al. (2014) Schmaltz, C., Peter, P., Mainberger, M., Ebel, F., Weickert, J., and Bruhn, A.  Understanding, optimising, and extending data compression with anisotropic diffusion.  _International Journal of Computer Vision_ , 108(3):222–240, jul 2014.  ISSN 0920-5691.
  * Teed & Deng (2020) Teed, Z. and Deng, J.  Raft: Recurrent all-pairs field transforms for optical flow.  In _Proceedings of the 12th European Conference on Computer Vision – Volume Part II_ , pp. 402–419, Berlin, Heidelberg, August 2020. Springer-Verlag.  ISBN 978-3-030-58535-8.
  * Vašata et al. (2021) Vašata, D., Halama, T., and Friedjungová, M.  Image inpainting using wasserstein generative adversarial imputation network.  In _Artificial Neural Networks and Machine Learning – ICANN 2021: 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 14–17, 2021, Proceedings, Part II_ , pp. 575–586, Berlin, Heidelberg, 2021. Springer-Verlag.  ISBN 978-3-030-86339-5.
  * Weickert (1994) Weickert, J.  Theoretical foundations of anisotropic diffusion in image processing.  In _Proceedings of the 7th TFCV on Theoretical Foundations of Computer Vision_ , pp. 221–236, Berlin, Heidelberg, 1994. Springer-Verlag.  ISBN 3211827307.
  * Weickert (1998) Weickert, J.  _Anisotropic Diffusion in Image Processing_.  Teubner Stuttgart, 1998.  URL https://www.mia.uni-saarland.de/weickert/Papers/book.pdf.
  * Weickert & Schnörr (2001) Weickert, J. and Schnörr, C.  A theoretical framework for convex regularizers in pde-based computation of image motion.  _International Journal of Computer Vision_ , 45(3):245–264, December 2001.  ISSN 0920-5691.
  * Weickert et al. (2013) Weickert, J., Welk, M., and Wickert, M.  L2-stable nonstandard finite differences for anisotropic diffusion.  In _Scale Space and Variational Methods in Computer Vision_ , volume 7893, pp. 380–391. Springer, 2013.
  * Xu et al. (2022) Xu, H., Zhang, J., Cai, J., Rezatofighi, H., and Tao, D.  Gmflow: Learning optical flow via global matching.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 8121–8130, 2022.

##  Appendix A Training Details

In this section, we provide further details about the training procedure of
our method. Unless otherwise specified, these details also apply to the
considered baselines we will discuss in later sections.

We evaluate our methods on a combination of two datasets. We train on the
FlyingThings3D (flyingthings) subset that removes overly large displacements
and evaluate on the Sintel dataset (Sintel). For both datasets, we use the
final versions that include image degradations such as motion or depth-of-
field blur. As the evaluation metric, we choose the average Endpoint Error
(EPE), as it is customary for optical flow evaluation. While training, the
mask is drawn from a uniform distribution and binarized to adhere to the
desired density. During evaluation, we use a fixed mask for each image. To
speed up the training, we use center-cropped images of size 384×384 pixels.

All methods are implemented in PyTorch and trained on an Nvidia A100 GPU. We
trained the neural networks for a total of $900{,}000$ iterations using a
batch size of $16$. For the optimizer, we choose Adam (adam) with the default
parameter configuration $\beta_{1}=0.9,\beta_{2}=0.999$. We use an initial
learning rate of $0.0001$ that is halved every $100{,}000$ iterations after
the first $300{,}000$.

##  Appendix B Implementation Details

In this section, we provide some additional implementation details of our
methods.

The CNN architecture we use to estimate parameters to the diffusion process is
a simple UNet (Unet) architecture. Table 4 shows the layers and corresponding
channel dimensions in our Diffusion Tensor Module.

###  B.1 ResNet implementation of Diffusion Evolutions

As stated by DiffBlockConn, a discretization of any higher order diffusion
evolution can be formulated as a variant of a ResNet (resnet) block.
DiffBlockConn introduced their diffusion blocks for nonlinear diffusion
processes where the diffusion tensor is determined from the evolving signal.
To translate ResNet into diffusion blocks, we will follow their formulation
and consider such a diffusion process. However, in the following subsection we
will show how to adapt these diffusion blocks into the linear, image-driven
diffusion process we considered as part of our ablation study.

Starting from the conventional version of a ResNet block, we are now
constructing a nonlinear diffusion process from it. To this end, we
reintroduce the activation function we used in the main section of the paper:

$\Phi(\bm{Ku})=g\Bigl{(}\underbrace{\sum_{i=0}^{c}(\bm{Ku})_{i}(\bm{Ku})_{i}^{\top}}_{\bm{S}}\Bigr{)}(\bm{Ku}).$
(11)

Note the change in notation, where we removed the first argument that
determines the structure tensor. We do this, to make the translation into the
ResNet block more intuitive.

A normal ResNet block can be written in the following form:

$\bm{u}=\sigma_{2}(\bm{f}+\bm{W}_{2}\sigma_{1}(\bm{W}_{1}\bm{f}+\bm{b}_{1},~{}\bm{y})+\bm{b}_{2},~{}\bm{y}),$
(12)

with $\bm{W}_{1},\bm{W}_{2}$ denoting the application of a convolution kernel,
$\bm{b}_{1},\bm{b}_{2}$ denoting the respective biases, and
$\sigma_{1},\sigma_{2}$ denoting arbitrary activation functions, such as the
ReLU (relu).

$\displaystyle\sigma_{1}(\bm{x})=\tau\Phi(\bm{x}),\hskip 28.45274pt$
$\displaystyle\sigma_{2}(\bm{x})=\bm{x},$ (13)
$\displaystyle\bm{W}_{1}=\bm{K},\hskip 28.45274pt$
$\displaystyle\bm{W}_{2}=-\bm{K}^{T}$

and with $\bm{b}_{1}=\bm{b}_{2}=\bm{0}$, we can transform our diffusion
process into a ResNet architecture.

Notably, the convolution kernels share their weights since they implement the
same operator $\bm{K}$. In practice, this is implemented by maintaining one
kernel $\bm{W}$ that resembles the inner convolution. The outer convolution
kernel can be obtained by mirroring and negating $\bm{W}$ (DiffBlockConn).
When dealing with more than one input channel, the PDE formulation suggests
that inter-channel communication should only happen through the joint
diffusion tensor in the activation and not the derivative (e.g. the
convolution). This behavior can be realized by implementing the convolution
kernel $\bm{W}$ as grouped convolutions (alexnet) with an equal number of
groups to channels.

When learning the finite difference operators in the diffusion block,
stability of the diffusion process can be harmed. To avoid this, DiffBlockConn
suggested a weight normalization process that rescales the convolution kernels
after each optimization step. The stability assumptions hold, as long as the
maximal absolute eigenvalue of $\bm{K}$ is less or equal to $1$. In practice,
this constraint can be satisfied by rescaling each grouped convolution by
$\sqrt{C}\lVert\bm{W}\rVert_{2}^{2}$, where $C$ is the number of channels of
the considered signal $\bm{u}$. For more information about diffusion blocks,
we kindly refer the reader to the original publication (DiffBlockConn).

Table 4: Architecture of the Diffusion Tensor Module

Encoder
---
Layer | in-ch | out-ch | stride
cnv0 | 3 | 44 | 1
cnv1 | 44 | 44 | 2
cnv1_1 | 44 | 44 | 1
cnv2 | 44 | 88 | 2
cnv2_1 | 88 | 88 | 1
cnv3 | 88 | 176 | 2
cnv3_1 | 176 | 176 | 1
cnv4 | 176 | 352 | 2

Decoder
---
Layer | in-ch | out-ch | input
dcnv4 | 352 | 176 | cnv4
dcnv3 | 352 | 176 | dcnv4+cnv3_1
dcnv2 | 264 | 88 | dcnv3+cnv2_1
dcnv1 | 132 | 44 | dcnv2+cnv1_1
dt4 | 352 | 5 | dcnv4+cnv3_1
dt3 | 352 | 5 | dcnv3+cnv2_1
dt2 | 264 | 5 | dcnv2+cnv1_1
dt1 | 132 | 5 | dcnv1+cnv0

Table 5: All required convolution operators per diffusion block. The outer
convolution is obtained by mirroring around the center of the kernel and
multiplying by $-1$.

$\bm{W}^{1}_{x}$ =  -1 1 0 0 $\bm{W}^{2}_{x}$ =  0 0 -1 1 $\bm{W}^{1}_{y}$ =
-1 0 1 0 $\bm{W}^{2}_{y}$ =  0 -1 0 1

###  B.2 Diffusion Block formulation of our Scheme

To translate our considered diffusion process into a ResNet-style
architecture, we need to construct the diffusion blocks that correspond to the
discretization of WWW and adapt the activation function shown in Equation 11.

Adapting the activation is straightforward. We only need to go back to our
initial formulation in the main section of the paper and let the diffusion
block accept an additional input, which corresponds to the structure tensor of
the reference image. Since the reference image does not change during the
diffusion evolution, the diffusion tensor can be precomputed for each level in
the coarse-to-fine pyramid, allowing for a more efficient scheme. By accepting
the reference image in the activation, it brings us back to the original
definition of the activation function $\Phi(\bm{I,~{}K}\bm{u})$.

To design the diffusion blocks, we need to decompose the
$3{\mkern-2.0mu\times\mkern-2.0mu}3$-stencil of WWW into the individual finite
difference operators. WWW propose a weighted average of two
$2{\mkern-2.0mu\times\mkern-2.0mu}2$ for each $x-$ and $y-$derivative.
Consequently, they leverage $4$ total finite differences per discrete gradient
operator. Therefore, each diffusion block requires $4$ convolution kernels.
The required kernels are shown in Table 5.

Let in the following $\bm{D}=\begin{pmatrix}a&b\\\ b&c\end{pmatrix}$ denote
the considered diffusion tensor based on the reference image $\bm{I}$. The
discrete divergence term is then implemented as

$\bm{K}^{\top}\Phi(\bm{I,~{}K}\bm{u})=\bm{w}^{\top}\bm{Hw},$ (14)

where
$\bm{w}:=(\bm{W}_{x}^{1}\bm{u},\bm{W}_{x}^{2}\bm{u},\bm{W}_{y}^{1}\bm{u},\bm{W}_{y}^{2}\bm{u})^{\top}$.
The construction of the matrix $\bm{H}$ introduces the discretization
parameters $\alpha$ and $\beta$ which will be used to weight the influence of
the finite difference operators:

$\bm{H}:=\begin{pmatrix}\frac{1-\alpha}{2}a&\frac{\alpha}{2}a&\frac{1-\beta}{4}b&\frac{1+\beta}{4}b\\\
\frac{\alpha}{2}a&\frac{1-\alpha}{2}a&\frac{1+\beta}{4}b&\frac{1-\beta}{4}b\\\
\frac{1-\beta}{4}b&\frac{1+\beta}{4}b&\frac{1-\alpha}{2}c&\frac{\alpha}{2}c\\\
\frac{1+\beta}{4}b&\frac{1-\beta}{4}b&\frac{\alpha}{2}c&\frac{1-\alpha}{2}c\end{pmatrix}$
(15)

In this formulation, the role of the discretization parameters also become
more clear. $\alpha$ and $\beta$ are used to determine the relative importance
of the diagonal and off-diagonal entries of the diffusion tensor respectively.
For more details about the discretization, we refer the reader to the original
publication of WWW.

Although this scheme can be implemented very efficiently in its traditional
form, translating it into a diffusion block introduces a severe computational
overhead. When considering the original stencil, one diffusion step requires a
single $3{\mkern-2.0mu\times\mkern-2.0mu}3$ convolution of the input signal.
In the diffusion block formulation where each finite difference operator can
be learned, one diffusion step requires a total of $8$
$2{\mkern-2.0mu\times\mkern-2.0mu}2$ convolutions. This does not only slow
down the effective inference time, it also bloats the computational graph
severely when optimizing each convolution kernel. Since we also did not see a
meaningful performance benefit to learning the operators, we opted against the
use of the diffusion blocks in our final model.

##  Appendix C Baseline details

###  C.1 Edge-Enhancing Diffusion Inpainting

We implement the EED inpainting baseline in PyTorch using the same
discretization of WWW. The reference images are normalized to the range
$[0,1]$. For faster convergence, we use the Fast-Semi-Iterative (FSI) scheme
(FSI) and the same coarse-to-fine setup as in our method. However, instead of
a fixed number of iterations as in our learned approach, we let each the
inpainting process converge at each resolution. We determine a sufficiently
converged state by observing the relative residual and stop the inpainting
once it has decreased below $10^{-6}$.

To make for a fair comparison with the deep learning methods, we optimize the
free parameters $\lambda,\alpha$ for each considered density on a subset of
the training data and keep them fixed during evaluation. Parameters are
determined via grid search on $128$ samples and we consider the best
parameters as the ones that minimize the EPE with the ground truth. Compared
to the final evaluation, we stop the inpainting during the parameter
optimization once the relative residual decreased by $10^{-5}$. We show the
chosen parameter per resolution and the considered interval in Table 6

Table 6: Grid Search to determine the optimal EED-parameters for each density.
Step denotes how many evenly spaced values we consider within the search
space.

Parameter $10\%$ $5\%$ $1\%$ Search Space Step $\lambda$ $10^{-4}$ $10^{-4}$
$10^{-4}$ $[10^{-6},10^{-2}]$ $9$ $\alpha$ $0.1$ $0.3$ $0.42$ $[0.001,0.5]$
$14$

###  C.2 FlowNetS Training Details

In the inpainting setting we start with a sparse initialization of correct
displacements, whereas FlowNetS (flownet) needs to find identifiable
correspondences given sequential images. Consequently, with over $15.6$
million parameters FlowNetS might be unneccessarily complex for our inpainting
task. In its original form, FlowNetS estimates the flow at a lower spatial
resolution and upsamples the initial estimation with bilinear interpolation.
This was done to achieve optical flow estimation in real time. Since runtime
is not a critical factor for us, we extend the decoder to the full output
resolution with two additional transposed convolution layers. The number of
channels per layer is reduced throughout the whole network, such that we end
up with roughly $8.8$ million learnable parameters.

To train the network, we follow mostly the same approach as discussed in A. In
addition to that, we added weight decay with weighting parameter $0.0004$ and
a deep supervision approach for the loss as proposed in (flownet). Concretely,
this means that we predict a (low resolution) flow at the last $4$ layers in
the decoder and compute the EPE with downsampled versions of the flow. All
losses are aggregated as a weighted combination, where we used the weights
$[0.32,0.08,0.04,0.02,0.01]$ going from coarse to fine resolution.

###  C.3 WGAIN Training Details

WGAIN (wgain) does not adapt well to the flow setting in its original form. We
noticed extremely unstable training with diverging loss after a few hundred
iterations. We suspect, that this is due to the combination of dealing with
(potentially large) flow values and the gradient clipping introduced in
(wasserstein) and used in (wgain). As a way of mitigating the outliers in the
flow, we divide the flow fields by $100$ to largely contain them in the range
of $[-1,1]$, but keep the relative distribution the same. To stabilize the
training, we replaced the gradient clipping operation with a gradient penalty
term (wganGP) in the training objective. As discussed in (wganGP), we also
introduce layer normalization (layernorm) in the critic for additional
stability. The generator remained largely unchanged, with the exception of the
removal of the hard-sigmoid function after the output layer. We adopted the
rest of the training procedure from (wgain), with the exception of using the
EPE instead of the Mean Absolute Error (MAE) and choosing $\lambda_{g}=1$. The
model was trained for the same number of iterations as our method.

###  C.4 Probabilistic Diffusion Training Details

With the exception of RePaint (repaint), sparse mask inpainting with
probabilistic diffusion has rarely been addressed. Since RePaint is only
applied during inference, any type of PD model for conditional image
generation can be used for our task. We chose the efficient UNet architecture
of Imagen (imagen) and adopted their cascading image generation pipeline. They
propose to generate a low-resolution image initially and compose super-
resolution models to transform it to the desired resolution. In our case, we
generate the initial image at resolution
$96{\mkern-2.0mu\times\mkern-2.0mu}96$ and chain one super-resolution net to
obtain the final flow at resolution $384{\mkern-2.0mu\times\mkern-2.0mu}384$.

Both networks obtain the reference image as conditioning signal and are
otherwise trained for conditional image generation. We used the proposed
training parameters in (imagen), but observed suboptimal results and slow
convergence during inference times. Consequently, we adopted the novel
training procedure from (elucidating) which yielded more effective training
and significantly reduced the sampling time during inference. As can be seen
in Table 7, this work adds several parameters to control the noise
distribution. We kept most of them the same as the optimal parameters in
(elucidating), but we noticed some improvements by increasing $\sigma_{max}$
and $\sigma_{data}$.

During inference, we perform $48$ sampling steps and apply the RePaint
(repaint) inpainting at both resolutions. RePaint introduces two parameters,
the number of resampling steps and the jump length. In (repaint) the jump
length was introduced to avoid blurred outputs. However, we observed sharp
edges with a jump length of $1$ and therefore kept this parameter fixed. The
resampling steps, on the other hand, are more critical. They provide a
tradeoff between added runtime during sampling and increased conditioning on
the known pixels. In (repaint) the masks were dense compared to our setting.
We noticed that the proposed number of $10$ resampling steps in (repaint)
yields poor inpainting quality with mask densities below $10\%$. To achieve
competetive performance on low densities, we had to increase the number of
steps and lower the inference time even further. We show the additional
parameters we used in Table 7.

Table 7: Added hyperparameters for training and inference of our Probabilistic Diffusion baseline Parameter | Value | Source
---|---|---
$\sigma_{min}$ | $0.002$ | (elucidating)
$\sigma_{max}$ | $(120,480)$ | (elucidating)
$\sigma_{data}$ | $1$ | (elucidating)
$\rho$ | $7$ | (elucidating)
$P_{mean}$ | $-1.2$ | (elucidating)
$P_{std}$ | $1.2$ | (elucidating)
$S_{churn}$ | $80$ | (elucidating)
$S_{tmin}$ | $0.05$ | (elucidating)
$S_{tmax}$ | $50$ | (elucidating)
$S_{noise}$ | $1.003$ | (elucidating)
Jump Length | 1 | (repaint)
Resampling Steps | 45 | (repaint)


# Probing flow fluctuations through factorization breaking of harmonic flows
in heavy-ion collision††thanks: Presented at Quark Matter 2022 : XXIXth
International Conference on Ultra-relativistic Nucleus-Nucleus Collisions

Piotr Bożek  Rupam Samanta
AGH University of Science and Technology, Faculty of Physics and Applied
Computer Science, aleja Mickiewicza 30, 30-059 Cracow, Poland

###### Abstract

We study factorization-breaking coefficients between the momentum dependent
and momentum averaged flow vectors to probe flow fluctuations caused by
initial-state fluctuations in heavy-ion collision. The coefficients for the
flow vector squared and flow magnitude squared could be used for the
extraction of flow angle decorrelations. We compare our model results with
preliminary experimental data. We also present the predictions for the
momentum dependent correlation between mixed flow harmonics.

##  1 Introduction

The dense matter created in the interaction region of a heavy-ion collision,
expands rapidly. Due to the asymmetry of the initial source distribution, the
collective expansion leads to an azimuthal momentum anisotropy of the spectra
of particles,

$\frac{dN}{dpd\phi}=\frac{dN}{2\pi
dp}(1+2\sum_{n=1}^{\infty}V_{n}(p)e^{in\Psi_{n}})$ (1)

where the Fourier coefficients $V_{n}(p)^{\prime}s$ are called the harmonic
flow coefficients.

The initial source fluctuates event-by-event (ebe), leading to the ebe
fluctuation of the harmonic flow coefficients in the final state [1, 2]. These
flow fluctuations can be probed by studying the correlation coefficient
between the harmonic flow vectors in two different transverse momentum bins
[3], the factorization-breaking coefficients. The fluctuations result in the
decorrelation (deviation of the correlation coefficients from 1) between these
flow vectors, which manifests the combined effect of flow magnitude
decorrelation as well as the flow angle decorrelation[4, 5]. These two effects
can be separated by constructing four particle correlators between two
transverse momentum bins. However, to avoid the experimental difficulty due to
the low statistics at higher momenta, the coefficients are constructed keeping
two particles fixed at transverse momentum bins (differential flow) and the
other two are momentum averaged (integrated flow)[6]. We compare our model
results with the preliminary results by ALICE collaboration. We also study the
similar factorization coefficients between the mixed flow harmonics e.g.
$V_{2}^{2}$ \- $V_{4}(p)$ or $V_{2}V_{3}$ \- $V_{5}(p)$ , which serve as
measures of the non-linear hydrodynamic response of the medium.

##  2 Factorization Breaking for harmonic flow vector, magnitude and angle

The complex flow vector $V_{n}(p)=v_{n}(p)e^{in\Psi_{n}(p)}$ in Eq. 1 denotes
the harmonic flow vector of order $n$ for particles emitted at the transverse
momentum $p$; $v_{n}$ and $\Psi_{n}$ are the corresponding flow magnitude and
angle. In general , one can construct the four-particle correlators, measuring
the factorization breaking coefficients between two flow vectors squared at
two different transverse momentum bins as [3]

$r_{n;2}(p_{1},p_{2})=\frac{\langle V_{n}^{2}(p_{1})V_{n}^{\star\
}(p_{2})^{2}\rangle}{\sqrt{\langle v_{n}^{4}(p_{1})\rangle\langle
v_{n}^{4}(p_{2})\rangle}}$ (2)

and similarly for the flow magnitude squared , the factorization-breaking
coefficient can be constructed as [7],

$r_{n}^{v_{n}^{2}}(p_{1},p_{2})=\frac{\langle|V_{n}(p_{1})|^{2}|V_{n}(p_{2})|^{2}\rangle}{\sqrt{\langle
v_{n}^{4}(p_{1})\rangle\langle v_{n}^{4}(p_{2})\rangle}}=\frac{\langle
v_{n}^{2}(p_{1})v_{n}^{2}(p_{2})\rangle}{\sqrt{\langle
v_{n}^{4}(p_{1})\rangle\langle v_{n}^{4}(p_{2})\rangle}}$ (3)

However, in practice, the above formulas are difficult to use in experiment
due to the low statistics in higher momentum bins.

Figure 1: The factorization breaking coefficient between flow vectors squared
$V_{2}(p)^{2}$ and $V_{2}^{2}$ (left) and for the flow magnitudes squared
$v_{2}(p)^{2}$ and $v_{2}^{2}$ (right) for the elliptic flow. The red squares
and black triangles denote the results obtained with the Glauber model and the
TRENTO model initial conditions respectively. The experimental data of the
ALICE Collaboration are represented using blue dots. (Fig. from [8])

To ease this difficulty, an alternative approach proposed by the ALICE
collaboration, would be to keep only one of the flow in a transverse momentum
bin and the other flow averaged over the momentum range. Following this, we
construct the factorization-breaking coefficient using the four-particle
correlator for the flow vector squared and flow magnitude squared as[6],

$r_{n;2}(p)=\frac{\langle V_{n}^{2}V_{n}^{\star}(p)^{2}\rangle}{\sqrt{\langle
v_{n}^{4}\rangle\langle v_{n}^{4}(p)\rangle}}\ \ and\ \
r_{n}^{v_{n}^{2}}(p)=\frac{\langle v_{n}^{2}v_{n}^{2}(p)\rangle}{\sqrt{\langle
v_{n}^{4}\rangle\langle v_{n}^{4}(p)\rangle}}\ .$ (4)

We use the boost invariant version of the hydrodynamic model MUSIC [9] for
Pb+Pb collisions at $5.02$ TeV. For the initial state, two models are used; a
two-component Glauber Monte Carlo model [10] and the TRENTO model [11]. We use
a constant shear viscosity to entropy density ratio $\eta/s=0.08$, unless
otherwise specified. In Fig. 1 are shown the factorization breaking
coefficients between the flow vectors squared (left) and flow magnitudes
squared (right) for the elliptic flow. For 0-5 % centrality our model result
corroborate the ALICE preliminary data. However, for semi-central collision
(30-40 %), our model results do not fully describe the data. Also one can see
that the decorrelation for the magnitudes $r_{2}^{v_{2}^{2}}(p)$, is
approximately one half of the decorrelation for the vectors $r_{2;2}(p)$ in 1,
which manifest the fact that the total flow vector decorrelation involves
equal contribution from the flow magnitude and the flow angle decorrelation;
$[1-r_{n;2}(p)]\simeq 2[1-r_{n}^{v_{n}^{2}}(p)]$.

Figure 2: The flow angle correlation estimated as the ratio of the flow vector
and the flow magnitude factorization breaking coefficients (Eq. 5). The
symbols with the dashed lines denote the angle correlations calculated
directly from the simulated events (Eq. 7). The green squares represent the
simple average of the decorrelation angle (Eq. 8).

Once we have the flow vector and flow magnitude factorization-breaking
coefficients separately, we can estimate the flow angle decorrelation by
constructing the correlator for flow angle, taking the ratio of the four-
particle flow vector correlator and flow magnitude correlator Eq. 4,

$F_{n}(p)=\frac{\langle V_{n}^{2}V_{n}^{\star}(p)^{2}\rangle}{\langle
v_{n}^{2}v_{n}^{2}(p)\rangle}\ .$ (5)

The above correlator is an experimental measure of the flow angle
decorrelation:

$F_{n}(p)=\frac{\langle
v_{n}^{2}v_{n}(p)^{2}\cos\left[2n\left(\Psi_{n}(p)-\Psi_{n}\right)\right]\rangle}{\langle
v_{n}^{2}v_{n}(p)^{2}\rangle}\ .$ (6)

However, the actual angle decorrelation could be defined as [7],

$\frac{\langle
v_{n}^{4}\cos\left[2n\left(\Psi_{n}(p)-\Psi_{n}\right)\right]\rangle}{\langle
v_{n}^{4}\rangle}\ .$ (7)

The results presented in Fig. 2 indicate that the two formulas give similar
results, suggesting that the experimental measure 5 could be used to estimate
the weighted flow angle decorrelation. Our model results describe the
experimental angle decorrelation. On the other hand, the simple average of the
decorrelation angle

$\langle\cos\left[2n\left(\Psi_{n}(p)-\Psi_{n}\right)\right]\rangle$ (8)

is very different and cannot be estimated directly from the data.

##  3 Mixed flow correlations

Similar correlations coefficients can be constructed between mixed flow
harmonics, as a measure of non-linear coupling between the flow harmonics.
Such correlation coefficients are sensitive to the non-linear response of the
hot dense medium [12]. For example, one can construct the correlation
coefficients between $V_{2}^{2}$ and $V_{4}(p)$ or between $V_{2}V_{3}$ and
$V_{5}(p)$. However, to extract the angle decorrelation between these mixed
flow, one needs to construct the correlation coefficient (or factorization-
breaking coefficients) between the squares of the harmonics used. The mixed
flow vector-vector, magnitude-magnitude and angle decorrelation are given by,

$\frac{\langle V_{2}^{4}V_{4}^{\star}(p)^{2}\rangle}{\sqrt{{\langle
v_{2}^{8}\rangle}{\langle v_{4}^{4}(p)\rangle}}}\ ,\frac{\langle
v_{2}^{4}v_{4}(p)^{2}\rangle}{\sqrt{{\langle v_{2}^{8}\rangle}{\langle
v_{4}^{4}(p)\rangle}}}\ \ \rm{and}\ \ \frac{\langle
V_{2}^{4}V_{4}^{\star}(p)^{2}\rangle}{\langle v_{2}^{4}v_{4}(p)^{2}\rangle}\
.$ (9)

Figure 3: The flow vector (left) and flow magnitude correlation (right)
between $V_{2}^{4}$ and $V_{4}(p)^{2}$. For the TRENTO model initial
conditions, the solid, dashed and dotted lines represent results with
$\eta/s=0.08,\ 0.12,\ 0.16$ respectively. The horizontal lines represent the
corresponding correlation coefficients for momentum averaged flow vectors
$V_{2}^{4}$ and $V_{4}^{2}$. (Fig. from [8]) Figure 4: The flow angle
correlation between $V_{2}^{4}$ and $V_{4}(p)^{2}$. (Fig. from [8])

In Figs. 3 and 4, are shown the results predicted in the model for the mixed
flow factorization-breaking coefficients between $V_{2}^{4}$ and $V_{4}^{2}$
for flow vector, magnitude and angle decorrelations. Again, we notice that the
flow magnitude decorrelation is approximately one half of the flow vector
decorrelation. The horizontal lines denote the baseline for the correlations
between the momentum averaged flow harmonics.

##  4 Summary and outlook

We study the factorization-breaking coefficients between the harmonic flow
coefficient in a fixed transverse momentum bin and the average flow harmonic.
Such factorization coefficients show significant decorrelations which is a
direct consequence of initial state fluctuation. The separate measurement of
the flow vector and magnitude decorrelation lead to the extraction of flow
angle decorrelation, where the magnitude and the flow angle decorrelations
constitute approximately equal parts of the total flow vector decorrelation.
Our model results well reproduce the ALICE preliminary data in central
collisions, but deviate in case of semi-central collisions. We also present
predictions for the momentum dependent mixed flow correlations, which measure
the non-linear response of the hydro response and put additional constrain on
the models. Our results could be verified in experiments.

## Acknowledgments

This research was partly supported by the Polish National Science Centre
grant: 2019/35/O/ST2/00357.

## References

  * [1] B. Alver et al., “Importance of correlations and fluctuations on the initial source eccentricity in high-energy nucleus-nucleus collisions,” Phys. Rev. C, vol. 77, p. 014906, 2008.
  * [2] B. Alver and G. Roland, “Collision geometry fluctuations and triangular flow in heavy-ion collisions,” Phys. Rev., vol. C81, p. 054905, 2010.
  * [3] F. G. Gardim, F. Grassi, M. Luzum, and J.-Y. Ollitrault, “Breaking of factorization of two-particle correlations in hydrodynamics,” Phys.Rev., vol. C87, no. 3, p. 031901, 2013.
  * [4] J. Jia and P. Huo, “Forward-backward eccentricity and participant-plane angle fluctuations and their influences on longitudinal dynamics of collective flow,” Phys.Rev., vol. C90, p. 034915, 2014.
  * [5] P. Bożek and W. Broniowski, “Longitudinal decorrelation measures of flow magnitude and event-plane angles in ultrarelativistic nuclear collisions,” Phys. Rev., vol. C97, no. 3, p. 034913, 2018.
  * [6] E. Nielsen talk presented at 6th International Conference on the Initial Stages in High-Energy Nuclear Collisions. Rehovot, 10-15 January , 2021.
  * [7] P. Bożek, “Angle and magnitude decorrelation in the factorization breaking of collective flow,” Phys. Rev., vol. C98, no. 6, p. 064906, 2018\.
  * [8] P. Bozek and R. Samanta, “Factorization breaking for higher moments of harmonic flow,” Phys. Rev. C, vol. 105, no. 3, p. 034904, 2022.
  * [9] B. Schenke, S. Jeon, and C. Gale, “(3+1)D hydrodynamic simulation of relativistic heavy-ion collisions,” Phys. Rev. C, vol. 82, p. 014903, 2010\.
  * [10] P. Bożek, W. Broniowski, M. Rybczynski, and G. Stefanek, “GLISSANDO 3: GLauber Initial-State Simulation AND mOre…, ver. 3,” Comput. Phys. Commun., vol. 245, p. 106850, 2019.
  * [11] J. S. Moreland, J. E. Bernhard, and S. A. Bass, “Alternative ansatz to wounded nucleon and binary collision scaling in high-energy nuclear collisions,” Phys. Rev. C, vol. 92, no. 1, p. 011901(R), 2015.
  * [12] R. S. Bhalerao, J.-Y. Ollitrault, and S. Pal, “Event-plane correlators,” Phys. Rev., vol. C88, p. 024909, 2013.