# Compositional Neural Textures

Peihan Tu  University of MarylandCollege ParkMarylandUnited States ,  Li-Yi
Wei  Adobe ResearchSan JoseCaliforniaUnited States and  Matthias Zwicker
University of MarylandCollege ParkMarylandUnited States

###### Abstract.

Texture plays a vital role in enhancing visual richness in both real
photographs and computer-generated imagery. However, the process of editing
textures often involves laborious and repetitive manual adjustments of
textons, which are the small, recurring local patterns that define textures.
In this work, we introduce a fully unsupervised approach for representing
textures using a compositional neural model that captures individual textons.
We represent each texton as a 2D Gaussian function whose spatial support
approximates its shape, and an associated feature that encodes its detailed
appearance. By modeling a texture as a discrete composition of Gaussian
textons, the representation offers both expressiveness and ease of editing.
Textures can be edited by modifying the compositional Gaussians within the
latent space, and new textures can be efficiently synthesized by feeding the
modified Gaussians through a generator network in a feed-forward manner. This
approach enables a wide range of applications, including transferring
appearance from an image texture to another image, diversifying textures,
texture interpolation, revealing/modifying texture variations, edit
propagation, texture animation, and direct texton manipulation. The proposed
approach contributes to advancing texture analysis, modeling, and editing
techniques, and opens up new possibilities for creating visually appealing
images with controllable textures.

texture, editing, machine learning, neural networks

††journal: TOG††journalyear: 2024††journalvolume: 36††journalnumber:
4††article: 1††publicationmonth: 7††submissionid: papers_xxx††copyright:
acmlicensed††ccs: Computing methodologies Texturing††ccs: Computing
methodologies Learning latent representations††ccs: Computing methodologies
Neural networks††ccs: Computing methodologies Unsupervised learning

---
|  |
|  |

(a) Overview

(b) Texture transfer (struct./appear.$①/②$and$②/①$)

(c) Image stylization $①$

(d) Interpolation & morphing $①②$

(e) Revealing/modifying variations $③$

(f) Edit propagation ($④$)

(g) Editing Gaussian textons $④$

Figure 1. Compositional neural representation for texture editing.
LABEL:sub@fig:teaser:overview We present a neural representation that
decomposes textures into geometric elements and appearance features in a fully
unsupervised manner. Geometric elements are represented by Gaussian functions
computed from an intermediate segmentation corresponding to individual
textons, and appearance is represented by an appearance feature vector for
each Gaussian. Our network architecture and objectives are designed to
encourage extraction of recurring texture elements (textons) and separation of
geometry/structure and appearance, while preserving faithful reconstruction of
the original texture. This facilitates a variety of editing operations and
synthesis applications, such as transferring appearance from one texture
towards the structure of another texture LABEL:sub@fig:teaser:texture_transfer
or image LABEL:sub@fig:teaser:image_stylization, texture interpolation and
morphing LABEL:sub@fig:teaser:interp_morphing, manipulating the variations
within a texture LABEL:sub@fig:teaser:variations, edit propagation
LABEL:sub@fig:teaser:propagation, and direct manipulation (e.g. moving and
transforming) of the textons LABEL:sub@fig:teaser:gaussians.

##  1\. Introduction

Texture plays a vital role in enhancing the visual richness and overall appeal
of various forms of visual content, including real photographs and computer-
generated imagery. Textures provide intricate details, patterns, and surface
characteristics that contribute to the overall perception and aesthetics of an
image. In practice, users often need to create or adjust textures in an image,
yet texture editing can be a challenging and time-consuming process, often
requiring laborious manual adjustments due to the repetitive details.

Previous research in texture manipulation has explored various applications,
ranging from global control [Hertzmann et al., 2001; Rosenberger et al., 2009;
Kaspar et al., 2015], to regional variations [Matusik et al., 2005; Frühstück
et al., 2019; Yu et al., 2019; Bellini et al., 2016; Dekel et al., 2015], and
individual textons [Brooks and Dodgson, 2002]. These approaches rely on
domain-specific priors for different applications or categories of textures.
Ideally, we would like to have a general-purpose approach that can be applied
to a wide range of textures and editing tasks across different scales.

In this paper, we introduce a texture representation that decomposes textures
into a composition of recurring neural elements (Figure 1a), disentangling the
geometry and appearance of each element. We obtain this representation by
developing an autoencoder architecture that is trained in a fully unsupervised
manner. Our representation is human-intelligible, applicable to a wide range
of textures that exhibit sufficient self-repetitions, and enhancing a variety
of editing techniques (Figure 1). Textures can be edited by manipulating the
compositional neural representations in deep latent space, and edited textures
can be efficiently generated by feeding the manipulated representations
through a generator (decoder) network in a feed-forward manner. The supported
editing operations include, but are not limited to, transferring appearance
from an image texture to another image (Figures 1b and 1c), diversifying
textures (Figure 3), texture interpolation and morphing (Figure 1d),
revealing/modifying texture variations (Figure 1e), edit propagation (Figure
1f), texture animation (Figure 13), and direct manipulation of textons (Figure
1g).

Our main idea of representing textures using a compositional neural model is
rooted in the understanding that textures are characterized by their recurring
patterns, which can vary from regular to irregular formations. Within each
texture, basic recurring patterns are known as textons, acting as building
blocks that define the texture’s unique appearance and structure. However,
extracting textons from diverse textures poses considerable challenges due to
the ambiguous nature of the texton concept, let alone defining them rigorously
in mathematical terms. One possible texton extraction approach is to use
supervised learning, for example, by training image segmentation networks.
While supervised approaches have achieved state-of-the-art performance for
extracting well-defined concepts from images [Krizhevsky et al., 2012; Minaee
et al., 2021] by learning from massive, human-annotated datasets, they are not
suitable for extracting textons. Because it is unclear how to consistently
annotate textons within a texture due to their ambiguous definition, it is
hard to obtain high-quality datasets required for supervised training.
Furthermore, the process of annotating any dataset can be both costly and
time-consuming. Therefore, we develop a fully unsupervised learning approach
to extract neural textons.

First, we propose a set of desired properties of (neural) textons detailed in
Section 3.1, which we leverage to effectively train the autoencoder (Figure
1a), including 1) discrete objectness, 2) spatial compactness, 3) reshuffling
invariance and 4) transformation equivariance. These poperties enable the
textons to represent diverse sets of textures while maintaining human
interpretability for intuitive texture editing applications. Second, we
represent each neural texton as a 2D Gaussian with an associated appearance
feature. A Gaussian approximates a texton’s shape via its spatial support, and
the Gaussian appearance feature encodes the texton’s exact appearance. This
seperation facilitates the enforcement of geometric transformation
equivariance. Further, the separate encoding of rough spatial and detailed
appearance information in Gaussian covariances and associated features offers
comprehensibility and intuitve user control over textons [Hertz et al., 2022;
Epstein et al., 2022]. Third, we develop unsupervised training methods to
instill the desired properties in neural textons. For each desired property we
construct a corresponding loss function that, when minimized, ensures the
incorporation of that property. In particular, the reshuffling invariance
property dictates that randomly reshuffling the appearances of textons, as
encoded by Gaussian appearance features, should not modify the texture’s
overall appearance. In line with this, we specifically design one of the
training losses to assess the texture similarity between the input texture,
encoded as Gaussians, and an image reconstructed from the Gaussians with
reshuffled appearance features. Moreover, we observe that the training losses
may present contradictory behaviors, where minimizing one could lead to
explosion of another, and address this issue by carefully adjusting the
training schedules to achieve a balanced trade-off between these losses.

We evaluate our proposed compositional neural texture representation using a
variety of texture editing applications, ablation studies, and comparative
analysis against existing methods. The results demonstrate that our approach
is versatile in various texture editing applications and surpasses existing
methods in texture analysis, modeling, and specific texture editing tasks.

##  2\. Related Work

###  2.1. Neural Texture Representations

Recent advancements in deep texture representations have yielded impressive
results in various applications, including texture classification [Zhai et
al., 2020; Xue et al., 2018; Zhang et al., 2017], segmentation [Minaee et al.,
2021; Li et al., 2022], synthesis [Gatys et al., 2015; Zhou et al., 2018],
interpolation [Yu et al., 2019; Bergmann et al., 2017; Henzler et al., 2020]
and rectification [Hao et al., 2023]. However, these representations, while
engineered to encode a broad spectrum of appearance information, exhibit very
limited or no disentanglement when applied to perception and uncontrolled
synthesis tasks. This entanglement considerably limits their practical utility
in texture editing tasks.

Texture interpolation, a global texture editing task, necessitates a
representation that effectively disentangles local and global texture
appearances. Bergmann et al. [Bergmann et al., 2017] introduce a periodic
spatial GAN (PSGAN). PSGAN maps noise to a texture through noise dimensions
that are disentangled into global, local, and spatially periodic components.
Yu et al. [2019] develop an autoencoder to encode an image into a disentangled
latent space. This space is divided into components that separately represent
local and global aspects of texture appearance. Specifically, the local
components of these representations encode the stochastic variations within a
texture, while the global components capture the overall appearance. This
separation makes these representations particularly effective for texture
interpolation tasks. It facilitates the interpolation of the overall texture
appearance, without compromising the inherent stochasticity within textures.
Henzler et al. [2020] model stochastic textures using noise fields and an MLP
decoder. Similarity to [Bergmann et al., 2017; Yu et al., 2019], this approach
effectively disentangles local stochastic and global appearance components. It
achieves this by encoding stochasticity with noise fields and the overall
appearance with a global texture code. This disentanglement facilitates
applications in texture interpolation. However, the level of disentanglement
in the representations currently used for interpolation is still inadequate,
limiting their extension to other applications across a diverse range of
textures.

To address this shortfall and broaden the scope in texture editing, a more
effectively disentangled representation is essential. We propose a
compositional neural texture representation by modeling textures as neural
textons in deep latent space with disentangled structure and appearance. The
proposed compositional representation is applicable to a number of texture
editing applications and a wide variety of textures.

###  2.2. Texture Editing

Previous research in texture manipulation has explored various applications,
such as controlled synthesis [Hertzmann et al., 2001; Rosenberger et al.,
2009; Kaspar et al., 2015], interpolation [Matusik et al., 2005; Frühstück et
al., 2019; Yu et al., 2019; Bellini et al., 2016], revealing/modifying
variations [Dekel et al., 2015], edit propagation [Brooks and Dodgson, 2002],
texture weathering [Bellini et al., 2016] and splicing [Liu et al., 2009].
These approaches rely on domain-specific priors for specific applications or
categories of textures. They leverage image smoothness [Kwatra et al., 2003]
or patch similarity priors [Lefebvre and Hoppe, 2005; Zhou et al., 2023], or
procedural noise [Henzler et al., 2020] defined in image pixel space [Lefebvre
and Hoppe, 2005; Brooks and Dodgson, 2002], hand-crafted feature space
[Lefebvre and Hoppe, 2006; Liu et al., 2004; Matusik et al., 2005; Bellini et
al., 2016], or learned feature spaces [Frühstück et al., 2019; Yu et al.,
2019; Bergmann et al., 2017; Henzler et al., 2020; Zhou et al., 2023], for
local [Wei et al., 2009], stochastic [Henzler et al., 2020], regular/near-
regular [Liu et al., 2004] or non-stationary textures [Zhou et al., 2018].

These texture editing methods can be classified into three categories: non-
parametric [Kwatra et al., 2003; Dekel et al., 2015; Wei et al., 2009;
Lefebvre and Hoppe, 2006], parametric [Matusik et al., 2005; Liu et al., 2004]
and more recent deep learning-based approaches [Yu et al., 2019; Henzler et
al., 2020]. Non-parametric methods, known for their flexibility, can adapt to
a diverse range of texture types. However, their editing results are less
predictable, and these methods are typically confined to specific applications
[Wei et al., 2009; Bellini et al., 2016; Brooks and Dodgson, 2002; Dekel et
al., 2015; Lu et al., 2009].

In contrast, parametric methods are anchored in well-defined mathematical
models, which makes them more interpretable, predictable, and controllable.
However, these methods encounter difficulties when dealing with textures that
are rich in complexity, encompassing intricate details and sophisticated
spatial arrangements [Matusik et al., 2005; Liu et al., 2004]. Recent deep
learning-based methods employ parametric neural networks, which excel at
representing textures, yet they often fall short in providing the
interpretability and control crucial for various user editing operations [Yu
et al., 2019; Bergmann et al., 2017; Henzler et al., 2020].

Another line of active research focuses on transforming image textures into
parametric, node-graph-based procedural textures [Li et al., 2023; Shi et al.,
2020; Guerrero et al., 2022; Hu et al., 2023]. These procedural textures, once
created, offer a professional-grade level of control and quality over texture
appearances. However, current methods are limited to a narrow range of texture
categories and encounter difficulties in synthesizing graph structures.
Moreover, the accuracy of the generated graphs and their parameters falls
short, failing to accurately and faithfully reproduce the appearance features
of the input image textures.

Consequently, existing approaches are either constrained to a limited set of
texture editing capabilities, or their applicability tends to be confined to
specific types of textures. Our proposed approach utilizes an autoencoder
network characterized by a disentangled, compositional latent space. This
latent space offers interpretability and expressiveness, unlocking a wide
range of editing capabilities for diverse sets of textures.

###  2.3. Compositional Neural Representations

In certain data domains such as programming languages, vector graphics, social
networks, and engineering designs, the underlying compositional structure is
either inherently present or can be parsed using specific rules. However, in
other domains like 2D images and 3D reconstructions, extracting and
representing compositional structures remains a challenging task due to its
elusive nature.

To address this, compositional neural representations have been proposed,
utilizing a neural latent space to represent various data types
compositionally. This approach has been explored in several raster data types,
such as 2D images [Epstein et al., 2022; Locatello et al., 2020; Jiang et al.,
2023; Zhang et al., 2018a], videos [Elsayed et al., 2022], 3D shapes [Hertz et
al., 2022; Nash and Williams, 2017], and scenes [Niemeyer and Geiger, 2021;
Sajjadi et al., 2022]. These representations are derived using various
methodologies, ranging from unsupervised [Epstein et al., 2022; Locatello et
al., 2020; Jiang et al., 2023; Zhang et al., 2018a] and semi-supervised
[Elsayed et al., 2022], to fully supervised methods [Nash and Williams, 2017].
The structures of these neural representations include disentangled object
slots [Locatello et al., 2020; Jiang et al., 2023], spatial points [Zhang et
al., 2018a], graphs [He et al., 2022a; Ost et al., 2021], Gaussians [Hertz et
al., 2022; Epstein et al., 2022], and image segments [He et al., 2022b].

In this work, we introduce a compositional neural representation for image
textures. This representation is characterized by compositions of Gaussians
within a disentangled neural latent space, and is learned without external
supervision.

##  3\. Method

###  3.1. Overview

At the core of our approach is a texture autoencoder with a disentangled,
compositional latent space, trained in a fully unsupervised manner. The latent
space consists of 2D Gaussians with neural appearance features, each
representing a texton. The Gaussian mean and covariance encode the location
and approximate (elliptical) shape of a texton. In addition, each Gaussian has
an associated neural feature vector that encodes the detailed appearance of a
texton. Users can edit textures by manipulating individual textons represented
as Gaussians in the latent space, such as editing their centers, covariances,
and appearance features.

(a) Network

(b) Encoder $\mathcal{E}$

Figure 2. Architecture of the proposed method.
LABEL:sub@fig:method_architecture:network Our network is composed of
autoencoding (Sections 3.2, 3.3 and 3.4), texton reshuffling (Section 3.5.1),
and structure-appearance disentangling (Section 3.5.2). In the autoencoding
part, the encoder $\mathcal{E}$ encodes an image $I$ into latent Gaussians
$\\{\textbf{g}\\}$. The Gaussians are splatted $h$ onto feature maps, which
are fed into the generator $\mathcal{G}$ to reconstruct the original input as
$I_{r}$. We calculate the reconstruction $L_{R}$ and GAN loss $L_{GAN}$
between $I$ and $I_{r}$ to ensure accurate reconstruction. In the texton
reshuffling part, we randomly reshuffle, denoted as function $\pi$, appearance
features within $\\{\textbf{g}\\}$ to produce $\\{\tilde{\textbf{g}}\\}$,
which are subsequently reconstructed as $\tilde{I}_{r}$. We compute the
texture loss $L_{T}$ and the patch GAN $L_{PGAN}$ between $I$ and
$\tilde{I}_{r}$, both of which are used to measure texture similarity. This is
done with the expectation that reshuffling texton appearances will not alter a
texture’s overall appearance (3 (Property 3 (Reshuffling Invarance).)). In the
disentangling part, we transform image $I$ to obtain $\hat{I}$ via
transformation $T$. $\\{\hat{\textbf{g}}\\}$ is obtained from encoding
$\hat{I}$. The same transformation $T$ is applied to $\\{\textbf{g}\\}$,
obtaining $T(\\{\textbf{g}\\})$. We compute the consistency loss $L_{Csis}$ to
enforce the similarity between latent Gaussians $\\{\hat{\textbf{g}}\\}$ and
$T(\\{\textbf{g}\\})$, and aim for reconstructing $\hat{I}$ from
$T(\\{\textbf{g}\\})$ by minimizing reconstruction loss $L^{\prime}_{R}$
between $\hat{I}$ and $\mathcal{G}(h(T(\\{\textbf{g}\\})))$. disentangling
encourages the encoding of image transformation within the transformation in
Gaussians (4 (Property 4 (Transformation Equivariance).)), thereby promoting
the disentanglement of structure and appearance.
LABEL:sub@fig:method_architecture:encoder illustrates a detailed view of the
encoder (Section 3.2), which comprises an appearance encoder
$\mathcal{E}_{FC}$, a segmentation network $\mathcal{E}_{s}$, and a Gaussian
parameter estimation layer $\mathcal{E}_{g}$. $\mathcal{E}_{FC}$ calculates an
appearance feature map $\mathbf{F}_{a}$ and direction map $\mathbf{V}$, which,
along with segmentations $\mathbf{S}$ generated from $\mathcal{E}_{s}$, are
used to compute latent Gaussians $\\{\textbf{g}\\}$. In accordance with 1
(Property 1 (Discrete Objectness).) and 2 (Property 2 (Spatial Compactness).),
entropy $L_{E}$ and compactness $L_{C}$ losses are applied to $\mathbf{S}$ to
foster hard and localized segmentation masks in order to inject these desired
properties into textons. Segmentations $\mathbf{S}$ are also employed in
computing $L_{Csis}$, as detailed in Section 3.5.2.

Figure 2 shows an overview of our autoencoder and unsupervised training
strategy, consisting of three major components: autoencoding, appearance
reshuffling, and disentangling. In autoencoding (Figure 2 autoencoding), the
encoder takes an input texture $I$ and generates its latent Gaussians
$\\{\textbf{g}\\}$. Then, the decoder employs a Gaussian splatting layer $h$
to convert latent Gaussians into a feature map [Epstein et al., 2022] that is
fed into the generator $\mathcal{G}$ to reconstruct the input texture $I_{r}$.
However, autoencoding is not sufficient to reliably learn the extraction of
textons as well as separation of texton geometry and appearance in a fully
unsupervised manner. While textons are not well-defined both conceptually and
mathematically, our unsupervised approach defines and extracts textons based
on a set of four desired properties. We then design the appearance reshuffling
and disentangling components to maintain these properties.

###### Property 1 (Discrete Objectness).

A texton embodies a distinct element within the image that can be identified
as either present or absent in its entirety, and a texture can be fully
described by a discrete set of textons. This discrete set representation
facilitates intuitive and interpretable image editing, allowing users to
engage in a more natural and user-friendly editing experience.

###### Property 2 (Spatial Compactness).

A neural texton describes the spatial extent and appearance of a local region
within a texture.

###### Property 3 (Reshuffling Invarance).

Textures inherently possess a stochastic characteristic that exhibits spatial
stationarity. Random reshuffling of appearances among a set of given textons
should span the stochastic space of a texture without modifying the texture’s
overall appearance. That is, reshuffled texton appearances should result in a
different version of the original texture.

###### Property 4 (Transformation Equivariance).

When neural textons undergo a spatial transformation, the resulting image
should reflect a consistent and corresponding transformation, preserving the
spatial relationships and patterns of the textons.

Our approach promotes 1 (Property 1 (Discrete Objectness).) and 2 (Property 2
(Spatial Compactness).) by applying corresponding compactness $L_{C}$ and
entropy $L_{E}$ losses in the autoencoding part (Figure 2 autoencoding). In
addition to the losses, we employ the inherently compact Gaussians to
represent textons. Aligned with 3 (Property 3 (Reshuffling Invarance).), the
reshuffling component modifies the Gaussian appearance features (Figure 2
reshuffling). These features are associated with input latent Gaussians
$\\{\textbf{g}\\}$, which are derived by encoding the input texture. The
generator $\mathcal{G}$ generates a new image $\tilde{I}_{r}$ from the
reshuffled latent Gaussians $\\{\tilde{\textbf{g}}\\}$. The generated image
$\tilde{I}_{r}$ is expected to closely resemble the input texture $I$ in terms
of texture similarity. The disentangling component (Figure 2 disentangling) is
to promote 4 (Property 4 (Transformation Equivariance).). We transform the
input texture and encode the tranformed texture $\hat{I}$ as latent Gaussians
$\\{\hat{\textbf{g}}\\}$. The same transformation $T$ is applied directly to
input latent Gaussians $\\{\textbf{g}\\}$. In accordance with 4 (Property 4
(Transformation Equivariance).), we expect that the Gaussians
$\\{\hat{\textbf{g}}\\}$ derived from transformed input textures should
closely match the transformed Gaussians $T(\\{\textbf{g}\\})$ from the
original input textures.

We next detail autoencoding including encoding textures as Gaussians (Section
3.2), Gaussian splatting (Section 3.3), and decoding splatted Gaussians as
textures (Section 3.4). In Section 3.5, we then introduce appearance
reshuffling (Section 3.5.1) and disentangling (Section 3.5.2).

###  3.2. Encoding Textures as Gaussians

####  3.2.1. Latent Gaussians

The encoder $\mathcal{E}$ maps an input texture $I\in\mathbb{R}^{H\times
W\times 3}$ into our structured latent representation $\mathbf{z}$, a
composition of Gaussians $\\{\textbf{g}_{i}\\}_{i=1}^{n}$ where each Gaussian
$\textbf{g}_{i}$ represents a texton and $n$ is the (maximum) number of
Gaussians. To facilitate learning the optimal number of Gaussians required for
a texture, we assign a weight parameter to each Gaussian $\delta_{i}\in[0,1]$.
As we model textures as a discrete set of textons (1 (Property 1 (Discrete
Objectness).)), $\textbf{g}_{i}$ exists in the set if $\delta_{i}$ is close to
1 and otherwise if $\delta_{i}$ is near 0. Each Gaussian is given by its
center $\boldsymbol{\mu}_{i}$ and covariance matrix $\mathbf{U}_{i}$ that
encodes the approximated shape of the texton. To enhance the encoding of
texton geometry and the subsequent geometry-appearance disentanglement, we
include a 2D unit-vector $\mathbf{\nu}_{i}\in\mathbb{R}^{2}$
($\|\mathbf{\nu}_{i}\|=1$) representing the anisotropic direction of textons
in addition to Gaussian covariance. Furthermore, $\textbf{g}_{i}$ contains an
appearance parameter given by a feature vector
$\mathbf{f}_{i}\in\mathbb{R}^{n_{f}}$, which is used to encode local texton
appearance. In summary, our compositional Gaussian representation $\mathbf{z}$
is defined as

(1) $\displaystyle\mathbf{z}=\ $
$\displaystyle\\{\textbf{g}_{i}\\}_{i=1}^{n}=\mathcal{E}(I)$

where

(2) $\displaystyle\textbf{g}_{i}=\ $
$\displaystyle\\{\delta_{i},\boldsymbol{\mu}_{i},\mathbf{U}_{i},\mathbf{\nu}_{i},\mathbf{f}_{i}\\}.$

####  3.2.2. Encoder Network Architecture

To encode image $I$ into latent Gaussians
$\mathbf{z}=\\{\textbf{g}_{i}\\}_{i=1}^{n}$, there are two options for
generating the Gaussian parameters (Equation 2): the first is to directly
predict these parameters via linear or MLP heads; the second, which we choose,
is to compute them indirectly from predicted heatmaps or segmentation maps.
The second option has demonstrated better performance in previous tasks such
as human pose estimation and unsupervised landmark detection [Zheng et al.,
2023; Zhang et al., 2018a]. Therefore, the encoder $\mathcal{E}$ includes
three components: an image encoder $\mathcal{E}_{FC}$, a segmentation network
$\mathcal{E}_{s}$, and a Gaussian parameter estimation layer
$\mathcal{E}_{g}$. The encoder $\mathcal{E}_{FC}$ is a shallow, fully
convolutional network (FCNN) in order to avoid encoding non-local structure
information. $\mathcal{E}_{FC}$ takes input images $I$ to generate feature
maps, and has two branches: the appearance branch produces an appearance
feature map $\mathbf{F}_{a}\in\mathbb{R}^{H\times W\times d_{a}}$ and the
geometry branch produces a direction map $\mathbf{V}\in\mathbb{R}^{H\times
W\times 2}$. The Mask2Former architecture is used [Cheng et al., 2022] as
$\mathcal{E}_{s}$ and trained end-to-end with the other networks.
$\mathcal{E}_{s}$ takes the image $I$ and generates pixel-wise, softmax-
normalized $n$ segmentation masks $\mathbf{S}\in[0,1]^{n\times H\times W}$.
Each segment is used to compute a Gaussian, and $n$ is also the number of
Gaussians. For more details about $\mathcal{E}_{FC}$ and $\mathcal{E}_{s}$,
please refer to Section A.7.1.

Given the $n$-segmentation masks $\mathbf{S}$, the appearance feature map
$\mathbf{F}_{a}$, and the directional map $\mathbf{V}$, $\mathcal{E}_{g}$
computes the composition of Gaussians $\\{\textbf{g}_{i}\\}_{i=1}^{n}$. The
mean $\boldsymbol{\mu}_{i}$ and covariance $\mathbf{U}_{i}$ of the
$i^{\text{th}}$ Gaussian $\textbf{g}_{i}$ is computed as the average and
covariance of image coordinates $\mathbf{p}=[x,y]$ weighted by the
$i^{\text{th}}$ segmentation mask $\mathbf{S}_{i}$,

(3) $\displaystyle\boldsymbol{\mu}_{i}=$
$\displaystyle\frac{\sum_{\mathbf{p}}\mathbf{p}\cdot\mathbf{S}_{i}(\mathbf{p})}{\mathfrak{S}_{i}}$
(4) $\displaystyle\mathbf{U}_{i}=$
$\displaystyle\frac{\sum_{\mathbf{p}}(\mathbf{p}-\boldsymbol{\mu}_{i})^{T}(\mathbf{p}-\boldsymbol{\mu}_{i})\cdot\mathbf{S}_{i}(\mathbf{p})}{\mathfrak{S}_{i}}$

where $\mathfrak{S}_{i}=\sum_{\mathbf{p}}\mathbf{S}_{i}(\mathbf{p})$ is the
normalization factor computed from the ${i}^{\text{th}}$ segment.

The Gaussian weight $\delta_{i}$ is used to adaptively adjust the number of
total Gaussians by indicating the existence of each Gaussian. While this
existence weight should ideally be binary, during model training it is crucial
to ensure that gradients are effectively back-propagated through the entire
network. Hence, we introduce a differentiable, randomized sampling mechanism
for computing $\delta_{i}\sim B(p_{i})$ as a sample from Bernoulli
distribution $B$, which takes the value 1 with probability $p_{i}$ computed as

(5)
$p_{i}=\frac{\sum_{\mathbf{p}}\mathbf{S}_{i}(\mathbf{p})\mathbf{S}_{i}(\mathbf{p})}{\mathfrak{S}_{i}}.$

Equation 5 computes $p_{i}$ as the average of mask $\mathbf{S}_{i}$ weighted
by $\mathbf{S}_{i}$. By computing the average of these mask probabilities
weighted by the probabilities themselves, we essentially emphasize the higher
probabilities, as higher values will contribute more to the average. If the
mask has either high probability values (such as in a local region) or zero
values (like those outside the region), which might indicate a generally
reliable mask, $p_{i}$ will be high and $\delta_{i}\sim B(p_{i})$, thus the
Gaussian $\textbf{g}_{i}$ is more likely to exist ($\delta_{i}=1$).
Conversely, if many pixels have lower but non-zero probability values,
implying uncertainty in the segmentation, $p_{i}$ will be low and the Gaussian
$\textbf{g}_{i}$ is more likely not to exist ($\delta_{i}=0$).

Because sampling from a (categorical) Bernoulli distribution is not
differentiable, during training we apply the Gumbel-softmax trick [Jang et
al., 2016] to approximate the hard sampling process from a categorical
distribution. During inference, we round the continuous values of
$p_{i}\in[0,1]$ to obtain binary values $\delta_{i}\in\\{0,1\\}$, aligning
with 1 (Property 1 (Discrete Objectness).). We find that using the categorical
sampling mechanism when computing Gaussian weights (Equation 5) generates more
aligned and more reliable texton segmentations and thus higher-quality latent
Gaussians. We hypothesize that the sampling mechanism inherently promotes
weights to converge towards 0 or 1, thereby ensuring binary outcomes, while
simultaneously maintaining exploration capabilities through randomized
sampling.

Finally, the appearance feature $\mathbf{f}_{i}$ and direction
$\mathbf{\nu}_{i}$ are computed as the weighted average (pooling) of image
appearance features $\mathbf{F}_{a}$ and directional map $\mathbf{V}$ within
each image segment as

(6) $\displaystyle\mathbf{f}_{i}=$
$\displaystyle\frac{\sum_{\mathbf{p}}\mathbf{F}_{a}(\mathbf{p})\mathbf{S}_{i}(\mathbf{p})}{\mathfrak{S}_{i}}$
(7) $\displaystyle\mathbf{\nu}_{i}=$
$\displaystyle\frac{\sum_{\mathbf{p}}\mathbf{V}(\mathbf{p})\mathbf{S}_{i}(\mathbf{p})}{\mathfrak{S}_{i}}$

followed by normalization $\mathbf{f}_{i}=\mathbf{f}_{i}/\|\mathbf{f}_{i}\|$,
$\mathbf{\nu}_{i}=\mathbf{\nu}_{i}/\|\mathbf{\nu}_{i}\|$.

####  3.2.3. Losses

In the process of encoding textures as Gaussisans
$\\{\textbf{g}_{i}\\}_{i=1}^{n}$, we calculate several losses on the
intermediate segmentation masks $\mathbf{S}$ during end-to-end training. This
includes the entropy loss $L_{E}$, which encourages reliable hard segmentation
$\mathbf{S}$ and thus 1 (Property 1 (Discrete Objectness).),

(8)
$L_{E}=-\frac{1}{HW}\sum_{\mathbf{p}}\sum_{i=1}^{n}\mathbf{S}_{i}(\mathbf{p})\log(\mathbf{S}_{i}(\mathbf{p})).$

$L_{E}$ is minimized when the softmax-normalized probabilities in segmentation
mask $\mathbf{S}$ are hard with values either 0 or 1. We also apply a
compactness loss $L_{C}$ [Yang et al., 2020a], especially during the initial
phase of training on the segmentation masks so that each segment
$\mathbf{S}_{i}$ and thus $\textbf{g}_{i}$ is spatially compact and localized,
in accordance to 2 (Property 2 (Spatial Compactness).),

(9)
$\displaystyle\mathbf{p}^{\prime}=\sum_{i}\boldsymbol{\mu}_{i}\cdot\mathbf{S}_{i}(\mathbf{p})$
(10) $\displaystyle
L_{C}=\frac{1}{HW}\sum_{\mathbf{p}}\lVert\mathbf{p}^{\prime}-\mathbf{p}\rVert_{2}^{2}.$

###  3.3. Gaussian Splatting

After obtaining latent Gaussians $\\{\textbf{g}_{i}\\}_{i=1}^{n}$, the
Gaussian splatting (GS) layer converts $\\{\textbf{g}_{i}\\}_{i=1}^{n}$ into a
feature grid $\mathbf{F}_{gs}$, which is then fed into a generator to
reconstruct the input image as $I_{r}$. To define the Gaussian splatting
function $h:\\{\textbf{g}_{i}\\}_{i=1}^{n}\rightarrow\mathbf{F}_{gs}$, we
first compute the squared Mahalanobis distance $M_{i}^{2}$ of $\mathbf{p}$ to
the Gaussian center/mean $\boldsymbol{\mu}_{i}$,

(11)
$M_{i}^{2}(\mathbf{p})=(\mathbf{p}-\boldsymbol{\mu}_{i})\mathbf{U}_{i}^{-1}(\mathbf{p}-\boldsymbol{\mu}_{i})^{T}.$

Gaussians are derived from pixel-wise normalized image segmentation masks.
This approach inherently suggests that each pixel is exclusively associated
with a single segmentation mask, and consequently, with a single latent
Gaussian. However, in the context of texture editing applications, we permit
modifications to these Gaussians during inference, which may result in
potential overlaps between the edited Gaussians. For managing these overlaps,
we adopt alpha-compositing in Gaussian splatting, as in [Epstein et al.,
2022]. Alpha-compositing takes into account the order of Gaussians in
instances of overlap. It is important to note that in cases where there is no
overlap, for example, during training, the ordering of the Gaussians does not
influence the splatting results.

In the alpha-compositing process, it is necessary to define the opacity $o$
for each Gaussian. $o_{i}(\mathbf{p})$ at $\mathbf{p}$ for $i^{\text{th}}$
Gaussian $\textbf{g}_{i}$ is calculated as

(12) $o_{i}(\mathbf{p})=\delta_{i}\exp({-M_{i}^{2}(\mathbf{p})}).$

Note that we use $-M^{2}$ instead of $-M^{2}/2$ in Equation 12 to obtain
sharper Gaussian splatting. In addition, the compositional alpha
$\alpha_{i}(\mathbf{p})$ at $\mathbf{p}$ for the $i^{\text{th}}$ Gaussian is

(13)
$\alpha_{i}(\mathbf{p})=o_{i}(\mathbf{p})\prod_{j=i+1}^{n}\left(1-o_{j}(\mathbf{p})\right).$

Finally, the Gaussian splatting function
$h:\\{\textbf{g}_{i}\\}_{i=1}^{n}\rightarrow\mathbf{F}_{gs}$ is defined as

(14)
$\mathbf{F}_{gs}(\mathbf{p})=\sum_{i}\alpha_{i}(\mathbf{p})\textsf{concat}(\mathbf{f}_{i},\mathbf{\nu}_{i}),$

where the function concat concatenates the feature
$\mathbf{f}_{i}\in\mathbb{R}^{n_{f}}$ and direction vectors
$\mathbf{\nu}_{i}\in\mathbb{R}^{2}$, and
$\mathbf{F}_{gs}\in\mathbb{R}^{H\times W\times(n_{f}+2)}$ is the alpha-
composited result of these Gaussian splatted features.

###  3.4. Decoding Images from Splatted Gaussians

####  3.4.1. Network architecture

We adopt the SPADE generator [Park et al., 2019] as the generator
$\mathcal{G}$ to reconstruct $I_{r}$ from $\mathbf{F}_{gs}$. The original
SPADE generator takes a learned constant feature map and is injected with
resized $\mathbf{F}_{gs}$ at multiple levels. Instead, we replace the learned
constant feature map with downscaled $\mathbf{F}_{input}=\max_{i}\alpha_{i}$
as input to SPADE so it can generate textures of arbitrary resolutions. For
more details, please refer to Section A.7.1.

####  3.4.2. Losses

The reconstruction is ensured using a mixed reconstruction loss function

(15) $L_{R}=w_{1}L_{1}+w_{LPIPS}L_{LPIPS}$

where $L_{1}=\|I-I_{r}\|_{1}$ and $L_{LPIPS}$ denotes LPIPS loss [Zhang et
al., 2018b] computed between $I$ and $I_{r}$. We set $w_{1}=2$ and
$w_{LPIPS}=0.2$. In addition, we wish for the reconstructed images $I_{r}$ to
be realistic by employing a discriminator $\mathcal{D}$. We apply the non-
saturating adversarial loss on the reconstructed images to train the encoder
$\mathcal{E}$ and generator $\mathcal{G}$

(16) $L_{GAN}=-\log\left(\mathcal{D}\left(I_{r}\right)\right).$

For details about the discriminator $\mathcal{D}$ and its losses, please see
Section A.7.

###  3.5. Unsupervised Compositional Textons Learning

We utilize Gaussians to represent textons which we wish for exhibiting four
properties, as discussed in Section 3.1. We have introduced mechanisms to
enforce 1 (Property 1 (Discrete Objectness).) and 2 (Property 2 (Spatial
Compactness).) in Section 3.2. In this section, we introduce our approach to
inject 3 (Property 3 (Reshuffling Invarance).) and 4 (Property 4
(Transformation Equivariance).) into the neural textons.

####  3.5.1. Reshuffling-invariant neural textons

In accordance with 3 (Property 3 (Reshuffling Invarance).), random reshuffling
of texton appearances should maintain the original geometry encoding,
resulting in a different version of original texture without altering overall
texture appearance. More specifically, we randomly reshuffle, denoted as
$\pi$, appearance feature $\mathbf{f}_{i}$ associated with $\textbf{g}_{i}$
within $\\{\textbf{g}_{i}\\}$ to obtain
$\\{\tilde{\textbf{g}}_{i}\\}=\pi(\\{\textbf{g}_{i}\\})$, which is then
reconstructed as image $\tilde{I}_{r}$ that should resemble the input image
$I$. During inference, when dealing with valid discrete textons where the
Gaussian weight $\delta$ is rounded to 1, we can describe the reshuffling
process using a square binary permutation matrix. In the training phase,
however, as the network evolves to determine the ideal number of Gaussians by
continuously adjusting the Gaussian weight $\delta$, it is crucial to restrict
feature reshuffling exclusively to Gaussians that effectively represent
individual textons. For example, when the Gaussian weight/existence
$\delta_{i}/p_{i}$ or the area of its texton segmentation mask
$A_{i}=\sum_{\mathbf{p}}\mathbf{S}_{i}$ is very small or approaching 0
($p_{i}\approx 0$ or $\delta_{i}\approx 0$ or $A_{i}\approx 0$), we anticipate
that the latent Gaussian $\textbf{g}_{i}$ and its associated appearance
feature $\mathbf{f}_{i}$ do not effectively represent a texton and should,
therefore, be discouraged from appearance feature reshuffling. In this
randomized reshuffling process
$\\{\tilde{\textbf{g}}_{i}\\}_{i=1}^{n}=\pi(\\{\textbf{g}_{i}\\}_{i=1}^{n})$,
say the appearance feature $\mathbf{f}_{i}$ in $\textbf{g}_{i}$ is exchanged
with $\mathbf{f}_{j}$ in $\textbf{g}_{j}$, $\textbf{g}_{i}/\textbf{g}_{j}$
becomes $\tilde{\textbf{g}}_{i}/\tilde{\textbf{g}}_{j}$ and the resulting
appearance feature $\tilde{\mathbf{f}_{j}}$ in $\tilde{\textbf{g}}_{j}$ is
calculated as

(17)
$\tilde{\mathbf{f}_{j}}=\tau_{ij}\mathbf{f}_{i}+(1-\tau_{ij})\mathbf{f}_{j}$

where

(18)
$\tau_{ij}=\left(\min\left(\max\left(\frac{A_{i}}{A_{j}},\frac{p_{i}}{p_{j}}\right),1\right)\right)^{\gamma}.$

Equation 17 and Equation 18 define the basic pairwise feature reshuffling
operation, namely feature swapping, used in the feature reshuffling $\pi$
process. $\tau_{ij}\in[0,1]$ is the swapping coefficient between
$\mathbf{f}_{i}$ and $\mathbf{f}_{j}$. $\gamma=0.5$ is a non-negative
temperature hyperparameter, which will be further discussed later, controlling
the softness of the swapping/reshuffling operation. Since $\delta_{i}$ and
$p_{i}$ are similar concepts, we find that using one of them $p_{i}$ is
sufficient for computing $\tau_{ij}$ in Equation 18. A Gaussian can either
effectively represent a neural texton or not. We consider a Gaussian g to
effectively represent a texton if both associated $A\gg 0$ and $p\gg 0$.
Conversely, a Gaussian g is deemed ineffective in representing a texton if
$A\approx 0$ or $p\approx 0$. Accordingly, there are four distinct cases to
consider when computing the reshuffling coefficient $\tau_{ij}$ (Equation 17)
between a pair of Gaussians $\textbf{g}_{i},\textbf{g}_{j}$, which are
detailed below.

  1. (1)

If both $\textbf{g}_{i}$ ($A_{i}\gg 0$ and $p_{i}\gg 0$) and $\textbf{g}_{j}$
($A_{j}\gg 0$ and $p_{j}\gg 0$) are effective in representing textons, the
swapping is effective with $\tau_{ij}\gg 0$.

  2. (2)

If $\textbf{g}_{i}$ is effective ($A_{i}\gg 0$ and $p_{i}\gg 0$), but
$\textbf{g}_{j}$ is not ($A_{j}\approx 0$ or $p_{j}\approx 0$), Equation 18
gives us $\tau_{ij}=1$ and thus the swapping is effective. But if
$\tilde{p}_{j}=p_{j}\approx 0$ ($\tilde{p}_{j}\in\tilde{\textbf{g}}_{j}$), the
swapping has no effect on the reconstructed images and is in fact ineffective,
because the Gaussian splatting (Section 3.3) can ignore Gaussians with
$\tilde{p}_{j}\approx 0$ (and thus $\tilde{\delta}_{j}\approx 0$).

  3. (3)

If $\textbf{g}_{i}$ is not effective ($A_{i}\approx 0$ or $p_{i}\approx 0$),
but $\textbf{g}_{j}$ is effective ($A_{j}\gg 0$ and $p_{j}\gg 0$), there are
three subcases: the swapping is not effective with $\tau_{ij}\approx 0$ when
both $A_{i}\approx 0$ and $p_{i}\approx 0$, the swapping is effective with
$\tau_{ij}\gg 0$ but has no effect on the reconstruction when $A_{i}\gg 0$ and
$p_{i}\approx 0$, and otherwise the swapping is effective.

  4. (4)

If neither $\textbf{g}_{i}$ ($A_{i}\approx 0$ or $p_{i}\approx 0$) nor
$\textbf{g}_{j}$ ($A_{j}\approx 0$ or $p_{j}\approx 0$) are effective,
$\tau_{ij}$ can be any values between $[0,1]$. Even if it is possible that
$\tau_{ij}\gg 0$, the swapping may still be ineffective if
$\tilde{p}_{j}=p_{j}\approx 0$ and thus $\tilde{\delta}_{j}\approx 0$.

Next, we discuss the effect of $\gamma$. When $\gamma=1$, Equation 18 behaves
normally; when $\gamma<1$, reshuffling is harder and tends to include
Gaussians that may not effectively represent a texton: an extreme case is when
$\gamma=0$, and the operation is equivalent to hard, discrete reshuffling
because we always have $\tau_{ij}=1$ with
$\tilde{\mathbf{f}_{j}}=\mathbf{f}_{i}$.

We can describe the randomized reshuffling process $\pi$ in a matrix form by
extending pairwise feature swapping (Equation 17). $\pi$ can be described as a
convex combination of both unshuffled and hard reshuffled (permuted) features
with the coefficient matrix $\mathbf{\tau}$ of size $n\times n$ as

(19)
$\tilde{\mathbf{F}}=\mathbf{\tau}\mathbf{F}+(1-\mathbf{\tau})\mathbf{P}\mathbf{F}$

where
$\mathbf{F}=[\mathbf{f}_{1}^{T},\mathbf{f}_{2}^{T},\ldots,\mathbf{f}_{n}^{T}]^{T}$
are the unshuffled features, $\mathbf{P}\in\mathbf{P}_{n}$ is a random
permutation matrix, and $\mathbf{P}_{n}$ is the set of all permutation
matrices of size $n\times n$. The multiplication $\mathbf{P}\mathbf{F}$ yields
the hard reshuffled features.

In accordance with 3 (Property 3 (Reshuffling Invarance).), appearance feature
reshuffling should not alter overall texture appearance. To this end, we
calculate texture loss $L_{T}$ between the original input $I$ and image
$\tilde{I}_{r}=\mathcal{G}\left(h(\\{\tilde{\textbf{g}}_{i}\\})\right)$
reconstructed from Gaussians $\\{\tilde{\textbf{g}}_{i}\\}$ with reshuffled
features by using the patch-based texture loss defined in [Zhou et al., 2023].
In addition, we use non-saturating GAN losses on image patches [Park et al.,
2020] to train $\mathcal{E}$ and $\mathcal{G}$

(20)
$L_{PGAN}=-\textsf{mean}\left(\log\left(\mathcal{D}_{p}\left(\textsf{randcrop}(\tilde{I}_{r})|\textsf{randcrop}(I)\right)\right)\right)$

where the function randcrop indicates random image cropping. randcrop crops
random image patches of size $\frac{H}{2}\times\frac{W}{2}$. $\mathcal{D}_{p}$
is a patch discriminator performing on local patches of
$\textsf{randcrop}(\tilde{I}_{r})$. Given real cropped patches
$\textsf{randcrop}(I)$ from input texture $I$ as conditions, $\mathcal{D}_{p}$
is optimized to discriminate if patches are from the same texture $I$ (real)
or not (fake), and $\mathcal{E}$ and $\mathcal{G}$ are optimized to fool
$\mathcal{D}_{p}$ by generating $\tilde{I}_{r}$ with similar local patches as
$I$. In the implementation, the real patches are concatenated to real/fake
patches before feeding into $\mathcal{D}_{p}$. 8 crops are made for each (real
or fake) image. The function mean calculates the mean of a vector because
$\mathcal{D}_{p}$ executes on patches and returns a vector of predictions.
Please refer to Section A.7 for more implementation details.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(a) Input

(b) Seg. overlay

(c) Gaussians

(d) Output 1

(e) Output 2

(f) Output 3

Figure 3. Texture diversification. Given the input
LABEL:sub@fig:diverse_texture_synthesis:exemplar, our network processes it to
extract textons via segmentation
LABEL:sub@fig:diverse_texture_synthesis:seg_overlay. Each texton is
represented by a Gaussian calculated from segmentations, as illustrated in
LABEL:sub@fig:diverse_texture_synthesis:Gaussians. We can generate different
versions LABEL:sub@fig:diverse_texture_synthesis:output1
LABEL:sub@fig:diverse_texture_synthesis:output2
LABEL:sub@fig:diverse_texture_synthesis:output3 of the same texture by
randomly reshuffling appearance features in latent Gaussians.

####  3.5.2. Equivariance of geometric transformation

Per 4 (Property 4 (Transformation Equivariance).), our objective is to achieve
consistent results by first extracting the Gaussians and then applying
transformations, as compared to first transforming the image and subsequently
transforming the Gaussians. The consistency guarantees that the latent
Gaussians effectively separate the encoding of texton geometry and appearance,
and maintain robustness against transformations applied to the image.
Concretely, a consistency loss $L_{Csis}$ is defined to measure the
discrepancies between two sets of Gaussians $T(\mathcal{E}(I))$ and
$\mathcal{E}(T(I))$. The first set, $T(\mathcal{E}(I))$, is derived by
initially extracting Gaussians from the input texture $I$ using our encoder
$\mathcal{E}$, and subsequently applying the transformation $T$; the second
set, $\mathcal{E}(T(I))$, is obtained by first applying the transformation $T$
to the input texture $I$, and then extracting Gaussians with the encoder
$\mathcal{E}$. A matching $\sigma$ is required to compute the consistency loss
between the two sets of Gaussians. Before presenting the formal definition of
the consistency loss $L_{Csis}$, we first outline the procedure for computing
$\sigma$ between, in order to simplify notations, two arbitrary sets of
Gaussians
$\\{\textbf{g}_{i}\\}_{i=1}^{n},\\{\textbf{g}^{\prime}_{j}\\}_{j=1}^{n}$.

As in [Carion et al., 2020], we use the Hungarian algorithm [Kuhn, 1955] to
compute the matching $\sigma$. The Hungarian algorithm processes a cost
matrix, where each entry specifies the cost associated with matching a
Gaussian $\textbf{g}_{i}$ from the set $\\{\textbf{g}_{i}\\}_{i=1}^{n}$ to a
Gaussian $\textbf{g}^{\prime}_{j}$ from the other set
$\\{\textbf{g}^{\prime}_{j}\\}_{j=1}^{n}$. The entry $\mathbf{C}_{ij}$ at
$i^{\text{th}}$ row, $j^{\text{th}}$ column of the cost matrix
$\mathbf{C}\in\mathbb{R}^{n\times n}$ is computed from $\textbf{g}_{i}$ and
$\textbf{g}^{\prime}_{j}$ as

(21) $\displaystyle\mathbf{C}_{ij}=$ $\displaystyle
w_{ij}\beta_{ij}(\lambda_{\boldsymbol{\mu}}\|\boldsymbol{\mu}_{i}-\boldsymbol{\mu}_{j}^{\prime}\|_{2}+\lambda_{\mathbf{U}}\|\mathbf{U}_{i}-\mathbf{U}_{j}^{\prime}\|_{2}+\lambda_{\mathbf{\nu}}\|\mathbf{\nu}_{i}-\mathbf{\nu}^{\prime}_{j}\|_{2}$
$\displaystyle+$
$\displaystyle\lambda_{\mathbf{f}}\|\mathbf{f}_{i}-\mathbf{f}_{j}^{\prime}\|_{2})+\lambda_{p}\|p_{i}-p_{j}^{\prime}\|_{2}+\lambda_{\mathbf{S}}\|\mathbf{S}_{i}-\mathbf{S}^{\prime}_{j}\|_{2}$

where $w_{ij}=p_{i}p_{j}^{\prime}$ is employed to ignore low-existence
Gaussians when computing differences in “non-existence” Gaussian parameters
$\boldsymbol{\mu},\mathbf{U},\mathbf{f},\mathbf{\nu}$, and
$\beta_{ij}\in[0.1,1]$ is the weight for reducing the impact of matching
lower-accuracy textons near the image boundaries. $\beta_{ij}$ is lower for
pairs of Gaussians located near image boundaries and equals 1 for other
Gaussian pairs. The term computing the difference in $p$ encourages the number
of Gaussian textons to be consistent across image transformations, and the
difference in $\mathbf{S}$, which can increase the matching robustness,
promotes equivariant segmentations. Then the matching cost between two sets of
Gaussians $\\{\textbf{g}\\}$, $\\{\textbf{g}^{\prime}\\}$ is computed as the
minimum sum of matching cost between the two sets

(22)
$\mathcal{C}(\\{\textbf{g}_{i}\\}_{i=1}^{n},\\{\textbf{g}^{\prime}_{j}\\}_{j=1}^{n})=\min_{\sigma}\sum_{i,\sigma(i)}\mathbf{C}_{ij}.$

Using the matching cost defined between two arbitrary sets of Gaussians, we
define the consistency loss between $T(\mathcal{E}(I))$ and
$\mathcal{E}(T(I))$ as

(23)
$L_{Csis}=\mathcal{C}\left(T\left(\mathcal{E}(I)\right),\mathcal{E}\left(T(I)\right)\right).$

We filter out Gaussians and segmentation masks transformed outside of image
boundaries prior to applying Equation 21. In our paper, we set
$\lambda_{\boldsymbol{\mu}}=1.2$, $\lambda_{\mathbf{U}}=2$,
$\lambda_{\mathbf{f}}=10$, $\lambda_{\mathbf{\nu}}=0.01$, $\lambda_{p}=4$,
$\lambda_{\mathbf{S}}=200$.

In addition, we calculate the reconstruction loss $L^{\prime}_{R}$, using the
same formulation as in Equation 15, but between transformed images
$\hat{I}=T(I)$ and images reconstructed from $T\left(\mathcal{E}(I)\right)$.
Again, we mask out image regions transformed outside image boundaries.

###  3.6. Overall Losses and Training Procedure

To sum up, we train the generation network ($\mathcal{E},h$ and $\mathcal{G}$)
with the following losses

(24)
$\begin{split}L=L_{R}+L^{\prime}_{R}+w_{E}L_{E}+w_{C}L_{C}+w_{Csis}L_{Csis}+w_{T}L_{T}+\\\
w_{GAN}L_{GAN}+w_{PGAN}L_{PGAN}\end{split}$

During training, we fix $w_{T}=0.01,w_{GAN}=0.1,w_{PGAN}=0.1$. We adjust
$w_{E},w_{C},w_{Csis}$ during training to maintain a balance in the
unsupervised training process. This ensures that the reduction of one loss
does not disproportionately occur at the expense of the others.

We train our model using $128\times 128$ images for 950K steps and then fine-
tune it using $256\times 256$ images for 75K steps. Since some of the network
components, including segmentation network $\mathcal{E}_{s}$ and the
discriminator $\mathcal{D}$, cannot take arbitrary-resolution input, we
downsample $256\times 256$ images back to $128\times 128$ before feeding into
$\mathcal{E}_{s}$, randomly crop $128\times 128$ patches before feeding into
$\mathcal{D}$ and resize as $96\times 96$ before feeding into
$\mathcal{D}_{p}$. $\mathcal{E}_{s}$ is freezed during fine-tuning. We want to
emphasize $\mathcal{E}_{s}$ is primarily employed to generate spatial Gaussian
parameters. In this context, the resolution of the segmentation is not a
critical factor affecting the overall quality of the output. Our results and
comparisons are in $256\times 256$ except ablation studies where $128\times
128$ images are sufficient to show the differences. See Section A.7 for more
implementation details about our network architectures, losses, etc.

###  3.7. Dataset

We collect a high-quality texture dataset consisting of 377K synthetic and
1.2K real stock images. The synthetic data is from Adobe Firefly’s text-to-
image model using 2500 detailed texture descriptions generated with ChatGPT.
We use 90% images for training and 10% images for testing (applications and
analysis). The dataset is further augmented with random affine transformations
and color histogram matching with a random reference image from the dataset.
Please see Section A.1 for some random data samples.

##  4\. Applications

In this section, we showcase a variety of applications for our compositional
neural texture representation, including texture diversification (Section
4.1), texture transfer and image stylization (Section 4.2), modifying texture
variations (Section 4.3), texture interpolation (Section 4.4), direct
manipulation of textons (Section 4.5), and animated textures (Section 4.6).
These applications involve manipulating the compositional latent Gaussians,
and efficiently generating output textures by feeding the manipulated
Gaussians through the generator network in a feed-forward manner. For
reference, the feed-forward processing of a $256\times 256$ image using our
autoencoder takes 0.16 seonds on a single NVIDIA A10G (24GB) in PyTorch, and
this is achieved without any specific optimizations for speed.

appearancestructure

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

Seg. overlay

(u)

(v)

(w)

(x)

(y)

Figure 4. Texture transfer. This application involves appearance-providing
(left column) and structure-providing (top row) textures. The corresponding
transfer results are shown at the row-column intersections. We also visualize
the segmentation masks $\mathbf{S}$ of the structure-providing inputs (bottom
row).

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 5. Stylizing an image by tranferring textures. Our approach is also adaptable to image stylization. In each row of the presented results, the original content is shown on the left, followed by the stylized outputs with the corresponding texture overlays in the top right corners.  |  |
---|---
|
|
|  |
---|---
Figure 6. Spatial control in texture transfer. We can apply textures to
specific areas of an image as designated by image masks. From left to right:
masks, textures, original images, stylized outputs.

(a)

(b)

(c)

(d)

Figure 7. Scale control in texture transfer. Content image is shown in the
first column. From second to last columns, the transferred results with large,
median (2/3 of large scale) and small (2/3 of median scale) textons. The
texture is shown on top right corner of the second column.

###  4.1. Texture Diversification

Textures can be characterized by their inherent randomness, meaning that there
are countless variations of the same texture while maintaining its overall
look. Our method treats a texture as a combination of neural textons, and the
randomness is (partially) encapsulated in the distribution of appearance
features. By reshuffling these appearance features among effective textons
(with $\delta=1$), we can create diverse versions of the same texture, as
shown in Figures 3d, 3e and 3f. This method validates that our neural textons
accurately encapsulate the texture’s composition, showing that feature
reshuffling creates a subspace of variations of a texture.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k) $\Delta_{\mathbf{f}}=0$

(l) $\Delta_{\mathbf{f}}=0.5$

(m) Input

(n) $\Delta_{\mathbf{f}}=1.5$

(o) $\Delta_{\mathbf{f}}=2$

Figure 8. Revealing and modifying appearance variations. Our method is capable
of either reducing or increasing variations in the appearance of a texture by
correspondingly changing the appearance feature variations. With the input
images shown in the center column, we demonstrate reduced variations towards
the left and increased variations towards the right.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p) $\Delta_{\mathbf{U}}=0$

(q) $\Delta_{\mathbf{U}}=1/\sqrt{2}$

(r) Input

(s) $\Delta_{\mathbf{U}}=\sqrt{2}$

(t) $\Delta_{\mathbf{U}}=2$

Figure 9. Revealing and modifying geometric variations. Our method is capable
of either decreasing or increasing variations in the geometry of textons by
changing the the variations in Gaussian covariances. With the input images and
the corresponding visualized Gaussians in the center, we demonstrate reduced
and increased variations towards the left and right.

(a)

(b)

(c)

(d)

(e)

(f)

(a) source

(b) $\eta=0.25$

(c) $\eta=0.5$

(d) $\eta=0.75$

(e) target

(f) morphing

Figure 10. Texture interpolation and morphing. Given a source texture
LABEL:sub@fig:texture_interpolation:source and a target texture
LABEL:sub@fig:texture_interpolation:target, we can generate interpolations at
various degrees between the source and the target, controlled by interpolation
coefficients of $\eta=0.25$ LABEL:sub@fig:texture_interpolation:0p25,
$\eta=0.5$ LABEL:sub@fig:texture_interpolation:0p5, and $\eta=0.75$
LABEL:sub@fig:texture_interpolation:0p75. Furthermore, the interpolation
coefficients can vary spatially within one output, such as linearly increasing
from left to right. This variation leads to texture morphing, as demonstrated
in LABEL:sub@fig:texture_interpolation:morphing.

###  4.2. Texture Transfer

Texture transfer combines the structure and appearance of two different
textures: the first provides the structure while the second provides the
appearance. Our compositional texture representations effectively separate the
spatial structure and appearance characteristics within a texture, which is
key for successful texture transfer. Our transferred results are created by
taking the Gaussians from the structure-providing texture, appearance features
from the appearance-providing texture, and we only adjust the means of
Gaussian appearance features of the structure-providing texture to match those
of the appearance-providing texture. Figure 4 shows a grid of texture transfer
results. Note for texture transfer, we can also adjust other statistics beyond
the means. For example, we can directly replace appearance features from
structure-providing textures with those from appearance-providing textures.
This alternative generates transfer results that preserve less appearance
information from the structure-providing textures (see Section A.4 for
comparisons and Figure 1b (left)). We will leave studying other transfer
mechanisms as future works. Unless otherwise noted, we transfer textures by
adjusting means of appearance features by default.

In addition, we can stylize a natural image using textures. While our network
is not trained on natural images, a small patch of a natural image can be
effectively treated as a texture. Initially, we run our encoder $\mathcal{E}$
on overlapping content image patches to generate compositional Gaussians for
each patch. These Gaussian patches are merged in overlapping areas to ensure
smooth transitions, similarly to [Efros and Freeman, 2001]. Concretely, we
build correspondences between Gaussians from two patches in the overlapping
area, and then we interpolate their parameters based on the correspondences.
Subsequently, we adjust the feature mean of these smoothly merged Gaussians to
match that of the appearance-providing texture. Finally, the stylized image is
reconstructed from these mean-aligned Gaussians, resulting in a harmonious
blend of the texture appearance and structure from the natural image. Figure 5
shows the image stylization results. Users can also transfer different
textures to image regions specified by control masks, which could be later
used to create interactive brush tools. In this case, we adjust the means of
appearance features within each image region respectively to match those of
the corresponding appearance-providing texture. Figure 6 shows the transfer of
three different textures to three image regions.

Moreover, users can control the scale of transferred textons, as demonstrated
in Figure 7 and more in Section A.4. This control is achieved by varying the
scale of Gaussians derived from content image patches. To produce Gaussians on
a smaller scale for a content image, we initially sample reduced-size patches
from the content image. These smaller patches are then upscaled to the
standard resolution of $128\times 128$ pixels, which corresponds to the
resolution of input images required for the segmentation network
$\mathcal{E}_{s}$. Post-upscaling, these patches are encoded into latent
Gaussians. These latent Gaussians are subsequently downscaled and positioned
at the corresponding locations within original sampled patches. They are then
merged with Gaussians from other patches to maintain smooth transitions.
Conversely, to generate Gaussians on a larger scale, the process is reversed,
involving the sampling of larger patches and appropriate scaling adjustments.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u) Input

(v) Move

(w) Scale

(x) Rotate

Figure 11. Texton editing. In order to edit individual textons within a
texture, we can apply a sequence of edits (move, scale and rotate) to its
latent Gaussians.

(a)

(b)

(c)

(d)

(a) Input

(b) Edited

(c) Propagated edits

(d) Seg. overlay

Figure 12. Edit propagation. Given a texture LABEL:sub@fig:edit_prop:input, we
can edit it LABEL:sub@fig:edit_prop:edited and the edits can be propagated to
other similar locations LABEL:sub@fig:edit_prop:output based on extracted
textons LABEL:sub@fig:edit_prop:seg, for both stationary (bottom) and non-
stationary (top) structures.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p) Initial frame

(q) Frame at 1/4 length

(r) Frame at 1/2 length

(s) Frame at 3/4 length

(t) Final frame

Figure 13. Texture animation. The top shows the animation created from a
texture and perturbed shear flow. The bottom uses vortex flow. Please see the
supplementary video for dynamic effects.

(a)

(b)

(c)

(d)

(e)

(f)

(g) Input

(h) [Meta, 2023]

(i) Ours

Figure 14. Comparison to segment anything (SA). SA is trained with supervision
of 1 billion masks. However, SA is incapable of detecting and extracting
textons from textures. Our model can extract textons from textures using fully
unsupervised training.

###  4.3. Revealing and Modifying Texture Variations

We can modify non-local texture variations by editing variations within the
Gaussian’s appearance and geometry parameters. To quantify the variations, we
first calculate the mean of Gaussian appearance features $\mathbf{f}_{i}$ and
covariances $\mathbf{U}_{i}$ as

(25)
$\displaystyle\overline{\mathbf{f}}=\frac{\sum_{i}\mathbf{f}_{i}\delta_{i}}{\sum_{i}\delta_{i}}$
(26)
$\displaystyle\overline{\mathbf{U}}=\frac{\sum_{i}\mathbf{U}_{i}\delta_{i}}{\sum_{i}\delta_{i}},$

where $\delta_{i}\in\\{0,1\\}$ in the inference process. Equation 26 ignores
ineffective Gaussians when computing the means. The variations in edited
appearance features $\mathbf{f}_{i}^{*}$ can be controlled by a coefficient
$\Delta_{\mathbf{f}}>0$,

(27)
$\mathbf{f}_{i}^{*}=\overline{\mathbf{f}}+\Delta_{\mathbf{f}}(\mathbf{f}_{i}-\overline{\mathbf{f}}).$

Note when $\Delta_{\mathbf{f}}=1$, the features are not edited as
$\mathbf{f}_{i}^{*}=\mathbf{f}_{i}$ and the input image is reconstructed.
While the differences in appearance features can be computed as simple vector
differences, we measure the differences between covariances $\mathbf{U}_{i}$
and $\overline{\mathbf{U}}$ by estimating the transformation $W_{i}$ between
them using Cholesky decomposition, as

(28) $L_{i}L_{i}^{T}=\mathbf{U}_{i},\
\overline{L}\overline{L}^{T}=\overline{\mathbf{U}}$

and

(29) $W_{i}=L_{i}{\overline{L}}^{-1}.$

Then the variations in edited covariances $\mathbf{U}_{i}^{*}$ can be
controlled by $\Delta_{\mathbf{U}}>0$,

(30)
$\mathbf{U}_{i}^{*}=(W_{i})^{\Delta_{\mathbf{U}}}\overline{\mathbf{U}}\left((W_{i})^{\Delta_{\mathbf{U}}}\right)^{T},$

where $(W_{i})^{\Delta_{\mathbf{U}}}$ computes the fractional matrix power of
$W_{i}$. Note when $\Delta_{\mathbf{U}}=1$, the covariances are not edited as
$\mathbf{U}_{i}^{*}=\mathbf{U}_{i}$ and the input image is reconstructed.
Figures 8 and 9 show the edited image when we reduce and increase the
variations in appearances features and covariances (shapes) respectively.

###  4.4. Texture Interpolation

Our representation can be applied for interpolating textures by interpolating
latent Gaussians. Given two sets of Gaussians $\\{\textbf{g}_{i}\\}_{i=1}^{n}$
and $\\{\textbf{g}^{\prime}_{j}\\}_{j=1}^{n}$, we first build correspondences
$\phi$ between them ($\textbf{g}_{i}$ matches
$\textbf{g}^{\prime}_{\phi(i)}$), by minimizing the sum of differences between
their centers and weights using the Hungarian algorithm [Kuhn, 1955]. The
parameters in interpolated Gaussian $\textbf{g}_{i}^{\dagger}$ are the convex
blending of those in $\textbf{g}_{i}$ and $\textbf{g}^{\prime}_{\phi(i)}$
controlled by a coefficient $\eta\in[0,1]$. For example, interpolated Gaussian
center $\boldsymbol{\mu}_{i}^{\dagger}$ is defined as

(31)
$\boldsymbol{\mu}_{i}^{\dagger}=(1-\eta)\boldsymbol{\mu}_{i}+\eta\boldsymbol{\mu}_{\phi(i)}^{\prime}.$

Since $\delta_{i}^{*}$ is supposed to be binary, being either 0 or 1 during
inference, we treat the interpolated value as a probablity of being 1 in a
Bernoulli distribution and $\delta_{i}^{*}$ is sampled from the distribution.
The interpolated image is reconstructed from the interpolated Gaussians
$\\{\textbf{g}_{i}^{\dagger}\\}$. Figure 10 shows interpolated images with
various $\eta$.

Our method can also generate smooth transition of two textures in one image by
spatially interpolating Gaussians, as shown in Figure 10f, where $\eta$
linearly increases from 0 (left) to 1 (right).

###  4.5. Texton-centric Editing

Since we extract individual neural textons, our approach can also enable more
direct manipulation of individual textons.

####  4.5.1. Editing Gaussians

Users can move, scale and rotate textons by transforming their geometric
parameters ($\mathbf{U}_{i}$ and $\mathbf{\nu}_{i}$), as shown in Figure 11.

####  4.5.2. Edit propagation

When users make edits to a specific area of a texture, these changes can be
automatically propagated to other similar areas. This process is demonstrated
in Figure 12. The procedure begins by calculating the difference image between
the edited and the original images. The difference image encodes the edits in
the pixel space. Next, the edit propagation involves identifying the specific
neural Gaussian texton that corresponds to the edited region which is
implemented by querying the nearest Gaussian to the edits. To propagate these
edits to another similar region or texton, we 1) apply a transformation to the
difference image; this transformation aligns the Gaussian of the edited and
the target textons; and 2) add the transformed difference images to the
corresponding region of the original image, which creates the final edit-
propagated image. The edits can be consistently and accurately replicated
across similar textural features, the locations and shapes of which are
identified by extracted neural textons, within the image.

##### Discussions

We observe that propagating edits directly in the pixel space results in
clearer and more precise edits compared to using the latent feature space.
This distinction arises from two key factors. First, edits like text or other
novel elements, which the network might not have encountered during training,
can lead to poor reconstruction quality when processed in the latent feature
space. This, in turn, affects the quality of edit propagation. Second, even if
edits in the image space can be theoretically represented in the appearance
feature space, another issue is that the appearance feature space is not
explicitly trained to handle robust feature arithmetic, that is, the image-
space differences are not guaranteed to be describable and transferable as
differences in the appearance feature space. This lack of specialized training
can lead to less accurate or predictable edit propagation in feature space
when compared to direct pixel space manipulation.

###  4.6. Texture Animation

In Section 4.5.1, we describe how our framework can handle image editing at
the texton level, with each latent Gaussian aligned with a corresponding
texton in the image. This technique also enables animating Gaussians using
flow fields [Kwatra et al., 2005; Chen et al., 2011], which are then
reconstructed as animated textures. Figure 13 shows various frames from such
dynamic visualizations. Please refer to our supplementary video for dynamic
experiences and refer to Section A.7.3 for further details about the flow
fields used in these animations.

##  5\. Analysis

We compare our method to related approaches and evaluate our approach via
ablation studies, both qualitatively and quantitatively.

###  5.1. Comparison to “Segment Anything”

The fundamental aspect of our methodology lies in its ability to autonomously
extract compositional textons from textures using unsupervised techniques. In
our encoder $\mathcal{E}$, the unsupervised segmentation network
$\mathcal{E}_{s}$ is used to segment textons and convert them into Gaussians
in the latent space for subsequent texture editing applications. One obvious
alternative approach to extracting textons is by training a supervised
segmentation model tailored for textons. However, the challenge with
supervised segmentation, which relies on ground-truth labels, is its
incompatibility with the ambiguous nature of textons. Since the concept of a
texton is not universally agreed upon, achieving consistency in ground-truth
texton labels is difficult. Furthermore, labeling textures can be a labor-
intensive process, even more so than natural photos.

To evaluate the quality of our extracted textons, we compare our segmentation
against the “Segment Anything” (SA) model [Meta, 2023]. This model, noted for
its remarkable ability to segment with zero prior knowledge, has been trained
on an extensive dataset of 1 billion masks. Figure 14 shows the comparisons
with the SA model. See Section A.5 for more comparisons. These results
indicate that SA is incapable of reliably detecting and extracting textons, a
fundamental aspect that contributes to the versatility of our method across a
variety of applications.

###  5.2. Comparisons on Texture Transfer

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s) Structure

(t) Appearance

(u) [Kolkin et al., 2019]

(v) [Tumanyan et al., 2022]

(w) [Cheng et al., 2023]

(x) ours

Figure 15. Comparisons with existing approaches on texture transfer. Our
approach can preserve the better structure from the structure-providing
texture than [Tumanyan et al., 2022] and more accurate texton appearance than
[Kolkin et al., 2019]. Cheng et al. [2023] could not transfer the texton
appearance to the structure image due to its use of text-to-image models while
it might not be suitable to describe detailed textures using abstract human
language.

In this section, we aim to demonstrate our approach’s unique capacity in
disentangled structure-appearance texture modeling using latent Gaussians
based on segmented structures. To this end, we compare our approach with prior
methods [Kolkin et al., 2019; Tumanyan et al., 2022; Cheng et al., 2023]
tailored for combining appearance from one image and content/structure from
another image. Figure 15 shows comparisons against prior methods. Our method
generates more visually appealing results without clear artifacts, effectively
capturing both the structure and appearance from the respective structure-
providing and appearance-providing textures. Please see Section A.5 for more
comparisons. Note we not only disentangle structure from texture appearance
but also explicitly represent them as different Gaussian parameters, which
enables a variety of applications (Section 4) beyond texture transfer.

###  5.3. Ablation Studies

We evaluate our method via ablation studies to demonstrate the importance of
each individual component in our approach, both qualitatively and
quantitatively.

####  5.3.1. Quantitative metrics

For quantitative evaluation, first, we evaluate how robust and meaningful the
compositional representation is for different transforms/edits. Since there is
no ground-truth benchmarks for texton extractions, we consider to evaluate our
method based on a cycle consistency (CC) metric. That is, the estimated
compositional representation $\mathcal{E}(I^{*})$ of the edited images
$I^{*}=\mathcal{G}(\\{\textbf{g}^{*}\\})$ reconstructed from edited Gaussians
$\\{\textbf{g}^{*}\\}$ is supposed to be consistent to the edited Gaussians
$\\{\textbf{g}^{*}\\}$. Formally, CC metric is defined as

(32) $\text{CC}=L_{Csis}(\\{\textbf{g}^{*}\\},\mathcal{E}(I^{*}))$

where $L_{Csis}$, the consistency loss to measure differences between two sets
of Gaussians during training, is reused. Second, we evaluate the quality of
our edited images $I^{*}$ using Fréchet inception distance (FID) by comparing
it to unedited input images $I$. We also compute CC and FID, between input
images and their reconstructions for calibrations, in which case, there is no
edits applied to $\\{\textbf{g}^{*}\\}$ and
$\\{\textbf{g}^{*}\\}=\\{\textbf{g}\\}$ when computing CC. In addition, we
calculate a diversity (DVC) metric for measuring the diversity of textures
generated from randomly reshuffled appearance features (Section 4.1). Please
see Section A.2 for more evaluation details.

| Ablation studies | Full
---|---|---
| Absent losses | Absent network components
| $L_{C}$ | $L_{E}$ | $L_{Csis}$ | $L_{T}$ | $L_{GAN}$ | $L_{PGAN}$ | segmentation | latent Gaussians
CC (R) | 0.15 | 1.29 | 1.40 | 0.99 | 0.54 | 1.49 | 0.62 | 0.05 | 0.60
CC (TD) | 0.15 | 1.30 | 7.82 | 1.20 | 0.65 | 1.63 | 0.60 | 0.11 | 0.99
CC (TT/S) | 0.16 | 1.73 | 3.28 | 1.20 | 0.79 | 1.70 | 0.63 | 0.13 | 0.99
CC (TT/A) | 0.15 | 1.39 | 3.29 | 1.27 | 0.68 | 1.95 | 0.61 | 0.11 | 1.04
CC (TE) | 2.71 | 1.50 | 3.59 | 1.17 | 0.73 | 1.78 | 0.75 | - | 1.13
CC (TI) | 0.08 | 4.92 | 10.34 | 6.11 | 5.01 | 6.02 | 0.99 | - | 6.13
FID (R) $\downarrow$ | 67.26 | 129.29 | 17.40 | 53.32 | 69.30 | 112.93 | 219.59 | 11.93 | 16.95
FID (TD) $\downarrow$ | 67.26 | 129.97 | 59.13 | 63.59 | 87.81 | 116.90 | 239.18 | 16.87 | 44.73
FID (TT/S) $\downarrow$ | 67.11 | 139.14 | 53.33 | 70.01 | 93.17 | 122.38 | 221.78 | 27.44 | 41.72
FID (TT/A) $\downarrow$ | 67.26 | 133.75 | 53.53 | 100.80 | 79.43 | 130.32 | 220.76 | 16.64 | 53.58
FID (TE) $\downarrow$ | 178.31 | 139.23 | 50.81 | 57.13 | 77.97 | 116.61 | 221.94 | - | 47.18
FID (TI) $\downarrow$ | 97.35 | 142.84 | 134.32 | 96.35 | 113.20 | 153.64 | 246.39 | - | 70.92
DVS (TD) $\downarrow$ | 0.0 | 0.69 | 4.56 | 2.30 | 1.47 | 1.00 | 0.87 | 0.12 | 2.63
Table 1. Quantitative evaluations of ablation studies. R, TD, TT, TE, TI stand
for reconstruction, texture diversification (Section 4.1), texture transfer
(Section 4.2), texton (Gaussian) editing (Section 4.5) and texture
interpolation (Section 4.4). TT/S and TT/A indicate the metrics are computed
between the transferred output (TT) and the structure (S) or appearance (A)
image. For TI, metrics are computed between the interpolation using $\eta=0.5$
and one of its source/target images. FID score is better when it is lower. For
CC (cycle consistency), a very low value (e.g. w/o $L_{C}$) may suggest the
extracted neural textons do not vary regardless of the input textures while a
high value could suggest the extracted textons lack consistency in response to
edit/transform. A moderate CC value is preferred, as it indicates a balanced
and reliable extraction of neural textons that are both responsive to
different input texture structures and consistent with respect to edits.
Similarly, a moderate DVS (diversity) value is preferred. For example, w/o
$L_{C}$, DVS equals 0 as a texture is represented by one latent Gaussian; w/o
$L_{Csis}$, DVS is the largest as the network fails to extract correct texton
compositions, and therefore random reshuffling of appearance features could
destroy the texture’s overall appearance. In ablaion studies of network
components, without latent Gaussians, it’s unclear how to manipulate textons
and conduct interpolation on segmentations; even if FID scores might be lower,
the editing results may not properly capture texture structures, as shown in
Figure 17.

####  5.3.2. Ablation studies on losses

We illustrate the significance of each loss term in our model by removing each
one and observing the effects. Figure 16 presents a visual comparison of the
results in terms of segmentation, Gaussians, reconstruction, and the texture
transfer application. See Section A.6 for more visual comparisons for the
ablation studies. Without compact loss $L_{C}$, our model struggles to extract
textons, leading to the entire image being represented by a single segment or
Gaussian. The absence of entropy loss $L_{E}$ results in less defined
segmentations and blurry reconstructions. Without consistency loss $L_{Csis}$,
the segmentation degenerates and is always grid-like caused by excessive
minimization of compact $L_{C}$ and entropy $L_{E}$ losses; while its quality
of texture reconstruction is similar to that of using full losses, applying
its extracted latent Gaussian textons for texture transfer applications
generates low-quality, blurry results without capturing structure and
appearance respectively from two input images. Without texture $L_{T}$ and
patch GAN losses $L_{PGAN}$, both of which are applied to encourage similarity
of textures between images reconstructed from appearance-reshuffled Gaussians
and input images for injecting 3 (Property 3 (Reshuffling Invarance).), the
extracted textons are less accurate. The reconstruction quality is also
degraded as a result of the disrupted equilibrium in unsupervised training.
Without GAN loss $L_{GAN}$, the resulting reconstructions exhibit more blurry
artifacts, which interferes with the texton extraction quality. In addition,
we provide detailed quantitative analyses of the significance of various loss
terms for different texture applications on the left side of Table 1.

|  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |

(a) Input

Seg. Overlay

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

Gaussians

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

Recon.

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

Texture Transfer

(aa)

(ab)

(ac)

(ad)

(ae)

(af)

(ag)

(ah)

(ai) Appearance

(aj)

(ak)

(al)

(am)

(an)

(ao)

(ap)

(aq) Input

Seg. Overlay

(ar)

(as)

(at)

(au)

(av)

(aw)

(ax)

(ay)

Gaussians

(az)

(ba)

(bb)

(bc)

(bd)

(be)

(bf)

(bg)

Recon.

(bh)

(bi)

(bj)

(bk)

(bl)

(bm)

(bn)

(bo)

(bp)

Texture Transfer

(bq)

(br)

(bs)

(bt)

(bu)

(bv)

(bw)

(bx)

(by) Appearance

(bz) w/o $L_{C}$

(ca) w/o $L_{E}$

(cb) w/o $L_{Csis}$

(cc) w/o $L_{T}$

(cd) w/o $L_{GAN}$

(ce) w/o $L_{PGAN}$

(cf) Full

Figure 16. Ablation studies on losses. We ablate each loss from our
implementation and visually compare their results with those using full
losses. We visualize the segmentations $\mathbf{S}$, Gaussians g,
reconstructions $I_{r}$ and texture transfer results given appearance-
providing textures.

####  5.3.3. Ablation studies on neural architectures

The segmentation model and the intermediate compositional Gaussian
representation are the two pivotal components in our model architectures. We
evaluate the importance of both components below. Figure 17 shows our results
of the ablation studies. The right side of Table 1 reports the quantitative
evaluation results.

|  |  Gaussian/Seg. |  |  |
---|---|---|---|---
|  |  |  |
|  Texture transfer |  |  |
|  |  |
|  |  Gaussian/Seg. |  |  |
---|---|---|---|---
|  |  |  |
|  Texture transfer |  |  |
|  |  |

(a) Input

(b)

(c)

(d)

Recon.

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m) Appearance

(n) w/o seg.

(o) w/o Gauss.

(p) Full

(q) Input

(r)

(s)

(t)

Recon.

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad) w/o seg.

(ae) w/o Gauss.

(af) Full

Figure 17. Ablation studies on network components. We visualize the
Gaussians/segmentation masks, reconstruction and texture transfer results
given appearance-providing textures.

##### Disabling segmentation module

We replace the segmentation model with a set of linear layers that directly
regress the Gaussian parameters. In the encoder, we replace the segmentation
model with transformer + MLP to generate Gaussian parameters. With the absense
of segmentations, we replace the losses $L_{C}$ and $L_{E}$ originally
computed on segmentation masks with similar losses but defined on Gaussians
directly. As shown in Figure 17, without the segmentation network, we observe
that direct generation of Gaussians cannot converge and does not produce
meaningful compositional textons. This is in accordance with what is observed
in keypoint detection research where heatmap-based approaches outperform
regression-based approaches, since direct regression involves more non-
linearity and thus is hard to train even in supervised settings [Zheng et al.,
2023; Zhang et al., 2018a].

##### Disabling latent Gaussians

Instead of using the Gaussians, we splat features directly using predicted
segmentation masks. Figure 17 shows that, without the Gaussian bottleneck, the
segmentation network segments as simple grids instead of each individual
texton. Also, it is unclear how to directly manipulate textons and conduct
interpolation using segmentation masks. Therefore, Table 1 does not report the
metrics for texton editing and interpolation applications. These results
indicate the Gaussian bottleneck not only provides an easy and intuitive way
for editing applications, but also serves as an implicit regularization for
clean segmentation. We hypothesize this is because without the bottleneck, the
model tends to find shortcuts to encode texture appearance information
directly via high-dimensional segmentation masks without extracting meaningful
texton geometry.

##  6\. Limitations and Future Work

Depending on the editing operations used, our current method may struggle with
handling patterns with elongated, continuous structures. In these patterns,
the notion of textons might not be clear and each elongated structure might be
represented by multiple textons in our current implementation as we impose 2
(Property 2 (Spatial Compactness).) on these learned neural textons. Figure 18
shows such an example where texton editing can disrupt continuous structures
in textures. In addition, our approach relies on the assumption of random
reshuffling of texton appearances (3 (Property 3 (Reshuffling Invarance).))
that can result in a different version of the original texture without
altering its overall appearance. Consequently, while our method can extract
non-stationary structure of textons within textures, it is suitable for those
that exhibit consistent texton appearances throughout the image domain.

(a) Input

(b) Gaussians

(c) Edited (rotated) Gaussians

(d) Edited

Figure 18. Improper compositional representation for elongated structures. For
elongated, continuous structures, it might be improper to model them as
localized textons. Editing individual textons might discrupt the continuous
structures.

In this paper, we propose a framework for compositional modeling of repetitive
patterns in images. Using our framework, we present a broad range of image
editing applications using simple, baseline approaches, which could be
improved with more dedicated mechanisms in the future. For example, in texture
interpolation applications (Section 4.4), we could interpolate Gaussians via a
softer mechanism using optimal transport instead of via the hard approach by
solving assignment problems. Additionally, the potential uses of our approach
extend beyond the presented examples, such as texture rectification [Hao et
al., 2023], by analyzing and rectifying Gaussian centers, directions and
shapes, and weathering [Bellini et al., 2016], by analyzing and manipulating
the appearance features, font stylization by focusing on font shape, and video
stylization by imposing intuitive coherence directly on latent Gaussians.
Furthermore, our framework could be adapted to model repetitive phenomena in
various other data types, such as materials, vector patterns, 3D geometry, and
audio.

## References

  * [1]
  * Bellini et al. [2016] Rachele Bellini, Yanir Kleiman, and Daniel Cohen-Or. 2016\.  Time-varying weathering in texture space.  _ACM Transactions on Graphics (TOG)_ 35, 4 (2016), 1–11.
  * Bergmann et al. [2017] Urs Bergmann, Nikolay Jetchev, and Roland Vollgraf. 2017\.  Learning texture manifolds with the periodic spatial GAN.  _arXiv preprint arXiv:1705.06566_ (2017).
  * Brooks and Dodgson [2002] Stephen Brooks and Neil Dodgson. 2002.  Self-similarity based texture editing.  _ACM Transactions on Graphics (TOG)_ 21, 3 (2002), 653–656.
  * Cao et al. [2022] Jiezhang Cao, Jingyun Liang, Kai Zhang, Yawei Li, Yulun Zhang, Wenguan Wang, and Luc Van Gool. 2022.  Reference-based image super-resolution with deformable attention transformer. In _European conference on computer vision_. Springer, 325–342.
  * Carion et al. [2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020.  End-to-end object detection with transformers. In _European Conference on Computer Vision_. 213–229.
  * Chen et al. [2011] Guoning Chen, Vivek Kwatra, Li-Yi Wei, Charles D Hansen, and Eugene Zhang. 2011.  Design of 2d time-varying vector fields.  _IEEE Transactions on Visualization and Computer Graphics_ 18, 10 (2011), 1717–1730.
  * Cheng et al. [2023] Bin Cheng, Zuhao Liu, Yunbo Peng, and Yue Lin. 2023\.  General image-to-image translation with one-shot image guidance. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 22736–22746.
  * Cheng et al. [2022] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. 2022.  Masked-attention mask transformer for universal image segmentation. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 1290–1299.
  * Dekel et al. [2015] Tali Dekel, Tomer Michaeli, Michal Irani, and William T. Freeman. 2015.  Revealing and Modifying Non-Local Variations in a Single Image.  _ACM Transactions on Graphics (Proc. SIGGRAPH Asia)_ (2015).
  * Efros and Freeman [2001] Alexei A. Efros and William T. Freeman. 2001.  Image Quilting for Texture Synthesis and Transfer. In _Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques_ _(SIGGRAPH ’01)_. Association for Computing Machinery, New York, NY, USA, 341–346.  https://doi.org/10.1145/383259.383296
  * Elsayed et al. [2022] Gamaleldin F Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C Mozer, and Thomas Kipf. 2022\.  Savi++: Towards end-to-end object-centric learning from real-world videos.  _arXiv preprint arXiv:2206.07764_ (2022).
  * Epstein et al. [2022] Dave Epstein, Taesung Park, Richard Zhang, Eli Shechtman, and Alexei A Efros. 2022.  BlobGAN: Spatially disentangled scene representations.  _ECCV_ (2022).
  * Freeman et al. [2002] William T Freeman, Thouis R Jones, and Egon C Pasztor. 2002\.  Example-based super-resolution.  _IEEE Computer graphics and Applications_ 22, 2 (2002), 56–65.
  * Frühstück et al. [2019] Anna Frühstück, Ibraheem Alhashim, and Peter Wonka. 2019\.  Tilegan: synthesis of large-scale non-homogeneous textures.  _ACM Transactions on Graphics (ToG)_ 38, 4 (2019), 1–11.
  * Gatys et al. [2015] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2015.  Texture synthesis using convolutional neural networks. In _International Conference on Neural Information Processing Systems_. 262–270.
  * Guerrero et al. [2022] Paul Guerrero, Miloš Hašan, Kalyan Sunkavalli, Radomír Měch, Tamy Boubekeur, and Niloy J Mitra. 2022.  MatFormer: A generative model for procedural materials.  _arXiv preprint arXiv:2207.01044_ (2022).
  * Hao et al. [2023] Guoqing Hao, Satoshi Iizuka, Kensho Hara, Edgar Simo-Serra, Hirokatsu Kataoka, and Kazuhiro Fukui. 2023\.  Diffusion-based Holistic Texture Rectification and Synthesis.  _SIGGRAPH Asia Conference Paper_ (2023).
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * He et al. [2022a] Xingzhe He, Bastian Wandt, and Helge Rhodin. 2022a.  Autolink: Self-supervised learning of human skeletons and object outlines by linking keypoints.  _arXiv preprint arXiv:2205.10636_ (2022).
  * He et al. [2022b] Xingzhe He, Bastian Wandt, and Helge Rhodin. 2022b.  Ganseg: Learning to segment by unsupervised hierarchical image generation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 1225–1235.
  * Henzler et al. [2020] Philipp Henzler, Niloy J Mitra, and Tobias Ritschel. 2020\.  Learning a neural 3d texture space from 2d exemplars. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 8356–8364.
  * Hertz et al. [2022] Amir Hertz, Or Perel, Raja Giryes, Olga Sorkine-Hornung, and Daniel Cohen-Or. 2022.  Spaghetti: Editing implicit shapes through part aware generation.  _ACM Transactions on Graphics (TOG)_ 41, 4 (2022), 1–20.
  * Hertzmann et al. [2001] Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David H. Salesin. 2001\.  Image Analogies. In _SIGGRAPH ’01_. Association for Computing Machinery, New York, NY, USA, 327–340.  https://doi.org/10.1145/383259.383295
  * Hu et al. [2023] Yiwei Hu, Paul Guerrero, Milos Hasan, Holly Rushmeier, and Valentin Deschaintre. 2023.  Generating Procedural Materials from Text or Image Prompts. In _ACM SIGGRAPH 2023 Conference Proceedings_. 1–11.
  * Isola et al. [2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017.  Image-to-image translation with conditional adversarial networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 1125–1134.
  * Jang et al. [2016] Eric Jang, Shixiang Gu, and Ben Poole. 2016.  Categorical Reparameterization with Gumbel-Softmax. In _International Conference on Learning Representations_.
  * Jiang et al. [2023] Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. 2023\.  Object-Centric Slot Diffusion.  _NeurIPS_ (2023).
  * Karras et al. [2020] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2020\.  Analyzing and improving the image quality of stylegan. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 8110–8119.
  * Kaspar et al. [2015] Alexandre Kaspar, Boris Neubert, Dani Lischinski, Mark Pauly, and Johannes Kopf. 2015.  Self Tuning Texture Optimization.  _Comput. Graph. Forum_ 34, 2 (May 2015), 349–359.  https://doi.org/10.1111/cgf.12565
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Kolkin et al. [2019] Nicholas Kolkin, Jason Salavon, and Gregory Shakhnarovich. 2019\.  Style transfer by relaxed optimal transport and self-similarity. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10051–10060.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012\.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ 25 (2012).
  * Kuhn [1955] Harold W Kuhn. 1955\.  The Hungarian method for the assignment problem.  _Naval research logistics quarterly_ 2, 1-2 (1955), 83–97.
  * Kwatra et al. [2005] Vivek Kwatra, Irfan Essa, Aaron Bobick, and Nipun Kwatra. 2005\.  Texture Optimization for Example-Based Synthesis.  _ACM Trans. Graph._ 24, 3 (jul 2005), 795–802.  https://doi.org/10.1145/1073204.1073263
  * Kwatra et al. [2003] Vivek Kwatra, Arno Schödl, Irfan Essa, Greg Turk, and Aaron Bobick. 2003.  Graphcut textures: Image and video synthesis using graph cuts.  _Acm transactions on graphics (tog)_ 22, 3 (2003), 277–286.
  * Lefebvre and Hoppe [2005] Sylvain Lefebvre and Hugues Hoppe. 2005.  Parallel controllable texture synthesis.  In _ACM SIGGRAPH 2005 Papers_. 777–786.
  * Lefebvre and Hoppe [2006] Sylvain Lefebvre and Hugues Hoppe. 2006.  Appearance-space Texture Synthesis. In _ACM SIGGRAPH 2006 Papers_ (Boston, Massachusetts) _(SIGGRAPH ’06)_. ACM, New York, NY, USA, 541–548.  https://doi.org/10.1145/1179352.1141921
  * Li et al. [2023] Beichen Li, Liang Shi, and Wojciech Matusik. 2023.  End-to-End Procedural Material Capture with Proxy-Free Mixed-Integer Optimization.  _ACM Transactions on Graphics (TOG)_ 42, 4 (2023), 1–15.
  * Li et al. [2022] Xueting Li, Xiaolong Wang, Ming-Hsuan Yang, Alexei A Efros, and Sifei Liu. 2022.  Scraping Textures from Natural Images for Synthesis and Editing. In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XV_. Springer, 391–408.
  * Liu et al. [2004] Yanxi Liu, Wen-Chieh Lin, and James Hays. 2004.  Near-regular texture analysis and manipulation.  _ACM Transactions on Graphics (TOG)_ 23, 3 (2004), 368–376.
  * Liu et al. [2009] Yiming Liu, Jiaping Wang, Su Xue, Xin Tong, Sing Bing Kang, and Baining Guo. 2009\.  Texture splicing. In _Computer Graphics Forum_ , Vol. 28. Wiley Online Library, 1907–1915.
  * Locatello et al. [2020] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. 2020.  Object-centric learning with slot attention.  _Advances in Neural Information Processing Systems_ 33 (2020), 11525–11538.
  * Lu et al. [2009] Jianye Lu, Julie Dorsey, and Holly Rushmeier. 2009.  Dominant texture and diffusion distance manifolds. In _Computer Graphics Forum_ , Vol. 28. Wiley Online Library, 667–676.
  * Matusik et al. [2005] Wojciech Matusik, Matthias Zwicker, and Frédo Durand. 2005\.  Texture Design Using a Simplicial Complex of Morphable Textures.  _ACM Trans. Graph._ 24, 3 (July 2005), 787–794.  https://doi.org/10.1145/1073204.1073262
  * Meta [2023] Meta. 2023.  Segment Anything.  https://segment-anything.com/
  * Minaee et al. [2021] Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. 2021.  Image segmentation using deep learning: A survey.  _IEEE transactions on pattern analysis and machine intelligence_ 44, 7 (2021), 3523–3542.
  * Nash and Williams [2017] Charlie Nash and Christopher KI Williams. 2017.  The shape variational autoencoder: A deep generative model of part-segmented 3D objects. In _Computer Graphics Forum_ , Vol. 36. Wiley Online Library, 1–12.
  * Niemeyer and Geiger [2021] Michael Niemeyer and Andreas Geiger. 2021.  Giraffe: Representing scenes as compositional generative neural feature fields. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 11453–11464.
  * Ost et al. [2021] Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. 2021.  Neural Scene Graphs for Dynamic Scenes. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 2856–2865.
  * Park et al. [2019] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. 2019\.  Semantic image synthesis with spatially-adaptive normalization. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 2337–2346.
  * Park et al. [2020] Taesung Park, Jun-Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei Efros, and Richard Zhang. 2020.  Swapping autoencoder for deep image manipulation.  _Advances in Neural Information Processing Systems_ 33 (2020), 7198–7211.
  * Rosenberger et al. [2009] Amir Rosenberger, Daniel Cohen-Or, and Dani Lischinski. 2009\.  Layered Shape Synthesis: Automatic Generation of Control Maps for Non-Stationary Textures.  _ACM Trans. Graph._ 28, 5 (Dec. 2009), 1–9.  https://doi.org/10.1145/1618452.1618453
  * Sajjadi et al. [2022] Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetic, Mario Lucic, Leonidas J Guibas, Klaus Greff, and Thomas Kipf. 2022.  Object scene representation transformer.  _Advances in Neural Information Processing Systems_ 35 (2022), 9512–9524.
  * Shi et al. [2020] Liang Shi, Beichen Li, Miloš Hašan, Kalyan Sunkavalli, Tamy Boubekeur, Radomir Mech, and Wojciech Matusik. 2020\.  Match: Differentiable material graphs for procedural material capture.  _ACM Transactions on Graphics (TOG)_ 39, 6 (2020), 1–15.
  * Tumanyan et al. [2022] Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel. 2022\.  Splicing vit features for semantic appearance transfer. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10748–10757.
  * Wei et al. [2009] Li-Yi Wei, Sylvain Lefebvre, Vivek Kwatra, and Greg Turk. 2009.  State of the art in example-based texture synthesis. In _Eurographics 2009, State of the Art Report, EG-STAR_. Eurographics Association, 93–117.
  * Xue et al. [2018] Jia Xue, Hang Zhang, and Kristin Dana. 2018.  Deep texture manifold for ground terrain recognition. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 558–567.
  * Yang et al. [2020a] Fengting Yang, Qian Sun, Hailin Jin, and Zihan Zhou. 2020a.  Superpixel segmentation with fully convolutional networks. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 13964–13973.
  * Yang et al. [2020b] Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo. 2020b.  Learning texture transformer network for image super-resolution. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 5791–5800.
  * Yu et al. [2019] Ning Yu, Connelly Barnes, Eli Shechtman, Sohrab Amirghodsi, and Michal Lukáč. 2019.  Texture Mixer: A Network for Controllable Synthesis and Interpolation of Texture. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zhai et al. [2020] Wei Zhai, Yang Cao, Zheng-Jun Zha, HaiYong Xie, and Feng Wu. 2020.  Deep structure-revealed network for texture recognition. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 11010–11019.
  * Zhang et al. [2017] Hang Zhang, Jia Xue, and Kristin Dana. 2017.  Deep ten: Texture encoding network. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 708–717.
  * Zhang et al. [2018b] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018b.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In _CVPR_.
  * Zhang et al. [2018a] Yuting Zhang, Yijie Guo, Yixin Jin, Yijun Luo, Zhiyuan He, and Honglak Lee. 2018a.  Unsupervised discovery of object landmarks as structural representations. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 2694–2703.
  * Zhang et al. [2019] Zhifei Zhang, Zhaowen Wang, Zhe Lin, and Hairong Qi. 2019\.  Image super-resolution by neural texture transfer. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 7982–7991.
  * Zheng et al. [2023] Ce Zheng, Wenhan Wu, Chen Chen, Taojiannan Yang, Sijie Zhu, Ju Shen, Nasser Kehtarnavaz, and Mubarak Shah. 2023\.  Deep learning-based human pose estimation: A survey.  _Comput. Surveys_ 56, 1 (2023), 1–37.
  * Zheng et al. [2018] Haitian Zheng, Mengqi Ji, Haoqian Wang, Yebin Liu, and Lu Fang. 2018.  Crossnet: An end-to-end reference-based super resolution network using cross-scale warping. In _Proceedings of the European conference on computer vision (ECCV)_. 88–104.
  * Zhou et al. [2023] Yang Zhou, Kaijian Chen, Rongjun Xiao, and Hui Huang. 2023\.  Neural Texture Synthesis With Guided Correspondence. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 18095–18104.
  * Zhou et al. [2018] Yang Zhou, Zhen Zhu, Xiang Bai, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. 2018\.  Non-stationary texture synthesis by adversarial expansion.  _ACM Transactions on Graphics (TOG)_ 37, 4 (2018), 1–13.

##  Appendix A Appendix

###  A.1. Dataset

We have partitioned the dataset into training and testing subsets through a
random split, aiming to maintain a consistent data distribution across both
training and testing scenarios. Figure 19 shows 100 images, randomly sampling
from the training subset, without processing with data augmentation (i.e.
various shape and color transforms of the originals), to provide a glimpse
into the dataset’s diversity. This dataset spans an extensive range of
textures, varying from stochastic to regular patterns.

Figure 19. 100 original dataset samples without augmentation.

###  A.2. Evaluation details

The metrics in Table 1 are calculated across 5400 images or image pairs. For
texture diversification (TD), the Fréchet Inception Distance (FID) is computed
between pairs of input image and one of its diversified version. To calculate
the diversity (DVS) metric, we encode input textures as latent Gaussians. Then
we create 4 unique variations by reshuffling appearance features. These
variations are then processed through a VGG-19 network. The DVS is calculated
as the average differences in the network’s responses from its 2nd, 4th, 8th,
and 12th layers. In texture transfer (TT), we use 5400 pairs of images. One
image in each pair provides the appearance, and the other provides the
structure. The metrics are derived by comparing the transferred results with
the appearance- and structure-providing inputs. For texton editing, we encode
images into latent Gaussians and introduce random geometric edits. These edits
include random movements (with normal-distributed displacements having a mean
of 0 and a standard deviation of 0.1 $\times$ the average distance to the
nearest other Gaussians), scaling (by a factor ranging from 0.8 to 1.25), and
rotating (within a range of -30 to 30 degrees). We use these edits and their
reconstructions to compute the CC and FID metrics.

###  A.3. Texton segmentation

Our methodology centers on the ability to segment individual textons from a
wide variety of textures, which facilitates a number of applications. Figures
20 and 21 show 120 uncurated segmentation results, demonstrating the
effectiveness of our approach across diverse textures.

Figure 20. Uncurated texton segmentation results.

Figure 21. Continuation from Figure 20.

###  A.4. Texture transfer

By default, our texture transfer results are generated by “mean alignment”
mechanism by “adjusting the mean of Gaussian appearance features of the
structure-providing texture to match those of the appearance-providing
texture” (Section 4.2). However, our framework also supports other mechanisms
for texture transfer. For example, one can replace each appearance feature of
the structure-providing texture with a randomly sampled feature from the
appearance-providing texture. Figure 22 presents a comparison of the results
generated using both these mechanisms. As expected, the replacement method
tends to preserve less appearance information from the structure-providing
texture compared to the mean alignment method. However, both methods are
effective. In addition, Figure 23 shows more results about scale control in
transferring texture to images.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z) Structure

(aa) Appearance

(ab) Replacement

(ac) Mean alignment (default)

(ad) Seg. overlay

Figure 22. Two texture transfer mechanisms. Segmentations of the structure-
providing textures are displayed on the right.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

Figure 23. Scale control in texture transfer. The first row, from left to
right: content image, large, median (2/3 of large scale) and small (2/3 of
median scale) latent Gaussians. The remaining rows, from left to right:
texture, transferred results with large, median and small textons.

###  A.5. Comparisons

In Figure 24, we present a broader range of uncurated comparisons utilizing
the segment anything method. Further, we show more comparisons on texture
transfer with existing approaches. This involves randomly sampling a set of
images and merging them in such a way that one provides the structure while
the other provides the appearance. Some notable examples of these pairings are
shown in Figures 25 and 26. Additionally, we include uncurated combinations of
these images in Figures 27, 28, 29, 30, 31 and 32.

Figure 24. Uncurated comparisons with SA. Each set contains 3 images: the
input (left), result from SA [Meta, 2023] trained with 1 billion masks
(middle), and our model’s output (right). Notably, despite SA’s reliance on
extensive supervision, our model successfully extracts textons without
supervision.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

(ae) Structure

(af) Appearance

(ag) [Kolkin et al., 2019]

(ah) [Tumanyan et al., 2022]

(ai) [Cheng et al., 2023]

(aj) ours

Figure 25. Comparison with state-of-the-art approaches on texture transfer.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

(ae) Structure

(af) Appearance

(ag) [Kolkin et al., 2019]

(ah) [Tumanyan et al., 2022]

(ai) [Cheng et al., 2023]

(aj) ours

Figure 26. Continuation from Figure 25.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

(ae) Structure

(af) Appearance

(ag) [Kolkin et al., 2019]

(ah) [Tumanyan et al., 2022]

(ai) [Cheng et al., 2023]

(aj) ours

Figure 27. Uncurated comparisons with state-of-the-art approaches on texture
transfer.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

(ae) Structure

(af) Appearance

(ag) [Kolkin et al., 2019]

(ah) [Tumanyan et al., 2022]

(ai) [Cheng et al., 2023]

(aj) ours

Figure 28. Continuation from Figure 27.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

(ae) Structure

(af) Appearance

(ag) [Kolkin et al., 2019]

(ah) [Tumanyan et al., 2022]

(ai) [Cheng et al., 2023]

(aj) ours

Figure 29. Continuation from Figure 27.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

(ae) Structure

(af) Appearance

(ag) [Kolkin et al., 2019]

(ah) [Tumanyan et al., 2022]

(ai) [Cheng et al., 2023]

(aj) ours

Figure 30. Continuation from Figure 27.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

(ae) Structure

(af) Appearance

(ag) [Kolkin et al., 2019]

(ah) [Tumanyan et al., 2022]

(ai) [Cheng et al., 2023]

(aj) ours

Figure 31. Continuation from Figure 27.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac)

(ad)

(ae) Structure

(af) Appearance

(ag) [Kolkin et al., 2019]

(ah) [Tumanyan et al., 2022]

(ai) [Cheng et al., 2023]

(aj) ours

Figure 32. Continuation from Figure 27.

###  A.6. Ablation studies

Figures 33 and 34 shows more results for ablation studies of losses. Figure 35
shows more results for ablation studies of neural architectures.

|  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |

(a) Input

Seg. Overlay

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

Gaussians

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

Recon.

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

Texture Transfer

(aa)

(ab)

(ac)

(ad)

(ae)

(af)

(ag)

(ah)

(ai) Appearance

(aj)

(ak)

(al)

(am)

(an)

(ao)

(ap)

(aq) Input

Seg. Overlay

(ar)

(as)

(at)

(au)

(av)

(aw)

(ax)

(ay)

Gaussians

(az)

(ba)

(bb)

(bc)

(bd)

(be)

(bf)

(bg)

Recon.

(bh)

(bi)

(bj)

(bk)

(bl)

(bm)

(bn)

(bo)

(bp)

Texture Transfer

(bq)

(br)

(bs)

(bt)

(bu)

(bv)

(bw)

(bx)

(by) Appearance

(bz) w/o $L_{C}$

(ca) w/o $L_{E}$

(cb) w/o $L_{Csis}$

(cc) w/o $L_{T}$

(cd) w/o $L_{GAN}$

(ce) w/o $L_{PGAN}$

(cf) Full

Figure 33. Ablation studies on losses. We ablate each loss from our
implementation and visually compare their results with those using full
losses. We visualize the segmentations $\mathbf{S}$, Gaussians g,
reconstructions $I_{r}$ and texture transfer results given appearance-
providing textures.

|  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |

(a) Input

Seg. Overlay

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

Gaussians

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

Recon.

(r)

(s)

(t)

(u)

(v)

(w)

(x)

(y)

(z)

Texture Transfer

(aa)

(ab)

(ac)

(ad)

(ae)

(af)

(ag)

(ah)

(ai) Appearance

(aj)

(ak)

(al)

(am)

(an)

(ao)

(ap)

(aq) Input

Seg. Overlay

(ar)

(as)

(at)

(au)

(av)

(aw)

(ax)

(ay)

Gaussians

(az)

(ba)

(bb)

(bc)

(bd)

(be)

(bf)

(bg)

Recon.

(bh)

(bi)

(bj)

(bk)

(bl)

(bm)

(bn)

(bo)

(bp)

Texture Transfer

(bq)

(br)

(bs)

(bt)

(bu)

(bv)

(bw)

(bx)

(by) Appearance

(bz) w/o $L_{C}$

(ca) w/o $L_{E}$

(cb) w/o $L_{Csis}$

(cc) w/o $L_{T}$

(cd) w/o $L_{GAN}$

(ce) w/o $L_{PGAN}$

(cf) Full

Figure 34. Continuation of Figure 33.

|  |  Gaussian/Seg. |  |  |
---|---|---|---|---
|  |  |  |
|  Texture transfer |  |  |
|  |  |
|  Gaussian/Seg. |  |  |
|  |  |  |
|  Texture transfer |  |  |
|  |  |
|  Gaussian/Seg. |  |  |
---|---|---|---|---
|  |  |  |
|  Texture transfer |  |  |
|  |  |
|  Gaussian/Seg. |  |  |
|  |  |  |
|  Texture transfer |  |  |
|  |  |

(a) Input

(b)

(c)

(d)

Recon.

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m) Appearance

(n) w/o seg.

(o) w/o Gauss.

(p) Full

(q) Input

(r)

(s)

(t)

Recon.

(u)

(v)

(w)

(x)

(y)

(z)

(aa)

(ab)

(ac) Appearance

(ad) w/o seg.

(ae) w/o Gauss.

(af) Full

(ag) Input

(ah)

(ai)

(aj)

Recon.

(ak)

(al)

(am)

(an)

(ao)

(ap)

(aq)

(ar) w/o seg.

(as) w/o Gauss.

(at) Full

(au) Input

(av)

(aw)

(ax)

Recon.

(ay)

(az)

(ba)

(bb)

(bc)

(bd)

(be)

(bf) w/o seg.

(bg) w/o Gauss.

(bh) Full

Figure 35. Ablation studies on network components. We visualize the
Gaussians/segmentation masks, reconstruction and texture transfer results
given appearance-providing textures.

###  A.7. Implementation details

####  A.7.1. Network architectures

##### Encoder $\mathcal{E}$

We utilize Mask2Former model [Cheng et al., 2022] as the segmentation network
$\mathcal{E}_{s}$. This network employs ResNet-50 [He et al., 2016] as the
backbone and set up with 100 object quries and other default parameters as
specified in [Cheng et al., 2022] except that the number of transformer
decoder layers is reduced to 4 and the number of classes is set as 1 given
that texton is the sole semantic class under consideration. The fully
convolutional encoder $\mathcal{E}_{FC}$ processes an input image to generate
an appearance feature map $\mathbf{F}_{a}$ and a direction map $\mathbf{V}$,
as illustrated in Table 2. To generate $\mathbf{F}_{a}$, $\mathcal{E}_{FC}$
comprises four downsampling residual blocks followed by two upsampling
convolutional blocks. To generate the direction map, the network diverges
after the residual blocks, incorporating an additional set of two upsampling
convolutional blocks. Reflection padding is employed in $\mathcal{E}_{FC}$ to
prevent the leakage of absolute positional information. For an input image of
size $128\times 128$, the segmentation masks generated by Mask2Former are of
dimensions $32\times 32\times n$, where $n=100$ represents the maximum number
of Gaussians or the number of queries utilized in Mask2Former. The appearance
feature map $\mathbf{F}_{a}$ has dimensions of $32\times 32\times d_{a}$ with
$d_{a}=n_{f}=382$. The direction map $\mathbf{V}$ is sized $32\times 32\times
2$. The segmentation masks, appearance feature map and the direction map are
upscaled to match the original image size ($128\times 128$ or $256\times 256$
during the fine-tuning phase) before feeding into the Gaussian parameter
estimation layer $\mathcal{E}_{g}$.

FC image encoder $\mathcal{E}_{FC}$ | Upsample ConvBlock $\textsf{UpBlock}(n_{i},n_{o})$
---|---
$I\in\mathbb{R}^{H\times W\times 3}$ | $x\in\mathbb{R}^{H_{x}\times W_{x}\times C_{x}}$
$\textsf{ResBlock}(3,64)\rightarrow\mathbb{R}^{H/2\times W/2\times 64}$ | $\textsf{Upsample}(2)\rightarrow\mathbb{R}^{2H_{x}\times 2W_{x}\times n_{i}}$
$\textsf{ResBlock}(64,128)\rightarrow\mathbb{R}^{H/4\times W/4\times 128}$ | $\textsf{Conv}(n_{i},n_{o})\rightarrow\mathbb{R}^{2H_{x}\times 2W_{x}\times n_{o}}$
$\textsf{ResBlock}(128,256)\rightarrow\mathbb{R}^{H/8\times W/8\times 256}$ | $\textsf{BatchNorm2D}(n_{o})\xrightarrow{\textsf{LeakyReLU}(0.2)}\mathbb{R}^{2H_{x}\times 2W_{x}\times n_{o}}$
$\textsf{ResBlock}(256,512)\rightarrow\mathbb{R}^{H/16\times W/16\times 512}$ |
$\textsf{UpBlock}(512,256)\rightarrow\textsf{UpBlock}(256,382)\rightarrow\mathbf{F}_{a}\in\mathbb{R}^{H/4\times W/4\times 382}$ |
$\textsf{UpBlock}(512,256)\rightarrow\textsf{UpBlock}(256,2)\rightarrow\mathbf{V}\in\mathbb{R}^{H/4\times W/4\times 2}$ |
Table 2. The architecture of $\mathcal{E}_{FC}$.

##### Generator

We adopt the SPADE generator as described in [Park et al., 2019], configured
with four SPADE ResBlocks. Instead of using the learnable constant feature map
originally employed in SPADE, we replace it with a downscaled version of
$\mathbf{F}_{input}=\max_{i}\alpha_{i}$, which has dimensions that are 1/4 the
size of the input image. When the input image resolution is $H\times W$, the
intermediate feature maps produced by the sequence of SPADE ResBlock are sized
as follows: the first block generates $H/4\times W/4\times 384$
($n_{f}+$length of $\mathbf{\nu}$ (2)$=384$), the second block $H/2\times
H/2\times 192$, the third block $H\times W\times 96$, and the final block
outputs $H\times W\times 3$.

##### Discriminators

We have two discriminators $\mathcal{D}$ and $\mathcal{D}_{p}$, each serving a
different purpose. The discriminator $\mathcal{D}$ is identical to StyleGAN2
[Karras et al., 2020] without minibatch discrimination [Park et al., 2020]. We
employs the patch discriminator $\mathcal{D}_{p}$ consisting of 4 layers from
[Isola et al., 2017].

####  A.7.2. Training details

In each iteration, we sample a minibatch of size N, generating N reconstructed
images, N transformed images, and N/2 images reconstructed from reshuffled
features for saving memory. For the GAN loss specifics, we conform to the
StyleGAN2 setting [Karras et al., 2020], including the non-saturating GAN loss
and lazy R1 regularization executed every 16 steps. for both the $\mathcal{D}$
and the $\mathcal{D}_{p}$. The R1 regularization weight is 10.0 for the image
discriminator $\mathcal{D}$ and 0.1 for the patch discriminator
$\mathcal{D}_{p}$.

Initially, the weights are set as follows: $w_{Csis}=0.008$, $w_{E}=0.08$, and
$w_{C}=100$. These weights are dynamically adjusted at specific training
milestones. After 5,000 steps, $w_{Csis}$ is increased to $0.08$.
Subsequently, at the 100000 step, $w_{E}$ is adjusted to $0.1$. Meanwhile, the
value of $w_{C}$ undergoes two changes: it is reduced to $1$ after 1000 steps,
and then further lowered to $0.2$ after 5,000 steps. Regarding the texture
loss, we employ the default parameters specified in [Zhou et al., 2023], with
the exception of the patch sizes. Initially, the patch size is set to 2, which
is then increased to 4 after 40,000 steps and further to 6 after 80,000 steps.
During the fine-tuning phase, all parameters are maintained at their final
adjusted values from the initial training phase. We use ADAM [Kingma and Ba,
2014] with $5e^{-5}$ learning rate, $\beta{1}=0.5$ and $\beta{2}=0.999$.
During the initial training phase on $128\times 128$ images, we use batch size
64 on 8 A100 (40GB) GPUs. For fine-tuning on $256\times 256$ images, we use
batch size 56 on 8 A100 (80GB) GPUs.

####  A.7.3. Texture animations

We create texture animations using two types of flow fields: perturbed shear
flow and vortex flow. Given a point of initial location $(x_{init},y_{init})$
($x_{init},y_{init}\in[-1,1]$), we compute the subsequent position of this
point $(x_{t},y_{t})$ at time $t$ induced by the perturbed shear flow as

(33) $x_{t}=\begin{cases}x_{init}-vt+\epsilon_{t}\qquad&y_{init}<-1/3\\\
x_{init}+vt+\epsilon_{t}\qquad&y_{init}>1/3\\\ 0\qquad&otherswise\end{cases}$

and $y_{t}=y_{init}$, where $v$ is the velocity of point; $\epsilon_{t}$ is
the perturbation at time $t$. To compute $\epsilon_{t}$, we initially
uniformally sample 10 frames; for each of these frames, we sample a value for
$\epsilon$ from a normal distribution that has mean of 0 and a standard
deviation of 1/30. After obtaining these samples, we perform interpolation of
$\epsilon_{t}$ values in the time domain to establish a continuous temporal
profile of the perturbation. If a point moves beyond the image boundary, it
will re-enter the image from the opposite side. To define our used vortex
flow, for simplifying notations, we convert the coordinates into polar
coordinate systems as $r=\sqrt{x^{2}+y^{2}}$ and $\phi=\textsf{atan2}(y/x)$,
then at time $t$,

(34) $\displaystyle\phi_{t}=\phi_{init}+\omega t$

and $r_{t}=r_{init}$, where $\omega$ is the angular velocity of point.

##  Appendix B Options for High-quality Productions

We showcase the flexibility and effectiveness of our compositional texture
representation across various texture manipulation tasks. To integrate this
technology into products, it’s crucial to achieve outputs of high quality and
resolution, retaining all details from the source images. A direct approach
would be to upscale our model to train on large-scale, high-resolution images.
However, we propose that training and implementing a high-resolution, sizable,
dedicated end-to-end network for compositional texture editing isn’t the most
efficient for production. This approach demands excessive demands on a single
network for both control and quality. Instead, we discuss some alternative,
potentially more efficient methodologies that can be applied in the post-
processing stage.

When working with a high-resolution (HR) input image, users first need to
downsample it to match our network’s capacity. This step allows for the
creation of a low-resolution (LR) edited image by modifying encoded latent
Gaussians. The process of converting LR edited images back to high-resolution,
using the original HR input images, is a classic example of a reference-based
super-resolution (SR) problem [Freeman et al., 2002]. Generally, for effective
results, the reference images used should possess a texture and/or content
structure similar to the LR image [Zhang et al., 2019]. Recent advancements in
general reference-based super-resolution (SR) methods, which are potentially
instrumental for deploying our techniques into production, have been detailed
in [Zheng et al., 2018; Zhang et al., 2019; Yang et al., 2020b; Cao et al.,
2022]. Moreover, given that both our inputs and edited outputs are textures,
an alternative strategy could involve performing optimization at test-time.
This would be aimed at maximizing the similarity in texture between the HR
input and the LR edited output. Such an approach could effectively bridge the
gap between differing resolutions, ensuring a closer match in texture quality
and details between the input and output.