(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: CVSSP, University of Surrey, United Kingdom 22institutetext:
iFlyTek-Surrey Joint Research Centre 33institutetext: Surrey Institute for
People-Centred AI
33email<EMAIL_ADDRESS>
# PartCraft: Crafting Creative Objects by Parts

Kam Woh Ng 1122 0000-0002-9309-563X Xiatian Zhu 1133 0000-0002-9284-2955 Yi-
Zhe Song 1122 0000-0001-5908-3275 Tao Xiang 1122

###### Abstract

This paper propels creative control in generative visual AI by allowing users
to “select”. Departing from traditional text or sketch-based methods, we for
the first time allow users to choose visual concepts by parts for their
creative endeavors. The outcome is fine-grained generation that precisely
captures selected visual concepts, ensuring a holistically faithful and
plausible result. To achieve this, we first parse objects into parts through
unsupervised feature clustering. Then, we encode parts into text tokens and
introduce an entropy-based normalized attention loss that operates on them.
This loss design enables our model to learn generic prior topology knowledge
about object’s part composition, and further generalize to novel part
compositions to ensure the generation looks holistically faithful. Lastly, we
employ a bottleneck encoder to project the part tokens. This not only enhances
fidelity but also accelerates learning, by leveraging shared knowledge and
facilitating information exchange among instances. Visual results in the paper
and supplementary material showcase the compelling power of PartCraft in
crafting highly customized, innovative creations, exemplified by the
“charming” and creative birds in Fig. 1. Code is released at
https://github.com/kamwoh/partcraft.

###### Keywords:

Part Composition Controllable Text-to-image Generation

##  1 Introduction

Figure 1:  People often form creative concepts based on existing ones [70, 46,
59, 10]. For instance, a bird enthusiast may want to craft a unique bird with
different parts (_e.g_., heads, bodies, and wings) from common bird types
(_e.g_., blue jay, cardinal and song sparrow). (a) Using text prompts in T2I
models often results in a lack of control and deviation from the intended
details, especially those visual details that are difficult to describe. (b)
While sketching is a direct way, not everyone possesses the ability to sketch,
particularly in intricate detail. (c) Even with reference images, existing
methods (_e.g_., DreamBooth [58]) did not consider learning object parts, thus
cannot generate with desired parts. (d) Using additional control is even
cumbersome, requiring too many inputs! (e) We aim to create an object by
simply selecting desired parts. PartCraft learns from visual examples to
generate the object with a faithful holistic structure, seamlessly integrating
the chosen parts into a natural and coherent entity.

Humans are creators; AI, on the other hand, hallucinates. Creativity,
arguably, is the very force driving humanity forward. Recently, generative AI
has garnered considerable attention for its perceived “creativity” [57, 54, 7,
21, 47, 60, 20, 51, 52, 29]. Despite its promise, the challenge of control has
swiftly surfaced – how can humans infuse their creativity into the generation
process and regulate the extent to which AI hallucinates?

Creativity starts with an idea. The immediate challenge is how to articulate
that idea and integrate it with generative AI. Text is the most commonly
employed medium. For instance, imagine being a bird enthusiast wanting to
craft the most unique bird akin to Fig. 1. The go-to approach would be
furnishing Stable Diffusion [57] (or an equivalent model) with the following
textual prompt (the idea): “generate a bird with the head of X, wings of Y,
and body of Z”. While you might be presented with remarkably looking birds
like Fig. 1(a), they may bear little resemblance to the envisioned concept.
Recent literature suggests that a swift sketch could serve as a viable
alternative [44, 76, 68], providing fine-grained shape control. However, the
caveat is that not everyone possesses the ability to sketch, particularly in
intricate detail.

In this paper, our primary focus revolves around addressing the issue of
“control” in generative AI. We advance by introducing fine-grained control
into the generative process, inviting you to “select”. While this selection
mechanism might seem modest at first, it closely mirrors the human creative
process, where new concepts often emerge from existing ones [70, 46, 59, 10].
Recall those moments when you desired an “ideal” pair of shoes with selected
features from different pairs, or when you aimed to get creative with a cat
(for that matter!)?

It follows that rather than relying on writing (text) or drawing (sketch), all
that is required is to choose the distinct visual concepts you specifically
desire in your creative endeavor. Our model then ensures that all selected
concepts are seamlessly and precisely composed into a faithful novel object in
the final generation, without resorting to additional control such as bounding
boxes [71, 8, 44, 41, 76]. To illustrate with the “unique bird” example once
more, our approach literally involves selecting the head of X, wings of Y, and
body of Z! (see Fig. 1(e) and 2).

Our solution is intuitive and centers around the well-studied computer vision
concept of objects and their parts [9, 24, 38, 15, 30]. The challenge then
boils down to two aspects: (i) how to parse known objects into their parts
(_i.e_., recognizing that birds have eyes and tails), and (ii) how to assemble
parts from different visual concepts to form a faithful creative concept
(_i.e_., ensuring the output is recognizably a bird).

The former is easier. We make clever use of DINOv2 [49] feature maps and
perform unsupervised clustering to identify common parts. The idea is that
each cluster will then correspond to a semantic part of a common object
(_e.g_., head and wings of birds). We specifically chose DINOv2 for its
superior fine-grained perception [3] compared to others [63, 55, 62]. Further,
this way has a higher flexibility to enhance fine-grained parsing by using a
higher cluster count.

Our major contribution lies in addressing the latter challenge. The solution
is intuitive – it essentially revolves around the fine-grained selection and
placement of the chosen parts. Inspired by recent efforts in personalization
[58, 27, 69, 39, 4], primarily designed for learning entire objects, we
introduce a tailored attention loss that specifically operates on parts. With
this loss, our model learns the object part composition, ensuring the final
generation appears holistically faithful with mixed parts (_i.e_., head and
wings of a bird actually appear at the right places).

Figure 2:  Two sets of images were generated from their original parts
(sources A and B). We can integrate a specific part (_e.g_., body, head, or
even background (BG) of source B to target A seamlessly without effort.

More specifically, we introduce an entropy-based attention loss that maximizes
the attention of a specific part at a particular location while minimizing the
attention where no parts appear. This is achieved by first selecting the
attention maps that correspond to the parts. A normalization over all parts is
then performed to ensure that each image region is occupied by no more than
one part. Finally, we minimize the entropy loss between each normalized
individual part and the semantic maps obtained during object parsing,
containing the correct part location. This not only facilitates stronger part
disentanglement but is also the key to generating a faithful holistic
structure of an object as it learns a generic prior topology knowledge about
object parts.

To enhance generation fidelity, we further employ a bottleneck encoder to
project the text tokens. This approach accelerates learning by leveraging
shared knowledge (common parts) and facilitating information exchange among
instances in each part. Each instance adjusts slightly to adapt to the fine-
grained part details during optimization.

Our contributions are as follows: (i) We pioneer a unique approach for fine-
grained part-level control in Text-to-Image (T2I) models, empowering users to
craft objects by selecting desired parts. This method streamlines the creative
process, marking a significant advancement in our capacity to manipulate and
reimagine visual content. (ii) We introduce PartCraft, a technique that
autonomously parses object parts and orchestrates them from different visual
concepts, resulting in the faithful creation of a novel object. (iii) For
enhanced part disentanglement and generation fidelity, we propose an entropy-
based normalized attention loss and leverage a bottleneck encoder. (iv) We
present two problem-specific quantitative metrics. Comprehensive experiments
on CUB-200-2011 (birds) and the Stanford Dogs dataset demonstrate the superior
performance of our method in generating novel objects, surpassing alternative
approaches in both qualitative and quantitative evaluations.

##  2 Related Work

Creative editing and generation. Creativity involves generating innovative
ideas or artifacts across various domains [13]. Extensive research has
explored the integration of creativity into Generative Adversarial Networks
(GANs) [23, 48, 61] and Variational Autoencoders (VAEs) [19, 17]. For example,
DoodlerGAN [28] learns and combines fine-level part components to create
sketches of new species. A recent study by [65] demonstrated decomposing
personalized concepts into distinct visual aspects, creatively recombined
through diffusion models. InstructPix2Pix [11] allows creative image editing
through instructions, while ConceptLab [56] aims to identify novel concepts
within a specified category, deviating from existing concepts. Different from
these works where editing/generation usually focuses on the whole object, we
instead focus on training a text-to-image generative model that can understand
parts, thus able to creatively generate new objects by seamlessly composing
different parts simply through selection.

Text-to-image generation. Recent advancements in large text-to-image (T2I)
diffusion models [57, 54, 7, 21, 47, 60, 20, 22] have made significant
improvements over conventional methods [77, 64, 72, 75, 50, 40] in producing
high-fidelity images from text prompts. Their application scope has expanded
to include both global [11, 45, 35] and localized [18, 73, 31, 5] image
editing tasks, demonstrating versatility. Methods such as [26, 6, 41, 76, 8,
37, 14, 74, 25, 16, 71] have introduced more granular spatial control, such as
incorporating semantic segmentation masks or bounding boxes, into large
pretrained diffusion models to guide image generation. Contrary to these
approaches, we focus on enhancing the control by enabling a straightforward
discrete selection of desired parts. We minimize the complexity and manual
intervention required by spatial controls, yet the model can compose selected
parts as a coherent object seamlessly. This not only simplifies the user’s
role in the generative process but also ensures that the compositional logic
and coherence are inherently managed by the model’s ability. As such, our
research is distinguished by focusing on the model’s inherent ability to
understand and apply part relationships instead of providing additional
controls.

Abstracting visual knowledge as a text token. The effectiveness of T2I models
is constrained by the user’s ability to articulate their desired image through
text. These models face challenges in faithfully replicating visual
characteristics from a reference set and generating innovative interpretations
in diverse contexts, even with detailed textual descriptions. To address this
challenge, various personalization techniques have been developed. These
techniques obtain a new word embedding from multiple images depicting the same
concept [27, 66, 2, 58, 39] or multiple new word embeddings for various
concepts within a single image [4] through inversion. The learned visual
concepts can be creatively reused in many image editing tasks. Nonetheless,
most of these approaches struggle to learn object parts, often not able to
follow the part selections due to part entanglement as they were designed to
learn object as a whole. In this work, we introduce a customized attention
loss that serves a dual purpose: ensure accurate positioning of each part and
enforce each image region occupied by no more than one part. This greatly
improves the part disentanglement, creating novel concepts with correct
appearances (see Fig. 2).

##  3 Methodology

Figure 3:  Overview of our PartCraft. All parts are organized into a
dictionary, and their semantic embeddings are learned through a textual
inversion approach. For instance, a text description like “a photo of a
[Head,42] [Wing,87]…” guides the optimization of the corresponding textual
embedding by reconstructing the associated image. To improve generation
fidelity, we incorporate a bottleneck encoder $f$ (MLP) to compute the
embedding $y$ (Eq. (4)) as input to the text encoder. To promote
disentanglement among learned parts, we minimize a specially designed
attention loss, denoted as $\mathcal{L}_{attn}$.

Given a set of unlabeled images depicting the same object (_e.g_., bird) with
different part details, we aim to train a T2I generative model that decomposes
parts of objects into text tokens and can recompose them in a novel way. To
that end, we propose PartCraft, as depicted in Fig. 3.

We start by discovering the parts in a three-tier hierarchy, as detailed in
Sec. 3.1. Paired with the training images $\\{x_{i}\\}^{N}_{i=1}$, this
semantic hierarchy subsequently serves as the supervision to fine-tune a pre-
trained text-to-image model, say a latent diffusion model [57], denoted as
$\\{\epsilon_{\theta},\tau_{\theta},\mathcal{E},\mathcal{D}\\}$, where
$\epsilon_{\theta}$ represents the diffusion denoiser, $\tau_{\theta}$ the
text encoder, and $\mathcal{E}/\mathcal{D}$ the autoencoder respectively. We
adopt the textual inversion technique [27]. Concretely, we learn a set of
pseudo-words $p^{*}$ for each part in the word embedding space with:

$\displaystyle\mathcal{L}_{ldm}$
$\displaystyle=\mathbb{E}_{z,t,p,\epsilon}\big{[}||\epsilon-\epsilon_{\theta}(z_{t},t,\tau_{\theta}(y_{p}))||^{2}_{2}\big{]},$
(1) $\displaystyle p^{*}$
$\displaystyle=\operatorname*{argmin}_{p}\,\mathcal{L}_{ldm},$ (2)

where $\epsilon\sim\mathcal{N}(0,1)$ denotes the unscaled noise, $t$ is the
time step, $z=\mathcal{E}(x)$ is the latent representation of the image,
$z_{t}$ is the latent noise at time $t$, and $y_{p}$ is the text condition
that includes $p$ as part of the text tokens. $\mathcal{L}_{ldm}$ is a
standard diffusion loss [33] to reconstruct the parts. As each object is
composed of a set of parts, its reconstruction is achieved by the
reconstruction of the associated set of parts. In other words, when all parts
are reconstructed properly, it will become a valid object.

###  3.1 Unsupervised Part Discovery

Figure 4: Part discovery within a semantic hierarchy involves partitioning
each image into distinct parts and forming semantic clusters across unlabeled
training data.

To minimize the labeling cost, we develop a scalable process to reveal the
underlying semantic hierarchy with parts in an unsupervised fashion. We
leverage the off-the-shelf vision model for image decomposition and
clustering. Specifically, given an image $x_{i}$, we employ DINOv2 [49] to
extract the feature map $F=\\{F_{i}=\mathrm{DINOv2}(x_{i})\\}_{i}^{N}$. We
then conduct three-level hierarchical clustering (see Fig. 4): (i) At the top
level, $k$-means is applied on all patches in $F$ with $k=2$ to separate
foregrounds and backgrounds $B$. (ii) At the middle level, $k$-means is
further applied on all foreground patches to acquire $M$ clusters representing
common parts, such as the heads of birds. (iii) At the bottom level, we
further group each of the $M$ clusters as well as the background cluster $B$
into $K$ splits. Each split refers to finer meanings, such as the head of a
specific bird species, or a specific background style. Lastly, each region of
an image will be tagged with the corresponding cluster index. We represent
these cluster tags as follows:

$\displaystyle p=(0,k_{0}),(1,k_{1}),...,(M,k_{M}),$ (3)

where the first pair refers to the background style, and the following $M$
pairs denote the combinations of $M$ parts (_e.g_., head, body, wings) each
associated with a specific object (_e.g_., sparrow), and
$k\in\\{1,\ldots,K\\}$. This description will be used as the textual prompt in
model training, such as “a photo of a [$p$]”. Please refer to the
supplementary material for more examples of the discovered semantic hierarchy.
This process also yields the segmentation mask of each $m$-th part, which we
define as $S_{m}$.

Motivation. While we can leverage off-the-shelf segmentation models such as
VLPart [63], the robustness relies on the generalizability of the model and
the part segmentation result is usually pre-defined and may be unstable for
unseen domains. As a result, we rely on our feature clustering method to
obtain the segmentation map, which also has a higher flexibility in choosing
the number of clusters (parts).

###  3.2 Part Token Bottleneck

In contrast to prior text inversion studies [27], our task requires learning a
greater quantity at the same time – specifically, $(M+1)K$-of word tokens
derived from a collection of discovered parts marked by inherent imperfections
(such as partial overlap and over splitting). This makes the learning task
more demanding. To enhance the learning process, we propose a neural network
$f(\cdot)$ comprising a two-layer MLP with ReLU activation:

$\displaystyle y_{p}=f(e(p)),$ (4)

where $y_{p}$ will be subsequently used as the input111Word templates such as
“a photo of a [*]” will be used. to the text encoder $\tau_{\theta}$ and
$e\in\mathbb{R}^{MK\times D}$ is a learnable word embedding dictionary that
maps $p$ to their respective embeddings.

Our design demonstrates quicker convergence than directly learning the final
word embeddings $e(\cdot)$ [27] (see Fig. 10). This could be attributed to the
entanglement of word embeddings in the conventional design, where there is no
information exchange among them during optimization. For instance, each token
doesn’t know they are learning for a specific part of a specific species. This
lack of communication leads to lower data efficiency and slower learning. With
the bottleneck $f$, it will first project the token into a common part
embedding space (_e.g_., head), then slightly adjust itself to adapt the fine-
grained part details. It’s worth noting that the conventional design is a
specific instance of our approach when $f$ is an identity function.

###  3.3 Learning to Craft by Parts

Fine-tuning the T2I model, rather than sorely learning pseudo-words, has been
shown to achieve better reconstruction of target concepts as demonstrated in
[58, 39]. However, this comes with a significant training cost. Thus, we apply
LoRA (low-rank adaptation) [34] to the cross-attention block for efficient
training. We then minimize the diffusion loss $\mathcal{L}_{ldm}$ (Eq. (1)) to
learn both pseudo-words and LoRA adapters.

While training with only $\mathcal{L}_{ldm}$, entanglement happens between
parts, as evident from the attention maps in the cross-attention block of the
denoiser $\epsilon_{\theta}$ (see Fig. 9). This entanglement arises due to the
correlation between parts (_e.g_., a bird head code is consistently paired
with a bird body code to represent the same species). To address this issue,
we introduce an entropy-based attention loss as regularization:

$\displaystyle\mathcal{L}_{attn}$
$\displaystyle=\mathbb{E}_{z,t,m}\big{[}-\big{(}S_{m}\log\hat{A}_{m}+(1-S_{m})\log(1-\hat{A}_{m})\big{)}\big{]},$
(5) $\displaystyle\hat{A}_{m,i,j}$
$\displaystyle=\frac{\bar{A}_{m,i,j}}{\sum_{k}\bar{A}_{k,i,j}},\quad\bar{A}_{m}=\frac{1}{L}\sum_{l}^{L}A_{l,m},$
(6)

where $A\in[0,1]^{M\times HW}$ represents the cross-attention map between the
$m$-th part and the noisy latent $z_{t}$, $L$ represents the number of
specific layers to select attention maps, $\hat{A}\in[0,1]^{M\times HW}$
represents the averaged and normalized cross-attention map over all parts and
$S_{m}\in\\{0,1\\}^{M\times HW}$ serves as the mask that indicates the
location of $m$-th part. In cases where the part is not present in the image
(_e.g_., occluded), we set both $S_{m}$ and $\hat{A}_{m}$ as 0 to exclude
them. Thus, the overall learning objective is defined as:

$\displaystyle\mathcal{L}_{total}=\mathcal{L}_{ldm}+\lambda_{attn}\mathcal{L}_{attn},$
(7)

where $\lambda_{attn}=0.01$. We focus on attention maps at the resolution of
$16\times 16$ where rich semantic information is captured [31]. Normalization
is performed at each location to ensure that the sum of a patch location
equals 1. This aims to maximize the attention of a specific part at a
particular location which implicitly minimizing the attention of other parts
similar to a softmax classification task. Compared to the mean-square based
attention loss [4], this intuitively ensures that a part only appears once at
a particular location, facilitating stronger disentanglement from other parts
during the denoising operation. When generating a part for a particular
location, the diffusion model $\epsilon_{\theta}$ should only attend to the
part instead of other non-related parts.

##  4 Experiments

Datasets. We demonstrate our selection task on two fine-grained object
datasets: CUB-200-2011 (birds) [67] which contains 5,994 training images, and
Stanford Dogs [36] which contains 12,000 training images.

Implementation. For part composition, we assess the model’s ability to combine
up to 4 different parts from 4 distinct species/objects. We set $M=5$ for bird
generation (head, front body/breast area, wings, legs, tail) and $M=7$ for dog
generation (forehead, eyes, mouth/nose, ears, neck, body/tail, legs). For both
datasets, $K$ is set as 256, ensuring sufficient coverage of all fine-grained
classes (_i.e_., 200 for birds and 120 for dogs). We randomly generate 500
images by sampling 500 sets of parts. For each image, we randomly replace an
original part with any part from another 500 non-overlapping sets of parts.
The resulting set of parts may take the form of “$(0,k_{A})$ $(1,k_{B})$
$(2,k_{C})$ … $(M,k_{D})$”, representing a composition from species A, B, C,
and D. Stable Diffusion v1.5 [57] is used. Please see the supplementary
material for further training details.

Method Learnable Token Fine Tune Disentanglement Bottleneck Textual Inversion
[27] ✓ ✗ ✗ ✗ DreamBooth [58] ✓ LoRA* ✗ ✗ CustomDiffusion [39] ✓ $K/V$ ✗ ✗
Break-a-scene [4] ✓ LoRA* MSE ✗ PartCraft (Ours) ✓ LoRA Eq. (5) ✓

Table 1: Comparing our and alternative methods in design properties. *: We
fine-tuned the added LoRA [34] adapter rather than the entire diffusion model
$\epsilon_{\theta}$ due to resource limit. MSE is a mean-square based
attention loss used in [4].

We compare our method with the recent personalization methods: Textual
Inversion (TI) [27], DreamBooth (DB) [58], Custom Diffusion (CD) [39], Break-
a-scene (BaS) [4]. These personalization methods were designed to take single
or multiple images with associated labeled objects as input. The text prompt
for each image is as simple as “a photo of [$p$]” where $p$ is expressed in
Eq. (3), since we do not rely on complex prompts. We employ the official
implementations released by the original authors for training. We summarize
the main design properties of all compared methods in Tab. 1.

Evaluation metrics. To assess a model’s ability to disentangle and composite
parts, we introduce two metrics: (a) exact matching rate (EMR) and (b) cosine
similarity (CoSim) between the $k$-means embeddings of the parts of real and
generated images. Utilizing the pre-trained $k$-means from Sec. 3.1, we
predict the parts of generated images. EMR quantifies how accurately the
cluster index of parts of generated images matches the parts of the
corresponding real images whereas CoSim measures the cosine similarity between
the $k$-means centroid vector that the part belongs to between generated and
real images. These metrics assess the model’s ability to follow the input
parts and accurately reconstruct them, with perfect disentanglement indicated
by EMR of 1 and CoSim of 1. A detailed algorithm is provided in the
supplementary material. We also measure image generation quality using FID
[32] to assess model performance in terms of image distribution. Additionally,
we compute the average pairwise cosine similarity between CLIP [53]/DINO [12]
embeddings of generated and real class-specific images following [58]. Each
generated image is conditioned on the parts of the corresponding real image.
This results in 5,994 generated images for birds and 12,000 generated images
for dogs.

###  4.1 Part Reconstruction and Image Quality Evaluation

We first assess the ability of different methods to learn parts as text tokens
by evaluating how well they can accurately reconstruct the parts (this also
means image generation with original parts).

Method Birds: CUB-200-2011 Dogs: Stanford Dogs FID CLIP DINO EMR CoSim FID
CLIP DINO EMR CoSim Textual Inversion [27] 10.10 0.784 0.607 0.305 0.842 23.36
0.652 0.532 0.218 0.754 DreamBooth [58] 12.94 0.775 0.594 0.355 0.856 22.65
0.660 0.563 0.275 0.777 Custom Diffusion [39] 37.61 0.694 0.504 0.338 0.833
42.41 0.593 0.491 0.253 0.755 Break-a-Scene [4] 20.05 0.742 0.549 0.390 0.854
24.20 0.633 0.532 0.300 0.775 PartCraft (Ours) 12.86 0.783 0.618 0.460 0.882
16.92 0.669 0.573 0.358 0.796

Table 2: Quantitative comparison for part reconstruction.

(a) TI

(b) DB

(c) CD

(d) BaS

(e) PartCraft

Figure 5: Visual comparison under the part reconstruction setting. All images
are generated by using the original parts of respective objects.

In Tab. 2, we summarize the performance of respective methods on the bird and
dog generation, respectively. We highlight four observations: (i) Textual
Inversion performs quite well compared to DreamBooth, CustomDiffusion, and
Break-a-scene in terms of FID, CLIP, and DINO scores although did not fine-
tune the diffusion model $\epsilon_{\theta}$. This may be due to the potential
risk of overfitting when fine-tuning $\epsilon_{\theta}$ especially when
learning a vast array of new concepts with many update iterations. It is also
not uncommon to carefully tune the learning rate and the training iterations
in these models when fine-tuning new concepts (_e.g_., only 800-1000 steps of
updates to learn a new concept in [4]). (ii) Nonetheless, fine-tuning the
diffusion model $\epsilon_{\theta}$ can help improve the ability to follow
prompts as shown by increased EMR and CoSim scores (_e.g_., EMR of at least 5%
in DreamBooth). (iii) Break-a-scene has a better ability to reconstruct the
parts as shown by EMR and CoSim, this is due to the attention loss explicitly
forcing the parts to focus on the respective semantic region. (iv) PartCraft
achieves the best performance in DINO, EMR, and CoSim scores (_e.g_., 7%
better in EMR compared to Break-a-scene). This indicates that not only does
our image-generation ability perform comparably well with textual inversion,
but PartCraft is also able to disentangle the parts learning so that it can
follow the prompt instructions more accurately to generate the parts in a
cohort. In Fig. 5, we present generated images from different methods, with
CustomDiffusion exhibiting high-contrast images, possibly due to unconstrained
fine-tuning on the cross-attention components $K/V$ and resulting in worse FID
scores.

(a) CUB-200-2011

(b) Stanford Dogs

Figure 6: Quantitative comparisons of part composition in terms of EMR and
CoSim.

Figure 7: Visual comparison on 4-species (specified on the top row) mixed
generation. The last column indicates generated images with different styles
(_i.e_., DSLR, Van Gogh, Oil Painting, Pencil Drawing).

###  4.2 Part Composition Evaluation

In this section, we assess the part composition ability of different methods.
In this experiment, we generate the image by mixing different parts from
different species. Our findings, as shown in Fig. 6, can be summarized as
follows: (i) As the number of composited parts increases, EMR and CoSim
decrease, reflecting the challenge of composing multiple diverse parts. (ii)
Break-a-scene and PartCraft achieve notably higher EMR and CoSim scores,
thanks to disentanglement through attention loss minimization. (iii) PartCraft
outperforms Break-a-scene significantly by token bottleneck and tailored
attention loss.

In Fig. 7, we visualize the results of composing 4 different parts. While all
images appear realistic, most methods struggle to assemble all 4 parts. For
instance, Break-a-scene missed out on the flurry body of kerry blue terrier
(rightmost column). In contrast, our methods successfully combine 4 different
parts from 4 different species, demonstrating the superior ability of our
approach to part composition. We also visualize additional examples of our
method in the supplementary material.

Furthermore, we explore the versatility of the adapted model by generating
images with simple styles such as pencil drawing. While most methods
successfully incorporate specific styles into the generated image, Custom
Diffusion often fails to do so, possibly due to the unconstrained fine-tuning
of the cross-attention components $K/V$.

Figure 8: Ablation on our part token bottleneck and attention loss under the
part composition on CUB-200-2011 birds.

Figure 9: Cross-attention map of each part (top) without and (bottom) with our
attention loss.

Figure 10: Generated images of cardinal over different stages of training.

###  4.3 Ablation Studies

Component analysis. In Fig. 8, we evaluate the effect of our proposed
components (token bottleneck and attention loss) on creating novel bird
species. (i) Removing the bottleneck outlined in Eq. (4) degrades the
generation quality as evidenced by a higher FID score (12.86 $\rightarrow$
16.36) even though both EMR and CoSim remain. (ii) By replacing our
$\mathcal{L}_{attn}$ with the MSE loss as proposed in [4], we observe
significant deterioration in both EMR and CoSim. (iii) Finally, incorporating
both the projector and our attention loss performs the best. This improvement
highlights the necessity of incorporating interactions between multiple parts
to achieve more effective part disentanglement and optimization.

Cross-attention visualization. Our attention loss plays a crucial role in
token disentanglement. We demonstrate the impact of this loss in Fig. 9, where
we observe significantly enhanced disentanglement after explicitly guiding
attention to focus on distinct semantic regions.

(a) Textual Inversion

(b) Break-a-scene

(c) PartCraft (Ours)

Figure 11: 2D tSNE [43] projection of word embeddings. Different colors
represent different common parts. (correspond to the segmentation mask in Fig.
4).

Part word embedding space. We visualize the word embeddings of learned tokens
of Textual Inversion [27], Break-a-scene [4] and our PartCraft for birds
generation (CUB-200-2011 [67]) through tSNE [43] in Fig. 11. In our PartCraft,
the word embeddings are the projected embeddings through Eq. (4). We can see
that our projected version has a better semantic meaning such that the part
embeddings are clustered together by their semantic meaning (_e.g_., head). We
believe this is one of the reasons that our PartCraft outperforms previous
methods in which we can compose all parts seamlessly yet with higher quality.

Convergence analysis. We present a visual comparison of images generated by
various methods in Fig. 10, spanning from the initial to the final stages of
training. Notably, our PartCraft demonstrates an ability to learn new concepts
at even the early stages of training. Without the bottleneck encoder, we
observe that the learning speed drops significantly (as evidenced by
generating wrong part detail), indicating how the bottleneck serves as an
important component when learning new tokens that have shared properties
(_e.g_., common part).

Figure 12: (Top): Using learned parts to modify the property of other domains
such as cat, and lion with prompt such as “A cat with [beagle’s ear]”.
(Bottom) We can also repurpose learned parts for creative image generation
using prior knowledge in Stable Diffusion with prompt such as “A robot
designed inspired by [red header woodpecker’s head] and [blue jay’s body]”.

Transferability for Creativity. (i) In Fig. 12, we demonstrate that not only
it can compose parts within the training domain (_e.g_., birds), but it can
also transfer the learned parts to and combine with other domains (_e.g_.,
cat). This enables the creation of unique combinations, such as a cat with a
dog’s ear. (ii) Leveraging the prior knowledge embedded in Stable Diffusion,
PartCraft can also repurpose learned parts for creative image generation. An
example of this is the generation of a bird-shaped robot adorned with various
parts. These examples showcase PartCraft’s immense potential for diverse and
limitless creative applications. Please see the supplementary material for
more examples.

##  5 Conclusion

We propose a new way of control in generative AI. Instead of text or sketch,
we “select” desired parts to create an object. We addressed the challenge of
learning parts in T2I models by introducing a customized attention loss. This
loss serves a dual purpose: to ensure parts are at the right location and to
ensure each location is occupied by not more than one part. This greatly
improves the part disentanglement. We further employ a non-linear bottleneck
encoder to improve generation fidelity. Our model, PartCraft, can seamlessly
compose different parts from different objects, creating objects that do not
exist yet holistically correct and plausible objects by mixing them. Extensive
experiments demonstrated PartCraft’s superior performance in both qualitative
and quantitative evaluation. Moreover, the learned parts demonstrate strong
transferability. We hope that our PartCraft will empower artists, designers,
and enthusiasts to bring the creations of their dreams to reality.

##  6 Limitations and Future Works

It is worth noting that the accuracy of obtained parts may be affected by
using a self-supervised pre-trained feature extractor. Future work may explore
the incorporation of encoders, such as [69], to improve part accuracy. We also
observed challenges in composing relatively small parts, like tails and legs,
which require further investigation. Additionally, we are also exploring
cross-domain generation, _i.e_., combining learned parts from different
datasets to create objects with even more diverse parts. For instance, we can
merge non-rigid parts (_e.g_., parts from quadrupled animals) with rigid parts
(_e.g_., parts from cars/airplanes) and form a creative structure (_e.g_., a
car that has horse legs instead of wheels). This not only further improves the
applicability of PartCraft but also serves as a stepping stone to creative
generation, as generative AI progresses, continually expanding the limits of
achievable creativity and artistic expression.

##  7 Acknowledgements

We extend our special thanks to Jia Wei Sii for her help in creating figures
and discussing the main concept. We are also grateful to Ruoyi Du, Zhiyu Qu,
Chee Seng Chan, and the reviewers for their fruitful comments and corrections
on our draft, methodology, and experiments.

## References

  * [1] Sims4 faces. https://huggingface.co/datasets/rocca/sims4-faces (2022)
  * [2] Alaluf, Y., Richardson, E., Metzer, G., Cohen-Or, D.: A neural space-time representation for text-to-image personalization. arXiv preprint arXiv:2305.15391 (2023)
  * [3] Amir, S., Gandelsman, Y., Bagon, S., Dekel, T.: On the effectiveness of vit features as local semantic descriptors. In: ECCV (2022)
  * [4] Avrahami, O., Aberman, K., Fried, O., Cohen-Or, D., Lischinski, D.: Break-a-scene: Extracting multiple concepts from a single image. In: SIGGRAPH Asia (2023)
  * [5] Avrahami, O., Fried, O., Lischinski, D.: Blended latent diffusion. ACM TOG (2023)
  * [6] Avrahami, O., Hayes, T., Gafni, O., Gupta, S., Taigman, Y., Parikh, D., Lischinski, D., Fried, O., Yin, X.: Spatext: Spatio-textual representation for controllable image generation. In: CVPR (2023)
  * [7] Balaji, Y., Nah, S., Huang, X., Vahdat, A., Song, J., Kreis, K., Aittala, M., Aila, T., Laine, S., Catanzaro, B., et al.: ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 (2022)
  * [8] Bar-Tal, O., Yariv, L., Lipman, Y., Dekel, T.: Multidiffusion: Fusing diffusion paths for controlled image generation. In: ICML (2023)
  * [9] Biederman, I.: Recognition-by-components: a theory of human image understanding. Psychological review (1987)
  * [10] Bonnardel, N., Marmèche, E.: Towards supporting evocation processes in creative design: A cognitive approach. International Journal of Human-Computer Studies (2005)
  * [11] Brooks, T., Holynski, A., Efros, A.A.: Instructpix2pix: Learning to follow image editing instructions. In: CVPR (2023)
  * [12] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: ICCV (2021)
  * [13] Cetinic, E., She, J.: Understanding and creating art with ai: Review and outlook. ACM Transactions on Multimedia Computing, Communications, and Applications (2022)
  * [14] Chefer, H., Alaluf, Y., Vinker, Y., Wolf, L., Cohen-Or, D.: Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM TOG (2023)
  * [15] Chen, C., Li, O., Tao, D., Barnett, A., Rudin, C., Su, J.K.: This looks like that: deep learning for interpretable image recognition. NeurIPS (2019)
  * [16] Chen, M., Laina, I., Vedaldi, A.: Training-free layout control with cross-attention guidance (2024)
  * [17] Cintas, C., Das, P., Quanz, B., Tadesse, G.A., Speakman, S., Chen, P.Y.: Towards creativity characterization of generative models via group-based subset scanning. In: IJCAI (2022)
  * [18] Couairon, G., Verbeek, J., Schwenk, H., Cord, M.: Diffedit: Diffusion-based semantic image editing with mask guidance. In: ICLR (2023)
  * [19] Das, P., Quanz, B., Chen, P.Y., Ahn, J.w., Shah, D.: Toward a neuro-inspired creative decoder. In: IJCAI (2020)
  * [20] Ding, M., Yang, Z., Hong, W., Zheng, W., Zhou, C., Yin, D., Lin, J., Zou, X., Shao, Z., Yang, H., et al.: Cogview: Mastering text-to-image generation via transformers. In: NeurIPS (2021)
  * [21] Ding, M., Zheng, W., Hong, W., Tang, J.: Cogview2: Faster and better text-to-image generation via hierarchical transformers. In: NeurIPS. vol. 35 (2022)
  * [22] Du, R., Chang, D., Hospedales, T., Song, Y.Z., Ma, Z.: Demofusion: Democratising high-resolution image generation with no $$$. In: CVPR (2024)
  * [23] Elgammal, A., Liu, B., Elhoseiny, M., Mazzone, M.: Can: Creative adversarial networks generating "art" by learning about styles and deviating from style norms. In: ICCC (2017)
  * [24] Felzenszwalb, P.F., Huttenlocher, D.P.: Pictorial structures for object recognition. IJCV (2005)
  * [25] Feng, W., He, X., Fu, T.J., Jampani, V., Akula, A.R., Narayana, P., Basu, S., Wang, X.E., Wang, W.Y.: Training-free structured diffusion guidance for compositional text-to-image synthesis. In: ICLR (2022)
  * [26] Gafni, O., Polyak, A., Ashual, O., Sheynin, S., Parikh, D., Taigman, Y.: Make-a-scene: Scene-based text-to-image generation with human priors. In: ECCV (2022)
  * [27] Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A.H., Chechik, G., Cohen-Or, D.: An image is worth one word: Personalizing text-to-image generation using textual inversion. In: ICLR (2023)
  * [28] Ge, S., Goswami, V., Zitnick, C.L., Parikh, D.: Creative sketch generation. In: ICLR (2021)
  * [29] Han, X., Cao, Y., Han, K., Zhu, X., Deng, J., Song, Y.Z., Xiang, T., Wong, K.Y.K.: Headsculpt: Crafting 3d head avatars with text. In: NeurIPS (2024)
  * [30] He, J., Chen, J., Lin, M.X., Yu, Q., Yuille, A.L.: Compositor: Bottom-up clustering and compositing for robust part and object segmentation. In: CVPR (2023)
  * [31] Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to-prompt image editing with cross attention control. In: ICLR (2023)
  * [32] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS 30 (2017)
  * [33] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: NeurIPS (2020)
  * [34] Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-rank adaptation of large language models. In: ICLR (2022)
  * [35] Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., Irani, M.: Imagic: Text-based real image editing with diffusion models. In: CVPR (2023)
  * [36] Khosla, A., Jayadevaprakash, N., Yao, B., Li, F.F.: Novel dataset for fine-grained image categorization: Stanford dogs. In: CVPRW (2011)
  * [37] Kim, Y., Lee, J., Kim, J.H., Ha, J.W., Zhu, J.Y.: Dense text-to-image generation with attention modulation. In: ICCV (2023)
  * [38] Krause, J., Jin, H., Yang, J., Fei-Fei, L.: Fine-grained recognition without part annotations. In: CVPR (2015)
  * [39] Kumari, N., Zhang, B., Zhang, R., Shechtman, E., Zhu, J.Y.: Multi-concept customization of text-to-image diffusion. In: CVPR (2023)
  * [40] Li, B., Qi, X., Lukasiewicz, T., Torr, P.: Controllable text-to-image generation. In: NeurIPS (2019)
  * [41] Li, Y., Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., Li, C., Lee, Y.J.: Gligen: Open-set grounded text-to-image generation. In: CVPR (2023)
  * [42] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: ICLR (2018)
  * [43] Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of Machine Learning Research (2008)
  * [44] Mo, S., Mu, F., Lin, K.H., Liu, Y., Guan, B., Li, Y., Zhou, B.: Freecontrol: Training-free spatial control of any text-to-image diffusion model with any condition. arXiv preprint arXiv:2312.07536 (2023)
  * [45] Mokady, R., Hertz, A., Aberman, K., Pritch, Y., Cohen-Or, D.: Null-text inversion for editing real images using guided diffusion models. In: CVPR (2023)
  * [46] Nagai, Y., Taura, T., Mukai, F.: Concept blending and dissimilarity: factors for creative concept generation process. Design studies (2009)
  * [47] Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., Chen, M.: Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 (2021)
  * [48] Nobari, A.H., Rashad, M.F., Ahmed, F.: Creativegan: Editing generative adversarial networks for creative design synthesis. arXiv preprint arXiv:2103.06242 (2021)
  * [49] Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.Y., Li, S.W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., Bojanowski, P.: Dinov2: Learning robust visual features without supervision (2023)
  * [50] Qiao, T., Zhang, J., Xu, D., Tao, D.: Mirrorgan: Learning text-to-image generation by redescription. In: CVPR (2019)
  * [51] Qu, Z., Xiang, T., Song, Y.Z.: Sketchdreamer: Interactive text-augmented creative sketch ideation. In: BMVC (2023)
  * [52] Qu, Z., Yang, L., Zhang, H., Xiang, T., Pang, K., Song, Y.Z.: Wired perspectives: Multi-view wire art embraces generative ai. In: CVPR (2024)
  * [53] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML (2021)
  * [54] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 (2022)
  * [55] Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F., Zeng, Z., Zhang, H., Li, F., Yang, J., Li, H., Jiang, Q., Zhang, L.: Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159 (2024)
  * [56] Richardson, E., Goldberg, K., Alaluf, Y., Cohen-Or, D.: Conceptlab: Creative generation using diffusion prior constraints. arXiv preprint arXiv:2308.02669 (2023)
  * [57] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR (2022)
  * [58] Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., Aberman, K.: Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In: CVPR (2023)
  * [59] Runco, M.A., Jaeger, G.J.: The standard definition of creativity. Creativity research journal (2012)
  * [60] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.: Photorealistic text-to-image diffusion models with deep language understanding. In: NeurIPS. vol. 35 (2022)
  * [61] Sbai, O., Elhoseiny, M., Bordes, A., LeCun, Y., Couprie, C.: Design: Design inspiration from generative networks. In: ECCVW (2019)
  * [62] Sun, P., Chen, S., Luo, P.: Grounded segment anything: From objects to parts. https://github.com/Cheems-Seminar/grounded-segment-any-parts (2023)
  * [63] Sun, P., Chen, S., Zhu, C., Xiao, F., Luo, P., Xie, S., Yan, Z.: Going denser with open-vocabulary part segmentation. arXiv preprint arXiv:2305.11173 (2023)
  * [64] Tao, M., Tang, H., Wu, F., Jing, X.Y., Bao, B.K., Xu, C.: Df-gan: A simple and effective baseline for text-to-image synthesis. In: CVPR (2022)
  * [65] Vinker, Y., Voynov, A., Cohen-Or, D., Shamir, A.: Concept decomposition for visual exploration and inspiration. In: SIGGRAPH Asia (2023)
  * [66] Voynov, A., Chu, Q., Cohen-Or, D., Aberman, K.: P+: Extended textual conditioning in text-to-image generation. arXiv preprint arXiv:2303.09522 (2023)
  * [67] Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The Caltech-UCSD Birds-200-2011 Dataset (2011)
  * [68] Wang, X., Darrell, T., Rambhatla, S.S., Girdhar, R., Misra, I.: Instancediffusion: Instance-level control for image generation. arXiv preprint arXiv:2402.03290 (2024)
  * [69] Wei, Y., Zhang, Y., Ji, Z., Bai, J., Zhang, L., Zuo, W.: Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. In: ICCV (2023)
  * [70] Wilkenfeld, M.J., Ward, T.B.: Similarity and emergence in conceptual combination. Journal of Memory and Language (2001)
  * [71] Xie, J., Li, Y., Huang, Y., Liu, H., Zhang, W., Zheng, Y., Shou, M.Z.: Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion. In: ICCV (2023)
  * [72] Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., He, X.: Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In: CVPR (2018)
  * [73] Yang, B., Gu, S., Zhang, B., Zhang, T., Chen, X., Sun, X., Chen, D., Wen, F.: Paint by example: Exemplar-based image editing with diffusion models. In: CVPR (2023)
  * [74] Yang, Z., Wang, J., Gan, Z., Li, L., Lin, K., Wu, C., Duan, N., Liu, Z., Liu, C., Zeng, M., et al.: Reco: Region-controlled text-to-image generation. In: CVPR (2023)
  * [75] Yin, G., Liu, B., Sheng, L., Yu, N., Wang, X., Shao, J.: Semantics disentangling for text-to-image generation. In: CVPR (2019)
  * [76] Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: ICCV (2023)
  * [77] Zhu, M., Pan, P., Chen, W., Yang, Y.: Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. In: CVPR (2019)

##  Appendix 0.A Implementation Details

We conducted training on a single GeForce RTX 3090 GPU with a batch size of 2
over 100 epochs. AdamW [42] optimizer was employed with a constant learning
rate of 0.0001 and weight decay of 0.01. Only random horizontal flip
augmentation is used. $512\times 512$ image resolution is applied.

We adopted the LoRA design [34] from diffusers
library222https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora.py,
in which the low-rank adapters were added to the $QKV$ and $out$ components of
all cross-attention modules.

Regarding the attention loss (see Eq. (5)), we selected cross-attention maps
with a feature map size of $16\times 16$. The specific layers chosen for this
purpose were as follows:

  * •

down_blocks.2.attentions.0.transformer_blocks.0.attn2

  * •

down_blocks.2.attentions.1.transformer_blocks.0.attn2

  * •

up_blocks.1.attentions.0.transformer_blocks.0.attn2

  * •

up_blocks.1.attentions.1.transformer_blocks.0.attn2

  * •

up_blocks.1.attentions.2.transformer_blocks.0.attn2

##  Appendix 0.B Implementation of EMR and CoSim

Algorithm 1 EMR and CoSim for part composition

⬇

1# part_predictor: obtained via Sec 3.1

2# pipeline: diffusion generation pipeline

3# real_xs: real image (Nx512x512x3) where N up to 4

4# M: number of parts

5# D: number of dino feature dimension

6

7# obtain the prompt of the real image (Eq.3)

8p_input = part_predictor.predict(real_xs[0]) # (M+1)

9p_idxs = [0,1,...,M]

10

11for real_x in real_xs[1:]:

12 p_real = part_predictor.predict(real_x) # (M+1)

13

14 # replace one part

15 rand_idx = randint(len(p_idxs))

16 rand_pop = p_idxs.pop(rand_idx)

17 p_input[rand_pop] = p_real[rand_pop]

18

19gen_x = pipeline(p_input)

20p_gen = part_predictor.predict(gen_x) # (M+1)

21

22p_input_embs = part_predictor.get_centroids(p_input) # (M+1,D)

23p_gen_embs = part_predictor.get_centroids(p_gen) # (M+1,D)

24

25EMR = average(p_input == p_gen)

26CoSim = average(cossim(p_input_embs, p_gen_embs))

Algorithm 2 EMR and CoSim for part reconstruction

⬇

1# part_predictor: obtained via Sec 3.1

2# pipeline: diffusion generation pipeline

3# real_x: real image (512x512x3)

4# M: number of parts

5# D: number of dino feature dimension

6

7# obtain the prompt of the real image (Eq.3)

8p_real = part_predictor.predict(real_x) # (M+1)

9gen_x = pipeline(p_real)

10p_gen = part_predictor.predict(gen_x) # (M+1)

11

12# an example of "p" is [4, 222, 55, 23, 98, 22]

13# in the "pipeline", we prepend word template like "a photo of a "

14# e.g., "a photo of a [0,4] [1,222] ... [M,K]"

15# the token [*,*] will be replaced by its embedding computed via Eq.4

16

17p_real_embs = part_predictor.get_centroids(p_real) # (M+1,D)

18p_gen_embs = part_predictor.get_centroids(p_gen) # (M+1,D)

19

20EMR = average(p_real == p_gen)

21CoSim = average(cossim(p_real_embs, p_gen_embs))

Our evaluation algorithms for the Exact Matching Rate (EMR) and Cosine
Similarity (CoSim) between generated and real images are presented in
Algorithms 1 and 2, respectively. Each algorithm is designed to evaluate a
single sample. For the evaluations in Section 4.1, we computed the average
results over 500 iterations using Algorithm 1. Similarly, for Section 4.2, we
averaged the outcomes over 5,994 and 12,000 iterations for the CUB-200-2011
(birds) and Stanford Dogs datasets, respectively.

##  Appendix 0.C Examples of our part discovery

In Fig. 13, we display a few examples of our obtained segmentation masks and
associated sets of parts.

Figure 13: Three example outputs of our part discovery. Note that all these
discrete IDs can be translated easily with minimal effort. For instance, 0 is
background, 1 is tail, etc.

##  Appendix 0.D More examples

(a)

(b)

Figure 14: We present additional examples of creative generation. (a) displays
the effects of transferring learned parts, _e.g_., replacing a leopard head
with a chow’s head. (b) displays using the learned parts to inspire some
character/product designs.

(a)

(b)

Figure 15: We present additional examples of images generated by our
PartCraft, featuring a random selection of parts.

###  0.D.1 PartCraft for Character Face Creation

Figure 16: We present additional examples of images generated by our
PartCraft, using data from Sims4-Faces [1]. We transfer three different parts
(_i.e_., hair, eyes, mouth) from source B to target A.

To further demonstrate the capability of PartCraft in learning parts, we
conducted training on the publicly available Sims4-Faces [1] dataset. We
selected a subset of 200 faces (100 men and 100 women) from the dataset and
applied our part discovery algorithm to parse the faces into various parts
(_e.g_., hair, eyes, mouth, ear, neck). In Fig. 16, two sets of images were
generated from their original parts (sources A and B), and a specific part
from source B can be integrated into target A. This experiment shows that our
PartCraft can work on domains where the creation and manipulation of parts are
essential.

##  Appendix 0.E Further Analysis

###  0.E.1 Attention loss weight

In Fig. 17 and Tab. 3, we summarize the results of our ablation study on the
impact of $\lambda_{attn}$. We observed that $\lambda_{attn}=0.01$ frequently
yields the best EMR and CoSim scores, while also delivering comparable FID
scores. Consequently, we have adopted this value as the default in our
experiments.

Figure 17: Ablation on the effect of $\lambda_{attn}$ for virtual creature generation on CUB-2011 birds. Different colors represent different numbers of composited parts. $\lambda_{attn}$ | 0.1 | 0.01 | 0.001 | 0.0001 | 0.00001
---|---|---|---|---|---
FID ($\downarrow$) | 19.08 | 12.86 | 12.44 | 11.78 | 11.64
EMR ($\uparrow$) | 0.339 | 0.460 | 0.445 | 0.425 | 0.397
CoSim ($\uparrow$) | 0.851 | 0.882 | 0.880 | 0.878 | 0.872
Table 3: Ablation on the effect of $\lambda_{attn}$ for conventional
generation on CUB-200-2011 birds.

###  0.E.2 Hyperparameter of Break-a-Scene

We show the $\lambda=0.1/0.001$ result (CUB-200-2011). We hypothesize that
stronger attention loss may cause overfitting and neglect the generation
quality, thus lowering EMR/CoSim. Thus, we use $\lambda=0.01$ for Break-a-
Scene experiments.

$\lambda$ 0.1 0.01 0.001 EMR 0.366 0.390 0.344 CoSim 0.841 0.854 0.832