# MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout
Guidance††thanks: Work done during internship of X. Wang and Q. Huang at
Alibaba Group.

X. Wang1,2,+ Siming Fu1,+ Qihan Huang1,2 Wanggui He1 Hao Jiang1,†
1Alibaba Group 2Zhejiang University
+Equal contribution †Corresponding author
https://MS-Diffusion.github.io

###### Abstract

Recent advancements in text-to-image generation models have dramatically
enhanced the generation of photorealistic images from textual prompts, leading
to an increased interest in personalized text-to-image applications,
particularly in multi-subject scenarios. However, these advances are hindered
by two main challenges: firstly, the need to accurately maintain the details
of each referenced subject in accordance with the textual descriptions; and
secondly, the difficulty in achieving a cohesive representation of multiple
subjects in a single image without introducing inconsistencies. To address
these concerns, our research introduces the MS-Diffusion framework for layout-
guided zero-shot image personalization with multi-subjects. This innovative
approach integrates grounding tokens with the feature resampler to maintain
detail fidelity among subjects. With the layout guidance, MS-Diffusion further
improves the cross-attention to adapt to the multi-subject inputs, ensuring
that each subject condition acts on specific areas. The proposed multi-subject
cross-attention orchestrates harmonious inter-subject compositions while
preserving the control of texts. Comprehensive quantitative and qualitative
experiments affirm that this method surpasses existing models in both image
and text fidelity, promoting the development of personalized text-to-image
generation.

Figure 1: Representative outputs showcase the capabilities of MS-Diffusion in
typical applications. The MS-Diffusion framework facilitates personalization
across both single-subject scenarios (the upper panel) and multi-subject
contexts (the lower panel). Notably, while preserving the intricacies of
subject detail, MS-Diffusion achieves a marked enhancement in textual
fidelity.

##  1 Introduction

Recent advancements in text-to-image (T2I) diffusion methodologies [27, 29, 3]
have propelled the field to new heights, realizing unprecedented levels of
photorealism while demonstrating a refined ability to conform to textual
prompts. These achievements have spurred the development of a broad spectrum
of applications, most prominently in the domain of personalized T2I (P-T2I)
models, which are tasked with the complex undertaking of assimilating and
regenerating novel visual concepts or subjects across diverse contexts with a
heightened demand for conceptual and compositional fidelity. Despite fine-
tuning-based techniques such as DreamBooth [28] and Textual Inversion [9]
yield results with considerable accuracy, they necessitate extensive resources
for tuning individual instances and for the storage of multiple models, which
renders them less feasible for widespread application. To circumvent these
resource-intensive requirements, fine-tuning-free alternatives have come to
the fore.

Single-subject driven personalization methods, IP-Adapter [38] and ELITE [36]
for instance, introduce a specialized cross-attention mechanism that
distinctly processes text and image features, thereby affording the
possibility to employ reference images directly as visual prompts within the
model. Furthermore, recent works have employed multi-subject driven
customization methodologies to concatenate visual concepts with textual
prompts, offering a glimpse of the potential in techniques like SSR-Encoder
[40], $\lambda$-ECLIPSE [20], Emu2 [30], and KOSMOS-G [19]. These models
harness identity data and amalgamate it with text via cross-attention,
exhibiting proficiency in adjusting textures. Nevertheless, above zero-shot
personalization methods encounter limitations, notably in adapting a pressing
question pertains to the congruence of granular details between the subject
depicted in the synthesized imagery and its corresponding subjects, along with
the degree of semblance between the content of the generated image and
associated textual descriptions. The challenge is further amplified in
scenarios requiring the personalization of multiple subjects. Especially, the
challenge of ensuring harmonious representation when multiple subjects are
incorporated—specifically, the elucidation of whether the resultant image
manifests any discordant elements or deleterious interactions in accordance
with textual directives and multi-subject referential controls. As illustrated
in Figure 2, multi-subject personalization methods frequently incur notable
detail inaccuracies in a fine-tuning-free framework and often lead to subject
neglect, subject overcontrol, and subject-subject conflict issue within the
generated images.

To confront these identified challenges, we are the first to introduce the
layout-guided zero-shot image personalization with multiple subjects (MS-
Diffusion) framework, which consolidates the accommodation of multiple
subjects, the incorporation of zero-shot learning capabilities, the provision
of layout guidance, and the preservation of the foundational model’s
parameters. Firstly, we design the grounding resampler to extract the subject
detailed features and integrate them with grounding information containing
entities and boxes. As an image projection module, the proposed grounding
resampler can enhance the subject fidelity while appending semantic and
positional priors. Secondly, we propose a novel cross-attention mechanism for
multiple subjects, which confines subjects to represent themselves in specific
areas. This confluence not only facilitates the efficacious infusion of multi-
subject data into the model but also mitigates conflicts between text and
image subject control conditions. Such an approach culminates in the refined
granularity of control over the image’s multi-subject composition. The
experimental results demonstrate our method consistently outperforms the
state-of-the-art approaches on all the benchmarks. We conclude the previous
P-T2I works and provide an overall comparison in Table 1. The contributions
can be summarized as follows:

Table 1: An overview of previous studies of P-T2I tasks. MS-Diffusion is the first approach to support multi-reference zero-shot P-T2I generation with layout guidance and base model freezing. Method |  Zero Shot |  Multi Subject |  Base Model Freezing |  MLLM Free |  Layout Guidance
---|---|---|---|---|---
Textual Inversion [9] |  ✗ |  ✗ |  ✓ |  ✓ |  ✗
DreamBooth [28] |  ✗ |  ✗ |  ✗ |  ✓ |  ✗
ELITE [36] |  ✓ |  ✗ |  ✗ |  ✓ |  ✗
BLIP-Diffusion [15] |  ✓ |  ✗ |  ✗ |  ✗ |  ✗
IP-Adapter [38] |  ✓ |  ✗ |  ✓ |  ✓ |  ✗
Emu2 [30] |  ✓ |  ✓ |  ✗ |  ✗ |  ✗
Kosmos-G [19] |  ✓ |  ✓ |  ✓ |  ✗ |  ✗
$\lambda$-ECLIPSE [20] |  ✓ |  ✓ |  ✓ |  ✗ |  ✗
SSR-Encoder [40] |  ✓ |  ✓ |  ✓ |  ✓ |  ✗
MS-Diffusion |  ✓ |  ✓ |  ✓ |  ✓ |  ✓
Figure 2: Challenges inherent in multi-subject personalization approaches.
Through the explicit layout guidance, MS-Diffusion addresses these challenges
by directing subject conditioning to specific areas, simultaneously
maintaining high image fidelity.

  * •

We introduce a layout-guided, zero-shot multi-subject image personalization
framework within the diffusion model paradigm, designated as ’MS-Diffusion’.
This innovation streamlines the complex process of preserving detailed subject
references. Moreover, it seamlessly integrates multiple subjects into a
coherent and harmonious personalized image.

  * •

The ’Grounding Resampler’ is advanced as a novel feature refinement mechanism.
This construct enriches the detail extraction from images by ascertaining the
correlative content and fusing it with box embeddings that demarcate the
anticipated spatial zones for each subject. Additionally, we introduce a
specialized multi-subject cross-attention mechanism, confronting and
rectifying prevalent complications in multi-subject personalization, including
inadvertent subject neglect, disproportionate subject dominance, and
internecine subject conflicts.

  * •

The capabilities of ’MS-Diffusion’ are empirically substantiated through its
ability to synthesize a broader spectrum of images with notable fidelity. The
paper further delineates comprehensive ablation studies, underpinning the
rationale behind our design decisions and affirming the efficacy of our
proposed approach.

##  2 Related Work

###  2.1 Text-to-image Generation

Text-to-image generative models [27, 29, 3, 2] are capable of producing high-
quality images using user-provided text prompts. In recent times, diffusion-
based models have shown strong performance in text-to-image tasks. Stable
Diffusion [27] proposes conducting the diffusion process in latent space
rather than pixel space, which reduces the sampling steps without compromising
image quality. Kandinsky [26] takes both text embedding and image embedding as
conditions to generate images more controllablely. DALLE-3 [3] recaptions the
training data pairs and utilizes T5-XXL [7] as the text encoder to strengthen
the prompt-following ability. StableCascade [22] presents a cascaded
architecture to leverage outputs of front stages as priors, further reducing
the latent space. PixArt-$\alpha$ [5] also employs a large T5 text encoder and
replaces the original U-Net backbone with a transformer [21]. These models
focus on the basic text-to-image ability and cannot handle the situation when
users provide specific subjects.

To finely control text-to-image generation, some diffusion models [41, 17, 35]
support users in providing layout guidance. Layout Diffusion [41] and GLIGEN
[17] input positions and labels of bounding boxes into the diffusion model and
train it to learn the layout information. DenseDiffusion [12] develops a
training-free method and modulates the attention maps in the inference phase.
Instance Diffusion [34] and MIGC [42] extend the layout-conditioned diffusion
to the instance level, enabling the model to generate multiple objects with
precise quantities. While layout-guided diffusion models have robust
controllability, they cannot reference specific concepts, which is emphasized
in personalized text-to-image generation.

###  2.2 Text-to-image Personalization

Text-to-image personalization [28, 9, 14, 10, 24] has attracted much attention
in the community for its powerful referencing capability of both text and
image prompts. Textual inversion [9] and DreamBooth [28] utilize an identifier
in the text to bind the visual concept through fine-tuning. IP-Adapter [38]
proposes a zero-shot personalized model by projecting the image embedding to
the cross-attention layers. InstanceID [33] develops an approach for identity
personalization, replacing the image encoder with a face encoder and employing
ControlNet [39] to integrate the face landmarks. To narrow the gap between
image and text prompts, Kosmos-G [19] and $\lambda$-ECLIPSE [20] conduct
multi-modal training to unite the inputs by text-image interleaving. SSR-
Encoder [40] design a query network to extract a single subject from images
with multiple subjects for personalization.

Though past research in this field has significantly enhanced the ability to
reference single subjects, few zero-shot multi-subject personalized models.
Moreover, existing related works struggle to address conflicts in personalized
generation with multiple subjects and generate bad results, which is precisely
the focus of our work, to discuss and resolve these issues.

Figure 3: The overall pipeline of MS-Diffusion. It introduces two pivotal
enhancements to the model: the grounding resampler and multi-subject cross-
attention mechanisms. Firstly, the grounding resampler adeptly assimilates
visual information, correlating it with specific entities and spatial
constraints. Subsequently, a targeted cross-attention mechanism facilitates
precise interactions between the image condition and the diffusion latent
within the multi-subject attention layers. Throughout the training phase, all
components of the pre-existing diffusion model remain frozen.

##  3 Method

The pipeline of MS-Diffusion is shown in Figure 3. Through an improved data
construction strategy, we can get multiple subject images together with the
corresponding entities and layouts as input. We propose a grounding resampler
to separately extract the image features and integrate them with phrase
embedding and box embedding for condition enhancement. Inside the cross-
attention layers, we further introduce the masked cross-attention to guide the
generation with layout priors and alleviate conflicts of multiple subjects.
The training needs no pre-trained weights to be optimized and remains plug-
and-play in various base models.

###  3.1 Preliminaries

#### Stable Diffusion with Image Prompt.

As a widely used diffusion model, Stable Diffusion (SD) [27] conducts the
diffusion process in the latent space. Given an image and a text prompt, SD
encodes them into latent code $\mathbf{z}$ and condition embedding
$\mathbf{c_{t}}$ utilizing VAE [32] and CLIP [25] text encoder, respectively.
In zero-shot image personalization architectures like IP-adapter [38], images
can also be considered a condition of the diffusion model. Specifically, a
subject image is encoded to image embeddings by an image encoder and then
projected into the original condition space of the diffusion model denoted as
$\mathbf{c}_{i}$. For a timestep $t$ which is uniformly sampled from a fixed
range, the model $\theta$ predicts the noise $\boldsymbol{\epsilon}_{\theta}$
and is optimized through the objective:

$\mathcal{L}_{IP}=\mathbb{E}_{\mathbf{z},\mathbf{c},\epsilon,t}\left[\left\|{\epsilon-\boldsymbol{\epsilon}}_{\theta}\left(\mathbf{z}_{t}\mid\mathbf{c_{t}},\mathbf{c_{i}},t\right)\right\|_{2}^{2}\right]$
(1)

where $\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})$. In this work, we
employ $\mathcal{L}_{IP}$ with SDXL [23] as the pre-trained model, which
contains two CLIP text encoders and additional condition inputs besides
$\mathbf{c}$ and $t$, omitted for brevity.

#### Cross-attention.

In SD [27], both $\mathbf{c}_{i}$ and $\mathbf{c}_{t}$ are integrated into the
U-Net backbone through cross-attention layers:

$\text{Attn}\left(\mathbf{Q},\mathbf{K}_{i},\mathbf{K}_{t},\mathbf{V}_{i},\mathbf{V}_{t}\right)=\gamma\cdot\mathbf{z}_{img}+\mathbf{z}_{txt}=\gamma\cdot\text{Softmax}\left(\frac{\mathbf{Q}\mathbf{K}_{i}}{\sqrt{d}}\right)\mathbf{V}_{i}+\text{Softmax}\left(\frac{\mathbf{Q}\mathbf{K}_{t}}{\sqrt{d}}\right)\mathbf{V}_{t}$
(2)

where $\mathbf{Q}=\mathbf{z}W_{q}$, $\mathbf{K_{i}}=\mathbf{c}_{i}W_{k}^{i}$,
$\mathbf{K_{t}}=\mathbf{c}_{t}W_{k}^{t}$,
$\mathbf{V_{i}}=\mathbf{c}_{i}W_{v}^{i}$,
$\mathbf{V_{t}}=\mathbf{c}_{t}W_{v}^{t}$, and $W_{q}$, $W_{k}$, $W_{v}$ are
corresponding projection weight matrices. And $d$ represents the
dimensionality of the key vectors Note that the key and value matrix of
$\mathbf{c}_{i}$ and $\mathbf{c}_{t}$ are independent of each other to
decouple conditions of different modalities. Previous studies [11, 31] have
found that attention maps
$\mathbf{A}=\text{Softmax}\left(\mathbf{Q}\mathbf{K}/\sqrt{d}\right)$ can
reflect the attribution relation between generated images and conditions,
which means that they determine the effect of condition controls.

###  3.2 Data Construction

In the field of multi-subject personalization, creating a robust dataset
architecture is a significant challenge, especially when no pre-existing
dataset includes a variety of reference subjects with their validated truths.
Our method starts with applying a Named Entity Recognition (NER) protocol to
textual data to extract relevant entities. These entities are then used within
a detection framework to define the corresponding bounding boxes. This step
generates training samples that encompass a range of [subjects, entities,
spatial layouts].

Previous studies have mostly created training examples from stand-alone images
which is essentially a reconstruction task, leading the resulting models to
favor replication, often resulting in ’copy-and-paste’ artifacts [6]. To
address this issue, our enhanced approach involves extracting subjects from a
single video frame and using another frame from the same sequence as a
reference for the truth. This technique effectively separates the personalized
references from the target images. Due to possible variations in subjects
between frames, we use a specialized subject-matching algorithm to ensure
accurate matching. We provide a detailed description of this data processing
pipeline in Section A.

###  3.3 Grounding Resampler

Different from text embedding, image embedding generally contains more
information and is sparser, making projection into the condition space
challenging. Leveraging embeddings from all image patches primarily control
the condition, but the pooled output from the image encoder tends to omit many
details. Different from [1] and [38], we propose the integration of a
grounding resampler that functions as an alternative form of image projector.
Utilizing a set of learnable tokens, the grounding resampler queries and
distills pertinent information from the image features. Specifically, with an
image embedding $f_{i}$ and a learnable query $f_{q}$, the resampler comprises
several attention layers:

$\text{RSAttn}=\text{Softmax}\left(\frac{\mathbf{Q}\left(f_{q}\right)\mathbf{K}\left(\left[f_{i},f_{q}\right]\right)}{\sqrt{d}}\right)\mathbf{V}\left(\left[f_{i},f_{q}\right]\right)$
(3)

where $[f_{i},f_{q}]$ denotes the concatenation of the image embedding $f_{i}$
and the learnable query $f_{q}$. The architecture incorporates fully connected
feedforward networks (FFNs), analogous to those utilized in standard vision
transformers [8].

As detailed in 3.2, we can obtain entities of multiple referenced subjects and
their target area boxes in the generated image. We present to initialize the
query $f_{q}$ with grounding tokens $f_{g}$ derived from text embedding of
entities and Fourier embedding of boxes. Entities are related to the semantic
information of images and boxes indicate the areas where the subjects are
supposed to be. It would be helpful for the resampler to extract the image
features appropriately and the cross-attention layers to condition the
generation finely. To prevent the model from becoming dependent on the
grounding tokens during inference, we randomly replace them with the original
learnable queries in the training. For the input of n subjects, the projection
processes of different subject images do not affect each other. The resulting
n queries will be concatenated and input into the subsequent model as
$\mathbf{c_{i}}$ with $N=n*n_{t}$ tokens, where $n_{t}$ is the token quantity
per subject.

###  3.4 Multi-subject Cross-attention

In scenarios involving the generation of multiple subjects, challenges
frequently arise that are not exclusive to personalization tasks. These
include discordances between subjects and their backgrounds and amongst the
subjects themselves. A viable solution to mitigate such conflicts leverages
attention masks, contingent upon the availability of layout priors. The
incorporation of attention masks within cross-attention mechanisms facilitates
the exclusion of padding tokens from the condition, thus minimizing their
impact.

To confine the context of each subject to a designated spatial domain, we
propose an enhancement to the conventional attention mask, denoted as
$\mathbf{M}$. This adjustment involves the bilateral neglection of tokens
within both the query and key matrices, applied specifically for the $j$th
subject as follows:

$\mathbf{M}_{j}(x,y)=\begin{cases}0&\text{if }[x,y]\in B_{j}\\\
-\infty&\text{if }[x,y]\notin B_{j}\end{cases}$ (4)

Here, $B_{j}$ signifies the coordinate set of bounding boxes pertaining to the
$j$th subject. By this means, the conditional image latent
$\hat{\mathbf{z}}_{img}$ is derived through:

$\hat{\mathbf{z}}_{img}=\text{Softmax}\left(\frac{\mathbf{Q}\mathbf{K}_{i}}{\sqrt{d}}+\mathbf{M}\right)\mathbf{V}_{i}$
(5)

Herein, $\mathbf{M}$ represents the amalgamation of all subject-specific
masks, $\text{Concat}(\mathbf{M}_{0},\ldots,\mathbf{M}_{n})$.

However, an inherent limitation arises when a query patch token is
ubiquitously masked across all referenced subjects or remains unmasked (in
instances of overlapping bounding boxes), thereby diminishing the intended
efficacy of multi-subject cross-attention. To counteract this, we introduce
dummy tokens initialized at random preceding the image tokens to symbolize the
background. This strategy is instrumental in ensuring that text conditions
predominantly govern areas devoid of any guided layout, thereby considered
irrelevant to the referenced images. Following the acquisition of
$\mathbf{A}$, we apply $\mathbf{M}_{bg}$ to seamlessly mask these tokens
within $\hat{\mathbf{z}}_{img}$, as illustrated:

$\mathbf{z}_{img}=(1-\mathbf{M}_{bg})\cdot\hat{\mathbf{z}}_{img}$ (6)

where $\mathbf{M}_{bg}$ is articulated as a binary mask, with elements within
the subject bounding boxes designated as zero. Notably, certain studies have
sought to resolve these conflicts through the application of objectives on
attention maps within the cross-attention mechanism. A series of rigorous
experiments have been conducted to substantiate our design, with the details
elucidated in Section 4.4.

##  4 Experiments

Figure 4: Qualitative results of MS-Diffusion and baselines in single-subject personalization. Table 2: Quantitative comparison on MS-Diffusion and baselines. Bold and underline represent the highest and second-highest metrics, respectively. For single-subject, the results for IP-Adapter, Emu2, and Kosmos-G were obtained from [20], while the rest are reported in the corresponding papers. * denotes the model is fine-tuned on DreamBench. Method | Single-subject | Multi-subject
---|---|---
CLIP-I |  DINO |  CLIP-T |  CLIP-I |  DINO |  M-DINO |  CLIP-T
Textual Inversion |  0.780 |  0.569 |  0.255 |  - |  - |  - |  -
DreamBooth |  0.803 |  0.668 |  0.305 |  - |  - |  - |  -
BLIP-Diffusion* |  0.805 |  0.670 |  0.302 |  - |  - |  - |  -
$\lambda$-ECLIPSE* |  0.796 |  0.682 |  0.304 |  - |  - |  - |  -
MS-Diffusion* |  0.805 |  0.702 |  0.313 |  - |  - |  - |  -
ELITE |  0.771 |  0.621 |  0.293 |  - |  - |  - |  -
BLIP-Diffusion |  0.779 |  0.594 |  0.300 |  - |  - |  - |  -
IP-Adapter |  0.810 |  0.613 |  0.292 |  - |  - |  - |  -
Emu2 |  0.765 |  0.563 |  0.273 |  - |  - |  - |  -
Kosmos-G |  0.822 |  0.618 |  0.250 |  - |  - |  - |  -
$\lambda$-ECLIPSE |  0.783 |  0.613 |  0.307 |  0.724 |  0.419 |  0.094 |  0.316
SSR-Encoder |  0.821 |  0.612 |  0.308 |  0.725 |  0.425 |  0.107 |  0.303
MS-Diffusion |  0.792 |  0.671 |  0.321 |  0.698 |  0.425 |  0.108 |  0.341
Figure 5: Qualitative results of MS-Diffusion and baselines in multi-subject
personalization.

###  4.1 Experiment Setup

#### Datasets.

For training, we utilize an in-house video dataset that contains 3.6M video
clips. For evaluation, we measure the single-subject and multi-subject
performance on DreamBench [28] and MS-Bench, respectively. DreamBench includes
30 subjects and 25 prompts and we preset all input boxes to [0.25, 0.25, 0.75,
0.75]. To thoroughly assess the performance of multi-subject personalization,
we have established a new evaluation standard, MS-Bench, which includes 40
subjects and 1148 combinations, each combination having up to 6 prompts,
totaling 4488 distinct test samples. Details of datasets are provided in
Section A and Section B.

#### Evaluation metrics.

Following previous works, we measure the performance through image and text
fidelity. To assess image fidelity, we employ cosine similarity measures
between generated images and subject images within CLIP [25] and DINO [4]
spaces, referred to as CLIP-I and DINO, respectively. For text fidelity, we
utilize cosine similarity between generated images and text prompts in CLIP
space, denoted as CLIP-T. In multi-subject personalization, using the average
fidelity to reflect image fidelity is insufficient, as it fails to reveal
cases of subject neglect. We further employ the product of multi-subject DINO,
denoted as M-DINO, to indicate whether each subject has been recreated in the
results.

#### Baselines.

For single-subject personalization, we compare our model with methods
mentioned in Table 1. Emu2 [30], Kosmos-G [19], and $\lambda$-ECLIPSE [20] are
all MLLM-based methods, while $\lambda$-ECLIPSE is reported to have better
performance. Therefore, we select SSR-Encoder [40] and $\lambda$-ECLIPSE [20]
as baselines for multi-subject personalization. The implementation details of
MS-Diffusion and these methods are contained in Section C.

###  4.2 Single-subject Comparison

In the single-subject comparison, a detailed examination is carried out
utilizing both qualitative and quantitative comparisons to gauge the
performance of different methodologies. On the qualitative front, as shown in
Figure 4, MS-Diffusion shows an exceptional ability to generate single-subject
images with high fidelity and detail retention. In quantitative results
provided in Table 2, MS-Diffusion also achieves competitive scores, with
obviously the highest DINO and CLIP-T scores at 0.671 and 0.321 respectively
on zero-shot scenarios, and leading CLIP-I score of 0.792. For tuning methods,
MS-Diffusion outperforms baselines in all metrics. As discussed in [28], DINO
more accurately captures the similarity in details between the results and the
labels, whereas CLIP-I may exhibit high scores in situations of background
overfitting, resulting in a clear advantage for DINO, but a slight
disadvantage for CLIP-I of MS-Diffusion.

###  4.3 Multi-subject Comparison

From a qualitative perspective in Figure 5, MS-Diffusion manages to maintain
natural interactions among subjects in generated images while ensuring each
subject retains its distinctiveness and recognizability. Quantitatively,
results in Table 2 demonstrate the strength of MS-Diffusion in DINO, M-DINO,
and CLIP-T. Unlike in single-subject personalization, there is a larger gap in
text fidelity between MS-Diffusion and the baselines in multi-subject
personalization, demonstrating that MS-Diffusion not only effectively
generates the multiple subjects outlined in the text but also excellently
preserves the text control capabilities inherent to SD. Additionally, the
image fidelity of MS-Diffusion is comparable, highlighting its superior
ability to retain details, particularly significant as low text fidelity is
commonly associated with overfitting.

Table 3: Ablation study of MS-Diffusion. RS, GRS, MCA, TAL, IAL, and LG represent resampler, grounding resampler, multi-subject cross-attention, text attention loss, image attention loss, and layout guidance, respectively. Method | Single-subject | Multi-subject
---|---|---
CLIP-I |  DINO |  CLIP-T |  CLIP-I |  DINO |  M-DINO |  CLIP-T
MS-Diffusion |  0.792 |  0.671 |  0.321 |  0.698 |  0.425 |  0.108 |  0.341
w/o RS |  0.775 |  0.583 |  0.320 |  0.680 |  0.372 |  0.082 |  0.336
w/o GRS |  0.777 |  0.646 |  0.320 |  0.681 |  0.389 |  0.090 |  0.331
w/o MCA |  0.798 |  0.662 |  0.312 |  0.693 |  0.422 |  0.100 |  0.309
w/o LG w/ IAL |  0.761 |  0.577 |  0.284 |  0.675 |  0.377 |  0.080 |  0.305
w/o LG w/ IAL&TAL |  0.809 |  0.660 |  0.293 |  0.687 |  0.413 |  0.093 |  0.316

###  4.4 Ablation Study

#### Module ablation.

We conduct an ablation experiment on the proposed two modules, grounding
resampler (GRS) and multi-subject cross-attention (MCA), to validate their
effects. For GRS, we replace it with a linear projection layer and a normal
resampler [1, 38]. Results in Table 3 indicate that the resampler-like image
projector significantly enhances the details, as evidenced by DINO being
obviouly higher than the linear projector. Moreover, The substantial
improvement in multi-object image fidelity by GRS reflects the critical role
of the information carried by grounding tokens in multi-object generation. As
a key module for resolving conflicts, removing MCA results in a noticeable
degradation of text fidelity, especially in multi-subject generation. The
combined use of both modules ensures that MS-Diffusion maintains high image
and text fidelity simultaneously.

#### Layout guidance.

As mentioned in Section 3.4, we have explored the indispensable role of
explicit layout guidance (LG), including grounding tokens and MCA. A
straightforward approach to implicitly utilizing layout involves incorporating
an attention loss during training. Besides the image attention loss (IAL), we
also introduce text attention loss (TAL) to training by setting the original
cross-attention layers trainable. The detailed loss definition is provided in
Section F. As illustrated in Table 3, an objective to guide the image cross-
attention helps the personalization hardly at all. TAL has somewhat resolved
the conflict issues, but its performance is inferior to MS-Diffusion while
introducing additional training parameters. We consider the inclusion of LG
necessary and rational, not merely for the performance enhancements it offers,
but also because it effectively resolves the various multi-object generation
issues highlighted in Figure 2.

##  5 Conclusion

This study makes a significant contribution to the field of P-T2I diffusion
models with the development of MS-Diffusion. This zero-shot framework excels
at capturing intricate subject details and smoothly blending multiple subjects
into a single coherent image. Equipped with the innovative Grounding Resampler
and Multi-subject Cross-attention mechanisms, our model effectively overcomes
common multi-subject personalization issues, such as subject neglect and
conflict. Extensive ablation studies underscore MS-Diffusion’s enhanced
performance in image synthesis fidelity compared to existing models. It stands
as a groundbreaking approach for P-T2I applications that are free from the
need for fine-tuning and require layout guidance.

## References

  * [1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan.  Flamingo: a visual language model for few-shot learning.  In NeurIPS, 2022.
  * [2] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu.  All are worth words: A vit backbone for diffusion models.  In CVPR, pages 22669–22679. IEEE, 2023.
  * [3] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al.  Improving image generation with better captions.  Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2:3, 2023.
  * [4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In ICCV, 2021.
  * [5] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James T. Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li.  Pixart-$\alpha$: Fast training of diffusion transformer for photorealistic text-to-image synthesis.  CoRR, abs/2310.00426, 2023.
  * [6] Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao.  Anydoor: Zero-shot object-level image customization.  CoRR, abs/2307.09481, 2023.
  * [7] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  arXiv preprint arXiv:2210.11416, 2022.
  * [8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR. OpenReview.net, 2021.
  * [9] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  In ICLR. OpenReview.net, 2023.
  * [10] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris N. Metaxas, and Feng Yang.  Svdiff: Compact parameter space for diffusion fine-tuning.  In ICCV, pages 7289–7300. IEEE, 2023.
  * [11] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross-attention control.  In ICLR. OpenReview.net, 2023.
  * [12] Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Ha, and Jun-Yan Zhu.  Dense text-to-image generation with attention modulation.  In ICCV, 2023.
  * [13] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick.  Segment anything.  In ICCV, pages 3992–4003. IEEE, 2023.
  * [14] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  In CVPR, pages 1931–1941. IEEE, 2023.
  * [15] Dongxu Li, Junnan Li, and Steven C. H. Hoi.  Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing.  In NeurIPS, 2023.
  * [16] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi.  BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models.  In ICML, volume 202 of Proceedings of Machine Learning Research, pages 19730–19742. PMLR, 2023.
  * [17] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee.  Gligen: Open-set grounded text-to-image generation.  CVPR, 2023.
  * [18] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  arXiv preprint arXiv:2303.05499, 2023.
  * [19] Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei.  Kosmos-g: Generating images in context with multimodal large language models.  CoRR, abs/2310.02992, 2023.
  * [20] Maitreya Patel, Sangmin Jung, Chitta Baral, and Yezhou Yang.  $\lambda$-eclipse: Multi-concept personalized text-to-image diffusion models by leveraging CLIP latent space.  CoRR, abs/2402.05195, 2024.
  * [21] William Peebles and Saining Xie.  Scalable diffusion models with transformers.  In ICCV, pages 4172–4182. IEEE, 2023.
  * [22] Pablo Pernias, Dominic Rampas, Mats Leon Richter, Christopher Pal, and Marc Aubreville.  Würstchen: An efficient architecture for large-scale text-to-image diffusion models.  In The Twelfth International Conference on Learning Representations, 2023.
  * [23] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  SDXL: improving latent diffusion models for high-resolution image synthesis.  CoRR, abs/2307.01952, 2023.
  * [24] Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf.  Controlling text-to-image diffusion by orthogonal finetuning.  In NeurIPS, 2023.
  * [25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In ICML, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR, 2021.
  * [26] Anton Razzhigaev, Arseniy Shakhmatov, Anastasia Maltseva, Vladimir Arkhipkin, Igor Pavlov, Ilya Ryabov, Angelina Kuts, Alexander Panchenko, Andrey Kuznetsov, and Denis Dimitrov.  Kandinsky: An improved text-to-image synthesis with image prior and latent diffusion.  In EMNLP, pages 286–295. Association for Computational Linguistics, 2023.
  * [27] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, pages 10674–10685. IEEE, 2022.
  * [28] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In CVPR, pages 22500–22510. IEEE, 2023.
  * [29] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.  In NeurIPS, 2022.
  * [30] Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang.  Generative multimodal models are in-context learners.  CoRR, abs/2312.13286, 2023.
  * [31] Raphael Tang, Linqing Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Pontus Stenetorp, Jimmy Lin, and Ferhan Ture.  What the DAAM: interpreting stable diffusion using cross attention.  In ACL, pages 5644–5659. Association for Computational Linguistics, 2023.
  * [32] Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu.  Neural discrete representation learning.  In NeurIPS, pages 6306–6315, 2017.
  * [33] Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen.  Instantid: Zero-shot identity-preserving generation in seconds.  arXiv preprint arXiv:2401.07519, 2024.
  * [34] Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra.  Instancediffusion: Instance-level control for image generation.  CoRR, abs/2402.03290, 2024.
  * [35] Zirui Wang, Zhizhou Sha, Zheng Ding, Yilin Wang, and Zhuowen Tu.  Tokencompose: Grounding diffusion with token-level supervision.  CoRR, abs/2312.03626, 2023.
  * [36] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo.  ELITE: encoding visual concepts into textual embeddings for customized text-to-image generation.  In ICCV, pages 15897–15907. IEEE, 2023.
  * [37] Zhichao Wei, Qingkun Su, Long Qin, and Weizhi Wang.  Mm-diff: High-fidelity image personalization via multi-modal condition integration.  CoRR, abs/2403.15059, 2024.
  * [38] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang.  Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models.  CoRR, abs/2308.06721, 2023.
  * [39] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 3813–3824. IEEE, 2023.
  * [40] Yuxuan Zhang, Jiaming Liu, Yiren Song, Rui Wang, Hao Tang, Jinpeng Yu, Huaxia Li, Xu Tang, Yao Hu, Han Pan, and Zhongliang Jing.  Ssr-encoder: Encoding selective subject representation for subject-driven generation.  CoRR, abs/2312.16272, 2023.
  * [41] Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li.  Layoutdiffusion: Controllable diffusion model for layout-to-image generation.  In CVPR, pages 22490–22499. IEEE, 2023.
  * [42] Dewei Zhou, You Li, Fan Ma, Zongting Zhang, and Yi Yang.  MIGC: multi-instance generation controller for text-to-image synthesis.  CoRR, abs/2402.05408, 2024.

##  Appendix A Training Dataset Construction Pipeline

Figure 6 illustrates the construction pipeline of the training dataset.
Firstly, we randomly select two frames from a video clip, one as the reference
and the other as the ground truth. Both frames are captioned by BLIP-2 [16].
Secondly, we utilize a NER model111https://spacy.io/ to extract entities from
the caption. Entities and images are then input into Grounding DINO [18] to
obtain the boxes, which are parts of the final input of the model. Taking the
boxes as prompts of SAM [13], we can further obtain segmentation masks to
extract subjects from the reference image. Since the entities in different
frames can be different, we design a subject matcher, which finds the
correspondence between the frames by conducting Hungarian Algorithm on the
entity image embeddings. Frames in a video typically contain the same entities
but exhibit clear differences in details such as angles and poses. This makes
them highly suitable as training data for personalized image generation
models, which can help mitigate the model’s tendency to copy-and-paste.

Our dataset comprises 2.8M general scenario videos and 0.8M product
demonstration videos, where the former covers more scenarios and the latter
has more clear subjects. 2-5 frames for each are adopted in the training. In
practice, there may only be 1-2 subjects successfully matched. To ensure that
the training data contains a sufficient number of reference subjects, for the
targets where matching fails, we directly use the corresponding parts of the
ground truth as references. Subjects that are too small, too large, or have
imbalanced proportions are filtered out. Each training sample can have up to 4
subjects, and we pad in the ones with fewer than 4.

Figure 6: Data construction pipeline of our work. For the input of two frames,
we can get subject images, entities, and boxes. Note that the entities and
boxes are from the ground truth frame since they indicate the information in
the generated result.

##  Appendix B Details of MS-Bench

Table 4: Explanation of MS-Bench. Each combination type has preset prompts and boxes. [S] represents prompt variations about the scene, including "in a room", "in the jungle", "in the snow", "on the beach", "on the grass", and "on a cobblestone street". Type |  Prompt |  Boxes
---|---|---
living+living |  a {0} and a {1} [S] |  [0.00, 0.25, 0.50, 0.75] [0.50, 0.25, 1.00, 0.75]
living+object |  |
object+object |  |
living+upwearing |  a {0} wearing a {1} [S] |  [0.25, 0.25, 0.75, 0.75] [0.25, 0.00, 0.75, 0.25]
living+midwearing |  a {0} wearing a {1} [S] |  [0.25, 0.25, 0.75, 0.75] [0.25, 0.25, 0.75, 0.75]
living+wholewearing |  |
midwearing+downwearing |  a woman wearing a {0} and a {1} [S] |  [0.25, 0.25, 0.75, 0.60] [0.25, 0.60, 0.75, 1.00]
living+scene |  a {0} with a {1} in the background |  [0.25, 0.25, 0.75, 0.75] [0.00, 0.00, 1.00, 1.00]
object+scene |  |
living+living+living object+object+object |  a {0}, a {1}, and a {2} [S] |  [0.00, 0.25, 0.35, 0.75] [0.35, 0.25, 0.65, 0.75] [0.65, 0.25, 1.00, 0.75]
|
|
living+object+scene |  a {0} and a {1} with a {2} in the background |  [0.00, 0.25, 0.50, 0.75] [0.50, 0.25, 1.00, 0.75] [0.00, 0.00, 1.00, 1.00]
|
|
upwearing+midwearing+ downwearing |  a woman wearing a {0}, a {1}, and a {2} [S] |  [0.25, 0.00, 0.75, 0.25] [0.25, 0.25, 0.75, 0.60] [0.25, 0.60, 0.75, 1.00]
|
|

To construct MS-Bench, we collect subjects from [28, 9, 14], the
Internet222https://unsplash.com/, and an internal dataset that does not
overlap with the training set. MS-Bench contains four data types and 13
combination types with two or three subjects. We provide the details in Table
4. Each combination type other than those related to the scene has 6 prompt
variations. There are 1148 combinations and 4488 evaluation samples, where
entities and boxes are subject categories and preset layouts. Compared to
other multi-subject benchmarks, our MS-Bench ensures that the model
performance can be reflected comprehensively in abundant cases.

##  Appendix C Experiment Settings

#### Training and Inference.

The pre-trained model employed in MS-Diffusion is Stable Diffusion XL (SDXL)
[23]. Implemented by Pytorch 2.0.1 and Diffusers 0.23.1, our model is trained
on 16 A100 GPUs for 120k steps with a batch size of 8 and a learning rate of
1e-4. Following the training of [38], we set $\gamma=1.0$ in cross-attention
layers and dropped the text and image condition using the same probability. To
ensure the model is not dependent on the grounding tokens (Section 3.3), we
also randomly drop them with a probability of 0.1. We generate five images for
each sample during the inference, with unconditional guidance scale and
$\gamma$ set to 7.5 and 0.6, respectively, to get better results.

#### Comparative methods.

Here we provide the details of the baselines compared in qualitative and
quantitative experiments:

  * •

BLIP-Diffusion [15] utilizes BLIP-2 [16] to unite the text and image
embeddings. We implement it in the qualitative comparison using Diffusers.

  * •

IP-Adapter [38] also uses image prompt as the condition. We run qualitative
samples on their official code with the scale set to 0.5 recommended by the
paper. Considering fairness, we use the result of SDXL in Table 2.

  * •

SSR-Encoder [40] design a query network to extract the specified subject of a
single image, which enables it to finish multi-subject generation. We leverage
it as one of the baselines in multi-subject personalization. For performance
comparison, we employed the official code provided, alongside the default
hyperparameters specified in the code repository.

  * •

$\lambda$-ECLIPSE [20] trains an independent multi-modal encoder and employs
Kandinsky as the generative backbone. Since it outperforms other MLLM-based
approaches, we choose it to be the representative. To facilitate a comparative
analysis of performance, the study utilized the officially provided code, in
conjunction with the default hyperparameters delineated within the
corresponding code repository.

##  Appendix D More Results of Single-subject Personalization

Additional qualitative results on DreamBench are provided in Figure
LABEL:fig:single_app. MS-Diffusion shows excellent text fidelity in all
subjects while keeping subject details, especially the living ones (dogs). It
can be noticed that some elements in the background (the third line and the
fourth line) also occur in the results (the grass and the teapot holder) since
the entire images are referenced during the generation. Their scope of action
depends on the input bounding box. In practical applications, using masked
images as a condition is recommended.

##  Appendix E More Results of Multi-subject Personalization

We provide additional multi-subject personalized images based on MS-Bench in
Figure LABEL:fig:multi_app. The results encompass various combination types,
fully demonstrating the generalizability and robustness of MS-Diffusion. When
the scene changes freely according to the text, the details of the subject are
preserved without being affected. In addition to common parallel combinations,
MS-Diffusion also performs well in personalized generation for combinations
with certain overlapping areas, such as "living+midwearing" and
"object+scene".

##  Appendix F Layout Guidance

Cross-attention maps can intuitively reflect the condition-image attribution
relation [31, 11]. Recent works [35, 37] have studied utilizing an objective
on the cross-attention maps in multi-subject generation. The objective exists
only in training, considered implicit and insufficient to handle multi-subject
conflicts. We have provided the performance comparison between our explicit
layout guidance and attention loss in Section 4.4. For a single cross-
attention layer, the attention loss of the $j$th subject is calculated by:

$\mathcal{L}_{am}^{j}=\left(1-\frac{\sum_{\left[x,y\right]\in
B_{j}}\mathbf{A}_{\left[x,y\right],j}}{\sum_{\left[x,y\right]}\mathbf{A}_{\left[x,y\right],j}}\right)^{2}$
(7)

where $\left[x,y\right]$ corresponds to a latent token in $\mathbf{Q}$ and
$B_{j}$ is the bounding box of the $j$th subject. This objective aims to
promote the activation of attention maps within specific boxes. We average
$\mathcal{L}_{am}^{j}$ across layers and subjects and set its weight to 0.01
in the final loss. To validate the text attention loss, we also optimize the
text cross-attention layers in training, increasing approximately 70% in
learnable parameters.

Although our model provides explicit layout guidance, it still significantly
differs from layout-based diffusion. Firstly, the information of boxes in the
grounding resampler is prior, and its conditional effect is relatively weak.
We have also reduced the model’s reliance on this input by randomly dropping
grounding tokens. Secondly, the multi-subject cross-attention only exists in
image cross-attention, inherently controlling the action of image conditions
in specified areas, but cannot determine the whole generation of the diffusion
model. Our goal is not to develop a method that fully supports layout control
but to utilize layout information to guide the model in resolving conflicts in
multi-subject generation.

To further explore the layout control capability of MS-Diffusion, we provide
qualitative results in Figure LABEL:fig:layout. It can be demonstrated that
MS-Diffusion can generate images that adhere to layout conditions, even in the
case of two instances of the same category. However, the generated positions
are not entirely accurate, especially in "a cat and a cat on the grass",
illustrating that the layout condition is relatively weak compared to text and
image prompts in the personalization task.

##  Appendix G Integration with ControlNet

In the realm of text-to-image diffusion models, a notable application is the
enhancement of structural control within image generation. Our MS-Diffusoin
maintains the original network architecture unchanged, thus ensuring full
compatibility with existing controllable tools. Consequently, this allows for
the generation of images that are not only prompted by images but also
governed by additional conditions. By integrating our MS-Diffusion with
established controllable mechanisms such as ControlNet, we demonstrate the
capacity to produce images under varied structural directives. Figure
LABEL:fig:control displays an array of images synthesized with image prompts
coupled with distinct structural controls(depth, canny edge, openpose). This
seamless cooperation between our method and these tools facilitates the
creation of highly controllable images without necessitating fine-tuning.

##  Appendix H Text-Image Attribution Analysis

In MS-Diffusion, our focus is primarily on resolving conflicts between
subjects without altering the control mechanism of the text. While some
approaches [35, 34, 42, 12] in non-personalized text-to-image tasks address
multi-object generation conflicts through text cross-attention, this
inevitably requires tuning the diffusion model’s parameters, thereby affecting
the plug-and-play nature, which is not preferred by us. As demonstrated in the
text-image attribution analysis presented in Figure 7, the control of multiple
objects by text in our model is also quite evident. This may be related to the
explicit layout guidance for subjects, since the images and text condition
jointly in the generation process. We also attempted to control text cross-
attention using the same mechanism as in Section 3.4, but no differences were
observed in the results.

Figure 7: Text-image attribution analysis of MS-Diffusion. We average the
attention maps corresponding to the subjects and translate them to normalized
heat maps.

##  Appendix I Subject Interpolation

The blending of two distinct subject representations to yield composite
subjects with hybrid characteristics is feasible through the navigation of the
embedding space linking the subjects. As depicted in Figure
LABEL:fig:interpolation, linear interpolations are conducted among dog and hat
representations, subsequently rendering the interpolated subject in an
unaccustomed context. The visualization reveals a natural gradation of subject
appearance along the interpolated trajectory that harmonizes with the
surrounding environment. This technique proves beneficial when applied in
subject fusion and style transfer.

##  Appendix J Limitations

There are certain limitations in MS-Diffusion. The box-based indication of
positions lacks precision, making it challenging to work effectively when the
interaction between subjects is stronger. Moreover, the model requires
explicit layout input during inference, and generating complex scenes becomes
difficult. Though MS-Diffusion beats SOTA personalized diffusion methods in
both single-subject and multi-subject generation, it still suffers from the
influence of background in subject images. In the future, we will tackle the
above limitations and extend the model to more applications.

##  Appendix K Societal Impact

As an image personalization method, MS-Diffusion aims to customize images
based on user-provided subjects without fine-tuning. Additionally, the multi-
subject reference capability of MS-Diffusion allows users to freely combine
and re-create different concepts. However, MS-Diffusion can also be used to
generate deceptive images, especially those involving subject combinations
that would not exist in reality, an issue that remains to be addressed in the
future.