# The science of ultra-high energy cosmic rays after more than 15 years of
operation of the Pierre Auger Observatory

Olivier Deligny*, for the Pierre Auger Collaboration** * Laboratoire de
Physique des 2 Infinis Irène Joliot-Curie (IJCLab)
CNRS/IN2P3, Université Paris-Saclay, Orsay, France
** Full author list: https://www.auger.org/archive/authors_2022_09.html *
<EMAIL_ADDRESS>**<EMAIL_ADDRESS>
###### Abstract

The Pierre Auger Observatory has been detecting ultra-high energy cosmic rays
(UHECRs) for more than fifteen years. An essential feature of the Observatory
is its hybrid design: cosmic rays above $100~{}$PeV are detected through the
observation of the associated air showers with different and complementary
techniques, from surface detector arrays and fluorescence telescopes to radio
antennas. The analyses of the multi-detector data have enabled high-statistics
and high-precision studies of the energy spectrum, mass composition and
distribution of arrival directions of UHECRs. The resulting picture is
summarized in this contribution. While no discrete source of UHECRs has been
identified so far, the extragalactic origin of the particles has been recently
determined from the arrival directions above 8 EeV, and the ring is closing
around nearby astrophysical sites at higher energies. Also, the established
upper limits on fluxes of UHE neutrinos and photons have implications on dark
matter and cosmological aspects that are also presented in this contribution.

  * September 2022

The Pierre Auger Observatory. Ultra-high energy cosmic rays (UHECRs) are the
most energetic particles produced in nature, with energies in excess of
$100~{}$EeV. Uncovering their origin is a persistent task since their
discovery, mostly because of the very small value of their intensity on Earth
and because of the magnetic deflections they experience en route to Earth. The
Pierre Auger Observatory, located in the province of Mendoza (Argentina) and
covering 3000 km2, is the present flagship experiment studying UHECRs [1]. Two
techniques of detection are combined to measure the extensive air showers
(EAS). A surface detector (SD) array, consisting of 1660 autonomously operated
water-Cherenkov detectors, provides a lateral sampling of the EAS at the
ground level. The detectors are arranged on a triangular grid of 1500 m
spacing (SD-1500 m), except for a denser infill area of $\simeq 30~{}$km2,
where the spacing is 750 m (SD-750 m). The atmosphere above the SD array is
overlooked by fluorescence detectors (FD), which consists of telescopes that
detect the faint UV light emitted by nitrogen molecules previously excited by
the charged particles from the EAS. This technique provides the opportunity of
performing shower calorimetry by mapping the ionization content along the
shower tracks, and of measuring the primary energy on a nearly model-
independent basis. The FD can only operate during dark, moonless nights with a
field of view free of clouds. Online and long-term performances of the
detectors and data quality are monitored continuously, and a set of high-
quality devices installed in the Observatory array monitor the atmospheric
conditions during operation.

The SD operates with a quasi-permanent duty cycle and thus provides a harvest
of data. Yet, assessing the energy of the observed events requires assumptions
about the primary mass and the hadronic processes that control the cascade
development. This proves to be a difficult task as the primary mass on an
event-by-event basis is unknown and the centre-of-mass energy reached at the
LHC corresponds only to that of a proton of $\simeq 100~{}$PeV colliding with
a nitrogen nucleus. To circumvent these limitations, the energies are obtained
by making use of a subset of events detected simultaneously by the FD and the
SD. This “hybrid” approach allows a calorimetric estimate of the energy for
events recorded during periods when the FD cannot be operated.

Figure 1: Flux map at energies above 40 EeV with a top-hat smoothing radius of
25∘ in Galactic coordinates. The supergalactic plane is shown as a gray line.
The blank area is outside the field of view of the Observatory. From [2].

Arrival directions. The identification of UHECR sources relies primarily on
capturing in the arrival directions a pattern suggestive in an evident way of
a class of astrophysical objects. Such a capture is still eluding our grasp,
but some recent observations have confirmed the long-lasting broad statement
of the extragalactic origin of UHECRs above the so-called ankle energy. On the
one hand, an anisotropy at large scales has been revealed above
${\simeq}\>8\>$EeV [3], the amplitude and the direction of which are
consistent with expectations drawn from sources distributed in a similar
manner to the extragalactic matter [3, 4]. On the other hand, at higher
energies, the energy losses of UHECRs limit the horizon of the highest-energy
particles. For small-enough magnetic deflections, the distribution of the
arrival directions of UHECRs above $\simeq 40~{}$EeV could mirror the
inhomogeneous distribution of the nearby extragalactic matter. Data have been
subjected to comprehensive anisotropy searches for different energy thresholds
above $32~{}$EeV, and within different angular windows, between $1^{\circ}$
and $30^{\circ}$ [2]. Searches for significant excesses anywhere in the sky
have been performed, as well as searches for correlations with known
astrophysical structures and with objects that are considered plausible
candidates for UHECR sources. Out of all the searches performed, above
$\simeq~{}38~{}$EeV, the flux patterns of massive, star-forming of active
galaxies within 200 Mpc provide a 4$\sigma$ evidence for anisotropy [2].
Overall, the ring is closing around nearby astrophysical sites to explain the
main features of the flux sky map shown in Galactic coordinates in Fig. 1,
obtained by filtering the data at $25^{\circ}$ above 40 EeV [2].

Energy spectrum and mass composition. The energy spectrum and chemical
composition of UHECRs observed on Earth result from the emission processes at
play, which encompass acceleration mechanisms, losses and escape from the
source environments as well as propagation effects. Differently from, and
complementary to anisotropies, these two observables provide constraints
helping infer the properties of the acceleration processes, the energetics of
the sources, and the abundances of elements in the source environments.

Figure 2: Top: Energy spectrum after combining the individual measurements by
the SD-750 and the SD-1500 scaled by $E^{2.6}$ – from [5]. Bottom: Composition
fractions observed on Earth derived from fitting templates of four mass groups
to the $X_{\mathrm{max}}$ distributions – adapted from [6].

The measurement of the energy spectrum above $100~{}$PeV is emblematic of the
power of using multiple detectors: it has been performed by combining the
independent spectra from the two different SD arrays. The different data
streams are complementary: data from the SD-750 m array allow for the
determination of the energy spectrum down to $100~{}$PeV [5]; data with events
from the SD-1500 m array are crucial above the energy of full efficiency of
$3~{}$EeV up to the highest energies [7]. The two spectra, in agreement within
uncertainties, are combined into a unique one shown in figure 2, top panel,
taking into account the systematics of the individual measurements. Beyond the
ankle (the hardening at $\simeq 5~{}$EeV) and suppression (steepening at
$\simeq 50~{}$EeV) spectral features already established previously, the
second-knee feature is observed to extend over a wide energy range (not fully
covered in this measurement) while a steepening at $\simeq 10~{}$EeV, dubbed
as the instep feature, has been uncovered thanks to the exposure and energy
resolution reached at the Observatory.

The measurement of the depth of the shower maximum, $X_{\mathrm{max}}$, is the
most robust mass-sensitive EAS observable. While $X_{\mathrm{max}}$ data cover
the region of the ankle well, they do not extend, currently, into the region
of flux suppression due to the intrinsically limited duty cycle of the FD.
Between $\simeq 150~{}$PeV and $2~{}$EeV, $\langle X_{\mathrm{max}}\rangle$
increases by around 77 g cm-2 per decade of energy [8]. This is larger than
that expected for a constant mass composition (60 g cm-2 per decade) and thus
indicates that the mean primary mass is becoming lighter over this energy
range. On the other hand, above $2\times 10^{18}~{}$eV, the rate of change of
$\langle X_{\mathrm{max}}\rangle$ becomes significantly smaller (26 g cm-2 per
decade [8]). The trend is thus inverted and the composition gets heavier. The
fluctuations of $X_{\mathrm{max}}$ start to decrease above the same energy,
$2\times 10^{18}~{}$eV, being rather constant below [8]. Composition fractions
observed on Earth can be derived from fitting templates of four mass groups to
the $X_{\mathrm{max}}$ distributions. The results are shown in Fig. 2 –
adapted from [6] – as a function of energy. Error bars denote statistical
uncertainties and lines were added to guide the eye. The two interpretations
of the data with EPOS-LHC [9] and Sibyll2.3 [10] are shown as closed and open
symbols with solid and dashed lines styles respectively. Overall, heavier mass
groups are observed to take over one after the other so that the all-particle
flux gets dominated by one specific mass group in every energy range beyond
the ankle energy. As of today, no composition fractions are available around
and above $100~{}$EeV.

Figure 3: Energy density obtained with the best fit parameters of the
benchmark scenario used for illustration, as described in the text. The dashed
curve shows the energy range that is not used in the fit and where an
additional component is needed for describing the spectrum. From [11].

These results fit a scenario in which several nuclear components contribute to
the total intensity and in which electromagnetic fields permeate source
environments where nuclei are accelerated to a maximum energy proportional to
their charge ($Z$). Without distraction by the many details a full model
scenario would require, we illustrate this scenario in Fig. 3 by considering
several nuclear components injected at the sources with a power-law spectrum
and with the maximal energy of the sources modeled with an exponential cut-off
[12, 11]. For simplicity, the sources are assumed to be stationary and uniform
in a co-moving volume. The best reproduction of the data is shown by
simultaneously fitting the energy spectrum above $5~{}$EeV and the
distribution of $X_{\rm max}$ (using EPOS LHC as a model of hadronic
interactions in their interpretation). The abundance of nuclear elements at
the sources is dominated by intermediate-mass nuclei accelerated to $\approx
5~{}Z{\times}~{}$EeV and escaping from the source environments with a very
hard spectral index, which is in turn determined by the quasi-monoelemental
increase of the average mass with energy. In this scenario, the steepening
observed above $\approx 50$ EeV results from the combination of the maximum
energy of acceleration of the heaviest nuclei at the sources and the GZK
effect [13, 14]. The steepening at $\approx 10$ EeV reflects the interplay
between the flux contributions of the helium and carbon-nitrogen-oxygen
components injected at the source with their distinct cut-off energies, shaped
by photo-disintegration during the propagation.

Beyond the UHECR science. The reach of the data of the Observatory goes beyond
the study of UHECRs: measurement of p-air and p-p cross section [15], study of
muon number in EAS [16, 17, 18, 19], study of hadronic interactions [20],
searches for multi-messengers of high-energy phenomena in the universe [21,
22, 23], study of electric phenomena in the atmosphere [24, 25, 26], etc.
Among other results, the upper limits obtained on photon [27, 28, 29] and
neutrino [30] fluxes in the EeV range and above are of special interest. As an
example of the power of the limits on photon fluxes, we highlight the
constraints that can be inferred on the reduced coupling constant of gauge
interactions in a dark sector of super-heavy particles. Full details can be
found in [31, 29]. Compelling evidence for the observation of the decay of
such dark-matter particles would be the detection of a flux of astrophysical
photons with energies in excess of ${\simeq}10^{8}~{}\mathrm{GeV}$, in
particular from regions of denser DM density such as the center of our Galaxy.

Stability of super-heavy particles is calling for a new quantum number
conserved in the dark sector so as to protect the particles from decaying.
Nevertheless, even stable particles in the perturbative domain will in general
eventually decay due to non-perturbative effects (instantons) in non-abelian
gauge theories. Instanton-induced decay can thus make observable a dark sector
that would otherwise be totally hidden by the conservation of a quantum number
[32]. Assuming quarks and leptons carry this quantum number and so contribute
to anomaly relationships with contributions from the dark sector, they will be
secondary products in the decays of SHDM together with the lightest hidden
fermion. The lifetime of the decaying particle, $\tau_{X}\simeq
M_{X}^{-1}\exp{\left(4\pi/\alpha_{X}\right)}$, is governed by $\alpha_{X}$,
the reduced coupling constant of the hidden gauge interaction [33]. As photons
would result in the secondaries of the decay process, the limits on their flux
can be translated into a bound on the reduced coupling constant:
$\alpha_{X}<0.09$, for $10^{9}<M_{X}/{\rm GeV}<10^{19}$. This upper limit on
$\alpha_{X}$ is the best-ever obtained limit on a proxy of instanton strength.
Implications of this result in terms of dark-matter production in the early
universe are discussed elsewhere [29].

## References

  * [1] Aab A et al. (Pierre Auger Collaboration) 2015 Nucl. Instrum. Meth. A 798 172–213 (Preprint 1502.01323)
  * [2] Abreu P et al. (Pierre Auger Collaboration) 2022 Astrophys. J. 935 170 (Preprint 2206.13492)
  * [3] Aab A et al. (Pierre Auger Collaboration) 2017 Science 357 1266–1270 (Preprint 1709.07321)
  * [4] Aab A et al. (Pierre Auger Collaboration) 2018 Astrophys. J. 868 4 (Preprint 1808.03579)
  * [5] Abreu P et al. (Pierre Auger Collaboration) 2021 Eur. Phys. J. C 81 966 (Preprint 2109.13400)
  * [6] Bellido J and Pierre Auger Collaboration (Pierre Auger Collaboration) 2018 PoS ICRC2017 506
  * [7] Aab A et al. (Pierre Auger Collaboration) 2020 Phys. Rev. D 102 062005 (Preprint 2008.06486)
  * [8] 2019 The Pierre Auger Observatory: Contributions to the 36th International Cosmic Ray Conference (ICRC 2019): Madison, Wisconsin, USA, July 24- August 1, 2019 (Preprint 1909.09073)
  * [9] Pierog T, Karpenko I, Katzy J M, Yatsenko E and Werner K 2015 Phys. Rev. C 92 034906 (Preprint 1306.0121)
  * [10] Riehn F, Dembinski H P, Engel R, Fedynitch A, Gaisser T and Stanev T 2017 PoS ICRC2017 301 (Preprint 1709.07227)
  * [11] Aab A et al. (Pierre Auger Collaboration) 2020 Phys. Rev. Lett. 125 121106 (Preprint 2008.06488)
  * [12] Aab A et al. (Pierre Auger Collaboration) 2017 JCAP 04 038 [Erratum: JCAP 03, E02 (2018)] (Preprint 1612.07155)
  * [13] Greisen K 1966 Phys. Rev. Lett. 16 748–750
  * [14] Zatsepin G T and Kuzmin V A 1966 JETP Lett. 4 78–80
  * [15] Abreu P et al. (Pierre Auger Collaboration) 2012 Phys. Rev. Lett. 109 062002 (Preprint 1208.1520)
  * [16] Aab A et al. (Pierre Auger Collaboration) 2016 Phys. Rev. Lett. 117 192001 (Preprint 1610.08509)
  * [17] Aab A et al. (Pierre Auger Collaboration) 2015 Phys. Rev. D 91 032003 [Erratum: Phys.Rev.D 91, 059901 (2015)] (Preprint 1408.1421)
  * [18] Aab A et al. (Pierre Auger Collaboration) 2020 Eur. Phys. J. C 80 751
  * [19] Aab A et al. (Pierre Auger Collaboration) 2021 Phys. Rev. Lett. 126 152002 (Preprint 2102.07797)
  * [20] Abreu P et al. (Pierre Auger Collaboration) 2021 PoS ICRC2021 310
  * [21] Aab A et al. (Pierre Auger Collaboration) 2019 Front. Astron. Space Sci. 6 24 (Preprint 1904.11918)
  * [22] Aab A et al. (Pierre Auger Collaboration) 2020 Astrophys. J. 902 105 (Preprint 2010.10953)
  * [23] Abreu P et al. (Pierre Auger Collaboration) 2021 PoS ICRC2021 973
  * [24] Aab A et al. (Pierre Auger Collaboration) 2020 Earth Space Sci. 7 e2019EA000582
  * [25] Abreu P et al. (Pierre Auger Collaboration) 2021 PoS ICRC2021 395
  * [26] Vásquez Ramírez A et al. (Pierre Auger Collaboration) 2021 PoS ICRC2021 327
  * [27] Abreu P et al. (Pierre Auger Collaboration) 2022 Astrophys. J. 933 125 (Preprint 2205.14864)
  * [28] Abreu P et al. (Pierre Auger Collaboration) 2021 PoS ICRC2021 373
  * [29] Abreu P et al. (Pierre Auger Collaboration) 2022 (Preprint 2209.05926)
  * [30] Aab A et al. (Pierre Auger Collaboration) 2019 JCAP 10 022 (Preprint 1906.07422)
  * [31] Abreu P et al. (Pierre Auger Collaboration) 2022 (Preprint 2203.08854)
  * [32] Kuzmin V A and Rubakov V A 1998 Phys. Atom. Nucl. 61 1028 (Preprint astro-ph/9709187)
  * [33] ’t Hooft G 1976 Phys. Rev. Lett. 37 8–11


(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: 1Mohamed bin Zayed University of AI 2Technology Innovation
Institute 11email<EMAIL_ADDRESS>
# Multi-modal Generation via Cross-Modal In-Context Learning

Amandeep Kumar1 Muzammal Naseer1 Sanath Narayan2
Rao Muhammad Anwer1 Salman Khan1 Hisham Cholakkal1

###### Abstract

In this work, we study the problem of generating novel images from complex
multimodal prompt sequences. While existing methods achieve promising results
for text-to-image generation, they often struggle to capture fine-grained
details from lengthy prompts and maintain contextual coherence within prompt
sequences. Moreover, they often result in misaligned image generation for
prompt sequences featuring multiple objects. To address this, we propose a
Multi-modal Generation via Cross-Modal In-Context Learning (MGCC) method that
generates novel images from complex multimodal prompt sequences by leveraging
the combined capabilities of large language models (LLMs) and diffusion
models. Our MGCC comprises a novel Cross-Modal Refinement module to explicitly
learn cross-modal dependencies between the text and image in the LLM embedding
space, and a contextual object grounding module to generate object bounding
boxes specifically targeting scenes with multiple objects. Our MGCC
demonstrates a diverse range of multimodal capabilities, like novel image
generation, the facilitation of multimodal dialogue, and generation of texts.
Experimental evaluations on two benchmark datasets, demonstrate the
effectiveness of our method. On Visual Story Generation (VIST) dataset with
multimodal inputs, our MGCC achieves a CLIP Similarity score of $0.652$
compared to SOTA GILL $0.641$. Similarly, on Visual Dialogue Context (VisDial)
having lengthy dialogue sequences, our MGCC achieves an impressive CLIP score
of $0.660$, largely outperforming existing SOTA method scoring $0.645$. Code:
https://github.com/VIROBO-15/MGCC

Figure 1: Example illustration of image generation based on complex multimodal
prompt sequences. The top row shows challenges faced by diffusion in aligning
the image with the prompt, while state-of-the-art GILL generates a holistic
image combining all prompts. Our method surpasses both by aligning with the
final text and incorporating the visual appearance of pasta" from the first
image in the sequence, resulting in a more contextually accurate image. In the
bottom row, both diffusion and GILL fail to capture the context of the
mentioned object in the dialogue. In contrast, our method generates the
elephant" as described in the dialogue and preserves count information,
illustrating a comprehensive and context-aware image generation process (best
viewed in zoom).

##  1 Introduction

The advancement of large language models (LLMs) [34, 31] trained on extensive
textual corpora has enabled remarkable adaptability across various modalities.
Earlier works demonstrated the effectiveness of grounding text-only LLMs to
images for vision-and-language tasks [57, 9, 46, 20, 27], as well as in
embodied settings for robotics [14, 2] and beyond. These methods leverage the
capabilities of LLMs that are trained on large scale text-only data, while
keeping the LLM weights frozen. In this work, we tackle the problem of
generating novel images with lengthy text descriptions or complex sequence of
text prompts by leveraging the capabilities of both LLMs [54] and diffusion
models [37, 38].

Recent advances in text-to-image generation methods [38, 37, 39, 40] have
demonstrated impressive results in generating novel images. However, these
approaches tend to overlook fine-grained details in the case of lengthy text
prompts or complex text sequences that require understanding the previous
context of the prompts.

Generally, these approaches struggle in these scenarios likely due to two
reasons: (a) they rely on the CLIP text encoder [36] that is limited to
handling 77 tokens at a time, leading to loss of crucial information in
lengthy text prompts, and (b) they cannot process interleaved text-image
sequences as input. To overcome these challenges, GILL [22] proposed to
utilize pretrained LLMs. First, for handling image inputs, it learns to
transform images to the LLM vocabulary space. Then, to generate images, it
aligns the LLM output embedding space to CLIP text encoder output space [36]
via a transformer encoder-decoder module [47]. Such an alignment allows
conditioning diffusion on the LLM embedding for generating images.

While GILL is capable of generating images with lengthy prompt descriptions
and complex sequence of prompts, it still struggles to generate accurate
images aligned with the prompts sequence. This can be attributed to the use of
pre-trained LLMs that are implicitly designed to handle dependence within the
sequence token but not explicitly designed for handling the cross-modal
context, such as image and text tokens. A straightforward solution is to fine-
tune the LLM. However, fine-tuning the LLM requires large amount of
interleaved image text pairs and extensive compute resource [3, 1]. This
approach can also lead to a loss of generalization, which was learned from the
large text corpus. In this work, we address the aforementioned limitations by
training on the image-captions [1] alone.

Contributions: We propose an approach named Multi-modal Generation via Cross-
Modal In-Context Learning (MGCC) that learns to generate multimodal outputs
given lengthy multimodal inputs. To this end, we introduce a novel Cross-Modal
Refinement Module to enable learning the cross-modal dependencies between text
and image in the LLMs embedding space during training. This module aids the
pre-trained LLM to explicitly learn the correspondence between text and image
tokens using cross-attentions. By leveraging the refinement module, the model
gains semantic understanding of the scene based on the input prompt sequence.
Moreover, to enhance the fine grained details in the output, we incorporate a
contextual object grounding module. Utilizing the in-context learning [29, 5],
we predict bounding boxes of the objects present in the prompt while
maintaining the temporal consistency of the prompt sequence. Thereby, we
collectively solve the problem of the object present in the scene and their
count.

Extensive quantitative and qualitative experiments are conducted on two
datasets: Visual Story Generation (VIST) [19] and Visual Dialogue Context
(VisDial) [12]. Our MGCC performs favorably against text-to-image generation
methods and state-of-the-art GILL [22]. When handling multimodal context in
VIST dataset, our MGCC outperforms the state-of-the-art approach in terms of
both CLIP Similarity from $0.641$ to $0.652$ and LPIPS score from $0.693$ to
$0.679$. Similarly, on challenging VisDial dataset with long dialogue prompts,
our MGCC achieves a CLIP Similarity score of $0.660$ largely outperforming the
SOTA GILL with $0.645$. Fig. 1 shows the generation of novel images by our
MGCC, illustrating the improved alignment of our generated images with the
prompts while maintaining temporal consistency.

##  2 Related Works:

Figure 2: Example images depicting the impact of progressively integrating our
cross-modal refinement module (CMRM) and contextual object grounding module
(COGM) into the baseline. In first row, the baseline generates an image of
“cookies and coffee in a plate" which doesn’t align with the earlier prompts
“the boss is teaching the new employee to prepare coffee and snack." Although
the integration of our CMRM module to baseline improves semantic
understanding, the generated image still fails to include the person instance
in the scene. Finally, by incorporating our GCO (grounding contextual
objects), we achieve better alignment with the ground truth, resulting in an
image that accurately matches the number of “persons" mentioned in the earlier
prompt. Similarly, in second row, baseline struggles to generate an image
consistent with the text “the glowing embers of a campfire is so relaxing".
Our refinement module comprehends the prompts and generates “people and
campfire", although the last prompt is most aligned with the “campfire". Our
grounding module generates bounding boxes for the “campfire", resulting in a
more aligned image with the specified context.

Multimodal language model: Our work builds upon recent advancements in large-
scale Transformer-based Language Models (LLMs). These models exhibit
remarkable properties learning from few-shot in-context examples [6, 8] and
the ability to handle lengthy text inputs. Some of the recent LLMs, like
OpenAI’s ChatGPT and GPT4 [34], have showcased impressive language
comprehension and reasoning capabilities through techniques like instruction
tuning [35, 48, 31, 55] and reinforcement learning from human feedback (RLHF)
[42]. Moreover, a range of open-source LLMs, such as Flan-T5 [11], Vicuna
[10], LLaMA [45], and Alpaca [44], have significantly accelerated progress and
have made valuable contributions to the broader community. Subsequently, there
have been efforts to develop multimodal LLMs (MLLLMs) that can handle both
multimodal inputs (image and text) and tasks.

Most of the work in multimodal language models (MLLLMs) [18, 57, 43, 22, 48],
align pre-trained encoders of various modalities with the textual feature
space of LLMs, allowing LLMs to effectively process other modal inputs,
demonstrating compelling few-shot, captioning, and question-answering
capabilities. Other approaches have built on this concept by introducing
adapters [15], increasing model and data sizes [3], improving visual encoders
[3, 27], fine-tuning on instructions [31], and training unified models with
multi-task objectives [32, 51]. For example, Flamingo [3] trained on 1535 TPUs
for 15 days, while RA-CM3 [50] utilized 256 GPUs for 5 days. Recent work,
FROMAGe [23], trained a multimodal LLM capable of processing arbitrarily
interleaved image and text inputs to generate text interwoven with retrieved
images. A closely related work to ours is GILL [22], which requires multimodal
LLMs (MLLMs) to generate conditional embeddings, that are explicitly designed
to align with a pre-trained CLIP encoder. These embeddings can subsequently be
utilized with a pre-trained Stable Diffusion (SD) model [38].

Text-to-Image Generation: The task of generating high-quality images [4, 26,
24, 25, 13] based on textual descriptions has gained popularity [37, 39, 38].
Latent Stable Diffusion [38] introduces denoising in the latent space and then
decodes these denoised latents into high-resolution pixel space. However,
These models fail to handle the complex and lengthy prompt. Recently, research
studies like [29] and [16] have employed LLMs to tackle the challenges posed
by lengthy text sequences. These methods use LLMs to generate layout using
lengthy prompts.

Given prior visual knowledge like layout, segmentation map, poses and stroke
are used to condition to generate the novel image. ControlNet [52], GLIGEN
[28], and ReCo [49] have proposed training-based adaptations for spatially-
conditioned image generation within the framework of diffusion models.
However, these methods depend on external annotated datasets, like COCO [30],
which provide images with annotations such as bounding boxes and captions.
Moreover, relying on training-based adaptation has dual implications. It not
only results in the model’s incompatibility because of add-ons such as
pretrained LoRA [17] weights but also introduces complexities when attempting
to train a new LoRA model as it requires additional training data to finetune.
In contrast, [29] and [16] use training-free generation through existing text-
to-image generation. Different from these methods which only condition the
visual knowledge like layout to the existing text-to-image generator, we
propose a cross-modal refinement module and layout generation using incontext
examples to generate novel images.

##  3 Method

Figure 3: Overall framework of our model, MGCC to generate novel images using
multimodal prompts. During training, (a) our model first align the image into
the LLM token embedding space. (b) To generate the novel images we introduce
special image token $[I]$ to the LLM Vocabulary. We refine these image token
$[I]$ in the LLM feature space by introducing a novel cross-modal refinement
module (CMRM), and then align these refined features in the clip text encoder
space. The refined image token $\hat{F}_{I}$ are then taken as input to the
Transformer Mapper $S_{w}$ to map the tokens into the clip text embedding
space as $f_{g}(y)$. (c) During inference, we use Contextual Object Grounding,
to generate the bounding boxes for the objects present in the scene
$\\{b_{i}\\}_{i=1}^{p}$. We condition theses bounding boxes
$\\{b_{i}\\}_{i=1}^{p}$ along with refined image tokens embedding $f_{g}$ on
the diffusion $D$ to generate the final image $I_{g}$.

Problem Statement: Given a sequence of text prompts or interleaved image-text
prompts (eg. story sequence) presented over multiple instances from $t_{1}$ to
$t_{n-1}$ (Fig. 2), our task is to generate the image at time $t_{n}$ while
maintaining the context of the earlier text and image prompt sequence. Here,
$n$ represents the length of the sequence. Formally, our aim is to process
interleaved sequences of text $y=\\{y_{a}\\}_{a=1}^{n}$ and images
$x=\\{x_{a}\\}_{a=1}^{n-1}$ pairs, where $y$ and $x$ represent text and image
respectively. Then, our objective is to generate a novel image at time
$t_{n}$, retaining the context of earlier prompts. In this work, we leverage
the capabilities of pre-trained and frozen large language models [35, 48, 31]
and diffusion models [37, 38] to generate these images with minimal training
efforts.
Baseline: Our method builds upon the recent GILL approach [22]. In contrast to
conventional diffusion models utilizing clip text encoders [38], GILL adopts a
pre-trained LLM. This fusion of LLM with the diffusion model enables image
generation within extensive multimodal input. In processing text-image
sequences, GILL [22] initially transforms the image into LLM embedding space.
Additionally, it introduces specialized image tokens within the LLM’s
vocabulary to represent the final image to be generated by the model. These
image tokens are aligned with the clip text encoder through a learnable
transformer module named GILLMapper. Subsequently, GILLMapper’s output serves
as input to the diffusion model [22] during inference.

While our baseline GILL enables using lengthy multimodal story sequences, it
faces several limitations: (a) The output of GILLMapper, serving as the pre-
trained diffusion model’s input, tends to generate holistic images
representing all prompts in a story sequence. This results in the loss of
fine-grained details specific to the current prompt at $t_{n}$. For instance,
as shown in (Fig. 1 row 1), the baseline generates a holistic image combining
information from various prompts, such as party foods and fruits, even when
the final prompt corresponds solely to a pasta salad. (b) With increasing
sequence length, GILL struggles to maintain coherent narrative and context,
evident in its inability to generate elephants in the final image (Fig. 1 row
2). (c) Performance deteriorates, particularly with lengthy descriptions and
scenes featuring multiple objects, impacting accurate image generation in such
scenarios.

Motivation: As mentioned earlier, our baseline GILL introduces the image
tokens within the vocabulary of a pretrained Language Model (LLM) to handle
complex generation problems such as story sequence. While pretrained language
models (LLM) are designed to capture dependencies within sequences of tokens,
they are not explicitly optimized for capturing cross-modal relationships
between text and special image tokens. In order to address this limitation and
establish multi-model dependencies, we introduce a cross-modal refinement
module (Fig. 3). This module enables the model to explicitly attend to
relevant parts of the input when generating text and image tokens. Our
proposed refinement is based on cross-attention and aims to refine the image
token within LLM vocabulary such that the diffusion model does not generate a
holistic image corresponding to all the prompts in the story sequence. It can
produce images that contain both the semantics of the last prompt and the
context of previous prompts of the sequence.

Although the refinement of image tokens improve the semantic understanding of
the scene (Fig. 2), the model still lag behind the fine-grained understanding
of the objects and their counts. To address this, we introduce grounding of
contextual objects with LLMs, to predict the layout of objects present in the
last sequence of the story. By doing so, we are solving not only the problem
of temporal consistency but also the problem of missing objects’ in the scene
and their inaccurate count, where both baseline GILL and the diffusion models
are sub-optimal. [22, 28, 16].

###  3.1 Overall Framework

For a given training image $x$ and its caption $y$, we first perform an image
alignment as shown in Fig. 3 (a). While, the input caption $y$ are tokenized
as $(c_{1}\cdots c_{T})$, the input image $x$ is passed through a pretrained
clip visual encoder $g_{\phi}(x)$ to obtain the image embedding
$g_{\phi}(x)\in\mathbb{R}^{d}$. Here $d$ is the dimension of the embeddings.
The goal is to map these image embeddings $g_{\phi}(x)$ into a sequence of $k$
$e$-dimensional vectors, which serve as inputs to the pretrained LLMs. Here,
$e$ represents the embedding dimension of the LLM. We learn a linear mapping
$H_{cap}\in\mathbb{R}^{d\times k_{e}}$ using the given image $x$ and captions
$y$, to translate the $x$ into the token embedding space of the LLMs. This
results in a mapping between the CLIP vision encoder and LLM (Fig. 3 (a)).

To further enable the LLM to generate image outputs, a special set of tokens,
named image tokens $[I]=\text{{[I\\{1\\}]}},\ldots,\text{{[I\\{n\\}]}}$ are
introduced into the vocabulary of the pretrained LLMs as shown (Fig. 3 (b)).
Here, the image tokens correspond to images that the model should generate as
in [56, 23, 22]. The embedding matrix of LLMs, which maps words or tokens to
continuous vector representations, is enhanced with an additional trainable
matrix $\mathbf{Emd}\in\mathbb{R}^{n\times e}$. This trainable matrix allows
the model to better incorporate the specific characteristics to the final
generated image $I_{g}$.

We further introduce a Cross Modal Refinement Network that explicitly learn
the cross-modal alignment to get the refined image token. Theses refined image
token features obtained from the LLM are further aligned to the clip text
encoder, which then serves as an input to a diffusion model $D$ conditioned on
bounding boxes [28]. We use a $4-$layer encoder-decoder transformer $S_{w}$
with the learnable weights [22] to learn the clip alignment. The transformer
$S_{w}$ is conditioned on the refined image tokens processed by the LLM and a
learnable query embedding $(q_{1},\cdots,q_{L})\in\mathbb{T}^{L\times m}$ to
extract $L$ features from LLM hidden states. Here, $m$ is embedding length of
the transformer and $L$ is the maximum sequence length of the Diffusion model
$D$ (similar to DETR [7] and BLIP2 [27]). During Inference (see Fig. Fig. 3
(c)), we introduce a contextual object grounding module (COGM) to predict the
bounding boxes which are used along with the clip aligned features to
condition the diffusion model. Next, we describe our cross model refinement
contextual object grounding modules.

###  3.2 Cross Modal Refinement Module

As discussed before, the frozen and pretrained LLMs are not explicitly
designed to understand the cross-modal relationship between text and distinct
image tokens representing an image within the LLM embedding space. This
results in the loss of fine-grained details specific to the final prompt
within the generated images. For instance, in Fig. 2, our baseline GILL model
[22] generates “coffee and snack" on the table which doesn’t align with prompt
“the boss is teaching the new employee to prepare the coffee and snack" and
also lose the context of the earlier prompts “like shop and customers".

To solve this problem, we introduce a refinement network that explicitly
learns the cross-modal dependencies using the cross-attention between the
special image token and the text in the LLM embedding space. To learn the
refinement module, we pass text $y$ and the image tokens $[I]$ to the LLM and
obtain a multimodal feature embedding $f_{mm}$ in the LLM space. This
multimodal feature contains the embedding representation of both text and
image tokens. We first separate the embedding of image tokens $f_{I}\subset
f_{mm}$ before applying cross-attention between $f_{I}$ and $f_{mm}$.

$\mathrm{Attn^{joint}}=\left(\frac{proj_{q,I}(f_{I})proj_{q,{mm}}(f_{mm})^{T}}{\sqrt{d^{k}}}\right),$
(1)

where $proj_{q,I_{n}}$ and $proj_{q,y}$ are the query projections for the
image token and text features.

$\displaystyle
F_{I}=\mathrm{FFN}^{m}_{I}(\mathrm{softmax}(\mathrm{Attn^{joint}})proj_{t}(f_{mm}))),$
(2) $\displaystyle
F_{mm}=\mathrm{FFN}^{m}_{y}(\mathrm{softmax}(\mathrm{Attn^{joint}}^{T})proj_{I}(f_{I}))),$

where $\mathrm{FFN}$ denotes learnable linear layer, $F_{I}$ and $F_{mm}$ are
refined features by our refinement module. Then, we apply the following
operation to obtain the final image tokens.

$\displaystyle\hat{F_{I}}=(F_{mm}\odot m_{I})+F_{I},$ (3)

where $m_{I}$ is a mask with $0$s for the text tokens and $1$s for the image
tokens. Finally, we pass the refined image tokens $\hat{F_{I}}$ to transformer
$S_{w}$ to align them to the clip text encoder space,

$\displaystyle
f_{g}(y)=s_{w}(\hat{F_{I_{1}}},\hat{F_{I_{2}}},\cdots,\hat{F_{I_{n}}},(q_{1},\cdots,q_{L})).$
(4)

Here $f_{g}\in\mathbb{R}^{1\times L}$ is the output of the transformer.

###  3.3  Contextual Object Grounding Module

We perform in-context learning during inference to generate images that
captures fine-grained details within all prompt sequences. Although the cross-
modal refinement module improves the semantics, the generated images still
lack behinds several aspects. For example, in Fig. 2, the model fails to
generate relevant objects present in the previous prompts of this sequence.
During the in-context learning, our contextual object grounding module detects
the “campfire" in the whole image and generates it accordingly as shown in our
final results.

Specifically, during inference, given a story sequence $(y_{1},y_{2}\cdots
y_{n})$, we generate the bounding boxes $b_{1}\cdots b_{N}$ of relevant
objects along with their class labels. Here, $b_{i}=[x,y,w,h]$ where $w,h$ are
the height and width of the bounding boxes. These bounding boxes are obtained
from the LLMs thorough specific prompting explained below.

Prompting. We designed a prompt for LLMs as follows:

  1. 1.

Description of the task:

You are an intelligent bounding box generator. I will provide you with a
entire story sequence for a occasion. Your task is to generate the bounding
boxes for the last sequence remaining the context of the earlier sequence

  2. 2.

Detail of the image:

The images are of size $512\times 512$ … Format of the bounding boxes should
be fixed … If needed, take the context of the previous sequence and have the
guess.

Similar to the [29, 5], we prompt the Large Language Model (LLM) with manually
curated context examples subsequent to predict the bounding boxes. These
examples serve to elucidate the layout representation and help eliminate
potential ambiguities. An example is provided below:

Story sequence: We took my son on a roadtrip. We stopped to look at the golden
gate bridge. He had a lot of fun in the go carts. We stopped in the desert and
took a picture. He was excited to get home.
Objects: [(‘a car’, [482, 100, 27, 18]), (‘a child’, [102, 107, 201, 402])]

The LLM generates the bounding boxes for the last prompt in the sequence as we
can see in the above example where both “car" and “child" are not mentioned in
the last prompt but LLMs need to have the understanding of the previous
sequence to predict the final content in the scene. The bounding boxes $B$ and
features learned by our alignment network $f_{g}$ are then passed through the
Diffusion model $D$ to generate the final image $I_{g}$ which contains the
semantics as well as the fine-grained details about the sequence.

##  4 Experiment

Datatset: The proposed MGCC method is evaluated on two datasets: Visual Story
Generation (VIST) and Visual Dialogue Context (VisDial).
VIST [19]: The VIST dataset contains a collection of sequences for vision-and-
language tasks, featuring 5 text-image pairs that form a cohesive story.
Similar to [23, 22], our evaluation is performed by generating the final image
in the sequence of texts under three different input conditions: (a) _Single
Caption_ : The input comprises of only the last text description. This
scenario mirrors standard text-to-image generation, where the model is
conditioned on a single caption to produce an image. (b) _Multiple Captions (5
captions)_ : The input encompasses text descriptions from the entire story
sequence. This assessment evaluates the models’ capability to handle longer
and temporally dependent text descriptions. (c) _Multimodal Context (5
Captions, 4 Images)_ : The input encompasses all the image-text pairs
preceding the final image, and additionally the last text description. This
evaluation assesses the models’ proficiency in processing multimodal context
during image generation. VisDial [12] contains sequences of question-answer
(Q$\&$A) pairs related to a specific image that simulate a dialogue between
two individuals discussing the image. Each example incorporates up to 10
rounds of Q$\&$A pairs. This evaluates the model’s generalizability to
dialogue-like text and its ability to process long-text sequences.

Evaluation Metrics: Our evaluation focuses on assessing the capability of the
model to handle complex prompt descriptions. To measure the relevance of the
generated image content, we employ two standard evaluation metrics, CLIP
Similarity and LPIPS. CLIP Similarity utilizes the CLIP [36] ViT-L image
encoder, to extract feature representations for both the real and generated
images, and calculates the cosine similarity between them. A higher score
indicates greater similarity. Learned Perceptual Image Patch Similarity
(LPIPS) [53] measures the distance between image patches, assessing the
dissimilarity between real and generated images. A lower LPIPS value signifies
a closer perceptual resemblance, while a higher value indicates greater
dissimilarity.

Figure 4: The images on the left showcase examples illustrating the multimodal
generation capabilities of our MGCC, which operates on sequential multimodal
input dialogues arranged from top to bottom. On the right-hand side, the
images demonstrate: (a) the model’s ability to perform grounded generation,
(b) its proficiency in following instructions, and (c) its capability in
generating descriptive captions for images.

###  4.1 Implementation Details

Following [22, 23], we train on the Conceptual Captions (CC3M) dataset [41]
comprising of 3.3 million image-text pairs. The OPT-6.7B model [54] serves as
the language model with hidden state embedding dimension $e=4096$. To align
the input image in the LLMs token embedding space we employ CLIP [36] ViT-L
image encoder. For the text-to-image generation module ($D$) we employ Gligen
[28] backbone network. All the weights from the pre-trained models are kept
frozen, updating only the linear layer $\mathbf{H}_{\text{cap}}$, embedding
matrix $\mathbf{Emb}_{\text{img}}$, cross-modal refinement module and the
transformer mapper $\mathbf{S_{w}}$. Similar to [22], we use $k=4$ visual
tokens and $n=8$ learned $[I]$ tokens. The embedding dimension of the
learnable query $q$ is set to $m=512$. The total number of refinement layers
is set to $4$. We optimize using Adam [21] with a learning rate of 0.001 and
parameters $\beta_{1}=0.9$ and $\beta_{2}=0.95$. The total number of in-
context examples to generate the image bounding boxes used during inference is
set to $5$. We train this network with the same losses as our baseline [22]
including the cross entropy (CE) and the mean squared (MSE) losses.

Table 1: Performance comparison with existing approaches on VIST [19]. For
single caption inputs, compared to Stable Diffusion, our approach performs on
par in terms of CLIP similarity, while performing favorably in terms of LPIPS.
Furthermore, our MGCC outperforms both Stable Diffusion and GILL, in terms of
both metrics for long sequence of captions (5 captions) and multimodal inputs
(5 caps, 4 images), highlighting our approach’s improved alignment in
generating context-aware images while maintaining temporal consistency.

|  |  CLIP Similarity ($\uparrow$) |  |  LPIPS ($\downarrow$)
---|---|---|---|---
Model |  | 1 caption | 5 captions | 5 caps, 4 images |  | 1 caption | 5 captions | 5 caps, 4 images
GLIDE [33] |  | 0.582 | 0.591 | - |  | 0.753 | 0.745 | -
Stable Diffusion [38] |  |  0.592 $\pm 0.0007$ | $0.598\pm 0.0006$ | - |  | 0.703 $\pm 0.0003$ | $0.704\pm 0.0004$ | -
GILL [22] |  | 0.581 $\pm 0.0005$ | 0.612 $\pm 0.0011$ | 0.641 $\pm 0.0011$ |  | 0.702 $\pm 0.0004$ | 0.696 $\pm 0.0008$ | 0.693 $\pm 0.0008$
Ours: MGCC |  | 0.591 $\pm 0.0002$ | 0.637 $\pm 0.0007$ | 0.652 $\pm 0.0009$ |  | 0.699 $\pm 0.0015$ | 0.682 $\pm 0.0018$ | 0.679 $\pm 0.0012$

Figure 5: In the first row, the baseline model produces a holistic
representation of the scene, including the “statue" and the “flowers".
However, our model excels in generating “hue" flower picture. In the second
row, the baseline model fails to comprehend the context sequence about the
“room" and the “guest", whereas our model successfully captures this context,
resulting in generating the “room having the bed" with the help of the context
of “relaxing". Moving to the third row, the baseline and diffusion loses the
context as the prompt sequence increases and generates “trees" and “old lady"
whereas our model can generate the images very much aligned with the text
“barrels in the aging room" groundtruth.

###  4.2 Experimental Results

Quantitative and Qualitative Results: We present the quantitative comparison
of our proposed approach MGCC on datasets VIST [19] and VisDial [12] in Tab. 1
and Tab. 2 respectively. In Tab. 1 we observe that when a single caption is
provided, our model’s performance closely aligns with stable diffusion [38],
while marginally outperforming the baseline GILL. However, when a sequence of
5 captions from the story is given as input, our model surpasses both stable
diffusion and GILL, improving CLIP Similarity from $0.612$ to $0.637$ and
LPIPS from $0.696$ to $0.682$. Further investigation with multimodal story
sequences (5 captions and 4 images) improves the CLIP Similarity score from
$0.641$ to $0.652$ and LPIPS from $0.693$ to $0.679$. This demonstrates that
our model is capable of capturing multimodal inputs and a sequence of lengthy
prompts to generate images that are contextually aligned, and the qualitative
results for the same can be seen in Fig. 5. In Tab. 2, we observe that for
short rounds of dialogue, stable diffusion outperforms both GILL and MGCC.
However, for long dialogues sequences, MGCC outperforms both GILL and stable
diffusion, improving CLIP Similarity from $0.645$ to $0.660$ and LPIPS from
$0.714$ to $0.699$. These results indicate that our model is able to handle
the lengthy prompts sequence and dialogue-like inputs better. As shown in Fig.
6 MGCC demonstrates a keen understanding of objects and their count within the
dialogue. This could be attributed to the cross-modal refinement module which
enhances the image tokens for better semantics, and our contextual object
grounding module contributes to generating fine-grained details in the images.
Our model MGCC, can process multimodal dialogue to generate multimodal (images
and text) outputs as shown in Fig. 4.

Table 2: Performance comparison with existing approaches on the VisDial
dataset [12], in terms of CLIP similarity and LPIPS. While Stable Diffusion
performs favorably for short rounds of dialogue, our MGCC approach outperforms
both Stable Diffusion and GILL for long dialogue sequences (5 rounds, 10
rounds), indicating that our approach handles lengthy prompts and dialogue-
like inputs better.

|  |  CLIP Similarity ($\uparrow$) |  |  LPIPS ($\downarrow$)
---|---|---|---|---
Model |  | 1 round | 5 rounds | 10 rounds |  | 1 round | 5 rounds | 10 rounds
GLIDE [33] |  | 0.562 | 0.595 | 0.587 |  | 0.800 | 0.794 | 0.799
Stable Diffusion [38] |  | 0.552 $\pm 0.0015$ | $0.629\pm 0.0015$ | 0.622 $\pm 0.0012$ |  |  0.642 $\pm 0.0010$ | $0.722\pm 0.0012$ | $0.723\pm 0.0008$
GILL [22] |  | 0.528 $\pm 0.0014$ | 0.621 $\pm 0.0009$ | 0.645 $\pm 0.0010$ |  | 0.742 $\pm 0.0004$ | 0.718 $\pm 0.0028$ | 0.714 $\pm 0.0006$
Ours: MGCC |  | 0.539 $\pm 0.0009$ | 0.639 $\pm 0.0010$ | 0.660 $\pm 0.0003$ |  | 0.712 $\pm 0.0019$ | 0.704 $\pm 0.0015$ | 0.699 $\pm 0.0012$

Table 3: Image generation performance on CC3M [41] and VIST [19] with our
proposed contribution onto the baseline.

|  | CC3M |  | VIST
---|---|---|---|---
Model |  |  FID ($\downarrow$) |  |  CLIP Sim ($\uparrow$)
Stable Diffusion [38] |  | 13.94 |  | 0.598
Baseline |  | 15.31 |  | 0.641
Baseline + COGM |  | 14.98 |  | 0.644
Baseline + CMRM |  | 14.67 |  | 0.646
Ours(Baseline + COGM + CMRM) |  | 14.23 |  | 0.652

Table 4: Image generation result on different number of layer of cross modal
refinement module (CMRM) .

|  | CC3M |  | VIST
---|---|---|---|---
$N$ |  |  FID ($\downarrow$) |  |  CLIP Sim ($\uparrow$)
1 |  | 15.11 |  | 0.643
2 |  | 14.83 |  | 0.6470
4 |  | 14.23 |  | 0.652

Figure 6: In the first example, baseline and diffusion models gets confused
between the “fence" and “zebra" whereas our model is able to get the “two
zebra" with fine grained details like “can see whole body of only one". In the
second case, the baseline and diffusion model failed to comprehend the scene
and only generated the “chair" and a “bed" as its output. In contrast, our
model demonstrated its capability by generating the image of “bunky beds" with
fine grained details like floor of bunkbed. In the third row, the baseline and
diffusion fails to generate the wooden floor kitchen whereas our model is able
to generate image aligned with the ground truth.

Ablation Study:

Here, we present the impact of the two proposed modules: cross-modal
refinement (CMRM) and contextual object grounding (COGM) on the CC3M [41] and
VIST [19] datasets in Sec. 4.2. When integrating the CMRM and COGM into the
baseline, we observe progressive improvement in FID and CLIP Similarity
scores. This highlights the importance of learning the cross-modal
dependencies across two different modalities (image and text) and learning
fine-grained details of the objects. Finally, our proposed approach, which
simultaneously integrates the two modules takes advantage of both these
capabilities to generate fine-grained semantically aligned images. This is
reflected in the improvements in the FID from $15.31$ to $14.23$ and CLIP
Similarity from $0.641$ to $0.652$. In Tab. 4, we ablate the impact of the
number of cross-modal refinement module in MGCC. We observe that with the
increase in the number of modules, the FID improves from $15.11$ to $14.23$
and CLIP Similarity from $0.643$ to $0.652$ for module $N=1$ to $N=4$
respectively. This indicates the models’ ability to capture improved cross-
modal dependencies.

###  4.3 Conclusion

We present MGCC, a method designed to generate images from lengthy and complex
multimodal prompt sequences while maintaining temporal consistency. Our
approach involves a cross-modal refinement module explicitly learning
correspondence between multimodal inputs (image and text) and integrates
contextual object grounding for precise control of object layout and count.
Quantitative and qualitative results on two benchmark datasets demonstrate the
merits of our contributions. On both datasets, our method demonstrates
superior image generation quality and alignment with ground truth compared to
existing approaches.

## References

  * [1] Aghajanyan, A., Huang, B., Ross, C., Karpukhin, V., Xu, H., Goyal, N., Okhonko, D., Joshi, M., Ghosh, G., Lewis, M., et al.: Cm3: A causal masked multimodal model of the internet. arXiv preprint arXiv:2201.07520 (2022)
  * [2] Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al.: Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 (2022)
  * [3] Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems 35, 23716–23736 (2022)
  * [4] Bhunia, A.K., Koley, S., Kumar, A., Sain, A., Chowdhury, P.N., Xiang, T., Song, Y.Z.: Sketch2saliency: Learning to detect salient objects from human drawings. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2733–2743 (June 2023)
  * [5] Brooks, T., Holynski, A., Efros, A.A.: Instructpix2pix: Learning to follow image editing instructions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18392–18402 (2023)
  * [6] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020)
  * [7] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: European conference on computer vision. pp. 213–229. Springer (2020)
  * [8] Chan, S.C., Dasgupta, I., Kim, J., Kumaran, D., Lampinen, A.K., Hill, F.: Transformers generalize differently from information stored in context vs in weights. arXiv preprint arXiv:2210.05675 (2022)
  * [9] Chen, J., Zhu, D., Shen, X., Li, X., Liu, Z., Zhang, P., Krishnamoorthi, R., Chandra, V., Xiong, Y., Elhoseiny, M.: Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478 (2023)
  * [10] Chiang, W.L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J.E., et al.: Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) (2023)
  * [11] Chung, H.W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al.: Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 (2022)
  * [12] Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J.M., Parikh, D., Batra, D.: Visual dialog. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 326–335 (2017)
  * [13] Dong, Q., Muhammad, A., Zhou, F., Xie, C., Hu, T., Yang, Y., Bae, S.H., Li, Z.: Zood: Exploiting model zoo for out-of-distribution generalization. In: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A. (eds.) Advances in Neural Information Processing Systems. vol. 35, pp. 31583–31598. Curran Associates, Inc. (2022), https://proceedings.neurips.cc/paper_files/paper/2022/file/cd305fdee96836d5cc1de94577d71b61-Paper-Conference.pdf
  * [14] Driess, D., Xia, F., Sajjadi, M.S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al.: Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 (2023)
  * [15] Eichenberg, C., Black, S., Weinbach, S., Parcalabescu, L., Frank, A.: Magma–multimodal augmentation of generative models through adapter-based finetuning. arXiv preprint arXiv:2112.05253 (2021)
  * [16] Feng, W., Zhu, W., Fu, T.j., Jampani, V., Akula, A., He, X., Basu, S., Wang, X.E., Wang, W.Y.: Layoutgpt: Compositional visual planning and generation with large language models. arXiv preprint arXiv:2305.15393 (2023)
  * [17] Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)
  * [18] Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O.K., Liu, Q., et al.: Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302.14045 (2023)
  * [19] Huang, T.H., Ferraro, F., Mostafazadeh, N., Misra, I., Agrawal, A., Devlin, J., Girshick, R., He, X., Kohli, P., Batra, D., et al.: Visual storytelling. In: Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies. pp. 1233–1239 (2016)
  * [20] Ilharco, G., Zellers, R., Farhadi, A., Hajishirzi, H.: Probing contextual language models for common ground with visual representations. arXiv preprint arXiv:2005.00619 (2020)
  * [21] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [22] Koh, J.Y., Fried, D., Salakhutdinov, R.: Generating images with multimodal language models. NeurIPS (2023)
  * [23] Koh, J.Y., Salakhutdinov, R., Fried, D.: Grounding language models to images for multimodal inputs and outputs (2023)
  * [24] Kumar, A., Bhunia, A.K., Narayan, S., Cholakkal, H., Anwer, R.M., Khan, S., Yang, M.H., Khan, F.S.: Generative multiplane neural radiance for 3d-aware image generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7388–7398 (2023)
  * [25] Kumar, A., Bhunia, A.K., Narayan, S., Cholakkal, H., Anwer, R.M., Laaksonen, J., Khan, F.S.: Cross-modulated few-shot image generation for colorectal tissue classification. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 128–137. Springer (2023)
  * [26] Kumar, A., Ghose, S., Chowdhury, P.N., Roy, P.P., Pal, U.: Udbnet: Unsupervised document binarization network via adversarial game. In: 2020 25th International Conference on Pattern Recognition (ICPR). pp. 7817–7824. IEEE (2021)
  * [27] Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 (2023)
  * [28] Li, Y., Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., Li, C., Lee, Y.J.: Gligen: Open-set grounded text-to-image generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22511–22521 (2023)
  * [29] Lian, L., Li, B., Yala, A., Darrell, T.: Llm-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint arXiv:2305.13655 (2023)
  * [30] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740–755. Springer (2014)
  * [31] Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. arXiv preprint arXiv:2304.08485 (2023)
  * [32] Lu, J., Clark, C., Zellers, R., Mottaghi, R., Kembhavi, A.: Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08916 (2022)
  * [33] Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., Chen, M.: Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 (2021)
  * [34] OpenAI, R.: Gpt-4 technical report. arxiv 2303.08774. View in Article (2023)
  * [35] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.: Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35, 27730–27744 (2022)
  * [36] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [37] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents, 2022. URL https://arxiv. org/abs/2204.06125 7 (2022)
  * [38] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
  * [39] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.: Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems 35, 36479–36494 (2022)
  * [40] Shamshad, F., Naseer, M., Nandakumar, K.: Clip2protect: Protecting facial privacy using text-guided makeup via adversarial latent search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20595–20605 (2023)
  * [41] Sharma, P., Ding, N., Goodman, S., Soricut, R.: Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 2556–2565 (2018)
  * [42] Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., Christiano, P.F.: Learning to summarize with human feedback. Advances in Neural Information Processing Systems 33, 3008–3021 (2020)
  * [43] Su, Y., Lan, T., Liu, Y., Liu, F., Yogatama, D., Wang, Y., Kong, L., Collier, N.: Language models can see: Plugging visual controls in text generation. arXiv preprint arXiv:2205.02655 (2022)
  * [44] Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., Hashimoto, T.B.: Stanford alpaca: An instruction-following llama model (2023)
  * [45] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)
  * [46] Tsimpoukelli, M., Menick, J.L., Cabi, S., Eslami, S., Vinyals, O., Hill, F.: Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems 34, 200–212 (2021)
  * [47] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [48] Wu, S., Fei, H., Qu, L., Ji, W., Chua, T.S.: Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519 (2023)
  * [49] Yang, Z., Wang, J., Gan, Z., Li, L., Lin, K., Wu, C., Duan, N., Liu, Z., Liu, C., Zeng, M., et al.: Reco: Region-controlled text-to-image generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14246–14255 (2023)
  * [50] Yasunaga, M., Aghajanyan, A., Shi, W., James, R., Leskovec, J., Liang, P., Lewis, M., Zettlemoyer, L., Yih, W.t.: Retrieval-augmented multimodal language modeling (2023)
  * [51] You, H., Guo, M., Wang, Z., Chang, K.W., Baldridge, J., Yu, J.: Cobit: A contrastive bi-directional image-text generation model. arXiv preprint arXiv:2303.13455 (2023)
  * [52] Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3836–3847 (2023)
  * [53] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 586–595 (2018)
  * [54] Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X.V., et al.: Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 (2022)
  * [55] Zhang, Y., Zhang, R., Gu, J., Zhou, Y., Lipka, N., Yang, D., Sun, T.: Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107 (2023)
  * [56] Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Learning to prompt for vision-language models. International Journal of Computer Vision 130(9), 2337–2348 (2022)
  * [57] Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 (2023)