(eccv) Package eccv Warning: Package ‘hyperref’ is not loaded, but highly
recommended for camera-ready version

11institutetext:  1ByteDance 2UCAS
11email<EMAIL_ADDRESS>
11email<EMAIL_ADDRESS>
# Graphic Design with Large Multimodal Model

Yutao Cheng* 11 0009-0002-3618-1344 Zhao Zhang* 11 0000-0002-1521-8163 Maoke
Yang* 11 0000-0002-5400-4117
Hui Nie 1122 Chunyuan Li 11 Xinglong Wu 11 Jie Shao🖂 11

###### Abstract

In the field of graphic design, automating the integration of design elements
into a cohesive multi-layered artwork not only boosts productivity but also
paves the way for the democratization of graphic design. One existing practice
is Graphic Layout Generation (GLG), which aims to layout sequential design
elements. It has been constrained by the necessity for a predefined correct
sequence of layers, thus limiting creative potential and increasing user
workload. In this paper, we present Hierarchical Layout Generation (HLG) as a
more flexible and pragmatic setup, which creates graphic composition from
unordered sets of design elements. To tackle the HLG task, we introduce
Graphist, the first layout generation model based on large multimodal models.
Graphist efficiently reframes the HLG as a sequence generation problem,
utilizing RGB-A images as input, outputs a JSON draft protocol, indicating the
coordinates, size, and order of each element. We develop multiple evaluation
metrics for HLG. Graphist outperforms prior arts and establishes a strong
baseline for this field. Project homepage: https://github.com/graphic-design-
ai/graphist

###### Keywords:

Graphic design Layout generation LMM MLLM

\begin{overpic}[width=429.28616pt]{teaser_.pdf} \end{overpic} Figure 1:
Schematic diagram of hierarchical layout generation. (Left) A comparison
between the traditional GLG task and the newly proposed HLG task, with the
major difference in that HLG relaxes the constraint of GLG, so that unordered
multimodal input elements can be processed. (Right) Errors in either layer
sequencing or spatial positioning can significantly impact the overall quality
of the design.

##  1 Introduction

Graphic design [41] fundamentally serves as a form of visual communication. It
involves the creation and combination of symbols, images, and text to express
certain ideas or messages. This field requires significant expertise and time
investment to produce aesthetically pleasing graphic compositions. Recently,
there is a significant paradigm shift in leveraging AI for automating the
layout of given design elements into cohesive graphic compositions heralds
[100, 90, 27, 26]. This could potentially reduce the workload of professional
designers and provide an avenue for beginners to create their design pieces,
making graphic design more democratize and effective.

A preliminary attempt to automate this process is observed in the task known
as Graphic Layout Generation (GLG) [86, 40, 25, 54, 35, 97, 53, 39, 83, 16,
30, 52]. GLG attempts to intelligently arrange provided elements into
attractive compositions under the assumption of a predefined order of layers.
However, establishing the appropriate ordering of these layers is a design
cornerstone that, if mismanaged, can fracture the visual hierarchy, leading to
disarray in the intended message delivery. Requiring users to prescribe an
accurate layer sequence prior to layout not only burdens them with foresight
and planning but also stifles layout algorithms, restricting their capacity to
transcend such confines in the pursuit of innovative and aesthetically
superior outcomes.

In pursuit of more practical applications, this paper introduces a new task,
Hierarchical Layout Generation (HLG). HLG to craft a visually appealing
graphic composition from a collection of unordered elements by meticulously
considering both their spatial arrangement and the sequencing of layers. The
adjective hierarchical emphasizes the significance of element ordering,
setting HLG apart from conventional layout generation practices. The
distinction between GLG and HLG is illustrated in the left panel of Figure 1,
while the right panel of the same figure captures the impact of layer
sequencing and spatial positioning on the visual aesthetics of graphic design.

For the HLG task, we introduce Graphist, the first layout generation model
built upon Large Multimodal Model (LMM) [49, 18, 57, 104, 13, 70]. The layout
generation task is challenging that it digests diverse input elements such as
RGB-A materials, RGB images, and texts, and the desired outcomes must
precisely reflect the intricate relationships among these multimodal input
elements. LMMs are well-suited for this task, as they can unify different
modalities, like images, text, coordinates [64, 13] into tokens. This allows
for flexible configuration of various tasks, such as HLG, GLG, and more
variants. The knowledge and strong reasoning ability stored in pre-trained
model parameters are also beneficial for layout tasks. Furthermore, LMMs
demonstrate significant potential for scaling [42, 61, 73], enabling the
pursuit of enhanced performance through the use of larger models and more
extensive datasets. For these reasons, LMMs were a natural choice for our
foundational architecture in developing Graphist. In our specific approach for
HLG, we train Graphist with graphic composition data, in which, as shown in
Figure 2, each design element is represented as an RGB-A image as input, the
model then generates a JSON draft protocol, which specifies the coordinates,
size, order and other attributes of each design elements in end-to-end manner.

To evaluate the HLG task, we introduce two novel metrics: Inverse Order Pair
Ratio (IOPR) and GPT-4V Eval. The former assesses the accuracy of the layer
order in the graphic composition, while the GPT-4V Eval, leveraging the
capabilities of GPT-4V [62], quantifies the overall aesthetic quality.
Additionally, we have incorporated a human rating score to align with the
subjective perceptions of real individuals. Graphist has emerged as the SoTA
solution, excelling across all these metrics. Furthermore, Graphist
consistently delivers impressive performance on conventional GLG task.

\begin{overpic}[width=429.28616pt]{pipeline_.pdf} \end{overpic} Figure 2:
Graphist Pipeline. Graphist comprises three components: RGB-A Encoder, Visual
Shrinker, and a LLM. It accepts a variety of design elements and generates a
graphic composition in JSON format end-to-end.

We summarize our contributions as follows:

  * •

We introduce the Hierarchical Layout Generation (HLG) task, which creates
graphic compositions from unordered design elements. HLG overcomes the
constraints of the Graphic Layout Generation that requires pre-determined
layer ordering. The new setup allows more flexible and practical AI-assisted
graphic designs.

  * •

We present Graphist, the first LMM-parameterized layout generation model that
can be trained end-to-end. Graphist accepts a variety of vision-and-language
multimodal design elements and generates a graphic compositions in JSON code
format, which can be further automatically rendered into the final design.

  * •

We develop evaluation metrics for HLG, including IOPR and GPT-4V Eval.
Graphist shows superior performance in these metrics, setting a robust
benchmark for the field.

##  2 Related Work

\begin{overpic}[width=433.62pt]{one_case_.pdf} \end{overpic} Figure 3:  A
user-generated case via graphist web demo. The top-left figure represents the
input design elements to Graphist. Below it, we present the corresponding
output JSON code generated by Graphist. The final two images in the top row
illustrate the visualized results: first is the layout visualization, and the
second is the graphic composition by putting these elements according to the
JSON protocol. Additional examples are available in Figure 6.

###  2.1 Graphic Layout Generation

Graphic design is a form of visual art that combines multimodal elements
(_e.g_., images, texts, and symbols) to create aesthetically pleasing
compositions which can effectively convey information to the audience. As one
of the core technologies in automated graphic design, layout generation
methods has been widely used in various scenarios, such as document [100, 39],
UI [19, 6, 58, 82], social media snippets [90], banners [44], poster and
advertisement [30, 52, 55], icon/logo [11, 24, 85, 59], CAD Sketches [63, 27],
slides [68, 32, 26].

As people deepen their understanding of layout generation tasks, the modeling
of layout generation problems is becoming increasingly complex. Early methods
for layout generation typically relied on aesthetic rules [5, 87] or
constraints [34, 37]. However, the diversity and aesthetics of generated
compositions are primarily restricted by the formalized human prior knowledge,
and are hard to scale up for production. Later on, some data-driven methods
[47, 44, 2] emerged, utilizing learning-based generation models for layout
generation. These approaches are dedicated to fitting the data distribution of
the topological relationship between elements, and focus on the alignment of
elements, but neglecting the content of the elements. Recent methods [101, 92,
12, 29, 46, 55] have recognized the importance of element content, and have
explored ways to layout text or decorations on a given base graphic without
obstructing important content. Drawing from this observation, attempts [36,
99] have also been made to incorporate multimodal inputs in order to better
understand the content of each layer. These methods often assume that the
layering order of the elements has been predefined [86, 36], which is
difficult to achieve in the practical application. Therefore, we argue that
the layer ordering needs to be fully considered in the layout generation
problem.

The advancement of layout generation techniques is closely intertwined with
the progress of fundamental algorithms. During the early exploration [5, 87,
34, 37], researchers attempted to define beauty using formulas and
constraints. Subsequently, generative methods based on GANs [47, 48, 31], VAEs
[44, 2], and diffusions [97, 46, 82] have been widely attempted. Besides,
LayoutDETR [92] combining DETR [10] and generative models has been attempted
to deal with the problem of element arrangement on a given image. In the era
of Transformer [77], several methods [36, 39, 53] have adopted the powerful
learning capabilities of Transformers to address layout and related control
problems [39, 53]. FlexDM [36] models graphic design using a BERT-like [20]
approach. The rise of large language models [9, 75, 76] has also captured the
attention of researchers in the field of graphic design, with LayoutPrompter
[54] and LayoutNUWA [72] introducing language models into layout generation in
a zero-shot manner. COLE [38] fuses large language models and diffusion models
to build a hierarchical graphic design framework. Our approach is also
inspired by large language models [76, 98] and multimodal models [57, 56, 104,
13]. However, COLE [38] focuses on utilizing the generating power of diffusion
models to directly produce multi-layer graphic designs based on user intent,
while our approach prioritizes the use of existing design elements for layout
generation.

###  2.2 Large Multimodal Models

As a bridge between language and vision, LMM [45, 89, 95] has received
widespread attention recently. Related topic including autonomous driving [21,
17], video understanding [71, 96, 50], open-vocabulary vision understanding
[93, 84, 51], multimodal agent [88, 78], image generation [7, 69, 94],
embodied intelligence [23, 60, 8] and so on. To the best of our knowledge,
Graphist is the first work to address the task of layout generation using LMM
in an end-to-end manner. While COLE [38] likewise employs multimodal models in
their pipeline, their main emphasis lies in generating text content and styles
on predetermined base image.

Common architectures of LMM typically encompass a pre-trained visual encoder
[22, 67] for extracting visual features, a pre-trained LLM[75, 76, 74] for
interpreting user commands and generating responses, as well as a vision-
language cross-modal connector[1, 57, 49, 28, 103] for aligning the visual
encoder outputs to the language model. Given that the task of hierarchical
layout generation involves organizing design elements on a blank canvas, the
compatibility of coordinates is of paramount importance. Pix2Seq [15] was the
first to explore the use of a discretization and serialization approach in
object detection problems, in order to convert coordinates into token
sequences that can ultimately be used in sequence generation mode. Some work,
like OFA [79], VisionLLM [81], Kosmos-2 [33], build upon this form by
introducing special coordinate tokens in their vocabulary. Alternatively,
PerceptionGPT [65] implements an additional vision encoder and decoder
specifically for processing and predicting coordinates. Most notable among
these approaches is Shikra [13], which represents spatial positions using
numerical values in natural language. It has been proven succinct and
effective, as echoed by related research [4, 91, 80]. Inspired by Shikra, our
work adopts the numerical representation for coordinates in the natural
language sequence.

##  3 Task Formulation

###  3.1 Graphic Layout Generation

For a unified representation, we assume that a text box can be represented by
an RGB-A image, which can usually be achieved through rendering or similar
methods. Under this assumption, GLG is defined as finding an optimal
arrangement for a collection of RGBA design assets
$\mathcal{M}=\\{M_{i}\in\mathbb{R}^{h_{i}\times w_{i}\times 4}\\}_{i=1}^{n}$.
This endeavor seeks to spatially organize the set in a coherent graphic
composition $\mathcal{S}=\\{s_{i}\\}_{i=1}^{n}=f(\mathcal{M})$, where
$f(\cdot)$ embodies our Graphist layout methodology. Specifically, the output
constitutes a set of transformations wherein for each asset $M_{i}$, a
quadruple $(x_{i},y_{i},w_{i},h_{i})$ expresses its upper-left corner
placement along with the respective width and height adjustments within the
composition.

###  3.2 Hierarchical Layout Generation

Given a set of RGB-A design elements
$\mathcal{M}=\\{M_{i}\in\mathbb{R}^{h_{i}\times w_{i}\times 4}\\}_{i=1}^{n}$,
HLG seeks to arrange them into a well-constructed graphic composition
$\mathcal{S}=\\{s_{i}\\}_{i=1}^{n}=f(\mathcal{M})$. In this context,
$f(\cdot)$ denotes the layout action implemented by our method, which we refer
to as Graphist. Each element $s_{i}$ in the layout prediction encompasses five
numerical values $(x_{i},y_{i},w_{i},h_{i},l_{i})$, associated with $M_{i}$
top/left coordinates, width, height and hierarchy respectively. As shown in
Figure 1, compare to conventional GLG task [86], HLG framework emphasizes the
significance of element stratification by imposing a hierarchy $l_{i}$ that
dictates not only the placement but also the $z$-order of each element, thus
ensuring a compositionally harmonious layering.

##  4 Proposed Method

###  4.1 Graphist Architecture

Graphist is constructed using a LMM, which receives the input of multimodal
design elements and predicts a JSON fragment formatted like Figure 2. Graphist
comprises three components: $(i)$ RGBA-Encoder. We utilize a ViT-L/14 with
$224\times 224$ four-channel input as RGBA-Encoder, initialized with visual
tower parameters of CLIP [67], excluding the alpha channel. $(ii)$ LLM. Our
LLM foundation incorporates Qwen1.5-0.5B/7B [3]. $(iii)$ Visual Shrinker. To
manage the extensive elements processing requirements of Graphist, the Visual
Shrinker compresses ViT’s $16\times 16+1~{}\text{(cls-token)}$ output grid
feature tokens into only 5 tokens, thereby saving computational costs.
Specifically, it compress the 2D output of ViT’s $16\times 16$ into $2\times
2$ tokens via 2D average pooling and concatenates it with the cls-token to
form $\mathbf{V}\in\mathbb{R}^{5\times D}$. Consequently, it uses one MLP
layer to map the $\mathbf{V}$ to $\mathbf{V^{\prime}}\in\mathbb{R}^{5\times
D^{\prime}}$ for modal alignment and dimension matching of LLM, where
$D^{\prime}$ is the word embedding dimension defined by LLM. Visual embedding
can be inserted into anywhere of input sequence. Inspired by Shikra [13], we
employ the digital tokens to represent coordinates, devoid of any vocab,
specialty encoders, or any pre-/post-detectors for encoding position
information. Model variants include Graphist-Tiny utilizing Qwen1.5-0.5B and
Graphist-Base engaging Qwen1.5-7B.

###  4.2 Training Strategy

The proposed Graphist is trained in three stages. Initially, Stage-1 targets
the patch embedding layer and the projector layer. The focal training tasks
during this phase predominantly encompass image captioning and the HLG task,
leveraging a substantial batch size coupled with a reduced sequence length for
efficiency. The primary objectives is to calibrate the visual encoder to
interpret the alpha channel in RGB-A imagery and to align the projector with
both visual and linguistic features.

Progressing to Stage-2, the training expands to encompass both the projector
layer and the full LLM. Training tasks from the initial stage are retained;
however, the HLG task is administered with greater frequency to highlight the
model’s fundamental understanding of graphic layout.

Finally, in Stage-3, maintains the focus on the projector layer and the
complete LLM, but aiming to adapt the model to a broader range of graphic
design task types. Traditional GLG task is also regarded as a specific task
type in this stage. We list primary configurations during training in Table 1.

Table 1:  Implementation details including training stages, datasets, and essential parameters: “BS” for batch size, “Length” for total sequence length. "Graphist" is trained on academic data; “Graphist*” on proprietary data. “PE.” denotes the Patch Embedding Layer. “Cap.” signifies image captioning task. “Vari.” denotes the task variants in Section 5.1.1. “IH-RGBA” is in-house image-text dataset, where images with alpha channel. “IH-Design” is 80k in-house Graphic Design Dataset like Crello [86].  Stage | BS | Length |  Tune Part |  Task | Training datasets
---|---|---|---|---|---
Graphist |  Graphist*
1 | 128 | 1536 |  PE
Projector |  Cap.
HLG |  ShareGPT4v[14]
Flickr30k[66]
Crello[86] |  \+ IH-RGBA
2 | 64 | 2048 |  Projector
LLM |  Cap.
HLG |  ShareGPT4v[14]
Flickr30k[66]
Crello[86] |  \+ IH-RGBA
\+ IH-Design
3 | 64 | 3584 |  Projector
LLM |  Cap.
HLG
GLG
Vari. |  ShareGPT4v[14]
Flickr30k[66]
Crello[86]
CGL-V2[46] |  \+ IH-RGBA
\+ IH-Design

To train the model to arrange inputs layer order and spatial coordinates, we
randomly shuffle the input elements with a 0.75 probability for all of the
Graphic Design datasets used in the training process. In these three stages,
the first stage trains 10k steps, while the second and third stages both train
20k steps. The training for the Graphist-Base is over the course of 5 days,
whereas for the Graphist-Tiny model, a shorter duration of 2 days is
anticipated.

##  5 Experiment

###  5.1 Datasets

####  5.1.1 Crello. [86]

Crello dataset 111https://huggingface.co/datasets/cyberagent/crello furnishes
an array of graphic compositions derived from a web-based design utility,
namely Crello (Now change the name to
VistaCreate222https://create.vista.com/). It covers an extensive range of
graphic compositions suited for various applications such as social media
infographics, digital banner ads, blog headers, and printed poster templates.
Inside the dataset, each graphic composition includes detailed information
about the layering order, spatial positioning, and categorical details of the
design elements. The dataset is advantageous for tasks like GLG and has been
the foundation for several methods [86, 36]. Additionally, it is a good
playground for HLG task. In Flex-DM [36], the dataset is partitioned into
19,095 training, 1,951 validation, and 2,375 testing examples. However, they
used the Crello v2, but since the current version released by Crello is v4, we
used the intersection of all parts in the two version test sets, a total of
242 graphic compositions as the test set in experiments.

CGL-Dataset V2 is an extension of the CGL-Dataset [102] and provides a dataset
for generating advertising poster layouts, including 60,548 training samples
and 1,035 testing samples. Previous work [46] has built two tasks based on
this dataset: the Content-Aware task and the Content-Agnostic task. In the
Content-Aware task, the model is challenged with an input image and textual
content, and is required to output the positions of all textual content in the
image while autonomously adding underlay and decorative information. The
Content-Agnostic task, on the other hand, only requires an image as input. In
this case, the model autonomously determines the output, which includes text,
underlay, decoration, and other relevant layering information. Graphist has
integrated these tasks alongside HLG/GLG tasks to facilitate comparison across
a broader spectrum of layout methods [47, 101, 43, 35, 46].

###  5.2 Evaluation Metrics

####  5.2.1 Inverse order pair ratio (IOPR).

The appropriate order of layers is crucial for the results of HLG. We develop
IOPR, which is a ratio representing the fraction of overlap element pairs that
are in inverse order according to the model’s predictions out of all possible
overlapping element pairs. It is calculated as

$\text{IOPR}=\frac{\sum_{i=1}^{n-1}\sum_{j=i+1}^{n}\mathbbm{1}\left(\mathcal{O}_{j}<\mathcal{O}_{i}\land\mathbbm{1}\left(i,j\right)\right)}{\sum_{i=0}^{n-1}\sum_{j=i+1}^{n}\mathbbm{1}},$
(1)

where $n$ is the number of layers in the hierarchical structure. $\mathbbm{1}$
is an indicator function that returns 1 if the argument condition is true and
0 otherwise. $\mathcal{O}$ denotes the output order or predicted order of the
layers as determined by the model. $\mathcal{O}_{i}$ and $\mathcal{O}_{j}$
correspond to the predicted order positions of the $i^{th}$ and $j^{th}$
layers, respectively. $\mathtt{overlap}(i,j)$ is a predicate function that
determines whether the $i^{th}$ and $j^{th}$ layers overlap. A low IOPR would
suggest that the model is quite accurate in predicting the correct order of
layers because there are few inverse order pairs. Contrastingly, a high IOPR
would suggest the model often predicts the wrong sequence, indicating lower
prediction accuracy. In our experiments, we employed
$\text{IOPR}_{\text{min}}$ and $\text{IOPR}_{\text{avg}}$ to evaluate the
performance of the model, representing respectively the average score and the
minimum score in the test dataset. While the average IOPR across a dataset can
provide significant insights, it should be noted that variances in layer
orders for specific graphic compositions, when compared to ground truth, do
not inherently imply subpar layout performance.

####  5.2.2 GPT-4V Eval.

In addition to the layers order, the overall aesthetic quality and harmony of
the elements in the graphic composition are vitally important. We utilize
GPT-4V to evaluate our approach and compare it with other alternatives.
Following COLE [38], we use four scores named GPT-4V rating including
$S_{DL}$, $S_{GI}$, $S_{IO}$ and $S_{TV}$ use GPT-4V to evaluate the quality
of the graphic composition we generate.

  * •

$S_{DL}$ means the graphic design should present a clean, balanced, and
consistent layout. The organization of elements should enhance the message,
with clear paths for the eye to follow.

  * •

$S_{GI}$ reflects that any graphics or images used should enhance the design
rather than distract from it. They should be high quality, relevant, and
harmonious with other elements.

  * •

$S_{IO}$ evaluates the innovation level of the design.

  * •

$S_{TV}$ represents text readability. A lower score would be assigned if the
readability of the text is poor due to the color of the text being similar to
the background color or overlapping of the text.

As a supplement to GPT-4V rating [38], we propose GPT-4V voting. Here, GPT-4V
also partakes in a comparative analysis. It selects the most proficient
graphic composition when confronted with two competing outputs. This
preference distribution acts as a testament to the discernibility of GPT-4V in
recognizing and preferring one method over the other across a range of
comparative samples. GPT-4V rating [38] and GPT-4V voting together form the
GPT-4V Eval for evaluating layout ability.

###  5.3 Comparison with SoTA

We trained our model using the approach outlined in Section 4.2 and compared
it to other state-of-the-art methods. In the following comparison, we
represent the model trained only on public datasets as “Graphist”, while
“Graphist*” denotes the model trained on in-house data in addition to public
datasets.

####  5.3.1 Results in GLG & HLG tasks.

Table 2: GPT-4V Eval on Crello. The table demonstrates the performance of different methods on the Crello dataset. Graphist is Graphist-Base built upon Qwen1.5-7B. The scores on the left are GPT-4V rating and $\text{IOPR}_{\text{avg}}$, while the chart on the right showcases a comparative evaluation using GPT-4V voting against the FlexDM [36], Gemini-1.5-Pro (Gemini) and GPT-4V. Flex-DM is compared based on the GLG task (a, b), while Gemini-1.5-Pro and GPT-4V are based on the HLG task (c, d). Method |  Task |  $S_{DL}$ |  $S_{GI}$ |  $S_{IO}$ |  $S_{TV}$ |  IOPR |
---|---|---|---|---|---|---|---
Flex-DM |  GLG |  5.43 |  6.13 |  4.69 |  4.60 |  -
GPT-4V |  GLG |  5.10 |  5.83 |  4.54 |  3.84 |  -
Gemini |  GLG |  4.87 |  5.62 |  4.39 |  4.09 |  -
Graphist |  GLG |  5.60 |  6.64 |  4.84 |  4.86 |  -
Graphist$*$ |  GLG |  5.72 |  6.70 |  5.03 |  5.37 |  - |
GPT-4V |  GLG |  4.19 |  4.66 |  3.71 |  3.25 |  0.45 |
Gemini |  GLG |  5.06 |  5.94 |  4.33 |  4.21 |  0.70 |
Graphist |  HLG |  5.66 |  6.60 |  5.02 |  4.93 |  0.96 |
Graphist$*$ |  HLG |  5.85 |  6.90 |  5.10 |  5.24 |  0.97 |

The Crello dataset evaluation underscores the dominance of Graphist* and
Graphist over Flex-DM across both tasks. In the GLG task, Graphist* achieved
the highest scores in three out of four domains, with marked gains in
$S_{GI}$, attaining a commanding 6.70, and similarly impressive performance in
$S_{DL}$ (5.72) and $S_{IO}$ (5.03). In $S_{TV}$, Graphist* rendered a
commendable score, highlighting the method’s robustness in maintaining text
readability amidst complex designs. Shifting to the more challenging HLG task,
Graphist* not only continued its strong showing but exhibited even more
dramatic enhancements, scoring uniformly higher over the counterpart, Flex-DM.
Noteworthy is the peak score of 5.85 for $S_{DL}$, underscoring Graphist*’s
proficiency in clean and balanced layout generation. This version also
excelled notably in graphic-image synergy with a top score of 6.90 in $S_{GI}$
and innovation with a superior score of 5.10 in $S_{IO}$. Moreover, the
$S_{TV}$ metric remains above Flex-DM, reflecting the model’s diligence in
preserving text clarity across design variations. We also evaluate the
performance of top-tier proprietary linear mixed models, specifically
Gemini-1.5-Pro (Gemini) and GPT-4V, on both tasks. Although they offer greater
versatility, there is a significant performance gap compared to our Graphist
on layout-centric tasks, such as GLG and HLG. Regarding the evaluation
details, there may be situations where GPT-4V and Gemini-1.5-Pro cannot
correctly infer the required JSON format. We have prepared format post-
processing for their common return result types, and we give these two methods
five inference opportunities for each test case. If all are unsuccessful, then
skip. In the end, GPT-4V retained 120 results, and Gemini-1.5-Pro retained 240
results. We reported their scores on these subsets.

\begin{overpic}[width=429.28616pt]{crellocompare.pdf} \end{overpic} Figure 4:
Results visualization of the GLG task on the Crello dataset. The results for
Flex-DM were derived from their open-source code, whereas the results for
GPT-4V and Gemini-1.5-Pro are obtained in zero-shot manner.

In the visualization of the GLG outcomes in Figure 4, Graphist outperforms
competitors including Flex-DM, GPT-4V, and Gemini-1.5-Pro. The other
methodologies grapple with challenges such as text overlap and image
distortion. Furthermore, we evaluated real-world design elements, contrasting
the performance of Gemini-1.5-Pro with Graphist, as depicted in Figure 5. For
complex design tasks involving over ten layers, our approach yield more
refined design outcomes. Although Gemini-1.5-Pro managed to accomplish the HLG
task, the design acuity is markedly distinct from that of Graphist.

\begin{overpic}[width=424.94574pt]{gemini_cc.pdf} \end{overpic} Figure 5:
Comparison with SoTA method on the HLG task with real-world design elements.
Both Gemini-1.5-Pro and Graphist models are tasked with HLG using identical
real-world design elements. The outcomes indicate superior design quality
achieved by our Graphist* when compared with Gemini-1.5-Pro.

####  5.3.2 Results on CGL-V2.

Table 3: Comparison with content-aware methods on CGL-V2. In this table, we compare Graphist based on Qwen1.5-7B[3] against other content-aware methods. Content-aware refers to generating a layout based on the input background image and text content. C-R means Composition-relevant. Method | C-R measures | Graphic measures
---|---|---
$R_{com}\downarrow$ |  $R_{occ}\uparrow$ |  $R_{ali}\downarrow$ |  $R_{ove}\downarrow$ |  $R_{und}\uparrow$
ContentGAN [47] |  31.930 |  1.000 |  0.009 |  0.065 |  0.840
CGL-GAN [101] |  16.040 |  0.875 |  0.007 |  0.081 |  0.732
RADM [46] |  10.260 |  0.997 |  0.008 |  0.046 |  0.983
Graphist* |  12.150 |  1.000 |  0.004 |  0.002 |  0.996
Table 4: Comparison with content-agnostic methods on CGL-V2. In this table, we compare Graphist based on Qwen1.5-7B[3] against other content-agnostic methods. Content-agnostic refers to generating a layout based solely on the background image. C-R means Composition-relevant. Method | C-R measures | Graphic measures
---|---|---
$R_{com}\downarrow$ |  $R_{occ}\uparrow$ |  $R_{ali}\downarrow$ |  $R_{ove}\downarrow$ |  $R_{und}\uparrow$
BLT [43] |  28.540 |  1.000 |  0.004 |  0.002 |  0.993
LayoutDM [35] |  21.300 |  1.000 |  0.006 |  0.039 |  0.896
RADM [46] |  10.260 |  0.997 |  0.008 |  0.046 |  0.983
Graphist* |  7.074 |  1.000 |  0.0003 |  0.001 |  1.000
\begin{overpic}[width=429.28616pt]{visulize.pdf} \end{overpic} Figure 6:  More
user-generated designs via graphist web demo. To evaluate usability, we
invited numerous non-expert volunteers to upload their design components to
our Graphist web demo, resulting in the creation of their own design projects.
The image displays a selection of these high-quality outcomes alongside their
respective layouts. Within the layout map, various colors signify distinct
layer attributes as recognized by the model.

Besides the HLG and GLG tasks, our model can adapt to various other design
tasks through flexible configurations. We conducted Content-Aware and Content-
Agnostic layout generation experiments on the CGL-V2 dataset based on
Qwen1.5-7B and following the experiment settings and evaluation metrics of
[101].

In the Content-Aware layout design, from Table 3, Graphist stands out with the
lowest $R_{com}$ of 12.150, the highest $R_{occ}$ tied with ContentGAN at
1.000, and leading performances in all graphic measures
($R_{ali}=0.004,R_{ove}=0.002,R_{und}=0.996$), indicating superior grapic
visual balance, non-emptiness, alignment, overlap avoidance, and underlay
success.

Similarly, within Content-Agnostic layout task, Graphist again showcases its
robust capability by achieving the best $R_{com}$ score of 7.074 and perfect
$R_{occ}$ score, which is illustrated in Table 4. It also surpasses
competitors in graphic measures, underpinning its strength in alignment
($R_{ali}=0.0003$), minimizing elements overlap ($R_{ove}=0.001$), and
maximizing the informative underlay usage ($R_{und}=1.0$). These results
cement Graphist as a versatile and potent solution for automatic layout
generation, exhibiting notable advancements in synthesizing information-rich
graphic compositions.

###  5.4 Ablation Studies

In this section, we delve into a series of ablation studies to examine the
underlying factors that influence the layout quality of Graphist. The specific
aspects under investigation include the sequencing of input design elements,
the choice of language model, the number of visual tokens, and the encoding of
transparency through RGB-A image channels.

####  5.4.1 Influence of input layer sequencing.

In Table 2, we show the performance comparison of our method with structured
versus unstructured input sequences. The findings indicate that our approach
achieves better results in the metrics $S_{DL}$, $S_{GI}$, and $S_{IO}$ when
dealing with an unordered input. The flexibility introduced by allowing
randomness in input ordering contributes to the superior performance observed
in $S_{DL}$ and $S_{IO}$ metrics versus a structured sequence. Nevertheless,
this unordered approach increases the complexity of text layer placement,
reflected by a reduced score in $S_{TV}$.

####  5.4.2 Impact of LLM.

Table 5: Different LLM model. Comparison of Qwen1.5-0.5B, Qwen1.5-7B, and InternLM2-7B performance on the Crello dataset. LLM |  $S_{DL}$ |  $S_{GI}$ |  $S_{IO}$ |  $S_{TV}$ |  $\text{IOPR}_{\text{min}}$ |  $\text{IOPR}_{\text{avg}}$
---|---|---|---|---|---|---
Qwen1.5-0.5B |  5.44 |  6.35 |  4.94 |  4.60 |  0.641 |  0.963
Qwen1.5-7B |  5.85 |  6.90 |  5.10 |  5.37 |  0.667 |  0.971
InternLM2-7B |  5.60 |  6.50 |  4.90 |  5.00 |  0.476 |  0.966
Table 6: Different LLM model. Comparison of Qwen1.5-7B and InternLM2-7B performance on the CGL-V2 dataset. LLM |  Task | C-R measures | Graphic measures
---|---|---|---
$R_{com}\downarrow$ |  $R_{occ}\uparrow$ |  $R_{ali}\downarrow$ |  $R_{ove}\downarrow$ |  $R_{und}\uparrow$
InternLM2-7B |  Aware |  12.803 |  1.000 |  0.004 |  0.002 |  0.988
Qwen1.5-7B |  Aware |  12.150 |  1.000 |  0.004 |  0.002 |  0.996
InternLM2-7B |  Agnostic |  9.510 |  1.000 |  0.004 |  0.001 |  0.999
Qwen1.5-7B |  Agnostic |  7.074 |  1.000 |  0.0003 |  0.001 |  1.000

In this evaluation, we explore the influence of varying LLM configurations on
our layout generation results. Similarly, we carry out experiments using the
Crello dataset and the CGL-V2 dataset. We evaluate models with differing
capacities: Qwen1.5-0.5B, Qwen1.5-7B, and InternLM2-7B. As demonstrated in
Table 5 and Table 6, our experiments underline that within the domain of our
tasks, the larger models consistently outperform their smaller counterparts.
In addition, different models of the same size also reflect performance
differences.

Table 7: Different visual token length. Comparison of different sequence length performance on the Crello dataset based on Graphist-Tiny. From our observation, five tokens are sufficient for layout generation.  Length |  $S_{DL}$ |  $S_{GI}$ |  $S_{IO}$ |  $S_{TV}$ |  $\text{IOPR}_{\text{min}}$ |  $\text{IOPR}_{\text{avg}}$
---|---|---|---|---|---|---
5 tokens |  5.44 |  6.35 |  4.94 |  4.60 |  0.641 |  0.963
17 tokens |  5.47 |  6.28 |  4.88 |  4.67 |  0.667 |  0.965
Table 8: Different input channel. Comparison of RGB input and RGB-A input performance on the Crello dataset. The results indicate that RGB-A performs better, particularly in $S_{IO}$ and $S_{TV}$. This experiment based on Graphist-Tiny model. Method |  $S_{DL}$ |  $S_{GI}$ |  $S_{IO}$ |  $S_{TV}$ |  $\text{IOPR}_{\text{min}}$ |  $\text{IOPR}_{\text{avg}}$
---|---|---|---|---|---|---
RGB-A |  5.44 |  6.35 |  4.94 |  4.60 |  0.641 |  0.963
RGB |  5.24 |  6.14 |  4.62 |  4.22 |  0.502 |  0.951

####  5.4.3 Influence of visual token length.

In our method, we typically represent an image using a sequence of 5 visual
tokens. To evaluate the effects of increased token quantity, potentially
enhancing the image representation’s granularity, we conducted experiments
using a 17 tokens format. The composition of 17 tokens includes 1 cls token
and 16 visual tokens, achieved by using a $4\times 4$ pooling kernel size. The
results presented in Table 7 suggests that lengthening the visual token
sequence does not necessarily lead to performance improvements. According to
the outcomes, the representation of an image with a quintet of tokens
sufficiently captures the necessary information for this specific task.

####  5.4.4 RGB _vs_. RGB-A.

To assess the effect of omitting the alpha channel, we have also tested the
model with traditional three-channel RGB images. As illustrated in Table 8,
inputs with the additional alpha channel (RGB-A) yield higher-quality outputs
with more accurate layer ordering when compared to RGB inputs. The alpha
channel provides detailed information that aids the model in discerning
textural elements and gradients within the image layers. Importantly, when
processing text layers, the alpha channel allows the model to isolate text
from potentially distracting backgrounds, facilitating improved clarity and
precision in text placement.

###  5.5 Real-World Evaluation

To assess the real-world applicability of our method, we engaged a group of
non-expert volunteers to submit their design assets via our Graphist web demo
and chose a selection of representative outputs for analysis. As illustrated
in Figure 6, our model successfully generates visually appealing and cohesive
designs across varying canvas dimensions and design elements.

###  5.6 Limitations & Negative Impact

In order to create an efficient design assistant that adheres to user intent,
Graphist only represents a starting point. The automatic generation of
complete sets of high-quality materials that meet design intent requirements,
and the generation of designs that more closely align with human aesthetic
preferences, are areas that require further exploration. As we explore more
intelligent graphic design systems, potential negative consequences primarily
involve generating homogeneous design results and the environmental impact
associated with the carbon consumption of model training.

##  6 Conclusion

This paper represents a step forward from traditional graphic layout
generation by introducing the hierarchical layout generation task, which
enhances graphic design automation by effectively handling disordered design
elements, thereby increasing creative potential and efficiency. To address
this more challenging task, we proposed Graphist, a novel LMM that tackles HLG
tasks as sequence generation challenges. Graphist takes RGB-A images as input
and produces JSON draft protocols that define the layout parameters of graphic
compositions. To appropriately evaluate HLG tasks, we introduced two metrics:
the Inverse Order Pair Ratio and GPT-4V Eval. Our evaluation metrics
demonstrate that Graphist achieves state-of-the-art results, providing a
strong baseline for generating automated graphic designs that are more
creative and diverse.

## References

  * [1] Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems 35, 23716–23736 (2022)
  * [2] Arroyo, D.M., Postels, J., Tombari, F.: Variational transformer networks for layout generation. In: CVPR. pp. 13642–13652 (2021)
  * [3] Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., Tu, J., Wang, P., Wang, S., Wang, W., Wu, S., Xu, B., Xu, J., Yang, A., Yang, H., Yang, J., Yang, S., Yao, Y., Yu, B., Yuan, H., Yuan, Z., Zhang, J., Zhang, X., Zhang, Y., Zhang, Z., Zhou, C., Zhou, J., Zhou, X., Zhu, T.: Qwen technical report. arXiv preprint arXiv:2309.16609 (2023)
  * [4] Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., Zhou, J.: Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966 (2023)
  * [5] Bauerly, M., Liu, Y.: Computational modeling and experimental investigation of effects of compositional elements on interface and design aesthetics. International journal of human-computer studies 64(8), 670–682 (2006)
  * [6] Beltramelli, T.: pix2code: Generating code from a graphical user interface screenshot. In: Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems. pp. 1–6 (2018)
  * [7] Betker, J., Goh, G., Jing, L., Brooks, T., Wang, J., Li, L., Ouyang, L., Zhuang, J., Lee, J., Guo, Y., et al.: Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf 2(3), 8 (2023)
  * [8] Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al.: Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818 (2023)
  * [9] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020)
  * [10] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16. pp. 213–229. Springer (2020)
  * [11] Carlier, A., Danelljan, M., Alahi, A., Timofte, R.: Deepsvg: A hierarchical generative network for vector graphics animation. Advances in Neural Information Processing Systems 33, 16351–16361 (2020)
  * [12] Chai, S., Zhuang, L., Yan, F., Zhou, Z.: Two-stage content-aware layout generation for poster designs. In: ACM MM. pp. 8415–8423 (2023)
  * [13] Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., Zhao, R.: Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195 (2023)
  * [14] Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., Lin, D.: Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793 (2023)
  * [15] Chen, T., Saxena, S., Li, L., Fleet, D.J., Hinton, G.: Pix2seq: A language modeling framework for object detection. arXiv preprint arXiv:2109.10852 (2021)
  * [16] Cheng, C.Y., Huang, F., Li, G., Li, Y.: Play: Parametrically conditioned layout generation using latent diffusion. arXiv preprint arXiv:2301.11529 (2023)
  * [17] Cui, C., Ma, Y., Cao, X., Ye, W., Zhou, Y., Liang, K., Chen, J., Lu, J., Yang, Z., Liao, K.D., et al.: A survey on multimodal large language models for autonomous driving. In: WACV. pp. 958–979 (2024)
  * [18] Dai, W., Li, J., Li, D., Tiong, A.M.H., Zhao, J., Wang, W., Li, B., Fung, P., Hoi, S.: Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500 (2023)
  * [19] Deka, B., Huang, Z., Franzen, C., Hibschman, J., Afergan, D., Li, Y., Nichols, J., Kumar, R.: Rico: A mobile app dataset for building data-driven design applications. In: Proceedings of the 30th annual ACM symposium on user interface software and technology. pp. 845–854 (2017)
  * [20] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
  * [21] Ding, X., Han, J., Xu, H., Liang, X., Zhang, W., Li, X.: Holistic autonomous driving understanding by bird’s-eye-view injected multi-modal large models. arXiv preprint arXiv:2401.00988 (2024)
  * [22] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [23] Driess, D., Xia, F., Sajjadi, M.S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al.: PaLM-E: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 (2023)
  * [24] Feng, S., Jiang, M., Zhou, T., Zhen, Y., Chen, C.: Auto-icon+: An automated end-to-end code generation tool for icon designs in ui development. ACM Transactions on Interactive Intelligent Systems 12(4), 1–26 (2022)
  * [25] Feng, W., Zhu, W., Fu, T.j., Jampani, V., Akula, A., He, X., Basu, S., Wang, X.E., Wang, W.Y.: LayoutGPT: Compositional visual planning and generation with large language models. arXiv preprint arXiv:2305.15393 (2023)
  * [26] Fu, T.J., Wang, W.Y., McDuff, D., Song, Y.: DOC2PPT: automatic presentation slides generation from scientific documents. In: AAAI. pp. 634–642 (2022)
  * [27] Ganin, Y., Bartunov, S., Li, Y., Keller, E., Saliceti, S.: Computer-aided design as language. Advances in Neural Information Processing Systems 34, 5885–5897 (2021)
  * [28] Gao, P., Han, J., Zhang, R., Lin, Z., Geng, S., Zhou, A., Zhang, W., Lu, P., He, C., Yue, X., et al.: Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010 (2023)
  * [29] Gao, Y., Lin, J., Zhou, M., Liu, C., Xie, H., Ge, T., Jiang, Y.: Textpainter: Multimodal text image generation with visual-harmony and text-comprehension for poster design. In: ACM MM. pp. 7236–7246 (2023)
  * [30] Hsu, H.Y., He, X., Peng, Y., Kong, H., Zhang, Q.: PosterLayout: A new benchmark and approach for content-aware visual-textual presentation layout. In: CVPR. pp. 6018–6026 (2023)
  * [31] Hsu, H., He, X., Peng, Y.: Densitylayout: Density-conditioned layout gan for visual-textual presentation designs. In: International Conference on Image and Graphics. pp. 187–199. Springer (2023)
  * [32] Hu, Y., Wan, X.: Ppsgen: Learning-based presentation slides generation for academic papers. IEEE transactions on knowledge and data engineering 27(4), 1085–1097 (2014)
  * [33] Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O.K., Liu, Q., et al.: Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302.14045 (2023)
  * [34] Hurst, N., Li, W., Marriott, K.: Review of automatic document formatting. In: Proceedings of the 9th ACM symposium on Document engineering. pp. 99–108 (2009)
  * [35] Inoue, N., Kikuchi, K., Simo-Serra, E., Otani, M., Yamaguchi, K.: LayoutDM: Discrete diffusion model for controllable layout generation. In: CVPR. pp. 10167–10176 (2023)
  * [36] Inoue, N., Kikuchi, K., Simo-Serra, E., Otani, M., Yamaguchi, K.: Towards flexible multi-modal document models. In: CVPR. pp. 14287–14296 (2023)
  * [37] Jahanian, A., Liu, J., Lin, Q., Tretter, D., O’Brien-Strain, E., Lee, S.C., Lyons, N., Allebach, J.: Recommendation system for automatic design of magazine covers. In: Proceedings of the 2013 international conference on Intelligent user interfaces. pp. 95–106 (2013)
  * [38] Jia, P., Li, C., Liu, Z., Shen, Y., Chen, X., Yuan, Y., Zheng, Y., Chen, D., Li, J., Xie, X., et al.: Cole: A hierarchical generation framework for graphic design. arXiv preprint arXiv:2311.16974 (2023)
  * [39] Jiang, Z., Guo, J., Sun, S., Deng, H., Wu, Z., Mijovic, V., Yang, Z.J., Lou, J.G., Zhang, D.: LayoutFormer++: Conditional graphic layout generation via constraint serialization and decoding space restriction. In: CVPR. pp. 18403–18412 (2023)
  * [40] Jiang, Z., Sun, S., Zhu, J., Lou, J.G., Zhang, D.: Coarse-to-fine generative modeling for graphic layouts. In: AAAI. pp. 1096–1103 (2022)
  * [41] Jobling, P., Crowley, D.: Graphic design: reproduction and representation since 1800. Manchester University Press (1996)
  * [42] Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., Amodei, D.: Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020)
  * [43] Kong, X., Jiang, L., Chang, H., Zhang, H., Hao, Y., Gong, H., Essa, I.: Blt: bidirectional layout transformer for controllable layout generation. In: European Conference on Computer Vision. pp. 474–490. Springer (2022)
  * [44] Lee, H.Y., Jiang, L., Essa, I., Le, P.B., Gong, H., Yang, M.H., Yang, W.: Neural design network: Graphic layout generation with constraints. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16. pp. 491–506. Springer (2020)
  * [45] Li, C., Gan, Z., Yang, Z., Yang, J., Li, L., Wang, L., Gao, J.: Multimodal foundation models: From specialists to general-purpose assistants. arXiv preprint arXiv:2309.10020 1(2), 2 (2023)
  * [46] Li, F., Liu, A., Feng, W., Zhu, H., Li, Y., Zhang, Z., Lv, J., Zhu, X., Shen, J., Lin, Z.: Relation-aware diffusion model for controllable poster layout generation. In: Proceedings of the 32nd ACM international conference on information & knowledge management. pp. 1249–1258 (2023)
  * [47] Li, J., Yang, J., Hertzmann, A., Zhang, J., Xu, T.: Layoutgan: Generating graphic layouts with wireframe discriminators. arXiv preprint arXiv:1901.06767 (2019)
  * [48] Li, J., Yang, J., Zhang, J., Liu, C., Wang, C., Xu, T.: Attribute-conditioned layout gan for automatic graphic design. IEEE Transactions on Visualization and Computer Graphics 27(10), 4039–4048 (2020)
  * [49] Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 (2023)
  * [50] Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L., Qiao, Y.: Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355 (2023)
  * [51] Li, X., Yuan, H., Li, W., Ding, H., Wu, S., Zhang, W., Li, Y., Chen, K., Loy, C.C.: Omg-seg: Is one model good enough for all segmentation? arXiv (2024)
  * [52] Li, Z., Li, F., Feng, W., Zhu, H., Liu, A., Li, Y., Zhang, Z., Lv, J., Zhu, X., Shen, J., et al.: Planning and Rendering: Towards end-to-end product poster generation. arXiv preprint arXiv:2312.08822 (2023)
  * [53] Lin, J., Guo, J., Sun, S., Xu, W., Liu, T., Lou, J.G., Zhang, D.: A parse-then-place approach for generating graphic layouts from textual descriptions. In: ICCV. pp. 23622–23631 (2023)
  * [54] Lin, J., Guo, J., Sun, S., Yang, Z.J., Lou, J.G., Zhang, D.: LayoutPrompter: Awaken the design ability of large language models. arXiv preprint arXiv:2311.06495 (2023)
  * [55] Lin, J., Zhou, M., Ma, Y., Gao, Y., Fei, C., Chen, Y., Yu, Z., Ge, T.: Autoposter: A highly automatic and content-aware design system for advertising poster generation. In: Proceedings of the 31st ACM International Conference on Multimedia. pp. 1250–1260 (2023)
  * [56] Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 (2023)
  * [57] Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. arXiv preprint arXiv:2304.08485 (2023)
  * [58] Lu, Y., Tong, Z., Zhao, Q., Zhang, C., Li, T.J.J.: UI layout generation with llms guided by ui grammar. arXiv preprint arXiv:2310.15455 (2023)
  * [59] Mateja, D., Armbruster, R., Baumert, J., Bleil, T., Langenbahn, J., Schwedhelm, J.C., Sester, S., Heinzl, A.: Animatesvg: autonomous creation and aesthetics evaluation of scalable vector graphics animations for the case of brand logos. In: AAAI. pp. 15710–15716 (2023)
  * [60] Mu, Y., Zhang, Q., Hu, M., Wang, W., Ding, M., Jin, J., Wang, B., Dai, J., Qiao, Y., Luo, P.: Embodiedgpt: Vision-language pre-training via embodied chain of thought. NeurIPS 36 (2024)
  * [61] OpenAI: Gpt-4 technical report (2023)
  * [62] OpenAI: Gpt-4v(ision) system card (2023), https://openai.com/research/gpt-4v-system-card
  * [63] Para, W., Bhat, S., Guerrero, P., Kelly, T., Mitra, N., Guibas, L.J., Wonka, P.: Sketchgen: Generating constrained cad sketches. Advances in Neural Information Processing Systems 34, 5077–5088 (2021)
  * [64] Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., Wei, F.: Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824 (2023)
  * [65] Pi, R., Yao, L., Gao, J., Zhang, J., Zhang, T.: PerceptionGPT: Effectively fusing visual perception into llm. arXiv preprint arXiv:2311.06612 (2023)
  * [66] Plummer, B.A., Wang, L., Cervantes, C.M., Caicedo, J.C., Hockenmaier, J., Lazebnik, S.: Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In: ICCV. pp. 2641–2649 (2015)
  * [67] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748–8763. PMLR (2021)
  * [68] Shibata, T., Kurohashi, S.: Automatic slide generation based on discourse structure analysis. In: ICON. pp. 754–766. Springer (2005)
  * [69] Sun, Q., Yu, Q., Cui, Y., Zhang, F., Zhang, X., Wang, Y., Gao, H., Liu, J., Huang, T., Wang, X.: Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222 (2023)
  * [70] Tai, Y., Fan, W., Zhang, Z., Liu, Z.: Link-context learning for multimodal llms. In: CVPR (2024)
  * [71] Tang, Y., Bi, J., Xu, S., Song, L., Liang, S., Wang, T., Zhang, D., An, J., Lin, J., Zhu, R., et al.: Video understanding with large language models: A survey. arXiv preprint arXiv:2312.17432 (2023)
  * [72] Tang, Z., Wu, C., Li, J., Duan, N.: Layoutnuwa: Revealing the hidden layout expertise of large language models. arXiv preprint arXiv:2309.09506 (2023)
  * [73] Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., Hauth, A., et al.: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)
  * [74] Team, I.: Internlm: A multilingual language model with progressively enhanced capabilities. https://github.com/InternLM/InternLM (2023)
  * [75] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)
  * [76] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)
  * [77] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [78] Wang, J., Xu, H., Ye, J., Yan, M., Shen, W., Zhang, J., Huang, F., Sang, J.: Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint arXiv:2401.16158 (2024)
  * [79] Wang, P., Yang, A., Men, R., Lin, J., Bai, S., Li, Z., Ma, J., Zhou, C., Zhou, J., Yang, H.: Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In: ICML. pp. 23318–23340. PMLR (2022)
  * [80] Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., Song, X., et al.: Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079 (2023)
  * [81] Wang, W., Chen, Z., Chen, X., Wu, J., Zhu, X., Zeng, G., Luo, P., Lu, T., Zhou, J., Qiao, Y., et al.: Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. arXiv preprint arXiv:2305.11175 (2023)
  * [82] Wei, J., Courbis, A.L., Lambolais, T., Xu, B., Bernard, P.L., Dray, G.: Boosting gui prototyping with diffusion models. arXiv preprint arXiv:2306.06233 (2023)
  * [83] Weng, H., Huang, D., Zhang, T., Lin, C.Y.: Learn and sample together: Collaborative generation for graphic design layout. In: IJCAI. pp. 5851–5859 (2023)
  * [84] Wu, J., Li, X., Xu, S., Yuan, H., Ding, H., Yang, Y., Li, X., Zhang, J., Tong, Y., Jiang, X., et al.: Towards open vocabulary learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)
  * [85] Wu, R., Su, W., Ma, K., Liao, J.: Iconshop: Text-guided vector icon synthesis with autoregressive transformers. ACM Transactions on Graphics (TOG) 42(6), 1–14 (2023)
  * [86] Yamaguchi, K.: Canvasvae: Learning to generate vector graphic documents. In: ICCV. pp. 5481–5489 (2021)
  * [87] Yang, X., Mei, T., Xu, Y.Q., Rui, Y., Li, S.: Automatic generation of visual-textual presentation layout. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 12(2), 1–22 (2016)
  * [88] Yang, Z., Liu, J., Han, Y., Chen, X., Huang, Z., Fu, B., Yu, G.: Appagent: Multimodal agents as smartphone users. arXiv preprint arXiv:2312.13771 (2023)
  * [89] Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., Chen, E.: A survey on multimodal large language models. arXiv preprint arXiv:2306.13549 (2023)
  * [90] Yin, W., Mei, T., Chen, C.W.: Automatic generation of social media snippets for mobile browsing. In: Proceedings of the 21st ACM international conference on Multimedia. pp. 927–936 (2013)
  * [91] You, H., Zhang, H., Gan, Z., Du, X., Zhang, B., Wang, Z., Cao, L., Chang, S.F., Yang, Y.: Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704 (2023)
  * [92] Yu, N., Chen, C.C., Chen, Z., Meng, R., Wu, G., Josel, P., Niebles, J.C., Xiong, C., Xu, R.: Layoutdetr: Detection transformer is a good multimodal layout designer. arXiv preprint arXiv:2212.09877 (2022)
  * [93] Yuan, Y., Li, W., Liu, J., Tang, D., Luo, X., Qin, C., Zhang, L., Zhu, J.: Osprey: Pixel understanding with visual instruction tuning. arXiv preprint arXiv:2312.10032 (2023)
  * [94] Zhan, J., Dai, J., Ye, J., Zhou, Y., Zhang, D., Liu, Z., Zhang, X., Yuan, R., Zhang, G., Li, L., et al.: Anygpt: Unified multimodal llm with discrete sequence modeling. arXiv preprint arXiv:2402.12226 (2024)
  * [95] Zhang, D., Yu, Y., Li, C., Dong, J., Su, D., Chu, C., Yu, D.: Mm-llms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601 (2024)
  * [96] Zhang, H., Li, X., Bing, L.: Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858 (2023)
  * [97] Zhang, J., Guo, J., Sun, S., Lou, J.G., Zhang, D.: LayoutDiffusion: Improving graphic layout generation by discrete diffusion probabilistic models. In: ICCV (2023)
  * [98] Zhang, P., Zeng, G., Wang, T., Lu, W.: TinyLLaMA: An open-source small language model. arXiv preprint arXiv:2401.02385 (2024)
  * [99] Zheng, X., Qiao, X., Cao, Y., Lau, R.W.: Content-aware generative modeling of graphic design layouts. ACM Transactions on Graphics (TOG) 38(4), 1–15 (2019)
  * [100] Zhong, X., Tang, J., Yepes, A.J.: Publaynet: largest dataset ever for document layout analysis. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 1015–1022. IEEE (2019)
  * [101] Zhou, M., Xu, C., Ma, Y., Ge, T., Jiang, Y., Xu, W.: Composition-aware graphic layout gan for visual-textual presentation designs. arXiv preprint arXiv:2205.00303 (2022)
  * [102] Zhou, M., Xu, C., Ma, Y., Ge, T., Jiang, Y., Xu, W.: Composition-aware graphic layout GAN for visual-textual presentation designs. In: IJCAI. pp. 4995–5001. ijcai.org (2022)
  * [103] Zhou, Q., Wang, Z., Chu, W., Xu, Y., Li, H., Qi, Y.: InfMLLM: A unified framework for visual-language tasks. arXiv preprint arXiv:2311.06791 (2023)
  * [104] Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 (2023)


CP-violating inflation

its cosmological imprints

Venus Keus[E-mail<EMAIL_ADDRESS>^1,2,
Kimmo Tuominen[E-mail<EMAIL_ADDRESS>^1

^1 Department of Physics, University of Helsinki,

P.O.Box 64, FI-00014 Helsinki, Finland Helsinki, Finland

^2 School of Physics and Astronomy, University of Southampton,

Southampton, SO17 1BJ, United Kingdom

We study models with several $SU(2)$ scalar doublets where the inert doublets
have a non-minimal coupling to gravity and play the role of the inflaton.
We allow for this coupling to be complex, thereby introducing CP-violation
- a necessary source of the baryon asymmetry - in the Higgs–inflaton couplings.
We investigate the inflationary dynamics of the model and discuss how the
CP-violation of the model is imprinted on the particle asymmetries
after inflation in the hot big bang universe.

§ INTRODUCTION

The Standard Model (SM) of particle physics has been extensively tested and is in great agreement with experimental data, with
its last missing particle – the Higgs boson – discovered by ATLAS and CMS experiments at the CERN Large Hadron Collider (LHC) [1, 2].
Although the properties of the observed scalar are in agreement with those of the SM-Higgs boson, it may just be one member of an extended scalar sector.
Even though so far no signs of
new physics have been detected, it is well understood that the SM of particle physics is incomplete.

Cosmological and astrophysical observations imply a large dark matter (DM) component
in the energy budget of the universe.
Within the particle physics setting, this would be a particle which is stable on cosmological time scales, cold, non-baryonic, neutral and weakly interacting [3].
A particle with such characteristics does not exist in the SM.

Another shortcoming of the SM is the lack of an explanation for the origin of the observed matter-antimatter asymmetry in the universe.
One of the most promising baryogenesis scenarios is electroweak baryogenesis (EWBG) [4], which produces the baryon excess during the electroweak phase transition (EWPT). Although the SM in principle contains all required ingredients for EWBG, it is unable to explain the observed baryon excess due to its insufficient amount of CP-violation [5, 6, 7] and the lack of a first-order phase transition [8].

Furthermore, in its current form, the SM fails to incorporate cosmic inflation
in a satisfactory manner.
Inflation is a well-motivated theory predicting a period of exponential expansion
in the early universe
which explains the generation of primordial density fluctuations seeding structure formation,
flatness, homogeneity and isotropy of the universe
[9, 10, 11, 12].
The simplest models of inflation in best agreement with observations are those driven by a
scalar field, the inflaton, with a standard kinetic term, slowly rolling down its
smooth potential.
At the end of inflation, the inflaton which naturally is assumed to have couplings with the SM-
Higgs, dumps its energy into the SM bath during the reheating process which populates
the universe with SM particles.

Scalars with non-minimal couplings to gravity

are well-motivated inflaton candidates since they acquire fluctuations proportional to the inflationary scale and can drive the inflation process in the early universe,
as in the Higgs-inflation model [13] where
the SM-Higgs plays the role of the inflaton, and $s$-inflation models [14, 15] where the SM is extended by a singlet scalar.
Extensive studies have been carried out in simple one singlet or one doublet scalar extensions of the SM (see e.g. [16, 17, 18, 19] and references therein).
These models, however, by construction can only partly provide a solution to the
main drawbacks of the SM.
For example, to incorporate both CP-violation and DM into the model
one has to go beyond simple scalar extensions of the SM [20];
see also e.g. [21, 22, 23, 24, 25, 26].

It is therefore theoretically appealing to have a more coherent setting where
different motivations of beyond SM (BSM) frameworks could be simultaneously investigated.
For example, in non-minimal Higgs frameworks with conserved discrete symmetries one
can accommodate stabilised DM candidates. Moreover, the extended scalar potential
could provide new sources of CP-violation and accommodate a strong first order
phase transition [27]. Collider searches can constrain these model frameworks by
excluding or discovering the existence of the spectrum of new states.

In this paper we introduce a model where a source of CP-violation
originates from the couplings of the inflation.
Through the process of reheating this is transmitted to an asymmetry within the
SM and can furthermore seed
the generation of an excess of matter over antimatter during the
evolution of the early universe.
We describe these dynamics in the context of a $Z_2$ symmetric
3-Higgs Doublet Model (3HDM) with a CP-violating extended dark sector,
which also provides a viable DM candidate, new sources of CP-violation and
a strong first-order EWPT [21, 20, 22, 23, 24, 25].
We study the inflationary dynamics of this set-up and outline its main
consequences. In a future work we aim to continue to complement this study
by more thorough analysis of EWBG and DM observables as well as a phenomenological analysis
towards LHC searches for new physics.

The paper is organized as follows.
In Section <ref> we present the scalar potential and explore the inflationary
dynamics. In Section <ref>, we discuss the inflationary imprints of our novel CP
violating inflation phenomena. In Section <ref>, we discuss the inflaton decay into the SM particles and possible consequences.
In Section <ref> we draw our conclusions and discuss the outlook for further

§ THE SCALAR POTENTIAL

§.§ General definitions

A 3HDM scalar potential which is symmetric under a group $G$ of phase rotations, can be written as the sum of two parts: $V_0$ with terms symmetric under any phase rotation, and $V_G$ with terms symmetric under $G$ [28, 29]. As a result, a $Z_2$-symmetric 3HDM can be written as[We ignore additional $Z_2$-symmetric terms that can be added to the potential, e.g.,
as they do not change the phenomenology of the model [23].]:

V   =   V_0+V_Z_2,

V_0    =    - μ^2_1 (ϕ_1^†ϕ_1) -μ^2_2 (ϕ_2^†ϕ_2) - μ^2_3(ϕ_3^†ϕ_3)

+ λ_11 (ϕ_1^†ϕ_1)^2+ λ_22 (ϕ_2^†ϕ_2)^2  + λ_33 (ϕ_3^†ϕ_3)^2

+ λ_12  (ϕ_1^†ϕ_1)(ϕ_2^†ϕ_2)
+ λ_23  (ϕ_2^†ϕ_2)(ϕ_3^†ϕ_3) + λ_31 (ϕ_3^†ϕ_3)(ϕ_1^†ϕ_1)

+ λ'_12 (ϕ_1^†ϕ_2)(ϕ_2^†ϕ_1)
+ λ'_23 (ϕ_2^†ϕ_3)(ϕ_3^†ϕ_2) + λ'_31 (ϕ_3^†ϕ_1)(ϕ_1^†ϕ_3),

V_Z_2    =    -μ^2_12(ϕ_1^†ϕ_2)+  λ_1(ϕ_1^†ϕ_2)^2 + λ_2(ϕ_2^†ϕ_3)^2 + λ_3(ϕ_3^†ϕ_1)^2  + h.c. where the three Higgs doublets, $\phi_{1},\phi_2,\phi_3$, transform under the $Z_2$ group, respectively, as

g_Z_2=  diag(-1, -1, +1 ).
The parameters of the $V_0$ part of the potential are real by construction. We allow for the parameters of $V_{Z_2}$ to be complex, using the following notation throughout the paper

λ_j = |λ_j|   e^i  θ_j   (j = 1,2,3),
μ^2_12 = |μ^2_12|   e^i  θ_12 .
The composition of the doublets is as follows:
ϕ_1= $\begin{scriptsize}$ H^+_1 $\end{scriptsize}$H_1+iA_1/√(2),  ϕ_2= $\begin{scriptsize}$ H^+_2 $\end{scriptsize}$H_2+iA_2/√(2),   ϕ_3= $\begin{scriptsize}$ G^+ $\end{scriptsize}$v+h+iG^0/√(2),

where $\phi_1$ and $\phi_2$ are the $Z_2$-odd inert doublets,
$\langle \phi_1 \rangle = \langle \phi_2 \rangle =0$, and $\phi_3$ is the one
$Z_2$-even active doublet, which at low energy attains a vacuum expectation
value (VEV)
$\langle \phi_3 \rangle =v/$$ \sqrt{2} $ $ \neq 0$.
The doublet $\phi_3$ plays the role of the SM Higgs doublet, with $h$ being the
SM Higgs boson and $G^\pm,~ G^0$ the would-be Goldstone bosons.
Note that
according to the $Z_2$ generator in Eq. (<ref>)
the symmetry of the potential is respected by the vacuum
$(0,0,v/$$ \sqrt{2} $$)$. In this paper we consider the scenario where
the components of the inert doublets act as inflation candidates and reheat the
universe at the end of inflation through their interactions with the SM-Higgs and
gauge bosons. Note that at the scales relevant for inflation we can take the VEV
of the active doublet to be zero, $\langle \phi_3 \rangle = 0$.

Furthermore, CP-violation is only introduced in the inert sector which is forbidden
from mixing with the active sector by the conservation of the $Z_2$ symmetry. As a
result, the amount of CP-violation is not limited by electric dipole moments [21].
The lightest particle amongst the CP-mixed neutral fields from the inert doublets is a
viable DM candidate and stable due to the unbroken $Z_2$ symmetry.
In this paper, we focus on the inflationary dynamics of the model and shall not discuss DM
implications of the model any further.

§.§ Potential for the inflaton

We start by rewriting the doublets in the unitary gauge and ignore the charged scalars
(since they do not affect the inflationary dynamics).
ϕ_1= 1/√(2) (

h_1 +i η_1
,  ϕ_2= 1/√(2) (

h_2 + i η_2
,   ϕ_3= 1/√(2) (

The action of the model in the Jordan frame is

S_J    =    ∫d^4x √(-g)
[  -1/2 M^2_pl R - D_μϕ_1^†D^μϕ_1
- D_μϕ_2^†D^μϕ_2
- D_μϕ_3^†D^μϕ_3

- V(ϕ_1, ϕ_2,ϕ_3)
- (ξ_1 |ϕ_1|^2  +ξ_2 |ϕ_2|^2 +ξ_3 |ϕ_3|^2
+ ξ_4 (ϕ^†_1 ϕ_2)  +ξ_4^* (ϕ^†_2 ϕ_1 ) )R
where $R$ is the Ricci scalar, $M_{pl}$ is the reduced Planck mass and the parameters $\xi_i$ are dimensionless couplings of the scalar doublets to gravity.

Note that, in principle, $\xi_4$ could be a complex parameter for which we use the notation

ξ_4 = |ξ_4|  e^iθ_4 .

In Eq. (<ref>) the covariant derivative, $D_\mu$, contains couplings of the scalars with the gauge bosons. However, for the dynamics during the inflation,

the covariant derivative is reduced to the normal derivative
$D_\mu\rightarrow \partial_\mu$. The minus sign in the kinetic terms follows the metric convention of $(-,+,+,+)$.

Since we identify the two inert doublets with inflaton, we assume that the energy density of
$\phi_3$ is sub-dominant during inflation. Therefore, the

part of the potential relevant for inflation

V    =    - μ^2_1 (ϕ_1^†ϕ_1) -μ^2_2 (ϕ_2^†ϕ_2)
+ λ_11 (ϕ_1^†ϕ_1)^2+ λ_22 (ϕ_2^†ϕ_2)^2

+ λ_12  (ϕ_1^†ϕ_1)(ϕ_2^†ϕ_2)
+ λ'_12 (ϕ_1^†ϕ_2)(ϕ_2^†ϕ_1)
-μ^2_12(ϕ_1^†ϕ_2)+  λ_1(ϕ_1^†ϕ_2)^2 + h.c. Due to local SU(2) invariance, we can rotate away one of the CP-odd fields,
say $\eta_2$.

Such a transformation is equivalent to taking the $\eta_2 \to 0$ limit, and
we assume this limit to be taken when writing the fields in terms
of components in Eq. (<ref>).

To facilitate the analysis, we apply a conformal transformation from
the Jordan frame, which contains terms with scalar-gravity quadratic couplings,
to the Einstein frame with no explicit couplings to gravity [30].
Physical observables are invariant under this frame transformation. The two frames
are equivalent after the end of inflation when the transformation parameter equals unity.

The action in the Einstein frame can be written as
S_E=∫d^4x √(-g̃)[-1/2M_pl^2R̃-1/2g̃^μν  G_ij  ∂_μφ_i ∂_νφ_j-Ṽ],

where $\tilde{V} = V/\Omega^4$ is the potential in the Einstein frame following
the conformal transformation
g̃_μν   =
Ω^2 g_μν,

G_ij   =
1/Ω^2δ_ij+3/2M_pl^2/Ω^4∂ Ω^2/∂ φ_i∂ Ω^2/∂ φ_j,

where $\varphi_k=h_{1},h_{2},\eta_{1}$, and the transformation parameter

Ω^2 =

1 +ξ_1/M_pl^2(h_1^2+η_1^2)+ξ_2/M_pl^2h_2^2+2 |ξ_4|/M_pl^2
( h_1h_2 c_θ_4 +η_1 h_2 s_θ_4)
using the shorthand notation $c_{\theta_k} = \cos\theta_k $ and $s_{\theta_k} = \sin\theta_k $ throughout the paper.

The prefactor $G_{ij}$ in Eq. (<ref>) leads to mixed kinetic terms.
We introduce the reparametrisation

A =√(3/2)   M_pl   log(Ω^2)
∂ Ω^2/∂ φ_k =
√(2/3)  Ω^2/M_pl   dA/dφ_k
which reduces the kinetic terms to the diagonal form

g̃_μν   G_ij    ∂_μφ_i ∂_νφ_j
Ω^2 g_μν ( δ_ij/Ω^2
+  ∂A/∂φ_i    ∂A/∂φ_j )
∂_μφ_i ∂_νφ_j
∂_μφ_i ∂_μφ_i
+ Ω^2   ∂_μA    ∂_μA

To write the potential in the Einstein frame,

we keep only terms in the potential in Eq. (<ref>)
which are quartic in $h_{1,2}$ and $\eta_{1}$.
This reduces the potential to
Ṽ    ≈    1/4   Ω^4 [
λ_11 (h_1^2 + η_1^2)^2
+λ_22  h_2^4
+(λ_12+λ'_12)(h_1^2 + η_1^2)h_2^2

+ 2 |λ_1| ( c_θ_1
(h_2^2(h_1^2 - η_1^2) )
+ 2  s_θ_1 h_2^2h_1η_1  )
where $\theta_1$ is the CP-violating phase of the $\lambda_1$ parameter.

Further, we introduce another reparametrisation

η_1 = β_1   h_1 ,
h_2 = β_2   h_1   ,
with $\beta_1,\beta_2$ as field dependent values, to rewrite the potential as

Ṽ ≈h_1^4/4   Ω^4 [
λ_11  (1+ β_1^2)^2
+λ_22    β_2^4
+ (
(λ_12+λ'_12)(1 + β_1^2)
+ 2 |λ_1| ( c_θ_1
(1 - β_1^2)
+ 2  s_θ_1 β_1 )) β_2^2
Using this reparametrisation, one can also simplify the $\Omega^2$ parameter in Eq. (<ref>) as

Ω^2 =
1+ (ξ_1/M_pl^2(1+β_1^2)+ξ_2/M_pl^2β_2^2+2  |ξ_4|/M_pl^2
β_2(c_θ_4 +β_1 s_θ_4))   h_1^2≡1+B/M_pl^2h_1^2.
From Eq. (<ref>), recall that $\Omega^2= \exp(\tilde{A})$ using the shorthand notation
One can then write the field $h_1$ in terms of the reparametrised field $\tilde{A}$

h_1^2 =

(e^Ã -1 ) .
Therefore, expressing $h_1^2$ and $\Omega^2$ in terms of $\tilde{A}$ allows us to write the
potential in Eq. (<ref>) in the form
Ṽ∼(1-e^-Ã)^2 X(β_1,β_2).

We will be interested in the effect of the non-minimal coupling $\xi_4$ and
the associated phase $\theta_4$. Therefore, we will set $\xi_1=\xi_2=0$
and assume that the initial field values are such that $\Omega^2>0$ is guaranteed.
Therefore, with these assumptions,
the potential in Eq. (<ref>) can be written as

(M_pl^2/2  |ξ_4| )^2

X(β_1,β_2) =
λ_11  (1+ β_1^2)^2
+λ_22    β_2^4
+ (
(λ_12+λ'_12)(1 + β_1^2)
+ 2 |λ_1| ( c_θ_1
(1 - β_1^2)
+ 2  s_θ_1 β_1 )) β_2^2 /4β_2^2(c_θ_4 + β_1  s_θ_4)^2   .

The second order derivative of the function $X(\beta_1,\beta_2)$ with respect to $\beta_2$ at the minimum ($\partial X/\partial \beta_2=0$) on the left and the second order derivative of the function $X(\beta_1)$ with respect to $\beta_1$ at the minimum ($\partial X/\partial \beta_1=0$) on the right (all $\lambda_i \sim 0.001$). The white area on the left panel corresponds to where the denominator in Eq. (<ref>) becomes zero.

Following the procedure in [16], to find the direction of inflation, we first minimise the $X(\beta_1,\beta_2)$ function with respect to $\beta_2$ which occurs at
⇒  β_2^2 = √(λ_11/λ_22 )   (1+β_1^2)

The second order derivative at this point is
∂^2 X(β_1,β_2)/∂β_2^ 2 =
2  λ_22/(c_θ_4+β_1 s_θ_4)^2

which is always positive provided $\lambda_{22}>0$, as shown in the left panel in Figure <ref>.

Using the $\beta_2$ value in Eq. (<ref>), we can write the $X(\beta_1,\beta_2)$ function solely in terms of $\beta_1$,

(1+ β_1^2)   Λ+ 2  ( (1-β_1^2) c_θ_1
+ 2β_1 s_θ_1) |λ_1|/4 (c_θ_4 + β_1  s_θ_4)^2
with $\Lambda = \lambda_{12}+\lambda'_{12}+2\sqrt{\lambda_{11}\lambda_{22}}\,$.

We repeat the same treatment and minimise the $X(\beta_1)$ function with respect to $\beta_1$.

⇒  β_1 =
(Λ+2 |λ_1| c_θ_1) s_θ_4 - 2 |λ_1| c_θ_4 s_θ_1/(Λ-2 |λ_1| c_θ_1) c_θ_4 - 2 |λ_1| s_θ_4 s_θ_1
We check the positivity of the second order derivative at the minimum point which is satisfied for all $\theta_1,\theta_4$ values as shown in the right panel of Figure <ref>.

Replacing the $\beta_1$ value which minimises the $X(\beta_1)$ function back into the $X(\beta_1)$ function itself, yields the form of $X$ independent of $\beta_1$ and $\beta_2$ with only $\theta_1$ and $\theta_4$ as variables:

X(θ_1,θ_4) = 1/4Λ^2 - λ_1^2  /Λ- 2λ_1 cos(θ_1-2θ_4)

The left panel in Figure <ref> shows the $X(\theta_1,\theta_4)$ function for allowed values of $\theta_1$ and $\theta_4$.
At each point in the plots, one can derive the values of $\beta_1$ and consequently $\beta_2$ using Eq. (<ref>) for given values of $\theta_1$ and $\theta_4$.
The right panel in Figure <ref> shows the values of $\beta_1$ for varying values of $\theta_1$ and $\theta_4$.

The $X(\theta_1,\theta_4)$ function on the left and the values of $\beta_1$ on the right for varying values of $\theta_1$ and $\theta_4$ (all $\lambda_i \sim 0.001$). The white region in the right panel shows a discontinuity where $\beta_1$ values tend to plus infinity approaching from the bottom and to minus infinity approaching from the top of the plot.

§ INFLATIONARY DYNAMICS

With the procedure used in the previous section,

the dynamics is essentially that of a single field inflation.
The full inflationary potential in Eq. (<ref>) can be written as

Ṽ= (M_pl^2/2  |ξ_4| )^2
Figure <ref> shows the inflationary potential for different values of  $\theta_1$ and $\theta_4$. Note that the potential is almost flat at high field values which ensures a slow roll inflation.

The inflationary potential for different values of $\theta_1$ and $\theta_4$ (all $\lambda_i \sim 0.001$).

For the usual slow roll parameters in this case the function $X$ is irrelevant,
since it cancels in the expressions for $\epsilon$ and $\eta$, which are
ϵ   =
1/2M_pl^2  (1/ṼdṼ/dA)^2 =
4/3( 1- e^Ã   )^2 ,

η   =
M_pl^2   1/Ṽd^2 Ṽ/dA^2
4(2-e^Ã)  /3( 1- e^Ã   )^2 .

For field values $A\gg M_{pl}$ (or equivalently $\tilde{A}\gg 1 $), both parameters $\epsilon,\eta \ll 1$ which satisfies the slow roll condition. Inflation ends when $\epsilon \simeq 1$.
To calculate the values of $A$ at the beginning and end of inflation,  $A_i$ and $A_f$ respectively, one needs to calculate the number of e-folds $N_e$, i.e. the number of times the universe expanded by $e$ times its own size. $N_e$ is calculated to be

N_e = 1/M_pl^2∫_A_f^A_i Ṽ/Ṽ' dA
[ Ã_f - Ã_i
-e^Ã_f + e^Ã_i
] ,
where $\tilde{V}'=\frac{d\tilde{V}}{dA}$ and $A_{i}$ ($\tilde{A}_i$) is the value of $A$ ($\tilde{A}$) at the beginning of inflation and $A_{f}$  ($\tilde{A}_f$) is the value of $A$ ($\tilde{A}$) at the end of the inflation.
Since inflation ends when $\epsilon\simeq 1$, one can calculate $A_{f}$, which yields:
e^Ã_f = exp(√(2/3)A_f/M_pl)  ≃2.1547
⇒  Ã_f = √(2/3) A_f/M_pl ≃0.7676   .

To calculate $A_{i}$, one could plug in the $A_{f}$ value into Eq. (<ref>) assuming $N_e=60$, which results in
[-Ã_i+  e^Ã_i
]- 1.0403
⇒  Ã_i = √(2/3)A_i/M_pl

At this point we can also check
the field values in terms of the
original field $h_1$ using Eq. (<ref>).

This gives

h_1_f  =
1.85 ×10^18  /√(|ξ_4| β_2(c_θ_4 + β_1  s_θ_4) ) ,
h_1_i  =
1.59 ×10^19 /√(|ξ_4| β_2(c_θ_4 + β_1  s_θ_4) )  .
In the case of Higgs-inflation where the non-minimal coupling to gravity, $\xi$, is forced to be of the order $\sim 10^{4}$ GeV,
the $h$ field values during inflation are as large as $10^{16}$ GeV or so. In our case
the situation is similar.

Having fixed $N_e$ to 60, and calculated the $A$ field value at the start of inflation, we can derive the scalar power spectrum, $P_s$, the tensor to scalar ratio $r$ and the spectral index $n_s$ as follows:
P_s =1/12 π^2   M_pl^6  (Ṽ)^3/(Ṽ')^2
( (1-e^Ã)^4/128   π^2  e^2Ã)X(θ_1,θ_4)/|ξ_4|^2
5.565 ×X(θ_1,θ_4)/|ξ_4|^2 ,

r =16 ϵ   =    0.00296,

n_s =1-6ϵ+2η   =    0.9678,
where $\tilde{V}'$ is the derivative of $\tilde{V}$ with respect to $A$ and both $\tilde{V}$ and $\tilde{V}'$ are calculated at the $A_{i}$.
Figure <ref> shows the slow roll parameters $N_e$, $n_s$ and $r$ with respect to $\tilde{A}$ with the grid-lines highlighting the $55 <N_e <65$ values. We show the inflationary parameters over a range of $N_e$, since there is no reason for $N_e$ to be precisely $60$.
The values of $r$ and $n_s$ are well within the Plank bounds of $n_s=0.9677\pm 0.0060 $ at $1\sigma$ level and $r<0.11$ at $95\%$ confidence level [31]. Note that the spectral index and the tensor to scalar ratio are in agreement with the Planck bounds over the full range of $N_e$.
Figure <ref> shows the 1$\sigma$ and 2$\sigma$ regions allowed by Planck observations in the $r$-$n_s$ plane and   the theoretical predictions of our framework for $N_e$ values of 55 and 65.

The slow roll parameters: the number of $e$-folds $N_e$ (left), spectral index $n_s$ (center) and tensor to scalar ratio $r$ (right) as a function of $\tilde{A}$ with the grid-lines highlighting the $55 <N_e <65$ values.

The 1$\sigma$ and 2$\sigma$ regions for $n_s$ and $r$
from Planck observation compared to the theoretical
prediction of our framework.

Observations from WMAP7 [32] constrain the scalar power spectrum which put a bound on the $\xi_4$  coupling and angles $\theta_1,\theta_4$,

5.565 ×X(θ_1,θ_4)/|ξ_4|^2   .
In the left panel of Figure <ref>, we show $P_s$ values for the fixed $\theta_1=\pi/3$ angle and varying values of $\xi_4$ and $\theta_4$ up to $3\sigma$ standard deviation from the central value in Eq. (<ref>). In the right panel, we fix $P_s$ to the WMAP7 central value for fixed values of $\lambda_i \sim 0.001$ to get

|ξ_4|= 4.785×10^4  √(X(θ_1,θ_4) )
and show contours of $\xi_4$ for varying values of $\theta_1$ and $\theta_4$.
Note that every point in the plot yields the exact $P_s$ central value.

This is a very important feature of our framework. To satisfy the bounds on the scalar power spectrum, the function $X(\theta_1,\theta_4)$ allows for a wide range of $\xi_4$ values as shown in Figure <ref>. This is in contrast to
the Higgs-inflation models where $P_s \propto {\lambda}/{\xi^2}$
with $\lambda$ the Higgs self-coupling which is fixed to be $\sim 0.12$ at the electroweak scale. Thus, for $P_s$ to agree with observations at the inflationary scale, $\xi$ will have to be very large $\mathcal{O}(10^4)$.

In our set-up, a combination of parameters $\lambda_{1}, \lambda_{11}, \lambda_{22}, \lambda_{12},\lambda'_{12}$ appears in the $X(\theta_1,\theta_4)$ function. The only constraint limiting these parameters is the stability of the potential requiring
λ_ii > 0,     λ_ij + λ'_ij > -2 √(λ_iiλ_jj),     |λ_i| ≤|λ_ii|, |λ_ij|, |λ'_ij| ,   i≠j = 1,2,3  ,
which allows for very small values of $\lambda_i \sim 0.001$ which, in turn, allows for much smaller values of $\xi_4$, at least one order of magnitude than the $\xi$ value in Higgs-inflation models.

Left panel: $P_s$ values for the fixed $\theta_1=\pi/3$ angle and varying values of $\xi_4$ and $\theta_4$ up to $3\sigma$ standard deviation from the observed central value. Right panel: Contours of $\xi_4$ in the $\theta_1$-$\theta_4$ plane which lead to $P_s$ central values (all $\lambda_i \sim 0.001$).

§ REHEATING AND SCALAR ASYMMETRIES

At the end of inflation, the energy stored in the inflaton disperses as the inflaton decays/annihilates into the SM particles through processes mediated by the SM-Higgs and gauge bosons in our case, during the so-called reheating phase [33]. There are numerous
details on how the inflaton decays and creates the initial condition for the
conventional hot early universe. Here our main interest is to discuss how the CP asymmetry
originating from the non-minimal coupling, is transferred to the SM
degrees of freedom.

For the discussion of the scalar asymmetries, let's focus on the neutral components of the $\phi_1$ doublets acquiring an initial non-vanishing expectation value at the exit from
inflaton. We write the field fluctuations around the initial conditions as

ϕ_1 →ϕ_1 - a_1 e^i  α   ,  ϕ_1^†→ϕ_1^* - a_1 e^-i  α

ϕ_2 →ϕ_2 - a_2   , e^i  α   ϕ_2^†→ϕ_2^* - a_2  e^-i  α

ϕ_3 →ϕ_3 - a_3   , e^i  α  ϕ_3^†→ϕ_3^* - a_3  e^-i  α

The phase $\alpha$ here is related to the CP-violating phases of inflation.
Note that at the end of inflation the $h_1$ field has taken the value $h_{1f}$ according to Eq. (<ref>) which is dependant on the inflationary dynamics, namely $\theta_4$, $\beta_1$ and $\beta_2$ which are dependant on $\theta_1$.
Since $h_1$ is the real part of the complex field $\phi_1$, its value is what feeds the
$a_1 \cos\alpha$ component of fluctuations in
Eq. (<ref>). The imaginary part of $\phi_1$, represented by
$\eta_1$, takes a value proportional to $\eta_{1f}=\beta_1 h_{1f}$ as shown in
Eq. (<ref>), and feeds the $a_1 \sin\alpha$ component of the field fluctuations.
Recall that one can obtain the values of $\beta_1$ and $\beta_2$ for any given value of $\theta_1$ and $\theta_4$ from Eq. (<ref>) and Eq. (<ref>).
Explicitly, one can write
tanα=a_1 sinα/a_1 cosα = η_1f/h_1f =   β_1 =
(Λ+2 |λ_1| c_θ_1) s_θ_4 - 2 |λ_1| c_θ_4 s_θ_1/(Λ-2 |λ_1| c_θ_1) c_θ_4 - 2 |λ_1| s_θ_4 s_θ_1 ,
with $\Lambda = \lambda_{12}+\lambda'_{12}+2\sqrt{\lambda_{11}\lambda_{22}}\,$ as mentioned before.
However, to keep the present discussion more transparent, we retain
a generic phase $\alpha$ here.

To discuss the consequences of this complex phase, we now
assume instant reheating. Since the field $\phi_3$ is light with respect to the inflaton
degrees of freedom, we expect the latter to quickly decay to
$\phi_3$. The asymmetry arising from the values of the fields
in Eq. (<ref>) will manifest in creation of unequal number
of $\phi_3$ and $\phi_3^\ast$ quanta as follows.

Let us study the decay process $\phi_1 \to \phi_3^* \phi_3^*$ in detail.
From the potential in Eq. (<ref>), the amplitude of the tree-level process is proportional to
ℳ_(ϕ_1 →ϕ_3^* ϕ_3^*)   ∝  - 2 a_1 λ_3   e^i (α+θ_3)
ℳ_(ϕ_1^* →ϕ_3 ϕ_3)   ∝  - 2 a_1 λ_3   e^-i (α+θ_3) .
The generation of the asymmetry is sensitive to the interference between the tree and loop diagrams [34, 35]. Hence, we need to sketch what happens at loop level.
At one loop level, there are many diagrams that contribute to this decay process.

For the purpose of demonstration, we consider the bubble diagrams which convert $\phi_1$ to $\phi_3$ with only $\phi_1$ and $\phi_1^{*}$ in the loop, as shown in Figure <ref>.
Clearly one needs to take into account all diagrams contributing to this decay process, specially since there may be interferences cancelling the CP asymmetry.
However, since all triple scalar couplings in the potential can be different, one can ensure that such cancellation does not occur.
More careful analysis of these effects is deferred to a future work.

(1.5,-1.25) – node[black,above,xshift=-0.1cm,yshift=0.0cm]  (1.5,-1.25);
[dashed] (0,0) – node[black,above,xshift=-1cm,yshift=-0.25cm] $\phi_1$ (1.5,0);
[dashed] (1.5,0) – node[black,above,xshift=0.8cm,yshift=0cm] $\phi_3^*$ (2.5,0.6);
[dashed] (1.5,0) – node[black,above,yshift=-0.6cm,xshift=0.8cm] $\phi_3^*$ (2.5,-0.6);

[dashed] (0,0) – node[black,above,sloped,yshift=-0.25cm,xshift=-0.8cm] $\phi_1$ (1,0);
[dashed]  (1,0) node[black,above,sloped,yshift=-1.25cm,xshift=0.6cm] $\phi_1^*$  arc (180:0:0.6cm) ;
[dashed]  (1,0) node[black,above,sloped,yshift=0.55cm,xshift=0.6cm] $\phi_1$  arc (-180:0:0.6cm) ;

[dashed] (2.2,0) – node[black,above,sloped,yshift=0cm,xshift=0cm] $\phi_3$ (3.2,0);

[dashed] (3.2,0) – node[black,above,xshift=0.8cm,yshift=0cm] $\phi_3^*$ (4.2,0.6);
[dashed] (3.2,0) – node[black,above,yshift=-0.6cm,xshift=0.8cm] $\phi_3^*$ (4.2,-0.6);

The tree level decay process $\phi_1 \to \phi_3^* \phi_3^*$ and the one-loop bubble diagram with $\phi_1$ and $\phi_1^*$ in the loop.

The amplitude of the loop process with $\phi_1$ and $\phi_1^*$ running in the loop is proportional to
ℳ_(ϕ_1 →ϕ_3 →ϕ_3^* ϕ_3^*)     ∝
-4 a_1 a_3^2 λ_11λ_33 (λ_31+λ'_31) e^-iα ,

ℳ_(ϕ_1^* →ϕ_3^* →ϕ_3 ϕ_3)     ∝
-4 a_1 a_3^2 λ_11λ_33
(λ_31+λ'_31) e^iα .
Due to the interference of the tree and loop diagrams, the decay processes are CP-violating and result in unequal number of $\phi_3$ and $\phi_3^*$ states.

Consequently, we define the asymmetry $A^1_{CP}$ as the difference between the $\phi_1$ decay rate and its conjugate, and we find
A^1_CP = Γ^tree+loop_(ϕ_1 →ϕ_3^* ϕ_3^*) -  Γ^tree+loop_(ϕ_1^* →ϕ_3 ϕ_3)
=  -1/16√(3)π^2 a_1^2 a_3^2 λ_3 λ_11λ_33 (λ_31+λ'_31)sin(2α+θ_3) .

This asymmetry in the scalar sector is then transferred to the fermion sector through the couplings of the Higgs field (the $\phi_3$ doublet) with the fermions.
For example, assuming the existence of right-handed neutrinos, the Yukawa interactions
between neutrinos and $\phi_3$ will generate an asymmetry between $\nu_L$ and $\bar{\nu}_R$,
which would be further translated into baryon asymmetry by the electroweak sphalerons.

§ CONCLUSION AND OUTLOOK

Scalar fields which have non-minimal couplings to gravity are well-motivated inflaton candidates. Paradigmatic examples are the Higgs-inflation [13]
and $s$-inflation models [15].

In this paper we have considered a scenario where several non-minimally coupled scalars
contribute to the inflationary dynamics. In particular we investigated a model where
these scalars are electroweak doublets and therefore generalize the Higgs inflation.
We focused on a setting where the dominant non-minimal coupling is allowed to be complex
and investigated the effect that this would have on CP-violation in our universe.

We determined the inflationary dynamics in the regime where the model essentially
conforms to the predictions of single field inflation. The essential difference is
that the inflaton obtains a non-zero phase representing possible source of CP-violation
for subsequent post-inflationary evolution.

At the end of inflation, the inflaton particle which is naturally assumed to have couplings with the SM Higgs, dumps its energy into the SM particle bath through the process of
reheating, which populates the universe with the SM particles. We sketched how
the complex value of the inflaton field leads to an asymmetry in the scalar sector decays,
and how this asymmetry will further be transmitted to the fermion sector.

There are numerous details in our scenario which can be investigated in more detail.
These include the multi-field dynamics during the inflation as well as the details
of reheating and subsequent particle decays. Also the detailed analysis of the
effects on the generation of baryon asymmetry need to be addressed in more detail.
We will consider these in future work on the model introduced in this paper.

§.§.§ Acknowledgements

VK acknowledges financial support from Academy of Finland projects “Particle cosmology and gravitational waves” No. 320123
and “Particle cosmology beyond the Standard Model” No. 310130.

[1]
G. Aad et al. [ATLAS],

Phys. Lett. B 716, 1-29 (2012)

[arXiv:1207.7214 [hep-ex]].

[2]
S. Chatrchyan et al. [CMS],

Phys. Lett. B 716, 30-61 (2012)

[arXiv:1207.7235 [hep-ex]].

[3]
P. A. R. Ade et al. [Planck],

Astron. Astrophys. 594, A13 (2016)

[arXiv:1502.01589 [astro-ph.CO]].

[4]
D. E. Morrissey and M. J. Ramsey-Musolf,

New J. Phys. 14, 125003 (2012)

[arXiv:1206.2942 [hep-ph]].

[5]
M. B. Gavela, P. Hernandez, J. Orloff and O. Pene,

Mod. Phys. Lett. A 9, 795-810 (1994)

[arXiv:hep-ph/9312215 [hep-ph]].

[6]
P. Huet and E. Sather,

Phys. Rev. D 51, 379-394 (1995)

[arXiv:hep-ph/9404302 [hep-ph]].

[7]
M. B. Gavela, P. Hernandez, J. Orloff, O. Pene and C. Quimbay,

Nucl. Phys. B 430, 382-426 (1994)

[arXiv:hep-ph/9406289 [hep-ph]].

[8]
K. Kajantie, M. Laine, K. Rummukainen and M. E. Shaposhnikov,

Nucl. Phys. B 466, 189-258 (1996)

[arXiv:hep-lat/9510020 [hep-lat]].

[9]
S. W. Hawking,

Phys. Lett. B 115, 295 (1982)

[10]
A. A. Starobinsky,

Phys. Lett. B 117, 175-178 (1982)

[11]
M. Sasaki,

Prog. Theor. Phys. 76, 1036 (1986)

[12]
V. F. Mukhanov,

Sov. Phys. JETP 67, 1297-1302 (1988)

[13]
F. L. Bezrukov and M. Shaposhnikov,

Phys. Lett. B 659, 703-706 (2008)

[arXiv:0710.3755 [hep-th]].

[14]
R. N. Lerner and J. McDonald,

Phys. Rev. D 80, 123507 (2009)

[arXiv:0909.0520 [hep-ph]].

[15]
K. Enqvist, S. Nurmi, T. Tenkanen and K. Tuominen,

JCAP 08, 035 (2014)

[arXiv:1407.0659 [astro-ph.CO]].

[16]
J. O. Gong, H. M. Lee and S. K. Kang,

JHEP 04, 128 (2012)

[arXiv:1202.0288 [hep-ph]].

[17]
S. Choubey and A. Kumar,

JHEP 11, 080 (2017)

[arXiv:1707.06587 [hep-ph]].

[18]
C. Englert, T. Plehn, D. Zerwas and P. M. Zerwas,

Phys. Lett. B 703, 298-305 (2011)

[arXiv:1106.3097 [hep-ph]].

[19]
G. C. Branco, P. M. Ferreira, L. Lavoura, M. N. Rebelo, M. Sher and J. P. Silva,

Phys. Rept. 516, 1-102 (2012)

[arXiv:1106.0034 [hep-ph]].

[20]
V. Keus,

PoS CHARGED2016, 017 (2016)

[arXiv:1612.03629 [hep-ph]].

[21]
A. Cordero-Cid, J. Hernández-Sánchez, V. Keus, S. F. King, S. Moretti, D. Rojas and D. Sokołowska,

JHEP 12, 014 (2016)

[arXiv:1608.01673 [hep-ph]].

[22]
A. Cordero, J. Hernandez-Sanchez, V. Keus, S. F. King, S. Moretti, D. Rojas and D. Sokolowska,

JHEP 05, 030 (2018)

[arXiv:1712.09598 [hep-ph]].

[23]
A. Cordero-Cid, J. Hernández-Sánchez, V. Keus, S. Moretti, D. Rojas and D. Sokołowska,

Eur. Phys. J. C 80, no.2, 135 (2020)

[arXiv:1812.00820 [hep-ph]].

[24]
A. Cordero-Cid, J. Hernández-Sánchez, V. Keus, S. Moretti, D. Rojas-Ciofalo and D. Sokołowska,

Phys. Rev. D 101, no.9, 095023 (2020)

[arXiv:2002.04616 [hep-ph]].

[25]
V. Keus,

PoS CORFU2019, 059 (2020)

[arXiv:2003.02141 [hep-ph]].

[26]
V. Keus,

Phys. Rev. D 101, no.7, 073007 (2020)

[arXiv:1909.09234 [hep-ph]].

[27]
A. Ahriche, G. Faisel, S. Y. Ho, S. Nasri and J. Tandean,

Phys. Rev. D 92, no.3, 035020 (2015)

[arXiv:1501.06605 [hep-ph]].

[28]
I. P. Ivanov, V. Keus and E. Vdovin,

J. Phys. A 45, 215201 (2012)

[arXiv:1112.1660 [math-ph]].

[29]
V. Keus, S. F. King and S. Moretti,

JHEP 01, 052 (2014)

[arXiv:1310.8253 [hep-ph]].

[30]
D. I. Kaiser,

Phys. Rev. D 81, 084044 (2010)

[arXiv:1003.1159 [gr-qc]].

[31]
P. A. R. Ade et al. [Planck],

Astron. Astrophys. 594, A20 (2016)

[arXiv:1502.02114 [astro-ph.CO]].

[32]
E. Komatsu et al. [WMAP],

Astrophys. J. Suppl. 192, 18 (2011)

[arXiv:1001.4538 [astro-ph.CO]].

[33]
A. D. Linde,

Phys. Lett. B 108, 389-393 (1982)

[34]
K. R. S. Balaji, T. Biswas, R. H. Brandenberger and D. London,

Phys. Lett. B 595, 22-27 (2004)

[arXiv:hep-ph/0403014 [hep-ph]].

[35]
K. R. S. Balaji, T. Biswas, R. H. Brandenberger and D. London,

Phys. Rev. D 72, 056005 (2005)

[arXiv:hep-ph/0506013 [hep-ph]].