#  TextBind: Multi-turn Interleaved Multimodal Instruction-following in the
Wild

Huayang Li♡♠ Siheng Li♡♣11footnotemark: 1 Deng Cai♡11footnotemark: 1 Longyue
Wang♡ Lemao Liu♡ Taro Watanabe♠ Yujiu Yang♣ Shuming Shi♡

♡Tencent AI Lab ♠Nara Institute of Science and Technology ♣Tsinghua University
https://textbind.github.io
Equal Contribution. Work done during HL and SL’s internships at Tencent AI
Lab. Correspondence to DC (jcykcai@tencent.com).

###### Abstract

Large language models with instruction-following abilities have revolutionized
the field of artificial intelligence. These models show exceptional
generalizability to tackle various real-world tasks through their natural
language interfaces. However, their performance heavily relies on high-quality
exemplar data, which is often difficult to obtain. This challenge is further
exacerbated when it comes to multimodal instruction following. We introduce
TextBind, an almost annotation-free framework for empowering larger language
models with the multi-turn interleaved multimodal instruction-following
capabilities. Our approach requires only image-caption pairs and generates
multi-turn multimodal instruction-response conversations from a language
model. To accommodate interleaved image-text inputs and outputs, we devise
MIM, a language model-centric architecture that seamlessly integrates image
encoder and decoder models. We release our dataset, model, and demo to foster
future research in the area of multimodal instruction following.

##  1 Introduction

The field of artificial intelligence (AI) has experienced a significant
paradigm shift with the rise of large language models (LLMs). These models are
capable of processing a wide range of natural language processing (NLP)
applications through natural language interactions with users [27, 26].
Despite their remarkable performance, these models cannot process and generate
visual content.

Recently, a number of efforts have been made to augment LLMs with visual
perception and understanding abilities. Prior work uses template-based
instruction-following datasets for training [45, 9, 20]. These datasets
comprise a variety of classic computer vision (CV) tasks, e.g., object
detection, with each task being converted into an instructional format using a
handful of human-written natural language instructions. However, classic CV
tasks often represent manageable and focused abstractions or simplifications
of real-world tasks [25], they generally fall short in representing the true
variety and complexity of real-world tasks and capturing the lexical diversity
of human language. For example, most of them are single-turn inquiries about a
single input image, whereas a small fraction supports multi-turn textual
interactions or multiple image inputs. Consequently, the instruction-following
capabilities of models trained on these datasets remain limited in open-world
scenarios [44]. This is reminiscent of the early development of instruction
tuning in NLP, where public NLP tasks were eventually superseded by high-
quality, diverse open-world instruction data [29]. Nevertheless, collecting
such data for multimodal models can be extremely costly.

In this paper, we address the above challenge by introducing TextBind, an
almost annotation-free framework for augmenting LLMs with multi-turn
interleaved multimodal instruction-following capabilities. The main idea is to
represent images through their textual descriptions, e.g., captions, and
utilize an LLM to generate multi-turn instructions and responses. To ensure
the coherence and meaningfulness of the constructed multi-turn conversations,
we propose a series of strategies such as topic-aware image sampling and
human-in-the-loop refinement of in-context demonstrations. TextBind can
harvest large-scale datasets given the abundance of public image-caption
pairs.

Figure 1: Comparison among models trained on different datasets.  Dataset | Multi-turn |  Arbitrarily Interleaved | Multimodal |  Instruction Following |  Image Generation
---|---|---|---|---|---
Platypus | ✗ | ✗ | ✗ | ✓ | ✗
Flamingo | ✗ | ✓ | ✓ | ✗ | ✓
MultiInstruct | ✗ | ✗ | ✓ | ✓ | ✗
MiniGPT-4 | ✗ | ✗ | ✓ | ✓ | ✗
LLaVA | ✓ | ✗ | ✓ | ✓ | ✗
TextBind | ✓ | ✓ | ✓ | ✓ | ✓
Table 1: Comparison with previous datasets

TextBind provides examples of processing and generating arbitrarily
interleaved image-and-text content. To accommodate interleaved image-text
inputs and outputs, we devise MIM, a multimodal model that emphasizes the
reasoning abilities of LLMs and seamlessly integrates image encoder and
decoder models. Experiments show that MIM trained on TextBind can perform a
wide range of tasks, including composing engaging stories inspired by a set of
images (Figure 1), comparing the common and different parts in multiple images
(Figure 8(b)), explaining concepts with vivid images (Figure 7(a)), generating
long coherent stories with illustrations (Figure 6), etc. Most interestingly,
the core innovation of our model is its capability to naturally interact with
users. The comparison of TextBind and previous representative datasets is
shown in Table 1, accompanied with an illustration of the models trained on
different datasets in Figure 1. For instance, rather than requiring users to
supply the model with explicit descriptions of the desired image, our model
can spontaneously generate images in proper conversation contexts. We hope
TextBind serves as an initial step towards building AGI that can interact with
humans flexibly in different modalities and in broad real-world scenarios.

##  2 Related Work

#### Multimodal Datasets

Existing multimodal datasets can be broadly classified into two categories:
conventional tasks and generalist instruction tuning. Conventional vision-
language tasks include image captioning [8, 1, 48] and visually-grounded
question answering [14, 24, 37, 23, 51, 12, 13], which is originally
introduced for specific applications, and used to train and evaluate
traditional specialist models correspondingly.

Recently, there has been a growing interest in transforming conventional
datasets into a unified instructional format. For instance, MultiInstruct
[45], InstructBLIP [9], and M3IT [20] convert existing vision-language
datasets with handcrafted templates. This approach is reminiscent of the early
explorations on instruction tuning in NLP [42, 33], where language models are
fine-tuned on a mixture of existing NLP tasks phrased as instructions.
However, it has been reported that such instruction-tuned multimodal models
still generalize poorly to open-world scenarios [44]. This finding is also in
line with the observations in NLP [29], where template-based instruction
tuning is less effective than instruction tuning data collected from real-
world scenarios due to its restricted diversity. There are also some attempts
to convert the output of existing vision-language models into natural language
answers for constructing instruction tuning data [21, 52, 6].

Compared to existing instruction-tuning data, the examples in TextBind (1)
generally exhibits greater task and lexicon diversity; (2) typically involve
multiple images scattered throughout a multi-urn conversation; (3) support
multimodal output (image generation).

#### Multimodal Models

One straightforward way to augment existing LLMs with visual abilities is to
employ off-the-shelf vision models as external tools. That is, the LLM calls
expert vision models through their language interfaces for completing specific
visual tasks when needed, including VisualChatGPT [43], HuggingGPT [36], Cola
[7], X-GPT [53], MM-REACT [46], and ViperGPT [39]. However, these approaches
may suffer from cross-modal information loss and lack of generality.

Recently, end-to-end multimodal language models has garnered significant
interest. Flamingo [2] is a pioneering work on extending LLMs to vision-
language pretraining by inserting additional cross-attention layers for visual
input. OpenFlamingo [2] is an open-source effort to replicate Flamingo.
Different from training from scratch, subsequent research efforts have focused
on integrating pretrained vision and language models. BLIP-2 [19] proposes
Qformer to map the features extracted from vision models to the hidden space
of language models. To date, various network architectures and training
strategies have been proposed to bridge different combinations of vision and
language models, including MiniGPT-4 [52], LLaVA [21], mPLUG-Owl [47], Otter
[18], LLaMA-Adapters[50], VisualGLM [10], Shikra [6], and InstructBLIP [9].
However, these models are limited to use visual content as input. Our work is
inspired by recent work on LLM-empowered image retrieval or generation [17,
16]. Contrary to prior work, we aim to present the first instruction-following
model capable of processing and generating arbitrarily interleaved image-text
inputs and outputs.

#### Evaluation

Conventional vision datasets designed for specific tasks and scenarios may
suffer from data contamination issues for evaluating LLMs. Recently, efforts
have been made to provide systematic evaluations with a broader coverage of
diverse visual abilities and real-world scenarios. MME [11] is an evaluation
dataset containing visually-grounded Yes/No questions. OwlEval [47] is a
benchmark comprising 82 questions based on 50 images and relies on human
feedback evaluation to address the open-ended nature of many questions. The
test size is limited and the results may suffer from subjective bias. In
response to these challenges, MMbench [22] and MM-Vet [49] are two recent
benchmarks aiming to offer more comprehensive evaluations by incorporating the
use of ChatGPT/GPT4 for answer verification. LVLM Arena [44], an online
evaluation framework that ranks different models using human judgment, is also
introduced. However, the above benchmarks primarily focus on question
answering based on a single image at the beginning of a conversation.

##  3 TextBind

In this work, we seek to enhance the multi-turn instruction-following
capabilities of a language model in the context of arbitrarily interleaved
images and text. Constructing such datasets poses significant challenges: 1)
it demands inventive thinking for devising high-quality visually-grounded
instructions and their responses; 2) it requires specialized expertise to
craft appropriate images. To tackle these issues, we introduce TextBind, a
method that predominantly resorts to existing text-only language
models111Although OpenAI claims that GPT4 supports visual input, this feature
is yet to be made public. to produce the desired data.

###  3.1 Definition of Data

The goal of TextBind is to construct a collection of multi-turn conversation
such as
$[\boldsymbol{x}^{1}_{u},\boldsymbol{x}^{1}_{a},\ldots,\boldsymbol{x}^{T}_{u},\boldsymbol{x}^{T}_{a}]$,
where $T$ is the number of turns, $\boldsymbol{x}^{i}_{u}$ denotes the $i$-th
instruction from the user, and $\boldsymbol{x}^{i}_{a}$ represents the $i$-th
response from the assistant. The conversation is also accompanied with an
image set $\\{\boldsymbol{m}_{1},\ldots,\boldsymbol{m}_{n}\\}$, where $n$ is
the number of unique images in this conversation. Each instruction
$\boldsymbol{x}^{i}_{u}$ or response $\boldsymbol{x}^{i}_{a}$ is a sequence of
tokens in $\mathcal{V}_{\text{lang}}\cup\mathcal{V}_{\text{img}}$, where
$\mathcal{V}_{\text{lang}}$ is the ordinary vocabulary of a language model and
$\mathcal{V}_{\text{img}}$ contains $n$ distinct pointers to the images
$\boldsymbol{m}_{1},\ldots,\boldsymbol{m}_{n}$ respectively. It is worth
noting that every image can appear at any point within the conversation.

Figure 2: Illustration of the TextBind method. In the top-left corner, we
display five representative images from each of three example clusters,
obtained via unsupervised clustering. On the right-hand side, a conversation
is showcased, constructed using two randomly sampled images from the cartoon
cluster. In the bottom-left corner, we outline the additional TextBind
pipeline, which include human-in-the-loop refinement and post-processing
stages.

###  3.2 Automatic Data Generation

Our method capitalizes on the powerful language understanding and generation
abilities of LLMs as effective annotators. However, LLM cannot "see". To
circumvent the problem, we use image captions as textualized proxies of images
and perform data construction entirely in language space. This flexibility
allows for the creation of arbitrarily interleaved text-and-image content.

Specifically, TextBind consists of a three-step pipeline: 1) topic-aware image
sampling for ensuring the coherence of each constructed conversation and the
diversity across all conversations; 2) LLM-empowered multi-turn instruction-
response generation to create natural and practical conversations; 3) post-
processing and filtering to eliminate low-quality data. An overview of the
TextBind pipeline can be found in Figure 2.

#### Topic-Aware Image Sampling

The initial step in our method, TextBind, entails assembling groups of images
that will serve as the foundation for generating multi-turn instructions and
responses. In order to facilitate coherent, meaningful, and practical
conversations, it is expected that the images within each group exhibit
meaningful interconnections. Furthermore, to guarantee a comprehensive
representation of diverse real-world scenarios, the topics of images across
different conversations should demonstrate a wide range of diversity.

Following the above inspirations, we employ unsupervised clustering algorithms
to group the images in our dataset into clusters and execute a two-step image
sampling process for each conversation. Concretely, we use the image encoder
of the CLIP model [31] to extract the semantics of images. Each image is
represented by a vector obtained through average pooling over the output of
the image encoder. Subsequently, we execute the $k$-means algorithm to
classify all images into $K$ clusters. We find that each cluster can be
considered a topic. Examples of such clusters are given in Figure 2. To
construct each conversation, we first randomly sample a cluster from the
available $K$ clusters, followed by sampling $n$ images from the chosen
cluster.

#### Generation of Multi-turn Conversation

After selecting a list of images, we proceed to leverage a text-only LLM, such
as GPT-4, to simulate a conversation between a user and an assistant based on
the chosen images. The core idea is to let LLMs receive and process textual
descriptions of images as if they see the actual images adhere to the
descriptions. Given the abundance of publicly available image-caption pairs,
we propose representing an image with an XML-like string <imgX> DESCRIPTION
</imgX>, where DESCRIPTION serves as a placeholder for the image caption,
<imgX> and </imgX> mark the caption boundaries, and X denotes the image index
in the input image list. After generating the conversation, we replace the
XML-like strings in the conversation with the original images. Importantly, to
ensure that a caption faithfully describes its corresponding image, we employ
the CLIP model [31] to filter out image-caption pairs with matching scores
below a high threshold.

The detailed prompt can be found in Appendix A, and examples of generated
conversations before mapping the textual descriptions back to visual images
are shown in Appendix B. In the prompt, we also provide in-context examples to
improve the generation quality. We collect the in-context examples through a
human-in-the-loop refinement process, which is elaborated in Section 3.3. By
using our conversation generation strategy, the percentage of "Poor" data
annotated by humans declines from $30\%$ to $9\%$. More details about the
quality of generated data are in Section 4.

#### Post-processing and Low-quality Filtering

We present two crucial rules to exclude low-quality conversations. First, the
LLM may alter image descriptions during the generation process. Since changes
in descriptions might result in a semantic mismatch between the image and the
description, we compute the edit distance between the original descriptions
and those in the conversation and filter out conversations with an edit
distance higher than $0.1$. Second, we exclude conversations containing image
descriptions not present in the provided image list. Besides those errors, we
also filter out instances that contain formatting errors such as co-reference
errors and invalid image tags.

Annotation | Labels |  Description
---|---|---
Overall Quality | Excellent |  This conversation is very interesting, practical, or intricate.
Satisfactory |  This conversation is coherent and reasonable without any factual errors.
Poor |  At least one turn in the conversation is unreasonable in some aspects, e.g., unrealistic content, illegal formats, etc.
Characteristic | Image Creation |  To create new images in appropriate contexts.
Image Comparison |  To combine, relate, or compare the information in different images.
Intrinsic Image Understanding |  To identify and recognize the objects, colors, shapes, and patterns in images.
Extrinsic Image Understanding |  To interpret the underlying meaning of images, e.g., the context, emotions, symbolism, or narrative conveyed by the images. It goes beyond merely recognizing the elements in the images and often requires external knowledge and/or deep analysis.

Table 2: Human annotation guideline.

###  3.3 Human-in-the-loop Refinement

In-context learning has been demonstrated to be crucial for enhancing the
generation quality of LLMs [3, 41]. Therefore, we also construct a seed set
$\mathcal{S}$ of high-quality in-context examples. The seed set $\mathcal{S}$
begins as an empty set and is iteratively updated with human feedback. In each
iteration, we follow the steps detailed below:

  1. 1.

We employ the latest $\mathcal{S}$ and the template in Appendix A, and
generate 100 new conversations using TextBind (Section 3).

  2. 2.

We manually analyze the generated conversations. Each conversation is assigned
a quality label ("Excellent", "Satisfactory" or "Poor"). Besides, we also
label the characteristics of each conversation based on the visual abilities
it requires. The annotation guideline is outlined in Table 2. We also follow
the same guideline and perform a comprehensive analysis on the final dataset
we construct in Section 4.

  3. 3.

We add the generated conversations with "Excellent" or "Satisfactory" labels
to $\mathcal{S}$.

Statistics |
---|---
# of conversations | $25,629$
Avg. # turns in conversations | $3.36$
Avg. # images |
in conversations | $2.46$
in instructions | $0.94$
in responses | $1.52$
Avg. # words |
in conversations | $285.90$
in instructions | $78.66$
in responses | $207.24$
Table 3: Statistics of the constructed dataset by applying TextBind to GPT-4.

To ensure diversity among different conversations, for each conversation, we
randomly sample three in-context examples from the seed set with the following
constraints. First, we require that at least one of the sampled in-context
examples be labeled "Excellent". Second, the three sampled examples should
encompass all four characteristic labels. These constraints are designed to
offer valuable in-context guidance for diverse examples. After three
iterations, we fix the seed set and employ it to generate instructions and
responses for the remaining data.

##  4 TextBind Data from GPT4

We apply TextBind to GPT4 as a case study. In this section, we conduct
comprehensive analyses of the constructed dataset.

###  4.1 Implementation Details

We construct our TextBind dataset based on the Conceptual Captions 3M (CC3M)
[34, 4] dataset, which only provides image-caption pairs. In our experiments,
we employ the clip-vit-base-patch16 model released by
OpenAI222https://huggingface.co/openai/clip-vit-base-patch16 [31] to filter
out image-caption pairs with matching scores lower than $30$. We use the
$k$-means clustering algorithm implemented by FAISS [15] toolkit to classify
the cleaned CC3M dataset into 4096 clusters. The features used for $k$-means
clustering are the hidden representations of images encoded by clip-vit-base-
patch16 model. In addition, clusters with less than $32$ images are regarded
as outliers and will not be considered. The number of images, hyper-parameter
$n$, in each conversation is sampled from $\\{2,3,4\\}$. We access the GPT-4
model through the OpenAI API333https://openai.com/blog/openai-api, and set
top_p and temperature hyper-parameters to $1.0$.

(a) Distribution of quality labels.

(b) Distribution of characteristic labels.

Figure 3: Distribution of human annotated data. The percent of those labels
are evaluated by humans on 100 randomly sampled conversations as described in
section 4. Note that a multi-turn conversation may have multiple
characteristics.

Dataset | Instruct | Response | Overall
---|---|---|---
LLaVA | $1.56$ | $1.84$ | $1.70$
MiniGPT-4 | $0.00$ | $1.11$ | $0.89$
MultiInstruct | $0.51$ | $1.69$ | $0.51$
Platypus | $0.98$ | $0.75$ | $0.78$
Shikra | $0.89$ | $1.08$ | $0.87$
TextBind | $\mathbf{1.76}$ | $\mathbf{1.92}$ | $\mathbf{1.84}$

Table 4: Averaged diversity scores of roles in various datasets. The diversity
scores are defined in section 4.3 and averaged across multiple turns.

###  4.2 Statistics

As depicted in Table 3, our constructed dataset comprises $25,629$
conversations. In this dataset, each turn is defined as a pair of instruction
and response. The average number of turns per conversation is $3.36$.
Additionally, the mean number of images in each conversation is $2.46$.

###  4.3 Diversity

In this section, we examine the diversity of the constructed dataset from
three perspectives: data characteristics, text, and images.

Diverse data characteristics may equip a model with various capabilities.
Therefore, understanding the types and distribution of those data
characteristics is essential. As demonstrated in Table 2, we identify four
types of data characteristics in our constructed dataset. In Figure 3(b), we
also display the characteristic distribution of 100 data randomly sampled from
the constructed dataset. We observe that TextBind method forces a significant
portion of conversations in the dataset focusing on more insightful and
informative characteristics, such as extrinsic understanding and image
comparison.

We further investigate the diversity of generated text, including both
instructions and responses. The diversity of text in a dataset is defined as
$\sum_{n=2}^{4}(\frac{\\#\text{Unique n-grams}}{\\#\text{Total n-grams}})$, in
line with previous works [38]. As shown in Table 4, our dataset achieves
better diversity scores than most existing visual-language datasets,
indicating that the language used in our dataset is more informative.

Error Type | Percentage
---|---
Img-Cap Mismatch | $0.03$
Incoherence | $0.03$
Hallucination | $0.03$
Table 5: Error types and percentage in dataset constructed by TextBind.

Additionally, we also study the diversity of the topics and positions of
images appearing in the conversation. In Figure 2, we display five randomly
sampled images from multiple randomly sampled clusters. The implicit
connections between the sampled images in each cluster enable the GPT-4 model
to construct coherent and meaningful discussions between humans and the
assistant. Furthermore, we examine the diversity of image positions within the
conversation. As depicted in Figure 4, most images in instructions and
responses appear within the first three turns.

Figure 4: The averaged number of images in each turn. A turn is defined as a
pair of instruction and response.

###  4.4 Quality

TextBind uses several strategies to enhance data quality, such as building in-
context examples with humans in the loop, filtering out image-caption pairs
with low matching scores, and post-processing. To gain a deeper understanding
of the data quality in our constructed dataset, we perform a comprehensive
error analysis in this section.

We conduct the quality evaluation on the same set of randomly sampled
conversations as in section 4.3. As shown in Figure 3(a), only $9\%$
conversations in the dataset are labeled with “Poor” quality. Notably, for a
multi-turn conversation, we will label the whole conversation as “Poor” if any
of its turns has a problem. Therefore, we also conduct an in-depth analysis of
the reasons for the “Poor” conversations. In Table 5, we identify three
typical errors present in the constructed dataset. Despite setting a high
threshold to filter out mismatched image-caption pairs, some mismatched cases
cannot be detected by the CLIP model [31]. A few conversations suffering from
incoherence and hallucinations may be attributed to the GPT-4 model. Overall,
while a small number of conversations are affected by errors that are
difficult to detect using rules, the majority of generated conversations
exhibit high quality. In Appendix C, we present several cases labeled with
“Poor”. We can find that most of those “Poor” cases only have minor and non-
obvious problems.

##  5 Augmenting LLMs with Visual I/O

###  5.1 Model

To support multi-turn interleaved multimodal instruction-following abilities,
we supplement large language models with visual input and output modules.
Specifically, LLama2-Chat444https://huggingface.co/meta-llama/Llama-2-7b-chat-
hf [40] is employed as the backbone language model. For visual input, we use
the vision encoder from
BLIP2555https://huggingface.co/Salesforce/blip2-flan-t5-xxl, our vision
encoder and Q-Former are initialized from BLIP-2 [19]. The features extracted
from the vision model are mapped into the embedding space of the language
model through the pretrained Q-Former model [19]. The core challenge lies on
the ability to produce images. Inspired by GILL [16], we attempt to learn a
mapping from the output space of the language model to the input space of a
stable diffusion (SD) model [32] (in this work, the embeddings produced by the
text encoder of Stable Diffusion XL [30]). To this end, we explore three model
variants in our preliminary experiments.

  * •

Q-Former as Medium. We add a special token <IMG> to the vocabulary of the
language model, indicating an image should be generated when it is emitted. We
then use a Q-Former [19] that takes all previous hidden states of the language
model as input and outputs the SD embeddings.

  * •

Q-Former with Prompt Tokens as Medium. To further leverage the reasoning
abilities of the language model, we incorporate a series of special tokens
(<IMG1>, …, <IMG{r}>), instead of a single token (<IMG>), to the language
model. When <IMG1> is emitted, the generation of the special token sequence is
enforced, serving as additional reasoning steps for predicting the forthcoming
image. Subsequently, the Q-Former only accepts the hidden states of special
tokens as input.

  * •

Language Description as Medium. The previous two variants try to align the
continuous hidden spaces of different models. An alternative is to use
discrete language descriptions for information exchange, as depicted in Figure
5. Specifically, we add two special tokens, <start> and <end>, and encode the
generated text between these two tokens using the text encoder in the SD
model.

Similar to GILL [16], we optimize the first two variants by minimizing the
mean squared error (MSE) loss between the output embeddings and the SD
embeddings. For the third variant, we employ the standard cross-entropy loss.
Empirically, we find that the first two methods generalize poorly and only the
third method demonstrates satisfactory performance. We hypothesize that this
disparity arises from the one-to-many nature of image generation in real-world
conversations. That is, unlike the conventional text-to-image model that
generates images from the explicit descriptions, there can exist more image
candidates that are suitable for a given conversation context in the solution
space. Operating in the hidden space may inadvertently average all
possibilities, resulting in the generation of ambiguous or noisy images. On
the other hand, the last method mitigates the one-to-many issue by taking full
advantage of the autoregressive generation of language models for decision-
making. We name the last variant MIM for its strong performance on multi-turn
interleaved multimodal instruction-following.

Figure 5: The architecture of MIM. It integrates a vision model, a language
model, and a stable diffusion model. MIM is able to process multi-turn
interleaved multimodal inputs and outputs.

###  5.2 Training

Our training process consists of two stages, the multimodal alignment stage
and the multimodal instruction tuning stage. The first stage aims to align the
feature spaces of the vision model and language models. The second stage
focuses on training the joint model on multimodal instruction tuning data for
improving its instruction following capabilities.

#### Multimodal Alignment

We utilize massive image-caption pairs for training, drawing from datasets
such as Conceptual Captions [5, 35] and SBU [28]. During training, only the
Q-Former connecting the vision and language models is optimized while other
model components remain frozen.

#### Multimodal Instruction Following

The model is further trained on our TextBind dataset. The Q-Former model and
LLM are optimized in this stage.

##  6 Demonstrations

The four high-level characteristics of the TextBind dataset equips MIM with a
variety of capabilities. We demonstrate those capabilities with concrete user
cases.

#### Image Creation

One core innovation of TextBind is that it enables the model to create images
based on the conversation context without explicit dictations from the users.
This characteristic is extremely useful for open-world scenarios, because in
many cases people may just have an implicit intention and have no clear
thoughts about what the images should be. We observe that our model can
explain concepts and ideas for users with vivid images (Figure 7(a)), creating
images with correct emotions (Figure 7(b)), and editing images based on the
whole context (Figure 7(c) and 7(d)). Furthermore, as shown in Figure 6, we
discover that our model is proficient in generating long stories featuring
interleaved text and images while maintaining exceptional coherence.

#### Image Comparison

Another interesting feature of TextBind is that it can compare or relate the
information in multiple images. For example, our model can correctly explain
the different and common parts between images in Figure 8.

#### Intrinsic & Extrinsic Image Understanding

The model trained on TextBind can understand the content in images precisely
in a multi-turn conversation. In all the three sub-figures of Figure 9, the
model precisely follows the human instructions and explains the details of the
images to users. Moreover, TextBind also enables the model to explore the
meaning of an image beyond the symbols in it. For example, the model also
explains the the influence of Bob Dylan’s album in Figure 9(b) and the impact
of iPhone in Figure 9(c).

Figure 6: Generation of a long story with interleaved text and images.

(a) Explaining concepts with multiple images.

(b) Creating images with correct emotions.

(c) Editing images based on context.

(d) Creating images based on context.

Figure 7: User cases of creating images.

(a) Comparing music styles.

(b) Relating images.

(c) Comparing movies.

(d) Comparing different concepts.

Figure 8: User cases of comparing images.

(a)

(b)

(c)

(d)

Figure 9: User cases of understanding both intrinsic & extrinsic information
in the images.

##  7 Limitations

Despite the amazing instruction-following capabilities demonstrated by our
models trained on TextBind, there are many avenues for further improvement.

  1. 1.

Our best-performing models use textual descriptions as a medium for image
generation. Nevertheless, it is impossible to describe everything in an image
through words. Thus, our current models have difficulties completing tasks
such as precise image editing. Investigating fine-grained multimodal feature
combinations may enhance performance in these areas.

  2. 2.

The multimodal capabilities of TextBind can be further enriched by
incorporating additional modalities. For example, future work may use audio-
text pairs for data synthesis.

  3. 3.

New benchmarks to evaluate the performance of multi-turn interleaved
multimodal instruction-following capabilities are demanded.

  4. 4.

Similar to existing multimodal language models, models trained on TextBind
exhibit some common deficiencies such as hallucination, toxicity, and
stereotypes.

## References

  * Agrawal et al. [2019] Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. 2019.  Nocaps: Novel object captioning at scale.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 8948–8957.
  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Changpinyo et al. [2021a] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021a.  Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _CVPR_.
  * Changpinyo et al. [2021b] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021b.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021_ , pages 3558–3568. Computer Vision Foundation / IEEE.
  * Chen et al. [2023a] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. 2023a.  Shikra: Unleashing multimodal llm’s referential dialogue magic.  _arXiv preprint arXiv:2306.15195_.
  * Chen et al. [2023b] Liangyu Chen, Bo Li, Sheng Shen, Jingkang Yang, Chunyuan Li, Kurt Keutzer, Trevor Darrell, and Ziwei Liu. 2023b.  Language models are visual reasoning coordinators.  In _ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models_.
  * Chen et al. [2015] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015.  Microsoft coco captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_.
  * Dai et al. [2023] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  _arXiv preprint arXiv:2305.06500_.
  * Du et al. [2022] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022.  Glm: General language model pretraining with autoregressive blank infilling.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 320–335.
  * Fu et al. [2023] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. 2023.  Mme: A comprehensive evaluation benchmark for multimodal large language models.  _ArXiv_ , abs/2306.13394.
  * Goyal et al. [2017] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6904–6913.
  * Gurari et al. [2018] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. 2018.  Vizwiz grand challenge: Answering visual questions from blind people.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3608–3617.
  * Hudson and Manning [2019] Drew A Hudson and Christopher D Manning. 2019.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 6700–6709.
  * Johnson et al. [2019] Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019.  Billion-scale similarity search with GPUs.  _IEEE Transactions on Big Data_ , 7(3):535–547.
  * Koh et al. [2023a] Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. 2023a.  Generating images with multimodal language models.  _arXiv preprint arXiv:2305.17216_.
  * Koh et al. [2023b] Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. 2023b.  Grounding language models to images for multimodal inputs and outputs.  _ICML_.
  * Li et al. [2023a] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. 2023a.  Otter: A multi-modal model with in-context instruction tuning.  _arXiv preprint arXiv:2305.03726_.
  * Li et al. [2023b] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_.
  * Li et al. [2023c] Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, et al. 2023c.  M3it: A large-scale dataset towards multi-modal multilingual instruction tuning.  _arXiv preprint arXiv:2306.04387_.
  * Liu et al. [2023a] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023a.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_.
  * Liu et al. [2023b] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2023b.  Mmbench: Is your multi-modal model an all-around player?  _arXiv preprint arXiv:2307.06281_.
  * Lu et al. [2022] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022.  Learn to explain: Multimodal reasoning via thought chains for science question answering.  _Advances in Neural Information Processing Systems_ , 35:2507–2521.
  * Marino et al. [2019] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019.  Ok-vqa: A visual question answering benchmark requiring external knowledge.  In _Proceedings of the IEEE/cvf conference on computer vision and pattern recognition_ , pages 3195–3204.
  * Marr [2010] David Marr. 2010.  _Vision: A computational investigation into the human representation and processing of visual information_.  MIT press.
  * OpenAI [2022a] OpenAI. 2022a.  Gpt-4 technical report.
  * OpenAI [2022b] OpenAI. 2022b.  Introducing chatgpt.
  * Ordonez et al. [2011] Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. 2011.  Im2text: Describing images using 1 million captioned photographs.  In _Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12-14 December 2011, Granada, Spain_ , pages 1143–1151.
  * Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Podell et al. [2023] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. 2023.  Sdxl: Improving latent diffusion models for high-resolution image synthesis.  _arXiv preprint arXiv:2307.01952_.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8748–8763. PMLR.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695.
  * Sanh et al. [2021] Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021.  Multitask prompted training enables zero-shot task generalization.  _arXiv preprint arXiv:2110.08207_.
  * Sharma et al. [2018a] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018a.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2556–2565, Melbourne, Australia. Association for Computational Linguistics.
  * Sharma et al. [2018b] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018b.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers_ , pages 2556–2565. Association for Computational Linguistics.
  * Shen et al. [2023] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023.  Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface.  _arXiv preprint arXiv:2303.17580_.
  * Singh et al. [2019] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019.  Towards vqa models that can read.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 8317–8326.
  * Su et al. [2022] Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. 2022.  A contrastive framework for neural text generation.  In _Advances in Neural Information Processing Systems_.
  * Surís et al. [2023] Dídac Surís, Sachit Menon, and Carl Vondrick. 2023.  Vipergpt: Visual inference via python execution for reasoning.  _arXiv preprint arXiv:2303.08128_.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Wang et al. [2023] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023.  Self-instruct: Aligning language models with self-generated instructions.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 13484–13508, Toronto, Canada. Association for Computational Linguistics.
  * Wei et al. [2021] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021.  Finetuned language models are zero-shot learners.  _arXiv preprint arXiv:2109.01652_.
  * Wu et al. [2023] Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. 2023.  Visual chatgpt: Talking, drawing and editing with visual foundation models.  _arXiv preprint arXiv:2303.04671_.
  * Xu et al. [2023a] Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. 2023a.  Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models.  _arXiv preprint arXiv:2306.09265_.
  * Xu et al. [2023b] Zhiyang Xu, Ying Shen, and Lifu Huang. 2023b.  MultiInstruct: Improving multi-modal zero-shot learning via instruction tuning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 11445–11465, Toronto, Canada.
  * Yang et al. [2023] Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. 2023.  Mm-react: Prompting chatgpt for multimodal reasoning and action.  _arXiv preprint arXiv:2303.11381_.
  * Ye et al. [2023] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. 2023.  mplug-owl: Modularization empowers large language models with multimodality.  _arXiv preprint arXiv:2304.14178_.
  * Young et al. [2014] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  _Transactions of the Association for Computational Linguistics_ , 2:67–78.
  * Yu et al. [2023] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023.  Mm-vet: Evaluating large multimodal models for integrated capabilities.  _arXiv preprint arXiv:2308.02490_.
  * Zhang et al. [2023] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. 2023.  Llama-adapter: Efficient fine-tuning of language models with zero-init attention.  _arXiv preprint arXiv:2303.16199_.
  * Zhou et al. [2018] Luowei Zhou, Chenliang Xu, and Jason Corso. 2018.  Towards automatic learning of procedures from web instructional videos.  In _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * Zhu et al. [2023] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_.
  * Zou et al. [2022] Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. 2022.  Generalized decoding for pixel, image, and language.  _arXiv preprint arXiv:2212.11270_.

##  Appendix A Prompt of TextBind

GPT-4 Prompt Please construct a dialogue between a human and a helpful, honest
and harmless assistant. The dialogue contains interleaved text and images.
Each image is represented by <imgX> DESCRIPTION </imgX>, where DESCRIPTION is
a textual description of the image and X is an index of the image. Please do
not assume any further visual information beyond the description. The
constructed dialogues must and can only contain the following input images:
<img0> museum - the 1st nuclear submarine </img0> <img1> response to the
production of heavy </img1> Characteristics about the assistant:
1\. The assistant is trained to understand text, images, and their
combinations. 2\. The assistant can reply the human with images and/or text.
3\. The assistant has exceptional world knowledge and commonsense reasoning
capabilities. 4\. The assistant does not have access to the Internet or any
other external tools. 5\. If the assistant is asked to create an image, it can
only show the image in the provided image list. 6\. Please do not copy the
images appearing in the dialogue. The assistant should refer to the previously
mentioned image by natural language. Characteristics about the human:
1\. The human may send images and/or text to the assistant. 2\. The human may
ask questions requiring visual reasoning and/or understanding the relations
between multiple images. 3\. The human may ask the assistant to show images
based on his/her intention. 4\. The human may ask the assitant to do
interesting things, rather than simply describing the content of the image.
Properties of a bad dialogue:
1\. Simply describing or analyzing the content in the image. 2\. Dialogue
without a good logic. Properties of a good dialogue:
1\. Introducing extrinsic and attractive information of stuff shown in the
image. 2\. Discovering the connection between multiple images. 3\. The
dialogue happens in a reasonable context. Examples:
======== Case 1: Input Images: … Output Dialogue: … ——— Case 2: Input Images:
… Output Dialogue: … ——— Case 3: Input Images: … Output Dialogue: … ======
Please directly give the dialogue if you understand. The number of turns of
the dialogue should be less than 6. The dialogue should be self-contained. Do
NOT assume any previous dialogue between the human and the assistant. Please
use the same format <imgX> DESCRIPTION </imgX> to denote images in the
dialogue and do not modify the description of the image. Note that the
dialogue can stop at any time without an explicit ending.

Figure 10: Illustration of the prompt used for TextBind.

##  Appendix B Examples of Constructed Conversations

GPT-4 Ouput Example #1
Human: Hi! I’m trying to write a cartoon story for my kids featuring a grumpy
clown and a lovable cupcake. Can you help me craft this? Assistant: Of course!
That sounds like a wonderful idea. We can start by establishing our
characters. The grumpy clown could be someone who struggles finding joy in his
own performances. <img0> a cartoon illustration of a clown looking angry
</img0> Human: That’s a wonderful idea! How about the cute cupcake? Assistant:
The cupcake, on the other hand, is always upbeat and tries to help the clown
find his happiness. Despite being a tiny cupcake, it has a big heart and a
contagious joy. <img1> cartoon illustration of a cupcake with a happy
expression </img1> Human: Interesting! What should be their adventure?
Assistant: They could go on a journey to find the ultimate joke that
guarantees laughter. The lovable cupcake, with his relentless positivity and
cheer, helps the grumpy clown see the lighter side of life and finally find
his joy. Example #2
Human: Hi, I am researching small business operations. Can you show me an
example of informal small business? Assistant: Of course. Here’s an example of
informal small business: <img0> a mother and daughter selling gum and
cigarettes in person </img0>. This concept of business is most commonly found
in rural or underdeveloped areas where formal establishments might not be as
prevalent. Human: Interesting! On a somewhat related note, what sells well in
such arrangements? Assistant: It depends on the location and the demand.
Mostly, such businesses sell products of daily use such as food items, candy,
toiletries etc. For example an ice cream cart is a common sight in many areas,
as it’s a popular treat among people of all ages. Human: Can you show me an
image of a kid indulging in something like that? Assistant: Absolutely! Here’s
a delightful image of <img1> a child eating ice cream </img1>. Ice cream and
similar snacks are often popular products for small informal businesses.
Example #3
Human: Hi there! I’m preparing a speech on unity and cooperation in global
issues. I have a few images but I’m not sure how to describe them in a way
that fits my theme. Assistant: I’d be happy to help with that! Could you show
me the images? Human: Sure, here they are. <img0> large group of people in the
shape of flag </img0> and <img1> rear view of a male boxer holding globe with
flag painted on his back </img1> Assistant: The first image can be seen as a
symbol of unity. Here, numerous individuals come together to form the shape of
a flag, showing that collective action can lead to a powerful representation,
like a flag - often a symbol of identity and unity. The second image
illustrates the idea of global responsibility and commitment. The male boxer,
suggesting strength and determination, has a globe – representing the world –
in his hands, which could indicate a keen sense of global responsibility. The
flag painted on his back can symbolize allegiance or commitment to his
country, and by extension to global harmony if we’re expanding the theme to
encompass international cooperation. Human: That sounds great! I also have
this image. Not sure if it fits. <img2> diplomatic handshake between countries
: flags overprinted the hands stock photo </img2> Assistant: This image very
much fits your theme. It can represent a formal agreement or cohesion between
countries. The flags overprinted on the shaking hands signify the countries
involved in the meet. This image can be used to illustrate diplomatic dialogue
or mutual understanding which is an integral component to solving global
issues through cooperation and unity.

Figure 11: Illustration of in-context examples of TextBind.

##  Appendix C Constructed Conversations with “Poor” Label

(a) Hallucination Case: The topic is about rain boot. However, not all the
boots shown in the second image are rain boots.

(b) Incoherence Case: The first turn discusses about relaxing while the second
turn is about outdoor activities. There are no connections.

(c) Image-caption Mismatch Case: The first image only shows a bathroom, but
the caption is “double room with private bathroom in a cottage”.

Figure 12: Constructed conversations with “Poor” Label. The caption is shown
below the image with gray color.