# Large Language Models Meet Computer Vision; A Brief Survey

Raby Hamadi
Artificial Intelligence Department, Tahakom

<EMAIL_ADDRESS>

###### Abstract

Recently, the intersection of Large Language Models (LLMs) and Computer Vision
(CV) has emerged as a pivotal area of research, driving significant
advancements in the field of Artificial Intelligence (AI). As transformers
have become the backbone of many state-of-the-art models in both Natural
Language Processing (NLP) and CV, understanding their evolution and potential
enhancements is crucial. This survey paper delves into the latest progressions
in the domain of transformers and their subsequent successors, emphasizing
their potential to revolutionize Vision Transformers (ViTs) and LLMs. This
survey also presents a comparative analysis, juxtaposing the performance
metrics of several leading paid and open-source LLMs, shedding light on their
strengths and areas of improvement as well as a literature review on how LLMs
are being used to tackle vision related tasks. Furthermore, the survey
presents a comprehensive collection of datasets employed to train LLMs,
offering insights into the diverse data available to achieve high performance
in various pre-training and downstream tasks of LLMs. The survey is concluded
by highlighting open directions in the field, suggesting potential venues for
future research and development. This survey aims to underscores the profound
intersection of LLMs on CV, leading to a new era of integrated and advanced AI
models.

_K_ eywords Large Language Models $\cdot$ Vision Transformer $\cdot$ Retention
Networks

##  1 Introduction

The field of AI is witnessing a great fusion of CV and NLP. At the middle of
this exciting combination are LLMs, which have come to be a key participant in
AI. These models, that were first created to understand and produce human
language, are actually expanding to encompass the interpretation of visual
data. This shows how flexible and adaptable LLMs are, as they begin to combine
the understanding of text with seeing and interpreting visual data. LLMs,
along with CV are set to revolutionize the way machines understand and
describe the visible world. This fusion of LLMs and CV is leading to an era
where AI systems can see the world, understand it, and communicate about it
nearly like humans do. By bringing together these technologies, we can make
the way humans and machines interact smoother and more natural. For example,
in security, this could mean systems that not only see something unusual but
also explain it clearly, helping to deal with threats faster. In healthcare,
combining LLMs and computer vision could lead to better diagnosis and
treatment by joining visual information with a wide range of medical
knowledge. In retail, AI could watch over inventory levels and use LLMs to
make useful reports and predictions. In manufacturing, these technologies
could improve how we check for product quality by spotting defects with
computer vision and using LLMs to give detailed advice on fixing them. Looking
ahead, the fusion of LLMs and computer vision is set to be a landmark in AI’s
progression, bringing us closer to creating highly autonomous digital
assistants that can engage with the world comprehensively. This new era of AI
envisions machines as proactive participants, interpreting and narrating the
visual world around them.

A pivotal advancement in the fields of NLP and CV was the invention of the
transformer architecture [1], which was crucial at managing long sequences in
NLP tasks through the innovative attention mechanism [1]. This mechanism
allows the model to weigh the importance of different segments of input data.
The breakthrough came when researchers stated that if transformers could
interpret language as sequences of words, they might also process images as
sequences of pixel patches. This insight gave rise to VT, a step forward in CV
methodologies. In our survey, we aim to present an exhaustive exploration of
LLMs as they venture into the realm of CV. We will delve into the nuances of
transformer architectures and their successors, highlighting their role in
shaping an integrated AI future. Our literature review will meticulously cover
recent works and groundbreaking research that have charted the course for LLMs
in visual applications. A comparative analysis will illuminate the strengths
and potential enhancements of both paid and open-source models. Additionally,
we will explore the diverse datasets that are used in the literature to train
these models. The survey is then concluded by some of the challenges that face
LLMs and some open research directions.

##  2 Background

###  2.1 RNNs

Recurrent Neural Networks (RNNs) [2] are a class of artificial neural networks
designed for processing sequences of data. Unlike traditional feedforward
neural networks, RNNs possess loops that allow information to be passed from
one step in the sequence to the next, making them inherently suited for tasks
where temporal dynamics and context from earlier steps are crucial. The
foundational idea behind RNNs is the concept of memory. At each step in a
sequence, an RNN takes in a new input and the previous hidden state (or
memory) to produce a new hidden state and an output. This recurrent nature
allows RNNs to maintain a form of "memory" about previous steps in the
sequence, enabling them to capture patterns over time.

Figure 1: Dual form of RetNet. “GN” is short for GroupNorm [3].

###  2.2 Transformers

Introduced in 2017 by Vaswani et al. [4], Transformers are a type of deep
learning models that excel at handling sequential data, making them
particularly suitable for natural language processing tasks. The key
breakthrough in Transformers lies within their self-attention mechanism [5],
which allows the model to assess the significance of different segments within
an input sequence relative to one another. This unique capability enables the
model to capture complex patterns and dependencies over long distances in the
data. Unlike conventional RNNs, Transformers process input information
concurrently rather than sequentially, resulting in notable enhancements in
both training efficiency and scalability. Thus, it enables researchers to
build large models with billions of parameters that can generate long coherent
and contextual stories, answer questions accurately, and execute tasks that
require logic and understanding. While they are efficient during training due
to their ability to handle parallelization, one of their drawbacks is that
they can be slow during inference. This is because the self-attention
mechanism in Transformers requires computation over all elements of the input
sequence, leading to a quadratic increase in computation as the sequence
length grows. This can be particularly challenging for real-time applications
or scenarios where quick response times are essential.

###  2.3 RetNet

The Retentive Network (RetNet) [3] emerges as a foundational architecture for
large language models, offering a unique blend of training parallelism, cost-
effective inference, and robust performance. At its core lies the retention
mechanism tailored for sequence modeling, which seamlessly integrates three
computational paradigms: parallel, recurrent, and chunkwise recurrent. This
multifaceted approach ensures that the parallel representation fosters
training parallelism, the recurrent representation guarantees low-cost
inference, enhancing decoding throughput, latency, and GPU memory efficiency
without compromising on performance. Furthermore, the chunkwise recurrent
representation is adept at modeling long sequences with linear complexity,
encoding each chunk in parallel but summarizing them in a recurrent manner.
These distinctive features position RetNet as a potential successor to the
Transformer architecture for large language models. The retention mechanism
treats an 1D input sequence in a recurrent, unidirectional manner as shown in
Eq 1.

$o_{n}=\sum_{m=1}^{n}\gamma^{n-m}(Q_{n}e^{in\theta})(K_{m}e^{im\theta})^{\dagger}v_{m},$
(1)

where $\dagger$ is the conjugate transpose. To allow parallel training, the
above equation 1 is written as follows given an input $X$:

$Q=(XW_{Q})\odot\Theta,\quad K=(XW_{K})\odot\bar{\Theta},\quad
V=XW_{V},\quad\Theta_{n}=e^{in\theta},\quad
D_{nm}=\begin{cases}\gamma^{n-m},&\text{if }n\geq m\\\ 0,&\text{if
}n<m\end{cases},$ (2) $Retention(X)=(QK^{T}\odot D)V$ (3)

where $\bar{\Theta}$ is the complex conjugate of $\Theta$ and
$D\in\mathrm{R}^{|x|\times|x|}$ combines causal masking and exponential decay
along relative distance as one matrix. The recurrent and parallel
representations of the RetNet block is shown in Fig 1.

Figure 2: RetNet-based Vision Transformer [6]

###  2.4 Vision Transformers

ViTs [7] represent a significant shift in the field of computer vision.
Traditionally, Convolutional Neural Networks (CNNs) [8] have been the dominant
architecture for image-related tasks, leveraging their ability to process
spatial hierarchies in images. However, the success of the Transformer
architecture in natural language processing led researchers to explore its
potential in the domain of vision. ViTs approach image tasks by dividing
inputs into fixed-size patches, linearly embedding them, and then processing
them as a sequence using the Transformer architecture. The self-attention
mechanism in Transformers, which had been pivotal in capturing long-range
dependencies in text, proved equally adept at handling spatial relationships
in images.

###  2.5 RetNet-Based Vision Transformers

As previously discussed, RetNet is mainly designed for 1D data and it treats
the inputs in an unidirectional way. Efforts have been made [6], as shown in
Fig. 2, to extrapolate this to the 2D space in order to be able to handle
images and treat them in a bidirectional manner. The bidirectional extension
of retention in Eq. 1 for each token can be written as follows:

$o_{n}=\sum_{m=1}^{N}\gamma^{|n-m|}(Q_{n}e^{in\theta})(K_{m}e^{im\theta})^{\dagger}v_{m},$
(4)

where $\dagger$ is the conjugate transpose, and thus the parallel version is
written as:

$BiRetention(X)=(QK^{T}\odot D^{B_{i}})V,\quad\text{where}\quad
D^{B_{i}}_{nm}=\gamma^{|n-m|}$ (5)

Now, to extrapolate to the two-dimensional space, let’s denote by
$(x_{n},y_{n})$ the coordinate of the $n$th token of an image. The $D$ matrix
is modified to contain the Manhattan distance between token pairs as follows:

$D^{2d}_{nm}=\gamma^{|x_{n}-x_{m}|+|y_{n}-y_{m}|}$ (6)

and finally, the 2D retention can be written as:

$ReSA(x)=(Softmax(QK^{T})\odot D^{2d})V$ (7)

###  2.6 Vision-Language Models (VLMs)

A visual language model is an AI-driven mechanism tailored to interpret and
derive insights from visual content like images and videos. By training on
vast collections of visual content, these models become adept at identifying
and categorizing various visual components, from objects to individuals to
entire scenes. VLMs have evolved significantly over the years, transitioning
from the use of hand-crafted image descriptors and pre-trained word vectors to
the adoption of advanced transformer architectures for both image and text
encoding. Central to these models are three core components: an image encoder,
a text encoder, and a fusion mechanism to integrate information from both
domains as shown in Fig 3. The contemporary approach emphasizes strategic pre-
training objectives to enhance transfer performance across various tasks.
Among the notable pre-training strategies are:

  * •

Contrastive Learning: Aligns images and texts in a joint feature space.

  * •

PrefixLM: Uses images as a prefix to a language model, enabling joint learning
of image and text embeddings.

  * •

Multi-modal Fusing with Cross Attention: Integrates visual data into language
model layers.

  * •

MLM/ITM: Focuses on aligning specific image parts with corresponding text
using masked-language modeling and image-text matching.

  * •

No Training approach: Leverages standalone vision and language models through
iterative optimization.

Figure 3: Vision-Language Models (VLMs) [9]

##  3 Related Work

In this survey [10], the authors presents a deep comprehensive background of
LLMs, their scaling laws, and their emergent capabilities. The authors also
focus on the evolution of GPT-series models and discuss their capabilities in
details. Moreover, the authors discuss some available resources, datasets,
available checkpoints, and APIs to help develop and train LLMs. In [11] the
authors offer an extensive examination of LLMs, which are at the forefront of
AI advancements, particularly in the realm of natural language processing. The
historical progression of LLMs is charted, detailing their evolution and the
various training methodologies that have been employed to enhance their
capabilities. A broad spectrum of LLM applications is explored, spanning
sectors such as healthcare, education, finance, and engineering, illustrating
the transformative impact LLMs have on these fields. Furthermore, the paper
addresses the challenges that come with implementing LLMs in practical
settings. These challenges include ethical dilemmas, inherent model biases,
issues with interpretability, and the substantial computational resources
required for their operation. It also sheds light on strategies to improve LLM
robustness and control, as well as methods to mitigate biases and enhance the
quality of generated content. However, the authors of the previously
mentionned surveys do not specify how these LLMs can be used in the field of
computer vision.

In [12] the authors provide a critical examination of evaluation methods for
LLMs, addressing the ’what’, ’where’, and ’how’ of LLM assessment. It covers
the range of tasks LLMs are tested on, including language processing,
reasoning, and applications in various fields like medicine and education. The
paper also explores the environments and benchmarks used for evaluation, and
the methodologies employed to assess performance, highlighting both successes
and failures of LLMs. The aim is to offer a comprehensive understanding of LLM
capabilities and guide future evaluation strategies. In [13], the authors
provide a comprehensive overview of the significant strides made in the domain
of Visual-Language Pretrained Models (VLPMs), which have emerged as a powerful
approach for learning joint representations of visual and linguistic content.
The paper begins by defining the general tasks and the overarching
architecture of VLPMs, then delves into the specifics of language and vision
data encoding methods. It also examines the core structures of mainstream
VLPMs, detailing the nuances of their design and function. Essential
strategies for both pretraining and fine-tuning these models are summarized,
offering a clear roadmap for how they are developed and optimized for various
tasks. However, a limitation of this paper is that it does not encompass the
very latest advances in NLP, particularly the next generation of transformer
models. As the field of NLP is advancing rapidly, with new models and
techniques continually emerging, any overview can quickly become outdated.
This paper, while thorough in its current scope, may not capture the most
cutting-edge developments that have occurred since its writing, which could be
crucial for researchers looking to build upon the most recent and advanced
work in the field.

##  4 Latest LLMs

Large language models, are cutting-edge AI systems that can grasp and produce
human-like language. Drawing inspiration from the intricacies of the human
brain, they’re built on intricate neural networks, such as transformer models.
By learning from massive amounts of data, these models can understand context,
making their text outputs feel more natural, whether they’re answering your
questions or spinning a story. In essence, a large language model is a
sophisticated AI buddy, designed to chat and understand just like humans.
While they exhibit significant potential, not all are straightforward in their
implementation. LLMs, even when optimally configured, struggle with tasks
necessitating precise adherence to instructions. LlamaIndex [14] provides
compatibility with a vast majority of LLMs. However, it remains ambiguous
whether a specific LLM will operate efficiently immediately upon integration
or if additional adjustments are imperative. The subsequent tables endeavor to
assess the preliminary experience associated with diverse LlamaIndex
functionalities across various LLMs. These evaluations aim to measure
performance and determine the extent of modifications required to ensure
optimal functionality. Typically, commercial APIs, such as OpenAI, are
perceived as more dependable. Nonetheless, local open-source models are
increasingly favored due to their adaptability and commitment to transparency.
A comparison of various open-source and paid LLMs is shown in Table 2 and
Table 1.

Model Name |  Basic
Query
Engines |  Router
Query
Engine |  Sub
Question
Query
Engine |  Text2SQL |  Pydantic
Programs |  Data
Agents
---|---|---|---|---|---|---
gpt-3.5-turbo |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$
gpt-3.5-turbo-instruct |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\bigtriangleup$
gpt-4 |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$
claude-2 |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\bigtriangleup$
claude-instant-1.2 |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\bigtriangleup$
Legend: $\checkmark$ \- Supported, •\- Not Supported, $\bigtriangleup$\-
Partially Supported
Table 1: Paid LLMs Benchmark [14] Model Name |  Basic
Query
Engines |  Router
Query
Engine |  Sub
Question
Query Engine |  Text2SQL |  Pydantic
Programs |  Data
Agents
---|---|---|---|---|---|---
llama2-chat-7b 4bit |  $\checkmark$ |  • |  • |  • |  • |  $\bigtriangleup$
llama2-13b-chat |  $\checkmark$ |  $\checkmark$ |  • |  $\checkmark$ |  • |  •
llama2-70b-chat |  $\checkmark$ |  $\checkmark$ |  ✓ |  $\checkmark$ |  • |  $\bigtriangleup$
Mistral-7B-instruct-v0.1 4bit |  $\checkmark$ |  • |  • |  $\bigtriangleup$ |  $\bigtriangleup$ |  $\bigtriangleup$
zephyr-7b-alpha |  $\checkmark$ |  ✓ |  ✓ |  $\checkmark$ |  ✓ |  $\bigtriangleup$
zephyr-7b-beta |  $\checkmark$ |  $\checkmark$ |  ✓ |  $\checkmark$ |  • |  $\checkmark$
Legend: $\checkmark$ \- Supported, •\- Not Supported, $\bigtriangleup$\-
Partially Supported
Table 2: Open-Source LLMs Benchmark [14]

###  4.1 Applications of LLMs

In [15], the authors introduce MiniGPT-v2, a model designed to serve as a
unified interface for various vision-language tasks such as image description,
visual question answering, and visual grounding. The primary challenge
addressed is the use of a single model to effectively perform diverse vision-
language tasks using multi-modal instructions. To achieve this, the authors
propose the use of unique identifiers for different tasks during training.
These identifiers help the model distinguish between task instructions and
improve its learning efficiency. The model undergoes a three-stage training
process. Experimental results demonstrate that MiniGPT-v2 outperforms other
vision-language generalist models in various benchmarks. The architecture of
MiniGPT-v2 includes a visual backbone, a linear projection layer, and a large
language model. To reduce ambiguity across tasks, the authors introduce task-
specific tokens in their multi-task instruction template. The model also
represents spatial locations using textual formatting of bounding boxes.

Zhao et al. [16], introduces LAVILA, a new approach to learning video-language
representations by leveraging Large Language Models. The paper repurpose pre-
trained LLMs to be conditioned on visual input and fine-tune them to create
automatic video narrators. The auto-generated narrations offer advantages such
as dense coverage of long videos, better temporal synchronization of visual
information and text, and higher diversity of text. The video-language
embedding learned with these narrations outperforms previous state-of-the-art
on multiple video tasks. LAVILA obtains significant gains in classification
and retrieval benchmarks. The method shows positive scaling behavior with
increasing pre-training data and model size.

Doveh et al. [17], address the challenge of enhancing VLMs’ understanding of
complex language structures. They introduce the concept of Structured Vision
and Language Concepts (SVLC), which encompasses object attributes, relations,
and states present in both text and images. The novel contribution of this
work is a data-driven approach that leverages the inherent structure of
language. By manipulating the textual components of existing VLM datasets, the
authors emphasize SVLCs in VLM training. They generate alternative negative or
positive texts, which instruct the model about SVLCs. They utilize LLMs for
"unmasking" tasks, where given a sentence with a missing word, the model
suggests contextually appropriate alternatives. This property is employed to
create plausible negative examples, enhancing the training data for VLMs.
Additionally, by prompting LLMs, the authors generate text analogies—sentences
that convey similar meanings but with varied wording. This method of
structured textual data manipulation, powered by LLMs, aims to emphasize the
importance of SVLC in VLM training, leading to significant improvements in the
model’s comprehension of these concepts. Through evaluations on multiple
datasets, the approach demonstrates improvements of up to 15% in the VLMs’
understanding of SVLCs, while ensuring minimal degradation in their zero-shot
object recognition capabilities.

Yining et Al. [18], made a significant push towards integrating 3D spatial
understanding into LLMs. Their work introduced a 3D-LLM framework, designed
explicitly for training LLMs to comprehend and respond to 3D scenes and
objects. This framework ingeniously extracts 3D features from 2D multi-view
images and combines them with language prompts to generate relevant responses.
One of the primary challenges addressed by the authors was the difficulty in
training 3D-LLMs from scratch, primarily due to the absence of a comprehensive
3D-language dataset. To circumvent this, they proposed a method that leverages
pre-trained image encoders to extract features from 2D images and map these to
the 3D domain. These mapped 3D features are then seamlessly integrated into
pre-trained 2D vision-language models, which subsequently serve as the
foundational architecture for the 3D-LLMs. Furthermore, the study employed
ChatGPT [19] to generate a diverse range of 3D-language data, emphasizing its
ability to form holistic 3D descriptions from multiple image views. The
results from this research underscored the potential of 3D-LLMs, showcasing
their capability to generate answers closely aligned with ground-truth data
and perform intricate visual reasoning tasks. This innovative approach not
only bridges the existing gap between 2D image understanding and 3D spatial
reasoning but also sets a promising direction for future advancements in AI-
driven 3D scene comprehension.

Zhenfei et al., [20] introduce an advancement in multi-modal large language
models (MLLMs) by incorporating point cloud data, presenting the LAMM-Dataset
and LAMM-Benchmark specifically designed for 2D image and 3D point cloud
understanding. These resources are intended to support high-level vision tasks
across both dimensions. They validate the effectiveness of their dataset and
benchmark through extensive experiments. Furthermore, they detail the process
of creating instruction-tuning datasets and benchmarks for MLLMs, aiming to
expedite future research and expansion into new domains, tasks, and
modalities. The paper also introduces a novel training framework for MLLMs
that is optimized for the inclusion of additional modalities. Alongside this
framework, they provide baseline models as well as comprehensive experimental
observations and analyses to serve as a foundation for accelerating subsequent
research in the field.

In this paper [21], the authors address the challenge of product understanding
in the context of online shopping, a task that involves responding to a wide
range of queries based on multi-modal product information. Unlike traditional
methods that use different model architectures for each sub-task, the authors
introduce PUMGPT, a unified vision-language model designed to handle all
aspects of product understanding within a single model structure. To
effectively combine visual and textual data, they propose a novel technique
called Layer-wise Adapters (LA), which aligns vision and text representations
more efficiently with fewer visual tokens and allows for parameter-efficient
fine-tuning. This fine-tuning capability enables PUMGPT to adapt quickly to
new tasks and products. The authors also create diverse product instruction
datasets using instruction templates and enhance the model’s performance and
generalization by incorporating open-domain datasets in the training process.
PUMGPT is thoroughly evaluated and shows superior performance across a variety
of product understanding tasks, including product captioning, category
question-answering, attribute extraction, attribute question-answering, and
free-form question-answering about products.

In this work [22], the authors tackle the limitations of vision foundation
models (VFMs) in computer vision, which, despite their power, lack the open-
ended task flexibility of large language models (LLMs). They introduce
VisionLLM, a novel LLM-based framework designed to handle vision-centric
tasks. VisionLLM treats images as if they were a foreign language, aligning
vision tasks with language tasks that can be defined and manipulated through
language instructions. This allows for a high degree of task customization
using an LLM-based decoder that interprets these instructions to make
predictions for a variety of open-ended tasks. The authors demonstrate through
extensive experiments that VisionLLM can handle a range of customizations,
from detailed object-level to broader task-level adjustments, with impressive
results. Remarkably, the model achieves over 60% mean Average Precision (mAP)
on the COCO dataset, which is competitive with models dedicated to detection
tasks. The authors aim for VisionLLM to establish a new standard for
generalist models that integrate vision and language.

##  5 Datasets

A suite of datasets was introduced to enhance the training and evaluation of
Large Language Models (LLMs). These datasets target various functionalities of
LLMs such as API interaction, reasoning, extended context comprehension, and
language understanding. A concise overview of the latest dataset is provided
below:

  * •

LAMM-Dataset [20]: A diverse multi-modal dataset featuring 186K pairs of
language-image instructions and responses, alongside 10K language-3D pairings,
sourced from 8 image and 4 point cloud datasets. It includes four distinct
types of multi-modal instruction-response pairings.

  * •

function_calling_extended: A dataset consisting of English code pairs,
designed to refine the API usage capabilities of LLMs. It is a human-created
dataset focused on improving the interface between language models and coding
queries.

  * •

AmericanStories [23]: This dataset is a comprehensive corpus from the US
Library of Congress for pretraining LLMs, enriching them with a wealth of
American cultural and historical narratives.

  * •

dolma [24, 25]: Employed for OLMo pretraining, the dolma dataset encompasses a
3 trillion token repository, aimed at providing a robust, diverse corpus for
foundational language model training.

  * •

Platypus [26]: Part of the Platypus2 series, this dataset with 25,000 English
entries is tailored for enhancing STEM reasoning capabilities within LLMs,
fostering their ability to process and reason with scientific and technical
information.

  * •

Puffin: The Redmond-Puffin Series is a dialog dataset with about 3,000
entries, characterized by its long conversation contexts and multi-turn
dialogue structure, essential for training LLMs in sustaining coherent,
context-aware interactions.

  * •

tiny_series: This dataset contains concise codes or texts in English and is
devised to sharpen an LLM’s problem-solving and reasoning processes through
targeted, precise examples.

  * •

LongBench [27]: Serving as an evaluative benchmark, LongBench comprises 17
tasks in both English and Chinese, testing the long-context understanding and
retention in LLMs.

  * •

orca-chat [28]: This dataset, consisting of 198,463 English dialog entries, is
an Orca-style dialog dataset aimed at improving LLM’s long-context
conversational ability.

  * •

DialogStudio [29]: A collection of diverse multilingual datasets aimed at
building conversational Chatbots.

  * •

chatbot_arena_conversations [30]: With 33k multilingual dialog conversations,
this dataset collected cleaned conversations with pairwise human preferences
on Chatbot Arena. It is used for Reinforcement Learning with Human Feedback
(RLHF).

  * •

WebGLM-qa [31]: Used by WebGLM, this English pairs dataset contains 43.6k
entries. It is a QA system based on LLM and the Internet where each entry
comprises a question, a response, and a reference, with the response being
grounded in the reference.

  * •

phi-1 [32]: An English dialog dataset generated using the method outlined in
"Textbooks Are All You Need," focusing on math and computer science problems.

  * •

Linly-pretraining-dataset: A Chinese pretraining dataset used by the Linly
series model, it comprises 3.4GB of data from ClueCorpusSmall, CSL news-crawl,
and more.

  * •

FineGrainedRLHF [33]: Containing approximately 5,000 English examples, this
repository aims to develop a new framework to collect human feedback to
improve LLMs in aspects such as factual correctness and topic relevance.

  * •

dolphin: With 4.5 million English pair entries, this dataset attempts to
replicate Microsoft’s Orca and is based on FLANv2.

  * •

openchat_sharegpt4_dataset [34]: A high-quality English dialog dataset with 6k
dialogs generated using GPT-4 to complete refined ShareGPT prompts, used by
OpenChat.

Each dataset is uniquely curated to address specific training objectives,
significantly contributing to the progression of language models in handling
complex tasks and bolstering their overall proficiency. More datasets can be
found in Table 3

Dataset Description Size SBU [35] Images from Flicker with captions 1M COCO
Caption [36] Images from MS COCO with two versions (c5 and c40) 330k YFCC100M
[37] Multimedia dataset with images and videos 100M VG [38] Images with
object-level info, scene graphs, and etc. 108k CC3M [39] Image-text pairs from
the web 3.3M CC12M [40] Image-text pairs for VLM pretraining 12M LR [41] Image
captioning with local multi-modal annotations 848,749 WIT [42] Multi-modal
multilingual dataset from Wikipedia 37.6M Red Caps [43] Image-text pairs from
Reddit 12M LAION400M [44] Image-text pairs filtered by CLIP 400M LAION5B [45]
Image-text pairs in multiple languages 5.8B WuKong [46] Chinese multi-modal
dataset 100M CLIP [47] Web image-text dataset 400M ALIGN [48] Noisy image-text
pairs 1.8B FILIP [49] Image-text pairs from the internet 300M WebLI [50]
Multilingual image-text dataset from the web 10B

Table 3: Summary of Image-Text Datasets

##  6 LLMs and VLMs: challnges and future research directions

##  7 LLMs

Designing Artificial General Intelligence (AGI) Benchmarks: Developing
benchmarks for AGI necessitates a deep exploration of the distinctions between
human and machine intelligence. Human intelligence, with its nuances,
adaptability, and context sensitivity, differs significantly from machine
intelligence, which excels in specific tasks but often lacks adaptability and
context awareness. To create comprehensive tests, these benchmarks must assess
an AGI’s ability to generalize knowledge across domains, reason abstractly,
and learn dynamically in new, unseen scenarios. These tests should measure an
AGI’s adaptability, reasoning prowess, and capacity for learning on the fly.
Moreover, these benchmarks should not merely aim for superhuman performance in
isolated tasks but should also evaluate a balanced spectrum of abilities akin
to human intelligence. This includes assessing emotional understanding,
ethical decision-making, and the nuanced judgment that characterizes human-
like intelligence. Crafting benchmarks that encapsulate these multi-
dimensional aspects is crucial in evaluating AGI’s depth, adaptability, and
nuanced understanding compared to human intelligence.

Complete Behavioral Evaluation: Evaluating language models like LLMs solely on
narrow tasks has indeed limited our understanding of their true capabilities.
Shifting the focus towards holistic assessments, such as integrating LLMs with
physical environments through robot control, marks a significant stride in
understanding their adaptability and real-world functionality. By
orchestrating a scenario where the LLM navigates and interacts in a physical
setting, we delve into its capacity to process diverse streams of
information—from visual and auditory cues to tactile feedback. This
multifaceted evaluation tests the model’s ability to synthesize various
modalities seamlessly, enabling it to respond dynamically to unforeseen
circumstances. It not only gauges the LLM’s linguistic prowess but also sheds
light on its practical application, providing a richer and more comprehensive
view of its behavioral capabilities in complex, unstructured environments.

Robustness Evaluation: Assessing the adaptability of language models to
diverse inputs like different dialects, slang, and varying grammar structures
is a multifaceted challenge. First, creating comprehensive datasets that
encapsulate this linguistic diversity is pivotal. These datasets must
encompass a wide array of linguistic variations, capturing colloquialisms,
regional nuances, and grammatical deviations. Secondly, devising evaluation
metrics becomes crucial in measuring a model’s robustness across these
variations. Metrics should focus on consistency in understanding and
generating outputs, evaluating not just accuracy but also coherence and
contextual relevance. Developing these metrics requires a deep understanding
of linguistic nuances and the ability to quantify these qualitative aspects
objectively. Ultimately, this concerted effort aims to enhance the model’s
adaptability to the dynamic nature of human language, ensuring it performs
reliably across diverse linguistic landscapes.

Dynamic and Evolving Evaluation: The rapid advancement of language models
poses a challenge in evaluating their true capabilities. The static
benchmarks, while useful at a point in time, tend to become outdated as models
evolve. Creating evaluation protocols that dynamically adapt is crucial to
ensure that these models are not merely memorizing data but are consistently
learning and adapting to new challenges. By constantly presenting them with
novel and unseen tasks, we can gauge their capacity to generalize knowledge,
understand context, and apply reasoning skills. This dynamic evaluation
approach fosters a deeper understanding of how well these models grasp the
essence of language, encouraging continuous improvement and pushing the
boundaries of their learning capabilities. It also enables researchers and
developers to identify areas for enhancement and refinement, ensuring that
these language models progress in a meaningful and sustainable manner, rather
than stagnating with static benchmarks that fail to capture their evolving
potential.

Unified Evaluation for All LLM Tasks: LLMs indeed face a formidable challenge
in catering to an extensive range of tasks, each with its distinct set of
demands. One crucial aspect lies in creating an evaluation framework that can
seamlessly adapt to this diversity. This framework must offer a consistent
method to gauge performance, regardless of whether the task involves
straightforward question answering or delves into the intricate realms of
complex reasoning and creative generation. The complexity inherent in these
tasks necessitates an evaluation system that not only acknowledges but
effectively measures the nuances inherent in simple, direct responses as well
as the intricacies demanded by abstract thinking, logical inference, and the
generation of novel, imaginative content. Crafting a unified evaluation
framework capable of accommodating this breadth of tasks is fundamental for
comprehensively assessing the capabilities of language models, offering
insights into their strengths across diverse domains and guiding their
continual enhancement and refinement.

Beyond Evaluation: LLMs Enhancement: Evaluations are not just a final
assessment but a crucial tool for continual enhancement in LLMs development.
They serve as the compass guiding improvements, not merely by flagging
weaknesses but by dissecting the intricacies behind robust or vulnerable
responses. Understanding the reasons behind these outcomes provides a roadmap
for enhancement. By delving into the ’why’ behind both strengths and
weaknesses, developers can pinpoint the underlying mechanisms governing
language understanding. This insight is pivotal—it informs the iterative
process, directing focus towards refining specific areas rather than a broad,
generalized overhaul. The integration of evaluation insights into the
development cycle enables a more targeted and effective approach, ultimately
leading to the evolution of more proficient and nuanced Language Models.

###  7.1 VLMs

Vision Language Interaction Modeling: The challenge here is to create a more
nuanced alignment between visual and linguistic representations. Current
models often use task-level or input-level masking, which obscures parts of
the data to encourage the model to learn from context. However, this does not
necessarily ensure that the model is aligning the underlying features of the
image with the corresponding textual features. The paper suggests that
embedding-level masking, where the model learns to predict masked features
directly within the embedding space, could be more effective. This approach
could lead to a more granular understanding of the relationships between
visual elements and their linguistic counterparts, but it requires careful
consideration of how to best implement such a strategy to truly enhance the
model’s representational capabilities.

VLM Pretraining Strategy: The pretraining phase for VLMs is crucial as it sets
the foundation for how well the model will perform on downstream tasks. The
paper points out that there is a lack of comprehensive research on how to
synergize various tasks during pretraining to benefit the model’s overall
performance. Multi-stage training, which involves sequentially training models
on different tasks or datasets, has been explored by few but could hold the
key to unlocking more effective VLMs. The challenge is multifaceted:
researchers must select the right mix of datasets, design tasks that
complement each other, and sequence these tasks in a way that builds upon
previous learning. This process is complex and requires a deep understanding
of how different tasks interact and how they can be layered to reinforce the
model’s learning. The ultimate goal is to develop a pretraining strategy that
is tailored to enhance performance on specific tasks or domains, which would
be a significant step forward for the field.

Training Evaluation: The current practice of evaluating VLMs primarily during
downstream tasks can be inefficient and costly. If a model has fundamental
flaws, these may only become apparent after significant computational
resources have been expended. The paper suggests that developing intermediate
metrics, akin to perplexity in language models, could provide early indicators
of a model’s potential performance. Such metrics would allow researchers to
make adjustments during the training process, rather than after the fact,
saving time and computational power. This proactive approach to evaluation
could help ensure that models are on the right track before they are fully
trained, but developing these metrics is challenging. They must be predictive
of downstream success and sensitive enough to guide the training process
effectively. This requires a deep understanding of what makes a model
successful and how that success can be measured incrementally as the model
learns.

##  8 Conclusion

In this survey, we effectively highlighted the significant convergence of LLMs
and CV, marking a transformative phase in the realm of AI. By focusing on the
evolution and advancements of transformers and their impact on ViTs and LLMs,
the survey underscores the groundbreaking potential in this area. The
comprehensive comparative analysis of various leading paid and open-source
LLMs provides valuable insights into their performance, strengths, and areas
needing improvement, while the discussion on the use of LLMs in vision-related
tasks adds depth to our understanding of their practical applications. The
detailed presentation of diverse datasets employed in training LLMs emphasizes
the importance of varied data in enhancing the performance of these models.
The survey not only serves as a rich resource for current state-of-the-art
practices in the field but also opens up new avenues for future research and
development.

## References

  * [1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  2017\.
  * [2] Alex Sherstinsky.  Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network.  Physica D: Nonlinear Phenomena, 404:132306, 2020.
  * [3] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei.  Retentive network: A successor to transformer for large language models.  arXiv preprint arXiv:2307.08621, 2023.
  * [4] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [5] Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun.  Exploring self-attention for image recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10076–10085, 2020.
  * [6] Qihang Fan, Huaibo Huang, Mingrui Chen, Hongmin Liu, and Ran He.  Rmt: Retentive networks meet vision transformers.  arXiv preprint arXiv:2309.11523, 2023.
  * [7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [8] Keiron O’Shea and Ryan Nash.  An introduction to convolutional neural networks.  arXiv preprint arXiv:1511.08458, 2015.
  * [9] contenteratechspace.  What is a visual language model, 2023.  Accessed: 2023-10-29.
  * [10] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al.  A survey of large language models.  arXiv preprint arXiv:2303.18223, 2023.
  * [11] Muhammad Usman Hadi, R Qureshi, A Shah, M Irfan, A Zafar, MB Shaikh, N Akhtar, J Wu, and S Mirjalili.  A survey on large language models: Applications, challenges, limitations, and practical usage.  TechRxiv, 2023.
  * [12] Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie.  A survey on evaluation of large language models, 2023.
  * [13] Siqu Long, Feiqi Cao, Soyeon Caren Han, and Haiqin Yang.  Vision-and-language pretrained models: A survey.  arXiv preprint arXiv:2204.07356, 2022.
  * [14] llamaindex.  llamaindex website, 2023.  Accessed: 2023-10-29.
  * [15] Jun Chen, Deyao Zhu1 Xiaoqian Shen1 Xiang Li, Zechun Liu2 Pengchuan Zhang, Raghuraman Krishnamoorthi2 Vikas Chandra2 Yunyang Xiong, and Mohamed Elhoseiny.  Minigpt-v2: Large language model as a unified interface for vision-language multi-task learning.  arXiv preprint arXiv:2310.09478, 2023.
  * [16] Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar.  Learning video representations from large language models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6586–6597, June 2023.
  * [17] Sivan Doveh, Assaf Arbelle, Sivan Harary, Eli Schwartz, Roei Herzig, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, and Leonid Karlinsky.  Teaching structured vision & language concepts to vision & language models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2657–2668, 2023.
  * [18] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan.  3d-llm: Injecting the 3d world into large language models.  arXiv preprint arXiv:2307.12981, 2023.
  * [19] Partha Pratim Ray.  Chatgpt: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope.  Internet of Things and Cyber-Physical Systems, 2023.
  * [20] Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Lu Sheng, Lei Bai, Xiaoshui Huang, Zhiyong Wang, et al.  Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark.  arXiv preprint arXiv:2306.06687, 2023.
  * [21] Shuhui Wu, Zengming Tang, Zongyi Guo, Weiwei Zhang, Baoliang Cui, Haihong Tang, and Weiming Lu.  Pumgpt: A large vision-language model for product understanding.  arXiv preprint arXiv:2308.09568, 2023.
  * [22] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al.  Visionllm: Large language model is also an open-ended decoder for vision-centric tasks.  arXiv preprint arXiv:2305.11175, 2023.
  * [23] Melissa Dell, Jacob Carlson, Tom Bryan, Emily Silcock, Abhishek Arora, Zejiang Shen, Luca D’Amico-Wong, Quan Le, Pablo Querubin, and Leander Heldring.  American stories: A large-scale structured text dataset of historical us newspapers.  arXiv preprint arXiv:2308.12477, 2023.
  * [24] Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Khyathi Chandu, Jennifer Dumas, Li Lucy, Xinxi Lyu, Ian Magnusson, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Pete Walsh, Hannaneh Hajishirzi, Noah A. Smith, Luke Zettlemoyer, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo.  Dolma: An Open Corpus of 3 Trillion Tokens for Language Model Pretraining Research.  Technical report, Allen Institute for AI, 2023.  Released under ImpACT License as Medium Risk artifact, https://github.com/allenai/dolma.
  * [25] Soldaini, Luca and Lo, Kyle and Kinney, Rodney and Naik, Aakanksha and Ravichander, Abhilasha and Bhagia, Akshita and Groeneveld, Dirk and Schwenk, Dustin and Magnusson, Ian and Chandu, Khyathi.  The Dolma Toolkit, 2023.  Apache 2.0 License, Version 0.9.0, https://github.com/allenai/dolma.
  * [26] Ariel N. Lee, Cole J. Hunter, and Nataniel Ruiz.  Platypus: Quick, cheap, and powerful refinement of llms.  2023\.
  * [27] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li.  Longbench: A bilingual, multitask benchmark for long context understanding, 2023.
  * [28] Shahul Es.  Orca-chat: A high-quality explanation-style chat dataset.  https://huggingface.co/datasets/shahules786/orca-chat/, 2023.
  * [29] Jianguo Zhang, Kun Qian, Zhiwei Liu, Shelby Heinecke, Rui Meng, Ye Liu, Zhou Yu, , Huan Wang, Silvio Savarese, and Caiming Xiong.  Dialogstudio: Towards richest and most diverse unified dataset collection for conversational ai, 2023.
  * [30] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.  Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
  * [31] Xiao Liu, Hanyu Lai, Hao Yu, Yifan Xu, Aohan Zeng, Zhengxiao Du, Peng Zhang, Yuxiao Dong, and Jie Tang.  Webglm: Towards an efficient web-enhanced question answering system with human preferences, 2023.
  * [32] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al.  Textbooks are all you need.  arXiv preprint arXiv:2306.11644, 2023.
  * [33] Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi.  Fine-grained human feedback gives better rewards for language model training.  arXiv preprint arXiv:2306.01693, 2023.
  * [34] Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu.  Openchat: Advancing open-source language models with mixed-quality data.  arXiv preprint arXiv:2309.11235, 2023.
  * [35] Vicente Ordonez, Girish Kulkarni, and Tamara Berg.  Im2text: Describing images using 1 million captioned photographs.  Advances in neural information processing systems, 24, 2011.
  * [36] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server. arxiv 2015\.  arXiv preprint arXiv:1504.00325, 2015.
  * [37] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li.  Yfcc100m: The new data in multimedia research.  Communications of the ACM, 59(2):64–73, 2016.
  * [38] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International journal of computer vision, 123:32–73, 2017.
  * [39] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018.
  * [40] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558–3568, 2021.
  * [41] Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari.  Connecting vision and language with localized narratives.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16, pages 647–664. Springer, 2020.
  * [42] Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork.  Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning.  In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2443–2449, 2021.
  * [43] Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson.  Redcaps: Web-curated image-text data created by the people, for the people.  arXiv preprint arXiv:2111.11431, 2021.
  * [44] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  arXiv preprint arXiv:2111.02114, 2021.
  * [45] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  Advances in Neural Information Processing Systems, 35:25278–25294, 2022.
  * [46] Jiaxi Gu, Xiaojun Meng, Guansong Lu, Lu Hou, Niu Minzhe, Xiaodan Liang, Lewei Yao, Runhui Huang, Wei Zhang, Xin Jiang, et al.  Wukong: A 100 million large-scale chinese cross-modal pre-training benchmark.  Advances in Neural Information Processing Systems, 35:26418–26431, 2022.
  * [47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [48] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In International conference on machine learning, pages 4904–4916. PMLR, 2021.
  * [49] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.  Filip: Fine-grained interactive language-image pre-training.  arXiv preprint arXiv:2111.07783, 2021.
  * [50] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al.  Pali: A jointly-scaled multilingual language-image model.  arXiv preprint arXiv:2209.06794, 2022.


# Mining Images in Biomedical Publications: Detection and Analysis of Gel
Diagrams

Tobias Kuhn  1 Mate Levente Nagy  Tobias Kuhn –<EMAIL_ADDRESS>3
ThaiBinh Luong  Mate Levente Nagy —<EMAIL_ADDRESS>2 and Michael
Krauthammer2,3 ThaiBinh Luong –<EMAIL_ADDRESS>Michael Krauthammer –
<EMAIL_ADDRESS>(1)Department of Humanities, Social and Political
Sciences, ETH Zurich, Switzerland. (2)Department of Pathology, Yale University
School of Medicine, New Haven, CT, USA. (3)Program for Computational Biology
and Bioinformatics, Yale University, New Haven, CT, USA.

###### Abstract

Authors of biomedical publications use gel images to report experimental
results such as protein-protein interactions or protein expressions under
different conditions. Gel images offer a concise way to communicate such
findings, not all of which need to be explicitly discussed in the article
text. This fact together with the abundance of gel images and their shared
common patterns makes them prime candidates for automated image mining and
parsing. We introduce an approach for the detection of gel images, and present
a workflow to analyze them. We are able to detect gel segments and panels at
high accuracy, and present preliminary results for the identification of gene
names in these images. While we cannot provide a complete solution at this
point, we present evidence that this kind of image mining is feasible.

## Introduction

A recent trend in the area of literature mining is the inclusion of images in
the form of figures from biomedical publications [1, 2, 3]. This development
benefits from the fact that an increasing number of scientific articles are
published as open access publications. This means that not just the abstracts
but the complete texts including images are available for data analysis. Among
other things, this enabled the development of query engines for biomedical
images like the Yale Image Finder [4] and the BioText Search Engine [5].
Below, we present our approach to detect and access gel diagrams. This is an
extended version of a previous workshop paper [6].

As a preparatory evaluation to decide which image type to focus on, we built a
corpus of 3 000 figures that allows us to reliably estimate the numbers and
types of images in biomedical articles. These figures were drawn randomly from
the open access subset of PubMed Central and then manually annotated. They
were split into subfigures when the figure consisted of several components.
Figure 1 shows the resulting categories and subcategories. This classification
scheme is based on five basic image categories: Experimental/Microscopy,
Graph, Diagram, Clinical and Picture, each divided into multiple
subcategories. It shows that bar graphs (12.4%), black-on-white gels (12.0%),
fluorescence microscopy images (9.4%), and line graphs (8.1%) are the most
frequent subfigure types (all percentages are relative to the entire set of
images).

Figure 1: Categorization of images from open access articles of PubMed
Central.

We targeted different kinds of graphs (i.e., diagrams with axes) in previous
work [7], and we decided to focus this work on the second most common type of
images: gel diagrams. They are the result of gel electrophoresis, which is a
common method to analyze DNA, RNA and proteins. Southern, Western and Northern
blotting [8, 9, 10] are among the most common applications of gel
electrophoresis. The resulting experimental artifacts are often shown in
biomedical publications in the form of gel images as evidence for the
discussed findings such as protein-protein interactions or protein expressions
under different conditions. Often, not all details of the results shown in
these images are explicitly stated in the caption or the article text. For
these reasons, it would be of high value to be able to reliably mine the
relations encoded in these images.

A closer look at gel images reveals that they follow regular patterns to
encode their semantic relations. Figure 2 shows two typical examples of gel
images together with a table representation of the involved relations. The
ultimate objective of our approach (for which we can only present a partial
solution here) is to automatically extract at least some of these relations
from the respective images, possibly in conjunction with classical text mining
techniques. The first example shows a Western blot for detecting two proteins
(14-3-3$\sigma$ and $\beta$-actin as a control) in four different cell lines
(MDA-MB-231, NHEM, C8161.9, and LOX, the first of which is used as a control).
There are two rectangular gel segments arranged in a way to form a $2\times 4$
grid for the individual eight measurements combining each protein with each
cell line. A gel diagram can be considered a kind of matrix with pictures of
experimental artifacts as content. The tables to the right illustrate the
semantic relations encoded in the gel diagrams. Each relation instance
consists of a condition, a measurement and a result. The proteins are the
entities being measured under the conditions of the different cell lines. The
result is a certain degree of expression indicated by the darkness of the
spots (or brightness in the case of white-on-black gels). The second example
is a slightly more complex one. Several proteins are tested against each other
in a way that involves more than two dimensions. In this case, the use of “+”
and “–” labels is a frequent technique to denote the different possible
combinations of a number of conditions. Apart from that, the principles are
the same. In this case, however, the number of relations is much larger. Only
the first eight of a total of 32 relation instances are shown in the table to
the right. In such cases, the text rarely mentions all these relations in an
explicit way, and the image is therefore the only accessible source.

|
---
| Condition | Measurement | Result
---|---|---
MDA-MB-231 |  14-3-3$\sigma$ | high expression
NHEM |  14-3-3$\sigma$ | no expression
C8161.9 |  14-3-3$\sigma$ | high expression
LOX |  14-3-3$\sigma$ | low expression
MDA-MB-231 |  $\beta$-actin | high expression
NHEM |  $\beta$-actin | high expression
C8161.9 |  $\beta$-actin | high expression
LOX |  $\beta$-actin | high expression
|
---
| Condition | Measurement | Result
---|---|---
IL-1$\beta$ (–) | DEX (–) | RU486 (–) | p-p38 | low expression
IL-1$\beta$ (+) | DEX (–) | RU486 (–) | p-p38 | high expression
IL-1$\beta$ (–) | DEX (+) | RU486 (–) | p-p38 | no expression
IL-1$\beta$ (+) | DEX (+) | RU486 (–) | p-p38 | low expression
IL-1$\beta$ (–) | DEX (–) | RU486 (+) | p-p38 | no expression
IL-1$\beta$ (+) | DEX (–) | RU486 (+) | p-p38 | high expression
IL-1$\beta$ (–) | DEX (+) | RU486 (+) | p-p38 | low expression
IL-1$\beta$ (+) | DEX (+) | RU486 (+) | p-p38 | high expression
… |  |  | … | …
Figure 2: Two examples of gel images from biomedical publications (PMID
19473536 and 15125785) with tables showing the relations that could be
extracted from them

## Background

In principle, image mining involves the same processes as classical literature
mining [11]: document categorization, named entity tagging, fact extraction,
and collection-wide analysis. However, there are some subtle differences.
Document categorization corresponds to image categorization, which is
different in the sense that it has to deal with features based on the two-
dimensional space of pixels, but otherwise the same principles of automatic
categorization apply. Named entity tagging is different in two ways:
pinpointing the mention of an entity is more difficult with images (a large
number of pixels versus a couple of characters), and OCR errors have to be
considered. Fact extraction in classical literature mining involves the
analysis of the syntactic structure of the sentences. In images, in contrast,
there are rarely complete sentences, but the semantics is rather encoded by
graphical means. Thus, instead of parsing sentences, one has to analyze
graphical elements and their relation to each other. The last process,
collection-wide analysis, is a higher-level problem, and therefore no
fundamental differences can be expected. Thus, image mining builds upon the
same general stages as classical text mining, but with some subtle yet
important differences.

Image mining on biomedical publications is not a new idea. It has been applied
for the extraction of subcellular location information [12], the detection of
panels of fluorescence microscopy images [13], the extraction of pathway
information from diagrams [14], and the detection of axis diagrams [7]. Also,
there is a large amount of existing work on how to process gel images [15, 16,
17, 18, 19] and databases have been proposed to store the results of gel
analyses [20]. These techniques, however, take as input plain gel images,
which are not readily accessible from biomedical papers, because they make up
just parts of the figures. Furthermore, these tools are designed for
researchers who want to analyze their gel images and not to read gel diagrams
that have already been analyzed and annotated by a researcher. Therefore,
these approaches do not tackle the problem of recognizing and analyzing the
labels of gel images. Some attempts to classify biomedical images include gel
figures [21], which is, however, just the first step in locating them and
analyzing their labels and their structure. To our knowledge, nobody has yet
tried to perform image mining on gel diagrams.

## Approach and Methods

Figure 3 shows the procedure of our approach to image mining from gel
diagrams. It consists of seven steps: figure extraction, segmentation, text
recognition, gel detection, gel panel detection, named entity recognition and
relation extraction.111Due to the fact that many figures consist of multiple
panels of different types, we go straight to gel segment detection without
first classifying entire images. Most gel panels share their figure with other
panels, which makes automated classification difficult at the image level.

Figure 3: The procedure of our approach: (1) figure extraction, (2)
segmentation, (3) text recognition, (4) gel detection, (5) gel panel
detection, (6) named entity recognition, and (7) relation extraction.

Using structured article representations, the first step is trivial. For steps
two and three, we rely on existing work. The main focus of this paper lies on
steps four and five: the detection of gels and gel panels. In the discussion
section, we present some preliminary results on step six of recognizing named
entities, and sketch how step seven could be implemented, for which we cannot
provide a concrete solution at this point.

To practically evaluate our approach, we ran our pipeline on the entire open
access subset of PubMed Central (though not all figures made it through the
whole pipeline due to technical difficulties).

### Figure Extraction

A large portion of the articles of the open access subset of the PubMed
Central database are available as structured XML files with additional image
files for the figures. We only use these articles so far, which makes the
figure extraction task very easy. It would be more difficult, though
definitely feasible, to extract the figures from PDF files or even bitmaps of
scanned articles (see [22] and http://pdfjailbreak.com for approaches on
extracting the structure of articles in PDF format).

### Segmentation and Text Recognition

For the next two steps — segment detection and subsequent text recognition —,
we rely on our previous work [23, 24]. This method includes the detection of
layout elements, edge detection, and text recognition with a novel pivoting
approach. For optical character recognition (OCR), the Microsoft Document
Imaging package is used, which is available as part of Microsoft Office 2003.
Overall, this approach has been shown to perform better than other existing
approaches for the images found in biomedical publications [23]. We do not go
into the details here, as this paper focuses on the subsequent steps.

Due to some limitations of the segmentation algorithm when it comes to
rectangles with low internal contrast (like gels), we applied a complementary
very simple rectangle detection algorithm.

### Gel Segment Detection

Based on the results of the above-mentioned steps, we try to identify gel
segments. Such gel segments typically have rectangular shapes with darker
spots on a light gray background, or — less commonly — white spots on a dark
background. We decided to use machine learning techniques to generate
classifiers to detect such gel segments. To do so, we defined 39 numerical
features for image segments: the coordinates of the relative position (within
the image), the relative and absolute width and height, 16 grayscale histogram
features, three color features (for red, green and blue), 13 texture features
(coarseness, presence of ripples, etc.) based on [25], and the number of
recognized characters.

To train the classifiers, we took a random sample of 500 figures, for which we
manually annotated the gel segments. In the same way, we obtained a second
sample of another 500 figures for testing the classifiers.222We double-checked
these manual annotations to check their quality, which revealed only four
misclassified segments in total for the training and test samples (0.016% of
all segments). We used the Weka toolkit and opted for random forest
classifiers based on 75 random trees. Using different thresholds to adjust the
trade-off between precision and recall, we generated a classifier with good
precision and another one with good recall. Both of them are used in the next
step. We tried other types of classifiers including naive Bayes, Bayesian
networks [26], PART decision lists [27], and convolutional networks [28], but
we achieved the best results with random forests.

### Gel Panel Detection

A gel panel typically consists of several gel segments and comes with labels
describing the involved genes, proteins, and conditions. For our goal, it is
not sufficient to just detect the figures that contain gel panels, but we also
have to extract their positions within the figures and to access their labels.
This is not a simple classification task, and therefore machine learning
techniques do not apply that easily. For that reason, we used a detection
procedure based on hand-coded rules.

In a first step, we group gel segments to find contiguous gel regions that
form the center part of gel panels. To do so, we start with looking for
segments that our high-precision classifier detects as gel segments. Then, we
repeatedly look for adjacent gel segments, this time applying the high-recall
classifier, and merge them. Two segments are considered neighbors if they are
at most 50 pixels apart333We are using absolute distance values at this point.
A more refined algorithm could apply some sort of relative measure. However,
the resolution of the images does not vary that much, which is why absolute
values worked out well so far. and do not have any text segment between them.
Thus, segments which could be gel segments according to the high-recall
classifier make it into a gel panel only if there is at least one high-
precision segment in their group. The goal is to detect panels with high
precision, but also to detect the complete panels and not just parts of them.
We focus here on precision because low recall can be leveraged by the large
number of available gel images. Furthermore, as the open access part of PubMed
Central only makes up a small subset of all biomedical publications, recall in
a more general sense is anyway limited by the proportion of open access
publications.

As a next step, we collect the labels in the form of text segments located
around the detected gel regions. For a text segment to be attributed to a
certain gel panel, its nearest edge must be at most 30 pixels away from the
border of the gel region and its farthest edge must not be more than 150
pixels away. We end up with a representation of a gel panel consisting of two
parts: a center region containing a number of gel segments and a set of labels
in the form of text segments located around the center region.

To evaluate this algorithm, we collected yet another sample of 500 figures, in
which 106 gel panels in 61 different figures were revealed by manual
annotation.444Again, these manual annotations were double-checked to ensure
their quality. Five errors were found and fixed in this process. Based on this
sample, we manually checked whether our algorithm is able to detect the
presence and the (approximate) position of the gel panels.

## Results

The top part of Table 1 shows the result of the gel detection classifier. We
generated three different classifiers from the training data, one for each of
the threshold values 0.15, 0.3 and 0.6. Lower threshold values lead to higher
recall at the cost of precision, and vice versa. In the balanced case, we
achieved an F-score of 75%. To get classifiers with precision or recall over
90%, F-score goes down significantly, but stays in a sensible range. These two
classifiers (thresholds 0.15 and 0.6) are used in the next step. To interpret
these values, one has to consider that gel segments are greatly outnumbered by
non-gel segments. Concretely, only about 3% are gel segments. More
sophisticated accuracy measures for classifier performance, such as the area
under the ROC curve [29], take this into account. For the presented
classifiers, the area under the ROC curve is 98.0% (on a scale from 50% for a
trivial, worthless classifier to 100% for a perfect one).

| Method | Threshold | Precision | Recall | F-score | ROC area
---|---|---|---|---|---|---
Segments | Random forests | 0.15 | 0.439 | 0.909 | 0.592 | $\left.\begin{matrix}~{}\\\ ~{}\\\ \end{matrix}\right\\}$ 0.980
0.30 | 0.765 | 0.739 | 0.752
0.60 | 0.926 | 0.301 | 0.455
Naive Bayes |  | 0.172 | 0.739 | 0.279 | 0.883
Bayesian network |  | 0.394 | 0.531 | 0.452 | 0.914
PART decision list |  | 0.631 | 0.496 | 0.555 | 0.777
Convolutional networks |  | 0.142 | 0.949 | 0.248 |
Panels | Hand-coded rules |  | 0.951 | 0.368 | 0.530 |
Table 1: The results of the gel segment detection classifiers (top) and the
gel panel detection algorithm (bottom)

The results of the gel panel detection algorithm are shown in the bottom part
of Table 1. The precision is 95% at a recall of 37%, leading to an F-score of
53%. The comparatively low recall is mainly due to the general problem of
pipeline-based approaches that the various errors from the earlier steps
accumulate and are hard to correct at a later stage in the pipeline.

Table 2 shows the results of running the pipeline on PubMed Central. We
started with about 410 000 articles, the entire open access subset of PubMed
Central at the time we downloaded them (February 2012). We successfully parsed
the XML files of 94% of these articles (for the remaining articles, the XML
file was missing or not well-formed, or other unexpected errors occurred). The
successful articles contained around 1 100 000 figures, for some of which our
segment detection step encountered image formatting errors or other internal
errors, or was just not able to detect any segments. We ended up with more
than 880 000 figures, in which we detected about 86 000 gel panels, i.e.
roughly ten out of 100 figures. For each of them, we found on average 3.6
labels with recognized text. After tokenization, we identified about 76 000
gene names in these gel labels, which corresponds to 6.8% of the tokens.
Considering all text segments (including but not restricted to gel labels),
only 3.3% of the tokens are detected as gene names.555The low numbers are
partially due to the fact that a considerable part of the tokens are “junk
tokens” produced by the OCR step when trying to recognize characters in
segments that do not contain text.

Total articles | 410 950
---|---
Processed articles | 386 428
Total figures from processed articles | 1 110 643
Processed figures | 884 152
Detected gel panels | 85 942
Detected gel panels per figure | 0.097
Detected gel labels | 309 340
Detected gel labels per panel | 3.599
Detected gene tokens | 1 854 609
Detected gene tokens in gel labels | 75 610
Gene token ratio | 0.033
Gene token ratio in gel labels | 0.068
Table 2: The results of running the pipeline on the open access subset of
PubMed Central

## Discussion

The presented results show that we are able to detect gel segments with high
accuracy, which allows us to subsequently detect whole gel panels at a high
precision. The recall of the panel detection step is relatively low, but with
about 37% still in a reasonable range. As mentioned above, we can leverage the
high number of available figures, which makes precision more important than
recall. Running our pipeline on the whole set of open access articles from
PubMed Central, we were able to retrieve 85 942 potential gel panels (around
95% of which we can expect to be correctly detected).

The next step would be to recognize the named entities mentioned in the gel
labels. To this aim, we did a preliminary study to investigate whether we are
able to extract the names of genes and proteins from gel diagrams. To do so,
we tokenized the label texts and looked for entries in the Entrez Gene
database to match the tokens. This look-up was done in a case-sensitive way,
because many names in gel labels are acronyms, where the specific
capitalization pattern can be critical to identify the respective entity. We
excluded tokens that have less than three characters, are numbers (Arabic or
Latin), or correspond to common short words (retrieved from a list of the 100
most frequent words in biomedical articles). In addition, we extended this
exclusion list with 22 general words that are frequently used in the context
of gel diagrams, some of which coincide with gene names according to
Entrez.666These words are: _min_ , _hrs_ , _line_ , _type_ , _protein_ , _DNA_
, _RNA_ , _mRNA_ , _membrane_ , _gel_ , _fold_ , _fragment_ , _antigen_ ,
_enzyme_ , _kinase_ , _cleavage_ , _factor_ , _blot_ , _pro_ , _pre_ ,
_peptide_ , and _cell_. Since gel electrophoresis is a method to analyze genes
and proteins, we would expect to find more such mentions in gel labels than in
other text segments of a figure. By measuring this, we get an idea of whether
the approach works out or not. In addition, we manually checked the gene and
protein names extracted from gel labels after running our pipeline on 2 000
random figures. In 124 of these figures, at least one gel panel was detected.
Table 3 shows the results of this preliminary evaluation. Almost two-thirds of
the detected gene/protein tokens (65.3%) were correctly identified. 9% thereof
were correct but could be more specific, e.g. when only “actin” was recognized
for “$\beta$-actin” (which is not incorrect but of course much harder to map
to a meaningful identifier). The incorrect cases (34.6%) can be split into two
classes of roughly the same size: some recognized tokens were actually not
mentioned in the figure but emerged from OCR errors; other tokens were
correctly recognized but incorrectly classified as gene or protein references.

| absolute | relative
---|---|---
Total | 156 | 100.0%
Incorrect | 54 | 34.6%
– Not mentioned (OCR errors) | 28 | 17.9%
– Not references to genes or proteins | 26 | 16.7%
Correct | 102 | 65.3%
– Partially correct (could be more specific) | 14 | 9.0%
– Fully correct | 88 | 56.4%
Table 3: Numbers of recognized gene/protein tokens in 2 000 random figures

Although there is certainly much room for improvement, this simple gene
detection step seems to perform reasonably well.

For the last step, relation extraction, we cannot present any concrete results
at this point. After recognizing the named entities, we would have to
disambiguate them, identify their semantic roles (condition, measurement or
something else), align the gel images with the labels, and ultimately quantify
the degree of expression. To improve the quality of the results, combinations
with classical text mining techniques should be considered. This is all future
work. We expect to be able to profit to a large extent from existing work to
disambiguate protein and gene names [30, 31] and to detect and analyze gel
spots (see the existing work mentioned above).

It seems reasonable to assume that these results can be combined with existing
techniques of term disambiguation and gel spot detection at a satisfactory
level of accuracy. We plan to investigate this in future work.

As mentioned above, we have started to investigate how the gel segment
detection step could be improved by the use of the image recognition technique
of convolutional networks (ConvNet) [28]. We started with a simplified
approach to the one presented in [32]. In this approach, images are tiled into
small quadratic pieces. We used a single network (and not several parallel
networks), based on $48\times 48$ input tile images with three layers of
convolutions. The first layer takes eight $5\times 5$ convolutions and is
followed by a $2\times 2$ sub-sampling. The second layer takes twenty four
$5\times 5$ convolutions and is followed by a $3\times 3$ sub-sampling. The
last layer takes seventy two $6\times 6$ convolutions, which leads to a fully
connected layer. We trained our ConvNet on the 500 images of the training set,
where we manually annotated the tiles as _gel_ and _non-gel_. With the use of
EBLearn [33], this trained ConvNet classified the tiles of the 500 images of
our testing set. The classified tiles can then be reconstructed into a mask
image, as shown in Figure 4. A manual check of the clusters of recognized gel
tiles led to the results shown in Table 1. Recall is very good (95%) but
precision is very poor (14%), leading to an F-score of 25%. This is much worse
than the results we got with our random forest approach, which is why ConvNet
is currently not part of our pipeline. We hope, however, that we can further
optimize this ConvNet approach and combine it with random forests to exploit
their (hopefully) complementary benefits. Using ConvNet to classify complete
images as _gel-image_ or _non-gel-image_ and adjusting the classification to
account for unbalanced classes, we were able to obtain an F-score of 74%,
which makes us confident that a combination of the two approaches could lead
to a significant improvement of our gel segment detection step. As an
alternative approach, we will try to run ConvNet on down-scaled entire panels
rather than small tiles, as described in [34]. Furthermore, we will experiment
with parallel networks instead of single ones to improve accuracy.

Figure 4: Original and mask image after ConvNet classification for an
exemplary image from PMID 14993249. Green means _gel_ ; brown means _other_ ;
and white means _not enough gradient information_.

The results obtained from our gel recognition pipeline indicate that it is
feasible to extract relations from gel images, but it is clear that this
procedure is far from perfect. The automatic analysis of bitmap images seems
to be the only efficient way to extract such relations from existing
publications, but other publishing techniques should be considered for the
future. The use of vector graphics instead of bitmaps would already greatly
improve any subsequent attempts of automatic analysis. A further improvement
would be to establish accepted standards for different types of biomedical
diagrams in the spirit of the Unified Modeling Language, a graphical language
widely applied in software engineering since the 1990s. Ideally, the resulting
images could directly include semantic relations in a formal notation, which
would make relation mining a trivial procedure. If authors are supported by
good tools to draw diagrams like gel images, this approach could turn out to
be feasible even in the near future.

Concretely, we would like to take the opportunity to postulate the following
actions, which we think should be carried out to make the content of images in
biomedical articles more accessible:

  * •

Stop pressing diagrams into bitmaps! Unless the image only consists of one
single photograph, screenshot, or another kind of picture that only has bitmap
representation, vector graphics should be used for article figures.

  * •

Let data and metadata travel from the tools that generate diagrams to the
final articles! Whenever the specific tool that is used to generate the
diagram “knows” that a certain graphical element refers to an organism, a
gene, an interaction, a point in time, or another kind of entity, then this
information should be stored in the image file, passed on, and finally
published with the article.

  * •

Use RDF vocabularies to embed semantic annotations in diagrams! Tools for
creating scientific diagrams should use RDF notation and stick to existing
standardized schemas (or define new ones if required) to annotate the diagram
files they create.

  * •

Define standards for scientific diagrams! In the spirit of the Unified
Modeling Language, the biomedical community should come up with standards that
define the appearance and meaning of different types of diagrams.

Obviously, different groups of people need to be involved in these actions,
namely article authors, journal editors, and tool developers. It is relatively
inexpensive to follow these postulates (though it might require some time),
which in turn would greatly improve data sharing, image mining, and scientific
communication in general. Standardized diagrams could be the long sought
solution to the problem of how to let authors publish computer-processable
formal representations for (part of) their results. This can build upon the
efforts of establishing an open annotation model [35, 36].

## Conclusions

Successful image mining from gel diagrams in biomedical publications would
unlock a large amount of valuable data. Our results show that gel panels and
their labels can be detected with high accuracy, applying machine learning
techniques and hand-coded rules. We also showed that genes and proteins can be
detected in the gel labels with satisfactory precision.

Based on these results, we believe that this kind of image mining is a
promising and viable approach to provide more powerful query interfaces for
researchers, to gather relations such as protein-protein interactions, and to
generally complement existing text mining approaches. At the same time, we
believe that an effort towards standardization of scientific diagrams such as
gel images would greatly improve the efficiency and precision of image mining
at relatively low additional costs at the time of publication.

## Competing Interests

The authors declare that they have no competing interests.

## Authors’ Contributions

TK was the main author and main contributor of the presented work. He was
responsible for designing and implementing the pipeline, gathering the data,
performing the evaluation, and analyzing the results. MLN applied, trained,
and evaluated the ConvNet classifier, and contributed to the annotation of the
test sets. TL built and analyzed the corpus for the preparatory evaluation. MK
contributed to the conception and the design of the approach and to the
analysis of the results. All authors have been involved in drafting or
revising the manuscript, and all authors read and approved the final
manuscript.

## Acknowledgments

This study has been supported by the National Library of Medicine grant
5R01LM009956.

## References

  * [1] Yu H, Lee M: Accessing bioscience images from abstract sentences. _Bioinformatics_ 2006, 22(14):e547–e556.
  * [2] Zweigenbaum P, Demner-Fushman D, Yu H, Cohen KB: Frontiers of biomedical text mining: current progress. _Briefings in Bioinformatics_ 2007, 8(5):358–375.
  * [3] Peng H: Bioimage informatics: a new area of engineering biology. _Bioinformatics_ 2008, 24(17):1827–1836, [http://dx.doi.org/10.1093/bioinformatics/btn346].
  * [4] Xu S, McCusker J, Krauthammer M: Yale Image Finder (YIF): a new search engine for retrieving biomedical images. _Bioinformatics_ 2008, 24(17):1968–1970, [http://dx.doi.org/10.1093/bioinformatics/btn340].
  * [5] Hearst MA, Divoli A, Guturu H, Ksikes A, Nakov P, Wooldridge MA, Ye J: BioText Search Engine. _Bioinformatics_ 2007, 23(16):2196–2197, [http://dx.doi.org/10.1093/bioinformatics/btm301].
  * [6] Kuhn T, Krauthammer M: Image Mining from Gel Diagrams in Biomedical Publications. In _Proceedings of the 5th International Symposium on Semantic Mining in Biomedicine (SMBM 2012)_ , University of Zurich (Zurich, Switzerland) 2012:26–33, [http://www.zora.uzh.ch/64476/].
  * [7] Kuhn T, Luong T, Krauthammer M: Finding and Accessing Diagrams in Biomedical Publications. In _Proceedings of the American Medical Informatics Association (AMIA) 2012 Annual Symposium_ , American Medical Informatics Association (Bethesda, MD, USA) 2012.
  * [8] Southern E: Detection of specific sequences among DNA fragments separated by gel electrophoresis. _J. Mol. Biol._ 1975, 98(3):503–517.
  * [9] Alwine JC, Kemp DJ, Stark GR: Method for detection of specific RNAs in agarose gels by transfer to diazobenzyloxymethyl-paper and hybridization with DNA probes. _Proceedings of the National Academy of Sciences_ 1977, 74(12):5350.
  * [10] Burnette WN: Western Blotting: Electrophoretic transfer of proteins from sodium dodecly sulfate-polyacrylamide gels to unmodified nitrocellulose and radiographic detection with antibody and radioiodinated protein A. _Anal. Biochem._ 1981, 112:195–203.
  * [11] De Bruijn B, Martin J: Getting to the (c)ore of knowledge: mining biomedical literature. _International Journal of Medical Informatics_ 2002, 67(1-3):7–18.
  * [12] Murphy RF, Kou Z, Hua J, Joffe M, Cohen WW: Extracting and Structuring Subcellular Location Information from On-line Journal Articles: The Subcellular Location Image Finder. In _Proceedings of the IASTED International Conference on Knowledge Sharing and Collaborative Engineering (KSCE-2004)_ , ACTA Press (Calgary, AB, Canada) 2004:109–114, [http://www.actapress.com/Abstract.aspx?paperId=17244].
  * [13] Qian Y, Murphy RF: Improved recognition of figures containing fluorescence microscope images in online journal articles using graphical models. _Bioinformatics_ 2008, 24(4):569–576, [http://dx.doi.org/10.1093/bioinformatics/btm561].
  * [14] Kozhenkov S, Baitaluk M: Mining and integration of pathway diagrams from imaging data. _Bioinformatics_ 2012, 28(5):739–742, [http://dx.doi.org/10.1093/bioinformatics/bts018].
  * [15] Lemkin PF: Comparing two-dimensional electrophoretic gel images across the Internet. _Electrophoresis_ 1997, 18(3-4):461–470.
  * [16] Luhn S, Berth M, Hecker M, Bernhardt J: Using standard positions and image fusion to create proteome maps from collections of two-dimensional gel electrophoresis images. _Proteomics_ 2003, 3(7):1117–1127.
  * [17] Cutler P, Heald G, White IR, Ruan J: A novel approach to spot detection for two-dimensional gel electrophoresis images using pixel value collection. _Proteomics_ 2003, 3(4):392–401.
  * [18] Rogers M, Graham J, Tonge RP: Statistical models of shape for the analysis of protein spots in two-dimensional electrophoresis gel images. _Proteomics_ 2003, 3(6):887–896.
  * [19] Zerr T, Henikoff S: Automated band mapping in electrophoretic gel images using background information. _Nucleic Acids Research_ 2005, 33(9):2806–2812.
  * [20] Schlamp K, Weinmann A, Krupp M, Maass T, Galle P, Teufel A: BlotBase: a northern blot database. _Gene_ 2008, 427(1-2):47–50.
  * [21] Rodriguez-Esteban R, Iossifov I: Figure mining for biomedical research. _Bioinformatics_ 2009, 25(16):2082–2084, [http://dx.doi.org/10.1093/bioinformatics/btp318].
  * [22] Ramakrishnan C, Patnia A, Hovy EH, Burns GA, et al.: Layout-aware text extraction from full-text PDF of scientific articles. _Source code for biology and medicine_ 2012, 7:255–258.
  * [23] Xu S, Krauthammer M: A new pivoting and iterative text detection algorithm for biomedical images. _J. of Biomedical Informatics_ 2010, 43(6):924–931, [http://dx.doi.org/10.1016/j.jbi.2010.09.006].
  * [24] Xu S, Krauthammer M: Boosting text extraction from biomedical images using text region detection. In _Biomedical Sciences and Engineering Conference (BSEC), 2011_ , IEEE (New York City, NY, USA) 2011:1–4.
  * [25] Haralick RM, Shanmugam K, Dinstein I: Textural Features for Image Classification. _IEEE Transactions on Systems, Man, and Cybernetics_ 1973, 3(6):610–621, [http://dx.doi.org/10.1109/TSMC.1973.4309314].
  * [26] Cooper GF, Herskovits E: A Bayesian method for the induction of probabilistic networks from data. _Machine learning_ 1992, 9(4):309–347.
  * [27] Frank E, Witten IH: Generating Accurate Rule Sets Without Global Optimization. In _Proceedings of the Fifteenth International Conference on Machine Learning_ , Morgan Kaufmann Publishers (Burlington, MA, USA) 1998:144–151.
  * [28] LeCun Y, Bengio Y: Convolutional networks for images, speech, and time series. _The handbook of brain theory and neural networks_ 1995, 3361.
  * [29] Bradley AP: The use of the area under the ROC curve in the evaluation of machine learning algorithms. _Pattern recognition_ 1997, 30(7):1145–1159.
  * [30] Tanabe L, Wilbur WJ: Tagging gene and protein names in biomedical text. _Bioinformatics_ 2002, 18(8):1124–1132.
  * [31] Lu Z, Kao HY, Wei CH, Huang M, Liu J, Kuo CJ, Hsu CN, Tsai R, Dai HJ, Okazaki N, et al.: The gene normalization task in BioCreative III. _BMC bioinformatics_ 2011, 12(Suppl 8):S2.
  * [32] Barbano PE, Nagy ML, Krauthammer M: Energy-Based Architecture for Classification of Publication Figures. In _Proceedings of the Biomedical Science and Engineering Center Conference (BSEC 2013)_ , IEEE (New York City, NY, USA) 2013.
  * [33] Sermanet P, Kavukcuoglu K, LeCun Y: EBlearn: Open-source energy-based learning in C++. In _Proceedings of the 21st International Conference on Tools with Artificial Intelligence (ICTAI’09)_ , IEEE (New York City, NY, USA) 2009:693–697.
  * [34] Krizhevsky A, Sutskever I, Hinton G: ImageNet classification with deep convolutional neural networks. _Advances in Neural Information Processing Systems_ 2012, 25:1106–1114.
  * [35] Ciccarese P, Ocana M, Garcia Castro LJ, Das S, Clark T: An open annotation ontology for science on web 3.0. _J Biomed Semantics_ 2011, 2(Suppl 2):S4.
  * [36] Sanderson R, Ciccarese P, Van de Sompel H: Open Annotation Data Model. Community draft, W3C 2013, [http://www.openannotation.org/spec/core/20130208/index.html].