# CheXagent: Towards a Foundation Model
for Chest X-Ray Interpretation

Zhihong Chen1∗ Maya Varma1∗ Jean-Benoit Delbrouck1∗ Magdalini Paschali1
&Louis Blankemeier1 Dave Van Veen1 Jeya Maria Jose Valanarasu1 Alaa Youssef1
&Joseph Paul Cohen${}^{1\text{\textdagger}}$ Eduardo Pontes Reis1 Emily B.
Tsai1 Andrew Johnston1
&Cameron Olsen1 Tanishq Mathew Abraham2 Sergios Gatidis1
&Akshay S. Chaudhari1 Curtis Langlotz1

1Stanford University 2Stability AI
<EMAIL_ADDRESS>
###### Abstract

Chest X-rays (CXRs) are the most frequently performed imaging test in clinical
practice. Recent advances in the development of vision-language foundation
models (FMs) give rise to the possibility of performing automated CXR
interpretation, which can assist physicians with clinical decision-making and
improve patient outcomes. However, developing FMs that can accurately
interpret CXRs is challenging due to the (1) limited availability of large-
scale vision-language datasets in the medical image domain, (2) lack of vision
and language encoders that can capture the complexities of medical data, and
(3) absence of evaluation frameworks for benchmarking the abilities of FMs on
CXR interpretation. In this work, we address these challenges by first
introducing _CheXinstruct_ \- a large-scale instruction-tuning dataset curated
from 28 publicly-available datasets. We then present _CheXagent_ \- an
instruction-tuned FM capable of analyzing and summarizing CXRs. To build
CheXagent, we design a clinical large language model (LLM) for parsing
radiology reports, a vision encoder for representing CXR images, and a network
to bridge the vision and language modalities. Finally, we introduce
_CheXbench_ \- a novel benchmark designed to systematically evaluate FMs
across 8 clinically-relevant CXR interpretation tasks. Extensive quantitative
evaluations and qualitative reviews with five expert radiologists demonstrate
that CheXagent outperforms previously-developed general- and medical-domain
FMs on CheXbench tasks. Furthermore, in an effort to improve model
transparency, we perform a fairness evaluation across factors of sex, race and
age to highlight potential performance disparities. Our project is at
https://stanford-aimi.github.io/chexagent.html.

**footnotetext: Equal contributions.††footnotetext: Work not related to
position at Amazon.

##  1 Introduction

Figure 1: Overview of the proposed pipeline: CheXinstruct is a curation of
datasets for instruction-tuning across various CXR tasks, CheXagent is our
clinical FM for CXR interpretation, and CheXbench is our comprehensive FM
evaluation benchmark. Two example CXR interpretation tasks include local
findings generation and open-ended visual question answering (VQA).

Foundation models (FMs) have recently emerged as a powerful class of models
capable of performing a diverse range of reasoning and comprehension tasks
[9]. The rise of FMs presents a major opportunity to re-imagine complex
healthcare workflows that commonly require posing multi-faceted questions from
inherently multi-modal data. One particular clinical workflow is the analysis
of medical imaging data. Take for example chest X-ray (CXR) interpretation -
the most common medical imaging study, with 70+ million CXRs performed
annually in the US [33]. Here, radiologists interpret hundreds of images
daily, translate imaging insights into textual descriptions, and summarize
image findings succinctly to other clinicians, while maximizing accuracy and
minimizing bias. Moreover, patient-facing tasks for CXRs might include
answering clarifying questions about these generated findings. A CXR FM
capable of automating or increasing the efficiency of these tasks can
substantially improve clinical decision-making as well as patient satisfaction
and outcomes [67, 75, 96].

High-quality CXR FMs must bridge the gap between vision and language. Most
prior approaches for building vision-language FMs focus on natural image
settings, where high-performing methods align image encoders with pretrained
large language models (LLMs). Such instruction-tuned multimodal LLMs
demonstrate superior capabilities across a range of perception and generation
tasks. However, developing instruction-tuned multimodal LLMs for medical
imaging is challenging for the following reasons:

  1. 1.

There is a lack of vision-language medical imaging datasets. Instruction-tuned
multimodal LLMs require large-scale, diverse training datasets with data
triplets consisting of instructions, images, and answers. Whereas existing
approaches in the natural image domain leverage millions of training samples
from datasets like LAION-5B [66], the availability of such data is severely
limited in the medical domain due to patient privacy concerns that prevent
dissemination of large corpora of medical text. As a result, existing
instruction-tuned multimodal LLMs developed for CXR interpretation are trained
on small datasets with limited instruction diversity [74, 89].

  2. 2.

Existing vision and language encoders fail to capture the complexities of
medical data. Multimodal LLMs require powerful image and text encoders.
However, CXRs represent a large domain shift from natural images and text, and
thus, existing pretrained vision and language encoders struggle with medical
knowledge grounding.

  3. 3.

Performing rigorous evaluations of multimodal LLMs in medicine is challenging.
Multimodal LLMs generate open-ended responses, and evaluating free-text
responses for factual correctness and completeness is arduous, particularly
for large-scale evaluations that require domain expertise. Previous FMs
developed for CXR interpretation are primarily evaluated using visual-
question-answer (VQA) or text generation tasks. To the best of our knowledge,
there are no existing benchmarks for quantitative and qualitative evaluation
of multimodal LLMs across diverse, clinically-relevant CXR interpretation
tasks.

In this work, we address these challenges by (i) introducing _CheXinstruct_ ,
a large-scale instruction-tuning dataset with instruction-image-answer
triplets, (ii) developing _CheXagent_ , an instruction-tuned multimodal LLM
for CXR interpretation, and (iii) curating _CheXbench_ , a novel benchmark to
enable systematic comparisons of FMs across 8 clinically-relevant CXR
interpretation tasks. Below we outline our key contributions, also summarized
in Fig. 1, that can help create capable and robust CXR FMs:

  1. 1.

CheXinstruct is an instruction-tuning dataset with 6M instruction-image-answer
triplets designed to improve the ability of FMs to interpret CXRs. We collect
instructions from 34 tasks and 65 unique datasets, spanning categories
including coarse- and fine-grained image understanding, question answering,
and text generation.

  2. 2.

CheXagent is an instruction-tuned foundation model with 8B parameters capable
of analyzing images, understanding text, and generating responses. Our
methodology for developing CheXagent includes training (1) a clinical LLM
capable of understanding radiology reports, (2) a vision encoder capable of
reading CXRs, and (3) a network to bridge the vision and language modalities.
We then perform instruction-tuning using data from CheXinstruct.

  3. 3.

CheXbench is a novel benchmark designed to rigorously evaluate FMs across two
evaluation axes: image perception and textual understanding. We introduce 8
tasks across 7 CXR datasets, and we evaluate performance using close-ended
multiple-choice predictions as well as open-ended text generation.

We use CheXbench to compare CheXagent with six prior general-domain and
medical-domain FMs. Across six visual tasks, the performance of CheXagent
surpasses general-domain FMs by 97.5% and medical-domain FMs by 55.7%. Across
two text generation tasks, CheXagent provides medical text evaluated via
automated quantitative metrics and qualitative metrics from five expert
radiologists. We further provide an evaluation of potential model bias and
highlight performance disparities across demographic factors of sex, race and
age to improve model transparency.

##  2 Related Work

Figure 2:  Collection of datasets and tasks comprising CheXinstruct (Left).
The four-stage training process of CheXagent, starting from adapting a general
LLM for clinical use, through training a CXR vision encoder and a vision-
language bridger, to the final stage of instruction tuning on diverse CXR
tasks (Right).

###  2.1 Foundation Models

The surge in available data and computational resources has enabled the
creation of FMs that are versatile in addressing a wide array of tasks with a
single generalist model.

Language: Significant strides in FMs were first seen in natural language
processing (NLP) because large-scale text data was easily accessible online.
LLMs like GPT-3 [10], ChatGPT, GPT-4 [54], FLAN-T5 [18], Llama-2 [76, 77],
Mistral 7B [35], and PaLM-2 [17, 3] excel at multiple text-based tasks using
only prompting, enabling new possibilities, such as outperforming human
experts for clinical text summarization [81].

Vision: In vision, FMs like Stable Diffusion [63], DALL-E [61, 60], and Imagen
[64] were proposed for the task of text-to-image generation. Models Like
Segment Anything Model (SAM) [38] and Segment Everything Everywhere Model
(SEEM) [98] have been developed to perform in-the-wild segmentation. To
achieve LLM-like scaling in vision, ViT-22B [19], a scaled-up version of ViT
[23] was introduced as a large vision encoder. In the medical domain, there
are several datasets designed for training FMs [49].

Vision-Language: Given the inherent connection between different modalities
such as language and vision, inter-modality supervision has been critical in
the development of many vision-language foundation models (VLMs), like CLIP
[59]. Several other large-scale VLMs like Flamingo [1], Coca [94], Qwen-VL
[5], BLIP [44], LLaVA [47], PaLI-X [14], and CogVLM [83] have been introduced
with a focus on developing models powerful enough to perceive and understand
both text and images. To evaluate the performance of these models, benchmarks
like Multimodal Large Language Model Evaluation (MME) [25] and SEED-Bench [42]
have been introduced. These benchmarks are carefully curated using manually
generated instruction-answer pairs to avoid any data leakage.

###  2.2 Medical Foundation Models

Biomedical language models (LMs) like BioNLP [41], BioBERT [40], PubMedBERT
[29], BioGPT [48], Med-PaLM [40] and Med-PaLM 2 [71] have shown that LMs can
be fine-tuned on curated biomedical corpora to perform competitively on
medical question answering tasks. Specifically, Med-PaLM 2 [71] achieved high
scores in questions in the style of the US Medical Licensing Examination
(USMLE) highlighting rapid progress towards physician-level performance for
medical question answering. Since medicine inherently involves multiple
modalities and covers a broad range of tasks, multi-modal FMs are particularly
suitable for this field. Works like LLaVA-Med [43], Med-Flamingo [51], and
Med-PaLM M [78] have proposed generalist models spanning across modalities
like imaging and clinical text for tasks in radiology, dermatology, pathology,
etc. RadFM [88] introduced a foundation model for radiology for both 2D and 3D
imaging tasks. XrayGPT [74] aligned a MedCLIP [85] visual encoder with Vicuna
[16], which was finetuned using clinical text. RoentGen, a fine-tuned version
of Stable Diffusion, allows generating CXR images using radiology report text
prompts [13, 12].

Med-PaLM M, Med-Flamingo, RadFM, and LLaVA-Med aim to establish generalist
frameworks that empower models to process data from various image modalities.
In contrast, CheXagent is designed to excel in handling multiple tasks related
to CXRs within a single model.

###  2.3 Most Related Work

Our approach is most similar to the following works. (i) BLIP-2 [44]: our
model architecture is closely aligned with that of BLIP-2, which uses a
Querying Transformer (QFormer) to bridge the vision-language modality gap;
(ii) FLAN [86] and MultiInstruct [90]: our development of instruction tuning
datasets derived from existing annotated datasets is inspired by these works.

##  3 CheXinstruct: Instruction-Tuning Dataset

Motivation CheXinstruct seeks to cover a broad range of tasks shown in Fig. 2a
to support training CXR FMs. These tasks can either (i) improve the abilities
of FMs to understand CXRs or (ii) improve clinical decision making.

Design Scheme CheXinstruct is organized into four levels:

  1. 1.

Capability Level, where we specify the essential skills and competencies that
a CXR FM should possess.

  2. 2.

Task Level, where we outline a wide array of specific tasks that align with
each identified capability, providing a clear framework for what the FM should
be able to accomplish in the context of CXRs.

  3. 3.

Dataset Level, where we identify and categorize various CXR datasets, each
associated with a particular task. We focus on ensuring that the datasets are
relevant and appropriately matched to the tasks they are intended to support.

  4. 4.

Instance Level, where we define individual instances within each dataset at
the most granular level. Each instance comprises an input (such as a CXR
image) and its corresponding labels, forming the basic units for training and
evaluating the FM.

Tasks CheXinstruct consists of five task categories according to their
capabilities:

  * •

Coarse-grained Image Understanding, which defines the overall understanding of
CXRs, e.g., view classification [36], and disease classification [84, 32, 62,
55, 30, 6, 34, 11, 69].

  * •

Fine-grained Image Understanding, which defines the localized understanding of
CXRs, e.g., abnormality detection [53, 58], abnormality grounding [8], and
foreign object detection [91].

  * •

Question Answering, which defines the ability to respond to a question related
to CXRs, e.g., close-ended visual question answering (VQA) [97, 57], open-
ended VQA [7, 4], difference VQA [31], and text QA.

  * •

Text Generation, which defines the ability to generate radiology report
sections, including a description of the findings [21, 82, 56], impression
generation [24], findings summarization [15], and local findings generation
[36].

  * •

Miscellaneous: This category defines the miscellaneous abilities that are
critical for a CXR FM, e.g., report evaluation [93, 50], and natural language
explanation [37].111Additional details about the tasks are presented in
Appendix A.

Dataset Sources With the aforementioned taxonomy of tasks, we created
CheXinstruct by either (i) collecting existing publicly available datasets or
(ii) curating a dataset with new labels from existing datasets.

For every dataset, we derive relevant information to formulate distinct tasks.
All the datasets are split following their corresponding official splits if
applicable.222More details of the dataset sources are presented in Appendix B.

Task Instruction Creation Inspired by [86], we create instructions by manually
writing ten instruction templates for each task.333The annotators are
researchers in radiology and computer vision (CV). Each instruction template
contains placeholders (e.g., <IMAGE>, <QUESTION>, and <OPTIONS>), which are
replaced with specific values when creating instruction-following instances.

Finally, each CheXinstruct instance is a triplet consisting of an image (or
none, in the case of non-image-based tasks), a question, and an answer.

Data Analysis

Table 1: The statistics of CheXinstruct, where the total and unique numbers of
questions and images are shown along with the average number of times each
unique image is reused in the dataset (reuse).

Split | Questions | Images
---|---|---
Total | Unique | Reuse | Total | Unique | Reuse
Train | 6.1M | 1.9M | 3.2 | 9.3M | 1.1M | 8.6
Val | 203.3K | 82.9K | 2.5 | 228.9K | 31.7K | 7.2
Test | 188.3K | 72.8K | 2.6 | 233.7K | 49.5K | 4.7

We describe the overall statistics of CheXinstruct, including the number of
questions-answer pairs and CXRs in Table 1. There are 6.1M question-answer
pairs in CheXinstruct. Furthermore, since images from one dataset may be used
across multiple tasks, we present additional information to provide intuition
about the potential “overlap” problem among different datasets.444More details
about the data analysis are reported in Appendix C.

##  4 CheXagent: Instruction-Following CXR FM

###  4.1 Problem Setup

The aim of CheXagent is a model that can “see” images $x_{I}$ and/or “read”
text $x_{T}$ and generate “responses” $y$. To this end, we introduce three
components of our model: a vision encoder $\mathcal{M}_{v}$, a vision-language
bridger $\mathcal{M}_{b}$, and a language decoder $\mathcal{M}_{l}$.
Therefore, the ultimate formulation of the desired model is:

$y=\mathcal{M}_{l}(\mathcal{M}_{b}(\mathcal{M}_{v}(x_{I})),x_{T}).$ (1)

###  4.2 Training CheXagent

In this subsection, we present the four training stages of CheXagent, which
are illustrated in Figure 2b.

Stage 0: Train a clinical LLM Numerous open-source biomedical large language
models (LLMs) exist, such as BioMegatron [70], GatorTron [92], BioGPT [48],
BioMedLM555https://huggingface.co/stanford-crfm/BioMedLM, and PMC-LLaMA [87].
These models are predominantly trained on PubMed Central (PMC) articles rather
than clinical texts. To address this gap, we focus on developing a clinical
LLM by adapting a general-domain LLM. Our starting point is Mistral-7B-v0.1
[35], chosen for its demonstrated strong reasoning capabilities across various
benchmarks.

To infuse the model with comprehensive medical and clinical knowledge, we
utilize five distinct text sources for training: (i) PMC article abstracts,
(ii) radiology reports from MIMIC-IV, (iii) MIMIC-IV discharge summaries, (iv)
medical terms from Wikipedia, and (v) CXRs from CheXinstruct. Importantly, for
MIMIC-IV data, we meticulously exclude any studies that are part of the
validation and test sets of MIMIC-CXR to prevent data leakage.

Stage 1: Train a vision encoder for CXR Drawing inspiration from the works of
[45] and [94], we train our vision encoder using a variety of visual pre-
training objectives, namely image-text contrastive (ITC) and image captioning
(IC). Our model architecture mirrors that of [45]. For training purposes, we
utilize datasets comprising image-text pairs, specifically from MIMIC-CXR,
PadChest, and BIMCV-COVID-19. Preliminary studies indicate that employing a
combination of ITC and IC, akin to the approach in [94], yields enhanced
performance in our specific context.

Stage 2: Train a vision-language bridger Following the training of the
clinical LLM and the CXR vision encoder, we focus on developing a bridger
model, $\mathcal{M}_{b}$. This model is designed to map visual data to the
corresponding language (semantic) space. During training, we keep both the
clinical LLM, ${\mathcal{M}_{l}}$, and the CXR vision encoder,
${\mathcal{M}_{v}}$ frozen. This approach is crucial for preventing
catastrophic forgetting of prior knowledge during the image-text alignment
process. For training $\mathcal{M}_{b}$, we employ the same datasets as in
Stage 1, applying an image captioning objective for learning.

Stage 3: Instruction tuning Upon completing Stage 2, we obtain a multi-modal
LLM tailored for CXR interpretation. In this stage, our focus shifts to
training the model on a variety of tasks within the CheXinstruct framework.
Prior to training, we consider two key aspects: (i) reserving certain task-
dataset pairs exclusively for evaluation purposes, and (ii) determining
optimal dataset ratios to ensure balanced training across different
capabilities. For the first, we sequester datasets including OpenI, SLAKE, and
SIIM, which facilitates a more streamlined evaluation process (as discussed in
§5). For the second, we heuristically establish dataset ratios by carefully
assessing the quality and diversity of each dataset. This method leaves room
for future exploration into automated dataset selection and balancing.666More
details on dataset ratios can be found in Appendix D. This training is
conducted using a next-word prediction objective, with the loss computation
being limited to answers.

Implementation Details For model architecture, we use EVA-CLIP-g [73] for the
vision encoder and BERT [22] for the Qformer, a linear layer for the bridger,
and Mistral for the LLM. The optimization of trainable parameters is
structured across various stages: (a) the entire LLM is trained in Stage 0;
(b) in Stage 1, we train both the LoRA parameters of the vision encoder and
the entire BERT encoder; (c) Stage 2 involves training the vision-language
bridger; and (d) in Stage 3, we focus on training both the bridger and the
entire LLM. For optimization across all these stages, we employ the AdamW
optimizer, with each stage having its own hyper-parameters777More details on
the implementation can be found in Appendix E..

##  5 CheXbench: A Benchmark for Evaluating FMs on CXR Interpretation

Table 2: Results of Evaluation Axis 1 of CheXbench for tasks associated with
image perception comparing CheXagent with general domain and medical domain
FMs on several CXR datasets. For each task, we report accuracy.

|  | General-domain FMs | Medical-domain FMs | CheXagent
---|---|---|---|---
Task | Dataset | BLIP-2 | InstructBLIP | XrayGPT | MedFlamingo | RadFM | LLaVA-Med | (Ours)
| MIMIC-CXR | 28.8 | 25.3 | 24.0 | 25.0 | 28.5 | 23.8 | 97.5
View Classification | CheXpert | 38.0 | 34.0 | 33.0 | 39.0 | 37.0 | 30.0 | 96.7
| SIIM | 53.0 | 54.0 | 50.0 | 50.0 | 50.0 | 49.0 | 64.0
| RSNA | 50.0 | 60.0 | 50.0 | 50.0 | 50.0 | 44.0 | 81.0
Binary Disease Classification | CheXpert | 51.5 | 53.2 | 51.5 | 48.5 | 55.8 | 47.6 | 76.0
| OpenI | 40.2 | 40.2 | 45.4 | 39.0 | 42.2 | 43.8 | 47.0
| MIMIC-CXR | 25.6 | 22.6 | 24.1 | 25.6 | 27.2 | 26.7 | 30.3
Single Disease Identification | CheXpert | 21.3 | 19.5 | 23.7 | 26.0 | 26.6 | 26.0 | 29.6
| OpenI | 48.5 | 54.4 | 57.7 | 46.1 | 52.8 | 53.9 | 55.6
| MIMIC-CXR | 30.0 | 25.3 | 39.0 | 14.7 | 22.3 | 28.7 | 55.3
Multi Disease Identification | CheXpert | 4.3 | 6.1 | 3.9 | 7.1 | 23.6 | 2.1 | 52.1
| Rad-Restruct | 41.2 | 42.4 | 38.6 | 45.5 | 48.5 | 34.9 | 57.1
Visual Question Answering | SLAKE | 74.3 | 86.4 | 52.4 | 64.8 | 85.0 | 55.5 | 78.1
Image-Text Reasoning | OpenI | 47.9 | 52.6 | 52.4 | 54.7 | 54.0 | 45.8 | 59.0

In this section, we introduce CheXbench, an evaluation benchmark for enabling
systematic comparisons of FMs across 8 clinically-relevant CXR interpretation
tasks.

###  5.1 Benchmark Design

CheXbench is structured with two evaluation axes, crafted to assess crucial
aspects of CXR interpretation: image perception and textual understanding. The
evaluations within CheXbench are conducted on a specific subset of the
CheXinstruct test set.888This is comprehensively detailed in Appendix F. In
the following sections, we describe the tasks associated with each evaluation
axis.

Evaluation Axis 1 - Image Perception: We first aim to evaluate the ability of
FMs to understand the visual content of CXRs. Our goals are to (1) evaluate
the ability of FMs to generalize to a variety of data distributions and (2)
include a range of challenging, clinically-relevant tasks. To this end, we
introduce 6 tasks across 7 datasets; in particular, we note that 3 datasets
(SIIM, SLAKE, and OpenI) were completely held-out from CheXagent training to
avoid any potential data leakage. In line with prior benchmarks designed for
general domain FMs [42, 26], we use a multiple-choice format, where an image
and a question are posed to the FM and multiple options are considered. Since
open-ended, free-text outputs from FMs are challenging to evaluate, we instead
compute log-likelihood scores associated with each option; the option with the
highest score is then selected as the response. For each task, we report
accuracy.

  * •

View Classification (700 samples): Given a CXR, the FM is tasked with
identifying the imaging view. We perform view classification on the CheXpert
test set with three possible options (AP, PA, and Lateral) as well as the
MIMIC-CXR test set with four possible options (AP, PA, Lateral, and LL).

  * •

Binary Disease Classification (433 samples): Given a CXR and a disease label,
the FM is tasked with determining if the disease is present in the image. We
perform binary disease classification with twelve disease labels in the
CheXpert test set, one disease label in the RSNA dataset, and one disease
label in the SIIM dataset. There are two possible options (yes and no).

  * •

Single Disease Identification (864 samples): Given a CXR, the FM is tasked
with identifying the disease present in the image. We implement single disease
identification with the CheXpert test set (where disease labels are obtained
from expert radiologist annotations) and OpenI (where disease labels are
obtained from Medical Subject Headings (MeSH) codes). There are four possible
options associated with each question, and each option includes a single
disease label (e.g. “pneumonia").

  * •

Multi-Disease Identification (1387 samples): Given a CXR, the FM is tasked
with identifying a set of multiple diseases present in the image. We implement
multi-disease classification using CheXpert and OpenI. There are four possible
options associated with each question, and each option includes a set of
multiple diseases (e.g. “pneumonia, pleural effusion, cardiomegaly").

  * •

Visual-Question-Answering (1319 samples): We evaluate FMs across two standard
VQA benchmarks: SLAKE and Rad-Restruct. SLAKE consists of questions with two
options (yes and no), and Rad-Restruct consists of questions with between two
and four options.

  * •

Image-Text Reasoning (380 samples): Given a CXR, the FM is tasked with
identifying the disease in the image. In contrast to the single-disease
classification task, this task employs hard negatives; each question is
associated with two challenging options, distinguished only by a single word
indicating location or severity (e.g. “left-sided pleural effusion" vs.
“right-sided pleural effusion"). We implement image-text reasoning with the
OpenI dataset.

Table 3: Results of evaluation axis 2 of CheXbench for the task of findings
generation comparing CheXagent with baseline medical-domain FMs using various
metrics. Private Dataset MIMIC-CXR Model Size BERT-S CheXbert-S RadGraph-S
BERT-S CheXbert-S RadGraph-S MedFlamingo 8B 8.5 2.7 1.7 10.4 3.2 2.2 LLaVA-Med
8B 12.5 17.0 4.2 6.2 17.5 4.0 RadFM 14B 35.7 12.7 5.1 45.7 17.5 10.9 XrayGPT
8B 40.1 23.4 9.0 44.0 24.2 11.2 CheXagent 8B 46.6 23.7 14.6 50.4 24.9 18.6

Table 4: Results of eval. axis 2 on findings summarization. MIMIC-CXR Model
Size Rouge-L Llama-2 7B 20.3 Vicuna 7B 21.5 FLAN-T5 11B 42.5 FLAN-UL2 20B 42.1
CheXagent 8B 40.3

Evaluation Axis 2 - Textual Understanding: We additionally evaluate the
ability of CheXagent and baseline FMs to generate and summarize text. To this
end, we introduce the following 2 tasks: we evaluate open-ended responses
using a combination of automated metrics (ROUGE-L [46], CheXbert-Score [72],
BERT-Score [95], RadGraph-Score [20], and GPT-4) and human expert evaluations
from five radiologists.

  * •

Findings Section Generation: Given an image, the FM is tasked with generating
the findings section of the radiology report. This task involves identifying
key features of the image, such as the presence of abnormalities. We implement
the findings section generation task with MIMIC-CXR. Since existing medical
FMs are often trained on MIMIC-CXR, we also evaluate the models on a private
dataset.

  * •

Findings Summarization: Given the findings section of a radiology report, the
FM is tasked with summarizing the key observations into a concise statement.
This task does not include images. We evaluate Findings Summarization on
MIMIC-CXR.

###  5.2 Evaluation Results

In our study, we employ CheXbench to compare CheXagent against two general-
domain instruction-tuned FMs, InstructBLIP and BLIP2, which achieve state-of-
the-art performance in previous research [42]. Additionally, we compare
CheXagent with four medical FMs: XrayGPT, MedFlamingo, RadFM, and LLaVA-Med
[74, 51, 43, 88]. This comparison aims to provide a comprehensive
understanding of CheXagent’s performance in relation to both general and
medical-specific models.

Table 2 provides results on the six tasks associated with evaluation axis 1.
CheXagent demonstrates superior performance across image perception tasks,
achieving an average improvement of 97.5% over general-domain FMs and an
average improvement of 55.7% over medical FMs. We provide a detailed breakdown
of CheXagent performance:

  * •

On view classification, CheXagent achieves near perfect performance,
demonstrating a 68.7 point (238%) improvement over the closest baseline on
MIMIC-CXR and a 57.7 point (148%) improvement over the closest baseline on
CheXpert. The majority of general-domain and medical-domain FMs demonstrate
performance near random.

  * •

On binary disease classification, single disease, and multi-disease
identification, CheXagent demonstrates an average improvement of 11.6 points
over the closest baseline. In particular, we note that CheXagent demonstrates
superior performance on the SIIM and OpenI datasets, which were completely
held-out from training CheXagent; this suggests the ability of our model to
generalize to diverse CXRs. On multi-disease classification on OpenI, XrayGPT
outperforms CheXagent by 1.1 points; this could be attributed to the fact that
XrayGPT was fine-tuned on samples from OpenI, which would bias its
classification accuracy on this task.

  * •

On visual question answering, CheXagent outperforms baselines on Rad-Restruct
(achieving a 8.6 point improvement over the closest baseline) and is
competitive with existing approaches on SLAKE. We note that the SLAKE and Rad-
Restruct datasets were completely held-out from CheXagent training, whereas
RadFM was trained using samples from SLAKE.

  * •

On image-text reasoning, CheXagent achieves a 4.3 point (7.8%) improvement
over the closest baseline, MedFlamingo. We observe that the image-text
reasoning task, which was explicitly designed to include hard negatives, is
particularly challenging for all evaluated FMs.

Tables 5.1, 5.1 and Figure 3 provide results on the two tasks associated with
evaluation axis 2, medical text generation and summarization. We provide a
detailed breakdown of CheXagent performance below:

  * •

On findings section generation, CheXagent outperforms all medical FMs across
all metrics on both the private dataset and MIMIC-CXR. In particular,
CheXagent achieves an average improvment of 6.5 points on RadGraph scores and
0.5 points on CheXbert scores; this is notable since these metrics directly
evaluate factual correctness. Figure 3 shows results from automated
evaluations using GPT-4. For 152 randomly selected samples from our private
dataset, we provided GPT-4 with a reference report, the findings section
generated by CheXagent, and the findings section generated by each of the
medical FM baselines; GPT-4 was then prompted to select the report with the
highest accuracy. As shown in Figure 3, our GPT-4 evaluations demonstrate that
CheXagent generates high-quality reports when compared to other medical FMs.

  * •

For findings summarization, as shown in Table 5.1, CheXagent outperforms LLMs
of comparable size on the Rouge-L metric and achieves comparable performance
to LLMs with more than twice the number of parameters [80, 81].

Figure 3: GPT-4 evaluations demonstrate that the reports generated by
CheXagent outperform medical-domain FMs for the findings generation task on
MIMIC-CXR. Figure 4: Reader study with five radiologists. Top: Study design
comparing the findings sections of CheXagent vs. that of human experts for two
tasks (rows) across three attributes (columns). Bottom: Results. Compared to
humans, CheXagent achieves parity in report summarization, while the results
demonstrate a gap between CheXagent report generation and human-level
expertise. Figure 5: Comparison of CheXagent (top) against physician (bottom)
on report generation. Radiologist reports often refer to past studies
(purple), a context not available to CheXagent. The presence of these
references in the training set perhaps contributes to CheXagent’s error (red)
in an otherwise accurate report (green), motivating future work for vision-
language models.

##  6 Human Evaluation

To complement the quantitative results presented in Section 5.2, we conduct a
reader study in which five radiologists compare text generated by CheXagent
against text written by a physician. This study includes the two textual
understanding tasks described in Section 5.1: findings generation and findings
summarization on MIMIC-CXR. For both tasks, each radiologist views the same
set of 20 randomly selected samples, which contain the task input as well as
an A/B comparison (CheXagent vs. physician) of the task outputs. Radiologists
then grade these outputs using a five-point Likert scale across: completeness,
correctness, and conciseness.

Study results in Figure 4 show that CheXagent is comparable to physicians for
findings summarization and that text from human experts achieves higher scores
in terms of findings generation. Although CheXagent outperforms other vision-
language models in quantitative metrics, these findings indicate that further
improvement could facilitate closing the gap between CheXagent and human
radiologists.

We now discuss qualitative findings to better understand the opportunity for
future improvement in vision-language models. As shown in Figure 5,
physicians’ reports typically reference past patient studies to track changes
over time. Thus, the capabilities of CheXagent, which is trained on cross-
sectional reports, could be further improved by incorporating longitudinal
data during training.

We also observed that the model frequently produced a uniform distance
measurement of 3.5 cm in its generated text, irrespective of context. This
issue arises because CheXagent is not designed to directly estimate physical
distances or other quantities from images. Thus, addressing this limitation
represents yet another avenue for advancement in vision-language models.

##  7 Fairness Evaluation

Recent studies [28, 68] highlight the presence of biases in AI models used in
radiology, raising concerns about their equitable application across diverse
populations. For fairness evaluation, we test CheXagent on a subset of the
CheXpert public test set, annotated by two expert radiologists to avoid label
noise [65], using frontal view CXRs from individuals self-reporting as Asian,
White, or Black. In total we use 159 unique subjects labeled as “No Finding"
and “Cardiomegaly". To create a balanced test set, we resample with
replacement [27] to ensure balanced disease prevalence and subgroup
representation. We generate 2000 bootstrap samples to calculate mean and
standard deviation of F1-scores.

We evaluate model performance for the detection of cardiomegaly with the
prompt “Does this chest X-ray contain cardiomegaly?" with possible answers
“Yes" and “No". Detection of cardiomegaly is crucial for early diagnosis, and
treatment of heart conditions [2], with studies revealing significant
disparities in heart disease death rates based on demographic factors [79].

Figure 6: Evaluation of CheXagent subgroup performance on cardiomegaly
classification investigating potential model biases. F1 Scores vary across
sex, racial groups, and age categories.

Our findings, shown in Fig. 6 reveal disparities; F1-scores are higher for
males compared to females and vary across racial groups, with the model
performing best for the Black subgroup and worst for the Asian subgroup. This
could reflect inherent differences in the presentation of cardiomegaly across
races, and could be influenced by the limited samples of 14 Black and 30
unique Asian subjects included in the test set. Age-wise, the model performs
better for the 65+ age group compared to the 0-65 group, potentially due to a
higher prevalence of cardiomegaly in older patients, and age-related
physiological differences. These results are consistent with existing
literature [28] and underscore the need for continued efforts in mitigating
biases in AI models used in healthcare. An effective approach to address this
issue is curating larger and more diverse datasets, which can help in
developing models that are more representative and equitable across different
patient demographics.

##  8 Conclusion

In conclusion, our work represents progress towards automated CXR
interpretation. We introduce (i) CheXinstruct, an instruction-tuning dataset,
(ii) CheXagent, an 8B-parameter vision-language FM and demonstrate its
abilities through (iii) CheXbench, our benchmarking framework including 8
tasks over 7 datasets. CheXagent achieves improvement in visual perception and
text generation tasks compared to general- and medical-domain LLMs and is
validated by five expert radiologists. Furthermore, our fairness analysis
across sex, race, and age contributes to the ongoing efforts to enhance model
transparency in healthcare AI. The release of CheXinstruct, CheXagent, and
CheXbench to the public domain not only underscores our commitment to
advancing medical AI but also sets a new benchmark for future developments in
this critical area of research.

### Acknowledgements

MV is supported by graduate fellowship awards from the Department of Defense
(NDSEG) and the Knight-Hennessy Scholars program at Stanford University. AC
receives research support from the National Institutes of Health (grants - R01
HL167974, R01 AR077604, R01 EB002524, R01 AR079431, P41 EB027060, and
contracts 75N92020C00008, 75N92020C00021); and from GE Healthcare and Philips.
Research reported in this publication was made possible in part by the
National Institute of Biomedical Imaging and Bioengineering (NIBIB) of the
National Institutes of Health under contracts 75N92020C00008 and
75N92020C00021, and by grant #1R18HS028955 from the Agency for Health Research
and Quality. This work is supported in part by MIDRC (The Medical Imaging and
Data Resource Center), made possible by the National Institute of Biomedical
Imaging and Bioengineering (NIBIB) of the National Institutes of Health under
contract 75N92020D00021. We acknowledge support by Stability AI in providing
computational support for this work.

## References

  * Alayrac et al. [2022] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Alghamdi et al. [2020] S. S. Alghamdi, I. Abdelaziz, M. Albadri, S. Alyanbaawi, R. Aljondi, and A. Tajaldeen.  Study of cardiomegaly using chest x-ray.  _Journal of Radiation Research and Applied Sciences_ , 13(1):460–467, 2020.
  * Anil et al. [2023] R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al.  Palm 2 technical report.  _arXiv preprint arXiv:2305.10403_ , 2023.
  * Bae et al. [2023] S. Bae, D. Kyung, J. Ryu, E. Cho, G. Lee, S. Kweon, J. Oh, L. Ji, E. I. Chang, T. Kim, et al.  Ehrxqa: A multi-modal question answering dataset for electronic health records with chest x-ray images.  _arXiv preprint arXiv:2310.18652_ , 2023.
  * Bai et al. [2023] J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou.  Qwen-vl: A frontier large vision-language model with versatile abilities.  _arXiv preprint arXiv:2308.12966_ , 2023.
  * Bannur et al. [2023] S. Bannur, S. Hyland, Q. Liu, F. Pérez-García, M. Ilse, D. C. de Castro, B. Boecking, H. Sharma, K. Bouzid, A. Schwaighofer, et al.  Ms-cxr-t: Learning to exploit temporal structure for biomedical vision-language processing, 2023.
  * Ben Abacha et al. [2019] A. Ben Abacha, S. A. Hasan, V. V. Datla, D. Demner-Fushman, and H. Müller.  Vqa-med: Overview of the medical visual question answering task at imageclef 2019.  In _Proceedings of CLEF (Conference and Labs of the Evaluation Forum) 2019 Working Notes_. 9-12 September 2019, 2019.
  * Boecking et al. [2022] B. Boecking, N. Usuyama, S. Bannur, D. C. Castro, A. Schwaighofer, S. Hyland, M. Wetscherek, T. Naumann, A. Nori, J. Alvarez-Valle, et al.  Making the most of text semantics to improve biomedical vision–language processing.  In _European conference on computer vision_ , pages 1–21. Springer, 2022.
  * Bommasani et al. [2021] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021.
  * Brown et al. [2020] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Bustos et al. [2020] A. Bustos, A. Pertusa, J.-M. Salinas, and M. De La Iglesia-Vaya.  Padchest: A large chest x-ray image dataset with multi-label annotated reports.  _Medical image analysis_ , 66:101797, 2020.
  * Chambon et al. [2022a] P. Chambon, C. Bluethgen, J.-B. Delbrouck, R. Van der Sluijs, M. Połacin, J. M. Z. Chaves, T. M. Abraham, S. Purohit, C. P. Langlotz, and A. Chaudhari.  Roentgen: Vision-language foundation model for chest x-ray generation, 2022a.  URL https://arxiv.org/abs/2211.12737.
  * Chambon et al. [2022b] P. Chambon, C. Bluethgen, C. P. Langlotz, and A. Chaudhari.  Adapting pretrained vision-language foundational models to medical imaging domains, 2022b.
  * Chen et al. [2023a] X. Chen, J. Djolonga, P. Padlewski, B. Mustafa, S. Changpinyo, J. Wu, C. R. Ruiz, S. Goodman, X. Wang, Y. Tay, et al.  Pali-x: On scaling up a multilingual vision and language model.  _arXiv preprint arXiv:2305.18565_ , 2023a.
  * Chen et al. [2023b] Z. Chen, M. Varma, X. Wan, C. Langlotz, and J.-B. Delbrouck.  Toward expanding the scope of radiology report summarization to multiple anatomies and modalities.  In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 469–484, Toronto, Canada, July 2023b. Association for Computational Linguistics.
  * Chiang et al. [2023] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, et al.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  _See https://vicuna. lmsys. org (accessed 14 April 2023)_ , 2023.
  * Chowdhery et al. [2022] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Chung et al. [2022] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, et al.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_ , 2022.
  * Dehghani et al. [2023] M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. P. Steiner, M. Caron, R. Geirhos, I. Alabdulmohsin, et al.  Scaling vision transformers to 22 billion parameters.  In _International Conference on Machine Learning_ , pages 7480–7512. PMLR, 2023.
  * Delbrouck et al. [2022] J.-B. Delbrouck, P. Chambon, C. Bluethgen, E. Tsai, O. Almusa, and C. Langlotz.  Improving the factual correctness of radiology report generation with semantic rewards.  In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 4348–4360, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.findings-emnlp.319.  URL https://aclanthology.org/2022.findings-emnlp.319.
  * Demner-Fushman et al. [2012] D. Demner-Fushman, S. Antani, M. Simpson, and G. R. Thoma.  Design and development of a multimodal biomedical information retrieval system.  _Journal of Computing Science and Engineering_ , 6(2):168–177, 2012.
  * Devlin et al. [2019] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, 2019.
  * Dosovitskiy et al. [2020] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Feng et al. [2021] S. Feng, D. Azzollini, J. S. Kim, C.-K. Jin, S. P. Gordon, J. Yeoh, E. Kim, M. Han, A. Lee, A. Patel, et al.  Curation of the candid-ptx dataset with free-text reports.  _Radiology: Artificial Intelligence_ , 3(6):e210136, 2021.
  * Fu et al. [2023] C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, Z. Qiu, W. Lin, J. Yang, X. Zheng, et al.  Mme: A comprehensive evaluation benchmark for multimodal large language models.  _arXiv preprint arXiv:2306.13394_ , 2023.
  * Gao et al. [2021] L. Gao, J. Tow, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, K. McDonell, N. Muennighoff, J. Phang, L. Reynolds, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou.  A framework for few-shot language model evaluation, Sept. 2021.  URL https://doi.org/10.5281/zenodo.5371628.
  * Glocker et al. [2023a] B. Glocker, C. Jones, M. Bernhardt, and S. Winzeck.  Algorithmic encoding of protected characteristics in chest x-ray disease detection models.  _Ebiomedicine_ , 89, 2023a.
  * Glocker et al. [2023b] B. Glocker, C. Jones, M. Roschewitz, and S. Winzeck.  Risk of bias in chest radiography deep learning foundation models.  _Radiology: Artificial Intelligence_ , 5(6):e230060, 2023b.
  * Gu et al. [2021] Y. Gu, R. Tinn, H. Cheng, M. Lucas, N. Usuyama, X. Liu, T. Naumann, J. Gao, and H. Poon.  Domain-specific language model pretraining for biomedical natural language processing.  _ACM Transactions on Computing for Healthcare (HEALTH)_ , 3(1):1–23, 2021.
  * Holste et al. [2023] G. Holste, S. Wang, A. Jaiswal, Y. Yang, M. Lin, Y. Peng, and A. Wang.  Cxr-lt: Multi-label long-tailed classification on chest x-rays.  _PhysioNet_ , 2023.
  * Hu et al. [2023] X. Hu, L. Gu, Q. An, M. Zhang, L. Liu, K. Kobayashi, T. Harada, R. M. Summers, and Y. Zhu.  Expert knowledge-aware image difference graph representation learning for difference-aware medical visual question answering.  In _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , pages 4156–4165, 2023.
  * Irvin et al. [2019] J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya, et al.  Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pages 590–597, 2019.
  * Iyeke et al. [2022] L. Iyeke, R. Moss, R. Hall, J. Wang, L. Sandhu, B. Appold, E. Kalontar, D. Menoudakos, M. Ramnarine, S. P. LaVine, et al.  Reducing unnecessary ‘admission’chest x-rays: An initiative to minimize low-value care.  _Cureus_ , 14(10), 2022.
  * Jaeger et al. [2014] S. Jaeger, S. Candemir, S. Antani, Y.-X. J. Wáng, P.-X. Lu, and G. Thoma.  Two public chest x-ray datasets for computer-aided screening of pulmonary diseases.  _Quantitative imaging in medicine and surgery_ , 4(6):475, 2014.
  * Jiang et al. [2023] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_ , 2023.
  * Johnson et al. [2019] A. E. Johnson, T. J. Pollard, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, Y. Peng, Z. Lu, R. G. Mark, S. J. Berkowitz, and S. Horng.  Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs.  _arXiv preprint arXiv:1901.07042_ , 2019.
  * Kayser et al. [2022] M. Kayser, C. Emde, O.-M. Camburu, G. Parsons, B. Papiez, and T. Lukasiewicz.  Explaining chest x-ray pathologies in natural language.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 701–713. Springer, 2022.
  * Kirillov et al. [2023] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023.
  * Lau et al. [2018] J. J. Lau, S. Gayen, A. Ben Abacha, and D. Demner-Fushman.  A dataset of clinically generated visual questions and answers about radiology images.  _Scientific data_ , 5(1):1–10, 2018.
  * Lee et al. [2020] J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240, 2020.
  * Lewis et al. [2020] P. Lewis, M. Ott, J. Du, and V. Stoyanov.  Pretrained language models for biomedical and clinical tasks: understanding and extending the state-of-the-art.  In _Proceedings of the 3rd Clinical Natural Language Processing Workshop_ , pages 146–157, 2020.
  * Li et al. [2023a] B. Li, R. Wang, G. Wang, Y. Ge, Y. Ge, and Y. Shan.  Seed-bench: Benchmarking multimodal llms with generative comprehension.  _arXiv preprint arXiv:2307.16125_ , 2023a.
  * Li et al. [2023b] C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao.  Llava-med: Training a large language-and-vision assistant for biomedicine in one day.  _arXiv preprint arXiv:2306.00890_ , 2023b.
  * Li et al. [2022] J. Li, D. Li, C. Xiong, and S. Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pages 12888–12900. PMLR, 2022.
  * Li et al. [2023c] J. Li, D. Li, S. Savarese, and S. Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023c.
  * Lin [2004] C.-Y. Lin.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81, 2004.
  * Liu et al. [2023] H. Liu, C. Li, Q. Wu, and Y. J. Lee.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_ , 2023.
  * Luo et al. [2022] R. Luo, L. Sun, Y. Xia, T. Qin, S. Zhang, H. Poon, and T.-Y. Liu.  Biogpt: generative pre-trained transformer for biomedical text generation and mining.  _Briefings in Bioinformatics_ , 23(6):bbac409, 2022.
  * Mei et al. [2022] X. Mei, Z. Liu, P. M. Robson, B. Marinelli, M. Huang, A. Doshi, A. Jacobi, C. Cao, K. E. Link, T. Yang, et al.  Radimagenet: an open radiologic deep learning research dataset for effective transfer learning.  _Radiology: Artificial Intelligence_ , 4(5):e210315, 2022.
  * Miura et al. [2021] Y. Miura, Y. Zhang, E. Tsai, C. Langlotz, and D. Jurafsky.  Improving factual completeness and consistency of image-to-text radiology report generation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5288–5304, 2021.
  * Moor et al. [2023] M. Moor, Q. Huang, S. Wu, M. Yasunaga, C. Zakka, Y. Dalmia, E. P. Reis, P. Rajpurkar, and J. Leskovec.  Med-flamingo: a multimodal medical few-shot learner.  _arXiv preprint arXiv:2307.15189_ , 2023.
  * Mukherjee et al. [2023] S. Mukherjee, A. Mitra, G. Jawahar, S. Agarwal, H. Palangi, and A. Awadallah.  Orca: Progressive learning from complex explanation traces of gpt-4.  _arXiv preprint arXiv:2306.02707_ , 2023.
  * Nguyen et al. [2022] H. Q. Nguyen, K. Lam, L. T. Le, H. H. Pham, D. Q. Tran, D. B. Nguyen, D. D. Le, C. M. Pham, H. T. Tong, D. H. Dinh, et al.  Vindr-cxr: An open dataset of chest x-rays with radiologist’s annotations.  _Scientific Data_ , 9(1):429, 2022.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report, 2023.
  * Pavlova et al. [2022] M. Pavlova, T. Tuinstra, H. Aboutalebi, A. Zhao, H. Gunraj, and A. Wong.  Covidx cxr-3: a large-scale, open-source benchmark dataset of chest x-ray images for computer-aided covid-19 diagnostics.  _arXiv preprint arXiv:2206.03671_ , 2022.
  * Pelka et al. [2018] O. Pelka, S. Koitka, J. Rückert, F. Nensa, and C. M. Friedrich.  Radiology objects in context (roco): a multimodal image dataset.  In _Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis_ , pages 180–189. Springer, 2018.
  * Pellegrini et al. [2023] C. Pellegrini, M. Keicher, E. Özsoy, and N. Navab.  Rad-restruct: A novel vqa benchmark and method for structured radiology reporting.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 409–419. Springer, 2023.
  * Pham et al. [2022] H. H. Pham, T. T. Tran, and H. Q. Nguyen.  Vindr-pcxr: An open, large-scale pediatric chest x-ray dataset for interpretation of common thoracic diseases.  _PhysioNet_ , 2022.
  * Radford et al. [2021] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Ramesh et al. [2021] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pages 8821–8831. PMLR, 2021.
  * Ramesh et al. [2022] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022.
  * Reis et al. [2022] E. P. Reis, J. P. de Paiva, M. C. da Silva, G. A. Ribeiro, V. F. Paiva, L. Bulgarelli, H. M. Lee, P. V. Santos, V. M. Brito, L. T. Amaral, et al.  Brax, brazilian labeled chest x-ray dataset.  _Scientific Data_ , 9(1):487, 2022.
  * Rombach et al. [2022] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Saharia et al. [2022] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Saporta et al. [2022] A. Saporta, X. Gui, A. Agrawal, A. Pareek, S. Q. Truong, C. D. Nguyen, V.-D. Ngo, J. Seekins, F. G. Blankenberg, A. Y. Ng, et al.  Benchmarking saliency methods for chest x-ray interpretation.  _Nature Machine Intelligence_ , 4(10):867–878, 2022.
  * Schuhmann et al. [2022] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _Advances in Neural Information Processing Systems_ , 35:25278–25294, 2022.
  * Seah et al. [2021] J. C. Seah, C. H. Tang, Q. D. Buchlak, X. G. Holt, J. B. Wardman, A. Aimoldin, N. Esmaili, H. Ahmad, H. Pham, J. F. Lambert, et al.  Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: a retrospective, multireader multicase study.  _The Lancet Digital Health_ , 3(8):e496–e506, 2021.
  * Seyyed-Kalantari et al. [2021] L. Seyyed-Kalantari, H. Zhang, M. B. McDermott, I. Y. Chen, and M. Ghassemi.  Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations.  _Nature medicine_ , 27(12):2176–2182, 2021.
  * Shih et al. [2019] G. Shih, C. C. Wu, S. S. Halabi, M. D. Kohli, L. M. Prevedello, T. S. Cook, A. Sharma, J. K. Amorosa, V. Arteaga, M. Galperin-Aizenberg, et al.  Augmenting the national institutes of health chest radiograph dataset with expert annotations of possible pneumonia.  _Radiology: Artificial Intelligence_ , 1(1):e180041, 2019.
  * Shin et al. [2020] H.-C. Shin, Y. Zhang, E. Bakhturina, R. Puri, M. Patwary, M. Shoeybi, and R. Mani.  Biomegatron: Larger biomedical domain language model.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4700–4706, 2020.
  * Singhal et al. [2023] K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, L. Hou, K. Clark, S. Pfohl, H. Cole-Lewis, D. Neal, et al.  Towards expert-level medical question answering with large language models.  _arXiv preprint arXiv:2305.09617_ , 2023.
  * Smit et al. [2020] A. Smit, S. Jain, P. Rajpurkar, A. Pareek, A. Y. Ng, and M. Lungren.  Combining automatic labelers and expert annotations for accurate radiology report labeling using bert.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1500–1519, 2020.
  * Sun et al. [2023] Q. Sun, Y. Fang, L. Wu, X. Wang, and Y. Cao.  Eva-clip: Improved training techniques for clip at scale.  _arXiv preprint arXiv:2303.15389_ , 2023.
  * Thawkar et al. [2023] O. Thawkar, A. Shaker, S. S. Mullappilly, H. Cholakkal, R. M. Anwer, S. Khan, J. Laaksonen, and F. S. Khan.  Xraygpt: Chest radiographs summarization using medical vision-language models.  _arXiv preprint arXiv:2306.07971_ , 2023.
  * Tiu et al. [2022] E. Tiu, E. Talius, P. Patel, C. P. Langlotz, A. Y. Ng, and P. Rajpurkar.  Expert-level detection of pathologies from unannotated chest x-ray images via self-supervised learning.  _Nature Biomedical Engineering_ , 6(12):1399–1406, 2022.
  * Touvron et al. [2023a] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023a.
  * Touvron et al. [2023b] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023b.
  * Tu et al. [2023] T. Tu, S. Azizi, D. Driess, M. Schaekermann, M. Amin, P.-C. Chang, A. Carroll, C. Lau, R. Tanno, I. Ktena, et al.  Towards generalist biomedical ai.  _arXiv preprint arXiv:2307.14334_ , 2023.
  * Van Dyke et al. [2018] M. Van Dyke, S. Greer, E. Odom, L. Schieb, A. Vaughan, M. Kramer, and M. Casper.  Heart disease death rates among blacks and whites aged 35 years—united states, 1968–2015.  _MMWR Surveillance Summaries_ , 67(5):1, 2018.
  * Van Veen et al. [2023a] D. Van Veen, C. Van Uden, M. Attias, A. Pareek, C. Bluethgen, M. Polacin, W. Chiu, J.-B. Delbrouck, J. Zambrano Chaves, C. Langlotz, A. Chaudhari, and J. Pauly.  RadAdapt: Radiology report summarization via lightweight domain adaptation of large language models.  In D. Demner-fushman, S. Ananiadou, and K. Cohen, editors, _The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks_ , pages 449–460, Toronto, Canada, July 2023a. Association for Computational Linguistics.  doi: 10.18653/v1/2023.bionlp-1.42.  URL https://aclanthology.org/2023.bionlp-1.42.
  * Van Veen et al. [2023b] D. Van Veen, C. Van Uden, L. Blankemeier, J.-B. Delbrouck, A. Aali, C. Bluethgen, A. Pareek, M. Polacin, W. Collins, N. Ahuja, et al.  Clinical text summarization: Adapting large language models can outperform human experts.  _arXiv preprint arXiv:2309.07430_ , 2023b.
  * Vayá et al. [2020] M. D. L. I. Vayá, J. M. Saborit, J. A. Montell, A. Pertusa, A. Bustos, M. Cazorla, J. Galant, X. Barber, D. Orozco-Beltrán, F. García-García, et al.  Bimcv covid-19+: a large annotated dataset of rx and ct images from covid-19 patients.  _arXiv preprint arXiv:2006.01174_ , 2020.
  * Wang et al. [2023] W. Wang, Q. Lv, W. Yu, W. Hong, J. Qi, Y. Wang, J. Ji, Z. Yang, L. Zhao, X. Song, et al.  Cogvlm: Visual expert for pretrained language models.  _arXiv preprint arXiv:2311.03079_ , 2023.
  * Wang et al. [2017] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers.  Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2097–2106, 2017.
  * Wang et al. [2022] Z. Wang, Z. Wu, D. Agarwal, and J. Sun.  Medclip: Contrastive learning from unpaired medical images and text.  _arXiv preprint arXiv:2210.10163_ , 2022.
  * Wei et al. [2021] J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_ , 2021.
  * Wu et al. [2023a] C. Wu, W. Lin, X. Zhang, Y. Zhang, Y. Wang, and W. Xie.  Pmc-llama: Towards building open-source language models for medicine.  _arXiv preprint arXiv:2305.10415_ , 2023a.
  * Wu et al. [2023b] C. Wu, X. Zhang, Y. Zhang, Y. Wang, and W. Xie.  Towards generalist foundation model for radiology.  _arXiv preprint arXiv:2308.02463_ , 2023b.
  * Xu et al. [2023] S. Xu, L. Yang, C. Kelly, M. Sieniek, T. Kohlberger, M. Ma, W.-H. Weng, A. Kiraly, S. Kazemzadeh, Z. Melamed, et al.  Elixr: Towards a general purpose x-ray artificial intelligence system through alignment of large language models and radiology vision encoders.  _arXiv preprint arXiv:2308.01317_ , 2023.
  * Xu et al. [2022] Z. Xu, Y. Shen, and L. Huang.  Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning.  _arXiv preprint arXiv:2212.10773_ , 2022.
  * Xue et al. [2015] Z. Xue, S. Candemir, S. Antani, L. R. Long, S. Jaeger, D. Demner-Fushman, and G. R. Thoma.  Foreign object detection in chest x-rays.  In _2015 IEEE international conference on bioinformatics and biomedicine (BIBM)_ , pages 956–961. IEEE, 2015.
  * Yang et al. [2022] X. Yang, A. Chen, N. PourNejatian, H. C. Shin, K. E. Smith, C. Parisien, C. Compas, C. Martin, A. B. Costa, M. G. Flores, et al.  A large language model for electronic health records.  _NPJ Digital Medicine_ , 5(1):194, 2022.
  * Yu et al. [2023] F. Yu, M. Endo, R. Krishnan, I. Pan, A. Tsai, E. P. Reis, E. K. U. N. Fonseca, H. M. H. Lee, Z. S. H. Abad, A. Y. Ng, et al.  Evaluating progress in automatic chest x-ray radiology report generation.  _Patterns_ , 4(9), 2023.
  * Yu et al. [2022] J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu.  Coca: Contrastive captioners are image-text foundation models.  _arXiv preprint arXiv:2205.01917_ , 2022.
  * Zhang et al. [2019] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_ , 2019.
  * Zhang et al. [2023a] X. Zhang, C. Wu, Y. Zhang, W. Xie, and Y. Wang.  Knowledge-enhanced visual-language pre-training on chest radiology images.  _Nature Communications_ , 14(1):4542, 2023a.
  * Zhang et al. [2023b] X. Zhang, C. Wu, Z. Zhao, W. Lin, Y. Zhang, Y. Wang, and W. Xie.  Pmc-vqa: Visual instruction tuning for medical visual question answering.  _arXiv preprint arXiv:2305.10415_ , 2023b.
  * Zou et al. [2023] X. Zou, J. Yang, H. Zhang, F. Li, L. Li, J. Gao, and Y. J. Lee.  Segment everything everywhere all at once.  _arXiv preprint arXiv:2304.06718_ , 2023.

## A. Additional details on CheXinstruct tasks

Table 5: Detailed task information of CheXinstruct, where three levels (i.e.,
capabilities, tasks, datasets) and the corresponding task descriptions are
shown.

Capability | Task | Dataset | Description
---|---|---|---
Coarse-grained Image Understanding | Disease Classification | ChestX-ray14 | Given an <IMAGE>, the model is required to diagnose if the <DISEASE> exists.
CheXpert
MIMIC-CXR
PadChest
RSNA
COVIDX-CXR-3
CXR-LT
BRAX
NLM-TB
Temporal image classification | MS-CXR-T | Given the <PRIOR IMAGE> and <CURRENT IMAGE>, identify the progression <LABEL>.
View Classification | MIMIC-CXR | Given the <IMAGE>, identify its <VIEW>.
View Matching | MIMIC-CXR | Given the <IMAGE 1> and <IMAGE 2>, if they belong to the same study.
Fine-grained Image Understanding | Abnormality Detection | VinDr-CXR | Given the <IMAGE>, localize the <REGION> of abnormalities.
VinDr-PCXR
Abnormality Grounding | VinDr-CXR | Given the <IMAGE>, localize the <REGION> of <abnormality>.
VinDr-PCXR
Pneumothorax Segmentation | Candid | Given the <IMAGE>, segment the <REGION> of pneumothorax.
SIIM
Rib Fracture Segmentation | Candid | Given the <IMAGE>, segment the <REGION> of rib fractures.
Chest Tube Segmentation | Candid | Given the <IMAGE>, segment the <REGION> of Chest Tubes.
Foreign Object Detection | Object-CXR | Given the <IMAGE>, detect the <REGION> of external objects.
Phrase Grounding | MS-CXR | Given the <IMAGE> and <PHRASE>, identify the <REGION> of <PHRASE>.
Grounded Captioning | MS-CXR | Given the <IMAGE> and <REGION>, generate a <CAPTION>.
Grounded Diagnosis | MS-CXR | Given the <IMAGE> and <REGION>, generate a <DIAGNOSIS>.
Grounded Phrase Extraction | MS-CXR | Given the <IMAGE>, its <REPORT>, and <REGION>s, extract a <PHRASE>.
Phrase Extraction and Grounding | MS-CXR | Given the <IMAGE> and its <REPORT>, extract a <PHRASE> and localize its <REGION>.
Text Generation | Report Generation | PadChest | Given the <IMAGE>, generate its <REPORT>.
BIMCV-COVID19
Findings Generation | MIMIC-CXR | Given the <IMAGE>, generate its <FINDINGS>.
MIMIC-CXR-Struct
OpenI
Impression Generetion | MIMIC-CXR | Given the <IMAGE>, generate its <IMPRESSION>.
MIMIC-CXR-Struct
OpenI
Candid
Progression Findings Generation | MIMIC-CXR | Given the <REFERENCE IMAGE> and <MAIN IMAGE>, generate its <FINDINGS>.
Progression Impression Generation | MIMIC-CXR | Given the <REFERENCE IMAGE> and <MAIN IMAGE>, generate its <IMPRESSION>.
Findings Summarization | MIMIC-CXR | Given the <FINDINGS>, generate its <IMPRESSION>.
OpenI
MIMIC-III
Caption Generation | ROCO | Given the <IMAGE>, generate its <CAPTION>.
Local Findings Generation | MIMIC-CXR-Struct | Given the <IMAGE> and a anatomy, generate its <FINDINGS>.
Local Impression Generation | MIMIC-CXR-Struct | Given the <IMAGE> and a anatomy, generate its <IMPRESSION>.
Question Answering | Open-ended VQA | VQA-RAD | Given the content of the given <IMAGE>, answer the <QUESTION>.
SLAKE
MedVQA-2019
PMC-VQA
Rad-Restruct
MIMIC-CXR-VQA
Close-ended VQA | VQA-RAD | Given the content of the given <IMAGE>, choose one option from the <OPTIONS> to answer the <QUESTION>.
SLAKE
PMC-VQA
Rad-Restruct
MIMIC-CXR-VQA
Difference VQA | MIMIC-Diff-VQA | Given a <REFERENCE IMAGE> and a <MAIN IMAGE>, answer the <QUESTION>.
Text QA | RadQA | Given <PARAGRAPH>, answer the <QUSETION>.
Miscellaneous | Image-Text Matching | MIMIC-CXR | Given the <IMAGE> and <REPORT>, decide if they match.
ROCO
Image-Text Selection | MIMIC-CXR | Given the <IMAGE>, select the text that best matches the image from <OPTIONS>.
ROCO
Report Evaluation | ReXVal | Given a <REFENCE REPORT> and a <GENERATED REPORT>, identify the <ERROR>.
Natural Language Explanation | MIMIC-NLE | Gven an <IMAGE> and <DISEASE>, generate the natural language <EXPLANATION>.
Natural Language Inference | RadNLI | Given a <PREMISE REPORT>, determine whether a <HYPOTHESIS REPORT> is entailment, contradiction, or neutral.
Temporal Sentence Similarity | MS-CXR-T | Given <SENTENCE 1> and <SENTENCE 2>, identify their <SIMILARITY> in terms of disease progression.

We report the detail of each task in Table 5, where three levels (i.e.,
capabilities, tasks, datasets) and the corresponding task descriptions are
shown.

## B. Additional details on CheXinstruct dataset sources

We created CheXinstruct by either (i) collecting existing publicly available
datasets or (ii) curating a dataset with new labels from existing datasets.

We include the following publicly-available datasets: BIMCV-COVID-19 [82],
BRAX [62], CANDID-PTX [24], ChestXray14 [84], CheXpert [32], COVIDX-CXR-3
[55], CXR-LT [30], MedVQA-2019 [7], MIMIC-CXR-VQA [4], MIMIC-Diff-VQA [31],
MIMIC-RRS [15], MIMIC-NLE [37], MS-CXR [8], MS-CXR-T [6], NLM-TB [34], Object-
CXR [91], OpenI [21], PadChest [11], PMC-VQA [97], Rad-Restruct [57], RadNLI
[50], ReXVal [93], ROCO [56], RSNA [69],
SIIM999https://www.kaggle.com/datasets/jesperdramsch/siim-acr-pneumothorax-
segmentation-data, VinDr-CXR [53], VinDr-PCXR [58], and VQA-RAD [39].
Additionally, we use the OpenOrca101010https://huggingface.co/datasets/Open-
Orca/OpenOrca [52] text instruction tuning dataset to improve the diversity of
text instructions.

In order to curate datasets with new labels based on existing datasets, we
perform translation on the radiology reports of BIMCV-COVID-19 and PadChest
from Spanish to English using GPT-4. Additionally, we filter VQA-RAD, SLAKE,
MedVQA-2019, and PMC-VQA to include only chest X-ray images and their
corresponding question-answer pairs. For the MIMIC-CXR dataset, we use GPT-4
to format its Findings and Impression sections to structured ones (called
MIMIC-CXR-Struct), where each sentence is annotated with its corresponding
anatomical structure and paraphrased to eliminate comparisons to the patient’s
prior images.

## C. Extended analysis of CheXinstruct

Table 6: The basic statistics of CheXinstruct, where the numbers of instances
and images of each dataset are shown. The datasets are split following their
official or traditional ways.

Task-Dataset Pair | Instances (train) | Instances (val) | Instances (test) | Images (train) | Images (val) | Images (test)
---|---|---|---|---|---|---
(Open-Ended VQA) VQA-RAD | 713 | 119 | 119 | 73 | 50 | 50
(Open-Ended VQA) SLAKE | 1,093 | 233 | 220 | 90 | 19 | 17
(Open-Ended VQA) MedVQA-2019 | 78 | 11 | 1 | 78 | 11 | 1
(Open-Ended VQA) PMC-VQA | 747 | 0 | 229 | 646 | 0 | 193
(Open-Ended VQA) Rad-Restruct | 142,340 | 17,641 | 17,641 | 2,972 | 374 | 374
(Open-Ended VQA) MIMIC-CXR-VQA | 259,484 | 63,078 | 11,347 | 127,798 | 8,647 | 500
(Close-Ended VQA) VQA-RAD | 417 | 69 | 69 | 69 | 37 | 37
(Close-Ended VQA) SLAKE | 297 | 59 | 64 | 90 | 19 | 17
(Close-Ended VQA) MedVQA-2019 | 1 | 0 | 0 | 1 | 0 | 0
(Close-Ended VQA) PMC-VQA | 682 | 0 | 209 | 596 | 0 | 180
(Close-Ended VQA) Rad-Restruct | 142,340 | 17,641 | 17,641 | 2,972 | 374 | 374
(Close-Ended VQA) MIMIC-CXR-VQA | 162,577 | 39,376 | 6,952 | 106,209 | 8,586 | 500
(Difference VQA) MIMIC-Diff-VQA | 160,054 | 1,325 | 2,967 | 160,377 | 1,332 | 2,969
(Text QA) RadQA | 4,878 | 614 | 614 | 0 | 0 | 0
(Image Classification) ChestXray14 | 78,484 | 11,211 | 22,425 | 78,484 | 11,211 | 22,425
(Image Classification) CheXpert | 223,414 | 234 | 668 | 223,414 | 234 | 668
(Image Classification) MIMIC-CXR | 212,098 | 1,714 | 3,131 | 348,516 | 2,813 | 4,896
(Image Classification) PadChest | 109,845 | 0 | 0 | 160,742 | 0 | 0
(Image Classification) RSNA | 18,678 | 4,003 | 4,003 | 18,678 | 4,003 | 4,003
(Image Classification) COVIDX-CXR-3 | 29,986 | 0 | 400 | 29,986 | 0 | 400
(Image Classification) CXR-LT | 155,349 | 0 | 0 | 255,445 | 0 | 0
(Image Classification) Brax | 23,276 | 0 | 0 | 40,967 | 0 | 0
(Image Classification) NLM-TB | 800 | 0 | 0 | 800 | 0 | 0
(Temporal Image Classification) MS-CXR-T | 985 | 10 | 50 | 1,903 | 19 | 96
(View Classification) MIMIC-CXR | 353,640 | 2,867 | 4,834 | 353,640 | 2,867 | 4,834
(View Matching) MIMIC-CXR | 34,174 | 290 | 320 | 33,797 | 293 | 349
(Phrase Grounding) MS-CXR | 964 | 7 | 189 | 878 | 5 | 164
(Grounded Captioning) MS-CXR | 964 | 7 | 189 | 878 | 5 | 164
(Grounded Diagnosis) MS-CXR | 964 | 7 | 189 | 878 | 5 | 164
(Grounded Phrase Extraction) MS-CXR | 527 | 7 | 9 | 490 | 5 | 8
(Phrase Extraction and Grounding) MS-CXR | 527 | 7 | 9 | 490 | 5 | 8
(Abnormality Detection) VinDr-CXR | 15,000 | 0 | 3,000 | 15,000 | 0 | 3,000
(Abnormality Grounding) VinDr-CXR | 30,282 | 0 | 4,022 | 11,685 | 0 | 1,999
(Grounded Diagnosis) VinDr-CXR | 17,880 | 0 | 2,345 | 4,510 | 0 | 937
(Abnormality Detection) VinDr-PCXR | 7,728 | 0 | 1,397 | 7,728 | 0 | 1,397
(Abnormality Grounding) VinDr-PCXR | 6,648 | 0 | 1,218 | 4,477 | 0 | 809
(Grounded Diagnosis) VinDr-PCXR | 4,788 | 0 | 851 | 2,496 | 0 | 469
(Pneumothorax Segmentation) Candid | 8,195 | 0 | 0 | 8,195 | 0 | 0
(Pneumothorax Segmentation) SIIM | 7,621 | 1,709 | 1,704 | 7,621 | 1,709 | 1,704
(Rib Fracture Segmentation) Candid | 670 | 0 | 0 | 668 | 0 | 0
(Chest Tube Segmentation) Candid | 2,846 | 0 | 0 | 2,775 | 0 | 0
(Foreign Object Detection) Object-CXR | 8,000 | 1,000 | 0 | 8,000 | 1,000 | 0
(Report Generation) PadChest | 109,792 | 0 | 0 | 160,670 | 0 | 0
(Report Generation) BIMCV-COVID19 | 46,941 | 0 | 0 | 65,421 | 0 | 0
(Findings Generation) MIMIC-CXR | 152,173 | 1,196 | 2,347 | 270,790 | 2,130 | 3,858
(Findings Generation) MIMIC-CXR-Struct | 148,501 | 1,164 | 2,309 | 264,777 | 2,079 | 3,786
(Findings Generation) OpenI | 0 | 0 | 3,337 | 0 | 0 | 6,473
(Impression Generation) MIMIC-CXR | 185,816 | 1,521 | 2,224 | 316,684 | 2,573 | 3,724
(Impression Generation) MIMIC-CXR-Struct | 175,152 | 1,430 | 2,141 | 298,108 | 2,429 | 3,584
(Impression Generation) OpenI | 0 | 0 | 3,820 | 0 | 0 | 7,418
(Impression Generation) Candid | 18,307 | 0 | 0 | 19,206 | 0 | 0
(Findings Summarization) MIMIC-CXR | 125,417 | 991 | 1,624 | 0 | 0 | 0
(Progression Findings Generation) MIMIC-CXR | 19,107 | 138 | 483 | 34,354 | 258 | 851
(Progression Impression Generation) MIMIC-CXR | 26,646 | 216 | 313 | 46,227 | 369 | 560
(Findings Summarization) OpenI | 0 | 0 | 3,419 | 0 | 0 | 0
(Findings Summarization) MIMIC-III | 59,320 | 7,413 | 13,057 | 0 | 0 | 0
(Caption Generation) ROCO | 2,554 | 293 | 324 | 2,554 | 293 | 324
(Local Findings Generation) MIMIC-CXR-Struct | 1,059,903 | 8,299 | 16,275 | 264,688 | 2,078 | 3,785
(Local Impression Generation) MIMIC-CXR-Struct | 674,284 | 5,686 | 8,196 | 297,598 | 2,426 | 3,582
(Image-Text Matching) MIMIC-CXR | 675,978 | 5,434 | 9,142 | 354,619 | 2,866 | 4,710
(Image-Text Matching) ROCO | 5,108 | 586 | 648 | 2,554 | 293 | 324
(Image-Text Selection) MIMIC-CXR | 337,989 | 2,717 | 4,571 | 354,619 | 2,866 | 4,710
(Image-Text Selection) ROCO | 2,554 | 293 | 324 | 2,554 | 293 | 324
(Report Evaluation) ReXVal | 0 | 0 | 200 | 0 | 0 | 0
(Natural Language Explanation) MIMIC-NLE | 37,016 | 273 | 714 | 51,503 | 373 | 944
(Natural Language Inference) RadNLI | 0 | 480 | 480 | 0 | 0 | 0
(Temporal Sentence Similarity) MS-CXR-T | 0 | 0 | 361 | 0 | 0 | 0
(Named Entity Recognition) RadGraph | 541 | 9 | 50 | 0 | 0 | 0

We report CheXinstruct statistics in Table 6, where the number of instances
and number of images associated with each dataset are provided.

## D. CheXagent dataset ratios

Table 7: The sampling ratio of each dataset in the third-stage training of
CheXagent.

Task-Dataset Pair | Sampling Ratio
---|---
(Named Entity Recognition) RadGraph | 10.00
(Abnormality Grounding) VinDr-CXR | 1.00
(Abnormality Grounding) VinDr-PCXR | 1.00
(Caption Generation) ROCO | 1.00
(Close-Ended VQA) PMC-VQA | 1.00
(Close-Ended VQA) VQA-RAD | 1.00
(Foreign Object Detection) Object-CXR | 1.00
(Grounded Captioning) MS-CXR | 1.00
(Grounded Diagnosis) MS-CXR | 1.00
(Grounded Diagnosis) VinDr-CXR | 1.00
(Grounded Diagnosis) VinDr-PCXR | 1.00
(Grounded Phrase Extraction) MS-CXR | 1.00
(Image Classification) COVIDX-CXR-3 | 1.00
(Image Classification) NLM-TB | 1.00
(Image Classification) RSNA | 1.00
(Impression Generation) Candid | 1.00
(Open-Ended VQA) MedVQA-2019 | 1.00
(Open-Ended VQA) PMC-VQA | 1.00
(Open-Ended VQA) VQA-RAD | 1.00
(Phrase Grounding) MS-CXR | 1.00
(Pneumothorax Segmentation) Candid | 1.00
(Progression Findings Generation) MIMIC-CXR | 1.00
(Progression Impression Generation) MIMIC-CXR | 1.00
(Text QA) RadQA | 1.00
(View Matching) MIMIC-CXR | 0.50
(Findings Generation) MIMIC-CXR-Struct | 0.40
(Impression Generation) MIMIC-CXR-Struct | 0.30
(Natural Language Explanation) MIMIC-NLE | 0.30
(Report Generation) BIMCV-COVID19 | 0.25
(Findings Summarization) MIMIC-III | 0.20
(Image Classification) ChestXray14 | 0.20
(Close-Ended VQA) MIMIC-CXR-VQA | 0.10
(Findings Generation) MIMIC-CXR | 0.10
(Findings Summarization) MIMIC-CXR | 0.10
(Image Classification) Brax | 0.10
(Image Classification) CheXpert | 0.10
(Image Classification) PadChest | 0.10
(Image-Text Matching) ROCO | 0.10
(Image-Text Selection) ROCO | 0.10
(Impression Generation) MIMIC-CXR | 0.10
(Report Generation) PadChest | 0.10
(View Classification) MIMIC-CXR | 0.10
(Image Classification) MIMIC-CXR | 0.05
(Open-Ended VQA) MIMIC-CXR-VQA | 0.05
(Local Findings Generation) MIMIC-CXR-Struct | 0.02
(Local Impression Generation) MIMIC-CXR-Struct | 0.02
(Text Instructions) OpenOrca | 0.02
(Difference VQA) MIMIC-Diff-VQA | 0.01
(Image Classification) CXR-LT | 0.01
(Image-Text Matching) MIMIC-CXR | 0.01
(Image-Text Selection) MIMIC-CXR | 0.01
(Abnormality Detection) VinDr-CXR | 0.00
(Abnormality Detection) VinDr-PCXR | 0.00
(Chest Tube Segmentation) Candid | 0.00
(Close-Ended VQA) Rad-Restruct | 0.00
(Close-Ended VQA) SLAKE | 0.00
(Open-Ended VQA) Rad-Restruct | 0.00
(Open-Ended VQA) SLAKE | 0.00
(Phrase Extraction and Grounding) MS-CXR | 0.00
(Pneumothorax Segmentation) SIIM | 0.00
(Rib Fracture Segmentation) Candid | 0.00

In Table 7, we report the sampling ratios of each dataset used for Stage-3
training of CheXagent. The value “0” means that the corresponding dataset is
held out.

## E. CheXagent implementation details

Table 8: Training hyperparameters of CheXagent.

Configuration | Stage 0 | Stage 1 | Stage 2 | Stage 3
---|---|---|---|---
ViT init. | - | EVA01-CLIP-g-14-plus | ViT Stage 1 | ViT Stage 2
LLM init. | Mistral-7B-v0.1 | - | LLM Stage 0 | LLM Stage 2
Qformer init. | - | BERT Base | Qformer Stage 1 | Qformer Stage 2
Image resolution | - | 4482 | 4482 | 4482
ViT sequence length | - | 1024 | 1024 | 1024
LLM sequence length | 2048 | - | 512 | 512
Learnable query numbers | - | 128 | 128 | 128
Global batch size | 2048 | 256 | 2048 | 512
Optimizer | AdamW
Optimizer hyperparameter |  $\beta_{1}=0.9$, $\beta_{2}=0.999$, $eps=1e-8$
Peak learning rate | $5e-6$ | $1e-4$ | $1e-4$ | $1e-6$
Minimum learning rate | $5e-7$ | $1e-5$ | $1e-5$ | $1e-7$
Learning rate schedule | cosine
Weight decay | 0.05
Gradient clip | 1.0
Numerical precision | bf16
DeepSpeed | ZeRO stage 2 | - | - | ZeRO stage 2

The training of CheXagent involves four stages and we report the training
hyperparameters of each stage in Table 8.

## F. CheXbench Design

CheXbench evaluates two components of CXR interpretation: image perception and
textual understanding.

Evaluation Axis 1 - Image Perception: We assess FM performance on image
perception with six multiple-choice tasks across seven datasets. In
combination, these tasks include 5,083 samples spanning view classification,
binary disease classification, single disease identification, multi-disease
identification, visual-question-answering, and image-text reasoning. In Figure
7, we provide examples from each task. Table 9 provides a summary of all tasks
included in Evaluation Axis 1.

Table 9: Statistics for image perception tasks in CheXbench (Evaluation Axis 1). Task | Dataset | Num. Samples | Num. Options
---|---|---|---
| MIMIC-CXR | 400 | 4
View Classification | CheXpert | 300 | 3
| SIIM | 100 | 2
| RSNA | 100 | 2
Binary Disease Classification | CheXpert | 233 | 2
| OpenI | 500 | 4
| MIMIC-CXR | 195 | 4
Single Disease Identification | CheXpert | 169 | 4
| OpenI | 807 | 4
| MIMIC-CXR | 300 | 4
Multi Disease Identification | CheXpert | 280 | 4
| Rad-Restruct | 899 | 2-4
Visual Question Answering | SLAKE | 420 | 2
Image-Text Reasoning | OpenI | 380 | 2
Figure 7: Examples of CheXbench image perception tasks: CheXbench includes six
multiple-choice tasks that evaluate the ability of FMs to interpret chest
X-rays.

Evaluation Axis 2 - Textual Understanding: We assess FM performance on textual
understanding with two tasks: findings generation and findings summarization.
We evaluate performance using several automated metrics: ROUGE-L, CheXbert-
Score, BERT-Score, RadGraph-Score, and GPT-4. We provide prompts used for
GPT-4 evaluations in Figure 8.

The prompt for GPT-4 Evaluation Prompt: We would like to request your feedback
on the radiology reports generated by two AI assistants by comparing them to
the reference report written by radiologists. [Reference Report] [reference]
[End of Reference Report] [Assistant 1] [report1] [Assistant 1] [Assistant 2]
[report2] [End of Assistant 2] [Requirements] 1\. The length of the reports is
not important. 2\. The style of the reports is not important. 3\. The clinical
accuracy is important especially for positive findings (i.e., diseases).
Therefore, please focus on clinical accuracy instead of the length and style.
[End of Requirements] Please compare the accuracy of their generated reports.
You should tell me whether Assistant 1 is “better than”, “worse than”, or
“equal to” Assistant 2. Please first compare the generated reports with the
reference report to analyze which one is more in line with the given
requirements. In the last line, please output a single line containing only a
single label selecting from “Assistant 1 is better than Assistant 2”,
“Assistant 1 is worse than Assistant 2”, and “Assistant 1 is equal to
Assistant 2”. Figure 8: The prompt for GPT-4 Evaluation.