# Event Stream GPT: A Data Pre-processing and Modeling Library for Generative,
Pre-trained Transformers over Continuous-time Sequences of Complex Events

Matthew B. A. McDermott
Harvard University
<EMAIL_ADDRESS>
&Bret Nestor
Massachusetts Institute of Technology
&Peniel Argaw
Harvard University
&Isaac Kohane
Harvard Univiersity

###### Abstract

Generative, pre-trained transformers (GPTs, a.k.a. "Foundation Models") have
reshaped natural language processing (NLP) through their versatility in
diverse downstream tasks. However, their potential extends far beyond NLP.
This paper provides a software utility to help realize this potential,
extending the applicability of GPTs to continuous-time sequences of complex
events with internal dependencies, such as medical record datasets. Despite
their potential, the adoption of foundation models in these domains has been
hampered by the lack of suitable tools for model construction and evaluation.
To bridge this gap, we introduce Event Stream GPT (ESGPT), an open-source
library designed to streamline the end-to-end process for building GPTs for
continuous-time event sequences. ESGPT allows users to (1) build flexible,
foundation-model scale input datasets by specifying only a minimal
configuration file, (2) leverage a Hugging Face compatible modeling API for
GPTs over this modality that incorporates intra-event causal dependency
structures and autoregressive generation capabilities, and (3) evaluate models
via standardized processes that can assess few and _even zero-shot_
performance of pre-trained models on user-specified fine-tuning tasks.

##  1 Introduction

The unprecedented performance of large language models (LLMs) in natural
language processing (NLP) has ushered in a new paradigm in machine learning.
Instead of being dominated by single-task, supervised learning models, this
paradigm is characterized by general-purpose, pre-trained “foundation models.”
These models, exemplified by the “Generative, Pre-trained Transformer” (GPT)
architecture, deliver state-of-the-art performance on diverse downstream tasks
in a highly data-efficient manner [22]. The success of GPT models over natural
language data is driven in part by the fact that the generative, language
modeling forecasting task is “universal” across NLP downstream tasks—meaning
any natural language task can be recast as a language modeling task through
prompting. However, this property is not unique to NLP, and GPT-equivalent
architectures may hold significant promise in other domains as well. In
particular, this work explores the application of these models to the domain
of continuous-time sequences of complex, multi-modal events (event streams),
specifically electronic health record (EHR) datasets.

Two critical barriers inhibit the application of foundation model research to
modalities like EHR data. First, these data modalities do not come in a
single, unified format, and compelling datasets are often private and
unshareable. This hampers foundation model research by requiring dataset-
specific pre-processing pipelines and hindering one’s ability to assess the
model generalizability. Second, modeling continuous-time sequences of complex
events is more challenging than modeling ordinal sequences of tokens, as GPTs
over these modalities must account for non-ordinal inputs, complex, multi-
modal emission distributions, and intra-event causal relationships.

In this paper, we address these gaps with Event Stream GPT (ESGPT), an open
source software package111Code provided in Supplementary Material, API, and
evaluation utility for foundation models over event stream data. ESGPT can
represent diverse datasets across various sources in a unified manner, pre-
process very large datasets extremely quickly through its use of the Polars
library [37], and can Hyperparameter tune, pre-train, fine-tune, and perform
zero-shot evaluation for foundation models through a Hugging Face compatible
API. In the rest of this work, we will demonstrate the unique value Event
Stream GPT offers through the lens of a real-world, working example: building
foundation models over the MIMIC-IV electronic health record dataset [12]
(Section 2). We will walk through all aspects of the pipeline one would follow
to pursue this research, revealing in each how ESGPT fills a critical gap in
existing tools across data pre-processing (Section 3), model configuration and
running (Section 4), and evaluation (Section 5). Finally, we will close with
related work and concluding thoughts.

##  2 Working Example Problem Setup

#### Source Data

The MIMIC-IV dataset [12] is a publicly available dataset consisting of the
EHR data for all adult patients who were admitted to the emergency department
or an intensive care unit (ICU) at Beth Israel Deaconess Medical Center
between 2008 and 2019. This dataset contains approximately 300,000 patients
and consists of numerous modalities of health data, including diagnoses,
laboratory test results, medications, in and out of hospital mortality, and
many others, all localized continuously in time. For our running example, we
will define the internal covariates of each event to include the set of
modeling targets in Table 1.

#### Modeling Task

Our sample modeling challenge is to build a GPT model over the continuous-
time, complex event stream data contained in MIMIC-IV. This can also be seen
as a multi-variate marked temporal point process. In particular, given a
sequence of complex events $\boldsymbol{x}_{1},\ldots,\boldsymbol{x}_{N}$
(where each event $\boldsymbol{x}_{i}$ is a collection of occurrences of the
selected covariates in Table 1) which occur at continuous times
$t_{1},\ldots,t_{N}$, we wish to model the following probability distribution:

$\displaystyle
p(t_{i},\boldsymbol{x}_{i}|\underbrace{(t_{1},\boldsymbol{x}_{1}),\ldots,(t_{i-1},\boldsymbol{x}_{i-1})}_{\boldsymbol{h}_{i-1}})$

We will realize this through a transformer neural network architecture
parametrized by $\boldsymbol{\theta}$, such that
$f_{\boldsymbol{\theta}}(t_{i},\boldsymbol{x}_{i},\boldsymbol{h}_{i-1})=p(t_{i},\boldsymbol{x}_{i}|\boldsymbol{h}_{i-1})$.
Note that, unlike other GPT modeling settings, here it may be the case that
internal covariates of each event $\boldsymbol{x}_{i}$ have internal causal
dependencies. For example, if $\boldsymbol{x}_{i}^{(j)}$ is used to denote the
$j$th internal covariate of event $i$, then
$p(\boldsymbol{x}_{i}|\boldsymbol{h}_{i-1},t_{i})\neq\prod_{j}p(\boldsymbol{x}_{i}^{(j)}|\boldsymbol{h}_{i-1},t_{i})$.
Any full generative model will therefore need to account for these internal
causal dependencies.

Admission/Discharge | Demographics | Measurements/Actions
---|---|---
Admission Type | Language | Laboratory Tests∗
Admission Location | Race† | Procedures
ICU Care-unit | Marital Status | Medications
Discharge Destination | Insurance Type | Infusions∗
| Age | Diagnoses
| Time-of-day | Patient Weight
| Gender |
Table 1: The modalities of data we include in our MIMIC-IV working example.
Our generative model consists of continuous timeseries of partial observations
of any of these events, and we task the model with predicting which of the
times of these events, which modalities will be observed in those events, and
what values they will take on. Included modalities are not observed uniformly
across all events, in a non-random manner. $*$: Also includes continuous
regression components (laboratory test results and total infusion amount).

##  3 Data Extraction and Pre-processing

###  3.1 The Problem

To build a dataset suitable for performing our foundation modeling task
outlined in Section 2, we perform the following steps:

  1. 1.

Extract the raw data from its source (here, the MIMIC-IV postgresql database).

  2. 2.

Pre-process the data: learn outlier detection and normalization parameters for
numerical variables, filter out infrequently measured variables or outliers,
normalize data, etc.

  3. 3.

Join independent input datasets together into a format optimized for deep
learning.

  4. 4.

Build a PyTorch dataset, dataloader, and embedding layer for this deep-
learning friendly structure, such that we can use these data efficiently in
downstream pipelines.

###  3.2 Current State-of-the-art

If we wish to use current solutions to the data extraction problem, we must
find an existing tool that either is fully specialized to our dataset (MIMIC-
IV) and use-case (of which no such example exists), or is a general purpose
data-extraction, pre-processing, and/or modeling pipeline that will ease our
workflow. For such a general purpose tool to be useful, it must meet several
critical criteria:

  1. 1.

It must solve at least a majority of the challenges identified in Section 3.1.

  2. 2.

It must be flexible enough to handle not only the MIMIC-IV data, but also
various other input datasets, schemas, and covariate choices.

  3. 3.

It must be designed for efficiency in mind – both in data pre-processing and
in the output deep-learning representations we produce, so that we can pre-
train large models effectively.

  4. 4.

It must be actively maintained, well documented, and tested.

Unfortunately, as shown in Table 2, existing pipelines for extracting
Electronic Health Record (EHR) data for machine learning use do not fully meet
these criteria. Only one pipeline meets even a meaningful subset of these
critical desiderata, TemporAI [31], as it has strong development practices,
exposes a flexible input format, and meets our critical pre-processing feature
goals; however, TemporAI does not support native extraction from underlying
source files, and suffers in its efficiency, both of the core data pre-
processing pipeline and in its output deep learning representation, raising
concerns about its viability on larger datasets.

|  Usable on General Datasets |  Flexible Input Format |  Performs Extraction from Source |  Accepts Arbitrary Input Variables |  Fits Vocabularies |  Outlier Detection |  Pre-set censoring/outlier removal |  Normalization |  Filters Infrequent Measurements |  Efficiently Pre-processes Data using Modern Libraries‡ |  Produces Efficient Deep-Learning Representation∗ |  Produces PyTorch Datasets |  Provides Embedding Layer |  Actively Maintained⋆ |  Tested† |  Comprehensive Documentation
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Input | Pre-processing | PyTorch | Dev
MIMIC-Extract [38] |  |  | ✓ |  |  |  | ✓ | ✓ | ✓ |  |  |  |  |  |  |
Fiddle [36] | ✓ |  |  |  | ✓ |  |  | ✓ | ✓ |  |  |  |  |  |  |
MT-MIMIC [8] |  |  |  |  | ✓ |  |  | ✓ |  |  |  |  |  |  |  |
Clairvoyance [10] | ✓ | ✓ |  | ✓ | ✓ | ✓ |  | ✓ | ✓ |  |  | ✓ | ✓ |  |  | ✓
OMOP-Learn [13] | ✓ |  | ✓ |  |  |  |  | ✓ |  |  |  | ✓ |  |  |  |
TemporAI [31] | ✓ | ✓ |  | ✓ | ✓ | ✓ |  | ✓ | ✓ |  |  | ✓ | ✓ | ✓ | ✓ | ✓
ESGPT (Ours) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Table 2: A collection of existing data pre-processing and/or modelling
pipelines for temporal EHR data, categorized on whether or not they enable
various features. $\star$ Active maintenance is defined by the existence of a
commit between May 2022 and May 2023. $\dagger$ Tested is defined by the
presence of a reasonably comprehensive, automated test suite. $\ddagger$
Efficient data pre-processing is defined by whether or not the library relies
on Pandas [39, 23] for their dataframe manipulation, as opposed to more
modern, faster systems such as Polars [37] or DuckDB [19]. $*$ Efficient deep-
learning representation is defined by the extent to which sparsity is
leveraged in their deep learning representation files; efficient outputs
should only scale in size with the observations actually present in any given
batch of data, not with the number of total features in the space, total
possible sequence elements, etc.

###  3.3 Contributions of Event Stream GPT

ESGPT solves all these challenges in data extraction and pre-processing.
Figure 1 shows the entire pipeline in visual form; from a flexible, concise,
user-friendly configuration file, ESGPT can extract raw data from source, pre-
process categorical and numerical data elements, and produce efficient PyTorch
datasets for downstream analysis. Below, we detail three key advantages of the
ESGPT system, and full technical details of the pipeline and its usage an be
found in the SI.

Figure 1:  ESGPT’s end-to-end data pipeline, which spans raw data extraction
from source all the way to production of a PyTorch dataset and pre-built
embedder suitable for use in any deep learning pipeline. (a) An example of our
data modality; for a single subject $\boldsymbol{S}_{1}$, the data consists of
a series of events at continuous timestamps
$\boldsymbol{e}_{1},\boldsymbol{e}_{2},\ldots$ (such as medical visits), with
each event being composed of inter-related internal covariate measurements
$\boldsymbol{m}_{1},\boldsymbol{m}_{2},\ldots$ (such as laboratory test
results, medication perscripts, infusions, etc.). (b) These data elements can
be distributed among many input data sources in raw form. From a simple YAML
configuration file, ESGPT can extract these data elements from source and
compile them into an internal data model consisting of three key dataframes:
subjects_df for static data, events_df containing event timestamps and types
per subject, and dynamic_measurements_df, storing all observed measurements.
(c) ESGPT pre-processes these data-frames across several critical axes, doing
so efficiently through the use of the Polars library [37]. (d) ESGPT produces
a PyTorch dataset which yields batches whose size scales with the number of
observed data element per event, not with the input vocabulary size. (e) ESGPT
provides a default embedding layer capable of embedding these sparse batches
efficiently.

#### Event Stream GPT is Easy to Use and Flexible

To extract a machine-learning ready dataset from _any_ temporal, structued
dataset with Event Stream GPT, a user only needs to specify a modest
configuration file detailing the input data sources, desired modalities to
extract, and pre-processing pipeline parameters. This configuration file
instructs the pipeline how to extract the initial data from its raw source,
pre-process the extracted measurements, aggregate the data temporally, and
produce deep-learning efficient representation outputs suitable for downstream
modeling. A full configuration file for our working example over MIMIC-IV can
be found in the appendix.

#### Event Stream GPT can Efficiently Pre-process Large Datasets

Through the use of the Polars library [37] and careful design choices, ESGPT
is able to provide a dramatic improvement in data pre-processing speed and
output storage costs compared to existing libraries. For our working example,
_pre-processing the entire MIMIC-IV dataset (300,000 input patients) with our
pipeline takes only approximately 30 minutes, and outputs a dataset that takes
only 1.2 GB to store on disk_. While it is difficult to perform a robust
comparison across different data extraction pipelines, given they all have
differing features and capabilities, anecdotally it is the case that many
other pipelines take hours to perform extraction and pre-processing over
datasets of similar size, and often occupy much more space on disk. Further
details on the computational performance of the pipeline, and illustrations of
its performance on other datasets to demonstrate its flexibility can be found
in the SI.

#### Event Stream GPT Provides a Memory-efficient Deep Learning Representation

Beyond efficient pre-processing of data, ESGPT also provides an efficient deep
learning representation and an associated PyTorch Dataset and Data Embedding
Layer. This representation has three key benefits compared to existing
systems. First, The PyTorch Dataset view of ESGPT datasets is designed for
rapid retrieval of a subject’s timeseries data, in order, to minimize
dataloader iteration time, by storing the entirety of a subject’s timeseries
data in a single row in the output dataframe, and serializing that row to a
plain python list at instantiation. In contrast, existing systems like
TemporAI store temporal data as a multi-indexed pandas dataframe object, which
requires dynamic slicing within the multi-index levels to retrieve a patient’s
temporal data.

Second, a patient’s data in the deep-learning representation is represented in
an extremely sparse format in ESGPT, so that the memory required to represent
a single patient’s timeseries scales linearly _only_ with the total number of
observations that patient has. In contrast, existing systems, such as
TemporAI, largely have data storage scale linearly with the _total number of
features in the dataset_ , in some cases further multiplied by _the maximum
number of events any patient has in the dataset_. Given the long-tailed nature
of these data, this is an extremely large difference. In our MIMIC-IV working
example, for example, the average number of observations per patient event is
approximately 80. In contrast, the total number of features in this dataset is
6053 (this includes all measured modalities in Table 1). Therefore, our deep
learning batch representation strategy requires only 1.3% of the required
memory of traditional, less sparse solutions.

Beyond these two major improvements, ESGPT also is the only pipeline to fully
provide a pre-built PyTorch dataset with a customized embedding layer for our
data representation; this means that one can immediately feed any ESGPT
dataset into any sequential deep learning pipeline of their choice.

##  4 Building GPTs

###  4.1 The Problem

To produce the generative model outlined in Section 2, we must satisfy several
desiderata:

  1. 1.

We need to be able model historical dependencies
($p(t_{i},\boldsymbol{x}_{i}|\boldsymbol{h}_{i-1})$), temporal dependencies
($p(\boldsymbol{x}_{i}|t_{i})$), and intra-event dependencies
($p(\boldsymbol{x}_{i}^{(j+1)}|\boldsymbol{x}_{i}^{(1)},\ldots,\boldsymbol{x}_{i}^{(j-1)})$).
In contrast, traditional (sequential) GPTs only need to model historical
dependencies.

  2. 2.

We need to enable deep, sequential processing of event sequences that can take
into account event timestamps to build high-capacity representations.

  3. 3.

We need to produce categorical, temporal, and continuous output emission
distribution. In contrast, traditional GPTs only need to output categorical
emission distributions.

  4. 4.

Our model needs to be able to dynamically construct analytically determinable
time-dependent features (e.g., age) on the fly during generation.

###  4.2 Current State-of-the-art

There are three categories of existing tools that could help us build models:
General-purpose deep learning modeling frameworks with built in model
architectures (e.g., TemporAI [31]); other GPT architectures for different
problem settings, such as natural language or regularly sampled timeseries; or
existing models for autoregressive or generative modeling of EHR or related
modalities.

Unfortunately, across all three options, there are no existing tools which
sufficiently satisfy the criteria outlined in Section 4.1. Among existing,
general-purpose frameworks, such as TemporAI [31] or OMOP Learn [13], lack any
suitable built-in model architectures for this task. Existing GPT
architectures, on the other hand, can be very helpful, especially for
providing sources to adapt transformer architectures for core multi-layer
network structures, but lack key features specific to this modality, such as
accounting for unknown output event times and complex inter-event
dependencies. Finally, among existing, published generative models for EHR or
other similar modalities, all existing architectures have major limitations,
as outlined in Table 5 (see the SI for an expanded tables that includes a
broader class of models).

Model |  TTE |  Continuous Outputs |  Categorical Outputs |  Intra-event Dependencies |  Generative |  Fine-tuning |  Zero-shot
---|---|---|---|---|---|---|---
Models | Evaluation
Doctor AI [3] | ✓ |  | ✓ |  | ✓ |  |
CDT [16] |  |  | ✓ |  |  |  | ✓
MetaCare++ [35] | ✓ |  | ✓ |  | ✓ |  |
CLMBR [34] |  |  | ✓ |  |  | ✓ |
MedTPP [5] | ✓ |  | ✓ |  | ✓ |  |
MedGPT [14] | ✓ |  | ✓ |  | ✓ |  |
ESGPT (Ours) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Table 3: A summary of existing foundation models over EHR data, broken down by
what modalities of data they model and how they report evaluation results.

###  4.3 Contributions of Event Stream GPT

The ESGPT library includes a pre-built model architecture that models time-to-
event components, continuous and categorical output measurements, handles
intra-event dependencies in a user-configurable way, and provides a
HuggingFace-compatible API for generating new events with a pre-trained model.
We detail these capabilities below, and a full technical reference is in the
SI.

#### Event Stream GPT provides a HuggingFace API compatible interface for
building models capable of outputting complex, temporal distributions

To use one of the pre-defined model architectures in Event Stream GPT, users
simply build a specialized configuration object, which is a sub-class of the
standard, Hugging Face
PretrainedConfig222https://huggingface.co/docs/transformers/main_classes/configuration.
Once the config is defined, a model can be instantiated in the same way as any
other Hugging Face pre-trained model, either from scratch or from a pre-
trained source. In addition to the standard Hugging Face attributes, the ESGPT
config also contains several custom attributes specific to our setting,
including attributes describing the breakdown of the various measurements and
their respective vocabulary sizes, the implied dependency relationships
between intra-event covariates, and data embedding configuration parameters
(for a full specification, see the Supplement).

All pre-built ESGPT models for generative modeling output a unified model
output object. This object maps event generation covariate target names to
PyTorch distributions, from which we can easily compute losses or sample new
events. The unified ESGPT dataset class and this output object defines the API
expected by ESGPT models in further utilities like generative capabilities.
Full details can be found in the Supplement.

#### Event Stream GPT provides a modeling interface for encoding complex,
custom intra-event dependencies

ESGPT comes with two pre-defined model architectures. One (the
ConditionallyIndependent model) that matches much of the existing published
literature and only supports defining generative models where all intra-event
covariates are conditionally independent from one another given the patient’s
history and one (the NestedAttention model) that supports user-defined
dependency relationships between various internal features of
$\boldsymbol{x}_{i}$. Users specify these relationships through the
configuration file. For example, one of the models pre-trained in our MIMIC-IV
working example relies on the intra-event dependency chain shown in Figure 2.
This specification takes the form of a list of lists, and tells the model that
it should make the assumption that the internal covariates in a given inner
list of $\boldsymbol{x}_{i}$ are conditionally independent of one another
given both (1) the historical representation of all
$t_{\ell},\boldsymbol{x}_{\ell}$, $\ell<i$ and (2) the internal covariates in
all prior lists in the specified dependency graph. Concretely, this means that
the model configuration snippet shown in Figure 2 defines a generative model
such that the predicted type of an event will depend only on the historical
representation and the true time of that event, but the predicted values
observed for the laboratory tests ordered for this patient will depend on both
the time of the event, but also the event type, patient dynamic demographics,
and the categorical laboratory test item IDs that are actually ordered for
that patient.

We describe the architecture used to model this in full detail in the SI;
however, what is more essential for the utility of the ESGPT software library
is that this API for specifying internal dependencies, and the associated non-
architectural changes required to support it (e.g., enabling generation to
respect the multi-stage process implied by the intra-event dependency graph)
are general, and can be used across a variety of internal transformer
architectures. Thus, this provided model class can serve as a basis for
further model development that relies on internal event dependencies in future
research without invalidating the benefits of the other tools in ESGPT. All
aspects of this process are documented fully in the supplementary information.

⬇

structured_event_processing_mode nested_attention

measurements_per_dep_graph_level

- ["age", "time_of_day"] # Must start with FUNCTIONAL_TIME_DEPENDENT measurements.

- [

"event_type", "patientweight", "admission_type", "admission_location",

"race", "language", "marital_status", "insurance", "careunit",

["lab_itemid", "categorical_only"], ["infusion_itemid", "categorical_only"]

]

- [["lab_itemid", "numerical_only"], ["infusion_itemid", "numerical_only"]]

- ["procedure_itemid", "medication", "icd_code", "discharge_location"]

Figure 2: A sample dependency graph for our MIMIC-IV working example, shown in
YAML syntax. The dependency graph is encoded as a list of lists, such that all
measurements within any inner list are constrained to depend only on those
measurements in prior lists.

#### Event Stream GPT provides seamless generation capabilities for
unsupervised, zero-shot evaluation

A critical use-case for foundation models is the ability to generate new,
unseen predictions for the future of an input. To the best of our knowledge,
ESGPT is the first pipeline to bring that capability in a systemized fashion
to foundation models over event stream data like EHR systems. To do so, we
introduce several key modifications from the traditional Hugging Face
generation API, including the ability to support complex emission
distributions during generation, dynamic computation of time-dependent input
features for newly generated time-points, and generating events in sequence
while respecting the causal path. This enables us to use generation out of the
box on models over ESGPT datasets. Full details can be found in the SI.

##  5 Evaluating Foundation Models

###  5.1 The Problem

Even after building a foundation model on this non-standard domain, one still
has to evaluate that model, both as a concrete representation learning system
and in its possible viability as an early stage foundation model. In domains
of complex, timeseries of events, like medical record datasets, there are a
number of unique challenges that must be addressed in evaluation. We foresee
the following metrics to be critical points to assess in early stage
foundation models in such domains:

  1. 1.

Performance as a raw generative model.

  2. 2.

Viability as a foundation model

  3. 3.

Practical utility vs. current ML systems.

  4. 4.

Performance disparities across subject sub-populations

  5. 5.

Identifiability of subject-level data from pre-trained model parameters

###  5.2 Current State-of-the-art

Existing models in this space have significant gaps in their evaluation
strategies to date. Table 5, in addition to summarizing model modalities, also
summarizes evaluation methods used. While some techniques, like few-shot fine-
tuning analyses, are well represented, others are under-explored. _In
particular, at present there are no existing strategies to reliably assess
general, zero-shot performance of GPTs over event stream datasets, a gap which
we fill here._ 333The Clinical Decision Transformer (CDT) [16] does perform
some limited zero-shot evaluation strictly through its reinforcement learning
defined policy analysis, not on general tasks.

###  5.3 Contributions of Event Stream GPT

ESGPT focuses on providing usable, powerful tools to enable rapid evaluation
of GPTs over event stream datasets, focused currently on the assessment of a
model’s generative performance and in its viability as a foundation model in
few and zero-shot evaluation tasks. While further efforts on the other
evaluation criteria would be highly impactful, given the dearth of existing
utilities just on assessing these basic performance metrics, we leave the rest
to future work. In addition to these evaluation utilities, we also provide
pre-built scripts for running distributed, Bayesian hyperparameter
optimization sweeps over foundation models through weights and biases [2], to
help further standardize that portion of the optimization process.

#### Assessing Generative Model Performance & Hyperparameter Tuning

To make assessing generative performance easier, ESGPT comes pre-equipped with
a PyTorch Lightning [6] module for running pre-training models which tracks a
variety of metrics to assess generative model performance, including
traditional classification and regression metrics over all component internal
event covariates. These are logged by default to weights and biases,
permitting easy exploration of output model performance. These metrics are
further tracked and can be assessed visually in weights and biases reports
during hyperparameter tuning as well, which is run via a Weights and Biases
sweep with a provided default hyperparameter search space. A template report
for assessing the output of these sweeps is also provided alongside the code
for our working example over MIMIC-IV (accessible in the SI), allowing users
to easily design their own dashboards to monitor their model tuning runs.

#### Assessing Foundation Model Viability via Zero-shot Performance

While many systems already explore few-shot performance (and ESGPT comes pre-
equipped with lightning modules and utilities for performing similar
assessments), assessing zero-shot performance across general models has not
been previously studied. Enabling zero-shot evaluation for foundation models
trained on event stream data requires three things: First, a labeled dataset
corresponding to a well-defined task which can be used to evaluate zero-shot
produced labels; second, a method to perform unsupervised forecasting of a
patient’s record given an input window, which ESGPT provides; third, a
function to translate generated samples into empirical predictions for the
task in question.

The ESGPT system then provides a PyTorch Lightning utility to take as input a
dataset, task, pre-trained model, and labeling function, and perform zero-shot
evaluation over a specified data cohort. This system reports performance
metrics based on the output of this labeling function over the empirical
predictions generated for each patient’s input. _This provides any model
trained using the ESGPT API the ability to assess zero-shot performance in a
meaningful way, out of the box._ For examples of these evaluation utilities on
MIMIC-IV, see the SI.

##  6 Further Discussion

#### Limitations

While ESGPT provides significant utilities, it also has a number of
limitations. Firstly, it lacks some more advanced pre-processing capabilities
that other systems support, such as unit conversion, ontological aggregation,
or support for a wide variety of optional pre-processing algorithms. Secondly,
defining downstream tasks is currently only possible through manual creation /
user-defined code, as opposed to via a configuration language. Finally,
lastly, while the utilities ESGPT offers do significantly enhance evaluation,
even more is needed on that front, including dedicated metrics for fairness
and privacy sensitivity, methods to assess embedding space structure, and
methods to detect emergence of foundation model capabilities more cost
efficiently.

#### Related Work

As listed in Tables 2 & 5, there are a number of existing processing pipelines
and models of relevance here. Beyond those already discussed, there are also a
number of models that leverage non-generative losses that warrant
highlighting, such as contrastive methods like [1, 17, 21]. These provide a
valuable alternative to generative foundation models in various domains.
Additionally, other pipelines also exist in this and adjacent spaces, such as
time-series specific pipelines and tabular data embedding methodologies [11,
7, 4]. GPTs have also been used very successfully in other domains outside of
event stream data. These include both examples of foundation models outside of
either NLP or event stream data, such as on protein sequences [30] or point
clouds [40], but also other examples of temporal point processes outside of
the typical “foundation model” framing [32, 41]

##  7 Conclusion

In this paper, we introduced Event Stream GPT (ESGPT), a new, open source
software library that fills a critical gap in the foundation model literature:
namely, technological support for building generative, pre-trained
transformers (GPTs) over “event stream data”—datasets consisting of continuous
time sequences of complex events with multiple modalities and internal
dependencies. We’ve shown that Event Stream GPT can reliably address all
aspects of the research pipeline for this end, from data pre-processing at
scale, model construction in a manner accounting for the unique properties of
event stream data, and evaluation across all aspects of the foundation model
pipeline. We feel that ESGPT will significantly accelerate, standardize, and
improve the space of research into these models and will help identify the
best architectures and settings where the strengths of GPTs as seen in other
domains can be replicated in new domains.

## Acknowledgments and Disclosure of Funding

MBAM gratefully acknowledges support by a Berkowitz Postdoctoral Fellowship at
Harvard Medical School. In addition, this work has benefited from the advice
of Oleksander Shchur.

The All of Us Research Program is supported by the National Institutes of
Health, Office of the Director: Regional Medical Centers: 1 OT2 OD026549; 1
OT2 OD026554; 1 OT2 OD026557; 1 OT2 OD026556; 1 OT2 OD026550; 1 OT2 OD 026552;
1 OT2 OD026553; 1 OT2 OD026548; 1 OT2 OD026551; 1 OT2 OD026555; IAA #: AOD
16037; Federally Qualified Health Centers: HHSN 263201600085U; Data and
Research Center: 5 U2C OD023196; Biobank: 1 U24 OD023121; The Participant
Center: U24 OD023176; Participant Technology Systems Center: 1 U24 OD023163;
Communications and Engagement: 3 OT2 OD023205; 3 OT2 OD023206; and Community
Partners: 1 OT2 OD025277; 3 OT2 OD025315; 1 OT2 OD025337; 1 OT2 OD025276. In
addition, the All of Us Research Program would not be possible without the
partnership of its participants.

## References

  * [1] Monica N Agrawal, Hunter Lang, Michael Offin, Lior Gazit, and David Sontag.  Leveraging time irreversibility with order-contrastive pre-training.  In International Conference on Artificial Intelligence and Statistics, pages 2330–2353. PMLR, 2022.
  * [2] Lukas Biewald.  Experiment tracking with weights and biases, 2020.  Software available from wandb.com.
  * [3] Edward Choi, Mohammad Taha Bahadori, Andy Schuetz, Walter F. Stewart, and Jimeng Sun.  Doctor ai: Predicting clinical events via recurrent neural networks.  In Finale Doshi-Velez, Jim Fackler, David Kale, Byron Wallace, and Jenna Wiens, editors, Proceedings of the 1st Machine Learning for Healthcare Conference, volume 56 of Proceedings of Machine Learning Research, pages 301–318, Northeastern University, Boston, MA, USA, 18–19 Aug 2016. PMLR.
  * [4] Maximilian Christ, Nils Braun, Julius Neuffer, and Andreas W. Kempa-Liehr.  Time series feature extraction on basis of scalable hypothesis tests (tsfresh – a python package).  Neurocomput., 307(C):72–77, sep 2018.
  * [5] Joseph Enguehard, Dan Busbridge, Adam Bozson, Claire Woodcock, and Nils Hammerla.  Neural temporal point processes for modelling electronic health records.  In Machine Learning for Health, pages 85–113. PMLR, 2020.
  * [6] William Falcon and The PyTorch Lightning team.  PyTorch Lightning, March 2019.
  * [7] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko.  Revisiting deep learning models for tabular data.  Advances in Neural Information Processing Systems, 34:18932–18943, 2021.
  * [8] Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan.  Multitask learning and benchmarking with clinical time series data.  Scientific data, 6(1):96, 2019.
  * [9] AURP Investigators.  The “all of us” research program.  New England Journal of Medicine, 381(7):668–676, 2019.
  * [10] Daniel Jarrett, Jinsung Yoon, Ioana Bica, Zhaozhi Qian, Ari Ercole, and Mihaela van der Schaar.  Clairvoyance: A pipeline toolkit for medical time series.  In International Conference on Learning Representations, 2021.
  * [11] Xiaodong Jiang, Sudeep Srivastava, Sourav Chatterjee, Yang Yu, Jeffrey Handler, Peiyi Zhang, Rohan Bopardikar, Dawei Li, Yanjun Lin, Uttam Thakore, Michael Brundage, Ginger Holt, Caner Komurlu, Rakshita Nagalla, Zhichao Wang, Hechao Sun, Peng Gao, Wei Cheung, Jun Gao, Qi Wang, Marius Guerard, Morteza Kazemi, Yulin Chen, Chong Zhou, Sean Lee, Nikolay Laptev, Tihamér Levendovszky, Jake Taylor, Huijun Qian, Jian Zhang, Aida Shoydokova, Trisha Singh, Chengjun Zhu, Zeynep Baz, Christoph Bergmeir, Di Yu, Ahmet Koylan, Kun Jiang, Ploy Temiyasathit, and Emre Yurtbay.  Kats, 3 2022.
  * [12] Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Benjamin Moody, Brian Gow, Li-wei H Lehman, et al.  Mimic-iv, a freely accessible electronic health record dataset.  Scientific data, 10(1):1, 2023.
  * [13] Rohan Kodialam, Rebecca Boiarsky, Justin Lim, Aditya Sai, Neil Dixit, and David Sontag.  Deep contextual clinical prediction with reverse distillation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 249–258, 2021.
  * [14] Zeljko Kraljevic, Anthony Shek, Daniel Bean, Rebecca Bendayan, James Teo, and Richard Dobson.  Medgpt: Medical concept prediction from clinical narratives.  arXiv preprint arXiv:2107.03134, 2021.
  * [15] Chuhong Lahlou, Ancil Crayton, Caroline Trier, and Evan Willett.  Explainable health risk predictor with transformer-based medicare claim encoder.  arXiv preprint arXiv:2105.09428, 2021.
  * [16] Seunghyun Lee, Da Young Lee, Sujeong Im, Nan Hee Kim, and Sung-Min Park.  Clinical decision transformer: Intended treatment recommendation through goal prompting.  arXiv preprint arXiv:2302.00612, 2023.
  * [17] Yikuan Li, Mohammad Mamouei, Gholamreza Salimi-Khorshidi, Shishir Rao, Abdelaali Hassaine, Dexter Canoy, Thomas Lukasiewicz, and Kazem Rahimi.  Hi-behrt: Hierarchical transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records.  IEEE Journal of Biomedical and Health Informatics, 2022.
  * [18] Yikuan Li, Shishir Rao, José Roberto Ayala Solares, Abdelaali Hassaine, Rema Ramakrishnan, Dexter Canoy, Yajie Zhu, Kazem Rahimi, and Gholamreza Salimi-Khorshidi.  Behrt: transformer for electronic health records.  Scientific reports, 10(1):1–12, 2020.
  * [19] Mark, Hannes Mühleisen, Pedro Holanda, tiagokepe, Diego Gomes Tomé, Josh Wills, Richard Wesley, Kirill Müller, Till Döhmen, tania, Denis Hirn, simonasked, Azim Afroozeh, nantiamak, Patrick Schratz, Aris Koning, André Kohn, Chilarai, Gabor Szarnyas, Arjen P. de Vries, Sreeharsha Ramanavarapu, Andy Teucher, Dominik Moritz, Igor [hyperxor], Uwe L. Korn, travis leith, Aleksei Kashuba, Erwan Le Pennec, Y., and Jian Zhang.  duckdb/duckdb: 0.2.6 preview release "jamaicensis", May 2021.
  * [20] Matthew McDermott, Bret Nestor, Evan Kim, Wancong Zhang, Anna Goldenberg, Peter Szolovits, and Marzyeh Ghassemi.  A comprehensive ehr timeseries pre-training benchmark.  In Proceedings of the Conference on Health, Inference, and Learning, pages 257–278, 2021.
  * [21] Matthew McDermott, Brendan Yap, Peter Szolovits, and Marinka Zitnik.  Structure inducing pre-training.  arXiv preprint arXiv:2103.10334, 2021.
  * [22] OpenAI.  Gpt-4 technical report, 2023.
  * [23] The pandas development team.  pandas-dev/pandas: Pandas, April 2023.  If you use this software, please cite it as below.
  * [24] Chao Pang, Xinzhuo Jiang, Krishna S. Kalluri, Matthew Spotnitz, RuiJun Chen, Adler Perotte, and Karthik Natarajan.  Cehr-bert: Incorporating temporal information from structured ehr data to improve prediction tasks.  In Subhrajit Roy, Stephen Pfohl, Emma Rocheteau, Girmaw Abebe Tadesse, Luis Oala, Fabian Falck, Yuyin Zhou, Liyue Shen, Ghada Zamzmi, Purity Mugambi, Ayah Zirikly, Matthew B. A. McDermott, and Emily Alsentzer, editors, Proceedings of Machine Learning for Health, volume 158 of Proceedings of Machine Learning Research, pages 239–260. PMLR, 04 Dec 2021.
  * [25] Chantal Pellegrini, Nassir Navab, and Anees Kazi.  Unsupervised pre-training of graph transformers on patient population graphs.  arXiv preprint arXiv:2207.10603, 2022.
  * [26] Shishir Rao, Mohammad Mamouei, Gholamreza Salimi-Khorshidi, Yikuan Li, Rema Ramakrishnan, Abdelaali Hassaine, Dexter Canoy, and Kazem Rahimi.  Targeted-behrt: Deep learning for observational causal inference on longitudinal electronic health records.  IEEE Transactions on Neural Networks and Learning Systems, 2022\.
  * [27] Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi.  Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction.  NPJ digital medicine, 4(1):1–13, 2021.
  * [28] Houxing Ren, Jingyuan Wang, and Wayne Xin Zhao.  Generative adversarial networks enhanced pre-training for insufficient electronic health records modeling.  In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’22, page 3810–3818, New York, NY, USA, 2022\. Association for Computing Machinery.
  * [29] Houxing Ren, Jingyuan Wang, Wayne Xin Zhao, and Ning Wu.  Rapt: Pre-training of time-aware transformer for learning robust healthcare representation.  In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD ’21, page 3503–3511, New York, NY, USA, 2021\. Association for Computing Machinery.
  * [30] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus.  Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences.  Proceedings of the National Academy of Sciences, 118(15):e2016239118, 2021.
  * [31] Evgeny S Saveliev and Mihaela van der Schaar.  Temporai: Facilitating machine learning innovation in time domain tasks for medicine.  arXiv preprint arXiv:2301.12260, 2023.
  * [32] Oleksandr Shchur, Marin Biloš, and Stephan Günnemann.  Intensity-free learning of temporal point processes.  In International Conference on Learning Representations, 2020.
  * [33] Yuqi Si, Elmer V Bernstam, and Kirk Roberts.  Generalized and transferable patient language representation for phenotyping with limited data.  Journal of Biomedical Informatics, 116:103726, 2021.
  * [34] Ethan Steinberg, Ken Jung, Jason A Fries, Conor K Corbin, Stephen R Pfohl, and Nigam H Shah.  Language models are an effective representation learning technique for electronic health record data.  Journal of Biomedical Informatics, 113:103637, 2021.
  * [35] Yanchao Tan, Carl Yang, Xiangyu Wei, Chaochao Chen, Weiming Liu, Longfei Li, Jun Zhou, and Xiaolin Zheng.  Metacare++: Meta-learning with hierarchical subtyping for cold-start diagnosis prediction in healthcare data.  In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’22, page 449–459, New York, NY, USA, 2022. Association for Computing Machinery.
  * [36] Shengpu Tang, Parmida Davarmanesh, Yanmeng Song, Danai Koutra, Michael W Sjoding, and Jenna Wiens.  Democratizing EHR analyses with FIDDLE: a flexible data-driven preprocessing pipeline for structured clinical data.  Journal of the American Medical Informatics Association, 10 2020\.
  * [37] Ritchie Vink, Stijn de Gooijer, Alexander Beedie, J van Zundert, Gert Hulselmans, Cory Grinstead, Marco Edward Gorelli, Matteo Santamaria, Daniël Heres, ibENPC, Jorge Leitao, Marc van Heerden, Colin Jermain, Ryan Russell, Chris Pryer, Adrián Gallego Castellanos, Jeremy Goh, Moritz Wilksch, illumination k, Max Conradt, Liam Brannigan, Joshua Peek, Yu Ri Tan, elbaro, Nicolas Stalder, Søren Havelund Welling, Adam Gregory, paq, and Jakob Keller.  pola-rs/polars: Python polars 0.16.12, March 2023.
  * [38] Shirly Wang, Matthew B. A. McDermott, Geeticka Chauhan, Marzyeh Ghassemi, Michael C. Hughes, and Tristan Naumann.  Mimic-extract: A data extraction, preprocessing, and representation pipeline for mimic-iii.  In Proceedings of the ACM Conference on Health, Inference, and Learning, CHIL ’20, page 222–235, New York, NY, USA, 2020. Association for Computing Machinery.
  * [39] Wes McKinney.  Data Structures for Statistical Computing in Python.  In Stéfan van der Walt and Jarrod Millman, editors, Proceedings of the 9th Python in Science Conference, pages 56 – 61, 2010\.
  * [40] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu.  Point-bert: Pre-training 3d point cloud transformers with masked point modeling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19313–19322, 2022.
  * [41] Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha.  Transformer hawkes process.  In International conference on machine learning, pages 11692–11702. PMLR, 2020.

##  Appendix A Code and Full Technical Documentation of Event Stream GPT

Event Stream GPT is open source, available at
https://github.com/mmcdermott/EventStreamGPT, with documentation accessible at
https://eventstreamml.readthedocs.io.

#### Full Technical Documentation

Full technical documentation for ESGPT can be found in online at
https://eventstreamml.readthedocs.io. It features:

  1. 1.

API: A full API reference of all module classes, functions, configuration
files, etc.

  2. 2.

MIMIC-IV Tutorial: A full walk-through of our MIMIC-IV example, both in
written documentation and in code, with concrete examples of model and
evaluation output results over MIMIC-IV, as well as a link to a stand-alone
github repository for that example.

  3. 3.

Dataset Pipeline: A full description of the dataset pipeline and configuration
language.

  4. 4.

Pre-built Architectures: Full descriptions of our two pre-built model
architectures.

##  Appendix B Generalizing Beyond MIMIC-IV: Dataset Pipeline on Other
Datasets

The ESGPT pipeline is not specific to MIMIC-IV. We have used it on a number of
other datasets, spanning both public and private settings. In Table 4, we show
a summary of the pipeline’s usability across three different datasets: the
public MIMIC-IV dataset [12], the public All of Us Dataset [9], which is a
large, electronic health record (EHR) dataset consisting of broad spectrum
care over a large number of individuals from diverse areas, health statuses,
and backgrounds; and the “HF” dataset, a private dataset sourced from an
ongoing, IRB-approved study regarding the use of machine learning for analysis
of heart failure measurements from an academic research institution consisting
of labs, vitals, and cardiology derived measures for a large number of
patients. We see that the system is able to process large datasets very
quickly with only modest configuration inputs from the user with consistent
space reductions in all three settings.

Dataset | Public? | Input Format | Input Size
---|---|---|---
# Subjects | Disk (GB)
MIMIC-IV [12] | ✓ | PostgreSQL | 300k | 43
All of Us [9] | ✓ | Parquet | 159k | 11
HF |  | CSV | 134k | 32

Dataset | # Lines | CPUs | Time (min) | Mem (GB) | Output | Disk (GB)
---|---|---|---|---|---|---
Subj. | Events | Meas. | Non-DL | DL
MIMIC-IV | 153 | 10 | 30.78 ± 1.12 | 65.93 ± 0.14 | 12k | 3M | 222M | 1.19 | 0.49
All of us∗ | 98 | 64 | 62.03 ± 0.31 | 119.36 ± 0.93 | 145k | 28M | 330M | 2.95 | 3.99
HF | 78 | 128 | 24.96 ± 0.44 | 161.07 ± 0.23 | 116k | 25M | 244M | 3.6 | 1.54

Table 4: Top: Descriptions of the raw data for each of our four datasets.
Bottom: Statistics on computational cost to produce each dataset. “# Lines”
refers to the number of lines in the ‘yaml‘ configuration file used to produce
this dataset. Standard deviations are omitted when they are universally $\leq
0.0$. Memory was assessed via mprof for MIMIC-IV and HF, but due to technical
issues it was assessed via a shell loop with the ‘free‘ command for All of us.

##  Appendix C Further Details on Existing Models

There are a variety of existing foundation model attempts over EHR and related
datasets. In Tables 5 and 6, we summarize a variety of existing models over a
larger class of options than is shown in the main-body Table 3 (in particular,
here we also include Contrastive and Masking based pre-training systems). Even
across these broader examples, we see that evaluation is largely limited to
fine-tuning evaluations, establishing a clear gap regarding zero-shot
evaluation.

Style | Model |  Primary Care |  Hospital |  Intensive Care Unit |  Claims |  Clinical Notes |  Diagnoses |  Procedures |  Medications |  Lab Orders |  Lab Results |  Vitals | “Token” | Patients | T/P | C/T | # Codes
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Care Modalities | Event Types
Causal |  Doctor AI [3] | ✓ |  |  |  |  | ✓ | ✓ | ✓ |  |  |  | Visit | 0.3M | 54.6 | 3.2 | 1.8k
CDT [16] |  | ✓ |  |  |  |  |  | ✓ | ✓ | ✓ | ✓ | Event | 0.0M | 13.6 |  | 0.0k
MetaCare++ [35] |  |  | ✓ |  |  | ✓ |  |  |  |  |  | Visit | 0.0M | 2.2 | 4.8 | 0.9k
CLMBR [34] |  | ✓ |  |  |  | ✓ | ✓ | ✓ | ✓ |  |  | 1 day | 3.4M | 7.0 | 5.0 | 21.7k
MedTPP [5] |  |  | ✓ |  |  |  |  |  |  |  |  | Event | 0.0M | 4.0 |  | 0.1k
MedGPT [14] |  |  |  |  | ✓ | ✓ |  |  |  |  |  | Event | 0.6M |  |  |
Contrastive |  OCP [1] |  |  |  |  | ✓ |  |  |  |  |  |  | Event | 0.1M | 12.0 |  | 30.5k
Hi-BEHRT [17] | ✓ | ✓ |  |  |  | ✓ | ✓ | ✓ | ✓ |  |  | Visit | 2.8M | 62.0 | 4.5 | 3.7k
Masking |  Med-BERT [27] | ✓ | ✓ |  |  |  | ✓ |  |  |  |  |  | Visit | 28.5M |  |  | 82.0k
BEHRT [18] | ✓ | ✓ |  |  |  | ✓ |  |  |  |  |  | Visit | 1.6M |  |  | 0.3k
Graph-Transformer [25] |  |  | ✓ |  |  |  |  | ✓ | ✓ | ✓ | ✓ | 1 hr | 0.0M | 24.0 |  |
EHR-PT [20] |  |  | ✓ |  |  |  |  | ✓ | ✓ | ✓ | ✓ | 1 hr | 0.1M | 70.6 |  |
RAPT [29] |  | ✓ |  |  |  |  |  |  | ✓ | ✓ | ✓ | Visit | 0.1M | 6.8 |  |
CEHR-BERT [24] |  | ✓ |  |  |  | ✓ | ✓ | ✓ | ✓ |  |  | Visit | 2.4M | 14.0 | 5.4 |
GRACE [28] |  |  | ✓ |  |  |  |  |  | ✓ | ✓ | ✓ | Visit | 0.0M | 6.5 |  |
CMS LDS BERT [15] |  |  |  | ✓ |  | ✓ | ✓ |  |  |  |  | Event | 1.2M |  |  | 20.0k
MTL GPR [33] |  |  |  |  | ✓ |  |  |  |  |  |  | 1 hr | 0.0M | 38.0 |  |
T-BEHRT [26] | ✓ | ✓ |  |  |  | ✓ |  |  |  |  |  | Visit | 6.8M |  |  |
Table 5: A summary of existing MedLMs. T/P refers to “tokens” per patient, C/T to codes per “token,” and # Codes to the number of unique codes present in the dataset. Missing values reflect quantities not reported in the source publication. Style | Model |  TTE |  Continuous Outputs |  Categorical Outputs |  Intra-event Dependencies |  Generative |  Fine-tuning |  Zero-shot
---|---|---|---|---|---|---|---|---
Models | Evaluation
Causal |  Doctor AI [3] | ✓ |  | ✓ |  | ✓ |  |
CDT [16] |  |  | ✓ |  |  |  | ✓
MetaCare++ [35] | ✓ |  | ✓ |  | ✓ |  |
CLMBR [34] |  |  | ✓ |  |  | ✓ |
MedTPP [5] | ✓ |  | ✓ |  | ✓ |  |
MedGPT [14] | ✓ |  | ✓ |  | ✓ |  |
Contrastive |  OCP [1] | NA | NA | NA | NA |  | ✓ |
Hi-BEHRT [17] | NA | NA | NA | NA |  |  |
Masking |  Med-BERT [27] | NA |  | ✓ |  |  | ✓ |
BEHRT [18] | NA |  | ✓ |  |  | ✓ |
Graph-Transformer [25] | NA | ✓ | ✓ |  |  | ✓ |
EHR-PT [20] | NA | ✓ | ✓ |  |  | ✓ |
RAPT [29] | NA | ✓ | ✓ |  |  | ✓ |
CEHR-BERT [24] | NA |  | ✓ |  |  | ✓ |
GRACE [28] | NA | ✓ | ✓ |  |  | ✓ |
CMS LDS BERT [15] | NA |  | ✓ |  |  | ✓ |
MTL GPR [33] | NA |  | ✓ |  |  | ✓ |
T-BEHRT [26] | NA |  | ✓ |  |  | ✓ |
Causal | ESGPT (Ours) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Table 6: A summary of existing foundation models over EHR data, broken down by
what modalities of data they model and how they report evaluation results.