# Foundation Models for Electrocardiograms

Junho Song, Jong-Hwan Jang, Byeong Tak Lee, DongGyun Hong,
Joon-myoung Kwon, and Yong-Yeon Jo11footnotemark: 1

AI Group
Medical AI Co., Ltd.
Seoul, Republic of Korea
{jhsong11footnotemark: 1, jangood1122, bytaklee, dghong, cto,
yy.jo11footnotemark<EMAIL_ADDRESS>
Corresponding Author

###### Abstract

Foundation models, enhanced by self-supervised learning (SSL) techniques,
represent a cutting-edge frontier in biomedical signal analysis, particularly
for electrocardiograms (ECGs), crucial for cardiac health monitoring and
diagnosis. This study conducts a comprehensive analysis of foundation models
for ECGs by employing and refining innovative SSL methodologies—namely,
generative and contrastive learning—on a vast dataset of over 1.1 million ECG
samples. By customizing these methods to align with the intricate
characteristics of ECG signals, our research has successfully developed
foundation models that significantly elevate the precision and reliability of
cardiac diagnostics. These models are adept at representing the complex,
subtle nuances of ECG data, thus markedly enhancing diagnostic capabilities.
The results underscore the substantial potential of SSL-enhanced foundation
models in clinical settings and pave the way for extensive future
investigations into their scalable applications across a broader spectrum of
medical diagnostics. This work sets a benchmark in the ECG field,
demonstrating the profound impact of tailored, data-driven model training on
the efficacy and accuracy of medical diagnostics.

_Keywords_ Biosignal process $\cdot$ Foundation model $\cdot$ Generative
learning $\cdot$ Contrastive learning $\cdot$ Electrocardiogram

##  1 Introduction

Electrocardiograms (ECGs) are essential bio-signals that record the heart’s
electrical activity, providing critical information about heart rhythm,
strength, timing, and beat rate. These signals play a pivotal role in
diagnosing heart diseases, detecting abnormalities that may indicate
conditions such as myocardial infarction, arrhythmias, and other cardiac
disorders. Analyzing ECGs is crucial for effective diagnosis and treatment,
making advancements in this field highly impactful.

Deep Neural Networks (DNNs) have emerged as powerful tools for analyzing ECG
data, capable of performing various tasks such as prediction, classification,
and denoising. However, two significant challenges hinder their utilization.
Firstly, ECG data is sensitive medical information that requires strict
privacy protection, leading to a relative scarcity of available data.
Secondly, the low incidence of certain heart diseases results in insufficient
curated ECG datasets with annotated medical labels to effectively train DNNs.

Researchers have adopted foundation models with Self-Supervised Learning (SSL)
to address these challenges, showing great promise in various fields,
including natural language processing (Devlin et al., 2018; Achiam et al.,
2023), computer vision (Chen et al., 2020; He et al., 2022), and speech
recognition (Baevski et al., 2020). In the context of ECG analysis, foundation
models can generalize from unlabeled data and be fine-tuned with minimal
labeled data, making them highly flexible and efficient.

Foundation models with SSL offer several key benefits:

  * •

Reduced Need for Labeled Data: SSL requires only unlabeled data to train
foundation models, learning robust representations without the need for
extensive labeled datasets.

  * •

Enhanced Accuracy and Efficiency: These models can be fine-tuned for specific
downstream tasks, resulting in improved accuracy and faster convergence
compared to training models from scratch.

  * •

Resource Efficiency: Since foundation models are pre-trained, fine-tuning them
requires fewer computational resources, making the process more efficient.

The success of foundation models hinges on effective pre-training, which
relies heavily on SSL methods. Two main streams of SSL are widely adopted:
Contrastive Learning (CL) and Generative Learning (GL). CL enhances the
model’s ability to distinguish between similar and dissimilar instances by
bringing positive pairs closer and pushing negative pairs apart in the
representation space. GL, on the other hand, focuses on reconstructing data by
masking parts of the input, thereby improving the model’s ability to predict
missing information (Lai et al., 2023; Wang et al., 2024; Chien et al., 2022;
Na et al., 2024).

While promising, the application of foundation models for ECGs is still in its
early stages. There is a need for extensive investigation into various
aspects, such as representation schemes, SSL methods, and practical
applications, to fully realize their potential. Our study aims to guide the
development of foundation models for ECGs using a large-scale dataset of over
1.1 million samples. We investigate the models from three perspectives:

  * •

Representation scheme: Analyzing the architectural design based on the Vision
Transformer (ViT (Dosovitskiy et al., 2020)) to customize it for the unique
characteristics of ECG data.

  * •

SSL methods: Evaluating different SSL methodologies, including contrastive,
generative, and hybrid approaches, to determine the most effective strategy
for ECG data representation.

  * •

Practical applications: Validating the efficacy of these models in real-world
scenarios characterized by low incidence rates and restricted data
availability.

Our contributions include:

  * •

Large-Scale Pre-Training: To the best of our knowledge, this is the first work
using a large-scale dataset to pre-train foundation models on ECG data.

  * •

Hybrid SSL Framework: Introducing a novel combination of SSL techniques for
foundation models of ECGs.

  * •

Guidance for Foundation Models: Providing extensive insights through the
comprehensive analysis into the performance of ECG foundation models using
various SSL methods.

The findings of this study are expected to significantly enhance the
application of SSL-enhanced foundation models in ECG classification tasks,
providing a robust framework for future research and practical implementations
in clinical settings.

##  2 Related Work

The advent of foundation models has revolutionized various fields by
leveraging large-scale, unlabeled datasets to produce versatile and robust
data representations. These representations enable models to adapt efficiently
to specific downstream tasks with minimal reliance on labeled data,
significantly enhancing performance and efficiency. In the realm of bio-signal
analysis, particularly ECGs, the application of foundation models enhanced by
Self-Supervised Learning (SSL) techniques has shown immense promise
(Abbaspourazad et al., 2024; Yue et al., 2022).

###  2.1 Advanced Self-Supervised Learning Techniques for Foundation Models

Foundation models utilize advanced SSL techniques during their pre-training
phase to learn meaningful data representations without direct supervision.
These techniques are crucial for enabling versatile applications in subsequent
task-specific training:

  * •

Contrastive Learning (CL): CL improves a model’s ability to form embeddings by
bringing similar instances closer in the embedding space while pushing
dissimilar instances apart. This is typically achieved using a contrastive
loss function, such as InfoNCE, which helps the model develop rich,
semantically meaningful representations from unlabeled data. CL has been
successfully applied across various domains, enhancing model performance on
downstream tasks by improving the quality of learned representations (Chen et
al., 2020).

  * •

Generative Learning (GL): GL techniques, such as Masked AutoEncoders (MAE),
challenge models to reconstruct obscured segments of input data. This process
strengthens the model’s capacity to intuit and regenerate missing data
components; fostering a deeper understanding of complex data patterns. GL has
been particularly effective in applications like image and natural language
processing, where it aids in denoising and anomaly detection (He et al., 2022;
Dosovitskiy et al., 2020).

  * •

Hybrid Learning (HL): HL combines the strengths of both CL and GL, leveraging
the discriminative power of CL and the reconstructive ability of GL. This
synergistic approach has shown promising results in domains such as computer
vision and holds significant potential for ECG analysis and other medical
tasks. By balancing the advantages of both methods, HL aims to create robust
and flexible data representations (Huang et al., 2023).

###  2.2 Pioneering Applications of SSL in Bio-Signal Analysis

SSL methodologies have been increasingly adapted for bio-signal analysis,
particularly in ECG, demonstrating substantial improvements in model accuracy
and generalizability:

  * •

Diverse Approaches to ECG Analysis: SSL has been instrumental in developing
deep multi-task learning frameworks that significantly enhance emotion
recognition from ECG signals, achieving state-of-the-art classification
results. These frameworks illustrate the ability of SSL to capture complex
physiological patterns inherent in ECG data (Sarkar and Etemad, 2020).

  * •

Innovative ECG Classification Techniques: Techniques such as U-ResNet-based
SSL have shown meaningful improvements in ECG classification accuracy,
underscoring the utility of SSL in clinical diagnostics. These methods
leverage SSL pre-training to refine models, leading to better performance
compared to traditional supervised approaches (Gedon et al., 2021).

  * •

Addressing Data Challenges: SSL methods have been employed to overcome
challenges like label imbalance and data scarcity in ECG datasets. Self-
supervised pre-training has demonstrated superior performance in handling
complex bio-signal datasets, highlighting the effectiveness of SSL in managing
the inherent challenges of ECG data (Liu et al., 2021).

  * •

Lead-agnostic Learning Models: Developing SSL methods that are agnostic to ECG
leads and capable of learning both local and global features has shown
improved adaptability and performance on downstream tasks. These approaches
enhance the flexibility and generalizability of SSL in medical applications,
making them valuable for diverse clinical scenarios (Oh et al., 2022).

  * •

Non-contrastive Learning for ECG: The introduction of non-contrastive SSL
approaches, which do not rely on data augmentation or negative pairs, offers a
novel pathway for SSL application in bio-signal processing. These methods
provide an alternative to traditional contrastive techniques, simplifying the
training process and potentially improving model performance (Atienza et al.,
2023).

###  2.3 Research Goals

Despite the promising results and growing adoption of SSL for bio-signal
generalization, such as ECG, EEG, and PPG, there remains a significant gap in
comprehensive implementation strategies for applying foundation models using
diverse SSL frameworks in real-world healthcare settings. The existing
literature often revolves around closed datasets and controlled experimental
conditions, which may not accurately replicate the complexities encountered in
practical environments (Na et al., 2024, Abbaspourazad et al., 2024, Liu et
al., 2024).

Our research aims to bridge these gaps by providing detailed and empirically
supported guidance on pre-training foundation models. We meticulously evaluate
these models across a broad spectrum of architectural designs and SSL
methodologies, enhancing their efficacy for bio-signal applications such as
ECG. By extending our investigations to include a wide range of real-world
applications, we seek to refine these foundation models to better suit
clinical needs, thereby advancing their readiness for broader healthcare
applications and facilitating significant technological advancements in
medical practice.

##  3 Study Procedure

Figure 1: Study procedure

Our study’s methodology unfolds through a meticulously structured sequence of
steps, aiming to provide a comprehensive evaluation of foundation models for
ECG classification. The process includes data preparation, model training,
thorough evaluation, and analysis, ensuring a robust framework for developing
and assessing the performance of these models. The entire procedure of our
study is illustrated in Figure 1.

###  3.1 Data preparation

We utilized two types of datasets: unlabeled and labeled. The unlabeled
dataset, presented in Table 1, was compiled from five public repositories,
MIMIC (Johnson et al., 2016), CODE15 (Ribeiro et al., 2021), BIOBANK (Sudlow
et al., 2015), SAMI (Ribeiro et al., 2021), and IKEM (Seják et al., 2023), to
ensure demographic fairness by including data collected from various
continents. This dataset encompasses a total of 1,291,868 ECG samples from
442,736 distinct patients, aiding in mitigating potential biases and enhancing
the generalizability of the study results.

Table 1: Summary of the unlabeled dataset. | Category | Name | Demographic Information | Statistics
---|---|---|---|---
#Patients | Country | #Sample | #Leads | MV-unit | Duration (s) | Sample rate (Hz)
Unlabeled | Public | MIMIC | 161,352 | USA | 800,035 | 12 | 0.005 | 10 | 500
CODE15 | 233,770 | USA | 341,292 | 12 | 1 | 10 | 400
BIOBANK | 15,365 | UK | 50,780 | 12 | 0.005 | 10 | 500
SAMI | 1,959 | Brazil | 1,631 | 12 | 1 | 10 | 400
IKEM | 30,290 | Czech Republic | 98,130 | 12 | 0.004 | 10 | 500

For the classification tasks, we employed labeled datasets, specifically
utilizing ECGs from the PTB-XL dataset outlined in Table 2. This dataset
includes super-classes: MI (Myocardial Infarction), STTC (ST/T change), CD
(Conduction Disturbance), and HYP (Hypertrophy). These super-classes were
chosen to reflect the diverse morphologies, features, and rhythmic patterns
inherent in ECG data, providing a robust framework for assessing the efficacy
of our foundation models. The dataset comprises a total of 21,265 ECG samples
collected from 8,000 patients.

Table 2: Summary of the labeled dataset. | Category | Name | Demographic Information | Statistics
---|---|---|---|---
#Patients | Label | Case ratio | #Sample | #Leads | Length | Duration (s) | Sample rate (Hz)
Labeled | Public | PTB-XL | 5,466 | MI | 0.2524 | 6,886 | 12 | 5,000 | 10 | 500
5,107 | STTC | 0.2382 | 5,788 | 12 | 5,000 | 10 | 500
4,897 | CD | 0.2211 | 5,772 | 12 | 5,000 | 10 | 500
2,648 | HYP | 0.1209 | 2,819 | 12 | 5,000 | 10 | 500

Before analysis, all data underwent a standardization process to account for
inherent variances in sample rates, measuring durations, and MV-units among
different datasets. To standardize our comparisons and maintain consistency
across analyses, we adjusted the sample rate to 250 Hz, the measurement
duration to 10 seconds, and the MV-units to 1.0, rendering each ECG signal in
our dataset to consist of a 2500 length per lead.

###  3.2 Model Training

####  3.2.1 Foundation models

We adopt widely recognized Self-Supervised Learning (SSL) methodologies to
pre-train our foundation models, specifically focusing on Generative Learning
(GL), Contrastive Learning (CL), and Hybrid Learning (HL). These models
incorporate the Vision Transformer (ViT) (Dosovitskiy et al., 2020) as the
primary architectural backbone, modified with a one-dimensional convolution
projection layer specifically tailored for embedding ECG signals (Woo et al.,
2023). The comprehensive architecture of our foundation models, which
effectively combines these innovative self-supervised learning strategies.

Figure 2: Structure of foundation models

Figure 2 illustrates the overall structure: the preprocessor divides ECG
signals into patches, the encoder extracts features and understands global
contextual relationships, and the decoder reconstructs the original ECG
signals from the embeddings. By integrating these self-supervised learning
methodologies, especially the pioneering HL approach, the foundation models
effectively capture complex patterns in ECG data, significantly improving
their utility in various downstream tasks, such as classification and
prediction, and setting a new benchmark in the field.

In the training of foundation models for ECG analysis, we employ three primary
self-supervised learning (SSL) methodologies: Contrastive Learning (CL),
Generative Learning (GL), and Hybrid Learning (HL), each with a specific focus
and procedure. As depicted in Figure 2, the preprocessor divides the input ECG
signals into smaller patches using a one-dimensional convolution projection
layer.

For CL, these patches are augmented to create different views, forming
positive and negative pairs. The encoder, consisting of 1D-Convolutional
layers and a Transformer block, processes these patches to extract features
and generate embeddings. The contrastive learning objective is to bring the
embeddings of positive pairs closer while pushing the embeddings of negative
pairs apart, achieved using the contrastive loss function (InfoNCE loss):

$L_{CONT}=-\log\frac{\exp\left(\mathbf{z}_{i}\cdot\mathbf{z}_{j}/\tau\right)}{\sum_{k=0}^{N}\exp\left(\mathbf{z}_{i}\cdot\mathbf{z}_{k}/\tau\right)}$
(1)

where $z_{i}$ and $z_{j}$ are the embeddings of the positive pair, $\tau$ is a
temperature parameter, and $N$ is the number of negative samples. During
training, the encoder weights are updated to minimize this loss, enhancing the
model’s ability to distinguish between similar and dissimilar ECG signals.

For GL, the preprocessor randomly masks a portion of the patches, creating
incomplete ECG inputs. The encoder processes the unmasked patches, and the
resulting embeddings are input into the transformer decoder, which aims to
reconstruct the original ECG signals. The generative learning objective is to
minimize the reconstruction loss, quantified using the mean absolute error
(MAE):

$L_{RECON}=\sum_{i=1}^{N}|x_{i}-y_{i}|$ (2)

where $x_{i}$ is the original signal and $y_{i}$ is the reconstructed signal.
The encoder and decoder weights are updated to minimize this loss during
training, improving the model’s ability to predict missing information.

Hybrid Learning (HL) is a novel approach applied for the first time in
training foundation models for ECG data in this study. It combines the
strengths of both CL and GL to create a more robust and versatile learning
framework. The preprocessor creates both augmented pairs for CL and masked
patches for GL. The encoder processes the unmasked patches, generating
embeddings used in both the contrastive and generative learning objectives.
The total hybrid loss combines the contrastive loss and the reconstruction
loss:

$L_{HYBRID}=L_{RECON}+L_{CONT_{Patient}}+L_{CONT_{Sample}}$ (3)

where $L_{CONT_{Patient}}$ and $L_{CONT_{Sample}}$ are the contrastive losses
calculated at the patient and sample levels, respectively. The encoder and
decoder weights are updated to minimize this combined loss, leveraging the
benefits of both learning strategies to enhance the model’s representation
capabilities. This hybrid approach not only maximizes the representational
power of the models but also ensures they are adept at capturing and
categorizing complex patterns within ECG data.

####  3.2.2 Downstream models

After pre-training the foundation models using SSL, we employed two downstream
learning strategies to assess their performance: linear probing and fine-
tuning.

Linear Probing: Linear probing involves freezing the weights of the pre-
trained foundation model and adding a linear classifier on top of its output
layer. This linear classifier is trained using the labeled ECG dataset,
evaluating the quality of the learned representations without updating the
pre-trained model’s weights. This approach provides insights into how well the
foundation model’s pre-trained features can be used for classification tasks.

Fine-Tuning: Fine-tuning involves updating the weights of the entire model,
including both the pre-trained foundation model and the newly added linear
classifier. This step allows the model to adapt more specifically to the
classification tasks using the labeled dataset. Fine-tuning typically results
in better performance than linear probing, as it adjusts the pre-trained
features to better fit the specific characteristics of the labeled data.

###  3.3 Evaluation

The evaluation of these downstream models is conducted from two critical
viewpoints: optimization and applicability.

Optimization: We scrutinize the parameters utilized during the pre-training
phase of the foundation models, focusing on their architectural designs and
SSL methodologies, to determine their impact on the representation of ECG
data. The evaluation includes: Impact of Patch Size, Influence of Block Depth,
Role of Embedding Size.

Applicability: We assess the practical effectiveness of these models in ECG
classification tasks, considering variables such as data availability and case
ratios. This involves validating the efficacy of these models in real-world
scenarios characterized by low incidence rates and restricted data
availability.

###  3.4 Analysis

####  3.4.1 Research Questions

Our research endeavors to elucidate the potential of self-supervised learning
(SSL) in the context of electrocardiogram (ECG) modeling through foundation
models. It systematically addresses several pivotal research questions
designed to deepen our understanding of model development and integration
within practical settings:

RQ1. What architectural designs of foundation models are most effective for
capturing the nuanced semantics of ECG?
The foundation model is pivotal in generalizing electrocardiogram (ECG) data
effectively. Determining the most suitable architectural designs to capture
diverse ECG signal features—including beat rate, pqrst values, morphological
attributes, and temporal correlations—is critical. We conducted an extensive
grid search to investigate the impact of key parameters such as patch size,
depth, and embedding size on data representation. This search involved testing
patch sizes of 250, 125, and 60; depths of 2, 4, and 8; and embedding sizes of
256, 512, and 1024, with embedding sizes set to half that of the encoders.
This methodical exploration aims to identify the architectural designs that
most accurately capture the complex nuances of ECG signals.

RQ2. Which are the most adept at accurately encoding the complex
representations of ECGs among the SSL methods?
Self-supervised learning (SSL) techniques fundamentally influence the way
foundation models represent and process data. Recent studies suggest that the
effectiveness of models trained via generative learning (GL) and contrastive
learning (CL) varies depending on their application in specific downstream
tasks such as classification and prediction (Dubois et al., 2023). Our
research examines how different SSL methods impact the representation of ECG
data when tailored to these tasks. We assessed the effectiveness of each model
using four distinct metrics: approximation error, representation usability
error, probe generalization error, and encoder generalization error, as
outlined in the risk decomposition approach (Dubois et al., 2023). Detailed
methodologies for these metrics are further elaborated in Section B.1.

RQ3. How can foundation models be effectively implemented within clinical
environments to optimize diagnostic processes?
This aspect of our study addresses the deployment of foundation models in
scenarios characterized by limited data and scarce labeling resources. We
evaluated the performance of the optimal architectural designs identified
previously under various conditions of data scarcity. Specifically, we
simulated scenarios with case ratio percentages of 1%, 2%, and 5%, and data
usage percentages of 10%, 25%, and 50%. The objective is to determine the
adaptability and efficacy of these models in diverse real-world situations,
assessing their performance across different levels of data availability and
their potential reliability and utility in resource-constrained environments.

These questions guide our exploration into the practical applications and
theoretical implications of SSL techniques in medical informatics,
particularly within the realm of cardiac health diagnostics. The findings from
our study are expected to provide valuable insights into the development and
deployment of foundation models for ECG analysis, with significant
implications for clinical practice and future research directions.

##  4 Experimental Results

In this section, we provide a summary of the experimental results. Due to
paper length limitations, the complete results are included in Appendix C.

###  4.1 Dataset

For our study, the data was utilized in two primary stages: the foundation
model training and the downstream model training and testing.

We used all available unlabeled data for training the foundation model. This
extensive use of unlabeled data allowed the foundation model to learn robust
and generalized representations of ECG signals through self-supervised
learning techniques.

For the downstream tasks, we utilized labeled data. The labeled dataset was
split into two parts: 80% of the data was used for training and validation,
while the remaining 20% was reserved for the test. It is important to
emphasize that the test set was strictly not used during any training step of
both the foundation model and the downstream model. This separation ensures
that the performance evaluation of the downstream models reflects the
foundation models’ ability to generalize to unseen data, thereby providing an
unbiased assessment of the foundation models’ effectiveness.

###  4.2 Environment

The experiments were conducted using a high-performance computing
infrastructure equipped with a parallel distributed architecture, including 16
NVIDIA A100 GPUs. This setup facilitated the efficient handling of the
extensive datasets and computationally intensive processes required for our
study. To ensure consistent training conditions across all phases, we adopted
a uniform set of optimization parameters. Throughout the pre-training, linear
probing, and fine-tuning stages, we applied a fixed learning rate of $1\times
10^{-4}$ and a weight decay of $1\times 10^{-5}$. These parameters were
selected to optimize convergence and minimize the risk of overfitting.

The initial stage involved pre-training 81 distinct foundation models (all
combinations of 3 patch sizes, 3 block depth, 3 embedding sizes, and 3 SSL
methodologies) on a large dataset of 1.1 million unlabeled electrocardiograms
(ECGs). This extensive pre-training was conducted over approximately one week
for each, aiming to develop robust data representations that form the basis
for later supervised learning tasks.

Following SSL pre-training, the top layers of each model were replaced with
new linear classifiers, while the underlying base layers retained their pre-
trained weights. During this phase, training was restricted to these top
layers using a subset of labeled data to assess the effectiveness of the SSL-
derived features under the established optimization parameters. This process,
known as linear probing, evaluates the quality of the learned representations
without updating the pre-trained model’s weights.

After evaluating the models through linear probing, all layers of the models
were fine-tuned in a separate phase. This comprehensive fine-tuning phase
adapted the entire network’s weights to align more closely with the specific
requirements of ECG classification tasks. The consistent application of the
predetermined learning rate and weight decay during this stage facilitated a
uniform adjustment of the model parameters, enhancing their task-specific
performance.

The efficacy of the models was assessed using the Area Under the Receiver
Operating Characteristic (AUROC) and the Area Under the Precision-Recall Curve
(AUPRC). These metrics provide a robust framework for evaluating the models’
performance across various classification challenges, reflecting their
precision and reliability in handling medical data.

###  4.3 Overall Performance

####  4.3.1 Foundation Model Loss

Table 3: Variation in losses of foundation models by learning method. Hybrid Learning (HL) achieves lower patient and sample-level losses compared to Contrastive Learning (CL). | Patch | Depth | Dim | Total loss | Patient loss | Sample loss | Reconstruction loss
---|---|---|---|---|---|---|---
CL (Contrastive Learning) | 250 | 2 | 512 | 459.2073 | 262.5800 | 196.6271 | -
GL (Generative Learning) | 60 | 2 | 1024 | 11.3272 | - | - | 11.3272
HL (Hybrid Learning) | 60 | 2 | 1024 | 321.2631 | 135.0332 | 169.3104 | 16.9187

The results regarding the loss of foundation models, categorized by the
learning methods of Contrastive Learning (CL), Generative Learning (GL), and
Hybrid Learning (HL), are summarized in Table 3. Detailed evaluations of the
loss variations across different structure settings for these models were
extensively analyzed in Appendix C, with the complete results of the
evaluation presented in Tables 1, 2, and 3.

The findings illustrated that the structure settings significantly influenced
the efficiency of foundation models under different learning frameworks. CL-
based models registered the lowest loss with a patch size of 250, suggesting
that larger patch sizes enhanced the effectiveness of contrastive mechanisms
by encapsulating more comprehensive information cues. Conversely, models based
on GL and HL recorded minimal loss at a smaller patch size of 60, indicating a
preference for more granular data segmentation in these learning approaches.

Additionally, block depth emerged as a crucial factor affecting model
performance, with a depth of 2 blocks proving to be most effective across all
models. This highlighted an optimal balance between model complexity and
computational manageability. For embedding sizes, CL-based models demonstrated
optimal performance with an embedding size of 512, sufficient for capturing
essential features without excessive complexity. In contrast, GL and HL models
exhibited the best loss metrics at a higher embedding size of 1024, implying a
need for more detailed feature extraction to achieve effective learning of ECG
representation.

These findings underscored the intricate dependencies between model
architectural parameters and their operational efficacy within different SSL
paradigms, enriching the understanding of how specific architectural designs
could be optimized to enhance the performance of SSL-based foundation models
in various applications.

####  4.3.2 Linear Probing Performance

Table 4: Best performance of linear-probed downstream models and the corresponding structures of foundation model for each SSL method, using Random Init (random initialization of encoder weights) as the baseline. | Patch | Depth | Dim | AUROC | AUPRC | Criteria
---|---|---|---|---|---|---
MI | STTC | CD | HYP | MI | STTC | CD | HYP
| Random Initialization
---
(Baseline)
- | - | - | 0.6469 | 0.7872 | 0.7261 | 0.7979 | 0.4120 | 0.5844 | 0.5444 | 0.3724 | 4.8713
CL (Contrastive Learning) | 250 | 8 | 256 | 0.7849 | 0.8266 | 0.7998 | 0.8467 | 0.5448 | 0.5891 | 0.6019 | 0.4650 | 5.4628
GL (Generative Learning) | 125 | 8 | 512 | 0.6895 | 0.8458 | 0.7520 | 0.7859 | 0.4740 | 0.6442 | 0.5346 | 0.4299 | 5.1559
HL (Hybrid Learning) | 125 | 4 | 512 | 0.8318 | 0.8165 | 0.8411 | 0.8135 | 0.616 | 0.5911 | 0.635 | 0.4140 | 5.559

Table 4 provided a comprehensive comparison of the linear probing performance
with foundation models trained using Contrastive Learning (CL), Generative
Learning (GL), and Hybrid Learning (HL) against a baseline model initialized
with random encoder weights. The evaluation criteria included AUROC and AUPRC
across four downstream tasks: MI, STTC, CD, and HYP.

Performance of the AUROC revealed that the Hybrid Learning (HL) method
consistently outperformed the other approaches, achieving the highest AUROC
for MI (0.8318) and CD (0.8411). This indicated HL’s superior ability to
distinguish between classes in these tasks. Generative Learning (GL)
demonstrated the highest AUROC for STTC (0.8458), while Contrastive Learning
(CL) led in the HYP task (0.8467). When examining AUPRC, which evaluated the
precision-recall trade-off, HL again excelled with the highest AUROC for MI
(0.616) and CD (0.635), underscoring its robustness in these predictive tasks.
GL, on the other hand, showed the highest AUPRC for STTC (0.6442).

Overall, the performance criteria indicated that Hybrid Learning (HL) achieved
the highest score (5.559), suggesting it was the most balanced and effective
method among the three evaluated. Contrastive Learning (CL) followed with a
score of 5.4628, while Generative Learning (GL) scored 5.1559, both
outperforming the baseline of 4.8713. This comprehensive evaluation
highlighted the efficacy of Hybrid Learning (HL) in achieving superior
performance across various metrics, thereby making it a promising approach for
foundation model training in self-supervised learning contexts.

####  4.3.3 Fine-tuning Performance

Table 5: Best performance of fine-tuned downstream models and the corresponding structures of foundation models for each SSL method, with SL (end-to-end supervised model with labeled data) serving as the baseline. | Patch | Depth | Dim | AUROC | AUPRC | Criteria
---|---|---|---|---|---|---
MI | STTC | CD | HYP | MI | STTC | CD | HYP
SL (Supervised Learning) (Baseline) | 125 | 2 | 256 | 0.9369 | 0.9380 | 0.9384 | 0.9071 | 0.8571 | 0.8351 | 0.8810 | 0.6769 | 6.9715
125 | 2 | 512 | 0.9399 | 0.9309 | 0.9397 | 0.9065 | 0.8652 | 0.8101 | 0.8758 | 0.6820 | 6.9501
125 | 4 | 512 | 0.9404 | 0.9354 | 0.9389 | 0.9167 | 0.8634 | 0.8326 | 0.8780 | 0.6969 | 7.0023
CL (Contrastive Learning) | 125 | 2 | 256 | 0.9406 | 0.9426 | 0.9388 | 0.9272 | 0.8686 | 0.8439 | 0.8745 | 0.7086 | 7.0448
GL (Generative Learning) | 125 | 2 | 512 | 0.9389 | 0.9420 | 0.9362 | 0.9191 | 0.8665 | 0.8354 | 0.8753 | 0.6960 | 7.0094
HL (Hybrid Learning) | 125 | 4 | 512 | 0.9448 | 0.9398 | 0.9449 | 0.9307 | 0.8809 | 0.8339 | 0.8841 | 0.7153 | 7.0744

Table 5 illustrates the performance for fine-tuned models based on the various
foundation models, with a supervised learning (SL) model serving as the
baseline. Among the fine-tuned models, Hybrid Learning (HL) demonstrated the
highest AUROC for MI (0.9448) and CD (0.9449), reflecting its superior ability
to distinguish between classes in these tasks. Contrastive Learning (CL)
achieved the highest AUROC for STTC (0.9426), while also showing strong
performance in other tasks. In terms of AUPRC, which evaluates the precision-
recall trade-off, HL again excelled with the highest AUPRC for MI (0.8809) and
CD (0.8841), further underscoring its robustness in these predictive tasks.

The superior performance of the SSL-based models compared to the SL baseline
can be attributed to the inherent advantages of self-supervised learning. SSL-
based models leverage vast amounts of unlabeled data to pre-train models,
capturing a broad range of patterns and features that might not be as
effectively learned through limited labeled data in SL. This extensive pre-
training enables foundation models to develop more robust and generalizable
representations, which translate to improved performance when fine-tuned on
specific downstream tasks. Additionally, SSL approaches like HL and CL
facilitate better utilization of the data structure and intrinsic
relationships, further enhancing model performance beyond what is achievable
with end-to-end supervised learning alone.

Analyzing the task-specific characteristics, each performance of fine-tuned
models with the various foundation models reveals insights into their
strengths. For Myocardial Infarction (MI) detection, the highest AUROC and
AUPRC of fine-tuned model with the HL-based foundation model indicate its
effectiveness in identifying subtle variations in heart signals, likely due to
its balanced pre-training approach that captures a wide range of features. For
ST/T Changes (STTC) detection, the highest AUROC of fine-tuned model with the
CL-based foundation model suggests that its contrastive approach, which
emphasizes differences between samples, is particularly suited for identifying
specific changes in ECG waveforms. In the case of Conduction Disturbance (CD),
the highest AUROC and AUPRC of fine-tuned model with the HL-based foundation
model highlights its capability to generalize well from pre-training,
capturing necessary patterns indicative of conduction disturbances. For
Hypertrophy (HYP), detecting structural changes in the heart muscle higher
AUROC of fine-tuned model with the CL-based foundation model indicates that
contrastive learning is effective in distinguishing features associated with
hypertrophy, possibly due to its focus on capturing diverse structural
representations.

Overall, the performance criteria suggest that Hybrid Learning (HL) is the
most balanced and effective SSL method, achieving the highest overall score.
Contrastive Learning (CL) and Generative Learning (GL) also performed well,
surpassing the baseline model, but HL’s ability to consistently achieve high
scores across multiple tasks highlights its potential as a robust approach for
foundation model training in self-supervised learning contexts. The superior
performance of SSL methods across tasks underscores their ability to leverage
large-scale unlabeled data for pre-training, capturing complex patterns and
relationships that enhance downstream task performance.

###  4.4 Impact of pre-training strategies on AUROC

To answer the resaerch question, we aim to validate the linear probing and
fine-tuning processes on foundation models utilizing different learning
methodologies—contrastive learning (CL), generative learning (GL), and a
hybrid approach combining both (HL). The four binary classification
tasks—Myocardial Infarction (MI), ST/T Changes (STTC), Conduction Disturbance
(CD), and Hypertrophy (HYP)—serve as benchmarks to assess the efficacy of
these learning techniques under varying parameter settings, such as patch
size, embedding size, and block depth.

####  4.4.1 Patch Size

(a) CL-based

(b) GL-based

(c) HL-based

Figure 3: Impact of patch size on the linear probing models

(a) CL-based

(b) GL-based

(c) HL-based

Figure 4: Impact of patch size on the fine-tuning models

Foundation Model Loss: As shown in both Figure 3(a) and 4(a), the loss of CL-
based foundation model did not show a consistent trend with patch sizes (e.g.,
FM Loss showed v-shaped trend with 2-block depth and 256-embedding size, and
opposite with 4-block depth and 512-embedding size). The loss of GL-based
foundation model showed increasing trend, indicating less learning efficacy in
Figure 3(b) and 4(b). In last, the loss of HL-based foundation model also
increased with larger patch sizes but less than that of GL-based as depicted
in Figure 3(c) and 4(c).

Linear Probing Performance: As shown in Figure 3(a), for the linear probing
results based on the CL-based foundation model, AUROC for MI generally
decreased with larger patch sizes, with a noticeable drop at mid-range patch
sizes (e.g., MI AUROC dropped significantly around the 125 patch size). CD and
HYP showed slight decreases with larger patch sizes but were more stable
compared to MI, while STTC remained generally stable with slight fluctuations.
As shown in Figure 3(b), for the linear probing results based on the GL-based
foundation model, MI and CD showed improvement with larger patch sizes but
with some fluctuations (e.g., AUROC for MI improved from 0.62 to 0.67 as patch
size increased with 4-block depth and 512-embedding size), while STTC and HYP
showed slight decreases with larger patch sizes. As shown in Figure 3(c), for
the linear probing results based on the HL-based foundation model demonstrated
more stable AUROC for all tasks with slight decreases with larger patch sizes.

Fine-Tuning Performance: In Figure 4(a), for the fine-tuning results based on
the CL-based foundation model, high and stable AUROC for MI, STTC, CD, and HYP
were observed, with slight improvements as patch sizes increased (e.g., MI
AUROC remained above 0.93 and slightly improved). As shown in Figure 4(b), for
the fine-tuning results based on the GL-based foundation model, AUROC remained
high and stable across different patch sizes for all tasks, with slight
improvements as patch sizes increased (e.g., AUROC for HYP improved on all
parameter variations). As depicted in Figure 4(c), the HL-based foundation
model showed stable AUROC for STTC and CD with slight improvements as patch
sizes increased. While the AUROC of MI and HYP were decreased slightly as
patch sizes increased. However, the fine-tuning results based on the HL-based
foundation model achieved the highest AUROC for MI, STTC, CD, and HYP, further
improving with larger patch sizes and FM Loss decreased, indicating effective
learning.

####  4.4.2 Block Depth

(a) CL-based

(b) GL-based

(c) HL-based

Figure 5: Impact of block depth on the linear probing models

(a) CL-based

(b) GL-based

(c) HL-based

Figure 6: Impact of block depth on the fine-tuning models

Foundation Model Loss: As shown in both Figure 5 and 6, the loss of CL-, GL-,
and HL-based foundation model increased with deeper block depth, indicating
worse learning (e.g., CL-based FM Loss increased from 524 to 804 with
125-patch size and 256-embedding dims, GL-based FM Loss increased 12.6 to
14.3, and HL-based FM Loss increased from 363 to 368 as block depth
increased).

Linear Probing Performance: As shown in Figure 5(a), for the linear probing
results based on the CL-based foundation model, AUROC for MI, STTC, CD, and
HYP generally showed decreases with deeper block depth (e.g., AUROC for CD
decreased from 0.72 to 0.69 with deeper block depth). The FM Loss increased
with deeper block depth. The linear probing results based on the GL-based
foundation model in Figure 5(b) demonstrated the slight increasing trends,
with AUROC showing variability but generally improving with deeper block
depth. The linear probing results based on the HL-based foundation model in
Figure 5(c) exhibited better performance compared to other linear probing
methods, with AUROC improving slightly with deeper block depth.

Fine-Tuning Performance: For the fine-tuning results based on the CL-based
foundation model in Figure 6(a), high AUROC for all tasks were observed,
slightly improving with deeper block depth (e.g., AUROC for MI increased from
0.93 to 0.94). As shown in Figure 6(b), for the fine-tuning results based on
the GL-based foundation model, AUROC remained stable across different block
depths, with slight improvements. As shown in Figure 6(c), the fine-tuning
results based on the HL-based foundation model achieved the highest AUROC for
all tasks except for HYP, further improving with deeper block depth, while
AUROC of HYP slightly decreased with 125-patch size and 256-embedding size.

####  4.4.3 Embedding Size

(a) CL-based

(b) GL-based

(c) HL-based

Figure 7: Impact of embedding size on the linear probing models

(a) CL-based

(b) GL-based

(c) HL-based

Figure 8: Impact of embedding size on the fine-tuning models

Foundation Model Loss: As depicted in Figure 7 and 8, all FM Loss decreased
with larger embedding sizes, indicating better learning (e.g., CL-based FM
Loss decreased on all settings expcept for 125-patch size and 2-block depth,
GL-based FM Loss decreased on all settings, and HL-based FM Loss decreased on
all settings).

Linear Probing Performance: As shown in Figure 7(a), for the linear probing
results based on the CL-based foundation model, AUROC for MI, STTC, CD, and
HYP generally improved with larger embedding sizes (e.g., AUROC for MI
increased from 0.65 to 0.77 as embedding size increased). The linear probing
results based on the GL-based foundation model in Figure 7(b) demonstrated
similar trends in shallow block depth (2-depth), with AUROC improving with
larger embeddings but showing slight decreases in deep block depth (4-depth)
(e.g., AUROC for all tasks generally improved with larger embeddings with 60-,
125-patch sizes and 2-depth, while decreased with 60-, 125-patch sizes and
4-depth). As shown in Figure 7(c), the linear probing results based on the HL-
based foundation model showed improvements in AUROC for all tasks with 512
embedding size but slight decreases or consistency with 1,024 emebdding sizes.

Fine-Tuning Performance: For the fine-tuning results based on the CL-based
foundation model in Figure 8(a), AUROC for all tasks were slightly decreased
with larger embeddings (e.g., AUROC for HYP decreased in all parameter
settings). As shown in Figure 8(b), for the fine-tuning results based on the
GL-based foundation model, AUROC remained stable across different embedding
sizes for all tasks, with slight improvements as embedding sizes increased.
The fine-tuning results based on the HL-based foundation model in Figure 8(c)
showed slight improving with larger embedding sizes, further achieving the
highest AUROC for all tasks, while FM Loss decreased, indicating effective use
of larger embeddings.

###  4.5 Real-World Applicability

####  4.5.1 Limited Amount of Data

Table 6: Performance of fine-tuned downstream models based on varying percentages of labeled data usage, excluding 100% data usage detailed in Table 5. | Data Usage (%) | AUROC | AUPRC | Criteria
---|---|---|---|---
MI | STTC | CD | HYP | MI | STTC | CD | HYP
SL (Supervised Learning) (Baseline) | 50% | 0.9311 | 0.9292 | 0.9313 | 0.9049 | 0.8454 | 0.8112 | 0.8623 | 0.6618 | 6.8772
25% | 0.9170 | 0.8991 | 0.9089 | 0.8935 | 0.8184 | 0.7391 | 0.8111 | 0.6446 | 6.6317
10% | 0.9006 | 0.9047 | 0.9017 | 0.8609 | 0.7717 | 0.7545 | 0.8077 | 0.6028 | 6.5046
CL (Contrastive Learning) | 50% | 0.9361 | 0.9325 | 0.9339 | 0.9212 | 0.8561 | 0.8095 | 0.8583 | 0.6910 | 6.9386
25% | 0.9161 | 0.9249 | 0.9140 | 0.9145 | 0.8235 | 0.7859 | 0.8219 | 0.6817 | 6.7825
10% | 0.9022 | 0.9237 | 0.9064 | 0.9010 | 0.8015 | 0.7828 | 0.8029 | 0.6237 | 6.6442
GL (Generative Learning) | 50% | 0.9318 | 0.9338 | 0.9318 | 0.9055 | 0.8586 | 0.8155 | 0.8569 | 0.6689 | 6.9028
25% | 0.9067 | 0.9244 | 0.9236 | 0.8999 | 0.8091 | 0.7906 | 0.8419 | 0.6509 | 6.7471
10% | 0.8885 | 0.9204 | 0.8864 | 0.8883 | 0.7676 | 0.7834 | 0.7774 | 0.6367 | 6.5487
HL (Hybrid Learning) | 50% | 0.9420 | 0.9332 | 0.9350 | 0.9178 | 0.8760 | 0.8174 | 0.8609 | 0.6968 | 6.9791
25% | 0.9267 | 0.9259 | 0.9268 | 0.9158 | 0.8467 | 0.7882 | 0.8447 | 0.7084 | 6.8832
10% | 0.9182 | 0.9166 | 0.9112 | 0.8655 | 0.8304 | 0.7613 | 0.8064 | 0.5841 | 6.5937

Table 6 presents the performance metrics of fine-tuned downstream models based
on varying percentages of labeled data usage, excluding 100% data usage. The
models evaluated included an end-to-end supervised learning model (SL) and
three foundation models pre-trained for each downstream task using Contrastive
Learning (CL), Generative Learning (GL), and Hybrid Learning (HL). Performance
is measured using AUROC and AUPRC for four downstream tasks (MI, STTC, CD,
HYP) and an overall criteria score.

The analysis revealed that as the percentage of labeled data decreased, the
performance metrics for all methods dropped. This impact was most significant
at 10% data usage, where both AUROC and AUPRC were notably lower. SL
demonstrated the steepest decline, indicating its reliance on larger labeled
datasets. At 50% data usage, SL showed high AUROC and AUPRC, with a criteria
score of 6.8772. However, as data usage dropped to 25% and 10%, its
performance decreased, with criteria scores of 6.6317 and 6.5046,
respectively.

In contrast, the foundation models exhibited better resilience to reduced
labeled data. CL maintained robust performance across all data usage levels,
achieving a criteria score of 6.9386 at 50%, 6.7825 at 25%, and 6.6442 at 10%.
GL performed comparably to SL at higher data usage levels but showed a slight
advantage at lower data usage, with criteria scores of 6.9028 at 50%, 6.7471
at 25%, and 6.5487 at 10%. HL consistently outperformed all other methods,
demonstrating the highest AUROC and AUPRC across all tasks. It achieved the
highest criteria scores of 6.9791 at 50%, 6.8832 at 25%, and 6.5937 at 10%,
indicating superior performance and resilience.

In summary, foundation models, particularly HL, showed superior performance
and robustness to reduced labeled data compared to the end-to-end supervised
learning model. HL stood out as the most effective approach, maintaining high
performance across varying percentages of labeled data usage, followed by CL
and GL, while SL showed significant performance degradation with decreased
labeled data.

####  4.5.2 Limited Cased of Real-World

Table 7: Performance of fine-tuned downstream models according to varying case ratios, excluding unchanged case ratios detailed in Table 5. | Case Ratio (%) | AUROC | AUPRC | Criteria
---|---|---|---|---
MI | STTC | CD | HYP | MI | STTC | CD | HYP
SL | 5% | 0.9185 | 0.9173 | 0.9168 | 0.9039 | 0.8356 | 0.8001 | 0.8371 | 0.6667 | 6.7960
2% | 0.9045 | 0.8968 | 0.8934 | 0.9115 | 0.7974 | 0.7538 | 0.8017 | 0.6797 | 6.6388
1% | 0.8797 | 0.8969 | 0.8576 | 0.8928 | 0.7684 | 0.7462 | 0.7373 | 0.6281 | 6.407
CL | 5% | 0.9234 | 0.9191 | 0.9259 | 0.9208 | 0.8475 | 0.7917 | 0.8441 | 0.7006 | 6.8731
2% | 0.9176 | 0.9167 | 0.9104 | 0.9142 | 0.8237 | 0.7866 | 0.8188 | 0.6747 | 6.7627
1% | 0.8991 | 0.9030 | 0.8921 | 0.8995 | 0.7975 | 0.7442 | 0.7995 | 0.6477 | 6.5826
GL | 5% | 0.9231 | 0.9312 | 0.9247 | 0.9144 | 0.8408 | 0.8132 | 0.8474 | 0.6911 | 6.8859
2% | 0.8912 | 0.8850 | 0.9067 | 0.9052 | 0.7903 | 0.7103 | 0.8169 | 0.6759 | 6.5815
1% | 0.8756 | 0.8856 | 0.8729 | 0.8850 | 0.7536 | 0.7400 | 0.7666 | 0.6389 | 6.4182
HL | 5% | 0.9294 | 0.9278 | 0.9304 | 0.9246 | 0.8636 | 0.8094 | 0.8564 | 0.7080 | 6.9496
2% | 0.9116 | 0.9142 | 0.9113 | 0.9140 | 0.8349 | 0.7759 | 0.8181 | 0.6843 | 6.7643
1% | 0.9069 | 0.9124 | 0.9053 | 0.8816 | 0.8133 | 0.7773 | 0.8175 | 0.6166 | 6.6309

Table 7 shows the performance of fine-tuned downstream models that was
evaluated according to varying case ratios (5%, 2%, 1%). The models assessed
included an end-to-end supervised learning model (SL) and three foundation
models pre-trained using Contrastive Learning (CL), Generative Learning (GL),
and Hybrid Learning (HL). The fine-tuned models for each downstream task were
based on these foundation models. Performance was measured using AUROC (Area
Under the Receiver Operating Characteristic Curve) and AUPRC (Area Under the
Precision-Recall Curve) for four downstream tasks: MI, STTC, CD, and HYP, with
an additional overall criteria score for each model and case ratio.

Across all learning methods, both AUROC and AUPRC metrics generally decreased
as the case ratio decreased, indicating a drop in model performance with fewer
training cases. However, a key observation is that the Hybrid Learning (HL)
model exhibited less performance decline with the drop in case ratio compared
to other methods. For instance, in the HL model, the AUROC for MI decreased
from 0.9294 at 5% to 0.9069 at 1%, a drop of approximately 2.4%, while the
AUPRC for MI fell from 0.8636 to 0.8133, a decline of about 5.8%. This trend
indicates that HL models maintain robustness better than CL and GL models
under reduced data conditions. For comparison, the AUROC for MI in the CL
model decreased from 0.9234 at 5% to 0.8991 at 1%, a drop of around 2.6%,
while the AUPRC decreased from 0.8475 to 0.7975, a decline of approximately
5.9%.

Furthermore, the fine-tuned models based on foundation models (CL, GL, and HL)
consistently outperformed the end-to-end SL model across all tasks and case
ratios. For example, the CL model’s AUROC for MI was 0.9234 at 5% and 0.8991
at 1%, whereas the SL model’s AUROC for MI dropped from 0.9185 at 5% to 0.8797
at 1%, a decrease of about 4.2%. The overall criteria values, reflecting
aggregate performance, also decreased with lower case ratios across all
learning methods, aligning with the trends observed in AUROC and AUPRC
metrics.

Overall, the table demonstrates the robustness and effectiveness of different
learning methods under varying data availability conditions, with hybrid
learning showing the least performance degradation and fine-tuned models based
on foundation models exhibiting superior performance compared to the end-to-
end SL model.

###  4.6 Ablations

####  4.6.1 Risk decomposition

Table 8: Results of risk decomposition of the fine-tuned downstream models. | Approximation Error | Representation Usability Error
---|---|---
MI | STTC | CD | HYP | AVR. | MI | STTC | CD | HYP | AVR.
CL | 0.3600 | 0.3100 | 0.3400 | 0.1300 | 0.2850 | 0.0450 | 0.0200 | 0.0100 | -0.0050 | 0.0175
GL | 0.1300 | 0.5250 | 0.3700 | 0.3850 | 0.3525 | -0.0050 | -0.1650 | 0.0300 | -0.0050 | -0.0363
HL | 0.1000 | 0.4500 | 0.3600 | 0.3500 | 0.3150 | 0.0000 | -0.1400 | -0.0150 | 0.0300 | -0.03125
| Probe Generalization Error | Encoder Generalization Error
MI | STTC | CD | HYP | AVR. | MI | STTC | CD | HYP | AVR.
CL | -0.0650 | 0.0300 | -0.0100 | -0.0050 | -0.0125 | 0.0700 | 0.0000 | -0.0650 | 0.0250 | 0.0075
GL | 0.0350 | 0.2000 | -0.0200 | -0.0400 | 0.0438 | -0.0200 | -0.0600 | 0.0600 | 0.1050 | 0.0213
HL | -0.0600 | 0.1700 | 0.0150 | 0.0200 | 0.0363 | 0.0900 | 0.0200 | -0.0600 | 0.0800 | 0.0325

Table 8 presented a comprehensive risk decomposition analysis ( Dubois et al.,
2023) across various SSL methodologies—CL, GL, and HL—evaluated over multiple
ECG classification tasks (MI, STTC, CD, HYP). Each methodology was assessed
across four dimensions of error: approximation, representation usability,
probe generalization, and encoder generalization, with a summary of average
errors presented in the AVR column.

##### Approximation Error

Our findings revealed significant differences in approximation errors among
the methodologies. The CL methodology exhibited the highest approximation
errors across nearly all tasks, suggesting that its encoder’s architectural
capacity might be insufficient for the complexity of the tasks. This was
particularly evident in tasks with higher dimensional data requirements, where
the capacity to capture intricate patterns is crucial. In stark contrast, the
HL methodology demonstrated robust capabilities by consistently recording the
lowest approximation errors, notably excelling in the CD task. This suggests
that HL’s architecture is potentially more suited for handling the intricacies
of ECG data. The GL methodology, while generally performing moderately, showed
its best performance in the MI task, indicating a possible alignment of its
architectural strengths with the specific demands of this task.

##### Representation Usability Error

Representation usability error analysis further differentiated the
methodologies. HL consistently showed the lowest errors, indicating its
superior efficacy in generating linearly separable and usable representations
for downstream tasks. This is critical in clinical applications where the
reliability of ECG classification can directly impact diagnostic outcomes.
Conversely, GL’s highest usability error in the MI task suggests its potential
limitations in conditions requiring high discriminative power from the
representations, which might impede its applicability in more nuanced or
complex clinical scenarios.

##### Probe Generalization Error

The probe generalization error results provided insights into the models’
ability to generalize from learned representations to unseen data. The
negative values observed in CL’s performance on the STTC task highlight its
strengths in generalization, potentially making it suitable for applications
where robustness to new data is paramount. However, GL’s relatively high
generalization errors, especially in the CD task, underscore some challenges
it may face in environments with high variability or where the training data
does not comprehensively represent the test scenarios.

##### Encoder Generalization Error

In terms of encoder generalization, HL again proved to be superior,
particularly in the STTC and HYP tasks, which suggests that its pretraining
approach may better encapsulate the underlying patterns of the ECG data. This
robustness is invaluable for practical applications where models often
encounter data that differ from the training set. The higher encoder
generalization errors observed with the CL methodology indicate potential
overfitting to the training data, which could limit its effectiveness in
broader clinical implementations.

##  5 Discussion

###  5.1 RQ1. What architectural designs of foundation models are most
effective for capturing the nuanced semantics of ECG data?

Based on the results from Section 4.4.1, 4.4.2, and 4.4.3, the research
question can be answered as follows:

For patch size, our analysis shows that larger patch sizes generally improve
performance, especially in fine-tuned configurations. This means that
capturing greater contextual information within each patch helps the model
better understand broader patterns in the ECG data. For example, in fine
tuning, the results of the CL-based foundation model showed high and stable
AUROC for all tasks, although there was some improvement as patch size
increased. Similarly, the HL-based foundation model achieved the highest AUROC
with larger patch sizes. However, in the linear probing results of the GL
foundation-based model, the overall performance of all tasks decreased and FM
Loss tended to increase. Therefore, for GL-based foundation models, small
patch sizes are effective.

In the case of block depth, a slight performance improvement was confirmed in
the finetuning results, but these results are not efficient in terms of
resource use due to increased model complexity. In addition, linear probing
results showed a tendency for performance to decrease and FM loss to increase.
These results indicate that the generalizability of the foundation model is
reduced. Therefore, it can be said that shallow block depth is efficient for
the generalizability of the foundation model and the performance of the
donwsteam model.

For the embedding size, larger embedding sizes consistently improved AUROC
across both linear probing and fine-tuning configurations. This indicates that
a larger embedding size allows the model to represent more complex features
and patterns within the ECG data. For example, the CL-based method for fine-
tuning showed high and stable AUROC, improving with larger embeddings. The HL-
based method also achieved the highest AUROC with larger embeddings,
indicating that a richer representation space enhances the model’s ability to
capture nuanced ECG semantics.

Our analysis shows that the most effective architectural design for a
foundation model to capture nuances in ECG data is to use larger patch sizes,
shallow block depths, and larger embedding sizes. These design choices allow
the base model to capture broader contextual information, represent complex
features, and learn hierarchical patterns within ECG data.

###  5.2 RQ2. Which are the most adept at accurately encoding the complex
representations of ECGs among the SSL methods?

Based on the results from Section 4.3.1, 4.3.2, and 4.3.3, the research
question can be answered as follows:

Hybrid Learning (HL) emerges as the most adept method at accurately encoding
the complex representations of ECGs among the SSL methods evaluated. The table
on variation in losses (foundation model losses) showed that HL demonstrated
lower patient and sample losses compared to Contrastive Learning (CL).
Although HL did not have the lowest total loss, its balanced trade-offs
between different loss types suggested its capability to handle complex
representations effectively.

In Table 4, HL consistently achieved the highest AUROC and AUPRC in key tasks,
particularly for Myocardial Infarction (MI) and Conduction Disturbance (CD).
This indicated HL’s superior performance in initial linear probing, showcasing
its strength in encoding meaningful ECG representations. Furthermore, in Table
5, HL again outperformed other methods, achieving the highest AUROC for MI and
CD, and the highest AUPRC for MI and CD. While Contrastive Learning (CL)
excelled in ST/T Changes (STTC), HL’s overall performance was more balanced
and robust.

Therefore, HL’s ability to balance various loss types during pre-training,
combined with its superior performance in both linear-probed and fine-tuned
downstream tasks, highlights its effectiveness in capturing the intricate
features and patterns within ECG data. The consistent high performance of HL
across multiple evaluation criteria underscores its robustness and reliability
in encoding complex ECG representations, making it the most suitable method
for this purpose.

###  5.3 RQ3. How can foundation models be effectively implemented within
clinical environments to optimize diagnostic processes?

The analysis of results from Section 4.5 provides insights into effectively
implementing foundation models within clinical environments to enhance
diagnostic processes.

The study evaluated the performance of optimal architectural designs under
conditions of data scarcity by simulating scenarios with case ratio
percentages of 1%, 2%, and 5%, and data usage percentages of 10%, 25%, and
50%. These simulations aimed to determine the adaptability and efficacy of the
models in diverse real-world situations, assessing their performance across
different levels of data availability. The results indicated that Hybrid
Learning (HL)-based models maintained robust performance even with limited
data, highlighting their potential reliability and utility in resource-
constrained environments.

Foundation models, particularly those utilizing HL, have demonstrated
significant robustness when dealing with limited labeled data. This
adaptability is crucial in clinical environments where obtaining large volumes
of annotated medical data is challenging. The ability to perform well with
smaller datasets enhances the practicality of these models for clinical use,
where data scarcity is a common issue.

Implementing foundation models can enhance diagnostic accuracy. Leveraging
pre-trained models fine-tuned with smaller labeled datasets allows healthcare
providers to achieve higher diagnostic precision. This is particularly
beneficial for detecting conditions with low incidence rates, as the models
can be fine-tuned to recognize subtle variations in ECG signals indicative of
specific pathologies.

Once pre-trained, foundation models require fewer computational resources for
fine-tuning. This efficiency is advantageous for clinical settings where
computational resources may be limited. The pre-training phase, involving
extensive use of unlabeled data, can be conducted using high-performance
computing facilities, while the fine-tuning phase can be performed with more
modest computational setups available in clinical environments.

The scalability of foundation models facilitates their application across
various clinical tasks beyond ECG analysis. Their flexible architecture can be
adapted to other diagnostic processes, such as imaging and genetic data
analysis, providing a comprehensive tool for healthcare providers. This
versatility is vital for integrating advanced diagnostic models into routine
clinical workflows, enhancing overall healthcare delivery.

Incorporating data from diverse sources and demographics during the pre-
training phase ensures that foundation models generalize well across different
patient populations. This diversity mitigates biases and improves the models’
applicability in varied clinical settings, supporting more equitable
healthcare outcomes.

Implementing foundation models within clinical environments allows for
continuous learning and improvement. Regularly updating the models with new
data ensures that diagnostic tools evolve with emerging medical knowledge and
patient data trends, maintaining their relevance and accuracy over time.

In conclusion, effectively implementing foundation models within clinical
environments involves leveraging their adaptability to limited data,
optimizing diagnostic accuracy, ensuring resource efficiency, and maintaining
scalability and flexibility. These models provide a robust framework for
enhancing diagnostic processes, ultimately contributing to improved patient
care and outcomes.

###  5.4 Comprehensive Analysis

The observed discrepancies in optimal parameter settings across Tables 3, 4,
and 5 can be attributed to several factors inherent to the nature of the tasks
(loss minimization versus performance maximization) and the characteristics of
the different learning methods.

The objectives of these tasks are fundamentally distinct. In Table 3, the goal
is to identify parameter settings that minimize total loss, encompassing
patient loss, sample loss, and reconstruction loss. This minimization focuses
on reducing errors during the pre-training phase, which includes various types
of losses depending on the learning method. In contrast, Table 4 aims to
maximize the AUROC and AUPRC during linear probing, assessing the quality of
pre-trained representations by training a simple classifier on top of the
frozen pre-trained model. Table 5 seeks to maximize AUROC and AUPRC during
fine-tuning, which involves updating the weights of the entire model,
including pre-trained layers, to adapt to specific downstream tasks. The
distinct objectives of these tasks naturally lead to different optimal
parameter settings.

The nature of the learning methods significantly influences the optimal
settings. Contrastive Learning (CL) focuses on learning representations by
contrasting positive and negative pairs, implying that the optimal settings
for minimizing losses might differ from those providing the best downstream
task performance due to the emphasis on distinguishing between similar and
dissimilar samples. Generative Learning (GL) focuses on generating data
samples that align with the training data distribution. Consequently, the
optimal settings for minimizing reconstruction loss may not directly translate
to the best settings for discriminative tasks like linear probing and fine-
tuning. Hybrid Learning (HL) combines aspects of both contrastive and
generative learning, balancing between loss minimization and downstream task
performance, which results in different optimal settings for pre-training
losses and downstream performance metrics.

The impact of parameter settings also contributes to these differences.
Parameters such as patch size, block depth, and embedding size affect the
model’s capacity and ability to capture different levels of information.
Smaller patch sizes capture more detailed information, which is crucial for
loss minimization but might not always translate to optimal performance for
downstream tasks. For instance, linear probing results based on the CL-based
foundation model showed that AUROC for MI generally decreased with larger
patch sizes, while CD and HYP exhibited slight decreases and were more stable
compared to MI, and STTC remained generally stable with slight fluctuations
(Section 4.4.1, Figure 3(a)). Conversely, larger embedding sizes capture more
complex representations, which might benefit linear probing and fine-tuning
but could lead to higher pre-training losses if not managed well. The GL-based
foundation model demonstrated slight increasing trends, with AUROC generally
improving with deeper block depth but showing variability (Section 4.4.2,
Figure 5(b)). Similarly, the HL-based foundation model showed improvements in
AUROC for all tasks with a 512 embedding size but exhibited slight decreases
or consistency with a 1,024 embedding size (Section 4.4.3, Figure 7(c)).

Overfitting considerations are crucial in the context of Table 3. Overfitting
occurs when a model learns not only the underlying patterns but also the noise
in the training data, resulting in poor generalization to new, unseen data.
Overfitting risks vary across different parameter settings. In pre-training
loss minimization (Table 3), an overemphasis on minimizing total loss could
lead to overfitting if the model becomes too tailored to the specific details
and noise of the training data. Smaller patch sizes and larger embedding sizes
could exacerbate overfitting by capturing more granular details, including
noise. To mitigate overfitting, various techniques such as regularization,
dropout, and careful monitoring of validation loss can be applied during pre-
training.

In conclusion, the differences in optimal parameter settings across the tables
arise from the distinct goals and characteristics of the tasks (loss
minimization versus performance maximization) and the learning methods
(contrastive, generative, hybrid). Each method’s nature and the way parameters
influence different types of losses and performance metrics lead to varying
optimal settings for pre-training and downstream tasks. Overfitting is a
crucial factor that needs to be managed during pre-training to ensure that the
models generalize well to new, unseen data. Proper regularization, data
augmentation, and careful monitoring of performance metrics are essential to
mitigate the risks of overfitting.

###  5.5 Limitations and Future Works

##### Limitations

Despite the extensive dataset of over 1.1 million ECG samples, the inherent
imbalance in medical datasets, particularly in the representation of rare
conditions, poses a significant challenge. This imbalance can lead to biased
models that perform well on common conditions but poorly on rare ones.
Additionally, while the datasets used in this study are comprehensive, they
may not fully capture the diversity seen in global populations. Variations in
demographic factors such as age, gender, and ethnicity can affect ECG
characteristics, and models trained on limited demographic data might not
generalize well to all patient populations.

Another critical limitation is model interpretability. Foundation models,
especially those based on deep learning architectures like Vision Transformers
(ViT), often act as black boxes, making it difficult to interpret their
decision-making processes. This lack of transparency can hinder clinical
adoption, where understanding the rationale behind a diagnosis is crucial.
Furthermore, the pre-training and fine-tuning of large-scale foundation models
require significant computational resources, which may not be readily
available in all research or clinical settings. This limitation can restrict
the accessibility and scalability of these models. Lastly, handling large-
scale medical data, particularly ECGs, involves stringent privacy and security
considerations. Ensuring that data is anonymized and securely managed is
essential, yet challenging, especially when dealing with cross-institutional
data sharing.

##### Future Works

To address data imbalance, future work can explore advanced data augmentation
techniques and synthetic data generation to create a more balanced and
representative dataset, which can help improve model performance on rare
conditions. Increasing the diversity of training datasets by including more
samples from underrepresented demographic groups can enhance the
generalizability of the models. Collaborations with international healthcare
institutions could provide access to a more varied patient population.

Developing methods to make the decision-making processes of foundation models
more interpretable will be crucial for clinical adoption. Techniques such as
attention mechanism visualization, feature importance mapping, and explainable
AI (XAI) frameworks can be explored. Research into more efficient training
algorithms and model architectures that require less computational power can
make these models more accessible. Techniques such as model pruning,
quantization, and knowledge distillation can be investigated.

Future work should also focus on developing robust privacy-preserving
techniques, such as federated learning and differential privacy, to enable the
secure use of sensitive medical data without compromising patient
confidentiality. Conducting clinical trials to validate the performance of
these foundation models in real-world settings is essential. This will help
assess their practical utility and effectiveness in diverse clinical
environments. Finally, exploring cross-modal learning techniques that
integrate ECG data with other types of medical data (e.g., imaging, genetic
information) can provide a more holistic understanding of patient health and
improve diagnostic accuracy.

By addressing these limitations and exploring these future directions, the
robustness, generalizability, and clinical applicability of foundation models
for ECG analysis can be significantly enhanced.

##  6 Conclusion

This paper presents comprehensive findings on developing and applying
foundation models for electrocardiogram (ECG) using Self-Supervised Learning
(SSL). The study significantly contributes to the field by pioneering large-
scale foundation models specifically tailored for ECG data. These models are
trained using a hybrid SSL framework that integrates contrastive and
generative learning methods, providing robust and flexible tools for ECG
classification in practical healthcare settings. The effectiveness of these
models has been demonstrated through extensive testing and optimization
processes, focusing on various architectural designs and SSL methods to
enhance ECG representation. This involved exploring different parameter
architectural designs, which were crucial in determining the optimal settings
for achieving the best performance in practical applications, particularly in
scenarios with limited labeled data. One of the key conclusions is that while
the foundation models excel in ECG classification, the choice of the SSL
method significantly impacts their effectiveness. The hybrid learning
approach, which combines the strengths of both generative and contrastive
learning, has shown superior performance in many cases, suggesting a promising
direction for future research and application. Moreover, the study confirms
the practical viability of these models in real-world settings, emphasizing
the reduced need for labeled data and the potential for quicker, more
efficient diagnostic processes. This advancement is expected to facilitate
broader adoption and further innovation in the use of deep-learning techniques
in cardiology and other medical fields. Ultimately, the findings advocate for
continued exploration and development of foundation models in medical AI,
encouraging further studies to refine these models and expand their
applications across more varied and complex medical datasets.

## References

  * Devlin et al. [2018] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova,  Bert: Pre-training of deep bidirectional transformers for language understanding,  arXiv preprint arXiv:1810.04805 (2018).
  * Achiam et al. [2023] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al.,  Gpt-4 technical report,  arXiv preprint arXiv:2303.08774 (2023).
  * Chen et al. [2020] T. Chen, S. Kornblith, M. Norouzi, G. Hinton,  A simple framework for contrastive learning of visual representations,  in: International conference on machine learning, PMLR, 2020, pp. 1597–1607.
  * He et al. [2022] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, R. Girshick,  Masked autoencoders are scalable vision learners,  in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 16000–16009.
  * Baevski et al. [2020] A. Baevski, Y. Zhou, A. Mohamed, M. Auli,  wav2vec 2.0: A framework for self-supervised learning of speech representations,  Advances in neural information processing systems 33 (2020) 12449–12460.
  * Lai et al. [2023] J. Lai, H. Tan, J. Wang, L. Ji, J. Guo, B. Han, Y. Shi, Q. Feng, W. Yang,  Practical intelligent diagnostic algorithm for wearable 12-lead ecg via self-supervised learning on large-scale dataset,  Nature Communications 14 (2023) 3741.
  * Wang et al. [2024] Y. Wang, Y. Han, H. Wang, X. Zhang,  Contrast everything: A hierarchical contrastive framework for medical time-series,  Advances in Neural Information Processing Systems 36 (2024).
  * Chien et al. [2022] H.-Y. S. Chien, H. Goh, C. M. Sandino, J. Y. Cheng,  Maeeg: Masked auto-encoder for eeg representation learning,  arXiv preprint arXiv:2211.02625 (2022).
  * Na et al. [2024] Y. Na, M. Park, Y. Tae, S. Joo,  Guiding masked representation learning to capture spatio-temporal relationship of electrocardiogram,  arXiv preprint arXiv:2402.09450 (2024).
  * Dosovitskiy et al. [2020] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al.,  An image is worth 16x16 words: Transformers for image recognition at scale,  arXiv preprint arXiv:2010.11929 (2020).
  * Abbaspourazad et al. [2024] S. Abbaspourazad, O. Elachqar, A. Miller, S. Emrani, U. Nallasamy, I. Shapiro,  Large-scale training of foundation models for wearable biosignals,  in: The Twelfth International Conference on Learning Representations, 2024.
  * Yue et al. [2022] Z. Yue, Y. Wang, J. Duan, T. Yang, C. Huang, Y. Tong, B. Xu,  Ts2vec: Towards universal representation of time series,  in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 2022, pp. 8980–8987.
  * Huang et al. [2023] Z. Huang, X. Jin, C. Lu, Q. Hou, M.-M. Cheng, D. Fu, X. Shen, J. Feng,  Contrastive masked autoencoders are stronger vision learners,  IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).
  * Sarkar and Etemad [2020] P. Sarkar, A. Etemad,  Self-supervised ecg representation learning for emotion recognition,  IEEE Transactions on Affective Computing 13 (2020) 1541–1554.
  * Gedon et al. [2021] D. Gedon, A. H. Ribeiro, N. Wahlström, T. B. Schön,  First steps towards self-supervised pretraining of the 12-lead ecg,  2021 Computing in Cardiology (CinC) 48 (2021) 1–4.
  * Liu et al. [2021] H. Liu, Z. Zhao, Q. She,  Self-supervised ecg pre-training,  Biomed. Signal Process. Control. 70 (2021) 103010.
  * Oh et al. [2022] J. Oh, H. Chung, J. Kwon, D. Hong, E. Choi,  Lead-agnostic self-supervised learning for local and global representations of electrocardiogram,  ArXiv abs/2203.06889 (2022).
  * Atienza et al. [2023] A. Atienza, J. Bardram, S. Puthusserypady,  Subject-based non-contrastive self-supervised learning for ecg signal processing,  ArXiv abs/2305.10347 (2023).
  * Liu et al. [2024] Z. Liu, A. Alavi, M. Li, X. Zhang,  Self-supervised learning for time series: Contrastive or generative?,  arXiv preprint arXiv:2403.09809 (2024).
  * Johnson et al. [2016] A. E. Johnson, T. J. Pollard, L. Shen, L.-w. H. Lehman, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. Anthony Celi, R. G. Mark,  Mimic-iii, a freely accessible critical care database,  Scientific data 3 (2016) 1–9.
  * Ribeiro et al. [2021] A. H. Ribeiro, G. M. Paixao, E. M. Lima, M. Horta Ribeiro, M. M. Pinto Filho, P. R. Gomes, D. M. Oliveira, W. Meira Jr, T. B. Schon, A. L. P. Ribeiro,  CODE-15%: a large scale annotated dataset of 12-lead ECGs (2021).
  * Sudlow et al. [2015] C. Sudlow, J. Gallacher, N. Allen, V. Beral, P. Burton, J. Danesh, P. Downey, P. Elliott, J. Green, M. Landray, et al.,  Uk biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age,  PLoS medicine 12 (2015) e1001779.
  * Ribeiro et al. [2021] A. L. P. Ribeiro, A. H. Ribeiro, G. M. Paixao, E. M. Lima, M. Horta Ribeiro, M. M. Pinto Filho, P. R. Gomes, D. M. Oliveira, W. Meira Jr, T. B. Schon, E. C. Sabino,  Sami-Trop: 12-lead ECG traces with age and mortality annotations (2021).
  * Seják et al. [2023] M. Seják, J. Sido, D. Žahour,  Ikem dataset v1.0.0 (2023).
  * Woo et al. [2023] S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweon, S. Xie,  Convnext v2: Co-designing and scaling convnets with masked autoencoders,  in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 16133–16142.
  * Dubois et al. [2023] Y. Dubois, T. Hashimoto, P. Liang,  Evaluating self-supervised learning via risk decomposition,  arXiv preprint arXiv:2302.03068 (2023).

##  Appendix A Data sets

###  A.1 Summary of Data sets

This section provides a detailed description of the various datasets used in
the study, including MIMIC-IV, CODE15, UK Biobank, SaMi-Trop, and IKEM. Each
dataset description includes demographic information, the number of samples,
and specific details about the data collection and characteristics.

MIMIC-IV (Johnson et al., 2016) is an extensive database that contains
comprehensive data on patients admitted to a tertiary academic medical center
in Boston, MA, USA. The database provides a wealth of information about each
patient while hospitalized, including laboratory measurements, medications
administered, documented vital signs, and other relevant medical data. The
primary aim of this database is to support healthcare research.

CODE15 (Ribeiro et al., 2021) is a dataset of 345,779 ECG exams from 233,770
patients. The ECGs were sampled at a rate of 400 Hz and had varying durations,
but were standardized to ECGs of 4096 points by filling them with zeros on
both sides.

UK Biobank (Sudlow et al., 2015) is a publicly available dataset containing
data on 500,000 individuals from the United Kingdom. This population-based
study includes individuals aged 40 to 69 years old who were recruited between
2006 and 2010. Resting 12-lead ECGs and interval measurements were available
for 15,365 participants, and 13,314 were included in the analysis after
excluding those with AF on ECG or AF identified from self-reports at baseline
or electronic healthcare records.

SaMi-Trop (Ribeiro et al., 2021) is a prospective cohort study funded by the
NIH. It includes 1,959 patients with chronic Chagas cardiomyopathy and aims to
evaluate the usefulness of a clinical prediction rule based on ECG, brain
natriuretic peptide (BNP) levels, and other biomarkers in clinical practice. A
subset of the SaMi-Trop dataset with annotations of age and mortality and the
correspondent ECG traces is openly available.

IKEM (Seják et al., 2023) is a dataset of 12-lead ECG recordings collected
from patients examined by the cardiology or diabetology sections at the
Institute for Clinical and Experimental Medicine (IKEM) in Prague, Czech
Republic. The recordings were sampled at a rate of 500 Hz for 10 seconds each.
Each entry in the dataset includes raw ECG recordings paired with an
anonymized unique patient ID, which groups ECG recordings belonging to the
same patient.

###  A.2 Characteristics of ECG Annotations

This section elaborates on the annotated classes used in the study, including
Myocardial Infarction (MI), ST/T Changes (STTC), Conduction Disturbance (CD),
and Hypertrophy (HYP). It includes descriptions of each condition, their
significance, and the typical ECG characteristics associated with each.

##### MI (Myocardial Infarction)

  * •

Description: Myocardial Infarction, commonly known as a heart attack, involves
the interruption of blood supply to a part of the heart, causing heart cells
to die. This is typically caused by a blockage in one or more coronary
arteries due to plaque (a mix of fat, cholesterol, and other substances).

  * •

ECG Characteristics: The ECG of a person suffering from MI may show abnormal Q
waves, elevation or depression of the ST segment, and T wave inversions.
Diagnosing MI involves identifying these specific patterns on different parts
of the ECG.

##### STTC (ST/T Changes)

  * •

Description: ST/T changes on an ECG can indicate various conditions including
cardiac ischemia, electrolyte imbalances, and other myocardial injuries. The
ST segment and T wave are parts of the ECG cycle associated with the heart’s
electrical recovery (repolarization) following a contraction.

  * •

ECG Characteristics: ST segment elevations can indicate acute myocardial
injury (like in the case of an ongoing heart attack), while ST depressions and
T wave inversions can suggest coronary ischemia or other cardiac problems.

##### CD (Conduction Disturbance)

  * •

Description: Conduction disturbances occur when the heart’s electrical
impulses are delayed or blocked at certain points in the heart’s conduction
system (such as the AV node or the His-Purkinje system). This can lead to
arrhythmias or irregular heartbeats.

  * •

ECG Characteristics: ECG patterns may include prolonged PR intervals, widened
QRS complexes, or abnormal QRS morphologies depending on where the blockage
occurs. Conditions such as Bundle Branch Blocks or AV blocks are examples.

##### HYP (Hypertrophy)

  * •

Description: Hypertrophy in the cardiac context usually refers to the
thickening of the heart muscle, typically the left ventricle, which can be
caused by chronic high blood pressure or other diseases that increase the
workload of the heart.

  * •

ECG Characteristics: ECG signs of hypertrophy involve larger than the normal
voltage in QRS complexes, changes in the ST segment and T wave and a shift in
the axis of the QRS wave. These changes reflect the increased muscle mass and
altered electrical conduction due to the thickening walls.

Each of these tasks requires the AI model to recognize specific and often
subtle variations in ECG waveforms to accurately classify the type of cardiac
event or condition. The performance of these tasks in AI models, as measured
by AUROC, indicates how well these patterns can be detected and classified
against a backdrop of normal and other abnormal patterns.

##  Appendix B Evaluation Metrics

###  B.1 AUROC and AUPRC Calculations

An explanation of the evaluation metrics used in the study, including the
calculation of Area Under the Receiver Operating Characteristic (AUROC) and
Area Under the Precision-Recall Curve (AUPRC). This section provides the
formulas and interpretation guidelines for these metrics.

  * •

AUROC: The Area Under the Receiver Operating Characteristic (AUROC) curve is a
statistical measure that evaluates the performance of binary classification
models. AUROC plots the True Positive Rate (TPR) versus the False Positive
Rate (FPR) at different threshold settings. It represents the probability of a
classifier ranking a randomly chosen positive instance higher than a randomly
chosen negative one. An AUC of 1 indicates perfect classification, while an
AUC of 0.5 suggests performance equivalent to random guessing. AUROC is useful
for evaluating models on imbalanced datasets as it is not influenced by class
label distribution.

  * •

AUPRC: The Area Under the Precision-Recall Curve (AUPRC) provides a measure to
evaluate binary classification model performance, especially under class
imbalance. Unlike AUROC, which plots TPR against FPR, PRC plots Precision
(true positives to all predicted positives) against Recall (equivalent to
TPR). A higher AUPRC value represents better performance in distinguishing
between classes under imbalanced class distributions.

  * •

F1-Socre: The F1-Score, a harmonic mean of Precision and Recall, offers a
balance between the two metrics for binary classification models. It is
calculated as $2*(Precision*Recall)/(Precision+Recall)$, where Precision is
the ratio of true positives to the sum of true and false positives, and Recall
is the ratio of true positives to the sum of true positives and false
negatives. The F1-Score ranges from 0 to 1, where 1 indicates perfect
Precision and Recall, and 0 indicates the worst. The F1-Score is particularly
useful when the costs of false positives and false negatives are similar, and
when class distribution is imbalanced.ß

###  B.2 Risk Decomposition

Details on the risk decomposition approach (Dubois et al., 2023) used to
evaluate the self-supervised learning methods. It includes descriptions of
approximation error, representation usability error, probe generalization
error, and encoder generalization error, along with how these were measured
and analyzed.

  * •

Encoder generalization error: The encoder generalization error measures the
drop in performance due to pretraining the encoder on finite samples $U$
compared to the population $p_{un}$. Intuitively, it is small if: (i)
$A_{\phi}$ makes the pre-training sample efficient, or (ii) there are many
pre-training examples $|U|$.

  * •

Probe generalization error: The probe generalization error measures the drop
in performance due to training the probe on finite samples $S$ instead of
$p_{sup}$. Intuitively, it is small if: (i) the number of training samples
$|S|$ is large, or (ii) representations ensure that downstream probes are
sample efficient, e.g., by minimizing the margin between same-class examples.

  * •

Representation usability error: The representation usability error measures
errors due to learning representations via an SSL pipeline $A_{\phi}$,
$p_{un}$, rather than supervised learning. Intuitively, it is small if the SSL
algorithm ensures that representations retain information usable by probes
$F$, e.g., linearly separable classes.

  * •

Approximation error: The approximation error measures errors due to the
architecture of the encoder $\phi$ (e.g., ResNet50) and probes $F$ (e.g.,
linear) being too constrained to perform even the supervised task.
Intuitively, it decreases with the capacity of $\phi,F$.

##  Appendix C Experimental Results

This section provides an entire experimental results. The initial stage
involved pre-training 81 distinct foundation models (all combinations of 3
patch sizes, 3 block depth, 3 embedding sizes, and 3 SSL methodologies) on a
large dataset of 1.1 million unlabeled electrocardiograms (ECGs). Following
SSL pre-training, the top layers of each model were replaced with new linear
classifiers, while the underlying base layers retained their pre-trained
weights. After evaluating the models through linear probing, all layers of the
models were fine-tuned in a separate phase.

###  C.1 Pre-training Results

Detailed tables showing the losses of different foundation model architectures
(e.g., patch size, block depth, embedding size) in pre-training, demonstrating
how each parameter affects to pre-train.

Table 1: The result of CL-based foundation models. Case | Windows | Depth | Dims | Total loss | Patient loss | Sample loss | Reconstruction loss
---|---|---|---|---|---|---|---
0 | 60 | 2 | 256 | 775.3041 | 441.5192 | 333.7850 | N/A
1 | 60 | 2 | 512 | 699.6405 | 414.7109 | 284.9303 | N/A
2 | 60 | 2 | 1024 | 663.8318 | 544.7585 | 119.0735 | N/A
3 | 60 | 4 | 256 | 732.1746 | 418.5121 | 313.6624 | N/A
4 | 60 | 4 | 512 | 512.7451 | 291.5426 | 221.2027 | N/A
5 | 60 | 4 | 1024 | 568.3319 | 326.3327 | 241.9992 | N/A
6 | 60 | 8 | 256 | 777.0902 | 439.2020 | 337.8884 | N/A
7 | 60 | 8 | 512 | 703.6331 | 553.8072 | 149.8258 | N/A
8 | 60 | 8 | 1024 | OOM
9 | 125 | 2 | 256 | 524.5357 | 284.3040 | 240.2327 | N/A
10 | 125 | 2 | 512 | 677.8004 | 397.1795 | 280.6209 | N/A
11 | 125 | 2 | 1024 | 671.6511 | 399.2603 | 272.3903 | N/A
12 | 125 | 4 | 256 | 756.5930 | 426.3842 | 330.2094 | N/A
13 | 125 | 4 | 512 | 686.0510 | 511.8934 | 174.1582 | N/A
14 | 125 | 4 | 1024 | 723.0967 | 428.0925 | 295.0038 | N/A
15 | 125 | 8 | 256 | 804.1048 | 449.9247 | 354.1809 | N/A
16 | 125 | 8 | 512 | 720.7618 | 546.3441 | 174.4180 | N/A
17 | 125 | 8 | 1024 | OOM
18 | 250 | 2 | 256 | 729.0459 | 418.5083 | 310.5373 | N/A
19 | 250 | 2 | 512 | 459.2073 | 262.5800 | 196.6271 | N/A
20 | 250 | 2 | 1024 | 727.0769 | 455.8883 | 271.1887 | N/A
21 | 250 | 4 | 256 | 757.3063 | 439.1334 | 318.1726 | N/A
22 | 250 | 4 | 512 | 521.8998 | 297.6395 | 224.2606 | N/A
23 | 250 | 4 | 1024 | 673.0448 | 521.9521 | 151.0925 | N/A
24 | 250 | 8 | 256 | 767.0340 | 432.2636 | 334.7702 | N/A
25 | 250 | 8 | 512 | 724.7963 | 420.9260 | 303.8701 | N/A
26 | 250 | 8 | 1024 | OOM
Table 2: The result of GL-based foundation models. Case | Windows | Depth | Dims | Total loss | Patient loss | Sample loss | Reconstruction loss
---|---|---|---|---|---|---|---
0 | 60 | 2 | 256 | 13.0357 | N/A | N/A | 13.0357
1 | 60 | 2 | 512 | 12.3399 | N/A | N/A | 12.3399
2 | 60 | 2 | 1024 | 11.3272 | N/A | N/A | 11.3272
3 | 60 | 4 | 256 | 12.7696 | N/A | N/A | 12.7696
4 | 60 | 4 | 512 | 11.8257 | N/A | N/A | 11.8257
5 | 60 | 4 | 1024 | 11.6158 | N/A | N/A | 11.6158
6 | 60 | 8 | 256 | 12.9540 | N/A | N/A | 12.9540
7 | 60 | 8 | 512 | 13.8394 | N/A | N/A | 13.8394
8 | 60 | 8 | 1024 | 11.3917 | N/A | N/A | 11.3917
9 | 125 | 2 | 256 | 15.6771 | N/A | N/A | 15.6771
10 | 125 | 2 | 512 | 14.5915 | N/A | N/A | 14.5915
11 | 125 | 2 | 1024 | 13.7325 | N/A | N/A | 13.7325
12 | 125 | 4 | 256 | 15.7419 | N/A | N/A | 15.7419
13 | 125 | 4 | 512 | 14.6308 | N/A | N/A | 14.6308
14 | 125 | 4 | 1024 | 13.6501 | N/A | N/A | 13.6501
15 | 125 | 8 | 256 | 15.5542 | N/A | N/A | 15.5542
16 | 125 | 8 | 512 | 14.8993 | N/A | N/A | 14.8993
17 | 125 | 8 | 1024 | 13.1519 | N/A | N/A | 13.1519
18 | 250 | 2 | 256 | 17.4541 | N/A | N/A | 17.4541
19 | 250 | 2 | 512 | 17.5024 | N/A | N/A | 17.5024
20 | 250 | 2 | 1024 | 17.1858 | N/A | N/A | 17.1858
21 | 250 | 4 | 256 | 17.4811 | N/A | N/A | 17.4811
22 | 250 | 4 | 512 | 17.1457 | N/A | N/A | 17.1457
23 | 250 | 4 | 1024 | 16.9634 | N/A | N/A | 16.9634
24 | 250 | 8 | 256 | 17.5896 | N/A | N/A | 17.5896
25 | 250 | 8 | 512 | 17.3625 | N/A | N/A | 17.3625
26 | 250 | 8 | 1024 | 17.1834 | N/A | N/A | 17.1834
Table 3: The results of HL-based foundation models. Case | Windows | Depth | Dims | Total loss | Patient loss | Sample loss | Reconstruction loss
---|---|---|---|---|---|---|---
0 | 60 | 2 | 256 | 363.8454 | 153.4615 | 191.6084 | 18.7758
1 | 60 | 2 | 512 | 338.7160 | 143.0586 | 176.5117 | 19.1446
2 | 60 | 2 | 1024 | 321.2631 | 135.0332 | 169.3104 | 16.9187
3 | 60 | 4 | 256 | 368.1020 | 157.6347 | 191.3426 | 19.1249
4 | 60 | 4 | 512 | 339.9288 | 143.6038 | 178.7828 | 17.5419
5 | 60 | 4 | 1024 | 327.3949 | 139.2565 | 170.1599 | 17.9787
6 | 60 | 8 | 256 | 373.0694 | 158.4789 | 195.4308 | 19.1591
7 | 60 | 8 | 512 | 344.9142 | 147.0694 | 179.0293 | 18.8156
8 | 60 | 8 | 1024 | OOM
9 | 125 | 2 | 256 | 364.0501 | 149.6521 | 194.6952 | 19.7029
10 | 125 | 2 | 512 | 338.2749 | 138.8596 | 180.2473 | 19.1674
11 | 125 | 2 | 1024 | 323.5201 | 134.1882 | 170.3779 | 18.9548
12 | 125 | 4 | 256 | 369.9288 | 153.7398 | 196.6635 | 19.5256
13 | 125 | 4 | 512 | 342.8939 | 142.0322 | 181.1545 | 19.7073
14 | 125 | 4 | 1024 | 327.3968 | 138.0119 | 169.6444 | 19.7404
15 | 125 | 8 | 256 | 377.1915 | 158.0082 | 199.4338 | 19.7494
16 | 125 | 8 | 512 | 348.1791 | 146.8680 | 181.5616 | 19.7499
17 | 125 | 8 | 1024 | OOM
18 | 250 | 2 | 256 | 362.1963 | 148.8520 | 193.6936 | 19.6514
19 | 250 | 2 | 512 | 338.2755 | 137.9361 | 180.6892 | 19.6513
20 | 250 | 2 | 1024 | 324.4859 | 131.7396 | 173.0957 | 19.6511
21 | 250 | 4 | 256 | 368.3145 | 151.1946 | 197.4695 | 19.6514
22 | 250 | 4 | 512 | 343.0133 | 140.4411 | 182.9213 | 19.6512
23 | 250 | 4 | 1024 | 328.0620 | 138.0024 | 170.4090 | 19.6509
24 | 250 | 8 | 256 | OOM
25 | 250 | 8 | 512 | OOM
26 | 250 | 8 | 1024 | OOM

###  C.2 Linear Probing Results

Detailed tables showing the performance of different foundation model
architectures (e.g., patch size, block depth, embedding size) in linear
probing. This section should include AUROC and AUPRC for various settings,
demonstrating how each parameter affects model performance.

Table 4: The linear probing results of CL-based foundation models. Case | Windows | Depth | Dims | AUROC | AUPRC | Criteria
---|---|---|---|---|---|---
MI | STTC | CD | HYP | MI | STTC | CD | HYP
0 | 60 | 2 | 256 | 0.7653 | 0.7573 | 0.7443 | 0.7811 | 0.5147 | 0.4614 | 0.4073 | 0.2922 | 4.7236
1 | 60 | 2 | 512 | 0.7681 | 0.7928 | 0.7221 | 0.808 | 0.5421 | 0.5265 | 0.4909 | 0.3533 | 5.0038
2 | 60 | 2 | 1024 | 0.6574 | 0.7811 | 0.6621 | 0.7631 | 0.406 | 0.4776 | 0.3591 | 0.3008 | 4.4072
3 | 60 | 4 | 256 | 0.7511 | 0.7928 | 0.7667 | 0.8008 | 0.531 | 0.5268 | 0.5469 | 0.3331 | 5.0492
4 | 60 | 4 | 512 | 0.7427 | 0.7724 | 0.7712 | 0.7819 | 0.5341 | 0.5019 | 0.4991 | 0.309 | 4.9123
5 | 60 | 4 | 1024 | 0.7673 | 0.7984 | 0.7754 | 0.7974 | 0.5418 | 0.533 | 0.4994 | 0.341 | 5.0537
6 | 60 | 8 | 256 | 0.726 | 0.7844 | 0.7253 | 0.7769 | 0.4692 | 0.4961 | 0.4311 | 0.3017 | 4.7107
7 | 60 | 8 | 512 | 0.619 | 0.7829 | 0.6916 | 0.7967 | 0.3647 | 0.4956 | 0.427 | 0.339 | 4.5165
8 | 60 | 8 | 1024 | OOM
9 | 125 | 2 | 256 | 0.6864 | 0.7551 | 0.7279 | 0.729 | 0.3884 | 0.4815 | 0.4208 | 0.2498 | 4.4389
10 | 125 | 2 | 512 | 0.7285 | 0.7854 | 0.7239 | 0.7852 | 0.4609 | 0.4844 | 0.396 | 0.3033 | 4.6676
11 | 125 | 2 | 1024 | 0.7439 | 0.8029 | 0.7688 | 0.8024 | 0.483 | 0.5247 | 0.4732 | 0.336 | 4.9349
12 | 125 | 4 | 256 | 0.6399 | 0.7268 | 0.649 | 0.7313 | 0.3611 | 0.416 | 0.3065 | 0.2475 | 4.0781
13 | 125 | 4 | 512 | 0.6067 | 0.7618 | 0.6672 | 0.7537 | 0.3293 | 0.4993 | 0.3689 | 0.2952 | 4.2821
14 | 125 | 4 | 1024 | 0.7102 | 0.783 | 0.6927 | 0.7926 | 0.4333 | 0.4958 | 0.3641 | 0.3221 | 4.5938
15 | 125 | 8 | 256 | 0.6832 | 0.7414 | 0.7091 | 0.7461 | 0.4236 | 0.4705 | 0.3826 | 0.2563 | 4.4128
16 | 125 | 8 | 512 | 0.6579 | 0.8453 | 0.7323 | 0.8399 | 0.4166 | 0.6127 | 0.4673 | 0.4371 | 5.0091
17 | 125 | 8 | 1024 | OOM
18 | 250 | 2 | 256 | 0.7137 | 0.7851 | 0.7225 | 0.7945 | 0.4471 | 0.5159 | 0.4073 | 0.3621 | 4.7482
19 | 250 | 2 | 512 | 0.775 | 0.8177 | 0.8022 | 0.8077 | 0.5637 | 0.5982 | 0.5872 | 0.411 | 5.3627
20 | 250 | 2 | 1024 | 0.7283 | 0.8045 | 0.7382 | 0.8138 | 0.4571 | 0.5481 | 0.4335 | 0.3632 | 4.8867
21 | 250 | 4 | 256 | 0.6548 | 0.72 | 0.6273 | 0.6929 | 0.3687 | 0.432 | 0.2859 | 0.2178 | 3.9994
22 | 250 | 4 | 512 | 0.7056 | 0.762 | 0.7687 | 0.7846 | 0.4464 | 0.5035 | 0.5168 | 0.3299 | 4.8175
23 | 250 | 4 | 1024 | 0.7179 | 0.8048 | 0.7601 | 0.7971 | 0.489 | 0.5645 | 0.5163 | 0.3213 | 4.971
24 | 250 | 8 | 256 | 0.7849 | 0.8266 | 0.7998 | 0.8467 | 0.5488 | 0.5891 | 0.6019 | 0.465 | 5.4628
25 | 250 | 8 | 512 | 0.7383 | 0.8027 | 0.7844 | 0.8047 | 0.5081 | 0.5307 | 0.5225 | 0.3821 | 5.0735
26 | 250 | 8 | 1024 | OOM
Table 5: The linear probing results of GL-based foundation models. Case | Windows | Depth | Dims | AUROC | AUPRC | Criteria
---|---|---|---|---|---|---
MI | STTC | CD | HYP | MI | STTC | CD | HYP
0 | 60 | 2 | 256 | 0.6653 | 0.8424 | 0.7206 | 0.7545 | 0.46 | 0.6523 | 0.4777 | 0.3484 | 4.9212
1 | 60 | 2 | 512 | 0.6453 | 0.8375 | 0.7222 | 0.7655 | 0.4394 | 0.6571 | 0.4802 | 0.3689 | 4.9161
2 | 60 | 2 | 1024 | 0.6747 | 0.8415 | 0.7356 | 0.794 | 0.4748 | 0.6545 | 0.5087 | 0.4247 | 5.1085
3 | 60 | 4 | 256 | 0.6496 | 0.8342 | 0.7326 | 0.7709 | 0.4558 | 0.6376 | 0.4815 | 0.3877 | 4.9499
4 | 60 | 4 | 512 | 0.6237 | 0.8028 | 0.7197 | 0.7678 | 0.4054 | 0.6039 | 0.4751 | 0.3787 | 4.7771
5 | 60 | 4 | 1024 | 0.6574 | 0.7811 | 0.6621 | 0.7631 | 0.406 | 0.4776 | 0.3591 | 0.3008 | 4.4072
6 | 60 | 8 | 256 | 0.6674 | 0.8331 | 0.7191 | 0.7747 | 0.4685 | 0.644 | 0.469 | 0.4217 | 4.9975
7 | 60 | 8 | 512 | 0.6734 | 0.848 | 0.751 | 0.7731 | 0.4797 | 0.643 | 0.5452 | 0.4061 | 5.1195
8 | 60 | 8 | 1024 | 0.6418 | 0.8267 | 0.7404 | 0.7936 | 0.4336 | 0.6211 | 0.542 | 0.4029 | 5.0021
9 | 125 | 2 | 256 | 0.6372 | 0.8107 | 0.6981 | 0.7499 | 0.4241 | 0.6086 | 0.4531 | 0.3777 | 4.7594
10 | 125 | 2 | 512 | 0.6491 | 0.8423 | 0.7147 | 0.7575 | 0.439 | 0.6539 | 0.463 | 0.3695 | 4.889
11 | 125 | 2 | 1024 | 0.668 | 0.8355 | 0.7244 | 0.7811 | 0.4586 | 0.6401 | 0.4721 | 0.4191 | 4.9989
12 | 125 | 4 | 256 | 0.6526 | 0.8332 | 0.7278 | 0.7774 | 0.4395 | 0.6502 | 0.5002 | 0.41 | 4.9909
13 | 125 | 4 | 512 | 0.6319 | 0.8258 | 0.7036 | 0.7831 | 0.4237 | 0.6275 | 0.4505 | 0.4093 | 4.8554
14 | 125 | 4 | 1024 | 0.6073 | 0.8054 | 0.6583 | 0.7362 | 0.4028 | 0.6002 | 0.4081 | 0.3282 | 4.5465
15 | 125 | 8 | 256 | 0.625 | 0.8148 | 0.6917 | 0.745 | 0.4269 | 0.6121 | 0.4701 | 0.3674 | 4.753
16 | 125 | 8 | 512 | 0.6895 | 0.8458 | 0.752 | 0.7859 | 0.474 | 0.6442 | 0.5346 | 0.4299 | 5.1559
17 | 125 | 8 | 1024 | 0.6627 | 0.8294 | 0.7521 | 0.7745 | 0.4688 | 0.6112 | 0.5514 | 0.3924 | 5.0425
18 | 250 | 2 | 256 | 0.6518 | 0.7961 | 0.7198 | 0.6986 | 0.4272 | 0.5703 | 0.4891 | 0.3168 | 4.6697
19 | 250 | 2 | 512 | 0.6687 | 0.7963 | 0.7254 | 0.7245 | 0.4445 | 0.5769 | 0.5001 | 0.3333 | 4.7697
20 | 250 | 2 | 1024 | 0.6733 | 0.8091 | 0.7306 | 0.7475 | 0.4564 | 0.5872 | 0.5067 | 0.3594 | 4.8702
21 | 250 | 4 | 256 | 0.6109 | 0.7638 | 0.7067 | 0.7015 | 0.351 | 0.5324 | 0.4738 | 0.3006 | 4.4407
22 | 250 | 4 | 512 | 0.6793 | 0.8087 | 0.7329 | 0.7411 | 0.4496 | 0.5843 | 0.5165 | 0.3744 | 4.8868
23 | 250 | 4 | 1024 | 0.6758 | 0.8109 | 0.7104 | 0.751 | 0.4497 | 0.5968 | 0.4664 | 0.3725 | 4.8335
24 | 250 | 8 | 256 | 0.5489 | 0.7022 | 0.6514 | 0.6327 | 0.3123 | 0.4832 | 0.4112 | 0.259 | 4.0009
25 | 250 | 8 | 512 | 0.6544 | 0.7945 | 0.7151 | 0.7283 | 0.4294 | 0.5793 | 0.4957 | 0.3478 | 4.7445
26 | 250 | 8 | 1024 | 0.6412 | 0.77 | 0.6817 | 0.7473 | 0.406 | 0.5298 | 0.4278 | 0.34 | 4.5438
Table 6: The linear probing results of HL-based foundation models. Case | Windows | Depth | Dims | AUROC | AUPRC | Criteria
---|---|---|---|---|---|---
MI | STTC | CD | HYP | MI | STTC | CD | HYP
0 | 60 | 2 | 256 | 0.8314 | 0.7994 | 0.8158 | 0.7804 | 0.658 | 0.5861 | 0.6086 | 0.3635 | 5.4432
1 | 60 | 2 | 512 | 0.8053 | 0.7805 | 0.8251 | 0.7561 | 0.5942 | 0.5281 | 0.6074 | 0.308 | 5.2047
2 | 60 | 2 | 1024 | 0.7626 | 0.7391 | 0.7516 | 0.7305 | 0.5408 | 0.4679 | 0.4719 | 0.2982 | 4.7626
3 | 60 | 4 | 256 | 0.7799 | 0.744 | 0.7848 | 0.7089 | 0.5488 | 0.4986 | 0.5261 | 0.2702 | 4.8613
4 | 60 | 4 | 512 | 0.7869 | 0.802 | 0.8078 | 0.7573 | 0.5702 | 0.5417 | 0.5553 | 0.3002 | 5.1214
5 | 60 | 4 | 1024 | 0.8042 | 0.7765 | 0.7759 | 0.7352 | 0.5843 | 0.4947 | 0.5051 | 0.2522 | 4.9281
6 | 60 | 8 | 256 | 0.8357 | 0.8245 | 0.803 | 0.7585 | 0.6575 | 0.5799 | 0.5782 | 0.3395 | 5.3768
7 | 60 | 8 | 512 | 0.7908 | 0.7225 | 0.7616 | 0.7414 | 0.5368 | 0.439 | 0.4635 | 0.2875 | 4.7431
8 | 60 | 8 | 1024 | OOM
9 | 125 | 2 | 256 | 0.7559 | 0.7531 | 0.7449 | 0.6928 | 0.5257 | 0.4739 | 0.4317 | 0.225 | 4.603
10 | 125 | 2 | 512 | 0.7922 | 0.7768 | 0.8124 | 0.7533 | 0.5701 | 0.5317 | 0.573 | 0.2996 | 5.1091
11 | 125 | 2 | 1024 | 0.7601 | 0.7552 | 0.7939 | 0.7494 | 0.5311 | 0.4868 | 0.5193 | 0.2978 | 4.8936
12 | 125 | 4 | 256 | 0.782 | 0.7529 | 0.7927 | 0.7273 | 0.5664 | 0.4789 | 0.506 | 0.2593 | 4.8655
13 | 125 | 4 | 512 | 0.8318 | 0.8165 | 0.8411 | 0.8135 | 0.616 | 0.5911 | 0.635 | 0.414 | 5.559
14 | 125 | 4 | 1024 | 0.7997 | 0.7587 | 0.7804 | 0.7192 | 0.556 | 0.4941 | 0.5006 | 0.2733 | 4.882
15 | 125 | 8 | 256 | 0.7514 | 0.7209 | 0.7905 | 0.6954 | 0.5087 | 0.4441 | 0.5068 | 0.2722 | 4.69
16 | 125 | 8 | 512 | 0.8088 | 0.7817 | 0.8187 | 0.7457 | 0.6041 | 0.5396 | 0.5897 | 0.3041 | 5.1924
17 | 125 | 8 | 1024 | OOM
18 | 250 | 2 | 256 | 0.7925 | 0.7592 | 0.7964 | 0.7473 | 0.5825 | 0.4862 | 0.5088 | 0.2928 | 4.9657
19 | 250 | 2 | 512 | 0.783 | 0.7173 | 0.7896 | 0.7162 | 0.5506 | 0.4566 | 0.5378 | 0.2515 | 4.8026
20 | 250 | 2 | 1024 | 0.818 | 0.8061 | 0.8155 | 0.7702 | 0.597 | 0.5615 | 0.593 | 0.3473 | 5.3086
21 | 250 | 4 | 256 | 0.7356 | 0.6794 | 0.7391 | 0.6953 | 0.4949 | 0.3881 | 0.4304 | 0.2596 | 4.4224
22 | 250 | 4 | 512 | 0.7531 | 0.7177 | 0.7501 | 0.7093 | 0.5213 | 0.4241 | 0.4643 | 0.236 | 4.5759
23 | 250 | 4 | 1024 | 0.7815 | 0.7504 | 0.8015 | 0.7608 | 0.5264 | 0.459 | 0.5574 | 0.3165 | 4.9535
24 | 250 | 8 | 256 | OOM
25 | 250 | 8 | 512 | OOM
26 | 250 | 8 | 1024 | OOM

###  C.3 Fine-tuning Results

Detailed tables showing the performance of different foundation model
architectures (e.g., patch size, block depth, embedding size) in fine-tuning.
It provides a comprehensive comparison of AUROC and AUPRC across different
foundation model configurations and self-supervised learning methods.

Table 7: The fine-tuning results of CL-based foundation models. Case | Windows | Depth | Dims | AUROC | AUPRC | Criteria
---|---|---|---|---|---|---
MI | STTC | CD | HYP | MI | STTC | CD | HYP
0 | 60 | 2 | 256 | 0.9346 | 0.9381 | 0.9349 | 0.9273 | 0.8580 | 0.8263 | 0.8771 | 0.7015 | 6.9978
1 | 60 | 2 | 512 | 0.9406 | 0.9386 | 0.9389 | 0.9229 | 0.8647 | 0.8176 | 0.8752 | 0.6947 | 6.9932
2 | 60 | 2 | 1024 | 0.9324 | 0.9356 | 0.9320 | 0.9193 | 0.8549 | 0.8240 | 0.8636 | 0.6846 | 6.9464
3 | 60 | 4 | 256 | 0.9383 | 0.9366 | 0.9382 | 0.9285 | 0.8610 | 0.8215 | 0.8786 | 0.7011 | 7.0038
4 | 60 | 4 | 512 | 0.9333 | 0.9395 | 0.9349 | 0.9229 | 0.8456 | 0.8326 | 0.8654 | 0.6898 | 6.9640
5 | 60 | 4 | 1024 | 0.9355 | 0.9347 | 0.9398 | 0.9237 | 0.8589 | 0.8168 | 0.8770 | 0.6749 | 6.9613
6 | 60 | 8 | 256 | 0.9303 | 0.9393 | 0.9367 | 0.9210 | 0.8570 | 0.8304 | 0.8712 | 0.6883 | 6.9742
7 | 60 | 8 | 512 | 0.9330 | 0.9390 | 0.9342 | 0.9230 | 0.8586 | 0.8319 | 0.8709 | 0.6981 | 6.9887
8 | 60 | 8 | 1024 | OOM
9 | 125 | 2 | 256 | 0.9406 | 0.9426 | 0.9388 | 0.9272 | 0.8686 | 0.8439 | 0.8745 | 0.7086 | 7.0448
10 | 125 | 2 | 512 | 0.9345 | 0.9383 | 0.9379 | 0.9225 | 0.8527 | 0.8258 | 0.8813 | 0.6954 | 6.9884
11 | 125 | 2 | 1024 | 0.9397 | 0.9360 | 0.9338 | 0.9198 | 0.8668 | 0.8209 | 0.8729 | 0.6886 | 6.9785
12 | 125 | 4 | 256 | 0.9337 | 0.9354 | 0.9330 | 0.9174 | 0.8574 | 0.8199 | 0.8730 | 0.6675 | 6.9373
13 | 125 | 4 | 512 | 0.9377 | 0.9347 | 0.9352 | 0.9292 | 0.8634 | 0.8291 | 0.8711 | 0.7085 | 7.0089
14 | 125 | 4 | 1024 | 0.9375 | 0.9353 | 0.9393 | 0.9190 | 0.8638 | 0.8282 | 0.8806 | 0.6940 | 6.9977
15 | 125 | 8 | 256 | 0.9407 | 0.9384 | 0.9367 | 0.9254 | 0.8702 | 0.8255 | 0.8764 | 0.6972 | 7.0105
16 | 125 | 8 | 512 | 0.9402 | 0.9377 | 0.9384 | 0.9235 | 0.8651 | 0.8266 | 0.8747 | 0.6888 | 6.9950
17 | 125 | 8 | 1024 | OOM
18 | 250 | 2 | 256 | 0.9393 | 0.9429 | 0.9335 | 0.9276 | 0.8699 | 0.8410 | 0.8694 | 0.6984 | 7.0220
19 | 250 | 2 | 512 | 0.9405 | 0.9394 | 0.9382 | 0.9211 | 0.8687 | 0.8373 | 0.8730 | 0.7014 | 7.0196
20 | 250 | 2 | 1024 | 0.9366 | 0.9396 | 0.9318 | 0.9232 | 0.8625 | 0.8318 | 0.8661 | 0.6960 | 6.9876
21 | 250 | 4 | 256 | 0.9368 | 0.9339 | 0.9319 | 0.9234 | 0.8670 | 0.8158 | 0.8724 | 0.7022 | 6.9834
22 | 250 | 4 | 512 | 0.9409 | 0.9337 | 0.9438 | 0.9230 | 0.8679 | 0.8354 | 0.8825 | 0.7024 | 7.0296
23 | 250 | 4 | 1024 | 0.9349 | 0.9369 | 0.9333 | 0.9249 | 0.8568 | 0.8302 | 0.8693 | 0.7022 | 6.9885
24 | 250 | 8 | 256 | 0.9382 | 0.9397 | 0.9372 | 0.9219 | 0.8658 | 0.8364 | 0.8698 | 0.6929 | 7.0019
25 | 250 | 8 | 512 | 0.9391 | 0.9384 | 0.9361 | 0.9216 | 0.8677 | 0.8300 | 0.8772 | 0.7019 | 7.0120
26 | 250 | 8 | 1024 | OOM
Table 8: The fine-tuning results of GL-based foundation models. Case | Windows | Depth | Dims | AUROC | AUPRC | Criteria
---|---|---|---|---|---|---
MI | STTC | CD | HYP | MI | STTC | CD | HYP
0 | 60 | 2 | 256 | 0.9408 | 0.9335 | 0.9352 | 0.9198 | 0.8644 | 0.8164 | 0.8660 | 0.6910 | 6.9671
1 | 60 | 2 | 512 | 0.9384 | 0.9318 | 0.9352 | 0.9168 | 0.8619 | 0.8126 | 0.8686 | 0.6962 | 6.9615
2 | 60 | 2 | 1024 | 0.9353 | 0.9333 | 0.9368 | 0.9143 | 0.8542 | 0.8134 | 0.8657 | 0.6895 | 6.9425
3 | 60 | 4 | 256 | 0.9411 | 0.9351 | 0.9329 | 0.9128 | 0.8710 | 0.8207 | 0.8633 | 0.6698 | 6.9467
4 | 60 | 4 | 512 | 0.9300 | 0.9377 | 0.9367 | 0.9180 | 0.8464 | 0.8188 | 0.8697 | 0.6839 | 6.9412
5 | 60 | 4 | 1024 | 0.9365 | 0.9370 | 0.9360 | 0.9066 | 0.8629 | 0.8261 | 0.8644 | 0.6790 | 6.9485
6 | 60 | 8 | 256 | 0.9379 | 0.9349 | 0.9396 | 0.9196 | 0.8601 | 0.8246 | 0.8743 | 0.6999 | 6.9909
7 | 60 | 8 | 512 | 0.9390 | 0.9353 | 0.9333 | 0.9151 | 0.8653 | 0.8172 | 0.8653 | 0.6815 | 6.9520
8 | 60 | 8 | 1024 | 0.9364 | 0.9330 | 0.9364 | 0.9150 | 0.8588 | 0.8167 | 0.8675 | 0.6795 | 6.9433
9 | 125 | 2 | 256 | 0.9387 | 0.9385 | 0.9337 | 0.9167 | 0.8626 | 0.8317 | 0.8678 | 0.6801 | 6.9698
10 | 125 | 2 | 512 | 0.9389 | 0.9420 | 0.9362 | 0.9191 | 0.8665 | 0.8350 | 0.8753 | 0.6960 | 7.0090
11 | 125 | 2 | 1024 | 0.9427 | 0.9393 | 0.9400 | 0.9185 | 0.8728 | 0.8295 | 0.8740 | 0.6814 | 6.9982
12 | 125 | 4 | 256 | 0.9404 | 0.9361 | 0.9361 | 0.9137 | 0.8671 | 0.8223 | 0.8695 | 0.6713 | 6.9565
13 | 125 | 4 | 512 | 0.9396 | 0.9360 | 0.9363 | 0.9195 | 0.8649 | 0.8261 | 0.8714 | 0.7042 | 6.9980
14 | 125 | 4 | 1024 | 0.9387 | 0.9370 | 0.9369 | 0.9157 | 0.8683 | 0.8222 | 0.8737 | 0.6852 | 6.9777
15 | 125 | 8 | 256 | 0.9393 | 0.9359 | 0.9376 | 0.9220 | 0.8637 | 0.8176 | 0.8744 | 0.6791 | 6.9696
16 | 125 | 8 | 512 | 0.9330 | 0.9412 | 0.9367 | 0.9178 | 0.8551 | 0.8352 | 0.8725 | 0.6910 | 6.9825
17 | 125 | 8 | 1024 | 0.9410 | 0.9370 | 0.9344 | 0.9095 | 0.8714 | 0.8274 | 0.8646 | 0.6903 | 6.9756
18 | 250 | 2 | 256 | 0.9369 | 0.9390 | 0.9380 | 0.9230 | 0.8643 | 0.8347 | 0.8787 | 0.6941 | 7.0087
19 | 250 | 2 | 512 | 0.9346 | 0.9349 | 0.9348 | 0.9214 | 0.8576 | 0.8178 | 0.8742 | 0.6932 | 6.9685
20 | 250 | 2 | 1024 | 0.9346 | 0.9373 | 0.9389 | 0.9117 | 0.8562 | 0.8299 | 0.8713 | 0.6695 | 6.9494
21 | 250 | 4 | 256 | 0.9379 | 0.9346 | 0.9368 | 0.9183 | 0.8696 | 0.8217 | 0.8733 | 0.6779 | 6.9701
22 | 250 | 4 | 512 | 0.9346 | 0.9338 | 0.9358 | 0.9249 | 0.8542 | 0.8195 | 0.8720 | 0.6931 | 6.9679
23 | 250 | 4 | 1024 | 0.9409 | 0.9370 | 0.9365 | 0.9259 | 0.8703 | 0.8236 | 0.8691 | 0.6957 | 6.9990
24 | 250 | 8 | 256 | 0.9387 | 0.9326 | 0.9393 | 0.9272 | 0.8694 | 0.8182 | 0.8762 | 0.6930 | 6.9946
25 | 250 | 8 | 512 | 0.9377 | 0.9359 | 0.9441 | 0.9125 | 0.8600 | 0.8279 | 0.8854 | 0.6800 | 6.9835
26 | 250 | 8 | 1024 | 0.9409 | 0.9403 | 0.9350 | 0.9097 | 0.8684 | 0.8404 | 0.8699 | 0.6838 | 6.9884
Table 9: The fine-tuning results of HL-based foundation models. Case | Windows | Depth | Dims | AUROC | AUPRC | Criteria
---|---|---|---|---|---|---
MI | STTC | CD | HYP | MI | STTC | CD | HYP
0 | 60 | 2 | 256 | 0.9418 | 0.9413 | 0.9364 | 0.9257 | 0.8732 | 0.8366 | 0.8736 | 0.7018 | 7.0304
1 | 60 | 2 | 512 | 0.9446 | 0.9400 | 0.9400 | 0.9291 | 0.8814 | 0.8435 | 0.8778 | 0.7023 | 7.0587
2 | 60 | 2 | 1024 | 0.9444 | 0.9398 | 0.9360 | 0.9237 | 0.8786 | 0.8354 | 0.8742 | 0.6966 | 7.0287
3 | 60 | 4 | 256 | 0.9444 | 0.9364 | 0.9378 | 0.9274 | 0.8753 | 0.8236 | 0.8775 | 0.6976 | 7.0200
4 | 60 | 4 | 512 | 0.9472 | 0.9376 | 0.9399 | 0.9254 | 0.8810 | 0.8337 | 0.8826 | 0.6951 | 7.0425
5 | 60 | 4 | 1024 | 0.9427 | 0.9398 | 0.9365 | 0.9257 | 0.8746 | 0.8365 | 0.8788 | 0.6975 | 7.0321
6 | 60 | 8 | 256 | 0.9434 | 0.9434 | 0.9428 | 0.9291 | 0.8779 | 0.8389 | 0.8847 | 0.7003 | 7.0605
7 | 60 | 8 | 512 | 0.9453 | 0.9368 | 0.9366 | 0.9339 | 0.8828 | 0.8331 | 0.8766 | 0.7104 | 7.0555
8 | 60 | 8 | 1024 | OOM
9 | 125 | 2 | 256 | 0.9434 | 0.9396 | 0.9415 | 0.9234 | 0.8803 | 0.8339 | 0.8807 | 0.6948 | 7.0376
10 | 125 | 2 | 512 | 0.9406 | 0.9412 | 0.9391 | 0.9213 | 0.8735 | 0.8365 | 0.8796 | 0.6881 | 7.0199
11 | 125 | 2 | 1024 | 0.9452 | 0.9403 | 0.9425 | 0.9245 | 0.8799 | 0.8363 | 0.8835 | 0.6988 | 7.0510
12 | 125 | 4 | 256 | 0.9432 | 0.9370 | 0.9408 | 0.9252 | 0.8747 | 0.8315 | 0.8806 | 0.7027 | 7.0357
13 | 125 | 4 | 512 | 0.9448 | 0.9398 | 0.9449 | 0.9307 | 0.8809 | 0.8339 | 0.8841 | 0.7153 | 7.0744
14 | 125 | 4 | 1024 | 0.9461 | 0.9417 | 0.9390 | 0.9219 | 0.8828 | 0.8400 | 0.8794 | 0.6948 | 7.0457
15 | 125 | 8 | 256 | 0.9434 | 0.9413 | 0.9399 | 0.9163 | 0.8749 | 0.8373 | 0.8820 | 0.6824 | 7.0175
16 | 125 | 8 | 512 | 0.9433 | 0.9378 | 0.9446 | 0.9207 | 0.8782 | 0.8271 | 0.8867 | 0.6772 | 7.0156
17 | 125 | 8 | 1024 | OOM
18 | 250 | 2 | 256 | 0.9442 | 0.9406 | 0.9428 | 0.9253 | 0.8773 | 0.8412 | 0.8837 | 0.6960 | 7.0511
19 | 250 | 2 | 512 | 0.9418 | 0.9390 | 0.9410 | 0.9240 | 0.8737 | 0.8289 | 0.8835 | 0.6962 | 7.0281
20 | 250 | 2 | 1024 | 0.9435 | 0.9366 | 0.9381 | 0.9202 | 0.8788 | 0.8277 | 0.8715 | 0.6964 | 7.0128
21 | 250 | 4 | 256 | 0.9413 | 0.9394 | 0.9432 | 0.9256 | 0.8758 | 0.8399 | 0.8846 | 0.6979 | 7.0477
22 | 250 | 4 | 512 | 0.9437 | 0.9406 | 0.9415 | 0.9215 | 0.8791 | 0.8428 | 0.8823 | 0.6890 | 7.0405
23 | 250 | 4 | 1024 | 0.9452 | 0.9393 | 0.9393 | 0.9203 | 0.8800 | 0.8375 | 0.8773 | 0.6811 | 7.0200
24 | 250 | 8 | 256 | OOM
25 | 250 | 8 | 512 | OOM
26 | 250 | 8 | 1024 | OOM