# The NCI Imaging Data Commons as a platform for reproducible research
in computational pathology

Daniela P. Schacherer<EMAIL_ADDRESS>Markus D.
Herrmann  David A. Clunie  Henning Höfener  William Clifford  William J.R.
Longabaugh  Steve Pieper  Ron Kikinis  Andrey Fedorov  André Homeyer
<EMAIL_ADDRESS>Fraunhofer Institute for Digital Medicine
MEVIS, Bremen, Germany  Department of Pathology, Massachusetts General
Hospital and Harvard Medical School, Boston, Massachusetts, USA  PixelMed
Publishing LLC, Bangor, Pennsylvania, USA  Institute for Systems Biology,
Seattle, Washington, USA  Isomics Inc, Cambridge, Massachusetts, USA
Department of Radiology, Brigham and Women’s Hospital and Harvard Medical
School, Boston, Massachusetts, USA

###### Abstract

This is an outdated preprint. An updated version is published in the journal
Computer Methods and Programs in Biomedicine:
https://doi.org/10.1016/j.cmpb.2023.107839

Background and Objectives: Reproducibility is a major challenge in developing
machine learning (ML)-based solutions in computational pathology (CompPath).
The NCI Imaging Data Commons (IDC) provides >120 cancer image collections
according to the FAIR principles and is designed to be used with cloud ML
services. Here, we explore its potential to facilitate reproducibility in
CompPath research.

Methods: Using the IDC, we implemented two experiments in which a
representative ML-based method for classifying lung tumor tissue was trained
and/or evaluated on different datasets. To assess reproducibility, the
experiments were run multiple times with separate but identically configured
instances of common ML services.

Results: The AUC values of different runs of the same experiment were
generally consistent. However, we observed small variations in AUC values of
up to 0.045, indicating a practical limit to reproducibility.

Conclusions: We conclude that the IDC facilitates approaching the
reproducibility limit of CompPath research (i) by enabling researchers to
reuse exactly the same datasets and (ii) by integrating with cloud ML services
so that experiments can be run in identically configured computing
environments.

Keywords: reproducibility, computational pathology, FAIR, cloud computing,
machine learning, artificial intelligence

##  1 Introduction

Computational pathology (CompPath) is a new discipline that investigates the
use of computational methods for the interpretation of heterogeneous data in
clinical and anatomical pathology to improve health care in pathology
practice. A major focus area of CompPath is the computerized analysis of
digital tissue images [1]. These images show thin sections of surgical
specimens or biopsies that are stained to highlight relevant tissue
structures. To cope with the high level of complexity and variability of
tissue images, virtually all state-of-the-art methods use sophisticated
machine learning (ML) algorithms such as Convolutional Neural Networks (CNN)
[2].

Because CompPath is applicable in a wide variety of use cases, there has been
an explosion of research on ML-based tissue analysis methods [3, 4]. Many
methods are intended to assist pathologists in routine diagnostic tasks such
as the recognition of tissue patterns for disease classification [5, 6, 7, 8,
9]. Beyond that, CompPath methods have also shown promise for deriving novel
biomarkers from tissue patterns that can predict outcome, genetic mutations,
or therapy response [3].

###  1.1 Reproducibility challenges

In recent years, it has become increasingly clear that reproducing the results
of published ML studies is challenging [10, 11, 12, 13]. Reproducibility is
commonly defined as the ability to obtain “consistent results using the same
input data, computational steps, methods, and conditions of analysis” [14].
Difficulties related to reproducibility prevent other researchers from
verifying and reusing published results and are a critical barrier to
translating solutions into clinical practice [15]. In most cases,
reproducibility problems seem to stem not from a lack of scientific rigor, but
from challenges to convey all details and set-up of complex ML methods [15,
16, 12]. In the following, we provide an overview of the main challenges
related to ML reproducibility and the existing approaches to address them.

The first challenge is the specification of the analysis method itself. ML
algorithms have many variables, such as the network architecture,
hyperparameters, and performance metrics [17, 18, 16]. ML workflows usually
consist of multiple processing steps, e.g., data selection, preprocessing,
training, evaluation [18]. Small variations in these implementation details
can have significant effects on performance. To make all these details
transparent, it is crucial to publish the underlying source code [15].
Workflows should be automated as much as possible to avoid errors when
performing steps manually. Jupyter notebooks have emerged as the de facto
standard to implement and communicate ML workflows [19]. By combining software
code, intermediate results and explanatory texts into “computational
narratives” [20] that can be interactively run and validated, notebooks make
it easier for researchers to reproduce and understand the work of others [19].

The second challenge to reproducibility is the specification and setup of the
computing environment. ML workflows require significant computational
resources including, e.g., graphics or tensor processing units (GPUs or TPUs).
In addition, they often have many dependencies on specific software versions.
Minor variations in the computing environment can significantly affect the
results [13]. Setting up a consistent computational environment can be very
expensive and time consuming. This challenge can be partially solved by
embedding ML workflows in virtual machines or software containers like Docker
[21]. Both include all required software dependencies so that ML workflows can
be shared and run without additional installation effort. Cloud ML services,
like Google Vertex AI, Amazon SageMaker, or Microsoft Azure Machine Learning,
provide an even more comprehensive solution. By offering preconfigured
computing environments for ML research in combination with the required high-
performance hardware, such services can further reduce the setup effort and
enable the reproduction of computationally intensive ML workflows even if one
does not own the required hardware. They also typically provide web-based
graphical user interfaces through which Jupyter notebooks can be run and
shared directly in the cloud, making it easy for others to reproduce, verify,
and reuse ML workflows [21].

The third challenge related to ML reproducibility is the specification of data
and its accessibility. The performance of ML methods depends heavily on the
composition of their training, validation and test sets [13, 22]. For current
ML studies, it is rarely possible to reproduce this composition exactly as
studies are commonly based on specific, hand-curated datasets which are only
roughly described rather than explicitly defined [23, 17]. Also, the datasets
are often not made publicly available [15], or the criteria/identifiers used
to select subsets from publicly available datasets are missing. Stakeholders
from academia and industry have defined the Findability, Accessibility,
Interoperability, and Reusability (FAIR) principles [24], a set of
requirements to facilitate discovery and reuse of data. FAIR data provision is
now considered a “must” to make ML studies reproducible and the FAIR
principles are adopted by more and more public data infrastructure initiatives
and scientific journals [25].

Reproducing CompPath studies is particularly challenging. To reveal fine
cellular details, tissue sections are imaged at microscopic resolution,
resulting in gigapixel whole-slide images (WSI) [26]. Due to the complexity
and variability of tissue images [27], it takes many—often thousands—of
example WSI to develop and test reliable ML models. Processing and managing
such large amounts of data requires extensive computing power, storage
resources, and network bandwidth. Reproduction of CompPath studies is further
complicated by the large number of proprietary and incompatible WSI file
formats that often impede data access and make it difficult to combine
heterogeneous data from different studies or sites. The Digital Imaging and
Communications in Medicine (DICOM) standard [28] is an internationally
accepted standard for storage and communication of medical images. It is
universally used in radiology and other medical disciplines, and has great
potential to become the uniform standard for pathology images as well [29].
However, until now, there have been few pathology data collections provided in
DICOM format.

###  1.2 NCI Imaging Data Commons

The National Cancer Institute (NCI) Imaging Data Commons (IDC) is a new cloud-
based repository within the US national Cancer Research Data Commons (CRDC)
[30]. A central goal of the IDC is to improve the reproducibility of data-
driven cancer imaging research. For this purpose, the IDC provides large
public cancer image collections according to the FAIR principles.

Besides pathology images (brightfield and fluorescence) and their metadata,
the IDC includes radiology images (e.g., CT, MR, and PET) together with
associated image analysis results, image annotations, and clinical data
providing context about the images. At the time of writing this article, the
IDC contained 128 data collections with more than 63,000 cases and more than
38,000 WSI from different projects and sites. The collections cover common
tumor types, including carcinomas of the breast, colon, kidney, lung, and
prostate, as well as rarer cancers such as sarcomas or lymphomas. Most of the
WSI collections originate from The Cancer Genome Atlas (TCGA) [31] and
Clinical Proteomic Tumor Analysis Consortium (CPTAC) [32] projects and were
curated by The Cancer Imaging Archive (TCIA) [33]. These collections are
commonly used in the development of CompPath methods [7, 34, 35, 36].

The IDC implements the FAIR principles as follows:

Interoperability: While the original WSIs were provided in proprietary,
vendor-specific formats, the IDC harmonized the data and converted them into
the open, standard DICOM format [29]. DICOM defines data models and services
for storage and communication of medical image data and metadata, as well as
attributes for different real-world entities (e.g., patient, study) and
controlled terminologies for their values. In DICOM, a WSI corresponds to a
“series” of DICOM image objects that represent the digital slide at different
resolutions. Image metadata are stored as attributes directly within the DICOM
objects.

Accessibility: The IDC is implemented on the Google Cloud Platform (GCP),
enabling cohort selection and analysis directly in the cloud. Since IDC data
are provided as part of the Google Public Datasets Program, it can be freely
accessed from cloud or local computing environments. In the IDC, DICOM objects
are stored as individual DICOM files in Google Cloud Storage (GCS) buckets and
can be retrieved using open, free, and universally implementable tools.

Findability: Each DICOM file in the IDC has a persistent universally unique
identifier (UUID) [37]. DICOM files in storage buckets are referenced through
GCS URLs, consisting of the bucket URL and the UUID of the file. Images in the
IDC are described with rich metadata, including patient (e.g., age, sex),
disease (e.g., subtype, stage), study (e.g., therapy, outcome), and imaging-
related data (e.g., specimen handling, scanning). All DICOM and non-DICOM
metadata are indexed in a BigQuery database [38] that can be queried
programmatically using standard Structured Query Language (SQL) statements
(see section “IDC data access”), allowing for an exact and persistent
definition of cohorts for subsequent analysis.

Reusability: All image collections are associated with detailed provenance
information but stripped of patient-identifiable information. Most collections
are released under data usage licenses that allow unrestricted use in research
studies.

###  1.3 Objective

This paper explores how the IDC and cloud ML services can be used in
combination for CompPath studies and how this can facilitate reproducibility.
This paper is also intended as an introduction to how the IDC can be used for
reproducible CompPath research. Therefore, important aspects such as data
access are described in more detail in the Methods section.

##  2 Methods

###  2.1 Overview

We implemented two CompPath experiments using data collections from the IDC
and common ML services (Figure 1). Since the computing environments provided
by cloud ML services are all virtualized, two identically configured instances
may run different host hardware and software (e.g., system software versions,
compiler settings) [13]. To investigate if and how this affects
reproducibility, both experiments were executed multiple times, each in a new
instance of the respective ML service.

The experiments are based on a basic CompPath analysis method that addresses a
use case representative of common CompPath tasks [5, 6, 8, 9, 7]: the
automatic classification of entire WSI of hematoxylin and eosin (H&E)-stained
lung tissue sections into either non-neoplastic (normal), lung adenocarcinoma
(LUAD), or lung squamous cell carcinoma (LSCC/LUSC).

Experiment 1 replays the entire development process of the method, including
model training and validation. Experiment 2 performs inference with a trained
model on independent data. The model trained in Experiment 1 was used as the
basis for Experiment 2. The two experiments were conducted with different
collections in the IDC: TCGA-LUAD/LUSC [39, 40] and CPTAC-LUAD/LSCC [41, 42],
respectively. While both the TCGA and the CPTAC collections cover H&E-stained
lung tissue sections of the three classes considered (Figure 2), they were
created by different clinical institutions using different slide preparation
techniques.

Figure 1: Overview of the workflows of both experiments and their interactions
with the IDC. Figure 2: Example tiles of the three classes considered from the
TCGA and CPTAC datasets. The width of each tile is 256 µm. The black boxes
marked with arrows in the whole slide images on top show the boundaries of the
upper left tiles of the TCGA data set.

###  2.2 Implementation

Both experiments were implemented as standalone Jupyter notebooks that are
available open source [43]. To enable reproducibility, care was taken to make
operations deterministic, e.g., by seeding pseudo-random operations, fixing
initial weights for network training, and by iterating over unordered
container types in a defined order. Utility functionality was designed as
generic classes and functions that can be reused for similar use cases.

As the analysis method itself is not the focus of this paper, we adopted the
algorithmic steps and evaluation design of a lung tumor classification method
described in a widely cited study by Coudray et al. [7]. The method was chosen
because it is representative of common CompPath tasks and easy to understand.
Our implementation processed images at a lower resolution, which is
significantly less computationally expensive.

In our analysis workflow, a WSI was subdivided into non-overlapping
rectangular tiles, each measuring 256 pixels at a resolution of 1 µm/px. Tiles
containing less than 50% tissue, as determined by pixel value statistics, were
discarded. Each tile was assigned class probabilities by performing multi-
class classification using an InceptionV3 CNN [44]. The per-tile results were
finally aggregated to a single classification of the entire slide. The
workflow is visualized in Figure 3 and a detailed description is provided in
the respective notebooks.

Figure 3: Illustration of the CompPath analysis method. Slides were subdivided
into non-overlapping rectangular tiles discarding those with more background
than tissue. Each tile was assigned class probabilities using a neural network
performing multi-class classification. Slide-based class values were
determined by aggregating the tile-based results.

In Experiment 1, the considered slides were divided into training, validation,
and test sets with proportions of 70%, 15%, and 15%, respectively. To keep the
sets independent and avoid overoptimistic performance estimates [45], we
ensured that slides from a given patient were assigned to only one set, which
resulted in 705, 151 and 153 patients per subset. The data collections used
did not contain annotations of tumor regions, but only one reference class
value per WSI. Following the procedure used by Coudray et al., all tiles were
considered to belong to the reference class of their respective slide.
Training was performed using a categorical cross-entropy loss between the true
class labels and the predicted class probabilities, and the RMSProp optimizer
with minimal adjustments to the default hyperparameter values [46]. The epoch
with the highest area under the receiver operating characteristic (ROC) curve
(AUC) on the validation set was chosen for the final model.

###  2.3 IDC data access

For most CompPath studies, one of the first steps is to select relevant slides
using appropriate metadata. In the original data collections, parts of the
metadata were stored in the image files and other parts in separate files of
different formats (e.g., CSV, JSON files). In order to select relevant slides,
the image and metadata first had to be downloaded in their entirety and then
the metadata had to be processed using custom tools. With the IDC, data
selection can be done by filtering a rich set of DICOM attributes with
standard BigQuery SQL statements (Figure 4). The results are tables in which
rows represent DICOM files and columns represent selected metadata attributes.
As this facilitates the accurate and reproducible definition of the data
subsets used in the analysis, these statements are described in more detail
below.

Figure 4: Generic example of a BigQuery SQL statement for compiling slide
metadata. The result set is limited to slide microscopy images, as indicated
by the value “SM” of the DICOM attribute “Modality”, from the collections
“TCGA-LUAD” and “TCGA-LUSC”.

An SQL query for selecting WSI in the IDC generally consists of at least a
SELECT, a FROM and a WHERE clause. The SELECT clause specifies the metadata
attributes to be returned. The IDC provides a wealth of metadata attributes,
including image-, patient-, disease-, and study-level properties. The
attribute “gcs_url” is usually selected because it stores the GCS URL needed
to access the DICOM file. The FROM clause refers to a central table
“dicom_all” which summarizes all DICOM attributes of all DICOM files. This
table can be joined with other tables containing additional project-specific
metadata. Crucial to reproducibility is that all IDC data are versioned: Each
new release of the IDC is represented as a new BigQuery dataset, keeping the
metadata for the previous release and the corresponding DICOM files accessible
even if they are modified in the new release. The version to use is specified
via the dataset specifier in fully qualified table names. All experiments in
this manuscript were conducted against IDC data version 11, i.e., the BigQuery
table “bigquery-public-data.idc_v11.dicom_all”. The WHERE clause defines which
DICOM files are returned by imposing constraints for certain metadata
attributes. To guarantee reproducibility, it is essential to not use SQL
statements that are non-deterministic (e.g., those that utilize ANY_VALUE) and
conclude the statement with an ORDER BY clause, which ensures that results are
returned in a sorted order.

The two experiments considered in this paper also begin with the execution of
a BigQuery SQL statement to select appropriate slides and required metadata
from the IDC. A detailed description of the statements is given in the
respective notebooks. Experiment 1 queries specific H&E-stained tissue slides
from the TCGA-LUAD/LUSC collections, resulting in 2163 slides (591 normal, 819
LUAD, 753 LSCC). Experiment 2 uses a very similar statement to query the
slides from the CPTAC-LUAD/LSCC collections, resulting in 2086 slides (743
normal, 681 LUAD, 662 LSCC).

Once their GCS URLs are known, the selected DICOM files in the IDC can be
accessed efficiently using the open source tool “gsutil” [47] or any other
tool that supports the Simple Storage Service (S3) API. During training in
Experiment 1, image tiles of different WSI had to be accessed repeatedly in
random order. To speed up this process, all considered slides were
preprocessed and the resulting tiles were extracted from the DICOM files and
cached as individual PNG files on disk before training. In contrast, simply
applying the ML method in Experiment 2 required only a single pass over the
tiles of each WSI in sequential order. Therefore, it was feasible to access
the respective DICOM files and iterate over individual tiles at the time they
were needed for the application of the ML method.

###  2.4 Cloud ML services

The two experiments were conducted with two different cloud ML services of the
GCP—Vertex AI and Google Colaboratory. Both services offer virtual machines
(VMs) preconfigured with common ML libraries and a JupyterLab-like interface
that allows editing and running notebooks from the browser. They are both
backed with extensive computing resources including state-of-the-art GPUs or
TPUs. The costs of both services scale with the type and duration of use for
the utilized compute and storage resources. To use any of them with the IDC, a
custom Google Cloud project must be in place for secure authentication and
billing, if applicable.

Since training an ML model is much more computationally intensive than
performing inference, we conducted Experiment 1 with Vertex AI and Experiment
2 with Google Colaboratory. Vertex AI can be attached to efficient disks for
storage of large amounts of input and output data, making it more suitable for
memory-intensive and long-running experiments. Colaboratory, on the other
hand, offers several less expensive payment plans, with limitations in the
provided computing resources and guaranteed continuous usage times.
Colaboratory can even be used completely free of charge, with a significantly
limited guaranteed GPU usage time (12 hours at the time of writing). This
makes Colaboratory better suited for smaller experiments or exploratory
research.

###  2.5 Evaluation

Experiment 1 was performed using a common Vertex AI VM configuration (8 vCPU,
30 GB memory, NVIDIA T4 GPU, Tensorflow Enterprise 2.8 distribution).
Experiment 2 was performed with Colaboratory runtimes (2–8 vCPU, 12–30 GB
memory). When using Google Colaboratory for Experiment 2, we were able to
choose between different GPU types, including NVIDIA T4 and NVIDIA P100 GPUs.
Since it has been suggested that the particular type of GPU can affect results
[48], all runs of Experiment 2 were repeated on both GPUs, respectively. Runs
with NVIDIA T4 were performed with the free version of Colaboratory, while
runs with NVIDIA P100 were performed in combination with a paid GCE
Marketplace VM, which was necessary for guaranteed use of this GPU.

For each run of an experiment, classification accuracy was assessed in terms
of class-specific, one vs. rest AUC values based on the slide-level results.
In addition, 95% confidence intervals of the AUC values were computed by
1000-fold bootstrapping over the slide-level results.

To speed up Experiment 2, only a random subset of 300 of the selected slides
(100 normal, 100 LUAD, 100 LSCC) was considered in the analysis, which was
approximately the size of the test set in Experiment 1.

##  3 Results

The evaluation results of both experiments are summarized in Table 1. It
became apparent that none of the experiments was perfectly reproducible and
there were notable deviations in the results of repeated runs. In Experiment
1, AUC values differed by up to 0.045 between runs. In Experiment 2, there
were also minimal deviations in the AUC values of the different runs, but none
of these were greater than 0.001. These deviations occurred regardless of
whether the runs were executed on the same GPU type or not.

The classification accuracy of the method trained in Experiment 1 appears
satisfactory when evaluated on the TCGA test set and comparable to the results
of a similar study based on the same TCGA collections [7]. When applied to the
CPTAC test set in Experiment 2, the same model performed substantially worse
(Figure 5).

Experiment 1 took an order of magnitude longer to complete (mean runtime of 1
d 18 h ±1 h) than Experiment 2 (mean runtime of 1 h 54 min ±23 min with NVIDIA
T4 and mean runtime of 1 h 28 min ±8 min with NVIDIA P100). The ML service
usage charges for Experiment 1 were approximately US$ 32 per run. With the
free version of Colaboratory, Experiment 2 was performed at no cost, while
runs with the GCE Marketplace VM cost approximately US$ 2 per run.

|  |  | normal | LUAD | LSCC
---|---|---|---|---|---
Experiment | ML Service (GPU) | Run | AUC | CI | AUC | CI | AUC | CI
Experiment 1 | Vertex AI (T4) | 1 | 0.994 | [0.987, 0.999] | 0.942 | [0.914, 0.968] | 0.937 | [0.904, 0.964]
|  | 2 | 0.981 | [0.964, 0.994] | 0.898 | [0.860, 0.937] | 0.914 | [0.875, 0.946]
|  | 3 | 0.992 | [0.983, 0.999] | 0.939 | [0.909, 0.964] | 0.918 | [0.881, 0.949]
|  | 4 | 0.994 | [0.986, 0.999] | 0.928 | [0.895, 0.958] | 0.910 | [0.865, 0.947]
|  | 5 | 0.989 | [0.979, 0.997] | 0.930 | [0.895, 0.959] | 0.892 | [0.838, 0.934]
Experiment 2 | Colaboratory (T4) | 1 | 0.811 | [0.746, 0.871] | 0.698 | [0.633, 0.759] | 0.850 | [0.802, 0.899]
|  | 2 | 0.811 | [0.746, 0.871] | 0.698 | [0.633, 0.759] | 0.850 | [0.802, 0.899]
|  | 3 | 0.811 | [0.747, 0.870] | 0.698 | [0.636, 0.758] | 0.851 | [0.800, 0.896]
|  | 4 | 0.811 | [0.748, 0.869] | 0.698 | [0.632, 0.758] | 0.851 | [0.802, 0.896]
|  | 5 | 0.811 | [0.748, 0.872] | 0.698 | [0.627, 0.759] | 0.851 | [0.799, 0.896]
| Colaboratory (P100) | 1 | 0.811 | [0.746, 0.874] | 0.698 | [0.630, 0.758] | 0.851 | [0.802, 0.896]
|  | 2 | 0.811 | [0.747, 0.873] | 0.698 | [0.627, 0.760] | 0.850 | [0.802, 0.897]
|  | 3 | 0.811 | [0.747, 0.873] | 0.698 | [0.627, 0.760] | 0.850 | [0.802, 0.897]
|  | 4 | 0.811 | [0.747, 0.873] | 0.698 | [0.627, 0.760] | 0.850 | [0.802, 0.897]
|  | 5 | 0.811 | [0.747, 0.873] | 0.698 | [0.627, 0.760] | 0.850 | [0.802, 0.897]
Table 1: Class-specific, slide-based AUC values and 95% confidence intervals
(CI) obtained through multiple runs of both experiments. Figure 5: One-vs-rest
ROC curves for the multi-class classification as obtained in (a) the first run
of Experiment 1 using Vertex AI and (b) the first run of Experiment 2 using
Colaboratory (T4).

##  4 Discussion

The aim of this study was to investigate how CompPath studies can be made
reproducible through the use of cloud-based computing environments and the IDC
as the source of input data. Although the same code was run with the same data
using the same ML services and care was taken that operations were
deterministic (see section “Implementation”), we observed small deviations in
the results of repeated runs. We did not investigate whether the deviations
originate from differences in the hardware and software used by the hosts of
the virtual computing environments, or whether they are due to randomness
resulting from parallel processing [13]. The greater variability in the
results of Experiment 1 can possibly be explained by its higher computational
complexity. Although the observed deviations appear negligible for many
applications, they represent a practical upper limit for reproducibility. Such
issues are likely to occur in any computing environment. As outlined below, we
argue that the IDC can help to approach this reproducibility limit.

We chose Jupyter notebooks and cloud ML services to address the first two
reproducibility challenges mentioned in the Introduction: specifying the
analysis method and setting up the computing environment. With the IDC, we
were able to tackle the third reproducibility challenge with respect to the
special requirements of CompPath: specifying and accessing the data.

By providing imaging data collections according to the FAIR principles, the
IDC facilitates precise definition of the datasets used in the analysis and
ensures that the exact same data can be reused in follow-up studies. Since
metadata on acquisition and processing can be included as DICOM attributes
alongside the pixel data, the risk of data confusion can be greatly reduced.
The IDC also facilitated the use of cloud ML services because it makes
terabytes of WSI data efficiently accessible by on-demand compute resources.
We consider our experiments to be representative of common CompPath
applications. Therefore, the IDC should be similarly usable for other CompPath
studies.

The results of Experiment 2 also reveal the transferability of the model
trained in Experiment 1 to independent data. Although the majority of slides
were correctly classified, AUC values were significantly lower, indicating
that the model is only transferable to a limited extent and additional
training is needed. Since all IDC data collections (both the image pixel data
and the associated metadata) are harmonized into a standardized DICOM
representation, testing transferability to a different dataset required only
minor adjustments to our BigQuery SQL statement. In the same way, the IDC
makes it straightforward to use multiple datasets in one experiment or to
transfer an experimental design to other applications.

###  4.1 Limitations

Using cloud ML services comes with certain trade-offs. Conducting
computationally intensive experiments requires setting up a payment account
and paying a fee based on the type and duration of the computing resources
used. Furthermore, although the ML services are widely used and likely to be
supported for at least the next few years, there is no guarantee that they
will be supported in the long term and support the specific configuration of
the computing environment used (e.g., software version, libraries). Those who
do not want to make these compromises can also access IDC data collections
without using ML services, both in the cloud and on-premises. Even if this
means losing the previously mentioned advantages with regard to the first two
reproducibility challenges, the IDC can still help to specify the data used in
a clear and reproducible manner.

Independent of the implementation, a major obstacle to the reproducibility of
CompPath methods remains their high computational cost. A full training run
often takes several days, making reproduction by other scientists tedious.
Performing model inference is generally faster and less resource intensive
when compared to model training. Therefore, Experiment 2 runs well even with
the free version of Google Colaboratory, enabling others to reproduce it
without spending money. The notebook also provides a demo mode, which
completes in a few minutes, so anyone can easily experiment with applying the
inference workflow to arbitrary images from IDC.

At the moment, the IDC exclusively hosts public data collections. New data
must undergo rigorous curation to de-identify (done by TCIA or data submitter)
and harmonize images into standard representation (done by IDC), which can
require a significant effort. Therefore, only data collections that are of
general relevance and high quality are included in the IDC. As a result, the
data in the IDC were usually acquired for other purposes than a particular
CompPath application and cannot be guaranteed to be representative and free of
bias [49]. Compiling truly representative CompPath datasets is very
challenging [45]. Nevertheless, the data collections in the IDC can provide a
reasonable basis for exploring and prototyping CompPath methods.

###  4.2 Outlook

The IDC is under continuous development and its technical basis is constantly
being refined, e.g., to support new data types or to facilitate data selection
and access. Currently, DICOM files in the IDC can only be accessed as a whole
from their respective storage buckets. This introduces unnecessary overhead
when only certain regions of a slide need to be processed, and it may make it
necessary to temporarily cache slides to efficiently access multiple image
regions (see section “IDC data access”). Future work should therefore aim to
provide efficient random access to individual regions within a WSI. For
maximum portability, such access should ideally be possible via standard DICOM
network protocols such as DICOMweb [50, 29].

The IDC is continuously being expanded to support even more diverse CompPath
applications. For instance, images collected by the Human Tumor Atlas Network
(HTAN) that provide rich, multispectral information on subcellular processes
[51] have recently been added. The IDC is integrated with other components of
the CRDC, such as the Genomic Data Commons [52] or the Proteomic Data Commons
[53]. This opens up many more potential CompPath applications involving tissue
images and different types of molecular cancer data [54].

###  4.3 Conclusion

We demonstrated how the IDC can facilitate the reproducibility of CompPath
studies. Implementing future studies in a similar way can help other
researchers and peer reviewers to understand, validate and advance the
analysis approach.

##  5 Author Contributions

DPS and AH conceived and carried out the study. AH and AF supervised the
project. AF, MDH, DAC, HH, WC, WJRL, SP and RK supported the study in
different ways, e.g., by providing data, supporting set-up of the computing
infrastructure, interpretation of the results and giving general advice. AH
and DPS drafted the manuscript. All authors critically revised the manuscript
and expressed their consent to the final version.

##  6 Declaration of Competing Interest

The authors declare no conflicts of interest.

##  7 Acknowledgements

The authors thank Lars Ole Schwen for advice on deterministic implementations
of machine learning algorithms and Tim-Rasmus Kiehl for advice on tissue
morphology.

The results published here are in whole or part based upon data generated by
the TCGA Research Network and the National Cancer Institute Clinical Proteomic
Tumor Analysis Consortium (CPTAC).

This project has been funded in whole or in part with Federal funds from the
National Cancer Institute, National Institutes of Health, under Task Order No.
HHSN26110071 under Contract No. HHSN261201500003l.

## References

  * [1] D. N. Louis, M. Feldman, A. B. Carter, A. S. Dighe, J. D. Pfeifer, L. Bry, J. S. Almeida, J. Saltz, J. Braun, J. E. Tomaszewski, J. R. Gilbertson, J. H. Sinard, G. K. Gerber, S. J. Galli, J. A. Golden, M. J. Becich, Computational pathology: A path ahead, Archives of Pathology & Laboratory Medicine 140 (1) (2015) 41–50.  doi:10.5858/arpa.2015-0093-sa.
  * [2] M. K. K. Niazi, A. V. Parwani, M. N. Gurcan, Digital pathology and artificial intelligence, The Lancet Oncology 20 (5) (2019) e253–e261.  doi:10.1016/s1470-2045(19)30154-8.
  * [3] A. Echle, N. T. Rindtorff, T. J. Brinker, T. Luedde, A. T. Pearson, J. N. Kather, Deep learning in cancer pathology: a new generation of clinical biomarkers, British Journal of Cancer 124 (4) (2020) 686–696.  doi:10.1038/s41416-020-01122-x.
  * [4] M. Cui, D. Y. Zhang, Artificial intelligence and computational pathology, Laboratory Investigation 101 (4) (2021) 412–422.  doi:10.1038/s41374-020-00514-0.
  * [5] A. Cruz-Roa, H. Gilmore, A. Basavanhally, M. Feldman, S. Ganesan, N. N. Shih, J. Tomaszewski, F. A. González, A. Madabhushi, Accurate and reproducible invasive breast cancer detection in whole-slide images: A deep learning approach for quantifying tumor extent, Scientific Reports 7 (1) (Apr. 2017).  doi:10.1038/srep46450.
  * [6] G. Campanella, M. G. Hanna, L. Geneslaw, A. Miraflor, V. W. K. Silva, K. J. Busam, E. Brogi, V. E. Reuter, D. S. Klimstra, T. J. Fuchs, Clinical-grade computational pathology using weakly supervised deep learning on whole slide images, Nature Medicine 25 (8) (2019) 1301–1309.  doi:10.1038/s41591-019-0508-1.
  * [7] N. Coudray, P. S. Ocampo, T. Sakellaropoulos, N. Narula, M. Snuderl, D. Fenyö, A. L. Moreira, N. Razavian, A. Tsirigos, Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning, Nature Medicine 24 (10) (2018) 1559–1567.  doi:10.1038/s41591-018-0177-5.
  * [8] X. Wang, H. Chen, C. Gan, H. Lin, Q. Dou, E. Tsougenis, Q. Huang, M. Cai, P.-A. Heng, Weakly supervised deep learning for whole slide lung cancer image analysis, IEEE Transactions on Cybernetics 50 (9) (2020) 3950–3962.  doi:10.1109/tcyb.2019.2935141.
  * [9] O. Iizuka, F. Kanavati, K. Kato, M. Rambeau, K. Arihiro, M. Tsuneki, Deep learning models for histopathological classification of gastric and colonic epithelial tumours, Scientific Reports 10 (1) (Jan. 2020).  doi:10.1038/s41598-020-58467-9.
  * [10] C. Fell, M. Mohammadi, D. Morrison, O. Arandjelovic, P. Caie, D. Harris-Birtill, Reproducibility of deep learning in digital pathology whole slide image analysis, PLOS Digital Health 1 (12) (2022) e0000145.  doi:10.1371/journal.pdig.0000145.
URL https://doi.org/10.1371/journal.pdig.0000145

  * [11] M. Hutson, Artificial intelligence faces reproducibility crisis, Science 359 (6377) (2018) 725–726.  doi:10.1126/science.359.6377.725.
  * [12] E. Raff, A step toward quantifying independently reproducible machine learning research, in: H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, R. Garnett (Eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, 2019, pp. 5486–5496.
URL
https://proceedings.neurips.cc/paper/2019/hash/c429429bf1f2af051f2021dc92a8ebea-
Abstract.html

  * [13] O. E. Gundersen, S. Shamsaliei, R. J. Isdahl, Do machine learning platforms provide out-of-the-box reproducibility?, Future Generation Computer Systems 126 (2022) 34–47.  doi:10.1016/j.future.2021.06.014.
  * [14] National Academies of Sciences, Engineering, and Medicine, Reproducibility and Replicability in Science, The National Academies Press, Washington, DC, 2019\.  doi:10.17226/25303.
URL https://nap.nationalacademies.org/catalog/25303/reproducibility-and-
replicability-in-science

  * [15] B. Haibe-Kains, G. A. Adam, A. Hosny, F. Khodakarami, T. Shraddha, R. Kusko, S.-A. Sansone, W. Tong, R. D. Wolfinger, C. E. Mason, W. Jones, J. Dopazo, C. Furlanello, L. Waldron, B. Wang, C. McIntosh, A. Goldenberg, A. Kundaje, C. S. Greene, T. Broderick, M. M. Hoffman, J. T. Leek, K. Korthauer, W. Huber, A. Brazma, J. Pineau, R. Tibshirani, T. Hastie, J. P. A. Ioannidis, J. Quackenbush, H. J. W. L. A. and, Transparency and reproducibility in artificial intelligence, Nature 586 (7829) (2020) E14–E16.  doi:10.1038/s41586-020-2766-y.
  * [16] J. Pineau, P. Vincent-Lamarre, K. Sinha, V. Lariviere, A. Beygelzimer, F. d’Alche Buc, E. Fox, H. Larochelle, Improving reproducibility in machine learning research(a report from the neurips 2019 reproducibility program), Journal of Machine Learning Research 22 (164) (2021) 1–20.
URL http://jmlr.org/papers/v22/20-303.html

  * [17] M. Hartley, T. S. Olsson, dtoolAI: Reproducibility for deep learning, Patterns 1 (5) (2020) 100073.  doi:10.1016/j.patter.2020.100073.
  * [18] F. Renard, S. Guedria, N. D. Palma, N. Vuillerme, Variability and reproducibility in deep learning for medical image segmentation, Scientific Reports 10 (1) (Aug. 2020).  doi:10.1038/s41598-020-69920-0.
  * [19] J. M. Perkel, Why jupyter is data scientists’ computational notebook of choice, Nature 563 (7729) (2018) 145–146.  doi:10.1038/d41586-018-07196-1.
  * [20] A. Rule, A. Birmingham, C. Zuniga, I. Altintas, S.-C. Huang, R. Knight, N. Moshiri, M. H. Nguyen, S. B. Rosenthal, F. Pérez, P. W. Rose, Ten simple rules for writing and sharing computational analyses in jupyter notebooks, PLOS Computational Biology 15 (7) (2019) e1007007.  doi:10.1371/journal.pcbi.1007007.
  * [21] J. M. Perkel, Make code accessible with these cloud services, Nature 575 (7781) (2019) 247–248.  doi:10.1038/d41586-019-03366-x.
  * [22] L. Maier-Hein, M. Eisenmann, A. Reinke, S. Onogur, M. Stankovic, P. Scholz, T. Arbel, H. Bogunovic, A. P. Bradley, A. Carass, C. Feldmann, A. F. Frangi, P. M. Full, B. van Ginneken, A. Hanbury, K. Honauer, M. Kozubek, B. A. Landman, K. März, O. Maier, K. Maier-Hein, B. H. Menze, H. Müller, P. F. Neher, W. Niessen, N. Rajpoot, G. C. Sharp, K. Sirinukunwattana, S. Speidel, C. Stock, D. Stoyanov, A. A. Taha, F. van der Sommen, C.-W. Wang, M.-A. Weber, G. Zheng, P. Jannin, A. Kopp-Schneider, Why rankings of biomedical image analysis competitions should be interpreted with care, Nature Communications 9 (1) (Dec. 2018).  doi:10.1038/s41467-018-07619-7.
  * [23] O. E. Gundersen, S. Kjensmo, State of the art: Reproducibility in artificial intelligence, Proceedings of the AAAI Conference on Artificial Intelligence 32 (1) (Apr. 2018).  doi:10.1609/aaai.v32i1.11503.
  * [24] M. D. Wilkinson, M. Dumontier, I. J. Aalbersberg, G. Appleton, M. Axton, A. Baak, N. Blomberg, J.-W. Boiten, L. B. da Silva Santos, P. E. Bourne, J. Bouwman, A. J. Brookes, T. Clark, M. Crosas, I. Dillo, O. Dumon, S. Edmunds, C. T. Evelo, R. Finkers, A. Gonzalez-Beltran, A. J. Gray, P. Groth, C. Goble, J. S. Grethe, J. Heringa, P. A. ’t Hoen, R. Hooft, T. Kuhn, R. Kok, J. Kok, S. J. Lusher, M. E. Martone, A. Mons, A. L. Packer, B. Persson, P. Rocca-Serra, M. Roos, R. van Schaik, S.-A. Sansone, E. Schultes, T. Sengstag, T. Slater, G. Strawn, M. A. Swertz, M. Thompson, J. van der Lei, E. van Mulligen, J. Velterop, A. Waagmeester, P. Wittenburg, K. Wolstencroft, J. Zhao, B. Mons, The FAIR guiding principles for scientific data management and stewardship, Scientific Data 3 (1) (Mar. 2016).  doi:10.1038/sdata.2016.18.
  * [25] M. Scheffler, M. Aeschlimann, M. Albrecht, T. Bereau, H.-J. Bungartz, C. Felser, M. Greiner, A. Groß, C. T. Koch, K. Kremer, W. E. Nagel, M. Scheidgen, C. Wöll, C. Draxl, FAIR data enabling new horizons for materials research, Nature 604 (7907) (2022) 635–642.  doi:10.1038/s41586-022-04501-x.
  * [26] A. Patel, U. G. Balis, J. Cheng, Z. Li, G. Lujan, D. S. McClintock, L. Pantanowitz, A. Parwani, Contemporary whole slide imaging devices and their applications within the modern pathology department: A selected hardware review, Journal of Pathology Informatics 12 (1) (2021) 50.  doi:10.4103/jpi.jpi_66_21.
  * [27] M. T. McCann, J. A. Ozolek, C. A. Castro, B. Parvin, J. Kovacevic, Automated histology analysis: Opportunities for signal processing, IEEE Signal Processing Magazine 32 (1) (2015) 78–87.  doi:10.1109/msp.2014.2346443.
  * [28] W. D. Bidgood, S. C. Horii, F. W. Prior, D. E. V. Syckle, Understanding and using DICOM, the data interchange standard for biomedical imaging, Journal of the American Medical Informatics Association 4 (3) (1997) 199–212.  doi:10.1136/jamia.1997.0040199.
  * [29] M. D. Herrmann, D. A. Clunie, A. Fedorov, S. W. Doyle, S. Pieper, V. Klepeis, L. P. Le, G. L. Mutter, D. S. Milstone, T. J. Schultz, R. Kikinis, G. K. Kotecha, D. H. Hwang, K. P. Andriole, A. J. lafrate, J. A. Brink, G. W. Boland, K. J. Dreyer, M. Michalski, J. A. Golden, D. N. Louis, J. K. Lennerz, Implementing the DICOM standard for digital pathology, Journal of Pathology Informatics 9 (1) (2018) 37.  doi:10.4103/jpi.jpi_42_18.
  * [30] A. Fedorov, W. J. Longabaugh, D. Pot, D. A. Clunie, S. Pieper, H. J. Aerts, A. Homeyer, R. Lewis, A. Akbarzadeh, D. Bontempi, W. Clifford, M. D. Herrmann, H. Höfener, I. Octaviano, C. Osborne, S. Paquette, J. Petts, D. Punzo, M. Reyes, D. P. Schacherer, M. Tian, G. White, E. Ziegler, I. Shmulevich, T. Pihl, U. Wagner, K. Farahani, R. Kikinis, NCI imaging data commons, Cancer Research 81 (16) (2021) 4188–4193.  doi:10.1158/0008-5472.can-21-0950.
  * [31] The Cancer Genome Atlas Program [cited 2023-01-30].
URL https://www.cancer.gov/tcga

  * [32] The National Cancer Institute’s Clinical Proteomic Tumor Analysis Consortium [cited 2023-01-30].
URL https://proteomics.cancer.gov/programs/cptac

  * [33] K. Clark, B. Vendt, K. Smith, J. Freymann, J. Kirby, P. Koppel, S. Moore, S. Phillips, D. Maffitt, M. Pringle, L. Tarbox, F. Prior, The cancer imaging archive (TCIA): Maintaining and operating a public information repository, Journal of Digital Imaging 26 (6) (2013) 1045–1057.  doi:10.1007/s10278-013-9622-7.
  * [34] J. Saltz, R. Gupta, L. Hou, T. Kurc, P. Singh, V. Nguyen, D. Samaras, K. R. Shroyer, T. Zhao, R. Batiste, J. V. Arnam, Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images, Cell Reports 23 (1) (2018) 181–193.e7.  doi:10.1016/j.celrep.2018.03.086.
  * [35] P. Khosravi, E. Kazemi, M. Imielinski, O. Elemento, I. Hajirasouliha, Deep convolutional neural networks enable discrimination of heterogeneous digital pathology images, EBioMedicine 27 (2018) 317–328.  doi:10.1016/j.ebiom.2017.12.026.
  * [36] J. Noorbakhsh, S. Farahmand, A. F. pour, S. Namburi, D. Caruana, D. Rimm, M. Soltanieh-ha, K. Zarringhalam, J. H. Chuang, Deep learning-based cross-classifications reveal conserved spatial behaviors within tumor histological images, Nature Communications 11 (1) (Dec. 2020).  doi:10.1038/s41467-020-20030-5.
  * [37] P. Leach, M. Mealling, R. Salz, A universally unique IDentifier (UUID) URN namespace, Tech. rep. (Jul. 2005).  doi:10.17487/rfc4122.
  * [38] Clinical Proteomic Tumor Analysis Consortium [cited 2023-01-30].
URL https://cloud.google.com/healthcare/docs/how-tos/dicom-bigquery-schema

  * [39] B. Albertina, M. Watson, C. Holback, R. Jarosz, S. Kirk, Y. Lee, K. Rieger-Christ, J. Lemmerman, The Cancer Genome Atlas Lung Adenocarcinoma Collection (TCGA-LUAD) (2016).  doi:10.7937/K9/TCIA.2016.JGNIHEP5.
URL https://wiki.cancerimagingarchive.net/x/wgBp

  * [40] S. Kirk, Y. Lee, P. Kumar, J. Filippini, B. Albertina, M. Watson, K. Rieger-Christ, J. Lemmerman, The Cancer Genome Atlas Lung Squamous Cell Carcinoma Collection (TCGA-LUSC) (2016).  doi:10.7937/K9/TCIA.2016.TYGKKFMQ.
URL https://wiki.cancerimagingarchive.net/x/pAD1

  * [41] National Cancer Institute Clinical Proteomic Tumor Analysis Consortium (CPTAC), The Clinical Proteomic Tumor Analysis Consortium Lung Adenocarcinoma Collection (CPTAC-LUAD) (2018).  doi:10.7937/K9/TCIA.2018.PAT12TBS.
URL https://wiki.cancerimagingarchive.net/x/XQIGAg

  * [42] National Cancer Institute Clinical Proteomic Tumor Analysis Consortium (CPTAC), The Clinical Proteomic Tumor Analysis Consortium Lung Squamous Cell Carcinoma Collection (CPTAC-LSCC) (2018).  doi:10.7937/K9/TCIA.2018.6EMUB5L2.
URL https://wiki.cancerimagingarchive.net/x/WAIGAg

  * [43] Classification of lung tumor slide images with the NCI Imaging Data Commons [cited 2023-03-15].
URL https://github.com/ImagingDataCommons/idc-comppath-reproducibility.git

  * [44] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, Z. Wojna, Rethinking the inception architecture for computer vision (2015).  doi:10.48550/ARXIV.1512.00567.
  * [45] A. Homeyer, C. Geißler, L. O. Schwen, F. Zakrzewski, T. Evans, K. Strohmenger, M. Westphal, R. D. Bülow, M. Kargl, A. Karjauv, I. Munné-Bertran, C. O. Retzlaff, A. Romero-López, T. Sołtysiński, M. Plass, R. Carvalho, P. Steinbach, Y.-C. Lan, N. Bouteldja, D. Haber, M. Rojas-Carulla, A. V. Sadr, M. Kraft, D. Krüger, R. Fick, T. Lang, P. Boor, H. Müller, P. Hufnagl, N. Zerbe, Recommendations on compiling test datasets for evaluating artificial intelligence solutions in pathology, Modern Pathology 35 (12) (2022) 1759–1769.  doi:10.1038/s41379-022-01147-y.
  * [46] RMSprop class [cited 2023-01-30].
URL https://keras.io/api/optimizers/rmsprop

  * [47] gsutil tool [cited 2023-01-30].
URL https://cloud.google.com/storage/docs/gsutil

  * [48] P. Nagarajan, G. Warnell, P. Stone, Deterministic implementations for reproducibility in deep reinforcement learning (2018).  doi:10.48550/ARXIV.1809.05676.
  * [49] G. Varoquaux, V. Cheplygina, Machine learning for medical imaging: methodological failures and recommendations for the future, npj Digital Medicine 5 (1) (Apr. 2022).  doi:10.1038/s41746-022-00592-y.
  * [50] DICOMweb [cited 2023-01-30].
URL https://www.dicomstandard.org/using/dicomweb

  * [51] O. Rozenblatt-Rosen, A. Regev, P. Oberdoerffer, T. Nawy, A. Hupalowska, J. E. Rood, O. Ashenberg, E. Cerami, R. J. Coffey, E. Demir, L. Ding, E. D. Esplin, J. M. Ford, J. Goecks, S. Ghosh, J. W. Gray, J. Guinney, S. E. Hanlon, S. K. Hughes, E. S. Hwang, C. A. Iacobuzio-Donahue, J. Jané-Valbuena, B. E. Johnson, K. S. Lau, T. Lively, S. A. Mazzilli, D. Pe’er, S. Santagata, A. K. Shalek, D. Schapiro, M. P. Snyder, P. K. Sorger, A. E. Spira, S. Srivastava, K. Tan, R. B. West, E. H. Williams, D. Aberle, S. I. Achilefu, F. O. Ademuyiwa, A. C. Adey, R. L. Aft, R. Agarwal, R. A. Aguilar, F. Alikarami, V. Allaj, C. Amos, R. A. Anders, M. R. Angelo, K. Anton, O. Ashenberg, J. C. Aster, O. Babur, A. Bahmani, A. Balsubramani, D. Barrett, J. Beane, D. E. Bender, K. Bernt, L. Berry, C. B. Betts, J. Bletz, K. Blise, A. Boire, G. Boland, A. Borowsky, K. Bosse, M. Bott, E. Boyden, J. Brooks, R. Bueno, E. A. Burlingame, Q. Cai, J. Campbell, W. Caravan, E. Cerami, H. Chaib, J. M. Chan, Y. H. Chang, D. Chatterjee, O. Chaudhary, A. A. Chen, B. Chen, C. Chen, C. hui Chen, F. Chen, Y.-A. Chen, M. G. Chheda, K. Chin, R. Chiu, S.-K. Chu, R. Chuaqui, J. Chun, L. Cisneros, R. J. Coffey, G. A. Colditz, K. Cole, N. Collins, K. Contrepois, L. M. Coussens, A. L. Creason, D. Crichton, C. Curtis, T. Davidsen, S. R. Davies, I. de Bruijn, L. Dellostritto, A. D. Marzo, E. Demir, D. G. DeNardo, D. Diep, L. Ding, S. Diskin, X. Doan, J. Drewes, S. Dubinett, M. Dyer, J. Egger, J. Eng, B. Engelhardt, G. Erwin, E. D. Esplin, L. Esserman, A. Felmeister, H. S. Feiler, R. C. Fields, S. Fisher, K. Flaherty, J. Flournoy, J. M. Ford, A. Fortunato, A. Frangieh, J. L. Frye, R. S. Fulton, D. Galipeau, S. Gan, J. Gao, L. Gao, P. Gao, V. R. Gao, T. Geiger, A. George, G. Getz, S. Ghosh, M. Giannakis, D. L. Gibbs, W. E. Gillanders, J. Goecks, S. P. Goedegebuure, A. Gould, K. Gowers, J. W. Gray, W. Greenleaf, J. Gresham, J. L. Guerriero, T. K. Guha, A. R. Guimaraes, J. Guinney, D. Gutman, N. Hacohen, S. Hanlon, C. R. Hansen, O. Harismendy, K. A. Harris, A. Hata, A. Hayashi, C. Heiser, K. Helvie, J. M. Herndon, G. Hirst, F. Hodi, T. Hollmann, A. Horning, J. J. Hsieh, S. Hughes, W. J. Huh, S. Hunger, S. E. Hwang, C. A. Iacobuzio-Donahue, H. Ijaz, B. Izar, C. A. Jacobson, S. Janes, J. Jané-Valbuena, R. G. Jayasinghe, L. Jiang, B. E. Johnson, B. Johnson, T. Ju, H. Kadara, K. Kaestner, J. Kagan, L. Kalinke, R. Keith, A. Khan, W. Kibbe, A. H. Kim, E. Kim, J. Kim, A. Kolodzie, M. Kopytra, E. Kotler, R. Krueger, K. Krysan, A. Kundaje, U. Ladabaum, B. B. Lake, H. Lam, R. Laquindanum, K. S. Lau, A. M. Laughney, H. Lee, M. Lenburg, C. Leonard, I. Leshchiner, R. Levy, J. Li, C. G. Lian, K.-H. Lim, J.-R. Lin, Y. Lin, Q. Liu, R. Liu, T. Lively, W. J. Longabaugh, T. Longacre, C. X. Ma, M. C. Macedonia, T. Madison, C. A. Maher, A. Maitra, N. Makinen, D. Makowski, C. Maley, Z. Maliga, D. Mallo, J. Maris, N. Markham, J. Marks, D. Martinez, R. J. Mashl, I. Masilionais, J. Mason, J. Massagué, P. Massion, M. Mattar, R. Mazurchuk, L. Mazutis, S. A. Mazzilli, E. T. McKinley, J. F. McMichael, D. Merrick, M. Meyerson, J. R. Miessner, G. B. Mills, M. Mills, S. B. Mondal, M. Mori, Y. Mori, E. Moses, Y. Mosse, J. L. Muhlich, G. F. Murphy, N. E. Navin, T. Nawy, M. Nederlof, R. Ness, S. Nevins, M. Nikolov, A. J. Nirmal, G. Nolan, E. Novikov, P. Oberdoerffer, B. O’Connell, M. Offin, S. T. Oh, A. Olson, A. Ooms, M. Ossandon, K. Owzar, S. Parmar, T. Patel, G. J. Patti, D. Pe’er, I. Pe'er, T. Peng, D. Persson, M. Petty, H. Pfister, K. Polyak, K. Pourfarhangi, S. V. Puram, Q. Qiu, Á. Quintanal-Villalonga, A. Raj, M. Ramirez-Solano, R. Rashid, A. N. Reeb, A. Regev, M. Reid, A. Resnick, S. M. Reynolds, J. L. Riesterer, S. Rodig, J. T. Roland, S. Rosenfield, A. Rotem, S. Roy, O. Rozenblatt-Rosen, C. M. Rudin, M. D. Ryser, S. Santagata, M. Santi-Vicini, K. Sato, D. Schapiro, D. Schrag, N. Schultz, C. L. Sears, R. C. Sears, S. Sen, T. Sen, A. Shalek, J. Sheng, Q. Sheng, K. I. Shoghi, M. J. Shrubsole, Y. Shyr, A. B. Sibley, K. Siex, A. J. Simmons, D. S. Singer, S. Sivagnanam, M. Slyper, M. P. Snyder, A. Sokolov, S.-K. Song, P. K. Sorger, A. Southard-Smith, A. Spira, S. Srivastava, J. Stein, P. Storm, E. Stover, S. H. Strand, T. Su, D. Sudar, R. Sullivan, L. Surrey, M. Suvà, K. Tan, N. V. Terekhanova, L. Ternes, L. Thammavong, G. Thibault, G. V. Thomas, V. Thorsson, E. Todres, L. Tran, M. Tyler, Y. Uzun, A. Vachani, E. V. Allen, S. Vandekar, D. J. Veis, S. Vigneau, A. Vossough, A. Waanders, N. Wagle, L.-B. Wang, M. C. Wendl, R. West, E. H. Williams, C. yun Wu, H. Wu, H.-Y. Wu, M. A. Wyczalkowski, Y. Xie, X. Yang, C. Yapp, W. Yu, Y. Yuan, D. Zhang, K. Zhang, M. Zhang, N. Zhang, Y. Zhang, Y. Zhao, D. C. Zhou, Z. Zhou, H. Zhu, Q. Zhu, X. Zhu, Y. Zhu, X. Zhuang, The human tumor atlas network: Charting tumor transitions across space and time at single-cell resolution, Cell 181 (2) (2020) 236–249.  doi:10.1016/j.cell.2020.03.053.
  * [52] R. L. Grossman, A. P. Heath, V. Ferretti, H. E. Varmus, D. R. Lowy, W. A. Kibbe, L. M. Staudt, Toward a shared vision for cancer genomic data, New England Journal of Medicine 375 (12) (2016) 1109–1112.  doi:10.1056/nejmp1607591.
  * [53] Proteomic data commons [cited 2023-01-30].
URL https://pdc.cancer.gov

  * [54] L. Schneider, S. Laiouar-Pedari, S. Kuntz, E. Krieghoff-Henning, A. Hekler, J. N. Kather, T. Gaiser, S. Fröhling, T. J. Brinker, Integration of deep learning-based image analysis and genomic data in cancer pathology: A systematic review, European Journal of Cancer 160 (2022) 80–91.  doi:10.1016/j.ejca.2021.10.007.


New constraints on $H_{0}$ and $\Omega_{m}$ from SZE/X-RAY data and Baryon
Acoustic Oscillations

R. F. L<EMAIL_ADDRESS>J. V.
<EMAIL_ADDRESS>and J. A. S<EMAIL_ADDRESS>
Departamento de Astronomia, Universidade de São Paulo

Rua do Matão, 1226 - 05508-900, São Paulo, SP, Brazil

###### Abstract

The Hubble constant, $H_{0}$, sets the scale of the size and age of the
Universe and its determination from independent methods is still worthwhile to
be investigated. In this article, by using the Sunyaev-Zel‘dovich effect and
X-ray surface brightness data from 38 galaxy clusters observed by Bonamente et
al. (2006), we obtain a new estimate of $H_{0}$ in the context of a flat
$\Lambda$CDM model. There is a degeneracy on the mass density parameter
($\Omega_{m}$) which is broken by applying a joint analysis involving the
baryon acoustic oscillations (BAO) as given by Sloan Digital Sky Survey
(SDSS). This happens because the BAO signature does not depend on $H_{0}$. Our
basic finding is that a joint analysis involving these tests yield
$H_{0}=0.765^{+0.035}_{-0.033}$ km s-1 Mpc-1 and
$\Omega_{m}=0.27^{+0.03}_{-0.02}$. Since the hypothesis of spherical geometry
assumed by Bonamente et al. is questionable, we have also compared the above
results to a recent work where a sample of triaxial galaxy clusters has been
considered.

##  1 Introduction

Ten years ago, two different teams using SNe type Ia as standard candles
announced that the expansion of the Universe is speeding up and not slowing as
believed by the cosmology community[1]. The authors interpreted their results
as implying that the cosmological constant $\Lambda$ is greater than zero, an
assumption that is not only surviving but has been strengthened in the last
decade by many independent astronomical observations[2, 3, 4, 5, 6, 7] (see
also Peebles and Ratra[8], and Padmanabhan[9] for reviews).

On the other hand, until now there is no consensus about the present value of
the Hubble parameter even when the estimates of $H_{0}$ are limited to the
Cepheid period-luminosity relation and the correlation of the luminosity at
maximum light for SNe type Ia. For instance, Riess et al.[10] using a hybrid
sample composed by Cepheids in galaxies and recent type Ia SNe concluded that
$H_{0}=73\pm 6$ $km.s^{-1}.Mpc^{-1}$ while Sandage[11] and coworkers based on
data from the Hubble Space Telescope (HST) key project are claiming that
$H_{0}=62\pm 5$ $km.s^{-1}.Mpc^{-1}$. Therefore, is still worthwhile to
improve the direct local estimates of $H_{0}$, and, probably, more important,
to investigate their concordance with independent methods. An interesting one
is provided by the combination of Sunyaev-Zel’dovich Effect (SZE) and
measurements of X-ray surface brightness from galaxy clusters.

The so-called SZE is a small distortion on the Cosmic Microwave Background
(CMB) spectrum provoked by the inverse Compton scattering of the CMB photons
passing through a population of hot electrons[12, 13]. Observing the
temperature decrement of the CMB in the direction of galaxy clusters and the
X-ray spectrum emitted by the hot electron gas pervading the cluster, it is
possible to break the degeneracy between concentration and temperature[14],
and, therefore, to calculate the angular diameter distance (ADD)[15, 16, 17].
This technique for measuring distances is completely independent of other
methods (as the one provided by the luminosity distance), and it can be used
to measure distances at high redshifts directly. Review papers on this subject
have been published by Birkinshaw[18] and Carlstrom, Holder and Reese[19].
More recently, such a technique has been applied for a fairly large number of
clusters[20, 21, 22, 23, 24].

In 2006, Bonamente and collaborators[23] determined the ADD distance to 38
clusters of galaxies in the redshift range $0.14\leq z\leq 0.89$ using X-ray
data from Chandra and Sunyaev-Zeldovich effect data data from the Owens Valley
Radio Observatory and the Berkeley-Illinois-Maryland Association
interferometric arrays. Assuming spherical symmetry, the cluster plasma and
dark matter distributions were analyzed by using a hydrostatic equilibrium
model accounting for radial variations in density, temperature and abundances.
In addition, by fixing the earlier cosmic concordance $\Lambda$CDM model
($\Omega_{m}=0.3$, $\Omega_{\Lambda}=0.7$), they obtained for the Hubble
constant $H_{0}=76.9^{+3.5+10.0}_{-3.4-8.0}\,km.s^{-1}.Mpc^{-1}$ (statistical
followed by systematic uncertainty at 1$\sigma$ c. l.). The main advantage of
this method for estimating $H_{0}$ is that it does not rely on the
extragalactic distance ladder being fully independent of any local calibrator.
In a point of fact, once the angular diameter distance has been measured a
large amount of astrophysical and cosmological tests can be performed in order
to constrain the relevant free parameters[25, 26, 27].

It should be stressed, however, that in the above quoted determination of the
Hubble constant, a specific cosmology was fixed from the very beginning. This
is needed because the determination of the Hubble parameter from SZE/X-ray
technique is endowed with a strong degeneracy with respect to the matter
density parameter which can only be broken by considering additional
cosmological tests. An interesting possibility is given by the signature of
the baryon acoustic oscillations (BAO) from the last scattering surface (LSS).

In 2005, Eisenstein et al.[28] presented the large scale correlation function
from the Sloan Digital Sky Survey (SDSS) showing clear evidence for the baryon
acoustic peak at $100h^{-1}$ Mpc scale, a result in good agreement with the
analyses from CMB data[3]. The breaking on the degeneracy between $H_{0}$ and
$\Omega_{m}$ by applying this BAO signature is possible because it does not
depend on $H_{0}$ and is highly sensitive to the matter density parameter. The
combination SZE/X-ray with BAO has been recently discussed in the literature
by Cunha, Marassi and Lima[29].

In this article, we discuss the determinations of $H_{0}$ and $\Omega_{m}$ by
considering the same 38 clusters from Bonamente et al. sample where spherical
symmetry was assumed. Our basic findings follow from a joint analysis
involving the data from SZE and X-ray surface brightness with the recent SDSS
measurements of the baryon acoustic peak. The influence of the intrinsic
cluster geometry on $H_{0}$ it will be quantified by comparing the results
derived here with a recent determination of both parameters[29] based on De
Filippis et al. sample formed by 25 triaxial clusters[22].

##  2 Basic Equation and Statistical Analysis

Let us now consider that the Universe is described by a flat Friedmann-
Robertson-Walker (FRW) geometry

$ds^{2}=dt^{2}-a^{2}(t)\left[dr^{2}+r^{2}(d\theta^{2}+sin^{2}\theta
d\phi)\right],$ (1)

driven by cold dark matter plus a cosmological constant ($\Lambda$CDM).

In this background, the angular diameter distance, ${\cal{D}}_{A}$, can be
written as[29, 30, 31]

${\cal{D}}_{A}(z;h,\Omega_{m})=\frac{3000h^{-1}}{(1+z)}\int_{o}^{z}\frac{dz^{\prime}}{{\cal{H}}(z^{\prime};\Omega_{m})}\quad\mbox{Mpc},$
(2)

where $h=H_{0}/100$ km s-1 Mpc-1 and the dimensionless function
${\cal{H}}(z^{\prime};\Omega_{m})$ is given by
${\cal{H}}=\left[\Omega_{m}(1+z^{\prime})^{3}+(1-\Omega_{m})\right]^{1/2}.$ As
it appears, the above expression has only two free parameters
($h,\Omega_{m}$). In this way, we perform a statistical fit over the
$h-\Omega_{m}$ plane in light of Bonamente et al. data sample[23]. In our
analysis we use a maximum likelihood that can be determined by a $\chi^{2}$
statistics,

$\chi^{2}(z|\mathbf{p})=\sum_{i}{({\cal{D}}_{A}(z_{i};\mathbf{p})-{\cal{D}}_{Ao,i})^{2}\over\sigma_{{\cal{D}}_{Ao,i}}^{2}+\sigma_{stat}^{2}},$
(3)

where ${\cal{D}}_{Ao,i}$ is the observational ADD, $\sigma_{{\cal{D}}_{Ao,i}}$
is the uncertainty in the individual distance, $\sigma_{stat}$ is the
contribution of the statistical errors (see table 3 in Bonamente et al.[23]
(2006)) added in quadrature ($\approx 20$%) and the complete set of parameters
is given by $\mathbf{p}\equiv(h,\Omega_{m})$.

In what follows, we first consider the SZE/X-ray distances separately, and,
further, we present a joint analysis including the BAO signature from the SDSS
catalog. Note that a specific flat $\Lambda$CDM cosmology has not been fixed a
priori in the analysis below.

Figure 1: a) Angular diameter distance as a function of redshift for
$\Omega_{m}=0.3$, $\Omega_{\Lambda}=0.7$ and some selected values of the $h$
parameter. The data points correspond to the the SZE/X-ray distances for 38
clusters from Bonamente et al. sample. b) Confidence regions ($68.3$%, $95.4$%
and $99.7$%) in the $(\Omega_{m},h)$ plane provided by the SZE/X-ray data. The
best fit values are $h=0.743$ and $\Omega_{m}=0.37$. As remarked in the text,
the possible values of $H_{0}$ are heavily dependent on the allowed values of
$\Omega_{m}$, and, therefore, such a degeneracy need to be broken by adding a
new cosmological test.

###  2.1 Limits from SZE/X-ray

To begin with, let us consider the influence of the Hubble parameter on the
ADD once the cosmology is fixed.

In Fig. 1(a), we display the residual Hubble diagram and the galaxy cluster
data sample by considering a flat cosmic concordance model
($\Omega_{m}=0.3,\Omega_{\Lambda}=0.7$). As expected, for a given redshift,
the distances increase for smaller values of $H_{0}$.

In Fig. 1(b) we show the contours of constant likelihood (68.3%, 95.4% and
99.7%) in the space parameter $h-\Omega_{m}$ for the SZ/X-ray data above
discussed. Note that a large range for the $h$ parameter is allowed,
($0.65\leq h\leq 0.86$), at $1\sigma$ of confidence level. In particular, we
found $h=0.755^{+0.065}_{-0.065}$ with $\chi^{2}_{min}=35.2$ at $68.3$%
confidence level (c.l.). Naturally, such bounds on $h$ are reasonably
dependent on the cosmological model adopted. For example, if we fix
$\Omega_{m}=0.3$ we have $h=0.76$, for $\Omega_{m}=1.0$ we have $h=0.65$, and
both cases are permitted with high degree of confidence. This means that using
only the Bonamente et al. sample (spherical cluster hypothesis) we cannot
constrain severely the energetic components of the $\Lambda$CDM model with
basis on the SZE/X-ray data alone. As one may conclude, one additional
cosmological test (fixing $\Omega_{m}$) is necessary in order to break the
degeneracy on the $(\Omega_{m},h)$ plane.

###  2.2 Joint analysis for SZE/X-ray and BAO

As remarked in the introduction, more stringent constraints on the space
parameter ($h,\Omega_{m}$) can be obtained by combining the SZE/X-ray with the
BAO signature[28]. The peak detected (from a sample of 46748 luminous red
galaxies selected from the SDSS Main Sample) is predicted to arise precisely
at the measured scale of 100 $h^{-1}$ Mpc. Let us now consider it as an
additional cosmological test over the spherical cluster sample. Such a
measurement is characterized by

Figure 2: a) Contours in the $\Omega_{m}-h$ plane using the SZE/X-ray and BAO
joint analysis. The contours correspond to $68.3$%, $95.4$% and $99.7$%
confidence levels. The best-fit model converges to $h=0.765$ and
$\Omega_{m}=0.27$. b) Likelihood function for the $h$ parameter in a flat
$\Lambda$CDM universe from SZE/X-ray emission and BAO. The horizontal lines
are cuts in the regions of $68.3$% probability and $95.4$%.

$\displaystyle{\cal{A}}\equiv{\Omega_{\rm{m}}^{1/2}\over{{\cal{H}}(z_{\rm{*}})}^{1/3}}\left[\frac{1}{z_{\rm{*}}}\Gamma(z_{*})\right]^{2/3}=0.469\pm
0.017,$ (4)

where $z_{\rm{*}}=0.35$ is the redshift at which the acoustic scale has been
measured and $\Gamma(z_{*})$ is the dimensionless comoving distance to
$z_{*}$. Note that the above quantity is independent of the Hubble constant,
and, as such, the BAO signature alone constrains only the $\Omega_{m}$
parameter.

Table 1: Limits to $h$ using SZ/X-ray method from galaxy clusters ($\Lambda$CDM) Reference (data) | $\Omega_{m}$ |  $h$ ($1\sigma$) | $\chi^{2}$
---|---|---|---
Mason et al. 2001 (7 clusters).. | $0.3$ | $0.66^{+0.14}_{-0.11}$ | $\simeq 2$
Reese et al. 2002 (18 cluster).. | $0.3$ | $0.60^{+0.04}_{-0.04}$ | $16.5$
Reese 2004 (41 clusters).. | $0.3$ | $0.61^{+0.03}_{-0.03}$ | –
Jones et al. 2005 (5 clusters).. | $0.3$ | $0.66^{+0.11}_{-0.10}$ | –
Bonamente et al. 2006 (38 clusters).. | $0.3$ | $0.77^{+0.04}_{-0.03}$ | $31.6$
Cunha et al. 2007 (24 triaxial clusters) + BAO.. | $0.273^{+0.03}_{-0.02}$ | $0.738^{+0.042}_{-0.033}$ | $24.5$
This paper (38 clusters)+BAO.. | $0.273^{+0.03}_{-0.02}$ | $0.765^{+0.035}_{-0.033}$ | $35.3$

In Fig. 2(a), we show the confidence regions for the SZE/X-ray cluster
distance and BAO joint analysis. By comparing with Fig. 1(b), one may see how
the BAO signature breaks the degeneracy in the $(\Omega_{\rm{m}},h)$ plane. As
it appears, the BAO test presents a striking orthogonality centered at
$\Omega_{m}=0.274^{+0.036}_{-0.026}$ with respect to the angular diameter
distance data as determined from SZE/X-ray processes. We find
$h=0.765^{+0.035}_{-0.033}$ at $68.3$% c.l. and
$\Omega_{m}=0.273^{+0.03}_{-0.02}$ at $68.3$% c.l. for $1$ free parameter
($\chi^{2}_{min}=35.3$). In light of these results, the important lesson here
is that the combination of SZE/X-ray with BAO provides an interesting approach
to constrain the Hubble constant.

In Fig. 2(b), we have plotted the likelihood function for the $h$ parameter in
a flat $\Lambda$CDM universe for the SZE/X-ray + BAO data set. The dotted
lines are cuts in the regions of $68.3$% probability and $95.4$% (1 free
parameter).

At this point, it is interesting to compare our results with others recent
works. In table 1, a list of recent Hubble parameter determinations based on
cluster data is displayed. Note that in the first five works the $h$ values
were obtained by fixing the cosmology ($\Omega_{m}=0.3$,
$\Omega_{\Lambda}=0.7$), while in our estimate the SZE/X-ray + BAO technique
was used (no one specific cosmology has been fixed).

On the other hand, the importance of the intrinsic geometry of the cluster has
been emphasized by many authors[32, 33, 34, 35]. As a consequence, the
standard spherical geometry has been severely questioned, since Chandra and
XMM-Newton observations have shown that clusters usually exhibit an elliptical
surface brightness. In this concern, a previous determination of $H_{0}$ from
SZE/X-ray + BAO by Cunha et al.[29] was based in a smaller sample (25 triaxial
clusters) observed by De Fillipis and collaborators[22]. Their results
suggested that 15 clusters are in fact more elongated along the line of sight,
while the remaining 10 clusters are compressed. Actually, the assumed cluster
shape seems to affect considerably the SZE/X-ray distances, and, therefore,
the $H_{0}$ estimates.

As shown in table 1, the central value of $H_{0}$ in this case (triaxial
geometry) is exactly the same determined by Riess et al.[10] by using a hybrid
sample (Cepheids + Supernovae). It is also closer to the Hubble Space
Telescope (HST) key project determination of the Hubble parameter announced by
Friedman et al.[36] based only on Cepheids as distance calibrators.

##  3 Comments and Conclusions

In this paper we have discussed a new determination of the Hubble constant
based on the SZE/X-ray distance technique for a sample of 38 clusters as
compiled by Bonamente et al. assuming spherical symmetry[23]. The degeneracy
on the $\Omega_{m}$ parameter was broken trough a joint analysis applying the
baryon acoustic oscillation signature from the SDSS catalog. The Hubble
constant was constrained to be $h=0.765^{+0.035}_{-0.033}$ for $1\sigma$ and
$\Omega_{m}=0.273^{+0.03}_{-0.02}$. These limits were derived assuming the
flat $\Lambda$CDM scenario and a spherical $\beta$-model. The central $h$
value derived here is in perfect agreement with the Bonamente et al. value,
$h=0.769$ (assuming a hydrostatic equilibrium and fixing the concordance
model), as well as with others recent estimates coming from WMAP and Hubble
Space telescope Key Project, where $h=0.73$. However, it differs slightly of a
recent estimation by Cunha, Marassi & Lima, $h=0.74$, where the SZE/X-Ray +
BAO technique was applied for 24 triaxial galaxies clusters. In general
ground, such results are suggesting that the combination of these three
independent phenomena (SZE/X-ray and BAO) provides an interesting method to
constrain the Hubble constant.

##  4 Acknowledgements

The authors are very grateful to the organizers of the IWARA (International
Workshop on Relativistic Astrophysics, João Pessoa, Brazil, 2007). RFLH and
JVC are supported by FAPESP No. 07/5291-2 and 05/02809-5, respectively. JASL
is supported by CNPq and FAPESP No. 04/13668-0 (Brazilian Research Agencies).

## References

  * [1] S. Perlmutter et al., Nature,  391, 51 (1998); S. Perlmutter et al., ApJ. 517, 565 (1999); A. G. Riess et al., Astron. J. 116, 1009 (1998).
  * [2] P. Astier et al., Astron. Astrophys. 447, 31 (2006); A. G. Riess et al., Astrophys. J. 659, 98 (2007); T. M. Davis et al., Astrophys. J. 666, 716 (2007).
  * [3] D. N. Spergel et al., Astrop. J. Suppl. 148, 175 (2003); D. N. Spergel et al., Astrophys. J. Supl. 170, 377 (2007); J. Dunkley et al., arXiv:0803.0586 [astro-ph]; E. Komatsu et al., [ArXiv: 0803.0547v1].
  * [4] R. G. Carlberg et al., ApJ. 462, 32 (1996); A. Dekel, D. Burstein and S. D. M. White, In Critical Dialogues in Cosmology, edited by N. Turok World Scientific, Singapore (1997); P. J. E. Peebles, in Formation of Structure in the Universe, edited by A. Dekel and J. P. Ostriker, Cambridge UP, Cambridge (1999).
  * [5] D. J. Eisenstein et al., Astrophys. J. 633, 560 (2005).
  * [6] S. W. Allen, R. W. Schmidt and A. C. Fabian, MNRAS 334, L11 (2002); S. Ettori, P. Tozzi and P. Rosati, A&A 398, 879 (2003); J. A. S. Lima, J. V. Cunha and J. S. Alcaniz, Phys. Rev. D 68, 023510 (2003); J. V. Cunha, L. Marassi and R. C. Santos, IJMPD 16, 403 (2007).
  * [7] J. Dunlop et al., Nature 381, 581 (1996); Y. Yoshii, T. Tsujimoto and K. Kawara, ApJ 507, L133 (1998); J. S. Alcaniz and J. A. S. Lima, ApJ 521, L87 (1999)[astro-ph/9902298]; J. S. Alcaniz, J. A. S. Lima and J. V. Cunha, MNRAS 340, L39 (2003)[astro-ph/0301226]; J. V. Cunha and R. C. Santos, IJMP D 13, 1321 (2004)[astro-ph/0402169]; A. Friaca, J. S. Alcaniz and J. A. S. Lima, Mon. Not. R. Astron. Soc. 362, 1295 (2005), [astro-ph/0504031]; J. F. Jesus, Gen. Rel. Grav. 40, 791 (2008), [astro-ph/0603142]; J. A. S. Lima, J. F. Jesus and J. V. Vunha, arXiv:0709.2195 [astro-ph].
  * [8] P. J. E. Peebles and B. Ratra Rev. Mod. Phys. 75, 559 (2003).
  * [9] T. Padmanabhan, Phys. Rept. 380, 235 (2003).
  * [10] A. G. Riess et al., Astrophys. J., 627, 579 (2005).
  * [11] A. Sandage et al., Astrophys. J., 653, 843 (2006).
  * [12] R. A. Sunyaev and Ya. B. Zel’dovich, Comments Astrophys. Space Phys. 4 (1972) 173.
  * [13] N. Itoh, Y. Kohyama and S. Nozawa, Astrophys. J. 502, 7 (1998).
  * [14] The SZE and X-ray surface brightness are sensible to different combinations of the cluster electron gas density $n_{e}$ and its temperature $T_{e}$.
  * [15] J. Silk and S. D. M. White, Astrophys. J. 226, L103 (1978).
  * [16] J. G. Bartlett and J. Silk, Astrophys. J. 423, 12 (1994); J. G. Bartlett, Astrop. Space Sci. 290, 105 (2004).
  * [17] A. Cavaliere, L. Danese and G. De Zotti, Astron. and Astrophys. 75, 322 (1979).
  * [18] M. Birkinshaw, Phys. Rep. 310 (1999) 97.
  * [19] J. E. Carlstrom, G. P. Holder and E. D. Reese, ARA&A 40, 643 (2002).
  * [20] E. D. Reese et al., Astrophys. J. 581, 53 (2002).
  * [21] M. E. Jones et al., MNRAS 357, 518 (2002).
  * [22] E. De Filippis, M. Sereno, M.W. Bautz and G. Longo, Astrophys. J. 625, 108 (2005).
  * [23] M. Bonamente et al., Astrophys. J. 647, 25 (2006).
  * [24] S. LaRoque et al. Astrophys. J. 652, 917 (2006).
  * [25] J. G. Bartlett and J. Silk, Astrophys. J. 423, 12 (1994) .
  * [26] Y. Rephaeli, ARA&A, 33, 541 (1995).
  * [27] S. Kobayashy, S. Sasaki and Y. Suto, PASJ 48 L107 (1996).
  * [28] D. J. Eisenstein, Astrophys. J. 633, 560 (2005).
  * [29] J. V. Cunha, L. Marassi and J. A. S. Lima, MNRAS 379 (2007) L1[astro-ph/0611934].
  * [30] Note that we have replaced $H_{0}^{-1}=3000h^{-1}$ Mpc. See for instance, E. W. Kolb and M. S. Turner, The Early Universe, Addison-Wesley, New York (1990).
  * [31] J. A. S. Lima and J. S. Alcaniz, Astron. Astrophys. 357, 393 (2000)[astro-ph/0003189]; Astrophys. J. 566, 15 (2002)[astro-ph/0201056]; Astrophys. J. 618, 16 (2005)[astro-ph/0308465]; Z-H. Zhu and M.-K. Fujimoto, Astrophys. J., 581, 1 (2002); J. S. Alcaniz, Phys. Rev. D 69, 083521 (2004)[astro-ph/0312424]; J. V. Cunha, J. S. Alcaniz and J. A. S. Lima, Phys. Rev. D 69, 083501 (2004)[astro-ph/0306319].
  * [32] D. C. Fox and U.-L. Pen, Astrophys. J. 574, 38 (2002).
  * [33] Y. P. Jing and Y. Suto, Astrophys. J. 574, 538 (2002).
  * [34] M. Plionis, S. Basilakos and C. Ragone-Figueroa, Astrophys. J. 650, 770 (2006).
  * [35] M. Sereno, E. De Filippis, G. Longo and M. W. Bautz, Astrophys. J. 645, 170 (2006).
  * [36] W. L. Freedman et al., Astrophys. J., 553, 47 (2001).