# 3D Brain and Heart Volume Generative Models: A Survey

Yanbin Liu<EMAIL_ADDRESS>0000-0003-4724-8065 Harry Perkins Institute of
Medical Research, The University of Western AustraliaPerthWAAustralia6009 ,
Girish Dwivedi  Harry Perkins Institute of Medical Research, The University of
Western Australia, Fiona Stanley HospitalPerthWAAustralia
<EMAIL_ADDRESS>,  Farid Boussaid  Department of Electrical,
Electronic and Computer Engineering, The University of Western
AustraliaPerthWAAustralia<EMAIL_ADDRESS>and  Mohammed Bennamoun
Department of Computer Science and Software Engineering, The University of
Western AustraliaPerthWAAustralia<EMAIL_ADDRESS>
(2022)

###### Abstract.

Generative models such as generative adversarial networks and autoencoders
have gained a great deal of attention in the medical field due to their
excellent data generation capability. This paper provides a comprehensive
survey of generative models for three-dimensional (3D) volumes, focusing on
the brain and heart. A new and elaborate taxonomy of unconditional and
conditional generative models is proposed to cover diverse medical tasks for
the brain and heart: unconditional synthesis, classification, conditional
synthesis, segmentation, denoising, detection, and registration. We provide
relevant background, examine each task and also suggest potential future
directions. A list of the latest publications will be updated on Github to
keep up with the rapid influx of papers at
https://github.com/csyanbin/3D-Medical-Generative-Survey.

generative models, three dimension, medical images, brain and heart

††copyright: acmcopyright††journalyear: 2022††doi: XXXXXXX.XXXXXXX††journal:
JACM††publicationmonth: 9††ccs: Computing methodologies Computer vision††ccs:
Computing methodologies 3D imaging

##  1\. Introduction

A wide range of research fields has embraced deep learning (DL) in recent
years, including image processing (Krizhevsky et al., 2012; He et al., 2016,
2017; Girshick, 2015), speech recognition (Mikolov et al., 2011; Hinton et
al., 2012; Graves et al., 2013), natural language processing (Collobert et
al., 2011; Sutskever et al., 2014; Cho et al., 2014; Vaswani et al., 2017;
Kenton and Toutanova, 2019), and robotics (Pierson and Gashler, 2017;
Sünderhauf et al., 2018). Thus, the medical imaging community has put in
significant efforts to take advantage of deep learning advances, and medical
imaging research has made significant progress with respect to a variety of
applications including classification (Biffi et al., 2018; Zhang et al., 2017;
Kruthika et al., 2019; Pombo et al., 2019; Puyol-Antón et al., 2020),
segmentation (Niyas et al., 2022; Tritrong et al., 2021; Liu et al., 2020;
Enokiya et al., 2018; Tritrong et al., 2021), registration (Zhu et al., 2021;
Ramon et al., 2022), detection (Pinaya et al., 2022b; Uzunova et al., 2019;
Pinaya et al., 2022a), denoising (Wolterink et al., 2017; Wang et al., 2018c,
a; Ran et al., 2019; Zhou et al., 2022), and synthesis (Hong et al., 2021; Liu
et al., 2022; Liao et al., 2018; Huang et al., 2020; Dalmaz et al., 2021), as
well as with various imaging modalities, including Computed Tomography (CT)
(Würfl et al., 2016; Lell and Kachelrieß, 2020), ultrasound (Liu et al.,
2019a), Magnetic Resonance Imaging (MRI) (Lundervold and Lundervold, 2019;
Akkus et al., 2017), and Positron Emission Tomography (PET) (Reader et al.,
2020).

A large number of annotated training images, obtained with the aid of crowd-
sourcing annotation platforms like Amazon Mechanical Turk (Paolacci et al.,
2010), were required for deep learning to be successful in natural image
processing. However, the complexity of collection procedures, the lack of
experts, privacy concerns, and the mandatory requirement of consent from
patients make the annotation process a major bottleneck in medical imaging. In
order to mitigate this issue, deep generative models (e.g., generative
adversarial networks (GANs) (Goodfellow et al., 2014) and variational
autoencoder (VAE) (Kingma and Welling, 2013)) have been introduced to medical
imaging. In these generative models, the original data distribution is
mimicked so that realistic images are generated (Hong et al., 2021; Volokitin
et al., 2020; Sun et al., 2022) or cross-modality synthesis can be achieved
(Lin et al., 2021; Liao et al., 2018; Yu et al., 2020).

There have been numerous survey papers published on deep generative models for
medical imaging due to the rapid progress of the field (AlAmir and AlGhamdi,
2022; Yi et al., 2019; Kazeminia et al., 2020; Chen et al., 2022; Iqbal et
al., 2022; Jeong et al., 2022; Xiao et al., 2021; Ali et al., 2022). These
surveys cover different medical applications and provide an overall review of
GANs on general medical image analysis (AlAmir and AlGhamdi, 2022; Yi et al.,
2019; Kazeminia et al., 2020). Some focus on a specific application only, such
as augmentation (Chen et al., 2022), segmentation (Iqbal et al., 2022; Jeong
et al., 2022) and registration (Xiao et al., 2021). Others concentrate on a
specific image modality, such as MRI (Ali et al., 2022). Even though many
surveys exist, we find that there is a lack of comprehensive surveys on three-
dimensional (3D) medical volume, which is the original data format of many
medical modalities, such as MRI, CT, and PET, for example. Moreover, existing
surveys mainly focus on GANs, neglecting other effective generative models
such as Autoencoders (AEs) (Hinton, 1994) and Autoregressive models (Van Oord
et al., 2016).

This inspired us to conduct a comprehensive survey of generative models for 3D
medical volume images of the brain and heart. Since 3D volume is the intrinsic
representation of many medical imaging modalities, it displays the entire and
thorough anatomical structure of organs, whereas a 2D medical image only shows
a specific view/plane. GANs are widely used for 3D medical volumes, but there
has also been an increased interest in AEs (e.g., Diffusion Model (Kim and Ye,
2022; Pinaya et al., 2022a)) and Autoregressive models (e.g., Autoregressive
Transformers (Pinaya et al., 2022b)). As a result, our survey covers all three
types of generative models. As far as organs are concerned, we restrict our
interest to the brain and heart for the following reasons: (1) they are two
vital organs that control the mental and physiological functions of the human
body; (2) both organs involve a wide range of applications, e.g., segmentation
(Fig. 1(a)); (3) generative models are essential for both organs because of
their data scarcity; (4) by covering these 2 organs, we are able to cover
generative models for both static (i.e., brain) and dynamic organs (i.e.,
heart).

Figure 1. Statistics of the 3D brain and heart volume generative models. (a)
Statistics of all publications according to medical applications. (b)
Categorization by year of publication (2017-2022). Uncond. Syn.: Unconditional
Synthesis, Cond. Syn.: Conditional Synthesis.

Contributions. To provide a comprehensive and organized survey, we introduce a
new taxonomy (Fig. 2) that divides generative models into unconditional (only
taking a random variable as input) and conditional (taking an additional data
modality as input). In Figure 1, we provide a statistical analysis of the
proportion of publications per application and the number of publications per
year. Our contributions can be summarized as follows:

  * •

This is the _first survey_ on generative models for 3D medical volume images,
focusing on two important organs, i.e., the brain and the heart. It aims to
bridge the gap between the research of the 3D generative models community and
the research of the medical imaging community.

  * •

We provide a new taxonomy (Fig. 2) of 3D generative models by categorizing
them as unconditional or conditional generative models. Every category
includes several relevant medical applications.

  * •

Whilst most existing surveys focus on GANs only, we cover three main
categories of generative models: GANs, AEs, and Autoregressive models.

  * •

We discuss the key challenges and future directions of 3D medical generative
models and applications.

Paper Organization. The remainder of this paper is organized as follows. We
introduce the foundational techniques and challenges of 3D generative models
in Section 2. Section 3 comprehensively elaborates on the 3D medical
applications of both the unconditional and conditional generative models,
including unconditional synthesis, classification, conditional synthesis,
segmentation, denoising, detection, and registration. Then, Section 4
discusses the above surveyed applications and gives four future directions.
Finally, Section 5 concludes the paper.

Figure 2. Proposed taxonomy of 3D generative models for the brain and the
heart.

##  2\. Background

Recent deep generative models are mainly categorized into three types: GANs
(Goodfellow et al., 2014), AEs (Hinton, 1994), and Autoregressive models
(Gregor et al., 2014). The key challenge of generative models is how to
effectively model real data distributions. GANs devise a two-player (generator
and discriminator) game to approach the Nash equilibrium, where the real data
distribution equals the generated data distribution. AEs capture the intrinsic
data representation through a bottleneck structure (encoder-decoder).
Autoregressive models directly model the joint distribution of all pixels in
an image as the product of conditional distributions. Although GANs are the
most widely used models in various applications (e.g., synthetic face
generation), AEs and Autoregressive models are also indispensable for medical
research as we will show in this paper. While existing review papers (AlAmir
and AlGhamdi, 2022; Yi et al., 2019; Kazeminia et al., 2020) only focus on
GANs, we fill in this gap and cover all three types of generative models.

###  2.1. Unconditional and Conditional Generative Models

Figure 3. Generation process in unconditional and conditional generative
models.

In this paper, we divide generative models for 3D volume images into two
categories: unconditional model and conditional model, as shown in Fig. 3,
where we only show the generation process for simplicity.

In an unconditional generative model (Fig. 3(a)), the input is a random noise
variable $z$, and the output from the generation model is the generated image
X. Several model architectures belong to this type of model. A GAN generator,
for example, only uses random noise variables to synthesize images. Random
Gaussian variables are input to the decoder of the VAE model. The
unconditional generative model is used in several 3D medical applications,
including unconditional synthesis (Kwon et al., 2019; Hong et al., 2021; Liu
et al., 2022; Volokitin et al., 2020) and classification (Lan et al., 2020;
Pombo et al., 2019; Biffi et al., 2018).

In a conditional generative model (Fig. 3(b)), in addition to the random noise
$z$, the informative input data $X_{1}$ (e.g., semantic or visual input) is
also fed to the generation model to help generate the output image
${\mathbf{X}}_{2}$. Depending on the applications, a variety of data formats
are supported by $X_{1}$, including class labels (Mirza and Osindero, 2014),
attributes (Yan et al., 2016), texts (Reed et al., 2016), and images (Isola et
al., 2017). For example, the original cGAN (Mirza and Osindero, 2014)
generated synthetic MNIST (LeCun et al., 1998) images conditioned on the class
labels. Pixel-to-pixel (Isola et al., 2017) achieved image style transfer by
training a conditional GAN whose generator and discriminator were both based
on the input images. Many 3D medical applications take advantage of the
conditional generative model, including conditional synthesis (Pombo et al.,
2021; Yu et al., 2018; Pan et al., 2018; Liao et al., 2018; Yu et al., 2019;
Yang et al., 2021a; Dalmaz et al., 2021), segmentation (Rezaei et al., 2018;
Cirillo et al., 2020; Zhang et al., 2018b; Rezaei et al., 2018; Zhang et al.,
2021), denoising (Wolterink et al., 2017; Wang et al., 2018c, a; Ran et al.,
2019; Zhou et al., 2022), detection (Uzunova et al., 2019; Han et al., 2021;
Xiong et al., 2022; Pinaya et al., 2022b), and registration (Zhao et al.,
2019; Balakrishnan et al., 2019; Zhang et al., 2020; Ramon et al., 2022; Zhu
et al., 2022).

From the data distribution perspective, the unconditional generative model
captures the original realistic data distribution without requiring any
additional information other than a random noise. The conditional generative
model can be regarded as a transformation from the input data distribution
$p({\mathbf{X}}_{1})$ to the output image distribution $p({\mathbf{X}}_{2})$.
By proposing a new unconditional/conditional taxonomy perspective, we hope
researchers can gain valuable insights into future model design and apply them
to target medical applications in the future.

###  2.2. Generative Adversarial Networks

Generative Adversarial Networks were proposed by Goodfellow  _et al._
(Goodfellow et al., 2014) in 2014. The main idea is to design two networks (a
discriminator and a generator) to contest with each other in a zero-sum two-
layer game. Specifically, the generator $G$ takes a random noise variable $z$
as the input to generate the synthesized images $G(z)$. The role of the
discriminator is to distinguish between the realistic images $x$ and the
generated fake images $G(z)$. In an ideal case, the two-player game reaches a
Nash equilibrium (Farnia and Ozdaglar, 2020; Heusel et al., 2017) where the
synthesized images $G(z)$ are indistinguishable from real images $x$. The
equilibrium is difficult to achieve in practice, and GANs training suffers
from two problems, i.e., training instability (Tao and Wang, 2020; Saxena and
Cao, 2021) and mode collapse (Srivastava et al., 2017; Thanh-Tung and Tran,
2020). Diverse architectures and training strategies have been proposed to
address the two problems and improve GANs performance (Salimans et al., 2016;
Zhao et al., 2021c; Zhao et al., 2021b; Hinz et al., 2021). Below, we describe
the variants that are most relevant to the generation of 3D medical volume
images.

####  2.2.1. Vanilla GAN

The structure of the vanilla GAN is shown in Fig. 4(a). Based on a prior
distribution of the input noise variable $z\sim p_{z}(z)$, the generator
trains its distribution $p_{g}$ over $x$ to approximate the real data
distribution $p_{data}$. The discriminator maximizes the accuracy of
classifying real/fake images by optimizing over $D(x)$/$D(G(z))$. The minmax
optimization problem for $G$ and $D$ is defined as follow:

(1) $\min_{G}\max_{D}V(D,G)=\mathbb{E}_{x\sim p_{data}(x)}[\log
D(x)]+\mathbb{E}_{z\sim p_{z}(z)}[\log(1-D(G(z)))]\,.$

Theoretically, for arbitrary functions $G$ and $D$, the optimal solution
satisfies $p_{g}=p_{data}$ and $D(x)=D(G(z))=\frac{1}{2}$. But in practice,
$G$ and $D$ are usually implemented by deep neural networks (e.g., multilayer
perceptrons or convolution neural networks), which only have limited capacity
and cover a limited family of $p_{g}$ distributions.

Figure 4. The structure of (a) the vanilla (unconditional) and (b) the
conditional GAN.

####  2.2.2. Conditional GAN

The vanilla GAN is unconditional, which means it cannot use additional
information or control the modes of the generated data. Hence, conditional GAN
(cGAN) (Mirza and Osindero, 2014) was proposed in order to improve the
flexibility of vanilla GAN by conditioning on additional information. As shown
in Fig. 4(b), the conditional input $c$ is fed into both the generator $G$ and
discriminator $D$ (red line). The minmax optimization problem for $D$ and $G$
is then defined as follow:

(2) $\min_{G}\max_{D}V(D,G)=\mathbb{E}_{x\sim p_{data}(x)}[\log
D(x|c)]+\mathbb{E}_{z\sim p_{z}(z)}[\log(1-D(G(z|c)))]\,.$

The conditional GAN (Eq. 2) is different from the unconditional GAN (Eq. 1)
because it conditions on $c$ in the generation and discrimination process. $c$
is used in the original paper (Mirza and Osindero, 2014) to represent labels
in the MNIST (LeCun et al., 1998) image generation and to represent image
features in the generation of multi-model Flickr tags.

The conditional input $c$ can be instantiated with different information and
modalities so as to perform various image generation tasks. These tasks
include text-to-image synthesis (Zhang et al., 2017, 2018a), image-to-image
translation (Isola et al., 2017; Zhu et al., 2017), attribute editing (Yan et
al., 2016; Perarnau et al., 2016), image segmentation (Tritrong et al., 2021;
Enokiya et al., 2018), etc.

####  2.2.3. Pixel-to-pixel

Pixel-to-pixel was proposed in (Isola et al., 2017) to solve the general
image-to-image translation problems, e.g., edge to photo, day to night. In its
implementation, both the generator and discriminator are fed with input images
to construct a conditional GAN architecture. The objective function is as
follows:

(3) $\min_{G}\max_{D}\mathbb{E}_{x,y}[\log
D(x,y)]+\mathbb{E}_{x,z}[\log(1-D(x,G(x,z)))]+\lambda\mathbb{E}_{x,y,z}[\|y-G(x,z)\|_{1}]\,.$

Pixel-to-pixel is trained with image pairs $(x,y)$, and translates images from
$x$ domain to $y$ domain after training.

####  2.2.4. CycleGAN

Since Pixel-to-pixel (Isola et al., 2017) relies on a large number of
annotated image pairs, it is time-consuming and sometimes infeasible. CycleGAN
(Zhu et al., 2017) tackles this problem by taking advantage of the concept of
cycle-consistency. Two discriminators $G$ and $F$ are designed for forward and
backward mapping, and two discriminators $D_{X}$ and $D_{Y}$ are devised for
adversarial learning. The objective function is as follows:

(4) $L(G,F,D_{X},D_{Y})=L_{GAN}(G,D_{Y},X,Y)+L_{GAN}(F,D_{X},Y,X)+\lambda
L_{cyc}(G,F)\,,$

where $L_{GAN}$ is the adversarial loss, and $L_{cyc}$ is the cycle-
consistency loss.

####  2.2.5. WGAN and WGAN-GP

The GANs training objective can be non-continuous with respect to the
generator parameters, resulting in training instability. To alleviate this
issue, Wasserstein GAN (WGAN) (Arjovsky et al., 2017) was proposed to
leveraged the Wasserstein distance as its objective function. To further
improve the training stability and generation quality, (Gulrajani et al.,
2017) designed WGAN-GP, a GAN variant by adding the gradient penalty in the
WGAN training loss. The WGAN-GP objective function is as follows:

(5) $L=\mathbb{E}_{\tilde{{\mathbf{x}}}\sim
p_{g}}[D(\tilde{{\mathbf{x}}})]-\mathbb{E}_{{\mathbf{x}}\sim
p_{data}}[D({\mathbf{x}})]+\lambda\mathbb{E}_{\hat{{\mathbf{x}}}\sim
p_{\hat{{\mathbf{x}}}}}[(\|\nabla_{\hat{{\mathbf{x}}}}D(\hat{{\mathbf{x}}})\|_{2}-1)^{2}]\,,$

where $\hat{{\mathbf{x}}}$ are random samples used to calculate the gradient.

####  2.2.6. StyleGAN and StyleGAN2

A style-based generative model, such as StyleGAN (Karras et al., 2019), uses a
mapping network to produce style vectors for each convolution layer of the
generator. In StyleGAN, the Adaptive Instance Normalization (AdaIN) operation
takes the style vectors as input to control the generation style. With this
method, diverse and high-quality images can be generated with good
interpolation properties. For example, style vectors corresponding to specific
attributes can be jointly utilized to generate images of desired attributes,
such as pose, skin tone, and hair style. However, StyleGAN suffers from the
characteristic artifacts (e.g., water droplet -like artifacts), which are
corrected by the improved StyleGAN2 (Karras et al., 2020) model. By design,
StyleGAN2 designed the modulation and demodulation modules to incorporate
style vectors into the convolution operation. Moreover, it proposed lazy
regularization and alternative network designs (i.e., input/output skips and
residual nets). As a result of its good performance, the StyleGAN2 model has
been used in a wide range of medical imaging methods (Liu et al., 2022; Hong
et al., 2021).

###  2.3. Autoencoders

The concept of autoencoder dates back to early stages of neural networks
(Hinton, 1994; Bourlard and Kamp, 1988; LeCun, 1987). The original purpose of
AEs was to perform dimensionality reduction or feature learning in an
unsupervised manner. An autoencoder is composed of an encoder $E$ with
parameter $\phi$ and a decoder $D$ with parameter $\theta$. Typically, an
autoencoder has a bottleneck structure (a.k.a undercomplete), i.e., the latent
embedding dimension of the encoder is smaller than the original data $x$. With
this design, the autoencoder can capture intrinsic features of the data. The
training of an autoencoder can be achieved by minimizing the reconstruction
error (e.g., mean squared error) as follow:

(6) $\min_{\theta,\,\phi}\mathbb{E}_{x}\,\|x-D_{\theta}(E_{\phi}(x))\|^{2}\,.$

With the recent emergence of deep neural networks, a multitude of AE model
variants have been developed, such as Denoising Autoencoder (DAE) (Vincent et
al., 2010, 2008), VAE (Kingma and Welling, 2013), conditional VAE (Sohn et
al., 2015; Ramchandran et al., 2022), VQVAE (Van Den Oord et al., 2017), and
U-Net (Ronneberger et al., 2015). We cover below the most relevant AE models
for the synthetic generation of 3D medical volume images.

####  2.3.1. Variational Autoencoder (VAE)

Synthetic image generation was not the primary goal of the original
autoencoder, which instead focused on feature learning or dimensionality
reduction. The VAE (Kingma and Welling, 2013) introduced a new generative
model by combining the AE structure and the variational inference technique.
As shown in Fig. 5, the input data ${\mathbf{x}}$ is encoded to a Gaussian
distribution $\mathcal{N}({\mathbf{x}};{\mu}_{x},{\sigma}_{x}^{2})$ with the
Encoder $E_{\phi}({\mathbf{x}})$. Then, a variable ${\mathbf{z}}$ is sampled
from the Gaussian ${\mathbf{z}}\sim\mathcal{N}({\mu}_{x},{\sigma}_{x}^{2})$.
Finally, the decoder $D_{\theta}(z)$ takes ${\mathbf{z}}$ to generate the
reconstructed data $\tilde{{\mathbf{x}}}$. The intuition behind the VAE is to
encode an input as a distribution instead of a simple point. With proper
regularization of the distribution, the latent space of the VAE becomes more
continuous and complete, and can therefore be conveniently sampled to improve
the quality of the data generation.

Figure 5. The structure of Variational Autoencoder (VAE).

For effective training of the VAE model, the following Evidence lower bound
(ELBO) is optimized:

(7) $\log
p_{\theta}({\mathbf{x}})\geq\mathcal{L}(\theta,\phi;{\mathbf{x}})=-D_{KL}(q_{\phi}({\mathbf{z}}|{\bm{x}})\|p_{\theta}({\mathbf{z}}))+\mathbb{E}_{q_{\phi}({\mathbf{z}}|{\mathbf{x}})}[\log
p_{\theta}({\mathbf{x}}|{\mathbf{z}})]\,,$

where $q_{\phi}({\mathbf{z}}|{\mathbf{x}})$ is a model used to approximate the
intractable posterior distribution $p_{\theta}({\mathbf{z}}|{\mathbf{x}})$,
$D_{KL}$ is the Kullback–Leibler divergence between two distributions to
measure the divergence between two distributions, and
$p_{\theta}({\mathbf{x}}|{\mathbf{z}})$ is the reconstruction probability of
${\mathbf{x}}$ given ${\mathbf{z}}$. In a deep VAE framework (Fig. 5),
$q_{\phi}({\mathbf{z}}|{\mathbf{x}})$ is modeled by an Encoder
$E_{\phi}({\mathbf{x}})$ and $p_{\theta}({\mathbf{z}}|{\mathbf{x}})$ is
modeled by a Decoder $D_{\theta}({\mathbf{z}})$. $p_{\theta}({\mathbf{z}})$ is
set to an isotropic Gaussian to regularize the posterior distribution.
Moreover, the reparameterization trick is used to facilitate the model
training and the gradient estimation:
${\mathbf{z}}={\mu}_{x}+{\sigma}_{x}\epsilon,\epsilon\sim\mathcal{N}(0,1)$.

####  2.3.2. Conditional VAE (CVAE)

Similar to the Conditional GAN, a conditional version of the VAE model was
proposed in (Sohn et al., 2015). The model is not trained to reconstruct the
input ${\mathbf{x}}$. Instead, for the input ${\mathbf{x}}$, the latent
variable ${\mathbf{z}}$ is sampled from
$p_{\theta}({\mathbf{z}}|{\mathbf{x}})$, and then a different output
${\mathbf{y}}$ is generated according to
$p_{\theta}({\mathbf{y}}|{\mathbf{x}},{\mathbf{z}})$ rather than the original
reconstruction $\tilde{{\mathbf{x}}}$. Based on the ELBO of the VAE, the
variational lower bound of this conditional model is as follows:

(8) $\log p_{\theta}({\mathbf{y}}|{\mathbf{x}})\geq-
D_{KL}(q_{\phi}({\mathbf{z}}|{\mathbf{x}},{\mathbf{y}})\|p_{\theta}({\mathbf{z}}|{\mathbf{x}}))+\mathbb{E}_{q_{\phi}({\mathbf{z}}|{\mathbf{x}},{\mathbf{y}})}[\log
p_{\theta}({\mathbf{y}}|{\mathbf{x}},{\mathbf{z}})]\,.$

For a deep CVAE framework, $q_{\phi}({\mathbf{z}}|{\mathbf{x}},{\mathbf{y}})$
is modeled as a recognition network, $p_{\theta}({\mathbf{z}}|{\mathbf{x}})$
is modeled as a conditional prior network, and
$p_{\theta}({\mathbf{y}}|{\mathbf{x}},{\mathbf{z}})$ is modeled as the
generation network. All these networks can be instantiated as Multi-Layer
Perceptrons (MLPs) or Convolution Neural Networks (CNNs).

Various applications can be derived from different choices of ${\mathbf{x}}$
and ${\mathbf{y}}$. For example, if ${\mathbf{x}}$ represents the original
image and ${\mathbf{y}}$ represents the segmentation mask, then we are dealing
with an image segmentation task. If ${\mathbf{x}}$ represents an MRI image and
${\mathbf{y}}$ represents a CT image, then we are dealing with an MRI-to-CT
translation task.

####  2.3.3. U-Net

U-Net was first proposed in (Ronneberger et al., 2015) for biomedical image
segmentation. It has an encoder-decoder architecture, as shown in Figure 6.
Specifically, the encoder converts an image $X$ into a lower-dimension
representation. Then, the decoder takes these representations as inputs and
copies several intermediate feature maps from the encoder to generate the
segmentation mask $Y$. Due to its excellent performance, U-Net has been
widely-used in medical applications besides segmentation, including cross-
modality synthesis (El-Rewaidy et al., 2020), registration (Zhao et al., 2019;
Balakrishnan et al., 2019; Zhang et al., 2020), etc. For 3D medical
applications, the 3D U-Net (Çiçek et al., 2016) was designed by using 3D
convolution to process the 3D volume images. Moreover, the U-Net design can
also be used as the main structure of other generative models, e.g., the U-Net
generator in GANs.

Figure 6. A schematic view of the U-Net architecture. Several intermediate
feature maps from the encoder are copied and concatenated in the decoder.

####  2.3.4. Denoising Autoencoder (DAE)

DAE (Vincent et al., 2010) is an autoencoder that takes in the corrupted data
at the input to reconstruct the original uncorrupted data. A noise function
$T:\mathcal{X}\rightarrow\mathcal{X}$ is applied on the data $x\in\mathcal{X}$
of a standard autoencoder to construct a DAE. The objective of a DAE is to
reconstruct the original data $X$ from the noisy input $T(x)$. Thus, the
following optimization problem can be defined:

(9) $\min_{\theta,\,\phi}\mathbb{E}_{x}\,\|x-(D_{\phi}\circ E_{\theta}\circ
T)(x)\|^{2}\,.$

In a similar way to the bottleneck design, the denoising training strategy can
facilitate the learning of the inherent features of the original data. Besides
the manually designed noise function $T$, certain medical image collection and
processing devices can produce noisy and low-quality medical images, e.g.,
low-dose CT image. In this case, the idea of DAE can be applied to the medical
image denoising task (Gondara, 2016; Ahmed et al., 2021).

####  2.3.5. Denoising Diffusion Probabilistic Model (DDPM)

Inspired by non-equilibrium thermodynamics (Sohl-Dickstein et al., 2015),
diffusion models devise a Markov chain of diffusion steps by gradually adding
noise to the data (forward) and learning how to reverse the diffusion process
(reverse) so that an image can be generated from the noise. Recent research
has demonstrated that Denoising Diffusion Probabilistic Models (DDPMs) (Ho et
al., 2020) outperform GANs in terms of natural image synthesis (Dhariwal and
Nichol, 2021; Ho et al., 2022). Given data sample ${\mathbf{x}}_{0}\sim
q({\mathbf{x}}_{0})$, the forward diffusion process is defined as:
$q({\mathbf{x}}_{t}|{\mathbf{x}}_{t-1})=\mathcal{N}({\mathbf{x}}_{t};\sqrt{1-\beta_{t}}{\mathbf{x}}_{t-1},\beta_{t}{\mathbf{I}});q({\mathbf{x}}_{1:T}|{\mathbf{x}}_{0})=\prod_{t=1}^{T}q({\mathbf{x}}_{t}|{\mathbf{x}}_{t-1})$,
where $\beta_{t}$ controls the noise schedule such that
$q({\mathbf{x}}_{T}|{\mathbf{x}}_{0})\approx\mathcal{N}({\mathbf{x}}_{T};\mathbf{0},{\mathbf{I}}).$
In other words, the forward process converts a data sample into a zero-mean
isotropic Gaussian. The reverse denoising process is defined as:
$p({\mathbf{x}}_{T})=\mathcal{N}({\mathbf{x}}_{T};\mathbf{0},{\mathbf{I}});$
$p_{\theta}({\mathbf{x}}_{t-1}|{\mathbf{x}}_{t})=\mathcal{N}({\mathbf{x}}_{t-1};\mu_{\theta}({\mathbf{x}}_{t},t),\sigma_{t}^{2}{\mathbf{I}})\,,$
where $\mu_{\theta}({\mathbf{x}}_{t},t)$ is a trainable network to estimate
the mean and $\sigma_{t}^{2}$ is the variance schedule. The reverse process
generates realistic data from Gaussian noise.

Similar to VAE, the training of DDPM model uses the variational upper bound to
construct the following objective function:

(10)
$L=\mathbb{E}_{q}\big{[}\underbrace{D_{KL}(q({\mathbf{x}}_{T}|{\mathbf{x}}_{0})\|p({\mathbf{x}}_{T}))}_{L_{T}}+\sum_{t>1}\underbrace{D_{KL}(q({\mathbf{x}}_{t-1}|{\mathbf{x}}_{t},{\mathbf{x}}_{0})\|p_{\theta}({\mathbf{x}}_{t-1}|{\mathbf{x}}_{t}))}_{L_{t-1}}-\underbrace{\log
p_{\theta}({\mathbf{x}}_{0}|{\mathbf{x}}_{1}))}_{L_{0}}\big{]}\,.$

To improve the training of the objective function, a simplified version of
$L_{t-1}$ is proposed in (Ho et al., 2020):
$L_{\text{simple}}=\mathbb{E}_{{\mathbf{x}}_{0}\sim
q({\mathbf{x}}_{0}),\epsilon\sim\mathcal{N}(\mathbf{0},{\mathbf{I}}),t\sim\mathcal{U}(1,T)}[\|\epsilon-\epsilon_{\theta}(\sqrt{\bar{\alpha}_{t}}{\mathbf{x}}_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon,t)\|^{2}]$.
Here, $\epsilon_{\theta}$ is a function approximator to predict $\epsilon$
from ${\mathbf{x}}_{t}$, and $\bar{\alpha}_{t}=\prod_{s=1}^{t}(1-\beta_{s})$.
$L_{\text{simple}}$ achieved better generation quality on natural image
benchmarks.

In practice, the approximation functions $\mu_{\theta}({\mathbf{x}}_{t},t)$ or
$\epsilon_{\theta}$ can be implemented with U-net or DAE structures. Due to
the good generation performance, DDPM has been adopted in several recent
medical imaging papers (Kim and Ye, 2022; Pinaya et al., 2022a).

###  2.4. Autoregressive Models

The hallmark problem of generative models is to model the distribution of
images. GANs apply an adversarial strategy to indirectly approach the real
distribution, while VAE employs variational inference to approximate it. In
contrast, Autoregressive models (Gregor et al., 2014; Van Oord et al., 2016)
explicitly compute the joint distribution of pixels by factorizing it into
products of conditional distributions. As shown in Fig. 7, an $n\times n$
image is represented by a sequence of pixels $x_{1},x_{2},\dots,x_{n^{2}}$
according to the raster scan order111Rater scan order: row by row and pixel by
pixel within every row. The distribution of the forthcoming pixel $x_{i}$
depends on the above and left surrounding pixels, i.e.,
$p(x_{i}|x_{1},\dots,x_{i-1})$. In order to calculate the distribution of the
whole image, the chain rule is applied:
$p(x)=\prod_{i=1}^{n^{2}}p(x_{i}|x_{1},\dots,x_{i-1})$. For a color image,
each pixel $x_{i}$ is determined by three channels: Red, Green, and Blue. This
is followed by the introduction of the successive dependency:
$p(x_{i}|{\mathbf{x}}_{<i})=p(x_{i,R}|{\mathbf{x}}_{<i})p(x_{i,G}|{\mathbf{x}}_{<i},x_{i,R})p(x_{i,B}|{\mathbf{x}}_{<i},x_{i,R},x_{i,G})$.

Figure 7. A schematic illustration of an autoregressive model. An image
distribution is modeled as the distribution of all pixels, where each pixel is
conditioned on the left and above surrounding pixels.

Autoregressive models calculate the tractable image distribution in an
explicit and straightforward manner by maximizing the log likelihood function.
However, due to the factorization of multiple product terms, the direct
computation is inefficient. Thus, a variety of autoregressive models (Gregor
et al., 2014; Van Oord et al., 2016; Van den Oord et al., 2016; Esser et al.,
2021) have been proposed to enable fast and parallel computation. Below, we
introduce several methods that are relevant to the synthetic generation of 3D
medical volume images.

####  2.4.1. PixelRNN and PixelCNN

In order to improve the expressive power and efficiency of early
autoregressive models such as NADE (Larochelle and Murray, 2011), the work in
(Van Oord et al., 2016) used Recurrent Neural Networks (RNNs) to construct the
PixelRNN model for large-scale natural image generation. PixelRNN is designed
to stack multiple 2D Long Short-Term Memory (LSTM) layers. In each layer,
there is a fixed input-to-state component and a recurrent state-to-state
component to jointly compute the states and gates of an LSTM. Specifically,
two types of LSTM layers are designed to implement the state-to-state
components: Row LSTM and Diagonal BiLSTM. The Row LSTM processes the image row
by row with a $k\times 1$ convolution, while the Diagonal BiLSTM skews each
input row by one position and processes with a $2\times 1$ convolution.
Residual connections (He et al., 2016) are deployed to facilitate deep
PixelRNN training.

The Row LSTM and the Diagonal BiLSTM have the benefit of potentially unbounded
dependency range to model the global pixel correlations. However, they
introduce large computation costs. Therefore, another variant (called
PixelCNN) using standard convolution layers along with masks was proposed to
enable a fast and parallel training process. The speedup of PixelCNN, however,
degrades the generation performance when compared with the PixelRNN model.

####  2.4.2. Autoregressive Transformer

Compared with RNNs and CNNs, Transformers (Vaswani et al., 2017) have shown
superior performance in modeling the long-range interactions within sequences.
This inspired the Autoregressive Transformer proposed in (Esser et al., 2021).
Specifically, a Vector Quantised GAN (VQGAN) was designed to learn a
representative codebook from the patches of the training images. The
composition of the codes in an image was then modeled using an autoregressive
transformer. Finally, image generation was formulated as the autoregressive
prediction of the next image patch, similar to PixelRNN. Conditional synthesis
was also achieved in (Esser et al., 2021) by enforcing the conditional
information in the autoregressive transformer.

###  2.5. Challenges of 3D Generative Models

The aforementioned generative models have been applied to various two-
dimensional (2D) tasks, including natural image generation (Karras et al.,
2019; Karras et al., 2020), image style transfer (Isola et al., 2017; Zhu et
al., 2017), text-to-image generation (Reed et al., 2016), medical slice
synthesis (AlAmir and AlGhamdi, 2022), 2D cross-modality synthesis (Sikka et
al., 2021; Zhao et al., 2021a), etc. However, as a result of the intrinsic 3D
nature of medical volume images, there are practical challenges that are
beyond the design and training of 2D generative models.

Insufficient 3D medical volumes for training. A 2D generative model relies on
a large number of readily accessible images (e.g., natural images, slices of
medical volume images) in order to model real data distributions. For example,
the widely-used ImageNet (Deng et al., 2009) dataset contains 1.28 million
natural images. In the medical field, a big dataset for training can be
created by compiling all the 2D slices of a 3D volume. Comparatively, 3D
medical datasets are several orders of magnitude smaller. 3DSeg-8 (Chen et
al., 2019) consists of eight medical datasets, with each dataset containing
only tens to hundreds of volume images. For the training of generative models,
(Kwon et al., 2019) utilized only 991 brain T1-weighted image volumes from
ADNI (Weiner et al., 2017).

3D model parameters. 3D generative models usually utilize 3D convolution
architectures, such as 3D-U-Net (Çiçek et al., 2016). While 3D convolution
architectures are capable of extracting feature representations pertaining to
3D anatomical structures, they also require significantly more parameters
compared to their 2D counterparts. Consequently, a number of problems will
occur: (1) the deployment and training of 3D models are slow, (2) 3D models
consume a lot of GPU memory, which limits the training batch size, (3) the
lack of sufficient 3D volume images for training large 3D models leads to
overfitting.

Ultra-high dimension of the 3D volumes. In contrast to 2D images, 3D volume
images have a very high dimension, which makes modelling the 3D volume
distribution extremely difficult. In practice, it is infeasible to design a 3D
generative model that captures the 3D volume distribution exactly. As a
result, approximation techniques such as variational inference or distribution
factorization need to be adopted. The resolution of synthesized images is
usually limited to small values, such as $64\times 64\times 64$ or $128\times
128\times 128$, in order to reduce the data dimension. In contrast, 2D image
synthesis can use resolutions of up to $1024\times 1024$.

The existing 3D medical generative models attempt to address one or more of
the above challenges. In the next section, we will discuss how different
models address these issues with applications to the synthesis of the brain
and heart volume images.

##  3\. 3D Brain and Heart Applications of Generative Models

Various 3D medical applications, especially those related to the brain and
heart, can be implemented using the aforementioned model architectures. In
this section. we first examine the applications of unconditional generative
models, i.e., unconditional synthesis and classification. Unconditional
generation is the basic task performed by various generative models (e.g.,
vanilla GAN, VAE, and diffusion model), which maps a simple distribution
(e.g., Uniform or Gaussian) to capture the structure of the real data
distribution. Afterwards, we discuss the applications of conditional
generative models, such as conditional synthesis, segmentation, denoising,
detection, and registration. The conditional generation process transforms the
data distributions between different modalities, e.g., from image to
segmentation mask, from MRI to CT image, and from low-dose CT to normal-dose
CT.

In each of the following subsections, we describe the application background,
elaborate on the methods for brain and heart applications, and provide a
summary table showing the investigated organ, the model, the datasets, and the
performance metrics.

###  3.1. Unconditional Synthesis

Table 1. Summary of publications on 3D Unconditional Synthesis related to the
brain or heart.

Publication (Year) | Organ | Model | Dataset | Metrics
---|---|---|---|---
3D-$\alpha$-WGAN-GP (Kwon et al., 2019) (2019) | Brain |  $\alpha$-GAN (Rosca et al., 2017) | ADNI (Weiner et al., 2017), BRATS 2018 (Bakas et al., 2017) | MMD, MS-SSIM
|  | WGAN-GP (Gulrajani et al., 2017) | ATLAS (Liew et al., 2018) |
SlabGAN (Granstedt et al., 2021) (2019) | Brain | ProgressiveGAN (Karras et al., 2017) | fastMRI (Zbontar et al., 2018) | FID
Özbey _et al_. (Özbey et al., 2020) (2020) | Brain | GAN (Goodfellow et al., 2014) | IXI (Dataset, 2010) | PSNR, FID
2D Slice VAE (Volokitin et al., 2020) (2020) | Brain | VAE (Kingma and Welling, 2013) | HCP (Van Essen et al., 2013) | RAS, MMD, MS-SSIM
3D StyleGAN (Hong et al., 2021) (2021) | Brain | StyleGAN2 (Karras et al., 2020) | ADNI (Weiner et al., 2017), ABIDE (Di Martino et al., 2014), HABS (Dagley et al., 2017) | bMMD2, MS-SSIM, FID
|  |  | OASIS (Marcus et al., 2007), MCIC (Gollub et al., 2013), PPMI (Marek et al., 2011) |
|  |  | ADHD (consortium, 2012), Harvard, GSP (Holmes et al., 2015) |
DCR-$\alpha$-GAN (Segato et al., 2020) (2021) | Brain |  $\alpha$-GAN (Rosca et al., 2017) | ADNI (Weiner et al., 2017) | MMD, MS-SSIM, IoU
Shape+Texture GAN (Chong and Ho, 2021) (2021) | Brain | WGAN-GP (Gulrajani et al., 2017) | HCP (Van Essen et al., 2013), ADNI (Weiner et al., 2017) | MMD, MS-SSIM
Split&Shuffle-GAN (Liu et al., 2022) (2022) | Brain | StyleGAN2 (Karras et al., 2020) | COCA (AIMI, 2022), ADNI (Weiner et al., 2017) | FID
| Heart |  |  |
HA-GAN (Sun et al., 2022) (2022) | Brain | GAN (Goodfellow et al., 2014), AE (Hinton, 1994) | Harvard GSP (Holmes et al., 2015) | FID, MMD, IS
DDM (Kim and Ye, 2022) (2022) | Heart | DDPM (Ho et al., 2020) | ACDC (Bernard et al., 2018) | PSNR, NMSE, Dice

The most straightforward application of generative models for 3D medical
imaging is unconditional synthesis. By capturing the real data distributions
of the 3D medical volumes, high-quality synthesized images can be generated
for downstream tasks such as tumor prediction (Elazab et al., 2020) and
Alzheimer’s disease diagnosis (Kruthika et al., 2019; Lin et al., 2021). By
utilizing simulated images, the data-deficit and imbalance issues (Hu et al.,
2020) caused by the limitations of medical image collection (e.g., privacy,
ethics, required data annotation expertise, financial, etc.) can be addressed.
3D medical imaging relies more on image synthesis than 2D medical imaging. The
number of 3D volume images in existing medical datasets ranges from hundreds
(Chen et al., 2019) to thousands (Hong et al., 2021), which is not enough to
train a reliable deep diagnosis model for clinical use. With the rapid
development of generative models for natural images, a number of successful
architectures have been modified for 3D medical volume synthesis, e.g., VAE
(Kingma and Welling, 2013), Progressive GAN (Karras et al., 2017), StyleGAN
(Karras et al., 2020), $\alpha$-GAN, etc. There are a number of publications
that describe unconditional synthesis, as shown in Table 1.

####  3.1.1. AE and VAE

AE and VAE structures are commonly-used in medical generative models, either
alone or in combination with GANs. Based on a combined two-dimensional slice-
level VAE and slice correlation model, 2D Slice VAE (Volokitin et al., 2020)
generated 3D MRI brain volumes. The VAE was simply trained on MRI brain slices
to form a latent space, from which the latent code was sampled for slice
generation. To ensure completeness and consistency of the slices in the
generated volumes, the means and covariances of each dimension of the latent
code were separately estimated from the training slices. Moreover, (Volokitin
et al., 2020) proposed a Realistic Atlas Score (RAS) based on the affine-
registered segmentations of the generated volumes as an evaluation metric. As
a generalization of the VAE model, DDPM (Ho et al., 2020) was used in DDM (Kim
and Ye, 2022) for the task of 4-Dimensional (3D+t) temporal cardiac MRI
generation. A Diffusion module was trained to estimate the latent code $c$,
which was then leveraged by the Deformation module to generate deformed images
with the spatial transformation layer (Jaderberg et al., 2015).
3D-$\alpha$-WGAN-GP (Kwon et al., 2019) adapted the $\alpha$-GAN structure,
which combines both the VAE and GAN into a single model to generate 3D brain
MRI volumes from random vectors. The architecture consists of a Generator, a
Discriminator, an Encoder, and a Code Discriminator. To improve the training
stability, the model was trained with the WGAN-GP loss (Gulrajani et al.,
2017) and the reconstruction loss. The integration of VAE and GAN
simultaneously addressed the mode collapse and image blurriness problems.
Similarly, the $\alpha$-GAN structure was also employed in DCR-$\alpha$-GAN
(Segato et al., 2020), which integrated a Refiner Network with four ResNet
blocks after each generator to refine the generation process and produce more
realistic images.

####  3.1.2. GANs

In GANs, the progressive growing strategy (Karras et al., 2017) has been
verified to improve the quality and stability of image generation. SlabGAN
(Granstedt et al., 2021) directly extended the 2D ProgressiveGAN (Karras et
al., 2017) for the 3D generation of MRI brain images. Specifically, SlabGAN
replaced the original $4\times 4$ constant with $4\times 4\times 4$ to finally
generate volumes of $256\times 256\times 16$ resolution. HA-GAN (Sun et al.,
2022) designed a hierarchical generation structure containing two branches:
the full volume low-resolution generation branch and the sub-volume high-
resolution generation branch. Each branch consists of a GAN network and an AE
network with a shared decoder/generator. The two branches also share the low-
resolution generation block to facilitate progressive generation. Özbey _et
al_. (Özbey et al., 2020) progressively generated 3D volumes with three
sequential GAN models. An Axial GAN $G_{A}$ generated MRI volumes by
concatenating cross-sections (Dar et al., 2019) in the axial view. Taking the
generated volumes as inputs, a Coronal GAN $G_{C}$ was trained to enhance the
quality of the coronal view. Similarly, a Sagittal GAN $G_{s}$ enhanced the
previously generated volumes in the sagittal direction. The Shape+Texture GAN
(Chong and Ho, 2021) incorporated the geometric deformation technique (Lee et
al., 2022) in two GANs: Shape Network and Texture Network. Shape Network was
trained to generate the 3D deformation field from a random vector. In order to
obtain synthetic images, the generated 3D fields were applied to the Montreal
Neurological Institute (MNI) brain template to warp it. Both deformed and real
images were utilized to train the Texture Network composed of a U-Net
(Ronneberger et al., 2015) and a patch-GAN (Li and Wand, 2016) discriminator.

####  3.1.3. StyleGAN

As a result of the style-based generation technique (Karras et al., 2019;
Karras et al., 2020), StyleGAN (Karras et al., 2019) no longer requires the
progressive generation strategy (Karras et al., 2017). 3D StyleGAN (Hong et
al., 2021) extended the state-of-the-art 2D StyleGAN2 (Karras et al., 2020)
architecture for 3D medical images generation. Specifically, the mapping
network for generating each layer’s style vectors remained unchanged from 2D
to 3D. The generator was lifted to 3D with 3D convolution operations and the
$4\times 4$ constant input was replaced with $5\times 6\times 7$ to fit the 3D
brain volumes. The discriminator was also lifted with 3D convolutions. A total
of $7,392$ image volumes were collected from eight datasets to train the
model. The Split&Shuffle-GAN (Liu et al., 2022) addressed the training
challenges of the 3D StyleGAN from the perspective of the training strategy
and network architecture. First, a 2D CNN was trained, then five strategies
were proposed to inflate the 2D weights to 3D. The Channel Split&Shuffle
modules were respectively devised for the discriminator and the generator to
significantly reduce the number of parameters. Both the brain and heart
synthetic volume images were generated in (Liu et al., 2022).

####  3.1.4. Quality Assessment

In evaluating the quality of 3D synthetic volume images, it is important to
consider the 3D anatomical structure, which is not evident in a 2D image. As a
result, (Treder et al., 2022) assessed various image quality metrics on MRI
images generated by GANs, and proposed a Deep Quality Assessment (QA) model
that was trained according to the human ratings of MRI images. Among all,
Fréchet Inception Distance (FID) (Heusel et al., 2017), Naturalness Image
Quality Evaluator (NIQE) (Mittal et al., 2012), and Maximum Mean Discrepancy
(MMD) (Borji, 2019) showed good sensitivity to low-quality images. Deep
Quality Assessment managed to distinguish between high-quality images
exhibiting subtle variations.

###  3.2. Classification

Table 2. Summary of publications on 3D Classification related to the brain or
heart.

Publication (Year) | Organ | Model | Dataset | Metrics
---|---|---|---|---
SCGANs (Zhang et al., 2017) (2017) | Heart | GAN (Goodfellow et al., 2014) | UK Biobank (Sudlow et al., 2015) | ACC, Pre, Rec
Biffi _et al_. (Biffi et al., 2018) (2018) | Heart | VAE (Kingma and Welling, 2013) | Multi-centre (Biffi et al., 2018), ACDC (Bernard et al., 2018) | ACC
3D-CapsNet (Kruthika et al., 2019) (2019) | Brain | AE (Hinton, 1994) | ADNI (Weiner et al., 2017) | ACC, ROC
3DPixelCNN (Pombo et al., 2019) (2019) | Brain | PixelCNN (Van Oord et al., 2016) | DWI (Pombo et al., 2019), GM (Pombo et al., 2019) | Dice, ACC, MSE
Puyol-Antón _et al_. (Puyol-Antón et al., 2020) (2020) | Heart | VAE (Kingma and Welling, 2013) | Biobank (Sudlow et al., 2015) | Balanced ACC, Sen, Spe
ACC: accuracy. Pre: precision. Rec: recall. Sen: sensitivity. Spe:
specificity.

The classification of medical images has a multitude of applications,
including Alzheimer’s disease detection (Kruthika et al., 2019),
cardiovascular diseases diagnosis (Biffi et al., 2018), age/sex prediction
(Pombo et al., 2021, 2019), etc. For the highest level of accuracy, these
applications rely on anatomical structures within the 3D medical volume. In
order to apply deep learning techniques for classification of 3D medical
volumes, 3D CNNs are commonly used (Ji et al., 2012; Singh et al., 2020; Chen
et al., 2019; Niyas et al., 2022). Two problems arise when 3D CNNs are used
for 3D volume classification: insufficient training data, and unreliable
results explanations. Synthesizing volumes (either unconditionally or
conditionally) to augment the training data can address the issue of
insufficient data. Unreliable explanations need to be addressed in order to
facilitate deep learning approaches for clinical use, which can be
accomplished by interpreting the classifier or learning meaningful latent
spaces. A list of generative models for the 3D classification task is shown in
Table 2.

SCGANs (Zhang et al., 2017) tackled the problem of classifying missing apical
slices (MAS) and missing basal slices (MBS). A Semi-Coupled GANs was designed
to include two generators one for positive and one for negative data
generation. A shared discriminator was designed to perform two classification
tasks: real/fake and positive/negative. After GANs training, the MAS/MBS
classifier was trained on both generated and real data. Puyol-Antón _et al_.
(Puyol-Antón et al., 2020) performed interpretable Cardiac Resynchronisation
Therapy response prediction using a VAE and several classifiers. The volume
data was constructed by temporally sampling $T=25$ points to extract 2D
echocardiography images. VAE processed the slice-level segmentations and
concatenated all the $T$ latent codes. The primary classifier and the multiple
concept classifiers used the latent codes for prediction and interpretation,
respectively. In order to accurately detect Alzheimer’s disease in its early
stages with small-scale datasets, 3D-CapsNet (Kruthika et al., 2019) was
designed to perform Content Based Image Retrieval (CBIR). The model consisted
of a pre-trained sparse Autoencoder, a 3D Capsule Network (Sabour et al.,
2017), and a 3D Convolutional Neural Network (Ji et al., 2012). 3DPixelCNN
(Pombo et al., 2019) directly extended the PixelCNN (Van Oord et al., 2016)
model to handle 3D brain MRI volumes. Moreover, by applying SpatialDropout
(Tompson et al., 2015), the 3DPixelCNN became a Bayesian model capable of
performing uncertainty estimating (Gal and Ghahramani, 2016). The following
three semi-supervised tasks were evaluated: semantic segmentation of lesions,
age regression, and sex classification. Biffi _et al_. (Biffi et al., 2018)
proposed an interpretable model that can be used to automatically classify
cardiac diseases caused by structural remodeling, e.g., hypertrophic
cardiomyopathy. The segmentation masks from end-diastolic and end-systolic
phases were input to a VAE model to learn meaningful latent distributions.
Mean vectors of the latent distributions were used to train an MLP classifier.
The partial derivative of the class labels w.r.t the mean vector was
iteratively updated so as to generate interpretable visualization results.

###  3.3. Conditional Synthesis

Table 3. Summary of publications on 3D Conditional Synthesis related to the
brain or heart.

Publication (Year) | Organ | Model | Dataset | Metrics
---|---|---|---|---
PAN  _et al_. (Pan et al., 2018) (2018) | Brain | CycleGAN (Zhu et al., 2017) | ADNI (Weiner et al., 2017) | ACC, Sen, Spe, PSNR
|  |  |  | F1-score, AUC, MCC
3D cGAN (Yu et al., 2018) (2018) | Brain | cGAN (Mirza and Osindero, 2014) | BRATS (Bakas et al., 2017) | PSNR, NMSE, Dice
Liao  _et al_. (Liao et al., 2018) (2018) | Heart | GAN (Goodfellow et al., 2014) | Own (Liao et al., 2018) | Dice, ASSD
Joyce _et al_. (Joyce and Kozerke, 2019) (2019) | Heart | VAE (Kingma and Welling, 2013) | ACDC (Bernard et al., 2018) | Dice
Ea-GANs (Yu et al., 2019) (2019) | Brain | GAN (Goodfellow et al., 2014) | IXI (Dataset, 2010), BRATS (Bakas et al., 2017) | PSNR, NMSE, SSIM
Zhang _et al_. (Hongtao et al., 2020) (2020) | Brain | ESRGAN (Wang et al., 2018b) | IXI (Dataset, 2010) | PSNR, SSIM
dEa-SA-GAN (Yu et al., 2020) (2020) | Brain | GAN (Goodfellow et al., 2014), AE (Hinton, 1994) | BRATS (Bakas et al., 2017), SISS2015 (Maier et al., 2017) | PSNR, NMSE, SSIM
3D-RevGAN (Lin, 2020) (2020) | Brain | RevGAN (van der Ouderaa and Worrall, 2019) | ADNI (Weiner et al., 2017) | RMSE, PSNR, SSIM
|  |  |  | ACC, Sen, Spe, AUC
QSMGAN (Chen et al., 2020) (2020) | Brain | WGAN-GP (Gulrajani et al., 2017) | Own (Chen et al., 2020) | L1, PSNR, NMSW
|  |  |  | HFEN, SSIM
XCAT-GAN (Amirrajab et al., 2020) (2020) | Heart | cGAN (Mirza and Osindero, 2014) | ACDC (Bernard et al., 2018), SCD (Radau et al., 2009), York (Andreopoulos and Tsotsos, 2008) | DSC, HD
Mcmt-gan (Huang et al., 2020) (2020) | Brain | GAN (Goodfellow et al., 2014) | IXI (Dataset, 2010), NAMIC (Multimodality, 2022) | PSNR, SSIM, Dice
$\mathbb{C}$Net (El-Rewaidy et al., 2020) (2020) | Heart | U-Net (Ronneberger et al., 2015) | Own (El-Rewaidy et al., 2020) | MSE, SSIM
Lin  _et al_. (Lin et al., 2021) (2021) | Brain | RevGAN (van der Ouderaa and Worrall, 2019) | ADNI (Weiner et al., 2017) | PSNR, SSIM, Accuracy
|  |  |  | Sen, Spe, AUC
CAE-ACGAN (Yang et al., 2021a) (2021) | Brain | CVAE (Sohn et al., 2015), ACGAN (Odena et al., 2017) | Own (Yang et al., 2021a) | PSNR, SSIM, MAE
CACR-Net (Peng et al., 2021) (2021) | Brain | GAN (Goodfellow et al., 2014) | BRATS (Bakas et al., 2017) | NMSE, SSIM, PSNR
ProvoGAN (Yurt et al., 2022) (2022) | Brain | GAN (Goodfellow et al., 2014) | In vivo Brain (Yurt et al., 2022), IXI (Dataset, 2010) | PSNR, SSIM
ResViT (Dalmaz et al., 2021) (2022) | Brain | AE (Hinton, 1994) | IXI (Dataset, 2010), BRATS (Bakas et al., 2017) | PSNR, SSIM
Subramaniam  _et al_. (Subramaniam et al., 2022) (2022) | Brain | WGAN (Gulrajani et al., 2017) | PEGASUS (Mutke et al., 2014), 1000Plus (Hotter et al., 2009) | FID, PRD, DSC, bAVD
CounterSynth (Pombo et al., 2021) (2022) | Brain | StarGAN (Choi et al., 2018) | UK Biobank (Sudlow et al., 2015) | ACC, MAE, FID
Own: the dataset was originally collected and processed by the authors of the
paper.
ACC: accuracy. Sen: sensitivity. Spe: specificity. PRD: precision and recall
of distributions.
bAVD: balanced average Hausdorff distance. DSC: dice similarity coefficient.
ASSD: average symmetric surface distance. MCC: Matthews correlation
coefficient

In the case of unconditional synthesis, high-quality 3D medical volumes can be
generated solely from random input variables. This limits its applications to
a broader range of tasks such as class-conditional generation (Pombo et al.,
2021; Amirrajab et al., 2020), reconstruction (El-Rewaidy et al., 2020;
Hongtao et al., 2020), and cross-modality synthesis (Pan et al., 2018; Wang et
al., 2018a; Yu et al., 2020; Liao et al., 2018; Sikka et al., 2021; Zhao et
al., 2021a). These tasks can be modeled into the conditional synthesis
framework, where extra information can be conditioned for medical volume
generation (Fig. 3(b)). Depending on the conditional information, a vector or
an image modality can be used, resulting in models like Conditional GAN (Mirza
and Osindero, 2014) or image-to-image translation (Isola et al., 2017). The
formulation of the task and the design of the model depend heavily on how the
conditional information is incorporated, which will be elaborated below. A
list of relevant models for conditional synthesis is provided in Table 3.

####  3.3.1. Class / Attribute Condition.

CounterSynth (Pombo et al., 2021) proposed a counterfactual augmentation
approach to generate biological-plausible 3D brain volumes conditioned on the
demographic attributes. Specifically, the U-Net generator is conditioned on
the real brain image and counterfactual attributes (e.g., age, sex) to produce
the deformation fields, which are then applied to the real brain image to
obtain the counterfactual brain image. The discriminator consists of a
real/counterfactual classifier and an attribute classifier. By conditioning on
label maps instead of vectors, XCAT-GAN (Amirrajab et al., 2020) can
synthesize high-fidelity cardiac MRI images. First, a U-Net (Ronneberger et
al., 2015) was trained to obtain the 4-class or 8-class segmentation masks.
The real images and masks were then fed as inputs to conditional GANs for
image synthesis. The downstream cardiac cavity segmentation task was used to
evaluate the generated data after the GANs training. In order to produce 3D
cardiac synthetic MRI images, Joyce _et al_. (Joyce and Kozerke, 2019)
conditioned the synthesis on a learnt anatomical model. The samples from the
anatomical model and random Gaussian noises were input to the sequential
transform and render modules to generate synthetic volumes. All the model
parameters were trained through self-supervision with a reconstruction error.

####  3.3.2. Cross-modality Synthesis.

Among the early works, 3D cGAN (Yu et al., 2018) extended the 2D image-to-
image translation GANs model (Isola et al., 2017) to synthesize the 3D FLuid-
Attenuated Inversion Recovery (FLAIR) from 3D T1 MRI images. Local adaptive
fusion was applied to improve the FLAIR details. dEa-SA-GAN (Yu et al., 2020)
proposed sample adaptive GAN models to explore both the global space mapping
and the local space mapping. The whole model consists of two paths. The
baseline path is a standard 3D GAN used to learn the global mapping. The
sample-adaptive path uses the neighborhood relationships between the training
samples to force a local sample-specific learning. Two synthesis tasks were
conducted in dEa-SA-GAN: T1$\rightarrow$FLAIR and T1$\rightarrow$T2. CACR-Net
(Peng et al., 2021) designed a confidence-guided aggregation and cross-
modality refinement network for the multi-modality MRI synthesis task. The
Confidence-Guided Aggregation Module uses two U-Net generators to process two
different input modalities. The two outputs are then aggregated and input to
the Cross-Modality Refinement Module for further improvement. Three synthesis
tasks were conducted: T1+T2$\rightarrow$FLAIR, T1+FLAIR$\rightarrow$T2, and
T2+FLAIR$\rightarrow$T1. For CAE-ACGAN (Yang et al., 2021a), only a single CT
image was used to generate multi-contrast MRI images (i.e., fat, R2, and
water). The Conditional Variational AutoEncoder (CVAE) (Sohn et al., 2015) and
ACGAN (Odena et al., 2017) architectures were developed to take advantage of
the capabilities of both VAE and GAN. The structure of CAE-ACGAN consists of
an Encoder, a Generator, a Discriminator, and a Classifier. Mcmt-gan (Huang et
al., 2020) proposed an unsupervised modality transferable framework for 3D
brain image synthesis. To ensure robust and high-fidelity cross-modal
generation, Mcmt-gan introduced three losses (i.e., cycle-consistency loss,
bidirectional adversarial loss, and domain adapted loss) and a volumetric
manifold regularization. Moreover, a fault-aware discriminator was proposed to
make the model coherent with the segmentation task. The cross-modality
synthesis tasks include: Proton Density (PD)$\rightarrow$T2,
T2$\rightarrow$PD, T1$\rightarrow$T2, and T2$\rightarrow$T1. To reflect the
texture details of image structure, Ea-GANs (Yu et al., 2019) devised an edge-
aware generative adversarial network for cross-modality synthesis.
Specifically, the edge information was extracted by the Sobel (Gao et al.,
2010) edge detector and incorporated in the training process of GANs. Two
variants were proposed depending on where the edge information was included in
the objective function: gEa-GAN (generator) and dEa-GAN (generator and
discriminator). Both the T1$\rightarrow$FLAIR and T1$\rightarrow$T2 tasks were
evaluated. Recently, the Transformer (Vaswani et al., 2017) was adopted in
ResViT (Dalmaz et al., 2021) for the first time to conduct multi-modality
synthesis. The generator of ResViT has an encoder-decoder structure with
multiple Aggregated Residual Transformer (ART) blocks between the encoder and
the decoder. The weight-shared ART blocks are able to distill the task-
specific information, such as location and context. Three synthesis tasks were
conducted: T1,T2$\rightarrow$PD, T1,T2$\rightarrow$FLAIR, and
MRI$\rightarrow$CT.

By using cross-modality synthesis, volumetric data for a specific modality can
be completed. Liao  _et al_. (Liao et al., 2018) introduced a cross-modality
volume completion task to assist in the segmentation of IntraCardiac
Echocardiography images. A GAN model named 3D segmentation and completion
network (3D-SCNet) was proposed in (Liao et al., 2018) to perform simultaneous
completion and segmentation. A number of cross-modality completion methods
have been proposed to improve Alzheimer’s disease diagnosis (Pan et al., 2018;
van der Ouderaa and Worrall, 2019; Lin et al., 2021). PAN  _et al_. (Pan et
al., 2018) utilized the CycleGAN (Zhu et al., 2017) model to perform 3D PET
image imputation from MRI and likely incomplete PET images. Then, the imputed
PET and MRI were used by the Landmark-based Multi-Modal Multi-Instance
Learning (LM3IL) for Alzheimer’s disease classification. In 3D-RevGAN (Lin,
2020) and Lin  _et al_. (Lin et al., 2021), the 3D Reversible GAN (van der
Ouderaa and Worrall, 2019) was adopted to complete the missing PET data for
Alzheimer’s Disease diagnosis. Specifically, Lin  _et al_. (Lin et al., 2021)
proposed a two-step method: (1) learn the bidirectional mapping between PET
and MRI images with a 3D Reversible GAN, and (2) train a 3D CNN to classify
the disease. Reversible GAN has the advantage of only using one generator
instead of two in CycleGAN.

There are some other tasks that are related to the cross-modality/multi-
modality synthesis, e.g., quantitative susceptibility mapping (QSM) (Chen et
al., 2020) and Time-of-Flight Magnetic Resonance Angiography (TOF-MRA)
generation (Subramaniam et al., 2022; Chung et al., 2021). An architecture
based on the 3D U-Net was used by QSMGAN (Chen et al., 2020) to synthesize
QSM. WGAN with gradient penalty was used to improve the training stability.
Subramaniam  _et al_. (Subramaniam et al., 2022) simultaneously generated the
TOF-MRA patches along with the corresponding brain vessel segmentation labels.
To improve efficiency, four variants of the WGAN architecture were considered
and the mixed precision technique was applied.

####  3.3.3. Reconstruction/Recovery.

A complex-valued CNN (i.e., $\mathbb{C}$Net (El-Rewaidy et al., 2020)) was
used to reconstruct under-sampled 3D Late Gadolinium Enhancement (LGE) cardiac
MRI data. The overall architecture adopted a U-Net structure with several
complex-valued components: complex convolutional layer, radial Batch
Normalization (BN) layer, down-sampling and up-sampling layers, and complex
activation function. $\mathbb{C}$Net achieved better performance compared to
real-valued networks and faster speed compared to compressed sensing. Zhang
_et al_. (Hongtao et al., 2020) used a 2D super-resolution technique to
perform 3D Brain MRI super-resolution reconstruction. A super-resolution GAN
(i.e., ESRGAN (Wang et al., 2018b)) was applied to 2D brain slices, followed
by a Bilinear interpolation to complete the null values. ProvoGAN (Yurt et
al., 2022) proposed a progressively volumetric strategy to decompose the 3D
volume recovery into sequential 2D cross-sectional mappings. The three
rectilinear orientations (axial, coronal, sagittal) were gradually processed
in the first, second, and third progression step. In each step, a 2D GAN was
trained based on the previous step results. ProGAN outperformed both
volumetric and cross-sectional models on the MRI recovery and synthesis tasks.

###  3.4. Segmentation

Image segmentation is one of the most important medical imaging applications.
It can be utilized to either assign corresponding labels to different
anatomical structures (e.g., different cardiac chambers) or highlight specific
regions of interest (e.g., brain tumor or lesion areas). Automatic
segmentation liberates medical experts from the time-consuming and tedious
task of manually annotating the anatomical structures/disease regions. 3D
volume segmentation offers more detail and anatomical information than 2D
image segmentation, which is the focus of this subsection. This subsection is
organized according to different segmented structures/regions of interest in
the brain and heart. Hopefully, this will assist researchers in finding their
target models more quickly. Table 4 provides a list of the most relevant
methods.

Table 4. Summary of publications on 3D Segmentation related to the brain or
heart.

Publication (Year) | Organ | Model | Dataset | Metrics
---|---|---|---|---
Salehi  _et al_. (Salehi et al., 2017) (2017) | Brain | U-Net (Ronneberger et al., 2015) | MSSEG (Commowick et al., 2016) | DSC, Sen, Spe, F2, APR
Myronenko (Myronenko, 2018) (2018) | Brain | VAE (Kingma and Welling, 2013) | BRATS (Bakas et al., 2017) | Dice, HSD
S3D-UNet (Chen et al., 2018) (2018) | Brain | U-Net (Ronneberger et al., 2015) | BraTS (Bakas et al., 2017) | Dice, Sen, Spe, HD95
voxel-GAN (Rezaei et al., 2018) (2018) | Brain | cGAN (Mirza and Osindero, 2014), U-Net (Ronneberger et al., 2015) | BRATS (Bakas et al., 2017), ISLES (Maier et al., 2017) | Dice, HSD, Spe, Sen
Mondal  _et al_. (Mondal et al., 2018) (2018) | Brain | U-Net (Ronneberger et al., 2015), GAN (Goodfellow et al., 2014) | iSeg-2017 (iSeg 2017, 2022), MRBrainS (Mendrik et al., 2015) | DSC, ASD
Yang  _et al_. (Yang et al., 2018) (2018) | Heart | GAN (Goodfellow et al., 2014), U-Net (Ronneberger et al., 2015) | Own (Yang et al., 2018) | PSNR, MSE
MuTGAN (Xu et al., 2018) (2018) | Heart | GAN (Goodfellow et al., 2014) | Own (Xu et al., 2018) | ACC, Dice, Sen
|  |  |  | Spec, Infarct Size, etc.
VoxelAtlasGAN (Dong et al., 2018) (2018) | Heart | cGAN (Mirza and Osindero, 2014) | Own (Dong et al., 2018) | MSD, HSD, Dice, Corr of EF
Zhang _et al_. (Zhang et al., 2018b) (2018) | Heart | CycleGAN (Zhu et al., 2017) | Own (Zhang et al., 2018b) | Dice, S-score
Liu  _et al._ (Liu et al., 2019b) (2019) | Brain | GAN (Goodfellow et al., 2014), U-Net (Ronneberger et al., 2015) | BRATS (Bakas et al., 2017) | Dice
RP-Net (Wang et al., 2019) (2019) | Brain | U-Net (Ronneberger et al., 2015) | CANDI (Hodge et al., 2009), IBSR (Rohlfing et al., 2004) | DSC
DSTGAN (Xu et al., 2020) (2020) | Heart | GAN (Goodfellow et al., 2014) | Own (Xu et al., 2020) | ACC, Dice, Infarct size
|  |  |  | IoU, Per-size, Per-seg
PSCGAN (Xu et al., 2020) (2020) | Heart | GAN (Goodfellow et al., 2014) | Own (Xu et al., 2020) | SSIM, NRMSE, PSNR, Dice
|  |  |  | ACC, Sec, Spe
Yuan  _et al_. (Yuan et al., 2020) (2020) | Brain | GAN (Goodfellow et al., 2014) | Decathlon (Simpson et al., 2019) | Dice, RAVD, ASSD, MSSD
|  |  |  | HD95, Sen, Spe
3D DR-UNet (Vesal et al., 2020) (2020) | Heart | U-Net (Ronneberger et al., 2015) | ACDC (Bernard et al., 2018), LASC (Oakes et al., 2009) | DSC, HD, ASD
Peng  _et al_. (Peng et al., 2020) (2020) | Brain | U-Net (Ronneberger et al., 2015) | BraTS (Bakas et al., 2017) | Dice
SASSNet (Li et al., 2020) (2020) | Heart | GAN (Goodfellow et al., 2014) | Left Atrium (Xiong et al., 2021) | Dice, Jaccard, ASD, HD95
Vox2Vox (Cirillo et al., 2020) (2020) | Brain | Pixel-to-pixel (Isola et al., 2017) | BRATS (Bakas et al., 2017) | Dice, HD95
|  | U-Net (Ronneberger et al., 2015) |  |
Kolarik  _et al_. (Kolarik et al., 2021) (2021) | Brain | U-Net (Ronneberger et al., 2015) | MSSEG (Commowick et al., 2016) | Dice, Sen
FM-Pre-ResNet (Habijan et al., 2021) (2021) | Heart | VAE (Kingma and Welling, 2013), U-Net (Ronneberger et al., 2015) | MM-WHS (Zhuang and Shen, 2016) | DSC, Jaccard, SD, HD
Ullah  _et al_. (Ullah et al., 2021) (2021) | Brain | U-Net (Ronneberger et al., 2015) | BRATS (Bakas et al., 2017) | Dice
MVSGAN (Qi et al., 2021) (2021) | Heart | GAN (Goodfellow et al., 2014) | Own (Qi et al., 2021), STACOM2011 (Suinesiaputra et al., 2014) | Dice, Jaccard, HD
Zhang  _et al_. (Zhang et al., 2021) (2021) | Brain | CycleGAN (Zhu et al., 2017), U-Net (Ronneberger et al., 2015) | BRATS (Bakas et al., 2017) | Dice, HD95, Sen
DAR-UNet (Yao et al., 2022) (2022) | Brain | GAN (Goodfellow et al., 2014), U-Net (Ronneberger et al., 2015) | Vestibular Schwannoma (Shapey et al., 2021) | Dice, ASD
Bustamante  _et al_. (Bustamante et al., 2022) (2022) | Heart | U-Net (Ronneberger et al., 2015) | Own (Bustamante et al., 2022) | Dice, HD, ASD, Sen
|  |  |  | Precision, miss rate
Own: the dataset was originally collected and processed by the authors of the
paper.
ACC: accuracy. Sen: sensitivity. Spe: specificity. PPV: positive predictive
value. HD95: 95% Hausdorff distance.
DSC: dice similarity coefficient. MSD: mean surface distance. HSD: Hausdorff
surface distance.
ASD: average surface distance. EF: ejection fractions. Per-size: percentage of
infarct size, Per-seg: percentage of segments.

####  3.4.1. Brain

##### Tumor segmentation.

Brain tumor segmentation has gained great research interest recently (Liu et
al., 2022), especially for 3D brain volumes (Myronenko, 2018; Chen et al.,
2018; Peng et al., 2020; Cirillo et al., 2020; Zhang et al., 2021; Rezaei et
al., 2018; Yuan et al., 2020; Ullah et al., 2021). Myronenko (Myronenko, 2018)
used an encoder-decoder structure for tumor subregion segmentation, which
consists of a shared encoder, a segmentation decoder and a reconstruction VAE
decoder. S3D-UNet (Chen et al., 2018) proposed a separable U-Net architecture
that replaces the original 3D convolution with a three-branch separable 3D
convolution for brain tumor segmentation. Peng  _et al_. (Peng et al., 2020)
proposed a multi-scale 3D U-Net architecture to capture the features from
different spatial resolutions. The architecture consists of several 3D U-Net
blocks, with each block utilizing a 3D separable convolution instead of a
standard convolution to reduce the time and memory complexities. Vox2Vox
(Cirillo et al., 2020) devised a 3D volume-to-volume GAN to segment brain
tumors, composed of a U-Net generator to transform images to masks and a
PatchGAN (Li and Wand, 2016) discriminator for adversarial learning. Zhang
_et al_. (Zhang et al., 2021) proposed a cross-modality framework based on
CycleGAN (Zhu et al., 2017) to segment brain tumors from multi-modality brain
data. Specifically, two sub-modules were designed: a CycleGAN-based Cross-
modality Feature Transition module to perform cross-modality synthesis, and a
Cross-modality Feature Fusion module to integrate multi-modality features for
tumor segmentation. The conditional GAN model was used in voxel-GAN (Rezaei et
al., 2018) to address the imbalanced tumor segmentation task. In order to deal
with the imbalance issue, a cGAN that contained a segmentor and discriminator
was designed with a weighted adversarial loss. Yuan  _et al_. (Yuan et al.,
2020) conducted multi-modal segmentation with only unpaired 3D medical
volumes. The segmentation task and auxiliary cross-modality translation tasks
were tackled in a unified framework, consisting of modules for modality
translation and modality recovery. Each module contained a shared encoder, a
segmentation decoder, and a translation decoder. Ullah  _et al_. (Ullah et
al., 2021) evaluated a number of MRI image pre-processing techniques for
improving and segmenting MRI images. Gibbs ringing artifact removal was found
to be the best processing technique.

##### Sclerosis lesion segmentation.

To address the data imbalance problem of segmenting sclerosis lesions, Salehi
_et al_. (Salehi et al., 2017) used the U-Net structure and proposed a new
generalized loss function called Tversky loss. For better imbalanced
segmentation, the Tversky loss used both the Dice similarity coefficient (DSC)
and F1 score. Kolarik  _et al_. (Kolarik et al., 2021) proposed a transfer
learning framework for lesion segmentation in which pre-trained 2D CNN weights
were transferred to 3D networks. Specifically, a Planar 3D U-Net was proposed
based on the 3D Planar convolution to conduct lesion segmentation.

##### Brain tissue segmentation.

RP-Net (Wang et al., 2019) segmented the brain tissues into three classes:
cerebrospinal fluid (CSF), gray matter (GM), and white matter (WM). RP-Net
modified the U-Net structure in (Ronneberger et al., 2015) and introduced both
a pyramid pooling module and a recursive residual module. Then, multiple
stacks of deep networks were constructed with these modules. Mondal  _et al_.
(Mondal et al., 2018) performed the 3D multi-modal brain tissue segmentation
task in a few-shot setting where only few labeled data were provided.
Specifically, the labeled volumes were input to a U-Net generator to predict
the segmentation masks, while the unlabeled volumes underwent a variational
autoencoder to perform the reconstruction task.

##### Others.

Four MRI sequences (T1, T1c, T2, FLAIR) were used by Liu  _et al._ (Liu et
al., 2019b) to segment glioma subregions into three classes: whole tumor (WT),
enhancing tumor (ET) and tumor core (TC). Specifically, they proposed a GAN-
regularized 3D U-Net structure, which uses a pixel-to-pixel scheme to
concatenate the input with labels for effective training. DAR-UNet (Yao et
al., 2022) modeled the cross-modality medical image segmentation (e.g., brain
structure segmentation) within an Unsupervised Domain Adaptation (UDA)
framework. The first step was to train multiple GANs to disentangle the style
and content of both the source and target images at the same time. Then, these
styles were used to generate diverse target images, which were then used to
train the segmentation model.

####  3.4.2. Heart

##### Whole heart structure segmentation.

FM-Pre-ResNet (Habijan et al., 2021) proposed two novel designs for segmenting
the whole heart structure. First, the feature merge residual unit was designed
to improve existing pre-activation residual units. Second, 3D VAE was used to
perform the volume reconstruction task, which regularized and assisted the
segmentation task. Zhang _et al_. (Zhang et al., 2018b) proposed a cross-
modality synthesis approach with unpaired training data to boost the multi-
modal volume segmentation performance. For the cross-modality synthesis task,
two GANs were used to construct the CycleGAN (Zhu et al., 2017), and the
segmentation task utilized the synthesized data to augment the training data.
The model was trained with cycle-consistency loss, adversarial loss, and
shape-consistency loss. Heart segmentation was accomplished using a multi-
stage procedure in 3D DR-UNet (Vesal et al., 2020): a low-resolution localized
map was first generated with a Localization Network, then the high resolution
volume was cropped based on the map and fed into a Segmentation Network to
finally generate the segmentation mask. Both the Localization Network and the
Segmentation Network were built using the 3D Dilated Residual U-Net
architecture. For the automatic segmentation of the 4D flow MRI volume,
Bustamante  _et al_. (Bustamante et al., 2022) used the U-Net structure
instead of the regular 3D volume segmentation. With the proposed method, the
four cardiac chambers, the aorta, and pulmonary artery were all segmented.

##### Left ventricle (LV) / left atrium (LA) segmentation.

VoxelAtlasGAN (Dong et al., 2018) addressed the 3D LV segmentation task on
echocardiography images. A new voxel-to-voxel mode of the conditional GANs was
devised to include the 3D LV atlas as powerful condition information,
addressing issues with limited annotation and low contrast in
echocardiography. In addition, a consistent constraint was proposed to enhance
the discrimination loss. Yang  _et al_. (Yang et al., 2018) proposed a
probability-based method to segment the LV myocardium, avoiding the boundary
moving issues associated with threshold-based methods. A 3D U-Net generator
was used to propagate the partial 2D slice segmentation to the whole 3D
volume, and a 3D CNN discriminator was used to distinguish the ground truth
and the generated segmentation masks. MVSGAN (Qi et al., 2021) tackled the
short-axis (SAX) view segmentation of 3D cardiac MRI volumes, which was
hampered by the sparse spatial structure. Three modules were devised in
MVSGAN: (1) a residual adversarial fusion (RAF) module to examine the inter-
slice dependency and anatomical priors, (2) a structural perception-
aggregation (SPA) module to model the correspondence between segmentation
labels and cardiac models, and (3) a joint training objective for multi-task
learning. SASSNet (Li et al., 2020) proposed a shape-aware method for semi-
supervised 3D LA segmentation (Yu et al., 2019), incorporating geometric shape
constraints into the GANs framework. More specifically, the adversarial
generation of Signed Distance Map (SDM) serves as an auxiliary task to augment
the LA segmentation. The shared segmentation network (generator) adopts a
V-Net (Milletari et al., 2016) to simultaneously generate the SDM and
segmentation mask.

##### Myocardial infarction (MI) segmentation.

A generative and multi-task learning framework was proposed in MuTGAN (Xu et
al., 2018) for simultaneous quantification and segmentation of myocardial
infarction (MI). Specifically, the MuTGAN generator consists of two networks:
a spatial-temporal feature extraction network (SFEN) to extract the
comprehensive spatio-temporal correlations, and a joint feature learning
network to learn the interaction between segmentation and quantification. The
discriminator consists of a task relatedness network to learn the intrinsic
task patterns. DSTGAN (Xu et al., 2020) extended MuTGAN (Xu et al., 2018) with
several enhancements: a new feature learning network (spatio-temporal
variation encoder), a conditional GAN, and three iterative task-specific
discriminators.

##### Ischemic disease image segmentation.

PSCGAN (Xu et al., 2020) proposed the first one-stop contrast agent-free
framework to simultaneously synthesize and segment the Ischemic Heart Disease
(IHD) related cine MR images. The proposed framework is a progressive model
integrating three consecutive GAN phases: prior generation, conditional
synthesis, and fine segmentation. Each phase uses a similar sequential causal
learning framework with two pathways: a spatial perceptual pathway and a
temporal perceptual pathway. A synthetic regularization loss and a
segmentation auxiliary loss were specially designed to facilitate the model
training.

###  3.5. Denoising

Table 5. Summary of publications on 3D Denoising related to the brain or heart

Publication (Year) | Organ | Model | Dataset | Metrics
---|---|---|---|---
Wolterink  _et al_. (Wolterink et al., 2017) (2017) | Heart | GAN (Goodfellow et al., 2014) | Harder  _et al_. (den Harder et al., 2016) | PSNR
LA-GANs (Wang et al., 2018c) (2018) | Brain | GAN (Goodfellow et al., 2014) | MCI (Wang et al., 2018a) | PSNR, SSIM
3D c-GANs (Wang et al., 2018a) (2018) | Brain | cGAN (Mirza and Osindero, 2014) | MCI (Wang et al., 2018a) | PSNR, NMSE, SUV differences
RED-WGAN (Ran et al., 2019) (2019) | Brain | WGAN (Gulrajani et al., 2017) | IXI (Dataset, 2010) | PSNR, RMSE, SSIM, IFC
SGSGAN (Zhou et al., 2022) (2022) | Brain | StyleGAN (Karras et al., 2019) | Own (Zhou et al., 2022) | PSNR, SSIM, MAE, U-Net score
Own: the dataset was originally collected and processed by the authors of
paper.
IFC: information fidelity criterion.

Artifacts and noise may affect medical images collected in practical clinical
settings due to factors such as radiation dose, time constraints, and patient
discomfort. The purpose of image denoising is to reduce the amount of noise in
the original image (e.g., low-dose CT) and recover a high-quality medical
image (e.g., full-dose CT). Denoising Autoencoders and GANs can naturally be
applied for image denoising. A list of relevant 3D denoising models is
provided in Table 5.

In an early study, Wolterink  _et al_. (Wolterink et al., 2017) proposed the
use of GANs in low-dose CT for noise reduction. A generator network generated
routine-dose CT images from low-dose CT images. A discriminator was used to
differentiate between the real and generated routine-dose CT images. The GANs
were trained using a voxelwise loss. An encoder-decoder and WGAN technique
were used to denoise MRI images in RED-WGAN (Ran et al., 2019). The residual
autoencoder structure was devised for the generator network while three losses
were proposed to train the whole network: adversarial loss, MSE loss, and
perceptural loss. 3D c-GANs (Wang et al., 2018a) achieved high-quality (full-
dose) PET image synthesis from low-dose CT images using 3D conditional GANs.
The 3D U-Net was used as a generator and a concatenated progressive refinement
strategy was proposed. SGSGAN (Zhou et al., 2022) synthesized full-dose PET
images using StyleGAN (Karras et al., 2019) and a segmentation U-Net.
Specifically, a style-based generator converted low-dose PET images to full-
dose images, which were then processed by the segmentation network and the
discriminator network. Based on low-dose PETs and corresponding MRI images,
LA-GANs (Wang et al., 2018c) synthesized full-dose PETs. First, a local
adaptive fusion module was designed to integrate all input modalities into a
fused image. Then, a U-Net generator and a discriminator were adversarially
trained with the real/fake full-dose PET images.

###  3.6. Detection

Table 6. Summary of publications on 3D Detection related to the brain or heart

Publication (Year) | Organ | Model | Dataset | Metrics
---|---|---|---|---
Uzunova  _et al_. (Uzunova et al., 2019) (2019) | Brain | CVAE (Sohn et al., 2015) | BRATS (Bakas et al., 2017), LPBA40 (Shattuck et al., 2008) | Dice, Sen, Spec, AUC
|  |  |  | Jaccard
MADGAN (Han et al., 2021) (2021) | Brain | GAN (Goodfellow et al., 2014) | OASIS-3 (LaMontagne et al., 2019),Own (Han et al., 2021) | AUC
3D MTGA (Xiong et al., 2022) (2022) | Heart | U-Net (Ronneberger et al., 2015), cGAN (Mirza and Osindero, 2014) | Own (Xiong et al., 2022) | Sen, Spe, Pre, ACC
|  |  |  | SSIM, PSNR, MAE, RMSE
|  |  |  | DSC, USR, OSR, Jaccard
Pinaya  _et al_. (Pinaya et al., 2022b) (2022) | Brain | VQ-VAE (Van Den Oord et al., 2017) | MedNIST (Yang et al., 2021b), BRATS (Bakas et al., 2017) | Dice, AUPRC, AUROC
|  | Autoregressive Transformer (Esser et al., 2021) | WMH (Kuijf et al., 2019), MSLUB (Lesjak et al., 2018) | FPR80, FPR95, FPR99
|  |  | UK Biobank (Sudlow et al., 2015) |
Own: the dataset was originally collected and processed by the authors of the
paper.
Sen: sensitivity. Spe: specificity. Prec: precision. ACC: accuracy. FPR:
false- positive rates.
DSC: dice similarity coefficient. USR: under-segmentation rate. OSR: over-
segmentation rate.

In medical imaging, detection usually refers to the detection of anomalies or
pathologies, such as lesions or tumours. A supervised detection method
requires large amounts of healthy and anomalous data to effectively
differentiate between them. This is often not feasible especially due to the
scarcity of anomalous data and data imbalance. Therefore, unsupervised
detection methods have been proposed in order to use generative models to
capture the healthy data distribution. A list of relevant 3D detection models
is provided in Table 6 for the brain or the heart.

MADGAN (Han et al., 2021) designed a two-step GAN-based medical anomaly
detection method to detect different types of brain anomalies at various
stages, e.g., Alzheimer’s disease at early and late stages, as well as brain
metastasis. The generator uses three consecutive brain MRI slices to generate
the next three brain slices, while the discriminator classifies these
real/fake next three brain slices. After training, the reconstruction loss
(i.e., $L_{2}$) between the real and reconstructed three slices is computed to
detect anomalies. A multi-task generative framework was proposed in (Xiong et
al., 2022) to detect aortic dissection using non-contrast-enhanced CT (NCE-CT)
volumes. The framework consists of a 3D nnU-Net to perform aortas segmentation
of NCE-CT volumes and a 3D Multi-Task Generative Architecture (MTGA) to
perform CE-CT synthesis, true & false lumen segmentation, and simultaneous
aortic dissection detection. Uzunova  _et al_. (Uzunova et al., 2019)
performed unsupervised pathology detection with a 3D CVAE structure. The
pathologies were detected by the differences to the learned norm of the
healthy image volumes through CVAE reconstruction. Pinaya  _et al_. (Pinaya et
al., 2022b) applied a Transformer-based model for unsupervised 3D brain
anomaly detection and segmentation when a limited amount of data is required.
Specifically, a codebook of brain image representations is learnt with the
VQVAE (Van Den Oord et al., 2017) architecture. Then, an Autoregressive
Transformer uses the codebook to generate brain volumes in a similar way to
PixelRNN (Van Oord et al., 2016). Anomalies are considered for Autoregressive
Transformer output probabilities lower than a threshold.

###  3.7. Registration

Table 7. Summary of publications on 3D Registration related to the brain or
heart

Publication (Year) | Organ | Model | Dataset | Metrics
---|---|---|---|---
VTN (Zhao et al., 2019) (2019) | Brain | U-Net (Ronneberger et al., 2015) | ADNI (Weiner et al., 2017), ABIDE (Di Martino et al., 2014), ADHD (consortium, 2012) | Seg. IoU, Lm. Dist., Time
VoxelMorph (Balakrishnan et al., 2019) (2019) | Brain | U-Net (Ronneberger et al., 2015) | OASIS (Marcus et al., 2007), ABIDE (Di Martino et al., 2014), ADHD (consortium, 2012) | Dice, Jacobian matrix
|  |  | MCIC (Gollub et al., 2013), PPMI (Marek et al., 2011), Harward GSP (Holmes et al., 2015) |
|  |  | HABS (Dagley et al., 2017), FreeSurfer Buckner40 (Fischl, 2012) |
Deform-GAN (Zhang et al., 2020) (2020) | Brain | GAN (Goodfellow et al., 2014) | BRATS (Bakas et al., 2017) | RMSE, Dice
Zhu  _et al_. (Zhu et al., 2021) (2021) | Brain | U-Net (Ronneberger et al., 2015) | Mindboggle101 (Klein et al., 2017), LPBA40 (Shattuck et al., 2008), IXI (Dataset, 2010) | DSC, HD, ASSD
Ramon  _et al_. (Ramon et al., 2022) (2022) | Brain | GAN (Goodfellow et al., 2014) | ADNI (Weiner et al., 2017), NIREP (Christensen et al., 2006) | DSC
TGAN (Zhu et al., 2022) (2022) | Brain | GAN (Goodfellow et al., 2014) | BrainWeb (Cocosco et al., 1997), Atlas (Liew et al., 2018), RIRE (RIRE, 2022) | Dice, mean SSIM
|  | U-Net (Ronneberger et al., 2015) |  |
DSC: dice similarity coefficient. HD: Hausdorff distance. ASSD: average
symmetric surface distance.

The purpose of image registration is to find the spatial correspondence (e.g.,
affine matrix, deformation field) between an image pair. As a result of this
correspondence, geometric models like Spatial Transformer Network (STN)
(Jaderberg et al., 2015) can be used to warp the image. Since manually
annotating the corresponding points of image pairs is prohibitively laborious,
various unsupervised registration approaches based on generative models (e.g.,
U-Net (Ronneberger et al., 2015) or GAN (Goodfellow et al., 2014)) have been
proposed. Table 7 lists some of the most relevant 3D registration methods.

VTN (Zhao et al., 2019) proposed an unsupervised Volume Tweening Network (VTN)
for 3D volume registration. VTN was constructed by an end-to-end cascade of
registration subnetworks to generate deformation fields between images. The
registration subnetwork was implemented with either an affine registration
subnetwork or a dense deformable registration subnetwork. An invertibility
loss was further incorporated to ensure backward consistency. In an end-to-end
framework, Zhu  _et al_. (Zhu et al., 2021) performed joint affine alignment
and deformable registration. The framework contained an affine alignment
subnetwork to learn the affine matrix and a deformable registration subnetwork
to generate the deformation fields. An STN (Jaderberg et al., 2015) was used
to warp the volume. The VoxelMorph algorithm (Balakrishnan et al., 2019, 2018)
formulated registration as a mapping between image pairs and deformation
fields. A U-Net processes the input pair to get a registration field, which
was then used by STN to warp the image. Both the original image volumes and
the segmentation masks could be adopted to train the model. Deform-GAN (Zhang
et al., 2020) incorporated mono-modal registration and multi-modal
registration in an adversarial framework, which consists of a U-Net-like
Transformation Network and Generator, as well as a Discriminator. The
registration model was trained using both a GAN loss and a new local gradient
loss. Ramon  _et al_. (Ramon et al., 2022) combined a GAN with the traditional
LDDMM (Hernandez, 2014) approach and proposed two model variants for the
stationary and non-stationary parameterizations. TGAN (Zhu et al., 2022)
proposed a two-stage GAN framework for the multi-modal brain registration. The
GAN of the first stage extracted the structural representation of the
registered image, while the GAN of the second stage estimated the image
intensities after registration.

##  4\. Discussion and Future Directions

We have surveyed seven medical applications pertaining to 3D generative models
for the brain and heart (from Jan 2017 to Oct 2022): unconditional synthesis,
classification, conditional synthesis, segmentation, denoising, detection, and
registration. Among them, segmentation and conditional synthesis have the
largest number of publications: 25 and 19, showing the great research interest
and importance of these two applications. U-Net (Ronneberger et al., 2015) is
the most popular model for segmentation, either used individually or as a
generator/encoder network. Image-to-image translation (Isola et al., 2017; Zhu
et al., 2017) is naturally compatible with conditional synthesis, and is based
on models such as cGAN (Mirza and Osindero, 2014), CycleGAN (Zhu et al.,
2017), StarGAN (Choi et al., 2018), etc. Unconditional synthesis covers 10
papers, favoring WGAN-GP (Gulrajani et al., 2017), VAE (Kingma and Welling,
2013), and StyleGAN2 (Karras et al., 2020) models. In the literature, there
are only 6 papers on registration, 5 on denoising, and 4 on detection. They
are mainly based on the vanilla GAN (Goodfellow et al., 2014) and U-Net
(Ronneberger et al., 2015) models. The architecture and model design should
consider the application requirements, e.g., CycleGAN (Zhu et al., 2017) was
widely-used for unsupervised learning (Pan et al., 2018; Zhang et al., 2018b,
2021), and VAE (Kingma and Welling, 2013) was adopted for interpreting the
classification results (Biffi et al., 2018; Puyol-Antón et al., 2020). As well
as models, we also find that metrics and datasets are usually specific to
organs and applications, e.g., ADNI (Weiner et al., 2017) dataset for brain
applications, Dice metric for segmentation related tasks.

Although many publications show the effectiveness and application prospects of
3D generative models for medical volumes, this field is still rapidly growing
and embraces various potential future directions and techniques. The following
are four possible future directions.

###  4.1. Few-shot Learning

Few-shot learning (Wang et al., 2020) aims to leverage few available examples
to effectively train a deep model. It has been extensively researched in
computer vision (Liu et al., 2019b) and natural language processing (Hou et
al., 2021). For example, few-shot image generation (Ojha et al., 2021), few-
shot unsupervised image-to-image translation (Liu et al., 2019a) have been
investigated for natural images. As a result of the collection procedure,
patient consent, and annotation cost, medical applications usually lack
sufficient training data. Thus, few-shot learning is naturally suited to
medical applications of 3D generative models. Using few training samples,
Mondal  _et al_. (Mondal et al., 2018) achieved comparable segmentation
results for 3D multi-modal volumes, showing promising prospects for 3D medical
applications utilizing few-shot learning methods.

###  4.2. Self-supervised Learning

Self-supervised learning (SSL) (Liu et al., 2021) explores the structural and
contextual information lying in the data to provide supervision signals for
model training. Self-supervised learning has the advantage of not requiring
manual annotations, saving time and money. According to a recent study (He et
al., 2022), self-supervised learning can be as effective as or even superior
to supervised learning in various computer vision tasks. For 3D medical
applications, the utilization of self-supervised learning will mitigate the
burden of tedious and laborious annotation, while still achieving satisfactory
performance. Joyce _et al_. (Joyce and Kozerke, 2019) trained a 3D cardiac MRI
synthesis model with only unlabelled data through self-supervision
(reconstruction error). Cheng  _et al_. (Ouyang et al., 2022) showed that 2D
few-shot medical segmentation could be realized with self-supervised learning
techniques. In the future, we expect more studies to leverage the intrinsic 3D
structure of medical volumes as a powerful tool for self-supervised learning.

###  4.3. Efficient 3D Network Architecture

3D network architecture is another bottleneck for effective 3D generative
medical applications, in addition to data scarcity. The majority of 3D models
for medical volume generation use 3D convolution directly, which is slow and
parameter-inefficient. Due to this, existing methods have a small model
capacity (e.g., 64 convolution channels (Hong et al., 2021) and a small
synthesized resolution (e.g., $64\times 64\times 64$ (Kwon et al., 2019))).
The development of efficient network architectures for 3D generative models is
an important and practical research area. By using Split&Shuffle modules in
GANs, (Liu et al., 2022) reduced over half of the model parameters, and yet
achieved better performance on both heart and brain datasets. A comprehensive
analysis of resource-efficient 3D CNNs was conducted in (Kopuklu et al.,
2019). Additionally, neural architecture search (NAS) (Elsken et al., 2019) is
a promising direction for designing efficient 3D architectures.

###  4.4. Denoising Diffusion Probabilistic Model (DDPM)

Recent research has shown that DDPMs outperform GANs at generating natural
images (Dhariwal and Nichol, 2021; Ho et al., 2022). DDPM models a Markov
chain process to convert the simple Gaussian noise into a real data
distribution. When compared to GANs, it can better cover the overall data
distribution and have a stationary training objective. As a result, medical
applications can also benefit from the DDPM for effective image generation.
DDPM (Ho et al., 2020) was directly used in (Pinaya et al., 2022a) for
unsupervised brain anomaly detection on 2D CT and MRI images. In (Kim and Ye,
2022), DDPM was used to estimate the latent code for 4D cardiac MR image
generation. We believe that future 3D medical volume models can use DDPM as a
good alternative to GANs and VAEs.

##  5\. Conclusion

This paper reviews three types of generative models and their applications to
synthetic 3D medical volume generation. A new taxonomy of unconditional and
conditional generative models is proposed to cover seven medical applications:
unconditional synthesis, classification, conditional synthesis, segmentation,
denoising, detection, and registration. We have provided a corresponding table
with the organ, model, dataset, and evaluation metrics for each application to
make it easier for researchers to locate their model of interest. Finally,
with the hope of further progressing future research in this field, four
prospective directions are discussed.

## References

  * (1)
  * Ahmed et al. (2021) Aya Saleh Ahmed, Wessam H El-Behaidy, and Aliaa AA Youssif. 2021\.  Medical image denoising system based on stacked convolutional autoencoder for enhancing 2-dimensional gel electrophoresis noise reduction.  Biomedical Signal Processing and Control 69 (2021), 102842.
  * AIMI (2022) Stanford AIMI. 2022\.  COCA - Coronary Calcium and chest CT’s dataset.  (2022).  https://stanfordaimi.azurewebsites.net/datasets/e8ca74dc-8dd4-4340-815a-60b41f6cb2aa
  * Akkus et al. (2017) Zeynettin Akkus, Alfiia Galimzianova, Assaf Hoogi, Daniel L Rubin, and Bradley J Erickson. 2017\.  Deep learning for brain MRI segmentation: state of the art and future directions.  Journal of digital imaging 30, 4 (2017), 449–459.
  * AlAmir and AlGhamdi (2022) Manal AlAmir and Manal AlGhamdi. 2022.  The Role of Generative Adversarial Network in Medical Image Analysis: An in-depth survey.  ACM Computing Surveys (CSUR) (2022).
  * Ali et al. (2022) Hazrat Ali, Rafiul Biswas, Farida Ali, Uzair Shah, Asma Alamgir, Osama Mousa, and Zubair Shah. 2022.  The role of generative adversarial networks in brain MRI: a scoping review.  Insights into Imaging 13, 1 (2022), 1–15.
  * Amirrajab et al. (2020) Sina Amirrajab, Samaneh Abbasi-Sureshjani, Yasmina Al Khalil, Cristian Lorenz, Juergen Weese, Josien Pluim, and Marcel Breeuwer. 2020\.  Xcat-gan for synthesizing 3d consistent labeled cardiac mr images on anatomically variable xcat phantoms. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 128–137.
  * Andreopoulos and Tsotsos (2008) Alexander Andreopoulos and John K Tsotsos. 2008.  Efficient and generalizable statistical models of shape and appearance for analysis of cardiac MRI.  Medical image analysis 12, 3 (2008), 335–357.
  * Arjovsky et al. (2017) Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017\.  Wasserstein generative adversarial networks. In International conference on machine learning. PMLR, 214–223.
  * Bakas et al. (2017) Spyridon Bakas, Hamed Akbari, Aristeidis Sotiras, Michel Bilello, Martin Rozycki, Justin S Kirby, John B Freymann, Keyvan Farahani, and Christos Davatzikos. 2017.  Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features.  Scientific data 4, 1 (2017), 1–13.
  * Balakrishnan et al. (2018) Guha Balakrishnan, Amy Zhao, Mert R Sabuncu, John Guttag, and Adrian V Dalca. 2018.  An unsupervised learning model for deformable medical image registration. In Proceedings of the IEEE conference on computer vision and pattern recognition. 9252–9260.
  * Balakrishnan et al. (2019) Guha Balakrishnan, Amy Zhao, Mert R Sabuncu, John Guttag, and Adrian V Dalca. 2019.  VoxelMorph: a learning framework for deformable medical image registration.  IEEE transactions on medical imaging 38, 8 (2019), 1788–1800.
  * Bernard et al. (2018) Olivier Bernard, Alain Lalande, Clement Zotti, Frederick Cervenansky, Xin Yang, Pheng-Ann Heng, Irem Cetin, Karim Lekadir, Oscar Camara, Miguel Angel Gonzalez Ballester, et al. 2018\.  Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: is the problem solved?  IEEE transactions on medical imaging 37, 11 (2018), 2514–2525.
  * Biffi et al. (2018) Carlo Biffi, Ozan Oktay, Giacomo Tarroni, Wenjia Bai, Antonio De Marvao, Georgia Doumou, Martin Rajchl, Reem Bedair, Sanjay Prasad, Stuart Cook, et al. 2018\.  Learning interpretable anatomical features through deep generative models: Application to cardiac remodeling. In International conference on medical image computing and computer-assisted intervention. Springer, 464–471.
  * Borji (2019) Ali Borji. 2019\.  Pros and cons of gan evaluation measures.  Computer Vision and Image Understanding 179 (2019), 41–65.
  * Bourlard and Kamp (1988) Hervé Bourlard and Yves Kamp. 1988.  Auto-association by multilayer perceptrons and singular value decomposition.  Biological cybernetics 59, 4 (1988), 291–294.
  * Bustamante et al. (2022) Mariana Bustamante, Federica Viola, Jan Engvall, Carl-Johan Carlhäll, and Tino Ebbers. 2022.  Automatic Time-Resolved Cardiovascular Segmentation of 4D Flow MRI Using Deep Learning.  Journal of Magnetic Resonance Imaging (2022).
  * Chen et al. (2019) Sihong Chen, Kai Ma, and Yefeng Zheng. 2019.  Med3d: Transfer learning for 3d medical image analysis.  arXiv preprint arXiv:1904.00625 (2019).
  * Chen et al. (2018) Wei Chen, Boqiang Liu, Suting Peng, Jiawei Sun, and Xu Qiao. 2018.  S3D-UNet: separable 3D U-Net for brain tumor segmentation. In International MICCAI Brainlesion Workshop. Springer, 358–368.
  * Chen et al. (2020) Yicheng Chen, Angela Jakary, Sivakami Avadiappan, Christopher P Hess, and Janine M Lupo. 2020\.  QSMGAN: improved quantitative susceptibility mapping using 3D generative adversarial networks with increased receptive field.  NeuroImage 207 (2020), 116389.
  * Chen et al. (2022) Yizhou Chen, Xu-Hua Yang, Zihan Wei, Ali Asghar Heidari, Nenggan Zheng, Zhicheng Li, Huiling Chen, Haigen Hu, Qianwei Zhou, and Qiu Guan. 2022\.  Generative adversarial networks in medical image augmentation: a review.  Computers in Biology and Medicine (2022), 105382.
  * Cho et al. (2014) Kyunghyun Cho, B van Merrienboer, Caglar Gulcehre, F Bougares, H Schwenk, and Yoshua Bengio. 2014.  Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP 2014).
  * Choi et al. (2018) Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. 2018\.  Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 8789–8797.
  * Chong and Ho (2021) Chee Keong Chong and Eric Tatt Wei Ho. 2021.  Synthesis of 3D MRI brain images with shape and texture generative adversarial deep neural networks.  IEEE Access 9 (2021), 64747–64760.
  * Christensen et al. (2006) Gary E Christensen, Xiujuan Geng, Jon G Kuhl, Joel Bruss, Thomas J Grabowski, Imran A Pirwani, Michael W Vannier, John S Allen, and Hanna Damasio. 2006\.  Introduction to the non-rigid image registration evaluation project (NIREP). In International workshop on biomedical image registration. Springer, 128–135.
  * Chung et al. (2021) Hyungjin Chung, Eunju Cha, Leonard Sunwoo, and Jong Chul Ye. 2021.  Two-stage deep learning for accelerated 3D time-of-flight MRA without matched training data.  Medical Image Analysis 71 (2021), 102047.
  * Çiçek et al. (2016) Özgün Çiçek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger. 2016\.  3D U-Net: learning dense volumetric segmentation from sparse annotation. In International conference on medical image computing and computer-assisted intervention. Springer, 424–432.
  * Cirillo et al. (2020) Marco Domenico Cirillo, David Abramian, and Anders Eklund. 2020\.  Vox2Vox: 3D-GAN for brain tumour segmentation. In International MICCAI Brainlesion Workshop. Springer, 274–284.
  * Cocosco et al. (1997) Chris A Cocosco, Vasken Kollokian, Remi K-S Kwan, G Bruce Pike, and Alan C Evans. 1997.  Brainweb: Online interface to a 3D MRI simulated brain database. In NeuroImage. Citeseer.
  * Collobert et al. (2011) Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011\.  Natural language processing (almost) from scratch.  Journal of machine learning research 12, ARTICLE (2011), 2493–2537.
  * Commowick et al. (2016) Olivier Commowick, Frédéric Cervenansky, and Roxana Ameli. 2016.  MSSEG challenge proceedings: multiple sclerosis lesions segmentation challenge using a data management and processing infrastructure. In Miccai.
  * consortium (2012) ADHD-200 consortium. 2012\.  The ADHD-200 consortium: a model to advance the translational potential of neuroimaging in clinical neuroscience.  Frontiers in systems neuroscience 6 (2012), 62.
  * Dagley et al. (2017) Alexander Dagley, Molly LaPoint, Willem Huijbers, Trey Hedden, Donald G McLaren, Jasmeer P Chatwal, Kathryn V Papp, Rebecca E Amariglio, Deborah Blacker, Dorene M Rentz, et al. 2017\.  Harvard aging brain study: dataset and accessibility.  Neuroimage 144 (2017), 255–258.
  * Dalmaz et al. (2021) Onat Dalmaz, Mahmut Yurt, and Tolga Çukur. 2021.  ResViT: Residual vision transformers for multi-modal medical image synthesis.  arXiv preprint arXiv:2106.16031 (2021).
  * Dar et al. (2019) Salman UH Dar, Mahmut Yurt, Levent Karacan, Aykut Erdem, Erkut Erdem, and Tolga Cukur. 2019\.  Image synthesis in multi-contrast MRI with conditional generative adversarial networks.  IEEE transactions on medical imaging 38, 10 (2019), 2375–2388.
  * Dataset (2010) IXI Dataset. 2010\.  IXI brain development homepage.  (2010).  https://brain-development.org/ixi-dataset/
  * den Harder et al. (2016) Annemarie M den Harder, Jelmer M Wolterink, Martin J Willemink, Arnold MR Schilham, Pim A de Jong, Ricardo PJ Budde, Hendrik M Nathoe, Ivana Išgum, and Tim Leiner. 2016\.  Submillisievert coronary calcium quantification using model-based iterative reconstruction: a within-patient analysis.  European journal of radiology 85, 11 (2016), 2152–2159.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009\.  Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248–255.
  * Dhariwal and Nichol (2021) Prafulla Dhariwal and Alexander Nichol. 2021.  Diffusion models beat gans on image synthesis.  Advances in Neural Information Processing Systems 34 (2021), 8780–8794.
  * Di Martino et al. (2014) Adriana Di Martino, Chao-Gan Yan, Qingyang Li, Erin Denio, Francisco X Castellanos, Kaat Alaerts, Jeffrey S Anderson, Michal Assaf, Susan Y Bookheimer, Mirella Dapretto, et al. 2014\.  The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism.  Molecular psychiatry 19, 6 (2014), 659–667.
  * Dong et al. (2018) Suyu Dong, Gongning Luo, Kuanquan Wang, Shaodong Cao, Ashley Mercado, Olga Shmuilovich, Henggui Zhang, and Shuo Li. 2018\.  VoxelAtlasGAN: 3D left ventricle segmentation on echocardiography with atlas guided generation and voxel-to-voxel discrimination. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 622–629.
  * El-Rewaidy et al. (2020) Hossam El-Rewaidy, Ulf Neisius, Jennifer Mancio, Selcuk Kucukseymen, Jennifer Rodriguez, Amanda Paskavitz, Bjoern Menze, and Reza Nezafat. 2020.  Deep complex convolutional network for fast reconstruction of 3D late gadolinium enhancement cardiac MRI.  NMR in Biomedicine 33, 7 (2020), e4312.
  * Elazab et al. (2020) Ahmed Elazab, Changmiao Wang, Syed Jamal Safdar Gardezi, Hongmin Bai, Qingmao Hu, Tianfu Wang, Chunqi Chang, and Baiying Lei. 2020\.  GP-GAN: Brain tumor growth prediction using stacked 3D generative adversarial networks from longitudinal MR Images.  Neural Networks 132 (2020), 321–332.
  * Elsken et al. (2019) Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2019\.  Neural architecture search: A survey.  The Journal of Machine Learning Research 20, 1 (2019), 1997–2017.
  * Enokiya et al. (2018) Yuki Enokiya, Yutaro Iwamoto, Yen-Wei Chen, and Xian-Hua Han. 2018.  Automatic liver segmentation using U-Net with Wasserstein GANs.  Journal of Image and Graphics 6, 2 (2018), 152–159.
  * Esser et al. (2021) Patrick Esser, Robin Rombach, and Bjorn Ommer. 2021\.  Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 12873–12883.
  * Farnia and Ozdaglar (2020) Farzan Farnia and Asuman Ozdaglar. 2020.  Do GANs always have Nash equilibria?. In International Conference on Machine Learning. PMLR, 3029–3039.
  * Fischl (2012) Bruce Fischl. 2012\.  FreeSurfer.  Neuroimage 62, 2 (2012), 774–781.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning. PMLR, 1050–1059.
  * Gao et al. (2010) Wenshuo Gao, Xiaoguang Zhang, Lei Yang, and Huizhong Liu. 2010\.  An improved Sobel edge detection. In 2010 3rd International conference on computer science and information technology, Vol. 5. IEEE, 67–71.
  * Girshick (2015) Ross Girshick. 2015\.  Fast r-cnn. In Proceedings of the IEEE international conference on computer vision. 1440–1448.
  * Gollub et al. (2013) Randy L Gollub, Jody M Shoemaker, Margaret D King, Tonya White, Stefan Ehrlich, Scott R Sponheim, Vincent P Clark, Jessica A Turner, Bryon A Mueller, Vince Magnotta, et al. 2013\.  The MCIC collection: a shared repository of multi-modal, multi-site brain image data from a clinical investigation of schizophrenia.  Neuroinformatics 11, 3 (2013), 367–388.
  * Gondara (2016) Lovedeep Gondara. 2016\.  Medical image denoising using convolutional denoising autoencoders. In 2016 IEEE 16th international conference on data mining workshops (ICDMW). IEEE, 241–246.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014\.  Generative adversarial nets.  Advances in neural information processing systems 27 (2014).
  * Granstedt et al. (2021) Jason L Granstedt, Varun A Kelkar, Weimin Zhou, and Mark A Anastasio. 2021.  SlabGAN: a method for generating efficient 3D anisotropic medical volumes using generative adversarial networks. In Medical Imaging 2021: Image Processing, Vol. 11596. SPIE, 329–335.
  * Graves et al. (2013) Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013\.  Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing. Ieee, 6645–6649.
  * Gregor et al. (2014) Karol Gregor, Ivo Danihelka, Andriy Mnih, Charles Blundell, and Daan Wierstra. 2014\.  Deep autoregressive networks. In International Conference on Machine Learning. PMLR, 1242–1250.
  * Gulrajani et al. (2017) Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. 2017\.  Improved training of wasserstein gans.  Advances in neural information processing systems 30 (2017).
  * Habijan et al. (2021) Marija Habijan, Irena Galić, Hrvoje Leventić, and Krešimir Romić. 2021.  Whole Heart Segmentation Using 3D FM-Pre-ResNet Encoder–Decoder Based Architecture with Variational Autoencoder Regularization.  Applied Sciences 11, 9 (2021), 3912.
  * Han et al. (2021) Changhee Han, Leonardo Rundo, Kohei Murao, Tomoyuki Noguchi, Yuki Shimahara, Zoltán Ádám Milacski, Saori Koshino, Evis Sala, Hideki Nakayama, and Shin’ichi Satoh. 2021.  MADGAN: Unsupervised medical anomaly detection GAN using multiple adjacent brain MRI slice reconstruction.  BMC bioinformatics 22, 2 (2021), 1–20.
  * He et al. (2022) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022\.  Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16000–16009.
  * He et al. (2017) Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017.  Mask r-cnn. In Proceedings of the IEEE international conference on computer vision. 2961–2969.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
  * Hernandez (2014) Monica Hernandez. 2014\.  Gauss–Newton inspired preconditioned optimization in large deformation diffeomorphic metric mapping.  Physics in Medicine & Biology 59, 20 (2014), 6085\.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017\.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems 30 (2017).
  * Hinton (1994) GE Hinton. 1994\.  RS n emel. Autoencoders, minimum description length, and Helmholtz free energy.  J. D. C owan, G. Tesauro, and J. Alspector, editors, dvances in£ e u ra l I n f ormation rocessin g ystems q. Morgan K aufmann P ublishers, San Francisco, C A (1994).
  * Hinton et al. (2012) Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. 2012\.  Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups.  IEEE Signal processing magazine 29, 6 (2012), 82–97.
  * Hinz et al. (2021) Tobias Hinz, Matthew Fisher, Oliver Wang, and Stefan Wermter. 2021.  Improved techniques for training single-image gans. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 1300–1309.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising diffusion probabilistic models.  Advances in Neural Information Processing Systems 33 (2020), 6840–6851.
  * Ho et al. (2022) Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. 2022\.  Cascaded Diffusion Models for High Fidelity Image Generation.  J. Mach. Learn. Res. 23 (2022), 47–1.
  * Hodge et al. (2009) S Hodge, C Haselgrove, D Kennedy, and J Frazier. 2009\.  CANDIShare: A Resource for Pediatric Neuroimaging Data. In Front. Neuroinform. Conference Abstract: Neuroinformatics.
  * Holmes et al. (2015) Avram J Holmes, Marisa O Hollinshead, Timothy M O’keefe, Victor I Petrov, Gabriele R Fariello, Lawrence L Wald, Bruce Fischl, Bruce R Rosen, Ross W Mair, Joshua L Roffman, et al. 2015\.  Brain Genomics Superstruct Project initial data release with structural, functional, and behavioral measures.  Scientific data 2, 1 (2015), 1–16.
  * Hong et al. (2021) Sungmin Hong, Razvan Marinescu, Adrian V Dalca, Anna K Bonkhoff, Martin Bretzner, Natalia S Rost, and Polina Golland. 2021\.  3d-stylegan: A style-based generative adversarial network for generative modeling of three-dimensional medical images.  In Deep Generative Models, and Data Augmentation, Labelling, and Imperfections. Springer, 24–34.
  * Hongtao et al. (2020) Zhang Hongtao, Yuki Shinomiya, and Shinichi Yoshida. 2020\.  3D brain MRI reconstruction based on 2D super-resolution technology. In 2020 IEEE international conference on systems, man, and cybernetics (SMC). IEEE, 18–23.
  * Hotter et al. (2009) Benjamin Hotter, Sandra Pittl, Martin Ebinger, Gabriele Oepen, Kati Jegzentis, Kohsuke Kudo, Michal Rozanski, Wolf U Schmidt, Peter Brunecker, Chao Xu, et al. 2009\.  Prospective study on the mismatch concept in acute stroke patients within the first 24 h after symptom onset-1000Plus study.  BMC neurology 9, 1 (2009), 1–8.
  * Hou et al. (2021) Yutai Hou, Yongkui Lai, Yushan Wu, Wanxiang Che, and Ting Liu. 2021.  Few-shot learning for multi-label intent detection. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 13036–13044.
  * Hu et al. (2020) Shengye Hu, Wen Yu, Zhuo Chen, and Shuqiang Wang. 2020\.  Medical image reconstruction using generative adversarial network for Alzheimer disease assessment with class-imbalance problem. In 2020 IEEE 6th International Conference on Computer and Communications (ICCC). IEEE, 1323–1327.
  * Huang et al. (2020) Yawen Huang, Feng Zheng, Runmin Cong, Weilin Huang, Matthew R Scott, and Ling Shao. 2020\.  Mcmt-gan: multi-task coherent modality transferable gan for 3d brain image synthesis.  IEEE Transactions on Image Processing 29 (2020), 8187–8198.
  * Iqbal et al. (2022) Ahmed Iqbal, Muhammad Sharif, Mussarat Yasmin, Mudassar Raza, and Shabib Aftab. 2022.  Generative adversarial networks and its applications in the biomedical image segmentation: a comprehensive survey.  International Journal of Multimedia Information Retrieval (2022), 1–36.
  * iSeg 2017 (2022) iSeg 2017. 2022\.  6-month infant MRI brain segmentation.  (2022).  https://iseg2017.web.unc.edu/reference/
  * Isola et al. (2017) Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017.  Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1125–1134.
  * Jaderberg et al. (2015) Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. 2015\.  Spatial transformer networks.  Advances in neural information processing systems 28 (2015).
  * Jeong et al. (2022) Jiwoong J Jeong, Amara Tariq, Tobiloba Adejumo, Hari Trivedi, Judy W Gichoya, and Imon Banerjee. 2022\.  Systematic review of generative adversarial networks (gans) for medical image classification and segmentation.  Journal of Digital Imaging, 1–16.
  * Ji et al. (2012) Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 2012\.  3D convolutional neural networks for human action recognition.  IEEE transactions on pattern analysis and machine intelligence 35, 1 (2012), 221–231.
  * Joyce and Kozerke (2019) Thomas Joyce and Sebastian Kozerke. 2019.  3D medical image synthesis by factorised representation and deformable model learning. In International Workshop on Simulation and Synthesis in Medical Imaging. Springer, 110–119.
  * Karras et al. (2017) Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2017\.  Progressive growing of gans for improved quality, stability, and variation.  arXiv preprint arXiv:1710.10196 (2017).
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila. 2019.  A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 4401–4410.
  * Karras et al. (2020) Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2020\.  Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 8110–8119.
  * Kazeminia et al. (2020) Salome Kazeminia, Christoph Baur, Arjan Kuijper, Bram van Ginneken, Nassir Navab, Shadi Albarqouni, and Anirban Mukhopadhyay. 2020.  GANs for medical image analysis.  Artificial Intelligence in Medicine 109 (2020), 101938.
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT. 4171–4186.
  * Kim and Ye (2022) Boah Kim and Jong Chul Ye. 2022.  Diffusion Deformable Model for 4D Temporal Medical Image Generation.  arXiv preprint arXiv:2206.13295 (2022).
  * Kingma and Welling (2013) Diederik P Kingma and Max Welling. 2013.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114 (2013).
  * Klein et al. (2017) Arno Klein, Satrajit S Ghosh, Forrest S Bao, Joachim Giard, Yrjö Häme, Eliezer Stavsky, Noah Lee, Brian Rossa, Martin Reuter, Elias Chaibub Neto, et al. 2017\.  Mindboggling morphometry of human brains.  PLoS computational biology 13, 2 (2017), e1005350.
  * Kolarik et al. (2021) Martin Kolarik, Radim Burget, Carlos M Travieso-Gonzalez, and Jan Kocica. 2021.  Planar 3D Transfer Learning for End to End Unimodal MRI Unbalanced Data Segmentation. In 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 6051–6058.
  * Kopuklu et al. (2019) Okan Kopuklu, Neslihan Kose, Ahmet Gunduz, and Gerhard Rigoll. 2019.  Resource efficient 3d convolutional neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. 0–0.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012\.  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems 25 (2012).
  * Kruthika et al. (2019) KR Kruthika, HD Maheshappa, Alzheimer’s Disease Neuroimaging Initiative, et al. 2019\.  CBIR system using Capsule Networks and 3D CNN for Alzheimer’s disease diagnosis.  Informatics in Medicine Unlocked 14 (2019), 59–68.
  * Kuijf et al. (2019) Hugo J Kuijf, J Matthijs Biesbroek, Jeroen De Bresser, Rutger Heinen, Simon Andermatt, Mariana Bento, Matt Berseth, Mikhail Belyaev, M Jorge Cardoso, Adria Casamitjana, et al. 2019\.  Standardized assessment of automatic segmentation of white matter hyperintensities and results of the WMH segmentation challenge.  IEEE transactions on medical imaging 38, 11 (2019), 2556–2568.
  * Kwon et al. (2019) Gihyun Kwon, Chihye Han, and Dae-shik Kim. 2019.  Generation of 3D brain MRI using auto-encoding generative adversarial networks. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 118–126.
  * LaMontagne et al. (2019) Pamela J LaMontagne, Tammie LS Benzinger, John C Morris, Sarah Keefe, Russ Hornbeck, Chengjie Xiong, Elizabeth Grant, Jason Hassenstab, Krista Moulder, Andrei G Vlassenko, et al. 2019\.  OASIS-3: longitudinal neuroimaging, clinical, and cognitive dataset for normal aging and Alzheimer disease.  MedRxiv (2019).
  * Lan et al. (2020) Haoyu Lan, Arthur W Toga, Farshid Sepehrband, Alzheimer Disease Neuroimaging Initiative, et al. 2020\.  SC-GAN: 3D self-attention conditional GAN with spectral normalization for multi-modal neuroimaging synthesis.  bioRxiv (2020).
  * Larochelle and Murray (2011) Hugo Larochelle and Iain Murray. 2011.  The neural autoregressive distribution estimator. In Proceedings of the fourteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 29–37.
  * LeCun (1987) Yann LeCun. 1987\.  PhD thesis: Modeles connexionnistes de l’apprentissage (connectionist learning models).  (1987).
  * LeCun et al. (1998) Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998.  Gradient-based learning applied to document recognition.  Proc. IEEE 86, 11 (1998), 2278–2324.
  * Lee et al. (2022) Jihyun Lee, Minhyuk Sung, Hyunjin Kim, and Tae-Kyun Kim. 2022\.  Pop-Out Motion: 3D-Aware Image Deformation via Learning the Shape Laplacian. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18532–18541.
  * Lell and Kachelrieß (2020) Michael M Lell and Marc Kachelrieß. 2020.  Recent and upcoming technological developments in computed tomography: high speed, low dose, deep learning, multienergy.  Investigative radiology 55, 1 (2020), 8–19.
  * Lesjak et al. (2018) Žiga Lesjak, Alfiia Galimzianova, Aleš Koren, Matej Lukin, Franjo Pernuš, Boštjan Likar, and Žiga Špiclin. 2018\.  A novel public MR image dataset of multiple sclerosis patients with lesion segmentations based on multi-rater consensus.  Neuroinformatics 16, 1 (2018), 51–63.
  * Li and Wand (2016) Chuan Li and Michael Wand. 2016.  Precomputed real-time texture synthesis with markovian generative adversarial networks. In European conference on computer vision. Springer, 702–716.
  * Li et al. (2020) Shuailin Li, Chuyu Zhang, and Xuming He. 2020.  Shape-aware semi-supervised 3D semantic segmentation for medical images. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 552–561.
  * Liao et al. (2018) Haofu Liao, Yucheng Tang, Gareth Funka-Lea, Jiebo Luo, and Shaohua Kevin Zhou. 2018.  More knowledge is better: Cross-modality volume completion and 3D+ 2D segmentation for intracardiac echocardiography contouring. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 535–543.
  * Liew et al. (2018) Sook-Lei Liew, Julia M Anglin, Nick W Banks, Matt Sondag, Kaori L Ito, Hosung Kim, Jennifer Chan, Joyce Ito, Connie Jung, Nima Khoshab, et al. 2018\.  A large, open source dataset of stroke anatomical brain images and manual lesion segmentations.  Scientific data 5, 1 (2018), 1–11.
  * Lin (2020) Wanyun Lin. 2020\.  Synthesizing missing data using 3D reversible GAN for alzheimer’s disease. In Proceedings of the 2020 international symposium on artificial intelligence in medical sciences. 208–213.
  * Lin et al. (2021) Wanyun Lin, Weiming Lin, Gang Chen, Hejun Zhang, Qinquan Gao, Yechong Huang, Tong Tong, Min Du, and Alzheimer’s Disease Neuroimaging Initiative. 2021\.  Bidirectional mapping of brain MRI and PET with 3D reversible GAN for the diagnosis of Alzheimer’s disease.  Frontiers in Neuroscience 15 (2021), 646013.
  * Liu et al. (2019b) Jing Liu, Pengyu Yin, Xiaohui Wang, Wei Yang, and Kun Cheng. 2019b.  Glioma subregions segmentation with a discriminative adversarial regularized 3D Unet. In Proceedings of the Third International Symposium on Image Computing and Digital Medicine. 269–273.
  * Liu et al. (2019a) Ming-Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. 2019a.  Few-shot unsupervised image-to-image translation. In Proceedings of the IEEE/CVF international conference on computer vision. 10551–10560.
  * Liu et al. (2019a) Shengfeng Liu, Yi Wang, Xin Yang, Baiying Lei, Li Liu, Shawn Xiang Li, Dong Ni, and Tianfu Wang. 2019a.  Deep learning in medical ultrasound analysis: a review.  Engineering 5, 2 (2019), 261–275.
  * Liu et al. (2021) Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. 2021.  Self-supervised learning: Generative or contrastive.  IEEE Transactions on Knowledge and Data Engineering (2021).
  * Liu et al. (2022) Yanbin Liu, Girish Dwivedi, Farid Boussaid, Frank Sanfilippo, Makoto Yamada, and Mohammed Bennamoun. 2022.  Inflating 2D Convolution Weights for Efficient Generation of 3D Medical Images.  arXiv preprint arXiv:2208.03934 (2022).
  * Liu et al. (2019b) Y Liu, J Lee, M Park, S Kim, E Yang, SJ Hwang, and Y Yang. 2019b.  Learning to propagate labels: Transductive propagation network for few-shot learning. In 7th International Conference on Learning Representations, ICLR 2019.
  * Liu et al. (2020) Yilin Liu, Brendon M Nacewicz, Gengyan Zhao, Nagesh Adluru, Gregory R Kirk, Peter A Ferrazzano, Martin A Styner, and Andrew L Alexander. 2020.  A 3D fully convolutional neural network with top-down attention-guided refinement for accurate and robust automatic segmentation of amygdala and its subnuclei.  Frontiers in Neuroscience 14 (2020), 260.
  * Liu et al. (2022) Zhihua Liu, Lei Tong, Long Chen, Zheheng Jiang, Feixiang Zhou, Qianni Zhang, Xiangrong Zhang, Yaochu Jin, and Huiyu Zhou. 2022.  Deep learning based brain tumor segmentation: a survey.  Complex & Intelligent Systems (2022), 1–26.
  * Lundervold and Lundervold (2019) Alexander Selvikvåg Lundervold and Arvid Lundervold. 2019.  An overview of deep learning in medical imaging focusing on MRI.  Zeitschrift für Medizinische Physik 29, 2 (2019), 102–127.