11institutetext: University of Florida, Gainesville, FL 32611-6120, USA
11email<EMAIL_ADDRESS>

# Hybrid Generative Models for Two-Dimensional Datasets

Hoda Shajari🖂 Jaemoon Lee  Sanjay Ranka  Anand Rangarajan

###### Abstract

Two-dimensional array-based datasets are pervasive in a variety of domains.
Current approaches for generative modeling have typically been limited to
conventional image datasets and performed in the pixel domain which does not
explicitly capture the correlation between pixels. Additionally, these
approaches do not extend to scientific and other applications where each
element value is continuous and is not limited to a fixed range. In this
paper, we propose a novel approach for generating two-dimensional datasets by
moving the computations to the space of representation bases and show its
usefulness for two different datasets, one from imaging and another from
scientific computing. The proposed approach is general and can be applied to
any dataset, representation basis, or generative model. We provide a
comprehensive performance comparison of various combinations of generative
models and representation basis spaces. We also propose a new evaluation
metric which captures the deficiency of generating images in pixel space.

enerative Models, Image Representation Bases, Normalizing Flows, Independent
Component Analysis, Generative Adversarial Networks

###### Keywords:

G

##  1 Introduction

The high volume and unique requirements of scientific image datasets
necessitate the development of novel approaches for data modeling. The bedrock
assumption of all modeling methodologies is the existence of spatiotemporal
homogeneities in the data which can be exploited. However, in contrast to two-
dimensional image modeling, scientific data are underpinned by unusual
geometries and topologies. This “exotic setting” has to be leveraged and
addressed by machine learning methods in their quest to find homogeneities
which in turn can be efficiently exploited using representation bases.
Additionally, unlike image datasets where pixel values are discrete and within
a certain range, the elements of scientific datasets are continuous and can
vary for each data point. In this paper, we propose a novel approach for
modeling the probability distribution of two-dimensional datasets while
developing a new measure for evaluating the models. The proposed approach can
be applied to image and scientific datasets, with elements that are either
discrete or continuous valued. Generative models in machine learning have
drawn significant attention with many applications in different fields,
including, but not limited to, computer vision, and physics-based simulations
for scientific datasets. The importance of generative modeling and
approximating data distributions stems from the fact that unlabeled data are
relatively abundant compared to labeled data, and this has applications in
density estimation, outlier detection, and reinforcement learning. Deep
generative modeling also has emerged during the bloom of deep learning and
takes advantage of advances in computational power [16, 18]. However, these
models have not leveraged classical methods of data representation. These
models usually learn the probability distribution of the images directly in
pixel space, which is costly and inefficient while ignoring 50 years of image
representation bases used in the compression literature. Furthermore, learning
the distribution of the data in pixel space does not leverage the correlation
information among pixels.

Representation basis techniques aim to transform data in such a way that
useful aspects of data, for example statistical properties, are captured in
the transformed space. Principal Component Analysis (PCA), Independent
Component Analysis (ICA), and tensor decompositions using the higher order SVD
(which we henceforth encapsulate as the Tucker decomposition for the sake of
convenience) are among the widely used methods in this area. The other utility
of representation bases is dimensionality reduction, which can be considered
as a kind of lossy compression, i.e., it is possible to represent the data
with a subset of coefficients with desired accuracy. Therefore, dimensionality
reduction also brings a compression aspect into our approach.

We propose an approach to integrate image representation basis techniques in
generative image modeling and perform a comparison among three generative
models—generative adversarial networks (GANs), normalizing flows (NFs), and
Gaussian mixture modeling (GMM)—and analyze their performance. The results
suggest that this is a promising direction to pursue for efficient two-
dimensional dataset generative modeling, in particular for applications where
resources are scarce and speed of training matters. We summarize the
contributions of our work below:

  * •

We propose an approach for two-dimensional datasets which exploits
representation bases to capture the correlation among elements explicitly and
therefore makes the generation process fast and efficient. This approach is
general and can be applied to image and scientific datasets where the
underlying data are respectively discrete with a fixed range and continuous
with a free range.

  * •

We propose a new quantitative metric to compare the performance of our
approach for different choices of generative models and representation bases.
This metric seems to capture the quality of the learned probability
distribution better than conventional metrics, especially for scientific data.

The rest of the paper is outlined as follows. In section 2, we cover previous
work on generative models utilizing representation bases or compression
concepts. In section 3, we provide an overview of representation basis
approaches used in this work. In section 4, our methodology for hybrid
generative modeling is outlined. Implementation details and experiments are
discussed in section 5. Section 6 concludes the paper.

##  2 Previous Work

Learning the probability distribution of datasets is a long-standing problem,
and generative models in machine learning constitute an important class of
models with a rich literature. The Gaussian mixture model (GMM) is one of the
important and classical models for generative modeling [21] while deep
generative models rely on multilayer perceptrons (with deep architectures) for
learning the data distribution.

Despite the importance of image representation bases and their abundant
application in the compression literature, there are very few approaches which
learn image probability distributions by marrying image representation bases
(thereby moving away from pixel space) and deep learning-based generative
models. Our proposed work therefore bridges the gap between image
representation and deep learning-based image generation. As we will
demonstrate in section 4, our approach combines parametric and nonparametric
modules where the parametric component is based on representation bases, while
the nonparametric module is based on machine learning methods.

Generative Latent Optimization (GLO) [3] was proposed as a deep generative
model which learns a deep CNN generator to map latent vectors to data by a
Laplacian pyramid loss function while forcing latent representations to lie on
a unit sphere. In this model, however, it is not possible for the generator to
randomly sample from a known distribution. Implicit Maximum Likelihood
Estimation (IMLE) [20] uses a non-adversarial approach for discovering the
mapping between two densities. In this model, latent variables are mapped into
image space via a generator and for each training image, the nearest generated
image is found such that the $\ell_{2}$ distance between the image and mapping
is minimized and the generator is repeatedly optimized via a nearest neighbor-
based loss. IMLE optimization is costly and the generated images are typically
blurry. The work in [12] proposed the idea of combining GLO and IMLE to learn
a mapping for projecting images into a spherical latent space and learning a
network for mapping sampled points from latent space to pixel image space in a
non-adversarial fashion. Generative Latent Flow [25] learns the latent space
of data via an autoencoder and then maps the distribution of latent variables
to i.i.d. noise distributions. In the wavelet domain, SWAGAN [8] proposed a
wavelet-based progressive GAN for image generation which improves visual
quality by enforcing a frequency-aware latent representation. Our proposed
method is different from the aforementioned approaches because it can be
incorporated into any generative model and therefore allows for sampling from
a known distribution whenever necessary. Furthermore, it has a parametric
module and when used within a GAN architecture, it is trained via an
adversarial loss.

At the intersection of compression and deep generative models, Agustsson et
al. [2] proposed a framework based on GANs for generating images at lower
bitrates. The model learns an encoder which includes a quantization module
which is trained in combination with a multiscale discriminator. Kang et al.
[16] proposed a framework for generating JPEG images via GANs. They proposed a
generator with different layers for chroma sampling and residual blocks. Our
approach is also different from these approaches because image compression
concepts are directly used in the form of representation bases (like Tucker
etc.) coupled with dimensionality reduction.

The Tucker decomposition [23], PCA [15], and ICA [14] are among the widely-
used approaches which linearly transform data into a new space where data can
be presented in a more structured way and more efficiently represented.
Dimensionality reduction is an important byproduct of representation bases.
Choosing a subset of coefficients corresponding to the representation can
result in data compression and has been extensively used in the literature. As
discussed earlier, our use of these methods is for the purpose of converting
from the original data to a new space that captures the correlation between
elements as well as providing an efficient representation for further
processing. Therein lies the novelty of our work. As far as we know, this is
the first work that conducts a comprehensive comparison at the intersection of
generative models and representation bases while leveraging recent advances in
adversarial learning.

##  3 Representation Bases

Finding a proper representation of random multivariate data is a key to many
domains [14]. Linear transformations have specifically been of interest due to
their conceptual and computational simplicity. The following techniques have
been used in our work:

  1. 1.

Principal Component Analysis (PCA): PCA linearly transforms the data by
discovering orthogonal projections of high variance. Given a set of vectors
$\\{x_{i}\\}_{i=1}^{N},\>x_{i}\in\mathbb{R}^{D}$, a correlation matrix is
computed as $C=\Sigma_{i=1}^{N}{x_{i}x_{i}^{T}}$, which has the following
eigen decomposition: $C=E\Lambda
E^{T}=E\Lambda^{\frac{1}{2}}\Lambda^{\frac{1}{2}T}E^{T}=E\Lambda^{\frac{1}{2}}(E\Lambda^{\frac{1}{2}})^{T}$,
where $\Lambda$ is a diagonal matrix of eigenvalues, and their corresponding
eigenvectors constitute the columns of $E$. Dimensionality reduction is
performed by projecting data onto eigenvectors corresponding to the first $d$
maximum eigenvalues, which captures the maximum variance and is scaled with
corresponding eigenvalues, i.e., $y_{i}=Fx_{i}$, where $F$ is the top-left
block of $(E\Lambda^{\frac{1}{2}})^{T}$ with dimension $D\times d$. Data
reconstruction is performed via the operation of $F^{-1}$ on the obtained
coefficients.

  2. 2.

Independent Component Analysis (ICA): ICA attempts to decompose multivariate
data into maximally independent non-Gaussian components. Such a representation
seems to be able to capture the essential structure of the data and provide a
suitable representation which can be taken advantage of in neural networks
[14]. FastICA, used here, introduced a different measure for maximizing the
non-Gaussianity of rotated components [13].

  3. 3.

Tucker Decomposition: The Tucker decomposition decomposes a tensor $T$ of
order $N$ into a core tensor with the same order and $N$ unitary matrices. It
is viewed as a higher order singular value decomposition (HOSVD). If we
consider an image dataset as a tensor $T\in\mathbb{R}^{d_{1}\times d_{2}\times
d_{3}}$ of order $3$, its Tucker decomposition is
$T=\mathcal{T}\times_{1}U^{(1)}\times_{2}U^{(2)}\times_{3}U^{(3)}$, where
$\mathcal{T}\in\mathbb{R}^{d_{1}\times d_{2}\times d_{3}}$ is the core tensor
which, as a lower rank approximation of $T$, gives a representation basis for
it. Unitary or factor matrices are two-dimensional matrices which help in
projecting $T$ into bases $\mathcal{T}$. The Tucker decomposition is widely
used in compression by considering a subset of coefficients which carry most
of the information in the dataset and eliminating lower rank coefficients,
which typically has no adverse affect on tensor reconstruction.

We also utilize the Discrete Wavelet Transform (DWT) as a representation basis
for a “held out” model. We use the DWT to set up a probabilistic model which
can act as a basis for comparison for all generative approaches. The DWT
offers a suitable and general basis for image representation which captures
both frequency and location information. Therefore, it is highly viable as a
benchmark model. We calculated DWT coefficients of datasets with a symmetric
and biorthogonal $1.3$ scaling function. We trained a Gaussian mixture model
on all DWT coefficients (DWT-GMM) except the block of high frequency
coefficients. The DWT-GMM is used as a benchmark for all generative models and
is not used as a separate generative approach (but we plan to explore this
possibility in future work).

##  4 Methodology

Our two stage approach comprising representation basis projection and deep
learning is applicable to general 2D datasets. Below, we set up a cross-
product of approaches wherein representation bases are paired with deep
generative models. While we have elected not to explore variational
autoencoders (VAEs) in the present work, this can be easily accommodated in
the future.

  1. 1.

Data Projection: We begin by projecting images and two-dimensional datasets
into a representation basis space introduced in the previous section.
Depending on the nature of the dataset, generative model, or representation
basis approach, data preprocessing steps and some model customization are
required to improve the results. In section 5, we explain the preprocessing
steps or model specifications adopted for the datasets used in this study.

  2. 2.

Generative Modeling: Generative models are applied to learn the distribution
of a _subset_ of transformed coefficients obtained via one of the
dimensionality reduction procedures detailed in section 3. This is an
efficient use of the compression aspect of the representation basis which
makes the generative process fast and efficient. This way, the focus of
generative modeling shifts from learning the distribution of data in pixel
space to that of the distribution of coefficients in a more informative and
structured space.

The generative models used in this work for structured image generation are
GANs, NFs, and GMMs. The reason for this choice of models is the different
approaches they take towards learning the data distribution. These models are
briefly outlined below. For detailed explanation of generative models and
their variants, please see [9, 19].

###  4.0.1 GANs

are deep generative models which have shown promising results in generating
high-resolution images [9]. GANs are composed of two building blocks:
generator ($G$) and discriminator ($D$) networks which are trained in an
adversarial fashion to defeat each other. A GAN is formulated as a minimax
zero-sum game in which the generator and discriminator try to optimize the
value function $V$ from their own perspective:

$\begin{split}\min_{G}\max_{D}V(G,D)&=E_{y\sim p_{\mathbf{y}}}[\log
D(y)]+E_{z\sim p_{\mathbf{z}}}[\log(1-D(G(z)))],\end{split}$ (1)

where $p_{\mathbf{z}}$ is a predefined prior for the input noise variable $z$,
and $p_{\mathbf{y}}$ is the true distribution of the data. Despite their
impressive results on learning complex data distributions and generating
natural-looking images, GANs cannot perform inference and evaluation of the
probability density of new images and datasets—especially important in the
domain of scientific datasets.

###  4.0.2 NFs

were proposed as a generative model based on random variable transformations
to approximate a tractable probability distribution such that sampling and
inference is exact and efficient [22]. The basic idea of NFs is to transform a
simple probability distribution (typically Gaussian) into a complex one via
learning a sequence of invertible and differentiable mappings (bijectors).
This is the generative direction. Applying a chain of mappings (bijectors)
$f_{k}$, $k=1,2,\ldots,K$ on the random variable $\mathbf{z}_{0}\sim
p_{0}(z_{0})$ results in a random variable $\mathbf{z}_{K}=f_{K}\circ
f_{K-1}\circ\ldots\circ f_{1}(\mathbf{z}_{0})$ with probability distribution
$p_{K}$:

$p_{K}(\mathbf{z}_{K})=p_{0}(\mathbf{z}_{0})\prod_{k=0}^{K}\bigg{|}\text{det}\frac{\partial
f_{k}}{\partial\mathbf{z}_{k-1}}\bigg{|}^{-1}.$ (2)

In order for these transformations to be practical, determinants of their
Jacobians should be easy to compute. Some of the suggested approaches are
RealNVP [7], Glow [17], and FFJORD [10]. To implement NFs, we used the
probability library of TensorFlow [1] and its distributions module. Bijectors
were also trained by the FFJORD module in TensorFlow.

###  4.0.3 The GMM

is a parametric method for probability density function estimation. The
density function is represented as a weighted sum of Gaussian components [21].
The Gaussian mixture model represents data as normally distributed
subpopulations with a hidden, unknown digital membership. The density of $X$
is formulated as a weighted sum of $K$ Gaussian distribution
$N(\mu_{k},\Sigma_{k})$ as follows:

$p(x|\pi,\mu,\Sigma)=\sum_{k=1}^{K}\pi_{k}N(x;\mu_{k},\Sigma_{k});\;\;\text{with}\sum_{k=1}^{K}\pi_{k}=1.$
(3)

The parameters of the GMM model are estimated by maximum likelihood estimation
(MLE). Typically, an iterative Expectation-Maximization (EM) algorithm [6] is
applied which turns out to be reasonably efficient for this MLE problem.

##  5 Experimental Results

In this section, we detail the experimental evaluation of the two-step process
described in the previous section on two different datasets, one from image
processing and the other from scientific computing. To compare the performance
of generative models in representation bases, we executed a set of experiments
on the cross product of models and representation bases for these datasets.

With most image datasets, the pixel intensities range from $0$ to $255$ and
are frequently normalized to a different range like $[0,1]$ for training.
Unlike image datasets, scientific datasets are not visually meaningful to
human perception. Hence, measures like FID [11] developed for image quality of
GANs based on the Inception v3 model are not immediately applicable to
scientific datasets like XGC, where each two-dimensional slice has a different
range, so _there is no unified range_ in this dataset like there is in typical
image datasets. Based on this observation, we propose a likelihood-based
metric that we believe will be more suitable for this and other similar
scientific datasets.

###  5.0.1 Datasets.

We experimented with two datasets: Fashion MNIST [24] and XGC [5]. Fashion
MNIST is a standard, curated and widely used dataset consisting of ten classes
of clothing items. XGC consists of 16 planes corresponding to a doughnut’s
cross-sections. Each plane consists of $12,458$ nodes with each node
representing a histogram of perpendicular and parallel velocities of photons
at specific checkpoints (please see Figure 1). The histograms are not
necessarily normalized. The velocity histogram of one of these nodes is
depicted on the left in Figure 1. The goal is to derive a generative model to
simulate the two-dimensional velocity histograms of particles which are
represented as images in a compressed and efficient way.

Figure 1: Depiction of a node in the XGC dataset (left). The $x$ and $y$ axes
of the histogram represent the perpendicular and parallel velocities of
photons, respectively. Samples from dataset (right).

Figure 2: Fashion generated images via ICA-GAN (left) and pixel-GAN (right) at
epoch 10 (upper row) and epoch 50 (lower row). Image samples are randomly
drawn and not cherry-picked. ICA-GAN generates plausible images close to the
dataset from early iterations, with much fewer artifacts in terms of shape and
texture. Pixel-GAN takes many more iterations to converge, with some images
having artifacts.

###  5.0.2 Preprocessing.

Slightly differing approaches were taken for the two datasets for projection
to the PCA/ICA basis. Since the ICA bases generate an unconstrained range of
pixel values, for the Fashion dataset, we first project the image intensities
from a discrete range of $[0,255]$ to the continuous range $[0,1]$ and then
project these intensity values to a wider range using an inverse sigmoid
function $y(x)=\log\frac{x}{\beta(1-x)}$, where $\beta$ is a gradient slope
factor. PCA/ICA is then applied to this new range of values. Since the inverse
sigmoid is not defined at $0$ and $1$, we map all intensities to the interval
$[\epsilon,1-\epsilon]$ for some value of $\epsilon$ and then apply the
inverse sigmoid before applying PCA and ICA. In our experiments, $\epsilon$ is
set to $0.001$. The inverse sigmoid is used because the last nonlinear
activation function in the generator of the GAN architecture for this dataset
is sigmoidal, and therefore, we can match the intensities to the training
data.

For the XGC dataset, the values are normalized numbers of particles in
simulation which have a specific perpendicular and parallel velocity at a
checkpoint. These values are normalized by the mean and standard deviation of
each image separately (essentially a per image $Z$ score). Because each image
has a different range, it impacts the choice of architecture and activation
function in generative models.

###  5.0.3 Generative Models Architecture.

Figure 3 depicts the architecture of a GAN for the XGC dataset. We used
upsampling (conv2DTranspose) layers in the generator with a linear activation
function in the last layer to allow for the range of generated images to be
chosen freely for each image. This way, the generator is constrained to learn
coefficients such that, after image reconstruction (via the transformation
matrix), the values follow an acceptable range that is similar to the training
set. The GAN architecture for Fashion is very similar to XGC, except for the
number of filters and the use of sigmoidal activation instead of linear
activation at the last layer of the generator.

The advantage of representation bases and dimensionality reduction is more
tangible in NFs because these models are computationally expensive: when
input/output dimensions are increased, the number of training parameters grows
rapidly. We considered four layer of bijectors and 50 additional nodes in the
hidden layers of bijectors.

###  5.0.4 Dimensionality Reduction.

Our goal is to learn the distribution of a subset of coefficients as an
efficient approach to data generation. The number of top eigenvalues and
corresponding eigenvectors for each dataset was determined based on the
$\ell_{2}$ distance between the training dataset and reconstructed images. For
Fashion MNIST, more coefficients were needed to meet a certain error
threshold: $324$ and $400$ coefficients were chosen respectively for XGC and
Fashion datasets. We compare the performance of generative models in learning
the distribution of a subset of coefficients for each method via different
measures. We observed that for XGC, PCA, and ICA have similar performance
across the board. For Fashion however, ICA had better performance than PCA,
and therefore, we only focused on the performance of ICA-GAN for this dataset.

###  5.0.5 Metrics.

Many qualitative and quantitative measures have been proposed to evaluate
generated images and learned probability distributions of generative models
[4]. We consider two conventional and widely-used quantitative metrics—Frechet
inception distance (FID) and average log-likelihood (entropy) of samples in
kernel density estimation (KDE) [9] for evaluating the learned distributions
and generated samples from different models. FID was proposed as a statistical
metric to measure the similarity between two distributions. First, by running
the Inception v3 net on real and generated images, high level features (pool3
layer) are extracted as an embedding for images, and then a separate
multivariate Gaussian distribution is fitted to real and generated embeddings.
FID does not seem to be a suitable metric for evaluating scientific dataset
generation. The features extracted from a deep learning network trained on
real images which

Figure 3: The GAN architecture for the XGC dataset; generator upper row and
discriminator lower row. A subset of representation bases is chosen.

are perceptually meaningful to human vision _are not necessarily appropriate_
for scientific data (where perceptual quality is not used). Furthermore, FID
only provides a single scalar measure for the entire dataset and does not take
the actual likelihood of the training set or generated images into account.
For these reasons, we resort to metrics based on probability distributions and
the likelihood of generated samples with respect to a reference model. KDE
directly fits a probability density model to the generated images. We
calculated the average of the negative log-likelihood (NLL) i.e. entropy of
images sampled from learned distributions with respect to this reference
density model (Table 1). However, as mentioned above, it is much more
efficient to learn probability distributions in the space of a representation
basis than in pixel space. These considerations affect our choice as described
below. As mentioned in section 3, we consider a reference model which is
essentially a GMM on DWT coefficients. This model serves as a benchmark and is
_not used_ for generation or sampling. The number of coefficients used is
$3\times 256$ and $3\times 324$ coefficients for Fashion and XGC datasets,
respectively. To assess the learned density distributions via different
models, we use the average of NLL values of sampled images from learned
distributions via GANs, GMM, and NFs in the DWT-GMM space—essentially the DWT
entropy. Furthermore, we compute the $\ell_{1}$ distance between the density
curves obtained via KDE of the NLL values of generated images in the DWT-GMM
model (see Figures 4 and 5). Essentially, this distance is computed between
the density curve of each model and the density curve of the real dataset on
the interval that contains most of its density volume.

###  5.0.6 Results and Discussion

Table 1: Image generative models using representation bases with dimensionality reduction (324 and 400 coefficients: PCA, ICA, Tucker for XGC and Fashion datasets respectively). Numbers $10$ and $50$ in the $4^{\text{th}}$ column denote the learned distribution at that epoch number. Metrics: DWT and KDE entropies (DWT-E and KDE-E respectively), FID and the $\ell_{1}$ distance (scaled by 10-2). For all metrics, lower is better. Dataset | Model | Loss | Target Dist. | DWT-E | KDE-E | FID | $\ell_{1}$
---|---|---|---|---|---|---|---
XGC | GAN | ADV | ICA | $-2,239$ | $-82$ | $5.6$ | $4.0$
PCA | $-2,261$ | $-76$ | $5.6$ | $4.0$
Tucker | $25,745$ | $-33$ | $13.9$ | $5.0$
Pixel | $311,895$ | $207$ | $2.8$ | $5.0$
NF | MLE | ICA | $3,241$ | $477$ | $6.0$ | $4.8$
Tucker | $1,682$ | $1,105$ | $26$ | -
GMM | MLE | ICA | $-1,096$ | $39$ | $5.5$ | $4.3$
Tucker | $-2,474$ | $-455$ | $2.4$ | $3.0$
Fashion | GAN | ADV | ICA 10 | $-2,451$ | $-453$ | $4.9$ | $2.7$
ICA 50 | $-2,550$ | $-450$ | $2.2$ | $2.2$
Pixel 10 | $-1,576$ | $-307$ | $5.0$ | $3.5$
Pixel 50 | $-1,841$ | $-346$ | $1.3$ | $2.6$
Tucker | $-1,146$ | $-333$ | $2.2$ | $4.4$
NF | MLE | ICA | $-1,078$ | $-329$ | $2.9$ | $8.9$
GMM | MLE | ICA | $-1,033$ | $-361$ | $2.3$ | $5.7$
Tucker | $-2,660$ | $-480$ | $21$ | $2.6$

Experimental results for different combinations of modeling and generative
bases are provided in Table 1 for the two datasets. These results show that
ICA-GANs preserve the statistical properties of each dataset despite higher
FID scores compared to equivalent pixel-GANs. The _very high entropy_ of
pixel-GAN for the XGC dataset shows that the learned distribution of data via
pixel-GAN is far from the true distribution despite generating images which
are visually similar to the training set. This indicates that generating a
scientific dataset in pixel space may not be a reasonable approach. ICA-GAN
(at epoch 50) had the best performance on the Fashion dataset which is a
curated dataset with images being approximately registered within classes.
Note that we cannot expect scientific data to be pre-registered. The better
performance of the pixel-GAN on Fashion compared to XGC is partly because of
the unified range of pixel intensities for Fashion allowing for the use of a
single sigmoid activation function which confines the generated pixel values
within $[0,1]$. Overall, the results also show that GMMs with a representation
basis (after preprocessing) are powerful generative models for two dimensional
datasets, regardless of whether the data arise from standard imagery or from
scientific simulation.

Figure 2 shows that with a reasonable and simple architecture of GANs for
learning the distribution of ICA coefficients of the Fashion dataset, it is
possible to generate plausible looking images which are mostly texture and
shape artifact-free from early iterations. Furthermore, Figure 4 (and the
$\ell_{1}$ distance in Table 1) also indicate that the generated images by
ICA-GAN has the closest entropy to the Fashion dataset in both DWT-GMM and KDE
benchmark models among deep generative models and representation bases. The
pixel-GAN on the other hand does not produce close-to-dataset images until
later iterations (with an identical discriminator) while many of the images
have artifacts in terms of shape and texture. From Figure 5, it might seem
that the GMM has a better performance compared to the ICA-GAN for XGC data.
However, it is important to note that less than $\frac{2}{3}$ of the ICA-GMM
samples fall in the negative range of NLL while ICA-GAN shows a more
homogeneous behaviour and hence lower entropy.

Figure 4: Fashion negative log-likelihood (NLL) density distributions in the
DWT-GMM benchmark model. For better demonstration, only samples with negative
NLL are plotted (data density concentration). ICA-GAN samples depict NLL
values near NLL of Fashion dataset. Figure 5: XGC negative log-likelihood
(NLL) density distributions in the DWT-GMM benchmark model. ICA-GMM seems to
have better NLLs (lower entropy) however, GANs perform better on average since
ICA-GMM has only part of samples ($7\text{K}$ out of $12\text{K}$) in negative
range.

##  6 Conclusions

We proposed a framework for fast and efficient image generation that combines
a representation basis approach with deep generative modeling. Our rationale
was that learning a basis for data which preserves the statistical structure
and correlation among image pixels can be a useful preprocessing step for the
development of generative models. Furthermore, representation bases can be
deployed for data compression during generation which is a boon for
computationally intensive generative modeling frameworks. Immediate future
work will focus on using over-complete dictionaries and coefficient
compression within generative modeling.

## Acknowledgments

This material is based upon work supported by the U.S. Department of Energy,
Office of Advanced Scientific Computing Research, Scientific Discovery through
Advanced Computing (SciDAC) program under Award Number DE-SC0021320.

## References

  * [1] Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al.: TensorFlow: A system for large-scale machine learning. In: 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’16). pp. 265–283 (2016)
  * [2] Agustsson, E., Tschannen, M., Mentzer, F., Timofte, R., Gool, L.V.: Generative adversarial networks for extreme learned image compression. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 221–231 (2019)
  * [3] Bojanowski, P., Joulin, A., Lopez-Paz, D., Szlam, A.: Optimizing the latent space of generative networks. arXiv preprint arXiv:1707.05776 (2017)
  * [4] Borji, A.: Pros and cons of GAN evaluation measures. Computer Vision and Image Understanding 179, 41–65 (2019)
  * [5] Cole, M.D., Hager, R., Moritaka, T., Dominski, J., Kleiber, R., Ku, S., Lazerson, S., Riemann, J., Chang, C.: Verification of the global gyrokinetic stellarator code XGC-S for linear ion temperature gradient driven modes. Physics of Plasmas 26(8), 082501 (2019)
  * [6] Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B 39(1), 1–22 (1977)
  * [7] Dinh, L., Sohl-Dickstein, J., Bengio, S.: Density estimation using real NVP. arXiv preprint arXiv:1605.08803 (2016)
  * [8] Gal, R., Cohen, D., Bermano, A., Cohen-Or, D.: SWAGAN: A style-based wavelet-driven generative model. arXiv preprint arXiv:2102.06108 (2021)
  * [9] Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. arXiv:1406.2661 (2014)
  * [10] Grathwohl, W., Chen, R.T., Bettencourt, J., Sutskever, I., Duvenaud, D.: FFJORD: Free-form continuous dynamics for scalable reversible generative models. arXiv:1810.01367 (2018)
  * [11] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local Nash equilibrium. arXiv preprint arXiv:1706.08500 (2017)
  * [12] Hoshen, Y., Li, K., Malik, J.: Non-adversarial image synthesis with generative latent nearest neighbors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5811–5819 (2019)
  * [13] Hyvärinen, A.: Fast and robust fixed-point algorithms for independent component analysis. IEEE Transactions on Neural Networks 10(3), 626–634 (1999)
  * [14] Hyvärinen, A., Oja, E.: Independent component analysis: algorithms and applications. Neural Networks 13(4-5), 411–430 (2000)
  * [15] Jolliffe, I.T., Cadima, J.: Principal component analysis: a review and recent developments. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 374(2065), 20150202 (2016)
  * [16] Kang, B., Tripathi, S., Nguyen, T.Q.: Generating images in compressed domain using generative adversarial networks. IEEE Access 8, 180977–180991 (2020)
  * [17] Kingma, D.P., Dhariwal, P.: GLOW: Generative flow with invertible 1x1 convolutions. arXiv preprint arXiv:1807.03039 (2018)
  * [18] Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. arXiv:1312.6114 (2013)
  * [19] Kobyzev, I., Prince, S., Brubaker, M.: Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence (2020)
  * [20] Li, K., Malik, J.: Implicit maximum likelihood estimation. arXiv:1809.09087 (2018)
  * [21] Reynolds, D.A.: Gaussian mixture models. Encyclopedia of biometrics 741, 659–663 (2009)
  * [22] Rezende, D., Mohamed, S.: Variational inference with normalizing flows. In: International Conference on Machine Learning. pp. 1530–1538. PMLR (2015)
  * [23] Tucker, L.R.: Some mathematical notes on three-mode factor analysis. Psychometrika 31(3), 279–311 (1966)
  * [24] Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)
  * [25] Xiao, Z., Yan, Q., Amit, Y.: Generative latent flow. arXiv preprint arXiv:1905.10485 (2019)


# APPROXIMATING THE FULL-FIELD TEMPERATURE EVOLUTION IN 3D ELECTRONIC SYSTEMS
FROM RANDOMIZED “MINECRAFT” SYSTEMS

MONIKA STIPSITZ1 AND HÈLIOS SANCHIS-ALEPUZ1 1 Silicon Austria Labs GmbH
Inffeldgasse 33, 8010 Graz, Austria
e-mail: {monika.stipsitz<EMAIL_ADDRESS>
###### Abstract

Neural Networks as fast physics simulators have a large potential for many
engineering design tasks. Prerequisites for a wide-spread application are an
easy-to-use workflow for generating training datasets in a reasonable time,
and the capability of the network to generalize to unseen systems. In contrast
to most previous works where training systems are similar to the evaluation
dataset, we propose to adapt the type of training system to the network
architecture. Specifically, we apply a fully convolutional network and, thus,
design 3D systems of randomly located voxels with randomly assigned physical
properties. The idea is tested for the transient heat diffusion in electronic
systems. Training only on random “Minecraft” systems, we obtain good
generalization to electronic systems four times as large as the training
systems (one-step prediction error of $0.07\text{\,}\%$ vs $0.8\text{\,}\%$).

###### keywords:

3D geometries, Convolutional Neural Network, Transient heat equation, Image-
based physics-informed loss

Monika Stipsitz and Hèlios Sanchis-Alepuz

##  1 INTRODUCTION

The idea of using Neural Networks (NNs) as trainable physics simulators has
seen much attention sparked by recent impressive results [1, 2, 3]. Once
trained such a NN could predict physical properties much faster than any
standard simulation method. Potential benefits can be seen in many fields of
application, e.g. enabling large-scale, multi-domain, multi-objective design
optimization, providing close to real-time feedback during a manual design
process, as improved physics engine for computer games and in augmented
reality applications. However, one obstacle are the large datasets required
for the training process of such NNs. To be useful, for instance, in a design
optimization process the time for generating the dataset should be small
compared to the time a standard solver would spend in the optimization phase.

Regarding the data for the training of the NNs there are two main strategies:
(1) Physics-inspired Neural Networks (PINNs) [4], where the dataset is
completely or partially replaced by an objective function based on the
governing partial differential equation. This approach has proven successful
for many applications [5, 6] while convergence problems occurred for many
others [7, 8]. (2) A dataset is generated using a standard physics-based
simulation method like Finite Element Method (FEM), Finite Volume Method, etc.
[1, 2]. These datasets are often similar or slightly smaller versions of the
desired deployment systems.

Figure 1: Overview of the different system types: The NN is trained
exclusively on random “Minecraft” systems of two different types (top). The
generalization capabilities are validated on a set of electronic systems
(bottom). The electronic systems are four times as large as the Minecraft
systems.

In a previous work, we observed that training a NN directly on a training
dataset generated from the desired geometries might not be the best approach
[9]. First, building systems from predefined components limits the geometrical
diversity. The network tended to use the geometric shape of objects (e.g.
small cylindrical) to predict their temperature rather than the physical
properties (heat source, conductivity of surrounding region, etc.). Second,
generating systems of similar size as the desired validation systems is time-
consuming. Moreover, realistic systems in image representation take up most of
the available memory of a standard graphics card. Thus, there is limited space
for the network (and the gradients for the training process). Specifically, it
was found that standard networks are too large. To capture the long-range
correlations found in the heat propagation process a more intricate network
architecture is necessary which, however, makes training the network more
complicated.

Here, we suggest a different type of training dataset to overcome these
problems. We apply a convolutional neural network (CNN) to obtain a fast
approximation of the transient heat distribution in electronic systems. The
major contributions are:

  * •

We investigate three-dimensional electronic systems with complex geometries
consisting of different materials (see Fig. 1, bottom). The material
properties span a wide range (e.g. copper, epoxy). The temperature
distribution within the system is predicted.

  * •

Instead of training directly on electronic systems, we exploit the fact that a
fully convolutional network is used. We designed “Minecraft” systems
consisting of randomly located blocks of randomly selected materials (see Fig.
1, top). The network trains only on these random systems but generalizes well
to the electronic systems. The electronic systems are four times as large as
the training systems. Due to memory limitations it would not be possible to
train directly on the electronic systems.

##  2 METHODS

In this work we focus on the transient heat equation given by

$\rho(\vec{x})c_{P}(\vec{x})\frac{\partial T(\vec{x},t)}{\partial
t}-\vec{\nabla}\cdot\left(k(\vec{x})\vec{\nabla}T(\vec{x},t)\right)-\rho(\vec{x})h(\vec{x})=0~{},$
(1)

where the temperature $T$, the heat source power per mass $h$, and the
material properties density $\rho$, conductivity $k$ and heat capacity
$c_{P}$. $\rho$, $h$, $k$ and $c_{P}$ depend on the position $\vec{x}$ but are
constant over time $t$. All systems exhibit a bottom plate (see Fig. 2, left).
The boundary conditions consist of a constant temperature $T_{\mathrm{ext}}$
at the bottom of the PCB plate, and a Neumann boundary condition on all other
outer surfaces (heat transfer coefficient
$\alpha=$14\text{\,}\mathrm{W}\mathrm{/}\mathrm{(}\mathrm{m}^{2}\mathrm{K}\mathrm{)}$$,
external temperature $T_{\mathrm{ext}}$). Initially, the whole system is set
to a uniform temperature equalling the temperature of the Dirichlet boundary
condition ($T_{0}=T_{\mathrm{ext}}$).

Figure 2: System representation: Starting from a CAD design (left), the
components / the whole structure is split into voxels and combined to form an
image of the original design (center). The input to the network consists of
four channels, where each channel is a 3D image representing the distribution
of the conductivity $k$, heat source power $\rho\,h\,\mathrm{d}V$, initial
temperature $T_{n-1}$ or weighted capacity $\rho c_{\mathrm{P}}$ (right).

A NN is constructed to predict the temperature distribution $T_{n}(\vec{x})$
given the temperature at the previous time step $T_{n-1}(\vec{x})$ with a time
step of $\Delta t=$2\text{\,}\mathrm{s}$$. The temperature is represented as a
3D image, where each of the voxels is given the value of the temperature at
the central points of a regular hexahedral grid with grid size
$0.2\text{\,}\mathrm{m}\mathrm{m}$. A fully convolutional architecture was
chosen for the NN (Fig. 3, more details in [9]). The input to the network are
four channels, with each channel a 3D image containing a physical property
($1/k$, $\rho h~{}\mathrm{d}V$, $\rho c_{P}$), or the previous temperature
distribution $T_{n-1}$ (see Fig. 2, right for an illustration of the
channels). The physical properties are combined as they occur in Eq. 1, for
instance, the second channel $\rho h~{}\mathrm{d}V$ considers the integral
heating power of a voxel with volume $\mathrm{d}V$. The channels are maximum-
normalized using dataset statistics. The convolutional architecture naturally
preserves translational equivariance [10] and directly enables the network to
be applied to larger images than it was trained on. Rotational equivariance
was encouraged by data augmentation; the dataset is augmented by three 90
degree in-plane rotations of the images.

Figure 3: Fully convolutional architecture: Each “Fusion” block consists of 4
parallel 3D convolutions with different dilations $d$ and concatenated
outputs. For each 3D convolutional layer (Conv) and transpose convolutional
layer (ConvTrans) the kernel size $k$, stride $s$ and channel number $C$ of
the output are given. Leaky ReLU is used as activation function where not
stated otherwise. More details on the choice of architecture can be found in
[9].

Three different types of datasets are used within this work (compare Fig. 1):
Two types of random datasets (“Minecraft” systems) which are used for training
and testing, and a third dataset consisting of electronic systems which is
used for validation only. The network will only see the Minecraft systems for
training but should be able to generalize to the electronic systems.

Since the network will see only patterns of voxels, an ideal training system
would consist of completely randomly located voxels with randomly assigned
values. If enough of these systems would be generated, all possible
combinations of voxel values would occur on the level of the kernels of the
network so that it should be able to generalize to any kind of system that can
be defined as a stack of 3D images.

In practice there are, however, a number of limitations on how the training
systems can be designed. First, in order that FEM simulations can be performed
the system has to be uniquely connected. Thus, we decided to grow all systems
on a PCB plate. Second, to randomly create thousands of systems an automatized
workflow is necessary. Specifically, FreeCAD and python were used to generate
the systems. However, it was found that FreeCAD was not well accustomed to
process multiple thousands of independent components. Third, an immense number
of systems would be needed to represent all possible combinations. It is,
thus, more reasonable to generate systems better adjusted to the desired
validation systems.

Training dataset The Minecraft systems should capture the geometric features
of electronic systems. The electronic systems contain small details, e.g.
copper tracks or legs, in the order of individual voxels, as well as larger
components (e.g. molds), and parts (e.g. dies) completely enclosed in other
components. Two types of random systems were generated to cover this variety
of geometrical shapes (Fig. 1): “voxel systems” constructed of individual
voxels where each one is assigned a different random material, and “block
systems” made up of larger boxes leading to larger regions with the same
material properties. To increase the variation in geometrical shapes, the
block systems were generated from randomly sized 3D blocks which were randomly
placed on the PCB. The blocks could be overlapping, in which case the
structure was decomposed into multiple non-overlapping segments.

All random systems were generated with a size of
$6.4\text{\,}\mathrm{m}\mathrm{m}$ per dimension (a quarter of the size of the
electronic systems in the validation dataset). A voxel size of
$0.2\text{\,}\mathrm{m}\mathrm{m}$ was chosen. Each voxel/block is assigned a
homogeneous material. Thus, the voxel size limits the minimal size at which
structures are resolved.

Validation dataset The electronic systems had a size of
$25.6\text{\,}\mathrm{m}\mathrm{m}$ per dimension. They were generated by
randomly placing pre-designed components (one IC, one large and one small
capacitor, and copper layers of different shapes and sizes) on a PCB. Most of
these components have an internal structure, e.g. the large IC consists of a
die, mold base, thermal pad, legs and copper layers.

Finally, the training data for the supervised learning process was generated
using an automatized workflow employing the open-source packages FreeCAD for
system construction, gmsh for meshing and ElmerFem to solve for the
temperature distributions: One of the materials in Tab. 1 was assigned
randomly to each component of a system. (For the electronic systems the
materials were not assigned randomly but the composition of each component was
fixed, e.g. legs of copper, mold of epoxy.)

Table 1: Average material properties: The actual values are chosen randomly per component from a range of $[0.75\mathrm{Avg},1.25\mathrm{Avg}]$. Property | Avg. $k$ | Avg. $c_{P}$ | Avg. $\rho$
---|---|---|---
Unit | $(W/(mK))$ | $(J/(kgK)$ | $(kg/m^{3})$
Silicon | 148 | 705 | 2330
Copper | 384 | 385 | 8930
Epoxy | 0.881 | 952 | 1682
$\mathrm{FR}_{4}$ | 0.25 | 1200 | 1900
$\mathrm{Al}_{2}\mathrm{O}_{3}$ | 35 | 880 | 3890
Aluminium | 148 | 128 | 1930

The uniform initial temperature was chosen randomly from the range $[-40,200]$
°C. Heat sources were assigned randomly to some of the components. The
temperature was extracted at the central points of a hexahedral grid with
element sizes of $0.2\text{\,}\mathrm{m}\mathrm{m}$ $\times$
$0.2\text{\,}\mathrm{m}\mathrm{m}$ $\times$ $0.2\text{\,}\mathrm{m}\mathrm{m}$
and a temporal discretization of $\Delta t=$2\text{\,}\mathrm{s}$$. Thus, the
temperature at a given time step was represented as a 3D image of similar
dimensions as the input channels (compare Fig. 2). Unphysically hot systems
with $T_{\mathrm{max}}>$250\text{\,}\mathrm{K}$$ were discarded based on a
steady-state simulation. For the transient simulations, the systems started in
a uniform temperature state and were heated up until close to the steady-state
(after 15 time steps), from which four time-steps were randomly sampled for
the datasets. In total, 2440 valid Minecraft systems were generated (1129
voxel systems and 1311 block systems). A split of $75\text{\,}\%$ was used for
the train/test set.

###  2.1 Impact of training dataset type

To evaluate the generalization capabilities of the different types of training
datasets, the network was trained with three different Minecraft datasets: (1)
only voxel systems, (2) only block systems, (3) a combined dataset of
comparable size as (1) and (2) to enable a fair comparison. All training
procedures achieved comparable accuracy on the corresponding test datasets.
After training, the network was evaluated on the electronic systems to
evaluate the generalizability.

Fig. 4 reports the average and maximum $L_{1}$ error for the validation
dataset evaluated per system. The best $L_{1}$ error was achieved with the
combined dataset (see Fig. 4, rows 1-3), and also regarding the maximum error
found in the systems the combined dataset was clearly superior to both the
voxel and the block training datasets. The maximum $L_{1}$ error was nearly
$10\text{\,}\%$ larger when trained with only voxel or only block systems
(Fig. 4, right). The slightly worse errors in the combined dataset compared to
the final trained network indicate that the NN can profit from the larger
dataset (compare “Train set: Combined” and “$L_{1}+10^{-4}L_{\mathrm{heq}}$”
in Fig. 4).

Figure 4: Box plot of the $L_{1}$ error (left: average, right: maximum per
system) evaluated for the electronic systems. Rows 1-3 show the impact of
different types of random systems as training datasets. Rows 4-6 evaluate the
effect of different objective function compositions on the generalization
capabilities. For more information see Secs. 2.1 and 2.2, respectively.

Qualitatively, it is informative to look at the worst-case system, i.e. the
system with the largest $L_{1}$ error (Fig. 5). The idea of using two types of
random systems was that the voxel systems would teach the network the
importance of small details in the size of individual voxels while the block
systems would help it to learn the behaviour of larger heat generating blocks.
When trained with the voxel systems, it is clearly notable that the network is
not used to larger areas containing heat sources, leading it to predict much
hotter cores of the capacitors (Fig. 5, left). However, the role of the block
systems is less obvious (Fig. 5, center). Although the prediction of larger
heat generating components improved compared to the voxel systems and the
copper layers show slightly higher errors, large errors are located at
intermediately sized structures. Nevertheless, the combined dataset leads to
smaller, more uniform errors (Fig. 5, right).

Figure 5: Relative error in the electronic system with the worst average
$L_{1}$ error. The image shows only the lower half of the systems so that
errors within the structures are visible. The evaluations were performed with
three differently trained networks, once training using only the voxel systems
(left), once only the block systems (center) and once the combined dataset of
the same size (right). Note, that for visualization purposes the color scale
for the error is capped at $\pm$ $10\text{\,}\%$.

###  2.2 Optimization of learning scheme

Two types of objective functions were evaluated:

  * •

A normalized $L_{1}$ loss given by
$L_{1}=\mathrm{Average}\left(\frac{T^{\mathrm{FEM}}-T^{\mathrm{NN}}}{T^{\mathrm{FEM}}}\right)$,
where $T^{\mathrm{FEM}}$ and $T^{\mathrm{NN}}$ are the baseline temperature
map obtained from a FEM simulation and the predicted temperature map from the
NN, respectively.

  * •

A physics-based loss $L_{\mathrm{heq}}$ defined via an integrated and
discretized version of the heat equation (Eq. 1) inspired by [11]. The
differential operators were evaluated for the same coarse temporal and spatial
discretization used for the dataset. The $L_{\mathrm{heq}}$ was computed as
the sum of the absolute values of this local residual. Due to the coarse
discretization, $L_{\mathrm{heq}}$ evaluated for the FEM solutions was non-
negligible; on average $L_{\mathrm{heq}}(T^{\mathrm{FEM}})=1.95$.

Both objective functions were evaluated only for the image regions containing
components.

Different combinations of these two losses were tested. The network was
trained on the Minecraft systems, and afterwards the generalization
capabilities for the electronic systems was evaluated (Fig. 4, rows 4-6):
Training with $L_{1}$ loss, and training with $L_{1}$ loss plus a small
physics-based regularization (with scaling of $10^{-2}$ or $10^{-4}$) led to
similar average and maximum $L_{1}$ error distributions for the Minecraft
systems (average $L_{1}$ error of $0.07\text{\,}\%$, $0.08\text{\,}\%$ and
$0.07\text{\,}\%$, respectively). In contrast, training only with
$L_{\mathrm{heq}}$ led to much higher errors even on the Minecraft systems
($L_{1}=$10\text{\,}\%$$) and bad generalization to the electronic systems.
This was expected, since the definition of $L_{\mathrm{heq}}$ relies on a very
coarse discretization. However, adding $L_{\mathrm{heq}}$ as a physics-based
regularization term to the $L_{1}$ error significantly improved the
generalization capabilities compared to the $L_{1}$ loss only. It was
previously argued that physics-based regularization terms improve learning
because they add non-locality to the loss [12]. For instance, they could
encourage spatial smoothness often found in physical systems. However, it was
surprising that even a bad approximation of the physical laws can enhance the
learning process.

The results did not improve when random noise was added to the input channels
as suggested in [1].

##  3 RESULTS

The results reported in this section are based on the training setup labelled
“$L_{1}+10^{-4}L_{\mathrm{heq}}$” in Fig. 4. As training dataset a combination
of the voxel and block type random systems were used. All results given for
the Minecraft dataset refer to the test part of the dataset.

###  3.1 One-step prediction

The network was trained on individual time steps only; given the temperature
distribution at the current time step it was asked to predict the temperature
distribution at the next time step. The mean $L_{1}$ error for the one-step
prediction on the Minecraft dataset was $0.07\text{\,}\%$. For the one-step
prediction reasonably good generalization to the electronic systems was
achieved (average $L_{1}$ error of $0.8\text{\,}\%$) taking into account that
the electronic systems are four times as large as the training systems. In
more detail, the worst electronic system had an average error of
$2\text{\,}\%$ while the worst average error for Minecraft systems was
$0.9\text{\,}\%$. The maximum error for the electronic systems is only
slightly larger than for the Minecraft systems (see histogram of the maximum
errors per system in Fig. 6).

Qualitatively, the predicted temperature distributions compare well with the
baseline from the FEM simulations (Fig. 6). In some systems the predicted
temperature at the surface is slightly too low. This is most probably caused
by setting the background to zero in the input channels. Additionally, larger
structures like the large capacitor show slight non-uniformities in the
predicted temperature distributions.

Figure 6: Histogram of the maximum $L_{1}$ error per system and selected
temperature distribution predicted by the NN (bottom) and baseline from the
FEM simulation (top).

###  3.2 Multi-step prediction

After training on single-steps the NN was applied to predict the whole time
series from a uniform temperature distribution until close to steady-state
after 15 time steps. For this, the output of the NN was repeatedly fed back
into the network. For the Minecraft test dataset the average $L_{1}$ error
accumulates to $0.47\text{\,}\%$ $\pm$ $0.3\text{\,}\%$ after 15 time steps.
The $L_{\mathrm{heq}}$ based regularization term remains constant over the
unrolling steps ($(0.018\pm 0.0008)\times 10^{-4}$). Visually, the temperature
distribution in the Minecraft systems after 15 unrolling steps look still
similar to the baseline FEM simulations.

For the electronic systems the accumulated error is still bounded, although,
much larger than for the Minecraft systems (Fig. 7). After one step the error
is mostly distributed on the surface of the components (Fig. 7, right) and
spreads from there with increasing number of steps. Arguably, for the
application in electronic design the temperature at the center of components
is of most interest since over-heating may occur here.

Figure 7: The accumulated error (average and variance region) is obtained by
starting from an initially homogeneous temperature state, and feeding the
predicted temperature distribution repeatedly back to the network during
evaluation (left). Training was performed on single time steps only. The
center and right show selected sections of a system after one, five and 15
time steps.

It can be seen that the error accumulated in the first couple of steps is
higher than for later steps. This could indicate a bias in the training
dataset. To construct the Minecraft dataset the FEM simulations were sampled
at 15 time steps. The dataset was then constructed by randomly sampling four
of these one-step predictions. Step 15 was chosen to be close to the steady-
state distribution. This might lead to an over-representation of small
temperature changes per time step. This result indicates that the training
could in future improve by non-uniform selection of training data from
different initial time steps.

###  3.3 Performance

The time for generating the training dataset was significantly reduced with
the random dataset compared to a training dataset of electronic systems. The
time for one of the transient FEM simulation was significantly reduced from on
average $5200\text{\,}\mathrm{s}$ for an electronic system to about
$800\text{\,}\mathrm{s}$ for a Minecraft type system (on a single thread of an
Intel Xeon W-2145 CPU). The training was completed after 400 epochs á
$620\text{\,}\mathrm{s}$ (on an NVIDIA Titan RTX GPU). During deployment the
trained network can predict the 3D temperate distribution for a batch of 40
electronic systems in $70\text{\,}\mathrm{m}\mathrm{s}$ including the time to
move the systems to the GPU memory.

Also, the smaller system sizes of the Minecraft dataset enable the training of
reasonable large NNs (such as the one presented in this work) on a standard
GPU. Using the electronic systems directly the same network would not be
trainable because of the limited memory available on the GPU.

##  4 DISCUSSION & CONCLUSIONS

The task that we pose to the NN is challenging: The training systems are only
a quarter of the size of the electronic systems. Thus, longer range
correlations than the size of the training systems are never seen by the
network. The spatial and temporal discretization is much coarser than would be
applicable for a standard numerical method. The material properties span
multiple orders of magnitude (compare Tab. 1). The electronic systems consist
of small details (like copper layers or legs which can be as small as one
voxel) and larger homogeneous blocks. Additionally, the network is employed
for multi-step prediction on the electronic systems although it was trained on
one-step predictions of systems with random geometry only. Nevertheless, the
network generalizes well to the electronic systems for the one-step
predictions and leads to a bounded error in the unrolling.

To summarize, there are obvious advantages in using a randomly generated
dataset for training. The smaller system sizes lead to faster simulation times
to obtain the “true” temperature distributions of the training systems (Sec.
3.3). In comparison with a previous work [9], we found that the higher
geometrical variability of the Minecraft systems makes the network less likely
to lock to pattern recognition instead of using the physical properties.
Additionally, it should better generalize to a wider range of different
geometries than when trained on the goal systems only, as was shown for the
electronic systems. Thus, no retraining should be necessary when new
components / geometrical variations are added later on.

One clear limitation of the type of training dataset presented here is that
generating a good random dataset representing all the desired features of the
physical equation and desired geometry is a challenge on its own. For
instance, systems that consist of randomly located voxels of random material
properties and with heat sources assigned to some random voxels can exhibit
unphysically hot locations. This can occur, for instance, if a heat source was
placed among a mostly isolating region. Also, in order that the NN can
generalize to the electronic systems the input and output values should be of
the same range. In the case presented here, this means that all material
properties, the heat source power per voxel as well as the overall temperature
range and the temperature increase per time step should be of the same order.

Representing the systems as images has clear advantages for the type of
application we have in mind: In a manual design process, a designer could move
components and the NN could return a near-instantaneous feedback on the
thermal distribution. This is possible because the components can be pre-
voxelized so that generating the input channels for a new design takes only
fractions of a second. For this type of application a temperature feedback at
coarse grid points would be sufficient. With other data representations, like
meshes, the geometry could be much closer represented but moving components
would require re-meshing.

In this work, we presented a random dataset for training a fully convolutional
network to predict the transient 3D temperature distribution in electronic
systems. The NN was trained on the randomly generated geometries only but
generalized well to electronic systems which were four times as large. The
network was tested with a set of electronic systems with differently shaped
components, different placement of those components, different material
properties and different initial temperatures. Two points proved decisive to
obtain good generalization to the electronic systems: First, two types of
randomized datasets were employed to cover most geometrical features found in
the electronic systems. Second, the network did not train well based only on
the $L_{1}$ error but training and generalization capabilities were
significantly improved when a physics-based regularization term was added to
the objective function.

## Acknowledgment

This work has been supported by Silicon Austria Labs GmbH (SAL), owned by the
Republic of Austria, the Styrian Business Promotion Agency (SFG), the federal
state of Carinthia, the Upper Austrian Research (UAR), and the Austrian
Association for the Electric and Electronics Industry (FEEI).

## References

  * [1] A. Sanchez-Gonzalez, J. Godwin, T. Pfaff, R. Ying, J. Leskovec, and P. W. Battaglia, “Learning to Simulate Complex Physics with Graph Networks,” ICML, pp. 8459–8468, 2020.
  * [2] T. Pfaff, M. Fortunato, A. Sanchez-Gonzalez, and P. W. Battaglia, “Learning Mesh-Based Simulation with Graph Networks,” NeurIPS, oct 2020.
  * [3] M. Cranmer, A. Sanchez-Gonzalez, P. Battaglia, R. Xu, K. Cranmer, D. Spergel, and S. Ho, “Discovering Symbolic Models from Deep Learning with Inductive Biases,” NeurIPS, jun 2020.
  * [4] G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, and L. Yang, “Physics-informed machine learning,” Nature Reviews Physics, vol. 3, pp. 422–440, jun 2021.
  * [5] J. Lim and D. Psaltis, “MaxwellNet: Physics-driven deep neural network training based on Maxwell’s equations,” APL Photonics, vol. 7, p. 011301, jan 2022.
  * [6] Y. Zhu, N. Zabaras, P.-S. Koutsourelakis, and P. Perdikaris, “Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data,” Journal of Computational Physics, vol. 394, pp. 56–81, oct 2019.
  * [7] A. Krishnapriyan, A. Gholami, S. Zhe, R. Kirby, and M. W. Mahoney, “Characterizing possible failure modes in physics-informed neural networks,” in Advances in Neural Information Processing Systems, 2021.
  * [8] S. Wang, X. Yu, and P. Perdikaris, “When and why PINNs fail to train: A neural tangent kernel perspective,” Journal of Computational Physics, vol. 449, p. 110768, jan 2022.
  * [9] M. Stipsitz and H. Sanchis-Alepuz, “Approximating the Steady-State Temperature of 3D Electronic Systems with Convolutional Neural Networks,” Mathematical and Computational Applications, vol. 27, p. 7, jan 2022.
  * [10] P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, C. Gulcehre, F. Song, A. Ballard, J. Gilmer, G. Dahl, A. Vaswani, K. Allen, C. Nash, V. Langston, C. Dyer, N. Heess, D. Wierstra, P. Kohli, M. Botvinick, O. Vinyals, Y. Li, and R. Pascanu, “Relational inductive biases, deep learning, and graph networks,” jun 2018.
  * [11] P. Ren, C. Rao, Y. Liu, J. Wang, and H. Sun, “PhyCRNet: Physics-informed convolutional-recurrent network for solving spatiotemporal PDEs,” Computer Methods in Applied Mechanics and Engineering, vol. 389, p. 114399, feb 2022.
  * [12] M. Chen, R. Lupoiu, C. Mao, D. Huang, J. Jiang, P. Lalanne, and J. A. Fan, “Wavey-net: Physics-augmented deep learning for high-speed electromagnetic simulation and optimization,” CoRR, vol. abs/2203.01248, 2022.