# Prediction of Frequency-Dependent Optical Spectrum for Solid Materials: A
Multi-Output & Multi-Fidelity Machine Learning Approach

Akram Ibrahim  Department of Physics, University of Maryland Baltimore County,
1000 Hilltop Circle, Baltimore, MD 21250, USA  Can Ataca  Department of
Physics, University of Maryland Baltimore County, 1000 Hilltop Circle,
Baltimore, MD 21250, USA<EMAIL_ADDRESS>
###### Abstract

The frequency-dependent optical spectrum is pivotal for a broad range of
applications, from material characterization to optoelectronics and energy
harvesting. Data-driven surrogate models, trained on density functional theory
(DFT) data, have effectively alleviated the scalability limitations of DFT
while preserving its chemical accuracy, expediting material discovery.
However, prevailing machine learning (ML) efforts often focus on scalar
properties such as the band gap, overlooking the complexities of optical
spectra. In this work, we employ deep graph neural networks (GNNs) to predict
the frequency-dependent complex-valued dielectric function across the
infrared, visible, and ultraviolet spectra directly from crystal structures.
We explore multiple architectures for multi-output spectral representation of
the dielectric function and utilize various multi-fidelity learning
strategies, such as transfer learning and fidelity embedding, to address the
challenges associated with the scarcity of high-fidelity DFT data.
Additionally, we model key solar cell absorption efficiency metrics,
demonstrating that learning these parameters is enhanced when integrated
through a learning bias within the learning of the frequency-dependent
absorption coefficient. This study demonstrates that leveraging multi-output
and multi-fidelity ML techniques enables accurate predictions of optical
spectra from crystal structures, providing a versatile tool for rapidly
screening materials for optoelectronics, optical sensing, and solar energy
applications across an extensive frequency spectrum.

###### keywords:

Graph neural networks, Transfer learning, Fidelity embedding, Dielectric
function, Absorption coefficient, Solar cells

##  1 Introduction

Frequency-dependent optoelectronic properties provide essential insights
critical for the design and optimization of a wide array of devices spanning
various applications including photovoltaic (PV) cells 1, light-emitting
diodes 2, transparent electronics 3, optical sensors 4, optical coatings 5,
chemical analysis 6, and astrochemistry 7. The capability to accurately and
efficiently predict optical properties across a spectrum of frequencies is
critical for integrating materials into cutting-edge optoelectronic devices.
Computational approaches, mainly using DFT, can provide optical spectra with
accuracy comparable to experiments more cost-effectively. Additionally, DFT
optical spectra, generated with consistent calculation settings, can serve as
benchmarks to identify influences beyond band-to-band transitions, such as
experimental setups or substrate effects. However, the vast array of candidate
materials poses formidable computational challenges for DFT, necessitating the
exploration of data-driven predictive models for preliminary screening.

Nevertheless, a gap persists in the literature concerning ML surrogate models
capable of accurately predicting the frequency-dependent optical properties of
solid materials. Previous studies have exclusively concentrated on predicting
individual scalar properties, such as the band gap 8, 9, 10 and the static
dielectric constant 11, 12, without accounting for the frequency dependence of
optical properties. While the prediction of spectral properties has only
recently emerged in materials science, multiple studies have explored multi-
output learning for predicting the electronic and phononic density of states
13, 14, 15. In the context of optical spectra, a hierarchical-correlation
model was utilized to predict the absorption coefficient at different
frequencies within the visible range, solely based on the chemical composition
within a collection of $69$ three-cation metal oxides 16. Another study
utilized a Gaussian process model to predict the dielectric constant of
polymers using a dataset of $1210$ experimentally-measured values at different
frequencies. 17, 17. These studies, however, were confined to specific
material chemical spaces, employing composition-based features while excluding
crystal structure, and constrained their predictions to particular discrete
frequencies. To our knowledge, no published work has yet utilized ML to
directly predict the continuous, frequency-dependent dielectric function or
absorption coefficient across a general chemical space of materials based on
the crystal structure.

Early endeavors in predicting material properties relied on manual feature
engineering from composition, crystal structure, and electronic band structure
using featurization algorithms 18. Currently, state-of-the-art predictive
modeling of materials utilizes GNNs 19, 20, 21, which adeptly generate latent
feature representations from composition and structure, enabling automatic
learning of features specific to the target property. In this work, we use
GNNs to predict the frequency-dependent, complex-valued dielectric function of
solid materials directly from crystal structure data. The dielectric function,
a fundamental spectral output from ab initio calculations, determines
material’s response to electromagnetic waves. It also enables the calculation
of crucial practical frequency-dependent optical properties, such as the
refractive index, electron energy loss spectra 22, quality factors for
localized surface plasmon resonances and surface plasmon polaritons 23, and
the quantum efficiency of optical sensors and PV cells 24.

For material spectral properties like phonon or electronic density of states,
the full-energy density of occupied states is characterized by a known
integral for each material, attributed to its atom or electron count. This
facilitates modeling the spectrum as a probability distribution, simplifying
learning by correlating increases in intensity in one range with decreases in
another. However, the optical spectrum lacks this property, and the optical
response can vary significantly in magnitude among materials. Yet, scaling the
optical spectrum can still offer a way to establish a correlation within the
predicted output. We find that proper spectrum scaling can lead to improved
organization within the latent feature space, subsequently enhancing model’s
performance.

Furthermore, while training ML models on high-fidelity data yields more
accurate results, securing a sufficient volume of such data for effective
training presents a notable challenge. A viable solution is adopting multi-
fidelity learning frameworks that integrate data from both low-fidelity and
high-fidelity sources. The larger datasets employing low-fidelity DFT
functionals can enhance GNN models’ ability to learn better encodings of
crystal structures, consequently boosting performance in learning high-
fidelity data. In this study, we investigate two multi-fidelity learning
approaches: ”transfer learning” and ”fidelity embedding” 25, 26, 27,
demonstrating that multi-fidelity learning effectively addresses the
bottleneck caused by the scarcity of high-fidelity optical spectra.

Finally, we assess the potential to enhance the prediction accuracy of
physical features of interest alongside the overall spectrum prediction by
incorporating a physical learning bias during training. Focusing on PV cell
absorption metrics—short-circuit current, reverse saturation current, and the
spectroscopic limit of maximum efficiency (SLME), which represents the
theoretical maximum photoconversion efficiency of a single p-n junction PV
cell—we demonstrate that learning these properties within the context of
frequency-dependent absorption coefficient learning through learning biases is
more effective than directly learning them as standalone target properties.

##  2 Methods

Data Set: The dataset was obtained from the JARVIS-DFT Vienna Ab initio
Simulation Package (VASP) raw files on Figshare as of January 2024,
encompassing all publicly available optical spectra data 28, 29, 30. It
comprised $34,327$ calculations employing the OptB88vdW (OPT) functional and
$14,560$ calculations utilizing the meta-GGA modified Becke-Johnson (MBJ)
potential 31, 32, 33, following data cleansing. The complex dielectric tensor,
$\varepsilon(E)=\varepsilon_{1}(E)+i\varepsilon_{2}(E)$, is provided for each
material in the dataset across a $5,000$-point energy grid that spans the
entire range between the minimum and maximum Kohn-Sham eigenvalues. To ensure
uniformity in the energy values at which $\varepsilon(E)$ is evaluated for all
materials within the dataset, we employ cubic interpolation and then uniformly
extract the function values within the $0$ eV to $12.0$ eV range with a
resolution of $0.04$ eV. This energy range encompasses the infrared (IR),
visible, and ultraviolet (UV) spectral regions. Notably, the employed model
frameworks can be readily adapted to cover broader or different spectral
ranges.

Figure 1: Summary of the frequency-dependent optical dataset: (a) Heatmap
illustrating the presence of the periodic table elements in the combined
crystal structures of the OPT and MBJ datasets. Elements not present in the
dataset are marked in gray. (b) Proportions of metals versus non-metals and
distribution of band gap values for non-metals in both datasets. (c-e)
Distributions of crystal systems, number of distinct species in the unit cell,
and atom count in the unit cell. (f) Histogram distribution of the logarithm
of the imaginary dielectric function peak value.

The $\varepsilon(E)$ output from VASP is represented by a $3\times 3$ tensor,
computed for the primitive unit cell of each material. To simplify the
problem, we diagonalize $\varepsilon(E)$, yielding three eigenvalues, which
correspond to eigenvectors oriented along the principal crystallographic axes.
Our GNN models are then trained using the mean of these eigenvalues. It is
important to note that predicting the optical spectra along a specific
symmetry axis follows the same formalism. For example, in the context of van
der Waals layered materials, the model can be trained using either the
eigenvalues of the in-plane or out-of-plane axis. To streamline learning, we
utilize the imaginary part of the dielectric function ($\varepsilon_{2}(E)$)
and a reduced form of the absorption coefficient, rather than the real part
($\varepsilon_{1}(E)$), to model the complex optical spectra. This choice is
made due to commonalities shared by the imaginary part and the absorption
coefficient, such as their non-negativity and zero values for energies within
the band gap. The reduced adsorption coefficient is defined as

$\alpha(E)=-\varepsilon_{1}(E)+\sqrt{\varepsilon_{1}^{2}(E)+\varepsilon_{2}^{2}(E)}$
(1)

The total absorption coefficient is given by
$2\sqrt{2}\pi(E/hc)\sqrt{\alpha(E)}$, where $h$ and $c$ represent Planck’s
constant and the speed of light in vacuum, respectively. Notably, the real
part can be readily derived from both the imaginary part and the absorption
coefficient. Figure 1 illustrates a statistical distribution of all materials
within the extracted OPT and MBJ datasets, categorized according to the
frequency of elements they contain, band gap values, crystal systems,
diversity of constituent species, number of atoms in the unit cell, and
$\varepsilon_{2}$ peak magnitudes. More details about the dataset preparation
are provided in the Supporting Information (SI).

Figure 2: Overview of the GNN architecture and multi-fidelity learning
frameworks: a) Schematic of the GNN architecture, illustrating input
structures with embedded atomic numbers, Gaussian-expanded bond distances, and
state features. The multi-output layer outputs the predicted discretized
optical spectrum over the considered frequency range. b) Diagram of the multi-
fidelity learning frameworks: Transfer learning sequentially trains on OPT
data across the entire GNN, then employs the learned LFV as input to the dense
layers for subsequent MBJ data learning, while freezing the graph
convolutional layers; fidelity-embedding jointly learns OPT and MBJ data
throughout the entire GNN, using a trainable embedding matrix as the input
state feature to encode the DFT functional fidelity level (MBJ or OPT) for
each crystal structure.

Graph Neural Network Formalism: Figure 2-a) illustrates a general
architectural overview of the employed GNN models. We utilize the MEGNet graph
convolutional layers proposed by C. Chen et al. 19. Initially, each crystal
structure is transformed into a graph characterized by node features, edge
features, and global features, corresponding to atoms, bonds, and the overall
state of the structure, respectively. For atom (node) features, we use only
the atomic numbers of the constituent elements, which are then mapped to an
embedding layer to learn elemental embeddings. Bond (edge) features are
expressed through expanding interatomic bond distances using a Gaussian basis
with 180 centers, each $0.5$ Å wide, uniformly distributed between bonded
atoms. Atoms are considered bonded if their interatomic distance is less than
a cutoff radius of $5.5$ Å, which encompasses not only the nearest neighbors
but also interactions with more distant atoms. For state (global) features in
fidelity-embedding GNN models, which are jointly trained on multi-fidelity
data, the fidelity level of the DFT functional (OPT or MBJ) associated with
each data point is represented by an integer (0 or 1). This is followed by a
trainable embedding layer utilized to learn encodings for the fidelity levels.
In contrast, for single-fidelity (SF) learning—where the model is trained on
data from a single DFT functional—or in transfer learning—where the model is
trained on multi-fidelity data sequentially—only two placeholder nodes,
without embedding, are employed to facilitate global information exchange.

The input features undergo a preprocessing step through dense layers before
being forwarded to the consecutive graph convolutional layers. These
convolutional layers execute a sequence of update operations through
convolution and pooling layers, transforming an input graph
$G=(\mathbf{e},\mathbf{v},\mathbf{u})$ into an output graph
$G^{\prime}=(\mathbf{e^{\prime}},\mathbf{v^{\prime}},\mathbf{u^{\prime}})$,
where $\mathbf{e}$, $\mathbf{v}$, and $\mathbf{u}$ represent the edge, vertex,
and global features, respectively. A stack, comprising three repetitions of
the preprocessing dense layers and graph convolutional layers as shown in
Figure 2-a), is utilized to enhance model flexibility and indirectly access
information beyond the $5.5$ Å cutoff radius, enabling the model to capture
more intricate long-range interactions 19. A dropout layer follows the final
graph convolution layer to mitigate overfitting. Rather than padding the
structure graphs to uniformize the sizes of their atomic features, they are
assembled into a single, large disjoint union graph for training. In the final
stages of the model, a readout operation is performed on both the atom and
bond feature vectors using an order-invariant set2set model 19, 34. The
set2set layer combines atom and bond feature vectors with vectors denoting the
indices of these atoms and bonds within the disjoint union graph. After the
readout process, the atom, bond, and state feature vectors are concatenated to
form the latent feature vector (LFV), which is subsequently processed through
a series of dense layers to generate the multi-output prediction representing
the discretized optical spectrum over the considered frequency range. Further
details concerning the GNN architecture, hyperparameters, and MEGNet graph
convolutional layers are available in the SI.

Model Training: The model construction and training was executed though
employing the Keras API with the TensorFlow backend 35, 36. Our dataset was
partitioned into three segments, with $80\%$ allocated for training, $5\%$ for
validation, and $15\%$ for testing. The selection of hyperparameters,
including the size of atom and bond features, dimensions of hidden layers,
batch size, and dropout rate was meticulously chosen through Bayesian
hyperparameter optimization facilitated by Optuna 37 (See the SI). The best-
performing models were selected based on their performance on the validation
set and subsequently evaluated using the test set.

Training on multi-fidelity data can be approached through various methods.
Figure 2-b) presents a schematic of the two multi-fidelity frameworks
considered in this study: (1) transfer learning (TL) and (2) fidelity
embedding (FE). In the TL framework, multi-fidelity learning progresses
sequentially: the GNN model is initially trained on all of the OPT data, and
then the LFV is obtained. Following this, the set of dense layers after the
LFV is further optimized to accommodate additional layers and neurons, and
then trained on the MBJ dataset using an $80/5/15$ train/validation/test
split. This means that the weights of the layers before the LFV remain frozen,
and only the later dense layers of the GNN are trained on the high-fidelity
MBJ data. In the FE framework, multi-fidelity learning is approached jointly.
The fidelity level (MBJ or OPT) for each data point is encoded as an integer
and inputted into the GNN model through a trainable fidelity-embedding matrix,
serving as the input state feature vector. Optimal validation results for
$\varepsilon_{2}$ and $\alpha$ were achieved with fidelity-embedding vector
lengths of $20$ and $16$, respectively. The $80/5/15$ train/validation/test
split applies exclusively to the MBJ dataset, while the entire OPT dataset is
used for training. Further details regarding the architecture of the TL and FE
GNNs are available in the SI.

##  3 RESULTS AND DISCUSSION

Spectrum Multi-Output Architecture: In this section, we evaluate the
performance of different multi-output GNN architectures for representing the
optical spectrum, which involve various combinations of data scaling schemes
and training loss functions. All GNN models discussed in this section are
solely trained and evaluated utilizing MBJ data. Since we have no ground truth
regarding a physical scaling feature, such as electron/atom number for the
electronic/phonon density of states, we explore both scaling to a maximum
value of $1$ (MaxNorm) and normalization to a cumulative sum of $1$ (AvgNorm),
alongside the unnormalized spectrum (UnNorm) 15, 14. For the MaxNorm and
AvgNorm models, the loss function is formulated as
$\mathcal{L}=\mathcal{L}_{N}+w\mathcal{L}_{S}$, where $\mathcal{L}_{N}$
pertains to the error in the norm, $\mathcal{L}_{S}$ to the error in the
normalized spectrum, and $w$ is a hyperparameter that denotes the relative
weight of the two loss components during training, and its optimal value is
determined through a grid search process on the validation set (See SI). The
mean absolute error (MAE) loss function is utilized for the UnNorm model, for
$\mathcal{L}_{N}$ in both MaxNorm and AvgNorm models, and for
$\mathcal{L}_{S}$ in the MaxNorm model. In the AvgNorm model, where the
normalized spectrum represents a probability density function (PDF), we
experiment with training $\mathcal{L}_{S}$ using two distinct loss functions:
MAE and the Kullback-Leibler (KL) divergence loss 38. The four models {UnNorm,
MaxNorm, AvgNorm (KL), AvgNorm (MAE)} share the same architecture, differing
only in the output layer and/or the loss function. In the UnNorm model, the
output layer comprises a dense layer of $300$ neurons (representing the
considered $12$ eV range at $0.04$ eV resolution) and features a Rectified
Linear Unit (ReLU) activation. For the remaining three models, the output
layer consists of $301$ neurons, including an additional neuron representing
the norm with ReLU activation. For MaxNorm, a sigmoid activation function is
employed for the normalized spectrum ($S$), specifically applied to
($S$-$S_{\text{max}}/2$). For the two AvgNorm models, the normalized spectrum
utilizes a softmax activation function applied to ($S$-$S_{\text{max}}$). The
radar plots in Figure 3-a) summarize the performance of the four models on the
$\varepsilon_{2}$ and $\alpha$ MBJ test sets. The benchmarking metrics cover
various indicators to evaluate performance across different facets of the
spectrum. These metrics include, in order: the overall MAE and Pearson
correlation for the unnormalized spectrum ($\varepsilon_{2}$ and $\alpha$),
MAE for the maximum and average values, along with MAE, KL divergence,
Wasserstein distance (WD) 39, and first derivative MAE for the PDF-normalized
spectrum ($\overline{\varepsilon}_{2}$ and $\overline{\alpha}$). Additionally,
we define another metric for aggregated statistical moments, for instance, for
$\overline{\varepsilon}_{2}$, as
$\sum_{n}(E^{n}\cdot\overline{\varepsilon}_{2})^{(1/n)}$, encompassing the
first four moments. Numeric values of the radar plots, including uncertainties
represented by the interquartile range (IQR), are detailed in Tables S1 and
S2.

Figure 3: Effect of the multi-output architecture on model performance: a)
Radar charts illustrating the median errors, normalized to a maximum of one,
for the four considered multi-output GNN architectures. These GNN models
differ in their spectrum scaling method and/or the loss function. The shown
performance evaluations are conducted using the MBJ data test, detailing
various metrics for both the imaginary part ($\varepsilon_{2}$) and the
reduced absorption coefficient ($\alpha$). Numeric values of the median error
and the IQR are outlined in Tables S1 and S2. b) The Pearson ($r$) and
Spearman ($\rho$) correlation coefficients between the normalized Euclidean
pairwise distances of $\varepsilon_{2}$ targets (which include average norms
($\varepsilon_{2}^{\text{avg}}$) and PDF-normalized spectra
($\overline{\varepsilon}_{2}$)) and those of latent feature vectors are
depicted for both metals and non-metals in the whole MBJ dataset.

Figure 3-a) shows that the AvgNorm (MAE) model consistently outperforms the
other three models across all metrics. This superiority can be attributed to
several factors. First, the model benefits from extracting the average norm
and the transformation of the optical spectrum into a PDF. Figure 1-f)
indicates that the peak values in $\varepsilon_{2}$ span approximately six
orders of magnitude in our data set. Thus, this scaling approach enables the
model to effectively capture trends in the normalized spectra, thereby
preventing materials with high average values from disproportionately
influencing the training process. Second, the utilization of the softmax
activation function boosts the model’s performance by facilitating improved
learning of the spectral distribution by constraining the sum of predicted
values of the normalized spectrum to unity. This constraint introduces a
correlation along the predicted spectrum, where a high probability in some
range of the spectrum automatically leads to a reduction in the probability in
other ranges. Furthermore, the exponential function in softmax activation
responds to lower stimulations (present in the lower regions of the normalized
spectrum) with a more uniform distribution, while exponentially amplifying
higher stimulations, such as peaks and near-peak regions of the normalized
spectrum. This mechanism ensures that large probabilities predominate, while
still incorporating information from lower probabilities within the spectrum.

Training the PDF using KL divergence loss is observed to yield similar
performance, albeit slightly lower than when employing MAE loss, which,
surprisingly, results in a lower KL error on the test set compared to training
directly with KL divergence loss. This can be attributed to KL divergence
quantifying discrepancies between PDFs logarithmically, helping to enhance
sensitivity to low-probability areas but simultaneously reducing it in high-
probability regions due to the logarithmic function’s dampening effect on
large values. Thus, KL divergence may overlook finer details around peak areas
within the optical PDF.

We also experimented with the WD loss for the PDF-normalized spectrum within
the AvgNorm architecture but observed that the predicted spectra became
impractically noisy. This likely stems from WD penalizing errors in the
cumulative distribution function (CDF) rather than the PDF 40, leading to
precise resonant peak positioning but significant noise in regions such as the
baseline (ranges of no interaction with electromagnetic fields) or peak tails.
This occurs because the CDF loss permits fluctuations between underestimation
and overestimation at successive spectral points, resulting in a lower CDF
error comparable to a prediction that consistently under- or overestimates.
Given the necessity for accurate, noise-free spectral predictions for
analyzing the dielectric function, WD was deemed unsuitable as a standalone
loss function for the normalized spectrum and was therefore excluded from our
analysis.

On the other hand, extracting the maximum value and normalizing the spectrum
to a maximum of one results in inferior performance compared to the
unnormalized case. This decline can partly be attributed to the use of the
sigmoid function with the MaxNorm model. Although the sigmoid function offers
a smooth exponential form that confines the normalized spectrum within the
range of $0$ to $1$, unlike softmax, it does not impose constraints that can
correlate the spectrum points. The sigmoid activation for
$(S-S_{\text{max}}/2)$ treats values around the half-maximum almost linearly
but rapidly saturates values deviating upward from the half-maximum to $1$ and
values deviating downward from the half-maximum to zero. This saturation
effect reduces the resolution between points of higher values in the spectrum,
thereby contributing to the observed decrease in performance. We also
conducted experiments where the application of the sigmoid function in the
MaxNorm architecture was omitted, opting instead for a linear activation
function with clamping between $0$ and $1$. However, the resulting output did
not yield spectra that were deemed to be plausibly smooth. Further discussion
providing more insights into the reasons behind the observed effects of
spectrum scaling on GNN performance is discussed in the ”Latent Space
Visualization” section.

The efficacy of the AvgNorm model architecture implies that representing the
optical spectrum of arbitrary crystal structures through an average norm,
alongside a PDF, learned via uniform error penalization without weighting any
part of the distribution more heavily than another while using softmax
activation, enhances the model’s capacity to discern fundamental patterns in
optical spectra across diverse materials. Consequently, the AvgNorm (MAE)
architecture is utilized in subsequent analyses. Notably, the ramifications of
this scaling extend beyond mere post-processing of the output, profoundly
influencing the arrangement of materials in the latent feature space. This
phenomenon is illustrated in Figure 3-b), where we detail the distribution of
normalized Euclidean pairwise distances for $\varepsilon_{2}$ targets,
including average norms ($\varepsilon_{2}^{\text{avg}}$) and PDF-normalized
spectra ($\overline{\varepsilon}_{2}$), versus those of the LFVs across all
materials in the MBJ dataset, correlating them using both Pearson ($r$) and
Spearman ($\rho$) correlation coefficients. In the case of metals, the AvgNorm
model is observed to increase $r$ between features and both
$\varepsilon_{2}^{\text{avg}}$ and $\overline{\varepsilon}_{2}$ by about
$16\;\%$. Conversely, for non-metals, while the AvgNorm model increases $r$
between features and $\overline{\varepsilon}_{2}$ by around $7\;\%$, it
concurrently significantly reduces the correlation with
$\varepsilon_{2}^{\text{avg}}$ by about $56\;\%$. These alterations in the
correlation of features with targets underscore the profound influence of
optical spectrum scaling on the structural organization of the latent feature
space. In essence, the AvgNorm model orchestrates a rearrangement of the
materials within the latent space, fostering proximity among materials
exhibiting similar PDF-normalized spectra, and simultaneously
boosting/attenuating the arrangement based on the average norms for
metals/non-metals. Further discussion on the organization of materials in the
latent space is presented in the ”Latent Space Visualization” section.

Learning from Multi-Fidelity Data: Ideally, ML model training should rely on
experimental or high-fidelity computational data. However, given the inherent
scarcity of such high-fidelity data, practical limitations necessitate
leveraging the ample low-fidelity data to enhance predictive power through
improved feature learning.

Figure 4: Multi-fidelity learning: Radar charts depicting the median errors,
normalized to a maximum of one, for three GNNs: the single-fidelity (SF) model
(trained exclusively on MBJ data), the transfer learning (TL) model (initially
trained on all OPT data, followed by further training of the dense layers
post-LFV on $80\%$ of MBJ data), and the fidelity-embedding (FE) model
(trained jointly on all OPT data and $80\%$ of MBJ data). The shown
performance evaluations are conducted using the higher-fidelity MBJ data test
set, detailing various metrics for both the imaginary part ($\varepsilon_{2}$)
and the reduced absorption coefficient ($\alpha$). Numeric values of the
median error and the IQR are outlined in Tables S3 and S4.

The radar plots in Figure 4 illustrate the performance metrics for the SF GNN,
which is trained solely on MBJ data, as well as for the TL and FE GNNs for
both $\varepsilon_{2}$ and $\alpha$ (numeric values of median error and IQR
are detailed in Tables S3 and S4). All three models employ the AvgNorm (MAE)
architecture discussed in the previous section. The results indicate a
noticeable decrease across all error metrics and an increase in correlation
with the DFT (MBJ) spectra upon the incorporation of the lower-fidelity OPT
data. Notably, the FE model demonstrates a higher improvement, with the median
MAE of the unnormalized spectrum decreasing by $24.6\;\%$ and $25.0\;\%$ for
$\varepsilon_{2}$ and $\alpha$, respectively, compared to reductions of
$22.5\;\%$ and $20.0\;\%$ for TL. Both FE and TL exhibit comparable
improvements in the Pearson correlation, with $\varepsilon_{2}$ increases of
$2.5\;\%$ for FE versus $2.3\;\%$ for TL, and $\alpha$ increases of $3.9\;\%$
for FE versus $3.7\;\%$ for TL. The superior performance of the FE framework
can be attributed to its joint learning strategy, which simultaneously
integrates low- and high-fidelity data during training, in contrast to the
sequential learning approach utilized in TL. In TL, only the dense layers
post-LFV can detect the nuanced differences between OPT and MBJ data.
Conversely, FE empowers the entire GNN to fine-tune its weights for fitting
both OPT and MBJ data, thus achieving a broader optimization scope. Moreover,
while the larger size of the OPT dataset enables the TL model to glean a more
robust LFV compared to SF models trained solely on the smaller-sized MBJ data,
this approach inherently restricts the LFV learning to patterns of the lower-
fidelity dataset, potentially overlooking insights that could be gained from
holistic learning involving both data fidelities simultaneously. Therefore,
the joint learning strategy of the FE framework enables the GNN to exploit the
most extensive dataset resulting from the amalgamation of OPT and MBJ
datasets, thereby accessing a broader range of information and achieving a
more refined LFV compared to TL.

Latent Space Visualization: In essence, the enhanced performance of a GNN
model in predicting material properties implies an improved ordering of
materials within the latent feature space. While this concept is relatively
straightforward in single-target prediction scenarios, where an improved
latent space representation should manifest a more correlated ordering of
latent feature vectors with the target scalar, complexities emerge in spectral
multi-output prediction problems. In such cases, numerous scalar and vectorial
features derived from the spectrum can potentially organize the latent space.
The pertinent question then becomes: which feature holds greater significance
in organizing the latent space?

To gain deeper insights into the latent feature space, Figure 5-a) illustrates
a two-dimensional projection via t-distributed stochastic neighbor embedding
(t-SNE) 41 of the latent features of all structures within the MBJ dataset,
accompanied by heatmaps for certain physical scalar properties derived from
the optical spectrum. The latent feature vectors are derived from the best-
performing AvgNorm FE GNN model optimized for predicting $\varepsilon_{2}$.
Further quantitative insights into the latent feature space are obtained by
calculating the Spearman rank-order correlation between the normalized
Euclidean pairwise distances of latent features and the corresponding
normalized Euclidean pairwise distances of various target properties extracted
from the optical spectrum (0-12 eV range), as illustrated in Figure 5-b).

Figure 5: Significance of optical spectrum properties in organizing the latent
feature space of crystal structures: a) Two-dimensional t-SNE projection of
the latent features of all the structures in the MBJ dataset (perplexity =
150). The color maps represent DFT (MBJ) values for various scalar properties
of the multi-output optical spectrum. The latent feature vectors are derived
from the AvgNorm FE model optimized for predicting $\varepsilon_{2}$. b)
Spearman correlation coefficients between the normalized Euclidean pairwise
distances of latent feature vectors and those for different properties
extracted from the optical spectrum for both metals and non-metals.

The band gap (E${}_{\text{gap}}$) heatmap t-SNE plot in Figure 5-a) reveals
that the GNN model proficiently segregates metals from non-metals into two
distinctly discernible clusters within the latent space, thereby showcasing
the model’s adept comprehension of the distinctive optical characteristics of
these two material categories. Moreover, a prominent gradient in
E${}_{\text{gap}}$ is evident within the non-metal cluster, capturing the
diversity in band gaps among semiconductors and insulators. This is
demonstrated by a Spearman correlation of $\rho=0.48$ between
E${}_{\text{gap}}$ and the latent features, as illustrated in Figure 5-b). In
a similar vein, the model exhibits structured organization within the latent
space regarding additional dielectric properties, such as the logarithm of the
peak value of the imaginary dielectric function, denoted by
ln($\varepsilon_{2}^{\text{max}}$), and the corresponding energy at this peak,
denoted as E($\varepsilon_{2}^{\text{max}}$). A pronounced gradient in
ln($\varepsilon_{2}^{\text{max}}$) is evident within the metal cluster,
characterized by markedly elevated peaks at its boundary, distant from the
interface between metal and non-metal clusters. This pattern reveals that
metals distant from the metal/non-metal cluster interface display elevated
optical conductivity peaks, which diminish progressively toward the interface,
running counter to the band gap gradient direction within the non-metal
cluster; this is consistent with physical expectations, as one would expect
optical conductivity to exhibit patterns that are opposite to those of the
band gap. Conversely, within the non-metal cluster, a less pronounced gradient
is observed for ln($\varepsilon_{2}^{\text{max}}$). This can be attributed to
the fact that the imaginary dielectric function of semiconductors and
insulators exhibits a more complex dependence on band structure, reflecting
the intricate probabilities of valence to conduction interband transitions. In
contrast, the simpler intraband transitions of free electrons in metals,
particularly at lower energies, can be effectively characterized using bulk
properties such as electron density and scattering rate, as described by the
Drude model 42. Therefore, the optical spectra of non-metals are more
challenging to represent with simple dielectric magnitudes. This is
highlighted by the lower correlation coefficients of both maximum and average
values ($\varepsilon_{2}^{\text{max}}$ and $\varepsilon_{2}^{\text{avg}}$) for
non-metals compared to metals, as demonstrated in Figure 5-b), where non-
metals exhibit correlations of $0.17$ and $0.20$, compared to $0.26$ and
$0.50$ for metals.

Furthermore, a gradient is evident in E($\varepsilon_{2}^{\text{max}}$), with
metals generally manifesting lower values as indicated in Figure 5-a),
indicative of their lower natural frequencies due to the presence of
delocalized electrons. Conversely, semiconductors and insulators tend to
exhibit notably higher E($\varepsilon_{2}^{\text{max}}$) values owing to the
increased energies required to excite their tightly-bound valence electrons to
the conduction band, necessitating the overcoming of the band gap before
observing the peak. Interestingly, within the metal cluster, the latent space
of the imaginary part displays a noticeable gradient in the logarithm of the
static dielectric constant, ln($\varepsilon_{1}(0)$), which is related to the
real part. This suggests that the GNN effectively captures the physical
linkage between the imaginary and real parts of the dielectric function via
the Kramers-Kronig relation 43. In contrast, a less pronounced gradient is
observed among non-metals, again highlighting their more intricate dependence
on band structures rather than simple dielectric magnitudes. This is reflected
by a lower $\varepsilon_{1}(0)$ correlation value of $0.12$ for non-metals
compared to $0.27$ for metals, as shown in Figure 5-b).

To address our question regarding which feature from the optical spectrum is
most influential in structuring the latent space, an examination of Figure
5-b) indicates that for non-metals, the average-normalized spectrum
($\overline{\varepsilon}_{2}^{\text{avg}}$), interpreted as a PDF, emerges as
the property most strongly correlated with the latent features, exhibiting a
high $\rho$ of $0.76$, compared to the other evaluated properties, none of
which surpass $0.48$. Notably, the pronounced correlation of
$\overline{\varepsilon}_{2}^{\text{avg}}$ is not a consequence of training the
FE GNN model with the AvgNorm architecture. Evidence supporting this
observation is illustrated in Figure S6, which shows
$\overline{\varepsilon}_{2}^{\text{avg}}$ maintaining its status as the most
correlated property for non-metals across all SF models, regardless of the
employed output scaling architecture. Thus, the PDF-normalized
$\overline{\varepsilon}_{2}^{\text{avg}}$ can be deemed as a fundamental
descriptor for learning the optical spectra of semiconductors and insulators,
reinforcing our earlier observations regarding the superior performance of the
AvgNorm model compared to the UnNorm model. Moreover, the lower correlation of
features with the max-normalized spectrum
($\overline{\varepsilon}_{2}^{\text{max}}$), with $\rho=0.42$ for non-metals,
bolsters the previously observed superior performance of the AvgNorm
architecture over the MaxNorm. E${}_{\text{gap}}$ and statistical scalar
attributes, including E${}_{\text{mean}}$, E${}_{\text{variance}}$,
E${}_{\text{skewness}}$, E${}_{\text{kurtosis}}$, and
E$(\varepsilon_{2}^{\text{max}})$ (indicating the mode), rank as secondary in
significance for structuring non-metals’ latent space, exhibiting $\rho$
values between approximately $0.37$ and $0.48$. Nonetheless, these attributes
are comprehensively integrated within the normalized spectrum,
$\overline{\varepsilon}_{2}^{\text{avg}}$. Additional scalar properties,
including dielectric magnitudes such as the static dielectric constant
($\varepsilon_{1}(0)$), the average ($\varepsilon_{2}^{\text{avg}}$) and
maximum ($\varepsilon_{2}^{\text{max}}$) imaginary dielectric values, and
other generic features like the number of spectral peaks (defined as the
largest local maxima exceeding $0.25$ of the highest peak and separated by at
least $2$ eV), exhibit the lowest correlation scores with the latent features,
all with $\rho\leq 0.2$. This underscores their relatively minor role in
shaping the latent space of non-metals.

The correlation landscape for the latent space of metals exhibits a distinct
profile, with both $\overline{\varepsilon}_{2}^{\text{avg}}$ and
$\varepsilon_{2}^{\text{avg}}$ playing equally significant roles in
structuring the latent space, each exhibiting a correlation value of
$\rho=0.50$. As discussed, the spectral dielectric response of metals,
characterized by predominant intraband transitions at low energies within the
IR range, allows a simple dielectric magnitude, specifically
$\varepsilon_{2}^{\text{avg}}$, to serve as a key descriptor for metals,
emphasizing their optical conductivity. However, similar to non-metals,
$\overline{\varepsilon}_{2}^{\text{avg}}$ remains crucial for defining the
latent space and accurately representing the spectral energy distribution.
This consideration becomes particularly relevant given that metals may exhibit
additional peaks at higher frequencies due to potential interband transitions
in the late-visible and UV ranges. Consequently,
$\varepsilon_{2}^{\text{avg}}$ and $\overline{\varepsilon}_{2}^{\text{avg}}$
are identified as two pivotal descriptors for effectively learning metals’
optical spectra.

Figure 6: Graph neural network predictions of optical spectra: The
interpolated predictions for the dielectric function’s imaginary part
($\varepsilon_{2}$) and the reduced absorption coefficient ($\alpha$) are
shown against the DFT interpolations for samples of materials obtained from
the MBJ test set within $\pm 5\;\%$ around median MAE in the unnormalized
spectrum. The predictions are generated using the optimized AvgNorm (MAE) FE
models trained on all OPT data and $80\;\%$ of the MBJ data.

Prediction of Frequency-Dependent Optical Spectra & Solar Absorption
Efficiency: Figure 6 depicts predictions for both $\varepsilon_{2}$ and
$\alpha$ for materials selected from the MBJ test set, chosen within an error
margin of $\pm 5\;\%$ around the median MAE error in the unnormalized
spectrum. These predictions are generated using the optimized AvgNorm (MAE) FE
GNN models. The shown median performance highlights the efficacy of the GNN
models, enhanced by optimized spectrum multi-output scaling and multi-fidelity
learning, in accurately capturing the nuances in DFT optical spectra at the
meta-GGA MBJ level. Notably, the models exhibit proficiency in precisely
identifying peak values and their respective positions across the entire
spectrum, including the IR, visible, and UV regions, spanning metals,
semiconductors, and insulators. From $\varepsilon_{2}$ and $\alpha$,
$\varepsilon_{1}$ can be derived, yielding the complex dielectric function,
from which various significant frequency-dependent optical properties can be
calculated.

Having presented GNN models capable of generalizing across a broad range of
frequencies and diverse materials, it is noteworthy that specific practical
applications often necessitate focusing on particular material groups within
narrower spectral regions. For example, in solar cell applications, the
emphasis is on semiconductors with absorbance profiles efficiently capturing
incident solar irradiation. The SLME represents the theoretical maximum
photoconversion efficiency of a single p-n junction solar cell. While SLME is
a scalar property directly learnable through a single-output ML model, gaining
insight into the spectral absorbance characteristics contributing to this SLME
value is valuable. This deeper understanding can clarify why a material
exhibits lower or higher SLME, suggesting potential modifications or
exploration paths. Therefore, predicting SLME by forecasting the absorption
coefficient proves advantageous.

We demonstrate that a multi-output GNN model, trained to predict the
absorption coefficient, can accurately forecast SLME at levels comparable to,
or even surpassing, those of a single-output GNN model specifically designed
for SLME learning, while maintaining precise spectrum prediction capabilities.
By restricting our OPT and MBJ data to materials with band gaps in the range
of $0.1$ to $4.5$ eV (solar irradiation range), we retrain the AvgNorm FE GNN
model for predicting $\alpha$ as before, but now with two additional neurons
in the output layer for predicting the short-circuit current
(J${}_{\text{sc}}$) and the logarithm of the reverse saturation current
(log(J0)). The loss function is further regularized with two additional loss
terms for both J${}_{\text{sc}}$ and log(J0). By training the model using all
OPT and $80\;\%$ of the MBJ data, we evaluate its performance on the remaining
MBJ data, as shown in Figure 7. Utilizing the values of J${}_{\text{sc}}$ and
log(J0), we calculate SLME following the procedure proposed by K. Choudhary et
al. to maximize the power density output from a solar cell 44

$\text{SLME}=\frac{\max\left\\{\left(J_{\text{sc
}}-J_{0}\left(\mathrm{e}^{eV/kT}-1\right)\right)V\right\\}_{V}}{\int_{0}^{\infty}EI_{\text{sun
}}(E)\mathrm{d}E}$ (2)

where $J_{\text{sc}}=e\int_{0}^{\infty}a(E)I_{\text{sun}}(E)\,\mathrm{d}E$
with $I_{\text{sun}}$ representing the AM1.5G solar spectrum 44, 45. The
absorbance, $a(E)$, is determined from the absorption coefficient and the film
thickness ($L$) as
$a(E)=1-\mathrm{exp}({-2(2\sqrt{2}\pi(E/hc)\sqrt{\alpha(E)})L})$.
$J_{0}=e\pi\int_{0}^{\infty}a(E)I_{\text{bb}}(E,T)\,\mathrm{d}E$ accounts for
the radiative component of the electron-hole recombination current at
equilibrium in darkness, with $I_{\text{bb}}$ signifying the blackbody
irradiation. We assume thin films with a thickness of $50$ nm operating at
$300$ K for all materials.

Figure 7: Graph neural network prediction of solar energy absorption efficiency: Predictions of the short-circuit current (J${}_{\text{sc}}$), natural logarithm of the reverse saturation current (log(J0)), and the spectroscopic limit of maximum efficiency (SLME) are validated for materials with band gaps ranging from $0.1$ to $4.5$ eV against DFT (MBJ) values. J${}_{\text{sc}}$ and J0 are expressed in amperes (A). The employed GNN predicts the frequency-dependent absorption coefficient with extra learning bias to emphasize learning J${}_{\text{sc}}$ and log(J0). Table 1: Comparison of various GNN and non-graph-based ML models for predicting solar absorption efficiency parameters: Short-circuit current (J${}_{\text{sc}}$), natural logarithm of the reverse saturation current (log(J0)), and the spectroscopic limit of maximum efficiency (SLME)—against test set DFT (MBJ) values. The GNN models differ in their learning approach: they are either estimating the solar parameters from a learned frequency-dependent absorption coefficient, learning the solar parameters directly, or applying a learning bias to these parameters while concurrently learning the absorption coefficient. J${}_{\text{sc}}$ and J0 are measured in amperes. Model (Learnables) |  J${}_{\text{sc}}$ (R2/MAE) |  log(J0) (R2/MAE) |  SLME (R2/MAE)
---|---|---|---
GNN ($\alpha(E)$) |  0.87/33.00 |  -1.86/60.26 |  -0.15/5.04
GNN ($\\{\text{J}_{\text{sc}},\log(\text{J}_{0})\\}$) |  0.87/25.87 |  0.81/11.63 |  0.70/2.18
GNN ($\alpha(E)$ \+ $\\{\text{J}_{\text{sc}},\log(\text{J}_{0})\\}$ bias) |  0.90/22.42 |  0.87/10.05 |  0.76/2.04
Random Forest ($\\{\text{J}_{\text{sc}},\log(\text{J}_{0})\\}$) |  0.71/50.59 |  0.69/17.60 |  0.60/3.10
XGBoost ($\\{\text{J}_{\text{sc}},\log(\text{J}_{0})\\}$) |  0.71/46.95 |  0.71/15.96 |  0.59/2.93
LightGBM ($\\{\text{J}_{\text{sc}},\log(\text{J}_{0})\\}$) |  0.75/44.18 |  0.74/15.18 |  0.63/2.84

The efficacy of the GNN model in forecasting solar efficiency parameters is
demonstrated in Figure 7, where it achieves a MAE of $2.04$ and a coefficient
of determination (R2) exceeding $0.75$ for SLME prediction on unseen
materials. This performance demonstrates superiority over traditional non-
graph-based ML models. To benchmark this, we trained several non-graph-based
ML models, including random forests from scikit-Learn 46 and gradient boosting
decision trees from XGBoost and LightGBM packages 47, 48, on the MBJ dataset
using features derived from chemical composition and crystal structure
provided by the automatminer package 49. The LightGBM model achieves the best
performance on the test set among non-graph-based ML models, as outlined in
Table 1. Compared to the performance metrics of the solar-biased GNN model
presented in Figure 7 and denoted as GNN ($\alpha(E)$ \+
$\\{\text{J}_{\text{sc}},\log(\text{J}_{0})\\}$ bias) in Table 1, the GNN
model demonstrates superior efficacy. This is manifested by the MAE for the
best non-graph-based LightGBM model being approximately $97\%$, $51\%$, and
$39\%$ higher for J${}_{\text{sc}}$, log(J0), and SLME, respectively, compared
to the solar-biased GNN model. Further details about the GNNs, as well as the
features and hyperparameters used for the non-graph-based ML models, are
provided in the SI.

Furthermore, a GNN model with the same architecture, yet with an output layer
of only two neurons and a loss function tailored exclusively for predicting
J${}_{\text{sc}}$ and log(J0), denoted by GNN
($\\{\text{J}_{\text{sc}},\log(\text{J}_{0})\\}$) in Table 1, yields inferior
performance compared to the solar-biased GNN, with the MAE higher by
approximately $15\%$, $16\%$, and $7\%$ for J${}_{\text{sc}}$, log(J0), and
SLME, respectively. Thus, we can notice that leveraging a multi-output GNN
initially designed for learning the absorption spectrum, when further refined
with a learning bias to emphasize solar parameters’ learning, can yield
improved predictive performance in forecasting solar parameters compared to
directly learning them as singular targets. This improvement can be attributed
to the fact that these solar parameters result from convolution integrals with
the frequency-dependent absorption coefficient. By synergizing the learning of
the absorption coefficient with the learning of these solar parameters, the
latent feature space of the GNN becomes enriched with information on the
absorption spectrum, thus enhancing the model’s predictive accuracy regarding
solar parameters. On the other hand, a solar-unbiased GNN model with the same
architecture trained solely for predicting the absorption coefficient, denoted
by GNN ($\alpha(E)$) in Table 1, achieves $\text{MAE}=0.384$ and $r=0.948$ for
the unnormalized spectrum of $\alpha$. In comparison, the solar-biased GNN
model demonstrates an almost identical performance, with $\text{MAE}=0.386$
and $r=0.949$. However, the solar-unbiased model yields suboptimal results for
the solar parameters, as outlined in Table 1. The MAE increases by $47\%$,
$50\%$, and $147\%$ for J${}_{\text{sc}}$, log(J0), and SLME, respectively,
compared to the solar-biased model. This suggests that incorporating a
learning bias via simple regularization terms in the loss function, aimed at
emphasizing specific practical physical properties, can aid in distributing
the error of the multi-output prediction in a way that substantially enhances
the learning of these physical properties without compromising the overall
accuracy of predicting the absorption spectrum.

##  4 CONCLUSION

We have developed multi-output GNN models capable of predicting the frequency-
dependent imaginary dielectric function and absorption coefficient with
accuracy on par with meta-GGA DFT. This enables the derivation of the complete
dielectric response of any arbitrary material using only its atomic structure
as input. We considered a spectrum spanning a $12$ eV range—from IR to UV—yet
the employed GNN formalism offers easily adaptable spectrum ranges. We
investigated the effect of spectrum scaling on the formation of the latent
feature space and the GNNs’ predictive capacity by comparing various scaling
schemes, including UnNorm, MaxNorm, and AvgNorm models. Our findings highlight
that the AvgNorm GNN model, treating the optical spectrum of any material as
an average norm and a probability distribution function, along with a softmax
activation and an evenly-weighted-distribution loss function, exhibits
superior performance. Furthermore, our GNN models integrate multi-fidelity
learning schemes, such as transfer learning and fidelity embedding, and
utilize the whole low-fidelity OPT and high-fidelity MBJ optical spectral data
available in the JARVIS DFT database. We observe a notable decrease across all
error metrics and an increase in correlation with the DFT (MBJ) spectra upon
the incorporation of the lower-fidelity OPT data, with the fidelity embedding
approach achieving moderately superior accuracy to transfer learning
attributed to its broader multi-fidelity optimization scope.

We also demonstrated that the prediction of the frequency-dependent absorption
coefficient by GNNs can be fine-tuned to emphasize specific scalar
optoelectronic properties without compromising the overall spectrum multi-
output prediction, through simple learning biases applicable to any property
of interest. As a proof of concept, we optimized the learning of solar cell
performance parameters—short-circuit current, reverse saturation current, and
the corresponding solar cell efficiency SLME—showing that integrating these
properties into the multi-output learning of the absorption coefficient leads
to enhanced prediction of solar properties compared to learning them
separately. This synergy of multi-output and multi-fidelity learning, along
with the flexibility to apply target-specific learning biases, presents a
versatile tool for the rapid screening of solid materials for a wide range of
frequency-dependent optical device applications involving metals,
semiconductors, or insulators.

##  5 Supporting Information

To enhance reproducibility, the datasets and codes for model design and result
analysis are available on GitHub: https://github.com/UMBC-STEAM-LAB/GNN-
frequency-dependent-optical-properties.

##  6 Acknowledgements

The authors acknowledge the fund from the National Science Foundation (NSF)
under grant number NSF DMR$-2213398$ and the Department of Energy (DOE) under
grant number DE-SC$0024236$.

##  7 Notes

The authors declare no competing interests.

##  8 REFERENCES

## References

  * Day et al. 2019 Day, J.; Senthilarasu, S.; Mallick, T. K. Improving spectral modification for applications in solar cells: A review. _Renewable Energy_ 2019, _132_ , 186–205
  * Ren et al. 2021 Ren, A.; Wang, H.; Zhang, W.; Wu, J.; Wang, Z.; Penty, R. V.; White, I. H. Emerging light-emitting diodes for next-generation data communications. _Nature Electronics_ 2021, _4_ , 559–572
  * Won et al. 2023 Won, D.; Bang, J.; Choi, S. H.; Pyun, K. R.; Jeong, S.; Lee, Y.; Ko, S. H. Transparent electronics for wearable electronics application. _Chemical Reviews_ 2023, _123_ , 9982–10078
  * Li et al. 2015 Li, M.; Cushing, S. K.; Wu, N. Plasmon-enhanced optical sensors: a review. _Analyst_ 2015, _140_ , 386–406
  * Hu et al. 2019 Hu, C.; Guo, K.; Li, Y.; Gu, Z.; Quan, J.; Zhang, S.; Zheng, W. Optical coatings of durability based on transition metal nitrides. _Thin Solid Films_ 2019, _688_ , 137339
  * Babbe and Sutter-Fella 2020 Babbe, F.; Sutter-Fella, C. M. Optical absorption-based in situ characterization of halide perovskites. _Advanced Energy Materials_ 2020, _10_ , 1903587
  * Materese et al. 2020 Materese, C. K.; Gerakines, P. A.; Hudson, R. L. Laboratory Studies of Astronomical Ices: Reaction Chemistry and Spectroscopy. _Accounts of Chemical Research_ 2020, _54_ , 280–290
  * Wang et al. 2022 Wang, T.; Zhang, K.; Thé, J.; Yu, H. Accurate prediction of band gap of materials using stacking machine learning model. _Computational Materials Science_ 2022, _201_ , 110899
  * Rajan et al. 2018 Rajan, A. C.; Mishra, A.; Satsangi, S.; Vaish, R.; Mizuseki, H.; Lee, K.-R.; Singh, A. K. Machine-learning-assisted accurate band gap predictions of functionalized MXene. _Chemistry of Materials_ 2018, _30_ , 4031–4038
  * Zhuo et al. 2018 Zhuo, Y.; Mansouri Tehrani, A.; Brgoch, J. Predicting the band gaps of inorganic solids by machine learning. _The journal of physical chemistry letters_ 2018, _9_ , 1668–1673
  * Shimano et al. 2023 Shimano, Y.; Kutana, A.; Asahi, R. Machine learning and atomistic origin of high dielectric permittivity in oxides. _Scientific Reports_ 2023, _13_ , 22236
  * Takahashi et al. 2020 Takahashi, A.; Kumagai, Y.; Miyamoto, J.; Mochizuki, Y.; Oba, F. Machine learning models for predicting the dielectric constants of oxides based on high-throughput first-principles calculations. _Physical Review Materials_ 2020, _4_ , 103801
  * Mahmoud et al. 2020 Mahmoud, C. B.; Anelli, A.; Csányi, G.; Ceriotti, M. Learning the electronic density of states in condensed matter. _Physical Review B_ 2020, _102_ , 235130
  * Fung et al. 2022 Fung, V.; Ganesh, P.; Sumpter, B. G. Physically informed machine learning prediction of electronic density of states. _Chemistry of Materials_ 2022, _34_ , 4848–4855
  * Kong et al. 2022 Kong, S.; Ricci, F.; Guevarra, D.; Neaton, J. B.; Gomes, C. P.; Gregoire, J. M. Density of states prediction for materials discovery via contrastive learning from probabilistic embeddings. _Nature communications_ 2022, _13_ , 949
  * Kong et al. 2021 Kong, S.; Guevarra, D.; Gomes, C. P.; Gregoire, J. M. Materials representation and transfer learning for multi-property prediction. _Applied Physics Reviews_ 2021, _8_ , 021409
  * Chen et al. 2020 Chen, L.; Kim, C.; Batra, R.; Lightstone, J. P.; Wu, C.; Li, Z.; Deshmukh, A. A.; Wang, Y.; Tran, H. D.; Vashishta, P.; others Frequency-dependent dielectric constant prediction of polymers using machine learning. _npj Computational Materials_ 2020, _6_ , 61
  * Ward et al. 2018 Ward, L.; Dunn, A.; Faghaninia, A.; Zimmermann, N. E.; Bajaj, S.; Wang, Q.; Montoya, J.; Chen, J.; Bystrom, K.; Dylla, M.; others Matminer: An open source toolkit for materials data mining. _Computational Materials Science_ 2018, _152_ , 60–69
  * Chen et al. 2019 Chen, C.; Ye, W.; Zuo, Y.; Zheng, C.; Ong, S. P. Graph networks as a universal machine learning framework for molecules and crystals. _Chemistry of Materials_ 2019, _31_ , 3564–3572
  * Choudhary and DeCost 2021 Choudhary, K.; DeCost, B. Atomistic line graph neural network for improved materials property predictions. _npj Computational Materials_ 2021, _7_ , 185
  * Xie and Grossman 2018 Xie, T.; Grossman, J. C. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. _Physical review letters_ 2018, _120_ , 145301
  * Lewis et al. 2022 Lewis, N. R.; Jin, Y.; Tang, X.; Shah, V.; Doty, C.; Matthews, B. E.; Akers, S.; Spurgeon, S. R. Forecasting of in situ electron energy loss spectroscopy. _npj Computational Materials_ 2022, _8_ , 252
  * Shapera and Schleife 2022 Shapera, E. P.; Schleife, A. Discovery of New Plasmonic Metals via High-Throughput Machine Learning. _Advanced Optical Materials_ 2022, _10_ , 2200158
  * Chander et al. 2015 Chander, S.; Purohit, A.; Nehra, A.; Nehra, S.; Dhaka, M. A study on spectral response and external quantum efficiency of mono-crystalline silicon solar cell. _International journal of renewable energy research_ 2015, _5_ , 41–44
  * Jha et al. 2019 Jha, D.; Choudhary, K.; Tavazza, F.; Liao, W.-k.; Choudhary, A.; Campbell, C.; Agrawal, A. Enhancing materials property prediction by leveraging computational and experimental data using deep transfer learning. _Nature communications_ 2019, _10_ , 5316
  * Hoffmann et al. 2023 Hoffmann, N.; Schmidt, J.; Botti, S.; Marques, M. A. Transfer learning on large datasets for the accurate prediction of material properties. _Digital Discovery_ 2023, _2_ , 1368–1379
  * Chen et al. 2021 Chen, C.; Zuo, Y.; Ye, W.; Li, X.; Ong, S. P. Learning properties of ordered and disordered materials from multi-fidelity data. _Nature Computational Science_ 2021, _1_ , 46–53
  * Wines et al. 2023 Wines, D.; Gurunathan, R.; Garrity, K. F.; DeCost, B.; Biacchi, A. J.; Tavazza, F.; Choudhary, K. Recent progress in the JARVIS infrastructure for next-generation data-driven materials design. _Applied Physics Reviews_ 2023, _10_ , 041302
  * Choudhary et al. 2020 Choudhary, K.; Garrity, K. F.; Reid, A. C.; DeCost, B.; Biacchi, A. J.; Hight Walker, A. R.; Trautt, Z.; Hattrick-Simpers, J.; Kusne, A. G.; Centrone, A.; others The joint automated repository for various integrated simulations (JARVIS) for data-driven materials design. _npj computational materials_ 2020, _6_ , 173
  * Choudhary et al. 2018 Choudhary, K.; Zhang, Q.; Reid, A. C.; Chowdhury, S.; Van Nguyen, N.; Trautt, Z.; Newrock, M. W.; Congo, F. Y.; Tavazza, F. Computational screening of high-performance optoelectronic materials using OptB88vdW and TB-mBJ formalisms. _Scientific data_ 2018, _5_ , 1–12
  * Klimeš et al. 2009 Klimeš, J.; Bowler, D. R.; Michaelides, A. Chemical accuracy for the van der Waals density functional. _Journal of Physics: Condensed Matter_ 2009, _22_ , 022201
  * Becke and Johnson 2006 Becke, A. D.; Johnson, E. R. A simple effective potential for exchange. _The Journal of chemical physics_ 2006, _124_ , 221101
  * Tran and Blaha 2009 Tran, F.; Blaha, P. Accurate band gaps of semiconductors and insulators with a semilocal exchange-correlation potential. _Physical review letters_ 2009, _102_ , 226401
  * Vinyals et al. 2015 Vinyals, O.; Bengio, S.; Kudlur, M. Order Matters: Sequence to sequence for sets. 2015; https://arxiv.org/abs/1511.06391
  * Abadi et al. 2015 Abadi, M. et al. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. 2015; https://www.tensorflow.org/, Software available from tensorflow.org
  * Chollet et al. 2015 Chollet, F.; others Keras. https://keras.io, 2015
  * Akiba et al. 2019 Akiba, T.; Sano, S.; Yanase, T.; Ohta, T.; Koyama, M. Optuna: A next-generation hyperparameter optimization framework. Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. 2019; pp 2623–2631
  * Joyce 2011 Joyce, J. M. In _International Encyclopedia of Statistical Science_ ; Lovric, M., Ed.; Springer Berlin Heidelberg: Berlin, Heidelberg, 2011; pp 720–722
  * Frogner et al. 2015 Frogner, C.; Zhang, C.; Mobahi, H.; Araya-Polo, M.; Poggio, T. Learning with a Wasserstein Loss. 2015; https://arxiv.org/abs/1506.05439
  * Ramdas et al. 2017 Ramdas, A.; García Trillos, N.; Cuturi, M. On wasserstein two-sample testing and related families of nonparametric tests. _Entropy_ 2017, _19_ , 47
  * van der Maaten and Hinton 2008 van der Maaten, L.; Hinton, G. Visualizing Data using t-SNE. _Journal of Machine Learning Research_ 2008, _9_ , 2579–2605
  * Hilfiker and Tiwald 2018 Hilfiker, J. N.; Tiwald, T. Dielectric function modeling. _Spectroscopic Ellipsometry for Photovoltaics: Volume 1: Fundamental Principles and Solar Cell Characterization_ 2018, 115–153
  * Kuzmany and Kuzmany 1998 Kuzmany, H.; Kuzmany, H. The dielectric function. _Solid-State Spectroscopy: An Introduction_ 1998, 101–120
  * Choudhary et al. 2019 Choudhary, K.; Bercx, M.; Jiang, J.; Pachter, R.; Lamoen, D.; Tavazza, F. Accelerated discovery of efficient solar cell materials using quantum and machine-learning methods. _Chemistry of materials_ 2019, _31_ , 5900–5908
  * Collins et al. 1972 Collins, D. G.; Blättner, W. G.; Wells, M. B.; Horak, H. G. Backward Monte Carlo calculations of the polarization characteristics of the radiation emerging from spherical-shell atmospheres. _Applied Optics_ 1972, _11_ , 2684–2696
  * Pedregosa et al. 2011 Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; others Scikit-learn: Machine learning in Python. _Journal of machine learning research_ 2011, _12_ , 2825–2830
  * Chen and Guestrin 2016 Chen, T.; Guestrin, C. Xgboost: A scalable tree boosting system. Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 2016; pp 785–794
  * Ke et al. 2017 Ke, G.; Meng, Q.; Finley, T.; Wang, T.; Chen, W.; Ma, W.; Ye, Q.; Liu, T.-Y. Lightgbm: A highly efficient gradient boosting decision tree. _Advances in neural information processing systems_ 2017, _30_ , 3146–3154
  * Dunn et al. 2020 Dunn, A.; Wang, Q.; Ganose, A.; Dopp, D.; Jain, A. Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm. _npj Computational Materials_ 2020, _6_ , 138

TOC Graphic.


# From NeRFs to Gaussian Splats, and Back

Siming He*, Zach Osman*, Pratik Chaudhari

From NeRFs to Gaussian Splats, and Back
Siming He*, Zach Osman*, Pratik Chaudhari

Figure 1:  NeRF generalizes better than Gaussian splatting (GS) to views that
are very different from those in the training data. Training and validation
views are similar in many datasets (see Figure 2 Top), so GS preforms well on
both. But if we render more novel views, NeRFs perform better. NeRF-SH (Left)
and Splatfacto (Right) are trained on Aspen (Top) and Giannini Hall (Bottom)
from Nerfstudio dataset [1]. Even though both NeRF-SH and Splatfacto have a
high PSNR, NeRF-SH renders better RGB and depth at the novel view.  Figure 2:
All images are rendered using Gaussian splats obtained from NeRF-SH without
fine-tuning; called NeRFGS with 0 iterations in Tab. 1. Top: Aspen and
Giannini Hall scenes from Nerfstudio [1] contain validation views that are
similar to training views. Although Splatfacto has good rendering quality on
these data in Tab. 1, it does not accurately render more dissimilar views in
Figure 1. Bottom: We create two new scenes for this paper, named Wissahickon
and Locust Walk, where validation views are much more different than training
views. For these new scenes, Splatfacto has a much worse validation PSNR than
NeRF; see Tab. 1.

## Introduction

Radiance††*Equal Contribution. General Robotics, Automation, Sensing and
Perception (GRASP) Laboratory, University of Pennsylvania. Email: {siminghe,
osmanz<EMAIL_ADDRESS>††Code and data are in
https://github.com/grasp-lyrl/NeRFtoGSandBack. field scene representations are
promising for robotics, e.g., localization and mapping [2, 3, 4, 5, 6, 7, 8],
planning and control [9, 10, 11, 12, 13], scene understanding [14, 15], and
simulation [16, 17, 18]. Often, the key question in these applications is
whether one uses an implicit representation like a neural radiance field
(NeRF) [19, 20] or an explicit representation, like 3D Gaussian Splatting (GS)
[21, 22]. There are pros and cons for both.

Imagine a quadruped robot walking along a road. It is important to ensure that
the scene representation built from its ego-centric views generalizes to new
views, e.g., from the other side of the road. While non-parametric GS models
can do well on datasets where train and test views are rather adequate and
similar [19, 23], they do not generalize well to new views when train views
are limited. Parametric models like NeRFs work much better; see Figure 1.
Heuristics to move, merge, and split the Gaussians in GS are brittle for in-
the-wild data with exposure variations and motion blur [24]. In our
experience, training of NeRFs is more stable and results in better geometry
with limited views. NeRFs are also a more compact representation of the scene
and require less memory than GS. This is important for resource-constrained
robots. The difference is rather obvious for distilled feature fields. NeRF-
based methods [14, 25] can store high-dimensional features efficiently while
GS-based methods [26, 27, 28] need additional steps to compress the features.

Figure 3:  NeRFGS generalizes better than GS while having real-time rendering.
NeRFGS converts trained NeRF-SH into GS while maintaining the good
generalization in contrast to Splatfacto in Figure 1. The conversion takes
about 10 sec: 7 sec for spherical harmonics extraction and 3 sec for fine-
tuning. The conversion time can be further shortened if conversion is done
only around the robot and the sky regions are not fine-tuned. Rendered sky is
bad but it is not evaluated and does not matter in most robotics tasks.
Figure 4: GSNeRF can be used for saving memory and editing the NeRF. We
convert NeRFGS model as in Figure 3 back to NeRF-SH (Left). GSNeRF stores
memory intensive GS models as more compact NeRFs. Also, GSNeRF with NeRFGS
enables easy modifications to the scene represented by the NeRF by converting
into a GS, modifying the Gaussians, and converting it back. As an example, we
have edited out the lamp post from Giannini Hall in Figure 1 through GSNeRF
and rendered the image (Right) using NeRF-SH.

| Iterations | Aspen | Giannini Hall | Wissahickon | Locust Walk
---|---|---|---|---|---
|  ($\times 10^{2}$) | PSNR (Val) | SSIM | LPIPS | PSNR (Val) | SSIM | LPIPS | PSNR (Train/Val) | SSIM | LPIPS | PSNR (Train/Val) | SSIM | LPIPS
Nerfacto-big [1] | 300 | 17.75 | 0.5 | 0.43 | 20.11 | 0.68 | 0.3 | 22.17 / 20.75 | 0.75 | 0.26 | 22.29 / 21.49 | 0.8 | 0.3
Splatfacto [1] | 300 | 17.63 | 0.5 | 0.39 | 20.87 | 0.7 | 0.33 | 23.46 / 14.62 | 0.55 | 0.45 | 24.04 / 17.72 | 0.7 | 0.31
NeRF-SH | 300 | 17.73 | 0.48 | 0.45 | 19.89 | 0.65 | 0.32 | 22.41 / 17.46 | 0.61 | 0.39 | 21.73 / 18.74 | 0.7 | 0.33
RadGS [24] | 1 | 11.65 | 0.28 | 0.74 | 12.37 | 0.49 | 0.61 | 12.4 / 15.17 | 0.62 | 0.46 | 10.84 / 11.85 | 0.6 | 0.46
RadGS [24] | 10 | 17.85 | 0.51 | 0.44 | 20.84 | 0.72 | 0.3 | 20.7 / 20.73 | 0.76 | 0.29 | 21.15 / 21.04 | 0.8 | 0.25
NeRFGS | 0 | 13.96 | 0.3 | 0.58 | 16.19 | 0.47 | 0.49 | \- / 14.40 | 0.47 | 0.51 | \- / 14.87 | 0.51 | 0.47
NeRFGS | 1 | 14.06 | 0.34 | 0.57 | 15.73 | 0.53 | 0.46 | 16.62 / 17.07 | 0.63 | 0.4 | 15.7 / 17.22 | 0.65 | 0.37
NeRFGS | 10 | 17.7 | 0.51 | 0.4 | 21.05 | 0.73 | 0.26 | 20.67 / 20.64 | 0.75 | 0.27 | 21.11 / 21.14 | 0.8 | 0.24
GSNeRF | 50 | 18.1 | 0.44 | 0.44 | 21.22 | 0.69 | 0.31 | \- / 17.65 | 0.63 | 0.39 | \- / 19.32 | 0.71 | 0.33
GSNeRF | 300 | 18.58 | 0.51 | 0.36 | 23.71 | 0.82 | 0.17 | \- / 17.59 | 0.64 | 0.37 | \- / 19.32 | 0.72 | 0.31

Table 1: Quantitative Results: All GS based methods can render at more than 40
FPS on GeForce RTX 4090. Training for 100 iterations takes about 6 and 3
seconds respectively for NeRF-based and GS-based methods. GSNeRF was
supervised using the Gaussians obtained from NeRFGS with 5000 iterations. For
Wissahickon and Locust Walk where validation views are dissimilar to training
views, we notice a large gap between the training and validation PSNR for
Splatfacto. In contrast, the rendering quality at validation views is already
reasonable for NeRFGS after 100 iterations of fine-tuning. After training 1000
iterations, NeRFGS has better quality than Splatfacto and NeRF-SH. Compared to
RadGS, the convergence is faster, i.e., better results after 100 iterations,
also shown in Figure 5. GSNeRF also trains much faster and results in better
quality, also shown in Figure 5.

Explicit representations can achieve much faster rendering than implicit
representations like NeRF, because rasterization of Gaussians or triangles can
be done efficiently on a GPU. High-rendering speed is important for real-time
robot operations, for localization (which requires checking many views to
ascertain visual overlap with the current observation), planning (which
requires synthesizing new views along putative trajectories before one of them
is selected), etc. Robots need to model and operate in complex environments
with dynamic objects. Explicit representations can be modified easily, e.g.,
by updating the Gaussians. Modifying implicit representations requires
expensive re-training, complex modeling [29, 30, 31, 32, 33], or different
NeRFs for different objects [15, 34].

We develop an efficient procedure to go back and forth between implicit
representations and explicit ones—and thereby get the best of both approaches.
To our knowledge, this manuscript seems to be the first implementation of this
idea. We iteratively train NeRFs for mapping and feature distillation, then
convert the NeRF to GS for real-time localization, and handling changing
scenes. We also convert the GS back to NeRF to update the map and distilled
features. We evaluate the quality and efficiency of this approach using a
number of existing datasets; we also study this approach on views recorded
from an ego-centric camera along hiking trails in situations when evaluation
views are dissimilar to training views. This approach achieves the best of
both NeRFs (superior PSNR, SSIM, and LPIPS on the dissimilar views, and a
compact representation) and GS (real-time rendering and ability for easily
modifying the representation); the computational cost of these conversions is
minor compared to training the two from scratch.

## Approach and Results

We modify Nerfacto to predict spherical harmonics (degree 3, i.e., 16
coefficients) for each rgb channel. The volume rendering equation remains
unchanged; we simply calculate the RGB colors from the spherical harmonics
using the viewing direction while integrating them along the ray.

Given such a trained “NeRF-SH”, we calculate a point-cloud of the scene using
the median depth along 2$\times 10^{6}$ rays rendered from training views; we
ensure that these rays have high opacity and do not correspond to the sky.
Isotropic Gaussians are initialized at each of these points using the density
and spherical harmonics predicted by the NeRF-SH. The scale of each Gaussian
is half of the average distance between each point and its three nearest
neighbors. Without any further optimization, this “NeRFGS” already captures
geometric and photometric properties of the scene impressively well; see
Figures 2 and 1. We can fine-tune it further using training views; see Figures
3 and 1.

We render images using NeRFGS from training views, and fit or update a NeRF-
SH; we call this “GSNeRF”. We noticed that training NeRFs using the GS-
rendered views gives better PSNR, SSIM, and LPIPS as compared to using the
original images; see Figure 5 and Tab. 1. This is perhaps due to the absence
of high-frequency structures in the GS-rendered views. Converting an explicit
representation to an implicit one like this is useful to update the NeRFs or
update distilled features. We show an example in Figure 4 where we manually
edit out the lamp-post by selecting the corresponding splats in NeRFGS and
update NeRF through GSNeRF in 4.8 sec.

Figure 5:  Quality as a function of training progress on Aspen

## Discussion

We demonstrated a simple procedure to convert between implicit representations
of the scene such as NeRFs and explicit representations such as Gaussian
splatting (GS). These ideas are useful to handle situations with sparse views,
which are commonly encountered in robotics. There are many ways one might
build upon this work. For example, rendering speed of GS is very useful for
localization and planning while the minimal resource requirements of NeRFs are
more suited for mapping. Notice that in Tab. 1 the PSNR of NeRFGS without
fine-tuning is lower than that of NeRF-SH. This indicates that there is a
large degree of inefficiency in how we convert NeRF-SH into the explicit
representation.

## References

  * Tancik et al. [2023] Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa.  Nerfstudio: A modular framework for neural radiance field development.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , SIGGRAPH ’23, 2023.
  * Sucar et al. [2021] Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew J. Davison.  imap: Implicit mapping and positioning in real-time.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 6229–6238, October 2021.
  * Rosinol et al. [2023] Antoni Rosinol, John J. Leonard, and Luca Carlone.  Nerf-slam: Real-time dense monocular slam with neural radiance fields.  In _2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 3437–3444, 2023.  doi: 10.1109/IROS55552.2023.10341922.
  * Huang et al. [2024a] Huajian Huang, Longwei Li, Hui Cheng, and Sai-Kit Yeung.  Photo-slam: Real-time simultaneous localization and photorealistic mapping for monocular, stereo, and rgb-d cameras, 2024a.
  * Yugay et al. [2023] Vladimir Yugay, Yue Li, Theo Gevers, and Martin R. Oswald.  Gaussian-slam: Photo-realistic dense slam with gaussian splatting, 2023.
  * Keetha et al. [2024] Nikhil Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang, Sebastian Scherer, Deva Ramanan, and Jonathon Luiten.  Splatam: Splat, track & map 3d gaussians for dense rgb-d slam, 2024.
  * Matsuki et al. [2024] Hidenobu Matsuki, Riku Murai, Paul H. J. Kelly, and Andrew J. Davison.  Gaussian splatting slam, 2024.
  * Yan et al. [2024] Chi Yan, Delin Qu, Dan Xu, Bin Zhao, Zhigang Wang, Dong Wang, and Xuelong Li.  Gs-slam: Dense visual slam with 3d gaussian splatting, 2024.
  * Adamkiewicz et al. [2022] Michal Adamkiewicz, Timothy Chen, Adam Caccavale, Rachel Gardner, Preston Culbertson, Jeannette Bohg, and Mac Schwager.  Vision-only robot navigation in a neural radiance world.  _IEEE Robotics and Automation Letters_ , 7(2):4606–4613, 2022.  doi: 10.1109/LRA.2022.3150497.
  * He et al. [2024] Siming He, Christopher D. Hsu, Dexter Ong, Yifei Simon Shao, and Pratik Chaudhari.  Active perception using neural radiance fields, 2024.
  * Liu et al. [2024] Guangyi Liu, Wen Jiang, Boshu Lei, Vivek Pandey, Kostas Daniilidis, and Nader Motee.  Beyond uncertainty: Risk-aware active view acquisition for safe robot navigation and 3d scene understanding with fisherrf, 2024.
  * Lei et al. [2024] Xiaohan Lei, Min Wang, Wengang Zhou, and Houqiang Li.  Gaussnav: Gaussian splatting for visual navigation, 2024.
  * Li et al. [2021a] Yunzhu Li, Shuang Li, Vincent Sitzmann, Pulkit Agrawal, and Antonio Torralba.  3d neural scene representations for visuomotor control, 2021a.
  * Shen et al. [2023] William Shen, Ge Yang, Alan Yu, Jansen Wong, Leslie Pack Kaelbling, and Phillip Isola.  Distilled feature fields enable few-shot language-guided manipulation.  In _7th Annual Conference on Robot Learning_ , 2023.
  * Yang et al. [2023] Jiawei Yang, Boris Ivanovic, Or Litany, Xinshuo Weng, Seung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fidler, Marco Pavone, and Yue Wang.  Emernerf: Emergent spatial-temporal scene decomposition via self-supervision, 2023.
  * Byravan et al. [2022] Arunkumar Byravan, Jan Humplik, Leonard Hasenclever, Arthur Brussee, Francesco Nori, Tuomas Haarnoja, Ben Moran, Steven Bohez, Fereshteh Sadeghi, Bojan Vujatovic, and Nicolas Heess.  Nerf2real: Sim2real transfer of vision-guided bipedal motion skills using neural radiance fields, 2022.
  * Cleac’h et al. [2023] Simon Le Cleac’h, Hong-Xing Yu, Michelle Guo, Taylor A. Howell, Ruohan Gao, Jiajun Wu, Zachary Manchester, and Mac Schwager.  Differentiable physics simulation of dynamics-augmented neural objects, 2023.
  * Xu et al. [2024] Xiaohao Xu, Tianyi Zhang, Sibo Wang, Xiang Li, Yongqi Chen, Ye Li, Bhiksha Raj, Matthew Johnson-Roberson, and Xiaonan Huang.  Customizable perturbation synthesis for robust slam benchmarking, 2024.
  * Mildenhall et al. [2020] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _ECCV_ , 2020.
  * Müller et al. [2022] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Trans. Graph._ , 41(4):102:1–102:15, July 2022.  doi: 10.1145/3528223.3530127.  URL https://doi.org/10.1145/3528223.3530127.
  * Kerbl et al. [2023] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.  3d gaussian splatting for real-time radiance field rendering.  _ACM Transactions on Graphics_ , 42(4), July 2023.  URL https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/.
  * Huang et al. [2024b] Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao.  2d gaussian splatting for geometrically accurate radiance fields, 2024b.
  * Barron et al. [2022] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  _CVPR_ , 2022.
  * Niemeyer et al. [2024] Michael Niemeyer, Fabian Manhardt, Marie-Julie Rakotosaona, Michael Oechsle, Daniel Duckworth, Rama Gosula, Keisuke Tateno, John Bates, Dominik Kaeser, and Federico Tombari.  Radsplat: Radiance field-informed gaussian splatting for robust real-time rendering with 900+ fps.  _arXiv.org_ , 2024.
  * Kerr et al. [2023] Justin* Kerr, Chung Min* Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik.  Lerf: Language embedded radiance fields.  In _International Conference on Computer Vision (ICCV)_ , 2023.
  * Shi et al. [2023] Jin-Chuan Shi, Miao Wang, Hao-Bin Duan, and Shao-Hua Guan.  Language embedded 3d gaussians for open-vocabulary scene understanding.  _arXiv preprint arXiv:2311.18482_ , 2023.
  * Qin et al. [2023] Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister.  Langsplat: 3d language gaussian splatting.  _arXiv preprint arXiv:2312.16084_ , 2023.
  * Zuo et al. [2024] Xingxing Zuo, Pouya Samangouei, Yunwen Zhou, Yan Di, and Mingyang Li.  Fmgs: Foundation model embedded 3d gaussian splatting for holistic 3d scene understanding, 2024.
  * Park et al. [2021] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  _ICCV_ , 2021.
  * Pumarola et al. [2020] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-NeRF: Neural Radiance Fields for Dynamic Scenes.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * Li et al. [2021b] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang.  Neural scene flow fields for space-time view synthesis of dynamic scenes, 2021b.
  * Fang et al. [2022] Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian.  Fast dynamic radiance fields with time-aware neural voxels.  In _SIGGRAPH Asia 2022 Conference Papers_ , 2022.
  * Li et al. [2023] Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely.  Dynibar: Neural dynamic image-based rendering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Kong et al. [2023] Xin Kong, Shikun Liu, Marwan Taher, and Andrew J Davison.  vmap: Vectorised object mapping for neural field slam.  _arXiv preprint arXiv:2302.01838_ , 2023.