# Effective Diversity in Unsupervised Environment Design

Submission Number: 4071  Wenjun Li1 Pradeep Varakantham1 Dexun Li1 1Singapore
Management University

###### Abstract

Agent decision making using Reinforcement Learning (RL) heavily relies on
either a model or simulator of the environment (e.g., moving in an 8x8 maze
with three rooms, playing Chess on an 8x8 board). Due to this dependence,
small changes in the environment (e.g. positions of obstacles in the maze,
size of the board) can severely affect the effectiveness of the policy learnt
by the agent. To that end, existing work has proposed training RL agents on an
adaptive curriculum of environments (generated automatically) to improve
performance on out-of-distribution (OOD) test scenarios. Specifically,
existing research has employed the potential for the agent to learn in an
environment (captured using Generalized Advantage Estimation, GAE) as the key
factor to select the next environment(s) to train the agent. However, such a
mechanism can select similar environments (with a high potential to learn)
thereby making agent training redundant on all but one of those environments.
To that end, we provide a principled approach to adaptively identify diverse
environments based on a novel distance measure relevant to environment design.
We empirically demonstrate the versatility and effectiveness of our method in
comparison to multiple leading approaches for unsupervised environment design
on three distinct benchmark problems used in literature.

##  1 Introduction

Deep Reinforcement Learning (DRL) has had many successes in challenging tasks
(e.g. Atari Mnih et al. (2015), AlphaGO Silver et al. (2016), solving Rubik
cube Akkaya et al. (2019), chip design Mirhoseini et al. (2021)) in the last
decade. However, DRL agents have been proven to be brittle and often fail to
transfer well to environments only slightly different from those encountered
during training Zhang et al. (2018); Cobbe et al. (2019), e.g., changing size
of the maze, background in a game, positions of obstacles. To achieve robust
and generalizing policies, agents must be exposed to different types of
environments that assist in learning different challenges Cobbe et al. (2020).
To that end, existing research has proposed Unsupervised Environment Design
(UED) that can create a curriculum (or distribution) of training scenarios
that are adaptive with respect to agent policy Mehta et al. (2020); Portelas
et al. (2020); Wang et al. (2019a); Dennis et al. (2020); Jiang et al. (2021b,
a). Following the terminology in existing works, we will refer to a particular
environment configuration as a level, e.g., an arrangement of blocks in the
maze, shape/height of the obstacles in front of the agent, the geometric
configuration of racing tracks, etc.

Related work on UED: Existing works on UED can be categorized along two
threads. The first thread has focused on principled generation of levels and
was pioneered by the Protagonist Antagonist Induced Regret Environment Design
(PAIRED, Dennis et al. (2020)) algorithm. PAIRED introduced a multi-agent game
between level generator (teacher), antagonist (expert student) and protagonist
(normal student), and utilized the performance difference between antagonist
and protagonist as a reward signal to guide the generator to adaptively create
challenging training levels. The second thread has abandoned the idea of
principled level generation and instead championed: (a) randomly generating
levels and (b) replaying previously considered levels to deal with
catastrophic forgetting by the student agent. The algorithm is referred to as
Prioritized Level Replay (PLR, Jiang et al. (2021b)) and the levels are
replayed based on learning potential, captured using Generalized Advantage
Estimation (GAE, Schulman et al. (2015)). PLR has been empirically shown to be
more scalable and also is able to achieve more robust and better “out-of-
distribution” performance than PAIRED. In Jiang et al. (2021a), authors
combined randomized generator and replay mechanism together and proposed PLR⟂.
Empirically, PLR⟂ achieves the state-of-the-art in literature (as a method
that demands no human expertise), and thus we will adopt PLR⟂ as our primary
baseline in this paper. Another recent approach ACCEL Parker-Holder et al.
(2022) that builds on PLR performs edits (i.e. modifications based on human
expertise) on high regret environments to learn better. Both these threads of
research have significantly improved the state-of-the-art (domain
randomization Sadeghi and Levine (2016); Tobin et al. (2017) and adversarial
learning Pinto et al. (2017); Khirodkar et al. (2018)). However, to generalize
better, it is not sufficient to train the agent only on high regret/GAE
levels. We can have multiple high regret levels, which are very “similar” to
each other and agent does not learn a lot from being trained on similar
levels. Thus, levels also have to be sufficiently “different”, so that agent
can gain more perspective on different challenges that it can face.

To that end, in this paper, we introduce a diversity metric in UED, which is
defined based on the distance between occupancy distributions associated with
agent trajectories from different levels. We then provide a principled method,
referred to as Diversity Induced Prioritized Level Replay (DIPLR) to select
levels with the diversity metric to provide better generalization performance
than the state-of-the-art.

Related Work on Diversity in RL: There is existing literature on diversity
measurement in the field of transfer RL, e.g. Song et al. (2016); Agarwal et
al. (2021a); Wang et al. (2019a). However, these works conduct an exhaustive
comparison of any possible states between two underlying Markov Decision
Processes (MDPs), which does not take the current agent policy into account,
and may suffer from a large computational cost. In contrast, we use the
collected trajectories induced by the policy to approximate the occupancy
distribution of the different levels. Such a method can avoid massive
computation and more robustly indicate the difference between two levels being
explored by the same agent policy. There are a few works that have explored
the diversity between policies in DRL. 1) Hong et al. (2018) adopted distance
on pairwise policies to modify the loss function to enforce the agent to
attempt policies different from its prior policies. As an improved version on
Hong et al. (2018), Parker-Holder et al. (2020) computes the diversity on a
population of policies instead of pairwise distance, so that it avoids cycling
training behaviors and policy redundancy. 2) Lupu et al. (2021) proposed
trajectory diversity measurement based on Jensen-Shannon Divergence (JSD) and
included it in the loss function to maximize the difference between policies.
The JSD computes the divergence based on action distribution in trajectories.
In contrast to these works, our method computes both pairwise and population-
wise distance for the purpose of measuring the difference between levels.
Furthermore, we provide a diversity guided method to train a robust and well-
generalizing agent in the UED framework

Contributions:
Overall, our contributions are three-fold. First, we highlight the benefits of
diversity in UED tasks and formally present the definition of distance between
levels. Second, we employ Wasserstein distance to quantitatively measure the
distance and introduce the DIPLR algorithm. Finally, we empirically
demonstrate the versatility and effectiveness of DIPLR in comparison to other
leading approaches on benchmark problems. Notably, we also investigate the
relationship between diversity and regret (i.e. learning potential) and
conduct an ablation study to reveal their individual effectiveness.
Surprisingly, we find that diversity solely works better than regret in the
model, and they complement each other to achieve the best performance when
combined.

##  2 Background

In this section, we describe the problem of UED and also the details of the
leading approaches for solving UED.

###  2.1 Unsupervised Environment Design, UED

The goal of UED is to train a student agent that performs well across a large
set of different environments. To achieve this goal, there is a teacher agent
in UED that provides a curriculum of environment parameter values to train the
student agent to generalize well to unseen levels.

UED problem is formally described using an Underspecified Partially Observable
Markov Decision Process (UPOMDP). It is defined using the following tuple:

$\langle S,A,\Theta,I,O,T,R,\gamma\rangle$

$S$, $A$ and $O$ are the set of states, actions and observations respectively.
$R:S\rightarrow\mathbb{R}$ is the reward function and $\gamma$ is the discount
factor. The most important element in the tuple is $\Theta$, which is the set
of environment parameters. A particular parameter $\theta\in\Theta$ (can be a
vector or sequence of values) defines a level and can impact the reward model,
transition dynamics and the observation function, i.e.
$R:S\times\Theta\rightarrow\mathbb{R}$, $T:S\times A\times\Theta\rightarrow S$
and $I:S\times\Theta\rightarrow O$. UPOMDP is under specified, because we
cannot train with all values of $\theta$ ($\in\Theta$), as $\Theta$ can be
infinitely large. The goal of the student agent policy $\pi$ in a UPOMDP is to
maximize its discounted expected rewards for any given $\theta\in\Theta$:

$\mathop{\max}\limits_{\pi}V^{\theta}(\pi)=\mathop{\max}\limits_{\pi}\mathbb{E}_{\pi}V^{\theta}(\tau)=\mathop{\max}\limits_{\pi}\mathbb{E}_{\pi}\Big{[}\sum_{t=0}^{H}r_{t}^{\theta}\cdot\gamma^{t}\Big{]}$

where $r^{\theta}_{t}$ is the reward obtained by $\pi$ in a level with
environment parameter $\theta$ at time step $t$. Thus, the student agent needs
to be trained on a series of $\theta$ values that maximize its generalization
capability on all possible levels from $\Theta$. To that end, we employ the
teacher agent. The goal of the teacher agent policy $\Lambda$ is to generate a
distribution over the next set of environment parameter values to train the
student, i.e.,

$\Lambda:\Pi\rightarrow\Delta(\Theta)$

to achieve good generalization performance, where $\Pi$ is the set of possible
policies of the teacher.

###  2.2 Approaches for Solving UED

In all approaches for solving UED, student always optimizes a policy that
maximizes its value on the given $\theta$. The different approaches for
solving UED vary on the method adopted by the teacher agent. We now elaborate
on the $\Lambda$ employed by different approaches.

Two fundamental approaches to UED are Domain Randomization (DR) Jakobi (1997);
Tobin et al. (2017) and Minimax Morimoto and Doya (2005); Pinto et al. (2017).
Teacher in DR simply randomizes the environment configurations regardless of
the student’s policy, i.e.,

$\Lambda^{DR}(\pi)={\cal U}(\Theta)$

where ${\cal U}$ is the uniform distribution over all possible $\theta$
values. On the other hand, teacher in Minimax adversarially generates
challenging environments to minimize the rewards of the student’s policy,
i.e.,

$\Lambda^{MM}(\pi)=\arg\min_{\theta}V^{\theta}(\pi)$

The next set of approaches related to PAIRED Dennis et al. (2020); Jiang et
al. (2021a) rely on regret, which is defined approximately as the difference
between the maximum and the mean return of students’ policy, to generate new
levels:

$reg^{\theta}(\pi)\approx\max_{\tau\sim\pi}V^{\theta}(\tau)-\mathbb{E}_{\tau\sim\pi}V^{\theta}(\tau)$

Given this regret, teacher selects policies that maximize regret:

$\Lambda^{MR}(\pi)=\arg\max_{\theta}reg^{\theta}(\pi)$

From the original paper by Dennis et al. (2020), there have been multiple
major improvements Jiang et al. (2021b); Parker-Holder et al. (2022):

  1. 1.

New levels are generated randomly instead of using an optimizer.

  2. 2.

Level to be selected at a time step is decided based on an efficient
approximation of regret, referred to as positive value loss (a customized form
of Generalized Advantage Estimation (GAE)):

$gae^{\theta}(\pi)=\frac{1}{H}\sum_{t=0}^{H}\max\left(\sum_{k=t}^{H}(\gamma\lambda)^{k-t}\delta_{k},0\right)$
(1)

where $\gamma$ and $\lambda$ are the GAE and MDP discount factors
respectively, $H$ is the time horizon and $\delta_{k}$ is the TD-error at time
step $k$.

  3. 3.

Levels are replayed with a certain probability to ensure there is no
catastrophic forgetting.

  4. 4.

Edit randomly on generated levels through human-defined edits Parker-Holder et
al. (2022).

We will use learning potential, regret and GAE, interchangeably to represent
positive value loss in the rest of the paper.

###  2.3 Wasserstein Distance

We will propose a diversity measure that relies on distance between occupancy
distributions, where we utilize Wasserstein Distance. The problem of optimal
transport density between two distributions was initially proposed by Monge
(1781) and generalized by Kantorovich (1942). Wasserstein distance is
preferred by the machine learning community among several commonly used
divergence measures, e.g. Kullback-Leibler divergence, because it is 1) non-
zero and continuously differentiable when the support of two distributions is
disjoint and 2) symmetric, i.e., $\mathcal{W}(\mathcal{P},\mathcal{Q})$ is
equal to $\mathcal{W}(\mathcal{Q},\mathcal{P})$.

Wasserstein distance is the measurement between probability distributions
defined on a metric space $M(d,\mathcal{C})$ with a cost metric
$d:\mathcal{C}\times\mathcal{C}\mapsto\mathbb{R}_{+}$:

$\displaystyle\mathcal{W}_{p}(\mathcal{P},\mathcal{Q})$ $\displaystyle=$
$\displaystyle\left(\mathop{inf}\limits_{\gamma\in\Pi(\mathcal{P},\mathcal{Q})}\mathbb{E}_{(x,y)\sim\gamma}[d(x,y)^{p}]\right)^{1/p}$
(2)

where $\Pi(\mathcal{P},\mathcal{Q})$ is the set of all possible joint
probability distributions between $\mathcal{P}$ and $\mathcal{Q}$, and one
joint distribution $\gamma\in\Pi(\mathcal{P},\mathcal{Q})$ represents a
density transport plan from point $x$ to $y$ so as to make $x$ follows the
same probability distribution of $y$.

##  3 Approach: DIPLR

A key drawback of existing approaches for solving UED is the inherent
assumption that levels with high regret (or GAE) all have high learning
potential. However, if there are two levels111We will use levels and
environments interchangeably. which are very similar to each other, and they
both have high regret, training the student agent on one of those levels makes
training on the other unnecessary and irrelevant. Given our ultimate goal is
to let student agent policy transfer well on to a variety of few-shot or zero-
shot levels, we utilize diversity. Diversity has gained traction in Deep
Reinforcement Learning to improve the generalization performance (albeit in a
different problem setting than the one in the paper) of models (Parker-Holder
et al. (2020); Hong et al. (2018); Lupu et al. (2021)). More specifically, we
provide

  1. 1.

a scalable mechanism for quantitatively estimating similarity (or
dissimilarity) between two given levels given the current student policy. This
will then be indirectly used to generate diverse levels.

  2. 2.

a teacher agent for UED that generates diverse levels and trains student agent
on those diverse environments, so as to achieve strong generalization to ”out-
of-distribution” levels.

###  3.1 Estimating Similarity between Levels

The objective here is to estimate similarity between levels given the current
student agent policy. One potential option is to encode each level using one
of a variety of encoding methods (e.g., Variational Autoencoder) or using the
parameters associated with the level and then take distance between the
encodings of the levels. However, such an approach has multiple major issues:

  * $\bullet$

Encoding a level does not account for the sequential moves the student agent
will make through the level, which is of critical importance as the similarity
is with respect to student policy;

  * $\bullet$

There can be stochasticity in the level that is not captured by the parameters
of the level. For instance, in Bipedal-Walker, a complex yet well-
parameterized UPOMDP introduced by Wang et al. (2019b); Parker-Holder et al.
(2022), has multiple free parameters controlling the terrain. Because of the
existence of stochasticity in the level generation process, two levels could
be very different while we have near-zero distance measurement given their
environment parameter vectors.

  * $\bullet$

Distance between environment parameters requires normalization in each
parameter dimension and is domain-specific.

Since we collect several trajectories within current levels when approximating
the regret value, we can naturally get the state-action distribution induced
by the current policy. Therefore, we propose to evaluate similarity on the
different levels based on the distance between occupancy distributions of the
current student policy. The hypothesis is that if the levels are similar, then
the trajectories traversed by the student agent within the level will have
similar state-action distribution. Equation (3) provides the expression for
state-action distribution given a policy, $\pi$ and level $l_{\theta_{1}}$
($\theta_{1}$ represents the parameters corresponding to the level):

$\displaystyle\rho^{\pi}_{l_{\theta_{1}}}(s,a)$
$\displaystyle=(1-\gamma)\sum_{t=0}^{H}\Big{[}Pr(s_{t}=s,a_{t}=a|s_{0}\sim
p_{0}(.),$ $\displaystyle\hskip 28.90755pts_{t}\sim
p(.|s_{t-1},a_{t-1},\theta_{1}),a_{t}\sim\pi(.|s_{t})\Big{]}$ (3)

Similarly, we can derive $\rho^{\pi}_{l_{\theta_{2}}}$ for level,
$l_{\theta_{2}}$ with parameter $\theta_{2}$. Typically, KL divergence is
employed to measure the distance between state-action distributions.

$-D_{KL}(\rho^{\pi}_{l_{\theta_{1}}}||\rho^{\pi}_{l_{\theta_{2}}})=\mathbb{E}_{(s,a)\sim\rho^{\pi}_{\theta_{1}}}\Big{[}\log\frac{\rho^{\pi}_{l_{\theta_{2}}}}{\rho^{\pi}_{l_{\theta_{1}}}}\Big{]}$

However, KL divergence is not applicable in our setting, because of lack of
transition probabilities, the two different levels can result in two occupancy
distributions with disjoint supports. More importantly, KL divergence cannot
work without explicit estimates of the occupancy distributions. Therefore, we
employ the Wasserstein distance described in Equation (2), which can calculate
the distance between two distributions from empirical samples.

We can write the Wasserstein distance between two occupancy distributions as
Equation (4):

$\mathcal{W}(\rho^{\pi}_{l_{\theta_{1}}},\rho^{\pi}_{l_{\theta_{2}}})=\left(\mathop{inf}\limits_{\gamma\in\Pi(\rho^{\pi}_{l_{\theta_{1}}},\rho^{\pi}_{l_{\theta_{2}}})}\mathbb{E}_{(\phi_{1},\phi_{2})\sim\gamma}[d(\phi_{1},\phi_{2})^{p}]\right)^{1/p}$

(4)

where $\phi\in(S,A)$ is a sample from the occupancy distribution. By Equation
(4), we can collect state-action samples in trajectories to compute the
empirical Wasserstein distance between two levels, i.e. ${\cal
D}(l_{\theta_{1}},l_{\theta_{2}})\approx\mathcal{W}(\rho^{\pi}_{l_{\theta_{1}}},\rho^{\pi}_{l_{\theta_{2}}})$,
is our empirical estimation of the Wasserstein distance between two levels.

###  3.2 Diversity guided Training Agent

Figure 1: An Overview of DIPLR Algorithm

Now that we have a scalable way to compute the distance between levels, ${\cal
D}(\cdot,\cdot)$, we will utilize it in the teacher to generate a diverse
curriculum for the student agent. We build on the PLR training agent Jiang et
al. (2021a) that only considered regret/GAE and stateless to prioritize levels
for selection. Our training agent most crucially employs diversity to select
levels.

Our training agent maintains a buffer of high-potential levels for training.
At each iteration, we either: (a) generate a new level to be added to the
buffer; or (b) sample a mini-batch of levels for the student agent to train
on.

Diversity during generating a new level: For a new level, $l_{\theta}$, the
distance from $l_{\theta}$ to all the levels,
$L=\left<l_{\theta_{1}},l_{\theta_{2}},\cdots\right>$ in the buffer is given
by:

$\displaystyle{\cal D}(l_{\theta},L)=\min_{k}{\cal
D}(l_{\theta},l_{\theta_{k}})$ (5)

To increase diversity, we add new levels to the buffer that have the highest
value of this distance (amongst all the randomly generated levels). We can
also combine this distance measure with regret/GAE, when taking a decision to
include a level in the buffer, so that different and challenging environments
are added.

Diversity in sampling levels from buffer to train: To decide on levels to
train on, we maintain priority (probability of getting selected) for each
level that is computed based on their distance from other levels in the
buffer. The rank of a level, $l_{\theta_{i}}$ in terms of distance from other
levels in the buffer, $L$ is given by

$h({\cal D}_{i},\left<{\cal D}_{1},{\cal D}_{2},\cdots\right>)$

where ${\cal D}_{i}={\cal D}(l_{\theta_{i}}L\setminus l_{\theta_{i}})$. We
then convert the rank into a probability using Equation (6),

$\displaystyle P_{i}$ $\displaystyle=$ $\displaystyle\frac{1}{h({\cal
D}_{i},\mathbf{{\cal D}})^{\beta}}$ (6)

where $h(\cdot,\cdot)$ is the rank transformation that can find the rank of
${\cal D}_{i}$ in a set of values $\cal D$, and $\beta$ is a tunable
temperature parameter.

Combining Diversity with Regret/GAE: Instead of solely considering the
diversity aspect of the level buffer, we can also take learning potential into
account by using regrets in Equation (1) so that we will have a level buffer
that is not only filled up with diverse training levels but also challenging
levels that continuously push the student. We could assign different weights
to diversity and regret by letting the replay probability
$P_{replay}=\rho\cdot P_{D}+(1-\rho)\cdot P_{R}$, where $P_{D}$ and $P_{R}$
are the prioritization of diversity and regret respectively, and $\rho$ is the
tuning parameter.

An overview of our proposed algorithm is shown in Figure 1. When the level
generator creates a new level $l_{\theta_{i}}$, we collect trajectories
$\tau_{i}$ on $l_{\theta_{i}}$, and compute its distance ${\cal D}_{i}$ to the
trajectory buffer by Equation (5). If the replay probability of
$l_{\theta_{i}}$ is greater than that of any levels in the level buffer, we
insert $l_{\theta_{i}}$ into the level buffer and remove the level with
minimum replay probability from the level buffer.

The complete procedure of DIPLR is presented in Algorithm 1. To accelerate the
calculation process required by Wasserstein distance, we adopt the popular
empirical Wasserstein distance solver $D(\cdot,\cdot)$ from Flamary et al.
(2021). For simplicity, we use $\tau$ to represent the state-action samples
from trajectory $\tau$ in the pseudocode. To better reveal the relationship
between diversity and learning potential, we conduct an ablation study where
we only adopt the diversity metric to pick levels to fill up the level buffer,
i.e. set $P_{replay}=1\cdot P_{D}+0\cdot P_{R}$.

Algorithm 1 DIPLR

Input: Level buffer size $N$, level generator $\mathcal{G}$
Initialize: student policy $\pi_{\eta}$, level buffer $L$, traj buffer
$\Gamma$

1: Generate $N$ initial levels by $\mathcal{G}$ to populate $L$

2: Collect trajectories on each replay level in $L$ and fill up $\Gamma$

3: while not converged do

4: Sample replay-decision, $\epsilon\sim U[0,1]$

5: if $\epsilon\geq 0.5$ then

6: Generate a new level $l_{\theta_{i}}$ by $\mathcal{G}$

7: Collect trajectories $\tau_{i}$ on $l_{\theta_{i}}$, with stop-gradient
$\eta_{\perp}$

8: Compute the regret, staleness and distance for $l_{\theta_{i}}$

9: else

10: Sample a replay level $l_{\theta_{j}}\in L$ according to $P_{replay}$

11: Collect trajectories $\tau_{j}$ on $l_{\theta_{j}}$ and update
$\pi_{\eta}$ with rewards $R(\tau_{j})$

12: Compute the regret, staleness and distance for $l_{j}$

13: end if

14: Flush $\Gamma$ and collect trajectories on all replay levels to fill up
$\Gamma$

15: Update regret, staleness, and distance for $l_{\theta_{i}}$ or
$l_{\theta_{j}}$

16: Update $L$ with new level $l_{\theta_{i}}$ if its replay probability is
greater than any levels in $L$

17: Update replay probability $P_{replay}$

18: end while

##  4 Experiments

In this section, we compare DIPLR to the set of leading benchmarks for solving
UED: Domain Randomization, Minimax, PAIRED, PLR ⟂. We conduct extensive
experiments and empirically demonstrate the effectiveness and generality of
DIPLR on three popular yet highly distinct UPOMDP domains, Minigrid, Bipedal-
Walker and Car-Racing. Minigrid is a partially-observable navigation problem
under discrete control with sparse rewards, while Bipedal-Walker and Car-
Racing are partially-observable walking/driving under continuous control with
dense rewards (we provide more details in Appendix). In each domain, we train
the teacher and student agents with Proximal Policy Optimization (PPO,
Schulman et al. (2017)) and we will present the zero-shot out-of-distribution
(OOD) test performance of all algorithms in each domain. Besides, to make the
comparison more reliable and straightforward, we adopt the recently introduced
standardized DRL evaluation metric Agarwal et al. (2021b), with which we show
the aggregate inter-quartile mean (IQM) and optimality gap plots.
Specifically, IQM discards the bottom and top 25% of the runs and measures the
performance on the middle 50% of combined runs, and it is thus robust to
outlier scores and is a better indicator of overall performance; Optimality
gap captures the amount by which the algorithm fails to meet the ”best
performance”, i.e. it assumes that a score (e.g. =1.0) is a desirable target
beyond which improvements are not very important. We present the plots after
normalizing the performance with a min-max range of solved-rate/returns for
better visualization.

We also provide a detailed ablation analysis to demonstrate the utility of
diversity alone, by providing results with and without regret (or GAE). The
version with diversity alone is referred to as DIPLR- and the one which uses
both diversity and regret is referred to as DIPLR.

Figure 2: Examples of training environments in Minigrid Domain Figure 3:
Aggregate zero-shot OOD test performance in Minigrid Domain across 10
independent runs. Higher IQM scores and lower optimality gap are better.
Figure 4: Zero-shot transfer performance in eight human-designed test
environments. The plots are based on the median and interquartile range of
solved rates across 10 independent experiments.

###  4.1 Minigrid Domain

Introduced by Dennis et al. (2020), Minigrid is a good benchmark due to its
easy interpretability and customizability. In our experimental settings, the
teacher has a maximum budget of 60 steps to place small block cells and
another two steps to place the start position and goal position at the end.
The student agent is required to navigate to the goal cell within 256 steps
and it receives rewards=(num-steps/256) upon success, otherwise, it receives
zero rewards. Figure 2 displays a few training examples from the domain.

We train all the student agents for 30k PPO updates ($\sim$250M steps) and
evaluate their transfer capabilities on eight zero-shot OOD environments (see
the first row in Figure 2) used in the literature and that are crafted by
humans. We summarize the empirical results of our proposed algorithm and all
other benchmarks in Figure 3 and Figure 4. Here are the key observations from
both the figures:

  * $\bullet$

Figure 3: DIPLR surpasses all the benchmark approaches, and surpasses the
leading best approach, PLR⊥ by more than 20% with regards to IQM and
Optimality Gap.

  * $\bullet$

Figure 3: DIPLR- performs on-par or better than PLR⊥.

  * $\bullet$

Figure 3: PAIRED performs poorly, as the long horizon (60 steps) and sparse
reward task is super challenging for a guided level generator. While Domain
Randomization performs decently, minimax performs worse than PAIRED.

  * $\bullet$

Figure 4: With respect to zero-shot transfer results, DIPLR is clearly better
than PLR⊥ in every one of the testing scenarios. This clearly indicates the
utility of diversity.

Figure 5: Six examples of test environments in Bipedal-Walker domain. (a)
BipedalWalker, (b) Hardcore, (c) Stair, (d) PitGap, (e) Stump, (f) Roughness.
Note that these may not be the exact test environment terrains but only
extreme cases for demonstration. Figure 6: Aggregate test performance over 10
runs in Bipedal-Walker domain. Figure 7: Performance on test environments
during the training period over five independent experiments (mean and
standard error).

###  4.2 Bipedal-Walker Domain

We also conduct a comprehensive comparison on the Bipedal-Walker domain, which
is introduced in Wang et al. (2019b); Parker-Holder et al. (2022) and is
popular in the community as it is a well-parameterized environment with eight
variables (including ground roughness, the number of stair steps, min/max
range of pit gap width, min/max range of stump height, and min/max range of
stair height) conditioning the terrains and is thus straightforward to
interpret. The teacher in Bipedal-Walker can specify the values of the eight
environment parameters, but there will still be stochasticity in the
generation of a particular level. As for the student, i.e. walker agent, it
should determine the torques applied on its joints and is constrained by
partial observability where it only knows its horizontal speed, vertical
speed, angular speed, positions of joints, joints’ angular speed, etc.

Following the experiment settings in prior UED works, we train all the
algorithms for 30k PPO updates ($\sim$1B steps), and then evaluate their
generalization capability on six distinct test instances in Bipedal-Walker
domain, i.e. BidedalWalker, Hardcore, Stair, PitGap, Stump, and Roughness, as
illustrated in Figure 5. Among these, $\left\\{BipedalWalker\right\\}$ is the
basic level that only evaluates whether the agent can walk on flat ground,
$\left\\{Stair,PitGap,Stump,Roughness\right\\}$ challenges the walker for a
particular kind of obstacles and $\left\\{Hardcore\right\\}$ contains a
combination of all types of challenging obstacles.

To understand the evolution of transfer performance, we evaluate the student
every 100 student policy updates and plot the curves in Figure 7. As is shown,
our proposed method DIPLR outperforms all other benchmarks in each test
environment. The ablation model DIPLR- also surpasses our primary benchmark
PLR⟂ in five test environments, indicating that diversity metric alone
contributes to ultimate generalization performance more than regret in
continuous domains. PAIRED suffers from variance quite a bit and cannot
constantly create informative levels for the student agents in such a complex
domain. This is because of the nature of the multi-agent framework adopted in
PAIRED, where the convergence highly depends on the ability of the expert
student.

After training for 30k PPO updates, we collect trained models and conduct a
more rigorous evaluation based on 10 test episodes in each test environment,
and present the aggregate results after min-max normalization (with range=[0,
300] on Bipedal-Walker and [0, 150] on the other five test environments) in
Figure 6. Notably, our method DIPLR dominates all the benchmarks in both IQM
and optimality gap. Furthermore, DIPLR- performs better than PLR⊥.

Figure 8: Examples in Car-Racing domain. (a) A zoom-in snapshot of the car-
racing track; (b) a track generated by domain randomization; (c) one of the
test environments, F1-Singapore track  Figure 9: Zero-Shot Transfer
Performance in Car-Racing domain Figure 10: Aggregate test performance over 10
independent runs in Car-Racing domain.

###  4.3 Car-Racing Domain

To validate that our method is scalable and versatile, we further implement
DIPLR on the Car-Racing environment, which is introduced by Jiang et al.
(2021a) and has been used in existing UED papers. In this domain, the teacher
needs to create challenging racing tracks by using Bézier curves (via 12
control points) and the student drives on the track under continuous control
with dense reward. A zoom-in illustration of the track is shown in Figure 8
(a). After training the student for around 3k PPO updates ($\sim$ 5.5M steps),
we evaluate the student agent in four test scenarios, among which three are
challenging F1-tracks existing in the real world and the other one is a basic
vanilla track. Note that these tracks are absolutely OOD because they cannot
be defined with just 12 control points, for example, the F1-Singapore track in
Figure 8 (c) has more than 20 control points.

Figure 9 presents the complete training process of each algorithm on four test
environments, with an evaluation interval of 100 PPO updates. As a relatively
simpler domain, our method DIPLR and DIPLR- quickly converge to the optimum
while DR and PLR⟂ achieve a near-optimal generalization on the Vanilla and
F1-Italy track.

The aggregate performance after min-max normalization (with range=[200,800])
of all methods is summarized in Figure 10. Despite both PLR⟂ and DR agents can
learn well on this comparatively simple domain, DIPLR outperforms them on both
IQM and optimality gap.

##  5 Conclusion

We provided a novel method DIPLR for unsupervised environment design (UED),
where diversity in training environments is employed to improve generalization
capability of a student agent. By computing the distance between levels via
Wasserstein distance over occupancy distributions, we constantly expose the
agent to challenging yet diverse scenarios and thus improve its generalization
in zero-shot out-of-distribution test environments. In our experiments, we
validated that DIPLR is capable of training robust and generalizing agents and
significantly outperforms the best-performing baselines in three distinct UED
domains. Moreover, we explored the relationship between diveristy and learning
potential, and we discover that diversity alone benefits the algorithm more
than learning potential, and they complement each other to achieve the best
performance when combined.

## References

  * Agarwal et al. [2021a] Rishabh Agarwal, Marlos C Machado, Pablo Samuel Castro, and Marc G Bellemare.  Contrastive behavioral similarity embeddings for generalization in reinforcement learning.  arXiv preprint arXiv:2101.05265, 2021.
  * Agarwal et al. [2021b] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare.  Deep reinforcement learning at the edge of the statistical precipice.  Advances in neural information processing systems, 34:29304–29320, 2021.
  * Akkaya et al. [2019] Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al.  Solving rubik’s cube with a robot hand.  arXiv preprint arXiv:1910.07113, 2019.
  * Cobbe et al. [2019] Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman.  Quantifying generalization in reinforcement learning.  In International Conference on Machine Learning, pages 1282–1289. PMLR, 2019.
  * Cobbe et al. [2020] Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman.  Leveraging procedural generation to benchmark reinforcement learning.  In International conference on machine learning, pages 2048–2056. PMLR, 2020.
  * Dennis et al. [2020] Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine.  Emergent complexity and zero-shot transfer via unsupervised environment design.  Advances in neural information processing systems, 33:13049–13061, 2020.
  * Flamary et al. [2021] Remi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z Alaya, Aurelie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, et al.  Pot: Python optimal transport.  J. Mach. Learn. Res., 22(78):1–8, 2021.
  * Hong et al. [2018] Zhang-Wei Hong, Tzu-Yun Shann, Shih-Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, and Chun-Yi Lee.  Diversity-driven exploration strategy for deep reinforcement learning.  Advances in neural information processing systems, 31, 2018.
  * Jakobi [1997] Nick Jakobi.  Evolutionary robotics and the radical envelope-of-noise hypothesis.  Adaptive behavior, 6(2):325–368, 1997.
  * Jiang et al. [2021a] Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel.  Replay-guided adversarial environment design.  Advances in Neural Information Processing Systems, 34:1884–1897, 2021.
  * Jiang et al. [2021b] Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel.  Prioritized level replay.  In International Conference on Machine Learning, pages 4940–4950. PMLR, 2021.
  * Kantorovich [1942] LV Kantorovich.  Kantorovich: On the transfer of masses.  In Dokl. Akad. Nauk. SSSR, 1942.
  * Khirodkar et al. [2018] Rawal Khirodkar, Donghyun Yoo, and Kris M Kitani.  Vadra: Visual adversarial domain randomization and augmentation.  2018\.
  * Lupu et al. [2021] Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster.  Trajectory diversity for zero-shot coordination.  In International Conference on Machine Learning, pages 7204–7213. PMLR, 2021.
  * Mehta et al. [2020] Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J Pal, and Liam Paull.  Active domain randomization.  In Conference on Robot Learning, pages 1162–1176. PMLR, 2020.
  * Mirhoseini et al. [2021] Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Wenjie Jiang, Ebrahim Songhori, Shen Wang, Young-Joon Lee, Eric Johnson, Omkar Pathak, Azade Nazi, et al.  A graph placement methodology for fast chip design.  Nature, 594(7862):207–212, 2021.
  * Mnih et al. [2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  nature, 518(7540):529–533, 2015.
  * Monge [1781] Gaspard Monge.  Mémoire sur la théorie des déblais et des remblais.  Mem. Math. Phys. Acad. Royale Sci., pages 666–704, 1781.
  * Morimoto and Doya [2005] Jun Morimoto and Kenji Doya.  Robust reinforcement learning.  Neural computation, 17(2):335–359, 2005.
  * Parker-Holder et al. [2020] Jack Parker-Holder, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts.  Effective diversity in population based reinforcement learning.  Advances in Neural Information Processing Systems, 33:18050–18062, 2020.
  * Parker-Holder et al. [2022] Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel.  Evolving curricula with regret-based environment design.  arXiv preprint arXiv:2203.01302, 2022.
  * Pinto et al. [2017] Lerrel Pinto, James Davidson, and Abhinav Gupta.  Supervision via competition: Robot adversaries for learning tasks.  In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 1601–1608. IEEE, 2017.
  * Portelas et al. [2020] Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yves Oudeyer.  Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments.  In Conference on Robot Learning, pages 835–853. PMLR, 2020.
  * Sadeghi and Levine [2016] Fereshteh Sadeghi and Sergey Levine.  Cad2rl: Real single-image flight without a single real image.  arXiv preprint arXiv:1611.04201, 2016.
  * Schulman et al. [2015] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel.  High-dimensional continuous control using generalized advantage estimation.  arXiv preprint arXiv:1506.02438, 2015.
  * Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * Silver et al. [2016] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  nature, 529(7587):484–489, 2016.
  * Song et al. [2016] Jinhua Song, Yang Gao, Hao Wang, and Bo An.  Measuring the distance between finite markov decision processes.  In Proceedings of the 2016 international conference on autonomous agents & multiagent systems, pages 468–476, 2016.
  * Tobin et al. [2017] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel.  Domain randomization for transferring deep neural networks from simulation to the real world.  In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23–30. IEEE, 2017.
  * Wang et al. [2019a] Hao Wang, Shaokang Dong, and Ling Shao.  Measuring structural similarities in finite mdps.  In IJCAI, pages 3684–3690, 2019.
  * Wang et al. [2019b] Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley.  Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions.  arXiv preprint arXiv:1901.01753, 2019.
  * Zhang et al. [2018] Amy Zhang, Nicolas Ballas, and Joelle Pineau.  A dissection of overfitting and generalization in continuous reinforcement learning.  arXiv preprint arXiv:1806.07937, 2018.


# Crystallographic defects in Weyl semimetal LaAlGe

Inseo Kim1 Byungkyun Kang2 Hyunsoo Kim3 Minseok Choi1<EMAIL_ADDRESS>1Department of Physics, Inha University, Incheon 22212, Korea  2College of
Arts and Sciences, University of Delaware, Newark, Delaware 19716, United
States  3Department of Physics, Missouri University of Science and Technology,
Rolla, 65409, MO, United States

###### Abstract

Crystallographic defects in a topological semimetal can result in charge
doping, and the scattering due to the defects may mask its exotic transport
properties. Here, we investigate the possible crystallographic defects
including vacancy and antisite in Weyl semimetal LaAlGe using hybrid-density-
functional theory calculations. We show that a considerable concentration of
Al- and Ge-related defects naturally form during growth due to their low
formation enthalpy. Specifically, Al can be easily replaced by Ge in the
$I4_{1}md$ phase of LaAlGe, forming the Ge-on-Al antisite, GeAl. The
counterpart, Al-on-Ge (AlGe), is also probable. The most abundant defect GeAl
is donor-like, effectively electron-doping, and these defects are therefore
not only scattering centers in the electronic transport process but may also
induce the substantial vertical shift of the chemical potential. The results
imply that the naturally occurring defects hinder both spectroscopic and
transport features arising from the Weyl physics in LaAlGe. Our work can be
applied to the $R$AlGe family ($R$=rare earth) and help improve the quality of
single-crystal magnetic Weyl semimetal.

††preprint: MC

##  I Introduction

Solid states with relativistic low-energy excitations arising from the linear
band crossing are promising platforms for next-generation device technology
Yang (2016); Wang and Sun (2023). The first kind is the Dirac fermion emerging
when time-reversal and parity symmetries are preserved Armitage et al. (2018),
and the two-dimensional Dirac fermion is realized at the boundary of a
topological insulator Hasan and Kane (2010) and in graphene Geim and Novoselov
(2007). When, at least, one of the aforementioned symmetries is broken, and
the spin-degeneracy is lifted, the Weyl fermion emerges Xu et al. (2015). When
the band intersection is sufficiently close to the chemical potential, the
Weyl semimetal (WSM) exhibits unique properties, including the Veselago effect
Hills et al. (2017) and selective interfacial transmission Yesilyurt et al.
(2016); Hou et al. (2019), that allow unprecedented applications Guo et al.
(2023).

In the known WSMs, the position of the linear crossings, the Weyl nodes, is as
close as a few meV to the chemical potential Xu et al. (2015); Lv et al.
(2015). In the three-dimensional system, the bulk Weyl nodes exist as a pair
with opposite chirality, and they have topologically distinct characters with
opposite Chern numbers Armitage et al. (2018). The Weyl nodes are responsible
for negative magnetoresistance due to chiral anomaly when the magnetic field
is applied in the direction of the electrical current Ong and Liang (2021).
While the Weyl nodes are separated in bulk, they are connected through the
one-dimensional surface state, the Fermi arc Xu et al. (2015). Experimental
observations of chiral anomaly and the Fermi arc are regarded as direct
evidence of the WSM phase.

In LaAlGe, one of the WSM family $Ln$Al$X$ ($Ln$ = lanthanide, $X$ = Ge, Si),
the angle-resolved photoemission spectroscopy (ARPES) measurement successfully
resolved the Fermi arc electron states on the surface Xu et al. (2017a).
However, neither anomalous Hall effect nor chiral anomaly was observed to date
Hodovanets et al. (2018); Hu et al. (2020a). Quantum oscillations were not
observed in a typical laboratory field range ($<$10 T) while the other members
of the family ($Ln$ = Ce, Nd, and Pr) exhibit quantum oscillations at much
lower fields, which is counterintuitive considering the presence of excess
inelastic scattering in $Ln$Al$X$ due to localized moments. The normal state
transport property in LaAlGe is consistent with conventional metal, but the
residual resistivity is considerably high, which is comparable with that of a
magnetic counterpart CeAlGe Hodovanets et al. (2018). These experimental
observations can be attributed to the relatively high content of native
defects in LaAlGe.

Moreover, native crystallographic defects can be a charge dopant Yu et al.
(2017), and a significant amount of such defects can shift the chemical
potential. In topological semimetals, this issue may result in a significant
discrepancy between the bulk and surface chemical potentials Kim et al. (2018,
2022). In the WSMs, the energy level of Weyl nodes relative to the chemical
potential is crucial for Weyl physics, and it can be irrelevant if the nodes
are far away from the chemical potential.

The LaAlGe single crystals are typically grown out of Al self-flux because of
its low melting point Hodovanets et al. (2018); Hu et al. (2020a). The
volatile nature of Al may play a role and result in various crystallographic
defects such as Al vacancy and subsequent interstitial or antisite associated
with La and Ge that may occur during the crystal growth. Therefore, it is
crucial to understand the defect energetics at the growth level. Our aim here
is to theoretically identify the potential defects that affect crystal quality
and the physics of Weyl fermion in LaAlGe.

In the present work, we performed density functional theory calculation with a
hybrid functional, an approach that has been demonstrated to result in
accurate band structures and enthalpy of formation and has provided a reliable
description of defect formation energies and defect levels in materials
Alkauskas et al. (2012, 2016). Details of the calculations are provided in
Sec. II. The electronic and structural properties of LaAlGe are described in
Sec. III A, and electronic and structural properties of native defects are
addressed in Sec. III B and C. The impact of potential defects in the Weyl
physics of LaAlGe is discussed in Sec. III D.

##  II Computational approach

The calculations were based on density functional theory and the screened
hybrid functional of Heyd-Scuseria-Ernzerhof (HSE) Heyd et al. (2003); Krukau
et al. (2006), implemented with the projector augmented-wave method Blöchl
(1994) in the vasp code Kresse and Hafner (1993). The mixing parameter for the
nonlocal Fock-exchange was set to 32%. The tetragonal unitcell containing 12
atoms was considered for intrinsic property. The wavefunctions were expanded
in a plane-wave basis set with an energy cutoff of 360 eV, and integrations
over the Brillouin zone were carried out using $6\times 6\times 6$ $k$-mesh.
Atomic positions were relaxed until the Hellmann-Feynman forces were reduced
to less than 0.005 eV/Å. The effect of spin-orbit coupling was also considered
to look at the electronic structure of LaAlGe (See Fig. 1 in the supplementary
material sup ). However, it does not alter the defect properties (e.g. Sec.
III D), so we will mostly show results obtained without the spin-orbit
coupling here.

The validity of the HSE electronic structure of pristine LaAlGe was checked
using the $ab$-$initio$ linearized quasi-particle self-consistent GW (LQSGW)
method combined with dynamical mean field theory (DMFT) Tomczak (2015); Choi
et al. (2016, 2019), implemented in the ComDMFT code Choi et al. (2019) for
DMFT and the FlapwMBPT code Kutepov et al. (2017) for LQSGW part. The
electronic structure of the system was evaluated using LQSGW calculations
Kutepov et al. (2012, 2017), followed by diagrammatic DMFT corrections to the
local part of the GW self-energy Georges et al. (1996); Metzner and Vollhardt
(1989); Georges and Kotliar (1992). An explicit calculation of the double-
counting energy and Coulomb interaction tensor was performed. Local self-
energies for La 4$f$ and La 5$d$ were obtained by solving two different single
impurity models, with spin-orbital coupling included in all calculations. The
electronic temperature of the system was set to 300 K.

For defect simulation, the La vacancy ($V_{\rm La}$), the Al vacancy ($V_{\rm
Al}$), the Ge vacancy ($V_{\rm Ge}$), the La antisites (LaAl and LaGe), and
the Al antisites (AlLa and AlGe), and the Ge antisites (GeLa and GeAl) were
considered. A $3\times 3\times 1$ supercell containing 108 atoms was employed.
The wavefunctions were expanded in a plane-wave basis set with an energy
cutoff of 360 eV, and integrations over the Brillouin zone were carried out
using a $2\times 2\times 2$ $k$-mesh. Atomic positions were relaxed until the
Hellmann-Feynman forces were reduced to less than 0.02 eV/Å.

The formation energy of a defect $D$ in charge state $q$ is defined as Van de
Walle and Neugebauer (2004):

$\displaystyle E^{f}(D^{q})$ $\displaystyle=$ $\displaystyle E_{\rm
tot}(D^{q})-E_{\rm tot}({\rm LaAlGe})-\sum_{i}n_{i}\mu_{i}$ (1)

where $E_{\rm tot}(D^{q})$ is the total energy of a supercell containing a
defect $D$ in charge state $q$, and $E_{\rm tot}({\rm LaAlGe})$ is the total
energy of perfect LaAlGe supercell. $n_{i}$ is the number of atoms of type $i$
added to ($n_{i}$$>$0) and/or removed from ($n_{i}$$<$0) the perfect crystal
to form the defect, and $\mu_{i}$ ($i$ = La, Al, and Ge) are the atomic
chemical potentials consisting of $\mu_{i}^{0}+\Delta\mu_{i}$. The defect
formation energy depends on the atomic chemical potential $\mu_{i}$, which is
taken with respect to the total energy per atom of the standard phase of the
species $i$. I.e., $\mu_{\rm La}^{0}$ is referenced to the total energy per
atom of La bulk, $\mu_{\rm Al}^{0}$, that of Al bulk, and $\mu_{\rm Ge}^{0}$
is referenced to the total energy per atom of Ge bulk.

Table 1:  Calculated and experimental formation enthalpies of the reference materials for atomic chemical potentials.  Materials | Present work (eV) | Experiment (eV)
---|---|---
LaAl | –1.01 | –0.96 111Reference Cordfunke and Konings, 2001
LaAl2 | –1.76 | –1.58 111Reference Cordfunke and Konings, 2001
LaAl3 | –2.13 | –1.83 111Reference Cordfunke and Konings, 2001
La3Al11 | –6.51 | –5.97 111Reference Cordfunke and Konings, 2001
LaGe | –1.97 | –1.92 222Reference NIS,
La5Ge3 | –6.82 | –6.96 222Reference NIS,
LaAl2Ge2 | –2.81 | N/A

Figure 1:  Allowed values of the atomic chemical potentials defining the
stability of LaAlGe. The chemical potentials $\mu_{\mathrm{La}}$,
$\mu_{\mathrm{Al}}$, and $\mu_{\mathrm{Ge}}$ are limited by the formation of
the secondary phases as described in the text and Table. The filed region
indicates the conditions that enable to growth LaAlGe.

In this formalism the atomic chemical potentials $\mu_{i}$ are variable, yet
restricted by the formation of limiting phases containing La, Al, and Ge. The
atomic chemical potentials $\mu_{i}$ must satisfy the stability condition of
LaAlGe:

$\ \mu_{\rm La}+\mu_{\rm Al}+\mu_{\rm Ge}=\Delta H_{f}{\rm(LaAlGe)},$ (2)

with $\Delta\mu_{\rm La}$ $\leq$ 0, $\Delta\mu_{\rm Al}$ $\leq$ 0, and
$\Delta\mu_{\rm Ge}$ $\leq$ 0\. $\Delta H_{f}{\rm(LaAlGe)}$ is the formation
enthalpy of perfect LaAlGe crystal. The chemical potentials $\mu_{\rm La}$,
$\mu_{\rm Al}$, and $\mu_{\rm Ge}$ are further constrained by the formation of
the secondary phases, such as LaAl, LaAl2, LaAl3, La3Al11, LaGe, La5Ge3, and
La(AlGe)2 phases:

$\displaystyle\ \Delta\mu_{\rm La}+\Delta\mu_{\rm Al}<\Delta
H_{f}{\rm(LaAl)},$ (3) $\displaystyle\ \Delta\mu_{\rm La}+2\Delta\mu_{\rm
Al}<\Delta H_{f}{\rm(LaAl_{2})},$ (4) $\displaystyle\ \Delta\mu_{\rm
La}+3\Delta\mu_{\rm Al}<\Delta H_{f}{\rm(LaAl_{3})},$ (5) $\displaystyle\
\Delta\mu_{\rm La}+11\Delta\mu_{\rm Al}<\Delta H_{f}{\rm(LaAl_{11})},$ (6)
$\displaystyle\ \Delta\mu_{\rm La}+\Delta\mu_{\rm Ge}<\Delta
H_{f}{\rm(LaGe)},$ (7) $\displaystyle\ 5\Delta\mu_{\rm La}+3\Delta\mu_{\rm
Ge}<\Delta H_{f}{\rm(La_{5}Ge_{3})},$ (8) $\displaystyle\ \Delta\mu_{\rm
La}+2\Delta\mu_{\rm Al}+2\Delta\mu_{\rm Ge}<\Delta H_{f}{\rm(La(AlGe)_{2})},$
(9)

The calculated formation enthalpies of the phases are listed in Table 1. We
used the equations to define a region in which LaAlGe is thermodynamically
stable (See the filled regions in Fig. 1).

Table 2:  Lattice constants, interatomic distances, and formation enthalpy of LaAlGe obtained using the HSE and the generalized-gradient-approximation (GGA) functional of Perdew, Burke, and Ernzerhof (PBE) Perdew et al. (1996). The value in parenthesis is a number of the nearest neighboring bonds (distance). Experimental values are also shown for comparison.  Property | GGA | HSE | Experiment
---|---|---|---
$a$ (Å) | 4.380 | 4.348 |  4.336111Reference Guloy and Corbett, 1991 4.344222Reference Xu et al., 2017b 4.349333Reference Cheng and Navrotsky,2003
$c$ (Å) | 14.876 | 14.736 |  14.828111Reference Guloy and Corbett, 1991 14.812222Reference Xu et al., 2017b 14.829333Reference Cao et al., 2022
$d_{\rm La-Ge}$ (Å) |  3.303 ($\times$2) 3.339 ($\times$4) |  3.282 ($\times$2) 3.309 ($\times$4) |  3.280 ($\times$2)111Reference Guloy and Corbett, 1991 3.310 ($\times$4)111Reference Guloy and Corbett, 1991
$d_{\rm La-Al}$ (Å) |  3.310 ($\times$2) 3.335 ($\times$4) |  3.279 ($\times$2) 3.311 ($\times$4) |  3.300 ($\times$6)111Reference Guloy and Corbett, 1991
$d_{\rm Al-Ge}$ (Å) |  2.484 2.515 ($\times$2) |  2.454 2.498 ($\times$2) |  2.498111Reference Guloy and Corbett, 1991 2.470 ($\times$2)111Reference Guloy and Corbett, 1991
$\Delta H^{f}$ (eV/f.u.) | –2.138 | –2.562 | N/A

##  III Results and discussion

###  III.1 Properties of pristine LaAlGe

Prior to investigating native defects in LaAlGe, the structural and electronic
properties of pristine LaAlGe were examined. Figure 2 shows the conventional
unit cell of LaAlGe and its band structure and density of states (DOS)
obtained using the HSE functional. The calculated lattice parameters and
formation enthalpy of LaAlGe agree well with experimental values (Table
2)Howard et al. (2000). The electronic structure shows a semi-metallic feature
which is consistent with the experiment. LaAlGe has a low DOS near the Fermi
level with characteristics of La 5$d$. This indicates that as electrons are
added or removed, the position of the Fermi level readily moves up or down.

Figure 2:  (a) Crystal structure, (b) density of states, and (c) band
structure of LaAlGe obtained using the HSE functional without the spin-orbit
coupling.

In order to validity the electronic structure obtained using the HSE
functional, LQSGW+DMFT calculations were performed to obtain the DOS of
pristine LaAlGe which has been successfully utilized to address the electronic
structure of correlated material systems, including actinide-and lanthanide-
based compounds Siddiquee et al. (2023); Kang et al. (2023a, 2022, b, c); Kang
and Choi (2019). We find that overall features agree with the HSE results. An
important observation is the position of the main peak of La 4$f$ state near 5
eV above the Fermi level, which agrees with the HSE but differs from the PBE-
GGA result (See Fig. 2 in the supplementary material sup ).

Figure 3:  Local atomic structures of the lattice vacancies ((a) $V_{\rm Al}$,
(b) $V_{\rm La}$, and (c) $V_{\rm Ge}$) and three antisite defects ((d) GeAl,
(e) AlGe, and (f) GeLa), which are mostly likely to form in LaAlGe Momma and
Izumi (2011). Black sphere indicates the position of the vacant site, and the
interatomic distances near the defects are shown.

###  III.2 Vacancies

Figure 3 shows the obtained local vacancy structures. $V_{\rm Al}$ leads to
significant inward relaxation of the surrounding three Ge atoms, with $V_{\rm
Al}$–Ge distances of 2.421 Å for $\rm Ge_{1}$ and $\rm Ge_{2}$, and 2.334 Å
for $\rm Ge_{3}$; these are –3.08 % and –4.89 % smaller than the Al–Ge bond
lengths in pristine LaAlGe, respectively (Table 2). Similarly, $V_{\rm Al}$–La
distances are 3.269 Å for $\rm La_{1}$ to $\rm La_{4}$, and 3.213 Å for $\rm
La_{5}$ and $\rm La_{6}$, which are –1.27 % and –2.01 % smaller than the Al–La
bond lengths in pristine LaAlGe. The calculated formation energies of all the
native point defects in ternary LaAlGe for points indicated in the diagrams of
Fig. 1 were evaluated (Table 3). Al vacancies have the lowest formation
energies among the vacancies, even for Al-rich conditions (e.g., points D and
E in Fig. 1 (c)). However, their formation energies for all the conditions
considered here are higher than 4.5 eV, hence the formation of isolated
vacancies during growth may be not likely.

Table 3:  Calculated formation energy of the vacancies and antisites in LaAlGe for points indicated in the diagrams of Fig. 1. Chemical potentials of $\Delta\mu_{\rm La}$ vs $\Delta\mu_{\rm Al}$, $\Delta\mu_{\rm La}$ vs $\Delta\mu_{\rm Al}$, $\Delta\mu_{\rm La}$ vs $\Delta\mu_{\rm Al}$ correspond to Figs. 1(a), (b), and (c), respectively.  Chemical potential | Vacancies | Antisites
---|---|---
$\Delta\mu_{\rm La}$ vs $\Delta\mu_{\rm Al}$ | $V_{\rm La}$ | $V_{\rm Al}$ | $V_{\rm Ge}$ | LaAl | LaGe | AlLa | AlGe | GeLa | GeAl
A | 7.40 | 4.53 | 8.12 | 3.63 | 8.43 | 3.61 | 2.64 | 1.69 | –1.38
B | 8.80 | 4.53 | 6.73 | 2.23 | 5.64 | 5.01 | 1.24 | 4.48 | 0.02
C | 8.35 | 4.75 | 6.95 | 2.90 | 6.31 | 4.33 | 1.24 | 3.81 | 0.02
D | 7.43 | 5.06 | 7.56 | 4.13 | 7.84 | 3.10 | 1.55 | 2.27 | –0.29
E | 7.20 | 5.12 | 7.73 | 4.42 | 8.24 | 2.82 | 1.65 | 1.88 | –0.39
F | 7.05 | 5.12 | 7.88 | 4.57 | 8.53 | 2.67 | 1.80 | 1.58 | –0.54
G | 7.05 | 4.88 | 8.12 | 4.33 | 8.78 | 2.91 | 2.29 | 1.34 | –1.03
$\Delta\mu_{\rm La}$ vs $\Delta\mu_{\rm Ge}$ | $V_{\rm La}$ | $V_{\rm Al}$ | $V_{\rm Ge}$ | LaAl | LaGe | AlLa | AlGe | GeLa | GeAl
A | 7.20 | 5.12 | 7.73 | 4.42 | 8.24 | 2.82 | 1.65 | 1.88 | –0.39
B | 7.43 | 5.06 | 7.56 | 4.13 | 7.84 | 3.10 | 1.55 | 2.27 | –0.29
C | 8.35 | 4.75 | 6.95 | 2.90 | 6.31 | 4.33 | 1.24 | 3.81 | 0.02
D | 8.80 | 4.53 | 6.73 | 2.23 | 5.64 | 5.01 | 1.24 | 4.48 | 0.02
E | 7.40 | 4.53 | 8.12 | 3.63 | 8.43 | 3.61 | 2.64 | 1.69 | –1.38
F | 7.05 | 4.88 | 8.12 | 4.33 | 8.78 | 2.91 | 2.29 | 1.34 | –1.03
G | 7.05 | 5.12 | 7.88 | 4.57 | 8.53 | 2.67 | 1.80 | 1.58 | –0.54
$\Delta\mu_{\rm Al}$ vs $\Delta\mu_{\rm Ge}$ | $V_{\rm La}$ | $V_{\rm Al}$ | $V_{\rm Ge}$ | LaAl | LaGe | AlLa | AlGe | GeLa | GeAl
A | 8.94 | 4.55 | 6.57 | 2.10 | 5.34 | 5.13 | 1.07 | 4.78 | 0.19
B | 8.36 | 5.12 | 6.57 | 3.26 | 5.91 | 3.98 | 0.49 | 4.21 | 0.77
C | 7.05 | 5.12 | 7.88 | 4.57 | 8.53 | 2.67 | 1.80 | 1.58 | –0.54
D | 7.05 | 4.88 | 8.12 | 4.33 | 8.78 | 2.91 | 2.29 | 1.34 | –1.03
E | 7.40 | 4.53 | 8.12 | 3.63 | 8.43 | 3.61 | 2.64 | 1.69 | –1.38
F | 8.91 | 4.53 | 6.61 | 2.11 | 5.40 | 5.12 | 1.12 | 4.72 | 0.14

$V_{\rm La}$ leads to insignificant outward relaxation of the surrounding six
Ge atoms, with $V_{\rm La}$–Ge distances of 3.353 Å for $\rm Ge_{1}$, $\rm
Ge_{2}$, $\rm Ge_{3}$ and $\rm Ge_{4}$, and 3.343 Å for $\rm Ge_{5}$ and $\rm
Ge_{6}$; these are 1.33 % and 1.86 % larger than the La–Ge bond lengths in
pristine LaAlGe, respectively. However, the surrounding six Al atoms are
significantly inward relaxed. The $V_{\rm La}$–Al distances are 2.860 Å
(–13.62 %) for $\rm Al_{1}$ to $\rm Al_{4}$ and 2.882 Å (–12.11 %) for $\rm
Al_{1}$ and $\rm Al_{2}$.

Such significant inward relaxations are also observed for $V_{\rm Ge}$. The
$V_{\rm Ge}$–Al distances of 2.273 Å for $\rm Al_{1}$ and $\rm Al_{2}$, and
2.181 Å for $\rm Al_{3}$ correspond to –9.01 % and –11.12 % shortening
compared with the Ge–Al bond lengths in pristine LaAlGe, respectively. On the
other hand, the distances with the surrounding six La atoms are relatively
robust. The change in the Ge-La distances are 1.09 % for $\rm La_{1}$, $\rm
La_{2}$, $\rm La_{3}$ and $\rm La_{4}$, and –0.61 % for $\rm La_{5}$ and $\rm
La_{6}$.

In terms of energetics, the two vacancies $V_{\rm La}$ and $V_{\rm Ge}$ are
not a concern in LaAlGe because of their quite high formation energies. The
lowest formation energies are 7.05 eV for $V_{\rm La}$ and 6.57 eV for $V_{\rm
Ge}$, which indicates that their concentrations would be negligible.

###  III.3 Antisites

Similar to the case of vacancies, one specific defect, the GeAl antisite (Fig.
3) has the lowest formation energy among the six types of antisites in LaAlGe,
regardless of growth conditions (Table 3). The defect formation energies are
very low for most growth conditions (even negative for several conditions),
which indicates that the concentration of GeAl antisites should be quite high.
Structurally, the ${\rm Ge_{Al}}$–Ge distances are 2.558 Å for $\rm Ge_{1}$
and $\rm Ge_{2}$, and 2.483 Å for $\rm Ge_{3}$, which are 2.40 % and 1.18 %
larger than the Al–Ge bond lengths in pristine LaAlGe, respectively. The
surrounding six La atoms are inward relaxed. The ${\rm Ge_{Al}}$–La distances
are 3.284 Å (–0.82 %) for $\rm La_{1}$, $\rm La_{2}$, $\rm La_{3}$ and $\rm
La_{4}$, and 3.231 Å (–1.46 %) for $\rm La_{5}$ and $\rm La_{6}$.

The counterpart antisite of the GeAl antisite, the AlGe antisite also has
relatively low formation energies with the mean value of 1.66 eV, and its
formation energy becomes 0.49 eV under the Al-rich and Ge-poor condition
(point B in Fig 1 (c)). Even though GeAl has the lowest formation energy in
most conditions, the formation energy of GeAl is higher than that of AlGe at
the point B (Al rich, Ge poor condition). The interatomic distances between
AlGe and the nearest Al atoms increase a little, with the ${\rm Al_{Ge}}$–Al
distances of 2.502 Å (0.16 %) for $\rm Al_{1}$ and $\rm Al_{2}$, and 2.470 Å
(0.65 %) for $\rm Al_{3}$. The AlGe–La distances also slightly increases, with
the ${\rm Al_{Ge}}$–La distances of 3.324 Å (0.45 %) for $\rm La_{1}$ to $\rm
La_{4}$ and 3.297 Å (0.46 %) for $\rm La_{5}$ and $\rm La_{6}$.

Our results indicate that the formation of GeLa antisite cannot be excluded
when the LaAlGe samples are grown at La-poor or Ge-rich conditions since the
defect formation energies become lower than 2 eV. However, GeAl would be
dominant even for the growth conditions ($E^{f}({\rm Ge_{Al}})<0$).

On the other hand, the other antisites may be unlikely to form because of
their very high formation energies. Especially, the formation energies of LaGe
are in the range of 5 to 9 eV, which is huge. Presumably, these defects may be
able to form in LaAlGe only if growth techniques operated far from thermal
equilibrium (e.g., ion implantation) are used to obtain the specimens or to
introduce the defects.

###  III.4 Impact of defects on the properties of LaAlGe

Figure 4:  Calculated relative position of the Fermi level as a function of
carrier concentration in LaAlGe. The Fermi level in the pristine LaAlGe is
used as reference. The blue and orange regions denote LaAlGe doped by removing
and adding electron, respectively. The results obtained using GGA-PBE with and
without the spin-orbit coupling effect are shown for comparison.

Finally, we discussed the impact of native defects on the properties of
LaAlGe. As mentioned in the Introduction section, growing LaAlGe single
crystal is typically grown out of Al flux because of its low melting point due
to the volatile nature of Al. The present study supports the requirement of an
Al-rich environment to obtain high-quality LaAlGe crystal, which will affect
the physics of Weyl fermion in LaAlGe. Our results show that GeAl is dominant
defect and its formation cannot be unavoidable but the growth under Al-rich
condition increases its formation energy which could relatively hinder the
GeAl formation (e.g., point B in Fig. 1 (c)). This may help to obtain a better
crystal quality. In addition, the counterpart defect, AlGe could help to get
it. The formation energy of AlGe decreases under Al-rich condition, hence the
defect becomes more likely to form then partly compensate electron carriers
released from GeAl.

When the two defects form with moderate concentrations, there may exist an
interaction between them. Thus, the binding energy of a complex defect,
GeAl–AlGe, which is consist of GeAl and AlGe antisites, was evaluated using
$E_{\rm bind}[({\rm Ge_{Al}}-{\rm Al_{Ge}})]=E^{f}({\rm Ge_{Al}})+E^{f}({\rm
Al_{Ge}})-E^{f}[({\rm Ge_{Al}}-{\rm Al_{Ge}})]$. The computed value is
positive but low ($E_{\rm bind}[({\rm Ge_{Al}}-{\rm Al_{Ge}}$) = 0.52 eV).
Therefore, the complex would occur during cool down because complex formation
becomes energetically advantageous but can relatively easily dissociate. Note
that the formation energies do not depend on the atomic chemical potentials,
hence the values are constant regardless of growth conditions considered here.

Moving to electronic property, LaAlGe may have a large concentration of
electron carriers, since the formation of GeAl during growth is very likely at
high temperatures. GeAl can release one electron in the sample when the Ge4+
and Al3+ oxidation states assumed. Indeed, the measured electron
concentrations in an experiment are 1.08 $\times$ 1021 cm-3 at 10 K and 1.06
$\times$ 1021 cm-3 at 20 K with growth temperature $T$ = 700 ∘C Hu et al.
(2020b). Using the equation for the GeAl concentration ($c=N_{\rm
sites}~{}exp(-E^{f}/k_{B}T)$, where $N_{\rm sites}$ is the density of sites Al
sites, $E^{f}({\rm Ge_{Al}})$ corresponds to 2.2 meV and 4.5 meV, which are
close to the calculated values. Such electron concentrations in LaAlGe thin
films and single crystals place the Fermi level at significantly higher
positions compared to the expected value in the pristine materials, as shown
in Fig. 4. Our results indicate that the chemical potential can be pushed away
from the Fermi level position due to the intrinsic electron doping by native
defect, hence causing the unexpected experimental observation, different from
the other materials of the $Ln$AlX family.

These results suggest a strategy for tuning the electronic chemical potential
in these materials. Since GeAl is the main concern, increasing the counterpart
AlGe is the most straightforward to put the electronic chemical potential
position near the Fermi level in perfect LaAlGe crystal, by controlling the
growth environment close to the condition providing the formation energies of
the two antisites similar such as point B in Fig. 1 (c) ($\Delta\mu_{\rm Al}$
= 0 eV, $\Delta\mu_{\rm Ge}$ = –1.55 eV, and $\Delta\mu_{\rm La}$ = –1.01 eV).
At the growth condition, GeAl and AlGe are most likely to form with similar
formation energies, and therefore, the most electron carriers would be
compensated by holes towards the perfect LaAlGe, bringing back the Weyl
physics.

##  IV Summary

We have investigated native defects in the Weyl semimetal LaAlGe using hybrid
density functional calculations by analyzing the defect formation energies. We
find that vacancies are not likely to form during growth, whereas Ge- and Al-
related antisite defects are likely to form. This indicates that the antisite
defects can play a role in the growing high-quality crystal, which will affect
the Weyl physics of LaAlGe by shifting the position of the electronic chemical
potential. Especially, GeAl might provide high electron density to the samples
due to its low formation energy for all the atomic chemical potentials
considered here. The formation of AlGe is also comparable with specific growth
conditions. The estimated binding energy shows GeAl and AlGe would prefer to
bond and form a complex defect, but the complex may easily dissociate at high
temperatures. Our results propose that the position of the chemical potential
can be tuned by putting the atomic chemical potentials to the condition
resulting in the formation of the two antisites are comparable, and therefore
it might shift back to the Fermi level where the Weyl nodes are located.

###### Acknowledgements.

This work was supported by Inha University Research Grant (INHA-68917). We
also acknowledge the use of the KISTI supercomputing center (grant no.
KSC-2023-CRE-0421).

## References

  * Yang (2016) S. A. Yang, SPIN 06, 1640003 (2016), URL https://doi.org/10.1142/S2010324716400038.
  * Wang and Sun (2023) X. Wang and D. Sun, Light: Science & Applications 12, 53 (2023), URL https://doi.org/10.1038/s41377-023-01100-x.
  * Armitage et al. (2018) N. P. Armitage, E. J. Mele, and A. Vishwanath, Rev. Mod. Phys. 90, 015001 (2018), URL https://link.aps.org/doi/10.1103/RevModPhys.90.015001.
  * Hasan and Kane (2010) M. Z. Hasan and C. L. Kane, Rev. Mod. Phys. 82, 3045 (2010), URL https://link.aps.org/doi/10.1103/RevModPhys.82.3045.
  * Geim and Novoselov (2007) A. K. Geim and K. S. Novoselov, Nature Materials 6, 183 (2007), URL https://doi.org/10.1038/nmat1849.
  * Xu et al. (2015) S.-Y. Xu, I. Belopolski, N. Alidoust, M. Neupane, G. Bian, C. Zhang, R. Sankar, G. Chang, Z. Yuan, C.-C. Lee, et al., Science 349, 613 (2015), URL https://doi.org/10.1126/science.aaa9297.
  * Hills et al. (2017) R. D. Y. Hills, A. Kusmartseva, and F. V. Kusmartsev, Phys. Rev. B 95, 214103 (2017), URL https://link.aps.org/doi/10.1103/PhysRevB.95.214103.
  * Yesilyurt et al. (2016) C. Yesilyurt, S. G. Tan, G. Liang, and M. B. A. Jalil, Scientific Reports 6, 38862 (2016), URL https://doi.org/10.1038/srep38862.
  * Hou et al. (2019) Z. Hou, Y.-F. Zhou, N.-X. Yang, and Q.-F. Sun, Communications Physics 2, 86 (2019), URL https://doi.org/10.1038/s42005-019-0186-9.
  * Guo et al. (2023) C. Guo, V. S. Asadchy, B. Zhao, and S. Fan, eLight 3, 2 (2023), URL https://doi.org/10.1186/s43593-022-00036-w.
  * Lv et al. (2015) B. Q. Lv, H. M. Weng, B. B. Fu, X. P. Wang, H. Miao, J. Ma, P. Richard, X. C. Huang, L. X. Zhao, G. F. Chen, et al., Phys. Rev. X 5, 031013 (2015), URL https://link.aps.org/doi/10.1103/PhysRevX.5.031013.
  * Ong and Liang (2021) N. P. Ong and S. Liang, Nature Reviews Physics 3, 394 (2021), URL https://doi.org/10.1038/s42254-021-00310-9.
  * Xu et al. (2017a) S.-Y. Xu, N. Alidoust, G. Chang, H. Lu, B. Singh, I. Belopolski, D. S. Sanchez, X. Zhang, G. Bian, H. Zheng, et al., Science Advances 3, e1603266 (2017a), eprint https://www.science.org/doi/pdf/10.1126/sciadv.1603266, URL https://www.science.org/doi/abs/10.1126/sciadv.1603266.
  * Hodovanets et al. (2018) H. Hodovanets, C. J. Eckberg, P. Y. Zavalij, H. Kim, W.-C. Lin, M. Zic, D. J. Campbell, J. S. Higgins, and J. Paglione, Phys. Rev. B 98, 245132 (2018), URL https://link.aps.org/doi/10.1103/PhysRevB.98.245132.
  * Hu et al. (2020a) Z. Hu, Q. Du, Y. Liu, D. Graf, and C. Petrovic, Applied Physics Letters 117, 222410 (2020a), URL https://doi.org/10.1063/5.0035445.
  * Yu et al. (2017) Y. G. Yu, X. Zhang, and A. Zunger, Phys. Rev. B 95, 085201 (2017), URL https://link.aps.org/doi/10.1103/PhysRevB.95.085201.
  * Kim et al. (2018) H. Kim, K. Wang, Y. Nakajima, R. Hu, S. Ziemak, P. Syers, L. Wang, H. Hodovanets, J. D. Denlinger, P. M. R. Brydon, et al., Science Advances 4, eaao4513 (2018), eprint https://www.science.org/doi/pdf/10.1126/sciadv.aao4513, URL https://www.science.org/doi/abs/10.1126/sciadv.aao4513.
  * Kim et al. (2022) H. Kim, J. Lee, H. Hodovanets, K. Wang, J. D. Sau, and J. Paglione, Phys. Rev. Res. 4, 033169 (2022), URL https://link.aps.org/doi/10.1103/PhysRevResearch.4.033169.
  * Alkauskas et al. (2012) A. Alkauskas, J. L. Lyons, D. Steiauf, and C. G. Van de Walle, Phys. Rev. Lett. 109, 267401 (2012).
  * Alkauskas et al. (2016) A. Alkauskas, M. D. McCluskey, and C. G. Van de Walle, Journal of Applied Physics 119, 181101 (2016), ISSN 0021-8979, eprint https://pubs.aip.org/aip/jap/article-pdf/doi/10.1063/1.4948245/15179175/181101_1_online.pdf, URL https://doi.org/10.1063/1.4948245.
  * Heyd et al. (2003) J. Heyd, G. E. Scuseria, and M. Ernzerhof, J. Chem. Phys. 118, 8207 (2003).
  * Krukau et al. (2006) A. V. Krukau, O. A. Vydrov, A. F. Izmaylov, and G. E. Scuseria, J. Chem. Phys. 125, 224106 (2006).
  * Blöchl (1994) P. E. Blöchl, Phys. Rev. B 50, 17953 (1994).
  * Kresse and Hafner (1993) G. Kresse and J. Hafner, Phys. Rev. B 48, 13115 (1993).
  * (25) See Supplemental Material at http://link.aps.org/supplemental/ for more details.
  * Tomczak (2015) J. M. Tomczak, J. Phys. Conf. Ser. 592, 012055 (2015).
  * Choi et al. (2016) S. Choi, A. Kutepov, K. Haule, M. van Schilfgaarde, and G. Kotliar, npj Quantum Mater. 1, 16001 (2016), URL https://doi.org/10.1038/npjquantmats.2016.1.
  * Choi et al. (2019) S. Choi, P. Semon, B. Kang, A. Kutepov, and G. Kotliar, Comput. Phys. Commun. 244, 277 (2019), ISSN 0010-4655, URL https://www.sciencedirect.com/science/article/pii/S0010465519302140.
  * Kutepov et al. (2017) A. Kutepov, V. Oudovenko, and G. Kotliar, Comput. Phys. Commun. 219, 407 (2017), ISSN 0010-4655, URL https://www.sciencedirect.com/science/article/pii/S0010465517301947.
  * Kutepov et al. (2012) A. Kutepov, K. Haule, S. Y. Savrasov, and G. Kotliar, Phys. Rev. B 85, 155129 (2012), URL https://link.aps.org/doi/10.1103/PhysRevB.85.155129.
  * Georges et al. (1996) A. Georges, G. Kotliar, W. Krauth, and M. J. Rozenberg, Rev. Mod. Phys. 68, 13 (1996), URL https://link.aps.org/doi/10.1103/RevModPhys.68.13.
  * Metzner and Vollhardt (1989) W. Metzner and D. Vollhardt, Phys. Rev. Lett. 62, 324 (1989), URL https://link.aps.org/doi/10.1103/PhysRevLett.62.324.
  * Georges and Kotliar (1992) A. Georges and G. Kotliar, Phys. Rev. B 45, 6479 (1992), URL https://link.aps.org/doi/10.1103/PhysRevB.45.6479.
  * Van de Walle and Neugebauer (2004) C. G. Van de Walle and J. Neugebauer, J. Appl. Phys. 95, 3851 (2004).
  * Cordfunke and Konings (2001) E. Cordfunke and R. Konings, Thermochim. Acta 375, 65 (2001).
  * (36) M.W. Chase Jr., NIST-JANAF Thermochemical Tables, 4th ed. (AIP, 1998).
  * Cheng and Navrotsky (2003) J. Cheng and A. Navrotsky, J. Mater. Res. 18, 2501 (2003).
  * Perdew et al. (1996) J. P. Perdew, K. Burke, and M. Ernzerhof, Phys. Rev. Lett. 77, 3865 (1996).
  * Guloy and Corbett (1991) A. M. Guloy and J. D. Corbett, Inorganic Chemistry 30, 4789 (1991), eprint https://doi.org/10.1021/ic00025a021, URL https://doi.org/10.1021/ic00025a021.
  * Xu et al. (2017b) S. Y. Xu, N. Alidoust, G. Chang, H. Lu, B. Singh, I. Belopolski, D. S. Sanchez, X. Zhang, G. Bian, H. Zheng, et al., Science Advances 3 (2017b), ISSN 2375-2548, URL https://www.osti.gov/biblio/1473922.
  * Cao et al. (2022) W. Cao, N. Zhao, C. Pei, Q. Wang, Q. Zhang, T. Ying, Y. Zhao, L. Gao, C. Li, N. Yu, et al., Phys. Rev. B 105, 174502 (2022), URL https://link.aps.org/doi/10.1103/PhysRevB.105.174502.
  * Howard et al. (2000) C. J. Howard, B. J. Kennedy, and B. C. Chakoumakos, J. Phys.: Condens. Matter 12, 349 (2000).
  * Siddiquee et al. (2023) H. Siddiquee, C. Broyles, E. Kotta, S. Liu, S. Peng, T. Kong, B. Kang, Q. Zhu, Y. Lee, L. Ke, et al., Nat. Commun. 14, 527 (2023).
  * Kang et al. (2023a) B. Kang, Y. Lee, L. Ke, H. Kim, and M.-H. Kim, arXiv preprint arXiv:2305.08003 (2023a).
  * Kang et al. (2022) B. Kang, S. Choi, and H. Kim, npj Quantum Mater. 7, 64 (2022).
  * Kang et al. (2023b) B. Kang, H. Kim, Q. Zhu, and C. H. Park, Cell Rep. Phys. Sci. 4 (2023b).
  * Kang et al. (2023c) B. Kang, C. Melnick, P. Semon, S. Ryee, M. J. Han, G. Kotliar, and S. Choi, npj Quantum Mater. 8, 35 (2023c).
  * Kang and Choi (2019) B. Kang and S. Choi, arXiv preprint arXiv:1908.05643 (2019).
  * Momma and Izumi (2011) K. Momma and F. Izumi, J. Appl. Cryst. 44, 1272 (2011).
  * Hu et al. (2020b) Z. Hu, Q. Du, Y. Liu, D. Graf, and C. Petrovic, Applied Physics Letters 117, 222410 (2020b), ISSN 0003-6951, eprint https://pubs.aip.org/aip/apl/article-pdf/doi/10.1063/5.0035445/14542783/222410_1_online.pdf, URL https://doi.org/10.1063/5.0035445.