  * Citation

Z. Fowler, K. Kokilepersaud, M. Prabhushankar, G. AlRegib, ”Clinical Trial
Active Learning,” in Proceedings of the 14th ACM International Conference on
Bioinformatics, Computational Biology and Health Informatics (ACM-BCB), 2023.

  * Review

Date of Acceptance: July 12th 2023

  * Codes

https://github.com/olivesgatech/Clinical-Trial-Active-Learning.git

  * Bib

@inproceedings{fowler2023clinical,
title={Clinical Trial Active Learning},
author={Z. Fowler, K. Kokilepersaud, M. Prabhushankar, G. AlRegib},
booktitle={The 14th ACM Conference on Bioinformatics, Computational Biology
and Health Informatics (ACM-BCB)},
year={2023}}

  * Contact

{zfowler3, kpk6, mohit.p<EMAIL_ADDRESS>
https://ghassanalregib.info/

# Clinical Trial Active Learning

Zoe Fowler<EMAIL_ADDRESS>Georgia Institute of
TechnologyAtlantaGeorgiaUSA ,  Kiran Kokilepersaud<EMAIL_ADDRESS>Georgia
Institute of TechnologyAtlantaGeorgiaUSA ,  Mohit Prabhushankar
<EMAIL_ADDRESS>Georgia Institute of TechnologyAtlantaGeorgiaUSA and
Ghassan AlRegib<EMAIL_ADDRESS>Georgia Institute of
TechnologyAtlantaGeorgiaUSA

(2023; 11 June 2023; 6 July 2023; 12 July 2023)

###### Abstract.

This paper presents a novel approach to active learning that takes into
account the non-independent and identically distributed (non-i.i.d.) structure
of a clinical trial setting. There exists two types of clinical trials:
retrospective and prospective. Retrospective clinical trials analyze data
after treatment has been performed; prospective clinical trials collect data
as treatment is ongoing. Typically, active learning approaches assume the
dataset is i.i.d. when selecting training samples; however, in the case of
clinical trials, treatment results in a dependency between the data collected
at the current and past visits. Thus, we propose prospective active learning
to overcome the limitations present in traditional active learning methods and
apply it to disease detection in optical coherence tomography (OCT) images,
where we condition on the time an image was collected to enforce the i.i.d.
assumption. We compare our proposed method to the traditional active learning
paradigm, which we refer to as retrospective in nature. We demonstrate that
prospective active learning outperforms retrospective active learning in two
different types of test settings.

Active learning, Deep learning, Retinal disease detection, Prospective
clinical trials

††copyright: acmcopyright††journalyear: 2023††doi: XXXXXXX.XXXXXXX††ccs:
Computing methodologies Active learning settings††ccs: Applied computing
Health informatics

##  1\. Introduction

Clinical trials are research studies that evaluate the effects of medical
treatments (chien2022multi, 4). There are two different types of clinical
trials: retrospective and prospective as illustrated in Figure 1. In part a)
of Figure 1, we show a retrospective clinical trial, where data is collected
about participants after treatment has been administered. This type of
clinical trial has access to the entirety of the pre-existing data and has no
control over the design of the data collection process itself; it relies only
on previous data to draw conclusions. However, part b) of Figure 1 shows that
a prospective clinical trial collects data about participants over time as
they are receiving treatment (euser2009cohort, 8). Prospective clinical trials
are designed with a specific treatment and research plan, and only the data
collected through this specific clinical trial is used for analysis.

Figure 1. Retrospective vs Prospective Clinical Trial Description Figure 2.
Retrospective active learning: For each query strategy, the percentage of
samples queried at each training round from current and past visits versus
samples chosen from future visits is compared.

The prospective clinical trial procedure illustrated in Figure 1 has several
complications. The main complication is that doctors assist a large volume of
patients on any given day, leading to an increased demand when diagnosing
patients. This demand is demonstrated in a study from 2015
(mcdonald2015effects, 18), which concluded that the average radiologist is
tasked with interpreting an image every 3-4 seconds to satisfy their large
workload. This time requirement is typically not possible due to the time-
consuming and difficult nature of interpreting these images. This increased
workload leads to burnout, fatigue, and a large labeling error rate.

To alleviate these complications, machine learning is a potential solution to
assist doctors through many of the steps of a clinical trial. Machine learning
identifies meaningful features from data automatically that can be used to
obtain diagnostic decisions (erickson2017machine, 7). Thus, machine learning
allows model outputs to provide a second opinion to doctors when diagnosing
patients for certain diseases (kokilepersaud2022gradient, 13),
(kokilepersaud2023clinically, 12), (wang2017chestx, 28), (wu2019deep, 31).

Applying standard machine learning approaches to the medical field is hardly
straightforward. Specifically, there are two core challenges: 1) a large
labeled dataset requirement and 2) an i.i.d. assumption of the dataset
structure. Addressing the first point, most standard machine learning
algorithms require large amounts of labeled data samples in order to
accurately train a model. In the medical field, this labeled medical data can
take many forms, a common one being disease diagnosis. However, as stated
previously, this label is oftentimes time-consuming and challenging to provide
due to the requirement of a trained medical expert.

Active learning is a common approach for tackling the first issue of needing
large amounts of labeled data. Specifically, active learning does not require
a large labeled set to train a machine learning model; instead, it is an
iterative process that selects a smaller subset of training samples across
multiple training rounds. Active learning uses query strategies to acquire
this new selection of data and provides it to an external expert to label,
reducing the overall amount of data annotated by an expert while still
achieving better diagnosis with high accuracy (settles2009active, 23). The
query strategies aim to select the most informative samples, resulting in a
refined training set that quickly reduces the generalization error
(kumar2020active, 14). In addition, active learning algorithms mirror the data
collection process in prospective clinical trials as illustrated in Figure 1.
Namely, a patient deposits data on each visit, and an expert must choose a
subset of this data to annotate based on a fixed labeling budget. This
parallels the active learning paradigm, where the most informative samples are
selected by a query strategy and labeled by an expert for each training round.

Despite active learning bypassing the need for large amounts of labeled data,
prior query strategies assume that the data is i.i.d., as data is sampled
based on solely a query criteria (settles2009active, 23), (wang2015querying,
29). This leads to the second core problem: traditional active learning
attempts make i.i.d. assumptions about the dataset structure, disregarding
information about how data samples may be correlated across time
(scholkopf2021toward, 21). There are several issues with active learning
failing to account for this problem: 1) active learning disregards the
non-i.i.d. dependencies and consequences of these dependencies that arise
during the clinical trial procedure and 2) active learning does not mirror how
the data collection process is realistically performed in clinical trials.
Examining the first point, part b) of Figure 1 illustrates that patients
sometimes receive treatment at each visit during a prospective clinical trial.
Patients then experience a gap between their next visit, where this process is
repeated. Therefore, due to the treatment and the gap between visits, the
manifestation of the disease changes between visits. For some patients, they
may become more healthy, while others may become more ill. Hence, a patient’s
data at a particular visit can be viewed as a function of this previous visit
information and cannot be treated as an independent data sample.

Figure 3. Prospective clinical trial data collection process

The second challenge for using active learning in clinical trials is that the
data collection process in clinical trials is itself non-i.i.d. At a specific
visit in a clinical trial, the current patient information depends on the
patient’s medical data from previous visits. As a result, the sampling
strategies deployed during traditional active learning algorithms are often
unrealistic, as they sample images or data that the model would not
necessarily be able to access at that point in a clinical scenario. This point
is demonstrated in Figure 2. In this figure, we train an active learning model
on non-i.i.d. medical image data from the OLIVES dataset
(prabhushankar2022olives, 20) for 20 rounds, adding in new queried data at
each round. This figure shows the percentage of the total number of samples
queried from future visits versus current and past visits. We repeat this
calculation of the percentage at every training round for six different query
strategies. As shown in Figure 2, for all six query strategies the active
learning model relies heavily on future samples until later rounds, meaning
that active learning is _not_ being performed in a way that aligns with
prospective clinical trials despite the routine itself matching the data
collection process shown in Figure 1.

Traditional active learning algorithms further diverge from the prospective
clinical trial process through their evaluation procedure. In a realistic
prospective clinical trial setting, the treatment effect is evaluated on
patients’ current visit data. Thus, the data analyzed by medical experts is
dynamic and changes at each visit. Traditional active learning algorithms,
however, evaluate performance on a fixed test set across rounds, resulting in
the evaluation procedure common in these traditional active learning paradigms
to contribute further to how unrealistic these approaches are. Hence,
traditional attempts to apply active learning to medical data can be
considered as viewing the active learning process from a retrospective lens
(logan2022decal, 17), (smailagic2018medal, 24). We argue instead that the
_active learning process for medical data must be viewed from the prospective
clinical trial lens presented in Figure 1_ because of the nature of the
clinical trial data collection process. Throughout this paper, we refer to
traditional active learning methods as retrospective active learning, which
assumes access to the entirety of the collected data with no concern as to how
data samples may be correlated over time. On the other hand, we refer to
prospective active learning as active learning methods that follow the
structure of prospective clinical trials by querying data sequentially. To the
best of our knowledge, this is the first work that uses machine learning in a
prospective clinical trial setting.

In this paper, we address the aforementioned problems by introducing a
prospective active learning methodology that overcomes the inherent
limitations of existing active learning strategies to apply them in clinical
settings. Specifically, we consider the data collection process of clinical
trials that creates time-related dependencies across data samples in clinical
trial datasets, and we examine different model test settings by considering
realistic clinical setting conditions. Thus, the contributions of this paper
are as follows:

  1. (1)

We convert both retrospective and prospective clinical trials into active
learning frameworks by following the setup for both described in Figure 1.

  2. (2)

We demonstrate and compare performance for this setup against retrospective
active learning results on the classification of diseased Optical Coherence
Tomography (OCT) scans from an ophthalmology prospective clinical trial.

  3. (3)

We analyze our prospective active learning framework in different test
settings, including a fixed test set to demonstrate overall generalization
capabilities and a dynamic test set that generalizes to specific visits.

##  2\. Non-i.i.d. Explanation

In this section, we discuss in depth the non-i.i.d. structure of clinical
trial data. Consider an ophthalmology prospective clinical trial for Diabetic
Retinopathy (DR) and Diabetic Macular Edema (DME) assessment, illustrated in
Figure 3. In this clinical trial, assume we have a set of patients
$P=\\{p_{1},\cdots,p_{N}\\}$, where each $p_{n}\in P$ represents a unique
patient. Additionally, each $p_{n}$ is associated with a set of visits
$T=\\{t_{1},\cdots,t_{D}\\}$, where each $t_{d}\in T$. Figure 3 indicates that
at a particular visit $t_{d}$, a patient $p_{n}$ is assessed for DR and DME
through a medical expert’s examination of the patient’s OCT data of the form
$X_{n,d}$, which indicates the patient number $n$ and time step number $d$ the
data originated from. Then, depending on the expert’s assessment, patients
receive an eye injection $I_{1}$ or $I_{2}$ depending on which disease the
patient is diagnosed with. Thus, a patient’s OCT scans at a particular visit
are dependent on all previously administered treatment injections. We
represent this as $X_{n,d}=I_{1}(X_{n,d-1})$ or $X_{n,d}=I_{2}(X_{n,d-1})$, as
$X_{n,d}$ is a result of the treatment decision at patient $p_{n}$’s previous
visit $t_{d-1}$. Because a patient’s current visit data depends on the
previous visit, the data collected at each visit stage cannot be assumed to be
i.i.d.

The i.i.d. assumption prevalent in machine learning algorithms expects that
data samples from a particular dataset are independently drawn from the same
probability distribution (wu2022towards, 30). Throughout this paper, we
consider a clinical trial dataset to be i.i.d. if a patient’s data deposited
at a particular visit has no dependency on previous visit data. However, in
prospective clinical trial settings, the effect of a certain treatment is
observed by analyzing a patient’s data across several visits. By nature, the
resulting clinical trial dataset is non-i.i.d., which retrospective active
learning algorithms inherently disregard.

To understand how taking into account this non-i.i.d. structure affects active
learning performance, we analyze the aforementioned data collection process
through Figure 4. In Figure 4, assume a patient $p_{n}$ that has been
diagnosed with either DR or DME has completed the clinical trial, resulting in
a total of $d$ visits. Figure 4 visualizes the total collected OCT images
$X_{n}$ from this patient across their $d$ visits. Between each time step or
visit, a treatment, unique to the disease, has been administered. We term this
an intervention and represent it using $I_{1}$ for DR or $I_{2}$ for DME. This
intervention relates the OCT images at $t_{d}$ and $t_{d-1}$ for any
$d\in\\{1,\cdots,D\\}$. In the context of classification, estimating the
intervention function by utilizing machine learning algorithms is a viable way
to classify a data sample as belonging to a particular class.

Figure 4. Patient $p_{n}$’s total collected data $X_{n}$ across $d$ visits
(Intervention function applied between successive visits is dependent on which
disease class patient $p_{n}$ belongs to)

However, the overall order in which the data is analyzed to determine the
intervention function affects the overall estimate of this function and thus
classification decisions. We argue that if the data is observed in a
sequential manner, the intervention function $I_{1}$ or $I_{2}$ will be able
to be properly estimated. However, if the data is observed non-sequentially,
the intervention estimated is likely not equivalent to the actual
intervention, resulting in an incorrect classification result since the data
at a particular visit is dependent on all previous intervention functions
applied during past visits. By creating a framework that queries images
prospectively, i.e. in a sequential manner based on patient visit number, we
separate our work from previous active learning works that make an i.i.d.
assumption on the data. We show that with this framework, we attain increased
performance on a non-i.i.d. clinical trial dataset by exploiting the
relationship between data at each time step.

##  3\. Relation to Prior Work

###  3.1. Deep Learning in OCT

The proposed framework is applicable to any clinical trial. However, we
utilize OCT data as a case study. The clinical trial process can be laborious,
as demonstrated in an ophthalmology setting through Figure 3. Hence, there is
an interest in applying deep learning to ophthalmology to reduce diagnosis
time and increase diagnostic accuracy. For example, (temel2019relative, 25)
uses a transfer learning-based approach to detect abnormalities in relative
afferent pupillary screening. (choudhary2023deep, 5) is aimed towards using
machine learning for accurate disease classification of four different retinal
disease categories. (dai2021deep, 6) uses fundus images for labeling the
severity of DR and the presence of other retinal disease factors.
(tsuji2020classification, 26) uses a capsule network to exploit positional
information in OCT images for disease classification.

Other approaches have been designed to make deep learning more explainable
from a medical viewpoint. For example, (logan2022multi, 16) uses multi-modal
learning to incorporate OCT scans alongside physician insights for disease
classification. Additional methods aim to uncover meaningful clinical insights
and relationships. (kokilepersaud2023clinically, 12), for instance, uses
contrastive learning to discover a relationship between clinical values and
biomarker structures that manifest within OCT scans. All the aforementioned
methods demonstrate the vast potential for deep learning to be applied in
ophthalmology settings, despite the requirement of a large set of labeled
training data.

###  3.2.  Medical Active Learning

Figure 5. Active learning framework - I) Choose acquisition strategy and
sample from $X_{\text{pool}}$ accordingly II) Add sampled data $X^{*}$ to the
training set $X_{\text{train}}$ III) Evaluated trained model on test set

The premise behind active learning is that a machine learning algorithm can
achieve higher accuracy with fewer labeled samples (settles2009active, 23). By
choosing the most relevant data samples, active learning is able to reduce the
overall labeled data needed to train a model, while still experiencing
increased model performance. Active learning is performed iteratively; in each
iteration of training a machine learning model, a query strategy is applied to
determine the most informative samples to label. These samples are then added
to the model’s training set. The simplest strategy is to select unlabeled data
samples randomly, known as random sampling. Uncertainty-based methods are also
popular and include entropy uncertainty, margin sampling, and least confidence
sampling. Entropy presents a simple solution to querying samples by
identifying the top $B$ most uncertain samples to label (wang2014new, 27).
Margin sampling chooses samples that have the smallest margin between the two
highest predicted classes, and least confidence selects samples whose
predicted labels have the lowest probability (wang2014new, 27). Other methods
for querying samples focus on enhancing sample diversity. These diversity-
based strategies include Coreset, where new samples are queried at each round
using furthest-first traversal on prior labeled samples (sener2017active, 22),
(ash2019deep, 1). The BADGE query strategy (ash2019deep, 1), on the other
hand, takes into account both the diversity and uncertainty of samples. Active
learning is favorable in situations where acquiring labeled data samples is
challenging and time-consuming.

Using active learning in a medical setting is well-motivated by the fact that
labeling medical images is expensive due to requiring a trained medical expert
(bangert2021active, 2). Therefore, it is essential to develop algorithms that
reduce the labeling burden in order to assist medical experts in their
practice. Thus, there have been many previous efforts to apply active learning
for this purpose. Mainly, efforts revolve around altering active learning
frameworks to drastically reduce the number of labeled images needed in order
to achieve high accuracy. For example, (bangert2021active, 2) designed an
active learning framework by adding a pre-labeling model to provide human
annotators with estimated class labels to reduce overall labeling time. The
authors applied this method in many medical case studies, one of them being
the detection of Covid-19 in a dataset, and found that with their active
learning algorithm, they needed only $5\%$ of the images to be human-labeled
in order to achieve $93\%$ accuracy on the Covid-19 dataset.

Other efforts to incorporate active learning in a medical setting and reduce
the overall number of labeled images include (smailagic2018medal, 24),
(logan2022decal, 17), (logan2022patient, 15), (el2022multi, 9), and
(hoi2006batch, 11). (smailagic2018medal, 24) aims to sample unlabeled images
by combining entropy sampling and a distance function that maximizes the
average distance to all remaining images in the data pool in the feature
space. (logan2022decal, 17), (logan2022patient, 15) provides a plug-in
approach that combines patients’ clinical data with existing active learning
query strategies for the sampling of medical images. (el2022multi, 9) uses
active learning to diagnose heart disease and tests generalization capability,
and (hoi2006batch, 11) is an earlier work on active learning that uses the
Fisher information matrix to query batches of unlabeled samples and tests this
method’s ability to categorize medical images.

The aforementioned methods have many complications in a real-life clinical
setting. Mainly, all are evaluated retrospectively, with no discussion on
whether the medical data the algorithms have been both trained and evaluated
on exhibit any dependencies. All attempts consider each data sample as being
independent from others, even if the dataset utilized is from prospective
clinical trials.

##  4\. Proposed Method

###  4.1. Dataset

Figure 6. Retrospective active learning (RAL) vs Prospective active learning
(PAL) test accuracy results averaged over 5 seeds.

For our experiments, we utilize the OLIVES (prabhushankar2022olives, 20)
dataset. The OLIVES dataset consists of $78,189$ OCT images, collected from
two different clinical trials. The two clinical trials are denoted as PRIME,
which recruited patients with DR, and TREX, which recruited patients with DME.
The patients in PRIME and TREX received two distinct treatments, allowing the
assumptions from Figure 4 to reflect the structure of the actual dataset. The
DR patients in PRIME receive treatment $I_{1}$, whereas the DME patients in
TREX receive a different treatment $I_{2}$.

The clinical trial format of the OLIVES dataset overall follows closely to
what is described in Figure 3. Patients arrive at a clinic and have a series
of diagnostic exams and imaging procedures performed on them. Afterwards, they
are evaluated for treatment and wait until their next follow-up appointment to
repeat the data collection process. On average, patients in the dataset were
enrolled in the clinical trials for approximately 114 weeks. On average, this
equates to 16.6 visits per patient and 14.2 treatment injections. The number
of weeks between visits varies across the clinical trials; in most cases,
there are around 4 weeks between visits, though this amount increases if
patients miss appointments or if the clinical trial protocol adjusts this
length of time.

The OLIVES dataset reflects the clinical trial process by mapping each OCT
image to a visit number that indicates when in time an image was collected
during the clinical trials. Each OCT image also corresponds to a disease state
(DR/DME), anonymous patient identity number, and anonymous eye identity
number. The OLIVES dataset provides data from 96 unique eyes and 87 unique
patients. From the 96 unique eyes, we derive our train and test sets. For the
train set, we include 76 unique eyes and utilize the remaining 20 eyes for our
test set. The test set is balanced between disease classes.

###  4.2. Problem Formulation

Assume access to a dataset $X$ that contains a time-series component such that
the dataset can be represented as

(1) $X=\\{x_{t_{1}},x_{t_{2}},\dots,x_{t_{D}}\\},$

where $x_{t}$ represents the data collected at a specific time $t_{d}$ for any
$d\in\\{1,\cdots,D\\}$ from all $N$ associated patients
$P=\\{p_{1},\cdots,p_{N}\\}$. Thus, the resultant dataset has the form:

(2) $\displaystyle x_{t_{1}}=\\{p_{1,1},p_{1,2},\dots,p_{1,N}\\}$
$\displaystyle\vdots$ $\displaystyle
x_{t_{D}}=\\{p_{D,1},p_{D,2},\dots,p_{D,N}\\}$

Consider the context of active learning, where the objective is to iterate
through the unlabelled data pool $X_{\text{pool}}$ of dataset $X$ and select a
subset of data to add to the training set $X_{\text{train}}$. At each round of
training, query strategies select a batch of $b$ data samples $X^{*}$ until
the budget constraint $B$ is met. Following our definition of retrospective
active learning, we can adapt (benkert2023gaussian, 3) to define the selection
of such samples in the retrospective case as

(3) $X^{*}=\underset{x_{t_{1}},\dots,x_{t_{b}}\in
X_{\text{pool}}}{\arg\max}a_{retro}(x_{t_{1}},\dots,x_{t_{b}})$

where $a_{retro}$ is the acquisition function for the retrospective setting.
In the case of retrospective active learning, this is simply a pre-existing
query strategy (i.e., random, least confidence, etc.) that selects data from
$X_{\text{pool}}$ based off of a specific criteria unique to each query
strategy.

For prospective active learning, the non-i.i.d. structure of the data is taken
into consideration. Our selection of samples in this case is

(4) $X^{*}=\underset{x_{t_{1}},\dots,x_{t_{b}}\in
X_{\text{pool}}}{\arg\max}a_{pro}(x_{t_{1}},\dots,x_{t_{b}}|d)$

where we augment Equation 4 such that we condition the selection based on the
time or visit number $d$ of when the data was collected. This results in
$a_{pro}$, which constrains the selection space on which pre-existing
strategies can operate.

We design and implement an active learning framework to test the effect of
querying images in a prospective manner using our formulation of $a_{retro}$
and $a_{pro}$. We apply this framework to disease classification using the
OLIVES dataset, where OCT images from the dataset are classified as having
either DR or DME. The OLIVES dataset, like clinical trials, provides visit-
wise data. Hence, at the first visit, OCT scans will be collected from $N$
patients, resulting in $N\times 49$ OCT scans. Therefore, when performing
prospective active learning via $a_{pro}$, we only have $N\times 49$ OCT scans
to sample from in the first round. For the next visit, there are the remaining
OCT scans from the first visit that were not selected during the first round,
as well as an additional $N\times 49$ OCT scans corresponding to the patients’
second visit, that can be sampled from during the second round of training the
prospective active learning model. In other words, each round $d$ of training
an active learning model corresponds to patients’ visit $d$ and past (unused)
visit data. On the other hand, retrospective active learning will have access
to the $N\times 49$ OCT scans from the first visit, the $N\times 49$ OCT scans
from the second visit, and so forth. Hence at the first round itself there are
$D\times N\times 49$ OCT scans to choose from, where $D$ represents the total
number of visits recorded in the dataset. At the second round of training the
model, retrospective active learning has access to all the unused OCT scans
from the previous round. The retrospective case presents all OCT images in the
training pool as an option to be queried from, regardless of which patient
visit they were collected at.

We use Figure 5 to describe this experimental setup. Following the earlier
discussion, in part I of Figure 5 we iterate through $X_{\text{pool}}$ of the
dataset, sampling our b data samples $X^{*}$ prospectively via $a_{pro}$ or
retrospectively via $a_{retro}$. Once this new data has been queried, we add
this data to the training set $X_{\text{train}}$ in part II of Figure 5.
Afterwards, part III of Figure 5 shows the trained model being evaluated on
the test set for DR and DME classification. Finally, we repeat these steps,
where we query new images either prospectively or retrospectively at each
round of training until a total of $B$ images have been queried, with $B$
describing the overall budget constraint.

Figure 7. Retrospective active learning (RAL) vs Prospective active learning
(PAL) test accuracy results averaged over 5 seeds, where PAL queries visits
non-sequentially.

##  5\. Experimental Design and Results

###  5.1. Experimental Design

We implement the active learning framework described in Figure 5. We
initialize the model’s training set with 128 randomly selected data samples.
At each successive training round $d$, 256 new data samples are queried. These
data samples are selected either retrospectively or prospectively. The model’s
training set then consists of these newly queried samples in addition to
samples that were selected in previous rounds. After the model has been
trained with these new samples, the model is evaluated on the test set. This
process is then repeated for $20$ rounds (corresponding to a total of 20
visits), resulting in a budget $B=4,992$ images. With the current query size
selection of 256 images, the medical expert needs to label only 6.38$\%$ of
the total data collected, as the OLIVES dataset contains a total of 78,189 OCT
images. Our current query size selection takes into account a medical expert’s
limited labeling budget while still providing enough training data at each
round such that the model is able to learn the fine-grained differences
between DR and DME.

For our experiments, we utilize the Pytorch implementation of the Resnet18
architecture (he2016deep, 10), with the linear layer being altered to
accommodate binary classification. Thus, the input into the Resnet18 model is
a batch of OCT images, and the output is a binary classification decision
corresponding to the presence of either DR or DME. We train this model each
round for DR/DME detection until a minimum accuracy of 97$\%$ is met. The Adam
optimizer was used during training, with a learning rate of 0.00015. This
training regimen was adopted from (prabhushankar2022olives, 20) and
(logan2022patient, 15), which presented results for DR/DME classification
using these hyperparameter values. All OCT images were resized to $128\times
128$ and were normalized with $\mu=0.1706$ and $\sigma=0.2112$.

We define two different test sets to examine the impact of querying images
prospectively: 1) a fixed test set and 2) a dynamic test set. The fixed test
set aims to test the trained model’s overall generalization capabilities,
whereas the dynamic test set examines the trained model’s current and past
visit generalization performance. We differentiate between the two test sets
as follows:

####  5.1.1. Fixed Test Set

The fixed test set is composed of $2,000$ OCT images from the 20 eyes excluded
from the train set. The fixed test set is balanced between the two disease
classes and contains OCT images collected from patients’ Visit 1 through Visit
22 data. After each round of training both the prospective and retrospective
active learning model, the trained model is evaluated on this particular test
set. This test set evaluates how well the model can distinguish between two
different eye disease classes across time and varying levels of severity.

####  5.1.2. Dynamic Test Set

The dynamic test set increases in size after each round of training the active
learning model. At each round $d$ of training, 80 OCT images collected at
visit $d$ from the 20 eyes excluded from the train set are added to the
dynamic test set. At the final round of training, this results in a dynamic
test set size of a total of 1,600 OCT images. The OCT images added at each
round are class-balanced, and the composition of the dynamic test set is the
same when evaluating the performance of the retrospective and prospective
approaches.

In a prospective clinical trial setting, the treatment effect is evaluated
only on patients’ current visit data. Thus, the total amount of data that has
been evaluated by medical experts increases at each patient visit. The dynamic
test set contributes to realistically assessing the performance of both
retrospective and prospective active learning in a clinical trial setting, as
the model at each round is evaluated on data from that current visit in
addition to previous visits. The dynamic test set ensures that both active
learning setups are able to generalize to the current visit, while not losing
important information collected about previous visits.

###  5.2. Fixed Test Set Results

####  5.2.1. Retrospective vs Prospective Active Learning

A comparison between retrospective and prospective active learning for the
fixed test set is shown in Figure 6 for the following query strategies:
Random, Entropy, Margin, Least Confidence, Coreset, and BADGE. Results are
averaged over 5 seeds and presented in Figure 6. In these plots, the x-axis
corresponds to the number of training samples queried at each round, while the
y-axis corresponds to the performance accuracy on the test set. The standard
errors for each method are shown by the shaded regions.

For majority of the query strategies, the prospective active learning setting
outperforms the traditional retrospective setting, particularly during later
rounds. Because prospective active learning queries data in a sequential
manner, we hypothesize that prospective active learning distinguishes between
the two types of treatment administered to the two disease classes, resulting
in better classification decisions. To further validate this hypothesis, we
assess the impact of the order in which visits are integrated into the
sampling process in prospective active learning. We define a new experimental
setup, where the prospective active learning framework randomly queries new
visits non-sequentially at each round. Results for this setup are presented in
Figure 7.

As demonstrated in Figure 7, prospective active learning fails to consistently
outperform traditional retrospective active learning in this experimental
setup, implying that the order in which these images are added to the model’s
training pool is significant. In particular, it implies that the model must
receive the images consecutively in order to provide accuracy benefits over
retrospective active learning. This indicates that each patient visit is
dependent on the last visits and can not be treated independently, as is what
is assumed in typical active learning experiments. Thus, by adopting a
prospective active learning framework in this medical setting, the model is
able to understand and learn the interventions on the training data at each
visit, resulting in higher performance and generalization ability.

When analyzing Figure 6, we also notice that at Round 8, there is a clear
transition in efficiency between prospective and retrospective active learning
across all query strategies. The transition in efficiency between the two
methods is most likely due to the increased training set size at that
particular round. In Figure 6, Round 8 corresponds to 1,920 training images,
as the model is initialized with 128 images and queries 256 new images at
successive rounds. In previous rounds, it is harder to generalize well to the
test set due to a smaller training set size. This is because medical data has
fine-grained structures that are difficult for a model to understand
(logan2022decal, 17), (kokilepersaud2023clinically, 12); therefore, a larger
initial training set is needed for the model to accurately classify an image.
In addition, prospective active learning converges much quicker than
retrospective active learning. Figure 6 shows that the model does not
necessarily need to train for the full 20 rounds in order to perform well. The
BADGE and Margin query strategies, for instance, indicate that with
prospective active learning, achieving a test accuracy of over $90\%$ can be
achieved as early as rounds 9 or 10.

In Figure 6, we note that prospective active learning less consistently
outperforms retrospective active learning for some query strategies, such as
Random and Coreset. Potential reasons for the less consistent regions in
performance accuracy come from the nature of the query strategies themselves.
For example, Coreset relies on a learned representation of the data to query
from; thus, performance in the beginning can be low because it may be
utilizing poor representations of data to decide which samples to query
(muthakana2019uncertainty, 19). Figure 2 demonstrates that in retrospective
active learning, Coreset is mainly querying from future data samples, thus
allowing the model to access more diverse data, which results in an initial
performance increase over prospective active learning.

Other reasons for regions of lower accuracy when compared to the retrospective
setting come from the concept of catastrophic forgetting, which occurs when a
model is trained on a set of data samples but loses its ability to perform
well on this set when new samples are added. Catastrophic forgetting is
characterized by the negative fliprate (NFR) of a model, described as summing
the total number of model predictions that changed from correct to incorrect
after new data was added to the training set (yan2021positive, 32). We show
NFR results for Random and Coreset in Figure 8.

Figure 8. NFR for Coreset and Random (High NFR indicates more test samples
forgotten) Figure 9. Retrospective active learning (RAL) vs Prospective active
learning (PAL) test accuracy results averaged over 5 seeds for dynamic test
set.

For Figure 8, we commonly observe spikes in the NFR values for the prospective
active learning experiments, as well as the retrospective case. In many cases,
this appears to correlate with regions of lower accuracy noticed in Figure 6.
For example, when analyzing the Coreset query strategy for the prospective
case in Figure 6, at Round 11 we observe lower accuracy than what was observed
in previous rounds, which correlates to Figure 8’s NFR spike for the Coreset
query strategy at Round 11. This same correlation is seen in other query
strategies, like Random at Round 8 and 12 for the prospective active learning
curves. Note that this same trend is present in the retrospective case. For
example, Coreset at Round 14 experiences both a region of low test accuracy
and a spike in NFR values. Therefore, even though the nature of some of these
query strategies may can cause inconsistent regions of accuracy as discussed
previously, there appears to be a strong correlation between NFR spikes and
test accuracy performance deterioration. It is likely that prospective active
learning suffers from areas of high NFR due to the fact that it is being
tested on the fixed test set, which contains data from visits the model has
not been trained on yet. Therefore, this may cause inconsistency in
performance. However, despite some inconsistent regions of accuracy,
prospective active learning appears to consistently outperform retrospective
active learning after the ninth round of training, demonstrating better
generalization capability on the fixed test set.

###  5.3. Dynamic Test Set Results

####  5.3.1. Retrospective vs Prospective Active Learning

We present a comparison between retrospective and prospective active learning
for the same six query strategies for the dynamic test set, shown in Figure 9.
When analyzing the graphs, it becomes apparent that prospective active
learning typically outperforms retrospective active learning for most query
strategies in terms of test accuracy. This change is even more noticeable when
comparing the test accuracy results for the fixed test set in Figure 6, where
the results in Figure 9 show more consistent regions of high accuracy. This
change is most likely due to the fact that the dynamic test set only contains
data from visits that the model has been trained on, whereas the fixed test
set contains data from visits the model has not learned yet.

Query strategies like Entropy and BADGE noticeably perform much better than
retrospective active learning in Figure 9. BADGE, for instance, achieves over
$90\%$ accuracy as early as Round 9 using 2,432 training samples. When this
same query strategy is applied in the retrospective setting, however, the test
accuracy does not surpass $85\%$ throughout all 20 rounds. A similar trend is
present in the Entropy plot, where the prospective setting stabilizes to a
test accuracy of approximately $90\%$ during later rounds. Meanwhile, the
retrospective setting fluctuates frequently in test accuracy during later
rounds, appearing to be less stable than the prospective case. Other query
strategies like Coreset and Least Confidence exhibit similar performance.

Overall, these results demonstrate that even though retrospective active
learning has the entirety of the dataset to select data from, it does not
generalize as well to visit-specific data. Prospective active learning, on the
other hand, is able to achieve higher accuracy than the retrospective setting
by sampling images in a sequential manner, imitating the manner in which these
images were collected. These results further validate the claim that the
active learning process for clinical trial data must be viewed from a
prospective lens.

##  6\. Conclusion

In this work, we propose a prospective active learning framework applied to
clinical trials where the collected data is known to have interventional and
time dependencies. We analyze this method’s performance against traditional
retrospective active learning. We conclude that by conditioning on the time
(or visit, in the case of a clinical trial) of which an image was collected,
we enforce the i.i.d. assumption that most active learning query strategies
adhere by. Because of this, we observe an increase in performance on two
different types of test sets: a fixed test set and dynamic test set. The fixed
test set tests overall generalization performance, as it is composed of OCT
images from various visits. The dynamic test set expands each training round
and examines current and past visit generalization, as it is contains up to
the current visit of OCT images the model has been trained on. We use NFR as
an additional metric to examine how much information is retained over time,
comparing the NFR results to what is observed with the accuracy plots.
Furthermore, we perform experiments to understand the impact of the order in
which data from visits is sampled. We conclude that when analyzing data
collected from a clinical trial, active learning must be performed in a
prospective manner, as sampling sequentially is what allows the model to learn
the interventions across the training data. In addition, our method mirrors
how data is collected in a clinical trial setting, thus introducing more
realistic sampling. Future directions for this type of research include
further validating this framework on other clinical trial datasets beyond OCT,
as well as considering the effect of image alignment within these datasets
across patients’ visits. Furthermore, incorporating a healthy patient
population into the datasets may provide additional insights into the
performance of prospective active learning, where our current understanding
expects prospective and retrospective active learning to provide similar
results if minor changes are observed in the healthy cohort across visits. In
addition, the effect of different model architectures and query sizes can be
further investigated.

##  7\. Acknowledgements

This material is based upon work supported by the National Science Foundation
Graduate Research Fellowship under Grant No. DGE-2039655. Any opinion,
findings, and conclusions or recommendations expressed in this material are
those of the authors(s) and do not necessarily reflect the views of the
National Science Foundation.

## References

  * (1) Jordan T Ash et al.  “Deep batch active learning by diverse, uncertain gradient lower bounds”  In _arXiv preprint arXiv:1906.03671_ , 2019
  * (2) Patrick Bangert et al.  “Active learning performance in labeling radiology images is 90% effective”  In _Frontiers in radiology_ 1 Frontiers Media SA, 2021, pp. 748968
  * (3) Ryan Benkert et al.  “Gaussian Switch Sampling: A Second Order Approach to Active Learning”  In _IEEE Transactions on Artificial Intelligence_ IEEE, 2023
  * (4) Isabel Chien et al.  “Multi-disciplinary fairness considerations in machine learning for clinical trials”  In _2022 ACM Conference on Fairness, Accountability, and Transparency_ , 2022, pp. 906–924
  * (5) Amit Choudhary et al.  “A Deep Learning-Based Framework for Retinal Disease Classification”  In _Healthcare_ 11.2, 2023, pp. 212  MDPI
  * (6) Ling Dai et al.  “A deep learning system for detecting diabetic retinopathy across the disease spectrum”  In _Nature communications_ 12.1 Nature Publishing Group UK London, 2021, pp. 3242
  * (7) Bradley J Erickson, Panagiotis Korfiatis, Zeynettin Akkus and Timothy L Kline  “Machine learning for medical imaging”  In _Radiographics_ 37.2 Radiological Society of North America, 2017, pp. 505–515
  * (8) Anne M Euser, Carmine Zoccali, Kitty J Jager and Friedo W Dekker  “Cohort studies: prospective versus retrospective”  In _Nephron Clinical Practice_ 113.3 Karger Publishers, 2009, pp. c214–c217
  * (9) Ibrahim M El-Hasnony, Omar M Elzeki, Ali Alshehri and Hanaa Salem  “Multi-label active learning-based machine learning model for heart disease prediction”  In _Sensors_ 22.3 MDPI, 2022, pp. 1184
  * (10) Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun  “Deep residual learning for image recognition”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778
  * (11) Steven CH Hoi, Rong Jin, Jianke Zhu and Michael R Lyu  “Batch mode active learning and its application to medical image classification”  In _Proceedings of the 23rd international conference on Machine learning_ , 2006, pp. 417–424
  * (12) Kiran Kokilepersaud et al.  “Clinically Labeled Contrastive Learning for OCT Biomarker Classification”  In _IEEE Journal of Biomedical and Health Informatics_ IEEE, 2023
  * (13) Kiran Kokilepersaud et al.  “Gradient-based severity labeling for biomarker classification in oct”  In _2022 IEEE International Conference on Image Processing (ICIP)_ , 2022, pp. 3416–3420  IEEE
  * (14) Punit Kumar and Atul Gupta  “Active learning query strategies for classification, regression, and clustering: a survey”  In _Journal of Computer Science and Technology_ 35 Springer, 2020, pp. 913–945
  * (15) Yash-yee Logan, Ryan Benkert, Ahmad Mustafa and Ghassan AlRegib  “Patient aware active learning for fine-grained oct classification”  In _2022 IEEE International Conference on Image Processing (ICIP)_ , 2022, pp. 3908–3912  IEEE
  * (16) Yash-yee Logan et al.  “Multi-modal learning using physicians diagnostics for optical coherence tomography classification”  In _2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI)_ , 2022, pp. 1–5  IEEE
  * (17) Yash-yee Logan, Mohit Prabhushankar and Ghassan AlRegib  “Decal: Deployable clinical active learning”  In _arXiv preprint arXiv:2206.10120_ , 2022
  * (18) Robert J McDonald et al.  “The effects of changes in utilization and technological advancements of cross-sectional imaging on radiologist workload”  In _Academic radiology_ 22.9 Elsevier, 2015, pp. 1191–1198
  * (19) Hariank Muthakana  “Uncertainty and diversity in deep active image classification”, 2019
  * (20) Mohit Prabhushankar et al.  “Olives dataset: Ophthalmic labels for investigating visual eye semantics”  In _Advances in Neural Information Processing Systems_ 35, 2022, pp. 9201–9216
  * (21) Bernhard Schölkopf et al.  “Toward causal representation learning”  In _Proceedings of the IEEE_ 109.5 IEEE, 2021, pp. 612–634
  * (22) Ozan Sener and Silvio Savarese  “Active learning for convolutional neural networks: A core-set approach”  In _arXiv preprint arXiv:1708.00489_ , 2017
  * (23) Burr Settles  “Active learning literature survey”  University of Wisconsin-Madison Department of Computer Sciences, 2009
  * (24) Asim Smailagic et al.  “Medal: Deep active learning sampling method for medical image analysis”  In _arXiv preprint arXiv:1809.09287_ , 2018
  * (25) Dogancan Temel, Melvin J Mathew, Ghassan AlRegib and Yousuf M Khalifa  “Relative afferent pupillary defect screening through transfer learning”  In _IEEE Journal of Biomedical and Health Informatics_ 24.3 IEEE, 2019, pp. 788–795
  * (26) Takumasa Tsuji et al.  “Classification of optical coherence tomography images using a capsule network”  In _BMC ophthalmology_ 20.1 BioMed Central, 2020, pp. 1–9
  * (27) Dan Wang and Yi Shang  “A new active labeling method for deep learning”  In _2014 International joint conference on neural networks (IJCNN)_ , 2014, pp. 112–119  IEEE
  * (28) Xiaosong Wang et al.  “Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2097–2106
  * (29) Zheng Wang and Jieping Ye  “Querying discriminative and representative samples for batch mode active learning”  In _ACM Transactions on Knowledge Discovery from Data (TKDD)_ 9.3 ACM New York, NY, USA, 2015, pp. 1–23
  * (30) Jiqing Wu et al.  “Towards IID representation learning and its application on biomedical data”  In _arXiv preprint arXiv:2203.00332_ , 2022
  * (31) Nan Wu et al.  “Deep neural networks improve radiologists’ performance in breast cancer screening”  In _IEEE transactions on medical imaging_ 39.4 IEEE, 2019, pp. 1184–1194
  * (32) Sijie Yan et al.  “Positive-congruent training: Towards regression-free model updates”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 14299–14308


# Large magnetoresistance in type-$\rm{II}$ Weyl semimetal WP2

Aifeng Wang,1 D. Graf,2 Yu Liu,1 Qianheng Du,1,3 Jiabao Zheng,4,5 Hechang
Lei,6 and C. Petrovic1,3 1Condensed Matter Physics and Materials Science
Department, Brookhaven National Laboratory, Upton, New York 11973, USA
2National High Magnetic Field Laboratory, Florida State University,
Tallahassee, Florida 32306-4005, USA
32Department of Materials Science and Chemical Engineering, Stony Brook
University, Stony Brook, New York 11790, USA
4Department of Electrical Engineering, Columbia University, New York, New York
10027,USA
5Department of Electrical Engineering and Computer Science, Massachusetts
Institute of Technology, Cambridge, Massachusetts 02139, USA
6Department of Physics and Beijing Key Laboratory of Opto-electronic
Functional Materials and Micro-nano Devices, Renmin University of China,
Beijing 100872, People’s Republic of China

###### Abstract

We report magnetotransport study on type-II Weyl semimetal WP2 single
crystals. Magnetoresistance (MR) exhibits a nonsaturating $H^{n}$ field
dependence (14,300% at 2 K and 9 T) whereas systematic violation of Kohler’s
rule was observed. Quantum oscillations reveal a complex multiband electronic
structure. The cyclotron effective mass close to the mass of free electron me
was observed in quantum oscillations along $b$-axis, while reduced effective
mass of about 0.5$m_{e}$ was observed in $a$-axis quantum oscillations,
suggesting Fermi surface anisotropy. Temperature dependence of the resistivity
shows a large upturn that cannot be explained by the multi-band
magnetoresistance of conventional metals. Even though crystal structure of WP2
is not layered as in transition metal dichalcogenides, quantum oscillations
suggest partial two-dimensional character.

###### pacs:

72.20.My, 72.80.Jc, 75.47.Np

##  I INTRODUCTION

Weyl fermions originate from Dirac states by breaking either time-reversal
symmetry or space-inversion symmetry.Dirac ; WanX ; Balents ; Soluyanov ;
VafekO ; BansilA ; HasanM ; JiaS ; BurkovA ; LiuJ ; ZhengH ; DaiX  In addition
to type-$\rm{I}$Balents ; BansilA ; LiuJ  Weyl points with a close point-like
Fermi surface, novel type-$\rm{II}$Soluyanov ; ZhengH  Weyl fermions appear at
the boundaries between electron and hole pockets violating Lorentz invariance.
This results in an open Fermi surface and anisotropic chiral anomaly. Most of
the reported type-$\rm{II}$ Weyl semimetals exhibit characteristic
extraodinary magnetoresistance (XMR), observed for example in materials that
crystallize in C2/m space group of OsGe2-type structure, such as NbSb2, TaAs2,
and NbAs2.WangKF ; YuanZ

Figure 1: (Color online). (a) Powder X-ray diffraction pattern of WP2, inset
shows the photograph of as grown single crystal. (b) Temperature-dependent
transversal resistivity for WP2 in different magnetic fields. Inset shows
$\rho$(0T) fitted with Bloch-Grüneisen model. Heat capacity of WP2 (c,d).
The($\partial\rho/\partial T$) plotted as a function of temperature (e);
minima and sign change points in the $\partial\rho/\partial T$ are defined as
$T_{m}$ and $T_{i}$, respectively. Magnetic field dependence of $T_{m}$ and
$Ti$ (f).

WP2 crystallizes the nonsymmorphic space group Cmc21(36), which favors new
topological phases with Dirac states.Rundquist ; YoungS ; BzdusekT ; ZhaoYX
Indeed, WP2 was recently predicted to be a type-$\rm{II}$ Weyl semimetal with
four pairs of type-$\rm{II}$ Weyl points and long topological Fermi arcs.Autes
In this work, we have successfully grown single crystals of WP2, and have
investigated its magnetotransport properties. Magnetoresistance (MR =
[$\rho$(B) - $\rho$(0)]/$\rho$(0)$\times$100%) exhibits a nonsaturating
$\sim$$H^{1.8}$ field dependence up to 14,300% at 2 K and 9 T. WP2 shows a
systematic violation of Kohler’s rule and large magnetic field-induced
nonmetallic resistivity that cannot be explained by orbital magnetoresistance
of multiband metals. Fermi surface studies of of WP2 by quantum oscillations
reveal complex electronic structure. The $T-H$ phase diagram is consistent
with the universal phase diagram for extraordinary magnetoresistance (XMR)
materials,TaftiPNAS  however the Weyl points in WP2 have little influence on
the effective mass and mobility of the Fermi surfaces detected by quantum
oscillations.

##  II EXPERIMENTAL DETAILS

Single crystals of WP2 were grown by chemical vapor transport method using I2
as transport agent. First, WP2 polycrystals were obtained by annealing
stoichiometric W and P powder at 500 ∘C for 24 h, and then 750 ∘C for 48 h.
WP2 polycrystals were mixed with I2 (15 mg/ml), and then sealed in an
evacuated quartz tube. Single crystals were grown in a temperature gradient
1180 ∘C (source) to 1050 ∘C (sink) for a month. Shiny needle-like single
crystal with typical size 1.5 mm $\times$ 0.05 mm $\times$ 0.05 mm were
obtained. X-ray diffraction (XRD) measurements were performed using a Rigaku
Miniflex powder diffractometer. The element analysis was performed using an
energy-dispersive x-ray spectroscopy (EDX) in a JEOL LSM 6500 scanning
electron microscope. Heat capacity and magnetotransport measurement up to 9 T
were conducted in a Quantum Design PPMS-9. Magnetotransport measurement at
high magnetic fields up to 18 T were conducted at the National High Magnetic
Field Labortory (NHMFL) in Tallahassee. Resistivity was measured using a
standard four-probe configuration. Hall resistivity was measured by the four-
terminal technique by switching the polarity of the magnetic field to
eliminate the contribution of $\rho_{xx}$ due to the misalignment of the
voltage contacts.

##  III RESULTS AND DISCUSSIONS

Figure 2: (Color online). Magnetic field dependence of MR at various
temperatures (a) and magnetoresistance as a function of the titled angle from
the applied field (b). Dashed line is the fitting using the 2D model. Kohler
plot at different temperatures using $\Delta\rho$/$\rho(0)$ = $F[H/\rho(0)]$
$\simeq$ $H^{1.8}$ (c). MR versus temperature for WP2 at 9 T (left) and
temperature dependence of $A$ and $n$ (MR = $A\times H^{n}$) (right) (d).

Figure 1(a) shows the powder X-ray diffraction pattern of WP2. The data can be
fitted quite well by the space group Cmc21 with lattice parameter $a$ =
0.3164(2) nm, $b$ = 1.184(2) nm, and $c$ = 0.4980(2) nm, consistent with
previous report.Rundquist  The average atomic ratio determined by the EDX is
W: P = 1: 2. Figure 1(b) shows the temperature dependence of resistivity in
WP2 in different fields. The residual resistivity ratio (RRR) in the absence
of magnetic field is $\rho(300K)$/$\rho(2K)$ = 279 with $\rho(2K)$ = 0.19
$\mu\Omega$cm, indicating small defect scattering in the crystal. The metallic
resistivity in $H$ = 0 can be described by the Bloch-Grüneisen (BG)
model:Ziman

$\rho(T)=\rho_{0}+C(\frac{T}{\Theta_{D}})^{5}\int_{0}^{\frac{\Theta_{D}}{T}}\frac{x^{5}}{(e^{x}-1)(1-e^{-x})}dx$

where $\rho_{0}$ is the residual resistivity and $\Theta_{D}$ is the Debye
temperature, indicating phonon scattering in the whole temperature range. The
fitting results in $\Theta_{D}$ = 546(5) K. As shown in Fig. 1(b), with the
application of the magnetic field, the $\rho$(T) shows a large upturn which
saturates below $\sim$ 15 K. Similar resistivity behaviour was observed
recently in XMR materials such as WTe2, TaAs, LaSb, PtSn4, ZrSiSe, Ta3S2, but
also in graphite, bismuth, and MgB2.Tafti ; Hechang ; Cava ; HuangX ; MunE ;
Hosen ; ChenD ; Soule ; ZhuZ ; BudkoS  Heat capacity of WP2, shown in Fig.
1(c), is best described at low temperatures with a $C(T)/T=\gamma+\beta
T^{2}+\delta T^{4}$ where $\gamma$ = 1.50(5) mJ mol-1 K-1, $\beta$ = 0.0324(5)
mJ mol-1 K-2 and $\delta$ = 3.0(2) $\times$ 10-5 mJ mol-1 K-4 [Fig.
1(d)].Cezairliyan  The Debye temperature $\Theta_{D}=564(\pm 11)$ K is
obtained from $\Theta_{D}=(12\pi^{4}NR/5\beta)^{1/3}$ where N is the atomic
number in the chemical formula and R is the gas constant.

Based on the field dependence of $\partial\rho/\partial T$ [Fig. 1(e)] we plot
the temperature-field ($T-H$) phase diagram in Fig. 1(f), where $T_{m}$ and
$T_{i}$ are taken as the sign change point and minimum in
$\partial\rho/\partial T$. The phase diagram is consistent with extreme
magnetoresistance materials.TaftiPNAS

Figure 3: (Color online). (a) The magnetic field dependence of Hall
resistivity $\rho_{xy}$ at various temperatures. The temperature dependence of
carrier densities (b) and mobilities (c) of electrons and holes obtained by
fitting using two band model. (d) Measured temperature dependence of $\rho$
(T, B) at various field (solid lines), and $\rho$ (T, B) calculated using the
temperature dependence of $n_{e,h}$ and $\mu_{e,h}$ derived from the two-band
model (scatter symbols).

Figure 2(a) shows the field dependence of transverse magnetoresistance MR =
[$\rho$(B) - $\rho$(0)]/$\rho$(0)$\times$100% at various temperatures with
B//$b$. MR exhibits a non-saturating $\sim$$H^{1.8}$ dependence and reaches
14,300% at 2 K and 9 T. It decreases quickly with increasing temperature, and
becomes negligibly small above 100 K. The MR of WP2 is about one order of
magnitude smaller than in WTe2 and LaSb, but comparable even larger than those
in LaBi, Ta3S2, and TaIrTe4.Cava ; Tafti ; Hechang ; ChenD ; Khim  The MR of
solids only responds to the extremal cross section of the Fermi surface along
the field direction. Thus, angular dependence of MR in a material with a 2D
Fermi surface should be proportional to $\mid$cos($\theta$)$\mid$.Shoeneberg
As shown in Fig. 2(b), MR in different magnetic fields show two fold symmetry
which can be fitted well with $\mid$cos($\theta$)$\mid$, indicating a 2D Fermi
surface.

Field induced nonmetallic resistivity and XMR in topological materials has
been discussed lately in the literature; however its origin was ascribed to
different mechanisms. In WTe2 it was suggested that the band structure of a
compensated semimetal with perfect electron-hole symmetry is important whereas
in LaBi and LaSb the proposed mechanism involves a combination of compensated
electron-hole pockets and particular orbital texture on the electron
pocket.TaftiPNAS ; WangL ; NiuXH ; WuY ; NayakJ  There are also evidences for
exotic and multiple surface Dirac states in other materials. MR in PtSn4 has
been considered within the model of orbital MR in the long mean-free path
metals with single dominant scattering time.MunE  However, Dirac node arcs are
also found in PtSn4 and ZrSiSe(Te), suggesting that XMR could be connected
with suppressed backscattering of Dirac states.WuY2 ; MaoZQ

Figure 4: (Color online). Transverse resistivity vs field ($B//c$) at
different temperatures (a); inset shows the enlarged part in high magnetic
field. Magnetization ($B//a$) measured up to 18 T at various temperatures (b).
SdH (c), dHvA (d) oscillatory components obtained by smooth background
subtraction. FFT spectra for SdH (e)and dHvA (f), respectively. Temperature
dependence of oscillating amplitude at different frequencies for SdH(g) and
dHvA(h), respectively, Solid lines are fitted using Lifshitz-Kosevich formula.

Semiclassical transport theory based on the Boltzmann equation predicts
Kohler’s rule $\Delta\rho/\rho$(0) = $F[H/\rho(0)]$ to hold if there is a
single type of charge carrier and scattering time in a metal.Pippard
Violation of Kohler’s rule is common in XMR materials, such as LaBi, TaAs,
TaAs2, NbAs2, NbSb2.Hechang ; ZhangCL ; YuanZ ; WangKF  As shown in Fig.2(c),
MR in WP2 systematically deviates from Kohler’s rule above 10 K. The field
dependence of MR at different temperature can be well fitted with MR = $A$
$\times$ $H^{n}$, $n$ is 1.8 at 2 K. The exponent $n$ systematically decreases
to 1.5 with increasing temperature, as shown in Fig.2(d). The temperature
evolutions of $A$ and $n$ are similar to temperature dependence of MR(9T).

A common violation of Kohler’s rule in metals is multi-band electronic
transport i.e. existence of multiple scattering times. WP2 was predicted to be
a type-$\rm{II}$ Weyl semimetal; therefore there is a possibility that high
mobility bands with Dirac states contribute to small residual resistivity in
zero magnetic field and strong magnetoresistance. In the two-band model MR
is:SmithR

$\begin{array}[]{l}MR=\frac{n_{e}\mu_{e}n_{h}\mu_{h}(\mu_{e}+\mu_{h})^{2}(\mu_{0}H)^{2}}{{{{({\mu_{e}}{n_{h}}+{\mu_{h}}{n_{e}})}^{2}}+{{({\mu_{h}}{\mu_{e}})}^{2}}{{({\mu_{0}}H)}^{2}}{{({n_{h}}-{n_{e}})}^{2}}}}\end{array}$

For the compensated semimetal, where $n_{e}$ $\simeq$ $n_{h}$, we obtain MR =
$\mu_{e}\mu_{h}$$(\mu_{0}H)^{2}$. Since MR = $A$ $\times$ $H^{n}$ ($n$ =
1.5-1.8) [Fig. 2(c)] compensated two band model cannot completely explain the
MR. Next, we discuss electronic transport in WP2 from the perspective of two-
band model when $n_{e}$ $\neq$ $n_{h}$.

Figure 3(a) shows the field dependence of Hall resistivity $\rho_{xy}$ at
different temperatures. Clear quantum oscillation were observed at low
temperatures. The frequencies and effective mass derived from $\rho_{xy}$ are
consistent with that of $\rho_{xx}$, and we discuss this below. The
$\rho_{xy}$ can be fitted by two band model:

$\begin{array}[]{l}\frac{{{\rho_{xy}}}}{{{\mu_{0}}H}}=\mathop{R}\nolimits_{H}{\rm{}}=\frac{1}{e}\frac{{(\mu_{h}^{2}{n_{h}}-\mu_{e}^{2}{n_{e}})+{{({\mu_{h}}{\mu_{e}})}^{2}}{{({\mu_{0}}H)}^{2}}({n_{h}}-{n_{e}})}}{{{{({\mu_{e}}{n_{h}}+{\mu_{h}}{n_{e}})}^{2}}+{{({\mu_{h}}{\mu_{e}})}^{2}}{{({\mu_{0}}H)}^{2}}{{({n_{h}}-{n_{e}})}^{2}}}}\end{array}$

Where $n_{e}$ ($n_{h}$) and $\mu_{e}$ ($\mu_{h}$) denote the carrier
concentrations and mobilities of electron and hole, respectively. We find that
$\rho_{xy}$ can be well fitted by the two-band model. The obtained $n_{e}$
($n_{h}$) and $\mu_{e}$ ($\mu_{h}$) by fitting are shown in Fig. 3(b) and
3(c), respectively. WP2 shows relatively high carrier density and the dominant
carrier is electron-like. The mobility is similar to other Dirac materials
such as $A$MnBi2 ($A$ = Sr,Ca), LaBi and doped WTe2.KefengCa ; KefengSr ;
Hechang ; KrienerM  The large carrier density indicates that WP2 is different
from a compensated semimetal. We calculate the $\rho\;(T,B)$ using carrier
concentrations $n_{e}$ ($n_{h}$) and mobilities $\mu_{e}$ ($\mu_{h}$) obtained
by two band model fitting of $\rho_{xy}$, as shown in in Fig.3(d). A similar
temperature dependence between calculated and measured MR argues in favor of
the validity of multicarrier electronic transport [Fig. 3(d)]. However, in
contrast to LaBi, the $\rho\;(T,B)$ from two-band orbital magnetoresistance is
several orders of magnitude smaller than the measured $\rho\;(T,B)$. When
considering the multi-band behavior further fits of $\rho_{xy}$ using three-
band modelKimJS  reproduced $\rho_{xy}$ equally well. Moreover, nearly
identical MR is obtained. Therefore, in either case multiband MR is much
smaller than experiment results. This suggests possible contribution beyond
the conventional orbital magnetoresistance that amplifies MR.

Figure 5: (Color online). FFT spectra of the SdH oscillation with field
rotated in $bc$ plane (a) and FFT spectra of the dHvA with field rotated in
the $ab$ plane (b). Angular dependence of the oscillation frequency for SdH
(c), and dHvA (d). The spectra are nomalized and shifted vertically for
clearity, solid lines are fitted with 2D model 1/cos($\theta$)

MR for H//$b$, and cantilver for H//$a$ at different temperatures up to 18 T
are shown in Fig. 4(a) and 4(b). MR shows non-saturating $H^{n}$ up to 18 T,
consistent with result in Fig. 2. Clear oscillations were observed above 10 T.
The oscillatory components are plotted as a function of 1/B in Fig. 4(c) and
4(d). Both Fig. 4(c) and 4(d) exhibit beat patterns, indicating that multiple
frequencies contribute to oscillations. The fast Fourier transform (FFT)
spectra of the oscillatory components are shown in Fig. 4(e) and 4(f). Four
frequencies are observed in Fig. 4(e): 1384 T, 1799 T, 3183 T, and 3806 T.
There are several peaks around $F_{\alpha}$ = 1384 T, suggesting contribution
of several extrema. According the calculated Fermi surface of WP2,Autes ;
Kumar  $F_{\alpha}$ = 1384 T, and $F_{\beta}$ = 1799 T can be ascribed to hole
pockets, and $F_{\gamma}$ = 3183 T, and $F_{\eta}$ = 3806 T are from electron
pockets.

From the Onsager relation, $F=(\Phi_{0}/2\pi^{2})A_{F}$, where $\Phi_{0}$ is
the flux quantum and $A_{F}$ is the orthogonal cross-sectional area of the
Fermi surface, the Fermi surface is estimated to be 13 nm-2, 17 nm-2, 30 nm-2,
and 36 nm-2, corresponding to 5%, 7%, 12%, and 14% of the total area of the
Brillouin zone in the $ac$ plane. Assuming the circular cross section of Fermi
surface $A_{F}$ = $\pi$$k_{F}^{2}$, the band splitting induced by spin-orbital
coupling can be estimated to be $k_{\gamma}$ \- $k_{\eta}$ = 0.03
$\rm{\AA}$-1, which is larger than that in MoP, and about half of that in
giant Rashba effect material BiTeI.Shekhar ; Ishizaka  Strong spin-orbital
coupling in WP2 indicates that spin texture, which could forbid
backscattering, might play an important role in large magnetoresistance,
similar to that in WTe2.JiangJ  For dHvA oscillation along $a$ axis, the Fermi
surfaces from three frequencies 1217 T, 2949 T, and 3417 T are 12 nm-2, 28
nm-2, and 32 nm-2, corresponding to 18%, 42%, and 48% of the Brillouin zone.

In order to obtain the cyclotron mass for the main frequencies, the FFT
amplitude $A$ versus temperature was fitted using Lifshitz-Kosevich
formula,Shoeneberg  $A$$\sim$[$\alpha$$m^{*}$(T/B)/$\sinh$($\alpha$
$m^{*}$T/B)] where $\alpha$ = 2$\pi$2$k_{\rm B}$/e$\hbar$ $\approx$ 14.69 T/K,
$m^{*}$ = $m$/$m_{e}$ is the cyclotron mass ratio ($m_{e}$ is the mass of free
electron). As shown in Fig. 4(e) and 4(f), the cyclotron mass along $b$ axis
is close to the mass of free electron $m_{e}$, while cyclotron mass along $a$
axis is reduced to about 0.5$m_{e}$, which can be attributed to anisotropy of
the Fermi surface. We estimate the Dingle temperatures from the FFT frequency
with highest FFT peaks to be 2.4 K and 4.7 K for H//$b$ and H//$a$,
respectively. The corresponding scattering times are 5.0 $\times$ 10-13 s and
2.6 $\times$ 10-13 s. Then, the mobility estimated by $\mu_{q}$ =
$e\tau_{q}/m_{c}$ is 880 cm2V-1s-1 and 458 cm2V-1s-1, comparable to Fig. 3(c).
The effective mass and mobility suggest that the Fermi surfaces detected by
quantum oscillation are not under significant influence of the due the the
Weyl points which are located below the Fermi level.Autes

Angle dependence of SdH in $bc$ plane, and dHvA in $ab$ plane provides further
insight into the shape of the Fermi surface. The FFT spectra of the SdH is
presented in Fig. 5 (a), and dHvA in Fig. 5(b). For the SdH measurement, four
main peaks are observed, the FFT peaks increases with the angle tilt from
zero, and disappear above 60o. Solid lines in Fig.5(c) are fitted using 2D
Fermi surface [$F(0)/cos(\theta)$]. Hence low angle quantum oscillations
reveal quasi-2D character at low angles below 50o. When the field is rotated
in the $ab$ plane, with the field titled from $a$ axis, the frequency
decreases with the angle, indicating the enlongated Fermi surface in the $bc$
plane. The angle dependence of quantum oscillations agree well with the
calculated Fermi surface.Autes ; Kumar  The calculated Fermi surfaces of WP2
consist of spaghetti-like hole Fermi surfaces and bow-tie-like electron Fermi
surfaces, while the presence of strong spin-orbit coupling leads to splitting
of Fermi surfaces.Autes ; Kumar  The spaghetti-like hole Fermi surface extends
along $b$ axis, bends along $a$ axis, while it is flat along $c$ axis,
consistent with the quasi-2D enlonged $\alpha$ and $\beta$ bands in $bc$
plane. When magnetic field is applied in small angles around $b$ axis, SdH
oscillations of $\gamma$ and $\eta$ bands are from the orbits across whole
electron pockets in $ac$ plane, quasi-2D behavior of $\gamma$ and $\eta$ bands
can be then attributed to relatively flat wall of electron pockets. Moreover,
the appearance in pair of both the hole and electron Fermi surfaces with
almost same angle dependence might be the result of band splitting effect
which is induced by strong spin orbit coupling.

##  IV CONCLUSIONS

In conclusion, magnetotransport studies of WP2 confirm the presence of
multiple bands.Autes  The large increase of resistivity in magnetic field
cannot be explained by the multiband orbital magnetoresistance but the
effective mass and mobility detected by quantum oscilllations are not under
significant influence of Weyl points. Strong spin-orbit coupling effect was
observed in dHvA whereas the temperature and angle dependence of dHvA
measurements reveal anisotropic multiband characteristics and are in agreement
with calculations. Even though crystal structure in WP2 is not layered as in
transition metal dichalcogenides, Fermi surfaces do show partial quasi-2D
character.

## Acknowledgements

We thank John Warren for help with EDX measurements and Rongwei Hu for useful
discussions. Work at BNL was supported by the U.S. DOE-BES, Division of
Materials Science and Engineering, under Contract No. DE-SC0012704. Work at
the National High Magnetic Field Laboratory is supported by the NSF
Cooperative Agreement No. DMR-1157490, and by the state of Florida.

_Note added._ During the review of this paper, a related study on WP2 was
reported by R. Schönemann _et al_.,Schonemann  though the magnitude of
magnetoresistance and RRR of the crystal is somewhat different, the Fermi
surface and main conclusion is consistent with ours.

## References

  * (1) T.O. Wehling, A.M. Black-Schaffer and A.V. Balatsky, Advances in Physics 63, 1 (2014).
  * (2) Xiangang Wan, Ari M. Turner, Ashvin Vishwanath, and Sergey Y. Savrasov, Phys. Rev. B 83, 205101 (2011).
  * (3) Leon Balents, Physics 4, 36 (2011).
  * (4) A. A. Soluyanov, D. Gresch, Z. Wang, Q. Wu. M. Troyer, X. Dai, and B. A. Bernevig, Nature 527, 495 (2015).
  * (5) Oskar Vafek and Ashvin Vishwanath, Annu. Rev. Condens. Matter Phys. 5, 83 (2014).
  * (6) A. Bansil, Hsin Lin and Tanmoy Das, Rev. Mod. Phys. 88, 021004 (2016).
  * (7) M.Z. Hasan and C. L. Kane, Rev. Mod. Phys. 82, 3045 (2010).
  * (8) Shuang Jia, Su-Yang Xu and M. Zahid Hasan, Nature Materials 15, 1140 (2016).
  * (9) A. A. Burkov, Nature Materials 15, 1145 (2016).
  * (10) Jianpeng Liu and David Vanderbilt, Phys. Rev. B 90 155316 (2014).
  * (11) Hao Zheng, Guang Bian, Guoqing Chang, Hong Lu, Su-Yang Xu, Guangqiang Wang, Tay-Rong Chang, Songtian Zhang, Ilya Belopolski, Nasser Alidoust, Daniel S. Sanchez, Fengqi Song, Horng-Tay Jeng, Nan Yao, Arun Bansil, Shuang Jia, Hsin Lin, and M. Z. Hasan, Phys. Rev. Lett. 117, 266804 (2016).
  * (12) Xi Dai, Nature Materials 15, 5 (2015).
  * (13) K. Wang, D. Graf, L. Li, L. Wang, and C. Petrovic, Sci. Rep. 4, 7328 (2014).
  * (14) Z. Yuan, H. Lu, Y. Liu, J. Wang, and S. Jia, Phys. Rev. B 93, 184405 (2016).
  * (15) S. Rundqvist and T. Lundstrom, Acta Chem. Scand. 17, 37 (1963).
  * (16) S. M. Young and C. L. Kane, Phys. Rev. Lett. 115, 126803 (2015).
  * (17) T. Bzdusek, QuaanSheng Wu, A. Rüegg, M. Sigrist and A. A. Soluyanov, Nature 538, 75 (2016).
  * (18) Y. X. Zhao and A. P. Schnyder, Phys. Rev. B 94, 195109 (2016).
  * (19) G. Autes, D. Gresch, M. Troyer, A. A. Soluyanov, and O. V. Yazyev, Phys. Rev. Lett. 117, 066402 (2016).
  * (20) F. F. Tafti, Q. Gibson, S. Kushwaha, J. W. Krizan, N. Haldolaarachchige, and R. J. Cava, Proc. Natl. Acad. Sci. USA 113, E3475 (2016).
  * (21) M. Ziman, Electrons and Phonons (Clarendon Press, Oxford, 1962).
  * (22) F. F. Tafti, Q. D. Gibson, S. K. Kushwaha, N. Haldolaarachchige, and R. J. Cava, Nature Phys. 12, 272 (2015).
  * (23) Shanshan Sun, Qi Wang, Peng-Jie Guo, Kai Liu, and Hechang Lei, New J. Phys. 18, 082002 (2016).
  * (24) M. N. Ali, J. Xiong, S. Flynn, J. Tao, Q. D. Gibson, L. M. Schoop, T. Liang, N. Haldolaarachchige, M. Hirschberger, N. P. Ong, R. J. Cava, Nature 514, 205 (2014).
  * (25) D. Chen, L. X. Zhao, J. B. He, H. Liang, S. Zhang, C. H. Li, L. Shan, S. C. Wang, Z. A. Ren, C. Ren, and G. F. Chen, Phys. Rev. B 94, 174411 (2016).
  * (26) X. Huang, L. Zhao, Y. Long, P. Wang, D. Chen, Z. Yang, H. Liang, M. Xue, H. Weng, Z. Fang, X. Dai, and G. Chen, Phys. Rev. X 5, 031023 (2015).
  * (27) M. M. Hosen, K. Dimitri, I. Belopolski, P. Maldonado, R. Sankar, N. Dhakal, G. Dhakal, T. Cole, P. M. Oppeneer, D. Kaczorowski, F. Chou, M. Z. Hasan, T. Durakiewicz, and M. Neupane, Phys. Rev. B 95, 161101(R) (2017).
  * (28) E. Mun, H. Ko, G. J. Miller, G. D. Samolyuk, S. L. Bud’ko, and P. C. Canfield, Phys. Rev. B 85, 035135 (2012).
  * (29) D. E. Soule, Phys. Rev. 112, 698 (1958).
  * (30) Z. Zhu, A. Collaudin, B. Fauqué, W. Kang, and K. Behnia, Nature Phys. 8, 89 (2011).
  * (31) S. L. Bud′ko, C. Petrovic, G. Lapertot, C. E. Cunningham,, P. C. Canfield, M.-H. Jung and A. H. Lacerda Phys. Rev. B 63, 220503 (2001).
  * (32) A. Cezairliyan, Specific Heat of Solids Hemisphere Pub. Corp., New York (1988).
  * (33) Seunghyun Khim, Klaus Koepernik, Dmitry V. Efremov, J. Klotz, T. Förster, J. Wosnitza, Mihai I. Sturza, Sabine Wurmehl, Christian Hess, Jeroen van den Brink, and Bernd Büchner, Phys. Rev. B 94, 165145 (2016).
  * (34) D. Shoeneberg, Magnetic $Oscillation$ $in$ $Metals$ (Cambridge University Press, Cambridge, 1984).
  * (35) Lin Wang, Ignacio Gutierrez-Lezama, Celine Barreteau, Nicolas Ubrig, Enrico Giannini Alberto F. Morpurgo, Nature Comms. 6, 8892 (2015).
  * (36) X. H. Niu, D. F. Xu, Y. H. Bai, Q. Song, X. P. Shen, B. P. Xie, Z. Sun, Y. B. Huang, D. C. Peets, and D. L. Feng, Phys. Rev. B 94, 165163 (2016).
  * (37) Yun Wu, Tai Kong, Lin-Lin Wang, D. D. Johnson, Daixiang Mou, Lunan Huang, Benjamin Schrunk, S. L. Bud’ko, P. C. Canfield, and Adam Kaminski, Phys. Rev. B 94, 081108 (2016).
  * (38) Jayita Nayak, Shu-Chun Wu, Nitesh Kumar, Chandra Shekhar, Sanjay Singh, Jörg Fink, Emile E.D. Rienks, Gerhard H. Fecher, Stuart S.P. Parkin, Binghai Yan and Claudia Felser, Nature Comms. 8, 13942 (2016).
  * (39) Yun Wu, Lin-Lin Wang, Eundeok Mun, D. D. Johnson, Daixiang Mou, Lunan Huang, Yongbin Lee, S. L. Bud′ko, P. C. Canfield and Adam Kaminski, Nature Physics 12, 667 (2016).
  * (40) J. Hu, Z. Tang, J. Liu, X. Liu, Y. Zhu, D. Graf, K. Myhro, S. Tran, C. N. Lau, J. Wei, and Z. Mao, Phys. Rev. Lett. 117, 016602 (2016).
  * (41) A. B. Pippard, $Magnetoresistanceinmetals$ (Cambridge University Press, Cambridge, 1989).
  * (42) C. L. Zhang, Z. Yuan, Q. D. Jiang, B. Tong, C. Zhang, X. C. Xie, and S. Jia, Phys. Rev. B 95, 085202 (2017).
  * (43) R. A. Smith, Semiconductors (Cambridge University Press, 1978).
  * (44) Kefeng Wang, D. Graf, Hechang Lei, S. W. Tozer, and C. Petrovic, Phys. Rev. B 84, 220401(R) (2011).
  * (45) Kefeng Wang, D. Graf, Limin Wang, Hechang Lei, S. W. Tozer, and C. Petrovic, Phys. Rev. B 85, 041101(R) (2012).
  * (46) M. Kriener, A. Kikkawa, T. Suzuki, R. Akashi, R. Arita, Y. Tokura, and Y. Taguchi Phys. Rev. B 91, 075205 (2015).
  * (47) J. S. Kim, J. Appl. Phys. 86, 3187 (1999).
  * (48) N. Kumar, Y. Sun, K. Manna, V. Süss, I. Leermakers, O. Young, T. Förster, M. Schmidt, B. Yan, U. Zeitler, C. Felser, C. Shekhar, arXiv: 1703.04527
  * (49) C. Shekhar, Y. Sun, N. Kumar, M. Nicklas, K. Manna, V. Sü$\beta$, O. Young, I. Leermakers, T. Förster, M. Schmidt, L. Muechler, P. Werner, W. Schnelle, U. Zeitler, B. Yan, S. S. P. Parkin, C. Felser, arXiv: 1703.03736.
  * (50) K. Ishizaka, M. S. Bahramy, H. Murakawa, M. Sakano, T. Shimojima, T. Sonobe, K. Koizumi, S. Shin, H. Miyahara, A. Kimura, K. Miyamoto, T. Okuda, H. Namatame, M. Taniguchi, R. Arita, N. Nagaosa, K. Kobayashi, Y. Murakami, R. Kumai, Y. Kaneko, Y. Onose, and Y. Tokura, Nat. Mater. 10, 521 (2011).
  * (51) J. Jiang, F. Tang, X. C. Pan, H. M. Liu, X. H. Niu, Y. X. Wang, D. F. Xu, H. F. Yang, B. P. Xie, F. Q. Song, P. Dudin, T. K. Kim, M. Hoesch, P. K. Das, I. Vobornik, X. G. Wan, and D. L. Feng, Phys. Rev. Lett. 115, 166601 (2015).
  * (52) R. Schönemann, N. Aryal, Q. Zhou, Y. -C. Chiu, K. -W. Chen, T. J. Martin, G. T. McCandless, J. Y. Chan, E. Manousakis, and L. Balicas, arXiv: 1706.10135.