# Automatic Seizure Detection Using the Pulse Transit Time

Eric Fiege<EMAIL_ADDRESS>1234-5678-9012 ,  Salima Houta
<EMAIL_ADDRESS>1234-5678-9012 ,  Pinar Bisgin
<EMAIL_ADDRESS>1234-5678-9012 Fraunhofer Institute for
Software and Systems EngineeringEmil-Figge-Str. 91DortmundGermany ,  Rainer
Surges<EMAIL_ADDRESS>Department of Epileptology University of Bonn
Medical CenterBonnGermany and  Falk Howar<EMAIL_ADDRESS>TU
Dortmund UniversityDortmundGermany

###### Abstract.

Documentation of epileptic seizures plays an essential role in planning
medical therapy. Solutions for automated epileptic seizure detection can help
improve the current problem of incomplete and erroneous manual documentation
of epileptic seizures. In recent years, a number of wearable sensors have been
tested for this purpose. However, detecting seizures with subtle symptoms
remains difficult and current solutions tend to have a high false alarm rate.
Seizures can also affect the patient’s arterial blood pressure, which has not
yet been studied for detection with sensors. The pulse transit time (PTT)
provides a noninvasive estimate of arterial blood pressure. It can be obtained
by using to two sensors, which are measuring the time differences between
arrivals of the pulse waves. Due to separated time chips a clock drift
emerges, which is strongly influencing the PTT. In this work, we present an
algorithm which responds to alterations in the PTT, considering the clock
drift and enabling the noninvasive monitoring of blood pressure alterations
using separated sensors. Furthermore we investigated whether seizures can be
detected using the PTT. Our results indicate that using the algorithm, it is
possible to detect seizures with a Random Forest. Using the PTT along with
other signals in a multimodal approach, the detection of seizures with subtle
symptoms could thereby be improved.

Epilepsy, Seizures, Epileptic Seizure Detection, Wearables, Pulse Transit Time
(PTT), Classification, Random Forest

††ccs: Computing methodologies Machine learning

##  1\. Introduction

Epilepsy is one of the most common neurological disorders worldwide and limits
the autonomy of those affected by recurrent and unpredictable epileptic
seizures. If epileptic seizures are associated with impaired consciousness and
loss of control over bodily functions, the consequences can be life-
threatening (e.g., failure of the respiratory center, accidents, suffocation).
Taking appropriate safety precautions in time helps to avoid serious
consequences or even sudden unexpected death in epilepsy (SUDEP). By using
technological solutions to improve monitoring (e.g., video cameras, pulse
oximeters) or accommodating relatives, SUDEP incidence in an epilepsy center
shows a decreasing trend between $1981$ and $2016$. Accurate seizure recording
also supports individualized drug therapy planning. Documentation is currently
done in paper or web-based calendars (e.g., EPI-Vista®) (Rabending and Runge,
2007). However, studies show that about $50\%$ of seizures are not documented
and about two-thirds of patients provide incorrect information (Blum et al.,
1996; Hoope et al., 2007). The main reasons for incorrect seizure
documentation include impaired perception of one’s own seizures, error-prone
seizure documentation by relatives or caregivers (Inan et al., 2009; Nijsen et
al., 2005). Sensor-based systems that can accurately detect epileptic seizures
are therefore of great importance and are being increasingly explored.

Seizures can affect brain regions that contribute to regulation of autonomic
body functions such as heart rate, respiratory rate, sweating, and blood
pressure (Moseley et al., 2013; Hampel et al., 2016; Nass et al., 2019). While
motor phenomena, heart rate changes, and sympathetic skin response have been
used to detect seizures (Vandecasteele et al., 2017; Poh et al., 2012) with
wearable sensors, the detection of seizures with blood pressure (BP) has not
been investigated. Current sensors do not use this information because
noninvasive measurement of BP is challenging. The PTT is directly related to
the BP and can be determined noninvasively (Hennig and Patzak, 2013). It
describes the time required for a pulse wave to arrive in a peripheral part of
the body. In this work we investigate if seizures could automatically be
detected by using the PTT.

##  2\. Related Work

The gold standard for seizure detection is a combination of video and
electroencephalography (video-EEG). An EEG monitor can reliably detect
seizures using analysis of brain signals (Nigam and Graupe, 2004; Tzallas et
al., 2009; Jaiswal and Banka, 2018). Intracranial EEG sensors provide the most
reliable signals but are very invasive, not suitable for all forms of
epilepsy, and are less well accepted by patients (Hoppe et al., 2015). Because
existing EEG measurement solutions are not practical for use in the patient’s
home, research on mobile solutions using other biomarkers has been ongoing for
several years. Depending on the seizure type, each of which is associated with
different accompanying symptoms, this research usually focuses on detecting a
group of similar seizure types. Different research works provide a literature
review of seizure detection devices and their effectiveness for different
seizure types (Ulate-Campos et al., 2016; Kurada et al., 2019). Seizures
involving the muscles can be well detected with acceleration data. (Beniczky
et al., 2013; Picard et al., 2016) Many present sensors are used for the
detection of motor seizures, using acceleration data. However, there are also
nonmotor seizures that cannot be detected with acceleration data. Seizures
also affect the autonomic nervous system and the processes controlled by it.
With EDA, HR or BP, therefore, there are other signals, that can be utilized
to detect seizures. The difficulty is that changes in these signals occur
frequently, without a seizure, in the everyday life of the patient, leading to
a high false-alert rate. Under the use of multiple signals the false alarms
may be decreased.

The effects on the cardiovascular system have been investigated in many
studies. Some seizures may affect the patient’s HR. In most cases, they result
in tachycardia, rarely in bradycardia. The rise in HR may take a few seconds
before the seizure is visible in the EEG. The time period varies depending on
the type of present epilepsy. In addition, one in five patients is known to
have irregularities in the EEG (Leutmezer et al., 2003; Opherk et al., 2002).
Studies have shown that seizures can affect also the BP (Hampel et al., 2016;
Nass et al., 2019). However, current sensors do not use this information
because noninvasive measurement of BP is still a major challenge. The PTT is
directly related to the BP and can be determined non invasively (Hennig and
Patzak, 2013). One approach to determine PTT is to measure the time between
cardiac contraction with an ECG and arrival in the peripheral part of the body
with an PPG sensor. In this work, the PPG signal was recorded from a sensor
placed in the patient’s inner ear. The PTT is calculated by subtracting the
start times of RR intervals from the ECG and PPG signals.

##  3\. Methods

We created a data processing pipeline to structure the process of data
refinement and to grant insight for validation. The processing steps can be
categorized into three major categories. At first, we loaded and prepared RR-
intervals of both ECG and PPG signals. A matching combination of those signals
will be called a ”case”. Next, we calculated and further improved the PTT for
all cases. Finally, we generated the data set, trained a Random Forest and
calculated quality measures. The following sections describe the clinical
study and the data basis. Furthermore we describe the mentioned major
categories in detail. Finally, we present the methods used to validate the
performance of the trained models.

###  3.1. Clinical Study

Clinical trials are being conducted at participating specialty clinics for
derivation of biosignal patterns and algorithm development and validation. A
total of $200$ patients were recruited to test the in-ear sensor. Video EEG,
ECG, and in-ear sensor data (PPG, HR, temperature, acceleration data) were
collected from these patients over an average period of four days. Based on
the video EEG, physicians recorded seizures that occurred (time period, type
of seizure). Through July $2019$, $552$ seizures have been recorded. The
collected data form the basis for the present work.

###  3.2. RR interval Preprocessing

To find the parallel recorded ECG and PPG signals, first we loaded all
metadata files into a database and queried the parallel recorded ECG and PPG
filenames from the database. Then we loaded the raw data from the files. The
raw data contains a lot of outliers so, as a first refinement step, we removed
data points, if their difference to the empirical mean deviates more than
twice of the standard deviation on the entire signal. The parallel recorded
ECG and PPG signals also do not start and stop at the same time. But for the
calculation of the PTT it is mandatory to have both signals. Therefore we
clipped the parallel recorded signals to the overlapping data. Figure 1 shows
the result of the RR interval preprocessing.

Some cases show a systematic error in the RR interval calculation, resulting
in a great difference between the ECG and PPG interval lengths and are
therefore removed. The condition for a removal is that the difference of their
empirical mean values is greater than a constant $k$. In a real world
application the model should not be evaluated if the PTT cannot be calculated
properly.

Figure 1. Result of RR interval preprocessing. The outliers have been removed
and only parallel recorded data has been kept. A seizure is marked by the
dashed orange line.

###  3.3. Calculating the PTT

The preprocessed RR intervals could be used to calculate the PTT. But there
must be considered an effect occurring due to the fact that each recording
device has had its own time chip. This leads to a temporal offset between the
ECG and PPG signals, increasing with the length of the recordings. This
circumstance is called clock drift and has a huge impact on the PTT. As a
result the calculation of the absolute PTT with the collected data is not
possible. Relative alterations on the other side can be reconstructed. To
reconstruct the relative alterations the clock drift has to be removed as good
as possible. This however always effects the PTT. It is therefore important to
have a good balance between removing the clock drift and keeping the
alterations in the PTT. If too much clock drift is removed also alterations in
the PTT are removed and seizures may no longer be detected. In contrast, if a
too small amount of clock drift is removed, the changes in the PTT are masked
by the clock drift.

In this work we developed two algorithms. The first algorithm (PTT Algorithm)
calculates the PTT on small time intervals and will ignore the emerging clock
drift between both signals. The second algorithm (Reactive Algorithm) uses the
first algorithm and tries to remove the clock drift on longer time intervals.
The output of this algorithm is a time series that responds to alterations in
the PTT.

The PTT Algorithm searches for matching ECG and PPG entries. Each PPG entry
$(t_{PPG},RR_{PPG})$, where $t_{PPG}$ is the occurrence time of the PPG RR
interval and $RR_{PPG}$ is the length of the PPG RR interval, which follows
directly on an ECG entry $(t_{ECG},RR_{ECG})$ will be used to calculate the
$i$-th PTT entry as follows:

$PTT_{i}\coloneqq(t_{PTT},PTT)=(t_{ECG},t_{PPG}-t_{ECG})$

If the resulting PTT is greater than $RR_{ECG}$ the resulting PTT entry is
discarded. With perfectly aligned data tracks and without clock drift this
algorithm would calculate the absolute PTT.

The idea behind the Reactive Algorithm is to shift the start times of all PPG
entries by an offset $o$, in such way that for a limited time interval the
calculated PTT values are located in the middle of the ECG RR intervals. For
each ECG entry the following next few minutes of data are selected as a window
and the PTT is calculated with the PTT Algorithm. For a new PTT data point,
the mean value of the calculated PTT values on the window is used. The data
point is only used, if $20\%$ of the expected ECG- and PPG intervals are
available. After each calculation, the offset of the PPG time series is
adjusted. This is accomplished by subtracting the new PTT data point value
from the half of the mean of the ECG RR interval length. The mean value
$ecg\\_mean\\_rr\\_interval$ of the ECG RR interval is based on the entire
length of the time series, whereas the new PTT data point value $w_{PTT}$ is
calculated on a window of a few minutes. The PPG-offset is than given by:

$o\coloneqq\frac{ecg\\_mean\\_rr\\_interval/2-w_{PTT}}{c}.$

The constant $c$ can be used to control how much clock drift is removed. With
an increasing $c$ the algorithm removes less clock drift. All in all it is
ensured that the PPG start times are as centered as possible between the ECG
start times for all windows.

Figure 2. PTT-alterations calculated by the Reactive Algorithm for a CPS
seizure. The green line shows the added PPG-offset during the calculation.

###  3.4. Feature Generation and Training of Models

For those cases that contain a seizure, there was exactly one training pair
generated. Cases without seizure where used to create as many non-overlapping
training pairs as possible. We created the training pairs on a fixed time
interval and divided this time interval into windows. The window count can be
set to two or three windows. In addition, there may be added a time shift
$o_{s}$ for cases which contain a seizure. This shift should ensure that a
sudden change in the PTT is not located between two windows.

We generated the positive samples as follows. First we determined the temporal
middle $t_{m}$ of the seizure by the start and end time of the seizure. After
that, with $t_{c}=t_{m}+o_{s}$, the shift is added. Finally, we selected the
PTT around $t_{c}$ with the interval length $l$. If the start or end time of
the selected PTT is outside of the recorded PTT, we discarded the training
pair.

The input variables of the samples are based on statistical quantities derived
from the PTT and are calculated for each window. The length of the samples
inner windows depends on their number and is chosen in such way, that all of
them have the same length. For each window we calculated the mean, minimum,
maximum, variance, slope and the intercept of the regression line. Furthermore
we used the differences of the mean values of all windows. All these values
serve as input variables for the model. Finally we added the target variable
to the training pair.

With the resulting dataset a Random Forest has been trained and tested. We
optimized the hyperparameters, including the minimum of samples required to
split a node, the tree count and depth of trees. Possible parameters are drawn
from an equally distributed set of numbers. We then trained the Random Forest
with the drawn selection and tested with five-fold cross-validation.

###  3.5. Validation of the Performance

In order to validate the performance of our detection model we use the method
proposed by Beniczky et. al. (Beniczky and Ryvlin, 2018). To estimate the
accuracy and performance of seizure detection, Beniczky et al. define which
outcome measures are to be considered. The parameters also refer to the
handling and error-prone nature of the device. The focus of this work is the
basic applicability of the PTT for the detection and does not consider an
explicit device. Consequently, we limited our evaluation to the following
outcome measures:

  * •

Sensitivity number (and percentage) of all detected seizures/ number of all
seizures recorded during the study

  * •

False alarm rate: the number of false alarms per $24$ hours

  * •

Detection latency: time from seizure onset to the detection time

In addition we use the Receiver-Operating-Characteristic (ROC) curve and the
$F_{1}$-Score to evaluate the classifier performance (Zweig and Campbell,
1993).

##  4\. Results

The generated database contained $239$ parallel recorded ECG and PPG signals.
A seizure occurred in $149$ cases. The constant $k$, used for the removal of
cases with improper calculated RR-intervals, has been set to $200$
milliseconds. In this way $34$ cases were excluded from the further data
processing. We calculated the PTT using both PTT Algorithm and Reactive
Algorithm. Figure 2 shows an example for the calculated PTT-alterations using
the Reactive Algorithm during a CPS.

Using a window size of $8$ minutes, the generated data set contained $114$
positive and $1886$ negative samples. There were nine different seizure types
included and the features were generated using two inner windows. The $o_{s}$
parameter has been set to $30$ seconds. On this dataset we trained and tested
a Random Forest as a binary classifier with five-fold cross-validation. We
used the parameters with the best $F_{1}$-Score for a final evaluation. The
following hyperparameters are the results of the optimization:

  * •

Number of Trees: $1600$

  * •

Maximum depth: $20$

  * •

Minimum node split: $10$

Table 1 shows the overall classification performance for both PTT Algorithm
and Reactive Algorithm. The Reactive Algorithm yields better results than the
PTT Algorithm.

Table 1. Measures of the Performance for both PTT Algorithm and Reactive Algorithm using a Random Forest | PTT Algorithm | Reactive Algorithm
---|---|---
$Sensitivity$ | $16.39\%\pm 13.29\%$ | $42.12\%\pm 9.08\%$
$Specificity$ | $98.84\%\pm 0.01\%$ | $99.69\%\pm 0.34\%$
$Precision$ | $28.97\%\pm 10.83\%$ | $84.56\%\pm 13.41\%$
$F_{1}$-Score | $0.20\pm 0.13$ | $0.56\pm 0.05$
Figure 3. ROC curve under the usage of the Reactive Algorithm and a Random
Forest. The Random Forest is in $40$% of the seizures confident that it is a
seizure. The rest of the seizures is more difficult to distinguish from the
negative training pairs.

The ROC curve can be seen in Figure 3 and starts with a strong increase until
over $40$ % of the seizures have been classified correctly. Thus, the model is
in $40$% of the seizures confident that it is a seizure. The rest of the
seizures is more difficult to distinguish from the negative training pairs.
The further course of the ROC curve indicates a random process.

Table 2 shows the detection performance for specific seizure types. The best
model, based on the Reactive Algorithm, identified $37.31$ % of CPS, $21.43$ %
of SPS, and $20.00$ % of absence seizures. This shows that seizures can be
detected with the PTT. The classification results however depend on the
seizures types. The seizures involving the muscles ($10$ GTCS, $4$ tonic and
$1$ atonic) were all correctly detected. An exception to this are the
myoclonic seizures, which could not be detected.

Table 2. Detection Performance for Seizures Types using the Reactive Algorithm and a Random Forest.  Type | Correct | Total | Rate
---|---|---|---
$GTCS$ | $10$ | $10$ |  $100.00$ %
$tonic$ | $4$ | $4$ |  $100.00$%
$atonic$ | $1$ | $1$ |  $100.00$%
$CPS$ | $25$ | $67$ |  $37.31$%
$absence$ | $1$ | $5$ |  $20.00$%
$aura$ | $4$ | $23$ |  $17.39$%
$myoclonic$ | $0$ | $4$ |  $0.00$%

The high detection rate for motor seizures could be caused by a deterioration
in data quality. This is supported by the fact that the model uses the
variance of the training pairs to separate the classes. However, the increased
variance could also be caused by the valley output by the reactive algorithm
when the PTT drops. It should also be noted that the Reactive Algorithm can
lead to increased variance if the heart rate is increased. Accordingly, the
model could not only predict the seizures from the change in PTT, but also an
increased heart rate. The myoclonic seizures, even though they have not been
recognized by the model, could also affect blood pressure. They could lead to
minor alterations in the PTT and not be detected cause the removal of the
clock drift filtered those alterations too.

With a number of six false-positive training pairs, out of $266.67$ hours of
data, the false alarm rate is $0.54$ alarms per day. The true false alarm rate
however may be higher, because sections with a great difference between ECG
and PPG interval lengths were excluded. Due to the usage of $8$ minute windows
and $o_{s}=30s$ the detection latency is $210$ seconds.

##  5\. Conclusion and Future Work

The developed PTT algorithm takes the clock drift into account and responds to
alterations in the PTT. The degree of respondence to alterations and removal
of the clock-drift can be adjusted. The results show that seizures can be
detected by using the PTT. Using the algorithm, which is considering the clock
drift, the sensitivity increased from $16.39\%\pm 13.29\%$ to $42.12\%\pm
9.08\%$ compared to the simple PTT algorithm. The recognition rate depends on
the type of seizure. The best model was trained under use of a Random Forest
and detected almost all motor seizures. Furthermore, it detected $37.31$ % of
the CPS and $21.43$ % of the SPS. The $F_{1}$-Score is $0.56\pm 0.05$. The
Random Forest separates seizures mainly through the variance and the minimum,
for samples of $8$ minutes. The high detection rate with motor seizures can
have two reasons. On the one hand, the blood pressure in this kind of seizures
could fall more sharply. The separation over the variance could then be
explained by the increased variance caused by the valley. On the other hand,
the increased variance can also be explained by measurement errors that arise
from the movements during the seizure. The Random Forest could detect
seizures, not only by the PTT, but also by a worse data quality.

In future work, besides the use of a Random Forest, other methods can also be
used for model prediction. Furthermore the training of models for specific
seizures types could lead to a higher sensitivity. Features generated using
the PTT should be evaluated in a multimodal approach and may reduce the false
alarm rate. In addition to seizure detection, also other diseases can be
focused on using the same sensor technology and taking alterations of blood
pressure into account.

## References

  * (1)
  * Beniczky et al. (2013) Sándor Beniczky, Tilman Polster, Troels W. Kjaer, and Helle Hjalgrim. 2013.  Detection of generalized tonic-clonic seizures by a wireless wrist accelerometer: A prospective, multicenter study.  _Epilepsia_ 54, 4 (apr 2013), e58–e61.  https://doi.org/10.1111/epi.12120
  * Beniczky and Ryvlin (2018) Sándor Beniczky and Philippe Ryvlin. 2018.  Standards for testing and clinical validation of seizure detection devices.  _Epilepsia_ 59 (jun 2018), 9–13.  https://doi.org/10.1111/epi.14049
  * Blum et al. (1996) D E Blum, J Eskola, J J Bortz, and R S Fisher. 1996\.  Patient awareness of seizures.  _Neurology_ 47, 1 (1996), 260–264.
  * Hampel et al. (2016) Kevin Hampel, Amirhossein Jahanbekam, Christian E. Elger, and Rainer Surges. 2016.  Seizure‐related modulation of systemic arterial blood pressure in focal epilepsy.  _Epilepsia_ 57 (2016).
  * Hennig and Patzak (2013) A. Hennig and A. Patzak. 2013.  Continuous blood pressure measurement using pulse transit time.  _Somnologie - Schlafforschung und Schlafmedizin_ 17, 2 (jun 2013), 104–110.  https://doi.org/10.1007/s11818-013-0617-x
  * Hoope et al. (2007) Christian Hoope, Annkathrin Poepel, and Christian E Elger. 2007.  Epilepsy: accuracy of patient seizure counts.  _Achives of neurology_ 64, 11 (2007), 1595–1599.
  * Hoppe et al. (2015) Christian Hoppe, Mieke Feldmann, Barbara Blachut, Rainer Surges, Christian E. Elger, and Christoph Helmstaedter. 2015.  Novel techniques for automated seizure registration: Patients’ wants and needs.  _Epilepsy & Behavior_ 52, Pt A (nov 2015), 1–7.  https://doi.org/10.1016/j.yebeh.2015.08.006
  * Inan et al. (2009) Akman Cigdem Inan, Maria A Montenegro, Susan Jacob, Karen Eck, Claudia Chiriboga, and Frank Gilliam. 2009\.  Seizure frequency in children with epilepsy: factors influencing accuracy and parental awareness.  _Seizure_ 18, 7 (2009), 524–529.
  * Jaiswal and Banka (2018) Abeg Kumar Jaiswal and Haider Banka. 2018.  Epileptic seizure detection in EEG signal using machine learning techniques.  _Australasian Physical & Engineering Sciences in Medicine_ 41, 1 (mar 2018), 81–94.  https://doi.org/10.1007/s13246-017-0610-y
  * Kurada et al. (2019) Abhinav V. Kurada, Tarun Srinivasan, Sarah Hammond, Adriana Ulate-Campos, and Jonathan Bidwell. 2019\.  Seizure detection devices for use in antiseizure medication clinical trials: A systematic review.  , 61–69 pages.  https://doi.org/10.1016/j.seizure.2019.02.007
  * Leutmezer et al. (2003) Fritz Leutmezer, Christiana Schernthaner, Stefanie Lurger, Klaus Pötzelberger, and Christoph Baumgartner. 2003.  Electrocardiographic Changes at the Onset of Epileptic Seizures.  _Epilepsia_ 44, 3 (mar 2003), 348–354.  https://doi.org/10.1046/j.1528-1157.2003.34702.x
  * Moseley et al. (2013) Brian Moseley, Lisa Bateman, John J. Millichap, Elaine Wirrell, and Chrysostomos P. Panayiotopoulos. 2013\.  Autonomic epileptic seizures, autonomic effects of seizures, and SUDEP.  _Epilepsy & Behavior_ 26, 3 (mar 2013), 375–385.  https://doi.org/10.1016/j.yebeh.2012.08.020
  * Nass et al. (2019) Robert D. Nass, Kevin G. Hampel, Christian E. Elger, and Rainer Surges. 2019.  Blood Pressure in Seizures and Epilepsy.  _Frontiers in Neurology_ 10 (2019), 501.  https://doi.org/10.3389/FNEUR.2019.00501
  * Nigam and Graupe (2004) Vivek Prakash Nigam and Daniel Graupe. 2004.  A neural-network-based detection of epilepsy.  _Neurological Research_ 26, 1 (jan 2004), 55–60.  https://doi.org/10.1179/016164104773026534
  * Nijsen et al. (2005) Tamara M.E. Nijsen, Johan B.A.M. Arends, Paul A.M. Griep, and Pierre J.M. Cluitmans. 2005\.  The potential value of three-dimensional accelerometry for detection of motor seizures in severe epilepsy.  _Epilepsy & Behavior_ 7, 1 (aug 2005), 74–84.  https://doi.org/10.1016/J.YEBEH.2005.04.011
  * Opherk et al. (2002) Christian Opherk, James Coromilas, and Lawrence J Hirsch. 2002\.  Heart rate and EKG changes in 102 seizures: analysis of influencing factors.  _Epilepsy Research_ 52, 2 (dec 2002), 117–127.  https://doi.org/10.1016/S0920-1211(02)00215-2
  * Picard et al. (2016) Rosalind Picard, Giulia Regalia, Chiara Caborni, Matteo Migliorini, and Francesco Onorati. 2016\.  Embrace, a wearable convulsive seizure detection and alert system: First performance report of a case study in real-life settings.
  * Poh et al. (2012) Ming-Zher Poh, Tobias Loddenkemper, Claus Reinsberger, Nicholas C. Swenson, Shubhi Goyal, Mangwe C. Sabtala, Joseph R. Madsen, and Rosalind W. Picard. 2012.  Convulsive seizure detection using a wrist-worn electrodermal activity and accelerometry biosensor.  _Epilepsia_ 53, 5 (may 2012), e93–e97.  https://doi.org/10.1111/j.1528-1167.2012.03444.x
  * Rabending and Runge (2007) G Rabending and U Runge. 2007.  The electronic treatment diary for epileptic patients - A new instrument for therapeutical treatment.  7 (2007), 273–280.
  * Tzallas et al. (2009) A.T. Tzallas, M.G. Tsipouras, and D.I. Fotiadis. 2009\.  Epileptic Seizure Detection in EEGs Using Time–Frequency Analysis.  _IEEE Transactions on Information Technology in Biomedicine_ 13, 5 (sep 2009), 703–710.  https://doi.org/10.1109/TITB.2009.2017939
  * Ulate-Campos et al. (2016) A. Ulate-Campos, F. Coughlin, M. Gaínza-Lein, I. Sánchez Fernández, P.L. Pearl, and T. Loddenkemper. 2016\.  Automated seizure detection systems and their effectiveness for each type of seizure.  _Seizure_ 40 (aug 2016), 88–101.  https://doi.org/10.1016/J.SEIZURE.2016.06.008
  * Vandecasteele et al. (2017) Kaat Vandecasteele, Thomas De Cooman, Ying Gu, Evy Cleeren, Kasper Claes, Wim Paesschen, Sabine Huffel, and Borbála Hunyadi. 2017.  Automated Epileptic Seizure Detection Based on Wearable ECG and PPG in a Hospital Environment.  _Sensors_ 17, 10 (oct 2017), 2338\.  https://doi.org/10.3390/s17102338
  * Zweig and Campbell (1993) M. H. Zweig and G. Campbell. 1993.  Receiver-operating characteristic (ROC) plots: A fundamental evaluation tool in clinical medicine.  , 561–577 pages.  https://doi.org/10.1093/clinchem/39.4.561


††thanks: Present address: CAS Key Laboratory of Nanosystem and Hierarchical
Fabrication, CAS Center for Excellence in Nanoscience, National Center for
Nanoscience and Technology, Beijing 100190, China††thanks: Present address:
Physics Department, University of Bath, North Rd, Claverton Down, Bath BA2
7AY, UK

# Positive Seebeck Coefficient in Highly doped La2-xSrxCuO4 ($x$=0.33); Its
Origin and Implication

Hao Jin1 Alessandro Narduzzo2 Minoru Nohara3 Hidenori Takagi4,5 N. E.
Hussey2,6 Kamran Behnia1<EMAIL_ADDRESS>(1) LPEM (ESPCI - CNRS -
Sorbonne University), PSL University, 75005 Paris, France
(2) H. H. Wills Physics Laboratory, University of Bristol, Tyndall Avenue,
Bristol, BS8 1TL, UK
(3) Research Institute for Interdisciplinary Science, Okayama University,
Okayama 700-8530, Japan
(4) Max Planck Institute for Solid State Research, 70569 Stuttgart, Germany
(5)Department of Physics, University of Tokyo
Tokyo 113-0033, Japan
(6) High Field Magnet Laboratory (HFML-EMFL) and Institute for Molecules and
Materials, Radboud University, 6525 ED Nijmegen, The Netherlands

###### Abstract

We present a study of the thermoelectric (Seebeck and Nernst) response in
heavily overdoped, non-superconducting La1.67Sr0.33CuO4. In spite of the
electron-like curvature of the Fermi surface, the Seebeck coefficient is
positive at low temperatures. Such a feature, previously observed in copper,
silver, gold and lithium, is caused by a non-trivial energy dependence of the
scattering time. We argue that this feature implies a strong asymmetry between
the lifetime of occupied and unoccupied states along the zone diagonals and
such an electron-hole asymmetry impedes formation of Cooper pairs along the
nodal direction in the superconducting ground state emerging at lower doping
levels.

Cuprates are a family of layered materials each with a Mott insulating parent
which turns superconducting with a sizeable critical temperature upon doping
Lee et al. (2006). Their normal state transport properties exhibit highly
anomalous behaviour and a remarkable evolution with doping, temperature and
magnetic field (for recent reviews, see Hussey et al. (2018); Proust and
Taillefer (2019)). In hole-doped cuprates, superconductivity fades gradually
away beyond optimal doping $p\sim$ 0.16 before vanishing above a critical
threshold $p_{SC}\sim 0.30$, though only in La2-xSrxCuO4 (LSCO) has this
threshold ever been exceeded Nakamae et al. (2003); Cooper et al. (2009).

Figure 1: (Color online) Seebeck and Nernst coefficients: a) Temperature
dependence of the in-plane Seebeck coefficient $S$, in La1.67Sr0.33CuO4. Inset
shows the low temperature $S/T$ showing the uncertainty in the zero-
temperature slope. b) a) Temperature dependence of the in-plane Nernst
coefficient $\nu$ measured at $\mu_{0}H$ = 1 T.

In heavily overdoped La1.67Sr0.33CuO4 (LSCO33), the in-plane resistivity
$\rho_{ab}(T)$ follows the expected $T^{2}$ dependence of a correlated Fermi
liquid below 50 K Nakamae et al. (2003). Removing carriers from this metal
leads to the emergence of superconductivity as well as a ’strange’ metal
regime with a robust $T$-linear component in $\rho_{ab}(T)$ at low temperature
Cooper et al. (2009). Scrutinizing the non-superconducting metal above
$p_{SC}$ may provide clues for the origin of both.

In this paper, we present data on the thermoelectric response of LSCO33. In
the zero temperature limit, both the Seebeck and Nernst coefficients are
$T$-linear, as expected for a Fermi liquid. The sign and the slope of the
Seebeck coefficient, however, point to a remarkably large lifetime asymmetry
between occupied and unoccupied states. We argue that this impedes the
formation of Cooper pairs along the zone diagonal ($\pi$, $\pi$) and thus
provides the starting point for the emergence of a superconducting state with
a $d_{x^{2}-y^{2}}$ symmetry.

Figure 2: (Color online) The Fermi surface: a) Tight binding Fermi surface of
La2-xSrxCuO4 for $x$ = 0.22 and $x$ = 0.32. The Fermi surface is electron-
like. Note the large contrast between the doping evolution along nodal and
anti-nodal directions. b) The relative change in Fermi velocity $v_{F}$ at
azimuthal angle $\phi$ as one passes from $x$ = 0.30 to $x$ = 0.32. The change
is arrested along the nodal direction and is maximal around the anti-nodes.

The resistivity of the single crystal measured in the present work showed no
trace of superconductivity down to 95 mK (see Ref. Nakamae et al. (2003, 2009)
for more details about the crystal growth, annealing conditions and
characterization). The Seebeck and Nernst coefficients were subsequently
measured (in 2004) using a standard one-heater-two-thermometers technique.
Fig. 1 shows their $T$-dependence between 0.4 K and 45 K. Both coefficients
show a quasi $T$-linear dependence at low $T$. The asymptotic zero-temperature
slope of the Seebeck coefficient is $S/T$ = +0.21 $\mu$V.K-2, while for the
Nernst coefficient, the slope is $\nu/T$ = -1.5 nV.T-1.K-2.

The slope of the Nernst coefficient obtained in this measurement was first
discussed in Ref. Behnia (2009). There, it was argued that in the semi-
classical picture, the amplitude of the slope is given by a set of fundamental
constants ($\pi^{2}k_{B}/3e$) multiplied by the ratio of the mobility
$\mu_{\rm H}$ to the Fermi energy $E_{F}$. This picture is backed by available
experimental data on numerous metals Behnia (2009); Behnia and Aubin (2016).
In the specific case of LSCO33, the low-$T$ $\nu/T$ is in fair agreement with
estimates of $\mu_{\rm H}\approx 100$ (from magnetoresistance Nakamae et al.
(2003)) and $E_{F}\approx$ 5900 K (from specific heat Nakamae et al. (2003)).

In this report, we will focus on the Seebeck response and discuss the
significance and implications of both its sign and amplitude. The Seebeck
coefficient in cuprates has been the subject of numerous studies (See chapter
8 in Ref. Behnia (2015) for a review). In the case of LSCO, it was previously
studied in single crystals (up to $x$ = 0.3 by Nakamura and Uchida Nakamura
and Uchida (1993)) and in polycrystalline powders (up to $x$ = 0.45 by Cooper
and Loram Cooper and Loram (1996) and up to $x$ = 0.35 by Elizarova and
Gasumyants Elizarova and Gasumyants (2000)). Our LSCO33 data, which extends
down to sub-kelvin temperature, agrees well with these earlier studies in the
overlapping temperature ranges. Our data also smoothly connects to what was
recently reported by Collignon et al. in Eu-substituted LSCO for $x<0.26$
Collignon et al. (2020).

The first remarkable fact about the Seebeck coefficient of LSCO33 is its
positive sign. Angle resolved photoemission spectroscopy (ARPES) Yoshida et
al. (2007, 2006); Horio et al. (2018) studies have extensively documented the
emergence of an electron-like Fermi surface in LSCO for $x>0.2$, i.e. closed
around the $\Gamma$ point in the Brillouin zone. Fig. 2 shows the Fermi
surface derived from a tight binding model with nearest-neighbor hopping
parameters chosen to fit the ARPES-resolved Fermi surface Yoshida et al.
(2006); Horio et al. (2018). Thus, given the electron-like character of the
Fermi surface, one would naively expect the thermopower of LSCO33 to be
negative.

Interestingly, the Hall coefficient of LSCO33 is also positive Narduzzo et al.
(2008). This observation was explained by taking into account both the
curvature of the Fermi surface and the strong angle dependence of the
scattering time $\tau$ and mean-free-path of the mobile carriers at the Fermi
level Narduzzo et al. (2008). For the Seebeck coefficient, as we will see
below, it is the energy dependence of $\tau$ which matters.

In numerous Fermi liquids, there is an empirical correlation between the slope
of the diffusive Seebeck coefficient $S/T$ and the magnitude of the electronic
specific heat $\gamma$ Behnia et al. (2004). A dimensionless ratio of these
two quantities can be defined using Avogadro’s number $N_{Av}$ and the charge
of an electron, $e$:

$q=\frac{SN_{Av}e}{T\gamma}$ (1)

In dense Fermi liquids (i.e. those with roughly one mobile electron per
formula unit), $q$ is of order unity. This observation, first reported in 2004
Behnia et al. (2004) has been confirmed in numerous cases. The strength of
correlation among the conduction electrons tunes $\gamma$ over several orders
of magnitude ($\approx$ 1-1000 mJ.K-2.mol-1). Concomitantly, it modifies the
absolute value of $S/T$. Since both these quantities track the entropy
accumulated by electrons, such a correlation may not be surprising, though its
persistence in many multi-band metals with a Fermi surface consisting of
pockets of both signs remains a puzzle.

In LSCO33, where $S/T$ = +0.21 $\mu$V.K-2 and $\gamma$ = 6.9 mJ.K-2.mol-1
Nakamae et al. (2003), one finds $q=+2.8$. In dilute Fermi liquids, $q$ can be
significantly larger than unity, because the entropy per volume lags behind
entropy per carrier. In URu2Si2, for example, $|q|\simeq 11$ Zhu et al.
(2009). LSCO33, on the other hand, is a dense Fermi-liquid with 1.3 carriers
per formula unit. Hence, not only the sign, but also the enhanced value of $q$
demand an explanation.

As it turns out, this is not the first example of a system with a large simple
Fermi surface – occupying more than half the Brillouin zone – exhibiting an
anomalous Seebeck coefficient. In noble metals (Cu, Ag, Au), the Seebeck
coefficient is positive and $T$-linear well above the phonon drag peak
MacDonald (2006), despite their Fermi surfaces being electron-like Shoenberg
(2009). In 1967, Robinson called this puzzle ‘a nagging embarrassment to the
theory of the ordinary electronic transport properties of solids’ Robinson
(1967). Interestingly, the $q$ ratio is +0.75 in Cu, +0.81 in Ag and +0.86 in
Au Behnia (2015), i.e. $S/T$ has the right magnitude, just the wrong sign.

This ‘reversed sign thermopower’ puzzle in Cu, Ag, Au and Li (the alkali metal
also showing an unexpected positive Seebeck response) has been addressed by
Robinson Robinson (1967) and more recently by Xu, Di Gennaro and Verstraete Xu
and Verstraete (2014); Xu et al. (2020). Robinson argued that a mean-free-path
rapidly decreasing with increasing energy would provide a solution to the
puzzle and this can arise due to the structure of the electron-ion
pseudopotential. Xu et al. carried out first-principle calculations and found
that in Li, the sign reversal is driven by density and lifetime asymmetries
between states above and below the Fermi level, $E_{F}$ Xu and Verstraete
(2014). Similar calculations for noble metals also produced a positive sign
due to a non-trivial asymmetry in electron-phonon coupling for electronic
states at the two sides of the chemical potential Xu et al. (2020). These
results motivate us to search for a similar solution for the puzzle of the
‘wrong’ sign in LSCO33.

Figure 3: (Color online) Inverting the sign of the Seebeck response:  The
Seebeck coefficient is the result of the integration of three components over
the Fermi surface. These three ingredients, sketched as a function of energy
normalized to the chemical potential $\mu$, are: a) The pondering function; b)
The product of density of states and the square of velocity of a gas of free
electrons (blue) and free holes (red): c) The scattering time $\tau(\epsilon)$
in three distinct scenarios. When $\tau(\epsilon)$ is constant (black) or e-h
symmetric (red), the sign of the Seebeck coefficient remains positive for
holes and negative for electrons. When the energy dependence is such that the
unoccupied states are significantly more scattered than the occupied states
(blue) the sign will be inverted.

The Seebeck coefficient is defined as the ratio of thermoelectric conductivity
$\alpha$ to the electric conductivity $\sigma$. The Boltzmann equation links
both coefficients to $\tau$, the density of states $N(\epsilon)$ and the
velocity $v$ Ziman (1972); Behnia (2015):

$\alpha=-e\int\tau(\epsilon_{k})v(\epsilon_{k}).v(\epsilon_{k})N(\epsilon_{k})\frac{(\epsilon_{k}-\mu)}{T}\frac{\partial
f}{\partial\epsilon_{k}}\,d\epsilon_{k}$ (2)

$\sigma=-e^{2}\int\tau(\epsilon_{k})v(\epsilon_{k}).v(\epsilon_{k})N(\epsilon_{k})\frac{\partial
f}{\partial\epsilon}\,d\epsilon_{k}$ (3)

Here, the integrals are over the whole Fermi surface, $f$ is the Fermi-Dirac
distribution and $\mu$ is the chemical potential. The expression for $\alpha$,
contains a material-independent pondering factor together with material-
dependent parameters. As seen in Fig. 3a, the pondering factor is anti-
symmetric about the chemical potential. In the absence of electron-hole
asymmetry near the chemical potential, $\alpha$ would be zero. However, even
in a free electron gas, such asymmetry exists; both the velocity
($v(\epsilon_{k})\propto\sqrt{\epsilon_{k}}$) and the density of states
($N(\epsilon_{k})\propto\sqrt{\epsilon_{k}}$) grow with energy (See Fig. 3b).
As a result, $\alpha$ of a free electron (hole) gas is negative (positive).
Such a correspondence between the sign of the Seebeck coefficient and the sign
of carriers survives even in more complex Fermi surface geometries provided
that the energy dependence of the scattering time (or the mean-free-path) does
not alter the result. Note that Fermi’s golden rule, by linking the scattering
rate and the density of states, implies that features in $N(\epsilon_{k})$
will have counterparts in $\tau(\epsilon_{k})$. Note that $v$, $\tau$ and
$N(\epsilon_{k})$ can all have significant momentum dependence too. As we
shall see below, this $k$-space anisotropy also plays a prominent role here.

Fig. 3c shows three possible scenarios for the energy dependence of the
scattering time. In the first two cases, $\tau$ is constant or its energy
dependence is symmetric as one moves off the chemical potential and there is
no effect on the sign of $S$. In the third case, however, $\tau$ decreases
sufficiently rapidly with increasing energy that it inverts the overall
balance of the responses of the occupied and unoccupied states, as originally
shown by Robinson Robinson (1967).

Such an energy dependence can arise for different reasons. According to first
principle calculations on Li Xu and Verstraete (2014); Xu et al. (2020), a
feature in density of states just below the chemical potential skews the
available phase space for scattering Xu and Verstraete (2014). In copper, on
the other hand, the density of states is flat near the chemical potential Xu
et al. (2020), and it is the electron-phonon coupling that is energy
dependent. Robinson’s phenomenological model – invoking a screened electron-
ion pseudopotential – leads to a similar conclusion Robinson (1967).

Coming back to LSCO33, an energy-dependent scattering time would provide a
natural explanation for the positive sign of the Seebeck coefficient, though
the large magnitude of $q$ implies that the hole-particle asymmetry may be
even more pronounced than in noble metals. If one assumes a conventional
energy dispersion for a free electron gas but with an energy dependence of the
mean-free-path that follows a power law: $\ell\propto\epsilon^{-\delta}$, then
$\delta$ would be set by $3q/2+1$. In copper, for example, an inverted
$q=+0.75$ would then require $\delta\approx 2.1$ Behnia (2015), while an
inverted $q=+2.8$ would require $\delta$ exceeding 5. This may suggest that in
our present case, the energy dependence of the mean-free-path is stronger than
a simple power law.

The evolution of the Fermi surface with doping shown in Fig. 2 indicates a
very plausible nexus for a strong particle-hole asymmetry. The introduction of
additional holes between $x$ = 0.22 and $x$ =0.32, shifts the Fermi surface
almost exclusively along the anti-nodal direction and not at all along the
zone diagonal. This implies that along the nodal orientation the density of
states does not smoothly grow as a function of the chemical potential and
therefore, the phase space for scattering to unoccupied states above the
chemical potential is extremely reduced. As seen in Fig. 2b, the doping-
induced change of the Fermi velocity along the zone diagonal is negligible in
comparison with the anti-nodal direction. This indicates that the Seebeck
coefficient is dominated by the contribution of nodal quasi-particles with
strong asymmetry in the lifetime between occupied and unoccupied states.

The dichotomy between the nodal and anti-nodal contributions to transport has
been demonstrated by angle-dependent magnetoresistance (ADMR) studies, first
in Tl-2201 Hussey et al. (2003); Abdel-Jawad et al. (2006) and more recently
in Eu-LSCO Grissonnanche et al. (2020). However, the focus of attention of
both studies was the $T$-dependence of the anisotropic $\tau$ and not the
energy dependence of $\tau$ and its anisotropy. What causes the scattering
time to be strongly energy dependent along the nodes remains a question to be
addressed by microscopic theory. At this stage, let us point out that this
identification has a possible link with the superconducting gap structure and
pairing symmetry.

LSCO33 becomes a superconductor upon removal of mobile carriers. At the same
time, a $T$-linear scattering rate emerges below $p_{SC}=0.3$. The finite
positive $S/T$, on the other hand, does not appear to be affected by the
emergence of superconductivity. Indeed, the magnitude of $S/T$ in non-
superconducting LSCO33 is almost identical with what has been recently found
by Collignon et al. in superconducting Eu-LSCO for $0.23<x<0.26$ ($\approx$
+0.2 $\mu$V. K-2) Collignon et al. (2020).

In a BCS superconductor, Cooper pairs are formed by the superposition of
states above and below the Fermi level within an energy window of the size of
the gap Tinkham (1996). Strong asymmetry between quasiparticle lifetimes
across the occupation boundary would impede the formation of Cooper pairs. Our
analysis finds that in the nodal orientation, the density of states and the
quasi-particle lifetime do not evolve smoothly across the Fermi energy. Such
an electron-hole asymmetry along ($\pi$, $\pi$) would obliterate the
superconducting gap along the nodes, even if the attractive interaction
leading to the formation of Cooper pairs were isotropic. It remains to be seen
how the $d_{x^{2}-y^{2}}$ pairing symmetry of the superconductor Tsuei and
Kirtley (2000) and the anisotropic energy-dependent scattering phase space of
the strange metal connect to each other.

KB is supported by the Agence Nationale de la Recherche (ANR-18-CE92-0020-01;
ANR-19-CE30-0014-04). NEH is supported by the Netherlands Organisation for
Scientific Research (NWO) (Grant No. 16METL01)—“Strange Metals” and by the
European Research Council (ERC) under the European Union’s Horizon 2020
research and innovation programme (Grant Agreement No. 835279-Catch-22).

## References

  * Lee et al. (2006) P. A. Lee, N. Nagaosa, and X.-G. Wen, Rev. Mod. Phys. 78, 17 (2006).
  * Hussey et al. (2018) N. E. Hussey, J. Buhot, and S. Licciardello, Rep. Prog. Phys. 81, 052501 (2018).
  * Proust and Taillefer (2019) C. Proust and L. Taillefer, Ann. Rev. Condens. Matt. Phys. 10, 409 (2019).
  * Nakamae et al. (2003) S. Nakamae, K. Behnia, N. Mangkorntong, M. Nohara, H. Takagi, S. J. C. Yates, and N. E. Hussey, Phys. Rev. B 68, 100502 (2003).
  * Cooper et al. (2009) R. A. Cooper, Y. Wang, B. Vignolle, O. J. Lipscombe, S. M. Hayden, Y. Tanabe, T. Adachi, Y. Koike, M. Nohara, H. Takagi, et al., Science 323, 603 (2009).
  * Nakamae et al. (2009) S. Nakamae, K. Behnia, N. Mangkorntong, M. Nohara, H. Takagi, S. J. C. Yates, and N. E. Hussey, Phys. Rev. B 79, 219904 (2009).
  * Behnia (2009) K. Behnia, J. Phys.: Condens. Matter 21, 113101 (2009).
  * Behnia and Aubin (2016) K. Behnia and H. Aubin, Rep. Prog. Phys. 79, 046502 (2016).
  * Behnia (2015) K. Behnia, _Fundamentals of Thermoelectricity_ (Oxford University Press, 2015).
  * Nakamura and Uchida (1993) Y. Nakamura and S. Uchida, Phys. Rev. B 47, 8369 (1993).
  * Cooper and Loram (1996) J. R. Cooper and J. W. Loram, J. Phys. I France 6, 2237 (1996).
  * Elizarova and Gasumyants (2000) M. V. Elizarova and V. E. Gasumyants, Phys. Rev. B 62, 5989 (2000).
  * Collignon et al. (2020) C. Collignon, A. Ataei, A. Gourgout, S. Badoux, M. Lizaire, A. Legros, S. Licciardello, S. Wiedmann, J. Q. Yan, J. S. Zhou, et al., arXiv e-prints arXiv:2011.14927 (2020).
  * Yoshida et al. (2007) T. Yoshida, X. J. Zhou, D. H. Lu, S. Komiya, Y. Ando, H. Eisaki, T. Kakeshita, S. Uchida, Z. Hussain, Z.-X. Shen, et al., Journal of Physics: Condensed Matter 19, 125209 (2007).
  * Yoshida et al. (2006) T. Yoshida, X. J. Zhou, K. Tanaka, W. L. Yang, Z. Hussain, Z.-X. Shen, A. Fujimori, S. Sahrakorpi, M. Lindroos, R. S. Markiewicz, et al., Phys. Rev. B 74, 224510 (2006).
  * Horio et al. (2018) M. Horio, K. Hauser, Y. Sassa, Z. Mingazheva, D. Sutter, K. Kramer, A. Cook, E. Nocerino, O. K. Forslund, O. Tjernberg, et al., Phys. Rev. Lett. 121, 077004 (2018).
  * Narduzzo et al. (2008) A. Narduzzo, G. Albert, M. M. J. French, N. Mangkorntong, M. Nohara, H. Takagi, and N. E. Hussey, Phys. Rev. B 77, 220502 (2008).
  * Behnia et al. (2004) K. Behnia, D. Jaccard, and J. Flouquet, J. Phys.: Condens. Matter 16, 5187 (2004).
  * Zhu et al. (2009) Z. Zhu, E. Hassinger, Z. Xu, D. Aoki, J. Flouquet, and K. Behnia, Phys. Rev. B 80, 172501 (2009).
  * MacDonald (2006) D. K. C. MacDonald, _Thermoelectricity : an introduction to the principles_ (Dover Publications, 2006).
  * Shoenberg (2009) D. Shoenberg, _Magnetic Oscillations in Metals_ (Cambridge University Press, 2009).
  * Robinson (1967) J. E. Robinson, Phys. Rev. 161, 533 (1967).
  * Xu and Verstraete (2014) B. Xu and M. J. Verstraete, Phys. Rev. Lett. 112, 196603 (2014).
  * Xu et al. (2020) B. Xu, M. Di Gennaro, and M. J. Verstraete, Phys. Rev. B 102, 155128 (2020).
  * Ziman (1972) J. M. Ziman, _Principles of the Theory of Solids_ (Cambridge University Press, 1972), 2nd ed.
  * Hussey et al. (2003) N. E. Hussey, M. Abdel-Jawad, A. Carrington, A. P. Mackenzie, and L. Balicas, Nature 425, 814 (2003).
  * Abdel-Jawad et al. (2006) M. Abdel-Jawad, M. P. Kennett, L. Balicas, A. Carrington, A. P. Mackenzie, R. H. McKenzie, and N. E. Hussey, Nature Phys. 2, 821 (2006), ISSN 1745-2481.
  * Grissonnanche et al. (2020) G. Grissonnanche, Y. Fang, A. Legros, S. Verret, F. Laliberté, C. Collignon, J. Zhou, D. Graf, P. Goddard, L. Taillefer, et al., arXiv e-prints arXiv:2011.13054 (2020).
  * Tinkham (1996) M. Tinkham, _Introduction to superconductivity_ (Courier Corporation, 1996).
  * Tsuei and Kirtley (2000) C. C. Tsuei and J. R. Kirtley, Rev. Mod. Phys. 72, 969 (2000).


(cvpr) Package cvpr Warning: Package ‘hyperref’ is not loaded, but highly
recommended for camera-ready version

# Improving Face Recognition from Caption Supervision with Multi-Granular
Contextual Feature Aggregation

Md Mahedi Hasan and Nasser Nasrabadi
West Virginia University
Morgantown, West Virginia, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We introduce caption-guided face recognition (CGFR) as a new framework to
improve the performance of commercial-off-the-shelf (COTS) face recognition
(FR) systems. In contrast to combining soft biometrics (_e.g_., facial marks,
gender, and age) with face images, in this work, we use facial descriptions
provided by face examiners as a piece of auxiliary information. However, due
to the heterogeneity of the modalities, improving the performance by directly
fusing the textual and facial features is very challenging, as both lie in
different embedding spaces. In this paper, we propose a contextual feature
aggregation module (CFAM) that addresses this issue by effectively exploiting
the fine-grained word-region interaction and global image-caption association.
Specifically, CFAM adopts a self-attention and a cross-attention scheme for
improving the intra-modality and inter-modality relationship between the image
and textual features, respectively. Additionally, we design a textual feature
refinement module (TFRM) that refines the textual features of the pre-trained
BERT encoder by updating the contextual embeddings. This module enhances the
discriminative power of textual features with a cross-modal projection loss
and realigns the word and caption embeddings with visual features by
incorporating a visual-semantic alignment loss. We implemented the proposed
CGFR framework on two face recognition models (ArcFace and AdaFace) and
evaluated its performance on the Multi-Modal CelebA-HQ dataset. Our framework
significantly improves the performance of ArcFace in both 1:1 verification and
1:N identification protocol.

##  1 Introduction

Despite remarkable advancements in face recognition due to the adoption of
margin-based loss functions [2, 14, 25], face recognition in unconstrained
scenarios remains a challenging problem [40]. The presence of covariate
factors in an unconstrained environment, such as illumination, and pose
variation, affects the face image quality, thus, decreasing the recognition
performance. Providing auxiliary information, such as facial marks, ethnicity,
and skin color, to a face recognition (FR) system can improve its recognition
performance [5, 41]. For example, video surveillance environment, where a
prevalent commercially-off-the-shelf (COTS) system performs poorly [40, 43],
the application of soft biometrics has been proven to improve the performance
of hard biometrics [32, 5].

Natural language captions that describe the visual contents of a face are an
essential soft biometric trait for face recognition. Although captions are
rich, they face many challenges that limit their application in the biometric
system. As natural language contains high-dimensional information, it is often
much more abstract than images. A short textual description of a given face
consisting of a few sentences is insufficient to describe all the minute
details of the facial features. Consequently, CGFR is significantly different
from other tasks such as cross-modal image-text retrieval (ITR) [13, 17] and
image-text matching (ITM) [16], where the matching text has a description of
the various objects, background scenes, and styles, etc.

To improve the performance of the FR system using CGFR, it is essential to
find not only the semantic understanding of textual contents but also the
proper association between visual and textual modalities. This is because the
embedding space of images and text lies in different spaces due to the
heterogeneity of the two modalities [17]. Aligning the image features with
word embeddings is thus crucial, as it has a significant impact on the
performance of a cross-modal fusion algorithm [17]. In this work, we finetune
the state-of-the-art BERT model [4] to update the contextual associations
among words in the caption by incorporating a visual-semantic alignment loss
[38] and a cross-modal classification loss [42]. Finetuning the text encoder
is essential because the BERT model was trained with different objectives than
ours. Therefore, we finetune to achieve two objectives: (1) to learn visually
aligned text embedding, ı.e., to realign word and caption embeddings with
visual information, and (2) to enhance the discriminative power of textual
features.

However, a simple feature-level cross-modal fusion without fine-grained
interaction between image-text tokens does not perform well. Therefore, we
propose a novel module, namely, the contextual feature aggregation module
(CFAM), to effectively carry out the fine-grained word-region interaction and
global image-caption association on two different granularities. There are
mainly three networks in the proposed CFAM: caption-level context modeling,
word-level context modeling, and a feature aggregation network. Both context
modeling networks adopt a self-attention and a cross-attention mechanism [33].
The self-attention mechanism increases the intra-modality relationship within
each modality, while the cross-attention mechanism [33] improves the inter-
modality relationship between image and textual features. The inputs to the
feature aggregation network are the context-enhanced features from the word-
and caption-level context modeling.

We conduct our experiments on a benchmark text-to-face dataset, namely, Multi-
Modal CelebA-HQ [35] (MMCelebA). Sample image-caption pairs of the dataset are
illustrated in Figure 1. In fact, the dataset is based on a subset of the
CelebA dataset [22] that contains high-resolution images with very low
variation. In our experiment, we remove the crucial face-alignment step and
apply some pre-processing steps such as random sub-sampling, random rotation,
horizontal flip to augment our database as well as downgrade the image quality
in order to mimic real-world low-quality video surveillance scenarios. The
verification rate of FR models, drops drastically on this preprocessed dataset
because the images are corrupted with down-sampling and noise, which adversely
affect their facial analysis procedure [43]. We then apply our CGFR framework
to both models. The experimental results demonstrate a remarkable performance
leap over these FR models.

In this study, our contributions are: (1) exploring a new paradigm to improve
face recognition with natural language supervision, (2) proposing the CFAM
module to exploit fine-grained interaction among local and global features
using word- and caption-level of granularities, (3) designing a textual
feature refinement module (TFRM) to refine textual features and align them
with visual content by fine-tuning the BERT encoder, and (4) conducting
extensive experiments on the MMCelebA [35] dataset using the proposed CGFR
framework to demonstrate substantial improvements over existing FR models.
Finally, this work demonstrates excellent potential for caption-guided face
recognition and provides a promising approach for further research.

The rest of this paper is organized as follows: an overview of the related
works is presented in Section 2. A detailed description of the proposed
method, including steps to finetune the BERT encoder, is presented in Section
3. In Section 4, we demonstrate the experimental evaluation of the CGFR
framework. Finally, we summarize our results with some possible future
research directions in Section 5.

Figure 1: Sample image-caption pairs from the state-of-the-art Multi-Modal
CelebA-HQ text-to-face dataset.

##  2 Related Work

###  2.1 Soft Biometrics

Most of the prior works in the literature on improving face recognition using
soft biometrics have been based on using categorical labels [5, 41]. Zhang _et
al_. [41] integrated a set of five soft biometrics (ethnicity, gender,
eyebrow, eye color, and hair color) with hard biometric systems. Compared to
the baseline recognition rates at FAR = 1e-3, their verification rate improved
up to 15.5% when introducing all the soft biometrics and 16.4% when
introducing gender information on the ugly part of the GBU database.
Furthermore, authors in [5] empirically proved that a manual estimation of the
six most discriminative soft biometrics improves the relative performance of
the FR systems (COTS Face++ and VGG-face) up to 40% over the LFW database.

###  2.2 Caption-Guided Face Recognition

Several early works have been proposed for caption-supervised face recognition
[11, 7]. Huang _et al_. [11] improved state-of-the-art face recognition using
web-scale images with captions by learning the feature space in an iterative
label expansion process. However, they only employed captions to extrapolate
the labels of the face identity.

Recently, with the development of generative adversarial networks (GANs) [6]
and transformers [33], text-to-face synthesis [34, 31], and facial attribute
editing [9, 35] with textual descriptions have gained increasing popularity.
For example, TediGAN [35] uses latent code optimization of pre-trained
StyleGAN for caption-guided facial image generation and manipulation. In
contrast to these works, we introduce a new line of research by exploring the
idea of using natural language captions to improve the performance of the FR
models. As there are no publicly available datasets that contain large-scale
image-caption pairs for our task, we employ MMCelebA [35] dataset which has
been widely used for text-to-face synthesis.

###  2.3 Attention Techniques

Recently, different attention mechanisms, such as self-attention, cross-
attention, etc., have been extensively exploited in various multimodal tasks
[18, 16, 38, 39]. Cross-attention or co-attention is an attention mechanism
initially proposed in transformers [33] that interacts with two embedding
sequences from different modalities (_e.g_., text, image). Li _et al_. [18]
propose a latent co-attention mechanism in which spatial attention relates
each word to corresponding image regions. Also, Lee _et al_. [16] developed a
stacked cross-attention network that learns the cross-modal alignments among
all regions in an image and words in a sentence. Xu _et al_. [38] applied an
attention mechanism to guide the generator to focus on different words while
generating various image sub-regions. They also proposed a deep attentional
multimodal similarity model (DAMSM) to improve the similarity between the
generated images and the given descriptions. To re-weight the importance of
local image regions in tasks such as image synthesis [38], image caption
generation [37], image segmentation [30, 39], and image-text matching [16,
18], word-level attention has been employed. However, only employing word-
level attention cannot ensure global semantic consistency due to the diversity
of the text and image modalities. Global contextual information is also
important as it provides more information on the visual content of the image
and the context of the caption. It also drives the global features toward a
semantically well-aligned context.

###  2.4 Multimodal Representation Learning

In recent years, dual-stream approaches, where the image and text encoder are
trained on large-scale datasets individually with different cross-modality
loss functions, have become widely popular in tackling various multimodal
downstream tasks [27]. A lot of cross-modal loss functions such as contrastive
[27, 19], triplet [16], word-region alignment [38], cross-modal projection
[42], etc., have been proposed as part of the training objectives. Zhang _et
al_. [42] proposed a novel projection loss that consists of two losses: a
cross-modal projection matching (CMPM) loss for computing the similarity
between image-text pairs and a cross-modal projection classification (CMPC)
loss for learning a more discriminative visual-semantic embedding space. Also,
Liao _et al_. [20] proposed learning an optimal multimodal representation by
maximizing the mutual information between the local features of the images and
the sentence embedding. However, most of the dual-stream approaches in the
literature cannot effectively and accurately exploit the fine-grained
interaction among word-region features.

Furthermore, in other works, image and textual features, extracted from
separate encoders, are often concatenated to feed into a fusion module to
learn the joint representations [39]. However, a simple fusion scheme may not
be effective since the unaligned visual and word tokens lack prior
relationships. Therefore, cross-modal interaction of local and global contexts
is essential. For example, Niu _et al_. [26] map phrases-region and image-
caption into a joint embedding space using an image-text alignment method that
consists of three different granularities: global-global alignment, global-
local alignment, and local-local alignment.

In this work, we adopt a dual-stream approach to extract facial and textual
features from pre-trained encoders. We apply a visual semantic alignment loss,
known as DAMSM [38], to align the visual and word tokens locally and globally.
We also employ CMPC loss [42] to enhance the discriminative power of the
features. Finally, for fine-grained cross-modal interaction, we design CFAM.

Figure 2: An overview of our proposed CGFR framework: it contains an ArcFace
FR model and a pre-trained BERT encoder for extracting the facial features and
textual embeddings from the input image-caption pair, respectively. First,
TFRM updates contextual associations of the text embedding by finetuning the
text encoder using the state-of-the-art DAMSM loss [38] and a cross-modal
projection loss [42]. Next, CFAM fuses the facial features with textual
embeddings through cross-attention at both the word and caption-level of
granularities.

##  3 Framework

An overview of our proposed framework is depicted in Figure 2.

###  3.1 Facial Feature Extraction

We first employ the ArcFace model [2] as a feature extractor to extract the
facial features from the input image. Specifically, we choose ResNet18-IR [8,
2] for the backbone of the ArcFace model, which was pre-trained on the MS1MV3
dataset [3]. Here, we modify the ResNet18-IR architecture by replacing the
global average-pooling layer with a fully connected layer. The output of the
fully connected layer is a 512-dimensional feature vector, which is considered
as the global features $\textbf{v}~{}\epsilon~{}\mathbb{R}^{512}$ of the input
image, as it contains high-level visual information. We extract the local
features of the image $I~{}\epsilon~{}\mathbb{R}^{256\times 14\times 14}$ from
the output of the third IR block. The size of the input image is $3\times
112\times 112$. We further employ CGFR on the AdaFace model [14]. Here, the
backbone, ResNet18-IR, is similar to the ArcFace model; however, it was pre-
trained on the WebFace4M dataset [44]. In contrast to ArcFace, the input is a
BGR image.

###  3.2 Textual Feature Extraction

####  3.2.1 BiLSTM

Most of the works in the literature usually employ a long short-term memory
network (LSTM) [10] as an encoder to extract text embeddings from natural
language descriptions [37, 38]. Therefore, in this work, as a baseline, we
apply a bidirectional LSTM (BiLSTM) [29] as a text encoder to extract semantic
vectors from the input captions. The BiLSTM encoder encodes the input caption
as a matrix of $W~{}\epsilon~{}\mathbb{R}^{L\times D}$. Here, $D$ denotes the
dimension of the word vector, and $L$ denotes the maximum number of words in a
caption. In our experiment, for the BiLSTM encoder, we consider a maximum of
18 words per caption, and the dimension of the word embedding is 256.
Therefore, for an input caption of $L$ words, the word embeddings are
[${\mathbf{w_{1}},\mathbf{w_{2}},\cdots\mathbf{w_{L}}}$], where
$\mathbf{w_{L}}~{}\epsilon~{}\mathbb{R}^{D}$ is the caption embedding.

####  3.2.2 BERT

One of the limitations of traditional word embedding (such as word2vec) is
that they only have one context-independent embedding for each word. Devlin
_et al_. [4] introduced BERT, a deep bidirectional encoder that considers the
context of a word for each occurrence. In this work, we adopt a pre-trained
BERT-base model [4] with 12 encoder layers, each having 12 attention heads. It
obtains the contextual embedding of each word by exploiting all the words in
the caption. Furthermore, in addition to the input tokens, we add a
classification token, $[CLS]$, at the beginning and a separator token,
$[SEP]$, at the end of each sentence in the caption. The maximum length of the
token sequence, $L$, is set to $21$. Additional padding tokens, $[PAD]$, are
added for short-length captions after the last $[SEP]$ token. Extra tokens are
truncated if the length of the input tokens is higher than $L$. Therefore, the
input to the BERT-base model looks like this:

${[CLS],~{}w_{2},~{}w_{3},\cdots,~{}w_{L-3},~{}[SEP],~{}[PAD],~{}[PAD],\cdots}$

The output of the BERT layer gives a word matrix,
$W~{}\epsilon~{}\mathbb{R}^{L\times 768}$, where each contextualized token has
an embedding of 768 dimensions. Here, the first token, $[CLS]$, is a
classification token that represents the global embedding of the caption. The
remaining $L-1$ tokens represent the contextualized word embeddings. In
addition to BERT-base, we also experimented with other variants of BERT such
as BERT-large, DistilBERT-base [28], and RoBERTa-base [21]. However, in our
experiments, we found that the performance of these variants is almost the
same.

###  3.3 Textual Feature Refinement Module

In this subsection, we briefly describe the proposed textual feature
refinement module. Because our text encoder was pre-trained with objectives
that are totally different from ours and it creates embeddings that are
unaligned to the image features, we need to refine the textual features. As
shown in Figure 2, our TFRM consists of a convolution-based projection for
text embeddings, a projection head for local image features, and a module to
implement the visual-semantic alignment loss, DAMSM, and a cross-modal
projection classification loss.

####  3.3.1 Projection Heads

##### Convolution-based Projection:

As a caption has a natural order of word sequences, it is useful to extract
not only word-level features but also phrase-level features. Thus, we apply a
2D-convolution to the output of the BERT sequence to extract both word-level
and phrase-level information from the input caption. The first dimension of
the kernel size $K$ is set to 1, 2, and 3 to project uni-gram, bi-gram, and
tri-gram word sequences, respectively. For $K=$ 2 and 3, the word
representations, $W$, are appropriately padded to maintain the fixed length of
the sequence. All of these convolutions have a total of 64 filters with a
stride of 1. Next, we apply the max-pooling operation followed by an $L_{2}$
normalization across the outputs of the convolutions to generate the word
embeddings, $\mathbb{R}^{(L-1)\times 64}$. Figure 3 illustrates the proposed
scheme for creating word embeddings from the output of the BERT encoder.

Figure 3: The proposed convolution-based projection for creating the word
embeddings and global embedding for the input caption. 2D-convolutions with
three different kernel sizes are applied to the output representations of the
BERT encoder to extract both the word- and phrase-level information.

##### Caption Embedding:

There are multiple ways of creating the global embedding for the input
caption, $\textbf{c}~{}\epsilon~{}\mathbb{R}^{64}$. One common way to create
caption embedding is to employ a linear projection followed by a batch
normalization [12] on the $[CLS]$ token of the BERT output layer. We can also
create the caption embedding by applying the max-pooling operation across the
word embeddings of the convolution-based projection followed by an $L_{2}$
normalization. In our experiments, we achieved better results from the caption
embedding which was created by the latter scheme. Figure 3 also depicts the
scheme.

##### Projection Head for Image Features:

We project the local image features $I$ into a 64-dimensional space, which has
also been empirically found to be the optimal dimension for the word
embeddings. So, we design a projection head which consists of a $1\times 1$
convolution with 64 filters and a Leaky ReLU activation [36] for non-
linearity.

####  3.3.2 Objective Function

##### DAMSM Loss:

AttnGAN [38] introduced DAMSM loss to align image-caption pair by using word-
level and caption-level attention. Let $(W,I)$ denote an image-caption pair,
where $W~{}\epsilon~{}\mathbb{R}^{L\times D}$ represents the word embeddings,
and $I~{}\epsilon~{}\mathbb{R}^{H\times W\times D}$ represents the transposed
local image features. Next, we apply DAMSM loss [38] to perform cross-modal
contrastive learning between image-caption pair. The loss actually minimizes
the negative log posterior probability of the similarity scores between the
image-caption pair.

Figure 4: The block diagram of the proposed contextual feature aggregation
module. It applies cross-modal feature interaction on both word and caption
levels using a cross-attention mechanism. The module consists of three
networks. The first network, caption-level context modeling, produces a
64-dimensional global context-enhanced features whereas the second network
produces a 1024-dimensional regional context-enhanced features. The final
network aggregates the contextual features and finds an optimal representation
of it.

##### Cross-Modal Projection Classification Loss:

In order to produce discriminative textual features, we also apply a cross-
modal projection classification (CMPC) loss [42]. This loss first tries to
project the representations from one modality onto the corresponding features
from another modality and then classify them using normalized softmax loss.
The input to the CMPC is the global image features, v, and caption embeddings,
c. First, the image features are projected onto the normalized text
embeddings, $\bar{\textbf{c}}$. Therefore, the normalized softmax loss for the
image features, $L_{ipt}$, is given by:

$\small
L_{ipt}=\frac{1}{N}\sum_{i}-log(\frac{\exp(W^{T}_{y_{i}}\hat{\textbf{v}}_{i})}{\sum_{j}{\exp(W^{T}_{j}\hat{\textbf{v}}_{i})}}).$
(1)

Here,
$\hat{\textbf{v}}_{i}=\textbf{v}^{T}_{i}\bar{\textbf{c}_{i}}.\bar{\textbf{c}_{i}}$
denotes the vector projection of the image features. Now, let’s project the
text embeddings onto the normalized image features, $\bar{\textbf{v}}$.
Therefore, the text classification loss function, $L_{tpi}$, is given by:

$\small
L_{tpi}=\frac{1}{N}\sum_{i}-log(\frac{\exp(W^{T}_{y_{i}}\hat{\textbf{c}}_{i})}{\sum_{j}{\exp(W^{T}_{j}\hat{\textbf{c}}_{i})}}).$
(2)

Here,
$\hat{\textbf{c}}_{i}=\textbf{c}^{T}_{i}\bar{\textbf{v}}_{i}\bar{\textbf{v}}_{i}$
denotes the vector projection of the textual features. The total CMPC loss is
the summation of the two losses, as defined by Eq. 1 and Eq. 2.

##### Full Objective:

Our overall loss function is the weighted combination of the DAMSM and CMPC
losses:

$L_{loss}=\lambda_{1}L_{DAMSM}+\lambda_{2}L_{CMPC},$ (3)

where, $\lambda_{1}$ and $\lambda_{2}$ are the hyperparameters that control
the DAMSM and CMPC losses, respectively.

###  3.4 Contextual Feature Aggregation Module

In this study, we propose a contextual feature aggregation module (CFAM) that
applies cross-modal feature interactions in two different granularities: word
and caption. The block diagram of the proposed CFAM is illustrated in Figure
4.

####  3.4.1 Linear Fusion

First, we concatenate the global image features
$\textbf{v}~{}\epsilon~{}\mathbb{R}^{512}$ and the caption embedding
$\textbf{c}~{}\epsilon~{}\mathbb{R}^{64}$ from the convolution-based projected
head. Thus, we have a joint 576-dimensional multimodal representation. We then
apply a fully connected (FC) layer. This network serves as a fusion scheme for
our baseline approach.

####  3.4.2 Word-Level Context Modeling

In this network, we apply fine-grained cross-modal interactions between local
image features and word embeddings. Here, we use word embeddings as cues to
attend to the local image features extracted from the FR model. We also
experimented with image features as cues to attend to words. However, that did
not improve the performance, as words in a caption contain more abstract
information than image regions. Figure 4 illustrates the word-level context
modeling.

The inputs to the network are the word embeddings matrix,
$W~{}\epsilon~{}\mathbb{R}^{L-1\times 64}$, and local image features
$I~{}\epsilon~{}\mathbb{R}^{256\times 14\times 14}$. Batch normalization [12]
is applied to the image features, before feeding it to a convolution layer of
64 filters with a kernel of size 3, and padding of 2. A max-pooling layer with
a stride of size 2 is applied to the features map to reduce the spatial size
to ${64\times 8\times 8}$. Next, a self-attention layer with a $scale=0.5$ is
applied to increase the intra-modality relationship among the local image
features, followed by layer normalization [1].

Thus, due to the application of self-attention, each image region now contains
information about the whole image. In the self-attention layer, the $keys$,
$queries$, and $values$ are learned from $1\times 1$ convolutions. However,
the number of filters in $1\times 1$ convolutions for projecting $key$ and
$query$ are the multiplication of a $scale$ factor of the number filters of
the $1\times 1$ convolution for learning $values$. Note that the application
of normalization and self-attention in this network, as analyzed in Table 3,
is very crucial.

Contrary to image features, word embeddings $W$, have different dimensions.
Therefore, we, first, calculate the correlation of the word embeddings matrix,
$W^{T}W~{}\epsilon~{}\mathbb{R}^{64\times 64}$. Next, we reshape the
embeddings matrix to size ${64\times 8\times 8}$. Similar to image features,
we also experimented to implement self-attention to the reconstructed word
features, but that does not improve the performance. The reason for this could
be that as textual features are extracted from transformer-based BERT
architectures, the intra-modal relationship among the features is already
high. Afterward, the word embeddings and image features are fed into a cross-
attention scheme to increase the inter-modality relationship. Here, the
$queries$ are learned from the word embeddings matrix, and $keys$ and $values$
are learned from the image features using $1\times 1$ convolutions with a
$scale$ of 0.5. Finally, after applying another max-pooling layer, we flatten
the feature matrix to produce a 1024-dimensional output.

####  3.4.3 Caption-Level Context Modeling

Similar to the word level of granularity, we take the caption embedding as
cues to attend to the global image features. Multi-head cross-attention [33]
has been employed to explore inter-modal associations between global image
features and caption embedding. First, we reshape the global image features
into size $8\times 64$, $\textbf{v}~{}\epsilon~{}\mathbb{R}^{8\times 64}$.
Then, we calculate the $queries$ from caption embedding,
$\textbf{c}~{}\epsilon~{}\mathbb{R}^{64}$ and the $keys$, and $values$ from
global features v using linear transformation. The total number of heads is 4.
The output of the multi-head cross-attention is a 64-dimensional vector, which
is followed by a layer normalization [1].

####  3.4.4 Features Aggregation Network

At the final stage of CFAM, we aggregate the contextualized features from the
word-level CM and caption-level CM. A dense layer learns the optimal
representation in a joint multimodal feature space. In our experiment, we
found that the optimal dimension of the dense layer is 768. We also
experimented to implement the CFAM module on the textual features extracted
from the BiLSTM text encoder. However, it does not perform well as BiLSTM
encoder does not produce contextual embeddings for word tokens.

###  3.5 Training Strategy

We train our proposed framework in two phases. First, we train the TFRM module
to update the contextual embeddings of the BERT encoder using the objective
function mentioned in Equation 3. We finetune the BERT encoder for only 4
epochs and use a mini-batch AdamW optimizer [24] with a weight decay of 0.02.
The learning rate is initialized to $0.00001$ and is warmed up to $0.0001$
after 2,000 training iterations. We then decrease it using the cosine decay
strategy [23] to $0.00001$. The batch size is set to 16. For the projection
head of both visual and textual streams, we employ the Adam optimizer [15]
with $\beta_{1}=0.5$ and $\beta_{2}=0.99$. The initial learning rate, in this
case, is set to $0.001$. In the second phase, we train the whole framework
end-to-end for 24 more epochs. However, the text encoder and the projection
head were trained with a similar setup to the first phase except with a lower
learning rate. Note that, in all the phases, the parameters of the FR model
were fixed.

Table 1: The 1:1 verification and 1:N identification (Rank-1) results of our CGFR framework with ArcFace trained on the MMCelebA dataset. The top row represents the results of ArcFace when pre-trained on MS1MV3 dataset [3].  Architectures | ROC Curve | TPR@FPR | Id(%)
---|---|---|---
| AUC | EER | 1e-4 | 1e-3 | Rank-1
Pre-trained ArcFace [2] | 85.27 | 23.48 | 16.75 | 25.73 | 17.56
Baseline | 93.98 | 13.50 | 21.92 | 31.28 | 38.78
CGFR | 98.51 | 6.65 | 62.83 | 63.28 | 57.65
Figure 5: ROC curves of 1:1 verification protocol of the proposed CGFR
framework with (a) ArcFace, (b) AdaFace FR models.

##  4 Experiments

###  4.1 Dataset

The Mulit-Modal CelebA-HQ [35] (MMCelebA) is a large-scale text-to-face
dataset, originally built for face image generation and facial attributes
editing. It has a total of 30,000 high-resolution face images from the CelebA-
HQ dataset [22]. Each image has 10 auto-generated captions from a total of 38
facial attributes.

###  4.2 Preprocessing

First, we implement standard data augmentation techniques such as random sub-
sampling, color jittering, horizontal flipping, rotation, and Gaussian noise
to degrade the image quality of the MMCelebA dataset. Then, we resize all the
images to 3 x 112 x 112. Sample preprocessed images are shown in the Figure 1.
The top row of Table 1 and Table 2 represent the performance of pre-trained
ArcFace and AdaFace models on this preprocessed set, respectively.

###  4.3 Implementation

We implemented our architecture using two NVIDIA Titan RTX GPUs. In our
experiment, we empirically set the hyper-parameters in Equations 3 as follows:
$\lambda_{1}$ = 1, and $\lambda_{2}$ = 0.5. Since we employ pre-trained
encoders, training the proposed framework is very fast. Finetuning pre-trained
BERT model for 4 epochs takes approximately 80 minutes on the MMCelebA dataset
while training the whole network end-to-end takes 8 hours. Also, due to the
parallel strategy of our proposed framework, the model has a very low time
complexity during inference. The inference time, which requires only one
forward process, is 220ms for an image-caption pair.

Table 2: The 1:1 verification and 1:N identification (Rank-1) results of our CGFR framework with AdaFace trained on the MMCelebA dataset. The top row represent the results of AdaFace when pre-trained on the WebFace4M dataset [44].  Architectures | ROC Curve | TPR@FPR | Id(%)
---|---|---|---
| AUC | EER | 1e-4 | 1e-3 | Rank-1
Pre-trained AdaFace [14] | 85.55 | 22.88 | 11.46 | 20.00 | 8.45
Baseline | 93.97 | 12.88 | 24.28 | 33.00 | 22.55
CGFR | 98.10 | 7.52 | 53.08 | 54.12 | 43.23

###  4.4 Performance Evaluation

##### ArcFace System:

We compare our proposed CGFR to the pre-trained ArcFace and the baseline
approach, as shown in Table 1. Our baseline is a dual-stream model employing a
BiLSTM text encoder with a linear fusion. We evaluated the CGFR model on one-
to-one matching (1:1), wherein a face probe is compared to a single gallery
image for verification, and one-to-many matching (1:N), wherein a face probe
is compared to all gallery images for identification. In the 1:1 verification
protocol, the proposed CGFR achieves the highest verification rates (VR). It
improved the performance of pre-trained ArcFace by 71.68% and the baseline by
50.74% on the equal error rate (EER) metric. Also, on the true positive rate
(TPR) and false positive rate (FPR) metrics, as illustrated in Figure 5(a),
our proposed CGFR improves the VR(%) by a significant margin. In particular,
as compared to the pre-trained ArcFace model, our framework boosts
TPR(@FPR=1e-4) from 16.75% to 62.83%.

Similarly, when compared to the baseline approach, our framework improves the
TPR(@FPR=1e-4) from 21.92% to 62.83% and TPR(@FPR=1e-3) from 31.28% to 63.28%.
Furthermore, in the 1:N identification protocol, the proposed CFGR secures an
improvement of 48.66% and 228.30% on Rank-1 identification accuracy over
baseline and pre-trained ArcFace, respectively. Therefore, as the results
show, the ArcFace FR model, which performs poorly due to low quality and
noise, could be significantly improved using natural language supervision.

##### AdaFace System:

In Table 2, we conduct further experiments to evaluate the performance of our
CGFR framework with an AdaFace FR model. As illustrated in Figure 5(b), our
framework significantly improves the VR(%) over the baseline and pre-trained
AdaFace model. It improves the performance of pre-trained AdaFace by 67.13%
and the baseline by 41.61% on the EER metric. Also, in 1:1 verification
protocol, under the evaluation metric of TPR(@FPR=1e-4), our framework boosts
the performance of pre-trained AdaFace from 11.46% to 53.08% and
TPR(@FPR=1e-3) from 20.00% to 54.12%. Furthermore, as reported in Table 2, the
Rank-1 identification accuracy of our CGFR framework improves by 91.71% over
the baseline. Thus, the VR (%) of the above-mentioned experiments proves the
effectiveness and generalizability of the proposed framework.

###  4.5 Analysis of CFAM

Table 3: Ablation experiments of different networks on the CFAM module. Experimental results verifies the notion of fusing cross-modal features at multiple granularities improves 1:1 VR(%). Modules | ROC Curve | TPR@FPR
---|---|---
| AUC | EER | 1e-4 | 1e-3
w/o modules | 89.96 | 18.27 | 15.95 | 21.42
Word (w/o Norm) | 86.36 | 22.27 | 8.63 | 20.63
Word (w/o SA) | 96.30 | 10.42 | 27.02 | 33.13
Word (SA+Norm) | 96.86 | 9.88 | 53.42 | 54.45
Word + Caption | 97.22 | 9.38 | 60.0 | 60.75
Word + Caption + FAN | 98.51 | 6.65 | 62.83 | 63.28
Figure 6: Face verification evaluation on different modules of the proposed
CFAM using ROC curves.

We design an ablation experiment to evaluate the effectiveness of the proposed
CFAM module. Specifically, we analyze the role of individual granularities and
attention schemes. Table 3 demonstrates that a fusion scheme without any
granularity decreases the VR(%). Thus it proves the need for fusing contextual
features at multiple granularities. In fact, under the evaluation metric of
TPR, word-level contextual modeling (CM) increases the performance from 15.95%
to 53.42% at FPR=1e-4 over the simple concatenation of multimodal features.
However, the choice of adding normalization [12, 1] and self-attention are
crucial to the performance of this module. We observe a drastic performance
drop of 12.16% in AUC without normalization (one batch norm [12] and two layer
norm layer [1]). Also, adding self-attention to the image features reduces the
EER from 10.42% to 9.88%.

We also observe that the fusion of word-level and caption-level CM improves
the VR(%) by 5.33% on EER and 12.32% on TPR@FPR=1e-4 compared to word-level
CM. Furthermore, the ablation study shows that the implementation of the
feature aggregation network further boosts the VR(%), improving TPR from 60.0%
to 62.83% (@FPR=1e-4). Figure 6 depicts the performance comparison of these
networks on ROC curves. Figure 6 illustrates that the proposed CFAM, with both
CM networks and the feature aggregation network, achieves the highest VR(%),
proving the effectiveness of applying fine-grain word-region interaction and
image-caption associations.

###  4.6 Future Work

In this work, we tested CGFR on distorted, low-quality images to mimic
surveillance scenarios and evaluated the impact of the caption on two state-
of-the-art FR models. Overall, we achieved a remarkable performance leap in
this setup. In the future, we plan to evaluate our model on various
resolutions and qualities of input images. Additionally, we aim to employ
large-scale image-caption pair datasets to assess the generalizability of our
proposed method.

##  5 Conclusion

We have introduced a new framework, called the caption-guided face recognition
(CGFR) model, to improve the performance of FR models using captions. Our
framework is based on a dual-stream model with a TFRM, and a CFAM modules.
CFAM applies fine-grained cross-modal feature interaction at multiple
granularities using cross-attention. In contrast, TFRM helps the framework to
learn an effective joint multimodal embedding space by realigning the text
embeddings with visual features. Our CGFR has significantly improved the
performance of state-of-the-art FR models. It has also enhanced the robustness
and reliability of the FR models by offering higher resistance to spoofing
attacks. Since, only manipulating the face becomes insufficient to deceive the
CGFR model, as the captions also provide discriminative and unique information
about the face image.

##  6 Acknowledgment

This material is based upon a work supported by the Center for Identification
Technology Research and the National Science Foundation under Grant 1650474.

## References

  * [1] J. L. Ba, J. R. Kiros, and G. E. Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [2] J. Deng, J. Guo, N. Xue, and S. Zafeiriou.  ArcFace: additive angular margin loss for deep face recognition.  In Proceedings of the Conference on Computer Vision and Pattern Recognition, pages 4685–4694, 2019.
  * [3] J. Deng, J. Guo, D. Zhang, Y. Deng, X. Lu, and S. Shi.  Lightweight face recognition challenge.  In IEEE International Conference on Computer Vision Workshops, pages 2638–2646, 2019.
  * [4] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In North American Chapter of the Association for Computational Linguistics, pages 4171–4186, 2019.
  * [5] E. Gonzalez-Sosa, J. Fierrez, R. Vera-Rodriguez, and F. Alonso-Fernandez.  Facial soft biometrics for recognition in the wild: recent works, annotation, and COTS evaluation.  IEEE Transactions on Information Forensics and Security, 13(8):2001–2014, 2018.
  * [6] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, and et al.  Generative adversarial nets.  In Advances in Neural Information Processing Systems, 2014.
  * [7] M. Guillaumin, T. Mensink, J. Verbeek, and C. Schmid.  Face recognition from caption-based supervision.  International Journal of Computer Vision, 96(1):64–82, 2012.
  * [8] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
  * [9] Z. He, W. Zuo, M. Kan, S. Shan, and X. Chen.  AttGAN: Facial attribute editing by only changing what you want.  IEEE Transactions on Image Processing, 28(11):5464–5478, 2019.
  * [10] S. Hochreiter and J. Schmidhuber.  Long short-term memory.  Neural Computation, 9(8):1735–1780, 1997.
  * [11] Q. Huang, L. Yang, H. Huang, T. Wu, and D. Lin.  Caption-supervised face recognition: Training a state-of-the-art face model without manual annotation.  In European Conference on Computer Vision, pages 139–155, 2020\.
  * [12] S. Ioffe and C. Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In Proceedings of the International Conference on Machine Learning, pages 448–456. pmlr, 2015.
  * [13] D. B. J. Lu, D. Parikh, and S. Lee.  ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  Advances in Neural Information Processing Systems, 32, 2019.
  * [14] M. Kim, A. K. Jain, and X. Liu.  AdaFace: Quality adaptive margin for face recognition.  In Proceedings of the Conference on Computer Vision and Pattern Recognition, pages 18750–18759, 2022.
  * [15] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [16] K.-H. Lee, X. Chen, G. Hua, H. Hu, and X. He.  Stacked cross attention for image-text matching.  In European Conference on Computer Vision, pages 201–216, 2018\.
  * [17] J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. C. H. Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  Advances in Neural Information Processing Systems, 34:9694–9705, 2021.
  * [18] S. Li, T. Xiao, H. Li, W. Yang, and X. Wang.  Identity-aware textual-visual matching with latent co-attention.  In Proceedings of the Conference on Computer Vision and Pattern Recognition, pages 1908–1917, 2017.
  * [19] W. Li, C. Gao, G. Niu, X. Xiao, H. Liu, J. Liu, H. Wu, and H. Wang.  UNIMO: Towards unified-modal understanding and generation via cross-modal contrastive learning.  Proceedings of the Association for Computational Linguistics, pages 2592–2607, 2021.
  * [20] R. Liao, D. Moyer, M. Cha, K. Quigley, S. Berkowitz, S. Horng, and et al.  Multimodal representation learning via maximization of local mutual information.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 273–283, 2021.
  * [21] Y. Liu, M. Ott, N. G., J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyano.  RoBERTa: A robustly optimized BERT pretraining approach.  CoRR, abs/1907.11692, 2019.
  * [22] Z. Liu, P. Luo, X. Wang, and X. Tang.  Deep learning face attributes in the wild.  In Proceedings of the Conference on Computer Vision and Pattern Recognition, pages 3730–3738, 2015.
  * [23] I. Loshchilov and F. Hutter.  SGDR: stochastic gradient descent with warm restarts.  In International Conference on Learning Representations, 2017.
  * [24] I. Loshchilov and F. Hutter.  Decoupled weight decay regularization.  In International Conference on Learning Representations, 2019.
  * [25] Q. Meng, S. Zhao, Z. Huang, and F. Zhou.  MagFace: A universal representation for face recognition and quality assessment.  In Proceedings of the Conference on Computer Vision and Pattern Recognition, pages 14225–14234, 2021.
  * [26] K. Niu, Y. Huang, W. Ouyang, and L. Wang.  Improving description-based person re-identification by multi-granularity image-text alignments.  IEEE Transactions on Image Processing, 29:5542–5556, 2020.
  * [27] A. Radford, J. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, and et al.  Learning transferable visual models from natural language supervision.  In Proceedings of the International Conference on Machine Learning, pages 8748–8763, 2021.
  * [28] V. Sanh, L. Debut, J. Chaumond, and T. Wolf.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  arXiv preprint arXiv:1910.01108, 2019.
  * [29] M. Schuster and K. Paliwal.  Bidirectional recurrent neural networks.  IEEE Transactions on Signal Processing, 45(11):2673–2681, 1997\.
  * [30] H. Shi, H. Li, F. Meng, and Q. Wu.  Key-word-aware network for referring expression image segmentation.  In European Conference on Computer Vision, 2018.
  * [31] J. Sun, Q. Deng, Q. Li, M. Sun, M. Ren, and Z. Sun.  AnyFace: Free-style text-to-face synthesis and manipulation.  In Proceedings of the Conference on Computer Vision and Pattern Recognition, pages 18687–18696, 2022.
  * [32] P. Tome, J. Fierrez, R. Vera-Rodriguez, and M. S. Nixon.  Soft biometrics and their application in person recognition at a distance.  IEEE Transactions on Information Forensics and Security, 9(3):464–475, 2014.
  * [33] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, and et al.  Attention is all you need.  Advances in Neural Information Processing Systems, 30, 2017.
  * [34] T. Wang, T. Zhang, and B. Lovell.  Faces à la Carte: Text-to-face generation via attribute disentanglement.  In 2021 IEEE Winter Conference on Applications of Computer Vision, pages 3379–3387, 2021.
  * [35] W. Xia, Y. Yang, J.-H. Xue, and B. Wu.  TediGAN: Text-guided diverse face image generation and manipulation.  In Proceedings of the Conference on Computer Vision and Pattern Recognition, pages 2256–2265, 2021.
  * [36] B. Xu, N. Wang, T. Chen, and M. Li.  Empirical evaluation of rectified activations in convolutional network.  arXiv preprint arXiv:1505.00853, 2015.
  * [37] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, and et al.  Show, attend and tell: Neural image caption generation with visual attention.  In Proceedings of the International Conference on Machine Learning, pages 2048–2057, 2015.
  * [38] T. Xu, P. Zhang, Q. Huang, H. Zhang, Z. Gan, X. Huang, and X. He.  AttnGAN: Fine-grained text to image generation with attentional generative adversarial networks.  In Proceedings of the Conference on Computer Vision and Pattern Recognition, pages 1316–1324, 2018.
  * [39] L. Ye, M. Rochan, Z. Liu, and Y. Wang.  Cross-modal self-attention network for referring image segmentation.  In Proceedings of the Conference on Computer Vision and Pattern Recognition, pages 10494–10503, 2019.
  * [40] X. Yin, Y. Tai, Y. Huang, and X. Liu.  FAN: Feature adaptation network for surveillance face recognition and normalization.  In Asian Conference on Computer Vision, 2020.
  * [41] H. Zhang, J. R. Beveridge, B. A. Draper, and P. J. Phillips.  On the effectiveness of soft biometrics for increasing face verification rates.  Computer Vision and Image Understanding, 137:50–62, 2015.
  * [42] Y. Zhang and H. Lu.  Deep cross-modal projection learning for image-text matching.  In European Conference on Computer Vision, pages 686–701, 2018\.
  * [43] S. Zhu, S. Liu, C. C. Loy, and X. Tang.  Deep cascaded bi-network for face hallucination.  In European Conference on Computer Vision, pages 614–630, 2016\.
  * [44] Z. Zhu, G. Huang, J. Deng, Y. Ye, J. Huang, X. Chen, and et al.  WebFace260M: A benchmark unveiling the power of million-scale deep face recognition.  In Proceedings of the Conference on Computer Vision and Pattern Recognition, pages 10492–10502, 2021.