# Search for dark matter in the hidden-photon sector with a large spherical
mirror

${}^{\,a}$, Kai Daumiller${}^{\,a}$, Babette Döbrich${}^{\,b}$, Ralph
Engel${}^{\,a}$, Joerg Jaeckel${}^{\,c}$ Marek Kowalski${}^{\,d\,e}$, Axel
Lindner${}^{\,d}$, Hermann-Josef Mathes${}^{\,a}$, Javier Redondo${}^{\,f}$,
Markus Roth${}^{\,a}$, Christoph M. Schäfer${}^{\,a}$, Ralf Ulrich${}^{\,a}$
[The FUNK Experiment]
a Institute for Nuclear Physics, Karlsruhe Institute of Technology (KIT),
Germany
b Physics Department, CERN, Geneva, Switzerland
c Institute for Theoretical Physics, Heidelberg University, Germany
d Deutsches Elektronen Synchrotron DESY, Hamburg, Germany
e Department of Physics, Humboldt University, Berlin, Germany
f Department of Theoretical Physics, University of Zaragoza, Spain
∗ E-mail<EMAIL_ADDRESS>
###### Abstract:

If dark matter consists of hidden-sector photons which kinetically mix with
regular photons, a tiny oscillating electric-field component is present
wherever we have dark matter. In the surface of conducting materials this
induces a small probability to emit single photons almost perpendicular to the
surface, with the corresponding photon frequency matching the mass of the
hidden photons. We report on a construction of an experimental setup with a
large ${\sim}14$ m2 spherical metallic mirror that will allow for searches of
hidden-photon dark matter in the eV and sub-eV range by application of
different electromagnetic radiation detectors. We discuss sensitivity and
accessible regions in the dark matter parameter space.

##  1 Introduction

There is a number of convincing astrophysical and cosmological evidences that
a large fraction of the energy density in the universe must be composed of
invisible non-baryonic matter or dark matter (DM) [2]. The most explored
options for explaining DM are extensions of the Standard Model (SM) predicting
axions and weakly-interacting massive particles. In recent years attention has
been turned also to alternatives or weakly-interacting slim particles (WISP),
as e.g. axion-like particles or hidden photons (HP) [3]. WISP could be
nonthermally produced in the early universe and survive as cold DM until today
(see e.g. [4, 5, 6, 7] for reviews).

Figure 1: _Left:_ A $6{\times}6$-segmented spherical mirror [14], the same as
used for the fluorescence-detector telescopes of the Pierre Auger Observatory,
where each segment is 65 cm high with width gradually decreasing from 65 cm to
55 cm. _Right:_ View of the support structure for the segments at the back of
the mirror. The center of the mirror sphere is also marked with a point.

Here we consider HP, which are additional light U(1) gauge bosons that
kinetically mix with the (vis-à-vis uncharged) SM particles [8]. The effective
Lagrangian can be written as

$\mathcal{L}=-\tfrac{1}{4}(F_{\mu\nu}F^{\mu\nu}+X_{\mu\nu}X^{\mu\nu})+J^{\mu}A_{\mu}+\tfrac{m^{2}}{2}X_{\mu}X^{\mu}-\tfrac{\chi}{2}F_{\mu\nu}X^{\mu\nu},$
(1)

where $F_{\mu\nu}=\partial_{\mu}A_{\nu}-\partial_{\nu}A_{\mu}$ is the field
strength tensor of the photon and $X_{\mu\nu}$ that of the corresponding HP
field, and $J^{\mu}$ the electric current. Large regions of the HP mass $m$
and mixing parameter $\chi$ space are compatible with the observed DM
signatures [9, 10] and are already searched for with many experimental
methods: haloscopes, helioscopes, and light-shining-through-a-wall methods
[9]. Our experiment, Finding U(1)s of a Novel Kind (FUNK), is dedicated to
observe possible candidates for HP with the dish-antenna method [11, 12],
where the faint electromagnetic waves, emitted almost perpendicularly to
conducting surfaces, are focused by using a spherically shaped mirror. The HP
induced light emerging from any part of the mirror is thus gathered in the
center of the sphere where various detectors can be suitably placed. The DM
nature of the signal can be verified by observation of the expected small
daily and seasonal movements of the spot [13]. A similar experiment is
currently being performed in Tokyo [21, 22].

##  2 Mirror

Figure 2: _Left:_ Frosted glass screen (with a SMD LED on the front face),
linear stage with stepper motor, and a Raspberry Pi micro-computer as
controller. _Right-top:_ Schematical description of the sphere-center
reference point with two crossing laser beams (red along the mirror axis,
green laterally). Mirror segments were adjusted until focused and converging
into this point. _Right-bottom:_ Photo of the frosted glass and the crossing
laser beams as seen in darkness.

Figure 3: _Left:_ Back side of the mirrors exposing the mounts of the
segments. _Middle:_ Screws enabling longitudinal offset movement and fine
setting of the pointing direction via three-point tilting of the ball joint.
_Right:_ Already aligned segments are covered with non-reflecting foil to keep
only the misaligned beams in the reference point. At the bottom of the photo
the frosted-glass screen can be seen placed on a linear stage. Figure 4:
_Left:_ Contours of radial distance from the image center of mass (CM)
containing certain amount of total light (in steps of 10%) for the initial
assembly of the mirror. The three insets show spot images in various positions
marked by blue lines (middle for smallest spread). Note the focusing of
individual beams from segments but without their convergence (bottom inset).
_Right:_ The same as on the left but after the realignment.

Figure 5: _Left:_ Entropy of the image at different longitudinal positions of
the screen (blue points for the initial mirror assembly and green points after
the readjustment of individual segments). _Right:_ An image obtained when the
LED is driven off-center towards the mirror. Reflected beams are not converged
yet so that they form a matrix where each of the squares corresponds to
individual mirror segments, revealing their aberrations. The central black dot
is a shadow of the LED with two horizontal electricity cables.

For the purposes of the FUNK experiment we reused spherical mirror components
that were originally produced for the Schmidt telescopes of the fluorescence
detector of the Pierre Auger Observatory [14]. At the KIT Campus Nord we
assembled them inside of a windowless air-conditioned experimental hall of
${\sim}20$ m diameter with 2 m thick concrete walls. The whole mirror is
composed of three different segment shapes that are arranged into a
$6{\times}6$ matrix (see Fig. 1). The height of the segments is exactly 65 cm
but moving away from the mirror equator their width gradually decreases from
65 cm to 55 cm. The mirror segments were produced by milling a cast aluminium
backing to approximate spherical shape and gluing it at high temperatures with
a thinner AlMgSi sheet [15]. The front reflective surface was then precisely
milled with diamond tools and protected with electrochemical anodization,
resulting in a final reflectivity of $88\pm 1\%$ in visible and UV range. The
segments are attached to the mirror support structure via single-point ball
joint. The extreme four corners of the mirror form a $3.7{\times}3.1$ m
rectangle, nevertheless, due to the spherical geometry with radius 3.4 m, the
total area of the mirror front surface is 14.56 m2.

The initial assembly of the mirror was performed with help of an existing
laser tool according to the specifications used for the telescopes of the
Auger Observatory. Unfortunately, the prototype components from the
construction were mirror segments which mainly did not pass strict quality
requirements for their radius of curvature or other relevant parameters. To
quantify the optical quality of the mirror we built a movable imaging platform
at the center of the mirror sphere (see Fig. 2-left). It consists of a
frosted-glass screen with a yellow-green SMD LED as a light source. Note that
in the case of an ideal spherical mirror, light from a point source placed
exactly in the center of the sphere will be reflected and focused back onto
the source itself. With a slight lateral offset of the source the resulting
image can be observed in the screen next to it, with the offset-related
distortions being very small. This reflected light pattern was then observed
with a CCD camera. Placing the screen on a movable linear stage enabled us to
obtain cross-section images of the converging light beams from the mirror
segments in various longitudinal positions along the optical axis of the
mirror. In Fig. 4 such a scan through the central region is presented with
contour lines for distances (from the image center of mass) containing certain
fractions of the total light (in 10% steps). As seen in Fig. 4-left, the
initial assembly of the mirror produced in a minimal region (middle inset) a
very large focal spot size with radius ${\sim}6$ mm (for 90% of total light).
We also observed that most of the individual segments had good focus points
but at distances different than the nominal radius (see lower inset),
indicating that by realigning individual segments we can achieve better
convergence and smaller final spot size. This position was chosen as a new
dedicated mirror center and permanently marked for future reference with two
crossing laser beams (see Fig. 3-right) which will later-on also help to
accurately position the measuring equipment.

In a dedicated readjustment campaign we aligned all mirror segments, either by
offsetting them longitudinally or changing their pointing direction (see Fig.
3) until the sharpest possible spot has been produced by the corresponding
segment. In this way we compensated for the non-nominal values of the
curvature while other aberrations, nicely seen in an off-center image in Fig.
5-right, remained. Nevertheless, out of 36 segments, we identified only 5 that
had the individual spot sizes considerably larger than the LED source size,
mainly due to having two different radii of curvature in the two perpendicular
lateral directions, in Fig. 5-right identified as having more rectangular
instead of square shapes. The 90% spot radius now decreased to only ${\sim}2$
mm which enabled us to use a small CCD as a detector. Note that the seasonal
oscillation of the HP spot due to the relative movement of the mirror in the
DM frame is of the same order [12]. A way to quantify the sharpness of the
image with a single observable is using entropy, defined as
$S=-\sum_{i}p_{i}\ln p_{i}$ where $p_{i}$ is the (normalized) intensity in
image pixel $i$. From the entropy an effective number of pixels or area
containing relevant signal can be defined as $N_{\text{eff}}=\exp(S)$. In Fig.
5-left, entropy of the longitudinal scan images are compared. While the
initial mirror setup was exhibiting two entropy minima, a deeper one when
individual segments were focused and a shallower one for when the individual
beams crossed 20 mm away, the entropy after the readjustment shows only one
deep minimum with at least 6 times smaller effective spot area.

##  3 Preliminary results

Figure 6: _Left:_ PCO Sensicam camera used for the preliminary measurement.
_Right:_ Quantum efficiency of the camera CCD (see blue VGA curve) from [16].

Figure 7: _Left:_ The mounted camera with removed lens and its CCD placed in
the reference point. The Thorlabs Motorized Filter Flipper is used to drive a
shutter. _Right:_ The evolution of intensity (blue) with time in a longer
example run. Long term deviations are due to temperature (red) dependence of
the CCD intensity.

After the final preparation of the mirror we performed preliminary
measurements with the CCD camera that was used to obtain the images for Fig. 4
and 5 scans. The camera is a PCO sensicam with a half-inch diameter, VGA
resolution ($640{\times}480$) CCD sensor, internally Peltier-cooled to
$-15^{\circ}$C. The quantum efficiency is given in Fig. 6. The collected
charge to pixel count is specified [16] at $7.5\,e^{-}$/count, pixel readout
noise is ${\sim}14\,e^{-}$, and dark current per pixel is ${<}0.1\,e^{-}$/s.
We removed the lens, placed the CCD into the reference point, and added a
motorized filter flipper, see Fig. 7-left, to drive a large shutter. Since the
spot size is not much smaller than the whole CCD, the relevant quantity for
this sensor is the total intensity (integral over all pixels of the CCD). In
Fig. 7-right an example of a long-term intensity measurement with a closed
shutter is shown, together with the temperature of the camera enclosure (red),
revealing a temperature correlation on the order of $-1$%/K. In Fig. 8-left an
example of image intensities (in one of many long-term measurement runs we
performed) is shown for open and closed shutter. The signal is defined as the
difference of these two measurements. In Fig. 8-right a distribution of
possible signals is shown based on noise properties of individual images. The
actual signal is shown with a red vertical bar, being perfectly compatible
with the estimated noise.

No signal from HP is thus observed, allowing us to set in this test example
preliminary upper limits [13] on the mixing parameter $\chi$ in the
sensitivity range of the CCD, see Fig. 9.

Figure 8: Example of a 7 day measurement run with 1000 s exposures. _Left:_
Total image intensities for open (blue) and closed (red) shutter. _Right:_
Expected distribution of intensity differences (PDF) is shown in violet (side
lobes are artifacts of the FFT convolution) and the total observed difference
of the run with red vertical line). Figure 9: Exclusion plot in the mixing
parameter vs. hidden photon mass parameter space. Meaning of some labels:
measurements on cosmic microwave background (CMB), light-shining-through-wall
experiments (LSW), horizontal-branch stars (HBS), and the green line
corresponds to improved solar limits [19, 20]. Our preliminary result with the
CCD camera is shown in light green (Test result). Plot adapted from [21]
(Tokyo mirror), for summary and labels see [23].

##  4 Conclusions and future plans

As a first step of the FUNK experiment we assembled the segments of a
prototype mirror of the Pierre Auger Observatory to obtain a large ${\sim}14$
m2 spherical mirror, suitable for dark matter searches in the hidden photon
sector. With manual readjustments we managed to reduce the radius of the spot
size below ${\sim}2$ mm which enables us to use small and highly sensitive
detectors of electromagnetic radiation.

While the initial tests were made with a simple CCD camera we are preparing
high-sensitivity measurements with a low dark-current CCD and a low noise PMT.
Furthermore, in the near future we want to extend the search for possible HP
mass into the MHz [17], GHz [18], and THz range.

We gratefully acknowledge partial funding from the the Helmholtz Alliance for
Astroparticle physics (HAP), funded by the Initiative and Networking Fund of
the Helmholtz Association.

## References

  * [1]
  * [2] G.B. Gelmini, _TASI 2014 lectures: The hunt for dark matter_ , [arXiv: 1502.01320].
  * [3] A.E. Nelson and J. Scholtz, _Dark Light, Dark Matter and the Misalignment Mechanism_ , _Phys. Rev. D_ 84 (2011) 103501, [arXiv: 1105.2812].
  * [4] J. Jaeckel and A. Ringwald, _The Low-Energy Frontier of Particle Physics_ , _Ann. Rev. Nucl. Part. Sci._ 60 (2010) 405, [arXiv: 1002.0329].
  * [5] A. Ringwald, _Searching for axions and ALPs from string theory_ , _J. Phys. Conf. Ser._ 485 (2014) 012013, [arXiv: 1209.2299].
  * [6] J. Jaeckel, _A force beyond the Standard Model - Status of the quest for hidden photons_ , _Frascati Phys. Ser._ 56 (2012) 172, [arXiv: 1303.1821].
  * [7] K. Baker et al., _The quest for axions and other new light particles_ , _Ann. Phys._ 525 (2013) A93–A99.
  * [8] B. Holdom, _Two U(1)’s and Epsilon Charge Shifts_ , _Phys. Lett. B_ 166 (1986) 196.
  * [9] P. Arias, D. Cadamuro, M. Goodsell, J. Jaeckel, J. Redondo, and A. Ringwald, _WISPy cold dark matter_ , _JCAP_ 06 (2012) 013.
  * [10] P.W. Graham, J. Mardon, and S. Rajendran, _Vector Dark Matter from Inflationary Fluctuations_ , [arXiv: 1504.02102].
  * [11] D. Horns, J. Jaeckel, A. Lindner, A. Lobanov, J. Redondo, and A. Ringwald, _Searching for WISPy Cold Dark Matter with a Dish Antenna_ , _JCAP_ 04 (2013) 016.
  * [12] J. Jaeckel and J. Redondo, _An antenna for directional detection of WISPy dark matter_ , _JCAP_ 111 (2013) 016.
  * [13] B. Döbrich, K. Daumiller, R. Engel, M. Kowalski, A. Lindner, J. Redondo, and M.Roth, _Hidden Photon Dark Matter Search with a Large Metallic Mirror_ , _Proc. 10th Patras Workshop on Axions, WIMPs and WISPs, CERN, 29 June – 4 July 2014_ , [arXiv: 1410.0200].
  * [14] J. Abraham et al. [Pierre Auger Collaboration], _The Fluorescence Detector of the Pierre Auger Observatory_ , _Nucl. Instrum. Meth. A_ 620 (2010) 227–251.
  * [15] J. Abraham et al. [Pierre Auger Collaboration], _Properties and performance of the prototype instrument for the Pierre Auger Observatory_ , _Nucl. Instrum. Meth. A_ 523 (2004) 50–95.
  * [16] _Sensicam, Sensicam QE, Sensicam EM – Operating Instructions_ , PCO Imaging (2004); http://www.pco.de.
  * [17] A. Aab et al. [Pierre Auger Collaboration], _Probing the radio emission from air showers with polarization measurements_ , _Phys. Rev. D_ 89 (2014) 052002.
  * [18] R. Šmída et al. [CROME experiment], _First Experimental Characterization of Microwave Emission from Cosmic Ray Air Showers_ , _Phys. Rev. Lett._ 113 (2014) 221101.
  * [19] H. An, M. Pospelov, and J. Pradler, _New stellar constraints on dark photons_ , _Phys. Lett. B_ 725 (2013) 190, [arXiv: 1302.3884].
  * [20] J. Redondo and G. Raffelt, _Solar constraints on hidden photons re-visited_ , _JCAP_ 1308 (2013) 034, [arXiv: 1305.2920].
  * [21] J. Suzuki, T. Horie, Y. Inoue, and M. Minowa, _Experimental Search for Hidden Photon CDM in the eV mass range with a Dish Antenna_ , accepted for publication in _JCAP_ , [arXiv: 1504.00118].
  * [22] J. Suzuki, Y. Inoue, T. Horie, and M. Minowa, _Hidden photon CDM search at Tokyo_ , [arXiv: 1509.00785].
  * [23] M. Goodsell, J. Jaeckel, J. Redondo, and A. Ringwald, _Naturally light hidden photons in LARGE volume string compactifications_ , _JHEP_ 11 (2009) 027.


# Detecting Emerging Symptoms of COVID-19 using
Context-based Twitter Embeddings

Roshan Santosh1, H. Andrew Schwartz2, Johannes Eichstaedt3,
Lyle Ungar1, Sharath Chandra Guntuku1
1University of Pennsylvania 2Stonybrook University 3Stanford University
<EMAIL_ADDRESS>
###### Abstract

In this paper, we present an iterative graph-based approach for the detection
of symptoms of COVID-19, the pathology of which seems to be evolving. More
generally, the method can be applied to finding context-specific words and
texts (e.g. symptom mentions) in large imbalanced corpora (e.g. all tweets
mentioning #COVID-19). Given the novelty of COVID-19, we also test if the
proposed approach generalizes to the problem of detecting Adverse Drug
Reaction (ADR). We find that the approach applied to Twitter data can detect
symptom mentions substantially before being reported by the Centers for
Disease Control (CDC).

##  1 Introduction

The COVID-19 pandemic has interrupted many everyday behaviors. SARS-CoV-2 is a
relatively new virus and gaps in knowledge persist about how it affects the
body, and consequently, its symptoms and symptom severity. In the early phases
of the pandemic, patients and providers in affected areas used social media to
exchange information about symptoms and clinical treatment Iacobucci (2020);
Stokes et al. (2020). During COVID-19, the use of social media has increased
dramatically ($>$20%) as individuals shelter in place Venkatraman (2020).
While social media is non-representative and contains misinformation Singh et
al. (2020), it provides an open forum for individuals to share their
perceptions, concerns Giorgi et al. (2020), understanding of health and
science Yang et al. (2018), and mental health status during emergencies when
wide spread polling may not be available Guntuku et al. (2020).

Social media could enable early symptom discovery for diseases such as
COVID-19 where the pathology is not completely known and our knowledge of it
is evolving Del Rio and Malani (2020). The most prominent symptoms such as
fever, cough, and shortness of breath were known early on during the COVID-19
pandemic. However, others such as changes in smell/taste, body aches, and
diarrhea were added later to the symptom list by the CDC Grant et al. (2020).

Using social media to gather information on public health is a growing focus
of research, with a special emphasis on discovering side effects of drugs
(pharmacovigilance) O’Connor et al. (2014), often using labeled datasets to
build supervised machine learning models Luo et al. (2017).

We propose a graph-based natural language processing framework to
automatically detect emerging symptoms using Twitter data. Our approach is
built on the hypothesis that by identifying token embeddings that capture the
context of symptom mentions, new tokens used in a similar context can be
identified through embedding similarity Devlin et al. (2018). Our approach
shares similarities with the idea of lexicon development Bontcheva et al.
(2013), which uses an unsupervised graph-based approach for labeling new words
given a few labeled words. However, the graph is initiated with words of
interest that have already been identified.

Our method’s focus on a specific context allows it to search through large
imbalanced corpora to identify context-specific (e.g. symptoms) tweets. This
differentiates it from previous works by Wu et al. (2019); Mpouli et al.
(2020) that identify domain specific lexicon. Further, the approach by Wu et
al. (2019) relies on a domain specific corpus and topic modeling to build a
lexicon, which would require the construction of a symptom-specific COVID-19
corpus.

Figure 1: Comparative timeline of symptom detection by our approach against
CDC reporting

##  2 Method

As is the case with several applications involving creating word lists
associated with a construct or topic Das and Smith (2012), symptom mentions
associated with COVID-19 come in different forms and shapes - often difficult
to curate in advance Rúa (2007). The approach we propose assumes that we know
at least one word of interest (i.e., a seed word) along with few corresponding
seed texts where the seed word has been used in the desired context. For the
case of emerging symptom detection, cough, a seed text could be ‘I have a dry
cough, chest pain and feeling lethargic as hell plus a headache’.

###  2.1 Manual Context-Text Approach

Given the seed word and corresponding seed texts, BERT (bert-base) embeddings
Devlin et al. (2018) for the seed word are extracted from each of the texts.
The BERT embedding for each token was computed by summing the hidden states of
the last 4 layers of BERT. Individual embeddings from each of the seed texts
are then averaged to generate a representative embedding for the seed word. We
use 5 seed texts that capture part of the considerable variance associated
with the symptom context.

Using the representative embedding for the seed word, an exhaustive search is
performed across the dataset at a token level to identify tokens that are most
similar to the seed word, where similarity is measured using cosine similarity
(one minus cosine distance). All tokens with a similarity value less than a
minimum threshold (set empirically at 0.3) are excluded. Similarity scores of
all occurrences of a given word are averaged.

Training Details We fine-tuned a bert-base-uncased model using the masked
language modeling protocol with the following parameters: Num. Epochs: 5,
Learing Rate: 1e-5, Batch Size: 8, Max Sequence Length: 128. Embeddings were
extracted from the model by summing the hidden states of the last 4 BERT
layers, resulting in 768-dim embeddings. HuggingFace library Wolf et al.
(2019) was used for loading pre-trained model weights as well as Language
Model training.

###  2.2 Graph-based Iterative Training Approach

The previous model required text for every new seed word and did not allow
multiple runs with different seeds to learn from each other. To address this,
we propose an iterative search model to develop a similarity-based word graph,
retaining the search methodology of the earlier approach, but including a
graph element and a trainable search parameter that improves the detection of
context-specific words with increased iterations.

The directed and weighted word graph of the model represents connections
between tokens. Each node in the graph corresponds to a word and is
characterized by the representative embedding of the word. The edges have
weights corresponding to the similarity score between the connected words
(nodes). The second component of the model is the so-called ‘Context
Embedding’, $CEmb$. The context embedding is conceptualized to be an embedding
vector that represents the specific context that we are interested in.
Initialized by the representative embedding of the seed word, the context
embedding incorporates embeddings from other words over iterations, to develop
into a more robust representation of the specified context. 111Code available
at https://github.com/rsk2327/Covid-Symptoms-NLP.

###  2.3 Algorithm

##### Initialization

Initialize graph G by setting the root node with the representative embedding
of the seed word. Initialize a queue Q by adding the seed word to it. The
context embedding CEmb is also initialized to the representative embedding of
the seed word. $\textbf{CEmb}\leftarrow Emb\\{\text{Seed word}\\}$, where
$Emb\\{x\\}$ denotes the representative embedding of token $x$.

##### Procedure

The specific steps used in the algorithm are as follows:

  1. 1.

Pop next word from Q, denoted by $t$. Initialise a new node in G corresponding
to $t$ and set the node embedding to $Emb\\{t\\}$.

  2. 2.

Initialise the query embedding $q$ as $q\leftarrow
k*\textbf{CEmb}+(1-k)*Emb\\{t\\}$.

  3. 3.

Iterate through all tokens in the data, comparing their embeddings against the
query embedding $q$. All tokens with similarity less than the minimum
similarity threshold $minSimThresh$ are dropped.

  4. 4.

Select the top $n$ words based on their similarity to $q$. Add these words to
Q. Instantiate new nodes (if one does not already exist) for these words in G
and add outgoing edges from $t$ to these new nodes.

  5. 5.

If all words for given depth are explored, the top $m$ words corresponding to
that depth are selected based on similarity to CEmb. The context embedding is
then updated by averaging $CEmb$ with the representative embeddings of the
selected words (Eq 1).

$\displaystyle\textbf{CEmb}\leftarrow\frac{\textbf{CEmb}+\sum_{i=0}^{m}\text{Emb}\\{x_{i}\\}}{m+1}$
(1)

  6. 6.

Stop iterations when either Q is empty or when the maximum depth $maxDepth$ of
G is achieved. Otherwise, repeat from Step 1.

##  3 Experiments

###  3.1 Manual Context-Text Approach

We tested our approach on a Twitter dataset containing tweets related to
COVID-19, collected between March 12 to April 23. Our experiments were run on
a random subset of this dataset containing 1 Million tweets.

Table 1: Words returned (and their cosine similarity scores) by Manual
Approach described in Section 2.1 and Graph-based Iterative Approach described
in Section 2.2

Seed : Cough
---
(Manual Approach)
Word | Sim.
fever | 0.67
throat | 0.62
##tis (pneumonitis) | 0.61
headache | 0.61
nose | 0.59
breathing | 0.58
congestion | 0.57
##itis (bronchitis) | 0.57
taste | 0.55
##raine (migraine) | 0.54

Seed : Cough
---
(Graph Approach)
Word | Sim.
vomiting | 0.84
fever | 0.83
throat | 0.82
congestion | 0.75
headache | 0.71
coughing | 0.70
asthma | 0.69
##itis (bronchitis) | 0.64
nausea | 0.63
##hea (diarrhea) | 0.45

Our objective for this dataset is to identify new symptoms of COVID-19
mentioned in tweets. We ran tests with cough, an established symptom, as the
seed word. Seed texts (tweets) were selected where cough was used as a symptom
to ensure that the correct context is captured. Top 10 results are shown in
Table 1. The scores represent the cosine similarity scores of the context
embedding of cough with the averaged embeddings of the detected words. With
just a single seed word and corresponding text as input, the model could
identify key symptoms of COVID.

###  3.2 Graph-based Iterative Training Approach

We evaluate our graph-based approach on 2 different datasets, with each
dataset having a different context - 1) COVID-19 Symptom Detection; and 2)
Adverse Drug Reaction Identification.

####  3.2.1 COVID-19 Symptom Detection

With the 1 million COVID-19 tweet dataset, we use cough as the seed word, $k$
= 0.3, $maxDepth$ = 3 and $n$ = 5 to test our approach. The resulting graph
from our model is shown in Figure 2. The size of the nodes represent the
number of occurrences of a token as a symptom while the color intensity of the
nodes represent the similarity values computed for the node during the graph
building process (shown in Appendix).

Figure 2: Symptom model graph for COVID-19 Tweet dataset

We observe that the model identified a wide range of symptoms ranging from
common symptoms like fever, fatigue to less common ones like headache,
vomiting, (chest) congestion, nausea, mig-##raine.

##### Evaluation

Though a quantitative evaluation of our approach is not straightforward as
there is no definitive ‘ground truth’, we evaluate our approach by computing
the precision in detecting correct words that fit the specified context.

For the problem of symptom detection, precision is calculated as the
percentage of the actual symptoms detected by our model. Given that our model
outputs a ranked list of words, precision is computed by looking at the top
$p$ results, where $p$ represents the threshold for computing precision (Table
2).

Table 2: Precision for Symptom Detection Model | Seed word | Precision
---|---|---
p = 5 | p = 10 | p = 20
Manual | cough | 0.8 | 0.9 | 0.8
Manual | fever | 1.0 | 0.9 | 0.75
Manual | fatigue | 0.8 | 0.8 | 0.75
Graph | cough | 1.0 | 0.9 | 0.9

Through a manual inspection of the top 100 results, rare to-be-confirmed
symptoms like lack of appetite, skin/eye irritation, vertigo, anemia were
detected. This marks a key utility of our approach as it helps generate
potential symptom candidates which can guide further evaluation.

####  3.2.2 Adverse Drug Reaction (ADR) Detection

For the second task, we use an annotated ADR dataset Sarker and Gonzalez
(2015), where 13% of the tweets are labeled as ADR. The objective of this task
is the identification of words denoting adverse drug reactions. Therefore, the
specific context of interest is different from the previous dataset where it
was identification of symptoms of a disease. By testing our model on this
dataset, we also test the ability of our approach to generalize to new tasks.

For the experiment, the seed word used is pain. $k$ = 0.2, $maxDepth$ = 3, and
$n$ = 5. The resulting graph from our model is shown in Figure 3.

Figure 3: ADR model graph for Statins dataset

Some of the key ADR identified include inflammation, bleeding, muscle (pain),
(skin) lesions, tremors, discomfort, and (calcium) deposits.

##### Evaluation

Similar to COVID-19 symptom detection evaluation, we evaluate the model’s
performance for ADR detection, where a positive word represents an adverse
reaction to a drug (Table 3).

Table 3: Precision for ADR Detection Model | Seed word | Precision
---|---|---
p = 5 | p = 10 | p = 20
Manual | pain | 1.0 | 0.9 | 0.8
Graph | pain | 1.0 | 0.9 | 0.95

##  4 Discussion

The COVID-19 pandemic evolved in a global climate of confusion and
uncertainty. The professional and lay public alike speculated on disease
course, severity, and symptoms. The COVID-19 symptoms first observed appeared
to be largely non-specific to COVID-19 (e.g., cough, fever). Finding COVID-
specific symptoms (such as the sudden loss of the sense of smell) is important
and potentially of clinical significance as large populations are being risk-
assessed Wu and McGoogan (2020). The “digital exhaust” of social media encodes
informal case reports of symptoms and discussions of media content about the
virus alike. In principle, it could allow for the generation of a “master
list” of COVID-19 symptom candidates, which the public health and medical
community can, in turn, consider for further evaluation as COVID-specific
markers Chan et al. (2020).

In this study, we present an iterative learning approach to generate such a
“master” list of COVID-19 symptoms, using the identification of words matching
a specific symptom context. Our approach is intended to work at the
intersection of human-machine interaction and provide a ranked list of most
important keywords for a user to explore and validate. A typical workflow
would involve users from the medical/ pharmacovigilance domain inputting a
small number of known keywords of interest, detecting the emerging signals
(symptoms/ADR) output by the model, and validating them using the
corresponding text. This allows for very early detection of symptoms/ADR from
social media platforms without the need to manually scavenge large corpora.

Our initial experiments with Word2Vec Mikolov et al. (2017), Glove Pennington
et al. (2014), and latent Dirichlet allocation Blei et al. (2003) for the same
task, did not perform well since the similarity between tokens was the same
irrespective of the context in which the word was being used. So with ‘cough’
as the seed word, highly correlated words like ‘fever’ were detected by these
approaches, while words like ‘congestion’, ‘nausea’, ‘diarrhea’ were not.
Consequently, we developed an iterative graph-based training approach using
BERT embeddings that are able to take into account the context in which words
are used.

Through a preliminary evaluation, our approach shows high sensitivity in
detected words. The approach detected headache, chills, sore throat, diarrhea,
and other symptoms around a month before the CDC reported them (Figure 1).
Given the novelty of COVID-19, the current method is hard to evaluate. Recall
and other metrics that depend on false negatives are not useful as there is no
definitive ’ground truth’. This is even more true for COVID-19 as the symptoms
are evolving and newer symptoms are being observed even after several months
from the first case. We, therefore, considered the approach in the more
studied context of detecting adverse drug reactions and show that the approach
generalizes to this domain. The annotations in ADR vary from very specific
entities like ‘pain’, ‘headache’ to very broad phrases like ‘a tingling
sensation’ Nikfarjam et al. (2019). Though our approach does not extract
multi-word terms, given that the typical workflow is expected to focus on the
validation of symptoms using their corresponding full-length texts, the
utility of multi-word terms is minimal.

The approach relies on the Context Embedding contribution parameter ($k$). By
varying $k$, we observe a phenomenon analogous to ‘Exploration vs
Exploitation’ Coggan (2004), which, in principle, means that this method can
be calibrated for different use cases. In the early phase of the disease, for
example, a low $k$ parameter may be chosen to aid in the generation of symptom
candidates to be considered in light of the emerging clinical literature on
COVID-19 and known physiological and biological interactions in the human
body. A high $k$ may be chosen to yield the subset of COVID-19 symptoms that
are more robustly associated with the disease, at the cost of missing
infrequent (albeit potentially specific) disease markers.

In summary, this study demonstrates that the digital traces of social media
can be mined effectively to detect emerging symptoms in the population during
a public health emergency.

## References

  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003.  Latent dirichlet allocation.  _Journal of machine Learning research_ , 3(Jan):993–1022.
  * Bontcheva et al. (2013) Kalina Bontcheva, Leon Derczynski, Adam Funk, Mark A Greenwood, Diana Maynard, and Niraj Aswani. 2013.  Twitie: An open-source information extraction pipeline for microblog text.  In _Proceedings of the international conference recent advances in natural language processing RANLP 2013_ , pages 83–90.
  * Chan et al. (2020) AKM Chan, CP Nickson, JW Rudolph, A Lee, and GM Joynt. 2020.  Social media for rapid knowledge dissemination: early experience from the covid-19 pandemic.  _Anaesthesia_.
  * Coggan (2004) Melanie Coggan. 2004.  Exploration and exploitation in reinforcement learning.  _Research supervised by Prof. Doina Precup, CRA-W DMP Project at McGill University_.
  * Das and Smith (2012) Dipanjan Das and Noah A Smith. 2012.  Graph-based lexicon expansion with sparsity-inducing penalties.  In _Proceedings of the 2012 conference of the North American chapter of the Association for Computational Linguistics: human language technologies_ , pages 677–687.
  * Del Rio and Malani (2020) Carlos Del Rio and Preeti N Malani. 2020.  Covid-19—new insights on a rapidly changing epidemic.  _Jama_ , 323(14):1339–1340.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Giorgi et al. (2020) Salvatore Giorgi, Sharath Chandra Guntuku, Muhammad Rahman, McKenzie Himelein-Wachowiak, Amy Kwarteng, and Brenda Curtis. 2020.  Twitter corpus of the# blacklivesmatter movement and counter protests: 2013 to 2020.  _arXiv preprint arXiv:2009.00596_.
  * Grant et al. (2020) Michael C Grant, Luke Geoghegan, Marc Arbyn, Zakaria Mohammed, Luke McGuinness, Emily L Clarke, and Ryckie Wade. 2020.  The prevalence of symptoms in 24,410 adults infected by the novel coronavirus (sars-cov-2; covid-19): A systematic review and meta-analysis of 148 studies from 9 countries.  _Available at SSRN 3582819_.
  * Guntuku et al. (2020) Sharath Chandra Guntuku, Garrick Sherman, Daniel C Stokes, Anish K Agarwal, Emily Seltzer, Raina M Merchant, and Lyle H Ungar. 2020.  Tracking mental health and symptom mentions on twitter during covid-19.  _Journal of general internal medicine_ , pages 1–3.
  * Iacobucci (2020) Gareth Iacobucci. 2020.  Covid-19: diabetes clinicians set up social media account to help alleviate patients’ fears.  _BMJ_ , 368:m1262.
  * Luo et al. (2017) Yuan Luo, William K Thompson, Timothy M Herr, Zexian Zeng, Mark A Berendsen, Siddhartha R Jonnalagadda, Matthew B Carson, and Justin Starren. 2017.  Natural language processing for ehr-based pharmacovigilance: a structured review.  _Drug safety_ , 40(11):1075–1089.
  * Mikolov et al. (2017) Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2017.  Advances in pre-training distributed word representations.  _arXiv preprint arXiv:1712.09405_.
  * Mpouli et al. (2020) Suzanne Mpouli, Michel Beigbeder, and Christine Largeron. 2020.  Lexifield: a system for the automatic building of lexicons by semantic expansion of short word lists.  _Knowledge and Information Systems_ , pages 1–21.
  * Nikfarjam et al. (2019) Azadeh Nikfarjam, Julia D Ransohoff, Alison Callahan, Erik Jones, Brian Loew, Bernice Y Kwong, Kavita Y Sarin, and Nigam H Shah. 2019.  Early detection of adverse drug reactions in social health networks: a natural language processing pipeline for signal detection.  _JMIR public health and surveillance_ , 5(2):e11264.
  * O’Connor et al. (2014) Karen O’Connor, Pranoti Pimpalkhute, Azadeh Nikfarjam, Rachel Ginn, Karen L Smith, and Graciela Gonzalez. 2014.  Pharmacovigilance on twitter? mining tweets for adverse drug reactions.  In _AMIA annual symposium proceedings_ , volume 2014, page 924. American Medical Informatics Association.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1532–1543.
  * Rúa (2007) Paula López Rúa. 2007.  Keeping up with the times: lexical creativity in electronic communication.  _Lexical Creativity, Texts and Contexts/ed. by J. Munat.–Amsterdam_ , pages 137–159.
  * Sarker and Gonzalez (2015) Abeed Sarker and Graciela Gonzalez. 2015.  Portable automatic text classification for adverse drug reaction detection via multi-corpus training.  _Journal of biomedical informatics_ , 53:196–207.
  * Singh et al. (2020) Lisa Singh, Shweta Bansal, Leticia Bode, Ceren Budak, Guangqing Chi, Kornraphop Kawintiranon, Colton Padden, Rebecca Vanarsdall, Emily Vraga, and Yanchen Wang. 2020.  A first look at covid-19 information and misinformation sharing on twitter.  _arXiv preprint arXiv:2003.13907_.
  * Stokes et al. (2020) Daniel C Stokes, Anietie Andy, Sharath Chandra Guntuku, Lyle H Ungar, and Raina M Merchant. 2020.  Public priorities and concerns regarding covid-19 in an online discussion forum: Longitudinal topic modeling.  _Journal of general internal medicine_ , page 1.
  * Venkatraman (2020) A Venkatraman. 2020.  Weekly time spent in apps grows 20% year over year as people hunker down at home.  _App Annie_.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _ArXiv_ , pages arXiv–1910.
  * Wu et al. (2019) Sixing Wu, Fangzhao Wu, Yue Chang, Chuhan Wu, and Yongfeng Huang. 2019.  Automatic construction of target-specific sentiment lexicon.  _Expert Systems with Applications_ , 116:285–298.
  * Wu and McGoogan (2020) Zunyou Wu and Jennifer M McGoogan. 2020.  Characteristics of and important lessons from the coronavirus disease 2019 (covid-19) outbreak in china: summary of a report of 72 314 cases from the chinese center for disease control and prevention.  _Jama_ , 323(13):1239–1242.
  * Yang et al. (2018) Qinghua Yang, Christopher Tufts, Lyle Ungar, Sharath Guntuku, and Raina Merchant. 2018.  To retweet or not to retweet: Understanding what features of cardiovascular tweets influence their retransmission.  _Journal of health communication_ , 23(12):1026–1035.

## Appendix

##### COVID-19 Keywords

used to pull tweets from the streaming Twitter API: coronavirus, covid,
2019-ncov, covd, outbreak, pandemic, corona, corono, washyourhand,
handwashing, stayhome, stayathome, and sarscov.

### Illustrative graph building example


(cvpr) Package cvpr Warning: Package ‘hyperref’ is not loaded with option
‘pagebackref’, which is strongly recommended for review version

# Supplementary Document - We never go out of Style: Motion Disentanglement by
Subspace Decomposition of Latent Space

##  Appendix A Introduction

Please watch the video in the supplementary folder (main-video.mp4) for the
video version of the results presented in this document. This document
provides details about the related work, experimentation, dataset, and
detailed analysis of the motion subspaces. Further, we present additional
qualitative results. In the last section, we provide the details of the
dataset creation process for the CarsInCity image and video datasets.

##  Appendix B Related work

StyleGAN based editing. StyleGAN models have been shown to contain
disentangled semantics in their latent spaces. Multiple works
[shen2020interpreting, harkonen2020ganspace, stylespace,
patashnik2021styleclip, tewari2020stylerig, abdal2021styleflow,
unsupervised_discovery] have been proposed to perform image editing by using
pre-trained StyleGAN2 models. In one set of methods, a global latent direction
is estimated in the latent space, which, when traversed, results in attribute
editing [shen2020interpreting, harkonen2020ganspace, jahanian2019steerability,
spingarn2020gan]. In another set, each latent code is transformed differently
using normalizing flows [abdal2021styleflow, liang2021ssflow], transformer
networks [transformereditsg2, umetani2022user, xu2022transeditor] linear
transformation with explicit identity preservation losses [latentTransformer]
or using clip-guidance [patashnik2021styleclip, abdal2022clip2stylegan,
zheng2022bridging]. These methods either use pre-trained attribute classifiers
[shen2020interpreting, abdal2021styleflow, latentTransformer], guidance from
clip or estimate directions in an unsupervised manner [harkonen2020ganspace,
unsupervised_discovery, choi2021not, balakrishnan2022rayleigh,
jahanian2019steerability, spingarn2020gan, latentclr]. There is another set of
approaches that use a semantic mask or spatial regions [ling2021editgan,
zhu2022region], sketch [doner2022paintinstyle] or key-points
[wang2022rewriting] to perform controlled image editing. We guide the reader
to look at this comprehensive survey [bermano2022state] of other StyleGAN
editing methods.

GAN Encoder Models. To leverage the benefits of these StyleGAN editing methods
to edit real-world images, StyleGAN encoder models are proposed that encode a
given image into the latent code [psp, e4e, alaluf2021restyle, chai2021using,
abdal2019image2stylegan, abdal2020image2stylegan++, xu2021continuity,
chai2021using]. Another set of methods optimizes the generator to obtain a
highly detailed reconstruction of a given input image [roich2022pivotal,
tzaban2022stitch]. There are trade-offs between these image encoding methods’
reconstruction and editing abilities. Hyper-style [alaluf2021hyperstyle]
combines these two approaches to obtain fast, semantically rich encoding of
input images. In another set of methods, latent code is optimized directly to
obtain high-quality reconstructions [tewari2020pie, chai2021using]. In
StyleGAN models, $\mathcal{W}$ space has better editing capabilities but with
poor reconstruction quality. On the other hand, $\mathcal{W+}$ space is more
expressive, and encoding in $\mathcal{W+}$ results in high reconstruction
qualities. Hence, most of these encoder models map the image into
$\mathcal{W+}$ space for image editing. Recently, a few video encoders
[xu2021continuity, tzaban2022stitch] have been proposed to encode an input
video in a sequence of the latent codes in $\mathcal{W+}$.

Video generation. There have been several works that focus on generating
videos in the GAN framework. MoCoGAN [mocogan2018] proposed to explicitly
decompose video into motion and content and generate videos by mapping a
sequence of latent codes to generated frames. Building on this MoCoGAN-HD
[mocoganhd-iclr2021] and StyleGAN-V [stylegan-v-cvpr2022], use a pre-trained
StyleGAN model as the image generator where content is fixed and motion is
represented as a latent trajectory. All of these models are trained with
either two different discriminators for images and videos or a joint
discriminator on the videos. StyleVideoGAN [fox2021stylevideogan] trains a
generative sequence model in the latent space to learn a distribution of
latent sequences responsible for generating coherent videos. DiGAN [digan2022]
leverages implicit functions to predict pixel values and train them with image
and motion discriminators. Another line of work leverages transformers to
generate long videos [ge2022long, yan2021videogpt].

Video Editing. There have been a series of works on video editing that
leverage the pre-trained StyleGAN model to perform toonification
[yang2022vtoonify], face-reenactment [yin2022styleheat,
abdal2022video2stylegan, latentTransformer], or perform attribute editing in
temporally coherent manner [tzaban2022stitch, alaluf2022third,
xu2022temporally]. LIA [lia_animator] trained a video encoder-decoder model
for the task of face reenactment with a learned motion vector dictionary.
Recently, text2live [bar2022text2live] showed a novel direction of video
editing by generating an edit layer, which, when pasted on the original video,
generates realistic edits. All these methods majorly target spatial editing of
each frame and blending them coherently. However, for the first time in this
work, we propose to edit the motion in the video by decomposing it into
disentangled motion styles.

##  Appendix C Experiment details

We performed motion decomposition of talking face and car moving videos.
Talking face motion can be naturally decomposed into pose and expressions
motions. We show the generalization ability of our proposed method by
decomposition of the motion of a moving car into rotation and translation
components. The latter is very challenging, as the model has to dis-occlude
background regions, wherever required, to disentangle motion in a moving car.
Our proposed method requires a pretrained encoder and StyleGAN2 models.
Additionally, it also involves a set of $N$ videos with a single motion
component for obtaining motion subspaces. We will elaborate on each of these
requirements in the following sections. We encourage the readers to watch the
supplementary video for video results.

StyleGAN2 and Encoder Models. For face experiments, we used a pre-trained e4e
[e4e] encoder and StyleGAN2 model trained on FFHQ [karras2020analyzing]
dataset. Additionally, for motion decomposition, we performed PTI-based
optimization [roich2022pivotal] over the given test video to obtain a
temporally coherent latent trajectory during inference. For cars, we observed
that using off-the-shelf encoder [e4e] and StyleGAN2 models trained on real-
world cars cannot correctly encode moving car videos and result in temporally
inconsistent reconstructions. We noted that this is primarily because of the
significant variations in the real-world backgrounds. To this end, for
experimentation on cars, we created our dataset - CarsInCity, by placing
ShapeNet cars onto a simple road with a building in the background. We trained
StyleGAN2 and e4e encoder on the $25K$ images from CarsInCity and obtained an
FID of $6.770$ on the generated images and LPIPS value of $0.090$ for
reconstructions on the test set.

##  Appendix D Datasets

###  D.1 Video datasets for face motion

We decompose the face motion into two subspaces: expression and pose. To
obtain the videos with only pose variations, we use a state-of-the-art
3D-aware generative model [eg3d], to generate random faces rendered with
multiple camera positions with 3D consistency. We randomly sampled $100$ faces
of different identities and captured $120$ images for each identity. We stitch
the images from novel views to obtain a video with only pose variation. The
generated images are highly realistic and 3D consistent for each identity. The
videos for expression variations were obtained by randomly sampling videos of
five identities from the video-expression recognition dataset, GRID
[grid_dataset2006]. This dataset has very small videos with a single
expression, hence we sampled $100$ videos to create our expression dataset.
Note that, the total duration of the video samples is less than $15$ mins.

###  D.2 Cars.

CarsInCity. We generated a synthetic dataset comprising car images within the
3D Blender environment (ref Sec.N). The process involved randomly selecting a
3D car model from the ShapeNet cars dataset [chang2015shapenet] and placing it
within simple city settings. A virtual camera was added to capture the car’s
images from various angles. The camera is positioned such that the car remains
centered within the image at all times. Additional information regarding the
creation of the dataset is provided in the supplementary document. We rendered
$25K$ images of different cars in the dataset.

Videos for car motion. We use the same blender setup used for CarsInCity (ref
Sec.N), to obtain the videos of moving cars. The motion of moving cars is
decomposed into translation and rotation subspaces. We obtained the ground
truth videos with only translation motion by shifting the car along the road
with some velocity. For rotation, we rotated the car along its own axis by
$(-30,30)$ degrees. We captured images at uniform time steps with the camera
moving with the same velocity for translation and a fixed camera for rotation.
Totally, we captured $100$ videos for translation and rotation each $5-10s$
long. Additionally, as the test set, we generated another set of $100$ videos,
where both the translation and rotation are applied simultaneously.

##  Appendix E Details about the baselines.

We compared our method for motion decomposition against two baseline methods
Independent Component Analysis (ICA) and Latent Image Animator (LIA)
[lia_animator].

ICA. We performed ICA decomposition of a set of $100$ videos from CelebV-HQ,
having both the expressions and pose motion entangled. As ICA, estimates the
independent source present in a combined signal, we use it to decompose latent
directions for entangled motions to obtain independent motion directions. We
then annotate each of these directions to either pose or expressions, by
observing its impact on the output images. We used $6$ independent components
as increasing a large number of components resulted in very small motion in
all the components.

LIA [lia_animator]. LIA is trained to perform face-reenactment on videos.
However, LIA uses a learnable motion dictionary in the latent space and
considers any given motion as the linear combination of the vectors from this
dictionary. We argue that the learned motion dictionary can be decomposed into
two sets of vectors which span pose and expression subspaces. To this end, we
analyzed each of the $20$ vectors from the dictionary and assign them to
either pose or expressions. We found 0,1,4,8,9,10,15 and 18 vectors majorly
control pose and the remaining vectors control expressions. Interestingly, we
found some vectors that entangle both expression and pose.

##  Appendix F Aggregated Pose Motion (APM)

To evaluate the disentanglement we defined a novel metric based on the amount
of head pose motion: Aggregated Pose Motion. Given a video, we first estimate
facial landmark points for two consecutive frames using an off-the-shelf
landmark detector [dlib09]. Thereafter, we used the set of paired landmark
points to obtain a rotation matrix assuming the rigid motion of the face.
Finally, we compute a difference matrix between the estimated rotation matrix
and identity matrix and take the Frobenius norm of the difference matrix. We
define APM as the average of the Frobenius norm for all the consecutive frames
in a given video. Intuitively, when there is only head pose motion, APM should
be higher as the rotation matrix will be very different from the identity
matrix, conversely, if the head pose is fixed and only expressions are
changing, then the APM will be smaller. In the implementation, we subsample
every $10th$ for APM computation as neighboring frames have very small changes
in landmarks.

##  Appendix G Multiple subspace decomposition

Our method is naturally extendable to multiple subspaces following the
formulation given in Eq.3 in main paper. We present the results for combining
three fine-grained subspaces for yaw, pitch, and mouth motion in Fig. 1. We
highlight the transitions with significant motion. Observe that our method can
seamlessly merge these three motion subspaces. We obtained the ground truth
data for mouth motion by capturing only seven, one-minute talking mouth videos
with a smartphone camera. The superior quality of disentanglement results,
suggests that our method can seamlessly adapt to any new motion subspace given
a few set of videos.

Figure 1: Multi-subspace decomposition of the source sequence into fine motion
components of yaw, pitch, and mouth motion. We have highlighted the regions
with a significant change in the given subspace.

##  Appendix H Orthogonality of subspaces

Our method guarantees disentangled motion subspaces by using a small dataset
of ground truth videos with a single motion component (e.g., pose) and
performing PCA to approximate the corresponding motion subspace. We performed
the following experiment to evaluate the orthogonality among these subspaces.
Specifically, we extract the top $20$ principal components from the pose and
expression subspaces each. We then compute the pair-wise dot product between
each pose and expression component and plot the histogram in Fig. 2. Observe
that most of the vectors from the pose subspace are orthogonal to that of the
expression subspace, suggesting these two spaces are orthogonal.

Figure 2: Orthogonality analysis between the top $20$ principal components
derived from the subspaces of pose and expression motions.

##  Appendix I Analysis on Principal Components

We perform an ablation over the number of principal components required for
pose and expression subspace for faces as shown in Fig. 3 and Fig. 4,
respectively.

Figure 3: We perform an ablation experiment to analyze the effect of varying
the number of principal components in the pose subspace. For a lower number of
components, the head pose deviates from the source video. However, as the
number of components increases, the pose changes become more pronounced
compared to the reference video, as indicated by the green. Figure 4: We
perform an ablation study on the number of principal components for the
expression subspace. With five components, the mouth motion fails to transfer
accurately, as illustrated in the last frame. However, increasing the number
of components leads to temporal inconsistency in the generated video, as
indicated in green.

For the pose, increasing the number of components increases the degree of pose
change. Note that for $100$ components, the pose is changed more than that in
the source sequence (for better visual understanding, see main-video.mp4). In
the case of the expression, having a very large or very small number of
components results in temporal inconsistency and large mouth openings (visible
in the last frame). Based on this and the quantitative analysis presented in
Sec.$4.2$ in the main paper, we have used $35$ components for pose and $50$
for expression offering a trade-off between the disentanglement and visual
realism. For cars, we have found $10$ components for rotation and translation
work best and can faithfully disentangle the motion.

Additionally, we plot the variance captured by the selected principal
components for the rotation and translation motion subspaces for cars and pose
and expression motion subspaces for faces presented in Fig. 5. When given
enough components, we observe that the signal is reconstructed well for all
subspaces. This suggests that low-dimensional manifolds exist in the latent
spaces where each motion style is captured. Interestingly for pose subspace,
even with $200$ components, the variance captured is $\approx 70\%$. However,
we observed in the ablations that using $35$ components is enough to capture
the majority of pose variations in the dataset. This suggests that the
information captured in the tail components is not directly related to the
head pose but captures other motions also, such as abrupt camera motions.

Figure 5: Distribution of variance captured by the principal components for
the motion subspaces for cars and faces Figure 6: Motion decomposition for the
face dataset. We decompose the source video sequence into expression and pose
motion video. The results indicate a successful disentanglement of the
expressions and pose, while preserving the identity of the subject. Figure 7:
We perform an ablation study over the strength parameter $\alpha$ the degree
of expression motion. Observe that a smaller value of $\alpha$ leads to
partial eye motion in the final frame. Figure 8: Results with different
strength parameters for $\alpha$ for controlling the amount of pose variation.
Higher $\alpha$ values lead to significant changes in pose, ultimately
resulting in identity alteration for extreme poses, as evident from the last
row of the results. Figure 9: Here we show complete motion transfer results
from a driving video sequence to a source image. We show reenactment with
stylization (Top) using StyleGAN-NADA [gal2022stylegan] for ’Pixal style’ and
reenactment with age editing (Bottom) using [patashnik2021styleclip] edit
directions. Observe the high level of realism, and identity consistency in the
reenacted results. Figure 10: Here we show selective reenactment of the source
image given a driving video. We transfer only pose, only expressions and both
the motions combined to the source image. Observe that the reenacted sequence
follows the pose/expression motion from the driving frame correctly. Note that
the source image is not reconstructed perfectly with the e4e [e4e] encoder.
One can use advancement in optimization-based encoding methods to obtain a
better reconstruction quality and perform reenactment with our method. Figure
11: Here we show motion decomposition on the cars dataset. Given a source
sequence having entangled translation and rotation motions, we decompose into
disentangled videos of only translation and only expression motions. Observe
the yellow road marking moving back, giving the illusion of translation. Our
method is able to disentangle motion components well even in this challenging
scenario. Please watch the supplementary video to observe the decomposed
motion. Figure 12: We show reenactment on cars given a driving video where the
car is rotating on its own axis. We faithfully transfer the motion into a
given source image keeping the background unchanged and the identity of the
car intact. The reconstructions of the cars is not perfect as there is some
loss while encoding input images by [e4e] encoder.

##  Appendix J Comparison with state-of-the-art motion transfer methods

Although our method is designed for motion decomposition, it can be repurposed
to perform one-shot reenactment. We compare the performance of our proposed
one-shot face reenactment with state-of-the-art methods - LIA [lia_animator],
MRAA [mraa], and FOMM [fomm] in Tab. 1. Note that our method is trained on
only $15$ minutes of videos, unlike these methods, which are trained on large
video datasets of $20K$ videos further, as these methods are limited to a
smaller resolution of $512$ and take several days to train. In contrast, our
approach can scale to $1024$ resolution and is trained in less than $2$ hours
on a single GPU. Following [lia_animator], we compute AKD and AED metrics for
the exact identity reenactment and FID video for cross-identity reenactment.
We present quantitative results for reenactment on CelebV-HQ in Tab. 1.
Despite using 225x lesser data, our method achieves competitive performance
against the best-performing method on same identity reconstruction and cross-
identity transfer. Note that all the other methods are trained on videos;
however, our approach uses pretrained image GANs and only a few shot videos.

Table 1: Quantitative comparison for one-shot face reenactment for same and cross-identity videos.  | Same Identity | Cross Identity
---|---|---
Method | AKD | AED | FID-video
FOMM | 2.94 | 0.13 | 86.21
MRAA | 2.74 | 0.10 | 87.21
LIA | 2.14 | 0.12 | 71.36
Ours | 2.26 | 0.11 | 72.66

##  Appendix K Implementation details

Preprocessing. Given any input video, we first preprocess the video by
detecting faces and aligning the eyes to a fixed location. This is a standard
preprocessing step used by the StyleGAN2 models [karras2020analyzing]. For
faces videos, we have used Pivotal tuning PTI and stitch tuning following
[tzaban2022stitch] over the frames to obtain temporally coherent video
reconstructions. This is done in three steps: (a) given a video with $N$
frames, we first encode them into $N$ set of latent codes using e4e encoder
[e4e]. (b) Then, we fine-tune the StyleGAN2 generator to minimize the
reconstruction loss between the reconstructed and the ground truth frames. In
the second step, the encoded latent codes obtained from the encoder are kept
fixed. (c) After editing the face, we stitch the cropped face back to the
source video along with the next round of generator tuning for smooth
blending. This results in temporally smooth edited videos for the talking face
while keeping the subject’s identity intact. We performed PTI for $80$ steps
for face video datasets and stitch tuning for $100$ steps for the videos the
supplementary video.

However, PTI did not work for the moving car videos and resulted in generating
a fixed image for all the frames. It is primarily because car videos have a
significant background change across frames. Applying PTI results in
overfitting the generator towards only a fixed image, as it cannot fit all the
background variations in the video. Therefore, the encoded car videos could be
better and have minor inconsistencies. We do not perform any additional
preprocessing for the cars dataset.

Layer Selection. The initial few layers of StyleGAN models capture the coarse
structure of the image, such as the pose, gender, expression, and identity
(for faces), and the later layers capture the finer details, such as lighting
and skin-tone [yang2021semantic]. As we primarily focus on editing motion
while keeping the content preserved in the video, we applied our subspace
decomposition approach to only the latent codes corresponding to the first $L$
layers while keeping the other latent codes fixed. With experiments, we set
$L$ equal to $10$ for both datasets. We performed all of our experiments on
two NVIDIA-A5000 GPUs.

##  Appendix L Additional quantitative results

We evaluate the quality of the videos generated by motion decomposition using
Fre‘chet Video Distance (FVD) implementation from StyleGAN-V codebase
[stylegan-v-cvpr2022]. We generate motion decomposition for $500$ videos from
the CelebV-HQ dataset using ours, ICA, and LIA [lia_animator]. To compute FVD,
we used $1000$ ground truth videos from the CelebV-HQ dataset. Note that for
this experiment, we did not apply PTI as it is computationally inefficient to
run on a large number of videos. The results are shown in Table. 2.
Specifically, we compute $FVD_{1}6$ by calculating statistics using the first
$16$ frames from the video. All three methods have $FVD_{16}$ values in a
similar range, suggesting comparable video quality.

Table 2: Quantitative evaluation of Video quality using FVD. Method | ICA | LIA [lia_animator] | Ours
---|---|---|---
FVD | 207.24 | 220.40 | 218.33

##  Appendix M Qualitative results

In this section, we show additional qualitative results for our method. Please
have a look at the supplementary video to see the video version of these
outputs. In Fig. 6 and Fig. 11, we present results for motion decomposition
for faces and cars, respectively. Observe that our method can decompose the
video into disentangled video subspaces with high realism. For cars, observe
that the yellow road markings are moving back for translation while keeping
the pose of the car unchanged.

Further, we show an ablation over the strength parameter $\alpha$ defined in
Sec. $3$ in the main paper to selectively manipulate motion components in Fig.
7 and Fig. 8. For expressions, we can observe that with lower strength, the
eyes of the subject are not fully closed in the last frames. Similarly, we can
observe that increasing the strength to $1.5$ results in larger pose changes
than the source sequence. Finally, we present results for reenactment in Fig.
12 for cars, Fig. 10 and Fig. 9 for faces. As our method leverages StyleGAN2
as its backbone, we can perform stylization using StyleGAN-NADA
[gal2022stylegan] generator or perform semantic edits using StyleClip
[patashnik2021styleclip] directions on the reenacted video as shown in Fig. 9.

##  Appendix N CarsInCity

In our experiments, we have observed that using encoder and StyleGAN2 models
trained on real car images [e4e] results in temporally inconsistent
reconstructions when we encode a real-world moving car video. This is because
real-world car videos have large background variations and StyleGAN models
cannot reconstruct rich background details well. This results in unrealistic
reconstructions where background appearance and structure change across
frames. To this end, we proposed a new synthetic dataset of cars placed in
simple city background. The major advantage of this dataset is that we can
encode moving car videos with high fidelity and temporal coherence.
Specifically, we placed ShapeNet cars in the 3D city environment and captured
images and video using a virtual camera.

###  N.1 3D environment

We created a simplistic scene layout, with roads and buildings on the side.
The top view layout is shown in Fig. 13-a). The layout has two roads having
two lanes each, and they cross to form an intersection at a randomly sampled
location. On the sides, sidewalks and buildings are added. The placement
location of the car was randomly selected on any of the roads, and its
orientation was chosen based on the selected lane.

Figure 13: Dataset details for the CarsInCity dataset. We show the top view of
the road layout in a) which has roads, buildings, and sidewalks. We randomly
sample locations on the road to place ShapeNet cars following the orientation.
We add a repeated texture on buildings, roads and sidewalks as shown in b),
c), and d).

To realize this layout, we created a 3D environment in Blender [blender] and
represented buildings with cuboids of varying heights, sidewalks, and the road
with thin cuboids. We added a texture with yellow stripes on all the roads and
a single texture for the sidewalks and the buildings as shown in Fig. 13. For
each scene, we randomly sample a 3D car model from the ShapeNetV2
[chang2015shapenet] dataset and place it on the sampled location. In total,
ShapeNetV2 has $3533$ highly diverse car models, including sedans, hatchbacks,
buses, caravans, and trucks.

###  N.2 Image acquisition

We place a virtual camera on a cylinder enclosing the car with the same center
as the car. The diameter of the cylinder is kept at $1.5$ times the largest
side of the car’s bounding box. The camera location for a given height was
sampled in the range $(-30^{\circ},30^{\circ})$, when viewed from the side.
The look vector for the camera was always facing the center of the car’s
bounding box. Finally, we rendered the images using an inbuilt raytracing
engine from Blender with shadows and texture effects. The camera’s focal
length was selected such that the whole car is visible in the rendered image.

Post processing. We rendered the images at $256$x$384$ resolution, following
the natural shape of the cars. We added black padding to make the image square
followed by resizing to $256$x$256$ resolution. We added blue color in the
transparent region of the scene to represent the sky.

###  N.3 Video dataset

We generated $100$ videos each for rotation and translation motion.
Additionally, we generated $40$ video where both the rotation and translation
motion was occurring simultaneously. We explain the algorithm to generate the
videos for translation and rotation. For test videos that have both rotation
and translation, we simply combine these two algorithms.

Translation. To simulate the translation effect, we moved the car along the
road by a fixed amount at each time step. The camera was also driven by the
same amount keeping the camera stationary with respect to the car. The
illusion of translation motion is created as the background moves backward.

Rotation. Rotation was simulated by rotating the car from
$(-30^{\circ},30^{\circ})$ from its cannonical pose. To generate a video, the
car is rotated by $1^{\circ}$ at each time step.

##  Appendix O Conclusion

In conclusion, we propose a novel method for video motion decomposition using
pretrained StyleGAN models trained on images. We show the results of motion
decomposition in the video and its applications in tasks such as motion
editing and selective one-shot motion transfer.