# Overcoming Barriers to Skill Injection in Language Modeling: Case Study in
Arithmetic

Mandar Sharma
Virginia Tech
<EMAIL_ADDRESS>
Nikhil Muralidhar
Stevens Institute of Technology
<EMAIL_ADDRESS>
Naren Ramakrishnan
Virginia Tech
<EMAIL_ADDRESS>

###### Abstract

Through their transfer learning abilities, highly-parameterized large pre-
trained language models have dominated the NLP landscape for a multitude of
downstream language tasks. Though linguistically proficient, the inability of
these models to incorporate the learning of non-linguistic entities (numerals
and arithmetic reasoning) limits their usage for tasks that require numeric
comprehension or strict mathematical reasoning. However, as we illustrate in
this paper, building a general purpose language model that also happens to be
proficient in mathematical reasoning is not as straight-forward as training it
on a numeric dataset. In this work, we develop a novel framework that enables
language models to be mathematically proficient while retaining their
linguistic prowess. Specifically, we offer information-theoretic interventions
to overcome the catastrophic forgetting of linguistic skills that occurs while
injecting non-linguistic skills into language models.

##  1 Introduction

Numerals grant objectivity to language [15], thus, their incorporation to
language models, among other abilities, allows for better information
extraction and language inference [12, 14, 18]. With the rise of the Math-NLP
niche, several notable publications have investigated the deficiency of
inherent numerical skills induced in large language models through
unsupervised training [23, 8, 27]. Several more offer architectures and
training schemes to induce this skill through supervision [19, 7, 5, 20].
However, the overarching goal should remain to build numerically-capable
language models that still do what they were intended to do - model language.

Injecting non-linguistic skills often comes at the cost of losing linguistic
skills. Akin to all neural models, language models are susceptible to
catastrophic forgetting [10] when trained for multi-task learning, especially
when the two tasks are substantially different - linguistic vs non-linguistic.
Here, we first investigate the nature of catastrophic forgetting in the
context of language models through an information-theoretic lens and
subsequently establish interventions that prevent this phenomenon. Our models
perform substantially better arithmetic while retaining their linguistic
prowess.

##  2 Catastrophic Forgetting in Language Models

(a) Information Sensitivity
Self-attention Layer-1

(b) Information Sensitivity
Self-attention Layer-5

(c) Encoder Parameter Space
Self-attention Layer-1

(d) Encoder Parameter Space
Self-attention Layer-5

Figure 1: Parameter sensitivity (a & b) for the $n=800$ most vital parameters
of the encoder layers 1 & 5 for CoLA and MRPC. The higher sensitivity of these
parameters to CoLA vs MRPC lends an information-theoretic explanation of task-
specific performance degradation. 2D t-SNE visualizations of the
parameterization space (c & d) for the same models & layers - the distinction
between the parameterization space of models trained for Arithmetic task vs
Linguistic tasks offers another perspective into degradation of lingusitic
performance when trained on non-linguistic task.

In multi-task learning, when a model that is pre-trained on task $A$ is
subsequently trained on task $B$, the weights in the model that are vital for
task $A$ adapt their values to meet the objectives of task $B$. It has been
found that this frequently leads to performance degradation on task $A$. This
phenomenon is referred to as catastrophic forgetting [10]. While it has been
found that off-the-self pre-trained language models do establish a notion of
numeric scale from their unsupervised pre-training, these are not quite enough
to perform commonsense reasoning [27]. Furthermore, without additional
interventions, these notions of numeric scales fail to extrapolate for
numerals outside of the training set [23]. To meet the numerical comprehension
standards required for commonsense reasoning, these model have to be trained
further on numeric datasets, often resulting in the loss of their original
linguistic prowess through catastrophic forgetting.

Interestingly, for langauge models, the catastrophic forgetting tendencies are
not evenly spread across standardized GLUE [24] tasks - for instance, a base
DistilBERT model [17] trained for arithmetic computation faces significant
degradation in its grammatical prowess (CoLA [25]) while still retaining much
of its ability for semantic equivalence (MRPC [3]), as seen in the second row
of table 1.

In an effort to better understand this phenomenon, we adopt a two-pronged
approach. The analysis of the weights and biases of a converged model across
multiple datasets provides a parameterization-standpoint perspective towards
understanding their performance across shared tasks [6, 21]. Thus, we
visualize the parameterization space for models trained independently on each
of the aforementioned three tasks through figures 1(c) and 1(d). Through the
low-dimensional t-SNE [22] projections, we observe that the model
parameterizations live in different spaces when the tasks are linguistic (CoLA
and MRPC) vs non-linguistic (Arithmetic). This ties back to the premise of
catastrophic forgetting - where the parameters of the DistilBERT model trained
for arithmetic computation have re-adjusted their weights to a space that does
not comply well with linguistic tasks. These new parameter distributions for
arithmetic computation (orange) are consistent across the transformer encoder
self-attention layers - as seen for layer 1 in figure 1(c) and layer 5 in
figure 1(d). These two figures offer the first perspective on linguistic vs
non-linguistic performance degradation from a model parameterization stand-
point.

For a complementary perspective, we zoom into the contributions of the
individual model parameters for our set of shared tasks through an
information-theoretic lens. For a single sample $Y$ drawn from a distribution
with probability desnity $f(y;\theta)$, the Fisher information index
$I(\theta)$ (1) quantifies the sensitivity of the parameter $\theta$ to the
data instance $Y$. After selecting the top $n=800$ most vital parameters for
each encoder layer for the Arithmetic task based on their Fisher information
scores, we observe that the same model parameters are more sensitive to the
CoLA task than to the MRPC task, as shown in figures 1(a) and 1(b), again,
based on their respective task-specific sensitivity scores. Thus, offering a
complementary information-theoretic perspective on task-specific performance
degradation.

$I(\theta)=E(\frac{d\log f(Y;\theta)}{d\theta})^{2}=-E(\frac{d^{2}\log
f(Y;\theta)}{d\theta^{2}})$ (1)

##  3 Overcoming Catastrophic Forgetting

###  3.1 Elastic Weight Consolidation for Language Modeling

System-level consolidation often consists of stitching-together an amalgamated
dataset that consistutes of multiple-shared tasks [11]. However, for general-
purpose language models, the range of possible downstream tasks are so diverse
that the paradigm has remained to build large models that hold linguistic
prowess and are intended to be fine-tuned on a single downstream task [17, 9,
1]. Thus, being more suited to a continual learning paradigm. As these models,
by their stature, are highly parameterized - it can be ascertained that there
is a solution for task $B$, $\theta_{B}$, that is proximal to the solution
space for task $A$, $\theta_{A}$. For such continual learning, Elastic Weight
Consolidation (EWC) [10] regularizes learning on specific network weights
based on their importance to previously seen tasks, ensuring $\theta_{B}$
remains proximal to $\theta_{A}$.

Thus, with the posterior distribution of the DistilBERT model approximated
through its Fischer information matrix $F$ and Gaussian distribution mean from
parameters $\theta^{*}_{gen}$, we train the model to predict the correct
tokens representing the results of arithmetic operations through masked-
modeling loss $\mathcal{L}_{arith}$ such that changes to any model parameter
$i$ crucial to then general functionality of the model $\theta_{gen}$ is
penalized through a quadratic penalty scaled by $\lambda$ (2). The selection
of the hyperparameter $\lambda$ is crucial as it dictates both model
convergence and balances the learning of $\theta_{arith}$ with $\theta_{gen}$.
Thus, we gauge the sensitivity of model convergence with respect to $\lambda$
with a hyperparameter sweep raning from $\lambda$=1e-4 to 5e-11. Figure 2
shows the interplay between the weight consolidation loss $EWC$ and the Cross-
Entropy loss for learning arithmetic $CE$ (color-matched) for different values
of $\lambda$. We observe the first sign of model convergence at
$\lambda$=1e-8, and although the model convergence improves with decreasing
values of $\lambda$, we set $\lambda$ to 1e-8 for our experiments to promote
balanced learning of $\theta_{arith}$ with proper retention of $\theta_{gen}$.

$\mathcal{L}(\theta)=\mathcal{L}_{arith}(\theta)+\sum_{i}\frac{\lambda}{2}\
F_{i}(\theta_{i}-\theta_{gen,i}^{*})^{2}$ (2)

###  3.2 Experimental Setup and Results

Our in-house arithmetic dataset comprises of numerals modeled after the
numeral distributions in real-world datasets DROP [4] and EQAUTE [16]
(§Appendix 5.2) with 21,838 instances of arithmetic computations. All models
are trained for 50 epochs where the datasets are processed through the
standard sub-word tokenization scheme of BERT [9]. These models are then fine-
tuned and evaluated on the range of GLUE tasks [24]:

  * •

Single Sentence Tasks: Corpus of Linguistic Acceptability (CoLA) [25] for
grammatical fidelity with Matthews correlation coefficient as the metric and
the Stanford Sentiment Treebank (SST-2) [13] for sentiment analysis with
accuracy as the metric.

  * •

Similarity and Paraphrasing Tasks: The Microsoft Research Paraphrase Corpus
(MRPC) [3] for semantic equivalence with averaged accuracy and f1 scores as
the metric and the Semantic Textual Similarity Benchmark (STS-B) [2] for
sentence similarity with the Spearman rank correlation coefficient as the
metric.

  * •

Inference Task: The Multi-genre Natural Language Inference Corpus (MNLI) [26]
for textual entailment between a given premise and hypothesis with accuracy as
the metric.

The results presented in table 1 represent task-metrics for the models as
$\mu_{\sigma}$ where $\mu$ represents the mean value and $\sigma$ represents
the standard deviation across two runs with different seed values for random
initialization of the model weights. Please note that the base DistilBERT
model has been used off-the-shelf and thus has no standard deviation across
its runs.

Table 1: Comparative analysis between the base model, the base model trained on arithmetic, and our model. The results of the arithmetic computations are measured based on the ln RMSE of the model output and the ground-truth numeral. The results of the GLUE tasks follow the metrics as described in §3.1. |  ln RMSE | CoLA | MNLI | MRPC | SST-2 | STS-B
---|---|---|---|---|---|---
Base | $3.5367_{0.000}$ | $\textbf{0.4827}_{0.000}$ | $\textbf{0.8074}_{0.000}$ | $\textbf{0.8797}_{0.000}$ | $\textbf{0.8967}_{0.000}$ | $\textbf{0.8740}_{0.000}$
Base + Arithmetics | $0.4443_{0.011}$ | $0.0000_{0.000}$ | $0.3553_{0.001}$ | $0.7524_{0.000}$ | $0.8761_{0.003}$ | $0.3998_{0.079}$
Ours | $\textbf{0.4360}_{0.016}$ | $0.4193_{0.000}$ | $0.7951_{0.004}$ | $0.8570_{0.000}$ | $0.8962_{0.008}$ | $0.8626_{0.005}$
Figure 2: Interplay between the CE & the EWC loss (color-matched) as a
function of training iterations. The first sign of model convergence is
observed at $\lambda$=1e-8, and although the convergence improves with
decreasing values of $\lambda$, we choose $\lambda$ to 1e-8 to promote
balanced learning of $\theta_{arith}$ with proper retention of $\theta_{gen}$.

##  4 Discussion

From the results in table 1, we infer that our model, while performing the
best in arithmetic computation - orders of magnitude better than the base
model, closely competes with the linguistic prowess of the base model. Besides
preventing performance degradation on the MNLI, STS-B, and the MNLI tasks, for
the CoLA task that suffered significant degradation with its Matthews
correlation coefficient dropping to 0 after being trained on the arithmetic
dataset, our model was able to revitalize it to a value competitive to the
base model.

Returning to our original premise, this paper serves as a proof-of-concept
that the inherent barries to skill injection caused by the catastrophic
forgetting tendencies of large networks can be overcome with weight
consolidation schemes specifically tailored for language modeling. Please see
Appendix §5.1 for a disucssion on the limitations of our efforts. Our datasets
as well as the codebase is hosted at https://github.com/Mandar-
Sharma/OvercomingBarriers.

## References

  * [1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in Neural Information Processing Systems, 33:1877–1901, 2020.
  * [2] Daniel Cera, Mona Diabb, Eneko Agirrec, Inigo Lopez-Gazpioc, Lucia Speciad, and Basque Country Donostia.  Semeval-2017 task 1: Semantic textual similarity multilingual and cross-lingual focused evaluation.
  * [3] Bill Dolan and Chris Brockett.  Automatically constructing a corpus of sentential paraphrases.  In Third International Workshop on Paraphrasing (IWP2005), 2005\.
  * [4] Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner.  Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368–2378, 2019.
  * [5] Mor Geva, Ankit Gupta, and Jonathan Berant.  Injecting numerical reasoning skills into language models.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 946–958, 2020.
  * [6] Song Han, Jeff Pool, John Tran, and William Dally.  Learning both weights and connections for efficient neural network.  Advances in Neural Information Processing Systems, 28, 2015.
  * [7] Chengyue Jiang, Zhonglin Nian, Kaihao Guo, Shanbo Chu, Yinggong Zhao, Libin Shen, and Kewei Tu.  Learning numeral embedding.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2586–2599, 2020.
  * [8] Devin Johnson, Denise Mak, Andrew Barker, and Lexi Loessberg-Zahl.  Probing for multilingual numerical understanding in transformer-based language models.  In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 184–192, 2020.
  * [9] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of NAACL-HLT, pages 4171–4186, 2019.
  * [10] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017.
  * [11] Dharshan Kumaran, Demis Hassabis, and James L McClelland.  What learning systems do intelligent agents need? complementary learning systems theory updated.  Trends in Cognitive Sciences, 20(7):512–534, 2016.
  * [12] Aman Madaan, Ashish Mittal, Ganesh Ramakrishnan, Sunita Sarawagi, et al.  Numerical relation extraction with minimal supervision.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
  * [13] Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher.  Learned in translation: Contextualized word vectors.  Advances in neural information processing systems, 30, 2017.
  * [14] Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig.  Stress test evaluation for natural language inference.  In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353, 2018.
  * [15] Theodore M Porter.  Trust in numbers.  In Trust in Numbers. Princeton University Press, 1996.
  * [16] Abhilasha Ravichander, Aakanksha Naik, Carolyn Rose, and Eduard Hovy.  Equate: A benchmark evaluation framework for quantitative reasoning in natural language inference.  In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 349–361, 2019.
  * [17] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled bersion of bert: Smaller, faster, cheaper and lighter.  2019\.
  * [18] Mandar Sharma, Ajay Gogineni, and Naren Ramakrishnan.  Innovations in neural data-to-text generation.  arXiv preprint arXiv:2207.12571, 2022.
  * [19] Georgios Spithourakis and Sebastian Riedel.  Numeracy for language models: Evaluating and improving their ability to predict numbers.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2104–2115, 2018.
  * [20] Dhanasekar Sundararaman, Shijing Si, Vivek Subramanian, Guoyin Wang, Devamanyu Hazarika, and Lawrence Carin.  Methods for numeracy-preserving word embeddings.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4742–4753, 2020.
  * [21] Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin.  Predicting neural network accuracy from weights.  2020\.
  * [22] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of Machine Learning Research, 9(11), 2008.
  * [23] Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner.  Do nlp models know numbers? probing numeracy in embeddings.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5307–5315, 2019.
  * [24] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, 2018.
  * [25] Alex Warstadt, Amanpreet Singh, and Samuel R Bowman.  Cola: The corpus of linguistic acceptability.  2019\.
  * [26] Adina Williams, Nikita Nangia, and Samuel Bowman.  A broad-coverage challenge corpus for sentence understanding through inference.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, 2018.
  * [27] Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar, and Dan Roth.  Do language embeddings capture scales?  In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 292–299, 2020.

##  5 Appendix

###  5.1 Limitations

We believe this work to be a proof-of-concept for addressing the important
problem of non-linguistic skill injection in language modeling. Besides
mathematical reasoning, the set of non-linguistic skills can expand to
encompass logical inference, and dataset comprehension. Additionally, our in-
house arithmetic dataset comprises of addition and subtraction - we hope to
extend our highly generic proposed framework to incorporate other mathematical
operations like multiplication, division, and exponentiation, in addition to
incorporating decimal-point numerals in the dataset.

###  5.2 Numeral Distribution of the Arithmetic Dataset

(a) EQUATE

(b) DROP

Figure 3: Numeral distribution histograms of the EQUATE & DROP datasets based
on powers of 10.


#  Extraction of instantaneous frequencies and amplitudes in nonstationary
time-series data

Daniel E. Shea∗, Rajiv Giridharagopal∗∗, David S. Ginger∗∗,
Steven L. Brunton†, J. Nathan Kutz‡
∗ Department of Material Science, University of Washington, Seattle 98195
∗∗ Department of Chemistry, University of Washington, Seattle 98195
† Department of Mechanical Engineering, University of Washington, Seattle
98195
‡ Department of Applied Mathematics, University of Washington, Seattle 98195

###### Abstract

Time-series analysis is critical for a diversity of applications in science
and engineering. By leveraging the strengths of modern gradient descent
algorithms, the Fourier transform, multi-resolution analysis, and Bayesian
spectral analysis, we propose a data-driven approach to time-frequency
analysis that circumvents many of the shortcomings of classic approaches,
including the extraction of nonstationary signals with discontinuities in
their behavior. The method introduced is equivalent to a nonstationary Fourier
mode decomposition (NFMD) for nonstationary and nonlinear temporal signals,
allowing for the accurate identification of instantaneous frequencies and
their amplitudes. The method is demonstrated on a diversity of time-series
data, including on data from cantilever-based electrostatic force microscopy
to quantify the time-dependent evolution of charging dynamics at the
nanoscale.

##  1 Introduction

Time series data analysis is ubiquitous and foundational in scientific
analysis and engineering model design [1]. Indeed, it has revolutionized
nearly every scientific discipline by enabling the development of test models
for observed natural phenomena in diverse applications that include planetary
motion, chemical reactions, meteorological patterns, and transport phenomena.
In a typical scientific workflow, observations are made on a system and fit to
a time series model, which can include classical methods from statistics, such
as ARIMA and its variants [1], or more recent neural network based approaches
[2, 3], such as LSTM [4] (long-term, short-term memory), GRU (gated recurrent
units) [5], and echo-state networks [6]. These diverse mathematical strategies
regress to models fit to historical training data, often making assumptions
that the data is generated from a stationary process with Gaussian distributed
statistics. However, this workflow is often complicated by observations with
non-Guassian noise, the existence of nonstationary processes, and/or nonlinear
system dynamics. These challenges make forecasting exceptionally difficult,
requiring the re-training of models as new data becomes available. By
integrating elements of modern gradient descent algorithms, the Fourier
transform, multi-resolution analysis, and Bayesian spectral analysis [7], we
can learn an interpretable Fourier mode-based model for analyzing
nonstationary signals with periodic components, thus circumventing the
challenges normally associated with nonstationary processes and allowing for
accurate identification of instantaneous frequencies and their amplitudes.

Joseph Fourier revolutionized time series analysis with the introduction of
his eponymous transform in 1822, which he developed while studying heat
conduction [8]. The transform empowered an understanding of the frequency-
energy spectrum of time-series signals and spurred the development of Fourier
transform-powered time-frequency analyses [9]. These spectrum-based analysis
tools have been extensively applied in systems exhibiting periodic and quasi-
periodic behaviors, including oscillators and waves. So extensive are the
applications that the field of harmonic analysis has emerged as a consequence.
Harmonic analysis has been applied to a diverse range of problems spanning
many size and time scales, including mechanical vibrations and machine health
monitoring [10], speech and music recognition [11, 12], oceanographic tide
modeling [13], telecommunications and power systems [14], and quantum
mechanics [15].

Despite its widespread use and generality, the Fourier transform has a number
of restrictions that limit its utility in analyzing nonlinear and
nonstationary processes [16]. This was recognized by Denis Gabor in
considering radar technologies of the mid-20th century [17]. Indeed, Gabor
suggested circumventing these issues in part by using short-time, or windowed,
Fourier transforms. This led to improvements in time-frequency analysis and
eventually to the development of wavelet theory [18]. Gabor and wavelet
transforms provide rich visualizations of time-frequency representations with
spectrograms and scalograms respectively [19]. More recently, the Fourier
transform has inspired a new class of time-frequency methods, detailed below,
that complement traditional Fourier transform-based approaches[20, 21, 22, 23,
24, 25]. Among them, the Hilbert-Huang transform (HHT) [16] has become a
common tool for understanding nonstationary processes. The HHT combines
empirical mode decomposition (EMD), which separates a multi-component signal
into simpler periodic modes called intrinsic mode functions (IMFs), with
Hilbert spectral analysis. Hilbert spectral analysis leverages the Hilbert
transform to compute the analytic signal of each periodic mode identified by
EMD. In turn, the analytic signal can be leveraged to compute the
instantaneous phase, instantaneous frequency, and instantaneous amplitude of
the input signal. A significant theoretical framework has been developed
around applying the Hilbert transform to vibrational problems [26, 27, 28].
Although powerful in practice, the HHT is known to have difficulty with
signals that exhibit nonlinear phase evolution [29]. Furthermore, the
algorithm behind the EMD is empirical, thus lacking a broader theoretical
underpinning.

Our method is complementary with a newer generation of time-frequency analysis
algorithms that address specific shortcomings of classic approaches. The
Tycoon method was introduced to handle signals with extremely fast changing
frequencies [23], but focuses on signals that are (relatively) stationary. The
variational mode decomposition (VMD) aims to simultaneously identify periodic
modes with time-dependent non-linear phase functions by maximizing the
smoothness of the amplitude of the periodic modes [20]. VMD has been
successfully applied to a broad array of problems, though the importance
placed on smooth amplitude functions limits its applications for situations
with discontinuities in modes’ phase or amplitude functions. A wide array of
approaches have been built on the VMD and extend it to different types of
signals and multichannel measurements [30, 21, 10]. A different Fourier mode-
based algorithm has also been developed which uses an approach similar to
basis pursuit [22]. Other approaches focus on decomposition of specific
models, such as harmonic oscillators, and the parameters that describe them.
These approaches include both Hilbert vibrational decomposition and Kalman
filter-based approaches [27, 28, 26, 31] along with sparsity-promoting
decompositions [25, 24]. Wigner distribution-based approaches show great
promise for decomposing multicomponent signals with modes that have crossover
between frequencies [32].

In this work, we develop a method for extracting the instantaneous frequencies
and amplitudes from time-series data. The methods is equivalent to a
nonstationary Fourier mode decomposition (NFMD) for nonstationary and
nonlinear temporal signals. Importantly, it produces interpretable signal
decompositions that can handle signals with multiple periodic components, non-
linear phase functions, and sharp discontinuities in the phase function or
periodic mode amplitudes. Adopting the work of Lange et al [33], which
employed a similar architecture for future state prediction rather than
interpretable time-frequency analysis, the proposed method leverages modern
gradient descent optimization to fit temporally-local linear Fourier modes.
The approach resembles the short time Fourier transform (STFT) wherein smaller
temporal segments of the signal are analyzed independently, and the resulting
analyses are combined to provide a full time-frequency representation of the
signal. The NFMD fits Fourier modes to each signal segment, and computes the
mode frequency and amplitude for each signal segment through a gradient
descent optimization with a nonlinear Fourier basis objective function. The
method results in a superior time-frequency analysis to the HHT for
nonstationary signals, and improves both temporal and spatial resolutions
compared to the STFT. The NFMD can be applied to systems with fast-changing
frequencies and abrupt changes in the signal mean, such as machine health
monitoring, seismology, vibration-based imaging modalities, neurochemical and
biochemicals signals, and photonic sensing.

##  2 Methods

The NFMD analysis proposed combines elements of modern gradient descent
algorithms, the Fourier transform, Bayesian spectral analysis, and an
algorithm similar to STFT to learn an interpretable Fourier mode-based model
for analyzing nonstationary signals with periodic components. In general, we
aim to fit a model $\mathbf{y}_{t}$ to a measured signal $\mathbf{z}_{t}$. We
begin by framing the Fourier mode decomposition approach for an entire time
series signal. The subsequent section shows how the method is combined with a
segment-by-segment analysis, reminiscent of STFT, to propose a full time-
frequency analysis. Finally, the instantaneous signal parameters and
nonstationary part of the signal are addressed. We present a simple algorithm
for computing the nonstationary signal component from the learned Fourier mode
representations.

###  2.1 Fourier Mode Decomposition

Consider the time series data
$\mathbf{z}_{t}[\mathbf{z}(t_{1}),\mathbf{z}(t_{2}),\cdots,\mathbf{z}(t_{n})]$
sampled at $n$ discrete evenly-spaced times in $t\in\mathbb{R}^{+}$. The
signal is assumed to be periodic or quasi-periodic. The most common frequency-
domain signal analysis method is the Fourier transform, which allows the input
signal $\mathbf{z}_{t}$ to be represented as a Fourier series. The series
representation is a sum of sines and cosines that provides insight into the
frequency-energy spectrum of the signal.

The Fourier transform, typically implemented as the Fast Fourier Transform
(FFT), assumes a periodic input signal that satisfies the relationship
$\mathbf{z}(t_{i})=\mathbf{z}(t_{i}+P)$, where $P$ is the period of the
periodic data. This periodicity assumption imparts a limited frequency
resolution, $F_{s}/P$, that scales linearly with the sampling frequency of the
signal, $F_{s}$, and inversely with the period $P$. Note the period $P$ is
typically set to the total time $(t_{n}-t_{1})$ of the discrete input signal.

Algorithm 1 Fourier mode signal decomposition (FMD)

Input: Signal $\mathbf{z}_{t}$, Initial frequency guess $\boldsymbol{\omega}$,
Error tolerance $tol$
Output: Learned frequency $\boldsymbol{\omega}$, Learned amplitude
$\mathbf{A}$, Residual error $E(\mathbf{A},\boldsymbol{\omega})$

1:procedure

2: $\mathbf{A}\leftarrow\mathbf{z}_{t}(\Omega(\boldsymbol{\omega}t))^{-1}$
$\triangleright$ Amplitude from initial frequency guess

3: while $E(\mathbf{A},\boldsymbol{\omega})>tol$ do $\triangleright$ Use
gradient descent to optimize frequency vector $\boldsymbol{\omega}$

4:
$\boldsymbol{\omega}\leftarrow\operatorname*{arg\,min}_{\boldsymbol{\omega}^{*}}E(A,\boldsymbol{\omega}^{*})$

5: $\mathbf{A}\leftarrow\mathbf{z}_{t}(\Omega(\boldsymbol{\omega}t))^{-1}$

6: Update $E(\mathbf{A},\boldsymbol{\omega})$

7: end while

8: return $\boldsymbol{\omega}$, $A$, $E(A,\boldsymbol{\omega})$

9:end procedure

To overcome these limitations of the FFT, we adapt the Fourier series
representation and exploit some of its mathematical properties to enable the
NFMD time series analysis. First, the Fourier series model is framed in a more
general context which uses a finite number of modes and flexible mode
frequencies which need to be determined. The model

$\mathbf{y}_{t}(t)=\mathbf{y}_{t}=\sum_{k=1}^{K}F_{k}(t)=\sum_{k=1}^{K}a_{k}\cos(\omega_{k}t)+b_{k}\sin(\omega_{k}t)$

is used, where $F_{k}(t)$ is the general Fourier mode function, the
coefficients $a_{k}$ and $b_{k}$ are the weights for the cosine and sine
components of each mode, respectively, $\omega_{k}$ is the frequency of mode
$k$, and $K$ modes are considered. In matrix form, the model is

$\mathbf{y}_{t}(t)=\mathbf{A}\Omega(\boldsymbol{\omega}t)=\begin{bmatrix}a_{1}&\dots&a_{K}&b_{1}&\dots&b_{K}\end{bmatrix}\begin{bmatrix}\cos(\omega_{1}t)\\\
\vdots\\\ \cos(\omega_{K}t)\\\ \sin(\omega_{1}t)\\\ \vdots\\\
\sin(\omega_{K}t)\end{bmatrix},$ (1)

where $\mathbf{A}\in\mathbb{R}^{2\times K}$ is a vector of the coefficients,
and $\Omega(\boldsymbol{\omega}t)\in\mathbb{R}^{2\times K}$ is a vector of
cosines and sines with frequency $\boldsymbol{\omega}\in\mathbb{R}^{K}$. The
vector $\mathbf{A}$ is determined as the optimal coefficient vector given a
frequency vector $\boldsymbol{\omega}^{*}$, by fitting to the time-series data
$\mathbf{z}_{t}$, using the computation

$\mathbf{A}=\mathbf{z}_{t}(\Omega(\boldsymbol{\omega}^{*}t))^{-1}.$

This is a common approach for discovering component amplitudes in Bayesian
spectral analysis [7]. The vector
$\boldsymbol{\omega}=[\omega_{1},\dots,\omega_{K}]$ is determined by the
optimization

$\textnormal{minimize}\quad E(A,\boldsymbol{\omega})=\sum_{t\in
T}(\mathbf{z}_{t}-A\Omega(\boldsymbol{\omega}t))^{2}.$ (2)

This method has been previously demonstrated, including the theory behind the
optimization [33]. This approach does _not_ appear to be a convex optimization
objective, given the nonlinear objective with cosine and sine functions in
$\Omega(\cdot)$, and therefore should not yield globally optimal solutions.
Although this objective does lack global convexity, this pitfall is avoided by
using initial guesses near the optimal solutions by leveraging the FFT for an
initial guess. The initial guess frames this problem on an error surface that
is locally convex, and therefore allows for highly accurate frequency
estimation by gradient descent. Ultimately, this allows our input signal to be
fit to a model that is a superposition of Fourier modes with superior
frequency resolution to traditional Fourier transforms. The Fourier mode
decomposition (FMD) algorithm for decomposing a full time series signal
$\mathbf{z}_{t}$ is presented in Algorithm 1. This algorithm enables the time-
frequency analysis framework described in the next section.

###  2.2 Nonstationary Fourier Mode Decomposition

The NFMD algorithm builds upon Fourier decompositions to create a descriptive
time-frequency analysis (TFA) framework that exhibits improved frequency
resolution compared to traditional TFA techniques. The NFMD is similar in
principle to one of the most common traditional techniques for TFA, the STFT.
The STFT determines the frequency-domain energy spectrum of temporally local
intervals of a signal. Although effective, the reliance on the traditional
Fourier transform technique limits the frequency resolution of the STFT and
has limited interpretability for understanding the signal components. Figure 1
presents a graphical comparison between the time-frequency analysis approach
of the STFT and the NFMD algorithm. In the STFT, the original signal is
multiplied by a set of windowing functions, such as a Gaussian, that is
progressively applied to subsets of the signal. The signal subsets are then
analyzed by the FFT. In the NFMD, signal segments are sliced and analyzed
individually; the resultant Fourier modes are then fit by the FMD algorithm
described in Section 2.1, thereby enabling a time-frequency analysis.

Specifically, NFMD subsamples segments of the time series of length $\xi$.
These segments take the form $\boldsymbol{\chi}_{i}=\\{{\bf z}(t_{i}),{\bf
z}(t_{i+1})...,{\bf z}(t_{i+\xi/2})\\}$. There are $n-\xi$ segments considered
for an input signal of length $n$. The set of all segments in the signal is
the set ${\bf
X}=\\{\boldsymbol{\chi}_{1},\boldsymbol{\chi}_{2},...,\boldsymbol{\chi}_{i},...,\boldsymbol{\chi}_{n-\xi}\\}$.
The NFMD algorithm is presented in Algorithm 2.

Figure 1: Comparison between STFT and NFMD methods. The STFT computes the
Fourier transform of a convolution of the signal and a windowing function,
$\Psi(t)$. The example shown uses a Gaussian windowing function, which is
commonly referred to as the Gabor transform. The bottom left panel shows the
real (blue) and imaginary (orange) components of the Fourier transform. The
NFMD analyzes a specific segment of the signal, and fits a finite number of
Fourier modes to the signal segment. The modes fit to the signal segment are
shown in the bottom right panel, colored blue and orange. Algorithm 2
Nonstationary Fourier Mode Decomposition

Input: Signal segments $X$, Window size $\xi$, tolerance $tol$
Output: Frequency matrix, $\hat{\omega}$, Coefficients matrix
$\hat{\mathbf{A}}$, residual error vector $\hat{E}$

1:procedure

2: $\boldsymbol{\omega}_{-1}\leftarrow$ maxima(FFT($\boldsymbol{\chi}_{1}$))
$\triangleright$ Initial guess for frequencies using the FFT

3: for $\boldsymbol{\chi}_{i}\;\in\;X$ do

4: $\mathbf{A}_{i}$, $\boldsymbol{\omega}_{i}$, $E_{i}$ =
FMD($\boldsymbol{\chi}_{i}$, $\boldsymbol{\omega}_{-1}$, $tol$)
$\triangleright$ Learn Fourier modes for segment $\boldsymbol{\chi}_{i}$

5: $\boldsymbol{\omega}_{-1}\leftarrow\boldsymbol{\omega}_{i}$

6: end for

7:
$\hat{\mathbf{A}}=[\mathbf{A}_{1}^{T},\cdots,\mathbf{A}_{i}^{T},\cdots,\mathbf{A}_{n-\xi}^{T}]^{T}$
$\triangleright$ Coefficients matrix

8:
$\hat{\boldsymbol{\omega}}=[\boldsymbol{\omega}_{1}^{T},\cdots,\boldsymbol{\omega}_{i}^{T},\cdots,\boldsymbol{\omega}_{n-\xi}^{T}]^{T}$
$\triangleright$ Frequencies matrix

9: $\hat{\mathbf{E}}=[E_{1},\cdots,E_{i},\cdots,E_{n-\xi}]$ $\triangleright$
Residual errors vector

10: return $\hat{\mathbf{A}}$, $\hat{\boldsymbol{\omega}}$, $\hat{\mathbf{E}}$

11:end procedure

The NFMD algorithm takes in a set, ${\bf X}$, of window segments,
$\boldsymbol{\chi}_{i}$, and for each $\boldsymbol{\chi}_{i}\in X$ learns a
coefficient vector ${\bf A}_{i}$ and frequency vector
$\boldsymbol{\omega}_{i}$ using FMD (Algorithm 1). Importantly, the learned
frequency vector from each segment is used as the initial guess to the
subsequent segment. This significantly increased the speed of the algorithm,
by allowing each segment to start with a nominally close estimate of the
frequency vector $\boldsymbol{\omega_{i}}$. The learned vectors from all of
the segments are stored in matrices

$\hat{{\bf A}}=\begin{bmatrix}-\;{\mathbf{A}_{1}}\;-\\\ \vdots\\\
-\;{\mathbf{A}_{i}}\;-\\\ \vdots\\\ -\;{\mathbf{A}_{n-\xi}}\;-\end{bmatrix},$
(3)

and

$\hat{\boldsymbol{\omega}}=\begin{bmatrix}-\;\boldsymbol{\omega}_{1}\;-\\\
\vdots\\\ -\;\boldsymbol{\omega}_{i}\;-\\\ \vdots\\\
-\;\boldsymbol{\omega}_{n-\xi}\;-\end{bmatrix}.$ (4)

The learned frequency vectors,
$\boldsymbol{\omega}_{i}=[\omega_{1,i},...,\omega_{k,i},...\omega_{K,i}]$, and
coefficient vectors, $\mathbf{A}_{i}=[A_{1,i},...,A_{k,i},...,A_{K,i}]$,
contain the frequencies and coefficients for the $K$ fourier modes of each
signal segment $\boldsymbol{\chi}_{i}\in X$.

###  2.3 Instantaneous Frequency and Amplitude

The NFMD decomposes nonstationary, nonlinear signals by fitting a set of
Fourier modes to a set ${\bf X}$ of signal segments, as previously described.
Importantly, for each signal segment $\boldsymbol{\chi}_{i}$ the algorithm
yields a frequencies vector $\boldsymbol{\omega}_{i}$ and coefficients vector
$\mathbf{A}_{i}$. For each Fourier mode $k$, a vector of instantaneous
frequencies can be constructed by collecting the learned Fourier mode
frequency $\omega_{k,i}$ for each of the signal segments $i$ for a given mode
$k$. The instantaneous frequency vector takes the form

$\boldsymbol{\omega}_{k}=[\omega_{k,1},...,\omega_{k,i},...,\omega_{k,n-\xi}].$

The instantaneous amplitude of a Fourier mode can be computed from its
coefficients $a_{k,i}$ and $b_{k,i}$ with the relationship

$\phi_{k,i}=\sqrt{a_{k,i}^{2}+b_{k,i}^{2}}.$

Similar to the instantaneous frequency vector, an instantaneous amplitude
vector can be constructed for each mode $k$ with one element corresponding to
each segment $\boldsymbol{\chi}$,

$\boldsymbol{\phi}_{k}=[\phi_{k,1},...,\phi_{k,i},...,\phi_{k,n-\xi}].$

These metrics are useful for comparing the time-frequency analysis from NFMD
to other existing time-frequency analysis methods that report instantaneous
amplitude and instantaneous frequency of an input time series signal.

###  2.4 Nonstationary Signals

NFMD can provide insight into nonstationary signals of the form

$z(t)=\mu(t)+\sum_{l=1}^{L}A_{l}(t)\cos(\phi_{l}(t)),$ (5)

where $\mu(t)$ is an unknown, non-periodic function describing the
nonstationary part of the signal and there are $L$ periodic cosine modes with
the amplitude function $A_{l}(t)$ and instantaneous phase function
$\phi_{l}(t)$. The term $\mu(t)$ is only assumed to be continuous. We will
call the function $\mu(t)$ the signal’s _instantaneous mean_. The concept of
an instantaneous mean will prove useful in the analysis of nonstationary
signals where the signal mean drifts or trends away from zero.

The frequency and coefficient vectors allow computation of individual Fourier
modes $F_{k,i}=a_{k,i}\cos(\omega_{k,i}t)+b_{k,i}\cos(\omega_{k,i}t)$ for each
of the modes $k\in[1,K]$ and signal segments $\boldsymbol{\chi}_{i}$. In
signals that exhibit a moving mean, there is a mode that will account for the
nonstationary part of the signal. The instantaneous mean of the signal,
$\mu(t)$, can be estimated from the Fourier modes $F_{k,i}$ corresponding to
the mode that accounts for the nonstationary part of the signal. Many
interpolation strategies can be implemented to compute the instantaneous mean.
We implement a simple strategy of concatenating the value of the Fourier mode
$F_{k}$ at the median time from each time segment $\boldsymbol{\chi}_{i}$. To
visualize this implementation, imagine we construct the matrix

$\mathcal{M}=\begin{bmatrix}-\;F_{k,1}\;-\\\ \vdots\\\ -\;F_{k,i}\;-\\\
\vdots\\\
-\;F_{k,n-\xi}\;-\end{bmatrix}=\begin{bmatrix}F_{k,1}(t_{1})&\dots&F_{k,1}(t_{1+\xi/2})&\dots&F_{k,1}(t_{1+\xi})\\\
\vdots&&\vdots&&\vdots\\\
F_{k,i}(t_{i})&\dots&F_{k,i}(t_{i+\xi/2})&\dots&F_{k,i}(t_{i+\xi})\\\
\vdots&&\vdots&&\vdots\\\
F_{k,n-\xi}(t_{n-\xi})&\dots&F_{k,n-\xi}(t_{n-\xi/2})&\dots&F_{k,n-\xi}(t_{n})\end{bmatrix}.$

The approach is to use the center column of the matrix $\mathcal{M}$ as the
instantaneous mean. This mean is defined as

$\boldsymbol{\mu}=[F_{k,1}(t_{1+\xi/2}),...,F_{k,i}(t_{i+\xi/2}),...,F_{k,n-\xi}(t_{n-\xi/2})].$

The mode representing the mean can be challenging to identify, and generally
requires inspection of each of the modes identified by NFMD. In this work, the
instantaneous mean is always the lowest-frequency mode.

##  3 Results

We benchmark the NFMD against the Hilbert-Huang Transform for time-frequency
analysis of a series of nonstationary multi-component signals. The two methods
are compared with and without noise, and on signals with abrupt changes in
instantaneous frequency and instantaneous amplitude. After comparing NFMD with
the HHT, a pair of oscillator examples are provided to demonstrate how the
NFMD and the discovered instantaneous mean can provide insight into the
forcing function applied to the oscillator. Finally, the method is applied to
simulated and experimental data for a real-world oscillator-based microscopy
method. The instantaneous mean is proven to be effective in an experimental
data set by validating the form of the discovered instantaneous mean against
experimental control data with known forcing functions.

###  3.1 Synthetic Test Signals

Consider a basic multi-component signal with a non-periodic mean:

$\displaystyle z(t)$ $\displaystyle=z_{1}(t)+z_{2}(t)+\mu(t)$ $\displaystyle
z_{1}(t)$ $\displaystyle=A_{1}(t)\cos(2\pi\omega_{1}(t)t)$ $\displaystyle
z_{2}(t)$ $\displaystyle=A_{2}(t)\cos(2\pi\omega_{2}(t)t),$

with amplitude, phase, and instantaneous mean functions

$\displaystyle A_{1}(t)$ $\displaystyle=1+0.5\exp(-t/3)$
$\displaystyle\omega_{1}(t)$ $\displaystyle=360-10\exp(-t/0.5)$ $\displaystyle
A_{2}(t)$ $\displaystyle=8-0.5\exp(-t)$ $\displaystyle\omega_{2}(t)$
$\displaystyle=80-2t$ $\displaystyle\mu(t)$
$\displaystyle=1.5+2.5\exp(-x/1.5),$

where $z(t)$ is the signal, $z_{1}(t)$ and $z_{2}(t)$ are the periodic
components in $z(t)$, and $\mu(t)$ is the slow-moving non-periodic mean.
Figure 2 shows a traditional STFT spectrogram and PSD analysis of the signal,
and Figure 3 shows the true periodic modes, with instantaneous parameters, and
the mean $\mu(t)$. The signal is generated over one second with sampling
interval $\Delta t=2\times 10^{-4}$ s.

Figure 2: Example signal for demonstrating NFMD. The true signal is presented
with the short time Fourier transform and the power spectral density (PSD).
The signal has three maxima that appear in the PSD and STFT, indicating that
there are likely three modes to be considered. Figure 3: Instantaneous mean,
amplitude, and frequency of the example signal. Only part of the signal, from
$t=0$ to $t=0.1$ is shown. The instantaneous frequency, $\omega(t)$, and
amplitude, $A(t)$, are presented for the two periodic modes. The mean,
$\mu(t)$ of the signal is denoted on the signal itself.

Signal decomposition is performed with both the NFMD and HHT. The HHT employs
the EMD to identify a set of empirical modes. The EMD takes a number of
hyperparameters which adjust the outputs, including the number of identified
modes. The HHT used in this work used $\theta_{1}=0.05$, $\theta_{2}=0.5$,
$\alpha=0.05$, and identified four modes in the signal. The two most important
hyperparameters for the NFMD are the window size (similar to STFT) and the
number of modes to fit to the data. For this data, three modes are fit to a
window size of 250 points (or $0.05$ seconds). Figure 4 shows the decomposed
signal for both HHT and NFMD, where NFMD does a significantly better job of
identifying both the instantaneous frequency and instantaneous amplitude of
the signal. Adding noise to the signal significantly affects the HHT’s
decomposition when compared to the NFMD, as demonstrated in Figure 5. Noise in
the signal is added at a signal-to-noise ratio (SNR) of 35. The SNR in this
work is SNR $=10\log_{10}(\lVert
u(t)\rVert_{2}^{2}/\lVert(\tilde{u}(t)-u(t)\rVert_{2}^{2})$.

Figure 4: Decomposition of example signal by NFMD and HHT without noise. The
estimated instantaneous mean mode is shown for both the HHT and NFMD
decomposition methods. The instantaneous frequency and amplitude of the two
periodic components is also shown for both decomposition approaches. Figure 5:
Decomposition of example signal with added noise (SNR=35). The NFMD correctly
identifies the instantaneous frequency, amplitude, and mean of the signal,
although it does propagate noise in the system. The HHT begins to show erratic
behavior in its estimates of all instantaneous parameters. Figure 6:
Decomposition of signal with discontinuity in instantaneous frequency of a
periodic mode. Data contains additive white noise with SNR=25. The NFMD
correctly identifies the three modes, including the instantaneous mean and the
frequency and amplitudes of the periodic modes. The HHT fails to identify the
correct number of modes and assigns extra modes to the data. The periodic
modes uncovered by HHT do not qualitatively match any of the correct signal
modes.

NFMD enables signal decomposition even for noisy signals where the HHT begins
to have difficulty identifying intrinsic mode functions. The NFMD propagates
the noise from the signal through the decomposition process, showing some
noise-like error in the estimated instantaneous frequency vectors, amplitude
vectors, and mean. However, the NFMD provides clearly superior decomposition
of the signal.

One particular advantage of NFMD is the ability to correctly identify sharply-
changing instantaneous frequency in periodic modes or abrupt changes in the
instantaneous mean of an input signal. This is an advantage against other
modern optimization-based methods built on VMD, because VMD prioritizes
finding solutions with smooth amplitude functions [20, 10]. However, the NFMD
has no preference for smooth amplitudes and can accurately decompose these
types of signals with abrupt changes in the instantaneous frequency,
instantaneous amplitude, or mean. For example, consider a signal with an
abrupt change in the instantaneous frequency of one of its modes at $t=0.5$.
The model for this example is

$\displaystyle z(t)$ $\displaystyle=z_{1}(t)+z_{2}(t)+\mu(t)$ $\displaystyle
z_{1}(t)$ $\displaystyle=A_{1}(t)\cos(2\pi\omega_{1}(t)t)$ $\displaystyle
z_{2}(t)$ $\displaystyle=A_{2}(t)\cos(2\pi\omega_{2}(t)t),$

with amplitude, phase, and instantaneous mean functions

$\displaystyle A_{1}(t)$ $\displaystyle=2+\exp(-t/4)$
$\displaystyle\omega_{1}(t)$ $\displaystyle=400+10H(0.5)(1-\exp((t-0.5)/0.1)$
$\displaystyle A_{2}(t)$ $\displaystyle=2t+2$ $\displaystyle\omega_{2}(t)$
$\displaystyle=60-t$ $\displaystyle\mu(t)$
$\displaystyle=1.5+2.5\exp(-x/1.5),$

where $H(0.5)$ is the Heaviside function centered at $t=0.5$s. Noise is added
to the signal at an SNR ratio of 25 and decomposed by both HHT and NFMD. The
decomposed signal is shown in Figure 6. The NFMD correctly identifies the
sharp transition in frequency, and consequently obtains qualitatively good
estimates on the amplitude of both periodic modes. Results from the HHT are
presented, but it is important to note that in cases like this example, the
HHT often yields extra intrinsic mode functions. It is challenging to assign
which of the HHT modes are intended to represent which periodic mode, so the
presented results were quantitatively closest to the ’true’ mode by comparing
both instantaneous frequency and instantaneous amplitude vectors.
Additionally, all of the remaining modes from HHT are summed together to
estimate the instantaneous mean. Although the HHT yields a reasonable looking
instantaneous mean, neither of the periodic modes are correctly decomposed in
this example.

Figure 7: Decomposition of signal with discontinuity in instantaneous mean.
Data contains additive white noise with SNR=20. The NFMD correctly identifies
the periodic mode instantaneous frequency and amplitude, and the instantaneous
mean of the signal. The HHT fails to identify either the mean or the periodic
component.

As another example, we consider a situation where the instantaneous mean of
the signal changes abruptly in the middle of the signal ($t=0.5$ seconds). The
input signal uses the model

$\displaystyle z(t)$ $\displaystyle=z_{1}(t)+\mu(t)$ $\displaystyle z_{1}(t)$
$\displaystyle=A_{1}(t)\cos(2\pi\omega_{1}(t)t)$

with amplitude, phase, and instantaneous mean functions

$\displaystyle A_{1}(t)$ $\displaystyle=5-0.5\exp(-t/3)$
$\displaystyle\omega_{1}(t)$ $\displaystyle=245+10t^{2}$ $\displaystyle\mu(t)$
$\displaystyle=\left\\{\begin{array}[]{ll}\sin{}(2t)&\quad t\leq 0.25\\\
-2.5\cos{}(t)&\quad t>0.25.\end{array}\right.$

The signal has added Gaussian noise with SNR$=20$. As shown in Figure 7, the
NFMD accurately estimates instantaneous signal mean and identifies the correct
periodic mode. The HHT fails to find the instantaneous frequency of the
periodic mode, and errantly suggests a low-frequency wave as the instantaneous
mean.

Figure 8: Harmonic oscillator solutions with various forcings. The left column
shows the applied forcing, and the right column shows the solution to the
harmonic oscillator assuming it starts from rest. The top row shows a periodic
driving force, the middle row shows a non-periodic perturbation, and the
bottom row shows a superposition of the two forcings.

Having demonstrated that NFMD offers material advantages over HHT methods for
TFA, both in terms of mode extraction in noisy signals and for reacting to
sharp changes in instantaneous parameters, we next discuss applications to
realistic systems.

Figure 9: The signal mean (dashed) of the perturbed, periodically driven
oscillator matches the solution of the perturbation-only oscillator (orange).
This enables NFMD to directly probe non-oscillatory driving forces applied to
oscillators by accurately recovering the signal mean of a nonstationary
signal. Figure 10: Decomposition of forced oscillator signal by NFMD. The NFMD
decomposes a harmonic oscillator forced with a periodic forcing function and
non-periodic perturbation. The instantaneous mean of the signal, $\mu(t)$, is
directly correlated to the non-periodic forcing.

###  3.2 Applications

The example synthetic test signals above illustrate the power of NFMD to
decompose signals with abruptly changing instantaneous parameters and
accurately estimate the instantaneous mean of noisy signals. One application
of the instantaneous mean is estimating the non-periodic forces applied to an
oscillatory system. Consider the ubiquitous harmonic oscillator

$\ddot{x}+2\beta\omega_{0}\dot{x}+\omega_{0}^{2}=F(t)/m,$

where $x$ is the position of the oscillator, $\beta$ is a damping factor,
$\omega_{0}$ is the resonant frequency of the oscillator, $F(t)$ is an applied
forcing, and $m$ is the mass of the oscillator. We consider the solution to
this oscillator for three types of applied forcing functions: a periodic
driving force, $F_{d}(t)$, and a non-periodic perturbation forcing,
$F_{p}(t)$, and a combination of periodic and perturbation forcing functions,
$F_{d}(t)+F_{p}(t)$.

The driving force $F_{d}(t)$ is of the form $\alpha e^{i\omega t}$, where
$\omega$ is the frequency and $\alpha$ is the amplitude of the driving force.
The perturbation forcing function has the general form

$F_{p}(t)=+H(t-t^{\prime})\gamma(1-e^{-(t-t^{\prime})/\tau}),$ (6)

where $H$ is the Heaviside function, $t^{\prime}$ is a perturbation onset
time, and $\tau$ is a characteristic relaxation time constant. An oscillator
with this forcing function will have a solution of the form

$x_{p}(t)\propto\phi e^{i\omega t}+\psi e^{-(t-t^{\prime})/\tau},$ (7)

where $\phi$ is a prefactor determined by the parameters in the governing
equation. Forcing functions and solutions for an oscillator forced by
$F_{d}(t)$, $F_{p}(t)$, and $F_{p}(t)+F_{d}(t)$ are presented in Figure 8.

The solution to the combined case ($F(t)=F_{d}(t)+F_{p}(t)$) has an
instantaneous mean that is nearly identical to the solution for the
perturbation-only ($F(t)=F_{p}(t)$) case. This enables the NFMD signal
decomposition to provide insight into the form of the non-periodic forcing
applied to an oscillator by estimating the instantaneous mean of the signal.
The instantaneous mean and the perturbed solution are plotted on top of the
solution to the driven, perturbed solution in Figure 9.

We confirm this approach works by using a series of numerical simulations. The
same periodic driving force, $F_{d}(t)$, is used for all simulations while a
set of different perturbation forces, $F_{p}(t)$, is applied to each
oscillator. The perturbation forces have different relaxation times, $\tau$ in
equation 6. The relaxation times vary from $10^{-7}$ to $10^{-3}$ s. The
simulated oscillators are all subjected to the combined driving force and
perturbation force. The signals have added white noise with SNR$=100$.

Figure 10 shows the decomposition of a single simulated oscillator. Note the
time-varying mean of the signal correlates directly to the non-periodic
forcing function. Figure 11 shows the signal means discovered with the NFMD
and models fit to the instantaneous means. The model
$H(t-t^{\prime})\alpha(1-\exp{}((t-t^{\prime})/\tau)$ is fit to each of the
instantaneous means. This model makes it possible to identify the relaxation
time $\tau$ in the perturbation function. Below approximately $1\mu s$, the
estimated $\tau$ begins to flatten out. This is a result of the window size
($\xi$) used for model fitting, which is approximately a $1\mu s$ window
width.

Figure 11: Model fit comparison between instantaneous mean from NFMD and true
parameters from non-periodic forcing. The simulated oscillator is subjected to
a perturbation at $t=0.001$ seconds (left panel). A model is fit to the
discovered instantaneous mean mode $\mu(t)$. The time constant $\tau$ in the
fit model is compared to the time constant of the perturbation force
$F_{p}(t)$ (right panel). Figure 12: NFMD decomposition of experimental trEFM
control data with controlled perturbation time constants. A model is fit which
contains the time constant $\tau$ and compared with truth values.

In this proposed harmonic oscillator application, the NFMD can identify the
instantaneous mean of a signal. The instantaneous mean of the signal is then
fit to a model, which can provide insight to the form of the non-periodic
perturbation applied to the system.

###  3.3 Experimental Application

We test this application on a real-world cantilever based imaging modality
termed time-resolved electrostatic force microscopy (trEFM), which is an
electrical modification applied to the nanoscale imaging technique of atomic
force microscopy [34, 35, 36, 37, 38]. In trEFM, a periodically driven
metallic cantilever is brought into close proximity to a surface. An electric
field generated by an accumulation of electrical charge on the surface imparts
a force on the metallic cantilever, as well as an electrostatic force gradient
[39]. The electrostatic force is typically triggered by an external stimulus,
such as a voltage signal or photogenerated charge via an optical excitation
source. Such methods are useful for extracting dynamic information in
nanoscale measurements of photovoltaic or ionic conducting systems via the
effect of the force on the cantilever’s resonance frequency. In a typical
trEFM experiment, the desired outcome is extracting an unknown characteristic
time constant $\tau$, usually describing the time-dependent change in the
electrostatic force gradient. As a test case for NFMD, we apply a series of
perturbation forces to the cantilever of the form (6). In this experiment, the
relaxation time $\tau$ is controlled in the applied forcing, providing a set
of experimental data with different, known relaxation times.

The NFMD is used to decompose the experimentally-measured cantilever signal
into a single periodic mode (with instantaneous frequency near the periodic
driving force) and an instantaneous mean. The instantaneous mean can then be
fit to a model for the forcing term. The model we used is

$\mu(t)=H(t-t^{\prime})\alpha\exp{}(-\lambda(t-t^{\prime}))(1-\exp{}((t-t^{\prime})/\tau),$

where $H(t-t^{\prime})$ is the Heaviside function centered at $t-t^{\prime}$,
$t$ is the time, $t^{\prime}$ is the perturbation onset time, $\alpha$ is an
amplitude constant, $\lambda$ is a decay constant, and $\tau$ is the
relaxation time. The term with the decay constant $\lambda$ was included in
this model to fit a recurring pattern in the data that is likely a constant
related to the cantilever. Five different versions of this model were fit to
the data: one with a fixed perturbation time $t^{\prime}=0.168\mu$s, one with
a fixed decay constant $\lambda=1080$, one with both parameters fixed, one
with $\lambda$ set to zero, and one with all parameters fit by the model.
Figure 12 shows the result of fitting these models to the instantaneous means
of the experimental data.

Most of the models perform similarly, though the model with the decay constant
$\lambda$ set to zero tends to underestimate the correct relaxation time.
Similar to the simulation results, the same trend occurs around $\tau=1\mu s$,
where the model no longer fits the truth line and the predicted $\tau$
flattens out. Unfortunately, the size of the window is limited by the
frequency of the periodic components of the input signal. In both cases, the
$1\mu$s window was the minimum window size where the NFMD decomposition
successfully decomposes the signal.

It is important to compare this approach with the existing methods for trEFM
time constant estimation [38]. One current method is using the instantaneous
frequency vector [40], $\omega_{k}(t)$, of the periodic mode and estimating
the time between the perturbation onset and the next local minima in the
instantaneous frequency curve. Prior research showed an empirical correlation
between this time interval and the relaxation time of the perturbation. To
define the particular experiment, a calibration curve is used to estimate a
relaxation time given the instantaneous frequency vector. This approach is
effective, and enables identification of sub-microsecond relaxation time
constant [37]. However, the main drawback of the method in [40] is that the
correlation is indirect. Therefore, the external perturbation (namely, $\tau$)
is not directly learned, and the calibration curve will change based on
experimental variables such as the cantilever being used with a different set
of physical parameters like quality factor and spring constant. For more
complicated systems where the relevant timescales are more than single
exponential (modern photovoltaic systems with ionic transport and dielectric
relaxation in battery materials), the lack of a defined model in this
calibration curve can prove limiting [41, 42]. A chief advantage of the
proposed NFMD approach is that the forcing function can be detected directly
from experimental data via the instantaneous mean.

##  4 Conclusion

Time-frequency analysis methods are critically important in science and
engineering. In this work, we develop a data-driven approach to time-frequency
analysis that helps address shortcomings of classic approaches, including the
extraction of nonstationary signals with discontinuities in their behavior. By
integrating elements of modern gradient descent algorithms, the Fourier
transform, multi-resolution analysis, and Bayesian spectral analysis, we can
learn an interpretable Fourier mode-based model for analyzing nonstationary
signals with periodic components, thus circumventing the deleterious effects
normally associated with nonstationary processes and allowing for accurate
identification of instantaneous frequencies and their amplitudes. Indeed, our
method is equivalent to a nonstationary Fourier mode decomposition (NFMD) for
nonstationary and nonlinear temporal signals. Importantly, it produces
interpretable signal decompositions that can handle signals with multiple
periodic components, nonlinear phase functions, and sharp discontinuities in
the phase function or periodic mode amplitudes. The method results in a
superior time-frequency analysis to the HHT for nonstationary signals, and
improves both temporal and spatial resolutions compared to the STFT, thus
providing a viable and broadly applicable architecture for integration into a
diverse number of scientific processes.

## Acknowledgements

The authors acknowledge support from the NSF MRSEC program under a SuperSeed
award (1719797). DS, SLB, and JNK would especially like to acknowledge Henning
Lange for helpful discussion on the FMD decomposition algorithm. Part of this
work was conducted at the Molecular Analysis Facility, a National
Nanotechnology Coordinated Infrastructure site at the University of Washington
which is supported in part by the National Science Foundation (grant
NNCI-1542101), the University of Washington, the Molecular Engineering and
Sciences Institute, the Clean Energy Institute, and the National Institutes of
Health.

## References

  * [1] J.D. Hamilton and Princeton University Press “Time Series Analysis”, Time Series Analysis v. 10 Princeton University Press, 1994 URL: https://books.google.com/books?id=B8_1UBmqVUoC
  * [2] Yoshua Bengio, Ian Goodfellow and Aaron Courville “Deep learning” MIT press Massachusetts, USA:, 2017
  * [3] S.. Brunton and J.. Kutz “Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control” Cambridge University Press, 2019
  * [4] Sepp Hochreiter and Jürgen Schmidhuber “Long short-term memory” In _Neural computation_ 9.8, 1997, pp. 1735–1780
  * [5] Zhengping Che et al. “Recurrent Neural Networks for Multivariate Time Series with Missing Values” In _Scientific Reports_ 8.1, 2018, pp. 6085 DOI: 10.1038/s41598-018-24271-9
  * [6] Jaideep Pathak et al. “Model-free prediction of large spatiotemporally chaotic systems from data: a reservoir computing approach” In _Physical review letters_ 120.2, 2018, pp. 024102
  * [7] G. Bretthorst “Bayesian spectrum analysis and parameter estimation”, Lecture notes in statistics 48 New York: Springer-Verlag, 1988
  * [8] J.B.J. Fourier and Firmin père & fils Didot “Théorie analytique de la chaleur, par M. Fourier” chez Firmin Didot, pere et fils, 1822 URL: https://books.google.com/books?id=T3NcDFeURpEC
  * [9] J. Allen “Short term spectral analysis, synthesis, and modification by discrete Fourier transform” In _IEEE Transactions on Acoustics, Speech, and Signal Processing_ 25.3, 1977, pp. 235–238 DOI: 10.1109/TASSP.1977.1162950
  * [10] Guowei Tu et al. “Iterative nonlinear chirp mode decomposition: A Hilbert-Huang transform-like method in capturing intra-wave modulations of nonlinear responses” In _Journal of Sound and Vibration_ 485, 2020, pp. 115571 DOI: 10.1016/j.jsv.2020.115571
  * [11] S.. Hadjidimitriou and L.. Hadjileontiadis “Toward an EEG-Based Recognition of Music Liking Using Time-Frequency Analysis” In _IEEE Transactions on Biomedical Engineering_ 59.12, 2012, pp. 3498–3510 DOI: 10.1109/TBME.2012.2217495
  * [12] Marián Képesi and Luis Weruaga “Adaptive chirp-based time–frequency analysis of speech signals” In _Speech Communication_ 48.5, 2006, pp. 474–492 DOI: https://doi.org/10.1016/j.specom.2005.08.004
  * [13] Leicheng Guo et al. “River-Tide Dynamics: Exploration of Nonstationary and Nonlinear Tidal Behavior in the Yangtze River Estuary: River Tidal Dynamics” In _Journal of Geophysical Research: Oceans_ 120.5, 2015, pp. 3499–3521 DOI: 10.1002/2014JC010491
  * [14] Y. Lin and M. Tsai “Development of an Improved Time–Frequency Analysis-Based Nonintrusive Load Monitor for Load Demand Identification” In _IEEE Transactions on Instrumentation and Measurement_ 63.6, 2014, pp. 1470–1483 DOI: 10.1109/TIM.2013.2289700
  * [15] David J. Griffiths and Darrell F. Schroeter “Introduction to Quantum Mechanics” Cambridge University Press, 2018 DOI: 10.1017/9781316995433
  * [16] Norden E. Huang et al. “The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis” In _Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences_ 454.1971, 1998, pp. 903–995 DOI: 10.1098/rspa.1998.0193
  * [17] Dennis Gabor “Theory of communication. Part 1: The analysis of information” In _Journal of the Institution of Electrical Engineers-Part III: Radio and Communication Engineering_ 93.26 IET, 1946, pp. 429–441
  * [18] Stéphane Mallat “A wavelet tour of signal processing” Elsevier, 1999
  * [19] J Nathan Kutz “Data-driven modeling & scientific computation: methods for complex systems & big data” Oxford University Press, 2013
  * [20] Konstantin Dragomiretskiy and Dominique Zosso “Variational Mode Decomposition” In _IEEE Transactions on Signal Processing_ 62.3, 2014, pp. 531–544 DOI: 10.1109/TSP.2013.2288675
  * [21] Shiqian Chen et al. “Nonlinear Chirp Mode Decomposition: A Variational Method” Conference Name: IEEE Transactions on Signal Processing In _IEEE Transactions on Signal Processing_ 65.22, 2017, pp. 6024–6037 DOI: 10.1109/TSP.2017.2731300
  * [22] Pushpendra Singh, Shiv Dutt Joshi, Rakesh Kumar Patney and Kaushik Saha “The Fourier decomposition method for nonlinear and non-stationary time series analysis” In _Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences_ 473.2199, 2017, pp. 20160871 DOI: 10.1098/rspa.2016.0871
  * [23] Matthieu Kowalski, Adrien Meynard and Hau-tieng Wu “Convex Optimization approach to signals with fast varying instantaneous frequency” In _Applied and Computational Harmonic Analysis_ 44.1, 2018, pp. 89–122 DOI: 10.1016/j.acha.2016.03.008
  * [24] Seth M Hirsh, Bingni W Brunton and J Nathan Kutz “Data-driven spatiotemporal modal decomposition for time frequency analysis” In _Applied and Computational Harmonic Analysis_ 49.3 Elsevier, 2020, pp. 771–790
  * [25] Thomas Y Hou and Zuoqiang Shi “Sparse Time-Frequency decomposition for multiple signals with same frequencies” In _Advances in Data Science and Adaptive Analysis_ 9.04 World Scientific, 2017, pp. 1750010
  * [26] Michael Feldman “Hilbert transform in vibration analysis” In _Mechanical Systems and Signal Processing_ 25.3, 2011, pp. 735–802 DOI: 10.1016/j.ymssp.2010.07.018
  * [27] Michael Feldman “Non-linear system vibration analysis using Hilbert transform–II. Forced vibration analysis method ’Forcevib”’ In _Mechanical Systems and Signal Processing_ 8.3, 1994, pp. 309–318
  * [28] Michael Feldman “Non-linear system vibration analysis using Hilbert transform–I. Free vibration analysis method ’Freevib”’ In _Mechanical Systems and Signal Processing_ 8.3, 1994, pp. 119–127
  * [29] Ingrid Daubechies, Jianfeng Lu and Hau-Tieng Wu “Synchrosqueezed Wavelet Transforms: a Tool for Empirical Mode Decomposition” arXiv: 0912.2437 In _arXiv:0912.2437 [math]_ , 2009 URL: http://arxiv.org/abs/0912.2437
  * [30] Naveed ur Rehman and Hania Aftab “Multivariate Variational Mode Decomposition” In _IEEE Transactions on Signal Processing_ 67.23, 2019, pp. 6039–6052 DOI: 10.1109/TSP.2019.2951223
  * [31] Mehrdad Yazdanian, Ali Mehrizi-Sani and Mohsen Mojiri “Estimation of Electromechanical Oscillation Parameters Using an Extended Kalman Filter” In _IEEE Transactions on Power Systems_ 30.6, 2015, pp. 2994–3002 DOI: 10.1109/TPWRS.2015.2388694
  * [32] Ljubiša Stanković, Miloš Brajović, Miloš Daković and Danilo Mandic “On the decomposition of multichannel nonstationary multicomponent signals” In _Signal Processing_ 167, 2020, pp. 107261 DOI: 10.1016/j.sigpro.2019.107261
  * [33] Henning Lange, Steven L Brunton and J Nathan Kutz “From Fourier to Koopman: Spectral Methods for Long-term Time Series Prediction” In _Journal of Machine Learning Research_ 22.41, 2021, pp. 1–38 DOI: http://jmlr.org/papers/v22/20-406.html
  * [34] David C. Coffey and David S. Ginger “Time-resolved electrostatic force microscopy of polymer solar cells” Number: 9 Publisher: Nature Publishing Group In _Nature Materials_ 5.9, 2006, pp. 735–740 DOI: 10.1038/nmat1712
  * [35] Ryan P. Dwyer, Lee E. Harrell and John A. Marohn “Lagrangian and Impedance-Spectroscopy Treatments of Electric Force Microscopy” Publisher: American Physical Society In _Physical Review Applied_ 11.6, 2019, pp. 064020 DOI: 10.1103/PhysRevApplied.11.064020
  * [36] Ryan P. Dwyer, Sarah R. Nathan and John A. Marohn “Microsecond photocapacitance transients observed using a charged microcantilever as a gated mechanical integrator” Publisher: American Association for the Advancement of Science Section: Research Article In _Science Advances_ 3.6, 2017, pp. e1602951 DOI: 10.1126/sciadv.1602951
  * [37] Rajiv Giridharagopal et al. “Submicrosecond Time Resolution Atomic Force Microscopy for Probing Nanoscale Dynamics” Publisher: American Chemical Society In _Nano Letters_ 12.2, 2012, pp. 893–898 DOI: 10.1021/nl203956q
  * [38] Aaron Mascaro et al. “Review of time-resolved non-contact electrostatic force microscopy techniques with applications to ionic transport measurements” Publisher: Beilstein-Institut In _Beilstein Journal of Nanotechnology_ 10.1, 2019, pp. 617–633 DOI: 10.3762/bjnano.10.62
  * [39] S.V. Kalinin and A. Gruverman “Scanning Probe Microscopy: Electrical and Electromechanical Phenomena at the Nanoscale”, Scanning Probe Microscopy: Electrical and Electromechanical Phenomena at the Nanoscale v. 1 Springer New York, 2007 URL: https://books.google.com/books?id=Ua8fxzc2kTwC
  * [40] Durmus U. Karatay et al. “Fast time-resolved electrostatic force microscopy: Achieving sub-cycle time resolution” Publisher: American Institute of Physics In _Review of Scientific Instruments_ 87.5, 2016, pp. 053702 DOI: 10.1063/1.4948396
  * [41] Ali Moeed Tirmzi, Ryan P. Dwyer, Fangyuan Jiang and John A. Marohn “Light-Dependent Impedance Spectra and Transient Photoconductivity in a Ruddlesden–Popper 2D Lead–Halide Perovskite Revealed by Electrical Scanned Probe Microscopy and Accompanying Theory” Publisher: American Chemical Society In _The Journal of Physical Chemistry C_ 124.25, 2020, pp. 13639–13648 DOI: 10.1021/acs.jpcc.0c04467
  * [42] Rajiv Giridharagopal et al. “Time-Resolved Electrical Scanning Probe Microscopy of Layered Perovskites Reveals Spatial Variations in Photoinduced Ionic and Electronic Carrier Motion” Publisher: American Chemical Society In _ACS Nano_ 13.3, 2019, pp. 2812–2821 DOI: 10.1021/acsnano.8b08390


* Qu et al. (2018) Zhaowei Qu, Xiaomin Song, Shuqiang Zheng, Xiaoru Wang, Xiaohui Song, and Zuquan Li. 2018\.  Improved Bayes Method Based on TF-IDF Feature and Grade Factor Feature for Chinese Information Classification.  _2018 IEEE International Conference on Big Data and Smart Computing (BigComp)_ (2018), 677–680.
  * Quinlan (1987) J.R. Quinlan. 1987\.  Simplifying decision trees.  _International Journal of Man-Machine Studies_ 27, 3 (1987), 221 – 234.  https://doi.org/10.1016/S0020-7373(87)80053-6
  * Quinlan (1986) J. R. Quinlan. 1986\.  Induction of Decision Trees.  _Mach. Learn._ 1, 1 (March 1986), 81–106.  https://doi.org/10.1023/A:1022643204877
  * Radford et al. (2018) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018\.  Language Models are Unsupervised Multitask Learners.  (2018).  https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_ (2019).
  * Rehman et al. (2019) Arshia Rehman, Saeeda Naz, Usman Naseem, Imran Razzak, and Ibrahim A Hameed. 2019.  Deep AutoEncoder-Decoder Framework for Semantic Segmentation of Brain Tumor.  _Aust. J. Intell. Inf. Process. Syst._ 15, 3 (2019), 53–60.
  * Ren et al. (2016) Yafeng Ren, Yue Zhang, Meishan Zhang, and Donghong Ji. 2016\.  Context-sensitive Twitter Sentiment Classification Using Neural Network. In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_ (Phoenix, Arizona) _(AAAI’16)_. AAAI Press, 215–221.  http://dl.acm.org/citation.cfm?id=3015812.3015844
  * Reuter et al. (2016) Jack Reuter, Jhonata Pereira-Martins, and Jugal Kalita. 2016\.  Segmenting Twitter Hashtags.  _International Journal on Natural Language Computing_ 5 (08 2016), 23–36.  https://doi.org/10.5121/ijnlc.2016.5402
  * Rezaeinia et al. (2017a) Seyed Mahdi Rezaeinia, Ali Ghodsi, and Rouhollah Rahmani. 2017a.  Improving the Accuracy of Pre-trained Word Embeddings for Sentiment Analysis.  _CoRR_ abs/1711.08609 (2017).  arXiv:1711.08609 http://arxiv.org/abs/1711.08609
  * Rezaeinia et al. (2017b) Seyed Mahdi Rezaeinia, Ali Ghodsi, and Rouhollah Rahmani. 2017b.  Improving the Accuracy of Pre-trained Word Embeddings for Sentiment Analysis.  _CoRR_ abs/1711.08609 (2017).  arXiv:1711.08609 http://arxiv.org/abs/1711.08609
  * Saif et al. (2013) Hassan Saif, Marta Fernandez Andres, Yulan He, and Harith Alani. 2013.  Evaluation Datasets for Twitter Sentiment Analysis: A survey and a new dataset, the STS-Gold. In _ESSEM@AI*IA_.
  * Saloot et al. (2015) Mohammad Arshi Saloot, Norisma Idris, Nor Liyana Mohd Shuib, Ram Gopal Raj, and AiTi Aw. 2015\.  Toward Tweets Normalization Using Maximum Entropy. In _NUT@IJCNLP_.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_ (2019).
  * Scherer et al. (2010) Dominik Scherer, Andreas Müller, and Sven Behnke. 2010\.  Evaluation of Pooling Operations in Convolutional Architectures for Object Recognition. In _Artificial Neural Networks – ICANN 2010_ , Konstantinos Diamantaras, Wlodek Duch, and Lazaros S. Iliadis (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 92–101.
  * Seungil et al. (2017) Seungil, David Ding, Kevin Canini, Jan Pfeifer, and Maya Gupta. 2017.  Deep Lattice Networks and Partial Monotonic Functions.  _arXiv e-prints_ , Article arXiv:1709.06680 (Sep 2017), arXiv:1709.06680 pages.  arXiv:1709.06680 [stat.ML]
  * Severyn and Moschitti (2015) Aliaksei Severyn and Alessandro Moschitti. 2015.  Twitter Sentiment Analysis with Deep Convolutional Neural Networks. In _Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Santiago, Chile) _(SIGIR ’15)_. ACM, New York, NY, USA, 959–962.  https://doi.org/10.1145/2766462.2767830
  * Shoeybi et al. (2019) Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019\.  Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.  arXiv:1909.08053 [cs.CL]
  * Singh and Kumari (2016) Tajinder Singh and Madhu Kumari. 2016.  Role of Text Pre-processing in Twitter Sentiment Analysis.
  * Socher et al. (2013) R Socher, A Perelygin, J.Y. Wu, J Chuang, C.D. Manning, A.Y. Ng, and C Potts. 2013\.  Recursive deep models for semantic compositionality over a sentiment treebank.  _EMNLP_ 1631 (01 2013), 1631–1642.
  * Soheily-Khah et al. (2017) Saeid Soheily-Khah, Pierre-FranÃ§ois Marteau, and Nicolas Béchet. 2017.  Intrusion detection in network systems through hybrid supervised and unsupervised mining process- a detailed case study on the ISCX benchmark dataset -.
  * Song et al. (2019) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019.  Mass: Masked sequence to sequence pre-training for language generation.  _arXiv preprint arXiv:1905.02450_ (2019).
  * Sparck Jones (1988) Karen Sparck Jones. 1988\.  Document Retrieval Systems.  Taylor Graham Publishing, London, UK, UK, Chapter A Statistical Interpretation of Term Specificity and Its Application in Retrieval, 132–142.  http://dl.acm.org/citation.cfm?id=106765.106782
  * Speer et al. (2016) Robyn Speer, Joshua Chin, and Catherine Havasi. 2016.  ConceptNet 5.5: An Open Multilingual Graph of General Knowledge.  _CoRR_ abs/1612.03975 (2016).  arXiv:1612.03975 http://arxiv.org/abs/1612.03975
  * Sun et al. (2019) Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019\.  Ernie: Enhanced representation through knowledge integration.  _arXiv preprint arXiv:1904.09223_ (2019).
  * Sun et al. (2020) Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020.  ERNIE 2.0: A Continual Pre-Training Framework for Language Understanding.. In _AAAI_. 8968–8975.
  * Sutskever et al. (2011) Ilya Sutskever, James Martens, and Geoffrey Hinton. 2011\.  Generating Text with Recurrent Neural Networks. In _Proceedings of the 28th International Conference on International Conference on Machine Learning_ (Bellevue, Washington, USA) _(ICML’11)_. Omnipress, USA, 1017–1024.  http://dl.acm.org/citation.cfm?id=3104482.3104610
  * Suttles and Ide (2013) Jared Suttles and Nancy Ide. 2013.  Distant Supervision for Emotion Classification with Discrete Binary Values. In _Proceedings of the 14th International Conference on Computational Linguistics and Intelligent Text Processing - Volume 2_ (Samos, Greece) _(CICLing’13)_. Springer-Verlag, Berlin, Heidelberg, 121–136.  https://doi.org/10.1007/978-3-642-37256-8_11
  * Symeonidis et al. (2018) Symeon Symeonidis, Dimitrios Effrosynidis, and Avi Arampatzis. 2018.  A comparative evaluation of pre-processing techniques and their interactions for twitter sentiment analysis.  _Expert Systems with Applications_ 110 (2018), 298 – 310.  https://doi.org/10.1016/j.eswa.2018.06.022
  * Tang et al. (2015) Duyu Tang, Bing Qin, Furu Wei, Li Dong, Ting Liu, and Ming Zhou. 2015.  A Joint Segmentation and Classification Framework for Sentence Level Sentiment Classification.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 23, 11 (2015), 1750–61.
  * Tang et al. (2016a) Duyu Tang, Furu Wei, Bing Qin, Nan Yang, Ting Liu, and Ming Zhou. 2016a.  Sentiment Embeddings with Applications to Sentiment Analysis.  _IEEE Transactions on Knowledge and Data Engineering_ 28, 2 (2016), 496–509.
  * Tang et al. (2016b) Duyu Tang, Furu Wei, Bing Qin, Nan Yang, Ting Liu, and Ming Zhou. 2016b.  Sentiment Embeddings with Applications to Sentiment Analysis.  _IEEE Trans. on Knowl. and Data Eng._ 28, 2 (Feb. 2016), 496–509.  https://doi.org/10.1109/TKDE.2015.2489653
  * Tang et al. (2014) Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, and Bing Qin. 2014\.  Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Baltimore, Maryland, 1555–1565.  https://doi.org/10.3115/v1/P14-1146
  * Uysal and Günal (2014) Alper Kursat Uysal and Serkan Günal. 2014.  The impact of preprocessing on text classification.  _Inf. Process. Manage._ 50 (2014), 104–112.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is All You Need. In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ (Long Beach, California, USA) _(NIPS’17)_. Curran Associates Inc., USA, 6000–6010.  http://dl.acm.org/citation.cfm?id=3295222.3295349
  * Wallace (2017) Byron Wallace. 2017\.  A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification. In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Asian Federation of Natural Language Processing, Taipei, Taiwan, 253–263.
  * Wang et al. (2019) Wei Wang, Bin Bi, Ming Yan, Chen Wu, Zuyi Bao, Jiangnan Xia, Liwei Peng, and Luo Si. 2019.  StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding.  arXiv:1908.04577 [cs.CL]
  * Wang et al. (2012a) Wenbo Wang, Lu Chen, Krishnaprasad Thirunarayan, and Amit P. Sheth. 2012a.  Harnessing Twitter ”Big Data” for Automatic Emotion Identification.  _2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing_ (2012), 587–592.
  * Wang et al. (2016) Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao. 2016\.  Attention-based LSTM for Aspect-level Sentiment Classification. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Austin, Texas, 606–615.  https://doi.org/10.18653/v1/D16-1058
  * Wang et al. (2012b) Yuyang Wang, Roni Khardon, and Pavlos Protopapas. 2012b.  NONPARAMETRIC BAYESIAN ESTIMATION OF PERIODIC LIGHT CURVES.  _The Astrophysical Journal_ 756, 1 (aug 2012), 67.  https://doi.org/10.1088/0004-637x/756/1/67
  * Yamada et al. (2015) Ikuya Yamada, Hideaki Takeda, and Yoshiyasu Takefuji. 2015\.  Enhancing Named Entity Recognition in Twitter Messages Using Entity Linking. In _NUT@IJCNLP_.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019\.  XLNet: Generalized Autoregressive Pretraining for Language Understanding.  _CoRR_ abs/1906.08237 (2019).  arXiv:1906.08237 http://arxiv.org/abs/1906.08237
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books.  _CoRR_ abs/1506.06724 (2015).  arXiv:1506.06724 http://arxiv.org/abs/1506.06724