# One or Two Frequencies?
The Scattering Transform Answers ††thanks: This work is supported by the NSF
award 1633259 (BIRDVOX).

Vincent Lostanlen  Music and Audio Research Lab
New York University
New York, NY, USA
<EMAIL_ADDRESS>Alice Cohen-Hadria  IRIT
University of Toulouse — CNRS
Toulouse, France
<EMAIL_ADDRESS>Juan Pablo Bello  Music and Audio Research Lab
New York University
New York, NY, USA
<EMAIL_ADDRESS>
###### Abstract

With the aim of constructing a biologically plausible model of machine
listening, we study the representation of a multicomponent stationary signal
by a wavelet scattering network. First, we show that renormalizing second-
order nodes by their first-order parents gives a simple numerical criterion to
assess whether two neighboring components will interfere psychoacoustically.
Secondly, we run a manifold learning algorithm (Isomap) on scattering
coefficients to visualize the similarity space underlying parametric additive
synthesis. Thirdly, we generalize the “one or two components” framework to
three sine waves or more, and prove that the effective scattering depth of a
Fourier series grows in logarithmic proportion to its bandwidth.

###### Index Terms:

Audio systems, Amplitude modulation, Continuous wavelet transform, Fourier
series, Multi-layer neural network.

##  I Introduction

In the mammalian auditory system, cochlear hair cells operate like band-pass
filters whose equivalent rectangular bandwidth (ERB) grows in proportion to
their center frequency. Given two sine waves
$t\mapsto\bm{y_{1}}(t)=a_{1}\cos(f_{1}t+\varphi_{1})$ and
$t\mapsto\bm{y_{2}}(t)=a_{2}\cos(f_{2}t+\varphi_{2})$ of respective
frequencies $f_{1}>0$ and $f_{2}>0$, we perceive their mixture as a musical
chord insofar as $\bm{y_{1}}$ and $\bm{y_{2}}$ belong to disjoint critical
bands. However, if $a_{2}\ll a_{1}$ or $f_{2}\approx f_{1}$, then the tone
$\bm{y_{2}}$ is said to be _masked_ by $\bm{y_{1}}$. In lieu of two pure
tones, we hear a “beating tone”: i.e., a locally sinusoidal wave whose carrier
frequency is $\frac{1}{2}(f_{1}+f_{2})$ and whose modulation frequency is
$\frac{1}{2}|f_{1}-f_{2}|$. In humans, the resolution of beating tones
involves physiological processes beyond the cochlea, i.e., in the primary
auditory cortex.

The scattering transform ($\mathbf{S}$) is a deep convolutional operator which
alternates constant-$Q$ wavelet decompositions and the application of
pointwise complex modulus, up to some time scale $T$. Broadly speaking, its
first two layers ($\mathbf{S_{1}}$ and $\mathbf{S_{2}}$) resemble the
functioning of the cochlea and the primary auditory cortex, respectively. In
the context of audio classification, scattering transforms have been
succesfully employed to represent speech [2], environmental sounds [13], urban
sounds [20], musical instruments [10], rhythms [8], and playing techniques
[24]. Therefore, the scattering transform simultaneously enjoys a diverse
range of practical motivations, a firm rooting in wavelet theory, and a
plausible correspondence with neurophysiology.

This article discusses the response of the scattering transform operator to a
complex tone input $\bm{y}:t\mapsto\bm{y_{1}}(t)+\bm{y_{2}}(t)$, depending on
the sinusoidal parameters of $\bm{y_{1}}$ and $\bm{y_{2}}$. In this respect,
we follow a well-established methodology in nonstationary signal processing,
colloquially known as: “One or two frequencies? The X Answers”, where X is the
nonlinear operator of interest. The key idea is to identify transitional
regimes in the response of X with respect to variations in relative amplitude
($\frac{a_{2}}{a_{1}}$), relative frequency ($\frac{f_{2}}{f_{1}}$), and
relative phase ($\varphi_{2}-\varphi_{1}$). Prior publications have done so
for X being the empirical mode decomposition [19], the synchrosqueezing
transform [25], and the singular spectrum analysis operator [9]. We extend
this line of research to the case where X is the scattering transform in
dimension one.

##  II Wavelet-based recursive interferometry

Let $\bm{\psi}\in\mathbf{L}^{2}(\mathbb{R},\mathbb{C})$ a Hilbert-analytic
filter with null average, unit center frequency, and an ERB equal to $1/Q$. We
define a constant-$Q$ wavelet filterbank as the family
$\bm{\psi}_{\lambda}:t\mapsto\lambda\bm{\psi}(\lambda t)$. Each wavelet
$\bm{\psi}_{\lambda}$ has a center frequency of $\lambda$, an ERB of
$\lambda/Q$, and an effective receptive field of $(2\pi Q/\lambda)$ in the
time domain. In practice, the frequency variable $\lambda$ gets discretized
according to a geometric progression of common ratio $2^{\frac{1}{Q}}$.
Consequently, every continuous signal $\bm{y}$ that is bandlimited to
$[f_{\min},f_{\max}]$ activates a number of
$Q\log_{2}({\frac{f_{\max}}{f_{\min}}})$ wavelets $\bm{\psi_{\lambda}}$ at
most.

We define the scalogram of $\bm{y}$ as the squared complex modulus of its
constant-$Q$ transform (CQT):

$\mathbf{U_{1}}\bm{y}:(t,\lambda_{1})\longmapsto\left|\int_{-\infty}^{+\infty}\bm{y}(t^{\prime})\bm{\psi_{\lambda_{1}}}(t-t^{\prime})\;\mathrm{d}t^{\prime}\right|^{2}.$
(1)

Likewise, we define a second layer of nonlinear transformation for $\bm{y}$ as
the “scalogram of its scalogram”:

$\mathbf{U_{2}}\bm{y}:(t,\lambda_{1},\lambda_{2})\longmapsto\Big{|}\big{|}\bm{y}\ast\bm{\psi_{\lambda_{1}}}\big{|}^{2}\ast\bm{\psi_{\lambda_{2}}}\Big{|}^{2}(t),$
(2)

where the asterisk denotes a convolution product. This construct may be
iterated for every integer $m$ by “scattering” the multivariate signal
$\mathbf{U_{m}}\bm{y}$ into all wavelet subbands $\lambda_{m}<\lambda_{m-1}$:

$\mathbf{U_{m+1}}\bm{y}:(t,\lambda_{1}\ldots\lambda_{m+1})\longmapsto\\\
\Big{|}\mathbf{U_{m}}\bm{y}(t,\lambda_{1}\ldots\lambda_{m})\ast\bm{\psi}_{\lambda_{m}}\Big{|}^{2}(t,\lambda_{1}\ldots\lambda_{m}).$
(3)

Note that the original definition of the scattering transform adopts the
complex modulus ($|z|=\sqrt{z\bar{z}}$) rather its square ($|z|^{2}=z\bar{z}$)
as its activation function. This is to ensure that $\mathbf{U_{m}}$ is a non-
expansive map in terms of Lipschitz regularity. However, to simplify our
calculation and spare an intermediate stage of linearization of the square
root, we choose to employ a measure of power rather than amplitude. This idea
was initially proposed by [6] in the context of marine bioacoustics.

Every layer $m$ in this deep convolutional network composes an invariant
linear system (namely, the CQT) and a pointwise operation (the squared complex
modulus). Thus, by recurrence over the depth variable $m$, every tensor
$\mathbf{U_{m}}\bm{y}$ is equivariant to the action of delay operators. In
order to replace this equivariance property by an invariance property, we
integrate each $\mathbf{U_{m}}$ over some predefined time scale $T$, yielding
the invariant scattering transform:

$\displaystyle\mathbf{S_{m}}\bm{y}:(t,p)\longmapsto\int_{-\infty}^{+\infty}\mathbf{U_{m}}(t^{\prime},p)\bm{\phi}_{T}(t-t^{\prime})\;\mathrm{d}t^{\prime},$
(4)

where the $m$-tuple $p=(\lambda_{1}\ldots\lambda_{m})$ is a scattering path
and the signal $\bm{\phi}_{T}$ is a real-valued low-pass filter of time scale
$T$.

##  III Auditory masking in a scattering network

Figure 1:  Superimposed heatmaps of second-order masking coefficients
$\mathbf{\widetilde{S}_{2}}\bm{y}$ after a scattering transform of two sine
waves $\bm{y_{1}}$ and $\bm{y_{2}}$, measured around the frequency $f_{1}$, as
a function of relative amplitude $\frac{a_{2}}{a_{1}}$ and relative frequency
difference $\frac{|f_{2}-f_{1}|}{f_{1}}$. The color of each blot denotes the
resolution $\lambda_{2}$ at the second layer. Wavelets have an asymmetric
profile (Gammatone wavelets) and a quality factor $Q=4$. The second layer
covers an interval of nine octaves below $f_{1}$. For the sake of clarity, we
only display one interference pattern per octave.

Given $n\in\\{1,2\\}$, the convolution between every sine wave $\bm{y_{n}}$
and every wavelet $\bm{\psi_{\lambda_{1}}}$ writes as a multiplication in the
Fourier domain. Because $\bm{\psi_{\lambda_{1}}}$ is Hilbert-analytic, only
the analytic part
$\bm{y_{n}^{\mathrm{a}}}=\bm{y_{n}}+\mathrm{i}\mathcal{H}\\{\mathbf{y_{n}}\\}=a_{n}\exp(\mathrm{i}(f_{n}t+\varphi_{n}))$
of the real signal $\bm{y_{n}}$ is preserved in the CQT:

$\left(\bm{y_{n}}\ast\bm{\psi_{\lambda_{1}}}\right)(t)=\dfrac{1}{2}\bm{\widehat{\psi}_{\lambda_{1}}}(f_{n})\bm{y_{n}^{\mathrm{a}}}(t).$
(5)

By linearity of the CQT, we expand the interference between $\bm{y_{1}}$ and
$\bm{y_{2}}$ by heterodyning:

$\displaystyle\Big{|}(\bm{y_{1}}+\bm{y_{2}})\ast\bm{\psi_{\lambda_{1}}}\Big{|}^{2}(t)=\frac{1}{2}\Big{|}\bm{\widehat{\psi}}\big{(}\frac{f_{1}}{\lambda_{1}}\big{)}\Big{|}^{2}a_{1}^{2}+\frac{1}{2}\Big{|}\bm{\widehat{\psi}}\big{(}\frac{f_{2}}{\lambda_{1}}\big{)}\Big{|}^{2}a_{2}^{2}$
$\displaystyle+\mathfrak{R}\Bigg{(}\\!\bm{\widehat{\psi}}\Big{(}\frac{f_{1}}{\lambda_{1}}\Big{)}\bm{\widehat{\psi}^{\ast}}\Big{(}\frac{f_{2}}{\lambda_{1}}\Big{)}\\!\Bigg{)}a_{1}a_{2}\cos\big{(}(f_{2}-f_{1})t+(\varphi_{2}-\varphi_{1})\big{)}.$
(6)

Because the wavelet $\bm{\psi}$ has a null average, the two constant terms in
the equation above are absorbed by the first layer of the scattering network,
and disappear at deeper layers. However, the cross term, proportional to
$a_{1}a_{2}$, is a “difference tone” of fundamental frequency
$\Delta\\!f=|f_{2}-f_{1}|$.

The authors of a previous publication [3] have remarked that this difference
tone elicits a peak in second-order scattering coefficients for the path
$p=(\lambda_{1},\lambda_{2})=(f_{1},|f_{2}-f_{1}|)$. In the following, we
generalize their study to include the effect of the relative amplitude
$\frac{a_{2}}{a_{1}}$, the wavelet shape $\bm{\psi}$, the quality factor $Q$,
and the time scale of local stationarity $T$.

Equation 6 illustrates how the scalogram operator $\mathbf{U_{1}}$ converts a
complex tone (two frequencies $f_{1}$ and $f_{2}$) into a simple tone (one
frequency $|f_{2}-f_{1}|$). For this simple tone to carry a nonnegligible
amplitude in $\mathbf{U_{2}}$, three conditions must be satisfied. First, the
rectangular term $a_{1}a_{2}$ must be nonnegligible in comparison to the
square terms $a_{1}^{2}$ and $a_{2}^{2}$. Secondly, there must exist a wavelet
$\bm{\psi}_{\lambda_{1}}$ whose spectrum encompasses both frequencies $f_{1}$
and $f_{2}$. Said otherwise, $\lambda_{1}$ must satisfy the inequalities
$|\frac{f_{n}}{\lambda_{1}}-1|\ll\frac{1}{Q}$, both for $f_{n}=f_{1}$ and for
$f_{n}=f_{2}$. Thirdly, the frequency difference $|f_{2}-f_{1}|$ must belong
to the passband of some second-order wavelet $\bm{\psi_{\lambda_{2}}}$. Yet,
in practice, to guarantee the temporal localization of scattering coefficients
and restrict the filterbank to a finite number of octaves, the scaling factor
of every $\bm{\psi_{\lambda_{m}}}$ is upper-bounded by the temporal constant
$T$. Therefore, the period $\frac{2\pi}{|f_{2}-f_{1}|}$ of the difference tone
should be under the pseudo-period of the wavelet with support $T$; i.e., a
pseudo-period of $QT$. Hence the third condition: $|f_{2}-f_{1}|\ll\frac{2\pi
Q}{T}$.

One simple way of quantifying the amount of mutual interference between
signals $\bm{y_{1}}$ and $\bm{y_{2}}$ is to renormalize second-order
coefficients by their first-order “parent” coefficients:

$\mathbf{\widetilde{S}_{2}}\bm{y}(t,\lambda_{1},\lambda_{2})=\dfrac{\mathbf{S_{2}}\bm{y}(t,\lambda_{1},\lambda_{2})}{\mathbf{S_{1}}\bm{y}(t,\lambda_{1})}$
(7)

This operation, initially proposed by [2], is conceptually analogous to
classical methods in adaptive gain control, notably per-channel energy
normalization (PCEN) [14].

In accordance with the “one or two frequencies” methodology, Figure 1
illustrates the value of this ratio of energies in the subband
$\lambda_{1}=f_{1}$, for different values of relative amplitude
$\frac{a_{2}}{a_{1}}$ and relative frequency difference
$\frac{|f_{2}-f_{1}|}{f_{1}}$. We fixed $f_{2}<f_{1}$ without loss of
generality. As expected, we observe that, for $a_{2}\approx a_{1}$ and a
relative frequency difference between $\frac{Q}{f_{1}T}$ and $\frac{1}{Q}$,
second-layer wavelets $\bm{\psi_{\lambda_{2}}}$ resonate with the difference
tone as a result of the interference between signals $\bm{y_{1}}$ and
$\bm{y_{2}}$.

##  IV Application to manifold learning

To demonstrate the ability of the scattering transform to characterize
auditory masking, we build a dataset of complex tones according to the
following additive synthesis model:

$\bm{y}_{\alpha,r}(t)=\sum_{n=1}^{N}\dfrac{1+(-1)^{n}r}{n^{\alpha}}\cos(nf_{1}t)\bm{\phi}_{T}(t),$
(8)

where $\bm{\phi}_{T}$ is a Hann window of duration $T$. This additive
synthesis model depends upon two parameters: the Fourier decay $\alpha$ and
the relative odd-to-even amplitude difference $r$. Figure 2 displays the CQT
log-magnitude spectrum of $\bm{y}_{\alpha,r}$ for different values of $\alpha$
and $r$. In practice, we set $T$ to $1024$ samples, $N$ to $32$ harmonics, and
$f_{1}$ between $12$ and $24$ cycles.

Figure 2:  Log-magnitudes of synthetic musical tones as a function of wavelet
log-frequency ($\log\lambda$). Ticks of the vertical (resp. horizontal) axis
denote relative amplitude (resp. frequency) intervals of 10 dB (resp. one
octave). Parameters $\alpha$ and $r$ denote the Fourier decay exponent and the
relative odd-to-even amplitude difference $r$ respectively. See Equation 8 for
details.

Our synthetic dataset comprises $2500$ audio signals in total, corresponding
to $50$ values of $\alpha$ between $0$ and $2$ and $50$ values of $r$ between
$0$ and $1$, while $f_{1}$ is an integer chosen uniformly at random between
$12$ and $24$. We extract the scattering transform of each signal
$\bm{y}_{\alpha,r}$ up to order $M=2$, with $Q=1$ and $J=8$, by means of the
Kymatio Python package [4]. Concatenating $QJ$ first-order coefficients with
$\frac{1}{2}Q^{2}J(J-1)$ second-order coefficients yields a representation in
dimension $37$.

For visualization purposes, we bring the $37$-dimensional space of scattering
coefficients to the dimension three by means of the Isomap algorithm for
unsupervised manifold learning [22]. The appeal behind Isomap is that pairwise
Euclidean distances in the 3-D point cloud approximate the corresponding
geodesic distances over the $K$-nearest neighbor graph associated to the
dataset. Throughout this paper, we set the number of neighbors to $K=100$ and
measure neighboring relationships by comparing high-dimensional $\ell^{2}$
distances. Crucially, in the case of the scattering transform, these
$\ell^{2}$ distances are provably stable (i.e., Lipschitz-continuous) to the
action of diffeomorphisms [16, Theorem 2.12].

Scattering transform embedding

Audiovisual correspondence (Open-L3) embedding

Mel-frequency cepstral coefficient (MFCC) embedding

Color: $f_{1}\qquad\qquad$ Color: $\alpha\quad\qquad\qquad$ Color: $r\qquad$

Figure 3: Isomap embedding of synthetic musical notes, as described by their
scattering transform coefficients (top); their Open-L3 coefficients (center);
and their mel-frequency cepstral coefficients (MFCC, bottom). The color of a
dot, ranging from red to blue via white, denotes the fundamental frequency
$f_{1}$ (left), the Fourier decay exponent $\alpha$ (center), and the relative
odd-to-even amplitude difference $r$ (right) respectively. Note that all
methods are unsupervised: triplets ($f_{1}$, $\alpha$, $r$) are not directly
supplied to the models, but only serve for color grading post hoc. See Section
IV for details.

Figure 3 (top) illustrates our findings. We observe that, after scattering
transform and Isomap dimensionality reduction, the dataset appears as a 3-D
Cartesian mesh whose principal components align with $f_{1}$, $\alpha$, and
$r$ respectively. This result demonstrates that the scattering transform is
capable of disentangling and linearizing multiple factors of variability in
the spectral envelope of periodic signals, even if those factors are not
directly amenable to diffeomorphisms.

As a point of comparison, Figure 3 presents the outcome of Isomap on
alternative feature representations: Open-L3 embedding (center) and mel-
frequency cepstral coefficients (MFCC, bottom). The former results from
training a deep convolutional network (convnet) on a self-supervised task of
audiovisual correspondence, and yields $6177$ coefficients [7]. The latter
resuts from a log-mel-spectrogram representation, followed by a discrete
cosine transform (DCT) over the mel-frequency axis, and yields $12$
coefficients. We compute MFCC with librosa v0.7 [18] default parameters.

We observe that Open-L3 embeddings correctly disentangles boundary conditions
($r$) from fundamental frequency ($f_{1}$), but fails to disentangle Fourier
decay ($\alpha$) from $f_{1}$. Instead, correlations between $r$ and $f_{1}$
are positive for low-pitched sounds ($12$ to $16$ cycles) and negative for
high-pitched sounds ($16$ to $24$ cycles). Although this failure deserves a
more formal inquiry, we hypothesize that this it stems from the small
convolutional receptive field of the $L^{3}$-Net: $24$ mel subbands, i.e.,
roughly half an octave around $1$ kHz.

Figure 4: Energy decay as a function of wavelet scattering depth $m$, for
mixtures of $N$ components with equal amplitudes, equal phases, and evenly
spaced frequencies. The color of each line plot denotes the integer part of
$\log_{2}N$. In this experiment, wavelets have a sine cardinal profile
(Shannon wavelets) and a quality factor equal to $Q=1$. Each filterbank covers
seven octaves.

Moreover, in the case of MFCC, we find that the variability in fundamental
frequency ($f_{1}$) dominates the variability in spectral shape parameters
($\alpha$ and $r$), thus yielding a rectilinear embedding (top). This
observation is in line with a previous publication [11], which showed
statistically that MFCCs are overly sensitive to frequency transposition in
complex tones.

From this qualitative benchmark, it appears that the scattering transform is a
more interpretable representation of periodic signals than Open-L3, while
incurring a smaller computational cost. However, in the presence of aperiodic
signals such as environmental sounds, Open-L3 outperforms the scattering
transform in terms of classification accuracy with linear support vector
machines [5]. To remain competitive, the scattering transform must not only
capture heterodyne interference, but also joint spectrotemporal modulations
[1]. In this context, future work will strive to combine insights from
multiresolution analysis and deep self-supervised learning.

##  V Beyond pairwise interference:
full-depth scattering networks

In speech and music processing, pitched sounds are rarely approximable as a
mixture of merely two components. More often than not, they contain ten
components or more, and span across multiple octaves in the Fourier domain.
Thus, computing the masking coefficient at the second layer only provides a
crude description of the timbral content within each critical band. Indeed,
$\mathbf{S_{2}}$ encodes pairwise interference between sinusoidal components
but fails to characterize more intricate structures in the spectral envelope
of $\bm{y}$.

To address this issue, we propose to study the scattering transform beyond
order two, thus encompassing heterodyne structures of greater multiplicity.
For the sake of mathematical tractability, we consider the following mother
wavelet, hereafter called “complex Shannon wavelet” after [15, Section 7.2.2]:

$\bm{\psi}:t\longmapsto\dfrac{\exp(2\mathrm{i}t)-\exp(\mathrm{i}t)}{2\pi\mathrm{i}t}$
(9)

The definition of a scattering transform with complex Shannon wavelets
requires to resort to the theory of tempered distributions. We refer to [21]
for further mathematical details.

The following theorem, proven in the Appendix, describes the response of a
deep scattering network in the important particular case of a periodic signal
with finite bandwidth.

######  Theorem V.1.

Let $\bm{y}\in\mathcal{C}^{\infty}(\mathbb{R})$ a periodic signal of
fundamental frequency $f_{1}$. Let $\bm{\psi}$ the complex Shannon wavelet as
in Equation 9 and $\mathbf{U_{1}}$ its associated scalogram operator as in
Equation 1. If $\bm{y}$ has a finite bandwidth of $M$ octaves, then its
scattering coefficients $\mathbf{U_{m}}\bm{y}$ are zero for any $m>M$.

This result is in agreement with the theorem of exponential decay of
scattering coefficients [23]. Note, however, that [23] expresses an upper
bound on the energy at fixed depth for integrable signals, while we express an
upper bound on the depth at fixed bandwidth for periodic signals.

We apply the theorem above to the case of a signal containing $N$ components
of equal amplitudes, equal phases, and evenly spaced frequencies:
$\bm{y}:t\mapsto\sum_{n=1}^{N}a_{1}\cos(nf_{1}t+\varphi_{1})$. Figure 4
illustrates the decay of scatterered energy as a function of depth. The
conceptual analogy between depth and scale was originally proposed by [17] in
a theoretical effort to clarify the role of hierarchical symmetries in
convnets.

Although our findings support this analogy, we note that computing a
scattering transform with $M=\log_{2}T$ layers is often impractical. However,
if the Fourier series in $\bm{y}$ satisfies a self-similarity assumption, it
is possible to match the representational capacity of a full-depth scattering
network while keeping the depth to $M=2$. Indeed, spiral scattering performs
wavelet convolutions over time, over log-frequency, and across octaves,
thereby capturing the spectrotemporal periodicity of Shepard tones and
Shepard-Risset glissandos [12]. Further research is needed to integrate
broadband demodulation into deep convolutional architectures for machine
listening.

##  VI Conclusion

In this article, we have studied the role of every layer in a scattering
network by means of a well-established methodology, colloquially known as “one
or two components” [19]. We have come up with a numerical criterion of
psychoacoustic masking; demonstrated that the scattering transform
disentangles multiple factors of variability in the spectral envelope; and
proven that the effective scattered depth of Fourier series is bounded by the
logarithm of its bandwidth, thus emphasizing the importance of capturing
geometric regularity across temporal scales.

## References

  * [1] Joakim Andén, Vincent Lostanlen and Stéphane Mallat  “Joint time–frequency scattering”  In _IEEE Transactions on Signal Processing_ 67.14 IEEE, 2019, pp. 3704–3718
  * [2] Joakim Andén and Stéphane Mallat  “Deep scattering spectrum”  In _IEEE Trans. Signal Process._ 62.16 IEEE, 2014, pp. 4114–4128
  * [3] Joakim Andén and Stéphane Mallat  “Scattering representation of modulated sounds”  In _Proc. DAFx_ , 2012
  * [4] Mathieu Andreux et al.  “Kymatio: Scattering transforms in Python”  In _JMLR_ 21.60, 2020, pp. 1–6
  * [5] Relja Arandjelovic and Andrew Zisserman  “Look, listen and learn”  In _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 609–617
  * [6] Randall Balestriero and Hervé Glotin  “Linear time complexity deep Fourier scattering network and extension to nonlinear invariants”  In _arXiv_ 1707.05841, 2017
  * [7] Jason Cramer, Ho-Hsiang Wu, Justin Salamon and Juan Pablo Bello  “Look, listen, and learn more: Design choices for deep audio embeddings”  In _Proc. ICASSP_ , 2019, pp. 3852–3856  IEEE
  * [8] Daniel Haider and Peter Balazs  “Extraction of Rhythmical Features with the Gabor Scattering Transform”  In _Proc. CMMR_ , 2019
  * [9] Jinane Harmouche, Dominique Fourer, Pierre Auger and Patrick Flandrin  “Une ou deux composantes: la réponse de l’analyse spectrale singulière”  In _Actes du colloque GRETSI_ , 2015
  * [10] Vincent Lostanlen, Joakim Andén and Mathieu Lagrange  “Extended playing techniques: The next milestone in musical instrument recognition”  In _Proc. DLfM_ , 2018
  * [11] Vincent Lostanlen and Carmine-Emanuele Cella  “Deep convolutional networks on the pitch spiral for musical instrument recognition”  In _Proc. ISMIR_ , 2016
  * [12] Vincent Lostanlen and Stéphane Mallat  “Wavelet scattering on the pitch spiral”  In _Proc. DAFX_ , 2016
  * [13] Vincent Lostanlen, Grégoire Lafay, Joakim Andén and Mathieu Lagrange  “Relevance-based quantization of scattering features for unsupervised mining of environmental audio”  In _EURASIP J. Audio Speech Mus. Process._ 2018.1 Springer, 2018, pp. 15
  * [14] V. Lostanlen et al.  “Per-Channel Energy Normalization: Why and How”  In _IEEE Signal Proc. Let._ 26.1, 2019, pp. 39–43
  * [15] Stéphane Mallat  “A wavelet tour of signal processing: The sparse way”  Associated Press, 2008
  * [16] Stéphane Mallat  “Group invariant scattering”  In _Comm. Pure Appl. Math._ 65.10 Wiley Online Library, 2012, pp. 1331–1398
  * [17] Stéphane Mallat  “Understanding deep convolutional networks”  In _Phil. Trans. R. Soc._ 374.2065 The Royal Society Publishing, 2016, pp. 20150203
  * [18] Brian McFee et al.  “librosa: 0.7.2”, 2020  DOI: 10.5281/zenodo.3606573
  * [19] Gabriel Rilling and Patrick Flandrin  “One or two frequencies? The empirical mode decomposition answers”  In _IEEE Trans. Signal Process._ 56.1 IEEE, 2008, pp. 85–95
  * [20] Justin Salamon and Juan Pablo Bello  “Feature learning with deep scattering for urban sound analysis”  In _Proc. EUSIPCO_ , 2015, pp. 724–728  IEEE
  * [21] Robert S Strichartz  “A guide to distribution theory and Fourier transforms”  World Scientific Publishing Company, 2003
  * [22] Joshua B Tenenbaum, Vin De Silva and John C Langford  “A global geometric framework for nonlinear dimensionality reduction”  In _science_ 290.5500 American Association for the Advancement of Science, 2000, pp. 2319–2323
  * [23] Irène Waldspurger  “Exponential decay of scattering coefficients”  In _Proc. SampTA_ , 2017, pp. 143–146  IEEE
  * [24] Changhong Wang, Emmanouil Benetos, Vincent Lostanlen and Elaine Chew  “Adaptive time–frequency scattering for periodic modulation recognition in music signals”  In _Proc. ISMIR_ , 2019
  * [25] Hau-Tieng Wu, Patrick Flandrin and Ingrid Daubechies  “One or two frequencies? The synchrosqueezing answers”  In _Adv. Adapt. Data Anal._ 3.01–02 World Scientific, 2011, pp. 29–39

## Appendix: proof of Theorem V.1

###### Proof.

We reason by induction over the depth variable $M$. The base case ($M=1$)
leads to $\mathbf{U_{1}}\bm{y}(t,\lambda)=1$ if $\lambda<f_{1}\leq 2\lambda$
and zero otherwise. Because $\bm{\psi}$ has one vanishing moment, it follows
that $\mathbf{U_{2}}\bm{y}$ is zero, and likewise at deeper layers. To prove
the induction step at depth $M$, to decompose $\bm{y}$ into a low-pass
approximation $(\bm{y}\ast\bm{g_{M}})$ spanning the subband $[0;2^{M}f_{1}[$
and a high-pass detail $(\bm{y}\ast\bm{h_{M}})$ spanning the subband
$[2^{M}f_{1};2^{(M+1)}f_{1}[$. Denoting by $c_{n}$ the complex-valued Fourier
coefficients of $\bm{y}$, we have at every time $t\in\mathbb{R}$:

$\displaystyle\bm{y}(t)$ $\displaystyle=(\bm{y}\ast\bm{g_{M}})(t)$
$\displaystyle+$ $\displaystyle(\bm{y}\ast\bm{h_{M}})(t)$
$\displaystyle=\sum_{|n|\leq 2^{M}}c_{n}\exp(\mathrm{i}nf_{1}t)$
$\displaystyle+$ $\displaystyle\sum_{|n|>2^{M}}c_{n}\exp(\mathrm{i}nf_{1}t)$
(10)

On one hand, the coarse term $(\bm{y}\ast\bm{g_{M}})$ has a bandwidth of $M$
octaves. Therefore, by the induction hypothesis, we have
$\mathbf{U_{m}}(\bm{y}\ast\bm{g_{M}})=0$ for $m>M$, and _a fortiori_ for
$m>(M+1)$. On the other hand, we consider the complex Shannon scalogram of
$(\bm{y}\ast\bm{h_{M}})$ in some subband $\lambda>0$:

$\big{|}\bm{y}\ast\bm{h_{M}}\ast\bm{\psi}_{\lambda}\big{|}^{2}(t)\leq\big{|}\bm{y}\ast\bm{h_{M}}\ast\bm{\psi}_{2^{M}}\big{|}^{2}(t)\hfill\\\
\hfill=\sum_{n=(1+2^{M})}^{2^{M+1}}\sum_{k=(1+2^{M})}^{2^{M+1}}c_{n}c_{k}^{\ast}\exp\big{(}\mathrm{i}(n-k)f_{1}t\big{)}$
(11)

In the double sum above, all integer differences of the form $(n-k)$ range
between $-(2^{M}-1)$ and $(2^{M}-1)$. Thus,
$\big{|}\bm{y}\ast\bm{h_{M}}\ast\bm{\psi}_{2^{M}}\big{|}^{2}$ is a periodic
signal of fundamental frequency $f_{1}$ spanning $M$ octaves. Furthermore,
because $\bm{h_{M}}=\bm{\psi}_{2^{M}}$,
$\big{|}\bm{y}\ast\bm{h_{M}}\ast\bm{\psi}_{\lambda}\big{|}^{2}$ has a smaller
bandwidth than $\big{|}\bm{y}\ast\bm{h_{M}}\ast\bm{\psi}_{2^{M}}\big{|}^{2}$;
i.e., $M$ octaves or less. By the induction hypothesis, we have:

$\forall\lambda,\;\mathbf{U_{M+1}}\big{(}|\bm{y}\ast\bm{h_{M}}\ast\bm{\psi}_{\lambda}|^{2}\big{)}(\lambda_{1},\ldots,\lambda_{m+1})=0.$
(12)

In the equation above, we recognize the scattering path
$p=(\lambda,\lambda_{1},\ldots,\lambda_{M+1})$ of $\mathbf{U_{M+2}}$. Finally,
because the scattering transform is a nonexpansive operator [16, Prop. 2.5],
we have the inequality:

$\big{\|}\mathbf{U_{M+2}}(\bm{y})\big{\|}\leq\big{\|}\mathbf{U_{M+2}}(\bm{y}\ast\bm{g_{M}})\big{\|}+\big{\|}\mathbf{U_{M+2}}(\bm{y}\ast\bm{h_{M}})\big{\|}=0,$
(13)

which implies $\mathbf{U_{M+2}}\bm{y}=0$, and likewise at deeper layers. We
conclude by induction that the theorem holds for any $M$. ∎


Computer Science Department, Virginia Commonwealth University, Richmond, VA,
USA

# Extracting Adverse Drug Events from Clinical Notes

Darshini Mahendran and Bridget T. McInnes  Ph.D

Abstract

Adverse drug events (ADEs) are unexpected incidents caused by the
administration of a drug or medication. To identify and extract these events,
we require information about not just the drug itself but attributes
describing the drug (e.g., strength, dosage), the reason why the drug was
initially prescribed, and any adverse reaction to the drug. This paper
explores the relationship between a drug and its associated attributes using
relation extraction techniques. We explore three approaches: a rule-based
approach, a deep learning-based approach, and a contextualized language model-
based approach. We evaluate our system on the n2c2-2018 ADE extraction
dataset. Our experimental results demonstrate that the contextualized language
model-based approach outperformed other models overall and obtain the state-
of-the-art performance in ADE extraction with a Precision of 0.93, Recall of
0.96, and an $F_{1}$ score of 0.94; however, for certain relation types, the
rule-based approach obtained a higher Precision and Recall than either
learning approach.

##  1 Introduction

Adverse drug events (ADE) are unexpected incidents or accidents related to the
administration of a drug and its attributes. It includes overdoses, allergic
reactions, drug interactions, and medication errors. ADEs account for an
estimated 30% of all hospital adverse events [1], and the cost of managing
ADEs can be high because the clinical diagnosis of an ADE more often requires
additional laboratory tests or procedures to investigate the cause of a
patient’s symptoms. An ADE may cause a prolonged length of stay in the
hospital and increase the economic burden [2]. Some conditions can be caused
by undiscovered ADEs, which can increase the costs and risks further and
impact the patient economically and mentally. This can be prevented if we
identify the potential reason and the information regarding the drug on time.
Extracting relations from scientific publications and clinical narratives has
always been challenging due to the complexity of language and domain-specific
knowledge involved [3]. Processing the information from these clinical
narratives, which records patient medication history, known allergies,
reactions, and adverse events of the patient, allows a more thorough
assessment of potential ADEs before they happen [1].

If we can extract all possible interactions of a drug and warn the patient
when prescribing the drug, this would reduce the risks of an ADE taking place
[1]. ADEs are a world-wide health-related concern, and therefore, a
considerable amount of research and effort is dedicated to identifying
possible ADEs in different instances [4]. Information on drugs, its
attributes, and associated ADEs are stored in many databases; however, we
require information about not just the drug itself but attributes describing
the drug (e.g., strength, dosage), the reason why the drug was initially
prescribed (e.g., reason), and the relation between the drug and its
attributes.

However, manual extraction of ADEs is almost impossible [5] given the amount
of data gathered every year, therefore, there is an urgent need for automated
systems for ADE extraction. These data are more often unstructured and Natural
Language Processing (NLP) techniques are utilized for this significant task to
extract ADEs from the unstructured text. Relation Extraction (RE) is a sub-
field of NLP whose goal is to detect and classify relations between entities
in a text. In this work, we explore three RE approaches for ADE extraction: a
rule-based approach utilizing co-location information, a deep learning-based
approach utilizing Convolutional Neural Networks (CNNs), and a contextualized
language model-based approach utilizing Bidirectional Encoder Representations
from Transformers (BERT). We evaluate our system on the n2c2-2018 ADE
extraction dataset. Our experimental results demonstrate that the
contextualized language model-based approach outperformed other models overall
and obtained state-of-the-art performance in ADE extraction with a Precision
of 0.93, Recall of 0.96, and an $F_{1}$ score of 0.94; however, the rule-based
system obtained a higher Precision and Recall for certain relation types.

The remainder of this paper is structured as follows. First, we discuss the
previous works done in this area of research. Second, we describe the dataset
we use to evaluate our system. Third, we describe our three approaches.
Fourth, we present and analyze the results. Fifth, we conduct a comparison
between our approaches and previous work. Finally, we present the conclusions
we derive from this work and what we plan to do in the future.

##  2 Related Work

Adverse drug event (ADE) extraction is gaining attention recently among the
clinical NLP community. Many approaches have been explored and can be divided
into four paradigms: 1) rule-based, 2) machine learning-based 3) deep
learning-based, and 4) contextualized language model-based approaches.

Rule-based approaches. These systems use specified rules and patterns to
extract the information from texts. Li, et al. [6], used a rule-based method
to link drug names with their attributes. They used a string-based regular
expression matching to match the drug names to a prescription list and then
used the co-location information and RxNorm dictionary to determine whether
they matched.111https://www.nlm.nih.gov/research/umls/rxnorm/index.html.

Machine learning-based approaches. Traditional supervised machine learning
systems utilize large amounts of the annotated corpus for training. Previous
works have utilized learning algorithms such as Support Vector Machines (SVMs)
[7] and Random Forests (RFs) [1]. Miller, et al. [7] used an SVM-based system
and a neural system obtain for RE based on previous work on extracting
temporal narrative container relations from sentences [8]. Yang, et al. [9]
first applied heuristic rules to generate candidate pairs and then applied ML
models to classify the relations. They divided the relations into different
groups according to their cross-distance - defined as the number of sentence
boundaries between the two entities and developed multiple classifiers to
classify relations according to their cross-distance.

Deep learning-based approaches. These systems utilize multi-layer neural
networks typically with featureless embedding representations such as word
embeddings [10]. Recent work has explored using variations of Recurrent Neural
Network (RNN) architectures. Xu, et al. [11] proposed a cascaded sequence
labeling approach to recognize the entities and the relations simultaneously.
Sorokin, et al. [12] proposed using a Long Short Term Memory (LSTM)-based
encoder to jointly learn representations for all relations in a single
sentence. Henry, et al. [1] summarizes the work of participants in the
n2c2-2018 challenge who proposed an attention-based piecewise bidirectional
(bi-) LSTM with standard features and unique candidate pair generation.
Christopoulou, et al. [13] developed separate models for intra- and inter-
sentence relation extraction and combined them using an ensemble method. The
intra-sentence models use biLSTMs with attention mechanisms to capture
dependencies between multiple related pairs in the same sentence. For the
inter-sentence relations, they used the transformer network to improve
performance for longer sequences. Research on RNNs and its variants has been
studied however there are few exploring CNN architectures. Therefore, in this
work, we explore CNN based architectures for relation extraction.

Contextualized language model-based approaches. Pre-trained contextualized
language models have been shown to increase the performance for several NLP
tasks. Wei, et al. [14] and Alimova, et al. [15] applied pre-trained language
models of BERT to the ADE RE task. Wei, et al. [14] developed two BERT-based
methods: Fine-Tuned BERT (FT-BERT) and Feature Combined BERT (FC-BERT) to
determine relation categories for these candidate pairs. For the FT-BERT
models, they represent a candidate relation pair in an input sentence by
replacing the entity with its semantic type, and they added a linear
classification layer on the top of The BERT model to predict the labels. For
the FC-BERT, they represented the entities using the entity tags. Alimova, et
al. [15] proposed a machine learning model with a novel set of knowledge-based
and BioSentVec embedding [16] features. For comparison, they utilized three
BERT-based models: BERT_uncased, BioBERT, and Clinical BERT. They utilized the
entity texts combined with a context between them as an input for the BERT-
based models.

##  3 Dataset

We evaluate our approaches on the National NLP Clinical Challenges (n2c2) 2018
Adverse Drug Event Dataset [1]. The dataset contains ADE mentions, drug-
related attributes, and drug-related relations from 505 patient discharge
summaries drawn from the MIMIC-III database [17]. It consists of nine entity
types (Drug, Strength, Route, Form, ADE, Dosage, Reason, Frequency) and eight
relations between the drug entity and other non-drug entity types. Table 1
shows the number of relations in the training and test data. We use the gold
annotated entities of this dataset for RE.

Table 1: Relation type statistics of n2c2 2018 data sets.

n2c2 dataset
---
Relation | # Train instances | # Test instances
Strength-Drug | 6702 | 4244
Duration-Drug | 643 | 426
Route-Drug | 5538 | 3546
Form-Drug | 6654 | 4374
ADE-Drug | 1107 | 733
Dosage-Drug | 4225 | 2695
Reason-Drug | 5169 | 3410
Frequency-Drug | 6310 | 4034

##  4 Methods

In this work, we explore several approaches for ADE extraction: rule-based
approaches, two deep learning-based approaches, and a contextualized language
model-based approach. The remainder of this section describes the systems in
detail.

###  4.1 Rule-based approach

In our rule-based system, we utilize the co-location information between the
drug and the non-drug entity types to determine if the non-drug entity is
referring to the drug. We use a breadth-first search algorithm to find the
closest occurrence of the drug on either side of the non-drug entity. For each
non-drug entity, we traverse both sides until the closest occurrence of the
drug is found based on the provided span values of the entities. We explore
four traversal mechanisms and report the best traversal mechanism in our
results: 1) traverse left-only, 2) traverse right-only, 3) traverse left-
first-then-right, and 4) traverse right-first-then-left. This was conducted in
two modes: 1) limiting the traversal to only a single relation per relation
type (bounded), or 2) allowing for a drug to be linked to multiple entity
types with the same relation (unbounded). For example, the sentence Once her
hematocrit stabilized, she was started on a heparin gtt with coumadin overlap.
contains a non-drug entity, gtt (Route) and two drugs Heparin and Coumadin.
The non-drug entity has a relation with the closest drug occurrence Heparin
but not with Coumadin when applying the left-only traversal mechanism.

###  4.2 Deep learning-based approach

Here, we describe the CNN architectures used in this work. CNNs consist of
four main layers [18]: embedding, convolution, pooling, and feed-forward
layers. Initially, the convolution layer which is a filter learns using the
backpropagation algorithm and extracts features from the input. Then the max-
pooling layer uses the position information and helps to extract the most
significant features from the output of the convolution filter. Finally, the
feed-forward layer uses a softmax classifier that performs classification.
CNNs take pre-trained word vectors obtained from an external resource as
input. Here, we explore two word embedding types: word2vec [10] and GloVe
[19]. We treat the RE task as a binary classification task building a separate
model for each drug-entity type to determine whether a relation exists between
two entities.

Sentence CNN. In this architecture, for each drug-entity pair, we extract the
sentence containing the relation and feed it into a CNN where each word in the
sentence is represented as a vector embedding. We then apply the convolution
layer to learn the local features from the embedding vectors and then the max-
pooling layer to extract the most important features from the sentence.
Finally, the vector is fed into a softmax (fully-connected) layer to perform
the classification. The classification error is then back-propagated, and the
model is re-trained until the loss is minimized. Figure 1 shows an
illustration of the Sentence CNN architecture.

Figure 1: An illustration of our model for Sentence-CNN. It explains the
process of both single label and multi-label sentence CNN

Segment-CNN. In this architecture, the sentence is divided into segments and
trained by separate convolutional units. First, we extract the sentence
containing the relation, and we divide it into five segments: 1) preceding -
tokenized words before the first concept; 2) concept 1 - tokenized words in
the first concept; 3) middle - tokenized words between the two concepts 4)
concept 2 - tokenized words in the second concept; and 5) succeeding -
tokenized words after the second concept. Figure 2 explains how an extracted
input sentence is divided into five segments.

Figure 2: An example of an input sentence that illustrates how the
segmentation is done

We construct separate convolution units for each segment and concatenate them
before we feed the fixed-length vector into the dense layer that performs the
classification. Each convolution unit applies a sliding window that processes
the segment and feeds the output to the max-pooling layer to extract essential
features independent of their location. The output features of the max-pooling
layer of each segment are then flattened and concatenated into a vector before
feeding it into the fully connected feed-forward layer. The vector is finally
fed into a softmax layer to perform the classification. Figure 3 shows an
illustration of Segment-CNN architecture.

Figure 3: An illustration of our model for Segment-CNN

###  4.3 Contextualized language model-based approach

In this approach, we explore using Bidirectional Encoder Representations from
Transformers (BERT) [20] contextualized embeddings into a simple feed-forward
neural network. We first extract the sentence containing the relation and pass
it through a pre-trained BERT model. The output is then fed into a dropout
layer and then into a fully-connected dense layer for classification. As with
our deep learning-based approaches, we treat the RE as a binary classification
task building a separate model for each drug-entity type. We explore the
following BERT-based language models:

  * •

BERT [20] (-cased and -uncased). The original BERT models are trained on a
large corpus of English data: BookCorpus (800M words) and Wikipedia (2,500M
words) in a self-supervised manner (without human annotation). BERT-based
models are smaller BERT models intended for environments with limited
computational resources. BERT_uncased and BERT_cased have 2-heads, 12-layers,
768-hidden units/layer, and a total of 110 M parameters.

  * •

BioBERT [21]. This model is initialized with the general BERT and further
trained over a corpus of biomedical research articles from
PubMed222https://www.ncbi.nlm.nih.gov/pubmed/ abstracts and PubMed
Central333https://www.ncbi.nlm.nih.gov/pmc/ article full texts.

  * •

Clinical BERT [22]. This model is initialized with BioBERT and further fine-
tuned over the Medical Information Mart for Intensive Care-III [17] (MIMIC-
III) clinical note corpus.

##  5 Experimental design

Word representation. For our deep learning-based approaches we use Word2Vec
[10] and GloVe [19] representations. The Word2Vec algorithm is trained over
the Medical Information Mart for Intensive Care (MIMIC-III), an openly
available dataset developed by the MIT Lab for Computational Physiology,
comprising 2 million clinical notes from nearly 40,000 critical care patients.
The gloVe is trained over Wikipedia (2014) and Gigaword 5.

Text tokenization and vectorization. For the rule-based and the deep learning-
based approaches, we use SpaCy tokenizer444https://spacy.io/api/tokenizer and
Keras tokenizer. For the contextualized language model-based approaches, we
use BertTokenizer and AutoTokenizer [22].

Hyper-parameters. We define our model training hyper-parameters by adjusting
the batch size, learning rate, and the number of epochs. We use the batch size
of 512, rmsprop optimizer with the learning rate of 0.001, and train for 10-20
epochs for our deep learning-based approach. We used the
HuggingFaceTransformers555https://huggingface.co/transformers/ to build the
BERT model for our contextualized-learning-base approach with Tensorflow 2.0.
We use TFRecord to read data into a Dataset object efficiently. We use
SparseCategoricalCrossentropy as the loss function and Adam as the optimizer
to minimize the loss function.

##  6 Evaluation criteria

We evaluate our approaches using Precision (P), Recall (R), and $F_{1}$ score
(F). Precision calculates out of all instances how many instances are
predicted correct, and Recall calculates out of all the correct instances that
should have been predicted how many instances are correctly predicted. $F_{1}$
score is the harmonic mean of Precision and Recall. We also report the micro
and macro averages of the system performance. Micro average calculates metrics
globally by counting the total true positives, false negatives, and false
positives, whereas macro average calculates metrics for each label and the
unweighted mean as it does not take class imbalance into account.

##  7 Results and Discussion

In this section, we describe the results of our three approaches, discuss the
results across our three models, and compare previous work.

###  7.1 Individual model results

Rule-based approach results. Table 2 shows the Precision, Recall, and $F_{1}$
scores for our rule-based approach on the test set of the n2c2-2018 dataset
for the top three traversal mechanisms described in our method section.
Analysis of the various traversal mechanisms over all the non-drug entities
showed that the Left-only traversal mechanism obtained the best results except
for the entity-drug pair Duration-Drug, ADE-Drug, and Reason-Drug. Using the
Left-Right (unbounded) traversal mechanism obtained the highest $F_{1}$ score
for these three entities. This is mainly because all other drug attributes are
usually mentioned before the drug entity mentions, but the Duration, Reason,
and ADE are usually mentioned after the drug mentions. Overall, This approach
achieved an overall Precision of 0.88, Recall of 0.83, and $F_{1}$ score of
0.86.

Table 2: Results for our rule-based approaches over the n2c2-2018 test set

| Left-only | Left-Right (unbounded) | Left-Right (bounded)
---|---|---|---
P | R | F | P | R | F | P | R | F
Strength-Drug | 0.96 | 0.95 | 0.95 | 0.46 | 0.90 | 0.61 | 0.94 | 0.94 | 0.94
Duration-Drug | 0.78 | 0.69 | 0.73 | 0.58 | 0.74 | 0.65 | 0.46 | 0.41 | 0.43
Route-Drug | 0.90 | 0.89 | 0.89 | 0.45 | 0.64 | 0.53 | 0.37 | 0.36 | 0.37
Form-Drug | 0.98 | 0.98 | 0.98 | 0.62 | 0.63 | 0.63 | 0.67 | 0.66 | 0.67
ADE-Drug | 0.46 | 0.39 | 0.43 | 0.55 | 0.75 | 0.64 | 0.60 | 0.51 | 0.55
Dosage-Drug | 0.89 | 0.89 | 0.89 | 0.61 | 0.57 | 0.59 | 0.89 | 0.88 | 0.89
Reason-Drug | 0.48 | 0.35 | 0.41 | 0.61 | 0.57 | 0.59 | 0.39 | 0.28 | 0.33
Frequency-Drug | 0.98 | 0.98 | 0.98 | 0.39 | 0.62 | 0.48 | 0.10 | 0.10 | 0.10
System (Micro) | 0.88 | 0.83 | 0.86 | 0.50 | 0.67 | 0.57 | 0.56 | 0.53 | 0.55
System (Macro) | 0.85 | 0.80 | 0.83 | 0.61 | 0.70 | 0.63 | 0.58 | 0.53 | 0.55

The results indicate that for most entity-drug pairs, co-location information
is sufficient to identify most relations. However, the performance of the
entity-drug pair ADE-Drug and Reason-Drug are lower compared to the other
relation types. Our supposition for this is that the co-location information
was insufficient to identify the correct ADE or Reason when multiple drugs
were in the same sentence. For example, in the sentence ”Since no new
infection was found this was presumed steroids and the leukocytosis improved
with prednisone taper.” the non-drug entity leukocytosis (ADE) is associated
with both steroids (Drug) and prednisone (Drug).

Deep learning-based results. Table 3 shows the Precision (P), Recall (R) and
$F_{1}$ scores for our Segment-CNN and Sentence CNN models over the n2c2-2018
test set. The results show that both models performed comparatively similar.
In theory, we believed that Segment-CNN should have performed better because
the Sentence CNN cannot differentiate the inputs when multiple drug-entity
pairs are located in a sentence, but the results contradict the assumption. We
believe this is because we treat this as a binary classification problem and
build a separate model for each relation type.

Table 3: Results of our deep learning-based approaches over the n2c2-2018 test
set

| Segment-CNN | Sentence CNN
---|---|---
P | R | F | P | R | F
Strength-Drug | 0.91 | 0.88 | 0.90 | 0.90 | 0.91 | 0.90
Duration-Drug | 0.39 | 0.90 | 0.55 | 0.41 | 0.90 | 0.57
Route-Drug | 0.77 | 0.89 | 0.83 | 0.76 | 0.91 | 0.83
Form-Drug | 0.85 | 0.95 | 0.90 | 0.85 | 0.96 | 0.90
ADE-Drug | 0.32 | 0.85 | 0.46 | 0.32 | 0.85 | 0.46
Dosage-Drug | 0.83 | 0.92 | 0.87 | 0.82 | 0.93 | 0.87
Reason-Drug | 0.27 | 0.88 | 0.42 | 0.27 | 0.88 | 0.41
Frequency-Drug | 0.56 | 0.88 | 0.69 | 0.56 | 0.88 | 0.69
System (Micro) | 0.69 | 0.90 | 0.78 | 0.68 | 0.92 | 0.78
System (Macro) | 0.68 | 0.90 | 0.77 | 0.67 | 0.91 | 0.77

Segment-CNN performed well with word2vec, whereas Sentence-CNN performed well
with GloVe embeddings. We believe this is because the glove embeddings are
trained over a more extensive dataset, while the word2vec MIMIC-III embeddings
are trained in the same domain.

Contextualized language model-based results. Table 4 shows the Precision (P),
Recall (R) and $F_{1}$ scores of the four fine-tuned BERT models over the
n2c2-2018 test dataset. Results show that the models obtain a similar
performance overall and for each entity-drug pairs. Comparatively, BERT_cased
model performs better in some categories than the other models.

Table 4: Results of our contextualized langauge model-based approaches over
the n2c2-2018 test set

| BERT (uncased) | BERT (cased) | BioBERT | Clinical BERT
---|---|---|---|---
| P | R | F | P | R | F | P | R | F | P | R | F
Strength-Drug | 0.86 | 0.88 | 0.87 | 0.86 | 0.99 | 0.92 | 0.86 | 0.90 | 0.88 | 0.87 | 0.82 | 0.84
Duration-Drug | 0.95 | 0.93 | 0.94 | 0.96 | 0.93 | 0.94 | 0.96 | 0.93 | 0.95 | 0.96 | 0.92 | 0.94
Route-Drug | 0.92 | 0.99 | 0.95 | 0.92 | 0.97 | 0.97 | 0.92 | 0.97 | 0.94 | 0.92 | 0.95 | 0.93
Form-Drug | 0.96 | 0.97 | 0.97 | 0.96 | 0.95 | 0.96 | 0.96 | 0.97 | 0.96 | 0.96 | 0.97 | 0.97
ADE-Drug | 0.95 | 0.99 | 0.97 | 0.95 | 0.99 | 0.97 | 0.95 | 0.99 | 0.97 | 0.95 | 0.99 | 0.97
Dosage-Drug | 0.93 | 0.96 | 0.94 | 0.93 | 0.96 | 0.95 | 0.93 | 0.96 | 0.94 | 0.93 | 0.89 | 0.91
Reason-Drug | 0.96 | 0.98 | 0.97 | 0.96 | 0.98 | 0.97 | 0.96 | 0.99 | 0.97 | 0.96 | 0.99 | 0.97
Frequency-Drug | 0.93 | 0.96 | 0.94 | 0.93 | 0.92 | 0.93 | 0.93 | 0.95 | 0.94 | 0.93 | 0.95 | 0.94
System (Micro) | 0.93 | 0.96 | 0.94 | 0.93 | 0.96 | 0.94 | 0.93 | 0.95 | 0.94 | 0.93 | 0.96 | 0.94
System (Macro) | 0.92 | 0.95 | 0.93 | 0.92 | 0.96 | 0.93 | 0.92 | 0.95 | 0.93 | 0.92 | 0.95 | 0.93

Figure 4: Error analysis of each relation type during the binary
classification using BERT (uncased) model. The Blue and Brown bars represent
the positive and negative classes respectively.

###  7.2 Negation Analysis

In this work, we performed a binary classification for each class: 1) Positive
class - there is a relation between the drug and the entity, 2) Negative class
- there is no relation between the drug and the entity (no-relation). Figure 4
shows the breakdown of the performance of each class when the binary
classification is performed using the BERT_uncased model. We report Support,
Precision, Recall, and $F_{1}$ score for each class, and Blue and Brown bars
represent the positive and negative classes, respectively. The support shows
the number of actual occurrences of the classes. We can see the Precision,
Recall, and $F_{1}$ score of the positive classes are way higher than the
negative classes. We believe this explains the higher performance of the BERT
models. The performance of the negative (no-relation) classes is low due to
the data imbalance of the classes, as shown in the support. Positive classes
are significantly larger than the negative classes, and due to this, the poor
performance of the negative classes did not affect the performance of the
positive class.

###  7.3 Comparison across models

Table 5 shows the Precision (P), Recall (R), and $F_{1}$ score for the best
results of each of our three approaches: 1) rule-based approach using left-
only traversal mechanism; 2) deep learning approach using Segment-CNN, and 3)
contextualized language model-based approach using BioBERT. Comparing the
rule-based approach with our deep learning-based approach shows that the rule-
based approach obtained an overall higher Precision, Recall, and $F_{1}$ score
except for the classes ADE-Drug and Reason-Drug. BERT-based models outperform
the other two approaches except for the Strength-Drug, Frequency-Drug, and
Form-Drug pairs. The overall Precision and Recall are higher, especially for
the entity-drug pairs that performed poorly with the other approaches (ADE-
Drug, Reason-Drug, and Duration-Drug). Using pre-trained language
representations to fine-tune models is advantageous as they use minimal task-
specific parameters and are trained on the downstream tasks by simply fine-
tuning all the pre-trained parameters.

Table 5: Comparison across our approaches over the n2c2-2018 test set

| Train | Test | Rule-based | Segment-CNN | BioBERT
---|---|---|---|---|---
# | # | P | R | F | P | R | F | P | R | F
Strength-Drug | 6702 | 4244 | 0.96 | 0.95 | 0.95 | 0.91 | 0.88 | 0.90 | 0.86 | 0.90 | 0.88
Duration-Drug | 643 | 426 | 0.78 | 0.69 | 0.73 | 0.39 | 0.90 | 0.55 | 0.96 | 0.93 | 0.95
Route-Drug | 5538 | 3546 | 0.90 | 0.89 | 0.89 | 0.77 | 0.89 | 0.83 | 0.92 | 0.97 | 0.94
Form-Drug | 6654 | 4373 | 0.98 | 0.98 | 0.98 | 0.85 | 0.95 | 0.90 | 0.96 | 0.97 | 0.96
ADE-Drug | 1107 | 733 | 0.46 | 0.39 | 0.43 | 0.32 | 0.85 | 0.46 | 0.95 | 0.99 | 0.97
Dosage-Drug | 4255 | 2695 | 0.89 | 0.89 | 0.89 | 0.83 | 0.92 | 0.87 | 0.93 | 0.96 | 0.94
Reason-Drug | 5169 | 3410 | 0.48 | 0.35 | 0.41 | 0.27 | 0.88 | 0.42 | 0.96 | 0.99 | 0.97
Frequency-Drug | 6310 | 4034 | 0.98 | 0.98 | 0.98 | 0.56 | 0.88 | 0.69 | 0.93 | 0.95 | 0.94
System (Micro) | 0.88 | 0.83 | 0.86 | 0.69 | 0.90 | 0.78 | 0.93 | 0.95 | 0.94
System (Macro) | 0.85 | 0.80 | 0.83 | 0.68 | 0.90 | 0.77 | 0.92 | 0.95 | 0.93

###  7.4 Comparison with previous work

In this section, we compare our results with two previous works utilizing
BERT: Wei, et al. [14] and Alimova, et al. [15] To the best of our knowledge,
these are the only two works that have applied pre-trained language models of
BERT on the n2c2-2018 dataset. Table 6 shows the overall Precision, Recall,
and $F_{1}$ score of our fine-tuned BERT models with the reported results from
the other state-of-the-art BERT-based models on the n2c2-2018 dataset. The
$F_{1}$ score of all models of Wei et al’s and our three models is same, but
the Precision of Wei, et al’s models is higher whereas the Recall of our
models is higher. There is a notable difference between Alimova, et al. models
and ours. The $F_{1}$ score of all three models of Alimova, et al. are lower
than ours, and we believe this is due to the difference in the representation
of the inputs for the models.

Table 6: Overall results in comparison with previous work on the n2c2-2018
test data

| Our models | Wei, et al. [14] | Alimova, et al. [15]
---|---|---|---
| Cased | Uncased | Bio | Clinical | Cased | Uncased | Bio | Clinical | Uncased | Bio | Clinical
Precision | 0.93 | 0.93 | 0.93 | 0.93 | 0.98 | 0.98 | 0.98 | 0.98 | - | - | -
Recall | 0.96 | 0.96 | 0.95 | 0.93 | 0.90 | 0.90 | 0.90 | 0.90 | - | - | -
$F_{1}$ score | 0.94 | 0.94 | 0.94 | 0.93 | 0.94 | 0.94 | 0.94 | 0.94 | 0.56 | 0.75 | 0.75

Table 7: Comparison of $F_{1}$ score with previous work over each class of the
n2c2-2018 dataset.

| Our models | Wei, et al. [14] | Alimova, et al. [15]
---|---|---|---
| Cased | Uncased | Bio | Clinical | Cased | Uncased | Bio | Clinical | Uncased | Bio | Clinical
Strength-Drug | 0.87 | 0.87 | 0.88 | 0.84 | 0.98 | 0.99 | 0.98 | 0.99 | 0.58 | 0.68 | 0.68
Duration-Drug | 0.94 | 0.94 | 0.94 | 0.94 | 0.88 | 0.89 | 0.88 | 0.89 | 0.41 | 0.66 | 0.65
Route-Drug | 0.95 | 0.95 | 0.94 | 0.93 | 0.97 | 0.97 | 0.97 | 0.97 | 0.63 | 0.74 | 0.74
Form-Drug | 0.97 | 0.97 | 0.96 | 0.97 | 0.97 | 0.98 | 0.98 | 0.98 | 0.62 | 0.81 | 0.81
ADE-Drug | 0.97 | 0.97 | 0.97 | 0.97 | 0.80 | 0.80 | 0.81 | 0.81 | 0.10 | 0.62 | 0.62
Dosage-Drug | 0.94 | 0.94 | 0.94 | 0.91 | 0.97 | 0.97 | 0.97 | 0.97 | 0.67 | 0.82 | 0.82
Reason-Drug | 0.97 | 0.97 | 0.97 | 0.97 | 0.76 | 0.76 | 0.76 | 0.77 | 0.22 | 0.73 | 0.73
Frequency-Drug | 0.94 | 0.94 | 0.94 | 0.94 | 0.96 | 0.96 | 0.96 | 0.96 | 0.53 | 0.79 | 0.78

Table 7 shows a comparison of the $F_{1}$ score of our models the results
reported by Wei, et al. [14]’s and Alimova, et al. [15] for each class of the
dataset. The results show that Alimova, et al.’s models perform lower.
However, when comparing the results with Wei, et al., we found the results are
complementary; classes that did not perform well with Wei, et al.’s models
performed well with our models. Specifically, the classes Reason-Drug, ADE-
Drug, and Duration-Drug obtained a higher Precision, Recall, and $F-1$ score
than Wei, et al.’s. Meanwhile, the Precision, Recall, and $F-1$ score of the
class Strength-Drug are higher in Wei, et al. We believe this is due to three
differences between our systems: 1) Wei, et al. represent an entity-drug pair
in an input sentence using the semantic type of an entity to replace the
entity itself, whereas we do no such replacement; 2) they perform a multi-
class classification, whereas we perform binary classification creating a
separate model for each entity; and 3) Wei, et al. Clinical BERT
representations were fine-tuned with MIMIC-III over BERT (cased), whereas our
representations were fine-tuned over BioBERT.

Table 8 shows the comparison of the top five team results reported by the
n2c2-2018 challenge participants [1] and our best models. The UTH system
developed a joint learning biLSTM+CRF based architecture to identify the
entities and relations together; the VA team used a complex traditional
machine learning approach that utilized random forests; NaCT proposed a deep
learning-based ensemble method; and MDQ used a biLSTM with an attention
mechanism. The results show that our BERT based system obtained a higher
recall across all of the systems but a lower precision across the first four
teams.

Table 8: Our best results in comparison with the top 5 results of the
n2c2-2018 competition

| n2c2-2018 Teams | Our systems
---|---|---
| UTH | VA | NaCT | UFL | MDQ | BERT | Segment-CNN | Rule
Precision | 0.96 | 0.95 | 0.94 | 0.95 | 0.93 | 0.93 | 0.69 | 0.88
Recall | 0.95 | 0.94 | 0.94 | 0.92 | 0.94 | 0.96 | 0.90 | 0.83
$F_{1}$ score | 0.96 | 0.94 | 0.94 | 0.94 | 0.94 | 0.94 | 0.78 | 0.86

##  8 Conclusions and Future work

In this work, we have investigated diverse approaches to identify relations
between medication information and adverse drug events from clinical notes. We
explored a rule-based, deep learning-based, and contextualized language model-
based approaches. We evaluated our approaches on the n2c2-2018 dataset and
found overall the contextualized language model-based approach using BioBERT
outperformed the other approaches. However, our results also showed that the
rule-based approach which uses co-location information was sufficient to
identify relations between entities whose positions with respect to each other
were consistent throughout the text (e.g. Strength-Drug, Form-Drug and
Frequency-Drug). In our contextualized language model-based approaches, we
represent a drug-entity pair by the entire sentence, but this may not
advisable as we can have multiple drug-entity pairs within a sentence.
Therefore in the future, we plan to investigate effective ways of unique
representations of a drug-entity pair that can capture the positional
information of both the drug and entity. In this work, we developed a separate
model for each class of the dataset and performed binary classification
separately. In the future, we plan to investigate expanding the model to
perform multi-class classification for different datasets.

## References

  * 1. Henry S, Buchan K, Filannino M, Stubbs A, Uzuner O.  2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records.  Journal of the American Medical Informatics Association. 2020;27(1):3–12.
  * 2. Classen DC, Pestotnik SL, Evans RS, Lloyd JF, Burke JP.  Adverse drug events in hospitalized patients: excess length of stay, extra costs, and attributable mortality.  Jama. 1997;277(4):301–306.
  * 3. Luo Y.  Recurrent neural networks for classifying relations in clinical notes.  Journal of biomedical informatics. 2017;72:85–95.
  * 4. Schatz SN, Weber RJ.  Adverse drug reactions. ACCP (American College of Clinical Pharmacy). CNS.  Pharmacy Practice, PSAP. 2015;.
  * 5. Li F, Liu W, Yu H.  Extraction of information related to adverse drug events from electronic health record notes: design of an end-to-end model based on deep learning.  JMIR medical informatics. 2018;6(4):e12159.
  * 6. Li Q, Spooner SA, Kaiser M, Lingren N, Robbins J, Lingren T, et al.  An end-to-end hybrid algorithm for automated medication discrepancy detection.  BMC medical informatics and decision making. 2015;15(1):37.
  * 7. Miller T, Geva A, Dligach D.  Extracting adverse drug event information with minimal engineering.  In: Proceedings of the 2nd Clinical Natural Language Processing Workshop; 2019. p. 22–27.
  * 8. Dligach D, Bethard S, Becker L, Miller T, Savova GK.  Discovering body site and severity modifiers in clinical texts.  Journal of the American Medical Informatics Association. 2014;21(3):448–454.
  * 9. Yang X, Bian J, Fang R, Bjarnadottir RI, Hogan WR, Wu Y.  Identifying relations of medications with adverse drug events using recurrent convolutional neural networks and gradient boosting.  Journal of the American Medical Informatics Association. 2020;27(1):65–72.
  * 10. Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J.  Distributed representations of words and phrases and their compositionality.  In: Advances in neural information processing systems; 2013. p. 3111–3119.
  * 11. Xu J, Lee HJ, Ji Z, Wang J, Wei Q, Xu H.  UTH_CCB System for Adverse Drug Reaction Extraction from Drug Labels at TAC-ADR 2017.  In: Proceedings of the Text Analysis Conference; 2017. .
  * 12. Sorokin D, Gurevych I.  Context-aware representations for knowledge base relation extraction.  In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing; 2017. p. 1784–1789.
  * 13. Christopoulou F, Tran TT, Sahu SK, Miwa M, Ananiadou S.  Adverse drug events and medication relation extraction in electronic health records with ensemble deep learning methods.  Journal of the American Medical Informatics Association. 2020;27(1):39–46.
  * 14. Wei Q, Ji Z, Si Y, Du J, Wang J, Tiryaki F, et al.  Relation Extraction from Clinical Narratives Using Pre-trained Language Models.  In: AMIA Annual Symposium Proceedings. vol. 2019. American Medical Informatics Association; 2019. p. 1236.
  * 15. Alimova I, Tutubalina E.  Multiple features for clinical relation extraction: A machine learning approach.  Journal of Biomedical Informatics. 2020;103:103382.
  * 16. Chen Q, Peng Y, Lu Z.  BioSentVec: creating sentence embeddings for biomedical texts.  In: 2019 IEEE International Conference on Healthcare Informatics (ICHI). IEEE; 2019. p. 1–5.
  * 17. Johnson AE, Pollard TJ, Shen L, Li-wei HL, Feng M, Ghassemi M, et al.  MIMIC-III, a freely accessible critical care database.  Scientific data. 2016;3:160035.
  * 18. Nguyen TH, Grishman R.  Relation extraction: Perspective from convolutional neural networks.  In: Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing; 2015. p. 39–48.
  * 19. Pennington J, Socher R, Manning C.  Glove: Global vectors for word representation.  In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP); 2014. p. 1532–1543.
  * 20. Devlin J, Chang MW, Lee K, Toutanova K.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:181004805. 2018;.
  * 21. Lee J, Yoon W, Kim S, Kim D, Kim S, So CH, et al.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  Bioinformatics. 2020;36(4):1234–1240.
  * 22. Alsentzer E, Murphy JR, Boag W, Weng WH, Jin D, Naumann T, et al.  Publicly available clinical BERT embeddings.  arXiv preprint arXiv:190403323. 2019;.