#  Fluctuation-dissipation relations for spiking neurons

Benjamin Lindner  Bernstein Center for Computational Neuroscience Berlin,
Philippstr. 13, Haus 2, 10115 Berlin, Germany  Physics Department of Humboldt
University Berlin, Newtonstr. 15, 12489 Berlin, Germany

###### Abstract

Spontaneous fluctuations and stimulus response are essential features of
neural functioning but how they are connected is poorly understood. I derive
fluctuation-dissipation relations (FDR) between the spontaneous spike and
voltage correlations and the firing rate susceptibility for i) the leaky
integrate-and-fire (IF) model with white noise; ii) an IF model with arbitrary
voltage dependence, an adaptation current, and correlated noise. The FDRs can
be used to derive correlation statistics or to infer the system’s response
from observations of its spontaneous activity.

Small physical systems often display considerable fluctuations that can be
characterized by correlation functions or power spectra. Fluctuation-
dissipation relations connect the statistics of these spontaneous fluctuations
of certain observables to their mean response to a time-dependent
perturbation. Originally proposed for equilibrium thermodynamic systems Callen
and T.A.Welton (1951); Kubo (1966), they have been extended to nonequilibrium
setups with a steady state Agarwal (1972); Hänggi and Thomas (1982); Prost et
al. (2009); Gomez-Solano et al. (2009). Fluctuation-dissipation theorems can
be used to infer the response properties from observations of purely
spontaneous activity, to prove in a model-free way that a system operates
outside thermodynamic equilibrium Martin et al. (2001); Mizuno et al. (2007),
or to test whether a system obeys a Markovian description Dinis et al. (2012);
Willareth et al. (2017); for general reviews on applications of FDRs, see the
comprehensive reviews Marconi et al. (2008); Seifert (2012).

Fluctuations are especially prominent in neural systems, specifically in the
spike generation of neurons (nerve cells) in the brain, which is reflected in
a long history of stochastic modeling in neuroscience Tuckwell (1989);
Gerstner et al. (2014). Neurons are notoriously noisy due to intrinsic sources
of fluctuations (e.g. channel noise and unreliable synaptic transmission); in
the recurrent networks of the cortex, the nonlinear interactions among many
pulse-generating units lead to a strong chaotic variability (a network noise)
even if single units follow a completely deterministic dynamics (i.e. the
above mentioned channel noise, for instance, is neglected) and even if
external (noisy) stimulation is absent. Most importantly, the response to
external signals are of overarching importance for nerve cells, as it
characterizes the transmission and processing of information, which is the
main task of these cells. So, it is of vital importance to understand
potential connections between the statistics of spontaneous activity and the
response to a time-dependent perturbation in the case of spiking neurons.

Let us consider a paradigmatic stochastic model of computational neuroscience,
the leaky integrate-and-fire model with white noise $\xi(t)$ and a time-
dependent current signal $s(t)$:

$\frac{dv}{dt}=-v+\mu+s(t)+\sqrt{2D}\xi(t).$ (1)

The voltage across the nerve membrane, $v(t)$, upon reaching a threshold
$v_{T}$, is reset to $v_{R}<v_{T}$ and, simultaneously, the time instant is
registered as a spike time $t_{i}$. The most important output of this model is
the spike train, $x(t)=\sum\delta(t-t_{i})$ (this is what is communicated to
other cells). In Eq. (1) time and voltage are measured in multiples of the
membrane time constant $\tau_{m}$ and the threshold-reset distance,
respectively. The mean constant input $\mu$ and the intensity of the white
noise $D$ are important parameters that determine the stochastic regime of the
model Vilela and Lindner (2009).

For the spontaneous activity ($s(t)\equiv 0$) the power spectrum of the spike
train can be analytically calculated and expressed in terms of parabolic
cylinder functions $\mathcal{D}_{a}(x)$ (see e.g. Lindner (2002)):

$S_{xx}(\omega)=r_{0}\frac{|\mathcal{D}_{i\omega}(z_{T})|^{2}-e^{\frac{z_{R}^{2}-z_{T}^{2}}{2}}|\mathcal{D}_{i\omega}(z_{R})|^{2}}{|\mathcal{D}_{i\omega}(z_{T})-e^{\frac{z_{R}^{2}-z_{T}^{2}}{4}}\mathcal{D}_{i\omega}(z_{R})|^{2}}.$
(2)

Here $z_{T/R}=(\mu-v_{T/R})\sqrt{D}$ and $r_{0}=\left\langle
x(t)\right\rangle$ is the stationary firing rate not (a) with the angular
brackets indicating an ensemble average.

The response to a weak signal $s(t)$ is quantified by the time-dependent rate
modulation $r(t)\approx r_{0}+K_{x}*s(t)$, given in terms of a convolution
with the linear response function $K_{x}(t)$ or in terms of the susceptibility
$\chi_{x}(\omega)$ (the Fourier transform of $K_{x}$), which can be expressed
by confluent hypergeometric functions Brunel et al. (2001) or, equivalently,
again in terms of parabolic cylinder functions Lindner and Schimansky-Geier
(2001):

$\chi_{x}(\omega)\\!=\\!\frac{}{}\frac{ir_{0}\omega/\sqrt{D}}{i\omega-1}\frac{\mathcal{D}_{i\omega-1}(z_{T})-e^{\frac{z_{R}^{2}-z_{T}^{2}}{4}}\mathcal{D}_{i\omega-1}(z_{R})}{\mathcal{D}_{i\omega}(z_{T})-e^{\frac{z_{R}^{2}-z_{T}^{2}}{4}}\mathcal{D}_{i\omega}(z_{R})}.$
(3)

There is some structural similarity in the expressions for power spectrum and
susceptibility - both are given in terms of ratios of differences of parabolic
cylinder functions, but, apparently, it is not possible to express one in a
simple way by the other. So, even in this case, where we know the explicit
solutions for the two characteristics of spontaneous fluctuations and of the
response to a stimulus, it does not help us to connect them in a fluctuation-
dissipation relation. The situation is similar (analytical expressions are
known but cannot be related) for IF models with shot noise Richardson and
Swarbrick (2010); Droste and Lindner (2017a), with dichotomous background
noise Droste and Lindner (2017b), or escape noise Deger et al. (2010).

Here we connect the statistics of spontaneous spiking and the firing rate
response to a weak signal by means of a simple calculation, which is markedly
different to the typical derivation of the standard FDR not (b) and also to
recent calculations for IF models in discrete time and embedded in networks
Cessac et al. (2021). The approach here builds on two ideas: i) the reset can
be incorporated into the Langevin dynamics by means of the spike train (see
e.g. Laing and Chow (2001) or Meffin et al. (2001)), which permits to average
this and related equations, leading by the Rice method to equations for
spectral measures; ii) by means of the Furutsu-Novikov theorem Furutsu (1963);
Novikov (1965), we can relate the noise-spike-train correlator to the exact
linear response function. We first outline this calculation for the simple
model in Eq. (1) and then treat the biophysically more realistic and
dynamically richer exponential integrate-and-fire model endowed with
correlated (colored) current noise.

_LIF model with white noise.–_ Without signal current ($s(t)\equiv 0$), we can
rewrite Eq. (1) as follows

$\frac{dv}{dt}=-v+\mu+\sqrt{2D}\xi(t)-(v_{T}-v_{R})x(t),$ (4)

where the last term formally imposes the fire-and-reset rule: the Delta-
functions in $x(t)$ will push the voltage back from the threshold at $v_{T}$
to the reset point $v_{R}$. Having incorporated the reset rule into the
equation, we can now take averages over a stationary ensemble. If we, for
instance, directly average Eq. (4), we obtain

$\frac{d}{dt}\left\langle v\right\rangle=0=\mu-\left\langle
v\right\rangle-(v_{T}-v_{R})r_{0}.$ (5)

which leads to $\left\langle v\right\rangle=\mu-(v_{T}-v_{R})r_{0}$, a non-
trivial relation between mean membrane voltage and firing rate.

Next, we take Eq. (4) at time $t+\tau$, multiply with the spike train $x(t)$
at time $t$, and average. Using Eq. (5) and expressing the derivative by one
w.r.t. $\tau$, we arrive at

$\frac{dC_{xv}}{d\tau}=-C_{xv}-(v_{T}-v_{R})C_{xx}+\sqrt{2D}\left\langle
x(t)\xi(t+\tau)\right\rangle.$ (6)

Here we have introduced general correlation functions
$C_{yz}(\tau)=\left\langle y(t)z(t+\tau)\right\rangle-\left\langle
y\right\rangle\left\langle z\right\rangle$ for time series $y(t)$ and $z(t)$.
One crucial insight is now that the last term in the above equation, the
spike-train-white-noise correlation, is exactly proportional to the
susceptibility with respect to a weak signal _in the presence of the white
background noise of intensity $D$_. In fact, as a consequence of the Furutsu-
Novikov theorem this would hold true for any (in general, correlated) input
noise as long as it is Gaussian. To see this more directly for our problem,
imagine the (Gaussian) noise $\xi(t)=\sum\xi_{n}(t)$ being subdivided into $N$
independent Gaussian processes $\xi_{n}(t)$ with identical power spectra
$S_{nn}=S_{\xi,\xi}/N$. The above correlation is then the sum of the single
correlation functions $\left\langle x(t)\xi_{n}(t+\tau)\right\rangle$, in
which $\xi_{n}(t)$ represents a perfectly weak signal that is transmitted in
the presence of a noise of intensity $D$ (all the other $N-1$ terms amount now
to practically the entire noise process $\xi(t)$). In result, we obtain
$\sqrt{2D}\left\langle x(t)\xi(t+\tau)\right\rangle=2DK_{x}(-\tau)$ or, in the
Fourier domain, $\sqrt{2D}S_{x\xi}=S_{\xi\xi}\chi_{x}(\omega)$. Fourier
transformation of Eq. (6) then yields the following fluctuation-dissipation
relation for a stochastic LIF neuron with white noise

$\chi_{x}(\omega)=\frac{(v_{T}-v_{R})S_{xx}(\omega)+(1+i\omega)S_{xv}(\omega)}{2D}$
(7)

Figure 1: Confirmation of the FDR for a white-noise driven leaky IF model.
Real (top) and imaginary part (bottom) as functions of frequency for the left
side (response properties) and the right side (spontaneous activity) of Eq.
(7) for $\mu=0.8,D=0.1$ and a broadband stimulus (uniform power for
$|\omega|<2\pi\cdot 100$) and a small variance of $\left\langle
s^{2}(t)\right\rangle=0.1$. For both sets of simulations, $10^{4}$ trials, a
time step of $\Delta t=10^{-4}$ and a time window of $T\approx 100$ were used
(for real neurons with $\tau_{m}=10$ms, this would translate into a time
window of 1s).

On the left hand side, we find the susceptibility of the firing rate with
respect to a weak time-dependent signal, as can be, for instance, determined
by a periodic stimulation $s(t)=\varepsilon\cos(\omega_{s}t)$ from the rate
modulation $r(t)=\left\langle
x(t)\right\rangle=r_{0}+|\chi_{x}(\omega_{s})|\cos(\omega_{s}t-\arg(\chi_{x}(\omega_{s}))$
(here $\arg(\cdot)$ is the complex argument) or, equivalently, by a weak
broadband stimulus as was used in Fig. 1. On the right side of Eq. (7) are
statistics of the spontaneous activity ($s(t)\equiv 0$): Besides the spike
train power spectrum, the cross-spectrum between the subthreshold membrane
voltage and the generated spikes emerges - this is the missing link between
the spontaneous fluctuation statistics and the response statistics. For a
selected parameter set, the relation is tested and confirmed in Fig. 1.

Because we know most of the statistics by explicit expressions, we can use the
relation above to determine the cross-spectrum between $v(t)$ and $x(t)$
analytically:

$S_{xv}(\omega)=\frac{2D\chi_{x}(\omega)-(v_{T}-v_{R})S_{xx}(\omega)}{1+i\omega},$
(8)

which by virtue of Eq. (2) and Eq. (3) can be expressed by parabolic cylinder
functions and is confirmed in Fig. 2 by numerical simulations.

Figure 2: Cross-spectrum between subthreshold voltage and spike train.
Parameters as in Fig. 1.

The form Eq. (8) is instructive because it clearly shows the two sources of
cross-correlation between membrane voltage and spike train. For once, there
are the reset events occurring as steps at the spike times: The voltage
contains so to speak the integrated spike train and is thus in part correlated
to the spike train $x(t)$ as $x(t)$ is correlated to itself. Secondly, the
correlation with the noise is shared between voltage and spike train and
accounted for by the response function.

_Exponential IF model with adaptation current and colored noise.–_ We now turn
to a more general and biophysically more realistic model, which, in its
essential ingredients, has been justified on theoretical grounds Fourcaud-
Trocmé et al. (2003) but also extracted from data Badel et al. (2008). As
suggested by Brette and Gerstner Brette and Gerstner (2005), we include a
spike-triggered adaptation current Benda and Herz (2003), and instead of white
noise we allow for a Gaussian noise with arbitrary temporal correlations:

$\displaystyle\frac{dv}{dt}$ $\displaystyle=$ $\displaystyle
f(v)-a+\eta(t)-(v_{T}-v_{R})x(t)+s(t),$ (9)
$\displaystyle\tau_{a}\frac{da}{dt}$ $\displaystyle=$
$\displaystyle-a+\Delta_{a}\tau_{a}x(t).$ (10)

Here the white noise $\xi(t)$ has been replaced by a colored noise $\eta(t)$
with a prescribed power spectrum $S_{\eta\eta}(\omega)$. The simple linear
leak term has been replaced with the function
$f(v)=\mu-v+\Delta_{v}\exp((v-v_{t})/\Delta_{v}$ (see Fourcaud-Trocmé et al.
(2003)). We note that the parameter $v_{t}<v_{T}$ sets a kind of soft
threshold but we still keep a hard threshold at $v_{T}$ and a corresponding
reset rule (this has been already incorporated above). The variable $a(t)$
acts as an inhibitory current that pushes the voltage away from threshold. It
evolves according to the slow dynamics given in the second equation (the ratio
of its time constant to the membrane time constant is typically $\tau_{a}\gg
1$) but every spike generated by the model kicks the adaptation variable up by
the amount $\Delta_{a}$, which implements the negative feedback that results
in the spike-frequency adaptation seen in so many brain cells Benda and Herz
(2003).

We can again use the same methods to derive equations for correlation
functions and cross- and power spectra: Multiplication with $x(t)$ of the two
eqs. above taken at time $t+\tau$, averaging, using expressions for the mean
values, and, finally, expressing the noise-spike-train correlation function by
the linear response susceptibility. This gives us a fluctuation-dissipation
relation for the adapting exponential integrate-and-fire model:

$\chi_{x}=\frac{\left(v_{T}-v_{R}+\frac{\Delta\tau_{a}}{1+i\omega\tau_{a}}\right)S_{xx}+i\omega
S_{xv}-S_{xf(v)}}{S_{\eta\eta}}$ (11)

Again, on the left hand side we have exclusively the response to a weak
stimulus, which can be determined by means of a periodic or a broadband
stimulus $s(t)$. On the right hand side, we find exclusively statistics of the
spontaneous activity for $s(t)\equiv 0$. Several observations can be made: i)
the adaptation dynamics enters only by modifying the prefactor of the spike
train power spectrum in a frequency-specific manner; ii) instead of only the
cross-spectrum of the subthreshold membrane voltage and the spike train (which
still appears on the r.h.s.), we now also get the cross-spectrum of the spike
train $x(t)$ and the subthreshold nonlinearity $f(v(t))$, which in some
situations can be extracted from experiments Badel et al. (2008).

Figure 3: Confirmation of the FDR Eq. (11) for an exponential IF model with a
(colored) Ornstein-Uhlenbeck noise and a spike-triggered adaptation current.
Real (top) and imaginary part (bottom) as functions of frequency for the left
side (response property $\chi(\omega)$) and the right side (fluctuation
statistics) of Eq. (11) for
$\mu=0.8,\tau_{OUP}=10,\sigma^{2}=0.5,\tau_{a}=100,\left\langle
s^{2}(t)\right\rangle=0.2$ in (a) and
$\mu=4,\tau_{OUP}=1,\sigma^{2}=0.1,\tau_{a}=10,\left\langle
s^{2}(t)\right\rangle=0.1$ in (b). In both plots
$v_{t}=1,\Delta_{v}=0.2,\Delta_{a}=2,v_{R}=0,v_{T}=1$. For the determination
of the susceptibility a broadband stimulus (uniform power for
$|\omega|<2\pi\cdot 100$) was used. For all simulations, $10^{4}$ trials, a
time step of $\Delta t=10^{-3}$ (in a) and $\Delta t=10^{-4}$ (in b) were
used.

As examples we pick two cases in Fig. 3, which both confirm the FDR for the
adapting neuron with colored noise. For both cases it is difficult to
calculate analytically any of the statistics shown or used here.

In Fig. 3a the constant input current $\mu$ is so weak that the model is still
in the excitable regime (comparable to the LIF model) but we add a slow
adaptation current and low-pass filtered noise (Ornstein-Uhlenbeck noise with
an exponential correlation function and a correlation time that is ten times
the membrane time constant). In particular, the adaptation current leads to a
high-pass shape of the susceptibility Schwalger (2013). Remarkably, the
numerical fluctuations of the susceptibility determined by stimulation or from
the spontaneous statistics via Eq. (11) behave very differently: at small and
up to intermediate frequencies ($\omega<1$) the r.h.s. of Eq. (11) provides a
very reliable estimate of the susceptibility while the estimate obtained by
broadband stimulation is more noisy. In the high-frequency limit, it is the
other way around: the broadband stimulus yields a more reliable estimate of
the response while the estimate from the spontaneous statistics becomes really
noisy. Hence, if we want to know the response at specific frequencies, it will
depend on the frequency whether we can benefit from Eq. (11) or whether we
should resort to the simple stimulation method to get the susceptibility.

In Fig. 3b, the input current is increased, the Ornstein-Uhlenbeck noise is
reduced in standard deviation and correlation time (now equal to the membrane
time constant) such that the susceptibility displays pronounced resonances at
the firing rate (i.e. at $\omega=2\pi r_{0}$). Also in this dynamically very
different regime the relation between the fluctuation statistics and the
response, Eq. (11), is excellently confirmed.

One potential problem with verifying and/or exploiting the relationship Eq.
(11) for real neurons is that although the membrane voltage and the spike
train might be accessible, the same is not necessarily true for the
subthreshold nonlinearity $f(v)$. For non-adapting neurons, this can be
determined by _in vitro_ experiments Badel et al. (2008), however, such
experiments might not be possible or hampered by adaptation phenomena.
Comparison of fluctuation and response may help to infer appropriate values of
$v_{t}$ and $\Delta_{v}$, which would be one potential use for the FDR.

_Conclusions.–_ The relation between spontaneous fluctuations and the response
to external perturbations have been worked out for an important class of
spiking neuron models of the integrate-and-fire type. However, there is still
work to be done: In the same framework, it is easily possible to derive
relations for the susceptibility of the subthreshold membrane voltage and its
spontaneous power spectrum. The simple method introduced here can be applied
to all types of neuron model with spike-associated reset such as the two-
dimensional Izhikevich model Izhikevich (2007) or the generalized IF model
Richardson et al. (2003) if the voltage dynamics is driven by Gaussian noise.

We note that an entirely independent set of fluctuation-dissipation relations
can be derived by the more common approach to nonequilibrium thermodynamic
systems with a steady state going back to Agarwal Agarwal (1972); Hänggi and
Thomas (1982) and discussed also more recently in the literature Prost et al.
(2009); Dinis et al. (2012); Willareth et al. (2017). This approach will lead
to relations in terms of the correlation function of the conjugated variable
that is a highly nonlinear function of the membrane voltage steady state
distribution. Finally, there is also work on fluctuation-dissipation-relations
for integrate-and-fire models in discretized time by Cessac et al. Cessac et
al. (2021), the relation to results here has to be clarified.

The division into subthreshold voltage and spike train used in the derived
relations may appear somewhat conceived and as an artifact of the integrate-
and-fire framework. However, it is meaningful because the spike train
represents the important signal that is communicated to other neurons. In this
context, it will be also useful to generalize the analysis to conductance-
based neuron models of the Hodgkin-Huxley type. As the above mentioned
multidimensional IF models approximate conductance-based models in many
situations surprisingly well, I expect that the fluctuation-dissipation
relations found here may hold true (at least approximately) also for these
more detailed models of neural firing.

In the context of neural information transmission, it is worth mentioning that
both susceptibility and spike-train power spectra appear in the coherence
function which provides a frequency-resolved measure of signal transfer
Lindner (2016). Also in stochastic mean-field theories of neural networks,
both power spectra as well as susceptibilities play important roles (see e.g.
Brunel (2000); Knoll and Lindner (2022)). For these problems, fluctuation-
dissipation relations for spiking neurons may be used to simplify,
reformulate, and better understand the statistics of interest.

## References

  * Callen and T.A.Welton (1951) H. B. Callen and T.A.Welton, Phys. Rev. 83, 34 (1951).
  * Kubo (1966) R. Kubo, Rep. Prog. Phys. 29, 255 (1966).
  * Agarwal (1972) G. S. Agarwal, Z. Phys. 252, 25 (1972).
  * Hänggi and Thomas (1982) P. Hänggi and H. Thomas, Phys. Rep. 88, 207 (1982).
  * Prost et al. (2009) J. Prost, J.-F. Joanny, and J. R. Parrondo, Phys. Rev. Lett. 103, 090601 (2009).
  * Gomez-Solano et al. (2009) J. R. Gomez-Solano, A. Petrosyan, S. Ciliberto, R. Chetrite, and K. Gawedzki, Phys. Rev. Lett. 103, 040601 (2009).
  * Martin et al. (2001) P. Martin, A. J. Hudspeth, and F. Jülicher, Proc. Natl. Acad. Sci. USA 98, 14380 (2001).
  * Mizuno et al. (2007) D. Mizuno, C. Tardin, C. F. Schmidt, and F. C. MacKintosh, Science 315, 370 (2007).
  * Dinis et al. (2012) L. Dinis, P. Martin, J. Barral, J. Prost, and J. Joanny, Phys. Rev. Lett. 109, 160602 (2012).
  * Willareth et al. (2017) L. Willareth, I. Sokolov, Y. Roichman, and B. Lindner, EPL 118, 20001 (2017).
  * Marconi et al. (2008) U. M. B. Marconi, A. Puglisi, L. Rondoni, and A. Vulpiani, Phys. Rep. 461, 111 (2008).
  * Seifert (2012) U. Seifert, Rep. Prog. Phys. 75, 126001 (2012).
  * Tuckwell (1989) H. C. Tuckwell, _Stochastic Processes in the Neuroscience_ (SIAM, Philadelphia, Pennsylvania, 1989).
  * Gerstner et al. (2014) W. Gerstner, W. M. Kistler, R. Naud, and L. Paninski, _Neuronal Dynamics From single neurons to networks and models of cognition_ (Cambridge University Press, Cambridge, 2014).
  * Vilela and Lindner (2009) R. D. Vilela and B. Lindner, Phys. Rev. E. 80, 031909 (2009).
  * Lindner (2002) B. Lindner, _Coherence and Stochastic Resonance in Nonlinear Dynamical Systems_ (Logos-Verlag, Berlin, 2002).
  * not (a) The firing rate of the LIF model with white noise is given by $r_{0}^{-1}=\sqrt{\pi}\int_{z_{T}/\sqrt{2}}^{z_{R}/\sqrt{2}}e^{z^{2}}\operatorname{erfc}(z)dz$, see e.g. L. M. Ricciardi _Diffusion Processes and Related Topics on Biology_ Springer-Verlag (1977).
  * Brunel et al. (2001) N. Brunel, F. S. Chance, N. Fourcaud, and L. F. Abbott, Phys. Rev. Lett. 86, 2186 (2001).
  * Lindner and Schimansky-Geier (2001) B. Lindner and L. Schimansky-Geier, Phys. Rev. Lett. 86, 2934 (2001).
  * Richardson and Swarbrick (2010) M. J. E. Richardson and R. Swarbrick, Phys. Rev. Lett. 105, 178102 (2010).
  * Droste and Lindner (2017a) F. Droste and B. Lindner, J. Comp. Neurosci. 43, 81 (2017a).
  * Droste and Lindner (2017b) F. Droste and B. Lindner, Phys. Rev. E 95, 012411 (2017b).
  * Deger et al. (2010) M. Deger, T. Schwalger, R. Naud, and W. Gerstner, Phys. Rev. E. 82, 021129 (2010).
  * not (b) In the standard approach, the response to a step perturbation is expressed by the linear response on the one hand and by a Taylor expansion of the steady state under constant perturbation on the other hand.
  * Cessac et al. (2021) B. Cessac, I. Ampuero, and R. Cofré, Entropy 23, 155 (2021).
  * Laing and Chow (2001) C. R. Laing and C. C. Chow, Neural Comp. 13, 1473 (2001).
  * Meffin et al. (2001) H. Meffin, A. N. Burkitt, and D. B. Grayden, J. Comp. Neurosci 16, 159 (2001).
  * Furutsu (1963) K. Furutsu, Res. Natl Bur. Stand. D 67, 303 (1963).
  * Novikov (1965) E. A. Novikov, Sov. Phys. Jetp-USSR 20, 1290 (1965).
  * Fourcaud-Trocmé et al. (2003) N. Fourcaud-Trocmé, D. Hansel, C. van Vreeswijk, and N. Brunel, J. Neurosci. 23, 11628 (2003).
  * Badel et al. (2008) L. Badel, S. Lefort, R. Brette, C. C. H. Petersen, W. Gerstner, and M. J. E. Richardson, J. Neurophysiol. 99, 656 (2008).
  * Brette and Gerstner (2005) R. Brette and W. Gerstner, J. Neurophysiol. 94, 3637 (2005).
  * Benda and Herz (2003) J. Benda and A. V. M. Herz, Neural Comput. 15, 2523 (2003).
  * Schwalger (2013) T. Schwalger, Ph.D. thesis, Humboldt-Universität zu Berlin (2013).
  * Izhikevich (2007) E. M. Izhikevich, _Dynamical Systems in Neuroscience: The Geometry of Excitability and Bursting_ (The MIT Press, Cambridge, London, 2007).
  * Richardson et al. (2003) M. J. E. Richardson, N. Brunel, and V. Hakim, J. Neurophysiol. 89, 2538 (2003).
  * Lindner (2016) B. Lindner, IEEE Trans. Mol. Biol. Multi-Scale Commun. 2, 5 (2016).
  * Brunel (2000) N. Brunel, J. Comput. Neurosci. 8, 183 (2000).
  * Knoll and Lindner (2022) G. Knoll and B. Lindner, Phys. Rev. E (2022), accepted.


# Towards Contrastive Learning in Music Video Domain

Karel Veldkamp, Mariya Hendriksen, Zoltán Szlávik, Alexander Keijser

(2023)

###### Abstract.

Contrastive learning is a powerful way of learning multimodal representations
across various domains such as image-caption retrieval and audio-visual
representation learning. In this work, we investigate if these findings
generalize to the domain of music videos. Specifically, we create a dual
encoder for the audio and video modalities and train it using a bidirectional
contrastive loss. For the experiments, we use an industry dataset containing
550 000 music videos as well as the public Million Song Dataset, and evaluate
the quality of learned representations on the downstream tasks of music
tagging and genre classification. Our results indicate that pre-trained
networks without contrastive fine-tuning outperform our contrastive learning
approach when evaluated on both tasks. To gain a better understanding of the
reasons contrastive learning was not successful for music videos, we perform a
qualitative analysis of the learned representations, revealing why contrastive
learning might have difficulties uniting embeddings from two modalities. Based
on these findings, we outline possible directions for future work. To
facilitate the reproducibility of our results, we share our code and the pre-
trained model.

Music video recommendation, contrastive learning

††copyright: acmcopyright††journalyear: 2023

##  1\. Introduction

One of the most important challenges in music video recommendation is that
music videos are most relevant shortly after their release, resulting in the
need for an approach that works well for new items. A class of methods that
are known to work well for new items are content-based recommender (CBR)
systems. However, these methods rely on the quality of item representations
(Volkovs et al., 2017; Zhu et al., 2019). In the use case of music videos,
these representations should contain information from both the audio and video
domains. In contrast to music representation learning (Kim et al., 2020), the
topic of learning music video representations is relatively unexplored.

One method for multimodal representation learning that has been proven to be
powerful across various domains is contrastive learning (CL)(Chai and Wang,
2022; Shi et al., 2023). This class of methods uses pairs of data points from
different modalities to learn multimodal representations, thereby avoiding the
need for labels. The most renowned application of this methodology was in the
text-image domain(Radford et al., 2021; Hendriksen, 2022), but it has also
been applied in the audio-video domain (Wang et al., 2021; Ma et al., 2020).
In this paper, we examine whether CL can successfully be generalized to music
videos.

This research was carried out in collaboration with a music video streaming
platform. The platform provided a private dataset of 555 000 music videos.
Although this company dataset provides us with the unique opportunity to test
state-of-the-art (SOTA) methods on a large collection of real music videos, it
also means that we can not share our training data publicly, since it largely
consists of copyrighted music videos. However, to keep this research as
reproducible as possible, we evaluate our trained model on a subsection of the
publicly available Million Song Dataset (MSD) that overlaps with the company
dataset. Additionally, our code and trained models are made publicly
available, along with a description of the company dataset and a comparison to
the public MSD.

Research questions. We devise a method that learns audio-visual
representations for music videos based on their content using CL. We aim to
answer the following research questions:  (RQ1) Can CL of audio-video
representations be used to improve the quality of music video representations
relative to existing models when evaluated on the downstream tasks of genre
classification and music tagging?  (RQ2) Can our audio-visual representation
be used to calculate music-video similarity that is in line with the
subjective human judgment of similarity?

Main findings. We discover that our training procedure was not able to pull
embeddings for corresponding pairs of audio and video together. Consequently,
the contrastive model is outperformed by the baselines on the downstream
tasks, indicating that our approach was not able to generalize contrastive
learning to the music video setting. Additionally, we find that human judgment
is not in line with the similarity of representations.

To better understand these outcomes, we perform a qualitative analysis of the
learned representations. Specifically, we perform an error analysis by looking
at similarities and dissimilarities of music-video clips that were retrieved
based on various seed items. This reveals a potential explanation for the
failure of the contrastive loss function to pull embeddings from the two
modalities together.

Contributions. The principal contributions of our research are the following:
(1) We adopt a CL approach to learn music video representations for
recommendations and evaluate its ability to generalize to the music video
domain.  (2) We provide a quantitative evaluation of the learned
representations on two downstream tasks of genre classification and music
tagging.  (3) A qualitative evaluation of the representations based on item
similarity.  (4) To facilitate the reproducibility of our work, we provide the
code of our experiments, a pre-trained model, and a description of the company
dataset on GitHub111https://github.com/KarelVeldkamp/Multimodal-Musicvideo-
Representation.

##  2\. Related work

Multimodal learning. The main challenge for multimodal learning is that
feature vectors for different modalities encode different types of
information, which is referred to as the heterogeneity gap(Guo et al., 2019;
Hendriksen et al., 2023). A traditional approach to deal with this problem is
to project the different modalities into a single subspace while retaining
information from the different modalities(Baltrušaitis et al., 2018). This is
often done in a supervised (Jiang et al., 2017; Nojavanasghari et al., 2016;
Ortega et al., 2019) or unsupervised (Ngiam et al., 2011; Silberer and Lapata,
2014; Goei et al., 2021) manner. One recent approach gaining traction is
CL(Chai and Wang, 2022).

Contrastive learning. CL methods learn from pairs of samples by pulling
representations closer together for positive pairs of samples(Le-Khac et al.,
2020). In a unimodal setting pairs are created using augmentation(Chen et al.,
2020; Spijkervet and Burgoyne, 2021; Lorre et al., 2020; Shi and Lipani,
2023), whereas in a multimodal setting, positive pairs are matching samples
from different modalities. A renowned study in this field introduced
Contrastive Language-Image Pretraining (CLIP)(Radford et al., 2021). CLIP
learned a multimodal embedding space for images and text using CL on a large
dataset of image-caption pairs. The model attains performance on a variety of
vision-language tasks(Hendriksen and Overes, 2022). The approach works by
using separate modality-specific encoder networks, which are initialized using
a pre-trained network for the relevant modality. A projection head is added on
top, which consists of extra feedforward layers with the same output
dimensions across modalities. The network is trained contrastively, by
maximizing the similarity of the output layers for image-text pairs that
belong together and minimizing similarity for pairs that do not. This
seemingly simple training task results in informative representations of the
input data, evident from the high zero-shot performance on various tasks.
Several studies have been published using variations of this method including
both the audio and video domains(Guzhov et al., 2021; Wang et al., 2021; Ma et
al., 2020; Akbari et al., 2021; Hendriksen et al., 2022).

Although earlier research has shown the potential of CL in the audio-video
domain, it has not previously been applied for music video domain. Prior work
mainly focused on the kinetics human action dataset(Kay et al., 2017), the
AudioSet dataset(Gemmeke et al., 2017), and the HowTo100M dataset(Miech et
al., 2019). The relationship between a song and its video is more abstract and
more challenging to learn.

##  3\. Approach

Figure 1. Model architecture, comprised of pre-trained convolutional layers of
the _musicnn_ and R(2+1)D networks, and a dense projection head for each
modality. We freeze the convolutional layers and train the projection heads.

Figure1 presents an overview of our proposed architecture.

Audio encoding pipeline. Audio is encoded using the encoder network $f_{a}$
and a projection head $g_{a}$. We use the pre-trained _musicnn_ network as
encoder(Pons and Serra, 2019), freezing the weights during training. The
projection head consists of two dense feed-forward layers. We use a hidden
layer size of 512 and an embedding size of 256. These embedding sizes are
relatively small compared to some other multimodal networks(Radford et al.,
2021; Faghri et al., 2017) because we have relatively little training data.
The hidden layer uses a ReLU activation function and is trained using a
dropout rate of 0.3. The embedding layer uses a sigmoid activation function.

Video encoding pipeline. as video encoder $f_{v}$, we use the (2+1)D
convolutional neural network (CNN) fromTran et al. (2018). This encoder is
initialized with the pre-trained weights and is frozen during training. We use
the same projection head architecture for the video encoder as we did for the
audio encoder. We also experiment with several variations to the architecture
of our two projection heads, which are discussed in Section4.

Loss Function. To train the dense layers of the two projection heads, we use
the contrastive loss function from simCLR(Chen et al., 2020), adapted to the
audio-video scenario. For a batch of $N$ music videos, video-to-audio and
audio-to-video losses for a positive pair of audio embedding $a_{j}$ and video
embedding $v_{j}$ are defined respectively as:

(1) $l^{(v\rightarrow
a)}_{j}=-log\frac{\exp(f_{sim}(a_{j},v_{j})/\tau)}{\sum_{k=1}^{N}\mathbbm{1}_{[k\neq
j]}\exp(f_{sim}(a_{j},v_{k})/\tau)}.$ (2) $l^{(a\rightarrow
v)}_{j}=-log\frac{\exp(f_{sim}(v_{j},a_{j})/\tau)}{\sum_{k=1}^{N}\mathbbm{1}_{[k\neq
j]}\exp(f_{sim}(v_{j},a_{k})/\tau)}.$

Here $f_{sim}(x,y)$ is the cosine similarity function:
$f_{sim}(\mathbf{q},\mathbf{x})=\frac{\mathbf{q}}{||\mathbf{q}||}\frac{\mathbf{x}}{||\mathbf{x}||}$;
and $\mathbbm{1}$ is an indicator function. Finally, $\tau$ is a temperature
parameter that controls the strength of penalties on hard negative samples.
Prior research demonstrated that setting a low temperature parameter helps to
learn separable features, although setting it too low will result in
penalizing semantically similar negative samples too heavily(Wang and Liu,
2021). We experiment with different values for $\tau$, as detailed in
Section4. The overall loss is bidirectional, combining the values of the two
aforementioned loss functions:

(3) $\mathcal{L}=\frac{1}{\beta}\sum^{\beta}_{j=1}\Big{(}\ell^{(a\rightarrow
v)}_{j}+\ell^{(v\rightarrow a)}_{j}\Big{)}.$

##  4\. Experimental Setup

Datasets. The main dataset used is a company dataset which consists of a
collection of more than 90,000 music videos published on the music video
platform that we collaborate with. The second dataset is the MSD(Bertin-
Mahieux et al., 2011). This dataset is a benchmark dataset on the task of
music tagging. Note that the company dataset contains music videos, whereas
the MSD is audio only. For a detailed description and comparison of the two
datasets, we refer to the GitHub repository.

Evaluation method. The two main components of our work are training the model,
and evaluating the representations on downstream tasks. First, we train our
model on the company dataset and experiment with several different
configurations of the hyperparameters and architecture in this phase. Then, we
use the representations that we learned in the first step and use them on the
downstream tasks of genre classification and music tagging. This allows us to
compare performance using the CL representations to baselines from the
literature.

Metrics. FollowingZolfaghari et al. (2021), we use the median rank of cross-
modal retrieval to evaluate our embeddings. We report both the audio-to-video
median rank and the video-to-audio median rank. To evaluate performance on the
downstream tasks of genre classification and music tagging, we use area under
the curve (AUC), in line with prior workWon et al. (2020). Additionally, we
report $F_{1}$ score for downstream task performance.

Baselines. When evaluating the CL representations we use the pre-trained
backbone models as baselines. For the audio modality, we use _musicnn_(Pons
and Serra, 2019) as a baseline, and for the video modality, we use
_R(2+1)D_(Tran et al., 2018). This allows us to compare our representations to
representations that were not fine-tuned using CL.

Experiments. We run 3 experiments. In Experiment 1 we use CL to learn
representations for the dataset of music video segments. We start with a
baseline configuration of our model, and experiment with several variations to
this configuration:  (i) $CL_{BASE}$: A 2-layer projection head with hidden
layer size 512 and embedding size 256. We set temperature parameter $\tau$ to
one.  (ii) We increase the embedding size to 512.  (iii) We increase the depth
of the projection head to four layers.  (iv) We use a single projection head
for the video modality, which projects the video embeddings to the dimensions
of the _musicnn_ embeddings. This moves video representations closer to the
audio, rather than moving both closer to each other.  (v) We lower the
temperature parameter to .3, as suggested byWang and Liu (2021). This
penalizes hard negatives more heavily, which helps learn separable embeddings.
(vi) We train autoencoders to initialize the weights of the projection heads
for the two modalities. Previous research has shown that this can improve
performance on several deep learning tasks(Ferreira et al., 2020; Paine et
al., 2014).

In Experiment 2, we evaluate the representations on the downstream task of
genre classification. The most used dataset in this field is the GTZAN dataset
(Tzanetakis and Cook, 2002), but research has pointed out that the quality of
the labels in this dataset is poor (Pálmason et al., 2017). This lack of a
good common dataset makes it hard to compare different genre detection models
and to determine what methods are the SOTA(Ramírez and Flores, 2020). In our
study, we perform genre classification on the company dataset. This allows us
to test our model in a real practical context, and additionally, it allows us
to study the effect of the video modality on classification. We fit three
different models to predict genre based on our learned representations:  (i)
$CL_{a}$: Using the contrastive audio embeddings.  (ii) $CL_{v}$: Using the
contrastive video embeddings.  (iii) $CL_{agg}$: Using the aggregated audio
and video embeddings.  To evaluate the effectiveness of our learned
representations, we compare these models to three baseline models, which do
classification based on the pre-trained embeddings:  (i) _Musicnn_ : Using the
_musicnn_ embeddings (Pons and Serra, 2019).  (ii) _R(2+1)D_ : Using the
_R(2+1)D CNN_ embeddings (Tran et al., 2018).  (iii) _Musicnn+R(2+1)D_ : Using
the concatenated _musicnn_ and _R(2+1)D CNN_ embeddings.  All models are
three-layer perceptrons, with hidden layers sizes of 512 and 256 respectively.

In Experiment 3, we evaluate the embeddings on the downstream task of music
tagging. We use the same baselines and models as for genre classification.
However, to facilitate reproducibility, we use a subset of the public
MSD(Bertin-Mahieux et al., 2011) that overlaps with our company dataset. The
comprehensive list of the 9416 MSD IDs of the overlapping tracks is available
on the GitHub repository referenced above. The tagging task consists of
predicting several tags that were manually annotated based on the audio. These
tags contain genres (e.g. “blues”), eras (e.g. “80s”), and moods
(e.g.“happy”)(Pons and Serra, 2019; Choi et al., 2016). Since our training set
consists of just 9416 videos, we only use the top ten most common tags.

Implementation Details. In order to run our experiments in relatively quick
succession, we stored just six five-second non-overlap-
ping segments per music video, rather than the entire music video. We sampled
the segments by dividing the track up into six equally sized sections and
sampling a random five-second segment from each section, preventing overlap
between the segments, this procedure is represented schematically in Figure2.

Figure 2. Schematic representation of the sampling procedure.

Consecutive video frames have high redundancy (Wang et al., 2016). So in line
with previous work (Simonyan and Zisserman, 2014; Carreira and Zisserman,
2017) we choose to sample a random set of frames with equal temporal spacing
between them. We save 50 frames at a resolution of 768x432. The corresponding
audio was saved at a sampling rate of 16,000. Data was split up into a
training set of 80 percent, and a validation and testing set of 10 percent
each.

Separate segments of the same music video were constrained not to appear in
the same batch. In We applied random cropping to 112x112 in order to get the
correct input shape for the network, using resizing to prevent the cropped
videos to contain an unreasonably small fraction of the pixels in the original
video, which is a common data augmentation in visual deep learning (Takahashi
et al., 2018).

We use a batch size of 1000, with an initial learning rate of .01, and
exponential learning decay with a gamma parameter of .95, following Li and
Arora (2019). We implement an early stopping rule, stopping when the
validation loss does not decrease over a period of three epochs.

##  5\. Results

Experiment 1. In the first experiment, we explore different configurations of
our approach. Table1 presents the results of the experiment. The median rank
does not change across different selected configurations.

Table 1. Results of Experiment 1. The best performance is highlighted in bold. Configuration | $MR_{a\rightarrow v}$ | $MR_{v\rightarrow a}$
---|---|---
$CL_{BASE}$ | 507 | 502
Embedding size 512 | 494 | 498
Four projection layers | 503 | 501
Single projection head | 499 | 497
$\tau=0.3$ | 497 | 503
Autoencoder initiated | 501 | 500

Experiment 2 and 3. Table2 shows the average area under the curve and F1 score
for each baseline and model on the downstream task of music tagging. These
results present two main findings. First, the networks that are trained on
embeddings that include information about audio clearly outperform models that
are only based on a video. Second, the performance of the baseline models is
better than the performance of the models that were fine-tuned using CL. Given
the failure to unite embeddings from the different modalities, these results
are less surprising.

The results for the downstream task of genre classification are similar to the
results for music tagging. Table2 presents the same two statistics for the
task of genre classification. Similar to the results above, we see thfat
models including audio data outperform models that are solely based on a
video, and more importantly, that all three baselines outperform their CL
counterparts.

Table 2. Results of Experiments 2 and 3. The best performance is highlighted in bold. Model | Music tagging | Genre classification
---|---|---
AUC | $F_{1}$ | AUC | $F_{1}$
Musicnn | 0.78 | 0.22 | 0.79 | 0.18
R(2+1)D | 0.66 | 0.09 | 0.68 | 0.05
Musicnn+R(2+1)D | 0.78 | 0.24 | 0.82 | 0.21
$CL_{a}$ | 0.70 | 0.10 | 0.69 | 0.08
$CL_{v}$ | 0.61 | 0.09 | 0.63 | 0.04
$CL_{agg}$ | 0.71 | 0.11 | 0.68 | 0.07

##  6\. Qualitative analysis of representations

To better understand our results, we analyzed our representations
qualitatively on the level of entire music videos as well as individual
samples.

Music videos. We selected 25 random seed videos and retrieved the three most
similar music videos to each seed, based on the embeddings aggregated over the
audio and video modalities. In general, the most similar music videos did not
appear to be similar to the seed video at all in terms of both audio and
video. One exception to this rule was live performances, which are often most
similar to other live performances. When taking into account only the video
modality, rather than the aggregate over modalities, some other patterns were
also present: Some seed videos that were all black and white, as well as some
videos of people dancing or playing instruments, would result in similar
videos being retrieved. One thing that these exceptions have in common, is
that they are very consistent in terms of video across different parts of the
music video. This could point to the fact that aggregating representations of
different segments of the same music video is not a valid way to get a
representation of the music video, especially when different parts of the
music video are very inconsistent. This is particularly often the case for the
video modality, as music videos often consist of a wide variety of shots and
scenes.

Segments. We also examined similarity on the level of individual segments. We
took the same approach as before, this time selecting 20 random seed segments,
and retrieving the three most similar segments to this seed segment. As
expected, the results were better for single segments. There were more cases
in which the retrieved videos were similar to the seed, although in general
the retrieved videos were still not similar. For example, some seeds for which
the retrieved segments were similar were close-ups of women singing, shots of
people playing guitar, segments with a dark background with bright neon
accents, and shots of men talking. Overall similar segment retrieval was poor,
but there were more exceptions in which there was some specific similarity
between the videos.

##  7\. Discussion and limitations

Contrastive learning of audiovisual representations was not able to generalize
to the field of music videos. The contrastive audiovisual representations did
not improve performance on either of the downstream tasks, answering RQ1.
Additionally, with respect to RQ2, results indicated that our representations
can not be used to infer music video similarity. In this section, we discuss
some of the potential reasons for these issues, as well as some limitations to
our approach.

Discussion. The main challenge for multimodal learning consists of bridging
the heterogeneity gap(Guo et al., 2019). CL is one approach to this problem,
and some notable applications involve CL for images and their captions(Radford
et al., 2021), videos and their descriptions(Zolfaghari et al., 2021) and
videos and their audio streams (Wang et al., 2021). Compared to our use case,
the heterogeneity gap in these studies is small. An image and its caption are
closely related, as the caption often describes the image. The two modalities
both refer to the same concept. This is also true for videos and their
descriptions, or videos of an event and the sound it produces. However, the
relationship between music and its video is less direct. As opposed to the
other examples, the two modalities are not both directly referring to a single
concept. Rather than describing the music, the video modality serves as an
extra means to entertain the user and convey emotion. This large heterogeneity
gap is reflected in our qualitative analysis of similarity: The music network
encodes for features relating to the types of instruments used, and the video
network encodes for features like the color of the video as well as specific
actions like singing or dancing. The relationship between these features is
not always immediately apparent. Black and white videos vary greatly in the
type of instruments used, and the fact that a music video contains people
dancing, does not tell much about the instruments played. This is different
from the video-audio use case from previous research, where different videos
of similar events are likely to have similar audio streams since the event in
the video is often the source of the audio (Wang et al., 2021). We believe
that the fact that the two modalities in music videos are less tightly
connected is an important cause for the failure to generalize contrastive
learning to the music-video domain.

Limitations. One limitation of our approach concerns the input size of the
encoder networks. Due to the high dimensionality and complexity of the input
data, we were limited to training on segments of only three seconds. This
makes the task of retrieving corresponding segments from the other modality
significantly harder. Even for a human, retrieving the corresponding piece of
music based on a three-second video clip is difficult. Additionally, this
short input length makes it harder to compute the similarity between complete
music videos, which is necessary for CBR. We tried aggregating representations
of segments to come to a single representation per music video, but our
qualitative analysis of similarity revealed that this aggregation degraded the
performance, especially so when the input data was inconsistent across
segments, which is usually the case for video data from music videos. It might
be more fruitful to use a larger input size or to extract features that are
more consistent across the duration of the music video.

##  8\. Conclusions

In this work, we examined several CL bi-encoder architectures and explored
their generalizability towards the music video domain. We discovered that the
models struggled with A contrastive deep learning model was not able to unite
embeddings from the audio and video modalities in music videos. These results
demonstrate that aligning video and audio data is an open and challenging
problem.

Given the large heterogeneity gap, future research should look into multimodal
alternatives to CL that rely less heavily on the association between features
in different modalities. Multimodal fusion might for example be more suitable,
as these models combine features from different modalities based on a
supervised task, rather than solely based on the relationship between features
in the different modalities(Nojavanasghari et al., 2016).

Additionally, future research could focus on using encoder networks that learn
features that are more consistent over time. Most current research on video
processing is based on short video clips of single events, faces, or poses
(Ren et al., 2019; Santos et al., 2019). However, ideally, the video encoder
would learn features relating to the style of the video. Potentially, models
concerned with style transfer for video data (Chen et al., 2017; Li et al.,
2019) could learn features that are more closely related to the audio of the
song.

## References

  * (1)
  * Akbari et al. (2021) Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. 2021.  Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text.  _Advances in Neural Information Processing Systems_ 34 (2021).
  * Baltrušaitis et al. (2018) Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2018.  Multimodal machine learning: A survey and taxonomy.  _IEEE transactions on pattern analysis and machine intelligence_ 41, 2 (2018), 423–443.
  * Bertin-Mahieux et al. (2011) Thierry Bertin-Mahieux, Daniel PW Ellis, Brian Whitman, and Paul Lamere. 2011.  The million song dataset.  (2011).
  * Carreira and Zisserman (2017) Joao Carreira and Andrew Zisserman. 2017.  Quo vadis, action recognition? a new model and the kinetics dataset. In _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 6299–6308.
  * Chai and Wang (2022) Wenhao Chai and Gaoang Wang. 2022.  Deep Vision Multimodal Learning: Methodology, Benchmark, and Trend.  _Applied Sciences_ 12, 13 (2022), 6588.
  * Chen et al. (2017) Dongdong Chen, Jing Liao, Lu Yuan, Nenghai Yu, and Gang Hua. 2017.  Coherent online video style transfer. In _Proceedings of the IEEE International Conference on Computer Vision_. 1105–1114.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020.  A simple framework for contrastive learning of visual representations. In _International conference on machine learning_. PMLR, 1597–1607.
  * Choi et al. (2016) Keunwoo Choi, George Fazekas, and Mark Sandler. 2016\.  Automatic tagging using deep convolutional neural networks.  _arXiv preprint arXiv:1606.00298_ (2016).
  * Faghri et al. (2017) Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2017.  Vse++: Improving visual-semantic embeddings with hard negatives.  _arXiv preprint arXiv:1707.05612_ (2017).
  * Ferreira et al. (2020) Mafalda Falcao Ferreira, Rui Camacho, and Luis F Teixeira. 2020\.  Autoencoders as weight initialization of deep classification networks for cancer versus cancer studies.  _arXiv preprint arXiv:2001.05253_ (2020).
  * Gemmeke et al. (2017) Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. 2017\.  Audio set: An ontology and human-labeled dataset for audio events. In _2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 776–780.
  * Goei et al. (2021) Kenneth Goei, Mariya Hendriksen, and Maarten de Rijke. 2021\.  Tackling attribute fine-grainedness in cross-modal fashion search with multi-level features. In _SIGIR 2021 Workshop on eCommerce, ACM_.
  * Guo et al. (2019) Wenzhong Guo, Jianwen Wang, and Shiping Wang. 2019\.  Deep multimodal representation learning: A survey.  _IEEE Access_ 7 (2019), 63373–63394.
  * Guzhov et al. (2021) Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. 2021.  Audioclip: Extending clip to image, text and audio.  _arXiv preprint arXiv:2106.13043_ (2021).
  * Hendriksen (2022) Mariya Hendriksen. 2022\.  Multimodal Retrieval in E-Commerce: From Categories to Images, Text, and Back. In _European Conference on Information Retrieval_. Springer, 505–512.
  * Hendriksen et al. (2022) Mariya Hendriksen, Maurits Bleeker, Svitlana Vakulenko, Nanne van Noord, Ernst Kuiper, and Maarten de Rijke. 2022.  Extending CLIP for Category-to-image Retrieval in E-commerce. In _European Conference on Information Retrieval_. Springer, 289–303.
  * Hendriksen and Overes (2022) Mariya Hendriksen and Viggo Overes. 2022.  Unimodal vs. Multimodal Siamese Networks for Outfit Completion.  _arXiv preprint arXiv:2207.10355_ (2022).
  * Hendriksen et al. (2023) Mariya Hendriksen, Svitlana Vakulenko, Ernst Kuiper, and Maarten de Rijke. 2023.  Scene-Centric vs. Object-Centric Image-Text Cross-Modal Retrieval: A Reproducibility Study. In _European Conference on Information Retrieval_. Springer, 68–85.
  * Jiang et al. (2017) Yu-Gang Jiang, Zuxuan Wu, Jun Wang, Xiangyang Xue, and Shih-Fu Chang. 2017.  Exploiting feature and class relationships in video categorization with regularized deep neural networks.  _IEEE transactions on pattern analysis and machine intelligence_ 40, 2 (2017), 352–364.
  * Kay et al. (2017) Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. 2017\.  The kinetics human action video dataset.  _arXiv preprint arXiv:1705.06950_ (2017).
  * Kim et al. (2020) Jaehun Kim, Julián Urbano, Cynthia Liem, and Alan Hanjalic. 2020.  One deep music representation to rule them all? A comparative analysis of different representation learning strategies.  _Neural Computing and Applications_ 32, 4 (2020), 1067–1093.
  * Le-Khac et al. (2020) Phuc H Le-Khac, Graham Healy, and Alan F Smeaton. 2020\.  Contrastive representation learning: A framework and review.  _IEEE Access_ 8 (2020), 193907–193934.
  * Li et al. (2019) Xueting Li, Sifei Liu, Jan Kautz, and Ming-Hsuan Yang. 2019\.  Learning linear transformations for fast image and video style transfer. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 3809–3817.
  * Li and Arora (2019) Zhiyuan Li and Sanjeev Arora. 2019.  An exponential learning rate schedule for deep learning.  _arXiv preprint arXiv:1910.07454_ (2019).
  * Lorre et al. (2020) Guillaume Lorre, Jaonary Rabarisoa, Astrid Orcesi, Samia Ainouz, and Stephane Canu. 2020.  Temporal contrastive pretraining for video action recognition. In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_. 662–670.
  * Ma et al. (2020) Shuang Ma, Zhaoyang Zeng, Daniel McDuff, and Yale Song. 2020\.  Active contrastive learning of audio-visual video representations.  _arXiv preprint arXiv:2009.09805_ (2020).
  * Miech et al. (2019) Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019.  Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 2630–2640.
  * Ngiam et al. (2011) Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. 2011\.  Multimodal deep learning. In _ICML_.
  * Nojavanasghari et al. (2016) Behnaz Nojavanasghari, Deepak Gopinath, Jayanth Koushik, Tadas Baltrušaitis, and Louis-Philippe Morency. 2016.  Deep multimodal fusion for persuasiveness prediction. In _Proceedings of the 18th ACM International Conference on Multimodal Interaction_. 284–288.
  * Ortega et al. (2019) Juan DS Ortega, Mohammed Senoussaoui, Eric Granger, Marco Pedersoli, Patrick Cardinal, and Alessandro L Koerich. 2019.  Multimodal fusion with deep neural networks for audio-video emotion recognition.  _arXiv preprint arXiv:1907.03196_ (2019).
  * Paine et al. (2014) Tom Le Paine, Pooya Khorrami, Wei Han, and Thomas S Huang. 2014.  An analysis of unsupervised pre-training in light of recent advances.  _arXiv preprint arXiv:1412.6597_ (2014).
  * Pálmason et al. (2017) Haukur Pálmason, Björn Þór Jónsson, Markus Schedl, and Peter Knees. 2017\.  Music genre classification revisited: An in-depth examination guided by music experts. In _International Symposium on Computer Music Multidisciplinary Research_. Springer, 49–62.
  * Pons and Serra (2019) Jordi Pons and Xavier Serra. 2019.  musicnn: Pre-trained convolutional neural networks for music audio tagging.  _arXiv preprint arXiv:1909.06654_ (2019).
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International Conference on Machine Learning_. PMLR, 8748–8763.
  * Ramírez and Flores (2020) Jaime Ramírez and M Julia Flores. 2020.  Machine learning for music genre: multifaceted review and experimentation with audioset.  _Journal of Intelligent Information Systems_ 55, 3 (2020), 469–499.
  * Ren et al. (2019) Qiuyu Ren, Liang Bai, Haoran Wang, Zhihong Deng, Xiaoming Zhu, Han Li, and Can Luo. 2019.  A Survey on Video Classification Methods Based on Deep Learning.  _DEStech Transactions on Computer Science and Engineering_ cisnrc (2019).
  * Santos et al. (2019) Gabriel NP dos Santos, Pedro VA de Freitas, Antonio José G Busson, Álan LV Guedes, Ruy Milidiú, and Sérgio Colcher. 2019\.  Deep learning methods for video understanding. In _Proceedings of the 25th Brazillian Symposium on Multimedia and the Web_. 21–23.
  * Shi and Lipani (2023) Zhengxiang Shi and Aldo Lipani. 2023.  Rethink the Effectiveness of Text Data Augmentation: An Empirical Analysis.  _arXiv preprint arXiv:2306.07664_ (2023).
  * Shi et al. (2023) Zhengxiang Shi, Xi Wang, and Aldo Lipani. 2023.  Self contrastive learning for session-based recommendation.  _arXiv preprint arXiv:2306.01266_ (2023).
  * Silberer and Lapata (2014) Carina Silberer and Mirella Lapata. 2014.  Learning grounded meaning representations with autoencoders. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 721–732.
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Two-stream convolutional networks for action recognition in videos.  _Advances in neural information processing systems_ 27 (2014).
  * Spijkervet and Burgoyne (2021) Janne Spijkervet and John Ashley Burgoyne. 2021.  Contrastive learning of musical representations.  _arXiv preprint arXiv:2103.09410_ (2021).
  * Takahashi et al. (2018) Ryo Takahashi, Takashi Matsubara, and Kuniaki Uehara. 2018\.  Ricap: Random image cropping and patching data augmentation for deep cnns. In _Asian conference on machine learning_. PMLR, 786–798.
  * Tran et al. (2018) Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. 2018\.  A closer look at spatiotemporal convolutions for action recognition. In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_. 6450–6459.
  * Tzanetakis and Cook (2002) George Tzanetakis and Perry Cook. 2002.  Musical genre classification of audio signals.  _IEEE Transactions on speech and audio processing_ 10, 5 (2002), 293–302.
  * Volkovs et al. (2017) Maksims Volkovs, Guang Wei Yu, and Tomi Poutanen. 2017.  Content-based neighbor models for cold start in recommender systems.  In _Proceedings of the Recommender Systems Challenge 2017_. 1–6.
  * Wang and Liu (2021) Feng Wang and Huaping Liu. 2021.  Understanding the behaviour of contrastive loss. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 2495–2504.
  * Wang et al. (2021) Luyu Wang, Pauline Luc, Adria Recasens, Jean-Baptiste Alayrac, and Aaron van den Oord. 2021.  Multimodal self-supervised learning of general audio representations.  _arXiv preprint arXiv:2104.12807_ (2021).
  * Wang et al. (2016) Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. 2016.  Temporal segment networks: Towards good practices for deep action recognition. In _European conference on computer vision_. Springer, 20–36.
  * Won et al. (2020) Minz Won, Andres Ferraro, Dmitry Bogdanov, and Xavier Serra. 2020\.  Evaluation of CNN-based automatic music tagging models.  _arXiv preprint arXiv:2006.00751_ (2020).
  * Zhu et al. (2019) Yu Zhu, Jinghao Lin, Shibi He, Beidou Wang, Ziyu Guan, Haifeng Liu, and Deng Cai. 2019.  Addressing the item cold-start problem by attribute-driven active learning.  _IEEE Transactions on Knowledge and Data Engineering_ 32, 4 (2019), 631–644.
  * Zolfaghari et al. (2021) Mohammadreza Zolfaghari, Yi Zhu, Peter Gehler, and Thomas Brox. 2021.  Crossclr: Cross-modal contrastive learning for multi-modal video representations. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 1450–1459.

  *[CBR]: content-based recommender
  *[CL]: contrastive learning
  *[SOTA]: state-of-the-art
  *[MSD]: Million Song Dataset
  *[CLIP]: Contrastive Language-Image Pretraining
  *[CNN]: convolutional neural network
  *[AUC]: area under the curve