# Correlating $W$-Boson Mass Shift with Muon ${g-2}$ in the 2HDM

K.S. Babu<EMAIL_ADDRESS>Department of Physics, Oklahoma State University,
Stillwater, OK, 74078, USA  Sudip Jana<EMAIL_ADDRESS>Max-Planck-
Institut für Kernphysik, Saupfercheckweg 1, 69117 Heidelberg, Germany  Vishnu
P.K<EMAIL_ADDRESS>Department of Physics, Oklahoma State University,
Stillwater, OK, 74078, USA

###### Abstract

We show an interesting correlation between the recent high precision
measurement of the $W$-boson mass by the CDF collaboration and the muon
$(g-2)$ anomaly in the context of the two Higgs doublet model. One-loop
diagrams involving the exchange of neutral scalar bosons can explain the muon
$(g-2)$, which however requires significant mass splittings among members of
the second Higgs doublet. These splittings also generate a positive shift in
the mass of the $W$-boson, consistent with the recent CDF measurement. The
charged and neutral scalars of the model cannot be heavier than about 600 GeV
for a simultaneous explanation of the two anomalies. The entire parameter
space of the model can be tested at the LHC by a combination of same sign
dimuon signals in $pp\rightarrow(\mu^{+}\mu^{+}jj+{E\\!\\!\\!\\!/}_{T})$ and
$pp\rightarrow(\mu^{+}\mu^{-}\tau^{+}\tau^{-}+X)$ signals.

_Introduction_.– A high precision measurement of the $W$ boson mass has
recently been reported by the Fermilab CDF collaboration based on high yields
of $W$ bosons collected from 2002 to 2011 CDF:2022hxs . The reported value,
$M_{W}=(80,433.5\pm 9.4)\leavevmode\nobreak\ {\rm MeV}$, differs from the
Standard Model prediction of $M_{W}^{\rm SM}=(80,357\pm 6)\leavevmode\nobreak\
{\rm MeV}$ Awramik:2003rn  at the 7 sigma level. Although the CDF measurement
is not fully consistent with LHC measurements (ATLAS measures $M_{W}({\rm
ATLAS})=(80,370\pm 19)\leavevmode\nobreak\ {\rm MeV}$ ATLAS:2017rzl , LHCb
measures $M_{W}({\rm LHCb})=80,354\pm 32\leavevmode\nobreak\ {\rm MeV}$
LHCb:2021bjt ) its high precision taken together with the consistent cross
checks leading to the $Z$ boson mass in agreement with world average value may
be viewed as a tantalizing evidence for new physics beyond the Standard Model
(SM). It is sufficient for such new physics to show up as subtle quantum
corrections in order to explain this discrepancy.

Independently, the Fermilab muon ($g-2$) collaboration has reported a precise
measurement of the anomalous magnetic moment of the muon to be $a_{\mu}({\rm
FNAL})=116592040(54)\times 10^{-11}$ Muong-2:2021ojo , in agreement with the
prior Brookhaven $(g-2)$ collaboration measurement of $a_{\mu}({\rm
BNL})=116592089(63)\times 10^{-11}$ Muong-2:2006rrc . These measurements
differ from the standard model theoretical prediction of $a_{\mu}({\rm
theory})=116591810(43)\times 10^{-11}$ Aoyama:2020ynm  at a combined 4.2 sigma
confidence level. This anomaly may well be viewed as evidence for new physics.

It would be intriguing to seek frugal extensions of the SM which
simultaneously explain the upward shift in the $W$ boson mass and the
anomalous magnetic moment of the muon. The purpose of this paper is to show an
interesting correlation between the two anomalies in the context of two Higgs
doublet model (2HDM). A simple way to explain the muon $(g-2)$ in the 2HDM is
through one-loop diagrams involving neutral Higgs bosons from the second Higgs
doublet and the $\tau$-lepton. Such diagrams have a helicity enhancement
proportional to $m_{\tau}/m_{\mu}$ in the muon $(g-2)$ and thus can readily
explain the BNL and FNAL measurements. However, these diagrams require
significant splittings among the neutral scalar bosons – in the limit of
degenerate masses new contributions to $a_{\mu}$ would vanish. We show here
that interestingly these mass splittings also provide an upward shift in the
mass of the $W$ boson compared to the SM prediction, in agreement with the
recent CDF measurement. In presence of such splittings, the process
$pp\rightarrow\mu^{+}\mu^{+}jj+{E\\!\\!\\!\\!/}_{T}$ with same-sign dimuons
occurs with significant strength and can be used as a test of the model.
Additionally, the process $pp\rightarrow\mu^{+}\mu^{-}\tau^{+}\tau^{-}+X$ is
also within reach, given that the scalar particles cannot be heavier than
about 600 GeV in order for the anomalies to be explained. A combination of
these two processes can explore the entire parameter space of the model.

_Model_.– We work within the general 2HDM with no special symmetry imposed
(for a review and detailed list of references see Ref. Branco:2011iw ). In
this scenario one can make a field rotation so that only one among the two
neutral Higgs boson acquires a vacuum expectation value (VEV). In this Higgs
basis, the two Higgs doublets can be parameterized as:

$\displaystyle H_{1}=\begin{pmatrix}G^{+}\\\
\frac{1}{\sqrt{2}}(v+\phi_{1}^{0}+iG^{0})\end{pmatrix}\quad
H_{2}=\begin{pmatrix}H^{+}\\\
\frac{1}{\sqrt{2}}(\phi_{2}^{0}+iA)\end{pmatrix}$ (1)

Here $G^{+}$ and $G^{0}$ are unphysical Goldstone modes, whereas $H^{+}$ and
$\\{\phi_{1}^{0},\phi_{2}^{0},A\\}$ belong to the physical spectrum. The VEV
of $H_{1}$, $v\simeq 246$ GeV, governs the electroweak symmetry breaking
dynamics.

The physical scalar masses can be readily computed from the Higgs potential
(see Eq. (A1) in Appendix). For simplicity we take all the parameters of the
potential to be real. We are also interested in working close to the alignment
limit, where the SM-like Higgs $\phi_{1}^{0}\approx h$ decouples from the new
CP-even Higgs ($\phi_{2}^{0}\approx H$). The properties of $h$ in this limit
will not be modified much compared to the SM. In the alignment limit the
masses of the physical scalars can be expressed as (see for e.g. Ref.
Babu:2018uik ):

$\displaystyle
m^{2}_{h}=\lambda_{1}v^{2},\;m^{2}_{H}=m^{2}_{22}+\frac{v^{2}}{2}(\lambda_{3}+\lambda_{4}+\lambda_{5}),$
(2) $\displaystyle
m^{2}_{A}=m^{2}_{H}-v^{2}\lambda_{5},\;m^{2}_{H^{\pm}}=m^{2}_{H}-\frac{v^{2}}{2}(\lambda_{4}+\lambda_{5}).$
(3)

Note that the splittings among the masses of the second doublet are controlled
by the quartic couplings $\lambda_{4}$ and $\lambda_{5}.$ These splittings
will be relevant for both the muon $(g-2)$ anomaly and the $W$ boson mass
prediction.

The leptonic Yukawa Lagrangian of the model is given by

$\displaystyle-$ $\displaystyle\mathcal{L}_{\text{Y}uk}\supset
Y\overline{\ell}_{L}H_{1}\ell_{R}+\widetilde{Y}\overline{\ell}_{L}H_{2}\ell_{R}+h.c.$
(4)

The charged lepton mass matrix follows from Eq. (4) as $M_{\ell}=Yv/\sqrt{2}$.
Without loss of generality we choose the matrix $Y$ to be diagonal, so that
Eq. (4) is already in the charged lepton mass eigenbasis. Analogous couplings
exist in the quark sector. While the quark couplings to the $H_{1}$ field are
necessary to generate quark masses, their couplings to $H_{2}$ field may be
taken to be small, or even zero. In this limit, the second doublet behaves as
a leptophilic Higgs, for which low energy constraints from meson decays etc,
as well as high energy collider constraints are not that stringent. We shall
work in this leptophilic limit of the model.

In addition, in the spirit of explaining the muon $(g-2)$ anomaly, we shall
assume the following flavor structure for the Yukawa matrix $\tilde{Y}$ in Eq.
(4):

$\tilde{Y}_{\mu\tau}\neq 0,\leavevmode\nobreak\ \leavevmode\nobreak\
\tilde{Y}_{\tau\mu}\neq 0,\leavevmode\nobreak\
\tilde{Y}_{ij}=0\leavevmode\nobreak\ {\rm for\leavevmode\nobreak\
all\leavevmode\nobreak\ other}\leavevmode\nobreak\ i,j\leavevmode\nobreak\ .$
(5)

With this choice, lepton flavor violation constraints are automatically
satisfied. In fact, with this choice a $Z_{4}$ subgroup of
$U(1)_{L_{\mu}-L_{\tau}}$ remains unbroken in the Yukawa sector. This $Z_{4}$
symmetry, if imposed, would set $\lambda_{6}=\lambda_{7}=0$ in the Higgs
potential of Eq. (A1). Importantly, this $Z_{4}$ would admit the $\lambda_{5}$
coupling, which is crucial for the one-loop diagram for muon $(g-2)$. It is
worth noting that the diagonal couplings of the matrix $Y$ of Eq. (4) are
allowed by the $Z_{4}$, so that charged lepton masses are consistently
generated. There would be no tree-level flavor-changing neutral currents
mediated by neutral Higgs bosons in the limit of exact $Z_{4}$ symmetry. We
shall refer to this scenario as $\mu-\tau$ specific 2HDM. While this is an
interesting limit of the general 2HDM, we shall not explicitly need to adopt
this limit in our numerical study. Rather, the ansatz of Eq. (5) would suffice
for this purpose.

Muon $(g-2)$ has been studied within the 2HDM by various authors. One class of
study uses lepton flavor specific 2HDM where there are no FCNC at tree level
Abe:2015oca ; Jana:2020pxx ; Wang:2018hnw . Another possibility that has been
studied is muon-specific 2HDM Abe:2017jqo . In contrast, the
$\mu-\tau$-specific 2HDM that we have explored is closer to the 2HDM studied
in Ref. Omura:2015xcg .

_Muon anomalous magnetic moment_.–

Figure 1: The dominant contribution to the muon anomalous magnetic moment.

In our setup, both neutral scalars and the charged scalar would contribute to
the muon anomalous magnetic moment. However, since chirality flip occurs on
the external legs for the charged scalar mediated contribution to $\Delta
a_{\mu}$, the corresponding contribution is negligible compared to the
corrections from the neutral scalars. The dominant contribution to the muon
anomalous magnetic moment in our setup shown in Fig. 1 which generates a
$\Delta a_{\mu}$ given by Leveille:1977rc :

$\displaystyle\Delta a_{\mu}$
$\displaystyle=\frac{\widetilde{Y}_{\mu\tau}\widetilde{Y}_{\tau\mu}}{16\pi^{2}}\left(\frac{m_{\tau}}{m_{\mu}}\right)\left[G\left(\frac{m_{\tau}^{2}}{m^{2}_{\mu}},\frac{m_{H}^{2}}{m^{2}_{\mu}}\right)-G\left(\frac{m_{\tau}^{2}}{m^{2}_{\mu}},\frac{m_{A}^{2}}{m^{2}_{\mu}}\right)\right]$
(6)

where the loop function is defined as

$\displaystyle
G\left(\omega^{a},\omega^{b}\right)=\int_{0}^{1}dx\frac{x^{2}}{x^{2}+x(\omega^{a}-1)+\omega^{b}(1-x)}.$
(7)

Here we have not shown the neutral scalar contributions with external mass
flip of the muon, which are much smaller than the contribution shown in Eq.
(6). We shall use these expressions for computing $(g-2)$ of the muon. Note
that these contributions would require a mass splitting between the neutral
scalars $H$ and $A$, since in the degenrate limit their contributions exactly
cancel.

_Shift in the $W$-boson mass_.– The $W$-boson mass is calculable within the SM
in terms of the precisely measured input parameters $\\{G_{F},\,\alpha_{\rm
em},\,M_{Z}\\}$ as well as the masses of the top quark and the Higgs boson,
and the QCD coupling $\alpha_{s}(M_{Z})$ which appear in quantum corrections
to $M_{W}$. The $W$-boson mass is related to these parameters as

$M_{W}^{2}\left(1-\frac{M_{W}^{2}}{M_{Z}^{2}}\right)=\frac{\pi\alpha_{\rm
em}}{\sqrt{2}G_{F}}(1+\Delta r)$ (8)

where $\Delta r$ represents the quantum corrections. Here $M_{W}$ and $M_{Z}$
are the renormalized masses in the on-shell scheme. From Eq. (8) $M_{W}$ can
be calculated as

$M_{W}^{2}=\frac{M_{Z}^{2}}{2}\left[1+\sqrt{1-\frac{4\pi\alpha_{\rm
em}}{\sqrt{2}G_{F}M_{Z}^{2}}(1+\Delta r)}\right]\leavevmode\nobreak\ .$ (9)

The parameter $\Delta r$ itself is a function of $M_{W}^{2}$, which requires
an iterative solution to Eq. (9) to obtain the SM prediction for $M_{W}$. It
is conventional to write $\Delta r$ as

$\Delta r=\Delta\alpha-\frac{c_{W}^{2}}{s_{W}^{2}}\delta\rho+\Delta r_{\rm
rem}$ (10)

where $\Delta\alpha\simeq 0.06$ arises from the renormalization of
$\alpha_{\rm em}$ which is dominated by light fermions,
$\delta\rho=T/\alpha(M_{Z})$ is the oblique correction with a value of $\Delta
r^{[\Delta\rho]}\equiv-(c_{W}^{2}/s_{W}^{2})\delta\rho\simeq-0.03$ in the SM
primarily arising from top-bottom quark loop diagram, and $\Delta r_{\rm
rem}\simeq 0.01$ is the remaining box and vertex diagram contributions Lopez-
Val:2014jva . With the central values of the input parameter, $M_{Z}=91.1876$
GeV, $\alpha_{\rm em}^{-1}=137.036$, $s_{W}^{2}=0.2315$, $M_{t}=172.76$ GeV,
$M_{h}=125.25$ GeV, and $G_{F}=1.1663787\times 10^{-5}$ GeV-2, one finds
$\Delta r^{\rm SM}=0.038$, and consequently from Eq. (9) $M_{W}=80.3564$ GeV.
This is 7 $\sigma$ below the CDF value reported recently CDF:2022hxs . This
discrepancy may be resolved via quantum corrections that modify $\Delta r$
from its SM value. Indeed, a value of $\Delta r=0.03386$ would lead to
$M_{W}=80.433$ GeV, the central value from the CDF measurement.

Assuming that the new physics that modifies $\Delta r$ arises as oblique
corrections dominated by $T$-parameter, one would require a new positive
contribution to $\delta\rho^{\rm new}=0.001246$. Using
$\alpha(M_{Z})=1/127.935$, this would imply $T=0.15949$. The one sigma allowed
range for the $T$-parameter consistent with the CDF $W$ boson mass measurement
is

$T:\\{0.159,\leavevmode\nobreak\ 0.210\\}$ (11)

which corresponds to the full contribution to $\Delta r$ in the range
$\\{0.0326,\leavevmode\nobreak\ 0.0339\\}$. There is a sub-leading
contribution from the oblique $S$-parameter (discussed below), which we also
include in our numerical study.

There are new contributions to the $T$-parameter in the 2HDM, given, in the
alignment limit, by

$\displaystyle T=\scriptstyle\dfrac{1}{16\pi^{2}\alpha_{\rm
em}(M_{Z})v^{2}}\left\\{{\mathcal{F}(m_{H^{+}}^{2},m_{H}^{2})+\mathcal{F}(m_{H^{+}}^{2},m_{A}^{2})-\mathcal{F}(m_{H}^{2},m_{A}^{2})}\right\\}\,,$
(12)

where the symmetric function $\mathcal{F}$ is given by

$\mathcal{F}(m_{1}^{2},m_{2}^{2})\ \equiv\
\frac{1}{2}(m_{1}^{2}+m_{2}^{2})-\frac{m_{1}^{2}m_{2}^{2}}{m_{1}^{2}-m_{2}^{2}}\ln\left(\frac{m_{1}^{2}}{m_{2}^{2}}\right)\,.$
(13)

In addition to the $T$-parameter contribution, $S$-parameter can also modify
the $W$-boson mass. The corresponding shift in the $W$-boson mass is given by
Dugan:1991ck ; Peskin:1991sw

$\displaystyle\Delta
M_{W}\simeq-\dfrac{M_{W}\alpha(M_{Z})S}{4(c_{W}^{2}-s_{W}^{2})}.$ (14)

Here $M_{W}$ is the SM prediction for $M_{W}$. In the alignment limit, the
$S$-parameter in the 2HDM is given in Ref. Barbieri:2006dq  (see Eq. (B1) in
Appendix). We find that the shift in the $W$\- boson mass due to the
$S$-parameter is typically small, in the (5-10) MeV range, which we have
included in our analysis.

In our setup $G_{F}$ appearing in muon decay is modified due to new diagrams
affecting the $W$-$\mu$-$\nu_{\mu}$ coupling proportional to the Yukawa
coupling $\tilde{Y}_{\mu\tau}$ of Eq. (5). This shift in $G_{F}$ can be
obtained from analogous expressions given in Ref. Abe:2017jqo  (see Eq. (C1)
of Appendix). These correction would contribute to $\Delta r$ and thus would
modify he $W$-boson mass. However, we find that the shift in the $W$-boson
mass arising from these corrections is at most $1$ MeV in our setup, even with
the Yukawa coupling $|\tilde{Y}_{\mu\tau}|\simeq 1$.

_Other constraints_.– The charged Higgs boson in the model would mediate a new
decay for $\tau$ lepton:
$\tau^{-}\rightarrow\mu^{-}\nu_{\mu}\overline{\nu}_{\tau}$. The effective
Lagrangian for this decay can be written down (after a Fierz reordering) as

${\cal L}_{\rm
eff}=-\frac{\tilde{Y}_{\mu\tau}\tilde{Y}^{*}_{\tau\mu}}{2M_{H^{+}}^{2}}(\overline{\nu}_{\mu_{L}}\gamma_{\mu}\nu_{\tau_{L}})(\overline{\mu}_{R}\gamma^{\mu}\tau_{R})+h.c.$
(15)

Note that the $\tau^{-}\rightarrow\mu^{-}\nu_{\mu}\overline{\nu}_{\tau}$ decay
does not interfere with the standard decay
$\tau^{-}\rightarrow\mu^{-}\overline{\nu}_{\mu}\nu_{\tau}$. Even so, the
strength for the decay $\tau\rightarrow\mu\nu\nu$ will differ from that of the
usual muon decay which does not receive any new contribution. From the limit
$g_{\tau}/g_{\mu}=1.0011\pm 0.0015$ obtained from these two decays
Pich:2013lsa , one obtains

$|\epsilon_{\tau}|\equiv\left|\frac{\tilde{Y}_{\mu\tau}\tilde{Y}_{\tau\mu}^{*}}{g^{2}}\frac{M_{W}^{2}}{M_{H^{+}}^{2}}\right|\leq
0.089$ (16)

at the 2 sigma level. We shall impose this constraint in our numerical
analysis, which would serve as the most stringent constraint on the mass of
$H^{\pm}$. There is also a constraint $|\epsilon_{\tau}|<0.213$ from $\tau$
lifetime measurements, which is however weaker.

There are direct search limits on the mass of $H^{\pm}$ from LEP and LHC
experiments. These limits depend on the branching ratios of $H^{+}$ into
specific lepton flavor. In our scenario $H^{+}$ would decay into
$\tau^{+}\nu_{\mu}$ or $\mu^{+}\nu_{\tau}$. If the branching ratios for the
two decay modes are comparable, $H^{\pm}$ mass can be as low as about 100 GeV
Babu:2019mfe , while for a 100% branching ratio into $\mu^{+}\nu$ LHC limits
of order 250 GeV become applicable.

The neutral CP-even and CP-odd scalar fields can have mass as low as 100 GeV.
Once produced, these particles would decay dominantly to $\mu^{+}\tau^{-}$ or
$\mu^{-}\tau^{+}$. There are no dedicated searches by ATLAS or CMS for
resonances decaying this way. In our analysis we shall allow these masses to
be as low as 100 GeV.

Loop corrections to the $W$-$\mu$-$\nu_{\mu}$ vertex proportional to
$\tilde{Y}_{\mu\tau}$, as well as the oblique parameters $T$ and $S$ would
affect the partial decay width $\Gamma(W\rightarrow\mu\overline{\nu}_{\mu})$.
These shifts can be obtained from expressions given in Ref. Abe:2017jqo  with
appropriate changes in the couplings. For all the benchmark points listed in
Table 1, we find that the shift in the
$\Gamma(W\rightarrow\mu\overline{\nu}_{\mu})$ is less than $0.5$ MeV, for the
Yukawa coupling $|\tilde{Y}_{\mu\tau}|\simeq 1$. Similarly, the vertex
corrections to the $Z$-$\mu$-$\mu$ coupling and the oblique corrections would
modify the partial decay width $\Gamma(Z\rightarrow\mu\bar{\mu})$. However,
such effects are always very small in our setup. For
$|\tilde{Y}_{\mu\tau}|\simeq 1$ (or $|\tilde{Y}_{\tau\mu}|\simeq 1$), we find
that the shift in the $\Gamma(Z\rightarrow\mu\bar{\mu})$ to be less than 1
MeV.

Results.– In Fig. 2 we have shown a direct correlation between $W$\- boson
mass and muon anomalous magnetic moment within our framework. As noted
earlier, the muon anomalous magnetic moment and the shift in the $W$\- boson
mass depend on the parameters
$(m_{H},\lambda_{4},\lambda_{5},|\tilde{Y}_{\mu\tau}\tilde{Y}_{\tau\mu}|)$. In
Fig. 2, we choose three different values for the
$\sqrt{|\tilde{Y}_{\mu\tau}\tilde{Y}_{\tau\mu}|}$ (1.0, 0.5, and 0.3). We vary
the quartic couplings $\lambda_{4},\lambda_{5}$ uniformly in the range
$(-\sqrt{4\pi},\sqrt{4\pi})$ and $m_{H}\in(110$ GeV - 1 TeV). Then from this
data set we choose data points which satisfy $\tau$-decay universality
constraints, positivity criteria of the squared masses of the scalar states,
and the conditions $m_{A},m_{H^{\pm}}\geq 110$ GeV. Our results are shown in
Fig. 2, where we have plotted $M_{W}$ versus $\Delta a_{\mu}$. Here we also
show the FNAL + BNL one sigma range for $\Delta a_{\mu}$, as well as the 1
sigma range for $M_{W}$ from the CDF experiment. The yellow band in Fig. 2 is
the SM prediction for $M_{W}$. We see from Fig. 2 that there are large number
of points which explain both anomalies within 1 sigma.

In Table 1 we have listed several benchmark points for the various scalar
masses that satisfy both anomalies, corresponding to three sets of Yukawa
couplings. These benchmark points, as well as the results from our full scan,
show that the masses of the scalars cannot exceed about 600 GeV, making the
model fully testable at the LHC.

$\sqrt{|\tilde{Y}_{\mu\tau}\tilde{Y}_{\tau\mu}|}$ |  |  $m_{H}$ (GeV) |  $m_{A}$ (GeV) |  $m_{H^{\pm}}$ (GeV) |  $M_{W}$ (GeV) | $\Delta a_{\mu}\times 10^{9}$
---|---|---|---|---|---|---
| BP1 | 445.15 | 343.42 | 274.21 | 80.4372 | 3.04
0.3 | BP2 | 372.48 | 503.42 | 318.88 | 80.4241 | 2.95
| BP3 | 402.50 | 551.76 | 355.12 | 80.4203 | 2.66
| BP4 | 593.90 | 526.20 | 457.68 | 80.4227 | $2.08$
0.5 | BP5 | 397.48 | 430.29 | 511.94 | 80.4293 | 2.32
| BP6 | 128.68 | 126.95 | 233.72 | 80.4476 | $2.98$
| BP7 | 508.26 | 494.50 | 598.65 | 80.4292 | $2.28$
1.0 | BP8 | 400.32 | 409.43 | 501.73 | 80.4294 | $2.73$
| BP9 | 575.44 | 555.96 | 466.60 | 80.4257 | 2.31
Table 1: Benchmark points that simultaneously fit the $W$-boson mass ($M_{W}$)
and muon anomalous magnetic moment ($\Delta a_{\mu}$) within the model, along
with the predicted values of $M_{W}$ and $\Delta a_{\mu}$.  Figure 2:
Theoretical predictions and experimental measurements of $W$-boson mass and
muon anomalous magnetic moment in the 2HDM. The different colored scattered
points depict the correlated predictions for the $W$-boson mass ($M_{W}$) and
muon anomalous magnetic moment ($\Delta a_{\mu}$) in our framework for
different choices of Yukawa coupling:
$\sqrt{|\tilde{Y}_{\mu\tau}\tilde{Y}_{\tau\mu}|}=1.0$ (green), 0.5 (cyan), and
0.3 (blue).

Collider tests.– Since the masses of the neutral and charged scalar cannot be
more than about 600 GeV in order to explain the $W$-boson mass and the muon
$(g-2)$ anomaly, the model can be tested at the LHC with the high luminosity
run. One interesting process is $pp\rightarrow(\mu^{+}\mu^{+}jj+\not{E}_{T})$
which arises from the vector boson fusion (VBF) diagram shown in Fig. 3
Aiko:2019mww . The amplitude for this process is proportional to the mass-
splittings among the neutral scalar and pseudoscalar particles, which in turn
is constrained by muon $(g-2)$ and $W$-boson mass shift. Now, we analyze the
signal $pp\rightarrow\mu^{+}\mu^{+}jj+$ ${E\\!\\!\\!\\!/}_{T}$ (see Appendix,
for analysis details) and summarize the signal sensitivity in Fig.4 at 14
$\mathrm{TeV}$ LHC for two different integrated luminosities. For
$|m_{H}-m_{A}|=200\,(150)$ GeV, we find that at 3$\sigma$ level, the charged
scalar $H^{+}$ can be probed up to masses of 382 (289), and 591 (485) GeV
respectively for the integrated luminosities of $\mathcal{L}=1$ and 10 ab-1.
It is worth noting that mass splitting is necessary to account for the proper
sign and strength of the $W-$ mass shift, although, as shown by the benchmark
points, $\mathcal{O}(10)$ GeV mass splitting is adequate to account for the
shift. However, the sensitivity of the signal $pp\rightarrow\mu^{+}\mu^{+}jj+$
${E\\!\\!\\!\\!/}_{T}$ at the 14 TeV LHC is insufficient to test the scenario
in the case of such small mass-splittings. However, in the complementary
parameter space, when mass splitting is significant ($\gtrsim 50$ GeV) between
$H$ and $A$, as seen in Fig.4, this may be investigated during the forthcoming
run of the LHC. For instance, BP1, BP2, and BP3 can be fully explored with
future HL-LHC with 10 ab-1 luminosity.

Figure 3: Feynman diagram for the signal $pp\rightarrow\mu^{+}\mu^{+}jj+$
${E\\!\\!\\!\\!/}_{T}$ at the LHC. Figure 4: Discovery prospect of the signal
$pp\rightarrow\mu^{+}\mu^{+}jj+$ ${E\\!\\!\\!\\!/}_{T}$ at the 14 TeV LHC in
the mass-splitting vs. charged Higgs mass plane. Dashed and solid lines
correspond to a significance of 3$\sigma$ and 5 $\sigma$.

LHC could also look for pair production of $H$ and $A$ via $Z$-boson exchange.
Each of these fields would decay into $\mu^{\pm}\tau^{\mp}$. No dedicated
search has been carried out by the LHC collaborations. The signal strength for
this process has been studied in detail in Ref. Babu:2020ivd  and Ref.
Iguro:2019sly , where it has been shown that current data already would be
sensitive to $H$ and $A$ masses of about 500 GeV. For completeness, we analyze
the signal using the same technique as described in Ref. Babu:2020ivd . After
imposing the acceptance criteria specified in Ref. Babu:2020ivd , we estimate
that the number of events for the signal that passes the acceptance criteria
is 185, 270, 440, 90, 360, 39100, 148 ,390, 80 for the nine benchmark points,
respectively, whereas we obtain 560 events for the SM background at 14 TeV LHC
with an integrated luminosity of $\mathcal{L}=$ 1 ab-1. We find that all of
these benchmark points can be explored with a significance of more than
$3\sigma$ at 14 TeV LHC with an integrated luminosity of $\mathcal{L}=$ 1
ab-1. For instance, the significance values associated with the nine benchmark
points are 6.7, 9.3, 13.9, 3.5, 11.8, 196, 5.5, 12.6, and 3.2, respectively.
Thus the scenario presented can be fully explored in future searches at the
LHC.

Conclusions.– In this paper we have shown an intriguing connection between the
recently reported upward shift in the $W$-boson mass measurement by the CDF
collaboration and the muon $(g-2)$ anomaly within the 2HDM. Mass splittings
among the second Higgs multiplet are needed in the scenerio presented in order
to explain the muon $(g-2)$. These splittings also result in an upward shift
of $M_{W}$, consistent with the CDF results. The charged and neutral scalars
of the second Higgs doublet should have masses below about 600 GeV, which
makes the scenario testable at the high luminosity LHC. We have pointed out
that the mass splittings among the scalars also would lead to the signal
$pp\rightarrow(\mu^{+}\mu^{+}jj+\not{E}_{T})$ which arises from the vector
boson fusion diagrams. Additionally,
$pp\rightarrow(\mu^{+}\mu^{-}\tau^{+}\tau^{-}+X)$ can be looked for, with
signal strength within reach for the entire range of scalar mass parameters.

###### Acknowledgements.

Acknowledgments.– The work of KSB and VPK is in part supported by US
Department of Energy Grant Number DE-SC 0016013. SJ acknowledges Oklahoma
State Univeristy physics department for hospitality while this work was done.

##  Appendix A Scalar potential

The most general scalar potential in the Higgs basis can be written as

$\displaystyle
V=m_{11}^{2}H_{1}^{\dagger}H_{1}+m_{22}^{2}H_{2}^{\dagger}H_{2}-\\{m_{12}^{2}H_{1}^{\dagger}H_{2}+{\rm
h.c.}\\}+\frac{\lambda_{1}}{2}(H_{1}^{\dagger}H_{1})^{2}+\frac{\lambda_{2}}{2}(H_{2}^{\dagger}H_{2})^{2}+\lambda_{3}(H_{1}^{\dagger}H_{1})(H_{2}^{\dagger}H_{2})$
$\displaystyle+\lambda_{4}(H_{1}^{\dagger}H_{2})(H_{2}^{\dagger}H_{1})+\left\\{\frac{\lambda_{5}}{2}(H_{1}^{\dagger}H_{2})^{2}+{\rm
h.c.}\right\\}+\left\\{\big{[}\lambda_{6}(H_{1}^{\dagger}H_{1})+\lambda_{7}(H_{2}^{\dagger}H_{2})\big{]}H_{1}^{\dagger}H_{2}+{\rm
h.c.}\right\\}.$ (17)

Minimization of the potential leads to the conditions
$m_{11}^{2}+\frac{\lambda_{1}}{2}v^{2}=0$ and
$m_{12}^{2}-\frac{\lambda_{6}}{2}v^{2}=0$.

##  Appendix B Expression for $S$-parameter

In the alignment limit, the $S$-parameter in the 2HDM is given by
Barbieri:2006dq

$\displaystyle
S=\dfrac{1}{2\pi}\int_{0}^{1}dx\,x(1-x)\ln{\left(\frac{xm_{H}^{2}+(1-x)m_{A}^{2}}{m^{2}_{H^{\pm}}}\right)}.$
(18)

##  Appendix C Correction to $G_{F}$ in muon decay

The shift in the $G_{F}$ affecting muon decay due to one-loop wave function
corrections to the $\mu$ and $\nu_{\mu}$ lines is given by Abe:2017jqo

$\displaystyle\frac{\delta
G_{F}}{G_{F}}\simeq\frac{|\tilde{Y}_{\mu\tau}|^{2}}{32\pi^{2}}\left(1-f(m_{H}^{2},m^{2}_{H^{\pm}})-f(m_{A}^{2},m^{2}_{H^{\pm}})\right)\,,$
(19)

where the function $f$ is defined as

$f(m_{1}^{2},m_{2}^{2})\ \equiv\
\frac{m_{1}^{2}+m^{2}_{2}}{4(m_{1}^{2}-m^{2}_{2})}\ln{\frac{m_{1}^{2}}{m^{2}_{2}}}\,.$
(20)

##  Appendix D Details on collider analysis

For our collider analysis, we analyze the signal with MadGraph5aMC@NLO
Alwall:2011uj ; Alwall:2014hca  event generator, simulating the hadronization
and underlying event effects with Pythia8 Sjostrand:2007gs  and detector
effects with the Delphes3 deFavereau:2013fsa . For the signal
$pp\rightarrow(\mu^{+}\mu^{+}jj+\not{E}_{T})$, the dominant SM background
originates from same-sign $W-$boson pair-production. In order to optimize the
signal sensitivity, we employ the following selection criterias: (a) events
with at least two jets and two same-sign dimuon pairs with $p_{Tj}>30$ GeV,
$|\eta^{j}|<5.0$, $p_{T}^{\mu}>\leavevmode\nobreak\ 20\mathrm{GeV}$ and
$\left|\eta^{\mu}\right|<2.5$ are selected; (b) for the two leading jets,
invariant mass cut $m_{jj}>500\leavevmode\nobreak\ \mathrm{GeV}$ and
pseudorapidity separation cuts $\left|\Delta\eta_{jj}\right|>2.5$ are imposed.
We define the significance as $S/\sqrt{S+B}$, where $S$ and $B$ define the
number of events for signal and SM backgrounds.

## References

  * (1) CDF Collaboration, T. Aaltonen et al., “High-precision measurement of the W boson mass with the CDF II detector,” Science 376 no. 6589, (2022) 170–176.
  * (2) M. Awramik, M. Czakon, A. Freitas, and G. Weiglein, “Precise prediction for the W boson mass in the standard model,” Phys. Rev. D 69 (2004) 053006, arXiv:hep-ph/0311148.
  * (3) ATLAS Collaboration, M. Aaboud et al., “Measurement of the $W$-boson mass in pp collisions at $\sqrt{s}=7$ TeV with the ATLAS detector,” Eur. Phys. J. C 78 no. 2, (2018) 110, arXiv:1701.07240 [hep-ex]. [Erratum: Eur.Phys.J.C 78, 898 (2018)].
  * (4) LHCb Collaboration, R. Aaij et al., “Measurement of the W boson mass,” JHEP 01 (2022) 036, arXiv:2109.01113 [hep-ex].
  * (5) Muon g-2 Collaboration, B. Abi et al., “Measurement of the Positive Muon Anomalous Magnetic Moment to 0.46 ppm,” Phys. Rev. Lett. 126 no. 14, (2021) 141801, arXiv:2104.03281 [hep-ex].
  * (6) Muon g-2 Collaboration, G. W. Bennett et al., “Final Report of the Muon E821 Anomalous Magnetic Moment Measurement at BNL,” Phys. Rev. D 73 (2006) 072003, arXiv:hep-ex/0602035.
  * (7) T. Aoyama et al., “The anomalous magnetic moment of the muon in the Standard Model,” Phys. Rept. 887 (2020) 1–166, arXiv:2006.04822 [hep-ph].
  * (8) G. C. Branco, P. M. Ferreira, L. Lavoura, M. N. Rebelo, M. Sher, and J. P. Silva, “Theory and phenomenology of two-Higgs-doublet models,” Phys. Rept. 516 (2012) 1–102, arXiv:1106.0034 [hep-ph].
  * (9) K. S. Babu and S. Jana, “Enhanced Di-Higgs Production in the Two Higgs Doublet Model,” JHEP 02 (2019) 193, arXiv:1812.11943 [hep-ph].
  * (10) T. Abe, R. Sato, and K. Yagyu, “Lepton-specific two Higgs doublet model as a solution of muon g $-$ 2 anomaly,” JHEP 07 (2015) 064, arXiv:1504.07059 [hep-ph].
  * (11) S. Jana, V. P. K., and S. Saad, “Resolving electron and muon $g-2$ within the 2HDM,” Phys. Rev. D 101 no. 11, (2020) 115037, arXiv:2003.03386 [hep-ph].
  * (12) L. Wang, J. M. Yang, M. Zhang, and Y. Zhang, “Revisiting lepton-specific 2HDM in light of muon $g-2$ anomaly,” Phys. Lett. B 788 (2019) 519–529, arXiv:1809.05857 [hep-ph].
  * (13) T. Abe, R. Sato, and K. Yagyu, “Muon specific two-Higgs-doublet model,” JHEP 07 (2017) 012, arXiv:1705.01469 [hep-ph].
  * (14) Y. Omura, E. Senaha, and K. Tobe, “$\tau$\- and $\mu$-physics in a general two Higgs doublet model with $\mu-\tau$ flavor violation,” Phys. Rev. D 94 no. 5, (2016) 055019, arXiv:1511.08880 [hep-ph].
  * (15) J. P. Leveille, “The Second Order Weak Correction to (G-2) of the Muon in Arbitrary Gauge Models,” Nucl. Phys. B 137 (1978) 63–76.
  * (16) D. López-Val and T. Robens, “$\Delta$r and the W-boson mass in the singlet extension of the standard model,” Phys. Rev. D 90 (2014) 114018, arXiv:1406.1043 [hep-ph].
  * (17) M. J. Dugan and L. Randall, “The Sign of S from electroweak radiative corrections,” Phys. Lett. B 264 (1991) 154–160.
  * (18) M. E. Peskin and T. Takeuchi, “Estimation of oblique electroweak corrections,” Phys. Rev. D 46 (1992) 381–409.
  * (19) R. Barbieri, L. J. Hall, and V. S. Rychkov, “Improved naturalness with a heavy Higgs: An Alternative road to LHC physics,” Phys. Rev. D 74 (2006) 015007, arXiv:hep-ph/0603188.
  * (20) A. Pich, “Precision Tau Physics,” Prog. Part. Nucl. Phys. 75 (2014) 41–85, arXiv:1310.7922 [hep-ph].
  * (21) K. S. Babu, P. S. B. Dev, S. Jana, and A. Thapa, “Non-Standard Interactions in Radiative Neutrino Mass Models,” JHEP 03 (2020) 006, arXiv:1907.09498 [hep-ph].
  * (22) M. Aiko, S. Kanemura, and K. Mawatari, “Exploring the global symmetry structure of the Higgs potential via same-sign pair production of charged Higgs bosons,” Phys. Lett. B 797 (2019) 134854, arXiv:1906.09101 [hep-ph].
  * (23) K. S. Babu, S. Jana, and M. Lindner, “Large Neutrino Magnetic Moments in the Light of Recent Experiments,” JHEP 10 (2020) 040, arXiv:2007.04291 [hep-ph].
  * (24) S. Iguro, Y. Omura, and M. Takeuchi, “Testing the 2HDM explanation of the muon g – 2 anomaly at the LHC,” JHEP 11 (2019) 130, arXiv:1907.09845 [hep-ph].
  * (25) J. Alwall, M. Herquet, F. Maltoni, O. Mattelaer, and T. Stelzer, “MadGraph 5 : Going Beyond,” JHEP 06 (2011) 128, arXiv:1106.0522 [hep-ph].
  * (26) J. Alwall, R. Frederix, S. Frixione, V. Hirschi, F. Maltoni, O. Mattelaer, H. S. Shao, T. Stelzer, P. Torrielli, and M. Zaro, “The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations,” JHEP 07 (2014) 079, arXiv:1405.0301 [hep-ph].
  * (27) T. Sjostrand, S. Mrenna, and P. Z. Skands, “A Brief Introduction to PYTHIA 8.1,” Comput. Phys. Commun. 178 (2008) 852–867, arXiv:0710.3820 [hep-ph].
  * (28) DELPHES 3 Collaboration, J. de Favereau, C. Delaere, P. Demin, A. Giammanco, V. Lemaître, A. Mertens, and M. Selvaggi, “DELPHES 3, A modular framework for fast simulation of a generic collider experiment,” JHEP 02 (2014) 057, arXiv:1307.6346 [hep-ex].


# Play It Back: Iterative Attention for Audio Recognition

###### Abstract

A key function of auditory cognition is the association of characteristic
sounds with their corresponding semantics over time. Humans attempting to
discriminate between fine-grained audio categories, often replay the same
discriminative sounds to increase their prediction confidence. We propose an
end-to-end attention-based architecture that through selective repetition
attends over the most discriminative sounds across the audio sequence. Our
model initially uses the full audio sequence and iteratively refines the
temporal segments replayed based on slot attention. At each playback, the
selected segments are replayed using a smaller hop length which represents
higher resolution features within these segments. We show that our method can
consistently achieve state-of-the-art performance across three audio-
classification benchmarks: AudioSet, VGG-Sound, and EPIC-KITCHENS-100. 111Our
code is available at: tinyurl.com/playitback2023

††$\\!{}^{*}$Work was done while A. Stergiou was at the University of Bristol.

Index Terms—  Audio classification, playback, attention

##  1 Introduction

Audio recognition is the task of categorizing audio with discrete labels that
semantically represent the emitted sounds. This includes significant
challenges considering the similarity in object sounds (e.g. boat motors and
road vehicles), musical instruments (e.g. guitar, banjo, and ukulele), human
(e.g. wail and groan), or animal (e.g. yip and growl) sounds.

In everyday life, we repeat parts of songs or ask for someone to repeat
themselves to better understand audio. This relates to the development of
echoic memory which is responsible for the memorization of sounds [1, 2].
Therefore, repeated listens and replays of sound stimulants [3] are an
essential part of learning and associating sound patterns.

Fig. 1: Playback of discriminative sounds. Given an audio sequence, the most
relevant sounds are selected and played back at reduced hop length. The
generated playbacks attend solely informative sounds at a higher temporal
resolution.

Driven by the perception of sound through echoic memory and the recent success
of Vision Transformers (ViT) [4] at utilizing global context information, we
propose an end-to-end attention-based model that recognizes sounds through
discovering and playing back the most informative sounds from the audio
sequence, as shown in Figure 1. We use slots [5] to attend to category-
relevant sounds in the input sequence. These slots select the time segments to
be replayed. Coarser features from earlier playbacks are memorized alongside
finer (i.e. higher-temporal resolution) features from later playbacks with the
use of a transformer decoder.

Our contributions are as follows: i) We propose to select and replay relevant
audio features with decreased hop lengths, slowing down relevant parts of the
audio. ii) We propose an end-to-end transformer architecture for audio
recognition that jointly selects and attends to multiple audio replays, and
refines the final class predictions. iii) Our method achieves state-of-the-art
performance on AudioSet [6], VGG-Sound [7], and EPIC-KITCHENS-100 [8].

Fig. 2: PlayItBack architecture. The spectrogram of the full audio sequence
(top) is replayed by focusing on discriminative features and reducing the hop
length to capture finer temporal details (bottom). During each playback,
spectrogram patches are tokenized $\mathbf{x}_{i}$ and appended patch
(frequency and temporal) positional encodings ($P$). Several multi-head
attention layers are used to encode features $\mathbf{z}_{i}$. Slot attention
$\mathcal{G}(\mathbf{z}_{i})$ then discovers discriminative temporal segments.
These are considered input to the next playback. To combine decisions between
playbacks, a recurrent Transformer Decoder $\mathcal{D}(\mathbf{z}_{i})$ takes
previously decoded features from $i$ and the encoded features in $i+1$
playback appended with patch encodings ($PP$). PlayItBack is trained by
classification loss $\mathcal{L}_{CLS_{i}}$, regularized by the weighted sum
of ranking losses $\mathcal{L}_{rank_{i}}$ between $i$ and $\\{1,...,i-1\\}$
playbacks.

##  2 Related Work

Audio recognition. A popular approach for audio classification has been the
use of convolutional networks, previously used for image-based object
recognition [9, 10, 11] or video classification [12] tasks, to learn features
from audio spectrograms. The introduction of Transformer-based architectures
has further given rise to their adaptation for audio recognition by works
relying on hybrid architectures [13, 14, 15]. Similar attempts have also built
on image-pretrained Transformer models for attending audio spectrograms [16,
17]. [18] incorporated an additional video modality to improve performance.
Recently, [19] have also studied the effects of different hop lengths on the
temporal resolution of spectrograms.

In contrast to the majority of previous works, we focus on identifying
relevant and irrelevant sounds. The irrelevant sounds are removed, while
relevant segments are slowed and replayed with predictions calculated across
playbacks.

Selecting discriminative features. Modeling discriminative features has been a
central focus of image recognition methods. [20] generated features from
multiple scales selecting the best-suited features per scale. [21] proposed
the aggregation of features from image regions cropped based on class
saliencies. [22] applied distortion grids on the cropped regions. Most similar
to our work, [23] used a recurrent CNN to select image regions that are
attended to in follow-up scales. In their work, only a single region was
selected per scale. Instead, we identify multiple discriminative sound
segments which we combine to form the next playback.

We describe our PlayItBack method next.

##  3 Method

In this section, we describe the proposed PlayItBack architecture, depicted in
Figure 2. We compute the mel-spectrogram for a given audio sequence resulting
in an $F\\!\times\\!T$ representation of frequency $F$ and time $T$, and
extract $k$ non-overlapping patches. We project the patches to feature tokens
$\mathbf{x}_{i}\\!\\!\in\\!\\!\mathbb{R}^{D}$, where $D\\!=\\!FT$. A
transformer encoder $\mathcal{B}$ is used to encode these into features
$\mathbf{z}_{i}$. Slot attention $\mathcal{G}$ is applied to $\mathbf{z}_{i}$
to select the discriminative regions which, at the next playback, will be
slowed by decreasing the hop length in the spectrogram. The Decoder
$\mathcal{D}$ then relates features across playbacks.

Transformer Encoder. Given linear projections $\mathbf{x}_{i}$, we use
frequency and temporal patch positional encodings $P$. The encoder network
$\mathcal{B}$ extracts representations for each playback,
$\mathbf{z}_{i}\\!=\\!\mathcal{B}(\mathbf{x}_{i})\\!\in\\!\mathbb{R}^{d\times
C},\text{with}\;d\\!<\\!\\!FT$ resolution and $C$ channels.

Model | Backbone | Train set | mAP
---|---|---|---
Audio-only models
MAE-AST [24] | ViT-B [4] | mini-AS | 30.6
Perceiver [25] | Perceiver | AS-2M | 38.4
Conformer [14] | Conformer | AS-2M | 41.1
PANN [10] | ResNet38 [26] | AS-2M | 43.4
MBT [18] | ViT-B | AS-500K | 44.3
PSLA [9] | EffNet-B2 [27] | AS-2M | 44.4
PaSST [17] | DeiT-B [28] | AS-2M | 47.1
HTS-AT [16] | Swin-T [29] | AS-2M | 47.1
MaskSpec [30] | ViT-B | AS-2M | 47.1
Audio-MAE [31] | ViT-B | AS-2M | 47.3
PlayItBackX3 | MViTv2-B [32] | AS-500K | 47.7

Table 1: Comparisons to state-of-the-art audio-only models on AudioSet. We
report the mean average precision (mAP) alongside the backbone and training
set used.

Model | top-1 | top-5 | mAP | AUC | d-prime
---|---|---|---|---|---
Audio-only models
McDonnell & Gao [33] | 39.7 | 71.6 | 40.3 | 0.963 | 2.532
Peng et al. (A) [34] | 44.3 | - | 48.4 | - | -
ResNet-101 [12] | 45.6 | 72.3 | 47.6 | 0.968 | 2.615
Chen et al. [7] | 51.0 | 76.4 | 53.2 | 0.973 | 2.735
MBT (A) [18] | 52.3 | 78.1 | - | - | -
Slow-Fast [12] | 52.4 | 78.1 | 54.4 | 0.974 | 2.761
PlayItBackX3 | 53.7 | 79.2 | 56.1 | 0.978 |  2.846
Models trained with additional modalities
Peng et al. (AV) [34] | 50.6 | - | 53.9 | - | -
PolyViT [35] | 51.7 | - | - | - | -
MBT (AV) | 64.1 | 85.6 | - | - | -

Table 2: Comparisons to state-of-the-art models on VGG-Sound. We report the
top-1 and top-5 accuracies (%) alongside mAP, the AUC and d-prime.

Slot attention. We use slot attention [5] $\mathcal{G}$ to iteratively map the
resulting feature vectors $\mathbf{z}_{i}$ from each playback to two slot
vectors $\mathbf{s}_{lj}$ corresponding to the informative $\mathbf{s}_{1j}$
and uninformative $\mathbf{s}_{2j}$ temporal segments of the audio input
respectively. We use $j\\!\in\\!\\{1,...,J\\}$ to denote slot iterations. The
query $\mathbf{Q}_{lj}\\!=\\!MLP(LN(\textbf{s}_{lj-1}))$, key
$\mathbf{K}_{lj}\\!=\\!MLP(LN(\textbf{z}_{i}))$ and value
$\mathbf{V}_{lj}\\!=\\!MLP(LN(\textbf{z}_{i}))$ use Layer Normalization
$LN(\cdot)$ followed by Multi-Layer Perceptron $MLP(\cdot)$ to map the
features $\mathbf{z}_{i}$ and slots $\mathbf{s}_{lj}$ vectors to a common
dimension $d$. We set the softmax temperature based on a fixed value
$\sqrt{d}$.

$\displaystyle\mathbf{h}_{lj}=GRU\Bigl{(}\frac{\mathbf{a}_{lj}\>\mathbf{V}_{lj}}{\smashoperator[]{\sum_{m\in\\{1,2\\}}^{}}\mathbf{a}_{mj}}\Bigr{)},\,\text{where}\;\mathbf{a}_{lj}\\!=\\!Attn\Bigl{(}\frac{\mathbf{K}_{lj}\,\mathbf{Q}_{lj}^{T}}{\sqrt{d}}\Bigr{)}$
(1)

A Gated Recurrent Unit (GRU) with two hidden units is used at each slot
iteration updating the slot hidden states $\mathbf{h}_{lj}$ as in [5]. A
linear transformation alongside a residual connection is used for the slots
$\mathbf{s}_{lj}\\!=\\!\mathbf{s}_{lj-1}\\!+\\!MLP(LN(\mathbf{h}_{lj}))$.

We train the two slots so $\mathbf{s}_{1}$ attends to informative audio while
$\mathbf{s}_{2}$ captures the remaining audio. This is achieved by combining
$\mathcal{G}(\mathbf{z}_{i})_{1}\\!=\\!\mathbf{s}_{1}$ and the _inverse_ of
the uninformative slot $\mathcal{G}(\mathbf{z}_{i})_{2}\\!=\\!\mathbf{s}_{2}$
to create the attention matrix:
${\mathbf{M}\\!=\\!Attn(\mathcal{G}(\mathbf{z}_{i})_{1}^{T}\,\mathcal{G}(\mathbf{z}_{i})_{2}^{-1})}$.
We normalize and rescale the main diagonal $diag(\mathbf{M})$ by interpolation
so that it matches the temporal dimension of $\mathbf{x}_{i}$. Activations
above the normalized average ($>\\!0.5$) are selected for the segments in
$\mathbf{x}_{i+1}$.

Transformer Decoder. Given the extracted encoder features $\mathbf{z}_{i}$,
the decoder transformer $\mathcal{D}$ relates information across playbacks.
Positional encodings based on patches and the playback number are added to
$\mathbf{z}_{i}$. Considering the iterative nature of the PlayItBack model,
cross-attending [25] information over playbacks enables the model to retain
general features and associate patterns that are common. For the decoder, we
define the query from the previous playback as
$\mathbf{Q}_{i}\\!=\\!MLP(LN(\mathbf{v}_{i}))$, where $\textbf{v}_{1}$ is
initialized with a latent vector then updated at each playback
$\textbf{v}_{i}=\mathcal{D}(\textbf{z}_{i-1},\textbf{v}_{i-1})$, where $i>1,$
key $\mathbf{K}_{i}=MLP(LN(\textbf{z}_{i}))$ and value
$\mathbf{V}_{i}=MLP(LN(\textbf{z}_{i}))$ for the cross attention. This is
followed by a self-attention block. The decoder features are then passed to a
classifier shared across playbacks.

Classification and rank loss. We use an inter-playback weighted ranking loss
$\mathcal{L}_{\text{rank}(i)}$ for forcing the network to attain more
confident predictions in later playbacks. The ranking loss
$\mathcal{L}_{\text{rank}(i)}$, uses the pair-wise class probabilities
$p(\omega)_{i}$ and $p(\omega)_{m}\forall m\\!\in\\!\\{1,...,i\\!-\\!1\\}$ for
the correct class label $\omega$. We compute the probability difference
$\mathcal{L}_{\text{rank}(m\leftrightarrow i)}$ between the $i$th playback and
all previous playbacks.

$\mathcal{L}_{\text{rank}(i)}\\!=\\!\smashoperator[]{\sum_{m=1}^{i-1}}\lambda_{m}\,max(0,\gamma-p(\omega)_{i}+p(\omega)_{m})$
(2)

The ranking loss thus uses predictions from the previous playbacks as a
reference with the expectation that
$p(\omega)_{i}\\!>\\!p(\omega)_{m}+\gamma$, i.e. subsequent playbacks always
increase confidence, where $\gamma$ is the ranking loss’s margin. For
stability in training, we include a weight $\lambda_{m}=\frac{1}{i-m}$
computed based on the difference between the playback indices.

We combine $\mathcal{L}_{rank(i)}$ with an inter-playback cross-entropy loss
$\mathcal{L}_{\text{CLS}(i)}$ and define our multi-task loss as:

$\mathcal{L}=\mathcal{L}_{\text{CLS}(1)}+\smashoperator[]{\sum_{i=2}^{N}}\mathbf{\beta}\,\mathcal{L}_{\text{CLS}(i)}+(1-\mathbf{\beta})\,\mathcal{L}_{\text{rank}(i)}$
(3)

where $\mathbf{\beta}$ is a weighting parameter for the aggregation of the
cross-entropy and ranking losses. During inference, our model uses the average
of all predictions across playbacks.

##  4 Experiments

Datasets We evaluate our proposed PlayItBack architecture on three large-scale
datasets. AudioSet [6] is composed of 2M 10s audio clips from YouTube
annotated with 527 classes (AS-2M). Because of the high imbalance of the
dataset, we instead train with the proposed AS-500K [18]. VGG-Sound [7]
consists of 200k clips of 10s length with 309 labels corresponding to human
actions, objects and interactions. EPIC-KITCHENS-100 [8] includes 90k clips of
hand-object interactions labeled with 97 verb, 300 noun classes, and 4025
action classes. The clip length is variable and 2.6s on average.

Evaluation metrics. For AudioSet along the lines of previous works, we use the
mean average precision (mAP). For VGG-Sound, as in [12], we report the top-1/5
% accuracies, mAP, AUC, and d-prime. For EPIC-KITCHENS-100 we report the
top-1/5 % accuracies for the verb, noun, and action labels.

Implementation details. We use PlayItBackX3 with $N\\!\\!=\\!\\!3$ as our
model for comparative evaluation, with ablations showcasing that this produces
the best accuracy (top-1)/compute (GFLOPs) trade-off. We use the 24-layer
MViTv2-B [32] as our default encoder222The flattened vector size is d=50 and
the number of features is C=768. We note that due to the fixed number of 2D
patches used by MViTv2, the spectrogram dimensions remain constant throughout
playbacks. For all experiments, we set the ranking margin
$\gamma\\!\\!=\\!\\!0.05$, $J=3$ slot iterations, and
$\mathbf{\beta}\\!=\\!0.7$. As in [12] we use spectrograms with frequency
dimension of 128 corresponding to inputs of size $128\\!\\!\times\\!\\!100S$
for $S$ seconds of audio. Our initial spectrograms are created based on the
same hop length of 10ms, and 16kHz frequency as in [12, 18, 31]. For
subsequent iterations, we reduce the hop length by 1ms at each iteration.

We train for 50 epochs with Mixup [36] ($\alpha=0.3$) and base learning rate
of 0.5 for AudioSet and 0.01 for VGG-Sound & EPIC-KITCHENS-100. We use warm-up
for the first 2.5 epochs, a decayed cosine schedule, batch size of 64 with
SGD, momentum set to 0.9, and $1e^{-4}$ weight decay.

Model | GFLOPs | verb | noun | action
---|---|---|---|---
top-1 | top-5 | top-1 | top-5 | top-1 | top5
Damen et al. [8] | N/A | 42.6 | 75.8 | 22.3 | 44.6 | 14.5 | 28.2
MBT (A) [18] | 34.2 | 44.3 | - | 22.4 | - | 13.0 | \-
Slow-Fast [12] | 35.1 | 46.5 | 78.3 | 22.8 | 44.9 | 15.4 | 28.6
PlayItBackX3 | 122.8 | 47.0 | 78.7 | 23.1 | 45.1 | 15.9 |  29.2

Table 3: Comparisons to state-of-the-art for EPIC-KITCHENS-100. We report the
top-1 and top-5 accuracies for the verb, noun, and action labels.

Model | freq. | top-1 | top-5 | mAP | AUC | d-prime
---|---|---|---|---|---|---
PlayItBackX0 | 32kHz | 52.1 | 77.8 | 54.7 | 0.970 | 2.757
PlayItBackX0 | 16kHz | 51.8 | 77.4 | 54.3 | 0.966 | 2.743
PlayItBackX1 | 16kHz | 52.5 | 78.3 | 55.1 | 0.972 | 2.789
PlayItBackX2 | 16kHz | 53.2 | 78.7 | 55.5 | 0.976 | 2.810
PlayItBackX3 | 16kHz | 53.7 | 79.2 | 56.1 | 0.978 | 2.846

Table 4: Frequency to playbacks on VGG-Sound given top-1 and top-5 accuracies,
mAP, AUC, and d-prime.

Results. We compare PlayItBack to current state-of-the-art models on AudioSet
in Table 2. PlayItBackX3 achieves the best performance in comparison to other
models.

We report results on VGG-Sound in Table 2. PlayItBackX3 performs favorably to
in-domain audio models. Compared to the previously top-performing SlowFast
model [12], we observe a +1.3%p. top-1 accuracy improvement. Our model is only
outperformed by the multi-modal (audio-visual) version of MBT (AV) [18].
However, PlayItBackX3 outperforms MBT (A) in the audio-only setting.

In Table 3, we compare to audio-classification methods on EPIC-KITCHENS-100.
We observe that the relative improvement in performance varies across datasets
(higher performance gains are observed in AudioSet and VGG-Sound, while
somewhat smaller on EPIC-KITCHENS-100). We believe that this is due to EPIC-
KITCHENS-100 containing audio segments of 2.6s in length on average, compared
to 10s durations of AudioSet and VGG-Sound. As the segments are already
shorter, they intuitively benefit less from further playbacks by focusing on
discriminative regions. Even in such settings, PlayItBackX3 demonstrates a
moderate but consistent performance improvement.

$J$ | top-1 | GFLOPs
---|---|---
1 | 53.3 | 120.4
2 | 53.5 | 121.5
3 | 53.7 | 122.8

Table 5: Number of slot attention iterations ($J$) with respect to the top-1
accuracy and GFLOPs.

Fig. 3: VGG-Sound top-1 accuracy over different playbacknumbers (N) with
respect to the compute (in GFLOPs).

Ablations. Table 2 demonstrates that while PlayItBack uses a sampling
frequency of 16kHz, it can outperform HTS-AT [16] and PaSST [17] which are
trained on sampling frequencies of 32kHz. We confirm this by ablating the
impact on PlayItBack. Table 4 demonstrates that our proposed replays at 16kHz,
can be a better performing strategy than increasing the number of samples per
second over the entire audio sequence as in PlayItBackX0 trained with 32kHz.

Table 5 compares the performance achieved with different numbers of slot
iterations $J$ on VGG-Sound with PlayItBackX3. In general, moderate
performance improvements can be achieved by increasing the number of slot
attention iterations. The added computations also remain moderate with +2.6
GFLOPs from $J=1$ to $J=3$. In Figure 3, we investigate the impact of the
number of playbacks ($N$) on the model performance. We use a decoder-only
model ($N=0$), alongside PlayItBackX$N$. Performance improvements are shown
for $1\leq N\\!\leq\\!3$. Further $N$ increases, come with performance drops,
due to increased model complexity in tandem with the challenge of discovering
salient information in very deep playbacks - and thus very small hop lengths.
As showcased in this figure and across results, $N=3$ offers the best
performance. The performance remains consistent over multiple runs and across
datasets.

##  5 Conclusions

We propose an end-to-end attention-based architecture for audio recognition.
Our PlayItBack model uses information from the full audio sequence to
iteratively discover segments that are relevant to the sound. Audio segments
are discovered with slot attention and amplified in the next iteration
(playback). A transformer decoder is used to relate information across
playbacks. We demonstrate the advantages of our PlayItBack approach through
extensive experiments on AudioSet, VGG-Sound, and EPIC-KITCHENS-100 and
ablation studies.

Future work can explore the selection strategy for the number of playbacks,
which might vary per audio sample. We hope PlayItBack can trigger insights
into similar approaches for other audio signals such as speech as well as
audio-visual fine-grained understanding.

Acknowledgement. We use publicly available datasets. Research is funded by the
United Nation’s End Violence Fund (iCOP 2.0 project) and EPSRC UMPIRE
(EP/T004991/1).

## References

  * [1] Terry Clark,  “Echoic memory explored and applied,”  Journal of services marketing, 1987.
  * [2] Rael D Strous, Nelson Cowan, Walter Ritter, and Daniel C Javitt,  “Auditory sensory (” echoic”) memory dysfunction in schizophrenia.,”  The American journal of psychiatry, 1995.
  * [3] Gabriel A Radvansky,  Human Memory,  Psychology Press, 2005.
  * [4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, et al.,  “An image is worth 16x16 words: Transformers for image recognition at scale,”  in ICLR, 2020.
  * [5] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, et al.,  “Object-centric learning with slot attention,”  in NeuIPS, 2020.
  * [6] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, et al.,  “Audio set: An ontology and human-labeled dataset for audio events,”  in ICASSP, 2017.
  * [7] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman,  “VGGSound: A large-scale audio-visual dataset,”  in ICASSP, 2020.
  * [8] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, et al.,  “Rescaling egocentric vision: collection, pipeline and challenges for EPIC-KITCHENS-100,”  IJCV, 2022.
  * [9] Yuan Gong, Yu-An Chung, and James Glass,  “Psla: Improving audio tagging with pretraining, sampling, labeling, and aggregation,”  IEEE/ACM TASLP, 2021.
  * [10] Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, et al.,  “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,”  IEEE/ACM TASLP, 2020.
  * [11] Yun Wang, Juncheng Li, and Florian Metze,  “A comparison of five multiple instance learning pooling functions for sound event detection with weak labeling,”  in ICASSP, 2019.
  * [12] Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen,  “Slow-fast auditory streams for audio recognition,”  in ICASSP, 2021.
  * [13] Qiuqiang Kong, Yong Xu, Wenwu Wang, and Mark D Plumbley,  “Sound event detection of weakly labelled data with cnn-transformer and automatic threshold optimization,”  IEEE/ACM TASLP, 2020.
  * [14] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, et al.,  “Conformer: Convolution-augmented transformer for speech recognition,”  in Interspeech, 2020.
  * [15] Koichi Miyazaki, Tatsuya Komatsu, Tomoki Hayashi, Shinji Watanabe, et al.,  “Convolution augmented transformer for semi-supervised sound event detection,”  in DCASE, 2020.
  * [16] Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, et al.,  “Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection,”  in ICASSP, 2022.
  * [17] Khaled Koutini, Jan Schlüter, Hamid Eghbal-zadeh, and Gerhard Widmer,  “Efficient training of audio transformers with patchout,”  arXiv preprint, 2021.
  * [18] Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, et al.,  “Attention bottlenecks for multimodal fusion,”  NeurIPS, 2021.
  * [19] Haohe Liu, Xubo Liu, Qiuqiang Kong, Wenwu Wang, and Mark D Plumbley,  “Learning the spectrogram temporal resolution for audio classification,”  arXiv preprint arXiv:2210.01719, 2022.
  * [20] Shenlong Wang, Linjie Luo, Ning Zhang, and Jia Li,  “Autoscaler: Scale-attention networks for visual correspondence,”  in BMVC, 2017.
  * [21] Amir Rosenfeld and Shimon Ullman,  “Visual concept recognition and localization via iterative introspection,”  in ACCV, 2016.
  * [22] Adria Recasens, Petr Kellnhofer, Simon Stent, Wojciech Matusik, and Antonio Torralba,  “Learning to zoom: a saliency-based sampling layer for neural networks,”  in ECCV, 2018.
  * [23] Jianlong Fu, Heliang Zheng, and Tao Mei,  “Look closer to see better: Recurrent attention convolutional neural network for fine-grained image recognition,”  in CVPR, 2017.
  * [24] Alan Baade, Puyuan Peng, and David Harwath,  “Mae-ast: Masked autoencoding audio spectrogram transformer,”  arXiv preprint, 2022.
  * [25] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, et al.,  “Perceiver: General perception with iterative attention,”  in ICML, 2021.
  * [26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in CVPR, 2016.
  * [27] Mingxing Tan and Quoc Le,  “Efficientnet: Rethinking model scaling for convolutional neural networks,”  in ICML, 2019.
  * [28] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, et al.,  “Training data-efficient image transformers & distillation through attention,”  in ICML, 2021.
  * [29] Ze Liu, Yutong Lin, Yue Cao, Han Hu, et al.,  “Swin transformer: Hierarchical vision transformer using shifted windows,”  in ICCV, 2021.
  * [30] Dading Chong, Helin Wang, Peilin Zhou, and Qingcheng Zeng,  “Masked spectrogram prediction for self-supervised audio pre-training,”  arXiv preprint, 2022.
  * [31] Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, et al.,  “Masked autoencoders that listen,”  in NeurIPS, 2022.
  * [32] Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, et al.,  “Mvitv2: Improved multiscale vision transformers for classification and detection,”  in CVPR, 2022.
  * [33] Mark D McDonnell and Wei Gao,  “Acoustic scene classification using deep residual networks with late fusion of separated high and low frequency paths,”  in ICASSP, 2020.
  * [34] Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu,  “Balanced multimodal learning via on-the-fly gradient modulation,”  in CVPR, 2022.
  * [35] Valerii Likhosherstov, Anurag Arnab, Krzysztof Choromanski, Mario Lucic, et al.,  “Polyvit: Co-training vision transformers on images, videos and audio,”  arXiv preprint, 2021.
  * [36] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz,  “mixup: Beyond empirical risk minimization,”  in ICLR, 2018.