# A pre-explosion extended effervescent zone around core collapse supernova
progenitors

Noam Soker Department of Physics, Technion, Haifa, 3200003, Israel;
<EMAIL_ADDRESS>Guangdong Technion Israel Institute of
Technology, Shantou 515069, Guangdong Province, China

###### Abstract

I propose a scenario according to which the dense compact circumstellar matter
(CSM) that the ejecta of many core collapse supernovae (CCSNe) collide with
within several days after explosion results from a dense zone where in
addition to the stellar wind there is gas that does not reach the escape
velocity. In this effervescent zone around red supergiant (RSG) stars, there
are dense clumps that are ejected from the vicinity of the RSG surface, rise
to radii of tens of astronomical units, and then fall back. I consider two
simple velocity distributions of the ejected clumps. I find that the density
of the bound mass can be tens of times that of the escaping wind, and
therefore can mimic a very high mass loss rate. The dense effervescent compact
CSM zone can (1) explain the collision of the ejecta of many CCSNe with a
dense compact CSM days after explosion, (2) facilitate very high mass loss
rate if the star experiences powerful pre-explosion activity, (3) form dust
that obscures the progenitor in the visible band, and (4) lead to an efficient
mass transfer to a stellar companion at separations of tens of astronomical
units, if exists. The effervescent zone might exist for thousands of years and
more, and therefore the effervescent CSM model removes the requirement from
many type II CCSN progenitors to experience a very strong outburst just years
to months before explosion.

stars: massive – stars: mass-loss – supernovae: general

##  1 Introduction

There are two types of indications to the presence of compact circumstellar
matter (CSM) around many, but not all, progenitors of core collapse supernovae
(CCSNe) at explosion. One indication comes from pre-explosion outbursts years
to months before the explosion and the other is the collision of the CCSN
ejecta with compact CSM.

CCSNe are the explosions of stars that leave behind either a neutron star or a
black hole. Stars in the initial mass range of $\gtrsim 10M_{\odot}$ form an
iron core at the end of their nuclear burning phases. When the mass of the
iron core grows to about $1.2M_{\odot}$ the core collapses to form a NS (or
with more mass a black hole). A small fraction of the gravitational energy
that the formation of the neutron star releases explodes the rest of the star.
Stars in the initial mass range of $\simeq 8M_{\odot}$ to $\simeq 10M_{\odot}$
might collapse at an earlier phase, when their core is made up from ONeMg.
Electron capture by magnesium in the degenerate core reduces the pressure and
leads to core collapse, leaving behind a neutron star. In this study I
consider the compact CSM around red supergiant (RSG) progenitors of both types
of core collapse.

Pre-explosion outbursts that are accompanied by high mass loss rate episodes
might occur tens of years to only days prior to explosion (e.g., Foley et al.
2007; Pastorello et al. 2007; Smith et al. 2010; Pastorello et al. 2013;
Margutti et al. 2014; Ofek et al. 2014; Svirski & Nakar 2014; Tartaglia et al.
2016; Strotjohann et al. 2021). In some cases the pre-explosion outburst might
take place as early as carbon-burning phase (e.g., Moriya et al. 2014;
Margutti et al. 2017).

Yaron et al. (2017) argue that the progenitor of SN 2013fs had an enhanced
mass loss rate of $\simeq 0.3-4\times 10^{-3}M_{\odot}~{}\rm{yr}^{-1}$ for a
wind velocity of $v_{\rm w}\approx 15-100~{}\rm{km}~{}\rm{s}^{-1}$. They find
the wind velocity to be $v_{\rm w}\lesssim 100~{}\rm{km}~{}\rm{s}^{-1}$.
Hosseinzadeh et al. (2018) estimate the mass of the compact CSM around the
progenitor of SN 2016bkv to be $\approx 0.04M_{\odot}$. The typical sizes of
the compact CSM that, e.g., Bruch et al. (2020), find around CCSNe, on the
other hand, are $R_{\rm CSM}\approx 10^{15}~{}\rm{cm}$. If this compact CSM is
an outflowing wind, the high mass loss rate begins at about $\simeq R_{\rm
CSM}/v_{\rm w}\approx 3~{}\rm{yr}$, where $v_{\rm w}\simeq
100~{}\rm{km}~{}\rm{s}^{-1}$ is the wind velocity. In some cases the pre-
explosion wind starts only several months before explosion (e.g., Bruch et al.
2020). Prentice et al. (2020) claim that the CSM around the envelope-stripped
SN 2018gjx was ejected within four months from explosion. In that case the CSM
is non-spherical, most likely indicating a pre-explosion binary interaction
(Prentice et al., 2020).

There are theoretical studies that include an enhanced mass loss rate years to
months before explosion as a result of core activity (e.g., Morozova et al.
2020), either set by excitation of waves (e.g., Quataert & Shiode 2012; Shiode
& Quataert 2014; Fuller 2017; Fuller & Ro 2018) or by core-magnetic activity
(e.g., Soker & Gilkis 2017). There are two problems with such a core activity
as an explanation to all cases of SNe II with compact CSM. (1) In many cases
this core activity might lead to substantial envelope expansion, but without
much mass loss rate enhancement (e.g., Mcley & Soker 2014). (2) Bruch et al.
(2020) find that $>30\%$ of SNe II have compact CSM, while the fraction of
CCSNe that suffer a pre-explosion outburst, which the core activity should
excite, is only $\approx 10\%$ (e.g., Margutti et al. 2017).

Although pre-explosion outbursts do occur, in the present study I consider
another model to account for the frequent presence of compact CSM, the
effervescent CSM model. In this model there is a dense bound mass in an
extended zone around the RSG surface (an extended zone relative to the stellar
radius, but at the same time a compact CSM). Dessart et al. (2017) presented
the idea of a long-lived complex extended dense zone around RSG progenitors of
CCSNe that might have up to $\approx 0.01M_{\odot}$. There are some
similarities between their model and the effervescent CSM model. Before
presenting the effervescent CSM model for CCSN progenitors, I turn to describe
the motivation for introducing this model.

In low-mass asymptotic giant branch (AGB) stars that are potential progenitors
of planetary nebulae there are several observations that have lead to the
development of the effervescent CSM (wind) model (Soker, 2008). These
observations include complicated structures of SiO maser clumps (e.g., Cotton
et al. 2006) and a chaotic inflow-outflow motion around the surface of some
AGB stars (e.g., Diamond & Kemball 2003). Water maser observations that
explore regions at larger distances from the surface of some AGB stars,
$r\approx 100~{}\rm{AU}$, also indicate inhomogeneous outflows (e.g.,
Vlemmings et al. 2002). The distribution of dust close to some AGB stars is
also inhomogeneous, and might be related to the magnetic field in the
atmosphere of AGB stars (e.g., Khouri et al. 2020 and references therein).
Mira A is a pulsating AGB star with a radius of $R_{1}\simeq 500R_{\odot}$
(e.g., Wood & Karovska 2006), and with an inhomogeneous and clumpy
asymmetrical tens-AUs extended zone, that is, a compact CSM (e.g., Planesas et
al. 1990; Ryde & Schöier 2001; Lopez et al. 1997). Lopez et al. (1997)
considered a model where dusty clumps $\approx 100$ times denser than their
environment exist at tens of AUs from the AGB star Mira A. They assume these
clumps to explain the IR emission, but did not consider the motion of the
clumps.

The carbon AGB star IRC+10216 further motivates the introduction of the
effervescent CSM model as it has clumps within $\approx 10~{}\rm{AU}$ that
Fonfría et al. (2008) suggest move outward and inward at high velocities along
different radial directions; the fast outward moving clumps reaches distances
of $\gg 10~{}\rm{AU}$. The post-AGB star HD56126 has both outflowing and
inflowing gas around it at velocities of up to $\simeq
20~{}\rm{km}~{}\rm{s}^{-1}$ (Klochkova & Chentsov, 2007). These velocities are
non-negligible with respect to the escape velocity of $\simeq
60~{}\rm{km}~{}\rm{s}^{-1}$ from this star (Li, 2003).

More relevant to the present study are observations of inhomogeneous winds in
RSGs (e.g., Lobel & Dupree 2000; Humphreys et al. 2007), including massive
dust clumps around the RSG VY CMa (Kamiński, 2019). Observations (e.g.,
Josselin & Plez 2007) and theoretical studies (e.g., Freytag et al. 2002)
suggest that such inhomogeneous winds might result from convective cells in
the envelope and/or magnetic activity of the giant stars. Boian & Groh (2020)
argue that CCSNe with CSM tend to come from high mass RSGs, i.e., having zero
age main sequence mass of $M_{\rm ZAMS}\gtrsim 15M_{\odot}$. This also
suggests luminous CCSN progenitors that can lift gas above the photosphere.
There are studies of extended region where inflow and outflow coexist around
stars that are close to their Eddington luminosity limit (e.g., Owocki & van
Marle 2008; van Marle et al. 2009).

The above studies motivated me to introduce the effervescent CSM model to
upper AGB stars (Soker, 2008). The basic postulate of this model is that the
zone up to about 100 AUs is inhomogeneous and contains many clumps that do not
escape the star, but rather fall back.

Moriya et al. (2017) consider a dense compact CSM that results from the
acceleration zone of the wind. Namely, instead of a constant wind velocity
they consider a wind with a velocity that increases with radius, a profile
that makes the density higher than of a constant-velocity wind very close to
the stellar surface (also Moriya et al. 2018). The effervescent CSM model is
different in some key aspects that I discuss later.

Dessart et al. (2017) simulate CCSN explosion inside an extended complex CSM
that is bound or a dense wind. Numerically, they took either an atmosphere
with an extended scale height and/or a dense wind. The effervescent CSM model
share some properties and implications with their complex extended zone. They
cite as motivation for considering such an extended zone the observations of
the RSG Betelgeuse that show outflows and inflows (down-flows) out to several
RSG stellar radii (e.g., Ohnaka et al. 2011; Kervella et al. 2016). Dessart et
al. (2017) noted that the extended zone removes the requirement for a fine-
tuned stellar activity years to months before explosion. I also consider an
extended zone, and in that I overlap with their study, but I consider the
effervescent model of clumps that move out and fall back. I borrow the
effervescent CSM model from low mass AGB progenitors of planetary nebulae.
Nonetheless, I actually strengthen the claim of Dessart et al. (2017) that an
extended zone around the photosphere of many RSG progenitors of CCSNe might
explain observations.

In section 2 I present the basic condition for the presence of an effervescent
CSM, and in section 3 I estimate the outer boundary of the effervescent zone.
In section 4 I present one type of model for the effervescent zone and in
section 5 I present another type. I summarise in section 6.

##  2 The conditions for a dense effervescent zone

Stellar pulsations, magnetic activity, rotation, and/or strong convection
bring gas to the zone above and close to the photosphere. The stellar
radiation cannot bring all this gas to escape, but brings a large fraction to
almost escape velocities. This implies that the effervescent zone will be
dense when the stellar radiation cannot accelerate most of the gas around the
photosphere to escape velocity. In other words, the effervescent zone will
become extended when the radiation momentum flux is about equal to the wind
momentum flux. This might be the case during regular evolution of massive RSG
stars, and more so for all RSG when powerful convection in the core (e.g.,
Quataert & Shiode 2012; Fuller 2017; Fuller & Ro 2018) and/or a powerful
dynamo in the core (e.g., Soker & Gilkis 2017) drive stronger convection in
the envelope. In turn, the stronger envelope convection pushes more gas above
the photosphere. This process does not require the power of these activities
to be more than what convection in the envelope can carry as the direct mass
ejection mechanism requires. For that, this process of mass lifting can start
when core activity is still weak, implying many years, even thousands of years
and more, before the explosion. There is no need for a fine tuning of the
activity to form the compact CSM.

At that stage when the radiation cannot accelerate most of the gas to the
escape speed the mass loss rate in the wind is $\dot{M}_{\rm
wc}\simeq\eta_{\rm w}L/(cv_{w})$, where $v_{w}$ is the terminal wind speed,
$L$ the stellar luminosity, and $\eta_{\rm w}$ is the average net effective
times that a photon transfers momentum to the wind (only in the outward radial
direction). In most cases $\eta_{\rm w}<1$, but in dense and opaque winds it
can be somewhat larger than $1$. Substituting typical values gives

$\displaystyle\begin{aligned} \dot{M}_{\rm wc}&\simeq 4\times 10^{-5}\eta_{\rm
w}\left(\frac{L}{2\times 10^{5}L_{\odot}}\right)\\\
&\times\left(\frac{v_{w}}{100~{}\rm{km}~{}\rm{s}^{-1}}\right)^{-1}M_{\odot}~{}\rm{yr}^{-1}.\end{aligned}$
(1)

This shows that the effervescent zone becomes significant when mass loss rate
is high. But when we consider the effervescent zone, the mass loss rate into
the wind need not be as high as estimates that do not consider the
effervescent zone require.

##  3 A single clump

Let us compare the different forces that act on a clump that is $k_{b}$ times
denser than the ambient wind density, $\rho_{b}=k_{b}\rho_{w}$, and it moves
at radial distances of tens of $~{}\rm{AU}$, i.e., $r\gg R_{\ast}$, where
$R_{\ast}$ is the stellar radius. The wind density is
$\rho_{w}=\dot{M}_{w}/4\pi r^{2}v_{w}$, where $\dot{M}_{w}$ is the mass loss
rate into the wind, and $v_{w}$ the wind velocity. I further characterise the
clump with its cross section facing the star (perpendicular to the radial
direction) $A_{b}$ and its length in the radial direction $l_{b}$. I
schematically draw the effervescent zone in Figure 1.

---
Figure 1: A schematic drawing of the effervescent zone (not to scale). The
thick-blue arrows depict the escaping wind at its more than escape velocity
$v_{w}$. The red-oval clouds depict the dense clumps that rise and fall within
the effervescent zone. The orange sphere at the center is the RSG star of
radius $R_{\ast}$. The outer edge of the effervescent zone is at $R_{\rm
eff}$.

The (radial) gravitational force on the clump due to the RSG star of mass
$M_{\ast}$ is

$F_{g}=-\frac{GM_{\ast}}{r^{2}}\frac{\dot{M}_{w}}{4\pi
r^{2}v_{w}}k_{b}A_{b}l_{b}.$ (2)

The regular wind exerts drag force on the clump. In this simple treatment I
take this force to be

$F_{w}\simeq\frac{\dot{M}_{w}v_{w}}{4\pi r^{2}}A_{b}.$ (3)

Namely, due to non-smooth clump surface I assume that the clump absorbs all
the momentum of the wind that hits it. For an optical depth of $\tau_{b}$
along the radial direction of the clump the radiation exerts a force of

$F_{\rm rad}=A_{b}\frac{L}{4\pi r^{2}c}\left(1-e^{-\tau_{b}}\right).$ (4)

I assumed above that the radiation pressure can accelerate both the wind that
hits the clump, and the clump. The justification is that the radiation
accelerates the wind within several stellar radii, whereas I consider the
clumps at $r\gtrsim 10R_{\ast}$, where $R_{\ast}$ is the stellar radius. These
distances are outside the winds’ acceleration zone and the stellar radiation
is close to the photospheric radiation. Namely, I assume that the wind is
optically thin while the denser clumps are cooler and have dust. The opacity
of the dusty clump is $\kappa_{b}\approx 10~{}\rm{cm}^{2}~{}\rm{g}^{-1}$,
while that of the cool and partially neutral wind that contains much less dust
is much lower. Overall, at tens of AUs the optical depth in the clumps is much
larger than in the wind. However, further out the inner clumps make the
radiation redder, and for this longer wavelength band the dust opacity becomes
low.

The condition on the blob acceleration to be down (in the $-r$ direction) is
$F_{g}+F_{\rm rad}+F_{w}<0$. Using equations (2)-(4) I find this condition to
read

$\displaystyle\begin{aligned} 1>\frac{F_{\rm
rad}+F_{w}}{-F_{g}}&\simeq\frac{v^{2}_{w}}{v^{2}_{\rm
Kep}}\frac{r}{R_{\ast}}\frac{r}{l_{b}}k^{-1}_{b}\\\
&\times\left[\frac{L/c}{\dot{M}_{w}v_{w}}\left(1-e^{-\tau_{b}}\right)+1\right],\end{aligned}$
(5)

where $v_{\rm Kep}=\sqrt{GM_{\ast}/R_{\ast}}$ is the Keplerian velocity on the
surface of the star.

I assume that the clump expands radially, e.g., $A_{b}\propto r^{2}$, but that
its radial length $l_{b}$ stays constant. I consider two limits. If the
reddening of the stellar radiation by dust close to the star is significant,
then the low dust opacity to this band implies that $\tau_{b}$ might be low.
In addition, the low clump’s density at large distances also reduces
$\tau_{b}$. In that limit I neglect the first term in the square parenthesis
of equation (5), and so the requirement on the clump’s acceleration to be
negative (down to the star) reads

$\frac{r}{R_{\ast}}\lesssim\frac{v_{\rm
Kep}}{v_{w}}\left(\frac{l_{b}}{R_{\ast}}\right)^{1/2}k^{1/2}_{b}.$ (6)

The demand on the clumps to form an extended effervescent zone is that the
value of $k_{b}$ be large. I emphasise again here that it is not radiation
pressure that eject the clumps from the stellar surface, but rather stellar
pulsation, stellar convection, and magnetic activity in the outer envelope
where gravity is relatively weak.

If on the other hand radiation pressure exerts a larger outward force on the
clump than that of the wind, then I approximate $(1-e^{-\tau_{b}})^{1/2}\simeq
1$, and the condition reads

$\frac{r}{R_{\ast}}\lesssim\frac{v_{\rm
Kep}}{v_{w}}\left(\frac{\dot{M}_{w}v_{w}}{L/c}\right)^{1/2}\left(\frac{l_{b}}{R_{\ast}}\right)^{1/2}k^{1/2}_{b}.$
(7)

Since we take $\dot{M}_{w}v_{w}\simeq L/c$ (equation 1), equations (6) and (7)
are practically the same for the very evolved massive stars I study here.

The conclusion is that to form an extended effervescent zone around the star
the clumps should be hundreds to thousands of times denser than the average
density of the wind. The extension of the effervescent zone is

$\displaystyle\begin{aligned} R_{\rm eff}&\approx 6.7\times
10^{14}\left(\frac{k_{b}}{1000}\right)^{1/2}\left(\frac{R_{\ast}}{2~{}\rm{AU}}\right)\\\
&\times\left(\frac{l_{b}}{R_{\ast}}\right)^{1/2}\left(\frac{v_{w}}{v_{\rm
esc}}\right)^{-1}~{}\rm{cm},\end{aligned}$ (8)

where I assume that the wind is saturated in the sense that
$\dot{M}_{w}v_{w}\simeq L/c$ and that the wind speed is about the escape speed
from the stellar surface $v_{\rm esc}=2^{1/2}v_{\rm Kep}$.

The density in the clump is

$\displaystyle\begin{aligned} \rho_{b}\simeq&9\times
10^{-11}\left(\frac{\dot{M}_{\rm wc}}{4\times
10^{-5}}\right)\left(\frac{r}{1~{}\rm{AU}}\right)^{-2}\\\
&\times\left(\frac{v_{w}}{100~{}\rm{km}~{}\rm{s}^{-1}}\right)^{-1}\left(\frac{k_{b}}{1000}\right)~{}\rm{g}~{}\rm{cm}^{-3}.\end{aligned}$
(9)

This density is more than one order of magnitude smaller than the density at
the photosphere of RSG stars, $\rho_{p}$. As examples, a stellar model of zero
age main sequence star of $M_{\rm ZAMS}=15M_{\odot}$ has a photospheric
density of $\rho_{p}(1~{}\rm{AU})=5\times 10^{-9}~{}\rm{g}~{}\rm{cm}^{-3}$ and
$\rho_{p}(2~{}\rm{AU})=3\times 10^{-9}~{}\rm{g}~{}\rm{cm}^{-3}$, at stellar
radii of $R_{\ast}=1~{}\rm{AU}$ and $R_{\ast}=2~{}\rm{AU}$, respectively,
along its evolution (I obtained these values from simulating stellar models
with the open stellar evolution code mesa; Paxton et al. 2018). A stellar
model of $M_{\rm ZAMS}=30M_{\odot}$ has a photospheric density of
$\rho_{p}(2~{}\rm{AU})=4\times 10^{-9}~{}\rm{g}~{}\rm{cm}^{-3}$ and
$\rho_{p}(5~{}\rm{AU})=6\times 10^{-10}~{}\rm{g}~{}\rm{cm}^{-3}$, at stellar
radii of $R_{\ast}=2~{}\rm{AU}$ and $R_{\ast}=5~{}\rm{AU}$, respectively. This
implies that the star might lift such clumps above the photosphere.

##  4 An effervescent zone from a group of unique velocity clumps

###  4.1 The average density

Consider a case when the activity in the envelope of the RSG star (pulsation,
convection, rotation, magnetic activity, powerful radiation) ejects bound
mass, i.e., with less than the escape velocity, at a rate of

$\dot{M}_{\rm eff}=\beta\dot{M}_{w}.$ (10)

Consider also that the main force on the clumps is gravity, as the other
forces that we consider in section 2 play a role at very large radii where we
assume that the effervescent zone is already depleted. The time it takes a
clump to reach a maximum radius $R_{\rm eff}$ and fall back to the star is

$t_{\rm eff}\simeq\frac{2\pi}{2^{3/2}}\frac{R_{\ast}}{v_{\rm
Kep}}\left(\frac{R_{\rm eff}}{R_{\ast}}\right)^{3/2}.$ (11)

A wind element spends a time of $t_{w}(R_{\rm eff})\simeq R_{\rm eff}/v_{w}$
in the effervescent zone. The ratio of the average density of the bound mass
to that of the wind (total mass divided by total volume) in the effervescent
zone is therefore

$\displaystyle\begin{aligned} \frac{\bar{\rho}_{\rm
eff}}{\bar{\rho}_{w}}&\simeq\frac{\pi}{\sqrt{2}}\left(\frac{R_{\rm
eff}}{R_{\ast}}\right)^{1/2}\left(\frac{v_{w}}{v_{\rm Kep}}\right)\beta\\\
&=17\left(\frac{R_{\rm eff}}{30R_{\ast}}\right)^{1/2}\left(\frac{v_{w}}{v_{\rm
esc}}\right)\beta,\end{aligned}$ (12)

where in the second equality I scaled the wind velocity with the escape
velocity from the RSG star.

Of course, many clumps will reach much smaller radii and spend much less time
in the effervescent zone. On the other hand, for active RSG stars I expect
$\beta>1$. Namely, most of the gas that the envelope activity lifts around the
photosphere does not reach the escape velocity. The clumps move at a lower
velocity in the outer regions of the effervescent zone, so the density ratio
in the outer regions of the effervescent zone is larger even. For example,
$55\%$ of its round trip the clump spends in the outer 20 percent of the
effervescent zone, $0.8R_{\rm eff}$ to $R_{\rm eff}$. For the same parameters
I used in scaling equation (12) the ratio of average densities in this outer
region of the effervescent zone, $\left({\bar{\rho}_{\rm
eff}}/{\bar{\rho}_{w}}\right)_{0.8-1}\simeq 47$.

I emphasise two points. (1) This large density ratio exists despite that the
mass ejection rate into the effervescent zone is about equal to that in the
wind, $\beta\simeq 1$. (2) The derivation in this section assumes that the RSG
activity brings the wind to the escape velocity, but a similar amount of mass
to be close to, but below, the escape velocity. The process of acceleration
takes place along several stellar radii, and therefore the derivation here are
approximate. Nonetheless, equation (12) does give the general behavior of the
effervescent zone.

###  4.2 The density profile

I consider that all clumps are ejected to radius $R_{\rm eff}$. In that case
the average density at each radius, but not within a distance of $l_{b}$ from
$R_{\rm eff}$ is

$\rho_{\rm eff}(r)\simeq 2\frac{\dot{M}_{\rm eff}}{4\pi r^{2}v_{b}}\qquad{\rm
for}\qquad R_{\ast}<r<R_{\rm eff}-l_{b},$ (13)

where the factor 2 comes from that each clump moves outward and fall back. As
above, I neglect the forces on the clumps due to radiation and the wind, and
so the velocity of the clump is

$v_{b}\simeq\sqrt{\frac{2GM_{\ast}}{R_{\rm eff}}}\sqrt{\frac{R_{\rm
eff}}{r}-1}.$ (14)

With the aid of equations (10) and (14), equation (13) becomes

$\displaystyle\begin{aligned} &\rho_{\rm eff}(r)\simeq
2\frac{\beta\dot{M}_{w}}{4\pi r^{2}v_{\rm esc}}\left(\frac{R_{\rm
eff}}{R_{\ast}}\right)^{1/2}\left(\frac{R_{\rm eff}}{r}-1\right)^{-1/2}\\\
&=2\beta\rho_{w}(r)\left(\frac{v_{w}}{v_{\rm esc}}\right)\left(\frac{R_{\rm
eff}}{R_{\ast}}\right)^{1/2}\left(\frac{R_{\rm eff}}{r}-1\right)^{-1/2}\\\
&\qquad{\rm for}\qquad R_{\ast}<r<R_{\rm eff}-l_{b}.\end{aligned}$ (15)

I note the following properties of the above density profile.
(1) For $R_{\rm eff}\gg R_{\ast}$, as is the case in the present study, close
to the star, i.e. at $r\simeq R_{\ast}$ the density of the bound gas is
$\rho_{\rm eff}\simeq 2\beta\rho_{w}$.
(2) For the above simple density profile case, the minimum value of the
density is at $r=0.75R_{\rm eff}$. At that radius

$\left(\rho_{\rm eff}\right)_{\rm min}=\rho_{\rm eff}(0.75R_{\rm eff})\simeq
19\beta\left(\frac{R_{\rm eff}}{30R_{\ast}}\right)^{1/2}\rho_{w}.$ (16)

This ratio increases to, e.g., $33$ at $r=0.9R_{\rm eff}$, keeping other
parameters the same.
(3) The density profile is quite flat in an extended region (equation 15). As
three examples,

$\rho_{\rm eff}=\rho_{\rm eff}(0.75R_{\rm eff})\times\left\\{\begin{array}[]{
l r }1.57&{}\quad{\rm at}\quad r=0.95R_{\rm eff}\\\ 1.19&{}\quad{\rm at}\quad
r=0.55R_{\rm eff}\\\ 1.66&{}\quad{\rm at}\quad r=0.40R_{\rm
eff}.\end{array}\right.$ (17)

Overall, if we consider the increase in density near $R_{\rm eff}$ that gives
the high average density in the outer regions of the effervescent zone
(equation 12 and the discussion below it), we see that the ejection at a mass
rate about equal to that of the wind might mimic a short-lived wind with a
mass loss rate that is tens times larger than the real mass loss rate.

##  5 An example of velocity distribution

In the first paper on the effervescent zone that aimed at AGB stars (Soker
2008) I concentrated on a simple derivation of a density profile for a
specific case (for more details see that paper). I assumed that the bound gas
is ejected from a radius of $R_{0}\approx{\rm few}\times R_{\ast}$ (for the
motivation to take $R_{0}\approx{\rm few}\times R_{\ast}$ for the extended
dense zone see Soker 2008 and Dessart et al. 2017), and I took the mass
ejection rate in the velocity interval $v$ to $v+dv$ as

$d\dot{M}_{e}=f\dot{M}_{w}\left(\frac{v}{v_{\rm
esc,0}}\right)^{q}\frac{dv}{v_{\rm esc,0}}\quad{\rm for}\quad 0<v<v_{m},$ (18)

where $v_{\rm esc,0}$ is the escape speed from $R_{0}$ and $q$ ($-k_{v}$ in
Soker 2008) is a constant of the model. The maximum velocity $v_{m}$ is for
clumps that reach the outer boundary of the effervescent zone $R_{\rm eff}$,
with the relation $v_{m}={v_{\rm esc,0}}\left(1-R_{0}/R_{\rm
eff}\right)^{1/2}$ . The total rate of unbound mass that the star ejects to
the effervescence zone is

$\displaystyle\begin{aligned} \dot{M}_{\rm
eff}&=\frac{f}{1+q}\left(\frac{v_{m}}{v_{\rm
esc,0}}\right)^{1+q}\dot{M}_{w}\\\ &=\frac{f}{1+q}\left(1-\frac{R_{0}}{R_{\rm
eff}}\right)^{\frac{1+q}{2}}\dot{M}_{w},\end{aligned}$ (19)

From this, $f=(1+q)\left(1-R_{0}/R_{\rm eff}\right)^{-(1+q)/2}\beta$, where I
defined $\beta$ in equation (10).

As I expect the very luminous RSG stars to lift more gas closer to the escape
speed I take $q\gg 1$ (in the first paper I took low values of $q\approx 1$,
and even negative values). Under the very crude assumption that each dense
clump spends all the time at the maximum radius of its up and down trajectory
I derived (Soker 2008) the ratio of the bound density to the escaping wind
density as

$\displaystyle\begin{aligned} &\frac{\rho_{\rm
eff}}{\rho_{w}}\approx(1+q)\beta\left(1-\frac{R_{0}}{R_{\rm
eff}}\right)^{-\frac{1+q}{2}}\frac{\pi}{2}\left(\frac{v_{w}}{v_{\rm
esc}}\right)\left(\frac{R_{0}}{R_{\ast}}\right)\\\
&\times\left(\frac{r}{R_{\ast}}\right)^{-1/2}\left(1-\frac{R_{0}}{r}\right)^{\frac{q-1}{2}}\quad{\rm
for}\quad R_{0}<r<R_{\rm eff},\end{aligned}$ (20)

where as before $v_{\rm esc}$ is the escape velocity from the stellar surface.
I scale equation (20) to allow comparison to equations (16) and (17)

$\displaystyle\begin{aligned} &\frac{\rho_{\rm eff}}{\rho_{w}}\approx
9.3\frac{(1+q)\beta}{10}\left(\frac{v_{w}}{v_{\rm
esc}}\right)\left(\frac{R_{0}}{3R_{\ast}}\right)\left(\frac{r}{20R_{\ast}}\right)^{-1/2}\\\
&\times\left[\left(1-\frac{R_{0}}{R_{\rm
eff}}\right)^{-\frac{1+q}{2}}\frac{1}{1.69}\right]\\\
&\times\left[\left(1-\frac{R_{0}}{r}\right)^{\frac{q-1}{2}}\frac{1}{0.52}\right]\quad{\rm
for}\quad R_{0}<r<R_{\rm eff},\end{aligned}$ (21)

where I normalised terms for $q=9$, $r=(20/3)R_{0}=20R_{\ast}$, and $R_{\rm
eff}=10R_{0}=30R_{\ast}$ (for $R_{0}=3R_{\ast}$).

Keeping all other parameters the same, the ratio in equation (21) at two other
radii are ${\rho_{\rm eff}}/{\rho_{w}}\approx 8.4$ and $9.6$, at
$r=5R_{0}=15R_{\ast}$ and $r=10R_{0}=R_{\rm eff}$, respectively. Over all, the
density in the outer regions of this version of the effervescent zone
decreases more or less as $r^{-2}$, as the wind does. To increase the density
in the outer regions of the effervescent zone to tens times the wind density
for this distribution of clump velocities, would require taking $\beta>1$
and/or increasing further the value of $q$.

##  6 Summary and implications

I presented a phenomenological model for an extended high density zone around
RSG stars at the end of their life. The basic assumption is that dense clumps,
much denser than the escaping wind, move up and fall back. I term this the
effervescent CSM model. The effervescent zone extends to tens of stellar
radii, i.e., to $R_{\rm eff}\approx 10-100~{}\rm{AU}\approx
10^{14}-10^{15}~{}\rm{cm}$ for RSG stars. Such an effervescent zone might
exist only when the RSG is large, such that surface gravity is relatively
weak, and the radial momentum flux of the (escaping) wind is about equal to
the momentum flux of the stellar radiation (equation 1).

The effervescent CSM model assumes that during such a stellar evolutionary
phase the stellar activity (rotation, convection, magnetic activity,
disturbances from the vigorous core nuclear burning; see section1) lifts large
amounts of mass above the photosphere, but the stellar radiation manages to
unbound only about half of that mass or less. The rest almost escapes, but
falls back after reaching large radii. To fall back rather than be accelerated
by radiation and the wind, the effervescent zone extends to no more than tens
to about one hundred of AUs (equation 8).

I considered two phenomenological distributions of clumps’ velocities (after
accelerated out). In the first one (section 4), all clumps move at one
velocity very close to the escape velocity. I found that the average density
of the bound gas (equation 12) is tens times larger than that of the escaping
wind, in particular in the outer regions of the effervescent zone (equations
16, 17).

For example, consider a case where the outer radius of the effervescent zone
is 20 times the stellar radius of a large RSG star of stellar radius
$R_{\ast}=2~{}\rm{AU}$, i.e., $R_{\rm eff}\simeq 20R_{\ast}\simeq
40~{}\rm{AU}\simeq 6\times 10^{14}~{}\rm{cm}$. For the escaping wind
properties as given by equation (1), which for $v_{w}=v_{\rm esc}$ corresponds
to an RSG star of mass $M_{\ast}=11.3M_{\odot}$, the total wind mass inside
$r<R_{\rm eff}$ is $M_{w}(R_{\rm eff})\simeq 7\times 10^{-5}M_{\odot}$.
However, the mass of the bound gas is $14\beta$ times larger (equation 12),
i.e., $M_{\rm eff}\simeq 0.001\beta M_{\odot}$. However, in the very outer
parts of the effervescent zone this ratio can be as large as $\approx 100$,
mimicking a mass loss rate in the last year or so of $\approx
0.001-0.01M_{\odot}~{}\rm{yr}^{-1}$. This might explain some cases of CCSNe
with compact CSM, e.g., in SN 2013fs for which Yaron et al. (2017) estimated
an enhanced mass loss rate of $\simeq 0.3-4\times
10^{-3}M_{\odot}~{}\rm{yr}^{-1}$.

Dessart et al. (2017) already considered a dense bound gas around RSG
progenitors of CCSNe. In the present study I presented calculations of
outflowing and in-flowing gas to obtain the density profile, rather than
assuming it or taking an extended stellar atmosphere. Dessart et al. (2017)
estimate the mass around the progenitor of SN 2013fs to have been $\approx
0.01M_{\odot}$ spread over $\approx 2\times 10^{14}~{}\rm{cm}$. To account for
this amount of mass we would require $\beta\approx 10$ in the frame of the
effervescent CSM model that I studeid here. To account for a compact CSM of
$0.04M_{\odot}$ as in SN 2016bkv (Hosseinzadeh et al., 2018) or of
$0.07M_{\odot}$ as in SN 2018cuf (Dong et al., 2021), we would require to take
$\beta\simeq 10$. It is a large value, but not unreasonable in the
effervescent model if the RSG experiences strong stellar activity, e.g., due
to rapid rotation.

In section 5 I adopted the velocity distribution of the clumps (equation 18)
from my earlier paper (Soker, 2008). I followed the treatment from that paper,
and derived the density profile for this clumps’ velocities distribution in
equations (20) and (21). This case requires values of $\beta>1$ to achieve
high densities in the outer effervescent zone.

Prentice et al. (2020) study the type IIb (envelope-stripped) SN 2018gjx and
argue that its CSM was non-spherical, probably a torus, and extended to about
$20-30~{}\rm{AU}$ at explosion. They further estimate the mass in the CSM to
be $\approx 0.004-0.014M_{\odot}$ and the mass loss rate from the progenitor
that formed this CSM to be $\approx 0.01-0.05M_{\odot}~{}\rm{yr}^{-1}$. If
indeed the CSM is in a torus, this geometry suggests a strong binary
interaction (e.g., Gofman & Soker 2019) that probably spun-up the progenitor.
With rapid rotation the stellar progenitor might be able to lift a dense
equatorial effervescent zone with the required mass. I therefore raise the
possibility that even in these non type II CCSN the compact CSM might be an
effervescent zone rather than an intensive escaping wind.

The main motivation to consider the effervescent zone in RSG stars is to
remove both the requirement for fine-tuned and strong stellar activity years
to months before explosion. The effervescent CSM model can exist for a long
time, thousands of years and more before explosion, and the flow structure by
which the gas does not reach the escape speed does not require strong
activity. Moderate activity due to some extra rotation and/or moderate core
activity might be sufficient.

Another effect of the effervescent zone is an extra mass transfer to a
companion, if exists within and close to the effervescent zone (Soker, 2008).
To have a high mass transfer rate it is sufficient that the effervescent zone
overfills the Roche lobe of the RSG star, such that mass transfer through the
first Lagrangian point takes place (Harpaz et al. 1997; Mohamed &
Podsiadlowski 2007, 2012; Chen et al. 2017; Saladino et al. 2018; Chen et al.
2020; this is also termed wind-Roche lobe overflow). Due to the high specific
angular momentum of the transferred mass, it will form an accretion disk
around the secondary star (if it is not a giant). In turn, the accretion disk
is likely to launch two jets that will shape the CSM (e.g., Hillel et al.
2020).

## Acknowledgments

I thank Avishai Gilkis, Amit Kashi and an anonymous referee for useful
comments. I thank Aldana Grichener for simulating stellar models. This
research was supported by a grant from the Israel Science Foundation (420/16
and 769/20) and a grant from the Asher Space Research Fund at the Technion.

## References

  * Boian & Groh (2020) Boian, I. & Groh, J. H. 2020, MNRAS, 496, 1325
  * Bruch et al. (2020) Bruch, R. J., Gal-Yam, A., Schulze, S., et al. 2020, arXiv:2008.09986
  * Chen et al. (2017) Chen, Z., Frank, A., Blackman, E. G., Nordhaus, J., & Carroll-Nellenback J. 2017, MNRAS, 468, 4465
  * Chen et al. (2020) Chen, Z., Ivanova, N., & Carroll-Nellenback, J. 2020, ApJ, 892, 110
  * Cotton et al. (2006) Cotton, W. D., Vlemmings, W., Mennesson, B., et al. 2006, A&A, 456, 339
  * Dessart et al. (2017) Dessart, L., John Hillier, D., & Audit, E. 2017, A&A, 605, A83
  * Diamond & Kemball (2003) Diamond, P. J. & Kemball, A. J. 2003, ApJ, 599, 1372
  * Dong et al. (2021) Dong, Y., Valenti, S., Bostroem, K. A., et al. 2021, arXiv:2010.09764
  * Foley et al. (2007) Foley, R. J., Smith, N., Ganeshalingam, M., Li, W., Chornock, R., & Filippenko, A. V. 2007, ApJ, 657, L105
  * Fonfría et al. (2008) Fonfría, J. P., Cernicharo, J., Richter, M. J., & Lacy, J. H. 2008, ApJ, 673, 445
  * Freytag et al. (2002) Freytag, B., Steffen, M., & Dorch, B. 2002, Astronomische Nachrichten, 323, 213
  * Fuller (2017) Fuller, J. 2017, MNRAS, 470, 1642
  * Fuller & Ro (2018) Fuller, J. & Ro, S. 2018, MNRAS, 476, 1853
  * Gofman & Soker (2019) Gofman, R. A. & Soker, N. 2019, MNRAS, 488, 5854
  * Harpaz et al. (1997) Harpaz, A., Rappaport, S., & Soker, N. 1997, ApJ, 487, 809
  * Hillel et al. (2020) Hillel, S., Schreier, R., & Soker, N. 2020, ApJ, 891, 33
  * Hosseinzadeh et al. (2018) Hosseinzadeh, G., Valenti, S., McCully, C., et al. 2018, ApJ, 861, 63
  * Humphreys et al. (2007) Humphreys, R. M., Helton, L. A., & Jones, T. J. 2007, AJ, 133, 2716
  * Josselin & Plez (2007) Josselin, E. & Plez, B. 2007, A&A, 469, 671
  * Kamiński (2019) Kamiński, T. 2019, A&A, 627, A114
  * Kervella et al. (2016) Kervella, P., Lagadec, E., Montargès, M., et al. 2016, A&A, 585, A28
  * Khouri et al. (2020) Khouri, T., Vlemmings, W. H. T., Paladini, C., et al. 2020, A&A, 635, A200
  * Klochkova & Chentsov (2007) Klochkova, V. G. & Chentsov, Y. L. 2007, Astronomy Reports, 51, 994
  * Li (2003) Li, A. 2003, ApJ, 599, L45
  * Lobel & Dupree (2000) Lobel, A. & Dupree, A. K. 2000, ApJ, 545, 454
  * Lopez et al. (1997) Lopez, B., Danchi, W. C., Bester, M., et al. 1997, ApJ, 488, 807
  * Margutti et al. (2017) Margutti, R., Kamble, A., Milisavljevic, D., et al. 2017, ApJ, 835, 140
  * Margutti et al. (2014) Margutti, R., Milisavljevic, D., Soderberg, A. M., et al. 2014, ApJ, 780, 21
  * Mcley & Soker (2014) Mcley, L., & Soker, N. 2014, MNRAS, 445, 2492
  * Mohamed & Podsiadlowski (2007) Mohamed, S., & Podsiadlowski, P. 2007, 15th European Workshop on White Dwarfs, 372, 397
  * Mohamed & Podsiadlowski (2012) Mohamed, S., & Podsiadlowski, P. 2012, Baltic Astronomy, 21, 88
  * Moriya et al. (2018) Moriya, T. J., Förster, F., Yoon, S.-C., Gräfener, G., & Blinnikov, S. I., 2018, MNRAS, 476, 2840
  * Moriya et al. (2014) Moriya, T. J., Maeda, K., Taddia, F., Sollerman, J., Blinnikov, S. I., Sorokina, E. I. 2014, MNRAS, 439, 291
  * Moriya et al. (2017) Moriya, T. J., Yoon, S.-C., Gräfener, G., & Blinnikov, S. I. 2017, MNRAS, 469, L108
  * Morozova et al. (2020) Morozova, V., Piro, A. L., Fuller, J., & Van Dyk S. D. 2020, ApJ, 891, L32
  * Ofek et al. (2014) Ofek, E. O., Sullivan, M., Shaviv, N. J., et al. 2014, ApJ, 789, 104
  * Ohnaka et al. (2011) Ohnaka, K., Weigelt, G., Millour, F., et al. 2011, A&A, 529, A163
  * Owocki & van Marle (2008) Owocki, S. P., & van Marle, A. J., 2008, in Massive Stars as Cosmic Engines, IAU Symp 250, ed. F. Bresolin, P. A. Crowther, & J. Puls (Cambridge Univ. Press) Volume 250, p. 71-82.
  * Pastorello et al. (2013) Pastorello, A., Cappellaro, E., Inserra, C., et al. 2013, ApJ, 767, 1
  * Pastorello et al. (2007) Pastorello, A., Smartt, S. J., Mattila, S., et al. 2007, Nature, 447, 829
  * Paxton et al. (2018) Paxton, B., Schwab, J., Bauer, E. B., et al. 2018, The Astrophysical Journal Supplement Series, 234, 34.
  * Planesas et al. (1990) Planesas, P., Bachiller, R., Martin-Pintado, J., & Bujarrabal, V. 1990, ApJ, 351, 263
  * Prentice et al. (2020) Prentice, S. J., Maguire, K., Boian, I., et al. 2020, arXiv:2009.10509
  * Quataert & Shiode (2012) Quataert, E., & Shiode, J. 2012, MNRAS, 423, L92
  * Ryde & Schöier (2001) Ryde, N. & Schöier, F. L. 2001, ApJ, 547, 384
  * Saladino et al. (2018) Saladino, M. I., Pols, O. R., van der Helm, E., Pelupessy, I., & Portegies Zwart S. 2018, A&A, 618, A50
  * Shiode & Quataert (2014) Shiode, J. H., & Quataert, E. 2014, ApJ, 780, 96
  * Smith et al. (2010) Smith, N., Miller, A., Li, W., et al. 2010, AJ, 139, 1451
  * Soker (2008) Soker, N. 2008, New A, 13, 491
  * Soker & Gilkis (2017) Soker, N. & Gilkis, A. 2017, MNRAS, 464, 3249
  * Strotjohann et al. (2021) Strotjohann, N. L., Ofek, E. O., Gal-Yam, A., et al. 2021, arXiv:2010.11196
  * Svirski & Nakar (2014) Svirski, G., & Nakar, E. 2014, ApJ, 788, L14
  * Tartaglia et al. (2016) Tartaglia, L., Pastorello, A., Sullivan, M., et al. 2016, MNRAS, 459, 1039
  * van Marle et al. (2009) van Marle, A. J., Owocki, S. P., & Shaviv, N. J. 2009, MNRAS, 394, 595
  * Vlemmings et al. (2002) Vlemmings, W. H. T., Diamond, P. J., & van Langevelde, H. J. 2002, A&A, 394, 589
  * Wood & Karovska (2006) Wood, B. E. & Karovska, M. 2006, ApJ, 649, 410
  * Yaron et al. (2017) Yaron, O., Perley, D. A., Gal-Yam, A., et al. 2017, Nature Physics, 13, 510


# TK-KNN: A Balanced Distance-Based Pseudo Labeling Approach for Semi-
Supervised Intent Classification

Nicholas Botzer♣, David Vasquez♠, Tim Weninger♣, Issam Laradji♠
♣University of Notre Dame, ♠ServiceNow Research
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The ability to detect intent in dialogue systems has become increasingly
important in modern technology. These systems often generate a large amount of
unlabeled data, and manually labeling this data requires substantial human
effort. Semi-supervised methods attempt to remedy this cost by using a model
trained on a few labeled examples and then by assigning pseudo-labels to
further a subset of unlabeled examples that has a model prediction confidence
higher than a certain threshold. However, one particularly perilous
consequence of these methods is the risk of picking an imbalanced set of
examples across classes, which could lead to poor labels. In the present work,
we describe Top-K K-Nearest Neighbor (TK-KNN), which uses a more robust
pseudo-labeling approach based on distance in the embedding space while
maintaining a balanced set of pseudo-labeled examples across classes through a
ranking-based approach. Experiments on several datasets show that TK-KNN
outperforms existing models, particularly when labeled data is scarce on
popular datasets such as CLINC150 and Banking77. Code is available at
https://github.com/ServiceNow/tk-knn

##  1 Introduction.

Large language models like BERT (Devlin et al., 2018) have significantly
pushed the boundaries of Natural Language Understanding (NLU) and created
interesting applications such as automatic ticket resolution (Marcuzzo et al.,
2022). A key component of such systems is a virtual agent’s ability to
understand a user’s intent to respond appropriately. Successful implementation
and deployment of models for these systems require a large amount of labeled
data to be effective. Although deployment of these systems often generate a
large amount of data that could be used for fine-tuning, the cost of labeling
this data is high. Semi-supervised learning methodologies are an obvious
solution because they can significantly reduce the amount of human effort
required to train these kinds of models (Laradji et al., 2021; Zhu and
Goldberg, 2009) especially in image classification tasks (Zhai et al., 2019;
Ouali et al., 2020b; Yalniz et al., 2019). However, as well shall see,
applications of these models is difficult for NLU and intent classification
because of the label distribution.

Figure 1: Example of pseudo label selection when using a threshold (top)
versus the top-k sampling strategy (bottom). In this toy scenario, we chose
$k=2$, where each class is represented by a unique shape. As the threshold
selection strategy pseudo-labels data elements (shown as yellow) that exceed
the confidence level, the model tends to become biased towards classes that
are easier to predict. This bias causes a cascade of mis-labels that leads to
even more bias towards the majority class.

Indeed, research most closely realted to the present work is the Slot-List
model by Basu et al. (2021), which focuses on the meta-learning aspect of
semi-supervised learning rather than using unlabeled data. In a similar vein
the GAN-BERT (Croce et al., 2020) model shows that using an adversarial
learning regime can be devised to ensure that the extracted BERT features are
similar amongst the unlabeled and the labeled data sets and substantially
boost classification performance. Other methods have investigated how data
augmentation can be applied to the NLP domain to enforce consistency in the
models (Chen et al., 2020), and several other methods have been proposed from
the computer vision community. However, a recent empirical study found that
many of these methods do not provide the same benefit to NLU tasks as they
provide to computer vision tasks (Chen et al., 2021) and can even hinder
performance.

Intent classification remains a challenging problem for multiple reasons.
Generally, the number of intents a system must consider is relatively large,
with sixty classes or more. On top of that, most queries consists of only a
short sentence or two. This forces models to need many examples in order to
learn nuance between different intents within the same domain. In the semi-
supervised setting, many methods set a confidence threshold for the model and
assign pseudo-labels to the unlabeled data if their confidence is above the
threshold (Sohn et al., 2020). This strategy permits high-confidence pseudo-
labeled data elements to be included in the training set, which typically
results in performance gains. Unfortunately, this approach also causes the
model to become overconfident for classes that are easier to predict.

In the present work, we describe the Top-K K-Nearest Neighbor (TK-KNN) method
for training semi-supervised models. The main idea of this method is
illustrated in Figure 1. TK-KNN makes two improvements over other pseudo-
labeling approaches. First, to address the model overconfidence problem, we
use a top-k sampling strategy when assigning pseudo-labels to enforce a
balanced set of classes by taking the top-k predictions per class, not simply
the predictions that exceed a confidence threshold overall predictions.
Furthermore, when selecting the top-k examples the sampling strategy does not
simply rely on the model’s predictions, which tend to be noisy. Instead we
leverage the embedding space of the labeled and unlabeled examples to find
those with similar embeddings and combine them with the models’ predictions.
Experiments using standard performance metrics of intent classification are
performed on three datasets: CLINC150 (Larson et al., 2019), Banking77
(Casanueva et al., 2020), and Hwu64 (Liu et al., 2019). We find that the TK-
KNN method outperforms existing methods in all scenarios and performs
exceptionally well in the low-data scenarios.

##  2 Related Work

#### Intent Classification

The task of intent classification has attracted much attention in recent years
due to the increasing use of virtual customer service agents. Recent research
into intent classification systems has mainly focused on learning out of
distribution data (Zhan et al., 2021; Zhang et al., 2021b; Cheng et al., 2022;
Zhou et al., 2022). These techniques configure their experiments to learn from
a reduced number of the classes and treat the remaining classes as out-of-
distribution during testing. Although this research is indeed important in its
own regard, it deviates from the present work’s focus on semi-supervised
learning.

#### Pseudo Labeling

Pseudo labeling is a mainstay in semi-supervised learning (Lee et al., 2013;
Rizve et al., 2021; Cascante-Bonilla et al., 2021). In simple terms, pseudo
labelling uses the model itself to acquire hard labels for each of the
unlabeled data elements. This is achieved by taking the argmax of the models’
output and treating the resulting label as the example’s label. In this
learning regime, the hard labels are assigned to the unlabeled examples
without considering the confidence of the model’s predictions. These pseudo-
labeled examples are then combined with the labeled data to train the model
iteratively. The model is then expected to iteratively improve until
convergence. The main drawback of this method is that mislabeled data elements
early in training can severely degrade the performance of the system.

A common practice to help alleviate mislabeled samples is to use a threshold
$\tau$ to ensure that only high-quality (i.e., confident) labels are retained
(Sohn et al., 2020). The addition of confidence restrictions into the training
process (Sohn et al., 2020) has shown improvements but also restricts the data
used at inference time and introduces the confidence threshold value as yet
another hyperparameter that needs to be tuned.

Another major drawback of this selection method is that the model can become
very biased towards the easy classes in the early iterations of learning
(Arazo et al., 2020). Recent methods, such as FlexMatch (Zhang et al., 2021a),
have discussed this problem at length and attempted to address this issue with
a curriculum learning paradigm that allows each class to have its own
threshold. These thresholds tend to be higher for majority classes lower for
less-common classes. However, this only serves to exacerbate the problem
because the less-common classes will have less-confident labels. A previous
work by Zou et al. (2018) proposes a similar class balancing parameter to be
learned per class, but is applied to the task of unsupervised domain
adaptation. A close previous work to ours is co-training Nigam and Ghani
(2000) that iteratively adds a single example from each class throughout the
self-training. Another more recent work (Gera et al., 2022) also proposes a
balanced sampling mechanism for self-training, but starts from a zero-shot
perspective and limits to two cycles of self-training.

Another pertinent work is by Chen et al. (2022), who introduce ContrastNet, a
framework that leverages contrastive learning for few-shot text
classification. This is particularly relevant to our study considering the
challenges posed by datasets with a scarce number of labeled examples per
class. A notable work by Wang et al. (2022) employs a contrastive learning-
enhanced nearest neighbor mechanism for multi-label text classification, which
bears some resemblance to the KNN strategies discussed in our work.

The TK-KNN strategy described in the present work addresses these issues by
learning the decision boundaries for all classes in a balanced way while still
giving preference to accurate labels by considering the proximity between the
labeled and the unlabeled examples in the embedding space.

#### Distance-based Pseudo labeling

Another direction explored in recent work is to consider the smoothness and
clustering assumptions found in semi-supervised learning (Ouali et al., 2020a)
for pseudo labeling. The smoothness assumption states that if two points lie
in a high-density region their outputs should be the same. The clustering
assumption similarly states that if points are in the same cluster, they are
likely from the same class. Recent work by Zhu et al. (2022) propose a
training-free approach to detect corrupted labels. They use a k-style approach
to detect corrupted labels that share similar features. The results of this
work show that the smoothness and clustering assumptions are also applicable
in a latent embedding space and therefore data elements that are close in the
latent space are likely to share the same clean label.

Two other recent works have made use of these assumptions in semi-supervised
learning to improve their pseudo-labeling process. First, Taherkhani et al.
(2021) use the Wasserstein distance to match clusters of unlabeled examples to
labeled clusters for pseudo-labeling.

Second, the aptly-named feature affinity based pseudo-labeling (Ding et al.,
2019) method uses the cosine similarity between unlabeled examples and cluster
centers that have been discovered for each class. The selected pseudo label is
determined based on the highest similarity score calculated for the unlabeled
example.

Results from both of these works demonstrate that distance-based pseudo-
labeling strategies yield significant improvements over previous methods.
However, both of these methods depend on clusters formed from the labeled
data. In the intent classification task considered in the current study, the
datasets sometimes have an extremely limited number of labeled examples per
class, with instances where there is only one labeled example per class. This
scarcity of labeled data makes forming reliable clusters quite challenging.
Therefore, the TK-KNN model described in the present work adapted the
K-Nearest Neighbors search strategy to help guide our pseudo-labeling process.

##  3 Top-K KNN Semi-Supervised Learning

Figure 2: TK-KNN overview. The model is (1) trained on the small portion of
labeled data. Then, this model is used to predict (2) pseudo labels on the
unlabeled data. Then the cosine similarity (3) is calculated for each
unlabeled data point with respect to the labeled data points in each class.
Yellow shapes represent unlabeled data and green represent labeled data.
Similarities are computed and unlabeled examples are ranked (4) based on a
combination of their predicted probabilities and cosine similarities. Then,
the top-k ($k=2$) examples are selected (5) for each class. These examples are
finally added (6) to the labeled dataset to continue the iterative learning
process.

###  3.1 Problem Definition

We formulate the problem of semi-supervised intent classification as follows:

Given a set of labeled intents $X={(x_{n},y_{n})}:n\in(1,...,N)$ where $x_{n}$
represents the intent example and $y_{n}$ the corresponding intent class $c\in
C$ and a set of unlabeled intents $U={u_{m}}:m\in(1,...,M)$, where each
instance $u_{m}$ is an intent example lacking a label. Intents are fed to the
model as input and the model outputs a predicted intent class, denoted as
$p_{model}(c|x,\theta)$, where $\theta$ represents some pre-trained model
parameters. Our goal is to learn the optimal parameters for $\theta$.

###  3.2 Method Overview

As described above, we first employ pseudo labeling to iteratively train (and
re-train) a model based on its most confident-past predictions. In the first
training cycle, the model is trained on only the small portion of labeled data
$X$. In the subsequent cycles, the model is trained on the union of $X$ and a
subset of the unlabeled data $U$ that has been pseudo-labeled by the model in
the previous cycle. Figure 2 illustrates an example of this training regime
with the TK-KNN method.

We use the BERT-base Devlin et al. (2018) model with an added classification
head to the top. The classification head consists of a dropout layer followed
by a linear layer with dropout and ends with an output layer that represents
the dataset’s class set $C$. We select the BERT-base model for fair comparison
with other methods.

###  3.3 Top-K Sampling

When applying pseudo-labeling, it is often observed that some classes are
easier to predict than others. In practice, this causes the model to become
biased towards the easier classes (Arazo et al., 2020) and perform poorly on
the more difficult ones. The Top-K sampling process within the TK-KNN system
seeks to alleviate this issue by growing the pseudo-label set across all
labels together.

When we perform pseudo labeling, we select the top-k predictions per class
from the unlabeled data. This selection neither uses nor requires any
threshold; instead, it limits each class to choose the predictions with the
highest confidence. We rank each predicted data element with a score based on
the models predicted probability.

$\textrm{score}(u_{m})=p_{model}(y=c|u_{m};\theta)$ (1)

After each training cycle, the number of pseudo labels in the dataset will
have increased by $k$ times the number of classes. This process continues
until all examples are labeled or some number of pre-defined cycles has been
reached. We employ standard early stopping criteria (Prechelt, 1998) during
each training cycle to determine whether or not to stop training.

###  3.4 KNN-Alignment

Although our top-k selection strategy helps alleviate the model’s bias, it
still relies entirely on the model predictions. To enhance our top-k selection
strategy, we utilize a KNN search to modify the scoring function that is used
to rank which pseudo-labeled examples should be included in the next training
iteration. The intuition for the use of the KNN search comes from the findings
in Zhu et al. (2022) where "closer" instances are more likely to share the
same label based on the neighborhood information when some labels are
corrupted, which often occurs in semi-supervised learning from the pseudo-
labeling strategy.

Specifically, we extract a latent representation from each example in our
training dataset, both the labeled and unlabeled examples. We formulate this
latent representation in the same way as Sentence-BERT Reimers and Gurevych
(2019) to construct a robust sentence representation. This representation is
defined as the mean-pooled representation of the final BERT layer that we
formally define as:

$z=\texttt{mean}([CLS],T_{1},T_{2},...,T_{M})$ (2)

Where CLS is the class token, $T$ is each token in the sequence, $M$ is the
sequence length, and $z$ is the extracted latent representation. When we
perform our pseudo labeling process we extract the latent representation for
all of our labeled data $X$ as well as our unlabeled data $U$.

For each unlabeled example, we calculate the cosine similarity between its
latent representation and the latent representations of the labeled
counterparts belonging to the predicted class.

The highest cosine similarity score between the unlabeled example and its
labeled neighbors is used to calculate the score of an unlabeled example. Let
$z_{m}$ and $z_{n}$ be the latent representations of the unlabeled data point
$u_{m}$ and a labeled data point $x_{n}$, respectively. An additional
hyperparameter, $\beta$, permits the weighing of the model’s prediction and
the cosine similarity for the final scoring function.

$\displaystyle\textrm{score}(u_{m})={}$ $\displaystyle(1-\beta)\times
p_{model}(y|u_{m};\theta)+$ (3)
$\displaystyle\beta\times\textrm{sim}(z_{n},z_{m})$

In this equation $\textrm{sim}(z_{n},z_{m})$ computes the cosine similarity
between the latent representations of $u_{m}$ and its closest labeled
counterpart in the predicted class. With these scores we then follow the
previously discussed top-k selection strategy to ensure balanced classes. The
addition of the K-nearest neighbor search helps us to select more accurate
labels early in the learning process. We provide pseudo code for our pseudo-
labeling strategy in Appendix Algorithm 1.

###  3.5 Loss Function

As we use the cosine similarity to help our ranking method we want to ensure
that similar examples are grouped together in the latent space.While the cross
entropy loss is an ideal choice for classification, as it incentivizes the
model to produce accurate predictions, it does not guarantee that
discriminative features will be learned (Elsayed et al., 2018), which our
pseudo labeling relies on. To address this issue, we supplemented the cross-
entropy loss with a supervised contrastive loss Khosla et al. (2020) and a
differential entropy regularization loss Sablayrolles et al. (2019), and
trained the model using all three losses jointly.

$L_{CE}=-\sum_{i=1}^{C}y_{i}\log(\hat{y}_{i})$ (4)

We select the supervised contrastive loss as shown in Khosla et al. (2020).
This ensures that our model with learn good discriminative features in the
latent space that separate examples belonging to different classes. The
supervised contrastive loss relies on augmentations of the original examples.
To get this augmentation we simply apply a dropout layer of $0.2$ to the
representations that we extract from the model. As is standard for the
supervised contrastive loss we add a separate projection layer to our model to
align the representations. The representations fed to the projection layer is
the mean-pooled BERT representation as shown in Eq. 2.

$L_{SCL}=\sum_{i\in I}\frac{-1}{|P(i)|}\sum_{p\in
P(i)}log\frac{sim(z_{i},z_{p})/\tau}{\sum_{a\in A(i)}sim(z_{i},z_{a})/\tau}$
(5)

When adopting the contrastive loss previous works El-Nouby et al. (2021) have
discussed how the model can collapse in dimensions as a result of the loss. We
follow this work in adopting a differential entropy regularizer in order to
spread the representations our more uniformly. The method we use is based on
the Kozachenko and Leonenko (1987) differential entropy estimator:

$L_{KoLeo}=-\frac{1}{N}\sum_{i=1}^{N}\log(p_{i})$ (6)

Where $p_{i}=min_{(i\neq j)}||f(x_{i})-f(x_{j})||$. This regularization helps
to maximize the distance between each point and its neighbors. By doing so it
helps to alleviate the rank collapse issue. We combine this term with the
cross-entropy and contrastive objectives, weighting it using a coefficient
$\gamma$.

$L_{ALL}=L_{CE}+L_{SCL}+\gamma L_{KoLeo}$ (7)

The joint training of these individual components leads our model to have
better discriminative features that are more robust, that results in improved
generalization and prediction accuracy.

##  4 Experiments

###  4.1 Experimental Settings

#### Datasets

We use three well-known benchmark datasets to test and compare the TK-KNN
model against other models on the intent classification task. Our intent
classification datasets are CLINC150 (Larson et al., 2019) that contains 150
in-domain intents classes from ten different domains and one out-of-domain
class. BANKING77 (Casanueva et al., 2020) that contains 77 intents, all
related to the banking domain. HWU64 (Liu et al., 2019) which includes 64
intents coming from 21 different domains. Banking77 and Hwu64 do not provide
validation sets, so we created our own from the original training sets. All
datasets are in English. A breakdown of each dataset is shown in Table 1.

Dataset | Intents | Domain | Train | Val | Test
---|---|---|---|---|---
CLINC150 | 151 | 10 | 15,250 | 3,100 | 5,550
Banking77 | 77 | 1 | 9,002 | 1,001 | 3,080
Hwu64 | 64 | 21 | 8,884 | 1,076 | 1,076
Table 1: Breakdown of the intent classification datasets. Note that BANKING77
and HWU64 do not provide validation sets, so we generated a validation set
from the original training set.

We conducted our experiments with varying amounts of labeled data for each
dataset. All methods are run with five random seeds and the mean average
accuracy of their results are reported along with their 95% confidence
intervals Dror et al. (2018).

###  4.2 Baselines

To perform a proper and thorough comparison of TK-KNN with existing methods,
we implemented and repeated the experiments on the following models and
strategies.

  * •

Supervised: Use only labeled portion of dataset to train the model without any
semi-supervised training. This model constitutes a competitive lower bound of
performance because of the limits in the amount of labeled data.

  * •

Pseudo Labeling (PL) Lee et al. (2013): This strategy trains the model to
convergence then makes predictions on all of the unlabeled data examples.
These examples are then combined with the labeled data and used to re-train
the model in an iterative manner.

  * •

Pseudo Labeling with Threshold (PL-T) Sohn et al. (2020): This process follows
the pseudo labeling strategy but only selects unlabeled data elements which
are predicted above a threshold $\tau$. We use a $\tau$ of 0.95 based on the
findings from previous work.

  * •

Pseudo Labeling with Flexmatch (PL-Flex) Zhang et al. (2021a): Rather than
using a static threshold across all classes, a dynamic threshold is used for
each class based on a curriculum learning framework.

  * •

GAN-BERT Croce et al. (2020): This method applies generative adversarial
networks Goodfellow et al. (2020) to a pre-trained BERT model. The generator
is an MLP that takes in a noise vector. The output head added to the BERT
model acts as the discriminator and includes an extra class for predicting
whether a given data element is real or not.

  * •

MixText Chen et al. (2020): This method extends the MixUp Zhang et al. (2017)
framework to NLP and uses the hidden representation of BERT to mix together.
The method also takes advantage of consistency regularization in the form of
back translated examples.

  * •

TK-KNN : The method described in the present work using top-k sampling with a
weighted selection based on model predictions and cosine similarity to the
labeled samples.

###  4.3 Implementation Details

Each method uses the BERT base model with a classification head attached. We
use the base BERT implementation provided by Huggingface Wolf et al., 2019,
that contains a total of 110M parameters. All models are trained for 30 cycles
of self-training. The models are optimized with the AdamW optimizer with a
learning rate of 5e-5. Each model is trained until convergence by early
stopping applied according to the validation set. We use a batch size of 256
across experiments and limit the sequence length to 64 tokens. For TK-KNN, we
set $k=6$, $\beta=0.75$, and $\gamma=0.1$ and report the results for these
settings. An ablation study of these two hyperparameters is presented later.
For details on the settings used for MixText please see Appendix B.

#### Computational Use.

In total we estimate that we used around 18,000 GPU hours for this project.
For the final experiments and ablation studies we estimate that the TK-KNN
model used 4400 GPU hours. Experiments were carried out on Nvidia Tesla P100
GPUs that each had 12GB of memory and 16GB of memory.

##  5 Results

| Percent Labeled
---|---
Method | 1% | 2% | 5% | 10%
| CLINC150
Supervised | 27.35 $\pm 1.71$ | 49.15 $\pm 1.99$ | 67.96 $\pm 0.85$ | 75.05 $\pm 1.57$
PL | 24.51 $\pm 3.92$ | 48.58 $\pm 1.79$ | 69.19 $\pm 0.54$ | 76.92 $\pm 1.05$
PL-T | 39.05 $\pm 3.26$ | 56.65 $\pm 1.53$ | 71.25 $\pm 0.5$ | 79.29 $\pm 1.62$
PL-Flex | 42.81 $\pm 4.39$ | 60.07 $\pm 1.42$ | 73.42 $\pm 1.62$ | 78.86 $\pm 1.01$
GAN-BERT | 18.18 $\pm 0.0$ | 23.29 $\pm 11.42$ | 44.89 $\pm 24.39$ | 63.02 $\pm 25.1$
MixText | 12.86 $\pm 6.39$ | 37.93 $\pm 16.8$ | 61.39 $\pm 0.77$ | 74.29 $\pm 0.37$
TK-KNN |  53.73 $\pm 1.72$ |  65.87 $\pm 1.18$ |  74.31 $\pm 0.96$ |  79.45 $\pm 1.01$
| BANKING77
Supervised | 34.73 $\pm 1.5$ | 47.51 $\pm 2.89$ | 70.27 $\pm 1.08$ | 80.82 $\pm 0.41$
PL | 29.09 $\pm 3.83$ | 45.16 $\pm 2.71$ | 69.69 $\pm 2.16$ | 80.26 $\pm 0.49$
PL-T | 35.12 $\pm 3.86$ | 51.67 $\pm 3.14$ | 71.16 $\pm 1.98$ | 81.88 $\pm 0.43$
PL-Flex | 40.04 $\pm 3.4$ | 54.18 $\pm 3.31$ | 73.43 $\pm 1.55$ | 82.54 $\pm 0.84$
GAN-BERT | 5.4 $\pm 9.16$ | 16.98 $\pm 21.73$ | 54.09 $\pm 29.56$ | 79.64 $\pm 1.39$
MixText | 32.73 $\pm 6.02$ | 54.75 $\pm 3.15$ | 76.59 $\pm 1.05$ | 82.34 $\pm 0.94$
TK-KNN |  54.16 $\pm 4.56$ |  62.71 $\pm 2.30$ |  76.73 $\pm 01.46$ |  84.45 $\pm 0.52$
| HWU64
Supervised | 48.87 $\pm 1.55$ | 63.88 $\pm 1.6$ | 74.67 $\pm 1.91$ | 82.21 $\pm 1.72$
PL | 48.46 $\pm 1.86$ | 64.39 $\pm 1.66$ | 75.76 $\pm 1.69$ | 82.49 $\pm 0.94$
PL-T | 56.9 $\pm 1.64$ | 68.29 $\pm 1.79$ | 76.9 $\pm 1.1$ | 82.96 $\pm 1.69$
PL-Flex | 60.15 $\pm 3.27$ | 69.87 $\pm 0.93$ | 77.99 $\pm 1.4$ | 83.83 $\pm 1.2$
GAN-BERT | 33.36 $\pm 16.55$ | 32.9 $\pm 29.07$ | 72.32 $\pm 1.41$ | 81.78 $\pm 1.64$
MixText | 33.3 $\pm 8.98$ | 56.46 $\pm 11.08$ | 66.65 $\pm 7.28$ | 79.72 $\pm 1.27$
TK-KNN |  65.33 $\pm 2.29$ |  73.03 $\pm 1.31$ |  79.63 $\pm 0.56$ |  84.59 $\pm 0.58$
Table 2: Mean test accuracy results and their 95% confidence intervals across
5 repetitions with different different random seeds. All experiments used
$k=6$ and $\beta=0.75$. TK-KNN outperformed existing state of the art models,
especially when the label set is small. The confidence interval also shows
that the TK-KNN results were stable across repetitions.

Results from these experiments are shown in Table 2. These quantitative
results demonstrate that TK-KNN yielded the best performance on the benchmark
datasets. We observed the most significant performance gains for CLINC150 and
BANKING77, where these datasets have more classes. For instance, on the
CLINC150 dataset with 1% labeled data, our method performs 10.92% better than
the second best strategy, FlexMatch. As the portion of labeled data used
increases, we notice that the effectiveness of TK-KNN diminishes.

Another observation from these results is that the GAN-BERT model tends to be
unstable when the labeled data is limited. However, GAN-BERT does improve as
the proportion of labeled data increases. This is consistent with previous
findings on training GANs from computer vision (Salimans et al., 2016;
Arjovsky and Bottou, 2017). We also find that while the MixText method shows
improvements the benefits of consistency regularization are not as strong
compared to works from the computer vision domain.

These results demonstrate the benefits of TK-KNN’s balanced sampling strategy
and its use of the distances in the latent space.

$0$$10$$20$$30$$0.2$$0.3$$0.4$$0.5$CycleTest AccuracyPL-TPL-FlexTK-KNN Figure
3: Convergence analysis of pseudo-labelling strategies on CLINC150 at 1%
labeled data. TK-KNN clearly outperforms the other pseudo-labelling strategies
by balancing class pseudo labels after each training cycle.

###  5.1 Overconfidence in Pseudo-Labelling Regimes

A key observation we found throughout self-training was that the performance
of existing pseudo-labelling methods tended to degrade as the number of cycles
increased. An example of this is illustrated in Figure 3. Here we see that
when a pre-defined threshold is used, the model tends to improve performance
for the first few training cycles. After that point, the pseudo-labeling
becomes heavily biased towards the easier classes. This causes the model to
become overconfident in predictions for those classes and neglect more
difficult classes. PL-Flex corrects this issue but converges much earlier in
the learning process. TK-KNN achieves the best performance thanks to the
slower balanced pseudo-labeling approach. This process helps the model learn
clearer decision boundaries for all classes simultaneously and prevent
overconfidence in the model in some classes.

###  5.2 Ablation Study

Because TK-KNN is different from existing methods in two distinct ways: (1)
top-k balanced sampling and (2) KNN ranking, we perform a set of ablation
experiments to better understand how each of these affects performance.
Specifically, we test TK-KNN under three scenarios, top-k sampling without
balancing the classes, top-k sampling with balanced classes, and top-k KNN
without balancing for classes. When we perform top-k sampling in an unbalanced
manner, we ensure that the total data sampled is still equal to $k*C$, where
$C$ is the number of classes. We report the results for these experiments in
Table 3. We also conduct experiments on the addition of the different loss
objectives detailed in Appendix C

The results from the ablation study demonstrate both the effectiveness of
top-k sampling and KNN ranking. A comparison between our unbalanced sampling
top-k sampling and balanced versions show a drastic difference in performance
across all datasets. We highlight again that the performance difference is
greatest in the lowest resource setting, with a 12.47% increase in accuracy
for CLINC150 in the 1% setting.

Results from the TK-KNN method with unbalanced sampling also show an
improvement over unbalanced sampling alone. This increase in performance is
smaller than the difference between unbalanced and balanced sampling but still
highlights the benefits of leveraging the geometry for selective pseudo-
labeling. We also present an ablation of the Top-k sampling methodology when
passing correctly labeled examples in the Appendix A.

| Percent Labeled
---|---
Method | 1% | 2%
| CLINC150
Top-k U | 38.37 $\pm 1.08$ | 55.0 $\pm 1.44$
Top-k B | 51.36 $\pm 2.1$ | 64.99 $\pm 0.64$
Top-k KNN U | 41.24 $\pm 0.97$ | 55.01 $\pm 1.49$
Top-k KNN B |  53.73 $\pm 1.72$ |  65.87 $\pm 1.18$
| BANKING77
Top-k U | 41.56 $\pm 4.73$ | 54.78 $\pm 3.51$
Top-k B | 50.45 $\pm 4.53$ | 63.19 $\pm 1.78$
Top-k KNN U | 44.12 $\pm 3.14$ | 55.9 $\pm 2.65$
Top-k KNN B |  54.16 $\pm 4.56$ |  62.71 $\pm 2.30$
| HWU64
Top-k U | 54.87 $\pm 1.64$ | 64.85 $\pm 1.54$
Top-k B | 54.13 $\pm 6.0$ | 65.12 $\pm 0.35$
Top-k KNN U | 57.86 $\pm 2.25$ | 69.33 $\pm 0.96$
Top-k KNN B |  65.33 $\pm 2.29$ |  73.03 $\pm 1.31$
Table 3: Ablation study of top-k sampling. U stands for unbalanced sampling,
where classes are not balanced. B is for balanced sampling, and classes are
balanced with the top-k per class.
$0$$10$$20$$30$$0.5$$0.55$$0.6$$0.65$CycleTest Accuracy0.00.250.50.751.0
Figure 4: A comparison of TK-KNN on HWU64 with 1% labeled data as $\beta$
varies. $0$$10$$20$$30$$0.5$$0.55$$0.6$$0.65$CycleTest Accuracy468 Figure 5: A
comparison of TK-KNN on HWU64 with 1% labeled data as k varies.

###  5.3 Parameter Search

TK-KNN relies on two hyperparameters $k$ and $\beta$ that can affect
performance based on how they are configured. We explore experiments to gauge
their effect on learning by testing $k\in(4,6,8)$ and
$\beta\in(0.0,0.25,0.50,0.75,1.00)$. When varying $k$ we hold $\beta$ at 0.75.
For $\beta$ experiments we keep $k=6$. When $\beta=0.0$, this is equivalent to
just top-k sampling based on Equation 3. Alternatively, when $\beta=1.0$, this
is equivalent to only using the KNN similarity for ranking. Results from our
experiments are shown in Figures 4 for $\beta$ and 5 for $k$.

As we varied the $\beta$ parameter, we noticed that all configurations tended
to have similar training patterns. After we trained the model for the first
five cycles, the model tended to move in small jumps between subsequent
cycles. From the illustration, we can see that no single method was always the
best, but the model tended to perform worse when $\beta=0.0$, highlighting the
benefits of including our KNN similarity for ranking. The model reached the
best performance when $\beta=0.75$, which occurs about a third of the way
through the training process.

Comparison of values for $k$ show that TK-KNN is robust to adjustments in this
hyperparameter. We notice slight performance benefits from selecting a higher
$k$ of 6 and 8 in comparison to 4. When a higher value of $k$ is used the
model will see an increase in performance earlier in the self-training
process, as it has more examples to train from. This is only acheivable though
when high quality correct samples are selected across the entire class
distribution. If a $k$ value was selected that is too large, more bad examples
will be included early in the training process and may result in poor model
performance.

##  6 Conclusions

This paper introduces TK-KNN, a balanced distance-based pseudo-labeling
approach for semi-supervised intent classification. TK-KNN deviates from
previous pseudo-labeling methods as it does not rely on a threshold to select
the samples. Instead, we show that a balanced approach that takes the model
prediction and K-Nearest Neighbor similarity measure allows for more robust
decision boundaries to be learned. Experiments on three popular intent
classification datasets, CLINC150, Banking77, and Hwu64, demonstrate that our
method improved performance in all scenarios.

##  7 Limitations

While our method shows noticeable improvements, it is not without limitations.
Our method does not require searching for a good threshold but instead
requires two different hyperparameters, $k$ and $\beta$, that must be found.
We offer a reasonable method and findings for selecting both of these but
others may want to search for other combinations depending on the dataset. A
noticeable drawback from our self-training method is that more cycles of
training will need to be done, especially if the value of $k$ is small. This
requires much more GPU usage to converge to a good point. Further, we did not
explore any heavily imbalanced datasets, so we are unaware of how TK-KNN would
perform under those scenarios.

##  8 Ethics Statement

This work can be used to help improve current virtual assistant systems for
many of the businesses that may be seeking to use one. As the goal of these
systems is to understand a persons request, failures can lead to wrong actions
being taken that potentially impact an individual.

## References

  * Arazo et al. [2020] E. Arazo, D. Ortego, P. Albert, N. E. O’Connor, and K. McGuinness.  Pseudo-labeling and confirmation bias in deep semi-supervised learning.  In _IJCNN_ , 2020.
  * Arjovsky and Bottou [2017] M. Arjovsky and L. Bottou.  Towards principled methods for training generative adversarial networks.  _arXiv preprint arXiv:1701.04862_ , 2017.
  * Basu et al. [2021] S. Basu, A. Sharaf, A. Fischer, V. Rohra, M. Amoake, H. El-Hammamy, E. Nosakhare, V. Ramani, B. Han, et al.  Semi-supervised few-shot intent classification and slot filling.  _arXiv preprint arXiv:2109.08754_ , 2021.
  * Casanueva et al. [2020] I. Casanueva, T. Temčinas, D. Gerz, M. Henderson, and I. Vulić.  Efficient intent detection with dual sentence encoders.  In _Workshop on Natural Language Processing for Conversational AI_ , 2020.
  * Cascante-Bonilla et al. [2021] P. Cascante-Bonilla, F. Tan, Y. Qi, and V. Ordonez.  Curriculum labeling: Revisiting pseudo-labeling for semi-supervised learning.  In _AAAI_ , 2021.
  * Chen et al. [2020] J. Chen, Z. Yang, and D. Yang.  Mixtext: Linguistically-informed interpolation of hidden space for semi-supervised text classification.  In _ACL_ , 2020.
  * Chen et al. [2021] J. Chen, D. Tam, C. Raffel, M. Bansal, and D. Yang.  An empirical survey of data augmentation for limited data learning in nlp.  _arXiv preprint arXiv:2106.07499_ , 2021.
  * Chen et al. [2022] J. Chen, R. Zhang, Y. Mao, and J. Xu.  Contrastnet: A contrastive learning framework for few-shot text classification.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 10492–10500, 2022.
  * Cheng et al. [2022] Z. Cheng, Z. Jiang, Y. Yin, C. Wang, and Q. Gu.  Learning to classify open intent via soft labeling and manifold mixup.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 30:635–645, 2022.
  * Croce et al. [2020] D. Croce, G. Castellucci, and R. Basili.  Gan-bert: Generative adversarial learning for robust text classification with a bunch of labeled examples.  In _ACL_ , 2020.
  * Devlin et al. [2018] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Ding et al. [2019] G. Ding, S. Zhang, S. Khan, Z. Tang, J. Zhang, and F. Porikli.  Feature affinity-based pseudo labeling for semi-supervised person re-identification.  _IEEE Transactions on Multimedia_ , 21(11):2891–2902, 2019.
  * Dror et al. [2018] R. Dror, G. Baumer, S. Shlomov, and R. Reichart.  The hitchhiker’s guide to testing statistical significance in natural language processing.  In _ACL_ , 2018.
  * El-Nouby et al. [2021] A. El-Nouby, N. Neverova, I. Laptev, and H. Jégou.  Training vision transformers for image retrieval.  _arXiv preprint arXiv:2102.05644_ , 2021.
  * Elsayed et al. [2018] G. Elsayed, D. Krishnan, H. Mobahi, K. Regan, and S. Bengio.  Large margin deep networks for classification.  _Advances in neural information processing systems_ , 31, 2018.
  * Gera et al. [2022] A. Gera, A. Halfon, E. Shnarch, Y. Perlitz, L. Ein-Dor, and N. Slonim.  Zero-shot text classification with self-training.  _arXiv preprint arXiv:2210.17541_ , 2022.
  * Goodfellow et al. [2020] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial networks.  _Communications of the ACM_ , 63(11):139–144, 2020.
  * Khosla et al. [2020] P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan.  Supervised contrastive learning.  _Advances in neural information processing systems_ , 33:18661–18673, 2020.
  * Kozachenko and Leonenko [1987] L. F. Kozachenko and N. N. Leonenko.  Sample estimate of the entropy of a random vector.  _Problemy Peredachi Informatsii_ , 23(2):9–16, 1987.
  * Laradji et al. [2021] I. Laradji, P. Rodríguez, D. Vazquez, and D. Nowrouzezahrai.  Ssr: Semi-supervised soft rasterizer for single-view 2d to 3d reconstruction.  _arXiv preprint arXiv:2108.09593_ , 2021.
  * Larson et al. [2019] S. Larson, A. Mahendran, J. J. Peper, C. Clarke, A. Lee, P. Hill, J. K. Kummerfeld, K. Leach, M. A. Laurenzano, L. Tang, et al.  An evaluation dataset for intent classification and out-of-scope prediction.  In _EMNLP-IJCNLP_ , 2019.
  * Lee et al. [2013] D.-H. Lee et al.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In _Workshop on challenges in representation learning, ICML_ , 2013\.
  * Liu et al. [2019] X. Liu, A. Eshghi, P. Swietojanski, and V. Rieser.  Benchmarking natural language understanding services for building conversational agents.  _arXiv preprint arXiv:1903.05566_ , 2019.
  * Marcuzzo et al. [2022] M. Marcuzzo, A. Zangari, M. Schiavinato, L. Giudice, A. Gasparetto, and A. Albarelli.  A multi-level approach for hierarchical ticket classification.  In _W-NUT_ , 2022.
  * Nigam and Ghani [2000] K. Nigam and R. Ghani.  Analyzing the effectiveness and applicability of co-training.  In _Proceedings of the ninth international conference on Information and knowledge management_ , pages 86–93, 2000.
  * Ouali et al. [2020a] Y. Ouali, C. Hudelot, and M. Tami.  An overview of deep semi-supervised learning.  _arXiv preprint arXiv:2006.05278_ , 2020a.
  * Ouali et al. [2020b] Y. Ouali, C. Hudelot, and M. Tami.  Semi-supervised semantic segmentation with cross-consistency training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020b.
  * Prechelt [1998] L. Prechelt.  Early stopping-but when?  In _Neural Networks: Tricks of the trade_ , pages 55–69. Springer, 1998.
  * Reimers and Gurevych [2019] N. Reimers and I. Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, 2019.
  * Rizve et al. [2021] M. N. Rizve, K. Duarte, Y. S. Rawat, and M. Shah.  In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning.  _arXiv preprint arXiv:2101.06329_ , 2021.
  * Sablayrolles et al. [2019] A. Sablayrolles, M. Douze, C. Schmid, and H. Jégou.  Spreading vectors for similarity search.  In _ICLR 2019-7th International Conference on Learning Representations_ , pages 1–13, 2019.
  * Salimans et al. [2016] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen.  Improved techniques for training gans.  _Advances in neural information processing systems_ , 29, 2016.
  * Sohn et al. [2020] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  In _NeurIPS_ , 2020.
  * Taherkhani et al. [2021] F. Taherkhani, A. Dabouei, S. Soleymani, J. Dawson, and N. M. Nasrabadi.  Self-supervised wasserstein pseudo-labeling for semi-supervised image classification.  In _CVPR_ , 2021.
  * Wang et al. [2022] R. Wang, X. Dai, et al.  Contrastive learning-enhanced nearest neighbor mechanism for multi-label text classification.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 672–679, 2022\.
  * Wolf et al. [2019] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_ , 2019.
  * Yalniz et al. [2019] I. Z. Yalniz, H. Jégou, K. Chen, M. Paluri, and D. Mahajan.  Billion-scale semi-supervised learning for image classification.  _arXiv preprint arXiv:1905.00546_ , 2019.
  * Zhai et al. [2019] X. Zhai, A. Oliver, A. Kolesnikov, and L. Beyer.  S4l: Self-supervised semi-supervised learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019.
  * Zhan et al. [2021] L.-M. Zhan, H. Liang, B. Liu, L. Fan, X.-M. Wu, and A. Lam.  Out-of-scope intent detection with self-supervision and discriminative training.  _arXiv preprint arXiv:2106.08616_ , 2021.
  * Zhang et al. [2021a] B. Zhang, Y. Wang, W. Hou, H. Wu, J. Wang, M. Okumura, and T. Shinozaki.  Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling.  In _NeurIPS_ , 2021a.
  * Zhang et al. [2017] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz.  mixup: Beyond empirical risk minimization.  _arXiv preprint arXiv:1710.09412_ , 2017.
  * Zhang et al. [2021b] H. Zhang, H. Xu, and T.-E. Lin.  Deep open intent classification with adaptive decision boundary.  In _AAAI_ , 2021b.
  * Zhou et al. [2022] Y. Zhou, P. Liu, and X. Qiu.  Knn-contrastive learning for out-of-domain intent classification.  In _ACL_ , 2022.
  * Zhu and Goldberg [2009] X. Zhu and A. B. Goldberg.  Introduction to semi-supervised learning.  _Synthesis lectures on artificial intelligence and machine learning_ , 3(1):1–130, 2009.
  * Zhu et al. [2022] Z. Zhu, Z. Dong, and Y. Liu.  Detecting corrupted labels without training a model to predict.  In _ICML_ , 2022.
  * Zou et al. [2018] Y. Zou, Z. Yu, B. Kumar, and J. Wang.  Unsupervised domain adaptation for semantic segmentation via class-balanced self-training.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 289–305, 2018.

Algorithm 1 TK-KNN Sampling For a Cycle

1:Data of $X={x_{n},y_{n}}:n\in(1,...,N)$, $U={u_{m}}:m\in(1,...,M)$, $\beta$

2:Predict pseudo-labels for all $U$

3:Calculate cosine similarity via. Eq. (2) for all $U$ per class

4:Calculate score via Eq.(3)

5:Combine $X$ and top-k per class from $U$

##  Appendix A Upper Bound Analysis

We further ran experiments to gauge the performance of top-k sampling when
ground truth labels are fed to the model instead of predicted pseudo labels.
This experiment gives us an indicator as to how performance should increase
throughout the self-training process in an ideal pseudo-labeling scenario. We
present the results of this in Figure. 6. As expected, the model tends to
converge towards a fully supervised performance as the cycle increases and
more data is (pseudo-)labeled. Another point of interest is that the method’s
upper bound can continue learning with proper labels, while TK-KNN method
tends to converge earlier. The upper bound method also takes a significant
increase in the first few cycles as well. This highlights a need to
investigate methods for accurate pseudo label selection further, so that the
model can continue to improve.

$0$$10$$20$$30$$0.4$$0.6$$0.8$CycleTest AccuracyCLINC150

$0$$10$$20$$30$CycleBANKING77

$0$$10$$20$$30$CycleHWU64

6

Figure 6: Ablation results for each dataset using 1% labeled data.

##  Appendix B MixText Experiments

The MixText method requires a number of hyperparameters to be selected in
order to acheive good performance. The method also relies on data augmentation
of the original text. For our experiments we used the Helsinki-NLP models
available on the huggingface repository 111https://huggingface.co/Helsinki-
NLP/opus-mt-en-de https://huggingface.co/Helsinki-NLP/opus-mt-en-ru to perform
back translation. Following the original paper we performed back translation
into German (de) and Russian (ru). Our experiments used a labeled batch size
of 4 and unlabeled batch size of 8, the same as the original paper. The
Temperature parameter was set to 0.5.

##  Appendix C Ablation Table

We present a detailed breakdown of our full ablation results in Table 4. These
results include the performance when using only the cross-entropy loss, as
well as various combinations of the supervised contrastive loss and
differential entropy regularizer. The results in this table demonstrate that
the inclusion of additional loss objectives improves performance. This is
particularly evident when we add our KNN objective, as we observe an increase
of approximately 1-4%. Although the addition of the differential entropy
regularizer yields smaller performance improvements, it remains beneficial to
our method overall.

| Percent Labeled
---|---
Method | 1% | 2%
| CLINC150
Top-k U CE | 37.06 $\pm 2.09$ | 50.8 $\pm 1.79$
Top-k U CE CON | 38.67 $\pm 1.47$ | 55.09 $\pm 0.85$
Top-k U CE CON DER | 38.37 $\pm 1.08$ | 55.0 $\pm 1.44$
Top-k B CE | 49.53 $\pm 3.1$ | 62.45 $\pm 1.21$
Top-k B CE CON | 52.33 $\pm 2.53$ | 64.52 $\pm 1.63$
Top-k B CE CON DER | 51.36 $\pm 2.1$ | 64.99 $\pm 0.64$
Top-k KNN U CE | 40.58 $\pm 2.54$ | 53.45 $\pm 1.05$
Top-k KNN U CE CON | 42.3 $\pm 2.86$ | 56.07 $\pm 1.26$
Top-k KNN U CE CON DER | 41.24 $\pm 0.97$ | 55.01 $\pm 1.49$
Top-k KNN B CE | 50.28 $\pm 3.00$ | 63.99 $\pm 0.85$
Top-k KNN B CE CON | 53.23 $\pm 3.17$ | 65.37 $\pm 1.17$
Top-k KNN B CE CON DER |  53.73 $\pm 1.72$ |  65.87 $\pm 1.18$
| BANKING77
Top-k U CE | 36.19 $\pm 2.93$ | 51.69 $\pm 3.54$
Top-k U CE CON | 42.16 $\pm 4.61$ | 54.77 $\pm 2.48$
Top-k U CE CON DER | 41.56 $\pm 4.73$ | 54.78 $\pm 3.51$
Top-k B CE | 44.35 $\pm 3.82$ | 57.19 $\pm 3.23$
Top-k B CE CON | 50.48 $\pm 3.75$ | 62.34 $\pm 1.57$
Top-k B CE CON DER | 50.45 $\pm 4.53$ |  63.19 $\pm 1.78$
Top-k KNN U CE | 36.96 $\pm 4.78$ | 52.42 $\pm 2.75$
Top-k KNN U CE CON | 44.86 $\pm 3.53$ | 56.49 $\pm 2.68$
Top-k KNN U CE CON DER | 44.12 $\pm 3.14$ | 55.9 $\pm 2.65$
Top-k KNN B CE | 49.16 $\pm 3.02$ | 59.81 $\pm 1.69$
Top-k KNN B CE CON | 52.4 $\pm 3.91$ | 62.35 $\pm 2.18$
Top-k KNN B CE CON DER |  54.16 $\pm 4.56$ | 62.71 $\pm 2.30$
| HWU64
Top-k U CE | 51.45 $\pm 3.8$ | 62.79 $\pm 1.99$
Top-k U CE CON | 54.8 $\pm 2.1$ | 66.58 $\pm 1.17$
Top-k U CE CON DER | 54.87 $\pm 1.64$ | 64.85 $\pm 1.54$
Top-k B CE | 61.69 $\pm 3.15$ | 70.41 $\pm 0.66$
Top-k B CE CON | 63.05 $\pm 2.6$ | 72.55 $\pm 1.41$
Top-k B CE CON DER | 54.13 $\pm 6.0$ | 65.12 $\pm 0.35$
Top-k KNN U CE | 52.97 $\pm 1.45$ | 64.41 $\pm 1.58$
Top-k KNN U CE CON | 59.26 $\pm 2.99$ | 69.98 $\pm 1.44$
Top-k KNN U CE CON DER | 57.86 $\pm 2.25$ | 69.33 $\pm 0.96$
Top-k KNN B CE | 62.43 $\pm 2.78$ | 71.04 $\pm 0.93$
Top-k KNN B CE CON | 64.41 $\pm 2.26$ | 72.71 $\pm 1.07$
Top-k KNN B CE CON DER |  65.33 $\pm 2.29$ |  73.03 $\pm 1.31$

Table 4: Ablation study of top-k sampling. U stands for unbalanced sampling,
where classes are not balanced. B is for balanced sampling, and classes are
balanced with the top-k per class. CE stands for cross-entropy loss, CON for
contrastive loss, and DER differential entropy regularizer.