Another minor inaccurracy appears in the proof of the more general [Lan21,
Cor. 5.7], for, without some hypotheses on the $\mathbb{A}^{1}$-stack in
question, one cannot test the desired commutativity of the diagram of
morphisms over the affine line by checking it only over the origin and over
the unit, even in the presence of the $\mathbb{G}_{m}$-action. We thank Adrian
Langer for informing us that in fact the inaccuracy above can be by-passed in
his newer version to yield a complete proof. The proof of the properness
[Lan21, Thm. 5.9] is not affected by the inaccurracy in loc. cit. Cor. 5.7.

Theorem 1.1 proves that the stack of semistable $t$-connections with poles is
smooth over the Noetherian base $B.$ In this paper, we apply this smoothness
result to the case when $B$ is a field and when $B$ is a DVR, both of which
are reduced. Therefore, the proof given in [LP01, Prop. 3.2] works, with the
trivial modification stemming from the fact that: while in the case with no
poles loc. cit. uses, in the context of the elegant “Bost’s Trick,” the
elementary identity
$\partial_{x}^{[p]}:=\partial_{x}\circ\cdots\circ\partial_{x}$ ($p$ times)
$=0$, in the case with poles we can use the identity
$(x\partial_{x})^{[p]}=x\partial_{x}.$ In the end, while loc. cit. ends with a
factor $t$ in the case without poles, we end with a factor $tx$ in the case
with poles, and the logic to reach the desired conclusion, namely the
existence of the Hodge-Hitchin morphism for families of curves over a reduced
Noetherian scheme, is the same.

In this appendix, we remove the assumption made above of semistability, as
well as the assumtpion on the Noetehrian $B$ being reduced. We prove the
existence of the Hodge-Hitchin morphism (5) from the stack of $t$-connections,
with or without poles for a family of curves over a Noetherian base $B.$

The key step is to reduce to an auxiliary family of curves over a suitable
complete and reduced ring, where then the Lazlo-Pauly logic is valid, without
poles (factor $t$), and with poles (factor $tx$). We now give the details of
this key reduction step.

By Noetherian approximation (more precisely: choose a relative polarization
and then use the fact that the stack of polarized smooth geometrically
connected curves over $\mathbb{F}_{p}$ is locally of finite presentation over
$\mathbb{F}_{p}$ [Sta22, Tag 0DSS]\+ [Sta22, Tag 0E81]+[Sta22, Tag 0DQ0],
combined with [Sta22, Tag 0CMX] applied to a colimit
$B=\operatorname{colim}_{i}B_{i}$ as in [Sta22, Tag 01ZA]) the curve $C\to B$
fits into a Cartesian diagram:

${C}$${C^{\prime}}$${B}$${B^{\prime},}$

where $B^{\prime}$ is of finite type over the prime field $\mathbb{F}_{p}$ and
$C^{\prime}\to B^{\prime}$ is a smooth projective morphism with geometrically
connected fibers of dimension $1$. This means that the Hodge stacks fit into
the following diagram

${\mathcal{M}\text{Hodge}(C)}$${\mathcal{M}\text{Hodge}(C^{\prime})}$${B}$${B^{\prime},}$

Since the formation of the $p$-curvature morphism is compatible with base-
change in the base $B$, it suffices to show the desired factorization for
$\mathcal{M}\text{Hodge}(C^{\prime})$, and so we can assume without loss of
generality that $B$ is of finite type over $\mathbb{F}_{p}$.

Since the stack $\mathcal{M}\text{Hodge}(C)$ is locally of finite type over
$B$ [HLHJ21, Prop 2.2.2], it suffices to check the desired factorization for
any family over an affine scheme $\operatorname{Spec}(R)$ with $R$ of finite
type over $\mathbb{F}_{p}$. Such a point
$\operatorname{Spec}(R)\to\mathcal{M}\text{Hodge}(C)$ corresponds to a
function $t\in R$, a vector bundle $\mathcal{F}$ on $C_{R}$, and a
$t$-connection $\nabla$ on $\mathcal{F}$.

We can write $R=\mathbb{F}_{p}[t_{1},t_{2},\ldots t_{r}]/I$ for some ideal
$I$. We denote by $\hat{S}$ the completion of the polynomial ring
$\mathbb{F}_{p}[t_{1},t_{2},\ldots t_{r}]$ with respect to the ideal $I$.
Since $\mathbb{F}_{p}[t_{1},t_{2},\ldots t_{r}]$ is a reduced G-ring [Sta22,
Tag 07PX], the completion $\hat{S}$ is reduced ([Sta22, Tag 0AH2] \+ [Sta22,
Tag 0C21]). Choose a $\operatorname{Spec}(R)$-ample line bundle $\mathcal{L}$
on the family $C$. The deformation theory of smooth curves equipped with ample
line bundles is unobstructed ([Sta22, Tag 0AH2]+[Sta22, Tag 0E84]). Similarly
the deformation theory of the vector bundle $\mathcal{F}$ has obstructions in
the groups $H^{2}(C,I^{j}\otimes\mathcal{E}nd(\mathcal{F}))=0$ [FGI+05, Thm.
8.5.3], and so it is unobstructed as well. Therefore we can get a compatible
family of lifts of the triple $(C,\mathcal{L},\mathcal{F})$ for every
nilpotent thickening $\mathbb{F}_{p}[t_{1},t_{2},\ldots,t_{r}]/I^{j}$ as $j$
ranges over the positive integers. By Grothendieck’s existence and
algebraization theorems ([Sta22, Tag 089A]\+ [Sta22, Tag 03O]), we can
algebraize this formal tuple into a families
$(\widetilde{C},\widetilde{\mathcal{F}},\widetilde{\mathcal{L}})$ over
$\operatorname{Spec}(\hat{S})$. Therefore, we get a Cartesian diagram of
families of smooth curves:

${C}$${\widetilde{C}}$${\operatorname{Spec}(R)}$${\operatorname{Spec}(\hat{S}),}$$\hookrightarrow$

and a vector bundle $\widetilde{\mathcal{F}}$ on $\widetilde{C}$ such that its
restriction to $C$ recovers $\mathcal{F}$. Choose a lift
$\widetilde{t}\in\hat{S}$ of $t\in R$. In order to show the factorization of
the $p$-curvature morphism as in [LP01, Prop. 3.2], we need to show that
certain canonically defined sections of powers of the line bundle
$\omega_{C/S}$ vanish. This can be done Zariski locally on $C.$ Choose an
affine open covering $\widetilde{U}_{i}$ of $\widetilde{C}$ that trivializes
$\widetilde{\mathcal{F}}$. We fix trivializations of
$\widetilde{\mathcal{F}}|_{\widetilde{U}_{i}}$. We denote by $U_{i}$ the
restriction to $C$, which yields an affine open covering with trivializations
of the restriction $\mathcal{F}$. It suffices to show that the factorization
of the $p$-curvature map on every $U_{i}$. The $t$-connection $\nabla$ on the
trivial bundle $\mathcal{F}|_{U_{i}}$ can be written as $td_{U_{i}}+M$, where
$d_{U_{i}}:\mathcal{O}_{U_{i}}\to\Omega^{1}_{U_{i}/\hat{S}}$ denotes the
exterior derivative on $U_{i}$ and $M\in H^{0}(\omega_{U_{i}/R}^{\oplus
n^{2}})$ is a matrix of differentials. Choose a lift $\widetilde{M}\in
H^{0}(\omega_{\widetilde{U}_{i}/\hat{S}}^{\oplus n^{2}})$ of $M$, and define
$\widetilde{\nabla}$ to be the $\widetilde{t}$ connection
$\widetilde{t}d_{\widetilde{U}_{i}}+\widetilde{M}$ on the trivial bundle
$\widetilde{\mathcal{F}}_{\widetilde{U}_{i}}$. The $\widetilde{t}$-connection
$(\widetilde{\mathcal{F}}|_{\widetilde{U}_{i}},\widetilde{\nabla})$ on
$\widetilde{U}_{i}$ restricts to the $t$-connection
$(\mathcal{F}|_{U_{i}},\nabla)$ under the base-change by
$\operatorname{Spec}(R)\hookrightarrow\operatorname{Spec}(\hat{S})$. Since the
formation of the $p$-curvature is compatible with such base-change, it
suffices to show the desired factorization for the $\widetilde{t}$-connection
$(\widetilde{\mathcal{F}}|_{\widetilde{U}_{i}},\widetilde{\nabla})$ on
$\widetilde{U}_{i}$. Thus we can work over the reduced ring $\hat{S}$ and on
affine open subsets of $\widetilde{C}$ to prove the desired factorization. By
passing to each irreducible component of $\operatorname{Spec}(\hat{S})$, we
can furthermore assume that $\hat{S}$ is an integral domain. Hence we can use
the local computation outlined in the proof of [LP01, Prop. 3.2], which
assumes that the base ring is an integral domain. The calculation is carried
out in the case without poles by using the vector field $\partial_{x}.$ The
case with poles is analogous, once we replace the vector field $\partial_{x}$
with $x\partial_{x}.$ Note also that the case without poles implies directly
the case with poles: the sections we need to prove are trivial are trivial
away from the poles, hence are trivial across the poles.

## References

  * [Alp13] Jarod Alper.  Good moduli spaces for Artin stacks.  Ann. Inst. Fourier (Grenoble), 63(6):2349–2402, 2013.
  * [Alp14] Jarod Alper.  Adequate moduli spaces and geometrically reductive group schemes.  Algebr. Geom., 1(4):489–531, 2014.
  * [AOV08] Dan Abramovich, Martin Olsson, and Angelo Vistoli.  Tame stacks in positive characteristic.  Annales de l’Institut Fourier, 58(4):1057–1091, 2008.
  * [BNR89] Arnaud Beauville, M. S. Narasimhan, and S. Ramanan.  Spectral curves and the generalised theta divisor.  J. Reine Angew. Math., 398:169–179, 1989.
  * [BR94] I. Biswas and S. Ramanan.  An infinitesimal study of the moduli of Hitchin pairs.  J. London Math. Soc. (2), 49(2):219–231, 1994.
  * [BS06] Indranil Biswas and S. Subramanian.  Vector bundles on curves admitting a connection.  Q. J. Math., 57(2):143–150, 2006.
  * [CL16] Pierre-Henri Chaudouard and Gérard Laumon.  Un théorème du support pour la fibration de Hitchin.  Ann. Inst. Fourier (Grenoble), 66(2):711–727, 2016.
  * [CZ15] Tsao-Hsien Chen and Xinwen Zhu.  Non-abelian Hodge theory for algebraic curves in characteristic $p$.  Geom. Funct. Anal., 25(6):1706–1733, 2015.
  * [dC17] Mark Andrea de Cataldo.  A support theorem for the Hitchin fibration: the case of ${\rm SL}_{n}$.  Compos. Math., 153(6):1316–1347, 2017.
  * [dC22] Mark Andrea A. de Cataldo.  Perverse Leray filtration and specialisation with applications to the Hitchin morphism.  Math. Proc. Cambridge Philos. Soc., 172(2):443–487, 2022.
  * [dCZ21] Mark Andrea A. de Cataldo and Siqing Zhang.  A cohomological non abelian Hodge theorem in positive characteristic.  https://arxiv.org/abs/2104.12970, 2021.
  * [dCZ22] Mark Andrea A. de Cataldo and Siqing Zhang.  Projective completion of moduli of $t$-connections on curves in positive and mixed characteristic.  Adv. Math., 401:Paper No. 108329, 43, 2022.
  * [Del77] P. Deligne.  Cohomologie étale, volume 569 of Lecture Notes in Mathematics.  Springer-Verlag, Berlin, 1977.  Séminaire de géométrie algébrique du Bois-Marie SGA $4\frac{1}{2}$.
  * [DK73] Pierre Deligne and Nicholas Katz.  Groupes de monodromie en géométrie algébrique. II.  Lecture Notes in Mathematics, Vol 340. Springer-Verlag, 1973.  Séminaire de Géométrie Algébrique du Bois-Marie 1967–1969 (SGA 7, II).
  * [FGI+05] Barbara Fantechi, Lothar Göttsche, Luc Illusie, Steven L. Kleiman, Nitin Nitsure, and Angelo Vistoli.  Fundamental algebraic geometry, volume 123 of Mathematical Surveys and Monographs.  American Mathematical Society, Providence, RI, 2005.  Grothendieck’s FGA explained.
  * [Gro16] Michael Groechenig.  Moduli of flat connections in positive characteristic.  Math. Res. Lett., 23(4):989–1047, 2016.
  * [Har10] Robin Hartshorne.  Deformation theory, volume 257 of Graduate Texts in Mathematics.  Springer, New York, 2010.
  * [HLHJ21] Daniel Halpern-Leistner, Andres Fernandez Herrero, and Trevor Jones.  Moduli spaces of sheaves via affine Grassmannians.  https://arxiv.org/abs/2107.02172, 2021.
  * [Lan14] Adrian Langer.  Semistable modules over Lie algebroids in positive characteristic.  Doc. Math., 19:509–540, 2014.
  * [Lan21] Adrian Langer.  Moduli spaces of semistable modules over Lie algebroids.  https://arxiv.org/abs/2107.03128, 2021.
  * [LP01] Yves Laszlo and Christian Pauly.  On the Hitchin morphism in positive characteristic.  Internat. Math. Res. Notices, (3):129–143, 2001.
  * [Mar12] E. Martinengo.  Infinitesimal deformations of Hitchin pairs and Hitchin map.  Internat. J. Math., 23(7):1250053, 30, 2012.
  * [Moc09] Takuro Mochizuki.  Kobayashi-Hitchin correspondence for tame harmonic bundles. II.  Geom. Topol., 13(1):359–455, 2009.
  * [Nit91] Nitin Nitsure.  Moduli space of semistable pairs on a curve.  Proc. London Math. Soc. (3), 62(2):275–300, 1991.
  * [Nit93] Nitin Nitsure.  Moduli of semistable logarithmic connections.  J. Amer. Math. Soc., 6(3):597–609, 1993.
  * [OV07] A. Ogus and V. Vologodsky.  Nonabelian Hodge theory in characteristic $p$.  Publ. Math. Inst. Hautes Études Sci., (106):1–138, 2007.
  * [Sch98] Daniel Schaub.  Courbes spectrales et compactifications de jacobiennes.  Math. Z., 227(2):295–312, 1998.
  * [Sim90] Carlos T. Simpson.  Harmonic bundles on noncompact curves.  J. Amer. Math. Soc., 3(3):713–770, 1990.
  * [Sim94a] Carlos T. Simpson.  Moduli of representations of the fundamental group of a smooth projective variety. I.  Inst. Hautes Études Sci. Publ. Math., (79):47–129, 1994.
  * [Sim94b] Carlos T. Simpson.  Moduli of representations of the fundamental group of a smooth projective variety. II.  Inst. Hautes Études Sci. Publ. Math., (80):5–79 (1995), 1994\.
  * [Sta22] The Stacks Project Authors.  Stacks Project.  https://stacks.math.columbia.edu, 2022.
  * [Sun20] Hao Sun.  Moduli space of $\Lambda$-modules on projective Deligne-Mumford stacks.  https://arxiv.org/abs/2003.11674, 2020.
  * [Yok95] Kôji Yokogawa.  Infinitesimal deformation of parabolic Higgs sheaves.  Internat. J. Math., 6(1):125–148, 1995.

Mark Andrea de Cataldo, Department of Mathematics, Stony Brook University,
<EMAIL_ADDRESS>
Andres Fernandez Herrero, Department of Mathematics, Columbia university,
<EMAIL_ADDRESS>
Siqing Zhang, Department of Mathematics, Stony Brook University,

<EMAIL_ADDRESS>


# PILE: Pairwise Iterative Logits Ensemble for Multi-Teacher Labeled
Distillation

Lianshang Cai, Linhao Zhang11footnotemark: 1, Dehong Ma, Jun Fan
Daiting Shi, Yi Wu, Zhicong Cheng, Simiu Gu, Dawei Yin22footnotemark: 2
Baidu Inc., Beijing, China
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Equal contribution. Corresponding authors.

###### Abstract

Pre-trained language models have become a crucial part of ranking systems and
achieved very impressive effects recently. To maintain high performance while
keeping efficient computations, knowledge distillation is widely used. In this
paper, we focus on two key questions in knowledge distillation for ranking
models: 1) how to ensemble knowledge from multi-teacher; 2) how to utilize the
label information of data in the distillation process. We propose a unified
algorithm called Pairwise Iterative Logits Ensemble (PILE) to tackle these two
questions simultaneously. PILE ensembles multi-teacher logits supervised by
label information in an iterative way and achieved competitive performance in
both offline and online experiments. The proposed method has been deployed in
a real-world commercial search system.

##  1 Introduction

Search engines have been an infrastructure in the Information Age to satisfy
people’s needs for querying about information. In modern search engines,
multi-stage pipelines are usually employed where ranking is usually known as
the very final stage. It takes as input the shortlisted candidates of relevant
documents (i.e. web pages) retrieved from previous stages, and concentrates on
sorting Lin et al. (2021) based on the degree of match between the latent
semantics of documents and the search intent of the user’s query.

With the flourishing of pre-trained language models Devlin et al. (2018); Sun
et al. (2019b); Lan et al. (2019); He et al. (2020), BERT-based models have
achieved state-of-the-art performance in a broad range of downstream tasks and
text ranking is no exception. Pre-trained rankers based on BERT show
impressive performance in ranking tasks (Nogueira and Cho, 2019; Nogueira et
al., 2019; Zou et al., 2021).

Despite the state-of-the-art performance pre-trained models yield in
laboratories, it is hardly possible to apply them directly in real-world
search engines. Their large numbers of parameters go against computational
efficiency while the online environment is strictly restricted in resources.
Therefore, before deploying a pre-trained model online, one necessary
procedure is to reduce computational costs.

Knowledge distillation is one of the most commonly used methods to reduce the
model size (Cristian et al., 2006; Hinton et al., 2015a) and accelerate the
computation process. In a standard workflow of distillation, a large model
(i.e. teacher) is pre-trained and finetuned well in advance, and a small model
(i.e. student) imitates the teacher model’s behaviors. The knowledge learned
by the teacher model is then transferred to the student model.

One of the risk factors hindering the improvement of the student model is that
the knowledge acquired by a single teacher may be insufficient and biased. A
straightforward solution is using an ensemble of multiple teachers for
knowledge transfer. The ensemble process takes the predictions of multiple
teachers into account and provides comprehensive guidance that helps to
improve student performance (You et al., 2017). However, these teachers
sometimes conflict with each other, and the heuristic of treating them equally
by taking the mean of their predictions ignores the fact that they vary in
confidence and correctness, thus often leading to suboptimal performance (Du
et al., 2020). Besides, the valuable label information is ignored. Hence, how
to ensemble knowledge from multi-teacher and how to utilize the label
information are two key questions during the distillation process.

In this work, we introduce a unified algorithm called Pairwise Iterative
Logits Ensemble (PILE) to tackle these two questions simultaneously. The key
idea of PILE is to assign a higher weight to teachers that produce more
consistent soft targets with the golden labels. The resulting soft targets not
only retain the generalization information transferred from the teacher models
but also are integrated with the label information annotated by human experts.

We conduct both offline and online experiments and the results validate the
effectiveness of PILE. The main contributions of this paper can be summarized
as follows:

  * •

We propose PILE, a specially designed ensemble algorithm to tackle multi-
teacher distillation and labeled distillation. To the best of our knowledge,
PILE is the first work that addresses these two key questions simultaneously
in the ranking scenario.

  * •

We conduct extensive offline and online experiments to demonstrate the
effectiveness of our proposed method. The results show that PILE effectively
boosts a real-world search engine’s performance.

##  2 Related Work

Text Ranking The goal of text ranking is to generate an ordered list of texts
in response to a query. Conventional learning-to-rank (LTR) techniques Li
(2014) are widely used for text ranking, which plays an important role in a
wide range of applications, like search engines and recommender systems. LTR
techniques can be roughly categorized into three types: pointwise approach
Cooper et al. (1992); Li et al. (2007), pairwise approach Joachims (2002);
Zheng et al. (2007), and listwise approach Cao et al. (2007); Burges (2010).
The former two are more widely used in practice as they are easier to
optimize.

Recently, deep learning approaches have been widely adopted in ranking and
BERT-based ranking models achieve state-of-the-art ranking effectiveness. For
example, Nogueira and Cho (2019) use BERT-large Devlin et al. (2018) as the
backbone and feed the concatenation of query and passage text to estimate the
relevant scores for passage re-ranking. Nogueira et al. (2019) formulate the
ranking problem as a pointwise and pairwise classification problem and tackle
them with two BERT models in a multi-stage ranking pipeline. Yilmaz et al.
(2019) aggregate sentence-level information to estimate the relevance of the
documents and transfer the learned model to capture cross-domain notions of
relevance.

However, the performance improvement comes at the cost of efficiency, which
limits their real-world application. There are several ways to maintain high
performance while keeping efficient computations for BERT-based models, such
as knowledge distillation Hinton et al. (2015b), weight sharing Lan et al.
(2019), pruning Pasandi et al. (2020); Xu et al. (2020), and quantization
Hubara et al. (2017); Jacob et al. (2018). In this paper, we focus on
knowledge distillation which has proven a promising way to compress large
models while maintaining performance.

Knowledge Distillation The idea of knowledge distillation was first introduced
by Cristian et al. (2006) to train small and fast models to mimic cumbersome
and complex models, without much loss in performance. Hinton et al. (2015a)
developed this idea further by minimizing the difference between their soft
target distribution. With the rise of the pre-training and fine-tuning
paradigm, various work has later extended this idea to large-scale pre-trained
models and shown impressive results on multiple NLP tasks Wang et al. (2019);
Rajpurkar et al. (2018); Lai et al. (2017) with a significant gain in training
efficiency. Sanh et al. (2019) conducted knowledge transfer during the pre-
training phase, also known as a task-agnostic way. Sun et al. (2019a) proposed
an approach to transfer knowledge between intermediate layers in the fine-
tuning stage. Jiao et al. (2020) additionally used attention-based
distillation and hidden states-based distillation for students to imitate
teachers’ behaviors in intermediate layers. Wang et al. (2020) introduced
self-attention relation-based transfer and teacher assistants Mirzadeh et al.
(2020) to further improve the performance of students.

Ensemble Knowledge Distillation There is also some other work exploring the
issues of multi-teacher distillation. For example, Du et al. (2020) adaptively
ensemble knowledge distillation to ﬁnd a better optimizing direction for the
student network. Wu et al. (2021) designed a co-ﬁnetuning framework to jointly
finetune multiple teachers for better collaborative knowledge distillation. Li
et al. (2021) explored the influence of teacher model adoption which is
promising for improving student performance. Different from the above work, we
investigate the problem of ensemble knowledge distillation in ranking tasks
and use the golden label to supervise the ensemble process.

##  3 Methodology

###  3.1 Ranking Task Definition

In a search system, the ranking task aims to measure the relative order of a
set of documents $D_{q}=\\{d_{i}\\}_{i=1}^{N}$ given a query $q\in Q$, where
$Q$ is a set of user queries and $D_{q}\subset\mathbb{D}$ is a set of
$q$-related documents retrieved from a large document corpus $\mathbb{D}$ (Liu
et al., 2021). The ranking model determines the order of documents by
computing the relevance score $f(q,d;\theta)$ of each query-document pair
$\\{(q,d_{i})\\}_{i=1}^{N}$, where $f$ is a scoring function parameterized by
$\theta$ representing the relevance of query $q$ and document $d$ .

As regards training procedure, the ranking model is learned by minimizing the
empirical loss over the training data as

$\mathcal{L}=\sum_{q\in Q}l(Y_{D_{q}},F(q,D_{q})),$

where $l$ is the loss function in learning to rank, e.g. pointwise loss,
pairwise loss or listwise loss, and $F(q,D_{q})=\\{f(q,d_{i})\\}_{i=1}^{N}$ is
a set of relevance scores, $Y_{D_{q}}=\\{y_{i}\\}_{i=1}^{N}$ is a set of
labels. The label $y_{i}$ is often assigned an integer range from 0 to 4,
representing the relevance of the query-document pair $(q,d_{i})$.

###  3.2 Knowledge Distillation

Due to the resource constraint, the ranking model can not directly serve
online in a real production environment and we use knowledge distillation (KD)
to compress the model size. In a commonly used KD framework, a large teacher
model $T$ is pre-trained or finetuned well in advance, and the knowledge of
the teacher is transferred to a small student $S$ by minimization of the
difference between them, which can be formulated as:

$\mathcal{L}_{KD}=\sum_{x\in\mathcal{D}}L(f^{S}(x),f^{T}(x)),$

where $\mathcal{D}$ denotes the training dataset and $x$ is the input sample,
$f^{S}(\cdot)$ and $f^{T}(\cdot)$ represent behavior measurements of teacher
and student models, and $L(\cdot)$ is a loss function to measure the
difference between their behaviors. The behaviors are usually represented by
soft target distributions of the last layer, hidden state distributions or
other deep semantic features such as self-attention distributions and
embedding layer outputs Hinton et al. (2015a); Sun et al. (2020); Jiao et al.
(2020); Wang et al. (2020, 2021). The methods that transfer the knowledge
between the internal layers are limited in generality since the teachers and
students are required to have the same model structure or align with each
other in the number of layers or the size of hidden layers. Based on this
consideration of generality, we perform knowledge distillation on the last
layer only.

###  3.3 Pairwise Iterative Logits Ensemble

Figure 1: Procedures of PILE: 1) start with initializing the ensemble
distillation logits with equal weights and 2) update a pair of resulting
logits in reversed order by reassigning the weights of teachers.

Knowledge distillation by one single teacher may bring some bias to the
student model while simple yet common mitigation is distillation on the
average of logits output by $n$ multiple teachers. However, the teachers with
diversity may conflict with each other since the biased teacher contributes
equally as the unbiased teacher, which corrodes the confidence of distillation
logits. Since the logits produced by teachers on different data vary in the
degree of confidence, we conduct a dynamic weighting process for the ensemble.
In the ranking task, the logit predicted by the teacher for a document
represents a measurement of relevance with the query. The larger logits
represent more correlation between the query and the document, and the
correlation information is already annotated in its label. Thus, we consider
utilizing the golden label to direct the assignment of weight. The procedures
of the PILE algorithm are provided in Figure 1.

We start with initializing the ensembled distillation logit $e^{(0)}(q,d)$ for
each query $q$ and document $d$ by way of averaging each teacher’s outputs:

$\displaystyle w_{k}(q,d)$ $\displaystyle=1$ $\displaystyle e^{(0)}(q,d)$
$\displaystyle=\frac{1}{Z(q,d)}\sum_{k}w_{k}(q,d)f_{k}(q,d)$ $\displaystyle
Z(q,d)$ $\displaystyle=\sum_{k}w_{k}(q,d),$

where $f_{k}(q,d)$ represents the relevance score predicted by $k$-th teacher
and $w_{k}(q,d)$ is its weight w.r.t query $q$ and document $d$.

Then, we perform the iterations of the update procedure. At iteration $t$, we
randomly choose a pair of docs $(d_{i},d_{j})$ related to the same query $q$,
and check whether the magnitude of their ensemble logits is consistent with
their labels. More specifically, if label $y_{i}>y_{j}$ and ensemble logits
$e^{(t)}(q,d_{i})<e^{(t)}(q,d_{j})$, we call this pair of docs in reversed
order or a negative pair and consider the ensemble logits have been biased. We
modify the biased ensemble logits by reassigning to zero the weight of
teachers responsible for the reversed order error. The reassignment rule can
be formulated as follows:

$\displaystyle\begin{split}w_{k}(q,d_{i})&=\left\\{\begin{array}[]{ll}0,&f_{k}(q,d_{i})<e^{(t)}(q,d_{i})\\\
1,&otherwise\end{array}\right.\end{split}$
$\displaystyle\begin{split}w_{k}(q,d_{j})&=\left\\{\begin{array}[]{ll}0,&f_{k}(q,d_{j})>e^{(t)}(q,d_{j})\\\
1,&otherwise\end{array}\right.\end{split}$

where we assume the docs pair $(d_{i},d_{j})$ is in reversed order and label
$y_{i}>y_{j}$ for ease of explanation. Then, we update the ensemble logits
with an update rate $\lambda$:

$\displaystyle\tilde{e}^{(t+1)}(q,d)$
$\displaystyle=\frac{1}{Z(q,d)}\sum_{i}w_{k}(q,d)f_{k}(q,d)$ $\displaystyle
Z(q,d)$ $\displaystyle=\sum_{k}w_{k}(q,d)$ $\displaystyle e^{(t+1)}(q,d)$
$\displaystyle=(1-\lambda)e^{(t)}(q,d)+\lambda\tilde{e}^{(t+1)}(q,d)$

We repeat the updating in an iterative process until the magnitude of the
ensemble logits of each pair of docs is consistent with their labels or it
reaches the maximum iteration number. We use the final ensemble logits to
perform knowledge distillation for the student model.

##  4 Experiments

To investigate the effectiveness of our proposed method, we conduct offline
experiments with baseline models and deploy our proposed model in the real-
world production environment. In this section, we report the details of the
experiment setups, datasets we used, evaluation metrics, the results of the
experiments, and the case study.

###  4.1 Datasets

The datasets on which we pre-train, finetune, and evaluate our proposed method
are collected from the Baidu search engine. For the pre-training stage, we
collect a large-scale unlabeled dataset (log data) by means of the anonymous
search log. The dataset contains $2,970,692,361$ query-document pairs.

As regards finetuning stage, queries and documents are collected from the
search pipelines and manually labeled on Baidu’s crowdsourcing platform, where
a group of hired annotators assigned an integer label range from $0$ to $4$ to
each query-document pair, representing their relevance as {bad, fair, good,
excellent, perfect}. We repeat the same process for the test set. The dataset
information is summarized in Table 1.

Data | #Query | #Query-Doc Pair
---|---|---
log data | 635,420,390 | 2,970,692,361
train data | 432,410 | 8,794,863
test data | 12,044 | 289,835
Table 1:  Dataset statistics

###  4.2 Training details

We use a 12-layer Transformer (Vaswani et al., 2017) structure with 768 hidden
sizes and 12 attention heads as the backbone of teacher models and a 6-layer
Transformer structure with 768 hidden sizes and 12 attention heads as student
models, where the parameters are randomly initialized, pre-trained and
finetuned on the datasets described in section 4.1.

In order to obtain multiple teachers with similar performance and some
differences for ensemble knowledge distillation, we use the same pre-trained
checkpoint and finetune it on different samplings of the total train data.
Specifically, the teacher $T_{1}$ is finetuned on the whole train data, and
teacher $T_{2}$ and $T_{3}$ are finetuned on $80\%$ of the whole train data.
The intuition behind this is that we want all the teachers to perform
relatively well but not behave as the same one otherwise the ensemble of three
teachers may degenerate into a single model, which will compromise the
benefits of the ensemble knowledge distillation. As a result, $T_{1}$ performs
best on the test set and the other two teachers have a competitive performance
as $T_{1}$. The results of three teachers on the test set are shown in Table
2.

In the training procedure, we use the Adam optimizer (Kingma and Ba, 2014)
with $\beta_{1}=0.9$ and $\beta_{2}=0.99$. For both 12-layer and 6-layer
models, we set the learning rate as 2e-5, the batch size as 64, and the warm-
up step as 1000. In the PILE procedure, the maximum number of iterations is
set to $|D_{q}|^{\frac{3}{2}}$ where $|D_{q}|$ is the size of documents set
$D_{q}$ related to a given query $q$ and the update rate $\lambda$ is set to
$0.9$. In the knowledge distillation stage, we set the warm-up step as $1000$,
the learning rate as 2e-5 and the batch size as 1024.

###  4.3 Evaluation Metrics

The evaluation metrics we used in the experiments are as follows.

The Positive-Negative Ratio (PNR) measures the consistency between the golden
labels and the scores output by models. For a given query $q$ and a list of
$N$ associated documents ranked by model, the PNR can be calculated by this
formulation:

$PNR=\frac{\sum_{i,j\in[1,N]}I\\{y_{i}>y_{j}\\}I\\{f(q,d_{i})>f(q,d_{j})\\}}{\sum_{i,j\in[1,N]}I\\{y_{i}>y_{j}\\}I\\{f(q,d_{i})<f(q,d_{j})\\}},$

where $I\\{\cdot\\}$ is the indicator function, taking the value $1$ if the
internal statement is true or $0$ otherwise. For the test set that contains a
good many queries, we average PNR values over all queries.

The Discounted Cumulative Gain (DCG) (Järvelin and Kekäläinen, 2000) is a
widely used metric that evaluates the ranking result of search engines. More
specifically, DCG is calculated as a weighted sum of the document’s relevance
degree $G_{i}$ at each position $i$, where the weight is assigned according to
the document’s position in the ranking results:

$DCG=\sum_{i}\frac{G_{i}}{log_{2}(i+1)}$

The Interleaving (Chapelle et al., 2012; Chuklin et al., 2015) is extensively
used for comparing a new system with the base system in industrial information
systems evaluation. The results of two systems are interleaved and presented
together to the end users, whose clicks would be credited to the system that
provides the corresponding results. The gain of the new system $A$ over the
base system $B$ can be denoted as $\Delta_{AB}$:

$\Delta_{AB}=0.5*\frac{wins(A)-wins(B)}{wins(A)+wins(B)+ties(A,B)},$

where $wins(A)$(or $wins(B)$) counts the number of times when the results
produced by system $A$ (or $B$) are more preferred than the other system for a
given query and $ties(A,B)$ counts the number of times when the two systems
are tied.

We also conduct a comparison called Good vs. Same vs. Bad (GSB) between two
systems by inviting professional annotators to estimate which system produced
a greater ranking result for each given query (Zhao et al., 2011). The gain of
a new system can be formulated as:

$\Delta_{GSB}=\frac{\\#Good-\\#Bad}{\\#Good+\\#Bad+\\#Same},$

where $\\#Good$ (or $\\#Bad$) denotes the number of queries that the new (or
base) system provides better ranking results and $\\#Same$ for the number of
results that are equal in quality.

###  4.4 Offline Experimental Results

We conduct several comparison experiments to verify our proposed method. The
models in the offline comparison experiments include:

  * •

Base: We use an ERNIE-based ranking model as our base model, which is
finetuned with a pairwise loss using human-labeled query-document pairs
without any guidelines from teachers;

  * •

single-KD: In this setting we add knowledge distillation loss when training
the base model using the teacher that performs best on the test set;

  * •

AE-KD: Instead of using the single teacher, this variant uses an ensemble of 3
teachers with averaged weight to perform knowledge distillation;

  * •

PILE-KD: When performing knowledge distillation, PILE-KD uses human-annotated
labels with the help of the PILE algorithm to conduct a dynamic weighting
process for the ensemble of 3 teachers.

Method | PNR | Improvement
---|---|---
Teacher1 | 3.21 | -
Teacher2 | 3.20 | -
Teacher3 | 3.19 | -
Base | 3.11 | -
\+ single-KD | 3.15 | +1.29%
\+ AE-KD | 3.16 | +1.61%
\+ PILE-KD | 3.18 | +2.25%
Table 2:  Offline comparison of the proposed methods.

The results of each model are shown in Table 2 with the improvement compared
to the base model. We also report the performance of the teachers used in
knowledge distillation. As we expected, all the distilled models consistently
outperform finetuned base model thanks to teacher models’ guidance and
regularization. And besides, using an ensemble of teachers gains further
promotion than the single teacher distillation. After ensembling multiple
teachers by averaging distillation logits, the PNR reaches 3.16, exceeding the
base by 1.61%. This shows that the remission from biased distillation by the
cooperation of multiple teachers improves students in semantic matching.
Moreover, by applying the PILE algorithm, we can see that the student can beat
the base model by a large margin w.r.t PNR, where the value is improved to
3.18 by 2.25% improvement. It shows the effectiveness of dynamic reduction of
biased teachers’ weight in the ensemble process.

###  4.5 Online Experimental Results

| Random | Tail
---|---|---
$\Delta$DCG | +0.10% | +0.27%
$\Delta$GSB | +3.70% | +1.62%

| Query Type | Query Length
---|---|---
Random | Tail | short | long
$\Delta_{AB}$ | +0.022% | +0.029% | +0.01% | +0.039%

Table 3:  Online comparison of the proposed methods.

To investigate the effectiveness of our proposed method in the real production
environment, we deploy the proposed model in Baidu Search, a widely used
Chinese search engine, and conduct online experiments for comparison.

The results are presented in Table 3, which comprises the performance
comparison regarding $\Delta DCG$, $\Delta GSB$, and $\Delta_{AB}$. We
consider the queries from the perspective of types and lengths in the search
log. The tail queries and long queries are the queries whose search frequency
is lower than 10 times per week or whose length is greater than 10
respectively. Since the heterogeneous search queries follow long-tail
distributions, the tail queries make up a significant part of the queries in
the search engine. As we can see the proposed method improves the performance
of the online ranking system consistently. Particularly, we can observe that
the gains of tail queries in the $\Delta DCG$ and $\Delta GSB$ for our method
are 0.27% and 1.62% respectively. Compared with AE-KD, PILE-KD can enable
students to retain the ability of teachers as much as possible, and the
improvement on long-tail queries also confirms this.

###  4.6 Ablation Studies

To illustrate the detailed effects of the proposed algorithm, we take a deep
insight into the contributions of each setting.

Figure 2: The effect of the number of teachers. Figure 3: The effect of update
rate $\lambda$ in PILE.

Number of teachers. We first focus on the effect of the number of teachers and
the results are shown in Figure2. As expected, the PNRs under both AE-KD and
PILE-KD settings increase with the number of teachers and consistently
outperform the result of the single-KD model, showing the benefits of using
multiple teachers for distillation. Besides, the performance under PILE-KD
settings has always been better than it under AE-KD settings, showing the
benefits of reducing noise and bias with the help of the label information
annotated by human experts in the process of ensemble knowledge distillation.

PILE update rate. We further examine the effect of the PILE update rate
$\lambda$. As shown in section 3.3, the update rate $\lambda$ controls the
smoothness of two successive iterations. As we can see in Figure 3, the PNR
result increases with the $\lambda$ ranging from 0.1 to 1.0, until reaching
the apex at 0.9. The larger update rate $\lambda$ makes the reliable teachers
get more weight in a PILE iteration, resulting in more dependable soft
targets. The results further prove that the distillation process can benefit
from PILE iteration.

###  4.7 Case Study

To illustrate the effect of the PILE algorithm in the ensemble knowledge
distillation concretely, we show a case that corrects the ensemble logits of
two documents that are in reversed order.

As we can see in Table 4, two documents that are related to the same query are
labeled by annotators as $0$ and $3$ respectively, representing their
relevance with the query. The teachers’ predictions of the relevance are
listed in the table as $T_{1}\sim T_{3}$. In the last two rows of the table,
we show the ensemble results using averaged weight (AE) and the PILE method
respectively.

query |  北京驾校教练一个月能挣多少钱？(How much does a Beijing driving school coach earn a month?)
---|---
doc |  北京私人教练一个月能挣多少钱？(How much does a Beijing personal trainer make a month?) |  驾校教练工资一月多少？百度知道 (How much is the salary of a driving school coach per month? Baidu Knows)
label |  0 |  3
$T_{1}$ |  0.0589 |  0.0271
$T_{2}$ |  0.1923 |  0.0331
$T_{3}$ |  0.1057 |  0.0983
AE |  0.1190 |  0.0528
PILE |  0.0590 |  0.0981
Table 4:  The teachers’ logits and ensemble logits for two documents.

To get the ensemble logits for knowledge distillation, AE takes the mean of
teachers’ predictions. However, influenced by individual teachers, the result
ensemble logits of the two documents are contrary to their golden labels. In
other words, the document with higher relevance is scored lower than the
irrelevant one after the ensemble process, which will confuse the student in
the knowledge transfer process. Benefiting from the PILE iteration, the
teachers consistent with the golden label are assigned more weight. The
resulting soft targets not only retain the knowledge that transfers from
teachers but also are integrated with the label information annotated by human
experts, which is more promising for knowledge distillation.

##  5 Conclusion

In this work, we propose an easy-to-implement approach to multi-teacher
distillation for large-scale ranking models. Our algorithm ensembles multi-
teacher logits supervised by human-annotated labels in an iterative way. We
conduct the offline experiments as well as deploy our methods in an online
commercial search system which demonstrates its superiority.

## Acknowledgements

The authors would like to thank the colleagues at Baidu Inc. for their
constructive suggestions that helped improve the paper, and hope everything
goes well with their work. The authors are also indebted to the anonymous
reviewers for their valuable comments and suggestions on the paper.

## References

  * Burges (2010) Christopher JC Burges. 2010.  From ranknet to lambdarank to lambdamart: An overview.  _Learning_ , 11(23-581):81.
  * Cao et al. (2007) Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007.  Learning to rank: from pairwise approach to listwise approach.  In _Proceedings of the 24th international conference on Machine learning_ , pages 129–136.
  * Chapelle et al. (2012) Olivier Chapelle, Thorsten Joachims, Filip Radlinski, and Yisong Yue. 2012.  Large-scale validation and analysis of interleaved search evaluation.  _ACM Trans. Inf. Syst._ , 30:6:1–6:41.
  * Chuklin et al. (2015) Aleksandr Chuklin, Anne Schuth, Ke Zhou, and Maarten De Rijke. 2015.  A comparative analysis of interleaving methods for aggregated search.  _ACM Trans. Inf. Syst._ , 33(2).
  * Cooper et al. (1992) William S Cooper, Fredric C Gey, and Daniel P Dabney. 1992.  Probabilistic retrieval based on staged logistic regression.  In _Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval_ , pages 198–210.
  * Cristian et al. (2006) Bucila Cristian, Rich Caruana, and Alexandru Niculescu-Mizil. 2006.  Model compression.  In _Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Du et al. (2020) Shangchen Du, Shan You, Xiaojie Li, Jianlong Wu, Fei Wang, Chen Qian, and Changshui Zhang. 2020.  Agree to disagree: Adaptive ensemble knowledge distillation in gradient space.  _Advances in Neural Information Processing Systems_ , 33:12345–12355.
  * He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020.  Deberta: Decoding-enhanced bert with disentangled attention.  _arXiv preprint arXiv:2006.03654_.
  * Hinton et al. (2015a) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015a.  Distilling the knowledge in a neural network.  _Computer Science_ , 14(7):38–39.
  * Hinton et al. (2015b) Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015b.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2(7).
  * Hubara et al. (2017) Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2017.  Quantized neural networks: Training neural networks with low precision weights and activations.  _The Journal of Machine Learning Research_ , 18(1):6869–6898.
  * Jacob et al. (2018) Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018.  Quantization and training of neural networks for efficient integer-arithmetic-only inference.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2704–2713.
  * Järvelin and Kekäläinen (2000) Kalervo Järvelin and Jaana Kekäläinen. 2000.  Ir evaluation methods for retrieving highly relevant documents.  In _Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_.
  * Jiao et al. (2020) X. Jiao, Y. Yin, L. Shang, X. Jiang, and Q. Liu. 2020.  Tinybert: Distilling bert for natural language understanding.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_.
  * Joachims (2002) Thorsten Joachims. 2002.  Optimizing search engines using clickthrough data.  In _Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining_.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. 2017.  RACE: Large-scale reading comprehension dataset from examinations.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019.  Albert: A lite bert for self-supervised learning of language representations.  _arXiv preprint arXiv:1909.11942_.
  * Li (2014) Hang Li. 2014.  Learning to rank for information retrieval and natural language processing.  _Synthesis lectures on human language technologies_ , 7(3):1–121.
  * Li et al. (2021) Lei Li, Yankai Lin, Shuhuai Ren, Peng Li, Jie Zhou, and Xu Sun. 2021.  Dynamic knowledge distillation for pre-trained language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 379–389, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Li et al. (2007) Ping Li, Qiang Wu, and Christopher Burges. 2007.  Mcrank: Learning to rank using multiple classification and gradient boosting.  _Advances in neural information processing systems_ , 20.
  * Lin et al. (2021) Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2021.  Pretrained transformers for text ranking: Bert and beyond.  _Synthesis Lectures on Human Language Technologies_ , 14(4):1–325.
  * Liu et al. (2021) Yiding Liu, Weixue Lu, Suqi Cheng, Daiting Shi, Shuaiqiang Wang, Zhicong Cheng, and Dawei Yin. 2021.  Pre-trained language model for web-scale retrieval in baidu search.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, pages 3365–3375.
  * Mirzadeh et al. (2020) Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. 2020.  Improved knowledge distillation via teacher assistant.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 5191–5198.
  * Nogueira and Cho (2019) Rodrigo Nogueira and Kyunghyun Cho. 2019.  Passage re-ranking with bert.  _arXiv preprint arXiv:1901.04085_.
  * Nogueira et al. (2019) Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019.  Multi-stage document ranking with bert.  _arXiv preprint arXiv:1910.14424_.
  * Pasandi et al. (2020) Morteza Mousa Pasandi, Mohsen Hajabdollahi, Nader Karimi, and Shadrokh Samavi. 2020\.  Modeling of pruning techniques for deep neural networks simplification.  _arXiv preprint arXiv:2001.04062_.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for SQuAD.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 784–789, Melbourne, Australia. Association for Computational Linguistics.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Sun et al. (2019a) S. Sun, Y. Cheng, Z. Gan, and J. Liu. 2019a.  Patient knowledge distillation for bert model compression.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_.
  * Sun et al. (2019b) Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019b.  Ernie: Enhanced representation through knowledge integration.  _arXiv preprint arXiv:1904.09223_.
  * Sun et al. (2020) Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020\.  MobileBERT: a compact task-agnostic BERT for resource-limited devices.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2158–2170, Online. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _International Conference on Learning Representations_.
  * Wang et al. (2021) Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. 2021.  MiniLMv2: Multi-head self-attention relation distillation for compressing pretrained transformers.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 2140–2151, Online. Association for Computational Linguistics.
  * Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020.  Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.  _Advances in Neural Information Processing Systems_ , 33:5776–5788.
  * Wu et al. (2021) Chuhan Wu, Fangzhao Wu, and Yongfeng Huang. 2021.  One teacher is enough? pre-trained language model distillation from multiple teachers.  _arXiv preprint arXiv:2106.01023_.
  * Xu et al. (2020) Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. 2020.  Bert-of-theseus: Compressing bert by progressive module replacing.  _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Yilmaz et al. (2019) Zeynep Akkalyoncu Yilmaz, Wei Yang, Haotian Zhang, and Jimmy Lin. 2019.  Cross-domain modeling of sentence-level evidence for document retrieval.  In _Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP)_.
  * You et al. (2017) Shan You, Chang Xu, Chao Xu, and Dacheng Tao. 2017.  Learning from multiple teacher networks.  _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_.
  * Zhao et al. (2011) Shiqi Zhao, Haifeng Wang, Chao Li, Ting Liu, and Yi Guan. 2011.  Automatically generating questions from queries for community-based question answering.  In _Proceedings of 5th international joint conference on natural language processing_.
  * Zheng et al. (2007) Zhaohui Zheng, Keke Chen, Gordon Sun, and Hongyuan Zha. 2007.  A regression framework for learning ranking functions using relative relevance judgments.  In _Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval_.
  * Zou et al. (2021) Lixin Zou, Shengqiang Zhang, Hengyi Cai, Dehong Ma, Suqi Cheng, Shuaiqiang Wang, Daiting Shi, Zhicong Cheng, and Dawei Yin. 2021.  Pre-trained language model based ranking in baidu search.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, pages 4014–4022.