($l\in\Lambda$) of the teacher and student networks. Using cosine similarity
metric, these representations can be used to build a $k$-nearest neighbor
similarity graph for the teacher
$\mathcal{G}_{l}^{T}(\textbf{X})=<\textbf{X}_{l}^{T},\textbf{W}_{l}^{T}>$, and
for the student
$\mathcal{G}_{l}^{S}(\textbf{X})=<\textbf{X}_{l}^{S},\textbf{W}_{l}^{S}>$.
Note that $\textbf{W}_{l}^{T}$ and $\textbf{W}_{l}^{S}$ represent the edge
weights, which represent the similarity between the $i$-th and $j$-th elements
of $\textbf{X}_{l}^{T}$ and $\textbf{X}_{l}^{S}$. Based on graph
representation for both the teacher and the student, the KD loss in Eqn. 10
can be updated as follows:

$\mathcal{L}=\sum_{l\in\Lambda}D\left(\mathcal{G}_{l}^{S}(\textbf{X}),\mathcal{G}_{l}^{T}(\textbf{X})\right)$
(28)

where the distance metric $D$ is based on $L_{2}$ distance.

TABLE IX: A summary of KD methods via graph representations.  Method | Purpose | Graph type | Knowledge type | Distance metric | Graph embedding
---|---|---|---|---|---
GKD [212] | Model compression | Heterogeneous graph | Layer-wise feature | $L_{2}$ |  GSP [219]
IRG [208] | Model compression | Knowledge graph | Middle layers | $L_{2}$ | Instance relations
MHKD [207] | Model compression | Knowledge graph | Middle layers | KL |  SVD [71] \+ Attention
KTG [213] | Model compression | Directed graph | Network model |  $L_{1}$ \+ KP loss | –
GFL [214] | Few-shot learning | GNN | Class of nodes | Frobenius norm |  HGR [220]
HGKD [173] | Few-shot learning | GNN | Class of nodes | Wasserstein |  GraphSAGE [221]
GRL [215] | Multi-task leaning | GNN | Class of nodes | Cross-entropy |  HKS [222]
Yang [218] | Model compression | GNN | Topological info. | KL | Attention

IRG [208] essentially is similar to GKD [212] in the construction of the
graph, however, IRG also takes into account the instance of graph
transformations. The aim of introducing feature space transformation across
layers is because there may be too tight or dense constraints and descriptions
fitting on the teacher’s instance features at intermediate layers. The
transformation of the instance relation graph is composed of vertex
transformation and edge transformation from the $l_{1}l$-th layer to
$l_{2}$-th layer, as shown in Fig. 10 (b). Thus the loss in Eqn. 28 can be
extended to:

$\begin{split}\mathcal{L}=\sum_{l\in\Lambda}D_{1}(\mathcal{G}_{l}^{S}(\textbf{X}),\mathcal{G}_{l}^{T}(\textbf{X}))+\\\
D_{2}\left((\Theta_{T}(\mathcal{G}_{l}^{S}(\textbf{X})),\Theta_{S}(\mathcal{G}_{l}^{T}(\textbf{X}))\right)\end{split}$
(29)

where $\Theta_{T}$ and $\Theta_{S}$ are the transformation functions for the
teacher and the student, respectively, and $D_{1}$ and $D_{2}$ are the
distance metrics for instance relation and instance translation.

MHKD [207] is a method that enables distilling data based knowledge from a
teacher network to a graph using an attention network (see Fig. 10(d)). Like
IRG [18], feature transformation is also considered to capture the intra-data
relations. The KD loss is based on the KL-divergence loss using the embedded
graphs from the teacher and the student. KTG [213] also exploits graph
representation; however, it focuses on a different perspective of KD. The
knowledge transfer graph provides a unified view of KD, and has the potential
to represent diverse knowledge patterns. Interestingly, each node in the graph
represents the direction of knowledge transfer. On each edge, a loss function
is defined for transferring knowledge between two nodes linked by each edge.
Thus, combining different loss functions can represent collaborative knowledge
learning with pair-wise knowledge transfer. Fig. 10(c) shows the knowledge
graph of diverse collaborative distillation with three nodes, where $L_{s,t}$
represents the loss function used for the training node.

In addition, GFL [214], HGKT [173], GRL [215] and MHGD [207] all resort to GNN
for the purpose of KD. HGKT and GFL focus on transferring knowledge from seen
classes to unseen classes in few-shot learning [223, 224]. GFL [214] leverages
the knowledge learned by the auxiliary graphs to improve semi-supervised node
classification in the target graph. As shown in Fig. 10(e), GFL learns the
representation of a whole graph, and ensures the transfer of a similarly
structured knowledge. Auxiliary graph reconstruction is achieved by using a
graph autoencoder. HGTK aims to build a heterogeneous graph focusing on
transferring intra-class and inter-class knowledge simultaneously. Inspired by
modeling class distribution in adversarial learning [78, 123, 181, 189, 182,
225], in which instances with the same class are expected to have the same
distribution, the knowledge is transferred from seen classes to new unseen
classes based on learned aggregation and embedding functions, as shown in Fig.
10 (f). GRL [215] builds a multi-task KD method for representation learning
based on DeepGraph [222].This knowledge is based on GNN, and maps raw graphs
to metric values. The learned graph metrics are then used as auxiliary tasks,
and the knowledge of the network is distilled into graph representations (see
Fig. 10(g)). The graph representation structure is learned via a CNN by
feeding the graph descriptor to it. We denote pairs of graph and graph-level
labels as $\\{(G_{i},y_{i})\\}_{i=1}^{N}$, where $G_{i}\in\mathcal{G}$,
$y_{i}\in\mathcal{Y}$, and $\mathcal{G}$, $\mathcal{Y}$ are the cardinally of
all possible graphs and labels respectively. Then, the loss for learning the
model parameters are described as:

$\mathcal{L}=\operatorname{\mathbb{E}}[D(y_{i},f(G_{i};\theta))]$ (30)

where $\theta$ are the model parameters.

Open challenges Graph representations are of significant importance for
tackling KD problems because they better capture the hierarchical patterns in
locally connected structures. However, there are some challenges. Firstly,
graph representations are difficult to generalize because they are limited to
structured data or specific types of data. Secondly, it is challenging to
measure graph distances appropriately, since existing distance measure (e.g.,
$l_{2}$) may not fit well. Thirdly, layer-wise distillation is difficult to
achieve in graph KD, because graph representation models and network
structures in such cases are limited.

###  8.3 KD for semi-/self-supervised learning

Overall insight: KD with S-T learning aims to learn a rich representation by
training a model with a large number of unlabeled datasets, and limited amount
of labeled data.

Semi-supervised learning usually handles the problem of over-fitting due to
the lack of high-quality labels of training data. To this end, most methods
apply S-T learning that assumes a dual role as a teacher and a student. The
student model aims to learn the given data as before, and the teacher learns
from the noisy data and generates predicted targets. These are then
transferred to the student model via consistency cost. In self-supervised
learning, the student itself generates knowledge to be learned via various
approaches, and the knowledge is then transferred by the student to itself via
distillation losses. We now provide a detailed analysis of the technical
details of the existing methods.

####  8.3.1 Semi-supervised learning

The baseline S-T frameworks for semi-supervised learning was initialized by
Laine et al. [226] and Tarvainen et al. [2], as illustrated in Fig. 1(b). The
student and the teacher models have the same structures, and the teacher
learns from noise and transfers knowledge to the student via consistency cost.
Interestingly, in [2], the teacher’s weights are updated using the earth
moving average (EMA) of the student’s weights. Inspired by [2], Luo et al.
[227], Zhang et al. [228], French et al. [229], Choi et al. [230], Cai et al.
[231] and Xu et al. [232] all employ similar frameworks where the teacher’s
weights are updated using exponential moving average (EMA) of the student.
However, Ke et al.[157] mention that using a coupled EMA teacher is not
sufficient for the student, since the degree of coupling increases as the
training goes on. To tackle this problem, the teacher is replaced with another
student, and two students are optimized individually during training while a
stabilization constraint is provided for knowledge exchange (similar to mutual
learning [34]).

Instead of taking independent weights between the teacher and the student,
Hailat et al. [233] employ weight-sharing, in which the last two fully
connected layers of the teacher and the student are kept independent. The
teacher model plays the role of teaching the student, stabilizing the overall
model, and attempting to clean the noisy labels in the training dataset. In
contrast, Gong et al. [104] and Xie et al. [21] follow the conventional
distillation strategy proposed by [1], where a pretrained teacher is
introduced to generate learnable knowledge using unlabeled data, and utilizes
it as privileged knowledge to teach the student on labeled data. However,
during learning of the student, Xie et al. inject noise (e.g., dropout) to the
student such that it learns better than the teacher. Papernot et al. [38]
propose to distill from multiple teachers (an ensemble of teachers) on a
disjoint subset of sensitive data (augmented with noise) and to aggregate the
knowledge of teachers to guide the student on query data.

####  8.3.2 Self-supervised learning

Distilling knowledge for self-supervised learning aims to preserve the learned
representation for the student itself, as depicted in Fig. 1(c). Using pseudo
labels is the most common approach, as done in [71, 234]. Specifically, Lee et
al. [71] adopt self-supervised learning for KD, which not only ensures the
transferred knowledge does not vanish, but also provides an additional
performance improvement. In contrast, Noroozi et al. [234] propose to transfer
knowledge by reducing the learned representation (from a pretrained teacher
model) to pseudo-labels (via clustering) on the unlabeled dataset, which are
then utilized to learn a smaller student network. Another approach is based on
data augmentation (e.g., rotation, cropping, color permutation) [166, 164,
235], which has been mentioned in Sec. 6.2.3. In contrast to making the
‘positive’ and ‘negative’ (augmented) examples, BYOL [178] directly bootstraps
the representations with two neural networks, referred to as online and target
networks, that interact and learn from each other. This spirit is somehow
similar to mutual learning [34]; however, BYOL trains its online network to
predict the target network’s representation of another augmented view of the
same image. The promising performance of BYOL might point out a new direction
of KD with self-supervised learning via representation boostrap rather than
using negative examples.

####  8.3.3 Potentials and open challenges

Based on the technical analysis for the KD methods in semi-/self-supervised
learning, it is noticeable that online distillation is the mainstream.
However, there are several challenges. First, as pointed by [157], using EMA
for updating teacher’s weights might lead to less optimal learning of
knowledge. Second, no methods attempt to exploit the rich feature knowledge
from teacher models. Third, data augmentation methods in these distillation
methods are less effective compared to those proposed in Sec. 6.2, in which
the advantages of adversarial learning are distinctive. Fourth, the
representations of knowledge in these methods are limited and less effective.
BYOL [178] opens a door for representation boost, and there exists a potential
to further bind this idea with KD in the further research. Moreover, it has
the potential to exploit a better-structured data representation approach,
such as in GNNs. With these challenges, the future directions of KD for
semi-/self-supervised learning could gain inspirations from exploiting feature
knowledge and more sophisticated data augmentation methods together with more
robust representation approaches.

###  8.4 Few-shot learning

Insight:Is it possible possible to learn an effective student model to
classify unseen classes (query sets) by distilling knowledge from a teacher
model with the support set?

In contrast to the methods discussed in Sec. 5.2 focusing on distillation with
a few samples for training a student network (without learning to generalize
to new classes), this section stresses on analyzing the technical details of
few-shot learning with KD. Few-shot learning is to classify new data, having
seen from only a few training examples. Few-shot learning itself is a meta-
learning problem in which the DNN learns how to learn to classify, given a set
of training tasks, and evaluate using a set of test tasks. Here, the goal is
to discriminate between $N$ classes with $K$ examples of each (so-called
$N$-way-$K$-shot classification). In this setting, these training examples are
known as the support set. In addition, there are further examples of the same
classes, known as a query set. The approaches for learning prior knowledge of
a few-shot are usually based on three types: prior knowledge about similarity,
prior knowledge about learning procedure, and prior knowledge about data. We
now analyze the KD methods for few-shot learning [46, 236, 28, 126, 237] that
have been recently proposed.

Prior knowledge about similarity: Park et al. [28] propose distance-wise and
angle-wise distillation losses. The aim is to penalize the structural
differences in relation to the learned representations between the teacher and
the student for few-shot learning.

Prior knowledge about learning procedure: [236, 237] tackle the second type of
prior knowledge, namely learning procedure. To be specific, Flennerhag et al.
[236] focuses on transferring knowledge across the learning process, in which
the information from previous tasks is distilled to facilitate the learning on
new tasks. However, Jin et al. [237] address the problem of learning a meta-
learner that can automatically learn what knowledge to transfer from the
source network to where in the target network.

Prior knowledge about data: Dvornik et al. [46] and Liu et al. [126] address
the third type of prior knowledge, namely data variance. To be specific, in
[46], en ensemble of several teacher networks is elaborated to leverage the
variance of the classifiers and encouraged to cooperate while encouraging the
diversity of prediction. However, in [126], the goal is to preserve the
knowledge of the teacher (e.g., intra-class relationship) learned at the
pretraining stage by generating pseudo labels for training samples in the
fine-tuning set.

####  8.4.1 What’s challenging?

Based on our analysis, the existing techniques actually expose crucial
challenges. First, the overall performance of KD-based few-shot learning is
convincing, but the power of meta-learning is somehow degraded or exempted.
Second, transferring knowledge from multi-source networks is a potential, but
identifying what to learn and where to transfer is heavily based on the meta-
learner, and selecting which teacher to learn is computation-complex. Third,
all approaches focus on a task-specific distillation, but the performance
drops as the domain shifts. Thus, future works may focus more on handling
these problems.

###  8.5 Incremental Learning

Overall insight: KD for incremental learning mainly deals with two challenges:
maintaining the performance on old classes, and balancing between old and new
classes.

Incremental learning investigates learning the new knowledge continuously to
update the model’s knowledge, while maintaining the existing knowledge [238].
Many attempts [169, 239, 240, 238, 102, 241, 193] have been made to utilize KD
in addressing the challenge of maintaining the old knowledge. Based on the
number of teacher networks used for distillation, these methods can be
categorized into two types: distillation from a single teacher and
distillation from multiple teachers.

####  8.5.1 Distillation from a single teacher

Shmelkov et al. [241], Wu et al. [238], Michieli et al. [239] and Hou et al.
[169] focus on learning student networks for new classes, by distilling
knowledge (logits information) from pretrained teachers on old-class data.
Although these methods vary in tasks and distillation process, they follow
similar S-T structures. Usually, the pretrained model is taken as the teacher,
and the same network or a different network is employed to adapt for new
classes. Michieli et al. exploit the intermediate feature representations and
transfer them to the student.

####  8.5.2 Distillation from multiple teachers

Castro et al. [240], Zhou et al. [102] and Ammar et al. [242] concentrate on
learning an incremental model with multiple teachers. Specifically, Castro et
al. share the same feature extractor between teachers and the student. The
teachers contain old classes, and their logits are used for distillation and
classification. Interestingly, Zhou et al. propose a multi-model and multi-
level KD strategy in which all previous model snapshots are leveraged to learn
the last model (student). This approach is similar to born-again KD methods,
as mentioned in Sec. 6.2, where the student model at the last step, and is
updated using the assembled knowledge from all previous steps. However, the
assembled knowledge also depends on the intermediate feature representations.
Ammar et al. develop a cross-domain incremental RL framework, in which the
transferable knowledge is shared and projected to different task domains of
the task-specific student peers.

####  8.5.3 Open challenges

The existing methods rely on multi-step training (offline). However, it will
be more significant if the online (one-step) distillation approaches can be
utilized to improve the learning efficiency and performance. Moreover,
existing methods require accessing the previous data to avoid ambiguities
between the update steps. However, the possibility of data-free distillation
methods remains open. Furthermore, existing methods only tackle the
incremental learning of new classes in the same data domain, but it will be
fruitful if cross-domain distillation methods can be applied in this
direction.

###  8.6 Reinforcement learning

Overall insight: KD in reinforcement learning is to encourage polices (such as
students) in the ensemble to learn from the best policies (such as teachers),
thus enabling rapid improvement and continuous optimization.

Reinforcement learning (RL) is a learning problem that trains a policy to
interact with the environment in way that yields maximal reward. To use the
best policy to guide other policies, KD has been employed in [243, 18, 236,
244, 245, 246, 247, 107]. Based on the specialties of these methods, we divide
them into three categories, and provide an explicit analysis. We assume
familiarity with basics of RL, and skip the definitions of Deep Q-network and
A3C.

####  8.6.1 Collaborative distillation

Xue et al. [246], Hong et al. [245], and Lin et al. [247] focus on
collaborative distillation, which is similar to mutual learning [34]. In Xue
et al., the agents teach each other based on the reinforcement rule, and
teaching occurs between the value function of the agents (students and
teachers). Note that the knowledge is provided by a group of student peers
periodically, and assembled to enhance the learning speed and stability as in
[245]. However, Hong et al.[245] periodically distill the best-performing
policy to the rest of the ensemble. Lin et al. stress on collaborative
learning among heterogeneous learning agents, and incorporate the knowledge
into online training.

####  8.6.2 Model compression with RL-based distillation

Ashok et al. [243] tackle the problem of model compression via RL. The method
takes a larger teacher network, and outputs a compressed student network
derived from it. In particular, two recurrent policy networks are employed to
aggressively remove layers from the teacher network, and to carefully reduce
the size of each remaining layer. The learned student network is evaluated by
a reward, which is a score based on the accuracy and compression of the
teacher.

####  8.6.3 Random network distillation

Burda et al. [244] focus on a different perspective where the prediction
problem is randomly generated. The approach involves two networks: the target
(student) network, which is fixed and randomly initialized, and a predictor
(teacher) network trained on the data collected by the agent. With the
knowledge distilled from the predictor, the target network tends to have lower
prediction errors. Rusu et al. [107] also apply random initialization for the
target network. However, they focus more on online learning action policies,
which can be either single-task or muti-task.

####  8.6.4 Potentials of RL-based KD

We have analyzed existing RL-based KD methods in detail. Especially, we notice
that model compression via RL-based KD is promising due to its extraordinary
merits. First, RL-based KD better addresses the problem of scalability of
network models. This is similar to neural architecture search (NAS). Moreover,
the reward functions in RL-based KD better balance the accuracy-size trade-
off. It is also possible to transfer knowledge from a smaller model to a
larger model, which is a distinctive advantage over other KD methods.

##  9 Applications for visual intelligence

###  9.1 Semantic and motion segmentation

Insight: Semantic segmentation is a structured problem, and structure
information (e.g., spatial context structures) needs to be taken into account
when distilling knowledge for semantic segmentation networks.

Semantic segmentation is a special classification problem that predicts the
category label in a pixel-wise manner. As existing the state-of-the-art (SOTA)
methods such as fully convolutional networks [248] have large model sizes and
high computation costs, some methods [152, 197, 239, 249, 250, 251, 252, 117,
159] have been proposed to train lightweight networks via KD. Although these
methods vary in their learning methods, most of them share the same
distillation frameworks. Particularly, Xie et al. [252], Shan et al. [249],
and Michieli et al. [239] focused on pixel-wise, feature-based distillation
methods. Moreover, Liu et al.[197] and He et al. [250] both exploited
affinity-based distillation strategy using intermediate features. Liu et al.
also employed pixel-wise and holistic KD losses via adversarial learning. In
contrast, Dou et al. [152] focused on unpaired multi-modal segmentation, and
proposed an online KD method via mutual learning [34]. Chen et al. [251]
proposed a target-guided KD approach to learn the real image style by training
the student to imitate a teacher trained with real images. Mullapudi et al.
[159] trained a compact video segmentation model via online distillation, in
which a teacher’s output was used as a learning target to adapt the student
and select the next frame for supervision.

###  9.2 KD for visual detection and tracking

Insight: Challenges such as regression, region proposals, and less voluminous
labels must be considered when distilling visual detectors.

Visual detection is a crucial high-level task in computer vision. Speed and
accuracy are two key factors for visual detectors. KD is a potential choice to
achieve sped-up and lightweight network models. However, applying distillation
methods to detection is more challenging than applying classification methods.
First, detection performance degrades seriously after compression. Second,
detection classes are not equally important, and special considerations for
distillation have to be taken into account. Third, domain and data
generalization has to be considered for a distilled detector. To overcome
these challenges, several impressive KD methods [175, 65, 192, 253, 254, 255,
256, 237, 257, 258, 259, 260, 261, 262, 263, 264] have been proposed for
compressing visual detection networks. We categorize these methods according
to their specialties (e.g., pedestrian detection).

####  9.2.1 Generic object detection

[175, 192, 254, 253, 258, 256, 65, 265, 266] aimed to learn lightweight object
detectors with KD. Among these works, Chen et al. [254] and Hao et al. [253]
highlighted learning a class-incremental student detector by following the
generic KD framework (from a pretrained teacher). However, novel object
detection losses were adopted as strong impetus for learning new classes.
These losses handled classification results, location results, the detected
region of interest, and all intermediate region proposals. Moreover, Chen et
al. [175] learned a student detector by distilling knowledge from the
intermediate layer, logits, and regressor of the teacher, in contrast to [65],
in which only the intermediate layer of the teacher was utilized based on
fine-grained imitation masks to identify informative locations. Jin et al.
[258], Tang et al. [256], and Hong et al.[192] exploited multiple intermediate
layers as useful knowledge. Jin et al. designed an uncertainty-aware
distillation loss to learn the multiple-shot features from the teacher
network. However, Hong et al.and Tang et al. were based on one-stage KD
(online) via adversarial learning and semi-supervised learning, respectively.
In contrast, Liu et al. [266] combined single S-T learning and mutual learning
of students for learning lightweight tracking networks.

####  9.2.2 Pedestrian detection

While pedestrian detection is based on generic object detection, various sizes
and aspect ratios of pedestrians under extreme illumination conditions are
challenges. To learn an effective lightweight detector, Chen et al. [255]
suggested using the unified hierarchical knowledge via multiple intermediate
supervisions, in which not only the feature pyramid (from low-level to high-
level features) and region features, but also the logits information were
distilled. Kruthiventi et al. [261] learned an effective student detector in
challenging illumination conditions by extracting dark knowledge (both RGB and
thermal-like hint features) from a multi-modal teacher network.

####  9.2.3 Face detection

Ge et al. [262] and Karlekar et al. [267] compressed face detectors to
recognize low-resolution faces via selective KD (last hidden layer) from
teachers which were initialized to recognize high-resolution faces. In
contrast, Jin et al. [237], Luo et al. [263], and Feng et al. [264] used
single type of image. Jin et al.focused on compressing face detectors by using
the supervisory signal from the classification maps of teacher models and
regression maps of the ground truth. They identify that it is better to learn
a classification map of a larger model than that of smaller models. Feng et
al.presented a triplet KD method to transfer knowledge from a teacher model to
a student model, in which a triplet of samples, the anchor image, positive
image, and negative image, was used. The purpose of the triplet loss was to
minimize the feature similarity between the anchor and positive images, while
maximizing that between the anchor and negative images. Luo et al.addressed
the importance of neurons at the higher hidden layer of the teacher, and a
neuron selection method was applied to select neurons that were crucial for
teaching the student. Dong et al. [268] concentrated on the interaction
between the teacher and the students. Two students learned to generate pseudo
facial landmark labels, which were filtered and selected as the qualified
knowledge by the teacher.

####  9.2.4 Vehicle detection and driving learning

Lee et al. [259], Saputra et al. [257], and Xu et al. [182] focused more on
detection tasks for autonomous driving. In particular, Lee et al.focused on
compressing a vehicle maker classification system based on cascaded CNNs
(teacher) into a single CNN structure (student). The proposed distillation
method used the feature map as the transfer medium, and the teacher and
student were trained in parallel (online distillation). Although the detection
task was different, Xu et al. build a binary weight Yolo vehicle detector by
mincing the feature maps of the teacher network from easy tasks to difficult
ones progressively. Zhao et al. [269] exploited an S-T framework to encourage
the student to learn the teacher’s sufficient and invariant representation
knowledge (based on semantic segmentation) for driving.

####  9.2.5 Pose detection

Distilling human pose detectors has several challenges. First, lightweight
detectors have to deal with arbitrary person images/videos to determine joint
locations with unconstrained human appearances. Second, the detectors must be
robust in viewing conditions and background noises. Third, the detectors
should have fast inference speeds, and be memory-efficient. To this end, [270,
271, 145, 272, 273, 274, 275] formulated various distillation methods. Zhang
et al. [270] achieved effective knowledge transfer by distilling the joint
confidence maps from a pre-trained teacher model, whereas Huang et al. [272]
exploited the heat map and location map of a pretrained teacher as the
knowledge to be distilled. Furthermore, Xu et al. [273], Thoker et al. [145],
and Martinez et al. [271] focused on multi-person pose estimation. Thoker et
al. addressed cross-modality distillation problems, in which a novel framework
based on mutual learning [34] of two students supervised by one teacher was
initialized. Xu et al. [273] learned the integral knowledge, namely, the
feature, logits, and structured information via a discriminator under the
standard S-T framework. Martinez et al. [271] trained the student to mimic the
confidence maps, feature maps, and inner-stage predictions of a pre-trained
teacher with depth images. Wang et al. [274] trained a 3D pose estimation
network by distilling knowledge from non-rigid structure from motion using
only 2D landmark annotations. In contrast, Nie et al. [275] introduced an
online KD in which the pose kernels in videos were distilled by leveraging the
temporal cues from the previous frame in a one-shot learning manner.

###  9.3 Domain adaptation

Insight: Is it possible to distill knowledge of a teacher in one domain to a
student in another domain?

Domain adaptation (DA) addresses the problem of learning a target domain with
the help of a different but related source domain [276]. Since Lopez et al.
[277] and Gupta et al. [143] initially proposed the technique of transferring
knowledge between images from different modalities (called generalized
distillation), it is natural to ask if this novel technique be used to address
the problem of DA. The challenge of DA usually comes with transferring
knowledge from the source model (usually with labels) to the target domain
with unlabeled data. To address the problem, several KD methods based on S-T
frameworks [276, 278, 279, 232, 230, 280, 281, 231, 282] have been proposed
recently. Although these methods are focused on diverse tasks, technically,
they can be categorized into two types: unsupervised and semi-supervised DA
via KD.

####  9.3.1 Semi-supervised DA

French et al. [229], Choi et al. [230], Cai et al. [231], Xu et al. [232], and
Cho et al. [283] proposed similar S-T frameworks for semantic segmentation and
object detection. These frameworks were the updated methods of Mean-Teacher
[2], which is based on self-ensemble of the student networks (teacher and
student models have the same structure). Note that the weights of the teacher
models in these methods are the EMAs of the weights of the student models. In
contrast, Choi et al.added a target-guided generator to produce augmented
images instead of stochastic augmentation, as in [232, 229, 231]. Cai et al.
also exploited the feature knowledge from the teacher model, and applied
region-level and intra-graph consistency losses instead of the mean square
error loss.

In contrast, Ao et al. [276] proposed a generalized distillation DA method by
applying the generalized distillation information [277] to multiple teachers
to generate soft labels, which were then used to supervise the student model
(this framework is similar to online KD from multiple teachers as mentioned in
Sec. 4.2). Cho et al. [283] proposed an S-T learning framework, in which a
smaller depth prediction network was trained based on the supervision of the
auxiliary information (ensemble of multiple depth predictions) obtained from a
larger stereo matching network (teacher).

####  9.3.2 Unsupervised DA

Some methods such as [280, 278] distill the knowledge from the source domain
to the target domain based on adversarial learning [78] and image translation
[200, 123, 124]. Technically, images in the source domain are translated to
images in the target domain as data augmentation, and cross-domain consistency
losses are adopted to force the teacher and student models to produce
consistent predictions. Tsai et al. [281] and Deng et al. [282] focused on
aligning the feature similarities between teacher and student models, compared
with Meng et al. [279], who focused on aligning softmax outputs.

###  9.4 Depth and scene flow estimation

Insight: The challenges for distilling depth and flow estimation tasks come
with transferring the knowledge of data and labels.

Depth and optical flow estimations are low-level vision tasks aiming to
estimate the 3D structure and motion of the scene. There are several
challenges. First, in contrast to other tasks (e.g., semantic segmentation),
depth and flow estimations do not have class labels. Thus, applying existing
KD techniques directly may not work well. Moreover, learning a lightweight
student model usually requires a large amount of labeled data to achieve
robust generalization capability. However, acquiring these data is very
costly.

To address these challenges, Guo et al. [176], Pilzer et al. [284], and Tosi
et al. [285] proposed distillation-based approaches to learn monocular depth
estimation. These methods were focused on handling the second challenge,
namely, data distillation. Specifically, Pilzer et al. [284] proposed an
unsupervised distillation approach, where the left image was translated to the
right via the image translation framework [200, 123]. The inconsistencies
between the left and right images were used to improve depth estimation, which
was finally used to improve the student network via KD. In contrast, Guo et
al. and Tosi et al. focused on cross-domain KD, which aimed to distill the
proxy labels obtained from the stereo network (teacher) to learn a student
depth estimation network. Choi et al. [283] learned a student network for
monocular depth inference by distilling the knowledge of depth predictions
from a stereo teacher network via the data ensemble strategy.

Liu et al. [286] and Aleotti et al. [287] proposed data-distillation methods
for scene flow estimation. Liu et al. distilled reliable predictions from a
teacher network with unlabeled data, and used these predictions (for non-
occluded pixels) as annotations to guide a student network to learn the
optical flow. They proposed to leverage on the knowledge learned by the
teacher networks specialized in stereo to distill proxy annotations, which is
similar to the KD method for depth estimation in [176, 285]. Tosi et al. [288]
learned a compact network for predicting holistic scene understanding tasks
including depth, optical flow, and motion segmentation, based on distillation
of proxy semantic labels and semantic-aware self-distillation of optical
information.

###  9.5 Image translation

Insight: Distilling GAN frameworks for image translation has to consider three
factors: large number of parameters of the generators, no ground truth labels
for training data, and complex framework (both generator and discriminator).

Attempts were made in several works to compress GANs for image translation
with KD. Aguinaldo et al. [198] focused on unconditional GANs, and proposed to
learn a smaller student generator by distilling knowledge from the generated
images of a larger teacher generator using mean squared error (MSE). However,
the knowledge incorporated in the teacher discriminator was not investigated.
In contrast, Chen et al. [203] and Li et al. [199] focused on conditional
GANs, and exploited the knowledge from the teacher discriminator.
Specifically, Chen et al.included a student discriminator to measure the
distances between real images and images generated by the student and teacher
generators. The student GAN was then trained under the supervision of the
teacher GAN. In particular, Li et al. [199] adopted the discriminator of the
teacher as the student discriminator, and fine-tuned the discriminator
together with the compressed generator, which was automatically found with
significantly lower computation cost and fewer parameters, by using NAS. In
contrast, Wang et al. [289] focused on compressing encoder-decoder based
neural style transfer network via collaborative distillation (between the
encoder and its decoder), where the student was restricted to learn the linear
embedding of the teacher’s output.

###  9.6 KD for Video understanding

####  9.6.1 Video classification and recognition

Bhardwaj et al. [290] and Wang et al. [291] employed the general S-T learning
framework for video classification. The student was trained with processing
only a few frames of the video, and produced a representation similar to that
of the teacher. Gan et al. [292] focused on video concept learning for action
recognition and event detection by using web videos and images. The learned
knowledge from teacher network (Lead network) was used to filter out the noisy
images. These were then used to fine-tune the teacher network to obtain a
student network (Exceeding network). Gan et al. [293] explored geometry as a
new type of practical auxiliary knowledge for self-supervised learning of
video representations. Fu et al. [294] propose focused on video attention
prediction by leveraging both spatial and temporal knowledge. Farhadi et al.
[295] distill the temporal knowledge from a teacher model over the selected
video frames to a student model.

####  9.6.2 Video captioning

[296, 297] exploited the potential of graph-based S-T learning for image
captioning. Specifically, Zhang et al. [296] leveraged the object-level
information (teacher) to learn the scene feature representation (student) via
a spatio-temporal graph. Pan et al. [297] highlighted the importance of the
relational graph connecting for all the objects in the video, and forced the
caption model to learn the abundant linguistic language via teacher-
recommended learning.

##  10 Discussions

In this section, we discuss some fundamental questions and challenges that are
crucial for better understanding and improving KD.

###  10.1 Are bigger models better teachers?

The early assumption and idea behind KD are that soft labels (probabilities)
from a trained teacher reflect more about the distribution of data than the
ground truth labels [1]. If this is true, then it is expected that as the
teacher becomes more robust, the knowledge (soft labels) provided by the
teacher would be more reliable and better capture the distribution of classes.
That is, a more robust teacher provides constructive knowledge and supervision
to the student. Thus, the intuitive approach for learning a more accurate
student is to employ a bigger and more robust teacher. However, based on the
experimental results in [16], it is found out that a bigger and more robust
model does not always make a better teacher. As the teacher’s capacity grows,
the student’s accuracy rises to some extent, and then begins to drop. We
summarize two crucial reasons behind the lack of theoretical support for KD,
based on [16, 35].

  * •

The student is able to follow the teacher, but it cannot absorb useful
knowledge from the teacher. This indicates that there is a mismatch between
the KD losses and accuracy evaluation methods. As pointed in [35], the
optimization method used could have a large impact on the distillation risk.
Thus, optimization methods might be crucial for significant KD to the student.

  * •

Another reason comes from when the student is unable to follow the teacher due
to the large model capacity between the teacher and the student. It is stated
in [53, 1] that the S-T similarity is highly related to how well the student
can mimic the teacher. If the student is similar to the teacher, it will
produce outputs similar to the teacher.

Intermediate feature representations are also effective knowledge that can be
used to learn the student [52, 54]. The common approach for feature-based
distillation is to transfer the features into a type of representation that
the student can easily learn. In such a case, are bigger models are better
teachers? As pointed in [52], feature-based distillation is better than the
distillation of soft labels, and deeper students perform better than shallower
ones. In addition, the performance of the student increases upon increasing
the number of layers (feature representations) [54]. However, when the student
is fixed, a bigger teacher does not always teach a better student. When the
similarity between the teacher and student is relatively high, the student
tends to achieve plausible results.

###  10.2 Is a pretrained teacher important?

While most works focus on learning a smaller student based on the pretrained
teacher, the distillation is not always efficient and effective. When the
model capacity between the teacher and the student is large, it is hard for
the student to follow the teacher, thus inducing the difficulty of
optimization. Is a pretrained teacher important for learning a compact student
with plausible performance? [34, 40] propose learning from student peers, each
of which has the same model complicity. The greatest advantage of this
distillation approach is efficiency, since the pretraining of a high capacity
teacher is exempted. Instead of teaching, the student peers learn to cooperate
with each other to obtain an optimal learning solution. Surprisingly, learning
without the teacher even enables improving the performance. The question of
why learning without the teacher is better has been studied in [2]. Their
results indicate that the compact student may have a less chance of
overfitting. Moreover, [16] suggests that early stopping of training on
ImageNet [51] achieves better performance. The ensemble of students pool their
collective predictions, thus helping to converge at a more robust minima as
pointed in [34].

###  10.3 Is born-again self-distillation better?

Born-again network [30] is the initial self-distillation method in which the
student is trained sequentially, and the later step is supervised by the
earlier generation. At the end of the procedure, all the student generations
are assembled together to get an additional gain. So is self-distillation in
the generations better? [16] finds that network architecture heavily
determines the success of KD in generations. Although the ensemble of the
student models from all the generations outperforms a single model trained
from scratch, the ensemble does not outperform an ensemble of an equal number
of models trained from scratch.

Instead, recent works [168, 164, 163] shift the focus from sequential self-
distillation (multiple stages) to the one-stage (online) manner. The student
distills knowledge to itself without resorting to the teacher and heavy
computation. These methods show more efficiency, less computation costs, and
higher accuracy. The reason for such better performance has been pointed out
in [168, 163]. They have figured out that that online self-distillation can
help student models converge to flat minima. Moreover, self-distillation
prevents student models from the ‘vanishing gradient’ problem. Lastly, self-
distillation helps to extract more discriminative features. In summary, online
self-distillation shows significant advantages than sequential distillation
methods and is more generalizable.

###  10.4 Single teacher vs multiple teachers

It is noticeable that recent distillation methods turn to exploit the
potential of learning from multiple teachers. Is learning from multiple
teachers really better than learning from a single teacher? To answer this
question, [37] intuitively identified that the student can fuse different
predictions from multiple teachers to establish its own comprehensive
understanding of the knowledge. The intuition behind this is that by unifying
the knowledge from the ensemble of teachers, the relative similarity
relationship among teachers is maintained. This provides a more integrated
dark knowledge for the student. Similar to mutual learning [34, 40], the
ensemble of teachers collects the individual predictions (knowledge) together,
thus converging at minima that are more robust. Lastly, learning from multiple
teachers relieves training difficulties such as vanishing gradient problems.

###  10.5 Is data-free distillation effective enough?

In the absence of training data, some novel methods [116, 118, 112, 119] have
been proposed to achieve plausible results. A theoretical explanation for why
such methods are robust enough for learning a portable student has yet to be
proposed. These methods are focused only on classification, and the
generalization capability of such methods is still low. Most works employ
generators to generate the ‘latent’ images from noise via adversarial learning
[132, 123], but such methods are relatively hard to train and computationally
expensive.

###  10.6 Logits vs features

The knowledge defined in existing KD methods is from three aspects: logits,
feature maps (intermediate layers), and both. However, it is still unclear
which one of these represents better knowledge. While works such as [52, 54,
55, 53, 61] focus on better interpretation of feature representations and
claim that features might contain richer information; some other works [1, 34,
115, 15] mention that softened labels (logits) could represent each sample by
a class distribution, and a student can easily learn the intra-class
variations. However, it is noticeable that KD via logits has obvious
drawbacks. First, its effectiveness is limited to the softmax loss function,
and it relies on the number of classes (cannot be applied to low-level vision
tasks). Secondly, when the capacity between the teacher and the student is
big, it is hard for the student to follow the teacher’s class probabilities
[16]. Moreover, as studied in [61], semantically similar inputs tend to elicit
similar activation patterns in teacher networks, indicating that the
similarity-preserving knowledge from intermediate features express not only
the representation space, but also the activations of object category (similar
to class distributions). Thus, we can clearly see that features provide more
affluent knowledge than logits, and generalize better to the problems without
class labels.

###  10.7 Interpretability of KD

In Sec. 3, we provided a theoretical analysis of KD based on the information
maximization theory. It is commonly acknowledged that the teacher model’s dark
knowledge provides the privileged information on similarity of class
categories to improve the learning of students [1, 4]. However, why KD works
is also an important question. There are some methods that explore the
principles of KD from the view of label smoothing [172], visual concepts
[177], category similarity [1], etc. Specifically, [172] found that KD is a
learned label smoothing regularization (LSR), and LSR is an ad-hoc KD. Even a
poorly-trained teacher can improve the student’s performance, and the weak
student could improve the teacher. However, the findings in [172] only focus
on classification-related tasks, and these intriguing results do not apply to
the tasks without labels [199, 203]. In contrast, [177] claims that KD makes
DNN learn more task-related visual concepts and discard task-irreverent
concepts to learn discriminative features. From a general perspective, the
quantification of visual concepts in [177] provides a more intuitive
interpretation for the success of KD. However, there exists a strong need that
more intensive research needs to be done in this direction.

###  10.8 Network architecture vs effectiveness of KD.

It has been demonstrated that distillation position has a significant impact
on the effectiveness of KD [16, 53]. Most methods demonstrate this by
deploying the same network for both teacher and student. However, many fail to
transfer across very different teacher and student architectures. Recently,
[10] found that [57, 11, 52] perform poorly even on very similar student and
teacher architectures. [172] also reported an intriguing finding that a poorly
trained teacher also can improve the student’s performance. It is thus quite
imperative to excavate how network architecture affects the effectiveness of
KD and why KD fails to work when the network architectures of student and
teacher are different.

##  11 New outlooks and perspectives

In this section, we provide some ideas and discuss future directions of
knowledge distillation. We take the latest deep learning methods (e.g., neural
architecture search (NAS), graph neural network (GNN)), novel non-Euclidean
distances (e.g., hypersphere), better feature representation approaches, and
potential vision applications, such as 360∘ vision [298] and event-based
vision [158] into account.

###  11.1 Potential of NAS

In recent years, NAS has become a popular topic in deep learning. NAS has the
potential of automating the design of neural networks. Therefore, it can be
efficient for searching more compact student models. In such a way, NAS can be
incorporated with KD for model compression. This has been recently
demonstrated for GAN compression [199, 299]. It is shown to be effective for
finding efficient student model from the teacher with lower computation costs
and fewer parameters. It turns out that NAS improves the compression ratio and
accelerates the KD process. A similar approach is taken by [243] learning to
remove layers of teacher network based on reinforcement learning (RL). Thus,
we propose that NAS with RL can be a good direction of KD for model
compression. This might significantly relieve the complexity and enhance the
learning efficiency of existing methods, in which the student is manually
designed based on the teacher.

###  11.2 Potential of GNN

Although GNN has brought progress in KD learning under the S-T framework, some
challenges remain. This is because most methods rely on finding structured
data on which graph-based algorithms can be applied. [18] considers the
instance features and instance relationships as instance graphs, and [215]
builds an input graph representation for multi-task knowledge distillation.
However, in knowledge distillation, there exists non-structural knowledge in
addition to the structural knowledge (e.g., training data, logits,
intermediate features, and outputs of teacher), and it is necessary to
construct a flexible knowledge graph to tackle the non-structural distillation
process.

###  11.3 Non-Euclidean distillation measure

Existing KD losses are mostly dependent on Euclidean loss (e.g., $l_{1}$), and
have their own limitations. [300] has shown that algorithms that regularize
with Euclidean distance, (e.g.MSE loss) are easily confused by random
features. The difficulty arises when the model capacity between the teacher
and the student is large. Besides, $l_{2}$ regularization does not penalize
small weights enough. Inspired by a recent work [301] on GAN training, we
propose that it is useful to exploit the information of higher-order
statistics of data in non-Euclidean spaces (e.g., hypersphere). This is
because geometric constraints induced by the non-Euclidean distance might make
training more stable, thus improving the efficiency of KD.

###  11.4 Better feature representations

Existing methods that focus on KD with multiple teachers show potential for
handling cross-domain problems or other problems where the ground truth is not
available. However, the ensemble of feature representations [48, 64, 111] is
still challenging in some aspects. One critical challenge is fusing the
feature representations and balancing each of them with robust gating
mechanisms. Manually assigning weights to each component may hurt the
diversity and flexibility of individual feature representations, thus
impairing the effectiveness of ensemble knowledge. One possible solution is
attention gates, as demonstrated in some detection tasks [302, 303]. The aim
of this approach is to highlight the important feature dimensions and prune
feature responses to preserve only the activations relevant to the specific
task. Another approach is inspired by the gating mechanism used in long short-
term memory(LSTM) [304, 305]. That is, this gate unit in KD is elaborately
designed to remember features across different image regions and to control
the pass of each region feature as a whole by their contribution to the task
(e.g., classification) with the weight of importance.

###  11.5 A more constructive theoretical analysis

While KD shows impressive performance improvements in many tasks, the
intuition behind it is still unclear. Recently, [16] explained conventional KD
[1] using linear models, and [9, 8, 10] focus on explaining feature-based KD.
Mobahi et al.[163] provides theoretical analysis for self-distillation.
However, the mechanism behind data-free KD and KD from multiple teachers is
still unknown. Therefore, further theoretical studies on explaining the
principles of these methods should be undertaken.

###  11.6 Potentials for special vision problems

While existing KD techniques are mostly developed based on vision problems
(e.g., classification), they rarely exploit some special vision fields such as
360∘ vision [298] and event-based vision [158, 124, 306]. The biggest
challenge for both these vision fields is the lack of labeled data, and
learning in these needs a special change of inputs for neural networks. Thus,
the potential of KD, particularly cross-modal KD, for these two fields is
promising. By distilling knowledge from the teacher trained with RGB images or
frames to the student network specialized in learning to predict 360∘ images
or stacked event images, it not only handles the problem of lack of data, but
also achieves desirable results in the prediction tasks.

###  11.7 Integration of vision, speech and NLP.

As a potential, it is promising to apply KD to the integrated learning
problems of vision, speech, and NLP. Although recent attempts of cross-modal
KD [142, 146, 145, 147] focus on transferring the knowledge from one modality
(e.g., video) to the other (e.g., sound) on the end tasks, it is still
challenging to learn the end tasks for the integration of the three
modalities. The major challenge may come from collecting paired data of three
modalities; however, it is possible to apply GAN or representation learning
methods to unsupervised cross-modal KD for learning effective end tasks.

##  12 Conclusion

This review of KD and S-T learning has covered major technical details and
applications for visual intelligence. We provide a formal definition of the
problem, and introduce the taxonomy methods for existing KD approaches.
Drawing connections among these approaches, we identify a new active area of
research that is likely to create new methods that take advantage of the
strengths of each paradigm. Each taxonomy of the KD methods shows the current
technical status regarding its advantages and disadvantages. Based on explicit
analyses, we then discuss methods to overcome the challenges, and break the
bottlenecks by exploiting new deep learning methods, new KD losses, and new
vision application fields.

## Acknowledgments

This work was supported by the National Research Foundation of Korea(NRF)
grant funded by the Korea government(MSIT) (NRF-2018R1A2B3008640) and
Institute of Information & Communications Technology Planning &
Evaluation(IITP) grant funded by Korea government(MSIT) (No.2020-0-00440,
Development of Artificial Intelligence Technology that Continuously Improves
Itself as the Situation Changes in the Real World).

## References

  * [1] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” _NIPS Deep Learning Workshop_ , 2015.
  * [2] A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” in _Advances in neural information processing systems_ , 2017, pp. 1195–1204.
  * [3] S. Gutstein, O. Fuentes, and E. Freudenthal, “Knowledge transfer in deep convolutional neural nets,” _International Journal on Artificial Intelligence Tools_ , vol. 17, no. 03, pp. 555–567, 2008.
  * [4] C. Bucilua, R. Caruana, and A. Niculescu-Mizil, “Model compression,” in _Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining_. ACM, 2006, pp. 535–541.
  * [5] https://intellabs.github.io/distiller/knowledge_distillation.html.
  * [6] https://github.com/FLHonker/Awesome-Knowledge-Distillation.
  * [7] https://github.com/dkozlov/awesome-knowledge-distillation.
  * [8] S. Ahn, S. X. Hu, A. Damianou, N. D. Lawrence, and Z. Dai, “Variational information distillation for knowledge transfer,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * [9] S. Hegde, R. Prasad, R. Hebbalaguppe, and V. Kumar, “Variational student: Learning compact and sparser networks in knowledge distillation framework,” _ICASSP_ , 2020.
  * [10] Y. Tian, D. Krishnan, and P. Isola, “Contrastive representation distillation,” in _International Conference on Learning Representations_ , 2019.
  * [11] J. Ba and R. Caruana, “Do deep nets really need to be deep?” in _Advances in neural information processing systems_ , 2014, pp. 2654–2662.
  * [12] K. Mangalam and M. Salzamann, “On compressing u-net using knowledge distillation,” _arXiv preprint arXiv:1812.00249_ , 2018.
  * [13] Q. Ding, S. Wu, H. Sun, J. Guo, and S.-T. Xia, “Adaptive regularization of labels,” _arXiv preprint arXiv:1908.05474_ , 2019.
  * [14] J. Cho and M. Lee, “Building a compact convolutional neural network for embedded intelligent sensor systems using group sparsity and knowledge distillation,” _Sensors_ , vol. 19, no. 19, p. 4307, 2019.
  * [15] T. Wen, S. Lai, and X. Qian, “Preparing lessons: Improve knowledge distillation with better supervision,” _arXiv preprint arXiv:1911.07471_ , 2019.
  * [16] J. H. Cho and B. Hariharan, “On the efficacy of knowledge distillation,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 4794–4802.
  * [17] C. Yang, L. Xie, C. Su, and A. L. Yuille, “Snapshot distillation: Teacher-student optimization in one generation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2859–2868.
  * [18] I.-J. Liu, J. Peng, and A. G. Schwing, “Knowledge flow: Improve upon your teachers,” _Seventh International Conference on Learning Representations_ , 2019.
  * [19] A. Mishra and D. Marr, “Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy,” _arXiv preprint arXiv:1711.05852_ , 2017.
  * [20] Y. Li, J. Yang, Y. Song, L. Cao, J. Luo, and L.-J. Li, “Learning from noisy labels with distillation,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 1910–1918.
  * [21] Q. Xie, E. Hovy, M.-T. Luong, and Q. V. Le, “Self-training with noisy student improves imagenet classification,” _CVPR_ , 2020.
  * [22] Y. Xu, Y. Wang, H. Chen, K. Han, X. Chunjing, D. Tao, and C. Xu, “Positive-unlabeled compression on the cloud,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 2561–2570.
  * [23] F. Sarfraz, E. Arani, and B. Zonooz, “Noisy collaboration in knowledge distillation,” _openreview.net_ , 2019.
  * [24] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” _The journal of machine learning research_ , vol. 15, no. 1, pp. 1929–1958, 2014.
  * [25] C. Yang, L. Xie, S. Qiao, and A. L. Yuille, “Training deep neural networks in generations: A more tolerant teacher educates better students,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 5628–5635.
  * [26] L. Yu, V. O. Yazici, X. Liu, J. v. d. Weijer, Y. Cheng, and A. Ramisa, “Learning metrics from teachers: Compact networks for image embedding,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2907–2916.
  * [27] S. Arora, M. M. Khapra, and H. G. Ramaswamy, “On knowledge distillation from complex networks for response prediction,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 2019, pp. 3813–3822.
  * [28] W. Park, D. Kim, Y. Lu, and M. Cho, “Relational knowledge distillation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3967–3976.
  * [29] B. Peng, X. Jin, J. Liu, D. Li, Y. Wu, Y. Liu, S. Zhou, and Z. Zhang, “Correlation congruence for knowledge distillation,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 5007–5016.
  * [30] T. Furlanello, Z. C. Lipton, M. Tschannen, L. Itti, and A. Anandkumar, “Born again neural networks,” _ICML_ , 2018.
  * [31] D. Wang, Y. Li, Y. Lin, and Y. Zhuang, “Relational knowledge transfer for zero-shot learning,” in _Thirtieth AAAI Conference on Artificial Intelligence_ , 2016.
  * [32] S.-I. Mirzadeh, M. Farajtabar, A. Li, and H. Ghasemzadeh, “Improved knowledge distillation via teacher assistant: Bridging the gap between student and teacher,” _the Association for the Advancement of Artificial Intelligence_ , 2019.
  * [33] A. Malinin, B. Mlodozeniec, and M. Gales, “Ensemble distribution distillation,” _arXiv preprint arXiv:1905.00076_ , 2019.
  * [34] Y. Zhang, T. Xiang, T. M. Hospedales, and H. Lu, “Deep mutual learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4320–4328.
  * [35] M. Phuong and C. H. Lampert, “Distillation-based training for multi-exit architectures,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 1355–1364.
  * [36] S. Zagoruyko and N. Komodakis, “Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer,” _International Conference on Learning Representations_ , 2016.
  * [37] S. You, C. Xu, C. Xu, and D. Tao, “Learning from multiple teacher networks,” in _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 2017, pp. 1285–1294.
  * [38] N. Papernot, M. Abadi, U. Erlingsson, I. Goodfellow, and K. Talwar, “Semi-supervised knowledge transfer for deep learning from private training data,” _ICLR_ , 2017.
  * [39] B. B. Sau and V. N. Balasubramanian, “Deep model compression: Distilling knowledge from noisy teachers,” _arXiv preprint arXiv:1610.09650_ , 2016\.
  * [40] X. Lan, X. Zhu, and S. Gong, “Knowledge distillation by on-the-fly native ensemble,” in _Proceedings of the 32nd International Conference on Neural Information Processing Systems_. Curran Associates Inc., 2018, pp. 7528–7538.
  * [41] G. Song and W. Chai, “Collaborative learning for deep neural networks,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 1832–1841.
  * [42] I. Radosavovic, P. Dollár, R. Girshick, G. Gkioxari, and K. He, “Data distillation: Towards omni-supervised learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4119–4128.
  * [43] X. Tan, Y. Ren, D. He, T. Qin, Z. Zhao, and T.-Y. Liu, “Multilingual neural machine translation with knowledge distillation,” _ICLR_ , 2019.
  * [44] J. Vongkulbhisal, P. Vinayavekhin, and M. Visentini-Scarzanella, “Unifying heterogeneous classifiers with distillation,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * [45] A. Wu, W.-S. Zheng, X. Guo, and J.-H. Lai, “Distilled person re-identification: Towards a more scalable system,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 1187–1196.
  * [46] N. Dvornik, C. Schmid, and J. Mairal, “Diversity with cooperation: Ensemble methods for few-shot classification,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 3723–3731.
  * [47] Z. Yang, L. Shou, M. Gong, W. Lin, and D. Jiang, “Model compression with two-stage multi-teacher knowledge distillation for web question answering system,” _WSDM_ , 2019.
  * [48] S. Park and N. Kwak, “Feed: Feature-level ensemble for knowledge distillation,” _European Conference on Artificial Intelligence_ , 2019.
  * [49] K. Lee, L. T. Nguyen, and B. Shim, “Stochasticity and skip connections improve knowledge transfer,” 2019.
  * [50] D. Chen, J.-P. Mei, C. Wang, Y. Feng, and C. Chen, “Online knowledge distillation with diverse peers,” _Association for the Advancement of Artificial Intelligence_ , 2020.
  * [51] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [52] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, “Fitnets: Hints for thin deep nets,” _International Conference on Learning Representations_ , 2014.
  * [53] B. Heo, J. Kim, S. Yun, H. Park, N. Kwak, and J. Y. Choi, “A comprehensive overhaul of feature distillation,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 1921–1930.
  * [54] J. Kim, S. Park, and N. Kwak, “Paraphrasing complex network: Network compression via factor transfer,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 2760–2769.
  * [55] Z. Huang and N. Wang, “Like what you like: Knowledge distill via neuron selectivity transfer,” _International Conference on Learning Representations_ , 2017.
  * [56] Z. Zhang, G. Ning, and Z. He, “Knowledge projection for effective design of thinner and faster deep neural networks,” 2017.
  * [57] J. Yim, D. Joo, J. Bae, and J. Kim, “A gift from knowledge distillation: Fast optimization, network minimization and transfer learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 4133–4141.
  * [58] W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou, “Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers,” _arXiv preprint arXiv:2002.10957_ , 2020.
  * [59] S. Srinivas and F. Fleuret, “Knowledge transfer with jacobian matching,” _International Conference on Machine Learning_ , 2018.
  * [60] X. Jin, B. Peng, Y. Wu, Y. Liu, J. Liu, D. Liang, J. Yan, and X. Hu, “Knowledge distillation via route constrained optimization,” _ICCV_ , 2019\.
  * [61] F. Tung and G. Mori, “Similarity-preserving knowledge distillation,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 1365–1374.
  * [62] Z. Shen, Z. He, and X. Xue, “Meal: Multi-model ensemble via adversarial learning,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 4886–4893.
  * [63] B. Heo, M. Lee, S. Yun, and J. Y. Choi, “Knowledge transfer via distillation of activation boundaries formed by hidden neurons,” 2018.
  * [64] I. Chung, S. Park, J. Kim, and N. Kwak, “Feature-map-level online adversarial knowledge distillation,” 2020.
  * [65] T. Wang, L. Yuan, X. Zhang, and J. Feng, “Distilling object detectors with fine-grained feature imitation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4933–4942.
  * [66] S. Changyong, L. Peng, X. Yuan, Q. Yanyun, D. Longquan, and M. Lizhuang, “Knowledge squeezed adversarial network compression,” _arXiv preprint arXiv:1904.05100_ , 2019.
  * [67] A. Kulkarni, N. Panchi, and S. Chiddarwar, “Stagewise knowledge distillation,” _arXiv preprint arXiv:1911.06786_ , 2019.
  * [68] G. Aguilar, Y. Ling, Y. Zhang, B. Yao, X. Fan, and E. Guo, “Knowledge distillation from internal representations,” _AAAI_ , 2019.
  * [69] M. Gao, Y. Shen, Q. Li, and C. C. Loy, “Residual knowledge distillation,” _arXiv preprint arXiv:2002.09168_ , 2020.
  * [70] H.-T. Li, S.-C. Lin, C.-Y. Chen, and C.-K. Chiang, “Layer-level knowledge distillation for deep neural network learning,” _Applied Sciences_ , vol. 9, no. 10, p. 1966, 2019.
  * [71] S. H. Lee, D. H. Kim, and B. C. Song, “Self-supervised knowledge distillation using singular value decomposition,” in _European Conference on Computer Vision_. Springer, 2018, pp. 339–354.
  * [72] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [73] N. Passalis and A. Tefas, “Learning deep representations with probabilistic knowledge transfer,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 268–284.
  * [74] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” _arXiv preprint arXiv:1502.03167_ , 2015.
  * [75] S. Zagoruyko and N. Komodakis, “Wide residual networks,” _BMVC_ , 2016.
  * [76] D. Barber and F. V. Agakov, “The im algorithm: a variational approach to information maximization,” in _Advances in neural information processing systems_ , 2003, p. None.
  * [77] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in _Proceedings of the 26th annual international conference on machine learning_. ACM, 2009, pp. 41–48.
  * [78] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Advances in neural information processing systems_ , 2014, pp. 2672–2680.
  * [79] F. Horn and K.-R. Müller, “Learning similarity preserving representations with neural similarity and context encoders,” 2016.
  * [80] K. Clark, M.-T. Luong, U. Khandelwal, C. D. Manning, and Q. V. Le, “Bam! born-again multi-task networks for natural language understanding,” _ACL_ , 2019.
  * [81] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” in _International conference on machine learning_ , 2016, pp. 1928–1937.
  * [82] C. Zhang and Y. Peng, “Better and faster: knowledge transfer from multiple self-supervised learning tasks via graph distillation for video classification,” _arXiv preprint arXiv:1804.10069_ , 2018.
  * [83] C. Shen, M. Xue, X. Wang, J. Song, L. Sun, and M. Song, “Customizing student networks from heterogeneous teachers via adaptive knowledge amalgamation,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 3504–3513.
  * [84] S. Ruder, P. Ghaffari, and J. G. Breslin, “Knowledge adaptation: Teaching to adapt,” _arXiv preprint arXiv:1702.02052_ , 2017.
  * [85] X. Zhu, S. Gong _et al._ , “Knowledge distillation by on-the-fly native ensemble,” in _Advances in neural information processing systems_ , 2018, pp. 7517–7527.
  * [86] J. Vongkulbhisal, P. Vinayavekhin, and M. Visentini-Scarzanella, “Unifying heterogeneous classifiers with distillation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3175–3184.
  * [87] Z. Yang, L. Shou, M. Gong, W. Lin, and D. Jiang, “Model compression with two-stage multi-teacher knowledge distillation for web question answering system,” in _Proceedings of the 13th International Conference on Web Search and Data Mining_ , 2020, pp. 690–698.
  * [88] L. Tran, B. S. Veeling, K. Roth, J. Swiatkowski, J. V. Dillon, J. Snoek, S. Mandt, T. Salimans, S. Nowozin, and R. Jenatton, “Hydra: Preserving ensemble diversity for model distillation,” _arXiv preprint arXiv:2001.04694_ , 2020.
  * [89] A. Ruiz and J. Verbeek, “Distilled hierarchical neural ensembles with adaptive inference cost,” _arXiv preprint arXiv:2003.01474_ , 2020.
  * [90] M.-C. Wu and C.-T. Chiu, “Multi-teacher knowledge distillation for compressed video action recognition based on deep learning,” _Journal of Systems Architecture_ , vol. 103, p. 101695, 2020.
  * [91] T. Fukuda, M. Suzuki, G. Kurata, S. Thomas, J. Cui, and B. Ramabhadran, “Efficient knowledge distillation from an ensemble of teachers.” in _Interspeech_ , 2017, pp. 3697–3701.
  * [92] M. Mehak and V. N. Balasubramanian, “Knowledge distillation from multiple teachers using visual explanations,” Ph.D. dissertation, Indian Institute of Technology Hyderabad, 2018.
  * [93] J.-w. Jung, H. Heo, H.-j. Shim, and H.-J. Yu, “Distilling the knowledge of specialist deep neural networks in acoustic scene classification,” 2019.
  * [94] S. Sun, Y. Cheng, Z. Gan, and J. Liu, “Patient knowledge distillation for bert model compression,” _EMNLP_ , 2019.
  * [95] L. Liu, H. Wang, J. Lin, R. Socher, and C. Xiong, “Attentive student meets multi-task teacher: Improved knowledge distillation for pretrained models,” _arXiv preprint arXiv:1911.03588_ , 2019.
  * [96] X. Liu, P. He, W. Chen, and J. Gao, “Improving multi-task deep neural networks via knowledge distillation for natural language understanding,” _arXiv preprint arXiv:1904.09482_ , 2019.
  * [97] X. He, Z. Zhou, and L. Thiele, “Multi-task zipping via layer-wise neuron sharing,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 6016–6026.
  * [98] C. Shen, X. Wang, J. Song, L. Sun, and M. Song, “Amalgamating knowledge towards comprehensive classification,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 3068–3075.
  * [99] J. Ye, Y. Ji, X. Wang, K. Ou, D. Tao, and M. Song, “Student becoming the master: Knowledge amalgamation for joint scene parsing, depth estimation, and more,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2829–2838.
  * [100] S. Luo, X. Wang, G. Fang, Y. Hu, D. Tao, and M. Song, “Knowledge amalgamation from heterogeneous networks by common feature learning,” _IJCAI_ , 2019.
  * [101] R. Anil, G. Pereyra, A. Passos, R. Ormandi, G. E. Dahl, and G. E. Hinton, “Large scale distributed neural network training through online distillation,” _ICLR_ , 2018.
  * [102] P. Zhou, L. Mai, J. Zhang, N. Xu, Z. Wu, and L. S. Davis, “M2kd: Multi-model and multi-level knowledge distillation for incremental learning,” _The British Machine Vision Conference (BMVC)_ , 2019.
  * [103] L. Xiang and G. Ding, “Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification,” _ECCV_ , 2020.
  * [104] C. Gong, X. Chang, M. Fang, and J. Yang, “Teaching semi-supervised classifier via generalized distillation.” in _IJCAI_ , 2018, pp. 2156–2162.
  * [105] J. Ye, X. Wang, Y. Ji, K. Ou, and M. Song, “Amalgamating filtered knowledge: learning task-customized student from multi-task teachers,” _IJCAI_ , 2019\.
  * [106] J. Gao, Z. Li, R. Nevatia _et al._ , “Knowledge concentration: Learning 100k object classifiers in a single cnn,” _arXiv preprint arXiv:1711.07607_ , 2017.
  * [107] A. A. Rusu, S. G. Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukcuoglu, and R. Hadsell, “Policy distillation,” _arXiv preprint arXiv:1511.06295_ , 2015.
  * [108] K. Ahmed, M. H. Baig, and L. Torresani, “Network of experts for large-scale image categorization,” in _European Conference on Computer Vision_. Springer, 2016, pp. 516–532.
  * [109] I. M. Galván, P. Isasi, R. Aler, and J. M. Valls, “A selective learning method to improve the generalization of multilayer feedforward neural networks,” _International journal of neural systems_ , vol. 11, no. 02, pp. 167–177, 2001.
  * [110] J. Kim, M. Hyun, I. Chung, and N. Kwak, “Feature fusion for online mutual knowledge distillation,” _International Conference on Pattern Recognition_ , 2019.
  * [111] S. Hou, X. Liu, and Z. Wang, “Dualnet: Learn complementary features for image recognition,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 502–510.
  * [112] R. G. Lopes, S. Fenu, and T. Starner, “Data-free knowledge distillation for deep neural networks,” _NIPS Workshop_ , 2017.
  * [113] K. Bhardwaj, N. Suda, and R. Marculescu, “Dream distillation: A data-independent model compression framework,” _ICML Joint Workshop_ , 2019\.
  * [114] M. Haroush, I. Hubara, E. Hoffer, and D. Soudry, “The knowledge within: Methods for data-free model compression,” _CVPR_ , 2020.
  * [115] G. K. Nayak, K. R. Mopuri, V. Shaj, R. V. Babu, and A. Chakraborty, “Zero-shot knowledge distillation in deep networks,” _ICML_ , 2019.
  * [116] H. Chen, Y. Wang, C. Xu, Z. Yang, C. Liu, B. Shi, C. Xu, C. Xu, and Q. Tian, “Data-free learning of student networks,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 3514–3522.
  * [117] G. Fang, J. Song, C. Shen, X. Wang, D. Chen, and M. Song, “Data-free adversarial distillation,” _arXiv preprint arXiv:1912.11006_ , 2019.
  * [118] J. Ye, Y. Ji, X. Wang, X. Gao, and M. Song, “Data-free knowledge amalgamation via group-stack dual-gan,” _CVPR_ , 2020.
  * [119] J. Yoo, M. Cho, T. Kim, and U. Kang, “Knowledge extraction with no observable data,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 2701–2710.
  * [120] H. Yin, P. Molchanov, Z. Li, J. M. Alvarez, A. Mallya, D. Hoiem, N. K. Jha, and J. Kautz, “Dreaming to distill: Data-free knowledge transfer via deepinversion,” _CVPR_ , 2020.
  * [121] P. Micaelli and A. J. Storkey, “Zero-shot knowledge transfer via adversarial belief matching,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 9547–9557.
  * [122] M. Kulkarni, K. Patil, and S. Karande, “Knowledge distillation using unlabeled mismatched images,” _arXiv preprint arXiv:1703.07131_ , 2017.
  * [123] L. Wang, W. Cho, and K.-J. Yoon, “Deceiving image-to-image translation networks for autonomous driving with adversarial perturbations,” _IEEE Robotics and Automation Letters (RAL)_ , 2020.
  * [124] L. Wang, Y.-S. Ho, K.-J. Yoon _et al._ , “Event-based high dynamic range image and very high frame rate video generation using conditional generative adversarial networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10 081–10 090.
  * [125] A. Mordvintsev, C. Olah, and M. Tyka, “Inceptionism: Going deeper into neural networks,” 2015.
  * [126] Q. Liu, L. Xie, H. Wang, and A. L. Yuille, “Semantic-aware knowledge preservation for zero-shot sketch-based image retrieval,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 3662–3671.
  * [127] T. Li, J. Li, Z. Liu, and C. Zhang, “Few sample knowledge distillation for efficient network compression,” _arXiv preprint arXiv:1812.01839_ , 2018\.
  * [128] A. Kimura, Z. Ghahramani, K. Takeuchi, T. Iwata, and N. Ueda, “Few-shot learning of neural networks from scratch by pseudo example optimization,” _BMVC_ , 2018.
  * [129] H. Bai, J. Wu, I. King, and M. Lyu, “Few shot network compression via cross distillation,” _AAAI_ , 2020.
  * [130] E. Snelson and Z. Ghahramani, “Sparse gaussian processes using pseudo-inputs,” in _Advances in neural information processing systems_ , 2006, pp. 1257–1264.
  * [131] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1–9.
  * [132] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” _arXiv preprint arXiv:1412.6572_ , 2014.
  * [133] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proceedings of the IEEE_ , vol. 86, no. 11, pp. 2278–2324, 1998.
  * [134] A. Krizhevsky, G. Hinton _et al._ , “Learning multiple layers of features from tiny images,” 2009.
  * [135] M. Zhu and S. Gupta, “To prune, or not to prune: exploring the efficacy of pruning for model compression,” _ICLR Workshop_ , 2018.
  * [136] Y. Aytar, C. Vondrick, and A. Torralba, “Soundnet: Learning sound representations from unlabeled video,” in _Advances in neural information processing systems_ , 2016, pp. 892–900.
  * [137] J.-C. Su and S. Maji, “Adapting models to signal degradation using distillation,” _BMVC_ , 2016.
  * [138] A. Nagrani, S. Albanie, and A. Zisserman, “Learnable pins: Cross-modal embeddings for person identity,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 71–88.
  * [139] ——, “Seeing voices and hearing faces: Cross-modal biometric matching,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8427–8436.
  * [140] J. Hoffman, S. Gupta, J. Leong, S. Guadarrama, and T. Darrell, “Cross-modal adaptation for rgb-d detection,” in _2016 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2016, pp. 5032–5039.
  * [141] T. Afouras, J. S. Chung, and A. Zisserman, “Asr is all you need: cross-modal distillation for lip reading,” _ICASSP_ , 2020.
  * [142] S. Albanie, A. Nagrani, A. Vedaldi, and A. Zisserman, “Emotion recognition in speech using cross-modal transfer in the wild,” in _Proceedings of the 26th ACM international conference on Multimedia_ , 2018, pp. 292–301.
  * [143] S. Gupta, J. Hoffman, and J. Malik, “Cross modal distillation for supervision transfer,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2827–2836.
  * [144] T. Salem, C. Greenwell, H. Blanton, and N. Jacobs, “Learning to map nearly anything,” in _IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium_. IEEE, 2019, pp. 4803–4806.
  * [145] F. M. Thoker and J. Gall, “Cross-modal knowledge distillation for action recognition,” in _2019 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2019, pp. 6–10.
  * [146] M. Zhao, T. Li, M. Abu Alsheikh, Y. Tian, H. Zhao, A. Torralba, and D. Katabi, “Through-wall human pose estimation using radio signals,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 7356–7365.
  * [147] A. Owens, J. Wu, J. H. McDermott, W. T. Freeman, and A. Torralba, “Ambient sound provides supervision for visual learning,” in _European conference on computer vision_. Springer, 2016, pp. 801–816.
  * [148] R. Arandjelovic and A. Zisserman, “Look, listen and learn,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 609–617.
  * [149] T. Do, T.-T. Do, H. Tran, E. Tjiputra, and Q. D. Tran, “Compact trilinear interaction for visual question answering,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 392–401.
  * [150] Y. Aytar, C. Vondrick, and A. Torralba, “See, hear, and read: Deep aligned representations,” _arXiv preprint arXiv:1706.00932_ , 2017.
  * [151] C. Kim, H. V. Shin, T.-H. Oh, A. Kaspar, M. Elgharib, and W. Matusik, “On learning associations of faces and voices,” in _Asian Conference on Computer Vision_. Springer, 2018, pp. 276–292.
  * [152] Q. Dou, Q. Liu, P. A. Heng, and B. Glocker, “Unpaired multi-modal segmentation via knowledge distillation,” _IEEE TMI_ , 2020.
  * [153] F. Hafner, A. Bhuiyan, J. F. Kooij, and E. Granger, “A cross-modal distillation network for person re-identification in rgb-depth,” _arXiv preprint arXiv:1810.11641_ , 2018.
  * [154] C. Gan, H. Zhao, P. Chen, D. Cox, and A. Torralba, “Self-supervised moving vehicle tracking with stereo sound,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 7053–7062.
  * [155] A. Perez, V. Sanguineti, P. Morerio, and V. Murino, “Audio-visual model distillation using acoustic images,” in _The IEEE Winter Conference on Applications of Computer Vision_ , 2020, pp. 2854–2863.
  * [156] L. Zhao, X. Peng, Y. Chen, M. Kapadia, and D. N. Metaxas, “Knowledge as priors: Cross-modal knowledge generalization for datasets without superior knowledge,” _CVPR_ , 2020.
  * [157] Z. Ke, D. Wang, Q. Yan, J. Ren, and R. W. Lau, “Dual student: Breaking the limits of the teacher in semi-supervised learning,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 6728–6736.
  * [158] L. Wang, T.-K. Kim, and K.-J. Yoon, “Eventsr: From asynchronous events to image reconstruction, restoration, and super-resolution via end-to-end adversarial learning,” _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2020.
  * [159] R. T. Mullapudi, S. Chen, K. Zhang, D. Ramanan, and K. Fatahalian, “Online model distillation for efficient video inference,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 3573–3582.
  * [160] L. Gao, X. Lan, H. Mi, D. Feng, K. Xu, and Y. Peng, “Multistructure-based collaborative online distillation,” _Entropy_ , vol. 21, no. 4, p. 357, 2019\.
  * [161] R. Lin, J. Xiao, and J. Fan, “Mod: A deep mixture model with online knowledge distillation for large scale video temporal concept localization,” _ICCV YouTube8M workshop_ , 2019.
  * [162] A. Cioppa, A. Deliege, M. Istasse, C. De Vleeschouwer, and M. Van Droogenbroeck, “Arthus: Adaptive real-time human segmentation in sports through online distillation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2019, pp. 0–0.
  * [163] H. Mobahi, M. Farajtabar, and P. L. Bartlett, “Self-distillation amplifies regularization in hilbert space,” _arXiv preprint arXiv:2002.05715_ , 2020\.
  * [164] T.-B. Xu and C.-L. Liu, “Data-distortion guided self-distillation for deep neural networks,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 5565–5572.
  * [165] S. Hahn and H. Choi, “Self-knowledge distillation in natural language processing,” _arXiv preprint arXiv:1908.01851_ , 2019.
  * [166] H. Lee, S. J. Hwang, and J. Shin, “Rethinking data augmentation: Self-supervision and self-distillation,” _ICML_ , 2020.
  * [167] E. J. Crowley, G. Gray, and A. J. Storkey, “Moonshine: Distilling with cheap convolutions,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 2888–2898.
  * [168] L. Zhang, J. Song, A. Gao, J. Chen, C. Bao, and K. Ma, “Be your own teacher: Improve the performance of convolutional neural networks via self distillation,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 3713–3722.
  * [169] S. Hou, X. Pan, C. C. Loy, Z. Wang, and D. Lin, “Learning a unified classifier incrementally via rebalancing,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 831–839.
  * [170] Y. Luan, H. Zhao, Z. Yang, and Y. Dai, “Msd: Multi-self-distillation learning via multi-classifiers within deep neural networks,” _arXiv preprint arXiv:1911.09418_ , 2019.
  * [171] X. Lan, X. Zhu, and S. Gong, “Self-referenced deep learning,” in _Asian conference on computer vision_. Springer, 2018, pp. 284–300.
  * [172] L. Yuan, F. E. Tay, G. Li, T. Wang, and J. Feng, “Revisit knowledge distillation: a teacher-free framework,” _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019.
  * [173] J. Wang, X. Wang, B. Jin, J. Yan, W. Zhang, and H. Zha, “Heterogeneous graph-based knowledge transfer for generalized zero-shot learning,” 2019.
  * [174] J. Zhou, G. Cui, Z. Zhang, C. Yang, Z. Liu, L. Wang, C. Li, and M. Sun, “Graph neural networks: A review of methods and applications,” _arXiv preprint arXiv:1812.08434_ , 2018.
  * [175] G. Chen, W. Choi, X. Yu, T. Han, and M. Chandraker, “Learning efficient object detection models with knowledge distillation,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 742–751.
  * [176] X. Guo, H. Li, S. Yi, J. Ren, and X. Wang, “Learning monocular depth by distilling cross-domain stereo networks,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 484–500.
  * [177] X. Cheng, Z. Rao, Y. Chen, and Q. Zhang, “Explaining knowledge distillation by quantifying the knowledge,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 12 925–12 935.
  * [178] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar _et al._ , “Bootstrap your own latent: A new approach to self-supervised learning,” _arXiv preprint arXiv:2006.07733_ , 2020.
  * [179] M. Mirza and S. Osindero, “Conditional generative adversarial nets,” _arXiv preprint arXiv:1411.1784_ , 2014.
  * [180] L. Chongxuan, T. Xu, J. Zhu, and B. Zhang, “Triple generative adversarial nets,” in _Advances in neural information processing systems_ , 2017, pp. 4088–4098.
  * [181] X. Wang, R. Zhang, Y. Sun, and J. Qi, “Kdgan: knowledge distillation with generative adversarial networks,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 775–786.
  * [182] Z. Xu, Y.-C. Hsu, and J. Huang, “Training shallow and thin networks for acceleration via knowledge distillation with conditional adversarial networks,” _ICLR workshop_ , 2017.
  * [183] V. Belagiannis, A. Farshad, and F. Galasso, “Adversarial network compression,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 0–0.
  * [184] R. Liu, N. Fusi, and L. Mackey, “Teacher-student compression with generative adversarial networks,” _arXiv preprint arXiv:1812.02271_ , 2018.
  * [185] P. Liu, W. Liu, H. Ma, T. Mei, and M. Seok, “Ktan: knowledge transfer adversarial network,” _International Joint Conference on Neural Networks (IJCNN)_ , 2020.
  * [186] Y. Wang, C. Xu, C. Xu, and D. Tao, “Adversarial learning of portable student networks,” in _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018.
  * [187] S. Roheda, B. S. Riggan, H. Krim, and L. Dai, “Cross-modality distillation: A case for conditional generative adversarial networks,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 2926–2930.
  * [188] Z. Xu, Y.-C. Hsu, and J. Huang, “Training student networks for acceleration with conditional adversarial networks.” in _BMVC_ , 2018, p. 61.
  * [189] B. Heo, M. Lee, S. Yun, and J. Y. Choi, “Knowledge distillation with adversarial samples supporting decision boundary,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 3771–3778.
  * [190] J. Liu, Y. Chen, and K. Liu, “Exploiting the ground-truth: An adversarial imitation based knowledge distillation approach for event detection,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 6754–6761.
  * [191] M. Goldblum, L. Fowl, S. Feizi, and T. Goldstein, “Adversarially robust distillation,” _AAAI_ , 2020.
  * [192] W. Hong and J. Yu, “Gan-knowledge distillation for one-stage object detection,” _arXiv preprint arXiv:1906.08467_ , 2019.
  * [193] M. Zhai, L. Chen, F. Tung, J. He, M. Nawhal, and G. Mori, “Lifelong gan: Continual learning for conditional image generation,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 2759–2768.
  * [194] Y. Wang, A. Gonzalez-Garcia, D. Berga, L. Herranz, F. S. Khan, and J. van de Weijer, “Minegan: effective knowledge transfer from gans to target domains with few images,” _CVPR_ , 2019.
  * [195] L. Gao, H. Mi, B. Zhu, D. Feng, Y. Li, and Y. Peng, “An adversarial feature distillation method for audio classification,” _IEEE Access_ , vol. 7, pp. 105 319–105 330, 2019.
  * [196] Z. Shen, Z. He, W. Cui, J. Yu, Y. Zheng, C. Zhu, and M. Savvides, “Adversarial-based knowledge distillation for multi-model ensemble and noisy data refinement,” _AAAI_ , 2019.
  * [197] Y. Liu, K. Chen, C. Liu, Z. Qin, Z. Luo, and J. Wang, “Structured knowledge distillation for semantic segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2604–2613.
  * [198] A. Aguinaldo, P.-Y. Chiang, A. Gain, A. Patil, K. Pearson, and S. Feizi, “Compressing gans using knowledge distillation,” _arXiv preprint arXiv:1902.00159_ , 2019.
  * [199] M. Li, J. Lin, Y. Ding, Z. Liu, J.-Y. Zhu, and S. Han, “Gan compression: Efficient architectures for interactive conditional gans,” _CVPR_ , 2020\.
  * [200] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1125–1134.
  * [201] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley, “Least squares generative adversarial networks,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 2794–2802.
  * [202] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville, “Improved training of wasserstein gans,” in _Advances in neural information processing systems_ , 2017, pp. 5767–5777.
  * [203] H. Chen, Y. Wang, H. Shu, C. Wen, C. Xu, B. Shi, C. Xu, and C. Xu, “Distilling portable generative adversarial networks for image translation,” _AAAI_ , 2020.
  * [204] J.-Y. Zhu, R. Zhang, D. Pathak, T. Darrell, A. A. Efros, O. Wang, and E. Shechtman, “Toward multimodal image-to-image translation,” in _Advances in neural information processing systems_ , 2017, pp. 465–476.
  * [205] A. Odena, C. Olah, and J. Shlens, “Conditional image synthesis with auxiliary classifier gans,” in _Proceedings of the 34th International Conference on Machine Learning-Volume 70_. JMLR. org, 2017, pp. 2642–2651.
  * [206] Z. Xu, Y.-C. Hsu, and J. Huang, “Learning loss for knowledge distillation with conditional adversarial networks,” _arXiv preprint arXiv:1709.00513_ , 2017\.
  * [207] S. Lee and B. C. Song, “Graph-based knowledge distillation by multi-head self-attention network,” _BMVC_ , 2019.
  * [208] Y. Liu, J. Cao, B. Li, C. Yuan, W. Hu, Y. Li, and Y. Duan, “Knowledge distillation via instance relationship graph,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 7096–7104.
  * [209] H. Cai, V. W. Zheng, and K. C.-C. Chang, “A comprehensive survey of graph embedding: Problems, techniques, and applications,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 30, no. 9, pp. 1616–1637, 2018.
  * [210] W. L. Hamilton, R. Ying, and J. Leskovec, “Representation learning on graphs: Methods and applications,” _IEEE Data Engineering Bulletin_ , 2017.
  * [211] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini, “The graph neural network model,” _IEEE Transactions on Neural Networks_ , vol. 20, no. 1, pp. 61–80, 2008.
  * [212] C. Lassance, M. Bontonou, G. B. Hacene, V. Gripon, J. Tang, and A. Ortega, “Deep geometric knowledge distillation with graphs,” _ICASSP_ , 2020.
  * [213] S. Minami, T. Hirakawa, T. Yamashita, and H. Fujiyoshi, “Knowledge transfer graph for deep collaborative learning,” 2019.
  * [214] H. Yao, C. Zhang, Y. Wei, M. Jiang, S. Wang, J. Huang, N. V. Chawla, and Z. Li, “Graph few-shot learning via knowledge transfer,” _arXiv preprint arXiv:1910.03053_ , 2019.
  * [215] J. Ma and Q. Mei, “Graph representation learning via multi-task knowledge distillation,” _NeurIPS GRL Workshop_ , 2019.
  * [216] J. Liu, M. Shah, B. Kuipers, and S. Savarese, “Cross-view action recognition via view knowledge transfer,” in _CVPR 2011_. IEEE, 2011, pp. 3209–3216.
  * [217] Z. Luo, J.-T. Hsieh, L. Jiang, J. Carlos Niebles, and L. Fei-Fei, “Graph distillation for action detection with privileged modalities,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 166–183.
  * [218] Y. Yang, J. Qiu, M. Song, D. Tao, and X. Wang, “Distillating knowledge from graph convolutional networks,” _arXiv preprint arXiv:2003.10477_ , 2020.
  * [219] A. Ortega, P. Frossard, J. Kovačević, J. M. Moura, and P. Vandergheynst, “Graph signal processing: Overview, challenges, and applications,” _Proceedings of the IEEE_ , vol. 106, no. 5, pp. 808–828, 2018.
  * [220] Z. Ying, J. You, C. Morris, X. Ren, W. Hamilton, and J. Leskovec, “Hierarchical graph representation learning with differentiable pooling,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 4800–4810.
  * [221] W. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 1024–1034.
  * [222] C. Li, X. Guo, and Q. Mei, “Deepgraph: Graph structure predicts network growth,” _arXiv preprint arXiv:1610.06251_ , 2016.
  * [223] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” _ICLR_ , 2017.
  * [224] F. Sung, Y. Yang, L. Zhang, T. Xiang, P. H. Torr, and T. M. Hospedales, “Learning to compare: Relation network for few-shot learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 1199–1208.
  * [225] M. A. Haidar and M. Rezagholizadeh, “Textkd-gan: text generation using knowledge distillation and generative adversarial networks,” in _Canadian Conference on Artificial Intelligence_. Springer, 2019, pp. 107–118.
  * [226] S. Laine and T. Aila, “Temporal ensembling for semi-supervised learning,” _arXiv preprint arXiv:1610.02242_ , 2016.
  * [227] Y. Luo, J. Zhu, M. Li, Y. Ren, and B. Zhang, “Smooth neighbors on teacher graphs for semi-supervised learning,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8896–8905.
  * [228] S. Zhang, J. Li, and B. Zhang, “Pairwise teacher-student network for semi-supervised hashing,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2019, pp. 0–0.
  * [229] G. French, M. Mackiewicz, and M. Fisher, “Self-ensembling for visual domain adaptation,” _arXiv preprint arXiv:1706.05208_ , 2017.
  * [230] J. Choi, T. Kim, and C. Kim, “Self-ensembling with gan-based data augmentation for domain adaptation in semantic segmentation,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 6830–6840.
  * [231] Q. Cai, Y. Pan, C.-W. Ngo, X. Tian, L. Duan, and T. Yao, “Exploring object relation in mean teacher for cross-domain detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 11 457–11 466.
  * [232] Y. Xu, B. Du, L. Zhang, Q. Zhang, G. Wang, and L. Zhang, “Self-ensembling attention networks: Addressing domain shift for semantic segmentation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 5581–5588.
  * [233] Z. Hailat and X.-W. Chen, “Teacher/student deep semi-supervised learning for training with noisy labels,” in _2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA)_. IEEE, 2018, pp. 907–912.
  * [234] M. Noroozi, A. Vinjimoor, P. Favaro, and H. Pirsiavash, “Boosting self-supervised learning via knowledge transfer,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 9359–9367.
  * [235] G. Xu, Z. Liu, X. Li, and C. C. Loy, “Knowledge distillation meets self-supervision,” _European Conf. Comput. Vis.(ECCV)_ , 2020.
  * [236] S. Flennerhag, P. G. Moreno, N. D. Lawrence, and A. Damianou, “Transferring knowledge across learning processes,” _arXiv preprint arXiv:1812.01054_ , 2018.
  * [237] H. Jin, S. Zhang, X. Zhu, Y. Tang, Z. Lei, and S. Z. Li, “Learning lightweight face detector with knowledge distillation,” in _2019 International Conference on Biometrics (ICB)_. IEEE, 2019, pp. 1–7.
  * [238] Y. Wu, Y. Chen, L. Wang, Y. Ye, Z. Liu, Y. Guo, and Y. Fu, “Large scale incremental learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 374–382.
  * [239] U. Michieli and P. Zanuttigh, “Knowledge distillation for incremental learning in semantic segmentation,” _arXiv preprint arXiv:1911.03462_ , 2019.
  * [240] F. M. Castro, M. J. Marín-Jiménez, N. Guil, C. Schmid, and K. Alahari, “End-to-end incremental learning,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 233–248.
  * [241] K. Shmelkov, C. Schmid, and K. Alahari, “Incremental learning of object detectors without catastrophic forgetting,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 3400–3409.
  * [242] H. B. Ammar, E. Eaton, J. M. Luna, and P. Ruvolo, “Autonomous cross-domain knowledge transfer in lifelong policy gradient reinforcement learning,” in _Twenty-Fourth International Joint Conference on Artificial Intelligence_ , 2015.
  * [243] A. Ashok, N. Rhinehart, F. Beainy, and K. M. Kitani, “N2n learning: Network to network compression via policy gradient reinforcement learning,” _arXiv preprint arXiv:1709.06030_ , 2017.
  * [244] Y. Burda, H. Edwards, A. Storkey, and O. Klimov, “Exploration by random network distillation,” _arXiv preprint arXiv:1810.12894_ , 2018.
  * [245] Z.-W. Hong, P. Nagarajan, and G. Maeda, “Periodic intra-ensemble knowledge distillation for reinforcement learning,” _arXiv preprint arXiv:2002.00149_ , 2020.
  * [246] Z. Xue, S. Luo, C. Wu, P. Zhou, K. Bian, and W. Du, “Transfer heterogeneous knowledge among peer-to-peer teammates: A model distillation approach,” _arXiv preprint arXiv:2002.02202_ , 2020.
  * [247] K. Lin, S. Wang, and J. Zhou, “Collaborative deep reinforcement learning,” _arXiv preprint arXiv:1702.05796_ , 2017.
  * [248] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3431–3440.
  * [249] Y. Shan, “Distilling pixel-wise feature similarities for semantic segmentation,” _arXiv preprint arXiv:1910.14226_ , 2019.
  * [250] T. He, C. Shen, Z. Tian, D. Gong, C. Sun, and Y. Yan, “Knowledge adaptation for efficient semantic segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 578–587.
  * [251] Y. Chen, W. Li, and L. Van Gool, “Road: Reality oriented adaptation for semantic segmentation of urban scenes,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 7892–7901.
  * [252] J. Xie, B. Shuai, J.-F. Hu, J. Lin, and W.-S. Zheng, “Improving fast segmentation with teacher-student learning,” _arXiv preprint arXiv:1810.08476_ , 2018.
  * [253] Y. Hao, Y. Fu, Y.-G. Jiang, and Q. Tian, “An end-to-end architecture for class-incremental object detection with knowledge distillation,” in _2019 IEEE International Conference on Multimedia and Expo (ICME)_. IEEE, 2019, pp. 1–6.
  * [254] L. Chen, C. Yu, and L. Chen, “A new knowledge distillation for incremental object detection,” in _2019 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2019, pp. 1–7.
  * [255] R. Chen, H. Ai, C. Shang, L. Chen, and Z. Zhuang, “Learning lightweight pedestrian detector with hierarchical knowledge distillation,” in _2019 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2019, pp. 1645–1649.
  * [256] S. Tang, L. Feng, W. Shao, Z. Kuang, W. Zhang, and Y. Chen, “Learning efficient detector with semi-supervised adaptive distillation,” _arXiv preprint arXiv:1901.00366_ , 2019.
  * [257] M. R. U. Saputra, P. P. de Gusmao, Y. Almalioglu, A. Markham, and N. Trigoni, “Distilling knowledge from a deep pose regressor network,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 263–272.
  * [258] X. Jin, C. Lan, W. Zeng, and Z. Chen, “Uncertainty-aware multi-shot knowledge distillation for image-based object re-identification,” _arXiv preprint arXiv:2001.05197_ , 2020.
  * [259] Y. Lee, N. Ahn, J. H. Heo, S. Y. Jo, and S.-J. Kang, “Teaching where to see: Knowledge distillation-based attentive information transfer in vehicle maker classification,” _IEEE Access_ , vol. 7, pp. 86 412–86 420, 2019.
  * [260] J. Xu, Y. Nie, P. Wang, and A. M. López, “Training a binary weight object detector by knowledge transfer for autonomous driving,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 2379–2384.
  * [261] S. S. Kruthiventi, P. Sahay, and R. Biswal, “Low-light pedestrian detection from rgb images using multi-modal knowledge distillation,” in _2017 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2017, pp. 4207–4211.
  * [262] S. Ge, S. Zhao, C. Li, and J. Li, “Low-resolution face recognition in the wild via selective knowledge distillation,” _IEEE Transactions on Image Processing_ , vol. 28, no. 4, pp. 2051–2062, 2018.
  * [263] P. Luo, Z. Zhu, Z. Liu, X. Wang, and X. Tang, “Face model compression by distilling knowledge from neurons,” in _Thirtieth AAAI conference on artificial intelligence_ , 2016.
  * [264] Y. Feng, H. Wang, R. Hu, and D. T. Yi, “Triplet distillation for deep face recognition,” _ICML ODML-CDNNR Workshop_ , 2019.
  * [265] H. Felix, W. M. Rodrigues, D. Macêdo, F. Simões, A. L. Oliveira, V. Teichrieb, and C. Zanchettin, “Squeezed deep 6dof object detection using knowledge distillation,” _IJCNN_ , 2020.
  * [266] Y. Liu, X. Dong, W. Wang, and J. Shen, “Teacher-students knowledge distillation for siamese trackers,” _arXiv preprint arXiv:1907.10586_ , 2019\.
  * [267] J. Karlekar, J. Feng, Z. S. Wong, and S. Pranata, “Deep face recognition model compression via knowledge transfer and distillation,” _arXiv preprint arXiv:1906.00619_ , 2019.
  * [268] X. Dong and Y. Yang, “Teacher supervises students how to learn from partially labeled images for facial landmark detection,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 783–792.
  * [269] A. Zhao, T. He, Y. Liang, H. Huang, G. V. d. Broeck, and S. Soatto, “Lates: Latent space distillation for teacher-student driving policy learning,” _arXiv preprint arXiv:1912.02973_ , 2019.
  * [270] F. Zhang, X. Zhu, and M. Ye, “Fast human pose estimation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3517–3526.
  * [271] A. Martínez-González, M. Villamizar, O. Canévet, and J.-M. Odobez, “Efficient convolutional neural networks for depth-based multi-person pose estimation,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2019.
  * [272] D.-H. Hwang, S. Kim, N. Monet, H. Koike, and S. Bae, “Lightweight 3d human pose estimation network training using teacher-student learning,” _arXiv preprint arXiv:2001.05097_ , 2020.
  * [273] X. Xu, Q. Zou, X. Lin, Y. Huang, and Y. Tian, “Integral knowledge distillation for multi-person pose estimation,” _IEEE Signal Processing Letters_ , 2020\.
  * [274] C. Wang, C. Kong, and S. Lucey, “Distill knowledge from nrsfm for weakly supervised 3d pose learning,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 743–752.
  * [275] X. Nie, Y. Li, L. Luo, N. Zhang, and J. Feng, “Dynamic kernel distillation for efficient pose estimation in videos,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 6942–6950.
  * [276] S. Ao, X. Li, and C. X. Ling, “Fast generalized distillation for semi-supervised domain adaptation,” in _Thirty-First AAAI Conference on Artificial Intelligence_ , 2017.
  * [277] D. Lopez-Paz, L. Bottou, B. Schölkopf, and V. Vapnik, “Unifying distillation and privileged information,” _International Conference on Learning Representations_ , 2016.
  * [278] J. Hoffman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A. A. Efros, and T. Darrell, “Cycada: Cycle-consistent adversarial domain adaptation,” _CVPR_ , 2018.
  * [279] Z. Meng, J. Li, Y. Gaur, and Y. Gong, “Domain adaptation via teacher-student learning for end-to-end speech recognition,” in _2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 2019, pp. 268–275.
  * [280] Y.-C. Chen, Y.-Y. Lin, M.-H. Yang, and J.-B. Huang, “Crdoco: Pixel-level domain transfer with cross-domain consistency,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 1791–1800.
  * [281] Y.-H. Tsai, W.-C. Hung, S. Schulter, K. Sohn, M.-H. Yang, and M. Chandraker, “Learning to adapt structured output space for semantic segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 7472–7481.
  * [282] Z. Deng, Y. Luo, and J. Zhu, “Cluster alignment with a teacher for unsupervised domain adaptation,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 9944–9953.
  * [283] J. Cho, D. Min, Y. Kim, and K. Sohn, “A large rgb-d dataset for semi-supervised monocular depth estimation,” _IEEE Transactions on Image Processing_ , 2019.
  * [284] A. Pilzer, S. Lathuiliere, N. Sebe, and E. Ricci, “Refine and distill: Exploiting cycle-inconsistency and knowledge distillation for unsupervised monocular depth estimation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 9768–9777.
  * [285] F. Tosi, F. Aleotti, M. Poggi, and S. Mattoccia, “Learning monocular depth estimation infusing traditional stereo knowledge,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 9799–9809.
  * [286] P. Liu, I. King, M. R. Lyu, and J. Xu, “Ddflow: Learning optical flow with unlabeled data distillation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 8770–8777.
  * [287] F. Aleotti, M. Poggi, F. Tosi, and S. Mattoccia, “Learning end-to-end scene flow by distilling single tasks knowledge,” _arXiv preprint arXiv:1911.10090_ , 2019.
  * [288] F. Tosi, F. Aleotti, P. Z. Ramirez, M. Poggi, S. Salti, L. Di Stefano, and S. Mattoccia, “Distilled semantics for comprehensive scene understanding from videos,” _CVPR_ , 2020.
  * [289] H. Wang, Y. Li, Y. Wang, H. Hu, and M.-H. Yang, “Collaborative distillation for ultra-resolution universal style transfer,” _arXiv_ , pp. arXiv–2003, 2020.
  * [290] S. Bhardwaj, M. Srinivasan, and M. M. Khapra, “Efficient video classification using fewer frames,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 354–363.
  * [291] X. Wang, J.-F. Hu, J.-H. Lai, J. Zhang, and W.-S. Zheng, “Progressive teacher-student learning for early action prediction,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3556–3565.
  * [292] C. Gan, T. Yao, K. Yang, Y. Yang, and T. Mei, “You lead, we exceed: Labor-free video concept learning by jointly exploiting web videos and images,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 923–932.
  * [293] C. Gan, B. Gong, K. Liu, H. Su, and L. J. Guibas, “Geometry guided convolutional neural networks for self-supervised video representation learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 5589–5597.
  * [294] K. Fu, J. Li, Y. Song, Y. Zhang, S. Ge, and Y. Tian, “Ultrafast video attention prediction with coupled knowledge distillation,” _arXiv preprint arXiv:1904.04449_ , 2019.
  * [295] M. Farhadi and Y. Yang, “Tkd: Temporal knowledge distillation for active perception,” _WACV_ , 2020.
  * [296] Z. Zhang, Y. Shi, C. Yuan, B. Li, P. Wang, W. Hu, and Z. Zha, “Object relational graph with teacher-recommended learning for video captioning,” _CVPR_ , 2020.
  * [297] B. Pan, H. Cai, D.-A. Huang, K.-H. Lee, A. Gaidon, E. Adeli, and J. C. Niebles, “Spatio-temporal graph for video captioning with knowledge distillation,” _CVPR_ , 2020.
  * [298] Y. Lee, J. Jeong, J. Yun, W. Cho, and K.-J. Yoon, “Spherephd: Applying cnns on a spherical polyhedron representation of 360deg images,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 9181–9189.
  * [299] P. Bashivan, M. Tensen, and J. J. DiCarlo, “Teacher guided architecture search,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 5320–5329.
  * [300] M. Derezinski and M. K. Warmuth, “The limits of squared euclidean distance regularization,” in _Advances in Neural Information Processing Systems_ , 2014, pp. 2807–2815.
  * [301] S. W. Park and J. Kwon, “Sphere generative adversarial network based on geometric moment matching,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4292–4301.
  * [302] K. Li, Y. Zhang, K. Li, Y. Li, and Y. Fu, “Attention bridging network for knowledge transfer,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 5198–5207.
  * [303] J. Schlemper, O. Oktay, M. Schaap, M. Heinrich, B. Kainz, B. Glocker, and D. Rueckert, “Attention gated networks: Learning to leverage salient regions in medical images,” _Medical image analysis_ , vol. 53, pp. 197–207, 2019\.
  * [304] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [305] R.-W. Zhao, J. Li, Y. Chen, J.-M. Liu, Y.-G. Jiang, and X. Xue, “Regional gating neural networks for multi-label image classification.” in _BMVC_ , 2016, pp. 1–12.
  * [306] M. Mostafavi, L. Wang, and K.-J. Yoon, “Learning to reconstruct hdr images from events, with applications to depth and flow prediction,” _International Journal of Computer Vision_ , pp. 1–21.

|  Lin Wang is a Ph.D. student in Visual Intelligence Lab., Dept. of
Mechanical Engineering, Korea Advanced Institute of Science and Technology
(KAIST). His research interests include deep learning (especially adversarial
learning, transfer learning, semi-/self-supervised learning), event camera-
based vision, low-level vision (image super-solution and deblurring, etc.) and
computer vision for VR/AR.
---|---
|  Kuk-Jin Yoon received the B.S., M.S., and Ph.D. degrees in electrical
engineering and computer science from the Korea Advanced Institute of Science
and Technology in 1998, 2000, and 2006, respectively. He is now an Associate
Professor at the Department of Mechanical Engineering, Korea Advanced
Institute of Science and Technology (KAIST), South Korea, leading the Visual
Intelligence Laboratory. Before joining KAIST, he was a Post-Doctoral Fellow
in the PERCEPTION Team, INRIA, Grenoble, France, from 2006 to 2008, and was an
Assistant/Associate Professor at the School of Electrical Engineering and
Computer Science, Gwangju Institute of Science and Technology, South Korea,
from 2008 to 2018. His research interests include various topics in computer
vision such as multi-view stereo, visual object tracking, SLAM and structure-
from-motion, 360 camera and event-camera-base vision, sensor fusion.
---|---