On Cross-Layer Alignment for Model Fusion of Heterogeneous Neural Networks

Dang Nguyen⋄ |  Trang Nguyen⋆ |  Khai Nguyen† |  Dinh Phung‡ |  Hung Bui⋄ |  Nhat Ho†
---|---|---|---|---|---
VinAI Research⋄ | Hanoi University of Science and Technology⋆ | University of Texas, Austin†
---|---|---
| Monash University‡ |

###### Abstract

Layer-wise model fusion via optimal transport, named OTFusion, applies soft
neuron association for unifying different pre-trained networks to save
computational resources. While enjoying its success, OTFusion requires the
input networks to have the same number of layers. To address this issue, we
propose a novel model fusion framework, named _CLAFusion_ , to fuse neural
networks with a different number of layers, which we refer to as heterogeneous
neural networks, via cross-layer alignment. The cross-layer alignment problem,
which is an unbalanced assignment problem, can be solved efficiently using
dynamic programming. Based on the cross-layer alignment, our framework
balances the number of layers of neural networks before applying layer-wise
model fusion. Our experiments indicate that CLAFusion, with an extra
finetuning process, improves the accuracy of residual networks on the CIFAR10,
CIFAR100, and Tiny-ImageNet datasets. Furthermore, we explore its practical
usage for model compression and knowledge distillation when applying to the
teacher-student setting.

##  1 Introduction

The ubiquitous presence of deep neural networks raises the question, “Can we
combine the knowledge of two and more deep neural networks to improve
performance?" As one of the earliest successful methods, ensemble learning
aggregates the output over a collection of trained models, which is referred
to as an ensemble, to improve the generalization ability [11, 41]. However, it
is expensive in terms of computational resources because it requires storing
and running many trained neural networks during inference. To overcome the
previous hardness, we need to find a single neural network that can inherit
the knowledge from multiple pre-trained neural networks and have a small size
at the same time. To our best knowledge, there are two popular approaches for
this purpose: knowledge distillation and model fusion.

The first approach, _knowledge distillation_ , is a machine learning technique
that transfers knowledge from large networks (teachers) into a smaller one
(student) [13]. The smaller network is trained by the task-specific loss and
additional distillation losses that encourage the student network to mimic the
prediction of the teachers. Knowledge distillation is well-known as an
efficient model compression and acceleration technique [10]. The distillation
process, however, is computationally expensive because it runs forward
inference for both teacher and student networks at each training epoch. In
addition, simply distilling into a randomly initialized student network, in
most cases, does not yield high performance, thus a good initialization of the
student network is needed [32].

The second line of work is called _model fusion_ which is the problem of
merging a collection of pre-trained networks into a unified network [36, 28].
The simplest technique in model fusion is vanilla averaging [33, 29], which
computes the weighted average of pre-trained network parameters without the
need for retraining. Their methods fuse neural networks with the same
architecture without considering the permutation invariance nature of neural
networks. To mitigate this problem, the idea of solving a neuron alignment
problem before applying weight averaging is utilized [1, 23]. Recently, there
are two concurrent works that benefit from the idea of neuron alignment. FedMA
[36] formulated and solved the assignment problem to find a permutation matrix
that undoes the permutation of weight matrices. While OTFusion [28] viewed the
problem through the lens of optimal transport [24, 16] and used the transport
map to align the weight matrices. Though both methods can fuse multiple neural
networks, their applications are limited to networks with the same number of
layers.

Contributions. In this paper, we propose a model fusion framework for fusing
heterogeneous (_unequal width and unequal depth_) neural networks, which is
named as Cross-Layer Alignment Fusion (CLAFusion). CLAFusion consists of three
parts: cross-layer alignment, layer balancing method, and layer-wise model
fusion method. In summary, our main contributions are two-fold:

1\. We formulate the cross-layer alignment as an assignment problem using
layer representation and layer similarity, then propose a dynamic programming-
based algorithm to solve it. Next, we present two natural and fast methods to
balance the number of layers between two networks. Furthermore, we discuss the
application of our framework for the cases of more than two networks.

2\. Our experiments demonstrate the efficiency of CLAFusion on different
setups. The fused model from CLAFusion serves as an efficient initialization
when training residual networks. In addition, the components of CLAFusion can
be successfully applied to the heterogeneous model transfer task. Moreover,
CLAFusion shows potential applications for model compression and knowledge
distillation in the teacher-student setting.

Organization. The rest of the paper is organized as follows. After reviewing
some background in Section 2, we introduce CLAFusion in Section 3. We study
their performance in Section 4 on the image classification task with various
settings and followed by discussions in Section 5. Finally, experimental
settings and additional experiments are deferred to the supplementary
material.

##  2 Background

In this section, we first recall the layer representation and layer similarity
that have been widely used in applications. Then, we review available model
fusion methods and their limitations. Finally, we discuss OTFusion and the
challenge of fusing heterogeneous neural networks.

###  2.1 Layer Representation and Layer Similarity

A common layer representation is the matrix of activations [18]. Activation
matrix, also known as activation map, can be used as a type of knowledge to
guide the training in knowledge distillation [10]. Another choice is the
weight matrix of the trained neural networks, which has been shown to perform
well on intra-network comparison tasks [25].

The neural network exhibits the same output when its neurons in each layer are
permuted, this is so-called the permutation invariance nature of neural
network parameters. Due to the permutation invariance property, a meaningful
layer similarity index should be invariant to orthogonal transformations [18].
Their proposed Centered Kernel Alignment (CKA) effectively identifies the
relationship between layers of different architectures.

###  2.2 Model fusion

Despite showing good empirical results, vanilla averaging [33, 29] only works
in the case when the weights of individual networks are relatively close in
the weight space. As an effective remedy, several works on model fusion
considered performing permutation of neurons in hidden layers before applying
vanilla averaging. FBA-Wagging [1], Elastic Weight Consolidation [22], and
FedMA [36] formulate the neuron association problem as an assignment problem
and align the neurons. Nevertheless, those variants are not generalized to
heterogeneous neural networks.

There is limited literature devoted to the discussion of fusing heterogeneous
neural networks. NeuralMerger [5] is one of the few attempts to deal with
heterogeneous neural networks. However, there are two key differences in
NeuralMerger from our CLAFusion. First, their cross-layer alignment is hand-
crafted and dedicated to specific architectures. Secondly, they decompose
weights into lower dimensions and use vector quantization to merge the
weights. On the other hand, we introduce a systematic way to solve the cross-
layer alignment problem, and combining weights is done using the layer-wise
model fusion method.

###  2.3 Model fusion via Optimal Transport

Recently, Singh et al. [28] proposed OTFusion, which solves the neuron
association problem based on free-support barycenters [7]. OTFusion leads to a
noticeable improvement over vanilla averaging when all individual networks
have the same architecture. One advantage of OTFusion over other vanilla
averaging-based model fusion methods is that it can fuse networks with
different widths (i.e., number of neurons) thanks to the nature of optimal
transport.

Issues of OTFusion. OTFusion, however, is still a layer-wise approach that
cannot be directly employed for heterogeneous neural networks. We need to find
two corresponding layers, matching their neurons before averaging weight
matrices. There are two challenges to this scheme. Firstly, the one-to-one
mapping between layers is not available in advance. A naive one-to-one mapping
of layers with the same index as the layer-wise approach may cause a mismatch
when the numbers of layers are different. For example, we analyze two VGG
configurations [27, Table 1] with different depths: VGG11 and VGG13. The
second convolution layer of VGG11 has $128$ channels and an input image of
size $112\times 112$. While that of VGG13 has $64$ channels and an input image
of size $224\times 224$. Secondly, assume that we have successfully found a
one-to-one mapping, there still necessitates a special treatment for remaining
layers that has no counterparts. Simply removing those layers reduces the
performance of the deeper network, this may cumulatively degrade the fused
model.

---
Figure 1:  CLAFusion strategy:  In the first step, the cross-layer alignment
problem is solved for two pre-trained models. Two corresponding layers are
surrounded by two rounded rectangles of the same color. Based on the optimal
mapping obtained in the previous step, CLAFusion balances the number of layers
(adding layers in this figure). The red circles and lines indicate the newly
added neurons and weights (Zero weights are omitted). Finally, CLAFusion
applies a layer-wise model fusion method to produce the final output.

##  3 Cross-layer Alignment Model Fusion

To overcome the challenges of OTFusion, in this section, we propose Cross-
layer Alignment Model Fusion (in short, _CLAFusion_) framework for fusing
heterogeneous neural networks. In this section, we first provide an overview
of our framework, followed by the formal definition and an efficient solution
to the CLA problem. Next, we introduce two layer balancing methods that will
be in our experiments. Finally, we describe a simple yet effective extension
of our framework to the case of multiple neural networks.

Notation. We define $\llbracket{n}\rrbracket$ as a set of integers
$\\{1,\ldots,n\\}$ and $\mathbb{I}_{n}$ as an identity matrix of size $n\times
n$. We use A, B to denote the individual models and $\mathcal{F}$ to denote
the fused model. $l$ indicates the layer index. Layer $l$ of model A has
$p_{A}^{(l)}$ neurons and an activation function $f_{A}^{(l)}$. The pre-
activation and activation vector at layer $l$ of model A are denoted as
$\bm{z}_{A}^{(l)},\bm{x}_{A}^{(l)}\in\mathbb{R}^{p_{A}^{(l)}}$, respectively.
The weight matrix between layers $l$ and $l-1$ of model A is
$\bm{W}_{A}^{(l,l-1)}$. The following equations hold between two consecutive
layers of model A (we omit the bias terms here).

$\displaystyle\bm{x}^{(l)}=f_{A}^{(l)}(\bm{z}_{A}^{(l)})=f_{A}^{(l)}(\bm{W}_{A}^{(l,l-1)}\bm{x}^{(l-1)}).$
(1)

We stack the pre-activation vector over $t$ samples to form a $t-$row pre-
activation matrix. Let $\bm{Z}_{A}^{(l)}\in\mathbb{R}^{t\times p_{A}^{(l)}}$
denote a matrix of pre-activations at layer $l$ of model A for $t$ samples,
and $\bm{Z}_{B}^{(l)}\in\mathbb{R}^{t\times p_{B}^{(l)}}$ denote a matrix of
pre-activations at layer $l$ of model B for the same $t$ samples. The
activation matrices $\bm{X}_{A}^{(l)},\bm{X}_{B}^{(l)}$ are obtained by
applying the corresponding activation functions element-wise to the pre-
activation matrices.

Problem setting. Hereafter, we consider fusing two feed-forward networks A and
B of the _same architecture family_. Two networks have the same input and
output dimension but a different number of hidden layers. In each network, the
hidden layer has an activation function of ReLU, which is the general setting
in modern architectures. Additional work is required to handle bias terms and
the batch normalization layer properly so we leave them for future work. Let
$m$ and $n$ be the number of hidden layers of models A and B, respectively.
Taking the input and output layers into account, the numbers of layers are
$m+2$ and $n+2$, respectively. Without loss of generality, assume that $m\geq
n$. The layer index of model A and B are $\\{0,1,\ldots,{m+1}\\}$ and
$\\{0,1,\ldots,{n+1}\\}$, respectively.

General strategy. Our framework has three components, which are illustrated in
Figure 1. The first part is a cross-layer alignment which is a one-to-one
mapping from hidden layers of model B to hidden layers of model A. The second
part is a layer balancing method that equalizes the number of layers of two
models based on the cross-layer alignment. The last part is a layer-wise model
fusion method that is applied to same-sized models. The third part of our
framework adopts any model fusion methods that can fuse two neural networks
with the same number of layers.

---
Figure 2:  Cross-layer alignment example:  Two pre-trained neural networks are
given in (a). Model A has 4 hidden layers of size 5, 4, 3, 1 while model B has
3 hidden layers of size 3, 2, 1. (b) shows the cost matrix (squared
difference) between layer representations (number of neurons) for hidden
layers of two networks. Three color cells represent the solution of the CLA
problem. Note that the upper-left and lower-right cells are automatically
chosen by the constraints on the first and last hidden layers. (c) visualizes
the optimal mapping. Two rounded rectangles of the same colors represent two
corresponding layers in the optimal CLA.

###  3.1 Cross-layer alignment

Cross-layer alignment (CLA) is a nontrivial problem that also arises in the
context of the feature-based knowledge distillation approach [38, 31, 3]. CLA
requires finding a one-to-one function that matches similar layers between two
models. Despite considering CLA, prior works are based on the hand-crafted
layer association, which causes a lack of generalization. To address this
matter, we introduce an efficient method for solving the CLA problem. Because
the input and output layers of the two models are identical in both dimension
and functionality, we only align their hidden layers. In addition, the first
and last hidden layers usually play a critical role in the model performance
[40]. Therefore, we directly match the first and last hidden layers of the two
models. In case $n=1$, we prioritize the first layer over the last layer. The
importance of this constraint in CLA is studied in Appendix C. Furthermore,
two networks are feed-forward, so the mapping is necessary to keep the
sequential order of layers. All in all, we formulate the CLA problem as
follows.

######  Definition 1.

Assume that we have two layer representations for hidden layers of two models
as, $\bm{L}_{A}=\\{L_{A}^{(1)},\ldots,L_{A}^{(m)}\\}$ and
$\bm{L}_{B}=\\{L_{B}^{(1)},\ldots,L_{B}^{(n)}\\}$. Let
$\bm{C}\in\mathbb{R}^{m\times n}$ denote the cost matrix between them, i.e.,
$\bm{C}_{i,j}=d(L_{A}^{(i)},L_{B}^{(j)}),i\in\llbracket{m}\rrbracket,j\in\llbracket{n}\rrbracket$
where $d$ is a layer dissimilarity. The _optimal CLA_ is a strictly increasing
mapping $a:\llbracket{n}\rrbracket\mapsto\llbracket{m}\rrbracket$ satisfying
$a(1)=1,a(n)=m$ and minimizing $S(n,m)=\sum_{i=1}^{n}\bm{C}_{a(i),i}$.

Layer representation. Empirically, we used the pre-activation matrix, which is
also used in the activation-based alignment strategy of OTFusion [28, Section
4], as the layer representation. To measure the discrepancy between two pre-
activation matrices, we choose the dissimilarity index based on Linear CKA
[18]. Specifically, given two pre-activation matrices
${\bm{X}}\in\mathbb{R}^{t\times p_{A}}$ and ${\bm{Y}}\in\mathbb{R}^{t\times
p_{B}}$ for the same $t$ samples of two models A and B, their centered
versions are defined as $\bm{\hat{X}}={\bm{X}}{\bm{H}}_{p_{A}}$ and
$\bm{\hat{Y}}={\bm{Y}}{\bm{H}}_{p_{B}}$ where
${\bm{H}}_{p}=\mathbb{I}_{p}-\frac{1}{n}\mathbbm{1}_{p}\mathbbm{1}_{p}^{T}$.
Then, the similarity index Linear CKA between ${\bm{X}}$ and ${\bm{Y}}$ has
the following form

$\frac{\left\lVert\bm{\hat{Y}}^{T}\bm{\hat{X}}\right\rVert_{F}^{2}}{\left\lVert\bm{\hat{X}}^{T}\bm{\hat{X}}\right\rVert_{F}\left\lVert\bm{\hat{Y}}^{T}\bm{\hat{Y}}\right\rVert_{F}}$
(2)

where $\left\lVert\cdot\right\rVert_{F}$ is the Frobenius norm. Then, the
layer dissimilarity $d$ in Definition 1 is empirically set to

$1-\frac{\left\lVert\bm{\hat{Y}}^{T}\bm{\hat{X}}\right\rVert_{F}^{2}}{\left\lVert\bm{\hat{X}}^{T}\bm{\hat{X}}\right\rVert_{F}\left\lVert\bm{\hat{Y}}^{T}\bm{\hat{Y}}\right\rVert_{F}}.$
(3)

It is worth noting that besides the above option, our framework can work with
any appropriate alternatives, such as the number of neurons for layer
representation and Euclidean distance for dissimilarity.

Toy example. Figure 2 presents a CLA problem, in which two models in Figure 1
are utilized again. Here, the layer representation is the number of neurons
with the squared difference as the cost function. The optimal mapping $a$
obtained from solving the CLA problem is $a(1)=1,a(2)=3,a(3)=4$.

Discussion. The CLA problem in Definition 1 is a special case of the (linear)
unbalanced assignment problem [26]. The condition of strictly increasing
ensures the sequential order of layers when matching. If we have two
increasing layer representations in 1-D, the problem can be interpreted as the
Partial Optimal Transport in 1-D problem with uniform weights. It can be
solved efficiently in $\mathcal{O}(mn)$ using the proposed method in Bonneel
et al. [2]. However, the increasing constraint on layer representations is
quite strict and layer representations might not be in 1-D in general. On the
other hand, the unbalanced assignment problem can be solved using the
generalization of the Hungarian algorithm [20]. The time complexity of the
Hungarian algorithm in our CLA problem is $\mathcal{O}(mn^{2})$. The
importance of CLA in our framework is demonstrated in Appendix C where we
compare the performance of the fused model obtained using the optimal mapping
with those from other mappings.

Cross-layer alignment algorithm. We propose an efficient algorithm based on
dynamic programming to solve the CLA problem. Algorithm 1 details the pseudo-
code for our algorithm. Given the cost matrix between layer representations of
two networks, we want to find the optimal alignment from the shallower network
(model B) to the deeper network (model A). The optimal alignment is a strictly
increasing mapping that minimizes the total assignment cost. We denote
$S(i,j)$ as the optimal assignment cost from the first $i$ layers of model B
to the first $j$ layers of model A. We initialize $n\times m$ entities in
matrix $S$ to be $0$. Due to the strictly increasing property, when $i=j$, the
optimal alignment is trivial. Assume that we have already found all optimal
assignment costs before $S(i,j)(i<=j)$. There are two situations. If layer $j$
of model A is mapped to one layer in model B. Because the optimal alignment is
an increasing mapping, layer $j$ of model A must be assigned to layer $i$ of
model B. Otherwise, we can find another mapping that has a smaller assignment
cost. The optimal assignment cost in this situation is $S(i-1,j-1)+C_{j,i}$.
In the second situation, layer $j$ of model A is not mapped to any layers of
model B. Thus, the optimal assignment cost is $S(i,j-1)$. Considering all
scenarios, the optimal assignment cost is the lower cost between
$S(i-1,j-1)+C_{j,i}$ and $S(i,j-1)$. After finding the optimal assignment cost
$S(n,m)$ we can backtrack in the reverse direction to get the optimal CLA
mapping by finding pairs $(i,j)$ such that $S(i,j)=S(i-1,j-1)+C_{j,i}$. Note
that we have two constraints on the first and last hidden layers. Therefore,
$a(1)\leftarrow 1;a(n)\leftarrow m$.

Algorithm 1 Cross-layer alignment algorithm

Input: $\bm{C}=\left[\bm{C}_{i,j}\right]_{i,j}$

$S(i,i)\leftarrow\sum_{l=1}^{i}\bm{C}_{l,l},i\in\llbracket{n}\rrbracket$

$S(0,j)\leftarrow 0,j\in\llbracket{m}\rrbracket$ and $S(i,0)\leftarrow
0,i\in\llbracket{n}\rrbracket$

for $i=1$ to $n$ do

for $j=i+1$ to $m$ do

$S(i,j)\leftarrow\min\\{S(i,j-1),S(i-1,j-1)+\bm{C}_{j,i}\\}$

end for

end for

$a(1)\leftarrow 1;a(n)\leftarrow m;i\leftarrow n-1;j\leftarrow m-1$

while $i\geq 2$ do

while $j\geq i+1$ and $S(i,j)=S(i,j-1)$ do

$j\leftarrow j-1$

end while

$a(i)=j;i\leftarrow i-1;j\leftarrow j-1$

end while

Output: $a$

Given the cost matrix $\bm{C}$, the time complexity of our algorithm is only
$\mathcal{O}(mn)$. Note that the optimal alignment from Definition 1 is not
necessarily unique so we choose one which is obtained by backtracking.

CLA for Convolutional neural network (CNN). Following NeuralMerger, we do not
match fully connected layers and convolution layers. Naturally, a convolution
layer has different functionality from a fully connected layer. In addition,
there is a lack of an appropriate way to combine the weights of a fully
connected layer and a convolution layer. In this paper, we consider CNN
architecture which consists of consecutive convolution layers followed by
fully connected layers such as VGG [27] and RESNET [12]. The key idea is to
solve the CLA for same-type layers separately then combine two mappings. We
further break the convolution part into smaller groups and find the mapping
for each pair of groups. In particular, we divide into 5 groups separated by
the max-pooling layers in VGG. In RESNET, we group consecutive blocks with the
same number of channels, a.k.a. stage, and in each stage, align blocks instead
of layers. For the block representation, we choose the (pre-)activation matrix
of the second convolution layer in each block.

---
Figure 3:  Layer balancing examples:  In (a), a new layer is added between
hidden layers 1 and 2 of model B. Newly added neurons and weights are in red
(We omit zero weights). The new weight matrix between layers 1 and 2 of model
B is an identity matrix whose size equals the number of neurons at layer 1.
While the old weight matrix between layers 1 and 2 in model B becomes the new
weight matrix between layers 2 and 3. In (b), hidden layer 2 of model A is
removed. The red lines indicate the new weight matrix which is calculated
based on two old weight matrices and the activation matrix of the removed
layer.

###  3.2 Layer balancing methods

As mentioned above, some layers in model A may have no counterparts in model B
in the optimal mapping. Naturally, we have two opposite directions: reduce
layers in model A or add layers into model B. Assume that we have already
balanced the number of layers up to layer $l$ of model B. At layer $l+1$ of
model B, there are two possibilities. If $a(l+1)-a(l)=1$, the layer-wise
fusion method is ready to apply up to layer $l+1$. If $a(l+1)-a(l)>1$, we can
either merge layers between layers $a(l)$ and $a(l+1)$ of model A or add
layers between layers $l$ and $l+1$ of model B. Next, we discuss two methods
in the case $a(l+1)-a(l)=2$. When $a(l+1)-a(l)>2$, we can repeat the same
method $a(l+1)-a(l)-1$ times.

Add layers into model B. We add a layer $l^{\prime}$ between layers $l$ and
$l+1$ of model B. The new weight matrices are defined as
$\bm{W}_{B}^{(l^{\prime},l)}\leftarrow\mathbb{I}_{p_{B}^{(l)}}$ and
$\bm{W}_{B}^{(l+1,l^{\prime})}\leftarrow\bm{W}_{B}^{(l+1,l)}\in\mathbb{R}^{p_{B}^{(l+1)}\times
p_{B}^{(l)}}$. The new activation function is defined as
$f_{B}^{(l^{\prime})}\leftarrow f_{A}^{(a(l)+1)}$, which is ReLU. Because
$x_{B}^{(l)}\succeq 0\text{ for all }l\in\llbracket{n}\rrbracket$, from
Equation 1 we have

$x_{B}^{(l^{\prime})}=f_{B}^{(l^{\prime})}(\bm{W}_{B}^{(l^{\prime},l)}x_{B}^{(l)})=ReLU(x_{B}^{(l)})=x_{B}^{(l)}.$

Therefore, the information of model B remains unchanged after adding layer
$l^{\prime}$. Note that the new layer is just an identity mapping, which is a
trick that has been used in RESNET and NET2NET [4]. Also discussed in NET2NET,
the adding layers method is a function-preserving transformation that allows
us to generate a valuable initialization for training a larger network. We
later examine this hypothesis in our experiments.

Merge layers in model A. We merge layer $a(l)+1$ into layer $a(l)$ of model A
by directly connecting layer $a(l)$ to layer $a(l+1)$. Because
$f_{A}^{(a(l)+1)}$ is ReLU, the new weight matrix $\bm{W}_{A}^{(a(l+1),a(l))}$
can be written as

$\bm{W}_{A}^{(a(l+1),a(l))}=\bm{W}_{A}^{(a(l+1),a(l)+1)}\bm{D}_{A}^{(a(l)+1)}\bm{W}_{A}^{(a(l)+1,a(l))},$

where $\bm{D}_{A}^{(a(l)+1)}\in\mathbb{R}^{p_{A}^{(a(l)+1)}\times
p_{A}^{(a(l)+1)}}$ is an _input-dependent_ diagonal matrix with 0s and 1s on
its diagonal. We denote the sign (binary step) function as follow
$\text{step}(x)=1$ if $x>=0$ and $\text{step}(x)=0$ if $x<0$. Then, the
$i^{th}$ entry in the diagonal has a value of
$\text{step}(\bm{z}_{A}^{(a(l)+1)})$. Because the actual sign of neuron $i$ at
layer $a(l)+1$ varies by the input, we provide a simple estimation. Given that
the neuron $i\in\llbracket{p_{A}^{(a(l)+1)}}\rrbracket$ has a pre-activation
vector over t samples as $\bm{z}_{\bm{.},i}\in\mathbb{R}^{t}$, which is the
$i^{th}$ column of the pre-activation matrix $\bm{Z}_{A}^{(a(l)+1)}$. We
estimate the $i^{th}$ entry in the diagonal of matrix $\bm{D}_{A}^{(a(l)+1)}$
using either sign of sum (i.e., $\text{step}(\sum_{j=1}^{t}\bm{z}_{j,i})$) or
majority vote (i.e., 1 if at least $\frac{t}{2}$ samples have positive
activation values and 0 otherwise) or average of sign
($\frac{1}{t}\sum_{j=1}^{t}\text{step}(\bm{z}_{j,i})$).

Toy example. We illustrate two layer balancing methods by a toy example in
Figure 3. We used two neural networks with the same setting as in Figure 2.
Based on the optimal CLA, we can either add layers into Model B or merge
layers in Model A to equalize the number of layers.

Pros and cons of balance methods. An advantage of merging layers is that the
fused model has fewer layers. However, merging layers degrades the accuracy of
model A and it is slower than adding layers method that does not involve any
complex calculations. On the flip side, adding layers does not affect the
accuracy of model B but results in a deeper fused model. Surprisingly, merging
layers shows comparatively better performance than adding layers in our
experiments on the multilayer perceptron (MLP). Performance comparison between
the two methods will be provided in Appendix C and D.1. Note that we can use
more sophisticated model compression methods instead of the merging layers
method. Similarly, it is possible to replace the adding layers method with a
more advanced network expanding technique [37]. Here, we just introduce two
natural and fast ways to demonstrate our framework.

Balancing the number of layers for CNN. The same merging method does not work
in the case of CNN. On the other hand, the adding layers method can be easily
applied for convolution layers. For VGG, we can set all filters of a new
convolution layer to identity kernels. For RESNET, we add a new block in which
all filters of two convolution layers become zero kernels while the short-cut
connection remains as the identity mapping.

###  3.3 Extension to multiple neural networks

Solving the CLA problem for multiple neural networks is a non-trivial task. It
can be formulated as a multi-index assignment problem [30, 15]. In addition,
it has some connections to the multi-marginal optimal partial optimal
transport [8, 17]. Therefore, it is an interesting direction for our future
work. Here we only propose a simple yet effective approach to extend CLAFusion
to the case of multiple networks. Consider K pre-trained models
$\\{M_{i}\\}_{k=1}^{K}$. Our approach is similar to what they did in OTFusion
[28, Section 4]. Starting with an estimation of the fused model
$M_{\mathcal{F}}$, we apply the first and second parts of CLAFusion $K$ times
to depth-align $K$ pre-trained models with respect to the fused model. After
that, we apply OTFusion to those depth-aligned networks to produce the final
weights for the fused model. The choice of $M_{\mathcal{F}}$ plays an
important role in this approach. However, it is unclear how it is chosen in
OTFusion. In our experiments, the network with the most number of layers is
chosen as the initialization for $M_{\mathcal{F}}$.

##  4 Experiments

Outline. In this section, we showcase the practical usage of CLAFusion in
different setups. We first fuse a RESNET34 and a RESNET18 trained on CIFAR10,
CIFAR100 [19] and Tiny-ImageNet datasets [6]. Secondly, we incorporate CLA
into the framework of heterogeneous model transfer to boost the performance.
Finally, our method is applied to the teacher-student setting in which VGG
architectures are utilized. In all experiments, OTFusion is adopted as the
layer-wise model fusion in the third step of our framework. The detailed
settings including training hyperparameters, used assets, and computational
resources are deferred to Appendix B. We further conduct ablation studies to
validate the effectiveness of cross-layer alignment in all three types of
architectures (MLP, CNN, and RESNET) and compare two layer balancing methods
in Appendix C. In Appendix D, we provide additional experimental results
including skill transfer on the synthetic dataset and supplementary results
for experiments in the main text as well as highlight the computational
advantage of CLAFusion.

Comparison with NeuralMerger. Though NeuralMerger also considers the
heterogeneous model fusion setting, its settings are quite different from
ours. NeuralMerger combined VGG-16 with ZFNet [39] and LeNet [21] using hand-
designed cross-layer alignment. While our work only merged models of the same
architecture family (e.g., VGG-11 and VGG-13). In addition, its code is not
available to reproduce the methods in the paper. Furthermore, we also tried to
compare our optimal CLA mapping with its mapping by adopting its mapping into
our framework while keeping the second and third parts. However, OTFusion, the
third component used in our experiments, cannot combine with its mapping. This
is because OTFusion requires the height and width of output images (activation
matrix) at two corresponding layers of two models to be the same to calculate
the cost matrix if using activation-based alignment. If using weight-based
alignment, OTFusion requires kernels at two corresponding layers of two models
to have the same size. Neither of these two requirements is fulfilled by the
hand-crafted CLA in NeuralMerger. Note that NeuralMerger is able to combine
weights of two models because it used vector quantization to decompose weights
of two models into the same lower dimension while OTFusion is based on the
weight averaging mechanism.

Table 1: The results of fusing and finetuning RESNET34 ($M_{A}$) and RESNET18
($M_{B}$) on CIFAR10, CIFAR100, and Tiny-ImageNet datasets. Each entity shows
the average classification accuracy over 5 seeds. The best performing
initialization in each row is in bold. The detailed results are reported in
Table 14.

Dataset | $M_{A}$ | $M_{B}$ | Ensemble | $M_{\mathcal{F}}$ | Finetune
---|---|---|---|---|---
|  |  | Learning |  | $M_{A}$ | $M_{B}$ | $M_{\mathcal{F}}$ |  $M_{B}$ depth-aligned | Random RESNET34
CIFAR10 | 93.31 | 92.92 | 93.81 | 65.72 | 93.52 | 93.29 | 93.67 | 93.24 | 92.00
CIFAR100 | 65.93 | 65.33 | 68.49 | 27.93 | 66.87 | 66.19 | 67.59 | 66.48 | 64.59
Tiny-ImageNet | 54.84 | 53.29 | 58.70 | 17.19 | 55.17 | 53.81 | 55.46 | 53.94 | 54.16

###  4.1 Generate an efficient initialization

Settings. Following the experiments conducted in [28, Section 5.3], CLAFusion
can be used to generate an initialization when training the larger neural
network. We apply CLAFusion to pre-trained RESNET34 and RESNET18 on CIFAR10,
CIFAR100, and Tiny-ImageNet datasets. Since our purpose is not to obtain
state-of-the-art performance, we use the same architecture for both datasets.
Layer representation is the pre-activation matrix of $200$ samples while the
cost function is calculated by subtracting the linear CKA from 1. After
fusing, we also perform finetuning to improve the accuracy as observed in
previous model fusion works [5, 28]. The finetuning hyperparameters, which are
adopted from [28, Appendix S3.1.3], are reported in Table 8.

Baselines. We use NET2NET to generate a RESNET34 model, referred to as $M_{B}$
depth-aligned, which preserves the performance of $M_{B}$. To compare with our
method, we finetune the pre-trained models, $M_{B}$ depth-aligned, and a
RESNET34 from scratch. As a reference, we report the result of ensemble
learning that calculates the average predictions over all individual models.

Results. As can be seen from Table 1 retraining from the fused model gains
accuracies of $93.67,67.59$, and $55.46$, which are the highest among all
initializations on each dataset. This demonstrates the advantage of CLAFusion
over NET2NET when initializing a large model from a pre-trained small model.
Because our method allows combining the knowledge from the pre-trained large
model to generate a better initialization, rather than solely relying on the
knowledge of the small model. Although ensemble learning achieves higher
performance, it comes with the cost of a nearly $50\%$ increase in both memory
and inference time according to Table 20.

Table 2: The results of fusing and finetuning more than 2 residual networks on
CIFAR10, CIFAR100, and Tiny-ImageNet datasets. Pre-trained models are
alternately RESNET34 and RESNET18. Each entity shows the average
classification accuracy over 5 seeds. The best performing initialization in
each row is in bold. The detailed results are reported in Tables 15 and 16.

Dataset | Pre-trained models | Ensemble | $M_{\mathcal{F}}$ | Finetune
---|---|---|---|---
|  | Learning |  | Pre-trained models | $M_{\mathcal{F}}$
CIFAR10 | 93.31, 92.92, 93.16, 92.83 | 94.17 | 31.76 | 93.52, 93.29, 93.41, 93.20 | 93.76
| 93.31, 92.92, 93.16, 92.83, 93.18, 92.86 | 94.22 | 23.75 | 93.52, 93.29, 93.41, 93.20, 93.49, 93.24 | 93.78
CIFAR100 | 65.93, 65.33, 65.97, 65.28 | 70.27 | 8.47 | 66.87, 66.19, 66.77, 65.96 | 68.92
| 65.93, 65.33, 65.97, 65.28, 65.89, 65.30 | 70.62 | 4.39 | 66.87, 66.19, 66.77, 65.96, 66.56, 65.92 | 69.40
Tiny-ImageNet | 54.84, 53.29, 54.91, 53.23 | 61.28 | 4.05 | 55.17, 53.81, 55.29, 53.76 | 56.07
| 54.84, 53.29, 54.91, 53.23, 54.63, 53.47 | 62.32 | 2.47 | 55.17, 53.81, 55.29, 53.76, 55.06, 54.04 | 56.47

More than 2 models. We consider the same settings as in the 2-model scenario
but fuse 4(6) models instead. We train 2(3) RESNET34 and 2(3) RESNET18 models
and fuse them using CLAFusion before finetuning. In Table 2, the first, third
(and fifth) pre-trained models are RESNET34 while the second, fourth (and
sixth) pre-trained models are RESNET18. The fused model, after finetuning,
consistently yields the best performance on both datasets. On CIFAR100, for
example, the classification accuracy increases remarkably over finetuning pre-
train models by a margin of at least $2.05$ and $2.53$ for the cases of 4 and
6 models, respectively. In addition, increasing the number of pre-trained
models increases the final performance of CLAFusion. Our hypothesis is that
fusing multiple models which were trained at different random seeds can
aggregate different “knowledge” from these models. This is similar to the
outcome of ensemble learning. In terms of computational resources, the gap
between our model fusion method and ensemble learning is pushed even further.
Table 20 shows that ensemble learning uses approximately 3 and 4.5 times as
many resources as CLAFusion when combining 4 and 6 pre-trained RESNET models,
respectively.

Table 3: The results of heterogeneous model transfer from RESNET18 to
RESNET34. Each entity shows the average classification accuracy $\pm$ standard
deviation over 5 seeds. The best performing method in each row is in bold. The
detailed results can be found in Table 17.

Dataset | Method | HMT | HMT + Naive avg | HMT + OTFusion | HMT + CLA + OTFusion
---|---|---|---|---|---
CIFAR10 | Transfer | 10.79 $\pm$ 1.01 | 37.49 $\pm$ 4.59 | 18.32 $\pm$ 1.76 | 61.29 $\pm$ 3.69
| Transfer + Finetune | 93.10 $\pm$ 0.18 | 93.38 $\pm$ 0.19 | 93.66 $\pm$ 0.14 | 93.86 $\pm$ 0.14
CIFAR100 | Transfer | 2.70 $\pm$ 0.45 | 10.79 $\pm$ 1.36 | 5.14 $\pm$ 1.22 | 12.56 $\pm$ 2.79
| Transfer + Finetune | 65.15 $\pm$ 0.24 | 66.42 $\pm$ 0.25 | 67.23 $\pm$ 0.16 | 67.83 $\pm$ 0.21

####  4.1.1 Heterogeneous model transfer

Settings. Heterogeneous model transfer [35] is a branch of model transfer that
deals with heterogeneous neural networks. It is contrasted to homogeneous
model transfer, also known as transfer learning, in which the same network
architecture is used in both the pre-training and finetuning phase. In this
experiment, we apply CLA and OTFusion to the heterogeneous model transfer
method (in short, HMT). We transfer the pre-trained RESNET18 to the pre-
trained RESNET34 using different methods, then finetune and compare their
performance. We use the same pre-trained models, layer representation, layer
dissimilarity, and finetuning hyperparameters as in Section 4.1.

Baselines. First, the heterogeneous model transfer method (HMT) in the
original paper [35] is used. Model parameters of the pre-trained RESNET18 are
transferred to the pre-trained RESNET34. The weights that are not transferred
are set to the weights of the pre-trained RESNET34. In the second method (HMT
+ Naive avg), after transferring we apply naive averaging to the transferred
RESNET34, which is the result of HMT, and the pre-trained RESNET34. Naive
averaging is replaced with OTFusion in the third method (HMT + OTFusion).
Different from the third method, the last method utilizes the optimal mapping
from the CLA problem instead of the longest chain proposed in Wang et al. [35]
for the layer-to-layer transfer.

Results. Table 3 demonstrates the average performance of different transfer
methods. Using CLA improves the performance over the longest chain
substantially, with a significant gap of $42.97$ and $7.42$ between the third
and fourth methods on CIFAR10 and CIFAR100 datasets, respectively. After
finetuning, the combination of CLA and OTFusion leads to an improvement of
$0.76$ and $2.68$ compared to using HMT only on CIFAR10 and CIFAR100 datasets,
respectively. In addition, its final performance on CIFAR10 and CIFAR100
(93.86 and 67.83) is also better than finetuning pre-trained models in Table 1
(93.52 and 66.87) while that of HMT does not.

Table 4: The results of fusing teacher and student VGG models on the CIFAR10
dataset. The best performing student model is in bold. More results can be
found in Table 18.

# Params | Teacher | Students | Finetune
---|---|---|---
$(M_{A},M_{B},M_{\mathcal{F}})$ | $M_{A}$ | $M_{B}$ | $M_{\mathcal{F}}$ | $M_{A}$ | $M_{B}$ | $M_{\mathcal{F}}$ |  $M_{B}$ depth-aligned
(33M, 3M, 3M) | 92.70 | 89.92 | 82.66 | 92.65 | 89.89 | 90.96 | 90.84

Table 5: Knowledge distillation from teacher $M_{A}$ into student model
$M_{B}$. The "Best“ row shows the best performance for all combinations of
hyperparameters. The "Average" row illustrates the mean $\pm$ standard
deviation of the best accuracies obtained at different temperatures. The best
performing initialization in each row is in bold. The detailed results are
reported in Table 19.

| Distillation initialization
---|---
| Random $M_{B}$ | $M_{B}$ | $M_{\mathcal{F}}$ |  $M_{B}$ depth-aligned | Random $M_{\mathcal{F}}$
Best | 89.10 | 90.98 | 91.43 | 91.24 | 88.84
Average | 88.41 $\pm$ 0.77 | 90.73 $\pm$ 0.43 | 91.16 $\pm$ 0.25 | 91.09 $\pm$ 0.19 | 88.10 $\pm$ 0.68

###  4.2 Teacher-student fusion

Settings. In this experiment, we transfer the knowledge from the pre-trained
teacher model to the student model. We train two VGG models on the CIFAR10
dataset: model A has VGG13 architecture with a double number of channels at
each layer while model B has VGG11 architecture with a half number of channels
at each layer. After fusing the teacher and student models, we retrain the
fused model and compare it to the retraining result of the student model. The
layer measure and the layer dissimilarity are identical to Section 4.1. We
adopt the best hyperparameters as reported in [28, Appendix S11] for
retraining. All finetuning hyperparameters are reported in Table 8. Additional
experiments with teacher and student RESNET models on CIFAR100 are given in
Appendix D.4.

Results. The classification accuracies are summarized in Table 4. After
retraining, the fused model yields better performance than retraining other
student models. The compression ratio between the fused model and the teacher
is about 11 at a cost of reducing $1.74\%$ accuracy. This suggests that
CLAFusion can act as a network pruning method to compress a heavy model into a
lightweight one.

####  4.2.1 Knowledge distillation

Settings. It has been shown that pre-trained distillation [32], which performs
knowledge distillation from pre-trained large teacher models to a pre-trained
student model, outperforms both knowledge distillation to a randomly
initialized student and the conventional pre-training + finetuning scheme. As
discussed in OTFusion, the fused model can be used as an efficient
initialization for knowledge distillation. To examine the same application of
CLAFusion, we perform knowledge distillation to the above pre-trained VGG
models. We employ the method in Hinton et al. [13] to match the logit
distribution of the student model to that of the teacher model. For
initialization, we consider five different choices for the student model: (a)
randomly initialized $M_{B}$, (b) $M_{B}$, (c) $M_{\mathcal{F}}$, (d) $M_{B}$
depth-aligned, (e) randomly initialized $M_{\mathcal{F}}$. We sweep over a set
of hyperparameters to choose the best combination that maximizes the accuracy
of the student model. The sets of hyperparameters for distillation also
follows the setting in [28, Appendix S12]: temperature $T=\\{20,10,8,4,1\\}$
and loss-weight factor $\gamma=\\{0.05,0.1,0.5,0.7,0.95,0.99\\}$.

Results. The results for knowledge distillation are reported in Table 5. Using
the fused model as an initialization achieves the best performance among
different choices of initializations. It showcases the application of
CLAFusion as an efficient initialization for knowledge distillation. If
averaging over different temperatures, retraining the fused model $(90.96)$
works better than pre-trained distillation $(90.73)$. Even the best accuracy
of pre-trained distillation into the pre-trained student model $(90.98)$ is
slightly higher than that of retraining the fused model but comes with the
cost of hyperparameter tuning. Moreover, as reported in Table 21, an epoch in
pre-trained distillation takes 12.22 seconds while that of finetuning only
costs 7.48 seconds, which is around 1.63x speedup. It further strengthens the
claim that CLAFusion can serve as a reliable model compression method.

##  5 Conclusion

In the paper, we extend layer-wise model fusion methods to the setting of
heterogeneous neural networks by solving a cross-layer alignment problem,
followed by a layer balancing step. Finetuning the fused network from
CLAFusion achieves a better accuracy for RESNET trained on the CIFAR10,
CIFAR100, and Tiny-ImageNet datasets. In addition, our framework can be
incorporated into the heterogeneous model transfer framework to improve
performance. Furthermore, it shows potential applications for model
compression and knowledge distillation.

Supplement to “Model Fusion of Heterogeneous Neural Networks via Cross-Layer
Alignment”

In this supplementary material, we first present the limitations of our
method; then, detail the common settings, used assets, and computational
resources in Appendix B. Next, we illustrate the efficiency of CLA as well as
the importance of constraints on the first and last layers in Appendix C.
Finally, we provide additional results of our experiments in Appendix D.

##  Appendix A Limitations

The major limitations of our framework stem from our constraints on the
architecture of neural networks. Because in all experiments, OTFusion is used
in the third step of our framework, CLAFusion shares the same limitations as
OTFusion. Firstly, we do not consider networks that include bias and batch
normalization though they can be handled in the same way as discussed in
OTFusion [28, Appendix S1]. Secondly, our framework is not generalized to
complex architectures such as LSTM [14] or Transformer [34]. The main reason
is due to a lack of an effective approach for combining weights of complex
components such as LSTM cell or attention module in a reasonable way. This is
a common limitation for all model fusion methods based on weight averaging
mechanism [33, 29, 28]. Thirdly, while CLAFusion can combine two networks of
different depths and different widths, two networks must belong to the same
architecture family. This is because OTFusion requires the height and width of
output images (activation matrix) at two corresponding layers of two models to
be the same to calculate the cost matrix if using activation-based alignment.
If using weight-based alignment, OTFusion requires kernels at two
corresponding layers of two models to have the same size. Fourthly, similar to
other model fusion approaches [33, 29, 22, 36, 28], an additional fine-tuning
process is needed after performing model fusion to help the fused model
surpass the performance of the original pre-trained models. In terms of layer-
balancing techniques, merging layers degrades the performance of original
models and incurs computational overhead. In addition, our current merging
technique is not applicable to convolutional layers.

##  Appendix B Experiment settings

In all experiments, we use OTFusion to fuse same-length models after aligning
and balancing in the first two steps of CLAFusion. Following the same
alignment strategy in [28, Appendix S1], OTFusion is computed using the
activation-based alignment strategy in which the pre-activation matrix instead
of the activation matrix is used for the neuron measure. The hyperparameters
for pre-trained models are summarized in Table 6. Without any further
specification, the random seed for training or retraining is set to its
default value. The accuracy of pre-trained MLP is the accuracy of the last
epoch. While the accuracies of VGG and RESNET are reported as the best
performing checkpoint. For all finetuning experiments, we always chose the
best record among all epochs. The hyperparameters for finetuning RESNET and
VGG models are reported in Tables 8 and 8, respectively.

Table 6: The hyperparameters for training different network architectures.

| MLP | VGG | ResNet (CIFAR) | RESNET (Tiny-ImageNet)
---|---|---|---|---
Number of epochs | 10 | 300 | 300 | 120
Training batch size | 64 | 128 | 256 | 256
Test batch size | 1000 | 1000 | 1000 | 1000
Optimizer | SGD | SGD | SGD | SGD
Initial LR | 0.01 | 0.05 | 0.1 | 0.1
Momentum | 0.5 | 0.9 | 0.9 | 0.9
Weight decay |  | 0.0005 | 0.0001 | 0.0001
LR decay factor |  | 2 | 10 | 10
LR decay epochs |  | 30,60,$\ldots$,270 | 150,250 | 30,60,90
Default seed | 0 | 42 | 42 | 42
5 seeds | 0,1,2,3,4 | 40,41,42,43,44 | 40,41,42,43,44 | 40,41,42,43,44

Table 7: Finetuning RESNET hyperparameters.

| CIFAR | Tiny-ImageNet
---|---|---
Number of epochs | 120 | 90
Training batch size | 256 | 256
Test batch size | 1000 | 1000
Optimizer | SGD | SGD
Initial LR | 0.1 | 0.1 or 0.01
Momentum | 0.9 | 0.9
Weight decay | 0.0001 | 0.0001
LR decay factor | 2 | 10
LR decay epochs | 20,40,60,80,100 | 30,60,90

Table 8: Finetuning VGG hyperparameters.

| CIFAR |
---|---|---
Number of epochs | 120 |
Training batch size | 128 |
Test batch size | 1000 |
Optimizer | SGD |
Initial LR | 0.01 |
Momentum | 0.9 |
Weight decay | 0.0005 |
LR decay factor | 2 |
LR decay epochs | 20,40,60,80,100 |

Open source code. We adapt the official implementation of
OTFusion111https://github.com/sidak/otfusion [28] in our implementation. For
computing the optimal transport, we use Python Optimal Transport (POT)
library222https://github.com/PythonOT/POT [9]. For model transfer, the source
code of the original
paper333https://anonymous.4open.science/r/6ab184dc-3c64-4fdd-
ba6d-1e5097623dfd/a_hetero_model_transfer.py [35] is utilized.

Neural network architectures. Abusing the notation in VGG’s paper [27], we
represent several CNN architectures used in the experiments as follows.

_VGG13 doub_ : Input layer
$\rightarrow\text{conv3-64}\rightarrow\text{conv3-128}\rightarrow\text{maxpool}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{maxpool}\rightarrow\text{conv3-512}\rightarrow\text{conv3-512}\rightarrow\text{maxpool}\rightarrow\text{conv3-1028}\rightarrow\text{conv3-1028}\rightarrow\text{maxpool}\rightarrow\text{conv3-1028}\rightarrow\text{conv3-512}\rightarrow\text{maxpool}\rightarrow$
Output layer

_VGG11 half_ : Input layer
$\rightarrow\text{conv3-64}\rightarrow\text{maxpool}\rightarrow\text{conv3-64}\rightarrow\text{maxpool}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow\text{maxpool}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{maxpool}\rightarrow\text{conv3-256}\rightarrow\text{conv3-512}\rightarrow\text{maxpool}\rightarrow$
Output layer

_VGG13 stud_ : Input layer
$\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow\text{maxpool}\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow\text{maxpool}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow\text{maxpool}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{maxpool}\rightarrow\text{conv3-256}\rightarrow\text{conv3-512}\rightarrow\text{maxpool}\rightarrow$
Output layer

_RESNET34_ : Input layer
$\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow{\color[rgb]{1,0,0}conv1-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow{\color[rgb]{1,0,0}conv1-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-512}\rightarrow\text{conv3-512}\rightarrow{\color[rgb]{1,0,0}conv1-512}\rightarrow\text{conv3-512}\rightarrow\text{conv3-512}\rightarrow\text{conv3-512}\rightarrow\text{conv3-512}\rightarrow$
Output layer

_RESNET18_ : Input layer
$\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow{\color[rgb]{1,0,0}conv1-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow{\color[rgb]{1,0,0}conv1-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-512}\rightarrow\text{conv3-512}\rightarrow{\color[rgb]{1,0,0}conv1-512}\rightarrow\text{conv3-512}\rightarrow\text{conv3-512}\rightarrow$
Output layer

Note that the average-pooling layer is skipped in the representation. In
addition, the last fully-connected layers, which are usually called classifier
layers, are merged into the output layer. In RESNET models, convolution layers
in red color with a kernel size of 1 is the shortcut layer that connects two
stages of different channel sizes. These layers are represented by dotted
shortcuts in Figure 3 in RESNET’s paper [12].

Computing devices. All experiments are done on 1 NVIDIA A100 40GB GPU.

##  Appendix C Ablation studies

In this section, we show that the optimal CLA is superior to other strictly
increasing mappings by comparing the performance of the fused model which is
obtained by applying the second and third steps of CLAFusion with the
corresponding cross-layer mapping. We first study the importance of CLA and
two constraints to the classification accuracy of the fused model when the
pre-trained models are fully connected neural networks. In the latter part, we
validate the advantage of CLA in fusing convolutional neural networks.

Table 9: The optimal CLA for different combinations of layer representation
and cost function when fusing 2 MLPs on the MNIST dataset. Experiments were
run with 5 different seeds. Changing the random seed did not alter the optimal
mappings except for two cases which are in red.

Layer representation | Cost function | Both | Last layer constraint | First layer constraint | None
---|---|---|---|---|---
Number of neurons | squared difference | [1, 2, 5] | [1, 2, 5] | [1, 2, 3] | [1, 2, 3]
Pre-activation matrix | 1 - linear CKA | [1, 2, 5] | [1, 2, 5] | [1, 2, 3] | [1, 2, 3]
Pre-activation matrix | Wasserstein distance | [1, 2, 5] | [1, 2, 5] | [1, 2, 3] | [1, 2, 3]

###  C.1 Fully connected neural networks

Settings. We compare the performance of different combinations of layer
representation and layer balancing methods in our framework. To prove the
efficiency of the CLA step, we compare the result of fusing two models using
different mappings. We also consider removing the constraint on the first and
last hidden layers. We train two MLPs on the MNIST dataset [21] in 5 different
seeds. Model A has 5 hidden layers of size 400, 200, 100, 50, 25 while model B
has 3 hidden layers of size 400, 200, 100 as the following representation.

_Model A_ : Input layer $\rightarrow\underbrace{\text{FC-400}}_{\text{layer
1}}\rightarrow\underbrace{\text{FC-200}}_{\text{layer
2}}\rightarrow\underbrace{\text{FC-100}}_{\text{layer
3}}\rightarrow\underbrace{\text{FC-50}}_{\text{layer
4}}\rightarrow\underbrace{\text{FC-25}}_{\text{layer 5}}\rightarrow$ Output
layer

_Model B_ : Input layer $\rightarrow\underbrace{\text{FC-400}}_{\text{layer
1}}\rightarrow\underbrace{\text{FC-200}}_{\text{layer
2}}\rightarrow\underbrace{\text{FC-100}}_{\text{layer 3}}\rightarrow$ Output
layer

Table 10: Performance comparison between different combinations of mapping and
balancing methods when fusing 2 MLPs on the MNIST dataset. Experiments were
run with 5 different seeds. The mapping obtained from the CLA step and the
best performing mapping in columns $M_{\mathcal{F}}$ are in bold.

Mapping | $M_{A}$ | $M_{B}$ | $M_{\mathcal{F}}$
---|---|---|---
|  |  | Add | Merge (sum) | Merge (average)
$[1,2,3]$ | 96.95 $\pm$ 0.18 | 97.75 $\pm$ 0.04 | 92.33 $\pm$ 2.35 | 93.90 $\pm$ 1.57 | 91.84 $\pm$ 3.06
$[1,2,4]$ | 92.48 $\pm$ 2.07 | 93.41 $\pm$ 1.56 | 91.88 $\pm$ 1.62
$\bm{[1,2,5]}$ | 92.49 $\pm$ 2.64 | 94.00 $\pm$ 1.36 | 92.37 $\pm$ 2.94
$[1,3,4]$ | 90.69 $\pm$ 2.62 | 92.21 $\pm$ 1.86 | 89.17 $\pm$ 0.98
$[1,3,5]$ | 90.88 $\pm$ 2.61 | 93.04 $\pm$ 2.44 | 89.57 $\pm$ 3.84
$[1,4,5]$ | 90.98 $\pm$ 2.35 | 93.06 $\pm$ 2.07 | 90.26 $\pm$ 2.84
$[2,3,4]$ | 82.21 $\pm$ 1.79 | 51.26 $\pm$ 6.41 | 55.62 $\pm$ 5.97
$[2,3,5]$ | 83.14 $\pm$ 2.18 | 56.56 $\pm$ 5.87 | 59.46 $\pm$ 5.73
$[2,4,5]$ | 83.00 $\pm$ 2.01 | 54.16 $\pm$ 6.01 | 58.62 $\pm$ 5.60
$[3,4,5]$ | 79.49 $\pm$ 2.37 | 56.31 $\pm$ 4.79 | 55.90 $\pm$ 3.02

The optimal CLA. The cross-layer mappings obtained using Algorithm 1 with
different combinations are given in Table 9. According to Definition 1 in
Section 3.1, the optimal mapping is strictly increasing mapping
$a:\llbracket{n}\rrbracket\mapsto\llbracket{m}\rrbracket$ where $n$ and $m$
are the number of hidden layers of the shallower and deeper networks,
respectively. A mapping $a$ is represented as $[a(1),a(2),\ldots,a(n)]$ where
$a(i)=j$ indicates that layer i of model B is aligned to layer j of model A.
In this case, we have $m=5$ and $n=3$ and the mapping [1, 2, 5] means that
$a(1)=1,a(2)=2,$ and $a(3)=5$. We observe that changing the random seed barely
affects the result of CLA in this setting. When keeping both first and last
layer constraints, all three combinations yield the same mappings. All three
combinations share the same mappings in almost all cases even if none of the
two constraints are placed444When the random seed is 2, the optimal mapping
for ”pre-activation matrix“ + ”1 - linear CKA“ is $[1,2,5]$ instead of
$[1,2,3]$.. In this case, the first layer constraint does not affect the
results of the optimal mapping in all combinations.

The effectiveness of the optimal mapping. We fuse two MLPs for all 10 possible
mappings between two models A and B. For the layer balancing method, we try
both adding layers and merging layers (the sign of sum and the average of sign
estimations). The average performance of pre-trained models and the fused
model across 5 random seeds are summarized in Table 10. On either choice of
the layer balancing method, the best accuracy of the fused model is achieved
when imposing both the first and last layer constraints. Removing the last
layer constraint decreases slightly the performance of the fused model. When
the first layers of the two models do not match each other in the last 4 rows,
the accuracy drops dramatically. Because an input image may consist of
negative cell values due to normalization, adding a new layer as the first
hidden layer does not maintain the accuracy of the shallower model. Therefore,
it is necessary to match the first and last hidden layers of the two models.
Comparing layer balancing methods, the merging layers method runs slower but
interestingly leads to a higher accuracy than the adding layers in this
situation.

###  C.2 Convolutional neural networks

In this section, we conduct ablation studies on two common CNN architectures
including VGG and RESNET. For solving the CLA problem, layer representation is
the pre-activation matrix of 200 samples while the cost function is calculated
by subtracting the linear CKA from 1. In terms of the layer balancing method,
adding layers is chosen. After fusing two models, we perform finetuning with
hyperparameters in Tables 8 and 8.

####  C.2.1 VGG architecture

Settings. We consider the same two VGG models used in Section 4.2 on the
CIFAR10 dataset. Using the same notation as in VGG’s paper [27], the
architectures of the two models can be represented as follows.

_Model A_ : Input layer
$\rightarrow\Big{[}\underbrace{\text{conv3-64}}_{\text{layer
1}}\rightarrow\underbrace{\text{conv3-128}}_{\text{layer
2}}\Big{]}\rightarrow\text{maxpool}\rightarrow\Big{[}\underbrace{\text{conv3-256}}_{\text{layer
3}}\rightarrow\underbrace{\text{conv3-256}}_{\text{layer
4}}\Big{]}\rightarrow\text{maxpool}\rightarrow\Big{[}\underbrace{\text{conv3-512}}_{\text{layer
5}}\rightarrow\underbrace{\text{conv3-512}}_{\text{layer
6}}\Big{]}\rightarrow\text{maxpool}\rightarrow\Big{[}\underbrace{\text{conv3-1028}}_{\text{layer
7}}\rightarrow\underbrace{\text{conv3-1028}}_{\text{layer
8}}\Big{]}\rightarrow\text{maxpool}\rightarrow\Big{[}\underbrace{\text{conv3-1028}}_{\text{layer
9}}\rightarrow\underbrace{\text{conv3-512}}_{\text{layer
10}}\Big{]}\rightarrow\text{maxpool}\rightarrow$ Output layer

_Model B_ : Input layer
$\rightarrow\Big{[}\underbrace{\text{conv3-64}}_{\text{layer
1}}\Big{]}\rightarrow\text{maxpool}\rightarrow\Big{[}\underbrace{\text{conv3-64}}_{\text{layer
2}}\Big{]}\rightarrow\text{maxpool}\rightarrow\Big{[}\underbrace{\text{conv3-128}}_{\text{layer
3}}\rightarrow\underbrace{\text{conv3-128}}_{\text{layer
4}}\Big{]}\rightarrow\text{maxpool}\rightarrow\Big{[}\underbrace{\text{conv3-256}}_{\text{layer
5}}\rightarrow\underbrace{\text{conv3-256}}_{\text{layer
6}}\Big{]}\rightarrow\text{maxpool}\rightarrow\Big{[}\underbrace{\text{conv3-256}}_{\text{layer
7}}\rightarrow\underbrace{\text{conv3-512}}_{\text{layer
8}}\Big{]}\rightarrow\text{maxpool}\rightarrow$ Output layer

Note that following the official implementation of OTFusion, we did not modify
the number of channels of the first and last convolution layers.

Table 11: Performance comparison between different mappings when fusing 2 VGG
models on the CIFAR10 dataset. Experiments were run with 5 different seeds. To
make the notation succinct, the mapping only indicates the values of $a(1)$
and $a(2)$. The mapping obtained from the CLA step is in bold while two
additional inter-group mappings are in red. The best performing mapping in
columns $M_{\mathcal{F}}$ is in bold.

Mapping | $M_{A}$ | $M_{B}$ | $M_{\mathcal{F}}$
---|---|---|---
|  |  | Before finetuning | After finetuning
$[1,2]$ | 92.65 $\pm$ 0.16 | 89.72 $\pm$ 0.12 | 62.76 $\pm$ 1.92 | 90.56 $\pm$ 0.12
$\bm{[1,3]}$ | 82.03 $\pm$ 0.50 | 90.64 $\pm$ 0.17
$[1,4]$ | 79.37 $\pm$ 1.05 | 90.37 $\pm$ 0.20
$[2,3]$ | 46.64 $\pm$ 2.23 | 89.32 $\pm$ 0.15
$[2,4]$ | 49.63 $\pm$ 2.80 | 89.03 $\pm$ 0.19
$[3,4]$ | 46.93 $\pm$ 1.99 | 84.62 $\pm$ 0.16

The optimal CLA. Because two models have identical input, output, and fully
connected layers, we only need to consider matching their convolution layers.
The last layers of the two models are fully connected layers, which are
automatically assigned to each other in this setup as we solve the CLA problem
for different layer types (e.g., convolutional layers and fully connected
layers) independently. As discussed in Section 3.1, each VGG model is divided
into 5 groups which are separated by the max-pooling layers and enclosed in
square brackets in the above representation. For the last three pairs of
groups, the optimal mappings are trivial because each pair has the same number
of layers. Abusing the notation in Definition 1 in Section 3.1, we have
$a(3+i)=5+i,i\in\llbracket{5}\rrbracket$. Therefore, we only need to solve the
CLA problem for the first two groups, i.e., to find the values of $a(1)$ and
$a(2)$. In this case, we again observe that the optimal mapping remains the
same even if changing the random seed.

The effectiveness of the optimal mapping. Other than the optimal mapping
obtained from CLA, we fuse two VGG models for 3 intra-group mappings and 2
additional inter-group mappings. Here, intra-group means that layers in the
first(second) group of model B are only aligned to layers in the first(second)
group of model A. While inter-group means that layers in the first or second
group of model B can be aligned to layers in the both first and second groups
of model A. In essence, the inter-group mappings are discouraged due to the
inconsistency of the input image size as discussed in Section 2.3. In this
experiment only, we consider them for comparison purposes. Table 11
illustrates the average performance of the fused model for different mappings.
Before finetuning, it can be seen clearly that the optimal CLA shows a clear
margin of at least $2.66\%$ in the performance over other mappings. Similar to
the case of MLPs, the performance falls remarkably when not aligning the first
layers of two models in the last 3 rows. In terms of the performance after
finetuning, the optimal CLA still beats other mappings, resulting in an
accuracy of 90.64 on average.

####  C.2.2 RESNET architecture

Settings. We consider the same two RESNET models used in Section 4.1 on the
CIFAR10 dataset. Using the same notation as in VGG’s paper [27], the
architectures of the two models can be represented as follows.

_RESNET34_ : Input layer
$\rightarrow\text{conv3-64}\rightarrow\Big{[}(\text{conv3-64}\rightarrow\text{conv3-64})\rightarrow(\text{conv3-64}\rightarrow\text{conv3-64})\rightarrow(\text{conv3-64}\rightarrow\text{conv3-64})\Big{]}\rightarrow\Big{[}(\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow{\color[rgb]{1,0,0}conv1-128})\rightarrow(\text{conv3-128}\rightarrow\text{conv3-128})\\\
\rightarrow(\text{conv3-128}\rightarrow\text{conv3-128})\rightarrow(\text{conv3-128}\rightarrow\text{conv3-128})\Big{]}\rightarrow\Big{[}(\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow{\color[rgb]{1,0,0}conv1-256})\rightarrow(\text{conv3-256}\rightarrow\text{conv3-256})\rightarrow(\text{conv3-256}\rightarrow\text{conv3-256})\rightarrow(\text{conv3-256}\rightarrow\text{conv3-256})\\\
\rightarrow(\text{conv3-256}\rightarrow\text{conv3-256})\rightarrow(\text{conv3-256}\rightarrow\text{conv3-256})\Big{]}\rightarrow\Big{[}(\text{conv3-512}\rightarrow\text{conv3-512}\rightarrow{\color[rgb]{1,0,0}conv1-512})\rightarrow(\text{conv3-512}\rightarrow\text{conv3-512})\rightarrow(\text{conv3-512}\rightarrow\text{conv3-512})\Big{]}\rightarrow$
Output layer

_RESNET18_ : Input layer
$\rightarrow\text{conv3-64}\rightarrow\Big{[}(\text{conv3-64}\rightarrow\text{conv3-64})\rightarrow(\text{conv3-64}\rightarrow\text{conv3-64})\Big{]}\rightarrow\Big{[}(\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow{\color[rgb]{1,0,0}conv1-128})\rightarrow(\text{conv3-128}\rightarrow\text{conv3-128})\Big{]}\rightarrow\Big{[}(\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow{\color[rgb]{1,0,0}conv1-256})\rightarrow(\text{conv3-256}\rightarrow\text{conv3-256})\Big{]}\rightarrow\Big{[}(\text{conv3-512}\rightarrow\text{conv3-512}\rightarrow\\\
{\color[rgb]{1,0,0}conv1-512})\rightarrow(\text{conv3-512}\rightarrow\text{conv3-512})\Big{]}\rightarrow$
Output layer

Table 12: Performance comparison between different mappings when fusing 2
RESNET models on the CIFAR10 dataset. Experiments were run with 5 different
seeds. The term “CLA woc" stands for CLA without the last layer constraint.
The best performing mapping in columns $M_{\mathcal{F}}$ is in bold.

Mapping | $M_{A}$ | $M_{B}$ | $M_{\mathcal{F}}$
---|---|---|---
|  |  | Before finetuning | After finetuning
Naive | 93.31 $\pm$ 0.13 | 92.92 $\pm$ 0.20 | 62.43 $\pm$ 3.08 | 93.59 $\pm$ 0.18
Random | 64.04 $\pm$ 3.01 | 93.61 $\pm$ 0.12
CLA woc | 65.58 $\pm$ 2.61 | 93.60 $\pm$ 0.14
CLA | 65.72 $\pm$ 2.33 | 93.67 $\pm$ 0.07

The optimal CLA. Each RESNET model is also divided into 5 stages which are
enclosed in square brackets in the above representation. And each building
block is enclosed in parentheses. As discussed in Section 3.1, in each stage
we find the optimal mapping for blocks instead of layers. Because each stage
of RESNET18 has only two blocks, the first block is assigned to the first
block while the second one is assigned to the last block in the same stage of
RESNET34.

The effectiveness of the optimal mapping. We compare the optimal CLA with
three baseline mappings. The first baseline is the naive mapping of blocks
with the same index, i.e., the $i^{th}$ block of RESNET18 is aligned to the
$i^{th}$ block in the same stage of RESNET34. The second one is random mapping
which satisfies the first layer constraint. Because the total number of
strictly increasing mappings is quite large, we only sample 5 random mappings
for each random seed and take their average performance. The last mapping is
the one that is obtained by solving the CLA problem but removing the last
layer constraint. Table 12 compares the classification accuracy of the fused
model on different types of mapping. Before finetuning, using the optimal CLA
outperforms the naive and random mappings with a wide margin of 3.29 and 1.68,
respectively. Removing the last layer constraint decreases the performance
slightly but still yields better performance than the other two. After
finetuning, the optimal CLA produces slightly higher accuracy than the other
mappings. In terms of baselines, three mappings have a comparative performance
of approximately $93.60\%$.

##  Appendix D Additional experimental results

We first apply CLAFusion to the skill transfer experiment [28, Section5.1] but
for heterogeneous neural networks. In the subsequent sections, we report the
detailed results for different runs of experiments in the main text. Finally,
we showcase the efficiency of CLAFusion in terms of computational resources.

|
---|---
(a) Adding layers | (b) Merging layers (sum)
Figure 4:  The average performance of skill transfer using (a) adding layers
and (b) merging layers as the layer balancing method. Experiments were run
with 5 different seeds.

###  D.1 Skill transfer

Settings. In skill transfer, we aim to obtain a single model that can inherit
both overall and specialized skills from the two individual models. We adopt
the same heterogeneous data-split technique as in [28, Section 5.1] for two
MLPs A and B. The MNIST dataset is separated into two datasets. The dataset
for model A has all images of label $4$ and $10\%$ images of other labels
while that for model B contains the rest. We train an MLP with $3$ hidden
layers of size $400,200,100$ for model A and an MLP with $4$ hidden layers of
size $400,200,100,50$ for model B. The pre-activation matrix is obtained by
running inference for $400$ samples. We use the number of neurons as layer
representation and adding layers method for balancing. The weight proportion
of model A, which is referred to as $w_{A}$ (also known as fusion rate in
Smith et al. [29]), is swept over 11 different values:
$w_{A}=\\{0.0,0.1,\ldots,1.0\\}$.

Table 13: Performance comparison between two layer balancing methods on skill
transfer task. Experiments were run with 5 different seeds. The best
performing result in each row is in bold.

| Seed | $w_{A}$
---|---|---
|  | 0.0 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1.0
Add | 0 | 86.10 | 87.06 | 88.23 | 78.64 | 73.48 | 69.70 | 70.45 | 75.91 | 81.38 | 85.36 | 87.68
1 | 91.60 | 91.81 | 91.23 | 85.63 | 77.20 | 77.99 | 81.59 | 84.85 | 86.88 | 87.48 | 87.29
2 | 90.80 | 92.07 | 92.46 | 84.06 | 82.53 | 83.21 | 85.01 | 86.83 | 87.76 | 88.01 | 87.83
3 | 91.94 | 92.31 | 91.39 | 79.58 | 71.85 | 71.77 | 78.31 | 83.63 | 85.79 | 87.01 | 87.51
4 | 93.34 | 93.31 | 91.56 | 81.70 | 77.45 | 78.46 | 82.15 | 85.00 | 86.62 | 87.37 | 87.58
Avg | 90.76 | 91.31 | 90.97 | 81.92 | 76.50 | 76.23 | 79.50 | 83.24 | 85.69 | 87.05 | 87.58
Merge (sum) | 0 | 86.10 | 87.19 | 87.10 | 78.51 | 63.66 | 56.74 | 53.31 | 53.88 | 57.65 | 66.55 | 76.35
1 | 91.60 | 92.23 | 91.83 | 89.08 | 78.69 | 76.04 | 77.34 | 81.44 | 84.87 | 86.39 | 86.60
2 | 90.80 | 91.98 | 92.30 | 86.00 | 81.27 | 79.42 | 80.09 | 82.95 | 85.46 | 85.61 | 85.04
3 | 91.94 | 92.66 | 92.76 | 88.13 | 71.65 | 65.90 | 64.20 | 67.39 | 72.91 | 76.06 | 77.40
4 | 93.34 | 93.80 | 93.18 | 90.03 | 80.24 | 79.27 | 81.01 | 82.52 | 83.91 | 84.68 | 85.09
Avg | 90.76 | 91.57 | 91.43 | 86.35 | 75.10 | 71.47 | 71.19 | 73.64 | 76.96 | 79.86 | 82.10
Merge (average) | 0 | 86.10 | 87.55 | 88.65 | 85.23 | 69.70 | 60.47 | 57.45 | 59.01 | 63.90 | 75.26 | 84.55
1 | 91.60 | 92.16 | 91.62 | 89.28 | 79.00 | 75.20 | 76.80 | 81.53 | 85.00 | 86.75 | 87.10
2 | 90.80 | 91.96 | 92.35 | 88.33 | 81.11 | 79.69 | 80.60 | 83.80 | 86.28 | 86.64 | 86.42
3 | 91.94 | 92.74 | 92.60 | 87.63 | 74.14 | 68.91 | 68.37 | 74.89 | 80.75 | 84.04 | 85.61
4 | 93.34 | 93.83 | 92.80 | 89.48 | 80.96 | 78.68 | 80.50 | 83.19 | 84.82 | 86.11 | 86.56
Avg | 90.76 | 91.65 | 91.60 | 87.99 | 76.98 | 72.59 | 72.74 | 76.48 | 80.15 | 83.76 | 86.05

Results. Figure 4(a) shows that when the weight proportion of model B is small
($0.1$ or $0.2$), the fused model improves over both individual models. The
reason is that the specialist model A has higher accuracy than the generalist
model B. When distributing the weight proportion fairly ($0.5$), the fused
model performs worst due to the effect of heterogeneous training data. The
fused model has the best accuracy of $91.31\%$, which is higher than those of
models A ($90.76\%$) and B ( $87.58\%$). This claims that the knowledge has
been transferred successfully to the fused model without retraining.
Furthermore, the detailed results of skill transfer for 5 different random
seeds are reported in Table 13. The specialist model A generally has higher
accuracy than the generalist model B seeds because it has been trained on all
10 labels. In 4 out of 5 seeds, the fused model improves over both individual
models when the weight proportion of model B is small ($w_{A}=\\{0.1,0.2\\}$).

Comparison between two layer balancing methods. For comparison purposes, we
further conduct the same procedure but replace adding layers by merging layers
(the sign of sum and the average of sign estimation). Similar to the previous
experiment, the fused model attains the best performance when the weight
proportion of model B is small ($w_{A}=\\{0.1,0.2\\}$). Figure 4 illustrates
the average performance of two layer balancing methods on the skill transfer
task. When averaging over 5 random seeds, both methods perform best if
$w_{A}=0.1$. The best accuracy of merging layers $(91.57,91.65)$ is slightly
higher than that of adding layers $(91.31)$ even though the accuracy of
$M_{B}$ depth-aligned drops from $87.58$ to $82.10$ or $86.05$.

Table 14: The results of finetuning RESNET from different choices of
initialization across 5 different seeds. The best performing initialization in
each row is in bold.

Dataset | Seed | $M_{A}$ | $M_{B}$ | Ensemble | $M_{\mathcal{F}}$ | Finetune
---|---|---|---|---|---|---
|  |  |  | Learning |  | $M_{A}$ | $M_{B}$ | $M_{\mathcal{F}}$ |  $M_{B}$ depth-aligned | Random RESNET34
CIFAR10 | 40 | 93.42 | 93.21 | 94.10 | 67.83 | 93.50 | 93.47 | 93.61 | 93.47 | 91.95
| 41 | 93.37 | 92.90 | 93.81 | 63.66 | 93.61 | 93.26 | 93.74 | 93.32 | 91.93
| 42 | 93.31 | 93.05 | 93.94 | 68.30 | 93.61 | 93.36 | 93.71 | 93.43 | 92.25
| 43 | 93.39 | 92.78 | 93.71 | 66.45 | 93.50 | 93.30 | 93.73 | 93.00 | 91.76
| 44 | 93.06 | 92.65 | 93.49 | 62.35 | 93.37 | 93.05 | 93.55 | 92.97 | 92.12
| Avg | 93.31 | 92.92 | 93.81 | 65.72 | 93.52 | 93.29 | 93.67 | 93.24 | 92.00
CIFAR100 | 40 | 66.08 | 65.19 | 68.44 | 29.44 | 66.73 | 66.47 | 67.71 | 66.23 | 64.73
| 41 | 66.21 | 65.22 | 68.58 | 28.92 | 67.32 | 66.33 | 67.54 | 66.45 | 64.24
| 42 | 65.89 | 65.04 | 68.39 | 26.62 | 66.77 | 65.91 | 67.16 | 66.35 | 64.66
| 43 | 65.85 | 66.17 | 68.98 | 27.35 | 67.12 | 66.35 | 68.05 | 67.23 | 65.67
| 44 | 65.61 | 65.03 | 68.05 | 27.31 | 66.51 | 65.90 | 67.50 | 66.16 | 63.65
| Avg | 65.93 | 65.33 | 68.49 | 27.93 | 66.87 | 66.19 | 67.59 | 66.48 | 64.59
Tiny-ImageNet | 40 | 54.29 | 53.13 | 58.27 | 17.22 | 54.50 | 53.71 | 54.75 | 53.77 | 53.91
| 41 | 54.81 | 53.74 | 59.44 | 17.84 | 55.20 | 53.17 | 55.23 | 53.24 | 53.26
| 42 | 55.37 | 53.74 | 59.44 | 17.84 | 55.87 | 54.26 | 56.00 | 54.60 | 54.44
| 43 | 54.83 | 53.62 | 59.13 | 14.82 | 55.27 | 53.91 | 55.96 | 54.10 | 54.87
| 44 | 54.91 | 53.28 | 58.31 | 18.33 | 54.99 | 54.02 | 55.35 | 53.98 | 54.32
| Avg | 54.84 | 53.29 | 58.70 | 17.19 | 55.17 | 53.81 | 55.46 | 53.94 | 54.16

Table 15: The results of fusing and finetuning 4 RESNET from different choices
of initialization across 5 different seeds. The best performing initialization
in each row is in bold.

Dataset | Seeds | Pre-trained models | Ensemble | $M_{\mathcal{F}}$ | Finetune
---|---|---|---|---|---
|  | (RESNET34, RESNET18) x2 | Learning |  | Pre-trained models | $M_{\mathcal{F}}$
CIFAR10 | 40, 41 | 93.42, 93.21, 93.37, 92.90 | 94.36 | 34.55 | 93.50, 93.47, 93.61, 93.26 | 93.97
| 41, 42 | 93.37, 92.90, 93.31, 93.05 | 94.20 | 32.35 | 93.61, 93.26, 93.61, 93.36 | 93.72
| 42, 43 | 93.31, 93.05, 93.39, 92.78 | 94.10 | 34.01 | 93.61, 93.36, 93.50, 93.30 | 93.81
| 43, 44 | 93.39, 92.78, 93.06, 92.65 | 94.11 | 34.93 | 93.50, 93.30, 93.37, 93.05 | 93.73
| 44, 45 | 93.06, 92.65, 92.66, 92.76 | 94.06 | 22.98 | 93.37, 93.05, 92.94, 93.04 | 93.56
| Avg | 93.31, 92.92, 93.16, 92.83 | 94.17 | 31.76 | 93.52, 93.29, 93.41, 93.20 | 93.76
CIFAR100 | 40, 41 | 66.08, 65.19, 66.21, 65.22 | 70.24 | 8.28 | 66.73, 66.47, 67.23, 66.33 | 68.89
| 41, 42 | 66.21, 65.22, 65.89, 65.04 | 70.50 | 8.02 | 67.23, 66.33, 66.77, 65.91 | 68.70
| 42, 43 | 65.89, 65.04, 65.85, 66.17 | 70.46 | 7.75 | 66.77, 65.91, 67.12, 66.35 | 68.77
| 43, 44 | 65.85, 66.17, 65.61, 65.03 | 70.16 | 8.51 | 67.12, 66.35, 66.51, 65.90 | 69.08
| 44, 45 | 65.61, 65.03, 66.27, 64.92 | 69.98 | 9.77 | 66.51, 65.90, 66.24, 65.29 | 69.14
| Avg | 65.93, 65.33, 65.97, 65.28 | 70.27 | 8.47 | 66.87, 66.19, 66.77, 65.96 | 68.92
Tiny-ImageNet | 40, 41 | 54.29, 53.13, 54.81, 53.74 | 61.28 | 3.24 | 54.50, 53.71, 55.20, 53.17 | 55.67
| 41, 42 | 54.81, 53.74, 59.44, 55.37 | 61.47 | 4.70 | 55.20, 53.17, 55.87, 54.26 | 56.35
| 42, 43 | 55.37, 53.74, 54.83, 53.62 | 61.50 | 3.11 | 55.87, 54.26, 55.27, 53.91 | 56.14
| 43, 44 | 54.83, 53.62, 54.91, 53.28 | 61.23 | 4.13 | 55.27, 53.91, 54.99, 54.02 | 56.32
| 44, 45 | 54.91, 53.28, 54.65, 52.81 | 60.92 | 5.06 | 54.99, 54.02, 55.11, 53.44 | 55.86
| Avg | 54.84, 53.29, 54.91, 53.23 | 61.28 | 4.05 | 55.17, 53.81, 55.29, 53.76 | 56.07

###  D.2 Generate an efficient initialization

Results. Finetuning from the fused model always improves the accuracy over
finetuning from pre-trained models as illustrated in Table 14. In comparison
with other initializations, it yields the best performance in all 5 seeds on
both datasets. $M_{B}$ depth-aligned, which is the result of NET2NET [4]
operation, performs better than finetuning from scratch but only leads to
comparable performance as finetuning from $M_{B}$.

Pre-trained models for multiple model cases. For 4 and 6 models, we train two
and three pairs of RESNET34 and RESNET18 models, respectively. To utilize pre-
trained models in the two model scenario, we simply group multiple pre-trained
model pairs at different seeds. Specifically, in Table 15, the first pair (the
first and second pre-trained models) was trained at seed 40 and the second
pair (the third and fourth pre-trained models) was trained at seed 41. In all
pairs, the RESNET34 model is always put before the RESNET18 model. Their
accuracies can be found in Table 14 except for seeds 45 and 46, which are
newly trained for this setting.

CLAFusion for more than 2 models. Tables 15 and 16 detail the results in the
multiple-model scenario. Compared to the two-model scenario, fusing 4 RESNET
results in the initialization with a significant decrease in the accuracy due
to the increasing difficulty of the task. However, there is an increase in the
final performance after finetuning, with a figure growing from $93.67$ to
$93.76$ on the CIFAR10 dataset, from $67.59$ to $68.92$ on the CIFAR100
dataset, and from $55.46$ to $56.07$ on the Tiny-ImageNet dataset. In
addition, CLAFusion produces the best initialization in all 5 runs. For the
6-model case, CLAFusion also yields more favorable initialization in every run
on CIFAR10, CIFAR100, and Tiny-ImageNet datasets. It is worth mentioning that
increasing the number of pre-trained models increase the performance of
ensemble learning and the fused model after finetuning. This outcome is
expected since both methods combine knowledge from pre-trained models that
were trained at different random seeds.

Table 16: The results of fusing and finetuning 6 RESNET from different choices
of initialization across 5 different seeds. The best performing initialization
in each row is in bold.

Dataset | Seeds | Pre-trained models | Ensemble | $M_{\mathcal{F}}$ | Finetune
---|---|---|---|---|---
|  | (RESNET34, RESNET18) x3 | Learning |  | Pre-trained models | $M_{\mathcal{F}}$
CIFAR10 | 40, 41, 42 | 93.42, 93.21, 93.37, 92.90, 93.31, 93.05 | 94.28 | 26.32 | 93.50, 93.47, 93.61, 93.26, 93.61, 93.36 | 93.84
| 41, 42, 43 | 93.37, 92.90, 93.31, 93.05, 93.39, 92.78 | 94.19 | 24.22 | 93.61, 93.26, 93.61, 93.36, 93.50, 93.30 | 93.84
| 42, 43, 44 | 93.31, 93.05, 93.39, 92.78, 93.06, 92.65 | 94.13 | 24.37 | 93.61, 93.36, 93.50, 93.30, 93.37, 93.05 | 93.63
| 43, 44, 45 | 93.39, 92.78, 93.06, 92.65, 92.66, 92.76 | 94.14 | 23.78 | 93.50, 93.30, 93.37, 93.05, 92.94, 93.04 | 93.69
| 44, 45, 46 | 93.06, 92.65, 92.66, 92.76, 93.46, 93.05 | 94.36 | 20.08 | 93.37, 93.05, 92.94, 93.04, 93.41, 93.34 | 93.89
| Avg | 93.31, 92.92, 93.16, 92.83, 93.18, 92.86 | 94.22 | 23.75 | 93.52, 93.29, 93.41, 93.20, 93.49, 93.24 | 93.78
CIFAR100 | 40, 41, 42 | 66.08, 65.19, 66.21, 65.22, 65.89, 65.04 | 70.47 | 4.14 | 66.73, 66.47, 67.23, 66.33, 66.77, 65.91 | 68.80
| 41, 42, 43 | 66.21, 65.22, 65.89, 65.04, 65.85, 66.17 | 70.83 | 4.63 | 67.23, 66.33, 66.77, 65.91, 67.12, 66.35 | 69.34
| 42, 43, 44 | 65.89, 65.04, 65.85, 66.17, 65.61, 65.03 | 70.66 | 4.22 | 66.77, 65.91, 67.12, 66.35, 66.51, 65.90 | 69.72
| 43, 44, 45 | 65.85, 66.17, 65.61, 65.03, 66.27, 64.92 | 70.53 | 4.87 | 67.12, 66.35, 66.51, 65.90, 66.24, 65.29 | 69.56
| 44, 45, 46 | 65.61, 65.03, 66.27, 64.92, 65.82, 65.34 | 70.63 | 4.09 | 66.51, 65.90, 66.24, 65.29, 66.18, 66.16 | 69.57
| Avg | 65.93, 65.33, 65.97, 65.28, 65.89, 65.30 | 70.62 | 4.39 | 66.87, 66.19, 66.77, 65.96, 66.56, 65.92 | 69.40
Tiny-ImageNet | 40, 41, 42 | 54.29, 53.13, 54.81, 53.74, 59.44, 55.37 | 62.51 | 2.12 | 54.50, 53.71, 55.20, 53.17, 55.87, 54.26 | 56.15
| 41, 42, 43 | 54.81, 53.74, 59.44, 55.37, 54.83, 53.62 | 62.48 | 3.22 | 55.20, 53.17, 55.87, 54.26, 55.27, 53.91 | 57.05
| 42, 43, 44 | 55.37, 53.74, 54.83, 53.62, 54.91, 53.28 | 62.47 | 1.77 | 55.87, 54.26, 55.27, 53.91, 54.99, 54.02 | 56.51
| 43, 44, 45 | 54.83, 53.62, 54.91, 53.28, 54.65, 52.81 | 62.29 | 2.36 | 55.27, 53.91, 54.99, 54.02, 55.11, 53.44 | 56.44
| 44, 45, 46 | 54.91, 53.28, 54.65, 52.81, 53.39, 53.92 | 61.87 | 2.87 | 54.99, 54.02, 55.11, 53.44, 54.05, 54.55 | 56.21
| Avg | 54.84, 53.29, 54.91, 53.23, 54.63, 53.47 | 62.32 | 2.47 | 55.17, 53.81, 55.29, 53.76, 55.06, 54.04 | 56.47

Table 17: The results of model transfer from $M_{B}$ (RESNET18) to $M_{A}$
(RESNET34) across 5 different seeds. The best performing method in each row is
in bold.

Dataset | Method | Seed | HMT | HMT + Naive avg | HMT + OTFusion | HMT + CLA + OTFusion
---|---|---|---|---|---|---
CIFAR10 | Transfer | 40 | 9.55 | 37.80 | 20.17 | 62.94
|  | 41 | 10.13 | 37.66 | 17.66 | 62.27
|  | 42 | 10.45 | 43.44 | 19.25 | 60.86
|  | 43 | 12.44 | 39.24 | 15.20 | 65.74
|  | 44 | 11.37 | 29.33 | 19.30 | 54.63
|  | Avg | 10.79 | 37.49 | 18.32 | 61.29
| Transfer + Finetune | 40 | 93.29 | 93.63 | 93.61 | 93.75
|  | 41 | 93.14 | 93.28 | 93.69 | 94.09
|  | 42 | 93.08 | 93.16 | 93.78 | 93.68
|  | 43 | 93.20 | 93.27 | 93.80 | 93.92
|  | 44 | 92.77 | 93.58 | 93.41 | 93.84
|  | Avg | 93.10 | 93.38 | 93.66 | 93.86
CIFAR100 | Transfer | 40 | 2.98 | 9.69 | 7.32 | 12.69
|  | 41 | 2.33 | 10.92 | 4.38 | 14.49
|  | 42 | 3.03 | 12.90 | 4.13 | 7.33
|  | 43 | 3.17 | 11.44 | 5.65 | 15.35
|  | 44 | 2.00 | 9.00 | 4.22 | 12.92
|  | Avg | 2.70 | 10.79 | 5.14 | 12.56
| Transfer + Finetune | 40 | 64.78 | 65.96 | 67.10 | 67.74
|  | 41 | 65.25 | 66.39 | 67.37 | 67.95
|  | 42 | 65.15 | 66.56 | 67.22 | 67.51
|  | 43 | 65.53 | 66.71 | 67.44 | 68.13
|  | 44 | 65.06 | 66.46 | 67.00 | 67.80
|  | Avg | 65.15 | 66.42 | 67.23 | 67.83

###  D.3 Heterogeneous model transfer

Settings. The finetuning hyperparameters are similar to those for finetuning
RESNET models in Appendix D.2. However, we observe that retraining at the
initial learning rate of 0.1 in some cases causes the target model after
transferring to diverge. Hence, we additionally retrain at the initial
learning rate of 0.05 and report the best performing one between two initial
learning rates.

Results. Table 17 details the results of 4 different model transfer methods.
On the CIFAR10 dataset, combining CLA and OTFusion with HMT always results in
a great boost, at least $16.27$ (seed 42), in the accuracy of the target model
after transferring. After the finetuning phase, it gives the best performance
in 4 out of 5 random seeds. In terms of the CIFAR100 dataset, incorporating
CLAFusion into HMT leads to the best performance in 5 out of 5 runs.

###  D.4 Teacher-student fusion

Settings. We conduct an additional setting, which is fusing teacher and
student RESNET models on the CIFAR100 dataset. Teacher model A has RESNET34
architecture while model B has RESNET18 architecture with a half number of
channels at each layer. The architectures of $M_{A},M_{B},$ and
$M_{\mathcal{F}}$, which are in turn RESNET34, RESNET18 half, and RESNET34
half, can be represented as follows.

_RESNET34_ : Input layer
$\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow{\color[rgb]{1,0,0}conv1-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow{\color[rgb]{1,0,0}conv1-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-512}\rightarrow\text{conv3-512}\rightarrow{\color[rgb]{1,0,0}conv1-512}\rightarrow\text{conv3-512}\rightarrow\text{conv3-512}\rightarrow\text{conv3-512}\rightarrow\text{conv3-512}\rightarrow$
Output layer

_RESNET18 half_ : Input layer
$\rightarrow\text{conv3-32}\rightarrow\text{conv3-32}\rightarrow\text{conv3-32}\rightarrow\text{conv3-32}\rightarrow\text{conv3-32}\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow{\color[rgb]{1,0,0}conv1-64}\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow{\color[rgb]{1,0,0}conv1-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow{\color[rgb]{1,0,0}conv1-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow$
Output layer

_RESNET34 half_ : Input layer
$\rightarrow\text{conv3-32}\rightarrow\text{conv3-32}\rightarrow\text{conv3-32}\rightarrow\text{conv3-32}\rightarrow\text{conv3-32}\rightarrow\text{conv3-32}\rightarrow\text{conv3-32}\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow{\color[rgb]{1,0,0}conv1-64}\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\\\
\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow\text{conv3-64}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow{\color[rgb]{1,0,0}conv1-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-128}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow{\color[rgb]{1,0,0}conv1-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow\text{conv3-256}\rightarrow$
Output layer

Other settings for running CLAFusion and finetuning are similar to experiments
in Section 4.1.

Table 18: Finetuning teacher-student models across 5 different seeds. The best
performing student model in each row is in bold. The results of VGG reported
in Table 4 are run with the random seed of 42.

Dataset + | Seed | $M_{A}$ | $M_{B}$ | Ensemble | $M_{\mathcal{F}}$ | Retrain
---|---|---|---|---|---|---
Architecture |  |  |  | Learning |  | $M_{A}$ | $M_{B}$ | $M_{\mathcal{F}}$ |  $M_{B}$ depth-aligned
CIFAR10 + | 40 | 92.71 | 89.68 | 92.53 | 81.66 | 92.67 | 89.81 | 90.64 | 90.54
VGG | 41 | 92.34 | 89.61 | 92.68 | 82.59 | 92.50 | 89.57 | 90.59 | 90.42
| 42 | 92.70 | 89.92 | 92.86 | 82.66 | 92.65 | 89.89 | 90.96 | 90.84
| 43 | 92.79 | 89.78 | 92.82 | 81.79 | 92.77 | 90.10 | 90.55 | 90.61
| 44 | 92.70 | 89.62 | 92.51 | 81.43 | 92.56 | 89.46 | 90.44 | 90.19
| Avg | 92.65 | 89.72 | 92.68 | 82.03 | 92.63 | 89.77 | 90.64 | 90.52
CIFAR100 + | 40 | 66.08 | 61.68 | 67.14 | 28.72 | 66.73 | 63.10 | 64.89 | 63.54
RESNET | 41 | 66.21 | 61.57 | 67.11 | 27.98 | 67.23 | 62.95 | 64.59 | 62.94
| 42 | 65.89 | 61.36 | 67.53 | 29.31 | 66.77 | 62.97 | 65.10 | 62.48
| 43 | 65.85 | 61.56 | 67.14 | 28.03 | 67.12 | 62.23 | 63.83 | 62.96
| 44 | 65.61 | 61.46 | 66.94 | 28.30 | 66.51 | 62.59 | 63.85 | 62.84
| Avg | 65.93 | 61.53 | 67.17 | 28.47 | 66.87 | 62.77 | 64.45 | 62.95

Results. Table 18 illustrates the finetuning result of teacher and student
models across $5$ different seeds. The fused model $M_{\mathcal{F}}$ is the
most productive initialization of student models in at least 4 out of 5 seeds.
Retraining the fused model ($M_{\mathcal{F}}$) always yields higher accuracy
than continuing training the student model ($M_{B}$), leading to average
improvements of 0.87 and 1.68 on CIFAR10 and CIFAR100 datasets, respectively.

Discussion. At first glance, the first two steps of our approach may resemble
the NET2NET operations. Although both approaches increase the depth of the
student network and use the deeper student as a good initialization, there are
two major differences between ours and NET2NET. Firstly, we present a
systematic way of finding the location to add the identity mappings for
different types of network architectures while NET2NET is manually designed
for a specific type of network. Secondly, we do not enlarge the width of the
student network (equivalently, use only NET2DEEPERNET operation) so that the
student network remains more compact than the teacher network.

Table 19: Distillation results for different temperatures. For each type of
architecture, the entity in the first five rows is the best accuracy obtained
by varying the loss-weight factor while the last two rows report the best and
average performance across 5 different temperatures. The best performing
initialization in each row is in bold.

Dataset + | Temperature | Distillation initialization
---|---|---
Architecture | T | Random $M_{B}$ | $M_{B}$ | $M_{\mathcal{F}}$ |  $M_{B}$ depth-aligned | Random $M_{\mathcal{F}}$
CIFAR10 + | 20 | 88.99 (0.70) | 90.94 (0.70) |  91.29 (0.10) | 91.14 (0.70) | 88.20 (0.10)
VGG | 10 | 89.10 (0.70) | 90.97 (0.70) |  91.23 (0.50) |  91.23 (0.10) | 88.84 (0.50)
| 8 | 88.91 (0.70) | 90.98 (0.70) |  91.43 (0.70) | 91.24 (0.70) | 88.76 (0.50)
| 4 | 87.89 (0.70) | 90.81 (0.99) |  91.14 (0.50) | 91.10 (0.50) | 87.65 (0.05)
| 1 | 87.14 (0.10) | 89.97 (0.10) | 90.69 (0.05) |  90.73 (0.05) | 87.04 (0.05)
| Best | 89.10 | 90.98 | 91.43 | 91.24 | 88.84
| Avg | 88.41 | 90.73 | 91.16 | 91.09 | 88.10
CIFAR100 + | 20 | 63.19 (0.50) | 65.87 (0.95) |  66.56 (0.70) | 66.04 (0.70) | 64.37 (0.50)
RESNET | 10 | 63.99 (0.50) | 66.20 (0.95) |  66.79 (0.70) | 66.31 (0.99) | 64.84 (0.95)
| 8 | 63.01 (0.50) | 66.07 (0.99) |  66.50 (0.70) | 66.08 (0.99) | 64.63 (0.70)
| 4 | 62.41 (0.50) | 64.54 (0.50) |  65.26 (0.70) | 64.52 (0.50) | 62.79 (0.50)
| 1 | 61.64 (0.10) | 63.60 (0.10) |  64.43 (0.05) | 63.94 (0.50) | 62.01 (0.05)
| Best | 63.99 | 66.20 | 66.79 | 66.31 | 64.84
| Avg | 62.85 | 65.26 | 65.92 | 65.38 | 63.73

###  D.5 Knowledge distillation

In this experiment, we apply knowledge distillation to pre-trained models of
seed 42 in Table 18. For a fair comparison, the hyperparameters for training
student models are identical to finetune hyperparameters in Section 4.2.

Results. The results of distilling the teacher model into the student model
are given in Table 19. Knowledge distillation from the fused model leads to
the best accuracy in no fewer than 4 out of 5 temperatures. In addition, both
its average and best performance are the highest among all initializations.
This suggests that CLAFusion can also be used to generate an efficient
initialization for pre-trained distillation. Moving onto other initialization,
$M_{B}$ depth-aligned, which is obtained from the NET2NET operation, is the
second-best choice, followed by the pre-trained student model $M_{B}$.
Distilling knowledge into either randomly initialized student models, however,
yields a much lower accuracy. This result is consistent with the findings in
Turc et al. [32].

Table 20: The number of parameters and inference time on the CIFAR10 dataset
for different experiments. The inference time is the time taken in seconds to
run inference on the whole test dataset.

Experiment | Architecture | # Params | Inference time
---|---|---|---
| Pre-trained models | $M_{\mathcal{F}}$ | Pre-trained models | $M_{\mathcal{F}}$ | Ensemble | Pre-trained models | $M_{\mathcal{F}}$ | Ensemble
Fuse 2 RESNET | RESNET34, RESNET18 | RESNE34 | 21M, 11M | 21M | 32M | 1.78, 1.08 | 1.78 | 2.72
Fuse 4 RESNET | (RESNET34, RESNET18) x2 | RESNE34 | (21M, 11M) x2 | 21M | 64M | (1.78, 1.08) x2 | 1.78 | 5.30
Fuse 6 RESNET | (RESNET34, RESNET18) x3 | RESNE34 | (21M, 11M) x3 | 21M | 96M | (1.78, 1.08) x3 | 1.78 | 7.88

###  D.6 Computational efficiency of model fusion

In this section, we demonstrate that CLAFusion enhances the computational
efficiency substantially over ensemble learning and knowledge distillation in
all experiments.

Table 20 illustrates the model size and inference time on the CIFAR10 test
dataset of all models in Section 4.1. As discussed in Section 1, ensemble
learning is computationally expensive because it stores and runs all pre-
trained models during the inference stage. As the number of pre-trained models
increases, memory consumption and inference time of ensemble learning grow up
dramatically. In particular, the number of parameters multiplies by the same
factor as the number of pre-trained models, increasing from 32M to 96M. Though
rising at a lower speed, the inference time still increases by almost $190\%$,
from 2.72 to 7.88. On the other hands, the fused model produced by CLAFusion
remains compact no matter how many pre-trained models are fused.

Table 21: Training time for finetuning and knowledge distillation in the
teacher-student setting. The training time is the time taken in seconds to run
one epoch of training on the whole train dataset.

Experiment | Architecture | # Params | Finetuning | Distillation
---|---|---|---|---
| $M_{A}$ | $M_{B}$ | $M_{\mathcal{F}}$ | $M_{A}$ | $M_{B}$ | $M_{\mathcal{F}}$ | $M_{A}$ | $M_{B}$ | $M_{\mathcal{F}}$ | $M_{B}$ | $M_{\mathcal{F}}$
Teacher-student VGG | VGG13 doub | VGG11 half | VGG13 stud | 33M | 3M | 3M | 11.14 | 7.40 | 7.48 | 12.22 | 13.86
Teacher-student RESNET | RESNET34 | RESNET18 half | RESNET34 half | 21M | 3M | 5M | 31.73 | 7.50 | 12.05 | 36.39 | 41.36

Table 21 compares the training time between finetuning and knowledge
distillation in the teacher-student setting. It is undoubtedly that knowledge
distillation runs much more slowly than finetuning, causing a rise of at least
$65\%$ if using the same VGG initialization. In terms of RESNET architecture,
the speedup is even more significant, with increases of $385\%$ and $243\%$
for $M_{B}$ and $M_{\mathcal{F}}$, respectively. The convincing reason is that
in knowledge distillation the input data is fed in the forward direction
through both teacher and student models while finetuning only runs forward
inference for the student model whose size is much smaller than that of the
teacher model.

## References

  * [1] S. Ashmore and M. Gashler.  A method for finding similarity between multi-layer perceptrons by forward bipartite alignment.  In 2015 International Joint Conference on Neural Networks (IJCNN), pages 1–7. IEEE, 2015.
  * [2] N. Bonneel and D. Coeurjolly.  Sliced partial optimal transport.  ACM Transactions on Graphics (Proceedings of SIGGRAPH), 38(4), Jul 2019.
  * [3] D. Chen, J. Mei, Y. Zhang, C. Wang, Z. Wang, Y. Feng, and C. Chen.  Cross-layer distillation with semantic calibration.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7028–7036, 2021.
  * [4] T. Chen, I. Goodfellow, and J. Shlens.  Net2net: Accelerating learning via knowledge transfer.  In 4th International Conference on Learning Representations, 2016\.
  * [5] Y.-M. Chou, Y.-M. Chan, J.-H. Lee, C.-Y. Chiu, and C.-S. Chen.  Unifying and merging well-trained deep neural networks for inference stage.  In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 2049–2056. AAAI Press, 2018.
  * [6] P. Chrabaszcz, I. Loshchilov, and F. Hutter.  A downsampled variant of imagenet as an alternative to the cifar datasets.  arXiv preprint arXiv:1707.08819, 2017.
  * [7] M. Cuturi and A. Doucet.  Fast computation of Wasserstein barycenters.  In International conference on machine learning, pages 685–693. PMLR, 2014.
  * [8] A. Figalli.  The optimal partial transport problem.  Archive for rational mechanics and analysis, 195(2):533–560, 2010\.
  * [9] R. Flamary, N. Courty, A. Gramfort, M. Z. Alaya, A. Boisbunon, S. Chambon, L. Chapel, A. Corenflos, K. Fatras, N. Fournier, L. Gautheron, N. T. Gayraud, H. Janati, A. Rakotomamonjy, I. Redko, A. Rolet, A. Schutz, V. Seguy, D. J. Sutherland, R. Tavenard, A. Tong, and T. Vayer.  Pot: Python optimal transport.  Journal of Machine Learning Research, 22(78):1–8, 2021.
  * [10] J. Gou, B. Yu, S. J. Maybank, and D. Tao.  Knowledge distillation: A survey.  International Journal of Computer Vision, Mar 2021.
  * [11] L. K. Hansen and P. Salamon.  Neural network ensembles.  IEEE transactions on pattern analysis and machine intelligence, 12(10):993–1001, 1990.
  * [12] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [13] G. Hinton, O. Vinyals, and J. Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [14] S. Hochreiter and J. Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [15] G. Huang and A. Lim.  A hybrid genetic algorithm for three-index assignment problem.  In The 2003 Congress on Evolutionary Computation, 2003. CEC’03., volume 4, pages 2762–2768. IEEE, 2003.
  * [16] L. V. Kantorovich.  On the translocation of masses.  Journal of mathematical sciences, 133(4):1381–1382, 2006.
  * [17] J. Kitagawa and B. Pass.  The multi-marginal optimal partial transport problem.  In Forum of Mathematics, Sigma, volume 3. Cambridge University Press, 2015.
  * [18] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton.  Similarity of neural network representations revisited.  In International Conference on Machine Learning, pages 3519–3529. PMLR, 2019.
  * [19] A. Krizhevsky.  Learning multiple layers of features from tiny images.  Technical report, University of Toronto, Toronto, Ontario, 2009.
  * [20] H. W. Kuhn.  The Hungarian method for the assignment problem.  Naval research logistics quarterly, 2(1-2):83–97, 1955.
  * [21] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [22] M. I. Leontev, V. Islenteva, and S. V. Sukhov.  Non-iterative knowledge fusion in deep convolutional neural networks.  Neural Processing Letters, 51(1):1–22, Jul 2019.
  * [23] Y. Li, J. Yosinski, J. Clune, H. Lipson, and J. E. Hopcroft.  Convergent learning: Do different neural networks learn the same representations?  In Workshop on Feature Extraction: Modern Questions and Challenges, pages 196–212, 2015.
  * [24] G. Monge.  Mémoire sur la théorie des déblais et des remblais.  Histoire de l’Académie Royale des Sciences de Paris, 1781.
  * [25] J. O’Neill, G. V. Steeg, and A. Galstyan.  Layer-wise neural network compression via layer fusion.  In Asian Conference on Machine Learning, pages 1381–1396. PMLR, 2021.
  * [26] L. Ramshaw and R. E. Tarjan.  On minimum-cost assignments in unbalanced bipartite graphs.  HP Labs, Palo Alto, CA, USA, Tech. Rep. HPL-2012-40R1, 2012.
  * [27] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  In 3rd International Conference on Learning Representations, 2015\.
  * [28] S. P. Singh and M. Jaggi.  Model fusion via optimal transport.  Advances in Neural Information Processing Systems, 33, 2020.
  * [29] J. Smith and M. Gashler.  An investigation of how neural networks learn from the experiences of peers through periodic weight averaging.  In 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 731–736. IEEE, 2017.
  * [30] F. C. Spieksma.  Multi index assignment problems: complexity, approximation, applications.  In Nonlinear assignment problems, pages 1–12. Springer, 2000.
  * [31] F. Tung and G. Mori.  Similarity-preserving knowledge distillation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1365–1374, 2019.
  * [32] I. Turc, M.-W. Chang, K. Lee, and K. Toutanova.  Well-read students learn better: On the importance of pre-training compact models.  arXiv preprint arXiv:1908.08962, 2019.
  * [33] J. Utans.  Weight averaging for neural networks and local resampling schemes.  In Proc. AAAI-96 Workshop on Integrating Multiple Learned Models. AAAI Press, pages 133–138. Citeseer, 1996.
  * [34] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [35] G. Wang, J. Lai, W. Liang, and G. Wang.  Heterogeneous model transfer between different neural networks.  Submission at International Conference on Learning Representations, 2021.
  * [36] H. Wang, M. Yurochkin, Y. Sun, D. Papailiopoulos, and Y. Khazaeni.  Federated learning with matched averaging.  In International Conference on Learning Representations, 2020.
  * [37] T. Wei, C. Wang, Y. Rui, and C. W. Chen.  Network morphism.  In International Conference on Machine Learning, pages 564–572. PMLR, 2016.
  * [38] S. Zagoruyko and N. Komodakis.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  In International Conference on Learning Representations, 2017.
  * [39] M. D. Zeiler and R. Fergus.  Visualizing and understanding convolutional networks.  In European conference on computer vision, pages 818–833. Springer, 2014.
  * [40] C. Zhang, S. Bengio, and Y. Singer.  Are all layers created equal?  In Workshop on Identifying and Understanding Deep Learning Phenomena, 2019.
  * [41] Z.-H. Zhou.  Ensemble methods: Foundations and algorithms.  CRC press, 2012.