11institutetext:  Pattern Recognition and Intelligent Systems Lab., Beijing
University of Posts and Telecommunications, China. 11email:
<EMAIL_ADDRESS>22institutetext: The University of
Sydney, Australia. 22email<EMAIL_ADDRESS>33institutetext:
SenseTime Research Centre, China. 33email<EMAIL_ADDRESS>44institutetext: University of Science and Technology of China. 44email:
<EMAIL_ADDRESS>
# ScaleNet: Searching for the Model to Scale††thanks: This work was supported
in part by National Natural Science Foundation of China (NSFC) No. 61922015,
U19B2036, and in part by Beijing Natural Science Foundation Project No.
Z200002.

Jiyang Xie 11 Xiu Su 22 Shan You 33 Zhanyu Ma 11 Fei Wang 44
Chen Qian 33

###### Abstract

Recently, community has paid increasing attention on model scaling and
contributed to developing a model family with a wide spectrum of scales.
Current methods either simply resort to a one-shot NAS manner to construct a
non-structural and non-scalable model family or rely on a manual yet fixed
scaling strategy to scale an unnecessarily best base model. In this paper, we
bridge both two components and propose ScaleNet to jointly search base model
and scaling strategy so that the scaled large model can have more promising
performance. Concretely, we design a super-supernet to embody models with
different spectrum of sizes (_e.g._ , FLOPs). Then, the scaling strategy can
be learned interactively with the base model via a Markov chain-based
evolution algorithm and generalized to develop even larger models. To obtain a
decent super-supernet, we design a hierarchical sampling strategy to enhance
its training sufficiency and alleviate the disturbance. Experimental results
show our scaled networks enjoy significant performance superiority on various
FLOPs, but with at least $2.53\times$ reduction on search cost. Codes are
available at https://github.com/luminolx/ScaleNet.

###### Keywords:

Neural architecture search (NAS), model scaling, hierarchical sampling
strategy, Markov chain-based evolution algorithm

##  1 Introduction

Convolutional neural networks (CNNs) have achieved great performance in
computer vision with various model architectures [6, 7, 12, 15, 16, 30, 43,
44, 45, 46, 49, 51] proposed for better feature extraction abilities. Previous
work [31, 32, 33, 34, 48] usually focused on how to automatically design a
model architecture under a certain resource budget (_e.g._ , floating-point
operations per second, FLOPs) with neural architecture search (NAS) algorithms
and gained significant improvements. However, due to different levels of
budgets which may occur in various applications, multi-scale architectures
should be considered in practice and can be independently generated by the
NAS. Nevertheless, typical NAS methods [31, 32, 33, 34, 48] have to search one
a time for each scale, and the searching cost will be approximated linearly
scaled as well [38] (see baseline in Figure 1).

In contrast, recent work [4, 9, 11, 14, 22, 23, 24, 37, 40, 42, 50, 53] get
down to paying attention on model scale and designing a model family in a more
straightforward way. Two frameworks have been proposed as shown in Figure 1,
including one-shot NAS-based pipeline (_e.g._ , BigNAS [50] and Once-for-All
(OFA) [4]) and two-step pipeline (_e.g._ , EfficientNet [37] and
EfficientNet-X [22]). The former directly designed an overcomplete one-shot
supernet to embody multiple (finite) scales and searched models by NAS.
However, they are difficult to extend the searched models to a larger one,
since finding a specific scaling strategy that adapts all the non-structurally
searched architectures is infeasible. The latter decomposed the large model
generation with two steps,  _i.e._ , first acquiring an optimal base model,
then scaling it on three dimensions, including depth, width, and resolution,
using some pre-defined strategies,  _e.g._ , compound scaling [37] and fast
compound scaling [11]. However, the best base model is unnecessarily optimal
for scaling. How to combine the advantages of both,  _i.e._ , automatically
and jointly searching the base model and scaling strategy by NAS and freely
extending the scaling strategy into infinite scales, should be carefully
considered.

Figure 1: Comparisons of different methods to generate a model family
S$0$-S$3$. Red, blue, and green arrows are the scaling strategy searching for
scaling stage $1$, $2$, and $3$, respectively. The pure one-shot NAS
(baseline) independently searched various models without either scaling
strategy modeling between scaling stages or larger-scale architecture
generalization. BigNAS [50] jointly searched the model family from an entire
supernet, although it obtained non-structural then non-scalable architectures.
Then in EfficientNet [37], the base model (_i.e._ , S$0$) and the scaling
strategy to S$1$ were only searched independently for architecture
generalizing. In our ScaleNet, we combine the two components and jointly
search base model and all scaling strategies to scale the model into infinite
ones.

Different with manually designed rule-based scaling strategies [11, 37, 47],
we propose to directly discover the optimal scaling strategies within base
model search. One-shot NAS can search model architectures based on a trained
supernet that contains all the possible architectures (so-called paths). For
improving search efficiency, we apply an even-larger supernet dubbed super-
supernet to embody multi-scale networks. However, a common one-shot space
usually has a uni-modal distribution of FLOPs of paths under uniform sampling
[13, 50]. In this way, the super-supernet tends to favor the intermediate-
FLOPs and cannot accommodate all FLOPs budgets well, which will in turn
hampers the optimality of searched scaling strategies. Inspired by ancestral
sampling [2], we propose a hierarchical sampling strategy (HSS) that splits
the search space into partitions and the sampling is implemented respectively.
The search space and the sampling distribution of the super-supernet for FLOPs
are carefully designed according to the budgets of various scaling stages and
undertake a multi-modal form distribution.

Secondly, considering that our goal is to find a base model architecture with
the strongest scaling capability (instead of the best performance) and its
corresponding optimal scaling strategies, we propose a joint search for them,
dubbed Markov chain-based evolution algorithm (MCEA), by iteratively and
interactively optimizing both of them. After obtaining the searched scaling
strategies, we model the trends of depth, width, and resolution, respectively,
and generalize them to develop even larger models. We theoretically derive a
group of generalization functions in the three dimensions for larger-scale
architectures, with which moderate performance can be actually achieved.

The contributions of this paper are four-fold:

  * •

We propose ScaleNet to jointly search the base model and a group of the
scaling strategies based on one-shot NAS framework. The scaling strategies of
larger scales are generalized by the searched ones with our theoretically
derived generalization functions.

  * •

We carefully design the search space and a multi-modal distribution for FLOPs
budgets for hierarchical sampling strategy (HSS) in the one-shot NAS-based
scaling search algorithm to enhance the training sufficiency of paths in
super-supernet.

  * •

We propose a joint search algorithm for both the base model and the scaling
strategies, namely Markov chain-based evolution algorithm (MCEA), by
iteratively and interactively optimizing both of them.

  * •

Experimental results show that the searched architectures by the proposed
ScaleNet with various FLOPs budgets can outperform the referred methods on
various datasets including ImageNet-$1$k. Meanwhile, search time can be
significantly reduced at least $2.53\times$.

Figure 2: Framework of the proposed ScaleNet. Based on the carefully designed
search space with multiple scaling stage (the left box), we apply the proposed
hierarchical sampling strategy (HSS) for sampling paths (one path is generated
by a base model and a scaling strategy) in one-shot super-supernet training
(the upper box). Then, we utilize the proposed Markov chain-based evolution
algorithm (MCEA) to iteratively and iteractively search the optimal base model
and scaling strategies (the lower box). In each iteration, an evolution
procedure with crossover-mutation and evaluation is undertaken for searching
the optimal base model or scaling strategies based on the search space.
Finally, after obtaining the optimal ones, we generalize them to larger-scale
architectures by the estimations of the trends of depth, width, and
resolution, respectively (the right box). All the obtained architectures will
be applied for retraining and inference.

##  2 Related Work

###  2.1 One-shot NAS-based Model Family Searching

FBNets [9, 40, 42] optimized CNN architectures for mobile devices and
generated a family of models in order to avoid training individual
architectures separately and reduce resource consuming. Cai  _et al_. [4]
proposed to train a once-for-all (OFA) model that supports diverse
architectural settings by decoupling training and search, in order to reduce
the cost. BigNAS [50] challenged the conventional pipeline that post-
processing of model weights is necessary to achieve good performance and
constructed big single-stage models without extra retraining or post-
processing. However, the main drawback of these methods is that they only
searched for a model family by training a joint or even a group of independent
supernet(s), but did not analyze the structural relationship and explicit
scaling strategies between the architectures with different budgets in the
model family. It is difficult and even infeasible to extend the scaling
strategies to larger scales.

###  2.2 Model Scaling

Tan and Le [37] systematically studied model scaling and found that carefully
balancing depth, width, and resolution of a model can lead to better
performance. They proposed to empirically obtain the optimal compound scaling
that effectively scales a specific base model up to gain a model family,
_i.e._ , EfficientNet. Its variant versions, such as EfficientNetV$2$ [38] and
EfficientNet-X [22], improved it in trade-off on speed and accuracy. A simple
fast compound scaling strategy [11] was proposed to encourage to primarily
scale model width, while scaling depth and resolution to a lesser extent for
memory efficiency. Another work [23] built an greedy network enlarging method
based on the reallocation of computations in order to enlarge the capacity of
CNNs by improving the three dimensions on stage level. However, the
aforementioned work always estimated the optimal scaling strategy for
theoretical double-FLOPs budget by a small grid search, which is
computationally expensive and does not match the actual FLOPs budgets.
Meanwhile, they only considered to find the strategy for the smallest scaling
stage, which did not learn the dependency between larger scaling stages.
Furthermore, the relation between a base model and scaling strategies did not
be investigated, which means the base is not the optimal for scaling.

##  3 ScaleNet

Due to the drawbacks of the one-shot NAS-based model family searching and
compound scaling-based model scaling, we propose to combine their advantages
together and fill the gap between them. Here, ScaleNet jointly searches a base
model with the strongest scaling capability and the optimal scaling strategies
based on one-shot NAS framework by training a super-supernet as shown in
Figure 2. The super-supernet training and joint searching procedures are
carefully designed for the goal. Then, when obtaining the searched scaling
strategies, we model the trends of depth, width, and resolution, respectively,
and generalize to develop even larger architectures. All the searched and
generalized scaling strategies will be applied for the final model family
construction and training.

###  3.1 One-shot Joint Search Space for Model Scaling

Figure 3: Sampling distribution based on the proposed HSS, compared with that
of the original uniform sampling in [50]. We took $750,000$ paths for each to
simulate actual super-supernet training.

The FLOPs budget of the base model is selected according to the mean FLOPs of
the search space as shown in Figure 3 (Detailed information of the search
space are shown in supplementary material). Then for various scaling stages,
their FLOPs budgets are exponentially expanded by that of the base model
$\alpha$. As scaling strategies
$\left\\{\bm{S}_{j}=[d_{j},w_{j},r_{j}]\right\\}_{j=0}^{M}$ with maximum
scaling stage as $M$, including the change ratios of depth $d_{j}$, width
$w_{j}$, and resolution $r_{j}$ (real numbers that are not smaller than one),
have their corresponding FLOPs budgets, respectively, we assign a scaling
strategy to each base model architecture to compute the mean FLOPs and find
the center point of the search space of scaling stage $j$ according to its
FLOPs budget. The detailed settings of the whole search space are elaborated
in the supplementary material.

###  3.2 Hierarchical Sampling Strategy for Super-supernet Training

Applying the original one-shot NAS framework [50, 13] by utilizing uniform
sampling to train the super-supernet means each operation has equal
probability to be selected. It has two disadvantages: $1$) the original bell-
shape sampling distribution towards one specific FLOPs budget for the whole
search space is not suitable for the multiple scaling stages with various
budgets, where paths cannot be fairly trained in each scaling stage (see the
red histogram in Figure 3); $2$) the search space is almost $300\times$ larger
and the size of super-supernet is $8\times$ larger (statistics with base model
and three scaling stages trained) than before, which increase the difficulty
of super-supernet training. As we need to search architectures in different
scaling stages, the paths with the less selected FLOPs budgets in super-
supernet training are not sufficiently trained and those in one scaling stage
are not fairly trained. Here, we propose a hierarchical sampling strategy
(HSS) by implementing a multi-modal sampling distribution to address the above
issues (see the blue histogram in Figure 3).

As we assigned $d_{j}$, $w_{j}$, and $r_{j}$ into scaling stages in the search
space, we treat the target multi-modal sampling distribution
$p(\bm{\alpha},\bm{S})$ as a mixture model,  _i.e._ ,

$p(\bm{\alpha},\bm{S})=p(\bm{\alpha})\cdot
p(\bm{S})=\begin{matrix}p(\bm{\alpha})\cdot\left(\sum_{j=0}^{M}\eta_{j}p_{j}(\bm{S})\right)\end{matrix},$
(1)

where $p(\bm{\alpha})$ and $p_{j}(\bm{S})$ are the sampling distributions of
base model $\bm{\alpha}$ and scaling stage $j$, respectively, and $\eta_{j}$
is normalized component weight, $\sum_{j=0}^{M}\eta_{j}=1$ and $\eta_{j}\geq
0$. Here, we can empirically set equal component weights,  _i.e._ ,
$\eta_{j}=\tfrac{1}{M+1}$, or normalized combination ratios of scaling
strategies in the search space.

In sampling, we apply the ancestral sampling of a probabilistic graphic model
[2] that is a two-step hierarchical strategy for the scaling strategies. We
firstly select a scaling stage $m$ given the conditional distribution
$p(m|\eta_{1},\cdots,\eta_{M})$, which is a categorical distribution as

$p(m=j|\eta_{1},\cdots,\eta_{M})=\eta_{j}.$ (2)

Then, we uniformly sample a scaling strategy $\bm{S}$ in scaling stage $m$.
Meanwhile, a base model $\bm{\alpha}$ is sampled as well based on the original
uniform sampling.

The one-shot super-supernet training process based on the proposed HSS is

$\bm{W}^{*}=\operatorname*{argmin}_{\bm{W}}loss_{\bm{\alpha},\bm{S}\sim
p(\bm{\alpha},\bm{S})}\left(\bm{W}(\bm{\alpha},\bm{S});\bm{D}_{train}\right),$
(3)

where $\bm{W}$ is a set of super-supernet parameters, $\bm{W}^{*}$ is a set of
the optimal parameters, $loss$ is training loss function (commonly cross-
entropy loss), $\bm{W}(\bm{\alpha},\bm{S})$ means a path that is constructed
by $\bm{\alpha}$ and $\bm{S}$, and $\bm{D}_{train}$ is training set.

###  3.3 Interactive Search for Base Model and Multiple Scaling Strategy

After completing the super-supernet training, we usually search both base
model and a group of scaling strategies by an evolution algorithm (EA). The
original objective of the optimization is globally maximizing the weighted sum
of the validation accuracies $ACC$ of the $M$ scaling stages [41]] as

$\displaystyle\begin{matrix}\max_{\bm{\alpha},\\{\bm{S}_{j}\\}}\left[\sum_{j=1}^{M}\pi_{j}ACC\left(\bm{W}^{*}(\bm{\alpha},\bm{S}_{j});\bm{D}_{val}\right)\right]\end{matrix},$
(4) s.t.
$\displaystyle\;\;\bm{\alpha}\in\bm{\Omega}_{\bm{\alpha}},\bm{S}_{j}\in\bm{\Omega}_{j},\text{FLOPs}(\bm{W}^{*}(\bm{\alpha},\bm{S}_{j}))=f_{j},$

where $\pi_{j}$ is the normalized weight of scaling stage $j$ (constrained by
$\sum_{j=1}^{M}\pi_{j}=1$), $ACC$ is the validation accuracy of a path,
$\bm{D}_{val}$ is validation set, $\bm{\Omega}_{\bm{\alpha}}$ and
$\bm{\Omega}_{j}$ are the search space of base model and scaling stage $j$,
respectively, and $f_{j}$ is the FLOPs budget of scaling stage $j$.

However, both the too large search space and constrained computational
resource restrict the search. Meanwhile, searching the globally optimal group
of architectures from the search space is difficult and expensive, since
redundant information and noises may affect the search.

Figure 4: The interactive search in the proposed Markov chain-based evolution
algorithm (MCEA) with a coupled Markov chain. The coupled Markov chain
combines both the left ((5)-(6)) and the right ((7)-(8)) ones for the
interaction.

Here, inspired by Markov process, we propose a so-called Markov chain-based
evolution algorithm (MCEA) with a coupled Markov chain, which iteratively and
interactively optimizes $\bm{\alpha}$ and $\\{\bm{S}_{j}\\}$, to overcome the
global search issue. As shown in Figure 4, with maximum iteration number $T$
and iteration index $t=1,\cdots,T$, we transfer the optimization problem in
(4) to iteratively and interactively solving the limiting distributions
$\gamma(\bm{\alpha})$ and $\gamma(\\{\bm{S}_{j}\\})$ to obtain the optimal
$\bm{\alpha}^{*}$ and $\bm{S}_{j}^{*}$, respectively, as

$\displaystyle\bm{\alpha}^{*}=$
$\displaystyle\operatorname*{argmax}_{\bm{\alpha}}\gamma(\bm{\alpha})=\begin{matrix}\operatorname*{argmax}_{\bm{\alpha}}\left[\lim_{t\to\infty}p(\bm{\alpha}^{(t)})\right]\end{matrix},$
(5) $\displaystyle p(\bm{\alpha}^{(t)})=$
$\displaystyle\begin{matrix}\sum_{\bm{\alpha}^{(t-1)}}p(\bm{\alpha}^{(t)}|\bm{\alpha}^{(t-1)},\\{\bm{S}_{j}\\}_{j=1}^{M}=\\{\bm{S}_{j}^{(t-1)}\\}_{j=1}^{M})p(\bm{\alpha}^{(t-1)})\end{matrix},$
(6) $\displaystyle\bm{S}_{j}^{*}=$
$\displaystyle\operatorname*{argmax}_{\bm{S}_{j}}\gamma(\bm{S}_{j})=\begin{matrix}\operatorname*{argmax}_{\bm{S}_{j}}\left[\lim_{t\to\infty}p(\bm{S}_{j}^{(t)})\right]\end{matrix},$
(7) $\displaystyle p(\bm{S}_{j}^{(t)})=$
$\displaystyle\begin{matrix}\sum_{\bm{S}_{j}^{(t-1)}}p(\bm{S}_{j}^{(t)}|\bm{S}_{j}^{(t-1)},\bm{\alpha}=\bm{\alpha}^{(t)})p(\bm{S}_{j}^{(t-1)})\end{matrix},$
(8)

where $p(\bm{\alpha}^{(t-1)})$ and $p(\bm{S}_{j}^{(t-1)})$ are the state
probabilities of the discrete variable $\bm{\alpha}^{(t-1)}$ and
$\bm{S}_{j}^{(t-1)}$, respectively, with the enumerable search space as state
space.
$p(\bm{\alpha}^{(t)}|\bm{\alpha}^{(t-1)},\\{\bm{S}_{j}\\}_{j=1}^{M}=\\{\bm{S}_{j}^{(t-1)}\\}_{j=1}^{M})\propto\sum_{j=1}^{M}\pi_{j}ACC\left(\bm{W}^{*}(\bm{\alpha},\bm{S}_{j}^{(t-1)});\bm{D}_{val}\right)$
and
$p(\bm{S}_{j}^{(t)}|\bm{S}_{j}^{(t)},\bm{\alpha}=\bm{\alpha}^{(t-1)})\propto
ACC\left(\bm{W}^{*}(\bm{\alpha}^{(t)},\bm{S}_{j});\bm{D}_{val}\right)$ are
transition matrices and are approximated implemented by the crossover-mutation
process under FLOPs budgets given the obtained scaling strategies or base
model, respectively. $\bm{S}_{j}^{(0)}$ is the initial scaling strategy of the
$j^{th}$ scaling stage, where probability
$p(\bm{S}_{j}^{(0)})\propto\tfrac{1}{K}\sum_{k=1}^{K}ACC\left(\bm{W}^{*}(\bm{\alpha}_{k}^{(0)},\bm{S}_{j});\bm{D}_{val}\right)$
is obtained by a group of randomly selected base models
$\\{\bm{\alpha}_{k}^{(0)}\\}_{k=1}^{K}$ under the FLOPs budget of base model.

###  3.4 Larger-scale Architecture Generalization with Searched Scaling
Strategies

The scaling strategies of larger scales are generalized by the searched ones.
We define the optimal scales of $M+1$ scaling stages as
$\\{\bm{\hat{S}}_{j}\\}_{j=0}^{M}$. We should note that we pre-define
$\bm{\hat{S}}_{0}$ as $\hat{d}_{0}=\hat{w}_{0}=\hat{r}_{0}=1$ for the base
model.

We argue that depth, width, and resolution should have distinct growth rates,
respectively, since they perform different roles in the model scaling.
Meanwhile, $j$ is exponentially proportional to FLOPs budgets in our setting,
but $\bm{\hat{S}}_{j},j=1,\cdots,M$ are under almost linear- or quadratic-
level. Therefore, inspired by [37], we propose to utilize the independent
regression functions for depth, width, and resolution for the larger-scale
generalization with

$\left\\{\begin{array}[]{l}\hat{d}_{j}=a_{0}^{(d)}\cdot\left(\left(a_{1}^{(d)}\right)^{j}-1\right)+1\\\
\hat{w}_{j}=a_{0}^{(w)}\cdot\left(\left(a_{1}^{(w)}\right)^{j}-1\right)+1\\\
\hat{r}_{j}=a_{0}^{(r)}\cdot\left(\left(a_{1}^{(r)}\right)^{j}-1\right)+1\end{array}\right.$
(9)

to guarantee the values in $\bm{\hat{S}}_{0}$, where $a_{0}$ and $a_{1}$ are
parameters which can be directly optimized by stochastic gradient descent
(SGD) or other optimization algorithms. As we can learn different values of
the parameters, respectively, the three dimensions can obtain distinct growth
rates.

Derivation of Larger-scale Architecture Generalization Functions. We define
FLOPs budget as $f$ here. We can obtain the relation between $f$ and scaling
stage $j$ as $f\propto\ 2^{\theta\times j}$ where $\propto$ means
“proportional to”, $\theta>0$ is a parameter. As the depth $\hat{d}$, width
$\hat{w}$, resolution $\hat{r}$ are positively correlated with $f$, we have
$\theta^{(d)}$, $\theta^{(w)}$, and $\theta^{(r)}$ with
$\theta=\theta^{(d)}+\theta^{(w)}+\theta^{(r)}$ for the three, and obtain

$2^{(\theta^{(d)}+\theta^{(w)}+\theta^{(r)})\times j}\ \dot{\propto}\
\hat{d}\times\hat{w}^{2}\times\hat{r}^{2},$ (10)

where $\dot{\propto}$ means “positively correlated with”, but not
“proportional to”. We formulate the relation between $j$ and $\hat{d}$ as
$2^{\theta^{(d)}\times j}\ \dot{\propto}\ \hat{d}$ and introduce a linear
approximation as $2^{\theta^{(d)}\times
j}\approx\beta\hat{d}+\delta\Rightarrow\hat{d}\approx\tfrac{1}{\beta}\cdot
2^{\theta^{(d)}\times j}-\tfrac{\delta}{\beta}$, where $\beta$ and $\delta$
are parameters. Here, we define $a_{0}^{(d)}=\tfrac{1}{\beta},\
a_{1}^{(d)}=2^{\theta^{(d)}},\ a_{2}^{(d)}=-\tfrac{\delta}{\beta}$ and obtain

$\hat{d}=a_{0}^{(d)}\cdot\left(a_{1}^{(d)}\right)^{j}+a_{2}^{(d)}.$ (11)

Note that as $a_{0}^{(d)}$, $a_{1}^{(d)}$, and $a_{2}^{(d)}$ are parameters,
“$\approx$” can be transferred to “$=$”.

Table 1: Comparisons with other state-of-the-art methods on ImageNet-$1$k dataset. Top-$1$ and Top-$5$ accuracies (%), FLOPs (G), and numbers of parameters (#Param., M) are reported. The best results are highlighted in bold.  Model | Top-$1$ | Top-$5$ | FLOPs | #Param.
---|---|---|---|---
FBNetV$2$-L$1$ [40] | $77.2$ | N/A | $0.33$ | N/A
OFA-$80$ [4] | $76.8$ | $93.3$ | $0.35$ | $6.1$
GreedyNAS-A [48] | $77.1$ | $93.3$ | $0.37$ | $6.5$
EfficientNet-B$0$ [37] | $76.3$ | $93.2$ | $0.39$ | $5.3$
ScaleNet-S$\bm{0}$ (ours) | $\bm{77.5}$ | $\bm{93.7}$ | $\bm{0.35}$ | $\bm{4.4}$
EfficientNet-B$1$ [37] | $78.8$ | $94.4$ | $0.70$ | $7.8$
OFA-$200$ [4] | $79.0$ | $94.5$ | $0.78$ | $11.0$
RegNetY-$800$MF [28] | $76.3$ | N/A | $0.80$ | $6.3$
EfficientNet-X-B$0$ [22] | $77.3$ | N/A | $0.91$ | $7.6$
ScaleNet-S$\bm{1}$ (ours) | $\bm{79.9}$ | $\bm{94.8}$ | $\bm{0.80}$ | $\bm{7.4}$
EfficientNet-B$2$ [37] | $79.8$ | $94.9$ | $1.00$ | $9.2$
EfficientNet-B$2$ (re-impl) | $80.4$ | $95.1$ | $1.00$ | $9.2$
BigNASModel-XL [50] | $80.9$ | N/A | $1.04$ | $9.5$
EfficientNet-X-B$1$ [22] | $79.4$ | N/A | $1.58$ | $9.6$
ScaleNet-S$\bm{2}$ (ours) | $\bm{81.3}$ | $\bm{95.6}$ | $\bm{1.45}$ | $\bm{10.2}$
EfficientNet-B$3$ [37] | $81.1$ | $95.5$ | $1.80$ | $12.0$
EfficientNet-X-B$2$ [22] | $80.0$ | N/A | $2.30$ | $10.0$
RegNetY-$3.2$GF [28] | $79.0$ | N/A | $3.20$ | $19.4$
RegNetY-$4$GF [28] | $79.4$ | N/A | $4.00$ | $20.6$
ScaleNet-S$\bm{3}$ (ours) | $\bm{82.2}$ | $\bm{95.9}$ | $\bm{2.76}$ | $\bm{13.2}$
RegNetY-$500$M$\rightarrow\\!4$GF [11] | $81.7$ | N/A | $4.10$ | $36.2$
EfficientNet-B$4$ [37] | $82.6$ | $96.3$ | $4.20$ | $19.0$
EfficientNet-X-B$3$ [22] | $81.4$ | N/A | $4.30$ | $13.3$
RegNetY-$8$GF [28] | $81.7$ | N/A | $8.00$ | $39.2$
ScaleNet-S$\bm{4}$ (ours) | $\bm{83.2}$ | $\bm{96.6}$ | $\bm{5.97}$ | $\bm{16.1}$
EfficientNet-B$5$ [37] | $83.3$ | $96.7$ | $9.90$ | $30.0$
EfficientNet-X-B$4$ [22] | $83.0$ | N/A | $10.40$ | $21.6$
RegNetY-$500$M$\rightarrow\\!16$GF [11] | $83.1$ | N/A | $16.20$ | $134.8$
EfficientNet-B$0\rightarrow\\!16$GF [11] | $83.2$ | N/A | $16.20$ | $122.8$
ScaleNet-S$\bm{5}$ (ours) | $\bm{83.7}$ | $\bm{97.1}$ | $\bm{10.22}$ | $\bm{20.9}$

Then, due to $\hat{d}=1$ for the base model (_i.e._ , scaling stage $0$), we
should guarantee the relation. Thus, we put $\hat{d}=1,j=0$ into (11) and
obtain $a_{2}^{(d)}=1-a_{0}^{(d)}$. We re-put it into (11) and obtain the
depth function in (9) as

$\displaystyle\hat{d}=a_{0}^{(d)}\cdot\left(a_{1}^{(d)}\right)^{j}+\left(1-a_{0}^{(d)}\right)=a_{0}^{(d)}\cdot\left(\left(a_{1}^{(d)}\right)^{j}-1\right)+1.$
(12)

Similarly, we can achieve the relation between $i$ and $\hat{w}$, $\hat{r}$,
respectively, as

$\displaystyle\hat{w}$
$\displaystyle\approx\sqrt{\tfrac{1}{\beta^{\prime}}\cdot
2^{\theta^{(w)}\times
i}-\tfrac{\delta^{\prime}}{\beta^{\prime}}}\approx\tfrac{1}{\sqrt{\beta^{\prime}}}\sqrt{2}^{\theta^{(w)}\times
i}-\sqrt{\tfrac{\delta^{\prime}}{\beta^{\prime}}},$ (13)
$\displaystyle\hat{r}$
$\displaystyle\approx\sqrt{\tfrac{1}{\beta^{\prime\prime}}\cdot
2^{\theta^{(r)}\times
i}-\tfrac{\delta^{\prime\prime}}{\beta^{\prime\prime}}}\approx\tfrac{1}{\sqrt{\beta^{\prime\prime}}}\sqrt{2}^{\theta^{(r)}\times
i}-\sqrt{\tfrac{\delta^{\prime\prime}}{\beta^{\prime\prime}}},$ (14)

where $\beta^{\prime}$, $\delta^{\prime}$, $\beta^{\prime\prime}$, and
$\delta^{\prime\prime}$ are parameters.

##  4 Experimental Results and Discussions

###  4.1 Performance of ScaleNet on ImageNet-$\bm{1}$k

We conducted experiments on ImageNet-$1$k dataset [29] for the proposed
ScaleNet with recently proposed methods. Note that we divided a mini
validation set ($50$ images per class) from the training set for evaluation in
the MCEA. The search models are named by S$j$, where S$0$ is the base model,
S$1$, S$2$, and S$3$ are searched by the MCEA, and S$4$ and S$5$ are the
generalized ones. The FLOPs budgets are selected according to Figure 3.
Detailed settings are in the supplementary material. In Table 1 with different
FLOPs budgets, the searched models of our ScaleNet can achieve the best
performance among those with similar FLOPs.

Table 2: Performance in five fine-tuning tasks. Top-$1$ accuracies (%), FLOPs (G), parameter numbers (#Param., M) are reported. The best results are in bold. Dataset |  Model |  Top-$1$ |  FLOPs |  #Param.
---|---|---|---|---
FGVC Aircraft [26] | Inception-v$4$ [35] | $90.9$ | $13.00$ |  $41$
EfficientNet-B$3$ [37] | $90.7$ | $1.80$ |  $10$
ScaleNet-S$\bm{3}$ (ours) | $\bm{91.4}$ | $\bm{2.76}$ |  $\bm{11}$
Stanford Cars [19] | Inception-v$4$ [35] | $93.4$ | $13.00$ |  $41$
EfficientNet-B$3$ [37] | $93.6$ | $1.80$ |  $10$
ScaleNet-S$\bm{3}$ (ours) | $\bm{94.4}$ | $\bm{2.76}$ |  $\bm{11}$
Food-$101$ [3] | Inception-v$4$ [35] | $90.8$ | $13.00$ |  $41$
EfficientNet-B$4$ [37] | $91.5$ | $4.20$ |  $17$
ScaleNet-S$\bm{4}$ (ours) | $\bm{92.0}$ | $\bm{5.97}$ |  $\bm{14}$
CIFAR-$10$ [20] | NASNet-A [54] | $98.0$ | $42.00$ |  $85$
EfficientNet-B$0$ [37] | $98.1$ | $0.39$ |  $4$
ScaleNet-S$\bm{0}$ (ours) | $\bm{98.3}$ | $\bm{0.35}$ |  $\bm{3}$
CIFAR-$100$ [20] | NASNet-A [54] | $87.5$ | $42.00$ |  $85$
EfficientNet-B$0$ [37] | $88.1$ | $0.39$ |  $4$
ScaleNet-S$\bm{0}$ (ours) | $\bm{88.4}$ | $\bm{0.35}$ |  $\bm{3}$
Table 3: Ablation studies. Top-$1$ accuracies (%) of s$0$-s$4$ models on ImageNet-$100$ dataset are reported. In column “Sampl.”, “U” and “H” are the original uniform sampling and the proposed HSS, respectively. $M$ and $T$ are the maximum scaling stage and the iteration of the proposed MCEA. “Val” (for $T$ only) means the validation accuracy (%) in the MCEA. The best results are shown in bold. Sampl. |  $M$ |  $T$ |  Val |  s$0$ |  s$1$ |  s$2$ |  s$3$ |  s$4$
---|---|---|---|---|---|---|---|---
U | $3$ | $4$ | N/A | $84.16$ | $86.96$ | $87.72$ | $89.26$ | $90.02$
H | $1$ | $4$ | N/A | $84.06$ | $86.30$ | $87.93$ | $88.86$ | $90.30$
H | $2$ | $4$ | N/A | $84.42$ | $86.24$ | $88.02$ | $89.12$ | $90.14$
H | $3$ | $1$ | $63.58$ | $84.18$ | $86.34$ | $\bm{88.12}$ | $88.90$ | $89.76$
H | $3$ | $2$ | $63.38$ | $84.20$ | $85.86$ | $88.00$ | $89.44$ | $90.18$
H | $3$ | $4$ | $\bm{63.61}$ | $\bm{84.76}$ | $\bm{87.18}$ | $88.10$ | $\bm{89.90}$ | $90.46$
H | $3$ | $6$ | $63.59$ | $84.44$ | $86.42$ | $87.80$ | $89.54$ | $\bm{90.48}$
H | $3$ | $8$ | $63.53$ | $84.50$ | $86.48$ | $87.64$ | $89.30$ | $90.36$

###  4.2 Transferability to Fine-tuning Tasks

In addition to the experiments on ImageNet-$1$k, we also transferred the
searched architectures to fine-tuning tasks by fine-tuning our ImageNet-
pretrained models. Experimental settings can be found in the supplementary
material. Table 2 shows the transfer learning results. Ours can outperform
different referred models, respectively. When applying larger models, we can
gain further improvement.

Figure 5: Mean validation accuracies in the $4^{th}$ iteration of the MCEA by
using original uniform sampling in [50] and our HSS, respectively. The
accuracies are grouped based on FLOPs. Each non-overlapping group contains
recent $50$M FLOPs paths. We merely show the performance of the three scaling
stages, as we only evaluate them in the MCEA, except those of the base model.

###  4.3 Ablation Studies

We discuss the effect of the proposed components for the ScaleNet on
ImageNet-$100$ dataset [29, 39]. We divided a mini validation set ($50$ images
per class) from the training set. All the following validation accuracies were
calculated by the mini one. Searched models are named by s$j$, where s$0$
($120$M FLOPs) is the base model, s$1$ ($240$M FLOPs), s$2$ ($480$M FLOPs),
and s$3$ ($960$M FLOPs) are searched by the MCEA, and s$4$ ($1920$M FLOPs) is
the generalized one. The experimental results are shown in Table 3. Detailed
experimental settings and visualization of search space are in supplementary
material.

Effect of HSS: We compare the proposed HSS with the original uniform sampling
in [50]. Our HSS improves the searched results with better retrained
accuracies of s$0$-s$4$.

Furthermore, we illustrate the validation accuracies of paths by using both of
them to evaluate the sufficiency of the super-supernet training. In Figure 5,
the accuracies of our HSS are generally larger than those of the original one
in addition to the FLOPs interval of $[360,440]$, as the interval is the mode
of the original uniform sampling distribution. This means the proposed HSS can
improve the sufficiency of the super-supernet training.

We further analyze the Pearson, Spearman, and Kendall coefficients for
validation accuracies of the two ones, respectively. All of them are the
larger the better. Detailed settings are in the supplementary material. Table
4 shows the values of our HSS significantly outperform the corresponding ones
of the original. Specifically, our Pearson one is more than double of the
original’s.

Table 4: Performance comparison of three coefficients (%), including Pearson, Spearman, and Kendall coefficients, for validation accuracies by using original uniform sampling (“Original”) in [50] and our HSS, respectively, to evaluate the sampling strategies in super-supernet training. We sampled $6,000$ paths. Method | Pearson | Spearman | Kendall
---|---|---|---
Original | $35.3$ | $80.1$ | $64.1$
Our HSS | $\bm{73.6}$ | $\bm{83.9}$ | $\bm{66.2}$

(a) Base model.

(b) Scaling stage $1$.

(c) Scaling stage $2$.

(d) Scaling stage $3$.

Figure 6: Standard deviations (Stds) of validation accuracies in each
iteration of the MCEA. In different scaling stages, the Stds decrease
significantly in the first four iterations, while they then tend to be steady
and convergent.

Effect of Maximum Scaling Stage in Searching: We set the maximum scaling stage
$M$ to be one, two, or three in the MCEA for searching. ScaleNet can gain
better performance with larger $M$, which means a more suitable base model for
scaling can be found. More scaling stages can achieve better performance by
obtaining better base model architecture for scaling, which is a common sense.
This means we do not have to validate with much larger $M$.

Effect of Iteration in Searching: We set the iteration $T$ as one, two, four,
six, or eight in the MCEA for searching. When increasing $T$ from one to four,
better base models and scaling strategies can be obtained with top-$1$
accuracies improved in most of the scaling stages. This means that larger $T$
can improve the searched results. However, when increasing $T$ from four to
eight, similar performance can be found. This means about four iterations is
enough for the search. Meanwhile, we can find that the retraining accuracies
of s$0$-$4$ is relative to the validation accuracy in the MCEA, which shows
the effectiveness of it.

In addition, we show the standard deviations (Stds) of validation accuracies
in each iteration of the MCEA in Figure 6 to analyze their convergence. In
different scaling stages, the Stds decrease significantly in the first four
iterations, while they then tend to be steady and convergent. This shows that
our ScaleNet can effectively search the optimal ones and gradually minimize
the Stds.

Figure 7: Comparison of larger-scale architecture generalization functions.
The definitions of the compared ones are in the supplementary material. We
generalized the three ones of s$4$ and retrained the scaled models on
ImageNet-$100$ in (d).

Effect of Larger-scale Architecture Generalization: We experimentally compare
the proposed exponential one with commonly used polynomial functions, such as
linear and squared ones. As shown in Figure 7(a)-(c), three cases can
precisely fit the trends of the depth, width, and resolution, respectively.
Ours function can perform with different trends in the three dimensions. For
depth and resolution, ours obtains rapid increase similar to the squared one,
while it achieves gradual changes for width as the linear one. The total
trends of ours are various, which is similar to the conclusion in [37], but
the other two always perform the uppermost or the lowest, which are
unreasonable.

We also trained all the scaled s$4$ models with three generalized scaling
strategies, respectively, shown in Figure 7(d). The proposed one can achieve
the best top-$1$ performance as $90.46\%$, superior to the other two
functions. This shows the effectiveness of our larger-scale architecture
generalization.

Table 5: Comparisons of search cost (GPU/TPU days). Those of supernet (super-supernet) training and searching are compared, respectively. “Ratio-to-ScaleNet” is the ratio between total cost of a model to that of the proposed ScaleNet, the smaller the better. “$\dagger$” means we estimated the lower-bound time. “N/A” means the work does not have the step. “*” means the work has the step but did not specifically mentioned in the paper. The best result is in bold. Model | Device | Training | Searching | Total | Ratio-to-ScaleNet |
---|---|---|---|---|---|---
MnasNet [36]† |  TPUv$2$ | N/A | $211,571$ | $211,571$ | $436.23\times$ |
EfficientNet-X [22]† |  TPUv$3$ | N/A | $>1,765$ | $>1,765$ | $>3.64\times$ |
EfficientNet [37]† |  TPUv$3$ | N/A | $>1,714$ | $>1,714$ | $>3.53\times$ |
FBNetV$2$ [40]† |  V$100$ | * | * | $>1,633$ | $>3.37\times$ |
OFA [4]† |  V$100$ | * | * | $>1,486$ | $>3.06\times$ |
BigNAS [50]† |  TPUv$3$ | $>960$ | $>268$ | $>1,228$ | $>2.53\times$ |
ScaleNet (ours) | V$\bm{100}$ | $\bm{379}$ | $\bm{106}$ | $\bm{485}$ | $\bm{1\times}$ |

###  4.4 Discussion of Search Cost

We discuss the efficiency of our ScaleNet, compared with a few recent
strategies, including both one-shot NAS-based and two-step pipelines. We
estimated the search cost for the referred ones under our FLOPs budgets, as
they applied with various FLOPs budgets. The estimations are all shown in the
supplementary material. As shown in Table 5, the proposed ScaleNet can
remarkably reduce the total search cost, which contains the cost of
(super-)supernet training and searching. It can decrease at least $2.53\times$
and even $436.23\times$. Meanwhile, the proposed ScaleNet in the two parts of
cost can still significantly improve the efficiency, respectively. Note that
we used V$100$ for our experiments, while some others utilized TPUv$3$, which
are much better than ours. This means we can achieve a larger decrease on the
total search time under same resource conditions.

Figure 8: Trends of depth, width, and resolution on ImageNet-$1$k.

###  4.5 Discussion of the Trend of Scaling Strategies

We discuss the trend of learned scaling strategies for different scaling
stages (Figure 8) in order to promote further scaling strategy design.

  * •

Depth, width, and resolution change in an exponential order. They have
different values of their change rates among scaling stages. Depth and
resolution change similarly, while that of width is slightly smaller, which is
similar to EfficientNet [37].

  * •

Their values are not completely restricted by the theoretical constraint in
[37], but merely focus on the actual FLOPs budgets in each scaling stage. This
means the searching process of ours is more fair.

  * •

After obtaining the searched ones that are good enough under the corresponding
FLOPs budgets, the extended ones can be precisely constructed in FLOPs of
generalized architectures as well under our estimation and work well in
experiments.

##  5 Conclusion

In this paper, we proposed ScaleNet to jointly search the base model and a
group of the scaling strategies based on one-shot NAS framework. We improved
the super-supernet training by the proposed HSS. Then, we jointly searched the
base model and the scaling strategies by the proposed MCEA. The scaling
strategies of larger scales were decently generalized by the searched ones.
Experimental results show that the searched architectures by the proposed
ScaleNet with various FLOPs budgets can outperform the referred methods on
various datasets, including ImageNet-$1$k and fine-tuning tasks. Meanwhile,
the searching time can be significantly reduced, compared with those one-shot
NAS-based and manually designed two-step pipelines.

##  Appendix 0.A Detailed Experimental Settings

In this section, we describe the detailed experimental settings of the
proposed ScaleNet for super-supernet training, jointly base model and scaling
strategies searching, model retraining, and fine-tuning tasks, respectively.

FLOPs Budgets: For both ImageNet-$1$k and ImageNet-$100$ datasets [29], we
used same method to determine the FLOPs budgets for various scaling stages.

We first selected the FLOPs budget of base model by Monte Carlo simulation
according to the search space in Table 6. We randomly sampled about $100,000$
paths of base model and calculated the mean FLOPs of them. We chose a multiple
of $50$M near the mean as the FLOPs budget $f_{0}$ of base model. For scaling
stage $j$, we selected $2^{j}\cdot f_{0}$ as the FLOPs budget and searched the
optimal scaling strategy near it.

Super-supernet Training: For ImageNet-$1$k, we sampled the base model from the
search space in Table 6 and the scaling strategies in Table 7. We used a
stochastic gradient descent (SGD) optimizer with momentum as $0.9$ and weight
decay as $4\times 10^{-5}$ to train the super-supernet. Initial learning was
set as $0.12$ with a cosine annealing strategy for $750,000$ iterations.
Learning rate warm-up was also included for $3,750$ iterations, linearly
increasing from zero to $0.2$. We train the super-supernet with batch size as
$1024$. In data augmentation, we randomly resized the batches according to the
resolution values in the sampled scaling strategies, with common hyper-
parameters of the augmentation. Then, we resized the batch to the scaled
resolutions again. The maximum scaling stage $M$ was set as $3$ in the
experiments.

For ImageNet-$100$ dataset in ablation studies, we followed the class
selection [39] and reduced the channel number of the super-supernet to a half
with batch size as $256$, total iterations as $150,000$ and warm-up for $375$
iterations. All the other settings are same to those for ImageNet-$1$k
experiments. The visualization of search space in the ablation studies is
shown in Figure 9, which has similar trends with that for ImageNet-$1$k
experiments.

We divided a mini validation set ($50$ images per class) from the training set
for evaluation. The rest images in the training set were all used for super-
supernet training. All the validation accuracies that we have emphasized in
the paper were calculated by the mini validation set, respectively.

Table 6: The macro-structure of the search space of the base model. “$n$” is
the number of stacked building blocks, where “$n_{\text{min}}$” and
“$n_{\text{max}}$” are the minimum and maximum numbers, respectively.
“Operation” is the type of the block. “N/A” means “not applied”. “Y/N” means
“using or not” in the base model of the super-supernet training. “Input” is
the original resolution of input feature maps. “Channel” is the number of
output channels. “Stride” is the stride of the first block. “Scale” means
which dimensions of the stage need to be scaled, where “D” is depth, “W_I” is
input channel number, “W_O” is output channel number, “R” is resolution, “✓”
means having the part, “✗” means ignoring the one. “FC” is a fully connected
layer.

Stage | $n$ | Operation | Expand rate | SE | Input | Channel | Stride | Scale
---|---|---|---|---|---|---|---|---
$n_{\text{min}}$ | $n_{\text{max}}$ | D | W_I | W_O | R
Conv_stem | $1$ | $1$ |  $3\\!\times\\!3$ Conv. | N/A | N/A | $224\times 224\times 3$ | $112\times 112\times 32$ | $2$ | ✗ | ✗ | ✓ | ✓
Stage $1$ | $1$ | $1$ | MBConv | $1$ | Y/N | $112\times 112\times 32$ | $112\times 112\times 16$ | $1$ | ✓ | ✓ | ✓ | ✓
Stage $2$ | $1$ | $4$ | MBConv | $3/6$ | Y/N | $112\times 112\times 16$ | $56\times 56\times 32$ | $2$ | ✓ | ✓ | ✓ | ✓
Stage $3$ | $1$ | $4$ | MBConv | $3/6$ | Y/N | $56\times 56\times 32$ | $28\times 28\times 40$ | $2$ | ✓ | ✓ | ✓ | ✓
Stage $4$ | $1$ | $4$ | MBConv | $3/6$ | Y/N | $28\times 28\times 40$ | $14\times 14\times 80$ | $2$ | ✓ | ✓ | ✓ | ✓
Stage $5$ | $1$ | $4$ | MBConv | $3/6$ | Y/N | $14\times 14\times 80$ | $14\times 14\times 96$ | $1$ | ✓ | ✓ | ✓ | ✓
Stage $6$ | $1$ | $4$ | MBConv | $3/6$ | Y/N | $14\times 14\times 96$ | $7\times 7\times 192$ | $2$ | ✓ | ✓ | ✓ | ✓
Stage $7$ | $1$ | $1$ | MBConv | $3/6$ | Y/N | $7\times 7\times 192$ | $7\times 7\times 320$ | $1$ | ✓ | ✓ | ✓ | ✓
Conv_out | $1$ | $1$ |  $1\\!\times\\!1$ Conv. | N/A | N/A | $7\times 7\times 320$ | $7\times 7\times 1280$ | $1$ | ✗ | ✓ | ✗ | ✓
Pooling | $1$ | $1$ | Global avgpool | N/A | N/A | $7\times 7\times 1280$ | $1280$ | N/A | ✗ | ✗ | ✗ | ✗
Classifier | $1$ | $1$ | FC | N/A | N/A | $1280$ | $1000$ | N/A | ✗ | ✗ | ✗ | ✗

Jointly Base Model and Scaling Strategies Searching: We applied the proposed
MCEA for the search with iteration number $T$ as $8$ and sampling number of
base models for obtaining initial scaling strategies $S_{j}^{(0)}$ as $20$.

We first conducted an initial step for obtaining the $S_{j}^{(0)}$ based on
($7$) of the main body. Then, we iteratively searched the optimal base model
and scaling strategies for $T$ iterations. In each sub-optimization process of
the base model search steps, we undertook an evolution algorithm NSGA-II [10]
with population size $P$ as $50$ and generation size $N$ as $40$. Meanwhile,
although we can also employ the evolution algorithm for scaling strategy
search steps, we directly applied a small grid search on the trained super-
supernet as the search space of a scaling stage is small enough. Note that we
can also use small $N$ with large $T$ (such as $N=8,T=20$), but this may be
easy to fall into sub-optimal as the evolution algorithm needs to undertake
crossover-mutation steps.

Model Retraining: We followed previous work [21, 37] for obtaining the common
training recipe. The models were trained using a RMSProp optimizer with
momentum as $0.9$ and weight decay as $1\times 10^{-5}$. Initial learning was
set as $0.08$ with a step strategy for $300$ epochs. The learning rate was
increased from zero to $0.08$ linearly in the first $3$ epochs with batch size
$1024$, and then decayed to $0.97$ every $2.4$ epochs. In addition,
exponential moving average (EMA) on weights was adopted with a decay rate
$0.9999$. RandAugment [8] was also introduced.

Furthermore, for the ImageNet-$100$ dataset in ablation studies, we modified
batch size as $256$, decay rate of the EMA as $0.99$, and epoch number as
$200$.

Fine-tuning Tasks: We followed the model retraining settings for ImageNet-$1$k
experiments, but removed the EMA and reduced the batch size as $256$.

Table 7: The range of the search space of the scaling strategies. We take $M=3$ as an example. Depth and width use the same intervals of choices. Scaling stage $0$ refers to the base model. Scaling stage | Depth / Width | Resolution
---|---|---
$j$ | Min | Step | Max | Min | Step | Max
$0$ | $1.00$ | $0.00$ | $1.00$ | $1.00$ | $0.00$ | $1.00$
$1$ | $1.04$ | $0.04$ | $1.16$ | $1.00$ | $0.07$ | $1.14$
$2$ | $1.20$ | $0.04$ | $1.36$ | $1.21$ | $0.07$ | $1.35$
$3$ | $1.40$ | $0.04$ | $1.64$ | $1.43$ | $0.07$ | $1.57$
Figure 9: Sampling distribution based on the proposed HSS, compared with that
of the original uniform sampling for ablation studies (ImageNet-$100$). We
took $750,000$ paths for each to simulate actual super-supernet training. The
sampling distribution of the original uniform sampling performs as a bell-
shape single-modal one, where paths cannot be fairly trained In each scaling
stage (between black dashed lines). Meanwhile, paths in scaling stage $0$ and
$3$ cannot be sufficiently trained due to the too small sampling probabilities
of them. In contrary, our proposed HSS provides single-modal distributions for
each scaling stage and the modes of each locate near the corresponding target
FLOPs budgets.

##  Appendix 0.B Detailed Settings and Definitions of Comparisons of Pearson,
Spearman, and Kendall Coefficients

In Table $4$ of the main body, we evaluated the proposed HSS with three
coefficients, including Pearson $\rho_{\text{P}}$ [1], Spearman
$\rho_{\text{S}}$ [52], and Kendall $\tau$ [18], compared to the original
uniform sampling in [50], in order to indicate the super-supernet trained with
our HSS can provide more precisely ranking for architectures. In this section,
we explain the detailed settings and definitions of them, respectively.

Here, we assume the validation accuracies of the paths sampled the super-
supernet should be order preserving on FLOPs same to the actual performance of
those trained from scratch. This matches the assumption of the next
architecture search step in the one-shot NAS that the validation accuracies of
the paths can reflect on the actual performance. Thus, we utilized the three
ones to evaluate the effectiveness of our HSS on the order preservation of the
path performance.

We define a validation accuracy vector
$\bm{ACC}=[ACC_{1},\cdots,ACC_{Q}]^{\text{T}}$ and a FLOPs vector
$\bm{f}=[f_{1},\cdots,f_{Q}]^{\text{T}}$ for all the $Q$ sampled paths, where
$ACC_{q}$ and $f_{q}$ are both for one corresponding path. In our experiments,
$Q=6,000$. We can calculate the Pearson coefficient $\rho_{\text{P}}$ by

$\displaystyle\rho_{\text{P}}=\frac{\text{Cov}(\bm{ACC},\bm{f})}{\sigma_{ACC}\cdot\sigma_{f}}=\frac{\text{E}[(\bm{ACC}-\mu_{ACC})(\bm{f}-\mu_{f})]}{\sigma_{ACC}\cdot\sigma_{f}},$
(15)

where $\text{Cov}(\cdot,\cdot)$ is covariance function, $\sigma_{ACC}$ and
$\sigma_{f}$ are the standard deviations of the two vectors, respectively, and
$\mu_{ACC}$ and $\mu_{f}$ are the means of the two vectors, respectively.

For the Spearman coefficient $\rho_{\text{S}}$, as it replace the accuracy and
FLOPs vectors by their rank vectors $\tilde{\bm{ACC}}$ and $\tilde{\bm{f}}$,
respectively, we can compute it by

$\rho_{\text{S}}=\frac{\text{Cov}(\tilde{\bm{ACC}},\tilde{\bm{f}})}{\sigma_{\tilde{ACC}}\cdot\sigma_{\tilde{f}}},$
(16)

where $\sigma_{\tilde{ACC}}$ and $\sigma_{\tilde{f}}$ are the standard
deviations of the two rank vectors, respectively. As all the ranks are
distinct integers, (16) can be transferred to

$\rho_{\text{S}}=1-\frac{6\sum_{q=1}^{Q}(\tilde{ACC}_{q}-\tilde{f}_{q})^{2}}{Q(Q^{2}-1)}.$
(17)

Third, the Kendall coefficient $\tau$ is defined as

$\tau=\left|\frac{Q_{\text{concordant}}-Q_{\text{disconcordant}}}{Q_{\text{all}}}\right|,$
(18)

where

$\displaystyle Q_{\text{concordant}}$
$\displaystyle=\\#\\{(ACC_{q_{1}}-ACC_{q_{2}})(f_{q_{1}}-f_{q_{2}})>0,q_{1}<q_{2}\\},$
(19) $\displaystyle Q_{\text{disconcordant}}$
$\displaystyle=\\#\\{(ACC_{q_{1}}-ACC_{q_{2}})(f_{q_{1}}-f_{q_{2}})\leq
0,q_{1}<q_{2}\\},$ (20)

and $Q_{\text{all}}=\frac{Q(Q-1)}{2}$.

All the three coefficients are the larger the better in the interval of
$[0\%,100\%]$. More detailed explanations of them can be found in Wiki Pedia
for Pearson111https://en.wikipedia.org/wiki/Pearson_correlation_coefficient.,
Spearman222https://en.wikipedia.org/wiki/Spearman\%27s_rank_correlation_coefficient.,
and
Kendall333https://en.wikipedia.org/wiki/Kendall_rank_correlation_coefficient.
coefficients.

##  Appendix 0.C Definitions of Compared Functions in Lager-scale Architecture
Generalization

We have compared the proposed larger-scale architecture generalization
function in the ablation studies with linear and squared functions. Here, we
define the forms of the two compared functions and their optimization
processes in the ablation studies.

We define the optimal scales of $M+1$ scaling stages as
$\\{\bm{\hat{S}}_{j}\\}_{j=0}^{M}$. We also pre-define $\bm{\hat{S}}_{0}$ as
$\hat{d}_{0}=\hat{w}_{0}=\hat{r}_{0}=1$ for the base model. These are same to
those in the main body.

Linear Function. Here, we define the linear functions for the three
dimensions, respectively, as

$\left\\{\begin{array}[]{l}\hat{d}_{j}=a_{0}^{(d)}\cdot j+a_{1}^{(d)}\\\
\hat{w}_{j}=a_{0}^{(w)}\cdot j+a_{1}^{(w)}\\\ \hat{r}_{j}=a_{0}^{(r)}\cdot
j+a_{1}^{(r)}\end{array}\right.,$ (21)

where $a_{0}$ and $a_{1}$ are parameters.

In order to guarantee the relation between $j=0$ and $\bm{\hat{S}}_{0}$, we
put $\hat{d}=\hat{w}=\hat{r}=1,j=0$ into (21) and obtain

$\left\\{\begin{array}[]{l}a_{1}^{(d)}=1\\\ a_{1}^{(w)}=1\\\
a_{1}^{(r)}=1\end{array}\right..$ (22)

After that, we re-put (22) into (21) and obtain the linear functions for the
three dimensions as

$\left\\{\begin{array}[]{l}\hat{d}_{j}=a_{0}^{(d)}\cdot j+1\\\
\hat{w}_{j}=a_{0}^{(w)}\cdot j+1\\\ \hat{r}_{j}=a_{0}^{(r)}\cdot
j+1\end{array}\right..$ (23)

We should note that $a_{0}^{(d)}$, $a_{0}^{(w)}$, and $a_{0}^{(r)}$ are all
constrained to be positive as the trends of the three dimensions should be
monotonically increasing.

Squared Function. Similarly, we define the squared functions for the three
dimensions, respectively, as

$\left\\{\begin{array}[]{l}\hat{d}_{j}=a_{0}^{(d)}\cdot j^{2}+a_{1}^{(d)}\cdot
j+a_{2}^{(d)}\\\ \hat{w}_{j}=a_{0}^{(w)}\cdot j^{2}+a_{1}^{(w)}\cdot
j+a_{2}^{(w)}\\\ \hat{r}_{j}=a_{0}^{(r)}\cdot j^{2}+a_{1}^{(r)}\cdot
j+a_{2}^{(r)}\end{array}\right.,$ (24)

where $a_{0}$, $a_{1}$ ,and $a_{2}$ are parameters.

In the same way, we put $\hat{d}=\hat{w}=\hat{r}=1,j=0$ into (24) and obtain

$\left\\{\begin{array}[]{l}a_{2}^{(d)}=1\\\ a_{2}^{(w)}=1\\\
a_{2}^{(r)}=1\end{array}\right..$ (25)

We also re-put (25) into (24) and obtain the squared functions as

$\left\\{\begin{array}[]{l}\hat{d}_{j}=a_{0}^{(d)}\cdot j^{2}+a_{1}^{(d)}\cdot
j+1\\\ \hat{w}_{j}=a_{0}^{(w)}\cdot j^{2}+a_{1}^{(w)}\cdot j+1\\\
\hat{r}_{j}=a_{0}^{(r)}\cdot j^{2}+a_{1}^{(r)}\cdot j+1\end{array}\right.,$
(26)

where $a_{0}^{(d)}$, $a_{0}^{(w)}$, and $a_{0}^{(r)}$ are also constrained to
be positive.

Optimization Process. We define the optimization objective for one dimension
with a function $\text{func}(\cdot;a_{0},a_{1})$ as

$\displaystyle\operatorname*{argmin}_{a_{0},a_{1}}$
$\displaystyle\sum_{j=1}^{M}||y_{j}-\text{func}(j;a_{0},a_{1})||^{2},$ (27)
s.t. $\displaystyle\ a_{0}>0,$

where $\text{func}(\cdot;a_{0},a_{1})$ is selected from (23), (26), or (8) of
the main body and $y_{j}$ is the searched value of one dimension ($\hat{d}$,
$\hat{w}$, or $\hat{r}$) in scaling stage $j$. We can directly optimize by
stochastic gradient descent (SGD) or other optimization algorithms.

##  Appendix 0.D Search Space of Base Model

We introduce the search space of base model for model scaling in Table 6.
Super-supernet training will sample base models in paths according to the
search space. Following previous work [31, 48], we utilize mobile inverted
bottleneck MBConv [30] with expand rates as $3$ or $6$. Squeeze-and-excitation
network (SENet) [17] is also applied in the search space. No more than four
layers are assigned in each stage for the base model. Specifically, at least
one layer should be selected in each path of super-supernet training.

##  Appendix 0.E Search Space of Scaling Strategies

The search space of scaling strategy is defined in Table 7. We take maximum
scaling stage number $M=3$ as an example. Note that scaling stage $j=0$
represents the base model. As shown in Table 7, the three dimensions,  _i.e._
, depth, width, and resolution, are set as one for the base model,
respectively. For larger scaling stages, depth and width are selected in the
same intervals, while resolution has other different ones, which are
empirically designed.

For the three dimensions, we first set the center points of the intervals by
Monte Carlo simulation according to the FLOPs budgets, and then obtain the
ranges of the intervals based on the center points, respectively. In a
dimension, we define same value of steps for each scaling stage as $0.04$ for
both depth and width, and $0.07$ for resolution. For the former ones, a
certain and suitable step is enough for the super-supernet training, while for
the latter one, we empirically set them according to BigNAS [50]. Especially
for width, we need to guarantee the channel numbers of different paths to
uniformly change as we used the technique in [33] for width in super-supernet
training.

According to the search space of base model in Table 6 and that of scaling
strategy in Table 7, we can visualize the the whole search space to FLOPs by
Monte Carlo simulation, as shown in Figure $3$ of the main body for
ImageNet-$1$k experiments and Figure 9 for ablation studies on ImageNet-$100$
(half channel numbers for each layer). Our search spaces are both in similar
multi-modal forms of the histograms, instead of a simple bell-shape form of
the original uniform sampling. Note that although the respective sampling
distributions of each scaling stage are overlapped with contiguous ones, the
paths near the boundaries between scaling stages (_i.e._ , away from the FLOPs
Budgets) will be dropped in the search, as they do not satisfy the FLOPs
budgets. This means that they do not affect the joint base model and scaling
strategy search after the super-supernet training.

##  Appendix 0.F Path Generation with Sampled Base Model and Scaling Strategy

In the super-supernet training, we specifically sample a base model
architecture by uniform sampling and a scaling strategy by the proposed
hierarchical sampling strategy (HSS) for each batch. Thus, we should combine
them and generate a path for the super-supernet training.

For the depth, we simply copy the structure of the last non-identity block in
each stage that can be scaled (See line “D” in Table 6). “Conv_stem”,
“Pooling”, “Conv_out”, and “Classifier” cannot be scaled in depth. The number
$n_{\text{s}}$ of scaled blocks is defined as

$n_{\text{s}}=\left\lceil n\times d\right\rceil,$ (28)

where $n$ is the block number of a stage in a sampled path and $d$ is depth
coefficient.

Similarly for the width, we directly combine the added channels into each
layer that can be scaled in the base model. Note that the input channel number
of “Conv_stem” (three channels for RGB) and the output channel number of
“Conv_out” ($1280$ channels same to the input channel number of the fully
connected layer) are not modified (See line “W_I” and “W_O” in Table 6). The
latter one is different with that of EfficientNet [37]. The scaled channel
number $c_{\text{s}}$ of a layer is defined as

$c_{\text{s}}=\left\lceil c\times w\right\rceil,$ (29)

where $c$ is the channel number of a layer in a sampled path and $w$ is width
coefficient.

For the resolution, we directly resize the input images before augmentations
to the target size. It influences all the layers before “Pooling”. The scaled
resolution $r_{\text{s}}$ of a input image is defined as

$r_{\text{s}}=\left\lceil 224\times r\right\rceil,$ (30)

where $r$ is resolution coefficient.

The size of total search space in the super-supernet training is almost
$2\times 10^{26}$.

##  Appendix 0.G Search Cost Computation of Referred Methods

EfficientNet [37]: As they said in the paper that they conducted a small grid
search for $d_{1}$, $w_{1}$, and $r_{1}$ based on the constraint $d_{1}\times
w_{1}^{2}\times r_{1}^{2}\approx 2$, we sampled them in the interval of
$[1,2]$ with step as $0.01$ and analyzed all the cases. The ones that satisfy
the constraint $|d_{1}\times w_{1}^{2}\times r_{1}^{2}-2|\leq 0.1$ were
counted and $10,285$ cases were included. For the training time of a model, we
found that training an EfficientNet-B$4$ model ($4.2$G FLOPs) cost about one
TPU day. Thus, training an EfficientNet-B$1$ model ($0.7$G FLOPs) may cost
about $1/6$ TPU days. The total search cost for the compound scaling should be
$1,714$ TPU days.

Another work [40] said the lower bound of searching EfficientNet-B$0$ is
$91,000$ TPU hours,  _i.e._ , $3,792$ TPU days. However, we did not consider
the part.

EfficientNet-X [22]: They undertook a two-level grid search for $d_{1}$,
$w_{1}$, and $r_{1}$. We first sampled them in the interval of $[1,2]$ with
step as $0.1$ and remained all the cases. Then, a smaller grid search around
the best candidate with difference ranges in the interval of $[-0.1,0.1]$ and
step $0.01$. Finally, we can obtain $11^{3}+21^{3}-1=10,591$ cases. Similar to
the EfficientNet, the total search cost should be $1,756$ TPU days.

BigNAS [50]: As discussed in the paper, they train the supernet for $36$ hours
with $64$ TPUv$3$. Here, the model sized from $200$M to $2,000$M FLOPs and
they searched from $200$M to $1,000$M FLOPs. Thus, for searching a $10$G FLOPs
architecture, the supernet should size from $200$M to $20$G FLOPs,  _i.e._ ,
$100\times$ than before. The supernet training time should be also enlarged to
$10\times$ as $36\times 64\times 10=23,040$ TPU hours,  _i.e._ , $960$ TPU
days. Although the paper did not mention their cost on searching
architectures, we assumed their cost ratio of supernet training and searching
is same to ours and approximated their searching cost as
$960\times(106/379)=268$ TPU days.

FBNetV$2$ [40]: As shown in the paper, the authors searched the FBNetV$2$-L$1$
model ($0.33$M FLOPs) with total $0.6$k GPU hours. We linearly expand the
number by FLOPs and obtain $39,182$ GPU hours,  _i.e._ , $1,633$ GPU days.

OFA [4]: They first searched a large architecture ($40$ GPU hours), where
FLOPs budget was not mentioned in the paper and we assumed as $600$M FLOPs,
then trained the model for enough time ($1,200$ GPU hours), and searched
target ones from the large one by progressive shrinking and fine-tuning
($75\times 40$ GPU hours). For searching and training a $10$G FLOPs large
model, they need $(1,200+40)\times 10,000/600=20,667$ GPU hours. Then,
searching the five scaling stages needs at least $75\times 40\times 5=15,000$
GPU hours. The total time cost should be $20,667+15,000=35,667$ GPU hours,
_i.e._ , $1,486$ GPU days.

MnasNet [36]: Their total search cost is $91,000$ TPU hours for $388$M FLOPs
budget. Transferring to our total FLOPs budgets as
$10,000+6,000+3,000+1,500+800+350=21,650$M FLOPs, their search cost should be
$91,000\times 21,650/388=5,077,706$ TPU hours,  _i.e._ , $211,571$ TPU days.

Speed comparisons of Different Devices: We have reviewed the computation speed
of different devices, including V$100$ GPU and TPUv$2$/$3$, which were used in
previous work [4, 22, 36, 37, 40, 50] or this paper. TPUv$3$ is $2.7\times$
faster than TPUv$2$444See
https://www.linleygroup.com/newsletters/newsletter_detail.php?num=6203&year=2020&tag=3..
The speed of a TPUv$2$ core is similar to that of one V$100$555See page $24$
in https://storage.googleapis.com/nexttpu/index.html.. Combining all of them,
we can obtain the training speed ratios as $\text{a TPUv}3\text{ core}:\text{a
TPUv}2\text{ core}:\text{a V}100=2.7:1:1$.

Note that we only discuss the search cost, as we assume all the methods
require same cost on retraining, except for the BigNAS, which does not need to
retrain the searched architectures. Retraining needs about $280$ GPU days on
V$100$ and the ScaleNet totally costs $765$ GPU days, which is much smaller
than $1,228$ TPU days on TPUv$3$ of the BigNAS.

Table 8: The searched and generalized scaling strategies from ScaleNet-S$0$ to ScaleNet-S$5$. Scaling stage | Depth | Width | Resolution
---|---|---|---
$0$ | $1.000$ | $1.000$ | $1.000$
$1$ | $1.080$ | $1.040$ | $1.140$
$2$ | $1.360$ | $1.200$ | $1.355$
$3$ | $1.480$ | $1.400$ | $1.580$
$4$ | $1.653$ | $1.534$ | $2.042$
$5$ | $1.848$ | $1.688$ | $2.378$
Figure 10: Visualizations of searched and generalized architectures from
ScaleNet-S$0$ to ScaleNet-S$5$. “MB$e$_K$k$_SE” means the block uses $k\times
k$ depth-wise convolution as the intermediate convolution with expand rate as
$e$ and SE block. The architectures are split into stages (not scaling stages)
by the resolutions. All the downsampling blocks are employed in the first
layer of each stage.

##  Appendix 0.H Visualizations of Search Architectures

Table 8 shows the searched and generalized scaling strategies of the proposed
ScaleNet for the ImageNet-$1$k experiments. Moreover, Figure 10 shows the
corresponding searched and generalized architectures. Here, we find that they
have the following three properties:

  * •

Regarding kernels of convolutional layers, the base model usually prefers to
use more large kernels ($5\times 5$ or $7\times 7$ convolution layers) and
more $7\times 7$ kernels are applied in the deeper layers.

  * •

With different FLOPs budgets, two architectures may have same numbers of
layers, but with distinct width and resolution, respectively. As different
values of depth can respond to a same architecture.

  * •

Regarding resolutions of various scaling stages, the ScaleNet tends to use
higher resolutions than those in [11, 37] to improve the performance. They can
also reduce the numbers of parameters under certain FLOPs budgets to some
extent.

Table 9: Detailed comparisons between our ScaleNet and EfficientNet. “re-impl”
means we reimplemented the model with our settings. “$*$” used our scaling
strategies with EfficientNet-B$0$. The best results are highlighted in bold.

Model | Top-$1$ acc. (%) | Top-$5$ acc. (%) | #Params (M) | FLOPs (G) | Resolution
---|---|---|---|---|---
EfficientNet-B$1$ [37] | $78.8$ | $94.4$ | $7.8$ | $0.70$ | $240\\!\times\\!240$
EfficientNet-B$1$ (re-impl) | $78.7$ | $94.1$ | $7.8$ | $0.70$ | $240\\!\times\\!240$
EfficientNet-S$1^{*}$ | $79.2$ | $94.6$ | $8.3$ | $0.79$ | $256\\!\times\\!256$
ScaleNet-S$\bm{1}$ | $\bm{79.9}$ | $\bm{94.8}$ | $\bm{7.4}$ | $\bm{0.80}$ | $\bm{256\\!\times\\!256}$
EfficientNet-B$2$ [37] | $79.8$ | $94.9$ | $9.2$ | $1.00$ | $260\\!\times\\!260$
EfficientNet-B$2$ (re-impl) | $80.4$ | $95.1$ | $9.2$ | $1.00$ | $260\\!\times\\!260$
EfficientNet-S$2^{*}$ | $80.8$ | $95.4$ | $11.8$ | $1.58$ | $304\\!\times\\!304$
ScaleNet-S$\bm{2}$ | $\bm{81.3}$ | $\bm{95.6}$ | $\bm{10.2}$ | $\bm{1.45}$ | $\bm{304\\!\times\\!304}$
EfficientNet-B$3$ [37] | $81.1$ | $95.5$ | $12.0$ | $1.80$ | $300\\!\times\\!300$
EfficientNet-B$3$ (re-impl) | $81.1$ | $95.4$ | $12.0$ | $1.80$ | $300\\!\times\\!300$
EfficientNet-S$3^{*}$ | $82.1$ | $95.8$ | $15.4$ | $2.94$ | $354\\!\times\\!354$
ScaleNet-S$\bm{3}$ | $\bm{82.2}$ | $\bm{95.9}$ | $\bm{13.2}$ | $\bm{2.76}$ | $\bm{354\\!\times\\!354}$
EfficientNet-B$4$ [37] | $82.6$ | $96.3$ | $19.0$ | $4.20$ | $380\\!\times\\!380$
EfficientNet-B$4$ (re-impl) | $82.6$ | $96.3$ | $19.0$ | $4.20$ | $380\\!\times\\!380$
EfficientNet-S$4^{*}$ | $82.8$ | $96.3$ | $19.5$ | $6.34$ | $458\\!\times\\!458$
ScaleNet-S$\bm{4}$ | $\bm{83.2}$ | $\bm{96.6}$ | $\bm{16.1}$ | $\bm{5.97}$ | $\bm{458\\!\times\\!458}$
EfficientNet-B$5$ [37] | $83.3$ | $96.7$ | $30.0$ | $9.90$ | $456\\!\times\\!456$
EfficientNet-B$5$ (re-impl) | $83.2$ | $96.7$ | $30.0$ | $9.90$ | $456\\!\times\\!456$
EfficientNet-S$5^{*}$ | $83.2$ | $96.4$ | $25.8$ | $11.42$ | $533\\!\times\\!533$
ScaleNet-S$\bm{5}$ | $\bm{83.7}$ | $\bm{97.1}$ | $\bm{20.9}$ | $\bm{10.22}$ | $\bm{533\\!\times\\!533}$

##  Appendix 0.I Discussion of Maximum Scaling Stage in Searching

In this section, we discuss the maximum scaling stage issue in the joint
search of base model and scaling strategy. As discussed in the ablation
studies, searching on more scaling stages can obtain better scaling
strategies. This is given that directly searching the scaling strategy on a
scaling stage should have better performance than generalizing it. Although we
applied the maximum scaling stage $M=3$ and generalized the model on scaling
stage $4$ in the ablation studies, using $M=4$ and searching the fourth
scaling stage may obtain a better one.

However, larger $M$ implies consuming more resources, for example, training
the super-supernet under current settings on ImageNet-$1$k needs $32$ V$100$
cards with automatic mixed precision (AMP) or $64$ V$100$ cards with full
precision. $M$ influences the computational complexity in both super-supernet
training and searching. It is approximately exponential to the computational
complexity (the computational complexity is proportional to FLOPs). However,
with its increase, performance improvement of the generalized models should be
approximately logarithmically increased as in Table $3$. As current
experimental results have reached state-of-the-art performance, utilizing
double resources to training the super-supernet and searching with one more
scaling stage and achieving $0.5\%$ top-$1$ accuracy improvement on
ImageNet-$1$k (similar to s$4$ models in Table $3$ of the main body. We also
trained an s$5$ model and found that its top-$1$ accuracy is $90.74\%$,
slightly higher than $90.46\%$ of s$4$ model.) between a directly searched one
and a generalized one may not be necessary. Since the performance of models
tends to saturate with the increase of model scales, in practice a moderate
$M$ should be favored for better and efficient search. It can be theoretically
set as infinite, but a too large value of it is worthless. Thus, we only set
$M$ as three in the experiments.

(a) FLOPs versus top-$1$ accuracy.

(b) #Param. versus top-$1$ accuracy.

Figure 11: Comparisons between our ScaleNet and other baseline methods with on
ImageNet-$1$k dataset. Both (a) FLOPs and (b) number of parameters (#Param.)
are compared versus top-$1$ accuracy. Note that we do not illustrate ResNet in
(b) as it has too many parameters.

##  Appendix 0.J Comparisons with EfficientNet

Table 9 shows detailed comparisons between our ScaleNet and EfficientNet [37].
With same experimental settings, the searched and generalized models of our
ScaleNet can significantly outperform the re-implemented EfficientNet models
in each scaling stage with similar FLOPs and numbers of parameters,
respectively. As our models were implemented with larger resolutions than
EfficientNet ones, we can slightly reduce the numbers of parameters for each.

Meanwhile, we conducted a group of experiments on our scaling strategies with
EfficientNet-B$0$ as base model, namely EfficientNet-S$1$ to
EfficientNet-S$5$. Compared with the baseline models, they can achieve
remarkable or marginal performance improvement in various scaling stages. This
means our obtained scaling strategies is better than those of the baseline.
Furthermore, our results can also surpass those of them, which demonstrates
our searched base model is more suitable for scaling than the
EfficientNet-B$0$.

To intuitively compare the performance of our ScaleNet with other baseline
methods, we visualize the top-$1$ accuracies of our ScaleNet  _w.r.t._ FLOPs
and number of parameters, respectively, in Figure 11. As shown in Figure
11(a), the performance of the searched and generalized models can outperform
all the others as an upper bound. Similarly in Figure 11(b), our models also
perform best among the others with a larger margin on accuracy.

Table 10: Statistics of five fine-tuning datasets. “#Training”, “#Test”, and “#Class” are numbers of training samples, test samples, and classes, respectively. Dataset | #Training | #Test | #Class
---|---|---|---
FGVC Aircraft [26] | $6,667$ | $3,333$ | $100$
Stanford Cars [19] | $8,144$ | $8,041$ | $196$
Food-$101$ [3] | $75,750$ | $25,250$ | $101$
CIFAR-$10$ [20] | $50,000$ | $10,000$ | $10$
CIFAR-$100$ [20] | $50,000$ | $10,000$ | $100$
Table 11: Comparisons with other state-of-the-art methods in five fine-tuning tasks. Top-$1$ accuracies (%), FLOPs (G), numbers of parameters (#Param., M) are reported. The best results on each dataset are highlighted in bold. Dataset | Model |  Top-$1$ |  FLOPs |  #Param.
---|---|---|---|---
FGVC Aircraft | EfficientNet-B$3$ [37] | $90.7$ | $1.80$ | $10$
NAT-M$4$ [25] | $90.8$ | $0.58$ | $5$
Inception-v$4$ [35] | $90.9$ | $13.00$ | $41$
ScaleNet-S$3$ (ours) | $91.4$ | $2.76$ | $11$
ScaleNet-S$4$ (ours) | $92.8$ | $5.97$ | $14$
ScaleNet-S$\bm{5}$ (ours) | $\bm{92.9}$ | $\bm{10.22}$ | $\bm{19}$
Stanford Cars | NAT-M$4$ [25] | $92.9$ | $0.37$ | $6$
Inception-v$4$ [35] | $93.4$ | $13.00$ | $41$
EfficientNet-B$3$ [37] | $93.6$ | $1.80$ | $10$
EfficientNetV2-S [38] | $93.8$ | $8.80$ | $24$
EfficientNet-B$7$ [37] | $94.7$ | $37.00$ | $64$
DAT [27] | $94.8$ | - | -
ScaleNet-S$3$ (ours) | $94.4$ | $2.76$ | $11$
ScaleNet-S$4$ (ours) | $95.0$ | $5.97$ | $14$
ScaleNet-S$\bm{5}$ (ours) | $\bm{95.1}$ | $\bm{10.22}$ | $\bm{19}$
Food-$101$ | NAT-M$4$ [25] | $89.4$ | $0.36$ | $5$
Inception-v$4$ [35] | $90.8$ | $13.00$ | $41$
EfficientNet-B$4$ [37] | $91.5$ | $4.20$ | $17$
ScaleNet-S$3$ (ours) | $91.2$ | $2.76$ | $11$
ScaleNet-S$4$ (ours) | $92.0$ | $5.97$ | $14$
ScaleNet-S$\bm{5}$ (ours) | $\bm{92.2}$ | $\bm{10.22}$ | $\bm{19}$
CIFAR-$10$ | Proxyless-G+c/o [5] | $97.2$ | - | $6$
NASNet-A [54] | $98.0$ | $42.00$ | $85$
EfficientNet-B$0$ [37] | $98.1$ | $0.39$ | $4$
NAT-M$3$ [25] | $98.2$ | $0.39$ | $6$
ScaleNet-S$\bm{0}$ (ours) | $\bm{98.3}$ | $\bm{0.35}$ | $\bm{3}$
CIFAR-$100$ | NASNet-A [54] | $87.5$ | $42.00$ | $85$
NAT-M$3$ [25] | $87.7$ | $0.49$ | $8$
EfficientNet-B$0$ [37] | $88.1$ | $0.39$ | $4$
ScaleNet-S$\bm{0}$ (ours) | $\bm{88.4}$ | $\bm{0.35}$ | $\bm{3}$

##  Appendix 0.K More Comparisons on Fine-tuning Tasks

The statistics of five fine-tuning datasets are shown in Table 10, including
numbers of training samples, test samples, and classes. Table 11 shows
experimental results on the datasets. On FGVC Aircraft, Stanford Cars, and
Food-$101$, we trained ScaleNet-S$3/4/5$ with our ImageNet-pretrained models
and obtained state-of-the-art performance. Furthermore, on the other two CIFAR
datasets, we can also achieve superior accuracies compared with the baseline
models.

##  Appendix 0.L Definition Difference between “Stage” and “Scaling Stage”

“Stage” and “scaling stage” are in two different dimensions. ($1$) “Stage”
corresponds to depth in one model. As shown in Figure 10, a model can be
divided into seven stages with different channel numbers. ($2$) “Scaling
stage” is a holistic concept that refers to various models with different
FLOPs budgets,  _i.e._ , the subfigures in Figure 10.

## References

  * [1] Benesty, J., Chen, J., Huang, Y., Cohen, I.: Pearson Correlation Coefficient, pp. 1–4. Springer Berlin Heidelberg (2009)
  * [2] Bishop, C.M.: Pattern Recognition and Machine Learning. Springer Science+Business Media LLC. (2006)
  * [3] Bossard, L., Guillaumin, M., Van Gool, L.: Food-101 – mining discriminative components with random forests. In: ECCV. pp. 446–461 (2014)
  * [4] Cai, H., Gan, C., Wang, T., Zhang, Z., Han, S.: Once-for-All: Train one network and specialize it for efficient deployment. In: ICLR (2020)
  * [5] Cai, H., Zhu, L., Han, S.: ProxylessNAS: Direct neural architecture search on target task and hardware. In: ICLR (2019)
  * [6] Chang, D., Pang, K., Zheng, Y., Ma, Z., Song, Y.Z., Guo, J.: Your “flamingo” is my “bird”: Fine-grained, or not. In: CVPR. pp. 11476–11485 (2021)
  * [7] Cheng, Z., Su, X., Wang, X., You, S., Xu, C.: Sufficient vision transformer. In: KDD (2022)
  * [8] Cubuk, E.D., Zoph, B., Shlens, J., Le, Q.: Randaugment: Practical automated data augmentation with a reduced search space. In: NeurIPS. vol. 33, pp. 18613–18624 (2020)
  * [9] Dai, X., Wan, A., Zhang, P., Wu, B., He, Z., Wei, Z., Chen, K., Tian, Y., Yu, M., Vajda, P., Gonzalez, J.E.: FBNetV3: Joint architecture-recipe search using predictor pretraining. In: CVPR. pp. 16276–16285 (2021)
  * [10] Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.A.M.T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation 6(2), 182–197 (2002)
  * [11] Dollar, P., Singh, M., Girshick, R.: Fast and accurate model scaling. In: CVPR. pp. 924–932 (2021)
  * [12] Du, R., Xie, J., Ma, Z., Chang, D., Song, Y.Z., Guo, J.: Progressive learning of category-consistent multi-granularity features for fine-grained visual classification. IEEE TPAMI (2021)
  * [13] Guo, Z., Zhang, X., Mu, H., Heng, W., Liu, Z., Wei, Y., Sun, J.: Single path one-shot neural architecture search with uniform sampling. In: ECCV. pp. 544–560 (2020)
  * [14] Han, K., Wang, Y., Zhang, Q., Zhang, W., Xu, C., Zhang, T.: Model rubik’s cube: Twisting resolution, depth and width for TinyNets. In: NeurIPS (2020)
  * [15] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
  * [16] Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., Le, Q.V., Adam, H.: Searching for mobilenetV3. In: ICCV (October 2019)
  * [17] Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: CVPR (2018)
  * [18] Kendall, M.G.: A new measure of rank correlation. Biometrika 30(1/2), 81–93 (1938)
  * [19] Krause, J., Stark, M., Deng, J., Fei-Fei, L.: 3D object representations for fine-grained categorization. In: CVPRW. pp. 554–561 (2013)
  * [20] Krizhevsky, A.: Learning multiple layers of features from tiny images. techreport, CIFAR (2009)
  * [21] Li, C., Peng, J., Yuan, L., Wang, G., Liang, X., Lin, L., Chang, X.: Block-wisely supervised neural architecture search with knowledge distillation. In: CVPR (2020)
  * [22] Li, S., Tan, M., Pang, R., Li, A., Cheng, L., Le, Q.V., Jouppi, N.P.: Searching for fast model families on datacenter accelerators. In: CVPR. pp. 8085–8095 (2021)
  * [23] Liu, C., Han, K., Xiao, A., Deng, Y., Zhang, W., Xu, C., Wang, Y.: Greedy network enlarging. ArXiv preprint, arXiv:2108.00177 (2021)
  * [24] Lou, W., Xun, L., Sabet, A., Bi, J., Hare, J., Merrett, G.V.: Dynamic-OFA: Runtime DNN architecture switching for performance scaling on heterogeneous embedded platforms. In: CVPRW. pp. 3110–3118 (2021)
  * [25] Lu, Z., Sreekumar, G., Goodman, E., Banzhaf, W., Deb, K., Boddeti, V.N.: Neural architecture transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence 43(9), 2971––2989 (2021)
  * [26] Maji, S., Rahtu, E., Kannala, J., Blaschko, M., Vedaldi, A.: Fine-grained visual classification of aircraft. ArXiv preprint, arXiv:1306.5151 (2013)
  * [27] Ngiam, J., Peng, D., Vasudevan, V., Kornblith, S., Le, Q.V., Pang, R.: Domain adaptive transfer learning with specialist models. ArXiv preprint, arXiv:1811.07056 (2018)
  * [28] Radosavovic, I., Kosaraju, R.P., Girshick, R., He, K., Dollar, P.: Designing network design spaces. In: CVPR (2020)
  * [29] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet large scale visual recognition challenge. IJCV 115(3), 211–252 (2015)
  * [30] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: MobileNetV2: Inverted residuals and linear bottlenecks. In: CVPR (2018)
  * [31] Su, X., Huang, T., Li, Y., You, S., Wang, F., Qian, C., Zhang, C., Xu, C.: Prioritized architecture sampling with Monto-Carlo tree search. In: CVPR. pp. 10968–10977 (2021)
  * [32] Su, X., You, S., Huang, T., Wang, F., Qian, C., Zhang, C., Xu, C.: Locally free weight sharing for network width search. ArXiv preprint, arXiv:2102.05258 (2021)
  * [33] Su, X., You, S., Wang, F., Qian, C., Zhang, C., Xu, C.: Bcnet: Searching for network width with bilaterally coupled network. In: CVPR. pp. 2175–2184 (2021)
  * [34] Su, X., You, S., Xie, J., Zheng, M., Wang, F., Qian, C., Zhang, C., Wang, X., Xu, C.: ViTAS: Vision transformer architecture search. ArXiv preprint, arXiv:2106.13700 (2021)
  * [35] Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.A.: Inception-v4, Inception-ResNet and the impact of residual connections on learning. In: AAAI (2017)
  * [36] Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., Le, Q.V.: MnasNet: Platform-aware neural architecture search for mobile. In: CVPR (2019)
  * [37] Tan, M., Le, Q.: EfficientNet: Rethinking model scaling for convolutional neural networks. In: ICML. pp. 6105–6114 (2019)
  * [38] Tan, M., Le, Q.V.: EfficientNetV2: Smaller models and faster training. ArXiv preprint, arXiv:2104.00298 (2021)
  * [39] Tian, Y., Krishnan, D., Isola, P.: Contrastive multiview coding. In: ECCV. pp. 776–794 (2020)
  * [40] Wan, A., Dai, X., Zhang, P., He, Z., Tian, Y., Xie, S., Wu, B., Yu, M., Xu, T., Chen, K., Vajda, P., Gonzalez, J.E.: FBNetV2: Differentiable neural architecture search for spatial and channel dimensions. In: CVPR (2020)
  * [41] Wang, C., Xu, C., Yao, X., Tao, D.: Evolutionary generative adversarial networks. IEEE Transactions on Evolutionary Computation 23(6), 921–934 (2019)
  * [42] Wu, B., Dai, X., Zhang, P., Wang, Y., Sun, F., Wu, Y., Tian, Y., Vajda, P., Jia, Y., Keutzer, K.: FBNet: Hardware-aware efficient ConvNet design via differentiable neural architecture search. In: CVPR (2019)
  * [43] Xie, J., Ma, Z., Chang, D., Zhang, G., Guo, J.: GPCA: A probabilistic framework for Gaussian process embedded channel attention. IEEE TPAMI (2021)
  * [44] Xie, J., Ma, Z., Lei, J., Zhang, G., Xue, J.H., Tan, Z.H., Guo, J.: Advanced dropout: A model-free methodology for Bayesian dropout optimization. IEEE TPAMI (2021)
  * [45] Xie, J., Ma, Z., Xue, J.H., Zhang, G., Sun, J., Zheng, Y., Guo, J.: DS-UI: Dual-supervised mixture of Gaussian mixture models for uncertainty inference in image recognition. IEEE TIP 30, 9208–9219 (2021)
  * [46] Xu, H., Su, X., Wang, Y., Cai, H., Cui, K., Chen, X.: Automatic bridge crack detection using a convolutional neural network. Applied Sciences 9(14), 2867 (2019)
  * [47] Yang, Z., Liu, D., Wang, C., Yang, J., Tao, D.: Modeling image composition for complex scene generation. ArXiv preprint, arXiv:2206.00923 (2022)
  * [48] You, S., Huang, T., Yang, M., Wang, F., Qian, C., Zhang, C.: GreedyNAS: Towards fast one-shot nas with greedy supernet. In: CVPR (2020)
  * [49] You, S., Xu, C., Xu, C., Tao, D.: Learning from multiple teacher networks. In: KDD. pp. 1285–1294 (2017)
  * [50] Yu, J., Jin, P., Liu, H., Bender, G., Kindermans, P.J., Tan, M., Huang, T., Song, X., Pang, R., Le, Q.: BigNAS: Scaling up neural architecture search with big single-stage models. In: ECCV. pp. 702–717 (2020)
  * [51] Zagoruyko, S., Komodakis, N.: Wide residual networks. ArXiv preprint, arXiv:1605.07146 (2016)
  * [52] Zar, J.H.: Spearman rank correlation. Encyclopedia of biostatistics 7 (2005)
  * [53] Zhai, X., Kolesnikov, A., Houlsby, N., Beyer, L.: Scaling vision transformers. ArXiv preprint, arXiv:2106.04560 (2021)
  * [54] Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: CVPR (2018)