# MetaTune: Meta-Learning Based Cost Model
for Fast and Efficient Auto-tuning Frameworks

Jaehun Ryu  Hyojin Sung

###### Abstract

Deep learning compiler frameworks are gaining ground as a more portable back-
end for deep learning applications on increasingly diverse hardware. However,
they face the daunting challenge of matching performance offered by hand-tuned
target-specific libraries. While auto-tuning frameworks with statistical cost
models can provide dynamic and efficient code optimization, they suffer from
large space exploration and cost model training overheads. This paper proposes
MetaTune, a meta-learning based cost model that more quickly and accurately
predicts the performance of optimized codes with pre-trained model parameters.
MetaTune encodes convolution kernel codes as structurally similar graphs to
facilitate meta-learning, meta-trains a GNN model with a very small input data
set, and then predicts optimization parameters for unseen convolution
operations with varying sizes and structures during compilation. The resulting
framework with MetaTune provides 8 to 13% better inference time on average for
four CNN models with comparable or lower optimization time while outperforming
transfer learning by 10% in cross-platform cases.

Machine Learning, ICML

##  1 Introduction

Deep learning models have recently emerged as an application domain that
drives innovations in domain-specific compiler technologies. While many deep
learning programming frameworks (Abadi et al., 2015; Paszke et al., 2019; Chen
et al., 2015) rely on hand-optimized libraries such as NVIDIA cuDNN/cuBLAS or
Intel MKL for their back-ends, code-generating compiler frameworks (Chen et
al., 2018a; Rotem et al., 2019; Ragan-Kelley et al., 2013) are exploring its
potential as a more flexible and portable solution on increasingly diverse and
heterogeneous platforms.

The key challenge for deep-learning compilers lies in generating highly
optimized codes with comparable or better performance than hand-tuned
libraries.

To address the limitations of traditional rule-based heuristics that cannot
dynamically adapt to input and hardware characteristics, more data-driven
approaches have been proposed to determine optimization strategies (Chen et
al., 2018b; Ragan-Kelley et al., 2013; Ahn et al., 2020). These “auto-tuning”
compilers use statistical cost models to learn the correlation between
programs and runtime behaviors from profiling runs and guide the search of
ideal optimization configurations.

However, this approach incurs significant compilation overheads from searching
the huge optimization space even for simple convolution kernels and repeatedly
profiling optimized codes to train a cost model. The effort to address the
issue focused mainly on improving the search algorithms or
reducing/accelerating profiling runs (Ahn et al., 2020; Dhakal et al., 2020),
while cost models are still freshly trained from hardware measurements of each
tensor operation which has a unique optimization space. (Chen et al., 2018b)
reused pre-trained cost models for different tensor operations and with
transfer learning, but it still requires several hours and hundreds of
iterations for cost models to converge, and its effectiveness is task-
dependent.

In a search for a more general and systematic way of reusing knowledge
obtained from other tensor operations or platforms to fine-tune the cost
model, we observed that meta-learning can enable the cost model to “learn to
learn” code-performance correlations and very quickly adapt to the
optimization space of any tensor operations with structural similarity. A
performance regression task with code embeddings as input is not a typical
meta-learning application and has sparse prior work. However, with a more
flexible meta-learning method that works with any model with gradient descent
(Finn et al., 2017), a cost model may predict performance more accurately in
much fewer iterations with meta-learned hyperparameters.

Thus, we propose MetaTune, a meta-learning based cost model for more accurate
and efficient auto-tuning frameworks. We implemented MetaTune as an
alternative cost model for the auto-tuning framework in TVM (Chen et al.,
2018a). MetaTune includes (1) a feature generator that encodes code templates
of convolution kernels as graphs and augments the embeddings to improve
structural similarity, and (2) a meta-learning based cost model that meta-
trains with performance prediction tasks generated from a small data set
during pre-training and then fine-tunes with profiling runs of target kernel
during auto-tuning.

Our experimental results show that MetaTune outperforms existing cost models,
regardless of candidate search algorithms used together, and enables fast
adaptation in not only cross-operation but also cross-platform scenarios.
MetaTune with batch Bayesian optimization search we suggested in the paper
improves the inference time by 8% and 13% for all evaluated convolution
kernels, compared to (Chen et al., 2018b) and (Ahn et al., 2020) respectively.
Compared to transfer-learning based cost models, the MetaTune model provides
faster and better fine-tuning results (10% on average) on NVIDIA GPU’s of
different generations than the pre-training target hardware.

The contributions of the paper are as follows:

  * •

Super-graph input augmentation: Extending graph-based representation of codes
in prior work (Leary and Wang, 2017) (Tomczak et al., 2019), we augmented
graphs to fit in a common super-graph template and produced input data with
higher structural similarity. As a result, the same MetaTune model with
template augmentation provides 4.6% higher inference time on average than the
original input.

  * •

Meta-learning model formulation:  We designed meta-training tasks for
performance regression so that the model can be pre-trained for few-shot
learning. The result shows that the auto-tuning frameworks with MetaTune jump-
start to predict high-performing optimization parameters within the first
dozens of iterations.

  * •

Performance portability:  We showed that MetaTune can seamlessly interface
with existing search algorithms and consistently improve autotuning
framework’s overall efficiency on different GPU platforms.

  * •

Complete solution:  We implemented a batch Bayesian optimization search
algorithm to cater to the needs of MetaTune, which would not require as much
space exploration due to pre-training but incurs higher fine-tuning overheads.
Our proposed framework with MetaTune and batch Bayesian optimization achieves
the lowest inference time for all CNN models evaluated.

In the rest of the paper, we first provide an overview of auto-tuning
frameworks in deep learning compilers we assume for MetaTune, and present the
design and implementation of the MetaTune model in that context. Then we
evaluate the efficiency of MetaTune in various scenarios in the evaluation
section and wrap up with related work and conclusions.

Figure 1: The structure of deep learning compiler frameworks

##  2 Background

Knobs | Description |  # of knobs
---|---|---
tile_x |  Loop tiling parameter on the number of filter,height,weight of feature maps |  140
tile_y |  140
tile_f |  120
tile_rc |  Loop tile reduction parameter on the number of channels,height ,weight of feature maps |  8
tile_rx |  2
tile_ry |  2
auto_unroll _max_step | Guide max unroll iteration |  3
unroll_explicit | Turn on unroll loop |  2
Table 1: Description for knobs in TVM

###  2.1 Deep Learning Compiler Frameworks

Deep learning compiler frameworks consist of language front-ends and code-
generating back-ends, as shown in Figure 1. The front-end takes an input model
and translates it into high-level IR (often graph-based) to which target-
independent optimizations such as operator fusion and data layout
transformation are applied. The back-end takes optimized IR as input and goes
through target-dependent optimization passes that further transform IR to
better exploit target hardware features. Auto-tuning is implemented as a part
of such target-dependent optimization passes.

TVM (Chen et al., 2018a) is an open-source deep learning compilers with the
said structure that are widely adopted by industry and academia. Its auto-
tuning framework aims to match the performance of hand-tuned libraries and
showed promising results in prior work (Chen et al., 2018b; Ahn et al., 2020).
This paper focuses on proposing an alternative cost model for TVM to further
improve auto-tuning efficiency without changing high-level structures, while
our approach can be applied to any cost-model based auto-tuning effort.

###  2.2 Machine-learning Based Auto-tuning

The auto-tuning problem, in general, is about automatically generating a
search space of optimized codes and finding the best-performing version from
hardware measurements (Ding et al., 2015a, b). To efficiently navigate the
huge search space of all possible implementations, a cost model function is
often used to predict the performance of codes and guide the search. With a
code-generating function $\rho$ that uses optimization parameters $\phi$
(e.g., tile size), program operation $\sigma$ (e.g., 2D convolution) and
operation options $c$ (e.g., kernel size) to create a search space of
optimized codes $D$ and a cost model function $f$ predicts performance of
codes in $D$, the problem can be formulated as follows:

$\vspace{-0.2cm}\phi^{*}=\operatorname*{argmax}_{\phi}f(\rho(\phi|\sigma,c))$

In TVM, $\phi$ consists of a combination of parameters called knobs, so the
above expression turns into finding an optimal combination of knobs as shown
in Table 1 to maximize the performance predicted by $f$.

The accuracy of the cost model $f$ is crucial in locating ideal optimization
parameters, and many auto-tuning frameworks including TVM adopts machine-
learning based cost models that can more dynamically adapt to search results
than fixed heuristics or rule-based models.

As shown in Figure 1, Exploration module in TVM searches the space of
optimized codes for ideal optimization parameters and selects what to profile
next based on the predicted performance of the cost model which is in turn
updated based on hardware measurements. The key insight behind MetaTune is
that $f$ can be pre-conditioned with randomly sampled points in a small number
of similar optimization spaces so that it can quickly adapt to $f^{*}$, target
cost model function.

Figure 2: Graph-based input generation with super-graph Figure 3: MetaTune
model architecture Figure 4: Example of few-shot regression tasks for
MetaTune. The model learns the universal performance distribution from a pair
of graph-structured data which is sampled input and its label (the blue line
in the distribution).

##  3 Data Representation and Augmentation

Prior work on automatic optimizations emphasized the importance of finding
input features representative of program codes for machine-learning based cost
models to accurately understand code-performance correlations (Cummins et al.,
2017; Chen et al., 2018b; Fursin et al., 2011; Vasilache et al., 2018). When a
cost model is meta-learned, it requires input data to be sufficiently similar
in terms of its structure as well since the difference in data representation
in meta-training and meta-testing dataset can cause meta-overfitting (Finn et
al., 2017) leading to a learning failure. For example, meta-learning an image-
based model assumes input images of the same size where values for each pixel
are in the same value range. Thus, we focus on designing input data
representation of MetaTune to have to enable meta-learning.

###  3.1 Graph Representation

TVM offers three ways to generate input data for the cost model: a set of knob
parameters, “loop context vectors” which include computed statistics about
loops such as loop lengths, access patterns, and many different arithmetic
instructions, and curve features which summarize relational information about
loop context vectors (Chen et al., 2018b). Our attempt to design a meta-
learning based cost model with these representations consistently failed due
to a lack of structure in them.

They express structural information as second-hand statistics computed and
summarized from codes. As weakly structured data, they have variable lengths
for tensor operations with different structures. Such format divergence
interfered with meta-learning adaptation which assumes structurally similar
tasks. Inspired by prior work that showed more structured, tree/graph-based
code representation improves the modeling efficiency in general (Kaufman et
al., 2020; Tomczak et al., 2019; Sung et al., 2019), we use graph-based input
data for meta-learning and fine-tuning the MetaTune cost model.

Convolution kernels are expressed as code templates in TVM where kernels of
the same type (e.g., conv1d, conv2d+transponse) have the same code template.
Code templates are in turn represented as low-level abstract syntax trees
(AST).

As shown in Figure 2, we extract loop structures by recursively traversing the
AST and build a summarized graph representation in the middle. Nodes in the
graph can be a root node, for nodes, or iterval nodes. Directed edges between
the root node and all for nodes represent control flow, while edges between
for and iterval nodes match loops with detailed loop metadata. Each iterval
node has loop context vectors computed by TVM as its node feature. root and
for nodes, without any node features, serve the role of placing the loop
context vectors in a precise context.

###  3.2 Super-graph Augmentation

To produce more structurally uniform graphs, we augment the AST-based graph
input to fit in the shared super-graph template. This augmentation can be
viewed as a further format standardization of code templates (which are
already per-type format standardization) across all types of convolution
operations.

We perform a union of all possible graph representations and create a super-
graph template with placeholder nodes and edges between them. For super-graph
augmentation, we locate a corresponding iterval node in the template which is
uniquely identified and populate the node with context feature vectors. Figure
2 shows that only $i$, $j$, and $k$ iterval nodes corresponding to the nodes
in the original graph contain context feature vectors, while other iterval
nodes are null in the super-graph version on the right. To save the overhead
of augmentation, we pre-define the super-graph template for all convolution
kernels supported in TVM, identify mappings between matching nodes in a table,
and use the table to quickly generate final output. We plan to extend the
super-graph formulation process in the future to dynamically support a wider
range of tensor operations.

##  4 Model Architecture

The MetaTune cost model consists of a graph convolutional network (GCN) (Kipf
and Welling, 2017), an activation and aggregation layer, and three fully
connected (FC) layers with ReLU activation, as shown in Figure 3.

The GCN model serves as an embedding layer that takes augmented graphs as
input data and encodes relationships between performance-critical information
through convolutions on graph nodes. With the following ReLU activation and
weighted sum and max aggregation, low-dimensional, fixed-length embedding
vectors are generated as input for meta-training and fine-tuning tasks. The
regression model uses fully connected layers with non-linearity as exemplified
in (Finn et al., 2017) for regressions tasks. This model is meta-trained with
the model-agnostic meta-learning (MAML) method and continues to fine-tune with
inference queries during auto-tuning. The output of the model is the predicted
performance of input implementation in FLops.

##  5 Meta-learning Process

Algorithm 1 Metatune Meta Train-step

0: Input-Performance dataset $\mathcal{D}$ and distribution $p(\mathcal{T})$
over kernel performance

0: $\alpha$, $\beta$: step size

0: $n$,$\gamma$ : epochs, learning rate

1: Randomly initialize parameters $\theta$

2: for i …$n$ do

3: Sample input,output pair($knob,perf$) in $\mathcal{D}$

4: Evaluate training loss $\nabla_{\theta}\mathcal{L}(f_{\theta}(knob),perf)$

5: Update
$\theta\leftarrow\theta-\gamma\nabla_{\theta}\mathcal{L}(f_{\theta}(knob),perf))$

6: end for

7: while not done do

8: Sample batch of tasks $\mathcal{T}_{i}\sim p(\mathcal{T})$

9: for all $\mathcal{T}_{i}$ do

10: Evaluate training loss
$\nabla_{\theta}\mathcal{L}(f_{\theta}(knob_{i}),perf_{i})$

11: Compute adapted parameters with gradient
descent:$\theta_{i}^{\prime}=\theta-\alpha\nabla_{\theta}\mathcal{L}(f_{\theta}(knob_{i}),perf_{i})$

12: end for

13: Update
$\theta\leftarrow\theta-\beta\nabla_{\theta}\sum_{\mathcal{T}_{i}\sim
p(\mathcal{T})}\mathcal{L}(\theta_{i}^{\prime})$

14: end while

We trained our cost model using MAML (Finn et al., 2017; Koch, 2015) as shown
in Algorithm 1.

  1. 1.

Supervised pre-training [line 1-5]: As previous work (Soh et al., 2020) on
regression models learned with meta-learning proposed, we prepare a small
dataset with labels and pre-train the GCN embedding layer via supervised
learning to obtain stable embedding results from GCN and model weights that
are close to a target cost model function $f^{*}$.

  2. 2.

Input data preparation for meta-training [line 8]: We prepare another labeled
dataset for all types of convolution kernels with varying configurations (more
details on training dataset in Section 6) and randomly organized them into
meta-training tasks.

We create few-shot meta-training tasks to approximate a universal performance
distribution for knobs for all convolutional kernels. For the example shown in
Figure 4, a 3-way, 2-shot learning task includes two examples with different
optimization parameters from each of three classes for conv2d, winograd, and
depthwise operations with specific input and kernel size and stride.

  3. 3.

Meta-training [line 9-13]: The regression model with FC layers learns the
meta-training tasks with MAML (in this phase, we only use the pre-trained GCN
layer without further training). For each meta-training task, the model
computes loss (line 11) and updates $\theta_{i}$ with gradient descent (line
12), where $\alpha$ is the learning rate for local parameter updates within a
task. After all tasks are evaluated, $\theta$ of the cost model $f$ is updated
with gradient descent with the global learning rate $\beta$ and the sum of
$\theta_{i}$ (line 13).

The example in Figure 4 will sample a batch of tasks in each iteration and
trains the model with the training loss between predicted FLops and actual
FLops from each task. The model will learn to predict the performance of an
unseen implementation of the same task after being trained with other graphs
in the task through many epochs. At the end of training, the model will be in
a state (in terms of model weights) to need only a small number of samples to
regress a structurally similar but possibly unseen task.

During compilation, the pre-trained MetaTune model keeps improving with
adaptation to inference queries from the exploration module. Unlike meta-
training tasks, fine-tuning tasks include candidate samples for a single task
to be optimized. Online hardware measurement will be used for gradient
updates.

##  6 Evaluation Methodology

Operation |  Input size |  Input channel |  Output channel
---|---|---|---
conv1d, transpose1d |  150-600 |  32-128 |  32-512
conv2d, transpose2d, winograd |  7-224 |  3-128 |  16-128
Table 2: Parameters for MetaTune model training dataset

We implemented MetaTune as a plug-in cost model component for TVM version
0.7.dev. We replaced the input feature generator and cost model with MetaTune
implementation while experimenting with different search algorithms in the
exploration module as follows.

  * •

Batch Bayesian optimization:

The default search algorithm in TVM, simulated annealing (SA) (Bertsimas et
al., 1993), is known to effectively approximate global optimization in a large
search space with extensive random exploration. We observed that MetaTune
would not require as much space exploration but incur higher inference
overheads due to a complicated model structure than prior work, thus
implemented a batch Bayesian optimization (BO) algorithm inspired by
(Desautels et al., 2014; Wilson et al., 2017; Wu and Frazier, 2016). The BO is
more exploitation-centric and performs a search with a much lower number of
iterations than SA. Our implementation using botorch library defines each knob
to be the input of the BO to understand the knob relationships and executes
all BO operations and inference queries in parallel on GPU.

  * •

Adaptive exploration and sampling (Ahn et al., 2020): Reinforcement-learning
based search algorithms.

Evaluated models and data:  We evaluated inference times, total optimization
times, cost model prediction accuracy (MSE) for four CNN models: ResNet-18,
Vgg-16, Squeezenet_v1.0, and AlexNet.

For meta-training, we created a meta-training dataset from 47 convolution
kernels as shown in Table 2. Their input size and input/output channels are
randomly chosen within the range in the table, while the stride and padding
were fixed at 3 and 1 respectively. Then we randomly extracted 200 samples of
optimized codes for each class, from which meta-training tasks are created.

Evaluated frameworks:  We evaluated the following seven auto-tuning frameworks
in TVM including reproduction of prior work (Chen et al., 2018b; Ahn et al.,
2020).

  * •

xgb: The default TVM implementation with gradient boosting (Chen and Guestrin,
2016) for the cost model and SA for search algorithm as in (Chen et al.,
2018b).

  * •

xgb-Xfer:  xgb with transfer learning enabled (as in TVM v0.7.dev).

  * •

RL:  The reinforcement-learning based implementation in (Ahn et al., 2020).

  * •

meta-RL and meta-RL-T:  MetaTune with RL-based candidate selection. meta-RL-T
and meta-RL work with and without super-graph augmented data respectively.

  * •

meta-BO and meta-BO-T:  MetaTune with batch Bayesian optimization based
candidate selection. meta-BO-T and meta-BO work with and without super-graph
augmented data respectively.

Evaluation environment:  We conducted our primary performance evaluation for
all frameworks on a server with Intel Xeon CPUs and NVIDIA geForce 2080 Ti
GPUs. Hardware measurements, batch Bayesian optimization, and RL-based
implementation use a single GPU, while other components run on CPU. For
experiments on additional target hardware, we used an NVIDIA Titan XP GPU
server and an AMD Radeon RX VEGA 64 GPU server.

##  7 Experimental Results

Our experimental evaluation aims to answer the following questions: Can
MetaTune consistently adapt faster and find better solutions with different
search algorithms on different platforms? We answered the question by (1)
measuring inference time of CNN models optimized by auto-tuning frameworks
with MetaTune (evaluating the overall framework efficiency), (2) measuring the
model accuracy of MetaTune (isolating the cost model efficiency), and (3)
repeating the experiments on other platforms (showing portability). We
performed hyperparameter tuning for the MetaTune model with batch Bayesian
optimization and RL-based candidate search algorithms, and the details can be
found in the appendix.

(a) Resnet-18

(b) Vgg-16

(c) Squeezenet

(d) Alexnet

Figure 5: Output performance

(a) Resnet-18

(b) Vgg-16

(c) Squeezenet

(d) Alexnet

Figure 6: End-to-end evaluation of normalized inference time

(a) Resnet-18

(b) Alexnet

Figure 7: Output performance on NVIDIA Titan XP GPU

(a) Resnet-18

(b) Alexnet

Figure 8: Output performance on AMD Vega 64 GPU

Figure 9: Model accuracy in early iterations | xgb | RL | meta-BO | meta-BO-T
---|---|---|---|---
resnet-18 | 580.04 | 622.71 | 514.27 | 576.21
vgg-16 | 829.17 | 962.17 | 652.85 | 727.76
alexnet | 358.33 | 434.25 | 306.60 | 341.61
squeezenet | 966.78 | 1038.02 | 929.61 | 980.15
Table 3: End-to-end optimization time (minutes)

###  7.1 Model Output Performance

Figure 5 shows the output performance for four CNN models in FLops, while
Figure 6 compares highest FLops achieved for the models normalized to xgb.
Overall, meta-BO-T provides the best performance for all configurations,
outperforming xgb by 8% (up to 10%) and RL by 13% (up to 33%) on average.
meta-BO without super-graph augmentation also consistently improves xgb and RL
by 5% and 10% on average. meta-BO-T consistently outperforming meta-BO by 4%
on average supports our assumption that super-graph augmentation helps with a
quick adaptation of meta-learning models by increasing structural similarity
in input data.

You can see from the figure that both meta-BO and meta-BO-T adapt much faster
to the compiled kernel than the others, i.e., move to points more likely to
perform well and start candidate search from them, and reach the highest FLops
predicted by xgb earlier between 200 to 650 iterations (dotted lines). As a
result, MetaTune can search the space more effectively and widely with the
given number of iteration locating better-performing optimization parameters.

Both MetaTune and xgb-Xfer reuse previous experience for the current task.
While xgb-Xfer slightly improves output performance against xgb, all MetaTune
models show a more noticeable performance impact of the pre-trained model by
outperforming xgb-Xfer by 4 to 7% on average.

MetaTune with RL-based search algorithm in (Ahn et al., 2020) instead of batch
BO significantly outperforms RL by 7% on average (up to 28%), isolating the
performance impact of the cost model. In this scenario, meta-RL produces
better results than meta-RL-T for all evaluated models. We identified the
source in adaptive sampler in (Ahn et al., 2020), which performs much fewer
hardware measurements, thus adaptation for meta-RL-T than meta-RL after
filtering. We plan to investigate the issue further, but it is out of the
scope of this paper.

The evaluation results on AMD GPU as shown in Figure 8(b) and 8(a) shows 18%
and 5% performance improvement by meta-BO-T against xgb. This confirms that
meta-learning based cost models are a portable solution whose optimization
capability is not platform-dependent.

###  7.2 Optimization Time

Table 3 shows the end-to-end auto-tuning time for xgb, RL, meta-BO, and meta-
BO-T. While meta-BO is the fastest while all four are in a close range (We
could not reproduce the results for (Ahn et al., 2020)). meta-BO-T seems to
incur more inference overheads with larger super-graph data than meta-BO, but
is comparable or faster than xgb or RL.

network | method | MSE | MSE(D) | network | method | MSE | MSE(D)
---|---|---|---|---|---|---|---
alexnet | xgb | 0.1329 | 0.5884 | resnet-18 | xgb | 0.1175 | 0.6318
alexnet | xgb-Xfer | 0.1394 | 0.5289 | resnet-18 | xgb-Xfer | 0.1281 | 0.5666
alexnet | meta-BO | 0.1342 | 0.5888 | resnet-18 | meta-BO | 0.1048 | 0.5720
alexnet | meta-BO-T | 0.1134 | 0.4733 | resnet-18 | meta-BO-T | 0.1061 | 0.5373
vgg-16 | xgb | 0.1223 | 0.5859 | squeezenet | xgb | 0.1225 | 0.6058
vgg-16 | xgb-Xfer | 0.1388 | 0.5415 | squeezenet | xgb-Xfer | 0.1268 | 0.5442
vgg-16 | meta-BO | 0.1075 | 0.5150 | squeezenet | meta-BO | 0.1288 | 0.4212
vgg-16 | meta-BO-T | 0.1198 | 0.4754 | squeezenet | meta-BO-T | 0.1485 | 0.4044
alexnet(X) | xgb | 0.1292 | 0.4787 | resnet-18(X) | xgb | 0.1304 | 0.5600
alexnet(X) | xgb-Xfer | 0.1384 | 0.4621 | resnet-18(X) | xgb-Xfer | 0.1335 | 0.5471
alexnet(X) | meta-BO-T | 0.1337 | 0.3178 | resnet-18(X) | meta-BO-T | 0.1177 | 0.4319
alexnet(A) | xgb | 0.1018 | 0.7129 | resnet-18(A) | xgb | 0.1134 | 0.6264
alexnet(A) | meta-BO-T | 0.082 | 0.6150 | resnet-18(A) | meta-BO-T | 0.0942 | 0.6188
Table 4: Model error for xgb and xgb-Xfer vs. meta-BO-T. (X) for results on
Titan XP, (A) on AMD GPU.

###  7.3 Model Accuracy

Table 4 shows the prediction accuracy of the MetaTune cost model and the
gradient-boosting based cost model in TVM through 1,000 iterations. We
analyzed the inconsistency with MSE distribution and found out that infeasible
or erroneous candidates randomly appear during exploration, and failing to
predict performance for them unduly increases the model’s MSE. Therefore, we
introduced an additional accuracy metric than MSE; MSE(D) is a mean-square
error of fine-tuning instances whose hardware-measured FLops are in the top
25%. This metric is useful in focusing on how effective the cost model is in
identifying possibly best-performing candidates and guiding the search in the
right direction. meta-BO-T shows the highest MSE(D) for all models on all
platforms, which aligns with the output performance result.

We also investigated the model accuracy in early auto-tuning iterations in
detail to see if MetaTune adapts more quickly than non-meta-learning models.
From Figure 9, you can see that the MetaTune models start at a lower MSE and
maintain high accuracy while xgb and xgb-Xfer may explore further away from
ideal optimizations with more wrong predictions.

###  7.4 Cross-Platform Meta-Learning

We experimented with a pre-trained MetaTune model to see if it can adapt
cross-platform as well, i.e., if meta-learned model parameters can universally
work for performance distributions on which the model is not trained. Figure
7(a) and 7(b) show the output performance (inference time) for alexnet and
resnet-18 on NVIDIA Titan XP GPU where meta-BO-T are pre-trained on NVIDIA
2080 Ti GPU. meta-BO-T provides 14% and 6% higher performance than xgb and
xgb-Xfer showing that knowledge reuse across platforms is possible. Cross-
platform meta-learning across more heterogeneous platforms is a part of our
future work.

##  8 Related Work

### Adaptive exploration for automatic optimizations

There has been active research in improving the exploration module in auto-
tuning frameworks to reduce the total optimization time. Chameleon (Ahn et
al., 2020) addressed the issues of not selectively exploring feasible
optimizations only (Chen et al., 2018b) by proposing adaptive sampling and
exploration algorithms using reinforcement learning ALT (Zeng et al., 2020)
paid attention to the suboptimality caused by a premature cost model during
the exploration process. It proposes a filtering mechanism for useless
candidates that uses score prediction and active learning. AdaTune (Li et al.,
2020) addresses the exploration vs. exploitation problem that affects total
optimization and time and output quality. It proposes surrogate modeling to
solve the problem using uncertainty quantification and an evaluator that
determines early stopping based on the coefficient variation.

These are orthogonal to MetaTune focusing on the cost model and can be
combined with MetaTune for further optimizations.

### Auto-scheduling for tensor programs

FlexTensor (Zheng et al., 2020b) aims to generate dynamic schedules for tensor
operations and perform automatic optimization with them on heterogeneous
systems. Schedules are generated without code templates, focusing on the
search space of each operation at a time. While this approach may locate more
flexible and potentially efficient codes than using template-based schedules,
it incurs much higher overheads with fine-grained searches. Our results show
that the use of templates judiciously restricts the search space while not
hurting the quality of the solution. On a related note, Ansor (Zheng et al.,
2020a) also automatically generates optimization candidates hierarchically
without template-based restrictions. There are no predefined knobs per
operation, and a task scheduler learns the right level of hardware
measurements as well. This approach requires a larger number of hardware
measurements at each hierarchical level, ranging from 300 to 6,000 iterations.

### Meta learning for fast adaptation

Recent research used MAML to initialize model parameters for quick adaption in
various applications.

(Winata et al., 2020) used MAML to adapt to the high variability and complex
characteristics of accents for speech recognition. T-NAS (Lian et al., 2020)
leveraged MAML to handle multiple tasks in network architecture search, while
MAML is used to improve the accuracy of a super-resolution model for unseen
images in (Soh et al., 2020).

##  9 Conclusion and Future Work

In this paper, we presented a meta-learning based approach to improve the
efficiency and reduce the overheads of the auto-tuning process by reusing
prior experiences. Our experimental results showed that the resulting cost
model, MetaTune, can identify high-performing optimization parameters for
unseen convolution operations faster and more accurately than prior work,
regardless of search algorithms used together. We believe that the meta-
learning based cost model, or any machine-learning based model that can
systematically leverage previously learned knowledge and experiences with
similar programs, is a pivotal contribution to the auto-tuning solutions,
addressing the high optimization overheads and allowing them to be a more
widely-adopted compiler solution.

Our future work includes extending MetaTune to support a wider range of tensor
operations beyond convolution operations on different platforms. We believe
that the meta-learning based approach can show even stronger and more
consistent performance than related work with auto-tuning targets with varying
static and dynamic characteristics. We also consider going out of the boundary
of domain-specific compiler frameworks and looking more into solving more
general optimization problems with MetaTune approach.

## Acknowledgement

This work was supported by Institute for Information & communications
Technology Promotion (IITP) grant funded by the Korea government (MSIP) (No.
2019-0-01906, Artificial Intelligence Graduate School Program (POSTECH)) and
the Super Computer Development Leading Program of the National Research
Foundation of Korea (NRF) funded by the Korean government (Ministry of Science
and ICT (MSIT)) (No. 2020M3H6A1084853).

## References

  * (1)
  * Abadi et al. (2015) Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015\.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems.  https://www.tensorflow.org/ Software available from tensorflow.org.
  * Ahn et al. (2020) Byung Hoon Ahn, Prannoy Pilligundla, Amir Yazdanbakhsh, and Hadi Esmaeilzadeh. 2020.  Chameleon: Adaptive Code Optimization for Expedited Deep Neural Network Compilation.  arXiv:2001.08743 [cs.LG]
  * Bertsimas et al. (1993) Dimitris Bertsimas, John Tsitsiklis, et al. 1993\.  Simulated annealing.  _Statistical science_ 8, 1 (1993), 10–15.
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  XGBoost: A Scalable Tree Boosting System. In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ (San Francisco, California, USA) _(KDD ’16)_. Association for Computing Machinery, New York, NY, USA, 785–794.  https://doi.org/10.1145/2939672.2939785
  * Chen et al. (2015) Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. 2015\.  MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems.  _CoRR_ abs/1512.01274 (2015).  http://dblp.uni-trier.de/db/journals/corr/corr1512.html#ChenLLLWWXXZZ15
  * Chen et al. (2018a) Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Meghan Cowan, Haichen Shen, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. 2018a.  TVM: An Automated End-to-End Optimizing Compiler for Deep Learning. In _Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementation_ (Carlsbad, CA, USA) _(OSDI’18)_. USENIX Association, USA, 579–594.
  * Chen et al. (2018b) Tianqi Chen, Lianmin Zheng, Eddie Yan, Ziheng Jiang, Thierry Moreau, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. 2018b.  Learning to Optimize Tensor Programs. In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ (Montréal, Canada) _(NIPS’18)_. Curran Associates Inc., Red Hook, NY, USA, 3393–3404.
  * Cummins et al. (2017) C. Cummins, P. Petoumenos, Z. Wang, and H. Leather. 2017.  End-to-End Deep Learning of Optimization Heuristics. In _2017 26th International Conference on Parallel Architectures and Compilation Techniques (PACT)_. 219–232.  https://doi.org/10.1109/PACT.2017.24
  * Desautels et al. (2014) Thomas Desautels, Andreas Krause, and Joel W. Burdick. 2014\.  Parallelizing Exploration-Exploitation Tradeoffs in Gaussian Process Bandit Optimization.  _J. Mach. Learn. Res._ 15, 1 (Jan. 2014), 3873–3923.
  * Dhakal et al. (2020) Aditya Dhakal, Junguk Cho, Sameer G. Kulkarni, K. K. Ramakrishnan, and Puneet Sharma. 2020\.  Spatial Sharing of GPU for Autotuning DNN models.  arXiv:2008.03602 [cs.NE]
  * Ding et al. (2015a) Yufei Ding, Jason Ansel, Kalyan Veeramachaneni, Xipeng Shen, Una-May O’Reilly, and Saman Amarasinghe. 2015a.  Autotuning Algorithmic Choice for Input Sensitivity. In _Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation_ (Portland, OR, USA) _(PLDI ’15)_. Association for Computing Machinery, New York, NY, USA, 379–390.  https://doi.org/10.1145/2737924.2737969
  * Ding et al. (2015b) Yufei Ding, Jason Ansel, Kalyan Veeramachaneni, Xipeng Shen, Una-May O’Reilly, and Saman Amarasinghe. 2015b.  Autotuning Algorithmic Choice for Input Sensitivity.  _SIGPLAN Not._ 50, 6 (June 2015), 379–390.  https://doi.org/10.1145/2813885.2737969
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017\.  Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks.  arXiv:1703.03400 [cs.LG]
  * Fursin et al. (2011) Grigori Fursin, Yuriy Kashnikov, Abdul Wahid Memon, Zbigniew Chamski, Olivier Temam, Mircea Namolaru, Elad Yom-Tov, Bilha Mendelson, Ayal Zaks, Eric Courtois, François Bodin, Phil Barnard, Elton Ashton, Edwin V. Bonilla, John Thomson, Christopher K. I. Williams, and Michael F. P. O’Boyle. 2011\.  Milepost GCC: Machine Learning Enabled Self-tuning Compiler.  _Int. J. Parallel Program._ 39, 3 (2011), 296–327.  http://dblp.uni-trier.de/db/journals/ijpp/ijpp39.html#FursinKMCTNYMZCa11
  * Kaufman et al. (2020) Samuel J. Kaufman, Phitchaya Mangpo Phothilimthana, Yanqi Zhou, and Mike Burrows. 2020\.  A Learned Performance Model for the Tensor Processing Unit.  arXiv:2008.01040 [cs.PF]
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-Supervised Classification with Graph Convolutional Networks. In _Proceedings of the 5th International Conference on Learning Representations_ (Palais des Congrès Neptune, Toulon, France) _(ICLR ’17)_.  https://openreview.net/forum?id=SJU4ayYgl
  * Koch (2015) Gregory Koch. 2015\.  Siamese neural networks for one-shot image recognition.
  * Leary and Wang (2017) Chris Leary and Todd Wang. 2017.  XLA: TensorFlow, compiled.  _TensorFlow Dev Summit_ (2017).
  * Li et al. (2020) Menghao Li, Minjia Zhang, Chi Wang, and Mingqin Li. 2020\.  AdaTune: Adaptive Tensor Program Compilation Made Efficient. In _34th Conference on Neural Information Processing Systems (NeurIPS 2020)_.  https://www.microsoft.com/en-us/research/publication/adatune-adaptive-tensor-program-compilation-made-efficient/
  * Lian et al. (2020) Dongze Lian, Yintao Xu Yin Zheng, Yanxiong Lu, Leyu Lin, Peilin Zhao, Junzhou Huang, and Shenghua Gao. 2020.  Towards Fast Adaptation of Neural Architectures with Meta Learning.  _ICLR_ (2020).
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In _Advances in Neural Information Processing Systems 32_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 8024–8035.  http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * Ragan-Kelley et al. (2013) Jonathan Ragan-Kelley, Connelly Barnes, Andrew Adams, Sylvain Paris, Frédo Durand, and Saman Amarasinghe. 2013.  Halide: A Language and Compiler for Optimizing Parallelism, Locality, and Recomputation in Image Processing Pipelines. In _Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design and Implementation_ (Seattle, Washington, USA) _(PLDI ’13)_. Association for Computing Machinery, New York, NY, USA, 519–530.  https://doi.org/10.1145/2491956.2462176
  * Rotem et al. (2019) Nadav Rotem, Jordan Fix, Saleem Abdulrasool, Garret Catron, Summer Deng, Roman Dzhabarov, Nick Gibson, James Hegeman, Meghan Lele, Roman Levenstein, Jack Montgomery, Bert Maher, Satish Nadathur, Jakob Olesen, Jongsoo Park, Artem Rakhov, Misha Smelyanskiy, and Man Wang. 2019\.  Glow: Graph Lowering Compiler Techniques for Neural Networks.  arXiv:1805.00907 [cs.PL]
  * Soh et al. (2020) Jae Woong Soh, Sunwoo Cho, and Nam Ik Cho. 2020.  Meta-Transfer Learning for Zero-Shot Super-Resolution. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 3516–3525.
  * Sung et al. (2019) H. Sung, T. Chen, A. Eichenberger, and K. K. O’Brien. 2019.  POSTER: CogR: Exploiting Program Structures for Machine-Learning Based Runtime Solutions. In _2019 28th International Conference on Parallel Architectures and Compilation Techniques (PACT)_. 485–486.  https://doi.org/10.1109/PACT.2019.00057
  * Tomczak et al. (2019) Jakub M. Tomczak, Romain Lepert, and Auke Wiggers. 2019\.  Simulating Execution Time of Tensor Programs using Graph Neural Networks.  arXiv:arXiv:1904.11876
  * Vasilache et al. (2018) Nicolas Vasilache, Oleksandr Zinenko, Theodoros Theodoridis, Priya Goyal, Zachary DeVito, William S. Moses, Sven Verdoolaege, Andrew Adams, and Albert Cohen. 2018\.  Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions.  arXiv:1802.04730 [cs.PL]
  * Wilson et al. (2017) James T. Wilson, Riccardo Moriconi, Frank Hutter, and Marc Peter Deisenroth. 2017.  The reparameterization trick for acquisition functions.  arXiv:1712.00424 [stat.ML]
  * Winata et al. (2020) Genta Indra Winata, Samuel Cahyawijaya, Zihan Liu, Zhaojiang Lin, Andrea Madotto, Peng Xu, and Pascale Fung. 2020.  Learning Fast Adaptation on Cross-Accented Speech Recognition.  arXiv:2003.01901 [eess.AS]
  * Wu and Frazier (2016) Jian Wu and Peter I. Frazier. 2016.  The Parallel Knowledge Gradient Method for Batch Bayesian Optimization. In _Proceedings of the 30th International Conference on Neural Information Processing Systems_ (Barcelona, Spain) _(NIPS’16)_. Curran Associates Inc., Red Hook, NY, USA, 3134–3142.
  * Zeng et al. (2020) X. Zeng, T. Zhi, Z. Du, Q. Guo, N. Sun, and Y. Chen. 2020.  ALT: Optimizing Tensor Compilation in Deep Learning Compilers with Active Learning. In _2020 IEEE 38th International Conference on Computer Design (ICCD)_. 623–630.  https://doi.org/10.1109/ICCD50377.2020.00108
  * Zheng et al. (2020a) Lianmin Zheng, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu, Ameer Haj-Ali, Yida Wang, Jun Yang, Danyang Zhuo, Koushik Sen, Joseph E. Gonzalez, and Ion Stoica. 2020a.  Ansor: Generating High-Performance Tensor Programs for Deep Learning. In _14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20)_. USENIX Association, 863–879.  https://www.usenix.org/conference/osdi20/presentation/zheng
  * Zheng et al. (2020b) Size Zheng, Yun Liang, Shuo Wang, Renze Chen, and Kaiwen Sheng. 2020b.  FlexTensor: An Automatic Schedule Exploration and Optimization Framework for Tensor Computation on Heterogeneous System. In _Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems_ (Lausanne, Switzerland) _(ASPLOS ’20)_. Association for Computing Machinery, New York, NY, USA, 859–873.  https://doi.org/10.1145/3373376.3378508