# A Graph Deep Learning Framework for High-Level Synthesis Design Space
Exploration

Lorenzo Ferretti1, Andrea Cini211footnotemark: 1, Georgios Zacharopoulos3,
Cesare Alippi2,4, Laura Pozzi2
1University of California Los Angeles, 2Università della Svizzera italiana,
3 Hardvard University, 4 Politecnico di Milano
These two authors contributed equally. Corresponding author:
<EMAIL_ADDRESS>
###### Abstract

The design of efficient hardware accelerators for high-throughput data-
processing applications, e.g., deep neural networks, is a challenging task in
computer architecture design. In this regard, High-Level Synthesis (HLS)
emerges as a solution for fast prototyping application-specific hardware
starting from a C/C++ behavioural description of the application computational
flow. In the accelerator synthesis phase, designers apply HLS directives to
optimize the hardware implementation, by trading-off cost and performance.
This Design-Space Exploration (DSE) aims at identifying Pareto optimal
synthesis configurations whose exhaustive search is often unfeasible due to
the design-space dimensionality and the prohibitive computational cost of the
synthesis process. Within this framework, we effectively and efficiently
address the design problem by proposing, for the first time in the literature,
graph neural networks that jointly predict acceleration performance and
hardware costs of a synthesized behavioral specification given optimization
directives. The learned model can be used to rapidly approach the Pareto curve
by guiding the DSE, taking into account performance and cost estimates. The
proposed method outperforms traditional HLS-driven DSE approaches, by
accounting for arbitrary length of computer programs and the invariant
properties of the input. We propose a novel hybrid control and data flow graph
representation that enables training the graph neural network on
specifications of different hardware accelerators; the methodology naturally
transfers to unseen data-processing applications too. Moreover, we show that
our approach achieves prediction accuracy comparable with that of commonly
used simulators without having access to analytical models of the HLS compiler
and the target FPGA, while being orders of magnitude faster. Finally, the
learned representation can be exploited for DSE in unexplored configuration
spaces by fine-tuning on a small number of samples from the new target domain.
The outcome of the empirical evaluation of this transfer learning shows strong
results against state-of-the-art baselines in relevant benchmarks including
neural processing.

##  1 Introduction

In the last decade, specialised hardware has become a viable solution to
address the end of Moors’s Law [40] and the breakdown of Dennard scaling [9]
and deal with the constant growth in performance and efficiency requirements
of computer systems. Whitin this context, short time to market, higher design
productivity and reusability of existing modules are only a subset of the
challenges that have driven the Electronic Design Automation (EDA) industry
and research. In particular, the recent wide-spread of solutions based on
artificial intelligence and deep learning techniques [26, 41] has lead to an
increasing demand of hardware accelerators able to target a large variety of
computational workloads.

Automating the design process by exploiting the predictive power of modern
machine learning (ML) models is an appealing approach that, while accelerating
the development of computer architectures, would also allow the ML community
to benefit from the improved computing platforms. In fact, progress in one
field ripples through the other one, thus creating a positive feedback loop
and a virtuous cycle [10]. In this setting, graphs are natural candidates to
capture the functionally-dependant structure of software and hardware systems.
It is not surprising, then, that the advent of graph neural networks (GNNs)
[37, 4, 1] led to impressive developments in computer-aided hardware and
software design: from chip placement [32] to compilers for ML computational
graphs [56]. In this context, we advocate for the adoption of analogous
automation strategies to fill the gap between the increasing request for
efficient and effective hardware acceleration of existing applications and the
hardware design productivity.

Traditional Integrated Circuits (ICs) design methodologies rely on Hardware
Description Languages (HDLs) to describe logical components and their
interaction at a Register Transfer Level (RTL). This approach requires
designers to manually define the concurrent description of millions of
transistors working in parallel to carry out the desired computations. To
reduce the burden of this task, High-Level Synthesis (HLS) comes into play.
HLS tools enable to start the design process from high-level behavioural
specifications in C/C++/SystemC, hence avoiding designers to deal with the
tedious and error prone task of implementing the functionality at RTL level.
Besides specifying the desired behavior, as shown in Figure 1, designers can
guide the synthesis process by applying directives able to tune the resulting
RTL implementation according to target performance and cost requirements.
Synthesis directives allow to specify how to implement in hardware specific
software constructs such as loops, arrays and functions. For example, the
designer can use directives to tune the degree of hardware parallelization of
a loop by specifying a loop unrolling factor. While HLS allows to explore a
vast design space of micro-architectural variations by using different
directives, resulting performance and resource utilization of each
implementation cannot be determined a priori. In fact, exhaustive exploration
involves time-consuming syntheses, whose number grows exponentially w.r.t. the
number of applied optimizations. Moreover, among all the possible
configurations (i.e., combinations of directives), only a few are Pareto-
optimal from a performance and costs perspective. As designers are interested
in effective methodologies to automate the Design-Space Exploration (DSE)
process, the HLS-driven DSE problem consists in identifying as accurately as
possible the set of Pareto implementations while, at the same time, minimizing
the number of synthesis runs.

Figure 1: Example of HLS design flow. A behavioural description, synthesis
directives, technology library, and a target frequency are given to the HLS
tool, which generates as a result an RTL design, performance and cost reports,
as well as the synthesis scripts.

Recent works demonstrated the possibility to guide DSEs by exploiting the
notion of function similarity and the knowledge acquired from past
explorations performed on different functions [14, 24, 47] and have shown
strong empirical results besides the small number of source domains. In light
of this, we claim that data-driven approaches are the way forward for HLS-
driven DSE. In this work, we introduce the methodological framework where we
firmly believe that progress in the field is bound to happen. We propose both
a data representation able to capture the critical elements of the HLS process
and the data-processing tools to profit from such representation. At first, we
introduce a novel graph representation of computer programs, based on an
augmented hybrid control and data flow graph, capturing invariant properties
and relevant information from the HLS perspective. Then, we exploit this
representation to train a novel graph neural network model in a supervised
fashion, by fitting the model on a dataset of previously synthesized
configurations (behavioral specifications plus optimization directives) to
predict the latency and resource utilization corresponding to each point. To
the best of our knowledge, this is the first attempt to use graph
representation learning from software specifications to perform HLS-driven
design space exploration. We show that the learned model can be used for
effective DSE after fine-tuning on a small set of samples and that our method
compares favorably against state-of-the-art baselines on relevant benchmarks.
We refer to our framework as gnn4hls. We believe that the results achieved
here constitute a strong signal for the community that calls for a general
effort in collecting large datasets of syntheses to unlock the full potential
of graph deep learning solutions to the HLS-driven DSE problem. In order to
accellerate progress in this direction, methods and datasets presented here
together with a platform for data collection will be open sourced at the end
of the blind review process.

The rest of the paper is organized as follows. In Section 2 we define the
problem by introducing the main concepts and proper terminology for HLS-driven
DSE and graph neural networks. Then, in Section 3 we lay out the details of
our approach. In Section 4 we evaluate the proposed method on relevant
benchmarks. Finally, we discuss the related works in Section 5, draw our
conclusions and discuss future works in Section 6.

##  2 Background

###  2.1 High-Level Synthesis driven Design-Space Exploration

Given a software functionality, e.g., the sum_scan function from the Radix
Sort algorithm in Machsuite [36] (used as a running example in the rest of the
document), we define as _HLS design_ , or simply _design_ , the functionality
to be realized in hardware, and as _specifications_ the behavioural
description of the design in a high-level programming language such as C/C++.
The specification ($SW$) is given in input to the HLS tool together with the
target technology library ($Lib_{HLS}$) and a target frequency ($F$). The
result of the synthesis process is named _implementation_ , and it is an
automatically generated RTL code, usually in VHDL or Verilog. The resulting
RTL is coupled to a performance metric (e.g., latency or throughput), and a
cost metric (e.g., area or energy costs).

An implementation is generated by applying a set of _directives_ – specified
using compiler pragmas – to the $SW$, affecting the resulting performance and
costs. The set of directives affecting an implementation is named
_configuration_. Each directive is associated to a _target_ in the $SW$, which
can be either a label or a code construct (e.g., labeled statements, function
names, variables, etc.). In addition, a directive is characterized by its
_type_ and an associated _value_. Examples of directive types are loop
unrolling – affecting number of resources required to implement the loop body
in hardware and enabling its parallel execution – and array partitioning –
splitting the input array in multiple memory banks and enabling parallel
access to the data. The directive value forces a given directive type to a
specific value. As an example, an unrolling factor of $2$ doubles the logic
required to implement in hardware the loop body, enabling parallel hardware
execution of two iterations.

The HLS design flow and the different elements characterizing it are shown in
Figure 1. Figure 1(left) shows the inputs of the HLS design flow: a
behavioural description of an HLS-design – the sum_scan function in the Radix
Sort benchmark from MachSuite [36] – an example of pragmas applied to the
array and loop constructs of the specification, the technology library
adopted, and the target frequency. After being processed by the HLS tool, the
resulting implementation is generated as an RTL desing, with the synthesis
scripts and the performance and cost reports–Figure 1(right).

Given a _design_ $D$, a designer limits the set configurations to explore
during a DSE by defining a _configuration space_. The _configuration space_
$X_{D}$ is defined as the Cartesian product among the set of _directive
values_ $V_{i}$ associated to each $i$-th _directive_ , i.e.,
$X_{D}=V_{1}\times V_{2}\times...\times V_{N}$, where $N$ is the number of
considered _directives_. The size of the _configuration space_ is given by its
cardinality $|X_{D}|$. Given a configuration space $X_{D}$, a _design space_
$Y_{D}$ can be defined as the set of _implementations_ resulting from the
synthesis of the configuration in $X_{D}$.

##### Task formulation

The DSE problem is a Multi-Objective Optimization Problem (MOOP) having costs
and merit as objective functions. In the context of hardware design common
performance measure and cost are _latency_ , and _area_ or _power_
respectively. In this work we use as performance the _effective latency_
(LAT), i.e., the number of clock cycles required by the hardware
implementation to execute its functionality multiplied by the target clock of
the system. For cost we consider the percentage of resources (silicon)
utilization required to implement the IC in hardware. In this work, since our
target architecture is a Field Programmable Gate Array (FPGA), costs are
expressed in terms of number of Flip-Flops (FF), Look-Up Tables (LUT), Digital
Signal Processor (DSP), and Block RAM (BRAM)111For the HLS designs considered
in this work, the DSEs have not affected the number of BRAM; therefore, we
have not include BRAM estimation in our experiments.. In particular, the
objective is to identify a subset $P_{D}$ of the configuration space $X_{D}$
such that: $P_{D}=\\{x|x\in X_{D}$ and $x$ is Pareto$\\}$. A _Pareto
configuration_ ($p$) of a design $D$ is defined as: $p$ $\in$ $P$
$\Leftrightarrow$ $\nexists$ $x$ $\in$ $X_{D}$, $x$ $\neq$ $p$ $|$ $a_{x}$
$\leq$ $a_{p}$ $\wedge$ $l_{x}$ $\leq$ $l_{p}$. With $a_{p}$, $a_{x}$, and
$l_{p}$, $l_{x}$ being the cost ($a$) and merit ($l$) associated to the
implementation of $p$ and $x$ respectively.

###  2.2 Graph neural networks

Input behavioural specifications (programs) significantly differ in terms of
size, structure, constructs, and optimization directives. This variability is
hardly captured by vector representations which have been, we argue, one of
the main limiting factors of previous works in attempting to learn predictive
models of the HLS process [30, 47]. Differently, when modeling the problem in
the space of graphs we can use methods that naturally exploit existing
functional dependencies, account for the variability in the structure of
different specifications, and seamlessly transfer learned representations
across different configuration spaces. Furthermore, graph-processing
techniques permit to exploit the properties of graph representations (e.g.,
permutation invariance) inducing positive inductive biases that restrict the
hypothesis space explored by the learning system to plausible models.

We consider attributed directed graphs, with attributes associated to both
nodes and edges. In particular, a graph ${\mathcal{G}}$ is a tuple
${\left\langle{\bm{V}},{\bm{E}},{\bm{u}}\right\rangle}$, where
${\bm{V}}=\\{1,\dots,N\\}$ is a set of nodes,
${\bm{E}}=\\{(i,j)|i,j\in{\bm{V}}\\}$ is a set of edges and
${\bm{u}}\in{\mathbb{R}}^{u}$ is a global attribute vector. We denote by
${\bm{v}}_{i}\in{\mathbb{R}}^{v}$ the raw features associated with node
$i\in{\bm{V}}$ and with ${\bm{e}}_{i,j}\in{\mathbb{R}}^{e}$ the attribute
vector associated with edge $(i,j)\in{\bm{E}}$ connecting nodes $i$ and $j$.

Different works propose general frameworks to design GNNs: inspired by Gilmer
et al. [18] and Battaglia et al. [1], we consider a very general class of
message-passing neural networks (MPNNs) with global attributes where the
$t$-th propagation layer (or step) can be written as

$\displaystyle{\bm{v}}_{i}^{t}$
$\displaystyle=\tau^{t}_{\bm{v}}\left({\bm{v}}_{i}^{t-1},\texttt{AGGR}_{\bm{v}}\Big{\\{}\psi^{t}_{\bm{v}}\left({\bm{v}}_{j}^{t-1},{\bm{v}}_{i}^{t-1},{\bm{e}}_{j,i}\right);{(j,i)\in{\bm{E}}}\Big{\\}},{\bm{u}}^{t-1}\right)$
(1) $\displaystyle{\bm{u}}^{t}$
$\displaystyle=\tau_{\bm{u}}^{t}\left({\bm{u}}^{t-1},\texttt{AGGR}_{\bm{u}}\Big{\\{}\psi^{t}_{\bm{u}}\left({\bm{v}}_{i}^{t},{\bm{u}}^{t-1}\right);i\in{\bm{V}}\Big{\\}}\right),$
(2)

where the update ($\tau^{t}_{\bm{v}}$,$\tau^{t}_{\bm{u}}$) and message
($\psi^{t}_{\bm{v}}$,$\psi^{t}_{\bm{u}}$) functions can be implemented by any
differentiable function, e.g., Multi-Layer Perceptrons (MLPs). Aggregation
functions $\texttt{AGGR}_{\bm{u}}\\{{}\cdot{}\\}$ and
$\texttt{AGGR}_{\bm{v}}\\{{}\cdot{}\\}$ can be any permutation invariant
operation. Multi-layer GNNs are built by stacking several propagation layers,
which allow to aggregate at each node messages from different neighborhoods.

##  3 Methods

###  3.1 Data representation

Figure 2: Data representation. Control Flow Graph, Data flow graph and Hybrid
Control Data Flow Graph representations of the behavioural specification of
the function in Figure 1.

Control Flow Graphs (CFGs) are graphs representing the possible execution
paths in a program. The CFG is a directed graph ${\mathcal{G}}_{\texttt{CFG}}$
defined as the tuple
${\left\langle{\bm{V}}_{\texttt{CFG}},{\bm{E}}_{\texttt{CFG}}\right\rangle}$,
where ${\bm{V}}_{\texttt{CFG}}$ is the the set of nodes corresponding to the
basic blocks of the program, and ${\bm{E}}_{\texttt{CFG}}$ is the set of edges
representing the possible control flows among basic blocks. An example of CFG
for the sum_scan function is shown in Figure 2-(top). Traditionally, a basic
block is defined as a consecutive sequence of instructions without incoming
and outgoing branches except for the first and last instructions of the block
respectively. In this work we adopt a different definition of basic block. In
particular, in our proposed formulation, we differentiate basic blocks
according to the type of instructions they perform. We discriminate between
the following types of blocks: _loop blocks_ identifying basic blocks
including loop instructions, _read block_ including a single load instruction
from main memory, _writes block_ including a single store instruction to main
memory, _function block_ including a function invocation instruction, and
lastly _standard block_ being basic block including instructions performing
computations that do not belong to any of the above mentioned categories. The
effect of this representation and taxonomy affects the granularity of the CFG
representation, increasing the number of blocks w.r.t.the traditional one.
Figure 2-(bottom), shows the differences of the proposed CFG representation –
with a higher block granularity – w.r.t. the traditional one. This choice aims
at avoiding the limitation of approaches relying on a vector-based
representation of the program and directives [30], while, at the same time,
focusing only on the information that is more relevant from the HLS and DSE
perspective. Notably, recent works highlight the effectiveness of adopting a
similar taxonomy of basic blocks to capture similarities among points in
different configuration spaces [14]. Compared to previous works in program
analysis (e.g., Li et al. [29]), we include in our CFG representation only
corse-grained information on the types of operations performed in each block.
In addition to the node type, attribute vectors associated to each node
include block-type related information, such as: number of instruction in a
block, number of iterations of a loop block, presence of loop carried
dependencies, and other type of information extracted through static and
dynamic code analysis performed using custom LLVM [25] compiler passes (more
details in Section 4). While CFGs contain information about the execution flow
of a program, they do not model the flow of data and information. Data Flow
Graphs (DFGs) address this aspect. DFGs are used to represent the different
dependencies between the instructions of a program. In particular they
represent the use-def chains among variables in the program execution. The DFG
is a directed graph ${\mathcal{G}}_{\texttt{DFG}}$ defined as a tuple
${\left\langle{\bm{V}}_{\texttt{DFG}},{\bm{E}}_{\texttt{DFG}}\right\rangle}$,
where ${\bm{V}}_{\texttt{DFG}}$ is the a set of nodes corresponding to a
instruction in the input source code, and ${\bm{E}}_{\texttt{DFG}}$ is the set
of edges representing the possible data flow among instruction. The DFG
representation for the running example function is shown in Figure 2-(top).

##### Hybrid Control Data Flow Graph

HLS tools use instrumented Control Flow Graphs and Data Flow Graphs as program
representations to decide how to implement in hardware the design
functionality. In our approach we aim at using a similar representation
directly as input of a learning method. We propose a graph representation of
the software description including both CFG and the DFG information. In
particular, we augment the CFG representation by adding data flow edges and
nodes representing the input and output parameters of the function. Data flow
edges are added among the nodes involving operations affecting the input and
output parameters. These edges are identified by tracking the def-use chains
among parameter variables in the DFG and embedding them in the CFG. In
addition, edges among the parameter nodes and the CFG read and write blocks
are added to the set of edges (param flow). We indicate this Hybrid Control
Data Flow Graph as ${\mathcal{G}}_{\texttt{hls}}$ and we use it as input
representation in our methodology. To sum up, ${\mathcal{G}}_{\texttt{hls}}$
is defined as a tuple
${\left\langle{\bm{V}}_{\texttt{hls}},{\bm{E}}_{\texttt{hls}},{\bm{u}}\right\rangle}$,
where ${\bm{V}}_{\texttt{hls}}$ is the a set of nodes corresponding to basic
blocks and function parameters, and ${\bm{E}}_{\texttt{hls}}$ is the set of
attributed edges representing the control, data, and parameter flows. Each
edge attribute vector ${\bm{e}}_{i,j}$ is a $3$ dimensional feature vector
with a one hot encoding representation of the edge type. Lastly,
${\bm{v}}_{i}\in{\bm{V}}_{\texttt{hls}}$ is the feature vector associated to
each node with a one-hot-encoding representation of the nodes type and their
attributes, plus the value of each optimization directive. Figure 2-(bottom)
shows the Hybrid Control Data Flow Graph representations for the running
example function.

###  3.2 Graph neural networks for high-level synthesis design-space
exploration

Figure 3: gnn4hls. An encoder block maps the pre-process input representation
with standard MLPs. Propagation layers perform message passing and update the
global representation with multi-head attention layers. The final block, maps
the latent representation learned by the network into the regression targets.

Given a graph ${\mathcal{G}}_{\texttt{hls}}$ representing a program annotated
with optimization directives, we process it with message passing neural
networks by parametrizing the computation of messages exchanged between
neighboring nodes of the graph with MLPs. A schematic of the model is shown in
Figure 3: we describe at first each computational block in detail then discuss
the training procedure.

##### Encoder

The Encoder block maps node, edge, and global graph features into a first
hidden representation without performing any message-passing operation. The
Encoder is implemented by using standard MLPs as update functions

$\begin{split}{\bm{v}}^{0}_{i}=\texttt{MLP}_{{\bm{v}}}^{\texttt{enc}}\left({\bm{v}}_{i}\right),\end{split}\quad\begin{split}{\bm{u}}^{0}=\texttt{MLP}_{{\bm{u}}}^{\texttt{enc}}\left({\bm{u}}\right),\end{split}\quad\begin{split}{\bm{e}}^{\texttt{enc}}_{i,j}=\texttt{MLP}_{{\bm{e}}}^{\texttt{enc}}\left({\bm{e}}_{i,j}\right),\end{split}$

as using feed-forward layers before message passing has shown to be beneficial
to final performance in GNNs [49]. The Encoder block is followed by a stack of
propagation layers.

##### Propagation layer

Propagation layers are instantiated in the message-passing framework shown in
Equation 1. In particular, the node update and message functions are
implemented as:

${\bm{v}}_{i}^{t}={\texttt{MLP}}^{t}_{\tau_{\bm{v}}}\bigg{(}{\bm{v}}_{i}^{t-1}\mathbin{\|}\
\texttt{MEAN}\Big{\\{}{\texttt{MLP}}^{t}_{\psi_{\bm{v}}}\big{(}{\bm{v}}_{j}^{t-1}\mathbin{\|}\
{\bm{e}}^{\texttt{enc}}_{j,i}\big{)};(j,i)\in{\bm{E}}_{\texttt{hls}}\Big{\\}}\bigg{)},$
(3)

where $\mathbin{\|}$ is the vector concatenation operator and
$\texttt{MEAN}\\{{}\cdot{}\\}$ indicates that we aggregate incoming messages
by averaging them out. To update the global representation ${\bm{u}}^{t}$ we
use a MLP as update function that takes in input the concatenation of
${\bm{u}}$ at the previous propagation step and node attributes aggregated by
exploiting the attention mechanism [44, 45]. In particular, we use a MLP
($\texttt{MLP}_{\alpha}({}\cdot{})$) to compute a raw attention score for each
node attribute vector ${\bm{v}}_{i}^{t}$ given the global features
${\bm{u}}^{t-1}$; raw scores (i.e., logits) are then normalized over the nodes
of the graph with a softmax function. Normalized node scores are used to
aggregate node features processed by a third MLP. Putting all together, each
propagation layer updates the global representation as follows:

$\begin{gathered}\texttt{score}\left({\bm{u}},{\bm{v}}_{i}\right)=\texttt{softmax}\left\\{\texttt{MLP}_{\alpha}\Big{(}{\bm{u}},{\bm{v}}_{i}\Big{)};i\in{\bm{V}}_{\texttt{hls}}\right\\}\\\
{\bm{u}}^{t}={\texttt{MLP}}^{t}_{\tau_{\bm{u}}}\left({\bm{u}}^{t-1}\mathbin{\|}\
\texttt{SUM}\Big{\\{}\texttt{score}\left({\bm{u}}^{t-1},{\bm{v}}^{t}_{i}\right)\odot{\texttt{MLP}}^{t}_{\psi_{\bm{u}}}\big{(}{\bm{v}}_{i}^{t-1}\big{)};i\in{\bm{V}}_{\texttt{hls}}\Big{\\}}\right),\end{gathered}$
(4)

where ${}\odot{}$ is the element-wise multiplication operator and
$\texttt{SUM}\\{{}\cdot{}\\}$ indicates aggregation by graph-wise summation of
node features. In practice multiple attention heads can be used in parallel
for increased model capacity.

##### Regression head

After $T$ message-passing blocks, node representations are pooled in a single
vector using a permutation invariant aggregation function (e.g., by taking the
sum or the average of node representations, optionally weighted by learned
attention scores). The pooled representation is then concatenated to the
global attributes ${\bm{u}}^{T}$ leading to a vector representation of the
input graph. This feature vector is fed trough a last MLP which maps it to a
prediction of latency and resources as shown in Figure 3.

##### Training procedure and transfer learning

We train the GNN by supervised learning to predict the outcome of the HLS
procedure. We use data from several synthesized designs, with program
specifications relevant to different domains (we refer to Section 4 for more
details). While learning to predict the outcome of the synthesis process for
configurations spaces already partially explored is interesting from a
research perspective, we are interested in assessing the possibility of
exploiting the model for DSE. In particular, we aim at assessing if the
learned representation can support transfer to different domains (designs)
when only a few samples, or none, from the target configuration space are
available. We comment that our method differs from previous approaches, which
usually are domain specific and tied to the characteristics of the target
design space. Instead, our methodology is general and can easily incorporate
knowledge from different design spaces by simply including synthesized points
in the training dataset.

##  4 Experimental evaluation

The graph representation of HLS designs and the associated pragma values are
generated combining LLVM [25] compiler passes, Clang Abstract Syntax Tree
(AST) analysis, Frama-C [8] internal representation of program dependencies,
and HLS synthesis information from a recently published database of HLS-drived
DSEs [15]. The custom LLVM pass generates the CFG representation from the
compiler Intermediate Representation (IR) and performs static program analysis
to identify the block proprieties. In order to account for the information
lost in the LLVM IR representation, an AST visitor extracts and maps the SW
information to the CFG blocks. Data flow information is extracted from the
Frama-C program dependency analysis, and used to generate the data flow edges
of ${\mathcal{G}}_{\texttt{hls}}$. We generated graph representations from
$23$ different functions in MachSuite [36]. The considered functions include a
wide range of computational intensive applications such as: matrix-matrix
multiplication, sparse matrix-vector multiplication, sorting algorithms,
stencil computations, molecular dynamics, and forward passes of fully-
connected neural networks. For each design, we used configurations and
synthesis results available from db4hls [15], an open source data base of HLS
DSEs. The total number of configurations considered in this work is $103093$.
Configuration spaces contain from several hundreds, up to several thousand
design points. For the graph neural network, we use as global graph attributes
the number of LLVM instructions, the number of input parameters, and the
average value of each directive set within the configuration minus the mean
value of the directive sets computed over the entire configuration space. To
increase robustness to outliers, we also concatenate to the representation the
same values minus the median.

In the following at first we perform an experiment to asses the accuracy of
our model in inferring the performance and costs of unseen directive
combinations, then we switch to the DSE settings. Hyper parameters of the
models and full experimental setup are provided in the supplementary
materials.

###  4.1 Performance and cost estimation

Figure 4: Comparison among the gnn4hls approach proposed in this work and a
MLP ones adopted as a baseline.

Figure 5: Ablation study of gnn4hls graph structure.

Figure 6: Comparison between fine-tuned, prior-knowledge [14], and zero-shot
approaches.

We split the synthesized configuration space of each function in three folds,
keeping $70\%$ of the available points for training, $10\%$ for validation,
and $20\%$ for testing. Selecting a proper baseline for comparison is not easy
since none of the approaches existing in the literature can easily be extended
to our settings. The most similar approach is the one introduced by Kwon and
Carloni [24], where a MLP is trained in a multi-domain setting. However their
approach, based on multi-task learning [5], relies on training different input
and output layers for each domain, hence limiting flexibility. Furthermore,
they use only optimization directives as an input and predict normalized
scores for performance and costs instead of the actual latency and resources.
For these reasons we consider a DeepSets [50] model to be a more appropriate
baseline: in practice we use a node-level MLP followed by a permutation
invariant aggregation and a second MLP to process the aggregated features. The
network architectures were chosen to have a similar model complexity. The
boxplot in Fig. 6 shows, for each figure of merit, the median, $1^{st}$ and
$3^{rd}$ quartiles, and interquartile range of the mean absolute percentage
error (MAPE) of the two models over the 23 functions in the dataset. Results,
averaged over $5$ independent runs, show that our model drastically
outperforms the baseline by achieving an average MAPE, averaged over
performance and costs estimates, of $2.7\%$ against $15.4\%$: an improvement
of over $82\%$. Furthermore, the performance obtained by our model is
qualitatively similar to SoA simulator ones, which exploit an analytical model
of the HLS process, HLScope+ [6] for the latency, and MPSeeker [55] for
resources222A direct comparison with these methodologies was not possible
since these were not available open source. Thus, we compare w.r.t. the
performance of the original papers.. In particular, latency estimation
performance are comparable to the best from the SoA (1.1% MAPE of HLScope+ vs.
2.1% of gnn4hls), our area performance estimation outperforms the ones from
existing models. gnn4hls achieves 4.8%, 2.6% and 1.3% estimation for FFs,
LUTs, and DSPs compared to the 14.7%, 13.2%, 12.7% respectively from MPSeeker.
In addition, inference time is greatly reduced from seconds to tens of
milliseconds w.r.t. SoA alternatives. In particular, the network used here
requires $\approx 20\text{ms}$ to process a single point from the
get_delta_matrix_weights1 function on an Intel(R) Xeon(R) Silver CPU.

###  4.2 Ablation study

To evaluate the effectiveness of the proposed graph representation, we have
performed an ablation study on the graph edges. Fig. 6 shows results obtained
given 3 different ablations on the graph structure: no data flow edges, no
param edges, and both data flow and param edges removed. For all the
representation the same type of nodes and attributes have been considered.
Results show that the proposed Hybrid CDFG representation leads to the best
results. Note that the DFG can largely be inferred from Param edges, thus the
similar performance of this setting.

###  4.3 Design-space exploration

The second set of experiments aims at addressing DSE. Herein, we focus on
approximating the Pareto-frontier of a target HLS design, given the synthesis
outcomes of all the considered functions but the target one (i.e., we perform
a leave-one-out evaluation w.r.t. the available functions). We consider the
setting where the designer performs an initial naïve random sampling of the
configuration space, uses the synthesized points to fine tune the model and
then uses the model’s estimates over the configuration space to approximate
the Pareto curve.

In particular, we select only the configurations expected to be Pareto-optimal
by considering as cost the weighted sum of the utilized resources. We assessed
the quality of the DSEs measuring the Average Distance from Reference Set
(ADRS) [39, 14, 24] metric among the real Pareto-solutions and the one
estimated to be Pareto-optimal. A low value of ADRS implies a close
approximation of the real Pareto-frontier. To increase robustness to
prediction errors we iteratively select candidate Pareto-optimal points by
removing from the configuration space the already selected configurations and
recomputing the Pareto curve up to $5$ times. Results are shown in Figure 6.
In particular, we compare the performance of the fine-tuning approach against
the current state-of-the-art one on the considered dataset, namely the prior-
knowl. approach [14] (see Section 5). For the fine-tuning procedure we use a
maximum of $128$ points from the target domain, capped at $5\%$ of the
configuration space dimension; we fix the number of SGD updates to $150$, with
a batch size of $32$. We also compare the performance in the zero-shot setting
(no fine-tuning).

Figure 7: Effect of the inference from multiple Pareto-frontier on the ADRS
and the number of synthesis.

Figure 8: ADRS Comparison across different class of applications among gnn4hls
and prior knowl..

Figure 6 shows the performance obtained by the fine-tuned approach (averaged
over $40$ independent runs) w.r.t. prior-knowl. and the zero-shot model. As
previously mentioned, we consider up to the $5^{th}$ Pareto-front and compare
against the reported results for prior knowl. which iterates up to the
$10^{th}$ frontier. We considered only the portion of the configuration space
that is actually synthesizable (i.e., we considered only configurations
present in the database). Results show the distributions of the ADRS across
the $23$ considered functions. In particular, the fine-tuned model obtains
Pareto-frontier approximations comparable to the state of the art, while
reducing the number of outliers compared to prior knowl. and obtaining an
average ADRS of $0.20$ vs $0.45$: a remarkable improvement. This result is
particularly appealing when observing that our approach neither uses any
heuristic to perform the initial sampling, nor does it rely on domain
knowledge provided by the designer. Furthermore, our method provides the user
with performance and cost estimates of the candidate configurations, which
could be instrumental to further reduce the number of syntheses required to
obtain the desired performance and satisfy hardware constraints. Figure 8
shows how the ADRS scores change w.r.t. the number of iteratively estimated
Pareto frontiers during the DSE. While the ADRS decreases exponentially (plot
in logarithmic scale along the $y$ axis), the number of required syntheses
grows linearly. Compared to prior knowl. our approach requires a higher number
of syntheses. However, we argue that we might expect this gap to reduce
significantly when considering a larger dataset: the performance of the zero-
shot approach, in fact, should be considered in light of the fact that our
dataset contains only $23$ different designs. Finally, Figure 8 compares the
results of gnn4hls and prior knowl. across the different class of applications
considered in this work (see appendix for details on the taxonomy and the
complete set of results). Our method shows lower variance in the ML designs,
but a higher mean, and lower ADRS in linear algebra designs which are relevant
for deep learning.

##  5 Related works

##### Design Space Exploration approaches in High-Level Synthesis

In past years, the hardware design community has proposed different works to
address the HLS-driven DSE problem. A recent survey from Schafer and Wang [39]
summarize them. Among these works we can identify two main categories: _model-
based_ approaches, and _refinement-based_ ones. Model-based approaches [53,
35, 54, 52, 6] rely on estimates of performance and resource requirements of a
given optimization. These approaches require very few synthesis runs to
approximate the Pareto frontier, but, often, have difficulties while dealing
with multiple, interdependent optimizations. Conversely, refinement-based
methodologies are agnostic to the number and types of directives considered,
and rely on the outcome of few heuristically sampled synthesis runs as a
starting point for DSE. After the initial synthesis, the models aim to improve
the initial solutions using different strategies such as genetic algorithms
[38], simulated annealing [31], clustering [12] or local search techniques
[13]. Refinement-based approaches are not limited by the number and type of
synthesis directives pre-characterized, but usually converge more slowly to
the Pareto-frontier with respect to model-based approaches. Taking a different
stance, more recently Ferretti et al. [14] have proposed a DSE strategy able
to map the result of past DSE targeting different design to unseen ones. The
proposed approach searches for similarity in the source code of the already
explored designs, and, based on the result of past DSEs, decides how to
optimize the target one. This approach has been used as a baseline to compare
the DSEs performance of the experiment in Section 4.3. Similarly, a recent
work from [24] proposes a neural network model for mixed-sharing multi-domain
transfer learning to transfer the knowledge obtained from previously explored
design spaces in exploring a new target one.

##### Graph neural networks for hardware/software design.

First GNNs date back to models developed by Gori et al. [19] and Scarselli et
al. [37] and earlier ideas on how to process graph-structured data with
recurrent neural networks [43, 17]. In recent years, the field of graph deep
learning has surged in popularity and several architectural improvements and
variants of GNNs have found wide spread and adoption by the community [27, 23,
33, 20, 45, 2]. Among their many applications to structured data processing,
GNNs have been widely used as the learning system of choice in software
engineering to automate program analysis and code optimization [42, 29, 3,
56]. Furthermore, graphs have also been used to capture the structure of
hardware architectures. Notably, Mirhoseini et al. [32] used a GNN to learn a
transferable representation to tackle the critical hardware design problem of
chip placement. Finally, due to the flexibility of graph representations, GNNs
have be used to automate design processes in many other areas of science and
engineering: prime examples are in the discovery of new molecules [28, 48] and
in automatic robot design [46, 51].

##  6 Conclusion and future works

In this work we presented gnn4hls, a graph-based learning framework for HLS-
driven DSE. Compared against the state of the art, our method offers tools
that are general and that can easily be applied to any configuration space. A
key aspect of gnn4hls is its simplicity w.r.t.the its effectiveness. We show
that our method compares favorably against the state of the art w.r.t. both
quantitative and qualitative metrics. In the future, we plan to investigate
more advanced solutions from the few-shot learning literature to improve
transfer to unseen domains and to consider the a finer-grained representation
of basic blocks. Then, we argue for the possibility of replacing existing
heuristics to perform DSE with an exploration policy learned by (model-based)
reinforcement learning. To support breakthroughs in this direction, we renew
our invitation to the community in participating in a common effort for the
collection of datasets fully enabling the application of deep learning in the
context of HLS-driven DSE. We believe that this work represents a milestone
for the application of graph deep learning in EDA and a significant step
towards fully automated design of hardware accelerators with no human in the
loop.

## References

  * Battaglia et al. [2018] P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al.  Relational inductive biases, deep learning, and graph networks.  _arXiv preprint arXiv:1806.01261_ , 2018.
  * Bianchi et al. [2021] F. M. Bianchi, D. Grattarola, L. Livi, and C. Alippi.  Graph neural networks with convolutional ARMA filters.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * Bieber et al. [2020] D. Bieber, C. Sutton, H. Larochelle, and D. Tarlow.  Learning to execute programs with instruction pointer attention graph neural networks.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Bronstein et al. [2017] M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst.  Geometric deep learning: going beyond euclidean data.  _IEEE Signal Processing Magazine_ , 34(4):18–42, 2017.
  * Caruana [1997] R. Caruana.  Multitask learning.  _Machine learning_ , 28(1):41–75, 1997.
  * Choi et al. [2017] Y.-k. Choi, P. Zhang, P. Li, and J. Cong.  Hlscope+: Fast and accurate performance estimation for fpga hls.  In _2017 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)_ , pages 691–698. IEEE, 2017.
  * Clevert et al. [2016] D. Clevert, T. Unterthiner, and S. Hochreiter.  Fast and accurate deep network learning by Exponential Linear Units (ELUs).  In _4th International Conference on Learning Representations, ICLR_ , 2016.
  * Cuoq et al. [2012] P. Cuoq, F. Kirchner, N. Kosmatov, V. Prevosto, J. Signoles, and B. Yakobowski.  Frama-C: A software analysis perspective.  SEFM’12, page 233–247. Springer-Verlag, 2012.
  * Esmaeilzadeh et al. [2011] H. Esmaeilzadeh, E. Blem, R. St. Amant, K. Sankaralingam, and D. Burger.  Dark silicon and the end of multicore scaling.  In _Proceedings of the 38th Annual International Symposium on Computer Architecture_ , ISCA ’11, pages 365–376, 2011.
  * Fahim et al. [2021] F. Fahim, B. Hawks, C. Herwig, J. Hirschauer, S. Jindariani, N. Tran, L. P. Carloni, G. Di Guglielmo, P. Harris, J. Krupa, et al.  hls4ml: An open-source codesign workflow to empower scientific low-power machine learning devices.  _arXiv preprint arXiv:2103.05579_ , 2021.
  * Falcon [2019] e. a. Falcon, WA.  Pytorch lightning.  _GitHub. Note: https://github.com/PyTorchLightning/pytorch-lightning_ , 3, 2019.
  * Ferretti et al. [2018a] L. Ferretti, G. Ansaloni, and L. Pozzi.  Cluster-based heuristic for high level synthesis design space exploration.  _IEEE Transactions on Emerging Topics in Computing_ , (99):1–9, Jan. 2018a.
  * Ferretti et al. [2018b] L. Ferretti, G. Ansaloni, and L. Pozzi.  Lattice-traversing design space exploration for high level synthesis.  In _Proceedings of the International Conference on Computer Design_ , pages 210–217, Oct. 2018b.
  * Ferretti et al. [2020] L. Ferretti, J. Kwon, G. Ansaloni, G. D. Guglielmo, L. P. Carloni, and L. Pozzi.  Leveraging prior knowledge for effective design-space exploration in high-level synthesis.  _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , 39(11):3736–3747, 2020.
  * Ferretti et al. [2021] L. Ferretti, J. Kwon, G. Ansaloni, G. Di Guglielmo, L. Carloni, and L. Pozzi.  DB4HLS: A database of high-level synthesis design space explorations.  _arXiv preprint arXiv:2101.00587_ , 2021.
  * Fey and Lenssen [2019] M. Fey and J. E. Lenssen.  Fast graph representation learning with pytorch geometric.  _arXiv preprint arXiv:1903.02428_ , 2019.
  * Frasconi et al. [1998] P. Frasconi, M. Gori, and A. Sperduti.  A general framework for adaptive processing of data structures.  _IEEE transactions on Neural Networks_ , 9(5):768–786, 1998.
  * Gilmer et al. [2017] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl.  Neural message passing for quantum chemistry.  In _International Conference on Machine Learning_ , pages 1263–1272. PMLR, 2017.
  * Gori et al. [2005] M. Gori, G. Monfardini, and F. Scarselli.  A new model for learning in graph domains.  In _Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005._ , volume 2, pages 729–734. IEEE, 2005.
  * Hamilton et al. [2017] W. L. Hamilton, R. Ying, and J. Leskovec.  Inductive representation learning on large graphs.  In _NIPS_ , 2017.
  * Harris et al. [2020] C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, et al.  Array programming with numpy.  _Nature_ , 585(7825):357–362, 2020.
  * Kingma and Ba [2015] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations, ICLR_ , 2015.
  * Kipf and Welling [2017] T. N. Kipf and M. Welling.  Semi-supervised classification with graph convolutional networks.  In _International Conference on Learning Representations (ICLR)_ , 2017.
  * Kwon and Carloni [2020] J. Kwon and L. P. Carloni.  Transfer learning for design-space exploration with high-level synthesis.  In _Proceedings of the 2020 ACM/IEEE Workshop on Machine Learning for CAD_ , MLCAD ’20, page 163–168. Association for Computing Machinery, 2020.
  * Lattner and Adve [2004] C. Lattner and V. Adve.  LLVM: A compilation framework for lifelong program analysis & transformation.  In _Proceedings of the international symposium on Code generation and optimization_ , page 75, Mar. 2004.
  * LeCun et al. [2015] Y. LeCun, Y. Bengio, and G. Hinton.  Deep learning.  _nature_ , 521(7553):436–444, 2015.
  * Li et al. [2016] Y. Li, D. Tarlow, M. Brockschmidt, and R. S. Zemel.  Gated graph sequence neural networks.  In Y. Bengio and Y. LeCun, editors, _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_ , 2016.  URL http://arxiv.org/abs/1511.05493.
  * Li et al. [2018] Y. Li, O. Vinyals, C. Dyer, R. Pascanu, and P. Battaglia.  Learning deep generative models of graphs.  _arXiv preprint arXiv:1803.03324_ , 2018.
  * Li et al. [2019] Y. Li, C. Gu, T. Dullien, O. Vinyals, and P. Kohli.  Graph matching networks for learning the similarity of graph structured objects.  In _International Conference on Machine Learning_ , pages 3835–3845. PMLR, 2019.
  * Liu and Carloni [2013] H.-Y. Liu and L. P. Carloni.  On learning-based methods for design-space exploration with high-level synthesis.  In _Proceedings of the 50th Design Automation Conference_ , pages 1–6, June 2013.
  * Mahapatra and Schafer [2014] A. Mahapatra and B. C. Schafer.  Machine-learning based simulated annealer method for high level synthesis design space exploration.  In _Proceedings of the 2014 Electronic System Level Synthesis Conference_ , pages 1–6, 2014.
  * Mirhoseini et al. [2020] A. Mirhoseini, A. Goldie, M. Yazgan, J. Jiang, E. Songhori, S. Wang, Y.-J. Lee, E. Johnson, O. Pathak, S. Bae, et al.  Chip placement with deep reinforcement learning.  _arXiv preprint arXiv:2004.10746_ , 2020.
  * Monti et al. [2017] F. Monti, D. Boscaini, J. Masci, E. Rodola, J. Svoboda, and M. M. Bronstein.  Geometric deep learning on graphs and manifolds using mixture model cnns.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5115–5124, 2017.
  * Paszke et al. [2019] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems_ , pages 8024–8035. 2019.  URL http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf.
  * Pham et al. [2015] N. K. Pham, A. K. Singh, A. Kumar, and M. M. A. Khin.  Exploiting loop-array dependencies to accelerate the design space exploration with high level synthesis.  In _2015 Design, Automation Test in Europe Conference Exhibition (DATE)_ , pages 157–162, 2015.
  * Reagen et al. [2014] B. Reagen, R. Adolf, Y. S. Shao, G.-Y. Wei, and D. Brooks.  MachSuite: Benchmarks for accelerator design and customized architectures.  In _Proceedings of the IEEE International Symposium on Workload Characterization_ , pages 110–119, Oct. 2014.
  * Scarselli et al. [2008] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini.  The graph neural network model.  _IEEE transactions on neural networks_ , 20(1):61–80, 2008.
  * Schafer and Wakabayashi [2012] B. C. Schafer and K. Wakabayashi.  Machine learning predictive modelling high-level synthesis design space exploration.  _IET computers & digital techniques_, 6(3):153–159, May 2012.
  * Schafer and Wang [2020] B. C. Schafer and Z. Wang.  High-level synthesis design space exploration: Past, present, and future.  _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , 39(10):2628–2639, 2020.
  * Schaller [1997] R. R. Schaller.  Moore’s law: Past, present, and future.  _IEEE Spectr._ , 34(6):52–59, June 1997.
  * Schmidhuber [2015] J. Schmidhuber.  Deep learning in neural networks: An overview.  _Neural networks_ , 61:85–117, 2015.
  * Si et al. [2018] X. Si, H. Dai, M. Raghothaman, M. Naik, and L. Song.  Learning loop invariants for program verification.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc., 2018.
  * Sperduti and Starita [1997] A. Sperduti and A. Starita.  Supervised neural networks for the classification of structures.  _IEEE Transactions on Neural Networks_ , 8(3):714–735, 1997.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , pages 6000–6010, 2017.
  * Veličković et al. [2018] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio.  Graph Attention Networks.  _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=rJXMpikCZ.
  * Wang et al. [2019] T. Wang, Y. Zhou, S. Fidler, and J. Ba.  Neural graph evolution: Towards efficient automatic robot design.  _arXiv preprint arXiv:1906.05370_ , 2019.
  * Wang et al. [2020] Z. Wang, J. Chen, and B. C. Schafer.  Efficient and robust high-level synthesis design space exploration through offline micro-kernels pre-characterization.  In _2020 Design, Automation Test in Europe Conference Exhibition (DATE)_ , pages 145–150, 2020.
  * You et al. [2018] J. You, B. Liu, Z. Ying, V. S. Pande, and J. Leskovec.  Graph convolutional policy network for goal-directed molecular graph generation.  In _NeurIPS_ , pages 6412–6422, 2018.
  * You et al. [2020] J. You, Z. Ying, and J. Leskovec.  Design space for graph neural networks.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Zaheer et al. [2017] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola.  Deep sets.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper/2017/file/f22e4747da1aa27e363d86d40ff442fe-Paper.pdf.
  * Zhao et al. [2020] A. Zhao, J. Xu, M. Konaković-Luković, J. Hughes, A. Spielberg, D. Rus, and W. Matusik.  Robogrammar: graph grammar for terrain-optimized robot design.  _ACM Transactions on Graphics (TOG)_ , 39(6):1–16, 2020.
  * Zhao et al. [2017] J. Zhao, L. Feng, S. Sinha, W. Zhang, Y. Liang, and B. He.  COMBA: A comprehensive model-based analysis framework for high level synthesis of real applications.  In _Proceedings of the International Conference on Computer Aided Design_ , pages 430–437, Oct. 2017.
  * Zhong et al. [2014] G. Zhong, V. Venkataramani, Y. Liang, T. Mitra, and S. Niar.  Design space exploration of multiple loops on FPGAs using high level synthesis.  In _Proceedings of the International Conference on Computer Design_ , pages 456–463, Dec. 2014.
  * Zhong et al. [2016] G. Zhong, A. Prakash, Y. Liang, T. Mitra, and S. Niar.  Lin-analyzer: a high-level performance analysis tool for FPGA-based accelerators.  In _Proceedings of the 53rd Design Automation Conference_ , pages 136:1–136:6, June 2016.
  * Zhong et al. [2017] G. Zhong, A. Prakash, S. Wang, Y. Liang, T. Mitra, and S. Niar.  Design Space Exploration of fpga-based accelerators with multi-level parallelism.  In _Design, Automation & Test in Europe Conference & Exhibition (DATE), 2017_, pages 1141–1146. IEEE, 2017.
  * Zhou et al. [2020] Y. Zhou, S. Roy, A. Abdolrashidi, D. Wong, P. Ma, Q. Xu, H. Liu, P. Phothilimtha, S. Wang, A. Goldie, A. Mirhoseini, and J. Laudon.  Transferable graph optimizers for ml compilers.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 13844–13855. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/9f29450d2eb58feb555078bdefe28aa5-Paper.pdf.

##  Appendix A Dataset and experimental set-up

In this section we provide details on the dataset and the experimental setting
used for the experiments presented in the paper. All experiments were carried
out on a server with Intel Xeon Silver CPUs and Nvidia Titan Xp/V GPUs.

For developing the models and the infrastructure to run experiments we relied
on the following open-source tools and libraries:

  * •

LLVM [25];

  * •

FramaC [8];

  * •

numpy [21];

  * •

db4hls [15];

  * •

PyTorch [34];

  * •

PyTorch Geometric [16];

  * •

PyTorch Lightning [11];

The code used to run the experiments is attached as supplementary material and
it will be released on GitHub upon publication.

###  A.1 Dataset description

The synthesis data used to train and test the model come from the db4hls
database [15]. The database includes a collection of DSEs performed for HLS
designs of the MachSuite benchmark suite [36]. In this work we use the data
collected in the database according to the configuration space definition
described by Ferretti et al. [15]. The performed DSEs include up to $5$
different type of pragmas: resource type, array partitioning type, array
partitioning factor, loop unrolling, and function inlining. The pragma values
specified for array partitioning factor and loop unrolling are limited to
power of twos or integer divisors of the input/output array sizes and loop
trip-counts. We consider only a subset of the functions available in db4hls.
In particular, we select only HLS designs which do not require allocation of
arrays or structs. This limitation is due to the LLVM compiler analysis
performed to extract the CFG representation of the software specifications:
this limitation will be addressed in future works. The original designs from
MachSuite are compiled with the lowest optimisation level (-O0). Then, we
process the intermediate representation (IR) produced by LLVM through a custom
LLVM pass in order to generate the CFG and distinguish among the different
types of block. Due to some LLVM optimisation performed on the IR
representation, memory allocation are in some cases transformed into function
calls generating an undesired code transformation which affect the the
resulting CFG – i.e., additional call blocks are generated without an explicit
function invocation in the original specifications. We aim at addressing this
limitation in the next future.

In addition to the block type, the LLVM pass extracts block specific
information. This information is used to populate the block attribute vectors
used to represent each node of the hybrid control data flow graph
representation. In order to extract such information, we perform different
analyses of the LLVM IR, the Clang abstract syntax tree (AST) and the source
code. An LLVM pass generates the intermediate representation, which is used to
first identify basic blocks in the code and the traditional CFG
representation. Then, for each instruction in the basic blocks we check the
type of operation performed and we discriminate among the different types of
block. Basic blocks including store instructions are split separating the
store instruction from the predecessor and successor instructions. The newly
created store block is then is connected to the new basic blocks resulting
from the split. Read blocks are generated in the same way. Loop blocks are
identified from the IR. Loop information is extracted in order to model loop
carried dependencies, loop stride, and loop trip count. Similarly, call blocks
are identified from the IR. In this case information about the number of
parameters, number of function invocation, and number of LLVM instructions of
the invoked function are extracted and included in the attribute vector. From
the remaining uncategorized standard blocks, information about the number of
LLVM instructions executed in it are extracted and added to the attribute
vector.

Parameter blocks instead are generated combining information from the LLVM IR
and the AST analysis. In particular we extract information associated to the
parameter type (pointer or value), parameter data type – the size of the data
type –, and number of elements of pointer parameters. This last information is
required by the HLS tools in order to know the memory required by the hardware
accelerator. However, since this information is not preserved in the LLVM IR,
we have extracted it from a joint AST and source code analysis. To avoid large
numerical differences among attributes, we use logarithmic scale fot the
number of LLVM instructions, loop trip counts, array partitioning factors,
data types, and their related directive values.

Table 1 reports the designs considered in this work. For each function we
show: the application domain (used to group the applications in Figure 8), the
original benchmark name in MachSuite, the HLS design name, the size of the
designs in term of lines of code, the type of pragmas considered during the
exploration and the configuration space size.

Table 1: List of the functions considered in this work. The table shows:
application domain, the original benchmark from MachSuite [36], the target HLS
design to implement in hardware, number of lines of code, type of pragmas
considered for the exploration, configuration space size. Pragma types are
indicated by different symbols: resource $\blacktriangle$, array partition
type $\blacktriangledown$, array partition factor $\blacksquare$, loop
unrolling $\blacklozenge$, function inlining $\bigstar$.

Domain | Benchmark | HLS design | Lines of code | Type of pragmas | | CS |
---|---|---|---|---|---
|  | ncubed | 41 | $\blacktriangledown\quad\blacksquare\quad\blacklozenge\quad$ | 2744
| gemm | bbgemm | 45 | $\blacksquare\quad\blacklozenge\quad$ | 1600
Linear Algebra | spmv | ellpack | 28 | $\blacksquare\quad\blacklozenge$ | 1600
Sorting | radix sort | hist | 34 | $\blacktriangledown\quad\blacksquare\quad\blacklozenge$ | 4704
init | 29 | $\blacktriangle\quad\blacktriangledown\quad\blacksquare\quad\blacklozenge\quad$ | 484
last_step_scan | 32 | $\blacktriangledown\quad\blacksquare\quad\blacklozenge$ | 800
sum_scan | 31 | $\blacktriangle\quad\blacktriangledown\quad\blacksquare\quad\blacklozenge$ | 1280
local_scan | 32 | $\blacktriangledown\quad\blacksquare\quad\blacklozenge$ | 704
update | 36 | $\blacktriangledown\quad\blacksquare\quad\blacklozenge$ | 2400
ss_sort | 50 | $\blacksquare\quad\blacklozenge\quad\bigstar$ | 1792
Stencil |  | stencil2d | 39 | $\blacktriangledown\quad\blacksquare\quad\blacklozenge\quad$ | 1344
Stencil | stencil | stencil3d | 62 | $\blacktriangledown\quad\blacksquare\quad\blacklozenge\quad$ | 1536
Biology | md | knn | 71 | $\blacksquare\quad\blacklozenge$ | 1152
|  | get_delta_matrix_weights1 | 45 | $\blacktriangledown\quad\blacksquare\quad\blacklozenge$ | 21952
|  | get_delta_matrix_weights2 | 45 | $\blacktriangledown\quad\blacksquare\quad\blacklozenge$ | 31213
|  | get_delta_matrix_weights3 | 45 | $\blacktriangledown\quad\blacksquare\quad\blacklozenge$ | 21952
|  | product_with_bias_input_layer | 48 | $\blacksquare\quad\blacklozenge\quad\bigstar$ | 1372
|  | product_with_bias_second_layer | 48 | $\blacksquare\quad\blacklozenge\quad\bigstar$ | 686
|  | product_with_bias_output_layer | 48 | $\blacksquare\quad\blacklozenge\quad\bigstar$ | 392
|  | take_difference | 44 | $\blacktriangledown\quad\blacksquare\quad\blacklozenge$ | 512
|  | get_oracle_activations1 | 48 | $\blacksquare\quad\blacklozenge$ | 2401
|  | get_oracle_activations2 | 48 | $\blacksquare\quad\blacklozenge$ | 1372
Machine Learning | backprop | update_weights | 127 | $\blacksquare\quad\blacklozenge$ | 1024

###  A.2 Global attributes

In addition to the attribute vectors associated to each node, we introduce a
global representation of the configuration with respect to its configuration
space.

Given a design and its associated configuration space, we define
${\bm{s}}\in{\mathbb{R}}^{5}$ as the vector having for each component the
average among the directive values set associated to each pragma type. As an
example, given a configuration space having directive value sets with 2
resource types (one-hot encoded), 2 partitioning types (one-hot encoded),
partitioning factors of $1$, $2$, $4$, $8$, unrolling factor of $10$, $20$,
$30$, and 2 options for function inlining (one-hot encoded), the resulting
configuration space vector will be ${\bm{s}}$ = $[0.5,0.5,3.75,20,0.5]$. Then,
given a specific configuration, we generate the vector
${\bm{c}}\in{\mathbb{R}}^{5}$ as the average vector among directive values of
the same type in that particular configuration. Similarly, we generate vectors
${\bm{s}}^{\prime}$ and ${\bm{c}}^{\prime}$ using the median among the
directive set values instead of the mean. Finally, given the number of
instructions in the function $l$ and the number of input parameters $p$ we
define the global attribute vector of the configuration ${\bm{u}}$ as:

${\bm{u}}=l\mathbin{\|}p\mathbin{\|}({\bm{s}}-{\bm{c}})\mathbin{\|}({\bm{s}}^{\prime}-{\bm{c}}^{\prime}),$
(5)

where ${}\mathbin{\|}{}$ is the vector concatenation operation. Intuitively,
this vector representation captures how much a configuration leans toward a
region of the design space. In practice, the vectors $({\bm{s}}-{\bm{c}})$ and
$({\bm{s}}^{\prime}-{\bm{c}}^{\prime})$ are also normalized so that each
element has unitary variance across the configuration space.

####  A.2.1 Design space exploration performance metric

The results of the DSEs have been evaluated in term of Average Distance from
Reference Set (ADRS). The ADRS metric is used to quantify the distance between
a reference curve $P$, and an approximated one $\bar{P}$. In our case, the
reference curve is the Pareto-frontier ground truth available computed over
the synthesized design avaiable in the dataset, while the approximated one is
the Pareto-frontier resulting from the DSE performed by using the trained
model. The ADRS for two objective functions is defined as:

$\displaystyle ADRS(\bar{P},P)=\bigg{[}\frac{1}{|P|}\sum\limits_{p\in
P}\min_{\bar{p}\in\bar{P}}(d(\bar{p},p))\bigg{]},$ (6) $\displaystyle
d(\bar{p},p)=\max\\{0,(A_{\bar{p}}-A_{p})/A_{p},(L_{\bar{p}}-L_{p})/L_{p}\\}$
(7)

where $A_{\bar{p}}$ and $L_{\bar{p}}$ are the area and latency of an element
of the reference Pareto-frontier, while $A_{p}$ and $L_{p}$ are the area and
latency of the approximated one. Intuitively, lower ADRS values implies
proximity among the approximated curve and the reference one. We consider as
altency ($\mathit{LAT}$) the number of clock cycles required by the hardware
implementation to execute the functionality, and, as measure of the area
($A$), the number aggregated values of $\mathit{FF}$, $\mathit{LUT}$,
$\mathit{DSP}$ in form of a linear combination of their utilisation.

$A=\frac{\mathit{FF}}{\mathit{FF}_{available}}+\frac{\mathit{LUT}}{\mathit{LUT}_{available}}+\frac{\mathit{DSP}}{\mathit{DSP}_{available}}$
(8)

This formulation allows to obtain a unique metric for the area costs
evaluating the overall utilisation of the resources required by an
implementation ($\mathit{FF}$, $\mathit{LUT}$, and $\mathit{DSP}$) with
respect to the ones available on a specific FPGA ($\mathit{FF}_{available}$,
$\mathit{LUT}_{available}$, and $\mathit{DSP}_{available}$).

##  Appendix B Hyperparameters and additional results

In this section first we provide details on the hyperparameters and
architecture used for the different models and how training was performed,
then we show additional (more complete) experimental results.

###  B.1 Performance and cost estimation experiment

We use a GNN with $4$ propagation blocks. All the MLPs required to implement
encoding, update and message functions are implemented as networks with a
single hidden layer and ELU activation function [7]. The MLPs processing the
node representations all have $128$ hidden units, while we use a width of
$256$ to process global graph features, both in the propagation blocks and in
the regression head. The MLPs computing node messages
($\texttt{MLP}^{t}_{\psi_{\bm{v}}}$) have and additional linear layer after
the nonlinear one. Finally, the MLP used to compute attention scores has an
hidden layer of $256$ units and a number of output units equal to the number
of attention heads with LeakyReLU ($0.2$ negative slope) activation as in
[45]. We use $2$ attention heads in parallel and we concatenate their outputs
before processing with the global update function $\tau_{\bm{u}}^{t}$. For the
baseline we use a node-level MLP with $5$ hidden layers with $512$ units each
and ReLU activation function, followed by $\texttt{SUM}\\{{}\cdot{}\\}$
aggregation and a second global MLP with a single hidden layer with the same
activation and number of neurons.

For training, we use as target values the natural logarithm of the true
values, and the mean absolute error as loss function. Models are trained for
$800$ epochs with a batch-size of $128$ without early stopping and by taking
as final model the one that achieves the lowest validation error across the
training epochs. For optimization we use the Adam optimizer [22] with an
initial learning rate of $0.001$ and cosine annealing (with no restarts) as a
schedule with a minimum learning rate of $0.0001$. We also clip the gradient
norm to a maximum value of $3$ to avoid learning instabilities. We did not use
any form of regularization since we did not observe any sign of overfitting.
We do not perform any additional scaling to the input features w.r.t. the
preprocessing steps described in A.1.

Table 2 and Table 3 show the estimation accuracy for gnn4hls and the DeepSets
baseline respectively. In particular, we show performance in terms of Mean
Absolute Error (MAE) and Mean Absolute Percentage Error (MAPE) for all the
prediction targets for each design. Note that the MAPE of the DSP estimate is
not defined for some function where DSP is not used, tables report a $0.0\%$
error for both models in such cases. Results are averaged over $5$ independent
runs, we do not report standard deviations for the sake of presentation
clarity.

Table 2: Detailed results of the performance and costs estimation experiment
for gnnhls.

| gnnhls
---|---
| MAPE | MAE
HLS design | LAT | FF | LUT | DSP | LAT | FF | LUT | DSP
ncubed | 1.9% | 3.67% | 3.48% | 3.18% | 2.37E+05 | 6.06E+02 | 1.21E+03 | 4.80E+00
bbgemm | 1.35% | 2.8% | 2.24% | 0.46% | 2.42E+05 | 3.13E+02 | 2.58E+02 | 1.35E-01
ellpack | 2.69% | 2.58% | 2.41% | 2.21% | 1.14E+04 | 8.04E+02 | 6.08E+02 | 6.37E+00
hist | 3.68% | 8.54% | 5.41% | 0.0% | 6.53E+02 | 6.82E+02 | 1.64E+03 | 1.73E-03
init | 2.86% | 15.03% | 4.7% | 0.0% | 4.19E+01 | 4.04E+01 | 3.78E+02 | 1.95E-03
last_step_scan | 8.31% | 35.95% | 8.0% | 0.0% | 6.77E+02 | 7.72E+02 | 2.47E+03 | 2.02E-03
sum_scan | 0.8% | 1.19% | 1.05% | 0.0% | 5.31E+00 | 1.05E+01 | 3.53E+01 | 1.22E-03
local_scan | 6.5% | 11.61% | 7.15% | 0.0% | 8.94E+02 | 7.87E+02 | 1.92E+03 | 1.23E-03
update | 1.41% | 4.49% | 2.35% | 0.0% | 4.45E+02 | 1.84E+02 | 6.11E+02 | 8.17E-04
ss_sort | 0.68% | 1.24% | 1.27% | 0.0% | 1.37E+04 | 5.42E+01 | 3.47E+02 | 1.40E-03
stencil2d | 3.75% | 3.62% | 2.57% | 6.94% | 1.19E+04 | 3.07E+02 | 9.08E+02 | 3.01E+01
stencil3d | 4.21% | 8.65% | 6.35% | 6.68% | 1.74E+04 | 1.46E+03 | 2.75E+03 | 1.46E+01
knn | 0.99% | 1.6% | 1.72% | 1.38% | 8.17E+03 | 3.12E+02 | 6.50E+02 | 1.68E+00
get_delta_matrix_weights1 | 0.62% | 0.83% | 0.69% | 0.61% | 1.97E+02 | 8.47E+01 | 1.12E+02 | 2.04E+00
get_delta_matrix_weights2 | 1.51% | 2.45% | 2.04% | 1.36% | 1.40E+03 | 4.48E+02 | 5.05E+02 | 1.35E+01
get_delta_matrix_weights3 | 0.59% | 0.78% | 0.65% | 0.4% | 7.35E+01 | 2.92E+01 | 3.20E+01 | 7.06E-01
product_with_bias_input_layer | 0.67% | 0.82% | 1.5% | 0.56% | 3.61E+02 | 6.62E+01 | 1.61E+02 | 7.54E-01
product_with_bias_second_layer | 0.31% | 0.28% | 0.27% | 0.18% | 1.22E+03 | 7.71E+00 | 3.38E+01 | 2.82E-02
product_with_bias_output_layer | 0.29% | 0.29% | 0.38% | 0.2% | 5.23E+01 | 3.94E+00 | 7.29E+00 | 3.18E-02
take_difference | 0.35% | 0.18% | 0.13% | 0.21% | 6.10E-01 | 2.53E+00 | 2.27E+00 | 4.91E-02
get_oracle_activations1 | 2.48% | 2.7% | 2.79% | 2.67% | 4.32E+03 | 1.70E+02 | 3.19E+02 | 1.74E+00
get_oracle_activations2 | 2.42% | 1.46% | 2.13% | 1.85% | 1.93E+02 | 8.76E+01 | 1.51E+02 | 1.46E+00
update_weights | 0.39% | 0.72% | 0.83% | 0.25% | 7.76E+03 | 5.13E+01 | 1.05E+02 | 3.48E-02
Avg. | 2.12% | 4.85% | 2.61% | 1.27% | 2.43E+04 | 3.17E+02 | 6.61E+02 | 3.39E+00

Table 3: Detailed results of the performance and costs estimation experiment
for the baseline.

| baseline
---|---
| MAPE | MAE
HLS design | LAT | FF | LUT | DSP | LAT | FF | LUT | DSP
ncubed | 8.24% | 29.81% | 17.44% | 29.44% | 7.72E+05 | 4.39E+03 | 4.57E+03 | 2.12E+01
bbgemm | 5.34% | 11.26% | 7.05% | 4.39% | 9.32E+05 | 1.32E+03 | 8.01E+02 | 1.20E+00
ellpack | 29.99% | 36.96% | 21.09% | 92.78% | 1.50E+05 | 5.86E+03 | 3.67E+03 | 5.13E+01
hist | 18.67% | 49.28% | 43.0% | 0.0% | 2.64E+03 | 1.79E+03 | 8.35E+03 | 2.96E-02
init | 8.47% | 53.91% | 273.64% | 0.0% | 9.41E+01 | 1.94E+02 | 6.10E+02 | 5.36E-02
last_step_scan | 27.21% | 47.21% | 12.9% | 0.0% | 1.64E+03 | 1.52E+03 | 3.15E+03 | 3.97E-02
sum_scan | 4.32% | 7.61% | 3.61% | 0.0% | 3.08E+01 | 5.21E+01 | 1.08E+02 | 1.84E-02
local_scan | 17.13% | 14.32% | 8.67% | 0.0% | 1.93E+03 | 1.15E+03 | 2.23E+03 | 4.92E-02
update | 6.46% | 19.73% | 12.77% | 0.0% | 2.05E+03 | 6.72E+02 | 2.78E+03 | 2.02E-02
ss_sort | 1.79% | 2.47% | 2.43% | 0.0% | 3.56E+04 | 1.14E+02 | 6.63E+02 | 8.55E-03
stencil2d | 12.35% | 7.82% | 5.51% | 19.29% | 3.56E+04 | 7.70E+02 | 1.92E+03 | 6.65E+01
stencil3d | 23.16% | 25.92% | 20.36% | 41.69% | 6.96E+04 | 4.00E+03 | 8.97E+03 | 6.51E+01
knn | 3.02% | 4.92% | 20.33% | 2.8% | 1.80E+04 | 8.60E+02 | 8.35E+03 | 3.44E+00
get_delta_matrix_weights1 | 5.1% | 8.23% | 6.67% | 9.1% | 1.49E+03 | 6.27E+02 | 7.78E+02 | 1.58E+01
get_delta_matrix_weights2 | 40.05% | 31.02% | 32.77% | 60.15% | 2.11E+04 | 2.93E+03 | 8.26E+03 | 8.60E+01
get_delta_matrix_weights3 | 3.59% | 6.68% | 4.55% | 4.12% | 4.25E+02 | 2.67E+02 | 2.39E+02 | 6.76E+00
product_with_bias_input_layer | 1.53% | 2.23% | 3.91% | 1.39% | 8.43E+02 | 1.74E+02 | 4.84E+02 | 1.20E+00
product_with_bias_second_layer | 1.24% | 1.69% | 5.28% | 1.36% | 4.99E+03 | 3.43E+01 | 7.76E+02 | 2.11E-01
product_with_bias_output_layer | 0.79% | 4.08% | 3.89% | 0.5% | 1.44E+02 | 6.12E+01 | 8.85E+01 | 7.73E-02
take_difference | 2.26% | 1.22% | 1.17% | 1.29% | 3.48E+00 | 2.60E+01 | 3.07E+01 | 4.72E-01
get_oracle_activations1 | 4.74% | 5.76% | 5.75% | 5.73% | 9.97E+03 | 5.66E+02 | 1.09E+03 | 4.23E+00
get_oracle_activations2 | 8.85% | 4.46% | 5.96% | 4.37% | 6.97E+02 | 3.30E+02 | 4.60E+02 | 4.39E+00
update_weights | 1.18% | 1.5% | 5.17% | 0.31% | 2.33E+04 | 1.07E+02 | 6.40E+02 | 4.38E-02
Avg. | 10.24% | 16.44% | 22.78% | 12.12% | 9.06E+04 | 1.21E+03 | 2.56E+03 | 1.43E+01

Table 4: DSE result comparison among gnn4hls and prior-knowl.

HLS design | gnn4hls | prior-knowl
---|---|---
ADRS | # of syntehsis | ADRS | # of synthesis
ncubed | 0.043 $\pm$ 0.042 | 146 $\pm$ 27 | 0.012 | 35
bbgemm | 0.016 $\pm$ 0.018 | 148 $\pm$ 29 | 0.007 | 46
ellpack | 0.032 $\pm$ 0.02 | 115 $\pm$ 21 | 0.034 | 65
hist | 0.107 $\pm$ 0.199 | 119 $\pm$ 20 | 0.007 | 46
init | 1.3 $\pm$ 2.36 | 61 $\pm$ 30 | 0.078 | 68
last_step_scan | 0.211 $\pm$ 0.166 | 104 $\pm$ 19 | 0.004 | 90
sum_scan | 0.030 $\pm$ 0.02 | 143 $\pm$ 17 | 0.136 | 25
local_scan | 0.167 $\pm$ 0.24 | 88 $\pm$ 10 | 0.005 | 71
update | 0.002 $\pm$ 0.002 | 97 $\pm$ 17 | 0.009 | 28
ss_sort | 0.042 $\pm$ 0.02 | 42 $\pm$ 10 | 0.0005 | 21
stencil2d | 0.193 $\pm$ 0.24 | 90 $\pm$ 20 | 0.015 | 46
stencil3d | 0.115 $\pm$ 0.19 | 97 $\pm$ 28 | 1.88 | 16
knn | 0.006 $\pm$ 0.004 | 284 $\pm$ 42 | 0.006 | 25
get_delta_matrix_weights1 | 0.087 $\pm$ 0.036 | 427 $\pm$ 73 | 0.002 | 139
get_delta_matrix_weights2 | 0.054 $\pm$ 0.017 | 525 $\pm$ 71 | 0.010 | 77
get_delta_matrix_weights3 | 0.087 $\pm$ 0.037 | 568 $\pm$ 88 | 0.030 | 222
product_with_bias_input_layer | 0.005 $\pm$ 0.005 | 215 $\pm$ 36 | 3.560 | 3
product_with_bias_second_layer | 0.0001 $\pm$ 0.001 | 49 $\pm$ 18 | 0 | 30
product_with_bias_output_layer | 0.003 $\pm$ 0.013 | 71 $\pm$ 32 | 2.5E-5 | 24
take_difference | 0.002 $\pm$ 0.005 | 224 $\pm$ 56 | 0.0002 | 8
get_oracle_activations1 | 0.048 $\pm$ 0.022 | 220 $\pm$ 29 | 2.907 | 67
get_oracle_activations2 | 0.013 $\pm$ 0.009 | 244 $\pm$ 41 | 0.051 | 19
update_weights | 0.0002 $\pm$ 0.0002 | 63 $\pm$ 24 | 1.1E-5 | 3

###  B.2 Design space exploration experiment

For the DSE experiments we pretrained a model for each one of the available
functions using a leave-one-out approach. Then we finetuned each pretrained
model on the target domain as described in Section 4.3. The model architecture
here is the same used for the previous experiment. In the fine-tuning stage we
used Adam with a constant learning rate of $0.001$ and clipping the gradient
norm to $10$. To evaluate the impact of the initial random sampling the
finetuning runs were repeated $20$ times with different random seeds.

Table 4 reports the DSEs result obtained by the gnn4hls framework compared
against the prior-knowl. approach from Ferretti et al. [14]. The table lists,
for all the considered function, the ADRS values obtained and the number of
synthesis required by the methodologies. For gnn4hls results are averaged over
$40$ different runs and we report the standard deviations.