\item Case \Rule{Join-Symbolic}: we use \Rule{Le-ToSymbolic}.
\item Case \Rule{Join-Bottom}: we use \Rule{Le-MoveValue}. We need the assumption
$\wfjoinhole{\Omega_m[.]}{v_m}$ for the premise that the hole of $\Omega_l[.]$
or $\Omega_r[.]$ (depending on whether $v_l = \bot$ or $v_r = \bot$) is not inside a shared loan.
\item Case \Rule{Join-MutBorrows}: we the the induction hypothesis for the inner
value (with states $\Omega_l[\emborrow{\ell_0}{[.]}]$ and
$\Omega_r[\emborrow{\ell_1}{[.]}]$) then \Rule{Le-Reborrow-MutBorrow-Abs}
to introduce the fresh region abstraction with the reborrow.
\item Case \Rule{Join-SharedBorrows}: we use \Rule{Le-Reborrow-SharedBorrow}; the premises
stating that there are corresponding shared loans in the context are proven by using the
premise of \Rule{Join-SharedBorrows} in combination with the assumption
\item Case \Rule{Join-MutLoans}: we use \Rule{Le-Reborrow-MutLoan-Abs} to insert
a fresh mutable loan and move the current loan to a fresh region abstraction;
the rules for $\toabs$ (\Rule{ToAbs-MutBorrow} then \Rule{ToAbs-MutLoan}) allow
us to conclude that the fresh region abstraction has the proper shape.
\item Case \Rule{Join-SharedLoans}: we use \Rule{Le-Reborrow-SharedLoan}.
\item Cases \Rule{Join-MutLoan-Other}, \Rule{Join-Other-MutLoan}:
we use \Rule{Le-Fresh-MutLoan-Abs} to introduce a fresh loan on the side
which doesn't have one, then use the induction hypothesis.
\item Cases \Rule{Join-SharedLoan-Other}, \Rule{Join-Other-SharedLoan}:
we use \Rule{Le-Fresh-SharedLoan} to introduce a fresh loan on the side
which does't have one, then use the induction hypothesis.
\item Cases \Rule{Join-Tuple}, \Rule{Join-Sum}: trivial by the induction
\item Case \Rule{Join-Same-MutBorrow}: trivial by the induction hypothesis.
\item Case \Rule{Join-Same-SharedLoan}: trivial by the induction hypothesis;
we have to use the premise that there are no $\bot$ in the inner values
to prove that we can maintain the assumption $\sloansincl{\Omega^{0}_m}{\Omega_m}$.
\item Case \Rule{Join-MutBorrows-Precise}: same as for the case
\Rule{Join-MutBorrows-Precise}, but we have to introduce
additional region abstractions (with \Rule{Le-Reborrow-MutBorrow-Abs}).
\item Case \Rule{Join-SharedBorrows-Precise}: same as for the case
\Rule{Join-SharedBorrows-Precise}, but we have to introduce
additional region abstractions (with \Rule{Le-Reborrow-SharedBorrow-Abs}).
\end{itemize}

\begin{lemma}[Join-States-Le]
\label{thm:join-states-le}
For all $\Omega^{0}_l$, $\Omega^{0}_r$, $\Omega_{acc}$, $\Omega_l$, $\Omega_r$,
$\Omega_j$ we have:
\begin{align*}
&(\forall\; m \in \{l,\,r\},\,
\Omega^{0}_m \le ((\projmarked{m}{\Omega_{acc}}) \cup \Omega_m)
\wedge\\
&\sloansincl{\Omega^{0}_m}{((\projmarked{m}{\Omega_{acc}}) \cup \Omega_m}))
\Rightarrow\\
&\Omega^{0}_l,\, \Omega^{0}_r \vdash \join{\Omega_l}{\Omega_r} \rightsquigarrow \Omega_j
\Rightarrow\\
&\forall\; m \in \{l,\,r\},\,
\Omega^{0}_m \le \projmarked{m}{(\Omega_{acc} \cup \Omega_j)}
\end{align*}
\end{lemma}

\noindent\textbf{Proof}:\\
By induction on
$\Omega^{0}_l,\, \Omega^{0}_r \vdash \join{\Omega_l}{\Omega_r} \rightsquigarrow \Omega_j$.
\begin{itemize}
\item Case \Rule{Join-SameAbs}: we pose $\Omega'_{acc} := \Omega_{acc},\, A$ (we add
the region abstraction $A$ to $\Omega_{acc}$) and use
the induction hypothesis.
\item Case \Rule{Join-Same-Anon}: similar to case \Rule{Join-SameAbs}; we pose
$\Omega'_{acc} := \Omega_{acc}, \_ \rightarrow v$ and use the induction hypothesis.
\item Case \Rule{Join-AbsLeft}: we pose $\Omega'_{acc} := \Omega_{acc},\, \lbox{A}$
and use the induction hypothesis.
\item Case \Rule{Join-AbsRight}: we pose $\Omega'_{acc} := \Omega_{acc},\, \rbox{A}$
and use the induction hypothesis.
\item Case \Rule{Join-Var}: we use \Thm{Join-Values-Le}, pose
$\Omega'_{acc} := \Omega_{acc},\, \overrightarrow{A}$, use the fact that
$\sloansinclname$ is transitive, and use the induction hypothesis.
\end{itemize}

\begin{lemma}[Collapse-Merge-Abs-Le]
For all $\Omega_l$, $\Omega_r$, $\Omega_{acc}$, $\Omega_l$, $\Omega_r$,
$\Omega_j$ we have:
\begin{align*}
\vdash A_0 \MergeAbs A_1 \merge A \Rightarrow
\forall\; m \in \{l,\,r\},\,
\projmarked{m}{A_0} \MergeAbs \projmarked{m}{A_1} \merge \projmarked{m}{A}
\end{align*}
\end{lemma}

\noindent\textbf{Proof}:\\
By induction on $\vdash A_0 \MergeAbs A_1 \merge A$.
\begin{itemize}
\item Case \Rule{MergeAbs-Union}: by the induction hypothesis.
\item Case \Rule{MergeAbs-Mut}: by the induction hypothesis.
\item Case \Rule{MergeAbs-Shared}: by the induction hypothesis.
\item Case \Rule{MergeAbs-Mut-MarkedLeft}: we use \Rule{MergeAbs-Mut} then
the induction hypothesis for the left projection, and directly use the
induction hypothesis for the right projection.
\item Case \Rule{MergeAbs-Mut-MarkedRight}: symmetric of previous case.
\item Case \Rule{MergeAbs-Shared-MarkedLeft}: we use \Rule{MergeAbs-Shared}
then the induction hypothesis for the left projection, and directly use
the induction hypothesis for the right projection.
\item Case \Rule{MergeAbs-Shared-MarkedRight}: symmetric of previous case.
\end{itemize}

\begin{lemma}[Collapse-Le]
\label{thm:collapse-le}
For all $\Omega$, $\Omega'$ we have:
\begin{align*}
\vdash \Omega \collapse \Omega' \Rightarrow
\forall\; m \in \{l,\,r\},\,
\projmarked{m}{\Omega} \le \projmarked{m}{\Omega'}
\end{align*}
\end{lemma}

\noindent\textbf{Proof}:\\
By induction on $\vdash \Omega \collapse \Omega'$.
\begin{itemize}
\item Case \Rule{Collapse-Merge-Abs}. By \Thm{Collapse-Merge-Abs} and
\Rule{Le-MergeAbs}.
\item Case \Rule{Collapse-Dup-MutBorrow}.
We note that, for $m \in \{l,\,r\}$, we have:
\begin{align*}
\projmarked{m}{\left(A \cup \{\;
\lbox{\emborrow{\ell}{\_}},\;
\rbox{\emborrow{\ell}{\_}}
\;\}\right)} &=
(\projmarked{m}{A}) \cup \{\;
\emborrow{\ell}{\_}
\;\}\\
\projmarked{m}{(A \cup \{\;
\emborrow{\ell}{\_}
\;\})}
\end{align*}
\item Case \Rule{Collapse-Dup-MutLoan}. Similar to previous case.
\item Case \Rule{Collapse-Dup-SharedBorrow}. Similar to previous case.
\item Case \Rule{Collapse-Dup-SharedLoan}. Similar to previous case,
but we use \Rule{Le-ToSymbolic} if we need to introduce a fresh
symbolic value.
\end{itemize}

\medskip
Finally, the combination of theorems \ref{thm:join-states-le} and \ref{thm:collapse-le} gives
us the target theorem (\ref{thm:join-collapse-le}).

\section{Loops}
\label{appendix:loop-list}

We describe here a more realistic example which uses a loop.
In this example, we recursively dive into a list so as to get a mutable borrow
to its last element. We could then use this borrow to append another list, for instance.
We define the type list as: $\elist \tau := \mu X.\, () + (\tau,\, \ebox X)$.
We also pose $\enil := \eleft{()}$ and $\econs{x}{t} := \eright{(x,\, \ebox t)}$ as
syntactic shortcuts.
We start in an environment where \li+l+ mutably borrows a list (symbolic) value from another
variable \li+l0+.

\begin{minted}[mathescape=true,escapeinside=||]{rust}
// $l_0 \mapstoemloan{0},\; l \mapstoemborrow{0}{\sigma_0}$
loop {
match *l {
Nil => { // $l_0 \mapstoemloan{0},\; l \mapstoemborrow{0}{\enil}$
break; // No need to compute a join here
Cons => { // $l_0 \mapstoemloan{0},\; l \mapstoemborrow{0}{(\econs{\sigma_1}{\sigma_2})}$
l = &mut (*(*l as Cons).1); // $l_0 \mapstoemloan{0},\; \_\mapstoemborrow{0}{(\econs{\sigma_1}{\emloan{\ell_1}})},\; l \mapstoemborrow{1}{\sigma_2}$
continue; // We join with the environment at the entry of the loop
\end{minted}

At the continue, we convert the anonymous value to a region abstraction
(with \Rule{Le-ToAbs}, \Rule{ToAbs-MutBorrow}, \Rule{ToAbs-Sum}, \Rule{ToAbs-Box},
\Rule{ToAbs-MutLoan}).
We join the environment at the entry of the loop with the environment at the
\li+continue+. There is nothing to do for \li+l_0+ because its value is unchanged (we use .
\Rule{Join-Same}).
For $A_0$ we use \Rule{Join-AbsRight}. Finally, for \li+l+ we use
\Rule{Join-MutBorrows}. We get:

\begin{align*}
&l_0 \mapstoemloan{0},\\
&\AbsI{0}{\rbox{\emborrow{\ell_0}{\_}},\, \rbox{\emloan{\ell_1}} },\\
&l \mapstoemborrow{2}{\sigma_3}\\
&\AbsI{1}{ \lbox{\emborrow{\ell_0}{\_}},\, \rbox{\emborrow{\ell_1}{\_}},\, \emloan{\ell_2} }
\end{align*}

We collapse the environment by merging $A_0$ and $A_1$.
By \Rule{MergeAbs-Mut-MarkedLeft} we simplify $\rbox{\emloan{\ell_1}}$
and $\rbox{\emborrow{\ell_1}{\_}}$. We then use \Rule{Collapse-Dup-MutBorrow}
to simplify $\rbox{\emborrow{\ell_0}{\_}}$ and $\lbox{\emborrow{\ell_0}{\_}}$
into $\emborrow{\ell_0}{\_}$. We get the following environment, which is a fixed-point.

\begin{align*}
&l_0 \mapstoemloan{0},\\
&\AbsI{2}{ \emborrow{\ell_0}{\_},\, \emloan{\ell_2} }\\
&l \mapstoemborrow{2}{\sigma_3}\\
\end{align*}

\end{document}


# NonGEMM Bench: Understanding the Performance Horizon of the Latest ML
Workloads with NonGEMM Workloads

Rachid Karami  UC IrvineIrvineCAUSA ,  Hemanth Kota  UC IrvineIrvineCAUSA ,
Sheng-Chun Kao  GoogleMTVCAUSA and  Hyoukjun Kwon  UC IrvineIrvineCAUSA

###### Abstract.

Abstract: Machine Learning (ML) operators are the building blocks to design ML
models with various target applications. GEneral Matrix Multiplication (GEMM)
operators are the backbone of ML models. They are notorious for being
computationally expensive requiring billions of multiply-and-accumulate.
Therefore, significant effort has been put to study and optimize the GEMM
operators in order to speed up the execution of ML models. GPUs and
accelerators are widely deployed to accelerate ML workloads by optimizing the
execution of GEMM operators. Nonetheless, the performance of NonGEMM operators
have not been studied as thoroughly as GEMMs. Therefore, this paper describes
NonGEMM Bench, a benchmark to study NonGEMM operators. We first construct
NonGEMM Bench using popular ML workloads from different domains, then perform
case studies on various grade GPU platforms to analyze the behavior of NonGEMM
operators in GPU accelerated systems. Finally, we present some key takeaways
to bridge the gap between GEMM and NonGEMM operators and to offer the
community with potential new optimization directions.

##  1\. Introduction

The success of machine learning (ML) in various problem domains, such as
computer vision (he2016deep, ; he2017mask, ; sandler2018mobilenetv2, ;
kolesnikov2021ViT, ; liu2021swin, ) and natural language processing
(kenton2019bert, ; achiam2023gpt, ; touvron2023llama, ), made ML workloads
pervasive in various computing platforms from edge to cloud devices. ML models
typically involve billions of multiply-and-accumulate (MAC) operations (e.g.,
497 billions of MAC operations for ResNet 50 (he2016deep, )). Among many
operators used in ML models, GEneral Matrix Multiplication (GEMM)-based
operators, such as CONV2D, FC (fully-connected layer), Linear, and BMM
(batched matrix multiplication) dominate in terms of the computational cost,
as presented in Figure 1. Therefore, GPUs and accelerators have focused on the
optimization of the GEMM-based operators and greatly enhanced the
computational performance (e.g., latency and throughput) and energy efficiency
of end-to-end ML model inference.

Figure 1. The latency breakdown into GEMM and non-GEMM operators on AMD EPYC
7763 + NVIDIA A100 GPU. We measure the latency on two popular models from
HuggingFace (a) GPT2-XL (batch 1) (achiam2023gpt, ) and (b) Vision Transformer
(batch 1) (kolesnikov2021ViT, ).

However, because the GEMM operators are being accelerated, the non-GEMM
operators, such as memory operations (e.g., reshape, view, and transpose),
normalization, and logit computation functions (e.g., Softmax), now account
for a considerable amount of the end-to-end latency, compared to that of GEMM
operators. Figure 1 shows the profiled latency breakdown into GEMM and non-
GEMM operators running inferences on state-of-the-art large language (GPT2-XL
(achiam2023gpt, )) and image classification (Vision Transformer
(kolesnikov2021ViT, )) models. The motivational data imply a major shift of
the landscape of Amdahl’s law for ML acceleration, indicating that we now need
to consider non-GEMM operators in the ML system optimization. However, no
study thoroughly explored the non-GEMM operator performance in latest models,
although it is important to quantitatively understand the current status.

Therefore, we propose NonGEMM Bench, which focuses on non-GEMM operator
analysis in popular ML models today. Unlike previous benchmarks, NonGEMM Bench
thoroughly analyzes NonGEMM operator performance, which provides important
insights on the ML system optimization with powerful accelerators. NonGEMM
Bench consists of 18 distinct models collected from Hugging Face and
Torchvision on various computer vision and natural language processing tasks,
and the models were selected based on their popularity, which makes NonGEMM
Bench aligned with the realistic use cases.

To deliver more insights in NonGEMM operators, NonGEMM Bench provides a
microbenchmark suite of NonGEMM operators. NonGEMM Bench supports arbitrary
NonGEMM operators in any models supported by PyTorch FX (reed2022torch, ),
which is compatible with a majority of the ML models. The flow allows users to
analyze any NonGEMM operators without manually writing custom functions,
unlike previous benchmark (longtailbench, ). In addition, NonGEMM Bench
provides input argument specification extracted from real data. This helps
analyze NonGEMM operators in realistic settings, which is crucial for some
operators whose behavior is heavily dependent on input arguments

Using this benchmark, we perform case studies on 18 popular Hugging Face and
Torchvision models on image classification, object detection, segmentation,
and generative language models (e.g., Llama 2 (touvron2023llama, ) and GPT-2
(radford2019language, )). We study the performance on seven different hardware
configuration ranging from mobile, dekstop, to data center, which includes
four CPU and three GPU models. In our case studies, we evaluate the effect of
GPU acceleration on the latency distribution across operators for the selected
models highlighting the contribution of NonGEMM operators to the end to end
latency of the accelerated model.

We will open-source NonGEMM Bench at the time of publication to facilitate the
ML system research aligned with the current performance horizon. Including
that, we make the following contributions:

  * •

We shed a light on the changed landscape of Amdhal’s law in ML system design.

  * •

We propose NonGEMM Bench, which provides detailed analysis on NonGEMM
operators in the 16 latest popular models collected from Hugging Face and
Torchvision.

  * •

To facilitate NonGEMM operator-aware optimizations, we organize a
microbenchmark of NonGEMM operators collected in the popular models and
provide realistic input arguments collected from realistic input data.

  * •

Using NonGEMM Bench, we perform case studies on seven different hardware
configurations ranging from mobile, desktop, to data center, and show the
change in the impact of NonGEMM operators on the execution time of the models.

##  2\. Background

###  2.1. ML Operators

ML operators constitute the building blocks for creating machine learning
models. Operators consist of a computation or operation that generates an
output tensor from an input tensor. Examples include compute-intensive
operators such as convolution (CONV2D) and matrix multiplication (linear, BMM,
etc.), and other less expensive (computationally) operators like activation
(RELU), and normalization functions (LayerNorm) for example. The combination
of ML operators with weights (if applicable) creates layers, which are often
cascaded many times to make the machine learning model deep. The various
permutations of operators and layers define the architecture of the deep
neural network (DNN).

Figure 2. A description of Linear (a) and Conv1D (c) operators as GEMM
operators example, and that of non-maximum suppression (he2017mask, ) (b) and
Layer Normalization (ba2016layernorm, ) (d) as a non-GEMM operator example.

####  2.1.1. GEMM-based operations

GEMM operators are computationally heavy operators based on GEneral Matrix
Multiplication (GEMM) operations which compute MAC operations inside tightly
nested loops typically causing them to dominate the ML workload. Matrix
Multiplication, Convolution (CONV1D and CONV2D) and Linear (fully connected)
layers are examples of the mostly used GEMM operators in DNN. Figure 2 (a) and
Figure 2 (c) illustrate the algorimths of Linear and CONV1D operators
respectively. Both operators loop over the input tensor dimensions and
populate the output tensor by computing a MAC inside the inner most loop.
Despite having different semantics, GEMM operators compute MACs in tightly
nested loops and can be expressed as a series of matrix multiplications or dot
products. For example, 2-dimensional convolution can be mapped to a matrix
multiplication as shown in (chetlur2014cudnn, ), and linear layers compute a
vector-matrix multiplication between an input feature tensors and the layers’
weights.

####  2.1.2. Non-GEMM Operators

Non-GEMM operators are building blocks found in ML models that are different
than GEMMs. They span different functionalities and unlike GEMMs, they are not
centered around a common architecture (e.g. tight loop nest with MAC). They
encompass memory operations, as well as non-linear functions (e.g. element-
wise Squareroot in Layer Normalization shown in Figure 2 (d)). Some operators
also introduce non deterministic behaviors through conditional statements,
like non-maximum suppression which algorithm is shown in Figure 2 (b). We
label non-GEMM operators based on their functionality and summarize them
according to the groups in Table 2.

  * •

Normalization functions were introduced to regularize the input across a
certain dimension. Batch Norm (ioffe2015batchnorm, ) and Layer Norm
(ba2016layernorm, ) are traditional normalization functions heavily used in DL
models from computer vision and NLP applications (achiam2023gpt, ),
(he2017mask, ). Both of these functions normalize the intermediate feature
tensors by computing the mean and standard deviation statistics over a
specific tensor dimension. For a tensor with dimensions (Batch, Channel,
Height, Width), Batch Norm normalizes over the batch dimension B while Layer
Norm performs the normalization across the last dimension of the tensor
(Figure 2 (d)). RMS Norm (zhang2019root, ) is another function that has been
recently used in the large language model Llama (touvron2023llama, ) that
regularizes the feature tensors based on the value of the root mean square.

  * •

Activation Functions introduce non linearity into the model. Activation
functions span from the straightforward ReLU which filters out negative tensor
values, to the more sophisticated GELU (hendrycks2016gaussian, ) suggesting a
probabilistic gaussian approach to dropping data points.

  * •

Memory Operations consist of operations related to the allocation and the
”view” of a tensor inside the memory. A tensor ”view” operation does not
modify the original input tensor in memory nor creates a new copy. It provides
the user with a new ”view” of the tensor with new dimensions by modifying the
tensor shape and the strides to access the elements. Operators dealing with
indexing, reshaping and concatenating tensors fall into this category as well.
An example of memory oprations would be the Pytorch reshape function used in
the implementation of the attention block in Hugging Face transformers library
to ensure the correct dimensions of the Key matrix before multiplying it with
the Query matrix.

  * •

Element-Wise Arithmetic operators cover all element wise arithmetic tensor
operations. An example would be applying element wise division to scale the
elements of tensors in the attention block as shown in Figure 3.

  * •

RoI Selection functions are used in FasterRCNN (ren2015faster, ) and MaskRCNN
(he2017mask, ) detection models. They filter down bounding boxes proposed by
the region proposal network (Figure 3 (b)) and align the filtered boxes to the
objects detected in the image. Non-Maximum Suppression (NMS) is an RoI
selection operator used in RCNN models described in Figure 2. Given a list of
scores and bounding box information, it drops bounding boxes based on a
threshold score then filters out overlapping boxes based on the Intersection
over Union (IoU) metric.

Figure 3. The architecture of three popular families of ML models.

###  2.2. ML Models and Popular Tasks

The diversity in operators equipped ML algorithm designers with the building
blocks required to architect models excelling in a wide range of tasks.
Computer Vision (CV) and NLP are domains where deep learning has succeeded and
shown great potential. Therefore, developers have designed models with
specific architectures leveraging distinct operators for each task in CV and
NLP. Figure 3 shows the difference in architecture and operators used for
models from different tasks. Classical models in image classification consist
of deep convolution networks made by deeply cascading blocks of CONV2D,
normalization and activation operators (sandler2018mobilenetv2, ). Object
Detection models also rely on deep convolution network as a backbone and they
pair it with different heads and operators to as shown in Figure 3. On the
other hand, NLP and language models rely on a completely different
architecture. They use transformer models and attention mechanism introduced
in (vaswani2017attention, ). Figure 3 describe a decoder only transformer
architecture which is used in popular language models like GPT.

In NonGEMM Bench, we collect models from the computer vision and NLP domains
to breakdown their performance and understand the contributions of each
operator, especially the non-GEMM ones.

##  3\. NonGEMM Bench

Figure 4. Overview of the different stages of NonGEMM Bench

To better understand the performance of ML models across application domains,
we collect popular CV and NLP models and we evaluate their performance on
different target platforms. To gain more insight on their operator structure
and how the operator type is affecting performance, NonGEMM Bench allows the
user to profile the models at the operator level. Unlike LongTail bench that
focuses on micro-benchmarking operators without an ATEN or cuda kernel
(longtailbench, ), NonGEMM Bench allows the user to breakdown the model’s
performance at the computational graph level in end to end inference
scenarios. Moreover, NonGEMM Bench can also provide a finer breakdown of the
lower level kernels implementing the graph operators hence, exhibiting macro
and micro benchmarking capabilities. In the rest of this section, we present
NonGEMM Bench and describe its model selection process and different
benchmarking flows.

###  3.1. NonGEMM Bench Model Description

To make sure NonGEMM Bench captures the latest ML workloads, we select 18
highly-downloaded models from HuggingFace (hfmodels, ) for the
representativeness of NonGEMM Bench. NonGEMM Bench covers four major task
domains in ML, which include image classification, object detection, image
segmentation, and natural language processing.

As listed in Table 1, NonGEMM Bench includes 18 models based on state-of-the-
art CNN and Transformer architecture. The number of parameters of the models
ranges from 0.85M to 7B, which shows the diverse model coverage of NonGEMM
Bench. NonGEMM Bench also includes variants of the same models (e.g.,
ViTBase16, ViT large16, and ViT huge variants of the Vision Transformer
(kolesnikov2021ViT, )), which target diverse use cases in different platforms
(e.g., smartphone, workstation, and data center).

####  3.1.1. Computer Vision

Deep Learning is widely used in the computer vision domain (kolesnikov2021ViT,
) (he2016deep, ) (ren2015faster, ). NonGEMM Bench covers three important task
domain of the computer vision as follows:

  * •

Image Classification (IC): Image classification refers to a computer vision
task that identifies a class label of a given image. Image classification
models extract features (i.e., high-level and -dimensional information of the
input image) from an input image and reports the class label utilizing the
features. NonGEMM Bench includes eight most popular IC models in HuggingFace
(hfmodels, ), which includes ResNet50 (he2016deep, ), MobileNetv2
(sandler2018mobilenetv2, ), three variants of Vision Transformer
(kolesnikov2021ViT, ), and three variants of Swin Transformer (liu2021swin, ).

  * •

Image Segmentation (IS): Image segmentation refers to a computer vision task
that identifies the area in a image for each class. Like IC models, IS models
also extract features and utilize them to identify objects located in an image
and spatially separate them by highlighting pixels belong to each object.
NonGEMM Bench includes two state-of-the-art IS models: Segformer
(xie2021segformer, ) and MaskFormer (cheng2021per, ).

  * •

Object Detection (OD): Object detection refers to a computer vision task that
identifies the location of objects in an image and outputs the bounding box of
each object. Like aforementioned models, OD models extract features. In
addition to that, OD models generate region proposal, which refers to the
candidate location and bounding boxes for objects in an image. Using RoI
processing non-GEMM operators listed in Table 2, OD models refine the raw
region proposal generated by a region proposal network. The refined RoIs are
used as inputs to the classifier CNN at the end, and the classifier determines
the class label of object inside each refeined RoI. NonGEMM Bench includes
three popular OD models (hfdetection, ): FasterRCNN (ren2015faster, ),
MaskRCNN (he2017mask, ), and DETR (carion2020end, ).

####  3.1.2. Natural Language Processing

Natural Language Processing (NLP) refers to tasks involving the analysis and
understanding of human (natural) language. Transformer (vaswani2017attention,
) based DNNs have become the most dominant neural network models in NLP. They
are the backbone of popular state of the art generative large language models
like GPT (radford2019language, ) and Llama (touvron2023llama, ). Figure 3 (c)
shows the layer architecture of GPT’s transformer. It consists of diverse GEMM
and NonGEMM operators implementing a self attention block followed by a feed
forward network. NonGEMM Bench includes five popular NLP models (hfnlp, ):
Bert (kenton2019bert, ), three variants of GPT2 (achiam2023gpt, ), and
Llama2-7b (touvron2023llama, ).

####  3.1.3. Datasets

To evaluate the collected models, NonGEMM Bench utilizes real data from
popular datasets in each domain. We use ImageNet 2012 (imagenet_cvpr09, ) and
MS COCO (lin2014microsoft, ) for computer vision tasks. As for language
models, we use wikitext dataset available on HuggingFace (hfwikitext, ).

###  3.2. Benchmark Suite

NonGEMM Bench profiling flow runs the model with inputs from real datasets and
captures the latency performance at the operator level. Additional operator
data (e.g. input tensor shapes) is extracted and used to drive NonGEMM Bench
microbench flow. NonGEMM Bench microbench flow analyzes the energy cost of
non-GEMM operators by using input tensors with dimensions based on real data
collected during the profiling flow.

####  3.2.1. NonGEMM Bench Frontend

In the first stage of NonGEMM Bench, we start by preprocessing the inputs
retrieved from the appropriate dataset. The data is fetched in its raw format
as images or text. Tokenization is applied to text data and transformations
are applied to images to obtain the input tensors. The graph is then generated
using the torch FX (FX) (reed2022torch, ) library from Pytorch. The FX tracer
is the tool used to generate the computational graphs of NonGEMM Bench models.
It performs a symbolic execution using a proxy data structure to capture the
operators and construct the graph nodes. A limitation of the FX tracer is its
inability to trace through modules with dynamic control flow. The tracer fails
if the model has a conditional or iterative statement which depends on the
inputs or dynamically computed intermediate tensors. Hence, as shown in Figure
4, the preprocessed inputs are also fed into the graph generation tool to
overcome the FX tracer’s limitation. By utilizing the inputs to evaluate
dynamic control flow conditional statements, the tracer completes NonGEMM
Bench’s frontend by generating an input dependent operator graph.

####  3.2.2. NonGEMM Bench Profiler

After obtaining the computational graph, the model is ready to be analyzed and
profiled. We implement a custom Profiling Interpreter by integrating the
Pytorch Profiler with the FX Interpreter to enable performance analysis at the
graph node level as we evaluate the model. The base FX Interpreter iterates
over the computational graph and runs the nodes in Pytorch eager mode
sequentially to execute the model. The custom interpreter also executes the
model by computing the graph’s nodes iteratively. During the computation of
each node, it calls the Pytorch Profiler to instrument the node’s execution,
recording the performance metrics of each operator at the low level kernel
granularity.

####  3.2.3. NonGEMM Bench Post Processing

After the completion of NonGEMM Bench profiling stage, Post Processing cleans
and aggregates the collected data into performance reports and visualization
plots. This stage groups the operators into categories based on the operator
functionality and reports the performance analysis. The reports show a
breakdown of the latency across operators and devices (CPU/GPU) as well as
additional information about the number of FLOPS and input tensor shapes.

####  3.2.4. NonGEMM Operator MicroBenchmark

Based on the workload analysis obtained by running NonGEMM Bench, we notice a
shift in the latency distribution between GEMM and NonGEMM operators after
accelerating the inference (discussed in details in Section 4). The latency
for NonGEMM operators is becoming more significant which indicates new
potential bottleneck operators. To better study the potential bottlenecks, we
identify the NonGemm operators with the highest contributions to the models
latency, and construct a microbenchmark summarized in Table 2 with the
operators that had the highest contributions to the models latency. The
microbenchmark highlights the non-GEMM operators their implementation library
and realistic input tensors sizes based on the datasets used in the profiling
stage.

Table 1. NonGEMM Bench Tasks and Models Domain | Application | Models | Model Size (# Params.) | Dataset
---|---|---|---|---
Computer Vision | Image Classification | ResNet50 | 0.85M | ImageNet
MobileNetv2 | 3.4M | ImageNet
ViT Base 16 | 86M | ImageNet
ViT large 16 | 307M | ImageNet
ViT huge 14 | 632M | ImageNet
Swin Transformer tiny | 29M | ImageNet
Swin Transformer small | 50M | ImageNet
Swin Transformer base | 88M | ImageNet
Object Detection | FasterRCNN | 42M | COCO
MaskRCNN | 44M | COCO
DETR | 41M | COCO
Segmentation | Maskformer | 102M | COCO
SegFormer | 3.7M | COCO
NLP | Language Models | GPT2 | 117M | wikitext
GPT2 Large | 762M | wikitext
GPT2 X-Large | 1.5B | wikitext
Llama 2-7B | 7B | wikitext
Bert | 110M | wikitext
Table 2. A list of non-GEMM operators collected from eight popular model variants selected from Hugging Face listed in Table 1. Operator Group | Operator | Model | Implementation | Example Input Tensor Shape
---|---|---|---|---
Activation | ReLu | DETR | Torch.nn.modules.activation | [2,64,533],
GELU | ViT-l16 | Torch.nn.modules.activation | [1, 97, 4096]
GELU | GPT2-XL | transformers.activations.GELUActivation | [1, 8, 6400]
SiLu | Llama-2 | Torch.nn | [1, 10, 11008]
Normalization | LayerNorm | Segformer | Torch.nn.modules.normalization | [2, 16384, 32]
BatchNorm2d | Segformer | Torch.nn.modules.batchnorm | [2, 256, 128, 128]
LlamaRMSNorm | Llama | transformers.models.llama | [1, 10, 4096]
FrozenBatchNorm2d | MaskRCNN | torchvision.ops.misc | [1, 1024, 50,68]
FrozenBatchNorm2d | DETR | transformers.models.detr | [2, 850, 256]
LayerNorm | DETR | Torch.nn.modules.normalization | [2, 850, 256]
Elem-wise Arithmetic | Add | Segformer | Torch.add | [2, 16384, 32]
Mul | Llama-2 | Torch.mul | [1, 10, 11008]
Neg | Llama-2 | Torch.neg | [1, 32, 10, 64]
TrueDiv | Torch.true_divide | Segformer | [2, 1, 16384, 256]
TrueDiv | GPT2-XL | Torch.true_divide | [1, 25, 8, 8]
Memory | Contiguous | Segformer | Torch.Tensor.contiguous | [2, 32, 128, 128]
Contiguous | Llama-2 | Torch.Tensor | [1, 10, 32, 128]
Permute | ViT-b16 | Torch.permute | [1, 768, 196]
Permute | GPT2-XL | Torch.permute | [1, 8, 25, 64]
Split | GPT2-XL | Torch.split | [1, 8, 4800]
View | GPT2-XL | Torch.Tensor.view | [1, 8, 1600]
Reshape | ViT-b16 | Torch.reshape | [1, 768, 14, 14]
Expand | ViT-b16 | Torch.Tensor.expand | [1, 1, 768]
Squeeze | Llama-2 | Torch.squeeze | [1, 1, 10, 128]
Logit Computation | Softmax | DETR | Torch.nn.Functional.softmax | [1, 25, 8, 8]
Softmax | Segformer | Torch.nn.Functional.softmax | [2, 1, 16384, 256]
RoI Selection | NMS | MaskRCNN | torchvision.ops.nms | [4663, 4]
Interpolation | Interpolate | Segformer | Torch.nn.Functional | [2, 256, 128, 128]

##  4\. Case Studies

In this section, we perform a thorough analysis of the performance data
generated by running NonGEMM Bench on multiple hardware platforms show in
Table 3. We collect the data by running NonGEMM Bench on three systems
configuration ranging from mobile CPUs to data center grade CPUs and GPUs to
capture the behavior of the operators when subjected to different acceleration
capabilities. We notice that for most of the models, the time spent to compute
non-GEMM operators increases noticeably with respect to the total execution
time when running the model on the GPU.

Table 3. Hardware platform configurations used for NonGEMM Bench case studies. Class | CPU | GPU
---|---|---
Data Center | AMD EPYC 7763 | NVIDIA A100
Workstation | Intel i9-13900K | NVIDIA RTX 4090
Mobile | Intel i7-13700HU | NVIDIA RTX 4060m
Mobile | Apple M2 Max | Apple M2 Max GPU111Because the support from PyTorch is not mature yet, we only use the CPU for profiling on M2 Max
Table 4. NonGEMM Bench Model aliases used in Figure 9 and Figure 11. Models | Batch Size | Model Alias
---|---|---
ViT Base 16 | 1 | Vt-b b-1
8 | Vt-b b-8
ViT large 16 | 1 | Vt-l b-1
8 | Vt-l b-8
Swin Transformer tiny | 1 | Sw-t b-1
8 | Sw-t b-8
Swin Transformer small | 1 | Sw-s b-1
8 | Sw-s b-8
Swin Transformer base | 1 | Sw-b b-1
8 | Sw-b b-8
FasterRCNN | 1 | FRCNN b-1
2 | FRCNN b-2
8 | FRCNN b-8
MaskRCNN | 1 | MRCNN b-1
2 | MRCNN b-2
8 | MRCNN b-8
DETR | 2 | DETR b-2
GPT2 | 1 | gpt2 b-1
64 | gpt2 b-64
GPT2 Large | 1 | gpt2-L b-1
64 | gpt2-L b-64
GPT2 X-Large | 1 | GPT2-XL b-1
Llama2-7B | 1 | Llama2-7b b-1
Bert | 1 | bert b-1
64 | bert b-64

###  4.1. Image Classification

As can be seen in Figure 5, there is a clear shift in the latency distribution
across operators after running the models on the GPU. GEMM operators dominated
the computation for all the classification models running on CPU only
platforms. Upon enabling GPU computations, the latency to run GEMMs decreased
causing the NonGEMM operators to occupy a considerable proportion of the total
execution time.

Figure 5. The breakdown between GEMM and NonGEMM operators for image
classification workloads.

For a batch size of 1, the proportion of time spent to compute NonGEMM
operators for ViT-b16 and ViT-l16 increases from 23.5% and 18% when running on
a CPU only platform to 60% and 55% respectively on the data center GPU
platform. Similarly for the workstation GPU system, the NonGEMM operators
execution time proportion increases to 56% and 45% for ViT-b16 and ViT-l16 to
almost split the total execution time evenly with GEMM operators. For the
three Swin transformers with a batch size of 1, the GEMM operators latency
proportion dropped when running the inference on both the GPU systems
(datacenter and workstation). Following the trend observed in the vision
transformers, the NonGEMM operators contribution to the overall latency
increases as well. For example, the small Swin transformer model spends 55% of
the computation time executing NonGEMM operators on the data center GPU system
compared to a modest 33% contribution on the CPU only system. On the
workstation GPU system, the small Swin transformer also computes NonGEMM
operators for 57% of the time, a 2x increase compared to the workstation CPU
system. When increasing the batch size, the latency distribution across GEMM
and NonGEMM operators of classification models running on GPU follows the
trend observed for a batch size of 1. However, this trend is less noticeable
for the largest vision transformer, ViT huge (632M parameters). For a batch
size of 8, the non-GEMM operator latency is 43% and 51% for ViT-l16 and base
Swin transformer respectively (Figure 5) running on the data center GPU, an
3.9x and 3x increase compared to running on a CPU only. However, the huge
vision transformer only shows a 1.25x increase in the latency proportion of
non-GEMM operators when accelerating the inference on the GPU.

The non-GEMM operators with the most significance to the execution time of
classification models are presented in Figure 9. As shown in Table 5,
Normalization operators are the most expensive non-GEMM operator in
classification models when running on the data center GPU. It accounts for
19.1% of the total execution time on average. Figure 9 and Figure 11 show that
Arithmetic and Memory operators are also becoming expensive on GPU systems.

Based on NonGEMM Bench collected data, image classification models are
spending more time on NonGEMM operators on GPU systems with an average of 55%
for the evaluated models.

###  4.2. Object Detection

Figure 6. The breakdown between GEMM and NonGEMM operators for object
detection workloads.

Similar to image classification, non-GEMM operators latency becomes more
noticeable when running the models on GPU systems. On the data center, NonGEMM
operators dominated the execution time of FasterRCNN by consuming 82% of the
total inference time when we accelerate the workload compared to 46% when
running the CPU only (Figure 6). DETR and MaskRCNN showed a similar behavior
after acceleration, with a 2x increase in the time spend on non-GEMM
operators. DETR computed NonGEMM operators for 77% of the inference time on
the data center GPU, while MaskRCNN spent 80% on the data center and
workstation GPUs. For object detection models, Normalization operators exhibit
the highest execution time followed by activation and memory operators as
indicated in (Figure 9). All detection models in NonGEMM Bench use
normalization functions with custom implementations that do not leverage
available library normalization kernels. This leads leads to additional
overhead caused by launching and running multiple micro-kernels to execute the
normalization function. Overall, the evaluation data shows that detection
models get dominated by NonGEMM operators (78%) when executing the workloads
on a GPU system.

###  4.3. Segmentation

Figure 7. The breakdown between GEMM and NonGEMM operators for segmentation
workloads.

Interestingly, GEMM and NonGEMM operators take similar proportions of the
execution time when running on the data center and workstation CPU. We notice
in Figure 7 that the runtime for the Segformer model is slightly dominated by
NonGEMM operators on the workstation and data center CPUs. The execution time
follows a similar split between GEMM and NonGEMM operators on the mobile GPU
platform as well.

###  4.4. Language Models

Figure 8. The breakdown between GEMM and NonGEMM operators for language models
workloads.

On the mobile GPU, GPT2 and Bert executed NonGEMM operators for 77% and 70% of
the time with a 1.8x and 2.95x% increase from the CPU system respectively when
running on a single batch as indicated in Figure 8. For GPT2-large, the
NonGEMM operators clearly dominated the execution time on GPU systems as well.
Across the three GPU HW classes, non-GEMM operators occupy a higher ratio of
execution time (76% on average). Across GPU platforms and for a batch size of
1, Figure 8 highlights the NonGEMM operators domination of the runtime. Table
5 show that Activation (23% on GPT2) and element wise arithmetic (20% on Bert)
operators are the most expensive non-GEMM operators in language models running
on GPU.

Table 5. NonGEMM Bench Most Expensive NonGEMM Operator Group for selected NonGEMM Bench Models Running on Data Center GPU Platform. Application | Models | Batch Size | Operator Group | % of Exec. Time
---|---|---|---|---
Image Classification | ViT Base 16 | 1 | Normalization | 20
8 | Normalization | 17
ViT large 16 | 1 | Normalization | 20.5
8 | Elem-Wise Arithmetic | 17
Swin Transformer tiny | 1 | Normalization | 19
8 | Normalization | 18.5
Swin Transformer small | 1 | Normalization | 18
8 | Normalization | 17
Swin Transformer base | 1 | Normalization | 18
| 8 | Normalization | 18
Object Detection | FasterRCNN | 1 | Normalization | 60.5
2 | Normalization | 59
8 | Normalization | 49
MaskRCNN | 1 | Normalization | 58
2 | Normalization | 56
8 | Normalization | 49
DETR | 2 | Normalization | 40
Language Models | GPT2 | 1 | Activation | 23
64 | Activation | 16
GPT2 X-Large | 1 | Activation | 23
64 | Activation | 9
Llama2-7B | 1 | Elem-Wise Arithmetic | 23
Bert | 1 | Elem-Wise Arithmetic | 20
64 | Elem-Wise Arithmetic | 10
Figure 9. Breaking down the execution time of models running on the Data
Center (CPU only) and (CPU + GPU) configurations across operator groups.

####  4.4.1. Large Language Models

Figure 10. The breakdown between GEMM and NonGEMM operators for large language
models workloads. Figure 11. Breaking down the execution time of models
running on the workstation (CPU only) and (CPU + GPU) configurations across
operator groups.

Large language models latency is dominated by GEMMs when running on Data
center and mobile M2 CPUs (Figure 10), consisting of 79% of the total
inference time on average across platforms. When utilizing the GPU, the GEMM
latency no longer bottlenecks the total execution time. The GEMM operators
time drops to 30% and 22% for Llama2-7b and GPT2-XLarge respectively with non-
GEMM operators executing for 74% on the GPU configuration. Figure 12 breaks
down the execution time of the large language models and suggests that the
bottleneck are activation, arithmetic and memory operators. Element-wise
arithmetic operators are the most expensive NonGEMM operators for Llama2-7b
accounting for 23% of the total runtime (Table 5, while activation functions
occupy 23% of GPT2-XL total execution time.

Figure 12. The operator groups dominating the execution time of GPT2-XL and
LLama-2 on running on the data center GPU system with a batch size of 1.

GPT2 models implement their own GELU activation in the Hugging Face library
which does not have a direct mapping to a GELU GPU kernel when running in
Pytorch eager mode. This leads to additional overheads when launching
different kernels running to compute the activation function. Similarly, Llama
uses a normalization function that does not map to a GPU kernel in eager
Pytorch which leads to similar overheads that increase the latency of the non-
GEMM operators.

###  4.5. Key Observations

  * •

Accelerating the models using GPUs shifts the execution time distribution
across operators accentuating the proportion of NonGEMM operators. We average
the execution time proportions for GEMM and NonGEMM operators across all the
NonGEMM Bench experiments (percentages shown in Figure 5, Figure 6, Figure 7,
Figure 8, Figure 10). The average percentage of total inference time spent on
computing the NonGEMM operators increases from 27% across CPU systems to 55%
across GPU accelerated platforms.

  * •

NonGEMM operators’ total latency portion is increasing when running the model
on GPU irrespective of the grade of the GPU. NonGEMM Bench case studies
confirmed the increase of the significance of NonGEMM operators latency on the
three evaluation GPUs.

  * •

The significance of NonGEMM operators to the total runtime has increased when
accelerating the inference on GPUs for models with various architectures and
targeting different applications and task domains.

  * •

The most expensive NonGEMM operators after acceleration are not identical
across models from different appplications domains. Table 5, Figure 9 and
Figure 11 summarize the percentage of execution time occupied by NonGEMM
operators. In IC models, Normalization operators occupy the highest proportion
of execution time, with an average of 18.4%). In language models, Activation
and Element-wise Arithmetic operators are the most expensive NonGEMM
operators. They respectively account for 17.75% and 17.6% of the total
execution.

  * •

NonGEMM operators could be a potential new source of optimization to explore
when designing systems for ML. The results presented by NonGEMM Bench align
with the state of art work suggesting a novel architecture to accelerate
NonGEMM operators (ghodrati2024tandem, ).

##  5\. Related Works

ML workloads are rapidly evolving and are becoming more integrated in various
daily applications spanning different domains. Ensuring the best performance
possible of these workloads is of high importance to guarantee the quality of
service to the application user. Therefore, researchers have put a significant
effort in creating benchmarks to evaluate the performance of workloads from
different domains and understand their behavior across different platforms.

DeepBench (deepbench_repo, ) studies the performance of a very specific subset
of kernels and operators. While it may provide some useful optimization
insights at the operator level, it fails to capture the effect of operators
when running the model end to end.

MLPerf (reddi2020mlperf, ) is an industry standard state of the art inference
benchmark. It offers a framework to measure the end to end performance of
models on platforms ranging from high performance computers to mobile devices.
While it is ideal to measure end to end inference, MLPerf only support a
handful of models and does not provide an operator level performance
breakdown.

TorchBench (hao2023torchbench, ) presents a comprehensive benchmark suite for
the Pytorch framework consisting of a large number of DL models. However, it
is mainly designed to analyze the Pytorch software stack and identify
optimizations specific to it.

LongTail Bench (longtailbench, ) is a micro benchmark that studies the
operators without a specific compute library implementation. It focuses on
domain specific operators from CV models. It collects the performance metrics
of each operator by running it in a standalone fashion and feeding it
synthetic data. Despite shedding light on non-usually studied operators,
LongTail Bench isolates the operator and evaluates it on synthetically
generated data, hence, failing to capture the complexities of running the
operators within a model on real world data.

Given the limitations current state of the art benchamrks, and based on the
results shown in Section 4, we motivate the need for NonGEMM Bench to complete
the missing features in existing benchmarking works specified in Table 6.
NonGEMM Bench is an easily extensible popularity driven benchmark that
provides a performance breakdown at the operator level of popular ML models
evaluating inputs from real datasets.

Table 6. Comparing NonGEMM Bench with existing benchmarks. Benchmark | Latency Breakdown | Popularity Driven | Microbenchmark | Real dataset driven | Plug-model-and-profile
---|---|---|---|---|---
MLPerf (reddi2020mlperf, ) |  |  |  | ✓ |
LongTail (longtailbench, ) |  |  | ✓ |  |
TorchBench (hao2023torchbench, ) |  | ✓ |  | ✓ |
NonGEMM Bench | ✓ | ✓ | ✓ | ✓ | ✓

##  6\. Conclusion

In this work, we focus on analyzing NonGemm operators in popular machine
learning models. We construct NonGEMM Bench using popular models reflecting
realistic use scenarios to capture the performance of NonGEMM operators. We
evaluate NonGEMM Bench on different hardware platforms to offer the community
insights on the performance of accelerated ML workloads and potential new
optimization opportunities.

## References

  * [1] Hugging face models hub.  https://huggingface.co/models.  Accessed on March 8, 2024.
  * [2] Hugging face models hub - object detection.  https://huggingface.co/models?pipeline_tag=object-detection&sort=downloads.  Accessed on March 8, 2024.
  * [3] Hugging face models hub - text generation.  https://huggingface.co/models?pipeline_tag=text-generation&sort=downloads.  Accessed on March 8, 2024.
  * [4] WikiText-2 Dataset.  = ”https://huggingface.co/datasets/wikitext.  Accessed on April 12, 2024.
  * [5] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [6] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * [8] Bowen Cheng, Alex Schwing, and Alexander Kirillov.  Per-pixel classification is not all you need for semantic segmentation.  Advances in Neural Information Processing Systems, 34:17864–17875, 2021.
  * [9] Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer.  cudnn: Efficient primitives for deep learning.  arXiv preprint arXiv:1410.0759, 2014.
  * [10] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A Large-Scale Hierarchical Image Database.  In CVPR09, 2009.
  * [11] Soroush Ghodrati, Sean Kinzer, Hanyang Xu, Rohan Mahapatra, Yoonsung Kim, Byung Hoon Ahn, Dong Kai Wang, Lavanya Karthikeyan, Amir Yazdanbakhsh, Jongse Park, Nam Sung Kim, and Hadi Esmaeilzadeh.  Tandem processor: Grappling with emerging operators in neural networks.  In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 2024.
  * [12] Yueming Hao, Xu Zhao, Bin Bao, David Berard, Will Constable, Adnan Aziz, and Xu Liu.  Torchbench: Benchmarking pytorch with high api surface coverage.  arXiv preprint arXiv:2304.14226, 2023.
  * [13] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision (ICCV), pages 2961–2969, 2017.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [15] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  arXiv preprint arXiv:1606.08415, 2016.
  * [16] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. pmlr, 2015.
  * [17] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of NAACL-HLT, pages 4171–4186, 2019.
  * [18] Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Unterthiner, and Xiaohua Zhai.  An image is worth 16x16 words: Transformers for image recognition at scale.  In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
  * [19] Xiuhong Li, Shengen Yan, Lijuan Jiang, Ping Xu, Jinming Ma, Xingcheng Zhang, and Dahua Lin.  Longtail-bench: A benchmark suite for domain-specific operators in deep learning.  In 2022 IEEE International Symposium on Workload Characterization (IISWC), pages 282–295. IEEE, 2022.
  * [20] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [21] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF international conference on computer vision (ICCV), pages 10012–10022, 2021.
  * [22] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [23] Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, et al.  Mlperf inference benchmark.  In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), pages 446–459. IEEE, 2020.
  * [24] James Reed, Zachary DeVito, Horace He, Ansley Ussery, and Jason Ansel.  Torch. fx: Practical program capture and transformation for deep learning in python.  Proceedings of Machine Learning and Systems, 4:638–651, 2022.
  * [25] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [26] Baidu Research.  Deepbench, 2016.
  * [27] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 4510–4520, 2018.
  * [28] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [30] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo.  Segformer: Simple and efficient design for semantic segmentation with transformers.  Advances in Neural Information Processing Systems, 34:12077–12090, 2021.
  * [31] Biao Zhang and Rico Sennrich.  Root mean square layer normalization.  Advances in Neural Information Processing Systems, 32, 2019.


# Only long edges can erase the subcritical annulus-crossing phase of weight-
dependent random connection models

Emmanuel Jacob

> Abstract: This short note aims at complementing the results of the recent
> work [9], where Jahnel and Lüchtrath investigate the question of existence
> of a subcritical percolation phase for the annulus-crossing probabilities in
> a large class of continuum percolation models, namely the classical or
> generalized weight-dependent random connection models. Their work relates
> the absence of a subcritical phase to the occurrence of long edges in the
> graph, through a somewhat indirect criterium, that stays inconclusive in
> some models of interest. We provide a more direct and arguably simpler
> criterium, that is always conclusive when considering classical weight-
> dependent random connection models.

##  1\. Introduction

This note focuses on the existence of a subcritical annulus-crossing phase in
continuum percolation theory. We will consider classical or generalised
weight-dependent random connection models, but we first discuss the case of
the Poisson-boolean model (see [11, 4]) to motivate our work.

In the Poisson-boolean model, we consider a random graph
$\mathcal{G}^{\lambda}$ whose vertex set is given by a Poisson point process
of intensity $\lambda>0$ on $\mathbb{R}^{d}$. The vertices are given i.i.d.
radii (or weights), and two vertices $x$ and $y$ with radii $R_{x}$ and
$R_{y}$ are connected if $|x-y|<R_{x}+R_{y}$, namely if the balls $B(x,R_{x})$
and $B(y,R_{y})$ intersect. This model exhibits a phase transition at a
_critical percolation intensity_ $\lambda_{c}\in[0,+\infty)$: for
$\lambda>\lambda_{c},$ the graph $\mathcal{G}^{\lambda}$ has a.s. a (unique)
infinite component, while for $\lambda<\lambda_{c}$, the connected components
of $\mathcal{G}^{\lambda}$ are a.s. all finite. Of course, this is a true
phase transition only if there exists a subcritical phase, namely if
$\lambda_{c}>0$. This subcritical phase actually fails to exist if
$\mathbb{E}[R^{d}]=+\infty$, which is a degenerate case in the sense that the
vertices almost surely all have infinite degree, whatever the value of
$\lambda>0$ (see [11]). However, Gouéré proved in [4] that $\lambda_{c}>0$
whenever $\mathbb{E}[R^{d}]<+\infty$, showing the existence of a subcritical
phase transition for all nondegenerate models. His proof relies on a
renormalization lemma for the annulus-crossing probabilities. Taking
$\lambda>0$ small enough allows to initialize a simple renormalization
argument that shows that these probabilities tend to 0 when the size of the
annuli tend to infinity, which shows $\lambda\leq\lambda_{c}$. His technique
also leads naturally to the introduction of the _critical annulus-crossing
intensity_ $\widehat{\lambda}_{c}$, which automatically satisfies
$\widehat{\lambda}_{c}\leq\lambda_{c}$, as well as $\widehat{\lambda}_{c}>0$
by the aforementioned renormalization lemma. Interestingly, taking any
$\lambda<\widehat{\lambda}_{c}$ also provides renormalization techniques that
can be used to derive many properties of this phase, see [5]. It is thus
natural to ask whether we have $\widehat{\lambda}_{c}=\lambda_{c}$, namely
whether the two critical intensities coincide. This difficult question has
been investigated in [1, 2, 3], and it is proven in [2] that indeed
$\widehat{\lambda}_{c}=\lambda_{c}$ when $R$ follows a power-law, except
possibly for a countable number of exceptional power-law distributions.

We are however interested in a larger class of models that can feature long
edges and scale-free degree distribution, namely the classical weight-
dependent random connection models as in [7], or a generalized version in [9].
We first discuss the classical settings, which already include various
previously studied models like long-range percolation, scale-free percolation
or geometric inhomogeneous random graphs, and are closely related to
hyperbolic random geometric graph, see also [10] for a recent work on such
models. The vertices still have i.i.d. weights, but the probability of finding
an edge between a vertex $x$ of weight $W_{x}$ and a vertex $y$ of weight
$W_{y}$ is now of the form

$\varphi(W_{x},W_{y},|x-y|),$

for $\varphi$ some profile function satisfying some mild assumptions ensuring
the degrees are finite. Actually [7] use a more specific form

$\varphi(W_{x},W_{y},|x-y|)=\rho(g(W_{x},W_{y})|x-y|^{d}),$

but we will not require this form in our work. For these models, we still can
naturally define $\lambda_{c}$ and $\widehat{\lambda}_{c}$, that satisfy
$0\leq\widehat{\lambda}_{c}\leq\lambda_{c}$. The natural questions are now
whether we have equalities $\lambda_{c}=0$, or $\widehat{\lambda}_{c}=0$, or
$\lambda_{c}=\widehat{\lambda}_{c}$. The question of whether $\lambda_{c}=0$
has been investigated in various works, see in particular [8] for a fairly
general result.

In [9], it has been observed the interesting phenomenon that one may very-well
have $0=\widehat{\lambda}_{c}<\lambda_{c}$. This can be viewed as a “weak form
of degeneracy” for these models, that implies $\widehat{\lambda}_{c}=0$
without necessarily saying anything about $\lambda_{c}$. More precisely, they
relate the property $\widehat{\lambda}_{c}=0$ to the occurence of long edges
in the graph, as measured through a coefficient $\delta_{\rm{eff}}\geq 0$. In
short, they obtain that if $\delta_{\rm{eff}}>2$, then long edges are rare and
$\widehat{\lambda}_{c}>0$, while if $\delta_{\rm{eff}}<2$, long edges are
common and $\widehat{\lambda}_{c}=0$. Their results leave the case
$\delta_{\rm{eff}}=2$ open, which however includes some models of interest and
in particular the age-dependent random connection model of [6].

We elaborate on [9] by relating more directly the property
$\widehat{\lambda}_{c}=0$ to the occurence of long edges in the graph. Our
criterium always concludes whether we have $\widehat{\lambda}_{c}=0$ or not,
at least in the case of the classical weight-dependent random connection
models. Interestingly, [9] introduce and study a generalized version of
weight-dependent random connection models, where the connection probabilty
between two vertices is allowed to further depend on the surrounding vertices
of the graph. In this case we obtain partial results, which may complement
their work. Note however that in the full generality introduced in [9], it is
not necessarily the case that the percolation probabilities are monotonous in
$\lambda$. It is thus unclear that there is a single percolation phase
transition, and it seems dubious that much can be said without introducing
further assumptions.

We conclude this introduction with the following takeaway message from our
work: If it is likely to find long edges in our graphs, namely edges of length
of order $r$ in a ball of radius $r$, then this gives a weak form of
degeneracy that implies $\widehat{\lambda}_{c}=0$ without necessarily implying
$\lambda_{c}=0$. This leaves open the question whether we always have
$\lambda_{c}=\widehat{\lambda}_{c}>0$ whenever this weak form of degeneracy
does not occur.

##  2\. Framework

We use almost the same settings and notations as [9] to describe the
generalized weight-dependent random connection model. For $\lambda>0$, we
consider $\mathcal{G}^{\lambda}$ a random graph with vertex set $\mathcal{X}$
given by a Poisson Point Process (PPP) on $\mathbb{R}^{d}\times(0,1)$ with
intensity $\lambda>0$, where a vertex $\mathbf{x}=(x,u_{x})$ is said to be
located at position $x\in\mathbb{R}^{d}$ and with mark $u_{x}\in(0,1)$. We
then connect independently each unordered pair of vertices
$\mathbf{x},\mathbf{y}\in\mathcal{X}$ with probability

$\mathbf{p}(\mathbf{x},\mathbf{y},\mathcal{X}\backslash\\{\mathbf{x},\mathbf{y}\\}),$

where $\mathbf{p}$ is some functional invariant under translations and
rotations. It further satisfies the following integrability condition: for any
$\mathbf{x}$ and $\mathbf{y}$ in $\mathbb{R}^{d}\times(0,1)$, we have

$\mathbb{E}_{\lambda}[\mathbf{p}(\mathbf{x},\mathbf{y},\mathcal{X})]=\varphi_{\lambda}(u_{x},u_{y},|x-y|)$
(1)

for some measurable function
$\varphi_{\lambda}:(0,1)\times(0,1)\times(0,+\infty)$. Note $\mathbf{p}$ does
not depend on $\lambda$, but $\varphi_{\lambda}$ can depend on $\lambda$
because of the expectation taken with respect of a $\rm{PPP}$ with intensity
$\lambda$. We suppose:

  1. (1)

$\varphi_{\lambda}$ is symmetric in the first two arguments and non-increasing
in the third.

  2. (2)

The integral

$\int_{0}^{1}\int_{0}^{1}\int_{\mathbb{R}^{d}}\varphi_{\lambda}(s,t,|z|)\mathrm{d}z\mathrm{d}s\mathrm{d}t$

is finite.

We will sometimes introduce further hypotheses

  * •

Mixing hypothesis: We just repeat the hypothesis introduced in [9] to specify
local percolation and quantify the influence of far-apart vertices on the
connection mechanism. Write

$\displaystyle G(r,x)$ $\displaystyle:=\\{\text{there is a path in
}\mathcal{G}^{\lambda}\text{ from }B(x,r)\text{ to }B(x,2r)^{\complement}$
$\displaystyle\hbox{}\qquad\text{ using only vertices in }B(x,3r)\\}$
$\displaystyle G(r)$ $\displaystyle:=G(r,0).$

In these events we ask to find a crossing of some (large) annulus using only
vertices contained in a larger ball containing the annulus. We now say that
$\mathcal{G}^{\lambda}$ is mixing with index $\zeta>0$ if there exists
$C_{\rm{mix}}>0$ such that for all $\lambda>0$ and all $|x|>6r$, we have

$\left|\operatorname{Cov}({\mathbbm{1}}_{G(r)},{\mathbbm{1}}_{G(r,x)})\right|\leq
C_{\rm{mix}}\lambda r^{-\zeta}.$ (2)

  * •

Monotonicity hypothesis: We say the model satisfies the monotonicity
hypothesis if $\mathbf{p}$ is nondecreasing in the third argument, with the
understanding that adding vertices in your $\rm{PPP}$ cannot decrease the edge
probabilities of the existing vertices. If it holds, then $\varphi_{\lambda}$
is of course nondecreasing in $\lambda$, and more generally we can couple the
graphs $\mathcal{G}^{\lambda}$ and $\mathcal{G}^{\lambda^{\prime}}$ based on
the $\rm{PPP}(\lambda)$ and $\rm{PPP}(\lambda^{\prime})$ with
$\lambda<\lambda^{\prime}$, such that $\mathcal{G}^{\lambda}$ is a subgraph of
$\mathcal{G}^{\lambda^{\prime}}$. In particular, the percolation probability
becomes nondecreasing in $\lambda$ so that we can naturally introduce the
_classical critical percolation intensity_ $\lambda_{c}$, as well as the
_critical annulus-crossing intensity_ $\widehat{\lambda}_{c}$, defined by

$\displaystyle\widehat{\lambda}_{c}$
$\displaystyle:=\sup\left\\{\lambda>0,\mathbb{P}_{\lambda}\left(B(0,r)\leftrightarrow
B(0,2r)^{\complement}\right)\to 0\right\\}$
$\displaystyle=\inf\left\\{\lambda>0,\limsup\mathbb{P}_{\lambda}\left(B(0,r)\leftrightarrow
B(0,2r)^{\complement}\right)>0\right\\},$

where we write $A\leftrightarrow B$ for the existence in
$\mathcal{G}^{\lambda}$ of a path started at a vertex located in $A$ and
ending at a vertex located in $B$. Without the monotonicity assumption, we
still can define $\widehat{\lambda}_{c}$, but we cannot easily guarantee for
$\lambda>\widehat{\lambda}_{c}$ that the probabilities
$\mathbb{P}_{\lambda}\left(B(0,r)\leftrightarrow B(0,2r)^{\complement}\right)$
tend to 0 as $r$ tends to infinity. Note that [9] does not state the
monotonicty hypothesis, and introduce as an example a model that does not
satisfy this hypothesis.

  * •

Classical weight-dependent random connection model: In the classical model,
$\mathbf{p}$ does not depend on the surrounding vertices, so
$\mathbf{p}(\mathbf{x},\mathbf{y},\mathcal{X})=\mathbf{p}(\mathbf{x},\mathbf{y})=\varphi(u_{x},u_{y},|x-y|)$,
where of course $\varphi$ does not depend on $\lambda$. This is a strong
assumption, that implies both the mixing and monotonicity assumptions. This
assumption is also denoted as the “$\zeta=\infty$” case in [9].

This note aims at understanding in which cases we have a subcritical phase,
i.e. $\widehat{\lambda}_{c}>0$.

##  3\. Results

We claim that the main reason that can lead to an absence of a subcritical
phase is the presence of long edges in the graph. For $r>0$ and $c>0$, we
define the “long edge event”

$L(r,c):=\left\\{\exists x\sim y,|x|<r,|y-x|>cr\right\\},$ (3)

namely the event that we can find an edge of length at least $cr$ with one
endvertex in the ball of radius $r$ centered at the origin. We will also
consider $H(\lambda,c)$ the property

$\left\\{\limsup_{r\to\infty}\
\mathbb{P}_{\lambda}\left(L(r,c)\right)>0.\right\\}$

In words, $H(\lambda,c)$ holds if it is not unlikely to find long edges –
namely edges of length comparable to the size of the ball where we search for
them – when $r$ tends to infinity. The following simple property makes this
heuristic description actually quite pertinent:

######  Lemma 1.

$H(\lambda,c)$ does not depend on $c>0$.

###### Proof.

Clearly $H(\lambda,c)\Rightarrow H(\lambda,c^{\prime})$ for all
$c^{\prime}<c$. Consider now $c^{\prime}>c>0$. By spatial homogeneity, we have

$\mathbb{P}_{\lambda}\left(\exists x\sim y,x\in
B(0,r),|y-x|>c^{\prime}r\right)\geq\left(\frac{c}{c^{\prime}}\right)^{d}\mathbb{P}_{\lambda}\left(\exists
x\sim y,x\in B(0,c^{\prime}r/c),|y-x|>c^{\prime}r\right),$

and the result follows. ∎

In the following, we will write $H(\lambda)$ for the property $H(\lambda,1)$
(which by the lemma is actually the same as $H(\lambda,c)$ for any given value
$c>0$). Under the additional assumption that we work with a classical weight-
dependent random connection model, the following property makes the assumption
$H(\lambda)$ even more interesting.

######  Lemma 2.

For the classical weight-dependent random connection models, the property
$H(\lambda)$ does not depend on $\lambda>0$.

###### Proof.

For the classical weight-dependent random connection models and
$0<\lambda<\lambda^{\prime}$, we can couple $\mathcal{G}^{\lambda}$ and
$\mathcal{G}^{\lambda^{\prime}}$, so that $\mathcal{G}^{\lambda}$ is the
random subgraph of $\mathcal{G}^{\lambda^{\prime}}$ obtained by keeping each
vertex of $\mathcal{G}^{\lambda^{\prime}}$ independently with probability
$\lambda/\lambda^{\prime}$. Therefore

$(\lambda/\lambda^{\prime})^{2}\
\mathbb{P}_{\lambda^{\prime}}(L(r,1))\leq\mathbb{P}_{\lambda}(L(r,1))\leq\mathbb{P}_{\lambda^{\prime}}(L(r,1)).$

The result follows immediately. ∎

We now state our main result, which shows that the property $H(\lambda)$ is
intimately linked to the nonexistence of a subcritical phase.

######  Theorem 1.

  1. (1)

In the classical weight-dependent random connection model, the property
$H(\lambda)$ does not depend on $\lambda>0$, and is equivalent to
$\widehat{\lambda}_{c}=0$.

  2. (2)

In the generalized weight-dependent random connection model:

    1. (a)

If $H(\lambda)$ holds for all $\lambda>0$, then $\widehat{\lambda}_{c}=0$.

    2. (b)

If $H(\lambda)$ does not hold for some $\lambda>0$, and if additionally we
make the monotonicity and mixing assumptions, then $\widehat{\lambda}_{c}>0$.

Part (1) of the theorem directly follows from Part (2) together with Lemma 2,
so we only need to prove Part (2). Part (2)(a) follows from the simple
observation that the event $L(r,3)$ implies the existence of an edge between a
vertex in $B(0,r)$ and a vertex at distance larger than $3r$, thus necessarily
located outside $B(0,2r)$. Hence
$H(\lambda,3)\Rightarrow\lambda\geq\widehat{\lambda}_{c}$.

Finally, Part (2)(b) of the theorem is a consequence of the following lemma:

######  Lemma 3.

Suppose $H(\lambda,1/20)$ does not hold, and work under the monotonicity and
mixing assumptions. Then $\widehat{\lambda}_{c}>0$.

###### Proof.

Recall the event $G(r)$ defined in the mixing hypothesis, and introduce the
following events:

$\displaystyle C(r)$ $\displaystyle:=\\{B(x,r)\leftrightarrow
B(x,2r)^{\complement}\\},$ $\displaystyle F(r)$ $\displaystyle:=\\{\exists
x\sim y,x\in B(0,20r),|y-x|>r\\}.$

Using the mixing property, we obtain a renormalization lemma similar111The
only difference in our renormalization lemma is that we work with the events
$C(r)$ rather than with $G(r)$, which avoids us to separately have to control
the probability of $C(r)\backslash G(r)$. to the one stated in [9]:

$\displaystyle\mathbb{P}_{\lambda}(C(10r))$ $\displaystyle\leq
C\mathbb{P}_{\lambda}(G(r))^{2}+\mathbb{P}_{\lambda}(F(r))+CC_{mix}\lambda
r^{-\zeta}$ $\displaystyle\leq
C\mathbb{P}_{\lambda}(C(r))^{2}+\mathbb{P}_{\lambda}(F(r))+CC_{mix}\lambda
r^{-\zeta},$

where $C$ is an explicit constant depending only on the dimension. We do not
detail this classical argument, but just observe that for classical weight-
dependent random connection models, the last term with $C_{\rm{mix}}$
disappears, as in the original renormalization lemma of [4]. In order to use
such a renormalization lemma to deduce that $\mathbb{P}_{\lambda}(G(r))$ tends
to 0 as $r\to+\infty$, two ingredients are necessary:

  * •

We need the “error terms” $\mathbb{P}_{\lambda}(F(r))+CC_{mix}\lambda
r^{-\zeta}$ tending to 0 as $r\to\infty$. The mixing term clearly does, while
we obtain $\mathbb{P}_{\lambda}(F(r))\to 0$ by the hypothesis that
$H(\lambda,1/20)$ does not hold.

  * •

We need to initialize the renormalization argument by ensuring that
$\mathbb{P}_{\lambda}(C(r))$ is small on some interval $r\in[r_{0},10r_{0}]$.
To this end, we can simply bound roughly the probability of $C(r)$ by the
probability of finding any point of the $\rm{PPP}$ in the ball $B(0,r)$, and
_decrease_ sufficiently the value of $\lambda$ to make this probability small.
While decreasing the value of $\lambda$, we use the monotonicity property to
ensure the error term does not increase222Note that in models not satisfying
the monotonicity assumption, we could still conclude if we have a good control
of the probabilities $\mathbb{P}_{\lambda}(F(r))$, which in many cases should
not be too hard to obtain..

This classical renormalization technique thus allows to conclude. ∎

## References

  * [1] V. Ahlberg, Daniel; Tassion and A. Teixeira, “Sharpness of the phase transition for continuum percolation in $\mathbb{R}^{2}$,” Probability Theory and Related Fields 72 (2018) 25 – 581. https://doi.org/10.1007/s00440-017-0815-8.
  * [2] B. Dembin and V. Tassion, “Almost sharp sharpness for Poisson Boolean percolation,” arXiv:2209.00999 [math.PR].
  * [3] H. Duminil-Copin, A. Raoufi, and V. Tassion, “Subcritical phase of $d$-dimensional Poisson–Boolean percolation and its vacant set,” Annales Henri Lebesgue 3 (2020) 677–700. http://www.numdam.org/articles/10.5802/ahl.43/.
  * [4] J.-B. Gouéré, “Subcritical regimes in the Poisson Boolean model of continuum percolation,” The Annals of Probability 36 no. 4, (2008) 1209 – 1220. https://doi.org/10.1214/07-AOP352.
  * [5] J.-B. Gouéré and M. Théret, “Equivalence of some subcritical properties in continuum percolation,” Bernoulli 25 no. 4B, (2019) 3714 – 3733. https://doi.org/10.3150/19-BEJ1108.
  * [6] P. Gracar, A. Grauer, L. Lüchtrath, and P. Mörters, “The age-dependent random connection model,” Queueing Systems 93 (2019) 309–331. https://doi.org/10.1007/s11134-019-09625-y.
  * [7] P. Gracar, M. Heydenreich, C. Mönch, and P. Mörters, “Recurrence versus Transience for Weight-Dependent Random Connection Models,” arXiv:1911.04350 [math.PR].
  * [8] P. Gracar, L. Lüchtrath, and P. Mörters, “Percolation phase transition in weight-dependent random connection models,” arXiv:2003.04040 [math.PR].
  * [9] B. Jahnel and L. Lüchtrath, “Existence of subcritical percolation phases for generalised weight-dependent random connection models,” arXiv:2302.05396 [math.PR].
  * [10] J. Jorritsma, J. Komjáthy, and D. Mitsche, “Cluster-size decay in supercritical long-range percolation,” arXiv:2303.00712 [math.PR].
  * [11] R. Meester and R. Roy, Continuum Percolation.  Cambridge Tracts in Mathematics. Cambridge University Press, 1996.