11institutetext:  Laboratoire Informatique d’Avignon,
BP 91228 84911, Avignon, Cedex 09, France
11email<EMAIL_ADDRESS>22institutetext: École
Polytechnique de Montréal,
CP. 6128 succursale Centre-ville, Montréal, Québec, Canada

#  Artex is AnotheR TEXt summarizer

Juan-Manuel Torres-Moreno, 1122

###### Abstract

This paper describes Artex, another algorithm for Automatic Text
Summarization. In order to rank sentences, a simple inner product is
calculated between each sentence, a document vector (text topic) and a lexical
vector (vocabulary used by a sentence). Summaries are then generated by
assembling the highest ranked sentences. No ruled-based linguistic post-
processing is necessary in order to obtain summaries. Tests over several
datasets (coming from Document Understanding Conferences (DUC), Text Analysis
Conference (TAC), evaluation campaigns, etc.) in French, English and Spanish
have shown that Artex summarizer achieves interesting results.

Keywords: Automatic Text Summarization, Space Vector Model, Text extraction,
Ultra-stemming

##  0.1 Introduction

Automatic Text Summarization (ATS) is the process to automatically generate a
compressed version of a source document [15]. Query-oriented summaries focus
on a user’s request, and extract the information related to the specified
topic given explicitly in the form of a query [2]. Generic mono-document
summarization tries to cover as much as possible the information content.
Multi-document summarization is a oriented task to create a summary from a
heterogeneous set of documents on a focused topic. Over the past years,
extensive experiments on query-oriented multi-document summarization have been
carried out. Extractive Summarization produces summaries choosing a subset of
representative sentences from original documents. Sentences are ordered and
then assembled according to their relevance to generate the final summary
[10].

This article introduces a new method of summarization based in sentences
extraction on Vector Space Model (VSM). We score each sentence by calculating
their inner product with a pseudo-sentence vector and a pseudo-word vector.
Results show that Artex not only preserves the content of the summaries
generated using this new representation, but often, surprisingly the
performance can be improved. Artex could be an interesting and simple
algorithm using the extractive summarization paradigm. Our tests on trilingual
corpora (English, Spanish and French) evaluated by the Fresa algorithm
(without human references) confirm the good performance of Artex.

In this paper, related work is given in Section 0.2. Section 0.3 presents the
new algorithm of Automatic Text Summarization. Experiments are presented in
Section 0.4, followed by Results in Section 0.5 and Conclusions in Section
0.6.

##  0.2 Related works

Research in Automatic Text Summarization was introduced by H.P. Luhn in 1958
[9]. In the strategy proposed by Luhn, the sentences are scored for their
component word values as determined by tf*idf-like weights. Scored sentences
are then ranked and selected from the top until some summary length threshold
is reached. Finally, the summary is generated by assembling the selected
sentences in the original source order. Although fairly simple, this
extractive methodology is still used in current approaches. Later on, [3]
extended this work by adding simple heuristic features such as the position of
sentences in the text or some key phrases indicate the importance of the
sentences. As the range of possible features for source characterization
widened, choosing appropriate features, feature weights and feature
combinations have become a central issue.

A natural way to tackle this problem is to consider sentence extraction as a
classification task. To this end, several machine learning approaches that
uses document-summary pairs have been proposed [6, 12], An hybrid method
mixing statistical and linguistics algorithms is presented in [1]. [10] and
[15] propose a good state-of-art of Automatic Text Summarization tasks and
algorithms.

###  0.2.1 Document Pre-processing

The first step to represent documents in a suitable space is the pre-
processing. As we use extractive summarization, documents have to be chunked
into cohesive textual segments that will be assembled to produce the summary.
Pre-processing is very important because the selection of segments is based on
words or bigrams of words. The choice was made to split documents into full
sentences, in this way obtaining textual segments that are likely to be
grammatically corrects. Afterwards, sentences pass through several basic
normalization steps in order to reduce computational complexity.

The process is composed by the following steps:

  1. 1.

Sentence splitting. Simple rule-based method is used for sentence splitting.
Documents are chunked at the period, exclamation and question mark.

  2. 2.

Sentence filtering. Words lowercased and cleared up from sloppy punctuation.
Words with less than 2 occurrences ($f<2$) are eliminated (Hapax legomenon
presents once in a document). Words that do not carry meaning such as
functional or very common words are removed. Small stop-lists (depending of
language) are used in this step.

  3. 3.

Word normalization. Remaining words are replaced by their canonical form using
lemmatization, stemming, ultra-stemming or none of them (raw text). Four
methods of normalization were applied after filtering:

     * •

Lemmatization by simples dictionaries of morphological families. These
dictionaries have 1.32M, 208K and 316K words-entries in Spanish, English and
French, respectively.

     * •

Porter’s Stemming, available at Snowball (web site
http://snowball.tartarus.org/texts/stemmersoverview.html) for English,
Spanish, French among other languages.

     * •

Ultra-stemming. This normalization seems be very efficient and it produces a
compact matrix representation [16]. Ultra-stemming consider only the $n$ first
letters of each word. For example, in the case of ultra-stemming (first
letter, Fix1), inflected verbs like “sing”, “song”, “sings”, “singing”… or
proper names “smith”, “snowboard”, “sex”,… are replaced by the letter “s”.

  4. 4.

Text Vectorization. Documents are vectorized in a matrix $S_{[P\times N]}$ of
$P$ sentences and $N$ columns. Each element $s_{i,j}$ represents the
occurrences of an object $j$ (a letter in the case of ultra-stemming, a word
in the case of lemmatization or a stem for stemming), $j=1,2,...,N$ in the
sentence $i$, $i=1,2,...,P$.

##  0.3 AnotheR TEXt summarizer (Artex)

Artex111In French, Artex est un Autre Résumeur TEXtuel. is a simple extractive
algorithm for Automatic Text Summarization. The main idea is the next one:
First, we represent the text in a suitable space model (VSM). Then, we
construct an average document vector that represents the average (the “global
topic”) of all sentences vectors. At the same time, we obtain the “lexical
weight” for each sentence, i.e. the number of words in the sentence. After
that, it is calculated the angle between the average document and each
sentence; narrow angles indicate that the sentences near of the “global topic”
should be important and therefore extracted. See on the figure 1 the VSM of
words: $P$ vector sentences and the average “global topic” are represented in
a $N$ dimensional space of words.

Figure 1: The “global topic” in a Vector Space Model of $N$ words.

Next, a score for each sentence is calculated using their proximity with the
“global topic” and their “lexical weight”. In the figure 2, the “lexical
weight” is represented in a VSM of $P$ sentences.

Finally, the summary is generated concatenating the sentences with the highest
scores following their order in the original document.

Figure 2: The “lexical weight” in a Vector Space Model of $P$ sentences.

###  0.3.1 Algorithm

Formally, Artex algorithm computes the score of each sentence by calculating
the inner product between a sentence vector, an average pseudo-sentence vector
(the “global topic”) and an average pseudo-word vector (the “lexical weight”).

Once a pre-processing (word normalization and filtering of stop words) is
completed, it is created a matrix $S_{[P\times N]}$, using the Vector Space
Model, that contains $N$ words (or letters) and $P$ sentences.

Let $s_{i}=(s_{1},s_{2},...,s_{N})$ be a vector of the sentence $i$,
$i=1,2,...,P$. We defined ${\vec{a}}$ the average pseudo-word vector, as the
average number of occurrences of $N$ words used in the sentence $i$:

(1) $a_{i}=\frac{1}{N}\sum_{j}s_{i,j}$

and ${\vec{b}}$ the average pseudo-sentence vector as the average number of
occurrences of each word $j$ used trough the $P$ sentences:

(2) $b_{j}=\frac{1}{P}\sum_{i}s_{i,j}$

The score or weight of each sentence $s_{i}$ is calculated as follows:

(3) $\textrm{score}(s_{i})=\left(\vec{s}\times\vec{b}\right)\
\times\vec{a}=\frac{1}{NP}\left(\sum_{j}s_{i,j}\times b_{j}\right)\times
a_{i}\,;\,i=1,2,...,P\,;j=1,1,...,N$

The score($\bullet$) computed by equation 3 must be normalized between the
interval [0,1]. The calculation of $\vec{s}\times\vec{b}$ indicates the
proximity between the sentence $\vec{s}$ and the average pseudo-sentence
$\vec{b}$. The product ($\vec{s}\times\vec{b})\times\vec{a}$ weigh this
proximity using the average pseudo-word $a_{i}$.

If a sentence $s_{i}$ is near of $\vec{b}$ and their corresponding element
$a_{i}$ has a high value, $s_{i}$ will have, therefore, a high score.
Moreover, a sentence $i$ far of main topic (i.e. $\vec{s}_{i}\times\vec{b}$ is
near 0) or a less informative sentence $i$ (i.e. $a_{i}$ are near 0) will have
a low score.

In computational terms, it is not really necessary to divide the scalar
product by the constant $\frac{1}{NP}$, because the angle
$\alpha=\arccos{\vec{b}}.{\vec{s}}/|{\vec{b}}||{\vec{s}}|$ between ${\vec{b}}$
and $\vec{s}$ is the same if we use
${\vec{b}}={\vec{b}}^{\prime}=\sum_{i}s_{i,j}$. The element $a_{i}$ is only a
scale factor that does not modify $\alpha$.

In fact, if the matrix $S_{[P\times N]}$ is approximated to a binary
matrix222This is a reasonable approximation in this context, because
$S_{[P\times N]}$ is a sparsed matrix with many term occurrences equal to one
or zero. $S^{\prime}_{[P\times N]}$, where each element
$s^{\prime}_{i,j}=\\{0,1\\}$ has a probability of $p=\frac{1}{2}$, we can
normalize vectors $\vec{a}$, $\vec{b}$ and matrix $S$, as follows:

(4) $\displaystyle|\vec{a}|$ $\displaystyle=$
$\displaystyle\sum_{i}^{P}\sqrt{{s^{\prime}_{i,j}}^{2}}=\sum_{i}^{P}\sqrt{(\\{0,1\\}^{P})^{2}}=N\sqrt{P}$
(5) $\displaystyle|\vec{b}|$ $\displaystyle=$
$\displaystyle\sum_{j}^{N}\sqrt{{s^{\prime}_{i,j}}^{2}}=\sum_{j}^{N}\sqrt{(\\{0,1\\}^{N})^{2}}=\sqrt{N}P$
(6) $\displaystyle|\vec{s}_{i}|$ $\displaystyle=$
$\displaystyle\sum_{j}^{N}\sqrt{{s^{\prime}_{i,j}}^{2}}=\sum_{j}^{N}\sqrt{\\{0,1\\}^{2}}=N$

Vectors then will be represented in hyper-spheres of $N$ or $P$ dimensions,
and the normalized score’ in this space would be:

(7) $\displaystyle\textrm{score'}(s_{i})$ $\displaystyle=$
$\displaystyle\left(\frac{\vec{s}}{|\vec{s}|}\times\frac{\vec{b}}{|\vec{b}|}\right)\
\times\frac{\vec{a}}{|\vec{a}|}=\frac{1}{N\sqrt{N}PN\sqrt{P}}\left(\sum_{j}s_{i,j}\times
b_{j}\right)\times a_{i}\,$ $\displaystyle=$
$\displaystyle\frac{1}{\sqrt{N^{5}P^{3}}}\left(\sum_{j}s_{i,j}\times
b_{j}\right)\times a_{i}\,;\,i=1,2,...,P\,;j=1,2,...,N$

However, the term $1/\sqrt{N^{5}P^{3}}$ is a constant value (i.e. a simple
scale factor), and then the $\textrm{score}(\bullet)$ calculated using the
equation 3) and the $\textrm{score'}(\bullet)$ using the equation 7, are both
equivalent.

##  0.4 Experiments

Artex algorithm described in the previous section has been implemented and
evaluated in corpora in several languages.

We have conducted our experimentation with the following languages,
summarization tasks, summarizers and data sets: 1) Generic multi-document-
summarization in English with the corpus DUC’04; 2) Generic single-document
summarization in Spanish with the corpus Medicina Clínica and 3) Generic
single document summarization in French with the corpus Pistes.

We have applied the summarization algorithms and finally, the results have
been evaluated using Fresa while processing times for each summarizer have
been measured and compared.

The following subsections present formally the details of the summarizers,
corpora and evaluations studied in different experiments.

###  0.4.1 Other Summarizers

To compare the performances, two other summarization systems were used in our
experiments: Cortex and Enertex. To be in the same conditions, these two
systems have used exactly the same textual representation based on Vector
Space Model, described in Section 0.2.1.

  * •

Cortex is a single-document summarization system using several metrics and an
optimal decision algorithm [4, 14, 15, 18].

  * •

Enertex is a summarization system based in Textual Energy concept [5]: text is
represented as a spin system where spins $\uparrow$ represents words that
their occurrences are $f>1$ (spins $\downarrow$ if the word is not present).

###  0.4.2 Summarization Corpora Description

To study the impact of our summarizer, we used corpora in three languages:
English, Spanish and French. The corpora are heterogeneous, and different
tasks are representatives of Automatic Text Summarization: generic multi-
document summary and mono-document guided by a subject.

  * •

Corpus in English. Piloted by NIST in Document Understanding
Conference333http://duc.nist.gov (DUC) the Task 2 of DUC’04444http://www-
nlpir.nist.gov/projects/duc/guidelines/2004.html, aims to produce a short
summary of a cluster of related documents. We studied generic multi-document-
summarization in English using data from DUC’04. This corpus with 300K words
(17 780 types) is compound of 50 clusters, 10 documents each.

  * •

Corpus in Spanish. Generic single-document summarization using a corpus from
the scientific journal Medicina
Clínica555http://www.elsevier.es/revistas/ctl_servlet?_f=7032&revistaid=2,
which is composed of 50 medical articles in Spanish, each one with its
corresponding author abstract. This corpus contains 125K words (9 657 types).

  * •

Corpus in French. We have studied generic single-document summarization using
the Canadian French Sociological Articles corpus, generated from the journal
Perspectives interdisciplinaires sur le travail et la santé
(Pistes)666http://www.pistes.uqam.ca/. It contains 50 sociological articles in
French, each one with its corresponding author abstract. This corpus contains
near 400K words (18 887 types).

###  0.4.3 Summaries Content Evaluation

DUC conferences have introduced the ROUGE content evaluation [7], wich
measures the overlap of $n$-grams between a candidate summary and reference
summaries written by humans. However, to write the human summaries necessaries
for ROUGE is a very expensive task.

Recently metrics without references have been defined and experimented at DUC
and Text Analysis Conferences (TAC)777www.nist.gov/tac workshops.

Fresa content evaluation [13, 17] is similar to ROUGE evaluation, but human
reference summaries are not necessary. Fresa calculates the divergence of
probabilities between the candidate summary and the document source. Among
these metrics, Kullback-Leibler (KL) and Jensen-Shannon (JS) divergences have
been widely used by [8, 17] to evaluate the informativeness of summaries.

In this article, we use Fresa, based in KL divergence with Dirichlet
smoothing, like in the 2010 and 2011 INEX edition [11], to evaluate the
informative content of summaries by comparing their $n$-gram distributions
with those from source documents.

Fresa only considered absolute log-diff between the terms occurrences of the
source and the summary. Let $T$ be the set of terms in the source. For every
$t\in T$, we denote by $C_{t}^{T}$ its occurrences in the source and
$C_{t}^{S}$ its occurrences in the summary.

The Fresa package computed the divergence between the document source and the
summaries as follows:

(8) ${\mathcal{D}}(T||S)=\sum_{t\in
T}\left|\log\left(\frac{C_{t}^{T}}{|T|}+1\right)-\log\left(\frac{C_{t}^{S}}{|S|}+1\right)\right|$

To evaluate the information content (the “quality”) of the generated
summaries, after removing stop-words, several automatic measures were
computed: Fresa1 (Unigrams of single stems), Fresa2 (Bigrams of pairs of
consecutive stems), FresaSU4 (Bigrams with 2-gaps also made of pairs of
consecutive stems) and finally, $\langle\textsc{Fresa}\rangle$, i.e. the
average of all Fresa values.

The Fresa values (scores) are normalized between 0 and 1. High Fresa values
mean less divergence regarding the source document summary, reflecting a
greater amount of information content. All summaries produced by the systems
were evaluated automatically using Fresa package.

##  0.5 Results

In this section we present the results for each corpus with different
summarizers and the several normalization strategies used. Based on these
results, firstly, we have verified that ultra-stemming improves the
performance of summarizers. Secondly, we show that Artex is a system that has
a similar performances –in terms of information content and processing times–
to other state-of-art summarizers.

###  0.5.1 Content evaluation

  * •

English corpus. Figure 3 shows the performance of the three summarizers using
Fix1, stemming and lemmatization. Results show that ultra-stemming improves
the score of the three automatic summarizer systems. Artex and Cortex expose a
similar performances in information content.

Figure 3: Histogram plot of content evaluation for corpus DUC’04 Task 2, with
$\langle$Fresa$\rangle$ measures, for each summarizer and each normalization.

  * •

Spanish corpus. Spanish is a language with a greater variability than English.
Results in figure 4 shown that Artex summarizer outperforms Cortex and Enertex
if stemming or lemmatization are used as normalization.

Figure 4: Histogram plot of content evaluation for Spanish corpus Medicina
Clínica with $\langle$Fresa$\rangle$ scores for each summarizer.

  * •

French corpus. French is a language with a large variability too. Figure 5
shows the score $\langle$Fresa$\rangle$ on the French corpus Pistes. Results
show a similar behavior: Ultra-stemming improves the score of the three
automatic summarization systems used. In particular, the efficacy of Artex is
less sensible to word normalization than others summarizers.

Figure 5: Histogram plot of content evaluation for French corpus Pistes with
$\langle$Fresa$\rangle$ scores for each summarizer.

###  0.5.2 Processing Times Evaluation

Table 1 shows processing times for each corpus, following the normalization
method for Cortex, Artex and Enertex summarizers888All times are measured in a
7.8 GB of RAM computer, Core i7-2640M CPU @ 2.80GHz $\times$ 4 processor,
running under 32 bits GNU/Linux (Ubuntu Version 12.04).. Processing times of
ultra-stemming Fix1 are shorter compared to all others methods. By example,
Cortex is a very fast summarizer with $O(\log\rho^{2})$ (where $\rho=P\times
N$), and processing times for stemming and Fix1 are close. In other hand,
Enertex summarizer has a complexity of $O(\rho^{2})$, then it needs more time
to process the same corpus. Performances of Artex algorithm remain close to
Cortex.

| Summarizer Average Time
---|---
| (all corpora)
Normalization | Cortex | Artex | Enertex
Lemmatization | 1.60’ | 2.50’ | 10.42’
Stemming | 0.54’ | 1.29’ | 9.47’
fix1 | 0.32’ | 0.40’ | 4.25’
Table 1: Statistics of processing times (in minutes) of three summarizers over
three corpora.

##  0.6 Conclusions

In this article we have introduced and tested a simple method for Automatic
Text Summarization. Artex is a fast and very simple algorithm based in VSM
model and the extractive paradigm. The method uses a matrix representation to
calculate a normalized score for each sentence, using the inner product of
pseudo-(sentences|words) vectors. The algorithm retains the salient
information of each sentence of document. An important aspect of our approach
is that it does not requires linguistic knowledge or resources which makes it
a simple and efficient summarizer method to tackle the issue of Automatic Text
Summarization.

Summaries generated by Artex system are pertinents. The results obtained on
corpora in English, Spanish and French show that Artex can achieve good
results for content quality. Tests with other corpora (DUC and TAC evaluation
campaigns, INEX, etc.) in mono-and multi-document guided by a subject, using
content evaluation with (ROUGE evaluations) or without reference summaries
still in progress.

## References

  * [1] Iria da Cunha, Silvia Fernández, Patricia Velázquez-Morales, Jorge Vivaldi, Eric SanJuan, and Juan Manuel Torres-Moreno.  A new hybrid summarizer based on vector space model, statistical physics and linguistics.  In Proceedings of the 6th Mexican International Conference on Advances in Artificial Intelligence (MICAI’07), pages 872–882, Aguascalientes, Mexico, 2007. Springer-Verlag.
  * [2] Harold Daumé III.  Practical structured learning techniques for natural language processing.  PhD thesis, Los Angeles, CA, 2006.
  * [3] H. P. Edmundson.  New Methods in Automatic Extraction.  Journal of the Association for Computing Machinery, 16(2):264–285, 1969.
  * [4] B. Favre, F. Béchet, P. Bellot, F. Boudin, M. El-Bèze, L. Gillard, G. Lapalme, and J-M. Torres-Moreno.  The LIA-Thales summarization system at DUC-2006.  In Proceedings of the Document Understanding Conference (DUC’06), Brooklyn, New York, United States, 2006.  http://duc.nist.gov.
  * [5] Silvia Fernández, Eric SanJuan, and Juan-Manuel Torres-Moreno.  Textual Energy of Associative Memories: performants applications of Enertex algorithm in text summarization and topic segmentation.  In Proceedings of the Mexican International Conference on Artificial Intelligence (MICAI’07), pages 861–871, Aguascalientes, Mexico, 2007\. Springer-Verlag.
  * [6] J. Kupiec, J. Pedersen, and F. Chen.  A trainable document summarizer.  In Proceedings of the 18th Conference ACM Special Interest Group on Information Retrieval (SIGIR’95), pages 68–73, Seattle, WA, United States, 1995. ACM Press, New York.
  * [7] Chin-Yew Lin.  ROUGE: A Package for Automatic Evaluation of Summaries.  In Marie-Francine Moens and Stan Szpakowicz, editors, Proceedings of the Workshop Text Summarization Branches Out (ACL’04), pages 74–81, Barcelone, Spain, july 2004. ACL.
  * [8] Annie Louis and Ani Nenkova.  Automatic Summary Evaluation without Human Models.  In First Text Analysis Conference (TAC’08), Gaithersburg, MD, United States, 17-19 November 2008.
  * [9] H.P. Luhn.  The Automatic Creation of Literature Abstracts.  IBM Journal of Research and Development, 2(2):159–165, 1958.
  * [10] I. Mani and M. Mayburi.  Advances in Automatic Text Summarization.  MIT Press, Cambridge, 1999.
  * [11] Eric SanJuan, Patrice Bellot, Véronique Moriceau, and Xavier Tannier.  Overview of the INEX 2010 Question Answering Track (QA@INEX).  In Shlomo Geva, Jaap Kamps, Ralf Schenkel, and Andrew Trotman, editors, Comparative Evaluation of Focused Retrieval, volume 6932 of Lecture Notes in Computer Science, pages 269–281. Springer Berlin / Heidelberg, 2011.
  * [12] Simone Teufel and Marc Moens.  Sentence extraction as a classification task.  In I. Mani and M. Maybury, editors, Proceedings of the ACL/EACL’97 Workshop on Intelligent Scalable Text Summarization, Madrid, Spain, 11 July 1997.
  * [13] J.-M. Torres-Moreno, Horacio Saggion, I. da Cunha, P. Velazquez-Morales, and E. SanJuan.  Evaluation automatique de résumés avec et sans réferences.  In Proceedings de la conference Traitement Automatique des Langagues Naturelles (TALN’10), Montréal, QC, Canada, 19-23 July 2010. ATALA.
  * [14] J.-M. Torres-Moreno, P.-L. St-Onge, M. Gagnon, M. El-Bèze, and P. Bellot.  Automatic Summarization System coupled with a Question-Answering System (QAAS).  CoRR, abs/0905.2990, 2009.
  * [15] Juan-Manuel Torres-Moreno.  Résumé automatique de documents: une approche statistique.  Hermès-Lavoisier, Paris, 2011.
  * [16] Juan-Manuel Torres-Moreno.  Beyond Stemming and Lemmatization: Ultra-stemming to Improve Automatic Text Summarization.  CoRR, arXiv:1209.3126 [cs.IR], 2012.
  * [17] Juan-Manuel Torres-Moreno, Horacio Saggion, Iria da Cunha, and Eric SanJuan.  Summary Evaluation With and Without References.  Polibits: Research journal on Computer science and computer engineering with applications, 42:13–19, 2010.
  * [18] Juan-Manuel Torres-Moreno, Patricia Velázquez-Morales, and Jean-Guy Meunier.  Cortex : un algorithme pour la condensation automatique des textes.  In Proceedings of the Conference de l’Association pour la Recherche Cognitive, volume 2, pages 365–366, Lyon, France, 2001.


# Layer Folding: Neural Network Depth Reduction using Activation Linearization

Amir Ben Dror
Samsung Israel Research Center
<EMAIL_ADDRESS>
&Niv Zehngut
Samsung Israel Research Center
<EMAIL_ADDRESS>
&Avraham Raviv
Samsung Israel Research Center
<EMAIL_ADDRESS>
&Evgeny Artyomov
Samsung Israel Research Center
<EMAIL_ADDRESS>
&Ran Vitek
Samsung Israel Research Center
<EMAIL_ADDRESS>
&Roy J Jevnisek
Samsung Israel Research Center
<EMAIL_ADDRESS>

###### Abstract

Despite the increasing prevalence of deep neural networks, their applicability
in resource-constrained devices is limited due to their computational load.
While modern devices exhibit a high level of parallelism, real-time latency is
still highly dependent on networks’ depth. Although recent works show that
below a certain depth, the width of shallower networks must grow
exponentially, we presume that neural networks typically exceed this minimal
depth to accelerate convergence and incrementally increase accuracy. This
motivates us to transform pre-trained deep networks that already exploit such
advantages into shallower forms. We propose a method that learns whether non-
linear activations can be removed, allowing to fold consecutive linear layers
into one. We apply our method to networks pre-trained on CIFAR-10 and
CIFAR-100 and find that they can all be transformed into shallower forms that
share a similar depth. Finally, we use our method to provide more efficient
alternatives to MobileNetV2 and EfficientNet-Lite architectures on the
ImageNet classification task.

##  1 Introduction

Multiple works have studied the relation between expressiveness and neural
networks’ depth. Early works [1, 2, 3] showed that some deep neural networks
cannot be represented by shallower networks unless those networks are
significantly wider. More specifically, Telgarsky [4] showed that there are
networks of depth $k^{2}$, for which any approximating network of depth $k$
requires width of $\mathcal{O}(\mathrm{e}^{k})$. Such depth separation [5],
highlighting the exponential parameter growth of shallow networks compared to
deeper ones representing the same function, has been further studied for other
types of functions [6, 7, 8, 9, 10, 11]. While these findings suggest that a
certain depth is required to preserve performance on a given task, many
architectures are typically deeper than that. The role of the added layers can
be viewed by the unrolled iterative estimation [12] — a group of successive
layers iteratively refine their estimates of the same features instead of
computing entirely new representations. Many works have even enforced
iterative estimation by utilizing recurrent architectures instead of feed
forward architectures for non-sequential input data [13, 14]. In addition to
potentially higher accuracy by iterative refined representations, and despite
optimization barriers associated with increased depth [15, 16], deeper
networks have also been shown favorable to training in some cases, acting as
preconditioners that accelerate convergence [17]. Hence, some layers in deep
neural networks can be regarded as crucial for depth separation while others
for refining representations and facilitating optimization, implying varying
contribution to networks’ expressiveness. Such observation plays an important
role when efficiency is additionally considered as an optimization objective.
Particularly, while reducing a network’s depth below a certain level may
markedly impact the network’s accuracy, reduction up to that level may exhibit
lower and even no impact and yet considerably improve the network’s
efficiency. We treat this level as the Effective Degree of Non-Linearity
(EDNL), as the depth of feed forward networks is determined by the number of
their non-linear layers. We argue that some networks wield more layers than
their EDNL, suggesting that their expressive power can be maintained with
wider-but-shallower networks. This has been supported by recent works [18, 19]
showing that deeper networks have a simplicity bias and encourage low-rank
solutions. Shallower networks are particularly advantageous for hardware
accelerators and GPUs that leverage intra-layer parallelism and suffer from
inter-layer computational overhead.

We propose to learn which activations can be removed without incurring a
significant accuracy degradation. This allows us to merge adjacent linear
layers, and in turn, transform deep networks into shallow ones. Over-
parameterization in feed forward networks does not solely relate to their
depth but may also evident in layers’ width. Pruning methods [20, 21, 22] are
commonly used to optimize networks’ width, with the ability to prune an entire
layer. Similarly to pruning methods, we focus on optimizing a pre-trained
network. This stems from an optimization perspective; first, the fine-tuning
process involved in such optimization requires far less computational
resources than training the refined architecture from scratch. Second,
gradient-based optimization of the shallower network may better proceed from
the original deeper network. This allows the network to leverage the rich
representations and local minimum obtained by the deeper network form and
distill it to its shallow form. Recent pruning methods [23, 24] attenuate
neurons during a fine-tuning phase, gradually reducing the network’s size
while allowing it to compensate. However, when applied to layer pruning, these
methods force the network to gradually adopt new intermediate representations.
The deep and shallow forms of a network may reside in local minima, for which
traversing from one another may be challenging by gradient descent. In
contrast, our optimization method maintains the intermediate representations
of the original deep network during the fine-tuning phase, followed by a
functionality-preserving transformation. Recently, Ding et al. [25] suggested
a similar decoupling between training-time architecture and inference-time
architecture via structural re-parametrization to leverage parallel
connections during training.

In line with other optimization methods, we focus on Convolutional Neural
Networks (CNNs) for their prevalence in compute-intensive vision applications.
We learn to remove non-linear activations between consecutive convolution
layers, allowing their functionality-preserving merge. For layers with spatial
kernels of size $k\times k$, this result in a larger $(2k-1)\times(2k-1)$
kernel. We show how such a transformation, in spite of the added FLOPs, may
reduce latency on different hardware devices. Interestingly, this comes in
reverse of the common preference of multiple smaller kernels (e.g., $3\times
3$) than fewer larger ones (e.g., $5\times 5$) which originated in [26]. For
some cases, our approach can be regarded as an inverse of network
factorization methods that utilize tensor decompositions such as CP-
decomposition and Tucker decomposition [27, 28, 29, 30]. Indeed, merging
consecutive layers of depth-wise separable convolutions as in MobileNetV1 [31]
may lead to far more computations and negatively impact latency. In contrast,
for other architectures that incorporate bottlenecks, merging layers can in
fact reduce FLOPs, and latency furthermore. For example, merging layers of a
MobileNetV2 inverted bottleneck [32] into a single convolution layer may cut
FLOPs in half. We experiment on this architecture to show such potential gain.
The fact that many recent works [33, 34, 35, 36, 37] have relied on neural
architectures composed of inverted bottlenecks to achieve prominent
performance over the ImageNet classification task [38] highlights the
attractiveness of our method.

While most pruning methods may prune entire layers indirectly, we attend such
optimization explicitly. Specifically, pruning methods penalize weights
whereas our method penalizes activations. This implies that our method can be
used jointly with other pruning methods, improving efficiency even further. In
addition, our method result in merged weight kernels which are necessarily
low-rank. For instance, a $5\times 5$ weight kernel resulting from merging two
$3\times 3$ kernels is bounded to a dimension of $18$ rather than $25$.
Further optimization may leverage this redundancy to reduce computations even
further. For example, the aforementioned $5\times 5$ kernel may be more easily
distilled into a smaller $3\times 3$ kernel compared to an arbitrary $5\times
5$ kernel.

Our contributions are as follows:

  1. 1.

We propose _Layer Folding_ , a novel method to reduce the depth of a neural
network and fold consecutive linear layers by removing the non-linear
activations that separate them. We show how our method facilitates
optimization by maintaining the intermediate representations of the original
depth.

  2. 2.

We introduce EDNL, which provides information on the required depth for
networks to maintain their functionality. We reveal the EDNL of ReLU-activated
CNNs and show that it is closely related to the complexity of their task
rather than their original depth.

  3. 3.

We apply our method on efficient mobile networks over the ImageNet
classification task [38] and improve their latency even further without a
significant impact on their accuracy.

##  2 Related Work

### Pruning.

Similar to pruning, we modify the existing architecture of a pre-trained
network in order to improve its efficiency while maintaining its accuracy.
Pruning methods remove selected neurons – nodes, filters or layers – according
to pruning criteria. Works such as [20, 21, 39] rely on Taylor expansions
provided by the Hessian matrix of the loss function. Other works [22, 40, 41]
prune neurons according to their magnitude. Different importance assignment
methods have also been proposed, such as score propagation [42] or statistics
drawn from adjacent layers [43]. Resembling our focus on depth reduction,
dedicated layer pruning methods have also been proposed. Jordao et al. [44]
considered the discriminative power of feature maps as layer scoring which is
then used as scoring mechanism. Chen and Zhao [45] proposed to estimate
whether layers can be replaced with linear layers, sharing a similar
motivation to ours. In contrast, they optimize layers independently while we
train a network as a whole in an end-to-end manner, allowing layers to
compensate for those which are removed. Neill et al. [46] proposed to remove
layers based on layer similarity. They also empirically showed that there is a
bound on the amount of compression that can be achieved before an exponential
degradation in performance. We draw a different conclusion and show that such
a bound originates from a network’s depth rather than its size (i.e., number
of parameters). Our work is in line with pruning methods that perform fine-
tuning with additional loss which encourage more efficient networks, such as
[47]. However, while such methods require the network to learn new
intermediate representations, we maintain the rich representation space of the
original network. Moreover, according to [48], pruning entails masking or
removing weight elements of a given model, producing a smaller one. Our work
does not fall under this definition, as we introduce new weight elements, and
may even increase the model’s size.

### Neural Architecture Search (NAS).

As high-capacity and high-performing neural networks became feasible to train,
designing efficient architectures has gained high interest. Early works have
suggested various design principles, such as adopting multiple smaller spatial
kernels instead of larger ones [26], leveraging residual connections [49],
decomposing weight matrices [31] and utilizing bottlenecks [32]. Current top-
performing architectures are found using NAS. Since the search space spanned
by possible architectures is intractable, existing methods use reinforcement
learning [50], genetic algorithms [51], differentiable search [52] and other
methods [53, 54, 55] to traverse it. Similarly to our method, differentiable
methods learn architectural paths that allows the removal of entire layers,
and even compensate it with added width to the preceding ones. For example,
considering a reference architecture, all the following methods [33, 34, 35,
36, 37] may learn to remove a convolution layer while enlarging the kernel of
the preceding layer. However, NAS methods in general and these methods in
particular require costly computational resources for both training a super-
network from scratch and covering multiple search space dimensions. Our method
leverages a pre-trained network and focuses on a single search dimension:
depth. Both allow an expedited training time and facilitated convergence.

### Activation removal.

Our method uniquely combines activation removal with consecutive linear layer
folding. Nevertheless, activation removal alone has been used for other
purposes. He et al. [56] proposed PReLU, generalizing the ReLU activation
function. Ma et al. [57] extended this idea to arbitrary activation functions.
These methods allow learning whether activations should be shifted towards
identity. However, their method learns the extent of such shift with accuracy
optimization in mind. In contrast, we learn a binary decision – keeping the
original activation or replacing it with an identity – with efficiency
optimization in mind. Since our objective comprises depth minimization on top
of the original task loss, our method converges to different solutions.
Activation removal has gained further attention with the resurgence of Private
Inference (PI). PI performs inference on encrypted data, where latency is
hindered mostly by non-linear activations such as ReLU. Ghodsi et al. [58]
performed NAS to optimize the placement of skip connections and considered
pruning ReLU activations that follow them. Jha et al. [59] proposed to measure
ReLU criticality by evaluating a model’s performance with ReLUs of entire
stages or alternating layers being removed. While we leave it to future work,
our work can also be used to accelerate PI.

##  3 Method

We present a method which allows to reduce the number of non-linear
activations in a neural network. This effectively enables to merge adjacent
linear layers into a single one. We call this process _Layer Folding_. Given a
non-linear activation function $\sigma$, we define the parametric activation
$\sigma_{\alpha}()$ to be the linear combination of $\sigma$ and the identity
function:

$\sigma_{\alpha}(x)=\alpha x+(1-\alpha)\sigma(x),\quad 0\leq\alpha\leq 1$ (1)

where $\alpha$ is a trainable parameter which provides an interpolation
between $\sigma$ and the identity function. When $\sigma=ReLU$,
$\sigma_{\alpha}()$ is the common PReLU [56] activation. Given a trained
neural network $\mathcal{F}$, we construct a network $\mathcal{F}_{\alpha}$ by
transforming its activations into their corresponding parametric activations
initialized with $\alpha=0$. This ensures that $\mathcal{F}_{\alpha}$
maintains the same functionality of $\mathcal{F}$.

We perform a fine-tuning stage in which we optimize $\mathcal{F}_{\alpha}$
with respect to both the original task loss $\mathcal{L}_{t}$ and an auxiliary
loss $\mathcal{L}_{c}$ that penalizes smaller $\alpha$ values, encouraging
them to become $1$. We consider a general form of $\mathcal{L}_{c}$:

$\mathcal{L}_{c}=\sum_{l\in L}c_{l}\,h(\alpha_{l})$ (2)

where $\alpha_{l}$ corresponds to the $l$th activation and $h(\alpha)$ is a
monotonically decreasing function for $0\leq\alpha\leq 1$.
$\left\\{c_{l}\right\\}_{l\in L}$ weigh the contribution of each layer to
$\mathcal{L}_{c}$. These are used to depict a varying potential value for
folding different layers and can be set, for example, according to a measured
latency on a target device. When we simply want to encourage a shallow network
we set $c_{l}=1$, $l=1:L$.

While many forms of $h$ can be applied, we provide a simple suggestion for $h$
such that $\mathcal{L}_{c}$ becomes:

$\mathcal{L}_{c}=\sum_{l\in L}c_{l}\,(1-\alpha_{l}^{p})$ (3)

We select this form for the following reasons: after training, a layer can be
folded with its subsequent one only if its corresponding $\alpha$ is
sufficiently close to $1$. In particular, we are sensitive to small changes in
$\alpha_{l}$ near $1$, since the farther $\sigma_{\alpha}$ deviates from
identity the larger the error incurred from Layer Folding. Yet, we are
indifferent to small changes in $\alpha$ near $0$ since the matching layer
cannot be folded anyway. We would like $\mathcal{L}_{c}$ to represent these
ideas. $p>1$ is a hyperparameter controlling the flatness of the loss surface
around $\alpha_{l}=0$ and the strength in which larger $\alpha_{l}$ values are
pushed to $1$. Our final loss function is:

$\mathcal{L}=\mathcal{L}_{t}+\lambda_{c}\,\mathcal{L}_{c}$ (4)

where $\lambda_{c}$ is a hyperparameter that balances between the task loss
and the amount of layers that will be folded.

We define the folding of two adjacent linear layers that reside between the
feature maps $\left\\{X,Y,Z\right\\}$, $g_{1}:X\rightarrow Y$ and
$g_{2}:Y\rightarrow Z$, as their composite function, i.e., $g_{1}\circ
g_{2}:X\rightarrow Z$. We provide two examples for fully connected layers and
convolution layers while omitting the linear bias addition and batch
normalization operations for brevity. For
$g_{1}(\mathbf{x})=\mathbf{W}_{1}\mathbf{x}$,
$g_{2}(\mathbf{y})=\mathbf{W}_{2}\mathbf{y}$ fully connected layers where
$\mathbf{x}\in\mathbb{R}^{d_{X}}$, $\mathbf{y}\in\mathbb{R}^{d_{Y}}$,
$\mathbf{z}\in\mathbb{R}^{d_{Z}}$, $\mathbf{W}_{1}\in\mathbb{R}^{d_{Y}\times
d_{X}}$, $\mathbf{W}_{2}\in\mathbb{R}^{d_{Z}\times d_{Y}}$, their folding is
given by:

$g_{fold}(\mathbf{x})=\mathbf{W}_{2}\mathbf{W}_{1}\mathbf{x}$ (5)

For
$g_{1}(\mathbf{x})=\left\\{\sum_{i=1}^{c_{X}}\mathbf{W}_{1}^{i,j}\ast\mathbf{x}_{i}\right\\}_{j=1}^{c_{Y}}$,
$g_{2}(\mathbf{y})=\left\\{\sum_{j=1}^{c_{Y}}\mathbf{W}_{2}^{j,m}\ast\mathbf{y}_{j}\right\\}_{m=1}^{c_{Z}}$
convolution layers where $\mathbf{x}\in\mathbb{R}^{h\times w\times c_{X}}$,
$\mathbf{y}\in\mathbb{R}^{h\times w\times c_{Y}}$,
$\mathbf{z}\in\mathbb{R}^{h\times w\times c_{Z}}$,
$\mathbf{W}_{1}\in\mathbb{R}^{k\times k\times c_{Y}\times c_{X}}$,
$\mathbf{W}_{2}\in\mathbb{R}^{k\times k\times c_{Z}\times c_{Y}}$, their
folding is given by:

$g_{fold}(x)=\left\\{\sum_{i=1}^{c_{X}}\left(\sum_{j=1}^{c_{Y}}W_{1}^{i,j}\ast
W_{2}^{j,m}\right)\ast x_{i}\right\\}_{m=1}^{c_{Z}}$ (6)

Our method comprises 2 phases: _pre-folding_ and _post-folding_. In the pre-
folding phase we fine-tune $\mathcal{F}_{\alpha}$ with the loss defined in
(4). When training converges we remove activations whose $\alpha$s exceed a
threshold $\tau$ and fold the corresponding adjacent layers, resulting in a
shallower network $\mathcal{F}_{fold}$. In the post-folding phase, we fine-
tune $\mathcal{F}_{fold}$ once more for two main reasons. First, the
underlying function of $\mathcal{F}_{fold}$ may yet deviate from
$\mathcal{F}_{\alpha}$ due to various layers’ attributes such as padding,
resulting in a small accuracy decrease. Post-folding fine-tuning allows the
network to recover from it. Second, in some cases, $\mathcal{F}_{fold}$ result
in a larger number of weights. Further training of $\mathcal{F}_{fold}$ may
leverage the added capacity and increase accuracy. Our method is illustrated
in Figure 1.

Figure 1: Illustration of our method. We replace activations of a given
network $\mathcal{F}$ (a) with parametric activations according to Equation
(1), resulting in $\mathcal{F}_{\alpha}$ (b). We fine-tune
$\mathcal{F}_{\alpha}$ with the loss provided in Equation (4). When training
converges, we remove activations whose $\alpha\approx 1$ (c). We fold
consecutive linear operations (e.g., using Equation (5) and Equation (6)),
resulting in a shallower network $\mathcal{F}_{fold}$ (d). We then fine-tune
$\mathcal{F}_{fold}$.

##  4 Effective Degree of Non-Linearity Evaluation

In this section we use Layer Folding to evaluate the EDNL of several neural
networks over image classification tasks. The success of neural networks is
commonly attributed to their large capacity and non-linear nature. The
particular contribution of networks’ size in the training phase motivates the
pursuit for more efficient networks to be used for inference, and is often
carried out by pruning and NAS methods. The contribution of the non-linearity
of neural networks, however, is seldom assessed. Many works have shown that
for various cases, neural networks must employ a certain depth unless
exponentially increasing their width [4, 6, 7, 8, 9, 10, 11]. Since depth,
similarly to model’s size, may be stretched to aid the training phase [17],
networks’ depth can also be reduced for efficient inference. Unlike networks’
size, their depth accounts for a complexity measure that we regard as the
EDNL. Moreover, we expect the EDNL to be associated with a function that
yields a certain accuracy over a certain task, and as such, that similarly
performing networks will share an EDNL.

In order to show that a network possesses an EDNL, we show that its accuracy
is roughly maintained down to a certain depth and drops below it.
Particularly, we ensure that this holds true even when the network’s size
increases as its depth grows smaller. We further show that such depth knee-
point is shared for different networks over a particular task.

We perform our experiments on MNIST [60], CIFAR-10 and CIFAR-100 [61] image
classification tasks. For MNIST, we train fully-connected networks with depth
$L\in[2:10]$, ReLU activation and width $d=256$ for all layers. We denote
these networks by _FC- $L$_. For CIFAR-10 and CIFAR-100, we consider the
commonly used ResNet and VGG architectures [49, 26]. We use pre-trained ResNet
models with depth $L\in\left\\{20,32,44,56\right\\}$ and VGG models with depth
$L\in\left\\{16,19\right\\}$ [62]. For each of these networks, we apply Layer
Folding with $c_{l}=1$, $l=1:L$, $p=2$, $\tau=0.9$ while varying $\lambda_{c}$
to obtain shallower networks of varying depth.

Figure 2: Layer Folding applied on fully-connected networks with depth
$L\in[2:10]$. For each network, we gradually remove non-linear activations.

We first apply only pre-folding in order to evaluate the EDNL without
increasing the networks’ size. We count the number of remaining non-linear
layers, denoted by $\hat{L}$. As shown in Figure 2, an EDNL is clearly visible
for FC-$L$ networks, as they all retain their accuracy down to $\hat{L}=2$
which is then degraded for $\hat{L}=1$. Figure 3 shows our results for
CIFAR-10 and CIFAR-100. For CIFAR-10, all ResNet models exhibit a small
accuracy drop down to a depth of $\hat{L}=8$ and a large one below it. A
similar phenomenon is observed for CIFAR-100. As expected, the classification
task with the added classes exhibits a slightly larger EDNL. The gap between
these EDNLs may indicate the general complexity increase for classifying $100$
classes instead of $10$ or point to a more complex feature extraction required
for some of the added classes.

Figure 3: Layer Folding applied on ResNet and VGG architectures on CIFAR-10
(left) and CIFAR-100 (right). For each network, we gradually remove non-linear
layers. For CIFAR-10 (CIFAR-100), an accuracy drop is noticeable for number of
non-linear layers below 8 (10).

Figure 3 also shows our results for VGG networks. We expect that the
difference between VGG and ResNet architectures (e.g., different
dimensionality) will result in different EDNLs. Yet, the proximity between the
observed EDNLs of these models validates that EDNL is indeed attributed to the
task itself rather than the exact model. Interestingly, we could reduce the
depth of such architecture despite the lack of residual connections. We
accredit such success to the fact that our method can preserve the rich
intermediate representations of the deeper original network during fine-
tuning.

In Figure 4 we compare networks obtained by pre-folding and post-folding. The
results show that the EDNL of a folded network conforms to the one of the
original network. We note that for folded networks, shallower architectures
utilize more parameters, as the folding of two consecutive convolution layers
with weights $\mathbf{W}\in\mathbb{R}^{3\times 3\times c\times c}$ result in a
layer whose weights are $\mathbf{W}\in\mathbb{R}^{5\times 5\times c\times c}$,
i.e., its size grows by $40\%$. Hence, and in contrast to MNIST, this
experiment emphasizes the importance of depth even when it is disproportional
to model’s size, as networks were outperformed by deeper counterparts with
fewer parameters. In addition, the slight accuracy increase of the folded
networks quantifies the benefit of the post-folding phase. We believe that
this makes them favorable to further efficiency optimization such as kernel
size reduction. For example, we speculate that a $9\times 9$ kernel resulted
from folding four $3\times 3$ kernels can be successfully distilled into a
$7\times 7$ kernel that nonetheless holds more capacity ($49$) than the four
kernels altogether ($36$). We leave such optimization directions to future
work.

Figure 4: Relative contribution of pre-folding and post-folding. We perform
pre-folding fine-tuning on ResNet-20 and VGG-16 and compare them to their
shallower architectures resulting from post-folding. We also compare them to
training the folded architectures from scratch, denoted by ResNet-L and VGG-L.
Results are shown for CIFAR-10 (left) and CIFAR-100 (right).

We further compare our method to training randomly initialized networks with
different depths as commonly practiced in NAS methods. Figure 4 shows the
accuracy degradation of the folded architectures when trained from scratch
rather than derived from their deeper source by our method.

While Layer Folding can be applied with various loss functions in accordance
with Equation (2), we show that our chosen loss function in Equation (3) can
effectively remove non-linear layers. Figure 5 validates that $\alpha$ values
are indeed kept around zero or pushed to one, avoiding values in between.
Additionally, we show that our method is not biased towards folding layers
either in the beginning or the end of a given architecture. Table 1 shows the
indices of the removed layers for folded ResNet and VGG networks such that
their resulting depth corresponds to their EDNL.

Figure 5: Progression of $\alpha_{l}$ values corresponding to non-linear
layers in ResNet-20 (left) and ResNet-56 (right) throughout the pre-folding
phase with $\lambda_{c}=0.25$. As expected, all $\alpha$ values are either
kept around zero or pushed to one. Table 1: Folded networks.

Dataset | Model | Removed (white) and remaining (gray) activations | Depth | Acc. (%)
---|---|---|---|---
CIFAR-10 | ResNet-20 |  | 9 | 89.82
ResNet-32 |  | 9 | 90.02
ResNet-44 |  | 9 | 89.88
ResNet-56 |  | 10 | 90.29
VGG16 |  | 9 | 93.89
VGG19 |  | 8 | 93.23
CIFAR-100 | ResNet-20 |  | 11 | 67.88
ResNet-32 |  | 11 | 68.20
ResNet-44 |  | 11 | 67.96
ResNet-56 |  | 10 | 67.04
VGG16 |  | 12 | 72.82
VGG19 |  | 12 | 73.18

##  5 Depth Optimization for Efficient Networks

In this section we utilize our method to optimize networks with respect to
both accuracy and efficiency. We perform our experiments on the ImageNet image
classification task [38] and measure the latency of all models on NVIDIA Titan
X Pascal GPU.

We consider the commonly used MobileNetV2 [32] and EfficientNet-lite [63]. We
focus on these models for their attractiveness for hardware and edge devices,
mostly credited to their competitive latency and the exclusion of squeeze-and-
excite layers [64] employed by other state-of-the-art networks.

Both MobileNetV2 and EfficientNet-lite consist of multiple mobile inverted
bottleneck blocks (MBConv). An MBConv block is composed of three convolution
layers and two activations: (1) an expansion layer with
$\mathbf{W}\in\mathbb{R}^{1\times 1\times c\,t\times c}$ where $t$ denotes an
expansion factor over the input channel dimension $c$ followed by ReLU6, (2) a
depthwise layer with $\mathbf{W}\in\mathbb{R}^{3\times 3\times c}$ followed by
ReLU6, and (3) a projection layer with $\mathbf{W}\in\mathbb{R}^{1\times
1\times c\times c\,t}$. Removing only the first ReLU6 will allow us to fold
the expansion and depthwise layers into a convolution with
$\mathbf{W}\in\mathbb{R}^{3\times 3\times c\,t\times c}$. Interestingly,
MobileNetEdgeTPU [35] followed this exact approach and adopted fused inverted
bottleneck. In this model, expansion layers were folded with depthwise layers
despite a FLOPs increase, acknowledging the potential latency reduction. This
was performed on the the first MBConvs. Indeed, in the general case, such
folding may lead to a disadvantageous increase in FLOPs when $c\gg 9$.
Removing the second ReLU6 will result in a similar FLOPs increase. However, we
recognize that removing both ReLU6 activations will allow folding all three
convolutions into a single layer with $\mathbf{W}\in\mathbb{R}^{3\times
3\times c\times c}$. This effectively halve the computational load for blocks
in MobileNetV2 where $t=6$. In our experiments, we leverage this favorable
scenario by forcing folding of an entire MBConv blocks. For every block, we
share the same $\alpha$ among both of its activations. This ensures that they
are either removed together or remain.

Figure 6: Layer Folding results on ImageNet.

Table 2: Latency and FLOPs reduction obtained by applying Layer Folding on
MobileNetV2 (MNV2) and EfficientNet (EffNet) on ImageNet.

Model | Acc. (%) / | Latency | FLOPs
---|---|---|---
Acc. Drop (%) | Reduction | Reduction
MNV2-0.75 | 68.1 / 1.7 | 21% | 4%
MNV2-1.0 | 71.0 / 0.8 | 25% | 7%
MNV2-1.4 | 75.5 / 0.5 | 19% | 3%
EffNet-lite0 | 74.6 / 0.5 | 15% | 3%
EffNet-lite1 | 75.8 / 1.0 | 13% | 0%

We apply Layer Folding on MobileNetV2-1.4, MobileNetV2-1.0, MobileNetV2-0.75,
EfficientNet-lite0 and EfficientNet-lite1. Our implementation details are
provided in Appendix B. As shown in Figure 6, our folded models outperform
MobileNetV2 variants. For example, compared to MobileNetV2-0.75, we obtain a
model with a $1.2\%$ higher top-1 accuracy at a $14\%$ faster execution time.
Table 2 shows the latency reduction obtained for MobileNet and EfficientNet
models. These improvements should be viewed while taking the simplicity and
efficiency of our method into account.

In the experiment above, we improve networks’ latency mainly by revisiting
their activations. We suggest that other optimization techniques that operate
on different architectural components, such as pruning, can be used conjointly
with our method. That is, that their expected contribution is additive rather
than alternative to ours as they focus on width and remove weights while our
method focuses on depth and removes activations.

##  6 Conclusion

In this work we propose a novel method for removing non-linear activations.
Extensive experiments on several image classification tasks show that our
method can significantly reduce the depth of neural networks with a minor
effect on accuracy. We find that there is a minimal number of non-linear
layers to which networks can be reduced while retaining accuracy which we
denote as EDNL. We show that networks that vary by depth and architecture
share a similar EDNL, suggesting that EDNL is an attribute of a desired
mapping over a certain task rather than a specific model. Our work provides
empirical results that verify and bridge previous theoretical works on the
importance of depth and recent works that reason the profitability of
increased depth. The scope of this work is EDNL evaluation of CNNs with ReLU
activations. We leave further study of EDNL and the natural extension to other
architectures for future work. Finally, we show how reduced depth can aid
latency reduction on hardware devices and provide efficient alternatives to
mobile network architectures.

##  7 Broader Impact

The main positive impact of our work is the potential reduction of energy
consumption which aids environmental protection. This is reflected in three
aspects; first, our method allows to reduce the latency and power consumption
of neural network-based applications. Second, compared to NAS methods that
require training multiple models, our method requires fine-tuning of a single
model, reducing the cost of the optimization process. Third, our findings on
EDNL might sprout additional means to quantify and bound the expressiveness of
neural networks. Additionally, our method allows the deployment of neural
networks on computationally-constrained edge-devices that would not be
possible otherwise, along with its positive and negative societal
consequences.

## References

  * Delalleau and Bengio [2011] Olivier Delalleau and Yoshua Bengio.  Shallow vs. deep sum-product networks.  _Advances in neural information processing systems_ , 24:666–674, 2011.
  * Pascanu et al. [2014] Razvan Pascanu, Guido Montúfar, and Yoshua Bengio.  On the number of response regions of deep feed forward networks with piece-wise linear activations.  In _International Conference on Learning Representations (ICLR 2014)_ , 2014.
  * Bianchini and Scarselli [2014] Monica Bianchini and Franco Scarselli.  On the complexity of neural network classifiers: A comparison between shallow and deep architectures.  _IEEE transactions on neural networks and learning systems_ , 25(8):1553–1565, 2014.
  * Telgarsky [2016] Matus Telgarsky.  Benefits of depth in neural networks.  In _Conference on learning theory_ , pages 1517–1539. PMLR, 2016\.
  * Daniely [2017] Amit Daniely.  Depth separation for neural networks.  In _Conference on Learning Theory_ , pages 690–696. PMLR, 2017.
  * Liang and Srikant [2017] Shiyu Liang and R. Srikant.  Why deep neural networks for function approximation?  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net, 2017.  URL https://openreview.net/forum?id=SkpSlKIel.
  * Yarotsky [2017] Dmitry Yarotsky.  Error bounds for approximations with deep relu networks.  _Neural Networks_ , 94:103–114, 2017.
  * Safran and Shamir [2017] Itay Safran and Ohad Shamir.  Depth-width tradeoffs in approximating natural functions with neural networks.  In _International Conference on Machine Learning_ , pages 2979–2987. PMLR, 2017.
  * Poggio et al. [2017] Tomaso Poggio, Hrushikesh Mhaskar, Lorenzo Rosasco, Brando Miranda, and Qianli Liao.  Why and when can deep-but not shallow-networks avoid the curse of dimensionality: a review.  _International Journal of Automation and Computing_ , 14(5):503–519, 2017.
  * Petersen and Voigtlaender [2018] Philipp Petersen and Felix Voigtlaender.  Optimal approximation of piecewise smooth functions using deep relu neural networks.  _Neural Networks_ , 108:296–330, 2018.
  * Bolcskei et al. [2019] Helmut Bolcskei, Philipp Grohs, Gitta Kutyniok, and Philipp Petersen.  Optimal approximation with sparsely connected deep neural networks.  _SIAM Journal on Mathematics of Data Science_ , 1(1):8–45, 2019.
  * Greff et al. [2017] Klaus Greff, Rupesh Kumar Srivastava, and Jürgen Schmidhuber.  Highway and residual networks learn unrolled iterative estimation.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net, 2017.  URL https://openreview.net/forum?id=Skn9Shcxe.
  * Trigeorgis et al. [2016] George Trigeorgis, Patrick Snape, Mihalis A Nicolaou, Epameinondas Antonakos, and Stefanos Zafeiriou.  Mnemonic descent method: A recurrent process applied for end-to-end face alignment.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 4177–4187, 2016.
  * van Bergen and Kriegeskorte [2020] Ruben S van Bergen and Nikolaus Kriegeskorte.  Going in circles is the way forward: the role of recurrence in visual inference.  _Current Opinion in Neurobiology_ , 65:176–193, 2020.
  * Bengio et al. [1994] Yoshua Bengio, Patrice Simard, and Paolo Frasconi.  Learning long-term dependencies with gradient descent is difficult.  _IEEE transactions on neural networks_ , 5(2):157–166, 1994.
  * Glorot and Bengio [2010] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_ , pages 249–256. JMLR Workshop and Conference Proceedings, 2010.
  * Arora et al. [2018] Sanjeev Arora, Nadav Cohen, and Elad Hazan.  On the optimization of deep networks: Implicit acceleration by overparameterization.  In _International Conference on Machine Learning_ , pages 244–253. PMLR, 2018.
  * Pérez et al. [2018] Guillermo Valle Pérez, Chico Q Camargo, and Ard A Louis.  Deep learning generalizes because the parameter-function map is biased towards simple functions.  _stat_ , 1050:23, 2018.
  * Huh et al. [2021] Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola.  The low-rank simplicity bias in deep networks.  _arXiv preprint arXiv:2103.10427_ , 2021.
  * LeCun et al. [1990] Yann LeCun, John S Denker, and Sara A Solla.  Optimal brain damage.  In _Advances in neural information processing systems_ , pages 598–605, 1990.
  * Hassibi and Stork [1993] Babak Hassibi and David G Stork.  _Second order derivatives for network pruning: Optimal brain surgeon_.  Morgan Kaufmann, 1993.
  * Han et al. [2015] Song Han, Jeff Pool, John Tran, and William Dally.  Learning both weights and connections for efficient neural network.  In _Advances in Neural Information Processing Systems (NIPS)_ , pages 1135–1143, 2015.
  * Zhuang et al. [2020] Tao Zhuang, Zhixuan Zhang, Yuheng Huang, Xiaoyi Zeng, Kai Shuang, and Xiang Li.  Neuron-level structured pruning using polarization regularizer.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Tiwari et al. [2021] Rishabh Tiwari, Udbhav Bamba, Arnav Chavan, and Deepak Gupta.  Chipnet: Budget-aware pruning with heaviside continuous approximations.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=xCxXwTzx4L1.
  * Ding et al. [2021] Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun.  Repvgg: Making vgg-style convnets great again.  _arXiv preprint arXiv:2101.03697_ , 2021.
  * Simonyan and Zisserman [2015] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In Yoshua Bengio and Yann LeCun, editors, _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.  URL http://arxiv.org/abs/1409.1556.
  * Kim et al. [2016] Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin.  Compression of deep convolutional neural networks for fast and low power mobile applications.  In Yoshua Bengio and Yann LeCun, editors, _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_ , 2016.  URL http://arxiv.org/abs/1511.06530.
  * Novikov et al. [2015] Alexander Novikov, Dmitry Podoprikhin, Anton Osokin, and Dmitry P. Vetrov.  Tensorizing neural networks.  In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , pages 442–450, 2015.  URL https://proceedings.neurips.cc/paper/2015/hash/6855456e2fe46a9d49d3d3af4f57443d-Abstract.html.
  * Tai et al. [2016] Cheng Tai, Tong Xiao, Xiaogang Wang, and Weinan E.  Convolutional neural networks with low-rank regularization.  In Yoshua Bengio and Yann LeCun, editors, _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_ , 2016.  URL http://arxiv.org/abs/1511.06067.
  * Lebedev et al. [2015] Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan V. Oseledets, and Victor S. Lempitsky.  Speeding-up convolutional neural networks using fine-tuned cp-decomposition.  In Yoshua Bengio and Yann LeCun, editors, _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.  URL http://arxiv.org/abs/1412.6553.
  * Howard et al. [2017] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _arXiv preprint arXiv:1704.04861_ , 2017.
  * Sandler et al. [2018] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4510–4520, 2018.
  * Wu et al. [2019] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer.  Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10734–10742, 2019.
  * Stamoulis et al. [2019] Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu.  Single-path nas: Designing hardware-efficient convnets in less than 4 hours.  In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_ , pages 481–497. Springer, 2019.
  * Howard et al. [2019] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al.  Searching for mobilenetv3.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1314–1324, 2019.
  * Wan et al. [2020] Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, et al.  Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12965–12974, 2020.
  * Nayman et al. [2021] Niv Nayman, Yonathan Aflalo, Asaf Noy, and Lihi Zelnik-Manor.  Hardcore-nas: Hard constrained differentiable neural architecture search.  _arXiv preprint arXiv:2102.11646_ , 2021.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  _International Journal of Computer Vision (IJCV)_ , 115(3):211–252, 2015.  doi: 10.1007/s11263-015-0816-y.
  * Molchanov et al. [2017] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz.  Pruning convolutional neural networks for resource efficient inference.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net, 2017.  URL https://openreview.net/forum?id=SJGCiw5gl.
  * Han et al. [2016] Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A Horowitz, and William J Dally.  Eie: Efficient inference engine on compressed deep neural network.  _ACM SIGARCH Computer Architecture News_ , 44(3):243–254, 2016.
  * Wang et al. [2020] Huan Wang, Can Qin, Yulun Zhang, and Yun Fu.  Neural pruning via growing regularization.  _arXiv preprint arXiv:2012.09243_ , 2020.
  * Yu et al. [2018] Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad I Morariu, Xintong Han, Mingfei Gao, Ching-Yung Lin, and Larry S Davis.  Nisp: Pruning networks using neuron importance score propagation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 9194–9203, 2018.
  * Luo et al. [2018a] Jian-Hao Luo, Hao Zhang, Hong-Yu Zhou, Chen-Wei Xie, Jianxin Wu, and Weiyao Lin.  Thinet: pruning cnn filters for a thinner net.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(10):2525–2538, 2018a.
  * Jordao et al. [2020] Artur Jordao, Maiko Lie, and William Robson Schwartz.  Discriminative layer pruning for convolutional neural networks.  _IEEE Journal of Selected Topics in Signal Processing_ , 14(4):828–837, 2020.
  * Chen and Zhao [2018] Shi Chen and Qi Zhao.  Shallowing deep networks: Layer-wise pruning based on feature representations.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(12):3048–3056, 2018.
  * Neill et al. [2020] James O’ Neill, Greg Ver Steeg, and Aram Galstyan.  Compressing deep neural networks via layer fusion.  _arXiv preprint arXiv:2007.14917_ , 2020.
  * Xu et al. [2020] Pengtao Xu, Jian Cao, Fanhua Shang, Wenyu Sun, and Pu Li.  Layer pruning via fusible residual convolutional block for deep neural networks.  _arXiv preprint arXiv:2011.14356_ , 2020.
  * Blalock et al. [2020] Davis W. Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John V. Guttag.  What is the state of neural network pruning?  In Inderjit S. Dhillon, Dimitris S. Papailiopoulos, and Vivienne Sze, editors, _Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2-4, 2020_. mlsys.org, 2020.  URL https://proceedings.mlsys.org/book/296.pdf.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Zoph and Le [2016] Barret Zoph and Quoc V Le.  Neural architecture search with reinforcement learning.  _arXiv preprint arXiv:1611.01578_ , 2016.
  * Real et al. [2019] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le.  Regularized evolution for image classifier architecture search.  In _Proceedings of the aaai conference on artificial intelligence_ , volume 33, pages 4780–4789, 2019.
  * Liu et al. [2018a] Hanxiao Liu, Karen Simonyan, and Yiming Yang.  Darts: Differentiable architecture search.  _arXiv preprint arXiv:1806.09055_ , 2018a.
  * Kandasamy et al. [2018] Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabás Póczos, and Eric P. Xing.  Neural architecture search with bayesian optimisation and optimal transport.  In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , pages 2020–2029, 2018.  URL https://proceedings.neurips.cc/paper/2018/hash/f33ba15effa5c10e873bf3842afb46a6-Abstract.html.
  * Liu et al. [2018b] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy.  Progressive neural architecture search.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 19–34, 2018b.
  * Luo et al. [2018b] Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu.  Neural architecture optimization.  _arXiv preprint arXiv:1808.07233_ , 2018b.
  * He et al. [2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1026–1034, 2015.
  * Ma et al. [2020] Ningning Ma, Xiangyu Zhang, Ming Liu, and Jian Sun.  Activate or not: Learning customized activation.  _arXiv preprint arXiv:2009.04759_ , 2020.
  * Ghodsi et al. [2020] Zahra Ghodsi, Akshaj Kumar Veldanda, Brandon Reagen, and Siddharth Garg.  Cryptonas: Private inference on a relu budget.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 16961–16971. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/c519d47c329c79537fbb2b6f1c551ff0-Paper.pdf.
  * Jha et al. [2021] Nandan Kumar Jha, Zahra Ghodsi, Siddharth Garg, and Brandon Reagen.  Deepreduce: Relu reduction for fast private inference.  _CoRR_ , abs/2103.01396, 2021.  URL https://arxiv.org/abs/2103.01396.
  * LeCun et al. [1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Chen [2019] Yaofo Chen.  Pytorch cifar models.  https://github.com/chenyaofo/pytorch-cifar-models, commit:9751dd01a18d0c471b2c4522ae734757b6c94d8d, 2019.
  * Liu [2020] R Liu.  Higher accuracy on vision models with efficientnet-lite.  _TensorFlow Blog.[online] Available at: https://blog. tensorflow. org/2020/03/higher-accuracy-on-visionmodels-with-efficientnet-lite. html [Accessed 30 Apr. 2020]_ , 2020.
  * Hu et al. [2018] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7132–7141, 2018.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  _CoRR_ , abs/1912.01703, 2019.  URL http://arxiv.org/abs/1912.01703.
  * Cubuk et al. [2019] Ekin D. Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le.  Autoaugment: Learning augmentation strategies from data.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_ , pages 113–123. Computer Vision Foundation / IEEE, 2019.  doi: 10.1109/CVPR.2019.00020.  URL http://openaccess.thecvf.com/content_CVPR_2019/html/Cubuk_AutoAugment_Learning_Augmentation_Strategies_From_Data_CVPR_2019_paper.html.
  * Huang et al. [2016] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger.  Deep networks with stochastic depth.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, _Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV_ , volume 9908 of _Lecture Notes in Computer Science_ , pages 646–661. Springer, 2016.  doi: 10.1007/978-3-319-46493-0\\_39.  URL https://doi.org/10.1007/978-3-319-46493-0_39.

##  Appendix A Additional Results

Figure 7 shows the progression of $\alpha_{l}$ values on ResNet-20 and
ResNet-56 during the course of the pre-folding phase for different values of
$\lambda_{c}$. As can be seen, our method effectively remove non-linear layers
while maintaining accuracy, validating our choice of $h$ (see Equation (3),
Section 3). In addition, the resulting $\alpha$ values when pre-folding
concludes are either 0 or 1, indicating that $\tau$ values do not need special
tuning. In Figure 8 we show extreme cases where $\lambda_{c}$ values are set
such that none or all of the non-linear layers are removed.

(a) ResNet-20, $\lambda_{c}=0.1$

(b) ResNet-56, $\lambda_{c}=0.1$

(c) ResNet-20, $\lambda_{c}=0.2$

(d) ResNet-56, $\lambda_{c}=0.2$

(e) ResNet-20, $\lambda_{c}=0.4$

(f) ResNet-56, $\lambda_{c}=0.4$

Figure 7: Layer Folding with different values of $\lambda_{c}$.

(a) ResNet-20, $\lambda_{c}=0.01$

(b) ResNet-56, $\lambda_{c}=0.01$

(c) ResNet-20, $\lambda_{c}=1$

(d) ResNet-56, $\lambda_{c}=1$

Figure 8: Layer Folding with extreme values of $\lambda_{c}$ such that none
(top) or all (bottom) of the non-linear layers are removed.

##  Appendix B Implementation Details

For MNIST, we use fully-connected networks with depth $L\in[2:10]$, ReLU
activation and width $d=256$ for all layers. We train these networks for 10
epochs with a learning rate of $0.1$ and SGD with momentum of 0.9. For
CIFAR-10 and CIFAR-100, we use pre-trained models of ResNet-20, ResNet-32,
ResNet-44, ResNet-56, VGG16 and VGG19 from [62]. We apply Layer Folding with
$c_{l}=1$, $l=1:L$, $p=2$, $\tau=0.9$. We note that these values were chosen
for simplicity, and that from our experiments larger values of $p$ and $\tau$
provide similar results. We use $\lambda_{c}=1$ and multiply it by $2^{0.5}$
($2^{-0.5}$) to increase (decrease) the number of non-linear layers being
removed. The networks above were trained for 200 epochs, with a starting
learning rate of 0.1 which is reduced by one-tenth at the 100th and 150th
epochs. Hence, we run the pre-folding and post-folding fine-tuning phases for
100 epochs each with a learning rate of $10^{-3}$. When training from scratch
folded ResNet and VGG models (see Figure 4, Section 4), we match the training
schedule of the folded networks, i.e., a starting learning rate of 0.1 which
is reduced by one-tenth at the 100th and 150th epochs of the total 400 epochs.
As training depends on initialization, we repeat the training for five times
and take the average test accuracy. We implemented our code using PyTorch [65]
and NVIDIA GeForce GTX 1080Ti. In these settings, both pre-folding and post-
folding phases take less than an hour for each of the above networks.

For ImageNet, we use MobileNetV2-1.4, MobileNetV2-1.0, MobileNetV2-0.75,
EfficientNet-lite0 and EfficientNet-lite1. For all models we use augmentations
drawn from the models’ original training flow - standard augmentations for
MobileNetV2 and AutoAugment [66] for EfficientNet-lite. We use Adam optimizer
($\beta_{1}=0.9$, $\beta_{2}=0.999$), $0.1$ label smoothing and batch size of
$64$. We use weight decay of $2\mathrm{e}{-5}$. During the pre-folding phase
we use a learning rate of $\mathrm{1e-4}$ and during the post-folding phase we
use a cosine learning rate decay from $1\mathrm{e}{-5}$ to $1\mathrm{e}{-6}$.
For MobileNetV2 variants we use exponential moving average with a decay factor
of $0.9999$. The pre-folding phase consists of $5$ epochs and the post-folding
phase consists of $20$ epochs. For EfficientNet-lite we use Stochastic Depth
[67] with survival probability of $0.8$ and a dropout rate of $0.2$ during the
post-folding phase. The pre-folding phase consists of $10$ epochs and the
post-folding phase consists of $25$ epochs.

For latency measurements we used a batch size of $16$. We tested inference
latency using Tensorflow’s built-in benchmarking tool (`tf.test.Benchmark`) on
NVIDIA Titan X Pascal GPU, averaged on 100 iterations with 10 burn-in
iterations.

Our code is available at https://github.com/LayerFolding/Layer-Folding.


Although the analysis in Section <ref> was restricted to the case of a two-dimensional torus, it generalizes with minimal effort to the case of an $m$-dimensional torus simply by considering a rotation vector $\rho\in\mathbb{R}^{m-1}$ and a vector-valued angle $\phi\in\mathbb{S}^{m-1}$, integrals over $\mathbb{S}^{m-1}$ rather than $\mathbb{S}$, and additional phase conditions. For the transversally stable case, the results reduce to those of [Demir et al., 2010], which are presented there without derivation. A further generalization to the case of a piecewise-smooth dynamical system is open for investigation (cf. [Szalai and Osinga, 2009]).

For the case of dynamical systems with time delay, a preliminary analysis in [Ahsan et al., 2022] showed the relationship between an adjoint variable and the sensitivity of the orbital duration to violations of the periodic orbit constraint. This same analysis can, of course, be generalized according to the treatment in this paper without significant additional effort. More interesting is to consider the adjoint-based sensitivity analysis for a normally hyperbolic quasiperiodic invariant torus of a smooth dynamical system with delay. While the adjoint-based approach was used in [Ahsan et al., 2020] to search for optimal quasiperiodic invariant tori in an example system with delay, the analysis lacked the theoretical rigor of the present treatment, in particular as relates to the solvability of the adjoint conditions. This appears to be a worthwhile problem to pursue.

Finally, we briefly discuss opportunities associated with further development of coco to support analyses of the form considered here. For example, while the example in Section <ref> relied on low-level functionality (coco_add_func, coco_add_pars, coco_add_glue, and coco_add_adjt), it is straightforward to build a composite constructor that encapsulates the individual steps of constraint construction and a separate composite constructor that encapsulates the individual steps of constructing contributions to the adjoint conditions. Such development would then benefit from the encoding of utility functions for generating an initial solution guess, extracting solution data, and graphing families of solutions and sensitivities.

Indeed, a similar construction could apply to the case where the map $M$ is only implicitly defined by the solution to a differential equation. In this case, the zero problem $M_\mathrm{res}=0$ is replaced with the collocation problem for the corresponding trajectory segment. Rather than coco_add_func and coco_add_adjt, here calls to the constructors ode_isol2coll and adjt_isol2coll would instantiate individual trajectory segments and the corresponding contributions to the adjoint conditions. Finally, the index arrays muidx and maidx would be constructed from a subset of the indices associated with each trajectory instance, specifically those corresponding to the initial and final points on the trajectory segment and the corresponding parameter values. The implementation in coco of the higher-dimensional case should follow the same pattern, with necessary modifications to the discretization, now over $\mathbb{S}^{m-1}$, and the addition of a required number of phase conditions.

§ CODE AVAILABILITY

The code included in this paper constitutes fully executable scripts. Complete code, including that used to generate the results in Fig. <ref>, is available at <https://github.com/jansieber/adjoint-sensitivity2022-supp>.

§ ACKNOWLEDGMENTS

The second author's research is supported by the UK Engineering and Physical Sciences Research Council (EPSRC) grants EP/N023544/1 and EP/V04687X/1.

[Ahsan et al., 2020]
Z Ahsan, H Dankowicz, and J Sieber.
Optimization along families of periodic and quasiperiodic orbits in
dynamical systems with delay.
Nonlinear Dynamics, 990 (1):0 837–854, 2020.

[Ahsan et al., 2022]
Z Ahsan, H Dankowicz, M Li, and J Sieber.
Methods of continuation and their implementation in the coco software
platform with application to delay differential equations.
Nonlinear Dynamics, 2022.

[Bernardo et al., 2008]
M Bernardo, C Budd, A R Champneys, and P Kowalczyk.
Piecewise-smooth Dynamical Systems: Theory and Applications.
Springer, London, 2008.

[Cabré et al., 2003]
X. Cabré, E. Fontich, and R. de la Llave.
The parameterization method for invariant manifolds I
manifolds associated to non-resonant subspaces.
Indiana University mathematics journal, pages 283–328,

[Cabré et al., 2003]
X Cabré, E. Fontich, and R. de la Llave.
The parameterization method for invariant manifolds II
regularity with respect to parameters.
Indiana University mathematics journal, pages 329–360,

[Cabré et al., 2005]
X. Cabré, E. Fontich, and R. De La Llave.
The parameterization method for invariant manifolds III
overview and applications.
Journal of Differential Equations, 2180 (2):0
444–515, 2005.

[Corner et al., 2020]
S Corner, A Sandu, and C Sandu.
Adjoint sensitivity analysis of hybrid multibody dynamical systems.
Multibody System Dynamics, 490 (4):0 395–420,

[Dankowicz and Schilder, 2013]
H Dankowicz and F Schilder.
Recipes for continuation.
SIAM, 2013.

[de la Llave, 2001]
R de la Llave.
A tutorial on KAM theory.
In A Katok, R de la Llave, Y Pesin, and H Weiss, editors,
Smooth Ergodic Theory and Its Applications, volume 69 of
Proceedings of Symposia in Pure Mathematics, pages 175–292, 2001.

[Demir et al., 2010]
A. Demir, C. Gu, and J. Roychowdhury.
Phase equations for quasi-periodic oscillators.
In 2010 IEEE/ACM International Conference on Computer-Aided
Design (ICCAD), pages 292–297, 2010.

[Govaerts and Sautois, 2006]
W Govaerts and B Sautois.
Computation of the phase response curve: A direct numerical approach.
Neural Computation, 180 (4):0 817–847, 2006.

[Haro et al., 2016]
A. Haro, M. Canadell, J.-L. Figueras, A. Luque, and J.-M. Mondelo.
The Parameterization Method for Invariant Manifolds, volume
195 of Applied mathematical sciences.
Springer, 2016.

[Hirsch et al., 1977]
M Hirsch, C Pugh, and M Shub.
Invariant manifolds, volume 583 of Lecture Notes in
Springer-Verlag, Berlin, 1977.

[Kuznetsov, 2013]
Y A Kuznetsov.
Elements of applied bifurcation theory, volume 112.
Springer Science & Business Media, 2013.

[Li and Dankowicz, 2018]
M Li and H Dankowicz.
Staged construction of adjoints for constrained optimization of
integro-differential boundary-value problems.
SIAM Journal on Applied Dynamical Systems, 170
(2):0 1117–1151, 2018.

[Li and Dankowicz, 2020]
M Li and H Dankowicz.
Optimization with equality and inequality constraints using parameter
Applied Mathematics and Computation, 375:0 125058,

[Moser, 1966]
J Moser.
On the theory of quasiperiodic motions.
SIAM Review, 80 (2):0 145–172, 1966.

[Novičenko and Pyragas, 2012]
V Novičenko and K Pyragas.
Phase reduction of weakly perturbed limit cycle oscillations in
time-delay systems.
Physica D: Nonlinear Phenomena, 2410 (12):0
1090–1098, 2012.

[Park et al., 2018]
Y Park, K M Shaw, H J Chiel, and P J Thomas.
The infinitesimal phase response curves of oscillators in piecewise
smooth dynamical systems.
European Journal of Applied Mathematics, 290
(5):0 905–940, 2018.

[Rubino et al., 2018]
A Rubino, M Pini, P Colonna, T Albring, S Nimmagadda, T Economon, and J Alonso.
Adjoint-based fluid dynamic design optimization in quasi-periodic
unsteady flow problems using a harmonic balance method.
Journal of Computational Physics, 372:0 220–235,

[21]
F Schilder, H Dankowicz, and M Li.
Continuation Core and Toolboxes (COCO).
Accessed: 2022-02-26.

[Shirasaka et al., 2017]
S Shirasaka, W Kurebayashi, and H Nakao.
Phase reduction theory for hybrid nonlinear oscillators.
Physical Review E, 950 (1):0 012212, 2017.

[Szalai and Osinga, 2009]
R Szalai and H M Osinga.
Arnol'd tongues arising from a grazing-sliding bifurcation.
SIAM J. Appl. Dyn. Syst., 80 (4):0 1434–1461,

[Tortorelli and Michaleris, 1994]
D A Tortorelli and P Michaleris.
Design sensitivity analysis: Overview and review.
Inverse Problems in Engineering, 10 (1):0
71–105, 1994.

[Traverso and Magri, 2019]
T Traverso and L Magri.
Data assimilation in a nonlinear time-delayed dynamical system with
Lagrangian optimization.
Lecture Notes in Computer Science (including subseries Lecture
Notes in Artificial Intelligence and Lecture Notes in Bioinformatics),
11539:0 156–168, 2019.

Received xxxx 20xx; revised xxxx 20xx.