# Memory in cyclically crumpled sheets

Amit Dawadi  Department of Physics, Clark University, Worcester, Massachusetts
01610, USA  Arshad Kudrolli  Department of Physics, Clark University,
Worcester, Massachusetts 01610, USA

###### Abstract

We investigate the crumpling of a sheet as it is repeatedly crushed onto
itself by rolling it into a cylinder and twisting it axially while allowing
the end-to-end length to evolve freely. As deduced from its plastic
deformations, the sheet creases and collapses into structures which repeat and
sharpen over hundreds of cycles to a remarkable degree before forming new
configurations. The observed metastablilty increases with applied cycles
leading to recurrent structures over a significant range of loading before the
sheet tears, but reconfigurations continue to occur for large enough loading
amplitude. The evolution of the sheet structure as measured by the mean
curvature and the total crease length is found to increase logarithmically
with cycle number with a rate which increases with degree of compression. We
explain the overall extent of creasing using flat folding models, and show the
logarithmic growth as being a consequence of individual creases becoming
sharper with number of folding cycles, and due to the bifurcation in the
curvature field leading to the formation of new creases and folding pathways.
Thus, we show that elastoplastic sheets can follow complex folding pathways to
form convergent structures after a sufficiently large number of training
cycles provided material fatigue remains unimportant.

##  I Introduction

A crumpled sheet shows a distinct pattern of intersecting creases which carry
the subtle imprints of the complex buckling pathways followed as a result of
applied constraints [1, 2, 3, 4, 5]. The evolution of the observed disordered
ridges and facets can provide broad lessons on a system that can typically
only access metastable states [6, 7, 8]. While there have been considerable
number of experimental and theoretical studies on memory effects after the
application or removal of stress [9, 10, 11, 12], the evolution under repeated
loading cycles has only started garnering attention more recently. As a result
of sensitive dependence on the loading conditions, the sheet cannot be
expected to pass the same path if the sheet is crumpled again, thus leading to
further creases and hysteresis [13, 14]. Nonetheless, the presence of creases
influences how the sheet subsequently collapses when the loading cycle is
repeated as the sheet rigidifies along the length of the crease, while
becoming easier to fold about the crease [15], leading to guided folding
pathways as in origami [16]. Indeed, supervised strengthening folds and
weakening of misfolds in a repeatedly crumpled sheet has been suggested as a
paradigm for reinforced learning of structures in mechanical systems subjected
to classes of forces [17]. It has been argued based on numerical simulations,
that the energy landscape of a folded sheet can become deeper under repeated
loading, driving the sheet to unique folding pathways even under unsupervised
conditions [18]. However, investigations of these ideas have been conducted
with actual sheets remain limited [19].

In the case of athermal granular systems, logarithmic relaxation of
observables such as density can occur upon application of thousands of
vibration cycles [20]. It has been also shown that reversible motion and
relaxation to limit cycle can be observed in cyclically sheared colloidal
suspensions and amorphous solids that show irreversible motion above a
critical amount of deformation [21, 22]. In the case of a pre-crumpled sheet
under repeated loading and unloading, hysteresis and an approach to a limit
cycle have been observed when the loading is below the threshold required to
create additional crumples [23]. But, even a pre-creased ordered origami can
snap between different deployed structures because of their multistability
[24, 25, 26]. Furthermore, even single folds in elastoplastic sheets can age
[11, 24] impacting their compaction properties [27, 28]. Thus, the evolution
of the crumpled structure of a sheet subject to large number of repeated
loading and unloading cycles of varying strength, remains unclear.

To address the evolution of an elastoplastic sheet subject to large number of
training cycles, we examine the crumpling of a Mylar sheet which is clamped
onto circular end caps, and then repeatedly crushed by applying an axial twist
akin to wringing a towel [29]. Because of the inextensible nature of the
sheet, its length contracts axially, and the sheet collapses onto itself while
forming a bundle [30]. A hallmark of our design is that the applied strain can
be reversed back to zero, and thus the loading cycle can be repeated
essentially indefinitely under well prescribed conditions. This system enables
us to examine the effect of sheet training over a wide range of compaction
before material fatigue leads to fractures or tears in the sheet. Because no
constraint is imposed on the end-to-end distance, it can serve as a measure of
the macroscopic state of the system, which can evolve with cycle number,
enabling a subtle examination of the effect of cyclic loading. The broad
question we seek to understand is whether convergent structures can be
observed under unsupervised training cycles, and if so under what range of
conditions. As we discuss in the following, the sheets fold to form
increasingly long-lived metastable structures over sufficiently large number
of loading cycles even as the system ages logarithmically.

##  II Crumpling by twisting

Figure 1: Crumpling of an elastoplastic sheet with a twist. (A) Images of a
sheet ($L_{0}=16.5$ cm, $W_{0}=16.5$ cm, and $h=90$ $\mu$m) as it is twisted
and then untwisted about its axis through $\theta_{f}=720^{\circ}$. (B) The
crumples observed after one cycle corresponding to various $\theta_{f}$. The
mean curvature $H$ are superimposed with values denoted by the color bar. (C)
The end-to-end length $L$ as a function of $\theta$ shows hysteresis which
decreases with $n$. (D) The strain $\varepsilon$ versus $\theta_{f}$ can be
described by the exponential function
$\varepsilon=1-\exp(-\gamma_{n}\theta_{f})$ with $1/\gamma_{1}=588^{\circ}$
and $1/\gamma_{100}=636^{\circ}$ corresponding $n=1$ and 100, respectively.
(E) The distribution of the magnitude of the mean curvature $|H|$ broadens
with $n$. (F) The evolution of the average magnitude of mean curvature
$\langle|H|\rangle$ with $n$ is described by a logarithmic function
$\langle|H|\rangle=H_{1}+H_{2}\log n$. (G) $H_{1}$ and $H_{2}$ increase with
increasing $\theta_{f}$.

Figure 1A shows the evolution of a sheet as it is twisted quasi-statically up
to an axial angle $\theta_{f}=720^{\circ}$, and then untwisted (see Methods).
As the twist angle $\theta$ is increased, the sheet is observed to buckle with
folds slanted along the direction of twist in the central section between the
clamps ends, while the sections close to the clamps remain relatively
cylindrical [31, 32]. Thus, the twisting is observed to lead to the self-
collapse and contraction of the sheet which is not directly imposed by an
inward moving bounding surface as in a piston [13]. The buckling instability
is similar to those found in twisted long cylindrical shells [33] (see
Supplementary Information (SI) Section I). As the twist is increased further,
self-contact occurs, the initial folds split, and the sheet crumples. When the
applied twist is reversed, the crumpled structure show auxetic behavior [34]
with increases in both length and diameter as the sheet returns back close to
its initial cylindrical shape as the elastic components of folds relax while
showing imprints of plastic deformation. When the twist cycle is repeated, one
notes that the sheets folds largely along creases which were formed during the
first cycle (see Supplementary Movie S1).

Figure 1B shows scans of the sheet for increasing $\theta_{f}$ using laser
profilometry (see Methods) rendered with color corresponding to local mean
curvature of the surface $H$. In the case of $\theta_{f}=180^{\circ}$,
imprints of the initial folds that form can be observed even as the sheet
comes into self-contact and the folds start to bend and break to accommodate
the increasing twist. The sheet is observed to become more widely creased and
disordered as $\theta_{f}$ is increased to $360^{\circ}$ and
$\theta_{f}=720^{\circ}$. Thus, elastoplastic sheets become disordered, unlike
hyperelastic sheets which show ordered folds corresponding to Schläfli symbols
when twisted through similar $\theta$ [35]. We further find that the crumpled
structure largely repeated itself after application of a full cycle with
increasing $n$. But, formation of new creases and slow evolution of overall
structure can be also observed (Supplementary Movie S2). In the following, we
characterize and quantify the evolution of the system with $n$ as a function
of the degree of compaction by varying $\theta_{f}$.

We quantify the strain experienced by the system using the end-to-end length
$L$ of the sheet from one clamp edge to the other, and plot it in Fig. 1C as a
function of $\theta$. $L$ decreases faster with $\theta$ compared to a set of
infinitesimally thin inextensible strings that initially form a hyperbolic
hyperboloid (see SI Section II and SI Fig. S2). While untwisting, $L$
increases but follows a different path which is systematically higher compared
to the twisting phase, before crossing and reaching a length $L_{n}$ which is
slightly below $L_{0}$. When the sheet is again subjected to the same cycle,
the hysteresis is observed to decrease on average with each cycle (see Fig.
1C), while the length after a cycle continues to contract (see SI Section
III). We characterize the compression of the sheet by measuring the end to end
length $L_{c}$ when $\theta_{f}$ is reached, and calculating the axial strain
$\varepsilon=1-L_{c}/L_{0}$, and plotting it averaged over $3$ trails versus
$\theta$ in Fig. 1D. We observe that its increase can be described with the
function $\varepsilon=1-\exp(-\gamma_{n}\theta_{f})$, where $\gamma_{n}$
decreases slowly with $n$ as the plastic deformation in the sheet accumulate.
The exponential function is consistent with the sheet becoming increasingly
more difficult to compress with increasing compaction.

To quantify the evolution with applied loading cycles, we obtain the
distributions of the magnitude of the local mean curvature $P(|H|)$ and plot
them in Fig. 1E for various $n$. We observe that it is broadly distributed,
and broadens further as the sheet is repeatedly twisted showing that the sheet
is being increasingly creased not only with $\theta_{f}$ (see SI Section III),
but with $n$ as well (see SI Movie S2). We plot the average of the
distribution $\langle|H|\rangle$ in Fig. 1F and observe that
$\langle|H|\rangle$ grows systematically higher with increasing $\theta_{f}$.
Further, each $\langle|H|\rangle$ is well described by a logarithmic function
$\langle|H|\rangle=H_{1}+H_{2}\log n$, where $H_{1}$ is the magnitude of the
mean curvature observed after the application of the first cycle, and $H_{2}$
captures the growth with number of cycles. As shown in Fig. 1G, $H_{2}$
increasing approximately proportional to $H_{1}$. Thus, we find that not only
do the crumples formed depend on the applied twist, but that the rate of the
grow upon repeated loading is essentially proportional the creases formed
initially.

##  III Logarithmic growth of curvature

Figure 2: Evolution of crease curvature with applied loading. (A) The mean
curvature of a fold $H^{f}$ in sheet which is folded in half as a function of
applied force per unit cross section area, and the corresponding mean
curvature after the sheet is unfolded $H^{u}$ to measure the resulting plastic
deformation ($L_{0}=16.5$ cm). A fit to $H^{f}=\chi^{f}\sigma$ and
$H^{u}=\chi^{u}\sigma$ with $\chi^{f}=0.5$ mmN-1 and $\chi^{u}=0.06$ mmN-1,
respectively, are shown. Inset: A cross section of the folded and unfolded
sheets are also shown. (B) $H^{u}$ as a function of $n$ ($\sigma=0.7$ Nmm-2;
$T=30$ seconds). A fit to $H^{u}=H^{un}_{1}+H^{un}_{2}\log(n)$ is shown with
$H^{un}_{1}=0.049$ mm-1 and $H^{un}_{2}=0.10$ mm-1. (C) $H^{u}$ increases as a
function of time interval over which the same constant force is applied. A
function $H^{u}=H^{ut}_{1}+H^{ut}_{2}\log(t)$ is shown with $H^{ut}_{1}=0.051$
mm-1 at t = 30 seconds and $H^{ut}_{2}=0.014$ mm-1min-1 is shown. The effect
on $H^{u}$ is significantly lower over $t$ compared with $n$ while considering
the same time taken to apply the loading cycles.

We examine the curvature of a single crease upon repeated folding as a
starting point to understand the logarithmic growth, and any relation to
Arrhenius behavior that has been observed in response to application of stress
in crumpled sheets [11, 12, 36, 37]. While the slow relaxation of a fold after
application of a stress and its functional form as a function of time has been
well studied [11, 36], the functional evolution of the fold over a large
number of training cycles is not available. Therefore, a folded sheet is
placed between parallel plates and a normal force is applied for a fixed time,
and the force is then removed to relax the elastic component of the crease and
the unfolded sheet is placed on a flat-bed for scanning. A representative
cross-section of the sheet while folded and unfolded is shown in the inset to
Fig. 2A. We find the mean curvature along the crease while loaded and
unloaded, and plot them in Fig. 2A as a function of the stress $\sigma$
corresponding to the applied force per unit length of crease and the sheet
thickness. We observe that the curvatures for the folded and unfolded sheet
evolve linearly as a function of applied stress over the typical range of
curvatures observed in our crumpling experiments. Thus, even after loading is
removed, a fraction proportional to the maximum loading applied can be
identified, giving us confidence that the physical features of a crushed sheet
can be still identified after the sheet is relaxed due to the residual plastic
deformations.

Next, we perform measurements of the mean curvature of an unfolded sheet
$H^{u}$ for a typical crease as a function of number of folding and unfolding
cycles for $\sigma=0.7$ Nmm-2 applied for $30$ seconds, before unfolding and
placing the sheet on a flat-bed for scanning. As shown in Fig. 2B, we find
that the curvature increases logarithmically with the number of applied
loading cycles $n$, as $H^{u}=H^{un}_{1}+H^{un}_{2}\log(n)$ with a rate
$H_{2}^{un}=0.10$ mm-1. Because $\sigma=\chi^{-1}H^{u}$, and since
$n\sim\omega t$, one can see there is analogy between the logarithmic observed
here and those observed in time [24]. To probe this further, we measured the
evolution of curvature when the applied load is applied over varying time
interval $t$. As shown in Fig. 2C, we find that the increase in $H^{u}$ with
$t$ is significantly lower compared with the increase in curvature with $n$
over comparable cumulative loading time. Hence we find that the repeated
folding and unfolding of the crease leads to far greater plastic deformations
and is the dominant reason for the logarithmic increase in our experiments.
Thus, the evolution in our experiments appears to be consistent with memory
observed due to cyclic driving in sheared suspensions [21, 38], and different
from the time evolution reported previously in single folds and crumpled
sheets [11, 12, 37].

##  IV Evolution of Creases

Figure 1B shows the regions with high $H$ are organized along intersecting
networks of creases with twist crumpling, visually similar to those which have
been noted with other crumpling protocols [2, 13, 39]. To analyze the creases,
we identify these regions where most of the plastic deformation in the sheet
occurs (see SI Section IV). The total crease length $\ell$ normalized by
$L_{0}$ as a function of $n$ is plotted in Fig. 3A for various $\theta_{f}$.
We observe that it increases in all cases with number of cycles, and can be
fitted by logarithm function $\ell=\ell_{1}+\ell_{2}\log(n)$, where $\ell_{1}$
is the total crease length after the first loading cycle and $\ell_{2}$ is the
rate of growth corresponding to the creases with $n$. Both, $\ell_{1}$ and
$\ell_{2}$ increase with $\varepsilon$, as shown in Fig. 3B,C, respectively.

In order to gain an understanding of the growth of the creases with loading,
we studied complementary flat folding of a sheet into a smaller and smaller
volume (see SI Section VI). These prescribed folding method allows us to not
only estimate the total crease length but also practical differences due to
folding a sheet with finite thickness. We define a compaction in this square
folding case $\varepsilon_{s}=1-1/m$, where $L_{0}/m$ corresponds to length of
the sides after $m$ square folds and plot $\ell$ in Fig. 3A. In spite of the
differences in definition of compaction, we find $\ell$ increases roughly over
the same range of $\ell_{1}$ as the sheet is folded to form smaller and
smaller ordered squares as in twist crumpling. Growing disorder due to the
thickness of the sheet results in a somewhat faster growth compared to the
ideal square fold calculated limit.

A logarithmic growth of crease length has been reported in sheets crushed
uniaxially inside a piston by Gottesman, et al. [13], but with a different
two-parameter functional form. In those studies, the sheets were flattened
after each cycle, and thus random perturbations were introduced which leads to
the creation of new creases as the start point for the folding sequence is
different. Indeed, if we introduce similar perturbations in our system, by
unmounting and mounting after flipping the sheet inside out after each cycle,
we find logarithmic growth as well, but with far greater rate of increase (see
SI Section X) corresponding to the creation of larger numbers of fresh
creases. Thus, our study finds that creases grow under repeated loading even
when such random perturbations are absent due to the underlying plastic
evolution of the sheet curvature under repeated loading. This evolution leads
to sharpening curvature which then can exceed the thresholds used to identify
creases and their observed growth.

We plot the distribution of crease length between branch points for various
$\theta_{f}$ in Fig. 3D, and the mean ridge length in Fig. 3E. We find that
the tails in the ridge length distribution $P(\ell_{r})$ grows as the creases
spread more broadly between the twisted ends with increasing $\theta_{f}$ as
can be seen from Fig. 1B and Fig. S7. $P(\ell_{r})$ for the various
$\theta_{f}$ decay exponentially for sufficiently large $\ell_{r}$, consistent
with the gamma-distribution derived using a configurational entropy approach
[40, 41, 42]. Further, the average crease length $\langle\ell_{r}\rangle$
increases initially with $\varepsilon$ before reaching a broad peak and
decreasing as the crushed volume continues to shrink. Plotting the number of
ridges $n_{r}$ versus cycle number $n$ in Fig. 3F, we find that it grows
consistent with a logarithmic function. Thus, we find that the total crease
length grow because the total number of creases increase, even when
$\langle\ell_{r}\rangle$ does not change significantly with $n$.

Figure 3: Statistical evolution of creases with loading. (A) The evolution of
the total crease length $\ell$ with $n$ is well described by the logarithmic
function $\ell=\ell_{1}+\ell_{2}\log(n)$. (B) The crease length after the
first loading cycle $\ell_{1}$ increases with compaction $\varepsilon$. The
calculated and observed crease lengths when a sheet is folded into
increasingly small squares plotted versus compaction $\varepsilon_{s}$
estimated using the size of the folded sides. (C) The rate of increase of
crease length $\ell_{2}$ also increases with compression. (D) The distribution
of crease length ($\ell_{r}$) fitted with gamma function
$P(\ell_{r}|\alpha,\beta)={\beta^{{\alpha}}\ell_{r}^{\alpha-1}\exp(-\beta\ell_{r})}/{\Gamma(\alpha)}$,
where the fitted value of the shape parameter is $\alpha=1.55$ and the scale
parameter is $\beta=7.84$ mm-1. (E) The average length of creases increases
with compression as creases spread out from the central regions of the sheet,
and then starts to decrease as the ridges are broken as the sheet is forced
into a smaller volume. (F) The total number of ridges $n_{r}$ increases
logarithmically reflecting the growth in $\ell$.

##  V Metastability

Figure 4: Reversibility and irreversible transformation of sheet with
training. (A-C) The evolution of a sheet cross section shows creases becoming
sharper with $n$, and the formation of increasing number of new creases and
sudden changes in the surface profile with increasing $\theta_{f}$. (D) A
schematic representation of the evolving energy landscape of the sheet which
can give rise to slow evolution and rapid transitions in sheet structure. (E)
$\xi$ versus $n$ shows spikes corresponding to changes in the profile
($\theta_{f}=720^{\circ}$). (F) The number of peaks corresponding to
reconfigurations increases with $\theta_{f}$, but decreases with $n$ for
$\theta_{f}<720^{\circ}$.

Since the full surface scans rely on dismounting the sheet after starting with
a pristine sheet, it is only possible to understand statistical trends with
those measurements. As a way to interrelate the crumpled configuration of the
sheet after application of a training cycle, we examine the height profile
$h(x,n)$ of the sheets along a length-wise section $x$ after each cycle $n$.
Representative evolution of a section of the sheet corresponding to
$\theta=180^{\circ}$, $360^{\circ}$, and $720^{\circ}$ are shown in Fig. 4A-C,
respectively. While the sections evolve rapidly in each case over the first
tens of cycles, they converge to show a set of peaks and valleys which recur
in the case of $\theta_{f}=180^{\circ}$. Slow evolution of creases and
appearance of new creases can be noted due to the evolving constraints [43]
even as the sheet peaks and valleys mostly repeat and sharpen with $n$. It
should be noted that new creases do not have to arise at that location, but
changes in the structure elsewhere can also lead the sheet to move relative to
the fixed section being examined. While, one notes similar trends in the case
of $\theta_{f}=720^{\circ}$, sudden rearrangements occur as well, whereby the
entire cross-section undergoes a rapid snap-through event. Because the creases
become sharper with increasing number of cycles, the number of creases in
$h(x,n)$, identified by using a fixed curvature threshold criterion, continues
to grow over $n=1000$ even when reconfigurations are absent (see Fig. S10).

Thus, these observations further corroborate the discussion of the evolution
of a single crease following Fig. 2B, i.e. plastic damage accumulates in
regions which have sustained damage in previous cycles. As these constraints
change, they can lead to both creation and destruction of creases as the
curvature field adds pairs of peaks and valleys (see SI Fig. S11). Here we
observe that the continuous evolution of the crease curvatures due to ageing
can drive rapid reconfiguration of the crumpled structures even after the
sheet structure repeats over hundreds of cycle even while evolving slowly.
Thus, the picture which emerges from these observations is more subtle than
the continuous approach to a limit cycle over tens of cycles shown by Shohat,
et al. [23] with a precrumpled sheet subject to perturbative compression
cycles. In particular bifurcation and deepening of the stability landscape can
cause continuous evolution of the creases, as well as sudden transitions,
respectively, as shown schematically in Fig. 4D.

To quantify the metastability, we computed a measure
$\xi=\langle(h(x,n+2)-h(x,n))^{2}\rangle_{x}$, where $\langle..\rangle_{x}$
corresponds to average over $x$ and plot it in Fig. 4E in the case of
$\theta_{f}=720^{\circ}$. We observe that there are bursts of rearrangements
after a large number of cycles where the sheet appears to converge to a stable
structure. Further, these rearrangements show significant variations from
trial to trial, with long periods of quiescence between large rearrangements.
We quantify the relative intensity of the rearrangements over $n$ by finding
the number of peaks in $\xi$ above a given threshold in the first, second and
third part of the 1000 cycles, averaged over the three trails in the case of
each $\theta_{f}$. The number $\langle N_{\xi}\rangle$ is plotted in Fig. 4F
and shows that the rearrangements for $\theta_{f}\leq 540^{\circ}$ eventually
decrease and the folded shapes converge over time even as the creases sharpen.
However, for large enough deformations corresponding to
$\theta_{f}=720^{\circ}$, the rearrangements can continue to occur even as
they appear to converge over hundreds of cycles. In fact, we find that
rearrangements persist in the case of $\theta_{f}=720^{\circ}$ even as we
performed trials with $n=2000$ and the sheet started to tear along some of the
creases due to the weakening caused by repeated plastic deformations (see SI
Section IX). Thus, a recurrent folded structure does not always exist for
large enough deformations as fatigue leads the sheet to progressively tear up.

The overall convergence to a limit cycle over a transient number of cycles
that increase with increased $\theta_{f}$ is similar to that observed in
cyclically sheared amorphous materials [22]. There, a relaxation to a limit
cycle, and transitions between limit cycles was also observed with applied
shear amplitude. For large enough amplitudes only irreversible migration of
constituent particles was observed, suggesting a transition to chaos as in
various dynamical systems [22, 21]. In particular the convergence of the
folded structure and its evolution with increasing $\theta_{f}$ as captured in
Fig. 4 can be noted to be similar. Our system, with the slow evolution of
curvature folds with training cycle, further indicates as well that the broad
features of reversibility and irreversibility with applied amplitude can
survive sufficiently slow evolution of fatigue in the system.

##  VI Conclusions

In summary, we find that a repeatedly crumpled sheet can perform reversible
transformations even as they are crushed significantly over a wide range of
applied loading amplitudes. While temporal logarithmic relaxation has been
noted in crumpled sheets, here we demonstrate logarithmic evolution of
structure with training cycles which manifests itself in a wide range of
measurable quantities. These include the evolution of the curvatures of the
sheets, the crease length, the number length, and the crushed length reached
with training cycles. Even when the system displaces metastability, we find
that a convergent structure can be reached after a sufficiently large number
of training cycles depending on the depth of the compression cycle. Our
results echo the approach to limit cycle found in weakly perturbed pre-
crumpled sheets [23], but significantly broaden our understanding of when
reversible behavior can be expected. Our experiments demonstrate that
elastoplastic sheets can satisfy the proposed deepening of the energy
landscape which would give rise to programmable folding pathways leading to
unique complex structures using origami [18]. However, for sufficiently large
loading and compaction, a convergent structure is not reached even as tears
develop as the sheet fatigues. Overall, we find that the behavior of crumpled
sheet under repeated loading is similar to other disordered system, and in
particular the reversible transformation of athermal particles subjected to
cyclic shear [21, 22, 38]. There, reversibility is observed for a range of
sufficiently small strain amplitude, but irreversibility develops with
increasing amplitude depending on the Lyapunov exponents of the system. While,
a similar analysis is beyond the scope of this study, it may be anticipated
that our results can stimulate work in that direction.

## Methods

###  VI.1 Materials and Apparatus

Biaxially oriented polyethylene terephthalate (BoPET), also known as Mylar,
sheets of length $L_{0}=16.5$ cm, width $W_{0}=16.5$ cm, and thickness
$h_{0}=90$ $\mu$m were used in the study. These sheets have a bending modulus
$E=4.16$ GPa, obtained with standard beam bending measurements. The sheet is
rolled and clamped onto two circular rigid aluminum disks of diameter $D=52.5$
mm to form a cylinder of length $L_{0}$. The axis of one end is twisted
axially with a Parker stepper motor systems controlled by a computer through
an angle $\theta$ with a rate $\omega=3.67$ degrees/s. Thus, a loading cycle
takes approximately 6 minutes and 30 seconds when $\theta$ is increased to
$\theta_{f}=720^{\circ}$ and then returned back to $0^{\circ}$. The other end
is not allowed to twist, but allowed to move along its axis while mounted on
low-friction linear guides ($\lesssim 0.3N$). Although an axial load can be
prescribed, we perform measurements while allowing the sheet to contract or
expand freely.

###  VI.2 Laser Profilometry

The surface profile of the sheet is obtained by illuminating the sheet with a
640 nm red laser sheet orthogonal to the unperturbed surface. The illuminated
surface is then imaged at an angle with a Pixelink digital camera with a
resolution of $2592\times 2048$ pixels. The bright pixels are used to locate
the surface with a centroid algorithm to within a few microns. This method
yields a single height profile along the length of the sheet while mounted on
the apparatus. To obtain a full scan of the sheet surface, the sheet is
unmounted to relieve elastic stresses. It is then placed on a flat scanner bed
after being flattened somewhat to remove overhangs. Care is taken to minimize
introducing plastic deformations in the process. Because the sheet cannot be
remounted to have the same boundary conditions as when it was dismounted, we
start with a fresh sheet for each scan while measuring as a function of $n$ or
$\theta_{f}$.

Acknowledgements

This work was supported under U.S. National Science Foundation grant
DMR-2005090.

## References

  * [1] M. Ben Amar and Y. Pomeau.  Crumpled paper.  Proceedings: Mathematical, Physical and Engineering Sciences, 453(1959):729–755, 1997.
  * [2] Daniel L. Blair and Arshad Kudrolli.  Geometry of crumpled paper.  Phys. Rev. Lett., 94:166107, Apr 2005.
  * [3] L. Boué, M. Adda-Bedia, A. Boudaoud, D. Cassani, Y. Couder, A. Eddi, and M. Trejo.  Spiral patterns in the packing of flexible structures.  Phys. Rev. Lett., 97:166104, Oct 2006.
  * [4] T. A. Witten.  Stress focusing in elastic sheets.  Rev. Mod. Phys., 79:643–675, Apr 2007.
  * [5] Andrew Croll, Timothy Twohig, and Theresa Elder.  The compressive strength of crumpled matter.  Nature Communications, 10:1502, 04 2019.
  * [6] Kittiwit Matan, Rachel B. Williams, Thomas A. Witten, and Sidney R. Nagel.  Crumpling a thin sheet.  Phys. Rev. Lett., 88:076101, Jan 2002.
  * [7] M. C. Fokker, S. Janbaz, and A. A. Zadpoor.  Crumpling of thin sheets as a basis for creating mechanical metamaterials.  RSC Adv., 9:5174–5188, 2019.
  * [8] Shankar C. Venkataramani.  Buckling sheets open a door to understanding self-organization in soft matter.  Proceedings of the National Academy of Sciences, 116(5):1477–1479, 2019.
  * [9] B.N.J. Persson.  Sliding Friction: Physical Principles and Applications.  NanoScience and Technology. Springer Berlin Heidelberg, 2013.
  * [10] E. Bertin, Jean-Philippe Bouchaud, J-M Drouffe, and C Godreche.  The kovacs effect in model glasses.  Journal of Physics A General Physics, 36, 06 2003.
  * [11] B. Thiria and M. Adda-Bedia.  Relaxation mechanisms in the unfolding of thin sheets.  Phys. Rev. Lett., 107:025506, Jul 2011.
  * [12] Ariel Amir, Yuval Oreg, and Yoseph Imry.  On relaxations and aging of various glasses.  Proceedings of the National Academy of Sciences, 109(6):1850–1855, 2012.
  * [13] Omer Gottesman, Jovana Andrejevic, Chris H. Rycroft, and Shmuel M. Rubinstein.  A state variable for crumpled thin sheets.  Communications Physics, 1(1):70, 2018.
  * [14] Nathan C. Keim, Joseph D. Paulsen, Zorana Zeravcic, Srikanth Sastry, and Sidney R. Nagel.  Memory formation in matter.  Rev. Mod. Phys., 91:035002, Jul 2019.
  * [15] B. A. DiDonna and T. A. Witten.  Anomalous strength of membranes with elastic ridges.  Phys. Rev. Lett., 87:206105, Oct 2001.
  * [16] Christian D. Santangelo.  Extreme mechanics: Self-folding origami.  Annual Review of Condensed Matter Physics, 8(1):165–183, 2017.
  * [17] Menachem Stern, Chukwunonso Arinze, Leron Perez, Stephanie E. Palmer, and Arvind Murugan.  Supervised learning through physical changes in a mechanical system.  Proceedings of the National Academy of Sciences, 117(26):14843–14850, 2020.
  * [18] Menachem Stern, Viraaj Jayaram, and Arvind Murugan.  Shaping the topology of folding pathways in mechanical systems.  Nature Communications, 9:4303, 2018.
  * [19] Chukwunonso Arinze, Menachem Stern, Sidney R. Nagel, and Arvind Murugan.  Learning to self-fold at a bifurcation.  Phys. Rev. E, 107:025001, Feb 2023.
  * [20] Edmund R. Nowak, James B. Knight, Eli Ben-Naim, Heinrich M. Jaeger, and Sidney R. Nagel.  Density fluctuations in vibrated granular materials.  Phys. Rev. E, 57:1971–1982, Feb 1998.
  * [21] David Pine, J Gollub, J Brady, and Alexander Leshansky.  Chaos and threshold for irreversibility in sheared suspensions.  Nature, 438:997–1000, 2005.
  * [22] Ido Regev, Turab Lookman, and Charles Reichhardt.  Onset of irreversibility and chaos in amorphous solids under periodic shear.  Phys. Rev. E, 88:062401, Dec 2013.
  * [23] Dor Shohat, Daniel Hexner, and Yoav Lahini.  Memory from coupled instabilities in unfolded crumpled sheets.  Proceedings of the National Academy of Sciences, 119(28), 2022.
  * [24] Théo Jules, Austin Reid, Karen E. Daniels, Muhittin Mungan, and Frédéric Lechenault.  Delicate memory structure of origami switches.  Phys. Rev. Res., 4:013128, Feb 2022.
  * [25] Yutong Xia, Narayanan Kidambi, Evgueni Filipov, and K. W. Wang.  Deployment Dynamics of Miura Origami Sheets.  Journal of Computational and Nonlinear Dynamics, 17(7):071005, 04 2022.
  * [26] Francis Dunne, Kjell Westra, Caleb Mattox, and Jacob Leachman.  Fatigue life characterization of hand folded and vacuum formed kresling origami bellows at 77k.  IOP Conference Series: Materials Science and Engineering, 1241(1):012014, may 2022.
  * [27] T. Tallinen, J. A. Åström, and J. Timonen.  The effect of plasticity in crumpling of thin sheets.  Nature materials, 8 1:25–9, 2009.
  * [28] Mehdi Habibi, Mokhtar Adda-Bedia, and Daniel Bonn.  Effect of the material properties on the crumpling of a thin sheet.  Soft Matter, 13:4029–4034, 2017.
  * [29] Handika Dany Rahmayanti, Fisca Dian Utami, and Mikrajuddin Abdullah.  Physics model for wringing of wet cloth.  European Journal of Physics, 37(6):065806, sep 2016.
  * [30] Pan Dong, Mengfei He, Nathan C. Keim, and Joseph D. Paulsen.  Twisting a cylindrical sheet makes it a tunable locking material.  Phys. Rev. Lett., 131:148201, Oct 2023.
  * [31] Giles W. Hunt and Ichiro Ario.  Twist buckling and the foldable cylinder: an exercise in origami.  International Journal of Non-Linear Mechanics, 40(6):833–843, 2005\.
  * [32] Li-Min Wang, Sun-Ting Tsai, Chih-yu Lee, Pai-Yi Hsiao, Jia-Wei Deng, Hung-Chieh Fan Chiang, Yicheng Fei, and Tzay-Ming Hong.  Crumpling-origami transition for twisting cylindrical shells.  Phys. Rev. E, 101:053001, May 2020.
  * [33] L. H. Donnell.  Stability of thin-walled tubes under torsion.  Technical report, National Advisory Committee for Aeronautics - Report No. 479, 1933.
  * [34] Gerard Giménez-Ribes, Melika Motaghian, Erik van der Linden, and Mehdi Habibi.  Crumpled structures as robust disordered mechanical metamaterials.  Materials & Design, 232:112159, 2023.
  * [35] Julien Chopin and Arshad Kudrolli.  Tensional twist-folding of sheets into multilayered scrolled yarns.  Science Advances, 8(14):eabi8818, 2022.
  * [36] T. Jules, F. Lechenault, and M. Adda-Bedia.  Plasticity and aging of folded elastic sheets.  Phys. Rev. E, 102:033005, Sep 2020.
  * [37] Dor Shohat, Yaniv Friedman, and Yoav Lahini.  Logarithmic aging via instability cascades in disordered systems.  Nature Physics, 19(12):1890–1895, 2023.
  * [38] Joseph D. Paulsen, Nathan C. Keim, and Sidney R. Nagel.  Multiple transient memories in experiments on sheared non-brownian suspensions.  Phys. Rev. Lett., 113:068301, Aug 2014.
  * [39] Anne Dominique Cambou and Narayanan Menon.  Three-dimensional structure of a sheet crumpled into a ball.  Proceedings of the National Academy of Sciences, 108(36):14741–14745, 2011.
  * [40] Eric Sultan and Arezki Boudaoud.  Statistics of crumpled paper.  Phys. Rev. Lett., 96:136103, Apr 2006.
  * [41] Alexander S. Balankin and Leonardo Flores-Cano.  Edwards’s statistical mechanics of crumpling networks in crushed self-avoiding sheets with finite bending rigidity.  Phys. Rev. E, 91:032109, Mar 2015.
  * [42] Jovana Andrejevic, Lisa M. Lee, Shmuel M. Rubinstein, and Chris H. Rycroft.  A model for the fragmentation kinetics of crumpled thin sheets.  Nature Communications, 12(1), Mar 2021.
  * [43] Arezki Boudaoud, Pedro Patrício, Yves Couder, and Martine Ben Amar.  Dynamics of singularities in a constrained elastic plate.  Nature, 407:718–20, 11 2000.
  * [44] S. Kodama and N. Yamaki.  Postbuckling behavior of pressurized circular cylindrical shells under torsion—ii. theory.  International Journal of Non-Linear Mechanics, 16(3):355–370, 1981\.
  * [45] Wesley Wong and Sergio Pellegrino.  Wrinkled membranes, part ii: Analytical models.  Journal of Mechanics of Materials and Structures, 1:27–61, 2006\.
  * [46] John W. Harris and Horst Stöcker.  Handbook of Mathematics and Computational Science.  Springer, New York, 1998.
  * [47] Hongxun Song, Weixing Wang, Fengping Wang, Linchun Wu, and Zhiwei Wang.  Pavement crack detection by ridge detection on fractional calculus and dual-thresholds.  International Journal of Multimedia and Ubiquitous Engineering, 10:19–30, 2015.
  * [48] S. Deboeuf, E. Katzav, A. Boudaoud, D. Bonn, and M. Adda-Bedia.  Comparative study of crumpling and folding of thin sheets.  Phys. Rev. Lett., 110:104301, Mar 2013.

Supplementary Information

##  I Wrinkling instability and development of folds

According to Donnell’s stability analysis of thin-walled elastic tubes under
torsion [33, 44], the number of wrinkles,

$N_{w}=N_{0}\,\bigg{[}\frac{1}{\sqrt{1-\mu^{2}}}\frac{L_{0}^{2}h}{D^{3}}\bigg{]}^{-1/4},$
(S1)

where $N_{0}=2.62$, and $\mu$ is the Poisson ratio, which is approximately
$0.5$ in the case of Mylar. Fitting the data, we find $N_{0}=1.76$,
corresponding to a systematically lower number of wrinkles. The difference is
likely due to the fact that a cylindrical rolled sheet, rather than a
cylindrical shell, is twisted in the experiments. Based on work reported in
the case of sheared planar sheets [45], stress relaxation near the free edges
can lead to the angles that the wrinkles make with the direction of shear to
change, lowering the number of wrinkles. This fact also leads to disorder in
the formation of the slanted folds in the case of higher aspect ratio
cylinders, as opposed to those in thin cylindrical shells, where they are
ordered for a wide range of aspect ratios even as applied twist is increased
[33].

Figure S1: The number of wrinkles $N_{w}$ observed as a function of
$L_{0}/D_{0}$ and compared with Eq. (S1).

##  II Hyperbolic Hyperboloid

If the sheet was replaced by a set of infinitely thin inextensible filaments
of length $L_{0}$ arranged cylindrically with diameter $D_{0}$, they would
give rise to a hyperbolic hyperboloid when twisted up to $\theta<180^{\circ}$
[46]. Then, the distance between the ends as a function of twist $\theta$ is
given by

$L/L_{0}=\sqrt{1-\Big{(}\frac{D_{0}}{L_{0}}\Big{)}^{2}\sin^{2}(\theta/2)}\,.$
(S2)

Plotting $L/L_{0}$ over the range $0\leq\theta\leq\pi/2$ in Fig. S2, we find
that it is systematically higher than the end to end distance measured in the
case of the sheet.

As shown in Fig. 1, the deformation is initially localized in the central
portion of the structure. We can model the deformation before self-contact as
being localized to a central portion of the cylinder $\phi L_{0}$, with
essentially straight sections near the clamped edges. Then, we have

$L/L_{0}=(1-\phi)+\phi\sqrt{1-\Big{(}\frac{D_{0}}{\phi
L_{0}}\Big{)}^{2}\sin^{2}(\theta/2)}\,,$ (S3)

where $\phi=1$ corresponds to Eq. (S2). By fitting, we find $\phi\approx
0.29$, which corresponds to the twist being effectively localized to the
central 30% of the cylinder which roughly consistent with the image of the
deformation shown in Fig. 1A corresponding to $\theta=90^{\circ}$.

Figure S2: The measured end to end distance between clamped edges $L$ as a
function of twist angle $\theta$. Comparison with Eq. (S3) with $\phi=1$ and
$\phi=0.29$ are also shown.

##  III Hysteresis Characteristics

Figure S3 shows a plot of the compressed length $L_{c}$ when the sheet is
twisted by $\theta$, and then the length $L_{n}$ after untwisting. $L_{c}$
decreases rapidly after the first cycle, and then increases slowy with $n$
(see Fig. S3C). $L_{n}$ decreases relatively more rapidly with increasing $n$
as $\theta_{f}$ is increased (see Fig. S3E).

Figure S3: (A) $L/L_{0}$ as a function of $\theta$ in the case of three
different $\theta_{f}$ corresponding to $n=1$. While the path followed during
twist is the same, the return path during the untwisting varies leading to
growing differences with $\theta_{f}$. (B) The area enclosed by the hysteresis
curve as a function of $n$ for each case of $\theta_{f}$. The hysteresis
decreases slowly with $n$. The noise floor is approximately $10^{-1}$ for
these measurements, similar to $A$ measured in the case of
$\theta_{f}=180^{\circ}$. The compressed length $L_{c}$ versus $n$ (C), and
the corresponding strain $\varepsilon$ versus $n$ (D). (E) The end to end
length after each cycle $L_{n}$ observed as a function of loading cycle $n$.
(F) The mean curvature distribution $P(|H|)$ becomes broader with increasing
$\theta_{f}$ as the sheet is increasingly deformed.

##  IV Crease Detection

Figure S4: (A) The region of interest (ROI) considered for the statistical
analysis. (B) The creases longer than $2$ mm, detected assuming global
thresholds $Th=0.010$, $Th=0.012$, and $Th=0.014$. The blue (red) lines
correspond to valleys (ridges).

###  IV.1 Pixel Dimensions and Image Resizing

We first examine the total number of rows and columns in the matrix Z with the
scan data. If the number of rows exceeds the number of columns, we increase
the number of columns, transforming the pixel dimensions into a square shape
aligned with the initial pixel dimension along the rows. The same process is
applied in reverse if the number of columns is greater than the number of
rows. For example, if one has a matrix Z with order $R\times C$ and $R>C$, and
the pixel dimension is $r\times c$ along the row and columns respectively,
then we make the pixel dimension square by changing the number of columns to:

$N_{\text{new}}=\frac{c\times C}{r}$ (S4)

and vice versa. In this case, the individual pixel dimension becomes $r\times
r$. We resize the image as needed. In our case, the image is resized by a
factor of 0.5.

###  IV.2 Valley Detection

This is the main part of the ridge detection process. We define a two-
dimensional array having a size equal to the size of the $Z$ matrix with all
elements initialized to zero. Our goal is to set elements to 1 at positions
where there are creases. Creases are extracted by applying two levels of
thresholds: firstly, in the local level, and then in the global level [47].

Figure S5: (A)The $7\times 7$ matrix illustrating the selection of the
neighbors along eight different directions to obtain weighted means
($X_{0}^{\circ},X_{45}^{\circ},X_{90}^{\circ},X_{135}^{\circ},X_{180}^{\circ},X_{225}^{\circ},X_{270},X_{315}^{\circ}$).
(B) The resulting matrix A containing the 8 weighted means from the nearest 8
neighbors of the target pixel is calculated using the weighted mean formula
for three consecutive pixels along a given direction.

  * •

The weighted mean along eight directions
($0^{\circ},45^{\circ},90^{\circ},135^{\circ},180^{\circ},225^{\circ},270^{\circ},315^{\circ}$),
as shown in Figure S5, are calculated from the three consecutive matrix
elements along that direction:

$X_{0}=\frac{Z(i+1,j)+v\times Z(i+2,j)+\frac{v(v+1)}{2}\times
Z(i+3,j)}{W_{\text{factor}}},$ (S5) $X_{45}=\frac{Z(i+1,j-1)+v\times
Z(i+2,j-2)+\frac{v(v+1)}{2}\times Z(i+3,j-3)}{W_{\text{factor}}},$ (S6)
$X_{90}=\frac{Z(i,j-1)+v\times Z(i,j-2)+\frac{v(v+1)}{2}\times
Z(i,j-3)}{W_{\text{factor}}},$ (S7) $X_{135}=\frac{Z(i-1,j-1)+v\times
Z(i-2,j-2)+\frac{v(v+1)}{2}\times Z(i-3,j-3)}{W_{\text{factor}}},$ (S8)

… and so on.

Where the weighted factor $W_{\text{factor}}=1+v+\frac{v(v+1)}{2}$ is used
with a suitable value of $v$. In our case, we have set it to be 20.

  * •

From the $7\times 7$ matrix, we create a $3\times 3$ matrix (denoted as $A$)
with the central pixel value as the original height and other neighboring
elements with a weighted mean along corresponding directions, as shown in Fig.
S5.

  * •

Define the local threshold value ($Th_{\text{local}}$) by the following
formula:

$Th_{\text{local}}=\frac{\text{mean}(A)-Z(i,j)}{2}.$ (S9)

  * •

From the $3\times 3$ matrix $A$, we construct a new $1\times 4$ matrix,
denoted as $G$. To achieve this, we introduce a local threshold value as a
reference. In the process, we define pairs of mean values in opposite corners
as $X_{s\times 45^{\circ}}$ and $X_{s\times 45^{\circ}+180^{\circ}}$ for $s=0$
to $s=4$. The matrix $G$ is defined as
$G=[g_{s}]_{(s=1,2,3,4)}=[g_{1},g_{2},g_{3},g_{4}]$, where $g_{s}$ is
calculated as:

$g_{s}=\frac{(X_{s\times 45^{\circ}+180^{\circ}}-Z(i,j))+(X_{s\times
45^{\circ}}-Z(i,j))}{2},$ (S10)

if $\lvert(X_{s\times 45^{\circ}+180^{\circ}}-Z(i,j))\rvert>Th_{\text{local}}$
or $\lvert(X_{s\times 45^{\circ}}-Z(i,j))\rvert>Th_{\text{local}}$. Otherwise,
$g_{s}=0$.

  * •

Now we implement the global thresholding step to decide if the central pixel
is a valley or not by comparing the maximum value of matrix $G$ with the
global threshold value Th. If $max(G)>Th$, then Valley $(i,j)=1$. Otherwise,
Valley$(i,j)=0$. In our case, we have used $Th$: 0.010, 0.012, and 0.014, and
tested to check their effect of crease detection and effect on statistical
values of interest.

We invert the surface and apply the same algorithm to then extract the ridge
network. An example of the ridge and valley detected using various thresholds
is shown in Fig. S4.

###  IV.3 Crease Length Calculation

We count the number of occupied pixels $N_{\text{occupied}}$ and apply the
following formula to estimate the crease length:

$\ell=N_{\text{occupied}}\times\frac{\sqrt{2}+1}{2}\times r$ (S11)

Where $\ell$ is the ridge length, $r$ is the pixel dimension. While writing
Equation S11, we have considered that 50% of the non-zero pixels are connected
diagonally, and the remaining 50% linearly.

Figure S6: The effect of the global threshold $Th$ on measured evolution
parameters of the crease network.

As shown in Fig. S6, while the choice of threshold varies the values somewhat,
the trends as quantified by $\ell_{1}$ and $\ell_{2}$ remain robust.

##  V Spatial distribution of curvature

We plot the average $|H|$ as a function of distance from one clamped end to
the other in Fig. S7. As can be seen from Fig. S7A,B, $H$ is lower near in the
clamped edges because the sheet is forced to maintain the curvature of the end
disks on which it is mounted. When the sheet is repeatedly twisted, $H$
increase systematically everywhere and thus the variation across $x$ persists
(see Fig. S7C,D). Accordingly, we analyze the distribution of the measured $H$
over a $145$ mm by $115$ mm centrally cropped area of the sheet to avoid
direct boundary effects.

Figure S7:  Mean curvature of the crumpled sheet as a function of distance
along the sheet’s length for various $\theta_{f}$ after cycle $n=1$ (A) and
$n=100$ (B), respectively. (C,D) Mean curvature of the cross section as a
function of distance along the sheet’s length for increasing $n$ for
$\theta_{f}=180^{\circ}$ and $720^{\circ}$, respectively. The sheet is more
crumpled in the central regions between the clamps where sheet forms a tight
bundle when twisted.

##  VI Folding model of creases

Figure S8: (A) Folds become increasingly disordered as the sheet is folded
into smaller squares. (B) The detected ridges and valleys are superimposed on
the scans. Note that refolded creases become sharper and are detected more
fully as their curvature exceeds the threshold used to detect a crease. Figure
S9: The absolute mean curvature of the sheet increases with the compaction in
both flat folding and twist crumpling cases.

We performed complementary studies of crease growth using flat folding methods
[48]. First, we consider a square infinitesimally thin sheet with side
$L_{0}$, and calculate the total crease length upon flat folding the sheet
into smaller squares with sides $L_{0}/m$, with $m=1,2,4,..$. This flat folded
sheet can fit inside a piston corresponding compression
$\varepsilon_{s}=1-1/m$, and used to plot calculated $\ell$ in Fig. 3B.

Then, to understand the impact of sheet thickness and the crease detection
methods, we also performed the same square folding with $L_{0}\times L_{0}$
Mylar sheets for $m=1,2$, and 4. The scans are shown in Fig. S8, and show the
growing disorder and nonuniformity of the curvature along the creases even
though they were folded by an ordered sequence of folds. Plotting the average
mean curvature of the sheet $\langle|H|\rangle$ in Fig. S9, we observe that it
increases to a similar degree as the sheet is progressively folded and thus
compressed to contain within a piston which has been strained through
$\varepsilon$.

Next, we use the same threshold criteria used to detect the ridges as in the
twist experiments. Examining the detected ridge and valleys superposed on the
scans shown in Fig. S8(B), we observe that primary folds which was missed
initially was detected in their entirety with subsequent refolding. However,
the new folds corresponding to $m=2,3$ have overall lower curvature and were
detected to a lower degree.

Because of the thickness of the sheet, the sheet becomes increasingly more
difficult to fold, with increasing number of folds. Further, one observes the
appearance of curvature with opposite signs adjacent to the square folds as
the sheet is forced back to a planar configuration. While overall smaller in
magnitude, they can be noted to be similar in magnitude to the creases created
further along the folding sequence. One can further observe the appearance of
disorder and creases at the intersection of the creases created while folding,
earlier in the sequence.

Thus, while not all the creases expected corresponding to an ideal square
folding are detected as they elastically relax below the criteria set for
detecting ridges once unfolded to perform the scans, additional creases are
created due to the unfolding process of a sheet with finite thickness as
illustrated in Fig. S8. As seen from the comparison for the total ridge length
$\ell$ shown in Fig. 3B, these counter trends are observed to lead to an
overall higher total crease length in the experimentally obtained case
compared to the ideal square folds.

##  VII Evolution of creases with training cycles

Figure S10: The total number of creases counted along the length of the
cylinder evolves logarithmically with the loading cycles obtained from the
scanned surfaces (A), and using the cross-sections measured after each cycle
(B).

##  VIII Bifurcation and merging of folds

Figure S11: (A) The evolution of a cross-section with $n$ for
$\theta_{f}=360^{\circ}$. (B) A zoomed in view of the shaded region in Fig.
S11(A) showing the relative motion of the creases with increasing $n$,
including creation of peak and valley pairs (black circles) and destruction of
creases (black rectangle).

##  IX Fatigue with cycle number

Figure S12: $\xi$ tracking over the first 2000 cycles for two different cases,
$\theta_{f}=360^{\circ}$(A) and $720^{\circ}$(B). (C) An image of the sheet
after $n=2000$ twisting cycles shows tears along some of the creases
($\theta_{f}=720^{\circ}$).

Fig. S12 shows how the different loading condition affect the configuration of
the system even after thousands of repeated cycles. In the case of
$\theta_{f}=360^{\circ}$, we observe that $\xi$ decreases to zero and stays
that way after few hundred cycles even as $n$ is increased to 2000. Whereas,
in the case of $\theta_{f}=720^{\circ}$, we observe that $\xi$ continues to
large spikes indicating large rearrangements over the entire range of $n$. The
sheets is observed to show cracks or tears along some of the repeatedly folded
creases at approximately $n=1800$. Thus, no convergence of structure is
observed even as fatigue sets in over the course of the applied cycles.

##  X Effect of inter-cycle perturbations

In order to test the effect of introducing random perturbations between
training cycles, we performed experiments where we unmounted the sheet after
each cycle and then remounted it after flipping the cylindrical sheet surface
inside out. This can lead to a different folding sequence as the sheet
compresses since the applied constraints progress differently in detail. As
shown in Fig. S13, we find that the damage grows logarithmically as well, but
with a significantly higher growth rate $\ell_{2}/L_{0}=11.2$, as opposed to
$4.8$, when the sheet is simply twisted repeatedly without imposing any inter-
cycle perturbations.

Figure S13: $\ell/L_{0}$ for $\theta_{f}=360^{\circ}$ with two distinct
crumpling protocols: one involving the flipping the sheet surface after each
iteration and the other keeping the sheet clamped without unmounting
throughout the experiments. The graph exhibits a higher rate of increase, with
$\ell_{2}/L_{0}=11.2$, in the former case, whereas the rate is $4.8$ in the
latter case.

##  XI Supplementary Movies

  1. 1.

Supplementary Movie S1: A movie of a sheet as it is twisted twice about its
axis through $\theta_{f}=720^{\circ}$. Crumples develop as the sheet buckles
and folds with applied twist and gives rise to plastic deformations.

  2. 2.

Supplementary Movie S2: Snapshots of the sheet after the application of a
loading cycle over $100$ cycles corresponding to $\theta_{f}=720^{\circ}$. The
creases are observed to largely repeat but a few new creases and sheet
reconfigurations can be also observed.


Haoyu Wang1, Siyuan Wang1, Wei-Qiang Zhang1* Corresponding author, Hongbin
Suo2, Yulong Wan2††thanks: This work was supported by the National Natural
Science Foundation of China under Grant No. 62276153.

# Task-Agnostic Structured Pruning of Speech Representation Models

###### Abstract

Self-supervised pre-trained models such as Wav2vec2, Hubert, and WavLM have
been shown to significantly improve many speech tasks. However, their large
memory and strong computational requirements hinder their industrial
applicability. Structured pruning is a hardware-friendly model compression
technique but usually results in a larger loss of accuracy. In this paper, we
propose a fine-grained attention head pruning method to compensate for the
performance degradation. In addition, we also introduce the straight through
estimator into the $L_{0}$ regularization to further accelerate the pruned
model. Experiments on the SUPERB benchmark show that our model can achieve
comparable performance to the dense model in multiple tasks and outperforms
the Wav2vec 2.0 base model on average, with 72% fewer parameters and 2 times
faster inference speed.

Index Terms: Model pruning, knowledge distillation, model compression,
representation learning

##  1 Introduction

Recently, self-supervised pre-training has become one of the most attractive
topics in the speech domain [1, 2]. With this method, a large amount of
unlabeled data can be used to train a deep model to extract high-level
representations from raw audio, which can bring significant improvement to
many downstream tasks.

While pre-trained models provide a tremendous performance improvement, they
also require large amount of memory and computing power. Large self-supervised
pre-trained speech models such as Wav2vec2 [3], Hubert [4], and WavLM [5]
typically have hundreds of millions of parameters, making them unsuitable for
use on consumer products such as laptops and smartphones. This is an obstacle
to the application of these models in many real-world scenarios. As a result,
model compression has become a major concern for these large self-supervised
models.

Knowledge distillation usually uses a teacher model to guide a smaller student
model, and the structure of the student model must be carefully designed to
achieve better performance. DistilHubert [6] distills a 12-layer Hubert-based
model to obtain a 2-layer student model and significantly reduces the model
size. FitHubert [7], which is inspired by FitNets [8], designs a thin but deep
student network to provide better representation ability.

Model pruning attempts to discard the unimportant weights and obtain a
subnetwork from the pre-trained model. In unstructured pruning, these
discarded weights are randomly distributed in the matrices; in structured
pruning, network units such as attention heads or feed-forward layers are
removed entirely. Structurally pruned models do not require specially designed
hardware for acceleration, which may be more appropriate for consumer devices.
LightHubert treats model pruning as a neural architecture search problem and
significantly reduces the performance degradation, but the search process
still requires some time-consuming manual selections [9]. Peng et al. propose
a more flexible method by applying the $L_{0}$-regularization-based pruning
method [10] to the Wav2vec 2.0 model, but their method is task-specific and
comes at some additional cost when applied to downstream tasks [11].

We attempt to use a similar $L_{0}$-regularization-based method to obtain a
task-agnostic compressed model. However, learning the pruning masks using
$L_{0}$ regularization on unsupervised pre-training tasks such as contrastive
predictive coding [12] requires large computational resources. The combination
of distillation and pruning is a promising solution [13, 14]. The
representation provided by the pre-trained model not only reduces the training
effort of the downstream models, but also provides task-independent
information for model pruning.

Compared to existing unstructured pruning methods of the pre-trained speech
models [15, 16], structure pruning usually suffers from a larger performance
degradation [17]. The crux of this problem is that using structure rather than
individual weights as the basic unit of pruning reduces the degree of freedom,
resulting in the removal of some important weights. To compensate for the
performance degradation, we introduce a fine-grained attention head pruning
method that prunes each attention head separately. To promote the pruning of
coarse-grained structures and further speed up the pruned model, we also
introduce the straight through estimator (STE) [18] into the mutil-scale
structured pruning method [13] based on $L_{0}$ regularization.

Experiments on the SUPERB benchmark show the generalization ability of the
proposed model on different downstream tasks. With the help of the pre-trained
teacher, the proposed model is task-agnostic and can be directly fine-tuned to
many downstream tasks. Further contrast experiments demonstrate the
effectiveness of fine-grained attention head pruning and STE. Our model
outperforms the distilled baselines, and achieves comparable results to the
teacher model on multiple tasks, with 72% fewer parameters and 2 times faster
in speed.

##  2 Backgrounds

###  2.1 Pre-trained Speech Representation Models

Our experiment is mainly performed on WavLM [5], but the method can be easily
extended to Wav2vec 2.0 [3], data2vec [19], Hubert [4], and other models with
similar transformer-based structures.

WavLM is a set of state-of-the-art self-supervised pre-trained models. During
pre-training, offline clustered units are used as the training target and the
models learn to represent the continuous inputs by some discrete hidden units.
WavLM also introduces masked speech denoising and gated relative position bias
to improve the performance.

###  2.2 Pruning Based on the $L_{0}$ Regularization

Pruning based on $L_{0}$ regularization is one of the mask learning methods.
In some pruning methods, parameters are discarded according to some
artificially set criteria, such as the magnitude of weights or gradients. On
the other hand, mask learning methods tend to consider pruning as an
optimization problem [10]. As the name implies, $L_{0}$-regularization-based
pruning adds a mask to the parameters (or parameter groups) and uses the
$L_{0}$ norm of these pruning masks as a regularization term of the loss
function. For example, in our experiments, the training objective is:

$\displaystyle\mathcal{R}(\theta,\pi)=E_{z\sim{q(\pi)}}[\frac{1}{N}\sum^{N}_{i=1}\mathcal{L}(f_{s}(x_{i},\widetilde{\theta}),f_{t}(x_{i}))+\lambda||\widetilde{\theta}||_{0}]\text{,}$
(1)

where $f_{s}$ and $f_{t}$ are the student and teacher models for knowledge
distillation, $x_{i}$ is the $i$th input data, $\theta$ is the parameter set
of the student model, $z\in\\{0,1\\}$ is the pruning mask set,
$\widetilde{\theta}=\theta\odot z$ is the parameter set after masking. The
discrete random variable $z$ follows a Bernoulli distribution $q(\pi)$.

However, this objective function cannot be optimized by gradient descent
methods because the process of sampling $z$ for $q(\pi)$ is not
differentiable. Louizos et al. introduce a reparameterization trick to deal
with this problem [10]. After the reparameterization, z becomes a continuous
variable, determined by a learnable parameter $\alpha$ and an additional
random variable $u$ that ``collects" the randomness from $z$. Formally
speaking, $z$ is computed by:

$\displaystyle\begin{split}u&\sim U(0,1)\text{,
}s=\text{sigmoid}(\frac{1}{\beta}\text{log}(\frac{u}{1-u})+\text{log}\alpha)\\\
\bar{s}&={s(\zeta-\gamma)+\gamma}\text{,
}{z}=\text{hardtanh}({\bar{s}})\text{,}\end{split}$ (2)

where $u$ is sampled from a uniform distribution $U(0,1)$, $\zeta=1.1$,
$\gamma=-0.1$ are 2 constants to scale $s$ to a larger interval and make sure
$z$ can be exactly 0 or 1. $\beta$ controls the temperature, and $\alpha$ is
the learnable parameter.

(a)  (b)

Figure 1: (a) the possibility distribution of $z$ and $\bar{s}$. (b) $z$ and
$\bar{s}$ as a function of log$\alpha$, averaged on 500 samples. $z$ can be
exactly 0 or 1 or any value in between. In the shadow region, $\partial
z/\partial\bar{s}=0$.

Figure 1a shows the probability distribution of $z$ and $\bar{s}$, while
figure 1b shows their values as functions of log$\alpha$. We can see that the
reparameterization trick turns the discrete masks $z$ into continuous
variables while still allowing them to be exactly 0 or 1.

###  2.3 Multi-scale Structured Pruning

The $L_{0}$ regularization does not limit the grain of the pruning. If $z$
masks some structure, $L_{0}$ regularization can be used for structured
pruning. The grain can be as large as an entire layer or as small as a certain
dimension of a weight matrix. Recently, Xia et al. introduce a multi-scale
pruning method that removes fine-grained and coarse-grained structures in
parallel to promote the removal of large structures and achieve further
speedup [13]. We introduced this method to increase the possibility of
removing coarse-grained structures to compensate for the potential negative
effects of our fine-grained attention head pruning method on the inference
speed of the model.

##  3 Methods

###  3.1 Fine-grained Attention Head Pruning

In previous works [11, 13], the attention heads are used as the smallest units
for pruning. This may reduce the degree of freedom of pruning and lead to more
performance degradation. To make structure pruning more flexible, we propose a
fine-grained attention method that separately prunes each dimension of
matrices in the attention layer based on the multi-scale structured pruning
method of Xia et.al [13]. Formally speaking, a transformer block is masked as
follows:

$\displaystyle\begin{split}f_{\rm{MHA}}(X)&=z_{\rm{MHA}}\cdot\text{concat}(f_{\rm{ATT}}(X))\\\
f_{\rm{ATT}}(X)&=S_{c}\cdot(XW_{V}^{i})\cdot\text{diag}(z^{i}_{vo})\\\
S_{c}&=\text{softmax}((XW_{Q}^{i})\cdot\text{diag}(z^{i}_{qk})\cdot(XW_{K}^{i})^{T})\\\
f_{\rm{FFN}}(X)&=z_{\rm{FFN}}\cdot\text{gelu}(XW_{\rm{U}})\cdot\text{diag}(z_{\rm{int}})\cdot
W_{\rm{D}}\text{,}\end{split}$ (3)

where $X$ is the input data, $W_{Q}^{i}$, $W_{K}^{i}$, $W_{V}^{i}$, $W_{O}$ is
the query, key, value and output matrices, respectively. $z_{\rm{MHA}}$,
$z^{i}_{qk}$ , $z^{i}_{vo}$, $z_{\rm{FFN}}$, $z_{\rm{int}}$ denote the pruning
mask for multi-head attention layers, attention matrices, feed-forward layers,
and intermediate dimensions. We omit the scale factors in $f_{\rm{ATT}}(X)$
for clarity, and please note that $W_{O}$ should also be pruned according to
$z_{vo}^{i}$. For $W_{Q},W_{V}\in\mathbb{R}^{d_{\rm{hidden}}\times
d_{\rm{head}}}$, $z^{i}_{qk}$ and $z^{i}_{vo}$ will have $d_{\rm{head}}$
variables.

###  3.2 Optimizing Pruning Masks with STE

Although the reparameterization trick makes $z$ differentiable, the
introduction of hardtanh in Eq. 2 creates a new obstacle to optimization. As
shown in Figure 1b, when $\text{log}\alpha$ takes a value in the shaded
region, the presence of hardtanh makes $\partial z/\partial s=0$, and the
learnable parameter $\alpha$ cannot be updated. That is to say, the model
decides to keep a structure when $z$ is 1, but it cannot evaluate that
decision.

This problem becomes more obvious for multi-scale structured pruning. Figure
3a shows that the mean value of $z_{\text{FFN}}$ does not change during
training, which makes multi-scale pruning ineffective. The reason may be that
in the early stages of training, pruning the entire FFN layer can lead to a
huge performance degradation, so $\alpha$ may be optimized to a large positive
value, and difficult to update in the remaining training steps.

The failure to cut the coarse-scale structures will cause the sparse weight of
the pruning model to be too dispersed, resulting in lower acceleration ratio.
To address this problem, We apply the straight through estimator [18] to make
sure that the gradient can pass through the hardtanh function in Eq. 2. Since
the gradients from STE are not the gradients for the loss function, optimizing
in this direction may not lead to the most accurate student and may cause
instability near some local minima [20]. For the stability of training, we
define the gradient of STE such that:

$\displaystyle\frac{\partial\mathcal{L}}{\partial\bar{s}}=\begin{cases}1\text{,
if }\frac{\partial\mathcal{L}}{\partial z}>=1\text{; }\\\ -1\text{, if
}\frac{\partial\mathcal{L}}{\partial z}<-1\text{; }\\\
\frac{\partial\mathcal{L}}{\partial z}\text{, otherwise. }\end{cases}$ (4)

###  3.3 Training Objective

Hidden states of different layers contain different types of information [6,
21]. Therefore, we follow Xia et al. [13] to use learnable multi-task
knowledge distillation to learn the representation of different layers. We
also follow Wang et al. to change the 2nd term on the r.h.s of eq. 1 into a
Lagrangian term to better control the sparsity [22]. Our training objective is
as follows:

$\displaystyle\mathcal{L}=\frac{1}{N}\sum_{i=0}^{N}\sum_{(j,k)\in
D}\mathcal{L}_{\rm{MSE}}(h_{i}^{j},\hat{h}_{i}^{k})+\lambda_{1}(\hat{p}-p)+\lambda_{2}(\hat{p}-p)^{2}\text{,}$
(5)

where $\hat{p}$ is the approximate model sparsity, $p$ is the target sparsity.
$\lambda_{1}$ and $\lambda_{2}$ are learnable parameters for the Lagrangian
regularization. $D$ is the teacher-student layer pairing relation learned
during training [13], for sample $i$, $h_{i}^{j}$ and $\hat{h}_{i}^{k}$ are
the output of layer $j/k$ of the student and teacher models, respectively.

##  4 Experiments

###  4.1 SUPERB

SUPERB (Speech processing Universal PERformance Benchmark) is a benchmark for
evaluating the performance of speech pre-training models [23]. SUPERB provides
10 predefined speech tasks from different perspectives where the pre-trained
models are used as upstream feature extractors. These tasks include phoneme
recognition (PR), automatic speech recognition (ASR), keyword spotting (KS),
query-by-example spoken term detection (QbE), speaker identification (SID),
automatic speaker verification (SV), speaker diarization (SD), intent
classification (IC), slot filling (SF), and emotion recognition (ER).

###  4.2 Pruning setup

Model. Our model is initialized from the WavLM base model, which consists of a
7-layer CNN feature extractor and a 12-layer transformer encoder. For the
matrices in Eq. 3, $W_{Q}^{i},W_{K}^{i},W_{V}^{i}\in\mathbb{R}^{768\times
64}$, $W_{O}\in\mathbb{R}^{768\times 768}$, $W_{U}\in\mathbb{R}^{768\times
3072}$, and $W_{D}\in\mathbb{R}^{3072\times 768}$. For each transformer block,
we have 12 attention heads, leading to $12*64=768$ elements in $z_{qk}$ and
$z_{vo}$. We also have 3072 elements in $z_{\rm{int}}$ for each dimension in
the FFN layer, and 1 element in $z_{\rm{MHA}}$ and $z_{\rm{FNN}}$ to mask the
entire layer. The target pruning sparsity is set to 80%. The teacher model of
knowledge distillation is also the WavLM base model.

Data. We use the 960 hours Librispeech [24] corpus for pruning. For SUPERB
tasks, we use the dataset according to the official
guidelines111https://github.com/s3prl/s3prl/.

Pruning. Pruning is performed on an RTX 3090 GPU for 200k steps and takes
about 36 hours. Our training hyperparameters are chosen according to
DistilHuBERT [6] and Xia et al. [13]. The learning rate increases linearly to
2.0e-4 in the first 7% steps and decreases linearly to 0 in the remaining
steps, and the target sparsity increases linearly to 80% in the first 7% steps
and remains constant for the rest.

Table 1: Results on SUPERB of the proposed model, and other baselines. The performances are evaluated by Phoneme Error Rate (PER%), Accuracy (Acc%), Word Error Rate (WER%), Maximum Term Weighted Value(MTWV), F1 Score (F1%), Concept Error Rate (CER%), Equal Error Rate (EER%), and Diarization Error Rate (DER%). DistilWavLM is our reproduction of DistilHubert with the teacher changed to WavLM base; FAHP is the abbreviation for the proposed Fine-grained Attention Head Pruning method. Method | KS | IC | PR | ASR | ER | QbE | SF | SID | SV | SD | $\text{Superb}_{s}$↑
---|---|---|---|---|---|---|---|---|---|---|---
Acc↑ | Acc↑ | PER↓ | WER↓ | Acc↑ | MTWV↑ | F1↑/CER↓ | Acc↑ | EER↓ | DER↓
Baselines
wav2vec [25] | 95.59 | 84.92 | 31.58 | 15.86 | 59.79 | 4.85 | 76.37/43.71 | 56.56 | 7.99 | 9.90 | 491.59
w2v2 Base | 96.23 | 92.35 | 5.74 | 6.43 | 63.43 | 2.33 | 88.3/24.77 | 75.18 | 6.02 | 6.08 | 735.00
HuBERT Base | 96.30 | 98.34 | 5.41 | 6.42 | 64.92 | 7.36 | 88.53/25.2 | 81.42 | 5.11 | 5.88 | 837.63
WavLM Base | 96.79 | 98.63 | 4.84 | 6.21 | 65.94 | 8.70 | 89.38/22.86 | 84.51 | 4.69 | 4.55 | 895.99
Distilled Models
DistilHuBERT | 95.98 | 94.99 | 16.27 | 13.37 | 63.02 | 5.11 | 82.57/35.39 | 73.54 | 8.55 | 6.19 | 647.88
DistilWavLM | 96.40 | 96.39 | 14.18 | 13.24 | 63.69 | 7.07 | 85.27/31.80 | 71.00 | 8.87 | 7.2 | 668.39
Ours
Proposed | 96.57 | 98.08 | 9.09 | 10.61 | 63.61 | 7.40 | 87.14/27.13 | 74.56 | 6.17 | 6.11 | 769.62
w/o FAHP | 96.14 | 98.05 | 10.51 | 11.83 | 63.78 | 5.19 | 85.57/30.91 | 70.03 | 6.12 | 7.18 | 721.79

##  5 Results

Table 1 shows the evaluation results on the SUPERB downstream tasks. Our model
has comparable performance to the teacher model in KS, IC, ER, SV, and SD
tasks, demonstrating the effectiveness of our approach. The performance
degradation occurred mainly in PR, ASR, and SF tasks. These tasks require more
complex content-related information, which is more likely to be lost during
pruning. Using the same WavLM base teacher model, our method outperforms the
distilled models in most tasks, especially in content-related tasks such as
ASR, showing that our model better preserves the performance of the teacher
model.

In addition to the task-specific metrics, we also use the SUPERB score
($\text{superb}_{s}$) to provide an overall evaluation. The SUPERB score is an
average of the linear transformations of all the task-specific metrics, and is
determined by the SOTA model on the benchmark and a predefined FBANK baseline.
At the time of writing, the SOTA model is WavLM-Large222The performance of the
WavLM-Large model can be found at https://superbbenchmark.org/leaderboard..
Formally speaking, the SUPERB score is defined as:

$\displaystyle\text{superb}_{s}=\frac{1}{T}\sum_{t\in
T}\frac{1000}{m_{t}^{\text{sota}}-m_{t}^{\text{fbank}}}(m_{t}^{u}-m_{t}^{\text{fbank}})\text{,}$
(6)

where $m_{t}^{u}$ is the metric of task $t$ and model $u$,
$m_{t}^{\text{sota}}\equiv 1000$, $m_{t}^{\text{fbank}}\equiv 0$.

Figure 2 shows the relationship between the SUPERB score and the number of
parameters. Our model significantly outperforms the distillation models with
similar number of parameters, and even has superior performance to the Wav2vec
2.0 base model. These results show that the proposed method achieves a better
balance between performance and the number of parameters compared to the
distillation-based method.

Figure 2: The relationship between the SUPERB score and the number of
parameters.

We also compare our method with the previous pruning method which directly
removes the attention heads (w/o FAHP in Table 1). Again, the improvement is
mainly reflected in tasks such as ASR, suggesting that fine-grained attention
head pruning can help compensate for the loss of complex information in
structured pruning.

Figure 3a shows the average of the pruning masks $z_{\text{FFN}}$ and
$z_{\text{MHA}}$ during pruning. By introducing STE, the pruning masks of
coarse-grained structures change more frequently and eventually drop to lower
values, which proves the effectiveness of STE. Figure 3b shows the
distribution of the remaining weights of each layer after pruning. Since the
coarse-grained structures can be entirely removed, the remaining parameters
tend to be concentrated, leading to further acceleration.

(a) The average value of $z_{\rm{FFN}}$ and $z_{\rm{MHA}}$.

(b) Remaining (blue) parameters in $W_{V}^{0}$ for 12 layers.

Figure 3: The effectiveness of STE Table 2: Inference time measured on a RTX3090 GPU, by extracting features of librispeech dev-clean set and are averaged on 5 runs. Method | #Params | Infer. time
---|---|---
Millions | Seconds
WavLM base | 94.70 | 91.87(1.0x)
Proposed | 26.57 | 46.08(1.99x)
w/o STE | 26.37 | 67.78(1.35x)
Table 3: Influence of STE on accuracy. ASR, IC, ER, SID are representative of SUPERB content, paralinguistic, speaker, and semantic tasks. Methods | ASR | IC | ER | SID
---|---|---|---|---
WER$\downarrow$ | Acc$\uparrow$ | Acc$\uparrow$ | Acc$\uparrow$
proposed | 10.29 | 98.08 | 63.61 | 74.56
w/o STE | 10.61 | 97.07 | 64.17 | 74.65

In addition, the remaining weight is concentrated at the top of the network.
Since content-related information is more prominent in the features of the top
layers, this distribution of remaining weights may be one of the reasons for
the network's improvement in content-related tasks.

We also measure the inference time of the 2 models above. Table 2 shows the
speed effect of STE. It can be seen that the concentrated weight distribution
brought by STE significantly improves the inference speed of the model. With
STE, the pruned model is 1.4 times faster with a similar number of parameters.

Furthermore, we show the effect of STE on accuracy. Among these 4 tasks, STE
brings improvement in ASR and IC, while causing degradation in ER and SID, but
both the positive and negative influence are not significant. The degradation
in ER and SID may be due to the parameters removed from the lower layers that
are related to speaker or emotion information.

##  6 Conclusion

In this paper, we present a task-agnostic structured pruning method of pre-
trained speech representation models. By using fine-grained attention head
pruning, we retain the ability to represent content-level information and
reduce the performance degradation caused by structured pruning. We introduce
STE to multi-scale structured pruning to further accelerate the model. Our
experiments prove that the proposed model reduces 72% of the parameters while
having comparable performance to the dense model in multiple tasks, and
outperforms the Wav2vec2 base model in average performance.

## References

  * [1] A. Mohamed, H.-y. Lee, L. Borgholt, J. D. Havtorn, J. Edin, C. Igel, K. Kirchhoff, S.-W. Li, K. Livescu, L. Maaløe, T. N. Sainath, and S. Watanabe, ``Self-supervised speech representation learning: A review,'' _IEEE Journal of Selected Topics in Signal Processing_ , vol. 16, no. 6, pp. 1179–1210, Oct. 2022, conference Name: IEEE Journal of Selected Topics in Signal Processing. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/9893562
  * [2] J. Zhao and W.-Q. Zhang, ``Improving Automatic Speech Recognition Performance for Low-Resource Languages With Self-Supervised Models,'' _IEEE Journal of Selected Topics in Signal Processing_ , vol. 16, no. 6, pp. 1227–1241, Oct. 2022. [Online]. Available: https://ieeexplore.ieee.org/document/9801640/
  * [3] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, ``Wav2vec 2.0: A framework for self-supervised learning of speech representations,'' _Advances in Neural Information Processing Systems_ , vol. 33, pp. 12 449–12 460, 2020. [Online]. Available: https://dl.acm.org/doi/abs/10.5555/3495724.3496768
  * [4] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021. [Online]. Available: https://dl.acm.org/doi/abs/10.1109/TASLP.2021.3122291
  * [5] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao _et al._ , ``WavLM: Large-scale self-supervised pre-training for full stack speech processing,'' _IEEE Journal of Selected Topics in Signal Processing_ , vol. 16, no. 6, pp. 1505–1518, 2022. [Online]. Available: https://x-lance.sjtu.edu.cn/en/papers/2022/zyc97-jstsp22.pdf
  * [6] H.-J. Chang, S.-w. Yang, and H.-y. Lee, ``DistilHuBERT: Speech representation learning by layer-wise distillation of hidden-unit bert,'' in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 7087–7091. [Online]. Available: https://ieeexplore.ieee.org/document/9747490/
  * [7] Y. Lee, K. JANG, J. Goo, Y. Jung, and H.-R. Kim, ``FitHuBERT: Going thinner and deeper for knowledge distillation of speech self-supervised learning,'' in _23rd Annual Conference of the International Speech Communication Association, INTERSPEECH 2022_. ISCA, 2022, pp. 3588–3592. [Online]. Available: https://www.isca-speech.org/archive//pdfs/interspeech_2022/lee22p_interspeech.pdf
  * [8] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, ``Fitnets: Hints for thin deep nets,'' _arXiv preprint arXiv:1412.6550_ , 2014\. [Online]. Available: https://arxiv.org/abs/1412.6550
  * [9] R. Wang, Q. Bai, J. Ao, L. Zhou, Z. Xiong, Z. Wei, Y. Zhang, T. Ko, and H. Li, ``LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT,'' in _Interspeech 2022_. ISCA, Sep. 2022, pp. 1686–1690. [Online]. Available: https://www.isca-speech.org/archive/interspeech_2022/wang22t_interspeech.html
  * [10] C. Louizos, M. Welling, and D. Kingma, ``Learning sparse neural networks through l0 regularization.'' in _Sith International Conference on Learning Representations, 2018_ , 2018. [Online]. Available: https://openreview.net/pdf?id=H1Y8hhg0b
  * [11] Y. Peng, K. Kim, F. Wu, P. Sridhar, and S. Watanabe, ``Structured pruning of self-supervised pre-trained models for speech recognition and understanding,'' in _2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Jun. 2023, pp. 1–5. [Online]. Available: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10095780
  * [12] A. v. d. Oord, Y. Li, and O. Vinyals, ``Representation learning with contrastive predictive coding,'' _arXiv preprint arXiv:1807.03748_ , 2018\. [Online]. Available: https://arxiv.org/abs/1807.03748
  * [13] M. Xia, Z. Zhong, and D. Chen, ``Structured pruning learns compact and accurate models,'' in _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 1513–1528. [Online]. Available: https://aclanthology.org/2022.acl-long.107
  * [14] V. Sanh, T. Wolf, and A. Rush, ``Movement pruning: Adaptive sparsity by fine-tuning,'' _Advances in Neural Information Processing Systems_ , vol. 33, pp. 20 378–20 389, 2020. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2020/file/eae15aabaa768ae4a5993a8a4f4fa6e4-Paper.pdf
  * [15] M. Yang, A. Tjandra, C. Liu, D. Zhang, D. Le, and O. Kalinli, ``Learning ASR pathways: A sparse multilingual ASR model,'' in _2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , Jun. 2023, pp. 1–5. [Online]. Available: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10094300
  * [16] C.-I. J. Lai, Y. Zhang, A. H. Liu, S. Chang, Y.-L. Liao, Y.-S. Chuang, K. Qian, S. Khurana, D. Cox, and J. Glass, ``PARP: Prune, adjust and re-prune for self-supervised speech recognition,'' Oct. 2021, arXiv:2106.05933 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2106.05933
  * [17] Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell, ``Rethinking the value of network pruning,'' in _International Conference on Learning Representations_ , 2018. [Online]. Available: https://openreview.net/pdf?id=rJlnB3C5Ym
  * [18] Y. Bengio, N. Léonard, and A. Courville, ``Estimating or propagating gradients through stochastic neurons for conditional computation,'' _arXiv preprint arXiv:1308.3432_ , 2013. [Online]. Available: https://arxiv.org/abs/1308.3432
  * [19] A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, ``Data2vec: A general framework for self-supervised learning in speech, vision and language,'' in _International Conference on Machine Learning_. PMLR, 2022, pp. 1298–1312. [Online]. Available: https://proceedings.mlr.press/v162/baevski22a/baevski22a.pdf
  * [20] P. Yin, J. Lyu, S. Zhang, S. J. Osher, Y. Qi, and J. Xin, ``Understanding straight-through estimator in training activation quantized neural nets,'' in _International Conference on Learning Representations_ , 2019. [Online]. Available: https://openreview.net/forum?id=Skh4jRcKQ
  * [21] L. Chen, M. Asgari, and H. H. Dodge, ``Optimize Wav2vec2s architecture for small training set through analyzing its pre-trained models attention pattern,'' in _2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , May 2022, pp. 7112–7116. [Online]. Available: https://ieeexplore.ieee.org/document/9747831
  * [22] Z. Wang, J. Wohlwend, and T. Lei, ``Structured pruning of large language models,'' in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2020, pp. 6151–6162. [Online]. Available: https://aclanthology.org/2020.emnlp-main.496.pdf
  * [23] S. wen Yang, P.-H. Chi, Y.-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin, T.-H. Huang, W.-C. Tseng, K. tik Lee, D.-R. Liu, Z. Huang, S. Dong, S.-W. Li, S. Watanabe, A. Mohamed, and H. yi Lee, ``SUPERB: Speech Processing Universal PERformance Benchmark,'' in _Proc. Interspeech 2021_ , 2021, pp. 1194–1198. [Online]. Available: https://www.isca-speech.org/archive/interspeech_2021/yang21c_interspeech.html
  * [24] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, ``Librispeech: an ASR corpus based on public domain audio books,'' in _2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2015, pp. 5206–5210. [Online]. Available: https://www.danielpovey.com/files/2015_icassp_librispeech.pdf
  * [25] S. Schneider, A. Baevski, R. Collobert, and M. Auli, ``Wav2vec: Unsupervised Pre-Training for Speech Recognition,'' in _Proc. Interspeech 2019_ , 2019, pp. 3465–3469. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2019-1873