# Deformation and fracture of echinoderm collagen networks

Markus Ovaskaa, Zsolt Bertalanb, Amandine Miksica, Michela Sugnic,d, Cristiano
Di Benedettoc, Cinzia Ferrarioc, Livio Leggioc, Luca Guidettic,d, Mikko J.
Alavaa, Caterina A. M. La Porta ∗c,d, Stefano Zapperi∗d,e,b,f,a a Department
of Applied Physics, Aalto University, P.O. Box 11100, FIN-00076 Aalto, Espoo,
Finland
b Institute for Scientific Interchange Foundation, Via Alassio 11/C, 10126
Torino, Italy
c Department of Biosciences, University of Milan, via Celoria 26, 20133
Milano, Italy
d Center for Complexity and Biosystems, University of Milan, via Celoria 16,
20133 Milano, Italy.
e Department of Physics, University of Milan, via Celoria 16, 20133 Milano,
Italy.
f CNR - Consiglio Nazionale delle Ricerche, Istituto per l’Energetica e le
Interfasi, Via R. Cozzi 53, 20125 Milano, Italy
∗ Corresponding authors<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Collagen networks provide the main structural component of most tissues and
represent an important ingredient for bio-mimetic materials for bio-medical
applications. Here we study the mechanical properties of stiff collagen
networks derived from three different echinoderms and show that they exhibit
non-linear stiffening followed by brittle fracture. The disordered nature of
the network leads to strong sample-to-sample fluctuations in elasticity and
fracture strength. We perform numerical simulations of a three dimensional
model for the deformation of a cross-linked elastic fibril network which is
able to reproduce the macroscopic features of the experimental results and
provide insights on the internal mechanics of stiff collagen networks. Our
numerical model provides an avenue for the design of collagen membranes with
tunable mechanical properties.

###### keywords:

Collagen networks, echinoderms, elasticity, fracture, computational model

††journal: Journal of the Mechanical Behavior of Biomedical Materials

.

##  1 Introduction

Living systems provide a formidable source of inspiration for the design and
synthesis of new classes of materials with a potentially wide range of medical
and non-medical applications [1, 2]. The structural and mechanical properties
of animals mostly relies on collagen [3], one of the main protein constituents
of the extracellular matrix in connective tissues and an essential component
of mammalian skin, bones and tendons. To achieve remarkable mechanical
properties, long collagen molecules self-assemble into hierarchical structures
by forming fibrils that can then arrange into bundles, known as fibers, and
extended elastic networks. The mechanics of fiber networks has received a wide
attention both experimentally [4, 5, 6, 7, 8] and computationally [9, 10, 11,
12, 13, 14, 15] because of its relevance for applications but also as a model
system to understand soft and disordered matter in general.

Most of the experimental studies focus on shear deformation of very soft
collagen gels which are typically extremely soft at small deformations and
become stiffer as the deformation proceeds [16]. This peculiar non-linear
strain stiffening, observed widely in soft collagen gels under shear [4, 5, 6,
7, 8] but also present in stiffer skin tissues under tension [17], has
important mechanical and physiological implications since it provides at the
same time stability for large deformations and facilitates soft remodeling at
small deformations. The precise origin of non-linear stiffening is still under
scrutiny. In biopolymer networks this general behavior is usually attributed
to a crossover between bending dominated and stretching dominated elasticity
[18] or to the strain induced reorientation of the fibers [4, 12]. The
application of these ideas to collagen is still debated [8] and it is not
clear if the deformation is dominated by the non-linear response of individual
fibers or by collective properties of the network [19].

Marine-derived biomaterials in general — and marine collagens in particular —
are promising materials for applications to tissue engineering. The mutable
collagenous tissues (MCTs) of echinoderms, widespread in all five extant
echinoderm classes, display striking passive mechanical properties (viscosity,
tensile strength, and stiffness) that can be actively transformed within
seconds by active mechanisms controlled by the nervous system [20]. Hence, in
addition to the mechanical functions usually associated with ‘conventional’
collagenous structures (i.e. energy storage, transmission and dissipation),
MCTs allow for the detachment of appendages or body parts in response to
disease, trauma or predator attack [20] and for the energy-sparing maintenance
of posture [21]. As in mammalian connective tissues, most mutable collagenous
structures are formed by parallel aggregations of collagen fibrils attached to
proteoglycans [22, 23]. Collagen fibers are usually surrounded by an elastic
network further improving the structural and mechanical properties of the
tissue [20].

Figure 1: Echinoderm collagen networks. The fibril structure of collagen
networks as observed by SEM are reported in panels a),c),e) for sea urchin
(Paracentrotus lividus) (inset of b, photo by Federico Betti) starfish
(Echinaster sepositus, (inset of d, photo by Tato Grasso, CC 3.0) and sea
cucumber (Holothuria tubulosa (inset of f, photo by Roberto Pillon, CC 3.0).
The corresponding diameter distributions are reported in panels b),d),f).

Besides their physiological relevance, MCTs have been recently proposed as a
promising source of collagen for applications to regenerative medicine [24,
23]. In particular, collagen barrier-membranes of mammalian origin are
typically used to facilitate proper tissue regrowth in anatomically separated
compartments during guided tissue regeneration [25]. Marine collagen and MCTs
in particular could provide an innovative and safer alternative to mammalian
ones [26, 24, 27]. Moreover, other new intriguing applications could be
derived from the use of collagen derived from echinoderms: for example to
investigate cellular contractility as a biomechanics-related readout for
phenotypic drug assessment [28]

In this paper, we characterize the elastic tensile deformation and failure of
cross-linked collagen networks derived from three echinoderms (Fig. 1): sea
urchin (Paracentrotus lividus), starfish (Echinaster sepositus) and sea
cucumber (Holothuria tubulosa). We analyze the stiffening behavior of the
networks and characterize sample-to-sample fluctuations in the elastic modulus
and tensile strength. We compare our results with numerical simulations for a
three dimensional model for cross-linked elastic fibrils with parameters
chosen to mimic the microstructure of our samples as revealed by SEM
microscopy. Our model is similar in spirit to recent three dimensional models
of cross-linked fiber networks [9, 29, 11, 14, 13] but its features and
parameters are adapted to the stiff collagen networks we study experimentally.
In particular, we explicitly endow each cross-linker with bending, shear and
torsional rigidities [13]. Simulations of the model highlight the importance
of cross-linking in determining the mechanical properties of these networks,
setting the bases for a computational guided design for this kind of bio-
inspired materials.

Figure 2: Fracture surface of a collagen membrane. A fractured sea urchin
collagen membrane as observed by SEM. Notice the very smooth fracture surface
typical of brittle fracture. Figure 3: Elastic response of collagen membranes.
The stress-strain curves measured in collagen membranes for $n$ independent
experimental realizations and the corresponding average curves are non-linear
and display strain-stiffening. Data reported for a) sea urchin, b) starfish
and c) sea cucumber. The Young modulus exhibits an initial strain stiffening
and then reaches a plateau value $E^{*}$ (d). The error bar is the standard
error.

##  2 Experimental methods

###  2.1 Experimental animals

Adult specimens of the starfish Echinaster sepositus, the sea urchin
Paracentrotus lividus and the sea cucumber Holothuria tubulosa were collected
by scuba divers in Paraggi (Marine Protected Area of Portofino, Ligurian Sea,
Italy) and quickly transferred to the Department of Biosciences (University of
Milan) in order to be immediately dissected. The aboral body walls (ABW) of
the starfish, the peristomial membrane (PM) of the sea urchins and the whole
body walls (BW) of the sea cucumbers were dissected and stored at -20∘ C for
the subsequent collagen extraction protocol.

###  2.2 Echinoderm collagen extraction

Sea urchin and starfish collagen were extracted from the PM and ABW,
respectively, as previously described[24] with only slight modifications for
the latter (see below). Briefly, both starfish ABW and sea urchin PM were
dissected in small pieces, rinsed in artificial sea water (ASW), left in a
hypotonic buffer (10 mM Tris, 0,1% EDTA) for 12 h (RT) and then in a
decellularizing solution (10 mM Tris, 0,1% SDS) for 12 h (RT). After several
washings in PBS, samples were placed in a disaggregating solution (0.5 M NaCl,
0,1 M Tris-HCl pH 8,0, 0,1 M $\beta$-mercapto-ethanol, 0,05 M EDTA-Na).
Starfish samples underwent an additional washing step in citric acid (1 mM pH
3-4) between the decellularizing and the disaggregating solutions. All these
steps were performed under stirring (rotating) conditions. The obtained
collagen suspension was filtered and dialyzed against 0,5 M EDTA-Na solution
(pH 8,0) for 3 h (RT) and against dH2O overnight (RT). Sea cucumber BW was
dissected in small pieces, placed in PBS and gentamicin (40 $\mu$g/mL) and
left in stirring condition (RT) for at least 5 days until an homogeneous
collagen suspension was obtained, which was then subsequently filtered. The
suspensions obtained from the three experimental models were stored at -20∘C
until membrane preparation.

###  2.3 Echinoderm-derived collagen membrane production

Membranes of the three different collagen types were prepared as previously
described for sea urchin collagen matrices [24]. The collagen suspensions were
centrifuged for 10 minutes at 50x g and then for 20 minutes at 2000x g. The
pellet was re-suspended in autoclaved dH2O (final concentration 2mg/mL) and
8,7mL of this suspension were placed in 5cm x 3 cm rubber silicone molds and
left dry overnight at +37∘ C. The so obtained collagen membranes were then
immersed in a EDC/NHS cross-linker solution (EDC 30 mM/NHS 15 mM in MES buffer
100 mM pH 5,5) for 4 h and then washed with PBS, dH2O and ethanol 70%.

Figure 4: Stiffness and strength display sample-to-sample fluctuations. a) We
observe considerable sample-to-sample variations in the Young modulus $E^{*}$
and the tensile strength $\sigma_{c}$ which appear to be correlated
(correlation coefficient $r=0.73$, linear least square fit yields a slope
$b=0.12\pm 0.02$ and intercept $a=6\pm 9$ MPa). We also compute the cumulative
distributions b) for Young moduli $P(E^{*})$ and c) tensile strengths
$P(\sigma_{c})$. d) The strength distribution can be reasonably fit by the
Weibull law. We observe no significant difference between sea urchin and sea
cucumber derived membranes while membranes made out of starfish collagen
appear to be stiffer and stronger.  Figure 5: Cross-linked fibril model. a)
Schematic of the cross-linking between two fibrils A and B. The four cross-
linking springs are depicted in red. b) The initial configuration of the
simulated collagen network. A configuration in the (c) non-linear and (d)
linear elastic regimes.

###  2.4 Scanning Electron Microscopy

Substrates were fixed with 2% glutaraldehyde in 0.1 M cacodylate buffer for 2
h at 4∘C and post-fixed with 1% osmium tetroxide in 0.1 M sodium cacodylate
buffer (2 h, room temperature). After several washings with dH2O, they were
dehydrated in an increasing ethanol scale and treated with a series of
solutions of HMDS (Hexamethyldisilazane) and ethanol in different proportions
(1:3, 1:1, 3:1 and 100% HMDS), mounted on stubs, covered with pure gold (Agar
SEM Auto Sputter, Stansted, UK). The samples are observed under a scanning
electron microscope (SEM) (LEO-1430, Zeiss, Oberkochen, Germany). Examples of
the observed network structure is reported in Fig. 1ace. Images were analyzed
using the ImageJ plugin Diameter-J [30] to extract diameter distributions (see
Fig. 1bdf).

###  2.5 Tensile tests

The collagen membranes were cut into narrow strips by pressing them under an
assembly of evenly spaced razor blades. The ends of the collagen strips were
then glued between pieces of rough emery paper to facilitate clamping. The
length and width of the exposed collagen surface was 10 mm and 2.25 mm,
respectively. Before testing, the thickness of each sample was measured using
a LGF-01100L-B transmission-type photo-electric linear encode (Mitutoyo) with
100 nm resolution. Measurements were taken from three positions and averaged.
The average thickness was in the range $4.6-18.6\mu m$. We did not find any
systematic variation of the mechanical properties with the thickness of the
sample. The tensile tests were performed under controlled temperature (21-22
∘C) and humidity (64-69%) with a Kammrath-Weiss B.1205. A tensile tester
equipped with displacement gauge and DDS3 Controller (Kammrath & Weiss GmbH,
Dortmund, Germany) and a 100 N load cell. After clamping samples into place,
Leibovitz L-15 liquid was added on the surface in order to wet them
completely. Additional droplets of liquid were added every two minutes during
the tests to keep the samples from drying. All samples were tested under
uniaxial tension until failure, with a constant strain rate ($8.35\cdot
10^{-4}s^{-1}$). Fig. 2 reports a representative example of a broken sample
visualized under SEM microscopy. We observe a very smooth fracture surface,
indicative of brittle failure.

##  3 Experimental results

The individual stress-strain curves obtained experimentally for $n$ equivalent
samples are reported in Fig. 3a for sea urchin ($n=14$), in Fig. 3b for
starfish ($n=20$) and in Fig. 3c for sea cucumber ($n=7$). The curves display
a wide variability between sample to sample. The shape of each curve is,
however, similar in all cases and showing a characteristic stiffening behavior
with an initial non-linear regime, followed by a linear regime that ends in
abrupt failure, in qualitative agreement with previous experiments on softer
mammalian collagen networks [31]. To reveal the typical behavior of the
system, we compute the average stress strain curve by defining strain bins of
size $\Delta\epsilon=0.01$ and computing the average stress in each bin (Fig.
3). Using the same procedure, we estimate the strain-dependent Young modulus
$E(\epsilon)$ as the average slope of the stress-strain curves in each of the
strain bins. The result, plotted in Fig. 3d, shows a marked linear region
where $E(\epsilon)$ increases until it reaches a plateau $E^{*}$.

To better visualize the large variations observed from sample to sample, we
report in Fig. 4a the individual values of the Young moduli $E^{*}$ and
tensile strengths $\sigma_{c}$, defined as the peak stresses of each curve. As
reported for other natural and synthetic materials [2], strength and stiffness
are correlated, scaling in our case as $\sigma_{c}\simeq 0.12E^{*}$. The
sample to sample fluctuations can be quantified by measuring the cumulative
distributions of the Young moduli $P(E^{*})$ and of tensile strengths,
$P(\sigma_{c})$, reported in Fig. 4b and 4c, respectively. The functional form
of $P(\sigma_{c})$ is well described by the classical Weibull law
$P(x)=1-\exp(-(x/x_{0})^{k})$ (Fig. 4d) as commonly observed in the fracture
of disordered media [32]. Our results indicate that sea urchin and sea
cucumber collagen networks display comparable mechanical properties, while
starfish derived collagen membranes are slightly stronger and stiffer.

##  4 Numerical results

###  4.1 Model

We construct a three dimensional model of the deformation of cross-linked
elastic fibrils. Collagen fibrils are discretized as bead-spring polymers of
diameter $d$ with stretching and bending stiffness. The rest length of each
discretization segment is $a$ and the elastic energy of a fibril of $L$ nodes
can be written as

$E_{\textrm{ f}}=\frac{1}{2}\sum_{i=2}^{L}k_{\rm
f}(|\textbf{r}_{i}-\textbf{r}_{i-1}|-a)^{2}+K\sum_{j=1}^{L-1}\cos(\theta_{j}),$
(1)

where $k_{f}=\pi Ed^{2}/4a$ is the stretching stiffness, $K=3\pi Ed^{4}/64a$
is the bending stiffness, and $\theta_{j}$ is the angle between two fibril
segments. The fibrils are assumed to have no torsional resistance. The Young
modulus of an individual fibril is chosen to be $E=10$ GPa, as measured in AFM
tests in dry conditions [33]. To simulate fibril-fibril mutual hindrance,
fibril nodes also interact by a purely repulsive Lennard-Jones (LJ) potential
$U(r)=4\epsilon[(\sigma/r)^{12}-(\sigma/r)^{6}]$ for $r<\sigma$ and zero
otherwise. The parameter $\sigma$ is chosen to represent the typical fibril
diameter (e.g. $\sigma=75$nm).

To simulate cross-linking, we randomly place links between neighboring fibril
nodes. In particular, fibrils are cross-linked with probability $p$ when two
nodes are closer than a distance $b$. In order to provide the cross-linkers
with stretching, bending and torsional stiffness, we model each link by a set
of four springs (see Fig. 5a). Considering two fibrils, A and B, a link
involves four nodes, two on fibril A, $i-1$ and $i+1$ and two on fibril B,
$j-1$ and $j+1$, which are connected by four springs $i\pm 1\rightarrow j\pm
1$ (Fig. 5a). We notice that internal stresses are present in real samples
[34], but are not easily quantified in our case. Hence, for simplicity, we
assume that initially there is no stress in the system and set the rest length
of each of the four link connections to their initial distance
$b^{\alpha}_{i,j}$, with $\alpha=1,...4$. The cross-linking energy between
fibril $i$ and $j$ is given by

$E_{\textrm{link}}=\frac{1}{2}k_{\rm
l}\sum_{\alpha=1}^{4}(|\textbf{dr}^{\alpha}_{\\{ij\\}}|-b^{\alpha}_{\\{ij\\}})^{2},$
(2)

dr is the actual length of the link. In most of the simulations, we assume
that both fibrils and cross-linking bonds are linearly elastic and do not
consider fracture events or the formation of new cross-linkers. We also
perform simulations where the cross-linkers display non-linear elastic
behavior as recently shown in Ref. [14]. In particular, the cross-linker
$k_{l}$ stiffness in Eq. 2 is increased from $k_{l,1}=13.3$N/m to a value
$k_{l,2}=1330$N/m when the strain in the link is larger than a threshold
$\epsilon_{1}$.

Most simulations are performed with $N=1500$ fibrils randomly arranged in a
7.5 $\times$ 30 $\times$ 3,75 $\mu$m3 block with open boundary conditions (see
Fig. 5b for a typical starting configuration). We also perform additional
simulations with $N=750$ and $N=12000$ fibrils, with all the dimensions of the
simulation box doubled in the latter case. Other geometrical parameters,
estimated on the basis of the SEM micrographs, are reported in Table 1. We
deform the sample by holding a set of nodes on one edge of the sample fixed
and moving nodes at the other edge at constant velocity. The applied strain-
rate for all simulations is chosen to be $\dot{\epsilon}_{yy}=10$s-1. As it is
common in molecular dynamics simulations, we can only simulate strain rates
that are much larger than the experimental ones. The simulations are performed
with a linear damping term ($F=-\gamma v$), with damping coefficient
$\gamma=2\times 10^{-14}$ kg/s and the mass of each element is set to
$M=10^{-18}$ kg, consistent with the estimate of Ref. [3] for a smaller volume
element. The model is implemented in the LAMMPS molecular dynamics simulator
package [35] and the integration time step is set to $dt=6\times 10^{-8}$s.

Parameter | Symbol | SI value
---|---|---
fibril segment length | $a$ | 150 nm
fibril length | $50a$ | 7.5 $\mu$m
fibril diameter | $d$ | 75 nm
Young’s modulus | $E$ | 10 GPa
bending stiffness | $K$ |  $3.1\times 10^{-13}$ Nm
stretching stiffness | $k_{f}$ | 294.5 N/m
link stiffness | $k_{l}$ | 13-1330 N/m
link distance | $b$ | 150-450 nm
link probability | $p$ | 0.1-1.0
volume fraction | $\rho$ | 0.03-0.06
number of fibrils | $N$ | 750-12000
repulsion energy | $\epsilon$ | 1.65$\times 10^{-13}$ J
repulsion radius | $\sigma$ | 75 nm
damping parameter | $\gamma$ |  $2\times 10^{-14}$ kg/s
fibril segment mass | $M$ |  $10^{-18}$ kg
Table 1: Parameters used in the numerical simulations in SI units. Notice that
the parameter ranges (e.g. in $k_{l}$) represent the range that has been
explored in different simulations.

###  4.2 Simulations

Figure 6: Fibrils reorient during deformation. The distribution of fibril
angles $\theta$ as a function of strain for $N=12000$ fibrils. The data
indicate that the angles between between the loading axis and the fibrils
decrease with strain.

We perform numerical simulations of the network model using the parameters
reported in Table 1. The evolution of the deformation process is reported in
Fig. 5c-d and in video S1. We observe that fibrils are first reoriented in the
initial stage and then the sample becomes much thinner. This is quantified in
Fig. 6 displaying the evolution with strain of the distribution of the angle
$\theta$ formed by the tensile axis and fibril segments. Fig. 6 shows that as
the sample deforms the typical value of $\theta$ decreases, as shown by the
left-ward shift of the peak of the distributions. We checked that the effect
is more marked when the sample size is larger.

From the simulations, we notice that the amount of cross-linking plays a major
role in the mechanical response. This is illustrated in Fig. 7a where we show
that the stress-strain curves change substantially by varying the number of
cross-linkers $N_{link}$. The same curves display features that are in
qualitative agreement with experiments (Fig. 3). For each curve we compute the
local slope that we fit with an exponential function of the strain
$E(\epsilon)=E_{0}+(E^{*}-E_{0})(1-\exp(-\epsilon/\epsilon_{0}))$, which
allows to estimate the steady state Young modulus $E^{*}$ (Fig. 3b). Fig. 7cd
shows that the resulting Young modulus $E^{*}$ increases with the number of
cross-linkers $N_{link}$ and also depends on their stiffness $k_{link}$. The
results reported in Fig. 7c correspond to a set of $750$ fibrils, while those
in Fig. 7d correspond to $1500$ fibrils. Finally, we investigate how the
elastic energy is distributed in the system as it is loaded. In Fig. 8ab, we
report the distribution of elastic energies $E_{el}$ of fibrils (Eq. 1) and
cross-linkers (Eq. 2). Both distributions decay exponentially with a decay
constant that simply scales with the loading strain energy (e.g.
$E_{el}\propto\epsilon^{2}$) as illustrated in Fig. 8c. Finally Fig. 8d shows
the fraction of elastic energy carried by fibrils. As the sample is loaded a
part of the load carried by the cross-linkers is transferred to the fibrils,
in agreement with the observation of a reorientation of the latter.

We notice that the non-linearity of the simulated stress-strain curves and of
the resulting Young moduli is less pronounced than in experiments (compare
Fig. 3b with Fig. 7b). This suggests that geometrical effects due to fibril
arrangement and reorientation are not enough to fully account for the
experimentally observed stiffening. To overcome this problem, we explicitly
include non-linear effects in the cross-linker elasticity following Ref. [14].
Fig. 9a shows the stress-strain curves obtained for two values of the linear
cross-linker spring stiffnesses (i.e. $k_{l}=13.3$ N/m and $k_{l}=1330$N/m).
The curves are then compared with simulations with non-linear cross-linker
springs which interpolate between the two linear spring stiffness values. The
resulting stress-strain curves displays slopes that interpolates between the
curves obtained with linear cross-linker springs. The local Young moduli,
shown in Fig. 9b, now display a more pronounced non-linearity that is closer
to experimental results.

Figure 7: Mechanical role of the cross-linking probability. a) Stress-strain
curves for different values of the number of cross-linkers. b) The strain
dependent Young modulus is fit with exponential functions (see text). c) The
steady-state Young modulus $E^{*}$ as a function of $N_{link}$ for c) $N=750$
and d) $N=1500$ fibrils. The inset shows the dependence of slope with
$k_{link}$.  Figure 8: Elastic energy distributions. The distribution of
elastic energies for a) fibrils and b) cross-linkers as a function of strain.
c) The distributions in a) and b) can be simply rescaled in terms of
$\epsilon^{2}$. d) The fraction of elastic energy carried by fibrils increases
as a function of strain.  Figure 9: Role of the non-linear elasticity of
cross-linkers. a) Simulated stress-strain curves for a model with non-linear
cross-linker springs are compared with the ones obtained with linear cross-
linker springs. b) The corresponding strain dependent Young moduli.

##  5 Conclusions

In this paper, we investigate experimentally and computationally the
mechanical properties of collagen networks. We focus on echinoderm-derived
collagen networks, an alternative to the most widely used collagen of
mammalian origin, and show that their mechanical properties are very similar
to the latter. We notice that for the three types of collagen networks the
resulting mechanical properties are similar, with Young moduli in the 100-500
MPa range and tensile strength in the 20-50MPa range, values that are
remarkably close to what is measured in the human skin [36]. This feature is
possibly relevant in order to use marine collagen in biomedical applications
[24, 23]. Slight quantitative differences exist for starfish collagen that
appears to be stronger and stiffer than the other two. This could be
attributed to different cross-linking properties due to differences in the
sample preparation and extraction. Furthermore, as in other fiber networks
studied in the past [4, 5, 6, 7, 8], our samples also display non-linear
stiffening and sample-to-sample strength fluctuations typical of systems with
a heterogeneous microstructure [32].

To clarify the microscopic origin of the experimentally observed behavior, we
develop a computational model for the deformation of cross-linked elastic
fibril networks. The model allows to reproduce some key features of the
experimental results such as the non-linear stiffening behavior. If we use
linear-elastic cross-linkers, we find non-linear stress-strain curves induced
by a reorientation of the fibrils. The resulting non-linear strengthening is,
however, weaker than the one measured experimentally suggesting that other
mechanisms playing a role. When we include non-linear elastic cross-linkers in
the model, as suggested by Ref. [14], we obtain a more pronounced stiffening.
Additional sources of non-linearity may come from non-linear elastic
deformation of the fibril themselves [3] and from internal stresses due to
sample preparation [34]. Furthermore, our model highlights the crucial role
played by the amount of cross-linking in determining the mechanical response
of the sample. This suggests an interpretation from the experimentally
measured sample-to-sample fluctuations in the stress strain curves displaying
large variations (around 50%) in the Young modulus. We have performed
simulations of the model using the same parameters but different initial
arrangements for the fibrils. When we do this we find much smaller variations
of the resulting Young modulus (around 5%). A possible explanation is that the
experimental variations reflect internal fluctuations in the cross-linking
probability or in their stiffness, quantities that we have held constant in
the simulations but have shown (Fig. 7) to have a crucial influence on the
resulting Young modulus.

Our modeling strategy allows us to obtain an agreement with experimental
results and suggests mechanisms for the observed non-linear elasticity. The
current version of the computational model is presently not able to simulate
fracture, since simply breaking overstretched cross-linkers or fibrils leads
to discontinuities that are difficult to deal with in the present framework. A
possible solution would involve quasi-static approaches where the network
deformation is obtained by energy minimization as for other models for
fracture [32]. These types of models pave the way for a computationally based
design of bio-inspired materials where the mechanical properties can first be
explored in silico to guide the production of materials with desired
mechanical properties.

## Acknowledgements

MO, AM and MJA are supported by the Academy of Finland through its Centres of
Excellence Programme (2012-2017) under project no. 251748. ZB and SZ are
supported by ERC Advanced Grant n 291002 SIZEFFECTS. SZ thanks the Academy of
Finland FiDiPro progam, project 13282993. CAMLP thanks the visiting professor
program of Aalto University where part of this work was completed. The authors
are grateful to Lahja Martikainen and Matti Toivonen of the Molecular
Materials research group at Aalto University for assistance with the
mechanical testing.

## Authors Contributions

LG,LL,MS prepared samples. CF,MS performed SEM experiments. AM,MO performed
tensile tests. MO analyzed data. ZB,CDB,SZ designed the model. ZB wrote the
code. MO performed numerical simulations. CAMLP,SZ wrote the manuscript.
CAMLP,MJA,SZ designed and coordinated the project.

## References

  * [1] N. Huebsch, D. J. Mooney, Inspiration and application in the evolution of biomaterials, Nature 462 (7272) (2009) 426–32.  doi:10.1038/nature08601.
  * [2] U. G. K. Wegst, H. Bai, E. Saiz, A. P. Tomsia, R. O. Ritchie, Bioinspired structural materials, Nat Mater 14 (1) (2015) 23–36.  doi:10.1038/nmat4089.
  * [3] M. J. Buehler, Nature designs tough collagen: explaining the nanostructure of collagen fibrils, Proc Natl Acad Sci U S A 103 (33) (2006) 12285–90.  doi:10.1073/pnas.0603216103.
  * [4] D. Vader, A. Kabla, D. Weitz, L. Mahadevan, Strain-induced alignment in collagen gels, PLoS One 4 (6) (2009) e5902.  doi:10.1371/journal.pone.0005902.
  * [5] S. B. Lindström, D. A. Vader, A. Kulachenko, D. A. Weitz, Biopolymer network geometries: characterization, regeneration, and elastic properties, Phys Rev E Stat Nonlin Soft Matter Phys 82 (5 Pt 1) (2010) 051905.
  * [6] S. B. Lindstrom, A. Kulachenko, L. M. Jawerth, D. A. Vader, Finite-strain, finite-size mechanics of rigidly cross-linked biopolymer networks, Soft Matter 9 (2013) 7302–7313.  doi:10.1039/C3SM50451D.
URL http://dx.doi.org/10.1039/C3SM50451D

  * [7] S. Münster, L. M. Jawerth, B. A. Leslie, J. I. Weitz, B. Fabry, D. A. Weitz, Strain history dependence of the nonlinear stress response of fibrin and collagen networks, Proc Natl Acad Sci U S A 110 (30) (2013) 12197–202.  doi:10.1073/pnas.1222787110.
  * [8] A. J. Licup, S. Münster, A. Sharma, M. Sheinman, L. M. Jawerth, B. Fabry, D. A. Weitz, F. C. MacKintosh, Stress controls the mechanics of collagen networks, Proc Natl Acad Sci U S A 112 (31) (2015) 9573–8.  doi:10.1073/pnas.1504258112.
  * [9] Z. Qin, M. J. Buehler, Flaw tolerance of nuclear intermediate filament lamina under extreme mechanical deformation, ACS Nano 5 (4) (2011) 3034–42.  doi:10.1021/nn200107u.
  * [10] M. Das, D. A. Quint, J. M. Schwarz, Redundancy and cooperativity in the mechanics of compositely crosslinked filamentous networks, PLoS One 7 (5) (2012) e35939.  doi:10.1371/journal.pone.0035939.
  * [11] B. Lee, X. Zhou, K. Riching, K. W. Eliceiri, P. J. Keely, S. A. Guelcher, A. M. Weaver, Y. Jiang, A three-dimensional computational model of collagen network mechanics, PLoS One 9 (11) (2014) e111896.  doi:10.1371/journal.pone.0111896.
  * [12] J. Feng, H. Levine, X. Mao, L. M. Sander, Alignment and nonlinear elasticity in biopolymer gels, Phys Rev E Stat Nonlin Soft Matter Phys 91 (4) (2015) 042710\.
  * [13] G. Žagar, P. R. Onck, E. van der Giessen, Two fundamental mechanisms govern the stiffening of cross-linked networks, Biophys J 108 (6) (2015) 1470–9.  doi:10.1016/j.bpj.2015.02.015.
  * [14] B. Depalle, Z. Qin, S. J. Shefelbine, M. J. Buehler, Influence of cross-link structure, density and mechanical properties in the mesoscale deformation mechanisms of collagen fibrils, J Mech Behav Biomed Mater 52 (2015) 1–13.  doi:10.1016/j.jmbbm.2014.07.008.
  * [15] J. Feng, H. Levine, X. Mao, L. M. Sander, Nonlinear elasticity of disordered fiber networks, Soft Matter 12 (2016) 1419–1424.  doi:10.1039/C5SM01856K.
URL http://dx.doi.org/10.1039/C5SM01856K

  * [16] S. Motte, L. J. Kaufman, Strain stiffening in collagen i networks, Biopolymers 99 (1) (2013) 35–46.  doi:10.1002/bip.22133.
  * [17] W. Yang, V. R. Sherman, B. Gludovatz, E. Schaible, P. Stewart, R. O. Ritchie, M. A. Meyers, On the tear resistance of skin, Nat Commun 6 (2015) 6649.  doi:10.1038/ncomms7649.
  * [18] P. R. Onck, T. Koeman, T. van Dillen, E. van der Giessen, Alternative explanation of stiffening in cross-linked semiflexible networks, Phys Rev Lett 95 (17) (2005) 178102.
  * [19] M. L. Gardel, J. H. Shin, F. C. MacKintosh, L. Mahadevan, P. Matsudaira, D. A. Weitz, Elastic behavior of cross-linked and bundled actin networks, Science 304 (5675) (2004) 1301–5.  doi:10.1126/science.1095087.
  * [20] I. C. Wilkie, Mutable collagenous tissue: overview and biotechnological perspective, Prog Mol Subcell Biol 39 (2005) 221–50.
  * [21] N. Takemae, F. Nakaya, T. Motokawa, Low oxygen consumption and high body content of catch connective tissue contribute to low metabolic rate of sea cucumbers, Biol Bull 216 (1) (2009) 45–54.
  * [22] J. A. Trotter, F. A. Thurmond, T. J. Koob, Molecular structure and functional morphology of echinoderm collagen fibrils, Cell Tissue Res 275 (3) (1994) 451–8.
  * [23] M. Sugni, D. Fassini, A. Barbaglio, A. Biressi, C. Di Benedetto, S. Tricarico, F. Bonasoro, I. C. Wilkie, M. D. Candia Carnevali, Comparing dynamic connective tissue in echinoderms and sponges: morphological and mechanical aspects and environmental sensitivity, Mar Environ Res 93 (2014) 123–32.  doi:10.1016/j.marenvres.2013.07.010.
  * [24] C. Di Benedetto, A. Barbaglio, T. Martinello, V. Alongi, D. Fassini, E. Cullorà, M. Patruno, F. Bonasoro, M. A. Barbosa, M. D. C. Carnevali, M. Sugni, Production, characterization and biocompatibility of marine collagen matrices from an alternative and sustainable source: the sea urchin paracentrotus lividus, Mar Drugs 12 (9) (2014) 4912–33.  doi:10.3390/md12094912.
  * [25] J. Glowacki, S. Mizuno, Collagen scaffolds for tissue engineering, Biopolymers 89 (5) (2008) 338–44.  doi:10.1002/bip.20871.
  * [26] T. H. Silva, J. Moreira-Silva, A. L. P. Marques, A. Domingues, Y. Bayon, R. L. Reis, Marine origin collagens and its potential applications, Mar Drugs 12 (12) (2014) 5881–901.  doi:10.3390/md12125881.
  * [27] C. Ferrario, L. Leggio, R. Leone, C. Di Benedetto, L. Guidetti, V. Coccè, M. Ascagni, F. Bonasoro, C. A. M. La Porta, M. D. Candia Carnevali, M. Sugni, Marine-derived collagen biomaterials from echinoderm connective tissues, Mar Environ Resdoi:10.1016/j.marenvres.2016.03.007.
  * [28] H. Zhao, L. Zhou, Q. Zhang, X. Zhou, Y. Zhang, H. Chen, Y. Du, Bi-content micro-collagen chip provides contractility-based biomechanical readout for phenotypic drug screening with expanded and profiled targets, Lab Chip 15 (17) (2015) 3481–3494.  doi:10.1039/c5lc00589b.
  * [29] Z. Qin, M. J. Buehler, Impact tolerance in mussel thread networks by heterogeneous material distribution, Nat Commun 4 (2013) 2187.  doi:10.1038/ncomms3187.
  * [30] N. A. Hotaling, K. Bharti, H. Kriel, C. G. Simon, Jr, Diameterj: A validated open source nanofiber diameter measurement tool, Biomaterials 61 (2015) 327–38.  doi:10.1016/j.biomaterials.2015.05.015.
  * [31] B. A. Roeder, K. Kokini, J. E. Sturgis, J. P. Robinson, S. L. Voytik-Harbin, Tensile mechanical properties of three-dimensional type i collagen extracellular matrices with varied microstructure, Journal of Biomechanical Engineering 124 (2) (2002) 214–222.
URL http://dx.doi.org/10.1115/1.1449904

  * [32] M. J. Alava, P. K. V. V. Nukala, S. Zapperi, Statistical models of fracture, Advances in Physics 55 (2006) 349–476.  arXiv:cond-mat/0609650, doi:10.1080/00018730300741518.
  * [33] M. P. E. Wenger, L. Bozec, M. A. Horton, P. Mesquida, Mechanical properties of collagen fibrils, Biophys J 93 (4) (2007) 1255–63.  doi:10.1529/biophysj.106.103192.
  * [34] O. Lieleg, J. Kayser, G. Brambilla, L. Cipelletti, A. R. Bausch, Slow dynamics and internal stress relaxation in bundled cytoskeletal networks, Nat Mater 10 (3) (2011) 236–42.  doi:10.1038/nmat2939.
  * [35] S. Plimpton, Fast parallel algorithms for short-range molecular dynamics, Journal of Computational Physics 117 (1) (1995) 1–19.  doi:10.1006/jcph.1995.1039.
URL
http://www.sciencedirect.com/science/article/B6WHY-45NJN1B-3N/2/58aa2a309d2ebbbe60e0f417d398b0ef;http://www.bibsonomy.org/bibtex/238d1dbaa018ed3ab0d83ee353cf41f42/pmd

  * [36] M. D. Gilchrist, A. Ní Annaidh, M. Destrade, Characterising the anisotropic mechanical properties of excised human skin, Journal Of The Mechanical Behavior Of Biomedical Materials 5 (2012) 139–148.

## Supplementary information

Video S1: a representative example of a simulated fibril network under tensile
elongation.


# Feature Extraction for Temporal Signal Recognition: An Overview

Imad Rida
Normandie Univ, UNIROUEN, UNIHAVRE, INSA Rouen, LITIS
76 000, Rouen
France
<EMAIL_ADDRESS>

###### Abstract

Due to the huge progress of the recording devices, data from heterogeneous
nature can be recorded, such as spatial, temporal and spatio-temporal.
Nowadays, time-based data is of particular interest since it has the ability
to capture the characteristics evolution of the data over time. The temporal
data could be gait, auditory scene, piece of music, and so on. In this paper,
we are particularly interested in feature extraction for two different
temporal recognition applications namely, audio and human behavior analysis
and recognition. Indeed, relevant and discriminative features are of critical
and fundamental importance to achieve high performances in any automatic
pattern recognition system. This work is intended to provide researchers with
a brief overview of the different existing features through an understanding
of basic taxonomies which may serve as a reference to identify the adequate
features for a specific task.

_K_ eywords Signal $\cdot$ Security $\cdot$ Audio $\cdot$ Biometrics $\cdot$
Video

##  1 Introduction

Over the past two decades, there has been a massive and abundant amount of
data garnered from social media, data from internet-enabled devices (including
smartphones and tablets), video and voice recordings (digital cameras,
microphones), etc. The recorded data represents a huge and important resource
of information and knowledge which could be exploited in real life
applications such as, security, education, healthcare etc [1].

Despite the ability of recorded data to give useful information, it is not
always captured in ready and adequate format for analysis and interpretation
which clearly shows the need of novel efficient methods to address this
problem [2, 3]. However, doing this correctly and completely represents a
continuous challenging problem which took the effort and attention of
researchers. Temporal signals constitute a popular class of signals, where
data records are indexed by time. There is a large variety of examples in the
context of temporal signal recognition applications; within the most popular
ones we can find: audio signal recognition or human behavior analysis and
recognition.

Section 2 introduces audio signal recognition. Section 3 presents human
behavior analysis and recognition. Section 4 describes the architecture of an
automated recognition system. Section 5 explains audio feature extraction.
Section 6 reports human behavior analysis and recognition features extraction.
Finally, Section 7 concludes our paper.

##  2 Audio Signal Recognition

Human listeners are very good at all kinds of sound detection and
identification tasks, from understanding heavily accented speech to noticing a
ringing phone underneath music playing at full blast. Efforts to duplicate
these abilities on computer have been particularly intense in the area of
audio signal recognition. The beginning was with speech-based applications
[4], later extended to other audio recognition tasks, ranging from music
analysis [5] to the problems of analyzing the general "ambient" audio [6].

To tackle the problem of audio signal recognition, a development of auditory
signals taxonomy is needed. Gerhard [7] defines the sound as a pattern of air
pressure that is detectable (the average human can hear frequencies between 20
Hz and 15 000 Hz). He splitted the hearable sound into 5 main categories:
noise, natural sounds, artificial sounds, speech and music as is shown in
Figure 1.

Figure 1: Taxonomy of sounds [7].

From this taxonomy one can derive a broad range of audio signal recognition
problems based on speech, music and mixture of artificial and natural sounds.
In the following we briefly emphasize on three prominent applications namely
speech recognition, music transcription and computational auditory scene
analysis. These trends of research aim at building intelligent machines able
to interpret and infer based on audio information.

###  2.1 Speech

Speech has been one of the fundamental audio research topics for many years
now. There are three main topics in speech research in recognition context:
speaker, speech and language recognition. Speaker recognition is the general
term of discriminating one person from another based on the sound of their
voices. It was for instance a good biometric modality used as alternative of
conventional passwords, personal identification numbers (PINs) or smart cards
[8, 9, 10]. Speech recognition is the ability of a machine to convert a speech
signal to a readable sequence of words and phrases [11, 4, 12], while language
recognition refers to the process of automatically identifying the language
spoken in a speech sample [13, 14].

###  2.2 Automatic Music Transcription

In the past years, the problem of Automatic Music Transcription (AMT) has
known an increased interest due to many applications associated with it, such
as, interactive music systems, automatic search and annotation of musical
information, as well as musicological analysis [15, 16]. It corresponds to the
process of taking a sequence of sound waveform and extracting from it some
form of musical notation related to the high-level musical structures [17].
AMT machine generally follows three main stages, spectral estimation, pitch
detection and symbol formation [7]. Spectral estimation is usually done with
Fourier analysis and the detected pitch information is represented in
recognizable format by humans and computers such as Music Instrument Digital
Interface (MIDI). A melody line represented by a series of pitches could be
represented in any key signature.

The AMT problem can be divided into several subtasks such as, musical
instrument identification which seeks to identify the musical instrument(s)
playing in a music piece [18, 19]; onset detection which aims to find
beginnings of notes or events [20, 21] or music chord recognition [22, 23, 24,
25]. The latter represents the most fundamental structure and back-bone of the
tonal system which makes them deft to represent occidental music. Moreover
harmonic informations extracted from chord recognition task can serve as
features for high level tasks such as music genre classification or music
retrieval.

###  2.3 Computational Auditory Scene Analysis

Perception refers to the process of becoming aware of the elements of the
environment through physical sensation, which can include sensory input from
the eyes, ears, nose, tongue, or skin. While most of the efforts have focused
on vision perception (it represents the dominant sense in humans to build
intelligent artificial machines), there is now a growing interest based on
audio modality. Computational Auditory Scene Analysis (CASA) refers to the
computational analysis of an acoustic environment, and the recognition of
specific sounds and events in it. Automatic sound event detection (also called
acoustic event detection) and Computational Audio Scene Recognition (CASR)
represent two emerging topics in the general context of CASA [26]. The former
aims to process the continuous acoustic signals and convert them into symbolic
descriptions of the corresponding sound events present at the auditory scene
when the latter seeks to recognize the acoustic environment or context.
Applications that can specifically benefit from CASA include automatic tagging
in audio indexing [27], context-aware services [28], intelligent wearable
devices [29] and robotics navigation systems [30].

##  3 Human Behavior Analysis and Recognition

There is an increasing interest in video surveillance applications to propose
solutions able to analyze the human behaviors and identify individuals.
Currently, visual surveillance is one of the most active research areas in
computer vision and pattern recognition. The goal of visual surveillance is
not only to replace the human eyes by cameras but also to make the
surveillance task as automatic as possible. Applications in visual
surveillance can be divided into two main tasks, human behavior analysis and
person recognition.

###  3.1 Human Behavior Analysis

In the past years, a considerable number of surveillance cameras have been
installed in public places, train stations, airports and many research efforts
have been devoted to build intelligent systems able to analyze the visual data
in order to extract information about the humans behavior in scenes. Ideal
intelligent monitoring system should be able to automatically, analyze the
collected video data, detect the suspicious or endangering behaviors and give
out an early warning before the adverse event happens.

Many suspicious behaviors could be defined depending on the application
domain, such as loitering (waiting time to catch a bus longer than a threshold
time) illustrated in Figure 2 or fighting shown in Figure 3. Detection of
suspicious human behavior involves modeling and classification of human
activities based on predefined knowledge. However this task is not trivial due
to the randomness and complex nature of human movement [31, 32, 33, 34].

Figure 2: Example of suspicious (fighting) behave detection [32]. Figure 3:
Example of suspicious (loitering) behave detection [35].

###  3.2 Human Recognition in Surveillance Systems

A system which detects abnormal behavior should also be able to identify all
the suspicious persons in the scene, and track them across the zones.
Monitoring system requires not only to estimate the location and behavior, but
also to obtain the identity information [36]. Gait is the most suitable
biometric modality in the case of intelligent video surveillance [37, 38]. In
monitoring scenes, people are usually distant from cameras, which makes most
of biometric features not suitable even the use of face for identification.
The drawbacks are obvious, for example, view angle variations and occlusions
cause the impossibility to capture the full faces and distance brings low-
resolution face images. Therefore, face can not always achieve good
performances in practice. In contrast, gait is a behavioral biometric,
including not only individual appearance, such as limb, leg length, width, but
also the dynamic information of individual walking [39, 40]. Compared with
other biometric modalities, gait is remote accessed and difficult to imitate
or camouflage. Moreover, the capturing process does not require cooperation,
contact with special sensor, or high images resolution [41, 42, 43].

##  4 Architecture of Automated Recognition Systems

Assume that we have several objects associated with classes and that objects
belonging to the same class share the same features more than with objects in
other classes. The pattern recognition problem consists of assigning a new
unlabeled object to a class. It is accomplished by determining the features of
the object and identifying the class of which those features are most
correlated.

Given the goal of recognizing objects based on their features, the main task
of an automated recognition system can be divided into three basic subtasks
[44]: the description subtask which generates features of an object using
feature extraction techniques, mapping raw features into another
discriminative space where objects from different groups are well separated by
feature representation techniques and finally the classification subtask which
assigns a class label to the object based on those features and a trained
classifier [45].

As the ultimate goal of an automated recognition system is to discriminate the
class membership of the observed novel objects, a good functional automated
pattern recognition system should be able to classify the novel observed
objects with the minimum misclassification rate possible. Relevant and
discriminative features are of critical and fundamental importance to achieve
high performances in any automatic pattern recognition system [46]. Feature
extraction seeks to transform and fix the dimensionality of an initial input
raw data to generate a new set of features containing meaningful information
contributing to assign the observations to the correct corresponding either on
training samples or new unseen data class.

Different type of information can be extracted from the initial recorded raw
data (time, frequency, spatial information etc) depending on the nature of the
input raw data, the context and domain of the task. In the following we
present a general overview of the commonly used features in the domain of
audio and human behavior analysis application.

##  5 Audio Features Extraction

Humans have powerful brain capabilities to analyze and distinguish between
different sounds and assign them to a specific semantic class. Unfortunately
this is not possible for the machines due to the hidden nature of semantic
information in the recorded sounds. This motivates the researchers to
introduce several processing tools for audio signal which led to a large
variety of features for different applications, such as music transcription,
CASA, speech recognition etc.

Feature extraction is of extreme importance since the performance of the
system depends on the quality of the extracted features. The features,
determine which information and properties are available during the
recognition process. They should capture enough invariant audio properties
within the same class and variant ones between different classes.

Audio features represent specific characteristics of audio signals. Several
attributes have been introduced to describe different types of audio signals
from psychoacoustic point of view such as, duration, loudness, pitch, and
timbre [47].

Duration: represents the time between the beginning and the end of the audio
signal. The envelope of the sound over time can be divided into, Attack,
Decay, Sustain and Release (ADSR).

Loudness: is a psychoacoustic property of the sound, it represents our human
perception of how loud or soft sounds of various intensities are. The loudness
of a sound is subjective, it varies from person to person and measured by sone
and phon units [48].

Pitch: is a perceptual property. In [49] is defined as the intensive attribute
of auditory sensation in terms of which a sound may be ordered on a scale
extending from soft to loud. The pitch is measured with mel unit. In some
cases the pitch means the fundamental frequency [50].

Timbre: is defined as the attribute of auditory sensation which makes the
listener able to judge that two non-identical sounds which are presented
similarly and have the same loudness and pitch are dissimilar [49]. It is the
most complex attribute in the sound. For example, timbre helps to distinguish
between two different instruments playing the same note with same loudness.

Audio features extraction attempts to capture the aforementioned attributes
most adapted to the application domain. Audio features hold five main
properties [47]: signal format, domain, temporal scale, semantic meaning, and
the underlying model which will be further discussed in the following.

  * •

Signal format: there are two main categories, features based on linear coding
and based on lossy compression. The majority of audio features are linearly
coded based, however several works tried to introduce features in lossy
compression context (MPEG format) [51].

  * •

Domain: it represents the final domain of the extracted audio feature. The
features could belong to different domains such as, temporal, frequency,
cepstral, modulation frequency and reconstructed phase space [47].

  * •

Temporal scale: in this property, the features could belong to three different
categories, intraframe, interframe and global. In the intraframe features, the
signal is considered locally stationary. Each frame is taken in consideration
separately which results in one feature vector by frame. A well known example
of intraframe (or short-time) features is MFCCs. In contrast the interframe
features capture the temporal change of a given audio signal. An example of
the interframe features are rhythmic features. Note also the global features
which are computed from the whole signal.

  * •

Semantic meaning: it includes perceptual features which are based on the
aspects of human perception such as pitch, rhythm, and physical features
describing the audio signals based on physical and statistical properties
(Fourier transform).

  * •

Underlying model: there are two types of features, those based on
psychoacoustic model and those without it. An example of psychoacoustic model
is the incorporation of the filter banks [47].

From the previous description one can remark there is a various and large
variety of features to tackle the problem of audio signal recognition. This
shows the need to a taxonomy organization into hierarchical groups with shared
properties. Inspired by the taxonomy proposed by [47], we introduce the
following organization which divides the audio features into five main
domains, temporal, physical frequency, perceptual frequency, cepstral and
modulation frequency as illustrated in Figure 4.

Figure 4: Taxonomy of audio features.

###  5.1 Temporal Features

Temporal features are directly extracted from the audio raw data without any
transformation. The temporal features include:

  * •

Zero crossings: it is a very simple characteristic of the audio signals that
has been used in speech recognition. We can find features as, Zero Crossing
Rate (ZCR) [52], Linear Prediction Zero Crossing Ratio (LP-ZCR) [53], Zero
Crossing Peak Amplitude (ZCPA) [54] and Pitch Synchronous Zero Crossing Peak
Amplitude (PS-ZCPA) [55].

  * •

Amplitude: features are extracted from amplitude. An example is the Amplitude
Descriptor (AD) that has been introduced for animal sounds discrimination
[56].

  * •

Power: it represents the mean square of the input raw signal such as, Short
Time Energy (STE) [57] and volume [58].

###  5.2 Physical frequency features

The physical audio features are based on mathematical and statistical
formulations such as, Fourier and Wavelet transforms. The physical frequency
features are structured as follows:

  * •

Autoregression features: we can find features such as, Linear Predictive
Coding (LPC) [59] and Line Spectral Frequencies (LSF) [60].

  * •

Adaptive time-frequency decomposition features: they include features using
time-frequency representations based on wavelet transformation. The advantage
of the wavelet is the ability to provide variable frequency resolutions within
time [61].

  * •

Short time Fourier transform (STFT) features: these features calculated based
on the STFT can capture properties of spectral envelope and phase information,
such as subband energy ratio [62], spectral flux [63], spectral slope [64],
and spectral peaks [65].

####  5.2.1 Perceptual frequency features

Contrary to physical features, the perceptual ones try to include the semantic
in the feature extraction based on the human auditory system. The perceptual
features are organized below:

  * •

Brightness: brings information about the dominant frequency of the signal such
as, spectral centroid [66] and sharpness [67].

  * •

Tonality: it is the characteristic of the sound that distinguish noise in
tonal sounds including spectral dispersion [68] and spectral flatness [69].

  * •

Loudness: it includes integral loudness [70].

  * •

Pitch: several features have been introduced in this subgroup such as, pitch
histogram [71] and psychoacoustic pitch [72].

  * •

Chroma: the sensation of pitch is based on, tone height and chroma. The range
of chroma is divided into 12 pitch classes such as the Pitch Class Profile
(PCP) [22].

  * •

Harmonicity: it represents the Power Spectral Density (PSD) at integer
multiples of the fundamental frequency [73].

###  5.3 Cepstral features

Cepstral features have been widely used in speech analysis. They aim to
capture the timbral and pitch characteristics. We can find three main
subgroups:

  * •

Perceptual filter bank based features: they represent the Fourier transform of
logarithm of the magnitude spectrum. A representative of these features is the
widely used Mel-Frequency Cepstral Coefficients (MFCCs) and its extensions
such as Relative Autocorrelation Sequence MFCC (RAS-MFCC) and CHNRAS-MFCC
[74].

  * •

Advanced auditory model based features: these features try to model the
physiological human hearing process. An example is noise robust audio features
[75].

  * •

Autoregression based features: the features are calculated based on linear
predictive analysis such as, Perceptual Linear Prediction (PLP) [76], Relative
Spectral Perceptual Linear Prediction (RASTA-PLP) [77] and Linear Prediction
Cepstrum Coefficients (LPCC) [78].

###  5.4 Modulation Frequency Features

These features attempt to capture rhythm information. They represent a timbre
and energy change over time such as, beat spectrum [79] and pulse metric [63].

Table 1 summarizes different features along with their category and potential
applications. The use of these presented features is not restricted to the
reported applications. Extensions to other audio recognition tasks have been
explored in the literature in order to evaluate their efficiency and
genericity ability. The principal remark in this context is the fact that
features designed for music were only successfully applied to music based
application, in contrast to the speech and speaker recognition features which
have already shown good performances for auditory scene recognition [80]. This
is due to the ability of speech-based features to capture intrinsic
characteristics present in the audio scenes.

Table 1: Overview of audio features and their applications. SP: Speech Recognition, SR: Speaker Recognition, CASR: Computational Auditory Scene Recognition, MA: Music Analysis, AR: Animal Sound Recognition. Type | Examples | Application
---|---|---
1\. Temporal features |  |
• Zero crossings | ZCR, LP-ZCR, ZCA, PS-ZCA | SP, SR, CASR
• Amplitude | AD | AR
• Power | STE, Volume | CASR
2\. Physical frequency features |  |
• Autoregression | LPC, LSF | SP, SR, CASR
• Adaptive time-frequency decomposition | DWCH, ATFT | MA
• Short time Fourier transform | Spectral flux/slope/peaks | MA
3\. Perceptual frequency features |  |
• Brightness | Spectral Centroid Sharpness | MA
• Tonality | Spectral flatness/dispersion | MA
• Loudness | Integral loudness | CASR
• Pitch | Pitch histogram/psychoacoustic | MA
• Chroma crossings | PCP | MA
• Harmonicity | PSD | MA
4\. Cesptral features |  |
• Perceptual filter bank | MFCC, RAS-MFCC, CHNRAS-MFCC | SP, SR, CASR
• Advanced auditory model based | Noise robust | SP, SR
• Autoregression based | PLP, RASTA-PLP, LPCC | SP, SR, CASR
5\. Modulation frequency features |  |
• Rythm | Beat spectrum, Pulse metric | MA

##  6 Human Behavior Analysis and Recognition Features Extraction

Recognizing complex human behaviors and activities from video recorded data
helps to develop intelligent video monitoring systems. However human behavior
analysis and recognition represents one of the most challenging problems in
the domain of computer vision due to the view angle variations, occlusions and
the randomness of the activities. In visual perception based systems, the
features try to capture characteristics that describe the human object
segmented out from the raw video sequence such as, shape, silhouette, colors,
poses, and body motions [81].

We introduce a taxonomy which divides these features into four main groups:
space-time volumes, space-time trajectories, space-time local and body model
as is shown in Figure 5. The next subsections describe those features.

Figure 5: Taxanomy of human behavior analysis and recognition features.

###  6.1 Space-Time Volumes

Space-time volumes are constructed by stacking 2-D (XY) image frames along the
time axis (T) as a 3D (XYT) cube as shown in Figure 6. The space-time volumes
are able to capture both spatial and temporal information of the recorded
object. Mainly the images are stacked after a segmentation step which aims to
track the shape changes of the person in question [82]. Based on the training
video data, a space-time volume is constructed for different activities and
persons [83, 84].

Mainly, the space time volume features provide an efficient way to capture and
combine both spatial and temporal information; however this requires a good
preprocessing step of silhouette segmentations. Furthermore, viewpoint and
occlusion are factors that drastically affect the performances.

Figure 6: An example of the space-time volumes construction [85].

###  6.2 Space-time trajectories

These features seek to capture space-time trajectories by capturing the human
joint positions as a set of 2-dimensional (XY) or 3-dimensional (XYZ) points.
The trajectories are tracked over time which results 3-D XYT or 4-D XYZT
representations as shown in Figure 7. Several works have used these features
[86, 87, 88].

Figure 7: An example of trajectories in XYZ and XYT spaces [89].

###  6.3 Local Space-Time

3-D space-time volumes are considered as solid objects. This gives the ability
to extract some appropriate local characteristics to distinguish between them.
Several approaches are used to extract the local features: in [90, 91, 92],
the local features are extracted from each video frame, the resulting features
are concatenated over time to describe the human motion. In the other hand,
some approaches extract local features directly from the 3-D volumes as is
shown in Figure 8 [93, 94, 95]. The local features are extracted using
interest point detectors and descriptors such as, Harris operator, Laplacian
of Gaussian (LoG), Scale-Invariant Feature Transform (SIFT) and Histogram of
Oriented Gradients (HOG).

Figure 8: An example of 3-D volumes (XYT) used to extract local features [93].

###  6.4 Body Modeling

A human body model is developed to capture the 3D geometric and kinematic
structure of human body (see Figure 9). The model is supposed to extract
information such as degrees of joint angles, length, width etc [96]. There
have been several works using such features [97, 98].

Figure 9: An example of human body skeleton model [99].

The previously introduced features for human behavior analysis and recognition
try to capture the intrinsic characteristics of the moving subject and track
their evolution over time [100]. In the space-time volumes, whole body
silhouette is taken in consideration, it is simple to implement. However in
outdoor conditions the subjects suffer from different intra-class variations
caused by different conditions such as occlusion which make the segmentation
step very complicated [101]. The performance of space-time volume features is
affected by the quality of segmentation and can lead to very low performances
in case of poor segmentation. Features based on space-time trajectories follow
the same principle of the latter ones, however instead of taking the whole
silhouette, some key points are retained to construct the moving body
trajectories. The performance depends on the choice and amount of the
trajectories.

Motivated by the impact of segmentation on the performance of previous
features, local space-time features have been introduced; they are extracted
as local descriptors and are further concatenated to construct a feature
vector. Following the same idea, features capturing geometric and kinematic
structure of the human body have been suggested. This type of features showed
good performances however modeling the body is not a trivial task.

Once the features are extracted, finding a suitable feature representation
space is of extreme importance to achieve good classification performances.
The main techniques of feature representations have been studied in [102].

##  7 Conclusion

The recorded data represents a huge and important resource of information and
knowledge which could be exploited in different real life applications
including security, education, healthcare etc. Temporal signals have known a
particular interest thanks to their ability to capture the intrinsic
characteristics evolution over time. Indeed, temporal signals can be from
different nature such as, gait, auditory scene or even a piece of music. In
this paper, we have presented a general overview of the different extracted
features by proposing simple taxonomies. This intended to help researchers to
identify the most prominent features for different temporal signal recognition
applications.

## References

  * [1] Somaya Al Maadeed, Xudong Jiang, Imad Rida, and Ahmed Bouridane.  Palmprint identification using sparse and dense hybrid representation.  Multimedia Tools and Applications, pages 1–15, 2018.  DOI: 10.1007/s11042-018-5655-8.
  * [2] Imad Rida, Somaya Al-Maadeed, Arif Mahmood, Ahmed Bouridane, and Sambit Bakshi.  Palmprint identification using an ensemble of sparse representations.  IEEE Access, 6:3241–3248, 2018.  DOI: 10.1109/ACCESS.2017.2787666.
  * [3] Sima Shariatmadari, Somaya Al-maadeed, Younes Akbari, Imad Rida, and Sima Emadi.  Off-line persian signature verification using wavelet-based fractal dimension and one-class gaussian process.  In 2018 NASA/ESA Conference on Adaptive Hardware and Systems (AHS), pages 168–173. IEEE, 2018.
  * [4] Lawrence Rabiner and Biing-Hwang Juang.  Fundamentals of Speech Recognition.  Prentice-Hall, Inc., Upper Saddle River, NJ, USA, 1993.
  * [5] Meinard Muller, Daniel PW Ellis, Anssi Klapuri, and Gaël Richard.  Signal processing for music analysis.  IEEE Journal of Selected Topics in Signal Processing, 5(6):1088–1110, 2011.  DOI: 10.1109/JSTSP.2011.2112333.
  * [6] Mirco Rossi, Sebastian Feese, Oliver Amft, Nils Braune, Sandro Martis, and Gerhard Tröster.  Ambientsense: A real-time ambient sound recognition system for smartphones.  In IEEE International Conference on Pervasive Computing and Communications Workshops, 2013, pages 230–235, 2013.  DOI: 10.1109/PerComW.2013.6529487.
  * [7] David Gerhard.  Audio signal classification: History and current techniques.  Technical report, A. TEMKO, C. NADEU / PATTERN RECOGNITION 39 (2006) 682 – 694, 2003.
  * [8] Douglas A Reynolds and Richard C Rose.  Robust text-independent speaker identification using gaussian mixture speaker models.  IEEE Transactions on Speech and Audio Processing, 3(1):72–83, 1995\.  DOI: 10.1109/89.365379.
  * [9] Douglas A Reynolds, Thomas F Quatieri, and Robert B Dunn.  Speaker verification using adapted gaussian mixture models.  Digital Signal Processing, 10(1):19–41, 2000.  DOI: 10.1006/dspr.1999.0361.
  * [10] Douglas A Reynolds.  Speaker identification and verification using gaussian mixture speaker models.  Speech communication, 17(1):91–108, 1995.  DOI:10.1016/0167-6393(95)00009-D.
  * [11] Lawrence R Rabiner.  A tutorial on hidden markov models and selected applications in speech recognition.  Proceedings of the IEEE, 77(2):257–286, 1989.  DOI: 10.1109/5.18626.
  * [12] W Xiong, J Droppo, X Huang, F Seide, M Seltzer, A Stolcke, D Yu, and G Zweig.  Achieving human parity in conversational speech recognition.  arXiv preprint arXiv:1610.05256, 2016.
  * [13] Najim Dehak, Pedro A Torres-Carrasquillo, Douglas A Reynolds, and Reda Dehak.  Language recognition via i-vectors and dimensionality reduction.  In INTERSPEECH, pages 857–860, 2011.
  * [14] Haizhou Li, Bin Ma, and Kong Aik Lee.  Spoken language recognition: from fundamentals to practice.  Proceedings of the IEEE, 101(5):1136–1159, 2013.  DOI: 10.1109/JPROC.2012.2237151.
  * [15] Juan Pablo Bello Correa.  Towards the automated analysis of simple polyphonic music: A knowledge-based approach.  PhD thesis, University of London, 2003.
  * [16] Anssi Klapuri and Manuel Davy.  Signal Processing Methods for Music Transcription.  Springer Publishing Company, Incorporated, 1st edition, 2010.
  * [17] Juan Pablo Bello, Giuliano Monti, Mark Sandler, and Mark S.  Techniques for automatic music transcription.  In in International Symposium on Music Information Retrieval, pages 23–25, 2000.
  * [18] Perfecto Herrera-Boyer, Anssi Klapuri, and Manuel Davy.  Automatic classification of pitched musical instrument sounds.  In Signal Processing Methods for Music Transcription, pages 163–200. Springer, 2006.  DOI: 10.1007/0-387-32845-9_6.
  * [19] Mert Bay and James W Beauchamp.  Multiple-timbre fundamental frequency tracking using an instrument spectrum library.  The Journal of the Acoustical Society of America, 132(3):1886–1886, 2012.  DOI: 10.1121/1.4754916.
  * [20] Juan Pablo Bello, Laurent Daudet, Samer Abdallah, Chris Duxbury, Mike Davies, and Mark B Sandler.  A tutorial on onset detection in music signals.  IEEE Transactions on Speech and Audio Processing, 13(5):1035–1047, 2005.  DOI: 10.1109/TSA.2005.851998.
  * [21] Simon Dixon, Werner Goebl, and Emilios Cambouropoulos.  Perceptual smoothness of tempo in expressively performed music.  Music Perception: An Interdisciplinary Journal, 23(3):195–214, 2006\.  DOI: 10.1525/mp.2006.23.3.195.
  * [22] Takuya Fujishima.  Realtime chord recognition of musical sound: A system using common lisp music.  In Proc. ICMC, volume 1999, pages 464–467, 1999.
  * [23] Kyogu Lee and Malcolm Slaney.  Acoustic chord transcription and key extraction from audio using key-dependent hmms trained on synthesized audio.  IEEE Transactions on Audio, Speech, and Language Processing, 16(2):291–301, 2008.  DOI: 10.1109/TASL.2007.914399.
  * [24] Laurent Oudre, Yves Grenier, and Cédric Févotte.  Template-based chord recognition: Influence of the chord types.  In ISMIR, pages 153–158, 2009.
  * [25] Imad Rida, Romain Herault, and Gilles Gasso.  Supervised music chord recognition.  In 13th IEEE International Conference on Machine Learning and Applications (ICMLA), 2014, pages 336–341, 2014.  DOI: 10.1109/ICMLA.2014.60.
  * [26] DeLiang Wang and Guy J Brown.  Computational auditory scene analysis: Principles, algorithms, and applications.  Wiley-IEEE Press, 2006.
  * [27] Annamaria Mesaros, Toni Heittola, Antti Eronen, and Tuomas Virtanen.  Acoustic event detection in real life recordings.  In 18th European Signal Processing Conference, 2010, pages 1267–1271. IEEE, 2010.
  * [28] Bill Schilit, Norman Adams, and Roy Want.  Context-aware computing applications.  In First Workshop on Mobile Computing Systems and Applications, 1994, pages 85–90. IEEE, 1994.  DOI: 10.1109/MCSA.1994.512740.
  * [29] Yangsheng Xu, Wen Jung Li, and Ka Keung Lee.  Intelligent Wearable Interfaces.  Wiley-Interscience, New York, NY, USA, 2008.
  * [30] Selina Chu, Shrikanth Narayanan, CC Jay Kuo, and Maja J Matarić.  Where am i? scene recognition for mobile robots using audio features.  In IEEE International Conference on Multimedia and Expo, 2006, pages 885–888, 2006.  DOI: 10.1109/ICME.2006.262661.
  * [31] Yuri A Ivanov and Aaron F Bobick.  Recognition of visual activities and interactions by stochastic parsing.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8):852–872, 2000.  DOI: 10.1109/34.868686.
  * [32] François Brémond, Monique Thonnat, and Marcos Zúniga.  Video-understanding framework for automatic behavior recognition.  Behavior Research Methods, 38(3):416–426, 2006.  DOI: 10.3758/BF03192795.
  * [33] Charles J Cohen, Frank Morelli, and Katherine A Scott.  A surveillance system for the recognition of intent within individuals and crowds.  In IEEE Conference on Technologies for Homeland Security, 2008, pages 559–565, 2008.  DOI: 10.1109/THS.2008.4534514.
  * [34] Venkatesh Saligrama, Janusz Konrad, and Pierre-Marc Jodoin.  Video anomaly identification.  IEEE Signal Processing Magazine, 27(5):18–33, 2010.  DOI: 10.1109/MSP.2010.937393.
  * [35] Mei Kuan Lim, Szeling Tang, and Chee Seng Chan.  isurveillance: Intelligent framework for multiple events detection in surveillance videos.  Expert Systems with Applications, 41(10):4704–4715, 2014.  DOI: 10.1016/j.eswa.2014.02.003.
  * [36] Imad Rida, Ahmed Bouridane, Gian Luca Marcialis, and Pierluigi Tuveri.  Improved human gait recognition.  In International Conference on Image Analysis and Processing, pages 119–129. Springer, 2015.  DOI: 10.1007/978-3-319-23234-8_12.
  * [37] James B Hayfron-Acquah, Mark S Nixon, and John N Carter.  Automatic gait recognition by symmetry analysis.  Pattern Recognition Letters, 24(13):2175–2183, 2003.  DOI: 10.1016/S0167-8655(03)00086-2.
  * [38] Imad Rida, Noor Al-maadeed, and Somaya Al-maadeed.  Robust gait recognition: a comprehensive survey.  IET Biometrics, 2018.  DOI: 10.1049/iet-bmt.2018.5063.
  * [39] Imad Rida, Xudong Jiang, and Gian Luca Marcialis.  Human body part selection by group lasso of motion for model-free gait recognition.  IEEE Signal Processing Letters, 23(1):154–158, 2016.  DOI: 10.1109/LSP.2015.2507200.
  * [40] Imad Rida, Somaya Al Maadeed, and Ahmed Bouridane.  Unsupervised feature selection method for improved human gait recognition.  In 23rd European Signal Processing Conference (EUSIPCO), 2015, pages 1128–1132. IEEE, 2015.  DOI: 10.1109/EUSIPCO.2015.7362559.
  * [41] Mark S Nixon, John N Carter, D Cunado, Ping S Huang, and SV Stevenage.  Automatic gait recognition.  In Biometrics, pages 231–249. Springer, 1996.  DOI: 10.1049/ic:19990573.
  * [42] Nikolaos V Boulgouris, Dimitrios Hatzinakos, and Konstantinos N Plataniotis.  Gait recognition: a challenging signal processing technology for biometric identification.  IEEE Signal Processing Magazine, 22(6):78–90, 2005.  DOI: 10.1109/MSP.2005.1550191.
  * [43] Imad Rida, Somaya Almaadeed, and Ahmed Bouridane.  Gait recognition based on modified phase-only correlation.  Signal, Image and Video Processing, 10(3):463–470, 2016.  DOI: 10.1007/s11760-015-0766-4.
  * [44] Imad Rida, Romain Herault, Gian Luca Marcialis, and Gilles Gasso.  Palmprint recognition with an efficient data driven ensemble classifier.  Pattern Recognition Letters, 2018.  DOI: 10.1016/j.patrec.2018.04.033.
  * [45] Imad Rida, Somaya Al Maadeed, Xudong Jiang, Fei Lunke, and Abdelaziz Bensrhair.  An ensemble learning method based on random subspace sampling for palmprint identification.  In IEEE International conference on acoustics, speech and signal processing (ICASSP), 2018, pages 2047–2051. IEEE, 2018.  DOI: 10.1109/ICASSP.2018.8462051.
  * [46] Lunke Fei, Shaohua Teng, Jigang Wu, and Imad Rida.  Enhanced minutiae extraction for high-resolution palmprint recognition.  International Journal of Image and Graphics, 17(04):1750020, 2017\.  DOI: 10.1142/S0219467817500206.
  * [47] Dalibor Mitrović, Matthias Zeppelzauer, and Christian Breiteneder.  Features for content-based audio retrieval.  Advances in Computers, 78:71–150, 2010.  DOI: 10.1016/S0065-2458(10)78003-7.
  * [48] DW Robinson.  The relation between the sone and phon scales of loudness.  Acta Acustica united with Acustica, 3(5):344–358, 1953.
  * [49] Adrianus JM Houtsma.  Pitch and timbre: Definition, meaning and use.  Journal of New Music Research, 26(2):104–115, 1997.  DOI: 10.1080/09298219708570720.
  * [50] David Gerhard.  Pitch extraction and fundamental frequency: History and current techniques.  Regina: Department of Computer Science, University of Regina, 2003.
  * [51] Hualu Wang, Ajay Divakaran, Anthony Vetro, Shih-Fu Chang, and Huifang Sun.  Survey of compressed-domain features used in audio-visual indexing and analysis.  Journal of Visual Communication and Image Representation, 14(2):150–183, 2003.  DOI: 10.1016/S1047-3203(03)00019-1.
  * [52] Benjamin Kedem.  Spectral analysis and discrimination by zero-crossings.  Proceedings of the IEEE, 74(11):1477–1493, 1986.  DOI: 10.1109/PROC.1986.13663.
  * [53] Khaled El-Maleh, Mark Klein, Grace Petrucci, and Peter Kabal.  Speech/music discrimination for multimedia applications.  In IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 6, pages 2445–2448, 2000.  DOI: 10.1109/ICASSP.2000.859336.
  * [54] Doh-Suk Kim, Jae-Hoon Jeong, Jae Weon Kim, and Soo Young Lee.  Feature extraction based on zero-crossings with peak amplitudes for robust speech recognition in noisy environments.  In IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 61–64, 1996.  DOI: 10.1109/ICASSP.1996.540290.
  * [55] Muhammad Ghulam, Takashi Fukuda, Junsei Horikawa, and Tsuneo Nitta.  A noise-robust feature extraction method based on pitch-synchronous zcpa for asr.  In in Proc. ICSLP, pages 133–136, 2004.
  * [56] Dalibor Mitrovic, Matthias Zeppelzauer, and Christian Breiteneder.  Discrimination and retrieval of animal sounds.  In 12th IEEE International Multi-Media Modelling Conference, pages 5–pp, 2006.  DOI: 10.1109/MMMC.2006.1651344.
  * [57] Tong Zhang and CC Jay Kuo.  Content-based audio classification and retrieval for audiovisual data parsing, volume 606.  Springer Science & Business Media, 2013.  DOI: 10.1007/978-1-4757-3339-6.
  * [58] Hongchen Jiang, Junmei Bai, Shuwu Zhang, and Bo Xu.  Svm-based audio scene classification.  In IEEE International Conference on Natural Language Processing and Knowledge Engineering, 2005, pages 131–136, 2005.  DOI: 10.1109/NLPKE.2005.1598721.
  * [59] Lawrence R Rabiner and Ronald W Schafer.  Digital processing of speech signals.  Prentice Hall, 1978.
  * [60] Joseph P Campbell Jr.  Speaker recognition: a tutorial.  Proceedings of the IEEE, 85(9):1437–1462, 1997.  DOI: 10.1109/5.628714.
  * [61] Stephane Mallat.  A wavelet tour of signal processing: the sparse way.  Academic press, 2008.
  * [62] Mingchun Liu and Chunru Wan.  A study on content-based classification and retrieval of audio database.  In IEEE International Symposium on Database Engineering and Applications, 2001., pages 339–345, 2001.  DOI: 10.1109/IDEAS.2001.938102.
  * [63] Eric Scheirer and Malcoh Slaney.  Construction and evaluation of a robust multifeature speech/music discriminator.  In IEEE International Conference on Acoustics, Speech, and Signal Processing, 1997., volume 2, pages 1331–1334, 1997.  DOI: 10.1109/ICASSP.1997.596192.
  * [64] Fabian Mörchen, Alfred Ultsch, Michael Thies, and Ingo Löhken.  Modeling timbre distance with temporal statistics from polyphonic music.  IEEE Transactions on Audio, Speech, and Language Processing, 14(1):81–90, 2006.  DOI: 10.1109/TSA.2005.860352.
  * [65] Avery Wang et al.  An industrial strength audio search algorithm.  In ISMIR, pages 7–13, 2003.
  * [66] Tao Li and George Tzanetakis.  Factors in automatic musical genre classification of audio signals.  In IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, 2003., pages 143–146, 2003.  DOI: 10.1109/ASPAA.2003.1285840.
  * [67] Jürgen Herre, Eric Allamanche, and Chris Ertel.  How similar do songs sound? towards modeling human perception of musical similarity.  In Applications of Signal Processing to Audio and Acoustics, 2003 IEEE Workshop on., pages 83–86. IEEE, 2003.  DOI: 10.1109/ASPAA.2003.1285825.
  * [68] William A Sethares, Robin D Morris, and James C Sethares.  Beat tracking of musical performances using low-level audio features.  IEEE Transactions on Speech and Audio Processing, 13(2):275–285, 2005.  DOI: 10.1109/TSA.2004.841053.
  * [69] J.D. Gibson.  Digital coding of waveform: Principles and applications to speech and video.  Proceedings of the IEEE, pages 526–527, 1984.  DOI: 10.1109/PROC.1987.13765.
  * [70] R Lienbart, Silvia Pfeiffer, and Wolfgang Effelsberg.  Scene determination based on video and audio features.  In IEEE International Conference on Multimedia Computing and Systems, 1999., volume 1, pages 685–690. IEEE, 1999.  DOI: 10.1109/MMCS.1999.779282.
  * [71] George Tzanetakis and Perry Cook.  Musical genre classification of audio signals.  IEEE Transactions on Speech and Audio Processing, 10(5):293–302, 2002.  DOI: 10.1109/TSA.2002.800560.
  * [72] Ray Meddis and Lowel Mard.  A unitary model of pitch perception.  The Journal of the Acoustical Society of America, 102(3):1811–1820, 1997.  DOI: 10.1121/1.420088.
  * [73] Giulio Agostini, Maurizio Longari, and Emanuele Pollastri.  Musical instrument timbres classification with spectral features.  EURASIP Journal on Applied Signal Processing, 2003:5–14, 2003.  DOI: 10.1155/S1110865703210118.
  * [74] Kuo-Hwei Yuo, Tai-Hwei Hwang, and Hsiao-Chuan Wang.  Combination of autocorrelation-based features and projection measure technique for speaker identification.  IEEE Transactions on Speech and Audio Processing, 13(4):565–574, 2005.  DOI: 10.1109/TSA.2005.848893.
  * [75] Sourabh Ravindran, Kristopher Schlemmer, and David V Anderson.  A physiologically inspired method for audio classification.  EURASIP Journal on Advances in Signal Processing, 2005(9):1–8, 2005\.  DOI: 10.1155/ASP.2005.1374.
  * [76] Hynek Hermansky.  Perceptual linear predictive (plp) analysis of speech.  the Journal of the Acoustical Society of America, 87(4):1738–1752, 1990.  DOI: 10.1121/1.399423.
  * [77] Hynek Hermansky and Nelson Morgan.  Rasta processing of speech.  IEEE Transactions on Speech and Audio Processing, 2(4):578–589, 1994.  DOI: 10.1109/89.326616.
  * [78] Bishnu S Atal.  Effectiveness of linear prediction characteristics of the speech wave for automatic speaker identification and verification.  the Journal of the Acoustical Society of America, 55(6):1304–1312, 1974.  DOI: 110.1121/1.1914702.
  * [79] Jonathan Foote.  Automatic audio segmentation using a measure of audio novelty.  In IEEE International Conference on Multimedia and Expo, 2000., volume 1, pages 452–455, 2000.  DOI: 10.1109/ICME.2000.869637.
  * [80] Alain Rakotomamonjy and Gilles Gasso.  Histogram of gradients of time-frequency representations for audio scene classification.  IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 23(1):142–153, 2015.  DOI: 10.1109/TASLP.2014.2375575.
  * [81] Imad Rida, Noor Al Maadeed, and Somaya Al Maadeed.  A novel efficient classwise sparse and collaborative representation for holistic palmprint recognition.  In 2018 NASA/ESA Conference on Adaptive Hardware and Systems (AHS), pages 156–161. IEEE, 2018.
  * [82] Aaron E Bobick and Amos Y Johnson.  Gait recognition using static, activity-specific parameters.  In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001., volume 1, pages I–423, 2001.  DOI: 10.1109/CVPR.2001.990506.
  * [83] Eli Shechtman and Michal Irani.  Space-time behavior based correlation.  In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005., volume 1, pages 405–412, 2005.  DOI: 10.1109/CVPR.2005.328.
  * [84] Yan Ke, Rahul Sukthankar, and Martial Hebert.  Spatio-temporal shape and flow correlation for action recognition.  In IEEE Conference on Computer Vision and Pattern Recognition, 2007., pages 1–8, 2007.  DOI: 10.1109/CVPR.2007.383512.
  * [85] Jake K Aggarwal and Michael S Ryoo.  Human activity analysis: A review.  ACM Computing Surveys (CSUR), 43(3):16, 2011.  DOI: 10.1145/1922649.1922653.
  * [86] Sourabh A Niyogi and Edward H Adelson.  Analyzing and recognizing walking figures in xyt.  In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1994., pages 469–474, 1994.  DOI: 10.1109/CVPR.1994.323868.
  * [87] Cen Rao and Mubarak Shah.  View-invariance in action recognition.  In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001., volume 2, pages II–316, 2001.  DOI: 10.1109/CVPR.2001.990977.
  * [88] A Yilma and Mubarak Shah.  Recognizing human actions in videos acquired by uncalibrated moving cameras.  In IEEE International Conference on Computer Vision, 2005., volume 1, pages 150–157, 2005.  DOI: 10.1109/ICCV.2005.201.
  * [89] Yaser Sheikh, Mumtaz Sheikh, and Mubarak Shah.  Exploring the space of a human action.  In IEEE International Conference on Computer Vision, 2005., volume 1, pages 144–149, 2005.  DOI: 10.1109/ICCV.2005.90.
  * [90] Olivier Chomat and James L Crowley.  Probabilistic recognition of activity using local appearance.  In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1999., volume 2, pages 104–109, 1999.  DOI: 10.1109/CVPR.1999.784616.
  * [91] Lihi Zelnik-Manor and Michal Irani.  Event-based analysis of video.  In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001., volume 2, pages II–123, 2001.  DOI: 10.1109/CVPR.2001.990935.
  * [92] Moshe Blank, Lena Gorelick, Eli Shechtman, Michal Irani, and Ronen Basri.  Actions as space-time shapes.  In IEEE International Conference on Computer Vision, 2005., volume 2, pages 1395–1402, 2005.  DOI: 10.1109/ICCV.2005.28.
  * [93] Ivan Laptev and Tony Lindeberg.  Space-time interest points.  In IEEE International Conference on Computer Vision, 2003., pages 432–439. IEEE, 2003.
  * [94] Piotr Dollár, Vincent Rabaud, Garrison Cottrell, and Serge Belongie.  Behavior recognition via sparse spatio-temporal features.  In 2nd Joint IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, 2005., pages 65–72, 2005.  DOI: 10.1109/VSPETS.2005.1570899.
  * [95] Juan Carlos Niebles, Hongcheng Wang, and Li Fei-Fei.  Unsupervised learning of human action categories using spatial-temporal words.  International journal of computer vision, 79(3):299–318, 2008.  DOI: 10.1007/s11263-007-0122-4.
  * [96] Imad Rida, Noor Al Maadeed, Gian Luca Marcialis, Ahmed Bouridane, Romain Herault, and Gilles Gasso.  Improved model-free gait recognition based on human body part.  In Biometric Security and Privacy, pages 141–161. Springer, 2017\.  DOI: 10.1007/978-3-319-47301-7_6.
  * [97] Pavan Turaga, Rama Chellappa, Venkatramana S Subrahmanian, and Octavian Udrea.  Machine recognition of human activities: A survey.  IEEE Transactions on Circuits and Systems for Video Technology, 18(11):1473–1488, 2008.  DOI: 10.1109/TCSVT.2008.2005594.
  * [98] Grégory Rogez, José Jesús Guerrero, and Carlos Orrite.  View-invariant human feature extraction for video-surveillance applications.  In IEEE Conference on Advanced Video and Signal Based Surveillance, 2007., pages 324–329, 2007.  DOI: 10.1109/AVSS.2007.4425331.
  * [99] Suman Sedai, Mohammed Bennamoun, and D Huynh.  Context-based appearance descriptor for 3d human pose estimation from monocular images.  In Digital Image Computing: Techniques and Applications, 2009., pages 484–491. IEEE, 2009.  DOI: 10.1109/DICTA.2009.81.
  * [100] Imad Rida, Somaya Almaadeed, and Ahmed Bouridane.  Improved gait recognition based on gait energy images.  In 26th IEEE International Conference on Microelectronics (ICM), 2014, pages 40–43. IEEE, 2014.  DOI: 10.1109/ICM.2014.7071801.
  * [101] Imad Rida, Larbi Boubchir, Noor Al-Maadeed, Somaya Al-Maadeed, and Ahmed Bouridane.  Robust model-free gait recognition by statistical dependency feature selection and globality-locality preserving projections.  In 39th IEEE International Conference on Telecommunications and Signal Processing (TSP), 2016, pages 652–655, 2016.  DOI: 10.1109/TSP.2016.7760963.
  * [102] Imad Rida, Noor Al-Maadeed, Somaya Al-Maadeed, and Sambit Bakshi.  A comprehensive overview of feature representation for biometric recognition.  Multimedia Tools and Applications, pages 1–24, 2018.  DOI: 10.1007/s11042-018-6808-5.