# Towards the One Learning Algorithm Hypothesis: A System-theoretic Approach

Christos N. Mavridis, , and John S. Baras The authors are with the Department
of Electrical and Computer Engineering and the Institute for Systems Research,
University of Maryland, College Park, USA. emails:{mavridis,
<EMAIL_ADDRESS>partially supported by the Defense Advanced Research
Projects Agency (DARPA) under Agreement No. HR00111990027, by ONR grant
N00014-17-1-2622, and by a grant from Northrop Grumman Corporation.

###### Abstract

The existence of a universal learning architecture in human cognition is a
widely spread conjecture supported by experimental findings from neuroscience.
While no low-level implementation can be specified yet, an abstract outline of
human perception and learning is believed to entail three basic properties:
(a) hierarchical attention and processing, (b) memory-based knowledge
representation, and (c) progressive learning and knowledge compaction. We
approach the design of such a learning architecture from a system-theoretic
viewpoint, developing a closed-loop system with three main components: (i) a
multi-resolution analysis pre-processor, (ii) a group-invariant feature
extractor, and (iii) a progressive knowledge-based learning module. Multi-
resolution feedback loops are used for learning, i.e., for adapting the system
parameters to online observations. To design (i) and (ii), we build upon the
established theory of wavelet-based multi-resolution analysis and the
properties of group convolution operators. Regarding (iii), we introduce a
novel learning algorithm that constructs progressively growing knowledge
representations in multiple resolutions. The proposed algorithm is an
extension of the Online Deterministic Annealing (ODA) algorithm which can be
viewed as a competitive-learning neural network where the neurons represent
codevectors in the data space and the training rule is based on annealing
optimization, solved using gradient-free stochastic approximation. ODA has
inherent robustness and regularization properties and provides a means to
progressively increase the complexity of the learning model i.e. the number of
the neurons, as needed, through an intuitive bifurcation phenomenon. The
proposed multi-resolution approach is hierarchical, progressive, knowledge-
based, and interpretable, allowing the localization of the error-prone regions
of the data space. We illustrate the properties of the proposed architecture
in the context of the state-of-the-art learning algorithms and deep learning
methods.

###### Index Terms:

Hierarchical Learning, Progressive Learning, Annealing Optimization, Online
Deterministic Annealing, Multi-resoltuion Analysis, One Learning Algorithm
Hypothesis.

##  I Introduction

Learning from data samples has become an important component in the
advancement of numerous research fields including artificial intelligence,
computational physics, biology and control systems. While virtually all
learning problems can be formulated as constrained stochastic optimization
problems, the optimization methods can be intractable due to the nature of the
problems and the constraints originating from the model assumptions. For this
reason, some of the most impactful breakthroughs in machine learning research
have been a result of selecting appropriately structured learning models that
can be trained efficiently using existing optimization methods [1].

In search for appropriate learning models and inspired by established
experimental findings from neuroscience [2, 3], a widely spread conjecture has
been formed regarding the existence of a universal learning model in human
cognition. While no observations can yet lead to the specifics of the
implementation of such an architecture, at an abstract level it is believed
that human perception, cognition, and learning, are based on three main
properties: (a) hierarchical attention and information processing [4, 5, 3, 6,
7, 8], (b) memory-based knowledge representation [9], and (c) progressive
learning and knowledge compaction [10, 11].

Deep learning methods, currently dominating the field of machine learning due
to their performance in numerous applications, have made progress towards this
direction, especially regarding the hierarchical properties (a). They attempt
to learn feature representations from data, using biologically-inspired models
in artificial neural networks that mimic the low-level computational
architecture of the human brain units [11, 12, 13, 14]. They provide task-
specific end-to-end learning algorithms implemented by a non-von Neumann
computational architecture that does not discriminate between computational
and memory units. As a result, however, they do not provide a framework for
memory representation and knowledge compaction as in (b) and (c).

Figure 1: Abstract system-theoretic block diagram of a “universal learning
architecture” in human cognition.

On the other hand, it is understood that artificial intelligence systems need
not mimic the low-level architecture of the brain cells, but rather get
inspirations from abstract properties of human intelligence [15]. This becomes
especially important when considering that adopting black-box deep neural
network architectures results in using overly complex models of a great many
parameters in the expense of time, energy, data, memory and computational
resources [16, 17], Even in the applications when complexity is not an issue,
the lack of interpretability and mathematical understanding, and the
vulnerability to small perturbations and adversarial attacks [18, 19, 20], has
led to an emerging hesitation in the use of deep learning models outside
common benchmark datasets [21, 22], and, especially, in security critical
applications. These models are hard to analyze with current mathematical
tools, hard to train with current optimization methods, and their design
relies solely in experimental heuristics. As a result, researchers have been
trying to answer questions about their approximation power, the dynamics and
convergence of optimization, the properties of the intrinsically generated
representations, and the phenomena of overfitting, regularization, and
overparameterization, in an attempt to understand the conditions under which
deep learning methods work as expected [23, 24, 25, 10, 26].

To this end, we approach the design of a “universal learning architecture”
[27, 28] from a system-theoretic viewpoint, based on the aforementioned
architectural abstractions of the processing system of the auditory and visual
cortex of the human [4, 5, 3, 6, 7, 8]. The architecture consists of a closed-
loop system with three main components:

  * $(i)$

a hierarchical signal decomposition unit based on multi-resolution wavelet
analysis [24],

  * $(ii)$

a feature extraction module that, in combination with $(i)$, estimates group-
invariant representations in multiple resolutions, and

  * $(iii)$

a hierarchical, progressive, robust, and interpretable learning model for
supervised and unsupervised learning that is able to represent and
progressively compact accumulated knowledge.

Multi-resolution feedback loops are used for adaptive estimation of the
optimal parameters for $(i)$, $(ii)$, and $(iii)$. An abstract block diagram
is depicted in Fig. 1.

Regarding the subsystems $(i)$ and $(ii)$, we follow the established theory of
wavelet decomposition to build a multi-resolution representation of the input.
Wavelet decomposition is based on a cascade of convolution and downsampling
operators. Convolutions create translation equivariant representations which
are stable to deformations, and can be generalized to carry these properties
to any known compact Lie group [29]. Downsampling, on the other hand,
constitutes an averaging operation that creates local invariance, and can be
viewed as a pooling operation. Finally, creating a hierarchical cascade of
convolution operations followed by non-linear mappings (subsystem $(ii)$) and
averaging functions, has been shown to construct multi-scale group-invariant
representations [24].

Regarding $(iii)$, we build upon the Online Deterministic Annealing (ODA)
algorithm [30], a novel learning algorithm that constructs progressively
growing knowledge representations. ODA can be formulated as a series of soft-
clustering optimization problems

$\displaystyle\min_{M}~{}F_{T}(M):=D(M)-TH(M),$

parameterized by a temperature coefficient $T$. Here $D$ is the average
distortion $D(M):=\mathbb{E}\left[\sum_{i}p(\mu_{i}|X)d(X,\mu_{i})\right]$
measured by $d$, the proximity measure that defines the similarity between the
random input $X$ and a codevector $\mu_{i}$ with appropriately defined
association probabilities $p(\mu_{i}|X)$. $H$ represents the Shannon entropy
that characterizes the “purity” of the clusters induced by the codevectors,
and $T$ acts as a Lagrange multiplier controlling the trade-off between
minimizing the distortion $D$ and maximizing the entropy $H$. The codevectors
can be viewed as a set of neurons, the weights of which live in the data space
itself. The representation of the input in terms of memorized exemplars is an
intuitive approach which parallels similar concepts from cognitive psychology
and neuroscience and induces interpretability. Moreover, it resembles vector
quantization algorithms, which have shown impressive robustness against
adversarial attacks, suggesting suitability in security critical applications
[31]. By successively solving the optimization problems $\min_{M}~{}F_{T}(M)$
for decreasing values of $T$, the model undergoes a series of phase
transitions when the cardinality of the set of codevectors $M$ increases,
according to an intuitive bifurcation phenomenon. This resembles an annealing
process [32] that introduces inherent robustness and regularization
properties. The optimization problems are solved using gradient-free
stochastic approximation [33], provided that the proximity measure $d$ belongs
to the family of Bregman divergences: information-theoretic dissimilarity
measures (including the widely used Kullback-Leibler divergence) that have
been shown to play an important role in learning applications [34, 35].

Finally, to construct a hierarchical multi-resolution learning approach for
$(iii)$, we develop a tree-structured ODA learning algorithm. Traditional
tree-structured vector quantization algorithms partition the data space by
building a labeled binary tree using ad-hoc splitting criteria. This yields a
faster, yet suboptimal solution to the original problem. In contrast, we train
each layer of the tree with the ODA algorithm that progressively adjusts the
number of codevectors-children for each node according to the annealing
optimization process. Moreover, to make use of multiple-resolutions, we extend
this approach having each layer trained using progressively more detailed
representations according to the multi-resolution analysis of $(i)$ and
$(ii)$. As will be shown, the combination of the convolution-based multi-
resolution features with the proposed tree-structured multi-layer neural
network resembles the structure of Deep Convolutional Networks [11] and
Scattering Convolutional Networks [36], and provides a general system-
theoretic framework for progressive, hierarchical, interpretable, and
knowledge-based learning.

##  II Hierarchical and Invariant Representations

To understand how deep learning models work, it is important to mathematically
characterize the properties of the internally learned representations [37, 38,
39, 13] and its inductive bias, i.e., the class of tasks for which they are
predesigned to perform well. As an example, in computer vision tasks,
geometric stability has been shown to play an important role and can be
measured in terms of invariance to translations and stability to local
deformations, as will be explained later. Deep convolutional neural networks
have been shown to provide this geometrical stability [25] by following a
compositional approach with a hierarchically structured model [11, 40]. The
structure is mainly composed by successive operations of a linear operator (or
linear filter, e.g., convolutional layers), a nonlinear mapping (often a
rectifying function, e.g., ReLu), and a down-sampling step (e.g., max-
pooling), which produces the input for the next stage of the architecture [36,
24, 41]. This approach seems to reflect neuro-scientific findings [4, 40, 42],
and is also reminiscent of the multi-resolution representations produced by
the extensively studied wavelet transform. The wavelet transform [43], as well
as extensions including the wavelet packets [44], offer a solid mathematical
tool to understand the properties of the multi-resolution representation of a
signal. As a result, wavelets have been already used to study the mathematics
of deep learning, with the scattering transform [36, 24] having been shown to
preserve the aforementioned geometric stability properties, hinting that
convolution filters need not be learned from data, but can be defined a priori
with respect to the properties of the task at hand.

In this section, we briefly review the mathematics of the wavelet and wavelet
packet transform and show how they can be used to build multi-resolution
locally invariant representations with respect to compact Lie groups.

###  II-A Wavelet-based Multi-Resolution Analysis

Consider a measurable signal $f(t)$ with finite energy, i.e., $f(t)\in
L^{2}(\mathbb{R}^{n})$. We will focus in one-dimensional signals ($n=1$) for
simplicity of notation, although the analysis can be generalized for any
$n>0$. The continuous wavelet transform of $f$ corresponds to the set of
coeeficients of the atoms of a redundant dictionary of time-scale images (in
$(u,s)$):

$\left\\{\psi_{u,s}(t)=\frac{1}{\sqrt{s}}\psi(\frac{t-u}{s})\right\\}_{u\in\mathbb{R},s>0}$
(1)

where $\psi\in L^{2}(\mathbb{R})$ has zero average, i.e.,
$\int_{-\infty}^{\infty}\psi(t)dt=0$, and is called the mother wavelet. In
many useful cases, such as in directional wavelets for 2D images, the mother
wavelets are defined such that the redundant dictionaries constitute a
complete and stable signal representation, called a frame [44]. For simplicity
of notation, we will focus on orthonormal (Riesz) bases which remove all
redundancy and define stable representations. A wavelet orthonormal basis is
obtained by scaling a wavelet $\psi$ with a dyadic scale $s=2^{j}$ and
translating it by $u=2^{j}n$:

$\left\\{\psi_{j,n}(t)=\frac{1}{\sqrt{2^{j}}}\psi(\frac{t-2^{j}n}{2^{j}})\right\\}_{(j,n)\in\mathbb{Z}^{2}}$
(2)

The simplest orthonormal basis can be constructed using the piecewise constant
Haar wavelets, but regular wavelets of compact support can form an orthonormal
basis as well [44]. The signal $f$ can be expressed in the wavelet bases as:

$f=\sum_{j=-\infty}^{\infty}\sum_{n=-\infty}^{\infty}c_{j,n}\psi_{j,n}$ (3)

where $c_{j,n}=\langle f,\psi_{j,n}\rangle$. Moreover, $f$ can be expressed as
a direct sum:

$\displaystyle f$
$\displaystyle=\sum_{j>J}\sum_{n=-\infty}^{\infty}c_{j,n}\psi_{j,n}+\sum_{j\leq
J}\sum_{n=-\infty}^{\infty}c_{j,n}\psi_{j,n}$ (4)
$\displaystyle=\sum_{n=-\infty}^{\infty}a_{J,n}\phi_{J,n}+\sum_{j\leq
J}^{\infty}\sum_{n=-\infty}^{\infty}c_{j,n}\psi_{j,n}$ (5)

where $\phi_{j,n}$ is called the scaling function and is defined similar to
(2) according to a mother scaling function $\phi$. The set
$\left\\{\phi_{J,n}\right\\}$ is an orthonormal basis of the subspace
$V_{J}\subset L^{2}(\mathbb{R})$ which corresponds to the approximation of
$L^{2}(\mathbb{R})$ at resolution $J$, and has the following properties:

  * •

$V_{j}\subset V_{j-1}$, $\forall j\in\mathbb{Z}$,

  * •

$\cup_{j=-\infty}^{\infty}V_{j}$ is dense in $L^{2}(\mathbb{R})$ with
$\cap_{j=-\infty}^{\infty}V_{j}=\left\\{0\right\\}$,

  * •

$f(t)\in V_{j}\Leftrightarrow f(2t)\in V_{j-1}$, $\forall j\in\mathbb{Z}$, and

  * •

$f(t)\in V_{j}\implies f(t-2^{j}k)\in V_{j}$, $\forall k\in\mathbb{Z}$.

It is easy to see that the sum $\sum_{n=-\infty}^{\infty}c_{j,n}\psi_{j,n}$ is
an orthonormal basis for the subspace $W_{j}$ for which $V_{j}\oplus
W_{j}=V_{j-1}$ and $V_{J}=\oplus_{j>J}W_{j}$. In other words, the additional
information lost in the approximation $V_{J}$ at resolution $2^{J}$ is
contained in the subspace $W_{J}$.

In practice, a physical measurement device can only measure a signal at a
finite resolution, denoted by $j=0$. Then the coefficients
$\left\\{c_{j,n}\right\\}_{j\in\mathbb{N}}$ and
$\left\\{a_{j,n}\right\\}_{j\in\mathbb{N}}$ define the multi-resolution
representation of the wavelet transform. Moreover, the coefficients

$\displaystyle c_{j,n}=\langle
f,\psi_{j,n}\rangle=\int_{-\infty}^{\infty}f(t)\psi_{j,n}(t)dt$ (6)
$\displaystyle a_{j,n}=\langle
f,\phi_{j,n}\rangle=\int_{-\infty}^{\infty}f(t)\psi_{j,n}(t)dt$

can also be written as:

$\displaystyle\langle f,\psi_{j,n}\rangle$
$\displaystyle=(f\ast\psi_{2^{j}})(t)|_{t=2^{j}n}:=W_{j}f(t)|_{t=2^{j}n}$ (7)
$\displaystyle\langle f,\phi_{j,n}\rangle$
$\displaystyle=(f\ast\phi_{2^{j}})(t)|_{t=2^{j}n}$

where $\psi_{2^{j}}(x)=2^{-j/2}\psi(-2^{-j}x)$,
$\phi_{2^{j}}(x)=2^{-j/2}\phi(-2^{-j}x)$, and $(f\ast
g)(x)=\int_{-\infty}^{\infty}f(u)g(x-u)du$ is the convolution operation.
Taking into account that $\psi_{2^{j}},\phi_{2^{j}}\in V_{j-1}$, the
coefficients can be computed by:

$\displaystyle\langle f,\psi_{j,n}\rangle$ $\displaystyle=(g\ast\langle
f,\phi_{2^{j-1}}\rangle)(t)|_{t=2n}$ (8) $\displaystyle\langle
f,\phi_{j,n}\rangle$ $\displaystyle=(h\ast\langle
f,\phi_{2^{j-1}}\rangle)(t)|_{t=2n}$

Equation (8) reveals that the wavelet coefficients $\langle
f,\psi_{j,n}\rangle$, $\langle f,\psi_{j,n}\rangle$ can be computed by
hierarchical filtering with appropriately defined filters associated with the
mother wavelet $\psi$, followed by a uniform downsampling by a factor of $2$.
In particular, $h$ is a low-pass filter and $g$ is a high-pass filter. They
depend on $\phi$ and $\psi$, and define a pair of quadrature mirror filters,
an observation that has led to the fast filter-bank implementation of the
wavelet transform and its association with multi-resolution analysis [43].

###  II-B Wavelets, Translation Invariance, and Stability to Deformations

As shown above, the computation of the multi-resolution wavelet representation
of a signal consists of successive operations of a linear convolution
operator, followed by a downsampling step. This structure is similar to deep
convolutional neural networks, and has been very successful in multiple
applications, especially in sound and image recognition, where translation
invariance and stability to small deformations are important properties of the
representation [24]. The translation group is a Lie group of operators
$\left\\{T_{c}\right\\}_{c\in\mathbb{R}}$ such that $T_{c}f(t)=f(t-c)$, while
the group of deformations $D_{\tau}$ can be written as
$D_{\tau}f(t)=f(t-\tau(t))$, where the local function $\tau$ depends on $t$
and thus deforms the image. A representation $\Phi:L^{2}(\mathbb{R}\rightarrow
H$, where $H$ is a Hilbert space, is said to be invariant with respect to
$T_{c}$ if $\Phi(T_{c}f)=\Phi(f)$. The wavelet transform (eq. (7)) is based on
the convolution operation, and is therefore translation covariant (or
equivariant), i.e., it commutes with the translation operator and satisfies
the relation $W_{j}(T_{c}f)=T_{c}W_{j}(f)$. At the same time, the wavelet
transform, is stable to small deformations, which is a very important property
that can be expressed as a Lipschitz continuity condition:

$\|W_{j}(D_{\tau}f)-W_{j}(f)\|_{2}\leq C\|f\|_{2}\sup_{t}|\nabla\tau(t)|$ (9)

where $\|f\|_{2}=\int|f(t)|^{2}dt$ is the $L^{2}(\mathbb{R})$ norm, and
$|\nabla\tau(t)|$ is a measure of the deformation amplitude (it is assumed
that $|\nabla\tau(t)|<1$, which makes the small deformation an invertible
transformation [36]).

######  Remark 1.

We note that, in the control theory and signal processing communities,
convolutions are associated with systems described by the term ’linear time-
invariant’. To avoid confusion, with the terminology used here, these systems
are considered linear covariant (or equivariant) operators with respect to
translation in time.

To induce invariance with respect to translation, it is sufficient to
integrate (average) a translation-covariant representation over the
translation group. That is to say that the integral $\int W_{j}f(c)dc$ is
translation invariant. However, it turns out to be the trivial invariant
$\int_{-\infty}^{\infty}W_{j}f(t-c)dc=0$, since, by definition,
$\int_{-\infty}^{\infty}\psi(t)dt=0$. Therefore, it is necessary to insert a
non-linear operation between the wavelet transform and the integration [36].
It has been shown that a point-wise and stable non-linear operator $\rho$ (for
stability: $\|\rho f\|\leq\|f\|$ and $\|\rho f-\rho g\|\leq\|f-g\|$), that
also commutes with the translation (and the deformation) group, is the
$L^{1}(\mathbb{R})$ norm: $\|f\|_{1}=\int|f(t)|dt$, which acts as a rectifying
function, similar to the ReLu function in deep convolutional neural networks.
The choice $\rho=\|\cdot\|_{1}$ also preserves the deformation stability
properties of the wavelet transform.

The cascade of the wavelet transform, the non-linear operation, and the
averaging integral, results in the representation $\int\rho W_{j}(f)(t)dt$, in
every resolution $j$. However, global invariance is not always desired. In
contrast, it is often better to compute locally invariant representations for
translations up to a scale $2^{J}$ for some index $J$. This can be obtained by
substituting the integral by a convolution with a low-pass filter localized in
a spatial window scaled at $2^{J}$. The result is a representation that
belongs to $V_{J}$, and, in particular the projection of $\rho W_{j}(f)$ to
$V_{J}$, for $J\geq j$, which can be computed by the pyramid of filters of the
wavelet transform, as explained above. Moreover, during the computation of the
wavelet transform, the information lost in averaging (projecting to $V_{J}$)
is kept in the coefficients $\left\\{c_{j,n}\right\\}_{j<J}$ of the wavelet
transform. This information can be used to build new invariant features by
applying the wavelet transform again, creating a deep scattering network
introduced by Bruna in [36]. The implementation of the scattering transform is
based on a complex-valued convolutional neural network whose filters are fixed
wavelets and $\rho$ is a complex modulus operator as described above [45]. We
note that the features of the scattering transform do not belong to the
subspaces $\left\\{V_{j},j\in\mathbb{N}\right\\}$ that define the classical
wavelet hierarchy. However, they define a multi-level hierarchy of
successively more detailed information that can be used in a similar way.

###  II-C Building Invariance to compact Lie Groups

The invariance properties discussed above can be generalized to the action of
arbitrary compact Lie groups, such as a rotation and translation group [29].
Let $G$ be a compact Lie group and $L^{2}(G)$ be the space of measurable
functions $f(r)$ such that $\|f\|^{2}=\int_{G}|f(r)|^{2}dr<\infty$, where $dr$
is the Haar measure of $G$. The left action of $g\in G$ on $f\in L^{2}(G)$ is
defined by $L_{g}f(r)=f(g^{-1}r)$. As a special case, the action of the
translation group $T_{c}f(t)=f(t-c)$ translates the function $f$ to the right
by $c$, with $g^{-1}=-c$ translating the argument of $f$ to the left by $c$.
Similar to the usual convolution $(f\ast
h)(x)=\int_{-\infty}^{\infty}f(u)h(x-u)du$ that defines a linear translation
covariant operator, convolutions on a group appear naturally as linear
operators covariant to the action of a group:

$(f\ast h)(x)=\int_{G}f(g)h(g^{-1}x)dr$ (10)

where $dr$ is the Haar measure of $G$. As a result, an invariant
representation relatively to the action of a compact Lie group, can be
computed by averaging over covariant representations created by group
convolution with appropriately defined wavelets, similar to the methodology
explained above.

###  II-D Multi-resolution Feedback Loops

The methodology explained in Section II-C can be used to build invariant
features with respect to a known compact Lie Group. In the case of image and
sound recognition, it has been shown that local invariance with respect to the
groups of spatial translation and rotation is sufficient [24]. In other words,
the groups with respect to which we design the wavelet-based features
described above, are known a priori. In general, such information may not be
known, and may need to be estimated from the data. This is the purpose of the
multi-resolution feedback loops presented in Fig. 1, and specifically, the
first set of feedback loops shown in Fig. 3(a). The wavelet transform used can
be generalized to a redundant hierarchical dictionary, as well, the basis of
which can be estimated again from data via the same feedback loops. In the
case of a wavelet transform this reduces to estimating the mother wavelet
[46]. To estimate different non-linear functions $\rho$, the second set of
feedback loops can also be used. Therefore, the multi-resolution feedback
loops make the proposed approach a general framework for task-agnostic
learning. Estimating this non-linear mapping or the mother wavelet from data
is beyond the scope of this paper.

##  III Online Deterministic Annealing for Unsupervised and Supervised
Learning

To build a progressive learning algorithm with the properties mentioned above,
we will start our analysis with unsupervised learning, which can provide
valuable insights into the nature of the dataset at hand, and it plays an
important role in the context of visualization. Central to unsupervised
learning is the representation of data in a vector space by typical
representatives which is implemented with vector quantization variants [47].
Given a random variable $X:\Omega\rightarrow S$ defined in the probability
space $\left(\Omega,\mathcal{F},\mathbb{P}\right)$, a quantizer
$Q:S\rightarrow S$ is defined such that
$Q(X)=\sum_{h=1}^{K}\mu_{h}\mathds{1}_{\left[X\in S_{h}\right]}$, where
$V:=\left\\{S_{h}\right\\}_{h=1}^{K}$ forms a partition of $S$ and
$M:=\left\\{\mu_{h}\right\\}_{h=1}^{K}$ represents a set of codevectors such
that $\mu_{h}\in ri(S_{h})$, $h\in\left\\{1,\ldots,K\right\\}$. Given a
dissimilarity measure $d:S\times ri(S)\rightarrow\left[0,\infty\right)$ one
seeks the optimal $M,V$ in terms of minimum average distortion:

$\min_{M,V}~{}D(Q):=\mathbb{E}\left[d\left(X,Q(X)\right)\right]$ (11)

Vector quantization algorithms assume that $Q$ is a deterministic function of
$X$ and are proven to converge to locally optimal configurations even when
formulated as online learning algorithms [47]. However, their convergence
properties and final configuration depend heavily on two design parameters:
$(a)$ the number of clusters (neurons), and $(b)$ their initial configuration.
To deal with this phenomenon, the Online Deterministic Annealing approach [30]
makes use of a probabilistic framework, where input vectors are assigned to
clusters in probability, thus dropping the assumption that $Q$ is a
deterministic function of $X$. For the randomized partition, the expected
distortion becomes:

$\displaystyle
D=\mathbb{E}\left[d_{\phi}(X,Q)\right]=\mathbb{E}\left[\mathbb{E}\left[d_{\phi}(X,Q)|X\right]\right]$

The central idea of deterministic annealing is to seek the distribution that
minimizes $D$ subject to a specified level of randomness, measured by the
Shannon entropy

$\displaystyle H(X,M)=H(X)-\mathbb{E}\left[\mathbb{E}\left[\log
p(Q|X)|X\right]\right]$

with $p(\mu|x)$ representing the association probability relating the input
vector $x$ with the codevector $\mu$. This is essentially a realization of the
Jaynes’s maximum entropy principle [48]. The resulting multi-objective
optimization is conveniently formulated as the minimization of the Lagrangian

$F=D-TH$ (12)

where $T$ is the temperature parameter that acts as a Lagrange multiplier.
Clearly, (12) represents the scalarization method for trade-off analysis
between two performance metrics. For large values of $T$ we maximize the
entropy, and, as $T$ is lowered, we essentially transition from one Pareto
point to another in a naturally occurring direction that resembles an
annealing process. In this regard, the entropy $H$, which is closely related
to the “purity” of the clusters, acts as a regularization term which is given
progressively less weight as $T$ decreases. As is the case in vector
quantization, one minimizes $F$ via a coordinate block optimization algorithm.
Minimizing $F$ with respect to the association probabilities $p(\mu|x)$ is
straightforward and yields the Gibbs distribution

$p(\mu|x)=\frac{e^{-\frac{d(x,\mu)}{T}}}{\sum_{\mu}e^{-\frac{d(x,\mu)}{T}}}$
(13)

while, in order to minimize $F$ with respect to the codevector locations $\mu$
we set the gradients to zero

$\displaystyle\frac{d}{d\mu}D=0$
$\displaystyle\implies\frac{d}{d\mu}\mathbb{E}\left[\mathbb{E}\left[d(X,\mu)|X\right]\right]=0$
(14)

###  III-A Bifurcation Phenomena

Adding to the physical analogy, it is significant that, as the temperature is
lowered, the system undergoes a sequence of “phase transitions”, which
consists of natural cluster splits where the cardinality of the codebook
(number of prototypes) increases. This is a bifurcation phenomenon that
provides a useful tool for controlling the size of the model relating it to
the scale of the solution. At very high temperature ($T\rightarrow\infty$) the
optimization yields uniform association probabilities
$p(\mu|x)=\nicefrac{{1}}{{K}}$, and all the codevectors are located at the
same point. This is true regardless of the number of codevectors available. We
refer to the number of different codevectors resulting from the optimization
process as effective codevectors. These define the cardinality of the
codebook, which changes as we lower the temperature. The bifurcation occurs
when the solution above a critical temperature $T_{c}$ is no longer the
minimum of the free energy $F$ for $T<T_{c}$. A set of coincident codevectors
then splits into separate subsets. These critical temperatures $T_{c}$ can be
traced when the Hessian of $F$ loses its positive definite property, and are,
in some cases, computable (see Theorem 1 in [32]). In other words, an
algorithmic implementation needs only as many codevectors as the number of
effective codevectors, which depends only on the temperature parameter, i.e.
the Lagrange multiplier of the multi-objective minimization problem in (12).
As shown in [30], we can detect the bifurcation points by maintaining and
perturbing pairs of codevectors at each effective cluster so that they
separate only when a critical temperature is reached.

###  III-B Bregman Divergences as Dissimilarity Measures

The proximity measure $d$ need not be a metric, and can be generalized to more
general dissimilarity measures inspired by information theory and statistical
analysis. In particular, the family of Bregman divergences, which includes the
widely used Kullback-Leibler divergence, can offer numerous advantages in
learning applications compared to the Euclidean distance alone [34]. Notably,
in the case of deterministic annealing, Bregman divergences play an even more
important role, since we can show that, if $d$ is a Bregman divergence, the
solution to the second optimization step (14) can be analytically computed in
a convenient centroid form [30]:

$\mu^{*}=\mathbb{E}\left[X|\mu\right]$ (15)

###  III-C The online learning rule

In an offline approach, the approximation of the conditional expectation
$\mathbb{E}\left[X|\mu\right]$ is computed by the sample mean of the data
points weighted by their association probabilities $p(\mu|x)$. To define an
online training rule for the deterministic annealing framework, we formulate a
stochastic approximation algorithm to recursively estimate
$\mathbb{E}\left[X|\mu\right]$ directly. As a direct consequence of Theorem 4
in [30], the following corollary provides an online learning rule that solves
the optimization problem of the deterministic annealing algorithm.

######  Corollary 0.1.

The online training rule

$\begin{cases}\rho_{i}(n+1)&=\rho_{i}(n)+\beta(n)\left[\hat{p}(\mu_{i}|x_{n})-\rho_{i}(n)\right]\\\
\sigma_{i}(n+1)&=\sigma_{i}(n)+\beta(n)\left[x_{n}\hat{p}(\mu_{i}|x_{n})-\sigma_{i}(n)\right]\end{cases}$
(16)

where $\sum_{n}\beta(n)=\infty$, $\sum_{n}\beta^{2}(n)<\infty$, and the
quantities $\hat{p}(\mu_{i}|x_{n})$ and $\mu_{i}(n)$ are recursively updated
as follows:

$\displaystyle\mu_{i}(n)=\frac{\sigma_{i}(n)}{\rho_{i}(n)},\quad\hat{p}(\mu_{i}|x_{n})=\frac{\rho_{i}(n)e^{-\frac{d(x_{n},\mu_{i}(n))}{T}}}{\sum_{i}\rho_{i}(n)e^{-\frac{d(x_{n},\mu_{i}(n))}{T}}}$
(17)

converges almost surely to a solution of the block optimization (13), (15).

The learning rule (16), (17) is a stochastic approximation algorithm [33].

###  III-D Online Deterministic Annealing for Supervised Learning

We can extend the proposed learning algorithm to be used for classification as
well. For the classification problem, a pair of random variables
$\left\\{X,c\right\\}\in S\times\left\\{0,1\right\\}$ defined in a probability
space $\left(\Omega,\mathcal{F},\mathbb{P}\right)$, is observed with $c$
representing the class of $X$ and $S\subseteq\mathbb{R}^{d}$. Let
$M:=\left\\{\mu_{h}\right\\}_{h=1}^{K}$, where $\mu_{h}\in ri(S_{h})$
represent codevectors, and define the set
$C_{\mu}:=\left\\{c_{\mu_{h}}\right\\}_{h=1}^{K}$, such that
$c_{\mu_{h}}\in\left\\{0,1\right\\}$ represents the class of $\mu_{h}$ for all
$h\in\left\\{1,\ldots,K\right\\}$. The quantizer
$Q^{c}:S\rightarrow\left\\{0,1\right\\}$ is defined such that
$Q^{c}(X)=\sum_{h=1}^{k}c_{\mu_{h}}\mathds{1}_{\left[X\in S_{h}\right]}$.
Then, the minimum-error classification problem is formulated as

$\min_{\left\\{\mu_{h},S_{h}\right\\}_{h=1}^{K}}~{}J_{B}(Q^{c}):=\pi_{1}\sum_{H_{0}}\mathbb{P}_{1}\left[X\in
S_{h}\right]+\pi_{0}\sum_{H_{1}}\mathbb{P}_{0}\left[X\in S_{h}\right]$ (18)

where
$\pi_{i}:=\mathbb{P}\left[c=i\right],\mathbb{P}_{i}\left\\{\cdot\right\\}:=\mathbb{P}\left\\{\cdot|c=i\right\\}$,
and $H_{i}$ is defined as
$H_{i}:=\left\\{h\in\left\\{1,\ldots,K\right\\}:Q^{c}=i\right\\}$,
$i\in\left\\{0,1\right\\}$. In this case we can rewrite the expected
distortion as

$\displaystyle D=\mathbb{E}\left[d^{b}(c_{X},Q^{c})\right]$

where $d^{b}(c_{x},c_{\mu})=\mathds{1}_{\left[c_{x}\neq c_{\mu}\right]}$.
Because $d^{b}$ is not differentiable, using similar principles as in the case
of LVQ, we can instead approximate the optimal solution by solving the
minimization problem for the following distortion measure

$d^{c}(x,c_{x},\mu,c_{\mu})=\begin{cases}d(x,\mu),~{}c_{x}=c_{\mu}\\\
0,~{}c_{x}\neq c_{\mu}\end{cases}$ (19)

This particular choice for the distortion measure $d^{c}$ results in important
regularization properties (see Section III-E and [30]).

###  III-E The algorithm

The Online Deterministic Annealing (ODA) algorithm for clustering and
classification is depicted in Algorithm 1. In this section we briefly discuss
the properties of the algorithm, as well as the effect of the parameters used.
A detailed discussion on the implementation of Alg. 1 and the effect of its
parameters can be found in [30].

Temperature Schedule. The temperature schedule $T_{i}$ plays an important role
in the behavior of the algorithm. Starting at high temperature $T_{max}$
ensures the correct operation of the algorithm. The value of $T_{max}$ depends
on the domain of the data. The stopping temperature $T_{min}$ can be set a
priori or be decided online depending on the performance of the model at each
temperature level. It is common practice to use the geometric series
$T_{i+1}=\gamma T_{i}$.

Stochastic Approximation. Regarding the stochastic approximation stepsizes,
simple time-based learning rates, e.g. of the form
$\alpha_{n}=\nicefrac{{1}}{{a+bn}}$, have been sufficient for fast convergence
in all our experiments so far. Convergence is checked with the condition
$d_{\phi}(\mu_{i}^{n},\mu_{i}^{n-1})<\epsilon_{c}$ for a given threshold
$\epsilon_{c}$ that can depend on the domain of $X$.

Bifurcation and Perturbations. To every temperature level $T_{i}$, corresponds
a set of effective codevectors $\left\\{\mu_{j}\right\\}_{j=1}^{K_{i}}$, which
consist of the different solutions of the optimization problem (12) at
$T_{i}$. Bifurcation, at $T_{i}$, is detected by maintaining a pair of
perturbed codevectors $\left\\{\mu_{j}+\delta,\mu_{j}-\delta\right\\}$ for
each effective codevector $\mu_{j}$ generated at $T_{i-1}$, i.e. for
$j=1\ldots,K_{i-1}$. Using arguments from variational calculus [32], it is
easy to see that, upon convegence, the perturbed codevectors will merge if a
critical temperature has not been reached, and will get separated otherwise.
Therefore, the cardinality of the model is at most doubled at every
temperature level. For classification, a perturbed codevector for each class
is generated.

Regularization. Merging is detected by the condition
$d_{\phi}(\mu_{j},\mu_{i})<\epsilon_{n}$, where $\epsilon_{n}$ is a design
parameter that acts as a regularization term for the model. Large values for
$\epsilon_{n}$ (compared to the support of the data $X$) lead to fewer
effective codevectors, while small $\epsilon_{n}$ values lead to a fast growth
in the model size, which is connected to overfitting. An additional
regularization mechanism that comes as a natural consequence of the stochastic
approximation learning rule, is the detection of idle codevectors. To see
that, notice that the sequence $\rho_{i}(n)$ resembles an approximation of the
probability $p(\mu_{i},c_{\mu_{i}})$, and whether or not they become
negligible ($\rho_{i}(n)<\epsilon_{r}$) is a natural criterion for removing
the codevector $\mu_{i}$. The threshold $\epsilon_{r}$ is a parameter that
usually takes values near zero.

Complexity. The complexity of Alg. 1 for a fixed temperature coefficient
$T_{i}$ is $O(N_{c_{i}}(2K_{i})^{2}d)$, where $N_{c_{i}}$ is the number of
stochastic approximation iterations needed for convergence which corresponds
to the number of data samples observed, $K_{i}$ is the number of codevectors
of the model at temperature $T_{i}$, and $d$ is the dimension of the input
vectors, i.e., $x\in\mathbb{R}^{d}$. Therefore, assuming a training dataset of
$N$ samples and a temperature schedule
$\left\\{T_{1}=T_{max},T_{2},\ldots,T_{N_{T}}=T_{min}\right\\}$, the worst
case complexity of Algorithm 1 becomes:

$\displaystyle O(N_{c}(2\bar{K})^{2}d)$

where $N_{c}=\max_{i}\left\\{N_{c_{i}}\right\\}$ is an upper bound on the
number of data samples observed until convergence at each temperature level,
and

$\displaystyle
N_{T}\leq\bar{K}\leq\min\left\\{\sum_{n=0}^{N_{T}-1}2^{n},\sum_{n=0}^{\log_{2}K_{max}}2^{n}\right\\}<N_{T}K_{max}$

where the actual value of $\bar{K}$ depends on the bifurcations occurred as a
result of reaching critical temperatures and the effect of the regularization
mechanisms described above. Note that typically $N_{c}\ll N$ as a result of
the stochastic approximation algorithm, and $\bar{K}\ll N_{T}K_{max}$ as a
result of the progressive nature of the ODA algorithm.

Fine-Tuning. In practice, because the convergence to the Bayes decision
surface comes at the limit $(K,T)\rightarrow(\infty,0)$, a fine-tuning
mechanism should be designed to run on top of the proposed algorithm after
$T_{min}$. This can be either an LVQ algorithm [49] or some other local model.

Algorithm 1 Online Deterministic Annealing

Select Bregman divergence $d_{\phi}$

Set temperature schedule: $T_{max}$, $T_{min}$, $\gamma$

Decide maximum number of codevectors $K_{max}$

Set convergence parameters: $\left\\{\alpha_{n}\right\\}$, $\epsilon_{c}$,
$\epsilon_{n}$, $\epsilon_{r}$, $\delta$

Select initial configuration
$\left\\{\mu^{i}\right\\}:c_{\mu^{i}}=c,~{}\forall c\in\mathcal{C}$

Initialize: $K=1$, $T=T_{max}$

Initialize: $p(\mu^{i})=1$, $\sigma(\mu^{i})=\mu^{i}p(\mu_{i})$, $\forall i$

while $K<K_{max}$ and $T>T_{min}$ do

Perturb $\mu^{i}\leftarrow\left\\{\mu^{i}+\delta,\mu^{i}-\delta\right\\}$,
$\forall i$

Increment $K\leftarrow 2K$

Update $p(\mu^{i})$, $\sigma(\mu^{i})\leftarrow\mu^{i}p(\mu^{i})$, $\forall i$

Set $n\leftarrow 0$

repeat

Observe data point $x$ and class label $c$

for $i=1,\ldots,K$ do

Compute membership $s^{i}=\mathds{1}_{\left[c_{\mu^{i}}=c\right]}$

Update:  $\displaystyle p(\mu^{i}|x)$
$\displaystyle\leftarrow\frac{p(\mu^{i})e^{-\frac{d_{\phi}(x,\mu^{i})}{T}}}{\sum_{i}p(\mu^{i})e^{-\frac{d_{\phi}(x,\mu^{i})}{T}}}$
$\displaystyle p(\mu^{i})$ $\displaystyle\leftarrow
p(\mu^{i})+\alpha_{n}\left[s^{i}p(\mu^{i}|x)-p(\mu^{i})\right]$
$\displaystyle\sigma(\mu^{i})$
$\displaystyle\leftarrow\sigma(\mu^{i})+\alpha_{n}\left[s^{i}xp(\mu^{i}|x)-\sigma(\mu^{i})\right]$
$\displaystyle\mu^{i}$
$\displaystyle\leftarrow\frac{\sigma(\mu^{i})}{p(\mu^{i})}$

Increment $n\leftarrow n+1$

end for

until $d_{\phi}(\mu^{i}_{n},\mu^{i}_{n-1})<\epsilon_{c}$, $\forall i$

Keep effective codevectors: discard $\mu^{i}$ if
$d_{\phi}(\mu^{j},\mu^{i})<\epsilon_{n}$, $\forall i,j,i\neq j$

Remove idle codevectors: discard $\mu^{i}$ if $p(\mu^{i})<\epsilon_{r}$,
$\forall i$

Update $K$, $p(\mu^{i})$, $\sigma(\mu^{i})$, $\forall i$

Lower temperature $T\leftarrow\gamma T$

end while

##  IV Progressive Learning in Multiple Resolutions

As opposed to linear models or simple artificial neural networks where the
forward pass is often computationally inexpensive scaling linearly with
respect to the number of neurons, in prototpe-based methods, every codevector
has to be compared with all $K$ of the codevectors used, resulting in a
computational complexity that scales quadratically with respect to $K$. The
progressive nature of the online deterministic annealing approach partially
mitigates this problem, but as $K$ increases, the running time of the
algorithm is greatly affected. To counteract that, we impose a tree structure
in the growth of the ODA model, which will also serve as a means for
developing a multi-resolution online deterministic annealing learning
algorithm.

###  IV-A Tree-Structured Online Deterministic Annealing

Tree-structured vector quantizers (TSVQ) provide computationally efficient
means of compressing multivariate data. While lacking the optimality
properties of full-search techniques, TSVQ are easier to implement and give
rise to variable-rate codes that frequently outperform fixed-rate, full-search
techniques with the same average number of bits per sample.

The progressive nature of a greedy growing tree structure aligns perfectly
with the online deterministic annealing algorithm. In fact, as will be shown,
ODA constructs a tree of arbitrary many children per parent node thus dropping
the dependency of the number of codevectors to the number of cell splits,
which is associated with the depth of the tree [50, 51]. In other words, every
parent cell is split into a progressively growing number of children cells as
the temperature decreases. We refer to this as a horizontal split. Once the
temperature coefficient $T$ drops bellow a predefined temperature threshold,
then the partition is fixed and every children node will serve as a parent
node that will be divided into more cells by further application of the ODA
algorithm. We refer to this as a vertical split. We note that the simplicity
of the splitting criterion roots in the annealing nature of ODA. The splitting
criterion can even be generalized to involve additional terms such as the
percentage of improvement in accuracy or distortion reduction for every
temperature step.

As before, let $X:\Omega\rightarrow S$ be a random variable. A tree-structured
quantizer $Q_{\Delta}:S\rightarrow S$ is defined by a set of codevectors
$M:=\left\\{\mu_{h}\right\\}_{h=1}^{K}$ arranged in a tree structure $\Delta$
with a single root node $\nu_{0}$. A node $\nu_{i}$ represents a region
$S_{\nu_{i}}\subseteq S$. It is associated with a set of codevectors
$M_{\nu_{i}}:=\left\\{\mu_{j}\right\\}$, where $|M_{\nu_{i}}|=K_{\nu_{i}}$,
and a Voronoi partition $V_{\nu_{i}}:=\left\\{S_{j}\right\\}$ of $S_{\nu_{i}}$
with respect to $M_{\nu_{i}}$ and a divergence measure $d$ (see Section III).
A tree-structure is a special case of a connected, acyclic directed graph,
where each node has a single parent node (except for the root node) and an
arbitrary number of children nodes, that is, we do not restrict $\Delta$ to be
a binary tree. The set $C(\nu_{i})$ represents the nodes
$\left\\{nu_{j}\right\\}$ that are the children of $\nu_{i}$, while the set
$P(\nu_{j})$ represents the node $\nu_{i}$ for which $\nu_{j}\in C(\nu_{i})$.
The level $l\geq 0$ of a node $\nu_{h}\in\Delta$ is the length of the path
$\left\\{\nu_{0},\ldots,\nu_{i},\nu_{j},\ldots,\nu_{h}\right\\}$ leading from
the root node $\nu_{0}$ to $\nu_{h}$ such that $\nu_{j}\in C(\nu_{i})$. To
explicitly state the level $l$ of a node $\nu_{i}$ we will write
$\nu_{i}^{(l)}$. The terminal nodes
$\tilde{N}:=\left\\{\nu:C(\nu)=\emptyset\right\\}$ are called leaves, and the
union of their associated codevectors will be denoted
$\tilde{M}:=\left\\{\mu_{j}\right\\}$, where $|\tilde{M}|=\tilde{K}$ is the
number of leaf codevectors, and
$\tilde{l}:=\max\left\\{l:\mu_{\nu}^{(l)}\in\tilde{M},\
\forall\nu\right\\}<\infty$ will denote the maximum depth of the tree. The
quantizer $Q_{\Delta}$ defines a hierarchical partitioning scheme for the
domain $S$, such that for every node $\nu_{i}^{(l)}\in\Delta$ associated with
the region $S_{i}$, its children nodes $\left\\{\nu^{(l+1)}:\nu^{(l+1)}\in
C(\nu_{i}^{(l)})\right\\}$ are associated with the regions
$\left\\{S_{j}:S_{j}\in V_{\nu_{i}}\right\\}$ which form a Voronoi partition
of $S_{\nu_{i}}$ with respect to $M_{\nu_{i}}$.

After a finite number of vertical cell splits, the ODA tree $Q_{\Delta}$ will
have a depth $\tilde{l}<\infty$ and a finite number $\tilde{K}<\infty$ of leaf
cells. A final application of the ODA algorithm (Alg. 1) $\tilde{l}$ times,
results, in the limit $K\rightarrow\infty$, in a consistent density and risk
estimator, since all the assumptions needed (see [30]) carry over to the leaf
cells as subsets of the original domain. We note that due to the fact that the
tree-structured $Q_{\Delta}$ defines a partition of $S$, the training of the
nodes is done in parallel by doing one update of the ODA algorithm (Alg. 1) in
every region $S_{i}$ associated to $\nu_{i}$ asynchronously, depending on what
region each new observation $X$ belongs to. As a result, depending on the
density of the random variable $X$, some cells will be visited more often than
others, which will result in some branches of the tree growing faster than
others, inducing a variable-rate code that frequently outperforms fixed-rate,
full-search techniques with the same average number of bits per sample.
Alternatively, when learning offline using a dataset, all nodes can be trained
using parallel processes, which can be utilized by multi-core computational
units. Algorithm 2, presents the pseudocode for the greedy-growing Tree-
Structured Online Deterministic Annealing algorithm using multiple-
resolutions, as will be explained in the next section.

Algorithm 2 Multi-Resolution ODA Algorithm

Set temperature schedule:
$\overline{T}=\left\\{\overline{T}_{\tilde{l}},\overline{T}_{\tilde{l}-1},\ldots,\overline{T}_{0}\right\\}$,
$\underline{T}=\left\\{\underline{T}_{\tilde{l}},\underline{T}_{\tilde{l}-1},\ldots,\underline{T}_{0}\right\\}$

Initialize $\nu_{0}^{(0)}$, $M_{\nu_{0}}$, $V_{\nu_{0}}$.

repeat

Observe data point $(X,c)$

$w=\nu_{0}$, $l=\tilde{l}$, $x=X_{\tilde{l}}$

while $C(w)\neq\emptyset$ do

$w=v\in C(w)$ such that $x\in S_{v}$

$l=l-1$

$x=X_{l}$

end while

Update $M_{w}$ using Alg. 1 in $S_{w}$ with ($T_{max}=\overline{T}_{l}$,
$T_{min}=\underline{T}_{l}$)

if ODA in $S_{w}$ converged and $l<\tilde{l}$ then

Split $w$ to $C(w)$ with respect to $V_{w}$

end if

until Convergence

###  IV-B Hierarchical Learning with Multi-Resolution ODA

The tree-growing mechanism detailed above aligns with our intuition to use
more detailed information to train deeper layers of the tree, and can be used
to develop a hierarchical multi-resolution learning scheme. In the following
analysis, we assume that the multi-resolution representation is defined by the
subspaces $\left\\{V_{j},j\in\mathbb{N}\right\\}$ induced by the wavelet
transform described in Section II. This analysis can be directly generalized
to other hierarchical dictionaries, including the scattering transform.
Consider the high-resolution representation of the input $X_{0}\in V_{0}=S$
and the multi-resolution approximation defined by
$\left\\{X_{1},X_{2},\ldots,X_{\tilde{l}}\right\\}$, the projections of $X$ to
the subspaces $\left\\{V_{1},V_{2},\ldots,V_{\tilde{l}}\right\\}$, as
described in Section II. The input representations
$\left\\{X_{\tilde{l}},\ldots,X_{1},X_{0}\right\\}$, can be used to train a
sequence of vector quantizers with optimal sets of codevectors
$\left\\{M^{(\tilde{1})},\ldots,M^{(1)},M^{(0)}\right\\}$, respectively, where
$M^{(l)}:=\left\\{\mu_{h}^{(l)}\right\\}_{h=1}^{K_{l}}$, and $\mu_{h}^{(l)}\in
V_{l}$. Since $V_{\tilde{l}}\subset\ldots\subset V_{1}\subset V_{0}$, it
follows that $X_{l}\in V_{\lambda<l}$, and, as a result, $\mu_{h}^{(1)}\in
V_{\lambda<l}$, for all $l=1,\ldots,\tilde{l}$. In other words, since the
multi-resolution representation defines a sequence of spaces $V_{l}\subset
V_{l-1}$, both the low-resolution representations $X_{l}$ and the
corresponding codevectors $M^{(1)}$ belong to a subspace of $V_{\lambda<l}$,
and, of course, a subspace of the original input space $V_{0}$. Because the
codevectors $\mu_{h}^{(l)}\in V_{l}$ live in a subspace of $V_{0}$, we can
define the distortion $J(M^{(l)})$ (similar to eq. 11) induced by the optimal
codevectors in resolution $l$ and measured in $V_{0}$. Similarly, we define
the classification error
$L(M^{(l)})=\pi_{1}(l)\sum_{H_{0}(l)}\mathbb{P}_{1}\left[X_{0}\in
S_{h}(l)\right]+\pi_{0}(l)\sum_{H_{1}(l)}\mathbb{P}_{0}\left[X_{0}\in
S_{h}(l)\right]$ (similar to 18), as the probability of error induced by the
Voronoi partition of the codevectors $M^{(l)}$ and measured in $V_{0}$.
Because of the fact that $V_{l}\subset V_{0}$, it is easy to see that
$J(M^{(\lambda)})\geq J({M^{(l)}})$ and $L(M^{(\lambda)})\geq L({M^{(l)}})$
for $\lambda\geq l$.

In other words, learning with progressively better approximations yields
progressively better classification results, as one would expect. This fact
can be exploited to reduce the computational complexity of the learning
algorithm following a hierarchical architecture that mirrors the progressive
attention mechanism observed in human cognition [4, 5, 3, 6, 7, 8]. At this
point, the feature space that is created through the wavelet transform
(Section II), has a multi-resolution structure, while the learning module has
a progressively growing tree structure. We make a fusion of the two structures
by applying the using different input representations in each level of the
tree, by matching the resolution $(l)$ to the depth level $(\tilde{l}-l)$ of
each node. This idea was first introduced in [52] and [53]. More specifically,
we start with the representation corresponding to the coarsest resolution
$X_{\tilde{l}}$ of the input (which is a projection in $V_{\tilde{l}}$), and
apply Alg. 1 on the region $S_{\nu_{0}}$ of the root node $\nu_{0}$, until a
threshold temperature is reached. At this point, the level $l=1$ of the tree-
structured quantizer $Q_{\Delta}$ has been fixed, and this qualifies as the
first vertical split. The nodes $C(\nu_{0})$ are horizontally split by the
application of Alg. 1 using the input representations $X_{\tilde{l}-1}$, and
this procedure continuous until the desired distortion/accuracy level is
reached, or until a stopping criterion is met, depending on the cardinality of
the neurons (system resources) and the temperature of the annealing process.
The multi-resolution online deterministic annealing algorithm is depicted in
Alg. 2.

The resulting architecture shares similar properties with Deep Convolutional
Networks [11] and the Scattering Convolutional Networks [36], as shown in Fig.
2 and Fig. 3, and provides a general system-theoretic framework for
progressive, hierarchical, and knowledge-based learning. We illustrate this
framework in the block diagram of Fig. 3.

(a) DCNN.

(b) SCN.

Figure 2: The architectures of Deep Convolutional Neural Networks (DCNN) and
Scattering Convolutional Networks (SCN). The arrows represent a cascade of
convolution, rectifying, and downsampling operations.

(a) System-theoretic block diagram.

(b) Deep neural network architecture.

Figure 3: Block diagram of the proposed learning architecture as a closed-loop
system (a) and as a deep neural network (b). The arrows represent a cascade of
convolution, rectifying, and downsampling operations.

##  V Experimental Evaluation and Discussion

We illustrate the properties and evaluate the performance of the proposed
learning algorithm in widely used artificial and real datasets for clustering
and classification.

###  V-A ODA in One Resolution

(a) Gaussians.

(b) Concentric circles.

Figure 4: Illustration of the evolution of Alg. 1 for decreasing temperature
$T$ in binary classification in 2D. $(b)$ Showcasing robustness with respect
to bad initial conditions.

We first showcase how Alg. 1 works in a simple, but illustrative,
classification problem in two dimensions (Fig. 4). The underlying class
distributions are shaped as concentric circles and the dataset consist of
$1500$ samples. Since the objective is to give a geometric illustration of how
the algorithm works in the two-dimensional plane, the Euclidean distance is
used. The algorithm starts at high temperature with a single codevector for
each class. As the temperature coefficient gradually decreases (Fig. 4, from
left to right), the number of codevectors progressively increases. The
accuracy of the algorithm typically increases as well. As the temperature goes
to zero, the complexity of the model, i.e. the number of codevectors, rapidly
increases. This may, or may not, translate to a corresponding performance
boost. A single parameter –the temperature $T$– offers online control on this
complexity-accuracy trade-off. Finally, the robustness of the proposed
algorithm with respect to the initial configuration is showcased. Here the
codevectors are poorly initialized outside the support of the data, which is
not assumed known a priori (e.g. online observations of unknown domain). In
this example the LVQ algorithm has been shown to fail [54]. In contrast, the
entropy term $H$ in the optimization objective of Alg. 1, allows for the
online adaptation to the domain of the dataset and helps to prevent poor local
minima. Finally, the running time of the ODA algorithm for the classification
and clustering problems of the dataset in Fig. 4(a) is compared in Fig. 5
against state-of-the-art algorithms. All experiments were implemented on a
personal computer. The reader is referred to [30] for a more detailed
assessment of the ODA algorithm (Alg. 1) in both classification and
clustering, as well as a discussion on the choice of the parameters and the
limitations of the approach.

Figure 5: Running time of the algorithms in Fig. 4(a).

###  V-B Multi-Resolution Learning

In Fig. 6, Fig. 7, and Fig. 8, we illustrate the evolution of the multi-
resolution online deterministic annealing algorithm (Alg. 2) compared to the
single-resolution ODA algorithm and the tree-structured ODA algorithm, in a 2D
classification problem. The accuracy of the model, the number of observations
needed for convergence and the number of codevectors (neurons) used at each
temperature level, are also shown.

We stress that the computational benefit of using a tree-structured
architecture is significant. We remind the readers that the complexity of the
ODA algorithm is $O(N_{c}(2\bar{K})^{2}d)$, where $N_{c}$ is an upper bound on
the number of data samples observed until convergence, and $\bar{K}$ the
number of effective codevectors used by the algorithm. The partition of the
space in smaller sets (Voronoi regions) results in easier to solve, localized
ODA problems. In particular, the size of each Voronoi set decreases in a
geometric rate with respect to the depth of the tree. A decrease of similar
magnitude is observed on the number $N_{c}$. If an ODA algorithm needs
$\bar{K}$ codevectors on the original domain $S$, a tree-structured ODA
algorithm with $\tilde{l}$ layers will require $O(\bar{K}^{1/\tilde{l}})$
number of codevectors, assuming the same number of codevectors in each cell.

Another advantage of using a tree-structured learning module is the
localization properties. The Voronoi regions shrink quickly, and allow for the
use of local models, which is especially important in high-dimensional spaces.
Unlike most learning models, it is possible to locate the area of the data
space that presents the highest error rate and selectively split it by using
local ODA. This process can be iterated until the desired error rate (or
average distortion) is achieved. When using a training dataset for
classification, it is often possible to force accuracy of up to $100\%$ on the
training dataset. This is similar to an over-fitted classification and
regression tree (CART) [55].

However, over-fitting on the training dataset often adversely affects the
generalization properties of the model, the performance on the testing
dataset, and the robustness against adversarial attacks. Therefore, the
progressive process of ODA becomes important in establishing a robust way to
control the trade-off between performance and complexity, before you reach
that limit. Finally, an important question in tree-structured learning models
is the question of which cell to split next. An exhaustive search in the
entire tree to find the node that presents the largest error rate is possible
but is often not desired due to the large computational overhead. This is
automatically answered by the multi-resolution ODA algorithm (Alg. 2) as it
asynchronously updates all cells depending on the sequence of the online
observations. As a result, the regions of the data space that are more densely
populated with data samples are trained first, which results in a higher
percentage of performance increase per cell split. We stress that this
property makes the proposed algorithm completely dataset-agnostic, in the
sense that it does not require the knowledge of a training dataset a priori,
but instead operates completely online, i.e., using one observation at a time
to update its knowledge base.

Finally, we evaluate the performance of the algorithm in image recognition,
which is considered to be central to human perception and cognition. To
illustrate the interpretability of the representations learned by the model,
we show in Fig. 9 the weights of the neurons (codevectors) of the first two
layers of a multi-resolution ODA architecture trained on the MNIST dataset
[56] of hand-written digits. All $10$ neurons of the first layer are shown.
For the second layer, 10 of the neurons-children of each first-layer neuron
are randomly selected and shown. For the first two layers, a wavelet
representation of images of $7\times 7$ and $14\times 14$ pixels was used. The
testing performance for the first layer of $10$ neurons is $82\%$, and it goes
up to $89\%$ using both layers which add up to $212$ neurons. This is only
using the low-resolution wavelet representation of $14\times 14$ pixels.
Notice how different deformations of the hand-written digits are recognized by
the algorithm without explicitly having to specify these transformations a
priori. By studying the relationship between different codevectors that belong
to the same class, it is possible to reconstruct class-invariant
transformations and conserved quantities that will enable better feature
extraction. We continue the learning process using two more layers based on
the scattering transform provided by the ‘Kymatio’ software [45] using a
spatial resolution $J=2$ and ‘morlet’ complex directional wavelets of four
angles for the wavelet transform. We stop the process when the percentage
increment in accuracy is below a certain threshold which signifies that
further increase in the model complexity will likely not have a corresponding
effect in the performance, and may cause over-fitting. The final architecture
consists of $1,960$ neurons and achieves accuracy $97.21\%$, in a testing
dataset of $10,000$ samples. As discussed above, by following this process, a
knowledge representation has been built such that, if further performance
boost is required, we can locate the region of the data space that should be
given more emphasis. This property, in conjunction with the fact that the
neurons live in the data space itself (Fig. 9), make the proposed learning
architecture inherently interpretable.

Figure 6: Evolution of the ODA algorithm in 2D.

Figure 7: Evolution of the Tree-Structured ODA algorithm in a single
resolution in 2D.

Figure 8: Evolution of the Multi-Resolution ODA algorithm in multiple
resolutions (1D and 2D). The low-resolution representations consist of the
projections of the data points to the first principal component of the
dataset.

(a) First layer ($l=1$).

(b) $l=2$ (parent node: 0).

(c) $l=2$ (parent node: 1).

(d) $l=2$ (parent node: 2).

(e) $l=2$ (parent node: 3).

(f) $l=2$ (parent node: 4).

(g) $l=2$ (parent node: 5).

(h) $l=2$ (parent node: 6).

(i) $l=2$ (parent node: 7).

(j) $l=2$ (parent node: 8).

(k) $l=2$ (parent node: 9).

Figure 9: Learned representations of the first two layers a multi-resolution
Online Deterministic Annealing algorithm trained on the MNIST dataset.

###  V-C Source Code and Reproducibility

Code and Reproducibility: The source code is publicly available at
https://github.com/MavridisChristos/OnlineDeterministicAnnealing.

##  VI Conclusion

We used principles form mathematics, system theory, and optimization to
investigate the structure of a data-agnostic learning architecture that
resembles the “one learning algorithm” believed to exist in the visual and
auditory cortices of the human brain. Our approach consists of a closed-loop
system with three main components: (i) a multi-resolution analysis pre-
processor, (ii) a group-invariant feature extractor, and (iii) a progressive
knowledge-based learning module, along with multi-resolution feedback loops
that are used for learning. The design of (i) and (ii), is based on the well-
established theory of wavelet-based multi-resolution analysis and group
convolution operators. Regarding (iii), we introduced a multi-resolution
extension of the Online Deterministic Annealing algorithm, a novel learning
algorithm that constructs progressively growing knowledge representations. The
resulting architecture shares similar properties with Deep Convolutional
Networks and the Scattering Convolutional Networks but also provides a more
general system-theoretic framework with some significant properties. First, it
is dataset-agnostic, a term that in this case has a double meaning: it is not
tailored to a specific learning task, e.g., face recognition only, and does
not require the a priori knowledge of an entire dataset. In contrast, online
(or real-time) observations are used to train the algorithm. Second, it is
hierarchical as a result of the multi-resolution pre-processing with the
wavelet transform and the tree-structure of the learning algorithm. Third, it
is memory-based, since a knowledge base, i.e., the set of codevectors
representing the data space, is being built through training. In addition, it
is interpretable, a term that again has a double meaning. In the case of a
dataset that can be visualized by humans, the fact that the codevectors are
representatives of the data space is enough to convey the information of what
has been learned. Even in the general case when this is not possible, however,
the proposed approach offers a structured knowledge base that can be used to
localize the error-prone regions of the feature space, which leads to
understanding when and why the learned model fails. This information is
valuable to the designers of a learning model and can be used to incrementally
improve its performance without re-training the entire model from scratch. One
more property of the proposed approach is robustness with respect to initial
conditions, perturbations, and adversarial attacks. This is an aftermath of
the annealing optimization used to train the learning module, and the
established properties of the vector quantization variants. Finally, it is
progressive in the sense that the knowledge base is progressively growing,
starting from very few codevectors, and adding more as needed. This interplay
between compression and classification does not only help with the complexity
of the model, but also with the generalization properties and with the
sensitivity of the model with respect to input noise. It also provides one of
the first approaches to adaptively grow a neural network architecture, in
contrast to ad-hoc experimentation on the number of neurons and number of
layers to be used.

## References

  * [1] J. Bruna, A. Szlam, and Y. LeCun, “Learning stable group invariant representations with convolutional networks,” _arXiv preprint arXiv:1301.3537_ , 2013.
  * [2] A. W. Roe, S. L. Pallas, Y. H. Kwon, and M. Sur, “Visual projections routed to the auditory pathway in ferrets: receptive fields of visual neurons in primary auditory cortex,” _Journal of Neuroscience_ , vol. 12, no. 9, pp. 3651–3664, 1992.
  * [3] J. Sharma, A. Angelucci, and M. Sur, “Induction of visual orientation modules in auditory cortex,” _Nature_ , vol. 404, no. 6780, pp. 841–847, 2000.
  * [4] P. Taylor, J. Hobbs, J. Burroni, and H. Siegelmann, “The global landscape of cognition: hierarchical aggregation as an organizational principle of human cortical networks and functions,” _Scientific reports_ , vol. 5, no. 1, pp. 1–18, 2015.
  * [5] M. Riesenhuber and T. Poggio, “Hierarchical models of object recognition in cortex,” _Nature neuroscience_ , vol. 2, no. 11, pp. 1019–1025, 1999.
  * [6] M. Ito and H. Komatsu, “Representation of angles embedded within contour stimuli in area v2 of macaque monkeys,” _Journal of Neuroscience_ , vol. 24, no. 13, pp. 3313–3324, 2004.
  * [7] H. L. Read, J. A. Winer, and C. E. Schreiner, “Functional architecture of auditory cortex,” _Current opinion in neurobiology_ , vol. 12, no. 4, pp. 433–440, 2002.
  * [8] T. Chi, P. Ru, and S. A. Shamma, “Multiresolution spectrotemporal analysis of complex sounds,” _The Journal of the Acoustical Society of America_ , vol. 118, no. 2, pp. 887–906, 2005.
  * [9] G. A. Carpenter and S. Grossberg, “Adaptive resonance theory,” 2010.
  * [10] T. Poggio, T. A. Poggio, and F. Anselmi, _Visual cortex and deep networks: learning invariant representations_. MIT Press, 2016.
  * [11] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” _nature_ , vol. 521, no. 7553, pp. 436–444, 2015.
  * [12] G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm for deep belief nets,” _Neural computation_ , vol. 18, no. 7, pp. 1527–1554, 2006\.
  * [13] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in neural information processing systems_ , 2012, pp. 1097–1105.
  * [14] H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng, “Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations,” in _Proceedings of the 26th annual international conference on machine learning_ , 2009, pp. 609–616.
  * [15] M. I. Jordan, “Artificial intelligence—the revolution hasn’t happened yet,” _Harvard Data Science Review_ , vol. 1, no. 1, 7 2019, https://hdsr.mitpress.mit.edu/pub/wot7mkc1. [Online]. Available: https://hdsr.mitpress.mit.edu/pub/wot7mkc1
  * [16] N. C. Thompson, K. Greenewald, K. Lee, and G. F. Manso, “The computational limits of deep learning,” _arXiv preprint arXiv:2007.05558_ , 2020.
  * [17] E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy considerations for deep learning in nlp,” _arXiv preprint arXiv:1906.02243_ , 2019.
  * [18] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” _arXiv preprint arXiv:1312.6199_ , 2013.
  * [19] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in _2017 ieee symposium on security and privacy (sp)_. IEEE, 2017, pp. 39–57.
  * [20] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” _arXiv preprint arXiv:1706.06083_ , 2017.
  * [21] V. Sehwag, A. N. Bhagoji, L. Song, C. Sitawarin, D. Cullina, M. Chiang, and P. Mittal, “Analyzing the robustness of open-world machine learning,” in _Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security_ , 2019, pp. 105–116.
  * [22] C. G. Northcutt, A. Athalye, and J. Mueller, “Pervasive label errors in test sets destabilize machine learning benchmarks,” _arXiv preprint arXiv:2103.14749_ , 2021.
  * [23] T. Poggio, A. Banburski, and Q. Liao, “Theoretical issues in deep networks,” _Proceedings of the National Academy of Sciences_ , 2020.
  * [24] S. Mallat, “Understanding deep convolutional networks,” _Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences_ , vol. 374, no. 2065, p. 20150203, 2016.
  * [25] R. Vidal, J. Bruna, R. Giryes, and S. Soatto, “Mathematics of deep learning,” _arXiv preprint arXiv:1712.04741_ , 2017.
  * [26] C. Liu, L. Zhu, and M. Belkin, “Toward a theory of optimization for over-parameterized systems of non-linear equations: the lessons of deep learning,” _arXiv preprint arXiv:2003.00307_ , 2020.
  * [27] L. Von Melchner, S. L. Pallas, and M. Sur, “Visual behaviour mediated by retinal projections directed to the auditory pathway,” _Nature_ , vol. 404, no. 6780, pp. 871–876, 2000.
  * [28] L. Thaler, S. R. Arnott, and M. A. Goodale, “Neural correlates of natural human echolocation in early and late blind echolocation experts,” _PLoS one_ , vol. 6, no. 5, p. e20162, 2011.
  * [29] S. Mallat, “Group invariant scattering,” _Communications on Pure and Applied Mathematics_ , vol. 65, no. 10, pp. 1331–1398, 2012.
  * [30] C. Mavridis and J. Baras, “Online deterministic annealing for classification and clustering,” _arXiv preprint arXiv:2102.05836_ , 2021.
  * [31] S. Saralajew, L. Holdijk, M. Rees, and T. Villmann, “Robustness of generalized learning vector quantization models against adversarial attacks,” in _International Workshop on Self-Organizing Maps_. Springer, 2019, pp. 189–199.
  * [32] K. Rose, “Deterministic annealing for clustering, compression, classification, regression, and related optimization problems,” _Proceedings of the IEEE_ , vol. 86, no. 11, pp. 2210–2239, 1998.
  * [33] V. S. Borkar, _Stochastic approximation: a dynamical systems viewpoint_. Springer, 2009, vol. 48.
  * [34] A. Banerjee, S. Merugu, I. S. Dhillon, and J. Ghosh, “Clustering with bregman divergences,” _Journal of machine learning research_ , vol. 6, no. Oct, pp. 1705–1749, 2005.
  * [35] T. Villmann, S. Haase, F.-M. Schleif, B. Hammer, and M. Biehl, “The mathematics of divergence based online learning in vector quantization,” in _IAPR Workshop on Artificial Neural Networks in Pattern Recognition_. Springer, 2010, pp. 108–119.
  * [36] J. Bruna and S. Mallat, “Invariant scattering convolution networks,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 35, no. 8, pp. 1872–1886, 2013.
  * [37] J. Zhao, M. Mathieu, and Y. LeCun, “Energy-based generative adversarial network,” _arXiv preprint arXiv:1609.03126_ , 2016.
  * [38] Y. LeCun, “The next frontier in ai: Unsupervised learning,” _https://www.youtube.com/watch?v=IbjF5VjniVE_ , 2016.
  * [39] F. Anselmi, J. Z. Leibo, L. Rosasco, J. Mutch, A. Tacchetti, and T. Poggio, “Unsupervised learning of invariant representations,” _Theoretical Computer Science_ , vol. 633, pp. 112–121, 2016.
  * [40] S. Geman, “Compositionality in vision,” _The grammar of vision: probabilistic grammar-based models for visual scene understanding and object categorization_ , 2007.
  * [41] F. Anselmi, L. Rosasco, C. Tan, and T. Poggio, “Deep convolutional networks are hierarchical kernel machines,” _arXiv preprint arXiv:1508.01084_ , 2015\.
  * [42] A. L. Yuille and C. Liu, “Deep nets: What have they ever done for vision?” _arXiv preprint arXiv:1805.04025_ , 2018.
  * [43] S. G. Mallat, “A theory for multiresolution signal decomposition: the wavelet representation,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 11, no. 7, pp. 674–693, 1989.
  * [44] S. Mallat, _A wavelet tour of signal processing_. Elsevier, 1999.
  * [45] M. Andreux, T. Angles, G. Exarchakis, R. Leonarduzzi, G. Rochette, L. Thiry, J. Zarka, S. Mallat, J. Andén, E. Belilovsky, J. Bruna, V. Lostanlen, M. J. Hirn, E. Oyallon, S. Zhang, C. Cella, and M. Eickenberg, “Kymatio: Scattering transforms in python,” 2019.
  * [46] Y. Zhuang and J. S. Baras, “Optimal wavelet basis selection for signal representation,” in _Wavelet Applications_ , vol. 2242. International Society for Optics and Photonics, 1994, pp. 200–211.
  * [47] C. N. Mavridis and J. S. Baras, “Convergence of stochastic vector quantization and learning vector quantization with bregman divergences,” _IFAC-PapersOnLine_ , vol. 53, no. 2, pp. 2214–2219, 2020.
  * [48] E. T. Jaynes, “Information theory and statistical mechanics,” _Physical review_ , vol. 106, no. 4, p. 620, 1957.
  * [49] T. Kohonen, _Learning Vector Quantization_. Berlin, Heidelberg: Springer Berlin Heidelberg, 1995, pp. 175–189.
  * [50] E. A. Riskin and R. M. Gray, “A greedy tree growing algorithm for the design of variable rate vector quantizers (image compression),” _IEEE Transactions on Signal Processing_ , vol. 39, no. 11, pp. 2500–2507, 1991.
  * [51] A. B. Nobel and R. A. Olshen, “Termination and continuity of greedy growing for tree-structured vector quantizers,” _IEEE Transactions on Information Theory_ , vol. 42, no. 1, pp. 191–205, 1996.
  * [52] J. S. Baras and S. I. Wolk, “Efficient organization of large ship radar databases using wavelets and structured vector quantization,” in _Proceedings of 27th Asilomar Conference on Signals, Systems and Computers_. IEEE, 1993, pp. 491–498.
  * [53] ——, “Wavelet-based progressive classification of high-range resolution radar returns,” in _Wavelet Applications_ , H. H. Szu, Ed., vol. 2242, International Society for Optics and Photonics. SPIE, 1994, pp. 967 – 977. [Online]. Available: https://doi.org/10.1117/12.170034
  * [54] J. S. Baras and A. LaVigna, “Convergence of a neural network classifier,” in _Advances in Neural Information Processing Systems_ , 1991, pp. 839–845.
  * [55] L. Breiman, “Random forests,” _Machine learning_ , vol. 45, no. 1, pp. 5–32, 2001.
  * [56] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proceedings of the IEEE_ , vol. 86, no. 11, pp. 2278–2324, 1998.

|  Christos N. Mavridis (M’20) received the Diploma degree in electrical and
computer engineering from the National Technical University of Athens, Greece,
in 2017, and the M.S. and Ph.D. degrees in electrical and computer engineering
at the University of Maryland, College Park, MD, USA, in 2021. His research
interests include learning theory, stochastic optimization, systems and
control theory, multi-agent systems, and robotics. He has worked as a
researcher at the Department of Electrical and Computer Engineering at the
University of Maryland, College Park, MD, USA, and as a research intern for
the Math and Algorithms Research Group at Nokia Bell Labs, NJ, USA, and the
System Sciences Lab at Xerox Palo Alto Research Center (PARC), CA, USA. Dr.
Mavridis is an IEEE member, and a member of the Institute for Systems Research
(ISR) and the Autonomy, Robotics and Cognition (ARC) Lab. He received the Ann
G. Wylie Dissertation Fellowship in 2021, and the A. James Clark School of
Engineering Distinguished Graduate Fellowship, Outstanding Graduate Research
Assistant Award, and Future Faculty Fellowship, in 2017, 2020, and 2021,
respectively. He has been a finalist in the Qualcomm Innovation Fellowship US,
San Diego, CA, 2018, and he has received the Best Student Paper Award (1st
place) in the IEEE International Conference on Intelligent Transportation
Systems (ITSC), 2021.
---|---
|  John S. Baras (LF’13) received the Diploma degree in electrical and
mechanical engineering from the National Technical University of Athens,
Athens, Greece, in 1970, and the M.S. and Ph.D. degrees in applied mathematics
from Harvard University, Cambridge, MA, USA, in 1971 and 1973, respectively.
He is a Distinguished University Professor and holds the Lockheed Martin Chair
in Systems Engineering, with the Department of Electrical and Computer
Engineering and the Institute for Systems Research (ISR), at the University of
Maryland College Park. From 1985 to 1991, he was the Founding Director of the
ISR. Since 1992, he has been the Director of the Maryland Center for Hybrid
Networks (HYNET), which he co-founded. His research interests include systems
and control, optimization, communication networks, applied mathematics,
machine learning, artificial intelligence, signal processing, robotics,
computing systems, security, trust, systems biology, healthcare systems,
model-based systems engineering. Dr. Baras is a Fellow of IEEE (Life), SIAM,
AAAS, NAI, IFAC, AMS, AIAA, Member of the National Academy of Inventors and a
Foreign Member of the Royal Swedish Academy of Engineering Sciences. Major
honors include the 1980 George Axelby Award from the IEEE Control Systems
Society, the 2006 Leonard Abraham Prize from the IEEE Communications Society,
the 2017 IEEE Simon Ramo Medal, the 2017 AACC Richard E. Bellman Control
Heritage Award, the 2018 AIAA Aerospace Communications Award. In 2016 he was
inducted in the A. J. Clark School of Engineering Innovation Hall of Fame. In
2018 he was awarded a Doctorate Honoris Causa by his alma mater the National
Technical University of Athens, Greece.
---|---