# Delving Deeper Into Astromorphic Transformers

Md Zesun Ahmed Mia, , Malyaban Bal, Abhronil Sengupta MZA Mia, M. Bal and A.
Sengupta are with the School of Electrical Engineering and Computer Science,
The Pennsylvania State University, University Park, PA 16802, USA. E-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Preliminary attempts at incorporating the critical role of astrocytes - cells
that constitute more than 50% of human brain cells - in brain-inspired
neuromorphic computing remain in infancy. This paper seeks to delve deeper
into various key aspects of neuron-synapse-astrocyte interactions to mimic
self-attention mechanisms in Transformers. The cross-layer perspective
explored in this work involves bio-plausible modeling of Hebbian and pre-
synaptic plasticities in neuron-astrocyte networks, incorporating effects of
non-linearities and feedback along with algorithmic formulations to map the
neuron-astrocyte computations to self-attention mechanism and evaluating the
impact of incorporating bio-realistic effects from the machine learning
application side. Our analysis on sentiment and image classification tasks on
the IMDB and CIFAR10 datasets underscores the importance of constructing
Astromorphic Transformers from both accuracy and learning speed improvement
perspectives.

###### Index Terms:

Astromorphic Transformer, Tripartite Synapse, Hebbian Plasticity, Presynaptic
Plasticity, Self-Attention

##  I Introduction

Astrocytes, a type of glial cell, play a critical role in brain function
encompassing various processes such as homeostasis, metabolism, and synaptic
regulation [1]. Astrocytes detect and regulate synaptic activity in the
tripartite synapse through interactions with pre and postsynaptic neurons.
Investigating their impact on neural computation is currently an active
research field in neuroscience and underscores the critical need to move
beyond the neuro-synaptic perspective of current Artificial Intelligence (AI)
systems. Recent experimental findings on neuron-astrocyte interactions and
modulation have led to significant progress in computational neuroscience,
enabling the development of models that incorporate neuron-astrocyte
interactions within neural networks [2, 3]. Astrocytes have been found to
modulate bursting in neural circuitry through the release of gliotransmitters,
which have an impact on neuronal excitability and synaptic plasticity [4, 5].
Astrocytes possess the ability to encode information through calcium signaling
and regulate information processing, thereby actively engaging in neural
computation at the tripartite synapse level. Additionally, astrocytes possess
inherent capacity as memory components [6, 7] and plasticity regulators that
are capable of facilitating local sequential learning [8, 9].

Preliminary attempts at incorporating the critical role of astrocytes in
brain-inspired neuromorphic computing has started in earnest. Neuromorphic
systems [10] are bio-plausible AI models that attempt to emulate neuronal and
synaptic operations with a higher degree of bio-fidelity to tap into the huge
energy efficiency of the brain. Such systems are temporal and event-driven
where computation occurs through transmission of sparse spiking events between
neurons - thereby offering the potential of orders of magnitude power
efficiency when implemented on neuromorphic platforms [11]. However,
astromorphic computing encapsulating features of astrocytic regulation in
neuromorphic systems still remain in infancy with works primarily targeting
integration of astrocyte functionality with Spike-Timing-Dependent-Plasticity
(STDP) mechanisms [12, 13, 14, 15] \- an unsupervised learning framework that
remains limited to shallow networks and simple tasks [16]. Other attempts also
suffer from the pressing issue of scalability to complex tasks [17].

This work is based on a recent study [18] that has shown promise at emulating
the self-attention mechanism in transformers [19] using a simple two layered
network involving neuron-astrocyte interactions. The ability to design
Astromorphic Transformers is therefore highly relevant in the current era of
Large Language Models (LLMs), such as BERT [20] and GPT [21] \- paving the
pathway for designing explainable transformers from a neuroscience perspective
as well as addressing scaling challenges for utilizing the computational role
of astrocytes in neuromorphic computing. However, as we illustrate in this
work, prior work has neglected various key aspects of bi-directional
astrocytic signalling, feedback and non-linear temporal behavior \- all of
which are key to address the fundamental mapping of self-attention mechanism
to astrocyte-neuron interactions along with providing insights on whether such
intrinsic non-linear dynamics and plasticity modulation enable better learning
capabilities from the application side. The distinguishing feature of this
work therefore lies in the neuroscience-algorithm-application co-design
perspective seeking to answer the following questions: (i) Theoretical
neuroscience: What aspects of astrocyte functionality should be included in
neural architectures to better explain self-attention mechanism in
transformers? (ii) Algorithm: Can we map key algorithmic modules like relative
positional information encoding of tokens to astrocytic interactions by
drawing qualitative inspiration from theoretical neuroscience? (iii)
Application: What is the impact of incorporating such dynamic temporal bio-
plausible transformers from the machine learning application perspective?

##  II Related Works and Main Contributions

Understanding the role of astrocytes in neuromorphic computing, especially
from the algorithm perspective, is relatively nascent. While hardware
implementation of astrocyte functionality in CMOS [22, 23] as well as post-
CMOS technologies [24] have been explored, algorithm level works exploiting
the role of astrocytes and leveraging their computational role has largely
remain limited to self-repair functionality of faulty neural network hardware
[25, 12, 26, 14]. This is mainly inspired by the fact that astrocytes, through
their synaptic plasticity regulation and feedback mechanisms, enable neurons
to fire at a baseline ideal frequency. This property can be utilized to
mitigate neuromorphic hardware non-idealities with stuck-at-faults and drift
effects [14]. However, such approaches are still limited to single layered
unsupervised networks where astrocyte functionality is integrated with STDP
learning mechanisms.

Recent studies have also examined the influence of astrocytes on learning [9],
working memory [6], energy minimization in neural networks [27], robotic
locomotion control [28], structure learning [17], among others. Building upon
recent works [18] at emulating self-attention module in transformers through
astrocyte-neuron interactions, we develop a computational framework that
encapsulates key aspects of temporal non-linearities and feedback enabled by
astrocytes and underscore the importance of incorporating such bio-plausible
effects through a comprehensive evaluation of Astromorphic Transformers on a
benchmark evaluation set. Our work expands the computational capabilities of
astrocyte-neuron network architectures equipped with plasticity, for the first
time, to a broad category of sequential tasks.

##  III Methodology

###  III-A Transformers

Vanilla transformers [19] for classification tasks usually consist of an
encoder block processing the provided key$(K)$, query$(Q)$ and value$(V)$
generated from the input token embeddings $X$ (words in language tasks, pixels
in vision tasks, etc.). Appendix A in the supplementary material illustrates
the conventional configuration of a transformer, featuring a block referred to
as “Multi-Head Attention” that produces the contextual association among the
provided inputs. The Multi-Head Attention block in transformers uses self-
attention mechanisms for multiple input heads, allowing the model to allocate
varying importance to individual tokens, focusing on salient features for
precise output in specific contexts.

The transformer implements auto-regressive attention by computing the dot
product of key and query, and then multiplying the resulting scalar with the
corresponding value derived from the input token. Implementing the softmax
function (a type of distribution function) atop the dot product of key and
query (to enforce attention over the tokens) increases the order of
complexity, a predicament that can be alleviated by the use of linearized
attention [29]. Linearized attention exploits the distributive property of
matrix multiplication resulting in lower complexity for self-attention
computation. More details can be found in Appendix B of the supplementary
material. From the linear attention-based transformer (linear transformer)
perspective [30], the transformer equations can be defined in the following
form:

$\mathrm{SA}(X)=\frac{\phi(Q)(\phi(K)^{T}V)}{\phi(Q)[\sum_{t=1}^{t=N}\phi(k_{t})]^{T}}\qquad\qquad\qquad\qquad$
(1) $Y=\mathrm{LayerNorm}(\mathrm{SA}(X)+X)\qquad\qquad\qquad\qquad\;$ (2)
$Z=\mathrm{Softmax}(\mathrm{Linear}(\mathrm{LayerNorm}(\mathrm{FFN}(Y)+Y)))$
(3)

From Equations 2 and 3, we can see that the output of self-attention is $Y$,
which undergoes layer normalization after being augmented with residual
connections from the input layer. The transformer determines the final output
probabilities, denoted by $Z$. There is a $\phi(.)$ feature map that
corresponds to a positive similarity function. When two such functions are
multiplied, the normalized dot product maps to a distribution function [31]
(i.e. softmax). According to [30], $\phi(.)$ can be defined by Equation 4.
Here, the activation function $elu(x)$, which stands for exponential linear
unit, is utilized for efficient and precise learning [32].

$\phi(x)=\mathrm{elu}(x)+1;$ (4)

In the next section, we discuss the tripartite synapse model and aspects which
will be necessary in constructing astromorphic transformers.

###  III-B Tripartite Synapse Dynamics

Astrocytes play a crucial role in the modulation of synaptic plasticity by
releasing chemical signals that regulate the efficacy of synaptic connections.
This phenomenon plays a role in modulating the strength and duration of
synaptic connectivity and is crucial for maintaining the health of neurons.
The depiction of the tripartite synapse in Fig. 1, which incorporates
astrocytes, is a biologically plausible representation that embodies a
multifaceted and evolving comprehension of synaptic signaling within the
brain.

####  III-B1 Astrocyte dynamics

The tripartite synapse’s dynamics are based on Li and Rinzel $Ca^{2+}$
dynamics [33], gatekeeper model [34], Nadkarni and Jung model [35, 36], and
bidirectional study on astrocytic glutamate binding to postsynaptic neurons
(AN model) [37].

Figure 1: A model of synaptic communication in the brain. The tripartite
synapse consists of presynaptic neurons, postsynaptic neurons and astrocytes.
Astrocytes detect neuronal activity and respond bidirectionally by emitting
gliotransmitters, thereby modulating the intensity and duration of synaptic
communication.

According to the biophysical models, it is observed that an action potential
(AP) is generated by the presynaptic axon through the release of
neurotransmitters. Neurotransmitters bind to postsynaptic dendrites, causing
depolarization. This leads to the release of 2-Arachidonylglycerol (2-AG) from
postsynaptic neurons to type 1 Cannabinoid Receptors (CB1Rs) on the
presynaptic terminal. This process is called Depolarization-induced
Suppression of Excitation ($DSE$) and depresses the continuous release of
neurotransmitters by inhibiting presynaptic neurons [38]. Neurotransmitter
release also involves binding 2-AG to CB1Rs on an astrocyte, enveloping the
synapse and increasing Inositol 1,4,5-trisphosphate ($IP_{3}$) levels.
$IP_{3}$ facilitates the release of calcium ions into the cytoplasm, thereby,
increasing calcium concentration. Three currents, $J_{channel}$, $J_{leak}$,
and $J_{pump}$, are introduced to model the rate of change of calcium
concentration inside the astrocyte [33]. When calcium concentration crosses a
threshold, astrocytes discharge glutamate, which binds to presynaptic group I
metabotropic Glutamate Receptors (mGluRs), creating a process called eSP
(Indirect Feedback) from the astrocyte to the presynaptic neuron [39]. The
comprehensive description of the computational model (Equation 5 introduces
the basic equation of the model) can be found in the works of [40, 37, 41].

$\frac{d(Ca^{2+})}{dt}=J_{channel}+J_{leak}-J_{pump}$ (5)

####  III-B2 Neuron dynamics

In this work, we consider the passive Leaky Integrate and Fire (LIF) dynamics
to describe the neuron firing process, which is described as follows:

$\tau_{m}\frac{dv}{dt}=-v(t)+R_{m}\sum_{i=1}^{M}I_{total-syn}^{i}$ (6)

where, $\tau_{m}$ represents the membrane time constant, $v$ represents the
membrane potential, and $R_{m}$ represents the membrane resistance. $I_{total-
syn}^{i}$ denotes the total synaptic current at the $i^{th}$ synapse. $M$ is
the number of synapses connecting to the corresponding neuron. Total synaptic
current at the $i^{th}$ synapse can be defined as the summation of two
currents namely, Synaptic Current, $I_{syn}^{i}$ and Slow Inward Current,
$SIC(t)$ in Equation 7:

$I_{total-syn}^{i}=I_{syn}^{i}+SIC(t)$ (7)

$I_{syn}^{i}$ is the intrasynaptic current which varies linearly with the
amount of neurotransmitters present in that synapse at any time, $t$
($I_{syn}^{i}=A_{se}y_{i}(t)$ where, $A_{se}$ and $y_{i}(t)$ represent the
absolute synaptic efficacy and the amount of neurotransmitter discharged by
the synapse at a given time $t$, respectively). Thus, $I_{syn}$ depends on the
direct feedback (corresponds to $DSE$) as direct feedback is modulated by the
amount of neurotransmitters between the pre and postsynaptic neuron. On the
other hand, the extrasynaptic Slow Inward Current $(SIC(t))$ is generated when
astrocyte-released glutamate interacts with extrasynaptic NMDARs (located in
postsynaptic neuron) [42]. To elaborate, the activation of astrocytes leads to
the release of glutamate from these cells. Upon its release, glutamate
interacts with NMDARs located on the postsynaptic neuron, resulting in the
emergence of a distinct form of current known as Slow Inward Current
$(SIC(t))$ [37]. The details of the mathematical modeling of $SIC(t)$ can be
found in [37].

Next, we develop a network architectural framework to mimic the self-attention
mechanism in transformers using the astrocyte and neuron dynamics described
herein and evaluate the impact of inherent non-linearities and feedback.

Figure 2: The neural network architecture showing the three layers. As tokens
are presented to the network as a $d$ dimensional vector, there are $d$
neurons in both the input and output layers. Hidden layer has $m$ neurons.
Solid lines indicate active operations, therefore $H_{neuron}$ and $H_{astro}$
are learnt during write mode but utilized during read mode.

###  III-C Neural Network Emulating Self-Attention

Self-attention is a fundamental component of all transformer models. Based on
prior works of mimicking self-attention by astrocytes [18], we adopt a network
topology as shown in Fig. 2. The neural network architecture comprises three
distinct layers, namely the Input layer, Hidden layer, and Output layer. There
exists two distinct operational modes for the aforementioned neural network:
(i) write mode and (ii) read mode. Upon presentation of the tokens ($x_{t}\in
R^{1\times d}$: which may include word embeddings and image embeddings) to the
input layer, the write mode is initially employed, followed by the read mode
to generate the outputs. In write mode, we encode information into the network
by means of keys and values, whereas, in read mode, we retrieve the
information by providing queries.

During the write mode, the input layer sequentially transmits the keys
($k_{t}=x_{t}W_{K};\;W_{K}\in R^{d\times m}$) for each token to the hidden
layer and the values ($v_{t}=x_{t}W_{V};\;W_{V}\in R^{d\times d}$) to the
output layer. The hidden layer neurons introduce a nonlinearity within the
system by activating a feature map, denoted as $\phi(.)$, on the transmitted
keys. Use of $\phi(.)$ reduces the computational complexity and memory usage
inspired from linearized attention mechanism. More information on the
reduction of the order of complexity is discussed in Appendix B (with Fig. S2
illustrating the idea) in the supplementary material. A presynaptic neuron in
the hidden layer is intricately connected with an astrocyte and an output
layer neuron, forming a tripartite synapse. Similar to the mechanism by which
the presynaptic neuron stimulates the astrocyte through the release of
neurotransmitters, in this context, the astrocyte is being activated through
the transmission of keys from neurons in the hidden layer. The write mode can
be mathematically expressed as in Equation 8:

$\displaystyle f_{t}=x_{t}$ (8) $\displaystyle
h_{t}=\phi(f_{t}W_{K})=\phi(k_{t})$ $\displaystyle l_{t}=f_{t}W_{V}=v_{t}$

In order to establish the read mode equations, it is necessary to first
formulate the neural plasticities that occur among the components of the
tripartite synapse. Moreover, our work aims at exploring strong bio-plausible
correlations at modelling such plasticities, which ultimately influence our
neuroscience-algorithm-application co-design analysis. In a subsequent
section, we will examine the read mode which is responsible for retrieving the
encoded information.

###  III-D Mathematical Formulation of Plasticity

####  III-D1 Hebbian Plasticity

Hebbian plasticity is characterized by the sequential adjustment of synaptic
weights in accordance with the interconnection pattern between two neurons
(pre and postsynaptic neurons) at the synapse. Previous studies [18] have
exclusively focused on the plasticity occurring between two neurons neglecting
non-linearities and the plasticity that may occur between astrocytes and
postsynaptic neurons, which is also involved in the learning process.

Contribution I: In the context of Hebbian plasticity, we underscore the
importance of addition of astrocyte-postsynaptic neuron connected Hebbian-
inspired learning along with inherent non-linearities. To elaborate, the
synaptic weights ($H_{neuron}$) that establish connections between the
presynaptic and postsynaptic neurons are acquired through Hebbian plasticity
during the write mode. This learning occurs due to the connection between pre
and postsynaptic neuron which can be mapped to the synaptic current
($I_{syn}$) from the neuroscience perspective. As discussed previously,
$I_{syn}$ is modulated by DSE so, $H_{neuron}$ is inspired from the direct
feedback in the tripartite synapse. This leads to Hebbian plasticity between
the hidden layer (presynaptic neuron) and the output layer (postsynaptic
neuron). Hidden neuron outputs are denoted as $h_{t}=\phi(k_{t})$ whereas the
output (last) layer outputs are denoted as $l_{t}=v_{t}$ in the write mode
(refer to Equation 8). $H_{neuron}$ weights are learnt via Equation 9, where
$m$ acts as a scaling factor affecting the learning speed.

$H_{neuron,t}=H_{neuron,t-1}+\frac{1}{m}h_{t}^{T}l_{t}$ (9)

The astrocyte functions as the third constituent in the tripartite synapse,
enabling bidirectional communication with the synaptic neurons. Hence, we
propose that the connection between the astrocyte and the postsynaptic neuron
be also represented in this formulation through a learnable weight matrix
($H_{astro}$), in accordance with the Hebbian rule. Equation 10 shows the
learning mechanism of these weights in which the presynaptic neuron output
($h_{t}$) is replaced by the astrocytic activity, represented by $W_{astro}$,
as this connection incorporates the astrocyte and the postsynaptic neuron. The
formulation of $W_{astro}$ will be considered in the succeeding text.

$H_{astro,t}=H_{astro,t-1}+\frac{1}{m}W_{astro}l_{t}$ (10)

The weight, denoted as $H_{astro}$, is inspired from the extrasynaptic current
$SIC$ (refer to tripartite synapse dynamics section). According to Equation 7,
$SIC$ is added to $I_{syn}$ to generate the total synaptic current.
Consequently, the two distinct learnable weights, denoted as $H_{neuron}$ and
$H_{astro}$, are combined to produce the Hebbian weight
$H=H_{neuron}+H_{astro}$.

Astrocytes have a significant impact on the Hebbian weight due to their
inherent non-linearity. A recent study by [17] introduced an AstroNet model
that optimizes neural network connections through temporal and global
regulation mechanisms. The authors found that astrocytes introduce non-linear
effects, which compound the pre-existing nonlinearity of neurons. The concept
of Hebbian Plasticity is associated with temporal and global regulation, where
the current weight is influenced by past neuron weights and astrocytic
weights. To map the effect of non-linearity, we consider that the summation of
neuron and astrocyte Hebbian weights undergoes non-linear activation,
ultimately generating the Hebbian weight ($H$). As such, we propose the
integration of a sigmoid ($\sigma$) nonlinearity into the learning process.
Thus, the weight $H$ exhibits the collective Hebbian plasticity of a
tripartite synapse, as elucidated by Equation 11, where $K$ and $V$ correspond
to keys and values of all the $N$ tokens in a single sequence (in matrix form:
$K\in R^{N\times m}$, $V\in R^{N\times d}$). Assuming initial $H_{neuron,0}$
and $H_{astro,0}$ from Equations 9 and 10 are both zero, we can define the
Hebbian weight in its matrix form ($H$) as,

$\displaystyle H=\sigma(H_{neuron}+H_{astro})$ (11) $\displaystyle
H=\sigma(\frac{1}{m}(\phi(K)^{T}V+W_{astro}*V))$

Next, we consider formulation of the astrocytic activity parameter,
$W_{astro}$. We note that astrocytic $Ca^{2+}$ dynamics is a relatively slower
temporal process than the streaming rate of the input tokens. Therefore, it is
expected that the astrocytic activity should encode relative information among
successive tokens. In this context, we derive inspiration from algorithmic
formulations of relative positional information of tokens, popularly used in
Vision transformers [43, 44], and map it to the activity parameter
$W_{astro}$. The astrocytic response parameter is therefore defined to use the
edge of the tokens to capture relative position differences between input
elements [43]. The edge of the tokens considers the pairwise relationships
between input elements in the sense that the input is modeled as a labeled,
directed, fully-connected graph. The edges pertaining to relative positional
information are learnt using the temporal regulation property of astrocytes
through the activity parameter, $W_{astro}$. Consequently, $H_{astro}$ can be
considered as a representation of acquired positional knowledge derived from
the input tokens. $W_{astro}$ can be modeled using Equation 12 where, $a_{ij}$
is the edge between input tokens $x_{i}$ and $x_{j}$.

$W_{astro}=\phi(a_{ij})^{T}\\\ $ (12)

####  III-D2 Presynaptic Plasticity

The plasticity resulting from the induced calcium response inside the
astrocyte due to bidirectional communication between the astrocyte and
presynaptic neurons (hidden layer) can be defined as presynaptic plasticity.
As tokens are sequentially presented to the hidden layer during the write
mode, keys: $\phi(k_{t})$ are generated which can be mapped to an increasing
number of released neurotransmitters (implying increment in $J_{channel}$ as
per the computational neuroscience model), thereby resulting in elevation of
the calcium concentration within the astrocyte (evident from Equation 5).
Thus, presynaptic plasticity parameter $g$, can be mapped to calcium
concentration in astrocytes modulated by indirect feedback ($eSP$), and can be
expressed as the aggregate of the inputs ($h_{t}=\phi(k_{t})$) received by the
presynaptic neuron. Equation 13 defines the formulation of the presynaptic
plasticity parameter ($g$). Assuming the initial calcium concentration to be
zero ($g_{0}=0$), $g$ can be expressed as,

$\displaystyle
g_{t}=g_{t-1}+h_{t}=g_{t-1}+\phi(k_{t})\quad(\mathrm{pertoken\;form})$ (13)
$\displaystyle
g=\sum_{t=1}^{N}{\phi(k_{t})}\>\;\;\quad\qquad\qquad\qquad\quad\;(\mathrm{matrix\;form})$

Figure 3: As the input spike frequency in the presynaptic neurons increases,
the rate of increase in calcium concentration slows down, thereby exhibiting a
non-linear relationship between $Ca^{2+}$ ion concentration and input spike
rate.

Contribution II: Prior work [18] at modelling pre-synaptic plasticity has not
considered the intrinsic non-linear $Ca^{2+}$ dynamics of the astrocytes in
response to increasing firing rate of presynaptic neurons. Due to the indirect
feedback (eSP), the rate of increase in the release rate of neurotransmitters
gradually reduces in response to an increase in presynaptic neuron firing over
time. Hence, a non-linear relationship exists between the calcium
concentration and the frequency of spikes received from the presynaptic
neuron. We substantiate this observation by simulating the AN model [40, 37].
The associated non-linear variation is depicted in Fig. 3 which leads us to
modify the presynaptic plasticity parameter $g$ as,

$g=(\sum_{t=1}^{N}{\phi(k_{t})})^{\alpha}$ (14)

where, the exponent $\alpha$ encodes the degree of non-linearity. From our
computational modelling, we use a value of $\alpha=0.25$. The subsequent
section elaborates on the read mode operation which involves the process of
retrieving the information encoded during the write mode to implement self-
attention mechanism incorporating our key contributions towards more bio-
plausible formulations of Hebbian and presynaptic plasticity.

###  III-E Realization of Self-Attention

Self attention mechanism is realized when the encoded information is accessed
during the read mode to obtain the activations of the output layer neurons.
The hidden layer neurons receive queries ($q_{t}=x_{t}W_{Q}$) from the input
layer and access the information that was previously encoded within the
astrocyte. This information is stored via both the Hebbian weight ($H$) and
the presynaptic plasticity parameter ($g$) during write mode. However, during
read mode, $H$ is already learnt and not modulated by the transmitted queries
from the presynaptic neurons as it is directly connected to the postsynaptic
neurons (output layer). As $g$ is the presynaptic plasticity parameter, it has
no direct connection to the postsynaptic neuron. Therefore, the information
encoded in $g$ has to be retrieved by the queries transmitted from the hidden
layer during read mode by inducing a calcium response from the astrocytes. In
order to read the calcium response ($c_{t}$) of the astrocyte, the queries
($h_{t}$) are scaled by the corresponding presynaptic plasticity parameters
($g_{t}$) across all the $m$ hidden neurons and then summed together. This is
depicted in Equation 15, where the matrix form of the calcium response is
defined by $C$.

$\displaystyle c_{t}=\sum_{m}g_{t}h_{t}\quad(\mathrm{pertoken\;form})$ (15)
$\displaystyle C=hg^{T}\>\;\quad\qquad(\mathrm{matrix\;form})$

Let us define the weight modulating the connection between the astrocyte and
the output layer as $P$. According to studies regarding Tumor Necrosis Factor
- $\alpha$ ($TNF-\alpha$), astrocytes possess the ability to boost synaptic
weights in reaction to low neural activity and diminish weights in reaction to
high neural activity [18] by means of calcium response (neural activity is
proportional to released neurotransmitters). Consequently, it is possible to
infer an inverse correlation between $P$ and the calcium response ($C$).
Simplifying, this relation is given by,

$P=\frac{1}{\phi(Q)((\sum_{t=1}^{N}{\phi(k_{t})})^{\alpha})^{T}}$ (16)

Given that the presynaptic plasticity modulated weight ($P$) is ascertained,
we can define the network equations for the read mode. As seen from Fig. 2,
the hidden layer and the output layer are connected by the Hebbian weight $H$
and the astrocyte is imposing another learnt weight (from calcium response)
denoted by $P$ onto the existing weight $H$. Thus the weights are implemented
by the Hadamard multiplication (element-wise: $\odot$) of $H$ and $P$ onto the
output layer. Furthermore, the output layer receives the input ($F$) directly
from the input layer modulated by unity weight to implement the residual
connection present in the self-attention module of transformers. Considering
the network architecture, the read mode network equations can be defined in
Equation 17. It is important to highlight here that during the read mode, the
decoding process occurs in a singular timestep, as opposed to the sequential
nature of the write mode. Hence, all the parameters for input, hidden and
output layers are represented in their matrix format.

$\displaystyle F=X$ (17) $\displaystyle h=\phi(FW_{Q})=\phi(Q)$ $\displaystyle
L=h\times(H\odot P)+F=\phi(Q)(H\odot P)+F$

In Equation 17, all the parameters are in matrix mode, where the dimensions
are defined as follows: $W_{Q}\in R^{d\times m}$, $X\in R^{N\times d}$ (input
tokens) and $Q\in R^{N\times m}$ (queries). $L$ represents the generated
output from the last layer. Self-attention algorithm is defined when the
network completes a write and read mode as discussed next.

By substituting the values of $H$ and $P$ from Equation 11 and 16 into
Equation 17, the resulting Equation 18 can be obtained after the write and
read mode:

$L=\frac{\phi(Q)(\sigma(\phi(K)^{T}V+W_{astro}V))}{m\times\phi(Q)\sum_{t=1}^{N}[{\phi(k_{t})}^{\alpha}]^{T}}+X$
(18)

In Equation 18, $L$ implements identical operation as the transformer self-
attention from Equation 1 (along with the inclusion of an extra residual
connection from the input layer). Thus, with this implementation, the
astromorphic transformer is formulated.

TABLE I: Performance comparison of our proposed astromorphic attention-based transformer architecture against other benchmarks with varying degrees of bioplausibility on the IMDB and CIFAR10 datasets. Task |  Model |  Neuromorphic Implementation |  Bioplausibility |  Accuracy
---|---|---|---|---
Sentiment Classification (IMDB Dataset) |  Memristor SNN [45] |  Directly trained in SNN domain |  Yes |  $84.86\%$
Memristor SNN [45] |  Converted from ANN |  Yes |  $85.88\%$
ANN [45] |  No spiking activity |  No |  $86.02\%$
Spike-transformer [46] |  Rate coded approach using Spiking Neurons |  Yes |  $86.36\%$
Recurrent SNN [47] |  Presynaptic current based backpropagation |  Yes |  $86.82\%$
HRNaEMSA biGRU [48] |  Implements multi-headed attention |  No |  $87.3\%$
HRNaEMSA biLSTM [48] |  Implements multi-headed attention |  No |  $87.4\%$
CoRNN [49] |  Coupled oscillators modeled as neurons |  No |  $87.4\%$
LSTM [50] |  Memristor crossbars mimicking NN |  No |  $87.48\%$
SNN [51] |  Compute-in-memory macro using SNN |  Yes |  $88.15\%$
UniCoRNN [52] |  Solves vanishing gradient problem in BPTT |  No |  $88.4\%$
Transformer |  Softmax attention without bioplausibilities |  No |  $88.9\pm 0.1\%$
Astromorphic Transformer (Our Model) |  Implements astromorphic attention with non-linearity and relative positional information |  Yes |  $\mathbf{88.7\pm 0.2\%}$
Image Classification (CIFAR10 Dataset) |  BioLeaF [53] |  SNNs trained by BP-based rules |  Yes |  $86.88\%$
STBP in CIFARNET [54] |  Spatio-temporal backprop |  No |  $89.83\%$
VGG16-SNN [55] |  ANN-SNN conversion |  Yes |  $91.55\%$
MS-ResNet [56] |  SNN oriented ResNet-110 architecture |  No |  $91.72\%$
ANN [57] |  ResNet-19 (12.63M parameters) |  No |  $94.97\%$
ANN [57] |  Transformer-4-384 (9.32M parameters) |  No |  $96.73\%$
Spikformer [57] |  Spikformer-4-384 (9.32M parameters) |  Yes |  $95.19\%$
Spikingformer [54] |  Spikingformer-4-384 (9.32M parameters) |  Yes |  $95.61\%$
Transformer |  Softmax attention without bioplausibilities |  No |  $97.4\pm 0.1\%$
Astromorphic Transformer (Our Model) |  Implements astromorphic attention with non-linearity and relative positional information |  Yes |  $\mathbf{97.0\pm 0.2\%}$
Figure 4: Comparative analysis to examine the effects of incorporating non-
linearity and relative positional encoding in the astromorphic transformer on
the (a) IMDB and (b) CIFAR10 datasets. The yellow and purple lines in Fig. (b)
are nearly overlapping.

##  IV Results

Our astromorphic transformer is trained and tested on two different tasks.
These include sentiment classification on the IMDB dataset [58] and image
classification on the CIFAR10 dataset [59]. Appendix C in the supplementary
material lists all the associated hyper-parameters. To date, there has not
been any language or vision model that incorporates transformer architecture
in a manner that is bioplausible and derived from neuron-astrocyte
interactions. We report an accuracy of $88.7\pm 0.2\%$ for the sentiment
classification task on the IMDB dataset, utilizing the non-linearity and
relative positional information in the model. We also achieve $97.0\pm 0.2\%$
accuracy using an astromorphic vision transformer on the CIFAR10 dataset.

###  IV-A Hardware Specifications & Dataset Details

The astromorphic transformer under consideration has been executed through the
utilization of a GPU-based platform running PyTorch. The computational
procedures were executed on a hardware configuration comprising a $12$ core
Intel(R) Xeon(R) Platinum $8167M$ CPU @ 2.00GHz, in conjunction with an
NVIDIA® Tesla® V100 GPU possessing $16384$ Mega Bytes of graphics memory.
While simulating the model on the IMDB dataset, the model uses 1.7 GB of
memory whereas it uses 4.4 GB of memory on the CIFAR10 dataset.

The IMDB dataset, created by Stanford University, contains $50,000$ movie
reviews (each 250-500 words long) labeled as positive or negative. The data is
presented in a tab-delimited format, containing fields for movie title, review
text, and sentiment label. The CIFAR-10 dataset is a collection of computer
vision data, comprising $60,000$ color images with dimensions of $32\times 32$
pixels. These images are categorized into 10 distinct classes, with each class
containing 6,000 images.

###  IV-B Network Architecture

In the Sentiment classification task, a transformer with an encoder-based
architecture is employed. We use a similar architecture as depicted in the
Fig. S1 with 1 encoder layer and 4 headed self-attention. We use the
combination of GloVe word embeddings with one-hot positional encoding as input
tokens. A dropout layer is employed after the addition of residual connection
just before the layer normalization.

Meanwhile, for image classification, a convolutional and self-attention based
neural network (CoAtNet [60]) was utilized to implement our proposed
innovations. CoAtNet transformer architecture can best utilize the
effectiveness of relative positional information when trained on vision task
focused datasets. Furthermore, CoAtNet leverages the capabilities of
Convolutional Neural Networks (CNNs) and transformers to enhance its
performance. Our architecture implements a $3\times 3$ convolution before
creating patches from the images. We use a patch size of $2$ on the image
inputs and employ an 8-layered encoder stack with 8-headed self-attention. In
addition to the original CoAtNet architecture, we add additional non-linearity
and average pooling after encoder layers.

For both the architectures, AdamW optimizer and Cross Entropy Loss are used to
learn the weights of the neural network. For IMDB dataset tokenization, we
used the Glove pretrained word vectors with 840B tokens, 2.2M vocab (cased),
300D vectors.

###  IV-C Performance Evaluation

Table I compares performance of our astromorphic transformer against
neuromorphic as well as other non-bioplausible models on the IMDB and CIFAR10
datasets. Our model outperforms RNNs, LSTMs, and GRUs on the IMDB dataset and
other bioplausible neuromorphic architectures. However, we did not compare
with existing pretrained transformers (BERT, GPT, etc.) because these models
are pretrained using non-bioplausible self-attention and have customized
pretrained learnable embeddings for natural language processing, whereas we
use fixed GloVe embeddings [61] that are simple enough to implement in our
astromorphic attention-based transformer architecture. We also test the
competitiveness of astromorphic attention in vision transformer architecture.
As shown in Table I, our astromorphic vision transformer model competes with
artificial and outperforms bioplausible models on the CIFAR10 dataset.

###  IV-D Ablation Studies

####  IV-D1 Learning speed

The impact of the two primary contributions of our study is illustrated in
Fig. 4. It is evident from the Fig. that the absence of relative positional
information leads to a decrease in the learning speed of the model. Measuring
the number of epochs to reach $85\%$ ($85\%$) accuracy on the IMDB (CIFAR10)
dataset, we find that: (i) with both of our contributions (non-linearity and
relative positional information), it takes only $16\pm 2$ ($45\pm 3$) epochs,
whereas (ii) with only the non-linearity included, it takes $22\pm 2$ ($111\pm
4$) epochs, and (iii) with only relative positional information, it takes
$43\pm 4$ ($215\pm 2$) epochs, and (iv) without both, it takes $54\pm 4$
($215\pm 2$) epochs to reach the specified accuracy.

####  IV-D2 Final accuracy

We implement the theoretical model proposed by [18] to see how it compares
without the bioplausibilities proposed herein and find an accuracy of $88.0\pm
0.2\%$ and $87.4\pm 0.1\%$ on the IMDB and CIFAR10 datasets, respectively.
This implementation does not incorporate the non-linearity and relative
positional information that are integral to our model, which draws inspiration
from the dynamics between neurons and astrocytes, thereby underscoring the
strong need to consider such bioplausibilities in modeling astromorphic
transformers. Our astromorphic transformer also competes with an iso-
architecture standard non-bioplausible transformer with softmax attention (see
“Transformer” entry in Table 1).

##  V Conclusion

This study aims to provide a detailed explanation of the development of an
astromorphic transformer using bioplausible self-attention. The inspiration
for this approach is drawn from the dynamics of bidirectional communication
between astrocytes and neurons in a tripartite synapse. In this study, we
exploit the intrinsic Presynaptic and Hebbian plasticity of the astrocyte to
incorporate two novel advancements into the transformer architecture.
Subsequently, we demonstrate the effectiveness of our model on two distinct
machine learning tasks. The performance of Astromorphic Transformers on
sentiment and image classification tasks is presented for the first time on
the IMDB and CIFAR10 datasets and is found to perform at par with an iso-
architecture non-bioplausible transformer model. Implementation of
Astromorphic Transformers on neuromorphic hardware has the potential of
enabling scalable, robust, power and energy efficient computing paradigms that
are applicable to a broad range of sequential processing tasks - a significant
shift from the huge computational requirements of conventional deep learning
solutions like Large Language Models.

Future research in this area could involve the exploration of astrocyte-to-
astrocyte communication through gap junctions. This approach may enable the
parallelization of Presynaptic and Hebbian plasticity, potentially addressing
the limitation of a slower calcium response in comparison to neuron
activation. In addition, it is possible to modify our model to accommodate
large language models such as BERT and GPT. This can be achieved by
incorporating the functionalization of relative positional information with
the customized embedding of these models from a neuromorphic perspective.
Further, quantifying the energy efficiency benefits of implementing
Astromorphic Transformers on neuromorphic hardware can be an area of
exploration.

## References

  * [1] N. A. Perez-Catalan, C. Q. Doe, and S. D. Ackerman, “The role of astrocyte-mediated plasticity in neural circuit development and function,” _Neural Development_ , vol. 16, no. 1, pp. 1–14, 2021.
  * [2] T. Manninen, J. Aćimović, and M.-L. Linne, “Analysis of network models with neuron-astrocyte interactions,” _Neuroinformatics_ , vol. 21, no. 2, pp. 375–406, 2023.
  * [3] N. M. Wakida, V. Gomez-Godinez, H. Li, J. Nguyen _et al._ , “Calcium dynamics in astrocytes during cell injury,” _Frontiers in Bioengineering and Biotechnology_ , vol. 8, p. 912, 2020.
  * [4] C. Durkee, P. Kofuji, M. Navarrete, and A. Araque, “Astrocyte and neuron cooperation in long-term depression,” _Trends in Neurosciences_ , vol. 44, no. 10, pp. 837–848, 2021.
  * [5] K. Lenk, E. Satuvuori, J. Lallouette, A. Ladrón-de Guevara, H. Berry, and J. A. Hyttinen, “A computational model of interactions between neuronal and astrocytic networks: the role of astrocytes in the stability of the neuronal firing rate,” _Frontiers in computational neuroscience_ , vol. 13, p. 92, 2020.
  * [6] Y. Tsybina, I. Kastalskiy, M. Krivonosov, A. Zaikin _et al._ , “Astrocytes mediate analogous memory in a multi-layer neuron–astrocyte network,” _Neural Computing and Applications_ , vol. 34, no. 11, pp. 9147–9160, 2022\.
  * [7] S. Gordleeva, Y. A. Tsybina, M. I. Krivonosov, I. Y. Tyukin _et al._ , “Situation-based memory in spiking neuron-astrocyte network,” _arXiv preprint arXiv:2202.07218_ , 2022.
  * [8] M. E. Gibbs, D. Hutchinson, and L. Hertz, “Astrocytic involvement in learning and memory consolidation,” _Neuroscience & Biobehavioral Reviews_, vol. 32, no. 5, pp. 927–944, 2008.
  * [9] C. M. Alberini, E. Cruz, G. Descalzi, B. Bessières, and V. Gao, “Astrocyte glycogen and lactate: New insights into learning and memory mechanisms,” _Glia_ , vol. 66, no. 6, pp. 1244–1262, 2018.
  * [10] A. Sengupta, Y. Ye, R. Wang, C. Liu, and K. Roy, “Going deeper in spiking neural networks: VGG and residual architectures,” _Frontiers in neuroscience_ , vol. 13, 2019.
  * [11] M. Davies, A. Wild, G. Orchard, Y. Sandamirskaya _et al._ , “Advancing neuromorphic computing with loihi: A survey of results and outlook,” _Proceedings of the IEEE_ , vol. 109, no. 5, pp. 911–934, 2021.
  * [12] J. Liu, L. J. McDaid, J. Harkin, S. Karim, A. P. Johnson, A. G. Millard, J. Hilder, D. M. Halliday, A. M. Tyrrell, and J. Timmis, “Exploring self-repair in a coupled spiking astrocyte neural network,” _IEEE transactions on neural networks and learning systems_ , vol. 30, no. 3, pp. 865–875, 2018.
  * [13] M. Rastogi, S. Lu, N. Islam, and A. Sengupta, “On the Self-Repair Role of Astrocytes in STDP Enabled Unsupervised SNNs,” _Frontiers in Neuroscience_ , vol. 14, p. 603796, Jan. 2021. [Online]. Available: https://www.frontiersin.org/articles/10.3389/fnins.2020.603796/full
  * [14] Z. Han, N. Islam, and A. Sengupta, “Astromorphic self-repair of neuromorphic hardware systems,” _arXiv preprint arXiv:2209.07428_ , 2022.
  * [15] J. Liu, J. Harkin, L. P. Maguire, L. J. McDaid, and J. J. Wade, “Spanner: A self-repairing spiking neural network hardware architecture,” _IEEE transactions on neural networks and learning systems_ , vol. 29, no. 4, pp. 1287–1300, 2017.
  * [16] P. U. Diehl and M. Cook, “Unsupervised learning of digit recognition using spike-timing-dependent plasticity,” _Frontiers in computational neuroscience_ , vol. 9, p. 99, 2015.
  * [17] M. Han, L. Pan, and X. Liu, “Astronet: When astrocyte meets artificial neural network,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 20 258–20 268.
  * [18] L. Kozachkov, K. V. Kastanenka, and D. Krotov, “Building transformers from neurons and astrocytes,” _Proceedings of the National Academy of Sciences_ , vol. 120, no. 34, p. e2219150120, 2023.
  * [19] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [20] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [21] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding by generative pre-training,” 2018.
  * [22] M. Ranjbar and M. Amiri, “An analog astrocyte–neuron interaction circuit for neuromorphic applications,” _Journal of Computational Electronics_ , vol. 14, no. 3, pp. 694–706, 2015.
  * [23] G. Karimi, M. Ranjbar, M. Amirian, and A. Shahim-Aeen, “A neuromorphic real-time vlsi design of ca2+ dynamic in an astrocyte,” _Neurocomputing_ , vol. 272, pp. 197–203, 2018.
  * [24] U. Garg, K. Yang, and A. Sengupta, “Emulation of astrocyte induced neural phase synchrony in spin-orbit torque oscillator neurons,” _Frontiers in neuroscience_ , vol. 15, 2021.
  * [25] J. Wade, L. McDaid, J. Harkin, V. Crunelli, and S. Kelso, “Self-repair in a Bidirectionally Coupled Astrocyte-Neuron (AN) System based on Retrograde Signaling,” _Frontiers in Computational Neuroscience_ , vol. 6, 2012. [Online]. Available: https://www.frontiersin.org/article/10.3389/fncom.2012.00076
  * [26] M. Rastogi, S. Lu, N. Islam, and A. Sengupta, “On the Self-Repair Role of Astrocytes in STDP Enabled Unsupervised SNNs,” _Frontiers in Neuroscience_ , vol. 14, p. 1351, 2021.
  * [27] A. Fountain and C. Merkel, “Effect of biologically-motivated energy constraints on liquid state machine dynamics and classification performance,” _Neuromorphic Computing and Engineering_ , vol. 2, no. 2, p. 024005, 2022.
  * [28] I. Polykretis, G. Tang, and K. P. Michmizos, “An astrocyte-modulated neuromorphic central pattern generator for hexapod robot locomotion on intel’s loihi,” in _International Conference on Neuromorphic Systems 2020_ , 2020, pp. 1–9.
  * [29] I. Schlag, K. Irie, and J. Schmidhuber, “Linear transformers are secretly fast weight programmers,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 9355–9366.
  * [30] A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret, “Transformers are rnns: Fast autoregressive transformers with linear attention,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 5156–5165.
  * [31] S. Chaudhari, V. Mithal, G. Polatkan, and R. Ramanath, “An attentive survey of attention models,” _ACM Transactions on Intelligent Systems and Technology (TIST)_ , vol. 12, no. 5, pp. 1–32, 2021.
  * [32] D.-A. Clevert, T. Unterthiner, and S. Hochreiter, “Fast and accurate deep network learning by exponential linear units (elus),” _arXiv preprint arXiv:1511.07289_ , 2015.
  * [33] Y.-X. Li and J. Rinzel, “Equations for InsP3 receptor-mediated $Ca^{2+}$ i oscillations derived from a detailed kinetic model: a Hodgkin-Huxley like formalism,” _Journal of theoretical Biology_ , vol. 166, no. 4, pp. 461–473, 1994.
  * [34] V. Volman, E. Ben-Jacob, and H. Levine, “The astrocyte as a gatekeeper of synaptic information transfer,” _Neural computation_ , vol. 19, no. 2, pp. 303–326, 2007.
  * [35] S. Nadkarni and P. Jung, “Dressed neurons: modeling neural–glial interactions,” _Physical biology_ , vol. 1, no. 1, p. 35, 2004.
  * [36] ——, “Modeling synaptic transmission of the tripartite synapse,” _Physical biology_ , vol. 4, no. 1, p. 1, 2007.
  * [37] J. J. Wade, L. J. McDaid, J. Harkin, V. Crunelli, and J. S. Kelso, “Bidirectional coupling between astrocytes and neurons mediates learning and dynamic coordination in the brain: a multiple modeling approach,” _PloS one_ , vol. 6, no. 12, p. e29445, 2011.
  * [38] B. E. Alger, “Retrograde signaling in the regulation of synaptic transmission: focus on endocannabinoids,” _Progress in neurobiology_ , vol. 68, no. 4, pp. 247–286, 2002.
  * [39] M. Navarrete and A. Araque, “Endocannabinoids potentiate synaptic transmission through stimulation of astrocytes,” _Neuron_ , vol. 68, no. 1, pp. 113–126, 2010.
  * [40] J. Wade, L. J. McDaid, J. Harkin, V. Crunelli, and S. Kelso, “Self-repair in a bidirectionally coupled astrocyte-neuron (AN) system based on retrograde signaling,” _Frontiers in computational neuroscience_ , vol. 6, p. 76, 2012\.
  * [41] M. De Pittà, M. Goldberg, V. Volman, H. Berry, and E. Ben-Jacob, “Glutamate regulation of calcium and ip 3 oscillating and pulsating dynamics in astrocytes,” _Journal of biological physics_ , vol. 35, pp. 383–411, 2009\.
  * [42] P. G. Haydon and G. Carmignoto, “Astrocyte control of synaptic transmission and neurovascular coupling,” _Physiological reviews_ , vol. 86, no. 3, pp. 1009–1031, 2006.
  * [43] P. Shaw, J. Uszkoreit, and A. Vaswani, “Self-attention with relative position representations,” _arXiv preprint arXiv:1803.02155_ , 2018.
  * [44] P. Ramachandran, N. Parmar, A. Vaswani, I. Bello, A. Levskaya, and J. Shlens, “Stand-alone self-attention in vision models,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [45] J. Huang, A. Serb, S. Stathopoulos, and T. Prodromakis, “Text classification in memristor-based spiking neural networks,” _Neuromorphic Computing and Engineering_ , vol. 3, no. 1, p. 014003, 2023.
  * [46] E. Mueller, V. Studenyak, D. Auge, and A. Knoll, “Spiking transformer networks: A rate coded approach for processing sequential data,” in _2021 7th International Conference on Systems and Informatics (ICSAI)_. IEEE, 2021, pp. 1–5.
  * [47] Z. Wang, Y. Zhang, H. Shi, L. Cao, C. Yan, and G. Xu, “Recurrent spiking neural network with dynamic presynaptic currents based on backpropagation,” _International Journal of Intelligent Systems_ , vol. 37, no. 3, pp. 2242–2265, 2022.
  * [48] X.-L. Leng, X.-A. Miao, T. Liu _et al._ , “Using recurrent neural network structure with enhanced multi-head self-attention for sentiment analysis,” _Multimedia Tools and Applications_ , vol. 80, pp. 12 581–12 600, 2021\.
  * [49] T. K. Rusch and S. Mishra, “Coupled oscillatory recurrent neural network (cornn): An accurate and (gradient) stable architecture for learning long time dependencies,” _arXiv preprint arXiv:2010.00951_ , 2020.
  * [50] A. Aswani, R. Kumar, J. N. Tripathi, and A. James, “Performance of crossbar based long short term memory with aging memristors,” in _2021 IEEE 3rd International Conference on Artificial Intelligence Circuits and Systems (AICAS)_. IEEE, 2021, pp. 1–4.
  * [51] A. Agrawal, M. Ali, M. Koo, N. Rathi, A. Jaiswal, and K. Roy, “Impulse: A 65-nm digital compute-in-memory macro with fused weights and membrane potential for spike-based sequential learning tasks,” _IEEE Solid-State Circuits Letters_ , vol. 4, pp. 137–140, 2021.
  * [52] T. K. Rusch and S. Mishra, “Unicornn: A recurrent model for learning very long time dependencies,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 9168–9178.
  * [53] Y. Yang and P. Li, “Bioleaf: A bio-plausible learning framework for training of spiking neural networks,” _arXiv preprint arXiv:2111.13188_ , 2021.
  * [54] C. Zhou, L. Yu, Z. Zhou, H. Zhang, Z. Ma, H. Zhou, and Y. Tian, “Spikingformer: Spike-driven residual learning for transformer-based spiking neural network,” _arXiv preprint arXiv:2304.11954_ , 2023.
  * [55] G. Srinivasan, C. Lee, A. Sengupta, P. Panda, S. S. Sarwar, and K. Roy, “Training deep spiking neural networks for energy-efficient neuromorphic computing,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 8549–8553.
  * [56] Y. Hu, Y. Wu, L. Deng, and G. Li, “Advancing residual learning towards powerful deep spiking neural networks,” _arXiv e-prints_ , pp. arXiv–2112, 2021.
  * [57] Z. Zhou, Y. Zhu, C. He, Y. Wang, S. Yan, Y. Tian, and L. Yuan, “Spikformer: When spiking neural network meets transformer,” _arXiv preprint arXiv:2209.15425_ , 2022.
  * [58] A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts, “Learning word vectors for sentiment analysis,” in _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_. Portland, Oregon, USA: Association for Computational Linguistics, June 2011, pp. 142–150. [Online]. Available: http://www.aclweb.org/anthology/P11-1015
  * [59] A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” 2009.
  * [60] Z. Dai, H. Liu, Q. V. Le, and M. Tan, “Coatnet: Marrying convolution and attention for all data sizes,” _Advances in neural information processing systems_ , vol. 34, pp. 3965–3977, 2021.
  * [61] J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in _Empirical Methods in Natural Language Processing (EMNLP)_ , 2014, pp. 1532–1543. [Online]. Available: http://www.aclweb.org/anthology/D14-1162

## Supplementary Material

###  V-A Appendix A: Transformer Illustration

Transformer employs only encoder architecture for performing sentiment
classification and vision tasks. As illustrated in Fig. S5, the input
embeddings combined with the positional embeddings enter the encoder layer as
input tokens. Then key, query and value are generated from the input tokens
and after performing self attention on them, a skip connection is added from
the inputs and the overall result is layer normalized and fed into a feed
forward network. Another add & normalize operation is performed before feeding
them into a linear layer and softmax function to obtain the output
probabilities. Sometimes, there might be more than one encoder layer. For
different tasks, like machine translation and text generation, encoder-decoder
or only decoder type architecture of transformer can be used. However,
independent of the type of architecture needed, self-attention is a
fundamental mechanism defining transformer operation. The “Multi-Head
Attention” Block has self-attention modules parallelly running on different
heads obtained from the input embeddings.

Figure S5: Schematic representation of the Transformer Encoder architecture,
illustrating the sequential flow of operations. $N$ denotes the number of
encoder layers in the network.

###  V-B Appendix B: Memory Efficient Linearized Attention

Linearized attention involves utilizing feature maps with lower complexity
levels, thereby minimizing memory and computational resources. As discussed
before, the introduction of non-linearity in biological neurons is facilitated
by an activation function ($elu(X)$) in the hidden layer neurons. The
utilization of the positive similarity feature map, denoted by $\phi(.)$,
characterizes the implementation of the linearized attention operation as
described in the work by (Katharopoulos et al. 2020). The utilization of
linearized feature attention results in a reduction of computational
complexity associated with the computation of the attention score. The
advantage of utilizing feature map based linearized attention over the
conventional distribution function (i.e. softmax) based self-attention is
demonstrated in Fig. S6.

Usual self-attention employs softmax as the distribution function, however,
the dot product of Query ($Q$) and Key ($K$) poses an $O(N^{2}D)$ complexity
which overuses the memory. Whereas, in linearized attention the computation
multiplies Key ($K$) and Value ($V$) yielding an $O(ND^{2})$ complexity,
properly utilizing the memory and computational efficiency when token length
($N$) is larger than internal embedding size ($D$). Thus, the employment of
the linearized feature map, denoted as $\phi(.)$, enables the realization of
memory efficient linearized transformer.

TABLE S2: Hyper-parameter list for the IMDB dataset. Hyper-parameters |  Value |  Tested Range
---|---|---
$\alpha$ (Non-linearity parameter) |  $0.25$ |  $0.15$-$0.4$
Model size (D) |  $128$ |  $32$-$256$
(Scaling parameter $m$) |  $500$D |  ($1$-$1000$)D
Model size |  $128$ |  $32$-$256$
Sequence length |  $400$ |  $100$-$600$
Learning rate |  $0.00001$ |  $1e-4$ \- $1e-6$
Batch size |  $64$ |  $8$-$256$
Number of epochs |  $120$ |  $40$-$150$
Number of heads |  $4$ |  $1$-$12$
TABLE S3: Hyper-parameter list for the CIFAR10 dataset. Hyper-parameters |  Value |  Tested Range
---|---|---
$\alpha$ (Non-linearity parameter) |  $0.25$ |  $0.15$-$0.4$
Model size (D) |  $256$ |  $64$-$512$
(Scaling parameter $m$) |  $500$D |  ($1$-$1000$)D
Patch size |  $2$ |  $2$-$4$
Learning rate |  $0.001$ |  $1e-2$ \- $1e-4$
Batch size |  $32$ |  $8$-$128$
Number of epochs |  $250$ |  $80$-$300$
Number of heads |  $8$ |  $2$-$12$
Figure S6: Figure showing the comparison of order of complexity between usual
and linearized self-attention. Distribution function can be mapped to softmax
function used in usual transformers.

###  V-C Appendix C: Hyper-parameter Details

The hyper-parameters for the encoder-type astromorphic transformer for the
IMDB dataset are listed in Table S2 in the supplementary material. The hyper-
parameters for the encoder-type convolution and astromorphic self-attention
based transformer for the CIFAR10 dataset are listed in Table S3.


# End-to-End High Accuracy License Plate Recognition Based on Depthwise
Separable Convolution Networks

Song-Ren Wang, Hong-Yang Shih, Zheng-Yi Shen, and Wen-Kai Tai  Computer
Science and Information Engineering
National Taiwan University of Science and Technology
Taipei, Taiwan
Email: {B10415005, B10415045, M10715098<EMAIL_ADDRESS>
###### Abstract

Automatic license plate recognition plays a crucial role in modern
transportation systems such as for traffic monitoring and vehicle violation
detection. In real-world scenarios, license plate recognition still faces many
challenges and is impaired by unpredictable interference such as weather or
lighting conditions. Many machine learning based ALPR solutions have been
proposed to solve such challenges in recent years. However, most are not
convincing, either because their results are evaluated on small or simple
datasets that lack diverse surroundings, or because they require powerful
hardware to achieve a reasonable frames-per-second in real-world applications.
In this paper, we propose a novel segmentation-free framework for license
plate recognition and introduce NP-ALPR, a diverse and challenging dataset
which resembles real-world scenarios. The proposed network model consists of
the latest deep learning methods and state-of-the-art ideas, and benefits from
a novel network architecture. It achieves higher accuracy with lower
computational requirements than previous works. We evaluate the effectiveness
of the proposed method on three different datasets and show a recognition
accuracy of over 99% and over 70 fps, demonstrating that our method is not
only robust but also computationally efficient.

##  I Introduction

Automatic license plate recognition (ALPR) is an essential part of research
for intelligent transportation systems such as surveillance systems for access
control, traffic monitoring, traffic violation detection, and parking lot
management.

Despite the large body of approaches proposed for ALPR [1, 2, 3, 4, 5, 6, 7],
challenges still exist in real-world applications. For example, highly
distorted or blurred vehicle images, poor lighting conditions, and frigid
weather conditions all significantly influence the recognition process. As
most previous solutions rely on extra rules (e.g., the maximum number of
characters allowed in a plate) to enhance their accuracy, or validate their
methods on datasets that are environment-specific (images are collected using
a single camera or from identical viewing angles), or lack diversity (for
instance, they recognize only a single class of vehicles or plates with the
same background color), they perform well only in restricted scenarios. A
robust ALPR system, however, should address these common challenges and adapt
to diverse environmental conditions.

|  |  |  |
---|---|---|---|---
|  |  |  |
|  |  |  |
Figure 1: Samples of NP-ALPR dataset showing the dataset variety, including
different times of day, locations, lighting conditions, and various vehicle
types and positions. Yellow rectangles represent manually annotated LP
locations. Due to privacy concerns, sensitive information is blurred.

Traditional license plate recognition (LPR) methods are commonly based on
segmentation-based methods, that is, character detection followed by character
recognition. Such methods require accurate character segmentation: faults in
segmentation lead to misrecognition of license plates even when using a robust
character recognizer. However, in real-world scenarios, as blurry images and
environmental factors degrade the accuracy of character segmentation, these
methods are not suitable for real-world applications; moreover, separating
character detection and recognition brings with it additional computational
costs. With the development of deep learning techniques, more and more
innovative ideas for solving LPR without character segmentation have been
proposed [4, 8, 9]. Segmentation-free methods usually extract features from
license plates and deliver them to a CNN or RNN model to recognize the
character sequence. Practically, these methods have better performance and
accuracy and are more robust, compared to segmentation-based methods.

In this paper, we propose a novel segmentation-free neural network
architecture for LPR based on Xception [10] and Inception-ResNet v2 [11]. With
the deep learning infrastructure provided by both, the network learning is
deeper and requires fewer parameters; that is, it is efficient and fast,
compared to other machine learning methods. In addition, we integrate an
affine transformation model into the proposed recognition system to rectify
distorted images before they are submitted to the recognition model. Several
datasets are used to verify the proposed method, showing that it achieves 99%
accuracy with an average fps (frames per second) of over 75, indicating that
it is applicable under different environments and challenging conditions
without any implementation change.

Apart from using public datasets [1, 8, 5] to validate the proposed method, we
also describe NP-ALPR,111Due to local privacy policies, we cannot make NP-ALPR
publicly available. For access to NP-ALPR, please contact the first author.
another large dataset with over 10,000 vehicle images recorded from several
cameras under various conditions, including locations, vehicle distance,
lighting conditions, and times of day. Furthermore, this dataset contains
various types of vehicles (motorcycles, cars, buses, and trucks), license
plates with various background colors (white, green, yellow, and red), and
license plates with characters in various colors (black, white, and red).
Examples of the dataset are shown in Figure 1. Note that the dataset also
includes images with multiple vehicles in the same frame.

To summarize, the main contributions made by this work are as follows:

  * •

We propose a novel end-to-end LPR model without character segmentation based
on Xception and Inception ResNet v2.

  * •

The proposed LPR method recognizes license plate characters at 99% accuracy
and over 75 fps under different datasets, confirming that it is robust enough
to be applied in real-world applications and that it outperforms several
recent works.

  * •

We present NP-ALPR, a dataset that contains over 10,000 images of various
types of vehicles under various conditions.

In the next sections, we first discuss approaches raised in recent years to
recognize license plates and then present the network architecture and
training flow of the proposed method. We then present the experimental results
to verify the proposed method.

##  II Related Work

Common methods for LPR can be divided into segmentation-based and
segmentation-free methods. In this section, we briefly review recent work on
both LPR methods and present a brief description of work that our research
touches upon. Since this work mainly focuses on LPR, we do not include studies
on vehicle and license plate detection.

###  II-A Segmentation-based LPR

Most conventional LPR methods perform character detection first and then
recognize the segemented character (character recognition). Several recent
works [5, 6, 7] are based on this methodology and differ only in
implementation details.

Existing algorithms for character detection can be classified into two groups,
the first based on connected component techniques that build connected areas
from binarized images and regard each as a character [2, 12, 13], and the
other based on projection techniques that separate each character according to
the top/bottom boundaries obtained from a horizontal projection of binarized
images [14, 12, 7].

Various algorithms have been proposed for character recognition which are
based either on template matching or on machine learning methods. The former
has been used widely for character recognition by measuring the similarity
between the segmented character and template images [15, 16, 14]. For the
latter, [17, 18] proposed works that apply a scale-invariant feature transform
(SIFT) [19] to extract features from segmented characters for delivery to
support vector machines (SVMs) [20] for final classification. As this method
requires numerous sliding windows to extract license plate features, however,
it is more computationally costly. The regional convolutional neural network
(RCNN) avoids such unecessary computations by using selective search to
optimize the detection, but it requires more effort to train the models. Other
common machine learning methods include probabilistic neural networks (PNNs),
hidden Markov models (HMMs), and multilayer perceptrons (MLPs).

###  II-B Segmentation-free LPR method

Solutions for LPR without character detection emerged following advances in
machine learning and deep learning. Elimination of character detection enables
these approaches to achieve better performance relative to segmentation-based
methods. [13] propose the first segmentation-free LPR method by using
convolutional neural networks (CNNs). LP image features are extracted via
CNNs, and a recurrent neural network (RNN) with connectionist temporal
classification (CTC) is used to label the sequential data and to classify the
character sequence. Similar CNN-based work includes [21, 4, 8].

To the best of our knowledge, [9] proposed the first work to use advanced
semantic segmentation for license plate recognition based on a modified
version of DeepLabv2 ResNet-101. This method is based on pixel-wise
classification, which usually yields more robust accuracy but is more
computationally complex. It achieved outstanding results on the AOLP dataset
[1].

###  II-C Datasets for ALPR

The SSIG SegPlate Database [22], which contains 2k Brazilian license plate
images, is a commonly used benchmark for ALPR research, but as the images were
collected only on sunny days and from a static camera with monotonous
backgrounds, the data is neither representive nor convincing. The UFPR-ALPR
dataset [5] contains 4.5k images of different types of vehicles captured by
three different cameras, yet does not include variations such as daytime
differences and weather conditions. The AOLP dataset [1] contains three
subsets: Access Control (with 681 images), Traffic Law Enforcement (with 528
images), and Road Patrol (with 611 images). Although each subset contains
images under different conditions, such as various times of day, tilted
plates, and different viewing points, the total number of images is relatively
small. In [8], the author proposes the ReId dataset, which contains 76k
license plates. Although it contains a large number of images of license
plates, it lacks variations in tilt angels. To the best of our knowledge, [23]
propose CCPD, the largest-yet dataset, with over 250k unique car images in
China with detailed annotations under diverse environmental conditions.

|  |  |
---|---|---|---
|  |  |
|  |  |
Figure 2: Dataset augmentation generated by imgaug library with different
settings. The left-top image is the raw image. TABLE I: The detailed
architecture descriptions of the proposed network.

|  Layer |  Input/Parameter | Output
---|---|---|---
| Concat | (128, 32), (128, 32) | (128, 32, 2)
| Conv + BN + LeakyReLU | (128, 32, 2) | (64, 16, 32)
| Conv + BN + LeakyReLU | (64, 16, 32) | (64, 16, 64)
| Xception Module | (64, 16, 64) | (64, 16, 64)
| Xception Module | (64, 16, 64) | (64, 16, 64)
| Inception Module B | (64, 16, 64) | (64, 16, 64)
| Inception Module B | (64, 16, 64) | (64, 16, 64)
| Xception Reduce Module | (64, 16, 64) | (32, 8, 128)
| Xception Module | (32, 8, 128) | (32, 8, 128)
| Xception Module | (32, 8, 128) | (32, 8, 128)
| Inception Module B | (32, 8, 128) | (32, 8, 128)
| Inception Module B | (32, 8, 128) | (32, 8, 128)
| Xception Module | (32, 8, 128) | (32, 8, 128)
| Xception Module | (32, 8, 128) | (32, 8, 128)
| Permute | (32, 8, 128) | (8, 32, 128)
| GlobalAvgPool1D | (8, 32, 128) | (32, 128)
| Dropout | 0.4 ratio |
| LSTM | (32, 128) | (32, 38)
| BatchNorm |  |
| Softmax |  |
|  |  |

TABLE II: All possible combinations of Taiwan license plate; where ’A’ denotes
an arbitrary alphabet, ’N’ denotes an arbitrary number character.

|  LP with 4 characters |  LP with 5 characters |  LP with 6 characters |  LP with 7 characters
---|---|---|---|---
| AA-AN | AA-ANN | AA-NNNN | AAA-NNNN
| AA-NN | AA-NNN | AN-NNNN |
| AN-NN | AN-NNN | NA-NNNN |
| NA-NN | NA-NNN | AAA-NNN |
| NN-AA | NNN-AA | AAN-NNN |
| NN-AN | NNN-AN | ANA-NNN |
| NN-NA | NNN-NA | ANN-NNN |
|  |  | NAA-NNN |
|  |  | NNN-AAA |
|  |  | NNNN-AA |
|  |  | NNNN-AN |
|  |  |  |

##  III Proposed Method

In this section, we present the architectural overview and data flow of the
entire network, and also describe the training process and provide
implementation details.

###  III-A Overview

The main purpose of this work is to introduce a real-time and highly accurate
license plate recognition method. Unlike segmentation-based methods which
require character segmentation followed by character recognition to predict
the license character sequence, the proposed model processes the whole license
plate image without segmentation. We also use CTC loss for segmentation-free
training so that we do not need to annotate the positions of characters in
license plates. Here RGB license plate images cropped from raw images are
considered as the inputs for the model rather than the raw images with
vehicles and other background material. Practically, this is easily
accomplished with various detection solutions. For example, previous work [5]
uses YOLOv2 [24] or the latest YOLOv3 [25] as the license plate detector and
vehicle detector. In this work, we simply use YOLOv3 as a license plate
detector in several experiments that require license plate detection. Since
this work mainly focuses on license plate recognition, details on detection
methods are not presented here.

Existing powerful networks such as AlexNet, VGGNet, or GoogLeNet are widely
popular in recent work. However, to build a fast and lightweight network,
wholesale use of these monolithic networks is not the best option. In this
work, the basic building blocks of the proposed networks were inspired either
by Inception ResNet v2 [11] or Xception [10], or both. The Inception network
is a deep neural network that achieves outstanding performance with a modest
number of parameters; due to its complicated design, however, it is still too
computationally costly for use in license plate recognition applications. In
[11], improved versions of Inception with residual connections prevent
vanishing gradients; that is, Inception ResNet v1 and v2 achieve slightly
better performance than their predecessors. On the other hand, the Xception
network, although not as powerful as the former, benefits from the adoption of
depthwise separable convolution and is thus significantly more efficient and
requires fewer parameters to match the performance of the former. To strike a
good balance between accuracy and computational efficiency, we make use of
portions of both to construct a license plate recognition-specific network
rather than directly use both heavy networks. The building blocks used to
construct license plate recognition model are illustrated in Figure 3. Note
that in the original works [10, 11], few implementation details are given, and
ReLU is used for activation in both works. However, in several experiments, we
found that replacing ReLU with LeakyReLU improved overall efficiency. Hence in
this work we use LeakyReLU as the activation function.

---
Figure 3: Implementation details of Xception and Inception building blocks originally proposed in [10, 11]. As names of these building blocks were not given, we name them according to their functionalities. Also, we adopt LeakyReLU as the activation function, whereas in the original work they did not reveal the implementation details. |
---|---
(a). Affine transformation model | (b). Proposed license plate recognition model
Figure 4: Detailed overview of proposed neural network architecture, including
affine transformation model and license plate recognition model.

###  III-B Affine Transformation Module

Tilted license plate images are common in real-world scenarios and often lead
to inaccurate prediction. To take this into account, we correct the tilt of
input images before the license plate recognition phase. To rectify tilted
license plate images, we apply affine transformation to unwarp them, so that
in the license plate recognition phase, every license plate resembles one
captured from the frontal view. To precisely unwarp distorted license plate
images, we train an affine transformation model to capture the license plate’s
four endpoints and deliver them to the correction algorithm. We experimented
with several affine transformation algorithms, and found the one provided by
the OpenCV library to be more stable than others. Thus we adopted it in our
implementation. Figure 4 shows the detailed architecture of the affine
transformation model. The results show that the affine transform model yields
a roughly 1 to 3 percent increase in recognition accuracy.

###  III-C License Plate Recognition Module

The network architecture of the proposed segmentation-free license plate
recognition model is presented in Figure 4. To begin with, image features are
extracted using a pre-trained CNN model that slides across a license plate
bounding box with an input size of 128 $\times$ 32, and principal component
analysis (PCA) is applied to reduce the feature dimensions, yielding feature
maps of size 32 $\times$ 8, which are then delivered to the DNN model with a
bidirectional CudnnLSTM layer followed by a fully connected layer to obtain
sequential features. Lastly, CTC is applied to decode the sequential LSTM
features and to predict the character sequence. The parameters of the network
architecture are shown in Table I.

During the experiments, we observed that although higher input sizes yielded
more accurate results, we found that $128\times 32$ is the best choice for the
network as a trade-off between accuracy and performance. To increase
efficiency, we also attempted to reduce the feature map size to $16\times 4$,
but this led to significantly inaccurate results.

###  III-D Model Training

Accurate prediction usually requires model training on a large number of
samples. However, existing datasets are commonly small-scale and thus do not
satisfy this requirement. To take this into account, we first trained our
model using an autoencoder, an unsupervised learning algorithm that allows the
model be trained on only a few labeled samples, after which transfer learning
is using to enhance the model. Secondly, we made use of the imgaug library to
augment the training samples by generating blurred and distorted images.
Examples produced by the imgaug library are shown in Figure 2. Further,
because some license plate letters are relatively infrequent in our region
(for example, I and O), we synthesized infrequent characters to produce
fabricated license plate images and used them to train the model. To do so, we
first randomly chose segregated characters cropped from the dataset, and then
measured the similarity between the currently selected character and the
partly completed license plate to determine whether the character should be
kept or discarded. To make the fabricated license plate resemble a real one,
we followed all legal rules of character combinations of license plates shown
in Table II to fabricate license plates. The experiment results showed that
fabricated samples have a character distribution similar to real license
plates, and that they benefit not only accuracy for license plates with
infrequent characters but also overall performance.

##  IV Experiment

In this section, we present our experimental results on the AOLP dataset, the
UFPR-ALPR dataset, and our own NP dataset. Although the CCPD dataset includes
a huge number of license plate images, because non-ASCII characters are used
to represent the province code, we did not evaluate on this dataset as our
model currently does not support non-ASCII characters. Here we consider both
recognition accuracy and running time as performance metrics and provide
several comparisons with state-of-the-art methods.

All of our models were trained using an NVIDIA GeForce RTX 2080 Ti with
specific images provided by each dataset; testing was conducted on a GTX 1070.

###  IV-A AOLP Dataset

TABLE III: AOLP results

|  AC (%) |  LE (%) |  RP (%) | Avg (%) | fps |
---|---|---|---|---|---|---
Hsu (2013) [3] | 88.50 | 86.60 | 85.70 | 86.93 | 7.00 |
LSTMs (2016) [13] | 94.85 | 94.19 | 88.38 | 92.47 | None |
DeepFCN (2016) [26] | 97.90 | 97.60 | 98.20 | 97.90 | 69.40 |
Zhuang (2018) [9] | 99.41 | 99.31 | 99.02 | 99.25 | 38.00 |
Proposed | 99.13 | 99.20 | 99.21 | 99.18 | 78.41 |
|  |  |  |  |  |

TABLE IV: AOLP error study

Image |  Ground truth |  Predicted | Cause of failure
---|---|---|---
| 0750J0 | 0750JC |  Character ‘0’ covered and incomplete. |
| 3L2556 | 3L25565 |  Background treated as part of license plate. |
| Y88096 | Y8B096 |  Blurry image leads to prediction failure. |
| 2E5507 | 2F5507 |  ‘E’ character is incomplete. |
|  |  |  |

The application-oriented license plate benchmark dataset (AOLP) includes 2,049
Taiwan license plate images and is divided into access control (AC), law
enforcement (LE), and road patrol (RP) scenarios. In this dataset, of the
three subsets, the LE subset seems to be the most challenging and most similar
to real-world scenarios. The LE subset contains 757 images of vehicles
violating traffic laws that were captured from roadside cameras under various
illumination and weather conditions; the image backgrounds are cluttered with
road signs and sometimes single frames even contain multiple license plates.
The AC and RP subsets, in contrast, are more or less rigid scenarios in that
most vehicles are very close to the camera.

In this experiment, we compared the proposed method with four approaches: [3],
[13], [26], and [9]. We followed the same training/test split as in [13] and
[9]: two subsets for training and one for testing. The experimental results
are shown in Table III. They show that not only does the recognition accuracy
of the proposed method outperform three previous works [3], [13] and [26], but
the recognition speed is also faster than these methods, even on less powerful
hardware such as the GTX 1070. While the error rate of the proposed work is
slightly higher than that of [9], our running speed is twice as fast, which
suggests that there is still room for improvement. For example, we could
sacrifice running time by using a heavier or more complicated model for more
accurate predictions. In Table IV we analyze a few failure cases. Recognition
failures are caused mainly by poor image quality, as mentioned in [9].
Moreover, the dataset contains images that are difficult even for human beings
to recognize, such as the first sample in the table, in which the first and
last characters are blocked by the surroundings.

###  IV-B UFPR-ALPR Dataset

The UFPR dataset consists of 4.5k images: 1.8k for testing, 1.8k for training,
and 900 for validation. In this dataset, images extracted from videos that are
captured from static cameras are divided into different subsets, each of which
contains 30 images with only one vehicle. Actually, every image is a single
frame of the original video, so the background has no significant change
compared to the AOLP dataset. We conducted experiments with the same settings
as in [5] except that we used YOLOv3 for vehicle and license plate detection,
unlike the previous experiment which requires only license plate recognition.
This dataset is thus not as challenging as the AOLP dataset, and image quality
is also far better than that of AOLP (images in the UFPR dataset are
$1920\times 1080$, whereas those in AOLP are around $320\times 480$ or even
lower), but to verify the robustness and the ability of environment adoptions,
we still evaluate the method on various datasets.

TABLE V: UFPR results

|  Accuracy (%) |  |  fps |
---|---|---|---|---
[5] | 78.33 (47/60) |  | 35 |
Sightound [5] | 70.00 (72/60) |  | None |
OpenALPR [5] | 56.67 (34/60) |  | None |
Proposed | 76.66 (46/60) |  | 43 |
|  |  |  |

TABLE VI: UFPR error study

Image |  Ground truth |  Predicted | Cause of failure
---|---|---|---
|  AKT 8174 | 0G27 |  Completely failed on multiple-row license plates. |
|  ALJ 9348 | B7Q2 |  Completely failed on multiple-row license plates. |
| IOZ3616 | I0Z3616 |  ‘O’ character extremely similar to ‘0’. |
| AIQ1Q56 | ATQ1Q56 |  ‘I’ character is shadowed. |
|  |  |  |

In this experiment, we compared the proposed method with [5]. Because each
subset contains images of the same vehicle, we could simply use the majority
vote to produce the final prediction result of every subset as in [5]. The
results are shown in Table V: at first glance, it appears that our method
performs worse than [5]. After carefully inspection, we find that our
prediction model does not correctly predict license plates with two rows. Of
the 60 different vehicles in the testing set, 12 motorcycles are all equipped
with two-row license plates. In our network design, as the CTC model treats
the license plate as a collection of characters within a single row, the
current proposed method does not fully recognize license plates that have
multiple rows. Again, we demonstrate and analyze failed samples in Table VI.

TABLE VII: Secondary UFPR results

|  Accuracy (%) |  |  fps |
---|---|---|---|---
Proposed (without majority vote) | 86.13 (1292/1500) |  | 45.24 |
Proposed (with majority vote) | 94.00 (47/50) |  | 44.57 |
Proposed (without object detection and majority vote) | 93.13 (1397/1500) |  | 69.75 |
|  |  |  |

Due to time constraints, we were unable to modify the proposed method to
account for multiple-row license plates. We instead conducted another
experiment on the same dataset but without the motorcycles. We randomly chose
50 subsets that consisted only of images of non-motorcycle vehicles from the
entire dataset, with the remaining images for training. The results are shown
in Table VII: the results without multiple-row license plates indicate that
the proposed method is still robust and sufficiently accurate under various
environments in different countries, although it is currently limited to
single-row license plates. Due to the segmentation-free recognition
architecture and the latest YOLOv3 improvements, the proposed method clearly
achieves a significantly higher fps than [5]. If we entirely preclude license
plate detection, the proposed method achieves 94.00% accuracy, whereas [5]
yields 95.97% and 90.37% accuracies on character segmentation and recognition
respectively. Thus the proposed method indeed outperforms their work.

TABLE VIII: NP-ALPR error study

Image |  Ground truth |  Predicted | Cause of failure
---|---|---|---
|  ASM0177 | ASMB177 |  ‘0’ character distorted and shadowed. |
|  532NEQ | 532AEQ |  Image is distorted. |
| A1561D8 | A15610B |  Image is blurred and tilted |
| AKL3269 | AAL3269 |  Image is extremely blurred. |
| MAE9576 | M1E9576 |  Image is extremely blurred. |
| T6A769 | TBA769 |  Image is extremely blurred. |
| 7692KJ | 7092KJ |  Image is extremely blurred. |
| 3G7585 | 367585 |  Image is extremely blurred. |
|  |  |  |

###  IV-C NP-ALPR Dataset

The last experiment was conducted on the proposed NP-ALPR dataset. Because our
model was already pre-trained on a similar private dataset, for this
experiment we randomly selected 1500 images on which to evaluate the proposed
method. As mentioned in the previous section, as diverse conditions such as
weather conditions and times of day are covered in this dataset, this would be
the most practical test in this paper. The proposed method correctly
recognized 1479 license plates (98.60%) at more than 70 fps on a GTX 1070 GPU.
Analysis of the failed samples is given in Table VIII. In the experiment, we
observe that most error cases were at night, during which rear lights tend to
overexpose cameras, thus increasing the difficulty of license plate
recognition. Another difficulty is the long range and tilted angles between
the camera and vehicles; also, the captured images are often seriously blurred
or distorted despite the large $1920\times 1080$ images.

We note that these extremely distorted and blurred samples are difficult even
for humans to fully recognize. Nevertheless, the proposed method still
achieves over 98.60% recognition accuracy, which is outstanding in and of
itself.

##  V Conclusion

In this paper, we present a novel real-time network architecture combined with
the latest works which not only facilitates highly efficient recognition for
deployment in real-world applications but also yields improved overall
accuracy. We evaluate the proposed method on three different datasets and
compare the results with state-of-the-art approaches, showing that it achieves
outstanding recognition accuracy and fast running times. Even though our
method does not significantly outperfom all previous works, it runs on cheaper
hardware such as the GTX 1070 GPU, unlike most previous works which use the
GTX 1080 TI or higher-level hardware. As future work, we intend to introduce
deblurring algorithms into our method to improve accuracy under environments
such as that in Table VIII, and plan to further optimize the affine
transformation model to reduce the entire running time and errors caused by
incorrect tilting correction. In addition, we plan to recognize multiple-row
license plates to support scenarios such as that in UFPR-ALPR. Finally, we
also intend to support recognition of license plates with non-ASCII characters
such as those in China and Japan.

## References

  * [1] G.-S. Hsu, P. Alexandra, J.-C. Chen, F. Yeh, and M.-H. Chen, “License plate recognition for categorized applications,” in _Proceedings of 2011 IEEE International Conference on Vehicular Electronics and Safety_. IEEE, 2011, pp. 220–225.
  * [2] K. W. Maglad, “A vehicle license plate detection and recognition system,” _Journal of Computer Science_ , vol. 8, no. 3, p. 310, 2012.
  * [3] G.-S. Hsu, J.-C. Chen, and Y.-Z. Chung, “Application-oriented license plate recognition,” _IEEE transactions on vehicular technology_ , vol. 62, no. 2, pp. 552–561, 2012.
  * [4] T. K. Cheang, Y. S. Chong, and Y. H. Tay, “Segmentation-free vehicle license plate recognition using convnet-rnn,” _arXiv preprint arXiv:1701.06439_ , 2017.
  * [5] R. Laroca, E. Severo, L. A. Zanlorensi, L. S. Oliveira, G. R. Gonçalves, W. R. Schwartz, and D. Menotti, “A robust real-time automatic license plate recognition based on the yolo detector,” in _2018 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2018, pp. 1–10.
  * [6] C.-H. Lin, Y.-S. Lin, and W.-C. Liu, “An efficient license plate recognition system using convolution neural networks,” in _2018 IEEE International Conference on Applied System Invention (ICASI)_. IEEE, 2018, pp. 224–227.
  * [7] F. Xie, M. Zhang, J. Zhao, J. Yang, Y. Liu, and X. Yuan, “A robust license plate detection and character recognition algorithm based on a combined feature extraction model and bpnn,” _Journal of Advanced Transportation_ , vol. 2018, pp. 1–14, 09 2018.
  * [8] J. Špaňhel, J. Sochor, R. Juránek, A. Herout, L. Maršík, and P. Zemčík, “Holistic recognition of low quality license plates by cnn using track annotated data,” in _2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)_. IEEE, 2017, pp. 1–6.
  * [9] J. Zhuang, S. Hou, Z. Wang, and Z.-J. Zha, “Towards human-level license plate recognition,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 306–321.
  * [10] F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1251–1258.
  * [11] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning,” in _Thirty-First AAAI Conference on Artificial Intelligence_ , 2017.
  * [12] Z. Abderaouf, B. Nadjia, and O.-K. Saliha, “License plate character segmentation based on horizontal projection and connected component analysis,” in _2014 World Symposium on Computer Applications & Research (WSCAR)_. IEEE, 2014, pp. 1–5.
  * [13] H. Li and C. Shen, “Reading car license plates using deep convolutional neural networks and lstms,” _arXiv preprint arXiv:1601.05610_ , 2016.
  * [14] M. Khalil, “Car plate recognition using the template matching method,” _International Journal of Computer Theory and Engineering_ , vol. 2, no. 5, p. 683, 2010.
  * [15] E. R. Lee, P. K. Kim, and H. J. Kim, “Automatic recognition of a car license plate using color image processing,” in _Proceedings of 1st International Conference on Image Processing_ , vol. 2. IEEE, 1994, pp. 301–305.
  * [16] M.-A. Ko and Y.-M. Kim, “License plate surveillance system using weighted template matching,” in _32nd Applied Imagery Pattern Recognition Workshop, 2003. Proceedings._ IEEE, 2003, pp. 269–274.
  * [17] M. Zahedi and S. M. Salehi, “License plate recognition system based on sift features,” _Procedia Computer Science_ , vol. 3, pp. 998–1002, 2011.
  * [18] Y. Wang, X. Ban, J. Chen, B. Hu, and X. Yang, “License plate recognition based on sift feature,” _Optik_ , vol. 126, no. 21, pp. 2895–2901, 2015.
  * [19] A. E. Abdel-Hakim and A. A. Farag, “Csift: A sift descriptor with color invariant characteristics,” in _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)_ , vol. 2. Ieee, 2006, pp. 1978–1983.
  * [20] K. S. Durgesh and B. Lekha, “Data classification using support vector machine,” _Journal of theoretical and applied information technology_ , vol. 12, no. 1, pp. 1–7, 2010.
  * [21] D. Menotti, G. Chiachia, A. X. Falcão, and V. O. Neto, “Vehicle license plate recognition with random convolutional networks,” in _2014 27th SIBGRAPI Conference on Graphics, Patterns and Images_. IEEE, 2014, pp. 298–303.
  * [22] G. R. Gonçalves, S. P. G. da Silva, D. Menotti, and W. R. Schwartz, “Benchmark for license plate character segmentation,” _Journal of Electronic Imaging_ , vol. 25, no. 5, p. 053034, 2016.
  * [23] Z. Xu, W. Yang, A. Meng, N. Lu, H. Huang, C. Ying, and L. Huang, “Towards end-to-end license plate detection and recognition: A large dataset and baseline,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 255–271.
  * [24] J. Redmon and A. Farhadi, “Yolo9000: better, faster, stronger,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 7263–7271.
  * [25] ——, “Yolov3: An incremental improvement,” _arXiv preprint arXiv:1804.02767_ , 2018.
  * [26] Y. Wu and J. Li, “License plate recognition using deep fcn,” in _International Conference on Cognitive Systems and Signal Processing_. Springer, 2016, pp. 225–234.