# Deep Neural Networks on EEG Signals to Predict Auditory Attention Score
Using Gramian Angular Difference Field

Mahak Kothari1, Shreyansh Joshi1, Adarsh Nandanwar1, Aadetya Jaiswal1, Veeky
Baths2 1Department of Computer Science & Information Systems
2Cognitive Neuroscience Lab, Department of Biological Sciences
BITS Pilani K K Birla Goa Campus, Goa, India
{f20180232, f20180097, f20180396, f20180096<EMAIL_ADDRESS>
###### Abstract

Auditory attention is a selective type of hearing in which people focus their
attention intentionally on a specific source of a sound or spoken words whilst
ignoring or inhibiting other auditory stimuli. In some sense, the auditory
attention score of an individual shows the focus the person can have in
auditory tasks. The recent advancements in deep learning and in the non-
invasive technologies recording neural activity beg the question, can deep
learning along with technologies such as electroencephalography (EEG) be used
to predict the auditory attention score of an individual? In this paper, we
focus on this very problem of estimating a person’s auditory attention level
based on their brain’s electrical activity captured using 14-channeled EEG
signals. More specifically, we deal with attention estimation as a regression
problem. The work has been performed on the publicly available Phyaat dataset.
The concept of Gramian Angular Difference Field (GADF) has been used to
convert time-series EEG data into an image having 14 channels, enabling us to
train various deep learning models such as 2D CNN, 3D CNN, and convolutional
autoencoders. Their performances have been compared amongst themselves as well
as with the work done previously. Amongst the different models we tried, 2D
CNN gave the best performance. It outperformed the existing methods by a
decent margin of 0.22 mean absolute error (MAE).

###### Index Terms:

Auditory attention, Gramian Angular Difference Field, Electroencephalography.

##  I Introduction

Attention plays a major role in our daily lives. Almost every conscious move
we make requires attention. Clearly, understanding attention and improving it
can be the key to ameliorate the effectiveness of many tasks. Auditory
attention is a form of attention, a cognitive process wherein the attention of
the person is gauged by listening/hearing tasks. More specifically, it refers
to the process by which a person focuses selectively on a stimulus of interest
whilst ignoring other stimuli. It is important to note that such selective
hearing is not some kind of a physiological disorder like autism, dementia but
rather, it is the ability of humans to intentionally block out certain sounds
and noise and to focus on selective ones.

The next obvious question that arises is that why is auditory attention
relevant in the real world? Sounds in everyday life seldom appear in
isolation. Both humans and machines are constantly flooded with a cacophony of
sounds that need to be filtered and scoured for relevant information— famously
referred to as the ’cocktail party problem’.

This paper makes a conscious effort towards predicting the auditory attention
level of an individual by analyzing the EEG signals produced by the person’s
brain. Since the world is moving towards automation in almost every domain
these days, taking a step in that direction by automating the prediction of
the attention score of a person, just by using EEG signals from the brain, was
a big source of motivation for our work.

We, in this paper, have tried to show that level of attention (attention
score) can be predicted based on the EEG signals recorded from the brain. EEG
is a non-invasive technique used to measure the electrical activity of the
brain regions [1, 2]. A lot of research on attention has been done in the past
using EEG signals. Studies have shown that EEG data from different frequency
bands can be used to draw inferences on visual and auditory attention. In
2006, Sauseng et al. [3] showed that shift of visual-spatial attention is
selectively associated with human EEG alpha activity. In 2013, Gola et al. [4]
showed that EEG beta-band activity is related to attention and attentional
deficits in the visual performance of elderly subjects. Decreased activity of
beta-band reflected the difficulty in activation of attentional processes
(alertness deficits in short delay condition) and deficits in sustaining those
processes (longest delay). As recently as in 2018, Wang et al. [5] used deep
learning techniques like convolutional neural networks to work on visual
attention. This method was based on an end-to-end deep learning architecture,
which resulted in good performance while predicting human eye fixation with
view-free scenes.

However, unlike visual attention, auditory attention analysis based on EEG
signals is a relatively lesser-explored and newer field, although some efforts
have been made to analyze auditory attention in the past using EEG signals. In
2014 Treder et al. [6] showed that by observing the EEG signals, we could
infer the attention paid to a particular instrument in polyphonic music. With
a mean accuracy of 91%, they classified the attended instrument. Another study
in 2017 by Fiedler et al. [7] showed that using the EEG response recorded from
short-distance configurations consisting only of a single in-ear-EEG electrode
and an adjacent scalp-EEG electrode, an individual’s attention focus could be
detected. This information can be used in hearing aids to identify the
listener’s focus of attention in concurrent-listening (’cocktail party’)
scenarios. Similarly, by analyzing the electrode potential, we may be able to
estimate the auditory attention level. How noise affects the attention levels
of an individual was shown by Das et al. [8]. They showed that it is possible
to identify the speaker to which the listener is actively listening in an
environment with multiple speakers. They observed that attention in moderate
background noise is greater than in a no-background noise environment. A
further increase in the noise was found to bring about a remarkable drop in
attention level. In our work, the EEG data source [9] features recordings at 6
different signal-to-noise ratios (SNR). This noise data is crucial in
accurately estimating auditory attention levels.

In this paper, we aim to estimate auditory attention using deep learning
architectures like 2D CNN, 3D CNN, and convolutional autoencoders with random
forest and XGBoost regressor. This is not the first work in the field
involving convolutional neural networks (CNN) [10]. In [11], various CNN-based
approaches were used to decode attention and detect the attended speaker in a
multi-speaker scenario. The majority of time-series classification work on
signals till date (involving CNNs) has been done using 1D CNN. Kashiparekh et
al. [12], Fawaz et al. [13] and Wang et al. [14] succeeded in demonstrating
the significant results of 1D CNN. However, a major limitation in using 1D CNN
is the extensive hyperparameter tuning accompanying it [15]. Therefore, we
restrained ourselves from using 1D CNN. Recently, 2D CNNs have been applied to
time-series data by transforming them into 3D images. Hatami et al. [16] were
among the very first researchers to propose encoding of the time-series data
into images using Gramian Angular Fields (GAF) and then applied the CNN-based
architecture. GAF alongside CNN [17], has also been previously used for the
classification of motion imagery from EEG signals with high accuracy. In [18],
the EEG time-series data were transformed to RGB images using Gramian Angular
Summation Field (GASF), and various deep neural networks were used to diagnose
epilepsy. Using a similar method, we plan to transform the time-series data
into multi-channel images using Gramian Angular Difference Field (GADF).
The remaining part of the paper is organized as follows. Section 2 and 3 talk
about the dataset on which the work has been done and data preprocessing using
GADF, respectively. Section 4 contains the methodology, which is further
divided into three subsections. Section 5 talks about the evaluation metric
used. Section 6 presents the results and inferences we obtained from the
various experiments we performed, and finally, section 7 wraps up the paper
with the conclusion and future work.

##  II Dataset

In this paper, we have used the PhyAAt dataset [9], which contains the
physiological responses of 25 subjects collected from an experiment on
auditory attention. The chosen subjects were university students aged between
16 to 34. All participants were non-native English speakers, out of which 21
of them were male, and 4 were female.

The following procedure was performed to obtain the dataset. Each participant
was subjected to 3 tasks - listening, writing, and resting, across multiple
trials. More formally, the experiment consisted of 144 trials for each
subject, where each trial was divided into the 3 aforementioned tasks. During
each listening phase, an audio clip was played that contained English language
sentences of varying lengths (ranging from 3 to 13 words) with 6 different
levels of background noise. A good chunk of the sentences were semantic in
nature, while others were non-semantic.

During each writing phase, the subject was asked to write the words in the
correct order. The attention score was calculated using the number of words
present in the same positions in the listening and writing phase for a given
sentence. The formula for the same has been given in Eq. 1

$Attention\>score=\frac{number\>of\>correct\>words}{total\>number\>of\>words}$
(1)

A 14-channel EEG system was used to record the brain signals of the subjects
during the experiment at a sampling rate of 128Hz. A total of 144 stimuli were
collected for each participant. In our work, we have used signals only from
the listening phase for the prediction.

Phyaat dataset is one of the best publicly available datasets for determining
auditory attention using EEG signals. Albeit small in size, the diversity of
subjects, as well as the rigour with which the experiments were performed (for
instance, to better estimate the auditory attention, the experiment also
included some non-semantic English sentences as well as 6 different levels of
background noise) increases the possibility of a model to generalize well on
real-world data, after being trained on it.

##  III Data pre-processing using GAF

The deep learning boom is largely fueled by the success that deep learning
based models have had in computer vision [19] and speech recognition [20, 21].
The well-established success of CNNs in feature extraction from images was one
of our main motivations for choosing computer vision based models over RNNs
and LSTMs that are typically used while dealing with time-series data. To do
that, we had to first convert the time-series data into images, for which we
have used the technique of Gramian Angular Field (GAF).

The dataset contained the time-series data of EEG signals, which was converted
into a GAF image [22]. GAF is simply a polar coordinate representation of the
time-series data, which preserves the temporal dependency. Gramian Difference
Angular Field (GADF) is a type of GAF in which trigonometric difference
between each point is calculated to create a matrix with the temporal
correlation amongst different time intervals.

Given below is the step-by-step procedure, we followed to create a GADF image
from the EEG signals obtained from a particular electrode. Fig. 1. visually
depicts that procedure.

  * •

Step 1: Let $X$ (Eq. 2) be a time-series datum containing $k$ real values.
Apply a rescaling function $f$ (Eq. 3) with domain ${\mathbb{R}}$ and range
[-1,1]. This is essential so as to apply $arccos$ function, while converting
them into polar coordinates as shown in step 2.

$X=\\{x_{1},x_{2},...,x_{k}\\}\\\ $ (2)
$f(x_{i})=\frac{(x_{i}-max(X))+(x_{i}-min(X))}{max(X)-min(X)}$ (3)

  * •

Step 2: Convert the rescaled data into polar coordinates by encoding the value
as the angular cosine $\phi$ and the time stamp as the radius $r$, as can be
seen below.

$\begin{cases}\phi_{i}=arccos(f(x_{i})),\>1\leq i\leq k\\\
r=\frac{i}{k}\end{cases}$

  * •

Step 3: Calculate the GADF matrix

$GADF=\begin{bmatrix}\sin(\phi_{1}-\phi_{1})&..&\sin(\phi_{1}-\phi_{k})\\\
\sin(\phi_{2}-\phi_{1})&..&\sin(\phi_{2}-\phi_{k})\\\ :&:&:\\\
\sin(\phi_{k}-\phi_{1})&..&\sin(\phi_{k}-\phi_{k})\\\ \end{bmatrix}$

Figure 1: Time-series to GADF conversion.

The above-mentioned procedure was followed for the EEG signals from all the 14
electrodes planted in a subject’s head, resulting in 14 images which were
eventually stacked one behind the other, resulting in a single 14-channel GADF
image. However, these images were not of fixed dimensions. A way to deal with
this was to pad all those images so that they become of the same size.
However, we refrained from doing so as the padding might result in the loss of
information from some images, possibly due to the variability in image sizes.
Instead, the models were designed in a way that they could incorporate images
of different sizes.

##  IV Methodology

For a total of 25 subjects, we had about 3600 datapoints (144x25, 144 trials
each for 25 subjects) available for our work, out of which 300 datapoints were
chosen to be kept for validation, and the remaining 90% of the data would be
used for training purposes. It is worthwhile to note that this training and
testing data was not fixed during the entire duration of our experiment. The
dataset (consisting of 3600 data points as mentioned previously) was randomly
divided into 12 sets or ’folds’, and experiments were conducted by choosing
one fold at a time as the validation set, while the rest of the 11 folds
served as the training set.

This technique was employed to ensure more reliable and reproducible results
since not much work has been done in this field (on the Phyaat dataset) to
compare our work with. The value of 12 (for the number of folds) was chosen
after careful deliberation and taking into account the variance and standard
deviation of the results. A higher number of folds imply a larger computation
time and a greater variance, as each validation fold has lesser datapoints. On
the other hand, an extremely smaller number of folds is detrimental in the way
that a larger chunk of data would not be available for training, thus leading
to underfitting, with the model not being able to properly discern the
patterns present in the dataset. The sweet spot for the number of folds was
found by experimenting with different values. It was found that any value
between 11 to 13 was giving similar results. We chose 12, since it divides
3600 completely and so we could ensure that all datapoints get a chance to be
a part of validation set. For each of the following architectures, we employed
the 12-fold cross-validation and calculated the mean and standard deviation of
the MAE.

We trained and tested the following computer vision based models -

###  IV-A 2D CNN

A 2D CNN is capable of only extracting spatial features in images whilst
neglecting the temporal correlation since the kernel moves only in the spatial
directions. The architecture we used comprises 4 blocks of convolution layers
followed by maxpooling to reduce the dimensions as the depth of the network
increases. The number of filters increase with the depth, but the dimensions
of filters in both Conv2d layers, as well as the MaxPool2d layers, remain the
same ((3,3) and (2,2) respectively). The activation function ReLU was used
between the layers, as studies have proven empirically that deep networks,
when training with ReLU, tend to converge quickly and more reliably as against
any other activation function.

Generally, a CNN works on the image of a fixed dimension because the fully
connected layer requires a specific number of neurons, but in our case, as
explained previously, the dimensions of input images were not fixed, so in the
last of the 4 blocks described above, we introduced an adaptive maxpooling
layer (instead of the normal maxpooling) before flattening the representation.
’AdaptiveMaxPool2d’ converts the input of any dimension into the output of a
fixed dimension. Following it, our architecture has 3 dense layers with 128,
16 and 1 neurons. Dropout was used in the first of the 3 FC layers to combat
overfitting. We chose against using dropout in the convolutional layers as
those are the layers responsible for learning the features, and hence we did
not want any loss of useful information during the feature learning process.
Fig. 2 contains the detailed architecture.

Figure 2: The architecture of 2D CNN that gave the best result.

###  IV-B 3D CNN

Generally, 3D CNNs are used to learn the representations for volumetric data
like videos where temporal relations among different frames are captured along
with spatial relations of each frame. The third axis intrinsically links the
frames together and so cannot be ignored. Conventionally, a 3D CNN operates on
4 dimensions of an image - width, height, depth, channels as against 3 in the
case of 2D CNN. In contrast to 2D CNNs, where the filters are used only in
spatial directions, the 3D CNN uses the filters in a temporal direction also.

Since we were dealing with 14-channel images, we felt this architecture would
be ideal for our work. We believed that there might be some relationship
amongst the different channels of the EEG signals that can be captured using a
3D CNN (and probably not by 2D CNN). Here, we chose to go with a shallower
network, as compared to the 2D CNN one, for the simple reason that 3D CNN is
very computationally demanding, and we had limited memory resources at hand.
This time we had 3 blocks of convolution layers, followed by maxpooling. As in
the case of 2D CNN, the number of filters increase with the depth
(64,128,128), but the dimensions of filters in both Conv3d layers, as well as
the MaxPool3d layers remain the same ((3,5,5) and (1,3,3) respectively). The
activation function ReLU was used between the layers. The 3D version of
adaptive maxpooling, AdaptiveMaxPool3d was used before flattening the
extracted features. Following it, we used 3 dense layers having 512, 16 and 1.

###  IV-C Convolutional Autoencoder

Autoencoder [23] is a kind of artificial neural network used to encode and
compress data. They work by compressing the input first and later
reconstructing the output from the compressed representation of the data. In
this paper, we have used a nine-layered convolutional autoencoder to learn the
encoding of the data. Convolutional autoencoder is a variant of CNNs that are
used as the tools for unsupervised learning of convolution filters. They are
used to preserve the spatial information of an image and work differently from
general autoencoders that completely ignore the 2D image structure. Since our
inputs consisted of images (after applying GADF), it made sense to use CNN
(convnets) as encoders and decoders. To build any autoencoder, three things
are required: an encoding function, a decoding function, and a distance
function between the amount of information loss between the compressed
representation of the input data and the decompressed representation (i.e., a
”loss” function).

Our encoder is comprised of 3 blocks of convolution layers and maxpooling. As
in the case of 2D CNN and 3D CNN, the number of filters increase with the
depth of the network (64,128,128) and the kernel size remains the same
throughout for both Conv2d layers and MaxPool2d layers ((3,3) and (2,2),
respectively). The last of the 3 blocks had the MaxPool2d replaced with
AdaptiveMaxPool2d to deal with the variable size of input images. The encoder
finally terminates with an FC layer with 512 neurons. The decoder was designed
in the reverse order as the encoder. It begins with an FC layer, followed by 3
blocks of maxunpooling and transpose convolution. In this case, with the
increasing depth of the network, the number of filters keeps on decreasing
(128,128,64). The kernel size remains the same throughout for both MaxUnpool2d
and ConvTranspose2d ((2,2) and (3,3) respectively). The decoder finally
creates an image with the same shape as the input image to the encoder. For
both the encoder, as well as the decoder, the activation function ReLU was
used between the layers. The distance function used was MAE, which actually
calculates the pixel-wise difference between the original input image and the
new image generated by the decoder.

The encoded data was passed to two of the most promising machine learning
regression models: First, the random forest regressor [24], which is an
ensembling technique that uses the output of multiple decision trees and takes
the mean of all of them for the final output. Second, the XGBoost regressor
[25], a state-of-the-art machine learning algorithm that uses the boosting
ensemble technique. In boosting, multiple weak learners are trained
sequentially such that each tree tries to reduce the error of the previous one
[26]. The hyperparameters of both the machine learning models were found by
fine-tuning using GridSearchCV.

In all the 3 architectures, the models were trained using Adam optimizer for
about 15 epochs. It was observed that training any further, caused the models
to overfit. The initial learning rate was fixed at about 0.002-0.003 in each
case, and learning rate decay was used to help the model converge and
generalize better. Mean Squared Error (MSE) was the loss function used during
the training process.

##  V Evaluation Metric

The closeness of the actual value to the predicted value of the attention
score was used to evaluate the performance of our models. The metric we have
used for the attention score estimation is the mean absolute error or MAE that
captures the average magnitude of error in a set of predictions. MAE
calculates the mean of the absolute error between actual score and predicted
score as defined by the Eq. 4.

$MAE=\frac{1}{n}\sum_{j=1}^{n}|y_{j}-\hat{y_{j}}|$ (4)

Here $n$ is the number of testing samples, $y_{j}$ denotes the actual
attention score, and $\hat{y_{j}}$ is the predicted score of the j-th sample.

##  VI Results and inferences

Table I compares the performance of different models by displaying the mean
and standard deviation of the validation MAE each model obtained, after taking
into account the 12-fold cross-validation, we had performed during training.
While the mean provides a good measure of how accurately the model predicted
the attention score on average, the standard deviation shows the amount of
variance in the predicted values over the 12 folds of validation data. A
higher variance would typically mean that the trained model is prone to be
affected by outliers and might as well, have a lesser generalization ability.

2D CNN was found to provide the best results when it comes to the mean MAE
predicted over the validation set. We had expected 3D CNN to perform better
than 2D CNN, since it has the potential to capture the relations between the
14 channels of the image. The most plausible explanation for it not performing
up to our expectations is the shallowness of the network that we had trained
our model on. We had limited memory resources at our disposal and coupled with
the 14-channel input image, and it was not possible to train a deeper network
that could extract features in a better way. This is something we hope to
address in our future work.

TABLE I: Comparision of Different Models Models: | Validation MAE (mean $\pm$ std)
---|---
2D CNN | $\boldsymbol{29.43}\pm\boldsymbol{1.81}$
3D CNN | $29.72\pm 1.76$
Autoencoder with Random Forest | $30.16\pm 1.65$
Autoencoder with XGBoost | $29.99\pm 1.91$
SVM [9] |  $29.65\pm 4.75$ (approx.)

However, despite the shallowness of the network, it can be seen that 3D CNNs
have performed the second-best among the networks, and in fact, almost as good
as the owners of the Phyaat dataset were able to achieve. Even though it was
not able to perform up to its full potential because of the resource
constraints, it managed to outperform a couple of other architectures. Among
the convolutional autoencoders, the better results of the XGBoost model can be
ascribed to the fact that we found it to generalize better, thereby
overfitting lesser as compared to the model with random forest.

Figure 3: Model-wise plots of validation MAE for one of the 12 folds.

Another crucial observation we made during training was that for all
architectures (and in almost all 12 runs of cross-validation), the training
loss vs. epochs graphs were highly volatile, which seems to be a consequence
of the batch size we chose. This is corroborated by Fig. 3, which displays the
epoch-wise perfeormance our models achieved on one of the 12 folds used for
validation. The fact that Pytorch does not allow images of different
dimensions in a single batch forced us to use a batch size of 1. Essentially,
this annulled the concept of batch training in our work. Due to this, we could
not add batch normalization, which resulted in slower convergence of our
models. It was also observed that despite adding multiple layers of dropouts,
our models were somewhat prone to overfitting, possibly because we omitted
batch normalization layers in our architecture and because of the small size
of the dataset, containing only about 3600 data points.

##  VII Conclusion

In this paper, we proposed a way to predict the auditory attention score from
EEG signals of a person using computer vision based architectures. In order to
apply deep learning algorithms, we converted the time-series data into images
using a technique known as GADF. We used adaptive maxpooling in our
architectures to combat the varying sizes of images. It was found that 2D CNN
gave the best results among all the models, in fact even bettering existing
performances on the chosen dataset [9].

We plan to extend our work by training the models on a larger dataset, which
will give us more scope to experiment and improve the generalizability of our
models. We plan to train much deeper 3D CNN models (with better memory
resources at hand) and advanced architectures such as 2D CNNs + LSTMs that
would help us better capture interchannel information. We also plan to use
better data (signal) preprocessing techniques before converting the data into
images, allowing models to extract features better and consequently, learn
faster. We hope this work of ours serves as a platform to improvise and
innovate further and thereby contributing to the research community.

## References

  * [1] J. S. Kumar and P. Bhuvaneswari, “Analysis of electroencephalography (eeg) signals and its categorization–a study,” International Conference on Modelling Optimization and Computing, vol. 38, pp. 2525–2536, 2012.
  * [2] S. Vaid, P. Singh, and C. Kaur, “Eeg signal analysis for bci interface: A review,” in 2015 Fifth International Conference on Advanced Computing Communication Technologies, pp. 143–147, February 2015.
  * [3] P. Sauseng, W. Klimesch, W. Stadler, M. Schabus, M. Doppelmayr, S. Hanslmayr, W. Gruber, and N. Birbaumer, “A shift of visual spatial attention is selectively associated with human eeg alpha activity,” The European journal of neuroscience, vol. 22, pp. 2917–2926, January 2006.
  * [4] M. Gola, M. Magnuski, I. Szumska, and A. Wróbel, “Eeg beta band activity is related to attention and attentional deficits in the visual performance of elderly subjects,” Psychophysiology in Australasia - ASP conference - November 28-30 2012, vol. 89, no. 3, pp. 334–341, 2013.
  * [5] W. Wang and J. Shen, “Deep visual attention prediction,” IEEE Transactions on Image Processing, vol. 27, no. 5, pp. 2368–2378, 2018.
  * [6] M. S. Treder, H. Purwins, D. Miklody, I. Sturm, and B. Blankertz, “Decoding auditory attention to instruments in polyphonic music using single-trial EEG classification,” Journal of Neural Engineering, vol. 11, p. 026009, March 2014.
  * [7] L. Fiedler, M. Wöstmann, C. Graversen, A. Brandmeyer, T. Lunner, and J. Obleser, “Single-channel in-ear-EEG detects the focus of auditory attention to concurrent tone streams and mixed speech,” Journal of Neural Engineering, vol. 14, p. 036020, April 2017.
  * [8] N. Das, A. Bertrand, and T. Francart, “EEG-based auditory attention detection: boundary conditions for background noise and speaker positions,” Journal of Neural Engineering, vol. 15, p. 066017, October 2018.
  * [9] N. Bajaj, J. R. Carrión, and F. Bellotti, “Phyaat: Physiology of auditory attention to speech dataset,” 2020.
  * [10] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
  * [11] L. Deckers, N. Das, A. H. Ansari, A. Bertrand, and T. Francart, “Eeg-based detection of the attended speaker and the locus of auditory attention with convolutional neural networks,” 2018.
  * [12] K. Kashiparekh, J. Narwariya, P. Malhotra, L. Vig, and G. Shroff, “Convtimenet: A pre-trained deep convolutional neural network for time series classification,” in 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1–8, 2019.
  * [13] H. Ismail Fawaz, G. Forestier, J. Weber, L. Idoumghar, and P.-A. Muller, “Deep learning for time series classification: a review,” Data Mining and Knowledge Discovery, vol. 33, pp. 917–963, July 2019.
  * [14] Z. Wang, W. Yan, and T. Oates, “Time series classification from scratch with deep neural networks: A strong baseline,” in 2017 International Joint Conference on Neural Networks (IJCNN), pp. 1578–1585, 2017.
  * [15] W. Tang, G. Long, L. Liu, T. Zhou, J. Jiang, and M. Blumenstein, “Rethinking 1d-cnn for time series classification: A stronger baseline,” ArXiv, vol. abs/2002.10061, 2020.
  * [16] N. Hatami, Y. Gavet, and J. Debayle, “Classification of time-series images using deep convolutional neural networks,” in International Conference on Machine Vision, 2018.
  * [17] A. Bragin and V. Spitsyn, “Electroencephalogram analysis based on gramian angular field transformation,” 29th International Conference on Computer Graphics and Vision, GraphiCon 2019, vol. 2485, pp. 273–275, 2019.
  * [18] P. T. Krishnan, B. Parvathavarthini, U. Tanik, V. Rajinikanth, S. Kadry, and K. Kamalanand, “Implementation of deep neural networks to classify eeg signals using gramian angular summation field for epilepsy diagnosis,” ArXiv, vol. abs/2003.04534, 2020.
  * [19] A. Voulodimos, N. Doulamis, A. Doulamis, and E. Protopapadakis, “Deep learning for computer vision: A brief review,” Computational Intelligence and Neuroscience, vol. 2018, pp. 1–13, February 2018.
  * [20] A. B. Nassif, I. Shahin, I. Attili, M. Azzeh, and K. Shaalan, “Speech recognition using deep neural networks: A systematic review,” IEEE Access, vol. 7, pp. 19143–19165, 2019.
  * [21] A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, and A. Ng, “Deepspeech: Scaling up end-to-end speech recognition,” December 2014.
  * [22] Z. Wang and T. Oates, “Encoding time series as images for visual inspection and classification using tiled convolutional neural networks,” in Workshops at the Twenty-Ninth AAAI Conference on Artificial Intel- ligence, January 2015.
  * [23] G. Hinton and R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” Science, vol. 313, no. 5786, pp. 504 – 507, 2006.
  * [24] L. Breiman, “Random forests,” Machine Learning, vol. 45, pp. 5–32, October 2001.
  * [25] T. Chen and C. Guestrin, “Xgboost,” Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, August 2016\.
  * [26] Y. Freund and R. E. Schapire, “A short introduction to boosting,” in In Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence, pp. 1401–1406, Morgan Kaufmann, 1999.


# Systematically designing better instance counting models on cell images with
Neural Arithmetic Logic Units

Ashish Rana1, Taranveer Singh2, Harpreet Singh3, Neeraj Kumar4 and Prashant
Singh Rana5  Department of Computer Science, TIET Patiala, Punjab, India.
Email: 1{arana_be15, tsingh_me17, harpreet.s, neeraj.kumar and prashant.singh
<EMAIL_ADDRESS>
###### Abstract

The big problem for neural network models which are trained to count instances
is that whenever testing range data goes to higher counts than training range
data generalization error increases i.e. prediction error for images that are
outside training range increases. Consider the case of automating cell
counting process where more dense images with higher cell counts are commonly
encountered as compared to images used in training data. By making better
predictions for higher ranges of cell count we are aiming to create better
generalization systems for cell counting. With architecture proposal of neural
arithmetic logic units (NALU) for arithmetic operations, task of counting has
become feasible for higher numeric ranges which were not included in training
data with better accuracy. In our study we incorporate these units in already
existing Fully Convolutional Regression Network (FCRN) and U-Net architectures
in the form of residual concatenated layers. We carried out a systematic
comparative study with the newly proposed changes and earlier base
architectures. This comparative study results are evaluated in terms of
optimizing regression loss across cell density map over an image. We achieved
better results in cell counting tasks with our newly proposed architectures
having residual layer concatenation connections. We further validated our
results on custom created high count dataset created from BBBC005 synthetic
cell count dataset and obtained even better results where testing images have
higher counts. These results confirm that above mentioned numerically biased
units does help models to learn numeric quantities for better generalization
results on high count data.

###### Index Terms:

Neural Arithmetic Logic Units, Cell Counting, Fully Convolutional Regression
Networks, U-net.

##  I Introduction

Ability to generalize concepts is fundamental component of intelligence and
core for designing smart systems [1, 2]. Neural networks simulates this
behavior with hierarchical learning of concepts. When it comes to automation,
counting is an important task from machine vision application [3] to cell
counting [4]. While neural networks manipulates numerical quantities but it is
not associated with systematic generalization [5, 6]. These networks fail to
generalize as evident from high generalization error while predicting
quantities that lie outside the training numerical range [7]. This highlights
memorization behavior in neural networks instead of generalization abilities
for a given task. This is especially problematic for cell counting tasks as
images with higher cell counts that were not part of training are common to
encounter in real applications.

Neural accumulators (NAC) and neural arithmetic logic units (NALU) [7] are
biased to learn systematic numerical computation and performs relatively
better than non linear activation functions for arithmetic operations. This
numerical bias of learning computations makes them excellent choice for
counting tasks which are essentially is an increment addition operation only.
Deep learning models generally take either segmentation approach with explicit
counting trainer or end-to-end counting via a regression loss. In this paper
we will go through the latter approach [4] in detail for automation of cell
counting process. As cell counting is cumbersome task and dense cell images
with higher cell counts containing data outside training numeric range are
common in real world scenarios. Achieving true cell automation with less
generalization errors is the prime objective of this paper.

Figure 1: Abstract representation of proposed modified architecture (Right)
with either NAC or NALU as compared to previous Fully Convolution Regression
Networks (Left). Here, as clearly specified with dimensional analysis of
tensor blocks (Right) the layers in the newly proposed model are concatenated
and again squeezed into same size as earlier base model architecture.

In regression loss approach Fully convolutional regression networks [2] and
U-net [8] architectures learns mapping between an image I(x) and a density map
D(x), given by F$\colon$ I(x) $\rightarrow$ D(x) (I $\in$ Rm $\times$ n , D
$\in$ Rm $\times$ n ) for a m $\times$ n pixel image. Later on, variations of
these base architectures with different activation functions are implemented
in this paper and their prediction performance is compared with newly proposed
NAC/NALU concatenated architectures of FCRN and U-net. The concatenated
connections adds numerical bias to layers of the network and behaves in
similar manner like ResNets [9]. But, instead of addition of inputs from
previous layer the input from previous is passed through these numerical units
for capturing numerical bias and concatenated with the main network layer as
shown in figure 1. On surface it appears that these proposed architectural
changes leads to accuracy improvements due to increased model capacity with
these numerically biased units adding more parameters for learning. But, our
results with custom high count dataset for testing created from BBBC005 [10]
reflects increased generalization counting abilities. As for higher cell count
images our model gives higher relative improvement in mean absolute
error(MAE).

Our concatenation based residual architecture utilizes the fundamentals of
batch normalization like specified identity mapping architecture [11] in
ResNets. But, instead of using convolution operation directly this network
leverages numerical bias information obtained from NAC and NALU operations
applied on input layer and then finally uses convolution operation on the
concatenated layer. Before and after this concatenation of this numerical bias
learning operation, batch normalization is carried out and output of this
operation is added back again to our next main network layer, as shown in
figure 1.

With means of this paper we introduce changes in current regression based
model architectures for end-to-end counter training and produce systems with
improved accuracy. Also, we validate our trained models on a different
specially tailored validation dataset with approximately seven times higher
counts of cells as compared to training dataset created from BBBC005 synthetic
cell dataset [10]. Overall, experimental results demonstrates that
supplementing the base architectures with NAC and NALU helped in achieving
better results and improved relative MAE for higher count test images.

##  II Related Work

Intuitive numerical understanding is important in learning and by adjunct
important in deep learning [1] for creating better models with higher
generalization capabilities. Counting objects [12, 13, 14, 15, 16] in given
image is a widely studied task. Trained models for counting tasks either use a
deep learning model to segment instances of given object then count them in a
post-processing step [17] or learn end-to-end predict count via a regression
loss [4]. Networks like Count-ception [18] added the concept of average over
redundant predictions with its deep Inception family network. Also, recent
architectures like ResNets [19], Highway Networks [20] and Densenets [21]
advocate linear connections like Count-ception to promote better learning
bias. Such models have better performance, though additional computational
overhead due to increased depth of given architectures do arise. Our work
highlights the generalization capabilities of the network, that extrapolate
well on unseen parts of solution space which highlights underlying structure
of behavior governing-equations [22].

We introduce architectural changes in models that learns and preserves input
information which is numerically biased with reference to input layers. It is
somewhat similar to ResNets [9], which are easier to optimize and gain
accuracy with increasing depth. With our experiments we aim to highlight that
models with numerically biased concatenated residual functions helps in
achieving better results with their addition in the form of a comparative
study with original architectures. Also, with our results in this paper we
demonstrate with our results that backpropagation learns this numerical bias
without any explicit numeric quantity being provided as input implying that
better computer vision counters can be trained with this module when added to
existing convolutional neural network architectures.

Density based estimation doesn’t require prior object detection or
segmentation [15, 12, 23]. In previous years, several works have investigated
this approach. In [15], the problem is stated as density estimation with a
supervised learning algorithm, D(x) = cT$\phi$(x), where D(x) represents
ground-truth density map, and $\phi$(x) represents local features and
parameters c are learned by minimizing the error between predicted and true
density with quadratic programming over all possible sub-windows. In [23],
regression forest is used to exploit patch-based idea for learning structured
labels, then for new input image density map is estimated averaged over
structured patch-based predictions. Also, in [12] an algorithm is used that
allows fast interactive counting with ridge regression.

Figure 2: The left column represents training image in first row and
corresponding annotated image in second row from fluorescent synthetic dataset
[29]. The second column of highlights corresponding zoomed view for better
visual comprehension of the images in the dataset.

Cell counting [15] problem is classified into supervised learning problem that
learns mapping between an image I(x) and a density map D(x), denoted by
F$\colon$ I(x) $\rightarrow$ D(x) (I $\in$ Rm $\times$ n , D $\in$ Rm $\times$
n ) for a m $\times$ n pixel image, see figure 2. Density function D(x)
function is defined on pixels in given image, integrating this map over an
image region gives an estimate of number of cells in that region. CNNs [24,
25] are quite popular in the bio-medical imaging because of their simple
architecture and achieve great results. Like in mitosis detection [26],
neuronal membrane segmentation [27] and analysis of C. elegans embryos
development [28]. Previously, fully convolutional regression networks (FCRNs)
and Count-ception have given state-of-the-art results in cell counting, with
potential for cell detection of overlapping cells.

Also U-Nets [8], typically fully convolutional network uses a modified version
of architecture proposed by Ciresan et al. [27] as latter is slow and trade-
off between localization and use of context are present. In U-Nets pooling
operations are replaced by upsampling operations to supplement usual
contracting network. For localization high resolution features from
contracting path are combined with unsampled output. Based on this information
a successive convolution layer then learn to assemble more precise output. For
our experimentation we selected FCRN and U-net based on simplicity and
relative similarity in their architectures with the difference being that
U-net uses inputs from previous layer for better localization.

##  III Experiments

In this section first we conceptually explore NACs and NALUs. Then compare
their addition capabilities with multi-layer perceptrons equipped with
different activation functions. With this study we aim to select NAC/NALU
designed variants which can best approximate the counting behavior and compare
them with standard FCRN and U-net neural network architecture for regression
loss approach in the following experiment done on synthetic dataset [29]. For
validation of counting generalization capabilities, our trained models are
tested against different synthetic cell image dataset with approximately seven
times higher counts than training data.

###  III-A Visual understanding of NACs and NALUs

NACs [7] supports accumulation of numerical quantities additively, a desirable
bias for linear exploration while counting. It is special type of linear layer
with transformation matrix W being continuous and differentiable
parameterization for gradient descent. W =
$\tanh$($\hat{W}$)$\odot$$\sigma$($\hat{M}$) consists of elements in [-1, 1]
with bias close to −1, 0, and 1. See figure 3 for ideation of this concept
with following equations for NAC: a = Wx, W =
$\tanh$($\hat{W}$)$\odot$$\sigma$($\hat{M}$) where $\hat{W}$, $\hat{M}$ are
learning parameters and W is transformation matrix.

Figure 3:  Left: NACs are biased towards learning [-1, 0, 1] as highlighted
with large plateau regions around these values. This means its outputs are
either addition or subtraction of input vectors not scaling. Right:
Approximate surface curve of NALU with some dimensional constraints for 3-D
plotting. It highlights the ability to scale, along with earlier numerical
biases around [-1, 0, 1] as shown with plateau region surfaces.

For complex mathematical operations like multiplication and division we use
NALUs. It uses weighted sum of two sub-cells, one for addition or subtraction
and another of multiply, division or power functions. It demonstrates that
neural accumulators (NACs) can be extended for learning scaling operations
with gate-controlled sub-operations. See figure 3 for ideation of this concept
with following equations for NALU: y = g$\odot$a + (1-g)$\odot$m; m =
$\exp$W($\log$( $\mid$x$\mid$ \+ $\epsilon$)), g = $\sigma$(Gx) where m is
subcell that operates in log space and g is learned gate, both contains
learning parameters.

###  III-B Comparative analysis of addition operation

Here, we use neural networks with NACs/NALUs and multilayer perceptrons (MLP)
with different activation functions but same structures. These are trained
with two randomly generated inputs from uniform distribution a and b with each
having 214 data points for training. Prediction capabilities on test data with
values ranging up to 10 times the training range are evaluated as part of this
experiment. Refer figure 4 to observe architecture for both these trained
models in this comparative study.

Figure 4: The model on left represents neural network with NAC or NALU units
and on right, a MLP which is trained with different activation functions. Each
variant of above stated models have two inputs, three hidden units, one
output.

Comparative analysis is summarized in table I with MAE as an accuracy measure
for MLP variants, NAC, NALU and its variants with changed learned gate for
extrapolation. Also, in NALU-Tanh and NALU-Hard Sigmoid the learning gate g’s
is changed to observe any improvements in NALU’s performance based on this
change.

TABLE I: Comparative result summarization for multiple models Layer Configuration/Activations | Mean Absolute Error (a+b)
---|---
Linear MLP | $3.63\text{\times}{10}^{-06}$
Sigmoid MLP | $29.830$
Tanh MLP | $15.743$
ELu MLP | $0.019$
ReLU MLP | $0.001$
Leaky ReLU MLP | $9.83\text{\times}{10}^{-04}$
PReLU MLP | $0.001$
NAC | $2.70\text{\times}{10}^{-06}$
NALU | $2.71\text{\times}{10}^{-06}$
NALU-Hard Sigmoid | $3.24\text{\times}{10}^{-06}$
NALU-Tanh | $3.18\text{\times}{10}^{-06}$

From these results stated in table I and visualized in figure 5 we conclude
that Linear, LeakyReLU, ReLU activations and NAC, NALU, NALU-Tanh modules were
the top performers in extrapolation task for numeric addition operation task.
Hence, these top performers are used further in cell-counting task on
synthetic dataset for learning end-to-end counting mechanism.

Figure 5: Above visualization of MAE and log10(MAE) for different models
learning identity mapping, demonstrates that our standard activation functions
like sigmoid and tanh doesn’t perform that well with higher data ranges during
testing. Whereas, NACs and NALUs clearly overpowers the identity learning task
and linear, relu based activation functions somewhat manages to provide
acceptable results.

###  III-C Cell counting experiment

111Code repository: https://github.com/ashishrana160796/nalu-cell-counting

In this experiment section the first subsections elaborates the datasets used
for training and validation of our trained models, plus the data augmentation
techniques used in our experiment. After that we elaborate onto different
architectures used for training having different activation layers on standard
architectures and residual concatenated connection modules on modified
proposed model architectures.

####  III-C1 Datasets and data augmentation

Synthetic dataset which is generated by system [29]. 200 highly-realistic
synthetic fluorescence microscopic images of bacterial cells are used for
experimentation with a 75/25 train-test split for training each model
architecture and its variants. Images are having average of 174±64 cells.

For validation of trained models and checking true generalization capabilities
we use BBBC005 from the the Broad Institute’s Bioimage Benchmark Collection
[10]. This dataset is comprised of 600 images have a corresponding foreground
mask which alters the focus on these images and ground truth images are
completely in-focused version before any guassian filter application. We take
a subset of this dataset with highly focused F1 images only and their
corresponding ground truth image for validating our model. And after that we
coalesce each image 16 times in 4x4 grid manner with random vertical and
horizontal flips to create a final high cell count images for our validation
dataset, see figure 6. The ground truth images are accordingly changed with
same randomness and they are eroded also to match the fluorescent synthetic
dataset on which the models are trained on.

Figure 6: Left: Original image from BBBC005 dataset. Right: Repeated image
generated from a 4X4 grid repetition operation with random horizontal and
vertical flips of base image shown in left.

Data augmentation with elastic deformations to training images is applied for
teaching network the desired invariance and robustness properties, like
specified in figure 7. These elastic deformations are introduced in the form
of angular shear in the training images. Translation and rotation invariance
along with robustness to gray value variations and deformations is main focus
of augmentation process for microscopic images. Disfigurement using random
displacement vectors on a coarse 3x3 grid are also generated. These data
augmentation techniques especially are helpful for our custom data which is
created just by repeating the original image in order to supplement a more
robust dataset for the model to train on.

Figure 7: The above figure illustrates different cell augmentation techniques
in action like elastic deformations, random rotations and plane shifting for
the input images adding more robustness to the trained model.

####  III-C2 Defining regression task and architecture details

In training dataset ground truth is provided as dot annotation corresponding
to each cell image. For training, dot annotations are represented by Gaussian
and density surface D(x) which is formed from superposition of Gaussians. The
optimization task is to regress density surface from corresponding image I(x).
This is achieved by training convolutional neural networks (CNN) using mean
square error between output heat map and target density surface as the loss
function. Hence, at inference given an input I(x), the model predicts density
heat map D(x).

Figure 8: Network Architecture Left: FCRN with 3x3 convolution operations.
Network Architecture Right:  FCRN added with residual concatenation
connections of Ni numerically biased units. Also, for dimensional
compatibility Ni residual layers are fed to corresponding main network layers
and after that normalized for regularization. At last, 1x1 Conv operation
output in the form of density map of result image is generated .
Conv + BN + Pool: A 3x3 convolutional operation with batch normalization
regularization and 2x2 max pooling layer.
Unsample+Conv + BN: Unsampling the image and then applying convolutional
operation with batch normalization regularization.
Concat: Feature maps concatenated along depth dimension.
Ni: NAC or Variants of NALUs as residual concatenated connection.

FCRNs are inspired from VGG-net, we only used small kernels of size 3x3 pixels
for designing our network. Feature maps are increased for avoiding spatial
information loss. Activation layers like convolution-ReLU-Pooling are popular
in CNN architectures [24]. Here, we have altered these layers to create
different models which contains some numerical bias in the form of residual
connections and regularized by batch normalization. The first layers contains
convolutions-pooling operations, then we undo spatial reduction by upsampling
operations for end-to-end model training. Also, for dimensional compatibility
of residual NAC or NALU modules we append their output to corresponding main
network layer across depth dimension and batch normalize the output after
appropriate convolutional operation. See figure 8 for comparison between
earlier original model and newly proposed architecture along with architecture
parameter details.

Figure 9: Network Architecture Top: U-net with 3x3 convolution operations and
increasing depth of filters from 64 to 256 for feature abstraction & learning
and after that these layers are fed to upsampling layers. Network Architecture
Bottom:  U-nets main network added with residual concatenation connections of
Ni numerically biased units and after that batch normalized for
regularization. Similar, operation abbreviations used as stated in figure 8

U-net is modification upon the previously discussed FCRN architecture by
having large number of feature channels for upsampling to propagate context
information to high resolution layers. That makes expansive path almost
symmetric to contracting path yielding a u-shape. Similar to above FCRN’s
optimization problem formulation the evaluation parameters remains the same.
Residual concatenated connection addition with NACs and NALU units along with
batch normalization is done for adding numerical bias information to the main
network. Also, U-net architecture used in this paper is more computationally
expensive than FCRN having approximately thrice the number of parameters
leading to more feature learning capacity and is expected to perform better
for prediction tasks. See figure 9 for comparison between earlier original
U-net model and newly proposed architecture along with parameter details.

For the concatenation of residual connections of these units the dimensional
consistency is maintained by added them along the depth dimension with the
main network. After that upsampling, convolutional and batch normalization
regularization operations are applied accordingly for these units to merge
with the base network. FCRN’s implementation resembles that of MatConvNet [30]
as upsampling in Keras is implemented by repeating elements, instead of
bilinear sampling. In U-nets, low-level feature representations are fused
during upsampling, aiming to compensate the information loss due to max
pooling.

##  IV Results

MAE is the metric used in this paper for measuring results for cell counting
on the synthetic cell dataset [29] and custom BBBC005 synthetic modified high
cell count validation dataset.

  * •

Mean Absolute Error (MAE): Mean Absolute Error (MAE): The mean absolute error
is an average of the difference between the predicted value and true value.

$AE=\|e_{i}\|=\|y_{i}-x_{i}\|$ (1) $MAE=\sum_{i=1}^{n}\frac{|e_{i}|}{n}$ (2)

  * •

Relative Improvement Percentage Relative Improvement Percentage (RIP): Here,
in context of this paper it defined as percentage improvement in MAE of a
given model with respect to baseline ReLU models for FCRN and U-net
architectures. In below equation, Mr is MAE from baseline ReLU model and Mi is
model under consideration.

$RIP_{\%}=((M_{r}-M_{i})/M_{r})*100$ (3)

Result table II compares baseline FCRN, U-net architectures with new
numerically biased ResNet like connection modules with NACs and NALUs units
under current training setup. With our setup we able to obtain similar results
as mentioned in earlier reference papers and also we have equipped earlier
model architectures with different regularization activations as specified in
the table. From earlier ReLU implementation clearly Linear and LeakyReLU
activation regularization based baseline models have performed well. Also, for
both model structures NAC and NALUs residual modules have outperformed all the
earlier specified regular FCRN architecture. And similar results are extended
by U-net model evaluation where NALU layer concatenation based U-net
outperforms all the models trained for our experiment.

TABLE II: Result summarization for trained models FCRN-Models | MAE | U-Net-Models | MAE
---|---|---|---
ReLU | 3.43 | ReLU | 1.78
LeakyReLU | 3.39 | LeakyReLU | 1.74
Linear | 3.34 | Linear | 1.73
NALU-tanh | 3.21 | NALU/tanh | 1.56
NALU | 3.17 | NALU | 1.42
NAC | 3.23 | NAC | 1.63

Result table III compares performance of above trained models on a new
validation dataset containing much higher cell counts for measuring
performance on extrapolation capabilities counting tasks. For validation set
we have used 300 images of size 256x256 pixels with cell counts averaging
around 1200±12. Here also, NAC and NALU based residual concatenation module
based models outperforms earlier architectures for counting tasks. But, this
time the relative improvement is even more for NALU based FCRN and U-net
models showcasing better generalization abilities of trained models.

TABLE III: Validating trained models for extrapolation cell counting tasks FCRN-Models | MAE | U-Net-Models | MAE
---|---|---|---
ReLU | 3.04 | ReLU | 2.87
LeakyReLU | 2.99 | LeakyReLU | 2.62
Linear | 2.85 | Linear | 2.47
NALU-tanh | 2.32 | NALU-tanh | 1.95
NALU | 2.27 | NALU | 1.87
NAC | 2.40 | NAC | 1.92

Relative improvement in predictions is visualized in figure 10 against ReLU
based activation base result for comparison with other activation layer
changes in FCRNs/U-nets and concatenation layer NALU/NAC residual connection
addition in FCRNs & U-nets. It includes averaged out comparison from multiple
executions of training and testing runs for both interpolation testing and
extrapolation validation counting tasks for FCRN and U-net variant models with
respect to ReLU based FCRN and U-net model. From, this figure it is clearly
highlighted that models with NAC and NALUs residual modules have better
generalization capabilities for extrapolation counting tasks i.e. they are
better generalizers for this given cell counting task with increase in
relative improvement in prediction as compared to base ReLU implementation.
Here, for measuring generalization capabilities of model in extrapolation task
relative improvement metric is selected as it bring more perspective to the
improved performance on validation dataset. The performance has shown a
general increment for every model as validation dataset consist of highly
focused images. Hence, measuring relative improvement in results is more
appropriate decision to get understanding of true improvment in model
prediction capabilities.

Figure 10: Moving towards right in above figure and measuring RIP metric there
is sharp increase with NALU & NAC units based models demonstrating even more
increase in relative improvement with respect to baseline ReLU models for
extrapolation tasks. For interpolation task there is highest 9% improvement
and for extrapolation there is 23% improvement with NAC units as residual
connection.

This figure 10 shows more increase in relative improvement as we move right
towards horizontal axis for both testing and validation task with
extrapolation where in validation extrapolation task NAC/NALU models from
which we can conclude that trained models are having better generalization
abilities with some learned numerical bias in their trained weights with which
even better predictions for higher count cells is made.

##  V Summary

We were able to demonstrate that addition of newly proposed NAC and NALU units
in existing architectures in the form of residual concatenation connection
layer modules achieves better results. With numerically biased residual
connections, higher accuracy for more dense images having higher counts of
cells is achieved. Hence, producing more generalized cell counters that
provides better predictions for real life use-cases. Finally, for code
implementation details and other supplementary experimental results refer to
this paper’s github repository.

## References

  * [1] Dehaene, Stanislas. The number sense: How the mind creates mathematics. OUP USA, 2011.
  * [2] Gallistel, C. Randy. ”Finding numbers in the brain.” Philosophical Transactions of the Royal Society B: Biological Sciences 373.1740 (2018): 20170119.
  * [3] Baygin, Mehmet, et al. ”An Image Processing based Object Counting Approach for Machine Vision Application.” arXiv preprint arXiv:1802.05911 (2018).
  * [4] Xie, Weidi, J. Alison Noble, and Andrew Zisserman. ”Microscopy cell counting and detection with fully convolutional regression networks.” Computer methods in biomechanics and biomedical engineering: Imaging & Visualization 6.3 (2018): 283-292.
  * [5] Fodor, Jerry A., and Zenon W. Pylyshyn. ”Connectionism and cognitive architecture: A critical analysis.” Cognition 28.1-2 (1988): 3-71.
  * [6] Marcus, Gary F. The algebraic mind: Integrating connectionism and cognitive science. MIT press, 2018.
  * [7] Trask, Andrew, et al. ”Neural arithmetic logic units.” Advances in Neural Information Processing Systems. 2018.
  * [8] Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. ”U-net: Convolutional networks for biomedical image segmentation.” International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.
  * [9] He, Kaiming, et al. ”Deep residual learning for image recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
  * [10] Ljosa, Vebjorn, Katherine L. Sokolnicki, and Anne E. Carpenter. ”Annotated high-throughput microscopy image sets for validation.” Nature methods 9.7 (2012): 637-637.
  * [11] He, Kaiming, et al. ”Identity mappings in deep residual networks.” European conference on computer vision. Springer, Cham, 2016.
  * [12] Arteta, Carlos, et al. ”Interactive object counting.” European conference on computer vision. Springer, Cham, 2014.
  * [13] Chan, Antoni B., Zhang-Sheng John Liang, and Nuno Vasconcelos. ”Privacy preserving crowd monitoring: Counting people without people models or tracking.” 2008 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2008.
  * [14] Seguí, Santi, Oriol Pujol, and Jordi Vitria. ”Learning to count with deep object features.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2015.
  * [15] Lempitsky, Victor, and Andrew Zisserman. ”Learning to count objects in images.” Advances in neural information processing systems. 2010.
  * [16] Zhang, Cong, et al. ”Cross-scene crowd counting via deep convolutional neural networks.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
  * [17] Hernández, Carlos X., Mohammad M. Sultan, and Vijay S. Pande. ”Using Deep Learning for Segmentation and Counting within Microscopy Data.” arXiv preprint arXiv:1802.10548 (2018).
  * [18] Paul Cohen, Joseph, et al. ”Count-ception: Counting by fully convolutional redundant counting.” Proceedings of the IEEE International Conference on Computer Vision. 2017.
  * [19] He, Kaiming, et al. ”Identity mappings in deep residual networks.” European conference on computer vision. Springer, Cham, 2016.
  * [20] Srivastava, Rupesh Kumar, Klaus Greff, and Jürgen Schmidhuber. ”Highway networks.” arXiv preprint arXiv:1505.00387 (2015).
  * [21] Huang, Gao, et al. ”Densely connected convolutional networks.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.
  * [22] Brunton, Steven L., Joshua L. Proctor, and J. Nathan Kutz. ”Discovering governing equations from data by sparse identification of nonlinear dynamical systems.” Proceedings of the National Academy of Sciences 113.15 (2016): 3932-3937.
  * [23] Fiaschi, Luca, et al. ”Learning to count with regression forest and structured labels.” Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012). IEEE, 2012.
  * [24] Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. ”Imagenet classification with deep convolutional neural networks.” Advances in neural information processing systems. 2012.
  * [25] LeCun, Yann, et al. ”Gradient-based learning applied to document recognition.” Proceedings of the IEEE 86.11 (1998): 2278-2324.
  * [26] Cireşan, Dan C., et al. ”Mitosis detection in breast cancer histology images with deep neural networks.” International Conference on Medical Image Computing and Computer-assisted Intervention. Springer, Berlin, Heidelberg, 2013.
  * [27] Ciresan, Dan, et al. ”Deep neural networks segment neuronal membranes in electron microscopy images.” Advances in neural information processing systems. 2012.
  * [28] Ning, Feng, et al. ”Toward automatic phenotyping of developing embryos from videos.” IEEE Transactions on Image Processing 14 (2005): 1360-1371.
  * [29] Lehmussola, Antti, et al. ”Computational framework for simulating fluorescence microscope images with cell populations.” IEEE transactions on medical imaging 26.7 (2007): 1010-1016.
  * [30] Vedaldi, Andrea, and Karel Lenc. ”Matconvnet: Convolutional neural networks for matlab.” Proceedings of the 23rd ACM international conference on Multimedia. ACM, 2015.


* (74) S. Das, G. Goswami and C. Krishnan, Swampland, axions, and minimal warm inflation, Phys. Rev. D 101 (2020) no.10, 103529 doi:10.1103/PhysRevD.101.103529 [arXiv:1911.00323 [hep-th]].
  * (75) M. Laine and S. Procacci, Minimal warm inflation with complete medium response, JCAP 06 (2021), 031 doi:10.1088/1475-7516/2021/06/031 [arXiv:2102.09913 [hep-ph]].
  * (76) W. DeRocco, P. W. Graham and S. Kalia, Warming up cold inflation, JCAP 11 (2021), 011 doi:10.1088/1475-7516/2021/11/011 [arXiv:2107.07517 [hep-ph]].
  * (77) S. Das and R. O. Ramos, Running and running of the running of the scalar spectral index in warm inflation, Universe 9 (2023), 76 doi:10.3390/universe9020076 [arXiv:2212.13914 [astro-ph.CO]].
  * (78) R. O. Ramos and L. A. da Silva, Power spectrum for inflation models with quantum and thermal noises, JCAP 03 (2013), 032 doi:10.1088/1475-7516/2013/03/032 [arXiv:1302.3544 [astro-ph.CO]].
  * (79) M. Bastero-Gil, A. Berera, R. O. Ramos and J. G. Rosa, Adiabatic out-of-equilibrium solutions to the Boltzmann equation in warm inflation, JHEP 02 (2018), 063 doi:10.1007/JHEP02(2018)063 [arXiv:1711.09023 [hep-ph]].
  * (80) C. Graham and I. G. Moss, Density fluctuations from warm inflation, JCAP 07 (2009), 013 doi:10.1088/1475-7516/2009/07/013 [arXiv:0905.3500 [astro-ph.CO]].
  * (81) H. Kodama and M. Sasaki, Cosmological Perturbation Theory, Prog. Theor. Phys. Suppl. 78 (1984), 1-166 doi:10.1143/PTPS.78.1
  * (82) J. c. Hwang, Perturbations of the Robertson-Walker space - Multicomponent sources and generalized gravity, Astrophys. J. 375 (1991), 443-462 doi:10.1086/170206
  * (83) D. Baumann, Inflation, doi:10.1142/9789814327183_0010 [arXiv:0907.5424 [hep-th]].
  * (84) A. R. Liddle and S. M. Leach, How long before the end of inflation were observable perturbations produced?, Phys. Rev. D 68 (2003), 103503 doi:10.1103/PhysRevD.68.103503 [arXiv:astro-ph/0305263 [astro-ph]].
  * (85) H. P. De Oliveira, Density perturbations in warm inflation and COBE normalization, Phys. Lett. B 526 (2002), 1 doi:10.1016/S0370-2693(01)01496-4 [arXiv:gr-qc/0202045 [gr-qc]].
  * (86) M. Bastero-Gil, A. Berera and J. R. Calderón, Reexamination of the warm inflation curvature perturbations spectrum, JCAP 07 (2019), 019 doi:10.1088/1475-7516/2019/07/019 [arXiv:1904.04086 [astro-ph.CO]].
  * (87) M. Bastero-Gil, A. Berera, I. G. Moss and R. O. Ramos, Cosmological fluctuations of a random field and radiation fluid, JCAP 05 (2014), 004 doi:10.1088/1475-7516/2014/05/004 [arXiv:1401.1149 [astro-ph.CO]].
  * (88) M. Motaharfar and R. O. Ramos, Dirac-Born-Infeld warm inflation realization in the strong dissipation regime, Phys. Rev. D 104 (2021) no.4, 043522 doi:10.1103/PhysRevD.104.043522 [arXiv:2105.01131 [hep-th]].
  * (89) M. Motaharfar, E. Massaeli and H. R. Sepangi, Warm Higgs G-inflation: predictions and constraints from Planck 2015 likelihood, JCAP 10 (2018), 002 doi:10.1088/1475-7516/2018/10/002 [arXiv:1807.09548 [gr-qc]].
  * (90) Y. Qiu and L. Sorbo, Spectrum of tensor perturbations in warm inflation, Phys. Rev. D 104 (2021) no.8, 083542 doi:10.1103/PhysRevD.104.083542 [arXiv:2107.09754 [astro-ph.CO]].
  * (91) X. B. Li, H. Wang and J. Y. Zhu, Gravitational waves from warm inflation, Phys. Rev. D 97 (2018) no.6, 063516 doi:10.1103/PhysRevD.97.063516 [arXiv:1803.10074 [gr-qc]].
  * (92) I. Arraut, The Astrophysical Scales Set by the Cosmological Constant, Black-Hole Thermodynamics and Non-Linear Massive Gravity, Universe 3, no.2, 45 (2017) doi:10.3390/universe3020045 [arXiv:1305.0475 [gr-qc]].
  * (93) N. Aghanim et al. [Planck], Planck 2018 results. VI. Cosmological parameters, Astron. Astrophys. 641 (2020), A6 [erratum: Astron. Astrophys. 652 (2021), C4] doi:10.1051/0004-6361/201833910 [arXiv:1807.06209 [astro-ph.CO]].
  * (94) P. A. R. Ade et al. [BICEP and Keck], Phys. Rev. Lett. 127 (2021) no.15, 151301 doi:10.1103/PhysRevLett.127.151301 [arXiv:2110.00483 [astro-ph.CO]].
  * (95) I. G. Moss and C. Xiong, Non-Gaussianity in fluctuations from warm inflation, JCAP 04 (2007), 007 doi:10.1088/1475-7516/2007/04/007 [arXiv:astro-ph/0701302 [astro-ph]].
  * (96) I. G. Moss and T. Yeomans, Non-gaussianity in the strong regime of warm inflation, JCAP 08 (2011), 009 doi:10.1088/1475-7516/2011/08/009 [arXiv:1102.2833 [astro-ph.CO]].
  * (97) M. Bastero-Gil, A. Berera, I. G. Moss and R. O. Ramos, Theory of non-Gaussianity in warm inflation, JCAP 12 (2014), 008 doi:10.1088/1475-7516/2014/12/008 [arXiv:1408.4391 [astro-ph.CO]].
  * (98) M. Mirbabayi and A. Gruzinov, Shapes of non-Gaussianity in warm inflation, [arXiv:2205.13227 [astro-ph.CO]].
  * (99) E. Palti, The Swampland: Introduction and Review, Fortsch. Phys. 67 (2019) no.6, 1900037 doi:10.1002/prop.201900037 [arXiv:1903.06239 [hep-th]].
  * (100) H. Ooguri and C. Vafa, On the Geometry of the String Landscape and the Swampland, Nucl. Phys. B 766 (2007), 21-33 doi:10.1016/j.nuclphysb.2006.10.033 [arXiv:hep-th/0605264 [hep-th]].
  * (101) S. K. Garg and C. Krishnan, Bounds on Slow Roll and the de Sitter Swampland, JHEP 11 (2019), 075 doi:10.1007/JHEP11(2019)075 [arXiv:1807.05193 [hep-th]].
  * (102) R. Bousso, A Covariant entropy conjecture, JHEP 07 (1999), 004 doi:10.1088/1126-6708/1999/07/004 [arXiv:hep-th/9905177 [hep-th]].
  * (103) W. H. Kinney, Eternal Inflation and the Refined Swampland Conjecture, Phys. Rev. Lett. 122 (2019) no.8, 081302 doi:10.1103/PhysRevLett.122.081302 [arXiv:1811.11698 [astro-ph.CO]].
  * (104) D. Baumann and L. McAllister, Inflation and String Theory, Cambridge University Press, 2015, ISBN 978-1-107-08969-3, 978-1-316-23718-2 doi:10.1017/CBO9781316105733 [arXiv:1404.2601 [hep-th]].
  * (105) L. Aalsma and G. Shiu, Chaos and complementarity in de Sitter space, JHEP 05 (2020), 152 doi:10.1007/JHEP05(2020)152 [arXiv:2002.01326 [hep-th]].
  * (106) S. Mizuno, S. Mukohyama, S. Pi and Y. L. Zhang, Universal Upper Bound on the Inflationary Energy Scale from the Trans-Planckian Censorship Conjecture, Phys. Rev. D 102 (2020) no.2, 021301 doi:10.1103/PhysRevD.102.021301 [arXiv:1910.02979 [astro-ph.CO]].
  * (107) V. Kamali and R. Brandenberger, Relaxing the TCC Bound on Inflationary Cosmology?, Eur. Phys. J. C 80 (2020) no.4, 339 doi:10.1140/epjc/s10052-020-7908-8 [arXiv:2001.00040 [hep-th]].
  * (108) Y. Akrami et al. [Planck], Planck 2018 results. X. Constraints on inflation, Astron. Astrophys. 641 (2020), A10 doi:10.1051/0004-6361/201833887 [arXiv:1807.06211 [astro-ph.CO]].
  * (109) K. Dimopoulos, Phys. Rev. D 98 (2018) no.12, 123516 doi:10.1103/PhysRevD.98.123516 [arXiv:1810.03438 [gr-qc]].
  * (110) S. Brahma and S. Shandera, JHEP 11 (2019), 016 doi:10.1007/JHEP11(2019)016 [arXiv:1904.10979 [hep-th]].
  * (111) G. S. Vicente, L. A. da Silva and R. O. Ramos, Eternal inflation in a dissipative and radiation environment: Heated demise of eternity, Phys. Rev. D 93 (2016) no.6, 063509 doi:10.1103/PhysRevD.93.063509 [arXiv:1509.08983 [astro-ph.CO]].
  * (112) K. Freese, J. A. Frieman and A. V. Olinto, Natural inflation with pseudo - Nambu-Goldstone bosons, Phys. Rev. Lett. 65 (1990), 3233-3236 doi:10.1103/PhysRevLett.65.3233
  * (113) L. Visinelli, Natural Warm Inflation, JCAP 09 (2011), 013 doi:10.1088/1475-7516/2011/09/013 [arXiv:1107.3523 [astro-ph.CO]].
  * (114) H. Mishra, S. Mohanty and A. Nautiyal, Warm natural inflation, Phys. Lett. B 710 (2012), 245-250 doi:10.1016/j.physletb.2012.02.005 [arXiv:1106.3039 [hep-ph]].
  * (115) V. Kamali, Warm pseudoscalar inflation, Phys. Rev. D 100 (2019) no.4, 043520 doi:10.1103/PhysRevD.100.043520 [arXiv:1901.01897 [gr-qc]].
  * (116) G. Montefalcone, V. Aragam, L. Visinelli and K. Freese, Observational Constraints on Warm Natural Inflation, [arXiv:2212.04482 [gr-qc]].
  * (117) R. Herrera, S. del Campo and C. Campuzano, Tachyon warm inflationary universe models, JCAP 10 (2006), 009 doi:10.1088/1475-7516/2006/10/009 [arXiv:astro-ph/0610339 [astro-ph]].
  * (118) S. del Campo and R. Herrera, Warm-Intermediate inflationary universe model, JCAP 04 (2009), 005 doi:10.1088/1475-7516/2009/04/005 [arXiv:0903.4214 [astro-ph.CO]].
  * (119) R. Herrera and E. San Martin, Warm-intermediate inflationary universe model in braneworld cosmologies, Eur. Phys. J. C 71 (2011), 1701 doi:10.1140/epjc/s10052-011-1701-7 [arXiv:1108.1371 [gr-qc]].
  * (120) M. R. Setare and V. Kamali, Tachyon Warm-Intermediate Inflationary Universe Model in High Dissipative Regime, JCAP 08 (2012), 034 doi:10.1088/1475-7516/2012/08/034 [arXiv:1210.0742 [hep-th]].
  * (121) M. R. Setare and V. Kamali, Tachyon Warm-Logamediate Inflationary Universe Model in High Dissipative Regime, Phys. Rev. D 87 (2013), 083524 doi:10.1103/PhysRevD.87.083524 [arXiv:1305.0740 [hep-th]].
  * (122) M. R. Setare and V. Kamali, Warm-Intermediate Inflationary Universe Model with Viscous Pressure in High Dissipative Regime, Gen. Rel. Grav. 46 (2014), 1698 doi:10.1007/s10714-014-1698-y [arXiv:1403.0186 [gr-qc]].
  * (123) V. Kamali, S. Basilakos and A. Mehrabi, Tachyon warm-intermediate inflation in the light of Planck data, Eur. Phys. J. C 76 (2016) no.10, 525 doi:10.1140/epjc/s10052-016-4380-6 [arXiv:1604.05434 [gr-qc]].
  * (124) M. Motaharfar and H. R. Sepangi, Warm-tachyon Gauss–Bonnet inflation in the light of Planck 2015 data, Eur. Phys. J. C 76 (2016) no.11, 646 doi:10.1140/epjc/s10052-016-4474-1 [arXiv:1604.00453 [gr-qc]].
  * (125) A. Mohammadi, T. Golanbari, H. Sheikhahmadi, K. Sayar, L. Akhtari, M. A. Rasheed and K. Saaidi, Warm tachyon inflation and swampland criteria, Chin. Phys. C 44 (2020) no.9, 095101 doi:10.1088/1674-1137/44/9/095101 [arXiv:2001.10042 [gr-qc]].
  * (126) I. Dymnikova and M. Khlopov, Decay of cosmological constant as Bose condensate evaporation, Mod. Phys. Lett. A 15 (2000), 2305-2314 doi:10.1142/S0217732300002966 [arXiv:astro-ph/0102094 [astro-ph]].
  * (127) I. Dymnikova and M. Khlopov, Decay of cosmological constant in selfconsistent inflation, Eur. Phys. J. C 20 (2001), 139-146 doi:10.1007/s100520100625
  * (128) S. N. Gashti and J. Sadeghi, Refined swampland conjecture in warm vector hybrid inflationary scenario, Eur. Phys. J. Plus 137 (2022) no.6, 731 doi:10.1140/epjp/s13360-022-02961-8
  * (129) F. B. M. d. Santos, R. Silva, S. S. da Costa, M. Benetti and J. S. Alcaniz, Warm $\beta$-exponential inflation and the Swampland Conjectures, [arXiv:2209.06153 [astro-ph.CO]].
  * (130) A. Payaka, W. Amaek and P. Channuie, Warm deformed R2 inflation, Nucl. Phys. B 986 (2023), 116052 doi:10.1016/j.nuclphysb.2022.116052 [arXiv:2203.11041 [gr-qc]].
  * (131) Z. Bouabdallaoui, A. Errahmani, M. Bouhmadi-López and T. Ouali, Scalar warm inflation in holographic cosmology, Phys. Rev. D 105 (2022) no.4, 043513 doi:10.1103/PhysRevD.105.043513 [arXiv:2203.05047 [hep-th]].
  * (132) Y. Reyimuaji and X. Zhang, Warm-assisted natural inflation, JCAP 04 (2021), 077 doi:10.1088/1475-7516/2021/04/077 [arXiv:2012.07329 [astro-ph.CO]].
  * (133) L. L. Graef and R. O. Ramos, Probability of Warm Inflation in Loop Quantum Cosmology, Phys. Rev. D 98 (2018) no.2, 023531 doi:10.1103/PhysRevD.98.023531 [arXiv:1805.05985 [gr-qc]].
  * (134) A. Berera, J. Mabillard, M. Pieroni and R. O. Ramos, Identifying Universality in Warm Inflation, JCAP 07 (2018), 021 doi:10.1088/1475-7516/2018/07/021 [arXiv:1803.04982 [astro-ph.CO]].
  * (135) R. Herrera, Reconstructing warm inflation, Eur. Phys. J. C 78 (2018) no.3, 245 doi:10.1140/epjc/s10052-018-5741-0 [arXiv:1801.05138 [gr-qc]].
  * (136) Y. y. Wang, J. Y. Zhu and X. M. Zhang, Observational Constraints on Two-field Warm Inflation, Phys. Rev. D 99 (2019) no.10, 103529 doi:10.1103/PhysRevD.99.103529 [arXiv:1905.02414 [gr-qc]].
  * (137) T. Harko and H. Sheikhahmadi, Warm inflation with non-comoving scalar field and radiation fluid, Eur. Phys. J. C 81 (2021) no.2, 165 doi:10.1140/epjc/s10052-021-08964-6 [arXiv:2102.04728 [gr-qc]].
  * (138) A. Bose and S. Chakraborty, Does fractal universe favour warm inflation: Observational support?, Nucl. Phys. B 978 (2022), 115767 doi:10.1016/j.nuclphysb.2022.115767 [arXiv:2204.05712 [gr-qc]].
  * (139) T. Harko and H. Sheikhahmadi, Irreversible thermodynamical description of warm inflationary cosmological models, Phys. Dark Univ. 28 (2020), 100521 doi:10.1016/j.dark.2020.100521 [arXiv:2003.02257 [gr-qc]].
  * (140) H. Sheikhahmadi, A. Mohammadi, A. Aghamohammadi, T. Harko, R. Herrera, C. Corda, A. Abebe and K. Saaidi, Constraining chameleon field driven warm inflation with Planck 2018 data, Eur. Phys. J. C 79 (2019) no.12, 1038 doi:10.1140/epjc/s10052-019-7571-0 [arXiv:1907.10966 [gr-qc]].
  * (141) O. Bertolami and P. M. Sá, Multi-field cold and warm inflation and the de Sitter swampland conjectures, JCAP 09 (2022), 001 doi:10.1088/1475-7516/2022/09/001 [arXiv:2204.13794 [gr-qc]].
  * (142) M. Motaharfar and P. Singh, Role of dissipative effects in the quantum gravitational onset of warm Starobinsky inflation in a closed universe, Phys. Rev. D 104 (2021) no.10, 106006 doi:10.1103/PhysRevD.104.106006 [arXiv:2102.09578 [gr-qc]].
  * (143) M. AlHallak, K. K. A. Said, N. Chamoun and M. S. El-Daher, On Warm Natural Inflation and Planck 2018 constraints, [arXiv:2211.07775 [gr-qc]].
  * (144) G. Montefalcone, V. Aragam, L. Visinelli and K. Freese, Constraints on the scalar-field potential in warm inflation, [arXiv:2209.14908 [gr-qc]].
  * (145) H. P. de Oliveira and R. O. Ramos, Dynamical system analysis for inflation with dissipation, Phys. Rev. D 57 (1998), 741-749 doi:10.1103/PhysRevD.57.741 [arXiv:gr-qc/9710093 [gr-qc]].
  * (146) X. B. Li, Y. Y. Wang, H. Wang and J. Y. Zhu, Dynamic analysis of noncanonical warm inflation, Phys. Rev. D 98 (2018) no.4, 043510 doi:10.1103/PhysRevD.98.043510 [arXiv:1804.05360 [gr-qc]].
  * (147) A. Alho, V. Bessa and F. C. Mena, Dynamics of interacting monomial scalar field potentials and perfect fluids, [arXiv:2212.02942 [gr-qc]].
  * (148) R. Arya, A. Dasgupta, G. Goswami, J. Prasad and R. Rangarajan, Revisiting CMB constraints on warm inflation, JCAP 02 (2018), 043 doi:10.1088/1475-7516/2018/02/043 [arXiv:1710.11109 [astro-ph.CO]].
  * (149) M. Bastero-Gil, S. Bhattacharya, K. Dutta and M. R. Gangopadhyay, Constraining Warm Inflation with CMB data, JCAP 02 (2018), 054 doi:10.1088/1475-7516/2018/02/054 [arXiv:1710.10008 [astro-ph.CO]].
  * (150) R. Arya and R. Rangarajan, Study of warm inflationary models and their parameter estimation from CMB, Int. J. Mod. Phys. D 29 (2020) no.08, 2050055 doi:10.1142/S0218271820500558 [arXiv:1812.03107 [astro-ph.CO]].
  * (151) M. Benetti, L. Graef and R. O. Ramos, Observational Constraints on Warm Inflation in Loop Quantum Cosmology, JCAP 10 (2019), 066 doi:10.1088/1475-7516/2019/10/066 [arXiv:1907.03633 [astro-ph.CO]].
  * (152) A. Berera, T. W. Kephart and S. D. Wick, GUT cosmic magnetic fields in a warm inflationary universe, Phys. Rev. D 59 (1999), 043510 doi:10.1103/PhysRevD.59.043510 [arXiv:hep-ph/9809404 [hep-ph]].
  * (153) R. H. Brandenberger and M. Yamaguchi, Spontaneous baryogenesis in warm inflation, Phys. Rev. D 68 (2003), 023505 doi:10.1103/PhysRevD.68.023505 [arXiv:hep-ph/0301270 [hep-ph]].
  * (154) M. Bastero-Gil, A. Berera, R. O. Ramos and J. G. Rosa, Warm baryogenesis, Phys. Lett. B 712 (2012), 425-429 doi:10.1016/j.physletb.2012.05.032 [arXiv:1110.3971 [hep-ph]].
  * (155) M. Bastero-Gil, A. Berera, R. O. Ramos and J. G. Rosa, Observational implications of mattergenesis during inflation, JCAP 10 (2014), 053 doi:10.1088/1475-7516/2014/10/053 [arXiv:1404.4976 [astro-ph.CO]].
  * (156) J. C. Bueno Sanchez, M. Bastero-Gil, A. Berera, K. Dimopoulos and K. Kohri, The gravitino problem in supersymmetric warm inflation, JCAP 03 (2011), 020 doi:10.1088/1475-7516/2011/03/020 [arXiv:1011.2398 [hep-ph]].
  * (157) S. Bartrum, A. Berera and J. G. Rosa, Gravitino cosmology in supersymmetric warm inflation, Phys. Rev. D 86 (2012), 123525 doi:10.1103/PhysRevD.86.123525 [arXiv:1208.4276 [hep-ph]].
  * (158) K. Dimopoulos and L. Donaldson-Wood, Warm quintessential inflation, Phys. Lett. B 796 (2019), 26-31 doi:10.1016/j.physletb.2019.07.017 [arXiv:1906.09648 [gr-qc]].
  * (159) J. G. Rosa and L. B. Ventura, Warm Little Inflaton becomes Dark Energy, Phys. Lett. B 798 (2019), 134984 doi:10.1016/j.physletb.2019.134984 [arXiv:1906.11835 [hep-ph]].
  * (160) G. B. F. Lima and R. O. Ramos, Unified early and late Universe cosmology through dissipative effects in steep quintessential inflation potential models, Phys. Rev. D 100 (2019) no.12, 123529 doi:10.1103/PhysRevD.100.123529 [arXiv:1910.05185 [astro-ph.CO]].
  * (161) M. R. Gangopadhyay, S. Myrzakul, M. Sami and M. K. Sharma, Paradigm of warm quintessential inflation and production of relic gravity waves, Phys. Rev. D 103 (2021) no.4, 043505 doi:10.1103/PhysRevD.103.043505 [arXiv:2011.09155 [astro-ph.CO]].
  * (162) S. Basak, S. Bhattacharya, M. R. Gangopadhyay, N. Jaman, R. Rangarajan and M. Sami, The paradigm of warm quintessential inflation and spontaneous baryogenesis, JCAP 03 (2022) no.03, 063 doi:10.1088/1475-7516/2022/03/063 [arXiv:2110.00607 [astro-ph.CO]].
  * (163) R. Saleem and F. Mehmood, A study of warm inflation model inspired by some inhomogeneous dark energy fluids, Eur. Phys. J. Plus 136 (2021) no.5, 570 doi:10.1140/epjp/s13360-021-01535-4
  * (164) M. Levy, J. G. Rosa and L. B. Ventura, Warm inflation, neutrinos and dark matter: a minimal extension of the Standard Model, JHEP 12 (2021), 176 doi:10.1007/JHEP12(2021)176 [arXiv:2012.03988 [hep-ph]].
  * (165) X. M. Zhang, K. Li, Y. F. Guo, P. C. Chu, H. Liu and J. Y. Zhu, Two models unifying warm inflation with dark matter and dark energy, Phys. Rev. D 104 (2021) no.10, 103513 doi:10.1103/PhysRevD.104.103513 [arXiv:2111.14138 [gr-qc]].
  * (166) P. M. Sá, Triple unification of inflation, dark energy, and dark matter in two-scalar-field cosmology, Phys. Rev. D 102 (2020) no.10, 103519 doi:10.1103/PhysRevD.102.103519 [arXiv:2007.07109 [gr-qc]].
  * (167) R. D’Agostino and O. Luongo, Cosmological viability of a double field unified model from warm inflation, Phys. Lett. B 829 (2022), 137070 doi:10.1016/j.physletb.2022.137070 [arXiv:2112.12816 [astro-ph.CO]].
  * (168) R. Arya, Formation of Primordial Black Holes from Warm Inflation, JCAP 09 (2020), 042 doi:10.1088/1475-7516/2020/09/042 [arXiv:1910.05238 [astro-ph.CO]].
  * (169) M. Bastero-Gil and M. S. Díaz-Blanco, Gravity waves and primordial black holes in scalar warm little inflation, JCAP 12 (2021) no.12, 052 doi:10.1088/1475-7516/2021/12/052 [arXiv:2105.08045 [hep-ph]].
  * (170) M. Correa, M. R. Gangopadhyay, N. Jaman and G. J. Mathews, Primordial black-hole dark matter via warm natural inflation, Phys. Lett. B 835 (2022), 137510 doi:10.1016/j.physletb.2022.137510 [arXiv:2207.10394 [gr-qc]].
  * (171) R. Arya and A. K. Mishra, Scalar induced gravitational waves from warm inflation, Phys. Dark Univ. 37 (2022), 101116 doi:10.1016/j.dark.2022.101116 [arXiv:2204.02896 [astro-ph.CO]].
  * (172) G. Ballesteros, M. A. G. García, A. P. Rodríguez, M. Pierre and J. Rey, Primordial black holes and gravitational waves from dissipation during inflation, JCAP 12 (2022), 006 doi:10.1088/1475-7516/2022/12/006 [arXiv:2208.14978 [astro-ph.CO]].