11institutetext: Department of Computer Science and Engineering, Indian
Institute of Technology Bhilai, Chhattisgarh, India
11email<EMAIL_ADDRESS>
# Text2Gender: A Deep Learning Architecture for Analysis of Blogger’s Age and
Gender

Vishesh Thakur  Aneesh Tickoo

###### Abstract

> Deep learning techniques have gained a lot of traction in the field of NLP
> research. The aim of this paper is to predict the age and gender of an
> individual by inspecting their written text. We propose a supervised BERT-
> based classification technique in order to predict the age and gender of
> bloggers. The dataset used contains 681284 rows of data, with the
> information of the blogger’s age, gender, and text of the blog written by
> them. We compare our algorithm to previous works in the same domain and
> achieve a better accuracy and F1 score. The accuracy reported for the
> prediction of age group was 84.2%, while the accuracy for the prediction of
> gender was 86.32%. This study relies on the raw capabilities of BERT to
> predict the classes of textual data efficiently. This paper shows promising
> capability in predicting the demographics of the author with high accuracy
> and can have wide applicability across multiple domains.

##  1 Introduction

Gender estimation based on textual content has been a common area of research
in the field of natural language processing. Blogs, in particular, provide a
rich source of information for studying gender-related patterns, as authors
often express their thoughts, opinions, and experiences through their
writings. The ability to accurately estimate the gender of blog authors can
have significant implications in various domains such as marketing, social
sciences, and online content personalization.

In this study, we focus on estimating the gender of blog authors based on the
content they write. Our approach leverages the power of advanced language
models, specifically BERT (Bidirectional Encoder Representations from
Transformers)[4], for encoding the textual information present in the blogs.
BERT has gained immense popularity for its ability to capture deep contextual
representations of text, making it well-suited for various natural language
processing tasks.

To build our gender estimation model, we employ a supervised learning
approach. We utilize a large dataset of blogs where the gender of the authors
is known. The blogs are preprocessed and encoded using BERT, which generates
high-dimensional representations capturing the semantic and syntactic
properties of the text. These encoded representations serve as input features
for training a machine learning model, with gender as the categorical target
variable.

One key aspect of our work is the novel application of advanced sentence
models like BERT for gender estimation in text. While previous
studies[7][1][6] have explored gender estimation using traditional feature
engineering techniques[5] and simpler models, the utilization of BERT opens up
new possibilities for capturing more nuanced patterns and improving the
accuracy of gender predictions.

By employing BERT as the encoding backbone and training our model on a large
dataset, we aim to achieve robust and reliable gender estimation results.
Through rigorous evaluation and analysis, we will assess the performance of
our model and compare it with existing approaches in the field. The insights
gained from this research can contribute to a deeper understanding of the
relationship between language use and gender, furthering our knowledge in the
realm of computational gender studies.

In summary, this study presents a novel approach to estimating the gender of
blog authors based on their writings. By leveraging the power of BERT, we aim
to achieve accurate gender predictions and explore the potential of advanced
sentence models in the domain of Text2Age. Our work contributes to the growing
body of research in gender estimation and showcases the value of state-of-the-
art language models in addressing challenging natural language processing
tasks.

##  2 Related Work

Previous research has explored the usage of language patterns by different
social groups, although limited by the availability of annotated data. The
advent of the blogosphere and its accessibility for data collection has
greatly facilitated research in this area. Several studies have examined the
effects of bloggers’ age and gender on language usage in weblogs [11]; [3];
Yan, 2006; [2]; [13]; [8]).

However, a comprehensive analysis of bloggers’ age and gender based on more
advanced language models like BERT is lacking in the existing literature.
Previous studies have primarily relied on keyword analysis, parts of speech,
and other grammatical constructs for gender estimation and age determination.
While some progress has been made in gender estimation [11], less attention
has been given to age determination.

For instance, [9] and Pennebaker and [10] examined age-related variations in
language use, but their approaches were not specifically tailored to blog
data. Burger and Henderson (2006) also explored age-related patterns, but a
more comprehensive analysis using advanced language models is still needed.

Additionally, gender estimation has received more attention in previous
research. Koppel, Argamon, and Shimoni (2003) estimated authors’ gender using
the British National Corpus text, while Schler et al. (2006) achieved 80%
accuracy in classifying authors’ gender by analyzing function words and part-
of-speech. Their findings indicated that female authors tend to use pronouns
with high frequency, while male authors tend to use numerals and
representation-related numbers more frequently.

However, these previous studies were limited in their approaches and did not
leverage advanced sentence models like BERT, which have shown promising
results in various natural language processing tasks. Our research aims to
address this gap by employing BERT for gender estimation and exploring its
potential in determining the age of blog authors based on their writings.

By utilizing advanced language models and training our model on a large
annotated dataset, we aim to achieve higher accuracy in gender estimation and
age determination compared to previous studies. The incorporation of BERT
allows us to capture deeper contextual representations and uncover more
nuanced patterns in language use.

In summary, while previous research has examined the relationship between
language patterns and gender or age in the context of weblogs[8], our study
contributes to the existing literature by employing advanced language models
like BERT. This enables us to enhance the accuracy of gender estimation and
explore the feasibility of determining the age of blog authors. Our work aims
to advance the field of computational gender studies and provide valuable
insights into the analysis of bloggers’ gender and age based on their written
content.

##  3 Dataset and Computation Environment

The dataset[11] we used for this study is called the ’Blog Authorship Corpus’.
The Blog Authorship Corpus is a collection of posts written by 19,320
bloggers, which were obtained from blogger.com in August 2004. The corpus
comprises a total of 681,288 posts and more than 140 million words, which
translates to roughly 35 posts and 7250 words per person. Each blog is
presented in a separate file and is identified by a unique blogger ID, as well
as the blogger’s self-reported gender, age, industry, and astrological sign.
While all bloggers’ genders and ages are labeled, industry and sign
information is unknown for some. The corpus is divided into three age groups:
8240 "10s" blogs (ages 13-17), 8086 "20s" blogs (ages 23-27), and 2994 "30s"
blogs (ages 33-47), with an equal number of male and female bloggers in each
group. Additionally, each blog in the corpus contains at least 200 occurrences
of common English words. The corpus is devoid of all formatting except for two
exceptions, i.e., individual posts within a single blogger are separated by
the date of the subsequent post, and links within a post are marked by the
label URL link.

Age Group | Number of Subjects
---|---
10s (13 to 17) | 8240
20s (23 to 27) | 8086
30s (33 to 47) | 2994

Moreover, the population of Males and Females is equally distributed among all
the classes. Now let’s have a look at the final data that is presented in the
form of entire dataset. We have seen above that which age group has how many
unique participants or subjects. Now let’s have a look at what is the number
for the total dataset as although the participants are unique, the texts are
not. This essentially means that a single author may write more than one, or
multiple pieces of text, which is fairly understood. Let’s have a look at the
data about the demographics of all the texts. First, let’s see about the data
distribution of age groups.

Age Group | Number of Subjects
---|---
10s (13 to 17) | 236024
20s (23 to 27) | 321447
30s (33 to 47) | 98056

The distribution of data among the two genders is as below.

Gender | Number of Subjects
---|---
Male | 345193
Female | 336091

The data is almost fairly distributed among the male and female populations as
per the final dataset. To feed the data into the network, we had to break the
data into training and testing sets. We did a 4:1 split and the distribution
of our data into the training and testing set is as follows.

Training Set | 510964
---|---
Testing Set | 170320

The computing system used was a commercial laptop with 8GB RAM, 2 GB NVIDIA
GeForce MTX250 graphics card, and Intel i7 10th generation processor.

##  4 Preprocessing

Preprocessing of data played an important role in improving the accuracy of
the data. For the prediction of age group, accuracy without preprocessing was
67.6%, while after preprocessing it improved to 84.2%, while for prediction of
gender, before preprocessing it was 70.6%, and after preprocessing became
86.32%.

Input blog textRemoving unnecessary characters (hashtags, emojis etc.)Stop
words removalNormalization of text and lemmatization

Above is the text cleaning pipeline that was followed. Preprocessing was a
crucial step in order to increase accuracy and other metrics. The effects
observed by this step are documented in the table below.

| Accuracy Before | Accuracy After
---|---|---
Gender Prediction | 70.6% | 86.32%
Age Prediction | 67.6% | 84.2%

##  5
Architecture and Pipeline

After the text is cleaned and passed through the preprocessing pipeline, we
then again pass it through the BERT preprocess module, in order to maintain a
high quality of the final dataset that we feed to our network. Our next step
is to prepare the dataset and feed it into our model to fine-tune BERT. A
detailed step-by-step exploration of the architecture is shown below. We used
a total of three layers of BERT, and two layers of neural networks for
different purposes. First, let’s have a look at the BERT layers.

Input Layer (text input)Preprocessed Text (BERT preprocess(text input))Otput
Layer (BERT Encoder (Preprocessed Text))

Following are the layers of neural network that we use once the data is passed
through the BERT layers. We have a total of two layers, with one being a
dropout layer, which is used in order to reduce or prevent the overfitting of
the model. The second layer is a sigmoid activation layer. This is required
because in order to classify whether the input text is written by a male or a
female, we need some probabilities, and this layer does the same. The sigmoid
layer returns a number between 0 to 1. The representation of how a sigmoid
curve looks and how it maps the results between 0 to 1 is as below.

$\sigma=1/(1+exp(-x))$

$-6$$-4$$-2$$0$$2$$4$$6$$0.5$$1$$\sigma(x)$$\sigma^{\prime}(x)$

We have kept this as a convention that Male corresponds to 0, while Female
corresponds to 1. So, with a number between 0 to 1, we can decide whether it
maps to Male or Female.

Dropout LayerSigmoid Function Activation Layer

We use the default settings of BERT, with the number of layers at 12, with 12
attention heads. The optimizer used was ’Adam’, while the loss used was
’Binary Crossentropy’, whose formula is given as
$-{(y\log(p)+(1-y)\log(1-p))}$, where y is the binary indicator (0 or 1) if
class label c is the correct classification for observation o, and p is the
predicted probability observation o is of class c. Finally, let’s have a look
at our network in its entirety to get a clear picture of what are the pathways
our data will go through to get the final result.

Input Layer (text input)Preprocessed Text (BERT preprocess(text input))Otput
Layer (BERT Encoder (Preprocessed Text))Dropout LayerSigmoid Function
Activation Layer

##  6
Results and Conclusion

After feeding our data in the above pipeline, we get some interesting results.
The training was done for 150 epochs on a TensorFlow-based kernel. First,
we’ll discuss about the results we got while trying to predict age from the
given text. Let’s have a look at how the textual inputs were classified by our
model.

Figure 1: Confusion matrix for gender prediction

True Positive | 78456
---|---
True Negative | 68564
False Positive | 13865
False Negative | 9435

Using the above table, we can see the number of true positives, false
positives, true negatives, and false negatives. Using these values, we can
find some more useful metrics which can further inform us about how well our
model has performed or how badly has it erred. If we try to evaluate our model
on the basis of some standard metrics, the result is as follows:

Precision | Recall | Accuracy | F1 Score
---|---|---|---
0.8498 | 0.8926 | 0.8632 | 0.8706

Now, when we focus on the prediction class of age, then the results that we
have observed are as follows.

Figure 2: Confusion matrix for age prediction

True Positive | 80132
---|---
True Negative | 63277
False Positive | 17324
False Negative | 9587

if we have a look at the metrics, then we can also compare the two, age and
gender prediction among themselves and find out for which task our algorithm
does a better task.

Precision | Recall | Accuracy | F1 Score
---|---|---|---
0.8223 | 0.8932 | 0.8420 | 0.8562

The results we got are pretty good, especially if we consider the previous
works[5]. The great thing is that we have a healthily good score of both
precision and recall and an equally good F1 score. If we consider a comparison
between the prediction of age class and gender, then the results show that our
algorithm does a better job in predicting gender in comparison to predicting
the age class, or group. This may be attributed to the fact that gender data
had two labels, while the age class had 3 labels (10s, 20s, and 30s).

Feature | Precision | Recall | Accuracy | F1 Score
---|---|---|---|---
Age | 0.8223 | 0.8932 | 0.8420 | 0.8562
Gender | 0.8498 | 0.8926 | 0.8632 | 0.8706

The prediction of gender has better precision, accuracy, and F1 score, while
it lags marginally in recall in comparison to the prediction of age class.
We can conclude that BERT-based textual classification[12] works really well
for age and gender prediction tasks, and can be used in many real-case
scenarios. We also observe that this method is marginally better for gender
prediction, or in general we can say that this method works well for
classification if the dataset has a less number of classes.

## References

  * [1] Abdul-Mageed, M., Zhang, C., Rajendran, A., Elmadany, A., Przystupa, M., Ungar, L.: Sentence-level bert and multi-task learning of age and gender in social media (11 2019)
  * [2] Argamon, S., Shimoni, A.: Automatically categorizing written texts by author gender. Lit Linguistic Computing 17 (12 2003)
  * [3] Burger, J., Henderson, J., Kim, G., Zarrella, G.: Discriminating gender on twitter. pp. 1301–1309 (01 2011)
  * [4] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding (2019)
  * [5] Goswami, S., Sarkar, S., Rustagi, M.: Stylometric analysis of bloggers’ age and gender. Proceedings of the International AAAI Conference on Web and Social Media 3(1), 214–217 (Mar 2009). https://doi.org/10.1609/icwsm.v3i1.13992, https://ojs.aaai.org/index.php/ICWSM/article/view/13992
  * [6] Lopes Filho, J.A., Pasti, R., De Castro, L.: Gender Classification of Twitter Data Based on Textual Meta-Attributes Extraction, pp. 1025–1034 (03 2016). https://doi.org/10.1007/978-3-319-31232-3_97
  * [7] Mukherjee, S., BALA, P.: Gender classification of microblog text based on authorial style. Information Systems and e-Business Management 15 (02 2017). https://doi.org/10.1007/s10257-016-0312-0
  * [8] Nowson, S., Oberlander, J.: The identity of bloggers: Openness and gender in personal weblogs. pp. 163–167 (01 2006)
  * [9] Pennebaker, J., Francis, M., Booth, R.: Linguistic inquiry and word count (liwc) (01 1999)
  * [10] Pennebaker, J., Stone, L.: Words of wisdom: Language use over the life span. Journal of personality and social psychology 85, 291–301 (09 2003). https://doi.org/10.1037/0022-3514.85.2.291
  * [11] Schler, J., Koppel, M., Argamon, S., Pennebaker, J.: Effects of age and gender on blogging. pp. 199–205 (01 2006)
  * [12] Sun, C., Qiu, X., Xu, Y., Huang, X.: How to fine-tune bert for text classification? In: Sun, M., Huang, X., Ji, H., Liu, Z., Liu, Y. (eds.) Chinese Computational Linguistics. pp. 194–206. Springer International Publishing, Cham (2019)
  * [13] Yan, X., Yan, L.: Gender classification of weblog authors. In: AAAI Spring Symposium: Computational Approaches to Analyzing Weblogs (2006)


######  Lemma 9 (Corollary of [22, Lemma 2.1])

Let $\mathcal{C}$ be a distinctly-labelled $k$-DCN over alphabet $\Gamma$
(obtained as per Lemma 8), then there exists a single-state $(k+3)$-DCN
$\mathcal{U}$ over alphabet
$\Upsilon=\\{\gamma_{1},\gamma_{2},\gamma_{3}\mid\gamma\in\Gamma\\}$ such that
$\mathcal{L}(\mathcal{U})=\\{a_{1}a_{2}a_{3}\,b_{1}b_{2}b_{3}\cdots c_{1},$
$a_{1}a_{2}a_{3}\,b_{1}b_{2}b_{3}\cdots c_{1}c_{2},$
$a_{1}a_{2}a_{3}\,b_{1}b_{2}b_{3}\cdots c_{1}c_{2}c_{3}$ $\mid a\,b\cdots
c\in\mathcal{L}(\mathcal{C})\wedge a,b,\ldots,c\in\Gamma\\}$.

###### Proof

We give the construction of the VAS, this proof only differs from the proof
given by Hopcroft and Pansoit [22] by the inclusion of transition labels.
Assume that $\mathcal{C}$ has $n$ states $Q=\\{q_{1},\ldots,q_{n}\\}$. For
each $i\in\\{1,\ldots,n\\}$, let $a_{i}=i$ and $b_{i}=(n+1)(n+1-i)$. For each
transition $t=(q_{i},\sigma,\boldsymbol{x},q_{j})$ in $\mathcal{C}$, there are
three transitions
$t_{1}=\sigma_{1},(\boldsymbol{0},-a_{i},a_{n+1-i}-b_{i},b_{n+1-i})$,
$t_{2}=\sigma_{2},(\boldsymbol{0},b_{i},-a_{n+1-i})$, and
$t_{3}=\sigma_{3},(\boldsymbol{x},a_{j}-b_{i},b_{j},-a_{i})$ in $\mathcal{U}$
(we omit the states from the transitions, since there is only one state).

Observe that since each transition in $\mathcal{C}$ has a distinct label, then
each of the transitions in $\mathcal{U}$ also has a distinct label, making
$\mathcal{U}$ a DCN. It remains to show that $\mathcal{U}$ recognises the
language given in the statement of this lemma. Suppose the word
$\sigma_{1}\,\sigma_{2}\,\cdots\,\sigma_{k}\in\mathcal{L}(\mathcal{C})$, then
there is a path $\pi=(t_{i})_{i=1}^{k}$ in $\mathcal{C}$, so
$t_{i}=(q_{i},\sigma_{i},\boldsymbol{x}_{i},q_{i+1})$. Since $\mathcal{U}$
faithfully simulates $\mathcal{C}$, the path
$\pi^{\prime}=(t_{i,1}\,t_{i,2}\,t_{i,3})_{i=1}^{k}$ in $\mathcal{U}$
witnesses the word
$\sigma_{1,1}\,\,\\!\sigma_{1,2}\,\,\\!\sigma_{1,3}\;\sigma_{2,1}\,\,\\!\sigma_{2,2}\,\,\\!\sigma_{2,3}\,\cdots\,\sigma_{k,1}\,\,\\!\sigma_{k,2}\,\,\\!\sigma_{k,3}$.
In fact, the last three transitions need not all be executed; indeed
$\sigma_{1,1}\,\,\\!\sigma_{1,2}\,\,\\!\sigma_{1,3}\,\cdots\,\sigma_{k,1}\in\mathcal{L}(\mathcal{U})$,
$\sigma_{1,1}\sigma_{1,2}\sigma_{1,3}\,\cdots\,\sigma_{k,1}\,\,\\!\sigma_{k,2}\in\mathcal{L}(\mathcal{U})$,
and
$\sigma_{1,1}\,\,\\!\sigma_{1,2}\,\,\\!\sigma_{1,3}\,\cdots\,\sigma_{k,1}\,\,\\!\sigma_{k,2}\,\,\\!\sigma_{k,3}\in\mathcal{L}(\mathcal{U})$.
∎

###### Proposition 4

Let $L\subseteq\\{a,b,\ldots,c\\}^{*}$ and
$L^{\prime}\subseteq\\{a_{1},a_{2},a_{3},b_{1},b_{2},b_{3},\ldots,c_{1},c_{2},c_{3}\\}^{*}$
such that $L^{\prime}=\\{a_{1}a_{2}a_{3}\,b_{1}b_{2}b_{3}\cdots c_{1},$
$a_{1}a_{2}a_{3}\,b_{1}b_{2}b_{3}\cdots c_{1}c_{2},$
$a_{1}a_{2}a_{3}\,b_{1}b_{2}b_{3}\cdots c_{1}c_{2}c_{3}$ $\mid
a\,b\,\cdots\,c\in L\\}$, then $L$ is regular if and only if $L^{\prime}$ is
regular.

###### Proof

This is a basic exercise in automata. The direction $L$ is regular $\implies$
$L^{\prime}$ is regular is proved by constructing a DFA for $L^{\prime}$ that
verifies letters are read in triplets (i.e., $a_{1}a_{2}a_{3}$, possibly
stopping in the middle), and after every triplet simulates the corresponding
transition in a DFA for $L$.

The direction $L^{\prime}$ is regular $\implies$ $L$ is regular is proved by
constructing an NFA for $L$ that given letter $a$ simulates the transition of
a DFA for $L^{\prime}$ on $a_{1}a_{2}a_{3}$. ∎

By combining Lemma 8, Lemma 9, Proposition 4, and [3, Theorem 4.5], we can
conclude with the proof of Theorem 5.2.

#### Proof of Theorem 5.2.

Given a $k$-CN $\mathcal{D}$, construct a distinctly-labelled $k$-DCN
$\mathcal{C}$ as per Lemma 8. Next, apply Lemma 9 to obtain a single-state,
distinctly-labelled $k+3$-DVAS $\mathcal{U}$. Finally, treat $\mathcal{U}$ as
a VAS. Indeed, regularity of VAS is with respect to the transition “names”,
which are unique in $\mathcal{U}$ due to the distinct labels. We can then
decide the regularity of $\mathcal{U}$ using [3, Theorem 4.5].

We have that $\mathcal{U}$ is regular if and only if
$\mathcal{L}(\mathcal{D})$ is regular by Lemmas 8, 9 and 4. Moreover, our
construction can clearly be implemented in polynomial time, thus giving the
same complexity bound.∎