# BSpell: A CNN-blended BERT Based Bengali Spell Checker

Chowdhury Rafeed Rahman
MD. Hasibur Rahman
Samiha Zakir
Mohammad Rafsan
Mohammed Eunus Ali

###### Abstract

Bengali typing is mostly performed using English keyboard and can be highly
erroneous due to the presence of compound and similarly pronounced letters.
Spelling correction of a misspelled word requires understanding of word typing
pattern as well as the context of the word usage. We propose a specialized
BERT model, BSpell targeted towards word for word correction in sentence
level. BSpell contains an end-to-end trainable CNN sub-model named SemanticNet
along with specialized auxiliary loss. This allows BSpell to specialize in
highly inflected Bengali vocabulary in the presence of spelling errors. We
further propose hybrid pretraining scheme for BSpell combining word level and
character level masking. Utilizing this pretraining scheme, BSpell achieves
91.5% accuracy on real life Bengali spelling correction validation set.
Detailed comparison on two Bengali and one Hindi spelling correction dataset
shows the superiority of proposed BSpell over existing spell checkers.

##  1 Introduction

Bengali is the native language of 228 million people which makes it the sixth
most spoken language in the world
111https://www.babbel.com/en/magazine/the-10-most-spoken-languages-in-the-
world. This Sanskrit originated language has 11 vowels, 39 consonants, 11
modified vowels and 170 compound characters Sifat et al. (2020). There is vast
difference between Bengali grapheme representation and phonetic utterance for
many commonly used words. As a result, fast typing of Bengali yields frequent
spelling mistakes. Almost all Bengali native speakers type using English
QWERTY layout keyboard (Noyes, 1983) which makes it difficult to type Bengali
compound characters, phonetically similar single characters and similar
pronounced modified vowels correctly. Thus Bengali typing speed, if error-free
typing is desired, is really slow. An accurate spell checker (SC) can be a
solution to this problem.

Existing Bengali SCs include phonetic rule based (UzZaman and Khan, 2004,
2005) and clustering based methods (Mandal and Hossain, 2017). These methods
do not take misspelled word context into consideration. Another N-gram based
Bengali SC (Khan et al., 2014) takes only short range previous context into
consideration. Recent state-of-the-art (SOTA) SCs have been developed for
Chinese language, where a character level confusion set (similar characters)
guided seq2seq model has been proposed by Wang et al. (2019). Another research
used similarity mapping graph convolutional network in order to guide BERT
based character by character parallel correction (Cheng et al., 2020). Both
these methods require external knowledge and assumption about confusing
character pairs existing in the language. The most recent Chinese SC offers an
assumption free BERT architecture where error detection network based soft-
masking is included (Zhang et al., 2020). This model takes all $N$ characters
of a sentence as input and produces the correct version of these $N$
characters as output in a parallel manner.

Figure 1: Heterogeneous character number between error and correct spelling

One of the limitations in developing Bengali SC using SOTA BERT based
implementation (Zhang et al., 2020) is that number of input and output
characters in BERT have to be exactly the same. This scenario is problematic
due to the following reasons. First, such scheme is only capable of correcting
substitution type errors. Deletion or insertion will certainly alter word
length. As compound characters are commonly used in Bengali words, an error
made due to the substitution of such characters also changes word length. This
scenario has been shown in Figure 1. The correct version of the first word of
this figure contains 7 characters, whereas the misspelled version contains
only 5 characters due to typing mistake in place of the only compound
character of this word. So, character based BERT correction is not feasible
for Bengali. Hence, we introduce word level prediction in our proposed BERT
based model.

Figure 2: Sample words that are correctly spelled accidentally, but are
context-wise incorrect.

Figure 2 illustrates the importance of context in Bengali SC. Although the red
marked words of this figure are the misspelled versions of the corresponding
green marked correct words, these red words are valid Bengali words. But if we
check these red words based on sentence semantic context, we can realize that
these words have been produced accidentally because of spelling error. In
sentence one, the red word means to visit. But the sentence context hints
about a soldier riding towards battleground. A slight modification of this
word makes it correct, which now has the meaning of horse. We argue that in
order to correct a misspelled word, we need to consider the word pattern, its
prior context and its post context.

Figure 3: Necessity of understanding semantic meaning of existing erroneous
words for spelling correction of an individual misspelled word

Real life Bengali spelling errors while typing fast often span up to multiple
words in a sentence. Such a real life example has been provided in Figure 3,
where all four words (W1, W2, W3 and W4) have been misspelled during speed
typing. The correction of each word has context dependency on a few other
words of the same sentence. The table in the figure shows the words to look at
in order to correct each misspelled word. The problem is that the words which
provide correction context are also spelled incorrectly in the sentence. Our
proposed solution needs to have the ability of capturing the semantic meaning
of a word even if it is misspelled in order to provide the right correction
context.

We propose a word level BERT (Devlin et al., 2018) based model namely BSpell.
This model is capable of learning prior and post context dependency through
the use of multi-head attention mechanism of stacked Transformer encoders
(Vaswani et al., 2017). The model has the following major features.
SemanticNet: Each input word in a sentence has to pass through learnable CNN
sub-layers so that the model can capture semantic meaning even if the words
are misspelled. These learnable sub-layers form SemanticNet.
Auxiliary loss: We also introduce a specialized auxiliary loss to facilitate
SemanticNet sub-model learning in order to emphasize on the importance of user
writing pattern at word level. This loss has been placed at an intermediate
level of the model instead of the final level thus eliminating the gradient
vanishing problem prevalent in very deep architectures.
Hybrid pretraining: Existing BERT pretraining schemes (Liu et al., 2019) are
designed mainly to understand language context. In order to capture both
context and word error pattern, we introduce hybrid pretraining. Besides word
masking based fill in the gaps task, hybrid pretraining incorporates word
prediction task after blurring a few characters of some words. Such scheme
provides the BERT model with ideas about retrieving a word from its noisy
version.
We perform detailed evaluation on three error datasets that include a real
life Bengali error dataset. Our evaluation includes detailed analysis on
possible LSTM based SCs as well.

##  2 Related Works

Several studies on Bengali SC development have been conducted in spite of
Bengali being a low resource language. A phonetic encoding oriented Bengali
word level SC based on Soundex algorithm was proposed by UzZaman and Khan
(2004). This encoding scheme was later modified to develop a Double Metaphone
encoding based Bengali SC (UzZaman and Khan, 2005). They took into account
major context-sensitive rules and consonant clusters while performing their
encoding scheme. Another word level Bengali SC able to handle both
typographical and phonetic errors was proposed by Mandal and Hossain (2017)..
An N gram model was proposed by Khan et al. (2014) for checking sentence level
Bengali word correctness. They trained the model using a corpus containing one
million word tokens. An encoder-decoder based seq2seq model was proposed by
Islam et al. (2018) for Bengali sentence correction task which involved bad
arrangement of words and missing words, though this work did not include
incorrect spelling. A recent study on low resource language has included Hindi
and Telugu spell checker development, where mistakes are assumed to be made at
character level (Etoori et al., 2018). They have used attention based encoder
decoder modeling as their approach.

SOTA research in this domain involves Chinese SCs as it is an error prone
language due to its confusing word segmentation, phonetically and visually
similar but semantically different characters. A seq2seq model assisted by a
pointer network was employed for character level spell checking where the
network is guided by externally generated character confusion set (Wang et
al., 2019). Another research incorporated phonological and visual similarity
knowledge of Chinese characters into BERT based SC model by utilizing graph
convolutional network (Cheng et al., 2020). A recent BERT based SC has taken
advantage of GRU based soft masking mechanism and has achieved SOTA
performance in Chinese character level spell checking in spite of not
providing any external knowledge to the network (Zhang et al., 2020). Another
external knowledge free approach namely FASPell used BERT based seq2seq model
for Chinese spelling correction (Hong et al., 2019). HanSpeller++ is notable
among initially implemented Chinese SCs (Xiong et al., 2015). It was an
unified framework utilizing a hidden Markov model for candidate generation and
a filter to rank candidate solutions.

##  3 Our Approach

###  3.1 Problem Statement

Suppose, an input sentence consists of $n$ words – $Word_{1}$, $Word_{2}$, …,
$Word_{n}$. For each $Word_{i}$, we have to predict the right spelling, if
$Word_{i}$ exists in the top-word list of our corpus. If $Word_{i}$ is a rare
word (Proper Noun in most cases), we predict $UNK$ token denoting that we do
not make any correction to such words. For correcting a particular $Word_{i}$
in a paragraph, we only consider other words of the same sentence for context
information.

###  3.2 BSpell Architecture

Figure 4: BSpell architecture details

Figure 4 shows the details of BSpell architecture. Each input word of the
document is represented by a matrix (each character represented by an
embedding vector). Each word representative matrix is passed through our
proposed 1D CNN based SemanticNet sub-model. This sub-model returns us with a
SemanticVec vector representation for each input word. So, if there are $n$
input words, then we now have $n$ SemanticVec vectors. These vectors are then
passed onto two separate branches (main branch and secondary branch)
simultaneously. The main branch is similar to BERT_Base architecture (Gong et
al., 2019) which is basically a stack of Transformer encoders. This branch
provides us with the $n$ correct words corresponding to the $n$ input document
words at its output side. The secondary branch consists of an output dense
layer. The output layer of this branch is similar to the output layer of the
main branch. This branch is used for the sole purpose of imposing auxiliary
loss to facilitate SemanticNet sub-model learning of misspelled word patterns.
We describe each component of our BSpell architecture in more detail below.

####  3.2.1 SemanticNet Sub-Model

Correcting a particular word requires the understanding of other relevant
words in the same sentence. Unfortunately, those relevant words may also be
misspelled. As humans, we can understand the meaning of a word even if it is
misspelled because of our deep understanding at word syllable level and our
knowledge of usual spelling error pattern. We want our model to have similar
semantic level understanding of the words. We propose SemanticNet, a
sequential 1D CNN sub-model that is employed at each individual word level
with a view to learning intra word syllable pattern. Details of individual
word representation has been shown in bottom right corner of Figure 4. We
represent each input word $Word_{i}$ by a matrix $W_{i}$. $W_{i}$ consists of
$m$ vectors each of size $len_{C}$. These vectors
$\vec{C_{1}},\vec{C_{2}},\ldots\vec{C_{m}}$ represent one hot vector of
characters $C_{1},C_{2},\ldots C_{m}$, respectively of $Word_{i}$ including
padding. Applying 1D convolution (conv) layers on matrix $W_{i}$ helps in
deriving key local patterns and sub-word information of $Word_{i}$. 1D conv
operation starting from vector $\vec{C_{j}}$ of $W_{i}$ using a conv filter of
kernel size $K$ can be expressed as
$1D\\_Conv([\vec{C_{j}},\vec{C_{j+1}},\ldots\vec{C_{j+K-1}}],filter_{K})$
function. The output of this function is a scalar value. While using a stride
of $s$, the next conv operation starts from vector $\vec{C_{j+s}}$ of $W_{i}$
and will provide us with another scalar value. Thus we get a vector as a
result of applying one conv filter on matrix $W_{i}$. A conv layer has
multiple such conv filters. After passing $W_{i}$ matrix through the first
conv layer we obtain feature matrix $W^{1}_{i}$. Passing $W^{1}_{i}$ through
the second conv layer provides us with feature matrix $W^{2}_{i}$. So, the
$L^{th}$ conv layer provides us with feature matrix $W^{L}_{i}$. SemanticNet
sub-model consists of such 1D conv layers standing sequentially one after the
other. Such design helps in learning both short and long range intra-word
level features. Conv layers near matrix $W_{i}$ are responsible for
identifying key sub-word patterns of $Word_{i}$, while conv layers further
away focus on different combinations of these key sub-word patterns. Such word
level local pattern recognition plays key role in identifying semantic meaning
of a word irrespective of inflection or presence of spelling error. Each
subsequent conv layer has non-decreasing filter spatial size and increasing
number of filters. Finally, after applying global max pooling on feature
matrix $W^{L}_{i}$, we obtain vector $F_{i}$ of size $len_{F}$ for $Word_{i}$;
which we call SemanticVec representation of $Word_{i}$. We get a similar size
vector from each of our input word by applying the same SemanticNet sub-model
on the corresponding matrix representation.

While SemanticNet sub-model shares the word embedding idea of Kim et al.
(2016), it is different in at least two major cases – (a) it has no
integration with typical RNN based models; and (b) rather than using some
parallel convolution filters for word embedding, SemanticNet uses a sequential
1D CNN sub-model. Also Takase et al. (2019) shares a different idea of
attention based n-gram integration for word embedding. We do not use such
attention mechanism in our approach as our main goal is just to capture local
pattern of misspelled words.

####  3.2.2 BERT_Base as Main Branch

Each of the $n$ SemanticVecs obtained from $n$ input words are passed
individually through a dense layer. These modified vectors are then passed
parallelly on to our first Transformer encoder. The intuition behind such
dense layer based modification is to prepare the SemanticVecs for Transformer
based context information extraction. 12 such Transformer encoders are stacked
on top of each other. Each Transformer employs multi head attention mechanism,
layer normalization and dense layer specific modification on each input
vector. The $n$ vectors from $n$ words of the input sentence keep passing on
from one Transformer to another with attention oriented modification based on
their interaction with one another. Such interaction is vital for
understanding the context of each input word. Often context of a word can be
just as important as its error pattern for the spelling correction of that
word. We pass the final Transformer output vectors to a Softmax layer that
employs Softmax activation function on each vector at an individual manner.
So, now we have $n$ probability vectors. Each probability vector contains
$len_{P}$ values, where $len_{P}$ is one more than the total number of top
words considered. This one additional word represents rare words, also known
as UNK. The top word corresponding to the index of the maximum probability
value of $i^{th}$ probability vector represents the correct word for
$Word_{i}$ of the input sentence.

####  3.2.3 Auxiliary Loss in Secondary Branch

Gradient vanishing problem is a common phenomena in deep neural networks (NN),
where weights of the shallow layers are not updated sufficiently during
backpropagation. With the presence of 12 Transformer encoders on top of the
SemanticNet sub-model, the layers of this sub-model certainly lie in a shallow
position of a very deep NN. Although SemanticNet constitutes a small initial
portion of BSpell network, this portion is responsible for word pattern
learning, an important task of SC. In order to eliminate gradient vanishing
problem of SemanticNet and to turn it into an effective pattern based word
level SC, we introduce an auxiliary loss based secondary branch in BSpell.

Each of the $n$ SemanticVecs obtained from $n$ input words are passed
parallelly on to a Softmax layer without any further modification. The outputs
obtained from this branch are called auxiliary outputs. The outputs are
probability vectors exactly similar to the main branch output. The impact of
our auxiliary loss on weight update is as follows:

$L_{Total}=L_{Final}+\lambda\times L_{Auxiliary}$

$L_{Total}$ is the weighted total loss of BSpell model. We want our final loss
to have greater impact on model weight update as it is associated with the
final prediction made by BSpell. Hence, we impose the constraint
$0<\lambda<1$. As $\lambda$ gets closer to 0, the influence of $L_{Auxiliary}$
diminishes more and more on model weight update.

This secondary branch of BSpell does not have any Transformer encoders through
which the input words can interact to produce context information. The
prediction made from this branch is dependent solely on misspelled word
pattern information extracted by SemanticNet. This enables SemanticNet to
learn more meaningful word representation.

Auxiliary losses were first used in CNN architecture GoogleNet by Szegedy et
al. (2015). Our motivation of reducing gradient vanishing problem through
auxiliary loss is similar to GoogleNet. Our proposed auxiliary loss differs
from GoogleNet in at least three ways. First, the main motivation of this loss
is to facilitate independent learning of SemanticNet for word level
correction. Second, the loss is imposed on multiple simultaneous predictions
made by the model, it is not a single target prediction loss. Finally, The
loss has been provided in the context of 1D CNN (sequence) instead of 2D CNN
(image).

###  3.3 BERT Hybrid Pretraining

Figure 5: BERT hybrid pretraining

In contemporary BERT pretraining methods, each input word $Word_{i}$ maybe
kept intact or maybe replaced by a default mask word in a probabilistic manner
(Devlin et al., 2018; Liu et al., 2019). BERT has to predict the masked words
in addition to keeping the unmasked words in tact at the output side. Mistakes
from the BERT side will contribute to loss value accelerating backpropagation
based weight update. Basically, in this process, BERT learns to fill in the
gaps, which in turn teaches the model context of the language. Sun et al.
(2020) proposed incremental ways of pretraining the model for new NLP tasks.
We take a more task specific approach for masking. In SC, recognizing noisy
word pattern is important. But there is no provision for that in contemporary
pretraining. To mitigate the this limitation, we propose hybrid masking. Among
our $n$ input words in a sentence, we randomly replace $n_{W}$ words with a
mask word $Mask_{W}$; where $n>n_{W}$. Among the remaining $n-n_{W}$ words, we
choose $n_{C}$ words for character masking; where $(n-n_{W})>n_{C}$. Let us
now see how we implement character masking on these $n_{C}$ number of chosen
words. We choose $m_{C}$ characters at random from a word having $m$
characters to be replaced by a mask character $Mask_{C}$, where $m>m_{C}$.
Such masked characters introduce noise in words and helps BERT to understand
the probable semantic meaning of noisy words. Misspelled words are actually
similar to such noisy words and as a result, hybrid pretraining boosts
spelling correction specific fine tuning performance. Figure 5 shows a toy
example of hybrid masking on a sentence containing 5 words. Here, $Word_{2}$
(character no. 2) and $Word_{5}$ (character no. 1 and 4) are masked at
character level, while $Word_{4}$ is chosen for word level masking. $Word_{1}$
and $Word_{3}$ remain in tact.

##  4 Experimental Setup

###  4.1 Dataset Specification

Datasets |  | Unique
---
Word
| Unique
---
Char
| Top
---
Word
| Training
---
Sample
| Validation
---
Sample
| Unique
---
Error Word
| Error
---
Word %
| Prothom-Alo
---
(Synthetic Error)
262 K | 73 | 35 K | 1 M | 200 K | 450 K | 52%
| Real Error
---
(Bengali)
14.5 K | 73 | _ | 4.3 K | 2 K | 10 K | 36%
| Pretrain Corpus
---
(Bengali)
513 K | 73 | 40 K | 5.5 M | _ | _ | _
| ToolsForIL (Hindi
---
Synthetic Error)
20.5 K | 77 | 15 K | 75 K | 16 K | 5 K | 10%
| Pretrain Corpus
---
(Hindi)
370 K | 77 | 40 K | 5.5 M | _ | _ | _
Table 1: Dataset specification details

We have used one Bengali and one Hindi corpus with over 5 million (5 M)
sentences for BERT pretraining (see Table 1). There is no validation sample
used while implementing pretraining. Bengali pretraining corpus consists of
Prothom Alo (Rahman, ) articles dated from 2014-2017 and BDnews24 (Khalidi, )
articles dated from 2015-2017. The Hindi pretraining corpus consists of Hindi
Oscar Corpus (Thakur, ), preprocessed Wikipedia articles (Gaurav, ),
HindiEnCorp05 dataset (Bojar et al., 2014) and WMT Hindi News Crawl data
(Barrault et al., 2019).

We have used Prothom-Alo 2017 online newspaper dataset for Bengali SC training
and validation purpose. Our errors in this corpus have been produced
synthetically using the probabilistic algorithm described by Sifat et al.
(2020). The errors approximate realistic spelling errors, as the algorithm
considers phonetical similarity based confusions, compound letter typing
mistakes, keyboard mispress insertions and deletions. We further validate our
baselines and proposed methods on Hindi open source SC dataset, namely
ToolsForIL (Etoori et al., 2018). This synthetically generated dataset was
used for character level seq2seq Hindi SC development.

For real error dataset, we have collected a total of 6300 sentences from
Nayadiganta (Mohiuddin, ) online newspaper. Then we have distributed the
dataset among ten participants. They have typed (in regular speed) each
correct sentence using English QWERTY keyboard producing natural spelling
errors. It has taken 40 days to finish the labeling, where each of the
participant has labeled less than 20 sentences per day in order to ensure
quality in labeling. As this real error dataset has been used as a test set
after training on synthetic error dataset Prothom-Alo, we have considered its
top words to be the same as Prothom-Alo dataset. Top words have been taken
such that they cover at least 95% of the corresponding corpus.

###  4.2 Training and Validation Details

In case of Bengali spell checker, we randomly initialize the weights of our
BSpell model $M$. We use our large Bengali pretrain corpus for pretraining
utilizing our proposed hybrid masking. So, now we have a pretrained model
$M_{pre}$. Next we split our benchmark synthetic spelling error dataset
(synthetic error induced Prothom-Alo dataset) into 80%-20% training-validation
set. We fine tune $M_{pre}$ using the 80% training portion and obtain fine
tuned model $M_{fine}$. We report Bengali synthetic error spell checking
performance on the remaining 20% validation portion using $M_{fine}$. We have
constructed a real spelling error dataset for Bengali. We use this dataset in
two ways. First, we do not fine tune $M_{fine}$ at all on any real life
Bengali spelling error data and simply test $M_{fine}$ on the entire Bengali
real error dataset. In this case, this dataset works as an independent test
set (result reported with the title real error (no fine tune)). Second, we
split this real error dataset into 80%-20% training-validation and fine tune
$M_{fine}$ further using the 80% portion, then validate on the remaining 20%
(result reported with the title real error (fine tuned)).

In case of Hindi, the first two steps (pretraining and fine tuning) are the
same. We have not constructed any real life spelling error dataset for Hindi.
So, results are reported on the 20% hold out portion of the benchmark dataset.

We use accuracy and F1 score as our performance metric. We only correct top
words which cover at least 95% of the dataset. We do not correct rare UNK
words as most of them are names and proper nouns. Merely detecting rare words
correctly is considered sufficient.

###  4.3 BSpell Architecture Hyperparameters

SemanticNet sub-model of BSpell consists of a character level embedding layer
producing a 40 size vector from each character, then 5 consecutive layers each
consisting of 1D conv (batch normalization and Relu activation in between each
pair of conv layers) and finally, a 1D global max pooling in order to obtain
SemanticVec representation from each input word. The five 1D conv layers
consist of $(64,2),(64,3),(128,3),(128,3),(256,4)$ convolution, respectively.
The first and second element of each tuple denote number of conv filters and
kernel size, respectively. We provide a weight of 0.3 ($\lambda$ value of loss
function) to the auxiliary loss in order to limit the impact of this loss in
gradient update compared to the final loss.

The main branch of BSpell is similar to BERT_Base (Gong et al., 2019) in terms
of stacking 12 Transformer encoders. We pass each SemanticVec through a 768
hidden unit dense layer. The modified output obtained from this layer is then
passed on to the bottom-most Transformer of the stack. Each Transformer
consists of 12 attention heads and 768 hidden unit feed forward NNs. Attention
outputs from each Transformer is passed through a dropout layer (Srivastava et
al., 2014) with a dropout rate of 0.3 and then layer normalized (Ba et al.,
2016).

We use Stochastic Gradient Descent (SGD) Optimizer with a learning rate of
0.001 for our model weight update. We clip our gradient value and keep it
below 5.0 to avoid gradient exploding problem. Since we are dealing with multi
class classification, we use Categorical Cross-Entropy loss function, while at
both output layers (final and auxiliary), we use Softmax activation at
individual word level.

###  4.4 Implemented Pretraining Scheme Details

We have experimented with three types of masking based schemes which are as
follows:
Word Masking: We randomly select 15% words of a sentence and replace those
words with a fixed mask word.
Character Masking: We randomly select 50% words of a sentence. For each
selected word, we randomly mask 30% of its characters by replacing each of
them with our special mask character.
Hybrid Masking: We randomly select 15% words of a sentence and replace them
with a fixed mask word. We randomly select 40% words from the remaining words.
For these selected words, we randomly mask 25% of their characters.

##  5 Results and Discussion

###  5.1 BSpell vs Contemporary BERT Variants

Spell Checker Architecture |  | Synthetic Error
---
(Prothom-Alo)
| Real-Error
---
(No Fine Tune)
| Real-Error
---
(Fine Tuned)
| Synthetic Error
---
(Hindi)
ACC | F1 | ACC | F1 | ACC | F1 | ACC | F1
BERT Seq2seq | 31.6% | 0.305 | 24.5% | 0.224 | 29.3% | 0.278 | 22.8% | 0.209
BERT Base | 91.1% | 0.902 | 83% | 0.823 | 87.6% | 0.855 | 93.8% | 0.923
Soft Masked BERT | 92% | 0.919 | 84.2% | 0.832 | 88.1% | 0.862 | 94% | 0.933
BSpell | 94.7% | 0.934 | 86.1% | 0.859 | 90.1% | 0.898 | 96.2% | 0.96
Table 2: Comparing BERT based variants. Typical word masking based pretraining
has been used on all these variants (Liu et al., 2019). Real-Error (Fine
Tuned) denotes fine tuning of the Bengali synthetic error dataset trained
model on real error dataset, while Real-Error (No Fine Tune) means directly
validating synthetic error dataset trained model on real error dataset without
any further fine tuning.

We start with BERT Seq2seq where the encoder and decoder portion consist of 12
stacked Transformers (Devlin et al., 2018). Predictions are made at character
level. Similar architecture has been used in FASpell (Hong et al., 2019) for
Chinese SC. A word containing $m$ characters is considered wrong if even one
of its characters is predicted incorrectly. Hence character level seq2seq
modeling achieves poor result. Moreover, in most cases during sentence level
SC, the correct spelling of the $i^{th}$ word of input sentence has to be the
$i^{th}$ word in the output sentence as well. Such constraint is difficult to
follow through such architecture design. Hence we get poor result using BERT
Seq2seq operating at character level (see Table 2). Our implemented BERT Base
consisting of stacked Transformer encoders has two differences from the design
proposed by Cheng et al. (2020) \- (i) We make predictions at word level
instead of character level. For character level prediction, character number
of a misspelled sentence and its correct version have to be the same; which is
an uncommon scenario in Bengali (ii) We do not incorporate any external
knowledge about Bengali SC through the use of graph conv network since such
knowledge is not well established in Bengali SC domain. This approach achieves
good performance in all four cases. Soft Masked BERT learns to apply
specialized synthetic masking on error prone words in order to push the error
correction performance of BERT Base further. The error prone words are
detected using a GRU sub-model and the whole architecture is trained end to
end. Although Zhang et al. (2020) implemented this architecture to make
corrections at character level, our implementation does everything in word
level. It is to note that we have used popular FastText (Athiwaratkun et al.,
2018) word representation for both BERT Base and Soft Masked BERT. BSpell
shows decent performance improvement in all cases. SemanticNet sub-model, with
the help of our designed auxiliary loss is able to specialize on word error
pattern.

In Table 2, Bspell and Soft Masked BERT are the closest competitors. So, we
have further performed five fold cross validation on Prothom-Alo spelling
correction dataset based on these two models. We have applied one tailed
independent t-test based on these five accuracy values per model obtained from
five fold cross validation. BSpell performs significantly better than Soft
Masked BERT considering P-value threshold of 0.05.

###  5.2 Comparing BSpell Pretraining Schemes

Pretraining Scheme |  | Synthetic Error
---
(Prothom-Alo)
| Real-Error
---
(No Fine Tune)
| Real-Error
---
(Fine Tuned)
| Synthetic Error
---
(Hindi)
ACC | F1 | ACC | F1 | ACC | F1 | ACC | F1
Word Masking | 94.7% | 0.934 | 86.1% | 0.859 | 90.1% | 0.898 | 96.2% | 0.96
Character Masking | 95.6% | 0.952 | 85.3% | 0.851 | 89.2% | 0.889 | 96.4% | 0.963
Hybrid Masking | 97.6% | 0.971 | 87.8% | 0.873 | 91.5% | 0.911 | 97.2% | 0.97
Table 3: Comparing BSpell exposed to various pretraining schemes

We have implemented three different pretraining schemes (details provided in
Subsection 4.4) on BSpell before fine tuning on SC dataset. Word masking
teaches BSpell context of a language through a fill in the gaps sort of
approach. SC is not all about filling in the gaps. It is also about what the
writer wants to say, i.e. being able to predict a word even if some of its
characters are blank (masked). A writer will seldom write the most appropriate
word according to context, rather he will write what is appropriate to express
his opinion. Character masking takes a more drastic approach by completely
eliminating the fill in the gap task. This approach masks a few of the
characters residing in some of the input words of the sentence and asks BSpell
to predict these noisy words’ original correct version. The lack of context in
such pretraining scheme puts negative effect on performance over real error
dataset experiments, where harsh errors exist and context is the only feasible
way of correcting such errors (see Table 3). Hybrid masking focuses both on
filling in word gaps and on filling in character gaps through prediction of
correct word and helps BSpell achieve SOTA performance. It is to note that the
performance of BSpell reported in Table 2 is based on word masking based
pretraining, not hybrid pretraining; hence the reported performance of BSpell
is lower in that table.

###  5.3 BSpell vs Possible LSTM Variants

Spell Checker Architecture |  | Synthetic Error
---
(Prothom-Alo)
| Real-Error
---
(No Fine Tune)
| Real-Error
---
(Fine Tuned)
| Synthetic Error
---
(Hindi)
ACC | F1 | ACC | F1 | ACC | F1 | ACC | F1
Pre_LSTM | 15.5% | 0.133 | 12.3% | 0.109 | 14.2% | 0.131 | 17.4% | 0.165
Pre_LSTM + Word_CNN | 79.3% | 0.782 | 73.5% | 0.714 | 78.7% | 0.771 | 78.2% | 0.770
| Pre_LSTM + Word_CNN
---
\+ Post_LSTM
82.2% | 0.821 | 78.8% | 0.786 | 81.2% | 0.81 | 81.4% | 0.811
BiLSTM | 81.9% | 0.818 | 78.3% | 0.781 | 81.1% | 0.809 | 81.2% | 0.809
Stacked BiLSTM | 83.5% | 0.832 | 80.1% | 0.80 | 82.4% | 0.822 | 82.7% | 0.824
Attn_Seq2seq (Char) | 20.5% | 0.178 | 15.4% | 0.129 | 17.3% | 0.152 | 22.7% | 0.216
BSpell | 97.6% | 0.971 | 87.8% | 0.873 | 91.5% | 0.911 | 97.2% | 0.97
Table 4: Comparing LSTM based variants with hybrid pretrained BSpell. FastText
word representation has been used with LSTM portion of each architecture.

We start with Pre_LSTM, a generalized form of N-gram based word prediction
(Khan et al., 2014). Given $word_{1},word_{2},\ldots word_{i-1}$ of a
sentence, it predicts $word_{i}$. We assume this predicted word to be the
correct version of the given $word_{i}$. The result is poor on all datasets
(see Table 4). Pre_LSTM + Word_CNN coordinates between two NN branches for
correcting $word_{i}$. 1D conv based Word_CNN branch takes in $word_{i}$
representation matrix as input. Pre_LSTM branch provides a prior context
vector as output, whereas Word_CNN branch provides $word_{i}$ spelling pattern
summary vector as output. Both outputs are merged and passed to a coordinating
NN layer to get the correct version of $word_{i}$. Performance jumps up by a
large margin with the addition of Word_CNN branch as misspelled word spelling
pattern is crucial for SC. Pre_LSTM + Word_CNN + Post_LSTM is similar to the
previous model except for the presence of one extra LSTM based NN branch
Post_LSTM which takes in $word_{n},word_{n-1},\ldots word_{i+1}$ as input,
where $n$ is the number of words in the sentence. It produces a post context
vector for $word_{i}$ as output. The merging and prediction procedure is
similar to Pre_LSTM + Word_CNN. Performance sees another boost after the
addition of Post_LSTM branch, because the correction of a word requires the
word’s post context in some cases. All three of the discussed LSTM based
models correct one word at a time. BiLSTM is a many to many bidirectional LSTM
that takes in all $n$ words of a sentence at once and predicts their correct
version as output ($word_{i}$ is provided as input in time stamp $i$)
(Schuster and Paliwal, 1997). BiLSTM shows almost similar performance as it
also takes in both previous and post context into consideration besides the
writing pattern of each word. We have stacked two LSTM layers in the discussed
four models. However, in Stacked BiLSTM, we stack twelve many to many
bidirectional LSTMs instead of just two. SC performance improves by only
1-1.5% in spite of such large increase in parameter number. Attn_Seq2seq LSTM
model utilizes Bahdanu attention mechanism at decoder side (Bahdanau et al.,
2014). This model takes in misspelled sentence characters as input and
provides the correct sequence of characters as output (Etoori et al., 2018).
Due to word level spelling correction evaluation, this model faces the same
problems as BERT Seq2seq model discussed in Subsection 5.1. Such architecture
shows poor performance in Bengali and Hindi SC. Although Stacked BiLSTM shows
superior performance in LSTM paradigm, hybrid pretrained BSpell outperforms
this model by a large margin.

###  5.4 Ablation Study

BSpell Variants |  | Synthetic Error
---
(Prothom-Alo)
| Real-Error
---
(No Fine Tune)
| Real-Error
---
(Fine Tuned)
| Synthetic Error
---
(Hindi)
ACC | F1 | ACC | F1 | ACC | F1 | ACC | F1
Original BSpell | 97.6% | 0.971 | 87.8% | 0.873 | 91.5% | 0.911 | 97.2% | 0.97
| BSpell
---
(no auxiliary loss)
96.3% | 0.96 | 86.9% | 0.865 | 90.5% | 0.90 | 95.4% | 0.949
| BSpell
---
(no SemanticNet)
94.5% | 0.94 | 85.7% | 0.848 | 89.2% | 0.885 | 95.2% | 0.95
| BSpell (no hybrid
---
pretraining)
94.7% | 0.934 | 86.1% | 0.859 | 90.1% | 0.898 | 96.2% | 0.96
Table 5: Comparison of BSpell with its variants. Each variant has been created
by removing one of the novel features introduced in the original BSpell
architecture.

The original BSpell model proposed in this paper has three unique features -
(1) secondary branch with auxiliary loss, (2) 1D CNN based SemanticNet sub-
model and (3) hybrid pretraining. Table 5 demonstrates the results we obtain
after removing any one of these features. In all cases, the results show a
downward trend compared to the original architecture; which shows the
necessity of all the components. The first three models of Table 5 have been
put through hybrid pretraining, while the last one went through BERT specific
word level masking based pretraining. The secondary branch was removed from
the original BSpell to obtain BSpell (no auxiliary loss) variant. SemanticNet
sub-model based SemanticVec word representation was replaced by Byte Pair
Encoding (BPE) (Vaswani et al., 2017) based word representation to obtain
BSpell (no SemanticNet) variant from the original BSpell model.

###  5.5 Existing Bengali SC vs BSpell

Spell Checker |  | Synthetic Error
---
(Prothom-Alo)
| Real-Error
---
(No Fine Tune)
ACC | F1 | ACC | F1
Phonetic | 61.2% | 0.582 | 43.5% | 0.401
Clustering | 52.3% | 0.501 | 44.2% | 0.412
BSpell | 97.6% | 0.971 | 87.8% | 0.873
Table 6: Existing Bengali SCs vs BSpell

Phonetic rule based SC takes a Bengali phonetic rule based hard coded approach
(Saha et al., 2019), where a hybrid of Soundex (UzZaman and Khan, 2004) and
Metaphone (UzZaman and Khan, 2005) algorithm has been used for spell checking.
Clustering based SC on the other hand follows some predefined rules on word
cluster formation, distance measure and correct word suggestion (Mandal and
Hossain, 2017). Since these two SCs are not learning based, fine tuning is not
applicable for them. They do not take misspelled word context into
consideration while correcting that word. As a result, their performance is
poor especially in Bengali real error dataset (see Table 6). Our proposed
hybrid pretrained BSpell outperforms these Bengali SCs by a wide margin.

###  5.6 Is BSpell Language Specific?

Spelling error correction model BSpell has originally been designed keeping
the unique characteristics of Sanskrit originated languages such as Bengali
and Hindi in mind. Here we see how this model performs on English which is
very different from Bengali in terms of structure. We experiment on an English
spelling error dataset published by Jayanthi et al. (2020). The training set
consists of 1.6 million sentences. The authors created a confusion set
consisting of 109K misspelled-correct word pairs for 17K popular English
words. 20% of the words of the training set have been converted to spelling
error based on this confusion set. The authors created BEA-60K test set from
BEA-2019 shared task consisting of natural English spelling errors. The best
correction rate achieved by the authors was around 80% using LSTM based ELMo
model, whereas BSpell has achieved a correction rate of 86.2%. We have also
experimented with BERT_Base model on this test set where we have used byte
pair encoding as word representation. BERT_Base has achieved an error
correction rate of 85.6%. It is clear that BSpell and BERT_Base do not have
that much difference in performance when it comes to English compared to
Bengali and Hindi. Although BSpell achieves impressive result in English
spelling correction, its design is more suited for Sanskrit originated
languages such as Bengali and Hindi.

###  5.7 Effectiveness of SemanticNet

Figure 6: 10 popular words and three error variations of each of those words
have been plotted based on their SemanticVec representation. Each popular word
along with its three error variations form a separate cluster marked with a
unique color.

The main motivation behind the inclusion of SemanticNet in BSpell is to obtain
vector representations of error words as close as possible to their
corresponding correct words. Such ability imposes correct semantic meaning to
misspelled input words which is effective in cases where there are many error
words in a single sentence. We take 10 frequently occurring Bengali words and
collect three real life error variations of each of these words. We produce
SemanticVec representation of all 40 of these words using SemanticNet
(extracted from trained BSpell model). We use principal component analysis
(PCA) (Shlens, 2014) on each of these SemanticVecs and plot them in two
dimension. Finally, we implement K-Means Clustering algorithm using careful
initialization with $K=10$ (Chen and Xia, 2009). Figure 6 shows the 10
clusters obtained from this algorithm. Each cluster consists of a popular word
and its three error variations. In all cases, the correct word and its three
error versions are so close in the graph plot that they almost form a single
point. Error versions of a particular word are close to that word but are far
from other correct words.

###  5.8 BSpell Limitations

BSpell model provides a word for word correction, i.e., number of input words
and number of output words have to be exactly the same. Unfortunately, during
accidental word merging or word splitting, number of input and output words
differ and so in such cases BSpell will fail in resolving such errors. The
advantage is that this type of error is rare in Bengali and Hindi as the words
of these languages are clearly spaced in sentences. In case of error in rare
words (mostly proper nouns), BSpell simply provides UNK as output and does not
correct errors prevalent in such words which can be marked as another
limitation.

##  6 Conclusion

We have performed detail investigation on the limitations of existing Bengali
SCs as well as other SOTA SCs proposed for high resource languages. We have
proposed BSpell architecture that exploits the concept of hybrid masking based
pretraining. It uses SemanticVec representation of input misspelled words and
a specialized auxiliary loss for the enhancement of SC performance.
Experimental evaluation shows the effectiveness of our proposed method over
existing ones for Bengali and Hindi. The design of BSpell can be followed to
develop SCs for other languages as well. Future study may investigate on
personalized Bengali SC which has the ability to take user personal preference
and writing behaviour into account. Furthermore, grammar correction has gained
a lot of attention in English. State-of-the-art English grammar correction
models can be tested on Bengali and Hindi spell checking as part of future
research effort.

## References

  * Athiwaratkun et al. (2018) Ben Athiwaratkun, Andrew Gordon Wilson, and Anima Anandkumar. 2018.  Probabilistic fasttext for multi-sense word embeddings.  _arXiv preprint arXiv:1806.02901_.
  * Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016.  Layer normalization.  _arXiv preprint arXiv:1607.06450_.
  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_.
  * Barrault et al. (2019) Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019.  Findings of the 2019 conference on machine translation (WMT19).  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pages 1–61, Florence, Italy. Association for Computational Linguistics.
  * Bojar et al. (2014) Ondřej Bojar, Vojtěch Diatka, Pavel Straňák, Aleš Tamchyna, and Daniel Zeman. 2014.  HindEnCorp 0.5.  LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University.
  * Chen and Xia (2009) Zhang Chen and Shixiong Xia. 2009.  K-means clustering algorithm with improved initial center.  In _2009 Second International Workshop on Knowledge Discovery and Data Mining_ , pages 790–792. IEEE.
  * Cheng et al. (2020) Xingyi Cheng, Weidi Xu, Kunlong Chen, Shaohua Jiang, Feng Wang, Taifeng Wang, Wei Chu, and Yuan Qi. 2020.  Spellgcn: Incorporating phonological and visual similarities into language models for chinese spelling check.  _arXiv preprint arXiv:2004.14166_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Etoori et al. (2018) Pravallika Etoori, Manoj Chinnakotla, and Radhika Mamidi. 2018.  Automatic spelling correction for resource-scarce languages using deep learning.  In _Proceedings of ACL 2018, Student Research Workshop_ , pages 146–152, Melbourne, Australia. Association for Computational Linguistics.
  * (10) Gaurav.  Wikipedia.  https://www.kaggle.com/disisbig/hindi-wikipedia-articles-172k.
  * Gong et al. (2019) Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. 2019.  Efficient training of bert by progressively stacking.  In _International Conference on Machine Learning_ , pages 2337–2346. PMLR.
  * Hong et al. (2019) Yuzhong Hong, Xianguo Yu, Neng He, Nan Liu, and Junhui Liu. 2019.  Faspell: A fast, adaptable, simple, powerful chinese spell checker based on dae-decoder paradigm.  In _Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)_ , pages 160–169.
  * Islam et al. (2018) Sadidul Islam, Mst Farhana Sarkar, Towhid Hussain, Md Mehedi Hasan, Dewan Md Farid, and Swakkhar Shatabda. 2018.  Bangla sentence correction using deep neural network based sequence to sequence learning.  In _2018 21st International Conference of Computer and Information Technology (ICCIT)_ , pages 1–6. IEEE.
  * Jayanthi et al. (2020) Sai Muralidhar Jayanthi, Danish Pruthi, and Graham Neubig. 2020.  Neuspell: A neural spelling correction toolkit.  _arXiv preprint arXiv:2010.11085_.
  * (15) Toufique Imrose Khalidi.  Bdnews24.  https://bangla.bdnews24.com/.
  * Khan et al. (2014) Nur Hossain Khan, Gonesh Chandra Saha, Bappa Sarker, and Md Habibur Rahman. 2014\.  Checking the correctness of bangla words using n-gram.  _International Journal of Computer Application_ , 89(11).
  * Kim et al. (2016) Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016.  Character-aware neural language models.  In _Thirtieth AAAI conference on artificial intelligence_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Mandal and Hossain (2017) Prianka Mandal and BM Mainul Hossain. 2017.  Clustering-based bangla spell checker.  In _2017 IEEE International Conference on Imaging, Vision & Pattern Recognition (icIVPR)_, pages 1–6. IEEE.
  * (20) Alamgir Mohiuddin.  Nayadiganta.  https://www.dailynayadiganta.com/.
  * Noyes (1983) Jan Noyes. 1983.  The qwerty keyboard: A review.  _International Journal of Man-Machine Studies_ , 18(3):265–281.
  * (22) Matiur Rahman.  Prothom-alo.  https://www.prothomalo.com/.
  * Saha et al. (2019) Sourav Saha, Faria Tabassum, Kowshik Saha, and Marjana Akter. 2019.  _BANGLA SPELL CHECKER AND SUGGESTION GENERATOR_.  Ph.D. thesis, United International University.
  * Schuster and Paliwal (1997) Mike Schuster and Kuldip K Paliwal. 1997.  Bidirectional recurrent neural networks.  _IEEE transactions on Signal Processing_ , 45(11):2673–2681.
  * Shlens (2014) Jonathon Shlens. 2014.  A tutorial on principal component analysis.  _arXiv preprint arXiv:1404.1100_.
  * Sifat et al. (2020) Md Habibur Rahman Sifat, Chowdhury Rafeed Rahman, Mohammad Rafsan, and Hasibur Rahman. 2020.  Synthetic error dataset generation mimicking bengali writing pattern.  In _2020 IEEE Region 10 Symposium (TENSYMP)_ , pages 1363–1366. IEEE.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 15(1):1929–1958.
  * Sun et al. (2020) Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020.  Ernie 2.0: A continual pre-training framework for language understanding.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 8968–8975.
  * Szegedy et al. (2015) Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015.  Going deeper with convolutions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1–9.
  * Takase et al. (2019) Sho Takase, Jun Suzuki, and Masaaki Nagata. 2019.  Character n-gram embeddings to improve rnn language models.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 5074–5082.
  * (31) Abhishek Thakur.  Hindi oscar corpus.  https://www.kaggle.com/abhishek/hindi-oscar-corpus.
  * UzZaman and Khan (2004) Naushad UzZaman and Mumit Khan. 2004.  A bangla phonetic encoding for better spelling suggesions.  Technical report, BRAC University.
  * UzZaman and Khan (2005) Naushad UzZaman and Mumit Khan. 2005.  A double metaphone encoding for approximate name searching and matching in bangla.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Wang et al. (2019) Dingmin Wang, Yi Tay, and Li Zhong. 2019.  Confusionset-guided pointer networks for Chinese spelling check.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5780–5785, Florence, Italy. Association for Computational Linguistics.
  * Xiong et al. (2015) Jinhua Xiong, Qiao Zhang, Shuiyuan Zhang, Jianpeng Hou, and Xueqi Cheng. 2015.  Hanspeller: a unified framework for chinese spelling correction.  In _International Journal of Computational Linguistics & Chinese Language Processing, Volume 20, Number 1, June 2015-Special Issue on Chinese as a Foreign Language_.
  * Zhang et al. (2020) Shaohua Zhang, Haoran Huang, Jicong Liu, and Hang Li. 2020.  Spelling error correction with soft-masked bert.  _arXiv preprint arXiv:2005.07421_.