# Pay Attention to What You Read:
Non-recurrent Handwritten Text-Line Recognition

Lei Kang∗†, Pau Riba∗, Marçal Rusiñol∗, Alicia Fornés∗, Mauricio Villegas†
∗Computer Vision Center, Universitat Autònoma de Barcelona, Barcelona, Spain
{lkang, priba, marcal<EMAIL_ADDRESS>
†omni:us, Berlin, Germany
{lei<EMAIL_ADDRESS>
###### Abstract

The advent of recurrent neural networks for handwriting recognition marked an
important milestone reaching impressive recognition accuracies despite the
great variability that we observe across different writing styles. Sequential
architectures are a perfect fit to model text lines, not only because of the
inherent temporal aspect of text, but also to learn probability distributions
over sequences of characters and words. However, using such recurrent
paradigms comes at a cost at training stage, since their sequential pipelines
prevent parallelization. In this work, we introduce a non-recurrent approach
to recognize handwritten text by the use of transformer models. We propose a
novel method that bypasses any recurrence. By using multi-head self-attention
layers both at the visual and textual stages, we are able to tackle character
recognition as well as to learn language-related dependencies of the character
sequences to be decoded. Our model is unconstrained to any predefined
vocabulary, being able to recognize out-of-vocabulary words, _i.e._ words that
do not appear in the training vocabulary. We significantly advance over prior
art and demonstrate that satisfactory recognition accuracies are yielded even
in few-shot learning scenarios.

Figure 1: Handwriting text-line recognition with the proposed transformer
architecture. The attention mechanism focus at different locations to decode
character by character.

##  1 Introduction

Handwritten Text Recognition (HTR) frameworks aim to provide machines with the
ability to read and understand human calligraphy. From the applications
perspective, HTR is relevant both to digitize the textual contents from
ancient document images in historic archives as well as contemporary
administrative documentation such as cheques, forms, etc. Even though research
in HTR began in the early sixties [36], it is still considered as an unsolved
problem. The main challenge is the huge variability and ambiguity of the
strokes composing words encountered across different writers. Fortunately, in
most cases, the words to decipher do follow a well defined set of language
rules that should be also modelled and taken into account in order to discard
gibberish hypotheses and yield higher recognition accuracies. As a result, HTR
is often approached by combining technologies from both computer vision and
natural language processing communities.

Handwritten text is a sequential signal in nature. Texts are written from left
to right in Latin languages, and words are formed by an ordered sequence of
characters. Thus, HTR approaches usually adopted temporal pattern recognition
techniques to address it. The early approaches based on Hidden Markov Models
(HMM) [5] evolved towards the use of Deep Learning techniques, in which
Bidirectional Long Short-Term Memory (BLSTM) networks [22] became the standard
solution. Recently, inspired by their success in the applications such as
automatic translation or speech-to-text, Sequence-to-Sequence (Seq2Seq)
approaches, conformed by encoder-decoder networks led by attention mechanisms
have started to be applied for HTR [37]. All the above methods are not only a
good fit to process images sequentially, but also have, in principle, the
inherent power of language modelling, _i.e._ to learn which character is more
probable to be found after another in their respective decoding steps.
Nonetheless, this ability of language modelling has proven to be limited,
since recognition performances are in most cases still enhanced when using a
separate statistical language model as a post-processing step [46].

Despite the fact that attention-based encoder-decoder architectures have
started to be used for HTR with impressive results, one major drawback still
remains. In all of those cases, such attention mechanisms are still used in
conjunction with a recurrent network, either BLSTMs or Gated Recurrent Unit
(GRU) networks. The use of such sequential processing deters parallelization
at training stage, and severely affects the effectiveness when processing
longer sequence lengths by imposing substantial memory limitations.

Motivated by the above observations, Vaswani _et al._ proposed in [47] the
seminal work on the Transformer architecture. Transformers rely entirely on
attention mechanisms, relinquishing any recurrent designs. Stimulated by such
advantage, we propose to address the HTR problem by an architecture inspired
on transformers, which dispenses of any recurrent network. By using multi-head
self-attention layers both at the visual and textual stages, we aim to tackle
both the proper step of character recognition from images, as well as to learn
language-related dependencies of the character sequences to be decoded.

The use of transformers in different language and vision applications have
shown higher performances than recurrent networks while having the edge over
BLSTMs or GRUs by being more parallelizable and thus involving reduced
training times. Our method is, to the best of our knowledge, the first non-
recurrent approach for HTR. Moreover, the proposed transformer approach is
designed to work at character level, instead at the commonly used word level
in translation or speech recognition applications. By using such design we are
not restricted to any predefined fixed vocabulary, so we are able to recognize
out-of-vocabulary (OOV) words, _i.e._ never seen during training. Competitive
state-of-the-art results on the public IAM dataset are reached even when using
a small portion of training data.

The main contributions of our work are summarized as follows. i) For the first
time, we explore the use of transformers for the HTR task, bypassing any
recurrent architecture. We attempt to learn, with a single unified
architecture, to recognize character sequences from images as well as to model
language, providing context to distinguish between characters or words that
might look similar. The proposed architecture works at character level,
waiving the use of predefined lexicons. ii) By using a pre-training step using
synthetic data, the proposed approach is able to yield competitive results
with a limited amount of real annotated training data. iii) Extensive ablation
and comparative experiments are conducted in order to validate the
effectiveness of our approach. Our proposed HTR system achieves new state-of-
the-art performance on the public IAM dataset.

##  2 Related Work

The recognition of handwritten text has been commonly approached by the use of
sequential pattern recognition techniques. Text lines are processed along a
temporal sequence by learning models that leverage their sequence of internal
states as memory cells, in order to be able to tackle variable length input
signals. Whether we analyze the former approaches based on HMMs [5, 19, 21] or
the architectures based on deep neural networks such as BLSTMs [22],
Multidimensional LSTMs [23, 41] (MDLSTM) or encoder-decoder networks [7, 28,
44, 11, 37], they all follow the same paradigm. Although all those approaches
use recurrent architectures to properly conceal and learn serial information,
visually, but also from the language modelling perspective, they all suffer of
the lack of parallelization during the training stage. Moreover, in order to
efficiently train deep learning based approaches, a huge amount of labeled
training data is required. Some approaches like [4, 25, 32] alleviate the cost
and effort of collecting such amount of real annotated training data by using
synthetically generated cursive data with electronic true-type fonts. Which,
in turn, having unlimited annotated data for free and training models that are
less prone to overfit to a set of specific writing styles, exaggerate even
more the computational costs during the training process.

Vaswani _et al._ presented in [47] the Transformer architecture. Their
proposal relies entirely on the use of attention mechanisms, avoiding any
recurrent steps. Since the original publication, the use of transformers has
been popularized in many different computer vision and natural language
processing tasks such as automatic translation [12] or speech-to-text
applications [15]. Its use has started to eclipse recurrent architectures such
as BLSTMs or GRUs for such tasks, both by being more parallelizable,
facilitating training, and by having the ability to learn powerful language
modelling rules of the symbol sequences to be decoded.

However, to the best of our knowledge, the transformer architecture has not
yet been used to tackle the handwriting recognition problem. It has been
nonetheless used lately to recognize text in natural scenes [34]. In such
works, the original transformers architecture, often applied to one-
dimensional signals (_i.e._ text, speech, etc.), has been adapted to tackle
two-dimensional input images. Image features are extracted by the use of CNNs
[42], two-dimensional positional encodings [33, 6] or additional segmentation
modules [2] help the system locate textual information amidst background
clutter. However, all such works present some limitations when dealing with
handwritten text lines. On the one hand, all such architectures work with
fixed image size whereas for handwriting recognition we have to face variable
length inputs. On the other hand, they work at individual word level, whereas
in handwriting recognition we have to face much longer sequences. Finally,
despite also having its own great variability, scene text is often much
legible than cursive handwriting, since in most of the cases words are formed
by individual block letters, which, in turn, are easier to synthesize to
obtain large training volumes.

Summarizing, state-of-the-art handwriting recognition based on deep recurrent
networks have started to reach decent recognition results, but are too
computationally demanding at training stage. Moreover, albeit they shall have
the ability to model language-specific dependencies, they usually fall short
of inferring adequate language models and need further post-processing steps.
In this paper we propose, for the first time, the use of transformers for the
HTR task, bypassing any recurrent architecture. A single unified architecture,
both recognizes long character sequences from images as well as models
language at character level, waiving the use of predefined lexicons.

##  3 Proposed Method

Figure 2: Overview of the architecture of the proposed method. Text-line
images are processed by a CNN feature extractor followed by a set of multi-
headed self-attention layers. During training, the character-wise embeddings
of the transcriptions are also encoded by self-attentions and a final mutual
attention module aims to align both sources of information to decode the text-
line character by character.

###  3.1 Problem Formulation

Let $\\{\mathcal{X},\mathcal{Y}\\}$ be a handwritten text dataset, containing
images $\mathcal{X}$ of handwritten textlines, and their corresponding
transcription strings $\mathcal{Y}$. The alphabet defining all the possible
characters of $\mathcal{Y}$ (letters, digits, punctuation signs, white spaces,
etc.), is denoted as $\mathcal{A}$. Given pairs of images
$x_{i}\in\mathcal{X}$ and their corresponding strings $y_{i}\in\mathcal{Y}$,
the proposed recognizer has the ability to combine both sources of
information, learning both to interpret visual information and to model
language-specific rules.

The proposed method’s architecture is shown in Figure 2. It consists of two
main parts. On the one hand a visual feature encoder aimed at extracting the
relevant features from text-line images and at focusing its attention at the
different character locations. Subsequently, the text transcriber is devoted
to output the decoded characters by mutually attending both at the visual
features as well as the language-related features. The whole system is trained
in an end-to-end fashion, learning both to decipher handwritten images as well
as modelling language.

###  3.2 Visual Feature Encoder

The role of the visual feature encoder is to extract high-level feature
representations from an input handwritten image $x\in\mathcal{X}$. It will
encode both visual content as well as sequential order information. This
module is composed by the following three parts.

####  3.2.1 CNN Feature Encoder

Input images $x$ of handwritten text-lines, which might have arbitrary
lengths, are first processed by a Convolutional Neural Network. We obtain an
intermediate visual feature representation $F_{c}$ of size $f$. We use the
ResNet50 [26] as our backbone convolutional architecture. Such visual feature
representation has a contextualized global view of the whole input image while
remaining compact.

####  3.2.2 Temporal Encoding

Handwritten text images are sequential signals in nature, to be read in order
from left to right in Latin scripts. The temporal encoding steps are aimed to
leverage and encode such important information bypassing any recurrency.

In a first step, the three-dimensional feature $F_{c}$ is reshaped into a two-
dimensional feature by keeping its width, _i.e._ obtaining a feature shape
$(f\times h,w)$. This feature map is later fed into a fully connected layer in
order to reduce $f\times h$ back to $f$. The obtained feature
$F_{c}^{{}^{\prime}}$, with the shape of $(f,w)$, can be seen as a $w$-length
sequence of visual vectors.

However, we desire that the same character appearing at different positions of
the image has different feature representations, so that the attention
mechanisms are effectively and unequivocally guided. That is, we want that the
visual vectors $F_{c}^{{}^{\prime}}$ loose their horizontal shift invariance.
Following the proposal from Vaswani _et al._ [47], a one-dimensional
positional encoding using sine and cosine functions is applied.

$\displaystyle TE(pos,2i)$ $\displaystyle=$
$\displaystyle\sin{\left(\frac{pos}{10000^{2i/f}}\right)}$ $\displaystyle
TE(pos,2i+1)$ $\displaystyle=$
$\displaystyle\cos{\left(\frac{pos}{10000^{2i/f}}\right)},$ (1)

where $pos\in\\{0,1,2,\ldots,w-1\\}$ and $i\in\\{0,1,2,\ldots,f-1\\}$.

$F_{c}^{{}^{\prime}}$ and $TE$, sharing the same shape are added along the
width axis. A final fully connected layer produces an abscissa-sensitive
visual feature $\bar{F}_{c}$ with shape $(f,w)$.

####  3.2.3 Visual Self-Attention Module

To further distill the visual features, self-attention modules are applied
four times upon $\bar{F}_{c}$. The multi-head attention mechanism from [47] is
applied using eight heads. This attention module takes three inputs, namely
the query $Q_{c}$, key $K_{c}$ and value $V_{c}$, where
$Q_{c}=K_{c}=V_{c}=\bar{F}_{c}$. The correlation information is obtained by:

$\hat{v}_{c}^{i}=\operatorname{Softmax}\left(\frac{q_{c}^{i}K_{c}}{\sqrt{f}}\right)V_{c},$
(2)

where $q_{c}^{i}\in Q_{c}$ and $i\in\\{0,1,\ldots,w-1\\}$. The final high-
level visual feature is
$\hat{F}_{c}=\\{\hat{v}_{c}^{0},\hat{v}_{c}^{1},\ldots,\hat{v}_{c}^{w-1}\\}$.

###  3.3 Text Transcriber

The text transcriber is the second part of the proposed method. It is in
charge of outputting the decoded characters, attending to both the visual
features as well as the language-specific knowledge learnt form the textual
features. It is worth to note that unlike translation of speech-to-text
transformer architectures, our text transcriber works at character level
instead of word-level. It will thus learn $n$-gram like knowledge from the
transcriptions, _i.e._ predicting the next most probable character after a
sequence of decoded characters. The text transcriber consists of three steps,
the text encoding, the language self-attention step and the mutual-attention
module.

####  3.3.1 Text Encoding

Besides the different characters considered in alphabet $\mathcal{A}$, we
require some symbols without textual content for the correct processing of the
text-line string. Special character $\langle S\rangle$ denotes the start of
the sequence, $\langle E\rangle$ the end of the sequence, and $\langle
P\rangle$ is used for padding. The transcriptions $y\in\mathcal{Y}$ are
extended to a maximum length of $N$ characters in the prediction.

A character-level embedding is performed by means of a fully-connected layer
that maps each character from the input string to an $f$-dimensional vector.
The same temporal encoding introduced in eq. 1 is used here to obtain

$F_{t}=\operatorname{Embedding}\left(y\right)+TE,$ (3)

where $F_{t}$ has the shape of $(f,N)$.

In the decoding step of recurrent-based HTR approaches [28, 37] every decoded
character is iteratively fed again to the decoder, to predict the next
character, thus inhibiting its parallelization. Contrary, in the transformer
paradigm, all possible decoding steps are fed concurrently at once with a
masking operation [47]. To decode the $j$-th character from $y$, all
characters at positions greater than $j$ are masked so that the decoding only
depends on predictions produced prior to $j$. Such a parallel processing of
what used to be different time steps in recurrent approaches drastically
reduces training time.

####  3.3.2 Language Self-attention Module

This module follows the same architecture as in Section 3.2.3 and aims to
further distill the text information and learn language-specific properties.
$\hat{F}_{t}$ is obtained after the self-attention module implicitly delivers
$n$-gram-like features, since to decode the $j$-th character from $y$ all
character features prior to $j$ are visible.

---
a) Real data from IAM dataset
b) Synthetically rendered text-lines with truetype fonts
Figure 3: Examples of real and synthetic training handwritten text-line
images.

####  3.3.3 Mutual-attention Module

A final mutual self-attention step is devoted to align and combine the learned
features form the images as well as from the text strings. We follow again the
same architecture from Section 3.2.3, but now the query $Q_{t}$ comes from the
textual representation $\hat{F}_{t}$ while the key $K_{c}$ and value $V_{c}$
are fed with the visual representations $\hat{F}_{c}$

$\hat{v}_{ct}^{i}=\operatorname{Softmax}\left(\frac{q_{t}^{j}K_{c}}{\sqrt{f}}\right)V_{c},$
(4)

where $q_{t}^{j}\in Q_{t}$ and $j\in\\{0,1,\ldots,N-1\\}$. The final combined
representation is
$\hat{F}_{ct}=\\{\hat{v}_{ct}^{0},\hat{v}_{ct}^{1},\ldots,\hat{v}_{ct}^{N-1}\\}$.

The output $\hat{F}_{ct}$ is expected to be aligned with the transcription
$Y$. Thus, by feeding the $\hat{F}_{ct}$ into a linear module followed by a
softmax activation function, the final prediction is obtained.

###  3.4 Inference on Test Data

When evaluating on test data, the transcriptions $\mathcal{Y}$ are not
available. The text pipeline is initialized by feeding the start indicator
$\langle S\rangle$ and it predicts the first character by attending the
related visual part on the input handwritten text image. With the strategy of
greedy decoding, this first predicted character is fed back to the system,
which outputs the second predicted character. This inference process is
repeated in a loop until the end of sequence symbol $\langle E\rangle$ is
produced or when the maximum output length $N$ is reached.

##  4 Experimental Evaluation

###  4.1 Dataset and Performance Measures

We conduct our experiments on the popular IAM handwritten dataset [35],
composed of modern handwritten English texts. We use the RWTH partition, which
consists of $6482$, $976$ and $2914$ lines for training, validation and test,
respectively. The size of alphabet $|\mathcal{A}|$ is $83$, including special
symbols, and the maximum length of the output character sequence is set to
$89$. All the handwritten text images are resized to the same height of $64$
pixels while keeping the aspect ratio, which means that the textline images
have variable length. To pack images into mini-batches, we pad all the images
to the width of $2227$ pixels with blank pixels.

_Character Error Rate_ (CER) and _Word Error Rate_ (WER) [20] are used for the
performance measures. The CER is computed as the Levenshtein distance which is
the sum of the character substitutions ($S_{c}$), insertions ($I_{c}$) and
deletions ($D_{c}$) that are needed to transform one string into the other,
divided by the total number of characters in the groundtruth ($N_{c}$).
Formally,

$CER=\frac{S_{c}+I_{c}+D_{c}}{N_{c}}$ (5)

Similarly, the WER is computed as the sum of the word substitutions ($S_{w}$),
insertions ($I_{w}$) and deletions ($D_{w}$) that are required to transform
one string into the other, divided by the total number of words in the
groundtruth ($N_{w}$). Formally,

$WER=\frac{S_{w}+I_{w}+D_{w}}{N_{w}}$ (6)

###  4.2 Implementation Details

####  4.2.1 Hyper-Parameters of Networks

In the proposed architecture, the feature size $f$ is $1024$. We use four
blocks of visual and language self-attention modules, and each self-attention
module has eight heads. We use $0.1$ dropout setting for every dropout layer.
In the text transcriber, all the transcriptions include the extended special
symbols $\langle S\rangle$ and $\langle E\rangle$ at the beginning and at the
end, respectively. Then, they are padded to $89$ length with a special symbol
$\langle P\rangle$ to the right, which is the maximum number of characters in
the prediction $N$. The output size of the softmax is $83$, which is the size
of the alphabet $\mathcal{A}$, including upper/lower cased letters,
punctuation marks, blank space and special symbols.

####  4.2.2 Optimization Strategy

We adopt label smoothing mechanism [45] to prevent the system from making
over-confident predictions, which is also a way of regularization. As the
ground-truth are one-hot vectors with binary values, label smoothing is done
by replacing the $0$ and $1$ with $\dfrac{\varepsilon}{|\mathcal{A}|}$ and
$1-\dfrac{|\mathcal{A}|-1}{|\mathcal{A}|}\varepsilon$, where $\varepsilon$ is
set to $0.4$ in this paper. We utilize Adam optimizer [29] for the training
process with an initial learning rate of $2\cdot 10^{-4}$, while reducing the
learning rate by half every 20 epochs. The implementation of this system is
based on PyTorch [39] and performed on a NVIDIA Cluster. The code will be
publicly available.

###  4.3 Pre-training with Synthetic Data

Deep learning based methods need a large amount of labelled training data to
obtain a well generalized model. Thus, synthetic data is widely used to
compensate the scarcity of training data in the public datasets. There are
some popular synthetically generated handwriting datasets available [31, 27],
but they are at word level. For this reason we have created our own synthetic
data at line level for pre-training. First, we collect a text corpus in
English from online e-books and end up with over $130,000$ lines of text.
Second, we select 387 freely available electronic cursive fonts and use them
to randomly render text lines from the first step. Finally, by applying a set
of random augmentation techniques (blurring/sharpening, elastic transforming,
shearing, rotating, translating, scaling, gamma correcting and blending with
synthetic background textures), we obtain a synthetic dataset with $138,000$
lines. The comparison between the synthetic data and the real data is shown in
Figure 3.

###  4.4 Ablation Studies

In the ablation studies, all the experiments are trained from scratch with the
IAM training set at line-level, and then early-stopped by the CER of the
validation set, which is also utilized as an indicator to choose the hyper-
parameters as shown in Table 1 2 4.

####  4.4.1 Architecture of CNN Feature Encoder

We have explored different popular Convolutional Neural Networks for the
feature encoder detailed in Section 3.2.1. The best results were obtained with
ResNet models. We modified the original ResNet architecture to slighty
increase the final resolution of the features, by changing the stride
parameter from 2 to 1 in the last convolutional layer. From Table 1, the best
performance is achieved with a modified version of ResNet50.

Table 1: Ablation study on Convolutional architectures. ∗ indicates modified architectures. CNN | CER (%) | WER (%)
---|---|---
ResNet34 | 6.33 | 22.63
ResNet34∗ | 5.44 | 20.13
ResNet50 | 5.49 | 20.93
ResNet50∗ | 4.86 | 18.65

####  4.4.2 Function of Temporal Encoding

In both the visual feature encoder and the text transcriber, we have used
temporal encoding in order to enforce an order information to both visual and
textual features. Nonetheless we want to analyze its impact. In Table 2, it is
clear that using temporal encoding at text level boosts the performance
drastically from $7.72\%$ to $4.86\%$, and from $6.33\%$ to $5.52\%$,
depending on whether we use it at image level or not. The best performance is
reached when using the temporal encoding step both for image and text
representations.

Table 2: Ablation study on the use of temporal encoding in image and text levels. Image level | Text level | CER (%) | WER (%)
---|---|---|---
$-$ | $-$ | 6.33 | 21.64
✓ | $-$ | 7.72 | 24.70
$-$ | ✓ | 5.52 | 20.72
✓ | ✓ | 4.86 | 18.65
Table 3: Fine-tuning with different portions of real data (line-level test set with greedy decoding). | 20% |  | 40% |  | 60% |  | 80% |  | 100%
---|---|---|---|---|---|---|---|---|---
| CER | WER |  | CER | WER |  | CER | WER |  | CER | WER |  | CER | WER
Seq2Seq | 20.61 | 56.50 |  | 16.15 | 46.97 |  | 15.61 | 46.01 |  | 12.18 | 38.11 |  | 11.91 | 37.39
\+ Synth | 18.64 | 51.77 |  | 13.01 | 39.72 |  | 13.00 | 39.34 |  | 12.15 | 37.43 |  | 10.64 | 33.64
Ours | 73.81 | 132.74 |  | 17.34 | 42.57 |  | 10.14 | 30.34 |  | 10.11 | 29.90 |  | 7.62 | 24.54
\+ Synth | 6.51 | 20.53 |  | 6.20 | 19.69 |  | 5.54 | 17.71 |  | 4.90 | 16.44 |  | 4.67 | 15.45

####  4.4.3 Role of Self-Attention Modules

Self-attention modules have been applied in both image and text levels. In
Table 4 we analyze their effect in our system. We observe that the visual
self-attention module barely improves the performance. Nonetheless, for the
language self-attention module, it really plays an important role that
improves the performance from $7.71\%$ to $4.86\%$, and from $7.78\%$ to
$4.89\%$, with and without the visual self-attention module, respectively. Our
intuition is that the language self-attention module actually does learn
language-modelling information. This implicitly learned language model is at
character level and takes advantage of the contextual information of the whole
text-line, which not only boosts the recognition performance but also keep the
capability to predict out-of-vocabulary (OOV) words.

Table 4: Ablation study on visual and language self-attention modules. Image level | Text level | CER (%) | WER (%)
---|---|---|---
$-$ | $-$ | 7.78 | 29.78
✓ | $-$ | 7.71 | 28.50
$-$ | ✓ | 4.89 | 18.57
✓ | ✓ | 4.86 | 18.65
|
---|---
|
Figure 4: Performance of the transformer-based decodings for different amounts
of real training data.

We showcase in Figure 1 and Figure 5 some qualitative results on text-line
recognition, where we visualize the attention maps as well. The attention maps
are obtained by averaging the mini attention maps across different layers and
different heads. Those visualizations prove the successful alignment between
decoded characters and images.

---
Figure 5: Qualitative results on text-line recognition and visualization of
attention maps that coarsely align transcriptions and corresponding image
characters.

###  4.5 Detailed Comparison with Seq2Seq Model

In order to provide a fair comparison between the proposed architecture and
recurrent-based solutions, we re-implemented a state-of-the-art recurrent
handwriting recognition pipeline, and we train and evaluate those under the
exact same circumstances. Following the methods proposed in [28, 37] we built
a sequence-to-sequence recognizer composed of an encoder, a decoder and an
attention mechanism. The encoder consists of a VGG19-BN [43] and a two-layer
Bidirectional Gated Recurrent Units (BGRU) with feature size of $512$. The
decoder is a two-layer one directional GRU with feature size of $512$, and we
power the architecture with a location-based attention mechanism [10]. All the
dropout layers are set to $0.5$. Label smoothing technique is also used during
the training process. The maximum number of predicted characters is also set
to $89$. All the hyper-parameters in this sequence-to-sequence model are also
exhaustively validated by ablation studies with validation data.

We first provide in Table 5, the CER and WER rates on the IAM test set both
when training the networks from scratch and just using the IAM training data,
and when pre-training the networks with synthetic data for a later fine-tuning
step on real data. We also provide the model size and the time taken per epoch
during training. While the sequence-to-sequence model has much less
parameters, it still takes longer to train than the transformers-based one. We
also observe that both models benefit from the use of synthetic pre-training,
improving the final error rates quite noticeably for the transformers model,
although such boost is not so drastic for the sequence-to-sequence approach.

Table 5: Comparison between Recurrent and Transformers. Method | CER (%) | WER (%) | Time(s) | Param(M)
---|---|---|---|---
Seq2Seq | 11.91 | 37.39 | 338.7 | 37
\+ Synth | 10.64 | 33.64 | 338.7 | 37
Ours | 7.62 | 24.54 | 202.5 | 100
\+ Synth | 4.67 | 15.45 | 202.5 | 100

###  4.6 Few-shot Training

Due to the scarcity and the cost of producing large volumes of real annotated
data, we provide an analysis on the performance of the proposed approach when
dealing with a few-shot training setup, when compared again with the sequence-
to-sequence approach. To mimic a real scenario in which only a small portion
of real data is available, we randomly selected $20\%$, $40\%$, $60\%$ and
$80\%$ of the IAM training set.

As shown in Table 3, both sequence-to-sequence and transformer-based
approaches follow the same trend. The more real training data is available,
the better the performance is. Overall, the transformer-based method performs
better than the sequence-to-sequence, except for the extreme case of just
having a $20\%$ of real annotated training data available. The transfomer
approach, being a much larger model, struggles at such drastic data scarcity
conditions. However, when considering the models that have been pre-trained
with synthetic data, the transformer-based approach excels in few-shot setting
conditions. We provide in Figure 4 some qualitative examples of the
transcriptions provided by different models trained with reduced training
sets. All of the models were pre-trained with synthetic data.

###  4.7 Language Modelling Abilities

In order to validate whether the proposed approach indeed is able to model
language-specific knowledge besides its ability to decode handwritten
characters, we propose to test whether using a state-of-the-art language model
as a post-processing step actually improves the performance. We implement a
shallow fusion [24] language model, consisting of a recurrent network with
$2,400$ LSTM units. It has been trained on $130,000$ English text-lines. The
additive weight for the shallow fusion is set to $0.2$.

We observe in Table 6, that the use of such language modelling post-processing
is useless, somehow indicating that the proposed approach already incorporates
such language-specific contextual information within the language self-
attention module.

Table 6: Effect of using a post-processing langauge model. Method | CER (%) | WER (%)
---|---|---
Ours | 4.67 | 15.45
+LM | 4.66 | 15.47
Table 7: Comparison with the State-Of-The-Art approaches on IAM line level dataset. System | Method |  $\Omega$ (k) | CER (%) | WER (%)
---|---|---|---|---
HMM/ANN 2008 - now | Almazán  _et al._ [1] | $-$ | 11.27 | 20.01
España  _et al._ [19] | $-$ | 9.80 | 22.40
Dreuw  _et al._ [16] | 50 | 12.40 | 32.90
Bertolami  _et al._ [3] | 20 | $-$ | 32.83
Dreuw  _et al._ [17] | 50 | 10.30 | 29.20
Zamora  _et al._ [49] | 103 | 7.60 | 16.10
Pastor  _et al._ [38] | 103 | 7.50 | 19.00
España  _et al._ [19] | 5 | 6.90 | 15.50
Kozielski  _et al._ [14] | 50 | 5.10 | 13.30
Doetsch  _et al._ [13] | 50 | 4.70 | 12.20
RNN+CTC 2008 - now | Chen  _et al._ [9] | $-$ | 11.15 | 34.55
Pham  _et al._ [40] | $-$ | 10.80 | 35.10
Krishnan _et al._ [30] | $-$ | 9.78 | 32.89
Wigington  _et al._ [48] | $-$ | 6.40 | 23.20
Puigcerver [41] | $-$ | 5.80 | 18.40
Dutta  _et al._ [18] | $-$ | 5.70 | 17.82
Graves  _et al._ [22] | 20 | 18.20 | 25.90
Pham  _et al._ [40] | 50 | 5.10 | 13.60
Puigcerver [41] | 50 | 4.40 | 12.20
Bluche  _et al._ [8] | 50 | 3.20 | 10.50
Seq2Seq 2016 - now | Chowdhury [11] | $-$ | 8.10 | 16.70
Bluche [7] | $-$ | 7.90 | 24.60
Bluche [7] | 50 | 5.50 | 16.40
Transf. | Ours | $-$ | 4.67 | 15.45

###  4.8 Comparison with the State-Of-The-Art

Finally, we provide in Table 7 and extensive performance comparison with the
state of the art. Different approaches have been grouped into a taxonomy
depending on whether they are based on HMMs or early neural network
architectures, whether they use recurrent neural networks (usually different
flavours of LSTMs) followed by a Connectionist Temporal Classification (CTC)
layer, or if they are based on encoder-decoder sequence-to-sequence
architectures. Within each group, we differentiate results depending on
whether they make use of a closed vocabulary of size $\Omega$ or they are able
to decode OOV words. Bluche  _et al._ [8] achieves the best result among the
methods using a closed lexicon, while our proposed method obtains the best
result among the methods without using a closed lexicon, while still competing
with most of the closed-vocabulary approaches.

##  5 Conclusion

In this paper, we have proposed a novel non-recurrent and open-vocabulary
method for handwritten text-line recognition. As far as we know, it is the
first approach that adopts the transformer networks for the HTR task. We have
performed a detailed analysis and evaluation on each module, demonstrating the
suitability of the proposed approach. Indeed, the presented results prove that
our method not only achieves the state-of-the-art performance, but also has
the capability to deal with few-shot training scenarios, which further extends
its applicability to real industrial use cases. Finally, since the proposed
approach is designed to work at character level, we are not constrained to any
closed-vocabulary setting, and transformers shine at combining visual and
language-specific learned knowledge.

## References

  * [1] J. Almazán, A. Gordo, A. Fornés, and E. Valveny.  Word spotting and recognition with embedded attributes.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(12):2552–2566, 2014.
  * [2] C. Bartz, J. Bethge, H. Yang, and C. Meinel.  KISS: Keeping it simple for scene text recognition.  arXiv preprint arXiv:1911.08400, 2019.
  * [3] R. Bertolami and H. Bunke.  Hidden Markov model-based ensemble methods for offline handwritten text line recognition.  Pattern Recognition, 41(11):3452–3460, 2008.
  * [4] A. K. Bhunia, A. Das, A. K. Bhunia, P. S. R. Kishore, and P. P. Roy.  Handwriting recognition in low-resource scripts using adversarial learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  * [5] A.-L. Bianne-Bernard, F. Menasri, R. A.-H. Mohamad, C. Mokbel, C. Kermorvant, and L. Likforman-Sulem.  Dynamic and contextual information in HMM modeling for handwritten word recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(10):2066–2080, 2011.
  * [6] M. Bleeker and M. de Rijke.  Bidirectional scene text recognition with a single decoder.  arXiv preprint arXiv:1912.03656, 2019.
  * [7] T. Bluche.  Joint line segmentation and transcription for end-to-end handwritten paragraph recognition.  In Proceedings of the Neural Information Processing Systems Conference, 2016.
  * [8] T. Bluche and R. Messina.  Gated convolutional recurrent neural networks for multilingual handwriting recognition.  In Proceedings of the International Conference on Document Analysis and Recognition, 2017.
  * [9] Z. Chen, Y. Wu, F. Yin, and C.-L. Liu.  Simultaneous script identification and handwriting recognition via multi-task learning of recurrent neural networks.  In Proceedings of the International Conference on Document Analysis and Recognition, 2017.
  * [10] J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio.  Attention-based models for speech recognition.  In Proceedings of the Neural Information Processing Systems Conference, 2015.
  * [11] A. Chowdhury and L. Vig.  An efficient end-to-end neural model for handwritten text recognition.  In Proceedings of the British Machine Vision Conference, 2018.
  * [12] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [13] P. Doetsch, M. Kozielski, and H. Ney.  Fast and robust training of recurrent neural networks for offline handwriting recognition.  In Proceedings of the International Conference on Frontiers in Handwriting Recognition, 2014.
  * [14] P. Doetsch, H. Ney, et al.  Improvements in RWTH’s system for off-line handwriting recognition.  In Proceedings of the International Conference on Document Analysis and Recognition, 2013.
  * [15] L. Dong, S. Xu, and B. Xu.  Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition.  In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pages 5884–5888, 2018.
  * [16] P. Dreuw, P. Doetsch, C. Plahl, and H. Ney.  Hierarchical hybrid MLP/HMM or rather MLP features for a discriminatively trained Gaussian HMM: a comparison for offline handwriting recognition.  In Proceedings of the IEEE International Conference on Image Processing, pages 3541–3544, 2011.
  * [17] P. Dreuw, G. Heigold, and H. Ney.  Confidence-and margin-based mmi/mpe discriminative training for off-line handwriting recognition.  International Journal on Document Analysis and Recognition, 14(3):273, 2011.
  * [18] K. Dutta, P. Krishnan, M. Mathew, and C. Jawahar.  Improving CNN-RNN hybrid networks for handwriting recognition.  In Proceedings of the International Conference on Frontiers in Handwriting Recognition, 2018.
  * [19] S. España-Boquera, M. J. Castro-Bleda, J. Gorbe-Moya, and F. Zamora-Martínez.  Improving offline handwritten text recognition with hybrid HMM/ANN models.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(4):767–779, 2010.
  * [20] V. Frinken and H. Bunke.  Continuous handwritten script recognition.  In Handbook of Document Image Processing and Recognition, pages 391–425. 2014.
  * [21] A. Giménez, I. Khoury, J. Andrés-Ferrer, and A. Juan.  Handwriting word recognition using windowed Bernoulli HMMs.  Pattern Recognition Letters, 35:149–156, 2014.
  * [22] A. Graves, M. Liwicki, S. Fernández, R. Bertolami, H. Bunke, and J. Schmidhuber.  A novel connectionist system for unconstrained handwriting recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(5):855–868, 2008.
  * [23] A. Graves and J. Schmidhuber.  Offline handwriting recognition with multidimensional recurrent neural networks.  In Proceedings of the Neural Information Processing Systems Conference, 2009.
  * [24] C. Gulcehre, O. Firat, K. Xu, K. Cho, L. Barrault, H.-C. Lin, F. Bougares, H. Schwenk, and Y. Bengio.  On using monolingual corpora in neural machine translation.  arXiv preprint arXiv:1503.03535, 2015.
  * [25] N. Gurjar, S. Sudholt, and G. A. Fink.  Learning deep representations for word spotting under weak supervision.  In Proceedings of the IAPR International Workshop on Document Analysis Systems, 2018.
  * [26] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
  * [27] L. Kang, P. Riba, M. Villegas, A. Fornés, and M. Rusiñol.  Candidate fusion: Integrating language modelling into a sequence-to-sequence handwritten word recognition architecture.  arXiv preprint arXiv:1912.10308, 2019.
  * [28] L. Kang, J. I. Toledo, P. Riba, M. Villegas, A. Fornés, and M. Rusiñol.  Convolve, attend and spell: An attention-based sequence-to-sequence model for handwritten word recognition.  In Proceedings of the German Conference on Pattern Recognition, 2018\.
  * [29] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [30] P. Krishnan, K. Dutta, and C. Jawahar.  Word spotting and recognition using deep embedding.  In Proceedings of the IAPR International Workshop on Document Analysis Systems, 2018.
  * [31] P. Krishnan and C. Jawahar.  Generating synthetic data for text recognition.  arXiv preprint arXiv:1608.04224, 2016.
  * [32] P. Krishnan and C. Jawahar.  HWNet v2: An efficient word image representation for handwritten documents.  International Journal on Document Analysis and Recognition, 22(4):387–405, 2019.
  * [33] J. Lee, S. Park, J. Baek, S. J. Oh, S. Kim, and H. Lee.  On recognizing texts of arbitrary shapes with 2D self-attention.  arXiv preprint arXiv:1910.04396, 2019.
  * [34] N. Lu, W. Yu, X. Qi, Y. Chen, P. Gong, and R. Xiao.  Master: Multi-aspect non-local network for scene text recognition.  arXiv preprint arXiv:1910.02562, 2019.
  * [35] U.-V. Marti and H. Bunke.  The IAM-database: an English sentence database for offline handwriting recognition.  International Journal on Document Analysis and Recognition, 5(1):39–46, 2002.
  * [36] P. Mermelstein and M. Eyden.  A system for automatic recognition of handwritten words.  In Proceedings of the Fall Joint Computer Conference, 1964.
  * [37] J. Michael, R. Labahn, T. Grüning, and J. Zöllner.  Evaluating sequence-to-sequence models for handwritten text recognition.  arXiv preprint arXiv:1903.07377, 2019.
  * [38] J. Pastor-Pellicer, S. Espana-Boquera, M. J. Castro-Bleda, and F. Zamora-Martinez.  A combined convolutional neural network and dynamic programming approach for text line normalization.  In Proceedings of the International Conference on Document Analysis and Recognition, 2015.
  * [39] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer.  Automatic differentiation in PyTorch.  2017\.
  * [40] V. Pham, T. Bluche, C. Kermorvant, and J. Louradour.  Dropout improves recurrent neural networks for handwriting recognition.  In Proceedings of the International Conference on Frontiers in Handwriting Recognition, 2014.
  * [41] J. Puigcerver.  Are multidimensional recurrent layers really necessary for handwritten text recognition?  In Proceedings of the International Conference on Document Analysis and Recognition, 2017.
  * [42] F. Sheng, Z. Chen, and B. Xu.  NRTR: A no-recurrence sequence-to-sequence model for scene text recognition.  In Proceedings of the International Conference on Document Analysis and Recognition, pages 781–786, 2019.
  * [43] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [44] J. Sueiras, V. Ruiz, A. Sanchez, and J. F. Velez.  Offline continuous handwriting recognition using sequence to sequence neural networks.  Neurocomputing, 289:119–128, 2018.
  * [45] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna.  Rethinking the inception architecture for computer vision.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
  * [46] C. Tensmeyer, C. Wigington, B. Davis, S. Stewart, T. Martinez, and W. Barrett.  Language model supervision for handwriting recognition model adaptation.  In Proceedings of the International Conference on Frontiers in Handwriting Recognition, 2018.
  * [47] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  In Proceedings of the Neural Information Processing Systems Conference, 2017.
  * [48] C. Wigington, C. Tensmeyer, B. Davis, W. Barrett, B. Price, and S. Cohen.  Start, follow, read: End-to-end full-page handwriting recognition.  In Proceedings of the European Conference on Computer Vision, 2018\.
  * [49] F. Zamora-Martinez, V. Frinken, S. España-Boquera, M. J. Castro-Bleda, A. Fischer, and H. Bunke.  Neural network language models for off-line handwriting recognition.  Pattern Recognition, 47(4):1642–1652, 2014.