11institutetext: PRHLT Research Center, Universitat Politècnica de València,
Camí de Vera, s/n, València 46021, Spain 22institutetext: TEKLIA, Paris,
France 33institutetext: Departament d’Informàtica, Universitat de València,
València 46010, Spain

# Reading Order Independent Metrics for Information Extraction in Handwritten
Documents

David Villanova-Aparisi Authors contributed equally to this work.11
0000-0003-2301-6673 Solène Tarride0 22 0000-0001-6174-9865 Carlos-D. Martínez-
Hinarejos 11 0000-0002-6139-2891 Verónica Romero 33 0000-0002-1721-5732
Christopher Kermorvant 22 0000-0002-7508-4080 Moisés Pastor-Gadea 11

###### Abstract

Information Extraction processes in handwritten documents tend to rely on
obtaining an automatic transcription and performing Named Entity Recognition
(NER) over such transcription. For this reason, in publicly available
datasets, the performance of the systems is usually evaluated with metrics
particular to each dataset. Moreover, most of the metrics employed are
sensitive to reading order errors. Therefore, they do not reflect the expected
final application of the system and introduce biases in more complex
documents. In this paper, we propose and publicly release a set of reading
order independent metrics tailored to Information Extraction evaluation in
handwritten documents. In our experimentation, we perform an in-depth analysis
of the behavior of the metrics to recommend what we consider to be the minimal
set of metrics to evaluate a task correctly.

###### Keywords:

Information Extraction Evaluation Metrics Reading Order Full Page Recognition
End-to-End Model

##  1 Introduction

Information Extraction (IE) refers to the identification of parts of digital
text that contain specific knowledge. In particular, the task of Named Entity
Recognition (NER) [20, 7] aims to tag parts of the text that contain specific
semantic information with their corresponding categories. Traditionally, the
approach used to address this task on both handwritten or typeset documents
involves a first step of automatic transcription of the document, followed by
tagging of the resulting digital text [31, 21, 1]. Coupled approaches that
solve the task in one step have also seen successful application in recent
years [2, 3, 28] at solving the task, mainly when applied to historical
documents. Moreover, there is a growing trend in the scientific community
towards adopting full-page end-to-end models for document understanding [6,
14]. The challenge now is to determine effective evaluation methods for these
models to be compared with other approaches, as the currently employed metrics
are sensitive to reading order errors [37].

Researchers in the field of Automatic Text Recognition (ATR) are actively
tackling this issue, either focusing on ways to predict text with the correct
reading order [35, 6], or by designing reading order independent metrics [4].
In a recent proposal, Vidal et al. [32] discuss how reading order challenges
stemming from segmentation errors impact conventional text recognition
metrics. They propose several metrics to measure the transcription quality
independently of the reading order. Unlike traditional ATR metrics, which were
traditionally computed on smaller units like words or text lines to mitigate
reading order issues, these proposed metrics provide a more comprehensive
assessment of ATR on full documents. While there have been advancements in
designing metrics for ATR at full pages, a comparable focus in the realm of IE
is still in its early stages.

When performing IE with the intent of building a knowledge database, the goal
is to identify which Named Entities appear in each document. Therefore, as
long as the segmentation errors do not lead to splitting Named Entities, the
order in which they are recognized should not matter. Therefore, with this
expected application of IE, we claim that metrics should be independent of
reading order. In this paper, we present the following contributions:

  * •

We introduce a set of metrics that are not influenced by reading order when
evaluating the NER process in automatic transcriptions with flat entities
(i.e. we do not consider entities that appear inside other entities);

  * •

We conduct a thorough analysis of the considered metrics using a model trained
on four publicly available datasets and a real-world use case;

  * •

Based on our analysis, we recommend the adoption of the proposed OIECER,
OIEWER, and OINerval metrics for future benchmarks, identified as the most
effective;

  * •

We provide an open-source Python package111https://pypi.org/project/ie-eval/
including all the metric implementations.

##  2 Related Work

The evaluation of Information Extraction (IE) models from handwritten and
printed documents involves a variety of metrics designed to assess the
accuracy and efficiency of automatic models.

In Natural Language Processing tasks such as part-of-speech tagging, where
only the tagging itself is evaluated, metrics like Precision, Recall, and
F1-score can be computed directly without requiring alignment between ground
truth and predicted texts [7, 8, 16]. However, when working with automatic
transcriptions, the ground truth text and predicted text may differ due to
text recognition errors. To evaluate the model, then, it is necessary to
obtain the best match between the ground truth Named Entities and the
hypothesized ones. In this case, metrics introduce dependencies on word
positions or reading order depending on the expected application of the
system.

Yet, these dependencies are not required in IE applications, such as filling a
knowledge database. Moreover, these dependencies are likely to introduce a
bias depending on the segmentation level at which the system is evaluated when
using segmentation-free models [32].

In light of these considerations, this section reviews existing metrics for IE
of scanned handwritten or printed documents. Their application criteria, use
cases, and limitations are also discussed.

###  2.1 Metrics Based On Word Position Alignment

Information extraction metrics applied to printed documents often rely on word
positions to match ground truth and predicted entities based on their
Intersection over Union (IoU).

Once words are associated, two evaluation approaches can be used. Some
researchers evaluate the text recognition task (CER/WER) and the semantic
labeling task (Precision, Recall, and F1 scores) separately [30, 31]. In this
case, the F1 score measures whether the label assigned to Named Entities
matches the ground truth label independently of the content of the predicted
text.

The semantic labeling F1 has been used to evaluate IE systems on handwritten
datasets in which word positions are available [31, 30], notably IAM [17] and
Esposalles [23].

These metrics have two main limitations. First, in many cases, word positions
are not available to align ground truth and predicted words, especially in
handwritten documents where word segmentation is difficult to achieve. Second,
these metrics cannot be used to evaluate segmentation-free models working on
lines, paragraphs, or entire pages [6].

###  2.2 Metrics Based On Text Alignment

In scenarios where word positions are unavailable, the alignment between
ground truth and predicted entities is usually done using text alignment. In
other words, entities are matched according to the order in which they appear
and their edit distance from the ground truth entities.

The IEHHR metric was designed as part of the IEHHR
competition222https://rrc.cvc.uab.es/?ch=10&com=introduction [9], in which
participants had to automatically extract information from 17th-century
marriage records from the Esposalles dataset [23]. The metric considers text
recognition and semantic labeling and can operate at different levels: words,
text lines, records, or full pages. For each matched word in the marriage
record, the score is set to $1-CER$ if the semantic category is correct;
otherwise, it is set to 0. The ECER and EWER metrics [33] are an extension of
the IEHHR metric, with refined substitution costs depending on the
transcription quality and semantic tagging. This metric has been used to
assess the extraction of Named Entities from historical documents. The Nerval
metric [19] matches ground truth and predicted entities based on their CER and
then computes Precision, Recall, and F1 scores. A threshold of 30% CER is
generally used for this metric. Finally, a metric based on the Tree Edit
Distance (TED) [38] was also proposed [14]. For this metric, first, ground
truth and predicted documents are represented as ordered trees. Then, the TED
between ground truth and predicted trees is computed before being normalized
by the TED between the label and an empty tree. To compute the Tree Error Rate
$TER$, a semantic labeling error costs 1, while a text error is computed as
the text edit distance between ground truth and predicted leaves [14]. The
accuracy is obtained by taking $min(100,100-TER)$.

The IEHHR, ECER, EWER, and Nerval metrics have been used to evaluate IE models
on various handwritten documents [33, 21, 26, 29, 27], including tables (POPP
[5]), records (Balsac [29], Esposalles [23]), index cards (Simara [27]), and
books (IAM [17], HOME Alcar [24]). On the other hand, the TED accuracy has
been mainly used to evaluate IE models on printed documents [13, 14, 39], such
as forms (FUNSD [10]) and receipts (CORD [22], SROIE [12]).

Although text alignment-based metrics do not introduce a dependency on word
position, they do require the system to extract information in a particular
order. For structured documents with a fixed layout, this may not be a
problem, as the predicted fields can be reordered according to a template.
However, for other documents, such as records, the Named Entities included in
a document and the order in which they appear can vary.

###  2.3 Alignment-Free Metrics

To the best of our knowledge, only one alignment-free metric has been proposed
to evaluate IE models: the field extraction F1.

This metric [14, 36, 11] checks whether extracted fields appear in the ground
truth. If a single character is missed or added, the metric considers that the
field extraction has failed. It is referred to as word-level F1 when
considering tagged words [36], or entity-level F1 when considering tagged
entities [14].

It has been used to evaluate IE models on printed structured documents, such
as forms (FUNSD [10]) and receipts (CORD [22], SROIE [12]).

The field extraction F1 metric has the advantage of not relying on any
alignment between the label and prediction. However, it cannot assess the
structure of nested entities. Additionally, it does not consider partial
overlaps and is very strict for long entity blocks.

###  2.4 Discussion

A first striking observation is the use of different metrics to evaluate IE
models, particularly evident in the discrepancy between the metrics used on
printed and handwritten datasets. This creates a significant barrier to
effectively comparing different models, especially across datasets. As models
improve, the gap between handwriting recognition and printed text recognition
is narrowing, and we see a need to normalize the use of metrics in both
communities.

In addition, the dependence of some metrics on alignment, whether based on
word position or reading order, creates inherent challenges in the evaluation
process. In response to these issues, we advocate adapting existing metrics by
eliminating alignment dependencies, paving the way for fairer and more
reliable model evaluations. We intend to compare a set of metrics and analyze
their correlation and, from this study, select those that we consider the most
appropriate to normalize evaluation benchmarks.

##  3 Evaluated Metrics

From an Information Extraction perspective of the task, having a correct
reading order is not necessary. Furthermore, it may introduce a bias since
complex documents, such as those that include marginalia [25], are more prone
to reading order problems. This section briefly introduces the metrics we have
considered in our evaluation and the proposed reading order independent
metrics we implemented and tested.

###  3.1 Order Dependent Metrics

####  3.1.1 Character Error Rate (CER) and Word Error Rate (WER)

When evaluating automatic transcriptions of any kind, it is necessary to have
some criteria to determine their quality. CER and WER [34] are widespread
metrics that compute the distance between two transcriptions based on the
number of edit operations necessary to convert one text into the other.

When dealing with the evaluation of the NER process, CER and WER are not
particularly suitable metrics since they cannot incorporate semantics as they
consider tags as single tokens in the transcription. In our experiments, we
will use CER and WER metrics to assess the quality of the text recognition
task performed by our model.

Moreover, we have decomposed the CER and WER computation for each Named Entity
category. This way, we can evaluate the difficulty the model faces when
transcribing some particular categories. The results of this evaluation are
presented in the Annex 7.1.

####  3.1.2 Entity Character Error Rate (ECER) and Entity Word Error Rate
(EWER)

The ECER and EWER metrics were initially presented in [33] and are a
generalization of the evaluation criteria presented in [9]. To compute the
distance between two sequences of Named Entities, an edit distance is
calculated where the substitution cost depends first on the correctness of the
tagging and, finally, on the quality of the transcription of the Named Entity
if it was tagged correctly. In the computation of the ECER, the quality of the
transcription is assessed by computing the CER. In contrast, in the EWER, the
quality of the transcription is evaluated with the WER. For this article, we
employ a revised version of the ECER and EWER metrics in which the score is
normalized only by the length of the reference and the maximum substitution
cost is one. This revised version of the metric is more correlated with the
definition of the WER, where the score is normalized only by the number of
words of the reference. Moreover, this makes it directly comparable with the
reading order independent adaptation proposed in Section 3.2.

####  3.1.3 Soft-aligned Entity Precision, Recall and F1-Score (Nerval)

The Nerval[19] method for computing Precision, Recall, and F1 scores is also
based on an edit distance. In Nerval, tagged transcriptions are aligned at
Named Entity level and a threshold is set for the acceptable character error
percentage. In this alignment, True Positives are hypothesized Named Entities
that have been correctly tagged and transcribed when compared to their ground
truth counterpart. False Positives are hypothesized Named Entities that do not
find an acceptable match in the ground truth. False Negatives are ground truth
Named Entities for which no acceptable match was found in the hypothesis. Even
though, by default, the threshold is set to 30%, the metric gives interesting
information in the extremes. If we allow a 0% character error, then only
perfect matches are allowed. If a 100% character error is tolerated, the
metric solely evaluates the tagging.

###  3.2 Assignment-based Proposed Reading Order Independent Metrics

The first set of proposed reading order independent metrics are adaptations of
Nerval, ECER, and EWER in which sequentiality constraints have been eliminated
when computing the best alignment. More specifically, we will allow one-to-one
relations between Named Entities in any given order. Our formulation for the
problem shares the notation presented in [18] and is inspired by the concepts
presented in [32]. Given two sequences of Named Entities, $x$ and $y$, we want
to search for a non-sequential alignment $A(x,y)$, which is a sequence of
pairs of Named Entity indices $(j,k)$ such that $1\leq j\leq|x|$, $1\leq
k\leq|y|$, and for two distinct pairs $(j,k),(j^{\prime},k^{\prime})\in
A(x,y)$ it must be satisfied that $j\neq j^{\prime}\wedge k\neq k^{\prime}$.
From the best non-sequential alignment, we can compute the minimum distance
$d(x,y)$ between the two sequences by following the following equation:

$d(x,y)=\min_{A(x,y)}\sum_{j,k\in A(x,y)}\delta(x_{j},y_{k})$ (1)

Obtaining the best alignment can be reduced to an assignment problem in which
each hypothesized Named Entity $Y_{k}$ can be associated with each ground
truth Named Entity $X_{j}$ with a cost $\delta(x_{j},y_{k})$. To solve this
assignment problem, we employ the Hungarian Algorithm [15]. Since the
algorithm expects square cost matrices and the length of the Named Entity
sequences $x$ and $y$ may differ, we resort to padding whichever sequence is
smaller with dummy elements $\lambda$ with an assignment cost
$\delta(x_{j},y_{k})$ equal to inserting or deleting a Named Entity. How the
cost matrix is computed is what differentiates the proposed metrics.

####  3.2.1 Reading Order Independent Entity Character Error Rate (OIECER) and
Entity Word Error Rate (OIEWER)

In the adaptation of the ECER and EWER metrics [33], the assignment cost
$\delta(x_{j},y_{k})$ is equal to one if one of the Named Entities is a dummy
element $\lambda$ or equal to the cost of substituting $x_{j}$ with $y_{k}$ if
both Named Entities exist. For the cost of the substitution, the maximum
values of CER and WER are going to be capped to one. By saturating these
metrics, the substitution cost is never larger than one. Assuming that
$c(x_{j})$ and $c(y_{k})$ are the Named Entity categories of $x_{j}$ and
$y_{k}$ and that $t(x_{j})$ and $t(y_{k})$ are the transcriptions belonging to
each Named Entity, the costs can be computed as follows:

$\displaystyle\delta_{\mathrm{ECER}}(x_{j},y_{k})=\begin{cases}1\,&\text{if}\,x_{j}=\lambda\vee
y_{k}=\lambda\\\ 1\,&\text{if}\,c(x_{j})\neq c(y_{k})\\\
\min(1,\mathrm{CER}(t(x_{j}),t(y_{k})))\,&\mathrm{otherwise}\end{cases}$ (2)

$\displaystyle\delta_{\mathrm{EWER}}(x_{j},y_{k})=\begin{cases}1\,&\text{if}\,x_{j}=\lambda\vee
y_{k}=\lambda\\\ 1\,&\text{if}\,c(x_{j})\neq c(y_{k})\\\
\min(1,\mathrm{WER}(t(x_{j}),t(y_{k})))\,&\mathrm{otherwise}\end{cases}$ (3)

From the computed cost matrix, we can obtain the best alignment and,
consequently, the distance between both Named Entity sequences:

$\mathrm{OIECER}(x,y)=\min_{A(x,y)}\sum_{j,k\in
A(x,y)}\delta_{\mathrm{ECER}}(x_{j},y_{k})$ (4)

$\mathrm{OIEWER}(x,y)=\min_{A(x,y)}\sum_{j,k\in
A(x,y)}\delta_{\mathrm{EWER}}(x_{j},y_{k})$ (5)

From this definition of distance, we can extend the computation of the score
as a weighted average over each document $s$ belonging to a corpus $C$ by
considering the set of all hypothesized Named Entity sequences $Y$ and the
ground truth sequences $X$. For a given sample document $s$, its associated
hypothesized Named Entity sequence is $Y_{s}$ and the ground truth sequence is
$X_{s}$. From here, we compute the score for a given corpus $C$ as follows:

$\mathrm{OIECER}(C)=\frac{\sum_{s\in
C}{\mathrm{OIECER}(X_{s},Y_{s})}}{\sum_{s\in C}{|X_{s}|}}$ (6)

$\mathrm{OIEWER}(C)=\frac{\sum_{s\in
C}{\mathrm{OIEWER}(X_{s},Y_{s})}}{\sum_{s\in C}{|X_{s}|}}$ (7)

####  3.2.2 Reading Order Independent Soft-aligned Entity Precision, Recall
and F1-Score (OINerval)

In the adaptation of the Nerval computation for Precision, Recall, and F1
scores, the assignment cost $\delta(x_{j},y_{k})$ is computed from the
definition of $\delta_{\mathrm{ECER}}$, also considering a saturated version
of the CER between two transcriptions. For a given character error threshold
$M$, the computation of the cost is as follows:

$\displaystyle\delta_{\mathrm{NERVAL-M}}(x_{j},y_{k})=\begin{cases}1\,&\text{if}\,x_{j}=\lambda\vee
y_{k}=\lambda\\\ 2\,&\text{if}\,c(x_{j})\neq c(y_{k})\\\
\mathrm{otherwise}&\begin{cases}2\,&\text{if}\,\min(1,\mathrm{CER}(t(x_{j}),t(y_{k})))>\mathrm{M}\\\
0\,&\text{if}\,\min(1,\mathrm{CER}(t(x_{j}),t(y_{k})))\leq\mathrm{M}\\\
\end{cases}\end{cases}$ (8)

The assignment cost $\delta(x_{j},y_{k})$ determines the nature of the match.
If the cost is 0, the Named Entity must be considered a True Positive. Suppose
the cost is two due to category mismatch or excessive character error. In that
case, the hypothesized Named Entity $y_{k}$ must be considered a False
Positive, and the ground truth Named Entity $x_{j}$ as a False Negative. If
the cost is one due to $y_{k}$ being a dummy symbol $\lambda$, then $x_{j}$ is
a False Negative. Finally, if the cost is one due to $x_{j}$ being a dummy
symbol $\lambda$, then $y_{k}$ must be considered a False Positive.

This principle can be extended to whole corpora in which, for each document,
the number of TP, FP, and FN are calculated between the hypothesized sequence
of Named Entities and the ground truth. Calculating the micro Precision,
Recall, and F1 scores can be done traditionally:

$P=\frac{TP}{TP+FP}\;\;\;\;\;R=\frac{TP}{TP+FN}\;\;\;\;\;F_{1}=2\cdot\frac{P\cdot
R}{P+R}$ (9)

###  3.3 Bag-of-words-based Reading Order Independent Metrics

Our second set of reading order independent metrics is based on the bag-of-
words concept and it is inspired by the proposal of [32]. Here, we represent
the tagged text using a vector of counts in which, for each tagged token in
the vocabulary $v\in V^{\prime}$, we count the number of appearances of that
token in the reference $f_{X}(v)$ and in the hypothesis $f_{Y}(v)$. Note that,
in our case, we are considering the Named Entity category as a part of the
token and removing text that does not belong to any category, resulting in the
vocabulary $V^{\prime}$.

####  3.3.1 Bag-of-tagged-words Word Error Rate (btWER)

The bag-of-tagged-words Word Error Rate (btWER) is a direct adaptation from
the bag-of-words Word Error Rate (bWER) presented in [32]. Here, we represent
the output of the system and the reference text as sets of tagged words.
Instead of considering whole transcriptions in our adaptation, we focus on
Named Entities by removing words not tagged as belonging to a Named Entity.
With this modified vocabulary $V^{\prime}$ and considering that $|X|$ is the
number of tagged words in the reference and $|Y|$ is the number of tagged
words in the hypothesis, we can compute the btWER between a reference $X$ and
a hypothesis $Y$. For this, we consider the frequency with which each word $v$
appears in the reference, $f_{X}(v)$, and in the hypothesis, $f_{Y}(v)$. The
resulting equation is as follows:

$\text{btWER}(X,Y)=\frac{1}{2\left|X\right|}\left(\left|\left|X\right|-\left|Y\right|\right|\sum_{v\in
V^{\prime}}{\left|f_{X}(v)-f_{Y}(v)\right|}\right)$ (10)

####  3.3.2 Bag-of-tagged-words Precision, Recall and F1-Score

From this text representation, it is possible to compute Precision, Recall,
and F1-score. We can define the computation of the True Positives (TP), False
Positives (FP), and False Negatives (FN) as follows:

$\displaystyle\begin{split}\mathrm{TP}(X,Y)&=\sum_{v\in
V^{\prime}}{\text{min}(f_{X}(v),f_{Y}(v))}\\\ \mathrm{FP}(X,Y)&=\sum_{v\in
V^{\prime}}{\text{max}(f_{Y}(v)-f_{X}(v),0)}\\\ \mathrm{FN}(X,Y)&=\sum_{v\in
V^{\prime}}{\text{max}(f_{X}(v)-f_{Y}(v),0)}\end{split}$ (11)

Similarly to the Nerval evaluation, this computation can be extended to a
whole corpus if, for each document, we compute the amount of TP, FP, and FN
between the hypothesis and the ground truth and sum them. The micro Precision,
Recall, and F1 scores can be computed as in Equation 9.

###  3.4 Bag-of-entities Error Rate (beER)

Another representation level consists of, instead of considering words as the
minimal unit of information, taking whole Named Entities as the vocabulary for
the vector of counts. With this representation, we can follow the computation
for btWER as in Equation 10, resulting in a bag-of-entities Error Rate (beER).
With this approach, a single character error in the transcription of a word of
the Named Entity is penalized as a complete error.

###  3.5 Bag-of-entities Precision, Recall and F1-Score

With count vectors for whole Named Entities, Precision, Recall, and F1-Scores
can also be computed as in its bag-of-tagged-words counterpart (see Equation
11). As with the beER, a single character error in a word of the hypothesis
affects the whole Named Entity to which it belongs.

##  4 Experimental Setup

###  4.1 Datasets

Five datasets are used in this study: IAM [17], Simara [27], Esposalles [23],
POPP [5], and a private corpus consisting of French Military Records. An
overview of each dataset is presented in Fig. 1

####  4.1.1 IAM

The IAM dataset [17] contains modern English documents written by 500 authors.
It includes 747 training pages, 116 validation pages, and 336 test pages with
their corresponding transcriptions. In 2021, Tüselmann et al. annotated and
made available Named Entities in this dataset [31], with 18 tags: Cardinal,
Date, Event, FAC, GPE, Language, Law, Location, Money, NORP, Ordinal,
Organization, Person, Percent, Product, Quantity, Time and Work of art. We use
the RWTH split and train a model for the HTR+NER task on paragraphs for our
experiments. Note that we do not train on full pages to exclude the printed
instructions in the header.

####  4.1.2 Simara

The Simara dataset [27] is a corpus of finding aids from the National Archives
of France, dating from the 18th to 20th centuries. It includes 3778 training
pages, 811 validation pages, and 804 test pages, with their corresponding
information organized in 7 fields: Date, Title, Serie, Analysis,
Volume_number, Article_number, Arrangement. Documents come from 6 different
series where all documents contain the same type of information, but each
series has a specific layout. We train a model for key-value information
extraction on entire pages for our experiments.

####  4.1.3 Esposalles

The ESPOSALLES dataset [23] is a collection of historical marriage records
from the archives of the Cathedral of Barcelona, dating from the 17th century.
Each document is written in old Catalan by a single writer, with word, line,
and record segmentations. It includes 75 training pages, 25 validation pages,
and 25 test pages with their corresponding transcriptions and Named Entities.
Each word is also labeled with a semantic category (Name, Surname, Occupation,
Location, State, Other) and a person (Husband, Wife, Husband’s father,
Husband’s mother, Wife’s father, Wife’s mother, Other person, None). We train
a model for the HTR+NER task on records for our experiments. Note that the two
types of entities are merged (e.g., the tag HusbandName is used for a word
labeled as Husband and Name), for a total of 35 combined tags appearing in the
dataset.

####  4.1.4 POPP

The POPP dataset [5] is a collection of French handwritten tables from the
1926 Paris census. Each document contains 30 rows, and each row describes an
individual with 10 columns: Surname, Name, Birth date, Birth place,
Nationality, Civil status, Link, Education level, Occupation, and Employer. In
our experiments, we use the column name as a Named Entity. It includes 128
training pages, 16 validation pages, and 16 test pages with their
corresponding transcriptions and Named Entities. We train a model for key-
value information extraction on text lines for our experiments.

####  4.1.5 Real-World Application: French Military Records

French Military Records consist of handwritten tables detailing information
about soldiers in the French army during the 18th century. Annotations for
these records are collected via a crowdsourcing approach, where annotators
input information about individual soldiers using a form. However, these
annotations are not directly linked to specific areas in the corresponding
images, resulting in each two-page document being associated with an unordered
(and sometimes incomplete) set of annotations for different soldiers. This
annotation collection method is commonly employed in crowdsourcing
applications333Mémoires des Hommes - French Military Archives,
Geneanet/Ancestry, Filae. Our model is trained on 118 training and 29
validation full-page images to identify critical information including
Surname, Name, Birth date, Birth place, Date of enrollment, while disregarding
other information present in the documents.

(a) IAM [17]. Legend: Object; GPE; Date; Person.

(b) Simara [27]. Legend: Date; Title; Analysis; Arrangement; Serie; Article;
Volume.

(c) POPP [5]. Legend: Surname; Name; Birth date; Birth location; Link,
Occupation, Employer.

(d) Esposalles [23]. Legend: Name, Surname, Occupation, Location, State. Each
entity refers to a specific person, as illustrated in the image.

(e) Real use case: military records. Legend: Name, Surname, Bith place, Birth
date, Date of enrollment.

Figure 1: Illustration of the five datasets used in this study. Named Entities
are highlighted. Not all entity types appear in these examples.

###  4.2 Model

We train a single DAN model for each dataset for key-value information
extraction. DAN [6] is an open-source attention-based Transformer model for
handwritten text recognition that can work directly on pages. The encoder is
fully convolutional, while the decoder is a Transformer network. It is trained
with the cross-entropy loss function. The last layer is a linear layer with a
softmax activation function that computes probabilities associated with each
vocabulary character. The model is trained to recognize characters and unique
tokens representing Named Entities. In the case of Simara and POPP, where all
words are entities, we use opening tags before each entity, as illustrated in
Table 1. However, since IAM and Esposalles both include words that are not
Named Entities, each word is tagged individually with opening tags.

###  4.3 Evaluation Methodology

Once the model is trained, we use it to predict the test set of each dataset.
Labels and predictions on the test set are then converted into the IOB2
format444https://en.wikipedia.org/wiki/Inside-outside-beginning_(tagging).
Then, the selected metrics are computed on each dataset using a Python package
developed for this study. Finally, we shuffle predictions at Named Entity
level and re-evaluate them to study the impact of reading order in different
metrics. More precisely, we shuffle entity blocks across the document before
evaluating with the same metrics. However, the reading order within entity
blocks is preserved, as illustrated in Table 1.

Table 1: Example of a reference transcription from Simara, and the corresponding shuffled version. Note that we use opening tags to delimit Named Entities. Original |  <title>AUBERT Huissier priseur à Paris <analysis>Contre Baraise <date>10 mars 1773 <serie>X1A <article>4723 <reference>205
---|---
Shuffled |  <article>4723 <title>AUBERT Huissier priseur à Paris <serie>X1A <reference>205 <date>10 mars 1773 <analysis>Contre Baraise

To ensure reproducibility of this work, labels and predictions in IOB2 format
are publicly available555https://zenodo.org/records/11083657. Additionally, we
have released an open-source Python
package666https://gitlab.teklia.com/ner/metrics/ie-eval/ allowing to compute
multiple metrics. This package is available on
PyPi777https://pypi.org/project/ie-eval/

##  5 Results

Following the experimental method, the model’s performance has been evaluated
with all the considered metrics in all the available datasets, obtaining the
results presented in Table 2. In this section, we analyze the results and
present the main takeaways. Pearson and Spearman correlation matrix between
different metrics across all four datasets are also presented in Annex 7.2,
respectively in Figures 2 and 3.

Table 2: Comparison between metrics across four datasets. OI refers to Order-Independent metric. Reg. stands for regular data, whereas Shuf. stands for shuffled data. The largest 95% confidence interval computed is $5.6$ for the Nerval-P in IAM-18 (regular). Therefore, the OI version of the text-alignment metrics obtains a statistically significant difference compared to their original version.  | IAM-18 | Simara | Esposalles | POPP | Military
---|---|---|---|---|---
Metric | Reg. | Shuf. | Reg. | Shuf. | Reg. | Shuf. | Reg. | Shuf. | Shuf.
ECER $\downarrow$ | 86.5 | 98.2 | 5.0 | 74.0 | 3.6 | 88.5 | 9.2 | 79.2 | 93.0
OIECER $\downarrow$ | 83.7 | 83.7 | 5.0 | 5.0 | 3.5 | 3.5 | 9.2 | 9.2 | 68.1
EWER $\downarrow$ | 93.3 | 103.3 | 10.2 | 75.5 | 4.8 | 88.8 | 20.9 | 82.4 | 109.4
OIEWER $\downarrow$ | 91.5 | 91.5 | 10.2 | 10.2 | 4.7 | 4.7 | 20.9 | 20.9 | 95.1
Nerval-P $\uparrow$ | 29.0 | 2.9 | 95.0 | 37.7 | 96.3 | 17.0 | 89.3 | 35.0 | 14.3
OINerval-P $\uparrow$ | 29.0 | 29.0 | 95.0 | 95.0 | 96.3 | 96.3 | 89.3 | 89.3 | 33.9
Nerval-R $\uparrow$ | 26.6 | 2.7 | 95.3 | 37.9 | 97.1 | 17.2 | 90.0 | 35.3 | 15.1
OINerval-R $\uparrow$ | 26.6 | 26.6 | 95.4 | 95.4 | 97.1 | 97.1 | 90.0 | 90.0 | 36.0
Nerval-F1 $\uparrow$ | 27.7 | 2.8 | 95.1 | 37.8 | 96.7 | 17.1 | 89.6 | 35.1 | 14.7
OINerval-F1 $\uparrow$ | 27.7 | 27.7 | 95.2 | 95.2 | 96.7 | 96.7 | 89.6 | 89.6 | 34.9
btWER $\downarrow$ | 85.3 | 85.3 | 17.4 | 17.4 | 4.6 | 4.6 | 24.2 | 24.2 | 92.2
bt-P $\uparrow$ | 29.2 | 29.2 | 84.8 | 84.8 | 95.9 | 95.9 | 77.4 | 77.4 | 21.2
bt-R $\uparrow$ | 21.2 | 21.2 | 83.7 | 83.7 | 96.2 | 96.2 | 78.0 | 78.0 | 20.9
bt-F1 $\uparrow$ | 24.5 | 24.5 | 84.3 | 84.3 | 96.0 | 96.0 | 77.7 | 77.7 | 21.0
beER $\downarrow$ | 94.2 | 94.2 | 23.2 | 23.2 | 5.8 | 5.8 | 23.1 | 23.1 | 96.2
be-P $\uparrow$ | 21.6 | 21.6 | 77.1 | 77.1 | 94.4 | 94.4 | 77.4 | 77.4 | 18.8
be-R $\uparrow$ | 19.8 | 19.8 | 77.3 | 77.3 | 95.2 | 95.2 | 78.0 | 78.0 | 20.0
be-F1 $\uparrow$ | 20.7 | 20.7 | 77.2 | 77.2 | 94.8 | 94.8 | 77.7 | 77.7 | 19.4
CER $\downarrow$ | 5.0 | 74.2 | 6.4 | 39.7 | 0.4 | 68.1 | 8.8 | 67.5 | 80.8
WER $\downarrow$ | 15.5 | 94.0 | 17.3 | 54.2 | 1.5 | 84.2 | 23.6 | 80.5 | 106.1

###  5.1 Model Analysis Across Datasets

From Table 2, we can see the difference in difficulty between the datasets.
While the DAN model performs well enough regarding transcription quality
across all corpora, it seems almost unable to properly tag Named Entities in
the IAM-18 dataset, which was expected since there are few training examples
for each entity type. For Simara, POPP, and Esposalles, we consider that the
model performs well enough, as it obtains an OINerval-F1 score close to or
above 90% and an OIEWER score close to or below 20%. When comparing
correlations across different metrics, we found that DAN showed similar
correlation patterns on Simara and POPP. On Esposalles, the model exhibits
very strong correlations due to nearly flawless text recognition, while on
IAM, there is a notable lack of correlation between CER/WER and other metrics
due to well-recognized text but poorly identified entities.

###  5.2 Impact of Reading Order

Given the drop in score when using the original Nerval, ECER, and EWER metrics
in the shuffled versions of the data, it is clear that they are sensitive to
reading order errors.

Observing the results obtained by the proposed reading-order independent
adaptations of Nerval, ECER, and EWER in the case of regular data, we can
conclude that they have been well formulated and implemented since the results
are practically identical to those of the original metrics. In addition, we
observe a perfect correlation between Nerval and OINerval and between
ECER/EWER and OIECER/OIEWER metrics in terms of linear and rank correlations.
There are slight differences in some corpora, which are to be expected since
unconstrained alignments allow a slightly more optimistic evaluation in the
case of Named Entity repetition.

The correctness of the implementation is also supported by the fact that
OINerval, OIECER, and OIEWER give the same results regardless of whether the
data has been shuffled. Given that the proposed versions of the metrics
perform as expected while allowing for a more general use case, their use is
now recommended when evaluating information extraction tasks.

###  5.3 Metrics Similarity

We aim to assess the similarity between order-independent (OI) metrics.
Identifying redundancies between metrics is critical to determining which
metrics should be used as a reference in future evaluations. To achieve this,
we thoroughly analyze the correlations between all the metrics. Correlation
matrices are presented in Annex 7.2. Note that we rely on the absolute
correlation for our analysis, as the sign depends on the metric types: error
rates (best at 0) or detection rates (best at 100).

A first observation is that close metrics yield almost perfect correlation:
OIECER and OIWER, bt-F1 and btWER, be-F1 and b1-WER. Additionally, the CER/WER
have relatively low correlations with most IE metrics, as they only evaluate
text recognition on all words, not only Named Entities. Robust rank
correlations are observed for all IE metrics except Nerval, suggesting that it
provides complementary information about model performance. As most other
metrics have similar behavior, we recommend using ECER and EWER. These metrics
are more appropriate for IE tasks since they operate at the entity level
rather than the tagged word level, unlike bt-F1 and btWER, and manage to deal
effectively with split errors, unlike be-F1 and beER. These observations stand
for all datasets except for IAM, where the model produces many tagging errors.

In addition to this quantitative analysis, a qualitative analysis of five
prediction scenarios is presented and discussed in Table 7 of the Annex 7.3.

In conclusion, ECER/EWER and Nerval are complementary and do not correlate
strongly, suggesting they provide complementary information about model
performance. ECER/EWER evaluate text recognition and labeling errors, while
Nerval matches entities using a CER threshold. Two scenarios emerge. If the
final information extraction task allows a certain percentage of errors, we
recommend using Nerval with the appropriate CER threshold. However, for a more
global evaluation, or if the acceptable percentage of acceptable errors is not
known, ECER/EWER provides a more comprehensive evaluation.

###  5.4 Real-World Application: French Military Records

The French military records dataset presents challenges as the model must
extract partial information from complex full-page layouts, compounded by a
limited number of training images. While the model was trained on sequentially
ordered annotations, allowing it to learn the correct reading order, the test
set contains unordered annotations. The results on this dataset are shown in
Table 2. The model performs poorly, with only 20% of the entities being
perfectly extracted , which is comparable to the performance of DAN on IAM.
However, notable improvements are observed in the OIECER and OINerval-F1
metrics compared to IAM due to fewer tagging errors and more text recognition
errors. It should be noted that order-dependent metrics underestimate model
capabilities, primarily due to the misalignment of predictions with the ground
truth. In contrast, order-independent (OI) metrics yield more accurate
evaluation scores, highlighting the need for OI metrics to improve model
evaluation for real-world data with unordered annotations.

##  6 Conclusion

In this article, we have proposed reading-order-independent versions of the
Nerval, ECER, and EWER metrics and the adaptation of two bag-of-words-based
evaluation schemes for NER in scanned documents. The behavior of the
considered metrics has been evaluated in four publicly available corpora,
proving that our proposal mimics the original behavior of the metrics while
remaining robust to reading order errors. We have analyzed the correlation
between the proposed metrics and, considering this study and the expected
application of the system, we have chosen ECER and EWER as the general use
case metrics to employ from now on. However, if the threshold of tolerable
character error is known, we recommend the usage of Nerval. In future work, we
would like to extend the usage of these metrics to models that work at
different segmentation levels (i.e., page level, line level, and word level),
providing a benchmark for each approach. We also plan to extend these metrics
to handle nested entities.

## Acknowledgments

The authors would like to thank Yoann Schneider for his careful code review.
This work was supported by Grant PID2020-116813RB-I00 funded by MCIN/AEI/
10.13039/501100011033, by Grant ACIF/2021/436 funded by Generalitat Valenciana
and by Grant PID2021-124719OB-I00 funded by MCIN/AEI/10.13039/ 501100011033
and by ERDF, EU A way of making Europe.

## References

  * [1] Abadie, N., Carlinet, E., Chazalon, J., Duménieu, B.: A benchmark of named entity recognition approaches in historical documents application to 19th century french directories. In: Uchida, S., Barney, E., Eglin, V. (eds.) Document Analysis Systems. pp. 445–460. Springer International Publishing, Cham (2022)
  * [2] Boroş, E., Romero, V., Maarand, M., Zenklová, K., Křečková, J., Vidal, E., Stutzmann, D., Kermorvant, C.: A comparison of sequential and combined approaches for named entity recognition in a corpus of handwritten medieval charters. In: 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR). pp. 79–84. IEEE (2020)
  * [3] Carbonell, M., Villegas, M., Fornés, A., Lladós, J.: Joint recognition of handwritten text and named entities with a neural end-to-end model. In: 2018 13th IAPR International Workshop on Document Analysis Systems (DAS). pp. 399–404. IEEE (2018)
  * [4] Clausner, C., Pletschacher, S., Antonacopoulos, A.: Flexible character accuracy measure for reading-order-independent evaluation. Pattern Recognition Letters 131, 390–397 (2020). https://doi.org/https://doi.org/10.1016/j.patrec.2020.02.003
  * [5] Constum, T., Kempf, N., Paquet, T., Tranouez, P., Chatelain, C., Brée, S., Merveille, F.: Recognition and information extraction in historical handwritten tables: Toward understanding early 20th century paris census. In: Document Analysis Systems: 15th IAPR International Workshop, DAS 2022, La Rochelle, France, May 22–25, 2022, Proceedings. p. 143–157. Springer-Verlag, Berlin, Heidelberg (2022). https://doi.org/10.1007/978-3-031-06555-2_10
  * [6] Coquenet, D., Chatelain, C., Paquet, T.: Dan: a segmentation-free document attention network for handwritten document recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence p. 1–17 (2023). https://doi.org/10.1109/tpami.2023.3235826
  * [7] Ehrmann, M., Hamdi, A., Pontes, E.L., Romanello, M., Doucet, A.: Named entity recognition and classification in historical documents: A survey. ACM Comput. Surv. 56(2) (sep 2023). https://doi.org/10.1145/3604931, https://doi.org/10.1145/3604931
  * [8] Ehrmann, M., Romanello, M., Najem-Meyer, S., Doucet, A., Clematide, S.: ”overview of hipe-2022: Named entity recognition and linking in multilingual historical documents”. In: Experimental IR Meets Multilinguality, Multimodality, and Interaction: 13th International Conference of the CLEF Association, CLEF 2022, Bologna, Italy, September 5–8, 2022, Proceedings. p. 423–446. Springer-Verlag, Berlin, Heidelberg (2022). https://doi.org/10.1007/978-3-031-13643-6_26, https://doi.org/10.1007/978-3-031-13643-6_26
  * [9] Fornés, A., Romero, V., Baró, A., Toledo, J.I., Sánchez, J.A., Vidal, E., Lladós, J.: Icdar2017 competition on information extraction in historical handwritten records. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR). vol. 01, pp. 1389–1394 (2017). https://doi.org/10.1109/ICDAR.2017.227
  * [10] Guillaume Jaume, Hazim Kemal Ekenel, J.P.T.: Funsd: A dataset for form understanding in noisy scanned documents. In: Accepted to ICDAR-OST (2019)
  * [11] Hong, T., Kim, D., Ji, M., Hwang, W., Nam, D., Park, S.: Bros: A pre-trained language model focusing on text and layout for better key information extraction from documents. Proceedings of the AAAI Conference on Artificial Intelligence 36, 10767–10775 (06 2022). https://doi.org/10.1609/aaai.v36i10.21322
  * [12] Huang, Z., Chen, K., He, J., Bai, X., Karatzas, D., Lu, S., Jawahar, C.V.: Icdar2019 competition on scanned receipt ocr and information extraction. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 1516–1520 (2019). https://doi.org/10.1109/ICDAR.2019.00244
  * [13] Hwang, W., Yim, J., Park, S., Yang, S., Seo, M.: Spatial dependency parsing for semi-structured document information extraction. In: Zong, C., Xia, F., Li, W., Navigli, R. (eds.) Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. pp. 330–343. Association for Computational Linguistics, Online (Aug 2021). https://doi.org/10.18653/v1/2021.findings-acl.28
  * [14] Kim, G., Hong, T., Yim, M., Nam, J., Park, J., Yim, J., Hwang, W., Yun, S., Han, D., Park, S.: Ocr-free document understanding transformer. In: European Conference on Computer Vision (ECCV) (2022)
  * [15] Kuhn, H.W.: The Hungarian Method for the Assignment Problem. Naval Research Logistics Quarterly 2(1–2), 83–97 (March 1955). https://doi.org/10.1002/nav.3800020109
  * [16] Linhares Pontes, E., Hamdi, A., Sidere, N., Doucet, A.: Impact of ocr quality on named entity linking. In: Jatowt, A., Maeda, A., Syn, S.Y. (eds.) Digital Libraries at the Crossroads of Digital Information for the Future. pp. 102–115. Springer International Publishing, Cham (2019)
  * [17] Marti, U.V., Bunke, H.: The iam-database: an english sentence database for offline handwriting recognition. International Journal on Document Analysis and Recognition 5, 39–46 (2002), https://doi.org/10.1007/s100320200071
  * [18] Marzal, A., Vidal, E.: Computation of normalized edit distance and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence 15(9), 926 – 932 (1993). https://doi.org/10.1109/34.232078
  * [19] Miret, B., Kermorvant, C.: Nerval: a python library for named-entity recognition evaluation on noisy texts. https://gitlab.teklia.com/ner/nerval (2021)
  * [20] Mohit, B.: Named entity recognition. In: Natural language processing of semitic languages, pp. 221–245. Springer (2014)
  * [21] Monroc, C.B., Miret, B., Bonhomme, M.L., Kermorvant, C.: A comprehensive study of open-source libraries for named entity recognition on handwritten historical documents. In: Uchida, S., Barney, E., Eglin, V. (eds.) Document Analysis Systems. pp. 429–444. Springer International Publishing, Cham (2022)
  * [22] Park, S., Shin, S., Lee, B., Lee, J., Surh, J., Seo, M., Lee, H.: Cord: A consolidated receipt dataset for post-ocr parsing. In: Document Intelligence Workshop at Neural Information Processing Systems (2019)
  * [23] Romero, V., Fornés, A., Serrano, N., Sánchez, J.A., Toselli, A.H., Frinken, V., Vidal, E., Lladós, J.: The esposalles database: An ancient marriage license corpus for off-line handwriting recognition. Pattern Recognition 46(6), 1658–1669 (2013). https://doi.org/10.1016/j.patcog.2012.11.024
  * [24] Stutzmann, D., Torres Aguilar, S., Chaffenet, P.: HOME-Alcar: Aligned and Annotated Cartularies (2021). https://doi.org/10.5281/zenodo.5600884
  * [25] Sánchez, J.A., Romero, V., Toselli, A.H., Villegas, M., Vidal, E.: A set of benchmarks for handwritten text recognition on historical documents. Pattern Recognition 94, 122–134 (2019). https://doi.org/https://doi.org/10.1016/j.patcog.2019.05.025
  * [26] Tarride, S., Boillet, M., Kermorvant, C.: Key-value information extraction from full handwritten pages. In: Fink, G.A., Jain, R., Kise, K., Zanibbi, R. (eds.) Document Analysis and Recognition - ICDAR 2023. pp. 185–204. Springer Nature Switzerland, Cham (2023)
  * [27] Tarride, S., Boillet, M., Moufflet, J.F., Kermorvant, C.: Simara: A database for key-value information extraction from full-page handwritten documents. In: Fink, G.A., Jain, R., Kise, K., Zanibbi, R. (eds.) Document Analysis and Recognition - ICDAR 2023. pp. 421–437. Springer Nature Switzerland, Cham (2023)
  * [28] Tarride, S., Lemaitre, A., Coüasnon, B., Tardivel, S.: A comparative study of information extraction strategies using an attention-based neural network. In: International Workshop on Document Analysis Systems. pp. 644–658. Springer (2022)
  * [29] Tarride, S., Maarand, M., Boillet, M., McGrath, J., Capel, E., Vézina, H., Kermorvant, C.: Large-scale genealogical information extraction from handwritten quebec parish records. International Journal on Document Analysis and Recognition (IJDAR) 26(3), 255–272 (Jan 2023). https://doi.org/10.1007/s10032-023-00427-w
  * [30] Toledo, J., Carbonell, M., Fornés, A., Lladós, J.: Information extraction from historical handwritten document images with a context-aware neural model. Pattern Recognition 86 (08 2018). https://doi.org/10.1016/j.patcog.2018.08.020
  * [31] Tüselmann, O., Wolf, F., Fink, G.A.: Are end-to-end systems really necessary for ner on handwritten document images? In: Lladós, J., Lopresti, D., Uchida, S. (eds.) Document Analysis and Recognition – ICDAR 2021. pp. 808–822. Springer International Publishing, Cham (2021), https://doi.org/10.1007/978-3-030-86331-9_52
  * [32] Vidal, E., Toselli, A.H., Ríos-Vila, A., Calvo-Zaragoza, J.: End-to-end page-level assessment of handwritten text recognition. Pattern Recognition 142, 109695 (2023). https://doi.org/10.1016/j.patcog.2023.109695
  * [33] Villanova-Aparisi, D., Martínez-Hinarejos, C.D., Romero, V., Pastor-Gadea, M.: Evaluation of named entity recognition in handwritten documents. In: International Workshop on Document Analysis Systems. pp. 568–582. Springer (2022). https://doi.org/10.1007/978-3-031-06555-2_38
  * [34] Wagner, R.A., Fischer, M.J.: The string-to-string correction problem. Journal of the ACM (JACM) 21(1), 168 – 173 (1974). https://doi.org/10.1145/321796.321811
  * [35] Wang, Z., Xu, Y., Cui, L., Shang, J., Wei, F.: Layoutreader: Pre-training of text and layout for reading order detection. CoRR abs/2108.11591 (2021), https://arxiv.org/abs/2108.11591
  * [36] Xu, Y., Li, M., Cui, L., Huang, S., Wei, F., Zhou, M.: Layoutlm: Pre-training of text and layout for document image understanding. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. p. 1192–1200. KDD ’20, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3394486.3403172
  * [37] Zhang, C., Guo, Y., Tu, Y., Chen, H., Tang, J., Zhu, H., Zhang, Q., Gui, T.: Reading order matters: Information extraction from visually-rich documents by token path prediction. In: Bouamor, H., Pino, J., Bali, K. (eds.) Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 13716–13730. Association for Computational Linguistics, Singapore (Dec 2023). https://doi.org/10.18653/v1/2023.emnlp-main.846
  * [38] Zhang, K., Shasha, D.: Simple fast algorithms for the editing distance between trees and related problems. SIAM Journal on Computing 18(6), 1245–1262 (1989). https://doi.org/10.1137/0218082
  * [39] Zhong, X., ShafieiBavani, E., Jimeno Yepes, A.: Image-based table recognition: Data, model, and evaluation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M. (eds.) Computer Vision – ECCV 2020. pp. 564–580. Springer International Publishing, Cham (2020)

##  7 Annex

###  7.1 Detailed results by category

We present the details of most metrics for each category on Esposalles in
Table 3, on IAM in Table 4, on Simara in Table 5, and on POPP in Table 6.

Table 3: Metrics for each Named Entity category on the test set of the Esposalles database. Category | CER $\downarrow$ | WER $\downarrow$ | Number of entities
---|---|---|---
husband location | 3.1 | 5.4 | 250
husband name | 3.7 | 4.8 | 253
husband occupation | 4.7 | 7.0 | 250
husband state | 10.0 | 8.5 | 65
husband surname | 5.4 | 6.6 | 253
husband’s father location | 0.0 | 0.0 | 12
husband’s father name | 3.4 | 4.1 | 149
husband’s father occupation | 3.0 | 3.6 | 143
husband’s father surname | 5.7 | 7.4 | 143
husband’s mother name | 1.0 | 2.0 | 149
husband’s mother surname | 100.0 | 100.0 | 1
other person location | 100.0 | 100.0 | 2
other person name | 2.1 | 2.9 | 66
other person state | 100.0 | 100.0 | 1
other person surname | 4.0 | 7.3 | 65
wife location | 13.1 | 15.2 | 80
wife name | 0.8 | 2.5 | 253
wife occupation | 2.0 | 1.2 | 62
wife state | 3.0 | 4.3 | 248
wife surname | 43.5 | 50.0 | 3
wife’s father location | 1.6 | 3.1 | 148
wife’s father name | 4.4 | 5.2 | 186
wife’s father occupation | 2.7 | 4.0 | 179
wife’s father surname | 4.05 | 7.0 | 183
wife’s mother name | 0.2 | 1.1 | 184
total (including untagged words) | 0.4 | 1.5 | -
Table 4: Metrics for each Named Entity category on the IAM database. Category | CER $\downarrow$ | WER $\downarrow$ | Number of entities
---|---|---|---
cardinal | 66.0 | 70.0 | 70
date | 72.6 | 73.5 | 59
event | 98.9 | 100.0 | 7
fac | 96.2 | 97.2 | 28
gpe | 64.3 | 78.1 | 74
language | 100.0 | 100.0 | 4
location | 94.6 | 95.1 | 28
money | 100.0 | 100.0 | 2
norp | 74.3 | 79.8 | 44
ordinal | 37.6 | 39.5 | 31
org | 66.9 | 77.7 | 35
per | 63.8 | 76.9 | 211
percent | 68.2 | 75.0 | 1
product | 67.9 | 72.0 | 13
quantity | 97.4 | 98.5 | 19
time | 97.8 | 98.3 | 42
work of art | 93.2 | 97.3 | 38
total (including untagged words) | 5.0 | 15.5 | -
Table 5: Metrics for each Named Entity category on the Simara database. Category | CER $\downarrow$ | WER $\downarrow$ | Number of entities
---|---|---|---
analysis | 9.0 | 22.9 | 771
arrangement | 6.3 | 13.6 | 77
article | 3.1 | 4.2 | 676
serie | 3.1 | 3.2 | 676
date | 1.4 | 2.7 | 751
title | 7.7 | 21.0 | 804
volume | 12.4 | 14.1 | 675
total (including untagged words) | 6.4 | 17.3 | -
Table 6: Metrics for each Named Entity category on the POPP database. Category | CER $\downarrow$ | WER $\downarrow$ | Number of entities
---|---|---|---
birth date | 4.7 | 18.4 | 466
civil status | 14.4 | 15.7 | 225
employer | 14.5 | 34.5 | 396
firstname | 4.7 | 18.4 | 477
link | 14.4 | 15.7 | 408
location of birth | 15.2 | 25.9 | 443
occupation | 14.3 | 36.1 | 331
surname | 6.9 | 28.2 | 369
total (including untagged words) | 8.8 | 23.6 | -

###  7.2 Correlation between the different metrics

We present the correlation matrix between different metrics on all four
datasets. The linear correlation matrix is presented in Table 2, and the rank
correlation matrix in Table 3. Note that we provide absolute correlations, as
the sign of the correlation depends on the nature of the metrics, whether they
are error rates (best at 0) or recognition rates (best at 100).

(a) IAM [31]

(b) Simara [27]

(c) Esposalles [23]

(d) POPP [5]

Figure 2: Absolute linear correlation (Pearson) between the different metrics
across the four datasets. The correlation value appears on each cell, as well
as an indication of its p-value: * indicates a $p-value<0.05$, ** indicates a
$p-value<0.01$, and *** indicates a $p-value<0.001$. No star indicates that
the correlation is not significant.

(a) IAM [31]

(b) Simara [27]

(c) Esposalles [23]

(d) POPP [5]

Figure 3: Absolute rank correlation (Spearman) between the different metrics
across the four datasets. The correlation value appears on each cell, as well
as an indication of its p-value: * indicates a $p-value<0.05$, ** indicates a
$p-value<0.01$, and *** indicates a $p-value<0.001$. No star indicates that
the correlation is not significant.

###  7.3 Qualitative Analysis Of Different Metrics

Finally, we present in Table 7 a qualitative analysis of various metrics
presented in this study across five prediction scenarios on Simara.

Table 7: Qualitative analysis of metrics on specific examples. Label |  <title>COCATRIX (Jean-François) receveur du grenier à sel de Pithivers <analysis>Contre Denis QUIROT <date>26 mai 1770 <serie>X1A <article>4678 <reference>270
---|---
Case 1 |  Almost perfect prediction. As expected, all metrics yield very good scores when there are very few transcription and tagging errors.
|  <title>COCATRIX (Jean-François) receveur du grenier à sel de Pithivers
<analysis>Contre Denis QUIROT <date>26 mai 1770 <serie>X1A <article>4678
<reference>270
Case 2 |  Missing words in a long entity In this scenario, the text errors are concentrated in a single entity, which is consequently not matched. Entity-based metrics such as Nerval and the Bag-of-Entity F1 score are acceptable because only one entity is rejected. However, word-based metrics such as EWER and BoTW are low since many words are missed compared to the total number of words in the document.
|  <title>COCATRIX (Jean-François) <analysis>Contre Denis QUIROT <date>26 mai
1770<serie>X1A <article>4678 <reference>270
Case 3 |  Single word entity missing In this scenario, a single entity (reference) consisting of a single word is missed. In this case, entity-based metrics such as Nerval and the Bag-of-Entity F1 score are acceptable because only one entity is rejected. Word-based metrics such as EWER and BoTW would be better or equal to entity-based metrics because all other words are correctly recognized.
|  <title>COCATRIX (Jean-François) receveur du grenier à sel de Pithivers
<analysis>Contre Denis QUIROT <date>26 mai 1770 <serie>X1A <article>4678
Case 4 |  Uniform distribution of text errors. In this scenario, the text errors are evenly distributed throughout the document. The Nerval F1 will be perfect because all entities will be matched (below the 30% CER threshold). The ECER metric will be in an acceptable range. However, the word-based metrics (EWER, bt-F1, bt-WER) and the entity-based metrics (be-F1, be-WER) will be very low because many words contain at least one character error.
|  <title>COCATRIX (Jean Francois) receveur du greniers à sel de Pithiver
<analysis>Contre Denis QUIROT <date>26 mai 1771 <serie>X1A- <article>4678
<reference>270
Case 5 |  Tagging error. In this scenario, there is no text error, but the model has swapped two entity tags (analysis and title). In this case, all metrics are bad.
|  <analysis>COCATRIX (Jean-François) receveur du grenier à sel de Pithivers
<title>Contre Denis QUIROT <date>26 mai 1770 <serie>X1A <article>4678
<reference>270