# Online Gesture Recognition using Transformer
and Natural Language Processing

###### Abstract

The Transformer architecture is shown to provide a powerful machine
transduction framework for online handwritten gestures corresponding to glyph
strokes of natural language sentences. The attention mechanism is successfully
used to create latent representations of an end-to-end encoder-decoder model,
solving multi-level segmentation while also learning some language features
and syntax rules. The additional use of a large decoding space with some
learned Byte-Pair-Encoding (BPE) is shown to provide robustness to ablated
inputs and syntax rules. The encoder stack was directly fed with spatio-
temporal data tokens potentially forming an infinitely large input vocabulary,
an approach that finds applications beyond that of this work. Encoder transfer
learning capabilities is also demonstrated on several languages resulting in
faster optimisation and shared parameters. A new supervised dataset of online
handwriting gestures suitable for generic handwriting recognition tasks was
used to successfully train a small transformer model to an average normalised
Levenshtein accuracy of 96% on English or German sentences and 94% in French.

Index Terms—  Online Gesture Recognition, Transformer, Multilevel
Segmentation, Language Models, Transfer Learning, Multi-head Attention.

##  1 Introduction

Handwriting Character Recognition (HCR) when associated with touch-sensitive
display panels provides an intuitive and seamless input mechanism eschewing
the need for structured UIs such as virtual keyboards, often slow and error-
prone while also distant to the natural handwriting experience.

In this context, online gesture recognition of glyphs refers to the problem of
mapping a set of user gestures corresponding to sequences of spatio-temporal
samples into their corresponding symbolic representation. Each $n$-dimensional
sample individuates a touch. A coherent and consecutive sequence of touches
defines a stroke that can be combined to form glyphs. Glyphs correspond to
characters or symbols encoded in a language vocabulary. It also extends to a
wider context of symbols drawn from large alphabets composed of glyphs with
many strokes such as logographic systems or mathematical expressions. Table 1
formalises the terminology adopted in this work.

Table 1:  Terminology Term |  Definition
---|---
touch/ point |  $(x,y,[t,p])$ location on touch panel sampled at $t$ with finger pressure $p$
stroke |  Sequence of points where finger consecutively touches the panel
glyph |  List of one or more strokes individuating an element in the vocabulary
symbol |  Item in a vocabulary of 57 symbols $\\{{\tt a}-{\tt z},$ ${\tt A}-{\tt Z},$ ␣,[1200]⟨10ptunk[1200]⟩10pt,[1200]⟨10ptbos[1200]⟩10pt,[1200]⟨10pteos[1200]⟩10pt,[1200]⟨10ptpad[1200]⟩10pt$\\}$ or 2 000 BPE tokens trained from $\\{{\tt en},{\tt fr},{\tt de}\\}$ datasets

These applications must jointly solve a number of tasks, namely i) accurate
feature extraction in a multi-dimensional spatio-temporal space, ii)
segmentation of stroke sequences to identify glyphs, iii) glyph segmentation
for the purpose of word/sentence recognition, and iv) the encoding of syntax
rules and language patterns to form a correct symbolic output. An example of
an online gesture sequence is depicted in Fig. 1.

##  2 Related Work & Contributions

The field of HCR consists of techniques aiming at generating text directly
from handwritten inputs. Most solutions rely on offline data due to dataset
availability [1, 2]. However, the temporal dimension provides some valuable
information that may simplify stroke segmentation, avoiding recourse to
convoluted regression strategies such as text-line segmentation [3]. As a
result, online methods generally exhibit superior performance over offline
counterparts as reported in [4, 5, 6]. With the growing popularity of the
attention mechanism [7, 8], the field remains in constant development with
much effort and resources devoted to improving existing techniques [9, 10]. In
a related sub-problem, Handwritten Mathematical Expression Recognition (HMER)
consists in the generation of mathematical expressions using formal syntaxes,
with state-of-the-art HMER models reaching impressive levels of accuracy,
particularly when exploiting attention [11] and combining modalities from
online and offline data [12]. However, these models fail to learn the
intrinsic structure of expressions. This is somewhat addressed in [13] using
an RNN encoder along with a syntactic tree decoder.

In the context of sequence transduction tasks [14], the Transformer [15]
framework stands as the state-of-the-art on almost all NLP tasks [16, 17, 18,
19], eschewing recourse to recurrent or convolutional units. This powerful
sequence mapping architecture relies entirely on the attention mechanism [7],
allowing for significant parallelisation and unattenuated gradient flow. It
has also been successfully applied to a wider and more generic group of
sequence transduction problems [20, 21, 22, 23, 24]. The Transformer
popularity saw many proposals to revisit and optimise its design [25, 26, 27,
28] but very few are capable of clearly outperforming the original topology.

This work follows the seminal work by [15] and proposes to reformulate the
online gesture recognition problem as a neural transduction task, leveraging
the power of attention in the context of natural languages.

### Main Contributions

  1. (i)

New online datasets are proposed for handwritten text in natural languages
(cf. Section 3) and suitable for a wide range of supervised and unsupervised
machine learning applications.

  2. (ii)

The attention mechanism is shown to successfully learn and represent implicit
structures of spatio-temporal gesture data.

  3. (iii)

The power of transformers is demonstrated not only as language models but also
as a solution to several sequence mapping tasks, with transfer learning
behaviours observed for the encoder111Encoder with frozen parameters pre-
trained on English word dataset can be used on other languages with alphabets
using some potentially unknown glyphs..

  4. (iv)

A small footprint222Despite its small size, model can perform the tasks of
glyph segmentation, character recognition, word segmentation and sentence
construction at remarkable performance levels, learning efficiently the
input/output mapping. topology is proposed as an end-to-end model, with fast
optimisation, high accuracy and suitable for edge inference.

  5. (v)

Model robustness is demonstrated on ablated inputs with the ability to
generate grammatically compliant expressions in case of missing strokes.

  6. (vi)

Multi-level segmentation capability is highlighted in the correlation between
syntactically correct predictions and their explainability in cross-attention
visualisation.

##  3 Datasets

An important contribution of this work is that of online gesture datasets for
text in English (en), French (fr) or German (de), suitable for investigating
the task of HCR but also segmentation of touch, stroke or glyph, and
eventually grammatical/syntactical compliance of expressions in natural
languages. Our handwritten database is presented as a coherent collection of
tables composing a relational schema with spatio-temporal data for Roman
alphabets, Arabic numerals [6], mathematical and punctuation symbols,
collected from volunteers writing on touch panels. This stage saw the
contribution of over 600 subjects for a total of 69 278 labelled glyphs
composed by 93 330 strokes, with over 2 million touches. The dataset can be
used at different levels of granularity, namely touch, stroke and glyph. In
this work, we report results at the stroke level, leaving the burden of glyph
segmentation to the model.

Subjects have been split into training, validation and test sets (60/20/20
proportions) such that models were tested on unseen handwriting styles to
ensure accurate estimation of the generalisation power. When online HCR was
performed in the context of natural languages, the WMT19 news dataset [29] was
used to generate sequences of spatio-temporal gestures from our relational
schema for all available sentences ($\simeq 100\,000$ sentences chosen to be
no longer than 200 input strokes). Pre-processing was applied to substitute or
omit characters where gesture data was unavailable e.g. {é, è, ê}
$\rightarrow$ e in fr or ß $\rightarrow$ ss in de. While these datasets are
much smaller than those used to train large NLP models [30], it was found to
be sufficient to demonstrate the model’s ability to successful learn some
language features at the sentence level.

Fig. 1:  Online gesture example of an input stroke sequence (a) for the
sentence (b) and its corresponding output list – symbols in (c) and BPE tokens
in (d). Cells in (a) depict the linear interpolation of spatio-temporal points
that forms input tokens. Green and red cells denote the
$\stretchto[1200]{\langle}{10pt}$bos$\stretchto[1200]{\rangle}{10pt}$ and
$\stretchto[1200]{\langle}{10pt}$eos$\stretchto[1200]{\rangle}{10pt}$ input
token respectively. Stroke sequence segmentation, glyph segmentation and
parsing of language tokens are colour coded with with blue, green and red,
respectively.

##  4 Transformer Architecture & Experimental Details

Our model [31] leverages the original Transformer [15] architecture. However
crucial modifications are introduced to work with spatio-temporal data. Given
some input sequence, $X\in\mathbb{R}^{d_{\mathrm{f}}\times n}$, of $n$ stroke
tokens defined as interleaved spatio-temporal data with zero-padding of fixed-
length $d_{\mathrm{f}}$ —appropriately prefixed, suffixed and padded at the
end with
$\stretchto[1200]{\langle}{10pt}$eos$\stretchto[1200]{\rangle}{10pt}$,
$\stretchto[1200]{\langle}{10pt}$bos$\stretchto[1200]{\rangle}{10pt}$ and
$\stretchto[1200]{\langle}{10pt}$pad$\stretchto[1200]{\rangle}{10pt}$ tokens
respectively —, a mask $M_{x}$ is computed to ensure encoder’s attention is
only paid to valid online data tokens. Here, $d_{\mathrm{f}}=64\times 2$
allowing for a maximum of 64 $(x,y)$ touch samples per stroke.

As the input is composed of spatio-temporal information corresponding to
touches333Explicit sampling time and pressure $(t,p)$ features are observed to
have no impact on performance suggesting some level of information
redundancy., each encoder token embeds a stroke as $d_{\mathrm{f}}$ scalars
(cf. Fig. 1) resulting in the identification within a potentially unbounded
input vocabulary and therefore eschewing any form of embedding.

Positional encoding provides a strategy to embed the positional information of
input tokens in the encoder, a necessary operation since the attention
mechanism has no built-in concept of sequentiality. Frequency modulation is
proposed in [15]. However, since we observed no performance gain with such a
strategy, we use a learnable 1D embedding based on the token index. Stroke
positions are encoded in $P_{x}\in\mathbb{R}^{d_{\mathrm{f}}\times n}$.

The encoder is trained to learn some latent sequence representation
$Z=\operatorname{Enc}(X+\alpha P_{x},M_{x})\in\mathbb{R}^{d_{\mathrm{a}}\times
d_{\mathrm{h}}\times n}$ where $\alpha$ is a scaling factor blending input
data and positional information, $d_{\mathrm{a}}$ the number of attention
heads (2 to 4) and $d_{\mathrm{h}}$ the hidden state dimension of the
attention heads. The encoder consists of a stack of $l_{\mathrm{e}}$ identical
multi-head vanilla self-attention layers and a positional feed-forward network
of hidden dimension $d_{\mathrm{p}}$. Each layer is followed by a residual
connection before layer-normalisation.

When exploring transfer learning capabilities, an encoder with
$\Theta_{\mathrm{e}}\\!=\\!523\,520$ parameters is used as a feature extractor
resulting in a considerable speed-up during training and model optimisation.

The decoder generates a causal sequence of tokens in an auto-regressive manner
given some vocabulary and relative token encoding. It is initialised with the
$\stretchto[1200]{\langle}{10pt}$bos$\stretchto[1200]{\rangle}{10pt}$ token
and iteratively outputs a new token using greedy sampling of the decoder’s
softmax output until the
$\stretchto[1200]{\langle}{10pt}$eos$\stretchto[1200]{\rangle}{10pt}$ token is
predicted or the maximum sequence length, $m$, is reached. The decoder also
consists of $l_{\mathrm{d}}$ identical layers each composed by: i) a masked
self-attention layer (with $d_{\mathrm{a}}$ attention heads of hidden state
dimension $d_{\mathrm{h}}$) that prevents the decoder from peeking at the
subsequent tokens, ii) a cross-attention layer that attends over the encoder
output $Z$ to generate predictions, and iii) a feed-forward layer just as in
the encoder but of dimension $kd_{\mathrm{p}}$ where $k\\!=\\!1$ to $3$.

At each step, the decoder’s input, $Y_{i-1}$, is an auto-regressive sequence
of tokens adequately masked with $M_{y}$ and used to predict the next token of
the output sequence: $Y_{i}=\operatorname{Dec}(Y_{i-1},M_{y},Z)$. All
$\Theta_{\mathrm{d}}$ parameters of the decoder were trained from some
randomly initialised state.

Experimental details: All models were configured with
$d_{\mathrm{f}}\\!=\\!d_{\mathrm{a}}\times
d_{\mathrm{h}}\\!=\\!d_{\mathrm{p}}\\!=\\!128$. For
v$\,{}^{\texttt{\;\;\;rnd}}_{\\!61}$–v$\,{}^{\texttt{\;\;\;gow}}_{\\!68}$,
$n\\!=\\!2m\\!=\\!48$ and $k\\!=\\!1$. For
v$\,{}^{\texttt{\;\;\;en}}_{\\!74}$–v$\,{}^{\texttt{\;\;\;de}}_{\\!93}$,
$n\\!=\\!2m\\!=\\!200$ and $k\\!=\\!3$. Parameters of the attention layer
stack are detailed Tables 2 and 3. Models were trained on Nvidia TitanX
GPUs444Nvidia is acknowledged for the donation of GPUs, for a maximum of 5000
epochs, a 256 batch size, using cross-entropy loss and Adam optimiser with a
decay schedule (initial learning rate of $8\times 10^{-4}$ and halving every
30 epochs).

##  5 Experimental Results

A series of experiments were carried out to determine suitable model hyper-
parameters and investigate the benefits of increasing the decoder’s output
dimension using vocabularies (vocabs) of various sizes: i) a small vocab of 57
alphabetic symbols, and ii) larger vocabs with 2 000 Byte-Pair-Encoding (BPE)
symbols trained on three natural languages (en, fr, de). Models were evaluated
using a number of performance metrics on the test sets and results are
reported in terms of Cross-Entropy Loss (XEL) and two edit distance metrics,
namely: normalised Levenshtein distance [32] Accuracy (LA) and Character Error
Rate (CER).

Hyper-parameters were determined from XEL results obtained on models trained
with pseudorandom letter words as reported in Table 2. Since this only
required to solve the tasks of glyph segmentation and classification,
splitting hidden states in two multi-attention heads was sufficient. A five-
layer encoder abstraction performed best, a result consistent with previous
observations in an online classification context using convolutional
topologies [6]. Note that there are no benefits in increasing the decoder’s
layer abstraction since output tokens are i.i.d. in a small 57 symbol vocab,
with no symbol patterns to be exploited.

Table 2:  Hyper-parameter search for models trained on stroke sequence
corresponding to pseudorandom letter words.

Performance is reported in terms of Cross-Entropy Loss (XEL).

Name |  Enc ($l_{\mathrm{e}},d_{\mathrm{a}}$) |  Dec ($l_{\mathrm{d}},d_{\mathrm{a}}$) | $\Theta_{\mathrm{e}}+\Theta_{\mathrm{d}}$ | XEL
---|---|---|---|---
v$\,{}^{\texttt{\;\;\;rnd}}_{\\!61}$ | 2L, 4H | 2L, 4H | 554 552 | 0.761
v$\,{}^{\texttt{\;\;\;rnd}}_{\\!62}$ | 3L, 2H | 2L, 2H | 654 136 | 0.881
v$\,{}^{\texttt{\;\;\;rnd}}_{\\!63}$ | 4L, 4H | 4L, 4H | 1 085 496 | 0.914
v$\,{}^{\texttt{\;\;\;rnd}}_{\\!64}$ |  5L, 2H |  2L, 2H | 850 232 | 0.690
v$\,{}^{\texttt{\;\;\;rnd}}_{\\!65}$ | 5L, 2H | 4L, 2H | 1 185 080 | 1.556
v$\,{}^{\texttt{\;\;\;rnd}}_{\\!66}$ | 7L, 2H | 2L, 2H | 1 052 472 | 0.895
v$\,{}^{\texttt{\;\;\;rnd}}_{\\!67}$ | 7L, 2H | 4L, 2H | 1 384 248 | 1.332
Table 3:  Model performance reported in term of Cross-Entropy Loss (XEL),
normalised Levenshtein distance Accuracy (LA) and Character Error Rate (CER).
v$\,{}^{\texttt{\;\;\;gow}}_{\\!68}$ trained on a group of en words (57-symbol
vocab),
v$\,{}^{\texttt{\;\;\;en}}_{\\!80}$–v$\,{}^{\texttt{\;\;\;de}}_{\\!93}$
trained on sentences in three languages (2 000 BPE vocab).
$\Theta_{\mathrm{e}}=523\,520$ (all), $\Theta_{\mathrm{d}}=330\,041$
(v$\,{}^{\texttt{\;\;\;gow}}_{\\!68}$) and $1\,453\,520$ (others). Fine-tuned
models provide best performance.

Name |  Enc ($l_{\mathrm{e}},d_{\mathrm{a}}$) |  Dec ($l_{\mathrm{d}},d_{\mathrm{a}}$) | XEL | LA(%) | CER
---|---|---|---|---|---
v$\,{}^{\texttt{\;\;\;gow}}_{\\!68}$ |  5L, 4H‡ |  2L, 4H‡ | 0.282 | 91.07 | 0.089
v$\,{}^{\texttt{\;\;\;en}}_{\\!74}$ |  5L, 4H‡ |  4L, 4H‡ | 0.260 | 94.04 | 0.066
v$\,{}^{\texttt{\;\;\;en}}_{\\!80}$ |  5L, 4H∗ |  4L, 4H‡ | 0.183 | 95.90 | 0.045
v$\,{}^{\texttt{\;\;\;fr}}_{\\!82}$ |  5L, 4H∗ |  4L, 4H‡ | 0.313 | 93.21 | 0.074
v$\,{}^{\texttt{\;\;\;de}}_{\\!83}$ |  5L, 4H∗ |  4L, 4H‡ | 0.285 | 94.79 | 0.057
v$\,{}^{\texttt{\;\;\;fr}}_{\\!92}$ |  5L, 4H† |  4L, 4H† | 0.293 | 93.51 | 0.071
v$\,{}^{\texttt{\;\;\;de}}_{\\!93}$ |  5L, 4H† |  4L, 4H† | 0.298 | 94.52 | 0.060

‡ Trained from scratch from some randomly initialised state † Fined-tuned
parameters using transfer learning of v$\,{}^{\texttt{\;\;\;en}}_{\\!74}$
encoder ∗ Frozen parameters using transfer learning of
v$\,{}^{\texttt{\;\;\;en}}_{\\!74}$ encoder

Table 3 summarises the main results with hyper-parameter configuration and
performance evaluation for models trained on natural languages. Since this
requires solving multilevel segmentation tasks, the hidden states of a 4-layer
decoder abstraction were split into 4 attention heads. Model
v$\,{}^{\texttt{\;\;\;en}}_{\\!68}$ was trained on short en sentence chunks to
output symbols using the 57 symbol vocab, with some modest data augmentation
strategies using affine transformations. Despite its smaller size, model
demonstrates strong end-to-end recognition capabilities, learning some syntax
rules. Larger models
($\Theta_{\mathrm{e}}\\!+\\!\Theta_{\mathrm{d}}\\!=\\!\\!1.9\,$M) were
subsequently trained on full sentences in different languages.
v$\,{}^{\texttt{\;\;\;en}}_{\\!74}$ was optimised from a random state with the
same output configuration, exhibiting some improved performance over the
smaller v$\,{}^{\texttt{\;\;\;gow}}_{\\!68}$ model. Its encoder parameters
were then used as a feature extractor when training larger models outputting
BPE tokens (en, fr, de) with a significant optimisation speed-up. With a
performance reaching a LA of 96% 1(4.5% CER, 14.7% WER), the en model’s
improvement is attributed to the larger decoder output dimension that provides
some error correction mechanisms as confirmed in the robustness analysis of
Table 4, eliminating some spelling errors. Although the encoder was optimised
on inputs associated with en words, the de model
v$\,{}^{\texttt{\;\;\;de}}_{\\!83}$ fell just 0.4% short of the en model
accuracy, with strong performance also observed with the fr model
v$\,{}^{\texttt{\;\;\;fr}}_{\\!82}$ tailing by 1.5%. It is worth noting that
fine-tuning of the fr model over a couple of epochs resulted in modest
performance gains.

### Robustness and Visualisation

Model robustness is investigated with input stroke ablation and deliberate
erroneous input while observing the model’s ability to enforce syntax rules
and correct spelling.

As all dataset expressions end with some punctuation mark, the learning of
this rule is observed in all models as shown in Table 4. In addition,
interrogative forms are detected from the subject/verb inversion, with correct
insertion of a question mark as shown in the fr example  2. One can also note
the correct hyphenation token inferred between verb and subject pronoun as
expected in fr syntax, along with spacing before the question mark. The large
BPE vocab also provides some robustness to spelling errors as shown in the de
example  3. This was not observed to the same extent on models with smaller
output vocab such as v$\,{}^{\texttt{\;\;\;en}}_{\\!68}$. Finally, contraction
of en verbs is correctly detected and expanded as seen in the en example  1.
These observations demonstrate that the models are capable of learning non-
trivial valuable syntax and grammar rules along with some language features
despite the small dataset and model size.

Table 4: Robustness to elided input, spelling errors, and missing punctuation.
BPE models can infer correct expressions. Input ($X$) for glyph symbols ($S$)
and inference output ($\hat{Y}$)
---
$X\\!\\!=$
$S\\!=\\!\left\\{\texttt{I,t,',s,a,t,t,e,n,t,i,o}\right\\}$
v$\,{}^{\texttt{\;\;\;en}}_{\\!68}$
$\rightarrow\hat{Y}\\!\\!=\\!\left\\{\texttt{I,t,{\color[rgb]{0,0.7,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.7,0}{\textvisiblespace}}{},{\color[rgb]{0,0.7,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.7,0}{i}},s,{\color[rgb]{0,0.7,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.7,0}{\textvisiblespace}}{},a,t,t,e,n,t,i,o,{\color[rgb]{0,0.7,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.7,0}{n}}}\right\\}$
v$\,{}^{\texttt{\;\;\;en}}_{\\!80}$
$\rightarrow\hat{Y}\\!\\!=\\!\left\\{\texttt{It,{\color[rgb]{0,0.7,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.7,0}{\textvisiblespace}}{}{\color[rgb]{0,0.7,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.7,0}{i}}s,{\color[rgb]{0,0.7,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.7,0}{\textvisiblespace}}{}att,entio{\color[rgb]{0,0.7,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.7,0}{n}},{\color[rgb]{0,0.7,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.7,0}{.}}}\right\\}$
1
$X\\!\\!=$
$S\\!=\\!\left\\{\texttt{E,s,t,c,e,u,n,e,q,u,e,s,t,i}\right\\}$
v$\,{}^{\texttt{\;\;\;fr}}_{\\!82}$
$\rightarrow\hat{Y}\\!\\!=\\!\left\\{\texttt{Est,{\color[rgb]{0,0.7,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.7,0}{-}},ce,{\color[rgb]{0,0.7,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.7,0}{\textvisiblespace}}{}une,{\color[rgb]{0,0.7,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.7,0}{\textvisiblespace}}{}questi{\color[rgb]{0,0.7,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.7,0}{on}},{\color[rgb]{0,0.7,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.7,0}{\textvisiblespace}}{}{\color[rgb]{0,0.7,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.7,0}{?}}}\right\\}$
2
$X\\!\\!=$
$S\\!=\\!\left\\{\scalebox{1.0}[1.0]{{E,s,i,s,d,o,c,h,A,u,f,m,e,r,k,s,e,m,k,e,i,t}}\right\\}$
v$\,{}^{\texttt{\;\;\;de}}_{\\!83}$
$\rightarrow\hat{Y}\\!\\!=\\!\left\\{\scalebox{1.0}[1.0]{{Es,{\color[rgb]{0,0.7,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.7,0}{\textvisiblespace}}{}is{\color[rgb]{0,0.7,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.7,0}{t}},{\color[rgb]{0,0.7,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.7,0}{\textvisiblespace}}{}doch,{\color[rgb]{0,0.7,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.7,0}{\textvisiblespace}}{}Auf,mer,k,s{\color[rgb]{0,0.7,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.7,0}{a}}m,keit,{\color[rgb]{0,0.7,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.7,0}{.}}}}\right\\}$
3

Visualisation of the attention mechanisms provides some interesting insights
in the learning process. Fig. 2 depicts weights of the decoder’s cross-
attention over the encoder’s output. It shows that head 3 of layer 4 is key to
token segmentation with strong attention paid to the first token stroke
solving the word segmentation task. The large BPE space is also leveraged for
the auto-completion of the final text token, while insertion of the
punctuation mark is carried out by tracking the final stroke. The
$\stretchto[1200]{\langle}{10pt}$eos$\stretchto[1200]{\rangle}{10pt}$ token
output is seen focusing on both final stroke and
$\stretchto[1200]{\langle}{10pt}$eos$\stretchto[1200]{\rangle}{10pt}$ stroke
input.

Fig. 2:  Cross-attention plot of head 3–layer 4 for model
v$\,{}^{\texttt{\;\;\;en}}_{\\!80}$ showing output tokens tracking its first
stroke. Question mark token is focusing on input  token.

##  6 Conclusion

This work posed the problem of online HCR as a transduction task, using a
Transformer framework to learn the complex mapping of online input gesture
data corresponding to handwritten strokes of natural language sentences. The
encoder’s input was modified to receive spatio-temporal data as real-valued
tokens, operating at stroke level without the need for mapping on a fixed
input vocabulary. Models were shown to predict text with very high accuracy by
handling internally the multi-level segmentation of inputs (at glyph and word
levels), and also understanding and learning how to represent and enforce
syntactic and semantic rules of data. Index positional encoding was shown to
be as effective as cosine modulation yet standing as a simpler and more
natural encoding for the position information. Although the Transformer’s
ability to generate complex representations and learn non-trivial input/output
mapping between sequences is well established [16, 20], the challenge was
further pushed in this work in the absence of ad hoc syntax, semantic rules,
or engineered loss computation and architecture.

In addition, an encoder trained on a specific language was successfully used
as a frozen feature extractor in the optimisation of decoders in several other
language domains. Such transfer learning capabilities suggest that pre-trained
encoders can create general latent representations suitable for problems of
different nature, resulting in model size reduction, training acceleration
with no need for fine-tuning or explicit domain adaptation. This will also
benefit applications where computational power/time and dataset size are
limited.

The objective of this work was not so much to push out some state-of-the-art
model but rather to state some important considerations that may be the
starting points for future works on sequentiality of other signal types.
Neural transduction may be extended in this way to online data at different
granularity levels, with no need for separate input segmentation or complex
positional embeddings. With larger language datasets and computational
resources, this approach may reveal deeper language modelling capabilities to
similar levels observed in BERT or GPT [16, 30], and this straight from
digital signals. The end-to-end encoder-decoder models in this work achieved a
normalised Levenshtein accuracy of 94% to 96% at sentence levels on the three
languages considered and directly from online data.

## References

  * [1] R. Plamondon and S.N. Srihari,  “Online and off-line handwriting recognition: a comprehensive survey,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 1, pp. 63–84, 2000.
  * [2] Deepak Sinwar, Vijaypal Singh Dhaka, Pradhan, et al.,  “Offline script recognition from handwritten and printed multilingual documents: a survey,”  International Journal on Document Analysis and Recognition (IJDAR), vol. 24, no. 1, pp. 97–121, 2021.
  * [3] Berat Barakat, Ahmad Droby, Majeed Kassis, and Jihad El-Sana,  “Text line segmentation for challenging handwritten document images using fully convolutional network,”  in 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), 2018, pp. 374–379.
  * [4] JinHyung Kim and Bong-Kee Sin,  Handbook of Document Image Processing and Recognition, chapter Online Handwriting Recognition, pp. 887–915,  Springer London, 2014.
  * [5] Anchit Shrivastava, Isha Jaggi, Gupta, et al.,  “Handwritten digit recognition using machine learning: A review,”  in 2019 2nd International Conference on Power Energy, Environment and Intelligent Control (PEEIC), 2019, pp. 322–326.
  * [6] Philip J Corr, Guenole C Silvestre, and Chris J Bleakley,  “Open source dataset and deep learning models for online digit gesture recognition on touchscreens,”  in 2017 Irish Machine Vision and Image Processing Conference (IMVIP), 2017.
  * [7] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio,  “Neural machine translation by jointly learning to align and translate,”  in International Conference on Learning Representations (ICLR), Jul 2015.
  * [8] Jason Poulos and Rafael Valle,  “Character-based handwritten text transcription with attention networks,”  Neural Computing and Applications, vol. 33, no. 16, pp. 10563–10573, 2021.
  * [9] Daniel Keysers, Thomas Deselaers, Rowley, et al.,  “Multi-language online handwriting recognition,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, pp. 1180–1194, 2017.
  * [10] Alex Graves,  “Generating sequences with recurrent neural networks,”  arXiv preprint,  https://doi.org/10.48550/arXiv.1308.0850, 2013.
  * [11] Zhe Li, Lianwen Jin, Lai, et al.,  “Improving attention-based handwritten mathematical expression recognition with scale augmentation and drop attention,”  in 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), 2020, pp. 175–180.
  * [12] Jiaming Wang, Jun Du, Zhang, et al.,  “Multi-modal attention network for handwritten mathematical expression recognition,”  in 2019 International Conference on Document Analysis and Recognition (ICDAR), 2019, pp. 1181–1186.
  * [13] Jianshu Zhang, Jun Du, Yang, et al.,  “SRD: A tree structure based decoder for online handwritten mathematical expression recognition,”  IEEE Transactions on Multimedia, vol. 23, pp. 2471–2480, 2021.
  * [14] Ilya Sutskever, Oriol Vinyals, and Quoc V Le,  “Sequence to sequence learning with neural networks,”  arXiv preprint,  https://doi.org/10.48550/arXiv.1409.3215, 2014.
  * [15] Ashish Vaswani, Noam Shazeer, Parmar, et al.,  “Attention is all you need,”  in Advances in neural information processing systems, 2017, vol. 30, pp. 6000–6010.
  * [16] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova,  “BERT: Pre-training of deep bidirectional transformers for language understanding,”  in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics. Jun 2019, pp. 4171–4186, Association for Computational Linguistics.
  * [17] Tom Brown, Benjamin Mann, Ryder, et al.,  “Language models are few-shot learners,”  Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.
  * [18] Thomas Wolf, Lysandre Debut, Sanh, et al.,  “Transformers: State-of-the-art natural language processing,”  in Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, 2020, pp. 38–45.
  * [19] Ankush Garg and Mayank Agarwal,  “Machine translation: a literature review,”  arXiv preprint,  https://doi.org/10.48550/arXiv.1901.01122, 2018.
  * [20] Niki Parmar, Vaswani, et al.,  “Image transformer,”  in International conference on machine learning. PMLR, 2018, pp. 4055–4064.
  * [21] Cheng-Zhi Anna Huang, Ashish Vaswani, et al.,  “Music transformer: Generating music with long-term structure,”  in International Conference on Learning Representations (ICLR), 2019\.
  * [22] Hengshuang Zhao, Li Jiang, Jia, et al.,  “Point transformer,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 16259–16268.
  * [23] Alexander Kozlov, Vadim Andronov, and Yana Gritsenko,  “Lightweight network architecture for real-time action recognition,”  in Proceedings of the 35th Annual ACM Symposium on Applied Computing, 2020, pp. 2074–2080.
  * [24] Andrea D’Eusanio, Alessandro Simoni, Pini, et al.,  “A transformer-based network for dynamic hand gesture recognition,”  in 2020 International Conference on 3D Vision (3DV), 2020, pp. 623–632.
  * [25] Sinong Wang, Belinda Z Li, Khabsa, et al.,  “Linformer: Self-attention with linear complexity,”  arXiv preprint,  https://doi.org/10.48550/arXiv.2006.04768, 2020.
  * [26] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya,  “Reformer: The efficient transformer,”  in International Conference on Learning Representations, 2020.
  * [27] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, et al.,  “Rethinking attention with performers,”  in International Conference on Learning Representations, 2021.
  * [28] Roshan M Rao, Jason Liu, Verkuil, et al.,  “MSA transformer,”  in International Conference on Machine Learning. PMLR, 2021, pp. 8844–8856.
  * [29] Wikimedia Foundation,  “Shared task (WMT19): Machine translation of news,” ACL 2019,  http://www.statmt.org/wmt19/translation-task.html, 2019.
  * [30] Alec Radford, Jeffrey Wu, Rewon Child, et al.,  “Language models are unsupervised multitask learners,”  OpenAI blog, vol. 1, no. 8, pp. 9–20, 2019.
  * [31] Olumide Akinremi,  “Is this all about attention?,”  M.S. thesis, University College Dublin, May 2021.
  * [32] Li Yujian and Liu Bo,  “A normalized Levenshtein distance metric,”  IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 29, no. 6, pp. 1091–1095, 2007.


# Harvesting Textual and Structured Data from the HAL
Publication Repository

Francis Kulumba1, 2 Wissam Antoun1,2
Guillaume Vimont3 Laurent Romary1
1Inria, Paris, France 2Sorbonne Université, Paris, France 3IRIF, Paris, France
{francis.kulumba, wissam.antoun<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

HAL (Hyper Articles en Ligne) is the French national publication repository,
used by most higher education and research organizations for their open
science policy. As a digital library, it is a rich repository of scholarly
documents, but its potential for advanced research has been underutilized. We
present HALvest, a unique dataset that bridges the gap between citation
networks and the full text of papers submitted on HAL. We craft our dataset by
filtering HAL for scholarly publications, resulting in approximately 700,000
documents, spanning 34 languages across 13 identified domains, suitable for
language model training, and yielding approximately 16.5 billion tokens (with
8 billion in French and 7 billion in English, the most represented languages).
We transform the metadata of each paper into a citation network, producing a
directed heterogeneous graph. This graph includes uniquely identified authors
on HAL, as well as all open submitted papers, and their citations. We provide
a baseline for authorship attribution using the dataset, implement a range of
state-of-the-art models in graph representation learning for link prediction,
and discuss the usefulness of our generated knowledge graph structure.

Harvesting Textual and Structured Data from the HAL
Publication Repository

Francis Kulumba1, 2 Wissam Antoun1,2 Guillaume Vimont3 Laurent Romary1 1Inria,
Paris, France 2Sorbonne Université, Paris, France 3IRIF, Paris, France
{francis.kulumba, wissam.antoun<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Dataset | References | Full-text | Multilingual | Multi-domain
---|---|---|---|---
Curated Graph Datasets
ArnetMiner (Tang et al., 2012) | ✓ |  |  |
Large Graph Datasets
MAG (Wang et al., 2020) | ✓ |  |  | ✓
OpenAlex (Priem et al., 2022) | ✓ | ✓ | ✓ | ✓
Ours
HALvest | ✓ | ✓ | ✓ | ✓
Table 1: Comparison of information provided by HALvest to previous academic
graph datasets. References denote the presence of a citation list for a given
paper. Figure 1: HALvest’s citation network: a directed heterogeneous graph
with 4 node types and 4 edge types.

Publication repositories are the norm when storing and distributing scholarly
papers openly in a sovereign and sustainable way. When submitting a paper to a
repository, the depositor, an identified user, possesses a unique id in the
repository’s database. However, the depositor’s co-authors might not be
identified users, letting the repository perform id retrieval to map the
submission to the correct authors. Authorship attribution without human
interventions remains a non-trivial task to publication repositories (Tekles
and Bornmann, 2019). Researchers rapidly apprized the importance of co-
authorship to tackle authorship attribution, hence integrated structured data
to their proposed solutions (Shin et al., 2014; Ma et al., 2020; Xie et al.,
2022). However, the shortcomings of graph-based algorithms, in the absence of
co-authorship, drove researchers to exploit semantic and symbolic features
from papers (Tran et al., 2014; Müller, 2017; Kim et al., 2019; Cohan et al.,
2020; Boukhers et al., 2021; Pooja et al., 2021; Kojaku et al., 2021) and
investigating the semantic fingerprint from individuals (Han et al., 2017;
Zhai et al., 2019).

Recently, with the research in multimodal deep-learning mushrooming, several
architectures allow practitioners to now benefit from text and structured data
(Zhang et al., 2019, 2022; Yasunaga et al., 2022), thus exploiting citation
networks as well as semantic content from scholarly papers (Pooja et al.,
2022; Santini et al., 2022). By opening the data from Hyper Articles en Ligne
(HAL), we aim to enable researchers to implement and validate new methods
towards better multimodal architectures—exploiting graphs and text—and in an
ancillary manner tackling authorship identification. We introduce HALvest, a
structured and textual dataset: the structured part, called HALvest-Geometric,
is a heterogeneous citation network comprising 238,397 author nodes and
18,662,037 paper nodes, with 642,723 of them having their full-text available.
Furthermore, the gathered papers allow more than 16.5 billion tokens from
scholarly text, allowing for better language modeling in a given domain.

To validate HALvest’s suitability to not only train language models but also
graph representation learning architectures, we conduct a straightforward
experiment: we test several state-of-the-art graph neural networks (Scarselli
et al., 2009), that can fit heterogeneous graphs, to provide a baseline for
authorship attribution in a closed set-up, while also validating our graph’s
format.

Our contribution can be summarized as follows

  * •

A textual dataset comprising 17 billion tokens in 56 languages and 13 domains.

  * •

An academic citation network with 238,397 disambiguated authors and 18,662,037
scholarly papers.

The datasets 111https://huggingface.co/datasets/Madjakul/HALvest
222https://huggingface.co/datasets/Madjakul/HALvest-Geometric, as well as the
code used to craft them 333https://github.com/Madjakul/HALvesting
444https://github.com/Madjakul/HALvesting-Geometric are available online and
will be actively maintained and updated.

##  2 Related Work

##### Scholarly textual data

The rise in popularity of language modeling in natural language processing,
coupled with online publication repositories being an integral part of the
researcher’s toolkit, allowed the release of a non-negligible amount of
textual data. Although full-texts are not bound to the same licensing as
abstracts, publication libraries in science, technology, engineering, and math
are often at the forefront when it comes to opening their data when possible.
The likes of ArXiv 555https://arxiv.org/ and DBLP 666https://dblp.org/ PubMed
777https://pubmed.ncbi.nlm.nih.gov/download/ are allowing their publication’s
titles and abstract to be crawled (arXiv.org submitters, 2024; Wahle et al.,
2022; Sen et al., 2008; Doğan et al., 2014) while Semantic Scholar
888https://www.semanticscholar.org/ and a subset of PubMed (PMC)
999https://www.ncbi.nlm.nih.gov/pmc/, allow for a full-texts’ crawling (Ammar
et al., 2018; Lee et al., 2020) of their data. ACL
101010https://www.aclweb.org/portal/, on the other hand, opted for a cleaned
subset of 10,920 academic papers from ACL Anthology (Bird et al., 2008).
However, the aforementioned repositories are either limited in size or domains
(e.g., STEM for ArXiv, computer science for DBLP, and bio-medical for PubMed),
hence Lo et al. (2020) merged various sources in order to alleviate these
issues, while also offering full-texts from open some papers. Academic
unstructured data is also built and maintained by researchers’ communities as
seen in BC5CDR (Li et al., 2016) and NCBI (Doğan et al., 2014), and MEDLINE
(Vishnyakova et al., 2019) in the biomedical domain. All the aforementioned
datasets are confined to the English language and featured publications or
preprints, while the unstructured half of HALvest provides historical texts,
thesis, and dictionaries, for example, being more lengthy and domain-diverse.
Besides, the full texts are uniform in their formatting and referencing style.

##### Academic graph

When it comes to academic graph, ArnetMiner (Tang et al., 2008, 2012) and
Microsoft Academic Graph (MAG)(Wang et al., 2020) are often the go-to,
comprising of metadata from DBLP and ACM 111111https://dl.acm.org/. MAG,
unlike ArnetMiner, has not been manually curated but allows for a rich pool of
papers and authors (736,389 papers, 1,134,649 authors 8,740 institutions, and
59,965 fields of study nodes). One can also look upon OpenAlex (Priem et al.,
2022), offering more than 220 million publication nodes, and the graph
proposed by Ammar et al. (2018) to enable semantic features in Semantic
Scholar. All the graphs mentioned above come with various metadata, including
abstracts.

##### Combined Text and academic knowledge graph

To the best of our knowledge, the only ready-to-use academic graph dataset
mapped with full texts is from OpenAlex (Priem et al., 2022). However, ACL
(Cohan et al., 2019), and Lee et al. (2020)’s datasets can be easily mapped to
a citation network with little processing. Despite providing the full text of
every paper’s node, ACL and Lee et al. (2020) graphs are limited in size and
domain (computational linguistic and biomedical respectively).

##  3 Collection Methodology

Crafting HALvest is a four-step process. We start by fetching open pdf files
from HAL, before using GROBID (GROBID Repository, ) to obtain xml files and
easing the conversion to txt documents. We filter the French and English txt
files out of poorly encoded text, and use the remaining documents to build the
citation network. In the following, we introduce our pipeline in more detail.

###  3.1 Fetching data from HAL

HAL’s API allows for precise requests to the repository. One can acquire
structured xml-tei responses of each submitted document. Exploiting xml-teis
allows for standardized access to all the metadata available on HAL, and
therefore, more control over the desired document representation. We settled
for a list of eight features to represent submissions:

  * •

halid: submission’s unique identifier assigned by HAL.

  * •

lang: language of the document, as filled by the depositor.

  * •

title title of the document.

  * •

domain: list of field of studies 121212https://hal.science/browse/domain.

  * •

timestamp: time of access.

  * •

year: publication year of the document if relevant. Otherwise, it is set to
year 1.

  * •

url: URL to access the pdf.

  * •

authors: list of authors.

The author is comprised of at most nine features:

  * •

name: string for the author name, as filled by the depositor.

  * •

affiliations: list of unique identifiers attributed by HAL to the institutions
where the author belongs.

  * •

halauthorid: unique identifier assigned by HAL to each author registered on
the online repository. If an author is not registered to HAL, he is considered
unidentified and is assigned an halauthorid of "0".

  * •

Potentially six external identifiers, if provided by the registered authors,
comprising of arxiv, researcherid, idref, orcid, viaf and isni.

We design our query to only fetch submissions with an attached document in
open access, and apply further filtering to only fetch work submitted
alongside a pdf file. As of February 2024, the total number of open submitted
pdf files is 778,072.

We use GROBID GROBID Repository  to derive xml files from pdfs, hence easing
the conversion to plain text afterward. Reference markers in each document are
serialized following the scheme defined by Taylor et al. (2022). However,
math, SMILES formulas, as well as code, are encoded in plain utf-8 and left as
is.

###  3.2 Filtering

During the conversion process, if the fonts within a pdf lack Unicode tables
and do not employ standard encoding for mapping glyph indices to characters,
GROBID’s output ends up being gibberish (§ A.2), as the latter do not employ
optical character recognition to extract text from pdfs, but rather its
layout. To filter out gibberish documents, we use a set of heuristic
functions. Following Raffel et al. (2020); Wenzek et al. (2020); Rae et al.
(2022) and Penedo et al. (2023), we compute a set of statistics about each
document, effectively getting metrics like the number of lines, the average
word length or the ratio of unique words. We use the implementation provided
by RedPajama Repository . To compute the ratio of stop-words in a document, we
use stopwords-json (6, 2024).

We post-process the remaining documents, written in 34 languages, as follows :

  1. 1.

Documents with less than 3 words are discarded.

  2. 2.

Documents with more than 10% of words that are capitalized are discarded.

  3. 3.

Documents with more than 60% of words that are not alphanumeric are discarded.

  4. 4.

Documents with an average word length of 1.5 characters or below are
discarded.

  5. 5.

Document with no stop words are discarded. Stop words are strong indicators
for well-redacted documents, hence, lowering the chance of it being gibberish.
Besides, we use stop words as a language identifier, as the language provided
in the metadata is specified by a depositor, and human error can be
introduced.

  6. 6.

We compute the inverse fertility (Rust et al., 2021): the number of words in a
document divided by the number of tokens yielded by the mT5 tokenizer (Xue et
al., 2021). After removing special tokens, a tokenizer yields at most an
amount of tokens equal to the number of words, effectively bounding our
function between 0 and 1. An inverse fertility score close to zero is a strong
indication of over tokenization, and therefore, hinting at a gibberish
document. Documents with an inverse fertility score lower than 0.2 are
discarded.

###  3.3 From metadata to citation network

Building the citation network is straightforward, though, computationally
demanding. We define four node types, and four edge types, before iterating
through the metadata of each document. Proceeding this way leads to several
shortcomings: the computed graph represents a snapshot of HAL at a given time,
remaining static, leading to affiliations being cumulative—affiliation nodes
do not represent the affiliations of an author at a given time, but rather all
the institutions he has been part of. To alleviate those issues, we provide
optional features for the author and paper nodes, allowing practitioners to
only compute edges for a subset of the graph, based on a year or a language.

When it comes to domain identification, HAL provides a tree-like domain space,
with 13 domains, branching into several subdomains, and an additional one:
"other". We fetch the domains filled by the depositors and use the root node
of each one. We obtained 16 domains, as some depositors only provide a
subdomain that overlaps between two parent domains.

Regardless of how we compensate for the lack of precision in some of the
graph’s relations, computing the citations remains fuzzy. We retrieve each
document’s references through their xml document, collect their title and
publication year, and then use exact matching. However, this process can
induce inaccuracies in the title retrieved, as GROBID parsing can be inexact.

##  4 Dataset Composition

In the following section, we provide more details concerning the composition
of the two halves of HALvest: the unstructured part with text, and the
structured part with a heterogeneous citation network.

###  4.1 Unstructured data

Key |  Value
---|---
halid |  01744328
lang |  en
domain |  [ "info.info-ai" ]
timestamp |  2024/03/05 22:32:07
year |  2017
url |  https://hal.science/hal-01744328/file/ertek_chi_zhang_2017_RFID.pdf
text |  A Framework for Mining RFID Data From Schedule-Based Systems Gürdal Ertek, Xu Chi, Member, IEEE, and Allan N. Zhang Member, IEEE Abstract-A schedule-based system is a system that operates on or contains within a schedule of events and breaks at particular time intervals…

Table 2: Examples of text documents from HALph.

Although HALvest is mostly in English and French, the gathered 670,861 papers
are written in 56 languages across 16 domains for the unfiltered version,
accounting for approximately 17 billion tokens. HALvest’s text can also serve
as a valuable asset for low-resource languages, hosting documents in Basque,
Catalan, or Persian to mention a few (§B.1).

###  4.2 Structured Data

HALvest-Geometric is made up of a heterogeneous graph. Following Wang et al.
(2020), we compute 238,397 author nodes, 18,662,037 paper nodes, 96,105
institution nodes, and 16 domain nodes for a total of18,996,55 nodes as of
February 2024. We also define 4 different edge types, constituting 23,761,191
edges.

###  4.3 Do citations help, even when fuzzy?

In this subsection, we will evaluate the citations extracted from each
publication, as described in subsection 3.3. Because we fuzzily retrieve the
citations, the added information to the citation network can be nothing more
than noise. A straightforward way to estimate the usefulness of this retrieved
information is to use graph neural networks (GNN) and message passing. Message
passing allows GNNs to discover the graph’s structure from the way information
propagates on it. Another key concept to better understand our experiment is a
property called homophily: adjacent nodes should have similar features; in our
context since authors would most likely cite people from their fields, the
assumption that our data is highly homophilic is not far-fetched. Therefore,
the subsequent graph neural networks used to represent our graph should
perform better in setting with domain-related papers as adjacent nodes, than
with other papers two hopes away, bridged by a domain node.

####  4.3.1 Tasks & Settings

##### Authorship attribution

Given a document $s\in\mathcal{S}$ and a candidate author $a\in\mathcal{A}$,
we want to compute a probability

$\displaystyle p:\ $ $\displaystyle\mathcal{S}\times\mathcal{A}\mapsto[0,1]$
$\displaystyle s,a\rightarrow p(y|s,a)$

that the individual $a$ is an author of $s$.

##### Link prediction

When given a citation network, the authorship attribution problem can be
reformulated as a link prediction problem. Given a graph
$\mathcal{G}(\mathcal{V},\mathcal{E})$ with $V$ the set of all nodes and
$\mathcal{E}\subseteq|\mathcal{V}|\times|\mathcal{V}|$ the set of all the true
edges in $\mathcal{G}$. In our setting, given a potential author node $u$ and
a paper node $v$, we want to learn a classifier that predicts the probability
of the existence of an edge, by computing a score between the representations
of both incident nodes $\hat{y}_{u\sim v}=f(h_{u},h_{v})$ where $h_{u}$ and
$h_{v}$ are learned representation of said nodes. In this task we use a set of
candidate edges $\mathcal{E}^{\prime}$ comprising of positive and negative
edges between the incident nodes, allowing us to use a binary cross entropy
loss

$\displaystyle\mathcal{L}$ $\displaystyle=-\sum_{u\sim v\in
E^{\prime}}[y_{u\sim v}\log(\hat{y}_{u\sim v})$ $\displaystyle+(1-y_{u\sim
v})\log(1-\hat{y}_{u\sim v})]$

This closed setting, however, doesn’t account for name ambiguity and unknown
authors, as we only consider identified candidates from the citation network
to perform link prediction.

##### Baselines

HALvest’s heterogeneous graph is featureless, prompting us to learn a
representation for each node. We use several state-of-the-art graph neural
networks (GNN) architectures (Scarselli et al., 2009) to learn embedding for
each node while training models.

##### Evaluation

We report the area under the curve (AUC) as a measure of the quality of a link
prediction algorithm. Since our experiment is done in a closed setting, we
compute five random splits of the original citation network, keeping 10% of
the author $\leftrightarrow$ paper edges for validation and 20% for test
purposes.

####  4.3.2 Link Prediction

GNN | AUC | AUC (w/o citations)
---|---|---
GraphSAGE | $99.08_{\pm 0.05}$ | $89.56_{\pm 0.13}$
GAT | $98.44_{\pm 0.09}$ | $74.30_{\pm 5.10}$
RGGC | $99.3_{\pm 0.05}$ | $91.71_{\pm 0.29}$

Table 3: Area under the curve (with standard deviation) of each link
prediction model. The result is the average performance of 5 models trained
with 5 different random seeds. We use GraphSage (Hamilton et al., 2017), graph
attention network (Veličković et al., 2018) and residual gated graph
convolutions (Bresson and Laurent, 2018).

We build a simple link prediction architecture, comprising of an embedding
layer for each node type—of dimension
$\mathbb{R}^{|\mathcal{V}^{\prime}|\times 16}$ where $|\mathcal{V}^{\prime}|$
is the number of nodes of a given type—followed by two GNN layers, to learn an
inductive 16 dimensions representation of each node. The implemented GNNs are
GraphSAGE (Hamilton et al., 2017), a graph attention network (Veličković et
al., 2018), and a residual gated graph convolutional network (Bresson and
Laurent, 2018). Finally, we compute the cosine similarity between candidate
author and paper nodes, map it to probabilities, and rule out, or not the
presence of a link between each pair of nodes.

We follow (Fey and Lenssen, 2019)’s implementation and perform Bayesian
optimization to find the best hyperparameters (Li et al., 2020; Bergstra et
al., 2015). We report the training results in Table 3. The training
hyperparameters and learning metrics can be found in § C.1.

####  4.3.3 Results

The results of our experiments provide compelling evidence that incorporating
citations, even when retrieved fuzzily, enhances the performance of graph-
based models in authorship attribution tasks. The area under the curve (AUC)
scores, as presented in Table 3, illustrate this improvement across various
graph neural network (GNN) architectures.

##### Impact of Citations on Performance

Our evaluation metric, clearly indicates that the inclusion of citation
information significantly boosts the model’s performance by about 10% across
all models. These results underscore the hypothesis that citations, even when
not perfectly precise, contribute valuable information to the citation
network. This enhancement can be attributed to the property of homophily,
where nodes (authors and papers) tend to be more similar when they are closely
related or cited within the same domain.

##### Model Comparisons and Robustness

The comparative analysis of different GNN architectures further validates our
approach. All three models—GraphSAGE, GAT, and RGGC—exhibited significant
performance drops when citation information was removed. This consistency
across multiple architectures suggests that the benefits of incorporating
citations are not model-specific but rather a generalizable advantage.
Moreover, the standard deviations reported alongside the AUC scores reflect
the robustness of our models. The relatively low standard deviations indicate
stable performance across different random splits of the data, reinforcing the
reliability of our findings.

##  5 Limitations

##### PDF processing

Solely relying on GROBID to process the pdfs hindered the dataset creation in
some aspects. As only the pdf’s layout tokens matter to GROBID, and no optical
character recognition is performed, documents with odd layouts can not be
exploited after being converted, and are outright discarded. Most of the
documents are seamlessly converted, yet some of them have near gibberish span
of text within cleaner sentences. We designed our filtering process to remove
documents with a prior unusable layout, but not to remove the gibberish span
of text within clean data. Therefore, further processing effort is needed from
practitioners.

##### Multilingual documents

Some documents are written in two or more languages, but HAL imposes the
depositor to only fill one language. It is therefore necessary to perform
language identification at the sentence level to ensure only the needed
language is fetched from the dataset.

##### Accounting for references

When building the citation network, the references at the end of each paper
are not consolidated, resulting in a fuzzy process of deduplication afterward.
This leads our graph to lack correctness when it comes to modeling citations.

##  6 Conclusion

By processing approximately 700,000 documents in 56 languages across 13
domains from Hyper Articles en Ligne (HAL), we have created a unique resource
that maps natural language text to a directed heterogeneous graph: HALvest, a
comprehensive dataset that integrates citation networks with the full-text of
scholarly papers. This dataset encompasses 16.5 billion textual tokens for
18,996,55 nodes, enabling extensive research in multimodality, authorship
attribution, author name disambiguation, domain classification, and more.

We elaborate on the usefulness of HALvest, by performing authorship
attribution, using state-of-the-art GNN architectures, and discuss the added
value of retrieved citations, further confirming HALvest’s adequacy for the
natural language processing and graph representation learning fields.

Future work will focus on expanding the dataset and improving the
preprocessing pipeline, for greater accuracy representing the academic
interactions, and increased utility.

The dataset, along with associated benchmarks, are openly available, aiming to
democratize access to large-scale scholarly data.

## Acknowledgements

The authors are grateful to the CCSD staff, Achraf Azhar, and Patrice Lopez
for providing resources and support to better handle HAL and GROBID. We would
also like to thank Brahim Talb and Menel Mahamdi for the productive
discussions.

## References

  * 6 (2024) 6\. 2024.  stopwords-json.  Original-date: 2014-02-01T08:08:26Z.
  * Ammar et al. (2018) Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, Rodney Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler Murray, Hsu-Han Ooi, Matthew Peters, Joanna Power, Sam Skjonsberg, Lucy Lu Wang, Chris Wilhelm, Zheng Yuan, Madeleine van Zuylen, and Oren Etzioni. 2018.  Construction of the Literature Graph in Semantic Scholar.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers)_ , pages 84–91, New Orleans - Louisiana. Association for Computational Linguistics.
  * arXiv.org submitters (2024) arXiv.org submitters. 2024.  arxiv dataset.
  * Bergstra et al. (2015) James Bergstra, Brent Komer, Chris Eliasmith, Dan Yamins, and David D Cox. 2015.  Hyperopt: a python library for model selection and hyperparameter optimization.  _Computational Science & Discovery_, 8(1):014008.
  * Bird et al. (2008) Steven Bird, Robert Dale, Bonnie Dorr, Bryan Gibson, Mark Joseph, Min-Yen Kan, Dongwon Lee, Brett Powley, Dragomir Radev, and Yee Fan Tan. 2008.  The ACL Anthology Reference Corpus: A Reference Dataset for Bibliographic Research in Computational Linguistics.  In _Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08)_ , Marrakech, Morocco. European Language Resources Association (ELRA).
  * Boukhers et al. (2021) Zeyd Boukhers, Nagaraj Bahubali, Abinaya Thulsi Chandrasekaran, Adarsh Anand, Soniya Manchenahalli Gnanendra Prasadand, and Sriram Aralappa. 2021.  Bib2Auth: Deep Learning Approach for Author Disambiguation using Bibliographic Data.  _arXiv preprint_.  ArXiv:2107.04382 [cs].
  * Bresson and Laurent (2018) Xavier Bresson and Thomas Laurent. 2018.  Residual Gated Graph ConvNets.  _arXiv preprint_.  ArXiv:1711.07553 [cs, stat].
  * Cohan et al. (2019) Arman Cohan, Waleed Ammar, Madeleine van Zuylen, and Field Cady. 2019.  Structural Scaffolds for Citation Intent Classification in Scientific Publications.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3586–3596, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Cohan et al. (2020) Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. 2020.  SPECTER: Document-level Representation Learning using Citation-informed Transformers.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2270–2282, Online. Association for Computational Linguistics.
  * Doğan et al. (2014) Rezarta Islamaj Doğan, Robert Leaman, and Zhiyong Lu. 2014.  NCBI disease corpus: a resource for disease name recognition and concept normalization.  _Journal of Biomedical Informatics_ , 47:1–10.
  * Fey and Lenssen (2019) Matthias Fey and Jan E. Lenssen. 2019.  Fast graph representation learning with PyTorch Geometric.  In _ICLR Workshop on Representation Learning on Graphs and Manifolds_.
  * (12) GROBID Repository. 2008–2024.  Grobid.  https://github.com/kermitt2/grobid.  _Preprint_ , swh:1:dir:dab86b296e3c3216e2241968f0d63b68e8209d3c.
  * Hamilton et al. (2017) Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017.  Inductive Representation Learning on Large Graphs.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Han et al. (2017) Hongqi Han, Changqing Yao, Yuan Fu, Yongsheng Yu, Yunliang Zhang, and Shuo Xu. 2017.  Semantic fingerprints-based author name disambiguation in Chinese documents.  _Scientometrics_ , 111(3):1879–1896.
  * Kim et al. (2019) Kunho Kim, Shaurya Rohatgi, and C. Lee Giles. 2019.  Hybrid Deep Pairwise Classification for Author Name Disambiguation.  In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_ , CIKM ’19, pages 2369–2372, New York, NY, USA. Association for Computing Machinery.
  * Kojaku et al. (2021) Sadamori Kojaku, Xiaoran Yan, Jisung Yoon, Filipi N. Silva, Vincent Larivière, and Yong-Yeol Ahn. 2021.  DisamBERT: Author name disambiguation with BERT.
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240.
  * Li et al. (2016) Jiao Li, Yueping Sun, Robin J. Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J. Mattingly, Thomas C. Wiegers, and Zhiyong Lu. 2016.  BioCreative V CDR task corpus: a resource for chemical disease relation extraction.  _Database_ , 2016:baw068.
  * Li et al. (2020) Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-tzur, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. 2020.  A system for massively parallel hyperparameter tuning.  In _Proceedings of Machine Learning and Systems_ , volume 2, pages 230–246.
  * Lo et al. (2020) Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020.  S2ORC: The Semantic Scholar Open Research Corpus.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4969–4983, Online. Association for Computational Linguistics.
  * Ma et al. (2020) Yingying Ma, Youlong Wu, and Chengqiang Lu. 2020.  A Graph-Based Author Name Disambiguation Method and Analysis via Information Theory.  _Entropy_ , 22(4):416.
  * Müller (2017) Mark-Christoph Müller. 2017.  Semantic Author Name Disambiguation with Word Embeddings.  In _Research and Advanced Technology for Digital Libraries_ , Lecture Notes in Computer Science, pages 300–311, Cham. Springer International Publishing.
  * Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023.  The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data Only.  _Advances in Neural Information Processing Systems_ , 36:79155–79172.
  * Pooja et al. (2022) K. M. Pooja, Samrat Mondal, and Joydeep Chandra. 2022.  Online author name disambiguation in evolving digital library.  _Neurocomputing_ , 493:1–14.
  * Pooja et al. (2021) KM. Pooja, Samrat Mondal, and Joydeep Chandra. 2021.  Exploiting similarities across multiple dimensions for author name disambiguation.  _Scientometrics_ , 126(9):7525–7560.
  * Priem et al. (2022) Jason Priem, Heather Piwowar, and Richard Orr. 2022.  OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts.  _arXiv preprint_.  ArXiv:2205.01833 [cs].
  * Rae et al. (2022) Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. 2022.  Scaling Language Models: Methods, Analysis & Insights from Training Gopher.  _arXiv preprint_.  ArXiv:2112.11446 [cs].
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):140:5485–140:5551.
  * (29) RedPajama Repository. 2023.  RedPajama: an Open Dataset for Training Large Language Models.
  * Rust et al. (2021) Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021.  How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3118–3135, Online. Association for Computational Linguistics.
  * Santini et al. (2022) Cristian Santini, Genet Asefa Gesese, Silvio Peroni, Aldo Gangemi, Harald Sack, and Mehwish Alam. 2022.  A knowledge graph embeddings based approach for author name disambiguation using literals.  _Scientometrics_ , 127(8):4887–4912.
  * Scarselli et al. (2009) Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2009.  The Graph Neural Network Model.  _IEEE Transactions on Neural Networks_ , 20(1):61–80.  Conference Name: IEEE Transactions on Neural Networks.
  * Sen et al. (2008) Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. 2008.  Collective Classification in Network Data.  _AI Magazine_ , 29(3):93–93.  Number: 3.
  * Shin et al. (2014) Dongwook Shin, Taehwan Kim, Joongmin Choi, and Jungsun Kim. 2014.  Author name disambiguation using a graph model with node splitting and merging based on bibliographic information.  _Scientometrics_ , 100(1):15–50.
  * Tang et al. (2012) Jie Tang, Alvis C.M. Fong, Bo Wang, and Jing Zhang. 2012.  A Unified Probabilistic Framework for Name Disambiguation in Digital Library.  _IEEE Transactions on Knowledge and Data Engineering_ , 24(6):975–987.  Conference Name: IEEE Transactions on Knowledge and Data Engineering.
  * Tang et al. (2008) Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. 2008.  ArnetMiner: extraction and mining of academic social networks.  In _Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining_ , KDD ’08, pages 990–998, New York, NY, USA. Association for Computing Machinery.
  * Taylor et al. (2022) Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022.  Galactica: A Large Language Model for Science.  _arXiv preprint_.  ArXiv:2211.09085 [cs, stat].
  * Tekles and Bornmann (2019) Alexander Tekles and Lutz Bornmann. 2019.  Author name disambiguation of bibliometric data: A comparison of several unsupervised approaches.  _arXiv preprint_.  ArXiv:1904.12746 [cs].
  * Tran et al. (2014) Hung Nghiep Tran, Tin Huynh, and Tien Do. 2014.  Author Name Disambiguation by Using Deep Neural Network.  In _Intelligent Information and Database Systems_ , Lecture Notes in Computer Science, pages 123–132, Cham. Springer International Publishing.
  * Veličković et al. (2018) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018.  Graph attention networks.  In _International Conference on Learning Representations_.
  * Vishnyakova et al. (2019) Dina Vishnyakova, Raul Rodriguez-Esteban, and Fabio Rinaldi. 2019.  A new approach and gold standard toward author disambiguation in MEDLINE.  _Journal of the American Medical Informatics Association : JAMIA_ , 26(10):1037–1045.
  * Wahle et al. (2022) Jan Philip Wahle, Terry Ruas, Saif Mohammad, and Bela Gipp. 2022.  D3: A Massive Dataset of Scholarly Metadata for Analyzing the State of Computer Science Research.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 2642–2651, Marseille, France. European Language Resources Association.
  * Wang et al. (2020) Kuansan Wang, Zhihong Shen, Chiyuan Huang, Chieh-Han Wu, Yuxiao Dong, and Anshul Kanakia. 2020.  Microsoft Academic Graph: When experts are not enough.  _Quantitative Science Studies_ , 1(1):396–413.
  * Wenzek et al. (2020) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020.  CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 4003–4012, Marseille, France. European Language Resources Association.
  * Xie et al. (2022) Wenjin Xie, Siyuan Liu, Xiaomeng Wang, and Tao Jia. 2022.  Author Name Disambiguation via Heterogeneous Network Embedding from Structural and Semantic Perspectives.  In _2022 IEEE 34th International Conference on Tools with Artificial Intelligence (ICTAI)_ , pages 245–250.  ArXiv:2212.12715 [cs].
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Yasunaga et al. (2022) Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D. Manning, Percy S. Liang, and Jure Leskovec. 2022.  Deep Bidirectional Language-Knowledge Graph Pretraining.  _Advances in Neural Information Processing Systems_ , 35:37309–37323.
  * Zhai et al. (2019) Xiaorui Zhai, Hongqi Han, Zhong Li, and Yaxin Ran. 2019.  Research on Author Name Disambiguation Based on Fusion Features and Semantic Fingerprints.  _Journal of Physics: Conference Series_ , 1302(2):022013.  Publisher: IOP Publishing.
  * Zhang et al. (2022) Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D. Manning, and Jure Leskovec. 2022.  GreaseLM: Graph REASoning Enhanced Language Models for Question Answering.  _arXiv preprint_.  ArXiv:2201.08860 [cs].
  * Zhang et al. (2019) Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019.  ERNIE: Enhanced Language Representation with Informative Entities.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1441–1451, Florence, Italy. Association for Computational Linguistics.

##  Appendix A Collection Methodology

###  A.1 API request format

We use the following request to fetch open papers from HAL, yielding 743,160
documents: https://api.archives-
ouvertes.fr/search/?q=*&fq=dateLastIndexed_tdate:[2001-01-01T00:00:00ZTO2024-02-29T23:59:59Z]&fq=openAccess_bool:true&wt=xml-
tei&sort=docidasc&rows=500&cursorMark=*

###  A.2 Example of texts from pdf files with faulty unicode mappings

halid |  Sample Text
---|---
01762182 |  1 Introduction 1 IFI wodel nd prolem sttement F F F F F F F F F F F F F F F F F F I IFIFI ystem model F F F F F F F F F F F F F F F F F F F F F F F I IFIFP vrge dimensionl regime F F F F F F F F F…
00177057 |  , 4 @ / A -@ & $ B 4 & A / 2 @ - " 0 / -4 , / 3 $C" ’ 5$ B 5C % " +0 2 C % BB * ( ( $ - $ @" ’ + 0 D 0 E $ F $&$@" C % < = $ B 7 (#7 $ - $ @" 0 D 0 E @" @ C G $ ( > (( < 5HI 97>89 D (0 E " @C"@F C G $…
01770410 |  ACKNOWLEDGMENT S T h i s d o c t o r a l w o r k i s a b o u t m a k i n g o u r s w h a t w e s h a r e . Y e t , I c o u l d n o t h a v e r e a c h e d t h i s f i n a l s t a g e w i t h o u t s h…
01784066 |  II Theory L I S T O F T A B L E S Table Deterministic exact approaches to mean-variance portfolio selection problem (see also [START_REF] Mansini | Twenty Years of Linear Programming Based Portfolio Optimization[END_REF]…
Table 4: Examples of halid and output text from pdf files with no Unicode
mapping, leading to gibberish text. While the first two examples are from
discarded documents, the third one is kept, as only the section names are not
processed correctly.

See Table 4.

##  Appendix B Composition

###  B.1 Language composition

ISO-639 | Language | # Documents | # mT5 Tokens
---|---|---|---
en | English | 464,679 | 8,158,933,235
fr | French | 199,216 | 9,018,529,985
es | Spanish | 2,975 | 69,221,667
it | Italian | 1,172 | 48,747,986
pt | Portuguese | 934 | 32,918,832
de | German | 652 | 12,225,960
ru | Russian | 245 | 5,763,532
zh | Chinese | 160 | 2,861,585
eu | Basque | 113 | 2,297,485
ar | Arabic | 92 | 2,167,431
ja | Japanese | 92 | 547,861
el | Greek | 54 | 1,738,878
pl | Polish | 43 | 987,878
ro | Romanian | 39 | 1,298,901
uk | Ukrainian | 34 | 837,793
vi | Vietnamese | 29 | 436,660
ca | Catalan | 28 | 975,078
da | Danish | 27 | 961,955
oc | Occitan | 26 | 285,334
br | Breton | 24 | 998,088
sr | Serbian | 24 | 336,878
ko | Korean | 17 | 226,268
fa | Persian | 17 | 213,903
tr | Turkish | 17 | 149,718
hu | Hungarian | 14 | 577,568
eo | Esperanto | 14 | 105,286
hy | Armenian | 10 | 127,988
cs | Czech | 9 | 712,263
bg | Bulgarian | 9 | 208,763
sq | Albanian | 9 | 98,009
id | Indonesian | 9 | 53,075
he | Hebrew | 8 | 61,283
hr | Croatian | 8 | 40,621
et | Estonian | 7 | 20,405
sv | Swedish | 6 | 270,642
no | Norwegian | 6 | 62,767
az | Azerbaijani | 5 | 52,762
fi | Finnish | 4 | 60,507
tet | Tetum | 4 | 18,485
lt | Lithuanian | 3 | 16,572
mr | Marathi | 3 | 16,386
hi | Hindi | 3 | 3,490
ie | Interlingue | 2 | 140,383
ta | Tamil | 2 | 77,087
sw | Swahili | 2 | 73,921
tl | Tagalog | 2 | 35,962
gl | Galician | 2 | 29,688
mk | Macedonian | 2 | 14,654
th | Thai | 1 | 70,909
tk | Turkmen | 1 | 66,104
bs | Bosnian | 1 | 63,018
kk | Kazakh | 1 | 41,839
sl | Slovenian | 1 | 22,844
sk | Slovak | 1 | 12,997
co | Corsican | 1 | 9,083
gn | Guarani | 1 | 1,566
bo | Tibetan | 1 | 579

Table 5: Language statistics including ISO-639 codes, number of documents, and number of mT5 tokens for the raw version of HALvest. ISO-639 | Language | # Documents | # mT5 Tokens
---|---|---|---
en | English | 442,892 | 7,606,895,258
fr | French | 193,437 | 8,728,722,255
Table 6: Language statistics including ISO-639 codes, number of documents, and
number of mT5 tokens for the filtered version of HALvest.

See Tables 5 and 6.

###  B.2 Domain composition

Domain | Code | # Documents | # mT5 Tokens
---|---|---|---
Humanities and Social Sciences | shs | 156,566 | 5,614,423,171
Computer Science | info | 148,316 | 2,573,673,455
Life Sciences | sdv | 115,744 | 3,145,323,780
Engineering Sciences | spi | 102,751 | 2,254,653,825
Physics | phys | 65,991 | 1,503,190,749
Mathematics | math | 62,921 | 1,638,500,361
Chemical Science | chim | 40,012 | 899,507,319
Environmental Science | sde | 31,575 | 579,076,669
Sciences of the Universe | sdu | 23,557 | 682,356,264
Cognitive Science | scco | 11,772 | 227,487,096
Statistics | stat | 10,579 | 184,678,350
Quantitative Finance | qfin | 3,451 | 68,518,636
Nonlinear Sciences | nlin | 1,972 | 30,694,088
Table 7: Domain statistics including domain codes, number of documents, and
number of mT5 tokens for the raw version of HALvest.

See Table 7.

##  Appendix C Experiments

We train all the models for 2 epochs, with a batch size of 128. For message
passing, we sample 32 random neighbor nodes in the first hope and 16 nodes in
the second hope. We chose those numbers because the average number of
citations is 20, sampling 32 neighboring nodes from an author
$\leftrightarrow$ paper edge allows us to capture most of the cited papers as
well as the authors. For each positive author $\leftrightarrow$ paper, we
sample two negative pairs for the model to train on, those pairs are sampled
at each step.

###  C.1 Link prediction hyperparameters

We used the following hyperparameters for each moedl:

  * •

GraphSage:

    * –

Hidden channels: 64

    * –

Dropout: 0.1

    * –

Weight Decay: $1\times 10^{-7}$

    * –

Learning rate: $1\times 10^{-3}$

  * •

GAT:

    * –

Hidden channels: 64

    * –

Dropout: 0.5

    * –

Number of attention heads: 8

    * –

Weight Decay: $1\times 10^{-4}$

    * –

Learning rate: $5\times 10^{-3}$

  * •

RGGC:

    * –

Hidden channels: 16

    * –

Dropout: 0.1

    * –

Weight Decay: $1\times 10^{-4}$

    * –

Learning rate: $1\times 10^{-2}$


# Deep Convolutional Neural Network for Low Projection SPECT Imaging
Reconstruction

Charalambos Chrysostomou∗, Loizos Koutsantonis, Christos Lemesios, and Costas
N. Papanicolas C. Chrysostomou, L. Koutsantonis, C. Lemesios and C.N.
Papanicolas are with the computation-based Science and Technology Research
Center, The Cyprus Institute, 20 Konstantinou Kavafi Street, 2121, Aglantzia,
Nicosia, Cyprus

###### Abstract

In this paper, we present a novel method for tomographic image reconstruction
in SPECT imaging with a low number of projections. Deep convolutional neural
networks (CNN) are employed in the new reconstruction method. Projection data
from software phantoms were used to train the CNN network. For evaluation of
the efficacy of the proposed method, software phantoms and hardware phantoms
based on the FOV SPECT system were used. The resulting tomographic images are
compared to those produced by the ”Maximum Likelihood Expectation
Maximisation” (MLEM).

###### Index Terms:

Convolutional Neural Networks (CNN), Reconstruction, Single Photon Emission
Computerized Tomography (SPECT), SPECT angle interpolation

††Manuscript received December 20, 2020

##  I Introduction

Computed tomography (CT) techniques are useful nuclear imaging techniques that
are used most widely in medicine [1]. Single Photon Emission Computerized
Tomography (SPECT) [2, 3, 4] and Positron Emission Tomography (PET) [5, 6] are
two of the most used CT techniques for diagnosis and monitoring of numerous
diseases such as cancer [7] and cardiac diseases [8]. SPECT image
reconstructions have limited spatial resolution and underperform when a low
number of projections is available or when the measurements are low quality
and noisy. These difficulties produce an ill-posed inverse problem for image
reconstruction. In order to solve those issues, alternative reconstruction
algorithms were proposed and developed, such as ordered subset expectation
maximization (OSEM) [9] and maximization likelihood expectation maximization
(MLEM) [10]. Nevertheless, the improvement in the reconstructions is not ideal
in many cases. The reconstruction accuracy improves with an increase in the
number of iterations but with the downside of also increasing the noise. The
noise increase in the reconstruction is challenging to measure and may affect
the final reconstruction as a false detection. Therefore, a new method is
needed to provide high spatial resolution and handle high noise and a low
number of projections. In this paper, a novel method for tomographic image
reconstruction in SPECT imaging is proposed based on deep convolutional neural
networks (CNNR).

The paper is organised as follows: Section II-A presents the data generated
and used for training the proposed model, Section II-B, presents the proposed
model. Section III, presents the results and discussions and finally Section
IV is conclusions.

##  II Methods and Materials

###  II-A Training Data

A collection of sinograms was produced from randomly generated software
phantoms through $Y_{i}=\sum_{j=1}^{N\,PxN\,R}P_{ij}F_{j}$ where $N\,R$ is the
number of bin measurements per projection angle and $N\,P$ is the number of
projection angles. For the proposed project, 500,000 software phantoms were
generated on a rectangular grid of $128\times 128$ pixels size. Furthermore,
sinograms were created by simulating 24 projections, evenly spaced in 360
degrees, for the proposed model’s training. Examples of the produced phantoms
are presented in figure 1.

In order to have more robust and accurate results from the proposed model, the
produced sinograms were also randomised with a Poisson probability
distribution to produce the noisy sets of projections. The model was trained
for 1000 epochs, where 90% of the available data was used for training and 10%
for validation. Ultimately, Shepp-Logan phantom [11], as shown in figure 4(a)
was used to evaluate and assess the proposed method’s capabilities.

Figure 1: Sample software phantoms randomly generated in order to train the
proposed model

###  II-B Proposed Model

Figure 2: Structure of the proposed model based on deep convolutionl neural
networks

Artificial Neural networks and, more specifically, Convolutional neural
network (CNN) have been used with great success in multiple domains to analyse
visual representations. Initially, CNN’s were inspired by physiological and
biological processes, and the connectivity model between neurons compares to
the structure of the biological visual cortex [12]. CNN’s are designed and
require relatively minimum pre-processing of data compared to other
methodologies by employing multilayer perceptrons [13]. This minimum prior
knowledge provision provides an essential advantage of CNN’s over other
methodologies where prior knowledge and feedback from experts are required.
CNN’s have already successfully implemented in medical image analysis [14],
and classification [15, 16].

As shown in Figure 2, the proposed model structure consists of two parts, the
encoder and the decoder. The encoder consists of five blocks. The first three
blocks consist of two convolutional layers with 3x3 kernel and rectified
nonlinear activation function (Leaky ReLU) [17], followed by a batch
normalisation layer, and a 2x2 max pooling layer [18] and a Dropout layer of
30%. The fourth block consists of two convolutional layers with a 3x3 kernel,
a batch normalisation layer, a Leaky ReLU activation function, a Dropout layer
of 30% and a flatten layer. The fifth layer consists of a dense layer of 4096
neurons, a Leaky ReLU activation function, and a batch normalisation layer.
The number of kernels increases for each block, beginning with 32 kernels for
the first block, 64, 128 and 256 for the second, third and fourth blocks,
respectively.

Before the data are passed to the decoder, the decoder’s output is transformed
from 4096 features to a vector of 4x4x256. The decoder consists of six blocks.
The first five blocks consist of two convolutional layers and a transposed
convolutional layers, with a stride of 2, with a 3x3 kernel and Leaky ReLU
activation function and a batch normalisation layer. The final sixth layer
consists of a convolutional layer of 1x1 kernel and linear activation
function. The number of kernels decreases for each block, beginning with 256
kernels for the first block, 128, 64, 32, 8 and 1 for the second, third,
fourth, fifth and sixth blocks, respectively. Structural Similarity (SSIM)
Index [19] was used as the loss function for the training of the proposed
method, as shown in equation 1

$SSIM(x,y)=\frac{(2\mu_{x}\mu_{y}+C_{1})(2\sigma_{xy}+C_{2})}{(\mu_{x}^{2}+\mu_{y}^{2}+C_{1})(\sigma_{x}^{2}+\sigma_{y}^{2}+C_{2})}$
(1)

where $\mu_{x}$, $\mu_{y}$ and $\sigma_{x}^{2}$, $\sigma_{y}^{2}$ represent
the average and variance of $x$ and $y$ respectively while $\sigma_{{xy}}$
represent the covariance of $x$ and $y$; $c_{1}=(k_{1}L)^{2}$,
$c_{2}=(k_{2}L)^{2}$ two variables to stabilise the division with weak
denominator; $L$ the dynamic range of the pixel-values (typically this is
$2^{{\\#bits\ per\ pixel}}-1)$; $k_{1}=0.01$ and $k_{2}=0.03$ by default.

##  III Results and Discussions

The performance of the proposed method was compared against existing methods
by using the Mean Square Error (MSE), Structural Similarity (SSIM) Index [19],
and the Pearson Correlation Coefficient (PCC) [20]. As the results show and
illustrated in Table I, the proposed methodology outperforms MLEM with 0.003
versus 0.006, 0.92 versus 0.77 and 0.96 versus 0.93 for MSE, SSIM and PCC
respectively. The proposed method was also tested using hardware phantoms
generated using a small FOV SPECT system [21] of three parallel pillars, as
shown in Figure 3 with the activity of 21 $\mu Ci$ and variant background
activity of 190, 360 and 700 $\mu Ci$. The results are presented in figure 5.

Figure 3: Hardware phantoms of three parallel pillars, with activity of 21 $\mu Ci$ and variant background activity of 190, 360 and 700 $\mu Ci$ TABLE I: Results Methodology | MSE | SSIM | PCC
---|---|---|---
MLEM | 0.006 | 0.77 | 0.93
Proposed Model | 0.003 | 0.92 | 0.96

(a) True activity of the Shepp Logan Phantom

(b) MLEM Reconstruction of the Shepp Logan Phantom

(c) CNNR Reconstruction of the Shepp Logan Phantom

Figure 4: Evaluation of the proposed method results compared to the MLEM
method using the Shepp Logan Phantom. The results achieved using the proposed
method compare favourably to those obtained with the widely used MLEM method.
Figure 5: Evaluation of the proposed method results compared to the MLEM
method using the Hardware phantoms.

##  IV Conclusions

In this paper, we proposed a new method based on convolutional neural networks
to reconstruct SPECT imaging and demonstrate its capability to reconstruct
images with a limited number of projections. As the results show, the proposed
method can significantly outperform existing methods such as MLEM. Although
the results presented in this proposed work are appropriate for representing
the proposed method’s capabilities, further experimentation is needed, as
shown in Figure 3 were the size and shape of the reconstructed caterpillars
where overestimated.

## References

  * [1] A. B. De González, M. Mahesh, K.-P. Kim, M. Bhargavan, R. Lewis, F. Mettler, and C. Land, “Projected cancer risks from computed tomographic scans performed in the united states in 2007,” _Archives of internal medicine_ , vol. 169, no. 22, pp. 2071–2077, 2009.
  * [2] M. N. Wernick and J. N. Aarsvold, _Emission tomography: the fundamentals of PET and SPECT_. Elsevier, 2004.
  * [3] M. T. Madsen, “Recent advances in spect imaging,” _Journal of Nuclear Medicine_ , vol. 48, no. 4, p. 661, 2007.
  * [4] G. Mariani, L. Bruselli, T. Kuwert, E. E. Kim, A. Flotats, O. Israel, M. Dondi, and N. Watanabe, “A review on the clinical uses of spect/ct,” _European journal of nuclear medicine and molecular imaging_ , vol. 37, no. 10, pp. 1959–1985, 2010.
  * [5] S. R. Cherry, “Fundamentals of positron emission tomography and applications in preclinical drug development,” _The Journal of Clinical Pharmacology_ , vol. 41, no. 5, pp. 482–491, 2001.
  * [6] J. J. Vaquero and P. Kinahan, “Positron emission tomography: current challenges and opportunities for technological advances in clinical and preclinical imaging systems,” _Annual review of biomedical engineering_ , vol. 17, pp. 385–414, 2015.
  * [7] M. Perri, P. Erba, D. Volterrani, E. Lazzeri, G. Boni, M. Grosso, and G. Mariani, “Octreo-spect/ct imaging for accurate detection and localization of suspected neuroendocrine tumors,” _The Quarterly Journal of Nuclear Medicine and Molecular Imaging_ , vol. 52, no. 4, p. 323, 2008.
  * [8] J. Chen, E. V. Garcia, R. D. Folks, C. D. Cooke, T. L. Faber, E. L. Tauxe, and A. E. Iskandrian, “Onset of left ventricular mechanical contraction as determined by phase analysis of ecg-gated myocardial perfusion spect imaging: development of a diagnostic tool for assessment of cardiac mechanical dyssynchrony,” _Journal of nuclear cardiology_ , vol. 12, no. 6, pp. 687–695, 2005.
  * [9] H. M. Hudson and R. S. Larkin, “Accelerated image reconstruction using ordered subsets of projection data,” _IEEE transactions on medical imaging_ , vol. 13, no. 4, pp. 601–609, 1994.
  * [10] L. A. Shepp and Y. Vardi, “Maximum likelihood reconstruction for emission tomography,” _IEEE transactions on medical imaging_ , vol. 1, no. 2, pp. 113–122, 1982.
  * [11] L. A. Shepp and B. F. Logan, “The fourier reconstruction of a head section,” _IEEE Transactions on nuclear science_ , vol. 21, no. 3, pp. 21–43, 1974\.
  * [12] D. H. Hubel and T. N. Wiesel, “Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex,” _The Journal of physiology_ , vol. 160, no. 1, pp. 106–154, 1962.
  * [13] J. Schmidhuber, “Deep learning in neural networks: An overview,” _Neural networks_ , vol. 61, pp. 85–117, 2015.
  * [14] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. van der Laak, B. van Ginneken, and C. I. Sánchez, “A survey on deep learning in medical image analysis,” _Medical image analysis_ , vol. 42, pp. 60–88, 2017.
  * [15] C. Chrysostomou, L. Koutsantonis, C. Lemesios, and C. N. Papanicolas, “A reconstruction method based on deep convolutional neural network for spect imaging,” in _2018 IEEE Nuclear Science Symposium and Medical Imaging Conference Proceedings (NSS/MIC)_. IEEE, 2018, pp. 1–4.
  * [16] ——, “Spect imaging reconstruction method based on deep convolutional neural network,” in _2019 IEEE Nuclear Science Symposium and Medical Imaging Conference (NSS/MIC)_. IEEE, 2019, pp. 1–4.
  * [17] A. L. Maas, A. Y. Hannun, and A. Y. Ng, “Rectifier nonlinearities improve neural network acoustic models,” in _Proc. icml_ , vol. 30, no. 1, 2013, p. 3.
  * [18] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, _Deep learning_. MIT press Cambridge, 2016, vol. 1, no. 2.
  * [19] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE transactions on image processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [20] J. Benesty, J. Chen, Y. Huang, and I. Cohen, “Pearson correlation coefficient,” in _Noise reduction in speech processing_. Springer, 2009, pp. 1–4.
  * [21] V. Spanoudaki, N. Giokaris, A. Karabarbounis, G. Loudos, D. Maintas, C. Papanicolas, P. Paschalis, and E. Stiliaris, “Design and development of a position-sensitive $\gamma$-camera for spect imaging based on pci electronics,” _Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment_ , vol. 527, no. 1-2, pp. 151–156, 2004.