# Query-LIFE: Query-aware Language Image Fusion Embedding for E-Commerce
Relevance

Hai Zhu  1234-5678-9012 University of Science and Technology of
ChinaHefeiChina43017-6221<EMAIL_ADDRESS>and  Yuankai Guo,
Ronggang Dou, Kai Liu  LazadaHangzhouChina
<EMAIL_ADDRESS>
(2024)

###### Abstract.

Relevance module plays a fundamental role in e-commerce search as they are
responsible for selecting relevant products from thousands of items based on
user queries, thereby enhancing users experience and efficiency. The
traditional approach models the relevance based product titles and queries,
but the information in titles alone maybe insufficient to describe the
products completely. A more general optimization approach is to further
leverage product image information. In recent years, vision-language pre-
training models have achieved impressive results in many scenarios, which
leverage contrastive learning to map both textual and visual features into a
joint embedding space. In e-commerce, a common practice is to fine-tune on the
pre-trained model based on e-commerce data. However, the performance is sub-
optimal because the vision-language pre-training models lack of alignment
specifically designed for queries. In this paper, we propose a method called
Query-LIFE (Query-aware Language Image Fusion Embedding) to address these
challenges. Query-LIFE utilizes a query-based multimodal fusion to effectively
incorporate the image and title based on the product types. Additionally, it
employs query-aware modal alignment to enhance the accuracy of the
comprehensive representation of products. Furthermore, we design GenFilt,
which utilizes the generation capability of large models to filter out false
negative samples and further improve the overall performance of the
contrastive learning task in the model. Experiments have demonstrated that
Query-LIFE outperforms existing baselines. We have conducted ablation studies
and human evaluations to validate the effectiveness of each module within
Query-LIFE. Moreover, Query-LIFE has been deployed on Miravia Search111Miravia
is a local-to-local e-commerce platform in Spain incubated by Lazada, as one
part of Alibaba International Digital Commerce (AIDC) Group.
https://www.miravia.es/, resulting in improved both relevance and conversion
efficiency.

Relevance Modeling, Representation Learning, Contrastive Learning, Multi-modal
Representation

††copyright: acmcopyright††journalyear: 2024††doi:
XXXXXXX.XXXXXXX††conference: Companion Proceedings of the ACM Web Conference
2024; May 13–17,2024; Singapore††price: 15.00††isbn:
978-1-4503-XXXX-X/18/06††ccs: Information systems Relevance modeling

##  1\. Introduction

Nowadays, since Internet penetration is increased to a high level, online
shopping has become to a highly convenient manner for consumers. Everyday,
millions of users search & browse products, and maybe finally place orders in
e-commerce platforms. Consequently, the relevance of products exposed to users
triggered by their queries plays a crucial role in users shopping experiences,
and also the transaction efficiency. Therefore, it is essential to accurately
judge whether the candidate products are relevant to the user intentions for
an e-commerce search engine.

Traditionally, relevance model (Robertson et al., 2009; Huang et al., 2013;
Chang et al., 2021; Hu et al., 2014; Yao et al., 2021) have primarily relied
on textual information, such as query and product descriptions (title,
attribute, etc) to judge the relevance between queries and products. However
the product information also includes images which captures a significant
portion of user attention during browsing products, thus it is becoming
increasingly essential to incorporate image into relevance modeling. This
integration of both image and text data has the potential to provide a more
comprehensive understanding of the products and better capture user intent.

Some core information may be missing from product titles, just like the cases
in Table 1. In these cases, if only based on the product titles, it is hard to
accurately match the relevant product with search query. However, product
images can provide incremental valuable information for relevance judgement.
In recent years, there has been a surge of vision-language pre-training models
(VLP) (Li et al., 2023, 2021; Jia et al., 2021; Wang et al., 2023, 2021). As
shown in Figure 1(a), these VLP models typically consist of both text and
image encoders, and leverage text-image contrastive learning to align
representations across different modalities. They have demonstrated remarkable
performance in various general tasks, such as image captioning, visual
question answering, and text-image retrieval. It is worth noting that these
VLP models can extract image features to enhance the representation of
products with ambiguous titles, or correct the representation of products with
misleading titles in e-commerce relevance task. Current VLP models in
e-commerce relevance task often employ a divide-and-conquer approach. As shown
in Figure 1(b), they first conduct VLP model extracts query, title, and image
representations, and then add query-title similarity and query-image
similarity as the final relevance score. However, different types of products
contain varying weights of information in images and titles. For example,
electronic products often have important parameters listed in their titles,
while clothing items rely more on visual elements depicted in the images, such
as designed style, texture, material, color, etc.

Figure 1. (a) The relationship of relevance model, VLP model and Query-LIFE.
(b) VLP model divide-and-conquer approach for relevance task. (c) Query-LIFE
fusion approach for relevance task.

.

Table 1. Both product images and titles can be incorporated together to judge the search relevance with queries. Query |  Image |  Title
---|---|---
men’s winter coat |  |  Koroshi Jacket in two colors, water-repellent, with hood, for Men
air-conditioning |  |  Split 1x1 MUNDOCLIMA MUPR12 H11 3027frig R32
golden necklace |  |  Elegant necklace with col-layered pearl gent

In this paper, we propose a novel approach called Query-aware Language Image
Fusion Embedding(referred as Query-LIFE) for e-commerce relevance modeling. As
shown in Figure 1(a), it integrates query, title, and image into relevance
tasks. Firstly, we randomly sample ¡query,title,image¿ triplet data from the
online user behavior logs as pre-training data. Secondly, unlike the divide-
and-conquer approach, we propose a dynamic fusion as multi-modal
representation of product. Hence, we use inner product of ¡query, multi-modal
representation¿ to measure the relevance between one query and the multi-modal
representation above, as shown in Figure 1(c). Thirdly, we use supervised
contrastive learning and utilize generating ability both from multi-modal
large model and large language model to filter out the false negative samples.
Finally, we fine-tune the proposed model using the manually annotated triplet
data. Our contributions can be summarized as follows:

  * •

We propose query-aware multi-modal fusion tailored for e-commerce relevance
task. Instead of divide-and-conquer approach, Query-LIFE proposes multi-modal
representation and adopt ¡query, multi-modal representation¿ similarity as the
relevance score. It effectively generates dynamic fusion representations that
incorporate product images and text based on the product types.

  * •

We also propose query-based modal alignment module utilizes supervised
contrastive learning to align the multi-modal representation of products
guided by the search query. improves the representation quality and enhances
the matching between user queries and dynamic fusion representations of
products.

  * •

Generating and Filtering (GenFilt) is designed to mitigate the impact of false
negative samples during the training process. By generating additional
positive samples and filtering out false negative ones, it helps to improve
the quality of the training data, leading to enhanced model performance and
robustness.

  * •

We carried out extensive experiments on both offline and online A/B
experiments, which have shown that it outperforms existing VLP models and
relevance models in e-commerce relevance task. Our model has been successfully
deployed in Miravia Search.

##  2\. Related Work

###  2.1. Vision-Language Pre-training

The emergence of pre-training models, such as BERT (Kenton and Toutanova,
2019), GPT3 (Brown et al., 2020), and ViT (Dosovitskiy et al., 2021), has led
to significant advancements in NLP and CV tasks, achieving state-of-the-art
results. Recently, researchers have extended the pre-training approach to the
vision-language (VL) domain, resulting in the development of several
impressive VL models (e.g. CLIP (Radford et al., 2021a) and ALIGN (Jia et al.,
2021)). These VLP models have demonstrated impressive performance in various
multi-modal downstream tasks, such as image captioning, vision question
answering, and cross-modal retrieval. They achieve this by leveraging large-
scale image-text pairs and then employing contrastive learning to align images
and text in the joint embedding space. These VLP models are divided into two
categories: object-detector (OD)-based VLP models (e.g., UNITER (Chen et al.,
2020), OSCAR (Li et al., 2020)) and end-to-end VLP models (e.g., ALBEF (Li et
al., 2021),BLIP (Li et al., 2023))). OD-based VLP models rely on bounding box
annotations during pre-training and require high-resolution images for
inference, making them both annotation-expensive and computationally
demanding. In contrast, end-to-end VLP models directly utilize image patch
features as input to a pre-trained ViT model. This eliminates the need for
costly annotations and significantly improves inference speed. As a result,
end-to-end VLP models have gained traction in recent research (Chen et al.,
2021; Kim et al., 2021). Therefore, we also adopt the end-to-end VLP model in
this paper.

###  2.2. E-commerce VLP Model

There are also some VLP models specifically targeted at e-commerce scenarios.
FashionBERT (Gao et al., 2020) was the first vision-language pre-train model
which adopts mask language loss and title-image contrastive learning. Later
on, Kaleido-BERT (Zhuge et al., 2021) further adopts several self-supervised
tasks at different scales to focus more on title-image coherence. EI-CLIP (Ma
et al., 2022) proposed an intervention-based entity-aware contrastive learning
framework. KG-FLIP (Jia et al., 2023) proposed a knowledge-guided fashion-
domain language-image pre-training framework and utilizes external knowledge
to improve the pre-training efficiency. MAKE (Zheng et al., 2023) introduces
query encoder and propose modal adaptation and keyword enhancement modules to
improve text-to-multimodal matching. However, these e-commerce VLP models
focus on multi-modal retrieval rather than relevance task, so their losses are
designed for retrieval. On the contrary, our work focuses on relevance task
and our motivation is to deeply integrate the VLP model to enhance the
product’s relevance score in e-commerce scenarios.

##  3\. Method

###  3.1. Model Architecture

In this section, we will introduce our model architecture in detail. As shown
in Figure 2, The entire model training is divided into internal alignment and
external alignment. Internal alignment is used to align the features of
product titles and images. External alignment is used to align the relevance
between user queries and products. The model architecture consists of three
modules: an image preprocessing backbone, a universal modal encoder and
GenFilt. Visual transformer (ViT) (Dosovitskiy et al., 2021) is deployed as
our image preprocessing backbone, which divides the image into patches and
encodes them as a sequence of embeddings with an additional $[CLS]$ token to
represent the global image features. The universal modal encoder includes
self-attention layer, cross-attention layer and feed-forward layer. It
extracts different modalities features through the interaction of different
layers. For the text modal, the input undergoes tokenization and then
interacts with the self-attention layer and the feed-forward layer. Similarly,
for the image modal, the input image is first processed by the ViT and follows
the same process as the text modal. In the case of multimodal features, the
text modal interacts with the image modal through the cross-attention layer
after the self-attention layer. GenFilt is designed for filtering false
negative sample during in-batch sampling.

Inspired by the relevance learning framework proposed by Jiang et al (Jiang et
al., 2019), we also propose a three-stage training framework. In the first
stage, we leverage a large set of products title-image pairs for contrastive
learning, which is pre-trained for internal alignment of products in section
3.2. In the second stage, we sample million ¡query,title,image¿ positive pairs
from the online clicking log of Miravia Search, then pre-train for external
alignent between user queries and products in the section 3.3 and 3.4. In the
third stage, we utilize manually labeled ¡query, title, image¿ triplet data to
fine-tune the alignment between products and user queries furtherly.

Figure 2. Overview of Query-LIFE. The overall training process is divided into
internal aligment and external alignment.

.

###  3.2. Vision-Language Pre-training

The VLP model utilizes the Image-Text Contrastive (ITC) loss to align the
image features and the text features, which makes positive image-text pairs
have similar representations and reduces the similarity between negative pairs
(Radford et al., 2021b; Li et al., 2021). The ITC loss has proven to be an
effective objective for enhancing vision and language representation even in
the absence of labeled data. The formula is as follows:

(1) $\mathcal{L_{ITC}}=-\frac{1}{N}\sum_{i=1}^{N}log\frac{exp(Z_{T_{i}}\cdot
Z_{I_{i}}/\tau)}{\sum_{j=1}^{N}exp(Z_{T_{j}}\cdot Z_{I_{j}}/\tau)}.$

where $Z_{T}$ and $Z_{I}$ are normalized text and image embeddings,
$Z_{I_{i}}$ is the $i$-th positive image sample in the batch, and $N$ and
$\tau$ batch size and temperature parameter respectively.

###  3.3. Query-based Modal Alignment

In e-commerce search scenarios, the relevance of products is highly dependent
on the user queries. However, user queries are short and brief, while sellers
often add redundant keywords in titles to hack the search engine indexing.
Calculating relevance solely based on query and title can easily result in
errors in relevance scoring. If misleading words or insufficient information
are present in the title, this problem becomes even more severe.

Obviously, there is an imbalance problem between query and product information
in e-commerce relevance task. To mitigate the impact of above problem on
relevance scoring, we introduce image information to improve product
representation. In addition, we introduce the concept of title-image fusion
representation for products (referred as multi-modal representation or
$\mathcal{M}$ representation). $\mathcal{M}$ representation is defined as the
interaction between the product title and the image. Unlike divide-and-conquer
approach, we adopt universal modal encoder to represent $\mathcal{M}$ and use
inner product to calculate the relevance between the item and query. To
further align the $\mathcal{M}$ representation with user queries, we adopt the
query-$\mathcal{M}$ contrastive (QMC) loss. Additionally, we incorporate the
query-title contrastive (QTC) loss to align the query with title.
Simultaneously, the query-image contrastive (QIC) loss is utilized to further
align the product image with the query. These loss functions play a crucial
role in aligning user queries and different product modalities and enhancing
the relevance scoring.

Unsupervised contrastive learning utilizes large amounts of unlabeled data to
increase the similarity between positive samples while decrease the similarity
between negative samples. However, in e-commerce relevance task, the same
query often generates positive pairs with different products. Compared with
triplet loss and unsupervised contrastive learning, supervised contrastive
learning introduces labeled negative samples and can contain more positive
samples in a mini-batch, which is more suitable for relevance task. The
formula is defined as follows:

(2) $\mathcal{L}=-\frac{1}{N}\sum_{i=1}^{N}\left\\{\frac{1}{|P(i)|}\sum_{p\in
P(i)}\left[log\frac{exp(Q_{i}\cdot
Z^{x}_{p}/\tau)}{{\sum_{j=1}^{N}}exp(Q_{i}\cdot
Z^{x}_{j}/\tau)}\right]\right\\}.$

where $P(i)$ is all positive samples in the $i$ batch, $Q$ is normalized query
embedding and $Z^{x}_{p},x\in[I,T,M]$, $p\in P(i)$, are normalized
image/text/$\mathcal{M}$ modal embedding in positive samples. $\tau$ and $N$
are temperature parameter and batch size. Bringing in different modal by
$x\in[I,T,M]$, this loss function can represent QIC, QTC and QMC loss
respectively.

###  3.4. Query-based Modal Fusion

We utilize image-text matching (ITM) to learn the $\mathcal{M}$ representation
of the product. The objective of ITM is to learn a title-image fusion
representation that captures the alignment between the image and text
modalities. In ITM, we frame the task as a binary classification problem where
the model predicts whether an image-text pair is positive or negative. To
obtain the matching score, we pass the model’s output through a two-class
linear classifier, yielding a logit. We employ a hard negative mining strategy
(Jia et al., 2021) and leverage labeled data. Hard negative mining strategy
samples negative pairs with higher contrastive similarity within a batch.
Consequently, these informative negative pairs contribute to better align
between the image and text modalities. The ITM loss can be expressed as:

(3) $\mathcal{L}_{ITM}=-E_{(I,T)\sim P}[log\\{P(y_{(I,T)}|(I,T)\\}]$

where $P$ is a distribution of in-batch samples, $y_{(I,T)}\in(0,1)$
represents whether the image $I$ and the text $T$ are matched, and
$P(y_{(I,T)}|(I,T))$ is the output of the multi-modal embedding followed by a
two-class linear classifier.

We acknowledge that image-text matching alone may not be sufficient, as
different types of products contain varying amounts of information in their
images and titles. For example, electronic products often have important
parameters listed in their titles, while clothing items rely more on visual
attributes such as material, color, and size displayed in the images. To
enable the model to learn a more effective fusion representation, we introduce
query-$\mathcal{M}$ matching (QMM). QMM not only allows the model to extract
features from both the images and titles, but also giving different weights to
each modality based on the user queries. This enables the model to generate
fused representations with a query-aware bias. QMM and ITM share the same loss
function listed in Eq. 3. Finally, the Query-LIFE model loss contains vision-
language pre-training (VLP), query-based modal alignment (QMA) and query-based
modal fusion (QMF):

(4)
$\mathcal{L}_{total}=\underbrace{\mathcal{L}_{ITC}}_{VLP}+\underbrace{\mathcal{L}_{ITM}+\mathcal{L}_{QMM}}_{QMF}+\underbrace{\mathcal{L}_{QIC}+\mathcal{L}_{QTC}+\mathcal{L}_{QMC}}_{QMA}.$

###  3.5. GenFilt

Most VLP models adopt in-batch sampling to generate image-title negative
pairs. However, in the ¡query, title, image¿ triplet data, multiple user
queries may be relevant to the multiple products. In-batch sampling will
introduce false negative samples. Those similar even same queries are
mistakenly treated as negative samples and thus compromise relevance score.

Figure 3. Overview of GenFilt. GenFilt adopts LLM and InstructBLIP for feature
generation. Then compare the similarity of query-product pairs and correct the
false negative pairs. In addition, GenFilt can also calculate the similarity
of image-title pairs and correct false negative pairs.

.

Inspired by CapFilt (Li et al., 2023), we propose a method called Generating
and Filtering (GenFilt) to address the impact of false negative samples on the
training process. It enhances the quality of the training data using large
model generation capabilities. As illustrated in Figure 3, GenFilt consists of
two modules. The first module is generating, we employ a large language model
(LLM) and a multi-modal model (InstructBLIP) (Dai et al., 2023) to extract key
text features from the product title and image respectively. The second module
is filtering, we calculate the similarity between image feature and text
feature (I-T), the similarity between query feature and image feature (Q-I)
and the similarity between query feature and text feature (Q-T). Finally, we
set a threshold $\sigma$ based on these similarities, and the similarity of
query-product pairs ((Q-I+Q-T)/2) and image-text pairs (I-T) above the
threshold are also corrected as positive samples.

##  4\. Experiments

###  4.1. Baselines and Datasets

Large-scale Industrial Datasets. There are three training datasets. The first
dataset samples 5M product title-image pairs. The second dataset samples 1.3M
¡query,title,image¿ positive pairs from the online clicking log of Miravia
Search. The third dataset has 200,000 ¡query,title,image¿ labeled data, and
30K samples are selected as the evaluation set, where the ratio of positive to
negative is 1:1.

Baselines. In our experiments, we compare Query-LIFE with several strong
baselines, including BERT (Kenton and Toutanova, 2019), CLIP-zeroshot (Radford
et al., 2021a) and BLIP2-zeroshot and BLIP2-FT(finetune) (Li et al., 2023).
CLIP and BLIP2 are two-tower models, while BERT is a single-tower and text-
modal relevance model. Only the ¡query, title¿ pairs from the ¡query, title,
image¿ triplet data are used for training. This approach does not incorporate
image information. These baselines have shown impressive performance in their
respective domains. For the CLIP and BLIP2, we concatenate the query and title
as text information and train the models with images. To ensure fair
comparison, all models are trained on the 200,000 ¡query, title, image¿
triplet data samples.

###  4.2. Experiment Implementation

We select the state-of-the-art pre-trained vision transformer ViT-g/14 from
EVA-CLIP (Fang et al., 2023) and frozen it. We remove the last layer of the
ViT and use the second last layer’s output features, which leads to slightly
better performance and is the same as BLIP2 (Li et al., 2023) setting. The
universal modal encoder is composed with 12 layers of transformers, each layer
contains self-attention layer,cross-attention layer and feed forward layer. We
initialize the universal modal encoder with the pre-trained weights of BERT-
base (Kenton and Toutanova, 2019), whereas the cross-attention layers are
randomly initialized. In total, universal modal encoder contains 188M
parameters. We train the model for 10 epochs with a batch size of 512 on 16
NVIDIA A10 GPUs. We utilize the AdamW optimizer with
$\beta_{1}=0.9,\beta_{2}=0.98$, and a weight decay rate of 0.05. For learning
rate scheduling, we employ a cosine decay strategy with a maximum learning
rate of $1e^{-4}$ and a linear warmup of 2k steps. GenFilt threshold $\sigma$
is set to 0.9.

###  4.3. Evaluation Metrics

Offline Evaluation Metrics. We consider manual annotation as the ground truth,
where relevance is indicated by the labels of 1 (relevant) and 0 (irrelevant).
This task can be treated as a classification problem. In e-commerce scenarios,
the Area Under Curve (AUC) is commonly used as the evaluation metric (Yao et
al., 2021). Additionally, we utilize the Precision-Recall curve for
evaluation. This curve represents the trade-off between precision and recall,
with recall as the x-axis and precision as the y-axis. In addition, we employ
Recall@K (R@K) as the metric. Recall@K is widely used in search and
recommendation systems. To calculate Recall@K, we randomly select 10K unique
¡query,title,image¿ triplet data from the online clicking log of Miravia
Search. For each query, we consider the ground-truth product as well as 100
other randomly sampled products as the candidate rank set. We calculate the
similarity between the query $\rightarrow$ title, query $\rightarrow$ image,
and query $\rightarrow$ $\mathcal{M}$ and sort the candidate rank set based on
these similarities. Recall@K measures the percentage of ground-truth matches
that appear in the top-K ranked list (Gao et al., 2020).

Table 2. Offline results compared with different baselines. | Query $\rightarrow$ Title | Query $\rightarrow$ Image | Query $\rightarrow$ $\mathcal{M}$
---|---|---|---
R@5 | R@10 | R@20 | AUC | R@5 | R@10 | R@20 | AUC | R@5 | R@10 | R@20 | AUC
BERT | 0.142 | 0.186 | 0.351 | 0.871 | - | - | - | - | - | - | - | -
CLIP-Zeroshot | 0.068 | 0.125 | 0.272 | 0.542 | 0.068 | 0.147 | 0.272 | 0.554 | - | - | - | -
BLIP2-Zeroshot | 0.134 | 0.204 | 0.318 | 0.565 | 0.090 | 0.159 | 0.272 | 0.575 | - | - | - | -
BLIP2-FT | 0.113 | 0.170 | 0.272 | 0.752 | 0.056 | 0.159 | 0.316 | 0.771 | - | - | - | -
Query-LIFE | 0.125 | 0.215 | 0.340 | 0.865 | 0.079 | 0.204 | 0.329 | 0.871 | 0.102 | 0.215 | 0.386 | 0.891
Query-LIFE w/o QMA | 0.068 | 0.170 | 0.318 | 0.741 | 0.079 | 0.147 | 0.306 | 0.805 | 0.068 | 0.193 | 0.329 | 0.784
Query-LIFE w/o QMF | 0.136 | 0.207 | 0.318 | 0.856 | 0.090 | 0.147 | 0.306 | 0.863 | 0.113 | 0.193 | 0.295 | 0.877
Query-LIFE w/o GenFilt | 0.102 | 0.147 | 0.261 | 0.816 | 0.056 | 0.147 | 0.329 | 0.835 | 0.090 | 0.136 | 0.295 | 0.849

Online Evaluation Metrics. We adopt number of orders (Order_cnt), average
number of buyer (Order_uv), and GMV (Gross Merchandise Volume,total value of
sales) as online evaluation metrics. These metrics reflect changes in user
order transactions.

Human Evaluation. We sampled 1,000 queries, and selected the top-10 query-item
pairs of the exposure page for each query to perform human relevance
evaluation. The relevance of a query-item can be divided into three types:
Excellent, Fair and Bad. Excellent means same as the original highly standard,
the item’s core products, functional attributes and other attributes perfectly
match the query requirements. Fair means the core product is the same as
query, but the functional attributes are inconsistent. Bad means the core
products are different or the core products are the same, but the retained
attributes in the brand or other key industries are different. We count the
proportions of the three indicators of different models.

###  4.4. Offline Experiments

The previous models calculates the cosine similarity between query embedding
and title embedding (Query $\rightarrow$ Title) or image embedding (Query
$\rightarrow$ Image). In Query-LIFE, we introduce another method which
calculates the cosine similarity between query embedding and $\mathcal{M}$
embedding (Query $\rightarrow$ $\mathcal{M}$).

AUC of different models can be found in Table 2. It is worth noting that the
AUC of general VLP models (CLIP, BLIP2) is lower than that of BERT, because
these general VLP models only focus on the internal alignment of product
titles and images, ignoring the imbalance problem between query and products,
while BERT performs external alignment of query and product. This shows that
external alignment improves query and product relevance. In addition, AUC for
query$\rightarrow$$\mathcal{M}$ proposed by Query-LIFE is 0.021(0.891-0.871)
higher than BERT. It is shown that the relevance score is effectively improved
by introducing image information and external alignment of query-product.

Moreover, In Query-LIFE, when comparing the different relevance score inner
products, we observe that the AUC of query $\rightarrow$ $\mathcal{M}$ is also
the highest. This indicates that $\mathcal{M}$ representation can provide a
more comprehensive and robust representation compared to using a single
modality (text or image). To further evaluate these methods, we also plot
Precision-Recall (PR) curves of three relevance score inner products in Figure
4. It can be observed that the PR curve of query $\rightarrow$ $\mathcal{M}$
consistently outperforms the others, indicating the superiority of the query
$\rightarrow$ $\mathcal{M}$.

Figure 4. Precision-Recall curve of three types of representation

.

The R@K results for different models are also presented in Table 2. Firstly,
general VLP model’s R@K is lower than BERT and Query-LIFE, further
illustrating the impact of query-product external alignment on the relevance
score. Secondly, at R@10 and R@20, query$\rightarrow$$\mathcal{M}$ of Query-
LIFE is 0.029 (0.215-0.186), 0.035 (0.386-0.351) higher than
query$\rightarrow$title of BERT. Query-LIFE enhances the external alignment
between the product and query, and effectively enhance $\mathcal{M}$
representation by making full use of image information.

As shown in Figure 1(b), previous VLP models usually adopt divide-and-conquer
approach to utilize image information in e-commece relevance task. They
extract query, title, and image representations, and then add
query$\rightarrow$title (Q$\rightarrow$T) and query$\rightarrow$image
(Q$\rightarrow$I) as the relevance score. It assumes equal contribution of
product text and image for the relevance judgement given any types of
products. Hence, we compare the performance of query$\rightarrow$$\mathcal{M}$
and divide-and-conquer approach. As shown in Table 3,
query$\rightarrow$$\mathcal{M}$ outperforms the divide-and-conquer approach in
all metrics. It clearly demonstrates the advantage of dynamic weighting for
different types of products.

Table 3. The R@K and AUC of divide-and-conquer approach and query $\rightarrow$$\mathcal{M}$. | Model | R@5 | R@10 | R@20 | AUC
---|---|---|---|---|---
$\frac{Q\rightarrow T+Q\rightarrow I}{2}$ | BLIP2-FT | 0.090 | 0.181 | 0.329 | 0.781
Query-LIFE | 0.079 | 0.215 | 0.318 | 0.882
Query$\rightarrow$$\mathcal{M}$ | Query-LIFE | 0.102 | 0.215 | 0.386 | 0.891

###  4.5. Online Experiment

Furthermore, we carry out online A/B experiments, and BERT is the current
baseline in Miravia Search. Annotators are invited to evaluate whether the
relevance is improved by the proposed method. 10K query-item pairs are sampled
top-10 items both from the buckets of BERT and Query-LIFE respectively. The
results are shown in Table 4, compared with BERT, the main improvement is that
the Excellent ratio increased by 4.42% and the Bad ratio decreased by 2.79%.
Since the search relevance is one key aspect of user experience, the
conversion efficiency is also improved as a consequence. As shown in Table 5,
all the efficiency metrics are increased. The results verified that Query-LIFE
can attracts higher conversions for our platform. Query-LIFE has been deployed
online and brings stable conversion improvements for Miravia Search.

Table 4. Results of human evaluation. | Excellent | Fair | Bad
---|---|---|---
Query-LIFE | +4.42% | +2.17% | -2.79%
Table 5. Online A/B tests of Query-LIFE. | Order_cnt | Order_uv | GMV
---|---|---|---
Query-LIFE | +4.11% | +3.06% | +3.19%

##  5\. Ablation Studies

###  5.1. Effect of QMA and QMF

In this section, we will demonstrate what important role QMA and QMF play for
Query-LIFE. As listed in Table 2, Query-LIFE w/o QMA, Query-LIFE w/o QMF are
compared with Query-LIFE. It is seen that QMA significantly improves both R@K
and AUC by enhancing the similarity between query and products. Without QMA
module, AUC of query$\rightarrow$title, query$\rightarrow$image and
query$\rightarrow$$\mathcal{M}$ decreases 0.124 (0.865-0.741), 0.066
(0.871-0.805) and 0.107 (0.891-0.784) respectively. In
query$\rightarrow$$\mathcal{M}$, R@K decreases 0.034 (0.102-0.068), 0.022
(0.215-0.193) and 0.057 (0.386-0.329) respectively. Because in e-commerce
scenarios, product relevance is closely related to user queries, QMA aligns
user queries and different modalities of positive samples through contrastive
learning, strengthening relevance score.

QMF is designed to smartly extract information with different weights based on
product types. For verification, we sample several different types of products
and adopt Query-LIFE to extract text, image and $\mathcal{M}$ representation
respectively. Then we calculate the similarity between $\mathcal{M}$
representation and text&image representation respectively and then normalize
them. As shown in Table 6, without QMF, the model assigns equal weights to the
image and text representation of different types of products. However, Query-
LIFE dynamically adjusts the weights assigned to both image and text to better
represent the product. In particular, when it comes to dresses, the model
gives more attention to the images than text. Compared with title, image can
describe the material, size, pattern and other visual information of the dress
more intuitively. Similarly, it also works for other product types. In
addition, without QMF module, AUC of query$\rightarrow$title,
query$\rightarrow$image and query$\rightarrow$$\mathcal{M}$ drops 0.009
(0.865-0.856), 0.008 (0.871-0.863) and 0.014 (0.891-0.877) respectively.
Therefore it makes sense by giving more weight to the images.

Table 6. The weight of different models on image information. | Dress | Monitor | Phone
---|---|---|---
Query-LIFE | 0.71 | 0.56 | 0.35
Query-LIFE w/o QMF | 0.49 | 0.52 | 0.51

(a) BLIP2-zeroshot

(b) BLIP2-FT

(c) Query-LIFE w/o GenFilt

(d) Query-LIFE

Figure 5. Embedding space visualization of different models. The purple cross
is the different modal representation of the negative samples, the blue dot is
the query representation, the orange square is the $\mathcal{M}$
representation of the positive samples, the green five-pointed star is the
title representation of the positive samples, and the red diamond is the image
representation of the positive samples.

###  5.2. Effect of GenFilt

We fine-tune Query-LIFE w/o GenFilt and list AUC and R@K in Table 2. Taking
query $\rightarrow$ $\mathcal{M}$ as an example, compared with Query-LIFE, AUC
decreases by 0.042 (0.891-0.849), R@5, R@10, and R@20 decreases by 0.012
(0.102-0.090), 0.079 (0.215-0.136) and 0.091 (0.386-0.295) respectively. This
further shows that in the ¡query,title,image¿ data, in-batch sampling will
sample some false negative samples and affect model performance.

We visualize the embedding space of different relevance models
(BLIP2-zeroshot, BLIP2-FT, Query-LIFE w/o GenFilt and Query-LIFE) in Figure
5(a)-(d) respectively. In detail, we sampled positive samples and negative
samples for a query. In Figure 5(a), the similarity between the query and its
positive products does not widen the gap with other negative products. After
vision-language pre-training (BLIP2-FT), query and different modals of
positive products closer in Figure 5(b), while query and some negative samples
are still confused. While GenFilt is not used in Figure 5(c), the performance
is similar with BLIP2-FT in Figure 5(b). Figure 5(d) shows the Query-LIFE
results, positive samples are more clustered and are further away from
negative samples, compared with Figure 5(c), it intuitively demonstrates the
enhancement of model performance by GenFilt.

Finally, we select some cases to show the generation capability of LLM and
InstructBLIP, as shown in Table 7, GenFilt can extract key information such as
core products, materials, brand,colors, etc. from titles and images through
LLM and InstructBLIP. We adopt these prompts:

  * •

LLM: As a product search engine, please understand the input of the product
title, extract the core word, material, brand, color, and model parameters
from the title and provide structured output. The input title: title To solve
the problem, please execute the following steps: Firstly, understand the input
product title and extract the vocabulary that describes the main product as
the core word. Secondly, analyze the main material of the product and replace
it with ”NULL” if none is specified. Thirdly, analyze the brand of the product
and replace it with ”NULL” if none is specified. Firthly, analyze the color of
the product and replace it with ”NULL” if not specified. Finally, output the
structured parsing results.

  * •

InstructBLIP: Briefly summarize the items in the picture in a few words.

Table 7. GenFilt Results for some products. Image |  Title |  LLM |  InstructBLIP
---|---|---|---
|  Women’s long jacket with hood houndstooth print black and white casual autumn |  Women long white jacket |  Woman white jacket
|  Kid Princess Dress Girl Summer Fancy Party Clothes Girls Aurora Rapunzel Cinderella Sleeping Beauty Christmas Carnival Costume |  Dress princess summer fancy party |  A pink princess dress and accessories
|  EUDORA 925 Sterling Silver Viking Odin God Medal Necklace for Man Women Cool Vikings Circle Norse Runes Amulet Pendant Jewelry |  Necklace silver viking odin god medal |  A silver pendant with an angel on it
|  Gold Blade Chain Choker Necklace Women Sexy Flat Snake Chain Gift |  Gold necklace blade chain choker |  A gold chain necklace

##  6\. Conclusion

In this paper, we propose a novel approach for learning multi-modal
representation of product in e-commerce search relevance. We design a query-
based multi-modal fusion module which effectively generates dynamic fusion
representations that incorporate product image and text based on the product
types. We propose query-based modal alignment module which utilizes supervised
contrastive learning to align the multi-modal representation of products
guided by the search query. Additionally, we propose the GenFilt module,
leveraging LLM (large language model) and the generation capability extracting
information from image to text to address the problem of false negative
sampling in contrastive learning. Experimental results demonstrate that Query-
LIFE outperforms existing baselines in both the relevance task. Moreover,
Query- LIFE has been successfully deployed in Miravia Search, leading to
improvements both in search relevance and the conversion rate.

## References

  * (1)
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Chang et al. (2021) Wei-Cheng Chang, Daniel Jiang, Hsiang-Fu Yu, Choon Hui Teo, Jiong Zhang, Kai Zhong, Kedarnath Kolluri, Qie Hu, Nikhil Shandilya, Vyacheslav Ievgrafov, et al. 2021\.  Extreme multi-label learning for semantic matching in product search. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_. 2643–2651.
  * Chen et al. (2021) Kezhen Chen, Qiuyuan Huang, Yonatan Bisk, Daniel McDuff, and Jianfeng Gao. 2021.  Kb-vlp: Knowledge based vision and language pretraining. In _ICML, workshop_.
  * Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020.  UNITER: UNiversal Image-TExt Representation Learning. In _European Conference on Computer Vision_. 104–120.
  * Dai et al. (2023) Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023.  InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning.  arXiv:2305.06500
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2021\.  An image is worth 16x16 words: Transformers for image recognition at scale.  _ICLR_ (2021).
  * Fang et al. (2023) Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. 2023.  Eva: Exploring the limits of masked visual representation learning at scale. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 19358–19369.
  * Gao et al. (2020) Dehong Gao, Linbo Jin, Ben Chen, Minghui Qiu, Peng Li, Yi Wei, Yi Hu, and Hao Wang. 2020.  Fashionbert: Text and image matching with adaptive loss for cross-modal retrieval. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 2251–2260.
  * Hu et al. (2014) Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014.  Convolutional neural network architectures for matching natural language sentences.  _Advances in neural information processing systems_ 27 (2014).
  * Huang et al. (2013) Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013.  Learning deep structured semantic models for web search using clickthrough data. In _Proceedings of the 22nd ACM international conference on Information & Knowledge Management_. 2333–2338.
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021.  Scaling up visual and vision-language representation learning with noisy text supervision. In _International conference on machine learning_. PMLR, 4904–4916.
  * Jia et al. (2023) Qinjin Jia, Yang Liu, Daoping Wu, Shaoyuan Xu, Huidong Liu, Jinmiao Fu, Roland Vollgraf, and Bryan Wang. 2023.  KG-FLIP: Knowledge-guided Fashion-domain Language-Image Pre-training for E-commerce. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track)_. 81–88.
  * Jiang et al. (2019) Yunjiang Jiang, Yue Shang, Rui Li, Wen-Yun Yang, Guoyu Tang, Chaoyi Ma, Yun Xiao, and Eric Zhao. 2019.  A unified neural network approach to e-commerce relevance learning. In _Proceedings of the 1st International Workshop on Deep Learning Practice for High-Dimensional Sparse Data_. 1–7.
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of NAACL-HLT_. 4171–4186.
  * Kim et al. (2021) Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021.  Vilt: Vision-and-language transformer without convolution or region supervision. In _International Conference on Machine Learning_. PMLR, 5583–5594.
  * Li et al. (2023) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ (2023).
  * Li et al. (2021) Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021.  Align before fuse: Vision and language representation learning with momentum distillation.  _Advances in neural information processing systems_ 34 (2021), 9694–9705.
  * Li et al. (2020) Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020\.  Oscar: Object-semantics aligned pre-training for vision-language tasks. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX 16_. Springer, 121–137.
  * Ma et al. (2022) Haoyu Ma, Handong Zhao, Zhe Lin, Ajinkya Kale, Zhangyang Wang, Tong Yu, Jiuxiang Gu, Sunav Choudhary, and Xiaohui Xie. 2022.  Ei-clip: Entity-aware interventional contrastive learning for e-commerce cross-modal retrieval. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 18051–18061.
  * Radford et al. (2021a) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021a.  Learning transferable visual models from natural language supervision. In _International conference on machine learning_. PMLR, 8748–8763.
  * Radford et al. (2021b) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021b.  Learning transferable visual models from natural language supervision. In _International conference on machine learning_. PMLR, 8748–8763.
  * Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009\.  The probabilistic relevance framework: BM25 and beyond.  _Foundations and Trends® in Information Retrieval_ 3, 4 (2009), 333–389.
  * Wang et al. (2023) Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. 2023\.  Image as a Foreign Language: BEiT Pretraining for Vision and Vision-Language Tasks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 19175–19186.
  * Wang et al. (2021) Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2021.  SimVLM: Simple Visual Language Model Pretraining with Weak Supervision. In _International Conference on Learning Representations_.
  * Yao et al. (2021) Shaowei Yao, Jiwei Tan, Xi Chen, Keping Yang, Rong Xiao, Hongbo Deng, and Xiaojun Wan. 2021.  Learning a product relevance model from click-through data in e-commerce. In _Proceedings of the Web Conference 2021_. 2890–2899.
  * Zheng et al. (2023) Xiaoyang Zheng, Zilong Wang, Sen Li, Ke Xu, Tao Zhuang, Qingwen Liu, and Xiaoyi Zeng. 2023.  MAKE: Vision-Language Pre-training based Product Retrieval in Taobao Search. In _Companion Proceedings of the ACM Web Conference 2023_. 356–360.
  * Zhuge et al. (2021) Mingchen Zhuge, Dehong Gao, Deng-Ping Fan, Linbo Jin, Ben Chen, Haoming Zhou, Minghui Qiu, and Ling Shao. 2021.  Kaleido-bert: Vision-language pre-training on fashion domain. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 12647–12657.


# Towards Few-shot Entity Recognition in Document Images:
A Label-aware Sequence-to-Sequence Framework

Zilong Wang
University of California, San Diego
<EMAIL_ADDRESS>
&Jingbo Shang∗
University of California, San Diego
<EMAIL_ADDRESS>

###### Abstract

††$*$ Jingbo Shang is the corresponding author.

Entity recognition is a fundamental task in understanding document images.
Traditional sequence labeling frameworks treat the entity types as class IDs
and rely on extensive data and high-quality annotations to learn semantics
which are typically expensive in practice. In this paper, we aim to build an
entity recognition model requiring only a few shots of annotated document
images. To overcome the data limitation, we propose to leverage the label
surface names to better inform the model of the target entity type semantics
and also embed the labels into the spatial embedding space to capture the
spatial correspondence between regions and labels. Specifically, we go beyond
sequence labeling and develop a novel label-aware seq2seq framework, LASER.
The proposed model follows a new labeling scheme that generates the label
surface names word-by-word explicitly after generating the entities. During
training, LASER refines the label semantics by updating the label surface name
representations and also strengthens the label-region correlation. In this
way, LASER recognizes the entities from document images through both semantic
and layout correspondence. Extensive experiments on two benchmark datasets
demonstrate the superiority of LASER under the few-shot setting.

##  1 Introduction

Entity recognition lies in the foundation of document image understandings,
which aims at extracting word spans that perform certain roles from the
document images, such as header, question. Distinct from the text-only named
entity recognition task, the document images, such as forms, tables, receipts,
and multi-columns, provide a perfect scenario to apply multi-modal techniques
into practice where the rich layout formats in such document images serve as
the new, complementary signals for entity recognition performance in addition
to the existing textual data.

Recent methods (Xu et al., 2020; Hong et al., 2020; Garncarek et al., 2021)
follow the traditional sequence labeling framework to extract the word spans
using the standard IOBES tagging schemes (Marquez et al., 2005; Ratinov and
Roth, 2009) in named entity recognition tasks. Entity types are treated as
class IDs and the semantics of the label surface names are ignored. These
methods also largely extend the label space by including combinations of the
boundary identifiers (B, I, E, S) and entity types. For instance, when there
are 3 target entity types, the extended label space would have $13$ (i.e.,
$4\times 3+1$) dimensions. As a result, they fail to learn from the data
efficiently and require extensive datasets and high-quality annotations to
create the connection between entities and their entity types. Meanwhile,
document images typically include various formats and have a high diversity of
entities within each page. It is expensive or almost impossible to enumerate
all required entity types and obtain enough annotated data for them. Moreover,
ethical concerns would arise when it comes to the receipts or consent forms,
which makes it even harder to collect enough data.

Due to the inefficiency of traditional methods and the data limitation in real
application scenarios, it is necessary to resort to few-shot learning for
entity recognition in document images. We aim at exploiting the potential of a
limited number of training pages and try to generalize our model on the much
larger number of new pages for testing. In our method, we go beyond the
sequence labeling framework and reformulate the entity recognition as a
sequence-to-sequence task. Specifically, we propose a new generative labeling
scheme for entity recognition — the label surface name is explicitly generated
right after each entity as a part of the target sequence. In this way,
different entity types are no longer independent dimensions in the label space
and models can leverage the semantic connect between the entities and entity
types.

To this end, we propose a label-aware sequence-to-sequence framework for
entity recognition, LASER. Our implementation is based on pre-trained language
model LayoutReader 111Licensed under the MIT License (Wang et al., 2021),
which is a layout-aware pre-trained sequence-to-sequence model.

As shown in Figure 1, LASER extends the architecture of LayoutReader for our
proposed generative labeling scheme to better solve the few-shot entity
recognition task for document images. Specifically, after generating certain
word spans, the model can choose to generate either the following words in the
source sequence or label surface names. The entity labels are explicitly
inserted in the generated sequence so that the probability of the entity types
conditioned on the entity, ${P}(\mbox{type}|\mbox{entity})$, can be maximized
not only by the signals from the training data but also by the knowledge from
the pre-training of the language models. We also embed the label surface names
into the spatial embedding space, so the generation of labels is also aware of
the correlation between labels and the regions in the page.

Benefit from the novel generative labeling scheme and the semantics of labels,
LASER is able to effectively recognize entities in document images with only a
limited number of training samples. In contrast, the sequence labeling models
use less efficient tagging scheme, thus requiring more data and failing in the
few-shot settings.

We validate LASER using two benchmarks, FUNSD (Guillaume Jaume, 2019) and
CORD-Lv1 (Park et al., 2019). Both datasets are from real scenarios and fully-
annotated with textual contents and bounding boxes. We compare our model with
strong baselines and study the label-entity semantic and spatial correlations.
We summarize our contribution as follows.

  * •

We reformulate the entity recognition task and propose a new generative
labeling scheme that embeds the label surface names into the target sequence
to explicitly inform the model of the label semantics.

  * •

We propose a novel label-aware sequence-to-sequence framework LASER to better
handle few-shot entity recognition tasks for document images than the
traditional sequence labeling framework using both label semantics and layout
format learning.

  * •

Extensive experiments on two benchmark datasets demonstrate the effectiveness
of LASER under few-shot settings.

Reproducibility. We will release the code and datasets on
Github222github.com/zlwang-cs/LASER-release.

Figure 1: The Framework of LASER: [B], [E], [T] denote the boundaries; $\tau$,
$\tau^{\prime}$, $\tau^{\prime\prime}$ are the label surface names; (a) is the
process of generative labeling scheme; (b) shows the alignment of the spatial
identifiers and embeddings.

##  2 Problem Formulation

The few-shot entity recognition in the document images is to take the text and
layout inputs from a limited number of training samples to predict the
boundary of each entity and classify the entity into categories. Given a
document image page $\mathcal{P}$, the words within the page are annotated
with their textual contents $w$ and the bounding boxes
$B=(x_{0},y_{0},x_{1},y_{1})$ (top-left and bottom-right corners) by human
annotators or the OCR engines, and all the words and bounding boxes are listed
in a sequence serving as the inputs from textual and layout modalities. In
this way, the entities are spans of these words referring to precise concepts,
which makes it possible to conduct entity recognition using sequence labeling
or generative labeling scheme. We randomly select a small subset of training
samples and evaluate the performance under the $k$-shot training, where $k$
denotes the number of the training samples.

##  3 Our Generative Labeling Scheme

We propose our labeling scheme of entity recognition in the generative manner
which generates the entity boundaries and the label surface names explicitly.
Specifically, given an entity $e=[w_{i},w_{i+1}...,w_{j}]$, we use the [B] and
[E] to denote the boundary of the entity and append the label surface name
afterwards. Overall, the generative formulation is to generate:

$\displaystyle
w_{i-1},\textbf{{[B]}},w_{i},...,w_{j},\textbf{{[E]}},\mathbf{\tau_{1}},...,\mathbf{\tau_{k}},\textbf{{[T]}},w_{j+1}$

where [B] and [E] denote the start and end of the entity;
$\tau_{1}...\tau_{k}$ are the words in the label surface name; [T] denotes the
end of the label surface name. For example, “Sender” and “Charles Duggan” are
a pair of question and answer from a document image. According to the
generative labeling scheme, the corresponding generated sequence is that: [B]
Sender [E] question [T] [B] Charles Duggan [E] answer [T].

##  4 Our LASER Framework

In this section, we introduce our label-aware sequence-to-sequence framework
for entity recognition in document images. First, we introduce our method in a
bird’s eye view. Then we dive into the details of each part including the
multi-modal prefix language model, the label-aware generation.

###  4.1 Overview

Our proposed LASER is a label-aware sequence-to-sequence model for entity
recognition in document images. The framework is shown in Figure 1. The model
follows the prefix language model paradigm (Raffel et al., 2019; Dong et al.,
2019a; Bao et al., 2020) and is built upon the pre-trained language model,
LayoutReader (Wang et al., 2021). With extensive knowledge learned in pre-
training stage, the model leverages the semantic meaning of label surface
names during generation.

Since the functional tokens (e.g. [B], [E]) and the label surface names are
foreign words in the given page, their layout features are nonexistent. We use
trainable vectors as special layout identifiers for these extra tokens and
these vectors are well aligned into the spatial embedding space. In this way,
the spatial correspondence between layout formats and labels can be learned.

To reinforce the model to distinguish the functional tokens (e.g. [B], [E])
and ordinary words, an extra binary classification module is added, and the
probability is used in the next token prediction.

Equipped with all the components, our proposed model is able to conduct entity
recognition efficiently and effectively under the few-shot setting.

###  4.2 Multi-modal Prefix LM

LASER is built on the layout-aware prefix language model, LayoutReader (Wang
et al., 2021). Prefix language model refers to a multi-layered Transformer
where the source sequence and target sequence are packed together and a
“partially-triangle” mask is used to control the attention between tokens in
the two sequences. In LASER, the source sequence has full self-attention and
the target sequence only attends to the previous tokens so the conditional
generative probability is learned.

#### Input Embedding

The input embedding layer of LASER includes the word embedding, spatial
embedding, and positional embedding. We normalize and round the bounding box
coordinates to integers ranging from 0 to 1000, and embed them as trainable
vectors as spatial embeddings (Xu et al., 2020, 2021a, 2021b; Wang et al.,
2021). So the input embeddings of the ordinary words are as follows:

$\displaystyle
e_{w_{i}}=\mbox{WordEmb}(w_{i})+\mbox{SpatialEmb}(B_{i})+\mbox{PosEmb}(i)$

where WordEmb, SpatialEmb, PosEmb are the word embedding, the spatial
embedding, and the positional embedding lookup tables, respectively; $i$ is
the index of the word in the packed sequence.

The functional tokens and label surface names are new tokens in the given
page. We cannot extract the layout features from the bounding boxes of them
because their bounding boxes are nonexistent. Instead of the actual bounding
boxes, we design unique embedding vectors for each new tokens as their layout
identifiers. These identifiers can perform in the same way as real bounding
boxes during training to embed the functional tokens and label surface names
into the spatial embedding space. The input embedding replaces the spatial
embedding with the spatial identifiers:

$\displaystyle
e_{\lambda}=\mbox{WordEmb}(\lambda)+\mbox{SpatialID}(\lambda)+\mbox{PosEmb}(i)$

where SpatialID is the spatial identifier lookup table; $i$ is the index of
the word in the packed sequence;
$\lambda\in\\{\texttt{[B]},\texttt{[E]},\texttt{[T]},\tau_{1},...,\tau_{t}\\}$.

Within the input embedding layer, the pre-trained model learns the semantic
and layout formats from word embeddings or spatial features. The spatial
embeddings are already pre-trained and further fine-tuned in the downstream
tasks, and the spatial identifiers are new to the model and completely trained
in the downstream tasks.

#### Attention Mask

As mentioned, LASER depends on a “partially-triangle” mask to realize
sequence-to-sequence training within one encoder. To be more specific, the
“partially-triangle” attention mask has two parts, the source part and the
target part. In the source part, the tokens can attend to each other, which
enables the model to be aware of the entire sequence. In the target part, to
predict the next token in a sequence-to-sequence way, we design the “triangle”
mask which prevents the tokens from attending to the tokens after them.
Therefore, the generative probability conditioned on the previous tokens can
be computed.

#### Output Hidden States

To learn the conditional generative probability of the next token, we take the
output hidden states corresponding to the target sequence which is denoted as
$\mathbf{h}_{n+1},\mathbf{h}_{n+2},...,\mathbf{h}_{n+m}$, where $n+1$ is the
beginning of the target sequence in the packed sequence. According to the
“partially-triangle” attention mask, $\mathbf{h}_{n+k}$ is produced with the
attention to the source tokens and the previous target tokens, i.e., the input
embeddings whose index ranges from $1$ to $n+k$. Therefore, $\mathbf{h}_{n+k}$
is used to predict the $(k+1)$-th token in the target sequence.

###  4.3 Label-aware Generation

In the sequence-to-sequence setting, LASER estimates the probability of next
token conditioned on the previous context, i.e. $P(x_{k}|x_{<k})$ and
$x_{k}\in\mathcal{C}$, where
$\mathcal{C}=\\{w_{1}...w_{n}\\}\cup\\{\tau_{1}...\tau_{t}\\}\cup\\{\texttt{[B]},\texttt{[E]},\texttt{[T]}\\}$
is the set of all candidate words. Following LayoutReader, we restrain the
candidates within the source words instead of the whole dictionary, and we go
beyond it and extend the candidate set to include the functional tokens and
label surface names. Moreover, to distinguish whether the next word belongs to
the source or not, we design an extra binary classification module.

Specifically, we take the hidden states $h_{k}$ to predict whether the next
token is from the source or not. We denote the probability
$P(x_{k+1}\in\mbox{src})=p_{k+1}$. Then we use $p_{k+1}$ to weight the next
token prediction. The probability that the next token is the $i$-th word in
the source is computed as follows:

$\displaystyle P(x_{k+1}=w_{i}|x_{\leq
k})=\frac{p_{k+1}\exp{\left(\mathbf{e}_{w_{i}}^{T}\mathbf{h}_{k}+{b}_{k}\right)}}{\sum_{j}\exp{\left(\mathbf{e}_{w_{j}}^{T}\mathbf{h}_{k}+{b}_{k}\right)}}$

where $w_{i}$ is the $i$-th word in the source; $\mathbf{e}_{w_{i}}$ is the
input embedding of $w_{i}$; $\mathbf{b}_{k}$ is the bias.

Similarly, the probability that the next token is one of the functional tokens
or label surface names is computed as follows:

$\displaystyle P(x_{k+1}=\lambda|x_{\leq
k})=\frac{(1-p_{k+1})\exp{\left(\mathbf{e}_{\lambda}^{T}\mathbf{h}_{k}+{b}^{\prime}_{k}\right)}}{\sum_{\lambda^{\prime}}\exp{\left(\mathbf{e}_{\lambda^{\prime}}^{T}\mathbf{h}_{k}+{b}^{\prime}_{k}\right)}}$

where $\lambda$ is a functional token or label surface name, i.e.
$\lambda\in\\{\texttt{[B]},\texttt{[E]},\texttt{[T]},\tau_{1},...,\tau_{t}\\}$;
$1-p_{k+1}$ is the probability that $(k+1)$-th token is a functional token or
label surface name; $\mathbf{b}^{\prime}_{k}$ is the bias.

#### Label Semantics Learning

With the log likelihood loss of generative language modeling, the model
maximize the dot production between the hidden states $\mathbf{h}$ and the
input embeddings $e$. The semantic correlation is learned considering that the
input embeddings of the labels surface names are encoded in the word
embeddings.

#### Spatial Identifier Learning

From the layout format perspective, the input embedding of the label surface
names also includes the spatial identifiers. When predicting the next token,
the log likelihood also strengthens the relation between the spatial
identifiers and the layout context. In this way, LASER inserts the spatial
identifiers into the hyperspace of the spatial embeddings. In other words,
LASER predicts where a certain label is more likely to be. Similar to the
joint probability of language modeling, LASER maximizes the joint probability
of a mixture of spatial identifiers and spatial embeddings:
$P(...,B_{k-1},B_{k},\tau,B_{k+1},...)$ where $B_{k}$ is the bounding boxes of
the words in the page and the $\tau$ is the label to predict. Further
visualization is conducted in Section 5.7.

###  4.4 Sequential Decoding

After training, LASER follows the prefix language modeling paradigm and
generates the target sequence sequentially. We input the source sequence into
the model and take the last hidden states to predict the first token in the
target. Then we append the result to the end of input and repeatedly run the
generation. We cache the states of the model and achieve generation in linear
time.

##  5 Experiments

In this section, we conduct experiments and ablation study on FUNSD (Guillaume
Jaume, 2019) and CORD-Lv1 Park et al. (2019) under few-shot settings. We
replace the original label surface names with other tokens to study the
importance of semantic meaning. We also plot the heatmaps of the similarity
between the spatial identifiers and the spatial embeddings to interpret the
spatial correspondence. Case studies are also conducted.

###  5.1 Experimental Setups

All the experiments are under few-shot settings using 1, 2, 3, 4, 5, 6, 7
shots. We use 6 different random seeds to select the few-shot training samples
and the data augmentation is conducted to solve the data sparsity. We train
all the models using the same data and compute the average performance and the
standard deviation. We only report the result of 1, 3, 5, 7 shots for space
limitation. To evaluate our model, we first convert our results into IOBES
tagging style and compute the word-level precision, recall, and F-1 score
using the APIs from Nakayama (2018) so that all comparisons with sequence
labeling methods are under the same metrics. We believe such experiment
settings guarantee the results are representative.

###  5.2 Datasets

Our experiments are conducted on two real-world data collections: FUNSD and
CORD-Lv1. Both datasets provide rich annotations for the document image
understandings includes the words and the word-level bounding boxes. The
details and statistics of these two datasets are as follows.

  * •

FUNSD: FUNSD consists of 199 fully-annotated, noisy-scanned forms with various
appearance and format which makes the form understanding task more
challenging. The word spans in this datasets are annotated with three
different labels: header, question and answer, and the rest words are
annotated as other. We use the original label names.

  * •

CORD-Lv1: CORD consists of about 1000 receipts with annotations of bounding
boxes and textual contents. The entities have multi-level labels. We select
the first level and denote the dataset as CORD-Lv1. The first level includes
menu, void-menu, subtotal and total. We simplify subtotal as sub and void-menu
as void.

Table 1: Dataset Statistics.

Dataset | # Train Pages | # Test Pages | # Entities / Page
---|---|---|---
FUNSD | 149 | 50 | 42.86
CORD-Lv1 | 800 | 100 | 13.82

Table 2: Evaluation Results with Different Sizes of Few-shot Training Samples:
Bold denotes the best model; Underline denotes the second-best model.

$|\mathcal{P}|$ | Model | FUNSD | CORD-Lv1
---|---|---|---
Precision | Recall | F-1 | Precision | Recall | F-1
1 | BERT | 9.62$\pm$2.24 | 24.14$\pm$3.46 | 13.55$\pm$2.09 | 30.64$\pm$2.80 | 45.60$\pm$3.45 | 36.64$\pm$3.10
RoBERTa | 9.29$\pm$1.57 | 22.06$\pm$5.64 | 12.76$\pm$1.91 | 30.66$\pm$4.25 | 44.39$\pm$6.72 | 36.25$\pm$5.18
LayoutLM | 11.39$\pm$1.12 | 24.73$\pm$7.38 | 15.18$\pm$2.17 | 33.27$\pm$7.32 | 49.49$\pm$10.26 | 39.77$\pm$8.47
LayoutReader | 11.32$\pm$0.62 | 22.53$\pm$4.80 | 14.84$\pm$1.25 | 32.17$\pm$4.64 | 45.61$\pm$6.54 | 37.70$\pm$5.31
LASER | 30.40$\pm$4.89 | 35.20$\pm$7.20 | 32.36$\pm$5.14 | 47.63$\pm$3.90 | 45.52$\pm$5.84 | 46.24$\pm$3.01
3 | BERT | 16.42$\pm$4.30 | 34.74$\pm$5.36 | 22.19$\pm$5.05 | 39.62$\pm$3.99 | 56.65$\pm$4.03 | 46.58$\pm$3.94
RoBERTa | 16.71$\pm$3.63 | 31.28$\pm$3.55 | 21.66$\pm$3.84 | 44.51$\pm$4.69 | 60.18$\pm$4.69 | 51.15$\pm$4.70
LayoutLM | 28.67$\pm$6.56 | 47.22$\pm$8.31 | 35.42$\pm$7.00 | 47.68$\pm$7.49 | 63.93$\pm$7.04 | 54.57$\pm$7.46
LayoutReader | 22.37$\pm$2.03 | 35.19$\pm$4.97 | 27.19$\pm$2.56 | 43.85$\pm$4.72 | 56.90$\pm$2.47 | 49.47$\pm$3.95
LASER | 43.66$\pm$1.97 | 47.08$\pm$5.72 | 45.21$\pm$3.74 | 61.16$\pm$3.11 | 60.33$\pm$5.65 | 60.63$\pm$4.00
5 | BERT | 20.57$\pm$2.59 | 39.25$\pm$1.10 | 26.93$\pm$2.46 | 45.73$\pm$4.31 | 63.29$\pm$3.68 | 53.06$\pm$4.14
RoBERTa | 19.47$\pm$2.32 | 35.04$\pm$1.89 | 24.94$\pm$1.93 | 52.21$\pm$4.55 | 66.63$\pm$5.52 | 58.54$\pm$4.92
LayoutLM | 39.24$\pm$4.33 | 58.20$\pm$2.45 | 46.72$\pm$3.12 | 56.13$\pm$7.39 | 71.66$\pm$6.13 | 62.91$\pm$7.04
LayoutReader | 27.52$\pm$3.44 | 41.17$\pm$4.01 | 32.89$\pm$3.28 | 51.97$\pm$8.42 | 63.82$\pm$7.87 | 57.24$\pm$8.32
LASER | 47.25$\pm$1.93 | 52.85$\pm$1.22 | 49.87$\pm$1.29 | 65.62$\pm$3.79 | 64.90$\pm$5.78 | 65.23$\pm$4.70
7 | BERT | 21.44$\pm$2.07 | 40.87$\pm$3.79 | 28.09$\pm$2.48 | 50.13$\pm$4.35 | 66.67$\pm$3.67 | 57.20$\pm$4.07
RoBERTa | 23.68$\pm$3.06 | 38.74$\pm$3.54 | 29.32$\pm$3.08 | 55.14$\pm$4.49 | 69.35$\pm$4.16 | 61.43$\pm$4.42
LayoutLM | 43.23$\pm$5.27 | 61.73$\pm$5.97 | 50.76$\pm$5.30 | 62.87$\pm$3.98 | 76.38$\pm$2.72 | 68.96$\pm$3.49
LayoutReader | 31.22$\pm$3.14 | 45.08$\pm$3.83 | 36.85$\pm$3.26 | 54.43$\pm$5.89 | 65.48$\pm$5.34 | 59.42$\pm$5.68
LASER | 50.62$\pm$3.26 | 53.63$\pm$2.89 | 51.98$\pm$2.00 | 68.02$\pm$3.16 | 66.87$\pm$4.82 | 67.40$\pm$3.76

###  5.3 Compared Methods

We evaluate LASER against several strong sequence labeling methods as follows.

  * •

BERT (Devlin et al., 2018) is a text-only auto-encoding pre-trained language
model using the large-scale mask language modeling. We fine-tune the pre-
trained BERT-base model with the few-shot training samples on each datasets.

  * •

RoBERTa (Liu et al., 2019) extends the capacity of BERT and achieves better
performance in multiple natural language understanding tasks. We also conduct
the fine-tuning with few-shot training samples.

  * •

LayoutLM (Xu et al., 2020) is a multi-modal language model which includes the
layout and text information. It is built upon BERT and adds the extra spatial
embeddings into the BERT embedding layer. Following LayoutLM, LayoutLMv2 (Xu
et al., 2021a) leverages extra computer vision features and improves the
performance, which are strong signals but absent in our settings. For a fair
comparison, we do not include LayoutLMv2 in our comparative experiments.

  * •

LayoutReader (Wang et al., 2021) is a layout-aware sequence-to-sequence model
for reading order detection. We append a linear layer upon the hidden states
to conduct sequence labeling.

These compared methods are in their base version and follow the IOBES tagging
scheme.

(a) FUNSD (b) CORD-Lv1

Figure 2: F-1 Curves with Different Sizes of Few-shot Training Samples.

###  5.4 Implementation Details

We build LASER on the base of LayoutReader. We use the Transformers (Wolf et
al., 2019) and the s2s-ft toolkits from the repository of Dong et al. (2019a).
We use one NVIDIA A6000 to finetune with batch size of 8. We optimize the
model with AdamW optimizer and the learning rate is $5\times 10^{-5}$.

###  5.5 Experimental Results

From Table 2 and Figure 2, the results show that, under few-shot settings, our
proposed model, LASER, achieves the SOTA overall performance compared with
sequence labeling models. We conclude that the gain of performance comes
mostly from the generative labeling scheme since LASER largely outperforms
LayoutReader although both of them share the same backbone.

Specifically, compared with the second-best baseline, LASER improves the F-1
scores by 8.59% on FUNSD and by 3.32% on CORD-Lv1 on average across the
different shots and LASER (IRLVT) also surpasses the baselines under most
settings.

Moreover, the improvement on precision is remarkable. LASER improves the
precision by 12.35% on FUNSD and by 10.62% on CORD-Lv1 on average across the
different shots. Especially, under 1-shot setting, it surpasses the best
sequence labeling model on FUNSD by 19.01% on precision, 10.47% on recall and
17.18% on F-1 score.

We can also observe a drop in the improvement with the increasing number of
training samples. We conclude that, with enough training samples, the sequence
labeling learns the meaning of each label and the semantics of each label
surface names no longer provides extra useful information.

Based on these comparison, we safely come to the conclusion that our proposed
generative labeling scheme is superior to the traditional sequence labeling
scheme in few shot settings.

Table 3: Ablation Study of Different Label Surface Names in LASER. IRLVT uses
the irrelevant tokens as labels; ORIG uses the original label surface names;
Sub1 and Sub2 use some reasonable alternative label surface names. as
substitutes. Bold denotes the best model; Underline denotes the second-best
model.

$|\mathcal{P}|$ | FUNSD | CORD-Lv1
---|---|---
Label Surface Names | Precision | Recall | F-1 | Label Surface Names | Precision | Recall | F-1
1 | IRLVT [x, y, z] | 30.64$\pm$5.89 | 33.45$\pm$9.14 | 31.62$\pm$6.61 | IRLVT [w, x, y, z] | 48.57$\pm$4.93 | 44.12$\pm$6.36 | 45.84$\pm$3.57
ORIG [header, question, answer] | 30.40$\pm$4.89 | 35.20$\pm$7.20 | 32.36$\pm$5.14 | ORIG [menu, void, sub, total] | 47.63$\pm$3.90 | 45.52$\pm$5.84 | 46.24$\pm$3.01
Sub1 [title, key, value] | 31.78$\pm$4.75 | 34.21$\pm$7.44 | 32.66$\pm$5.10 | Sub1 [info, etc, small, number] | 48.12$\pm$4.15 | 48.47$\pm$6.60 | 48.04$\pm$4.06
Sub2 [page, topic, value] | 30.90$\pm$5.20 | 35.97$\pm$8.57 | 33.03$\pm$6.31 | Sub2 [page, non, part, price] | 45.59$\pm$5.68 | 44.09$\pm$7.87 | 44.38$\pm$5.39
3 | IRLVT [x, y, z] | 43.51$\pm$1.46 | 47.92$\pm$5.93 | 45.44$\pm$3.36 | IRLVT [w, x, y, z] | 61.50$\pm$2.52 | 59.17$\pm$4.11 | 60.27$\pm$2.99
ORIG [header, question, answer] | 43.66$\pm$1.97 | 47.08$\pm$5.72 | 45.21$\pm$3.74 | ORIG [menu, void, sub, total] | 61.16$\pm$3.11 | 60.33$\pm$5.65 | 60.63$\pm$4.00
Sub1 [title, key, value] | 43.87$\pm$1.33 | 47.11$\pm$6.07 | 45.26$\pm$3.44 | Sub1 [info, etc, small, number] | 61.54$\pm$2.76 | 58.79$\pm$6.76 | 60.00$\pm$4.57
Sub2 [page, topic, value] | 43.88$\pm$1.34 | 48.01$\pm$6.86 | 45.65$\pm$3.93 | Sub2 [page, non, part, price] | 61.85$\pm$2.16 | 60.29$\pm$2.85 | 61.03$\pm$2.10
5 | IRLVT [x, y, z] | 46.94$\pm$1.87 | 52.96$\pm$2.03 | 49.74$\pm$1.63 | IRLVT [w, x, y, z] | 63.67$\pm$3.82 | 61.10$\pm$5.21 | 62.33$\pm$4.48
ORIG [header, question, answer] | 47.25$\pm$1.93 | 52.85$\pm$1.22 | 49.87$\pm$1.29 | ORIG [menu, void, sub, total] | 65.62$\pm$3.79 | 64.90$\pm$5.78 | 65.23$\pm$4.70
Sub1 [title, key, value] | 47.43$\pm$2.29 | 52.19$\pm$2.09 | 49.68$\pm$1.98 | Sub1 [info, etc, small, number] | 65.05$\pm$5.59 | 63.64$\pm$7.16 | 64.31$\pm$6.34
Sub2 [page, topic, value] | 47.46$\pm$2.18 | 53.50$\pm$1.01 | 50.26$\pm$1.16 | Sub2 [page, non, part, price] | 65.57$\pm$3.04 | 64.71$\pm$3.97 | 65.12$\pm$3.38
7 | IRLVT [x, y, z] | 50.30$\pm$2.26 | 54.14$\pm$3.48 | 52.08$\pm$2.26 | IRLVT [w, x, y, z] | 66.08$\pm$3.26 | 64.73$\pm$5.08 | 65.32$\pm$3.74
ORIG [header, question, answer] | 50.62$\pm$3.26 | 53.63$\pm$2.89 | 51.98$\pm$2.00 | ORIG [menu, void, sub, total] | 68.02$\pm$3.16 | 66.87$\pm$4.82 | 67.40$\pm$3.76
Sub1 [title, key, value] | 50.22$\pm$3.20 | 53.79$\pm$3.13 | 51.88$\pm$2.56 | Sub1 [info, etc, small, number] | 67.61$\pm$4.19 | 66.64$\pm$5.72 | 67.08$\pm$4.72
Sub2 [page, topic, value] | 50.43$\pm$2.88 | 54.03$\pm$2.71 | 52.10$\pm$2.09 | Sub2 [page, non, part, price] | 66.64$\pm$3.97 | 63.59$\pm$7.00 | 65.02$\pm$5.47

###  5.6 Ablation Study

In the ablation study, we aim at study the role of the label surface names. We
introduce an ablation version, LASER (IRLVT), by replacing the label surface
names with irrelevant tokens. We also design more different sets of words as
substitutes denoted Sub1 and Sub2. The detailed substitutes are introduced in
Table 3.

To implement the ablation study, we simply replace the word embedding of label
surface names. For example, in LASER (Sub1) on FUNSD, we use the wording
embedding of title instead of the original header.

From Table 3, we compare the performance of all the ablation models. We
observe that LASER performs differently with distinct label semantics. In most
cases, the human-designed labels can provide stronger semantic correlation
with the entities than the irrelevant labels so they can further improve the
performance. However, there are also drops due to improper labels. Overall, we
conclude that the semantic meanings of the label surface names are useful to
bridge the gap between the labels and entities.

###  5.7 Spatial Correspondence Interpretation

In this section, we study the ability of LASER to capture the spatial
correspondence between certain areas and the labels. The experiment is based
on the results of LASER on FUNSD with 7 shots. As mentioned in Section 4.2, we
design unique spatial identifiers for the label surface names. The identifiers
are in the same form as the spatial embeddings and LASER inserts the
identifiers into the original spatial embedding space during sequence-to-
sequence training. Ideally, the model can learn where a certain label is more
likely to appear. To visualize such patterns, we compute the cosine similarity
matrix $M$ of identifiers and the spatial embeddings as
$M_{ij}=\cos{(\text{SpatialID}(\tau),\text{SpatialEmb((i, j))})}$ where
$(i,j)$ is the normalized coordinate pair;
$\tau\in\\{\tau_{1},...,\tau_{t}\\}$. Then we plot the heatmap of the
similarity matrix, where the highlight areas mean the higher similarities.

(a) Header

(b) Question

(c) Answer

Figure 3: Spatial correspondence visualization on FUNSD for different entity
types.

From Figure 3, we observe that the label header is more likely to be in the
middle column of the page and may appear in the bottom part as well when there
are multiple paragraphs. Intuitively, the label question and answer should
appear in pairs and it is observed in Figure 3 that their heatmaps are almost
complementary to each other. Several examples from FUNSD are selected to
demonstrate the visualization results in 4. Comparing the examples and the
visualization results, we conclude that the spatial identifiers of labels
capture the formats of pages and LASER leverages these features to better
extract the entities under few shot settings.

(a) Original Image

(b) Labeled Entities

(c) Original Image

(d) Labeled Entities

Figure 4: Layout Format Examples from FUNSD: Bl, Bl, Bl denotes question,
answer, header.

(a) Test Image and Expected Labels

(b) LASER Results

(c) LayoutLM Results

(d) Test Image and Expected Labels

(e) LASER Results

(f) LayoutLM Results

Figure 5: Case Studies. (a), (b), (c) from FUNSD; (d), (e), (f) from CORD-Lv1;
Bl, Bl, Bl, Bl denote menu, question, answer, other; Bl, Bl denote menu,
total; ///, /// denote the right, wrong predictions.

###  5.8 Case Study

We visualize cases from the 5-shot setting. From Figure 5, we observe LASER
can extract the entities correctly, and the errors of LayoutLM comes from the
failure to extract the entities or wrong entity type predictions. Since the
sequence labeling groups the words into spans through IOBES tagging, which
creates great uncertainty. Meanwhile, LASER also learns questions and answers
appear in pairs (see Figure 5(b)). It also properly predicts a numerical
string as menu even if numbers are likely to be total (see Figure 5(e)).

Table 4: Text-only Dataset Statistics

Dataset | # Train | # Test | # Entity Type
---|---|---|---
OntoNotes | 60.0k | 8.3k | 18
Mit Movie | 7.8k | 2.0k | 12

Table 5: Results of 10-way-5-shot Experiments

Model | OntoNotes | MIT Movie
---|---|---
F-1 | F-1
BERT | 60.79$\pm$0.97 | 47.88$\pm$0.97
RoBERTa (Huang et al., 2020) | 57.70 | 51.30
UniLM | 60.82$\pm$1.26 | 51.09$\pm$1.40
LASER | 61.11$\pm$1.08 | 51.88$\pm$1.27

###  5.9 Text-only Entity Recognition

LASER is designed for the entity recognition task in document images where
both text and layout can be leveraged to acquire essential information.
However, the generative labeling scheme is not constrained in the scenario of
document images. We briefly explore the potential of the generative labeling
scheme in text-only scenario. We initialize LASER with a text-only language
model, UniLM (Dong et al., 2019b), based on the experiments in Wang et al.
(2021), and apply it onto text-only entity recognition task. Following Huang
et al. (2020), we conduct 10-way-5-shot experiments on two datasets, OntoNotes
(Weischedel et al., 2013) and MIT Movie (Liu et al., 2013), which cover
general domains and review domains, respectively. The dataset statistics are
shown in Table 4 and the results are as shown in Table 5. We observe that our
method can also surpass the sequence labeling methods in these two datasets,
showing the great potential of the generative labeling scheme in the entity
recognition tasks.

##  6 Related Work

Layout-aware LMs. Since the post-OCR processing has great application
prospects, existing works propose to adapt the language pre-training to the
layout formats learning. LayoutLM (Xu et al., 2020) is the pioneer in this
area, which successfully uses the coordinates to represent the layout
information in the embedding layer of BERT (Devlin et al., 2018). Following
LayoutLM, the upgraded version, LayoutLMv2 (Xu et al., 2021a), is further
proposed to leverage the visual features and benefits from the alignment
between words and the regions in the page. LAMBERT (Garncarek et al., 2021)
and BROS (Hong et al., 2020) continue studying the layout representation which
uses the sinusoidal function or apply the relative positional biases from T5
(Raffel et al., 2019). LayoutReader (Wang et al., 2021) aims to predict the
reading order of words from the OCR results. ReadingBank (Wang et al., 2021)
is proposed to facilitate the pre-training of reading order detection, which
annotates the reading order of millions of pages.

Generalized Seq2Seq. Sequence-to-sequence architecture is basic in natural
language processing and is originally designed for machine translation. With
the rise of large pre-trained models, sequence-to-sequence models are
increasingly used with new problem formulation. Existing works exploit the
potential latent knowledge and stronger representation ability of sequence-to-
sequence modeling. GENRE (De Cao et al., 2020) creatively reformulates the
entity retrieval task into the sequence-to-sequence settings. It inferences
the lined entities using the generation of BART. Recent works on prompt
learning also leverage the pre-trained sequence-to-sequence language models to
conduct few shot learning (Liu et al., 2021; Puri and Catanzaro, 2019;
Hambardzumyan et al., 2021).

##  7 Conclusions and Future Work

In this paper, we present LASER, a label-aware sequence-to-sequence framework
for entity recognition in document images under few-shot settings. It benefits
from the generative labeling scheme which reformulates the entity recognition
task into the sequence-to-sequence setting. The label surface names are
embedded into the generated sequence. Compared with the sequence labeling
methods, LASER leverages the rich semantics of the label surface names and
overcome the limitation of training data. Moreover, we design spatial
identifiers for each label and well insert them into the spatial embedding
hyperspace. In this way, LASER can inference the entity labels from the layout
formats perspective and empirical experiments demonstrate our method can learn
the layout formats though limited number of training samples.

For further research, we will investigate the selection of label surface names
and how to better leverage the semantics from the pre-trained sequence-to-
sequence models. We also notice that such labeling scheme can cope with
unknown categories. We will focus on the generalization of our method.

## Acknowledgments

We want to thank the anonymous reviewers for their insightful comments. Zilong
would also like to thank the Powell Fellowship for the support during the
challenging years at the beginning of his Ph.D. program.

The research was sponsored in part by National Science Foundation Convergence
Accelerator under award OIA-2040727 as well as generous gifts from Google,
Adobe, and Teradata. Any opinions, findings, conclusions, or recommendations
expressed herein are those of the authors and should not be interpreted as
necessarily representing the views, either expressed or implied, of the U.S.
Government. The U.S. Government is authorized to reproduce and distribute
reprints for government purposes notwithstanding any copyright annotation
hereon.

## Ethical Consideration

This paper focuses on the entity recognition in document images under few-shot
setting. Our architecture are built upon open-source models and all the
datasets are available online. We will release the code and datasets on
https://github.com/zlwang-cs/LASER-release. Therefore, we do not anticipate
any major ethical concerns.

## References

  * Bao et al. (2020) Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, et al. 2020.  Unilmv2: Pseudo-masked language models for unified language model pre-training.  In _International Conference on Machine Learning_ , pages 642–652. PMLR.
  * De Cao et al. (2020) Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2020.  Autoregressive entity retrieval.  _arXiv preprint arXiv:2010.00904_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dong et al. (2019a) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019a.  Unified language model pre-training for natural language understanding and generation.  _arXiv preprint arXiv:1905.03197_.
  * Dong et al. (2019b) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019b.  Unified language model pre-training for natural language understanding and generation.  _Advances in Neural Information Processing Systems_ , 32.
  * Garncarek et al. (2021) Łukasz Garncarek, Rafał Powalski, Tomasz Stanisławek, Bartosz Topolski, Piotr Halama, Michał Turski, and Filip Graliński. 2021.  Lambert: Layout-aware language modeling for information extraction.  In _International Conference on Document Analysis and Recognition_ , pages 532–547. Springer.
  * Guillaume Jaume (2019) Jean-Philippe Thiran Guillaume Jaume, Hazim Kemal Ekenel. 2019.  Funsd: A dataset for form understanding in noisy scanned documents.  In _Accepted to ICDAR-OST_.
  * Hambardzumyan et al. (2021) Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021.  Warp: Word-level adversarial reprogramming.  _arXiv preprint arXiv:2101.00121_.
  * Hong et al. (2020) Teakgyu Hong, DongHyun Kim, Mingi Ji, Wonseok Hwang, Daehyun Nam, and Sungrae Park. 2020.  Bros: A pre-trained language model for understanding texts in document.
  * Huang et al. (2020) Jiaxin Huang, Chunyuan Li, Krishan Subudhi, Damien Jose, Shobana Balakrishnan, Weizhu Chen, Baolin Peng, Jianfeng Gao, and Jiawei Han. 2020.  Few-shot named entity recognition: A comprehensive study.  _arXiv preprint arXiv:2012.14978_.
  * Liu et al. (2013) Jingjing Liu, Panupong Pasupat, Yining Wang, Scott Cyphers, and Jim Glass. 2013\.  Query understanding enhanced by hierarchical parsing structures.  In _2013 IEEE Workshop on Automatic Speech Recognition and Understanding_ , pages 72–77. IEEE.
  * Liu et al. (2021) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _arXiv preprint arXiv:2107.13586_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Marquez et al. (2005) Lluis Marquez, Pere Comas, Jesús Giménez, and Neus Catala. 2005.  Semantic role labeling as sequential tagging.  In _Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL-2005)_ , pages 193–196.
  * Nakayama (2018) Hiroki Nakayama. 2018.  seqeval: A python framework for sequence labeling evaluation.  Software available from https://github.com/chakki-works/seqeval.
  * Park et al. (2019) Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwalsuk Lee. 2019.  Cord: A consolidated receipt dataset for post-ocr parsing.
  * Puri and Catanzaro (2019) Raul Puri and Bryan Catanzaro. 2019.  Zero-shot text classification with generative language models.  _arXiv preprint arXiv:1912.10165_.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_.
  * Ratinov and Roth (2009) Lev Ratinov and Dan Roth. 2009.  Design challenges and misconceptions in named entity recognition.  In _Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009)_ , pages 147–155.
  * Wang et al. (2021) Zilong Wang, Yiheng Xu, Lei Cui, Jingbo Shang, and Furu Wei. 2021.  Layoutreader: Pre-training of text and layout for reading order detection.
  * Weischedel et al. (2013) Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013.  Ontonotes release 5.0 ldc2013t19.  _Linguistic Data Consortium, Philadelphia, PA_ , 23.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.
  * Xu et al. (2021a) Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. 2021a.  Layoutlmv2: Multi-modal pre-training for visually-rich document understanding.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL) 2021_.
  * Xu et al. (2020) Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020.  Layoutlm: Pre-training of text and layout for document image understanding.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 1192–1200.
  * Xu et al. (2021b) Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, and Furu Wei. 2021b.  Layoutxlm: Multimodal pre-training for multilingual visually-rich document understanding.

## Appendix

##  Appendix A All Results

All results are listed in Table 6.

Table 6: Evaluation Results with Different Sizes of Few-shot Training Samples $|\mathcal{P}|$ | Model | FUNSD | CORD-Lv1
---|---|---|---
Precision | Recall | F-1 | Precision | Recall | F-1
1 | BERT | 9.62$\pm$2.24 | 24.14$\pm$3.46 | 13.55$\pm$2.09 | 30.64$\pm$2.80 | 45.60$\pm$3.45 | 36.64$\pm$3.10
RoBERTa | 9.29$\pm$1.57 | 22.06$\pm$5.64 | 12.76$\pm$1.91 | 30.66$\pm$4.25 | 44.39$\pm$6.72 | 36.25$\pm$5.18
LayoutLM | 11.39$\pm$1.12 | 24.73$\pm$7.38 | 15.18$\pm$2.17 | 33.27$\pm$7.32 | 49.49$\pm$10.26 | 39.77$\pm$8.47
LayoutReader | 11.32$\pm$0.62 | 22.53$\pm$4.80 | 14.84$\pm$1.25 | 32.17$\pm$4.64 | 45.61$\pm$6.54 | 37.70$\pm$5.31
LASER | 30.40$\pm$4.89 | 35.20$\pm$7.20 | 32.36$\pm$5.14 | 47.63$\pm$3.90 | 45.52$\pm$5.84 | 46.24$\pm$3.01
(IRLVT) | 30.64$\pm$5.89 | 33.45$\pm$9.14 | 31.62$\pm$6.61 | 48.57$\pm$4.93 | 44.12$\pm$6.36 | 45.84$\pm$3.57
(Sub1) | 31.78$\pm$4.75 | 34.21$\pm$7.44 | 32.66$\pm$5.10 | 48.12$\pm$4.15 | 48.47$\pm$6.60 | 48.04$\pm$4.06
(Sub2) | 30.90$\pm$5.20 | 35.97$\pm$8.57 | 33.03$\pm$6.31 | 45.59$\pm$5.68 | 44.09$\pm$7.87 | 44.38$\pm$5.39
2 | BERT | 12.49$\pm$3.24 | 30.01$\pm$4.55 | 17.53$\pm$3.89 | 37.66$\pm$3.79 | 55.43$\pm$3.41 | 44.82$\pm$3.78
RoBERTa | 13.12$\pm$3.08 | 27.63$\pm$5.16 | 17.56$\pm$3.59 | 40.66$\pm$3.60 | 56.60$\pm$4.87 | 47.30$\pm$4.05
LayoutLM | 19.73$\pm$5.43 | 37.32$\pm$9.58 | 25.41$\pm$6.25 | 42.92$\pm$8.27 | 60.47$\pm$8.15 | 50.15$\pm$8.44
LayoutReader | 16.66$\pm$2.58 | 29.26$\pm$5.34 | 20.96$\pm$3.01 | 44.14$\pm$7.77 | 58.50$\pm$8.32 | 50.26$\pm$8.04
LASER | 39.23$\pm$3.09 | 42.43$\pm$6.23 | 40.38$\pm$2.71 | 59.31$\pm$2.31 | 54.85$\pm$6.19 | 56.80$\pm$3.46
(IRLVT) | 38.75$\pm$3.74 | 40.37$\pm$8.17 | 39.19$\pm$4.84 | 57.05$\pm$2.87 | 53.49$\pm$7.01 | 55.01$\pm$4.19
(Sub1) | 38.47$\pm$2.95 | 41.11$\pm$7.03 | 39.43$\pm$3.83 | 57.46$\pm$4.22 | 54.72$\pm$5.87 | 55.98$\pm$4.68
(Sub2) | 37.52$\pm$2.22 | 43.23$\pm$6.48 | 39.90$\pm$3.12 | 57.03$\pm$2.80 | 55.75$\pm$5.48 | 56.28$\pm$3.48
3 | BERT | 16.42$\pm$4.30 | 34.74$\pm$5.36 | 22.19$\pm$5.05 | 39.62$\pm$3.99 | 56.65$\pm$4.03 | 46.58$\pm$3.94
RoBERTa | 16.71$\pm$3.63 | 31.28$\pm$3.55 | 21.66$\pm$3.84 | 44.51$\pm$4.69 | 60.18$\pm$4.69 | 51.15$\pm$4.70
LayoutLM | 28.67$\pm$6.56 | 47.22$\pm$8.31 | 35.42$\pm$7.00 | 47.68$\pm$7.49 | 63.93$\pm$7.04 | 54.57$\pm$7.46
LayoutReader | 22.37$\pm$2.03 | 35.19$\pm$4.97 | 27.19$\pm$2.56 | 43.85$\pm$4.72 | 56.90$\pm$2.47 | 49.47$\pm$3.95
LASER | 43.66$\pm$1.97 | 47.08$\pm$5.72 | 45.21$\pm$3.74 | 61.16$\pm$3.11 | 60.33$\pm$5.65 | 60.63$\pm$4.00
(IRLVT) | 43.51$\pm$1.46 | 47.92$\pm$5.93 | 45.44$\pm$3.36 | 61.50$\pm$2.52 | 59.17$\pm$4.11 | 60.27$\pm$2.99
(Sub1) | 43.87$\pm$1.33 | 47.11$\pm$6.07 | 45.26$\pm$3.44 | 61.54$\pm$2.76 | 58.79$\pm$6.76 | 60.00$\pm$4.57
(Sub2) | 43.88$\pm$1.34 | 48.01$\pm$6.86 | 45.65$\pm$3.93 | 61.85$\pm$2.16 | 60.29$\pm$2.85 | 61.03$\pm$2.10
4 | BERT | 18.25$\pm$3.30 | 37.90$\pm$2.93 | 24.55$\pm$3.59 | 43.94$\pm$4.14 | 61.13$\pm$4.18 | 51.09$\pm$4.12
RoBERTa | 17.99$\pm$2.84 | 34.38$\pm$4.09 | 23.52$\pm$3.07 | 49.56$\pm$4.89 | 65.19$\pm$4.85 | 56.29$\pm$4.86
LayoutLM | 33.38$\pm$3.62 | 53.71$\pm$3.24 | 41.00$\pm$2.71 | 52.15$\pm$7.90 | 68.06$\pm$6.86 | 58.99$\pm$7.66
LayoutReader | 24.61$\pm$2.58 | 38.28$\pm$3.05 | 29.89$\pm$2.51 | 48.27$\pm$9.14 | 61.29$\pm$8.49 | 53.96$\pm$9.06
LASER | 44.91$\pm$2.42 | 50.25$\pm$3.26 | 47.36$\pm$2.18 | 63.90$\pm$3.19 | 60.99$\pm$5.86 | 62.38$\pm$4.60
(IRLVT) | 46.31$\pm$1.91 | 51.74$\pm$2.55 | 48.83$\pm$1.67 | 63.68$\pm$3.72 | 60.39$\pm$7.11 | 61.93$\pm$5.50
(Sub1) | 45.58$\pm$1.63 | 51.47$\pm$2.97 | 48.29$\pm$1.65 | 63.22$\pm$3.31 | 60.39$\pm$8.23 | 61.65$\pm$5.93
(Sub2) | 45.43$\pm$2.08 | 51.74$\pm$2.64 | 48.33$\pm$1.79 | 62.85$\pm$3.83 | 60.07$\pm$8.00 | 61.31$\pm$5.97
5 | BERT | 20.57$\pm$2.59 | 39.25$\pm$1.10 | 26.93$\pm$2.46 | 45.73$\pm$4.31 | 63.29$\pm$3.68 | 53.06$\pm$4.14
RoBERTa | 19.47$\pm$2.32 | 35.04$\pm$1.89 | 24.94$\pm$1.93 | 52.21$\pm$4.55 | 66.63$\pm$5.52 | 58.54$\pm$4.92
LayoutLM | 39.24$\pm$4.33 | 58.20$\pm$2.45 | 46.72$\pm$3.12 | 56.13$\pm$7.39 | 71.66$\pm$6.13 | 62.91$\pm$7.04
LayoutReader | 27.52$\pm$3.44 | 41.17$\pm$4.01 | 32.89$\pm$3.28 | 51.97$\pm$8.42 | 63.82$\pm$7.87 | 57.24$\pm$8.32
LASER | 47.25$\pm$1.93 | 52.85$\pm$1.22 | 49.87$\pm$1.29 | 65.62$\pm$3.79 | 64.90$\pm$5.78 | 65.23$\pm$4.70
(IRLVT) | 46.94$\pm$1.87 | 52.96$\pm$2.03 | 49.74$\pm$1.63 | 63.67$\pm$3.82 | 61.10$\pm$5.21 | 62.33$\pm$4.48
(Sub1) | 47.43$\pm$2.29 | 52.19$\pm$2.09 | 49.68$\pm$1.98 | 65.05$\pm$5.59 | 63.64$\pm$7.16 | 64.31$\pm$6.34
(Sub2) | 47.46$\pm$2.18 | 53.50$\pm$1.01 | 50.26$\pm$1.16 | 65.57$\pm$3.04 | 64.71$\pm$3.97 | 65.12$\pm$3.38
6 | BERT | 20.37$\pm$2.08 | 39.58$\pm$2.90 | 26.85$\pm$2.27 | 46.39$\pm$4.56 | 63.61$\pm$4.81 | 53.62$\pm$4.65
RoBERTa | 21.70$\pm$2.37 | 37.14$\pm$1.34 | 27.32$\pm$1.96 | 51.80$\pm$5.60 | 66.54$\pm$5.87 | 58.25$\pm$5.77
LayoutLM | 41.75$\pm$3.83 | 60.47$\pm$3.23 | 49.28$\pm$3.02 | 59.31$\pm$4.23 | 74.48$\pm$3.16 | 66.02$\pm$3.86
LayoutReader | 29.19$\pm$2.10 | 43.21$\pm$2.32 | 34.82$\pm$2.09 | 52.76$\pm$4.45 | 65.62$\pm$3.47 | 58.45$\pm$3.99
LASER | 48.64$\pm$2.14 | 53.54$\pm$2.10 | 50.96$\pm$1.95 | 66.85$\pm$3.88 | 65.97$\pm$5.19 | 66.38$\pm$4.38
(IRLVT) | 48.33$\pm$2.15 | 52.68$\pm$1.56 | 50.36$\pm$1.08 | 66.01$\pm$4.07 | 63.85$\pm$5.59 | 64.87$\pm$4.63
(Sub1) | 48.49$\pm$1.96 | 52.85$\pm$1.67 | 50.53$\pm$0.99 | 65.19$\pm$4.22 | 63.37$\pm$7.10 | 64.21$\pm$5.68
(Sub2) | 49.14$\pm$1.97 | 53.42$\pm$1.20 | 51.16$\pm$1.22 | 66.44$\pm$3.11 | 64.05$\pm$4.31 | 65.21$\pm$3.67
7 | BERT | 21.44$\pm$2.07 | 40.87$\pm$3.79 | 28.09$\pm$2.48 | 50.13$\pm$4.35 | 66.67$\pm$3.67 | 57.20$\pm$4.07
RoBERTa | 23.68$\pm$3.06 | 38.74$\pm$3.54 | 29.32$\pm$3.08 | 55.14$\pm$4.49 | 69.35$\pm$4.16 | 61.43$\pm$4.42
LayoutLM | 43.23$\pm$5.27 | 61.73$\pm$5.97 | 50.76$\pm$5.30 | 62.87$\pm$3.98 | 76.38$\pm$2.72 | 68.96$\pm$3.49
LayoutReader | 31.22$\pm$3.14 | 45.08$\pm$3.83 | 36.85$\pm$3.26 | 54.43$\pm$5.89 | 65.48$\pm$5.34 | 59.42$\pm$5.68
LASER | 50.62$\pm$3.26 | 53.63$\pm$2.89 | 51.98$\pm$2.00 | 68.02$\pm$3.16 | 66.87$\pm$4.82 | 67.40$\pm$3.76
(IRLVT) | 50.30$\pm$2.26 | 54.14$\pm$3.48 | 52.08$\pm$2.26 | 66.08$\pm$3.26 | 64.73$\pm$5.08 | 65.32$\pm$3.74
(Sub1) | 50.22$\pm$3.20 | 53.79$\pm$3.13 | 51.88$\pm$2.56 | 67.61$\pm$4.19 | 66.64$\pm$5.72 | 67.08$\pm$4.72
(Sub2) | 50.43$\pm$2.88 | 54.03$\pm$2.71 | 52.10$\pm$2.09 | 66.64$\pm$3.97 | 63.59$\pm$7.00 | 65.02$\pm$5.47