# Less is More: Pay Less Attention in Vision Transformers

Zizheng Pan, Bohan Zhuang, Haoyu He, Jing Liu, Jianfei Cai
Dept of Data Science and AI, Monash University Corresponding author. E-mail:
$\tt<EMAIL_ADDRESS>
###### Abstract

Transformers [35] have become one of the dominant architectures in deep
learning, particularly as a powerful alternative to convolutional neural
networks (CNNs) in computer vision. However, Transformer training and
inference in previous works can be prohibitively expensive due to the
quadratic complexity of self-attention over a long sequence of
representations, especially for high-resolution dense prediction tasks. To
this end, we present a novel Less attention vIsion Transformer (LIT), building
upon the fact that convolutions, fully-connected (FC) layers, and self-
attentions have almost equivalent mathematical expressions for processing
image patch sequences. Specifically, we propose a hierarchical Transformer
where we use pure multi-layer perceptrons (MLPs) to encode rich local patterns
in the early stages while applying self-attention modules to capture longer
dependencies in deeper layers. Moreover, we further propose a learned
deformable token merging module to adaptively fuse informative patches in a
non-uniform manner. The proposed LIT achieves promising performance on image
recognition tasks, including image classification, object detection and
instance segmentation, serving as a strong backbone for many vision tasks.

##  1 Introduction

Transformers have made substantial strides in natural language processing
(NLP) (e.g., [35, 7]) and recently in the computer vision (CV) field (e.g.,
[8, 32]). Inspired by the pyramid design [18] in CNNs, recent hierarchical
vision Transformers (HVTs) [37, 21, 38, 40] divide transformer blocks into
several stages and progressively shrink feature maps as the network goes
deeper. However, high-resolution feature maps in the early stages result in
long token sequences, which brings huge computational cost and memory
consumption due to the quadratic complexity of self-attention. For instance, a
feature map of size $56\times 56\times 96$ costs 2.0G FLOPs in one Multi-head
Self-Attention (MSA) [35], while the entire model of ResNet-18 [11] only
requires 1.8G FLOPs. Such a huge computing cost makes it difficult to apply
Transformers into broad computer vision tasks.

Several emerging efforts have been made to reduce the computational cost in
the early stages of HVTs. For example, some works [38, 34] reduce the number
of self-attention heads in an MSA layer or further decreasing the number of
Transformer blocks [8]. Another line of works proposes to trade-off accuracy
and efficiency for MSA via heuristic approximation, such as spatial reduction
attention (SRA) [37] and shifted window based multi-head self-attention (SW-
MSA) [21]. There are also studies simply employ convolutional layers [30, 9]
when the resolution of feature maps are considerably large. However, how much
the early adoption of self-attention layers really contributes to the final
performance remains unclear.

In this paper, we present a Less attention vIsion Transformer (LIT) to address
the aforementioned problem for HVTs. Specifically, we propose to exclusively
use MLP layers to capture local patterns in the early stages while introducing
MSA layers with sufficient number of heads to handle long range dependencies
in the later stages. The motivation comes from two aspects. First, previous
studies in CNNs have shown that shallow layers focus on local patterns and
deeper layers tend to produce more discriminative features [39, 14, 43].
Second, from the theoretical perspective, a self-attention layer with
sufficient heads applied to images can express any convolutional layer [5],
and the size of the receptive field for an MSA layer is positively correlated
with the number of heads. Therefore, fewer heads in an MSA layer theoretically
result in a smaller receptive field, where the extreme case is as expressive
as a $1\times 1$ convolution that can be viewed as a standard FC layer applied
to each pixel independently. To be emphasized, by exploiting MLP blocks in the
early stages, the model avoids the huge computational cost and memory
footprint arising from self-attention on high-resolution feature maps.
Moreover, applying self-attention in the later stages to capture long range
dependencies is quite efficient due to the progressive shrinking pyramid. Our
comprehensive results show that such a simple architecture design brings a
sweet spot between model performance and efficiency.

Furthermore, recent HVTs either adopt a standard convolutional layer or a
linear projection layer to merge nearby tokens [37, 21], aiming to control the
scale of feature maps. However, such methods hinder the representational power
for vision Transformers to model geometric transformations, considering that
not every pixel equally contributes to an output unit [23]. To this end, we
propose a Deformable Token Merging (DTM) module, inspired by deformable
convolutions [6, 45], where we learn a grid of offsets to adaptively augment
the spatial sampling locations for merging neighboring patches from a sub-
window in a feature map. In this way, we can obtain more informative
downsampled tokens for subsequent processing.

Our contributions can be summarized as follows. First, we propose a simple HVT
structure with pure MLP blocks in the early stages, making it tractable for
Transformers to process high-resolution images, especially in dense prediction
tasks. Second, we propose a deformable token merging module to adaptively
merge more informative patches to deliver hierarchical representations, with
enhanced transformation modeling capability. Finally, we conduct extensive
experiments to show that the proposed LIT performs favorably against other
state-of-the-art vision Transformers with similar or even reduced
computational complexity and memory consumption.

##  2 Related Work

#### Vision Transformers.

Vision Transformers are models which adopt the self-attention mechanism [35]
into CV tasks. Recent works towards Vision Transformers either follow a hybrid
architecture that combines convolution and self-attention [2, 30], or design a
pure self-attention architecture without convolution [25, 15]. More recently,
Dosovitskiy et al. [8] propose a Vision Transformer (ViT) which achieves
promising results on ImageNet. Since then, a few subsequent works have been
proposed to improve ViT from different aspects. For example, some works [42,
17] seek to bring locality into ViT as they find ViT failed to model the
important local structures (e.g., edges, lines). Another line of works aims to
explore deeper architectures [44, 33] by stacking more Transformer blocks.

There is also a prevailing trend to introduce hierarchical representations
into ViT [24, 41, 37, 21, 38, 13, 34]. To do so, these works divide
Transformer blocks into several stages and downsample feature maps as the
network goes deeper. However, high-resolution feature maps in the early stages
inevitably result in high computational and memory costs due to the quadratic
complexity of the self-attention module. Targeting at this problem, Wang et
al. [37] propose to reduce the spatial dimensions of attention’s key and value
matrices. Liu et al. [21] propose to limit self-attention in non-overlapped
local windows. However, these replacements seek to shrink the global attention
maps for efficiency. In this paper, we elaborately design the shallow layers
with pure MLPs, that are powerful enough to encode local patterns. This neat
architecture design keeps the capability for modelling global dependencies in
the later stages while easing the prohibitively expensive complexity
introduced by high-resolution feature maps, especially in the dense prediction
tasks.

#### Deformable Convolutions.

Deformable convolutions (DC) are initially proposed by Dai et al. [6] in
object detection and semantic segmentation tasks. Different from the regular
sampling grid of a standard convolution, DC adaptively augments the spatial
sampling locations with learned offsets. One following work by Zhu et al. [45]
improves DC with a modulation mechanism, which modulates the input feature
amplitudes from different spatial locations. With the advantage on modeling
geometric transformations, many works adopt DC to target various CV problems.
For example, Zhu et al. [46] propose a deformable attention module for object
detection. Shim et al. [29] construct a similarity search and extraction
network built upon DC layers for single image super-resolution. Thomas et al.
[31] introduce a deformable kernel point convolution operator for point
clouds. In this paper, we propose a deformable token merging module to
adaptively merge more informative image tokens. Unlike previous works that
merge tokens from a regular grid, DTM introduces better transformation
modeling capability for HVTs.

Figure 1: Overall architecture of LIT. The model is divided into four stages,
where we apply MLP blocks in the first two stages and employ standard
Transformer blocks in the last two stages. “DTM” denotes our proposed
deformable token merging module.

##  3 Proposed Method

###  3.1 Overall Architecture

The overall architecture of LIT is illustrated in Figure 1. Let
${\mathcal{I}}\in\mathbb{R}^{H\times W\times 3}$ be an input RGB image, where
$H$ and $W$ represent the height and width, respectively. We first split
${\mathcal{I}}$ into non-overlapping patches with a patch size of $4\times 4$,
and thus the initial feature dimension of each patch is $4\times 4\times
3=48$. Next, a linear embedding layer is exploited to project each patch into
dimension $C_{1}$, serving as the initial input for the following pipeline.
The entire model is divided into 4 stages. Letting $s\in[1,2,3,4]$ be the
index of a stage, we employ $L_{s}$ blocks at each stage, where the first two
stages solely utilise MLP blocks to encode local representations and the last
two stages employ standard Transformer blocks [8] to handle longer
dependencies. At each stage, we scale the input feature maps into
$\frac{H_{s-1}}{P_{s}}\times\frac{W_{s-1}}{P_{s}}\times C_{s}$, where $P_{s}$
and $C_{s}$ represent the patch size and the hidden dimension at the $s$-th
stage, respectively. For the last two stages, we set $N_{s}$ self-attention
heads in each Transformer block.

###  3.2 Block Design in LIT

As shown in Figure 1, LIT employs two types of blocks: MLP blocks and
Transformer blocks. In the early stages, we apply MLP blocks. Concretely, an
MLP block is built upon an MLP which consists of two FC layers with GELU [12]
non-linearity in between. For each MLP at the $s$-th stage, an expansion ratio
of $E_{s}$ is used. Specifically, the first FC layer expands the dimension of
a token from $C_{s}$ to $E_{s}\times C_{s}$, and the other FC layer reduces
the dimension back to $C_{s}$. Formally, letting ${\bf
X}\in\mathbb{R}^{(\frac{H_{s-1}}{P_{s}}\times\frac{W_{s-1}}{P_{s}})\times
C_{s}}$ be the input of the $s$-th stage and $l$ be the index of a block, an
MLP block can be formulated as

${\bf X}_{l}={\bf X}_{l-1}+\mathrm{MLP}(\mathrm{LN}({\bf X}_{l-1})),$ (1)

where $\mathrm{LN}$ indicates the layer normalization [1] and $\mathrm{MLP}$
denotes an MLP. In the last stages, a Transformer block as described in ViT
[8] contains an MSA layer and an MLP, which can be expressed as

$\displaystyle{\bf X}^{{}^{\prime}}_{l-1}$ $\displaystyle={\bf
X}_{l-1}+\mathrm{MSA}(\mathrm{LN}({\bf X}_{l-1})),$ (2) $\displaystyle{\bf
X}_{l}$ $\displaystyle={\bf
X}^{{}^{\prime}}_{l-1}+\mathrm{MLP}(\mathrm{LN}({\bf
X}^{{}^{\prime}}_{l-1})).$ (3)

With this architecture, our model benefits from two main advantages: First, we
avoid the huge computational costs and memory footprint that are introduced by
long sequences in the early stages. Second, unlike recent works that shrink
the attention maps using sub-windows [21] or reduce the spatial dimensions of
the key and value matrices, we keep standard MSA layers in the last two stages
so as to maintain the capability of LIT to handle long range dependencies
while keeping mild FLOPs due to the pyramid structure. We will show in the
ablation study that our simple architecture design outperforms the state-of-
the-art hierarchical ViT variants on ImageNet with comparable FLOPs.

#### Remark.

Here we justify the rationality of applying pure MLP blocks in the early
stages by considering the relationship among a convolutional layer, an FC
layer and an MSA layer. Firstly, we begin with a review of a standard
convolutional layer. Let ${\bf X}\in\mathbb{R}^{H\times W\times C_{in}}$ be
the input feature map, and let ${\bf W}\in\mathbb{R}^{K\times K\times
C_{in}\times C_{out}}$ be the convolutional weight tensor, where $K$ is the
kernel size, $C_{in}$ and $C_{out}$ are the input and output channel
dimensions, respectively. For simplicity, we omit the bias term and use ${\bf
X}_{p,:}$ to represent ${\bf X}_{i,j,:,:}$, where $(i,j)$ denotes the pixel
index and $p\in[H]\times[W]$. Given a convolutional kernel of $K\times K$
sampling locations, the output for a pixel $p$ can be formulated as

$\mathrm{Conv}({\bf X})_{p,:}=\sum_{k\in[K\times K]}{\bf X}_{p+g(k),:}{\bf
W}_{g(k),:,:},$ (4)

where $g:[K\times K]\rightarrow\Delta_{K}$ is a bijective mapping of sampling
indexes onto the pre-specified offsets $\Delta_{K}$. For example, let
$\Delta_{K}=\\{(-1,-1),(-1,0),...,(0,1),(1,1)\\}$ be a $3\times 3$ kernel with
dilation 1, then $g(0)=(-1,-1)$ represents the first sampling offset.

When $K=1$, the weight tensor ${\bf W}$ is equivalent to a matrix, such that
${\bf W}\in\mathbb{R}^{C_{in}\times C_{out}}$. In this case, Eq. (4) can
express an FC layer, and the output for a pixel $p$ is defined by

$\mathrm{FC}({\bf X})_{p,:}={\bf X}_{p,:}{\bf W}_{:,:}.$ (5)

Last, let $N_{h}$ be the number of heads in an MSA layer and ${\bf
W}^{(h)}\in\mathbb{R}^{C_{in}\times C_{out}}$ be learnable parameters of the
$h$-th head. Under mild conditions, Cordonnier et al. [5] prove that the
output from an MSA layer at pixel $p$ is identical to

$\mathrm{MSA}({\bf X})_{p}=\sum_{h\in[N_{h}]}{\bf X}_{p+f(h),:}{\bf W}^{(h)},$
(6)

where $f:[N_{h}]\rightarrow\Delta_{K}$ is a bijective mapping of heads onto
pixel shifts. In that case, Eq. (6) can be seen to be equivalent to a
convolutional layer with a kernel size of $\sqrt{N_{h}}\times\sqrt{N_{h}}$. We
refer detailed explanations to [5].

From Eqs. (4)-(6), we observe almost equivalent mathematical formulations
among a convolutional layer, an FC layer and an MSA layer. In fact, the main
difference lies in the size of their receptive fields. Specifically, the size
of the receptive field in a convolutional layer is $K\times K$. For an FC
layer, $K$ is always $1$. For an MSA layer, $K$ is equivalent to
$\sqrt{N_{h}}$. The observation tells us that when $N_{h}$ is pretty small
(e.g., 1 or 2), an MSA layer theoretically has a limited receptive field that
is approximately or equal to an FC layer. Considering most recent HVTs [37,
38, 40] adopt very few heads in the early stages, this justifies our method of
applying pure MLP blocks in the first two stages without self-attention.

###  3.3 Deformable Token Merging

Previous works on HVTs [37, 21] rely on patch merging to achieve pyramid
feature representations. However, they merge patches from a regular grid and
neglect the fact that not every patch contributes equally to an output unit
[23]. Inspired by deformable convolutions [6, 45], we propose a deformable
token merging module to learn a grid of offsets to adaptively sample more
informative patches. Formally, a deformable convolution is formulated as

$\mathrm{DC}({\bf X})_{p,:}=\sum_{k\in[K\times K]}{\bf X}_{p+g(k)+\Delta
g(k),:}{\bf W}_{g(k),:,:}.$ (7)

Compared to a standard convolution operation as in Eq. (4), DC learns an
offset $\Delta g(k)$ for each pre-specified offset $g(k)$. Learning $\Delta
g(k)$ requires a separate convolutional layer, which is also applied over the
input feature map ${\bf X}$. To merge patches in an adaptive manner, we adopt
one DC layer in a DTM module, which can be formulated by

$\mathrm{DTM}({\bf X})=\mathrm{GELU}(\mathrm{BN}(\mathrm{DC}({\bf X}))),$ (8)

where $\mathrm{BN}$ denotes the batch normalization [16] and we employ the
GELU non-linearity. We will show in the ablation study that the sampling
locations in DTM are adaptively adjusted when objects’ scales and shapes
change, benefiting from the learned offsets. Also note that our light-weight
DTM introduces negligible FLOPs and parameters compared to regular grid
sampling in baselines, thus making it a plug-and-play module for recent HVTs.

Table 1: Architecture specifications of LIT. $P$ denotes the patch size and $C$ is the channel dimension. $L$ refers to the number of blocks applied at each stage. $N$ is the number of self-attention heads. As the first two stages do not have self-attention layers, we only show the numbers of $N_{3}$ and $N_{4}$. The expansion ratios of LIT-Ti at each stage are [8, 8, 4, 4]. We use an expansion ratio of 4 for all MLP layers in LIT-S, LIT-M and LIT-B. The numbers denoted under each stage index (e.g., $28\times 28$) represents the size of the feature maps at that stage, based on the input resolution of $224\times 224$. Stage | Layer Name | LIT-Ti | LIT-S | LIT-M | LIT-B
---|---|---|---|---|---
Stage 1 $56\times 56$ | Patch Embedding | $\begin{array}[]{l}P_{1}=4\\\ C_{1}=64\end{array}$ | $\begin{array}[]{l}P_{1}=4\\\ C_{1}=96\end{array}$ | $\begin{array}[]{l}P_{1}=4\\\ C_{1}=96\end{array}$ | $\begin{array}[]{l}P_{1}=4\\\ C_{1}=128\end{array}$
MLP Block | $\begin{array}[]{l}L_{1}=3\end{array}$ | $\begin{array}[]{l}L_{1}=2\end{array}$ | $\begin{array}[]{l}L_{1}=2\end{array}$ | $\begin{array}[]{l}L_{1}=2\end{array}$
Stage 2 $28\times 28$ | DTM | $\begin{array}[]{l}P_{2}=2\\\ C_{2}=128\end{array}$ | $\begin{array}[]{l}P_{2}=2\\\ C_{2}=192\end{array}$ | $\begin{array}[]{l}P_{2}=2\\\ C_{2}=192\end{array}$ | $\begin{array}[]{l}P_{2}=2\\\ C_{2}=256\end{array}$
MLP Block | $\begin{array}[]{l}L_{2}=4\end{array}$ | $\begin{array}[]{l}L_{2}=2\end{array}$ | $\begin{array}[]{l}L_{2}=2\end{array}$ | $\begin{array}[]{l}L_{2}=2\end{array}$
Stage 3 $14\times 14$ | DTM | $\begin{array}[]{l}P_{3}=2\\\ C_{3}=320\end{array}$ | $\begin{array}[]{l}P_{3}=2\\\ C_{3}=384\end{array}$ | $\begin{array}[]{l}P_{3}=2\\\ C_{3}=384\end{array}$ | $\begin{array}[]{l}P_{3}=2\\\ C_{3}=512\end{array}$
Transformer Block | $\begin{array}[]{l}N_{3}=5\\\ L_{3}=6\end{array}$ | $\begin{array}[]{l}N_{3}=12\\\ L_{3}=6\end{array}$ | $\begin{array}[]{l}N_{3}=12\\\ L_{3}=18\end{array}$ | $\begin{array}[]{l}N_{3}=16\\\ L_{3}=18\end{array}$
Stage 4 $7\times 7$ | DTM | $\begin{array}[]{l}P_{4}=2\\\ C_{4}=512\end{array}$ | $\begin{array}[]{l}P_{4}=2\\\ C_{4}=768\end{array}$ | $\begin{array}[]{l}P_{4}=2\\\ C_{4}=768\end{array}$ | $\begin{array}[]{l}P_{4}=2\\\ C_{4}=1024\end{array}$
Transformer Block | $\begin{array}[]{l}N_{4}=8\\\ L_{4}=3\end{array}$ | $\begin{array}[]{l}N_{4}=24\\\ L_{4}=2\end{array}$ | $\begin{array}[]{l}N_{4}=24\\\ L_{4}=2\end{array}$ | $\begin{array}[]{l}N_{4}=32\\\ L_{4}=2\end{array}$

##  4 Experiments

###  4.1 ImageNet Classification

We conduct experiments on ImageNet (ILSVRC2012) [28] dataset. ImageNet is a
large-scale dataset which has $\sim$1.2M training images from 1K categories
and 50K validation images. We compare with CNN-based ResNet [11] and
Transformer-based models including DeiT [32], PVT [37] and Swin [21]. For
simplicity, we denote them as “Model-Ti/S/M/B” to refer to their tiny, small,
medium and base variants. Similarly, we define four variants of our LIT
models: LIT-Ti, LIT-S, LIT-M and LIT-B. Detailed architecture settings are
shown in Table 1. For better comparison, we design LIT-Ti as a counterpart to
PVT-S, where both models adopt the absolute positional encoding. Our LIT-S,
LIT-M and LIT-B use the relative positional encoding, and these three models
can be seen as competitors to Swin-Ti, Swin-S, Swin-B, respectively.

Implementation details. In general, all models are trained on ImageNet with
300 epochs and a total batch size of 1024. For all ImageNet experiments,
training images are resized to $256\times 256$, and $224\times 224$ patches
are randomly cropped from an image or its horizontal flip, with the per-pixel
mean subtracted. We use the single-crop setting for testing. We use AdamW
optimizer [22] with a cosine decay learning rate scheduler. The initial
learning rate is $1\times 10^{-3}$, and the weight decay is set to $5\times
10^{-2}$. The initial values of learnable offsets in DTM are set to 0, and the
initial learning rate for offset parameters is set to $1\times 10^{-5}$. For a
fair comparison, we adopt the same training strategies as PVT and Swin when
comparing our models to each of them.

Table 2: Comparisons with state-of-the-art methods on ImageNet. All models are trained and evaluated with the input resolution of $224\times 224$. Method |  Params (M) |  FLOPs (G) |  Top-1 Acc. (%)
---|---|---|---
ResNet-18 [11] |  12 |  1.8 |  68.5
ResNet-50 |  26 |  4.1 |  78.5
ResNet-101 |  45 |  7.9 |  79.8
DeiT-Ti [32] |  5 |  1.3 |  72.2
DeiT-S |  22 |  4.6 |  79.8
DeiT-B |  86 |  17.5 |  81.8
PVT-Ti [37] |  13 |  1.9 |  75.1
PVT-S |  25 |  3.8 |  79.8
PVT-M |  44 |  6.7 |  81.2
PVT-L |  61 |  9.8 |  81.7
Swin-Ti [21] |  29 |  4.5 |  81.3
Swin-S |  50 |  8.7 |  83.0
Swin-B |  88 |  15.4 |  83.3
LIT-Ti |  19 |  3.6 |  81.1
LIT-S |  27 |  4.1 |  81.5
LIT-M |  48 |  8.6 |  83.0
LIT-B |  86 |  15.0 |  83.4

Table 3: Effect of our architecture design principle. * denotes the LIT model
which adopts the same uniform patch merging strategies as in PVT-S or Swin-Ti.

Model | Params | FLOPs | Top-1 Acc. (%)
---|---|---|---
PVT-S [37] | 25M | 3.8G | 79.8
LIT-Ti* | 19M | 3.6G | 80.4
Swin-Ti [21] | 28M | 4.5G | 81.3
LIT-S* | 27M | 4.1G | 81.3

Table 4: Effect of the proposed deformable token merging module. We replace
the uniform patch merging strategies in PVT-S and Swin-Ti with our DTM.

Model | Params | FLOPs | Top-1 Acc. (%)
---|---|---|---
PVT-S [37] | 25M | 3.8G | 79.8
PVT-S + DTM | 25M | 3.8G | 80.5
Swin-Ti [21] | 28M | 4.5G | 81.3
Swin-Ti + DTM | 28M | 4.5G | 81.6

Results on ImageNet. In Table 2, we compare LIT with several state-of-the-art
methods on ImageNet. In general, all LIT models achieve on par or better
performance than their counterparts while having fewer parameters and less
FLOPs, without applying any existing efficient self-attention mechanisms.
Specifically, our LIT-Ti outperforms PVT-S by 1.3% on the Top-1 accuracy while
the FLOPs is reduced by 0.2G. LIT-S surpasses Swin-Ti by 0.2% with 0.4G less
FLOPs. LIT-M achieves on par performance with Swin-S, whereas the FLOPs of
LIT-M is reduced. LIT-B brings 0.1% Top-1 accuracy increase over Swin-B while
using 0.4G less FLOPs. For ResNet and DeiT, LIT demonstrates better
performance when compared to them with the same magnitude of FLOPs and
parameters (e.g., ResNet-50 v.s. LIT-S, DeiT-B v.s. LIT-B).

###  4.2 Ablation Studies

Effect of the architecture design. To explore the effect of our architecture
design principle in LIT, we conduct experiments on ImageNet and compare the
architecture of LIT with two recent HVTs: PVT-S [37] and Swin-Ti [21]. The
results are shown in Table 4. In general, our architecture improves baseline
PVT-S by 0.6% in Top-1 accuracy while using less FLOPs (3.6G v.s. 3.8G). For
Swin-Ti, our method reduces the FLOPs by 0.4G while achieving on par
performance. It is also worth noting that the total number of parameters is
reduced for both PVT-S and Swin-Ti. The overall performance demonstrates the
effectiveness of the proposed architecture.

Effect of deformable token merging. To verify the effectiveness of our
proposed DTM strategy, we replace the default patch merging scheme in PVT-S
and Swin-Ti with DTM and train the models on ImageNet. The results are shown
in Table 4. We observe that for both models, DTM introduces negligible FLOPs
and parameters while improving PVT-S and Swin-Ti by 0.7% and 0.3% in terms of
the Top-1 accuracy, respectively. Furthermore, we visualize the learned
offsets in Figure 2. As it shows, unlike previous uniform patch merging
strategy where sampled locations are limited within the green rectangle, our
DTM adaptively merges patches according to objects’ scales and shapes (e.g.,
koala leg, keyboard, cat tail). This again emphasizes the power of LIT to
accommodate various geometric transformations.

Figure 2: Visualization of learned offsets by the proposed deformable token
merging modules. Each image shows $4^{3}$ sampling locations (red dots) in
three DTM modules with $2\times 2$ filter. The green rectangle outlines a
$32\times 32$ patch of the original image, which also indicates a regular
sampling field of previous methods. Best viewed in color.

Effect of MSA in each stage. To explore the effect of self-attention in recent
HVTs, we train PVT-S on ImageNet and gradually remove self-attention layers at
each stage. The results are presented in Table 5. First, after replacing the
SRA layers in PVT-S with standard MSA layers, we observe 1.1% improvement on
the Top-1 accuracy whereas the FLOPs is almost doubled. This indicates that
PVT makes a trade-off between performance and efficiency. Next, by gradually
removing MSA layers in the first two stages, the Top-1 accuracy only drops by
0.1%, 0.5%, respectively. It implies that the self-attention layers in the
early stages of PVT contribute less than expected to the final performance,
and they perform not much better than pure MLP layers. It can be attributed to
the fact that shallow layers focus more on encoding local patterns. However,
we observe a huge performance drop when removing self-attention layers in the
last two stages. The results show that the self-attention layers play an
important role in the later stages and capturing long range dependencies is
essential for well-performed hierarchical vision Transformers.

Table 5: Impact of the MSA layers at each stage. Note that PVT-S has four
stages, which adopts SRA at all blocks instead of MSA. Here we denote “w/ MSA”
as PVT-S with standard MSA layers. “Removed Stage” refers to the stages where
we remove all self-attention layers. For example, “1,2” means we remove the
self-attention layers in the first two stages. Note that “0” means a model
without removing any self-attention layers.

Model | Removed Stage | Params (M) | FLOPs (G) | Top-1 Acc. (%)
---|---|---|---|---
PVT-S [37] | 0 | 25 | 3.8 | 79.8
PVT-S w/ MSA | 0 | 20 | 8.4 | 80.9
PVT-S w/ MSA | 1 | 20 | 4.5 | 80.8
PVT-S w/ MSA | 1,2 | 19 | 3.6 | 80.4
PVT-S w/ MSA | 1,2,3 | 17 | 3.0 | 75.0
PVT-S w/ MSA | 1,2,3,4 | 14 | 2.8 | 66.8

To better understand the phenomenon, we visualize the attention probabilities
for PVT-S without removing any MSA layers, which are depicted in Figure 3.
First, the attention map at the first stage shows that the query pixel almost
pays no attention to other locations. At the second stage, the receptive field
of the query pixel is slightly enlarged, but similar to the first stage.
Considering that PVT-S only has one head at the first stage and two heads at
the second stage, this strongly supports our hypothesis that very few heads in
an MSA layer result in a smaller receptive field, such that a self-attention
layer is almost equivalent to an FC layer. Furthermore, we observe relatively
larger receptive fields from the attention maps of the last two stages. As a
large receptive field usually helps to model longer dependencies, this
explains the huge performance drop in Table 5 after we remove the MSA layers
in the last two stages.

Figure 3: Attention probabilities of PVT-S with standard MSA layers. For each
stage, we visualize the attention map of each head (columns) at each block
(rows). All attention maps are averaged over 100 validation images. Each map
shows the attention probabilities of a query pixel (green rectangle) to other
pixels. Darker color indicates higher attention probability and vice versa.
Best viewed in color. Table 6: Object detection performance on the COCO
val2017 split using the RetinaNet framework.

Backbone | RetinaNet
---|---
Params (M) | $\mathrm{AP}$ | $\mathrm{AP}_{50}$ | $\mathrm{AP}_{75}$ | $\mathrm{AP}_{S}$ | $\mathrm{AP}_{M}$ | $\mathrm{AP}_{L}$
ResNet-50 [11] | 38 | 36.3 | 55.3 | 38.6 | 19.3 | 40.0 | 48.8
PVT-S [37] | 34 | 40.4 | 61.3 | 43.0 | 25.0 | 42.9 | 55.7
LIT-Ti | 30 | 41.6 | 62.8 | 44.7 | 25.7 | 44.4 | 56.4
ResNet-101 [11] | 57 | 38.5 | 57.8 | 41.2 | 21.4 | 42.6 | 51.1
Swin-Ti [21] | 39 | 41.5 | 62.1 | 44.2 | 25.1 | 44.9 | 55.5
LIT-S | 39 | 41.6 | 62.7 | 44.1 | 25.6 | 44.7 | 56.5

Table 7: Object detection and instance segmentation performance on the COCO
val2017 split using the Mask R-CNN framework. $\mathrm{AP}^{b}$ and
$\mathrm{AP}^{m}$ denote the bounding box AP and mask AP, respectively.

Backbone | Mask R-CNN
---|---
Params (M) | $\mathrm{AP}^{b}$ | $\mathrm{AP}^{b}_{50}$ | $\mathrm{AP}^{b}_{75}$ | $\mathrm{AP}^{m}$ | $\mathrm{AP}^{m}_{50}$ | $\mathrm{AP}^{m}_{75}$
ResNet-50 [11] | 44 | 38.0 | 58.6 | 41.4 | 34.4 | 55.1 | 36.7
PVT-S [37] | 44 | 40.4 | 62.9 | 43.8 | 37.8 | 60.1 | 40.3
LIT-Ti | 40 | 42.0 | 64.9 | 45.6 | 39.1 | 61.9 | 41.9
ResNet-101 [11] | 63 | 40.4 | 61.1 | 44.2 | 36.4 | 57.7 | 38.8
Swin-Ti [21] | 48 | 42.2 | 64.6 | 46.2 | 39.1 | 61.6 | 42.0
LIT-S | 48 | 42.9 | 65.6 | 46.9 | 39.6 | 62.3 | 42.4

###  4.3 Object Detection and Instance Segmentation on COCO

In this section, we conduct experiments on COCO 2017 [20] dataset to show the
performance of LIT on object detection and instance segmentation. COCO is a
large-scale dataset which contains $\sim$118K images for the training set and
$\sim$5K images for the validation set. For a fair comparison, we evaluate
LIT-Ti and LIT-S on two base detectors: RetinaNet [19] and Mask R-CNN [10].
For the experiments with both detectors, we consider CNN-based ResNet [11] and
Transformer-based models including PVT-S [37] and Swin-Ti [21]. Following
common practice [2, 37], we measure the performance of all models by Average
Precision (AP) in COCO.

Implementation details. All models are trained on 8 V100 GPUs, with $1\times$
schedule (12 epochs) and a total batch size of 16. We use AdamW [22] optimizer
with a step decay learning rate scheduler. Following PVT [37], the initial
learning rates are set to $1\times 10^{-4}$ and $2\times 10^{-4}$ for
RetinaNet and Mask R-CNN, respectively. The weight decay is set to $1\times
10^{-4}$ for all models. Results of Swin-Ti based detectors are adopted from
Chu et al. [4]. At the training stage, we initialize the backbone with the
pretrained weights on ImageNet. The training images are resized to the shorter
size of 800 pixels, and the longer size is at most 1333 pixels. During
inference, we fix the shorter side of an image to 800 pixels.

Results on COCO. Table 6 shows the comparisons of different backbones on
object detection based on RetinaNet. By comparing LIT with PVT and ResNet
counterparts, we find that our model outperforms both backbones on object
detection in almost all metrics. Similar results can be found in Table 7,
where LIT again surpasses compared methods on object detection and instance
segmentation using the Mask R-CNN framework. The consistent improvement
demonstrates the effectiveness of LIT for dense prediction tasks.

##  5 Conclusion and Future Work

In this paper, we have introduced LIT, a hierarchical vision transformer which
pays less attention in the early stages to ease the huge computational cost of
self-attention modules over high-resolution representations. Specifically, LIT
applies MLP blocks in the first two stages to focus on local patterns while
employing standard Transformer blocks with sufficient heads in the later
stages to handle long range dependencies. Moreover, we have proposed a
deformable token merging module, which is learned to adaptively merge
informative patches to an output unit, with enhanced geometric
transformations. Extensive experiments on ImageNet and COCO have demonstrated
that LIT achieves better performance compared with existing state-of-the-art
HVT methods. Future works may include finding better architectural
configurations of LIT with neural architecture search (NAS) and improving MLP
blocks in the early stages to enhance the capability of LIT to encode local
patterns. Besides, one may also consider applying efficient self-attention
mechanisms in the later stages to achieve better efficiency, such as
kernelization [26], low-rank decomposition [36], memory [27], and sparsity [3]
schemes, etc. Also note that LIT potentially brings some negative societal
impacts, such as the unintended use for surveillance and the huge energy
consumption and carbon emissions of training LIT models on ImageNet.

## References

  * [1] Jimmy Ba, J. Kiros, and Geoffrey E. Hinton.  Layer normalization.  ArXiv, abs/1607.06450, 2016.
  * [2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In ECCV, pages 213–229, 2020.
  * [3] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever.  Generating long sequences with sparse transformers.  arXiv preprint arXiv:1904.10509, 2019.
  * [4] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen.  Twins: Revisiting the design of spatial attention in vision transformers.  Arxiv preprint 2104.13840, 2021.
  * [5] Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi.  On the relationship between self-attention and convolutional layers.  In ICLR, 2020.
  * [6] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  In ICCV, pages 764–773, 2017.
  * [7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In ACL, pages 4171–4186, 2019.
  * [8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  ICLR, 2021.
  * [9] Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze.  Levit: a vision transformer in convnet’s clothing for faster inference.  arXiv preprint arXiv:22104.01136, 2021.
  * [10] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick.  Mask R-CNN.  In ICCV, pages 2980–2988, 2017.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * [12] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  arXiv: Learning, 2016.
  * [13] Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh.  Rethinking spatial dimensions of vision transformers.  arXiv preprint arXiv:2103.16302, 2021.
  * [14] Qibin Hou, Ming-Ming Cheng, Xiaowei Hu, Ali Borji, Zhuowen Tu, and Philip H. S. Torr.  Deeply supervised salient object detection with short connections.  TPAMI, 41(4):815–828, 2019.
  * [15] Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin.  Local relation networks for image recognition.  In ICCV, pages 3463–3472, 2019.
  * [16] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In ICML, pages 448–456, 2015.
  * [17] Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool.  Localvit: Bringing locality to vision transformers.  arXiv preprint arXiv:2104.05707, 2021.
  * [18] Tsung-Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie.  Feature pyramid networks for object detection.  In CVPR, pages 936–944, 2017.
  * [19] Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In ICCV, pages 2999–3007, 2017.
  * [20] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: common objects in context.  In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, ECCV, pages 740–755, 2014.
  * [21] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  arXiv preprint arXiv:2103.14030, 2021.
  * [22] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In ICLR, 2019.
  * [23] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard S. Zemel.  Understanding the effective receptive field in deep convolutional neural networks.  In NeurIPS, pages 4898–4906, 2016.
  * [24] Zizheng Pan, Bohan Zhuang, Jing Liu, Haoyu He, and Jianfei Cai.  Scalable visual transformers with hierarchical pooling.  arXiv preprint arXiv:2103.10619, 2021.
  * [25] Niki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens.  Stand-alone self-attention in vision models.  In NeurIPS, pages 68–80, 2019.
  * [26] Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A Smith, and Lingpeng Kong.  Random feature attention.  In ICLR, 2021.
  * [27] Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap.  Compressive transformers for long-range sequence modelling.  In ICLR, 2020.
  * [28] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  IJCV, pages 211–252, 2015.
  * [29] Gyumin Shim, Jinsun Park, and In So Kweon.  Robust reference-based super-resolution with similarity-aware deformable convolution.  In CVPR, pages 8422–8431, 2020.
  * [30] Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani.  Bottleneck transformers for visual recognition.  arXiv preprint arXiv:2101.11605, 2021.
  * [31] Hugues Thomas, Charles R. Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J. Guibas.  Kpconv: Flexible and deformable convolution for point clouds.  In ICCV, pages 6410–6419, 2019.
  * [32] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  arXiv preprint arXiv:2012.12877, 2020.
  * [33] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou.  Going deeper with image transformers.  arXiv preprint arXiv:2103.17239, 2021.
  * [34] Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens.  Scaling local self-attention for parameter efficient visual backbones.  In CVPR, 2021.
  * [35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, pages 5998–6008, 2017.
  * [36] Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma.  Linformer: Self-attention with linear complexity.  arXiv preprint arXiv:2006.04768, 2020.
  * [37] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  arXiv preprint arXiv:2102.12122, 2021.
  * [38] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang.  Cvt: Introducing convolutions to vision transformers.  arXiv preprint arXiv:2103.15808, 2021.
  * [39] Zhe Wu, Li Su, and Qingming Huang.  Cascaded partial decoder for fast and accurate salient object detection.  In CVPR, pages 3907–3916, 2019.
  * [40] Haotian Yan, Zhe Li, Weijian Li, Changhu Wang, Ming Wu, and Chuang Zhang.  Contnet: Why not use convolution and transformer at the same time?  arXiv preprint arXiv:2104.13497, 2021.
  * [41] Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu.  Incorporating convolution designs into visual transformers.  arXiv preprint arXiv:2103.11816, 2021.
  * [42] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi Feng, and Shuicheng Yan.  Tokens-to-token vit: Training vision transformers from scratch on imagenet.  arXiv preprint arXiv:2101.11986, 2021.
  * [43] Matthew D. Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In ECCV, pages 818–833, 2014.
  * [44] Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Qibin Hou, and Jiashi Feng.  Deepvit: Towards deeper vision transformer.  arXiv preprint arXiv:2103.11886, 2021.
  * [45] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai.  Deformable convnets V2: more deformable, better results.  In CVPR, pages 9308–9316, 2019.
  * [46] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  In ICLR, 2021.


# Unsupervised Open-domain Keyphrase Generation

Lam Thanh Do♠ Pritom Saha Akash♡ Kevin Chen-Chuan Chang♡††footnotemark:
♠Hanoi University of Science and Technology, Viet Nam
♡University of Illinois at Urbana-Champaign, USA
<EMAIL_ADDRESS>
{pakash2<EMAIL_ADDRESS>Work done while visiting Cazoodle Inc.

###### Abstract

In this work, we study the problem of unsupervised open-domain keyphrase
generation, where the objective is a keyphrase generation model that can be
built without using human-labeled data and can perform consistently across
domains. To solve this problem, we propose a seq2seq model that consists of
two modules, namely phraseness and informativeness module, both of which can
be built in an unsupervised and open-domain fashion. The phraseness module
generates phrases, while the informativeness module guides the generation
towards those that represent the core concepts of the text. We thoroughly
evaluate our proposed method using eight benchmark datasets from different
domains. Results on in-domain datasets show that our approach achieves state-
of-the-art results compared with existing unsupervised models, and overall
narrows the gap between supervised and unsupervised methods down to about 16%.
Furthermore, we demonstrate that our model performs consistently across
domains, as it overall surpasses the baselines on out-of-domain
datasets.111Code and data will be available at
https://github.com/ForwardDataLab/UOKG..

##  1 Introduction

Figure 1: Overview of our proposed model

Keyphrases are short word sequences that describe the core concepts of the
text. The prediction of keyphrases for a text is a task that has received much
attention recently. It is a crucial problem as its outputs can be useful for a
variety of downstream tasks such as building digital libraries Gutwin et al.
(1999); Witten et al. (2009), document summarization Litvak and Last (2008),
document visualization Chuang et al. (2012) and so on.

There are mainly two approaches to keyphrase prediction, namely keyphrase
extraction Mihalcea and Tarau (2004); Florescu and Caragea (2017a); Bennani-
Smires et al. (2018) and keyphrase generation Meng et al. (2017); Chen et al.
(2019); Yuan et al. (2020); Shen et al. (2022). Keyphrase extraction
highlights keyphrases that appear within the text. On the other hand,
keyphrase generation generates keyphrases based on the understanding of the
given text and therefore allows predicting absent keyphrases alongside present
ones Meng et al. (2017). This ability has made keyphrase generation receive
more attention than keyphrase extraction in recent years, as human also tend
to use keyphrases that are absent from the text.

Most of the existing keyphrase generation models use manually labeled data for
training Meng et al. (2017); Chen et al. (2018, 2019); Yuan et al. (2020);
Ahmad et al. (2021). However, obtaining labeled data is often the most
expensive component of any machine learning model, and this is the same for
keyphrase generation. Compared to labeled data, access to unlabeled data is
easier and mostly available. For example, the arXiv dataset Clement et al.
(2019) containing metadata (e.g., title, abstract) of 1.7 million research
articles is readily available on Kaggle. Therefore, it is more desirable to
construct a keyphrase generation model in an unsupervised fashion.
Furthermore, in practice, the model may have to encounter texts that come from
various domains or even unseen ones. Therefore, another attractive property of
a keyphrase generation model is the ability to handle open-domain documents.

Considering the above scenario, we propose a new problem called Unsupervised
Open-domain Keyphrase Generation. Similar to every keyphrase generation
methods, the model of our objective is given a text $\boldsymbol{x}$ as input,
and as output, it generates a set of keyphrases $\\{\boldsymbol{y}\\}$. Both
$\boldsymbol{x}$ and $\boldsymbol{y}$ are word sequences. Furthermore, the
model of our objective should satisfy two requirements: 1) it can be built
using only an unlabeled corpus, denoted as $D$; 2) it can effectively handle
inputs from across domains.

This is a challenging task because we do not have access to labeled data from
which to learn the patterns for keyphrases. Additionally, we also need our
model to work across domains. This is difficult because there might exist
different patterns for keyphrases for different domains. None of the existing
work addresses these challenges. For instance, supervised keyphrase generation
models Meng et al. (2017); Chen et al. (2018, 2019); Yuan et al. (2020); Ahmad
et al. (2021) not only require manually labeled data for training but are also
known to perform poorly when being moved out-of-domain. On the other hand,
Shen et al. (2022) propose AutoKeyGen, which uses pseudo-labeled data to train
a seq2seq model in a weakly-supervised fashion, thereby removing the need for
human annotation effort. However, similar to supervised models, the weakly-
supervised approach taken by AutoKeyGen does not enable it to maintain
performance in unseen domains.

Therefore, to solve our problem, we propose an unsupervised keyphrase
generation model that can work across domains. The key idea is to modularize a
seq2seq model into two modules. The motivation for modularizing is to
decompose keyphrase generation into two simpler problems where each of which
can be addressed in an unsupervised and open-domain setting. The first module,
named the phraseness module, is responsible for generating phrases, while the
second module, named the informativeness module, guides the generation toward
the phrases that represent the most crucial concepts of the text.

The phraseness module is a retrieval-augmented seq2seq model, where the
retriever assists the seq2seq component in generating absent phrases alongside
present ones. This module can be built in an unsupervised fashion because it
leverages noun phrases to index the retriever and to train the seq2seq model,
which can easily be obtained using open-sourced software libraries such as
NLTK Bird et al. (2009), and therefore does not require human annotation
effort. Furthermore, the phraseness module can also be built in an open-domain
fashion, thanks to 1) the part-of-speech information incorporated into the
seq2seq model, which allows copying words to form grammatically correct noun
phrases regardless of domains; 2) the fact that the retriever can be further
indexed with domain-relevant information, to provide reliable references.

The informativeness module is another seq2seq model, where a phrase is likely
to be generated if it contains words that are informative to the given text.
Inspired by embedding-based unsupervised keyphrase extraction (UKE) methods,
we quantify informativeness of a word and a text based on their closeness in
meaning, which is measured via the similarity between their embeddings. We
choose this method of evaluating informativeness over other UKE methods (e.g.
graph-based, statistics based) since it supports not only present phrases, but
also absent ones. Similar to the phraseness module, the informativeness module
can also be built in an unsupervised and open-domain fashion. This is obtained
by using a domain-general, unsupervised text embedding model (e.g. Sent2Vec
Pagliardini et al. (2018)).

We summarize the contributions of our paper. Firstly, we propose a new problem
called unsupervised open-domain keyphrase generation. Secondly, we design a
model for solving the problem. Our proposed model is a seq2seq model that
consists of two modules, one is responsible for generating phrases and the
other guides the generation towards the phrases that represent the core
concepts of the text. Finally, we conduct extensive experiments on multiple
datasets across domains to demonstrate the effectiveness of our model as we
contrast it against multiple strong baselines.

##  2 Proposed method

Figure 1 illustrates our proposed framework. We propose a seq2seq model that
consists of two modules, namely phraseness and informativeness module. We
adopt the two terms phraseness and informativeness from Tomokiyo and Hurst
(2003), to describe the desirable criteria a keyphrase should satisfy.
Phraseness refers to the degree to which a word sequence is considered a
phrase, and informativeness refers to how well the phrase illustrates the core
concepts of the text. Each of the two modules guarantees a criterion mentioned
above. In particular, the phraseness module generates (present and absent)
phrases, while the informativeness module guides the generation toward phrases
that describe the core concepts of the text. In the following sections, we
will describe in detail the two modules, as well as how they are combined to
generate keyphrases.

Figure 2: Illustration of the retrieval of references.

###  2.1 Phraseness module

In order to generate keyphrases, it is crucial to know how to first generate
phrases. We emphasize the difference between a keyphrase and a phrase - the
former needs to be informative to the given text, while the latter does not.
It has been shown that keyphrases mostly take the form of noun phrases Chuang
et al. (2012). Also, recent work on keyphrase generation has shown that absent
keyphrases can often be retrieved from other texts Ye et al. (2021),
suggesting that absent phrases can be found similarly. Therefore, a simple
solution to obtaining phrases is to extract noun phrases as present phrases
and retrieve related noun phrases as absent ones.

However, this simple solution may not be optimal. Since the retrieved phrases
are originally used in other texts, they may not be suitable to describe the
concepts of the given text. We demonstrate this limitation using the example
in Figure 3(a). In this example, the absent phrases obtained via retrieval
describe concepts related to “topic modeling”. However, our desired outputs
need to also describe concepts related to “author modeling”.

The above problem could be mitigated if we also consider the given text
alongside the retrieved noun phrases. In the example above, relevant phrases
such as “author topic distributions” can be generated by combining “author”,
which is from the given text, and “topic distributions”, which is one of the
retrieved phrases. With this in mind, we employ a retrieval-augmented seq2seq
model as the phraseness module. First, a set of related but absent noun
phrases is retrieved, which we will now refer to as references. Then, a
seq2seq model generates noun phrases based on both the text and the
references.

((a)) Example of using retrieval to predict
absent phrases

((b)) Example of EmbedRank and TextRank’s prediction,
along with the groundtruth keyphrases

Figure 3: Examples

####  2.1.1 Retriever

Figure 2 describes the retrieval of references given a text. To obtain
references for the input, we leverage existing noun phrases observed in other
documents. We assume that a noun phrase is related to a text if it occurs in
contexts similar to that text. With this in mind, we collect noun phrases from
documents in the unlabeled corpus $D$ to form a phrase bank $B$. We index each
noun phrase $\boldsymbol{z}\in B$ with a context embedding, denoted as
$c_{\boldsymbol{z}}$, which is obtained by averaging the embeddings of the
documents in which $\boldsymbol{z}$ appears in. We obtain the embeddings of
texts by using Sent2Vec Pagliardini et al. (2018), an unsupervised sentence
embedding model. To retrieve references for a text $\boldsymbol{x}$, we first
use Sent2Vec to compute its embedding, denoted as $v_{\boldsymbol{x}}$, and
then retrieve the top-$k$ phrases $z$ based on the following retrieval score
$R_{\boldsymbol{x}}(\boldsymbol{z})=\cos(c_{z},v_{\boldsymbol{x}})$.
Furthermore, in order to prevent retrieving unreliable references, we filter
those whose retrieval scores are smaller than a threshold $\tau$. We denote
the set of references for $\boldsymbol{x}$ as $Z_{\boldsymbol{x}}$.

As mentioned above, we can further index the retriever with other corpora,
denoted as $D^{\prime}$, from different domains. To do this, all we need to do
is to update the phrase bank $B$ with new phrases from $D^{\prime}$ and update
the context embeddings of every phrase that occur in both $D$ and
$D^{\prime}$.

####  2.1.2 Seq2Seq model

Input representation. The seq2seq model takes as inputs not only the source
text $\boldsymbol{x}$ but also its references $Z_{\boldsymbol{x}}$, to
generate phrases. The text and its references are combined into a single input
$\boldsymbol{\tilde{x}}$, defined as

$\displaystyle\boldsymbol{\tilde{x}}=\text{{[BOS]}}\ \boldsymbol{x}\
\text{{[BOR]}}\ Z_{\boldsymbol{x}}\ \text{{[EOR]}}\ \text{{[EOS]}}$ (1)

where [BOS] and [EOS] are respectively the beginning and end of sentence
token. The two tokens [BOR] and [EOR] signals the start and end of the
reference block. In addition, the references are separated by a [SEP] token.

Model architecture. We employ Transformer Vaswani et al. (2017) with copy
mechanism Gu et al. (2016); See et al. (2017) as the architecture of our
seq2seq model. First, the encoder receives the word embeddings of the tokens
in $\boldsymbol{\tilde{x}}$, producing a sequence of encoder hidden states
$\boldsymbol{h}=\\{h_{i}\\}_{i=1}^{|\boldsymbol{\tilde{x}}|}$. The decoder
takes the embeddings of the previously generated words $\boldsymbol{y}_{<t}$
and the encoder hidden states, outputting the decoder hidden state $s_{t}$.
For each input, we build an extended vocabulary $V_{\boldsymbol{\tilde{x}}}$,
which is the union of the decoder’s vocabulary $V$ and the words in the
augmented input $\boldsymbol{\tilde{x}}$. Finally, we compute the phraseness
probability of predicting a word from $V_{\boldsymbol{\tilde{x}}}$ as
$P_{\text{pn}}(y_{t}|\boldsymbol{y}_{<t},\boldsymbol{\tilde{x}})=p_{\text{gen}}P^{V}_{\text{pn}}(y_{t}|\boldsymbol{y}_{<t},\boldsymbol{\tilde{x}})+(1-p_{\text{gen}})P^{C}_{\text{pn}}(y_{t}|\boldsymbol{y}_{<t},\boldsymbol{\tilde{x}})$.
Here,
$P^{V}_{\text{pn}}(y_{t}|\boldsymbol{y}_{<t},\boldsymbol{\tilde{x}})=\text{{softmax}}(W^{V}s_{t})$
is the distribution over the word vocabulary $V$,
$p_{\text{gen}}=\text{{sigmoid}}(W^{\text{g}}_{s}s_{t}+W^{\text{g}}_{y}y_{t-1})$
is the soft switch between generating and copy. All the $W$ terms are
trainable parameters, and we omit the bias terms for less cluttered notation.

We incorporate part-of-speech information to copy words from
$\boldsymbol{\tilde{x}}$. More formally,
$P^{C}_{\text{pn}}(y_{t}=w|\boldsymbol{y}_{<t},\boldsymbol{\tilde{x}})=\sum_{\tilde{x_{i}}=w}a_{i}^{t}$,
where $a_{i}^{t}=\text{{softmax}}(e_{i}^{t})$, and
$e_{i}^{t}=\text{{FF}}_{h}(\tilde{h_{i}}^{T})\text{{FF}}_{s}(\tilde{s_{t}})$.
Here, $\tilde{h_{i}}=\text{{concat}}(h_{i},l_{\tilde{x}_{i}})$ is the encoder
hidden state of $\tilde{x}_{i}$ enhanced with its part-of-speech embedding
$l_{\tilde{x}_{i}}$. Similarly,
$\tilde{s_{t}}=\text{{concat}}(s_{t},l_{y_{t}})$ is the decoder hidden state
enhanced with the part-of-speech embedding of the previously generated word.
$\text{{FF}}_{h}$ and $\text{{FF}}_{s}$ denotes the feedforward neural
networks, whose purposes are to help project $\tilde{h_{i}}$ and
$\tilde{s_{t}}$ into the same semantic space.

Model training. For every document $\boldsymbol{x_{i}}\in D$, we maximize
$\log P_{\text{pn}}(\boldsymbol{y}=\boldsymbol{z}|\boldsymbol{\tilde{x}})$,
where

$\displaystyle
P_{\text{pn}}(\boldsymbol{y}=\boldsymbol{z}|\boldsymbol{\tilde{x}})=\prod_{t=1}^{T}P_{\text{pn}}(y_{t}=z_{t}|\boldsymbol{y}_{<t},\boldsymbol{\tilde{x}})$
(2)

for the phrases $\boldsymbol{z}=\\{z_{t}\\}$, which include the present noun
phrases and the references. To encourage the model to generate absent phrases
instead of just copying it from the references, we randomly mask some
references and train the model to generate them.

###  2.2 Informativeness module

Knowing to generate phrases is not sufficient to obtain keyphrases. It is also
important to guide the generation towards the phrases that are informative to
the input. Previous work on unsupervised keyphrase extraction offer multiple
classes of methods, namely graph-based, statistics-based and embedding-based,
for evaluating informativeness of phrases (for more details, see Section 5).
Graph-based and statistics-based methods are not suitable in our setting.
These methods utilize only in-text information and therefore cannot determine
the informativeness of absent phrases. On the other hand, embedding-based
methods evaluate informativeness of a phrase based on its closeness in meaning
with the input text. As a result, these methods can support both present and
absent phrases. We therefore adopt the idea of embedding-based methods in
building our informativeness module.

Let us define
$S(\boldsymbol{a},\boldsymbol{b})=\max(0,v_{\boldsymbol{a}}^{T}v_{\boldsymbol{b}})$
as the similarity score between two pieces of text, where
$v_{\boldsymbol{a}}$, $v_{\boldsymbol{b}}$ are embeddings obtained using
Sent2Vec. Using this score, we define the informativeness distribution
$P_{\text{in}}(\boldsymbol{y}|\boldsymbol{x})$, by decomposing it into
conditional distributions of each word given the previous context. More
formally,
$P_{\text{in}}(\boldsymbol{y}|\boldsymbol{x})=\prod_{t=1}^{T}P_{\text{in}}(y_{t}|\boldsymbol{y}_{<t},\boldsymbol{x})$,
where

$\displaystyle
P_{\text{in}}(y_{t}=w|\boldsymbol{y}_{<t},\boldsymbol{x})\propto\begin{cases}S(w,\boldsymbol{x}),&\text{if
$w\neq\text{{[EOS]}}$}\\\
S(\boldsymbol{y}_{<t},\boldsymbol{x}),&\text{otherwise}\end{cases}$ (3)

The probability $P_{\text{in}}(y_{t}=w|\boldsymbol{y}_{<t},\boldsymbol{x})$ is
normalized over the extended word vocabulary $V_{\boldsymbol{\tilde{x}}}$,
which is the same one used by the phraseness module. Intuitively, a word has
high probability of being generated if that word has close meaning to the
text. The [EOS] token is likely to be generated if the currently generated
phrase $\boldsymbol{y}_{<t}$ already form an informative phrase.

###  2.3 Combining phraseness and informativeness

Generating keyphrases require us to enforce both phraseness and
informativeness on the output sequence. A simple solution is to adopt the
approaches taken by existing unsupervised keyphrase extraction methods, which
enforce the two criteria sequentially. In particular, they either 1) form
phrases first, then choose those that are most informative as keyphrases; or
2) choose informative words first, then form keyphrases using these words.
However, both approaches may not be optimal. The first approach may include
uninformative words in the prediction, while the second rigidly assume that a
keyphrase should only contain keywords. We illustrate the limitation of these
approaches using an example, shown in Figure 3(b). Here, we show the
predictions of EmbedRank Bennani-Smires et al. (2018), which takes approach 1)
and TextRank Mihalcea and Tarau (2004), which takes approach 2). Both of them
fail to predict the golden keyphrase “global illumination”. EmbedRank
redundantly include the word “algorithms”, while TextRank only outputs
“illumination”, as “global” is not predicted as a keyword.

This problem could be alleviated if both phraseness and informativeness is
considered when forming the keyphrase. In the example above, the word
“algorithms” should be excluded, since it neither contributes to the
informativeness of the phrase, nor it is required to make the phrase
understandable. On the other hand, the word “global” may not be among the most
informative words to the text, however, this word is essential as excluding it
results in a phrase with a different concept.

In light of this, we propose to generate keyphrases, one word at a time, where
each word is generated if it is predicted by both the phraseness and
informativeness module. To this end, we propose to combine the two modules in
a product-of-experts fashion Hinton (2002). In particular, the conditional
distribution of a keyphrase given a text is defined as follows

$\displaystyle\begin{split}P_{\text{kp}}(\boldsymbol{y}|\boldsymbol{x})&\propto
P_{\text{pn}}(\boldsymbol{y}|\boldsymbol{\tilde{x}})^{\lambda}\cdot
P_{\text{in}}(\boldsymbol{y}|\boldsymbol{x})\\\
&\propto\prod_{t=1}^{T}P_{\text{pn}}(y_{t}|\boldsymbol{y}_{<t},\boldsymbol{\tilde{x}})^{\lambda}\cdot
P_{\text{in}}(y_{t}|\boldsymbol{y}_{<t},\boldsymbol{x})\end{split}$ (4)

where $\lambda$ is a hyperparameter for balancing the two modules.

The idea of combining two language models using the product-of-experts has
previously been studied for the task of unsupervised abstractive summarization
Zhou and Rush (2019). To the best of our knowledge, we are the first to use
this idea in unsupervised keyphrase generation. In the above paragraphs, we
also discussed why it is a suitable choice.

###  2.4 Keyphrase decoding

To decode keyphrases, we employ beam search to search for keyphrases based on
$s(\boldsymbol{y})=-\log P_{\text{kp}}(\boldsymbol{y}|\boldsymbol{x})$. As
beam search tend to favor shorter keyphrases, we employ the length
normalization strategy similarly to that described in Sun et al. (2019), which
is to divide $s(\boldsymbol{y})$ by $|\boldsymbol{y}|+\alpha$, where $\alpha$
is a length penalty factor.

It has been shown in previous work that positional information is useful for
the prediction of present keyphrases Florescu and Caragea (2017b); Gallina et
al. (2020). Therefore, it is desirable to incorporate this feature into our
model. Furthermore, we found that the model tends to generate absent
keyphrases that are entirely new. This behavior may not be desirable for
downstream tasks such as document retrieval, where we need to associate
documents with common keyphrases. Based on the above discussion, we propose to
rerank the beam search results using the following score

$\displaystyle\hat{s}(\boldsymbol{y})=\frac{s(\boldsymbol{y})}{|\boldsymbol{y}|+\alpha}\times
b(\boldsymbol{y})$ (5)

$\displaystyle b(\boldsymbol{y})=\begin{cases}\beta,&\text{if $\boldsymbol{y}$
is absent and $\boldsymbol{y}\in B$}\\\ 1,&\text{if $\boldsymbol{y}$ is absent
and $\boldsymbol{y}\not\in B$}\\\
\frac{\log_{2}(1+\mathcal{P}_{\boldsymbol{x}}(\boldsymbol{y}))}{\log_{2}(1+\mathcal{P}_{\boldsymbol{x}}(\boldsymbol{y}))+1},&\text{if
$\boldsymbol{y}$ is present}\end{cases}$ (6)

where $b(\boldsymbol{y})$ is an adjustment weight, $\beta$ is a hyperparameter
for adjusting the scores of absent phrases that exist in the phrase bank $B$
($\beta<1$ indicates that we favor $\boldsymbol{y}\in B$), and
$\mathcal{P}_{\boldsymbol{x}}(\boldsymbol{y})$ is the word offset position of
the phrase $\boldsymbol{y}$ in the text $\boldsymbol{x}$. Intuitively,
$b(\boldsymbol{y})$ favors present keyphrases that appear earlier in the text,
and absent keyphrases that exist in the phrase bank $B$.

Dataset name | Language | Type | valid/test docs | #kps/doc | %absent | %overlap
---|---|---|---|---|---|---
SemEval | English | Scientific | 144/100 | 15.4 | 58.2 | 38.8
Inspec | English | Scientific | 1500/500 | 9.7 | 22.7 | 40
NUS | English | Scientific | 50/161 | 11.6 | 53.1 | 53.3
Krapivin | English | Scientific | 1844/460 | 5.3 | 50.5 | 53.5
StackExchange | English |  | Technical
---
Question
16000/16000 | 2.7 | 48.9 | 45.1
DUC-2001 | English | News | 50/268 | 8.1 | 2.7 | 14.7
KPTimes | English | News | 10000/20000 | 5 | 46.2 | 7.5
OpenKP | English | News | 6616/6614 | 2.2 | 10.9 | 12.7

Table 1: Statistics of testing datasets.

Present keyphrase generation
---
| SemEval | Inspec | NUS | Krapivin | StackExchange | Average
F1@3 | F1@5 | F1@3 | F1@5 | F1@3 | F1@5 | F1@3 | F1@5 | F1@3 | F1@5 | F1@3 | F1@5
TF-IDF | 19 | 23.9 | 18.7 | 24.8 | 22.7 |  25.9 | 15.9 | 15.7 |  18.8 |  16.5 | 19 |  21.4
TextRank | 13.8 | 17.2 | 17.7 | 25 | 14.9 | 18.9 | 11.1 | 13.3 | 8.5 | 8.4 | 13.2 | 16.6
MultipartiteRank | 18.9 | 21.4 | 23.1 | 26.5 | 22.7 | 24.9 | 19.3 | 18.5 | 13.9 | 13.6 |  19.6 | 21
EmbedRank | 17.9 | 21.2 | 26.2 | 32.6 | 17.5 | 20.8 | 13.5 | 15.2 | 11.8 | 12.6 | 17.4 | 20.5
Global-Local Rank | 20.4 |  23.6 |  24.5 |  30.6 | 22.4 | 23.7 | 15 | 15.2 | 10.2 | 9.8 | 18.5 | 20.6
AutoKeyGen | 16.64 | 22.14 | 19.42 | 23.13 |  23.24 | 25.73 |  19.57 |  20.65 | 148 | 14.96 | 18.53 | 21.32
Ours |  19.18 | 22.29 | 19.88 | 23.3${}_{1}1$ | 26.47 | 27.84 | 22.28 | 21.49 | 27.21 | 25.12 | 234 | 244
Supervised - CopyRNN | 26.14 | 29.75 | 19.15 | 22.85 | 35.513 | 37.94 | 30.39 | 30.16 | 24.16 | 22.45 | 276 | 28.63
Absent keyphrase generation
| SemEval | Inspec | NUS | Krapivin | StackExchange | Average
R@5 | R@10 | R@5 | R@10 | R@5 | R@10 | R@5 | R@10 | R@5 | R@10 | R@5 | R@10
UKE methods | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
AutoKeyGen | 0.72 | 1.23 | 1.72 | 2.84 | 12 | 1.95 | 2.42 | 3.85 | 1.22 | 1.91 | 1.41 | 2.32
Ours | 1.42 | 2.34 | 2.12 | 32 | 1.88 | 3.15 | 4.55 | 72 | 4.61 | 6.32 | 2.92 | 4.32
Supervised - CopyRNN | 2.13 | 2.73 | 3.73 | 5.33 | 4.44 | 6.47 | 7.92 | 10.77 | 2.32 | 3.53 | 4.11 | 5.73

Table 2: Keyphrase generation performance for in-domain datasets. The best and
second-best results for each category are bold and highlighted, respectively.
For AutoKeyGen, CopyRNN and our model, we run experiments five times with
different random seeds and report the average. The subscript denotes the
corresponding standard deviation (e.g. 26.47 indicates 26.4$\pm$ 0.7). We
report both F1 and Recall in percentage points.

##  3 Experiments

###  3.1 Datasets

We use the documents from the training set of KP20K Meng et al. (2017) to
train our model and to index the retriever in the training phase. It contains
the abstracts and titles of 514k scientific articles. In the testing phase, we
utilize 8 datasets, namely SemEval Kim et al. (2013), Inspec Hulth (2003), NUS
Nguyen and Kan (2007), Krapivin Krapivin et al. (2009), DUC-2001 Wan and Xiao
(2008), OpenKP Xiong et al. (2019), StackExchange Yuan et al. (2020) and
KPTimes Gallina et al. (2019). The title and abstract of an article are
concatenated to form a testing document.

The testing datasets are categorized into in-domain and out-of-domain, by
measuring the percentage of keyphrase overlap with the training corpus, i.e.
the percentage of golden keyphrases in the testing dataset that also appear in
some documents in KP20K. We choose the mean value of $\sim 33$ as a threshold
to classify the testing datasets. As a result, the in-domain datasets include
SemEval, Inspec, NUS, Krapivin and StackExchange, while the other three are
out-of-domain.

In the testing phase, besides using KP20K, we also use the training set of
StackExchange (300k documents) and KPTimes (260k documents) to further index
the phrase bank and the retriever. The purpose of adding these additional
sources in the testing phase is to test whether or not our model can easily
integrate additional information to work in domains unseen during training,
without having it re-trained.

Present keyphrase generation
---
| DUC-2001 | KPTimes | OpenKP | Average
F1@3 | F1@5 | F1@3 | F1@5 | F1@3 | F1@5 | F1@3 | F1@5
TF-IDF | 8.4 | 11.3 | 21.2 | 21.5 |  13.5 |  12.9 | 14.4 | 15.2
TextRank | 9.8 | 14 | 10.1 | 9.7 | 9.8 | 9.3 | 9.9 | 11
MultipartiteRank | 13.7 | 18.7 | 18 | 18.7 | 12.7 | 11.9 |  14.8 |  16.4
EmbedRank | 20 | 24.5 | 10 | 11.8 | 6.9 | 7.3 | 12.3 | 14.5
Global-Local Rank | 14.6 |  21.8 | 10.3 | 10.7 | 11.7 | 10.1 | 12.2 | 14.2
AutoKeyGen | 7.46 | 9.96 | 15.94 | 16.83 | 8.53 | 8.93 | 10.63 | 11.83
Ours |  15.23 | 18.24 |  20.11 |  21.11 | 15.98 | 14.22 | 173 | 17.82
Supervised - CopyRNN | 6.94 | 8.52 | 19.67 | 19.65 | 10.44 | 10.13 | 12.34 | 12.73
Absent keyphrase generation
| DUC-2001 | KPTimes | OpenKP |
R@5 | R@10 | R@5 | R@10 | R@5 | R@10
UKE methods | 0 | 0 | 0 | 0 | 0 | 0
AutoKeyGen | - | - | 0.20 | 0.30 | - | -
Ours | - | - | 31 | 3.60 | - | -
Supervised - CopyRNN | - | - | 0.20 | 0.41 | - | -

Table 3: Results on out-of-domain datasets.

###  3.2 Baselines & evaluation metrics

Baselines. We adopt five unsupervised keyphrase extraction (UKE) algorithms,
namely TF-IDF, TextRank222https://github.com/boudinfl/pke Mihalcea and Tarau
(2004), MultiPartiteRank333See footnote 1 Boudin (2018), EmbedRank Bennani-
Smires et al. (2018) and Global-Local
Rank444https://github.com/xnliang98/uke_ccrank Liang et al. (2021) as
baselines.

We also compare our model with AutoKeyGen Shen et al. (2022), which is the
only previous work on unsupervised keyphrase generation. With the permission
from the authors, we implemented and report the AutoKeyGen-Copy version.
Furthermore, we present CopyRNN Meng et al. (2017) as a supervised baseline.
We employ the Transformer-based pointer-generator network for both AutoKeyGen
and CopyRNN, with the same settings as described in A.1. Both AutoKeyGen and
CopyRNN are trained using KP20K.

Evaluation metrics. We follow the widely-used strategy and separate the
evaluation of present and absent keyphrase generation. We employ macro-average
F1 and macro-average Recall for evaluating present and absent keyphrase
generation, respectively. We evaluate present keyphrases at top 3 and 5
predictions; and absent keyphrases at top 5 and 10. The predictions as well as
the groundtruths are stemmed using Porter
Stemmer555https://github.com/nltk/nltk/blob/develop/nltk/stem/porter.py Porter
(1980) and duplicates are removed before evaluation.

###  3.3 Results

Figure 4: Two examples of the generated keyphrases from AutoKeyGen and our
proposed model. We illustrate the top 5 predictions. Correctly predicted
keyphrases are underlined.

####  3.3.1 Keyphrase generation for in-domain cases

Table 2 illustrates the performance of our proposed model and the baselines
for the five in-domain datasets. We also display the average performance
across datasets.

Present keyphrase generation. For predicting present keyphrases, our model is
best or second-best on most datasets. On SemEval, our model is slightly
inferior to TF-IDF and Global-Local Rank. The results on Inspec are worth
noting, as our proposed model is significantly outperformed by UKE methods.
This inferior performance may be due to this dataset not favoring generative
methods, as even CopyRNN, the supervised baseline, failed to compete with UKE
methods on Inspec. This behavior has also been observed in a recent work
Gallina et al. (2020). Although not being able to outperform existing methods
on all datasets, our proposed model still achieves the best weighted-average
results, outperforming the second-best by about 14% for top 3 predictions and
10% for top 5.

Absent keyphrase generation. For predicting absent keyphrases, our proposed
model outperforms existing work on all datasets. UKE methods cannot be
compared with our model, as they only extract present keyphrases. When
comparing with AutoKeyGen, we observe that our proposed model have
significantly better performance, except for the Inspec dataset where the
results are on par. On average, we outperform AutoKeyGen by nearly twice for
both top 5 and top 10 predictions.

####  3.3.2 Keyphrase generation for out-of-domain cases

One important objective of this work is the proposed model’s capability to
perform in out-of-domain settings. We show present and absent keyphrase
generation performance for out-of-domain datasets in table 3. For absent
keyphrase generation, we only report results on KPTimes, as DUC-2001 and
OpenKP mainly contain present keyphrases.

Present keyphrase generation. Our model achieves the best or second-best
results on all out-of-domain datasets. Similar to the in-domain cases, our
model achieves the best weighted-average results despite not being able to
outperform all baselines on all datasets. Of the two unsupervised keyphrase
generation methods, our proposed model achieves significantly better results
than AutoKeyGen in the out-of-domain setting.

Absent keyphrase generation.  In the out-of-domain setting. It can be seen
that AutoKeyGen fails to generate absent keyphrases, with the recall of only
$0.3\%$ for top 10 predictions. On the other hand, our model can recall 3.6%
of absent keyphrases. This improvement is significant considering that absent
keyphrase generation has been pointed out to be a “very challenging task” Meng
et al. (2017).

####  3.3.3 Comparison to supervised baseline

Although not being able to compete with the supervised baseline on the in-
domain datasets, our model has narrowed the gap between supervised and
unsupervised keyphrase generation methods. In addition, our model shows
remarkable performance on out-of-domain datasets, while the supervised
baseline shows poor generalization. It can be seen from Table 2 and 3 that the
performance of the supervised baseline plummets on out-of-domain datasets. On
the other hand, our model is able to retain performance across domains.

###  3.4 Ablation study

We perform an ablation study to further understand the role of the components
of our proposed model. In particular, we test our model with some components
removed, namely the adjustment weight defined in Equation 6, the references
and the part-of-speech information. We report the results in Table 4. For
KPTimes and OpenKP, we sample 200 and 500 documents from their original
validation and test set to perform the ablation study.

We observe that no model in the ablation study achieve the best performance in
all cases. However, the full version of our model shows to be more well-
rounded compared to its ablations.

Firstly, the adjustment weight $b(\boldsymbol{y})$ proves to be crucial, as
removing it cause our model’s performance to drop in most cases. This confirms
that the positional information is useful in predicting present keyphrases, as
has been pointed out by previous work Florescu and Caragea (2017b); Gallina et
al. (2020). Moreover, prioritizing phrases that exist in the phrase bank also
proves to be effective for predicting absent keyphrases. Next, removing the
references shows to heavily affect absent keyphrase generation, especially on
the out-of-domain dataset KPTimes. On the other hand, present keyphrase
generation seems not to be affected without using references. Finally, the
version of our model without part-of-speech information is able to maintain
present keyphrase generation performance for the in-domain dataset (Krapivin),
but slightly worsens when being moved out-of-domain. For absent keyphrase
generation, it seems that part-of-speech information does not help for
KPTimes. A possible explanation is that KPTimes mostly contains single-word
keyphrases and therefore grammatical information can offer little help in this
case.

Present keyphrase generation
---
| Krapivin | DUC-2001 | KPTimes | OpenKP
F1@5 | F1@5 | F1@5 | F1@5
No adjustment weight | 17.4 | 19.3 | 21.3 | 10
No references |  20.7 |  18.8 | 21.4 | 14.6
No POS | 21.1 | 17.6 | 21.5 | 13.1
Full | 20.5 | 18.2 | 21.8 |  14
Absent keyphrase generation
| Krapivin | DUC-2001 | KPTimes | OpenKP
R@10 | R@10 | R@10 | R@10
No adjustment weight | 5.8 | - | 2.5 | -
No references | 5.5 | - | 1 | -
No POS |  7.1 | - | 3.5 | -
Full | 7.2 | - |  3.2 | -

Table 4: Ablation study.

##  4 Case study

We display two examples of generated keyphrases from AutoKeyGen and our
proposed model in Figure 4. The first example is from Krapivin, an in-domain
dataset, while the second one is from KPTimes, an out-of-domain dataset. For
the first example, we observe that both the proposed model and AutoKeyGen
correctly predict the groundtruth (present and absent) keyphrases. However, it
can be seen that, for generating absent keyphrases, AutoKeyGen only reorders
words that are present in the given text. On the other hand, our model can
generate keyphrases whose component words are absent, such as “relational
models”, “categorical models” and “kripke models”.

In the second example, it is clear that our model predicts more correct
keyphrases. We observe that the absent keyphrases generated by AutoKeyGen are
highly irrelevant. On the other hand, our model successfully predicts “storms”
and also outputs other absent keyphrases that are relevant, although not being
within the ground truth keyphrases. This example help shows that our model is
better at handling documents from different domains.

##  5 Related work

###  5.1 Unsupervised keyphrase extraction

Unsupervised keyphrase extraction (UKE) aims at identifying keyphrases within
the text. Currently, there are three main classes of UKE methods, namely
statistics-based, graph-based and embedding-based. Statistics-based methods
Campos et al. (2018) employ features such as TF-IDF, word position and casing
aspect, to determine the relevance of a candidate phrase.

Graph-based methods typically build a graph from the source text, where a node
could be a word or a phrase. Then, different graph-theoretic measures are used
to estimate the importance of nodes, and finally phrases are formed based on
the top ranked nodes. TextRank Mihalcea and Tarau (2004) builds a word graph
where a link between two words exists if they co-occur within a window.
SingleRank Wan and Xiao (2008), CiteTextRank Gollapalli and Caragea (2014)
employs related documents to better measure similarity between word nodes.
TopicRank Bougouin et al. (2013), Topical PageRank Liu et al. (2010)
incorporate topical information in the graph ranking algorithm. Positional
information is used in PositionRank Florescu and Caragea (2017a) to favor
keyphrases that appear earlier in the text. Boudin (2018) utilizes the
structure of multi-partite graphs to extract diverse keyphrases.

Embedding-based methods utilize embedding spaces to measure informativeness of
candidates. EmbedRank Bennani-Smires et al. (2018) rank candidates by
measuring their distance to the source text in a pretrained sentence embedding
space, then an optional diversification step is performed using maximal-
marginal relevance to ensure diversity of extracted keyphrases. Liang et al.
(2021) jointly models local and global context of the document when ranking
candidates.

###  5.2 Unsupervised keyphrase generation

Keyphrase generation aims at predicting both present and absent keyphrases for
the source text. To our best knowledge, AutoKeyGen Shen et al. (2022) is
currently the only unsupervised keyphrase generation method. AutoKeyGen trains
a seq2seq model on automatically generated silver labeled document-keyphrase
pairs. The silver keyphrases are both present and absent, where present ones
are extracted, and the absent ones are constructed from the words present in
the text.

##  6 Conclusions

In this paper, we propose a new problem called unsupervised open-domain
keyphrase generation. We propose a seq2seq model that consists of two modules,
one is responsible for generating phrases while the other guides the
generation towards phrases that reflect the core concepts of the given text.
Our experiments on eight benchmark datasets from multiple domains demonstrate
that our model outperforms existing unsupervised methods and narrows the gap
between unsupervised and supervised keyphrase generation models. Furthermore,
we demonstrate that the proposed model can perform consistently across
domains.

## Limitations

One limitation of the proposed method is that it does not consider domain-
specific information to evaluate informativeness. The phraseness module has
access to domain-specific knowledge, which are the phrases that occur in
similar contexts, i.e. the references. On the other hand, the informativeness
module only employs a domain-general sentence embedding model to measure
informativeness of phrases. Therefore, the integration of both domain-specific
and domain-general information for the evaluation of informativeness may be
worth further investigation.

Another limitation of this work is that we only tested the proposed method on
short texts. Therefore, it is uncertain of the proposed framework’s
performance on long text documents. Handling long texts could be significantly
more difficult than short text, as long texts contain much more information
(can discuss a variety of topics).

The final limitation of this work is the absence of experiments on using
different sentence embedding models to construct the informativeness module.
Therefore, it might be useful to explore the impact of different sentence
embedding models on keyphrase generation performance. We leave this for future
work.

## References

  * Ahmad et al. (2021) Wasi Ahmad, Xiao Bai, Soomin Lee, and Kai-Wei Chang. 2021.  Select, extract and generate: Neural keyphrase generation with layer-wise coverage attention.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1389–1404, Online. Association for Computational Linguistics.
  * Bennani-Smires et al. (2018) Kamil Bennani-Smires, Claudiu Musat, Andreea Hossmann, Michael Baeriswyl, and Martin Jaggi. 2018.  Simple unsupervised keyphrase extraction using sentence embeddings.  In _Proceedings of the 22nd Conference on Computational Natural Language Learning_ , pages 221–229, Brussels, Belgium. Association for Computational Linguistics.
  * Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper. 2009.  _Natural language processing with Python: analyzing text with the natural language toolkit_.  " O’Reilly Media, Inc.".
  * Boudin (2018) Florian Boudin. 2018.  Unsupervised keyphrase extraction with multipartite graphs.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 667–672, New Orleans, Louisiana. Association for Computational Linguistics.
  * Bougouin et al. (2013) Adrien Bougouin, Florian Boudin, and Béatrice Daille. 2013.  Topicrank: Graph-based topic ranking for keyphrase extraction.  In _International joint conference on natural language processing (IJCNLP)_ , pages 543–551.
  * Campos et al. (2018) Ricardo Campos, Vítor Mangaravite, Arian Pasquali, Alípio Mário Jorge, Célia Nunes, and Adam Jatowt. 2018.  Yake! collection-independent automatic keyword extractor.  In _European Conference on Information Retrieval_ , pages 806–810. Springer.
  * Chen et al. (2018) Jun Chen, Xiaoming Zhang, Yu Wu, Zhao Yan, and Zhoujun Li. 2018.  Keyphrase generation with correlation constraints.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4057–4066, Brussels, Belgium. Association for Computational Linguistics.
  * Chen et al. (2019) Wang Chen, Yifan Gao, Jiani Zhang, Irwin King, and Michael R Lyu. 2019.  -guided encoding for keyphrase generation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 6268–6275.
  * Chuang et al. (2012) Jason Chuang, Christopher D Manning, and Jeffrey Heer. 2012.  “without the clutter of unimportant words” descriptive keyphrases for text visualization.  _ACM Transactions on Computer-Human Interaction (TOCHI)_ , 19(3):1–29.
  * Clement et al. (2019) Colin B. Clement, Matthew Bierbaum, Kevin P. O’Keeffe, and Alexander A. Alemi. 2019\.  On the use of arxiv as a dataset.
  * Florescu and Caragea (2017a) Corina Florescu and Cornelia Caragea. 2017a.  Positionrank: An unsupervised approach to keyphrase extraction from scholarly documents.  In _Proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers)_ , pages 1105–1115.
  * Florescu and Caragea (2017b) Corina Florescu and Cornelia Caragea. 2017b.  PositionRank: An unsupervised approach to keyphrase extraction from scholarly documents.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1105–1115, Vancouver, Canada. Association for Computational Linguistics.
  * Gallina et al. (2019) Ygor Gallina, Florian Boudin, and Beatrice Daille. 2019.  Kptimes: A large-scale dataset for keyphrase generation on news documents.  _arXiv preprint arXiv:1911.12559_.
  * Gallina et al. (2020) Ygor Gallina, Florian Boudin, and Béatrice Daille. 2020.  Large-scale evaluation of keyphrase extraction models.  In _Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020_ , pages 271–278.
  * Gollapalli and Caragea (2014) Sujatha Das Gollapalli and Cornelia Caragea. 2014.  Extracting keyphrases from research papers using citation networks.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 28.
  * Gu et al. (2016) Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016.  Incorporating copying mechanism in sequence-to-sequence learning.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1631–1640, Berlin, Germany. Association for Computational Linguistics.
  * Gutwin et al. (1999) Carl Gutwin, Gordon Paynter, Ian Witten, Craig Nevill-Manning, and Eibe Frank. 1999\.  Improving browsing in digital libraries with keyphrase indexes.  _Decision Support Systems_ , 27(1-2):81–104.
  * Hinton (2002) Geoffrey E Hinton. 2002.  Training products of experts by minimizing contrastive divergence.  _Neural computation_ , 14(8):1771–1800.
  * Hulth (2003) Anette Hulth. 2003.  Improved automatic keyword extraction given more linguistic knowledge.  In _Proceedings of the 2003 conference on Empirical methods in natural language processing_ , pages 216–223.
  * Johnson et al. (2019) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019.  Billion-scale similarity search with GPUs.  _IEEE Transactions on Big Data_ , 7(3):535–547.
  * Kim et al. (2013) Su Nam Kim, Olena Medelyan, Min-Yen Kan, and Timothy Baldwin. 2013.  Automatic keyphrase extraction from scientific articles.  _Language resources and evaluation_ , 47(3):723–742.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Krapivin et al. (2009) Mikalai Krapivin, Aliaksandr Autaeu, and Maurizio Marchese. 2009.  Large dataset for keyphrases extraction.
  * Liang et al. (2021) Xinnian Liang, Shuangzhi Wu, Mu Li, and Zhoujun Li. 2021.  Unsupervised keyphrase extraction by jointly modeling local and global context.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 155–164, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Litvak and Last (2008) Marina Litvak and Mark Last. 2008.  Graph-based keyword extraction for single-document summarization.  In _Coling 2008: Proceedings of the workshop Multi-source Multilingual Information Extraction and Summarization_ , pages 17–24, Manchester, UK. Coling 2008 Organizing Committee.
  * Liu et al. (2010) Zhiyuan Liu, Wenyi Huang, Yabin Zheng, and Maosong Sun. 2010.  Automatic keyphrase extraction via topic decomposition.  In _Proceedings of the 2010 conference on empirical methods in natural language processing_ , pages 366–376.
  * Meng et al. (2017) Rui Meng, Sanqiang Zhao, Shuguang Han, Daqing He, Peter Brusilovsky, and Yu Chi. 2017.  Deep keyphrase generation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 582–592, Vancouver, Canada. Association for Computational Linguistics.
  * Mihalcea and Tarau (2004) Rada Mihalcea and Paul Tarau. 2004.  TextRank: Bringing order into text.  In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_ , pages 404–411, Barcelona, Spain. Association for Computational Linguistics.
  * Nguyen and Kan (2007) Thuy Dung Nguyen and Min-Yen Kan. 2007.  Keyphrase extraction in scientific publications.  In _International conference on Asian digital libraries_ , pages 317–326. Springer.
  * Pagliardini et al. (2018) Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. 2018.  Unsupervised learning of sentence embeddings using compositional n-gram features.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 528–540, New Orleans, Louisiana. Association for Computational Linguistics.
  * Porter (1980) Martin F Porter. 1980.  An algorithm for suffix stripping.  _Program_.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017.  Get to the point: Summarization with pointer-generator networks.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics.
  * Shen et al. (2022) Xianjie Shen, Yinghan Wang, Rui Meng, and Jingbo Shang. 2022.  Unsupervised deep keyphrase generation.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 36(10):11303–11311.
  * Sun et al. (2019) Zhiqing Sun, Jian Tang, Pan Du, Zhi-Hong Deng, and Jian-Yun Nie. 2019.  Divgraphpointer: A graph pointer network for extracting diverse keyphrases.  In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 755–764.
  * Tomokiyo and Hurst (2003) Takashi Tomokiyo and Matthew Hurst. 2003.  A language model approach to keyphrase extraction.  In _Proceedings of the ACL 2003 Workshop on Multiword Expressions: Analysis, Acquisition and Treatment_ , pages 33–40, Sapporo, Japan. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wan and Xiao (2008) Xiaojun Wan and Jianguo Xiao. 2008.  Single document keyphrase extraction using neighborhood knowledge.  In _AAAI_ , volume 8, pages 855–860.
  * Witten et al. (2009) Ian H Witten, David Bainbridge, and David M Nichols. 2009.  _How to build a digital library_.  Morgan Kaufmann.
  * Xiong et al. (2019) Lee Xiong, Chuan Hu, Chenyan Xiong, Daniel Campos, and Arnold Overwijk. 2019.  Open domain web keyphrase extraction beyond language modeling.  _arXiv preprint arXiv:1911.02671_.
  * Ye et al. (2021) Jiacheng Ye, Ruijian Cai, Tao Gui, and Qi Zhang. 2021.  Heterogeneous graph neural networks for keyphrase generation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 2705–2715, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Yuan et al. (2020) Xingdi Yuan, Tong Wang, Rui Meng, Khushboo Thaker, Peter Brusilovsky, Daqing He, and Adam Trischler. 2020.  One size does not fit all: Generating and evaluating variable number of keyphrases.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7961–7975, Online. Association for Computational Linguistics.
  * Zhou and Rush (2019) Jiawei Zhou and Alexander Rush. 2019.  Simple unsupervised summarization by contextual matching.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5101–5106, Florence, Italy. Association for Computational Linguistics.

##  Appendix A Implementation details

###  A.1 Phraseness module

Retriever.  We extract noun phrases for the documents in the training set of
KP20K, StackExchange and KPTimes; and form the phrase bank by keeping the noun
phrases that occur in at least 5 documents. For obtaining embeddings of
documents, we employ the Sent2Vec pretrained model named
sent2vec_wiki_unigrams666https://github.com/epfml/sent2vec, a 600-dimensional
sentence embedding model trained on English Wikipedia. We utilize
Faiss777https://github.com/facebookresearch/faiss Johnson et al. (2019) for
indexing the phrases and their context embeddings.

Seq2seq model. Both the encoder and decoder of the seq2seq model contains 3
layers. The model dimensionality and the word embedding size are both set to
256, and the part-of-speech embedding size is set to 64. The seq2seq model
employs attention with 8 heads. The encoder and decoder have separate
vocabularies, both contain 40000 words. The encoder and decoder vocabulary
contains the frequent words in the unlabeled corpus and among the extracted
noun phrases, respectively. The seq2seq model is optimized using Adam
optimizer Kingma and Ba (2014), with a learning rate of 0.0001, gradient
clipping = 0.1 and a dropout rate of 0.1. We trained our model in 15 epochs.
After every 3 training epoch, the learning rate is reduced by $10\%$. The
seq2seq model contains 34M trainable parameters, and training it for 15 epochs
took about 7 hours on a single NVIDIA A40 GPU. We roughly estimate that
conducting our experiments, which include training the baseline models, took a
total of 150 GPU hours.

###  A.2 Informativeness module

For the informativeness module, we also employ the pretrained Sent2vec model
sent2vec_wiki_unigrams, to obtain embeddings of words and texts.

###  A.3 Keyphrase decoding

We employ beam search with beam size = 100 and beam depth = 6. The balancing
hyperparameter $\lambda$ is set to 0.75. Considering the adjustment weight in
Equation 6, we set $\beta=5/6$ to favor existing phrases in the phrase bank
$B$. For each text, we retrieve 15 references, some of which can be filtered
out by the threshold $\tau$, which is set to 0.7.

We use the validation set of each testing dataset to select the value for the
length penalty factor $\alpha$. In particular, we choose $\alpha\in$ $\\{$-1,
-0.75, -0.5, -0.25, 0, 0.25, 0.5, 0.75, 1$\\}$ that maximizes the geometric
mean of the evaluation metrics, i.e. F1 at top 3 and 5 for present keyphrases
and Recall at top 5 and 10 for absent keyphrases. Since the value range of
these metrics are different from one another, we divide each metric by its
maximum value (that can be found as we try different values of $\alpha$) for
normalization before taking the geometric mean. Table 5 provide the best
$\alpha$ values for each dataset in each run in our experiment.

| 1st run | 2nd run | 3rd run | 4th run | 5th run
---|---|---|---|---|---
SemEval | -1 | -1 | -0.5 | -1 | -1
Inspec | -1 | -1 | -1 | -0.75 | -1
NUS | -0.75 | -0.75 | -0.25 | -0.25 | -0.5
Krapivin | -0.25 | -0.5 | -0.25 | -0.25 | -0.5
StackExchange | 1 | 1 | 1 | 1 | 1
DUC-2001 | -1 | -1 | -1 | -1 | -1
KPTimes | 0.5 | 0.5 | 0.75 | 0.5 | 1
OpenKP | 0.5 | -0.25 | 0.25 | -0.25 | -0.25

Table 5: Best length penalty values for each dataset in each run.