# Semi-supervised Relation Extraction via
Data Augmentation and Consistency-training

Komal K. Teru
The Vanguard Group
<EMAIL_ADDRESS>

###### Abstract

Due to the semantic complexity of the Relation extraction (RE) task, obtaining
high-quality human labelled data is an expensive and noisy process. To improve
the sample efficiency of the models, semi-supervised learning (SSL) methods
aim to leverage unlabelled data in addition to learning from limited labelled
data points. Recently, strong data augmentation combined with consistency-
based semi-supervised learning methods have advanced the state of the art in
several SSL tasks. However, adapting these methods to the RE task has been
challenging due to the difficulty of data augmentation for RE. In this work,
we leverage the recent advances in controlled text generation to perform high
quality data augmentation for the RE task. We further introduce small but
significant changes to model architecture that allows for generation of more
training data by interpolating different data points in their latent space.
These data augmentations along with consistency training result in very
competitive results for semi-supervised relation extraction on four benchmark
datasets.

##  1 Introduction

Relation extraction is one of the essential components in constructing
structured knowledge bases Luan et al. (2018), performing interpretable
question answering Sun et al. (2021), improving web search, and many other
information extraction pipelines. It aims to discover the semantic relation
between a given head entity and tail entity based on the context in the input
sentence. For example, given a sentence "The battle led to panic on the
frontier, and settlers in the surrounding counties fled.", the goal is to
extract the Cause-Effect relation between the head entity ‘battle’ and the
tail entity ‘panic’. The RE task requires a high level of language
understanding and involves a significant level of semantic complexity Bach and
Badaskar (2007). Due to this semantic complexity it often requires extensive
and highly skilled human involvement to obtain good quality labelled data,
making data collection an expensive and noisy process. Unsurprisingly, because
of the same semantic complexity of the task, models typically require large
amounts of labelled data to give production-ready performance.

Input sentence: The battle led to panic on the frontier, and settlers in the
surrounding counties fled. Synonym-replacement: The struggle cause scare on
the frontier, and settlers in the surrounding counties fly. LM-based
augmentation: The battle reduced panic on the frontier, and settlers in the
surrounding counties relaxed. Vanilla BT: The war caused panic at the border,
and residents of the nearby counties fled. Our constrained BT: The battle
sparked panic at the border, with residents fleeing in surrounding counties.

Figure 1: Different data augmentation techniques applied to a sample datapoint
from SemEval dataset. Existing methods replace the head/tail entities
(highlited in red), change the original meaning or do not give very fluent
paraphrases.

A common strategy to improve the sample efficiency of machine learning models
is semi-supervised learning methods which leverage easily accessible
unlabelled data to improve the overall performance. While there are several
paradigms of semi-supervised learning methods, consistency training based
methods have advanced the state of the art in several SSL tasks Ghosh and
Thiery (2021). These methods can typically reach performances that are
comparable to their fully supervised counterparts while using only a fraction
of labelled data points. Recently, strong data augmentation combined with
consistency training algorithms have shown great success, even surpassing
fully supervised models, in low-data settings of various tasks Xie et al.
(2020). Adapting these methods to the task of relation extraction has been
challenging due to the difficulty of data augmentation for RE task. This is
because, in addition to the input sentence, each data point also consists of a
head entity and a tail entity contained in the input sentence. Typical data
augmentation techniques used in NLP such as back-translation, synonym-
replacement, language-model based augmentation, etc. Feng et al. (2021) can
not be easily applied to such ‘structured’ input as they do not guarantee the
integrity of either a) the entities in the input sentence or, b) the meaning
of the input sentence itself. Figure 1 shows that using synonym-replacement
and vanilla back-translation (BT) methods Sugiyama and Yoshinaga (2019) the
entities themselves could be paraphrased or replaced. Matching the new and the
old entities is a whole problem in itself. In the Language Model-based
augmentation method Anaby-Tavor et al. (2020), the semantic meaning of the
input sentence changes altogether, which makes it difficult to employ
consistency training.

Present work. In this work, we leverage the recent advances in controlled text
generation to perform high quality data augmentation for the relation
extraction task that not only keeps the meaning and the head/tail entities
intact but also produces fluent and diverse data points. In particular, we
modify back-translation to leverage lexically constrained decoding strategies
Post and Vilar (2018); Hu et al. (2019) in order to obtain paraphrased
sentences while retaining the head and the tail entities. We further propose
novel modifications to the widely popular relation extraction model
architecture, that allows for generation of more samples by interpolating
different data points in their latent space, a trick that has been very
successful in other domains and tasks Berthelot et al. (2019); Chen et al.
(2020b, a). Additionally, we leverage the entity types of the head and the
tail entities, when available, in a way that effectively exploits the
knowledge embedded in pre-trained language models. These data augmentations,
when applied to unlabelled data, let us employ consistency training techniques
to achieve very competitive results for semi-supervised relation extraction on
four benchmark datasets. To the best of our knowledge, this is the first study
to apply and show the merit of data augmentation and consistency training for
semi-supervised relation extraction task.

##  2 Related work

### Semi-supervised learning for NLP

Semi-supervised learning algorithms can be categorized into two broad
classes–1) self-ensembling methods and 2) self-training methods.

Self-ensembling methods leverage the smoothness and cluster/low-density
assumptions of the latent space Chapelle and Zien (2005). They train the
models to make consistent predictions under various kinds of perturbations to
either a) the data Miyato et al. (2019); Xie et al. (2020), or b) the model
parameters themselves Tarvainen and Valpola (2017). The former methods are
broadly referred to as consistency training methods and have resulted in
state-of-the-art performances for several semi-supervised NLP tasks. Sachan et
al. (2019) add adversarial noise to both labelled and unlabelled data and
train models to make consistent predictions on the original and the
corresponding noisy data-point. Many recent methods leverage large pre-trained
language models for more advanced data augmentation techniques, like back-
translation Edunov et al. (2018), and further improve performance in the low-
data regime Xie et al. (2020). Recently, Chen et al. (2020b, a) adapted the
Mixup algorithm Zhang et al. (2018) as another form of data augmentation for
textual data and show state-of-the-art performance on text classification and
NER tasks. Due to the difficulty of data augmentation for relation extraction
task (Figure 1), these methods have not been adapted for semi-supervised
relation extraction (SSRE) task so far. In this work, we fill that gap and
demonstrate the empirical success of consistency training for SSRE.

Self-training methods are the oldest heuristic methods of iteratively
expanding the labelled training set by including high-confidence pseudo-labels
from the unlabelled data. All of the existing works on SSRE fall under this
paradigm. These methods famously suffer from the confirmation bias problem
where the incorrect predictions of the initially trained model affect the
quality of pseudo-labels and eventually cause the label distribution to drift
away from the true data distribution, resulting in a semantic drift. Lin et
al. (2019) was one of the first works to address this by training two
different models and augmenting the labelled set with the ‘consensus’ set,
i.e., the data points which are labelled the same by both models. Several
works have developed on this idea of improving the pseudo-label quality via
various strategies like meta-learning Hu et al. (2021a) or reinforcement
learning Hu et al. (2021b). These set of methods constitute our baselines.

### Data augmentation for NLP

In this work, we concentrate on two major classes of data augmentation
techniques for NLP – sentence-level data augmentation and latent space
augmentations. Sentence-level data augmentation techniques include back-
translation Edunov et al. (2018), language-model based augmentations Anaby-
Tavor et al. (2020), and word-replacement strategies Zhang et al. (2015). We
adapt the back-translation techniques to the RE task.

In latent space augmentations one generates more samples by interpolating
between pairs of given data points in their latent space. This was originally
introduced for image classification Zhang et al. (2018); Verma et al. (2019);
Yun et al. (2019) as a data augmentation and regularization method. Previous
works have generalized this idea to the textual domain by proposing to
interpolate in embedding space Cheng et al. (2020) or the general latent space
Chen et al. (2020b, a) of textual data and applied the technique to NLP tasks
such as text classification, machine translation, NER task and achieved
significant improvements. We show that both these styles of augmentations can
be effectively applied to improve performance on SSRE task.

##  3 Background

Task formulation. In this work, we focus on the sentence-level relation
extraction task, i.e., given a relation statement
$\mathbf{x}:(\mathbf{s},e_{h},e_{t})$ consisting of a sentence, $\mathbf{s}$,
a head entity, $e_{h}$, and a tail entity, $e_{t}$ (both the entities are
mentioned in the given sentence $\mathbf{s}$), the goal is to predict a
relation $r\in\mathcal{R}\cup\\{\text{NA}\\}$ between the head and the tail
entity, where $\mathcal{R}$ is a pre-defined set of relations. If the sentence
does not express any relation from the set $\mathcal{R}$ between the two
entities, then the relation statement $\mathbf{x}$ is accordingly labelled NA.

This is typically done by learning a relation encoder model
$\mathcal{F}_{\theta}:\mathbf{x}\mapsto\mathbf{h}_{r}$ that maps an input
relation statement, $\mathbf{x}$, to a fixed length vector $\mathbf{h}_{r}$
that represents the relation expressed in $\mathbf{s}$ between $e_{h}$ and
$e_{t}$. This relation representation, $\mathbf{h}_{r}$, is then classified to
a relation $r\in\mathcal{R}\cup\\{\text{NA}\\}$ via an MLP classifier.

Base model architecture. Most recent methods for RE use a Transformer-based
architecture Devlin et al. (2019); Vaswani et al. (2017) for the relation
encoder model, $\mathcal{F}_{\theta}$. To represent the head and tail entities
in the input to the encoder, the widely accepted strategy is to augment the
input sentence $\mathbf{s}$ with entity marker tokens–[E1], [/E1], [E2],
[/E2]–to mark the start and end of both entities. Concretely, an input
sentence like "Lebron James currently plays for LA Lakers team." when
augmented with entity marker tokens becomes

$\text{{[E1]} {Lebron James} {[/E1]} currently}\\\ \text{plays for {[E2]} {LA
Lakers} {[/E2]} team.}$

This modified text is input to the Transformer-based sequence encoder. Next,
the encoder output representations111hidden state from the last layer of the
Transformer model of the tokens [E1] and [E2] are concatenated to give the
fixed length relation representation,
$\mathbf{h}_{r}=[\mathbf{h}_{[E1]}\oplus\mathbf{h}_{[E2]}]$. This fixed length
vector is in turn passed through an MLP classifier,
$p_{\phi}(\mathbf{h}_{r})$, to give a probability vector, $\mathbf{y}$, over
the relation set $\mathcal{R}\cup\\{\text{NA}\\}$.

##  4 Proposed approach

In our approach we build on the base model architecture described in §3 and
introduce additional model design elements that are necessary to obtain an
improved performance in semi-supervised relation extraction (SSRE) task.

We first describe the two data augmentation techniques we perform, and the
model architectural changes we introduce that facilitate these augmentations.
Then, we describe the training procedure we follow to leverage unlabelled data
and achieve state-of-the-art performance on three out of four benchmark
datasets for SSRE.

###  4.1 Constrained back-translation

Back-translation Edunov et al. (2018) generates diverse and fluent
augmentations while retaining the global semantics of the original input.
Specifically, one translates a given text into an intermediate language, say,
German, and translates it back to the source language, say English. Using
different intermediate languages and temperature-based sampling results in a
diverse set of paraphrases. Applying this back-translation technique in a
vanilla fashion is not possible for RE task because one has little control
over the retention of the head and tail entities (Figure 1). Thus, when
translating back to the source language from the intermediate language we
perform lexically-constrained decoding Hu et al. (2019), i.e., force the
inclusion of pre-specified words and phrases–positive constraint set–in the
output. In our case the original head and tail entity words/phrases make up
this positive constraints set. We use German and Russian as intermediate
languages and use the pre-trained WMT’19 English-German and English-Russian
translation models (in both directions) and their implementations provided by
Ott et al. (2019). This methodology generates diverse data augmentations for a
given sentence. For example, the sentence "The battle led to panic on the
frontier, and settlers in the surrounding counties fled." is converted to "The
battle sparked panic at the border, with residents fleeing in surrounding
counties" when back-translated via German, and to "The battle caused panic on
the border and settlers in nearby counties fled." when done via Russian. This
is illustrated in Figure 2. This strong data-augmentation technique for RE can
be applied to both labelled and unlabelled data opening the doors to
consistency training Xie et al. (2020) as we will see in §4.3.

Figure 2: Constrained back-translation process

###  4.2 Latent-space interpolation

Here, we adapt a mixup-based data augmentation technique to the RE task by
making necessary modifications to the base model architecture we described in
§3. As done in previous works Chen et al. (2020b, a), we sample two random
data points–$(\mathbf{x},\mathbf{y})$ and
$(\mathbf{x}^{\prime},\mathbf{y}^{\prime})$, where $\mathbf{x}$ and
$\mathbf{y}$ denote the relation statement and the corresponding relation
label–from the training data and separately compute the respective latent
representations, $\mathbf{h}^{m}$ and $\mathbf{h}^{\prime m}$, upto the layer
$m$ of the relation encoder $\mathcal{F}_{\theta}$ as follows:

$\displaystyle\mathbf{h}^{l}$
$\displaystyle=\mathcal{F}_{\theta}^{l}(\mathbf{h}^{l-1});\quad l\in[1,m],$
$\displaystyle\mathbf{h}^{\prime l}$
$\displaystyle=\mathcal{F}_{\theta}^{l}(\mathbf{h}^{\prime{l-1}});\quad
l\in[1,m],$

where $\mathbf{h}^{l}$ is the latent representation of all tokens in the
sentence $\mathbf{x}$ at the $l^{\text{th}}$ layer of the encoder. Next, the
latent representations of each token in $\mathbf{x}$ at the $m^{\text{th}}$
layer are linearly interpolated:

$\tilde{\mathbf{h}}^{m}=\lambda\mathbf{h}^{m}+(1-\lambda)\mathbf{h}^{\prime
m},$

where $\lambda$ is the mixing coefficient which is sampled from a Beta
distribution, i.e., $\lambda\sim\text{Beta}(\alpha,\beta)$. Then, the
interpolated latent representation is passed through the rest of the encoder
layers:

$\tilde{\mathbf{h}}^{l}=\mathcal{F}_{\theta}^{l}(\tilde{\mathbf{h}}^{l-1});\quad
l\in[m+1,L].$

This final encoder output representation, $\tilde{\mathbf{h}}^{L}$, can be
interpreted as the encoder output representation of a virtual input
$\tilde{\mathbf{x}}$, i.e.,
$\tilde{\mathbf{h}}^{L}=\mathcal{F}_{\theta}(\tilde{\mathbf{x}})$. We denote
this whole mixup operation222REMix entails the model architecture changes
discussed below. as
$\tilde{\mathbf{h}}^{L}:=\text{REMix}(\mathbf{x},\mathbf{x}^{\prime})$. The
label for this augmented virtual sample is given by the linear interpolation
of the respective labels, $\mathbf{y}$ and $\mathbf{y}^{\prime}$, with the
same mixing coefficient $\lambda$ i.e.,
$\tilde{\mathbf{y}}:=\text{mix}(\mathbf{y},\mathbf{y}^{\prime})=\lambda\mathbf{y}+(1-\lambda)\mathbf{y}^{\prime}$.
This virtual data point, $(\tilde{\mathbf{x}},\tilde{\mathbf{y}})$, is the
augmented data point and can be used as additional training data.

Proposed architecture change. Now, for the RE task we need to extract a fixed-
length relation representation from the encoder output representation of the
entire input sequence. The traditional way to do this for RE task is by
concatenating the encoder output representations of the entity marker tokens
[E1] and [E2]. However, it is challenging to do this for a virtual sample,
$\tilde{\mathbf{x}}$, as the entity markers are not clearly defined in this
case. We thus modify the relation representation to be the encoder output
representation of the [CLS] token. However, Baldini Soares et al. (2019) have
shown this choice to be sub-optimal compared to concatenation of marker
tokens. This is because the marker token representations provide direct access
to the contextual information of the respective entities. Although the [CLS]
token, in theory, has access to the entire context of the sentence, it might
be difficult to capture the nuances like the head entity type, tail entity
type, and the contextual information around the two entities all in a single
vector.

On the other hand, entity type information is easily accessible in most RE
benchmarks333From new datasets/applications viewpoint, when entities are
identified in a piece of text it is safe to assume that their types would also
be identified.. So, to compensate for the sub-optimal choice of using [CLS]
token representation as the relation representation, we modify how we
represent the entity spans in the input token to more effectively use the
easily accessible entity type information. In particular, we note that the
entity type labels can trivially be mapped to tokens from any pre-trained
language model’s vocabulary. For example, entity types like PERSON and
STATE_OR_PROVINCE can be tokenized into a word/phrase like ‘person’ and ‘state
or province’, respectively. In such cases when entity type information is
available, instead of using special marker tokens like [E1] and [E2] we
prepend the entity spans in the input sequence with the word/phrases
corresponding to their respective types and enclose these ‘type-words’ in
punctuation marks Zhou and Chen (2021). The modified input to the transformer
along with the [CLS] token looks as follows:

$\texttt{[CLS]}\text{@ * person * {Lebron James} @ plays}\\\ \text{for \& *
organization * {LA Lakers} \& team.}$

We use different punctuation symbols to distinguish between subject and object
entities. Specifically, we use ‘@’ for subject and ‘&’ for object entities.
This representation helps leverage the knowledge already contained in the pre-
trained large-language model about the type of the entity and offset some of
the downside of using a simplified relation representation in the [CLS] token.
As we will empirically see in §5.4, this use of entity type information is not
only effective but is necessary for the optimal functioning of our approach.
Zhou and Chen (2021) recently showed the success of this method in the fully
supervised setting. Here we use it in conjunction with a simplified relation
representation and show its merit in semi-supervised RE setting.

###  4.3 Consistency training for SSRE

Let the given limited labelled set be
$\mathbf{X}_{l}=\\{\mathbf{x}_{1}^{l},...,\mathbf{x}_{n}^{l}\\}$, with their
relation labels
$\mathbf{Y}_{l}=\\{\mathbf{y}_{1}^{l},...,\mathbf{y}_{n}^{l}\\}$, where
$\mathbf{y}_{i}^{l}\in\\{0,1\\}^{|\mathcal{R}\cup\\{\text{NA}\\}|}$ is a one-
hot vector and $\mathcal{R}$ is the set of pre-defined relations. Let
$\mathbf{X}_{u}=\\{\mathbf{x}_{1}^{u},...,\mathbf{x}_{m}^{u}\\}$ be a large
unlabelled set. The goal is to apply both the data augmentation techniques
described above and train a model with consistency loss to effectively
leverage unlabelled data along with the limited labelled data.

We largely adapt the semi-supervised training techniques introduced by Chen et
al. (2020b). For each $\mathbf{x}_{i}^{u}$ in the unlabelled set
$\mathbf{X}_{u}$, we generate $K$ augmentations
$\mathbf{x}_{i,k}^{a},k\in\\{1,2,...,K\\}$ using the constrained back
translation technique with different intermediate languages444In our specific
implementation K = 2; with German and Russian as intermediate languages..
These augmentations make up the set
$\mathbf{X}_{a}=\\{\mathbf{x}_{i,k}^{a}\\}$. For a given unlabelled data point
$\mathbf{x}_{i}^{u}$ and its $K$ augmentations $\mathbf{x}_{i,k}^{a}$ the
label is given by the average of current model’s predictions on all $K+1$ data
points:

$\mathbf{y}_{i}^{u}=\frac{1}{K+1}\left(p_{\phi}(\mathcal{F}_{\theta}(\mathbf{x}_{i}^{u}))+\sum_{k=1}^{K}p_{\phi}(\mathcal{F}_{\theta}(\mathbf{x}_{i,k}^{a}))\right),$

where $\mathbf{y}_{i}^{u}$ is a probability vector. This not only enforces the
constraint that the model should make consistent predictions for different
augmentations but also makes the predictions more robust by ensembling all the
predictions. We merge the unlabelled set and the augmented set into
$\mathbf{X}_{\text{ua}}=\mathbf{X}_{u}\cup\mathbf{X}_{a}$ and the
corresponding (pseudo-)labels are given by
$\mathbf{Y}_{\text{ua}}=\mathbf{Y}_{u}\cup\mathbf{Y}_{a}$, where
$\mathbf{Y}_{u}=\\{\mathbf{y}_{i}^{u}\\}$,
$\mathbf{Y}_{a}=\\{\mathbf{y}_{i,k}^{a}\\}$, and
$\mathbf{y}_{i,k}^{a}=\mathbf{y}_{i}^{u}\forall k\in\\{1,2,...,K\\}$, i.e.,
all the augmented data points share the same label as the original unlabelled
data point.

Given this cumulative set $\mathbf{X}_{\text{ua}}$ and their generated labels
$\mathbf{Y}_{\text{ua}}$ as additional training data, we employ the REMix
augmentation technique to generate arbitrary amounts of training data. In
particular, we randomly sample two data points
$\mathbf{x}_{s}^{\text{ua}},\mathbf{x}_{t}^{\text{ua}}\in\mathbf{X}_{\text{ua}}$,
and compute the encoder output representation of a new virtual data point with
$\text{REMix}(\mathbf{x}_{s}^{\text{ua}},\mathbf{x}_{t}^{\text{ua}})$ and the
corresponding target label with
$\text{mix}(\mathbf{y}_{s}^{\text{ua}},\mathbf{y}_{t}^{\text{ua}})$.

Additionally, while computing the final unsupervised loss in each training
iteration we filter out the unlabelled data points with prediction confidence
below a certain threshold $\gamma$ Xie et al. (2020). Finally, to encourage
low-entropy predictions on unlabelled data, we sharpen the predictions with a
sharpening coefficient $T$:

$\hat{\mathbf{y}}_{i}^{\text{ua}}=\frac{(\mathbf{y}_{i}^{\text{ua}})^{\frac{1}{T}}}{||(\mathbf{y}_{i}^{\text{ua}})^{\frac{1}{T}}||_{1}}.$

Everything put together, the final unsupervised loss in each training
iteration with mini-batch size $B$ is computed as:

$\displaystyle\mathcal{L}_{\text{unsp}}=\frac{1}{B}\sum_{\mathbf{x}_{s}^{\text{ua}},\mathbf{x}_{t}^{\text{ua}}\sim\mathbf{X}_{\text{ua}}}^{B}\texttt{m}(\mathbf{x}_{s}^{\text{ua}},\mathbf{x}_{t}^{\text{ua}})\mathcal{L}_{\text{mix}}(\mathbf{x}_{s}^{\text{ua}},\mathbf{x}_{t}^{\text{ua}}),$

where

$\displaystyle\begin{split}\mathcal{L}_{\text{mix}}(\mathbf{x}_{s}^{\text{ua}},\mathbf{x}_{t}^{\text{ua}})={}&\text{CE}(\text{mix}(\hat{\mathbf{y}}_{s}^{\text{ua}},\hat{\mathbf{y}}_{t}^{\text{ua}})||\\\
&\qquad
p_{\phi}(\text{REMix}(\mathbf{x}_{s}^{\text{ua}},\mathbf{x}_{t}^{\text{ua}}))),\end{split}$
$\displaystyle\texttt{m}(\mathbf{x}_{s}^{\text{ua}},\mathbf{x}_{t}^{\text{ua}})={}$
$\displaystyle
I(\max\mathbf{y}_{s}^{\text{ua}}>\gamma)I(\max\mathbf{y}_{t}^{\text{ua}}>\gamma).$

Here, $I(.)$ is an indicator function and $\texttt{m}(.)$ denotes the
confidence masking function which filters out the low-confidence datapoints.
In our implementation $p_{\phi}(.)$ is a two-layer MLP classifier on top of
the relation encoder model. CE denotes the cross entropy loss function.
555Note that we only apply the augmentation techniques on the unlabelled data
set. Initial experiments applying these to the labelled data set resulted in
only marginal improvements and even performance deterioration in some cases,
likely due to introduction of too much noise into an already limited labelled
set.

This combined with the traditional supervised loss,
$\mathcal{L}_{\text{sup}}=\sum_{\mathbf{x}_{i}\sim\mathbf{X}_{l}}^{B}\text{CE}(\mathbf{y}_{i}^{l}||p_{\phi}(\mathcal{F}_{\theta}(\mathbf{x}_{i})))$,
constitutes the total loss:

$\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{sup}}+\gamma_{m}\mathcal{L}_{\text{unsp}},$

where $\gamma_{m}$ is a parameter which controls the trade-off between
supervised and unsupervised loss.

##  5 Experiments

We perform experiments on four benchmark datasets for sentence-level RE and
compare the proposed model, REMix, against current state-of-the-art SSRE
approaches. We further conduct ablation studies and sensitivity analysis to
expose the significance of different design choices of REMix.

###  5.1 Datasets

We use two widely popular relation extraction benchmark datasets: SemEval 2010
Task 8 (SemEval) Hendrickx et al. (2010), and the TAC Relation Extraction
Dataset (TACRED) Zhang et al. (2017). SemEval is a standard benchmark dataset
for evaluating relation extraction models containing 10717 examples in total.
Each sentence is annotated with a pair of untyped nominals (concepts; example
in Figure 1) that are related via one of 19 semantic relation types (including
no_relation). TACRED is a large-scale crowd-sourced relation extraction
dataset with 106264 examples which is collected from all the prior TAC KBP
relation schema. Unlike SemEval, sentences in TACRED are labelled with pairs
of typed-entities that are related via one of 42 person- and organization-
oriented relation types (including no_relation). In addition to these standard
benchmark datasets, we also show results on two more datasets: RE-TACRED
Stoica et al. (2021) and KBP37 Zhang and Wang (2015). RE-TACRED is a re-
annotated version of the original TACRED dataset using an improved annotation
strategy to ensure high-quality labels. Zhou and Chen (2021) provide a
compelling analysis and recommend using this as the evaluation benchmark for
sentence-level RE. KBP37 is another sentence-level RE dataset with 21046 total
examples collected from 2010 and 2013 KBP documents as well as July 2013 dump
of Wikipedia. In terms of size, this falls between SemEval and TACRED. Similar
to SemEval the entity types are not available in this dataset, however the 37
relation types are person- and organization-oriented like in TACRED. This
dataset is thus a good segue between the two standard benchmarks. The
statistics of these datasets is given in Table 1.

Table 1: Dataset statistics Dataset | # rel. | examples | # no_relation
---|---|---|---
TACRED | 42 | 106264 | 79.51%
RE-TACRED | 40 | 91467 | 63.17%
KBP37 | 37 | 21046 | 10.33%
SemEval | 19 | 10717 | 17.39%
Table 2: F1 score with various amounts of labelled data and 50% unlabelled data. Mean and standard deviation of 5 different runs is reported. Best performance on each configuration is bolded and second best is underlined. | TACRED | KBP37
---|---|---
%labelled Data | 3% | 10% | 15% | 5% | 10% | 30%
MRefG | 43.81$\pm$ 1.44 | 55.42$\pm$ 1.40 | 58.21$\pm$ 0.71 | - | - | -
MetaSRE | 46.16$\pm$ 0.74 | 56.95$\pm$ 0.33 | 58.94$\pm$ 0.31 | 59.29$\pm$ 0.92 | 61.83$\pm$ 0.21 | 63.51$\pm$ 0.69
GradLRE |  47.37$\pm$ 0.74 |  58.20$\pm$ 0.33 |  59.93$\pm$ 0.31 |  59.98$\pm$ 0.37 |  62.67$\pm$ 0.54 |  66.41$\pm$ 0.28
REMix(ours) |  55.80$\pm$ 1.33 |  61.30$\pm$ 0.70 |  63.07$\pm$ 0.93 |  60.84$\pm$ 0.40 |  63.82$\pm$ 0.71 |  66.46$\pm$ 0.69
| RE-TACRED | SemEval
%labelled Data | 3% | 10% | 15% | 5% | 10% | 30%
MRefG | - | - | - | 75.48$\pm$ 1.34 | 77.96$\pm$ 0.90 | 83.24$\pm$ 0.71
MetaSRE | 44.42$\pm$ 3.02 | 58.71$\pm$ 1.70 | 61.71$\pm$ 3.70 |  78.33$\pm$ 0.92 | 80.09$\pm$ 0.78 | 84.81$\pm$ 0.44
GradLRE |  61.22$\pm$ 0.58 |  74.03$\pm$ 1.74 |  79.46$\pm$ 0.82 |  79.65$\pm$ 0.68 |  81.69$\pm$ 0.57 |  85.52$\pm$ 0.34
REMix(ours) |  71.33$\pm$ 1.22 |  77.94$\pm$ 0.59 |  79.76$\pm$ 0.47 | 77.58$\pm$ 0.59 |  81.13$\pm$ 0.82 |  85.51$\pm$ 0.38

###  5.2 Baselines and implementation details

We compare REMix with three state-of-the-art models that are representative of
the existing class of methods for SSRE: MRefG Li et al. (2021), MetaSRE Hu et
al. (2021a), and GradLRE Hu et al. (2021b). MRefG leverages the unlabelled
data by semantically or lexically connecting them to labelled data by
constructing reference graphs, such as entity reference or verb reference.
This approach heavily leverages the linguistic structure of the data and is
the only existing method that falls outside the self-training class of
methods. MetaSRE generates pseudo labels on unlabelled data by learning from
the mistakes of the classification model as an additional meta-objective.
GradLRE on the other hand generates pseudo label data to imitate the gradient
descent direction on labelled data and bootstrap its optimization capability
through trial and error Hu et al. (2021b). MetaSRE and GradLRE are two of the
strongest methods in the widely adapted self-training methods for SSRE.

Implementation details. We follow the established setting to use stratified
sampling to divide the training set into various proportions of labelled and
unlabelled sets so that the relation label distribution remains the same
across all subsets. Following existing work, we sample 5%, 10%, and 30% of the
training set as labelled data for the SemEval and KBP37 datasets, and 3%, 10%,
and 15% of the training set as labelled data for TACRED and RE-TACRED
datasets. For all datasets and experiments, unless otherwise specified, we
sample 50% of the training set as the unlabelled set. For TACRED and SemEval
datasets we take the performance numbers of all baseline models reported by Hu
et al. (2021b). For other datasets, we re-run the models with their best
configuration as provided in their respective implementations, when available.
To be consistent with all the baselines we initialize the text encoder of
REMix with the bert-base-cased model architecture and pre-trained weights.
Full details of all the hyperparameters can be found in Appendix A.2.

###  5.3 Main Results

Table 2 shows F1 results of all baseline models and our proposed model, REMix,
on the four datasets when leveraging various amounts of labelled data and 50%
unlabelled data. We report the mean and standard deviation of 5 different runs
(with different seeds) of training and testing. REMix gives state-of-the-art
performance on 10 out of 12 different configurations across all four datasets.
This reinforces the importance of consistency regularization beyond the
currently popular self-training methods for SSRE. Interestingly, the
performance gains are significantly higher for TACRED and RE-TACRED
datasets–we see an average improvement of as much as 17% when trained on 3%
labelled data. This can be attributed to the fact that entity type information
is available for these datasets and entity type markers are very effective in
exploiting the knowledge embedded in the pre-trained language models. We
revisit this observation in our ablation studies (§5.4) where we concretely
establish the benefits of using entity type markers.

###  5.4 Analysis and discussion

Figure 3: F1 Performance with various unlabelled data and 10% labelled data

Figure 4: F1 Performance of REMix with 50% unlabelled and 10% labelled data
with changing mixing coefficient $\lambda$

We first conduct experiments to empirically demonstrate the effectiveness of
three components of our proposed model: i) data augmentation by latent space
interpolation (Mix-DA), ii) data augmentation by constrained back-translation
(BT-DA), and iii) entity type markers (ET). In Table 3, we report the mean F1
score of five different runs for different variations of our model by removing
a certain combination of these components666We omit standard deviation values
for brevity. See the full results and comparisons in Appendix B. As can be
seen from Table 3, each of these components contributes to the overall success
of REMix. For contribution of just the Mix-DA: we compare i) row 1 v/s row 3,
and ii) row 2 v/s row 4. All comparisons show positive improvement. For
contribution of just the BT-DA: we compare i) row 1 v/s row 2, and ii) row 3
v/s row 4. We note that BT-DA results only in marginal improvements in most
cases. Upon closer inspection we note that the constrained-decoding algorithms
we implement for BT-DA are actually not perfect, especially when combined with
translation models. It sometimes misses the constraints and sometimes falls
into repetitive loops in an attempt to satisfy the constraint. With the ever-
improving language generation capabilities, we believe the quality of data
augmentation will only improve with time and result in more significant
performance improvements. For contribution of both DA techniques together: we
compare row 1 v/s row 4. All comparisons show significant improvements with
data augmentation. The contribution of entity type markers can be noted in
TACRED and RE-TACRED datasets. We see an average drop of 5.4% in F1 score
across all 8 comparisons. Although our data augmentation techniques are
effective, without the entity type information our method doesn’t result in
state-of-the-art performance. This reinforces our architectural choice to
include entity type markers when using [CLS] token for relation representation
(§4.2).

Table 3: Ablation results on all datasets using 10% labelled set and 50%
unlabelled set.

Mix. | BT-DA | ET | TACRED | RE-T | $\text{KBP37}^{*}$ | $\text{SemEval}^{*}$
---|---|---|---|---|---|---
✓ | ✓ | ✓ | 61.30 | 77.94 | 63.82 | 81.13
✗ | 56.82 | 75.11
✓ | ✗ | ✓ | 60.81 | 77.77 | 63.48 | 79.71
✗ | 56.35 | 74.67
✗ | ✓ | ✓ | 59.65 | 76.80 | 62.64 | 79.17
✗ | 55.52 | 73.78
✗ | ✗ | ✓ | 58.96 | 77.25 | 63.14 | 79.20
✗ | 55.25 | 74.58

* these datasets do not have entity type information

Next we examine the effect of using different amounts of unlabelled data. In
Figure 3, we report the average F1 score for different models trained with
different amounts of unlabelled data and 10% labelled data. REMix outperforms
the baselines in all settings except on SemEval dataset, and, interestingly,
the performance only marginally changes with the change in the amount of
unlabelled data. Note that we train the models until the performance on the
validation set stops improving for more than 5 epochs. Hence, REMix generates,
in principle, an infinite amount of unlabelled data via the mixup strategy.
Coupled with the fact that the label distribution remains the same in all
settings, adding more unlabelled data does not seem to add a lot of new
information. This explains why the model performance is relatively insensitive
to changing amounts of unlabelled data. This also implies that REMix can
leverage low amounts of unlabelled data better than the baselines.

Finally, in Figure 4 we show how the performance of REMix changes with a
change in the mean of the Beta distribution from which $\lambda$ is sampled on
each iteration. Note that a value near 0 and 1 for $\lambda$ means the
augmented virtual data point will be closer to one of the underlying data
points. As we get closer to 0.5 the virtual data points get further from the
original data manifold and become more ‘novel’. On TACRED and RE-TACRED
datasets the performance peaks at $\text{E}(\lambda)=0.15(\text{or 0.85})$ and
drops in the mid-values. This can be interpreted as: adding datapoints far
from the original data manifold is detrimental for these datasets.
Interestingly, on KBP37 and SemEval the pattern inverts, i.e., the performance
increases as $\text{E}(\lambda)$ approaches $0.5$, implying that more ‘novel’
augmentations help for these datasets.

##  6 Conclusion

In this paper, we propose a consistency-training-based semi-supervised
algorithm for relation extraction and empirically show the merit of this class
of methods in comparison to the current state-of-the-art self-training class
of methods. In future work, one could bootstrap the self-training methods with
consistency training as done in some previous works on vision tasks Pham et
al. (2021). Additionally, we show how the entity type information, when
available, can result in massive performance boosts in the semi-supervised
scenario. This is important because in most practical use cases when entities
have already been identified, the entity type information is easy available
and could be effectively leveraged in the proposed fashion.

##  7 Limitations

One of the key limitations of our proposed method compared to the baseline
methods is the tight dependence on a strong external translation system to get
good quality back-translated data augmentations. Secondly, since we use [CLS]
token embeddings instead of entity-specific embeddings for final
classification, it is more challenging to decipher entity-specific context.
This is evident from the fact that our method performs relatively the weakest
on the SemEval dataset which consists of untyped nominals (concepts) as
entities and abstract relations which we believe need more entity-specific
context to understand. Hence, our proposed method, REMix, shines bright when
the entities in the dataset are typed or named entities whose meaning or type
is relatively context-agnostic.

## Acknowledgements

The author would like to thank the reviewers for their thoughtful feedback,
the entire AI/ML Research team at Vanguard for their support, and especially
David Landsman for their valuable input and discussion on the initial drafts
of this work.

## Disclaimer

This material is provided for informational purposes only and is not intended
to be investment advice or a recommendation to take any particular investment
action. ©2022 The Vanguard Group, Inc. All rights reserved.

## References

  * Anaby-Tavor et al. (2020) Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2020.  Do not have enough data? deep learning to the rescue! _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(05):7383–7390.
  * Bach and Badaskar (2007) Nguyen Bach and Sameer Badaskar. 2007.  A review of relation extraction.
  * Baldini Soares et al. (2019) Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019\.  Matching the blanks: Distributional similarity for relation learning.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2895–2905, Florence, Italy. Association for Computational Linguistics.
  * Berthelot et al. (2019) David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. 2019.  Mixmatch: A holistic approach to semi-supervised learning.  In _Advances in Neural Information Processing Systems_ , volume 32.
  * Chapelle and Zien (2005) Olivier Chapelle and Alexander Zien. 2005.  Semi-supervised classification by low density separation.  In _Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics_ , volume R5 of _Proceedings of Machine Learning Research_ , pages 57–64. PMLR.
  * Chen et al. (2020a) Jiaao Chen, Zhenghui Wang, Ran Tian, Zichao Yang, and Diyi Yang. 2020a.  Local additivity based data augmentation for semi-supervised NER.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1241–1251, Online. Association for Computational Linguistics.
  * Chen et al. (2020b) Jiaao Chen, Zichao Yang, and Diyi Yang. 2020b.  MixText: Linguistically-informed interpolation of hidden space for semi-supervised text classification.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2147–2157, Online. Association for Computational Linguistics.
  * Cheng et al. (2020) Yong Cheng, Lu Jiang, Wolfgang Macherey, and Jacob Eisenstein. 2020.  AdvAug: Robust adversarial augmentation for neural machine translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5961–5970, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018.  Understanding back-translation at scale.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 489–500, Brussels, Belgium. Association for Computational Linguistics.
  * Feng et al. (2021) Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021.  A survey of data augmentation approaches for NLP.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 968–988, Online. Association for Computational Linguistics.
  * Ghosh and Thiery (2021) Atin Ghosh and Alexandre H. Thiery. 2021.  On data-augmentation and consistency-based semi-supervised learning.  In _International Conference on Learning Representations_.
  * Hendrickx et al. (2010) Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010.  SemEval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals.  In _Proceedings of the 5th International Workshop on Semantic Evaluation_ , pages 33–38, Uppsala, Sweden. Association for Computational Linguistics.
  * Hu et al. (2019) J. Edward Hu, Huda Khayrallah, Ryan Culkin, Patrick Xia, Tongfei Chen, Matt Post, and Benjamin Van Durme. 2019.  Improved lexically constrained decoding for translation and monolingual rewriting.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 839–850, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Hu et al. (2021a) Xuming Hu, Chenwei Zhang, Fukun Ma, Chenyao Liu, Lijie Wen, and Philip S. Yu. 2021a.  Semi-supervised relation extraction via incremental meta self-training.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 487–496, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Hu et al. (2021b) Xuming Hu, Chenwei Zhang, Yawen Yang, Xiaohe Li, Li Lin, Lijie Wen, and Philip S. Yu. 2021b.  Gradient imitation reinforcement learning for low resource relation extraction.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 2737–2746, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Jawahar et al. (2019) Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019.  What does BERT learn about the structure of language? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3651–3657, Florence, Italy. Association for Computational Linguistics.
  * Li et al. (2021) Wanli Li, Tieyun Qian, Xu Chen, Kejian Tang, Shaohui Zhan, and Tao Zhan. 2021.  Exploit a multi-head reference graph for semi-supervised relation extraction.  In _International Joint Conference on Neural Networks, IJCNN 2021, Shenzhen, China, July 18-22, 2021_ , pages 1–7. IEEE.
  * Lin et al. (2019) Hongtao Lin, Jun Yan, Meng Qu, and Xiang Ren. 2019.  Learning dual retrieval module for semi-supervised relation extraction.  In _The World Wide Web Conference_ , WWW ’19, page 1073–1083, New York, NY, USA. Association for Computing Machinery.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Luan et al. (2018) Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018.  Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3219–3232, Brussels, Belgium. Association for Computational Linguistics.
  * Miyato et al. (2019) Takeru Miyato, Shin ichi Maeda, Masanori Koyama, and Shin Ishii. 2019.  Virtual adversarial training: A regularization method for supervised and semi-supervised learning.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 41:1979–1993.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of NAACL-HLT 2019: Demonstrations_.
  * Pham et al. (2021) Hieu Pham, Qizhe Xie, Zihang Dai, and Quoc V. Le. 2021.  Meta pseudo labels.  _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 11552–11563.
  * Post and Vilar (2018) Matt Post and David Vilar. 2018.  Fast lexically constrained decoding with dynamic beam allocation for neural machine translation.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1314–1324, New Orleans, Louisiana. Association for Computational Linguistics.
  * Sachan et al. (2019) Devendra Singh Sachan, Manzil Zaheer, and Ruslan Salakhutdinov. 2019.  Revisiting lstm networks for semi-supervised text classification via mixed objective function.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 33(01):6940–6948.
  * Stoica et al. (2021) George Stoica, Emmanouil Antonios Platanios, and Barnabas Poczos. 2021.  Re-tacred: Addressing shortcomings of the tacred dataset.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(15):13843–13850.
  * Sugiyama and Yoshinaga (2019) Amane Sugiyama and Naoki Yoshinaga. 2019.  Data augmentation using back-translation for context-aware neural machine translation.  In _Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019)_ , pages 35–44, Hong Kong, China. Association for Computational Linguistics.
  * Sun et al. (2021) Haitian Sun, Patrick Verga, Bhuwan Dhingra, Ruslan Salakhutdinov, and William W Cohen. 2021.  Reasoning over virtual knowledge bases with open predicate relations.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 9966–9977. PMLR.
  * Tarvainen and Valpola (2017) Antti Tarvainen and Harri Valpola. 2017.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In _Advances in Neural Information Processing Systems_ , volume 30.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Verma et al. (2019) Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. 2019.  Manifold mixup: Better representations by interpolating hidden states.  In _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 6438–6447. PMLR.
  * Xie et al. (2020) Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. 2020.  Unsupervised data augmentation for consistency training.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 6256–6268.
  * Yun et al. (2019) Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Young Joon Yoo. 2019.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 6022–6031.
  * Zhang and Wang (2015) Dongxu Zhang and Dong Wang. 2015.  Relation classification via recurrent neural network.  _arXiv preprint arXiv:1508.01006_.
  * Zhang et al. (2018) Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. 2018.  mixup: Beyond empirical risk minimization.  In _International Conference on Learning Representations_.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _Advances in Neural Information Processing Systems_ , volume 28.
  * Zhang et al. (2017) Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017.  Position-aware attention and supervised data improve slot filling.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017)_ , pages 35–45.
  * Zhou and Chen (2021) Wenxuan Zhou and Muhao Chen. 2021.  An improved baseline for sentence-level relation extraction.  _ArXiv_ , abs/2102.01373.

##  Appendix A Reproducibility checklist

###  A.1 Datasets

The sources of all the datasets are given in Table 4. We use the given
train/validation/test splits for TACRED, RE-TACRED and KBP37 datasets. For
SemEval dataset, we use the same splits as all the baselines, i.e., we split
the original training set into 90% training set and 10% validation set.

Table 4: Dataset sources Dataset | Source
---|---
TACRED | https://catalog.ldc.upenn.edu/LDC2018T24
RE-TACRED | https://github.com/gstoica27/Re-TACRED
KBP37 | https://github.com/zhangdongxu/kbp37
SemEval | https://semeval2.fbk.eu/semeval2.php?location=data

###  A.2 Hyperparameters

We use the BERT tokenizer and set maximum sequence length to 256 to pre-
process all datasets. We use the AdamW optimizer Loshchilov and Hutter (2019)
with 5e-5 learning rate and 0.1 warmup ratio. We sweep over the following
hyperparameters: sharpening coefficient $T$, confidence threshold $\gamma$,
the Beta-distribution parameters ($\alpha$, $\beta$)777$\alpha$ is fixed to be
60 and we change the values of $\beta$ to control the mean of the
distribution, and the unsupervised loss weight $\gamma_{m}$. We perform
incremental grid search to get the best performing configuration based on the
F1 score on validation set. Table 5 shows the set of values we use for each
parameter. Table 6 shows the best parameter values on each dataset and
configuration. Following Chen et al. (2020b), we use {7, 9, 12} for the mixup
layer set; this layer subset contains most of the syntactic and semantic
information as suggested by Jawahar et al. (2019).

Table 5: Hyperparameter search values

Parameter | Values
---|---
$T$ | {0.4, 0.6, 0.8, 1.0}
$\gamma$ | {0, 0.15, 0.2, 0.25}
$\beta$ | {1, 10, 30, 60, 120, 190, 300, 600}*
$\gamma_{m}$ | {0.01, 0.1, 1}

* corresponding means of the sampled mixing coefficient, $\lambda$, are given by {0.04, 0.09, 0.17, 0.24, 0.33, 0.50, 0.67, 0.86, 0.98}

Table 6: Best hyperparameter values | TACRED | KBP37
---|---|---
| 3% | 10% | 15% | 5% | 10% | 30%
$T$ | 0.8 | 0.8 | 0.8 | 0.8 | 0.8 | 0.8
$\gamma$ | 0.15 | 0.15 | 0.90 | 0.25 | 0.25 | 0.70
$\beta$ | 10 | 10 | 10 | 120 | 120 | 190
$\gamma_{m}$ | 0.01 | 0.1 | 0.1 | 0.1 | 1.0 | 1.0
| RE-TACRED | SemEval
| 3% | 10% | 15% | 5% | 10% | 30%
$T$ | 0.4 | 0.4 | 0.4 | 0.8 | 0.8 | 0.8
$\gamma$ | 0.0 | 0.0 | 0.9 | 0.2 | 0.2 | 0.7
$\beta$ | 10 | 10 | 10 | 60 | 60 | 60
$\gamma_{m}$ | 0.01 | 0.1 | 0.1 | 0.1 | 1.0 | 1.0

###  A.3 Training details

We train each model on a single NVIDIA Tesla T4 GPU with 16GB memory. We
employ mixed precision training and gradient checkpointing techniques for
faster and memory-efficient training. Note that we train the models until the
performance on the validation set plateaus. The full REMix model roughly takes
about 6 hours to train on TACRED, 5 hours in RE-TACRED, 1 hour in KBP37, and
about 30 minutes on SemEval. Note that the training time slightly varies
($\pm$ 30 minutes) depending on the percentage of labelled and unlabelled data
we use. The number of parameters in all our models are largely dominated by
the bert-base-cased that we use as the text encoder. The relatively negligible
varying component is the MLP classifier that varies with the varying number of
relations in each dataset.

Table 7: F1 score with 10% of labelled data and 50% unlabelled data. Mean and standard deviation of 5 different runs is reported. | Mix. | BT aug. | Ent. type | TACRED | RE-TACRED | KBP37 | SemEval
---|---|---|---|---|---|---|---
a) | ✓ | ✓ | ✓ | 61.30$\pm$ 0.70 | 77.94$\pm$ 0.59 | 63.82$\pm$ 0.71 | 81.13$\pm$ 0.82
✗ | 56.82$\pm$ 0.64 | 75.11$\pm$ 1.16
b) | ✓ | ✗ | ✓ | 60.81$\pm$ 1.31 | 77.77$\pm$ 0.96 | 63.48$\pm$ 0.53 | 79.71$\pm$ 0.83
✗ | 56.35$\pm$ 0.97 | 74.67$\pm$ 1.04
c) | ✗ | ✓ | ✓ | 59.65$\pm$ 0.92 | 76.80$\pm$ 0.98 | 62.64$\pm$ 0.69 | 79.17$\pm$ 1.64
✗ | 55.52$\pm$ 0.89 | 73.78$\pm$ 1.34
d) | ✗ | ✗ | ✓ | 58.96$\pm$ 1.21 | 77.25$\pm$ 0.70 | 63.14$\pm$ 0.90 | 79.20$\pm$ 0.32
✗ | 55.25$\pm$ 1.53 | 74.58$\pm$ 0.91

##  Appendix B Full Ablation results

We conduct experiments to empirically demonstrate the effectiveness of three
components of our proposed model: i) data augmentation by latent space
interpolation, ii) data augmentation by constrained back-translation, and iii)
entity type markers. In Table 7, we report the performance of different
variations of our model by removing a certain combination of these components.
As can be seen from Table 7, each of these components contributes to the
overall success of REMix. To see the impact of data augmentation by latent
space interpolation, we compare the results in row ‘a’ with the corresponding
ones in row ‘c’, and similarly row ‘b’ with row ‘d’. We see a significant and
consistent drop in every comparison with and without that component.
Specifically, we see a drop of an average 1.6% in F1 score over all the 12
comparisons. Another interesting pattern that stands out is the significant
and consistent drop in performance (average 5.4% in F1 score across all 8
comparisons) when not using entity type markers. When we drop the entity type
markers from the input representation, we use the basic entity start and end
markers–[E1], [/E1], [E2], [/E2]. This only applies to TACRED and RE-TACRED
since only these datasets have entity type information readily available. Note
that when we do not use any of the data augmentations, consistency training is
not possible and we are effectively using only the labelled set for training
the model (fourth row in Table 7). Surprisingly, even without any unlabelled
data using the entity type markers alone gives better performance than the
current state-of-the-art results by GradLRE Hu et al. (2021b), i.e., an F1
score of 58.96 vs 58.20 on TACRED and 77.25 vs 74.03 on RE-TACRED. This proves
the effectiveness of using entity type information in the fashion we use,
i.e., by tokenizing the type words to leverage the knowledge embedded in pre-
trained language models.


# Homomorphism of independent random variable convolution and matrix
multiplication

Yue Liu<EMAIL_ADDRESS>School of Mathematics and Statistics, Fuzhou
University, Fuzhou, 350116, China

###### Abstract

A map is given showing that convolutions of independent random variables over
a finite group and matrix multiplications of doubly stochastic matrices are
homomorphic. As an application, a short proof is given to the theorem that the
limiting distributions of stochastic processes with stationary independent
increments over a finite group are always uniform. AMS classification: 60G50,
15B51

###### keywords:

convolution; doubly stochastic matrix; random walk; finite group

## The Result

Convolution of random variables is a basic operation in probability theory,
especially the convolutions of independent ones. In this paper, attention is
restricted to convolutions of independent random variables over finite groups.

Let $G=\\{g_{1},\ldots,g_{n}\\}$ be a finite group. By Caley Theorem, let

$\phi:\ G\rightarrow\Sigma,\ g_{k}\mapsto\sigma_{k}$

be an isomorphism, where $\sigma_{k}$ is the left transition of $G$ with the
element $g_{k}$, thus $\Sigma=\\{\sigma_{1},\ldots,\sigma_{n}\\}$ can be also
regarded as a permutation subgroup of the symmetric group $S_{n}$ over $[n]$.
For each $k$, there is an order $n$ permutation matrix
$P_{\sigma_{k}}=\left(p_{ij}^{(k)}\right)_{n\times n}$ corresponding to
$\sigma_{k}$, where

$p_{ij}^{(k)}=\left\\{\begin{array}[]{ll}1,&i=\sigma_{k}(j),\\\
0,&\text{otherwise.}\end{array}\right.$

Let $X$ be a random variable over $G$. Write $p_{k}=\mathbb{P}[X=g_{k}],\
k=1,\ldots,n$. Define the _convolution matrix_ of $X$, denoted by $Con(X)$, as

$Con(X)=\sum_{k=1}^{n}p_{k}P_{\sigma_{k}}.$

It is easy to see that $Con(X)$ is a doubly stochastic matrix. The following
lemma shows that convolutions of independent random variables over a finite
group and matrix multiplications of doubly stochastic matrices are
homomorphic.

######  Lemma 1.

Let $X,Y$ be two independent random variables over a finite group $G$. Then

$Con(X\cdot Y)=Con(X)Con(Y),$

where $X\cdot Y$ is the convolution of $X$ and $Y$.

###### Proof.

The lemma can be verified by a direct computation. ∎

Random walk is a typical stochastic process. Random walks with stationary and
independent increments over Euclidean spaces are called _Lévy processes_ (see
[1] for example). A more general framework for the study of such processes is
proposed in [2], where the random variables are assumed to be over any
topological semigroups, rather than just Euclidean spaces. It was shown that
the asymptotic behaviors of the processes over compact spaces and noncompact
spaces are quite different (see Chapter 2 and 3 in [2]).

As an application of Lemma 1, together with the well-knowm Perron Theorem
(Theorem 1), we give a short proof to the following Theorem 2, which
characterize the asymptotic behavior of random walks with stationary and
independent increments over finite groups.

######  Theorem 1.

(Perron Theorem, [3, Theorem 8.4.4]) Let $A\in M_{n}$ be irreducible and
nonnegative, and suppose that $n\geq 2$. Then

  1. 1.

$\rho(A)>0$.

  2. 2.

$\rho(A)$ is an algebraically simple eigenvalue of $A$.

  3. 3.

there is a unique real vector $x=[x_{i}]$ such that $Ax=\rho(A)x$ and
$x_{1}+\cdots+x_{n}=1$; this vector is positive.

  4. 4.

there is a unique real vector $y=[y_{i}]$ such that $y^{T}A=\rho(A)y^{T}$ and
$x_{1}y_{1}+\cdots+x_{n}y_{n}=1$; this vector is positive.

Let $X$ be a random variable. $\mathcal{L}(X)$ is used to denote the
distribution law of $X$.

######  Theorem 2.

Let $G=\\{g_{1},\ldots,g_{n}\\}$ be a finite group. $(X_{m})_{m\geq 1}$ be a
$G$-valued stochastic process such that $X_{1}=\xi_{1}$ and
$X_{m+1}=X_{m}\xi_{m+1},\ m\geq 1,$ where $(\xi_{m})_{m\geq 1}$ are i.i.d.
random variables whose supports are $G$. Then

$\lim_{m\to\infty}\mathcal{L}(X_{m})=\mathcal{L}$

exists, and $\mathcal{L}$ is convolution invariant, i.e., $\mathcal{L}$ is a
uniform distribution over $G$.

###### Proof.

Let $P_{\sigma_{1}},\ldots,P_{\sigma_{n}}$ be the permutation matrices as in
the definition of convolution matrices. Write $P_{\sigma_{k}}=(p_{ij}^{(k)})$.
By the definitions, $p_{ij}^{(k)}=1$ means $g_{k}=g_{j}\cdot g_{i}^{-1}$.
Thus, for every pair of $i,j\in[n]$, there exists only one $k\in[n]$ such that
$p_{ij}^{(k)}=1$. Then $P_{\sigma_{1}},\ldots,P_{\sigma_{n}}$ are linearly
independent, and $\sum_{k=1}^{n}P_{\sigma_{k}}$ is the matrix $J$ whose
entries are all 1.

Since $(\xi_{m})_{m\geq 1}$ are i.i.d. random variables, their convolution
matrices are the same. Write $A=Con(\xi_{m})$. Since the support of $\xi_{m}$
is $G$, every $p_{k}=\mathbb{P}[\xi_{m}=g_{k}]$ is positive for
$k=1,\ldots,n$, which means $A$ is a positive matrix.

By Lemma 1, $Con(X_{m})=A^{m}$. Since $P_{\sigma_{1}},\ldots,P_{\sigma_{n}}$
are linearly independent, $\lim_{m\to\infty}\mathcal{L}(X_{m})$ exists is
equivalent to $\lim_{m\to\infty}A^{m}$ exists, and the limiting distribution
is uniquely determined by $\lim_{m\to\infty}A^{m}$.

Since $A$ is a positive doubly stochastic matrix, by Perron Theorem, the
spectral radius $\rho(A)=1$ is an algebraically simple eigenvalue, and
$x=\frac{1}{n}[1,\ldots,1]^{T},y^{T}=[1,\ldots,1]$ are the unique left and
right eigenvectors corresponding to $\rho(A)$ as described in Theorem 1,
respectively. Then

$\lim_{m\to\infty}A^{m}=x\cdot
y^{T}=\frac{1}{n}J=\frac{1}{n}P_{\sigma_{1}}+\cdots+\frac{1}{n}P_{\sigma_{n}},$

yielding that the limiting distribution of $(X_{m})_{m\geq 1}$ is the uniform
distribution. ∎

## Acknowledgments

The author would like to thank Prof. Jian Wang of Funjian Normal University
for the helpful discussions and suggestions, especially for providing the
background of random walks.

## References

  * Applebaum [2009] D. Applebaum, Lévy processes and stochastic calculus, Cambridge university press, 2009\.
  * Högnäs and Mukherjea [2011] G. Högnäs, A. Mukherjea, Probability measures on semigroups: convolution products, random walks, and random matrices, Springer, 2011\.
  * Horn and Johnson [2012] R. Horn, C. Johnson, Matrix Analysis(2nd Edition), Cambridge University Press, 2012\.