# ReMask: A Robust Information-Masking Approach for Domain Counterfactual
Generation

Pengfei Hong1∗, Rishabh Bhardwaj1∗, Navonil Majumdar1, Somak Aditya2, Soujanya
Poria1

1 ISTD, Singapore University of Technology and Design, 2 Department of CSE,
IIT Kharagpur
{pengfei_hong<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Domain shift††∗Equal Contribution is a big challenge in NLP, thus, many
approaches resort to learning domain-invariant features to mitigate the
inference phase domain shift. Such methods, however, fail to leverage the
domain-specific nuances relevant to the task at hand. To avoid such drawbacks,
domain counterfactual generation aims to transform a text from the source
domain to a given target domain. However, due to the limited availability of
data, such frequency-based methods often miss and lead to some valid and
spurious domain-token associations. Hence, we employ a three-step domain
obfuscation approach that involves frequency and attention norm-based masking,
to mask domain-specific cues, and unmasking to regain the domain generic
context. Our experiments empirically show that the counterfactual samples
sourced from our masked text lead to improved domain transfer on 10 out of 12
domain sentiment classification settings, with an average of 2% accuracy
improvement over the state-of-the-art for unsupervised domain adaptation
(UDA). Further, our model outperforms the state-of-the-art by achieving 1.4%
average accuracy improvement in the adversarial domain adaptation (ADA)
setting. Moreover, our model also shows its domain adaptation efficacy on a
large multi-domain intent classification dataset where it attains state-of-
the-art results. We release the codes publicly at https://github.com/declare-
lab/remask.

##  1 Introduction

Despite significant advances in unsupervised representation learning, natural
language processing (NLP) systems often strongly rely on expensive human-
annotated datasets. These (labeled) datasets, however, are only available in
specific domains. Systems trained on such datasets usually significantly
under-perform on out-of-domain (OOD) samples during inference, due to strong
reliance on the dataset-specific token- or feature-label correlations. These
correlations do not often generalize beyond the domain of the training
dataset. These correlations can even be spurious annotation artifacts. For
example, in a sentiment-labeled dataset on restaurant reviews, the token
_food_ may frequently appear in samples with negative sentiment score, due to
selection bias Veitch et al. (2021). Such biases are particularly prevalent in
low resource settings, where training instances are scarce Nan et al. (2021).

In response, annotating samples from new domains may prove to be expensive and
infeasible in the long term. To address these issues, many domain adaptations
(DA) techniques have been proposed Roark and Bacchiani (2003); Daumé and Marcu
(2006); Ben-David et al. (2010); Jiang and Zhai (2007); Rush et al. (2012);
Schnabel and Schütze (2014). Many of such techniques Blitzer et al. (2007);
Ziser and Reichart (2016); Ganin et al. (2015); Ben-David et al. (2020) learn
domain-invariant features that sidestep the pitfall of relying on domain-
specific features when faced with OOD inputs. This obviously also hinders
performance on in-domain samples where domain-specific features may be
relevant. To address this issue, Calderon et al. (2022) recently proposed
domain-counterfactual generation (DoCoGen) to transform given in-domain
samples into out-domain samples.

DoCoGen masks the source-domain-specific n-grams in the input using a token-
frequency-based approach. These masks are filled with target-domain-specific
tokens using a conditional text generation language model that is fine-tuned
with an unsupervised sentence reconstruction objective. However, the
frequency-based masking approach does not account for the context to identify
the source-domain-specific tokens. Moreover, this approach is limited by the
statistics of an incomplete training set. These shortcomings may cause the
approach to overlook many valid domain-token associations. For instance, in
Fig. 1 the word _tearjerker_ is not masked by DoCoGen, where this word may be
associated with the DVD of a tragedy movie. Furthermore, such frequency-based
masking fails to generalize well to out-of-vocabulary words.

Therefore, we propose a robust masking method that can achieve better
retention of non-domain information, while removing domain-specific
information. Our method includes three phases: frequency-based mask
initialization Calderon et al. (2022), over-the-top (OOT) masking, and
unmasking. We first phase is simply to initialize the mask using the
frequency-based strategy by Calderon et al. (2022). These masks are based on
token-domain association consisting of two factors: the probability of the
presence of a target token in a particular domain and the non-uniformity of
this probability distribution. Tokens surpassing a certain association score,
w.r.t. both source and target domain, are masked. In the second phase, we
leverage the encoded knowledge in language models (LM) Liu et al. (2019) to
identify additional token-domain associations, improving recall at the risk of
hurting precision. We found that a token with high attention-norm Kobayashi et
al. (2020), in a fine-tuned LM as domain classifier, could be strongly
associated with the predicted domain. Such tokens with high enough attention
norms are masked. Finally, we sequentially unmask the masked tokens, guided by
the confidence score of a domain classifier. To minimize spurious token-domain
associations from the prior two masking phases that hurt precision, the tokens
that cause low domain confidence upon unmasking are kept unmasked.

The overall contributions of this work are: i) a robust masking strategy to
mask more domain-specific tokens, as compared to the state of the art (SOTA)
Calderon et al. (2022), while mitigating spuriously masked tokens; ii) domain
counterfactuals (D-CON) from our model outperform the SOTA on binary and
multi-label classification tasks, under unsupervised domain adaptation (UDA)
and adversarial domain adaptation (ADA) settings, under almost all domain
shifts.

Figure 1: This diagram outlines the flow of our approach (top part) and
illustrates it with an example (bottom part); the input Text (source) is
sequentially fed through _Base_ , _OTT_ , and _Unmasking_ steps to produce
source-domain unaware masked text, which is passed to a T5-based generator to
obtain domain counterfactual.

##  2 Methodology

#### Task Description.

Let $X_{\mathcal{D}}$ denote a text sampled from domain $\mathcal{D}$. We
define a counterfactual generator function
$f_{c}^{\mathcal{D}\rightarrow\mathcal{D}^{\prime}}(X_{\mathcal{D}})$ as a
mapping from a text $X_{\mathcal{D}}\sim\mathcal{D}$ to a text
$X_{\mathcal{D^{\prime}}}\in\mathcal{D}^{\prime}$, such that, all but domain
$\mathcal{D}$-specific information is preserved in $X_{\mathcal{D}}^{\prime}$.
We denote the domain-agnostic information by $X$ and the domain-specific text
to be a specific fusion of $X$ and $\mathcal{D}$. We define a fusion function
$\mathcal{I}$, such that,
$X_{\mathcal{D}}\coloneqq\mathcal{I}(X;\mathcal{D})$. An inverse mapping
$\mathcal{I}^{-1}$ makes it feasible to efficiently disentangle and extract
the domain-generic information $X$. Therefore, the task of domain
counterfactual generation aims to find a function that essentially
disentangles the domain-generic information from $X_{\mathcal{D}}$ to obtain
$X$, followed by mapping $X$ to a domain-specific text
$X_{\mathcal{D}^{\prime}}$. Thus, an ideal
$f_{c}^{\mathcal{D}\rightarrow\mathcal{D}^{\prime}}$ can be considered to be
functionally equivalent to
$\mathcal{I}(\mathcal{I}^{-1}(X_{\mathcal{D}}),\mathcal{D}^{\prime})$.

#### Overview.

The proposed approach consists of two phases:

  * •

Domain Obfuscation: This phase masks a text $\mathcal{X}_{D}$, such that, it
becomes void of source-domain-specific information. To this end, we propose a
three-step masking approach.

  * •

Domain Counterfactual Generation: As Calderon et al. (2022), we feed the
masked text to a T5-based encoder-decoder model $f_{c}$ to generate a
counterfactual in the target domain $\mathcal{D}^{\prime}$.

###  2.1 Domain Obfuscation

We propose a novel three-step approach for domain corruption to mask more
words carrying domain-specific cues and unmask words that are not specific to
any domain. Masking words carrying domain-specific information helps the model
learn to rely more on the input prompt, allowing us to have higher control
over the generated text. Moreover, unmasking domain-generic words preserve the
context information of text so as to keep the model-generated output an
equivalent text to input but in the specified domain.

#### Step 1 (Base Masking).

We begin with the frequency-based (heuristic) masks, proposed by Calderon et
al. (2022). It assigns an affinity-based score to a word $w\in X$. The word
($w$)-domain($\mathcal{D}$) affinity is defined by

$\rho(w,\mathcal{D})=P(\mathcal{D}\mid w)\cdot\left(1-\frac{H(D\mid w)}{\log
N}\right),$

where $D\in\\{1,\ldots,N\\}$ is a random variable representing a $N$ domain
classes. $H(D|w)$ denotes the entropy of random variable $D|w$. A low value of
$H(D|w)$ signifies that the spread of word $w$ is skewed towards fewer
domains, and thus domain-specific. $P(\mathcal{D}{\mid}w)$ dictates the
chances of domain being $\mathcal{D}$ given the word $w$. Thus, a word is said
to have a high affinity with domain $\mathcal{D}$ if $\rho(w,\mathcal{D})$
returns a high value. To perform $X_{\mathcal{D}}\rightarrow
X_{\mathcal{D}}^{\prime}$ counterfactual generation, we mask a word $w\in X$
if the word has a high affinity towards $\mathcal{D}$, relative to
$\mathcal{D}^{\prime}$. Thus, the domain transfer (masking) score can be
defined by

$\displaystyle\mathrm{m}_{a}\left(w,\mathcal{D},\mathcal{D}^{\prime}\right){\coloneqq}\rho(w,\mathcal{D}){-}\rho\left(w,\mathcal{D}^{\prime}\right).$

The higher the value of
$\mathrm{m}_{a}\left(w,\mathcal{D},\mathcal{D}^{\prime}\right)$, the harder it
is to perform its $\mathcal{D}\rightarrow\mathcal{D}^{\prime}$ transfer. A
word is masked if its $\mathrm{m}_{a}$-score is above a predefined threshold
$\tau_{1}$. Generalizing the approach to n-grams, we first mask all the
unigram words. This is followed by identifying the bigrams that do not overlap
with the masked unigram, and then trigrams that do not overlap with the masked
tokens111Threshold $\tau_{1}$ is a hyperparameter, best found to be 0.08.

#### Step 2 (Over-The-Top Masking).

After heuristic masking, which is word context agnostic, we perform attention-
based over-the-top (OTT) masking. The OTT masking serves two purposes—1)
Context awareness: based on their context information, it reconsiders the
words for masking that failed $m_{a}$-scoring and remained unmasked; and 2)
Induction: it generalizes the notion of a domain to a more comprehensive set
of word vocabulary and their word co-occurrences in which the domain-specific
corpus is prone to be deficient. The context-aware attribute masks generic
words, that under a given context and style of text, become domain
informative. The inductive masking attribute makes use of the underlying
language model to exploit the learned word-word co-occurrences to reconsider
the unmasked words. Thus, considering words missing in the domain-specific
corpus as well as words whose frequencies in the corpus do not truly reflect
their domain affinity. This results in improved recall in domain-specific
token extraction, although at the risk of hurting precision.

To perform OTT masking, we train a domain classifier
$f_{d}:\mathcal{D}\rightarrow D$ that learns to classify an
$X_{\mathcal{D}}\sim\mathcal{D}$ to a domain label $d\in D$. For finding OTT
words to mask, we use an attention-based architecture as domain classifier
$f_{d}$. Norm-based attention analysis has been seen to efficiently capture
the importance of a word to the model’s prediction Kobayashi et al. (2020).
Following this observation, we define the attention-based domain affinity of a
word as

$\displaystyle\mathrm{m}_{b}\left(w;l\right)\coloneqq||\alpha_{w}^{l}\bm{v}_{w}^{l}||,$

where $||\cdot||$ computes Euclidean norm. In layer $l$ operations, let
$\mathbf{W}_{V}^{l}$, $\bm{b}_{V}^{l}$ denote the value matrix and biases,
$\mathbf{W}_{O}^{l}$ be the output projection. For a word $w$, let
$\bm{y}_{w}^{l}$ be its representation at the output of layer $l$, we define
$\bm{v}_{w}^{l}=(\bm{y}_{w}^{l-1}\mathbf{W}_{V}^{l}+\bm{b}_{V}^{l})\mathbf{W}_{O}^{l}$,
$\alpha_{w}^{l}$ specifies how much the classification token222Classification
token for RoBERTa is <s> in layer $l$ attends to $w$. We mask all the words
whose $\mathrm{m}_{b}$ score is higher than a threshold denoted by $\tau_{2}$.
$\tau_{2}$ is set as a hyperparameter based on each $X_{\mathcal{D}}$, this
will be explained further in step 4.

#### Step 3 (Unmasking).

One of the aims of domain-counterfactual text generation is to preserve
everything —- including the task label — except domain information. Therefore
to make the minimal edits to its original text Calderon et al. (2022), we
reduce the number of edits to by restoring the masked tokens from prior steps.
Meanwhile, we posit that domain information masked by OTT (Step 2) is not
unmasked in Step 3. The latter is controlled by thresholding the domain
classification score of the masked sentence produced in this step 2. Analogous
to counterfactually-augmented data (CAD) generation Kaushik et al. (2020),
which is done by minimally intervening on examples to change their ground
truth label. In our case, we only want to use a minimal number of masks to
remove the amount of domain-specific terms in the text. Specifically, we use
the domain score given by the domain classifier to measure the amount of
domain information contained in the example and make sure it is below a
threshold 333In our initial experiments, we found the threshold value 0.4
works best. we also tried thresholding based on misclassification, in our
observations, it tends to unmask a lot of context-specific words..

Additionally, Step 1 and Step 2 are prone to excessive masking from spurious
domain-token correlations, resulting in the improved recall at the risk of
poor precision. To account for these factors, we aim to unmask tokens that do
not provide domain-specific cues, hence, restoring more contextual
information; this is inspired by Meng et al. (2022), where _causal tracing_ is
used to determine the association between different factors and the output in
a large language model. Given input text $X_{\mathcal{D}}$, following Step 1
and Step 2, we obtain a masked text $\tilde{X}^{K}_{\mathcal{D}}$, where
$K=\\{k_{1},...,k_{n}\\}$ represents $n$ positions of the masked tokens. We
define a subset of indices $U=\\{k_{i},...,k_{j}\\}\subseteq K$ containing the
positions of the words to be unmasked. Thus, $\tilde{X}^{K-U}$ represents the
rectified masked output.

First, we independently intervene on the masked text $\tilde{X}_{\mathcal{D}}$
by unmasking a token at position $i$ and perform domain classification using
$f_{d}$, to obtain the change in domain-label probability of $D$. We also
define

$\mathrm{m}_{u}(w_{k_{i}}){\coloneqq}f_{d}^{\mathcal{D}}(\tilde{X}_{\mathcal{D}}^{K-\\{k_{i}\\}}){-}f_{d}^{\mathcal{D}}(\tilde{X}_{\mathcal{D}}),$

where $f_{d}^{\mathcal{D}}(\cdot)$ returns a probability score of the origin
domain $\mathcal{D}$. Per iteration, we sequentially restore the tokens in
$K$, in the ascending order of $\mathrm{m}_{u}(w_{k_{i}})$. We stop the
unmasking iteration when the condition
$f_{d}^{\mathcal{D}}(\tilde{X}_{\mathcal{D}}^{K-U})<\tau_{3}$ is violated.
Then we get the unmasked example. We posit that the domain specificity is
highly correlated with $\mathrm{m}_{u}(w_{k_{i}})$. Therefore, we find taking
the greedy approach to unmasking from the most domain-descriptive tokens
robust in finding the optimal $U$.

Original from [dvd]: Pippin dvd, I loved Ben Vereen in the show. Wanted the
music for my ipod, too. Very satisfying! Masking [DoCoGen] to [book]: Pipin
<m>, I loved Ben Vereen in the <m>. Wanted <m> <m> for my ipod too. Very
satisfying! Masking [Ours] to [book]: Pipin <m>, I loved <m> <m> in the <m>.
Wanted the for my <m> too. Very satisfying! Original from [electronics]: sony
rm - ax4000 this is a poorly designed remote. i have six devices connected to
my hd television set. the software depends on the user assigning positions to
the various inputs to the tv and, in my case, routinely activated the wrong
device when i used the remote. i purchased a logistics harmony remote and it
works perfectly. Masking [DoCoGen] to [kitchen]: <m> \- ax4000 this is a
poorly <m> . i have six <m> to my <m> set . the <m> depends on the <m>
assigning positions to the various <m> to <m> , in <m> routinely <m> wrong <m>
when <m> the <m> . <m> logistics <m> and <m> . Masking [Ours] to [kitchen]:
<m> \- <m> this is a poorly designed <m> . i have six <m> to my <m> set . the
<m> depends on the <m> assigning <m> to the various <m> , in my case ,
routinely <m> the wrong <m> when i <m> . i purchased a <m> and it works
perfectly.

Table 1: Domain Obfuscation (Masking).

###  2.2 Domain Counterfactual Generation

#### Training.

We do not have access to parallel domain counterfactuals. Thus, following
Calderon et al. (2022), we train a T5-based encoder-decoder model $M$ in an
unsupervised manner. We essentially train the model to reconstruct the
original text $X$ in the source domain from its masked form $\tilde{X}$, which
is presumably purged of source-domain-specific cues. To have control on the
domain of the generated text, we follow DoCoGen’s approach of prepending a
soft prompt $v$, indicating the domain, to the masked text in the input. The
soft prompt is initialized using the embedding of the word444Following
DoCoGen, we choose the most common word in that domain as the representative
of that domain, we list the words we use in the appendix representative of the
target domain $\mathcal{D}$ and trained together with the text reconstruction
objective. During training, the model learns to generate the original example,
given the domain obfuscated text and the soft prompt:
$M(\tilde{X},v)\rightarrow X$.

#### Inference.

Given $(x,\mathcal{D},\mathcal{D}^{\prime})$, we first feed through the $x$
through our domain obfuscation process to get $\tilde{x}$ and select the soft
prompt $v^{\prime}$ to represent $\mathcal{D}^{\prime}$. We feed
$(\tilde{x},v^{\prime})$ to the model to generate the domain counterfactual
$x^{\prime}$. We used beam search with a beam size of four for decoding
$x^{\prime}$.

##  3 Intrinsic Evaluation

Conditional generation is difficult to automatically evaluate. Therefore, we
asked three Ph.D. students trained in natural language processing to manually
evaluate the dataset considering the following evaluation measures: (1) Domain
Relevance (D.REL) - whether the topic of the generated text is related to the
target domain; (2) Label Preservation (L.PRES) - what is the label of the
generated text and if the original label preserved; (3) Linguistic
Acceptability (ACCPT) - how logical and grammatical the example is (on a 1-5
scale); and (4) Word Error Rate (WER) - what is the minimum number of word
substitutions, deletions, and insertions required to make the example logical
and grammatical. The test is conducted on 20 reviews, uniformly distributed
among four domains (A, D, E, K) and 60 generated domain counterfactuals using
DoCoGen and ReMask.

Table 2 shows that our masking achieves better scores than its baseline
DoCoGen method, especially in preserving the original example label, it
achieves a score of 95%, surpassing its heuristic counterpart. We suspect
ReMask keeps more information for the generation model to infer the label thus
preserving label-invariance. We analyse the reason for this further in Section
6

Model |  $\uparrow$ D.REL |  $\uparrow$ L.PRES |  $\uparrow$ ACCPT |  $\downarrow$ WER
---|---|---|---|---
DoCoGen | 85 | 87.5 | 4.21 | 1.05
ReMask | 92.5 | 95.0 | 4.33 | 1.0
Original Reviews | 100.0 | 100.0 | 4.94 | 0

Table 2: Human intrinsic evaluation. Up arrows ( $\uparrow$ ) represent
metrics where higher scores are better, and down arrows $(\downarrow)$
represent the opposite.

##  4 Experimental Settings

###  4.1 Tasks and Settings

We focused on two low-resource scenarios: unsupervised domain adaptation (UDA)
and any domain adaptation (ADA). UDA assumes the availability of large
unlabeled data from a source domain and target domain, as well as access to
the limited number of labeled examples in the source domain. We follow
Calderon et al. (2022) and choose to sample 100 labeled data in the source
domain for both of the tasks mentioned below. An even more challenging and
potentially more realistic setting is ADA, which assumes no access to both
labeled data and unlabeled data from the target domain in training time. In
other words, the model will not have any information about the target domain
distribution. Following a large body of DA work, we benchmark the proposed
ReMask on cross-domain sentiment classification task and cross-domain multi-
label intent classification task. For brevity, we refer readers to Calderon et
al. (2022) for a detailed description of each dataset.

#### Sentiment Classification

In this task, we combine 3 datasets together to form a dataset of 6 domains.
The dataset includes 4 domains - Books(B), DVDs(D), Electronic items(E), and
Kitchen Appliances(K) from product review multi-domain dataset Blitzer et al.
(2007); the challenging airline review dataset (A) air ; and the restaurant
(R) domain from Yelp dataset challenge Wei and Zou (2019). We benchmark the
sentiment classification task on UDA and ADA settings. In UDA, we focus on
four of the six domains: A, D, E, and K. This will result in 12 (3$\times$4)
cross-domain pairs. In this setting, the model can access all of the unlabeled
source domain and target domain data. whereas For ADA, where an unlabeled
dataset from the target domain is not within reach, our experiment uses A, D,
E, and K as source domains; B, and R as target domains, thus resulting in a
total of 8 (4$\times$2) domain pairs for ADA. Further, To facilitate the
comparison with previous work, we focused on low resource settings by randomly
sampling 100 labeled examples each from the four domains to as labeled source
dataset. The reported scores are averaged across 5 training and development
sets using different seeds.

#### Multi-Label Intent Prediction.

The second task is to predict the potential intents of the utterances arising
from information-seeking dialogs, as there could be multiple intents
originating from the same utterances, it is treated as a multi-label
classification problem. We choose the MANtIS dataset Penha et al. (2019),
inside each utterance could have eight potential intent labels. Following
Calderon et al. (2022), we only consider 5 most common labels: Further Detail
(FD), potential answer (PA), Information Request (IR), Greetings and Gratitude
(GG), Original Question (OQ).
The MANtIS dataset is a multi-domain dataset with 14 domains. We experiment
with the UDA setting. To ease comparison with DoCoGen, we choose the first six
domains: Apple (AP), DBA (DB), Electronics (EL), Physics(PH), Statistics (ST),
askubuntu (UB) with available unlabeled data to form 30 (5$\times$6) cross-
domain pairs for UDA.

Source Domain $\rightarrow$ Target Domain
($\mathcal{D}\rightarrow\mathcal{D}^{\prime}$) Model
$\mathbf{A}{\rightarrow}\mathbf{D}$ $\mathbf{A}{\rightarrow}\mathbf{E}$
$\mathbf{A}{\rightarrow}\mathbf{K}$ $\mathbf{D}{\rightarrow}\mathbf{A}$
$\mathbf{D}{\rightarrow}\mathbf{E}$ $\mathbf{D}{\rightarrow}\mathbf{K}$
$\mathbf{E}{\rightarrow}\mathbf{A}$ $\mathbf{E}{\rightarrow}\mathbf{D}$
$\mathbf{E}{\rightarrow}\mathbf{K}$ $\mathbf{K}{\rightarrow}\mathbf{A}$
$\mathbf{K}{\rightarrow}\mathbf{D}$ $\mathbf{K}{\rightarrow}\mathbf{E}$
$\mathbf{AVG}$ NoDA 69.4 78.6 78.2 72.3 80.2 82.4 81.0 79.8 87.6 72.5 78.6
85.4 78.8 RM-RR 69.5 80.1 80.0 72.3 81.0 83.8 79.6 79.5 88.4 70.6 79.1 84.5
79.0 PERL 72.9 81.1 83.6 81.5 83.0 86.9 81.1 81.7 88.5 77.9 78.2 86.1 81.9
DoCoGen 70.6 79.7 79.8 75.8 82.8 84.4 83.0 82.0 89.3 81.2 82.2 87.3 81.5
DoCoGen-Rob 62.7 73.9 74.9 76.9 85.7 87.7 82.7 82.1 90.3 81.9 83.0 88.9 80.9
2-ASM 68.7 81.8 79.5 81.0 88.3 88.9 82.0 84.4 91.4 80.7 80.9 88.1 82.9 3WO
65.4 78.4 81.5 80.3 81.0 80.0 85.4 75.6 85.2 84.2 80.0 82.1 79.9 NoInit 62.7
73.9 74.9 76.9 85.7 87.7 82.7 82.1 90.3 81.9 82.95 88.9 80.9 NoUnmask 69.1
77.2 71.1 78.5 79.6 78.9 76.4 80.0 83.4 66.4 74.1 82.0 76.4 NoOTT 62.7 73.9
74.9 76.9 85.7 82.7 82.1 90.3 81.9 83.0 88.9 80.9 80.8 OUR 74.4 85.3 78.5 78.2
88.5 88.9 81.0 82.9 89.7 81.5 83.4 89.7 83.5 Oracle-Gen 83.8 88.4 88.9 83.6
89.3 90.0 84.9 84.6 90.7 84.1 82.2 89.0 86.6

Table 3: Unsupervised Domain Adaptation results for sentiment classification
in the Blitzer et al. (2006) dataset. DoCoGen-Rob uses a RoBERTa-base task
classifier instead of T5 for a fair comparison to our model.

Source $\xrightarrow{}$ | $\mathbf{A}$ | $\mathbf{D}$ | $\mathbf{E}$ | $\mathbf{K}$ | AVG
---|---|---|---|---|---
Target $\xrightarrow{}$ | $\mathbf{B}$ | $\mathbf{R}$ | $\mathbf{B}$ | $\mathbf{R}$ | $\mathbf{B}$ | $\mathbf{R}$ | $\mathbf{B}$ | $\mathbf{R}$
NoDA | 69.1 | 76.5 | 82.3 | 82.8 | 81.5 | 84.5 | 82.4 | 85.2 | 80.5
RM-RR | 69.4 | 78.4 | 83.8 | 83.5 | 81.9 | 85.6 | 83.7 | 85.4 | 81.5
DoCoGen | 70.9 | 78.1 | 84.4 | 82.9 | 83.9 | 86.0 | 84.5 | 85.7 | 82.1
DoCoGen-Rob | 70.4 | 82.1 | 79.5 | 82.4 | 82.9 | 86.6 | 85.5 | 86.7 | 82.0
2-ASM | 65.7 | 76.3 | 87.1 | 82.3 | 80.8 | 84.9 | 85.0 | 82.3 | 80.7
NoInit | 71.4 | 82.4 | 87.3 | 82.2 | 84.0 | 84.3 | 81.5 | 85.2 | 82.3
NoUnmask | 62.4 | 74.9 | 80.4 | 75.3 | 74.2 | 74.0 | 80.5 | 79.4 | 75.1
Our | 72.9 | 83.4 | 91.6 | 82.4 | 84.0 | 85.5 | 82.5 | 86.3 | 83.5
Oracle | 84.4 | 85.2 | 91.6 | 86.1 | 86.0 | 86.5 | 85.3 | 86.5 | 86.4

Table 4: Adversarial Domain Adaptation (ADA) results on the Blitzer et al.
(2006) dataset.

Model | AP | DB | EL | PH | ST | UB | AVG
---|---|---|---|---|---|---|---
NoDA | 71.1 | 67.1 | 72.0 | 47.3 | 66.0 | 72.8 | 66.1
DoCoGen | 69.8 | 73.9 | 70.7 | 62.0 | 67.8 | 73.4 | 70.1
ReMask | 70.7 | 73.2 | 72.7 | 60.3 | 68.1 | 73.4 | 70.9
Oracle-Gen | 77.1 | 75.0 | 76.2 | 73.6 | 70.1 | 72.3 | 74.1

Table 5: UDA results on the MANtIS dataset Penha et al. (2019).

##  5 Models and Baselines

We tested on three types of models 555Experiment 3 is not applicable to the
ADA setting due to lack of labeled data.: (a) baseline models (b) variants for
each step in our masking approach c) an upper-bound model to approximate the
best performance using domain counterfactuals (D-CON) on the downstream tasks.
Unless otherwise stated, all the domain classifiers and sentiment classifiers
use the same model, based on a pretrained Roberta-base model, and all
generation models are based on a pretrained T5-base.

#### Baseline DA Models.

We experiment with four baselines: (1) No-Domain-Adaptation (NoDA), the model
only trained on available training data from the source domain; (2) Random-
masking Random-Reconstruction (RMRR) Ng et al. (2020), randomly masks token
from the input example and then fills the masks by masked language modeling
head. 3) PERL Ben-David et al. (2020), a SOTA for the UDA setup. 4) DoCoGen:
Use the same model and training strategy as we do, but only use the first step
of our masking procedure.

#### Ablations.

We consider three types of ablations in our masking procedure. First, we show
ablations for step 2 in our procedure. (5) 2-Attention-Score-Masking (2-ASM)
where we use attention score rather than attention norm to perform OOT masking
step, (6) 3WO where we show the performance of unmasking by following word
order in the sentence. And finally, we show (7) NoInit, where we remove the
heuristic masking step, and only use OOT masking with unmasking. (8) No-
Unmask, where we only have step 1,2 and removed step 3 unmasking.

#### Upper-Bound.

To approximate the upper-bound for D-CON augmentation, we use Oracle-Matching
(Oracle) which can access to target domain labeled data Calderon et al.
(2022). Given an example from a source domain, Oracle looks for the most
similar example with the same label in the target domain as training data.

##  6 Results and Discussions

Tables 3 and 4 present sentiment classification accuracy results for the 12
UDA and 8 ADA setups respectively. Table 5 presents the average intent
prediction F1 scores for each source domain, taken across 5 target domains for
the UDA setup.

#### Results Comparison.

For sentiment classification, our method ReMask, outperforms all baseline
models in 10 of 12 UDA setups and in 6 of 8 ADA setups, gaining an average of
2% and 1.4% over the baseline masking approach in UDA and ADA settings,
respectively. Furthermore, in two ADA setups on sentiment classification, our
model outperforms the Oracle-Gen scores which suggests our model can
effectively remove domain-specific information in these domain pairs. For
intent prediction 666lack of working code implementation and specifications to
reproduce the reported results of DoCoGen, our model outperforms four out of
six setups, reaching an average gain of 0.8% across domains. Moreover, NoInit
result demonstrates that our model performance degrades if there is no initial
mask provided by step 1. One key observation on the masking process is that
Step 2 of our method fails to identify some domain-specific tokens in the
presence of too many such tokens in the text. We suspect that the thresholding
attention norm is not robust enough when the attention norm might be thinly
spread among many domain-specific tokens. On the other hand, ReMask
outperforms DoCoGen by 0.8% on the MANtIS dataset under the UDA setup. This
further confirms the efficacy of our method. We found that our ReMask have
less performance improvement for domains where the domain specific words are
distinctive like UB and have higher performance boost in domains that have
more complex text, e.g. EL and PH.

#### Discussion on ReMask’s ability to retain contextual information and
remove domain-specific information.

To test our proposed method’s effectiveness in removing domain-specific
information, we train a domain classifier (based on the Roberta-base model) to
predict the domain from the masked text. Table 6 shows that the accuracy of
the domain classifier trained on text masked by ReMask is consistently lower
than DoCoGen, which proves the efficiency of our model ability in the
obfuscation of the domain. The difference shrinks as the number of training
examples increases. This may suggest that the model is still able to exploit
the spurious correlation between linguistic features and the domain of the
dataset.

Model / #Train Samples | 400 | 1k | 10k
---|---|---|---
DoCoGen | 40.6 | 70.4 | 86.6
ReMask | 15.4 | 44.1 | 85.2
Table 6: Domain classification accuracy on the masked text between ReMask vs
DoCoGen masking.

Domain | A | D | E | K
---|---|---|---|---
Step1 | +Step2 | +Step3 | Step1 | +Step2 | +Step3 | Step1 | +Step2 | +Step3 | Step1 | +Step2 | +Step3
A | 15.2 | 27.8 | 16.3 | 37.9 | 48.16 | 44.5 | 37.3 | 54.2 | 30.9 | 38.0 | 56.1 | 33.3
D | 25.0 | 34.1 | 31.3 | 16.5 | 27.6 | 24.0 | 24.0 | 44.5 | 23.2 | 23.9 | 45.2 | 26.1
E | 27.8 | 33.4 | 26.5 | 26.7 | 30.1 | 24.5 | 15.7 | 30.2 | 16.4 | 19.7 | 22.0 | 19.6
K | 30.2 | 40.2 | 38.7 | 28.7 | 45.8 | 35.6 | 21.1 | 33.5 | 21.4 | 15.7 | 27.0 | 22.8

Table 7: The average number of masks in each step.

#### The Need of Step 2 and Step 3.

As presented in Table 3, ReMask without Step 3 (NoUnmask) performs very
poorly, worse than DoCoGen. This might give us the impression that Step 2 is
unnecessary. However, we hypothesize that Step 1 alone is not enough to
correctly distinguish all the domain-indicative tokens — as Step 1 is not
context-sensitive. Step 2 addresses this issue which essentially aims at
increasing the recall of the token-level masking. Although Step 2 can increase
recall of toke-level masking, it has no control over the precision and hence
can produce imperfect masks that are domain neutral or key to preserving
contextual cues. One should also note that counterfactual generators should
try to generate counterfactuals with minimal edits. So, we try to spot and
remove these imperfect masks by a greedy unmasking strategy in Step 3. As
such, Step 3 aims to produce masks without losing recall in Step 2 (OTT), but
to improve the precision of Step 2-produced masks significantly. We
empirically verify the functioning of these steps in Table 7. We can see that
the number of masks after Step 2 increases significantly, thus risking low
precision and high recall (comparable to information retrieval problems). Step
3 unmasks such imperfect token-level masks resulting in fewer token-level
masks than Step 2 in Table 7. We also show qualitative examples in Table 8 to
depict the usefulness of Step 2 and Step 3, where Step 3 unmasks critical
domain neutral and contextual cue-bearing tokens as well as correctly retains
important domain-dependant masks. This, as a result, maintains the recall of
step 2 but increases its precision. Although we do not have a direct
measurement of this precision and recall hypothesis due to the lack of ground
truth labels of these masks — we indirectly prove so using our experiments in
Table 3. The number of masks after Step 3 lies between Step 1 and Step 2
indicating Step 1 may not capture all the relevant masks and Step 2 might be
excessively masking. If we further link this observation with Table 3, we can
conclude that Step 3 is required to improve performance over Step 1 (DoCoGen).

Source domain: dvd; destination domain: electronics
---
Original Text: Pippin dvd, I loved Ben Vereen in the show. Wanted the music
for my ipod, too. Very satisfying!
First Step Maksing: Pipin <m>, I loved Ben Vereen in the <m>. Wanted the <m>
for my ipod, too. Very satisfying!
Second Step OTT Masking: <m> <m>, I loved <m> in the <m>. Wanted the <m> for
my ipod, too. Very <m>!
Third Step Unmasking: Pipin <m>, I loved <m> in <m>. Wanted the <m> for my
ipod too. Very satisfying!
Generation: Pipin is great, I loved green color in the earphone. Wanted the
earphones for my ipod too. Very satisfying!
Source domain: airline, destination domain: kitchen
Original Text: 40 am departure time became 4. 30 am due to a crew issue. the
plane was an older 737. the empower jacks at our seats were dead and there was
grime on the hard part of the seat. there was no special meal service for
business class - everybody got a small ham and cheese sandwich. the front
toilet quickly got foul.
First Step Heuristic Masking:
<m> time became 4 <m> to a <m> issue . the <m> . the empower jacks at our <m>
dead and <m> grime on the hard part of the <m> . <m> special <m> for <m> \-
everybody got a small <m> and <m> . the <m> quickly got foul.
Second Step OTT Masking:
<m> <m> became 4 <m> to a <m> <m>. the <m> <m> <m> . the <m> at our <m> <m>
and <m> grime on the hard part of the <m> . <m> special <m> for <m> \- <m> got
a <m> <m> and <m> . the <m> quickly got <m>.
Third Step Unmasking:
<m> <m> became 4 <m> to a <m> issue. the <m> was <m> . the <m> at our dead and
<m> grime on the hard part of the <m> . <m> special <m> for <m> \- <m> got a
<m> and <m> . the <m> quickly got <m>.

Table 8: A few examples of the masks produced by different steps of our
approach; the masks highlighted with <m> are masks added in Step 2, but
removed(unmasked) in Step 3, whereas <m> remains masked after Step 3;
consecutive masks are at times merged into one mask for brevity.

##  7 Related Work

Our work takes inspiration from several established lines of research, namely
Domain Adaptation, Counterfactual Data augmentation, and counterfactual text
generation.

Domain Adaptation. Domain Adaptation (DA; Farahani et al. (2021)) deviates
from the assumption that test data comes from the same distribution as
training and is aimed at improving the performance of models in target domains
(possibly) with a different distribution than the training source domain. In
NLP, various DA setups are considered Approaches in Unsupervised Domain
Adaptation Blitzer et al. (2006) assumes the availability of unlabeled data
from both source and target domains, as well as the existence of labeled data
in the source domain. In adversarial domain adaptation Wang et al. (2019), the
assumptions are the same as UDA, but adversarial learning is employed to learn
domain-invariant representations. In contrast, and domain adaptation (ADA;
Ben-David et al. (2022)) approaches assumes no knowledge of the target domains
at training time. We consider UDA and Adversarial DA settings in our work.

Counterfactual Data Augmentation. Data augmentation, in general, is an
important sub-field of NLP that aims to mitigate problems introduced by the
low-resource availability of text data. It aims to increase the number of
examples for training without any explicit efforts to collect new samples.
There have been rule-based approaches that work by modifying the underlying
text by using some pre-defined heuristics Wei and Zou (2019). Beyond
significant heuristic efforts, there have been model-based approaches to
modify the words or generate prior distribution-based completely new samples
Kobayashi (2018). Counterfactual Data Augmentation is a more sophisticated
approach that performs a minimal text intervention (such as a specific
concept), thus constructing an example with a modified label Kaushik et al.
(2020).

(Counterfactual) Controlled Text Generation. In conditioned or controllable
text generation Prabhumoye et al. (2020), the task is to generate text that
satisfies certain pre-specified conditions (such as topic, sentiment or
domain). Its useful for a vast range of applications including data
augmentation and domain adaptation. Previous approaches involved finetuning
the LM outputs using re-inforcement learning Ziegler et al. (2019), training
Generative Adversarial Networks (Yu et al., 2017), or training conditional
generative models. Recent approaches adapted causal text modeling Feder et al.
(2022) and uses counterfactual editing of text towards controlled text
generation Madaan et al. (2021); Wu et al. (2021). Most of these models
experiment with shorter text and properties that are easier to ground in the
text (more specifically replacing one or two spans suffice).

##  8 Conclusion

This work empirically shows that the addition of attention norm-based masking
leads to additional domain-token associations missed by the SOTA masking
strategy. Furthermore, our iterative unmasking approach leads to the removal
of spurious domain-token associations, resulting in improved domain
obfuscation. These two innovations collectively allow better domain transfer
on sentiment and intent classifications tasks.

## Limitations

We discuss the limitations of our work:

  * •

While the three-step masking is shown to be beneficial, masking (base+OTT)
followed by unmasking may introduce several redundant computations as a token
masked in the first two steps might get unmasked in step 3.

  * •

When compared against the DoCoGen baseline Calderon et al. (2022), iterative
masking steps increase the time complexity of the domain obfuscation which
leads to masking latency. Moreover, as the domain classifier is a critical
part of the domain obfuscation, the approach has extra memory footprints.

  * •

The proposed masking approach introduces two extra hyperparameters $\tau_{2}$
and $\tau_{3}$ on top of the hyperparameters introduced by DoCoGen. While we
identify them as a fixed scalar value working for all kinds of input, we posit
that one can propose dynamic input or source domain adaptive thresholding.
Currently, we classify it as a limitation of the proposed work.

## Ethics Statement

For intrinsic evaluation, we engage three Ph.D. students who are fairly
compensated. This qualitative evaluation project passed ethics review of our
IRB as it does not contain any confidential data.

## References

  * (1) Skytrax user reviews dataset.  https://github.com/quankiquanki/skytrax-reviews-dataset.  Accessed: 2010-09-30.
  * Ben-David et al. (2022) Eyal Ben-David, Nadav Oved, and Roi Reichart. 2022.  PADA: example-based prompt learning for on-the-fly adaptation to unseen domains.  _Trans. Assoc. Comput. Linguistics_ , 10:414–433.
  * Ben-David et al. (2020) Eyal Ben-David, Carmel Rabinovitz, and Roi Reichart. 2020.  Perl: Pivot-based domain adaptation for pre-trained deep contextualized embedding models.  _Transactions of the Association for Computational Linguistics_ , 8:504–521.
  * Ben-David et al. (2010) Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando C Pereira, and Jennifer Wortman Vaughan. 2010.  A theory of learning from different domains.  _Machine Learning_ , 79:151–175.
  * Blitzer et al. (2007) John Blitzer, Mark Dredze, and Fernando Pereira. 2007.  Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification.  pages 440–447.
  * Blitzer et al. (2006) John Blitzer, Ryan T. McDonald, and Fernando C Pereira. 2006.  Domain adaptation with structural correspondence learning.  In _Conference on Empirical Methods in Natural Language Processing_.
  * Calderon et al. (2022) Nitay Calderon, Eyal Ben-David, Amir Feder, and Roi Reichart. 2022.  Docogen: Domain counterfactual generation for low resource domain adaptation.  _arXiv preprint arXiv:2202.12350_.
  * Daumé and Marcu (2006) Hal Daumé and Daniel Marcu. 2006.  Domain adaptation for statistical classifiers.  _J. Artif. Intell. Res._ , 26:101–126.
  * Farahani et al. (2021) Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R Arabnia. 2021.  A brief review of domain adaptation.  _Advances in data science and information engineering_ , pages 877–894.
  * Feder et al. (2022) Amir Feder, Katherine A. Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E. Roberts, Brandon M. Stewart, Victor Veitch, and Diyi Yang. 2022.  Causal inference in natural language processing: Estimation, prediction, interpretation and beyond.  _Trans. Assoc. Comput. Linguistics_ , 10:1138–1158.
  * Ganin et al. (2015) Yaroslav Ganin, E. Ustinova, Hana Ajakan, Pascal Germain, H. Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky. 2015.  Domain-adversarial training of neural networks.  In _Journal of machine learning research_.
  * Jiang and Zhai (2007) Jing Jiang and ChengXiang Zhai. 2007.  Instance weighting for domain adaptation in nlp.
  * Kaushik et al. (2020) Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2020.  Learning the difference that makes a difference with counterfactually-augmented data.  In _International Conference on Learning Representations_.
  * Kobayashi et al. (2020) Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020.  Attention is not only a weight: Analyzing transformers with vector norms.  _arXiv preprint arXiv:2004.10102_.
  * Kobayashi (2018) Sosuke Kobayashi. 2018.  Contextual augmentation: Data augmentation by words with paradigmatic relations.  _arXiv preprint arXiv:1805.06201_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv_ , abs/1907.11692.
  * Madaan et al. (2021) Nishtha Madaan, Inkit Padhi, Naveen Panwar, and Diptikalyan Saha. 2021.  Generate your counterfactuals: Towards controlled counterfactual generation for text.  In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_ , pages 13516–13524. AAAI Press.
  * Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022.  Locating and editing factual associations in GPT.  _Advances in Neural Information Processing Systems_ , 35.
  * Nan et al. (2021) Guoshun Nan, Jiaqi Zeng, Rui Qiao, Zhijiang Guo, and Wei Lu. 2021.  Uncovering main causalities for long-tailed information extraction.
  * Ng et al. (2020) Nathan Ng, Kyunghyun Cho, and Marzyeh Ghassemi. 2020.  SSMBA: Self-supervised manifold based data augmentation for improving out-of-domain robustness.
  * Penha et al. (2019) Gustavo Penha, Alex Bălan, and Claudia Hauff. 2019.  Introducing mantis: a novel multi-domain information seeking dialogues dataset.  _ArXiv_ , abs/1912.04639.
  * Prabhumoye et al. (2020) Shrimai Prabhumoye, Alan W Black, and Ruslan Salakhutdinov. 2020.  Exploring controllable text generation techniques.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 1–14, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Roark and Bacchiani (2003) Brian Roark and Michiel Bacchiani. 2003.  Supervised and unsupervised PCFG adaptation to novel domains.  pages 205–212.
  * Rush et al. (2012) Alexander M. Rush, Roi Reichart, Michael Collins, and Amir Globerson. 2012.  Improved parsing and pos tagging using inter-sentence consistency constraints.
  * Schnabel and Schütze (2014) Tobias Schnabel and Hinrich Schütze. 2014.  FLORS: Fast and simple domain adaptation for part-of-speech tagging.  _Transactions of the Association for Computational Linguistics_ , 2\.
  * Veitch et al. (2021) Victor Veitch, Alexander D’Amour, Steve Yadlowsky, and Jacob Eisenstein. 2021.  Counterfactual invariance to spurious correlations: Why and how to pass stress tests.  _ArXiv_ , abs/2106.00545.
  * Wang et al. (2019) Huazheng Wang, Zhe Gan, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, and Hongning Wang. 2019.  Adversarial domain adaptation for machine reading comprehension.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 2510–2520. Association for Computational Linguistics.
  * Wei and Zou (2019) Jason Wei and Kai Zou. 2019.  EDA: Easy data augmentation techniques for boosting performance on text classification tasks.  pages 6382–6388.
  * Wu et al. (2021) Tongshuang Wu, Marco Túlio Ribeiro, Jeffrey Heer, and Daniel S. Weld. 2021\.  Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models.  In _ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , pages 6707–6723. Association for Computational Linguistics.
  * Ziegler et al. (2019) Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. 2019.  Fine-tuning language models from human preferences.  _CoRR_ , abs/1909.08593.
  * Ziser and Reichart (2016) Yftah Ziser and Roi Reichart. 2016.  Neural structural correspondence learning for domain adaptation.  _ArXiv_ , abs/1610.01588.

##  Appendix A Appendix

###  A.1 Hyperparameters and Setups

#### Data Preprocessing

We follow previous work in data preprocessing and truncate each example to 96
tokens, using the HuggingFace T5-base tokenizer. The hyper-parameter was set
to 96 due to computation reasons and since the median number of words in the
labeled examples was 89. When an example is longer than 96 tokens, we keep the
first 96 tokens. For example from the Airline domain, before truncating, we
remove the first sentence since it mostly contains details about the flight
(like “from JPK to LAX”).

#### Heuristic Masking

We estimate $P(\mathcal{D}|w)$ for uni-grams, bi-grams and tri-grams which
appear in the unlabeled data in at least 10 examples. We use the NLTK Snowball
stemmer to stem each word of the n-grams. The smoothing hyperparameters in the
computation of $P(\mathcal{D}|w)$ are set to be 1, 5, and 7 for uni-grams, bi-
grams and tri-grams, respectively. We use a $\tau=0.08$ threshold and mask an
additional 5% of the training examples (in order to add noise between training
epochs). We set $\tau=0.08$ since it resulted in the successful domain
alternation of more than 80% examples.

#### Step 2 OTT

The domain classifier to get a domain-related score is based on the pretrained
HuggingFace distilroberta model. It has total of 6 layers and we use the
attention norm in the middle layer (layer 3, 4) to perform Over-The-Top
masking

#### Domain Specific Prompt

We choose four words representing the domain and initialize the domain-
specific prompt with their word embeddings. The following is the word we use
for each domain: Airline: airline, flight, seat, staff; DVD: dvd, character,
actor, plot; Electronics: electronics, ipod, router, software; Kitchen:
kitchen, dishwasher, pan, oven; and for the MANtIS dataset: Apple: apple,
itunes, iphone, nacbook; askubuntu: askubuntu, ubuntu, apt, deb; DBA: dba,
database, SQL, query; Electronics: electronics, schematics, voltage, circuit;
Physics: physics, gravity, particle, quantom; Statistics: stats, regression,
logits, variance;

#### Training of D-CONs generation model

The controllable model is based on a pre-trained HuggingFace T5-base model. We
train it on the unlabeled data for 20 epochs and pick the model whose
generated examples for an unlabeled held-out set are of the highest domain
accuracy (D.REL). 5 Training is performed with the AdamW optimizer learning
rate parameter of 5e-5 and a weight decay parameter of 1e-5.

#### Task Classifiers

Task classifiers are based on the Huggingface’s
RobertaForSequenceClassification model. We train the classifiers for 5 epochs
with a batch size of 16 and pick the best model based on the performance on
the validation set. Training is performed using the AdamW optimizer with
learning rate parameters of 5e-5 for the encoder blocks and of 5e-4 for the
linear layer and weight decay parameter of 1e-5.

###  A.2 Samples

In this section, we include masked text from each stage of our mask for our
generated D-CONS.
(1) Original, Airline
Was traveing with my partner, we got bored halfway through the flight as there
is no inflight entertainment system . hot food was served, overall a positive
experience .
Step 1, Airline $\rightarrow$ Electronics
Was traveing with my partner, <m> halfway through the <m> is no <m> . <m> was
<m> , <m> positive experience .
Step 2, Airline $\rightarrow$ Electronics
Was <m> with my <m>, <m> through the <m> is no <m> . <m> was <m> , <m>
positive <m> .
Step 3, Airline $\rightarrow$ Electronics
Was <m> with my partner, <m> through the <m> is no <m> system. <m> was <m> ,
overall a positive experience .

(2) Original, Airline
both legs departed and arrived in time . approx 75 % full with a mix of
business leisure and Disney visitors . buy on board food is reasonably priced
and the pre - order Irish breakfast was delicious . fare was just €100 return
. recommended .
Step 1, Airline $\rightarrow$ kitchen
both legs <m> and <m> in time . <m> 75 <m> a mix of <m> leisure and Disney
visitors . buy <m> is reasonably priced and the <m> irish <m> was <m> . <m>
was just <m> return . <m>
Step 2, Airline $\rightarrow$ kitchen
both <m> and <m> . <m> 75 <m> a mix of <m> and <m> . buy on <m> is reasonably
priced and the <m> was <m> . <m> was just <m> return . <m>
Step 3, Airline $\rightarrow$ kitchen
both <m> and <m> . <m> 75 % <m> a mix of <m> and <m> . buy on <m> is resonably
priced and the pre - order Irish <m> was <m> . <m> was just <m> return .
recommended

(3) Original, Dvd
Startrek Voyager : I am a very avid " star trek " fan , and find the dvd ’ s
very worthwhile and interesting . ( my star trek interests go back to the
original series with william shatner, leonard nimoy, james doohan, et. al.
Step 1, Dvd $\rightarrow$ Electronics
<m> Voyager : I am a very avid <m> " <m>, and find the <m> ’ s very worthwhile
and <m> . (my star <m> go back to <m> with <m> , <m> nimoy , <m> doohan <m> .
<m> .
Step 2, Dvd $\rightarrow$ Electronics
<m> : I am a very avid <m>, and <m> ’s <m> worthwhile and <m> . (my <m> go
back to <m> .
<m> : I am a very avid “ <m> “ <m> , and <m> ’ s <m> worthwhile and
interesting . ( my <m> go back to <m> .

(4) Original, Dvd
Tell all your friends! This is one of my favorite movies and i find it
unfortunate that not too many people know about it. i think it is important
for people to know this story. Everyone should watch this movie simply because
it is fantastic .
Step 1, Dvd $\rightarrow$ Kitchen Tell all your friends! This <m> of my <m>
and i find it unfortunate that not too many people know about it . i think it
is important for people to know this <m> . <m> this <m> simply because it is
fantastic .
Step 2, Dvd $\rightarrow$ Kitchen <m> friends ! This <m> of my <m> and i <m>
unfortunate that not too many <m> it. i think it is important for <m> . <m>
this <m> because it is <m> .
Step 3, Dvd $\rightarrow$ Kitchen Tell all your friends! This is one of my
favorite <m> and i find it unfortunate that not too many people know about it.
i think it is important for people to know this <m>. Everyone should <m> this
<m> simply because it is fantastic .

(5) Original, Electronics
Great service, cartridge like a new one ordering was fast and easy . could not
detect any inferiority in the cartridges and will order them again
Step 1, Electronics $\rightarrow$ Airline
Great service , cartridge like a <m> ordering was fast and easy . could not
<m> any inferiority in the cartridges and will order them again .
Step 2, Electronics $\rightarrow$ Airline
Great <m> , <m> like <m> was fast and <m> . could not <m> any <m> in the <m>
and will <m> again .
Step 3, Electronics $\rightarrow$ Airline
Great <m>, <m> like a new one <m> was fast and easy . could not <m> any
inferiority in the <m> and will order them again

(6) Original, Kitchen
Conducts heat really well this set conducts heat really well and everything
heats up fast but be careful not to burn anything which i did the first time
using it . also it takes a bit of scrubbing if you burn it bt other than that
it’s a great product
Step 1, Kitchen $\rightarrow$ Electronics
conducts <M> really well this <M> conducts <M> really well and everything <M>
up fast but be careful not to burn anything which i did the first time using
it . also it takes a bit of <M> if you burn it bt other than that it ’ s a
great <M>
Step 2, Kitchen $\rightarrow$ Electronics
<M> really well this <M> really well and everything <M> up fast but <m> to <m>
anything which i did the first time <m> . also it takes a bit of <M> if you
<m> it <m> other than that it ’ s a great <M>
Step 3, Kitchen $\rightarrow$ Electronics
<M> really well this <M> really well and everything <M> up fast but <m> to <m>
anything which i did the first time using it . also it takes a bit of <M> if
you <m> it <m> other than that it ’ s a great product


# Leveraging Task Dependency and Contrastive Learning for Case Outcome
Classification on European Court of Human Rights Cases

Santosh T.Y.S.S, Marcel Perez San Blas, Phillip Kemper, Matthias Grabmair
School of Computation, Information, and Technology;
Technical University of Munich, Germany
{santosh.tokala, marcel.perez, phillip.kemper<EMAIL_ADDRESS>
###### Abstract

We report on an experiment in case outcome classification on European Court of
Human Rights cases where our model first learns to identify the convention
articles allegedly violated by the state from case facts descriptions, and
subsequently uses that information to classify whether the court finds a
violation of those articles. We assess the dependency between these two tasks
at the feature and outcome level. Furthermore, we leverage a hierarchical
contrastive loss to pull together article-specific representations of cases at
the higher level, leading to distinctive article clusters. The cases in each
article cluster are further pulled closer based on their outcome, leading to
sub-clusters of cases with similar outcomes. Our experiment results
demonstrate that, given a static pre-trained encoder, our models produce a
small but consistent improvement in classification performance over single-
task and joint models without contrastive loss.

##  1 Introduction

The NLP task of classifying case outcome information from a textual statement
of case facts is generally referred to as Legal Judgement Prediction
(LJP)(e.g., Aletras et al. 2016; Chalkidis et al. 2019). It has been studied
using corpora from different jurisdictions, such as the European Court of
Human Rights (ECtHR) Chalkidis et al. (2019, 2021, 2022b); Aletras et al.
(2016); Medvedeva et al. (2020); Santosh et al. (2022, 2023), Chinese Criminal
Courts Luo et al. (2017); Zhong et al. (2018); Yue et al. (2021); Zhong et al.
(2020); Yang et al. (2019), US Supreme Court Katz et al. (2017); Kaufman et
al. (2019), Indian Supreme Court Malik et al. (2021); Shaikh et al. (2020),
the French court of Cassation Şulea et al. (2017), Brazilian courts Bertalan
and Ruiz (2020), the Federal Supreme Court of Switzerland Niklaus et al.
(2021), the Turkish Constitutional court Sert et al. (2021); Mumcuoğlu et al.
(2021), UK courts Strickson and De La Iglesia (2020), German courts Waltl et
al. (2017), the Philippine Supreme court Virtucio et al. (2018), and the
Thailand Supreme Court Kowsrihawat et al. (2018).

In this work, we experiment with classifying case outcomes in the ECtHR A and
B benchmark tasks introduced by LexGLUE Chalkidis et al. (2022b). Task B is to
identify the set of articles of the European Convention of Human Rights (ECHR)
that the claimant alleges to have been violated, while Task A’s goal is to
classify which of the convention’s articles has been deemed violated by the
court. The input for both tasks is the case’s fact description that has been
extracted from the published judgement document. It should be noted that,
despite these tasks being typically referred to as instances of ‘legal
judgement prediction’, the fact statements are typically not finalized until
the decision outcome is known, making the task effectively one of
retrospective classification rather than prediction Medvedeva et al. (2021).
While this does lead to distracting and confounding phenomena (see our prior
work in Santosh et al. 2022), the dataset remains a useful resource for the
development of NLP models that analyze these fact statements for text patterns
that correspond to specific convention articles as drafted by the court.
Consequently, we speak of case outcome classification (COC).

Positive instances for Task A are cases in which an article was deemed
violated by the court. Negative instances, however, usually encompass not only
cases in which that particular article was alleged and considered not
violated, but also the cases in which the particular article was not alleged
in the first place. Given that the conditional probability of a positive task
A label (violation) given its task B label (allegation) can be very high in
the LexGLUE dataset, we posit that models can fall into this pitfall of
identifying dominant articles with high conditional violation probability,
missing information specific to violation classification. Thus, we experiment
with multi-task architectures to decouple these reasoning steps involved in
Task A . This is similar to work in Chinese criminal cases judgement
prediction carried out in Zhong et al. 2018; Yang et al. 2019; Xu et al. 2020;
Yue et al. 2021 that coordinated multiple outcome variables (law articles,
criminal charges, and penalty terms).

In the ECtHR context, most of the previous works Chalkidis et al. (2019, 2021,
2022b); Clavié and Alphonsus (2021); Santosh et al. (2022, 2023) employed
independent models for allegation identification and violation classification.
Concurrent work by Valvoda et al. 2023 has recently explored multi-task joint
models that learn Task A and Task B simultaneously. Their _Claim-Outcome
model_ decomposes them into two independent classifiers with separate encoders
for each allegation and violation classification given allegation information.
Our work differs in three ways: (i) We learn the joint representation for both
tasks through a shared encoder following our intuition that the shared
representation is beneficial as both tasks require similar features at the
lower level; (ii) We model dependency from allegation to violation
classification at both the allegation outcome and feature levels. Conditioning
violation classification on the allegation outcome leads to higher
performance, but we find that passing along feature level information yields
additional improvement. This way, the violation branch will focus on
identifying specific information required for determining violation than
falling prey to only allegation level features. (iii) We also model
correlations among different articles that tend to be concurrently alleged.

Inspired by recent advances in contrastive learning to learn effective
representations Khosla et al. (2020), we further devise a two-level
hierarchical contrastive loss for COC. First, we strive to maximize the latent
space distance between different article representations, which assists the
model in learning distinct article-specific views of case facts. Second, we
apply contrastive learning within each article latent space to form distinct
sub-clusters of similar outcome cases. Unlike our two-level hierarchical
contrastive learning, a single-level one has been explored for LJP in the
concurrent work by Zhang et al. 2023.

Our experiments demonstrate that, given a static pre-trained encoder, our
models outperform single-task and joint models without contrastive loss by a
small but consistent margin, with larger improvements for sparse classes in
classification performance.

##  2 Method

Our model takes as input the case fact description as token encodings
$x=\\{x_{1},x_{2},\ldots,x_{m}\\}$ where
$x_{i}=\\{x_{i1},x_{i2},\ldots,x_{in}\\}$ and outputs the set of articles
claimed to be alleged by the applicant (Task B) and set of articles deemed to
be violated by the court (Task A) as multi-hot vectors. $x_{i}$ and $x_{ip}$
denote $i^{\text{th}}$ sentence and $p^{\text{th}}$ token of $i^{\text{th}}$
sentence of the case facts respectively. $m$ and $n$ denote the number of
sentences and tokens in the $i^{\text{th}}$ fact sentence, respectively. Our
model is built on top of a hierarchical attention network Yang et al. (2016),
which we chose to be able to process the very long fact description inputs.
Our model contains three main components: (1) encoding layer, (2) Article-
Specific Case representation and (3) Task Dependency learning. See Fig. 1 for
an overview of our model.

Figure 1: Our architecture capturing task dependency from allegation to
violation classification branch.

Encoding layer: We encode each sentence in the case facts description with
LegalBERT Chalkidis et al. (2020) to obtain the token level representations
$\\{z_{i1},z_{i2},\ldots,z_{in}\\}$. Sentence level representations are then
computed using a token-level attention mechanism as follows:

$\displaystyle
u_{it}=\text{tanh}(W_{w}z_{it}+b_{w})~{}~{}\&~{}~{}\alpha_{it}=\frac{\text{exp}(u_{it}u_{w})}{\sum_{t}\text{exp}(u_{it}u_{w})}~{}~{}\&~{}~{}f_{i}=\sum_{t=1}^{n}\alpha_{it}z_{it}$

(1)

where $W_{w}$, $b_{w}$ and $u_{w}$ are trainable parameters. The sentence
level representations $\\{f_{1},\ldots,f_{m}\\}$ are passed through a GRU
encoder to obtain context-aware representations of the case facts
$h=\\{h_{1},h_{2},\ldots,h_{m}\\}$.

Article-Specific Case Representation: We disentangle the input case facts into
multiple article-view-representations $c=\\{c_{1},c_{2},\ldots,c_{k}\\}$ where
$k$ is the total number of modeled convention articles, through aggregating
context-aware sentence-level representations $h$ using $k$ sentence-level
attention mechanisms similar to eq. 1 for every article individually. Through
this article-specific attention mechanism, sentences relevant to a specific
article are emphasized and intended to aid in fine-grained reasoning of
outcome prediction for every article. This is helpful especially for sparser
classes and mitigates the tendency of models to focus on skewed dominant
articles. This is distinct from previous works Chalkidis et al. (2019, 2021,
2022b); Santosh et al. (2022) which use a single vector representation for
case facts.

Task Dependency Learning: To capture both the inter-article correlations for
allegation, violation classification, and the inter-task dependency, we again
use a two-step architecture. First, we apply a multi-head self-attention layer
Vaswani et al. (2017) to the obtained article-specific representations of case
$c$ to allow interactions among articles (e.g., some articles are typically
alleged together while others usually occur in isolation) and obtain article-
interaction-aware allegation feature representations
$c^{\prime}_{b}=\\{c^{\prime}_{b1},c^{\prime}_{b2},\ldots,c^{\prime}_{bk}\\}$.
These $c^{\prime}_{b}$ are passed through $k$ article-specific Task B
classification layers to obtain the binary outcome $o_{b}$ corresponding to
each article for allegation classification, which are then concatenated into a
multi-hot vector.

Then we utilize the task B allegation information to enhance task A violation
classification to capture the task dependency. We concatenate the obtained
article-interaction-aware allegation representations $c^{\prime}_{b}$ and
allegation label probability logits $o^{\prime}_{b}$ 111We use non-binarized
response (i.e. probability logits) as it avoids the information loss that can
occur in the probability space due to binarization. with article-specific case
representations $c$ to obtain the enhanced article-specific representation for
Task A as $c_{ai}=[c_{i},c^{\prime}_{bi},o^{\prime}_{bi}]$. We again employ
multi-head self-attention mechanism on these enhanced article-aware
representations $c_{a}$ for violation prediction to capture the correlations
that exist between violation of different articles. Finally, the obtained
representations are passed through $k$ article-specific Task A classification
layers to obtain the binary outcome $o_{a}$ corresponding to each article’s
violation output. To be able to evaluate the effectiveness of our interaction
architecture in a clean way, we freeze the LegalBERT encoder weights in all
our experiments.

###  2.1 Hierarchical Contrastive Loss

Contrastive learning has recently gained attention as a technique to obtain
effective representations. In essence, it involves pulling together an ‘anchor
point’ and its related samples while pushing it away from unrelated samples in
the embedding space. Originally developed in self-supervised learning Chen et
al. (2020); Henaff (2020), it has since been adopted in supervised settings
Khosla et al. (2020) where samples with the same/different labels are deemed
related/unrelated with respect to an anchor.

In this work, we use a hierarchical contrastive loss Liang et al. (2022)
alongside the standard binary cross entropy loss on the task outcome
probability. On the higher level, this is intended to form distinctive
clusters of article-specific case representations. We hypothesize that this
distinctiveness maximization constraint in turn helps the article-specific
representation component to extract salient information with respect to each
article more effectively. At the lower level, inside the latent space for each
article, we further perform contrastive learning among cases based on their
outcome for tasks A and B, respectively. This allows the positive outcome
representations of cases under a specific article to stay closer and separate
from the negative outcome cases, leading to formation of sub-clusters. We
apply contrastive losses for Task A and Task B separately on the interaction-
aware representations. It is calculated as the mean loss computed based on
every article based representation as an anchor point. The loss for the
interaction-aware representation of the $j^{\text{th}}$ case specific to the
$i^{\text{th}}$ article for Task B $c^{\prime}_{bi_{j}}$ as an anchor point is
calculated as follows:

$\displaystyle\begin{aligned} l(c_{bi_{j}}^{\prime})=&\text{log}(\sum_{f\in
Q(i,j)}\frac{\text{exp}(c^{\prime}_{bi_{j}}\cdot f/\tau_{a})}{\sum_{g\in
P(i,j)}\text{exp}(c^{\prime}_{bi_{j}}\cdot g/\tau_{a})})\\\
&\times\alpha\sum_{f\in R(i,j)}\frac{\text{exp}(c^{\prime}_{bi_{j}}\cdot
f/\tau_{c})}{\sum_{g\in Q(i,j)}\text{exp}(c^{\prime}_{bi_{j}}\cdot
g/\tau_{c})})\end{aligned}$

(2)

where $P(i,j)=\\{c^{\prime}_{bl_{m}}|l\neq i~{}~{}\&~{}~{}m\neq j\\}$ (all
representations except the anchor point),
$Q(i,j)=\\{c^{\prime}_{bl_{m}}|l=i~{}~{}\&~{}~{}m\neq j)\\}$ (all
representations which share the same article as the anchor point),
$R(i,j)=\\{c^{\prime}_{bl_{m}}|l=i~{}~{}\&~{}~{}y_{i_{j}}=y_{l_{m}}~{}~{}\&~{}~{}m\neq
j\\}$, $y_{i_{j}}$ the denotes binary outcome label of case $j$ with respect
to article $i$ (all representations which share the same article and outcome
of that article as the anchor point). $\tau_{a}$ and $\tau_{c}$ are scalar
temperature parameters that control the penalties on negative samples. The
first term in the above equation 2 denotes the contrastive loss among article
representations (i.e., for a given anchor point, positive pairs are obtained
by article-aware representations of cases in the batch which share the same
article with the anchor point; negative pairs are the remaining article
representations). The second term contrasts the cases based on their task
outcome within specific article representations. Contrastive learning has
shown to be more effective with larger batch sizes Radford et al. (2021); Chen
et al. (2020). To account for smaller batch size due to computational
constraints incurred with our hierarchical setup, we use a memory bank Wu et
al. (2018) which progressively reuses the representations from previous
batches in computing the contrastive loss.

##  3 Experiments & Discussion

###  3.1 Dataset & Baseline Models

We experiment on the ECHR task A and B datasets of LexGLUE Chalkidis et al.
(2022b), which consist of 11k case fact descriptions chronologically split
into training (2001–2016, 9k cases), validation (2016–2017, 1k cases), and
test sets (2017-2019, 1k cases). The label set for both tasks include 10
prominent ECHR articles. Implementation details are given in Appendix A. We
compare our models against the following baselines:

Task A/B only: We train single-task models only on task A and B labels,
respectively. Their architecture is based on our model with contrastive loss,
task dependency and article-specific representation components removed.

Multi-task: Using the same architecture, we also develop a model for both task
A and B that is trained in classic multi-task fashion with separate
classification heads without article-specific representations and task
dependency components.

###  3.2 Performance Evaluation

Following Chalkidis et al. 2022b, we report micro-F1 ($\mu$-F1) and macro-F1
(m-F1) scores across the 10 ECtHR articles contained in the dataset. We also
report hard-macro-F1 (hm-F1) for Task A following Santosh et al. 2022, which
is the mean F1-score computed for each article where cases with that article
having been violated are considered as positive instances, and cases with that
article being alleged but not found to have been violated as negative
instances. This forms the most critical measure for violation classification
as it conditions on allegation.

We compare the performance of our method with the baselines in the first four
rows of Table 1. The multi-task method performs better than the individual
tasks alone, validating the dependency between them. Our method performs
better than the multi-task architecture, highlighting the effectiveness of our
feature and label dependency components, as well as of the hierarchical
contrastive loss. It scores higher by a small margin in Task B and Task A
micro-F1 but achieves a larger benefit (>5%) in task A macro-F1 and hard
macro-F1 metrics. This suggests that our model is of particular utility for
sparser classes.

Table 1: Results on Task A and B. ‘feat.’, ‘lab,’. ‘Cont.’ indicate feature,
label and contrastive, respectively

| Task B | Task A
---|---|---
Model |  $\mu$-F1 |  m-F1 |  $\mu$-F1 |  m-F1 |  hm-F1
Task B only |  76.20 |  67.15 |  - |  - |  -
Task A only |  - |  - |  68.42 |  56.26 |  54.14
Multi-task |  78.17 |  69.16 |  69.29 |  58.05 |  55.57
Our Method |  79.29 |  70.97 |  71.26 |  65.24 |  60.90
w/o feat. & lab. |  78.93 |  71.45 |  70.07 |  59.14 |  57.09
w/o feat. |  78.59 |  71.56 |  70.68 |  63.93 |  59.28
w/o label |  79.09 |  71.38 |  70.32 |  64.12 |  61.70
only gold lab. |  78.21 |  70.03 |  81.46 |  78.93 |  66.59
gold lab. + feat. |  77.68 |  70.40 |  83.19 |  78.79 |  67.42
w/o outcome Cont. |  78.42 |  69.48 |  69.86 |  60.84 |  57.62
w/o article Cont. |  79.02 |  71.14 |  71.16 |  64.68 |  59.86
Task A Cont. |  - |  - |  70.16 |  62.14 |  58.12
Task B Cont. |  78.16 |  69.42 |  - |  - |  -

###  3.3 Analysis

####  3.3.1 Ablation on Task Dependency Learning

We create three ablation conditions by removing one interaction mechanism in
each of them: (i) _w/o feature & label_ (article-specific representation but
with no concatenation with features or labels) (ii) _w/o feature_ (article-
specific representation concatenated with task B label as classified by the
model) and (iii) _w/o label_ (article-specific representation concatenated
with task B features).

From the second section of Table 1, we observe that even the performance of
_w/o feature and label_ is better than _multi-task_ model indicated by a small
but consistent margin, indicating that article-specific case representation is
a competitive component. Both _w/o feature_ , _w/o label_ models perform
better than _w/o feature & label_ in Task A, demonstrating the benefit of
passing on task B model information explicitly. Between them, _w/o label_
performs better in m-F1 and hm-F1 scores of Task A than _w/o feature_ showing
that providing the fine-grained representation of features from the allegation
identification model is more useful than the predicted allegation labels only.

Further, to evaluate the impact of allegation identification performance on
downstream violation classification, we conduct a control experiment in which
we provide the actual gold allegation set of articles as input to the
violation component in place of the predicted ones. We create two variants:
_only gold labels_ and _gold labels + features_ (gold labels along with task B
features). From the third section of Table 1, we observe that performance on
Task A increases substantially in micro-F1 and macro-F1, which is intuitive as
the model gets access to perfect allegation information. Interestingly, when
adding task allegation feature information, we notice an additional small
increase in hard macro-F1, indicating that adding feature information can have
beneficial effect even in the presence of gold task B labels. These results
form an upper bound of the benefit of using task dependency information from
allegation branch to violation branch in our architecture, and the size of the
performance gap motivates future work on accurate allegation classification.

####  3.3.2 Ablation Hierarchical Contrastive Loss

We carry out an ablation experiment for each component of our hierarchical
contrastive loss: (i) disable article-level contrastive learning to obtain
distinctive article representations (‘w/o article contrastive’) and (ii)
disable outcome-based contrastive learning to separate cases based on outcome
within each article cluster (‘w/o outcome contrastive’). Intuitively, from the
fourth section of Table 1, we observe that removing the outcome contrastive
loss has the larger effect on performance, as it directly relates to the
predicted label. The smaller but consistent drop in performance when removing
article-level contrastive learning supports our hypothesis that maximizing the
distinctiveness among article representations encourages the model to learn
how to extract article-specific salient information from case facts.

Finally, to study the impact of our contrastive loss component alone, we
evaluate single-task models applying our hierarchical contrastive loss
referred to as ‘Task A/B Contrastive’. From the last section of Table 1, we
observe that they outperform their baseline counterparts in both tasks, but
still stay behind our model.

##  4 Conclusion

We improve ECtHR article violation classification from fact statements by
leveraging feature and label information from an allegation classification
model. We also leveraged hierarchical contrastive loss to contrast between
different article representations and case representations based on outcome
with respect to a specific article. Given a static pre-trained encoder, our
models outperform a straightforward multi-task architecture by a small but
consistent margin, with larger improvements for sparse classes. These results
suggest that the tasks of allegation and violation classification on ECtHR
fact statements interrelate in a way that may not be optimally captured using
straightforward multi-task architectures, and motivate further research on
dependency modeling between related legal classification tasks.

## Limitations

In this work, we have demonstrated improvements in violation classification
for ECtHR cases by leveraging the dependency from allegation classification at
the feature and label level. While our feature transfer and contrastive
learning techniques are general, our experimental contributions are contextual
to the court. The nature of the fact statements as being influenced by the
eventual case outcome and not suitable for prospective prediction has already
been addressed in section 1.

While similar tasks (i.e., allegations paired with findings) exist in many
other jurisdictions, they will differ in, for example, legal nature, semantic
difficulty, and sub-task dependency. In our case, we choose the allegation
classification as the auxiliary task which is closely related and also forms a
necessary sub-part of deriving the final outcome related to the main task
(violation prediction) at hand. We leave an exploration of the relatedness of
auxiliary tasks and their impact on LJP/COC for future work.

One major challenge is dealing with long input case facts description, which
is currently handled with hierarchical model in this work. These hierarchical
models do not allow tokens in one sentence to attend to tokens in other
sentence which leads to sub-optimal interaction modelling. This modeling
impact on performance is still underexplored except some preliminary empirical
work in Dai et al. 2022; Chalkidis et al. 2022a. Additionally, we freeze the
weights in the LegalBERT encoder, both to save computational resources and to
evaluate the effectiveness of our dependency mechanism in a clean way.

## Ethics Statement

We employ pre-trained language models and do not train them from scratch, thus
inheriting the biases they may have acquired from their training corpus. Our
experiments have been carried out on a dataset of ECtHR decisions which is
publicly available as the part of LexGLUE benchmark Chalkidis et al. (2022b)
and has been derived from the public court database
HUDOC222https://hudoc.echr.coe.int. Though these decisions are not anonymized
and contain the real names of the involved parties, we do not foresee any harm
incurred by our experiments beyond making this information available. This
collection of decision documents is of course historical data and using it to
train model may result in classifiers that exhibit behavior that may be
considered biased. For example, Chalkidis et al. 2022c explores disparities in
classification performance with regard to an applicant’s gender, their age,
and the identity of the respondent state. If COC models are deployed as part
of a decision support systems, then they of course must be screened for
performance/error differences in between groups that are to be treated
equally.

The task of LJP/COC in itself raises serious ethical and legal concerns, both
in general and specific to the European Court of Human Rights. However, we do
not advocate for the practical adoption of LJP/COC systems by courts. Our
prior work in Santosh et al. 2022 demonstrates that these systems rely on
several shallow surface-level spurious signals that are statistically
predictive but legally irrelevant. This highlights the risk of using
predictive systems in high stakes domains such as law. In the same work, we
argue that models leveraging the case outcome signal for analytical purposes
must be developed mindfully and with the goal of aligning their inferences
with legal expert reasoning. This further parallels the broader legal NLP
community increasingly addressing ethical aspects of developed systems in the
context of technical research (e.g., Wang et al. 2021; Medvedeva et al. 2021,
2022; Tsarapatsanis and Aletras 2021; Leins et al. 2020).

In this work, we use COC as a technical benchmarking task that allows the
development and study of neural NLP models on legal text. We focus on how to
leverage dependencies on two successive tasks (allegation identification and
violation classification) based on case facts, as well as on learning
effective representations of these facts using contrastive learning. Our
results are hence to be understood as technical contributions in pursuit of
the overarching goal of developing models capable of deriving insight from
data that can be used legally, ethically, and mindfully by experts in solving
problems arising in legal research and practice.

All experiments were carried out using Google Colab. We did not track
computation hours.

## References

  * Aletras et al. (2016) Nikolaos Aletras, Dimitrios Tsarapatsanis, Daniel Preoţiuc-Pietro, and Vasileios Lampos. 2016.  Predicting judicial decisions of the european court of human rights: A natural language processing perspective.  _PeerJ Computer Science_ , 2:e93.
  * Bertalan and Ruiz (2020) Vithor Gomes Ferreira Bertalan and Evandro Eduardo Seron Ruiz. 2020.  Predicting judicial outcomes in the brazilian legal system using textual features.  In _DHandNLP@ PROPOR_ , pages 22–32.
  * Chalkidis et al. (2019) Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. 2019.  Neural legal judgment prediction in english.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4317–4323.
  * Chalkidis et al. (2022a) Ilias Chalkidis, Xiang Dai, Manos Fergadiotis, Prodromos Malakasiotis, and Desmond Elliott. 2022a.  An exploration of hierarchical attention transformers for efficient long document classification.  _arXiv preprint arXiv:2210.05529_.
  * Chalkidis et al. (2020) Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020.  Legal-bert: The muppets straight out of law school.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2898–2904.
  * Chalkidis et al. (2021) Ilias Chalkidis, Manos Fergadiotis, Dimitrios Tsarapatsanis, Nikolaos Aletras, Ion Androutsopoulos, and Prodromos Malakasiotis. 2021.  Paragraph-level rationale extraction through regularization: A case study on european court of human rights cases.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 226–241.
  * Chalkidis et al. (2022b) Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. 2022b.  Lexglue: A benchmark dataset for legal language understanding in english.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4310–4330.
  * Chalkidis et al. (2022c) Ilias Chalkidis, Tommaso Pasini, Sheng Zhang, Letizia Tomada, Sebastian Schwemer, and Anders Søgaard. 2022c.  Fairlex: A multilingual benchmark for evaluating fairness in legal text processing.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4389–4406.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR.
  * Clavié and Alphonsus (2021) Benjamin Clavié and Marc Alphonsus. 2021.  The unreasonable effectiveness of the baseline: Discussing svms in legal text classification.  In _Legal Knowledge and Information Systems_ , pages 58–61. IOS Press.
  * Dai et al. (2022) Xiang Dai, Ilias Chalkidis, Sune Darkner, and Desmond Elliott. 2022.  Revisiting transformer-based models for long document classification.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 7212–7230, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Henaff (2020) Olivier Henaff. 2020.  Data-efficient image recognition with contrastive predictive coding.  In _International conference on machine learning_ , pages 4182–4192. PMLR.
  * Katz et al. (2017) Daniel Martin Katz, Michael J Bommarito, and Josh Blackman. 2017.  A general approach for predicting the behavior of the supreme court of the united states.  _PloS one_ , 12(4):e0174698.
  * Kaufman et al. (2019) Aaron Russell Kaufman, Peter Kraft, and Maya Sen. 2019.  Improving supreme court forecasting using boosted decision trees.  _Political Analysis_ , 27(3):381–387.
  * Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020.  Supervised contrastive learning.  _Advances in Neural Information Processing Systems_ , 33:18661–18673.
  * Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba. 2015.  Adam: a method for stochastic optimization 3rd int.  In _International Conference on Learning Representations_.
  * Kowsrihawat et al. (2018) Kankawin Kowsrihawat, Peerapon Vateekul, and Prachya Boonkwan. 2018.  Predicting judicial decisions of criminal cases from thai supreme court using bi-directional gru with attention mechanism.  In _2018 5th Asian Conference on Defense Technology (ACDT)_ , pages 50–55. IEEE.
  * Leins et al. (2020) Kobi Leins, Jey Han Lau, and Timothy Baldwin. 2020.  Give me convenience and give her death: Who should decide what uses of nlp are appropriate, and on what basis?  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2908–2913.
  * Liang et al. (2022) Bin Liang, Zixiao Chen, Lin Gui, Yulan He, Min Yang, and Ruifeng Xu. 2022.  Zero-shot stance detection via contrastive learning.  In _Proceedings of the ACM Web Conference 2022_ , pages 2738–2747.
  * Luo et al. (2017) Bingfeng Luo, Yansong Feng, Jianbo Xu, Xiang Zhang, and Dongyan Zhao. 2017.  Learning to predict charges for criminal cases with legal basis.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2727–2736.
  * Malik et al. (2021) Vijit Malik, Rishabh Sanjay, Shubham Kumar Nigam, Kripabandhu Ghosh, Shouvik Kumar Guha, Arnab Bhattacharya, and Ashutosh Modi. 2021.  Ildc for cjpe: Indian legal documents corpus for court judgment prediction and explanation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4046–4062.
  * Medvedeva et al. (2021) Masha Medvedeva, Ahmet Üstün, Xiao Xu, Michel Vols, and Martijn Wieling. 2021.  Automatic judgement forecasting for pending applications of the european court of human rights.  In _ASAIL/LegalAIIA@ ICAIL_.
  * Medvedeva et al. (2020) Masha Medvedeva, Michel Vols, and Martijn Wieling. 2020.  Using machine learning to predict decisions of the european court of human rights.  _Artificial Intelligence and Law_ , 28(2):237–266.
  * Medvedeva et al. (2022) Masha Medvedeva, Martijn Wieling, and Michel Vols. 2022.  Rethinking the field of automatic prediction of court decisions.  _Artificial Intelligence and Law_ , pages 1–18.
  * Mumcuoğlu et al. (2021) Emre Mumcuoğlu, Ceyhun E Öztürk, Haldun M Ozaktas, and Aykut Koç. 2021.  Natural language processing in law: Prediction of outcomes in the higher courts of turkey.  _Information Processing & Management_, 58(5):102684.
  * Niklaus et al. (2021) Joel Niklaus, Ilias Chalkidis, and Matthias Stürmer. 2021.  Swiss-judgment-prediction: A multilingual legal judgment prediction benchmark.  In _Proceedings of the Natural Legal Language Processing Workshop 2021_ , pages 19–35.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pages 8748–8763. PMLR.
  * Santosh et al. (2023) T. Y. S. S Santosh, Oana Ichim, and Matthias Grabmair. 2023.  Zero shot transfer of article-aware legal outcome classification for european court of human rights cases.  _arXiv preprint arXiv:2302.00609_.
  * Santosh et al. (2022) T.y.s.s Santosh, Shanshan Xu, Oana Ichim, and Matthias Grabmair. 2022.  Deconfounding legal judgment prediction for European court of human rights cases towards better alignment with experts.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 1120–1138, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Sert et al. (2021) Mehmet Fatih Sert, Engin Yıldırım, and İrfan Haşlak. 2021.  Using artificial intelligence to predict decisions of the turkish constitutional court.  _Social Science Computer Review_ , page 08944393211010398.
  * Shaikh et al. (2020) Rafe Athar Shaikh, Tirath Prasad Sahu, and Veena Anand. 2020.  Predicting outcomes of legal cases based on legal factors using classifiers.  _Procedia Computer Science_ , 167:2393–2402.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 15(1):1929–1958.
  * Strickson and De La Iglesia (2020) Benjamin Strickson and Beatriz De La Iglesia. 2020.  Legal judgement prediction for uk courts.  In _Proceedings of the 2020 the 3rd international conference on information science and system_ , pages 204–209.
  * Şulea et al. (2017) Octavia-Maria Şulea, Marcos Zampieri, Mihaela Vela, and Josef van Genabith. 2017.  Predicting the law area and decisions of french supreme court cases.  In _Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017_ , pages 716–722.
  * Tsarapatsanis and Aletras (2021) Dimitrios Tsarapatsanis and Nikolaos Aletras. 2021.  On the ethical limits of natural language processing on legal text.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 3590–3599.
  * Valvoda et al. (2023) Josef Valvoda, Ryan Cotterell, and Simone Teufel. 2023.  On the role of negative precedent in legal outcome prediction.  _Transactions of the Association for Computational Linguistics_ , 11:34–48.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Virtucio et al. (2018) Michael Benedict L Virtucio, Jeffrey A Aborot, John Kevin C Abonita, Roxanne S Avinante, Rother Jay B Copino, Michelle P Neverida, Vanesa O Osiana, Elmer C Peramo, Joanna G Syjuco, and Glenn Brian A Tan. 2018.  Predicting decisions of the philippine supreme court using natural language processing and machine learning.  In _2018 IEEE 42nd annual computer software and applications conference (COMPSAC)_ , volume 2, pages 130–135. IEEE.
  * Waltl et al. (2017) Bernhard Waltl, Georg Bonczek, Elena Scepankova, Jörg Landthaler, and Florian Matthes. 2017.  Predicting the outcome of appeal decisions in germany’s tax law.  In _International conference on electronic participation_ , pages 89–99. Springer.
  * Wang et al. (2021) Yuzhong Wang, Chaojun Xiao, Shirong Ma, Haoxi Zhong, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2021.  Equality before the law: Legal judgment consistency analysis for fairness.  _arXiv e-prints_ , pages arXiv–2103.
  * Wu et al. (2018) Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. 2018.  Unsupervised feature learning via non-parametric instance discrimination.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3733–3742.
  * Xu et al. (2020) Nuo Xu, Pinghui Wang, Long Chen, Li Pan, Xiaoyan Wang, and Junzhou Zhao. 2020.  Distinguish confusing law articles for legal judgment prediction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3086–3095.
  * Yang et al. (2019) Wenmian Yang, Weijia Jia, Xiaojie Zhou, and Yutao Luo. 2019.  Legal judgment prediction via multi-perspective bi-feedback network.  In _Proceedings of the 28th International Joint Conference on Artificial Intelligence_ , pages 4085–4091.
  * Yang et al. (2016) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016\.  Hierarchical attention networks for document classification.  In _Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies_ , pages 1480–1489.
  * Yue et al. (2021) Linan Yue, Qi Liu, Binbin Jin, Han Wu, Kai Zhang, Yanqing An, Mingyue Cheng, Biao Yin, and Dayong Wu. 2021.  Neurjudge: a circumstance-aware neural framework for legal judgment prediction.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 973–982.
  * Zhang et al. (2023) Han Zhang, Zhicheng Dou, Yutao Zhu, and Ji-Rong Wen. 2023.  Contrastive learning for legal judgment prediction.  _ACM Transactions on Information Systems_.
  * Zhong et al. (2018) Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. 2018.  Legal judgment prediction via topological learning.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3540–3549.
  * Zhong et al. (2020) Haoxi Zhong, Yuzhong Wang, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020.  Iteratively questioning and answering for interpretable legal judgment prediction.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 1250–1257.

##  Appendix A Implementation Details

Our models compute word embeddings of size 768. Our word level attention
context vector size is 300. The sentence level GRU encoder dimension is 200,
thus giving a bidirectional embedding of size 400, and a sentence level
attention vector dimension of 200. The final dense classifier for all tasks
has 100 hidden units. We use a mini batches size of 32 and the model is
optimized end-to-end using Adam Kingma and Ba (2015). The dropout rate
Srivastava et al. (2014) in all layers is 0.1. We determine the best learning
rate using grid search on the development set and use early stopping based on
the development set m-F1 score. We finetuned $\tau_{a}$, $\tau_{c}$ with an
additional constraint of $\tau_{a}$ < $\tau_{c}$ among the values of
$\\{0.07,0.1,0.14,0.2,0.25,0.3\\}$ so that it aids in pulling together the
representations belonging to the same article in latent space (leading to
distinct article clusters) and also in further slightly pulling together the
representations of cases belonging to the same outcome in each separated
article-specific embedding latent space compared to the other outcome cases in
that same article. We set $\alpha$ to be 0.5. We use a memory bank of size 32
per article and outcome, and store only the most recent examples per article
and its corresponding outcome.