# On Text Style Transfer via Style Masked Language Models

Sharan Narasimhan, Pooja Shekar, Suvodip Dey, Maunendra Sankar Desarkar
Department of Computer Science and Engineering
Indian Institute of Technology Hyderabad, India

{sharan.n21<EMAIL_ADDRESS>
{cs19resch01003<EMAIL_ADDRESS>
###### Abstract

Text Style Transfer (TST) is performable through approaches such as latent
space disentanglement, cycle-consistency losses, prototype editing etc. The
prototype editing approach, which is known to be quite successful in TST,
involves two key phases a) Masking of source style-associated tokens and b)
Reconstruction of this source-style masked sentence conditioned with the
target style. We follow a similar transduction method, in which we transpose
the more difficult direct source to target TST task to a simpler Style-Masked
Language Model (SMLM) Task, wherein, similar to BERT Devlin et al. (2019a),
the goal of our model is now to reconstruct the source sentence from its
style-masked version. We arrive at the SMLM mechanism naturally by formulating
prototype editing/ transduction methods in a probabilistic framework, where
TST resolves into estimating a hypothetical parallel dataset from a partially
observed parallel dataset, wherein each domain is assumed to have a common
latent style-masked prior. To generate this style-masked prior, we use
“Explainable Attention” as our choice of attribution for a more precise style-
masking step and also introduce a cost-effective and accurate “Attribution-
Surplus” method of determining the position of masks from any arbitrary
attribution model in O(1) time. We empirically show that this non-generational
approach well suites the “content preserving” criteria for a task like TST,
even for a complex style like Discourse Manipulation. Our model, the Style
MLM, outperforms strong TST baselines and is on par with state-of-the-art TST
models, which use complex architectures and orders of more parameters.

##  1 Introduction

Text Style Transfer (TST) can be thought of as a subset of the Controllable
Language Generation Task Hu et al. (2017) with tighter criteria. TST involves
converting input text possessing some source style into an output possessing
the target style, meanwhile preserving style-independent content and
maintaining fluency. Style is usually defined by the class labels present in
an annotated dataset, commonly considered to be Sentiment, Formality, Toxicity
etc. We consider the unsupervised setting where only non-parallel datasets are
available, i.e. the style-transferred output is not available. Past work
focuses on various common paradigms such as disentanglement Hu et al. (2017);
Shen et al. (2017), cycle-consistency losses Yi et al. (2020); Luo et al.
(2019); Dai et al. (2019); Liu et al. (2021), induction Narasimhan et al.
(2022); Shen et al. (2020) etc. We focus on a sentence to sentence
“transduction” (or prototype editing) method, a solution which naturally
emerges when following a probabilistic formulation consisting of a single
transduction model with a latent prior consisting of a style-absent corpus.
Our approach is also inspired by BERT’s MLM Devlin et al. (2019a)111A key
difference to BERT, ours is not using a self-supervised approach of pre-
training over large corpora and fine-tuning for a downstream task., modified
to incorporate style information to enable TST. This notion can be seen from
various viewpoints.

Task |  Positive to Negative |  Contradiction to Entailment
---|---|---
Input |  this movie is by far one of the best urban crime dramas i ’ve seen . |  a woman is sitting outside at a table using a knife to cut into a sandwich . a woman is sitting inside
Style Masked |  this movie is by mask one of the mask urban crime mask i ’ve seen . |  a woman is sitting outside at a table using a knife to cut into a sandwich . a woman mask mask mask
Output |  this movie is by far one of the worst urban crime garbage i ’ve seen . |  a woman is sitting outside at a table using a knife to cut into a sandwich . a woman is a outside
Table 1: Examples of Sentiment and Discourse TST by the SMLM on the IMDb and
SNLI datasets respectively.

Another MLM. By making use of various text attribution models, it is possible
to determine the fraction of tokens that significant contribution to the style
label. Masking out these tokens and training a bi-directional transformer
encoder with self-attention with the task of reconstructing the original
source style sentence is intuitive and closely resembles BERT-based MLM’s
Devlin et al. (2019a); Jin et al. (2020). Instead of random masking or
perplexity based masking Jin et al. (2020), by incorporating style-masking and
concatenation of control codes that inform the encoder about the target style
needed, the resultant MLM now resembles a rough TST model. This notion of
"filling in only the style words" also does well to automatically ensure that
other content words are preserved, and fluency of the original sentence is
maintained. Unlike most generational approaches, this method has an easier
objective by inferring what to fill in by looking at both directions, similar
to BERT.

Parallel vs Non Parallel. A "parallel" dataset, in which every instance in the
dataset has associated sentences with all given style labels, is naturally
much easier to salvage but harder to come by. In this case, TST resolves into
the supervised task of simply learning the input distribution corresponding to
this parallel dataset. In reality, most available datasets are non-
parallel/unsupervised (E.g. Yelp). TST models must infer the notion of "Style"
present in the dataset using non-aligned sentences of the same styles,
differing in content, with no direct supervision signal.
A Hypothetical Parallel. In this case, TST resolves into the task of
estimating the missing portion of a hypothetical parallel dataset without
explicit knowledge of the complete input distribution as in the parallel case.
We adopt a probabilistic formulation of TST, in which we assume each domain in
this hypothetical parallel dataset originates from a common discrete latent
prior, a latent style-masked prior in the case of SMLM. Salvaging a non-
parallel dataset to learn a simple "Same-Style Reconstruction" task using the
SMLM also indirectly approximates the cross-domain TST task, which we only
need to fine-tune for a single epoch to make it comparable to larger SOTA
models.

Contributions. a) We introduce the Style Masked Langauge Model (SMLM), another
type of MLM capable of being of strong TST model by initially pretrained on an
unsupervised same-style reconstruction task and fine-tuned for TST metrics. b)
We use an accurate and effective style masking step and present an analysis to
support our choice. c) We empirically show that the SMLM beats strong
baselines and compares to state of the art, using far lesser parameters. d) We
adopt a novel "Discourse" TST task by salvaging NLI datasets and show that our
style-masking is also capable of working with more complex "Inter-sentence’
styles to manipulate the flow of logic.

##  2 Related Work

Jin et al. (2021) and Hu et al. (2020) provide surveys detailing the current
state of TST and lay down useful taxonomies to structure the field. In this
section, we only discuss recent work similar to ours (a prototype editing
approach) assuming the same unsupervised setting.

Li et al. (2018) present the earliest known work using the prototype editing
method, in which a "delete" operation is performed on tokens based on simple
count-based methods, and the retrieval of the target word is done by
considering TF-IDF weighted word overlap. Malmi (2020) first train MLMs for
the source and target domains and perform TST by first masking text spans
where the models disagree (in terms of perplexity) the most, and use the
target domain MLM to fill these spans. Wu et al. (2019b) introduce the
Attribute-Conditional MLM, which most closely aligns to the working of the
MLM, also uses an attention classifier for attribution scores, a count and
frequency-based method to perform masking, and a pretrained BERT model fine-
tuned on the TST task. Lee (2020) and Wu et al. (2020) also follows roughly
the same pipeline but uses a generational transformer encoder-decoder approach
and also fine-tunes using signals from a pretrained classifier. Wu et al.
(2019a) uses a hierarchical reinforced sequence operation method is used to
iteratively revise the words of original sentences. Madaan et al. (2020) uses
n-gram TF-IDF based methods to identify style tokens and modify them as either
"add" or "replace" TAG tokens, which are then substituted by the decoder to
perform TST. Similar to the SMLM, Xu et al. (2018) also uses attribution-based
methods from a self-attention classifier. However, they use an LSTM Hochreiter
and Schmidhuber (1997) based approach, one to generate sentences from each
domain.

##  3 Method

###  3.1 Notation

Let $S$ denote the set of all style labels for a supervised dataset $D$ of the
form $\\{(x_{0},l_{0}),(x_{1},l_{1})\dots(x_{n},l_{n})\\}$ where $x_{i}$
denotes the input sentence and $l_{i}\in S$ denotes the label corresponding to
$x_{i}$. The set of all sentences of style $s$ in $D$ is denoted by
$\hat{x^{s}}=\\{x_{j}:\forall{j}\ where(x_{j},s)\in{D}\\}$. We use a special
meta label $m_{s}$ to represents the "style-masked" label class having $s$ the
original style. Subsequently, $x^{m_{s}}$ refers to the set of all style-
masked sentences which had source style $s$. The set of all style-masked
sentences from $D$ is given by $x^{m}=Union(x^{m_{s}}:\forall{s}\in{S})$. Let
$x^{s}$ denote the model’s estimation of $\hat{x^{s}}$.

Probabilistic Overview. We treat the non-parallel dataset as a partially
observed parallel dataset, where the SMLM has to estimate the entire joint
distribution $P(x^{s},x^{\hat{s}})$. The SMLM assumes that every sentence from
each style domain is a result of, a) sampling from a latent style-masked
prior, $p(x^{m_{s}})$, which we estimate using the Style-Masking model,
$p(x^{m_{s}}|x^{s},\theta_{SMM})$ (conditional posterior), and is then b)
conditionally reconstructed to form the sentence with target style using the
SMLM, $q(s^{s},x^{\overline{s}}|x^{m_{s}})$ (the style-conditioned
likelihood). The posterior in this case is easily estimatable by using various
intuitive methods. The likelihood function, represented by the SMLM, has a
simple task of performing the same-style reconstruction task using a non-
parallel dataset. Using control codes to guide which domain to sample from,
enables the SMLM to also estimate the unseen section of the hypothetical
parallel and therefore work as a rought TST model. The probabilistic
formulation of the overall model is given in Fig. 1.

###  3.2 Explainable Attention as Attribution Markers

Many prototype editing methods use Vanilla Attention (VA) as attribute scores
Wu et al. (2019b); Zhang et al. (2018); Wu et al. (2020). However, it has been
shown that attention is not explanation, at least for the purpose of human-
aligned interpretability Jain and Wallace (2019). VA does not correlate well
with other well-known attribution methods (such as Integrated Gradients
Sundararajan et al. (2017)). We instead use "Explainable Attention" (EA)
scores from a Diversity-LSTM classifier Mohankumar et al. (2020); Nema et al.
(2017) which have been shown to correlate better with other attribution
methods. EA scores as attribution also correlates better with human judgement
as compared to VA. We discuss the intuitive reasoning as to why VA over LSTM
hidden states fail as attribution scores, the intuition of the Diversity-LSTM,
its mechanism and loss equations in Section The Diversity-LSTM and Explainable
Attention. We also quantitatively compare the efficacy of the style-masking
step between EA and VA in Table 2.

###  3.3 The Style-Masking module

Even with having accurate attribution scores, effective style-masking requires
careful selection of a policy which satisfies certain criteria. The primary
criteria being that only tokens which significantly contribute to the style of
a sentence must be masked, and leaving other tokens, to ensure content is also
preserved. Similar to the masking policy in Wu et al. (2020), it is natural to
consider a "top $k$ tokens" scheme in which the top $k$ tokens with highest
attribution are masked. However, this static approach fails for sentences
which do not have $k$ style-contributing tokens, leading to either partial
style masking or erroneous masking of content tokens. For the same reason,
even a sentence length aware scheme such as "top 15%" masking fails.
Furthermore, all such policies require sorting, leading to O(nlogn) time
complexity for style masking of each sentence in a batch.
An "Attention Surplus" policy.  Let $A=\\{A_{i}\dots A_{n}\\}$ denote the
attention distribution of a sentence of size $n$. Intuitively, we can reason
that all "special" tokens which might contribute more to style should have an
attribution greater than the average base attribution of the sentence, given
by $A^{mean}=1/n$. Generalising this further, We refer to tokens with
$A^{i}\geq A^{baseline}$ as tokens with "attention surplus" w.r.t to a
sentence-length sensitive baseline attention $A^{baseline}$ given by:

$\displaystyle A^{baseline}=(1+\lambda_{\epsilon})*A^{mean}$ (1)

where $\lambda_{\epsilon}$ is a hyperparameter of range $0-1.0$. This chosen
threshold $A^{baseline}$ is sensitive to the sentence length as well and
subsequently ensures that the number of style-significant tokens can be
dynamically determined, without need of an elaborate algorithm. As a sanity
check, we observe that even in the adversarial case where all tokens might be
equally important to style, $A$ resolves into a $UniformDistribution(n)$ and
our policy correctly resorts to masking all tokens111Assuming
$\lambda_{\epsilon}=0$, whereas in practice we find
$\lambda_{\epsilon}=0.15,0.5$ giving optimal masking for the Sentiment and
Discourse TST, respectively. More is discussed in 5.2. Let $Mask$ denote the
token mask matrix of size $n$ initialised with zeros.

$\displaystyle\text{Mask}[A_{i}\geq A^{baseline}]=1$ (2)

Using a vectorised approach as given in the above equation, we can achieve
style-masking for an entire batch in O(1) as compared to O(nlogn) per sentence
in previous approaches.

###  3.4 A Style-aware MLM via Transformer Encoder Block

The SMLM. For the sake of convenience in notation, we assume binary style
labels, $S=\\{0,1\\}$222In theory, this can be extended to any number of
styles.. The task of TST now resolves into estimating
$q(\hat{x^{\overline{s}}}|x^{m_{s}},\theta_{SMLM})$. The style-masking module
from the previous step gives us access to the discrete prior $p(x^{m_{s}}$)
i.e. available as pairs of the form $(x^{s}_{i},x^{m_{s}}_{i})$. The task of
reconstructing a sentence $x^{m_{0}}_{i}$ e.g. "The food was <blank>" into the
original sentence $x^{0}_{i}$, e.g. "The food was good" resembles a Masked
Language Model (MLM) objective similar to Devlin et al. (2019b) where
additionally the target style is contextualised as well. We thus refer to this
task of "Style-aware" MLM (SMLM), which consist of estimating two
distributions i.e. $q(x^{\overline{s}}|x^{m_{s}},\theta_{SMLM})$, and
$q(x^{s}|x^{m_{s}},\theta_{SMLM})$.
Control Codes. We concatenate two special meta-tokens $[dst]$, $[src]$ to
$x^{m_{s}}_{i}$ i.e. the source and destination style labels respectively.
This is denoted by the function $concat(x^{m_{s}}_{i},[src],[dst])$. Choice of
$[src]$, $[dst]$ triggers the SMLM model to sample different distributions,
either $q(x^{s}|x^{m_{s}})$ or $q(x^{\overline{s}}|x^{m_{s}}$), corresponding
to the "Same-Style" Reconstruction and TST Task respectively. The final model
is defined as:

$\displaystyle
q_{SMLM}=\begin{cases}q(x^{s}_{i}|x^{m_{s}}_{i},\theta_{SMLM}),&\text{if
}[src]=[dst]\\\
q({x^{\overline{s}_{i}|}x^{m_{s}}_{i}},\theta_{SMLM}),&\text{if
}[dst]=\overline{[src]}\end{cases}$

Figure 1: Probabilistic overview of Transduction Model assuming common Latent
Style-Masked Variable as prior.

2-Stage Training. Similar to Liu et al. (2021), we use a two stage method of
training our TST model. In the first stage, we bootstrap a rough TST model and
then finetune to improve selected metrics for the TST task.

####  3.4.1 Bootstrapping on Same Style Reconstruction Task

The SMLM model is first trained to estimate the likelihood
$q(x^{s}|x^{m_{s}})$ and therefore set $[src]=[dst]=s$ in $concat$. This
corresponds to the task of reconstructing original sentences $x^{s}$ from
their corresponding style-masked sentence $x^{m_{s}}$ produced by the SM
module. Due to the nature of the SMLM task in which input and output pairs
($x^{m_{s}}$, $x^{s}$) always have the same length, the Transformer Vaswani et
al. (2017) encoder block posses as a natural choice since it enforces same-
dimension output transformations 333This however constrains the output size to
always match the input. Consequently, our SMLM model is not generational. In
section 6.6, we further motivate this decision.. The Transformer Encoder Block
$E$ minimizes the NLL given below:

$\displaystyle L_{BS}(\theta_{SMLM})=-\mbox{log}q_{SMLM}(\hat{x^{s}}|x^{s})$

####  3.4.2 Fine-tuning for TST

We now fine-tune the SMLM’s performance for the certains metric in the TST
task i.e. $q(x^{\overline{s}}|x^{m_{s}}$) which is triggered by setting
$[dst]=\overline{[src]}$. We fine-tune for TST% by jointly training a style
classifier $cls$ by minimising the NLL taking the same-style reconstructions
of SMLM i.e. $x^{s}$ as input.444We use a single feed-forward layer with input
being the average of the last layer embeddings of $X^{s}_{i}$ (excluding the
meta-labels).

$\displaystyle L_{cls}(\theta_{cls})=-\mbox{log}p_{cls}(s|x^{s})$

This classifier now provides a supervision signal that the SMLM uses to align
$x^{\overline{s}}$ to style attribute $\overline{s}$. Whereas the
discriminator tries to discourage this. This is formulated as min max
objective $L_{FT}$ of the form:

$\displaystyle\min_{\theta_{SMLM}}\max_{\theta_{cls}}-logp_{cls}(\overline{s}|x^{\overline{s}})$

We do not fine-tune for content-preservation or fluency as we do not observe
any lacking in these metrics after bootstrapping.

Figure 2: Overview of Model Architecture considering sentences of style $s$
i.e. $x^{s}$. In reality, this is applied all styles $x^{s},\forall{s}\in{S}$

##  4 Datasets and Tasks

We report the split and label wise statistics of each dataset in 14.
Sentiment TST: Following many past studies, we evaluate our model for the
sentiment TST task using three review datasets, Yelp, Amazon and IMDb All
three datasets are annotated with two labels corresponding to positive or
negative reviews and are non-parallel. 555For Yelp, Amazon and IMDb, we used
the pre-processed version specified in https://github.com/yixinL7/Direct-
Style-Transfer.

Discourse TST: To illustrate that our model also performs well in such
situations, we introduce the Discourse TST task by performing TST on the SNLI
Bowman et al. (2015) dataset. Each instance consists of two sentences, which
either contradict, entail (or agree) or are neutral (no relationship) to each
other. We consider the task of converting the discourse style between two
sentences, from contradiction to entailment and vice-versa. Some TST tasks are
more complex than others and have different levels of granularity Lyu et al.
(2021). Unlike the sentiment task, which is "intra-sentence", where the style
can be attributed to a select set of words, the discourse task is "inter-
sentence" and requires the model to be cognizant of the context and detect the
flow of logic rather than just a few style words (especially for the
Contradiction to Entailment Task).

##  5 Analysis of Style-Masking approach

In this section, we evaluate and justify our choice of style-masking
architecture, i.e. "Explainable Attention" + "Attention-Surplus" masking
policy. We explore the question "What actually constitutes a good style-
masking step?". Intuitively, we can reason that our style-masking approach
must a) produce accurate attribution scores for each token b) use an
appropriate masking policy that determines which tokens to mask using these
attribution scores. The final style-masked sequences (latent prior to the
SMLM) must actually be devoid of style information and accurate, i.e. not done
at the expense of content information.

###  5.1 Quant. Analysis of Various Attribution Methods

We consider various other attribution methods for our analysis i.e. Vanilla
Gradients, Integrated Gradients Sundararajan et al. (2017), Vanilla Attention,
Attention * X (or inputs) and Explainable Attention Mohankumar et al. (2020).
We do not consider techniques such as LIME Ribeiro et al. (2016), LRP Bach et
al. (2015), DeepLIFT Shrikumar et al. (2017) as they are relatively more
computationally expensive during inference time. As the style-masking policy,
use the "attribution-surplus" policy to determine the final style-masked
sequences.
In Table 2,  we compute the Accuracy% and s-BLEU on the final style-masked
sequences produced by each attribution method on the test split for all four
datasets. We can reason that an ideal style-masking method should be able to
produce sentences that completely mask out style, thereby fooling a pretrained
classifier (minimizing its Accuracy%) and also preserving content information
(maximizing the s-BLEU between the source and style-masked sentences). We see
that though Vanilla Attention is able to generally produce the lowest
Accuracy%, it does so at the expense of preserving content, reflected as lower
s-BLEU compared to EA. EA, on the other hand, has the best content-preserving
style masking throughout all datasets and comes as a close second in terms of
Accuracy %. Other gradient-based methods do not perform favourably in any
aspect.

| Yelp | IMDb | Amazon | SNLI
---|---|---|---|---
Attribution Model | Acc.% | s-BLEU | Acc.% | s-BLEU | Acc.% | s-BLEU | Acc.% | s-BLEU
Vanilla Attention (VA) | 73.8 | 62.41 | 69.8 | 62.4 | 70 | 57.54 | 50.76 | 66
Explainable Attention (EA) | 71.3 | 64.32 | 75.25 | 70 | 77.36 | 73.21 | 66.5 | 85.14
Vanilla Gradients | 74.2 | 38.8 | 81.5 | 54.47 | 74.64 | 44.19 | 61.36 | 39
Gradients * X | 97.2 | 37 | 93 | 50.35 | 84.92 | 40.37 | 70.14 | 39
Integrated Gradients | 77.7 | 37.29 | 81.75 | 42.42 | 71 | 40.77 | 74.73 | 43
No Masking | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100
Table 2: Comparison of quality of Style-Masking produced using various
attribution models. We found that $\lambda_{\epsilon}=0$ worked best with all
gradient-based methods. For attention based methods (VA and EA), we found that
$\lambda_{\epsilon}=0.15,0.5$ worked best for {Yelp, IMDb, Amazon}, SNLI
respectively.

###  5.2 Effect of $\lambda_{\epsilon}$ on Style-Masking

We can intuitively reason that s-BLEU metric of the style-masked sentences
serves as a rough upper bound of the s-BLEU we achieve on the output sentences
after TST. As expected (from Eq. 1), we observe a positive correlation between
$\lambda_{\epsilon}$ and both s-BLEU and Accuracy% as seen in Table 3. It is
desirable to carefully choose $\lambda_{\epsilon}$ to be a high enough to
boost future s-BLEU scores on the TST task and also ensure that the sentences
are sufficiently style masked with low Accuracy% scores. Only manual checking,
we observed that $\lambda_{\epsilon}=0.15$ served best to accurately style-
mask sentences for the Yelp, IMDb and Amazon datasets. SNLI required a higher
$\lambda_{\epsilon}$ of 0.5 to ensure content information was preserved
appropriately.

Figure 3: Effect of $\lambda_{\epsilon}$ on the resultant style-masked
sentences using our "EA+AS" method. We compute s-BLEU(Top) and
Accuracy%(Bottom) of the style-masked sentences on the test split of each
dataset.

##  6 Experiments

We perform TST on all the datasets delineated in Section 14 and analyse the
results.

###  6.1 Automatic Evaluation Metrics

Typically, metrics for TST include Style Transfer Accuracy (using a pretrained
classifier), BLEU for content preservation, and perplexity (using a pretrained
LM) to check the fluency of outputs. Xu et al. (2020) show that this
traditional set of metrics can be gamed. For fluency, we use the "Naturalness"
metric Mir et al. (2019) instead of PPL as it is shown to correlate better
with human judgement. Apart from using BLEU for content preservation, we also
report METEOR Banerjee and Lavie (2005), ROUGE-L Lin (2004) and CIDEr Vedantam
et al. (2015) scores in Table 11.

###  6.2 Baselines and Hyperparameters

Baseline Selection. As baselines, we choose DirR Liu et al. (2021), Stable Lee
(2020), Transforming Sudhakar et al. (2019), Tag Madaan et al. (2020),
CrossAligned Shen et al. (2017), CycleRL Xu et al. (2018), StyleEmbedding Fu
et al. (2018), D&R Li et al. (2018) and CycleMulti Dai et al. (2019). For the
hyperparameters of each baseline, we consider the optimal parameters of the
best models for each dataset reported in each respective work. Whenever
available, we directly make use of the flagship TST outputs published as part
of the original work of each reference paper to ensure that a fair comparison
is done.
SMLM Versions. We consider three version of the SMLM model i.e. without fine-
tuning (SMLM), with fine-tuning for 1 epoch (SMLM-F). We also consider a
Generational version of SMLM with an additional auto-regressive Transformer-
based decoder (SMLM-G).

SMLM Hyperparameters.  SMLM consists of a lightweight Transformer Encoder
block with 2 layers, 8 heads and embeddings of size 512. We find that training
it for 15 epochs on the Same-Style Reconstruction Task and fine-tuning it for
1 epoch on for TST produces optimal results. During fine-tuning,
$\lambda_{sta}$ was set to 1 and gradient-clipping with a threshold of
$10^{-3}$ was set to prevent gradients from exploding.

Diversity-LSTM Hyperparameters.  As mentioned earlier, we use the
"explainable" attention scores of the Diversity-LSTM classifier in our style-
masking step. The Diversity-LSTM’s mechanism, auxiliary losses and
hyperparameters chosen is presented in Section The Diversity-LSTM and
Explainable Attention.

Setup and #Parameters.  The Tag Madaan et al. (2020) and DirR Liu et al.
(2021) models (the two best performing baselines) have 50M and 1.5B parameters
respectively. The SMLM has 45M parameters, 30x lesser parameters than DirR’s
fine-tuned GPT-2 model, and roughly the same number of parameters as Tag, but
outperforming it in the IMDb and SNLI datasets. We report details on training
time required and infrastructure used in Section Computational Expense and
Infrastructure used.

###  6.3 Quant. metrics for TST task

We compute TST% (percentage of sentences with target style) using a Bi-LSTM
based pretrained classifier trained on each dataset (refer Details of
Pretrained Classifier for classifier details). r-BLEU and s-BLEU refer to the
BLEU score taken between the output sentences and the human reference and
ground truth sentences, respectively. For fluency, we measure the mean
"Naturalness" score (the "Nat." column) as the mean classification score of a
pretrained fluency discriminator 666We use the pretrained naturalness
classifier available in https://github.com/passeul/style-transfer-model-
evaluation.. We also add a "Mean" score consisting of the average of TST%,
s-BLEU and Nat. (normalised to 100) columns to denote a rough measure of the
overall quality of each TST model.

###  6.4 Sentiment TST

Sentiment TST is performed on Yelp (Table 3), IMDb (Table 5) and Amazon (Table
4). We observe that for Yelp and IMDb, SMLM-FT and DirR are the best
performing models according to the Mean score. In IMDb, DirR performs better
than SMLM-FT in content preservation metrics but slightly lags behind in
naturalness scores. In IMDb, SMLM-FT achieves a significantly high TST% score
at a reasonable s-BLEU of 60.9. In Amazon,  Tag and SMLM-FT are the best
performing, with Tag having a significantly high TST% score at competitive
content and fluency metrics. Overall we observe SMLM-FT, DirR and Tag as the
best performing models.

###  6.5 Discourse TST

It might be presumed that Prototype editing methods such as SMLM are only
capable of working well on "course-grained" styles, i.e. where the presence of
style is determined by the presence of a set of certain words that convey the
style. To inspect if this is true and gauge the ability of the SMLM to operate
on more cognitive and complex tasks, we consider "Discourse TST" by using
Natural Language Inference (NLI) datasets.

We report statistics for the SMLM in Table 6. We observe that the SMLM-FT
model does well overall in this task and obtains a strong mean score of 85.53
(higher than the sentiment TST tasks). It lags behind a little mainly in the
TST% metric, but with a strong s-BLEU and Naturalness score. We also report
qualitative examples of the discourse TST task in Table 9 and 12.

Model | TST% | r-BLEU | s-BLEU | Nat. | Mean
---|---|---|---|---|---
DirR | 92.9 | 23.5 | 60.8 | 0.84 | 79.27
Stable | 81.6 | 15.6 | 39.2 | 0.73 | 64.6
Transforming | 84.8 | 18.1 | 44.7 | 0.83 | 70.9
Tag | 87.7 | 16.9 | 47 | 0.83 | 72.57
CrossAligned | 74.4 | 6.8 | 20.2 | 0.68 | 54.2
CycleRL | 51.1 | 14.8 | 46.1 | 0.86 | 61.07
StyleEmbedding | 8.59 | 16.7 | 67.6 | 0.87 | 54.4
D&R | 88 | 12.6 | 36.8 | 0.89 | 71.27
cycleMulti | 83.8 | 22.5 | 63 | 0.86 | 77.6
SMLM | 56.5 | 20.5 | 63.2 | 0.85 | 68.23
SMLM-G | 63.4 | 20.3 | 61.3 | 0.83 | 69.23
SMLM-FT | 91.2 | 18.3 | 53.4 | 0.88 | 77.6
Table 3: Quantitative metrics for the Yelp Dataset. Model | TST% | r-BLEU | s-BLEU | Nat. | Mean
---|---|---|---|---|---
DirR | 58.2 | 30.1 | 60.6 | 0.91 | 69.93
Stable | 57.2 | 24.9 | 50 | 0.83 | 63.4
Transforming | 58.7 | 25.5 | 52.3 | 0.92 | 67.5
Tag | 75 | 32.6 | 68.3 | 0.91 | 78.1
CrossAligned | 73.9 | 1.5 | 2.5 | 0.62 | 46.1
StyleEmbedding | 41.1 | 13.4 | 31.2 | 0.92 | 54.8
DR | 52 | 27.2 | 56.6 | 0.92 | 66.9
SMLM | 52.4 | 31 | 71 | 0.91 | 71.47
SMLM-G | 53.4 | 31 | 69.6 | 0.88 | 70.33
SMLM-FT | 63.9 | 29.6 | 69.8 | 0.92 | 75.03
Table 4: Quantitative metrics for the Amazon dataset. Model | TST% | s-BLEU | Nat. | Mean
---|---|---|---|---
DirR | 80.3 | 67.9 | 0.92 | 80.05
cycleMulti | 67.2 | 73.7 | 0.93 | 77.95
SMLM | 66.8 | 69.2 | 0.92 | 76
SMLM-G | 68.9 | 65.6 | 0.93 | 75.83
SMLM-FT | 87.9 | 60.9 | 0.92 | 80.27
Table 5: Quantitative metrics for the IMDb dataset. Model | TST% | s-BLEU | Nat. | Mean
---|---|---|---|---
Tag | 48.3 | 90.2 | 0.98 | 78.83
SMLM | 52.2 | 88.5 | 0.98 | 79.57
SMLM-G | 58 | 86.7 | 0.98 | 80.9
SMLM-FT | 76.3 | 86.3 | 0.94 | 85.53
Table 6: Quantitative metrics for the SNLI dataset.

###  6.6 Ablation Studies

Without Finetuning.  As expected, The "SMLM" model in Table 3, 5, 4 and 6 has
substantially better r-BLEU and s-BLEU scores compared to the fine-tuned SMLM
but lacks in TST% metrics.
Using Decoder.  The generational non fine-tuned "SMLM-G" model performs quite
similarly to the non-fined-tuned decoder-less "SMLM" model with respect to the
mean score. Practically, we observed that trying to fine-tune SMLM-G for a
better TST% metric is difficult and leads to gradient exploding even with a
large amount of gradient clipping. The SMLM-FT model appears to be more stable
during fine-tuning with a lesser computational expense.

###  6.7 Additional Content Metrics.

Past work does not tend to clarify the meaning and prioritise the presence of
"content-preservation" abilities in TST models Lee et al. (2021). In this
effort, a more thorough analysis of content preservation abilities of DirR,
Tag and SMLM-FT is given in Section 11.

###  6.8 Qualitative Examples

Examples of the TST task performed using the SMLM for the IMDb and SNLI
dataset are given in Table 8 and 9 respectively.

###  6.9 Human Evaluations

We only consider the relatively unexplored Discourse TST (Entailment to
Contradiction and vice versa) task for human evaluations. We were unable to
reproduce the DirR baseline to run over the SNLI dataset. Therefore, we only
compare the next strongest performing models i.e., Tag and SMLM. Three
volunteers were given the task of voting on 200 instances (equally split for
the E to C and C to E task) from the test set. A vote consists of four options
i.e., "Model 1 better", "Model 2 better" or "Both Good", "Both Bad", where the
models were randomised. To determine the decision an each instance, a majority
considered by taking three separate votes, one from each volunteer. In the
case of no majority, "No agreement" decision was taken. As seen in Table 7,
the SMLM performs better in both tasks by a significant margin. We report some
qualitative examples taken from the evaluation in Table 12.

Direction |  | Tag
---
better
| SMLM
---
better
| Both
---
Good
| Both
---
Bad
| No
---
Agreement
E to C | 11 | 55 | 8 | 18 | 8
C to E | 8 | 44 | 2 | 35 | 11
Table 7: Human Evaluations done to compare Tag and SMLM on Discourse TST task
on SNLI dataset. "E" and "C" denote "Entailment" and "Contradiction"
respectively.

##  7 Conclusion

We introduce the SMLM, a modification of the standard MLM, which we show is
capable of performing TST by using a style-masked input and performing a
simple same-style reconstruction task with a lightweight Transformer Encoder
block. On fine-tuning the SMLM for the TST%, it is on par with state of the
art models with orders of more parameters and sophisticated architectures in
the Sentiment TST task. We show that complex styles such as flow of logic/
discourse can be manipulated even with using this simple style masking
assumption. We empirically show that the SMLM performs well in this Discourses
Manipulation task and outperforms another strong baseline in this task, also
seen through human evaluations.

##  8 Limitations

The apparent limitation with all prototype editing models, including the SMLM,
is that it encourages the model to only fill in necessary style words and
preserve the length and structure of the original sentence. In the case of
SMLM, the word-to-word input-output mapping while training the encoder
prevents the output sentence length from changing. Though it can be argued
that this even works for a relatively cognitive style like discourse, in the
future, there might exist styles which explicitly require the
addition/deletion of words/phrases in order to successfully alter the style.
Future work will therefore focus on enable variable length TST outputs,
similar to the Madaan et al. (2020) approach or by incorporating a padded
masked language model Malmi (2020).

## References

  * Bach et al. (2015) Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. 2015.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  _PLoS ONE_ , 10.
  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  _CoRR_ , abs/1409.0473.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In _IEEvaluation@ACL_.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _EMNLP_.
  * Chandrahas et al. (2018) Chandrahas, Aditya Sharma, and Partha P. Talukdar. 2018.  Towards understanding the geometry of knowledge graph embeddings.  In _ACL_.
  * Dai et al. (2019) Ning Dai, Jianze Liang, Xipeng Qiu, and Xuanjing Huang. 2019.  Style transformer: Unpaired text style transfer without disentangled latent representation.  In _ACL_.
  * Devlin et al. (2019a) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019a.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _ArXiv_ , abs/1810.04805.
  * Devlin et al. (2019b) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019b.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL_.
  * Fu et al. (2018) Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018.  Style transfer in text: Exploration and evaluation.  In _AAAI_.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural Computation_ , 9:1735–1780.
  * Hu et al. (2020) Zhiqiang Hu, Roy Ka-Wei Lee, and Charu C. Aggarwal. 2020.  Text style transfer: A review and experiment evaluation.  _ArXiv_ , abs/2010.12742.
  * Hu et al. (2017) Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017\.  Toward controlled generation of text.  In _ICML_.
  * Jain and Wallace (2019) Sarthak Jain and Byron C. Wallace. 2019.  Attention is not explanation.  In _NAACL_.
  * Jin et al. (2021) Di Jin, Zhijing Jin, Zhiting Hu, Olga Vechtomova, and Rada Mihalcea. 2021.  Deep learning for text style transfer: A survey.  _ArXiv_ , abs/2011.00416.
  * Jin et al. (2020) Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020.  Is bert really robust? a strong baseline for natural language attack on text classification and entailment.  In _AAAI_.
  * Joulin et al. (2017) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017.  Bag of tricks for efficient text classification.  In _EACL_.
  * Lee et al. (2021) Dongkyu Lee, Zhiliang Tian, Lanqing Xue, and Nevin Lianwen Zhang. 2021.  Enhancing content preservation in text style transfer using reverse attention and conditional layer normalization.  In _ACL_.
  * Lee (2020) Joosung Lee. 2020.  Stable style transformer: Delete and generate approach with encoder-decoder for text style transfer.  In _INLG_.
  * Li et al. (2018) Juncen Li, Robin Jia, He He, and Percy Liang. 2018.  Delete, retrieve, generate: a simple approach to sentiment and style transfer.  In _NAACL_.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _ACL 2004_.
  * Liu et al. (2021) Yixin Liu, Graham Neubig, and John Wieting. 2021.  On learning text style transfer with direct rewards.  In _NAACL_.
  * Luo et al. (2019) Fuli Luo, Peng Li, Jie Zhou, Pengcheng Yang, Baobao Chang, Zhifang Sui, and Xu Sun. 2019.  A dual reinforcement learning framework for unsupervised text style transfer.  _ArXiv_ , abs/1905.10060.
  * Lyu et al. (2021) Yiwei Lyu, Paul Pu Liang, Hai Xuan Pham, Eduard H. Hovy, Barnab’as P’oczos, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2021.  Styleptb: A compositional benchmark for fine-grained controllable text style transfer.  _ArXiv_ , abs/2104.05196.
  * Madaan et al. (2020) Aman Madaan, Amrith Rajagopal Setlur, Tanmay Parekh, Barnabás Póczos, Graham Neubig, Yiming Yang, Ruslan Salakhutdinov, Alan W. Black, and Shrimai Prabhumoye. 2020.  Politeness transfer: A tag and generate approach.  In _ACL_.
  * Malmi (2020) Eric Malmi. 2020.  Unsupervised text style transfer with padded masked language models.
  * Mir et al. (2019) Remi Mir, Bjarke Felbo, Nick Obradovich, and Iyad Rahwan. 2019.  Evaluating style transfer for text.  _ArXiv_ , abs/1904.02295.
  * Mohankumar et al. (2020) Akash Kumar Mohankumar, Preksha Nema, Sharan Narasimhan, Mitesh M. Khapra, Balaji Vasan Srinivasan, and Balaraman Ravindran. 2020.  Towards transparent and explainable attention models.  In _ACL_.
  * Narasimhan et al. (2022) Sharan Narasimhan, Suvodip Dey, and Maunendra Sankar Desarkar. 2022.  Towards robust and semantically organised latent representations for unsupervised text style transfer.  _ArXiv_ , abs/2205.02309.
  * Nema et al. (2017) Preksha Nema, Mitesh M. Khapra, Anirban Laha, and Balaraman Ravindran. 2017.  Diversity driven attention model for query-based abstractive summarization.  In _ACL_.
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016.  "why should i trust you?": Explaining the predictions of any classifier.  _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_.
  * Sai et al. (2019) Ananya B. Sai, Mithun Das Gupta, Mitesh M. Khapra, and Mukundhan Srinivasan. 2019\.  Re-evaluating adem: A deeper look at scoring dialogue responses.  _ArXiv_ , abs/1902.08832.
  * Shen et al. (2017) Tianxiao Shen, Tao Lei, Regina Barzilay, and T. Jaakkola. 2017.  Style transfer from non-parallel text by cross-alignment.  In _NIPS_.
  * Shen et al. (2020) Tianxiao Shen, Jonas Mueller, Regina Barzilay, and T. Jaakkola. 2020.  Educating text autoencoders: Latent representation guidance via denoising.  In _ICML_.
  * Shrikumar et al. (2017) Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017.  Learning important features through propagating activation differences.  In _ICML_.
  * Sudhakar et al. (2019) Akhilesh Sudhakar, Bhargav Upadhyay, and Arjun Maheswaran. 2019.  “transforming” delete, retrieve, generate approach for controlled text style transfer.  In _EMNLP_.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic attribution for deep networks.  _ArXiv_ , abs/1703.01365.
  * Vaswani et al. (2017) Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _ArXiv_ , abs/1706.03762.
  * Vedantam et al. (2015) Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015.  Cider: Consensus-based image description evaluation.  _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4566–4575.
  * Wiegreffe and Pinter (2019) Sarah Wiegreffe and Yuval Pinter. 2019.  Attention is not not explanation.  In _EMNLP_.
  * Wu et al. (2019a) Chen Henry Wu, Xuancheng Ren, Fuli Luo, and Xu Sun. 2019a.  A hierarchical reinforced sequence operation method for unsupervised text style transfer.  _ArXiv_ , abs/1906.01833.
  * Wu et al. (2020) Chunhua Wu, Xiaolong Chen, and Xingbiao Li. 2020.  Mask transformer: Unpaired text style transfer based on masked language.  _Applied Sciences_ , 10:6196.
  * Wu et al. (2019b) Xing Wu, Tao Zhang, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019b.  Mask and infill: Applying masked language model for sentiment transfer.  In _IJCAI_.
  * Xu et al. (2018) Jingjing Xu, Xu Sun, Qi Zeng, Xuancheng Ren, Xiaodong Zhang, Houfeng Wang, and Wenjie Li. 2018.  Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach.  In _ACL_.
  * Xu et al. (2020) Peng Xu, Jackie Chi Kit Cheung, and Yanshuai Cao. 2020.  On variational learning of controllable representations for text without supervision.  In _ICML_.
  * Yi et al. (2020) Xiaoyuan Yi, Zhenghao Liu, Wenhao Li, and Maosong Sun. 2020.  Text style transfer via learning style instance supported latent space.  In _IJCAI_.
  * Zhang et al. (2018) Y. Zhang, Jingjing Xu, Pengcheng Yang, and Xu Sun. 2018.  Learning sentiment memories for sentiment modification without parallel data.  In _EMNLP_.

Appendix

## Qualitative examples of TST

Direction |  Negative to Positive |  Positive to Negative
---|---|---
Input |  ben affleck is back to making the same boring bad acting films . |  this movie is by far one of the best urban crime dramas i ’ve seen .
Style Masked |  ben affleck is back to making the same <mask> <mask> acting films . |  this movie is by <mask> one of the <mask> urban crime <mask> i ’ve seen .
Output |  ben affleck is back to making the same truly great acting films . |  this movie is by far one of the worst urban crime garbage i ’ve seen .
Table 8: Example of Sentiment TST on the IMDb dataset. Direction |  Entailment to Contradiction |  Contradiction to Entailment
---|---|---
Input |  a guy in a red jacket is snowboarding in midair . a guy is outside in the snow |  a woman is sitting outside at a table using a knife to cut into a sandwich . a woman is sitting inside
Style Masked |  a guy in a red jacket is snowboarding in midair . a guy is <mask> in the <mask> |  a woman is sitting outside at a table using a knife to cut into a sandwich . a woman <mask> <mask> <mask>
Output |  a guy in a red jacket is snowboarding in midair . a guy is swimming in the park |  a woman is sitting outside at a table using a knife to cut into a sandwich . a woman is a outside
Table 9: Example of Discourse TST on the SNLI dataset.

## Details of Pretrained Classifier

We use a Bi-LSTM as our choice of classifier as it performs comparably to
FastText Joulin et al. (2017) and outperforms it in the SNLI dataset. A
comparison of the two models is given in Table Details of Pretrained
Classifier.

Dataset | Model | Acc.%
---|---|---
Yelp | FastText | 97.6
| Bi-LSTM | 97
IMDb | FastText | 99.35
| Bi-LSTM | 99
Amazon | FastText | 92.1
| Bi-LSTM | 93
SNLI | FastText | 72.5
| Bi-LSTM | 84
Table 10: Comparison of FastText, Bi-LSTM models for classification task on
all datasets.

## Additional Content Preservation Metrics

We present more content preservation metrics in Table Additional Content
Preservation Metrics to compare the top three performing models i.e., SMLM,
Tag Madaan et al. (2020) and DirR Liu et al. (2021).

Dataset |  Model |  METEOR |  ROUGE-L |  CIDEr |  Embedding Avg. Cosine Sim. |  Vector Extrema Cosine Sim. |  Greedy Matching Score
---|---|---|---|---|---|---|---
Yelp | SMLM-FT | 0.376 | 0.739 | 4.934 | 0.939 | 0.767 | 0.867
| DirR | 0.444 | 0.83 | 5.813 | 0.969 | 0.867 | 0.926
| Tag | 0.362 | 0.707 | 4.326 | 0.934 | 0.765 | 0.867
IMDb | SMLM-FT | 0.414 | 0.8 | 5.657 | 0.96 | 0.755 | 0.891
| DirR | 0.472 | 0.852 | 6.344 | 0.978 | 0.847 | 0.933
Amazon | SMLM-FT | 0.464 | 0.868 | 6.725 | 0.964 | 0.782 | 0.921
| DirR | 0.469 | 0.823 | 6.612 | 0.967 | 0.821 | 0.929
| Tag | 0.453 | 0.835 | 6.548 | 0.966 | 0.781 | 0.917
SNLI | SMLM-FT | 0.563 | 0.906 | 8.297 | 0.986 | 0.886 | 0.96
| Tag | 0.606 | 0.944 | 8.619 | 0.992 | 0.921 | 0.972
Table 11: Content Preservation metrics for all datasets comparing top
performing models

## Additional examples of Discourse TST

We focus more again on the Discourse TST task and report more examples of the
SMLM and the Tag Madaan et al. (2020) baseline in Table 12.

Direction |  Entailment to Contradiction |  Contradiction to Entailment
---|---|---
Input |  a black women holding a sign that says free hugs in the city . a woman is holding a sign |  a tan dog chases a black and white soccer ball . a dog is chasing after a cat
Output (Tag) |  a black women holding a sign that says free hugs in the city . a woman is holding a sign |  a tan dog chases a black and white soccer ball . a dog is chasing after a sport
Output (SMLM-FT) |  a black women holding a sign that says free hugs in the city . a woman is holding a book |  a tan dog chases a black and white soccer ball . a dog is outside after a ball
Input |  a man is doing a task by a body of water on a farm . the man is doing something by the water |  a dad with his child and an apple pie . a dad and his daughter with an blueberry pie
Output (Tag) |  a man is doing a nap by a body of water on a farm . the man is doing pushups by the water |  a dad with his child and an apple outside . a dad and his daughter with an acousticelling outside
Output (SMLM-FT) |  a man is doing a task by a body of water on a farm . the man is doing nothing by the beach |  a dad with his child and an apple pie . a dad and his daughter with an apple outside
Table 12: Examples of Discourse TST on SNLI of SMLM vs Tag Madaan et al.
(2020)

## Computational Expense and Infrastructure used

The most parameter-heavy SMLM model model was from the SNLI dataset. Therefore
we report statistics for this model to gauge the overall computational
expenses the SMLM demands. The model has 45 million parameters and each epoch
took approximately 224 seconds to train on an Nvidia V100-SMX2 GPU and an
Intel(R) Xeon(R) E5-2698 CPU. For complete details, we will make the code open
source which will also contain the models we trained along with log files with
all metadata about the model architecture and training.

## Ethics Statement

Any TST model can be used for illicit purposes. Therefore, it is important we
keep in mind a code of ethics (e.g. https://www.acm.org/code-of-ethics). We
will make all our code open-source and will contain all details of
experimentation and implementation, training time, additional hyperparameters
used in the form of log files included inside the directories of our saved
models, which can also be used to replicate results.

## The Diversity-LSTM and Explainable Attention

Effective style-masking requires an attribution model with a high degree of
plausibility, which motivates our use of "explainable" attention scores
Mohankumar et al. (2020) as choice for the style-masking step.
Why not use standard attention? Vanilla attention scores do not serve as
accurate attribution scores. Attention scores over RNN hidden states for the
classification task do not correlate well with other standard interpretation
metrics Jain and Wallace (2019), such as gradient and occlusion based methods.
Feeding alternative adversarial/random attention distributions lead to only a
modest effects are the model’s decision Wiegreffe and Pinter (2019). However
Wiegreffe and Pinter (2019) shows that these adversarial distributions, if
properly produced, do induce poorer performance showing that vanilla attention
is still partially faithful to its explanation. Mohankumar et al. (2020)
postulate that attention scores over hidden states (H) are not explainable due
to information mixing and subsequent entanglement/coupling and mutual
information among H in RNNs. To mitigate this entanglement, diversity driven
learning (inspired by results in Nema et al. (2017)) is enforced among H. This
promotes the attention mechanism over such diversity-enforced H to satisfy
"faithfulness" and "plausibility" properties when interpreted as attribution
scores, which we refer to as "Explainable attention" (EA). Mohankumar et al.
(2020) empirically show that EA does not suffer any loss in performance in the
downstream task. Supporting plausibility, a) EA scores correlate better with
strong attribution tools such as Integrated Gradients b) On analysis over POS
tags, EA attends more to tags which are contextually important w.r.t the given
task and c) Correlates better to human judgement than vanilla attention.

The Diversity Driven LSTM.  The Diversity LSTM consists of an LSTM-based
classifier with attention Bahdanau et al. (2015) over the H. The final context
vector is fed through a feedforward layer to generate the output.

$\displaystyle\tilde{\alpha}_{t}$
$\displaystyle=\mathbf{v}^{T}\mbox{tanh}(\mathbf{W}\mathbf{h}+\mathbf{b})\;\;\forall
t\in[1,m]$ $\displaystyle\alpha_{t}$
$\displaystyle=\mbox{softmax}(\tilde{\alpha}_{t})$
$\displaystyle\mathbf{c}_{\alpha}$
$\displaystyle=\sum_{t=1}^{m}\alpha_{t}{\mathbf{h}}_{t}$

To enforce the H of the LSTM to be "diverse" i.e. more disentangled w.r.t each
other, the conicity (Chandrahas et al. (2018), Sai et al. (2019)) metric is
used as an auxillary loss and is defined as the mean of "Alignment to Mean"
(ATM) for all vectors $\mathbf{v}_{i}\in\mathbf{V}$:

$\displaystyle\text{ATM}(\mathbf{v}_{i},\mathbf{V})=\text{cosine}(\mathbf{v}_{i},\frac{1}{m}\sum_{j=1}^{m}\mathbf{v}_{j})$
$\displaystyle\text{conicity}(\mathbf{V})=\frac{1}{m}\sum_{i=1}^{m}\text{ATM}(\mathbf{v}_{i},\mathbf{V})$

The attention mechanism over a Diversity LSTM’s H is now encouraged to be
faithful to a particular set of scores, thus promoting the model to move
towards more faithful and plausible attributions. The final loss is given as:

$\displaystyle
L(\theta_{Div})=-\mbox{log}p_{Div}(y|P)+\lambda_{c}on\;\mbox{conicity}(\mathbf{H}^{P})$

EA requires only training an additional diversity driven RNN classifier over
the given dataset. After which, a single forward pass is required to obtain
attribution scores. This is unlike other methods such as IG, Lime, DeepLift,
Occlusion, wherein each generating each explanation requires comparatively
more operations.

## Diversity-LSTM Hyperparameter Selection

Table Diversity-LSTM Hyperparameter Selection gives details of the Diversity-
LSTM classifier used during the style-masking step. We also note that the
Diversity-LSTM’s performance is comparable to a Bi-LSTM and FastText
classifier (as shown in Details of Pretrained Classifier).

Dataset | Acc.% | $Loss_{con}$ | $\lambda_{con}$
---|---|---|---
Yelp | 96 | 0.06 | 10
IMDb | 100 | 0.09 | 10
Amazon | 89 | 0.03 | 20
SNLI | 82 | 0.18 | 10
Table 13: Classification task statistics and choice of $\lambda_{con}$ for each dataset. Dataset | Style Label | Train | Dev | Test
---|---|---|---|---
Yelp |  | Positive
---
Negative
| 266K
---
177K
| 2000
---
2000
| 500
---
500
IMBb |  | Positive
---
Negative
| 178K
---
187K
| 2000
---
2000
| 1000
---
1000
Amazon |  | Positive
---
Negative
| 277K
---
179K
| 985
---
1015
| 500
---
500
SNLI |  | Entailment
---
Contradiction
| 183K
---
183K
| 3329
---
3278
| 3368
---
3237
Table 14: Split and label wise statistics of each dataset.

## Training Algorithm

1 foreach  _bootstrap_epoch_ do

2 foreach  _mini-batch_ do

3 minimize $L_{BS}$ w.r.t $\theta_{SMLM}$

4 end foreach

5

6 end foreach

7

8foreach  _finetune_epoch_ do

9 foreach  _mini-batch_ do

10 minimize $L_{BS}$ w.r.t $\theta_{SMLM},\theta_{cls}$

11 minimize $L_{cls}$ w.r.t $\theta_{cls}$

12 end foreach

13

14 end foreach

Algorithm 1 Training process.