# EROS: Entity-Driven Controlled Policy Document Summarization

###### Abstract

Privacy policy documents have a crucial role in educating individuals about
the collection, usage, and protection of users’ personal data by
organizations. However, they are notorious for their lengthy, complex, and
convoluted language especially involving privacy-related entities. Hence, they
pose a significant challenge to users who attempt to comprehend organization’s
data usage policy. In this paper, we propose to enhance the interpretability
and readability of policy documents by using controlled abstractive
summarization – we enforce the generated summaries to include critical
privacy-related entities (e.g., data and medium) and organization’s rationale
(e.g., target and reason) in collecting those entities. To achieve this, we
develop PD-Sum, a policy-document summarization dataset with marked privacy-
related entity labels. Our proposed model, EROS, identifies critical entities
through a span-based entity extraction model and employs them to control the
information content of the summaries using proximal policy optimization (PPO).
Comparison shows encouraging improvement over various baselines. Furthermore,
we furnish qualitative and human evaluations to establish the efficacy of
EROS.

Keywords: Controlled Summarization, Entity Extraction, Reinforcement Learning,
PPO

EROS: Entity-Driven Controlled Policy Document Summarization

Joykirat Singh∗††thanks: ∗First two authors contributed equally, Sehban
Fazili∗, Rohan Jain, Md Shad Akhtar
---
Indraprastha Institute of Information Technology Delhi (IIIT Delhi), India.
{joykirat19166, sehban21143, rohan19095<EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

In today’s Internet era, access to information has never been so convenient.
Every day, an overwhelming amount of users are exploring the Internet horizon
for entertainment or business purposes. Realizing an excellent opportunity to
increase their customer base, many organizations offer their products or
services in a convenient online setting. In majority of the cases, customers
are required to sign up to acquire the services on offer and while doing so,
they have to agree to the term-and-conditions (T&C) or policies of the service
providers.

A privacy policy is a crucial component of any organization that allows it to
legally collect, process, store, and/or distribute personal information. It
outlines how an organization will handle personal data and how it will comply
with applicable data protection laws and regulations. Little that they know,
on many occasions, customers are, advertently or inadvertently, granting full
access to their sensitive and private data (e.g., name, contact information,
location, etc.) to the service providers without reading or understanding the
privacy policy document. Moreover, some companies collect data with
distributional rights as well and make a fortune by selling user’s data to
third parties without their realization but with their inadvertent
consent111Brave under fire for alleged sale of copyrighted
data${}^{\text{,}}$222Twitter fined 150m in US for selling user’s data. The
primary reason for such ignorance on the user’s part is their busy and packed
schedule as well as lengthy and technical/legal language, which are usually
difficult to comprehend by a common user.

Entity | Explanation
---|---
Data Compulsory | Data which is compulsory for the source to enter
Data Optional | Data in which the source has the option to provide to the target
Data Others | Information that belongs to the source
Source Direct | The entity that directly provides the data to the target
Source Indirect | The entity that indirectly provides the data to the target
Target Direct | The entity to which the source directly provides the data
Target Indirect | The entity to which the source indirectly provides the data
Reason | Reason for why the data is being collected by the target
Medium | How the data will be collected by the target

(a)

(b)

Paragraph: When {you}${}_{\text{source-direct}}$ {visit the
site}${}_{\text{medium}}$, {we}${}_{\text{target-direct}}$ also collect {web
site usage information , the type and version of browser and operating
system}${}_{\text{data-compulsory}}$ {you}${}_{\text{source-direct}}$ use, {if
you arrived at trainchinese.com via a link from another website , the URL of
the linking page}${}_{\text{data-compulsory}}$. {We}${}_{\text{target-
direct}}$ use this information to {ensure our site is compatible with the
browsers used by most of our visitors}${}_{\text{reason}}$ and to {improve the
customer experience}${}_{\text{reason}}$.
---

(c)

Figure 1: Annotation labels with their definitions, inter-relationships, and
an annotated instance. Best viewed in colors.

#### Motivation and Problem Definition:

Privacy policies are essential for both businesses and individuals. For
businesses, having a privacy policy can protect them from legal issues related
to data privacy and usage. On the other hand, it provides transparency to
individuals about how their personal information will be managed and protected
by the organizations; thus enabling them to make informed decisions prior to
registering for the service. Despite its importance, very few users read these
lengthy and non-trivial documents and fall prey to their inadvertent consent.

Summarizing these documents is a straightforward remedy of the lengthy
document but it needs to ensure that every aspect of the data usage/management
must also be present in the summary to make it useful. However, given the
complicated nature of the policy document, it’s non-trivial to obtain every
critical privacy-related information in a summary. For instance, some policy
documents define different data items (viz. name, age, contact details, etc.)
at the beginning of the document but refrain from reporting the management of
data items until the end of the document or in different paragraph or context;
thus making it challenging for any summarization system to deal with such
cases. In such cases, controlled abstractive summarization techniques He et
al. (2020); Liu and Chen (2021); Zhang et al. (2023) can potentially enhance
accessibility and transparency by generating concise and coherent summaries of
policy documents. Acknowledging the severity of the problem, in this paper, we
propose an Entity-dRiven cOntrlled policy document Summarization system aka.
EROS. EROS operates in two stages: 1) it extracts various entities or data
items and their rationales through a BERT and XLNet-based entity-extraction
module; and 2) leveraging the extracted entities, it mandates a BART-based
summarization module to include these entities and their rationales through a
proximal policy optimization (PPO) framework.

We developed a dataset, namely PD-Sum, of 1900 policy documents and manually
annotated them with abstractive summaries along with privacy-related entities
and their rationales. At first, we mark all entities present in the document
and also identify what, why, and how they are being collected. To achieve
this, we proposed and followed a schema for the identification of critical
privacy-related information in a policy document as depicted in Figure
LABEL:fig:annotation-relations (c.f. Section 3 for details). It includes the
data being collected, who would be the source of data, through which medium
data will be collected, who will consume (target) the data, and what is the
reason of data collection. An example with annotated entities is shown in
Figure LABEL:fig:annotation-example. In the next step, we write a summary of
the document mandating the presence of the entities and their rationales along
with other relevant information.

Our experimental results demonstrate that EROS achieves state-of-the-art
performance on the proposed PD-Sum dataset against several baseline systems.
We also perform qualitative and error analyses to assess the capability of
EROS in ensuring various aspects of the privacy-related information in the
generated summaries.

#### Contribution:

The main contributions of this paper are summarized as follows:

  * •

We propose a BART-based entity-driven controlled policy document summarization
(EROS) to mitigate the concerns of general public over the data privacy and
security issues.

  * •

To identify privacy-related relevant information in a policy document, we
developed an entity extraction model, Entity Extraction from Policy Documents
(EEPD).

  * •

We introduce a personalized loss function and a reinforcement learning
framework using Proximal Policy Optimization (PPO) to manage the relevance and
length of the generated summaries.

  * •

We introduce a new dataset (PD-Sum) of privacy policy documents with their
summaries and privacy-bounded entities and rationales.

  * •

We also establish performance benchmarks for the proposed approach against
several baselines.

  * •

Finally, we perform qualitative and error analyses to assess the quality of
summaries.

#### Reproducibility:

Code and dataset are available at https://github.com/joykirat18/EROS.

## 2\. Related Work

Pretrained encoders have become pivotal in recent summarization approaches.
Liu et al. (2020) introduced a BERT-based unsupervised text summarization
model, achieving state-of-the-art performance on benchmark datasets. Wang et
al. (2021) explored fine-tuning strategies for language models like BERT and
GPT, revealing substantial performance gains through limited labelled data
utilization. Dong et al. (2021) proposed a hierarchical transformer model for
summarizing lengthy documents, achieving leading results on multiple
benchmarks. Zhang et al. (2020) devised PEGASUS, leveraging gap sentence
extraction and transformer-based gap filling pre-training, attaining state-of-
the-art performance on various benchmarks. These studies depict the impactful
role of pre-trained encoders in advancing summarization techniques.

Entity extraction is a fundamental task in information extraction. The problem
has been modelled in multiple ways such as sequence labelling Lin et al.
(2020); Bui et al. (2021), span level prediction Eberts and Ulges (2020);
Zhong and Chen (2020); Zhu and Li (2022), question answering Li et al. (2020)
as well as dependency parsing task Yu et al. (2020).

Recently, a paradigm shift has been observed from sequence labelling tasks to
span-based prediction of entities. In span-based task, such as Eberts and
Ulges (2020), all possible spans are selected and further classified whether
that span represents an entity or not followed by relation classification, if
required. To tackle the problem of exact spans being treated correctly and
partial spans being treated incorrectly, Zhu and Li (2022), proposes a way to
regularize span-based prediction tasks. The annotated spans are assigned a
full probability and the nearby tokens are also assigned some probability of
being correct. Zhong and Chen (2020) extracts entities along with relation
instead of the traditional approach of extracting entities and then using the
extracted entities for relation classification.

Reinforcement learning (RL) has gained traction in summarization Wang and et
al. (2018); Wan and et al. (2018). Rondeau and et al. (2018) introduced RL-
driven translation with simulated human feedback. Liu and et al. (2020)
addressed RL’s reward scarcity using human feedback. Gunasekara and et al.
(2021) presented a versatile framework using RL for abstractive summarization
through question-answering rewards. These efforts highlight RL’s effectiveness
in improving summarization. In another work, He et al. (2020) proposed
controlled summarization to let the user interact with the summary either in
the form of keywords or direct prompts. Saito et al. (2020) develops a
combination model consisting of a saliency model that extracts a token
sequence from a source text and a seq-to-seq model that takes the sequence as
an additional input text.

In comparison to existing works, our approach integrates pre-trained encoders,
reinforcement learning, and modified loss functions. Our model incorporates a
penalty mechanism in addition to a refined BART architecture for controlled
summarization. In order to provide more exact and controlled summaries, this
combined technique builds on the advantages of each component resulting in
more precise and controlled summary generation. Additionally, our method
uniquely incorporates insights from an Entity Extraction task, enhancing the
model’s ability to capture information from policy documents.

## 3\. Dataset Construction

To the best of our knowledge, the domain of privacy-driven policy document
summarization has not been studied so far; hence, we recognize the need for a
dataset to facilitate our research and thereby, develop PD-Sum, tailor-made
dataset for the policy document summarization.

#### Data collection and Filtering:

We collect policy documents of different websites curated by Amos et al.
(2021). These documents outline how websites manage (i.e., collect, use, or
disclose) personal information. After collection, we observed several issues
and hence, applied a filter to discard policy documents as follows:

  * •

Many websites have identical policy documents, we discard all but one.

  * •

In case there are URLs linking to other websites, disregard them.

  * •

Skip documents that lack meaningful/significant information or are incomplete.

  * •

Refrain from including any policy content that is not relevant to the topic at
hand.

Subsequent to the filtering process, 1921 policy document remains in PD-Sum.

#### Data Annotation:

To facilitate the entity-driven controlled summarization, we need two sets of
annotations: a) identification of privacy-related entities; and b) a summary
of the document. Considering the users’ concerns, we identified five
fundamental entities regarding data privacy and security and proposed a schema
(c.f. Figure LABEL:fig:annotation-relations) to capture their relationships:

  * •

Data: It defines the type of information that an organization usually collects
– name, email, contact number, address, location, photos, system details,
browsing history, search queries/patterns, keystrokes, etc. Further, we
observe that some of these data are compulsory as part of the service
agreement, while others are optional and users can deny access without any
interruption in service. We identify data items according to these two
categories, i.e., ‘data compulsory’ and ‘data optional’. Further, we identify
data items as ‘data others’ which are not associated with a target requesting
the data explicitly may not highlight its usage, e.g., the data item ‘personal
information’ in the sentence ‘We may share personal information with our
clients’.

  * •

Source: It signifies the provider of the information. While a majority of the
time, the user (e.g., ‘you’) is the direct source, in some cases, the source
can be indirect, e.g., "inviting your friends to join the website by sharing
contact information "(friends will be source indirect), "requiring someone
else information to ship products to their address"(someone else is source
indirect). These entities are very low in number and are majorly associated
with sharing someone else information.

  * •

Medium: It defines the way data is collected such as ‘while visiting the
website’, ‘responding to a survey’, ‘filling a form’, etc.

  * •

Target: It specifies who will consume the data. Similar to the source entity,
a target can be direct (the organization itself) or indirect (any third-party
vendor outside the organization). Though the direct target is somewhat benign
as the users know their data are being used for some specific purposes by the
service provider, the indirect target can be extremely detrimental as there is
no transparency about the usage of data in an unknown capacity.

  * •

Reason: It clarifies the purpose of data collection by the parent organization
such as ‘improving customer experience’. We observe that with indirect
targets, reasons are usually hidden or extremely vague.

Following the above schema, two annotators333Annotators were undergraduate
student volunteers and in the age group of 20-30. with good English
proficiency annotated the whole dataset using LabelBox444https://labelbox.com/
as the annotation tool. At first, we tokenize the sentences using NLTK
tokenizer Bird et al. (2009), and subsequently, for each identified entity, we
record their start and end indices as span. Further, to ensure the consistency
of annotations between them, the annotators independently annotated a small
set of documents in the pilot phase and discussed their common understanding.
Next, they annotated $10$ documents separately and achieved a Cohen Kappa
inter-annotator agreement score of $0.74$. Subsequently, we manually annotate
$150$ documents having $8000$ sentences with $9094$ distinct entity labels. In
the next step, we learn an entity-recognition model trained on the annotated
$150$ documents to obtain pseudo entity labels for the remaining $1771$
documents. An illustrative example of annotation for a paragraph of the
document is presented in Figure LABEL:fig:annotation-example. Table 1 lists
the distribution of the entities in PD-Sum.

|  | Train | Test
---|---|---|---
Documents | #Doc | 1536 | 385
#Sent in Doc | 116251 | 27487
Avg. token/Doc | 1455.6 | 1375.82
Total entities (Doc) | 130856 | 31146
– Data other | 26836 | 6458
– Data Compulsory | 7715 | 1866
– Data Optional | 274 | 68
– Reason | 10720 | 2451
– Medium | 11924 | 2922
– Target Direct | 33927 | 8159
– Target Indirect | 12421 | 3129
– Source Direct | 27002 | 6080
|  – Source Indirect | 37 | 13
Summary | #Sent in Summ | 18691 | 4500
Avg. token/Summ | 198.9 | 192.46
Total entities (Summ) | 28197 | 6952
– Data other | 6244 | 1580
– Data Compulsory | 1784 | 497
– Data Optional | 0 | 0
– Reason | 1449 | 338
– Medium | 2469 | 586
– Target Direct | 7991 | 1938
– Target Indirect | 2453 | 648
– Source Direct | 5807 | 1365
– Source Indirect | 0 | 0

Table 1: Dataset statistics of PD-Sum.

In the second stage, we focus on manually annotating $1921$ documents with a
brief yet informative summary. We provide clear guidelines to our annotators,
encouraging them to extract key data-related phrases and concepts from the
document while preserving the essence of the content in a concise and clear
manner. The goal is to keep the summaries as crisp and to the point as
possible. We emphasize using simple and everyday language, steering away from
complex or overly technical terminology. Our aim is to make the summaries
accessible and easily understandable to a broad audience. This approach
ensures that the information conveyed in the summaries remains approachable
and digestible. In terms of document length, we’ve found that policy documents
typically contain an average of $1500$ tokens. On the other hand, the
annotated summaries, following our guidelines, average about $200$ tokens.
This significant reduction in token count maintains a balance between
providing necessary information and keeping the summaries succinct and user-
friendly.

Figure 2: Left: Proposed Model for EROS. The reference model is a frozen pre-
trained BART-based model with modified loss. We initialize the controlled
summarization model in a similar way, which is subsequently updated through a
PPO framework on a combination of rewards and KL-divergence loss. Right:
Entity extraction model jointly learns a entity classification and entity
identification module with the assistance of contrastive loss. Further to
minimize the effect of false positives in identification, we supplement it
with a entity classification module in a joint framework.

## 4\. Proposed Methodology - EROS

Our proposed controlled summarization model, EROS, is depicted in Figure 2. It
works in two stages. At first, we train an entity extraction model that aims
to predicts all spans of entities in a given document. This model employ BERT-
based span prediction framework with the contrastive loss. Additionally, we
supplement the prediction via an entity classification model in a joint
learning setup. In the second stage, we employ a BART-based summarizer model
to generate a summary. To assess the quality of generated summary and to
ensure induction of critical entity information in the summary, we introduce
our pretrained entity extraction module in the pipeline. The extracted
entities are then compared with the true (gold) entities of the reference
summary and an entity reward is computed. Moreover, we compute two other
rewards (ROUGE-L and summary length) to ensure a relevant and concise summary.
The accumulated reward is then added with the KL-divergence score between the
log-probabilities of the controlled summarizer and a reference model (a BART-
based summarizer trained on a huge out-of-domain corpus). Subsequently, we
employ PPO to update the controlled summarizer. This process repeats for a few
step till (near-)convergence. In the following subsections, we elaborate on
these steps.

### Entity extraction module

Recent years have seen a paradigm shift in the task of entity recognition from
token-level tagging –which conceptualizes it as a sequence labelling task– to
span-level prediction Fu et al. (2021).

A span in a sentence is represented by the start and end token of a sentence.
Given a sentence $X=\left\\{x_{1},\cdots,x_{n}\right\\}$ with $n$ tokens, we
define a span of an entity as
$s_{i}^{y}=\left\\{x_{b_{i}},x_{b_{i}+1},\cdots,x_{e_{i}}\right\\}$, where
$b_{i}$ and $e_{i}$ denote the start and end index of the span $s_{i}$ with a
corresponding tag $y\in\\{$source-direct, source-indirect, data-optional,
data-compulsory, medium, target-direct, target-indirect, reason$\\}$.

Spans come in different lengths. To avoid over-fitting for a particular
length, we adopt an enumeration strategy, where all the possible $m$ spans
with a maximum length $l$ are being considered as valid spans for predicting
entities. For example, in the sentence, “we will collect name” with maximum
span length as 4, possible spans are: $s_{1}^{y}=\\{x_{1},x_{1}\\}$,
$s_{2}^{y}=\\{x_{1},x_{2}\\}$, $s_{3}^{y}=\\{x_{1},x_{3}\\}$,
$s_{4}^{y}=\\{x_{1},x_{4}\\}$, $s_{5}^{y}=\\{x_{2},x_{2}\\}$,
$s_{6}^{y}=\\{x_{2},x_{3}\\}$, $s_{7}^{y}=\\{x_{2},x_{4}\\}$,
$s_{8}^{y}=\\{x_{3},x_{3}\\}$, $s_{9}^{y}=\\{x_{3},x_{4}\\}$, and
$s_{10}^{y}=\\{x_{4},x_{4}\\}$. From gold labels, we know that out of these 10
spans, only $s_{1}$ and $s_{4}$ are valid spans; therefore, their $y$ labels
will be source-target and data-compulsory, respectively. For all other spans,
the $y$ labels will be “invalid (0)”.

We obtain token representation from BERT and subsequently, compute span
embeddings as
$\mathbf{z}_{i}^{b}=\left[\mathbf{h}_{b_{i}};\mathbf{h}_{e_{i}}\right]$
Additionally, to provide information regarding the width of each span, we
induce a learnable width encoding vector $z_{i}^{w}$ according to their width.
Thus, the final representation becomes
$\mathbf{s}_{i}=\left[\mathbf{z}_{i}^{b};\mathbf{z}_{i}^{w}\right]$. Our
initial experiments showed encouraging results; however, we also observe a
significant number of false-positive, especially, for a sentence with no
entity at all, in our predictions. To mitigate such issue, we incorporate a
binary classification task to identify if the sentence contains an entity in a
joint learning framework.

#### Contrastive Loss:

We compute a similarity score between pairs of input spans, and then minimize
the distance between similar pairs while maximizing the distance between
dissimilar pairs using the contrastive loss:

$\displaystyle\mathrm{P}\left(\mathbf{y}\mid
s_{i}\right)=\frac{\operatorname{score}\left(\mathbf{s}_{i},\mathbf{y}\right)}{\sum_{\mathbf{y}^{\prime}\in\mathcal{Y}}\operatorname{score}\left(\mathbf{s}_{i},\mathbf{y}^{\prime}\right)}$
(1)

where
$\operatorname{score}\left(\mathbf{s}_{i},\mathbf{y}_{k}\right)=\exp\left(\mathbf{s}_{i}^{T}\mathbf{y}_{k}\right)$,
is a function that measures the compatibility between a learnable label
representation of the class k $\mathbf{y}_{k}$ and span $\mathbf{s}_{i}$.

#### Span Prediction:

Finally, the span representations $s_{i}$ are fed into a softmax function to
get the probability considering the label $y$. For optimization, we combine
the three losses – entity identification cross-entropy loss ($\ell_{1}$),
binary classification loss ($\ell_{2}$), and the contrastive loss ($\ell_{3}$)
through the following weighting mechanism, where $\alpha_{1}$, $\alpha_{2}$,
and $\alpha_{3}$ are hyperparameters.

$\displaystyle\mathcal{L}^{e}=\alpha_{1}\ell_{1}+\alpha_{2}\ell_{2}+\alpha_{3}\ell_{3},\quad\text{where}\sum\alpha_{i}=1$
(2)

### Entity-Driven Controlled Summarization

As the foundational model, we employ BART in our experiment. Further, we
induced a modified loss function specially designed for the entity-driven
controlled summary generation. To elaborate, we obtain gold entities ($e_{i}$)
from the PD-Sum dataset and integrate them into the loss function of the BART
model. This entails augmenting the traditional cross-entropy loss with a
penalty component derived from the extracted entities. It enables the BART
model to comprehend the presence and importance of entities in the summaries,
thereby refining its summary generation capabilities while maintaining control
over the process. Mathematically it can be seen as follows:

CE $\displaystyle=-\sum(y\cdot\log(x))$ $\displaystyle TP$
$\displaystyle=\sum_{e_{i}}(1.0-\text{step}(e_{i}\in S_{G}))$
$\displaystyle\mathcal{L}^{s}$
$\displaystyle=\lambda\cdot\text{CE}+(1-\lambda)\cdot\text{TP}$ (3)

where CE, TP, $\lambda$, and $S_{G}$ are the cross-entropy, token penalty
score, weight of the loss, and the generated summary, respectively. We compute
TP by penalizing the model for each missing entity $e_{i}$ in $S_{G}$. The
step function will return 1 only if the entity is part of the summary,
otherwise, a value of 0 will be returned.

Further to supplement the controlled summary generation process, we adopted a
feedback mechanism, in the form of reinforcement learning, to reward/penalize
the model for inducing/not-inducing the privacy-related entities in the
summaries. We use proximal policy optimization (PPO) to enforce the model to
improve the generation quality. First introduced by Schulman et al. (2017),
PPO refines policy adjustments by combining ratio-based enhancement with a
clipped surrogate objective; thus, ensuring controlled updates. Incorporating
an auxiliary value function, PPO enhances policy updates by estimating
advantages and rewards more accurately, particularly in complex scenarios.

The proposed model shown in Figure 2, contains a policy model (i.e., the
controlled summarizer model that is being trained), a reference model, a
reward model, and a value function. The value function is used to describe the
reward at timestep $t$. On the other hand, the reference model is used to
calculate the KL divergence between the original model and the policy model.
The main idea is to ensure that the active model does not deviate a lot from
its original distribution.

#### Reward calculation:

We compute three rewards to maintain the coverage, conciseness, and relevance
in the summary. The coverage reward ensures the readability of the generated
summary –we compute the ROUGE-L score, which is based on the longest common
subsequence (LCS) between two sequences, as the first reward
($R1=\text{ROUGE-L}(S_{G},S_{R})$). A longer LCS indicates that the generated
summary conveys similar meaning and concepts as the reference summary. The
conciseness reward ($R2$) limits the model to generate an adequate length
summary and avoid generating lengthy jargon.

$\displaystyle
R_{2}=\frac{1-|(len(S_{G})-len(S_{R}))|}{max(len(S_{G}),len(S_{R}))}$ (4)

Finally, we compute the entity reward ($R3$) as follows: Let $E_{total}$ be
the total number of entities predicted from the generated summary, and
$E_{correct}$ and $E_{incorrect}$ be the number of entities present and not
present in the gold summary respectively.

$\displaystyle R_{3}=\frac{E_{correct}-\beta*E_{incorrect}}{E_{total}}$ (5)

where $\beta$ is a negative factor for penalizing incorrect entities. We set
$\beta=0.3$ for our experiments.

Training EEPD | Training EROS | Generating Text (EROS)
---|---|---
Hyper-parameter | Value | Hyper-parameter | Value | Hyper-parameter | Value
n_class | 10 | warmup | 0.1 | max_seq_length | 1024
bert_dropout | 0.2 | learning_rate | 5.41e-6 | min_new_tokens | 200
xlnet_dropout | 0.2 | adaptive_kl_coef | True | top_p | 0.9
LR | 1e-5 | gamma | 0.99 | do_sample | True
maxlen | 512 | batch_size | 8 | top_k | 10
maxnorm | 1.0 | mini_batch_size | 2 | use_cache | True
batchSize | 4 |  |  | num_beams | 1
max_spanLen | 10 |  |  |  |
spanLen_emb_dim | 300 |  |  |  |
$\alpha_{1}$ | 0.5 |  |  |  |
$\alpha_{2}$ | 0.25 |  |  |  |
$\alpha_{3}$ | 0.25 |  |  |  |
l | 10 |  |  |  |

Table 2: Hyperparameters for training EEPD, EROS, and generating summary from
EROS

Model | Rouge |  | BLEU | METEOR | BS
---|---|---|---|---|---
R1 | R2 | RL |  | B1 | B2 | B3 | B4
Extractive Oracle  | 0.43 | 0.30 | 0.42 |  | 0.14 | 0.12 | 0.11 | 0.10 | 0.25 | 0.881
Bert2Bert  | 0.25 | 0.04 | 0.22 |  | 0.22 | 0.08 | 0.10 | 0.16 | 0.25 | 0.818
T5-Summarizer  | 0.44 | 0.24 | 0.42 |  | 0.32 | 0.24 | 0.19 | 0.17 | 0.34 | 0.877
PEGASUS  | 0.35 | 0.17 | 0.32 |  | 0.18 | 0.12 | 0.09 | 0.08 | 0.23 | 0.859
BART  | 0.46 | 0.29 | 0.44 |  | 0.31 | 0.25 | 0.22 | 0.20 | 0.33 | 0.882
CTRL-SUM [bigpatent] | 0.200 | 0.037 | 0.180 |  | 0.174 | 0.067 | 0.022 | 0.008 | 0.142 | 0.798
CTRL-SUM [cnndm] | 0.340 | 0.148 | 0.318 |  | 0.266 | 0.164 | 0.121 | 0.099 | 0.246 | 0.837
CTRL-SUM [PD-Sum] | 0.355 | 0.124 | 0.327 |  | 0.244 | 0.134 | 0.077 | 0.044 | 0.216 | 0.848
T5-Loss | 0.45 | 0.26 | 0.43 |  | 0.32 | 0.24 | 0.19 | 0.17 | 0.34 | 0.877
PEGASUS-Loss | 0.38 | 0.21 | 0.36 |  | 0.23 | 0.17 | 0.14 | 0.12 | 0.26 | 0.865
BART-Loss | 0.48 | 0.31 | 0.46 |  | 0.31 | 0.25 | 0.22 | 0.20 | 0.34 | 0.889
EROS | 0.519 | 0.341 | 0.500 |  | 0.424 | 0.337 | 0.292 | 0.262 | 0.438 | 0.891
EROS w/o R1 | 0.434 | 0.232 | 0.412 |  | 0.351 | 0.241 | 0.185 | 0.148 | 0.33 | 0.871
EROS w/o R2 | 0.414 | 0.211 | 0.395 |  | 0.328 | 0.217 | 0.16 | 0.124 | 0.295 | 0.8648
EROS w/o R3 | 0.510 | 0.324 | 0.491 |  | 0.412 | 0.322 | 0.275 | 0.244 | 0.415 | 0.888

Table 3: Rouge, Bleu, Meteor and BertScore scores for baselines and our
proposed EROS model. The term “Loss" signifies application of a customized
loss function (c.f. equation 4). CTRL-SUM [$D$] signifies that the model is
trained on dataset $D$.

## 5\. Experiments and Results

We train EROS on 1536 documents, while we use 385 documents for evaluating the
performance. A summary of the hyperparameters used for training is listed in
Table 2. We evaluate both components of our model separately and their results
are furnished in Tables 3 and 4, respectively – we compute precision, recall,
and F1-score for the entity extraction module, we measure the quality of
summarization through BLEU, ROUGE, METEOR, and BERTScore. We also perform
extensive comparative analysis against the following baselines for both
models. In all cases, we re-train/fine-tune these models on PD-Sum.

#### Baselines:

  * •

Entity Extraction: We evaluate EEPD against six entity extraction models
covering both sequence-labelling and span-based frameworks: Sequence-
labelling:  Taggin each token as Begin, Intermedite, or Other – BERT Devlin et
al. (2018), SpanBERT Joshi et al. (2020), and PrivBERT Srinath et al. (2021).
Span-based: Start/end index-based extraction models – Boundary Smoothing Zhu
and Li (2022), PURE Zhong and Chen (2020), SPERT Eberts and Ulges (2020).

  * •

Entity-Driven Controlled Summarization: For the controlled summarization
model, we compare EROS’s effectiveness against the following baseline
approaches: Extractive Oracle Hirao et al. (2017): It employs extractive
methods to gather essential information from the source text for
summarization. This model offers efficiency by avoiding new sentence
generation and at the expense of missing overall context and flow. PEGASUS
Zhang et al. (2020): Pre-trained transformer-based sequence-to-sequence
architecture designed by Google AI. The model uses a novel pre-training
objective known as "gap-sentences generation". CTRL-SUM He et al. (2020): It
uses control tokens at inference time to enable user to control the summary.
In the original work, it has been trained on two huge datasets – BIGPATENT
patent documents Sharma et al. (2019) and CNN/Dailymail news articles Hermann
et al. (2015). Further, in this work, we also train CTRL-SUM on PD-Sum. In
addition, we also compare with Bert2Bert Chen et al. (2022), T5 Raffel et al.
(2020), and BART Lewis et al. (2019) summarizers.

|  | Precision | Recall | F1-score
---|---|---|---|---
BIO | BERT | 0.31 | 0.38 | 0.34
SpanBERT | 0.31 | 0.39 | 0.35
PrivBERT | 0.37 | 0.44 | 0.40
Span Based | SPERT | 0.10 | 0.68 | 0.17
Boundary Smoothing | 0.40 | 0.48 | 0.44
PURE | 0.35 | 0.12 | 0.17
SpanNER | 0.49 | 0.56 | 0.52
SpanNER + Identification | 0.47 | 0.66 | 0.55
EEPD | 0.54 | 0.62 | 0.58
– Identification | 0.48 | 0.57 | 0.52

Table 4: Comparative results of entity extraction module.

#### Result Analysis:

Table 4 contains the comparative result of EEPD and various baselines.
PrivBert reports the best F1-score of $0.40$ in the sequence-labelling
framework (i.e., in a BIO setup), whereas, BoundarySmoothing yields $+4\%$
better F1-score at $0.44$ in the span-based setting. Further, SpanNER, with
the identification module, records the best F1-score of $0.58$ among all
baselines. In comparison, EEPD reports the state-of-the-art performance at
$0.58$ F1-score – an increment of $+3\%$ over the best baseline. We also
observe the effect of the entity identification module on the overall
performance – a decrement of $-6\%$ is observed on removing the identification
component from EEPD.

Ref Summary:  When you register with the Site or use any of our Services, you
may be asked to provide us with Personal Information. It is entirely optional
to provide this information. If you do not provide the requested information,
you may be unable to use some or all of the Site’s features. To improve the
Site and Services, we use cookies (a small text file placed on your computer
to identify your computer and browser) and Web beacons (electronic files
placed on a Web site that monitor usage). If you delete or disable cookies,
some features of the Site or Services may not function properly. You have the
option of making some of your Personal Information available to others. Users
of the Site and commercial search engines such as Google, Yahoo!, and Bing may
access the information you choose to make available. Some third-party services
may provide us with information from your accounts, allowing us to improve and
personalise your experience on the Site. When you visit our Site, we may allow
third-party companies to serve ads and/or collect certain anonymous
information….
---
Entities in Ref summary: Data compulsory: [e-mail, personally identifiable
information]; Data others: [personal information]; Source Direct: [you,
<EMAIL_ADDRESS>Target Direct: [we, us]; Target Indirect: [service
providers, third parties]; Medium: [register with the Site or use any of our
Services, cookies, web beacons, visit our site]; Reason: [improve the site and
services, improve and personalise your experience on the Site, Usage Data
regarding the Site and Services].
Entities in BART summary: Data compulsory: [None]; Data others: [None]; Source
Direct: [you<EMAIL_ADDRESS>Target Direct: [we, us]; Target Indirect:
[None]; Medium: [using the site or services, cookies, web beacons, visit our
site]; Reason: [improve the experience of the site and services].
Entities in BART-Loss summary: Data compulsory: [e-mail, personally
identifiable information]; Data others: [None]; Source Direct: [you]; Target
Direct: [we, us]; Target Indirect: [service providers]; Medium: [cookies, web
beacons,]; Reason: [None].
Entities in EROS w/o R3 summary: Data compulsory: [e-mail, personally
identifiable information]; Data others: [personal information]; Source Direct:
[you]; Target Direct: [we, us]; Target Indirect: [service providers]; Medium:
[register with the Site or use any of our Services, cookies, web beacons,
visit our site]; Reason: [None].
Entities in EROS summary: Data compulsory: [e-mail, personally identifiable
information]; Data others: [personal information]; Source Direct: [you,
<EMAIL_ADDRESS>Target Direct: [we, us]; Target Indirect: [service
providers, third parties]; Medium: [register with the Site or use any of our
Services, cookies, web beacons, visit our site]; Reason: [Usage Data regarding
the Site and Services].

Table 5: Qualitative analysis of generated summaries. Due to space
constraints, we could not provide the generated summaries of models. source
direct, target direct, medium - blue, reason, data, data compulsory, target
indirect, data optional. Best viewed in color.

For the controlled summarization task, we furnish the results in Table 3. We
compute the traditional ROUGE, METEOR, BLEU and BertScore scores to evaluate
the generated summaries. Pretrained Ctrl-sum He et al. (2020) failed to
perform well when compared to fine-tuned models. Among all baselines (except
with the modified loss, TP (c.f. Section 4)), BART reports the best
performance across the three metrics – ROUGE-L score ($0.44$), BLEU-4
($0.20$), METEOR ($0.33$) and BertScore($0.882$). Further, we observe that the
incorporation of modified TP loss obtains comparable results in the majority
of the cases and better several setups – ROUGE-L: T5, BART, and PEGASUS
improved; BLEU-4: PEGASUS improved; METEOR: BART and PEGASUS improved and
BertScore: BART and PEGASUS improved. On the other hand, EROS yields the best
scores across all metrics – improvement of $+0.04$ in ROUGE-L ($0.50$),
$+0.062$ in BLEU-4 ($0.262$), $+0.098$ in METEOR ($0.438$) and $+0.0017$ in
BertScore ($0.8907$).

Model | En / Sum | Len dev
---|---|---
Gold Summary | $18.057$ | -
BART | $10.329$ | $-0.2308$
BART-Loss | $10.355$ | $-0.266$
EROS | $14.436$ | $+0.163$
Table 6: Rate of entities and conciseness of the generated against the gold
summary.

Table 6 shows a comparative analysis on the conciseness and the rate of
captured entities in the generated summaries. We observe $+4\%$ increase in
rate of entities in EROS against the two baselines.

#### Human Evaluation:

We performed a human evaluation on a subset of randomly chosen samples from
the PD-Sum’s test set. We compare the summaries of EROS and two baselines
i.e., BART and BART-Loss. We ask our evaluators to assess the generated
summaries against the reference summaries on four parameters – the
informativeness of the summary (INF), its conciseness (CON), its fluency and
grammatical correctness (FL), and the inclusion of relevant entities (EC). The
first two metrics assess the quantity of the information content in the
generated summary, whereas the third metric ensures the linguistics quality of
the summary. Additionally, the last metric explicitly evaluates the presence
of privacy-related entities in the summary. For each parameter, all evaluators
assign a rating on a scale of 1 (worst) to 5 (best) based on the quality of
the summaries.

Model | INFO | CON | FLU | EC
---|---|---|---|---
BART | 3.0 | 3.35 | 3.75 | 2.90
BART-Loss | 3.75 | 3.70 | 4.0 | 3.40
EROS | 4.20 | 3.15 | 4.05 | 4.15
Table 7: Human evaluation on informativeness, conciseness, fluency, and entity
coverage.

Subsequently, we aggregate the scores through averaging and report the
observations in Table 7. We observe that EROS outperforms the other two
baseline models in three out of four metrics. It records a comparatively
inferior score for conciseness, suggesting that EROS’s summaries are
relatively lengthier than others. However, it is better in informativeness,
grammatical correctness, and inclusion of relevant entities.

## 6\. Conclusion

In this work, we presented a novel approach for abstractive summarization of
privacy policy documents. Our approach aimed to address the challenge of
generating controlled and informative summaries that capture the essence of
complex privacy policies. To achieve this, we introduced a customized loss
function and incorporated a reinforcement learning framework, enabling us to
optimize the relevance of the generated summaries. To facilitate the
evaluation and advancement of research in this domain, we also introduced a
new dataset for controlled summarization generation. The experimental results
obtained from our comprehensive evaluations highlight the effectiveness of the
proposed approach. Our model achieved state-of-the-art performance on the
custom dataset. The controlled generation of summaries allows for improved
accessibility and transparency for users, enabling them to quickly grasp the
key points of privacy policies without getting overwhelmed by excessive
information. The findings of our work demonstrate the potential of our
approach to make a significant impact in the field of privacy policy
summarization. By addressing the critical need for concise and user-friendly
representations of privacy policies, we contribute to enhancing user
understanding. The implications of our work extend to various domains where
privacy policies play a crucial role, including data protection, online
services, and legal compliance.

## Limitation

Our research initiative necessitates the utilization of proprietary and
sensitive data, encompassing company privacy policies, to train our
specialized model. This dataset is comprehensive, containing detailed policies
of various companies. This inclusion is vital for our model to effectively
understand and learn the intricacies of privacy policies and their
implementations. However, the training process for our model is
computationally intensive, requiring substantial computing resources and time.
The utilization of a reinforcement learning approach, while effective,
significantly extends the duration needed for the training phase. This
computational demand poses a notable challenge in terms of time and resource
allocation. We understand that the relevant entities, such as data items,
target usage, etc., are extremely critical in comprehending the policy
document and hence, any model must include these components in the generated
summaries. However, in some cases, the generated summaries may not be
exhaustive in terms of these non-trivial entities and care must be taken prior
to agreeing to the terms and conditions of the policy document.

## Ethical Consideration

In this section, we address the ethical concerns and safeguards associated
with our research on privacy policy text summarization. Ensuring the ethical
conduct of our research is of paramount importance, given the sensitive nature
of the data and the potential implications for privacy.

  * •

Data Collection and Usage: Data was obtained from publicly available sources
or with explicit permission, and sensitive information was excluded from our
dataset. We respect any limitations set by data sources concerning data usage.

  * •

Transparency and Explainability: We recognize the importance of providing
transparent and understandable summaries. Efforts were made to ensure that our
summarization process can be explained to users, and address any algorithmic
biases that may arise, striving for fairness.

  * •

Fairness and Bias Mitigation: We are mindful of potential biases and employ
measures to mitigate them, particularly those related to gender, race, and
other sensitive attributes. Fairness in our summaries is a core consideration.

  * •

Compliance with Regulations: Our research adhered to all relevant data
protection laws and regulations, such as GDPR, HIPAA, or regional data privacy
laws, to ensure the ethical use of data.

  * •

Potential Risks and Mitigation Strategies: We acknowledge the potential risks,
such as unintended disclosure of sensitive information. To mitigate these
risks, we implement strict controls and perform thorough risk assessments.

By taking these ethical considerations into account, we aimed to conduct our
research on privacy policy text summarization with the utmost respect for
privacy, transparency, and ethical integrity.

## Acknowledgement

Authors acknowledge the partial support of Infosys foundation through Center
for AI (CAI)-IIIT Delhi.

## Bibliographical References

##

  * Amos et al. (2021) Ryan Amos, Gunes Acar, Eli Lucherini, Mihir Kshirsagar, Arvind Narayanan, and Jonathan Mayer. 2021.  Privacy policies over time: Curation and analysis of a million-document dataset.  In _Proceedings of the Web Conference 2021_ , pages 2165–2176.
  * Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper. 2009.  _Natural language processing with Python: analyzing text with the natural language toolkit_.  " O’Reilly Media, Inc.".
  * Bui et al. (2021) Duc Bui, Kang G Shin, Jong-Min Choi, and Junbum Shin. 2021.  Automated extraction and presentation of data practices in privacy policies.  _Proc. Priv. Enhancing Technol._ , 2021(2):88–110.
  * Chen et al. (2022) Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, and Qun Liu. 2022.  bert2BERT: Towards reusable pretrained language models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2134–2148, Dublin, Ireland. Association for Computational Linguistics.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dong et al. (2021) Li Dong, Furu Wei, Ming Zhou, and Ke Xu. 2021.  Hierarchical transformers for long document summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics_.
  * Eberts and Ulges (2020) Markus Eberts and Adrian Ulges. 2020.  Span-based joint entity and relation extraction with transformer pre-training.
  * Fu et al. (2021) Jinlan Fu, Xuanjing Huang, and Pengfei Liu. 2021.  SpanNER: Named entity re-/recognition as span prediction.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7183–7195, Online. Association for Computational Linguistics.
  * Gunasekara and et al. (2021) Chamara Gunasekara and et al. 2021.  Using question answering rewards to improve abstractive summarization.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 518–526.
  * He et al. (2020) Junxian He, Wojciech Kryscinski, Bryan McCann, Nazneen Rajani, and Caiming Xiong. 2020.  Ctrlsum: Towards generic controllable text summarization.  _ArXiv_ , abs/2012.04281.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  _Advances in neural information processing systems_ , 28.
  * Hirao et al. (2017) Tsutomu Hirao, Masaaki Nishino, Jun Suzuki, and Masaaki Nagata. 2017.  Enumeration of extractive oracle summaries.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_ , pages 386–396, Valencia, Spain. Association for Computational Linguistics.
  * Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020.  Spanbert: Improving pre-training by representing and predicting spans.  _Transactions of the Association for Computational Linguistics_ , 8:64–77.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.
  * Li et al. (2020) Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. 2020.  A unified MRC framework for named entity recognition.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5849–5859, Online. Association for Computational Linguistics.
  * Lin et al. (2020) Bill Yuchen Lin, Dong-Ho Lee, Ming Shen, Ryan Moreno, Xiao Huang, Prashant Shiralkar, and Xiang Ren. 2020.  Triggerner: Learning with entity triggers as explanations for named entity recognition.  _CoRR_ , abs/2004.07493.
  * Liu and et al. (2020) Fei Liu and et al. 2020.  Learning to summarize from human feedback.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Liu et al. (2020) Yang Liu, Ming Li, Xin Liu, Yao Chen, and Maosong Sun. 2020.  Presumm: A bert-based unsupervised text summarization model.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Liu and Chen (2021) Zhengyuan Liu and Nancy F. Chen. 2021.  Controllable neural dialogue summarization with personal named entity planning.  In _Conference on Empirical Methods in Natural Language Processing_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.
  * Rondeau and et al. (2018) Mathieu-Auguste Rondeau and et al. 2018.  Reinforcement learning for bandit neural machine translation with simulated human feedback.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_.
  * Saito et al. (2020) Itsumi Saito, Kyosuke Nishida, Kosuke Nishida, and Junji Tomita. 2020.  Abstractive summarization with combination of pre-trained sequence-to-sequence and saliency models.  _arXiv preprint arXiv:2003.13028_.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.  Proximal policy optimization algorithms.  _CoRR_ , abs/1707.06347.
  * Sharma et al. (2019) Eva Sharma, Chen Li, and Lu Wang. 2019.  Bigpatent: A large-scale dataset for abstractive and coherent summarization.  _arXiv preprint arXiv:1906.03741_.
  * Srinath et al. (2021) Mukund Srinath, Shomir Wilson, and C Lee Giles. 2021.  Privacy at scale: Introducing the PrivaSeer corpus of web privacy policies.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6829–6839, Online. Association for Computational Linguistics.
  * Wan and et al. (2018) Xiaojun Wan and et al. 2018.  Improving abstractive document summarization with salient information modeling.  In _Proceedings of the 27th International Conference on Computational Linguistics_.
  * Wang and et al. (2018) Chenguang Wang and et al. 2018.  A hierarchical reinforced sequence operation method for abstractive summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_.
  * Wang et al. (2021) Liyuan Wang, Yue Zhang, Fuli Xu, and Xu Chen. 2021.  Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping.  _arXiv preprint arXiv:2103.05447_.
  * Yu et al. (2020) Juntao Yu, Bernd Bohnet, and Massimo Poesio. 2020.  Named entity recognition as dependency parsing.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6470–6476, Online. Association for Computational Linguistics.
  * Zhang et al. (2020) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.
  * Zhang et al. (2023) Yusen Zhang, Yang Liu, Ziyi Yang, Yuwei Fang, Yulong Chen, Dragomir Radev, Chenguang Zhu, Michael Zeng, and Rui Zhang. 2023.  Macsum: Controllable summarization with mixed attributes.
  * Zhong and Chen (2020) Zexuan Zhong and Danqi Chen. 2020.  A frustratingly easy approach for entity and relation extraction.  _arXiv preprint arXiv:2010.12812_.
  * Zhu and Li (2022) Enwei Zhu and Jinpeng Li. 2022.  Boundary smoothing for named entity recognition.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7096–7108, Dublin, Ireland. Association for Computational Linguistics.


# Simplifying Dataflow Dialogue Design

Joram Meron
Telepathy Labs GmBH, Zurich, Switzerland
<EMAIL_ADDRESS>
###### Abstract

In (Semantic Machines et al., 2020), a dataflow (DF) based dialogue system was
introduced, showing clear advantages compared to many commonly used current
systems. This was accompanied by the release of SMCalFlow, a practically
relevant, manually annotated dataset, more detailed and much larger than any
comparable dialogue dataset. Despite these remarkable contributions, the
community has not shown further interest in this direction. What are the
reasons for this lack of interest? And how can the community be encouraged to
engage in research in this direction?

One explanation may be the perception that this approach is too complex - both
the the annotation and the system. This paper argues that this perception is
wrong: 1) Suggestions for a simplified format for the annotation of the
dataset are presented, 2) An implementation of the DF execution engine is
released111https://github.com/telepathylabsai/OpenDF, which can serve as a
sandbox allowing researchers to easily implement, and experiment with, new DF
dialogue designs. The hope is that these contributions will help engage more
practitioners in exploring new ideas and designs for DF based dialogue
systems.

Simplifying Dataflow Dialogue Design

Joram Meron Telepathy Labs GmBH, Zurich, Switzerland<EMAIL_ADDRESS>

##  1 Introduction

Traditional task oriented dialogue systems implement a pipeline consisting of
NLU (natural language understanding, converting the user input to a structured
representation), which feeds into a DM (dialogue manager), implementing
dialogue state tracking (DST) as well as a policy to decide what action to
take next. Both the NLU and DM components can take different forms.

Modern machine learning approaches have demonstrated large improvements in
prediction accuracy for both the NLU and DM components (as well as end-to-end
pipelines and other architectures), but for industrial applications, where
strict control of the system behaviour is required, it is very common that the
NLU is implemented as a machine learning based intent/entity classifier, while
the DM is conceptually based on a finite state machine (FSM)-like paradigm,
typically with the states and transitions manually defined.

The combination of intent/entity NLU, and FSM based DM offer controllability
and reduces the training data requirements, but come at the cost of reduced
expressiveness of user requests, as well as practical difficulty in scaling up
the dialogue design as more interaction scenarios are added.

##  2 Dataflow Dialogues

Microsoft’s Semantic Machines (SM) introduced a dataflow dialogue system in
(Semantic Machines et al., 2020), which represents the user requests as rich
compositional (hierarchical) expressions, which encode computational graphs.
An engine executes these computations, which results in manipulating the
computational graphs, generating an answer (possibly an error message), and
optionally producing some side effects through API’s to external services.

The prominent features of this system are:

  * •

The system represents the dialogue history as a set of graphs, where each
computational graph typically represents one user turn.

  * •

it has a refer operation to search over the current computational graphs (as
well as external resources) which allows easy look-up and re-use of graph
nodes which occurred previously in the dialogue.

  * •

it has a revise operation which allows modification and reuse of previous
computations

  * •

it has an exception mechanism which allows convenient interaction with the
user (e.g. asking for missing information, and resumption of the computation
once the information is supplied).

These features correspond to essential phenomena in natural conversations
(referring to previous turns, modifying previous requests, reacting to wrong
information, etc.), which allows the system to handle these phenomena more
effectively, and help mitigate the problem of DM scaling, compared to FSM-
based DMs222No claim is being made here that the DF approach can theoretically
represent more complex user requests than the FSM approach, rather that it can
practically be significantly easier to handle complex requests using DF..

Extracting richer representation from the user requests, does increase the
difficulty for the NLU task. To address this, the developers of the DF system
released SMCalFlow - the largest dialogue dataset to date, manually annotated
using the expressive compositional DF format, presumably the result of
thousands of man hours of work. The dialogues in this dataset cover diverse
common tasks, which can be relevant and interesting for developers of
practical task oriented dialogue systems.

In the same paper, they also show that using the DF paradigm can be
advantageous even without requiring manual preparation of rich annotations
(demonstrated on the MultiWOZ dataset Budzianowski et al. (2018)). Instead,
the original intent/entity style annotation is automatically translated into
DF expressions.

##  3 Practical Applications

Developers of actual applications, constrained by business goals, are often
unable to take advantage of the recent advances achieved in dialogue system
research. One of the most common reason for that is the need for the
developers to be able to closely control the behaviour of the application, and
be able to quickly adapt it whenever errors are discovered or specifications
evolve.

For this reason, approaches allowing manual control (as opposed to black box
approaches where control can only be asserted through engineering of the
training data) are often preferred.

One example is the use of FSM based DMs. These can be closely controlled, as
they are based on manually created rules, which makes them very easy to use
when creating simple applications. However, FSMs are notoriously difficult to
scale up, especially when it comes to tasks with multiple possible execution
paths.

The DF paradigm can be used for such applications, using manually crafted
rules to closely control the application, while potentially reducing developer
efforts as applications grow.

##  4 Complexity

More than a year has passed since the publication of the original DF paper,
the release of SMCalFlow, and a related public leader-board. Despite the
obvious theoretical and practical advantages of the DF approach, and the
potential for further improvements, the community has shown only limited
interest in following this work.

Why is this the case? This paper hypothesizes that the reason may be a
perception that the DF approach is inherently too complex (both the
annotation, and the implementation), and that this perception is a result of
two factors. First, the complexity of the annotation of SMCalFlow (which is
the only dedicated DF dataset available), and second, the fact that the
implementation of the SMCalFlow executable functions is excluded from the
released code. The lack of documentation and of explanation of the design
decisions behind the annotation, only exacerbates this perception.

In order to encourage the community to engage in this line of research, this
paper tries to counter this misconception, by following two directions:

  1. 1.

By releasing a basic implementation of the SMCalFlow execution engine. Since
this represents the work of one person during a limited time, only a part of
the functionality of SMCalFlow is implemented, but it should allow readers to
get familiar enough with the DF approach, so that they can easily implement
their own ideas in this framework.

  2. 2.

Explore (and demonstrate) ways to simplify SMCalFlow. Specifically,
simplifying the annotation format in a way which reduces the effort for
annotators (and readers), as well as for the NLU component, while maintaining
its ability to be correctly executed.

This paper also tries to emphasize that DF is not a monolithic system, with
fixed functionality, applications, tools, formats and implementation. Rather,
it is a general approach; a paradigm, which can have different interpretations
and flavours, and which is ready for further research as well as practical
uses.

##  5 Simplifying SMCalFlow

The DF approach is an object-oriented paradigm, which describes user requests
as executable programs. Annotation is practically a programming task,
translating a user’s request to a program, whose execution will satisfy the
user’s request. These programs consist of two types of functions: base DF-
framework functions (like refer and revise), and application specific
functions (e.g. CreateEvent for SMCalFlow).

In designing the application specific functions, the developers need to come
up with a consistent design for a set of functions, which can be combined
together to describe and execute user requests.

As mentioned above, multiple alternative designs are possible. Each design
results in a trade-off between coverage of possible user requests, good
programming practices (such as clarity, modularity and low complexity of the
programs, and hence of the annotations), efficiency of execution, the demands
it puts on the NLU component, etc.

In the following, a simplified annotation is presented, with the motivation to
dispel the perception that SMCalFlow is too complex. The starting point of
this work is SMCalFlow, since creating a new dataset was beyond the resources
available for this work. The price to pay for this is twofold. First, the new
design is tied to the original style. Creating a new dataset (or a completely
new annotation for an existing dataset) would allow complete freedom in
suggesting a new annotation style, but would require intensive labour. Second,
once a new design has been defined for the annotation style, the original
dataset needs to be transformed to the new design. The transformation process
is a necessary step, but not the primary focus of this paper.

While DF is not inherently complicated, finding a good design is a challenging
task. A novel aspect of this challenge is the need for the design to function
correctly within the DF paradigm, e.g. use the refer and revise operators.
Indeed, one of the motivations of this work is the hope that the community can
suggest interesting new designs, which can serve as templates for further
applications.

The work progressed by taking one original annotation at a time, defining the
simplified annotation for it, and then implementing the needed DF functions
(iteratively modifying the implementation to ensure new functions can interact
consistently with previously implemented functions). Due to limited resources,
only a small portion of the original expressions were inspected, concentrating
on the domain of event creation/update, which covers only part of SMCalFlow’s
more than 300 functions. Hopefully, the released implementation can serve as a
useful example, and motivate interested researchers to expand this work.

Note that SM released a modified version of SMCalFlow (Platanios et al.,
2021), which also represents a kind of annotation simplification, but focusing
on improving NLU accuracy, rather than understandability. This paper uses the
original SMCalFlow version.

###  5.1 Simplification Mechanism

Due to the size of the dataset, manual simplification is clearly not feasible,
therefore all simplifications were done programmatically. The full dataset was
simplified, without trying to filter out some of the "irregular" annotations
in the original dataset.

For convenience, the simplified format uses Python style expressions (as
opposed to the Lisp style S-expressions in the original dataset), as this
format is generally more familiar (the released system itself is written in
Python).

The simplification was practically done by implementing a set of tree
transformation rules, which convert specified sub-trees of the original
expressions into simplified sub-trees (the transformation code is part of the
release). A few tens of such rules were implemented (additional rules will be
needed as more SMCalFlow functions are implemented).

The simplification is applied to the whole dataset, resulting in a simplified
dataset, which can be used by the same pipeline used in the original paper.

###  5.2 Simplification Approach

The design principles (which are heuristic, and not mutually exclusive) for
the simplifications were: 1) Retain only necessary information in the
annotation, 2) Avoid explicit logical steps, 3) Move logic from the annotation
to the implementation, 4) Group and reuse repeating sequences of functions.

Practically this means: Try to omit any information which can be
deterministically inferred from context (or possibly, which can be guessed).
Keep only information which can not be inferred. Specifically, logical steps
which can be inferred from context, are moved from the annotation into the
implementation of the functions. For example, explicit type casts which are
clear from the context can be omitted. Similarly, when needed information is
missing in the user input, but can be inferred from the context, the
simplified annotation should then include only the supplied information, and
the logic to infer the missing information is left for the function
implementation.

Finally, the simplified annotation tries to avoid fragments of the original
annotation which serve only "formal" purposes, and instead tries to style the
annotation to be closer to a more natural/comprehensible description of the
user requests. Please see the appendix for examples.

###  5.3 Executing Simplified Annotations

At execution time, an additional step (which can be viewed as the inverse of
the dataset simplification step) translates the simplified annotation to a
fully executable expression. This is done, again, by implementing tree
transformation rules (for each function), which can add deterministically
inferable missing information/steps (e.g. casting input to the right type, or
performing other conversions/functions based on input type).

###  5.4 Simplification Results

Since the original code to execute SMCalFlow was not released (and
documentation not supplied), it is impossible to verify that the suggested
simplifications in fact correctly implement the same logic (nor what that
logic actually is) (in fact this was one of the motivations for this paper).
Some examples of simplification are shown in the appendix, but it can only be
left to the readers to inspect the simplified annotations and the code and
draw their own conclusions.

| Program Length
---|---
Original Annotation | (11, 37, 58)
Simplified Annotation | (2, 13, 22)
Table 1:  Program length of the two annotation styles. Length is measured as
number of seq2seq target tokens, when translating user request to annotation.
Showing (.25, .50, .75) quantiles over the entire dataset.

Table 1 shows the results of a comparison of the annotation lengths of the
original and simplified annotations, confirming that the simplification does
make the annotation significantly shorter (adding more simplification rules,
as more functions are implemented, is expected to further reduce the length).
In addition to being shorter, the simplified annotations are also more
understandable (unfortunately, this can not be shown with simple objective
measures), which should reduce annotation efforts when creating new training
data.

To verify that the simplified annotations do not increase the burden on the
NLU component, the translation pipeline of the original paper was used to
train and evaluate seq2seq models using the simplified dataset. The result
shows no degradation in translation accuracy (in fact a slight improvement in
exact match, from 72.8% to 73.8%). With the significant decrease in annotation
length, it could be expected that translation accuracy would actually improve.
Possible explanations why this did not happen could be that the
simplifications are highly regular, or the fact that the coverage of the
simplification rules is only partial.

##  6 Further Work

With the released execution code, deeper probing and exploration can become
possible, compared to having access only to the SMCalFlow dataset.

Particular areas of interest may include:

Evaluation: in addition to the exact-match metric for translation accuracy,
other metrics can be used, such as comparison of execution results, graph
structure similarity, etc.

Using the graph structure: the graph structure (at different points of the
execution) can be used by prediction models. For example, the user request
could be recursively translated into a hierarchical graph using graph self
attention as well as attention to previous graphs before/after execution.

Different design patterns which are beneficial to specific parts of the
system. For example, the execution of a computation graph could emit various
types of information which would then be useful for subsequent prediction
models.

And of course, completing the design and implementation of a full working
SMCalFlow system.

##  7 Conclusion

This paper argues that dataflow dialogue systems are worthy of more attention
from the community.

In order to lower the barrier of entry into this field, a simplified version
of a DF dataset, which is designed to be more easily understandable, has been
described. This is accompanied by the release of a basic implementation of a
DF system, which should allow interested researchers to easily implement their
own designs and extensions.

## References

  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018.  MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5016–5026, Brussels, Belgium. Association for Computational Linguistics.
  * Mansimov and Zhang (2021) Elman Mansimov and Yi Zhang. 2021.  Semantic parsing in task-oriented dialog with recursive insertion-based encoder.
  * Platanios et al. (2021) Emmanouil Antonios Platanios, Adam Pauls, Subhro Roy, Yuchen Zhang, Alexander Kyte, Alan Guo, Sam Thomson, Jayant Krishnamurthy, Jason Wolfe, Jacob Andreas, and Dan Klein. 2021.  Value-agnostic conversational semantic parsing.  In _ACL-IJCNLP 2021_.
  * Semantic Machines et al. (2020) Semantic Machines et al. 2020.  Task-oriented dialogue as dataflow synthesis.  _Transactions of the Association for Computational Linguistics_ , 8:556–571.
  * Yin et al. (2021) Pengcheng Yin, Hao Fang, Graham Neubig, Adam Pauls, Emmanouil Antonios Platanios, Yu Su, Sam Thomson, and Jacob Andreas. 2021.  Compositional generalization for neural semantic parsing via span-level supervised attention.  In _2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2810–2823. Association for Computational Linguistics.

##  Appendix A Annotation Simplification Examples

In this section, two examples are given for annotation simplification.

The annotations are shown in both the text and the computational graph
formats. The graphs are constructed from the text, and are basically
equivalent to the text format, but slight modifications occur during the graph
construction process.

Some details are given as to how the simplification principles mentioned in
section 5.2 are applied to the example annotations.

These examples are intended to demonstrate the feasibility of automatically
simplifying the original annotation (further, or different, simplification is
clearly possible).

More important than the simplification process is the final form of the
simplified annotations. The goal of this paper is to demonstrate that concise,
understandable, and yet fully descriptive and executable annotations are
possible, and that implementing them does not have to be complex.

The simplified form of the annotation can be used by the pipeline described in
(Semantic Machines et al., 2020), or by the methods described e.g. in
(Platanios et al., 2021), (Yin et al., 2021), (Mansimov and Zhang, 2021).

###  A.1 Example 1

In this example, the user request is:

"Cancel my 10 AM tomorrow"

Figure 1 and 3 show the original annotation for this request in text and graph
forms, respectively, while figure 2 and 4 show the same for the simplified
annotation.

As explained in 5.2, simplification rules are used to remove operations which
can be inferred from context. In this example:

Figure 1:  Example 1 - Original annotation Figure 2:  Example 1 - Simplified
annotation

  * •

The Yield function is removed, as it is added as the top level function in
every annotation. At execution time, a wrapping Yield is automatically added
back.

  * •

The event deletion operation is originally realized as a two step process
(DeletePreflightEventWrapper, DeleteCommitEventWrapper), first ensuring the
correct event is found, requesting confirmation from the user, and then
deleting it through a call to an external API. In the original annotation,
these steps are explicitly mentioned, but in the simplified version, this is
replaced with one operation (DeleteEvent). At execution time, this is
converted back to two separate functions.

  * •

The input to the delete event function is originally an event id (presumably a
unique integer identifying the event), and the id is explicitly extracted from
the found event in a separate step. In the simplified annotation, there is no
need for this additional step. Instead, the implementation of DeleteEvent
knows how to handle different input types. Specifically, it can apply the
necessary conversions: if the input is an Int, it will be used as the event
id. If the input is an Event, it will be used as is. If the input is multiple
Events, then the singleton function will be added to wrap the input. If the
input is a specification of an event, then a FindEvents function is added to
wrap the input.

  * •

The original annotation calls the function DateAtTimeWithDefaults, presumably
applying some logic to fill missing time information. In the simplified
annotation, this kind of logic is moved from the annotation to the function
implementation.

  * •

The original function EventOnDateTime, with an empty "formal" parameter
"Constraint[Event]" is removed in the simplification.

  * •

The new function starts_at (which can handle various types of input) replaces
the old functions to specify the time constraints.

The resulting simplified annotation is more concise. More importantly, it is,
arguably, more understandable, and "closer" to the user request, without
losing any necessary information.

Figure 3:  Example 1 - Original graph Figure 4:  Example 1 - Simplified graph

###  A.2 Example 2

In this example, the user request is:

"Change on Sunday at Jeffs from 10:00 to 10:30 AM to 10:00 am to 2:00 pm."

Figure 5 and 7 show the original annotation, in text and graph formats,
respectively, for the second example. Figures 6 and 8 show the same for the
simplified annotation.

This example annotation uses an assignment construct, which defines separate
mini-graphs which can be re-used within one annotation. While useful, it can
make the annotation more difficult to understand, and require more effort to
annotate.

Assignments are kept in the simplified annotation, but specific simplification
rules try to remove unnecessary use of assignments. Alternative annotation
designs could altogether avoid the definition of separate mini-graphs,
directly reusing parts of the main graph.

In this example, the assignments can automatically be completely removed by
the simplification rules, as after successive simplification steps, no mini-
graph is used in multiple places.

Explanation of the original annotation and its simplification:

  * •

Assigns label x0 to "Sunday at 10:00AM"

  * •

Assigns the label x1 to the computation: "find an event at location ’Jeffs’,
starting at x0, and ending at 10:30AM after x0". The fragment "start at x0 and
end after x0" is simplified to just "start at x0". After this, x0 appears only
once in the graph, so the x0 mini-graph can be directly attached to the x1
mini-graph.

  * •

Assigns the label x2 to "the date of x1 (the found event), and time 10AM"

  * •

The main computation then tries to update x1 (the found event) to start at x2
(10AM at the date of x1), and end at 2PM after x2. Updating x1 to have the
date of x1 is automatically removed, leaving just "start at 10AM, end at 2PM".
After this, x2 is not used any more, and x1 is used only once, so it is
attached to the main computation.

Again, the simplified annotation is shorter and, arguably, clearer, while
retaining all the necessary information of the user request.

Figure 5:  Example 2 - Original annotation Figure 6:  Example 2 - Simplified
annotation Figure 7:  Example 2 - Original graph Figure 8:  Example 2 -
Simplified graph

##  Appendix B Example Dataflow Function

In order to demonstrate that writing DF programs is not complex, this section
shows an example implementation of a simple DF function, and using it in a
non-trivial mini dialogue, taking advantage of the DF framework functionality.

The code snippet implements the simple function of integer addition. The
Python code in figure 9 is (almost) executable by the released DF execution
code (minor details are modified for the sake of clarity).

The definition consists of Three main blocks:

  * •

Function signature definition. In this case, declaring two inputs of type Int
(named pos1, pos2), and setting the output type to Int.

  * •

Validity checks on the inputs (optional). In this example, missing input will
trigger the exception mechanism.

  * •

The code executing the function (optional). In this case, the values of the
inputs are added, and a new Int node, with its value set to the calculated
sum, is created and attached as result.

Figure 9:  Code snippet implementing ’Add’

Once this function is defined, the execution engine can execute DF expressions
using this function. For example, executing the expressions:

    ’Add(2,Add(3,5))’
    ’revise(old=Int?(3), new=Int(6))’

(corresponding to the user requests: "add 2 to the sum of 3 and 5", followed
by

"make it 6 instead of 3"), builds the graphs shown in figure 10.

For the first expression, it checks that inputs to all nodes are valid, and
recursively executes the calculation, setting the results of all the nodes to
the corresponding addition results.

Figure 10:  Graph context after executing two user turns. Right: result of
first turn, using the Add function (blue dashed arrows point to result nodes).
Left: Result of executing the revise turn: the graph whose root is node
$Add_{0}$ is duplicated as a graph whose root is $Add_{11}$, sharing inputs
with the old graph, but replacing the original input (3) to node $Add_{2}$ by
the new input (6) to $Add_{10}$ (which is the duplication of $Add_{2}$).

For the second expression, the system finds the relevant computational graph
and the node to be replaced, and then creates a duplicate of the graph,
replacing the old input node with the new one, and finally executing
(evaluating) the new graph.