# Response-act Guided Reinforced Dialogue Generation
for Mental Health Counseling

Aseem Srivastava$1$, Ishan Pandey$1$, Md. Shad Akhtar$1$, Tanmoy
Chakraborty$2$ $1$IIIT-Delhi, India; $2$IIT Delhi, India  aseems, ishan20304,
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract.

Virtual Mental Health Assistants (VMHAs) have become a prevalent method for
receiving mental health counseling in the digital healthcare space. An
assistive counseling conversation commences with natural open-ended topics to
familiarize the client with the environment and later converges into more
fine-grained domain-specific topics. Unlike other conversational systems,
which are categorized as open-domain or task-oriented systems, VMHAs possess a
hybrid conversational flow. These counseling bots need to comprehend various
aspects of the conversation, such as dialogue-acts, intents, etc., to engage
the client in an effective and appropriate conversation. Although the surge in
digital health research highlights applications of many general-purpose
response generation systems, they are barely suitable in the mental health
domain – the prime reason is the lack of understanding in the mental health
counseling conversation. Moreover, in general, dialogue-act guided response
generators are either limited to a template-based paradigm or lack appropriate
semantics in dialogue generation. To this end, we propose READER – a REsponse-
Act guided reinforced Dialogue genERation model for the mental health
counseling conversations. READER is built on transformer to jointly predict a
potential dialogue-act $d_{t+1}$ for the next utterance (aka response-act) and
to generate an appropriate response ($u_{t+1}$). Through the transformer-
reinforcement-learning (TRL) with Proximal Policy Optimization (PPO), we guide
the response generator to abide by $d_{t+1}$ and ensure the semantic richness
of the responses via BERTScore in our reward computation. We evaluate READER
on HOPE, a benchmark counseling conversation dataset and observe that it
outperforms several baselines across several evaluation metrics – METEOR,
ROUGE, and BERTScore. We also furnish extensive qualitative and quantitative
analyses on results, including error analysis, human evaluation, etc.

††copyright: none††conference: Make sure to enter the correct conference title
from your rights confirmation emai; April 03–May 04, 2023; Texas, USA Figure
1. A sample counseling conversation along with associated dialogue-acts. The
proposed model – READER takes utterance- and dialogue-act context to predict
response-act and subsequently generate a response.

##  1\. Introduction

Virtual Mental Health Assistants (VMHAs) are the backbone of the new-age
digital healthcare industry. More than 60% of therapies conducted in the past
three years are via virtual assistants. This massive spike in the number of
users using VMHAs to gain mental health assistance is due to the ease and
safety of access to AI-based therapist-bots (Abd-Alrazaq et al., 2021).
Numerous potential platforms, viz. Weobot111https://woebothealth.com/,
Wysa222https://www.wysa.io/, etc. in the digital health space are developing
practical and effective ways for the common public. More popular than ever,
VMHAs are now becoming an instant solution to millions of clients struggling
with mental health
issues333https://psychnews.psychiatryonline.org/doi/10.1176/appi.pn.2022.05.4.50.

Limitations of existing methods. Many such conversational agents fail to (a)
understand the directives of the client with whom they are in active
conversation, and (b) take the conversation in the required
direction444https://www.bbc.com/news/technology-46507900. This is resemblant
to the fact that even human therapists find it impossible to reply to
something they do not understand from the help-seeker. Therefore, clients’
directives directly impact the response generation capability. Current open
domain conversational systems viz. XiaoIce (Zhou et al., 2020) and GPT-3-based
systems (Zhang and Li, 2021) generate semantically and grammatically rich
responses. However, these open-domain counseling systems lack contextual
understanding in the response generation process, which includes being unable
to respond with the intended dialogue-act. Therefore, modeling this problem
using open-domain dialogue systems cannot suffice the task of goal-oriented
dialogue systems. To mitigate the issue in the mental healthcare domain, there
is a need to harmonize VMHA’s dialogue with individual intentions to be useful
for clinical practice. A very generic solution to this is to design a model
that gauges the dialogue context and predicts the next dialogue-act (aka
response-act), which collectively helps generate the next utterance.

Our approach. Our work focuses on response generation by exploiting response-
acts. To understand the problem better, Figure 1 shows an example of a
counseling conversation. As we observe, the dialogue contains both therapist’s
and client’s utterances, each possessing a dialogue-act that is critical in
maintaining the flow of the conversation. Evidently, the dialogue-acts of the
utterances generally form a pattern. For instance, the dialogue-act of the
third utterance from the therapist is information-request, which in succession
is followed by information-delivery in the fourth utterance by the client.
Earlier approaches (Noble and Maraev, 2021; Chao et al., 2021; Saha et al.,
2020) exploited dialogue-act and context to build rich representation for
several tasks on dialogue system. Another work proposes a dialogue management
strategy in order to improvise on response generation task exploiting fine-
grained belief states (Hosseini-Asl et al., 2020). These fine-grained belief
states are task-specific, and their proposed system, SimpleTOD, needs
relatively more information (specific slots) in order to generate a response.
At the same time, exploiting the slot-filling task to generate responses
supports the dialogue system in most goal-oriented cases. However, counseling
conversations cannot be categorized either into open-ended or goal-oriented
dialogue and hence needs a separate focus on the hybrid conversational
pattern. On the other hand, several studies utilize other guiding factors such
as keyword, target, etc. for the response generation task (Gupta et al., 2022;
Tang et al., 2019). In our work, we predict the dialogue-act of the next
utterance aka response-act and take advantage of state-of-the-art language
models to generate relevant responses. At the same time, contextual
information in the conversation plays an essential role in developing a full-
fledged conversational system. To this end, we propose a response-act guided
dialogue generation model, named, READER. It comprises a foundation language
model, on top of which we deploy three unique heads, namely, the response-act
head (RAC-Head), the language model head (LM-Head) and the value head
(V-Head). These three heads jointly learn to optimize the reinforced loss and
primarily perform the response generation task. READER learns by optimizing
Proximal Policy Optimization (PPO), for which we curate a unique reward
function.

Evaluation. We benchmark READER on the HOPE dataset (Malhotra et al., 2022),
which is a dyadic counseling conversation dataset containing $13k$ utterances
from therapist and client. We observe that READER outperforms several
baselines across three relevant quantitative metrics -– METEOR, ROUGE, and
BERTScore, with improvements in the range of $0.82-11.53\%$. In addition, we
also present an extensive qualitative and quantitative analyses of the
performance, error analysis and human evaluation. Furthermore, to evaluate the
generalizability of the READER, we benchmark it over the Switchboard Dialogue-
act corpus (SWDA) (Stolcke et al., 2000) and obtain better results than
baselines by $0.1-9.4\%$.

Major Contributions. Below, we summarize the contributions:

  * •

We exploit future dialogue-acts (aka response-acts) in guiding the response
generation model to generate the intended response and maintain the flow of
counselling conversation in the mental-health domain. To the best of our
knowledge, ours is one of the first attempts that exploits response-acts to
generate precise responses in VMHAs or any other dialogue systems.

  * •

We propose a novel transformer-reinforcement-learning (TRL) driven response-
act guided model, READER to generate response in mental health counseling
conversations.

  * •

Our evaluation on the HOPE dataset shows significant improvements in the
performance of response generation over several competing baselines. We also
perform extensive ablation analysis and justify the choice of various
components of READER.

  * •

We conduct a through and qualitative human evaluation on the generated
responses and establish that the proposed approach is qualitatively efficient
as well.

  * •

We also show the effective generalizability of READER on another dataset,
i.e., the Switchboard Dialogue-act dataset.

We have open-sourced the code for READER and a sample of the dataset on an
anonymous link555Code and dataset sample: https://bit.ly/3CqpcrK. We commit to
publicly release the source codes and datasets upon acceptance of the paper..

##  2\. Related Work

To bring more clarity in understanding the role of dialogue-acts, we present
relevant studies in two broad areas – (i) dialogue-act classification, and
(ii) dialogue/response generation. We intend to comprehend how dialogue-acts
could bring effective innovation in building a conversational system for a
dedicated task.

Dialogue-act Classification. Earlier studies by Budzianowski et al. (2018) and
Su et al. (2015) employ a sparse representation of each dialogue-act in the
form of triple vectors (domain-action-slot); this triple vector is represented
as a one-hot encoding. However, acts become very large with the use of such
sparse representations. Later, Chen et al. (2019) addressed the issue by
considering dialogue-act structures. Further, the authors represented
dialogue-acts considering the act structures with level-wise vectorization on
a one-hot scale where a binary classifier predicts each dimension of vectors.
Their methods are further improved in a recent work by Pei et al. (2019). The
authors exploited a separate expert decoder for different areas and dialogue-
acts to fuse them with a main chair decoder. A recent work applies a fusion
approach to fuse their language model with a next utterance generation decoder
(Mehri et al., 2019). Several other studies use reinforcement learning to
generate dialogue responses (Ziegler et al., 2019).

Response Generation. Studies on dialogue generation (Peters et al., 2018;
Devlin et al., 2019) showed improved performance by leveraging data corpus
size, which in turn resulted in learning better context-sensitive features
from large language models. Yang et al. (2019) extended this idea further by
deploying models with large parameters. They used a similar idea on XLNet, a
generalized autoregressive pretrained model, in order to (i) maximizing the
expected likelihood over all permutations of the factorization order allowing
learning of bidirectional contexts, and (ii) coping up with the drawbacks of
BERT by leveraging the proposed approach’s autoregressive formulation.

Later Radford et al. (2019) explored their hypotheses of the zero-shot
learning capacity of large language models as multi-task learners on the task
of response generation using GPT-2. The authors also showed an intuitive
qualitative analysis of a sample to fetch quality insights. The analysis shows
the reflection of coherent responses to prompts. The result presents a better
path toward building a response generation system that learns to perform the
task from their naturally occurring demonstrations. A recent study on
transformer-based models has been fine-tuned for dialogue modeling through
various data modification techniques. This includes methods such as adding
information about the user’s persona, masking, etc. (Wolf et al., 2019).

Figure 2. Architecture of READER. It contains three heads on top of the
foundation language model, GPT-2: (a) RAC-Head classifies the response-act
trained on context-aware representations, (b) LM-Head generates the response,
and (c) V-Head calculates the final reward and initiates Proximal Policy
Optimization (PPO).

At the same time, studies by Xu et al. (2019) control responses using meta-
words and manually controlled features (viz. length of response, specificity,
etc.). They defined a meta-word as an organized record. The authors further
described the response attributes. This allows them to model the relationship
(one-to-many) within task-independent conversations and execute the problem of
generating a response in an explainable and controllable manner. Hosseini-Asl
et al. (2020) proposed a simpler architecture which relies on the belief-
states generated by the dialogue management module. These belief states are
similar to fine-grained intents and slots, exploiting which the authors aimed
for the dialogue-generation task. Further, Khalifa et al. (2021) proposed a
distributional method for handling text generation in a controllable manner by
exploiting language models. This method allows point-wise specification of
details and distributional constraints on the target language model in one
standard framework. Their work is the first effort into this concept while
minimizing relative entropy from the earlier proposed language model
distribution. They uniquely defined the optimal target distribution as an
explicit EBM (Energy-Based Model) representation. Moreover, using those
optimal representations, we train a target-controlled autoregressive language
model through an adaptive distributional variant of the policy gradient. They
conducted experiments on point-wise constraints and showed the advantages of
their method over traditional fine-tuning methods. Furthermore, one of the
studies on dialogue modeling (Lippe et al., 2020) propose to combine the
merits of template-based and corpus-based DRGs by introducing a prototype-
based, paraphrasing neural network, called P2-Net, which aims to enhance
quality of the responses in terms of both precision and diversity. Instead of
generating a response from scratch, they generate system responses by
paraphrasing template-based responses. Their approach learns to separate a
response into its semantics, context influence, and paraphrasing noise, and to
keep the semantics unchanged during paraphrasing.

##  3\. Proposed Approach: READER

In a regular conversation, dialogue-acts of the interlocutors tend to form a
pattern. For instance, if person $A$ seeks some clarification from person $B$,
the most probable response from $B$ would be to elucidate the clarification
raised by $A$. Therefore, leveraging the above connotation, we propose to
utilize the next dialog-act (or response-act) in the response generation task.
Formally, we formulate the problem as follows:

  * Given a counseling dialogue containing utterances and their corresponding dialogue-acts as $U\in\\{u_{0},u_{1},...,u_{t-1},u_{t}\\}$ and $DA\in\\{d_{0},d_{1},...,d_{t-1},d_{t}\\}$ respectively, where $t$ is the time step, our twofold jointly-learned tasks are – (a) to predict the response-act $d_{t+1}$ (auxiliary), and (b) to generate a response $u_{t+1}$ in the dialogue abiding by the predicted response-act $d_{t+1}$ (primary).

To this end, we propose READER, a novel response-act guided reinforced
response generation model. The architecture of READER is presented in Figure
2. READER leans on the joint transfer-reinforcement-learning (TRL) paradigm
for generating response-acts and responses. Our method of transformer
reinforcement learning takes inspiration from an earlier work (Ziegler et al.,
2019) . Moreover, we train the foundation language model with Proximal Policy
Optimization (PPO) (Schulman et al., 2017). We define a vocabulary $\Sigma$
and the foundation language model $\theta$ (in our case, GPT2) that defines a
probability distribution over sequences of tokens.

On top of the foundation language model, we place three task-cum-learning
specific heads. First, the language model head (LM-Head) is generalized for
text generation tasks. Secondly, we introduce a response-act classification
head (RAC-Head), an encoder-only model to classify response-acts. At last, we
have a value head (V-Head) to compute the reward to send back to the
foundation model. Next, we train the model jointly to generate responses from
LM-Head and predict response-acts from RAC-Head simultaneously. Subsequently,
V-Head computes the reward considering the scores of LM-Head and RAC-Head,
which in turn is optimized via PPO. We furnish details related to each head
and the reward computation in subsequent sections.

RAC-Head. Dialogue-acts play an essential role in articulating dialogue flow.
RAC-Head is a transformer-based encoder-only module on top of the foundation
language model that learns to predict the future response-act. The head
exploits the last hidden representations of the foundation language model. We
feed the hidden representations to a GRU to exploit the contextual pattern of
the dialog. In parallel, we obtain linear projections of the hidden
representation. Next, these contextually-rich representations are passed
through a multi-head attention module in which we treat the GRU
representations as the query and the linear projections as the key and value.
Finally, we apply softmax to classify a response-act. The prediction
calibrates READER to adapt the PPO optimization through the RAC-Head’s logits,
thus allowing LM-Head to generate an appropriate response.

LM-Head. We use GPT-2666https://openai.com/blog/better-language-models/ as our
foundation language model. It has been established as one of the preferred
models for a variety of generative tasks (Dingliwal et al., 2021; Gretz et
al., 2020; Lin et al., 2021; Yang et al., 2020).

###  3.1. Reference Models for Reward Computation

We aim to augment the response by inheriting adequate semantics and response-
acts. To maintain the stability of the reward function, we deploy state-of-
the-art reference models to compare the outcomes for both tasks. For the
language model head (LM-Head), we employ the pre-trained GPT-2 model as the
reference model, whereas, SPARTA (Malhotra et al., 2022) is used for the
response-act head (RAC-Head). Subsequently, we compute ROUGE ($R$), BERTScore
($BS$), and relative entropy ($RE$) between the proposed and the reference
language model’s outputs. Unlike primitive methods of RL-training with a
standard reward function where the model deviates to learn biased features in
order to maximize the reward, where for instance, the model may start copying
text from reference text to maximize ROUGE, Our approach employs each metric
to calculate the reward function and tracks the relative entropy with the
performance of reference model in parallel. It ensures that READER’s
prediction does not deviate significantly and leverages the semantic richness
of the pre-trained reference language model. We calculate the relative entropy
as follows:

(1) $RE=\mathbb{E}_{z\sim P_{lm}}[\log P_{lm_{p}}(z)-\log P_{lm_{ref}}(z)]$

where $z$ is sampled from $P_{lm}$, and $lm_{p}$ refers to the proposed
language model; whereas $lm_{ref}$ refers to the reference model. A lower $RE$
score demonstrates better generations; therefore, we employ $RE$ as a direct
parameter in the reward computation.

Similarly, we utilize SPARTA (Malhotra et al., 2022) to compute the logit
values for the predicted dialogue-act and apply mean-pooling for the reward
computation.

V-Head. The value head (V-Head) is responsible for accumulating the reward
parameters from other heads to yield the reward and subsequently, use it to
reinforce the READER.

For our reward function, we use metrics including BERTScore and ROGUE Score.
Along with these, we use known response-acts to train a reward model (SPARTA),
and then optimize that reward model.

Our proposed reward function accumulates the weighted Rouge score ($R$),
BERTScore ($BS$), the relative entropy ($RE$), and the DAC-Head’s logit value
($\rho$). The former three components ($R,BS,RE$) offer feedback on the
semantic and syntactic richness of the current state, while the last component
($\rho$) guides the model towards the desired response, exhibiting the
predicted response-act. We compute the reward as follows:

(2)
$Reward=[\lambda{{}_{1}}(R)+\lambda{{}_{2}}(BS)+\lambda{{}_{3}}(\rho)-\lambda{{}_{4}}RE]$

where $\lambda_{1},\lambda_{2},\lambda_{3},$ and $\lambda_{4}$ are
hyperparameters and tuned to optimize and maximize the reward. Subsequently,
we reinforce the yielded reward to optimize the current state using PPO.

###  3.2. Training and Proximal Policy Optimization

Similar to the optimization policy explored by Ziegler et al. (2019) on a
general-purpose task, the training of GPT-2 with PPO in READER is a three-step
process:

  * •

Initiate RAC-Head and LM-Head: Given $u_{t}$, $d_{t}$ along with the context
$\\{<u_{t-k},d_{t-k}>,...,<u_{t-1},d_{t-1}>\\}$, where $k$ is the context
size, READER generates a response-act and the response.

  * •

Evaluate Outcomes: In this step, we calculate the log-probability distribution
of logits from the active component (trainable model) of LM-Head and RAC-Head.
Subsequently, we obtain the difference in the log probability distribution of
reference model which is used to impose penalty and ensure coherency of the
outputs.

  * •

PPO Optimization: We choose ROUGE, BERTScore, and Relative entropy to assess
the quality of the generated response and the max-logit scores in case of the
response-act classification task. We receive the reward score from V-Head
(c.f. Equation 2). In order to optimize READER, we first compute the relative
entropy from LM-Head. At the same time, LM-Head and RAC-Head yield the
remaining reward parameters. At last, V-Head accumulates and computes the
reward from each head as per Equation 2. We perform optimization that
subsequently allows READER to learn to penalize/reward the foundation language
model.

This approach is extended from an earlier work (Ziegler et al., 2019). The
authors initialized a policy $\pi=\theta$, and then fine-tuned $\pi$ to
operate on downstream tasks using PPO. If the task is defined by a reward
function $(r:X\times Y\rightarrow R)$, then the authors used PPO to optimize
the expected reward. In their algorithm, PPO utilizes clipped surrogate
objective, and the model maximizes a surrogate objective. Another study
Ziegler et al. (2019) exploited the usage of PPO algorithm to further define
the downstream task to optimize the main objective function. The authors opted
for minimum of the clipped and unclipped objective. Hence the final objective
is lower bound (i.e., a pessimistic bound) on the unclipped objective. With
this scheme, we observe that only probability ratio is ignored when it
improves the objective.

As a result, we then exploit the above mentioned PPO method to optimize our
PPO algorithm (Ziegler et al., 2019) with the following equation.

(3) $R(x,y)=r(x,y)-\beta\cdot\log\pi(y|x)\cdot\theta(y|x)$

where $r$ and $\theta$ represent reward function and foundation language
model, respectively. In our case, we experiment with a constant as well as
dynamic $\beta$ to achieve a favorable value of $RE(\pi,\theta)$. The relative
entropy plays the role of an entropy bonus; it prevents the policy from moving
too far from the range where $r$ is valid. We rely on the relative entropy to
sync with the fine-tuned reference model’s coherent responses.

Table 1. Statistics of the HOPE dataset (Malhotra et al., 2022). The dyadic
counseling conversational dataset contains a total of 12.8k utterances, each
associated with one of the 12 dialogue-act labels.

HOPE | Train | Validation | Test | Total
---|---|---|---|---
Dialogue Sessions | 149 | 21 | 43 | 212
Client Utterances | 4668 | 595 | 1119 | 6382
Therapist Utterances | 4751 | 599 | 1122 | 6472
1-5 #Total Utterances | 9419 | 1194 | 2241 | 12854

##  4\. Experiments

In this section, we first discuss the counseling dataset, HOPE. We then define
the baseline systems and evaluation metrics which we use to compare the
performance of the proposed model and baselines.

###  4.1. Dataset

We use HOPE (Malhotra et al., 2022), a mental health counseling conversation
dataset. It contains $12.8K$ utterances from $212$ dyadic counseling sessions
between therapists and clients, publicly available on a video sharing
platform. The conversation encompasses diverse demographic groups with
distinct mental health discussions. Malhotra et al. (2022) extracted
transcriptions of the utterances and processed them to remove any noise and/or
transcription issues. The collected dialogues are dyadic in nature, i.e.,
clients and therapists are the only interlocutors. Each utterance in the HOPE
dataset is annotated with one of the twelve dialogue-acts – information-
delivery (ID), information-request (IRQ), yes/no-question (YNQ),
clarification-request (CRQ), opinion-request (ORQ), clarification-delivery
(CD), positive-answer (PA), negative-answer (NA), opinion-delivery (OD),
greeting (GT), acknowledgment (ACK), general chit-chat (GC). A detailed
statistics of the HOPE dataset is presented in Table 1. Furthermore, we
demonstrate the relation between the dialogue-acts of current and next
utterances in the HOPE dataset in Appendix (c.f. Figure 6). Evidently, the
dataset shows a high correlation between certain pairs of dialogue- and
response-acts. For instance, an utterance requesting-information (labelled
IRQ) is mostly followed by an utterance delivering the information (labelled
IRD).

###  4.2. Baselines and Evaluation Metrics

We compare READER’s performance with various competitive baselines in the
domain of dialogue generation. Moreover, to have a fair comparison with
READER, we choose systems which leverage and exploit the dialogue context for
the response generation. To the best of our knowledge, no other systems have
reinforced the response-act for the dialog generation. We choose the following
baselines in this work. DialoGPT (Zhang et al., 2020) is a pretrained
transformer model dedicated for response-generation task. GPT-2 (Radford et
al., 2019) is a decoder only model trained on a large corpora. A vanilla
finetuned version of GPT-2 works well in our use case. DialogVED (Chen et al.,
2022) introduces continuous latent variables into the encoder-decoder pre-
training framework to increase the relevance and diversity of responses.
ProphetNet-Dialog (Qi et al., 2021) focuses on pretraining of dialogue
specific corpus to generate coherent response. HRED (Serban et al., 2016) is
based on generative modeling to develop conversational response containing
hierarchical encoder-decoder paradigm. It is trained on a large dialogue
corpus for the utterance generation task. HRED with Speaker and Utterance
Encoder (Zhao and Kawahara, 2019) adds speaker and utterance level information
to the hierarchical encoder-decoder (HRED) setup. It leverages the
personalization parameters in a dialogue system. It is also trained on a large
dialogue corpus. VHCR  (Park et al., 2018) uses variational hierarchical RNNs
for the conversation-only setup. It is trained on a large conversation corpus
for the dialogue modeling task.

Table 2. Results obtained on the HOPE dataset. We show ROUGE (1, 2, L),
BERTScore (BS), and METEOR to assess the performance of the READER. $Rew(x)$
is the reward function, where $x$ is the parameter.

|  | R1 | R2 | RL | BS | METEOR
---|---|---|---|---|---|---
| P | R | F1 | P | R | F1 | P | R | F1
Baselines | DialoGPT (Zhang et al., 2020) | 12.34 | 40.48 | 15.72 | 2.92 | 11.83 | 4.42 | 12.23 | 38.60 | 15.76 | 0.7603 | 0.2021
GPT2 (Radford et al., 2019) | 12.70 | 32.63 | 14.98 | 3.08 | 7.92 | 3.51 | 13.74 | 32.05 | 15.87 | 0.7445 | 0.1754
DialogVED (Chen et al., 2022) | 12.48 | 31.74 | 12.8 | 0.98 | 2.45 | 1.22 | 12.45 | 31.11 | 14.46 | 0.7189 | 0.2000
ProphetNet (Qi et al., 2021) | 12.15 | 34.29 | 14.48 | 3.30 | 10.41 | 4.17 | 12.24 | 33.12 | 15.18 | 0.6707 | 0.1901
VHCR (Park et al., 2018) | 11.29 | 21.33 | 11.81 | 2.66 | 3.49 | 3.00 | 10.01 | 19.72 | 10.99 | 0.5953 | 0.1041
HRED (Serban et al., 2016) | 11.52 | 21.51 | 10.72 | 1.89 | 6.42 | 2.92 | 12.12 | 24.36 | 13.56 | 0.6259 | 0.1425
HRED w/ Sp. Utt. Encoder (Zhao and Kawahara, 2019) | 11.77 | 28.63 | 10.08 | 1.29 | 4.19 | 2.06 | 12.25 | 21.27 | 12.72 | 0.6171 | 0.1801
Ours | RagRes w/ DialoGPT | 12.41 | 43.91 | 16.12 | 3.70 | 13.72 | 4.98 | 11.92 | 41.02 | 16.30 | 0.7656 | 0.2098
READER – RAC-Head | 12.64 | 41.48 | 15.78 | 3.60 | 11.83 | 4.58 | 12.3 | 38.64 | 15.90 | 0.7628 | 0.2039
READER | 12.82 | 43.93 | 16.15 | 3.77 | 13.67 | 4.93 | 12.51 | 40.82 | 16.32 | 0.7666 | 0.2103
Ablations |  – $Rew($R$)$ | $11.73$ | $38.82$ | $14.65$ | 2.28 | $8.45$ | $2.96$ | 11.21 | 35.76 | 14.53 | 0.7561 | 0.1840
– $Rew($RAC$)$ | 12.36 | 40.71 | 15.43 | 3.13 | 11.12 | 4.06 | 11.91 | 37.63 | 15.40 | 0.7609 | 0.2000
– $Rew($RAC + R$)$ | 11.92 | 38.06 | 14.70 | 2.43 | 8.26 | 3.11 | 11.40 | 34.98 | 14.58 | 0.7530 | 0.1874
– $Rew($R + BS$)$ | 12.48 | 41.13 | 15.57 | 3.52 | 11.85 | 4.47 | 12.22 | 38.29 | 15.77 | 0.7527 | 0.2092
– $Rew($RAC + BS$)$ | 12.01 | 40.45 | 15.18 | 2.72 | 9.93 | 3.52 | 11.46 | 37.05 | 14.97 | 0.7577 | 0.1908
| $\Delta_{\texttt{READER}-BEST}(\%)$ | $\uparrow 0.94$ | $\uparrow 8.5$ | $\uparrow 2.73$ | $\uparrow 14.24$ | $\uparrow 15.50$ | $\uparrow 11.53$ | $\downarrow 8.90$ | $\uparrow 5.69$ | $\uparrow 2.83$ | $\uparrow 0.82$ | $\uparrow 4.05$

For evaluating the performances of READER and other comparative systems, we
employ ROUGE, METEOR, and BERTScore as evaluation metrics. We use py-
rouge777https://pypi.org/project/py-rouge/, nltk-
meteor888https://www.nltk.org/api/nltk.translate.meteor_score.html, and
Hugging Face - BERTScore999https://huggingface.co/spaces/evaluate-
metric/bertscore for computing the scores.

##  5\. Results and Analysis

In this section, we discuss the results obtained from the READER model and
aforementioned baselines. Table 2 summarizes the comparative and ablation
results on the HOPE dataset.

###  5.1. Performance Comparison

Our evaluation shows superior performance of READER across a majority of the
metrics. Evidently, there is a significant increase in the recall of the
ROUGE-2 score – our model receives a ROUGE-2 recall of $13.67$, which is
$+15.50\%$ as compared to the second best performer, DialoGPT ($11.83$). At
the same time, our model yields $43.93,40.82,\text{ and }76.66$ scores of
ROUGE-1 recall, ROUGE-L recall, and METEOR, respectively, with an increase of
$+3.45,+2.22,\text{and}+0.63$ points as compared to the best baseline, i.e.
DialoGPT. On the other hand, to evaluate the linguistic properties in the
generated utterances, we calculate METEOR on READER’s generations. Similar to
the earlier cases, READER reports an improved BERTScore of $0.2103$, $+4.05\%$
points better than DialoGPT.

Ablation on Foundation Model. Among all baselines, DialoGPT performs the best
on average with GPT-2 closely competing with it. However, in our case, we find
it suitable to choose GPT-2 as the foundation language model due to the
marginally better performance in READER. We also experiment by swapping it
with DialoGPT and report the results in Table 2. We observe that READER with
GPT-2 performs better on recall scores of R1 $(+0.41)$, R2 $(+0.02)$, RL
$(+0.2)$, BS $(+0.0139)$ as compared to READER with DialoGPT.

Model Component Ablation Study. One of the prime contributions of this work
involves the role of the RAC-Head. READER relies on the RAC-Head to determine
the response-act of the generated utterances. Further, RAC-Head gradually
allows the response generation to adapt according to the predicted response-
acts during the PPO optimization. We perform ablation on RAC-Head and present
the results in Table 2. We observe that READER without RAC-Head scores
$41.48$, $11.83$, $38.64$ on ROUGE-1, 2, L (recall scores), respectively. The
final model performs relatively better with an increased score
($+2.45,+1.84,+2.18$) on the same metrics as compared to the READER without
RAC-Head.

Table 3. An example showing the generated response along with the reference
response.

Context |  Next utterance | RAC
---|---|---
Therapist: Oh great, which is a little different. But yeah, that’s great. That’s a really good goal. |  Gold: Okay, so you’re getting A’s and B’s in really good numbers. So what did you discover about this? | CRQ
Client: Yeah, thanks. |  READER: Okay. Okay. What classes are you not getting A’s and B’s in? | CRQ
Therapist: It’s hard work to get into law school. Are you doing well in school? What kind of grades you get? |  GPT2: Anyway, I should mention I’m probably going to get through that anyway. That might not be fun. | –
Client: I’m doing okay. I’m getting mostly A’s and B’s so I’m pretty good. |  DialoGPT: Are you going to law school? | –
Client: My employer just said I wasn’t a good fit. |  Gold: My anxiety is really bad. | ID
Therapist: wasn’t a good fit. So this so these symptoms have caused you some real problems. |  READER: Oh, I get really anxious. just scared that something’s going to happen and your anxiety is going to be costly. | ID
Client: Yeah. |  GPT2:That’s it. Lerner. It’is a honest fact and it is one piece of information about your mind. Lerner | –
Therapist: With employment and how about with your anxiety level?
DialoGPT: I’m not sure if you’re joking or not. | –

Discussion on Reward Selection and Reward Ablation. We meticulously conduct
experiments on several hypotheses to design a reward function that optimizes
the PPO policy and penalizes the model for every shortcoming. While
experimentation, we consider several metric scores as a parameter to the
reward function (c.f. Equation 2). However, most of the parametric
configurations deteriorate the results. We show various possible ablations on
the final set of parameters, i.e., ROUGE, BERTScore, and RAC-Head’s logits in
the lower half of Table 2. We observe that a combination of RAC-Head along
with BERT Score, ROUGE scores, and relative entropy yields the best result. In
addition, using only BERTScore or ROUGE scores in the reward function
deteriorates the results significantly. We also observe a decrease in the
metrics after detaching RAC-Head. This justifies the contribution of both the
heads of READER toward the generation task. Moreover, Figure 3 shows the
incremental graph of the reward function during the PPO optimization.
Evidently, the plot demonstrates the progressive learning curve considering
the mean reward score, and supports our claim that the model improves with the
increasing step.

###  5.2. Qualitative Analysis

To further assess the quality of the generated responses of READER, we present
a qualitative analysis in Table 3. In comparison with the reference utterance,
we observe that semantics and grammar are well established in the generated
response. Additionally, we observe that the response-act of the generated
response is in accordance to the intended act. It suggests the effectiveness
of RAC-head in reward computation. We also show generated utterances for the
two best performing baselines (DialoGPT and GPT). Though the outputs are
syntactically correct, they are mostly incoherent with the dialogue context.

Figure 3. The increment flow of reward in the PPO optimization of READER.
Table 4. Human evaluation on the responses generated from READER when compared
to the top two best performing baselines. We observe that the performance of
READER across all metrics is up to the mark and slightly better than the best-
performing dialogue models.

Model | Relevance | Consistency | Fluency | Coherence
---|---|---|---|---
DialoGPT | 2.11 | 2.42 | 2.90 | 2.30
GPT2 | 2.70 | 3.00 | 3.01 | 2.44
1-5 READER | 2.85 | 3.05 | 3.05 | 2.95

Human Evaluation We also perform human evaluation on a subset of model
outcomes on linguistic ground. We use four linguistic parameters, namely,
relevance, consistency, fluency, and coherence, to perform the human
evaluation (van der Lee et al., 2021). We define these parameters as follows:
Fluency demonstrates the linguistic quality of the generated responses;
Coherence shows the structure and organization of the generated responses;
Relevance shows the selection of relevant content in the generated response
considering the reference utterance; and Consistency evaluates the factual
alignment between the generated response and the source utterance.

In total, we take 50 randomly-selected instances and ask 10 human evaluators
to assign a score on a scale of $[1,5]$ to each of the four parameters, where
$5$ represents the best outcome. All human evaluators are linguistic experts,
aged between 20 to 35. For comparison, we repeat the exercise for DialoGPT and
GPT2 as well. Finally, we compute the average score and report the findings in
Table 4. Our analysis shows that READER’s outputs are also qualitatively
better than baselines in each dimension.

###  5.3. Application of READER: Dialogue Generation

In this section, we present the application of READER for generating
counseling dialogues. To do so, we adopt two setups: a) Natural: an end-to-end
conversation between a client and an agent (READERTherapist); and b)
Synthetic: an end-to-end conversation between two agents, i.e.,
READERTherapist and READERClient. The first setup is a natural configuration
for VMHAs, when deployed at the application stage, it generates therapist
utterances to interact with real-time clients having mental health issues. To
do so, at every step $i$ of the response generation, we provide actual client
inputs and the previously-generated READER’s outputs for therapist
($\\{i-n,\cdots,i-2,i-1\\}$) as the recurring context to READER. On the other
hand, the second setup is an analysis configuration to assess the
effectiveness of READER in handling diverse inputs (e.g., generated by an
agent). Moreover, this can also be viewed as a data augmentation technique to
generate synthetic dialogues. In this setup, we provide READER’s generated
outputs for both client and therapist as context.

Furthermore, in both setups, we assume that a context is present to instigate
the conversation, such that the agent (or READER in our case) understands the
dynamics of the conversation and starts generating responses that are aligned
with the conversation. This approach is similar to existing VMHAs, like
WoeBot101010https://woebothealth.com/, where the agent collects initial
information from the client in terms of template-based questions and propels
the conversation further with the provided details. A snippet of the generated
dialogue for the two setups are presented in Figure 4 and Figure 5,
respectively. Evidently, we observe that the proposed model is able to
comprehend the context of the conversation in both setups and generates
aligned responses.

###  5.4. Generalizability

READER outperforms several baselines across most of the metrics on the HOPE
dataset. Further, to assess the model’s generalizability, we extend our
experiments and evaluate READER on the Switchboard Dialog-act corpus (Stolcke
et al., 2000). We observe that READER improves the performances of two best
performing baselines (DialoGPT and GPT2) by $0.1\%-9.4\%$ in $10$ out of $11$
metrics. In particular, we observe a significant improvement of $9.4\%$ in
BERTScore; thus suggesting that READER’s outputs are semantically richer than
other baselines along with the marginal improvements in textual similarity. We
argue that in the presence of the information of dialogue-acts, READER
harnesses the context in an efficient way for generating semantically-richer
responses. In conclusion, we posit that READER generalized well over other
domains as well. Due to the space constraints, we furnish the results along
with the baseline’s performance on the Switchboard dataset in Appendix (c.f.
Table 6).

##  6\. Discussion

Societal Impact and Deployment. Our work acts as a support to the mental
health community and ongoing research by leveraging the advancements in AI-
based dialogue systems for counseling. Such advancements in the mental health
domain are likely to bring a high social impact. Further, to put this paper’s
ideas into practice, we are in active collaboration with a prominent mental
health service provider. Collaborators have verified the model’s applicability
in the real world and agreed to extend READER on a bigger corpus and
commercialize it. The results of the A/B testing are suppressed due to
company’s privacy issues.

Ethical Considerations and Future Work. Considering the severity of the
research area, we make sure that at each step, we maintain the privacy of the
personal data of clients. In future, we plan to extend our work in the
expansion of Virtual Mental Health Assistants (VMHAs) modules and scale the
idea of including dialogue components such as empathetic understanding.

Figure 4. Application of READER – Natural setup. Given a context, at each
step, READER generates an output for the therapist. This align with the
natural configuration of VMHAs, where a client seeks help from a bot or a
virtual agent. Figure 5. Application of READER – Synthetic setup. Given a
context, at each step, READER takes turn to generates outputs for the
therapist and the client. This setup is an analysis configuration to assess
the effectiveness of READER in handling diverse inputs (e.g., generated by an
agent).

##  7\. Conclusion

The continuous need to face the shortage in the number of mental health
experts is becoming a significant challenge every coming year. With new AI-
based therapist-bots coming into the picture, clients receive much support
with ease of access. However, one of the critical tasks for such
conversational agents is to generate an accurate yet effective response for
the clients possessing intended dialogue-act towards the client. To this end,
we proposed a novel response-act guided dialogue generation model, READER. We
designed a unique reward function that exploits several linguistic properties
to train the model using transformer-reinforcement learning (TRL) and further
improvised the PPO optimization. We added three heads on top of the foundation
language model: RAC-Head, Value-Head, and LM-Head, which collectively curate
the reward. We compared the performance of READER with several baselines. Our
model outperformed several baselines across five metrics: ROUGE (1, 2, & L),
METEOR, and BERTScore. At last, we demonstrated an extensive ablation study
and concluded the paper with a discussion on ethical considerations and
generalizability.

###### Acknowledgements.

The authors acknowledge the support of ihub-Anubhuti-iiitd Foundation set up
under the NM-ICPS scheme of the DST.

## References

  * (1)
  * Abd-Alrazaq et al. (2021) Alaa A Abd-Alrazaq, Mohannad Alajlani, Nashva Ali, Kerstin Denecke, Bridgette M Bewick, and Mowafa Househ. 2021.  Perceptions and Opinions of Patients About Mental Health Chatbots: Scoping Review.  _J Med Internet Res_ 23, 1 (13 Jan 2021), e17828.  https://doi.org/10.2196/17828
  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018.  MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Brussels, Belgium, 5016–5026.  https://doi.org/10.18653/v1/D18-1547
  * Chao et al. (2021) Chi Hsiang Chao, Xi Jie Hou, and Yu Ching Chiu. 2021\.  Improve Chit-Chat and QA Sentence Classification in User Messages of Dialogue System using Dialogue Act Embedding. In _Proceedings of the 33rd Conference on Computational Linguistics and Speech Processing (ROCLING 2021)_. The Association for Computational Linguistics and Chinese Language Processing (ACLCLP), Taoyuan, Taiwan, 138–143.  https://aclanthology.org/2021.rocling-1.19
  * Chen et al. (2022) Wei Chen, Yeyun Gong, Song Wang, Bolun Yao, Weizhen Qi, Zhongyu Wei, Xiaowu Hu, Bartuer Zhou, Yi Mao, Weizhu Chen, Biao Cheng, and Nan Duan. 2022\.  DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Dublin, Ireland, 4852–4864.  https://doi.org/10.18653/v1/2022.acl-long.333
  * Chen et al. (2019) Xiuyi Chen, Jiaming Xu, and Bo Xu. 2019.  A Working Memory Model for Task-oriented Dialog Response Generation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 2687–2693.  https://doi.org/10.18653/v1/P19-1258
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.  https://doi.org/10.18653/v1/N19-1423
  * Dingliwal et al. (2021) Saket Dingliwal, Ashish Shenoy, Sravan Bodapati, Ankur Gandhe, Ravi Teja Gadde, and Katrin Kirchhoff. 2021\.  Prompt Tuning GPT-2 language model for parameter-efficient domain adaptation of ASR systems.  https://doi.org/10.48550/ARXIV.2112.08718
  * Gretz et al. (2020) Shai Gretz, Yonatan Bilu, Edo Cohen-Karlik, and Noam Slonim. 2020\.  The workweek is the best time to start a family – A Study of GPT-2 Based Claim Generation.  https://doi.org/10.48550/ARXIV.2010.06185
  * Gupta et al. (2022) Prakhar Gupta, Harsh Jhamtani, and Jeffrey Bigham. 2022\.  Target-Guided Dialogue Response Generation Using Commonsense and Data Augmentation. In _Findings of the Association for Computational Linguistics: NAACL 2022_. Association for Computational Linguistics, Seattle, United States, 1301–1317.  https://doi.org/10.18653/v1/2022.findings-naacl.97
  * Hosseini-Asl et al. (2020) Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020.  A Simple Language Model for Task-Oriented Dialogue. In _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 20179–20191.  https://proceedings.neurips.cc/paper/2020/file/e946209592563be0f01c844ab2170f0c-Paper.pdf
  * Khalifa et al. (2021) Muhammad Khalifa, Hady Elsahar, and Marc Dymetman. 2021\.  A Distributional Approach to Controlled Text Generation.  https://openreview.net/forum?id=jWkw45-9AbL
  * Lin et al. (2021) Weizhe Lin, Bo-Hsiang Tseng, and Bill Byrne. 2021.  Knowledge-Aware Graph-Enhanced GPT-2 for Dialogue State Tracking. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 7871–7881.  https://doi.org/10.18653/v1/2021.emnlp-main.620
  * Lippe et al. (2020) Phillip Lippe, Pengjie Ren, Hinda Haned, Bart Voorn, and Maarten de Rijke. 2020.  Diversifying Task-oriented Dialogue Response Generation with Prototype Guided Paraphrasing.  _CoRR_ abs/2008.03391 (2020).  arXiv:2008.03391 https://arxiv.org/abs/2008.03391
  * Malhotra et al. (2022) Ganeshan Malhotra, Abdul Waheed, Aseem Srivastava, Md Shad Akhtar, and Tanmoy Chakraborty. 2022\.  Speaker and Time-Aware Joint Contextual Learning for Dialogue-Act Classification in Counselling Conversations. In _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining_ (Virtual Event, AZ, USA) _(WSDM ’22)_. Association for Computing Machinery, New York, NY, USA, 735–745.  https://doi.org/10.1145/3488560.3498509
  * Mehri et al. (2019) Shikib Mehri, Tejas Srinivasan, and Maxine Eskenazi. 2019\.  Structured Fusion Networks for Dialog. 165–177.  https://doi.org/10.18653/v1/W19-5921
  * Noble and Maraev (2021) Bill Noble and Vladislav Maraev. 2021.  Large-scale text pre-training helps with dialogue act recognition, but not without fine-tuning. In _Proceedings of the 14th International Conference on Computational Semantics (IWCS)_. Association for Computational Linguistics, Groningen, The Netherlands (online), 166–172.  https://aclanthology.org/2021.iwcs-1.16
  * Park et al. (2018) Yookoon Park, Jaemin Cho, and Gunhee Kim. 2018.  A Hierarchical Latent Structure for Variational Conversation Modeling. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_. Association for Computational Linguistics, New Orleans, Louisiana, 1792–1801.  https://doi.org/10.18653/v1/N18-1162
  * Pei et al. (2019) Jiahuan Pei, Pengjie Ren, Christof Monz, and Maarten de Rijke. 2019\.  Retrospective and Prospective Mixture-of-Generators for Task-oriented Dialogue Response Generation.  _CoRR_ abs/1911.08151 (2019).  arXiv:1911.08151 http://arxiv.org/abs/1911.08151
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep Contextualized Word Representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_. Association for Computational Linguistics, New Orleans, Louisiana, 2227–2237.  https://doi.org/10.18653/v1/N18-1202
  * Qi et al. (2021) Weizhen Qi, Yeyun Gong, Yu Yan, Can Xu, Bolun Yao, Bartuer Zhou, Biao Cheng, Daxin Jiang, Jiusheng Chen, Ruofei Zhang, Houqiang Li, and Nan Duan. 2021.  ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_. Association for Computational Linguistics, Online, 232–239.  https://doi.org/10.18653/v1/2021.acl-demo.28
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019\.  Language Models are Unsupervised Multitask Learners.  _OpenAI blog_ (2019).
  * Saha et al. (2020) Tulika Saha, Dhawal Gupta, Sriparna Saha, and Pushpak Bhattacharyya. 2020.  Emotion Aided Dialogue Act Classification for Task-Independent Conversations in a Multi-modal Framework.  _Cognitive Computation_ (2020), 1–13.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.  Proximal Policy Optimization Algorithms.  https://doi.org/10.48550/ARXIV.1707.06347
  * Serban et al. (2016) Iulian V. Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016\.  Building End-to-End Dialogue Systems Using Generative Hierarchical Neural Network Models. In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_ (Phoenix, Arizona) _(AAAI’16)_. AAAI Press, 3776–3783.
  * Stolcke et al. (2000) Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000.  Dialogue act modeling for automatic tagging and recognition of conversational speech.  _Computational Linguistics_ 26, 3 (2000), 339–374.  https://aclanthology.org/J00-3003
  * Su et al. (2015) Pei-Hao Su, David Vandyke, Milica Gaši´c, Dongho Kim, Nikola Mrkši´c, Tsung Hsien Wen, and Steve Young. 2015.  Learning from Real Users: Rating Dialogue Success with Neural Networks for Reinforcement Learning in Spoken Dialogue Systems.  https://doi.org/10.21437/Interspeech.2015-456
  * Tang et al. (2019) Jianheng Tang, Tiancheng Zhao, Chenyan Xiong, Xiaodan Liang, Eric Xing, and Zhiting Hu. 2019\.  Target-Guided Open-Domain Conversation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 5624–5634.  https://doi.org/10.18653/v1/P19-1565
  * van der Lee et al. (2021) Chris van der Lee, Albert Gatt, Emiel van Miltenburg, and Emiel Krahmer. 2021.  Human evaluation of automatically generated text: Current trends and best practice guidelines.  _Computer Speech & Language_ 67 (2021), 101151.  https://doi.org/10.1016/j.csl.2020.101151
  * Wolf et al. (2019) Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019\.  TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents.  _ArXiv_ abs/1901.08149 (2019).
  * Xu et al. (2019) Can Xu, Wei Wu, Chongyang Tao, Huang Hu, Matt Schuerman, and Ying Wang. 2019\.  Neural Response Generation with Meta-words. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 5416–5426.  https://doi.org/10.18653/v1/P19-1538
  * Yang et al. (2020) Yunyi Yang, Yunhao Li, and Xiaojun Quan. 2020.  UBAR: Towards Fully End-to-End Task-Oriented Dialog Systems with GPT-2.  https://doi.org/10.48550/ARXIV.2012.03539
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019\.  _XLNet: Generalized Autoregressive Pretraining for Language Understanding_.  Curran Associates Inc., Red Hook, NY, USA.
  * Zhang and Li (2021) Min Zhang and Juntao Li. 2021.  A commentary of GPT-3 in MIT Technology Review 2021\.  _Fundamental Research_ 1, 6 (2021), 831–833.  https://doi.org/10.1016/j.fmre.2021.11.011
  * Zhang et al. (2020) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020.  DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_. Association for Computational Linguistics, Online, 270–278.  https://doi.org/10.18653/v1/2020.acl-demos.30
  * Zhao and Kawahara (2019) Tianyu Zhao and Tatsuya Kawahara. 2019.  Effective Incorporation of Speaker Information in Utterance Encoding in Dialog.  https://doi.org/10.48550/ARXIV.1907.05599
  * Zhou et al. (2020) Li Zhou, Jianfeng Gao, Di Li, and Heung-Yeung Shum. 2020\.  The Design and Implementation of XiaoIce, an Empathetic Social Chatbot.  _Computational Linguistics_ 46 (01 2020), 1–62.  https://doi.org/10.1162/COLI_a_00368
  * Ziegler et al. (2019) Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. 2019.  Fine-Tuning Language Models from Human Preferences.  _CoRR_ abs/1909.08593 (2019).  arXiv:1909.08593 http://arxiv.org/abs/1909.08593

## Appendix

##  Appendix A Discussion on Optimization

Our model, READER is a result of numerous experiments on the selection of
rewards and other hyper-parameters. To scale this model’s applicability in the
real world, it is equally crucial to understand what worked and what didn’t.
In this section, we analyze the model’s behavior for different configurations.
Learning Rate (lr):  READER behaves unstably with higher learning rate and
often causes the reward to collapse significantly. Consequently, we observe
that both the convergence rate and the stability of the model start
deteriorating.

After fine tuning $lr$ to $2\times 10^{-7}$, we find an optimal tradeoff point
between the convergence of the model and the model’s stability. Batch-size: We
observe that a large batch size helps with the model’s stability for a
continuous action space without collapsing the reward. Relative Entropy (RE):
We also scaled down the RE value by a factor of 1000 for the reward
computation against the standard
recommended111111https://github.com/lvwerra/trl values of subjective responses
and highest metrics.

Reward. a) Employing the ROUGE-1 score explicitly as a reward is highly prone
to the collapse, and as a consequence, the text generation deteriorates
through a repetition of similar phrases. b) At the same time, if we use
BERTScore only, READER becomes unstable and starts generating arbitrarily long
responses. c) Using an external reference classifier (i.e., SPARTA) to
supplement the reward shows significant improvement in generation quality and
benchmark metrics. It ensures that the generated responses are consistent with
the response-acts gold labels.

Figure 6. Inspired by the work of Malhotra et al. (2022), we take their
proposed relationship among dialogue-act classes. Here, the directed path
$U_{t}^{x}\rightarrow U_{t+1}^{y}$ demonstrates the current dialogue-act and
response-act pair. Note: We use this diagram directly from their work. Table
5.  The table demonstrate the results of RAC-Head on the response-act
classification task. The comparision of the head is three-fold: a) reference
model (SOTA) vs gold dialogue-act labels, b) RAC-Head’s prediction vs gold
labels, and c) RAC-Head’s prediction vs reference model (SOTA). We show the
accuracy along with weighted precision, recall, and F1 scores.

Model | Precision | Recall | F1 | Accuracy
---|---|---|---|---
SOTA vs Gold Labels | 0.69 | 0.55 | 0.52 | 0.55
RAC-Head vs Gold Labels | 0.49 | 0.49 | 0.42 | 0.49
RAC-Head vs SOTA | 0.50 | 0.45 | 0.41 | 0.45

##  Appendix B Experimental Setup

We perform numerous experiments using various combinations of the
autoregressive language modeling loss, the dialogue-act loss, the value loss,
and the policy loss. Moreover, we conduct extensive hyper-parameter tuning to
correctly optimize the PPO trainer and scaling of the relative entropy reward.
Further, we extensively experimented with the reward function and observe that
Rouge, BERTScore, and RAC-Head’s logits along with the relative entropy ($RE$)
contribute towards most optimal policy learning.

We perform all experiments on an Nvidia A6000 GPU. We tune our hyperparameters
to find the optimal configurations. We utilize the learning rate of
$2x10^{-6}$, batch size of $128$, which we run for $4$ PPO-epochs. We use the
Adam optimizer and train the reference READER for 50 epochs. We also perform
hyperparameter tuning on values of $\lambda$ in Equation 2 and observe that
READER works best with
$\lambda_{1}=0.5;\lambda_{2}=0.15;\lambda_{3}=0.15;\lambda_{4}=0.2$.

Table 6. Results obtained on the Switchboard Dialogue-act dataset. We show
Rouge (1, 2, L), BERTScore (BS), and Meteor to assess the generalizability of
the READER on datasets similar to HOPE.

| R1 | R2 | RL | BertScore | Meteor
---|---|---|---|---|---
P | R | F1 | P | R | F1 | P | R | F1
DialoGPT | 21.64 | 27.16 | 23.87 | 11.41 | 14.43 | 12.47 | 19.53 | 23.78 | 21.46 | 0.6615 | 0.1836
GPT2 | 22.25 | 27.95 | 24.14 | 11.89 | 14.49 | 12.75 | 20.12 | 24.76 | 21.80 | 0.6608 | 0.1822
READER | 22.32 | 27.92 | 24.18 | 11.97 | 14.49 | 12.79 | 20.20 | 24.76 | 21.84 | 0.7295 | 0.1850
$\Delta_{\texttt{READER}-BEST}(\%)$ | $\uparrow 0.3$ | $\downarrow 0.1$ | $\uparrow 0.1$ | $\uparrow 0.6$ | $\uparrow 0.0$ | $\uparrow 0.3$ | $\uparrow 0.3$ | $\uparrow 0.0$ | $\uparrow 0.1$ | $\uparrow 9.4$ | $\uparrow 1.5$

##  Appendix C Performance of RAC-Head

We deploy RAC-Head to perform the task of response-act classification on the
HOPE dataset. RAC-Head and LM-Head jointly learns to optimize the READER.
Further, to analyze the performance of RAC-Head, we present the results in
Table 5.

##  Appendix D Analysis

We present a detailed analysis of responses generated by READER in Table $7$.
We see the model is able to correctly incorporate the context. Further
observations shows that the quality of the generations is up to the mark and
READER is capable to carry out full-fledged counseling. However, considering
the severity of the matter, we never intend to eliminate the human in the
loop.

##  Appendix E Generalizability

We discuss the generalizability of our proposed model, READER on the
Switchboard Dialogue-act Corpus for the response-generation task. We present
the results in the Table 6. The results show that the performance of the model
is better on majority of the metrics.

##  Appendix F Reproducibility Checklist

We upload our code in a zip file to reproduce the results of READER. In this
section we show the directory structure. The root directory consist of
modified trl sub directories consisting of model file named $gpt2.py$, ppo
trainer file named $ppo.py$ along with additional helper functions in
$core.py$ file. The training program is stored in $Train.py$ file and the
latest saved checkpoint can be located in $model\\_train\\_final.pt$. Further
details can be found in an enclosed README.md file.