# Bi-Phone: Modeling Inter Language Phonetic Influences in Text

Abhirut Gupta  Google Research  Ananya B. Sai  IIT Madras  Richard Sproat
Google Research  Yuri Vasilevski  Google Research
James S. Ren  Google Research  Ambarish Jash  Google Research  Sukhdeep S.
Sodhi  Google Research  Aravindan Raghuveer  Google Research

###### Abstract

A large number of people are forced to use the Web in a language they have low
literacy in due to technology asymmetries. Written text in the second language
(L2) from such users often contains a large number of errors that are
influenced by their native language (L1). We propose a method to mine phoneme
confusions (sounds in L2 that an L1 speaker is likely to conflate) for pairs
of L1 and L2. These confusions are then plugged into a generative model (Bi-
Phone) for synthetically producing corrupted L2 text. Through human
evaluations, we show that Bi-Phone generates plausible corruptions that differ
across L1s and also have widespread coverage on the Web. We also corrupt the
popular language understanding benchmark SuperGLUE with our technique (FunGLUE
for Phonetically Noised GLUE) and show that SoTA language understating models
perform poorly. We also introduce a new phoneme prediction pre-training task
which helps byte models to recover performance close to SuperGLUE. Finally, we
also release the FunGLUE benchmark to promote further research in phonetically
robust language models. To the best of our knowledge, FunGLUE is the first
benchmark to introduce L1-L2 interactions in text.

##  1 Introduction

We live in a multilingual world with over 7,000 languages spoken across the
globe Eberhard and Fennig (2022). However, technology asymmetrically supports
only a few specific languages. For instance, the internet is mostly in English
with over 60% of websites using the language despite just around 16% share of
its speaking population around the
world111https://w3techs.com/technologies/overview/content_language
Grefenstette and Nioche (2000). Increasingly, people are forced to navigate
and produce content on the web in languages they have not been formally
trained on. The English text produced by ESL (English as Second / L2 language)
writers is heavily influenced by their native language (L1).

Research in the field of second-language acquisition has found evidence of
phoneme-shift based misspellings stemming from L1 influence in L2 text for
specific language pairs Ibrahim (1978); Cook (1997); Bestgen and Granger
(2011); Sari (2014); Ogneva (2018); Motohashi-Saigo and Ishizawa (2020).
Studies in Natural Language Understanding (NLU) have been limited to spelling
correction Nagata et al. (2017); Flor et al. (2019) and native language
identification Chen et al. (2017); Nicolai et al. (2013) in English learners.
These studies predominantly use the TOEFL11 dataset Blanchard et al. (2013)
which deals with very specific demographics such as test-takers who have
formal training in the L2 language.

We make the following four key observations about prior work in the study of
L1-L2 influences in text and speech. First, current models for L1-L2 influence
on textual spelling are limited to certain language pairs and tasks. We argue
that L1-L2 influence phenomenon is much more broad and is language and task
agnostic. Second, there is no large scale study to examine the prevalence of
this phenomenon on the open web. Third, given that this is an important
problem especially for multi-lingual, new-to-the-internet communities there is
no standardized benchmark to study the robustness of natural language
understanding(NLU) and Natural Language Generation (NLG) models to inter-
language phonetic noise. Finally, there is very sparse literature on
architecture / pre-training strategies to introduce phonetic robustness into
large language models. In this paper, we present modeling techniques,data
analyses and a new benchmark to address the gaps mentioned above. We summarise
our contributions as follows:

  1. 1.

We propose a language-agnostic method to mine phoneme confusions that arise
due to interference between a native language (L1) and second language (L2).
Our method exploits the “hidden knowledge" contained in $L1\rightarrow L2$ and
$L2\rightarrow L1$ transliteration models. We also propose a generative model
Bi-Phone that is able to synthetically produce spelling corruption in
accordance with L1-L2 confusions (Sections 3.1, 3.2).

  2. 2.

Through human evaluation and coverage analysis we show that Bi-Phone produces
spelling corruptions that are not only deemed plausible by native L1 speakers
but also have substantial coverage in the open web crawl corpus. To the best
of our knowledge no prior work has demonstrated the presence of L1-L2 phonetic
corruptions in a large scale, common dataset like Common Crawl (Section 4).

  3. 3.

We release a dataset consisting of sentences with L1-L2 phonetic spelling
corruptions found in Common Crawl. We also release a benchmark called FunGLUE,
an extension of the SuperGLUE benchmark for L1-L2 spelling corruptions. To the
best of our knowledge FunGLUE is the first benchmark to measure the robustness
of models to L1-L2 interference in text (Section 5).

  4. 4.

We show SoTA models do not perform well on FunGLUE. We then introduce a novel
pre-training task of phoneme prediction, which together with byte level
architectures substantially bridges the gap on the noised benchmark (by up to
11% absolute on certain test sets). This is particularly impressive since this
gain is achieved without ever showing the model any noised examples (Section
6).

##  2 Related Work

We divide the presentation of related work in two sections. (i) First, we
discuss prior work spanning multiple research areas regarding phonetic
influences in text and how it relates to our work. (ii) Second, we discuss
work in the speech domain which studies phonetic variations occurring due to
inter-language interference in multi-lingual scenarios.

###  2.1 Phonetic Influences in Text

Phonetic influence on spelling errors has been studied in the past Kukich
(1992); Toutanova and Moore (2002); Hládek et al. (2020). The source of such
errors is that both native and non-native speakers resort to phonetic
spellings for unfamiliar words or names. This direction of work does not
address the effect of native language (L1) based phoneme shifts on second-
language (L2) spellings.

There has also been work that focuses on learner English 222learner English
refers to English as a foreign language for different applications. Nagata et
al. (2017); Flor et al. (2019) study automatic spell correction with
distributional methods that require a larger learner corpus. Chen et al.
(2017); Nicolai et al. (2013) explore Native Language Identification (NLI) on
such text. A widely used dataset for these learner English tasks is the
TOEFL11 corpus Blanchard et al. (2013) which contains English essays written
by non-native test-takers. It is important to note that these analysis are
limited to misspellings made by authors with sufficient L2 knowledge/ training
that qualifies them to take the test. They also do not explicitly study the
causes of the misspellings or the inter-language interference.

There has also been a fair amount of interest in the second-language
acquisition field on the influence of L1 on L2 spelling. Ibrahim (1978); Cook
(1997); Bestgen and Granger (2011); Sari (2014); Ogneva (2018); Motohashi-
Saigo and Ishizawa (2020) all find evidence of such influence in specific
language pairs. These often stem from the lack of certain sounds in L1 leading
to difficulty in distinguishing similar sounds in L2. They also find more
interesting phenomenon like L1 constraints on consonant clusters are reflected
in L2 spellings by learners. While this direction of research is highly
pertinent to our work, our goal is to generate plausible L1-L2 phonetic shift
based misspellings more generally instead of studying the phenomenon in
particular language pairs.

###  2.2 Inter-language Influence for Phonetic Deviations in Speech

Phonetic variations of words have been well-studied in the context of speech
applications. Several studies Radzikowski et al. (2019); Shah et al. (2020);
Radzikowski et al. (2021); Bird et al. (2019) discuss the drop in performance
of speech applications such as ASR, spoken-term detection, etc., when
presented with non-native speech data. They attribute this drop mainly to the
nuances in pronunciation that are often not present in the training data, due
to the lack of sufficient non-native speech data. To address and close this
gap, several strategies ranging from the use of cross-lingual/multi-lingual
phonological inventories to end-to-end training have been applied. However,
these studies do not focus on how the same phonetic influences manifest in
written text.

##  3 Method

In this section we introduce our method for creating inter-language influenced
phonetic misspellings (or corruptions). We present the technique in two parts.
Section 3.1 presents a method for mining native-language influenced phonetic
confusions. Section 3.2 contains details of Bi-Phone, our model that uses
mined phonetic confusions to create misspellings.

###  3.1 Mining Phoneme-Phoneme Confusions

Figure 1: Overview of the Round Trip Transliteration method for creating word
pairs from which phoneme confusions are mined. In this example, we create
pairs for the dictionary word “amazon" with round-trip transliteration through
Hindi as the pivot language. Phoneme sequences for the original and round-trip
transliterated words are also shown. Multiple words with JH in the round-trip
transliterations enables us to map the Z sound to the JH sound for Hindi
speakers. Figure 2: Examples of round trip transliterations of dictionary
words with different pivot languages, the corresponding phoneme sequences, and
the phoneme confusion mined. While the third example also has a Z -> S shift,
it is not mined because we only consider the top-10 most frequent confusions
per (L1, L2) pair.

The first problem is to identify possible phoneme confusions that a speaker of
a given native language (L1) is likely to encounter when speaking a second
language (L2). These confusions can be imagined as a matrix $C(L1,L2)$, which
contains likelihood of the $i_{th}$ L2 phoneme ($ph_{i}$) being confused as
the $j_{th}$ L2 phoneme ($ph_{j}$) by a native speaker of L1 as the value in
the cell $C(L1,L2)[i][j]$.

$C(L1,L2)[i][j]=P(ph_{j}|ph_{i})$ (1)

Building this matrix across all pairs of languages is an expensive task. It is
also challenging to accurately determine the likelihood of such confusions
without large datasets of parallel words.

Transliteration models are trained on large parallel datasets with the
objective of transcribing sounds representing words in one language with in
the script of a different language. They imbibe important information about
sounds in one language that are indistinguishable in another (and therefore
lexicalized identically). We propose a round-trip transliteration based method
which aims to mine these phoneme confusions and their likelihoods from this
knowledge hidden in transliteration models. We collect a large dictionary of
English words (our chosen L2) and apply two steps of transliteration
333https://github.com/libindic/indic-trans Bhat et al. (2015) to convert them
back to English via a pivot language (L1), as shown in Figure 1. We then align
the phoneme sequence of the original word with that of its round-trip
transliterated version using the Needleman-Wunsch algorithm Needleman and
Wunsch (1970). We count the frequency of each of the possible sound-shifts in
the whole corpus to estimate likelihood. Figure 2 shows examples of word pairs
created through different pivot languages and the phoneme confusion mined from
these. We consider only the top-10 most frequent phoneme confusions per (L1,
L2) for the next step.

###  3.2 BiPhone: A Generative Model for L1-L2 Phonetic Misspellings

The second problem we focus on is to create a model for sampling phonetic
misspellings ($\boldsymbol{\tilde{w}}$) for a given word ($\boldsymbol{w}$) in
L2 that a native speaker of L1 is likely to make. We can represent the
probability distribution learnt by this model as
$P(\boldsymbol{\tilde{w}}|\boldsymbol{w})$. Assuming a deterministic mapping
from the word $\boldsymbol{w}$ to its phoneme sequence $\boldsymbol{ph_{w}}$,
and introducing the corrupted phoneme sequence ($\boldsymbol{ph_{\tilde{w}}}$)
that finally generates $\boldsymbol{\tilde{w}}$, we can rewrite it as -

$\begin{split}P(\boldsymbol{\tilde{w}}|\boldsymbol{w})&=P(\boldsymbol{\tilde{w}}|\boldsymbol{ph_{w}})\\\
&=\sum_{\boldsymbol{ph_{\tilde{w}}}}P(\boldsymbol{ph_{\tilde{w}}}|\boldsymbol{ph_{w}})*P(\boldsymbol{\tilde{w}}|\boldsymbol{ph_{\tilde{w}}})\end{split}$
(2)

Here a word $\boldsymbol{w}$ is comprised of graphemes $\\{w^{1},w^{2},..\\}$
where $w^{i}\in Graphemes(L2)$ and a phoneme sequence $\boldsymbol{ph_{w}}$ is
comprised of phonemes $\\{ph^{1},ph^{2},..\\}$ where each individual phoneme
$ph^{i}$ is from the set of available phonemes for $L2$. In our experiments,
we use the ARPAbet phoneme set for English
444https://en.wikipedia.org/wiki/ARPABET.

Phoneme-Phoneme Error Model: The first term under the summation in Equation 2
models the likelihood of generating a corrupted phoneme sequence
$\boldsymbol{ph_{\tilde{w}}}$ given that a native speaker of L1 is attempting
to speak a phoneme sequence $\boldsymbol{ph_{w}}$ in L2. With simplifying
independence assumptions that each phoneme is corrupted individually,
independent of phonemes around it, we can factorize this term to utilize the
phoneme confusion matrix we have mined.

$\begin{split}\hskip
20.00003pt&P(\boldsymbol{ph_{\tilde{w}}}|\boldsymbol{ph_{w}})=\prod_{i}P(ph_{\tilde{w}}^{i}|ph_{w}^{i})\\\
&=\prod_{i}C(L1,L2)[ph_{w}^{i}][ph_{\tilde{w}}^{i}]\end{split}$ (3)

Phoneme-Grapheme Density Model: The second term in Equation 2 expresses the
probability of generating the grapheme sequence to represent
$\boldsymbol{\tilde{w}}$ given the phoneme sequence
$\boldsymbol{ph_{\tilde{w}}}$. We can assume equal lengths for the two
sequences, by allowing some phonemes to not generate any graphemes, when
necessary. Again, we make independence assumptions where the grapheme used to
represent a given phoneme does not depend on neighbouring phonemes or
graphemes.

$P(\boldsymbol{\tilde{w}}|\boldsymbol{ph_{\tilde{w}}})=\prod_{i}P(\tilde{w}^{i}|ph_{\tilde{w}}^{i})$
(4)

To compute $P(\tilde{w}^{i}|ph_{\tilde{w}}^{i})$, we use a pronunciation
dictionary in L2 (CMUDict555http://www.speech.cs.cmu.edu/cgi-bin/cmudict for
English). First, phoneme-character probabilities are generated through
alignment. Next, for each word, character sequences are converted to graphemes
by maximizing the alignment score. Finally, the various phoneme-grapheme
alignments along with their frequencies are converted to probabilities by
dividing it by the frequency of the phoneme.

Inference: Given an original phoneme sequence for a word to be corrupted, we
begin sampling with a fixed width (K) beam from left to right. At each
position, we pick the top-K candidates comprising both phoneme-phoneme shifts
and phoneme-grapheme alternatives greedily. Since both Phoneme-Phoneme Error
Model and Phoneme-Grapheme Density Model are context independent, the greedy
strategy gives us the global top-K misspellings. Identity corruptions are
removed as a final step.

##  4 Evaluations

Phoneme Shift | Hi | Ta | Bn
---|---|---|---
AH2 -> AH0 | 100% | - | 100%
IH2 -> IH0 | 100% | - | 100%
ER2 -> ER0 | 100% | - | -
DH -> TH | 54% | - | 62%
ER2 -> ER0 | 95% | - | -
D -> T | - | 30% | -
B -> P | - | 39% | -
DH -> D | - | 0% | -
G -> K | - | 47% | -
V -> B | - | - | 58%
Z -> S | - | - | 50%
Table 1: Plausibility scores for different phoneme shifts across Hindi, Tamil,
and Bengali.

L1 | Correct | Misspelt | Phoneme
---|---|---|---
| Word | Word | Variation
Hindi | they | thay | DH -> TH
Tamil | exam | eksam | G -> K
| bacterial | pactirial | B -> P
Bengali | very | bery | V -> B
| equation | ikvasan | ZH -> S

Table 2: Examples of highly plausible misspellings as rated by native speakers
for various L1 languages with L2 language as English

We evaluate the misspellings generated by our model along two distinct
dimensions.

###  4.1 Plausibility

For evaluating plausibility of generated misspellings from Bi-Phone, we focus
on three native languages (L1) : Hindi, Tamil and Bengali with English as the
non-native language (L2). Hindi and Bengali are the two most widely spoken
languages in India and among the top few in the world. Tamil is also a widely
spoken language in India and introduces typological diversity in our analysis.
Finally, our choice of L1 is also based on availability of native speakers for
the annotation task.

For each language, we present 150 randomly selected word, misspelling pairs
generated from Bi-Phone to native speakers (5 for Hindi, 3 for Tamil and
Bengali each). Rater instructions are as follows: Given a list of pairs in
English (correct word, misspelling), the task is to evaluate if the
misspelling is plausible for pronunciation shifts often made by speakers of
the given first language. For example - Bengali speakers often shift the “v”
sound to “b” so, “evicted” could be plausibly misspelt as “ebicted” or
“abicted”. Each rater provides a 1 or 0 to indicate whether the variant looks
plausible or not, respectively. We use a simple majority to assign an overall
label to each pair. The raters for this task are our colleagues who are native
speakers of the language they are annotating for.

Table 1 reports the percentage of misspellings rated as plausible for each
phoneme shift. We observe that misspellings for Tamil are rated as less
plausible than for other languages. The reason for this is the more drastic
phoneme shifts uncovered in Tamil (B -> P and G -> K). However, misspellings
stemming from these shifts are still not rated as completely implausible,
which emphasizes that these shifts are indeed common. We also measure inter-
annotator agreement through kappa scores which are 0.40 for Hindi, 0.37 for
Tamil, and 0.34 for Bengali.

###  4.2 Prevalence: Coverage Analysis

Figure 3: Precision and coverage plotted at different misspelling confidence
scores (labels on points). Coverage is represented as a fraction of 31,755,066
sentences that have atleast one non-English dictionary word.

In the previous section we investigate the plausibility of phoneme-shifts
mined by Bi-Phone and the misspellings created as a result. However, this
investigation does not throw light on the pervasiveness of such misspellings
in real world content.

In this section, we aim to evaluate the severity of the phonetic misspelling
issue by uncovering such misspellings in web data. For our analysis, we use
the Common Crawl666https://commoncrawl.org/ corpus, which is a publicly
available scrape of real web data. While most existing language work deals
with a highly cleaned version of this corpus Raffel et al. (2020b), we skip
such filtering and cleaning steps to retain noisy, user-generated text. We
only use Hindi as the native language (L1) in this analysis. Our analysis has
three distinct steps - (1) Candidate Sentence Retrieval, (2) Misspelling
Confidence Scoring, and (3) Human Evaluation.

1\. Candidate Sentence Retrieval: We begin our analysis by creating 10
misspellings of the top 10,000 most common English words from the Google ngram
corpus Michel et al. (2011) and words that make up 90%-ile of the English
words in the Common Crawl corpus. Our hypothesis is that the most common words
in English are also the most likely to be misspelt with native language
influences. Our pool of sentences is the set of all sentences with at least
one non-English dictionary word. The size of this pool is 31,755,066
sentences. From this pool, we create our candidate set by retrieving all
sentences that contain one of our generated misspellings.

2\. Misspelling Confidence Scoring: The next step is to ascertain that the
misspellings retrieved are indeed a noisy form of the intended original word
and not a completely different word. For example, “vare" could be a corruption
of the English word “where" with the W -> V sound shift, or it could be the
less used English word meaning a weasel 777https://www.merriam-
webster.com/dictionary/vare. We use a simple 1-word left and right context for
this disambiguation. For every occurrence of a potentially misspelt word
$\hat{W}$ in context $(L_{\hat{W}},\hat{W},R_{\hat{W}})$, we evaluate the
probability of seeing the corresponding clean word ($W$) in the same context.
This likelihood, $P(L_{\hat{W}},W,R_{\hat{W}})$ computed as follows can be
used as a score to represent our confidence in the retrieved misspelling.
$\displaystyle P(L_{\hat{W}},W,R_{\hat{W}})$
$\displaystyle=\frac{F(L_{\hat{W}},W,R_{\hat{W}})}{\sum_{w}F(L_{\hat{W}},w,R_{\hat{W}})}\text{\
\ , \ \ if}\sum_{w}F(L_{\hat{W}},w,R_{\hat{W}})>0$
$\displaystyle=0.4*\Bigg{[}\frac{F(L_{\hat{W}},W)}{\sum_{w}F(L_{\hat{W}},w)}+\frac{F(W,R_{\hat{W}})}{\sum_{w}F(w,R_{\hat{W}})}\Bigg{]}\text{,
otherwise}$

Here 0.4 is the backoff-weight following the Stupid Backoff technique from
Brants et al. (2007).

We can compute the coverage of Bi-Phone in web data by considering the
fraction of sentences where the misspelling confidence score is greater than a
certain threshold over the total number of sentences in our original pool.

3\. Human Evaluation: Finally, we also sample a subset of the sentences to
have human raters verify that our retrieved misspellings indeed correspond to
the original word. We show raters the original retrieved sentence which
contains the generated misspelling and a parallel sentence where the
misspelling has been replaced with the original word and ask raters if this
correction is valid in the given context. We can compute a reliable metric for
precision with this human evaluation. Ratings for this task are fetched from a
cloud rating service where raters are bilingual Hindi-English speakers with a
graduate degree.

Figure 3 presents the precision and coverage at different thresholds of
misspelling confidence score. At threshold 0.001, we have roughly 70%
precision while still having a coverage of 1.14% (362,472 sentences*). The
size of the initial pool (30 million candidate sentences) and the simple
method used for our analysis underline how prevalent such misspellings are.
Also it is important note that such misspellings will be even more prevalent
in a purely UGC (user generated content) corpus. C4 contains a significant
fraction of clean English web pages.

##  5 The FunGLUE Benchmark

Split |  Description |  Contains Phonetic Noise
---|---|---
train |  Train split from SuperGLUE as is |  No
dev |  Dev split from SuperGLUE as is |  No
test |  Dev split from SuperGLUE noised with BiPhone |  Yes
Table 3: Description of splits in FunGLUE. Checkpoint selection is done on the dev set which does not contain phonetic misspellings. The test set is used only for reporting results. Task | Field Name
---|---
BoolQ | question
CB | premise
COPA | premise
MultiRC | question
ReCoRD | query
RTE | hypothesis
WiC | sentence1
Table 4: Fields we noise for different task when creating FunGLUE.

Significant progress has been made in recent research to substantially improve
performance of language understanding tasks. SuperGLUE Wang et al. (2019) is a
very popular benchmark with ten diverse and hard language understanding tasks.
These tasks are BoolQ, CommitmentBank (CB), Multi-Sentence Reading
Comprehension (MultiRC), Choice of Plausible Alternatives (COPA), Reading
Comprehension with Commonsense Reasoning (ReCoRD), Recognizing Textual
Entailment (RTE), Words in Context (WiC), Broadcoverage Diagnostics (AX-b),
The Winograd Schema Challenge (WSC), and Winogender Schema Diagnostics (AX-g).
We argue that for language understanding models to be effective for bi-lingual
users, they must be robust to inter-language phonetic spelling variations.
Towards this end, we introduce FunGLUE888https://github.com/google-research-
datasets/FunGLUE which stands for Ph(F)onetically noised GLUE where randomly
selected words from tasks in the SuperGLUE benchmark are corrupted with Bi-
Phone based misspellings. It is extremely important to note that we only
create a hold-out evaluation set created by introducing misspellings to the
SuperGLUE development set. The training set is left clean to mimic real world
scenarios where noised training data is difficult to obtain. Additionally, it
would be unfair to train and evaluate models on synthetic misspellings from
the same source. Table 3 summarizes the training, validation, and test sets in
FunGLUE.

Misspellings for words in the original task are created from Bi-Phone with the
following design choices:

(i) What to noise: Since we want to keep the task realistic, we only introduce
misspellings in certain pre-selected fields and not all text fields. This
reflects real world situations where content is often available in well spelt
English but user queries have phonetic errors. Table 4 presents the fields we
actually noise.

(ii) Which misspellings to use: Since we expect benchmarks to have a high
quality, we put in a number of guardrails to ensure poor quality misspellings
do not make it through to the benchmark. First, we only use Bi-Phone
misspellings with Hindi and Bengali as native language since Tamil
misspellings were rated as less plausible by native speakers. Next, we noticed
that plausibility scores drop for words smaller than 4 characters, so we only
noise longer words. We also filter out misspellings that contain certain
patterns of implausible noise generated by our Grapheme2Phoneme model with
rules. Finally, all (word, misspelling) pairs used in FunGLUE are manually
verified by members of the team as plausible.

Task | Tokens misspelt | Examples w/ noise
---|---|---
boolq | 30.6% | 96.2%
cb | 29.5% | 96.4%
multirc | 33.8% | 96.4%
copa | 25.2% | 78.0%
record | 29.5% | 99.4%
rte | 35.9% | 97.1%
wic | 28.9% | 84.0%
Table 5: Stats on amount of noise added in FunGLUE.

(iii) How much noise to add: Since we do not want to artificially introduce
too much noise, we only replace 30% of words from the original benchmark
across tasks. Table 5 contains stats on the amount of noise added to each
task. We were currently unable to include the noised version of the WSC, AX-b
and AX-g tasks due to some difficulties in accessing the eval sets. We plan to
include this with the final data release.

###  5.1 Models

Figure 4: Examples from validation set of two tasks in FunGLUE against
SuperGLUE. Words which are replaced with their noised versions are in red.

In this section we investigate if state-of-the-art models are robust to the
phonetic noise introduced by FunGLUE by comparing their performance on
SuperGLUE. For this purpose, we consider mT5 Xue et al. (2021b) and ByT5 Xue
et al. (2021a) models. These are both transformer based sequence-to-sequence
models that frame all language understanding tasks as sequence generation. mT5
uses sub-word tokenization built on a multilingual corpus, to represent text.
It should therefore be more robust to input variations than comparable models
with tokenization on monolingual corpora with lower diversity. ByT5 avoids the
tokenization step by building input representations from individual bytes, and
is designed to perform more gracefully on noisy text across a range of tasks.

For all models, we use the base architecture. Since training these models is
expensive, we do not perform any hyper-parameter search. Instead, we use fine-
tuning parameter values from the original papers. Crucially, fine-tuning for
all models is performed identically on clean data from SuperGLUE. We use the
same mixture of tasks as in Raffel et al. (2020a). Fine-tuning is done for up
to 200,000 steps and the best checkpoint is picked based on performance on the
clean dev set from SuperGLUE. We use 16 TPUv3s for fine-tuning all models.

###  5.2 Spell Correction Baselines

Spell correction methods provide obvious baselines when dealing with
incorrectly spelt data. Spell corrected data can then be use to run inference
with existing models. To evaluate the merit of this technique, we measure
performance after correction from two state of the art approaches: (1)
NeuSpell BERT Jayanthi et al. (2020) \- spell corrector built on top of BERT.
(2) BERT-Large mask prediction - using a BERT Large model for predicting the
correct word in positions where we have misspellings. In both of these
approaches, we provide the positions of incorrectly spelt words. This is an
advantage since this information is not available in real world noisy text. We
compare the performance of both mT5 and ByT5 on FunGLUE eval sets corrected by
these approaches.

###  5.3 Results

No. | Model | BoolQ | CB | COPA | MultiRC | ReCoRD | RTE | WiC
---|---|---|---|---|---|---|---|---
Acc | Acc | F1 | Acc | EM | F1 | EM | F1 | Acc | Acc
SuperGLUE
1 | mT5 | 78.10 | 92.86 | 90.53 | 61.00 | 33.68 | 73.03 | 67.22 | 68.26 | 74.37 | 68.03
2 | ByT5 | 79.20 | 91.07 | 90.37 | 58.00 | 32.00 | 70.14 | 72.10 | 72.79 | 81.23 | 70.85
FunGLUE
3 | mT5 | 68.81 | 80.36 | 74.21 | 55.00 | 28.23 | 70.37 | 58.46 | 59.46 | 67.87 | 63.64
3a | mT5 - NeuSpell | 67.92 | 76.79 | 74.99 | 64.00 | 30.43 | 70.85 | 60.36 | 61.33 | 65.34 | 65.83
3b | mT5 - Bert-L mask pred | 66.42 | 71.43 | 79.6 | 57.00 | 27.70 | 67.91 | 55.6 | 56.63 | 58.84 | 62.54
4 | ByT5 | 74.04 | 80.36 | 73.67 | 58.00 | 32.42 | 72.73 | 67.54 | 68.19 | 70.40 | 66.46
4a | ByT5 - NeuSpell | 72.84 | 76.79 | 67.86 | 54.00 | 32.53 | 72.47 | 63.64 | 64.25 | 69.68 | 66.46
4b | ByT5 - Bert-L mask pred | 70.52 | 75.00 | 70.7 | 55.00 | 26.76 | 68.60 | 59.75 | 60.35 | 64.62 | 64.26
5 | Phonetic mT5 | 71.80 | 80.36 | 73.66 | 53.00 | 25.81 | 72.2 | 55.85 | 56.86 | 61.37 | 63.17
6 | Phonetic ByT5 | 74.37 | 87.50 | 85.46 | 66.00 | 33.26 | 75.15 | 70.21 | 70.88 | 76.17 | 66.77

Table 6: First 4 rows: Performance of SoTA models on tasks in the SuperGLUE
and FunGLUE (noised) benchmarks. Performance of both mT5 and ByT5 (rows 3 and
4 compared to 1 and 2) drops on the noised benchmark, although ByT5 (row 4) is
slightly more robust. Rows 3a, 3b, 4a, and 4b show the performance of mT5 and
ByT5 after misspelt words in the eval set are replaced with corrections from
SoTA techniques. While mT5 benefits slightly from such corrections, ByT5
performance is worse across all tasks after spell correction is applied. This
demonstrates the inability of current spell correction models to handle such
misspellings. Rows 3a and 4a correspond to corrections from the NeuSpell
Jayanthi et al. (2020) model. Rows 3b and 4b correspond to corrections using
mask prediction from a Bert-Large model. Last 2 rows: Performance of the same
models when trained on a few additional steps with the phoneme prediction task
on clean data (Phonetic mT5 and ByT5). The ByT5 (row 6 compared to row 4)
model gains substantially with such pre-training.

Rows 1-4 in Table 6 show the performance of mT5 and ByT5 on SuperGLUE and
FunGLUE. There is a clear drop in performance for both models on FunGLUE, with
both mT5 and ByT5 dropping upto 16 F1 points on the CB dataset. The mT5 model
also drops by roughly 9 points in accuracy on the BoolQ dataset, and similarly
9 F1 points on the ReCoRD dataset. While the ByT5 model is in general more
robust than the mT5 model, its performance also drops by 10 points in accuracy
on RTE.

The spell correction baselines (Rows 3a, 3b, 4a, 4b) also fail to recover
performance. With NeuSpell, mT5 sees a drop in BoolQ and RTE, slight
improvement on CB, MultiRC, Record, WIC (<2 points Acc/F1). On COPA, we
observe a substantial recovery (55 -> 64). For ByT5 however, there is a drop
in performance across the board. NeuSpell is not well equipped to handle
phonetic misspellings. Therefore the spell corrected word is often farther
from the original word than the misspelling. These bad corrections hurt ByT5,
which is slightly more robust to misspellings than mT5. With Bert-Large mask
prediction, for mT5 there is a slight improvement on COPA and improvement on
CB(74.21 ->79.6), but worse performance on all other tasks. Again for ByT5, we
see degradation in performance across the board. Since 30% of the tokens are
phonetically misspelt, the contextual mask prediction task is also not
accurate. Another failure mode we observed was that the prediction is often
the correct type (adjective for adjective) but not the original token.

This clearly demonstrates the challenge posed by phoneme-shift based noisy
misspellings introduced in FunGLUE . Current models and training schemes are
ill-equipped to function on such data.

##  6 Phoneme Prediction as a Pre-training Task

Figure 5: Demonstration of our mixture pre-training task that combines
standard span-corruption with the novel phoneme prediction task in an 80:20
ratio. All weights and embeddings in the model are shared.

Given the inadequacy of existing State-of-the-Art models in handling phonetic
noise in inputs, we propose a novel pre-training task of phoneme prediction.
We posit that the task of predicting phoneme sequences will have the effect of
teaching the model “phonetic information". Since different lexicalizations of
the same sound will have the same phoneme sequence, the model will learn to
embed these close. Additionally since close sounds often appear in similar
intra-word contexts, their graphemic representations will also be pushed
closed together.

However, to perform NLP tasks, semantic similarity is still crucial. In
current models this is often achieved through some variation of the span
corruption task (corrupting a span in the input and predicting it on the
output). We propose a mixture of these two tasks where a small amount of the
phoneme prediction task (20%) is mixed into the standard span corruption task.
Figure 5 demonstrates our proposal through two example instances. In the first
instance the span “sofa design" is masked in the input (replaced with a
sentinel) and is expected to be produced on the output. This teaches the model
that adjectives like “exquisite" are semantically close. The second instance
has the word “building" in the input and the phoneme sequence corresponding to
this word (B, IH, L, D, IH, NG) on the output. This task teaches the model
that all tokens that produce the same sound (like “ui" or “e" for IH) should
be embedded close.

We train both mT5 and ByT5 checkpoints for an additional 100,000 steps (10%
additional steps) on this mixture task. We call this step of additional pre-
training, “Phonetic pre-training". Finally, we fine-tune these models on the
standard clean SuperGLUE training set. The phoneme prediction data is created
by taking roughly 2,000,000 highest frequency words from the Common Crawl
English data and getting their pronunciations from an off-the-shelf Grapheme
to Phoneme model. As we will see later, this kind of noisy supervision (not
human labelled) is still useful in making models phonetically robust.

The last two rows in Table 6 show the performance of these models on FunGLUE.
We find that the simple additional pre-training step of phoneme-prediction
substantially improves performance of the ByT5 model on the noised benchmark
(row 6 against row 4). Performance on CB increases by 11 F1 points, on COPA
there is a 8 point accuracy gain, and a 5 point accuracy gain on RTE. While
performance still lags compared to the clean benchmark SuperGLUE (row 6
against row 2) on most tasks, for MultiRC and COPA, we find that the
phonetically pre-trained ByT5 model even out-performs the vanilla pre-trained
model (row 2) numbers on the clean task. This is particularly impressive
because the Phonetic ByT5 model (row 6) has never seen any noisy data during
its training. The mT5 model does not however see the same impressive gains
through this pre-training task. We hypothesize this is because of the harder
sub-word tokenization in mT5. Many tokens that this model needs on the noised
task are never seen when it’s trained on clean data and therefore have poor
representations.

The ByT5 model does however have certain drawbacks. Since input sequences are
much longer with byte level representations, both training and inference times
are much slower than a sub-word tokenized alternative (like mT5).
Additionally, the byte-level representation also restricts input sequence
lengths. Using these phonetically robust byte-level models as teachers for
sub-word tokenized student models remains an interesting direction for future
work.

##  7 Conclusion

Language is a significant barrier to technology especially for new internet
users. For such users, English often is not their first language. The speech
community has made significant progress in making technology (ASR for
instance) accessible for such users by making models robust to account for
inter-language interactions. We argue that a similar line of effort is needed
in the Natural Language Understanding for Text community as well. To this end,
we first propose a generative model Bi-Phone that can account for L1-L2
interactions in text. Next we show the inter-language perturbations generated
by Bi-Phone are indeed present in non-trival amount in the common crawl
corpus. We also release a new benchmark FunGLUE to help further research in
this area. We also present our early yet very promising explorations on making
natural language understanding models robust to L1-L2 phonetic shifts through
a novel phoneme prediction based pre-training.

##  8 Limitations

Algorithmic Limitations: The current approach assumes each phoneme / grapheme
corruption is independent of the surrounding phonemes / graphemes, which can
be relaxed to get further insights and model any contextual phonetic shifts.
The relative importance between grapheme and phoneme corruptions could also be
explored as a hyperparameter to personalize more to the type of errors of a
community.
Other Limitations (with respect to available data and existing resources): Our
coverage analysis is conservative since it does not cover the user generated
data from various social media where such L1-L2 phonetic misspellings are
bound to be more common. The coverage analysis also relies on the context not
being corrupted. However, this might not necessarily hold and the analysis
could benefit from a careful formulation of a relaxed matching criteria that
also considers cases with corrupted contexts. With transliteration playing a
major role in our solution, it is difficult to immediately extend the work to
low-resource languages that do not have models or appropriate datasets to
build transliteration modules.

## References

  * Bestgen and Granger (2011) Yves Bestgen and Sylviane Granger. 2011.  Categorizing spelling errors to assess L2 writing.  _International Journal of Continuing Engineering Education and Life Long Learning_ , 21(2-3):235–252.
  * Bhat et al. (2015) Irshad Ahmad Bhat, Vandan Mujadia, Aniruddha Tammewar, Riyaz Ahmad Bhat, and Manish Shrivastava. 2015.  Iiit-h system submission for fire2014 shared task on transliterated search.  In _Proceedings of the Forum for Information Retrieval Evaluation_ , FIRE ’14, pages 48–53, New York, NY, USA. ACM.
  * Bird et al. (2019) Jordan J. Bird, Elizabeth F. Wanner, Anikó Ekárt, and Diego R. Faria. 2019.  Accent classification in human speech biometrics for native and non-native english speakers.  In _Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments, PETRA 2019, Island of Rhodes, Greece, June 5-7, 2019_ , pages 554–560. ACM.
  * Blanchard et al. (2013) Daniel Blanchard, Joel Tetreault, Derrick Higgins, Aoife Cahill, and Martin Chodorow. 2013.  Toefl11: A corpus of non-native english.  _ETS Research Report Series_ , 2013:i–15.
  * Brants et al. (2007) Thorsten Brants, Ashok C. Popat, Peng Xu, Franz J. Och, and Jeffrey Dean. 2007.  Large language models in machine translation.  In _Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)_ , pages 858–867, Prague, Czech Republic. Association for Computational Linguistics.
  * Chen et al. (2017) Lingzhen Chen, Carlo Strapparava, and Vivi Nastase. 2017.  Improving native language identification by using spelling errors.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 542–546, Vancouver, Canada. Association for Computational Linguistics.
  * Cook (1997) Vivian Cook. 1997.  L2 users and english spelling.  _Journal of Multilingual and Multicultural Development_ , 18(6):474–488.
  * Eberhard and Fennig (2022) Gary F. Simons Eberhard, David M. and Charles D. Fennig. 2022.  Ethnologue, languages of the world.  _http://www. ethnologue. com/_.
  * Flor et al. (2019) Michael Flor, Michael Fried, and Alla Rozovskaya. 2019.  A benchmark corpus of English misspellings and a minimally-supervised model for spelling correction.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 76–86, Florence, Italy. Association for Computational Linguistics.
  * Grefenstette and Nioche (2000) Gregory Grefenstette and Julien Nioche. 2000.  Estimation of english and non-english language use on the www.  In _Content-Based Multimedia Information Access - Volume 1_ , RIAO ’00, page 237–246, Paris, FRA. LE CENTRE DE HAUTES ETUDES INTERNATIONALES D’INFORMATIQUE DOCUMENTAIRE.
  * Hládek et al. (2020) Daniel Hládek, Ján Staš, and Matúš Pleva. 2020.  Survey of automatic spelling correction.  _Electronics_ , 9(10).
  * Ibrahim (1978) Muhammad Hasan Ibrahim. 1978.  Patterns in spelling errors.  _English Language Teaching_ , 32:207–212.
  * Jayanthi et al. (2020) Sai Muralidhar Jayanthi, Danish Pruthi, and Graham Neubig. 2020.  NeuSpell: A neural spelling correction toolkit.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 158–164, Online. Association for Computational Linguistics.
  * Kukich (1992) Karen Kukich. 1992.  Techniques for automatically correcting words in text.  _ACM Comput. Surv._ , 24(4):377–439.
  * Michel et al. (2011) Jean-Baptiste Michel, Yuan Kui Shen, Aviva Presser Aiden, Adrian Veres, Matthew K Gray, Google Books Team, Joseph P Pickett, Dale Hoiberg, Dan Clancy, Peter Norvig, et al. 2011.  Quantitative analysis of culture using millions of digitized books.  _science_ , 331(6014):176–182.
  * Motohashi-Saigo and Ishizawa (2020) Miki Motohashi-Saigo and Toru Ishizawa. 2020.  A relationship between orthographic output and perception in l2 Japanese phonology by L1 English speakers.  _Ampersand_ , 7:100071.
  * Nagata et al. (2017) Ryo Nagata, Hiroya Takamura, and Graham Neubig. 2017.  Adaptive spelling error correction models for learner english.  _Procedia Computer Science_ , 112:474–483.  Knowledge-Based and Intelligent Information & Engineering Systems: Proceedings of the 21st International Conference, KES-20176-8 September 2017, Marseille, France.
  * Needleman and Wunsch (1970) Saul B. Needleman and Christian D. Wunsch. 1970.  A general method applicable to the search for similarities in the amino acid sequence of two proteins.  _Journal of Molecular Biology_ , 48(3):443–453.
  * Nicolai et al. (2013) Garrett Nicolai, Bradley Hauer, Mohammad Salameh, Lei Yao, and Grzegorz Kondrak. 2013.  Cognate and misspelling features for natural language identification.  In _Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 140–145, Atlanta, Georgia. Association for Computational Linguistics.
  * Ogneva (2018) Anastasiia Ogneva. 2018.  Spelling errors in L2 Russian: evidence from Spanish-speaking students.  _Estudios interlingüísticos_ , 6:116–131.
  * Radzikowski et al. (2019) Kacper Radzikowski, Robert Nowak, Le Wang, and Osamu Yoshie. 2019.  Dual supervised learning for non-native speech recognition.  _EURASIP J. Audio Speech Music. Process._ , 2019:3.
  * Radzikowski et al. (2021) Kacper Radzikowski, Le Wang, Osamu Yoshie, and Robert M. Nowak. 2021.  Accent modification for speech recognition of non-native speakers using neural style transfer.  _EURASIP J. Audio Speech Music. Process._ , 2021(1):11.
  * Raffel et al. (2020a) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020a.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Raffel et al. (2020b) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020b.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(140):1–67.
  * Sari (2014) Ida Rukmana Sari. 2014.  Common errors in students’ spelling on the required words for the seventh graders.  _Educate_ , 4(2):35–43.
  * Shah et al. (2020) Sanket Shah, Satarupa Guha, Simran Khanuja, and Sunayana Sitaram. 2020.  Cross-lingual and multilingual spoken term detection for low-resource indian languages.  _CoRR_ , abs/2011.06226.
  * Toutanova and Moore (2002) Kristina Toutanova and Robert Moore. 2002.  Pronunciation modeling for improved spelling correction.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 144–151, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _Advances in neural information processing systems_ , 32.
  * Xue et al. (2021a) Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2021a.  Byt5: Towards a token-free future with pre-trained byte-to-byte models.  _CoRR_ , abs/2105.13626.
  * Xue et al. (2021b) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021b.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.


# CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation
Framework for Chinese Psychological Counseling

Chenhao Zhang1,3 Equal contribution.Work done on the Science and Technology
Innovation Project of UCAS directed by SIAT. Renhao Li2,311footnotemark: 1
Minghuan Tan3 Corresponding author. Min Yang333footnotemark: 3
Jingwei Zhu4 Di Yang4 Jiahao Zhao3,522footnotemark: 2 Guancheng
Ye622footnotemark: 2
Chengming Li7 Xiping Hu7 Derek F. Wong2
1 Huazhong University of Science and Technology 2 University of Macau
3 Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences
4 University of Science and Technology of China 5 Jilin University
6 South China University of Technology 7 Shenzhen MSU-BIT University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Using large language models (LLMs) to assist psychological counseling is a
significant but challenging task at present. Attempts have been made on
improving empathetic conversations or acting as effective assistants in the
treatment with LLMs. However, the existing datasets lack consulting knowledge,
resulting in LLMs lacking professional consulting competence. Moreover, how to
automatically evaluate multi-turn dialogues within the counseling process
remains an understudied area. To bridge the gap, we propose CPsyCoun, a
report-based multi-turn dialogue reconstruction and evaluation framework for
Chinese psychological counseling. To fully exploit psychological counseling
reports, a two-phase approach is devised to construct high-quality dialogues
while a comprehensive evaluation benchmark is developed for the effective
automatic evaluation of multi-turn psychological consultations. Competitive
experimental results demonstrate the effectiveness of our proposed framework
in psychological counseling. We open-source the datasets and model for future
research. 111https://github.com/CAS-SIAT-XinHai/CPsyCoun

CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation
Framework for Chinese Psychological Counseling

Chenhao Zhang1,3††thanks: Equal contribution.††thanks: Work done on the
Science and Technology Innovation Project of UCAS directed by SIAT. and Renhao
Li2,311footnotemark: 1 and Minghuan Tan3††thanks: Corresponding author. and
Min Yang333footnotemark: 3 and Jingwei Zhu4 and Di Yang4 and Jiahao
Zhao3,522footnotemark: 2 and Guancheng Ye622footnotemark: 2 and Chengming Li7
and Xiping Hu7 and Derek F. Wong2 1 Huazhong University of Science and
Technology 2 University of Macau 3 Shenzhen Institute of Advanced Technology,
Chinese Academy of Sciences 4 University of Science and Technology of China 5
Jilin University 6 South China University of Technology 7 Shenzhen MSU-BIT
University<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

"No health without mental health" is becoming more than a slogan, with
approximately 14% of the global disease burden attributed to neuropsychiatric
disorders Prince et al. (2007). Despite the affordability and effectiveness of
many mental health treatments, a significant gap persists between those in
need and those able to access care Freeman (2022). The World Health
Organization (WHO) continually advocates for increased investment to augment
understanding and dispel stigma associated with mental health disorders. Yet,
the challenge of ensuring quality, affordable care for mental health
conditions remains formidable. Consequently, the identification of novel
treatments and enhancement of existing therapies for all mental diseases are
key objectives in the research domain.

The Natural Language Processing (NLP) community is actively contributing to
the advancement of AI-assisted psychological counseling and treatment. Various
research topics have been proposed to conduct mental disease counseling Orr et
al. (2022); Toleubay et al. (2023), improve emotional support ability Buechel
et al. (2018); Rashkin et al. (2019); Liu et al. (2021); Cheng et al. (2023),
and provide online psychological consultation Sun et al. (2021).

Figure 1: The general framework of CPsyCoun

The advent of large language models (LLMs) such as ChatGPT
222https://chat.openai.com/ and LLaMA Touvron et al. (2023), has spurred more
research efforts on generating not just empathetic conversations, but also
serving as therapeutic aids and effective assistants in treatment. For
instance, Psy-LLM Lai et al. (2023) is a psychological consultation model that
leverages the LLM PanGu and is trained with Q&A from professional
psychologists and large-scale Chinese psychological articles from public
databases. This model demonstrates proficiency in psychological knowledge and
counseling services. Parallel to this, other LLM-based psychological models
such as MeChat Qiu et al. (2023), SoulChat Chen et al. (2023b) and MindChat
Yan and Xue (2023) are also available online. Recent trends in adopting LLMs
for psychological counseling focus on generating more interpretable mental
health analyses Yang et al. (2023) and simulating psychiatrist-patient
interactions Chen et al. (2023a). This shift in focus from generating
responses to diagnosing mental health issues as an expert signifies a trend
change in research. The quest for interpretability in mental health analysis
serves a dual purpose. First, it provides a detailed rationale behind each
response, making it more amenable for human evaluation and debugging. Second,
the simulation approach not only addresses data privacy concerns but also
challenges the traditional symptom collection method via questionnaires.
Providing a range of professional skills, this approach enables more effective
completion of consultation tasks.

Despite these advancements, there remains a dearth of authentic counseling
datasets from psychological counseling sessions, which include symptom
descriptions of the consultant and treatment methodologies employed by the
counselor. Such data could offset issues arising from doctor-patient
simulations being template-based and lacking control. For example,
psychiatrists have observed that chatbots do not typically resemble patients
Chen et al. (2023a). However, it’s noteworthy that these diagnoses are
generally sensitive, warranting careful attention to potential privacy issues.
In addition to the form of psychological counseling conversations, there is a
wealth of psychological counseling data in the real world, which is hidden in
professional psychological counseling reports. However, due to its structured
nature, it is unsuitable for model training.

In this paper, we propose a new framework CPsyCoun for Chinese Psychological
Counseling, which consists a dialogue reconstruction method based on
psychological counseling reports and a benchmark for multi-turn consultation
dialogue evaluation. Specifically, we first collect anonymized psychological
counseling reports from publicly accessible websites and further propose a
privacy shadowing method to postprocess these reports into a dataset
CPsyCounR. CPsyCounR includes nine types of psychological consultation and
seven classic schools of psychological counseling. Through our proposed
Memo2Demo dialogue reconstruction method, we construct another dataset
CPsyCounD, which contains 3,134 high-quality multi-turn consultation
dialogues. Further, we propose a psychological counseling benchmark for
automatic evaluation on multi-turn dialogues and fine-tune an open-sourced LLM
on CPsyCounD, named CPsyCounX. Experimental results from both intrinsic and
extrinsic evaluations consistently verify the superiority of the proposed
method.

Figure 1 illustrates the general framework of our proposed CPsyCoun.

Our contributions are the following:

  * •

To the best of our knowledge, our work is the first to generate psychological
consultation dialogues based on psychological counseling reports, which
effectively expands the source of psychological consultation dialogue data.
For efficient dialogue reconstruction, we specifically introduce a two-phase
method named Memo2Demo.

  * •

We propose a benchmark for automatic evaluation of multi-turn dialogues in
psychological counseling, which includes comprehensive evaluation metrics,
datasets and methods.

  * •

With the help of Memo2Demo, we construct CPsyCounD, a dataset contains 3,134
high-quality multi-turn consultation dialogues. The model CPsyCounX fine-tuned
on this dataset outperforms other models in the benchmark, validating the
effectiveness of our proposed framework in psychological counseling.

##  2 Related Work

###  2.1 Dialogue Generation and Reconstruction using LLMs

Dialogue generation and reconstruction using LLMs have been proven to be
effective in data augmentation and conversation denoising. For example, SAFARI
Wang et al. (2023a) harnesses the planning and understanding capabilities of
LLMs to generate persona-consistent and knowledge-enhanced responses. In the
medical domain, DISC-MedLLM Bao et al. (2023) undertakes real-world dialogue
reconstruction for consultation records sourced from medical forums. This
process addresses issues of informal language usage and unregulated expressive
styles. In the realm of psychology, numerous studies concentrate on augmenting
emotional support capability by enhancing empathy.Qian et al. (2023) amplifies
empathetic responses by enriching the dialogue context with a commonsense
knowledge graph, thereby stimulating the relevant knowledge encoded by LLMs.

In this work, we propose a two-phase method for efficient dialogue
reconstruction.

###  2.2 Evaluation of Generated Dialogues using LLMs

The search for better automatic evaluation metrics in natural language
generation (NLG) has been a hot topic for the natural language processing
(NLP) community. Compared to conventional lexicon-based metrics like BLEU
Papineni et al. (2002) and Rouge Lin (2004), these new metrics capture deeper
semantic meaning and usually have better alignment with human judgments.

There have been a series of transformer-based evaluation metrics available in
the community, such as BERTScore Zhang et al. (2020), BARTScore Yuan et al.
(2021) and GPTScore Fu et al. (2023). In specific domains, there are also
derivatives of such metrics tailored for the domain. For example,
CodeBERTScore Zhou et al. (2023) is proposed to achieve a higher correlation
with human preference and with functional correctness. CBERTScore Shor et al.
(2023) can penalize clinically-relevant mistakes more than others.

The same trend continues with LLMs. Wang et al. (2023b) shows that ChatGPT
achieves state-of-the-art or competitive correlation with human judgments in
most cases. A new framework constructed over GPT-4 called G-Eval Liu et al.
(2023b) makes use of LLMs with chain-of-thoughts (CoT) and a form-filling
paradigm to assess the quality of NLG outputs, outperforming all previous
methods by a large margin.

In this work, we design a psychological counseling benchmark for automatic
evaluation.

##  3 CPsyCoun

(a) Distribution of counseling topics. (b) Distribution of psychological
counseling schools.

Figure 2: Statistics of collected cases.

###  3.1 Data Collection

We conduct a survey of publicly available psychological counseling cases
online and collect data from well-known Chinese psychological communities. The
online communities used in this work are: (1) Yidianling
333https://www.ydl.com, a top-tier mental health platform in China, serves
approximately 39 million users, backed by a robust network of over 6,000
professional counselors. (2) Psy525 444https://www.psy525.cn, another
prominent mental health platform in China, caters to over 1 million users and
is supported by nearly 30,000 professional counselors.

As the data are anonymized by the websites, there’s a low privacy risk. To
enhance the security of the collected data, we further conduct an analysis of
privacy and security issues about the data. The procedures adopted during data
collecting to ensure no sensitive or privacy-related content in the dataset
include rule-based cleaning, manual rewriting, and human proofreading. After
cleaning procedures, relevant private information has been completely removed,
and we ensure that relevant private information is protected.

In total, we collected 4,700 psychological counseling reports in different
formats, with a variety of types and counseling methods. These reports will
not be released to the public unless a Privacy Data Protection Agreement is
signed upon reasonable request.

###  3.2 Data Processing

To construct a high-quality dataset, we carefully selected 3,134 psychological
counseling reports. They contain complete methods and types, clear case
briefs, detailed consultation processes and experience thoughts. In the
selection process, we found that some of the collected reports contained
several counseling cases in one report. We did not select this type of report
due to multiple cases in one report where the background information of the
client and the consultation processes are incomplete. Therefore, among the
selected 3,134 psychological counseling reports, each report corresponds to
only one case. This high-quality report dataset is named CPsyCounR.

Figure 3: Illustration of the dialogue reconstruction method Memo2Demo

#### Data Format

Considering the differences in data sources of our collection, we need to
reformat these collected reports according to a uniform standard. To build a
comprehensive and helpful dataset, we combine the case format of China’s
National Class 2 Psychological Counselor Examination and other psychological
counseling literature to regularize collected reports, where the following 6
components are included: Title, Type, Method, Case Brief, Consultation Process
and Experience Thoughts. Note that the consultation process is written from a
third-person perspective and does not contain specific dialog. For detailed
description of components and examples of psychological counseling report,
please refer to Appendix A.

#### Data analysis

According to statistics, there are about 230 types of psychological counseling
cases and more than 250 counseling methods used in psychological counseling.
Considering that the classification of the original types is too detailed, we
further summarized the case types into 9 representative topics based on common
scenarios of psychological counseling. The distribution of counseling topics
is shown in Figure 2a.

Based on relevant information from the American Psychological Association
(APA) and the International Academy of Psychotherapy (IACP), we have
categorized the professional counselor’s methods utilized in psychological
counseling reports into 7 classic schools of psychological counseling. The
distribution of methods is shown in Figure 2b.

###  3.3 Dialogue Generation Method for Psychological Counseling

#### Baseline Method

Direct role-play prompting is utilized as our baseline method for generating
multiple rounds of dialogue from a single round, which has been successfully
used in previous work on dialogue generation Qiu et al. (2023); Chen et al.
(2023b). However, we believe that there are still aspects for improvement when
applying direct role-play prompting to multi-round dialogue generation in the
field of psychological counseling: (1) Comprehensiveness: Despite presenting
high-quality counseling reports to the language model, it may fail to focus on
the significant descriptions of the client’s situation within the report,
leading to subsequent dialogues that lack completeness. (2) Professionalism:
Role-playing prompted dialogues merely reference psychological methods in
generated dialogues. We hope that language models could integrate these
methods into the problem-solving process, thereby obtaining reconstructed
dialogue with professionalism. (3) Authenticity: Dialogue constructed by the
baseline method lacks the emotional interaction between the client and the
psychological counselor present in real scenarios, leaving a deficit in terms
of authenticity. We present the detailed prompt of role-play method in Figure
6 in the appendix.

#### Memo2Demo

To address the aforementioned issues of the baseline method, we propose a two-
phase framework named Memo2Demo to generate high-quality psychological
consultation dialogue from counseling reports. Mirroring real-life scenarios,
we incorporate two key roles into this framework: a psychological supervisor
together with a psychological counselor. The psychological supervisor guides
the psychological counselor on counseling techniques while ensuring the
privacy of the clients during the counseling process. Meanwhile, the
psychological counselor engages in direct dialogue with the clients to conduct
specific psychological counseling. Figure 3 illustrates the general framework
of our proposed method Memo2Demo, where a psychological counseling report is
first converted into a counseling note by the psychological supervisor, then
the psychological counselor generates the multi-turn consultation dialogue
based on both the report and the note. We present detailed prompts used for
Memo2Demo in Figure 7 and 8 in the appendix.

##### Memo Conversion

In this phase, we first assign the role of psychological supervisor to a large
language model, then prompt it to convert psychological counseling reports to
counseling notes. Specifically, the psychological supervisor comes up with a
counseling note based on the report, including basic information of counseling
and an elaborate consultation plan. One of the objectives of the counseling
note is to offer enhanced professional insights pertinent to the case,
employing distinct psychological counseling techniques to tackle the client’s
issue. In the meanwhile, it also condenses the core information related to the
client, thus improving the comprehensiveness of the subsequent psychological
counseling process. In this paper, we build the psychological supervisor based
on GLM-4 Zeng et al. (2023). Format of counseling notes is shown in the yellow
table of Figure 3.

##### Demo Generation

In this phase, we first assign the role of psychological counselor to a large
language model, then prompt it to generate multi-turn consultation dialogues
based on the psychological counseling report and the converted counseling
note. We simplify a four-stage consultation framework according to the actual
psychological counseling process. Leveraging this consultation framework, we
enhance our control over the direction of dialogue generation, improving the
professionalism of psychological counselors in multi-turn consultation
dialogues. In addition, the consultation framework is designed to efficiently
restore real scenarios and enhance the authenticity of the multi-round
consultation dialogues. In this paper, we build the psychological counselor
based on GLM-4 Zeng et al. (2023). Format of the consultation framework is
shown in the green table of Figure 3.

###  3.4 Automatic Evaluation of LLM-based Psychological Counseling

In the field of psychological counseling, assessing the quality of multi-turn
consultation dialogues has always been a challenging task. Despite having
successfully generated high-quality counseling dialogues from case reports
using Memo2Demo, we still need to verify the impact of these dialogues on
subsequent tasks. To this end, we elect to utilize CPsyCounD for supervised
fine-tuning on publicly accessible LLMs. This allows us to assess the changes
in the psychological counseling competency before and after the use of data.

Nonetheless, the multi-turn consultation dialogue that characterizes the
psychological counseling process is complex to evaluate without the input of
human experts. To address this, we first introduce evaluation metrics tailored
for multi-turn consultation dialogue. Then a turn-based dialogue evaluation
method is proposed for automatic evaluation of the psychological counseling
process. Moreover, we acknowledge the current shortfall of a comprehensive,
general multi-turn dialogue evaluation dataset within the psychological
counseling community. Such a dataset is vital for assessing LLM-based
psychological counseling. To bridge this gap, we present CPsyCounE, a general
multi-turn dialogue evaluation dataset.

#### Evaluation Metrics

In psychological counseling, the evaluation metrics remain diverse and not
universally standardized. For instance, SoulChat Chen et al. (2023b) proposes
evaluation metrics: Content, Empathy, Helpfulness and Safety. Some of these
metrics hinge on expert evaluations and lack specific scoring criteria,
favoring manual rather than objective and automatic evaluations. Similarly,
ChatCounselor Liu et al. (2023a) introduces the Counseling Bench, encompassing
seven different perspectives. While these metrics are designed to cater to the
model’s specific dialogue strategies, they lack the ability to evaluate the
overall dialogue effect. They are more adapted to single-round dialogue
evaluations and are not suitable for multi-turn dialogues.

Recognizing the aforementioned limitations in evaluating consultation
dialogues, and in order to analyze the counseling case used for dialogue
generation, we propose new evaluation metrics for multi-turn consultation
dialogues in psychological counseling. These metrics encompass four different
perspectives: Comprehensiveness, Professionalism, Authenticity, and Safety,
which are used for automatic evaluation in the rest of this paper. For each
perspective, we give its description and corresponding score criterion in
Appendix C.

#### Turn-Based Dialogue Evaluation

We propose a turn-based dialogue evaluation approach to effectively evaluate
multi-turn consultation dialogues. Denote a $m$-turn dialogue as a set of
paired elements $\\{(q_{i},r_{i})|i=1,2,...,m\\}$, where each $q_{i}$
represents a query from the client, and each corresponding $r_{i}$ represents
the counselor’s reply. We first split it into $m$ single-turn dialogue, then
prompt the model with query together with its dialogue history in each single-
turn dialogue, resulting in the corresponding single-turn response:

$\hat{r}_{i}=\left\\{\begin{aligned} f_{\mathit{LLM}}(q_{i})&,&i=1\\\
f_{\mathit{LLM}}(h_{i},q_{i})&,&1<i\leq m\end{aligned}\right.$ (1)

where $h_{i}=\\{(q_{j},r_{j})|j=1,2,...,i-1\\}$ signifies the dialogue history
before $i$-th turn, and $f_{\mathit{LLM}}(\cdot)$ denotes the inference
process of LLMs.

To automatically obtain reliable evaluation results, We employ GPT-4 Achiam et
al. (2023) to assess these responses, utilizing the evaluation metrics we
previously proposed. Concretely, we ask the model to assign an evaluation
score $\hat{s}_{i}$ for a single-turn response $\hat{r}_{i}$. Then we average
them to yield the total evaluation score of the current $m$-turn dialogue:

$s_{i}=\frac{1}{m}\sum_{i=1}^{m}\hat{s}_{i},$ (2)

For detailed prompts of single-turn response generation, please refer to
Figure 11 in the appendix.

#### CPsyCounE

SMILECHAT Qiu et al. (2023), a richly diverse and realistic multi-turn
dialogue dataset, comprises 56k multi-turn counseling dialogues, averaging
6.36 rounds per dialogue. Given its wide range of dialogue types, we choose it
as our base dataset. However, the open-source data of this dataset is not
classified by topic type. To address this limitation and conduct a more
comprehensive and explainable evaluation of models’ capabilities, we construct
a general multi-turn dialogue evaluation dataset with clear topic
classification - CPsyCounE. Leveraging the nine common counseling topics we
introduce in CPsyCounR, we manually select the five most representative
dialogues from SMILECHAT for each topic, resulting in a comprehensive
evaluation dataset of 45 cases.

##  4 Experiments

Dialogues | Role-play | Memo2Demo
---|---|---
Avg. Number of Turns | 8.2 | 8.7
Avg. Length of Client | 24.5 | 30.4
Avg. Length of Counselor | 40.2 | 49.7
Avg. Length of Dialogue | 545.8 | 622.3
Table 1: Statistics of generated dialogues. Method | School | Metrics
---|---|---
Comprehensiveness | Professionalism | Authenticity | Safety
Role-play | Psychoanalytic Therapy | 1.35 | 2.48 | 2.23 | 1.00
Cognitive Behavioral Therapy | 1.35 | 2.45 | 2.15 | 1.00
Humanistic Therapy | 1.30 | 2.15 | 1.98 | 1.00
Family Therapy | 1.28 | 2.18 | 2.00 | 1.00
Postmodern Therapy | 1.25 | 2.15 | 1.98 | 1.00
Integrative Therapy | 1.28 | 2.10 | 1.88 | 1.00
Other Therapies | 1.30 | 2.25 | 2.03 | 1.00
Memo2Demo | Psychoanalytic Therapy | 2.00 | 3.35 | 2.65 | 1.00
Cognitive Behavioral Therapy | 2.00 | 3.43 | 2.68 | 1.00
Humanistic Therapy | 2.00 | 3.55 | 2.65 | 1.00
Family Therapy | 2.00 | 3.48 | 2.70 | 1.00
Postmodern Therapy | 2.00 | 3.53 | 2.58 | 1.00
Integrative Therapy | 2.00 | 3.50 | 2.58 | 1.00
Other Therapies | 2.00 | 3.23 | 2.63 | 1.00
| Avg. Role-play | 1.30 | 2.25 | 2.04 | 1.00
Avg. Memo2Demo | 2.00 | 3.44 | 2.64 | 1.00
Improv. | +53% | +53% | +30% | -
Table 2: Results of the intrinsic evaluation on CPsyCoun. In the last row of
the table, we present the percentage improvement of metrics for Memo2Demo
compared to role-play method.

###  4.1 CPsyCounD

To validate the effectiveness of our proposed dialogue reconstruction
approach, we adopt direct role-play prompting and Memo2Demo to generate
dialogues from CPsyCounR respectively. We denote the set of dialogues
generated by Memo2Demo as CPsyCounD, which has a total of 3,134 multi-turn
consultation dialogues, covering nine topics and seven classic schools of
psychological counseling. For statistical information, please refer to Table
1.

###  4.2 Intrinsic Evaluation of CPsyCoun

To ensure comprehensiveness and diversity of the evaluation dataset, we
randomly select 20 cases from each of the seven classic schools of
psychological counseling in CPsyCounR, acquiring a total of 140 cases. Then we
adopt direct role-play prompting and Memo2Demo method respectively for
dialogue generation, and instruct GPT-4 Achiam et al. (2023) to conduct a
comparative evaluation of the above two multi-turn consultation dialogues. The
evaluation standard refers to the evaluation metrics shows in Table 4. For
detailed evaluation prompt, please refer to Figure 9 in the appendix.

Table 2 illustrates the results of intrinsic evaluation on CPsyCoun. For each
school, Memo2Demo method outperforms direct role-play prompting in terms of
Comprehensiveness, Professionalism, and Authenticity. When comparing the
overall average scores, Memo2Demo method exhibits a remarkable improvement of
53%, 53%, and 30% in these metrics respectively, when juxtaposed with direct
role-play prompting. Note that both methods get full scores in Safety, which
shows the advantage of report-based data construction methods for privacy
protection. In general, our proposed method Memo2Demo significantly enhances
the quality of reconstructed multi-turn consultation dialogues.

###  4.3 Extrinsic Evaluation of CPsyCoun

#### CPsyCounX

To delve deeper into whether the proposed dataset can effectively enhance the
psychological counseling capabilities of LLMs, we further fine-tune
InternLM2-7B-Chat Team (2023) on CPsyCounD and derive a chat model CPsyCounX
tailored specifically for psychological counseling.

CPsyCounX is fine-tuning for 9 epochs with the batch size set to 448, and the
learning rate set to ${1\times 10^{-6}}$. During fine-tuning, we adopt the
InternLM2-style template to concatenate queries and responses within the
multi-turn dialogue.

(a) Comprehensiveness (b) Professionalism (c) Authenticity

Figure 4: Radar plot of detailed scores of CPsyCounX and other baselines on 9
counseling topics on the benchmark.

Model | Metrics
---|---
Comprehensiveness | Professionalism | Authenticity | Safety
InternLM2-7B-Chat | 1.30 | 2.16 | 1.48 | 1.00
SoulChat | 1.22 | 2.18 | 2.24 | 1.00
ChatGPT | 1.32 | 2.25 | 2.09 | 1.00
GLM-4 | 1.44 | 2.36 | 2.22 | 1.00
CPsyCounX | 1.39 | 2.65 | 2.29 | 1.00

Table 3: Results of the extrinsic evaluation on CPsyCoun. The best score for
each metric is in-bold, while the second best score is underlined.

#### Automatic Evaluation

The turn-based dialogue evaluation method is adopted on CPsyCounE for the
following extrinsic evaluation. We include InternLM2-7B-Chat Team (2023),
SoulChat Chen et al. (2023b), ChatGPT and GLM-4 Zeng et al. (2023) as major
baseline models. The evaluation standard refers to the evaluation metrics in
Table 4 in the appendix. To accommodate multi-turn dialogues, we adjust the
authenticity criterion accordingly. For detailed evaluation prompt, please
refer to Figure 10 in the appendix.

We present the overall results of extrinsic evaluation on CPsyCoun in Table 3,
where CPsyCounX surpasses other models in terms of Professionalism and
Authenticity, falling behind GLM-4 only slightly in terms of
Comprehensiveness. Figure 4 further shows detailed scores of CPsyCounX and
other baselines, where CPsyCounX significantly outperforms nearly all other
baselines on Professionalism, demonstrating the efficacy of proposed method
Memo2Demo. While judging by the topic distribution, CPsyCounX leads in all
metrics in the topic "Mental Disease", demonstrating its high usability in the
field of psychological counseling. For full results, please refer to Appendix
E.

Upon evaluation, we find that GLM-4 scores the highest in Comprehensiveness,
largely because its single-turn dialogues encompass vast information. A manual
investigation reveals that GLM-4 prioritizes summarizing previous dialogues in
each turn, accounting for its high scores in Comprehensiveness. However, our
evaluation shows that excessive content tends to compromise Authenticity
scores. In psychological counseling, Authenticity and Comprehensiveness need a
balanced consideration. In our experiments, we prioritize natural and
authentic dialogues that contain key information. Consequently, while
CPsyCounX scores lower than GLM-4 in Comprehensiveness, it surpasses GLM-4 in
Authenticity.

These results highlight that fine-tuning on CPsyCounD enables the model to
naturally acquire professional psychological counseling techniques used in
counseling dialogues. Moreover, the model can learn the conversational style
of psychological counselors in real-life psychological counseling scenarios,
ensuring the dialogue’s authenticity.

##  5 Conclusion

In this paper, we introduce CPsyCoun, an innovative framework for report-based
multi-turn dialogue reconstruction and evaluation in Chinese psychological
counseling. Our research encompasses data collection, effective data
construction methods, and domain evaluation benchmarks. To harness the full
potential of psychological counseling reports, we design a two-phase approach
to construct high-quality consultation dialogues. Concurrently, we propose a
comprehensive evaluation benchmark for multi-turn consultation dialogue,
inclusive of metrics, datasets and methods. Experimental results validate the
effectiveness of our proposed framework, demonstrating its superiority in
building a comprehensive, professional, and authentic psychological counseling
assistant. All datasets and model weights developed in this paper are publicly
available. For future work, a more refined balance between authenticity and
professional knowledge in dialogue generation needs to be achieved. We aspire
this work will furnish fresh perspectives and references for the development
of LLMs in the field of psychological counseling.

## Limitations

In this work, we proposed CPsyCoun, a report-based multi-turn dialogue
reconstruction and evaluation framework for Chinese psychological counseling.
Although the experimental results demonstrate that our framework is viable,
there are still some limitations need to be considered. In real-life
scenarios, psychological counseling is complex. Counselors not only need to
empathize with clients, but also need to use psychological counseling
techniques at the appropriate time. For example, if a counselor can only
empathize, he will not be able to solve the client’s problems, and if
psychological counseling techniques are used at the wrong time, it may also
harm the solution of the problem. Therefore, dialogue generative methods and
dialogue evaluation methods need to further consider the balance between
authenticity and professional knowledge.

## Ethics Statement

### Data Privacy

We have implemented rigorous data sanitization procedures to construct the
dataset and safeguard privacy. These measures encompass rule-based cleaning,
manual rewriting, and human proofreading to guarantee the absence of sensitive
or privacy-related content. For instance, the initial data collection
contained private information of psychological counselors, including personal
specifics, contact details, residential addresses, and workplaces. Post-
sanitization, all such sensitive information has been entirely expunged,
ensuring the protection of relevant private information. Following the data
copyright formulated by Qiu et al. (2023), we release the multi-turn dialogue
evaluation dataset publicly available for research purposes only.

### Potential Risks of the Model

We carried out a safety assessment specifically for the model’s output during
the evaluation phase, the results of which are presented in Table 3. Given the
absence of human feedback during the model fine-tuning phase, it is inevitable
that some responses might potentially harm users. In the event of no
noticeable improvement after user interaction with the CPsyCounX model,
trained with multi-turn consultation dialogues, CPsyCounD, we strongly
recommend seeking assistance from a professional counselor or psychiatrist
promptly. It is critical to remember that a virtual dialogue agent may not
serve as a replacement for real-world therapy. Furthermore, when implementing
this model in downstream applications, it is essential to inform users
beforehand that the AI model generates the responses they see, and these
should be used only as references.

## Acknowledgements

This work was partially supported by the China Postdoctoral Science Foundation
(2023M733654) and Guangdong Basic and Applied Basic Research Foundation
(2023A1515110496).

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _ArXiv preprint_ , abs/2303.08774.
  * Bao et al. (2023) Zhijie Bao, Wei Chen, Shengze Xiao, Kuang Ren, Jiaao Wu, Cheng Zhong, Jiajie Peng, Xuanjing Huang, and Zhongyu Wei. 2023.  Disc-medllm: Bridging general large language models and real-world medical consultation.  _arXiv preprint arXiv:2308.14346_.
  * Buechel et al. (2018) Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc. 2018\.  Modeling empathy and distress in reaction to news stories.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4758–4765, Brussels, Belgium. Association for Computational Linguistics.
  * Chen et al. (2023a) Siyuan Chen, Mengyue Wu, Kenny Q Zhu, Kunyao Lan, Zhiling Zhang, and Lyuchun Cui. 2023a.  Llm-empowered chatbots for psychiatrist and patient simulation: Application and evaluation.  _arXiv preprint arXiv:2305.13614_.
  * Chen et al. (2023b) Yirong Chen, Xiaofen Xing, Jingkai Lin, Huimin Zheng, Zhenyu Wang, Qi Liu, and Xiangmin Xu. 2023b.  SoulChat: Improving LLMs’ empathy, listening, and comfort abilities through fine-tuning with multi-turn empathy conversations.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 1170–1183, Singapore. Association for Computational Linguistics.
  * Cheng et al. (2023) Jiale Cheng, Sahand Sabour, Hao Sun, Zhuang Chen, and Minlie Huang. 2023.  PAL: Persona-augmented emotional support conversation generation.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 535–554, Toronto, Canada. Association for Computational Linguistics.
  * Freeman (2022) M. Freeman. 2022.  The World Mental Health Report: transforming mental health for all.  _World Psychiatry_ , 21(3):391–392.
  * Fu et al. (2023) Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023.  GPTscore: Evaluate as you desire.  _arXiv preprint arXiv:2302.04166_.
  * Lai et al. (2023) Tin Lai, Yukun Shi, Zicong Du, Jiajie Wu, Ken Fu, Yichao Dou, and Ziqi Wang. 2023\.  Psy-llm: Scaling up global mental health psychological services with ai-based large language models.  _arXiv preprint arXiv:2307.11991_.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2023a) June M Liu, Donghao Li, He Cao, Tianhe Ren, Zeyi Liao, and Jiamin Wu. 2023a.  Chatcounselor: A large language models for mental health support.  _ArXiv preprint_ , abs/2309.15461.
  * Liu et al. (2021) Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. 2021.  Towards emotional support dialog systems.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3469–3483, Online. Association for Computational Linguistics.
  * Liu et al. (2023b) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023b.  G-eval: NLG evaluation using gpt-4 with better human alignment.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 2511–2522, Singapore. Association for Computational Linguistics.
  * Orr et al. (2022) Martin Orr, Kirsten Van Kessel, and Dave Parry. 2022.  The ethical role of computational linguistics in digital psychological formulation and suicide prevention. In _Proceedings of the Eighth Workshop on Computational Linguistics and Clinical Psychology_ , pages 17–29, Seattle, USA. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Prince et al. (2007) Martin Prince, Vikram Patel, Shekhar Saxena, Mario Maj, Joanna Maselko, Michael R Phillips, and Atif Rahman. 2007.  No health without mental health.  _The Lancet_ , 370(9590):859–877.
  * Qian et al. (2023) Yushan Qian, Weinan Zhang, and Ting Liu. 2023.  Harnessing the power of large language models for empathetic response generation: Empirical investigations and improvements.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 6516–6528, Singapore. Association for Computational Linguistics.
  * Qiu et al. (2023) Huachuan Qiu, Hongliang He, Shuai Zhang, Anqi Li, and Zhenzhong Lan. 2023.  Smile: Single-turn to multi-turn inclusive language expansion via chatgpt for mental health support.  _arXiv preprint arXiv:2305.00450_.
  * Rashkin et al. (2019) Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019.  Towards empathetic open-domain conversation models: A new benchmark and dataset.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5370–5381, Florence, Italy. Association for Computational Linguistics.
  * Shor et al. (2023) Joel Shor, Ruyue Agnes Bi, Subhashini Venugopalan, Steven Ibara, Roman Goldenberg, and Ehud Rivlin. 2023.  Clinical BERTScore: An improved measure of automatic speech recognition performance in clinical settings.  In _Proceedings of the 5th Clinical Natural Language Processing Workshop_ , pages 1–7, Toronto, Canada. Association for Computational Linguistics.
  * Sun et al. (2021) Hao Sun, Zhenru Lin, Chujie Zheng, Siyang Liu, and Minlie Huang. 2021.  PsyQA: A Chinese dataset for generating long counseling text for mental health support.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 1489–1503, Online. Association for Computational Linguistics.
  * Team (2023) InternLM Team. 2023.  Internlm: A multilingual language model with progressively enhanced capabilities.  https://github.com/InternLM/InternLM.
  * Toleubay et al. (2023) Yeldar Toleubay, Don Joven Agravante, Daiki Kimura, Baihan Lin, Djallel Bouneffouf, and Michiaki Tatsubori. 2023.  Utterance classification with logical neural network: Explainable AI for mental disorder diagnosis.  In _Proceedings of the 5th Clinical Natural Language Processing Workshop_ , pages 439–446, Toronto, Canada. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Wang et al. (2023a) Hongru Wang, Minda Hu, Yang Deng, Rui Wang, Fei Mi, Weichao Wang, Yasheng Wang, Wai-Chung Kwan, Irwin King, and Kam-Fai Wong. 2023a.  Large language models as source planner for personalized knowledge-grounded dialogues.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 9556–9569, Singapore. Association for Computational Linguistics.
  * Wang et al. (2023b) Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023b.  Is ChatGPT a good NLG evaluator? a preliminary study.  In _Proceedings of the 4th New Frontiers in Summarization Workshop_ , pages 1–11, Singapore. Association for Computational Linguistics.
  * Yan and Xue (2023) Xin Yan and Dong Xue. 2023.  Mindchat: Psychological large language model.  https://github.com/X-D-Lab/MindChat.
  * Yang et al. (2023) Kailai Yang, Shaoxiong Ji, Tianlin Zhang, Qianqian Xie, Ziyan Kuang, and Sophia Ananiadou. 2023.  Towards interpretable mental health analysis with large language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 6056–6077, Singapore. Association for Computational Linguistics.
  * Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021.  Bartscore: Evaluating generated text as text generation.  In _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , pages 27263–27277.
  * Zeng et al. (2023) Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. 2023.  GLM-130b: An open bilingual pre-trained model.  In _The Eleventh International Conference on Learning Representations_.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  Bertscore: Evaluating text generation with BERT.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Zhou et al. (2023) Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. 2023.  CodeBERTScore: Evaluating code generation with pretrained models of code.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 13921–13937, Singapore. Association for Computational Linguistics.

##  Appendix A Unified Format of Psychological Counseling Reports

Considering the differences in data sources of our collection, we need to
reformat these collected reports according to a uniform standard. To
regularize the psychological counseling reports collected from different data
sources, we include six components within each report. In Figure 5, we offer
detailed description of these components, accompanied by real examples from
collected reports. Note that the original case is written in Chinese, while
the English version is translated by us for reference.

Figure 5: Description of report format together with an example from our
collection

##  Appendix B Prompts of Dialogue Reconstruction Methods

In this paper, we adopt direct role-play prompting as the baseline method for
dialogue reconstruction. Detailed prompt used for this approach is given in
Figure 6. To further improve the role-play method, we propose a two-phase
method Memo2Demo for effective dialogue reconstruction, which consists of memo
conversion and demo generation. Detailed prompts used in these phases are
given in Figure 7 and Figure 8, respectively.

Figure 6: The prompt of direct role-play prompting method Figure 7: The prompt
of memo conversion phase in Memo2Demo method Figure 8: The prompt of demo
generation phase in Memo2Demo method

##  Appendix C Evaluation Metrics and Score Criterion

Perspective |  Description | Criterion | Score
---|---|---|---
Comprehensiveness |  The client’s situation and the degree to which psychological problems are reflected in the dialogues. |  1.1 Does the dialogue reflect the basic information about the client? | 1 | 2
1.2 Does the dialogue reflect the client’s psychological problems? | 1
Professionalism |  The professionalism of the psychological counselor during the dialogues. |  2.1 Does the counselor demonstrate professional ability to diagnose psychological problems? | 0.5 | 4
2.2 Does the counselor use professional psychological counseling techniques? | 0.5
2.3 Is the counselor’s language professional and is there a guided dialogue? | 0.5
2.4 Does the dialogue reflect the client’s purpose of consultation? | 0.5
2.5 Does the dialogue proceed in the order of the professional consultation framework (Reception and inquiry stage, Diagnostic stage, Consultation stage, Consolidation and ending stage)? | 1
2.6 Is there a specific implementation process for psychological counseling technology, as detailed and clear as possible? | 1
Authenticity |  The degree of authenticity between the client and the counselor in the dialogues. |  3.1 Does the client express emotions and their evolution that fit the scenario? | 1 | 3
3.2 Does the counselor listen to, understand, and empathize with the client? | 0.5
3.3 Does the dialogue avoid expressions that may cause misunderstanding or discomfort? | 0.5
3.4 Does the dialogue avoid long statements and is consistent with real psychological counseling scenarios? | 1
Safety |  The degree of privacy protection of clients. |  4.1 Does the dialogue comply with psychological counseling privacy guidelines and avoid disclosing sensitive information (personal name, workplace, contact information, home address)? | 0.5 | 1
4.2 Does the dialogue respect the client’s thoughts and emotions? | 0.5
Table 4: Evaluation Metrics and Corresponding Score Criterion.

To fill the blank of evaluating multi-turn dialogues in psychological
counseling, we specifically propose four metrics for automatic evaluation. We
give the detailed description of these metrics together with the score
criterion in Table 4.

##  Appendix D Prompts of Automatic Evaluation

In this paper, we adopt GPT-4 as a judge for automatic evaluation from both
intrinsic and extrinsic perspectives. For intrinsic evaluation, we ask the
judge to compare consultation dialogues reconstructed by different methods,
which is shown in Figure 9 in detail. For extrinsic evaluation, we first fine-
tune a chat model CPsyCounX with the proposed high-quality multi-turn dialogue
dataset. Then we ask the judge to compare responses from CPsyCounX and other
baseline models, which is shown in Figure 10 in detail. The prompt to generate
single-turn responses from different models is given in Figure 11.

Figure 9: The prompt of intrinsic evaluation on CPsyCoun Figure 10: The prompt
of extrinsic evaluation on CPsyCoun Figure 11: The prompt of single-turn
response generation in extrinsic evaluation

##  Appendix E Full Results of extrinsic evaluation

For CPsyCounX and other baseline models, we give the full results of extrinsic
evaluation on different topics in Table 5.

Topic | Model | Metrics
---|---|---
Comprehensiveness | Professionalism | Authenticity | Safety
Self-growth | InternLM2-7B-Chat | 1.31 | 2.31 | 1.38 | 1.00
SoulChat | 1.25 | 2.19 | 2.25 | 1.00
ChatGPT | 1.38 | 2.38 | 2.06 | 1.00
GLM-4 | 1.50 | 2.75 | 2.19 | 1.00
CPsyCounX | 1.44 | 2.69 | 2.38 | 1.00
Emotion&Stress | InternLM2-7B-Chat | 1.19 | 2.19 | 1.31 | 1.00
SoulChat | 1.25 | 2.13 | 2.19 | 1.00
ChatGPT | 1.25 | 2.31 | 2.13 | 1.00
GLM-4 | 1.38 | 2.50 | 2.38 | 1.00
CPsyCounX | 1.38 | 2.75 | 2.38 | 1.00
Education | InternLM2-7B-Chat | 1.36 | 2.21 | 1.36 | 1.00
SoulChat | 1.21 | 2.14 | 2.21 | 1.00
ChatGPT | 1.29 | 2.36 | 2.14 | 1.00
GLM-4 | 1.50 | 2.43 | 2.29 | 1.00
CPsyCounX | 1.43 | 2.86 | 2.21 | 1.00
Love&Marriage | InternLM2-7B-Chat | 1.31 | 1.94 | 1.63 | 1.00
SoulChat | 1.19 | 2.13 | 2.25 | 1.00
ChatGPT | 1.25 | 2.19 | 2.06 | 1.00
GLM-4 | 1.38 | 2.25 | 2.31 | 1.00
CPsyCounX | 1.31 | 2.44 | 2.44 | 1.00
Family Relationship | InternLM2-7B-Chat | 1.31 | 2.06 | 1.50 | 1.00
SoulChat | 1.25 | 2.19 | 2.25 | 1.00
ChatGPT | 1.38 | 2.13 | 2.00 | 1.00
GLM-4 | 1.63 | 2.38 | 2.06 | 1.00
CPsyCounX | 1.44 | 2.81 | 2.25 | 1.00
Social Relationship | InternLM2-7B-Chat | 1.31 | 2.13 | 1.81 | 1.00
SoulChat | 1.25 | 2.31 | 2.31 | 1.00
ChatGPT | 1.38 | 2.13 | 2.06 | 1.00
GLM-4 | 1.38 | 2.19 | 2.25 | 1.00
CPsyCounX | 1.44 | 2.69 | 2.25 | 1.00
Sex | InternLM2-7B-Chat | 1.29 | 2.14 | 1.71 | 1.00
SoulChat | 1.21 | 2.14 | 2.21 | 1.00
ChatGPT | 1.43 | 2.29 | 2.29 | 1.00
GLM-4 | 1.36 | 2.14 | 2.14 | 1.00
CPsyCounX | 1.29 | 2.29 | 2.21 | 1.00
Career | InternLM2-7B-Chat | 1.38 | 2.25 | 1.19 | 1.00
SoulChat | 1.19 | 2.19 | 2.25 | 1.00
ChatGPT | 1.31 | 2.25 | 2.00 | 1.00
GLM-4 | 1.44 | 2.31 | 2.19 | 1.00
CPsyCounX | 1.31 | 2.56 | 2.13 | 1.00
Mental Disease | InternLM2-7B-Chat | 1.25 | 2.25 | 1.42 | 1.00
SoulChat | 1.17 | 2.17 | 2.25 | 1.00
ChatGPT | 1.25 | 2.25 | 2.08 | 1.00
GLM-4 | 1.42 | 2.33 | 2.17 | 1.00
CPsyCounX | 1.50 | 2.75 | 2.33 | 1.00
Total Average | InternLM2-7B-Chat | 1.30 | 2.16 | 1.48 | 1.00
SoulChat | 1.22 | 2.18 | 2.24 | 1.00
ChatGPT | 1.32 | 2.25 | 2.09 | 1.00
GLM-4 | 1.44 | 2.36 | 2.22 | 1.00
CPsyCounX | 1.39 | 2.65 | 2.29 | 1.00

Table 5: Full results of extrinsic evaluation on CPsyCounX and other baseline
models. The best score for each counseling topic is in-bold, while the second
best score is underlined.