# Current Limitations of Language Models: What You Need is Retrieval

Aran Komatsuzaki
Georgia Institute of Technology
EleutherAI
<EMAIL_ADDRESS>

###### Abstract

We classify and re-examine some of the current approaches to improve the
performance-computes trade-off of language models, including (1) non-causal
models (such as masked language models), (2) extension of batch length with
efficient attention, (3) recurrence, (4) conditional computation and (5)
retrieval. We identify some limitations (1) - (4) suffer from. For example,
(1) currently struggles with open-ended text generation with the output
loosely constrained by the input as well as performing general textual tasks
like GPT-2/3 due to its need for a specific fine-tuning dataset. (2) and (3)
do not improve the prediction of the first $\sim 10^{3}$ tokens. Scaling up a
model size (e.g. efficiently with (4)) still results in poor performance
scaling for some tasks. We argue (5) would resolve many of these limitations,
and it can (a) reduce the amount of supervision and (b) efficiently extend the
context over the entire training dataset and the entire past of the current
sample. We speculate how to modify MARGE to perform unsupervised causal
modeling that achieves (b) with the retriever jointly trained.

##  1 Classification of Recent Language Model Approaches

We consider some of the successful recent language model approaches to improve
the performance-computes trade-off of language model (Kaplan et al., 2020) and
classify them for the ease of our argument. Each category tends to be
orthogonal to each other, so that they can often be used in combination for
further gain. Fig. 1 summarizes our classification. There are four major
categories: non-causal models, extension of batch length with efficient
attention, memory and retrieval. We note that batch length in this paper is
defined to be the length of a minibatch across timestep dimension. We avoid
calling it context length to be more specific. In this paper, the default
modality of our consideration is text unless specified otherwise. Also, memory
and retriever are defined to possess a certain property addressed later in
this section, so their usage may differ from the usual one. Efficient
attention also refers to non-attention alternatives with better complexity,
such as variants of convolution, for the convenience of our argument.

Non-causal models in this paper are defined to be the model that predicts a
token using future information. The example includes various BERT variants,
including BART and T5 (Devlin et al., 2018; Raffel et al., 2019; Yang et al.,
2019; Liu et al., 2019; Beltagy et al., 2020; Zaheer et al., 2020; Lewis et
al., 2019; Hendrycks et al., 2020). On the other hand, causal models include
the original Transformer encoder-decoder (Vaswani et al., 2017) and GPT-2/3.
Though we usually consider the decoder-only model as the latter for a causal
model, it should be clear from the context whether a causal model refers to
the former, the latter or both.

The most long-range language modeling approaches that rely on efficient
attention without recurrence or memory, including Reformer (Kitaev et al.,
2020), Routing Transformer (Roy et al., 2020), Big Bird (Zaheer et al., 2020)
and many others (Beltagy et al., 2020; Tay et al., 2020; Sukhbaatar et al.,
2019; Child et al., 2019), extend batch length with efficient attention
without suffering from quadratic complexity in terms of length. Notably, the
approaches as Transformer-XL (Dai et al., 2019) and Compressive Transformer
(Rae et al., 2019) are not included in this category, since they are recurrent
and their goal is to extend the context rather than batch length per se.

It is reasonable to assume that simply extending batch length may not be
enough to cover all the information needed for prediction. To extend the
context beyond batch length, one needs to resort to either memory or
retriever, with which the model can attend to stored relevant information
without increased batch length. Essentially, these methods enable attention
over the entire training dataset and the context of the current sample.

In this paper, memory-based methods and retrieval-based methods refer to the
models that are equipped with memory and retriever, respectively. Memory is
defined to retrieve relevant information from its own dense tensor that
encodes the training dataset and the context of the current sample, whereas
retriever is defined to retrieve relevant segments of data from the training
dataset and the context of the current sample in the original, raw format.

Notably, memory is more straightforward, since the parameters of a neural
network can be considered as a form of memory, as they are updated through
backpropagation so as to incorporate the input information to improve their
prediction in subsequent iterations. We refer to this type of memory as
implicit memory and any method of improving the capacity to store the input
information in this way beyond the baseline as an implicit memory approach. On
the other hand, if a model combines the input information with the stored
activations from the previous iteration to use it for the next iteration in
the forward pass, the sent activations is called explicit memory. In other
words, the memory used in recurrence at the level of TBPTT segment (e.g.
Transformer-XL) is referred to as explicit memory. It is also possible to
store the activations and keep them as trainable parameters, which can be
called hybrid memory, and they share the same properties as both types. For
example, explicit memory approach includes Transformer-XL, Compressive
Transformer and many others (Yang et al., 2019; Burtsev & Sapunov, 2020),
while implicit memory approach includes simply increasing the parameter count
and conditional computation approach, such as Sparsely-gated MoE (MoE)
(Shazeer et al., 2017; Lepikhin et al., 2020), Product Key Memory (PKM)
(Lample et al., 2019) and others (Dehghani et al., 2018; Lugosch et al.,
2020).

On the other hand, it is less clear how to design a good retriever, since the
data to be retrieved is stored in a raw format unlike for memory. The
retrieval-based approach can be distinguished by whether the retriever and the
language model are trained jointly (end-to-end differentiably) or not, where
the former refers to models as MARGE, KIF (Fan et al., 2020), REALM (Guu et
al., 2020) and CRISS (Tran et al., 2020), and the latter refers to the rest of
retrieval-based approaches. If they are not trained jointly, the retriever is
usually either a tf-idf variant such as BM25 (Robertson & Zaragoza, 2009)
(e.g. (Wang & McAllester, 2020)), DPR (Karpukhin et al., 2020) (e.g. Fusion-
in-Decoder (Izacard & Grave, 2020), RAG (Lewis et al., 2020b)) or a pre-
trained language model (e.g. knn-LM and BERT-knn (Kassner & Schütze, 2020)).
While many of these models have not been applied for unsupervised causal
modeling, we will pose a general idea to convert them for unsupervised causal
modeling.

For discussion about some relatively successful examples of attempts to
improve the performance-computes trade-off outside of the categories
mentioned, the reader can refer to Appendix 6.1.

(a)  Figure 1: Classification of recent language model approaches to improve
performance-computes trade-off. Blue indicates a major category. Orange
indicates an example. Batch length in this paper is defined to be the length
of a minibatch across timestep dimension.

##  2 Non-Causal Models

For various tasks in NLP, non-causal models tend to show better performance-
computes trade-off than causal models. For example, T5 outperforms the causal
baseline not only in discriminative tasks but also generative tasks such as
summarization and translation. BART achieves lower perplexity over the causal
baseline on the task of fine-tuning to a conversational dataset. Furthermore,
it was recently reported that XLNet achieves a superior performance to the
GPT-3 of the corresponding size at Winogrande with few-shot learning (Davison,
2020). UnifiedQA, a T5-like pre-trained model fine-tuned with a general QA
dataset, is evaluated with few-shot learning examples a la GPT-3, and it
performs nearly on par with about 30 times larger GPT-3 (that is not fine-
tuned) (Hendrycks et al., 2020).

However, the current approach of non-causal models suffer from some
limitations.

Firstly, many real-life tasks are open-ended text generation with the output
loosely constrained by the input in a way similar to many of the prompt-output
generation tasks a la GPT-2 and long-range unsupervised text generation.
However, non-causal models are rarely evaluated on this sort of tasks. This
practice unfairly favors a certain family of models, including masked language
models, since they tend to perform poorly compared with the causal models.
More details can be found in Appendix 6.2.1. While non-causal models excel at
reading/encoder-intensive tasks, they perform poorly on some writing/decoder-
intensive tasks. There needs to be more effort spent on seamlessly combining
causal and non-causal models to achieve the best of both worlds.

Secondly, it is unclear how to fine-tune a non-causal model to perform an
arbitrary textual task as GPT-2/3 at least as efficiently as GPT-2/3. Unlike
GPT-2/3, non-causal models rely on the availability of a fine-tuning dataset
relevant to a given task, which is usually not known beforehand. Hence, non-
causal models are likely to perform poorly on the tasks where there is no
fine-tuning dataset available for. This cannot be naively resolved by using a
very general or large fine-tuning dataset as WebText to deal with various
tasks, since the specificity of fine-tuning dataset is crucial for the power
of fine-tuning. For example, Raffel et al. (2019) shows that conventionally
fine-tuned T5 substantially outperforms the baseline fine-tuned with all the
fine-tuning datasets combined, which implies that fine-tuning dataset needs to
be specific. They also show that, when a fine-tuning dataset is large enough,
the fine-tuned model degenerates into a causal model in terms of its
performance, as the benefit of pre-training vanishes (Fig. 2). Thus, there
needs to be more investigation into how non-causal models would perform on
previously unknown tasks compared with causal counterparts and how to improve
the performance if necessary.

(a)  Figure 2: (Raffel et al., 2019). The performance of the baseline T5 with
and without fine-tuning on various datasets. Pre-training is beneficial for
all fine-tuning datasets but WMT EnFr, which is the largest dataset.

##  3 Extension of Batch Length With Efficient Attention

Before beginning this section, we have a few caveats. Since the modality of
our primary concern in this paper is text, our conclusion regarding the
approach of extending batch length with efficient attention would differ if
the modality of interest is different, especially if it has a structure more
exploitable with an inductive bias such as image and video. In particular, we
believe that, unlike text, such exploitable modalities will benefit
substantially more from better efficient attention or related architectural
modification than text. However, this is out of the scope of this paper.

There have been many recent works that propose a language model that simply
replaces full attention with an efficient attention to improve the
computational complexity in hope of extending the batch length, i.e., from
$O(Ld_{model}^{2}+L^{2}d_{model})$ to $O(Ld_{model}^{2}+f(L)d_{model})$, where
$L$ is the batch length, and $f(L)$ is some function such that
$f(L)=o(L^{2})$. Notable example of $f(L)$ includes linear, log-linear and
$O(L^{3/2})$. Since we have $Ld_{model}^{2}>L^{3/2}d_{model}$ if
$L<d_{model}^{2}\sim 10^{6}$, in practice even $f(L)=O(L^{3/2})$ is sufficient
to make the contribution of attention not to dominate that of linear layer.
Thus, the computational complexity of many efficient attention models is
already empirically on par with that of linear layers. In other words, one can
no longer practically gain from improving the complexity of efficient
attention any further.

Most naturally available samples as well as the reasonable output of most
tasks have rather limited length, though others (e.g. books) do not. For
example, the average sample length of WebText is only about $1000$ tokens, and
likewise for the webpages in Common Crawl. Also, when batch length is extended
from $L$ (e.g. $1024$) to $L^{\prime}$, this does not tend to result in
substantial improvement in the loss of the first $L$ tokens of a sample. For
example, Fig. 3 (left) demonstrates this. On the other hand, other approaches
such as increasing the number of parameters (implicit memory) would improve
the loss for early tokens as shown in Fig. 3 (right). It can be found on
Appendix 6.3.4 a more detail on how each approach improves or does not improve
the loss of early (the first $L$) tokens of each sample. In particular,
Appendix 6.3.4 concludes that any approach that satisfies a certain condition
(including retrieval and implicit memory approach) and improves the loss does
improve the early token prediction. Since many text data have rather limited
sample length, the gain from longer batch length due to efficient attention
can be observed only on a small subset of entire text data. Even for the long
samples, the prediction is poor for its early tokens, which results in poor
generation at the beginning and cannot be mitigated by better loss of latter
tokens.

The approach of simply replacing full attention with efficient attention to
extend the batch length has been successful to some extent. We summarize our
observations about this approach in this section as well as that of Appendix
6.3 (including the one subsections not mentioned above) as follows:

  1. 1.

In practice, one cannot improve the effective computational complexity of
efficient attention models any significantly further due to the contribution
from linear layers.

  2. 2.

Appendix 6.3.1 shows that improvement from extending the batch length would be
limited at some point by the trade-off between model size and batch size that
follows from Kaplan et al. (2020).

  3. 3.

This approach is unlikely to result in any improvement for modeling the first
$L\sim 10^{3}$ tokens of a sample, i.e., the entirety of many samples of our
interest, that tend to have short length, and the early tokens of long
samples, whose suboptimal generation would also affect the generation of the
later tokens.

  4. 4.

Appendix 6.3.3 argues that it is possible to combine efficient attention with
retrieval-based approach to increase $k$ of $k$NNs retrieved for possibly
further improvement of performance. However, it is uncertain if further
improvement in efficient attention would bring a notable gain for this use
over the existing efficient attention.

(a)

(b)

Figure 3: (Kaplan et al., 2020). (Left): $n$-th per-token loss for various $n$
on models with various parameter counts and batch lengths. Training runs with
shorter batch length $L=8$ (dashed lines) perform better on early tokens,
which implies that the model with longer batch length does not improve theIf
anyone loss of early tokens that have shorter context. (Right): $n$-th per-
token loss for various $n$ on models with various parameter counts with
$L=1024$. Even for early tokens, increasing the parameter count results in a
significant improvement in their loss.

##  4 Memory-Based Approach

There are both pros and cons for each of explicit approach and implicit
approach. Explicit memory (recurrence) is more sensitive to the recent past in
contrast to implicit memory. For example, the perplexity of Transformer does
not improve whether the minibatch of the previous iteration contains the
immediate past of a sample in the current minibatch, while the performance of
Transformer-XL (explicit memory) improves if its memory contains the
activations of the immediate past of the current sample. This property is an
advantage for explicit memory. However, explicit memory often cannot utilize
information of the distant past such as the information from samples processed
many iterations ago. While implicit memory (e.g. increasing the parameter
count) can utilize such information by storing it in more parameters as can be
seen from the fact that it improves the loss of early tokens (Fig. 3), this is
not the case for existing explicit memory approaches such as Transformer-XL.
The issue that recurrent models cannot utilize information well beyond the
range of batch length through recurrence has been known for a long time, and
there has been little progress in this direction. Thus, many of our
conclusions for the approach of extending batch length with efficient
attention apply to explicit memory as well.

One may counter the above argument by noting that self-attention is known to
perform poorly on some tasks recurrence excel at. We believe this is not a
valid counter-argument as addressed in Appendix 6.4.1.

Implicit memory approach, by definition, aims for compressing as much
information as possible in the parameters in a way such that it can retrieve
them efficiently. Since having more parameters allows storing more
information, conditional computation is more advantageous. The most successful
conditional computation approach thus far is presumably MoE. Both MoE and PKM
essentially expand the effective size of $d_{ff}$ and sparsely accesses to
each unit as discussed in Appendix 6.4.2. Obviously, there are other
dimensions of Transformer that can be expanded with conditional computation,
including depth and heads. We can also regard, for example, Routing
Transformer as a conditional computation approach to let each head to attend
to some timesteps only, though we separate them in this paper for the ease of
argument. Though we have tried to conditionally expand depth and the heads, it
has not saved performance-computes trade-off thus far. There are many works
that have tried to expand depth conditionally (Dehghani et al., 2018; Elbayad
et al., 2019), but there has been no such work that demonstrated a robust
scaling and a substantial improvement in performance-computes trade-off. We
hope this problem will be resolved soon.

Since conditional computation aims for saving the computes and adding the
parameters, the parameter GPU memory may eventually become a bottleneck if the
model size keeps increasing. However, L2L proposed in Pudipeddi et al. (2020)
reduces the parameter GPU memory from $O(L)$ to $O(1)$ by saving the
parameters in CPU, where $L$ is the number of layers, while maintaining
comparable speed.

Despite the success of conditional computation, the models that are made
larger and more capacious with conditional computation do not necessarily
outperform retrieval-based methods, as argued in Appendix 6.4.3. This
motivates for the use of retrieval-based approach in conjunction to
conditional computation.

This section can be summarized as follows:

  1. 1.

Explicit memory approach suffers from many of the same problems as extending
batch length with efficient attention. Hence, most of our conclusions in the
previous section applies to this method as well.

  2. 2.

Conditional computation, as a branch of implicit memory approach, brings a
dramatic improvement of performance-computes trade-off, usually by efficiently
approximating a Transformer with larger hyperparameters such as $d_{ff}$.

  3. 3.

Given that the vanilla models that spend more computes perform poorly at some
tasks compared with substantially smaller retrieval-based models, retrieval-
based approach should also be needed, though there still is a space left for
conditional computation to bring more gains.

##  5 Retrieval-Based Approach

###  5.1 Retrieval as Self-supervision

The supervision as few-shot learning and fine-tuning may not be always
possible in the general real-life tasks, since there are numerous, diverse
tasks, and naively finding a set of relevant samples for each task accurately
may not be feasible. This necessitates the use of self-supervision to reduce
the required amount of supervision.

The conventional method of construction of a dataset or feeding samples into a
model can be thought of as a form of retrieval. Retrieval with a language
model would partially automate this process and reduce the amount of
supervision required to solve a given task. For example, CRISS, a retrieval-
based model and the current state-of-the-art of unsupervised machine
translation (even without back-translation) over various language directions
with substantial improvement over its baseline, retrieves the target for a
given input sequence. The retriever essentially is trained to learn what
humans would feed for a given input sequence. Due to the ubiquity of this
procedure in machine learning, retrieval should be a crucial component.

###  5.2 Retrieval Resolves Many Limitations Other Approaches Suffer From

As stated at the end of the previous section and Appendix 6.4.3, retrieval-
based models, such as Fusion-in-Decoder and MARGE, often substantially
outperform the Transformer language models that either have a larger model
size or spend more computes on various tasks, which implies that it is not
only sufficient to scale up a model but also equip it with a retriever to
improve the performance of language model. Hence, retrieval complements
conditional computation in some tasks. Retrieval also improves early token
prediction as argued in Appendix 6.3.4. Retrieval may also help non-causal
models perform general textual task through self-supervision.

As argued previously, efficient attention and recurrence have limited range of
context both within the same sample and outside of it. However, retrieval can
achieve indefinite context length within the same sample and the entire
training dataset. For example, knn-LM achieves the state-of-the-art in
Wikitext-103 for the amount of computes thanks to its context over the entire
training dataset enabled by $k$NN search with faiss. While there has been no
demonstration of achieving indefinite context length within the same sample
with retrieval, in principle this can be similarly done as suggested later.

Retrieval can also self-supervise few-shot learning of GPT-3 by feeding
retrieved relevant samples. In fact, our proposed modified MARGE presented
below trains the retriever that retrieves few-shot learning examples with the
language model jointly, so that both components can benefit from the
improvement of each other.

###  5.3 MARGE and Its Significance

For retrieval-based models, it is more desirable to train the retriever and
the language model jointly in a way such that the objective of the retriever
is to maximize the performance of the language model, since models with
separately trained retriever, such as knn-LM. There are various models with
jointly trained retriever, but we focus on MARGE below, as MARGE also
incorporates the trait of Fusion-in-Decoder, the attention bias for finding
the similarity between source segments and target segments efficiently. MARGE
performs respectably, for example, even without fine-tuning or back
translation on unsupervised zero-shot machine translation with relatively
small computes spent, not to mention that it also achieves the state-of-the-
art performance on some tasks on translation and summarization with smaller
computes spent.

MARGE has the added attention bias set in a way such that it becomes larger
between a source segment and a target segment closer to each other in terms of
cosine similarity of their embeddings, and it computes the prediction loss of
the target segments that are conditioned on the source segments. This design
allows gradient update to make the embedder to embed a target segment and a
source segment closer in terms of cosine similarity if the perplexity of the
target segment conditioned on the source segment is lower. Thus, the model can
find the source segments that give a lower perplexity to a given target
segment, and simultaneously the language model conditioned on retrieved
contexts can be trained. After many iterations, the clusters are no longer up-
to-date to the trained embedder. Hence, the embedder is used to find more up-
to-date clusters of related segments using $k$NN search of the newly computed
embeddings.

We outline the notable aspects of MARGE, while skipping the details irrelevant
to our subsequent discussion:

  * •

We first divide each sample from the dataset into segments, each of which
consists of $N$ consecutive tokens, where $N=512$ in the paper, and treat them
as samples.

  * •

We heuristically create a set of shards of segments, so that each segment is
closely related to others in the same shard.

  * •

MARGE uses the conventional Transformer encoder-decoder architecture with the
usual prediction loss on the decoder. The normalized output of the first half
layers of the encoder at the first timestep is used as the embedding of each
segment for weight sharing.

  * •

After each $I$ iterations, where $I=10000$ in the paper, we compute the
embedding of each segment and the similarity between each embedding within
each shard and sample clusters of $k$-nearest neighbors for batch
construction.

  * •

Each batch consists of sub-batches, each of which consists of the same number
of evidence (source) segments and target segments, which are similar to each
other within the same shard and processed by the encoder and the decoder,
respectively.

  * •

The encoder-decoder attention comes from every source segment to every target
segment within the same sub-batch. The encoder-decoder attention from the
source segment $z_{i}$ to the target segment $x_{j}$ is modified in a way such
that $q\cdot k\to q\cdot k+\beta e(x_{j})\cdot e(z_{i})$ for a query $q$, a
key $k$, the embedder $e$ and a trainable scalar $\beta$ with softmax taken
over all the tokens in the source segments within the same sub-batch.

###  5.4 Modified MARGE

For combining retrieval with unsupervised causal modeling, the general idea is
that, as in Wang & McAllester (2020), the current context is used (embedded in
this case) to retrieve similar segments across the dataset and the past of the
current sample, which are added to the context for the prediction. Unlike Wang
& McAllester (2020), we would like to retrieve the relevant contexts multiple
times per sample so as to retrieve the contexts more relevant to the most
recently available context. Hence, when we divide a sample into multiple
segments, each of which has length $N$, (i.e., a sequence $S_{0},S_{1},\ldots
S_{t},\ldots$, where $S_{t}$ is the $t$-th segment of the sample in the order
of timestep), we want to make the value of $N$ reasonably small, e.g., $N\ll
512$, but large enough so that each segment makes sense by itself. If $S_{t}$
is a source segment, $S_{t+1}$ is used as its target segment. Using faiss,
this allows unsupervised causal modeling over the entire training dataset as
well as the entire past of the current sample. Fig. 4 compares the
conventional unsupervised causal modeling and our modified MARGE.

(a)  Figure 4: The conventional unsupervised causal modeling (left) vs. our
modified MARGE (right). Note that each block consists of multiple tokens. If a
model tries to predict the tokens in last two blocks (red), the attention is
paid to the past tokens in the red region as well as the tokens in context
(green). The modified MARGE reads the immediate context (purple) to find the
relevant parts across the dataset and the past of the current sample in order
to use them as a context.

While the original MARGE uses a heuristics exploiting the metadata of each
document to collect similar segments into a shard, for more general case
without metadata or to find similarity among samples with not necessarily
similar metadata, the use of $k$NN search with faiss (Johnson et al., 2017) is
preferred. At the beginning of the training, we can let each sub-batch to
consist of the segments from the same sample (or from similar samples), since
poorly initialized clusters are known to result in slow improvement. The model
also needs to mask the encoder-decoder attention from a segment to its past
segment in the same sample in order to preserve causality.

#### Acknowledgments

I wish to record my deep sense of gratitude and profound thanks to Loren
Lugosch, Madison May, Lukasz Kaiser and Phillip Wang for their helpful,
inspiring, well-thought feedback.

## References

  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan.  Longformer: The Long-Document Transformer.  _arXiv e-prints_ , art. arXiv:2004.05150, April 2020.
  * Burtsev & Sapunov (2020) Mikhail S. Burtsev and Grigory V. Sapunov.  Memory Transformer.  _arXiv e-prints_ , art. arXiv:2006.11527, June 2020.
  * Child et al. (2019) Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever.  Generating Long Sequences with Sparse Transformers.  _arXiv e-prints_ , art. arXiv:1904.10509, Apr 2019.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov.  Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context.  _arXiv e-prints_ , art. arXiv:1901.02860, Jan 2019.
  * Davison (2020) Joe Davison.  Twitter post: Comparison of GPT-3 with XLNet with Few-Shot Learning.  2020\.  URL https://twitter.com/joeddav/status/1285238992724787200.
  * Dehghani et al. (2018) Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser.  Universal Transformers.  _arXiv e-prints_ , art. arXiv:1807.03819, July 2018.
  * Devlin et al. (2018) J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _ArXiv e-prints_ , October 2018.
  * Elbayad et al. (2019) Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli.  Depth-Adaptive Transformer.  _arXiv e-prints_ , art. arXiv:1910.10073, October 2019.
  * Fan et al. (2020) Angela Fan, Claire Gardent, Chloe Braud, and Antoine Bordes.  Augmenting Transformers with KNN-Based Composite Memory for Dialogue.  _arXiv e-prints_ , art. arXiv:2004.12744, April 2020.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang.  REALM: Retrieval-Augmented Language Model Pre-Training.  _arXiv e-prints_ , art. arXiv:2002.08909, February 2020.
  * Hahn (2019) Michael Hahn.  Theoretical Limitations of Self-Attention in Neural Sequence Models.  _arXiv e-prints_ , art. arXiv:1906.06755, June 2019.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring Massive Multitask Language Understanding.  _arXiv e-prints_ , art. arXiv:2009.03300, September 2020.
  * Izacard & Grave (2020) Gautier Izacard and Edouard Grave.  Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering.  _arXiv e-prints_ , art. arXiv:2007.01282, July 2020.
  * Johnson et al. (2017) Jeff Johnson, Matthijs Douze, and Hervé Jégou.  Billion-scale similarity search with GPUs.  _arXiv e-prints_ , art. arXiv:1702.08734, February 2017.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling Laws for Neural Language Models.  _arXiv e-prints_ , art. arXiv:2001.08361, Jan 2020.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.  Dense Passage Retrieval for Open-Domain Question Answering.  _arXiv e-prints_ , art. arXiv:2004.04906, April 2020.
  * Kassner & Schütze (2020) Nora Kassner and Hinrich Schütze.  BERT-kNN: Adding a kNN Search Component to Pretrained Language Models for Better QA.  _arXiv e-prints_ , art. arXiv:2005.00766, May 2020.
  * Kitaev et al. (2020) Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya.  Reformer: The Efficient Transformer.  _arXiv e-prints_ , art. arXiv:2001.04451, Jan 2020.
  * Lample et al. (2019) Guillaume Lample, Alexandre Sablayrolles, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou.  Large Memory Layers with Product Keys.  _arXiv e-prints_ , art. arXiv:1907.05242, July 2019.
  * Lepikhin et al. (2020) Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen.  GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding.  _arXiv e-prints_ , art. arXiv:2006.16668, June 2020.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.  _arXiv e-prints_ , art. arXiv:1910.13461, October 2019.
  * Lewis et al. (2020a) Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer.  Pre-training via Paraphrasing.  _arXiv e-prints_ , art. arXiv:2006.15020, June 2020a.
  * Lewis et al. (2020b) Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela.  Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.  _arXiv e-prints_ , art. arXiv:2005.11401, May 2020b.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _arXiv e-prints_ , art. arXiv:1907.11692, July 2019.
  * Lugosch et al. (2020) Loren Lugosch, Derek Nowrouzezahrai, and Brett H. Meyer.  Surprisal-Triggered Conditional Computation with Neural Networks.  _arXiv e-prints_ , art. arXiv:2006.01659, June 2020.
  * Mehta et al. (2020) Sachin Mehta, Marjan Ghazvininejad, Srinivasan Iyer, Luke Zettlemoyer, and Hannaneh Hajishirzi.  DeLighT: Very Deep and Light-weight Transformer.  _arXiv e-prints_ , art. arXiv:2008.00623, August 2020.
  * Pudipeddi et al. (2020) Bharadwaj Pudipeddi, Maral Mesmakhosroshahi, Jinwen Xi, and Sujeeth Bharadwaj.  Training Large Neural Networks with Constant Memory using a New Execution Algorithm.  _arXiv e-prints_ , art. arXiv:2002.05645, February 2020.
  * Rae et al. (2019) Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and Timothy P. Lillicrap.  Compressive Transformers for Long-Range Sequence Modelling.  _arXiv e-prints_ , art. arXiv:1911.05507, November 2019.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _arXiv e-prints_ , art. arXiv:1910.10683, Oct 2019.
  * Robertson & Zaragoza (2009) Stephen Robertson and Hugo Zaragoza.  The probabilistic relevance framework: Bm25 and beyond.  _Found. Trends Inf. Retr._ , 3(4):333–389, April 2009.  ISSN 1554-0669.  doi: 10.1561/1500000019.  URL https://doi.org/10.1561/1500000019.
  * Roy et al. (2020) Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier.  Efficient Content-Based Sparse Attention with Routing Transformers.  _arXiv e-prints_ , art. arXiv:2003.05997, March 2020.
  * Shazeer et al. (2017) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean.  Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.  _arXiv e-prints_ , art. arXiv:1701.06538, Jan 2017.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano.  Learning to summarize from human feedback.  _arXiv e-prints_ , art. arXiv:2009.01325, September 2020.
  * Sukhbaatar et al. (2019) Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin.  Adaptive Attention Span in Transformers.  _arXiv e-prints_ , art. arXiv:1905.07799, May 2019.
  * Tan et al. (2020) Bowen Tan, Zichao Yang, Maruan AI-Shedivat, Eric P. Xing, and Zhiting Hu.  Progressive Generation of Long Text.  _arXiv e-prints_ , art. arXiv:2006.15720, June 2020.
  * Tay et al. (2020) Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng.  Synthesizer: Rethinking Self-Attention in Transformer Models.  _arXiv e-prints_ , art. arXiv:2005.00743, May 2020.
  * Tran et al. (2020) Chau Tran, Yuqing Tang, Xian Li, and Jiatao Gu.  Cross-lingual Retrieval for Iterative Self-Supervised Training.  _arXiv e-prints_ , art. arXiv:2006.09526, June 2020.
  * Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin.  Attention Is All You Need.  _ArXiv e-prints_ , June 2017.
  * Wang & McAllester (2020) Hai Wang and David McAllester.  On-The-Fly Information Retrieval Augmentation for Language Models.  _arXiv e-prints_ , art. arXiv:2007.01528, July 2020.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le.  XLNet: Generalized Autoregressive Pretraining for Language Understanding.  _arXiv e-prints_ , art. arXiv:1906.08237, June 2019.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed.  Big Bird: Transformers for Longer Sequences.  _arXiv e-prints_ , art. arXiv:2007.14062, July 2020.

##  6 Appendix

###  6.1 Some Other Approaches

####  6.1.1 Learning to Summarize from Human Feedback (Stiennon et al., 2020)

This work achieves super-human-level summarization on TL;DR dataset by
training a reward function on human feedback and fine-tuning a pre-trained
causal model with PPO. This approach is very general for being applicable to
any rank-able tasks with human evaluation easily available. Since modeling
human-generated contents with MLE may be merely an imitation of human behavior
and may not allow super-human performance, it is possible that this approach
is more efficient for achieving super-human performance than MLE modeling.
However, due to its need for a large amount of supervision, we do not
investigate this approach further in our paper.

####  6.1.2 Progressive Generation of Long Text (Tan et al., 2020)

Progressive Generation of Long Text achieves substantially better text
generation (human-evaluated) quality and diversity than GPT-2 by pre-training
multiple Seq2Seq models, each of which ”super-resolutes” a text, ultimately
from a set of keywords into a complete text. However, since the evaluation is
based on fine-tuning to a domain-specific dataset, the same remark as non-
causal models apply. There needs more investigation into how it would cope
with general textual tasks.

####  6.1.3 DeLighT: Very Deep and Light-weight Transformer (Mehta et al.,
2020)

DeLighT, a Transformer language model, each of whose layer consists of a
smaller feedforward network, a single head self-attention and DExTra, a module
consisting of many layers of thin group linear layers, performs on par with
the baseline Transformer language model with twice more parameters. Since
group linear layer is slow without a custom CUDA kernel, the model would
become faster than the baseline for a given performance if there is a custom
CUDA kernel implemented for DExTra. However, since we often observe that a
deep thin model is slower than a shallow wide model with the same parameter
count using GPUs regardless of using a custom CUDA kernel, we are not certain
if this would improve the performance-computes trade-off. Furthermore, since
the source of improvement seems to stem from the use of DExTra, a block that
has no interaction between different timesteps, we can assume that this module
augments the feedforward network likely without any effect on the attention
layer. Hence, DExTra is in a direct competition with MoE. While it is likely
that they can be combined for further gain, the attraction of DExTra may
diminish if MoE is also taken into the consideration.

###  6.2 More Details of Sec. 2: Non-causal Models

####  6.2.1 Open-ended text generation with the output loosely constrained by
the input

Let us consider the task of open-ended text generation in the sense that the
input provided (e.g. prompt) is short and not very informative, while the
expected output is substantially longer and more informative, in a way similar
to how GPT-2 generates an article given an input. Since we expect a model to
have the same autonomy as humans, the model has to be able to generate without
detailed specifications from humans. This is in contrast to many of the tasks
non-causal models are evaluated on (BART as a notable exception), which are
often either discriminative tasks or generative tasks whose input sequence
contains substantial information about the output, such as summarization and
translation. In fact, Lewis et al. (2019) shows that BART and BERT variants
perform poorly on none but the causal generative modeling of ELI5 dataset
compared with the causal model, as they argue that the task on this dataset
belongs to the aforementioned family of tasks.

###  6.3 More Details of Sec. 3: Extension of Batch Length With Efficient
Attention

####  6.3.1 Improvement from extending the batch length would be limited at
some point by the trade-off between model size and batch size

According to Kaplan et al. (2020), the compute-optimal training strategy of
causal Transformer language model is to allocate most of the increase in
available computes ($C$) into parameter count ($N$) rather than batch size
($B$) or the number of iterations ($S$):

$N\propto C^{0.73},\quad B\propto C^{0.24},\quad S\propto C^{0.03}.$

Note that we have $B=bL$, where $b$ is the number of sequences per minibatch.
In practice, $b$ has to be sufficiently large; otherwise, it is often observed
that the resulting perplexity degrades due to lack of diversity in minibatch.
Since optimal size of $B$ for even the largest model explored in Kaplan et al.
(2020) is about several millions of tokens, it is reasonable to assume that
$L=\frac{B}{b}$ cannot be made arbitrarily large, possibly at most the order
of million. Thus, it is safe to argue that an improvement from extending the
batch length would be limited by the trade-off between model size and batch
size.

However, a caveat follows. The aforementioned compute-optimal strategy is
under the assumption that the batch length is fixed ($L=1024$) and that
WebText is used as the dataset. If $L$ can be enlarged and if a dataset with
longer average sample size is used, the scaling exponents for compute-optimal
training should change, though given the importance of batch size to the
performance, it is reasonable to assume that the change is small.

####  6.3.2 Why does extension of batch length with efficient attention not
improve the early token prediction?

To our assertion that extension of batch length does not improve the early
token prediction, there is a good counter-argument: the loss of early tokens
may be substantially improved if batch length is extended from $L\sim 1024$ to
$L^{\prime}\gg L$, so that numerous samples are covered and the model will
utilize inter-sample information a la implicit memory approach and retrieval-
based approach. As addressed before, it is very likely that $L^{\prime}$ is
bounded above (possibly at the order of million). Hence, if samples are
ordered randomly, the samples in the same context are far from the $k$NNs from
each other. Therefore, the performance gain from inter-sample information is
likely to be negligible compared with the gain with retrieval-based approach,
which can find the $k$-nearest neighbors.

####  6.3.3 Combination of retrieval-based approach with the approach of
extending batch length with efficient attention

It is possible to combine efficient attention with retrieval-based approach in
order to increase $k$ of $k$NN samples to be used for prediction with a
retriever, since a larger $k$ notably results in a better performance to some
extent according to Izacard & Grave (2020). However, there are two caveats.

Firstly, the existing efficient attention may be already efficient enough that
any further improvement of the method may not be beneficial for this use, as
the empirical complexity of efficient attention would not be improved further
according to Sec. 3.

Secondly, the design of such efficient attention may depend on how the
retrieval-based model is designed. For example, MARGE embeds each sample, so
that the samples with the closest embeddings can attend to each other through
attention. In this case, instead of applying a method as Routing Transformer,
one can simply approximate the full attention with top-$k^{\prime}$
approximation at sample-level for some reasonable $k^{\prime}$, which does not
require any sophisticated efficient attention method.

####  6.3.4 How to assess if a given method improves early token prediction

In this paper, we define ”the improvement of early token prediction” to be the
improvement of the loss of the first $L$ tokens of each sample for a small
enough $L$. We set $L\sim 1024$, since the average length of a sample in the
currently used gigantic text datasets is about $1000$ (for being a subset of
Common Crawl) and since setting $L\sim d_{model}$ is a reasonable choice for
making the full-attention computes comparable to that of the linear layer (as
$d_{model}\sim 1024$).

The most straightforward method to verify if a given model improves early
token prediction or not is to compare the $n$-th per-token loss of early
tokens for various $n$, which we have performed on various models, which led
to our conclusion. However, we can also demonstrate to some extent that a
given method improves the early token prediction without comparing the per-
token loss for some cases. Models without recurrence tend to divide each
sample into segments, each of which has length $L$, and they tend to be
evaluated on a segment without being conditioned on the past tokens outside of
the segment within the same sample. In this case, each segment is essentially
treated as a sample for the lack of dependence on each other. Hence, the usual
loss of the test dataset is equal to the loss of the first $L$ tokens of each
sample. In other words, for a model without recurrence, if the segment length
of the model is equal to that of its baseline, it suffices to compare the loss
of the model with that of the baseline to see if the model improves the early
token prediction, since ”early token” cont.

For example, knn-LM and its baseline are the same vanilla Transformer, except
that the former is also conditioned on the distribution of the relevant tokens
from the training dataset. The computes spent for the former is only
marginally larger than that of the latter, while the improvement in the loss
is substantial; hence, knn-LM improves the early token prediction for the
given computes. A similar conclusion can be made for the GPT-2 with a
retriever as in Wang & McAllester (2020), which is an augmented GPT-2 fine-
tuned to predict conditioned on the sentences fed by a retriever. Since the
fine-tuning computes are marginal relative to that of the pre-training and
since the improvement in performance is large enough, we can argue that they
too improve early token prediction for the given computes. As discussed later,
many implicit memory models are the same as its baseline except for being an
efficient approximation to a Transformer with a larger model size. Hence, this
implies that such implicit memory models that improve the performance-computes
trade-off, including MoE and PKM, improve the early token prediction for the
given computes.

###  6.4 More Details of Sec. 4: Memory-Based Approach

####  6.4.1 The tasks recurrence excels at and self-attention struggles with

Self-attention is known to perform poorly on some tasks recurrence excel at.
For example, Hahn (2019) theoretically shows the former struggles with
predicting the parity of a given long bitstring the latter excels at. However,
TBPTT-segment-wise recurrence as explicit memory approach with self-attention
usually needs to have long enough batch length (e.g. $128$ tokens) to make its
performance-computes trade-off reasonably high on various tasks of practical
interest. With this batch length, the local behavior of explicit memory model
should be identical to that of self-attention. Hence, it should struggle just
as self-attention. Furthermore, the tasks self-attention struggles with and
recurrence excels at tend to be difficult even for humans. For example, if
humans, without provided any instruction, are forced to find a pattern in an
extremely long bitstring to figure out that the problem asks for the parity,
they too would struggle with the task.

####  6.4.2 More details about MoE and PKM

Let us consider a feedforward layer with a very large $d_{ff}$. MoE can
imitate this by dividing this layer into many feedforward layers with smaller
$d_{ff}$, each of which is an expert of MoE, and the gating of MoE controls
which expert to be activated for a given input vector. On the other hand, PKM
can imitate a large feedforward network by exploiting the following structural
sparsity of feedforward layer we have observed: masking every units but the
top-$k$ units of $d_{ff}$ units by zero does not affect the training of the
model substantially even if $k\ll d_{model}$ (we observed that setting
$k=\frac{d_{ff}}{10}$ does not affect the perplexity at all in some cases).
Hence, PKM tries to predict which units are the top-$k$ units for a given
input vector by its product key mechanism and efficiently computes the output
by fused gather-matmul, which in practice results in slower processing than
MoE due to memory bottleneck.

Scaling up the model size by evenly increasing $d_{ff}$, $d_{model}$, $h$ (the
number of heads) and depth, without increasing $d_{k}$ (the dimension of each
head) tends to beat other approaches. This is presumably why GPT-2 and GPT-3
were scaled up in a similar manner. Let us call this optimal model scaling.
Hence, increasing $d_{ff}$ only for scaling is suboptimal, and it is likely
that the scaling law using this scaling with respect to the parameter count
may not hold as long as that of the optimal scaling, not to mention that the
power law exponent is likely worse. However, the gain from MoE and PKM is
still substantial.

####  6.4.3 Limitations of conditional computation

More successful conditional computation methods, such as MoE, are mostly an
efficient approximation to a Transformer architecture with larger model
hyperparameters, such as $d_{ff}$. Hence, its performance is, at best, equal
to that of the larger Transformer. However, scaling up model hyperparameters
does not seem to solely resolve all the problems. Fig. 5 shows that T5
performs significantly worse than some open-domain QA models despite
substantially larger computes spent for the training. Likewise, Lewis et al.
(2020a) shows that MARGE outperforms various baselines that consume
substantially more computes and use more task-specific human intervention.
Thus, we may not only need conditional computation but also retrieval-based
approaches in order to improve the performance-computes trade-off further.

(a)  Figure 5: (Izacard & Grave, 2020). Comparison of the state-of-the-art QA
models.


# Learning to Automate Follow-up Question Generation using Process Knowledge
for Depression Triage on Reddit Posts

Shrey Gupta1 , Anmol Agarwal1††footnotemark:  , Manas Gaur2,
Kaushik Roy2, Vignesh Narayanan2, Ponnurangam Kumaraguru1, Amit Sheth2
1International Institute of Information Technology, Hyderabad, India
{shrey.gupta<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
2AI Institute, University of South Carolina, SC, USA
{mgaur<EMAIL_ADDRESS>{vignar<EMAIL_ADDRESS>
Authors contributed equally

###### Abstract

Conversational Agents (CAs) powered with deep language models (DLMs) have
shown tremendous promise in the domain of mental health. Prominently, the CAs
have been used to provide informational or therapeutic services (e.g.,
cognitive behavioral therapy) to patients. However, the utility of CAs to
assist in mental health triaging has not been explored in the existing work as
it requires a controlled generation of follow-up questions (FQs), which are
often initiated and guided by the mental health professionals (MHPs) in
clinical settings. In the context of ‘depression’, our experiments show that
DLMs coupled with process knowledge in a mental health questionnaire generate
12.54% and 9.37% better FQs based on similarity and longest common subsequence
matches to questions in the PHQ-9 dataset respectively, when compared with
DLMs without process knowledge support. Despite coupling with process
knowledge, we find that DLMs are still prone to hallucination, i.e.,
generating redundant, irrelevant, and unsafe FQs. We demonstrate the challenge
of using existing datasets to train a DLM for generating FQs that adhere to
clinical process knowledge. To address this limitation, we prepared an
extended PHQ-9 based dataset, PRIMATE, in collaboration with MHPs. PRIMATE
contains annotations regarding whether a particular question in the PHQ-9
dataset has already been answered in the user’s initial description of the
mental health condition. We used PRIMATE to train a DLM in a supervised
setting to identify which of the PHQ-9 questions can be answered directly from
the user’s post and which ones would require more information from the user.
Using performance analysis based on MCC scores, we show that PRIMATE is
appropriate for identifying questions in PHQ-9 that could guide generative
DLMs towards controlled FQ generation (with minimal hallucination) suitable
for aiding triaging. The dataset created as a part of this research can be
obtained from here.

##  1 Introduction

Conversational agents (CAs) powered by DLMs are software designed to interact
with human users for specific tasks. For mental health purposes, particularly
depression, CAs have been studied extensively in prior work for helping
patients follow generic mental health guidelines, typically by providing
reminders to assist patients in adhering to the medication and therapy
strategy outlined by a mental health professional
(MHP)111https://tinyurl.com/yfp3bhr2222https://woebothealth.com/. However,
previous work on depression have not examined the use of CAs for triaging. For
the purpose of triaging, CAs should learn to generate controlled and clinical
process knowledge-guided discourse that can assist MHPs in diagnosis. Our
research suggests a clinically grounded and explainable methodology to develop
conversational information-seeking tools, first to learn “what symptoms the
user is suffering” and “what extra information is needed for triaging.”

CAs are susceptible to irrelevant and sometimes harmful questions when
generating FQs or responses to a patient suffering from depression Miner et
al. (2016). The primary reason for irrelevant and harmful questions is that
CAs cannot incorporate contextual information in generating appropriate
follow-up questions (FQs) (see Figure 1). Further, the sensitivity of the
conversation and a controlled generation process are essential characteristics
of patient-clinician interactions, which are difficult to embed in DLM-based
CAs. Therefore, question generation (QG) in mental health is challenging, and
research to develop CAs for automating triage has not been explored.

Figure 1:  Reddit is a rich source for bringing crowd perspective in training
DLMs over conversational data. On the left is a sample post from
r/depression_help which sees inquisitive interaction from other Reddit users.
At the top-right are the FQs asked by the Reddit users in the comments. These
FQs are aimed at understanding the severity of the mental health situation of
the user and are hence, diagnostically relevant. At the bottom-right are the
questions generated by DLMs. It can be seen that these are not suitable FQs.

Procedures for generating semantically related and logically ordered questions
in the mental health domain are a form of process knowledge manifested in
various clinical instruments for mental health triage. For example, the
severity of depression is measured using Patient Health Questionnaire (PHQ-9).
Enforcing DLMs to follow process knowledge, like in PHQ-9, would make CAs
generate FQs similar to an MHP when they are seeking information from the
patient Karasz et al. (2012). Unfortunately, datasets that meet this criterion
are currently unavailable. Though clinical diagnostic interviews exist, they
are not rich, sufficiently dense, and varied to train DLMs Manas et al.
(2021); Gratch et al. (2014). Further, we require dataset(s) that includes
support seeking queries and natural questions that show help providing
behavior. For this purpose, anonymized user-generated conversational data in
Mental Health support communities on Reddit provides a rich source of fine-
grained, contextual, and diverse information suitable for fine-tuning DLMs.
Specific to depression, we explored posts and comments in r/depression_help.

In the current research, we emphasize the limitations of T5, a state-of-the-
art DLM333Current DLMs are either variants of T5 or built from T5 to generate
process knowledge-like FQs using the data from r/depression_help Raffel et al.
(2019). We filtered the dataset by retaining only posts with at least one
comment that seeks additional information from the user seeking support.
Further filtering of comments was performed using PHQ-9 to assist T5 in
generating relevant FQs (see Figure 2). We found that the outcome is
substantial for the single turn question answering model; however, not
suitable for mental health triage, which is a discourse. We conducted a series
of experiments keeping our focus on ’depression’ and leveraged its associated
process knowledge for mental health triage: the PHQ-9 Kroenke et al. (2001).
To the best of our knowledge, FQ generation relating to depression has never
been studied using PHQ-9 for discourse modeling and generation.

We make the following key contributions: (a) Extending PHQ-9: PHQ-9 questions
are limited in scope for common NLP tasks like finetuning. In collaboration
with MHPs, we prepared a list of 134 sub-questions for nine PHQ-9 questions
for better fine-tuning of T5. (b) We analyzed the performance of three
variants of T5 using BLEURT Sellam et al. (2020) and ROUGE-L scores that
measure semantic relatedness and exact match similarity of generated question
to sub-questions of PHQ-9. (c) PRIMATE Dataset: Lessons learned during our
experiments suggested that T5 must be trained in a supervised setting to
capture ‘what the user has already mentioned about his/her depression
condition in the post-text’ and then generate FQs. Along with MHPs, we
constructed a novel PRIMATE (PRocess knowledge Integrated Mental heAlth
daTasEt) dataset that would train DLMs to capture PHQ-9-answerable information
from user text. In this research, we restrict our experiments and discussion
on whether PRIMATE can help capture context from the user post relevant to
some PHQ-9 questions and pointing out which other PHQ-9 questions would form
candidates to direct FQ generation. Our approach and insights have
applications to Anxiety (GAD-7), Suicide (C-SSRS), and other mental health
disorders as well.

Figure 2: An illustration of our pipeline for developing Model 2 and Model 3
using T5 as the deep language models. Starting with posts (including comments)
from r/depression_help, we filter out comments that are neither interrogative
nor information seeking in nature to yield a posts-questions dataset for fine-
tuning T5. This dataset was further filtered using extended PHQ-9 before using
it to fine-tune T5 (Model 3).

##  2 Related Work

Recently, DLMs have attracted much attention for question answering, thanks to
their successes in NLP applications Thoppilan et al. (2022); Borgeaud et al.
(2021). Research on question generation has focused on improving the
legibility and relevance of questions. This is because DLMs continue to
hallucinate while generating questions in general-purpose domains, which can
lead to factually incorrect responses. This can have severe consequences in
the mental health domain Thoppilan et al. (2022). Recently, inappropriate and
toxic behaviors of language models have been extensively studied and reported
in the literature Dinan et al. (2021); Weidinger et al. (2021). Solutions
around fine-tuning, augmenting a neural retriever to support generation, and
rules on generation quality have been defined as possible remedies Manas et
al. (2021). These have been effective for the general-purpose domain; however,
the research surrounding DLMs is yet to unfold in mental health. ELIZA
Weizenbaum (1983) could transform users’ statements into questions but employs
labor-intensive templates to generate safe and relevant questions. Models like
RAG and REALM were developed to include external knowledge to support question
generation Lewis et al. (2020); Guu et al. (2020). However, these models are
still susceptible to incoherent and irrelevant FQ generation . Further, their
end-to-end learning approach is rigid to support process-guided question
generation and discourse, often followed in a clinical setting for triage Gaur
et al. (2021).

In theory, DLMs should be capable enough of extracting pieces of information
from user description that portrays the understanding of the user and leverage
it for generating the next FQ. For such a task, supervised training of DLMs
with process knowledge and coupling it with information retrieval over domain-
specific mental health knowledge is a viable solution. This is because mental
health knowledge sources (e.g., SCID (Structured Clinical Interviews for
DSM-5) have structured/semi-structured information on how interviews are
performed Brodey et al. (2018). Our research substantiates that DLMs (e.g.,
T5) generate low quality follow-up questions in the context of depression for
triage, and granting external knowledge through PHQ-9 reduces the rate at
which models generate meaningless FQs Thoppilan et al. (2022); Komeili et al.
(2021). In the current research, we define an approach for supervised training
of DLMs on a specific dataset that would yield probability distribution over
PHQ-9 (with support from Extended PHQ-9). These probabilities will confirm
whether the DLM can identify cues from user text that can inform a set of
PHQ-9 questions. Remaining PHQ-9 questions are potential FQs.

### Datasets:

Prior datasets such as Counsel Chat CounselChat , Counseling Conversations
Huang (2015), Role Play Demasi et al. (2019), Crisis Text Line Althoff et al.
(2016) and Reddit C-SSRS Gaur et al. (2019) have been created to train CA for
mental health counseling. Trained CAs can engage in a single turn question
answering; however, conducting a conversation requires capturing user context
and leveraging clinical instruments to guide the generation of FQs.

Model 1 QGs |  Model 2 QGs |  Model 3 QGs
---|---|---
• How long ago did the worsening chronic fatigue continue to cripple my life?
[*User already mentions in the title of the post he has been facing the
fatigue for years]
• What was the estimated dose of suicide? [*Unsafe incoherent question]
• How could body parts feel far less hot in other parts of my life?
[*Incoherent question]
• What has caused the gradual cognitive decline? [*It’s the physician who
needs to answer this question after the conversation]
• What effect is that I’m less optimistic, and it’s embarrassing ?
[*Incoherent attempt to generate a question from the sentence in the post
which mentions “I am still pretty optimistic , but I ’m just too out of it ,
it ’s embarrassing”]
• What was the reason that increasing depression has a negative effect on me?
[*Irrelevant and incoherent]
• A sleep study consists of what? [*Irrelevant to the diagnostic procedure]
• What did Caffeine feel in my body? [*Wrong pronoun used and irrelevant to the diagnosis] |  • Have you ever tried many depressants or mental health things? [*Redundant question as user has already mentioned in the post text that he has tried antidepressants]
• Are you seeing a therapist? [*Redundant question as user had already
mentioned in the post text that he has seen a therapist in the past]
• Do you even ever try to focus on negative things and/or pain, relief, and
patience? [*Incoherent question]
• And what if you forgot to fucking realize that, then it will be fucking so
much that you’ll be fucking fucking can we ask what they want. [*Unsafe
objectionable question]
• Do you mean heavy bipolar disorder? [*Irrelevant to the context of the post]
• Are you aware about the qualities of antidepressants? [*Inquisitive in
nature but irrelevant from a diagnostic perspective]
• What does antidepressant work at? [*Irrelevant from a diagnostic
perspective]
• Do you have a close friend you can go to who you can talk with, that way you can get out of the house? [*FQ] |  Are you on any antidepressants? [*User mentions in the post he has tried them before but no mention has been made if he is still on them]
• Do you have any positive thoughts? [*FQ]
• Are you in any danger of hurting yourself? [*Slightly unsafe]
• Is it that you aren’t happy with your feelings? [*Irrelevant question]
• Have you tried some exercise? [*Redundant question as user already mentions
he has tried it]
• Do you wake up frequently? [*FQ]
• How is your sleep quality? [*FQ]
• When you wake up, what do you do? [*FQ]
• Is there anything that helps you calm the symptoms for now? [*FQ]
• What are your hobbies? [*Generic FQ]
• What are your interests? [*Generic FQ]
Table 1: Examples of questions generated by T5 when tasked to generate FQs when the user query for the post in Figure 1 was provided as input. Model 1, which is a pre-trained T5 Raffel et al. (2019), often generates questions which are irrelevant, unsafe, incoherent, and redundant. Model 2, which is T5 fine-tuned on r/depression_help seems to be relatively coherent and inquisitive compared to Model 1. However, both models generate questions about the topic that user has discussed in their query. As a result, we see that pre-trained and fine-tuned DLMs fail to generate FQs. By enforcing FQ generation using using a dataset curated using extended PHQ-9, generated questions have been mostly inquisitive. This is shown by Model 3. Still, a lot of generations are around the problem the user mentioned. Post text |  Model 1 QGs |  Model 2 QGs |  Model 3 QGs
---|---|---|---
I feel depressed . I have felt depressed for a while. I have done a lot of things to try and help myself through therapy ,[…] , staying busy , going to university . all I have found is that each time I complete any of these things that I still feel like I am empty . When I do not do those things for more than 10 mins by myself that my thoughts just get dark and I be little everything I have ever done . [….] Also I get depressed because I have dreams where I ’m constantly bombarded by women who are not my wife who want to have relationships with me and it ’s a nightly battle to not give in to it when I ’m dreaming . I know it sounds silly but I fear that if I give into the dreams that I might do it in real life . I do n’t want to tell my wife because she does n’t understand other weird dreams that I have had in the past . |  • What if I do not do it for more than 10 mins by myself? [*Closed book question already answered in the post and irrelevant to the diagnostic procedure]
• What might happen to any women that want to have relationships with me?
[*Irrelevant to the diagnostic process]
• What do the women at work do? [*Irrelevant to the diagnostic process)]
• What kind of dreams are I was depressed? [*Closed book question generated directly from a statement in the post] |  Would you be happy with them? [*Irrelevant to the diagnostic process)]
• What’s your job? [*Irrelevant]
• Are you able to reach out to your doctor or psychiatrist? [*User has already
mentioned in the original post that he has talked to his therapist]
• what are your dreams? [*Closed book question already answered in the post]
• What do you enjoy about your dreams? [*FQ]
• Do you want to talk to people about what dreams are ? [*FQ]
• Do you feel as satisfied as yourself when you are dreams? [*FQ] |  Have you talked to a therapist? [*Already answered by the user in the post itself]
• Is there anything that helps you calm your feeling for now? [*FQ]
• “Have you ever gotten any help?” [*Already answered in the query]
• Do you feel isolated? [*FQ]
• What are your hobbies? [*FQ]
• What are your interests? [*FQ]
• How long have you been waiting for your wife to talk about these dreams?
[*FQ]
• Have you told your wife you’re depressed or not? [*Inquisitive in nature but
already answered by the user in original post]
Table 2: In this example, the generated questions from both Model 2 and Model
3 seem to be relevant FQs, but they are not assessing the severity of the
mental health condition, despite Model 3 being fine-tuned on a dataset
filtered by PHQ-9 questions. In comparison to the qualitative outcome in Table
1, this showcases the inability of T5 to support mental health triage.

##  3 Question Generation (QG)

### Dataset for QG:

Our approach to data collection involves scraping posts and comments from
r/depression_help, a subreddit on Reddit, which is meant to provide advice and
support to help individuals suffering from depression. The posts on this
subreddit contain flair tags such as SEEKING HELP, SEEKING ADVICE, and
REQUESTING SUPPORT. We filter down the data curated from this subreddit based
on the flair tag attribute to retain only advice, help or support seeking
posts and their comments. After filtering, our dataset had approximately
21,000 posts. Each post contains a title, description, and comments. On
average, each post has 5 comments. Next, we chunked the main text of each post
into smaller groups of sentences (chunks) of less than 512 tokens while making
sure no sentence is segmented in between. The motivation for chunking is to
ensure no context is lost from the post due to the limitation of T5 to process
512 tokens as input (DLMs in general suffer from such representation limits).
We also appended the post title to each chunk to ensure that main idea of each
post was captured in it’s chunks. This curated dataset tests T5’s capability
to generate FQs similar to any of the questions in the extended PHQ-9
questionnaire.

### Extending PHQ-9 to support FQ generation:

PHQ-9 questions are subject to different interpretations depending on patient-
MHP interaction. Additionally, nine questions are limited in scope for use in
tasks like fine-tuning and similarity-based performance evaluations.
Therefore, to increase the strength of PHQ-9, we collaborated with MHPs to
create sub-questions for each question in PHQ-9. First, we used Google SERP
API444https://serpapi.com/ and Microsoft Bing Search
API555https://www.microsoft.com/en-us/bing/apis/bing-web-search-api to
retrieve “People-Also-Ask” questions. For each question, we retrieved 40
questions by manually searching and assessing their relevance to PHQ-9
questions. Next, we provided the set of 360 questions to three MHPs for
assessment. MHPs evaluated the questions on two grounds:(a) Whether they would
ask such a question to a patient? (relevance) (b) If yes, when should such a
question be asked? (rank). Based on their ratings, we created a final set of
134 sub-questions for the nine questions in PHQ-9666Questions in extended
PHQ-9 : link resulting in a total of 143 questions.

### Models for FQ Generation:

We used an off-the-bench T5-base QG model that was fine-tuned on the SQuAD 2.0
question generation dataset Rajpurkar et al. (2018) [Model 1]. Next, we fine-
tuned Model 1 on r/depression_help posts and comments. To align with our task
of making T5 generate relevant FQs, we filtered out comments which were non-
interrogative. We kept only the interrogative statements asked by Reddit users
in the comments [Model 2]. Not all interrogative comments by Reddit users are
diagnostically relevant FQs (Eg: “Can you use MS Excel?”, “Were you
interactions on FaceTime?”). To remove such questions, we further filtered the
dataset by calculating the maximum BLEURT score between the question (present
in the comments) and the questions in extended PHQ-9. We applied a threshold
of $0.60$ to this score777empirically judged. This removed harmful and
diagnostically irrelevant questions while preserving contextual, semantically
relevant, and legible questions [Model 3]. See Fig 1 for examples of
diagnostically relevant questions.

$|\hat{Q}|$($\downarrow$) | Hit Rate on BLEURT |  | Hit Rate on Rouge-L
---|---|---|---
$\delta$($\rightarrow$) | 0.4 | 0.5 | 0.7 |  | 0.4 | 0.5 | 0.7
Model 1: Pre-trained T5
5 | 0.5417 | 0.1233 | 0.0020 |  | 0.1241 | 0.0386 | 0.0005
10 | 0.5400 | 0.1203 | 0.0010 |  | 0.1290 | 0.0400 | 0.0010
15 | 0.5368 | 0.1250 | 0.0013 |  | 0.1266 | 0.0384 | 0.0009
Model 2: Fine-Tuned T5 on r/depression_help
5 | 0.6657 | 0.2804 | 0.0097 |  | 0.3445 | 0.1560 | 0.0100
10 | 0.6691 | 0.2792 | 0.0104 |  | 0.3481 | 0.1590 | 0.0098
15 | 0.6726 | 0.2787 | 0.0104 |  | 0.3476 | 0.1588 | 0.0094
Model 3: T5 Fine-tuned on r/depression_help filtered by PHQ-9
5 | 0.9489 | 0.7088 | 0.1261 |  | 0.7457 | 0.4937 | 0.0903
10 | 0.9542 | 0.7126 | 0.1272 |  | 0.7460 | 0.5002 | 0.0947
15 | 0.9514 | 0.7098 | 0.1274 |  | 0.7484 | 0.4945 | 0.0916
Table 3: Experimental results comparing different models in generating
questions that match the sub-questions in PHQ-9. $\hat{Q}$ is the set of
generated questions in each chunk. The performance is recorded over all the
generated questions ($\mathbf{\hat{Q}}$). $\delta$ was used as the threshold
on the similarity between generated question and PHQ-9 sub-questions while
calculating hit rate. BLEURT records semantic similarity, whereas Rouge-L
records the longest common subsequence exact match between generated question
and PHQ-9 sub-questions. The highest performance on semantic and string
similarity is bolded. Acceptable performance in Model 3 achieved using PHQ-9
motivated us to prepare PRIMATE. Figure 3: A post in PRIMATE which is
annotated with PHQ-9.The questions marked “YES” are answerable by DLMs using
the mental health specific cues from user text. The questions marked “NO” are
the questions a DLM should consider asking as FQs. Sentences within [] were
taken as signals that the “YES” marked questions had already been answered in
the post .

### Analysis of Models for Question Generation:

Out of the 21k questions, performance of Models 1, 2, and 3 were examined on
those 2003 posts that had at least one interrogative comment. Each of the
three models was made to generate FQs in sets of 5, 10, and 15 through nucleus
sampling Holtzman et al. (2019). For a generated question, BLEURT score was
computed with each question in Extended PHQ-9 and the maximum among those
scores was taken as the score for the generated question. A clear distinction
between models 1, 2, and 3 is the nature of the questions asked. Model 1
generated closed book questions, whereas Model 2 and 3 seem to show some
inquisitive nature and seem more focused on the mental health domain, which
can be attributed to the after effect of finetuning on Reddit (see Table 1 and
2). We captured the performance of the models quantitatively using ’hit rate’
as a metric. For a generated question ($\hat{q}$), we denote :

$\displaystyle score(\hat{q})=max(bleurt\\_score(\hat{q},q_{1}),$
$\displaystyle
bleurt\\_score(\hat{q},q_{2}),...,bleurt\\_score(\hat{q},q_{143})),$

where $q_{1},q_{2}\text{...}q_{143}\in\text{Extended-PHQ-9}$. Across all 2003
posts, we had $C$ = 2575 chunks888Chunking was done as DLM accepts a maximum
input length of 512 tokens.. Let total number of questions generated by a
model be $|\mathbf{\hat{Q}}|$ and $|\hat{Q}|$ denote the number of question
generated by the model for a given chunk. For experimentation, we set
$|\hat{Q}|$ to have values {5, 10, 15}. Thus,
$|\mathbf{\hat{Q}}|=|\hat{Q}|*C$. Then the Hit Rate for a model was computed
as:

$\text{Hit
Rate(model},|\hat{Q}|)=\frac{\sum\limits_{\hat{q}~{}\epsilon~{}\mathbf{\hat{Q}}}\mathbf{I}(score(\hat{q})>\delta)}{|\mathbf{\hat{Q}}|},$

where $\delta$ is the threshold on the similarity between generated question
in a chunk and sub-questions in PHQ-9 and $I[\varphi]$ is the indicator
function taking values 0 or 1 for a predicate $\varphi$ (Table 3 has the
scores).

### Inference:

(1) Regardless of fine-tuning and filtering based on PHQ-9 questions,
inherently, T5 does not capture the meaning and usage of the words in the
mental health context. Moreover, T5 fails to generate legible and relevant FQs
as safe as PHQ-9 questions. Therefore, we scrutinize the generated FQs by
mapping them to most similar questions in extended PHQ-9. Examples of
irrelevant generations by T5 that it thought were relevant are: (a) “Wtf?”
(generated FQ) was found most similar to “Do you have hope?” (PHQ-9) (b) “What
did Boyfriend suffocate me with during his break up a week after I got a
diagnosis?” (generated FQ) was found most similar to “What do you think makes
you a failure” (PHQ-9). The previous generated question is redundant as the
answer to it was already present in the original post. (2) Many generated
questions contain extreme language due to the informal nature of the Reddit
platform, which is very sensitive issue, especially in the mental health
domain. Examples are: “Did you f***ing realize that f***ing people are f***ing
too?” (generated FQ) was found to be the most similar to “What do you think
makes you a failure?”. Thus, T5 and its variants need to capture “what the
user knows and has already mentioned in his post” by checking which PHQ-9
questions are already answerable using the user’s post before generating the
next probable FQs in order to avoid redundancy.

Figure 4: 1\. Answerability evaluator: A BERT model is trained in a supervised
setting to be an evaluator of whether a PHQ-9 question can be answered in a
given user post (binary) using PRIMATE. For nine PHQ-9 questions, we require
nine such evaluators. 2. Follow up questions: PHQ-9 questions that are not
already answerable using the user post form candidates for follow up. 3. SCID:
Corresponding to each PHQ-9 question, the SCID describes a clinician approved
sub-sequence of questions to obtain the answer to the follow up question. 4.
Use existing PHQ-9 and DSM-5 lexicons Yazdavar et al. (2017) to filter the
question to be generated. 5. Generate FQs using T5 fine-tuned on external
domain-specific knowledge and the large-scale depression support conversation
dataset created from Reddit and PRIMATE.

##  4 PRIMATE for FQ Generation

We conceptualize our approach on the duality of data and the process knowledge
contained in PHQ-9 (see Figure 4). First, a BERT Answerability Evaluator
identifies which questions in PHQ-9 are already answerable (using the user’s
initial description of his/her condition in the post) and which ones need more
information to be answerable. The latter type of questions form candidates for
training a generative DLM for FQ generation. We present PRIMATE, a dataset
consisting of Reddit posts containing user situations describing their health
conditions and whether the questions in PHQ-9 are answerable using the content
in the posts. Each question is attributed with a binary “yes” or “no” label
stating whether the user’s description already contains the answer to that
question (see Table 4). PRIMATE was created from a month long annotation-
evaluation cycle between MHPs and crowd workers. A total of five crowd workers
performed this task, achieving an initial annotator agreement of 67% using
Fleiss kappa. Subsequently, the MHPs assessed the quality of annotations and
provided their suggestion for improvement, leading to an acceptable agreement
score of 85%. A sample annotated post in PRIMATE is shown in Figure 3.

PHQ-9 | Number of Posts
---|---
Questions |  With Answer (Yes) |  W/o Answer (No)
Q1 |  1679 |  324
Q2 |  1664 |  339
Q3 |  686 |  1317
Q4 |  949 |  1054
Q5 |  530 |  1473
Q6 |  195 |  1808
Q7 |  741 |  1262
Q8 |  196 |  1807
Q9 |  374 |  1629
Table 4: Distribution of 2003 posts in PRIMATE according to whether the text
in the post answers a particular PHQ-9 question. Through this imbalance,
PRIMATE presents its importance in training DLM(s) to identify potential FQs
in PHQ-9 that would guide a generative DLM(s) to conduct a discourse with a
patient with a vision to assist MHPs in triage. Q1-Q9 are described in Figure
3

### BERT as Answerability Evaluator:

While Model 3 shows respectable performance (Table 3), even the FQs generated
by Model 3 may not yield the most efficient capture of the PHQ-9 related
questions (evident from the low hit rate at a higher threshold) ($\delta$).
The MHPs would probably have a more streamlined, focused questioning strategy.
For efficient MHPs and AI collaboration, we propose to guide the questioning
in a more systematic way by predicting if the user post already has answers to
the PHQ-9 questions. This is first posed as a binary classification problem
over nine PHQ-9 questions. Thereafter, the approach is to generate questions
similar to the PHQ-9 questions that do not have answers in the post. Thus, we
train BERT999BERT end-to-end training perform well compared to baselines
ElectraClark et al. (2019), and MedBERTGu et al. (2021) (a transformer-based
DLM) as a classifier on the PRIMATE dataset. We plan to further use the
classification outcome from the BERT model to drive the direction of further
questioning with the patient in a more controlled manner. This process can
lead to high efficiency and completion of the mental health triaging in as few
questions as possible.

$\delta$ ($\rightarrow$) | 0.5 | 0.7 | 0.9 | Class-
---|---|---|---|---
PHQ-9($\downarrow$) | MCC | MCC | MCC | Type
Q1 | 0.0 | 0.17 | 0.17 | W
Q2 | 0.43 | 0.45 | 0.52 | S
Q3 | 0.41 | 0.46 | 0.33 | M
Q4 | 0.14 | 0.19 | 0.13 | W
Q5 | 0.63 | 0.65 | 0.66 | S
Q6 | 0.47 | 0.43 | 0.27 | W
Q7 | 0.66 | 0.68 | 0.7 | S
Q8 | 0.1 | 0.0 | 0.0 | W
Q9 | 0.62 | 0.56 | 0.39 | M
Table 5: We record the Matthews Correlation Coefficient (MCC) to measure the
performance of the Evaluator (see Figure 4). The MCC score for all 9 questions
across different thresholds is in the range 0 to +1 (low to high positive
relationships). The MCC for some configurations runs into a divide by zero
error, and we replace this value with 0.0. W: model is unable to learn cues to
determine answerability in a post. M: model is uncertain whether a particular
PHQ-9 question is answerable or not. S: answerability can be determined by the
model with high reliability. Class-Type: Classification Type when
$\delta=\>0.9$

### Performance Analysis:

We report the Matthews Correlation Coefficient (MCC) scores in table 5. MCC is
a reliable metric to assess a model’s classification over an imbalanced
dataset, particularly useful when we are interested in all four categories of
confusion matrix: true positives (answerable questions (AQ)), true negatives
(FQ candidates), and false alarms (false negatives and positives). As PRIMATE
shows a disproportional distribution of AQs (yes) and FQs (no), MCC is an
appropriate metric Chicco and Jurman (2020). We base our analysis on the
consistency of BERT classifier on varying threshold ($\delta$) in table 5. A
score between 0.0 to 0.30 (Type W: Weak) on MCC means the model is only able
to find a negligible to weak positive relationship between input and output.
In our context, a score in this range for a particular PHQ-9 question means
that model is unable to effectively learn the cues needed to judge the
answerability of that question in user posts. A score between 0.30 and 0.40
(Type M: Maybe) means that the model is able to learn a moderately positive
relationship, interpreted as ambiguity in the model to judge whether a
particular PHQ-9 question is answerable from user posts. MCC scores between
0.40 to 0.70 (Type S: Strong) for a question in PHQ-9 means that the model can
effectively judge whether that question is answerable in user posts . Any
score above 0.70 makes the model’s judgements even more reliable. This
experiment completes steps 1 and 2 in Figure 4. Steps 3, 4 and 5 are concerned
with the task of FQ generation by fine-tuning the T5 DLM as a generator over
r/depression_help and other depression support communities on Reddit. The FQ
generations will be controlled using the process knowledge in SCID which is
consulted for interviewing by MHPs. Further, PHQ-9 lexicons are leveraged for
promoting diversity and filtering irrelevant FQ generations. We leave this
process of FQ generations to shape discourse as future work.

##  5 Conclusion

This paper demonstrated the importance of data and process knowledge to adapt
DLMs for generating FQs that would assist MHPs in triaging depression. Our
experiments show that without process knowledge, DLMs hallucinate by
generating unsafe, incoherent, and irrelevant questions that are not helpful
for MHPs in pre-screening or triaging. The challenge lies in the inability of
the DLMs to judge from the set of generated questions, which is a potential
effective FQ to ask based on the user information. The improved question
generation performance of DLMs fine-tuned on conversational data filtered by
process knowledge encouraged us to prepare PRIMATE. PRIMATE can train DLMs to
judge ‘whether a user’s description of their mental health condition already
contains an answer to a particular question in PHQ-9’, which would eventually
guide coherent FQ generations. We leave our approach for FQ generation as
future work, but provide sufficient details on the broader forms of knowledge
needed in realizing such a pipeline.

### Limitations:

We are yet to scale our understanding to other mental health disorders, such
as anxiety using GAD-7 and Suicidality using C-SSRS Jiang et al. (2020).
Further, we are yet to investigate whether PRIMATE, along with the knowledge
in SCID can make DLMs transferable across multiple mental health disorders,
especially the ones comorbid with depression. Also, there is a need for a
clinically explainable safety metric for our task.

### Ethical Considerations:

Mental health communities on Reddit offer a crowd perspective on various
disorders wherein the FQs in the comments highlight the good intentions of
Reddit users to help users with conditions, such as depression. We take such
interactions as a proxy for improving patient-MHP interactions. Benton et al.
(2017) described that studies involving user-generated content are exempted
from the IRB requirement as long as the data source is public and the user’s
identity is not recognizable. Apart from being publicly available, Reddit
users are anonymous, and we further work with random user IDs. Since we make
PRIMATE public for research use, we use a Data Use Agreement Losada and
Crestani (2016) for responsible dissemination of the dataset.

##  6 Acknowledgment

We acknowledge partial support from the National Science Foundation (NSF)
award #2133842 “EAGER: Advancing Neuro-symbolic AI with Deep Knowledge-infused
Learning,” with PI Dr. Amit Sheth. Any opinions, findings, and conclusions or
recommendations expressed in this material are those of the author(s) and do
not necessarily reflect the views of the NSF.

## References

  * Althoff et al. (2016) Tim Althoff, Kevin Clark, and Jure Leskovec. 2016.  Large-scale analysis of counseling conversations: An application of natural language processing to mental health.  _Transactions of the Association for Computational Linguistics_ , 4:463–476.
  * Benton et al. (2017) Adrian Benton, Glen Coppersmith, and Mark Dredze. 2017.  Ethical research protocols for social media health research.  In _Proceedings of the First ACL Workshop on Ethics in Natural Language Processing_ , pages 94–102, Valencia, Spain. Association for Computational Linguistics.
  * Borgeaud et al. (2021) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. 2021\.  Improving language models by retrieving from trillions of tokens.  _CoRR_ , abs/2112.04426.
  * Brodey et al. (2018) Benjamin Brodey, Susan E Purcell, Karen Rhea, Philip Maier, Michael First, Lisa Zweede, Manuela Sinisterra, M Brad Nunn, Marie-Paule Austin, and Inger S Brodey. 2018.  Rapid and accurate behavioral health diagnostic screening: initial validation study of a web-based, self-report tool (the sage-sr).  _Journal of Medical Internet Research_ , 20(3):e9428.
  * Chicco and Jurman (2020) Davide Chicco and Giuseppe Jurman. 2020.  The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation.  _BMC genomics_ , 21(1):1–13.
  * Clark et al. (2019) Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2019.  Electra: Pre-training text encoders as discriminators rather than generators.  In _International Conference on Learning Representations_.
  * (7) CounselChat.  Mental health answers from counselors.
  * Demasi et al. (2019) Orianna Demasi, Marti A Hearst, and Benjamin Recht. 2019.  Towards augmenting crisis counselor training by improving message retrieval.  In _Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology_ , pages 1–11.
  * Dinan et al. (2021) Emily Dinan, Gavin Abercrombie, A. Stevie Bergman, Shannon L. Spruit, Dirk Hovy, Y-Lan Boureau, and Verena Rieser. 2021.  Anticipating safety issues in E2E conversational AI: framework and tooling.  _CoRR_ , abs/2107.03451.
  * Gaur et al. (2019) Manas Gaur, Amanuel Alambo, Joy Prakash Sain, Ugur Kursuncu, Krishnaprasad Thirunarayan, Ramakanth Kavuluru, Amit Sheth, Randy Welton, and Jyotishman Pathak. 2019.  Knowledge-aware assessment of severity of suicide risk for early intervention.  In _The World Wide Web Conference_ , pages 514–525.
  * Gaur et al. (2021) Manas Gaur, Kalpa Gunaratna, Vijay Srinivasan, and Hongxia Jin. 2021.  Iseeq: Information seeking question generation using dynamic meta-information retrieval and knowledge graphs.  _arXiv preprint arXiv:2112.07622_.
  * Gratch et al. (2014) Jonathan Gratch, Ron Artstein, Gale Lucas, Giota Stratou, Stefan Scherer, Angela Nazarian, Rachel Wood, Jill Boberg, David DeVault, Stacy Marsella, et al. 2014.  The distress analysis interview corpus of human and computer interviews.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 3123–3128.
  * Gu et al. (2021) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021.  Domain-specific language model pretraining for biomedical natural language processing.  _ACM Transactions on Computing for Healthcare (HEALTH)_ , 3(1):1–23.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  Realm: Retrieval-augmented language model pre-training.  _arXiv preprint arXiv:2002.08909_.
  * Holtzman et al. (2019) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019.  The curious case of neural text degeneration.  _arXiv preprint arXiv:1904.09751_.
  * Huang (2015) Rongyao Huang. 2015.  _Language use in teenage crisis intervention and the immediate outcome: A machine automated analysis of large scale text data_.  Ph.D. thesis, Master’s thesis, Columbia University.
  * Jiang et al. (2020) Zheng Ping Jiang, Sarah Ita Levitan, Jonathan Zomick, and Julia Hirschberg. 2020\.  Detection of mental health from reddit via deep contextualized representations.  In _Proceedings of the 11th International Workshop on Health Text Mining and Information Analysis_ , pages 147–156.
  * Karasz et al. (2012) Alison Karasz, Christopher Dowrick, Richard Byng, Marta Buszewicz, Lucia Ferri, Tim C Olde Hartman, Sandra Van Dulmen, Evelyn van Weel-Baumgarten, and Joanne Reeve. 2012.  What we talk about when we talk about depression: doctor-patient conversations and treatment decision outcomes.  _British Journal of General Practice_ , 62(594):e55–e63.
  * Komeili et al. (2021) Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2021.  Internet-augmented dialogue generation.  _CoRR_ , abs/2107.07566.
  * Kroenke et al. (2001) Kurt Kroenke, Robert L Spitzer, and Janet BW Williams. 2001.  The phq-9: validity of a brief depression severity measure.  _Journal of general internal medicine_ , 16(9):606–613.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474.
  * Losada and Crestani (2016) D. Losada and F. Crestani. 2016.  A test collection for research on depression and language use.  In _Proc. of Experimental IR Meets Multilinguality, Multimodality, and Interaction, 7th International Conference of the CLEF Association, CLEF 2016_ , pages 28–39, Evora, Portugal.
  * Manas et al. (2021) Gaur Manas, Vamsi Aribandi, Ugur Kursuncu, Amanuel Alambo, Valerie L Shalin, Krishnaprasad Thirunarayan, Jonathan Beich, Meera Narasimhan, Amit Sheth, et al. 2021.  Knowledge-infused abstractive summarization of clinical diagnostic interviews: Framework development study.  _JMIR Mental Health_ , 8(5):e20865.
  * Miner et al. (2016) Adam S Miner, Arnold Milstein, Stephen Schueller, Roshini Hegde, Christina Mangurian, and Eleni Linos. 2016.  Smartphone-based conversational agents and responses to questions about mental health, interpersonal violence, and physical health.  _JAMA internal medicine_ , 176(5):619–625.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for squad.  _CoRR_ , abs/1806.03822.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  Bleurt: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892.
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Kathleen S. Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022.  Lamda: Language models for dialog applications.  _CoRR_ , abs/2201.08239.
  * Weidinger et al. (2021) Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William S. Isaac, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2021.  Ethical and social risks of harm from language models.  _CoRR_ , abs/2112.04359.
  * Weizenbaum (1983) Joseph Weizenbaum. 1983.  Eliza — a computer program for the study of natural language communication between man and machine.  _Commun. ACM_ , 26(1):23–28.
  * Yazdavar et al. (2017) Amir Hossein Yazdavar, Hussein S Al-Olimat, Monireh Ebrahimi, Goonmeet Bajaj, Tanvi Banerjee, Krishnaprasad Thirunarayan, Jyotishman Pathak, and Amit Sheth. 2017.  Semi-supervised approach to monitoring clinical depressive symptoms in social media.  In _Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017_ , pages 1191–1198.