# Learning Personalized User Preference from Cold Start in Multi-turn
Conversations

Deguang Kong, Abhay Jha and Lei Yun<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Amazon Alexa AI, Sunnyvale, CA, U.S.A

(2023)

###### Abstract.

This paper presents a novel teachable conversation interaction system (TAI for
short) that is capable of learning users’ preferences from cold start by
gradually adapting to personal preferences. In particular, the TAI system is
able to automatically identify and label users’ preference in live
interactions, manage dialogue flows for interactive teaching sessions, and
reuse learned preference for preference elicitation. We develop the TAI system
by leveraging BERT encoder models to encode both dialogue and relevant context
information, and build action prediction (AP), argument filling (AF) and named
entity recognition (NER) models to understand the teaching session. We adopt a
seeker-provider interaction loop mechanism to generate diverse dialogues from
cold-start. TAI is capable of learning users’ preference, which achieves
91.22% turn level accuracy on out-of-sample dataset, and has been successfully
adopted in production.

preference, personalization, cold-start, dialogue, LLM

††copyright: acmcopyright††journalyear: 2023††doi: XXXXXXX.XXXXXXX††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computing methodologies Discourse,
dialogue and pragmatics

##  1\. Introduction

In conversation AI (Hosseini-Asl et al., 2020), the interaction between users
and conversation agents is helpful to guide users to achieve certain tasks.
Customers may desire and expect the conversation agent to provide proactive
personalized experiences in order to fulfill an advertised promise of a
helpful, personal assistant (Labutov et al., 2018). One significant effort
would be accelerating personalization by harmonizing disparate preference
learning experiences across the conversation agent.

In this paper, we present a teachable conversation agent system that augments
the task oriented AI agents with an interactive teaching capability. In
particular, the _challenges_ we are facing here include _(C1)_ how to allow
users to naturally start with describing the preferences and its conditionals
at a high-level, and then recursively teach the agent through the
conversations, _(C2)_ how to understand diverse related entities and
conversation flow users may have for their preferences in a multi-turn dialog,
_(C3)_ how to train the conversation model given scare training data in cold-
start environment, _(C4)_ how to reuse the previously taught user preferences
across different domains, etc.

Figure 1. The working flow of learning user preferences conversational system
(including teaching stage and reuse stage). Figure 2. Overview of conversation
understanding and dialogue management

Despite some works (Allen et al., 2007; Li et al., 2019; Yin and Neubig, 2017;
Jia et al., 2017; Hou et al., 2018; Kobayashi et al., 2019; Xu and Hu, 2018;
Zhao and Feng, 2018; Yang et al., 2020; Chen et al., 2019; Goo et al., 2018;
Takanobu and etc, 2019; Zhou et al., 2017) about learning from user
interactions, they are not sufficient to address the above challenges. For
example, interaction learning (Wang et al., 2017) is widely used in learning
visual scene concept (Mao et al., 2019) about attributes (such as color,
shape, etc), learning semantic concept (e.g., ”set my room to cozy”) (Ping et
al., 2020) using concept parser, as well as games (Kirk and Laird, 2017) using
the symbolic execution for state progression and scene understanding. However,
the method is not directly applicable for preference teaching. While
reinforcement learning approach (Liu and Lane, 2017) views the dialogue
conversation as a sequential decision making process that learns optimal
policy, it is over-complicated and difficult to customize for different domain
teaching sessions.

To our knowledge, this work is the _first_ work that builds a conversation
agent which allows the users to share their preferences across different
topics and get more relevant and personalized results. The _contribution_ of
this paper is summarized as follows.

$\bullet$ To address _challenge C1_ , a multi-turn domain agnostic dialog
system is designed to support user initialized teaching, allowing users to
describe the preference, clarify ambiguities from explicit instructions.

$\bullet$ To address _challenge C2_ , the conversation agent performs a multi-
turn action prediction loop upon named entity recognition, action prediction
and argument filling models by encoding dialog context using BERT and Seq2Seq
model pipeline.

$\bullet$ To address _challenge C3_ , we designed a dialogue simulator module
to generate synthetic training data with adequate annotations guided by the
seeker-provider interaction loop to achieve the seeker goals from cold-start.

$\bullet$ To address _challenge C4_ , we enable centralized preference
management, which stores the taught preference in a persistent knowledge base,
facilitating effective reusability and generalizability of learned knowledge.

##  2\. System Overview

TAI consists of two stages (i.e., teaching stage and reuse stage). In the
teaching stage, given user utterance, TAI will call a dialogue management
module to learn users’ preferences from interaction. Users’ preferences will
be stored at the knowledge base after standardizing the schema for future use.
After enabling reuse, when a user asks the specific questions, the domain will
retrieve the relevant preference (stored in the knowledge base) and further
infer additional latent customer preferences to curate more proactive
personalized experiences to users. Fig. 1 depicts how the TAI system works.
The purpose of dialogue management (Fig. 2) is to conduct interactive sessions
with users for learning preferences and drive the conversations based on the
seeker goals to complete the objectives. Essentially, it consists of several
components:

$\bullet$ Preference parser: Conversational agent will bridge the prediction
gap to understand and interpret the preference for the given utterance from
the user. For example, it will understand the slot relevant to preference in
utterance such as I love Giants [sportsteam]. This is mainly achieved by NER
illustrated in Section 4.1.

$\bullet$ Dialogue management: After the session has been initiated by the
user in the conversation, the dialog management module will predict the right
clarification question to ask the user, based on the context of utterance and
the interactions. Then the user will respond for clarification (if needed) and
continue the multi-turn conversations until the respective preferences are
grounded before storage of them for future re-use. This is achieved by a
multi-turn action prediction loop illustrated in Section 4.2.

$\bullet$ Dialogue simulation from cold start: we build the simulator based on
seeker provider interaction loop to generate diverse dialogues that are beyond
these seed examples by enriching them with surface form (i.e. utterance)
variations, other dialogue flows and possible unhappy cases. This is mainly
achieved by simulation in Section 3.

$\bullet$ Centralized Knowledge Storage: The taught preference is stored in a
persistent knowledge graph, which facilitates effective reusability and
generalizability of learned knowledge. In particular, it supports the
following functionalities:

⬇

Update_KB(User_Id, &PrefList) \\\

Retrieve_KB(User_Id, &PrefList)

⬇

Retrieve_KB(User_Id, &PrefList)

Update_KB(User_Id, &PrefList)

which enables the TAI to store, update and retrieve the [user_id,
preference_list] pairs in the knowledge base. Through these interactions, it
leads to more complex personal interactions and action requests that surface
more nuanced gaps in conversation understanding.

###  2.1. Utterance Examples

The sampled utterance example is shown in Table 1.

Table 1. Sample golden utterances Domain | Goal | Utterances
---|---|---
sports | like sport team | The Yankees are my favorite baseball team
sports | like sport team | My favorite teams are the Yankees and The Knicks
sports | like sport team | Add the Warriors to my favorites
sports | like sport team | I am a new York giant fan.
sports | dislike sport team | Remove the Yankees from my favorite sports teams
sports | dislike sport team | Unfollow San Francisco giant.
restaurant | restaurant/cuisine one loves | I like to dine at Thai restaurant
restaurant | restaurant/cuisine one loves | I enjoy seafood restaurant.
preference management | reset preference | Forget everything you’ve learned about my preferences
preference management | reset preference | Delete all my preferences
preference management | delete preference | Remove Mexican food from our food preferences.
preference management | delete preference | I want you to forget my preference for vegetarian food.
weather provider | conditional preference | I prefer big sky for weather update.
weather provider | conditional preference | Always use RADAR for weather forecasting.

##  3\. Simulating Dialogues to Address Cold Start Issues

We design and implement the dialogue simulator module (“simulator” for short)
to generate synthetic training dialogues with adequate annotations since we do
not have any user data before product launch. For this purpose, the input to
the simulator module would be small number of annotated seed dialogues with
descriptions, and the output would be diverse dialogues that are generated
beyond these seed examples by enriching them with surface form (i.e.
utterance) variations, other dialogue flows and possible unhappy cases. In the
conversation, the user is annotated as a seeker who seeks to accomplish a
goal. The conversation agent (annotated as a provider) provides information or
a service to help the seeker complete the goal. The simulation is mainly
achieved by seek-provider interaction.

Stage 1: Seeker goal In this stage, a fixed seeker goal is sampled from the
goal sampler. To describe the seeker goal of a conversation that consists of a
sequence of API calls by the provider to fulfill the seeker’s goal, we adopt
an _entity transfer graph_ , i.e., a _directed acyclic graph (DAG)_ where the
vertices consists of the entities supplied by the seeker and the API calls of
the provider, and the edges encode the way that seeker entities and entities
returned by API calls are transferred (i.e. carried over), such that the
arguments of each API call in the goal are therefore encoded as parents in the
graph. Fig. 3 gives an example of entity transfer graph. It draws an API by
following a Markov chain distribution (denoted as $A_{1:i}$ for API sequences
[1:i]) estimated from the seed conversation, i.e,

(1) $\Pr(A_{1:i})=\prod_{j}\Pr(A_{i}|A_{j|{j<i}})$

where the probability of goal $\Pr(A_{1:j})$ is achieved by sampling a
sequence of intents (via APIs) in the sequential order (with index $i$, $j$).

Then starting with an empty entity transfer graph, seeker goals are sampled
incrementally by repeatedly determining if a new vertex should be added. In
particular, it draws an API by following a Markov chain distribution estimated
from the seed conversation, i.e,

(2) $Pr(\text{goal})=\Pi_{i}Pr(\text{API}_{i}|$

where the probability of goal $Pr(\text{goal})$ is achieved by sampling a
sequence of intents (via APIs) in the sequential order (e.g., $i$, $j$). The
sampling sequence $j$ is restricted to be $i-1$ for Markov sampling. For each
API argument, it randomly transfers existing entities or elicits new entities
from the seeker, and then adds the entities elicited (if any) and adds the new
API vertex with parent edges from the transferred entities.

Figure 3. An example of entity transfer graph for the utterance I prefer the
weather app for daily updating and rain forecasting

Stage 2: Seeker-Provider Interaction Loop The provider and seeker agents
exchange a series of messages composed of their respective actions and
utterances. On one hand, the seeker incrementally reveals the goal to the
provider with actions corresponding to dialogue acts. On the other hand, the
provider collects all the arguments necessary to make API calls, makes those
calls and informs the seeker of the results with NLG template actions and API
calls. Starting with the seeker, the conversation agent executes at each turn
for a) action sampling (i.e., a sequence of actions is drawn according to a
distribution of possible actions), b) NLG: an utterance corresponding to the
dialogue acts associated to the sampled actions is drawn. Both parts provide
the necessary annotations.

###  3.1. Design principle

To dive deep into the simulator mechanics, the design is guided by generating
diverse dialogues. It could be generated in the follow ways.

First, dialog variations can be introduced by resampling the entities of the
conversation using catalogs and using alternative utterance wordings. For
example, the slot sport_team in “I love sport_team” would be substituted by
other entities in catalog. Second, the user could change the information
provided in prompts in each turn. For example, when a user says ”I don’t like
the Gaint”, the generated clarification would be ”which gaint do you mean, New
York Gaint or San Francisco gaint?” The benefit is to proactive suggest users’
preferences for feature discovery. Third, it is important to simulate cases
with sufficient randomness or even errors, since neither the conversation
agent or the user will behave optimally. For example, when a user says ”what’s
my sports update?”, the conversation agents may see the situation that a user
has not set his preference yet, and asked ”Please set up your sport preference
before getting notifications.” The clarification sub-dialogues is to reduce
uncertainty in conversation agents’ estimation of the dialogue state. Finally,
to avoid overfitting during model training, it is important to generate a
previously unseen conversation to “realistic” simulator potential errors or
misunderstandings.

###  3.2. Architecture

Based on the above design principle, we illustrate the components of the
dialogue simulator and how they interact with the modeling component.

The simulator adopts the concept of dialogue acts to express the semantics
associated to utterances. For example, inform is the act used by the seeker to
communicate their intent and by the provider to describe the result of their
actions. Dialogue acts are used at the interface between the natural language
generation (NLG) unit, which converts the dialogue content and direction to a
corresponding language realization, and conversation agent.

Procedure Natural Language Generation component is currently based on
templates and is responsible for the generation of the seeker’s utterances
from dialogue acts, where seeker utterances are first categorized with
Dialogue Act classifier and then converted into NLG templates using simple
heuristics. In particular, it consists of three key steps:

Step 1: A small batch of 100 dialogues is simulated with the extracted seeker
NLG templates.

Step 2: A Mechanical Turk task is submitted to obtain variations of the
extracted seeker NLG templates by crowdsourcing, where paraphrasing is done
both at the utterance level and conversation level.

Step 3: A larger set (e.g., 50k) of annotated dialogues is simulated and used
for supervised training the conversation model.

###  3.3. Simulated dialogues for addressing cold-start issues

An example of simulated dialogue for addressed cold-start issue is shown
below.

U-1: ”type”: ”User_intent”, ”[String] User side utterance”: ”forget everything
I have taught you”, ”[List of string] UserNLGs associated with the user
utterance”: preferenceteaching.events.DeleteAllAffinityEvent();

U-2:”type”: ”User_intent”, ”partially_normalized_value”: ”[let it
be—SINGLESITE.Confirmation -¿ confirmation1]”, ”fully_normalized_value”:
[”com.cu.preferenceteaching.events.ConfirmationEvent
(confirmation=confirmation1)”];

S-1: ”type”: ”api”, ”[String] User side utterance”: ”getAllAffinityAction() -¿
getAllPreferenceResult1”,”, ”[List of string] UserNLGs associated with the
user utterance”: getAllAffinityAction() -¿ getAllPreferenceResult1”;

S-2: ”type”: ”nlg”,
”notify_com.cu.preferenceteaching.actions.getAllAffinityAction_success
(getAllAffinityResult=getAllPreferenceResult1)”;

S-3: ”type”: ”sys”, ”normalized_value”: ”wait_for_user_input()”;

S-4: ”type”: ”api”, ”normalized_value”:
”com.cu.preferenceteaching.actions.deleteAllAffinityAction
(confirmAction=confirmation1) -¿ deleteAllPreferenceResult1”;

S-5: ”type”: ”nlg”, ”fully_normalized_value”:
”notify_com.cu.preferenceteaching.actions.deleteAllAffinityAction _success
(deleteAllAffinityResult=deleteAllPreferenceResult1).

###  3.4. Preference teaching use case

In the context of preference teaching, the user goal in the goal-oriented
dialogue system is a sequence of APIs (such as
setDietOrCuisineAffinity, setSportAffinity) and their corresponding arguments
and values. One can sample the dialogue based on predefined templates and
perform sampling on the entities (e.g., I love {entity}). The arguments and
their values of APIs are sampled as well for more variations. We perform
sampling of API transitions in dialogue flows. For example, when computing API
transition weight, we consider factors such as API transitions observed in
dialog examples provided by the developer, shared entities amongst APIs,
input-output relation between APIs, etc. We then combine these weights using
the mixing ratio, treated as hyper parameters tuned for simulated dialogue
generation. An example of simulated dialog annotated with markup language
(where U- gives the user side utterance and S- gives the conversation agent
side actions) is illustrated in 3.3.

##  4\. Mutli-turn Action Prediction Loop

###  4.1. Overview

Figure 4. The model prediction flow.

Action Prediction The responsibility of prediction modeling is to (a)
understand dialogue context so that it can decide which actions should be
taken next; (b) fill arguments for selected actions with entities that
occurred in dialogue context, with the purpose of guiding the conversation in
the loop. In particular, the model predicts the actions in several types,
including intent API (e.g., SetMyPreference()) with argument, NLG service with
argument (e.g., the corresponding natural language response based on NLG
template and its argument, e.g., what is a restaurant type, and Sys system
actions (e.g., wait for user input wait_for_user() or end the dialog
end_dialogue), by leveraging the storage of dialogue context and input
utterance and context information to the conversational model. Another import
module is name entity recognition (NER) model on user utterances so as to
restrict the number of entities the model needs to consider in argument
filling for action prediction.

Multi-turn Prediction Loop Since the model continuously interacts with the
context storage in the conversation, the context storage is updated at dialog
turn level when the model makes a new prediction for the action before
handling the control between seeker and provider. For example, the user may
request call an intent/API triggered by forge everything I have taught you,
and the agent model will perform an API call for searching all users’
preferences by getAllPreference(), and then generate one NLG call that informs
the user successfully obtaining all user preferences by
getAllAffinityAction_success, before waiting for user input via a sys_call
like wait_for_user_input(). Then after prompting the user to provide the
confirmation of the deletion operation via ConfirmationEvent() API call, the
agent will call the intent API of  deleteAllAffinityAction(confirmAction)
followed by NLG generation of successful deletion of all preferences via
deleteAllAffinityAction_success. Fig. 4 illustrates the model prediction
operation loop.

###  4.2. Model breakdown

As is illustrated before, the conversation model is not a single monolithic
model, but consists of several key models instead.

Dialogue Context Encoder It encodes dialogue context into vector embeddings
used by other models. In particular, in dialog contexts, we extract (1)
current utterance (denoted as $E_{cu}$), (2) past utterances (denoted as
$E_{pu}$), (3) current entities (denoted as $E{ce}$), (4) past entities
(denoted as $E_{pe}$), (5) past actions (denoted as $E_{pa}$) as features, and
adopt BERT encoder (Devlin et al., 2019) as embeddings before concatenating
them into a unified context embedding $C$, i.e.,

(3) $\displaystyle C=[E_{cu},E_{pu},E_{ce},E_{pe},E_{pa}].$

Action Prediction (AP) Model  Given the embedding of context history, it
predicts $n$-best list of actions (API/NLG/SYS)111API denotes standard api to
achieve user goals, e.g., get_all_preference, set_all_preference, NLG denotes
the process of producing natural language output as the response, SYS denotes
system related operations, such as wait_for_user_input(), end the
conversation, etc from encoded dialogue context by computing the distribution
over possible actions (denoted as $A$) with a multi-layer feed-forward network
using ReLu activation and softmax over possible actions, i.e.,

(4) $A=\text{Softmax}(\text{ReLu}(WC+b)),$

where $W$ and $b$ are the weight matrix and bias vector learned for action
prediction, respectively. An example of action prediction output is shown
below: [(API:get_all_preference, 0.75), (API: set_all_preference, 0.25]

The responsibility of Argument Filling (AF) Model is to fill arguments of
predicted actions with entities in dialogues, whereas NER model recognizes the
entity in the utterance, which restricts search space for argument filling
model.

###  4.3. AF model and NER model

Argument Filling (AF) Model It fills arguments of predicted Actions with
entities in dialog context. A complete argument-filled action should be
consistent with API/NLG constraints defined in conversation goal descriptions
with encoded dialog context. Given each $c_{i}$ in a representation of the
$i$-th token in context encoder, the prediction is a scalar $p_{i}$ between 0
and 1, indicating the probability of $i$-th token being the argument value. To
make a better prediction, the embeddings of _argument_name_ , _argument_type_
, _action_name_ are fed into the model as the dialog context, attention scores
are calculated over all hidden states over dialogue context with entities that
violates API/NLG constraints masked out. Finally, the highest scoring entity
$p_{i}$ is chosen as the argument of all tokens of $x_{i}$, i.e.,

(5) $p_{i}=\text{sigmoid}(\text{Attention}(C,x_{i})).$

NER model It recognizes the entity in the utterance, which restricts search
space for argument filling model. Given the encoded context history, and the
embedding of each (e.g., $i$-th) token in current utterance using BERT, a
bidirectional LSTM-CRF (Huang et al., 2015) is employed to predict the
probability vector over all possible entities for inside, outside, beginning
tagging by optimizing the objectives:

(6)
$Pr(s|x;w)=\frac{exp(w\dot{\phi}(x,s))}{\sum_{s^{\prime}}exp(w\dot{\phi}(x,s^{\prime}))},$

where function $\phi()$ maps the input sequence $x$ to state sequence $s$ with
model parameter $w$, and $s^{\prime}$ ranges over all possible output sequence
with the scoring function $w\dot{\phi}(x,s))$ indicating how well the state
sequence fits the given input sequence that is modeled using a bidirectional
LSTM with parameter learned by back propagation.

###  4.4. NER model optimization via N-gram catalog feature

Suppose we have entity label space {sport_team, city}, the sport_team catalog
= {San Francisco Giant}, the city catalog = {San Francisco}, corresponding to
each token in the user utterance ‘‘I follow San Francisco Giant’’, to identify
the boundary and type of the entity, we generate the 1-gram catalog feature,
i.e.,

$[[0,0],[0,0],[0,0],[0,0],[0,0]],$

2-gram catalog feature, i.e.,

$[[[0,0],[0,0],[0,1],[0,0],[0,0]],$

and 3-gram catalog feature, i.e.,

$[[0,0],[0,0],[1,0],[0,0],[0,0]].$

##  5\. Performance Evaluations

###  5.1. Model Accuracy

Dialogues for training model To train the model, we generate training dataset
by sampling user goals (denoted as sequence of APIs, their arguments and
values) before generating dialogues from extension of these user goals. Table
1 (Appendix 2.1) lists examples of the goals we would like to support using
the corresponding utterances in the dialogue flow. Using dialogue simulation
(see Sec 3), This totally gives 50,000 dialogues, 546,509 actions and 151,208
turns (Table 2).

Model performance test on in-sample dialogues To evaluate the predictive
capabilities of the models, We generate an in-sample testing dataset using the
dialogue simulator, which gives 500 dialogues, 5673 actions and 1507 turns
(Table 3).

Model Robustness testing To evaluate the robustness of our model, we also
collect how users will describe these preferences by starting
AMT222https://www.mturk.com/ task to collect the variations of those goal-
oriented dialogues that would produce utterance level variance (i.e.,
different utterance for the same meaning) and dialog level variance (i.e.,
diverse dialogue working flow). Then we manually inspect each dialogue, and
remove the redundant and unreasonable ones.

After collecting these dialogues, to abide by the annotation contract, we
manually label these AMT dialogue by properly injecting the dialogues based on
dialogue definitions that clearly gives the entity boundaries, action
signatures with corresponding parameters in consistent with dialogue
generation contract. After data cleaning, finally it gives 500 dialogues and
1480 turns (Table 4). We call it “out-of-sample” evaluation dataset since it
consists of utterance-level and dialogue level variations that are _not_
covered in training dialogues. Compared to in-sample dataset, this dialogue
dataset has more variations at both utterance-level and dialogue level, which
may not match well with the training dialogues. That’s the reason we call it
as “out-of-sample” evaluation dataset. The annotated dialogue flows are in the
same format that can be processed by conversation understanding models.

Table 2. Statistics of training dataset #API | # Testing dialogues | # Action | # turns
---|---|---|---
12 | 50,000 | 546,509 | 151,208
Table 3. Statistics of in-sample evaluation dataset #API | # Testing dialogues | # Action | # turns
---|---|---|---
12 | 500 | 5673 | 1507
Table 4. Statistics of out-of-sample evaluation dataset #API | # Testing dialogues | # Action | # turns
---|---|---|---
12 | 500 | 5355 | 1480

Metrics We use _turn level accuracy_ to measure the performance of the models
for the multi-turn dialog system. One turn includes the user’s utterance plus
the conversation agent’s response, which measures a two-sided conversation in
a dialog. Each turn is considered as correct when all predictions of models
(including NER, AP, AF) in the turn are correct. For example, for the NER
model, we can compare the predictions of all entities against the ground-
truth, and define precision, recall, accuracy and F1-score based on the
percentage of correctly predicted entities against the ground-truth of
entities. The turn level accuracy for AP and AF are similarly defined. We also
show the model performance at each action level (i.e., at API, NLG, SYS
levels), and the accuracy is defined based on the percentage of correctly
predicted ones against the ground-truth at each-action level. We show model
prediction results for each model (such as NER, AP and AF). Also, we show the
end2end model evaluation result (denoted as NER+AP+AF), which indicates the it
is correct if and only if all NER/AP/AF are correct for an action (or a turn).

Result Tables 5, 6 show both the turn level accuracy and action-level accuracy
using both in-sample evaluation dataset and out-of-sample evaluation dataset.
The result is expected as the model performs reasonably well on an in-sample
dataset, whereas the model did not handle some entities correctly and
therefore affects argument filling (AF) model performance as well since these
entities might be out of dictionary.

Table 5. Accuracy (ACC) at per-turn level and per-action level for different models in _in-sample_ evaluation dataset model | ACC per-turn | ACC per-action
---|---|---
NER | 99.85% | 99.85%
AP | 100% | 100%
AF | 97.89% | 98.72%
AP + AF | 97.89% | 98.72%
NER + AP + AF | 97.40% | 97.65%
Table 6. Accuracy (ACC) at per-turn level and per-action level for different models in _out-of-sample_ evaluation dataset model | ACC per-turn | ACC per-action
---|---|---
NER | 94.48% | 95.89%
AP | 97.18% | 97.18%
AF | 93.92% | 95.24%
AP + AF | 92.57% | 94.92%
NER + AP + AF | 91.22% | 91.22%
Table 7. Accuracy (ACC) at per-turn level and per-action level for NER models in _out-of-sample_ evaluation dataset after adding catalog specific features (denoted as CF) model | ACC per-turn | ACC per-action
---|---|---
NER w/ CF | 96.79% | 98.30%
NER w/o CF | 94.48% | 95.89%

###  5.2. Lessons Learned

Increasing simulated dialogues for addressing cold start issues Thanks to the
dialogue simulator component that generates many training dialogues, we are
able to overcome the difficulty of insufficient training dialogues from cold-
start. However, when we did evaluation using AMT data collected from user
study, we still observed that some dialogue samples are _not_ covered by the
simulator. We may leverage the recent advances in generative Q&A (e.g., GPT-3
(Brown et al., 2020), OPT-175B model (Zhang et al., 2022)) to complement
current training dialogues from simulators. The training dialogues are
augmented by collecting online user dialogues as well after the product is
deployed. All these dialogues are put into _feedback loop_ to update the
current model.

More open and personalized preference suggestions When we train the model, the
entity type is given when setting the user preference. To move forward, some
users are interested in receiving more accurate suggestions after setting
preferences. We would provide more personalized suggestions based on
preferences learned from current users. We may even support open conversations
such as ‘‘What are my favorite ___" to see what is the entity list to be
added, rather than the fixed entity types.

Expanding catalogs When we do offline evaluation for the model, we see NER
model performs well on the in-sample evaluation dataset. However, it drops in
out-of-sample dataset. In user study, some users responds wish to be able to
add more options to their different categories of preferences. For example, we
may expand sports team options to include college sports and teams, and minor
league teams as well. We enabled _n-gram catalog feature_ to get catalog
specific representations particularly for the entity types that have free-form
catalog values which may not be well captured by BERT pre-trained models on
Wikipedia data. These catalog features (Appendix 4.4) are concatenated into
the embedding in LSTM-CRF models and Table 7 shows the performance improvement
in out-of-sample dataset for NER tasks.

###  5.3. User satisfaction

We conducted a user study to test the developed feature performance as well.
Overall, participants expressed satisfaction when teaching their preferences
for sports, restaurants and weather providers in a single session, emphasizing
how easy and seamless the experience was when setting different types of
preferences all at once. These findings are in line with the results observed
in model evaluations.

In particular, we notice some reasons for user dis-satisfactions including ASR
recognition error, issues with loops/crashes, and incorrect response, which
motivates us to further improve the end2end modeling performance. Fig. 5
demonstrates how the user interacts with the conversation agent in the
product.

Table 8. Top reasons of user dissatisfactions Reason | Ratio(%)
---|---
Issues with ASR recognition | 25%
Issues with loops/errors/crashes | 20%
incorrect/no response | 25%
Some entities are incorrectly recognized | 20%
Others | 10%
Figure 5. A snapshot for the conversation flow.

##  6\. Conclusion

This paper presents a conversation dialog system capable of teaching users’
preference for bridging the gaps of conversation understanding and dialog
management. The teachable dialogue system improves the language understanding
capabilities and provides more natural conversations to users. TAI
demonstrates good performance in teaching users’ preferences in terms of model
accuracy and user satisfaction. We are extending the application to various
domains by supporting conversation based reasoning and personalized
recommendation.

## References

  * (1)
  * Allen et al. (2007) James F. Allen, Nathanael Chambers, George Ferguson, Lucian Galescu, Hyuckchul Jung, Mary D. Swift, and William Taysom. 2007.  PLOW: A Collaborative Task Learning Agent. In _AAAI’07_ (Vancouver, British Columbia, Canada) _(AAAI’07)_. 1514–1519.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners. In _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901.  https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
  * Chen et al. (2019) Qian Chen, Zhu Zhuo, and Wen Wang. 2019.  BERT for Joint Intent Classification and Slot Filling.  _CoRR_ abs/1902.10909 (2019).  arXiv:1902.10909
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.  https://doi.org/10.18653/v1/N19-1423
  * Goo et al. (2018) Chih-Wen Goo, Guang Gao, and etc. 2018.  Slot-Gated Modeling for Joint Slot Filling and Intent Prediction. In _NAACL-HLT 2018_. 753–757.
  * Hosseini-Asl et al. (2020) Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020.  A Simple Language Model for Task-Oriented Dialogue. In _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 20179–20191.  https://proceedings.neurips.cc/paper/2020/file/e946209592563be0f01c844ab2170f0c-Paper.pdf
  * Hou et al. (2018) Mulan Hou, Xiaojie Wang, Caixia Yuan, Guohua Yang, Shuo Hu, and Yuanyuan Shi. 2018\.  Attention Based Joint Model with Negative Sampling for New Slot Values Recognition. In _IWSDS 2018_ , Vol. 579. Springer, 3–15.
  * Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015.  Bidirectional LSTM-CRF Models for Sequence Tagging.  _CoRR_ abs/1508.01991 (2015).  arXiv:1508.01991 http://arxiv.org/abs/1508.01991
  * Jia et al. (2017) Robin Jia, Larry Heck, Dilek Hakkani-Tür, and Georgi Nikolov. 2017.  Learning concepts through conversations in spoken dialogue systems. In _ICASSP2017_. 5725–5729.
  * Kirk and Laird (2017) James R. Kirk and John E. Laird. 2017.  Learning Hierarchical Symbolic Representations to Support Interactive Task Learning and Knowledge Transfer. In _IJCAI-19_. 6095–6102.
  * Kobayashi et al. (2019) Yuka Kobayashi, Takami Yoshida, Kenji Iwata1, and Hiroshi Fujimura. 2019.  Slot Filling with Weighted Multi-Encoders for Out-of-Domain Values. In _Interspeech_. 854–858.
  * Labutov et al. (2018) Igor Labutov, Shashank Srivastava, and Tom Mitchell. 2018\.  LIA: A Natural Language Programmable Personal Assistant. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_. Association for Computational Linguistics, Brussels, Belgium, 145–150.  https://doi.org/10.18653/v1/D18-2025
  * Li et al. (2019) Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M. Mitchell, and Brad A. Myers. 2019.  PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations. In _UIST2019_ (New Orleans, LA, USA) _(UIST ’19)_. 577–589.
  * Liu and Lane (2017) Bing Liu and Ian R. Lane. 2017.  Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Models.  _CoRR_ (2017).  arXiv:1709.06136
  * Mao et al. (2019) Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. 2019.  The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision. In _ICLR_.
  * Ping et al. (2020) Qing Ping, Feiyang Niu, Govind Thattai, Joel Chengottusseriyil, Qiaozi Gao, Aishwarya Reganti, Prashanth Rajagopal, Gokhan Tur, Dilek Hakkani-Tür, and Prem Natarajan. 2020.  Interactive teaching for conversational AI. In _NeurIPS 2020 Workshop on Human in the Loop Dialogue Systems_.  https://www.amazon.science/publications/interactive-teaching-for-conversational-ai
  * Takanobu and etc (2019) Ryuichi Takanobu and etc. 2019.  Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog. In _EMNLP-IJCNLP_. 100–110.
  * Wang et al. (2017) Sida I. Wang, Samuel Ginn, Percy Liang, and Christopher D. Manning. 2017.  Naturalizing a Programming Language via Interactive Learning. In _ACL_. 929–938.
  * Xu and Hu (2018) Puyang Xu and Qi Hu. 2018\.  An End-to-end Approach for Handling Unknown Slot Values in Dialogue State Tracking. In _ACL_. 1448–1457.
  * Yang et al. (2020) Puhai Yang, Heyan Huang, and Xianling Mao. 2020.  Context-Sensitive Generation Network for Handing Unknown Slot Values in Dialogue State Tracking.  _CoRR_ abs/2005.03923 (2020).  arXiv:2005.03923
  * Yin and Neubig (2017) Pengcheng Yin and Graham Neubig. 2017.  A Syntactic Neural Model for General-Purpose Code Generation. In _ACL_. Association for Computational Linguistics, 440–450.  https://doi.org/10.18653/v1/P17-1041
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022.  OPT: Open Pre-trained Transformer Language Models.  https://doi.org/10.48550/ARXIV.2205.01068
  * Zhao and Feng (2018) Lin Zhao and Zhe Feng. 2018\.  Improving Slot Filling in Spoken Language Understanding with Joint Pointer and Attention. In _ACL_. Association for Computational Linguistics, Melbourne, Australia, 426–431.  https://doi.org/10.18653/v1/P18-2068
  * Zhou et al. (2017) Li Zhou, Kevin Small, Oleg Rokhlenko, and Charles Elkan. 2017\.  End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradient.  _CoRR_ abs/1712.02838 (2017).  arXiv:1712.02838 http://arxiv.org/abs/1712.02838