# Self-Supervised Bot Play for Conversational Recommendation with
Justifications

Shuyang Li  UC San Diego<EMAIL_ADDRESS>,  Bodhisattwa Prasad Majumder  UC
San Diego<EMAIL_ADDRESS>and  Julian McAuley  UC San Diego
<EMAIL_ADDRESS>
(2021)

###### Abstract.

Conversational recommender systems offer the promise of interactive, engaging
ways for users to find items they enjoy. We seek to improve conversational
recommendation via three dimensions: 1) We aim to mimic a common mode of human
interaction for recommendation: experts justify their suggestions, a seeker
explains why they don’t like the item, and both parties iterate through the
dialog to find a suitable item. 2) We leverage ideas from conversational
critiquing to allow users to flexibly interact with natural language
justifications by critiquing subjective aspects. 3) We adapt conversational
recommendation to a wider range of domains where crowd-sourced ground truth
dialogs are not available. We develop a new two-part framework for training
conversational recommender systems. First, we train a recommender system to
jointly suggest items and justify its reasoning with subjective aspects. We
then fine-tune this model to incorporate iterative user feedback via self-
supervised bot-play. Experiments on three real-world datasets demonstrate that
our system can be applied to different recommendation models across diverse
domains to achieve superior performance in conversational recommendation
compared to state-of-the-art methods. We also evaluate our model on human
users, showing that systems trained under our framework provide more useful,
helpful, and knowledgeable recommendations in warm- and cold-start settings..

conversational recommendation, recommender systems, critiquing

††copyright: iw3c2w3g††journalyear: 2021

Figure 1. Conversational critiquing workflow. The system scores candidate
items and generates a justification for the top item. If the user rejects the
suggestion and critiques an aspect, the system uses the critique to update the
latent user representation.

##  1\. Introduction

Traditional recommender systems often return static recommendations, with no
way for users to meaningfully express their preferences and feedback. However,
interactivity and explainability can greatly affect a user’s trust of and
willingness to use a recommender system (Tintarev and Masthoff, 2011; Qiu and
Benbasat, 2009). This is reflected in human conversations: experts justify
their recommendations, customers critique suggestions, and both parties
iterate through the conversation to arrive at a satisfactory item.

Early work on interactive recommender systems focused on iteratively
presenting suggestions to the user based on simple “like” and “dislike”
feedback on individual items (Ko and Byun, 2002; Jin and French, 2003; Biswas
et al., 2019). Gradually, systems began to accommodate more fine-grained user
feedback—critiquing fixed attributes of an item (e.g. its color or brand)
(Chen and Pu, 2012). Recent models for conversational critiquing incorporate
user feedback on subjective aspects (e.g. taste and perception) (Wu et al.,
2019; Luo et al., 2020b, a). However, such methods are trained using a next-
item recommendation objective, and perform poorly when engaging with users
over multiple turns.

Another approach lies in training dialog agents to interact with the user over
multiple turns (Wärnestål, 2005). While such models are able to generate
convincing dialog in a vacuum, they require large corpora of transcripts from
crowd-sourced recommendation games (Li et al., 2018; Kang et al., 2019). To
create high-quality training dialogs, crowd-workers must be knowledgable about
many items in the target domain—this expertise requirement limits data
collection to a few common domains like movies. Additionally, these dialog
policies limit a user’s freedom to interact with the system by asking yes/no
questions about specific item attributes.

We thus desire a conversational recommender system that mimics characteristics
of human interactions not yet captured by existing systems:

  1. (1)

It can justify suggestions made to the user;

  2. (2)

It updates suggestions based on user feedback about subjective aspects; and

  3. (3)

It can be trained using review data that is easily harvestable from arbitrary
new domains.

To accomplish these goals, we present a two-part framework to train
conversational recommender systems. Ours is the first framework, to our
knowledge, that allows training of conversational recommender systems for
multi-turn settings without the need for supervised dialog examples. First,
using a next-item recommendation task we learn to encode historical user
preferences and generate justifications for our suggestions via sets of
subjective aspects. We then fine-tune our trained model via multiple turns of
bot-play in a recommendation game based on user reviews and simulated
critiques. We apply our framework to two base recommender systems (PLRec
(Sedhain et al., 2016) and BPR (Rendle et al., 2009)), and evaluate the
resulting PLRec-Bot and BPR-Bot models on three large real-world
recommendation datasets containing user reviews. Our method reaches the target
item with a higher success rate than state-of-the-art methods, and takes fewer
turns to do so, on average. We also conduct a study with real users, showing
that it can effectively help users find desired items in real time, even in a
cold-start setting.

We summarize our main contributions as follows: 1) We present a framework for
training conversational recommender systems using bot-play on historical user
reviews, without the need for large collections of human dialogs; 2) We apply
our framework to two popular recommendation models (BPR-Bot and PLRec-Bot),
with each showing superior or competitive performance in comparison to state-
of-the-art recommendation and critiquing methods; 3) We demonstrate that our
framework can be effectively combined with query refinement techniques to
quickly suggest desired items.

##  2\. Related Work

Table 1. Conversational critiquing systems (first section) are transcript-free but not equipped for multi-turn interactions. Dialog-based agents (second section) learn multi-turn interactions using large corpora of domain-specific dialog transcripts. Our framework allows us to train multi-turn interactive recommender systems without costly transcript data. | Justification | Multi-Turn | Transcript-Free
---|---|---|---
LLC (Luo et al., 2020a) | ✗ | ✗ | ✓
CE-VAE (Luo et al., 2020b) | ✓ | ✗ | ✓
M&M VAE (Antognini and Faltings, 2021) | ✓ | ✗ | ✓
Li et al. (2018) | ✗ | ✓ | ✗
Kang et al. (2019) | ✓ | ✓ | ✗
Zhou et al. (2020) | ✓ | ✓ | ✗
Ours | ✓ | ✓ | ✓

###  2.1. Recommendation with Justification

Users prefer recommendations that they perceive to be transparent or justified
(Sinha and Swearingen, 2002; Tintarev and Masthoff, 2011). Some early
recommender systems presented the objective attributes of suggested items to
users (Vig et al., 2009; Symeonidis et al., 2009), but did not attempt to
personalize the justifications. Another line of work considered the problem of
generating natural language explanations of recommendations. McAuley et al.
(2012) extract key aspects from the text of user reviews using topic
extraction. Such justifications have been expanded into full sentences based
on aspects of interest—constructed via template-filling (Zhang et al., 2014)
or recurrent language models (Ni et al., 2019). Due to the unstructured nature
of these justifications, however, sentence-level justifications have not been
used for iteratively refining recommendations. In this work, justifications
take the form of specific aspects that a user is interested in (e.g. that a
song is poetic) (McAuley et al., 2012; Wu et al., 2019; Antognini et al.,
2020). In Section 4.2 we describe how we extract such aspects from user
reviews in large recommendation datasets.

Table 2. Notation used in this paper. Notation | Description
---|---
$\mathbf{K}^{U}\in\mathbb{R}^{|U|\times|K|}$ |  | User aspect frequency; $\mathbf{k}^{U}_{u,a}$ is how often
---
user $u$ mentioned aspect $a$ in their reviews.
$\mathbf{K}^{I}\in\mathbbm{1}^{|I|\times|K|}$ |  | Binary matrix; $\mathbf{k}^{I}_{i,a}$ is 1 if and only if aspect $a$
---
was used in any review of item $i$
$\gamma_{u},\gamma_{i}\in\mathbb{R}^{h}$ | Learned $h$-dimension user/item embeddings.
$\hat{x}_{u,i}\in\mathbb{R}$ | The predicted score of item $i$ for user $u$.
$\hat{k}_{u,i}\in\mathbbm{1}^{|K|}$ | Predicted justification (binary for all aspects).
$c^{t}_{u}\in\mathbb{R}^{|K|}$ |  | Cumulative critique vector representing the
---
user’s evolving opinion about each aspect.
$m^{t}_{u}\in\mathbbm{1}^{|K|}$ |  | The user critique vector at turn $t$. $m^{t}_{u,a}$ is 1 if
---
and only if the user critiqued $a$ at turn $t$.

###  2.2. Conversational Recommendation

Users often seek to make informed decisions around consumption, and
controllability of a recommendation system is linked to improved user
satisfaction (Parra and Brusilovsky, 2015). We thus turn to conversational
recommendation as a way to iteratively engage with a user to learn their
preferences with the goal of recommending a suitable item (Burke et al.,
1997). We view recommenders as domain experts engaging with human customers,
able to elicit user preferences and requirements and suggest appropriate items
in the course of the conversation.

In early interactive recommender systems, users were only able to give binary
“like” and “dislike” feedback without further explanation (Ko and Byun, 2002).
One line of research used such feedback to refine the search space for
retrieving desired images from the web (Jin and French, 2003). Biswas et al.
(2019) extended this approach to interactive product search. More recently,
multi-armed bandit approaches to conversational recommendation (Zhao et al.,
2013; Zou et al., 2020) leverage exploration-exploitation algorithms to
maximize the information learned from feedback at each turn.

Another line of work treats conversational recommendation as a form of task-
oriented dialog where users express opinions about specific aspects of an
item. At each turn, the user is either a) asked if they prefer a specified
aspect; or b) recommended an item (Christakopoulou et al., 2016). Self-
supervised bot-play has been explored as a way to train such conversational
dialog agents (Li et al., 2018; Kang et al., 2019), but such approaches
require Wizard-of-Oz style data (Dahlbäck et al., 1993) with humans playing
the role of expert and seeker. The quality of such dialog data depends heavily
on the domain knowledge and competence of crowd-workers, which makes it
unsuitable for complex domains. Zhang et al. (2018) uses templated dialog
forms and trains a model to ask about aspects that are most informative of the
user’s preferences. However, this forces the user to answer yes/no questions
and restricts their flexibility when giving feedback. Instead, we explore
conversational critiquing, where a user is presented with items and
justifications, and is able to give feedback regarding any aspect in the
justification.

###  2.3. Conversational Critiquing

Critiquing systems aim to help users incrementally construct their preferences
in a way that mimics how humans refine their preferences and constraints
depending on conversation context (Tversky and Simonson, 1993). Early
critiquing methods relied on constraint-based programming to iteratively
shrink the search space of items as users provided more critiques (Burke et
al., 1996).

More recently, Wu et al. (2019) introduced a critiquing model with
justifications via a list of natural language aspects mined from user reviews.
In this setting, users are able to interact with any aspect present in the
generated justification. Antognini et al. (2020) generate a single sentence of
explanation alongside the set of aspects, but still require users to interact
with the aspect set. Luo et al. (2020b) use a variational auto-encoder (VAE)
(Kingma and Welling, 2014) in place of the collaborative filtering model,
learning a bi-directional mapping function between user latent representations
and aspects they have expressed in reviews. Such models can generate high
precision justifications but have shown poor multi-turn recommendation
performance.

Latent Linear Critiquing (LLC) methods do not generate justifications and
instead allow users to critique any aspect from the vocabulary (Luo et al.,
2020a; Li et al., 2020). After training a matrix factorization model to
predict ratings, these models then learn a linear regressor to recover user
embeddings from their historical aspect usage frequency. A linear programming
problem is then solved to weight a user’s critiques during each turn of the
conversation, which we observe to take an order of magnitude longer than VAE-
based methods and our own. Furthermore, while LLC assumes that user
preferences are fully explained by their review texts, recent studies have
shown that this assumption may be unfounded (Sachdeva and McAuley, 2020).

In Table 1 we compare our approach in context of recent frameworks for
training dialog agents for conversational recommendation and conversational
critiquing agents.

Figure 2. In (latent) conversational critiquing, user feedback about aspects
($c^{0},c^{1}$) modifies our prior latent user preference vector
$\gamma_{u}^{0}$ to bring it closer to the target item embedding.

##  3\. Model

Our model consists of three sections, as seen in Figure 3:

  1. (1)

A matrix factorization recommender model $M_{\text{rec}}$ that learns to embed
users and items in an $h$-dimensional latent space;

  2. (2)

A justification head $M_{\text{just}}$ that predicts the aspects of an item
toward which the user holds preferences;

  3. (3)

A critiquing function $f_{\text{crit}}$ that modifies a user’s preference
embedding based on user feedback about specific aspects.

Our model supports multi-turn critiquing as shown in Figure 2. At each turn of
a conversation, a user may provide explicit feedback about aspects they
dislike about the current set of recommendations in the form of a critique
($c^{t}$). The critiquing function $f_{\text{crit}}$ then uses this critique
to modify our latent user representation $\gamma_{u}$ in order to bring it
closer to the user’s target item.

###  3.1. Base Recommender System

Our method can be applied to any recommender system $M_{\text{rec}}$ that
learns user and item representations. We demonstrate its effectiveness using
two popular methods based on matrix factorization and linear recommendation.

_Bayesian Personalized Ranking_ (BPR) (Rendle et al., 2009) is a matrix
factorization recommender system that seeks to decompose the interaction
matrix $\textbf{R}\in\mathbb{R}^{|U|\times|I|}$ into user and item
representations (Koren et al., 2009). BPR optimizes a ranked list of items
given implicit feedback (a set of items with which a user has recorded a
binary interaction).

We learn $h$-dimensional user and item embeddings ($\gamma_{u}^{\text{MF}}$,
$\gamma_{i}^{\text{MF}}$), computing the score via the inner product:
$\hat{x}_{u,i}=\langle\gamma_{u}^{\text{MF}},\gamma_{i}^{\text{MF}}\rangle$.
At training time, the model is given a user $u$, observed item $i\in
I_{u}^{+}$, and unobserved item $j\in I_{u}^{-}$. We maximize the likelihood
that the user prefers the observed item $i$ to the unobserved item $j$:

(1) $\displaystyle\mathcal{L}_{R}\ =\ P(i>_{u}j|\Theta)\ =\
\sigma(\hat{x}_{u,i}-\hat{x}_{u,j})$

where $\sigma$ represents the sigmoid function $\frac{1}{1+e^{-x}}$.

_Projected Linear Recommendation_ (PLRec) is an SVD-based method to learn low-
rank representations for users and items via linear regression (Sedhain et
al., 2016). The PLRec objective minimizes the following:

(2) $\displaystyle\operatorname*{arg\,min}_{W}\sum_{u}\parallel
r_{u}-r_{u}VW^{T}\parallel^{2}_{2}+\Omega(W)$

where $V$ is a fixed matrix obtained by taking a low-rank SVD approximation of
$\mathbf{R}$ such that $\mathbf{R}=U\Sigma V^{T}$, and $W$ is a learned
embedding matrix. We thus obtain an $h$-dimensional user embedding
$\gamma_{u}^{\text{MF}}=r_{u}V$ and $h$-dimensional item embedding
$\gamma_{i}^{\text{MF}}=W_{i}$.

Figure 3. The proposed model architecture. Given a user, items, and critique
vector, our model encodes the critique $M_{\text{AE}}(c^{t}_{u})$ and fuses it
with the user embedding $\gamma^{\text{MF}}_{u}$. The fused user
representation $\gamma_{u}$ and item representation $\gamma_{i}$ are then used
to predict the justification and score items.

###  3.2. Generating Justifications

Our justification model $M_{\text{just}}$ consists of an aspect prediction
head: a fully connected network with two $h$-dimensional hidden layers that
predicts a score $s_{u,i,a}$ for each aspect $a$. This model takes as input
the sum of the learned user and item embeddings ($\gamma_{u},\gamma_{i}$).

At training time, we incorporate an aspect prediction loss $\mathcal{L}_{A}$
by computing the binary cross entropy (BCE) for each aspect:

(3) $\displaystyle\mathcal{L}_{A}$
$\displaystyle=-\frac{1}{|A|}\sum_{a=0}^{|A|}\mathbf{k}^{I}_{i,a}\cdot\log
p_{u,i,a}+(1-\mathbf{k}^{I}_{i,a})\cdot\log(1-p_{u,i,a})$

where $p_{u,i,a}=\sigma(s_{u,i,a})$ represents the likelihood of user $u$
caring about aspect $a$ in context of item $i$. At inference time, we again
compute the likelihood for each aspect ($p_{u,i,a}=\sigma(s_{u,i,a})$) and
sample from the Bernoulli distribution with $p_{u,i,a}$ to determine which
aspects $a$ appear in the justification.

###  3.3. Encoding Aspects

We posit that the user’s latent representation can be partially explained by
their written reviews. Thus, we jointly learn an aspect encoder
$M_{\text{AE}}$ alongside our recommendation model. This takes the form of a
linear projection from the aspect space to the user preference space:
$M_{\text{AE}}(c^{t}_{u})=W^{T}c^{t}_{u}+b$, where
$c^{t}_{u}\in\mathbb{Z}^{|K|}$ is the critique vector representing the
strength of a user’s preference for each aspect. We then fuse this aspect
encoding with the latent user embedding from $M_{\text{rec}}$ to form the
final user preference vector $\gamma_{u}$:
$\gamma_{u}=f(\gamma_{u}^{\text{MF}},M_{\text{AE}}(c^{t}_{u}))$

For the BPR-based model, we use $f(a,b)=a+b$ as a fusion function, and for the
PLRec-based model, we use $f(a,b)=\frac{a+b}{2}$. During training, the aspect
encoder takes in the user’s aspect history: $c^{t}_{u}=\textbf{k}^{U}_{u}$.

Recommendation and Justification models $M_{\text{rec}},M_{\text{just}}$;

Critique fusion function $f_{\text{crit}}$;

Seeker model $M_{\text{seeker}}$;

for  _each user $u$_ do

for  _goal item $g\in I_{u}^{+}$ (Evaluation set)_ do

initialize loss $\mathcal{L}$;

initialize $\gamma^{1}_{u}$ from $M_{\text{rec}}$;

for  _turn $t\in range(1,T)$_ do

compute scores $\hat{x}^{t}_{u,i}=M_{\text{rec}}(\gamma^{t}_{u},i)\ \forall\
i\in I$;

$\mathcal{L}\leftarrow\mathcal{L}+\delta^{t}\cdot\mathcal{L}_{\text{CE}}(g,\hat{x}^{t}_{u,i})$;

recommend item $\hat{i}^{t}=\operatorname*{arg\,max}_{i}{\hat{x}^{t}_{u,i}}$;

if  _$\hat{i}^{t}=g$_ then

break session with success;

end if

generate justification
$\hat{k}_{u,\hat{i}^{t}}=M_{\text{just}}(\gamma^{t}_{u},\gamma_{\hat{i}^{t}})$;

$M_{\text{seeker}}$ critiques justification: $c^{t}_{u}$;

$\gamma^{t+1}_{u}\leftarrow f_{\text{crit}}(\gamma^{t}_{u},c^{t}_{u})$;

end for

end for

end for

Algorithm 1 Bot play framework for fine-tuning conversational recommenders.

###  3.4. Training

To train our BPR-based model, we jointly optimize each component. Each
training example consists of a user $u$, an observed item $i\in I^{+}_{u}$
that the user has interacted with, and an unobserved item $j\in I^{-}_{u}$
that the user has not rated. We predict scores for items $i$ and $j$:

(4)
$\displaystyle\hat{x}_{u,i}=\langle\gamma_{u},\gamma_{i}\rangle=\langle\gamma_{u}^{\text{MF}}+M_{\text{AE}}(\textbf{k}^{U}_{u}),\gamma_{i}\rangle$

We first compute the BPR loss (see Section 3.1) with predicted scores
$\hat{x}_{u,i}$ and $\hat{x}_{u,j}$. We add the aspect prediction loss, scaled
by a constant $\lambda_{\text{KP}}$ to the ranking loss for our training
objective: $\mathcal{L}=\lambda_{\text{KP}}\mathcal{L}_{A}-\mathcal{L}_{R}$.
We find empirically that $\lambda_{\text{KP}}\in\\{0.5,1.0\\}$ works well.

To train our PLRec-based model, we follow Luo et al. (2020a) and separately
optimize $M_{\text{rec}}$, $M_{\text{just}}$, and $M_{\text{AE}}$. The user
and item embeddings are learned via eq. 2. We solve the following linear
regression problem to optimize $M_{\text{AE}}$:

(5)
$\displaystyle\operatorname*{arg\,min}_{W,b}\sum_{u}\parallel\gamma^{\text{MF}}_{u}-M_{\text{AE}}(\mathbf{k}^{U}_{u})\parallel^{2}_{2}+\Omega(W)$

Finally, we optimize the aspect prediction (justification) loss
$\mathcal{L}_{A}$ to train the justification head.

###  3.5. Conversational Critiquing with Our Models

To perform conversational critiquing with a model trained using our framework,
we adapt the latent critiquing formulation from Luo et al. (2020a), as shown
in Figure 1. Each conversation with a user $u$ consists of multiple turns. At
each turn $t$, the system assigns scores $\hat{x}^{t}_{u,i}$ for all candidate
items $i$, and presents the user with the highest scoring item $\hat{i}$. The
system also justifies its prediction with a set of predicted aspects
$\hat{k}^{t}_{u,i}$. The user may either accept the recommended item (ending
the conversation) or critique an aspect from the justification:
$a\in\\{a|\hat{k}_{u,i,a}=1\\}$.

Given a user critique, the system modifies the predicted scores for each item
and presents the user with a new item and justification:

(6) $\displaystyle\hat{x}^{t+1}_{u,i}$
$\displaystyle=M_{\text{rec}}(\hat{\gamma}_{u}^{t+1},i)$ (7)
$\displaystyle\hat{k}^{t+1}_{u,i}$
$\displaystyle=M_{\text{just}}(\hat{\gamma}_{u}^{t+1},i)$ (8)
$\displaystyle\hat{\gamma}_{u}^{t+1}$ $\displaystyle\leftarrow
f_{\text{crit}}(\hat{\gamma}_{u}^{t},c^{t}_{u})$

Effectively, a user critique modifies our prior for the user’s preferences; we
then re-rank the items presented to the user.

At inference time, $c^{t}_{u}$ is the cumulative critique vector, initialized
with the user’s aspect history:

(9) $\displaystyle c^{t}_{u}$
$\displaystyle=c^{t-1}_{u}-\max(\textbf{k}^{U}_{u},1)\odot m^{t}_{u};\quad
c^{0}_{u}=\textbf{k}^{U}_{u}$

where $\odot$ is element-wise multiplication. We use
$\max(\textbf{k}^{U}_{u},1)$ as the critique should match the strength of the
user’s previous opinion on the aspect—otherwise the encoding may have a small
magnitude. Even if a user has not mentioned an aspect in their previous
reviews, the $\max$ ensures a non-zero effect from each critique.

###  3.6. Learning to Critique via Bot Play

We propose a framework for critiquing via bot play that simulates
conversations when provided just a known set of user reviews. We first pre-
train our expert model (recommender model, justification model, and aspect
encoder). A seeker model $M_{\text{seeker}}$ is pre-trained via a simple user
prior: when provided with a known target item and justification, it selects
the most popular aspect present in the justification $\hat{k}_{u,\hat{i}}$ but
not the target’s historical aspects $\textbf{k}^{I}_{i}$ to critique.

For each training example (user $u$ and a goal item they have reviewed $g$),
we allow the expert and seeker models to converse with the goal of
recommending the goal item. We fine-tune the expert by maximizing its reward
(minimizing its loss) in the bot-play game (Algorithm 1). We end the dialog
after the goal item is recommended or a maximum session length of $T=10$ turns
is reached. We define the expert’s loss as the cross entropy loss of
recommendation scores per turn:

(10) $\displaystyle\mathcal{L}^{\text{expert}}$
$\displaystyle=\sum_{t}^{T}\delta^{t-1}\cdot\mathcal{L}_{\text{CE}}(g,\hat{x}^{t}_{u,i})$

where $\delta$ is a discount factor111We use a discount factor of $\delta=0.9$
to encourage successfully recommending the goal item at earlier turns.
$\mathcal{L}_{\text{CE}}(g,\hat{x}^{t}_{u,i})$ is the cross entropy loss
between predicted scores and the goal item:

(11) $\displaystyle\mathcal{L}_{\text{CE}}(g,\hat{x}^{t}_{u,i})$
$\displaystyle=-\sum_{i\in I}P(i)\log_{2}Q(i);\quad
Q(i)=\frac{e^{\hat{x}^{t}_{u,i}}}{\sum^{I}_{j}e^{\hat{x}^{t}_{u,j}}}$

where $P(i)$ is 1 if $g=i$ and 0 otherwise. As the cross-entropy loss is
continuous, we optimize the reward for each conversation ($u$, $i$).

##  4\. Experimental Setting

To train our initial model, we select hyperparameters via AUC on the
validation set. We select hyperparameters for bot-play fine-tuning by
evaluating the success rate at 1 (SR@1) on the validation set. We train each
model once, with three evaluation runs per experimental setting. For baseline
models, we re-used the authors’ code. We include additional training details
in the supplementary materials. We will make our code available upon
publication.

Table 3. Descriptive statistics of datasets, including average unique aspects expressed in reviews per item and user. | Users | Items | Reviews | Asp. | Asp./Item | Asp./User
---|---|---|---|---|---|---
Books | 13,889 | 7,649 | 654,975 | 75 | 27.0 | 25.0
Beer | 6,369 | 4,000 | 935,524 | 75 | 60.2 | 54.6
Music | 5,635 | 4,352 | 119,081 | 80 | 20.0 | 16.5

###  4.1. Datasets

We evaluate the quantitative performance of our model using three real-world,
publicly available recommendation datasets: Goodreads Fantasy (Books) (Wan and
McAuley, 2018), BeerAdvocate (Beer) (McAuley et al., 2012), and Amazon CDs &
Vinyl (Music) (He and McAuley, 2016; McAuley et al., 2015)—each with over 100K
reviews and ratings. We keep only reviews with positive ratings, setting
thresholds of $t>4.0$ for Beer and Music and $t>3.5$ for Books. We partition
each dataset into 50% training, 20% validation, and 30% test splits. Dataset
statistics are shown in Table 3.

###  4.2. Aspect Extraction

Our datasets do not contain pre-existing aspects, so we follow the pipeline of
(Wu et al., 2019) to extract subjective aspects from user reviews:

  1. (1)

Extract lists of high-frequency unigrams and bigrams (nouns and adjective
phrases only) from all user reviews;

  2. (2)

Prune the bigram keyphrase list using a Pointwise Mutual Information (PMI)
threshold, ensuring aspects are statistically unlikely to have randomly co-
occurred;

  3. (3)

Represent reviews as sparse binary vectors indicating whether each aspect was
expressed in the review.

Aspects describe a wide range of qualities; for beers, users commonly describe
the malt (e.g. roasted) and taste (e.g. citrus). For music, aspects range from
perceived genres (e.g. techno) to emotions (e.g. soulful). Users describe
books by reacting to character descriptions (e.g. strong female) and settings
(e.g. realistic)..

Recommendation and Justification models $M_{\text{rec}},M_{\text{just}}$;

Critique fusion function $f_{\text{crit}}$;

for  _each user $u$_ do

for  _goal item $g\in I_{u}^{+}$ (Evaluation set)_ do

initialize $\gamma^{1}_{u}$ from $M_{\text{rec}}$;

for  _turn $t\in range(1,T)$_ do

compute scores $\hat{x}^{t}_{u,i}=M_{\text{rec}}(\gamma^{t}_{u},i)\ \forall\
i\in I$;

recommend item $\hat{i}^{t}=\operatorname*{arg\,max}_{i}{\hat{x}^{t}_{u,i}}$;

generate justification
$\hat{k}^{t}_{u,\hat{i}}=M_{\text{just}}(\gamma^{t}_{u},\gamma_{\hat{i}^{t}})$;

if  _$\hat{i}^{t}=g$_ then

break session with success;

end if

user critiques justification: $c^{t}_{u}$;

$\gamma^{t+1}_{u}\leftarrow f_{\text{crit}}(\gamma^{t}_{u},c^{t}_{u})$;

end for

end for

end for

return _average success rate & length_

Algorithm 2 User Simulation Evaluation

###  4.3. Multi-Step Critiquing

Following prior work on conversational critiquing (Luo et al., 2020a; Li et
al., 2020), we simulate multi-step recommendation dialogs to assess model
performance. We randomly sample 500 user-item interactions from the test set
to conduct user simulations following Algorithm 2 for each user $u$ and goal
item $g$. At each turn, we recommend an item $\hat{i}^{t}$ to the user
alongside a set of aspects $\hat{k}_{u,\hat{i}^{t}}$. If the goal item is not
recommended, the user will critique an aspect $a$ from the justification that
is inconsistent with the goal item aspects: $a\in\\{a\ |\
\hat{k}^{t}_{u,\hat{i}^{t},a}=1\ \&\ \textbf{k}^{I}_{g,a}=0\\}$ We set a
maximum session limit of $T=10$ turns.

To evaluate how our models behave with different user behaviors, we simulate
each observation with three different critique selection strategies (Li et
al., 2020):

  * •

Random: We assume the user randomly chooses an aspect. This assumes no prior
knowledge on the part of the user.

  * •

Pop: We assume the user selects the most popular aspect used across all
training reviews.

  * •

Diff: We assume the user selects the aspect that deviates most from the goal
item reviews. In simulations, we select the aspect with the largest frequency
differential between the goal item and current item:
$\operatorname*{arg\,max}_{a}(\textbf{k}^{I}_{\hat{i}^{t},a}-\textbf{k}^{I}_{g,a})$

In all critiquing settings, a user may not critique the same aspect multiple
times in a session, and any recommended items are removed from consideration
in the following turns.

Figure 4. Success Rate @ N (% dialogs where target item rank $\leq$ N) across
datasets and user models. BPR-Bot (brown triangle) and PLRec-Bot (pink circle)
out-perform baselines (dashed) in all settings.

###  4.4. Candidate Algorithms

As our method can be applied to any base recommender system $M_{\text{rec}}$,
we apply our framework to train models based on BPR and PLRec (see Section
3.1)—BPR-Bot and PLRec-Bot, respectively.

We assess Latent Linear Critiquing (LLC) baselines, which embed critique
vectors $c^{t}_{u}$ in the same $h$-dimensional space as the latent user
representation $\gamma_{u}$. $f_{\text{crit}}$ is defined as a weighted sum of
the embedding for each critiqued aspect, alongside the original user
preference vector. UAC (Luo et al., 2020a) averages the initial user embedding
and all critiqued aspect embeddings. BAC (Luo et al., 2020a) first averages
critiqued aspects, and then averages the result with the initial user
embedding. LLC-Score (Luo et al., 2020a) learns the weights via a linear
program maximizing the posterior rating differences between items containing
critiqued aspects and those without. Instead of directly optimizing the
scoring margin, LLC-Rank (Li et al., 2020) minimizes the number of ranking
violations. These models cannot generate justifications; we binarize the
historical aspect frequency vector for the item
($\textbf{k}^{I}_{u,\hat{i}^{t}}$) as a justification at each turn. We compare
against these models to evaluate whether generating personalized
justifications can improve critiquing.

We also compare against a state-of-the-art variational conversational
recommender, CE-VAE (Luo et al., 2020b)—an improvement on the Wu et al. (2019)
justified critiquing model—which jointly learns to recommend and justify. CE-
VAE learns a VAE with a bidirectional mapping between critique vectors and the
user latent preference space. We compare our models to CE-VAE to assess how
justification quality impacts multi-turn critiquing performance.

##  5\. Experiments

RQ1: Can our framework enable multi-step critiquing? To measure multi-step
critiquing performance, we assess the average success rate and session length
following Luo et al. (2020a). Success rate measures the percentage of sessions
in which the target item reaches rank N, and session length measures the
average length of sessions with a limit of 10 iterations. Success rates and
session lengths for each dataset and user behavior model are shown in Figure 4
and Figure 5, respectively.

Figure 5. Avg. number of turns before the target item reaches rank N, across
datasets and user models. BPR-Bot (brown triangle) and PLRec-Bot (pink circle)
promote target items faster than baselines (dashed), especially for low N.

Our models are fine-tuned via bot-play with a seeker model that assumes one
particular user behavior: popularity-based critique selection. As such, we
expect it to perform better in the Pop user setting. However, BPR-Bot and
PLRec-Bot succeeds at a higher rate in fewer turns than baselines under all
user settings—including random aspect critiquing, which assumes no prior on
user behavior.

_Variational Baseline._ Despite its strong first-turn recommendation
performance and high-fidelity justifications, CE-VAE is out-performed by our
models in all nine settings across all metrics. This supports our observation
that the training method to learn a bi-directional mapping between latent user
preferences and a justification causes a trade-off between justification
quality and critiquing ability.

_Linear Baselines._ We further observe that linear critiquing models (UAC,
BAC, LLC-Score, and LLC-Rank) perform poorly on multi-step critiquing,
especially when trying to find the goal item outright ($N=1$). This confirms
our observation that the method of co-embedding aspect critiques with learned
user latent preferences ignores the existence of user preferences not
explained by review text. This additionally suggests that generating
personalized justifications helps users more effectively choose aspects to
critique.

Figure 6. Success Rate @ N (% dialogs where target item rank $\leq$ N),
comparing bot-play methods (orange) against non-bot-play ablations (blue).
Bot-play fine-tuning improves target item ranking across datasets compared to
the ablation, for both BPR-Bot (crosses) and PLRec-Bot (circles).

In general, the large item space makes it difficult for critiquing models to
reach the goal item within the turn limit, with the best model reaching the
goal item in only 6-15% of sessions. This suggests that practical
conversational critiquing systems may benefit from constraint-based filtering
as well as starting the session from an initial set of user requirements—while
users rarely enter a conversation knowing their full preference set (Pu and
Faltings, 2000), they often start with a limited set of broad requirements
(e.g. when buying a car, they want an SUV or a coupe). We demonstrate in RQ3
that our model can be combined with constraint-based query refinement to
quickly reach significantly higher success rates.

RQ2: Can our bot-play framework improve multi-step critiquing performance? We
next compare BPR-Bot (left) and PLRec-Bot (right) in Figure 6 against ablated
versions that were trained using the first step of our framework but _not_
fine-tuned via bot-play. For clarity, we display only results using the Pop
user behavioral model, as we observe the same trends with the Random and Diff
user models. In domains with relatively high aspect occurrence across reviews
(Books, Beer), we observe that bot-play confers a 3-6% improvement in success
rate for various N. This demonstrates that we can effectively train
conversational recommender systems using our bot-play framework using domains
with rich user reviews in lieu of crowd-sourced dialog transcripts.

Figure 7. Hit rate by turn for query refinement models on each dataset with
multi-step critiquing up to 10 turns.

In domains with more sparse coverage of subjective aspects (i.e. Music), we
observe lower improvement when using bot-play. Here, our model may not
encounter sufficient examples of rare aspects being critiqued. In future work,
we will explore methods to add noise to our user model to ensure that the bot-
play process encounters more rare aspects. We will also investigate additional
losses for bot-play, including ranking losses instead of cross entropy.

We confirm that our method is model-agnostic, as it improves conversational
recommendation success rates for both the matrix factorization-based (BPR) and
linear (PLRec) recommender systems. We also observe that models with a higher
latent dimensionality ($h\in[50,400]$ for PLRec-Bot vs. $h=10$ for BPR-Bot)
benefit more from bot-play, suggesting that our method effectively learns to
navigate complex user preference spaces.

RQ3: Can our models be effectively combined with query refinement? So far, we
have assumed that users provide _soft_ feedback via critiques: even if a user
has critiqued aspect $a$ during a session, future suggested items may still
contain aspect $a$. This assumption holds for some aspects: for example, even
if previous users mentioned that a song was dispassionate, a user may find it
emotional and enjoyable. However, in a real-world setting that user may reject
the suggestion after reading the reviews. Thus, we experiment with treating
critiques as hard feedback: if a user critiques some aspect $a$, we prune all
candidate items whose reviews mention $a$. We compare three models in this
setting, with the turn-0 ranked list of candidate items initialized from BPR-
Bot.

The Query baseline model suggests one item per turn and asks the user whether
they like aspect $a$. If the user answers yes, we prune all candidate items
whose reviews have not expressed $a$: $I^{t+1}\leftarrow\\{i\ \forall\ i\in
I^{t}|\textbf{k}^{I}_{i,a}=1\\}$. Otherwise, we prune all candidates whose
reviews have expressed the aspect: $I^{t+1}\leftarrow\\{i\ \forall\ i\in
I^{t}|\textbf{k}^{I}_{i,a}=0\\}$. At each turn, we pick the aspect that most
evenly divides the remaining candidate items:
$\operatorname*{arg\,min}_{a}||I^{+}_{a}|-|I^{-}_{a}||$ Effectively, we
perform binary search over our candidate space, and expect to find the target
item within $\log{|I|}$ turns.

In the Filter model, we suggest an item alongside a generated justification
per turn. When a user critiques aspect $a$, we prune candidate items whose
reviews have expressed $a$: $I^{t+1}\leftarrow\\{i\ \forall\ i\in
I^{t}|\textbf{k}^{I}_{i,a}=0\\}$. We extend this model via our learned
critiquing function $f_{\text{crit}}$ to further modify the user preference
vector and re-compute scores for the remaining items. This hybrid Filter+Re-
rank model then re-ranks the remaining candidate items for the next turn. We
conduct user simulations with the Pop user model following Algorithm 2, and
plot the success rate by turn—rate of achieving the goal item $g$ at or before
turn $t$—in Figure 7.

Binary queries are guaranteed to eventually find the answer, but the queried
aspect may not be related to suggested items. By allowing the user to provide
negative critiques, we can rapidly reduce the search space at early turns.
Across domains the success rate rises much faster in the first 6-10 turns for
Filter and Filter+Re-rank compared to binary querying. Re-ranking after
filtering improves performance across domains, suggesting that we have learned
how user critiques relate to their latent preferences for other aspects.

Table 4. Conversation-level human evaluation via ACUTE-EVAL. Win (W) and Loss
(L) percentages are reported while ties are not. All results statistically
significant with $p<0.05$.

BPR-Bot | Useful | Informative | Knowledgeable | Adaptive
---|---|---|---|---
vs | W | L | W | L | W | L | W | L
Ablation | 78 | 10 | 73 | 11 | 68 | 15 | 85 | 5
CE-VAE | 83 | 9 | 74 | 10 | 63 | 16 | 81 | 8
PLRec-Bot | Useful | Informative | Knowledgeable | Adaptive
vs | W | L | W | L | W | L | W | L
Ablation | 86 | 5 | 78 | 7 | 74 | 8 | 81 | 9
CE-VAE | 87 | 7 | 79 | 11 | 77 | 12 | 83 | 10

For the Beer and Books domains, the filtering approach reaches higher success
rates compared to binary querying same high success rate within the session
turn limit (70.7% vs. 69.7% and 57.0% vs. 55.2%, respectively). We see less of
a benefit in the Music domain. Relative aspect sparsity may play a role: per
Table 3, only 25% of possible aspects are expressed for the average item.
There also exists a longer tail of aspects expressed only for a small set of
items in Music compared to the other datasets. As such, user critiques prune
fewer candidate items on average in Music.

Our bot-play framework can be easily adapted to train models incorporating
hard critiquing constraints by pruning candidate items. One possible extension
involves masking the cross entropy (fine-tuning) loss to only adjust the
scores of non-pruned items, setting pruned item scores to a large negative
value: $\hat{x}_{u,i}=-1e15\ \forall\ i\in I^{+}_{a}$. We also wish to explore
fine-tuning with a ranking loss during bot-play, to encourage the model to
rank items containing a critiqued aspect $i\in I^{+}_{a}$ below those without.

##  6\. Human Study

Human Evaluation To assess the quality of the simulated conversations during
bot-play, we conduct human evaluations with 100 samples. Following ACUTE-EVAL
(Li et al., 2019), we conduct a comparative evaluation of each sample
conversation on four criteria: which agent seems more useful, informative,
knowledgeable and adaptive. We compare each bot-play model (BPR-Bot and PLRec-
Bot) against an ablative version (with no bot-play fine-tuning) and the best
baseline model (CE-VAE).

Each sample is evaluated by three annotators. We observe substantial (Landis
and Koch, 1977) inter-annotator agreement, with Fleiss Kappa (Fleiss and
Cohen, 1973) of 0.67, 0.79, 0.73, and 0.60 for the usefulness,
informativeness, knowledgeable, and adaptiveness criteria, respectively.

Table 5. Turn- and conversation-level feedback from cold-start user study. Statistically significant results are bolded. | Useful | Informative | Adaptive | Would use
---|---|---|---|---
Ablation | 0.67$\pm$0.24 | 0.75$\pm$0.21 | 0.64$\pm$0.27 | 41%
BPR-Bot | 0.79$\pm$0.24 | 0.88$\pm$0.18 | 0.78$\pm$0.23 | 69%

BPR-Bot and PLRec-Bot are judged to be significantly more informative and
knowledgeable compared to ablative models and CE-VAE, showing that our
justification module accurately predicts aspects of a recommended item. We
design the usefulness and adaptiveness criteria to capture how our framework
aids the user in achieving their conversational goal (i.e. recommending the
most relevant item within a minimum number of turns). Compared to the
alternatives, models trained under our bot-play framework are judged to be
more useful and adapt their recommendations in a manner more consistent with
critiques.

Our framework allows us to train conversational agents that are useful and
engaging for human users: evaluators overwhelmingly judged the models trained
via bot-play to be more useful, informative, knowledgeable, and adaptive
compared to CE-VAE and ablated variants.

Cold-Start User Study We conduct a user study using items and reviews from the
Books dataset to evaluate our model’s ability to provide useful conversational
recommendations in real-time. We recruited 32 real human users to interact
with our BPR-Bot recommender and another 32 to interact with the ablation
model (no conversational fine-tuning). As evaluators do not correspond to
users in our training data, we initialize each conversation with the average
of all learned latent user representations.

At each turn, the user is presented with the three top-ranked items and their
justifications (list of aspects), and is allowed to critique multiple aspects.
On average, users critiqued two aspects per turn—this suggests that when
training conversational models we should assume multiple critiques at each
turn.

We evaluate our systems following Li et al. (2019): at each turn, we ask our
users if the generated justifications are _informative_ , _useful_ in helping
to make a decision, and whether our system _adapted_ its suggestions in
response to the user’s feedback. We provide four options for each question:
yes, weak-yes, weak-no, and no, mapping these values to a score between 0 and
1 (Kayser et al., 2021). We display the normalized aggregated score for each
question in Table 5. We find that BPR-Bot significantly out-scores the
ablation model in all three metrics ($p<0.01$), showing that fine-tuning our
model on a bot-play framework instills a stronger ability to respond to
techniques and provide meaningful justifications—even for unseen users.

At the end of a conversation, we additionally ask the user how frequently (if
at all) they would choose to engage with our conversational agent in their
daily life. 69% of users indicated they would “often” or “always” use BPR-Bot
to find books, compared to 41% of users for the ablation model. We thus find
that fine-tuning our model via bot-play also makes it significantly ($p<0.05$)
more useful for new users.

##  7\. Conclusion

In this work, we aim to develop conversational agents for recommendation that
engage with users following common modes of human dialog: justifying why
suggestions were made and incorporating feedback about certain aspects of an
item to provide better recommendations at the next turn. We present a
framework for training conversational recommenders in this modality via self-
supervised bot-play. Our framework is model-agnostic and allows conversational
recommenders to be trained on any domain with review data. We use two popular
underlying recommender systems to train the BPR-Bot and PLRec-Bot
conversational agents using our framework, demonstrating quantitatively on
three datasets that our models 1) offer superior multi-turn recommendation
performance compared to current state-of-the-art methods; 2) can be
effectively combined with query refinement techniques to quickly converge on
suitable items; and 3) can iteratively refine suggestions in real-time, as
shown in user studies. In future work, we aim to adapt our framework to
natural language critiques (i.e. complete utterances), allowing users to
freely express their feedback in a less restrictive way.

## References

  * (1)
  * Antognini and Faltings (2021) Diego Antognini and Boi Faltings. 2021.  Fast Multi-Step Critiquing for VAE-based Recommender Systems.  _CoRR_ abs/2105.00774 (2021).  arXiv:2105.00774
  * Antognini et al. (2020) Diego Antognini, Claudiu Musat, and Boi Faltings. 2020\.  Interacting with Explanations through Critiquing.  _CoRR_ abs/2005.11067 (2020).  arXiv:2005.11067
  * Biswas et al. (2019) Ari Biswas, Thai T. Pham, Michael Vogelsong, Benjamin Snyder, and Houssam Nassif. 2019.  Seeker: Real-Time Interactive Search. In _KDD_.  https://doi.org/10.1145/3292500.3330733
  * Burke et al. (1996) Robin D. Burke, Kristian J. Hammond, and Benjamin C. Young. 1996.  Knowledge-Based Navigation of Complex Information Spaces. In _AAAI_.
  * Burke et al. (1997) Robin D. Burke, Kristian J. Hammond, and Benjamin C. Young. 1997.  The FindMe Approach to Assisted Browsing.  _IEEE Expert_ 12, 4 (1997), 32–40.  https://doi.org/10.1109/64.608186
  * Chen and Pu (2012) Li Chen and Pearl Pu. 2012\.  Critiquing-based recommenders: survey and emerging trends.  _UMUAI_ (2012).  https://doi.org/10.1007/s11257-011-9108-6
  * Christakopoulou et al. (2016) Konstantina Christakopoulou, Filip Radlinski, and Katja Hofmann. 2016\.  Towards Conversational Recommender Systems. In _KDD_.  https://doi.org/10.1145/2939672.2939746
  * Dahlbäck et al. (1993) Nils Dahlbäck, Arne Jönsson, and Lars Ahrenberg. 1993\.  Wizard of Oz studies: why and how. In _IUI_.  https://doi.org/10.1145/169891.169968
  * Fleiss and Cohen (1973) Joseph L Fleiss and Jacob Cohen. 1973.  The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability.  _Educational and psychological measurement_ 33, 3 (1973), 613–619.
  * He and McAuley (2016) Ruining He and Julian J. McAuley. 2016.  Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering. In _WWW_.  https://doi.org/10.1145/2872427.2883037
  * Jin and French (2003) Xiangyu Jin and James C. French. 2003.  Improving image retrieval effectiveness via multiple queries. In _MMDB_.  https://doi.org/10.1145/951676.951692
  * Kang et al. (2019) Dongyeop Kang, Anusha Balakrishnan, Pararth Shah, Paul A. Crook, Y-Lan Boureau, and Jason Weston. 2019.  Recommendation as a Communication Game: Self-Supervised Bot-Play for Goal-oriented Dialogue. In _EMNLP-IJCNLP_.  https://doi.org/10.18653/v1/D19-1203
  * Kayser et al. (2021) Maxime Kayser, Oana-Maria Camburu, Leonard Salewski, Cornelius Emde, Virginie Do, Zeynep Akata, and Thomas Lukasiewicz. 2021\.  e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks.  _CoRR_ abs/2105.03761 (2021).  arXiv:2105.03761
  * Kingma and Welling (2014) Diederik P. Kingma and Max Welling. 2014.  Auto-Encoding Variational Bayes. In _ICLR_.
  * Ko and Byun (2002) ByoungChul Ko and Hyeran Byun. 2002.  Probabilistic Neural Networks Supporting Multi-Class Relevance Feedback in Region-Based Image Retrieval. In _ICPR_.  https://doi.org/10.1109/ICPR.2002.1047418
  * Koren et al. (2009) Yehuda Koren, Robert M. Bell, and Chris Volinsky. 2009\.  Matrix Factorization Techniques for Recommender Systems.  _Computer_ 42, 8 (2009), 30–37.  https://doi.org/10.1109/MC.2009.263
  * Landis and Koch (1977) J. Richard Landis and Gary G. Koch. 1977.  The Measurement of Observer Agreement for Categorical Data.  _Biometrics_ 33, 1 (1977), 159–174.  http://www.jstor.org/stable/2529310
  * Li et al. (2020) Hanze Li, Scott Sanner, Kai Luo, and Ga Wu. 2020\.  A Ranking Optimization Approach to Latent Linear Critiquing for Conversational Recommender Systems. In _RecSys_.  https://doi.org/10.1145/3383313.3412240
  * Li et al. (2019) Margaret Li, Jason Weston, and Stephen Roller. 2019\.  ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons.  arXiv:1909.03087 [cs.CL]
  * Li et al. (2018) Raymond Li, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. 2018.  Towards Deep Conversational Recommendations. In _NeurIPS_.
  * Liu et al. (2020) Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2020.  On the Variance of the Adaptive Learning Rate and Beyond. In _ICLR_.
  * Luo et al. (2020a) Kai Luo, Scott Sanner, Ga Wu, Hanze Li, and Hojin Yang. 2020a.  Latent Linear Critiquing for Conversational Recommender Systems. In _WWW_.  https://doi.org/10.1145/3366423.3380003
  * Luo et al. (2020b) Kai Luo, Hojin Yang, Ga Wu, and Scott Sanner. 2020b.  Deep Critiquing for VAE-based Recommender Systems. In _SIGIR_.  https://doi.org/10.1145/3397271.3401091
  * McAuley et al. (2012) Julian J. McAuley, Jure Leskovec, and Dan Jurafsky. 2012\.  Learning Attitudes and Attributes from Multi-aspect Reviews. In _ICDM_.  https://doi.org/10.1109/ICDM.2012.110
  * McAuley et al. (2015) Julian J. McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. 2015.  Image-Based Recommendations on Styles and Substitutes. In _SIGIR_.  https://doi.org/10.1145/2766462.2767755
  * Ni et al. (2019) Jianmo Ni, Jiacheng Li, and Julian J. McAuley. 2019.  Justifying Recommendations using Distantly-Labeled Reviews and Fine-Grained Aspects. In _EMNLP_.  https://doi.org/10.18653/v1/D19-1018
  * Parra and Brusilovsky (2015) Denis Parra and Peter Brusilovsky. 2015.  User-controllable personalization: A case study with SetFusion.  _Int. J. Hum. Comput. Stud._ 78 (2015), 43–67.  https://doi.org/10.1016/j.ijhcs.2015.01.007
  * Pu and Faltings (2000) Pearl Pu and Boi Faltings. 2000.  Enriching buyers’ experiences: the SmartClient approach. In _CHI_.  https://doi.org/10.1145/332040.332446
  * Qiu and Benbasat (2009) Lingyun Qiu and Izak Benbasat. 2009.  Evaluating Anthropomorphic Product Recommendation Agents: A Social Relationship Perspective to Designing Information Systems.  _J. Manag. Inf. Syst._ 25, 4 (2009), 145–182.
  * Rendle et al. (2009) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009.  BPR: Bayesian Personalized Ranking from Implicit Feedback. In _UAI_.
  * Sachdeva and McAuley (2020) Noveen Sachdeva and Julian J. McAuley. 2020.  How Useful are Reviews for Recommendation? A Critical Review and Potential Improvements. In _SIGIR_.  https://doi.org/10.1145/3397271.3401281
  * Sedhain et al. (2016) Suvash Sedhain, Hung Bui, Jaya Kawale, Nikos Vlassis, Branislav Kveton, Aditya Krishna Menon, Trung Bui, and Scott Sanner. 2016\.  Practical linear models for large-scale one-class collaborative filtering. In _IJCAI_.
  * Sinha and Swearingen (2002) Rashmi R. Sinha and Kirsten Swearingen. 2002.  The role of transparency in recommender systems. In _CHI_.  https://doi.org/10.1145/506443.506619
  * Symeonidis et al. (2009) Panagiotis Symeonidis, Alexandros Nanopoulos, and Yannis Manolopoulos. 2009.  MoviExplain: a recommender system with explanations. In _RecSys_. ACM, 317–320.  https://doi.org/10.1145/1639714.1639777
  * Tintarev and Masthoff (2011) Nava Tintarev and Judith Masthoff. 2011.  Designing and Evaluating Explanations for Recommender Systems.  In _Recommender Systems Handbook_. Springer, 479–510.
  * Tversky and Simonson (1993) Amos Tversky and Itamar Simonson. 1993.  Context-Dependent Preferences.  _Management Science_ 39, 10 (1993), 1179–1189.
  * Vig et al. (2009) Jesse Vig, Shilad Sen, and John Riedl. 2009.  Tagsplanations: explaining recommendations using tags. In _IUI_.  https://doi.org/10.1145/1502650.1502661
  * Wan and McAuley (2018) Mengting Wan and Julian J. McAuley. 2018.  Item recommendation on monotonic behavior chains. In _RecSys_.  https://doi.org/10.1145/3240323.3240369
  * Wärnestål (2005) Pontus Wärnestål. 2005\.  Modeling a dialogue strategy for personalized movie recommendations. In _Beyond Personalization Workshop_. 77–82.
  * Wu et al. (2019) Ga Wu, Kai Luo, Scott Sanner, and Harold Soh. 2019\.  Deep Language-Based Critiquing for Recommender Systems. In _RecSys_ (Copenhagen, Denmark).  https://doi.org/10.1145/3298689.3347009
  * Zhang et al. (2018) Yongfeng Zhang, Xu Chen, Qingyao Ai, Liu Yang, and W. Bruce Croft. 2018.  Towards Conversational Search and Recommendation: System Ask, User Respond. In _CIKM_.  https://doi.org/10.1145/3269206.3271776
  * Zhang et al. (2014) Yongfeng Zhang, Guokun Lai, Min Zhang, Yi Zhang, Yiqun Liu, and Shaoping Ma. 2014\.  Explicit factor models for explainable recommendation based on phrase-level sentiment analysis. In _SIGIR_.  https://doi.org/10.1145/2600428.2609579
  * Zhao et al. (2013) Xiaoxue Zhao, Weinan Zhang, and Jun Wang. 2013.  Interactive collaborative filtering. In _CIKM_.  https://doi.org/10.1145/2505515.2505690
  * Zhou et al. (2020) Kun Zhou, Wayne Xin Zhao, Shuqing Bian, Yuanhang Zhou, Ji-Rong Wen, and Jingsong Yu. 2020\.  Improving Conversational Recommender Systems via Knowledge Graph based Semantic Fusion. In _KDD_. ACM, 1006–1014.
  * Zou et al. (2020) Lixin Zou, Long Xia, Yulong Gu, Xiangyu Zhao, Weidong Liu, Jimmy Xiangji Huang, and Dawei Yin. 2020.  Neural Interactive Collaborative Filtering. In _SIGIR_.  https://doi.org/10.1145/3397271.3401181

##  Appendix A Additional Training Details

All experiments were conducted on a machine with a 2.2GHz 40-core CPU, 132GB
memory and one RTX 2080Ti GPU. We use PyTorch version 1.4.0 and optimize our
models using the Rectified Adam (Liu et al., 2020) optimizer. Best
hyperparameters for each base recommender system model are shown in Table 7.

##  Appendix B Time Complexity

In Table 6, we report the mean and standard error of time taken per turn for
LLC-Score, CE-VAE, BPR-Bot, and PLRec-Bot. As baseline code does not leverage
the GPU, we also critique with PLRec-Bot and BPR-Bot on the CPU only. We
observe LLC-Score and PLRec-Bot to be an order of magnitude slower per
critiquing cycle compared to CE-VAE and BPR-Bot.

BPR-Bot shows acceptable latency for real-world applications (sub-10 ms), and
we observe empirically in our cold-start user study (Section 6) that we can
host BPR-Bot as a real-time recommendation service. Time trials were conducted
with batch size of 1; production throughput can be improved further with
parallel processing. Each model executes using a different framework (numpy
for LLC-Score, Tensorflow for CE-VAE, and Pytorch for PLRec-Bot/BPR-Bot),
which may contribute to differences in inference speed.

##  Appendix C User Evaluation

An image of the interface used for our cold-start user study (Section 6) is
shown in Figure 8.

Table 6. Mean and standard error of wall-clock time (ms) per turn of critiquing for linear (LLC-Score) and variational (CE-VAE) baselines vs. our models (BPR-Bot, BPR-PLRec) | LLC-Score | CE-VAE | BPR-Bot | PLRec-Bot
---|---|---|---|---
Books | 40.64 $\pm$ 20.46 | 4.61 $\pm$ 1.16 | 2.70 $\pm$ 3.95 | 48.84 $\pm$ 14.08
Beer | 15.94 $\pm$ 14.52 | 3.26 $\pm$ 1.18 | 2.54 $\pm$ 2.36 | 49.43 $\pm$ 14.81
Music | 42.21 $\pm$ 21.04 | 3.36 $\pm$ 1.37 | 2.25 $\pm$ 0.62 | 6.80 $\pm$ 7.53

Figure 8. User interface for user study, with turn-level feedback prompts and an example of a critiqued aspect (“Battle”) Table 7. Best hyperparameter settings for each base recommendation model. UAC, BAC, LLC-Score, LLC-Rank models use PLRec as a base model. BPR-Bot uses BPR as a base model. Dataset | Model | $h$ | LR | $\lambda_{\text{L2}}$ | $\lambda_{\text{KP}}$ | $\lambda_{c}$ | $\beta$ | Epoch | Dropout
---|---|---|---|---|---|---|---|---|---
Books | BPR | 10 | 0.001 | 0.01 | 0.5 | – | – | 200 | –
| PLRec | 50 | – | 80 | – | – | – | 10 | –
| CE-VAE | 100 | 0.0001 | 0.0001 | 0.01 | 0.01 | 0.001 | 300 | 0.5
Beer | BPR | 10 | 0.001 | 0.01 | 0.5 | – | – | 200 | –
| PLRec | 50 | – | 80 | – | – | – | 10 | –
| CE-VAE | 100 | 0.0001 | 0.0001 | 0.01 | 0.01 | 0.001 | 300 | 0.5
Music | BPR | 10 | 0.01 | 0.1 | 1.0 | – | – | 200 | –
| PLRec | 400 | – | 1000 | – | – | – | 10 | –
| CE-VAE | 200 | 0.0001 | 0.0001 | 0.001 | 0.001 | 0.0001 | 600 | 0.5