# Beyond Recommender: An Exploratory Study of the Effects of Different AI
Roles in AI-Assisted Decision Making

Shuai Ma  0000-0002-7658-292X The Hong Kong University of Science and
TechnologyHong KongChina<EMAIL_ADDRESS>,  Chenyi Zhang  ShanghaiTech
UniversityShanghaiChina<EMAIL_ADDRESS>,  Xinru Wang  Purdue
UniversityWest LafayetteIndianaUSA<EMAIL_ADDRESS>,  Xiaojuan Ma  The Hong
Kong University of Science and TechnologyHong KongChina<EMAIL_ADDRESS>and
Ming Yin  Purdue UniversityWest LafayetteIndianaUSA<EMAIL_ADDRESS>
###### Abstract.

Artificial Intelligence (AI) is increasingly employed in various decision-
making tasks, typically as a Recommender, providing recommendations that the
AI deems correct. However, recent studies suggest this may diminish human
analytical thinking and lead to humans’ inappropriate reliance on AI,
impairing the synergy in human-AI teams. In contrast, human advisors in group
decision-making perform various roles, such as analyzing alternative options
or criticizing decision-makers to encourage their critical thinking. This
diversity of roles has not yet been empirically explored in AI assistance. In
this paper, we examine three AI roles: Recommender, Analyzer, and Devil’s
Advocate, and evaluate their effects across two AI performance levels. Our
results show each role’s distinct strengths and limitations in task
performance, reliance appropriateness, and user experience. Notably, the
Recommender role is not always the most effective, especially if the AI
performance level is low, the Analyzer role may be preferable. These insights
offer valuable implications for designing AI assistants with adaptive
functional roles according to different situations.

AI-Assisted Decision-making, Human-AI Collaboration, Appropriate Reliance

††ccs: Human-centered computing Empirical studies in HCI

##  1\. Introduction

With remarkable technological advancements, AI has been increasingly used to
support people in making decisions in various domains, ranging from low-stakes
recommendations around entertainment (Wang and Wang, 2014; Ma et al., 2019) to
high-stakes decision tasks such as criminal justice (Dodge et al., 2019;
Dressel and Farid, 2018), admissions and employment (Cheng et al., 2019; Zhang
et al., 2023), financial investment (Green and Chen, 2019), medical diagnosis
(Cai et al., 2019b; Lee et al., 2021), etc. Due to AI’s imperfect accuracy,
and the concerns of safety, ethics, and accountability (Cai et al., 2019b; Lee
et al., 2021; Binns et al., 2018), the _AI-assisted decision-making_ paradigm
is widely adopted in real-world applications (Buçinca et al., 2021; Zhang et
al., 2020; Wang and Yin, 2021; Bansal et al., 2021). In this paradigm, AI
performs an assistive role by providing a recommendation, while human
decision-makers can choose to accept or reject AI’s suggestion in their final
decision (Lai et al., 2021).

However, empirical research reveals several limitations within the existing
AI-assisted decision-making framework, wherein AI acts primarily as a
recommender. One notable issue is that individuals, when passively receiving
AI suggestions, seldom engage in analytical thinking (Buçinca et al., 2020;
Rastogi et al., 2022; Bertrand et al., 2022). Furthermore, people frequently
inappropriately rely on the AI’s recommendations (such as over-reliance and
under-reliance) (Buçinca et al., 2021; Ma et al., 2023; Wang and Yin, 2021; Ma
et al., 2022b) and the mere provision of AI explanations can, paradoxically,
exacerbate overreliance (Bansal et al., 2021; Poursabzi-Sangdeh et al., 2021).

In comparison, in human-human decision-making, beyond recommenders, human
advisors sometimes play other types of roles, such as helping the decision-
makers analyze the pros and cons of different alternatives instead of directly
giving recommendations, or critically challenging decision-makers’ initial
views (Schwenk, 1984; Schweiger et al., 1989). Empirical studies show that
those roles sometimes help decision-makers achieve better decision-making
performance than just giving recommendations (Schwenk, 1984; Schweiger et al.,
1986; Cosier, 1978). Despite the potential of different advisor roles,
existing research on AI-assisted decision-making mainly allows AI to play the
role of recommender and little is known about the effects of AI playing other
roles in AI-assisted decision-making. Specifically, we pose the following
research questions.

  * •

RQ1: How will different AI assistant roles affect humans’ task performance?

  * •

RQ2: How will different AI assistant roles affect humans’ reliance and the
appropriateness of reliance?

  * •

RQ3: How will different AI assistant roles affect humans’ user experience in
the decision-making process?

Additionally, since the advisors’ expertise can influence the quality of the
support information provided and the decision-makers’ receptiveness to varying
types of suggestions (Dalal and Bonaccio, 2010; Bonaccio and Dalal, 2006), we
incorporate the factor of AI performance into exploration.

To investigate these research questions, we conducted a user study with
participants recruited from Prolific. They were tasked with classifying short
news paragraphs into one of three categories: business, entertainment, or
politics. We tested three distinct AI roles: (1) Recommender, which provides
the AI’s recommendation and explanations; (2) Analyzer, which aids in
analyzing each option without specific recommendations, offering evidence both
_for_ and _against_ each alternative; (3) Devil’s Advocate, where the AI
presents counter-arguments to the participant’s initial choice, even if it
aligns with the AI’s own judgment. Additionally, we implemented two levels of
AI performance: low (65% accuracy) and high (90% accuracy). Therefore, there
are six conditions (three AI roles * two levels of AI performance) in total.
The experimental results revealed that when AI performance is high, the
Recommender role tends to lead to better task performance, more appropriate
reliance, and better user experience. However, when AI performance is low, the
Analyzer role is more effective in contributing to higher task performance,
more appropriate reliance, and better user experience. These findings reveal
that there may not be a one-size-fits-all AI role and the choice of AI role
should be determined by other factors. Finally, we discussed our findings, the
implications for designers, and potential directions for future research.

##  2\. Related Work

Artificial Intelligence (AI) is increasingly used in decision-making across
various domains (Dastin, 2018; Dilsizian and Siegel, 2014; Khandani et al.,
2010; Wang and Wang, 2014; Yang et al., 2018). However, AI’s real-world
applications are not infallible, lacking 100% accuracy (Ma et al., 2022a; Lai
et al., 2021). This is especially concerning in high-stakes domains like
clinical diagnosis and criminal justice, leading to ethical and legal
complexities (Cai et al., 2019b; Lee et al., 2021; Binns et al., 2018). To
address this, the prevalent paradigm of AI-assisted decision-making has
emerged, drawing substantial attention in the Human-Computer Interaction (HCI)
and AI communities (Buçinca et al., 2021; Zhang et al., 2020; Wang and Yin,
2021; Bansal et al., 2021). In this paradigm, AI takes on a supportive role,
offering recommendations (as well as other information, such as explanations
(Bansal et al., 2021; Zhang et al., 2020)) for human decision-makers to accept
or reject in their final decisions (Lai et al., 2021).

Empirical research indicates that, in the existing AI-assisted decision-making
paradigm, humans often rely on heuristics rather than analytical thinking,
making it difficult to appropriately rely on AI’s suggestions (Buçinca et al.,
2020, 2021). This contrasts with human-human decision-making, where human
advisors can adopt various roles beyond merely recommending. They may assist
in weighing the pros and cons of different options, suggesting alternatives,
or actively challenging the initial opinions of decision-makers (Schwenk,
1984; Schweiger et al., 1989). Studies have demonstrated that these roles
(such as dialectical inquiry, devil’s advocate (Schwenk, 1984)) can sometimes
enhance decision-making effectiveness more than just providing recommendations
(Schweiger et al., 1986; Cosier, 1978). However, current research on AI-
assisted decision-making predominantly positions AI in the recommender role
(suggesting what AI thinks correct), with limited understanding of the impact
when AI assumes different advisory roles in the decision-making process.

In human decision-making processes, human advisors often act in three roles:
Recommender, Analyzer, and Devil’s Advocate. These three roles are the most
commonly assumed by human assistants in group or organizational decision
processes (Schwenk, 1984; Schweiger et al., 1989). Moreover, human advisers
sometimes take on additional roles. For example, acting as a (1) “facilitator”
to help decision-makers form or clarify the problem (Valacich and Schwenk,
1995) or teaching decision-makers how to make a decision (for instance,
suggesting that making a list of pros and cons could be useful) (Schweiger et
al., 1989), (2) “affirmation provider” because sometimes, decision-makers seek
not new options but confirmation that “you are right” (Sniezek and Buckley,
1995), (3) “social or emotional supporter” to give psychological support
(e.g., a human advisor may say “I understand this is really tough for you as
choosing a job is a very stressful decision”) (Arkes et al., 1991; Dalal and
Bonaccio, 2010), and (4) “information collector” to provide information
unknown to decision-makers or organizing information for them (Schweiger et
al., 1989). However, since the primary aim of AI-assisted decision-making is
to enhance task performance, merely agreeing with the decision-maker or
providing emotional support may not be suitable roles for AI to fulfill.
Furthermore, this paper focuses on decision-making scenarios based on current
task information and does not involve external knowledge, thus eliminating the
need for AI to help gather additional information. Therefore, in this paper,
we decided to choose the roles of Recommender, Analyzer, and Devil’s Advocate
for AI to play and investigate the effects on human decision-making.

In recent years, researchers in AI-assisted decision-making and XAI have
envisioned having AI play other roles. For example, in the discussion part of
Bansal et al.’s work (Bansal et al., 2021), they propose to make AI act as the
Devil’s Advocate. And in Miller’s paper (Miller, 2023), a new concept called
“hypothesis-driven” decision support is proposed where the AI will not give
suggestions but help users analyze their proposed hypotheses. However, there
is a lack of empirical investigation of the actual effects of different AI
roles in AI-assisted decision-making. In this paper, we focus on exploring the
effects of different AI roles on task performance, the appropriateness of
human reliance, and user experience. We also investigate the interaction
effects of some key factors, such as AI performance and human-independent
performance. We also conduct exploratory post-hoc analysis based on the
characteristics of humans.

##  3\. User Study

The goal of our study is to empirically understand the effects of different AI
roles and the interaction effects between AI roles and AI performance. We
begin our study when the task is easy for human decision-makers.

###  3.1. Task, AI Model, and AI Explanation

We selected _news category classification_ (Misra, 2022; Misra and Grover,
2021) as our testbed because text classification has been widely used in
several previous studies on human-AI decision-making (Lai and Tan, 2019; Lai
et al., 2020; Bansal et al., 2021) and it requires little domain expertise and
is thus amenable to crowdsourcing. Specifically, given a piece of short news,
participants were asked to classify the news into one of three categories:
business, entertainment, and politics.

After processing the textual data (such as removing the stopwords), we trained
two AI models to generate AI assistance with two levels of performance. One
used a logistic regression model with NLTK tokenizer to train the low-
performance AI. The other used a BERT model (bert-base-uncased) to train the
high-performance AI.

We used LIME to generate explanations for each model. For each of the three
news categories, LIME identified the top 10 important words in a news
paragraph and assigned an importance score to each word (a positive score
means a positive influence on that category while a negative score means a
negative influence). We leveraged words with positive importance scores as
AI’s collected evidence for a prediction and words with negative importance
scores as AI’s collected evidence against a prediction.

With pilot studies, we carefully selected 20 news with uniform distributed
labels (7 business, 7 entertainment, and 6 politics). Participants’ average
accuracy on these task instances is around 80%. The high-performance AI’s
accuracy is 90% while the low-performance AI’s accuracy is 65%.

###  3.2. Conditions

We adopt a two-factor between-subjects study design in which each participant
sees one of three AI role conditions in one of the two AI performance
conditions. So our experiment contains six conditions in total (3*2).

Figure 1. The interfaces of three AI roles. (a) Recommender: In this
interface, the AI directly presents a recommendation along with its
explanation. (b) Analyzer: The AI assists in evaluating the pros and cons of
each option. By default, it displays an overview of all options and highlights
two highly probable choices to help users refine their selection. Users can
explore detailed AI analyses for any specific option by clicking on its
corresponding tab. (c) Devil’s Advocate: This interface features the AI
providing arguments against the user’s current selection, irrespective of
whether it aligns with the AI’s own prediction. Additionally, the AI suggests
an alternative option, which represents the most viable alternative to the
user’s initial choice.

Factor 1: AI Role. We compare three types of assistance the AI can provide.
Note that the implementation of Analyzer and Devil’s Advocate is not fixed,
and we do not want to say that the current implementation is the best. We
encourage interested researchers to explore more interesting interface designs
for these AI roles.

  * •

Recommender: The AI model provides a recommendation (which category the AI
thinks with the highest probability) and its related explanations (words
highlighted for and against this recommendation) (Figure 1 (a)).

  * •

Analyzer: The AI model provides evidence for (words highlighted with positive
importance score) and again (words highlighted with negative importance score)
each category. To help users narrow the exploration space, the AI model also
highlights two options that the AI thinks possible (Figure 1 (b)).

  * •

Devil’s Advocate: The AI model provides evidence against what the user
predicts (by only highlighting words with negative importance scores). And the
AI model marks one option that needs to be explored other than what the user
has predicted (Figure 1 (c)). Note that the AI will always criticize the human
- even if the AI agrees with the users’ prediction, the AI still provides
negative evidence against the user’s selected option.

Factor 2: AI performance. The AI performance condition manipulates the
accuracy of AI’s predictions (although AI predictions are not directly
presented to users in some AI role conditions, the quality of AI gathered
evidence is affected by AI’s accuracy). We note that the definition of
performance level can be affected by many factors. For example, in some easy-
to-recognize tasks, 90% accuracy is still deemed as low performance, while in
some difficult tasks, even 65% accuracy can be a reasonable performance. In
this experiment, we define AI performance as high or low based on the relative
strength between humans and AI. Since from a pilot study, we found that, in
this task, humans’ accuracy is around 80%, we artificially set 90% accuracy as
high performance and 65% accuracy as low performance.

  * •

High-Performance AI: The AI model’s accuracy on the selected 20 task instances
is 90%.

  * •

Low-Performance AI: The AI model’s accuracy on the selected 20 task instances
is 65%.

For each AI performance condition, on the study interfaces, we explicitly
informed participants of AI’s performance - “The Al model is trained on a huge
news data set. We previously evaluated this Al model on a large data set and
its accuracy was 90% (65%) (i.e., the Al model’s predictions were correct on
90% (65%) of the news in that data set.)” In this way, we can avoid any
confounding factors that arise from the inconsistency between the stated AI
accuracy and the actual AI accuracy.

###  3.3. Procedure

We conducted a between-subjects study. After obtaining consent, participants
completed a questionnaire gathering their demographic data, AI expertise, and
other characteristics. Participants then went to an interactive tutorial,
practiced with one example task, and familiarized themselves with how the AI
offers assistance based on the assigned experimental condition. Following the
tutorial, qualification questions ensured understanding, with only those
answering all questions correctly proceeding to the main task. The main task
involved 20 task instances. For each task instance, participants were required
to first make their own predictions, then receive AI’s assistance, and finally
make the final predictions. In the exit survey, we collected participants’
user experience.

###  3.4. Participants

After obtaining institutional IRB approval, we recruited participants from
Prolific111www.prolific.co. Specific criteria we set for the participants
include: (1) a minimum 99% approval rate for previous submissions; (2) English
as their first language; (3) at least 1000 approved previous submissions; and
(4) using a desktop computer for the experiment. The study followed a between-
subjects design with no repeated participation. After filtering out
inattentive participants, we obtained 140 valid responses (more than 20
participants in each condition). Among the participants, there were 75 self-
reported males, 63 females, and 2 non-binary individuals. Age distribution
included 19 participants aged 18-29, 44 aged 30-39, 42 aged 40-49, 22 aged
50-59, and 13 aged over 59. Participants also self-rated their AI expertise:
17 had no knowledge, 68 knew basic AI concepts, 44 had experience with AI
algorithms, and 11 were experts in AI. As an incentive for quality work,
participants received a $1 bonus for an overall accuracy exceeding 95%. The
study took approximately 20 minutes, with participants earning an average wage
of $12 per hour.

###  3.5. Measurements and Analysis Methods

Based on the proposed three research questions, we measured participants’ task
performance, reliance and the appropriateness of reliance, and user
experience.

Task Performance We calculated participants’ initial accuracy before seeing
AI’s assistance and final accuracy after seeing AI’s assistance (Ma et al.,
2023).

Reliance and Its Appropriateness We measure the reliance of participants on
the AI system via two metrics (Zhang et al., 2020; He et al., 2023):

$\textnormal{{Agreement Fraction}}=\frac{\textnormal{Number of final decisions
same as the AI suggestion}}{\textnormal{Total number of decisions}},$

$\textnormal{{Switch Fraction}}=\frac{\textnormal{Number of decisions user
switched to agree with the AI model}}{\textnormal{Total number of decisions
with initial disagreement}},$

Based on literature (Wang and Yin, 2021; Bansal et al., 2021), we measure the
appropriateness of participants’ reliance by:

$\textnormal{{Over-Reliance}}=\frac{\textnormal{Number of incorrect human
final decisions with incorrect AI advice}}{\textnormal{Total number of
incorrect AI advice}},$

$\textnormal{{Under-Reliance}}=\frac{\textnormal{Number of incorrect human
final decisions with correct AI advice}}{\textnormal{Total number of correct
AI advice}},$

User Experience We measured participants’ _perceived autonomy_ (Dalal and
Bonaccio, 2010), _mental demand_ (Hart, 2006; Ma et al., 2023, 2022b),
_perceived complexity_ (Buçinca et al., 2021), _engagement_ (Boulton et al.,
2012), _future use_ (Ma et al., 2023, 2022a), _satisfaction_ (Buçinca et al.,
2021; Ghai et al., 2021), _perceived helpfulness_ (Laugwitz et al., 2008; Cai
et al., 2019a; Buçinca et al., 2020), _trust_ (Yin et al., 2019), _self-
efficacy_ (Hemmer et al., 2023) via 7-point Likert scales. The detailed
questions are shown in Table 1. For the data analysis, since the collected
user data did not pass the normality test, we used Kruskal-Wallis to compare
three AI roles and post-hoc pair-wise comparison with Bonferroni correction.

Table 1. Questionnaire used in our user study for assessing participants’ user experience. All the selected questions are selected and adapted based on verified scales. Aspect |  Question
---|---
Autonomy |  How much did your interaction with the AI model in these tasks help you maintain the freedom (i.e., autonomy) to make the news category predictions in the way you see fit? (1: Not at all, 7: Extremely)
Mental Demand |  How mentally demanding was your interaction with the AI model? (1: Very low, 7: Very high)
Complexity |  The decision-making process with this kind of interface to interact with AI models is complex. (1: Strongly disagree, 7: Strongly agree)
Engagement |  I am engaged in the decision-making task with the AI model. (1: Strongly disagree, 7: Strongly agree)
Future Use |  I would like to continue to work with the AI model. (1: Strongly disagree, 7: Strongly agree)
Satisfaction |  I am satisfied with the AI model’s assistance. (1: Strongly disagree, 7: Strongly agree)
Helpfulness |  I think the AI model’s assistance is helpful/useful for me to make good decisions. (1: Strongly disagree, 7: Strongly agree)
Trust |  I can trust the AI model. (1: Strongly disagree, 7: Strongly agree)
|  The AI model can be trusted to provide reliable decision support. (1:
Strongly disagree, 7: Strongly agree)
|  I trust the AI model to keep my best interests in mind. (1: Strongly
disagree, 7: Strongly agree)
|  In my opinion, the AI model is trustworthy. (1: Strongly disagree, 7:
Strongly agree)
Efficacy |  I am confident about my ability to complete news category classification tasks. (1: Strongly disagree, 7: Strongly agree)
|  I have mastered the skills necessary for completing news category
classification tasks. (1: Strongly disagree, 7: Strongly agree)
|  I am self-assured about my capabilities to perform news category
classification tasks. (1: Strongly disagree, 7: Strongly agree)

##  4\. Results

###  4.1. Effects of AI roles on task performance

Figure 2(a) demonstrates that with high-performance AI, the Recommender role
leads to higher final accuracy compared to both the Analyzer (p¡.001) and the
Devil’s Advocate (p¡.05). Conversely, at low AI performance levels, the
Analyzer tends to yield higher final accuracy. However, this difference is
statistically significant only when compared with the Devil’s Advocate
(p¡.001). Overall, these findings suggest that high-accuracy AI is best suited
for the Recommender role. In cases of suboptimal AI accuracy, it is less
appropriate for AI to directly provide recommendations. Instead, it proves
more effective in assisting users to analyze the advantages and disadvantages
of various decision options.

We also conducted a detailed analysis of task performance based on the
correctness of human initial prediction and AI recommendation. Results in
Figure 3 reveal that with high AI accuracy, the Devil’s Advocate approach
improves human decisions when initial predictions are correct but AI advice is
incorrect. Interestingly, under low AI accuracy, both Analyzer and Devil’s
Advocate surpass the Recommender by aiding better decisions in cases where
both human prediction and AI recommendation are wrong, or the human is right
but AI is wrong. This indicates that Analyzer and Devil’s Advocate remain
beneficial for decision-making even when AI recommendations are inaccurate.

Figure 2. The task accuracy in three AI role conditions. (a) When AI
performance is high. (b) When AI performance is low. Error bars represent
standard error. (+: $p$¡0.1, *: $p$¡0.05, **: $p$¡0.01, ***: $p$¡0.001)

Figure 3. A detailed analysis of participants’ final decision accuracy based
on the correctness of human initial prediction and AI recommendation. (a) When
AI performance is high. (b) When AI performance is low. Error bars represent
standard error. (Since categorizing participants’ prediction data into
different situations reduces the sample size, we did not perform statistical
analysis.)

Figure 4. Participants’ reliance and the appropriateness of their reliance on
AI. (a) When AI performance is high. (b) When AI performance is low. Error
bars represent standard error. (+: $p$¡0.1, *: $p$¡0.05, **: $p$¡0.01, ***:
$p$¡0.001)

###  4.2. Effects on Reliance and the Appropriateness of Reliance

Figure 4 shows that the agreement and switch fractions are highest when AI
plays a recommender role, irrespective of AI performance. This outcome is
attributed to the Recommender AI’s direct presentation of recommendations, in
contrast to the implicit decision support provided by the Analyzer and Devil’s
Advocate. As depicted in Figure 4 (a), high AI performance results in lower
over-reliance with the Devil’s Advocate compared to Analyzer (p¡.01). This may
be because the Devil’s Advocate raises challenges, particularly when AI
predictions are incorrect yet align with human predictions, thereby reducing
the likelihood of human error. Conversely, Figure 4 (b) reveals that with low
AI performance, over-reliance is less for the Analyzer and Devil’s Advocate
than for the Recommender (p¡.001 and p¡.05 respectively). This is probabaly
because, in these roles, AI does not directly present incorrect predictions
and participant’s lower overall trust in AI (due to the AI’s low performance)
mitigates the impact of AI misjudgments. Regarding under-reliance, Figure 4
(a) shows that high AI performance leads to lower under-reliance with the
Recommender role compared to the Analyzer (p¡.01) and Devil’s Advocate
(p¡.001). This is due to the Recommender’s direct provision of correct
predictions, facilitating the acceptance of AI’s accurate judgments. Figure 4
(b) indicates that low AI performance causes more under-reliance with the
Devil’s Advocate role, likely because this role prompts people to question
even correct AI predictions that align with their views.

In summary, the Devil’s Advocate role consistently reduces over-reliance,
albeit increasing under-reliance, regardless of AI performance. The
Recommender role decreases under-reliance but increases over-reliance.
Notably, the Analyzer role diminishes both over-reliance and under-reliance,
when AI performance is low. This suggests that the Analyzer role may be more
beneficial when AI performance is suboptimal, fostering a more appropriate
reliance on AI.

###  4.3. Effects on User Experience

Figure 5. Participants’ user experience. (a) When AI performance is high. (b)
When AI performance is low. Error bars represent standard error. (+: $p$¡0.1,
*: $p$¡0.05, **: $p$¡0.01, ***: $p$¡0.001)

Figure 5 shows the effects of different AI roles on participants’ user
experience. Results indicate that when AI performance is low, the Analyzer
role contributes the most to participants’ perceived autonomy. This likely
stems from people’s reluctance to engage with low-performance AI’s suggestions
or critiques. Instead, users seem to feel more control when AI just provides
analysis. When AI performance is high, the Devil’s Advocate role imposes
participants’ higher mental demands and perceived complexity of the system
than Analyzer. This might be due to users’ willingness to invest cognitive
resources in considering the AI’s challenges but they had to take AI’s
challenges into consideration due to AI’s high performance. Moreover, we found
that with low AI performance, users engage more with the Analyzer’s assistance
than with Devil’s Advocate. When the AI’s accuracy is low, participants show a
preference for continuing to work with the Analyzer. In comparison, there is a
consistent reluctance to use the Devil’s Advocate regardless of AI performance
level, possibly because users prefer affirmation over being questioned [ref].
Additionally, when AI performance is high, users exhibit greater satisfaction,
helpfulness, and trust in both the Recommender and Analyzer roles. However,
when AI performance is low, these positive responses are more pronounced with
the Analyzer role.

In summary, a high-performing AI enhances user experience with both the
Recommender and Analyzer roles. In contrast, a lower-performing AI leads to a
more favorable user experience when the Analyzer role is employed.

##  5\. Discussion

Stepping out of XAI-based AI-Assisted Decision-making. There is currently a
growing body of work proposing that AI should provide advice in ways other
than just giving suggestions and explanations. For instance, Danry et al.
(Danry et al., 2023) introduced AI-framed Questioning to enhance critical
thinking and human discernment of flawed statements while Miller advocates for
AI to “evaluate” people’s hypotheses instead of giving recommendations
(Miller, 2023). Our research aligns with these innovative approaches and
contributes empirical insights to the fields of AI-assisted decision-making
and human-AI collaboration. We encourage researchers to venture beyond
conventional XAI-based support and to investigate a broader range of novel AI-
assisted decision-making paradigms.

Adapting AI Roles to Suit Varying Situations. Our experiments indicate that a
high-performing AI is most effective in the Recommender role. Conversely, when
AI performance is low, the Analyzer role enhances task performance, fosters
more appropriate reliance, and improves the user experience. Since AI’s
calibrated confidence can serve as an indicator of its accuracy, it is
potentially effective to adaptively switch AI roles based on AI’s confidence
in a specific prediction. For instance, in scenarios where AI confidence is
low (suggesting a lower probability of correctness), the AI could assume an
analytical role, providing counsel rather than direct recommendation.

Choosing AI Roles Based on Specific Objectives. For instance, in high-risk
decision-making scenarios where reducing over-reliance on AI is crucial,
assigning AI the role of Devil’s Advocate might be beneficial, even when AI
performance is high. Conversely, to mitigate under-reliance on AI, the
Recommender role may be more effective.

Limitations and Future Work. Our study’s main limitation is its small
participant sample. In the next step, we plan to expand the number of
participants. Besides, in this paper, we only explored the interaction effects
between AI roles and AI performance. Future research will explore the effects
of the AI role with two critical factors: task difficulty and human-AI
complementarity. Task difficulty affects both independent human performance
and their response to various types of assistance, as well as their
willingness to accept advice. Besides, the effectiveness of AI roles like
Recommender, Analyzer, and Devil’s Advocate may vary based on the degree of
human-AI complementarity. Additionally, we plan to investigate how personal
characteristics (Schwenk, 1982; Bonaccio and Dalal, 2006), such as the need
for cognition, ambiguity tolerance, self-esteem, and agreeableness, influence
the effectiveness of different AI roles. This investigation will help us
understand if different individuals are more suited to be accompanied by
specific AI assistant roles. Moreover, in the next round of study, we will
collect participants’ open-ended feedback to qualitatively analyze their
nuanced perceptions of different AI roles and gain a deep understanding of the
root causes of the effects.

##  6\. Conclusion

This position paper empirically investigates the impact of AI assuming varied
roles in AI-assisted decision-making and its effects on human decision
processes. Our initial experiments reveal that different AI roles exhibit
distinct advantages and limitations, which are further influenced by the AI’s
performance. We aim for our findings to invigorate ongoing debates within the
AI-assisted decision-making community regarding the optimal forms of AI
assistance. Additionally, we hope to enrich discussions in the human-centered
XAI community about the appropriate and human-compatible ways for AI to
communicate explanations to users.

## References

  * (1)
  * Arkes et al. (1991) Hal R Arkes, Maya Bar-Hillel, Lee Roy Beach, Bemdt Brehmer, Jeanne B Brett, N John Castellan Jr, and W Edwards. 1991.  Organizational behavior and human decision processes.  (1991).
  * Bansal et al. (2021) Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021.  Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–16.
  * Bertrand et al. (2022) Astrid Bertrand, Rafik Belloum, James R Eagan, and Winston Maxwell. 2022.  How Cognitive Biases Affect XAI-assisted Decision-making: A Systematic Review. In _Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society_. 78–91.
  * Binns et al. (2018) Reuben Binns, Max Van Kleek, Michael Veale, Ulrik Lyngs, Jun Zhao, and Nigel Shadbolt. 2018.  ’It’s Reducing a Human Being to a Percentage’ Perceptions of Justice in Algorithmic Decisions. In _Proceedings of the 2018 Chi conference on human factors in computing systems_. 1–14.
  * Bonaccio and Dalal (2006) Silvia Bonaccio and Reeshad S Dalal. 2006.  Advice taking and decision-making: An integrative literature review, and implications for the organizational sciences.  _Organizational behavior and human decision processes_ 101, 2 (2006), 127–151.
  * Boulton et al. (2012) Mike Boulton, Julian Lloyd, James Down, and Hedda Marx. 2012.  Predicting undergraduates’ self-reported engagement in traditional and cyberbullying from attitudes.  _Cyberpsychology, Behavior, and Social Networking_ 15, 3 (2012), 141–147.
  * Buçinca et al. (2020) Zana Buçinca, Phoebe Lin, Krzysztof Z Gajos, and Elena L Glassman. 2020.  Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems. In _Proceedings of the 25th international conference on intelligent user interfaces_. 454–464.
  * Buçinca et al. (2021) Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021.  To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making.  _Proceedings of the ACM on Human-Computer Interaction_ 5, CSCW1 (2021), 1–21.
  * Cai et al. (2019a) Carrie J Cai, Emily Reif, Narayan Hegde, Jason Hipp, Been Kim, Daniel Smilkov, Martin Wattenberg, Fernanda Viegas, Greg S Corrado, Martin C Stumpe, et al. 2019a.  Human-centered tools for coping with imperfect algorithms during medical decision-making. In _Proceedings of the 2019 chi conference on human factors in computing systems_. 1–14.
  * Cai et al. (2019b) Carrie J Cai, Samantha Winter, David Steiner, Lauren Wilcox, and Michael Terry. 2019b.  ” Hello AI”: uncovering the onboarding needs of medical practitioners for human-AI collaborative decision-making.  _Proceedings of the ACM on Human-computer Interaction_ 3, CSCW (2019), 1–24.
  * Cheng et al. (2019) Hao-Fei Cheng, Ruotong Wang, Zheng Zhang, Fiona O’Connell, Terrance Gray, F Maxwell Harper, and Haiyi Zhu. 2019.  Explaining decision-making algorithms through UI: Strategies to help non-expert stakeholders. In _Proceedings of the 2019 chi conference on human factors in computing systems_. 1–12.
  * Cosier (1978) Richard A Cosier. 1978.  The effects of three potential aids for making strategic decisions on prediction accuracy.  _Organizational Behavior and Human Performance_ 22, 2 (1978), 295–306.
  * Dalal and Bonaccio (2010) Reeshad S Dalal and Silvia Bonaccio. 2010.  What types of advice do decision-makers prefer?  _Organizational Behavior and Human Decision Processes_ 112, 1 (2010), 11–23.
  * Danry et al. (2023) Valdemar Danry, Pat Pataranutaporn, Yaoli Mao, and Pattie Maes. 2023.  Don’t Just Tell Me, Ask Me: AI Systems that Intelligently Frame Explanations as Questions Improve Human Logical Discernment Accuracy over Causal AI explanations. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–13.
  * Dastin (2018) Jeffrey Dastin. 2018.  Amazon scraps secret AI recruiting tool that showed bias against women.  In _Ethics of Data and Analytics_. Auerbach Publications, 296–299.
  * Dilsizian and Siegel (2014) Steven E Dilsizian and Eliot L Siegel. 2014.  Artificial intelligence in medicine and cardiac imaging: harnessing big data and advanced computing to provide personalized medical diagnosis and treatment.  _Current cardiology reports_ 16, 1 (2014), 1–8.
  * Dodge et al. (2019) Jonathan Dodge, Q Vera Liao, Yunfeng Zhang, Rachel KE Bellamy, and Casey Dugan. 2019.  Explaining models: an empirical study of how explanations impact fairness judgment. In _Proceedings of the 24th international conference on intelligent user interfaces_. 275–285.
  * Dressel and Farid (2018) Julia Dressel and Hany Farid. 2018.  The accuracy, fairness, and limits of predicting recidivism.  _Science advances_ 4, 1 (2018), eaao5580.
  * Ghai et al. (2021) Bhavya Ghai, Q Vera Liao, Yunfeng Zhang, Rachel Bellamy, and Klaus Mueller. 2021.  Explainable active learning (xal) toward ai explanations as interfaces for machine teachers.  _Proceedings of the ACM on Human-Computer Interaction_ 4, CSCW3 (2021), 1–28.
  * Green and Chen (2019) Ben Green and Yiling Chen. 2019.  The principles and limits of algorithm-in-the-loop decision making.  _Proceedings of the ACM on Human-Computer Interaction_ 3, CSCW (2019), 1–24.
  * Hart (2006) Sandra G Hart. 2006.  NASA-task load index (NASA-TLX); 20 years later. In _Proceedings of the human factors and ergonomics society annual meeting_ , Vol. 50. Sage publications Sage CA: Los Angeles, CA, 904–908.
  * He et al. (2023) Gaole He, Lucie Kuiper, and Ujwal Gadiraju. 2023.  Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–18.
  * Hemmer et al. (2023) Patrick Hemmer, Monika Westphal, Max Schemmer, Sebastian Vetter, Michael Vössing, and Gerhard Satzger. 2023.  Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfaction. In _Proceedings of the 28th International Conference on Intelligent User Interfaces_. 453–463.
  * Khandani et al. (2010) Amir E Khandani, Adlar J Kim, and Andrew W Lo. 2010.  Consumer credit-risk models via machine-learning algorithms.  _Journal of Banking & Finance_ 34, 11 (2010), 2767–2787.
  * Lai et al. (2021) Vivian Lai, Chacha Chen, Q Vera Liao, Alison Smith-Renner, and Chenhao Tan. 2021.  Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies.  _arXiv preprint arXiv:2112.11471_ (2021).
  * Lai et al. (2020) Vivian Lai, Han Liu, and Chenhao Tan. 2020.  ” Why is’ Chicago’deceptive?” Towards Building Model-Driven Tutorials for Humans. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–13.
  * Lai and Tan (2019) Vivian Lai and Chenhao Tan. 2019.  On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In _Proceedings of the conference on fairness, accountability, and transparency_. 29–38.
  * Laugwitz et al. (2008) Bettina Laugwitz, Theo Held, and Martin Schrepp. 2008.  Construction and evaluation of a user experience questionnaire. In _Symposium of the Austrian HCI and usability engineering group_. Springer, 63–76.
  * Lee et al. (2021) Min Hun Lee, Daniel P Siewiorek, Asim Smailagic, Alexandre Bernardino, and Sergi Bermúdez i Badia. 2021.  A Human-AI Collaborative Approach for Clinical Decision Making on Rehabilitation Assessment. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–14.
  * Ma et al. (2023) Shuai Ma, Ying Lei, Xinru Wang, Chengbo Zheng, Chuhan Shi, Ming Yin, and Xiaojuan Ma. 2023.  Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Making. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–19.
  * Ma et al. (2022a) Shuai Ma, Mingfei Sun, and Xiaojuan Ma. 2022a.  Modeling Adaptive Expression of Robot Learning Engagement and Exploring its Effects on Human Teachers.  _ACM Transactions on Computer-Human Interaction_ (2022).
  * Ma et al. (2019) Shuai Ma, Zijun Wei, Feng Tian, Xiangmin Fan, Jianming Zhang, Xiaohui Shen, Zhe Lin, Jin Huang, Radomír Měch, Dimitris Samaras, et al. 2019\.  SmartEye: assisting instant photo taking via integrating user preference with deep view proposal network. In _Proceedings of the 2019 CHI conference on human factors in computing systems_. 1–12.
  * Ma et al. (2022b) Shuai Ma, Taichang Zhou, Fei Nie, and Xiaojuan Ma. 2022b.  Glancee: An Adaptable System for Instructors to Grasp Student Learning Status in Synchronous Online Classes. In _CHI Conference on Human Factors in Computing Systems_. 1–25.
  * Miller (2023) Tim Miller. 2023.  Explainable AI is Dead, Long Live Explainable AI! Hypothesis-driven Decision Support using Evaluative AI. In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_. 333–342.
  * Misra (2022) Rishabh Misra. 2022.  News Category Dataset.  _arXiv preprint arXiv:2209.11429_ (2022).
  * Misra and Grover (2021) Rishabh Misra and Jigyasa Grover. 2021.  _Sculpting Data for ML: The first act of Machine Learning_.
  * Poursabzi-Sangdeh et al. (2021) Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Wortman Vaughan, and Hanna Wallach. 2021.  Manipulating and measuring model interpretability. In _Proceedings of the 2021 CHI conference on human factors in computing systems_. 1–52.
  * Rastogi et al. (2022) Charvi Rastogi, Yunfeng Zhang, Dennis Wei, Kush R Varshney, Amit Dhurandhar, and Richard Tomsett. 2022.  Deciding fast and slow: The role of cognitive biases in ai-assisted decision-making.  _Proceedings of the ACM on Human-Computer Interaction_ 6, CSCW1 (2022), 1–22.
  * Schweiger et al. (1986) David M Schweiger, William R Sandberg, and James W Ragan. 1986.  Group approaches for improving strategic decision making: A comparative analysis of dialectical inquiry, devil’s advocacy, and consensus.  _Academy of management Journal_ 29, 1 (1986), 51–71.
  * Schweiger et al. (1989) David M Schweiger, William R Sandberg, and Paula L Rechner. 1989.  Experiential effects of dialectical inquiry, devil’s advocacy and consensus approaches to strategic decision making.  _Academy of Management journal_ 32, 4 (1989), 745–772.
  * Schwenk (1982) Charles R Schwenk. 1982.  Effects of inquiry methods and ambiguity tolerance on prediction performance.  _Decision Sciences_ 13, 2 (1982), 207–221.
  * Schwenk (1984) Charles R Schwenk. 1984.  Devil’s advocacy in managerial decision-making.  _Journal of Management Studies_ 21, 2 (1984), 153–168.
  * Sniezek and Buckley (1995) Janet A Sniezek and Timothy Buckley. 1995.  Cueing and cognitive conflict in judge-advisor decision making.  _Organizational behavior and human decision processes_ 62, 2 (1995), 159–174.
  * Valacich and Schwenk (1995) Joseph S Valacich and Charles Schwenk. 1995.  Structuring Conflict in Individual, Face-to-Face, and Computer-Mediated Group Decision Making: Carping Versus Objective Devil’s Advocacy.  _Decision Sciences_ 26, 3 (1995), 369–393.
  * Wang and Wang (2014) Xinxi Wang and Ye Wang. 2014.  Improving content-based and hybrid music recommendation using deep learning. In _Proceedings of the 22nd ACM international conference on Multimedia_. 627–636.
  * Wang and Yin (2021) Xinru Wang and Ming Yin. 2021.  Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making. In _26th International Conference on Intelligent User Interfaces_. 318–328.
  * Yang et al. (2018) Yi Yang, Wei Qian, and Hui Zou. 2018.  Insurance premium prediction via gradient tree-boosted Tweedie compound Poisson models.  _Journal of Business & Economic Statistics_ 36, 3 (2018), 456–470.
  * Yin et al. (2019) Ming Yin, Jennifer Wortman Vaughan, and Hanna Wallach. 2019.  Understanding the effect of accuracy on trust in machine learning models. In _Proceedings of the 2019 chi conference on human factors in computing systems_. 1–12.
  * Zhang et al. (2023) Angie Zhang, Olympia Walker, Kaci Nguyen, Jiajun Dai, Anqing Chen, and Min Kyung Lee. 2023.  Deliberating with AI: Improving Decision-Making for the Future through Participatory AI Design and Stakeholder Deliberation.  _Proceedings of the ACM on Human-Computer Interaction_ 7, CSCW1 (2023), 1–32.
  * Zhang et al. (2020) Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. 2020.  Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_. 295–305.