# Trustworthy Recommender Systems: An Overview

Shoujin Wang  University of Technology Sydney, RMIT UniversitySydneyAustralia
<EMAIL_ADDRESS>,  Xiuzhen Zhang  RMIT UniversityMelbourneAustralia
<EMAIL_ADDRESS>,  Yan Wang  Macquarie UniversitySydneyAustralia
<EMAIL_ADDRESS>,  Huan Liu  Arizona State UniversityTempeUnited States
<EMAIL_ADDRESS>and  Francesco Ricci  Free University of Bozen-
BolzanoBolzanoItaly<EMAIL_ADDRESS>
(2023)

###### Abstract.

Recommender systems (RSs) aim to help users to effectively retrieve items of
their interests from a large catalogue. For a quite long period of time,
researchers and practitioners have been focusing on developing accurate RSs.
Recent years have witnessed an increasing number of threats to RSs, coming
from attacks, system and user generated noise, system bias. As a result, it
has become clear that a strict focus on RS accuracy is limited and the
research must consider other important factors, e.g., trustworthiness. For end
users, a trustworthy RS (TRS) should not only be accurate, but also
transparent, unbiased and fair as well as robust to noise or attacks. These
observations actually led to a paradigm shift of the research on RSs: from
accuracy-oriented RSs to TRSs. However, researchers lack a systematic overview
and discussion of the literature in this novel and fast developing field of
TRSs. To this end, in this paper, we provide an overview of TRSs, including a
discussion of the motivation and basic concepts of TRSs, a presentation of the
challenges in building TRSs, and a perspective on the future directions in
this area. We also provide a novel conceptual framework to support the
construction of TRSs.

recommender systems, trustworthy recommendation

††journal: TIST††journalvolume: 1††journalnumber: 1††article: 1††journalyear:
2023††publicationmonth: 6††doi: 0000001.0000001††ccs: Information systems
Recommender systems

##  1\. Introduction

We are living in the era of information explosion and digital economy, where
the information overload problem has become increasingly important. As a
matter of fact, today people usually make choices from massive and rapidly
increasing catalogues of products and services (generally called items), while
consuming a large amount of time and resources to discriminate relevant from
not-relevant items. To make informed choices and decisions in a more effective
and efficient way, Recommender Systems (RSs) have been introduced into almost
every aspect of our daily life, work, business, study, entertainment and
socialization (Ricci et al., 2022; Wang et al., 2021). Today RSs are one of
the most important and popular application areas of artificial intelligence
(AI). Formally, RSs are software tools and techniques which provide
suggestions on items which may be of interest to end users. According to
McKinsey’s report111https://www.mckinsey.com/industries/retail/our-
insights/how-retailers-can-keep-up-with-consumers, 35% of what customers
purchase on Amazon and 75% of what users watch on Netflix come from
recommendations.

Since the release of the first RS, “Tapestry” in 1992, which was then termed
collaborative filtering technique (Goldberg et al., 1992), RSs have flourished
and achieved great success in the past 30 years. A series of RS approaches and
models including content-based filtering, collaborative filtering, and hybrid
approaches have been developed and deployed, most of which have achieved good
recommendation performance. In recent years, benefiting from the advancement
of machine learning, especially deep learning, more powerful and accurate RS
models have been proposed (Ricci et al., 2022; Zhang et al., 2019).

Although tremendous successes have been achieved in the RS area, the majority
of existing research works has focused on the improvement of the system
accuracy, while only some minor part of the literature has also taken some
other important properties and values of RSs into account, such as diversity
and novelty (Castells et al., 2015). In fact, existing accuracy-oriented RSs
have not properly considered end users in the broader context of human-machine
interaction where other important factors are commonly considered as critical
for the overall user experience (Konstan and Terveen, 2021). There are some
additional aspects that requires particular attention. Firstly, the cyberspace
where RSs are deployed is becoming increasingly complex and full of threats
coming from various sources and of diverse nature, including cyber-attacks (De
Arroyabe et al., 2023), noisy and fake information (Wang et al., 2022) and
system bias (Chen et al., 2023). This triggers the demand of trustworthy RSs
(TRSs) which aim to compete in the complex and challenging cyberspace.
Secondly, stakeholders including users, owners and regulators of RSs, have
increasingly higher demand for RSs (Abdollahpouri et al., 2020). These
stakeholders not only demand recommendation accuracy, but also need
trustworthiness, including robustness, fairness, explainability, privacy-
preservation, etc. Actually, trustworthiness is even more important than
accuracy in some critical and sensitive domains, including finance and
medicine, where highly reliable RSs are required. In practice, it has become a
consensus both in the academia and industry that accuracy should not be the
only focus of an RS, and trustworthiness must be prioritised. These analyses
have triggered the urgent demand of a new RS paradigm, i.e., TRSs.

Some researchers have surveyed the existing work in the area of TRSs. For
instance, Ge et al. (Ge et al., 2022a) systematically surveyed the techniques
for key aspects related to trustworthy recommendation, including
explainability, fairness, privacy-preserving, robustness and user
controllability in recommendation. Similarly, Fan et al. (Ge et al., 2022a)
conducted a comprehensive overview of TRSs, specially focusing on six most
important aspects including safety & robustness, nondiscrimination & fairness,
explainability, privacy, environmental well-being, and accountability &
auditability. Jha et al. (Jha et al., 2021) presented an exhaustive survey to
emphasize the current research in the field on TRS models. Jin et al. (Jin et
al., 2023) surveyed existing methodologies and practices of fairness in
recommender systems, which is an important aspect of TRSs. These surveys
mainly focus on summarizing existing work and progress in the filed of TRS,
while lacking a high-level overview and a unified framework for TRSs, totally
differentiating them from our work.

In practice, researchers lack a systematic overview of TRSs, a rapidly
developing field. There is neither existing work to systematically discuss the
fundamental concepts, critical challenges, nor work to provide a unified
framework for TRs, which are essentially quite important for the further
development of TRSs. Therefore, it is in an urgent demand to provide a high-
level overview on TRSs to sort out the fundamental concepts, key challenges
and whole picture in this emerging area. To this end, this paper aims to
provide a comprehensive overview of TRSs. To be specific, we discussed the
motivation of TRSs in this section. Then, in Section 2, we will explore the
fundamental concepts underlying TRSs, including the novel proposal of
trustworthy recommendation ecosystem, aspects of TRSs, and examples of TRSs.
In Section 3, we will present a systematic review of the paradigm shift in the
field of recommendation, followed by an illustration of a four-stage
perspective for TRS in Section 4. In Section 5, we will systematically analyse
the critical challenges faced in building TRSs in each of the four stages. In
Section 6, we will present a conceptual framework to build TRSs and then we
will share some future directions in this vibrant area in Section 7. We will
conclude this work in Section 8.

##  2\. The concept of trustworthy recommender systems

The term “trustworthy” describes an entity on which a subject can rely to be
good, honest, sincere (Dictionaries, 2022). Specific aspects of “trustworthy”
include: trustable, reliable, dependable, faithful, honourable, creditworthy,
responsible (Zhang et al., 2022). In general, a TRS refers to an RS on which
its stakeholder can rely to be good, trustable, reliable, dependable, and
faithful.

A recommendation process is complex and interactive; it involves several
entities, such as, users, items, the RS, and even the provider of items (e.g.,
sellers). All of them actually form a recommendation ecosystem which enables
the recommendation activities (Abdollahpouri et al., 2020). In practice, these
parties also correspond to the necessary elements for recommendations: data
(i.e., user/item/provider related information, e.g., user-item interactions),
approaches and models (i.e., RS).

TRSs are actually the result of trustworthy RS approaches and models built in
a trustworthy ecosystem. Although trustworthy RS approaches and models are
pivotal for enabling trustworthy recommendations, they cannot survive without
a trustworthy ecosystem. Most of studies in the literature only discussed
trustworthy RS approaches while ignoring the significance of trustworthy
ecosystem. For instance, Ge et al. (Ge et al., 2022a) provided a timely survey
on the various techniques for enabling robust, fair, explainable or secure RSs
while without touching trustworthy ecosystems or complex challenges in
building TRSs, which greatly differentiates it from our work. To bridge this
gap, we first propose a novel concept of trustworthy recommendation ecosystem,
which is the foundation for building trustworthy RSs. In addition, trustworthy
recommendation ecosystem can well differentiate trustworthy RSs from the
generally discussed trustworthy artificial intelligence (AI) since it is
derived from the core characteristics and properties of the recommendation
scenario. Then, we illustrate the various aspects to specify the
“trustworthiness” of RSs with an emphasis on the trustworthiness of
recommendation approaches and models. Finally, we describe some specific and
straightforward examples of trustworthy RSs in the real world.

###  2.1. Trustworthy recommendation ecosystem.

A recommendation process unfolds in a complex user-RS-item interaction
scenario. Items are provided by various providers, including sellers, news
source websites, and thus providers are an indispensable part of the process.
As mentioned above, these four parties together form a basic ecosystem for
recommendation activities and they interact with each other, possibly in a
collaborative form, to complete the recommendation task. A trustworthy
recommendation ecosystem includes trustworthy users, items, providers and RSs,
which will be illustrated successively in the following paragraphs.

#### Trustworthy users

Users consuming recommendations can possess diverse characteristics and
perform various tasks in the cyberspace. Some users may not be trustworthy,
and may try to obstacle the proper functioning of RSs. For instance, it is not
uncommon to identify special users (e.g., internet water army and internet
hacker) who may attempt to perform fake or false interactions with items to
intentionally bias the recommendation models for a given personal and even
malicious purpose (e.g., biased product promotions) (Zhang et al., 2013).
Therefore, in order to build a trustworthy RS, only trustworthy users should
be authorised to operate. Trustworthy users can be roughly characterized by
two aspects: (1) the user profile information should be true, accurate, and
users’ behaviour on the recommendation platforms should reflect genuine
preferences and intentions; (2) users should maintain a good reputation in
their interaction behaviour (Wang et al., 2008; Wang and Lin, 2006), e.g.,
they should write true and objective reviews or ratings for items, and pay for
the purchased items on time.

#### Trustworthy items

Usually, there are massive quantities and diverse types of items on an online
consumption platform. Items have diverse characteristics and can bring
different impact to the RS stakeholders, which are individual users, the local
community and even the whole society. Although most of the items can well
satisfy users’ needs and can have a positive impact on the stakeholders
interests in the long run, some items may produce a negative impact on either
the individual users, the community or the society. For example, additive
electronic games usually match young children’s preferences well. But these
games often create addiction and thus can negatively affect users’ lives, work
or studies, hence resulting in significant negative impact on users and their
family. Another example is fake news, which may match some minority users’
stance and reading preferences. But they mislead the public, and may cause
serious social problems, and should not be recommended to end users (Wang et
al., 2022). To this end, only trustworthy items should be recommended by a
trustworthy RS. Trustworthy items can be characterized by two aspects: (1) the
item-related information (e.g., item attributes) used for recommendations
should be true and precise, so that the intrinsic characteristics of all items
can be easily captured; (2) the candidate items to be recommended to end users
should be trustable and responsible, namely the items must not bring potential
negative impacts to the stakeholders of RSs.

#### Trustworthy providers

Providers are the party who provide items in online platforms. For example, a
provider may be a seller in an e-Commerce platform, or a news source website
which releases original news in the news domain. In the real world, some
providers are not trustworthy since they may provide fake or poor-quality
items, leading to negative impact to the end users and the society. For
instance, some sellers may sell fake products while some news websites may
release fake news. Obviously, trustworthy providers are necessary for building
trustworthy RSs and they can be characterised by two aspects: (1) the
providers should be responsible for the quality and potential impact of the
items they provide. For instance, the sellers should provide products or
services with guaranteed quality, while the news websites should provide
verified true news only while combating fake news; (2) it is important for
providers to maintain a good reputation through their historical transaction
or releasing behaviours (Zhang et al., 2015; Wang et al., 2015).

#### Trustworthy RSs

Recommendation approaches which are usually built on data mining or machine
learning models should be reliable and stable. Specifically, they should not
only be able to accurately capture users’ preferences and items’
characteristics so as to provide accurate recommendation services to end
users, but also perform stably all the time, even in the most complex, dynamic
and challenging context (e.g., the cyberspace facing frequent cyber-attacks).
Trustworthy approaches and models are the most important part in trustworthy
RSs and they can be characterised by multiple specific aspects, which will be
discussed in detail in the next part.

###  2.2. Aspects of trustworthy recommender systems.

In some related areas, such as, trustworthy AI, many different aspects have
been proposed to specify “trustworthy” from perspectives including
regulations, mechanism, and technology (Zhang et al., 2022). Several aspects
related to the technology perspective have been commonly recognized, including
robustness, fairness, explainability, transparency, privacy, accountability,
and responsibility. Since RSs are a specific application area of AI, the
aspects used to define trustworthy AI are also applicable to characterise
trustworthy RSs. However, due to the very special and specific data
characteristics, work mechanism and computation task of an RS, there are
additional aspects, such as, human’s perception and trustworthy evaluation,
which are specific to the RS scenario and can well distinguish trustworthy RSs
from trustworthy AI. Next, we illustrate the various aspects for
characterizing trustworthy RSs.

  * •

Robustness indicates an RS that has strong fault-tolerant capability to
survive attacks, noisy and fake information (e.g., such as users’ arbitrary
behaviours and fake reviews/ratings on items) (Mobasher et al., 2007; Liu et
al., 2022), which often commonly affect the input data used for
recommendations.

  * •

Fairness indicates that the RS is able to reduce or remove possible bias
towards certain groups of stakeholders (e.g., the disparities in treating user
with different demographics or items with low and high popularity) which may
be present in the input data, the recommendation model or in the
recommendation results (Mehrabi et al., 2021; Ge et al., 2022b; Wang et al.,
2023a). Consequently, the final recommendation results can be assessed as fair
from both the user side and the provider side.

  * •

Transparency generally means that the functioning mechanism and the
recommendation model are transparent to all the stakeholders of the RS, and it
is not perceived as a “black box” (Balog et al., 2019; Di Noia et al., 2022).
Transparency can greatly reduce the potential risks of deploying ineffective
RSs in real-world domains, especially in some sensitive and critical domains
like finance and healthcare domains.

  * •

Explainability means that the recommendation mechanism, recommendation models
and the recommendation results are well-explained so that the stakeholders of
the RS can properly understand how and why the recommendations are generated
(Zhang et al., 2020; Li et al., 2023).

  * •

Privacy and security means that the RS is able to effectively protect the
personal privacy of relevant stakeholders (Shin et al., 2018; Imran et al.,
2023). This aspect is even more important in RS, compared to other web
applications, since RSs are consuming a large amount of profile and behaviour,
which often contains a sensible users’ characteristics.

  * •

Responsibility. The development and deployment of RSs should be conducted in a
responsible manner (e.g., follow the regulations and law), so that the RS will
not explicitly or implicitly harm anybody. Moreover, the generated
recommendation results should be beneficial to the all stakeholders (Elahi et
al., 2021). For instance, fake news should not be recommended to any users
even if it may interest some ones, or addictive games should not be
recommended to young children either.

  * •

Human’s perception of trustworthiness, relates to the subjective perception,
which the relevant stakeholders have, of the trustworthiness of an RS. This is
unique to RSs and is of great significance. In fact, an RS is actually
involving a human-machine interaction (Jannach et al., 2016), and whether an
RS is trustworthy or not, ultimately depends on human’s feeling and
perception. However, almost all of the existing approaches to model
trustworthy RSs have been developed by assuming a single point of view, that
of the machine (Wu and et al., 2012), i.e., the RS models and algorithms,
while have ignored the human’s factor. Moreover, existing work often
mechanically computes a numerical measurement value to quantify the
trustworthiness of RSs. Such lack of consideration of the human’s factor
triggers the urgent demand to take human’s perception into account when
talking about trustworthy RSs.

Figure 1. The various aspects to characterise TRSs and the relations among
them. According to the characterization perspective, they can be organized
into three layers.

  * •

Trustworthiness integration. Although various aspects and perspectives have
been proposed to characterise trustworthy RSs, they are mostly separated from
each other. In addition, most of the existing studies on trustworthy RSs have
focused on one or two aspects of trustworthiness alone. For instance,
fairness-aware RSs and explainable RSs only focus on the fairness and
explainability aspect respectively. However, accounting for only one or two
aspects can not lead to a truly trustworthy RSs. In order to build truly
trustworthy RSs, all the aforementioned different aspects, including both
objective and subjective aspect, should be integrated together organically and
effectively towards to a unified trustworthy RS framework.

  * •

Trustworthy evaluations. On one hand, both the evaluation methods and the
evaluation metrics must be reliable to well and precisely reflect the
performance of the RS. One example is that most offline evaluations are
performed under a very ideal situation and thus they cannot actually indicate
how an RS will perform in the real-world online scenarios (Gunawardana and
Shani, 2015). On the other hand, new evaluation protocols and metrics are in
demand to evaluate the trustworthiness of an RS so that all the aforementioned
aspects can be appropriately validated.

Although these different aspects touch a variety of issues and often
correspond to different elements and stages of an RS, they are closely inter-
connected and must be jointly addressed to contribute to the “trustworthiness”
of the RS, as illustrated in Fig. 1. For instance, fairness often relates to
explainability. In fact, the fairness perspective (e.g., users’ gender or item
popularity) that is considered and how fairness is assured should be
explainable to the end users, so that they can clearly understand and accept
the generated fair recommendation results.

###  2.3. Examples of trustworthy recommender systems in the real world.

For decades, RSs have been widely exploited in nearly all information systems
that we use daily for work, study or entertainment. In recent years, along
with the occurrence of an increasing number of cyber threats including cyber
attacks, fake or false online information, bias in the cyber space,
trustworthy RSs are an evident need. There is a variety of examples of real-
world trustworthy RSs across different application domains. In E-commerce
domain, one typical example is the robust RS which can tame fake ratings and
reviews (Lyu et al., 2021) in platforms like Ebay or Amazon. This type of RSs
detects fake ratings and reviews inserted in the platform and discard them
when generating trustworthy recommendations. In media domain, a typical
example is a recently proposed fact check technology (e.g., Google fact check
tools) for the news domain, which introduces a series of techniques to check
the veracity of news on the web so that only verified true news will be
recommended to end users (Giansiracusa, 2021). Another typical example is a
veracity-aware news recommender system published at the 2022 Web Conference
(Wang et al., 2022). This system checks the veracity of each news piece before
generating recommendations, so that only checked true news will be recommended
to end users.

##  3\. The Recommendation Paradigm Shift

###  3.1. From accuracy-oriented RSs to trustworthy RSs

RS research can date back to 1990s when the first collaborative system
“Tapestry” was reported in the literature (Goldberg et al., 1992). From early
1990s to the middle of 2000s, several algorithms have been developed for
enabling content-based or collaborative filtering recommendations (Ricci et
al., 2022; Wu et al., 2022). From 2008 to 2016, driven by the popular Netflix
challenge, matrix factorization methods have dominated the scenario for a long
time. Since the middle of 2010s, benefiting from the rapid development of deep
learning techniques, deep learning based RSs have become the main stream in
recommendation domain (Zhang et al., 2019).

Along with that technological shift and revolution of RSs, the focused
research problems, challenges and evaluation mechanisms of RSs have also
changed. Since the first studies on RSs to early 2010s, the accuracy of
recommendation results has been the most important and often the only
evaluation criteria that was used to evaluate an RS. Specifically, that means
that the researchers tried to assess whether the recommended items were truly
preferred (e.g., clicked, purchased, highly rated) by users. During this
period, nearly all efforts were devoted to develop more accurate RS models and
algorithms, such as more accurate matrix factorization algorithms to predict
the unknown user-item ratings with a smaller and smaller prediction error.
Then, starting from early 2010s, more and more researchers realized that the
accuracy criterion is insufficient for evaluating an RS, and thus new
evaluation dimensions, including diversity and novelty were proposed
(Adomavicius and Kwon, 2011; Gunawardana and Shani, 2015). As a result, the
evaluation of an RS moved from single-criteria evaluations, based on accuracy,
to multi-criteria evaluations based on a range of dimensions, such as
accuracy, diversity and novelty, while accuracy still was considered as the
most important criteria.

Since late 2010s, due to an increasing level of threats in the cyberspace,
coming from cyber attacks, noises, and biases, the trustworthiness of RSs have
attracted increasing attention from both academia and industry. As a result, a
growing number of studies have focused on trustworthy RSs, from various
perspectives, such as robustness, fairness, explainability or privacy. For
example, some work aimed to build robust RSs which can fight against shilling
attacks (Gunes et al., 2014), and fairness-aware RSs to generate fair
recommendations for users with different demographics (Xiao et al., 2017).
Some other work have instead focused on building explainable RSs to generate
user-understandable recommendation results (Zhang et al., 2020), or secure and
privacy-preserved RSs which puts a special concern on users’ privacy (Shin et
al., 2018). Obviously, the research focus and the ultimate goal of these
studies is no longer centered on the improvement of recommendation accuracy.
Instead, they have changed the direction towards enhancing the trustworthiness
of RSs. This has actually revolutionized the research focus and the evaluation
perspectives, from accuracy-oriented recommendations towards trustworthiness-
oriented recommendations, leading to a recommendation paradigm shift.

In practice, the trustworthiness of RSs has been attracting much attention
from different sectors including academia, industry and government. A typical
example is that the Chinese government released regulations on recommendation
algorithms for internet information services in early 2022 (of China, 2022). A
large proportion of the regulations emphasised the significance of trustworthy
RSs to be deployed in the real-world business applications.

###  3.2. From trust-aware recommender systems to trustworthy recommender
systems

Trust-aware recommender systems mainly consider and leverage the trust
relationships between humans and organizations to enhance the recommendation
performance. Here the trust relationships can vary, some typical examples
include the trust relations between different users on online or offline
social networks (e.g., friends on facebook, or colleagues in one
organization), the trust relationships between users and comments, ratings on
items from others, the trust relationships between customers and sellers
(Zhang et al., 2012; Zheng et al., 2014). Since early 2000s, a variety of
studies (Massa and Avesani, 2007; Dong et al., 2022; Ahmadian et al., 2022;
Zhang et al., 2015) have been done to investigate how to well to model and
leverage such diverse trust relationships to improve the performance of RSs.

Differently from trust-aware RSs, trustworthy RSs aim to build reliable RSs
which can be accepted by humans, which were proposed in late 2000s for the
first time (Mobasher et al., 2007). However, early definitions on trustworthy
RSs mainly focused on one single aspect of trustworthiness, namely robustness
(Mobasher et al., 2007; Wu and et al., 2012). So trustworthy RSs in the early
stage mainly refers to those RSs which are robust while facing various types
of attacks in the cyberspace. However, they are not fully trustworthy since
there are some other important aspects contributing to trustworthiness that
were initially overlooked, including the aforementioned fairness,
explainability, privacy and responsibility.

Therefore, it is of great theoretical and practical significance to develop
the next-generation trustworthy RSs which go beyond the existing robustness-
oriented ones. To this end, in this paper, we propose novel trustworthy RSs in
which all the nine aforementioned aspects contributing to trustworthy RSs are
considered, leading to a significant step towards the implementation of truly
trustworthy RSs.

##  4\. Trustworthy Recommender Systems: A Four-stage Perspective

#### Classical Recommendation Process

Generally speaking, a typical machine learning-based recommendation process
comprises the following four successive and inter-connected stages:

  * •

Data preparation stage, including data collection and preprocessing;

  * •

Data representation stage, which often learns an informative representation of
the raw input data as the input of the downstream recommendation model;

  * •

Recommendation generation stage, which deploys some data mining or machine
learning based prediction models to predict the unknown user-item
interactions, e.g., ratings, clicks;

  * •

Performance evaluation stage, which employs various evaluation methods,
including online and offline evaluations, to evaluate the recommendation
results from different perspectives such as accuracy, diversity, and fairness.

#### Trustworthy Recommender Systems

To be trustworthy, an RS should be trustworthy in each of the aforementioned
four stages. Each stage has its unique computation task, characters and goal,
so the “trustworthiness” of different stages often has different specific
meanings and focuses on different aspects of the aforementioned nine aspects
(cf. Section 2.2). For instance, data preparation stage often focuses more on
the “robustness” aspect, since the data preparation methods should be robust
enough to survive to noisy and fake data, possible coming from attacks. While
the recommendation generation stage should focus more on aspects like
“fairness”, “transparency” and “robustness”, since fair, transparent and
robust RS models and algorithms are required in this stage for generating
recommendations. Next, we try to characterize a trustworthy RS from a four-
stage perspective while the specific meaning of “trustworthy” in each stage
will be highlighted.

  * •

Robust and secure data preparation, on one hand, is necessary to combat noisy
and fake information, and bias in the raw data, generated in the cyberspace,
and survive to possible attacks on the data. In this way, the prepared data
for recommendations can accurately reflect users’ preferences and items’
characteristics. On the other hand, the users’ data should be managed and
processed in a secure way so that sensitive and privacy information will not
be leaked.

  * •

Robust and explainable data representation, which employs robust and
explainable data representation models to precisely learn informative and
interpretable latent representations of the original input data in a stable
way regardless of the potential attacks. In most cases, the learned latent
representations are latent vectors without an explicit semantic meaning, which
then negatively impacts on explainability. In addition, it is often not very
clear which information is encoded in the representations. Therefore, the
“explainability” aspect should be emphasized in this stage.

  * •

Fair, transparent, explainable and responsible recommendation generation, in
which fair, transparent and explainable RS models should be adopted to provide
fair exposure opportunities of all items to different users in a transparent,
explainable and responsible way. In addition, the models should also be able
to provide straightforward explanations by using texts or images corresponding
to the recommendation results, to faithfully explain how the recommendations
are generated and why they are appropriate for a given user. More importantly,
only those items with positive impact to stakeholders can be recommended.

  * •

Trustworthy evaluations comprise two-sided evaluations: (1) technical
evaluation, and (2) ethical evaluation. The technical evaluation aims to
evaluate the recommendation performance from some technical perspectives such
as accuracy, diversity, novelty, and explainability. More trustworthy and
reliable evaluation protocols and metrics are required to evaluate such
technical performance. In contrast, the ethical evaluation aims to evaluate
recommendations from the ethical perspective such as the responsibility and
social impact. For instance, whether the recommendation activities and the
recommendation results are responsible and beneficial to the relevant
stakeholders. Towards trustworthy RSs, ethical evaluations are necessary and
of great significance to the sustainable development of the community and the
society. Some examples of unethical but maybe preferable recommendations
include the recommendations of fake news (Wang et al., 2022) and the
recommendation of addictive games to young children. To the best of our
knowledge, this is the first time the proposal of an ethical evaluation of RSs
is put forward in the literature.

##  5\. Challenges for Building Trustworthy Recommender Systems

Figure 2. The challenges and corresponding strategies for building TRSs.

Although the significance and urgency of trustworthy RSs have been commonly
recognised, the research in this area is still in an early stage. A variety of
challenges and research problems remain to be properly addressed in each of
the aforementioned four stages. We therefore analyze the particular challenges
of each stage.

#### Challenges in data preparation stage.

Before building an RS model, a necessary step is to collect and prepare data
to train and test the model (Wang et al., 2023b). Such data often contains
user-item interactions such as users’ clicks, purchases, ratings, or comments
on items, and side information such as users’ attributes, social relations,
and items’ features (Jannach et al., 2021). The data is often collected from
online consumption platforms such as amazon.com, youtube.com, and
abcnews.go.com. Due to the large amount of diverse users (e.g., 10M+ users) on
each platform, it is inevitable that some users’ behaviours may be noisy and
unreliable (e.g., randomly click or view a set of items on amazon.com), or
there may be some malicious users (e.g., internet water army) who have
performed malicious behaviours, e.g., posting fake ratings or comments to some
targeted items or sellers. These factors bring noise, bias, and fake
information into the interactions or side information, making the data used
for recommendations unclean and unreliable. As a result, the data can not
accurately reveal the users’ preferences and item characteristics, and thus
misleads the downstream recommendation models.

Therefore, a big challenge at the data preparation stage is to effectively
detect and remove critical data, e.g., noisy, fake, or biased data, and retain
the genuine data. This task is quite challenging since this critical data
usually accounts for a quite small proportion of the whole population, making
it very difficult to discover it, especially when critical data looks quite
similar to the genuine data. Possible strategies to address this challenge
include a set of conventional and emerging data preprocessing techniques. For
instance, some advanced data augmentation and data debias methods (Schnabel et
al., 2016), including counterfactual data augmentation (Wang and et al., 2021)
and contrastive learning (Tian et al., 2020), have been extensively studied to
remove the bias in raw data in recent years.

Another challenge at this stage is related to the data privacy and security
issue. This is especially important in real-world RS applications since the
data used for recommendations often contains a large amount of users’ private
information which should be protected. To be specific, on one hand, during the
collection of raw data, it is important to ensure that only the really
required data will be collected. On the other hand, when a large amount of
data is collected, it is important to build a safe and strong infrastructure
and mechanism to properly store and manage it, including secure and powerful
database, and rigorous data access mechanism. Another way to address the data
privacy and security issue is to introduce federated learning (Yang et al.,
2019) and edge computing (Shi and et al., 2016). In this way, the data used
for recommendations can be decentralized to various edge devices such as
users’ mobile phones where the individual users’ privacy can be well
protected.

#### Challenges in data representation learning stage.

Data representation aims to learn an informative numerical vector in a latent
space to well represent each data point in the original input dataset. For
instance, the well-known representation learning model Word2Vec (Rong, 2014)
learns a latent numerical low-dimensional vector to represent each word.

In advanced machine learning, especially in deep learning area, data
representation learning is a prior step of all the downstream learning models
and tasks. The quality of data representation directly determines the
performance of the learning models. Therefore, machine learning based RS
models often consists of two main steps: data representation learning and
prediction. The output of the first step is taken as input of the second for
the prediction task. For instance, in matrix factorization or neural network
based RS models, each user/item is represented by a K-dimensional latent
vector which is learned by mapping the user/item ID into a latent space (Ning
et al., 2015). The main challenges in representation learning stage lie in the
following three aspects.

First, how to learn accurate and informative representations to precisely
retain the original information in the raw data? For instance, the
similarities/distances between the raw data points should be accurately
reflected by their corresponding representations in the latent space. Towards
this challenge, some advanced representation learning techniques including
contrastive representation learning (Xiong et al., 2020) are able to learn
more accurate and informative representations.

Second, how to learn interpretable representations? One common and critical
issue with latent representations is the lack of explainability. It is often
not clear what the semantic meaning of the latent vectors is, making the
learned representations uninterpretable for humans. Sometimes, it is not clear
what kind of information is encoded in the latent vectors either, which brings
some potential risks for the real-world applications of RS models. To address
this challenge, some explainable representation learning models and techniques
including data visualization can be developed to disclose the patterns encoded
in the latent representations.

Third, how to combat attacks on representation learning models? In general,
nearly all machine learning models including representation learning models
may suffer from some external attacks, especially those deployed in real-world
applications. The attackers may quickly generate a large amount of fake data
to intentionally mislead the models and thus downgrade the quality of the
learned representations. Some possible strategies include robust
representation learning techniques, such as adversarial representation
learning which fights against attacks in an adversarial way (Donahue and
Simonyan, 2019).

#### Challenges in recommendation generation stage.

In this stage, the data representations learned in the previous stage will be
imported into the selected RS model for generating recommendations. The key
challenge lies in how to build a robust, fair, transparent, explainable and
responsible RS model which can work in a trustworthy way in a complex and
dynamic environment.

First, similarly to what was observed for the representation learning stage,
RS models also easily suffer from various attacks. So the first challenge is
how to build robust RS models which can survive various attacks especially
those malicious attacks and adversarial attacks in the complex cyberspace
where RSs are deployed.

Second, some particular design and work mechanisms of the RS models may lead
to biased RS models and thus generate biased recommendations. Therefore,
another challenge in this stage lies in how to effectively remove the bias.
Various model debias techniques like auto-debias which learns the debias
parameter from the data (Chen et al., 2021) can address this issue to some
degree.

Third, advanced machine learning based RS models, especially deep learning
based ones often run in a black-box mode, and thus the computations in the
model are not transparent and explainable to humans. This makes it hard to
understand how an RS model is working and what potential risks and
disadvantages the model may have. Hence, this may bring potential risks for
the applications of RS models in real-world, especially in some social and
economic domains which are critical, such as, healthcare and finance, where a
small mistake may lead to a large loss (e.g., the loss of a life or a large
sum of money). Hence, the third challenge in this stage lies in how to build
transparent and explainable RS models whose work mechanism are well
understandable to humans. In this way, how and why the recommendations are
generated can be well disclosed and visible to end users, which greatly
reduces the risks of the employment of RSs in real-world business.

Fourth, considering the real-world application context, some recommendation
behaviours may be profit-driven while ignoring the ethical or even the legal
issues (e.g., recommending additive games to children). Some recommendation
results may be accurate but not responsible, such as recommendations of fake
news to particular users (Wang et al., 2022). Hence, how to ensure responsible
recommendation behaviours and recommendation results which are for social good
is another big challenge in this stage.

#### Challenges in the evaluation stage.

In this stage, the recommendation results, e.g., predicted user-item rating
values, or a list of ranked items, generated by an RS model are evaluated to
measure its performance. Conventionally, the evaluation metrics could be a
variety of quantity metrics of concern such as accuracy, and diversity, which
can directly measure the performance from the technical perspective. However,
as mentioned in the Introduction, trustworthy evaluation should go beyond
conventional technical performance evaluation. Instead, they should contain
both technical evaluation and ethical evaluation.

For the technical evaluation, existing evaluation protocols and metrics for
RSs are mostly accuracy-oriented (Wu et al., 2022). They are not sufficient to
comprehensively evaluate an RS, especially in the complex cyberspace which
often contains noises, attacks, and bias (Huang et al., 2018). In such a
complex context, the stakeholders of an RS usually focus not only on accuracy,
but also on trustworthiness. Hence, the effective evaluation of the
trustworthiness of an RS has a great social and economic significance.
However, the evaluation of the trustworthiness of an RS is quite challenging,
and involves multiple aspects, such as the robustness, fairness,
explainability and privacy. Some of them are hard to be quantified in a
traditional online or offline experiment. In addition, different application
scenarios often have different requirements on the trustworthiness of the RS
and focus on different aspects. For instance, in an online e-commerce platform
where attacks and noise behaviours are routinely detected, the robustness of
the RS should be a primary attention point. Conversely, in an offline RS for
medicine treatment, the transparency of the RS model and the explainability of
the recommendation results may be more important. Therefore, it is of great
importance to define novel trustworthiness evaluation schemes for RSs,
including both new evaluation protocols and evaluation metrics for
comprehensively and systematically measuring the trustworthiness of RSs.

Ethical evaluation aims to evaluate an RS from the ethical and social
perspectives. Specifically, they measure whether the recommendation behaviours
and recommendation results will have a positive impact on the various
stakeholders. Here, the impact of an RS could be on the recognition or the
behaviours of end users, and it could be explicit or implicit, short-term or
long-term. Ethical evaluations are quite challenging since the various impact
and influence of RSs on both individual level and the society level are very
difficult to accurately capture and quantify (Hazrati and Ricci, 2022). New
and well-designed ethical evaluation methods are in demand for RSs, which may
combine both qualitative and quantitative evaluations.

##  6\. A Framework for Trustworthy Recommender Systems

To build a trustworthy RS is therefore a challenging task. Not only various
aspects of trustworthiness, such as robustness, fairness and security, and the
complex relations among them, should be well considered in a unified way, but
also new and trustworthy evaluation protocols and metrics are in order. Each
aspect often involves specific models and techniques. For instance, robustness
is often achieved via adversarial learning techniques such as generative
adversarial networks (Deldjoo et al., 2021). Due to its extreme complexity, it
is challenging to design a very specific model for trustworthy RSs which can
well handle all the aspects. However, a systematic methodology to build a
comprehensive trustworthy RS while considering all the different aspects in
different stages is in a compelling challenge and there is the need of a
unified framework for trustworthy RSs. Hence, to bridge this gap, we propose a
conceptual framework to support trustworthy RSs.

Consistent with the four stages of recommendation process discussed in Section
4, our proposed trustworthy RS framework consists of four stages: (1)
trustworthy data preparation, (2) robust and explainable data representation,
(3) fair, transparent, explainable and responsible recommendation generation,
and (4) trustworthy evaluation, as is is depicted in Figure 3.

Figure 3. A conceptual framework for building TRSs.

#### Trustworthy data preparation.

Trustworthy data preparation aims to first collect raw data (e.g., user-item
interaction data and side information data) from online platforms in a
trustworthy way (e.g., privacy-preserved) and then transfer it to trustworthy
input data (e.g., clean, unbiased and reliable data) for the downstream
recommendation tasks. Note that the collected raw data often contains noises,
bias and false information, etc. Some conventional and advanced data
preprocessing techniques can be employed to support trustworthy data
preparation. For instance, some data cleaning methods including pattern-based
error detection (He et al., 2005) and machine learning based outlier detection
(Pang et al., 2021) can be utilized to detect and remove those noisy and fake
data in the raw dataset. In recent years, some advanced machine learning
approaches like counterfactual data augmentation (Wang and et al., 2021) and
contrastive learning based data augmentation (Tian et al., 2020) have been
commonly utilized to reduce the bias in input data for recommendations.

#### Robust and explainable data representation.

In principle, there are two specific steps for achieving trustworthy data
representations: (1) representation learning to learn user and item
representations, and (2) representation quality evaluation to evaluate the
learned representations. In RS area, the evaluation step is often incorporated
into the downstream recommendation task, i.e., the representation quality is
measured by the quality of recommendation results.

For the representation learning step, as discussed in Section 4, trustworthy
can be specified into robust and explainable. On the one hand, to support the
robustness of data representation learning models so that they can perform
stably when facing attacks and noises, one typical solution is adversarial
representation learning models (Donahue and Simonyan, 2019). In addition, to
remove the possible biased and noisy information hidden in the
representations, some representative debiasing and denoising models including
Denoising Autoencoder (DAE) (Lu et al., 2013) can be deployed for more
accurate representation learning for recommendations. On the other hand, in
order to make the learned latent representations more interpretable for
humans, some data visualization techniques such as t-SNE (Van Maaten and et
al., 2008) can be helpful to explicitly reveal the hidden patterns in latent
data.

#### Fair, transparent, explainable and responsible recommendation generation.

In the recommendation generation stage, the core task is to design and develop
trustworthy RS models. As discussed in Section 4, trustworthy in this stage
can be characterised along four dimensions: fair, transparent, explainable,
and responsible. We illustrate them in the following.

First, a trustworthy RS model must be free of bias so that it can perform
equally well for different users and items. In support of this,
disentanglement learning based approaches (Ma et al., 2019) can be used for
removing bias in RS models. In addition to RS models, bias may also exist in
recommendation results, which could be reduced by re-ranking techniques based
on some particular constraints.

Second, trustworthy RS models should be transparent. To support the
transparency, some set-based techniques can be employed to explicitly present
the RS models to end users via natural language (Balog et al., 2019). Building
transparent RS models is quite challenging and more studies are deserved.

Third, to support the explainability of recommendation results, the output of
recommendation generation stage should be two-sided: (1) the normal
recommendation results, e.g., a list of selected candidate items, and (2)
corresponding explanations of the recommendation results to well explain how
and why the recommendations are generated. The explanations should be
straightforward and well-understandable, either in textual or visual formats
(Lu et al., 2019).

Last but not the least, to be trustworthy, the recommendation generation
process must be responsible in terms of both the recommendation behaviours and
the recommendation results. On one hand, some additional constraints and
criteria can be employed to ensure the above aspects towards social good. For
instance, some models can be developed to filter out those fake products,
additive games and fake news from recommendation lists. On the other hand,
some regulations may be enforced to guide the harmonic development and
deployment of RS techniques.

#### Trustworthy evaluation.

Evaluation is an important step for ensuring the quality of recommendation
results generated by RS models. As depicted in Figure 3 and discussed in
Section 4, both technical evaluation and ethical evaluation are required for a
trustworthy RS. Regarding technical evaluation, most of the existing
evaluations are accuracy-oriented, while trustworthiness evaluations are in
their early stage. Although a few metrics have been defined to measure one
specific aspect of trustworthiness, such as fairness (Leonhardt and et al.,
2018), there is no unified method to systematically evaluate multiple aspects
as discussed in Section 2. New evaluation protocols and metrics are in demand
for comprehensive trustworthiness evaluations.

Regarding ethical evaluation, to the best of our knowledge, there is no
existing work reported in the literature. Ethical evaluation mainly evaluates
the social impact of recommendation behaviours and results. For instance,
whether the recommendation behaviours are legal and ethical or if the
recommendation results lead to some good/bad impact (e.g., lead users to
become addicted to video games). Ethical evaluation is full of complexity and
dynamics, in which the users’ cognition, perception and behaviours should be
taken into account. To this end, some possible solutions include user studies,
and the combination of qualitative and quantitative studies.

##  7\. Future directions

Truly trustworthy recommender systems. Although there are existing work aiming
at building trustworthy RSs, nearly all of them focus on only one single
aspect (e.g., fairness or explainability). For instance, some early work on
trustworthy RSs only focuses on the robustness of RSs (Mobasher et al., 2007),
aiming to build resilient RSs to combat attacks. More recently, some other
work focuses on either explainability (Zhang et al., 2020), security (Lam et
al., 2006) or fairness (Leonhardt and et al., 2018). However, to be truly
trustworthy, usually all these various aspects (cf. 2) of trustworthiness
should be carefully considered in a unified way. Therefore, more analyses are
in demand to build truly trustworthy RSs.

Human-centered trustworthy recommender systems. Although some studies have
attempted to build trustworthy RSs, almost all of them only mechanically
developed the so-called trustworthy models from the machine perspective, while
ignoring the human factor. However, weather an RS is trustworthy or not should
be ultimately determined by the relevant stakeholders, rather than by the RS
model itself. Therefore, human’s perception and judgement should be well
incorporated when developing and evaluating a trustworthy RS, which deserves a
deeper investigation.

Trustworthy recommendation ecosystems. As discussed in Section 2, trustworthy
RSs can not survive without a trustworthy recommendation ecosystem. However,
almost all the existing work on trustworthy RSs only focus on the development
of trustworthy RS models while ignoring other important elements, e.g.,
trustworthy items and providers, in the ecosystem. Therefore, more studies in
the development of trustworthy recommendation ecosystems are in order.

Multi-granular fairness aware trustworthy recommender systems. Fairness is
essentially multigranular and exists at different levels (Mehrabi et al.,
2021), such as the high-level fairness between different groups, and low-level
fairness between different individuals within each group. A really fair RS
should be fair in all the different granularities rathern than only a single
granularity as done in most of the existing work. Hence, to comprehensively
model multigranular fairness is of great significance for building truly fair
RSs.

Fine-grained personalized recommender systems. Although it is a common sense
that personalization is the core aspect and the basis of RSs, the
recomemndation results of most existing RSs are not personalized enough since
they are mostly preferred by the majority users only while unsatisfied by the
minority ones. Therefore, it is in highly demand for designing innovative RS
models and evaluation approaches to ensure fine-grained personalization of
recommendation results which can satisfy nearly each user.

Responsible recommender systems. Most of existing RSs are technique-driven,
which are built on advanced machine learning techniques and are evaluated from
the technical perspective only. The social impact of RSs are hardly considered
during the design, development and evaluation of RSs. However, RSs are not
only a technique issue, but also a social issue. More studies are deserved to
explore how to well incorporate social impact and social science into the
whole life-cycle of RSs.

New evaluation protocols and metrics. As discussed in Section 2, building
trustworthy RSs triggers new challenges for RS evaluations. On one hand, new
technical evaluation protocols and measures are in demand to comprehensively
measure the trustworthiness of RSs. On the other hand, new ethical evaluation
methods are needed to effectively measure the social impact of RSs.

##  8\. Conclusions

Trustworthy recommender system (TRS) is an extremely challenging yet demanding
topic, which is of both great theoretical and practical values. In this paper,
we have provided a comprehensive overview of this topic. We comprehensively
characterized trustworthy RSs by thoroughly analyzing the various aspects
specifying the trustworthiness of RSs. We provided a novel four-stage
framework to systematically illustrate trustworthy RSs and analyzed the
corresponding challenges in each of the four stages in building trustworthy
RSs. We have also described a conceptual framework to support trustworthy RSs
and pointed out some future directions in this vibrant area. The research in
trustworthy RS is flourishing and it is our hope that this work can provide
readers with a comprehensive understanding of the key aspects, main
challenges, key techniques when building trustworthy RSs and shade some light
on future studies.

###### Acknowledgements.

Prof. Huan Liu from Arizona State University has provided some feedback on the
earlier version of this work. This work was supported by Australian Research
Council Discovery Project DP200101441.

## References

  * (1)
  * Abdollahpouri et al. (2020) Himan Abdollahpouri, Gediminas Adomavicius, Robin Burke, and et al. 2020.  Multistakeholder recommendation: Survey and research directions.  _User Model. User Adapt. Interact._ 30, 1 (2020), 127–158.
  * Adomavicius and Kwon (2011) Gediminas Adomavicius and YoungOk Kwon. 2011.  Improving aggregate recommendation diversity using ranking-based techniques.  _IEEE Trans Knowl Data Eng_ 24, 5 (2011), 896–911.
  * Ahmadian et al. (2022) Milad Ahmadian, Mahmood Ahmadi, and Sajad Ahmadian. 2022\.  A reliable deep representation learning to improve trust-aware recommendation systems.  _Expert Systems with Applications_ 197 (2022), 116697.
  * Balog et al. (2019) Krisztian Balog, Filip Radlinski, and Shushan Arakelyan. 2019\.  Transparent, scrutable and explainable user models for personalized recommendation. In _SIGIR_. 265–274.
  * Castells et al. (2015) Pablo Castells, Neil J. Hurley, and Saul Vargas. 2015\.  Novelty and diversity in recommender systems.  In _Recommender Systems Handbook_ , Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). Springer, 881–918.
  * Chen et al. (2021) Jiawei Chen, Hande Dong, and et al. 2021.  AutoDebias: learning to debias for recommendation. In _SIGIR_. 21–30.
  * Chen et al. (2023) Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. 2023\.  Bias and debias in recommender system: A survey and future directions.  _ACM Transactions on Information Systems_ 41, 3 (2023), 1–39.
  * De Arroyabe et al. (2023) Ignacio Fernandez De Arroyabe, Carlos FA Arranz, Marta F Arroyabe, and Juan Carlos Fernandez de Arroyabe. 2023\.  Cybersecurity capabilities and cyber-attacks as drivers of investment in cybersecurity systems: A UK survey for 2018 and 2019\.  _Computers & Security_ 124 (2023), 102954.
  * Deldjoo et al. (2021) Yashar Deldjoo, Tommaso Di Noia, and Felice Antonio Merra. 2021\.  A Survey on adversarial recommender systems: from attack/defense strategies to generative adversarial networks.  _ACM Comput. Surv._ 54, 2 (2021), 1–38.
  * Di Noia et al. (2022) Tommaso Di Noia, Nava Tintarev, Panagiota Fatourou, and Markus Schedl. 2022.  Recommender systems under European AI regulations.  _Commun. ACM_ 65, 4 (2022), 69–73.
  * Dictionaries (2022) O. L. Dictionaries. 2022\.  Definition of trustworthy.  https://www.oxfordlearnersdictionaries.com/definition/americanenglish/trustworthy.
  * Donahue and Simonyan (2019) Jeff Donahue and Karen Simonyan. 2019.  Large scale adversarial representation learning. In _NIPS_. 10542–10552.
  * Dong et al. (2022) Manqing Dong, Feng Yuan, Lina Yao, Xianzhi Wang, Xiwei Xu, and Liming Zhu. 2022\.  A survey for trust-aware recommender systems: a deep learning perspective.  _Knowledge-Based Systems_ (2022), 108954.
  * Elahi et al. (2021) Mehdi Elahi, Dietmar Jannach, Lars Skjærven, and et al. 2021.  Towards responsible media recommendation.  _AI and Ethics_ (2021), 1–12.
  * Ge et al. (2022a) Yingqiang Ge, Shuchang Liu, Zuohui Fu, and et al. 2022a.  A survey on trustworthy recommender systems.  _arXiv preprint arXiv:2207.12515_ (2022), 1–43.
  * Ge et al. (2022b) Yingqiang Ge, Juntao Tan, Yan Zhu, Yinglong Xia, Jiebo Luo, and et al. 2022b.  Explainable fairness in recommendation. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 681–691.
  * Giansiracusa (2021) Noah Giansiracusa. 2021\.  Tools for truth.  In _How Algorithms Create and Prevent Fake News_. Springer, 217–229.
  * Goldberg et al. (1992) David Goldberg, David Nichols, Brian M Oki, and Douglas Terry. 1992.  Using collaborative filtering to weave an information tapestry.  _Commun. ACM_ 35, 12 (1992), 61–70.
  * Gunawardana and Shani (2015) Asela Gunawardana and Guy Shani. 2015.  Evaluating Recommender Systems.  In _Recommender Systems Handbook_ , Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). Springer, 265–308.
  * Gunes et al. (2014) Ihsan Gunes, Cihan Kaleli, Alper Bilge, and Huseyin Polat. 2014.  Shilling attacks against recommender systems: a comprehensive survey.  _Artificial Intelligence Review_ 42, 4 (2014), 767–799.
  * Hazrati and Ricci (2022) Naieme Hazrati and Francesco Ricci. 2022.  Recommender systems effect on the evolution of users’ choices distribution.  _Inf. Process. Manag._ 59, 1 (2022), 102766.
  * He et al. (2005) Zengyou He, Xiaofei Xu, Zhexue Joshua Huang, and Shengchun Deng. 2005.  FP-outlier: frequent pattern based outlier detection.  _Computer Science and Information Systems_ 2, 1 (2005), 103–118.
  * Huang et al. (2018) Keman Huang, Michael Siegel, and Stuart Madnick. 2018\.  Systematically understanding the cyber attack business: a survey.  _Comput. Surveys_ 51, 4 (2018), 1–36.
  * Imran et al. (2023) Mubashir Imran, Hongzhi Yin, Tong Chen, Quoc Viet Hung Nguyen, Alexander Zhou, and Kai Zheng. 2023\.  ReFRS: Resource-efficient federated recommender system for dynamic and diversified user preferences.  _ACM Transactions on Information Systems_ 41, 3 (2023), 1–30.
  * Jannach et al. (2021) Dietmar Jannach, Pearl Pu, Francesco Ricci, and Markus Zanker. 2021.  Recommender systems: past, present, future.  _AI Mag._ 42, 3 (2021), 3–6.
  * Jannach et al. (2016) Dietmar Jannach, Paul Resnick, Alexander Tuzhilin, and Markus Zanker. 2016.  Recommender systems—beyond matrix completion.  _Commun. ACM_ 59, 11 (2016), 94–102.
  * Jha et al. (2021) Govind Kumar Jha, Manish Gaur, Preetish Ranjan, and Hardeo Kumar Thakur. 2021.  A survey on trustworthy model of recommender system.  _International Journal of System Assurance Engineering and Management_ (2021), 1–18.
  * Jin et al. (2023) Di Jin, Luzhi Wang, He Zhang, Yizhen Zheng, Weiping Ding, Feng Xia, and Shirui Pan. 2023.  A Survey on Fairness-aware Recommender Systems.  _arXiv preprint arXiv:2306.00403_ (2023).
  * Konstan and Terveen (2021) Joseph A. Konstan and Loren G. Terveen. 2021.  Human-Centered Recommender Systems: Origins, Advances, Challenges, and Opportunities.  _AI Mag._ 42, 3 (2021), 31–42.
  * Lam et al. (2006) Shyong K Lam, Dan Frankowski, and et al. 2006.  Do you trust your recommendations? An exploration of security and privacy issues in recommender systems. In _ETRICS_. 14–29.
  * Leonhardt and et al. (2018) Jurek Leonhardt and et al. 2018.  User fairness in recommender systems. In _WWW Companion_. 101–102.
  * Li et al. (2023) Lei Li, Yongfeng Zhang, and Li Chen. 2023.  Personalized prompt learning for explainable recommendation.  _ACM Transactions on Information Systems_ 41, 4 (2023), 1–26.
  * Liu et al. (2022) Zhiwei Liu, Liangwei Yang, Ziwei Fan, Hao Peng, and Philip S Yu. 2022.  Federated social recommendation with graph neural network.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ 13, 4 (2022), 1–24.
  * Lu et al. (2019) Wenpeng Lu, Fanqing Meng, Shoujin Wang, Guoqiang Zhang, Xu Zhang, Antai Ouyang, and Xiaodong Zhang. 2019.  Graph-Based Chinese Word Sense Disambiguation with Multi-Knowledge Integration.  _Computers, Materials & Continua_ 61, 1 (2019).
  * Lu et al. (2013) Xugang Lu, Yu Tsao, Shigeki Matsuda, and Chiori Hori. 2013\.  Speech enhancement based on deep denoising autoencoder. In _Interspeech_ , Vol. 2013. 436–440.
  * Lyu et al. (2021) Yanzhang Lyu, Hongzhi Yin, Jun Liu, Mengyue Liu, Huan Liu, and Shizhuo Deng. 2021\.  Reliable recommendation with review-level explanations. In _ICDE_. 1548–1558.
  * Ma et al. (2019) Jianxin Ma, Chang Zhou, Peng Cui, Hongxia Yang, and Wenwu Zhu. 2019.  Learning disentangled representations for recommendation. In _NIPS_.
  * Massa and Avesani (2007) Paolo Massa and Paolo Avesani. 2007.  Trust-aware recommender systems. In _RecSys_. 17–24.
  * Mehrabi et al. (2021) Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, and et al. 2021.  A survey on bias and fairness in machine learning.  _Comput. Surveys_ 54, 6 (2021), 1–35.
  * Mobasher et al. (2007) Bamshad Mobasher, Robin Burke, Runa Bhaumik, and et al. 2007\.  Toward trustworthy recommender systems: an analysis of attack models and algorithm robustness.  _ACM Transactions on Internet Technology_ 7, 4 (2007), 23–es.
  * Ning et al. (2015) Xia Ning, Christian Desrosiers, and et al. 2015.  A comprehensive survey of neighborhood-based recommendation methods.  In _Recommender Systems Handbook_ , Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). 37–76.
  * of China (2022) Cyberspace Administration of China. 2022\.  Regulation rules on the recommendation algorithm for internet information service.  http://www.cac.gov.cn/2022-01/04/c_1642894606364259.htm.
  * Pang et al. (2021) Guansong Pang, Chunhua Shen, Longbing Cao, and et al. 2021\.  Deep learning for anomaly detection: a review.  _Comput. Surveys_ 54, 2 (2021), 1–38.
  * Ricci et al. (2022) Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). 2022\.  _Recommender Systems Handbook_.  Springer.
  * Rong (2014) Xin Rong. 2014.  Word2vec parameter learning explained.  _arXiv preprint arXiv:1411.2738_ (2014).
  * Schnabel et al. (2016) Tobias Schnabel, Adith Swaminathan, and et al. 2016\.  Recommendations as treatments: debiasing learning and evaluation. In _ICML_. 1670–1679.
  * Shi and et al. (2016) Weisong Shi and et al. 2016\.  Edge computing: vision and challenges.  _IEEE Internet of Things Journal_ 3, 5 (2016), 637–646.
  * Shin et al. (2018) Hyejin Shin, Sungwook Kim, and et al. 2018.  Privacy enhanced matrix factorization for recommendation with local differential privacy.  _IEEE Trans Knowl Data Eng_ 30, 9 (2018), 1770–1782.
  * Tian et al. (2020) Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. 2020\.  What makes for good views for contrastive learning?. In _NIPS_. 6827–6839.
  * Van Maaten and et al. (2008) Laurens Van Maaten and et al. 2008.  Visualizing data using t-SNE.  _Journal of Machine Learning Research_ 9, 11 (2008).
  * Wang et al. (2021) Shoujin Wang, Longbing Cao, Yan Wang, Quan Z Sheng, Mehmet A Orgun, and Defu Lian. 2021\.  A survey on session-based recommender systems.  _Comput. Surveys_ 54, 7 (2021), 1–38.
  * Wang et al. (2023b) Shoujin Wang, Yan Wang, Fikret Sivrikaya, Sahin Albayrak, and Vito Walter Anelli. 2023b.  Data science for next-generation recommender systems.  _International Journal of Data Science and Analytics_ (2023).  https://doi.org/10.1007/s41060-023-00404-w
  * Wang et al. (2022) Shoujin Wang, Xiaofei Xu, Xiuzhen Zhang, Yan Wang, and et al. 2022.  Veracity-aware and event-driven personalized news recommendation for fake news mitigation. In _WWW_. 3673–3684.
  * Wang et al. (2015) Yan Wang, Lei Li, and Guanfeng Liu. 2015.  Social context-aware trust inference for trust enhancement in social network based recommendations on service providers.  _World Wide Web_ 18, 1 (2015), 159–184.
  * Wang and Lin (2006) Yan Wang and Fu-ren Lin. 2006.  Trust and risk evaluation of transactions with different amounts in peer-to-peer e-commerce environments. In _ICEBE_. 102–109.
  * Wang et al. (2023a) Yifan Wang, Weizhi Ma, Min Zhang, Yiqun Liu, and Shaoping Ma. 2023a.  A survey on the fairness of recommender systems.  _ACM Transactions on Information Systems_ 41, 3 (2023), 1–43.
  * Wang et al. (2008) Yan Wang, Duncan S Wong, and et al. 2008.  Evaluating transaction trust and risk levels in peer-to-peer e-commerce environments.  _Inf Syst E-Bus Manag_ 6, 1 (2008), 25–48.
  * Wang and et al. (2021) Zhenlei Wang and et al. 2021.  Counterfactual data-augmented sequential recommendation. In _SIGIR_. 347–356.
  * Wu et al. (2022) Le Wu, Xiangnan He, and et al. 2022.  A survey on accuracy-oriented neural recommendation: from collaborative filtering to information-rich recommendation.  _IEEE Trans Knowl Data Eng_ (2022).
  * Wu and et al. (2012) Zhiang Wu and et al. 2012\.  HySAD: A semi-supervised hybrid shilling attack detector for trustworthy product recommendation. In _KDD_. 985–993.
  * Xiao et al. (2017) Lin Xiao, Zhang Min, Zhang Yongfeng, Gu Zhaoquan, Liu Yiqun, and et al. 2017\.  Fairness-aware group recommendation with pareto-efficiency. In _RecSys_. 107–115.
  * Xiong et al. (2020) Yuwen Xiong, Mengye Ren, and et al. 2020.  Loco: local contrastive representation learning. In _NIPS_. 11142–11153.
  * Yang et al. (2019) Qiang Yang, Yang Liu, Yong Cheng, Yan Kang, Tianjian Chen, and Han Yu. 2019\.  Federated learning.  _Synthesis Lectures on Artificial Intelligence and Machine Learning_ 13, 3 (2019), 1–207.
  * Zhang et al. (2012) Haibin Zhang, Yan Wang, and Xiuzhen Zhang. 2012.  A trust vector approach to transaction context-aware trust evaluation in e-commerce and e-service environments. In _ICSOc_. 1–8.
  * Zhang et al. (2015) Haibin Zhang, Yan Wang, Xiuzhen Zhang, and et al. 2015\.  Reputationpro: the efficient approaches to contextual transaction trust computation in e-commerce environments.  _ACM Transactions on the Web_ 9, 1 (2015), 1–49.
  * Zhang et al. (2022) He Zhang, Bang Wu, Xingliang Yuan, Shirui Pan, and et al. 2022.  Trustworthy graph neural networks: aspects, methods and trends.  _arXiv preprint arXiv:2205.07424_ (2022).
  * Zhang et al. (2019) Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019\.  Deep learning based recommender system: a survey and new perspectives.  _Comput. Surveys_ 52, 1 (2019), 1–38.
  * Zhang et al. (2013) Xiuzhen Zhang, Lishan Cui, and Yan Wang. 2013.  Commtrust: computing multi-dimensional trust by mining e-commerce feedback comments.  _IEEE Trans Knowl Data Eng_ 26, 7 (2013), 1631–1643.
  * Zhang et al. (2020) Yongfeng Zhang, Xu Chen, et al. 2020\.  Explainable recommendation: a survey and new perspectives.  _Foundations and Trends® in Information Retrieval_ 14, 1 (2020), 1–101.
  * Zheng et al. (2014) Xiaoming Zheng, Yan Wang, Mehmet Orgun, and et al. 2014\.  Trust prediction with propagation and similarity regularization. In _AAAI_. 237–243.