# ACM Multimedia Grand Challenge on Detecting Cheapfakes

Shivangi Aneja  Technical University of Munich ,  Cise Midoglu  SimulaMet ,
Duc-Tien Dang-Nguyen  University of BergenKristiania University College ,
Sohail Ahmed Khan  University of Bergen ,  Michael Riegler  SimulaMet ,  Pål
Halvorsen  SimulaMet ,  Chris Bregler  Google AI and  Balu Adsumilli  YouTube

###### Abstract.

Cheapfake is a recently coined term that encompasses non-AI (“cheap”)
manipulations of multimedia content. Cheapfakes are known to be more prevalent
than deepfakes. Cheapfake media can be created using editing software for
image/video manipulations, or even without using any software, by simply
altering the context of an image/video by sharing the media alongside
misleading claims. This alteration of context is referred to as out-of-context
(OOC) misuse of media. OOC media is much harder to detect than fake media,
since the images and videos are not tampered. In this challenge, we focus on
detecting OOC images, and more specifically the misuse of real photographs
with conflicting image captions in news items. The aim of this challenge is to
develop and benchmark models that can be used to detect whether given samples
(news image and associated captions) are OOC, based on the recently compiled
COSMOS dataset.

cheapfakes, misinformation, news, out-of-context misuse,
re-contextualized media

††copyright: none Figure 1. Deepfakes (left) are defined as falsified media
created using sophisticated AI-based media manipulation tools and techniques.
Cheapfakes (right) include falsified media created with/without contemporary
non-AI based editing tools which are easily accessible. Photoshopping tools
can be used to tamper with images. Videos can be sped up or slowed down to
change the intent or misrepresent the person in the video. Re-contextualizing
includes associating falsified or unrelated claims with a genuine image to
misrepresent events or persons. This challenge is focused on detecting re-
contextualized cheapfakes. Image sources: (Rössler et al., 2019; Boredpanda,
2019; Waterson, 2019; House, 2014; Evon, 2020)

##  1\. Introduction

The last decade has seen a surge in the use of social media platforms as a
means of consuming news. In a recent study (Vorhaus, 2020), Forbes reports
that social media giant Facebook leads this trend with $36\%$ of its customers
using the platform for consuming news. Social media platforms come with a
freedom for users to upload and share posts, which has led to the
proliferation of fake media on these platforms.

Figure 2. Each image in the dataset is accompanied by one or two captions that
the image was circulated together with on the Internet. On the left, one of
the two captions is misleading with an alteration of context, indicating out-
of-context (OOC) misuse. On the right, none of the two captions are
misleading, hence not-out-of-context (NOOC). Image source: (Aneja et al.,
2021a)

Fake media (including audio, images, videos, and text) circulated on social
media platforms can be broadly grouped into two major categories: deepfakes
and cheapfakes, as shown in Figure 1. Deepfakes are falsified media, most
commonly facial videos created using sophisticated AI-based media manipulation
tools and techniques. Several deepfake detection methods (Rössler et al.,
2019; Nguyen et al., 2019; Li and Lyu, 2018; Yang et al., 2019; Cozzolino et
al., 2018; Afchar et al., 2018; Agarwal et al., 2019; Li et al., 2020;
Verdoliva, 2020; Aneja and Nießner, 2020; Cozzolino et al., 2020) are in place
to monitor and regulate the spread of deepfake videos. Cheapfake is a recently
coined general term that encompasses non-AI (“cheap”) manipulations of
multimedia content, created without using deep learning methods. Although a
lot of attention has been paid to the creation, detection, and misuse of
deepfakes in the last years, cheapfakes are actually known to be more
prevalent than deepfakes (Brennen et al., 2020; Schick, 2020; Luther, 2022).

Cheapfakes are created with or without contemporary editing tools which are
non-AI based and are easily accessible. Image manipulations, speeding/slowing
of videos, and deliberate alteration of the context of the multimedia asset
in, e.g., news captions, by sharing the media alongside misleading claims, are
some of the methods that are currently in use (see Figure 1). The latter is
referred to as out-of-context (OOC) misuse of media. OOC media are much harder
to detect than fake media, since the images and/or videos are not tampered. We
refer readers to the report by Paris et al. (Paris and Donovan, 2019) for an
overview of different types of cheapfakes surfacing the Internet.

Depending on the type of cheapfakes, different detection tools can be used.
Methods to detect image manipulations such as photoshopping and image splicing
have been investigated (Chen et al., 2017; Cozzolino et al., 2015; Huh et al.,
2018; Wang et al., 2019). Re-contextualization or OOC misuse, which include
associating falsified or unrelated claims with a genuine image in order to
misrepresent events or persons is, however, relatively niche and unexplored.
Very recently, Aneja et al. (Aneja et al., 2021a) introduced this task,
provided a dataset of real-world news posts called COSMOS, and proposed a
method for detecting cheapfakes, which was benchmarked using the COSMOS
dataset.

In this challenge, we focus on detecting OOC images, and more specifically the
misuse of real photographs with conflicting image captions, in news items. The
aim of this challenge is to develop and benchmark models that can be used to
detect whether given samples (news image and associated captions) are OOC,
based on a version of the COSMOS dataset.

##  2\. Challenge Tasks

### Task 1: Identification of Conflicting Image-Caption Triplets

An image serves as evidence of the event described by a news caption. If two
captions associated with an image are valid, then they should describe the
same event. If they align with the same object(s) in the image, then they
should be broadly conveying the same information. Based on these patterns, we
define out-of-context (OOC) use of an image as presenting the image as an
evidence of untrue and/or unrelated event(s). If the two captions refer to
same object(s) in the image, but are semantically different, i.e., associate
the same subject to different events, this indicates OOC use of the image.
However, if the captions correspond to the same event, irrespective of the
object(s) the captions describe, this is defined as not-out-of-context (NOOC)
use of the image. See Figure 2 for more details.

In this task, the participants are asked to come up with methods to detect
conflicting image-caption triplets, which indicates miscontextualization. More
specifically, given <Image,Caption1,Caption2> triplets as input, the proposed
model should predict corresponding class labels (OOC or NOOC). The end goal
for this task is not to identify which of the two captions is true/false, but
rather to detect the existence of miscontextualization. This kind of a setup
is considered particularly useful for assisting fact checkers, as highlighting
conflicting image-caption triplets allows them to narrow down their search
space.

### Task 2: Fake Caption Detection

A NOOC scenario from Task 1 makes no conclusions regarding the veracity of the
statements. In a practical scenario, multiple captions might not be available
for a given image. In such a scenario, the task boils down to figuring out
whether a given caption linked to the image is genuine or not. We argue that
this is a challenging task, even for human moderators, without prior knowledge
about the image origin. Luo et al. (Luo et al., 2021) verified this claim with
a study on human evaluators who were instructed not to use search engines,
where the average human accuracy was around $65\%$.

In this task, the participants are asked to come up with methods to determine
whether a given <Image,Caption> pair is genuine (real) or falsely generated
(fake). Since our dataset only contains real, non-photoshopped images, it is
suitable for a practical use case and challenging at the same time.

##  3\. Dataset

Aneja et al. (Aneja et al., 2021a) have created COSMOS, a large-scale dataset
of around $200K$ images which have been matched with $450K$ textual captions
from different news websites, blogs, and social media posts. Figure 3 presents
the category distribution of the images in the dataset, which were collected
from a wide variety of articles with a special focus on topics where
misinformation spread is prominent.

Figure 3. Distribution of the images in the COSMOS dataset per category.

For this challenge, a part of the COSMOS dataset (Aneja et al., 2021a) is
sampled and assigned as the public dataset. The public dataset, consisting of
the training, validation and public test splits, is provided openly to
participants for training and testing their algorithms. The remaining part of
the COSMOS dataset is augmented with new samples and modified to create the
hidden test splits, which are not made publicly available, and will be used by
the challenge organizers to evaluate the submissions. Details of the challenge
dataset are summarized in Table 1.

Table 1. Challenge dataset statistics. Task | Split | # Images | # Captions |  Context Annotation
---|---|---|---|---
| Training | $161752$ | $360749$ |  No
| Validation | $41006$ | $90036$ |  No
Task 1 | Public Test | $1000$ | $2000$ |  Yes
Task 2 | Public Test | $100$ | $100$ |  Yes

###  3.1. Training and Validation Splits

We provide common training and validation splits for both challenge tasks. The
training and validation splits are provided as JavaScript Object Notation
(JSON) formatted text files called `train.json` and `val.json`, where each
data sample is stored as a dictionary 111https://github.com/shivangi-
aneja/COSMOS. The attributes in `train.json` and `val.json` are as follows.

  * •

`img_local_path:` Source path for the image in the dataset directory.

  * •

`articles:` List of dictionaries containing metadata for every caption
associated with the image.

  * •

`caption:` Original caption scraped from the news website.

  * •

`article_url:` Link to the website from where the image and caption were
scraped.

  * •

`caption_modified:` Modified caption after applying Spacy
NER222https://spacy.io/api/entityrecognizer. Authors in (Aneja et al., 2021a)
use this caption as an input to their model during experiments.

  * •

`entity_list:` List of mappings between the modified named entities in caption
with the corresponding hypernyms.

  * •

`maskrcnn_bboxes:` List of detected bounding boxes corresponding to the image.
(x1,y1) refers to the start vertex of the rectangle and (x2, y2) refers to end
vertex of the rectangle. Note that for detecting bounding boxes, the authors
in (Aneja et al., 2021a) use the Detectron2 pretrained
model333https://github.com/facebookresearch/detectron2/blob/master/MODEL_ZOO.md
available
under444https://github.com/facebookresearch/detectron2/blob/master/configs/COCO-
Keypoints/keypoint_rcnn_X_101_32x8d_FPN_3x.yaml. They detect up to $10$
bounding boxes per image.

###  3.2. Test Splits

We provide separate test splits for evaluating the performance of the two
tasks.

Task 1 test split: The public test split is provided as a JSON formatted text
file called `test.json`, and has the structure described in1. The hidden test
split is structurally identical to the public test split. The attributes in
`test.json` are as follows.

  * •

`img_local_path:` Source path for the image in the dataset directory.

  * •

`caption1:` First caption associated with the image.

  * •

`caption1_modified:` Modified `caption1` after applying Spacy NER.

  * •

`caption1_entities:` List of mappings between the modified named entities in
`caption1` with the corresponding hypernyms.

  * •

`caption2:` Second caption associated with the image.

  * •

`caption2_modified:` Modified `caption2` after applying Spacy NER.

  * •

`caption2_entities:` List of mappings between the modified named entities in
`caption2` with the corresponding hypernyms.

  * •

`article_url:` Link to the website from where the image and caption were
scraped.

  * •

`label:` Class label indicating whether the two captions are out-of-context
with respect to the image, where $1$=out-of-context (OOC) and $0$=not-out-of-
context (NOOC).

  * •

`maskrcnn_bboxes:` List of detected bounding boxes corresponding to the image.
(x1,y1) refers to the start vertex of the rectangle and (x2, y2) refers to the
end vertex of the rectangle.

Task 2 test split: The public test split is provided as a JSON formatted text
file called `task_2.json`. The hidden test split is structurally identical to
the public test split. The attributes in `task_2.json` are as follows.

  * •

`img_local_path:` Source path for the image in the dataset directory.

  * •

`caption:` Text caption associated with the image.

  * •

`genuine:` Classification label, where 0=out-of-context (OOC) and 1=not-out-
of-context (NOOC).

##  4\. Evaluation Criteria

There are two considerations in the fulfillment of the above described tasks.
Participant models will be evaluated and ranked according to two aggregate
scores representing these considerations, each composed of $5$ and $3$ metrics
respectively.

Effectiveness: The first goal is to achieve high detection performance. This
speaks to effectiveness, which will be evaluated based on accuracy, precision,
recall, F1-score, and Matthews correlation coefficient (MCC). Participants are
asked to calculate these $5$ metrics for their model and include the values in
their submission.

Efficiency: In certain scenarios, having an idea about the potential misuse of
images in real-time and with minimal resources can be more important than the
detection performance itself. We take this aspect into consideration by
introducing an additional goal: having low latency and low complexity. This
speaks to efficiency, which will be evaluated based on latency555Average,
i.e., arithmetic mean of the runtime per sample, calculated over all samples
in the public test split (ms)., number of parameters666Number of trainable
parameters in the model (million)., and model size777Storage size for the
model (MB).. Participants are asked to calculate these $3$ metrics for their
model and include the values in their submission.

##  5\. Administrative Details

A challenge website888https://detecting-cheapfakes.github.io/ has been setup
with a commitment to be maintained at least for the next $3$ years. Along with
the official website, a Google
Group999https://groups.google.com/g/grandchallenge-cheapfakes and a GitHub
repository101010https://github.com/detecting-cheapfakes/detecting-cheapfakes-
code have been established to support prospective participants. Interested
participants can find the previously asked questions and join interactive
discussions on these platforms.

##  6\. Conclusion and Outlook

The ACM Multimedia Grand Challenge on Detecting Cheapfakes addresses the
relatively new but prominent problem of detecting cheapfake multimedia content
in news. More specifically, it focuses on the OOC misuse of images in news
items. As emphasized in Section 1, this is a relatively novel area of
research, in comparison to deepfakes, and is distinguishable from the wider
field of fake news in the following way: the term “fake news” traditionally
refers to the use of either fake multimedia content or false captions, whereas
OOC misuse refers to the scenario where the multimedia content in the news
item (images in this case) is decidedly not fake, and the captions may or may
not be false. Rather, the misinformation results from the OOC combination of
the two.

With this challenge, we firstly aim to motivate researchers to develop
different methods for addressing this particular problem, and to benchmark a
number of proposed models for detecting the OOC misuse of real photographs in
news items. Secondly, through the dissection and analysis of the COSMOS
dataset, we aim to encourage the in-depth understanding, and later generation
of (further), supervised datasets for the above described challenges.
Algorithmic benchmarking is an efficient approach to analyze the results from
different detection methods, but on top of evaluating the models themselves,
the comparison of different approaches can help identify the potentials as
well as the shortcomings of existing open datasets.

There is a growing interest from the scientific community towards addressing
the problem of misinformation in general, and towards the detection of
deepfakes in particular. We hope that this challenge can increase awareness
regarding the prominence of cheapfakes as well, and that in the future, the
methods presented within this context can evolve into systems that support
researchers, regulatory bodies, news consumers, and the general public in
their search for a safe and truthful information ecosystem.

The first edition of this challenge has been organized within the ACM
Multimedia Systems (MMSys) Conference as the “MMSys’21 Grand Challenge on
Detecting Cheapfakes” (Midoglu and Aneja, 2021; Aneja et al., 2021b). The
challenge will be sustained over the next years, as we believe that it is
specific enough to be relevant and timely, as well as generic enough to evolve
over time according to different needs. For instance, with the rising
importance of synthetic data for research and training purposes, a possible
future task could be the generation of fake captions.

## References

  * (1)
  * Afchar et al. (2018) D. Afchar, V. Nozick, J. Yamagishi, and I. Echizen. 2018\.  MesoNet: a Compact Facial Video Forgery Detection Network.  https://doi.org/10.1109/wifs.2018.8630761
  * Agarwal et al. (2019) S. Agarwal, H. Farid, Y. Gu, M. He, K. Nagano, and H. Li. 2019.  Protecting World Leaders Against Deep Fakes. In _CVPR Workshops_.
  * Aneja et al. (2021a) S. Aneja, C. Bregler, and M. Nießner. 2021a.  COSMOS: Catching Out-of-Context Misinformation with Self-Supervised Learning.  arXiv:2101.06278 [cs.CV]
  * Aneja et al. (2021b) S. Aneja, C. Midoglu, D. Dang-Nguyen, M. A. Riegler, P. Halvorsen, Ma. Niessner, B. Adsumilli, and C. Bregler. 2021b.  MMSys’21 Grand Challenge on Detecting Cheapfakes.  arXiv:2107.05297 [cs.MM]
  * Aneja and Nießner (2020) S. Aneja and M. Nießner. 2020.  Generalized Zero and Few-Shot Transfer for Facial Forgery Detection.  arXiv:2006.11863 [cs.CV]
  * Boredpanda (2019) Boredpanda. 2019\.  _30 Fake Viral Photos People Believed Were Real_.  https://www.boredpanda.com/fake-news-photos-viral-photoshop/
  * Brennen et al. (2020) J. S. Brennen, F. M. Simon, P. N. Howard, and R. K. Nielsen. 2020.  _Types, Sources, and Claims of COVID-19 Misinformation_.  http://www.primaonline.it/wp-content/uploads/2020/04/COVID-19_reuters.pdf
  * Chen et al. (2017) C. Chen, S. McCloskey, and J. Yu. 2017.  Image Splicing Detection via Camera Response Function Analysis.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ (2017), 1876–1885.
  * Cozzolino et al. (2015) D. Cozzolino, G. Poggi, and L. Verdoliva. 2015.  Splicebuster: A new blind image splicing detector.  _2015 IEEE International Workshop on Information Forensics and Security (WIFS)_ (2015), 1–6.
  * Cozzolino et al. (2020) D. Cozzolino, A. Rössler, J. Thies, M. Nießner, and L. Verdoliva. 2020.  ID-Reveal: Identity-aware DeepFake Video Detection.  arXiv:2012.02512 [cs.CV]
  * Cozzolino et al. (2018) D. Cozzolino, J. Thies, A. Rössler, C. Riess, M. Nießner, and L. Verdoliva. 2018\.  Forensictransfer: Weakly-supervised domain adaptation for forgery detection.  _arXiv preprint arXiv:1812.02510_ (2018).
  * Evon (2020) D. Evon. 2020.  _Is This Obama, Fauci, and Gates at a Wuhan Lab in 2015?_ Retrieved July 13, 2020 from https://www.snopes.com/fact-check/obama-fauci-gates-wuhan-lab/
  * House (2014) The White House. 2014\.  _President Obama tours a lab at the Vaccine Research Center at the National Institutes of Health_.  https://obamawhitehouse.archives.gov/photos-and-video/photo/2014/12/president-obama-tours-lab-vaccine-research-center-national-institutes
  * Huh et al. (2018) M. Huh, A. Liu, A. Owens, and A. A. Efros. 2018\.  Fighting Fake News: Image Splice Detection via Learned Self-Consistency.  arXiv:1805.04096 [cs.CV]
  * Li et al. (2020) L. Li, J. Bao, T. Zhang, H. Yang, D. Chen, F. Wen, and B. Guo. 2020\.  Face x-ray for more general face forgery detection. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5001–5010.
  * Li and Lyu (2018) Y. Li and S. Lyu. 2018\.  Exposing DeepFake Videos By Detecting Face Warping Artifacts.  arXiv:1811.00656 [cs.CV]
  * Luo et al. (2021) G. Luo, T. Darrell, and A. Rohrbach. 2021.  NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media.  arXiv:2104.05893 [cs.CV]
  * Luther (2022) Joel Luther. 2022\.  _MediaReview: A next step in solving the misinformation crisis_.  https://reporterslab.org/mediareview-a-next-step-in-solving-the-misinformation-crisis/
  * Midoglu and Aneja (2021) C. Midoglu and S. Aneja. 2021.  _Grand Challenge on Detecting Cheapfakes_.  https://2021.acmmmsys.org/cheapfake_challenge.php
  * Nguyen et al. (2019) H. H. Nguyen, J. Yamagishi, and I. Echizen. 2019\.  Capsule-forensics: Using Capsule Networks to Detect Forged Images and Videos.  https://doi.org/10.1109/icassp.2019.8682602
  * Paris and Donovan (2019) B. Paris and J. Donovan. 2019.  _Deepfakes and cheapfakes: The manipulation of audio and visual evidence_.  https://datasociety.net/wp-content/uploads/2019/09/DataSociety_Deepfakes_Cheap_Fakes.pdf
  * Rössler et al. (2019) A. Rössler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner. 2019\.  FaceForensics++: Learning to Detect Manipulated Facial Images.  arXiv:1901.08971 [cs.CV]
  * Schick (2020) N. Schick. 2020\.  _Don’t underestimate the cheapfake_.  Retrieved Dec 22, 2020 from https://www.technologyreview.com/2020/12/22/1015442/cheapfakes-more-political-damage-2020-election-than-deepfakes/
  * Verdoliva (2020) L. Verdoliva. 2020\.  Media Forensics and DeepFakes: An Overview.  _IEEE Journal of Selected Topics in Signal Processing_ 14, 5 (2020), 910–932.  https://doi.org/10.1109/JSTSP.2020.3002101
  * Vorhaus (2020) M. Vorhaus. 2020\.  _People Increasingly Turn To Social Media For News_.  Retrieved June 24, 2020 from https://www.forbes.com/sites/mikevorhaus/2020/06/24/people-increasingly-turn-to-social-media-for-news/
  * Wang et al. (2019) S. Wang, O. Wang, A. Owens, R. Zhang, and A. A. Efros. 2019.  Detecting Photoshopped Faces by Scripting Photoshop. In _ICCV_.
  * Waterson (2019) J. Waterson. 2019\.  _Facebook refuses to delete fake Pelosi video spread by Trump supporters_.  https://www.theguardian.com/technology/2019/may/24/facebook-leaves-fake-nancy-pelosi-video-on-site
  * Yang et al. (2019) X. Yang, Y. Li, and S. Lyu. 2019.  Exposing Deep Fakes Using Inconsistent Head Poses.  _ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ (2019).  https://doi.org/10.1109/icassp.2019.8683164


# Dual-interest Factorization-heads Attention for Sequential Recommendation

Guanyu Lin1, Chen Gao1†, Yu Zheng1, Jianxin Chang3, Yanan Niu3, Yang Song3,
Zhiheng Li2, Depeng Jin1, Yong Li1 1Department of Electronic Engineering,
Beijing National Research Center for Information Science and Technology,
Tsinghua University2International Graduate School, Tsinghua University3Beijing
Kuaishou Technology Co., Ltd.

(2023)

###### Abstract.

Accurate user interest modeling is vital for recommendation scenarios. One of
the effective solutions is the sequential recommendation that relies on click
behaviors, but this is not elegant in the video feed recommendation where
users are passive in receiving the streaming contents and return skip or no-
skip behaviors. Here skip and no-skip behaviors can be treated as negative and
positive feedback, respectively. With the mixture of positive and negative
feedback, it is challenging to capture the transition pattern of behavioral
sequence. To do so, FeedRec has exploited a shared vanilla Transformer, which
may be inelegant because head interaction of multi-heads attention does not
consider different types of feedback. In this paper, we propose Dual-interest
Factorization-heads Attention for Sequential Recommendation (short for DFAR)
consisting of feedback-aware encoding layer, dual-interest disentangling layer
and prediction layer. In the feedback-aware encoding layer, we first suppose
each head of multi-heads attention can capture specific feedback relations.
Then we further propose factorization-heads attention which can mask specific
head interaction and inject feedback information so as to factorize the
relation between different types of feedback. Additionally, we propose a dual-
interest disentangling layer to decouple positive and negative interests
before performing disentanglement on their representations. Finally, we evolve
the positive and negative interests by corresponding towers whose outputs are
contrastive by BPR loss. Experiments on two real-world datasets show the
superiority of our proposed method against state-of-the-art baselines. Further
ablation study and visualization also sustain its effectiveness. We release
the source code here: https://github.com/tsinghua-fib-lab/WWW2023-DFAR.

Sequential recommendation, User feedback, Contrastive Learning

$\dagger$Chen Gao is the corresponding author (chgao96@gmail.com).

††journalyear: 2023††copyright: rightsretained††conference: Proceedings of the
ACM Web Conference 2023; May 1–5, 2023; Austin, TX, USA††booktitle:
Proceedings of the ACM Web Conference 2023 (WWW ’23), May 1–5, 2023, Austin,
TX, USA††doi: 10.1145/3543507.3583278††isbn: 978-1-4503-9416-1/23/04††ccs:
Information systems Recommender systems††ccs: Computing methodologies Neural
networks

##  1\. Introduction

Online sequential recommendation (Wang et al., 2019) has achieved great
success for its time-aware personalized modeling and has been widely applied
in Web platforms, including micro-video, news, e-commerce, etc. Especially in
today’s video feed recommendation, users are attracted immensely by video
streaming which can be treated as a sequence of items. Formally speaking, the
sequential recommendation is defined as predicting the next interacted item by
calculating the matching probability between historical items and the target
item. As shown in Figure 1 (a), existing sequential recommendation models
often exploit click behaviors of users to infer their dynamic interests (Zhou
et al., 2018; Wang et al., 2020; Zhou et al., 2019; Hidasi et al., 2016; Kang
and McAuley, 2018), the optimization of which samples un-clicked items as
negative feedback. However, such an approach only inputs positive items into
the sequential model, and negative items are sampled as target items, ignoring
the transition pattern between historical positive and negative items.

---
Figure 1. Illustration of click-based sequential recommendation and our dual-
interest sequential recommendation which is hybrid with positive and negative
feedback.

In the video feed recommendation where a single item is exposed each time,
users either skip or do not skip the recommended items, as illustrated in
Figure 1 (b). Skip can be treated as a kind of negative feedback which means
users don’t want to receive certain items, while no-skip can be treated as a
kind of positive feedback. That is to say, users are passive in receiving the
recommended items without providing active click behaviors (Li et al., 2016;
Okura et al., 2017; Guo et al., 2019) in such video feed recommendations.
However, the existing click-based sequential recommendation does not consider
the transition pattern between positive and negative items. Indeed, there are
two key challenges when modeling positive and negative feedback in one
sequence.

  * •

Complex transition between positive and negative feedback. The transition
pattern among interacted items has become far more complex due to negative
feedback. A user may provide negative feedback only because she has consumed a
very similar item before, which makes accurate modeling of transition
essential and challenging.

  * •

Mixed interest in one behavioral sequence. The negative feedback in the
behavioral sequence brings significant challenges to interest learning. The
traditional methods of sequential recommendation always conduct a pooling
operation on user sequence to obtain the users’ current interest, which will
fail when the sequence is hybrid with positive and negative signals.

To address the above challenges, in this work, we propose a model named Dual-
interest Factorization-heads Attention for Sequential Recommendation (short
for DFAR), further extracting the transition pattern and pair-wise relation
between positive and negative interests. To address the first challenge, in
the feedback-aware encoding layer, we assume each head of multi-head attention
(Vaswani et al., 2017) tends to capture specific relations of certain feedback
(Voita et al., 2019). As different heads of multi-head attention (Vaswani et
al., 2017) are independent, it may fail to capture the transition pattern
between different feedback when positive feedback and negative feedback are
indeed not independent of each other. Thus we exploit talking-heads attention
(Shazeer et al., 2020) to implicitly extract the transition pattern between
positive and negative historical items. However, talking-heads attention may
mix different heads too much without sufficient prior knowledge. To explicitly
extract the transition pattern between positive and negative historical items,
we further propose feedback-aware factorization-heads attention which can even
incorporate the feedback information into the head interaction. To address the
second challenge, we propose a dual-interest disentangling layer and
prediction layer, respectively, to disentangle and extract the pair-wise
relation between positive and negative interests. Specifically, we first mask
and encode the sequence hybrid with positive feedback and negative feedback
into two single interest representations before performing disentanglement on
them to repel the dissimilar interests. Then we perform a prediction of each
interest with the corresponding positive or negative tower and apply
contrastive loss on them to extract their pair-wise relation.

In general, we make the following contributions in this work.

  * •

We have taken the pioneering step of fully considering the modeling of
negative feedback, along with its impact on transition patterns, to enhance
sequential recommendation.

  * •

We propose a feedback-aware encoding layer to capture the transition pattern,
dual-interest disentangling layer and prediction layer to perform
disentanglement and capture the pair-wise relation between positive and
negative historical items.

  * •

We conduct experiments on one benchmark dataset and one collected industrial
dataset, where the results show the superiority of our proposed method. A
further ablation study also sustains the effectiveness of our three
components.

##  2\. Problem Formulation

Click-based Sequential Recommendation. Given item sequence
$\mathcal{I}_{u}=(i_{1},i_{2},\ldots,i_{t})$ with only positive feedback, the
goal of traditional click-based sequential recommendation is accurately
predicting the probability that given user $u$ will click the target item
i.e., $i_{t+1}$. The traditional click-based sequential recommendation can be
formulated as follows.

Input: Item sequence $\mathcal{I}_{u}=(i_{1},i_{2},\ldots,i_{t})$ with only
positive feedback for a given user $u$.

Output: The predicted score that the given user $u$ will click the target item
$i_{t+1}$.

Dual-interest Sequential Recommendation. Given item sequence
$\mathcal{I}_{u}=(i_{1},i_{2},\ldots,i_{t})$ with both positive and negative
feedback, the dual-interest sequential recommendation aims to better predict
the probability that given user $u$ will skip or not skip the target item
i.e., $i_{t+1}$. The dual-interest sequential recommendation with both
positive and negative feedback can be formulated as follows.

Input: Item sequence $\mathcal{I}_{u}=(i_{1},i_{2},\ldots,i_{t})$ with
positive and negative feedbacks for a given user $u$.

Output: The predicted score that the given user $u$ will skip or do not skip
the target item $i_{t+1}$.

##  3\. Methodology

Our model captures the relation between positive feedback and negative
feedback at the transition level and interest level of sequential
recommendation, respectively, by the proposed Feedback-aware Encoding Layer,
Dual-interest Disentangling Layer and Prediction Layer, as shown in Figure 2.

---
Figure 2. Illustration of DFAR. ($i$) Feedback-aware Encoding Layer is linked
after the Feedback-aware Embedding Layer where each historical item is
injected with a label embedding according to the corresponding feedback; It
consists of linear transformation and feedback-aware factorization-heads
attention. In the linear transformation, input embeddings are transformed into
query, key and value matrices. In feedback-aware factorization-heads
attention, the transition relation between different items is factorized into
different heads which are masked according to the positive or negative
feedback. ($ii$) Dual-interest Disentangling Layer decouples positive and
negative interests and performs disentanglement to repel the dissimilar
representations of different feedback; ($iii$) Dual-interest Prediction Layer
evolves positive and negative interests with corresponding towers and perform
BPR loss to capture the pair-wise relation.

  * •

Feedback-aware Encoding Layer. We build item embeddings by item IDs and label
embeddings by item feedback and further propose feedback-aware factorization-
heads attention to capture the transition pattern between different feedback.

  * •

Dual-interest Disentangling Layer. We mask the sequence hybrid with both
positive and negative feedback into two sequences with solely positive or
negative feedback. After encoding two split sequences with independent
factorization-heads attention to extract the positive and negative interests,
we then disentangle them to repel the dissimilar interests.

  * •

Dual-interest Prediction Layer. We further extract the positive and negative
interests with independent towers and then perform contrastive loss on them to
extract the pair-wise relation.

###  3.1. Feedback-aware Encoding Layer

In the feedback-aware encoding layer, we first inject each historical item
embedding with corresponding feedback embeddings to incorporate the feedback
information into each historical item embedding. Then we further propose
talking-heads attention and feedback-aware factorization-heads attention to
capture the transition pattern between positive and negative historical items.

####  3.1.1. Feedback-aware Embedding Layer

To fully distinguish positive and negative feedback, we build a label
embedding matrix $\textbf{L}\in\mathbb{R}^{2\times D}$, besides the item
embedding matrix $\textbf{E}\in\mathbb{R}^{m\times D}$. Here $m$ denotes the
number of items, and $D$ is the dimensionality for the hidden state. Then we
inject the feedback information into the item embedding and obtain the
feedback-aware input embeddings as the model input. Therefore, given item
sequence $\mathcal{I}_{u}=(i_{1},i_{2},\ldots,i_{t})$, we can obtain the
feedback-aware item embeddings $\mathbf{E}^{f}\in\mathbb{R}^{T\times D}$ as:

(1) $\displaystyle\mathbf{E}^{f}$
$\displaystyle=[\textbf{E}_{i_{1}},\textbf{E}_{i_{2}},\ldots,\textbf{E}_{i_{t}}]+[\textbf{L}_{y_{u,i_{1}}},\textbf{L}_{y_{u,i_{2}}},\ldots,\textbf{L}_{y_{u,i_{t}}}],$

where $\\{y_{u,i_{1}},y_{u,i_{2}},\cdots,y_{u,i_{t}}\\}$ are feedback of items
$\\{i_{1},i_{2},\cdots,i_{t}\\}$. Here $y_{u,i_{1}}=1$ if $i_{1}$ is the no-
skip item, and $y_{u,i_{1}}=0$ if $i_{1}$ is the skip item. Note that if the
sequence length is less than $t$, we can pad $\mathbf{E}^{f}$ with zero
embedding (Kang and McAuley, 2018).

####  3.1.2. Talking-Heads Attention

After obtaining the input embeddings for positive and negative historical
items, we then capture the transition pattern between them. The existing work,
FeedRec (Wu et al., 2022), exploits vanilla Transformer to roughly capture
this transition pattern, of which multi-head attention (Kang and McAuley,
2018) is the essential part, having the following equation:

(2)
$\begin{array}[]{l}\textbf{S}=\operatorname{MHA}(\mathbf{Q},\mathbf{K},\mathbf{V})=\left[\mathbf{A}_{1}^{MHA}\mathbf{V}_{1},\ldots,\mathbf{A}_{H}^{MHA}\mathbf{V}_{H}\right]\mathbf{W}_{0},\end{array}$
(3)
$\mathbf{A}^{MHA}_{h}=\operatorname{softmax}\left(\frac{\mathbf{Q}_{h}{\mathbf{K}_{h}}^{T}}{\sqrt{d}}\right),$
(4)
$\mathbf{Q}_{h}=\mathbf{Q}\mathbf{W}_{h}^{Q},\mathbf{K}_{h}=\mathbf{K}\mathbf{W}_{h}^{K},\mathbf{V}_{h}=\mathbf{V}\mathbf{W}_{h}^{V},$

where $h\in\\{1,2,\cdots,H\\}$ is the number of heads.
$\mathbf{W}_{0}\in\mathbb{R}^{HD\times D}$ and $\mathbf{W}_{h}^{Q}$,
$\mathbf{W}_{h}^{K}$, $\mathbf{W}_{h}^{V}\in\mathbb{R}^{D\times D}$ are
parameters to be learned. $\operatorname{MHA}$ means multi-heads attention
(Vaswani et al., 2017). However, different heads of multi-head attention are
independent of each other, sharing no information across heads. If assuming
different heads capture specific relations between different feedback, then
this means there is no information sharing across different feedback. Thus we
first propose talking-heads attention (Shazeer et al., 2020) to address this
issue as below.

(5)
$\begin{array}[]{l}\textbf{S}=\operatorname{THA}(\mathbf{Q},\mathbf{K},\mathbf{V})=\left[\mathbf{A}_{1}^{THA}\mathbf{V}_{1},\ldots,\mathbf{A}_{H}^{THA}\mathbf{V}_{H}\right]\mathbf{W}_{0},\end{array}$
(6) $\displaystyle\left[\begin{array}[]{c}\mathbf{A}_{1}\\\ \mathbf{A}_{2}\\\
\vdots\\\
\mathbf{A}_{H^{\prime}}\end{array}\right]=\mathbf{W}_{THA}\left[\begin{array}[]{c}\frac{\mathbf{Q}_{1}{\mathbf{K}_{1}}^{T}}{\sqrt{d}}\\\
\frac{\mathbf{Q}_{2}{\mathbf{K}_{2}}^{T}}{\sqrt{d}}\\\ \vdots\\\
\frac{\mathbf{Q}_{H}{\mathbf{K}_{H}}^{T}}{\sqrt{d}}\end{array}\right],$ (7)
$\displaystyle\left[\begin{array}[]{c}\mathbf{A}_{1}^{THA}\\\
\mathbf{A}_{2}^{THA}\\\ \vdots\\\
\mathbf{A}_{H}^{THA}\end{array}\right]=\mathbf{W}_{THA}^{S}\left[\begin{array}[]{c}\operatorname{softmax}\left(\mathbf{A}_{1}\right)\\\
\operatorname{softmax}\left(\mathbf{A}_{2}\right)\\\ \vdots\\\
\operatorname{softmax}\left(\mathbf{A}_{H^{\prime}}\right)\end{array}\right],$

where $\mathbf{W}_{THA}\in\mathbb{R}^{H^{\prime}\times H}$,
$\mathbf{W}^{S}_{THA}\in\mathbb{R}^{H\times H^{\prime}}$ and
$\mathbf{W}_{0}\in\mathbb{R}^{HD\times D}$ are parameters to be learned. Here
$\operatorname{THA}$ refers to talking-heads attention. However, the
interaction between different heads in talking-heads attention is implicit,
which may confuse the task for each head and result in overfitting. Not to
mention, the two additional linear transformations (i.e. Eq.(6) and Eq.(7)) of
talking-heads attention will increase the computation cost.

####  3.1.3. Feedback-aware Factorization-heads Attention

In this part, we factorize the interaction between positive and negative
feedback. Traditional multi-heads attention assigns similar items with higher
attention weights. However, in our problem with both positive and negative
feedback, two similar items may have different attention weights due to the
feedback they have. For example, an NBA fan skips the recommended video about
basketball when he/she has watched a lot of basketball videos. But he/she
engages in the video about basketball when he/she only has watched a few
videos about basketball. In the first case we should repel the representations
between historical basketball videos and target basketball videos, while in
the second case we should attract them. That is to say, it is necessary to
inject the user’s feedback into the transition pattern between different
feedback. Here we suppose different heads can represent different transition
patterns for different feedback (Voita et al., 2019). To explicitly factorize
interaction across different heads, we further propose factorization-heads
attention as:

(8)
$\begin{array}[]{l}\textbf{S}=\operatorname{FHA}(\mathbf{Q},\mathbf{K},\mathbf{V})=\left[\mathbf{A}^{FHA}_{1,1}\mathbf{V}_{1},\ldots,\mathbf{A}^{FHA}_{{H},{H}}\mathbf{V}_{{H}}\right]\mathbf{W}_{0},\end{array}$
(9)
$\mathbf{A}^{FHA}_{h_{1},h_{2}}=\operatorname{softmax}\left(\frac{\mathbf{Q}_{h_{1}}{\mathbf{K}_{h_{2}}}^{T}}{\sqrt{d}}\right),$

where ${h_{1},h_{2}}\in\\{1,2,\cdots,H\\}$.
$\mathbf{W}_{0}\in\mathbb{R}^{HD\times D}$ are parameters to be learned. Here
$\operatorname{FHA}$ is our proposed factorization-heads attention. The
factorization-heads attention can represent $H\times H$ relations by $H$
heads. That is to say, our factorization-heads attention can reduce $\sqrt{H}$
times parameters if we want to represent $H$ head interaction relations like
talking-heads attention or multi-heads attention. Besides, to further inject
the prior feedback knowledge into the factorization-heads attention, we
propose feedback-aware factorization-heads attention with a label mask
$\mathbf{M}_{h_{1},h_{2}}\in\\{0,1\\}^{t\times t}$ as:

(10)
$\begin{array}[]{l}\textbf{S}=\operatorname{FFHA}(\mathbf{Q},\mathbf{K},\mathbf{V})=\left[\mathbf{A}^{FFHA}_{1,1}\mathbf{V}_{1},\ldots,\mathbf{A}^{FFHA}_{{H},{H}}\mathbf{V}_{{H}}\right]\mathbf{W}_{0},\end{array}$
(11)
$\mathbf{A}^{FFHA}_{h_{1},h_{2}}=\operatorname{softmax}\left(\mathbf{M}_{h_{1},h_{2}}\frac{\mathbf{Q}_{h_{1}}{\mathbf{K}_{h_{2}}}^{T}}{\sqrt{d}}\right),$

where $\mathbf{M}_{{h_{1},h_{2}},i,j}=1$, if
$h_{1}\in\\{\frac{y_{u,i}{H}}{2}+1,\frac{y_{u,i}{H}}{2}+2,\cdots,\frac{(y_{u,i}+1){H}}{2}\\},h_{2}\in\\{\frac{y_{u,j}{H}}{2}+1,\frac{y_{u,j}{H}}{2}+2,\cdots,\frac{(y_{u,j}+1){H}}{2}\\},i\in\\{1,2,\cdots,t\\},j\in\\{1,2,\cdots,t\\}$
and $\mathbf{M}_{{h_{1},h_{2}},i,j}=0$, otherwise. Here the first half of
heads w.r.t. $\\{1,2,\cdots,\frac{H}{2}\\}$ represent negative heads and
second half of heads w.r.t. $\\{\frac{H}{2}+1,\frac{H}{2}+2,\cdots,H\\}$
represent positive heads. For example, as shown in Figure 3, if item $i$ is
positive and item $j$ is negative (i.e., $y_{u,i}=1$ and $y_{u,j}=0$), $h_{1}$
in positive half and $h_{2}$ in negative half will be preserved, i.e.,
$\mathbf{M}_{{2,1},i,j}=1$, and
$\mathbf{M}_{{1,1},i,j},\mathbf{M}_{{1,2},i,j},\mathbf{M}_{{2,2},i,j}=0$.

---
Figure 3. Illustration of label mask $\mathbf{M}_{h_{1},h_{2}}$ on head
interaction. Here we show the comprehensible case with two heads, where the
first half of heads, i.e. head 1, represents negative head and second half of
heads, i.e. head 2, represents positive head.

Besides, $\operatorname{FFHA}$ is our proposed feedback-aware factorization-
heads attention. Apart from the advantage of explicit interaction between
different heads, unlike talking-heads attention, our factorization-heads
attention also improves the multi-heads attention without high computation
cost. We feed the input embedding into the feedback-aware factorization
attention module as:

(12)
$\begin{array}[]{l}\mathbf{S}=\operatorname{FFHA}(\mathbf{E}^{f},\mathbf{E}^{f},\mathbf{E}^{f}),\end{array}$

where $\mathbf{S}$ are the obtained feedback-aware sequential representations.
We put the pseudocode of FHA at Appendix A.1 and compare its complexity with
MHA and THA at Appendix A.1.5.

###  3.2. Dual-interest Disentangling Layer

Though feedback-aware factorization-heads attention has factorized the
transition relation between positive feedback and negative feedback, their
interest-level relations require further extracting. In this part, we decouple
the positive and negative interests and then perform disentanglement on them
to repel the dissimilar interests.

####  3.2.1. Dual-interest Decoupling Attention

After capturing the transition pattern between positive feedback and negative
feedback, we then filter out each feedback by a corresponding feedback mask as
follows,

(13) $\displaystyle\mathbf{S}^{P}$
$\displaystyle=[\textbf{S}_{i_{1}},\textbf{S}_{i_{2}},\ldots,\textbf{S}_{i_{t}}]*[{y_{u,i_{1}}},{y_{u,i_{2}}},\ldots,{y_{u,i_{t}}}],$
$\displaystyle\mathbf{S}^{N}$
$\displaystyle=[\textbf{S}_{i_{1}},\textbf{S}_{i_{2}},\ldots,\textbf{S}_{i_{t}}]*(1-[{y_{u,i_{1}}},{y_{u,i_{2}}},\ldots,{y_{u,i_{t}}}]),$

which are then fed into the corresponding factorization-heads attention
modules to enhance the transition pattern learning for each feedback as:

(14)
$\begin{array}[]{l}\mathbf{S}^{P}=\operatorname{FHA}(\mathbf{S}^{P},\mathbf{S}^{P},\mathbf{S}^{P}),\mathbf{S}^{N}=\operatorname{FHA}(\mathbf{S}^{N},\mathbf{S}^{N},\mathbf{S}^{N}),\par\end{array}$

where $\mathbf{S}^{P}$ (or $\mathbf{S}^{N}$) are the single-feedback
sequential representations for positive feedback (or negative feedback). In
the subsequent section, we will exploit these filtered representations to
further extract the interest-level relations.

####  3.2.2. Dual-interest Aggregation and Disentanglement

The positive and negative interests of a given user should be distinguished
from each other. Hence we aim to repel the positive and negative
representations of corresponding interests. Specifically, we assume the target
item is possibly either positive or negative. Then we assign the target item
with positive and negative label embeddings, respectively, in positive and
negative assumed cases. To calculate the attention scores of positive and
negative historical items, we fuse them with the target item in assumed
positive and negative cases as below.

(15)
$\mathbf{A}^{P}=\text{MLP}\left((\textbf{E}_{i_{t+1}}+\textbf{L}_{1})\|\mathbf{S}^{P}\right),\mathbf{A}^{N}=\text{MLP}\left((\textbf{E}_{i_{t+1}}+\textbf{L}_{0})\|\mathbf{S}^{N}\right),$

where $\mathbf{A}^{P}$ and $\mathbf{A}^{N}\in\mathbb{R}^{t\times D}$ are the
positive and negative attention scores. MLP is the multi-layer perceptron.
Here $\textbf{L}_{1}$ and $\textbf{L}_{0}$ are the label embeddings for
positive and negative feedback, respectively. With the calculated attention
scores by (15), we can then obtain the single-feedback aggregated
representations for positive and negative items, respectively, as,

(16)
$\small\mathbf{F}^{P}=\textbf{softmax}\left(\mathbf{A}^{P}\right)\mathbf{S}^{P},\mathbf{F}^{N}=\textbf{softmax}\left(\mathbf{A}^{N}\right)\mathbf{S}^{N},$
(17)
$\displaystyle\mathbf{f}^{P}=\sum_{j=1}^{t}\mathbf{F}^{P}_{j},\mathbf{f}^{N}=\sum_{j=1}^{t}\mathbf{F}^{N}_{j},$

which are then further disentangled with cosine distance as:

(18)
$\mathcal{L}^{D}=\frac{\mathbf{f}^{P}\cdot\mathbf{f}^{N}}{\left\|\mathbf{f}^{P}\right\|\times\left\|\mathbf{f}^{N}\right\|}.$

where $\|\cdot\|$ is the L2-norm. By this disentangling loss, we can repel the
aggregated positive and negative representations so as to capture the
dissimilar characteristics between them.

###  3.3. Dual-interest Prediction Layer

In this section, we predict the next item of different interests by positive
and negative towers. Finally, we further perform contrastive loss on the
outputs of positive and negative towers so as to extract the pair-wise
relation between them.

####  3.3.1. Dual-interest Prediction Towers

To extract the positive and negative interests, we fuse the feedback-aware
sequential representations, single-feedback sequential representations, and
single-feedback aggregated representations into the corresponding positive or
negative prediction tower. Before feeding different representations into the
final prediction towers, we first aggregate part of them by the sum pooling
as:

$\displaystyle\mathbf{s}=\sum_{j=1}^{t}\mathbf{S}_{j},\mathbf{s}^{P}=\sum_{j=1}^{t}\mathbf{S}^{P}_{j},\mathbf{s}^{N}=\sum_{j=1}^{t}\mathbf{S}^{N}_{j},$

which are then finally fed into the positive and negative prediction towers
as:

(19)
$\textit{logit}^{P}_{u,t}=\textbf{MLP}\left(\mathbf{s}\|\mathbf{s}^{P}\|\mathbf{f}^{P}\|(\textbf{E}_{i_{t+1}}+\textbf{L}_{1})\right),$
(20)
$\textit{logit}^{N}_{u,t}=\textbf{MLP}\left(\mathbf{s}\|\mathbf{s}^{N}\|\mathbf{f}^{N}\|(\textbf{E}_{i_{t+1}}+\textbf{L}_{0})\right).$

where $\textit{logit}^{P}_{u,t}$ and $\textit{logit}^{N}_{u,t}$ are positive
and negative predicted logits for user $u$ on time step $t$, aiming to capture
the positive and negative interests, respectively. Here $\mathbf{f}^{P}$ and
$\mathbf{f}^{N}$ are pooled at Eq.(17).

####  3.3.2. Pair-wise Contrastive Loss

When the target item is positive, the prediction logit of the positive tower
will be greater than that of the negative tower, and vice versa. After
obtaining the positive and negative prediction logits, we then perform BPR
loss (Rendle et al., 2009) on them as:

(21) $\mathcal{L}^{BPR}=\left\\{\begin{aligned}
-\log(\sigma(\textit{logit}^{P}_{u,t}-\textit{logit}^{N}_{u,t}))&,&{y}_{u,t}=1,\\\
-\log(\sigma(\textit{logit}^{N}_{u,t}-\textit{logit}^{P}_{u,t}))&,&{y}_{u,t}=0.\end{aligned}\right.$

where $\sigma$ denotes the sigmoid function. With this BPR loss, we can
extract the pair-wise relations between positive and negative logits.

###  3.4. Joint Optimization

Though we have positive and negative towers, in the optimization step, we only
need to optimize the next item prediction loss with the positive tower as:

(22)
$\mathcal{L}=-\frac{1}{|\mathcal{R}|}\sum_{(u,i_{t})\in\mathcal{R}}\left(y_{u,t}\log\hat{y}^{P}_{u,t}+\left(1-y_{u,t}\right)\log\left(1-\hat{y}^{P}_{u,t}\right)\right),$

where $\hat{y}^{P}_{u,t}=\sigma(\textit{logit}^{P}_{u,t})$ and $\mathcal{R}$
is the training set. The negative prediction tower $\hat{y}^{N}_{u,t}$ indeed
will be self-supervised and optimized by the contrastive loss of Eq.(21).
After obtaining the main loss for the next item prediction, disentangling loss
for repelling representations and BPR loss for pair-wise learning, we can then
jointly optimize them as:

(23)
$\mathcal{L}^{J}=\mathcal{L}+\lambda^{BPR}\mathcal{L}^{BPR}+\lambda^{D}\mathcal{L}^{D}+\lambda\|\Theta\|,$

where $\lambda^{BPR}$ and $\lambda^{D}$ are hyper-parameters for weighting
each loss. Here $\lambda$ is the regularization parameter, and $\Theta$
denotes the model parameters to be learned.

##  4\. Experiments

In this section, we experiment on a public dataset and an industrial dataset,
aiming to answer the following research questions (RQ):

  * •

RQ1: Is the proposed DFAR effective when compared with the state-of-the-art
sequential recommenders?

  * •

RQ2 : What is the effect of our proposed feedback-aware encoding layer, dual-
interest disentangling layer and prediction layer?

  * •

RQ3 : How do the heads of proposed feedback-aware factorization-heads
attention capture the transition pattern between different feedback?

  * •

RQ4: How does the proposed method perform compared with the sequential
recommenders under different sequence lengths?

We also look into the question: ”how do the auxiliary loss for disentanglement
and pair-wise contrastive learning perform under different weights?” in
Appendix A.4.

###  4.1. Experimental Settings

Table 1. Micro-video and Amazon data statistics. Dataset | Micro-video | Amazon
---|---|---
#Users | 37,497 | 6,919
#Items | 129,092 | 28,695
#Records | Positive | 6,413,396 | 99,753
Negative | 5,448,693 | 20,581
Avg. records per user | 316.35 | 17.39

Table 2. Overall evaluations for DFAR against baselines under Micro-video and Amazon datasets on four metrics. Here Improv. is the improvement. Bold is the highest result and underline is the second highest result. Models | DIN | Caser | GRU4REC | DIEN | SASRec | THA4Rec | DFN | FeedRec | Ours | Improv.
---|---|---|---|---|---|---|---|---|---|---
Micro-video | AUC | 0.7345 | 0.8113 | 0.7983 | 0.7446 | 0.8053 | 0.8104 |  0.8342 | 0.8119 | 0.8578 | 2.83%
MRR | 0.5876 | 0.6138 | 0.5927 | 0.5861 | 0.6046 | 0.6080 |  0.6321 | 0.6095 | 0.6568 | 3.91%
NDCG | 0.6876 | 0.7079 | 0.6916 | 0.6861 | 0.7009 | 0.7035 |  0.7222 | 0.7047 | 0.7410 | 2.60%
GAUC | 0.7703 | 0.8211 | 0.8041 | 0.7753 | 0.8120 | 0.8138 |  0.8362 | 0.8180 | 0.8545 | 2.19%
Amazon | AUC | 0.6595 | 0.7192 |  0.7278 | 0.6688 | 0.6903 | 0.7069 | 0.6998 | 0.7037 | 0.7333 | 0.76%
MRR | 0.4344 | 0.4846 |  0.4901 | 0.4547 | 0.4604 | 0.4599 | 0.4743 | 0.4675 | 0.4980 | 1.61%
NDCG | 0.5669 | 0.6073 |  0.6114 | 0.5832 | 0.5883 | 0.5879 | 0.5990 | 0.5938 | 0.6175 | 1.00%
GAUC | 0.6618 | 0.7245 |  0.7266 | 0.6859 | 0.7029 | 0.7021 | 0.7120 | 0.7079 | 0.7305 | 0.54%

####  4.1.1. Datasets

The data statistics of our experiments are illustrated in Table 1 where Micro-
video is a collected industrial dataset and Amazon is the public benchmark
dataset which is widely used in existing work for sequential recommendation
(Lin et al., 2022). The detailed descriptions of them are as below.

Micro-video This is a popular micro-video application dataset, which is
recorded from September 11 to September 22, 2021. In this platform, users
passively receive the recommended videos, and their feedbacks are mostly
either skip or no-skip. Skip can be treated as a form of negative feedback,
and no-skip can be treated as a form of positive feedback. That is to say, we
have hybrid positive and negative feedback in this sequential data which is
very rare in modern applications.

Amazon111https://www.amazon.com This is Toys domain from a widely used public
e-commerce dataset in recommendation. The rating score in Amazon ranges from 1
to 5, and we treat the rating score over three and under two as positive and
negative feedback, respectively, following existing work DenoisingRec (Wang et
al., 2021) which is not for the sequential recommendation.

For the Micro-video dataset, interactions before and after 12 pm of the last
day are split as the validation and test sets, respectively, while
interactions before the last day are used as the training set. For the Amazon
dataset, we split the last day as the test set and the second last day as the
validation set, while other days are split as the training set.

####  4.1.2. Baselines and Evaluation Metrics

We compare our DFAR with the following state-of-the-art methods for sequential
recommender systems.

  * •

DIN (Zhou et al., 2018): It aggregates the historical items via attention
score with the target item.

  * •

Caser (Tang and Wang, 2018): It captures the transition between historical
items via convolution.

  * •

GRU4REC (Hidasi et al., 2016): It captures the transition between historical
items via GRU (Chung et al., 2014).

  * •

DIEN (Zhou et al., 2019): It captures the transition between historical items
via interest extraction and evolution GRUs (Chung et al., 2014).

  * •

SASRec (Kang and McAuley, 2018): It captures the transition between historical
items via multi-heads attention (Vaswani et al., 2017).

  * •

THA4Rec: It means talking-heads attention (Shazeer et al., 2020) for the
sequential recommendation, which is firstly applied in the recommendation by
us.

  * •

DFN (Xie et al., 2021): It purifies unclick (weak feedback) by click (strong
positive feedback) and dislike (strong positive feedback).

  * •

FeedRec (Wu et al., 2022): It further performs disentanglement on the weak
positive and negative feedback.

Besides, Widely-used AUC and GAUC (Gunawardana and Shani, 2015) are adopted as
accuracy metrics here while MRR@10 and NDCG@10 (Lin et al., 2022) are used as
ranking metrics for performance evaluation. The detailed illustration of them
is in Appendix A.2.

####  4.1.3. Hyper-parameter Settings

Hyper-parameters are generally set following the default settings of
baselines. We strictly follow existing work for sequential recommendation (Lin
et al., 2022) and leverage Adam (Kingma and Ba, 2015) with the learning rate
of 0.0001 to weigh the gradients. The embedding sizes of all models are set as
32. We use batch sizes 20 and 200, respectively, on the Micro-video and Amazon
datasets. We search the loss weights for pair-wise contrastive loss in
$[10^{-4},10^{-3},10^{-2},10^{-1}]$.

###  4.2. Overall Performance Comparison(RQ1)

We compare our proposed method with eight competitive baselines, and the
results are shown as Table 2, where we can observe that:

  * •

Our method achieves the best performance.  The results on two datasets show
that our DFAR model achieves the best performance compared with these seven
baselines on all metrics. Specifically, GAUC is improved by about 2.0% on the
Micro-video dataset and 0.5% on the Amazon dataset and when comparing DFAR
with other baselines. Please note that 0.5% improvement on GAUC could be
claimed as significant, widely acknowledged by existing works (Zhou et al.,
2018). Besides, the improvement is more significant in the Micro-video with
more negative feedback, which means incorporating the negative feedback into
the historical item sequence can boost the recommendation performance.

  * •

Existing work roughly captures the relation between positive feedback and
negative feedback. FeedRec and DFN even underperform some traditional
sequential recommendation models like GRU4REC and Caser in Amazon dataset.
Besides, though they outperform other baselines in Micro-video dataset, the
improvement is still slight. In other words, their designs fail to capture the
relation between positive feedback and negative feedback, which motivates us
to further improve them from transition and interest perspectives.

###  4.3. Ablation Study (RQ2)

Table 3. Effectiveness study of our proposed components. FHA means factorization-heads attention; MO means label mask operation on heads; IDL means interest disentangling loss on positive and negative representations; IBL means interest BPR loss on positive and negative logits. Dataset | Micro-video
---|---
Methods | w/o FHA | w/o MO | w/o IDL | w/o IBL | Ours
AUC | 0.8360 | 0.8473 | 0.8475 | 0.8364 | 0.8578
MRR | 0.6198 | 0.6378 | 0.6377 | 0.6324 | 0.6568
NDCG | 0.7127 | 0.7264 | 0.7264 | 0.7212 | 0.7410
GAUC | 0.8319 | 0.8428 | 0.8436 | 0.8283 | 0.8545
Dataset | Amazon
AUC | 0.7133 | 0.7141 | 0.7284 | 0.7137 | 0.7333
MRR | 0.4782 | 0.4883 | 0.4855 | 0.4839 | 0.4980
NDCG | 0.6016 | 0.6095 | 0.6073 | 0.6057 | 0.6175
GAUC | 0.7054 | 0.7137 | 0.7128 | 0.7047 | 0.7305

We further study the impact of four proposed components as Table 3, where FHA
represents the factorization-heads attention, the MO represents the mask
operation on factorized heads for factorization-heads attention, IDL means the
interest disentanglement loss on the positive and negative interest
representations, and IBL means the interest BPR loss on the positive and
negative prediction logits. From this table, we can have the following
observations.

  * •

Factorization of heads for transition attention weights is important. Removing
FHA and MO both show significant performance drops, which means these two
components are both necessary to each other. Specifically, removing FHA means
it is impossible to apply the mask on the implicit head interaction of either
multi-heads attention or talking-heads attention. At the same time, removing
MO on FHA will cause it to fail to exploit the prior knowledge of labels for
historical items and degenerate to even as poor as multi-heads attention or
talking-heads attention in the Amazon dataset.

  * •

Pair-wise interest is more important than disentangling interest. Removing IDL
and IBL will both drop the performance, while removing IBL is more
significant. This is because contrastive learning by BPR loss can indeed
inject more self-supervised signals, while disentanglement solely tends to
repel the dissimilar representations of positive feedback and negative
feedback.

###  4.4. Visualization for Attention Weights of Heads (RQ3)

|
---|---
(a) Micro-video | (b) Amazon
Figure 4. Visualization of accumulated attention weights between different
heads. Here $h_{1}$ and $h_{2}$ represent the heads for the source and target
behaviors, respectively (i.e., if the source behavior is negative and target
behavior is positive, we have $h_{1}=0$ and $h_{2}=1$). This illustrates our
method can factorize and extract the relation between different feedback based
on the proposed factorization-heads attention.

As illustrated in Eq.(8), our proposed factorization-heads attention can
factorize the relation between different feedback, which makes it possible for
us to study the attention weights between them. Therefore, we perform
visualization on the attention weights between positive and negative heads in
Figure 4, where $h_{1}$ and $h_{2}$ (defined at (11)) represent heads for
source and target behaviors, respectively, with corresponding feedback. From
this figure, we can observe that: (1) For the collected Micro-video dataset,
users are still willing to watch videos even after they receive the disliked
videos. This may be because the negative recommended videos are of low cost
for users as they can easily skip the disliked videos, making no significant
impact on their later preferred videos; (2) For the e-commerce dataset about
Amazon, we can discover that when the source feedback is negative, the
probability of target feedback being negative will increase sharply. This may
be because the negative purchased items are of high cost in e-commerce for
users as it will waste their money, increasing their unsatisfied emotion
sharply.

###  4.5. The Impact of Sequence Length (RQ4)

|
---|---
(a) Micro-video | (b) Amazon
Figure 5. AUC performance comparisons under different sequence lengths on the
Micro-video and Amazon datasets.

On large-scale online platforms, active users often observe a lot of items and
generate very long historical item sequences, while cold-start users are
recorded with very short sequences. Long historical item sequences can bring
them more information but the problem of gradient vanishing will increase,
while short historical item sequence brings limited information and tends to
overfit the model. Thus, we divide historical item sequences into five groups
based on their lengths and further study how DFAR outperforms the attention-
based models under different lengths, under Micro-video and Amazon datasets,
as illustrated in Figure 5. From the visualization, we can observe that:

  * •

DFAR is superior under different sequence lengths. It is obvious that there is
always a significant performance gap between DFAR and other methods. In the
Amazon dataset, where the sequence length is relatively short, the AUC
performances increase with the growth of sequence length for all methods. This
means a longer sequence can bring more information. However, in the Micro-
video dataset where the sequence length is relatively long, the performances
of all methods improve with the increase of sequence length and reach their
peak at around 50-100. But then they all decline with the further increase in
length. Most importantly, our DFAR outperforms other methods significantly
throughout various sequence lengths.

  * •

DFAR is stable under different sequence lengths. DFAR is more stable with the
sequence length increasing or decreasing, even into very long or short. In the
Amazon dataset, other methods first increase with the sequence length but
fluctuate at 15-20 while DFAR increases steadily with the sequence length. In
the Micro-video dataset, All methods drop sharply when the sequence length is
too short or long, but our DFAR is more stable and still keeps a decent AUC
performance at 0.8382.

In summary, our DFAR is superior and robust under both long and short
historical item sequences.

##  5\. Related Work

Sequential Recommendation Sequential Recommendation (Wang et al., 2019)
predicts the next interacted item of the given user based on his/her
historical items. As the early work, FPMC (Rendle et al., 2010) exploits the
Markov chain to capture the transition pattern of historical item sequence in
the recommendation. Then some advanced deep learning methods such as RNN
(Chung et al., 2014; Hochreiter and Schmidhuber, 1997) and attentive network
(Vaswani et al., 2017) are applied in recommendation (Hidasi et al., 2016;
Zhou et al., 2019; Kang and McAuley, 2018; Zhou et al., 2018) to capture the
chronological transition patterns between historical items. While the
evolution of RNN-based methods should forward each hidden state one by one and
are difficult to parallel, attention-based methods can directly capture the
transition patterns among all historical items at any time step. Furthermore,
researchers also attempt to leverage convolution neural network (Krizhevsky et
al., 2012) to capture the union and point levels sequential pattern in
recommendation (Tang and Wang, 2018). Compared with CNN-based methods,
attention-based methods are more effective for their non-local view of self-
attention (Wang et al., 2018). However, the most existing sequential
recommendation is based on click behavior. Recently, there have been some
methods of achieving sequential recommendations beyond click behaviors (Ma et
al., 2022). For example, DFN (Xie et al., 2021) captures the sequential
patterns among click, unclick and dislike behaviors by an internal module for
each behavior and an external module to purify noisy feedback under the
guidance of precise but sparse feedback. CPRS (Wu et al., 2020) derives
reading satisfaction from the completion of users on certain news to
facilitate click-based modeling. Based on them, FeedRec (Wu et al., 2022)
further enhances sequential modeling by a heterogeneous transformer framework
to capture the transition patterns between user feedback such as click,
dislike, follow, etc. However, these works mainly focus on exploiting the
auxiliary feedback to enhance the modeling in the sequential recommendation,
which does not consider the most important characteristic - the transition
patterns between historical positive and negative feedback. Differently from
them, our approach can factorize the transition patterns between different
feedback, achieving more accurate modeling for sequential recommendation with
both positive and negative feedback. Additionally, our approach extracts the
relation between positive and negative feedback at interest level.

Explainable Attention Attention methods are popular in many machine learning
fields such as recommender systems (Kang and McAuley, 2018; Sun et al., 2019;
Zhang et al., 2020), computer vision (Fu et al., 2016; Wang et al., 2018; Li
et al., 2019; Dosovitskiy et al., 2021) and natural language processing
(Bahdanau et al., 2015; Vinyals et al., 2015), etc. Attention mechanisms are
often explainable and have been widely used in deep models to illustrate the
learned representation by visualizing the distribution of attention scores or
weights under specific inputs (Choi et al., 2016; Martins and Astudillo, 2016;
Wang et al., 2016). Some explainable attention methods are also generalizable
and can be equipped with many backbones. For example, L2X (Chen et al., 2018)
exploits Gumbel-softmax (Jang et al., 2017) for feature selection by instance,
with its hard attention design (Xu et al., 2015). Moreover, VIBI (Bang et al.,
2020) further propose a feature score constraint in a global prior so as to
simplify and purify the explainable representation learning. As self-attention
is popular (Vaswani et al., 2017; Devlin et al., 2019), there is also a work
that explains what heads learn and concludes that some redundant heads can be
pruned (Voita et al., 2019). In this work, we propose feedback-aware
factorization-heads attention to explicitly capture the transition pattern
between positive and negative feedback. The feedback mask matrix in our
attention module can be treated as hard attention based on feedback.

##  6\. Conclusions and Future Work

In this work, we considered the positive and negative feedback in the
historical item sequence for the sequential recommendation, while existing
works were mostly click-based and considered solely positive feedback. Such
exploration addressed the challenge of current multi-head attention for
different feedback interactions in one sequence. More specifically, we first
applied talking-heads attention in the sequential recommendation and further
proposed feedback-aware factorization-heads attention to explicitly achieve
interaction across different heads for self-attention. Secondly, we proposed
disentanglement and pair-wise contrastive learning to repel the dissimilar
interests and capture the pair-wise relation between positive and negative
feedback. In the future, we plan deploy the model in industrial applications
to validate online performance.

## Acknowledgment

This work is supported in part by the National Key Research and Development
Program of China under 2022YFB3104702, the National Natural Science Foundation
of China under 62272262, 61971267, U1936217 and 61972223, BNRist, the
Fellowship of China Postdoctoral Science Foundation under 2021TQ0027 and
2022M710006, and the Tsinghua University Guoqiang Institute under 2021GQG1005.

##  Appendix A Appendix for Reproducibility

###  A.1. Pseudocode

Listing 1: Pseudocode for Multi-heads Attention

⬇

1def MultiHeadAttention (

2X[n, d_X], # n vectors with dimensionality d_X

3M[m, d_M], # m vectors with dimensionality d_M

4P_q[d_X, d_k, h], # learned linear projection to produce queries

5P_k[d_M, d_k, h], # learned linear projection to produce keys

6P_v[d_M, d_v, h], # learned linear projection to produce values

7P_o[d_Y, d_v, h]): # learned linear projection of output

8 Q[n, d_k, h] = einsum (X[n, d_X], P_q[d_X, d_k, h])

9 K[m, d_k, h] = einsum (M[m, d_M], P_k[d_M, d_k, h])

10 V[m, d_v, h] = einsum (M[m, d_M], P_v[d_M, d_v, h])

11

12 L[n, m, h] = einsum (Q[n, d_k , h], K[m, d_k , h]) # logits h*n*m* d_k

13

14 W[n, m, h] = softmax (L[n, m, h], reduced_dim =m) # weights

15

16 O[n, d_v , h] = einsum (W[n, m, h], V[m, d_v , h]) # h*n*m* d_v

17 Y[n, d_Y ] = einsum (O[n, d_v , h], P_o[d_Y , d_v , h]) # output h*n* d_Y *
d_v

18 return Y[n, d_Y]

We follow talking-heads attention (Shazeer et al., 2020) and present the
following notation and pseudocode.

####  A.1.1. Notation

In our pseudocode, we follow talking-heads attention (Shazeer et al., 2020)
and have a notation as below.

  * •

The capital letters represent the variable names, and lower-case letters
represent the number of dimensions. Each variable of a tensor is presented
with its dimensions. For example, a tensor for an item sequence with batch
size $b$, sequence length $n$, hidden state $d$ is written as: X[b, n, d]
(Shazeer et al., 2020).

  * •

The einsum represents the generalized contractions between tensors without any
constraint on their dimension. Its computation process is: (1) Broadcasting
each input to have the union of all dimensions, (2) multiplying component-
wise, and (3) summing across all dimensions not in the output. The dimensions
are identified by the dimension-list annotations on the arguments and on the
result instead of being identified by an equation, as in TensorFlow and NumPy.
For example, multiplying two matrices is written as: Z[a, c] = einsum (X[a,
b], W[b, c]) (Shazeer et al., 2020).

####  A.1.2. Multi-heads Attention

The pseudocode for multi-heads attention (Vaswani et al., 2017) is as shown in
Pseudocode 1, where different heads for Q and K do not interact with each
other on line 12.

####  A.1.3. Talking-heads Attention

The pseudocode for talking-heads attention (Shazeer et al., 2020) is as shown
in Pseudocode 2, where different heads for Q and K achieve implicit
interaction by lines 15 and 18.

####  A.1.4. Factorization-heads Attention

The pseudocode for our proposed factorization-heads attention is as shown in
Pseudocode 3, where different heads for Q and K achieve explicit interaction
by line 16.

Listing 2: Pseudocode for Talking-heads Attention

⬇

1def TalkingHeadAttention (

2X[n, d_X], # n vectors with dimensionality d_X

3M[m, d_M], # m vectors with dimensionality d_M

4P_q[d_X, d_k, h_k], # learned linear projection to produce queries

5P_k[d_M, d_k, h_k], # learned linear projection to produce keys

6P_v[d_M, d_v, h_v], # learned linear projection to produce values

7P_o[d_Y, d_v, h_v]

8P_l [h_k , h], # talking - heads projection for logits

9P_w [h, h_v]): # talking - heads projection for weights

10 Q[n, d_k, h_k] = einsum (X[n, d_X], P_q[d_X, d_k, h_k])

11 K[m, d_k, h_k] = einsum (M[m, d_M], P_k[d_M, d_k, h_k])

12 V[m, d_v, h_v] = einsum (M[m, d_M], P_v[d_M, d_v, h_v])

13

14 J[n, m, h_k] = einsum (Q[n, d_k, h_k], K[m, d_k, h_k]) # dot prod . n*m*
d_k *h_k

15 L[n, m, h] = einsum (J[n, m, h_k], P_l [h_k, h]) # Talking - heads proj .
n*m*h* h_k

16

17 W[n, m, h] = softmax (L[n, m, h], reduced_dim=m) # Attention weights

18 U[n, m, h_v] = einsum (W[n, m, h], P_w [h, h_v]) # Talking - heads proj .
n*m*h* h_v

19

20 O[n, d_v, h_v] = einsum (U[n, m, h_v], V[m, d_v, h_v]) # n*m* d_v * h_v

21 Y[n, d_Y] = einsum (O[n, d_v, h_v], P_o [d_Y, d_v, h_v]) # n* d_Y * d_v *
h_v

22 return Y[n, d_Y]

####  A.1.5. Comparison

From these three Python pseudocodes, we can discover that our factorization-
heads attention achieves head interaction at a low cost. The comparison of it
with multi-heads attention and talking-heads attention are as below.

  * •

Comparing with Multi-heads Attention: our factorization-heads attention
incorporates the interaction between different heads with additional four
lines at lines 12-14 and 17, which are transpose and reshape operations and
with only $O(1)$ temporal complexity.
222https://stackoverflow.com/questions/58279082/time-complexity-of-numpy-
transpose .

  * •

Comparing with Talking-heads Attention: our factorization-heads attention
achieves explicit interaction with additional transpose and reshape operations
at $O(1)$ temporal complexity while talking-heads attention achieves implicit
interaction with two matrix multiplication operations at $O(m\times
h_{k}\times h)$ and $O(m\times h\times h_{v})$ temporal complexities
333https://en.wikipedia.org/wiki/Computational_complexity_of_matrix_multiplication,
respectively.

###  A.2. Evaluation Metrics

The detailed illustration of adopted evaluation metrics is as follows.

  * •

AUC: Randomly selecting one positive item and one negative item, it represents
the probability that the predicted score of the positive item is higher than
that of the negative item. It tests the model’s ability to classify the
positive and negative items.

  * •

GAUC: It weighs each user’s AUC based on his/her test set size. It tests the
model’s personalized classification ability on each user as recommender
systems indeed tend to rank preferred items for users individually.

  * •

MRR@K: It is the average of the reciprocal of the first hit item ranking.

  * •

NDCG@K: It assigns hit items that rank higher with more weights and thus tests
the model’s ability to rank the hit items in higher and more confident
positions.

###  A.3. Implementation Details

We implement all the models by a Microsoft
444https://github.com/microsoft/recommenders TensorFlow
555https://www.tensorflow.org framework in Python, which is accessible here
666https://anonymous.4open.science/r/DFAR-8B7B. We will publish the Micro-
video dataset to benefit the community in the future, and the public Amazon
dataset is accessible at this website
777http://jmcauley.ucsd.edu/data/amazon/index_2014.html.

The environment is as below.

  * •

Anaconda 3

  * •

Python 3.7.7

  * •

TensorFlow 1.15.0

Besides, for other parameters, we stop the model training with early stop step
2 and leverage the MLP layer sandwiched between two normalization layers as
the prediction tower for each model.

###  A.4. Hyper-parameter Study (RQ5)

|
---|---
(a) Micro-video | (b) Amazon
Figure 6. AUC performance of different auxiliary loss weights w.r.t
$\lambda^{BPR}$ and $\lambda^{D}$ under Micro-video and Amazon datasets.

We perform hyper-parameter study on the weights for loss of disentanglement
and pair-wise contrastive Learning (w.r.t. $\lambda^{BPR}$ and $\lambda^{D}$
at Eq.(23)) as Figure 6, varying the loss weights from $10^{-4}$ to $10^{-1}$.
From the figure, we can observe that the AUC performance reaches the peak at
$10^{-3}$ under the Amazon dataset while that reaches the peak at $10^{-2}$
under the Micro-video dataset. This is because the rating for Amazon is a
discrete value, but the playing time for Micro-video is a continuous value.
The partition of positive and negative feedback based on continuous value is
unclear and thus requires more contrastive learning. Based on the above
observation, we finally choose $10^{-3}$ and $10^{-2}$ as the best values for
the loss weights under Amazon and Micro-video datasets, respectively.

Listing 3: Pseudocode for Factorization-heads Attention

⬇

1def FactorizationHeadAttention (

2X[n, d_X], # n vectors with dimensionality d_X

3M[m, d_M], # m vectors with dimensionality d_M

4P_q[d_X, d_k, h], # learned linear projection to produce queries

5P_k[d_M, d_k, h], # learned linear projection to produce keys

6P_v[d_M, d_v, h], # learned linear projection to produce values

7P_o[d_Y, d_v, h]): # learned linear projection of output

8 Q[n, d_k, h] = einsum (X[n, d_X], P_q[d_X, d_k, h])

9 K[m, d_k, h] = einsum (M[m, d_M], P_k[d_M, d_k, h])

10 V[m, d_v, h] = einsum (M[m, d_M], P_v[d_M, d_v, h])

11

12 Q[n, h, d_k] = reshape(transpose(Q, [0, 2, 1]), [n * h, d_k]) # queries
h*n* d_X * d_k

13 K[d_k, h, m] = reshape(transpose(K, [1, 2, 0]), [d_k, h * m]) # keys h*m*
d_M * d_k

14 V[m, d_v, h * h] = tile(V[m, d_v, h], [1, 1, h]) # values h*m* d_M * d_v

15

16 L[n * h, h * m] = einsum (Q[n * h, d_k], K_[d_k, h * m])

17 L[n, h * h, m] = transpose(reshape(L, [n, h * h, m]), [0, 2, 1]) # logits
h*h*n*m* d_k

18

19 W[n, m, h * h] = softmax (L[n, m, h * h], reduced_dim=m) # weights

20 O[n, d_v , h * h] = einsum (W[n, m, h * h], V[m, d_v, h * h]) # h*h*n*m*
d_v

21 Y[n, d_Y] = einsum (O[n, d_v, h * h], P_o[d_Y, d_v, h * h]) # output h*h*n*
d_Y * d_v

22 return Y[n, d_Y]

## References

  * (1)
  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015\.  Neural Machine Translation by Jointly Learning to Align and Translate. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , Yoshua Bengio and Yann LeCun (Eds.).  http://arxiv.org/abs/1409.0473
  * Bang et al. (2020) Seojin Bang, Pengtao Xie, Heewook Lee, Wei Wu, and Eric Xing. 2020.  Explaining A Black-box By Using A Deep Variational Information Bottleneck Approach.  https://openreview.net/forum?id=BJlLdhNFPr
  * Chen et al. (2018) Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. 2018\.  Learning to explain: An information-theoretic perspective on model interpretation. In _International Conference on Machine Learning_. PMLR, 883–892.
  * Choi et al. (2016) Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. 2016\.  Retain: An interpretable predictive model for healthcare using reverse time attention mechanism.  _Advances in neural information processing systems_ 29 (2016).
  * Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2014.  Empirical evaluation of gated recurrent neural networks on sequence modeling. In _NIPS 2014 Workshop on Deep Learning, December 2014_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.  https://doi.org/10.18653/v1/N19-1423
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021\.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=YicbFdNTTy
  * Fu et al. (2016) Kun Fu, Junqi Jin, Runpeng Cui, Fei Sha, and Changshui Zhang. 2016.  Aligning where to see and what to tell: Image captioning with region-based attention and scene-specific contexts.  _IEEE transactions on pattern analysis and machine intelligence_ 39, 12 (2016), 2321–2334.
  * Gunawardana and Shani (2015) Asela Gunawardana and Guy Shani. 2015.  Evaluating Recommender Systems.  In _Recommender Systems Handbook_ , Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). Springer US, 265–308.  https://doi.org/10.1007/978-1-4899-7637-6_8
  * Guo et al. (2019) Lei Guo, Hongzhi Yin, Qinyong Wang, Tong Chen, Alexander Zhou, and Nguyen Quoc Viet Hung. 2019.  Streaming session-based recommendation. In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_. 1569–1577.
  * Hidasi et al. (2016) Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2016.  Session-based recommendations with recurrent neural networks. In _ICLR_.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Jang et al. (2017) Eric Jang, Shixiang Gu, and Ben Poole. 2017.  Categorical Reparameterization with Gumbel-Softmax. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=rkE3y85ee
  * Kang and McAuley (2018) Wang-Cheng Kang and Julian McAuley. 2018.  Self-attentive sequential recommendation. In _2018 IEEE International Conference on Data Mining (ICDM)_. IEEE, 197–206.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization. In _ICLR_.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012\.  Imagenet classification with deep convolutional neural networks.  25 (2012), 1097–1105.
  * Li et al. (2019) Yanwei Li, Xinze Chen, Zheng Zhu, Lingxi Xie, Guan Huang, Dalong Du, and Xingang Wang. 2019.  Attention-guided unified network for panoptic segmentation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 7026–7035.
  * Li et al. (2016) Yuchen Li, Dongxiang Zhang, Ziquan Lan, and Kian-Lee Tan. 2016\.  Context-aware advertisement recommendation for high-speed social news feeding. In _2016 IEEE 32nd International Conference on Data Engineering (ICDE)_. IEEE, 505–516.
  * Lin et al. (2022) Guanyu Lin, Chen Gao, Yinfeng Li, Yu Zheng, Zhiheng Li, Depeng Jin, and Yong Li. 2022.  Dual Contrastive Network for Sequential Recommendation. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 2686–2691.
  * Ma et al. (2022) Mingyuan Ma, Sen Na, Hongyu Wang, Congzhou Chen, and Jin Xu. 2022.  The graph-based behavior-aware recommendation for interactive news.  _Applied Intelligence_ 52, 2 (2022), 1913–1929.
  * Martins and Astudillo (2016) Andre Martins and Ramon Astudillo. 2016.  From softmax to sparsemax: A sparse model of attention and multi-label classification. In _International conference on machine learning_. PMLR, 1614–1623.
  * Okura et al. (2017) Shumpei Okura, Yukihiro Tagami, Shingo Ono, and Akira Tajima. 2017\.  Embedding-based news recommendation for millions of users. In _Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining_. 1933–1942.
  * Rendle et al. (2009) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Schmidt-Thie Lars. 2009.  BPR: Bayesian personalized ranking from implicit feedback. In _Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence_ (Montreal, Quebec, Canada) _(UAI ’09)_. AUAI Press, Arlington, Virginia, United States, 452–461.  http://dl.acm.org/citation.cfm?id=1795114.1795167
  * Rendle et al. (2010) Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2010\.  Factorizing personalized markov chains for next-basket recommendation. In _WWW_. 811–820.
  * Shazeer et al. (2020) Noam Shazeer, Zhenzhong Lan, Youlong Cheng, Nan Ding, and Le Hou. 2020.  Talking-heads attention.  _arXiv preprint arXiv:2003.02436_ (2020).
  * Sun et al. (2019) Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019.  BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In _Proceedings of the 28th ACM international conference on information and knowledge management_. 1441–1450.
  * Tang and Wang (2018) Jiaxi Tang and Ke Wang. 2018.  Personalized top-n sequential recommendation via convolutional sequence embedding. In _WWW_. 565–573.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _NeurIPS_. 5998–6008.
  * Vinyals et al. (2015) Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015\.  Grammar as a foreign language.  _Advances in neural information processing systems_ 28 (2015).
  * Voita et al. (2019) Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019.  Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 5797–5808.  https://doi.org/10.18653/v1/P19-1580
  * Wang et al. (2020) Heyuan Wang, Fangzhao Wu, Zheng Liu, and Xing Xie. 2020\.  Fine-grained interest matching for neural news recommendation. In _Proceedings of the 58th annual meeting of the association for computational linguistics_. 836–845.
  * Wang et al. (2019) Shoujin Wang, Liang Hu, Yan Wang, Longbing Cao, Quan Z. Sheng, and Mehmet Orgun. 2019\.  Sequential recommender systems: challenges, progress and prospects.  (2019).
  * Wang et al. (2021) Wenjie Wang, Fuli Feng, Xiangnan He, Liqiang Nie, and Tat-Seng Chua. 2021.  Denoising Implicit Feedback for Recommendation. In _Proceedings of the 14th ACM International Conference on Web Search and Data Mining_. ACM.  https://doi.org/10.1145/3437963.3441800
  * Wang et al. (2018) Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. 2018.  Non-local neural networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 7794–7803.
  * Wang et al. (2016) Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao. 2016\.  Attention-based LSTM for aspect-level sentiment classification. In _Proceedings of the 2016 conference on empirical methods in natural language processing_. 606–615.
  * Wu et al. (2020) Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2020\.  User Modeling with Click Preference and Reading Satisfaction for News Recommendation.. In _IJCAI_. 3023–3029.
  * Wu et al. (2022) Chuhan Wu, Fangzhao Wu, Tao Qi, Qi Liu, Xuan Tian, Jie Li, Wei He, Yongfeng Huang, and Xing Xie. 2022\.  Feedrec: News feed recommendation with various user feedbacks. In _Proceedings of the ACM Web Conference 2022_. 2088–2097.
  * Xie et al. (2021) Ruobing Xie, Cheng Ling, Yalong Wang, Rui Wang, Feng Xia, and Leyu Lin. 2021\.  Deep feedback network for recommendation. In _Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence_. 2519–2525.
  * Xu et al. (2015) Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015\.  Show, attend and tell: Neural image caption generation with visual attention. In _International conference on machine learning_. PMLR, 2048–2057.
  * Zhang et al. (2020) Junqi Zhang, Bing Bai, Ye Lin, Jian Liang, Kun Bai, and Fei Wang. 2020.  General-purpose user embeddings based on mobile app usage. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2831–2840.
  * Zhou et al. (2019) Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. 2019.  Deep interest evolution network for click-through rate prediction. In _AAAI_. 5941–5948.
  * Zhou et al. (2018) Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018\.  Deep interest network for click-through rate prediction. In _KDD_. 1059–1068.