# Better Generalization with Semantic IDs: A Case study in Ranking for
Recommendations

Anima Singh1 Trung Vu Raghunandan Keshavan Nikhil Mehta Xinyang Yi
Lichan Hong Lukasz Heldt Li Wei Ed H. Chi Maheswaran Sathiamoorthy2
1<EMAIL_ADDRESS>2<EMAIL_ADDRESS>
###### Abstract

Training good representations for items is critical in recommender models.
Typically, an item is assigned a unique randomly generated ID, and is commonly
represented by learning an embedding corresponding to the value of the random
ID. Although widely used, this approach have limitations when the number of
items are large and items are power-law distributed— typical characteristics
of real-world recommendation systems. This leads to the item cold-start
problem, where the model is unable to make reliable inferences for tail and
previously unseen items. Removing these ID features and their learned
embeddings altogether to combat cold-start issue severely degrades the
recommendation quality. Content-based item embeddings are more reliable, but
they are expensive to store and use, particularly for users’ past item
interaction sequence. In this paper, we use Semantic IDs, a compact discrete
item representations learned from content embeddings using RQ-VAE that
captures hierarchy of concepts in items. We showcase how we use them as a
replacement of item IDs in a resource-constrained ranking model used in an
industrial-scale video sharing platform. Moreover, we show how Semantic IDs
improves the generalization ability of our system, without sacrificing top-
level metrics.

##  1 Introduction

Recommender systems are widely used across the industry to serve personalized
content to users. They play a critical role in helping users discover novel
content such as apps (Cheng et al., 2016), music (Kim et al., 2007; Koren et
al., 2009) and videos (Covington et al., 2016; Zhao et al., 2019; Gomez-Uribe
and Hunt, 2015). In this paper, we consider a neural ranking model in a large
industrial-scale video recommendation system. Our recommendation corpus has
100s of millions of videos. Every video gets a unique identifier (called video
ID), which is a random string, devoid of any meaning. The ranking model gets
as input multiple video IDs, for example, what the user is watching and what
is to be recommended. Furthermore, users’ features are also typically
represented as a list of video IDs they have previously watched - a critical
signal for personalization. Given that the central goal of a recommendation
system is to connect users to videos, learning good representations of video
ID is critical.

The widely used technique to encode categorical features such as video ID is
to learn high-dimensional vectors (embeddings). Embedding learning is widely
used across many models and is well understood in domains such as natural
language processing. However, given the size of the video corpus and the
power-law distribution of the views, there are a number of challenges
associated with learning good embeddings for video IDs.

Large corpus size and power-law distribution: Given our extremely large video
corpus with 100s of millions of videos, learning one embedding vector per
video can be quite resource-intensive. More importantly, we have a long-tail
of videos with very few views. This leads to the cold-start problem, i.e., we
are unable to learn good representations for those videos. Furthermore, in our
recommendation system, many new videos get uploaded everyday, which makes it
hard to maintain a reliable 1:1 mapping.

Random collisions: The alternative approach is to use the hashing trick
(Weinberger et al., 2009) that maps many videos to the same row — a technique
we adopt in our ranking model. However, this causes random collisions because
the IDs of videos are random strings.

Due to these inherent limitations, we are motivated to develop better ways to
learn embeddings for the video features. A possible approach is use content-
based embeddings that we have access to in the system. These embeddings
capture the topicality of the video in a fine-grained manner based on audio-
visual features. But as we see in Section 5.2, replacing the embedding table
of video IDs with these content embeddings causes a large drop in the capacity
of the system, affecting model quality.

In this work, we replace video IDs with Semantic IDs (proposed in Rajput et
al. (2023)) and develop a methodology to learn video representations based on
these Semantic IDs. We use RQ-VAE to quantize content embeddings into tuples
of integers (e.g. (33, 202, 11, 2)) called Semantic IDs. Similar videos have
overlapping set of integers, allowing for semantic collisions when learning
embeddings.

Our contributions are as follows:

  1. 1.

We demonstrate how we obtain Semantic IDs for videos in our recommendation
system using content embeddings. We show that Semantic IDs capture meaningful
hierarchical relationships between videos.

  2. 2.

We propose a two-stage approach: 1) Efficient compression of content
embeddings into Semantic IDs and 2) Training downstream ranking model with
Semantic IDs. The efficient compression in Stage 1 means that storing Semantic
IDs use a small fraction of storage relative to content embeddings. This
unlocks the possibility of using content signals for personalization based on
users’ watch history, in resource-constrained production models.

  3. 3.

Through extensive experiments on data from our industrial video recommendation
platform, we demonstrate that semantically meaningful collisions are superior
to random hashing, given comparable model capacity. We also show that Semantic
IDs can replace video IDs to improve generalization of our industrial-scale
ranking model.

Since Semantic IDs offer generalization benefits, they also provide relief
from the popularity bias issue, wherein popular videos get better
representations due to their higher exposure, which causes them to be
recommended more.

##  2 Related Work

Embedding learning: Recommender models rely on learning good representation of
categorical features. A common technique to encode categorical features is to
train embeddings using one-hot embeddings. Word2vec (Mikolov et al., 2013)
popularized this in the context of language models. Hashing trick (Weinberger
et al., 2009) is typically used when the cardinality is high, but it causes
random collisions. Multiple hashing (Zhang et al., 2020) offers some relief,
but still leads to random collisions. Deep Hash Embedding (Kang et al., 2021)
circumvents this problem by not maintaining embedding tables but at the cost
of increased computation in the hidden layers. In contrast, we use Semantic
IDs — a compute efficient way to avoid random collisions during embedding
learning for item IDs. By enabling collisions between semantically related
items, Semantic IDs improve generalization in recommender models.

Cold-start and content information: Content-based recommender models have been
proposed to combat cold-start issues (e.g. Schein et al. (2002), Volkovs et
al. (2017b)). Recently, embeddings derived from content information are also
popular (e.g. DropoutNet (Volkovs et al., 2017a), CC-CC (Shi et al., 2019) and
Du et al. (2020)). Content embeddings have also been incorporated to improve
recommendations beyond cold-start applications. For example, PinSage (Ying et
al., 2018) aggregates visual, text and engagement information to represent
items. And PinnerFormer (Pancha et al., 2022) uses sequences of PinSage
embeddings corresponding to item history to build a sequential recommendation
model. In contrast to these efforts, our goal is to develop content-derived
representations that not only generalize well, but can also improve
performance relative to using video ID features — a significantly challenging
task. Furthermore, unlike PinnnerFormer (Pancha et al., 2022) which is used
for offline inference, our focus is to improve generalization of a ranking
model used for real-time inference. Therefore, approaches that significantly
increase resource costs (including storage, training and serving) make them
infeasible to deploy in production. Semantic IDs offer an efficient
compression of content embeddings into discrete tokens, making it feasible to
use content signals in production recommendation systems.

Discrete representations: Several techniques exist to discretize embeddings.
For instance, VQ-VAE (Van Den Oord et al., 2017), VQ-GAN (Esser et al., 2021)
and their variants are used for generative modeling: Parti (Yu et al., 2022)
uses ViT-VQGAN (Yu et al., 2021) for generative image modeling and SoundStream
(Zeghidour et al., 2021) uses RQ-VAE for generative audio modeling. TIGER
(Rajput et al., 2023) used RQ-VAE in the context of recommender applications,
which we utilize as well. Traditional techniques like Product Quantization
(Jegou et al., 2010) and it’s variants are used by many recommender models
(e.g. MGQE (Kang et al., 2020) and Hou et al. (2022)), but these do not offer
hierarchical semantics, which we leverage in our work.

##  3 Background

In this section, we present key aspects of our real-world industrial-scale
video ranking system. These factors provide context and define the constraints
and requirements that we need to consider.

Ranker Model: Our production ranking model is a multitask model for video
recommendation, which recommends the next video to watch given a video a user
is currently watching. Video ID features are the most important features to
the model. We represent the current video and the video to be ranked as video
IDs. Furthermore, we also represent a user as a sequence of videos IDs they
have interacted with - a critical signal for personalized recommendations.

Sequential Training: The model is trained continuously in a sequential manner,
i.e., the training is done using logged data in a chronological manner, and
the training continues as new data come in. Since the underlying video corpus
and users are constantly evolving, the model needs to be able to generalize
well under data-distribution shift. In this paper, we focus on data-
distribution shift of the video corpus.

Resource Constraints: Due to resource constraints and latency budgets dictated
by real-time inference, we cannot increase the resource usage by a lot. For
example, a possible approach to avoid using video IDs is to directly use
content embeddings as input to the model. However, this approach isn’t viable
due to high resource costs. For instance, storing 100s of 256-dimension floats
to represent users’ watch history per training example, for industrial
applications with billions of examples, does not scale well. Loading and
processing such large inputs also significantly slows down training and
increases latency at inference. Furthermore, if we use content embeddings to
replace video IDs, we lose the model capacity associated with the video
embedding parameters that can lead to model quality degradation. In theory, we
can increase the hidden layer capacity to compensate. However, in practice,
this exacerbates computational costs, since model capacity in form of dense
layers is much more compute-heavy than embedding parameters. Due to resource
constraints and latency budgets dictated by real-time inference, we cannot
adopt this approach.

##  4 Our Proposed Approach

###  4.1 Overview

Given content embeddings for our corpus of videos, we propose an efficient
two-stage approach***Using content embeddings directly in the ranking model
would be an example of a single-stage alternative. to leverage content signal
in a large scale ranking model, with the goal to improve generalization under
data-distribution shift.

  * •

Stage 1: Efficient compression of content embeddings into discrete Semantic
IDs. We use a Residual Quantization technique called RQ-VAE (Lee et al., 2022;
Zeghidour et al., 2021; Rajput et al., 2023) to quantize content embeddings -
a 256 dimensional floats, into discrete tokens to capture semantic information
about a video. We store all the tokens corresponding to a video using a single
64 bit integer. This translates to a $(256\times
4~{}\text{bytes})/(8~{}\text{bytes})=128\times$ compression in terms of
storage for a single instance of a video. This compression was imperative to
leverage content signals for personalization using user’s watch history. Once
trained, we freeze the trained RQ-VAE model and use it for training downstream
ranking model in Stage 2.

  * •

Stage 2: Training Ranking model with Semantic IDs. We use the quantization
model from Stage 1, to map each video to its Semantic ID and then train
embeddings for Semantic ID n-grams, along with the rest of the ranking model
(Section 4.3).

##### Design Choices

A key design choice in our proposal is to train and then freeze the RQ-VAE
model from Stage 1. The frozen RQ-VAE model is used to generate Semantic IDs
for training and serving the ranking model. As discussed in Section 3, the
ranking model is sequentially trained on recently logged data. Recent data may
include videos that did not exist when the RQ-VAE model was trained. This
raises a potential concern that freezing the model could hurt the performance
of the ranking model over time. In Section 5.3.3, we show that the performance
of ranking models that use Semantic IDs derived from the two RQ-VAE models
trained on older vs. more recent data perform comparably, suggesting that the
learned semantics are stable over time. In our work, we use RQ-VAE (Lee et
al., 2022; Zeghidour et al., 2021; Rajput et al., 2023) instead of VQ-VAE (Van
Den Oord et al., 2017) to quantize content embeddings (Section 4.2), since RQ-
VAE organically captures hierarchical semantic structure (Figure 2), offering
interpretability.

###  4.2 RQ-VAE for Semantic IDs

Given a content embedding vector for a video, we generate Semantic IDs using
Residual-Quantized Variational AutoEncoder (RQ-VAE) Lee et al. (2022);
Zeghidour et al. (2021); Rajput et al. (2023) that applies quantization on
residuals at multiple levels (see Figure 1).

There are three jointly-trained components: (1) an encoder ${\mathcal{E}}$
that maps the content embedding ${\bm{x}}\in\mathbb{R}^{D}$ to a latent vector
${\bm{z}}\in\mathbb{R}^{D^{\prime}}$, (2) a residual vector-quantizer with $L$
levels, each with a codebook
${\mathcal{C}}_{l}:=\\{{\bm{e}}^{l}_{k}\\}_{k=1}^{K}$, where
${\bm{e}}^{l}_{k}\in\mathbb{R}^{D^{\prime}}$ and $K$ is the codebook size; the
vector-quantizer recursively quantizes the residual ${\bm{r}}_{l}$ at each
level $l$ to the nearest codebook vector ${\bm{e}}_{c_{l}}$ (Figure 1) and (3)
a decoder ${\mathcal{D}}$ that maps the quantized latent $\hat{{\bm{z}}}$ back
to the original embedding space $\hat{x}$.

Figure 1: RQ-VAE: The input vector ${\bm{x}}$ is encoded into a latent
${\bm{z}}$, which is then recursively quantized by looking up the nearest
codebook vector of the residual at each level. Each orange box represents a
codebook. Within these boxes, each circle labeled with integer indices ($1$ to
$5$) represents a codebook vector, and each circle labeled with
${\bm{z}},{\bm{r}}_{1},{\bm{r}}_{2},{\bm{r}}_{3}$ represents the vector being
quantized. We compute the reconstruction vector $\hat{{\bm{x}}}$ by feeding
the quantized latent $\hat{{\bm{z}}}$ into the decoder. The sequence of
indices of the nearest codebook vector at each level represents the Semantic
ID for the item. In this figure, the item represented by ${\bm{x}}$ has
$(1,4,5,2)$ as its Semantic ID.

We use the following loss to train the RQ-VAE model:
${\mathcal{L}}={\mathcal{L}}_{recon}+{\mathcal{L}}_{rqvae}$, where
${\mathcal{L}}_{recon}=\|{\bm{x}}-\hat{{\bm{x}}}\|^{2}$ and
${\mathcal{L}}_{rqvae}=\sum_{l=1}^{L}\
\beta\|{\bm{r}}_{l}-\text{sg}[{\bm{e}}_{c_{l}}]\|^{2}+\|\text{sg}[{\bm{r}}_{l}]-{\bm{e}}_{c_{l}}\|^{2}$
and sg denotes the stop-gradient operator. ${\mathcal{L}}_{recon}$ aims to
reconstruct the content embedding ${\bm{x}}$. The first and the second terms
in ${\mathcal{L}}_{rqvae}$ encourages the encoder and the codebook vectors to
be trained such that ${\bm{r}}_{l}$ and ${\bm{e}}_{c_{l}}$ move towards each
other.

####  4.2.1 Semantic IDs as hierarchy of concepts

We illustrate the hierarchy of concepts captured by Semantic IDs from the
videos in our corpus. Section 5.1 details the hyper-parameters used to train
the RQ-VAE model. Intuitively, we can think of Semantic IDs as forming a trie
over videos, with higher levels representing coarser concepts and lower levels
representing more fine-grained concepts. Figure 2 shows an example subtrie
from our trained RQ-VAE model that captures a hierarchy within sports.

Figure 2: A subtrie that captures sports videos.

Table 1 shows the average pairwise cosine similarity in the content embedding
space for all videos with a shared Semantic ID prefix of length $n$ and its
corresponding subtrie size. We consider two videos with, say, Semantic IDs
$(1,2,3,4)$ and $(1,2,6,7)$ to have a shared prefix of length $2$. We observe
that as the shared prefix length increases, average pairwise cosine similarity
increases while the subtrie size decreases. These suggests that Semantic ID
prefixes represent increasingly granular concepts as their lengths increase.

Table 1: Aggregate metrics for videos sharing Semantic ID prefix of length $n$. The typical subtrie size refers to the 25th-75th percentile range (with rounding). Shared prefix length | Average pairwise cosine similarity | Typical subtrie size
---|---|---
1 | 0.41 | 150,000-450,000
2 | 0.68 | 20-150
3 | 0.91 | 1-5
4 | 0.97 | 1

###  4.3 Semantic ID based Video Representation in Ranking

In this section, we discuss how we generate a video representation derived
from Semantic IDs to use in the ranking model. For a given video $v$, an RQ-
VAE model (Section 4.2) with $L$ levels generates a Semantic ID
$(c^{v}_{1},...c^{v}_{L})$.

We propose an n-gram based representation of Semantic ID to represent a video.
An n-gram is a sequence of n-tokens. First, we extract each n-gram of the
Semantic ID for a given video. Let $|\text{n-grams}|$ represent the number of
n-grams per video. For example, for unigram, we extract $|\text{n-grams}|=L$
unigrams per video. Next, we associate a separate embedding row for each
distinct value of the n-gram. More specifically, for an n-gram of Semantic IDs
derived from an RQ-VAE model with codebook size $K$, we train
$|\text{n-grams}|$ embedding tables each with $K^{n}$ rows. As a result,
during training, videos that share Semantic ID tokens will collide to the same
embedding row. This allows us to generalize across related popular and long-
tail videos. Finally, we sum over $|\text{n-grams}|$ embeddings to generate a
feature representation for a video. We train the embedding tables along with
the ranking model.

##  5 Experiments

In this section, we evaluate the generalization performance of using content
signals to learn video representations for ranking. We either directly use
content-embeddings as the video representation or use embeddings derived from
Semantic IDs in lieu of video IDs. We compare their performance relative to
the baseline that relies on random hashing of video IDs.

###  5.1 Experimental Setup

####  5.1.1 Ranking Model

We use a simplified version of the multitask production ranking model (Section
3) for our experiments by only keeping an important subset of the tasks and
input features. The ranking model uses 10s of millions of buckets for random
hashing to accommodate 100s of millions of videos in our corpus and is trained
sequentially on engagement data. We use random hashing of video IDs for three
key features: users’ watch history, current video, and the candidate video to
be ranked.

####  5.1.2 RQ-VAE Model

We use a $256$ dimensional content embedding that incorporates visual and
audio information about the video as the input. For the RQ-VAE model, we use a
$3$-layer encoder with dimensions 256, 128 and 64 with ReLU activation for the
first 2 layers. Similarly, we use a 3-layer decoder with dimensions 64, 128,
and 256, with ReLU activation for the first 2 layers. We apply $L=4$ levels of
quantization using codebook size $K$222We omitted the value of $K$ here for
business-compliance reasons. for each. Given these settings, each video $v$
has $4$ unigrams, i.e., $\\{c^{v}_{1},c^{v}_{2},c^{v}_{3},c^{v}_{4}\\}$ and
$3$ bigrams, i.e., $\\{(c^{v}_{1}\times c^{v}_{2}),(c^{v}_{2}\times
c^{v}_{3}),(c^{v}_{3}\times c^{v}_{4})\\}$.

Vector quantization techniques are known to suffer from codebook collapse
(Dhariwal et al., 2020), where the model only uses a small proportion of
codebook vectors. We reset unused codebook vectors to a random vector within a
batch (Zeghidour et al., 2021) to improve codebook utilization. We used
$\beta=0.25$ to compute the loss and trained the model until the
reconstruction loss stabilized ($\approx$10s of millions steps for our
corpus).

####  5.1.3 Evaluation metrics

We sequentially train the ranking model on the first $N$ days of data, where
each day contains logged data generated from user interactions on that day. We
evaluate the model’s performance using AUC for CTR (indicated as CTR/AUC) —
one of the important training tasks, on the data from ($N+1$)-th day. We
further slice metric on items that are introduced on the ($N+1$)-th day. We
refer to this as CTR-1D/AUC. CTR/AUC and CTR-1D/AUC metrics evaluate model’s
ability to generalize over time in the face of data distribution shift and on
cold-start items, respectively. A $0.1\%$ change in CTR/AUC is considered
significant for our ranking model.

###  5.2 Performance of Content Embeddings

As discussed in Section 3, storing dense embeddings for each video in users’
watch history is extremely resource intensive. Hence, we were unable to train
a large-scale ranking model that directly uses content embeddings to model
users’ watch history. To study the performance of content embeddings, we train
a production-scale model that does not rely on users’ watch history signal,
and uses content embedding to represent the current video a user is watching,
and the candidate video to be ranked. For fair comparison, the baseline also
excludes users’ watch history signal, and uses random hashing of video IDs for
the other two features.

In Table 2, we show that directly using content embeddings, to replace random
hashing of video IDs, improves the performance on new videos. However, the
overall ranking performance drops significantly. This illustrates that using
content embeddings as input can improve generalization on cold-start items.
However, the overall performance drops. While it’s possible to improve model
performance by increasing dense layer parameters, this option would lead to
prohibitively high serving cost in our use case. Instead, we focus on better
leveraging content information with a fixed dense-layer size through
alternative methods, such as Semantic IDs, that can leverage content signals
more efficiently.

Table 2: Performance comparison of directly using content embeddings relative
to the baseline with Video-ID based random hashing.

|  %$\Delta$ CTR/AUC |  %$\Delta$ CTR-1D/AUC
---|---|---
Content embedding | -0.10 | 0.16

###  5.3 Performance of Semantic IDs

We report results from extensive experiments in order to answer the following
research questions:

  * •

RQ1: Are semantically meaningful collisions better than random collisions
given the same model capacity?

  * •

RQ2: Can Semantic IDs based content representation replace video ID in
production settings?

  * •

RQ3: Does freezing the RQ-VAE model affect production ranking performance over
time?

####  5.3.1 Semantic Collisions vs. Random Collisions (RQ1)

For this study, we use baselines that have the same model capacity, in terms
of dense and embedding layers, as the models that use different Semantic IDs-
based representation. While these baselines have significantly fewer random
hashing buckets that the production model described in Section 5.1.1, this
setting allows us to understand the benefits of collisions between
semantically related videos vs. random collisions. We defer the evaluation of
Semantic IDs relative to production settings to Section 5.3.2.

Table 3 shows that Semantic IDs consistently outperform random hashing of
video IDs, for both unigram and bigram based representation. The improvement
in CTR/AUC evaluated on the next day’s data, i.e., data not seen at training
time, offers evidence for better generalization under data-distribution shift.
We see a larger increase in performance for videos that are introduced within
24 hours (i.e., CTR-1D/AUC), suggesting that generalization benefits from
semantic collisons are higher for cold-start videos.

Table 3: Performance comparison of Semantic ID based representation relative
to a corresponding the baseline with the same model capacity. All models have
the same dense network architecture.

|  # of embedding rows |  embedding dimension |  %$\Delta$ CTR/AUC |  %$\Delta$ CTR-1D/AUC
---|---|---|---|---
VID-Random-hashing | $4\times{K}$ | 256 | - | -
SID-4Unigram-sum | $4\times{K}$ | 256 | +0.14 | +0.23
VID-Random-hashing | $3\times{K^{2}}$ | 256 | - | -
SID-3Bigram-sum | $3\times{K^{2}}$ | 256 | +0.12 | +0.27

####  5.3.2 Semantic IDs to Replace video ID in Production Settings (RQ2)

In this section, we evaluate the applicability of Semantic IDs†††We only
explore unigram and bigram based representation since trigram based
representation would significantly increase embedding parameters relative to
production settings. to replace video IDs in a large-scale production setting.
Hence, we use the production ranking model described in Section 5.1.1 as our
baseline.

Table 4 shows gains in CTR-1D/AUC for both Semantic IDs based representations,
showcasing their generalization ability on cold-start videos. With more
semantically meaningful collisions, SID-4unigram-sum is able to yield a even
higher gain for cold-start videos than SID-3bigram-sum. However, due to a
significant reduction in embedding parameters, SID-4unigram-sum underperforms
relative to the baseline in terms of overall model quality. On the other hand,
SID-3bigram-sum, with fewer embedding parameters than the baseline, is able to
improve overall performance. This highlights that SID-3bigram-sum is
parameter-efficient and can be used in production settings to replace video
IDs to improve generalization.

Table 4: Performance comparison of random hashing of video ID vs Semantic ID
based hashing in Production settings.

|  %$\Delta$ embedding parameters |  %$\Delta$ CTR/AUC |  %$\Delta$ CTR-1D/AUC
---|---|---|---
SID$-$4Unigram$-$sum | $\approx-99$ |  $-$0.05 | +0.35
SID$-$3Bigram$-$sum | $\approx-20$ | +0.12 | +0.19

####  5.3.3 Stability of Semantic IDs over time (RQ3)

To study the stability of Semantic IDs, we train two RQ-VAE models: RQ-VAEv0
and RQ-VAEv1, using data that are 6 months apart. Figure 3 shows that the
performance of production ranking model trained on recent engagement data
(using SID-3Bigram-sum) are comparable for Semantic IDs derived from both RQ-
VAEv0 and RQ-VAEv1. This confirms that Semantic token space for videos learned
via RQ-VAE are stable over time for use in downstream production ranking
model.

Figure 3: Comparison of ranking performance of SID-3Bigram-sum representation
derived using RQ-VAEv0 and RQ-VAEv1.

##  6 Conclusion and Future Work

In this paper, we discussed the disadvantages of using video ID features in
our ranking model used in an industrial video sharing platform. We proposed
the use of Semantic IDs derived from content embeddings and demonstrated how
they can improve generalization by introducing meaningful collisions. In
contrast to other work that directly uses content embedding as a feature,
Semantic IDs offers a compact content representation for videos that makes it
feasible to use content signals for user watch history features — critical for
personalization. We demonstrated approaches to make effective use of Semantic
IDs in our ranking model by employing a bigram scheme.

In the future, we plan to more thoroughly investigate the generalization
benefits of Semantic IDs with varying number of levels and codebook sizes.
Beyond applications in ranking models, we plan to explore Semantic IDs for
sequential recommendations.

## References

  * Cheng et al. (2016) H.-T. Cheng, L. Koc, J. Harmsen, T. Shaked, T. Chandra, H. Aradhye, G. Anderson, G. Corrado, W. Chai, M. Ispir, et al.  Wide & deep learning for recommender systems.  In _Proceedings of the 1st workshop on deep learning for recommender systems_ , pages 7–10, 2016.
  * Covington et al. (2016) P. Covington, J. Adams, and E. Sargin.  Deep neural networks for youtube recommendations.  In _Proceedings of the 10th ACM conference on recommender systems_ , pages 191–198, 2016.
  * Dhariwal et al. (2020) P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever.  Jukebox: A generative model for music, 2020.
  * Du et al. (2020) X. Du, X. Wang, X. He, Z. Li, J. Tang, and T.-S. Chua.  How to learn item representation for cold-start multimedia recommendation?  In _Proceedings of the 28th ACM International Conference on Multimedia_ , pages 3469–3477, 2020.
  * Esser et al. (2021) P. Esser, R. Rombach, and B. Ommer.  Taming transformers for high-resolution image synthesis.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 12873–12883, 2021.
  * Gomez-Uribe and Hunt (2015) C. A. Gomez-Uribe and N. Hunt.  The netflix recommender system: Algorithms, business value, and innovation.  _ACM Transactions on Management Information Systems (TMIS)_ , 6(4):1–19, 2015.
  * Hou et al. (2022) Y. Hou, Z. He, J. McAuley, and W. X. Zhao.  Learning vector-quantized item representation for transferable sequential recommenders.  _arXiv preprint arXiv:2210.12316_ , 2022.
  * Jegou et al. (2010) H. Jegou, M. Douze, and C. Schmid.  Product quantization for nearest neighbor search.  _IEEE transactions on pattern analysis and machine intelligence_ , 33(1):117–128, 2010.
  * Kang et al. (2020) W.-C. Kang, D. Z. Cheng, T. Chen, X. Yi, D. Lin, L. Hong, and E. H. Chi.  Learning multi-granular quantized embeddings for large-vocab categorical features in recommender systems.  In _Companion Proceedings of the Web Conference 2020_ , pages 562–566, 2020.
  * Kang et al. (2021) W.-C. Kang, D. Z. Cheng, T. Yao, X. Yi, T. Chen, L. Hong, and E. H. Chi.  Learning to embed categorical features without embedding tables for recommendation.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, pages 840–850, 2021.
  * Kim et al. (2007) D. Kim, K.-s. Kim, K.-H. Park, J.-H. Lee, and K. M. Lee.  A music recommendation system with a dynamic k-means clustering algorithm.  In _Sixth international conference on machine learning and applications (ICMLA 2007)_ , pages 399–403. IEEE, 2007.
  * Koren et al. (2009) Y. Koren, R. Bell, and C. Volinsky.  Matrix factorization techniques for recommender systems.  _Computer_ , 42(8):30–37, 2009.
  * Lee et al. (2022) D. Lee, C. Kim, S. Kim, M. Cho, and W.-S. Han.  Autoregressive image generation using residual quantization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11523–11532, 2022.
  * Mikolov et al. (2013) T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean.  Distributed representations of words and phrases and their compositionality.  _Advances in neural information processing systems_ , 26, 2013.
  * Pancha et al. (2022) N. Pancha, A. Zhai, J. Leskovec, and C. Rosenberg.  Pinnerformer: Sequence modeling for user representation at pinterest.  _arXiv preprint arXiv:2205.04507_ , 2022.
  * Rajput et al. (2023) S. Rajput, N. Mehta, A. Singh, R. Keshavan, T. Vu, L. Heldt, L. Hong, Y. Tay, V. Q. Tran, J. Samost, and M. Sathiamoorthy.  Recommender systems with generative retrieval.  _arXiv preprint arXiv:2305.05065_ , 2023.
  * Schein et al. (2002) A. I. Schein, A. Popescul, L. H. Ungar, and D. M. Pennock.  Methods and metrics for cold-start recommendations.  In _Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval_ , pages 253–260, 2002.
  * Shi et al. (2019) S. Shi, M. Zhang, X. Yu, Y. Zhang, B. Hao, Y. Liu, and S. Ma.  Adaptive feature sampling for recommendation with missing content feature values.  In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_ , pages 1451–1460, 2019.
  * Van Den Oord et al. (2017) A. Van Den Oord, O. Vinyals, et al.  Neural discrete representation learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Volkovs et al. (2017a) M. Volkovs, G. Yu, and T. Poutanen.  Dropoutnet: Addressing cold start in recommender systems.  _Advances in neural information processing systems_ , 30, 2017a.
  * Volkovs et al. (2017b) M. Volkovs, G. W. Yu, and T. Poutanen.  Content-based neighbor models for cold start in recommender systems.  In _Proceedings of the Recommender Systems Challenge 2017_ , pages 1–6. 2017b.
  * Weinberger et al. (2009) K. Weinberger, A. Dasgupta, J. Langford, A. Smola, and J. Attenberg.  Feature hashing for large scale multitask learning.  In _Proceedings of the 26th annual international conference on machine learning_ , pages 1113–1120, 2009.
  * Ying et al. (2018) R. Ying, R. He, K. Chen, P. Eksombatchai, W. L. Hamilton, and J. Leskovec.  Graph convolutional neural networks for web-scale recommender systems.  In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. ACM, jul 2018.  doi: 10.1145/3219819.3219890.  URL https://doi.org/10.1145%2F3219819.3219890.
  * Yu et al. (2021) J. Yu, X. Li, J. Y. Koh, H. Zhang, R. Pang, J. Qin, A. Ku, Y. Xu, J. Baldridge, and Y. Wu.  Vector-quantized image modeling with improved vqgan.  _arXiv preprint arXiv:2110.04627_ , 2021.
  * Yu et al. (2022) J. Yu, Y. Xu, J. Y. Koh, T. Luong, G. Baid, Z. Wang, V. Vasudevan, A. Ku, Y. Yang, B. K. Ayan, et al.  Scaling autoregressive models for content-rich text-to-image generation.  _arXiv preprint arXiv:2206.10789_ , 2022.
  * Zeghidour et al. (2021) N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi.  Soundstream: An end-to-end neural audio codec.  _CoRR_ , abs/2107.03312, 2021.  URL https://arxiv.org/abs/2107.03312.
  * Zhang et al. (2020) C. Zhang, Y. Liu, Y. Xie, S. I. Ktena, A. Tejani, A. Gupta, P. K. Myana, D. Dilipkumar, S. Paul, I. Ihara, et al.  Model size reduction using frequency based double hashing for recommender systems.  In _Proceedings of the 14th ACM Conference on Recommender Systems_ , pages 521–526, 2020.
  * Zhao et al. (2019) Z. Zhao, L. Hong, L. Wei, J. Chen, A. Nath, S. Andrews, A. Kumthekar, M. Sathiamoorthy, X. Yi, and E. Chi.  Recommending what video to watch next: a multitask ranking system.  In _Proceedings of the 13th ACM Conference on Recommender Systems_ , pages 43–51, 2019.