# Scenario-Adaptive Fine-Grained Personalization Network: Tailoring User
Behavior Representation to the Scenario Context

Moyu Zhang  Lazada GroupBeijingBeijingChina<EMAIL_ADDRESS>,  Yongxiang
Tang  UnaffiliatedBeijingBeijingChina<EMAIL_ADDRESS>,  Jinxin Hu
Lazada GroupBeijingBeijingChina<EMAIL_ADDRESS>and  Yu Zhang  Lazada
GroupBeijingBeijingChina<EMAIL_ADDRESS>
(2018)

###### Abstract.

As e-commerce has evolved, modern large-scale commercial platforms now
accommodate various scenarios to cater to the diverse shopping preferences of
users. To conserve resources, current multi-scenario methods often utilize a
unified framework to deliver personalized recommendations across various
scenarios. Given the overlap of users and items in multiple scenarios on
commercial platforms, current methods typically employ shared bottom
representations, capturing similarities and differences between scenarios
through adaptive adjustments. However, existing methods often adjust
representations adaptively only after aggregating user behavior sequences.
This coarse-grained approach to re-weighting the entire user sequence hampers
the model’s ability to accurately model the user interest migration across
different scenarios. To enhance the model’s capacity to capture user interests
from historical behavior sequences in each scenario, we develop a ranking
framework named the Scenario-Adaptive Fine-Grained Personalization Network
(SFPNet), which designs a kind of fine-grained method for multi-scenario
personalized recommendations. Specifically, SFPNet comprises a series of
blocks named as _Scenario-Tailoring Block_ , stacked sequentially. Each block
initially deploys a parameter personalization unit to integrate scenario
information at a coarse-grained level by redefining fundamental features.
Subsequently, we consolidate scenario-adaptively adjusted feature
representations to serve as context information. By employing residual
connection, we incorporate this context into the representation of each
historical behavior, allowing for context-aware fine-grained customization of
the behavior representations at the scenario-level, which in turn supports
scenario-aware user interest modeling. Ultimately, the effectiveness of our
proposed method is strongly substantiated by extensive experiments and online
A/B testing.

Multi-Scenario Recommendation, Scenario-Specific Behavior Representation,
Recommender System

††copyright: acmcopyright††journalyear: 2018††doi:
10.1145/1122445.1122456††conference: Woodstock ’18: ACM Symposium on Neural
Gaze Detection; June 03–05, 2018; Woodstock, NY††booktitle: Proceedings of the
47th International ACM SIGIR Conference on Research and Development in
Information Retrieval (SIGIR), July 14–18, 2024, Washington D.C., USA††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06 Figure 1. An illustrative example of
multi-scenarios for the online e-commerce platform Lazada.

##  1\. Introduction

In recent years, with the rise of electronic shopping as a dominant trend, a
multitude of e-commerce platforms have emerged, including Amazon, Alibaba’s
Taobao, Lazada, TikTok, and others (back1, ; back2, ; back3, ; back4, ;
causalint, ; adl, ). To cater to the growing diversity of user shopping
preferences, contemporary commercial platforms must offer personalized
services tailored to a variety of shopping contexts. For instance, the online
e-commerce platform depicted in Figure 1 offers multiple item recommendation
scenarios. The ”Homepage” scenario primarily suggests a broad range of
products potentially appealing to the user, based on their past behavior,
whereas the ”My Cart” and ”My Orders” scenarios depend more on the user’s
historical purchase records to generate recommendations. Compared to the
”Homepage” scenario, user interests in the ”My Cart” and ”My Orders” scenarios
become more specific. Consequently, these multi-scenario platforms present a
challenge to existing recommendation models: how to devise personalized
recommendations that consider the distinct characteristics of each scenario?

Traditional methods typically employ scenario-specific data to train distinct
models for each scenario, enhancing scenario differentiation for online
services. However, this isolated model training approach is resource-
intensive, often inefficient, and fails to capture inter-scenario
correlations, leading to suboptimal performance in data-sparse scenarios. In
practice, most current e-commerce platforms experience significant user and
item overlap across various scenarios. Consequently, despite the differences
in user and item performance across scenarios, substantial correlations
persist. Leveraging the correlations across scenarios, we can incorporate data
from other scenarios into the training process, alleviating the challenge of
data sparsity in individual scenarios. To model these inter-scenario
correlations, mainstream multi-scenario approaches often employ a unified
recommendation modeling framework, broadly categorized into two types: 1)
Scenario-specific network structures, influenced by multi-task learning (MTL)
(mtl1, ; mtl2, ; mtl3, ). Each scenario is treated as a distinct task, with a
common network capturing inter-scenario correlations and separate networks
modeling the unique aspects of each scenario. 2) Scenario-adaptive parameter
network structures (pepnet, ; adasparse, ), which differ from MTL by
acknowledging feature space variations. These methods apply scenario data
directly to the core embedding and prediction layers, enabling dynamic
adjustments of the feature space and prediction strategy in response to
scenario shifts. Compared to MTL methods that require learning distinct
network structures for new scenario, the adaptive parameter approach simply
incorporates new scenario-related features, enhancing its flexibility to
recognize new scenarios.

However, current scenario-adaptive parameter structures face a common issue:
they generally transfer information between scenarios coarsely, treating the
user’s entire historical behavior sequence as a single feature for
adjustments, which fails to consider the migration of user interests with the
scenario changes. As depicted in Figure 2(a), the weight adjustment method
uniformly applies the same weight to each behavior in the user sequence,
rather than tailoring scenario-specific information to individual behaviors,
thereby impairing the model’s capacity to accurately track user interest
shifts across scenarios. In fact, modeling user interests is critical in
recommendation systems, particularly as users do not always reveal their
intentions in non-search scenarios, making it essential to accurately track
the migration of user interests for effective multi-scenario recommendations.
For instance, on the ”Homepage” scenario, where user interests tend to be
broad, considering each click behavior within their historical behaviors is
necessary to provide diverse recommendations. Conversely, in scenarios guided
by specific interests, users may exhibit more defined preferences,
necessitating a focus on corresponding behaviors within their sequence.
Therefore, to model user interests across varied scenarios more effectively, a
fine-grained analysis of the relationships between the user’s historical
behaviors and scenarios is essential. Although the SAR-Net model (sar, ) ever
attempted to incorporate scenario information into sequence modeling using a
target-attention mechanism (din, ), as shown in Figure 2(b), this approach of
weighted summation does little to alter the actual representation of the user
sequence. Its ability to represent the complex interplay between real-life
scenarios and user behaviors is markedly limited. Consequently, adaptively and
reasonably adjusting the representations of user historical behavior to match
various scenarios remains an urgent challenge in contemporary multi-scenario
modeling.

Figure 2. Examples of previous methods and ours. In (a),
$\boldsymbol{V}_{1}=w_{1}\boldsymbol{V}$ and
$\boldsymbol{V}_{2}=w_{2}\boldsymbol{V}$, where
$\boldsymbol{V}=f(\boldsymbol{b}_{1},...,\boldsymbol{b}_{L})$. While in (b),
$\boldsymbol{V}_{1}=f(w_{1}^{s_{1}}\boldsymbol{b}_{1},...,w_{L}^{s_{1}}\boldsymbol{b}_{L})$
and
$\boldsymbol{V}_{2}=f(w_{1}^{s_{2}}\boldsymbol{b}_{1},...,w_{L}^{s_{2}}\boldsymbol{b}_{L})$.
In our method,
$\boldsymbol{V}_{1}=f(\boldsymbol{b}_{1}^{s_{1}},...,\boldsymbol{b}_{L}^{s_{1}})$,
where $\boldsymbol{b}_{1}^{s_{1}}$ is obtained by
$(\boldsymbol{V},\boldsymbol{C},\boldsymbol{S}_{1},\boldsymbol{b}_{1})$.

To tackle the above challenge of multi-scenario modeling, we propose a new
framework, the Scenario-Adaptive Fine-Grained Personalization Network
(SFPNet), designed to finely capture user interest changes across different
scenarios and enhance recommendation accuracy in a multi-scenario context.
SFPNet features a deep network architecture built from a series of stacked
blocks named as _Scenario-Tailoring Block_. Each block consists of two
critical modules: the Scenario-Adaptive Module and the Residual-Tailoring
Module. The former module adjusts foundational features to serve as instance-
specific contextual information, facilitating scenario awareness, while the
latter module uses this contextual information to tailor scenario
representations for each feature.

Specifically, as identified in numerous studies (pepnet, ; maria, ), the
foundational feature representation vector is frequently deemed the most
critical bottleneck in recommendation models. Therefore, SFPNet utilizes a
_Scenario-Adaptive Module (SAM)_ that dynamically scales foundational
features. This is achieved by inputting scenario prior features into a
scenario-specific gate, followed by an element-wise multiplication with the
original feature representations. These scaled representations embody
scenario-aware contextual information, signifying the input instance’s
presence in the current scenario. Notably, in the SAM, calculations are
performed after compressing the sequence into a multi-dimensional vector,
which aims to enhance the perception of the sequence’s contextual information
for subsequent fine-grained behavioral modeling. Upon acquiring the instance
context information, we devised a _Residual-Tailoring Module (RTM)_ in
response to concerns that the vector-level re-weighting summation to model the
shifts in user interests as reflected by user sequences across different
scenarios, as utilized by SAR-Net (sar, ), may inadequately capture the
intricate relationship between the scenario and user interests. To facilitate
scenario-level customization for each behavior in the sequence, we employ
residual connections (res, ) that depart from traditional feature-scenario
interactions. Instead, RTM integrates the contextual information of the input
instance and leverage a neural network to dynamically generate high-order
interactions, subsequently tailoring the behavior representation of the
sequence via residual computations. The integration of the above two modules
allows SFPNet to incorporate scenario-related information into the feature
representation vectors with high granularity, markedly enhancing the model’s
proficiency in discerning commonalities and distinctions across scenarios.

The contributions of our paper can be summarized as follows:

  * •

To our knowledge, SFPNet is the first multi-scenario work to offer fine-
grained scenario-specific customization for individual user behaviors, thereby
advancing the model’s capacity to track shifts in user interests across
various scenarios.

  * •

SFPNet features a novel block comprising two key modules, which capitalizes on
the context and scenario-specific information of each input instance to tailor
the representation vectors of user behaviors in the sequence and associated
context features of the input instance. By stacking the block, SFPNet
constructs a deep network that progressively bolsters its ability to model the
intricate interplay between scenarios and input instances.

  * •

Evaluations using offline datasets and online A/B testing have demonstrated
the superiority of the proposed SFPNet method over existing state-of-the-art
multi-scenario approaches.

##  2\. Related Work

With the evolution of e-commerce platforms and growing user engagement, a
single scenario is insufficient to satisfy the expanding needs of users (m5, ;
samd, ; hc2, ). Consequently, modern online e-commerce platforms are required
to cater to multiple scenarios simultaneously. Initially, recommendation
systems typically trained a distinct model for each scenario to cater to its
specific demands. However, with the proliferation of platform scenarios, the
drawbacks of this independent training approach have become increasingly
evident. This method is not only resource-intensive but also fails to leverage
the correlations between scenarios, leading to suboptimal model performance.
Recent advances in deep learning and the industry’s emphasis on
replicability—avoiding the integration of new elements with each additional
scenario—have led to a preference for a unified model structure, employing a
singular ranking framework for scenario-specific recommendations. Single model
structures are primarily categorized into two approaches: 1) Scenario-specific
network structures, and 2) Scenario-adaptive parameter network structures.
Though both strive to train a unified model capable of serving multiple
scenarios, their underlying philosophies differ.

Specifically, scenario-specific network structures draw inspiration from
multi-task learning (MTL) principles. These approaches seek to harness the
similarities between multi-task and multi-scenario applications by considering
each scenario as an independent task. Mixture-of-Experts (MoE) (moe, ) is
proposed to significantly increase model capacity and capability and is widely
used in the multi-scenario area. MMoE (mtl3, ) characterizes the task
correlation and learns the function of specific tasks based on shared bottom
feature representation. HMoE (hmoe, ) takes advantage of MMoE to implicitly
identify distinctions and commonalities between tasks, and improves the
performance with a stacked model learning task relationships in the label
space explicitly. Later, PLE (ple, ) shares experts in the share layer and
refines tasks uniquely to effectively alleviate the noise caused by other
scenarios and improves the effectiveness of feature extraction. M2M (m2m, )
utilizes expert networks to solve multi-scenario and multi-task problems based
on a meta network to express the scenario information explicitly. AESM2
(aesm2, ) proposes a novel expert network structure with automatic selection
of fine granularity by calculating the KL divergence to select the most
suitable sharing and exclusive experts. Beyond the MoE-based method, there are
alternative approaches like STAR (star, ), which introduces an extra tower for
each scenario, combining the parameters of scenario-specific tower with those
of the shared tower’s parameter.

However, scenario-specific approaches learn scenario information transfer in
quite implicit ways and often overlook differences in underlying
representations, which frequently constitute a bottleneck in recommendation
models. Therefore, SASS (sass, ) designs a scenario adaptive transfer module
to select and fuse effective transfer information from whole scenario to
individual scenario. Adasparse (adasparse, ) learns adaptively sparse
structure for each scenario, achieving better generalization across domains
with lower computational cost. HiNet (hinet, ) achieves hierarchical
extraction based on coarse-to-fine knowledge transfer scheme. DFFM (dffm, )
incorporates scenario-related information into the parameters of feature
interaction and user behavior modules, allowing scenario-specific learning of
these two aspects, which is essentially a scenario-adaptive parameterization
method. 3MN (3mn, ) proposes a novel three meta networks-based solution to
model the complicated task-task, scenario-scenario, and task-scenario
interrelations. PEPNet (pepnet, ) takes features with strong biases as input
and dynamically scales the bottom-layer embeddings and the top-layer DNN
hidden units in the model through a gate mechanism. MARIA (maria, ) proposes
to project the scenario semantic information into the bottom feature
representation to derive more discriminative feature representations.

##  3\. Problem Formulation

Given a set of scenarios $\mathcal{S}=\left\\{s_{m}\right\\}_{m=1}^{N_{s}}$
with a shared feature space $\mathcal{X}$ and label space $\mathcal{Y}$, the
multi-scenario recommendation task aims to devise a unified ranking formula
$\mathcal{F}:\mathcal{X}\rightarrow\mathcal{Y}$, to concurrently provide
accurate, personalized recommendations across $M$ scenarios. The common
feature space typically encompasses a variety of features that can be
categorized into distinct fields, such as user-centric and item-centric
features, which collectively represent an instance’s comprehensive contextual
information. Within this paper, the feature space $\mathcal{X}$ is constructed
as $[x_{1},x_{2},...,x_{N_{f}},\left\\{b_{i}\right\\}_{i=0}^{N_{b}}]$, where
for clarity in subsequent discussions, $\left\\{b_{i}\right\\}_{i=0}^{N_{b}}$
denote the user’s historical behavior sequence to distinguish it from other
$N_{f}$ features, and $N_{b}$ represents the number of the user’s historical
behaviors. Mathematically, the multi-scenario recommendation task involves
estimating the probability that the target user will interact with the target
item in a given scenario $s_{m}$, utilizing the contextual features, as
illustrated below:

(1)
$\displaystyle\hat{y}=P(y=1|\mathcal{X},s_{m})=\mathcal{F}(\boldsymbol{x}_{1},\boldsymbol{x}_{2},...,\boldsymbol{x}_{N_{f}},\left\\{\boldsymbol{b}_{i}\right\\}_{i=0}^{N_{b}},\boldsymbol{s}_{m})$

where $\boldsymbol{x}_{i}\in\mathbb{R}^{d}$ denotes the embedding of the
$i$-th feature, and $d$ denotes the embedding dimension of features.
$\boldsymbol{b}_{i}\in\mathbb{R}^{d}$ denotes the representation of $i$-th
behavior in the behavior sequence, typically derived from a pooled embedding
that combines the clicked item embedding and the embeddings of its associated
attribute features. $\boldsymbol{s}_{m}$ represents the representation vector
of $m$-th scenario features.

Figure 3. The network architecture of our proposed Scenario-Adaptive Fine-
Grained Personalization Network (SFPNet).

##  4\. Method

As previously discussed, contemporary multi-scenario models typically employ a
unified framework to deliver personalized recommendations across various
scenarios. While previous scene-adaptive parameter networks are lauded for
their flexibility in adjusting the model’s foundational representation to
excel in multi-scenario recommendations, they typically consider the entire
user historical behavior sequence as a singular feature field and conduct
scenario-based transformations within this sequence. This approach uniformly
adjusts all behaviors with the same weight, thus overlooking the distinct
semantic information that each behavior may signify in different scenarios. In
reality, users’ interests frequently shift with the scenario transition,
causing each behavior in the sequence to represent varying information across
different scenarios. Inadequate cross-scenario modeling precludes precise
extraction of user preferences, reflected in individual behaviors, for each
scenario, thereby compromising the accuracy of cross-scenario recommendations.
To more effectively model the intricate interplay between user historical
behaviors and scenarios, this paper introduces a novel framework named
Scenario-Adaptive Fine-Grained Personalization Network (SFPNet). It is
primarily built from stacked _Scenario-Tailoring Block_ and comprises two main
modules:
$\bullet$ Scenario-Adaptive Module (SAM), which employs a gating mechanism to
integrate scenario-related information into the base representations of
features, ensuring that the instance’s contextual information is strongly
guided by the scenario context.
$\bullet$ Residual-Tailoring Module (RTM), which aggregates scenario-aware
contextual information and, via residual connections, disaggregates it into
distinct representations for each behavior, allowing each behavior to capture
contextual-cross information.

###  4.1. Scenario Adaptive Module (SAM)

Drawing inspiration from the LHUC algorithm (lhuc, ), which focuses on
learning the distinct contributions of each speaker’s hidden units, a variety
of scenario-adaptive parameterization methods have been developed. These
methods involve integrating different scenario prior bias features to create
personalized network parameters for each instance, thereby enabling
distinction of features at the foundational level across different scenarios.
Consequently, within each Scenario-Tailoring Block, we also incorporate a
Scenario-Adaptive Module (SAM) that coarsely applies scenario prior
information to diverse feature representations. This ensures that while
serving as contextual information for instances, the distinctions between
scenarios are consistently maintained. Specifically, the input to the SAM in
the first _Scenario-Tailoring Block_ is the original embedding vector of the
features, and subsequent SAM inputs are typically the outputs from the
_Residual-Tailoring Module_ of the previous block. As the purpose of SAM is to
produce the contextual information for the instance, we compress the user
behavior sequence into a distinct feature field, enabling complete perception
of the sequence’s information during subsequent behavior customization. SAM
operates through a two-step process: sequence pooling, and gate
personalization. To elucidate, let’s examine the computation within SAM of the
$l$-th block, delineated as follows:

####  4.1.1. Sequence Pooling

Since that SAM’s output serves solely as contextual information for the
instance, our initial step is to pool the sequence. To avoid substantial
increases in computational complexity from stacking blocks, we opt for
computationally efficient aggregation methods—like summation, max pooling, or
average pooling—for user sequences. This contrasts with the more complex and
computationally intensive neural network structures used in DIN (din, ) to
capture sequence information related to target items. Nonetheless, the above
simple pooling methods result in a measure of sequence information loss,
meaning we are unable to accurately model the true distribution of sequences
from the pooled sequence representations alone. For instance, considering the
user sequence distribution adhering to a Gaussian distribution, we need to
know both the mean and the variance to reconstruct the true sequence
distribution, expressed as $b_{i}\verb|~|N(\mu,\sigma^{2})$. Consequently, we
refined the sequence pooling approach in this paper by developing a
_Distribution-Aware Pooling_ method, which compresses the sequence while
maximally preserving sequence information through the computation of both mean
and variance, with minimal additional computational load. The precise
computational procedure for pooling is delineated as follows:

(2)
$\displaystyle\boldsymbol{B}^{l}=\left(\frac{1}{N_{b}}\sum_{i=1}^{N_{b}}\boldsymbol{b}_{i}^{l-1},\sqrt{\frac{1}{N_{b}}\sum_{i=1}^{N_{b}}(\boldsymbol{b}_{i}^{l-1})^{2}-(\frac{1}{N_{b}}\sum_{i=1}^{N_{b}}\boldsymbol{b}_{i}^{l-1})^{2}}\right)$

where $\boldsymbol{B}^{l}\in\mathbb{R}^{2d_{l-1}}$ denotes the pooled
representation of user behavior sequence from the $(l-1)$-th block.
$\boldsymbol{b}_{i}^{l-1}\in\mathbb{R}^{d_{l-1}}$ is the updated
representation of $i$-th behavior output by the $(l-1)$-th block, and
$d_{l-1}$ is the dimension of features in the $(l-1)$-th block.

####  4.1.2. Gate Personalization

Given the scenario information for the $m$-th scenario, we have devised a
gating function that applies scenario-specific adjustments at the instance
level, tailored to each instance’s features, to aid in the encoding of
scenario semantics within instance feature representations. Specifically, we
concatenate all features—encompassing instance-specific, compressed sequence,
and scenario features as
$\boldsymbol{X}^{l}=(\boldsymbol{x}_{1}^{l-1}\oplus\boldsymbol{x}_{2}^{l-1}\oplus\cdots\oplus\boldsymbol{x}_{n}^{l-1}\oplus\boldsymbol{B}^{l}\oplus\boldsymbol{s}_{m})\in\mathbb{R}^{(n+2)d_{l-1}+d}$,
where $\boldsymbol{x}_{i}^{l-1}$ denotes the representation of $i$-th feature
from the $(l-1)$-th block—and employ two neural network layers to output a
vector matching the dimensionality of the input instance features. The weight
calculation for each element can be detailed as follows:

(3)
$\displaystyle\boldsymbol{A}^{l}=\sigma(\boldsymbol{W}_{1}^{l}(LN(ReLU(\boldsymbol{W}_{0}^{l}\boldsymbol{X}^{l}+\boldsymbol{b}_{0}^{l})))+\boldsymbol{b}_{1}^{l})$

where $\boldsymbol{A}^{l}\in\mathbb{R}^{(n+2)d_{l-1}}$ represents the weight
of the element corresponding to the feature, ranging from $0$ to $1$.
$\sigma(\cdot)$ denotes the sigmoid function. $LN(\cdot)$ denotes the layer-
norm calculation, which aims to help the model learn a more differentiated
weight distribution. $\boldsymbol{W}_{0}^{l}$, $\boldsymbol{W}_{1}^{l}$ and
$\boldsymbol{b}_{0}^{l}$, $\boldsymbol{b}_{1}^{l}$ are the weight and bias for
the gate network in the $l$-th block, respectively.

Subsequently, we apply a re-weighting operation to the input representation of
the $l$-th module within SAM, involving element-wise multiplication between
the original representation and the calculated weight values, resulting in a
new representation vector. Given that the gate’s input encompasses scenario
information, we can ensure that the re-weighted embeddings at each block layer
consistently reflect scenario-specific distinctions. The specific
computational formula is defined as follows:

(4)
$\displaystyle\hat{\boldsymbol{X}}^{l}=(\boldsymbol{x}_{1}^{l-1}\oplus\boldsymbol{x}_{2}^{l-1}\oplus...\oplus\boldsymbol{x}_{n}^{l-1}\oplus\boldsymbol{B}^{l})\otimes\boldsymbol{A}^{l}$

In this way, $\hat{\boldsymbol{X}}^{l}\in\mathbb{R}^{(n+2)d_{l-1}}$ as an
effective scenario signal will be further amplified in the following Residual-
Tailoring Module, allowing the model to better capture scenario differences.

###  4.2. Residual-Tailoring Module (RTM)

As previously noted, previous multi-scenario modeling methods often
differentiate features coarsely, thereby overlooking the intricate interplay
of individual user behaviors with different scenarios. This limitation stymies
the model’s capacity to precisely track the evolution of user interests across
scenarios, ultimately impairing the tailored recommendation of items that
align with users’ genuine interests in a cross-scenario context. To address
the above issue, this paper introduces a Residual-Tailoring Module (RTM) that
accepts the output of the Scenario-Adaptive Module (SAM). It leverages a
neural network to facilitate interaction with context-aware features,
producing a distinctive context representation vector for each feature.
Additionally, residual connections (res, ) are employed to integrate the
contextual interaction vector into the representation vector of each user
behavior, enabling fine-tuned scenario and context-aware customization for
every behavior representation. RTM encodes the global semantics of user
behavior within specific scenarios, thereby boosting the model’s proficiency
in detecting nuanced variations of user behavior across scenarios. It should
be noted that while our module focuses on customizing user behavior, the same
customization processes are applied to other features to augment their
capacity to depict scene divergences. Specifically, the RTM comprises two
principal steps: the aggregation interaction step and the context-aware
tailoring step. The computation within $l$-th block of the RTM is delineated
as follows:

####  4.2.1. Aggregation Interaction Step

Prior research (cross, ) on feature interactions has demonstrated that deep
neural networks adeptly capture implicit high-order feature interactions,
frequently resulting in enhanced model prediction performance. Consequently,
we integrate all scenario-aware contextual information from the input instance
and utilize a neural network to dynamically produce high-order global feature
interaction signals for each feature within the instance. This process unfolds
as follows:

(5)
$\displaystyle\boldsymbol{C}^{l}=\boldsymbol{W}_{3}^{l}(ReLU(\boldsymbol{W}_{2}^{l}\hat{\boldsymbol{X}}^{l}+\boldsymbol{b}_{2}^{l}))+\boldsymbol{b}_{3}^{l}$

where $\boldsymbol{C}^{l}\in\mathbb{R}^{(n+2)d_{l}}$ denotes the contextual
interaction information of the input features and include scenario information
due to the input representations that have been modified to be adaptive to the
scenario. $d_{l}$ is the dimension of the feature in $l$-th block.
$\boldsymbol{W}_{2}^{l}$ and $\boldsymbol{W}_{3}^{l}$ are the weights of
contextual information aggregation function, and $\boldsymbol{b}_{2}^{l}$ and
$\boldsymbol{b}_{3}^{l}$ are the bias of function.

####  4.2.2. Context-Aware Tailor Step

Drawing inspiration from residual architectures (res, ), we propose that once
contextual interaction information is acquired via the aggregation module,
residual connections are employed to merge the original feature representation
with its corresponding global interaction representation, thus enabling
recoding into a globally-aware representation. We begin by detailing the
calculation of residual connections for all features, excluding sequences. The
calculation process is as follows:

(6)
$\displaystyle\boldsymbol{x}_{i}^{l}=ReLU(\boldsymbol{W}_{x_{i},1}^{l}\boldsymbol{x}_{i}^{l-1}+\boldsymbol{b}_{x_{i},1}^{l})+\sigma(\boldsymbol{W}_{x_{i},2}^{l}\boldsymbol{x}_{i}^{l-1}+\boldsymbol{b}_{x_{i},2}^{l})\otimes\boldsymbol{c}_{i}^{l}$

where $\boldsymbol{x}_{i}^{l}\in\mathbb{R}^{d_{l}}$ denotes the context-aware
scenario-adaptive representation of the $i$-th feature generated by the $l$-th
block. $\boldsymbol{W}_{x_{i},1}^{l}$ and $\boldsymbol{W}_{x_{i},2}^{l}$
denote the weight matrix for the $i$-th feature.
$\boldsymbol{x}_{i}^{l-1}=\boldsymbol{X}^{l-1}[i*d_{l-1}:(i+1)*d_{l-1}]$
denotes the scenario-adaptive representation of the $i$-th feature output by
SAM. $\boldsymbol{c}_{i}^{l}=\boldsymbol{C}^{l}[i*d_{l}:(i+1)*d_{l}]$ denotes
the context-aware cross information for the $i$-th feature.

Fine-Grained Cross-Scenario Sequence Modeling. As emphasized throughout this
paper, our goal with sequence features is to recode specific behavioral
representations with fine granularity. The residual architecture is equally
adaptable for fine-grained coding of sequence features. The specific
computation involved in this fine-grained sequence coding is described as
follows:

(7)
$\displaystyle\boldsymbol{v}_{i}^{l}=ReLU(\boldsymbol{W}_{4}^{l}\boldsymbol{v}_{i}^{l-1}+\boldsymbol{b}_{4}^{l})+\sigma(\boldsymbol{W}_{5}^{l}\boldsymbol{v}_{i}^{l-1}+\boldsymbol{b}_{5}^{l})\otimes\boldsymbol{c}_{b}^{l}$

where $\boldsymbol{v}_{i}^{l}\in\mathbb{R}^{d_{l}}$ denotes the encoded
representation of the $i$-th behavior of user sequence with awareness of
context information and scenario information.
$\boldsymbol{c}_{b}^{l}=\boldsymbol{C}^{l}[n*d_{l}:(n+2)*d_{l}]$ denotes the
final context-aware scenario-adaptive representation of the sequence feature
generated by the $l$-th block. After re-encoding the sequence, each behavioral
representation is finely tuned in accordance with the scenario. Aggregating
the sequence at this juncture can substantially enhance the model’s
proficiency in extracting users’ interests across various scenarios, thereby
improving the model’s adeptness at recognizing different situations. This, in
turn, increases the accuracy of scenario-specific recommendations.

Scenarios | #A1 | #A2 | #A3 | #A4 | #A5 | #A6 | #A7 | #A8 | #A9 | #A10 | #A11 | #A12
---|---|---|---|---|---|---|---|---|---|---|---|---
#User | 9,233,770 | 3,375,959 | 3,275,513 | 232,018 | 3,613,010 | 4,066,414 | 4,474,204 | 2,968,181 | 3,064,679 | 783,071 | 1,352,973 | 157,645
#Item | 1,309,072 | 1,150,596 | 1,138,230 | 1,038,012 | 689,511 | 1,067,403 | 984,055 | 893,954 | 944,691 | 889,481 | 890,769 | 726,907
#Click | 14,366,092 | 3,817,960 | 3,726,107 | 4,788,877 | 5,156,626 | 2,279,444 | 1,498,577 | 1,662,265 | 774,141 | 1,428,080 | 1,869,587 | 490,689
#Impression | 613M | 184M | 166M | 156M | 135M | 115M | 90M | 64M | 48M | 46M | 44M | 29M
Table 1. Statistics of the industrial dataset. Scenarios | #B1 | #B2 | #B3 | #All
---|---|---|---|---
#User | 91,488 | 2,612 | 154,024 | 244,397
#Item | 535,711 | 198,651 | 537,937 | 538,376
#Click | 1,291,063 | 28,022 | 1,998,618 | 3,317,703
#Impression | 32,236,951 | 639,897 | 52,439,671 | 85,316,519
Table 2. Statistics of the Ali-CCP dataset.

###  4.3. Scenario-Specific Prediction

In the prediction layer, the sequence must first be compressed to facilitate
its integration with other feature vectors for subsequent processing by the
deep network. Given the article’s focus on sequence representation recoding,
we employ commonly used structures for the compression vector, such as the
target-attention mechanism of Deep Interest Network (DIN) (din, ):

(8)
$\displaystyle\boldsymbol{V}=DIN(\boldsymbol{v}_{1}^{L},\boldsymbol{v}_{2}^{L},...,\boldsymbol{v}_{N_{b}}^{L})$

where $L$ is the total number of block, and $\boldsymbol{v}_{i}^{L}$ is the
representation of $i$-th behavior outputted by the last block. Subsequently,
we leverage the benefits of a multi-tower architecture by incorporating
scenario-aware DNN towers at the prediction stage. These towers process
sequence features and other features in parallel by considering the scenario
bias each layer, to forecast the likelihood of users clicking on the target
item within the specific scenario $s_{m}$:

(9)
$\displaystyle\hat{y}=P(y=1|\mathcal{X},s_{m})=\sigma(\mbox{\emph{S-DNN}}(\boldsymbol{H}^{1},\boldsymbol{s}_{m}))$

where
$\boldsymbol{H}^{1}=(\boldsymbol{x}_{1}^{L}\oplus\boldsymbol{x}_{2}^{L}\oplus\cdots\oplus\boldsymbol{x}_{n}^{L}\oplus\boldsymbol{V})$
denotes the input of the first layer of the Scenario-Aware DNN (S-DNN). The
specific calculation of the $j$-th layer in the S-DNN structure is similar to
the calculation in the SAM, and can be depicted as follows:

(10)
$\displaystyle\boldsymbol{A}^{j}_{p}=\sigma(\boldsymbol{W}_{7}^{j}(LN(ReLU(\boldsymbol{W}_{6}^{j}(\boldsymbol{H}^{j-1}\oplus\boldsymbol{s}_{m})+\boldsymbol{b}_{6}^{j})))+\boldsymbol{b}_{7}^{j})$
(11)
$\displaystyle\boldsymbol{H}^{j}=ReLU(\boldsymbol{W}_{8}^{j}(\boldsymbol{H}^{j-1}\otimes\boldsymbol{A}^{j}_{p})+\boldsymbol{b}_{8}^{j})$

where $\boldsymbol{H}^{j}$ denotes the output of the $j$-th layer of S-DNN.
$\boldsymbol{W}_{6}^{j}$, $\boldsymbol{W}_{7}^{j}$ and
$\boldsymbol{W}_{8}^{j}$ are the weight for the S-DNN in the $j$-th layer.
$\boldsymbol{b}_{6}^{j}$, $\boldsymbol{b}_{7}^{j}$ and
$\boldsymbol{b}_{8}^{j}$ are the bias for S-DNN in the $j$-th layer.

###  4.4. Model Learning

The objective function applied in our model is the cross entropy loss
function, defined as:

(12)
$\displaystyle\mathcal{L}=-\frac{1}{N}\sum_{i=1}^{N}(y_{i}log(\hat{y}_{i})+(1-y_{i})log(1-\hat{y}_{i}))$

where $y_{i}\in\left\\{0,1\right\\}$ is the ground truth of instance.

##  5\. Experiments

In this section, we conduct extensive experiments on public bench- mark
datasets to validate the effectiveness of our proposed framework and answer
the following questions:
$\bullet$ RQ1: How does SFPNet perform compared with the state-of-the-art
baseline methods?
$\bullet$ RQ2: How does the performance of the SFPNet model, trained on
specific scenario data for each scene, compare to that of the SFPNet model
trained on a composite scenario dataset?
$\bullet$ RQ3: How about the impact of each part on the overall model?
$\bullet$ RQ4: How about the impact of hyper-parameters of our model?

###  5.1. Datasets

To validate the efficacy of our proposed method, two real-world large-scale
datasets covering diverse scenarios are used for performance evaluation. We
conduct experiments on both public available dataset and industrial dataset.
The descriptions and statistics of two dataset are detailed in Table 1 and 2,
respectively.
$\bullet$ Industrial Dataset. To assess our multi-scenario recommendation
approach in a real-world setting, we gathered a dataset from the Alibaba
international advertising platform, Lazada, that spans multiple scenarios from
December 10, 2023, to January 10, 2024. This dataset encompasses 12 scenarios,
denoted as #A1 through #A12. We allocated the user instances from the final
day for testing and the preceding instances for training.
$\bullet$ Ali-CCP 111https://tianchi.aliyun.com/dataset/408. It is widely used
in the relevant literature for multi-scenario recommendations (maria, ; plate,
; sass, ), which was collected Taobao’s recommender system under three
scenarios. It’s released by Taobao with prepared training and testing set, and
we can split the dataset into 3 scenarios according to the _scenario id_ ,
denoted as #B1, #B2, and #B3 for simplicity.

###  5.2. Competitors

We conduct experiments with several compared methods for the multi-scenario
recommendation task. These methods can be divided into three groups as
follows:

####  5.2.1. General Recommenders

Samples from all scenarios are utilized to co-train a universal recommender
system capable of multi-domain recommendations.
$\bullet$ BaseDNN. It is a multi-scenario model that shares the parameters of
the bottom layer. On top of the shared bottom layer, a single DNN is used for
prediction across scenarios.
$\bullet$ DeepFM (deepfm, ). The model resembles BaseDNN; however, it
substitutes the DNN architecture with DeepFM, which is a neural network
framework grounded in factorization machines.
$\bullet$ SharedBottom. It replaces the single DNN with multiple DNNs. That
is, an individual DNN is utilized for each scenario. And we add an auxiliary
tower to enhance the ability to characterize the scenario indicator.

####  5.2.2. Scenario-specific network structures

Each scenario is treated as a distinct task and inter-scenario correlations
are examined through dedicated networks for each scenario.
$\bullet$ MMoE (mtl3, ). It implicitly models task relationships for multi-
task learn- ing, where different tasks may have different label spaces. Here
we adapt MMoE for multi-scenario learning. The number of experts is equal to
the number of experts of Maria. The sum of weighted outputs from the experts
are fed into the individual tower for each scenario respectively.
$\bullet$ PLE (ple, ). It is a state-of-the-art multi-scenario/multi-task
model that organizes the experts into scenario-specific groups and scenario-
shared groups for the purpose of avoiding negative transfer or seesaw
phenomenon.
$\bullet$ STAR (star, ). It proposes a star topology to accommodate with the
scenario-specific characteristics. Specifically, a shared network works as the
center node for knowledge sharing and each scenario network connects only with
the center node.
$\bullet$ AESM2 (aesm2, ). It proposes a novel MMoE-based model with automatic
search towards the optimal network structure. In contrast to PLE and STAR, an
expert can be either scenario-shared or scenario- specific dynamically in an
instance-aware manner.

####  5.2.3. Scenario-adaptive parameter network structures

They disregard feature space differences and applying scenario information
directly to the core embedding layer and prediction module, thus allowing the
feature space and prediction strategies to dynamically adjust to scenario
variations.
$\bullet$ AdaSparse (adasparse, ). It learns adaptively sparse structures for
multi-scenario prediction and prunes redundant neurons via learned neuron-
level weighting factors to improve generalization.
$\bullet$ PEPNet (pepnet, ). It takes features with strong biases as input and
dynamically scales the bottom-layer embeddings and the top-layer DNN hidden
units in the model through a gate mechanism.
$\bullet$ MARIA (maria, ). It designs three components to enable
discriminative feature learning in a scenario-aware manner: feature scaling,
feature refinement, and feature correlation modeling.

Table 3. Prediction performance on the industrial dataset. * indicates p-value ¡ 0.05 in the significance test. Method | The industrial dataset (S-GAUC)
---|---
#A1 | #A2 | #A3 | #A4 | #A5 | #A6 | #A7 | #A8 | #A9 | #A10 | #A11 | #A12
BaseDNN | 0.6396 | 0.5501 | 0.6313 | 0.6322 | 0.6514 | 0.6176 | 0.6494 | 0.6571 | 0.6471 | 0.6390 | 0.6212 | 0.6298
DeepFM | 0.6398 | 0.5517 | 0.6309 | 0.6318 | 0.6523 | 0.6181 | 0.6499 | 0.6573 | 0.6468 | 0.6388 | 0.6208 | 0.6293
SharedBottom | 0.6352 | 0.5496 | 0.6313 | 0.6294 | 0.6486 | 0.6115 | 0.6437 | 0.6526 | 0.6415 | 0.6339 | 0.6192 | 0.6273
MMoE | 0.6424 | 0.5508 | 0.6321 | 0.6356 | 0.6558 | 0.6157 | 0.6535 | 0.6592 | 0.6499 | 0.6425 | 0.6234 | 0.6308
PLE | 0.6456 | 0.5529 | 0.6348 | 0.6353 | 0.6581 | 0.6175 | 0.6551 | 0.6602 | 0.6513 | 0.6457 | 0.6268 | 0.6317
STAR | 0.6412 | 0.5512 | 0.6304 | 0.6349 | 0.6536 | 0.6159 | 0.6529 | 0.6579 | 0.6481 | 0.6413 | 0.6217 | 0.6279
AESM2 | 0.6448 | 0.5524 | 0.6343 | 0.6341 | 0.6567 | 0.6168 | 0.6538 | 0.6579 | 0.6501 | 0.6423 | 0.6251 | 0.6301
AdaSparse | 0.6450 | 0.5531 | 0.6374 | 0.6362 | 0.6582 | 0.6201 | 0.6573 | 0.6605 | 0.6505 | 0.6451 | 0.6282 | 0.6313
PEPNet | 0.6463 | 0.5563 | 0.6402 | 0.6385 | 0.6595 | 0.6235 | 0.6612 | 0.6636 | 0.6527 | 0.6492 | 0.6306 | 0.6350
MARIA | 0.6472 | 0.5561 | 0.6418 | 0.6392 | 0.6601 | 0.6242 | 0.6615 | 0.6647 | 0.6531 | 0.6495 | 0.6312 | 0.6358
SFPNet |  0.6524* |  0.5612* |  0.6489* |  0.6447* |  0.6658* |  0.6309* |  0.6689* |  0.6714* |  0.6597* |  0.6548* |  0.6375* |  0.6432*
Table 4. Prediction performance on the Ali-CCP dataset. * indicates p-value ¡ 0.05 in the significance test. Method | Ali-CCP dataset (AUC)
---|---
#B1 | #B2 | #B3
BaseDNN | 0.5723 | 0.5887 | 0.5869
DeepFM | 0.5741 | 0.5903 | 0.5878
SharedBottom | 0.5685 | 0.5881 | 0.5817
MMoE | 0.5764 | 0.5916 | 0.5884
PLE | 0.5786 | 0.5828 | 0.5893
STAR | 0.5672 | 0.5869 | 0.5803
AESM2 | 0.5798 | 0.5963 | 0.5927
AdaSparse | 0.5808 | 0.5986 | 0.6024
PEPNet | 0.5839 | 0.6101 | 0.6052
MARIA | 0.5864 | 0.6108 | 0.6068
SFPNet |  0.5917* |  0.6153* |  0.6105*

Implementation Details. In offline experiments, we implement all the models
based on the TensorFlow framework (tensorflow, ). We use Adam (adam, ) for
optimization with an initial learning rate of 0.001 and a decay rate of 0.9.
The batch size is set as 512 and the embedding size is fixed to 40 for all
models. Xavier initialization (xavier, ) is used here to initialize the
parameters. All methods use a three-layer feedforward neural network with
hidden sizes of [256, 128, 64] for instance prediction. We apply careful grid-
search to find the best hyper-parameters. The number of experts in MMoE, PLE,
SharedBottom and its variants is searched in [2, 4, 6, 8]. All regularization
coefficients are searched in [1e-7, 1e-5, 1e-3].

Evaluation Metric. In line with prior studies, this paper employs the area
under the ROC curve (AUC) as the performance metric for the public Ali-CCP
dataset. However, for our industrial dataset, we adopt a variant of session-
weighted AUC. This metric assesses the quality of item rankings within
sessions by averaging the AUCs of a user’s individual session behaviors. This
variant, already implemented on our platform, has proven to align more closely
with the system’s online performance. For brevity, we refer to this metric as
S-GAUC, and its computation is as follows:

(13)
$\displaystyle\mbox{\emph{S-GAUC}}=\frac{\sum_{i=1}^{n}\\#impression_{i}\times
AUC_{i}}{\sum_{i=1}^{n}\\#impression_{i}}$

where $n$ is the number of sessions in the dataset. $\\#impression$ and
$AUC_{i}$ are the number of impressions and AUC corresponding to the $i$-th
session. A higher S-GAUC score usually denotes a better online recommendation
performance. We conduct a Mann-Whitney U test (m-test, ) under AUC and S-GAUC
metrics.

###  5.3. Comparison with Baselines (RQ1)

Table 3 and Table 4 displays the overall prediction performance of all methods
on the industrial and public dataset, respectively, along with the statistical
significance of our model against the best baseline model, with the highest
results highlighted in bold. From the results in two tables, we can see that
SFPNet outperforms all baselines for both datasets, indicating that the
Scenario-Tailoring Block can lead to multi-scenario prediction performance
improvements.

As for _General Recommenders_ , we can find that they often struggle to attain
satisfactory performance compared to other baseline models on the both
datasets. SharedBottom incorporates dedicated DNNs for each scenario to embed
scene-specific biases into the model. However, empirical findings indicate
that this straightforward approach of directly incorporating biases can
frequently yield detrimental outcomes. This outcome corroborates our
hypothesis that bottom-level sharing with a single top structure is inadequate
for capturing the complex interplay between scenarios. Consequently, with
datasets that exhibit significant scene variability, the performance of
_General Recommenders_ tends to be subpar.

To address the shortcomings of _General Recommenders_ , _Scenario-Specific
Network Structure Methods_ were proposed. According to experimental outcomes,
these methods have resulted in notable performance enhancements. By enhancing
the model’s capacity to distill shared knowledge across scenarios, both the
MMoE and STAR models outperformed the previously mentioned three baseline
models. However, similar to the seesaw effect observed in multi-task learning,
MMoE and PLE also exhibit a seesaw effect across multiple scenarios. This
indicates that these structures alone are insufficient for efficiently
handling scenarios with uneven data distribution, and achieving simultaneous
performance improvements across all scenarios. To mitigate the seesaw effect,
PLE, a variant of MMoE, introduces greater stability across scenarios by
segmenting the expert network into two distinct groups. Notably, PLE
demonstrates significant improvement over MMoE on the Ali-CCP dataset.
Additionally, AESM2 performs well on the Ali-CCP dataset, yet it underperforms
on the industrial dataset. We attribute this to the extensive variety of
industrial scenarios. AESM2 mechanism for automatically selecting expert
networks faces challenges in training with such a diversity of scenes. This
suggests that in more complex scenarios, the comparatively simpler strategy
employed by PLE may more readily achieve optimal performance.

Despite the focus on optimization at the top level by the aforementioned
methods, some studies have indicated that the bottom level often represents a
bottleneck in recommendation systems (maria, ; pepnet, ). As evidenced by the
results in Tables 3 and 4, the _Scenario-Adaptive Parameter Methods_ delivered
commendable predictive performance on both datasets, lending support to this
hypothesis. Notably, Adasparse’s performance is marginally inferior to that of
PEPNet. We surmise that this is because the mechanism of sparse neurons is
relatively challenging to learn. Sparse data scenarios in recommendations
frequently result in suboptimal performance. The MARIA model outshines others
on both datasets by optimizing both the lower and upper structures of the
model. However, it is apparent that these methods do not fully address the
pivotal task of modeling user interests across multi-scenarios. This oversight
is why our approach, SFPNet, secures the best prediction performance across
both datasets, emphatically underscoring the significance of finely-grained
modeling of user behaviors across scenarios.

###  5.4. Comparison with Training Alone (RQ2)

While our model SFPNet surpasses the baselines in scenario-based prediction
accuracy through multi-scenario joint modeling, we must ascertain whether
multi-scenario joint training confers benefits on each individual scenario.
Specifically, we question whether training a separate model for each scenario
using our custom module might yield better results than joint training. Should
some scenarios experience diminished performance following joint training, it
may be advantageous to model these scenarios independently to achieve better
online outcomes. This would also indicate that our model has room for
improvement in accurately modeling inter-scenario relationships. Consequently,
to investigate the potential issues outlined above, we trained SFPNet
individually on both public and industrial datasets. The detailed experimental
results, illustrated in the Figure 4, show a white box representing improved
performance when employing joint modeling as compared to separate training.
The results reveal that independent training not only requires more manpower
and resources but also underperforms compared to joint training. These
findings underscore the importance of leveraging data from various scenarios
for recommendation tasks, particularly those with sparse data, and also
validate our multi-scenario structural design’s ability to effectively
transfer knowledge between scenarios.

Figure 4. The impact of multi-scenario joint training. Table 5. The performance of five variants on the industrial dataset. We record the mean results over 5 runs.  Method | The industrial dataset (S-GAUC)
---|---
#A1 | #A2 | #A3 | #A4 | #A5 | #A6 | #A7 | #A8 | #A9 | #A10 | #A11 | #A12
SFPNet | 0.6524 | 0.5612 | 0.6489 | 0.6447 | 0.6658 | 0.6309 | 0.6689 | 0.6714 | 0.6597 | 0.6548 | 0.6375 | 0.6432
w/o DAP | 0.6506 | 0.5596 | 0.6478 | 0.6429 | 0.6649 | 0.6301 | 0.6677 | 0.6701 | 0.6583 | 0.6531 | 0.6364 | 0.6413
w/o SAM | 0.6491 | 0.5573 | 0.6462 | 0.6403 | 0.6621 | 0.6284 | 0.6651 | 0.6673 | 0.6558 | 0.6512 | 0.6339 | 0.6401
w/o RTM | 0.6470 | 0.5560 | 0.6424 | 0.6397 | 0.6605 | 0.6250 | 0.6621 | 0.6655 | 0.6528 | 0.6501 | 0.6318 | 0.6364
w/o S-DNN | 0.6497 | 0.5582 | 0.6470 | 0.6414 | 0.6628 | 0.6291 | 0.6660 | 0.6678 | 0.6567 | 0.6526 | 0.6351 | 0.6426
w/o STB | 0.6461 | 0.5553 | 0.6415 | 0.6384 | 0.6596 | 0.6241 | 0.6613 | 0.6643 | 0.6515 | 0.6492 | 0.6301 | 0.6350

###  5.5. Ablation Study (RQ3)

To verify the effectiveness of each module in the proposed model, i.e., the
SFPNet method, we conduct a series of ablation studies over the industrial
dataset. We have five variants as follows:
$\bullet$ w/o DAP removes the distribution-aware sequence pooling method and
directly adopts the average pooling of sequences.
$\bullet$ w/o SAM removes the Scenario-Adaptive Module, which means that the
block no longer emphasizes scenario information.
$\bullet$ w/o RTM removes the Residual-Tailoring Module in each block, which
implies that the block no longer provides finely-tuned, scenario-specific
representations for individual user behaviors.
$\bullet$ w/o S-DNN removes the scenario-aware DNN and replace it with a
normal deep neural network.
$\bullet$ w/o STB removes the Scenario-Tailoring Block and directly conducts
predictions based on S-DNN .

Table 5 presents the performance metrics of five SFPNet variant models. The
results indicate that SFPNet’s performance diminishes with the removal of any
module, underscoring the contribution of each component to predictive
accuracy. Specifically, the absence of distribution-aware sequence modeling
(w/o DAP) impacts prediction performance across all scenarios, aligning with
our hypothesis that mean pooling in sequence modeling may result in a loss of
critical information. Additionally, we observed that despite the presence of
an S-DNN atop SFPNet to assist with scene perception, the removal of SAM (w/o
SAM) still results in a decline in model performance, reinforcing our
understanding of the critical nature of the bottom representation proposed in
previous work—akin to the proverb ”a breach in a dike may cause a collapse”.
Furthermore, the Residual-Tailoring Module’s absence (w/o RTM) notably affects
the performance of each scenario. Indeed, without the RTM, our model reverts
to the scene adaptive parameter structure of prior multi-scenario modeling
methods, consequently forfeiting the capability for scenario-specific
customization of user behavior. The experimental data suggests that the
performance drop from removing either the RTM or the Scenario-Tailoring Block
(w/o STB) is quite comparable. This underscores the significance of the RTM
and robustly validates our rationale for concentrating on fine-grained
scenario modeling exploration. Lastly, the removal of S-DNN (w/o S-DNN) also
leads to a measurable reduction in the model’s prediction performance. This
reveals that, although we prioritize the bottom layer’s architecture, the top
structure’s design must not be overlooked. This reflects the initial design
motivation for the method, which was to clearly structure the scenario.

Figure 5. The performance of SFPNet under different predefined hyper-
parameters on the both datasets.

###  5.6. Hyper-Parameters Sensitivity Analysis (Q4)

This section examines the sensitivity of the hyper-parameter of Scenario-
Tailoring Block stacking times within the SFPNet model, i.e., $L$. We assessed
SFPNet’s performance on two datasets with five different values of $L$ (1, 2,
3, 4, and 5), with results presented in Figure 5(a) for the industrial dataset
and Figure 5(b) for the other. Analysis of the industrial dataset reveals that
$L$ values of 3 and 4 yield similar and optimal prediction performance;
however, an additional block layer increases computational demands.
Consequently, we recommend setting $L$ to 3. Theoretically, increasing the
number of stacks should deepen the model, potentially enhancing its capability
to model multiple scenarios. Nevertheless, given the dataset size, elevating
the stack count beyond a certain point complicates training and can
deteriorate prediction performance. Similarly, for the public dataset, setting
$L=2$ results in peak performance for SFPNet. This optimal setting is
attributable to the smaller size of the Ali-CCP dataset compared to the
industrial dataset. Excessive block stacking increases the likelihood of
overfitting in the model.

###  5.7. Online A/B Testing Results

To more robustly validate our model’s performance, we conducted an online A/B
test on an online e-commerce platform. The control in our test was a base
model—a model structure corresponds to the BaseDNN from our baseline
comparisons. We averaged the results of the 12 scenarios in our platform from
October 21 to 26, 2023, to determine the final outcome of the online test.
Specifically, following the implementation of the SFPNet model, we observed a
_6.4%_ increase in cumulative _Revenue_ and a _9.2%_ rise in user _Click-
Through Rate (CTR)_ compared to the base model. The online results further
corroborate the effectiveness of the proposed SFPNet approach for multi-
scenario recommendation.

##  6\. Conclusions

In this paper, we point out the optimizable space for modeling user sequences
within multi-scenario contexts in the multi-scenario recommendation field and
propose a novel model named Scenario-Adaptive Fine-Grained Personalization
Network (SFPNet) to explore the optimization space for modeling user sequences
with awareness of scenarios in a fine-grained way. SFPNet is is composed of
the Scenario-Tailoring Block stacked block by block, and each block comprises
two key components: the Scenario-Adaptive Module (SAM) and the Residual-
Tailoring Module (RTM). Specifically, SAM employs a gating mechanism to infuse
scenario-specific information into the instance’s contextual features, whereas
RTM harnesses the generated scenario-aware contextual information by SAM to
craft a distinctive representation vector for each behavior in the sequence,
utilizing residual connections for this purpose. This process ultimately
bolsters the model’s capacity to capture the evolving patterns of user
interests across different scenarios. Finally, extensive experiments validate
the superiority of SFPNet in the multi-scenario recommendation task.

## References

  * (1) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. Tensorflow: A system for large-scale machine learning. In _12th USENIX symposium on operating systems design and implementation (OSDI 16)_.
  * (2) Alex Beutel, Paul Covington, Sagar Jain, Can Xu, Jia Li, Vince Gatto, and Ed H. Chi. 2018. Latent Cross: Making Use of Context in Recurrent Recommender Systems. In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (WSDM)_. (Feb. 2018), 46-54.
  * (3) Yichao Wang, Huifeng Guo , Bo Chen, Weiwen Liu, Zhirong Liu, Qi Zhang, Zhicheng He, Hongkun Zhen, Weiwei Yao, Muyu Zhang, Zhenhua Dong, and Ruiming Tang. 2022. CausalInt: Causal Inspired Intervention for Multi-Scenario Recommendation. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD)_. (Aug. 2022), 4090-4099.
  * (4) Jianxin Chang, Chenbin Zhang, Yiqun Hui, Dewei Leng, Yanan Niu, Yang Song, and Kun Gai. 2023. PEPNet: Parameter and Embedding Personalized Network for Infusing with Personalized Prior Information. In _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD)_. (Aug. 2023), 3795-3804.
  * (5) Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. Deepfm: a factorization-machine based neural network for ctr prediction. In _Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI)_. Melbourne, Australia., 2782–2788.
  * (6) Wei Guo, Chenxu Zhu, Fan Yan, Bo Chen, Weiwen Liu, Huifeng Guo, Hongkun Zheng, Yong Liu, and Ruiming Tang. 2023. DFFM: Domain Facilitated Feature Modeling for CTR Prediction. In _Proceedings of the Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM)_. (Oct. 2023), 4602-4608.
  * (7) Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_. 249–256.
  * (8) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. (Jun. 2016), 770-778.
  * (9) Zhaoxin Huan, Ang Li, Xiaolu Zhang, Xu Min, Jieyu Yang, Yong He, and Jun Zhou. 2023. SAMD: An Industrial Framework for Heterogeneous Multi- Scenario Recommendation. In _Proceedings of the 29th ACM SIGKDD Con- ference on Knowledge Discovery and Data Mining (KDD)_. (Aug. 2023), 4175-4184.
  * (10) Alex Kendall, Yarin Gal, and Roberto Cipolla. 2018. Multi-task learning using uncertainty to weight losses for scene geometry and semantics. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. UT, USA, 7482–7491.
  * (11) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR.
  * (12) Jinyun Li, Huiwen Zheng, Yuanlin Liu, Minfang Lu, Lixia Wu, and Haoyuan Hu. 2023. ADL: Adaptive Distribution Learning Framework for Multi-Scenario CTR Prediction. In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)_. (Jul. 2023), 1786-1790.
  * (13) Pengcheng Li, Runze Li, Qing Da, An-Xiang Zeng, and Lijun Zhang. 2020. Improving Multi-Scenario Learning to Rank in E-commerce by Exploiting Task Relationships in the Label Space. In _Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM)_. (Oct. 2020), 2605-2612.
  * (14) Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial Multi-task Learning for Text Classification. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_. Vancouver, Canada, 1–10.
  * (15) Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. 2018. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD)_. London, UK, 1930–1939.
  * (16) Simon J Mason and Nicholas E Graham. 2002. Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation. _Quarterly Journal of the Royal Meteorological Society: A journal of the atmospheric sciences, applied meteorology and physical oceanography_. 128, 584 (2002), 2145–2166.
  * (17) Shanlei Mu, Penghui Wei, Wayne Xin Zhao, Shaoguo Liu, Liang Wang, Bo Zheng. 2023. Hybrid Contrastive Constraints for Multi-Scenario Ad Ranking. In _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM)_. (Oct. 2023), 1857-1866.
  * (18) Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2001. Item-based collaborative filtering recommendation algorithms. In _Proceedings of the 10th international conference on World Wide Web_. 285–295.
  * (19) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. In _Proceedings of the 5th International Conference on Learning Representations (ICLR)_. (Apr. 2017).
  * (20) Qijie Shen, Wanjie Tao, Jing Zhang, Hong Wen, Zulong Chen, and Quan Lu. 2021. SAR-Net: A Scenario-Aware Ranking Network for Personalized Fair Recommendation in Hundreds of Travel Scenarios. In _Proceedings of the 30th ACM International Conference on Information and Knowledge Management (CIKM)_. (Nov. 2021), 4094-4103.
  * (21) Pawel Swietojanski, Jinyu Li, and Steve Renals. 2016. Learning hidden unit contributions for unsupervised acoustic model adaptation. _IEEE/ACM Transactions on Audio, Speech, and Language Processing_. 24, 8 (2016), 1450–1463.
  * (22) Xiang-Rong Sheng, Liqin Zhao, Guorui Zhou, Xinyao Ding, Binding Dai, Qiang Luo, Siran Yang, Jingshan Lv, Chi Zhang, Hongbo Deng, et al. 2021. One model to serve all: Star topology adaptive recommender for multi-domain ctr prediction. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management (CIKM)_. 4104–4113.
  * (23) Hongyan Tang, Junning Liu, Ming Zhao, and Xudong Gong. 2020. Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations. In _Fourteenth ACM Conference on Recommender Systems_. 269– 278.
  * (24) Yu Tian, Bofang Li, Si Chen, Xubin Li, Hongbo Deng, Jian Xu, Bo Zheng, Qian Wang, and Chenliang Li. 2023. Multi-Scenario Ranking with Adaptive Feature Learning. In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)_. (Jul. 2023), 517-526.
  * (25) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser and Illia Polosukhin. 2017. Attention is All you Need. In _Proceedings of 30th Conference on Advances in Neural Information Processing Systems (NIPS)_. (Dec. 2017), 5998–6008.
  * (26) Hong Wen, Jing Zhang, Fuyu Lv, Wentian Bao, Tianyi Wang, and Zulong Chen. 2021. Hierarchically Modeling Micro and Macro Behaviors via Multi-Task Learn- ing for Conversion Rate Prediction. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)_.
  * (27) Hong Wen, Jing Zhang, Yuan Wang, Fuyu Lv, Wentian Bao, Quan Lin, and Keping Yang. 2020. Entire space multi-task modeling via post-click behavior decom- position for conversion rate prediction. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)_. 2377–2386.
  * (28) Yuhao Wang, Xiangyu Zhao, Bo Chen, Qidong Liu, Huifeng Guo, Huan-shuo Liu, Yichao Wang, Rui Zhang, and Ruiming Tang. 2023. PLATE: A Prompt-Enhanced Paradigm for Multi-Scenario Recommendations. In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)_. (Jul. 2023), 1498-1507.
  * (29) Xuanhua Yang, Xiaoyu Peng, Penghui Wei, Shaoguo Liu, Liang Wang and Bo Zheng. 2022. AdaSparse: Learning Adaptively Sparse Structures for Multi-Domain Click-Through Rate Prediction. In _Proceedings of the 31st ACM International Conference on Information and Knowledge Management (CIKM)_. (Oct. 2022), 4635-4639.
  * (30) Jing Zhang and Dacheng Tao. 2021. Empowering Things With Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things. _IEEE Internet of Things Journal_. 8(10), 7789–7817.
  * (31) Qianqian Zhang, Xinru Liao, Quan Liu, Jian Xu, Bo Zheng. 2022. Leaving No One Behind: A Multi-Scenario Multi-Task Meta Learning Approach for Advertiser Modeling. In _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining (WSDM)_. (Feb. 2022), 1368-1376.
  * (32) Pengyu Zhao, Xin Gao, Chunxu Xu, and Liang Chen. 2023. M5: Multi-Modal Multi-Interest Multi-Scenario Matching for Over-the-Top Recommendation. In _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD)_. (Aug. 2023), 5650-5659.
  * (33) Yifei Zhang, Hua Hua, Hui Guo, Shuangyang Wang, Chongyu Zhong, and Shijie Zhang. 2023. 3MN: Three Meta Networks for Multi-Scenario and Multi-Task Learning in Online Advertising Recommender Systems. In _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM)_. (Oct. 2023), 4945-4951.
  * (34) Yuanliang Zhang, Xiaofeng Wang, Jinxin Hu, Ke Gao, Chenyi Lei, and Fei Fang. 2022. Scenario-Adaptive and Self-Supervised Model for Multi-Scenario Personalized Recommendation. In _Proceedings of the 31st ACM International Conference on Information and Knowledge Management (CIKM)_. (Oct. 2022), 3674-3683.
  * (35) Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep interest network for click-through rate prediction. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD)_. 1059–1068.
  * (36) Jie Zhou, Xianshuai Cao, Wenhao Li, Lin Bo, Kun Zhang, Chuan Luo, and Qian Yu. 2023. HiNet: Novel Multi-Scenario & Multi-Task Learning with Hierarchical Information Extraction. In _Proceedings of the 39th IEEE International Conference on Data Engineering (ICDE)_. (Apr. 2023), 2969-2975.
  * (37) Xinyu Zou, Zhi Hu, Yiming Zhao, Xuchu Ding, Zhongyi Liu, Chenliang Li, Aixin Sun. 2022. Automatic Expert Selection for Multi-Scenario and Multi- Task Search. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)_. (Jul. 2022), 1535-1544.


# V2L: Leveraging Vision and Vision-language Models into Large-scale Product
Retrieval

Wenhao Wang∗ Yifan Sun∗ Zongxin Yang† Yi Yang†
∗Baidu Research  †Zhejiang University
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Product retrieval is of great importance in the ecommerce domain. This paper
introduces our 1st-place solution in eBay eProduct Visual Search Challenge
(FGVC9), which is featured for an ensemble of about 20 models from vision
models and vision-language models. While model ensemble is common, we show
that combining the vision models and vision-language models brings particular
benefits from their complementarity and is a key factor to our superiority.
Specifically, for the vision models, we use a two-stage training pipeline
which first learns from the coarse labels provided in the training set and
then conducts fine-grained self-supervised training, yielding a coarse-to-fine
metric learning manner. For the vision-language models, we use the textual
description of the training image as the supervision signals for fine-tuning
the image-encoder (feature extractor). With these designs, our solution
achieves 0.7623 MAR@10, ranking the first place among all the competitors. The
code is available at: V2L.

##  1 Introduction

Given a query image, the goal of product retrieval is to determine whether
there are the same product images from a reference dataset. It plays an
important role in the e-commerce domain. There are two main challenges. First,
because product retrieval is an instance of super fine-grained recognition, it
is very difficult for algorithms to distinguish two products with subtle
visual differences. Second, building large-scale product training datasets
with fine-grained level labels is very time-consuming and expensive.

This paper tries to leverage Vision and Vision-Language models (V2L) to
compete for eBay eProduct Visual Search Challenge (FGVC9) [21] at CVPR’22.
This competition builds a dataset with $2.5$ million product images. Different
from other similar datasets [1, 3], the dataset features: (1) It does not have
fine-grained labels, i.e. it only provides categorical labels rather than
product labels. In this way, we can build self-supervised learning algorithms
on our previous winning solutions [17, 18]. (2) It provides text information
(title) for each product image in the training set. In this way, we can
explore multimodal learning methods. To explore the highest performance, we do
not limit the amount of GPU and CPU resources. We use $6$ standard Nvidia
$8$-A100 GPU servers and about $100$ Intel 6240 CPUs (with $36$ cores per CPU)
in the competition.

The proposed approach consists of two parts, i.e. vision models, and vision-
language models. For vision models, we perform a coarse-to-fine training
strategy. First, backbones are trained with the provided $1,000$ categories
labels (without ImageNet pre-training). Then we adapt the strong self-
supervised baseline in [18] to accommodate the product retrieval task. We
choose many vision-language models, which are pre-trained and with suitable
licenses (both code and the models themselves), and then finetune them using
the product images and corresponding titles. That is how we perform multimodal
learning. The final submission is obtained by ensembling different models’
results from two types of approaches. The illustration of the proposed V2L is
shown in Fig. 1.

Figure 1: The designed V2L approach. We leverage vision models and vision-
language models into large-scale product retrieval. The final submission is
obtained by ensembling different models’ results from two types of approaches.

In summary, the main contributions of this paper are:

  1. 1.

The paper demonstrates the effectiveness of vision-language models in product
retrieval tasks for the first time.

  2. 2.

We explore the performance threshold of current product retrieval tasks by
using unimaginable resources.

  3. 3.

The proposed V2L approach ranks first in eBay eProduct Visual Search Challenge
(FGVC9).

##  2 Proposed Method

In this section, we introduce each important component in the proposed V2L. We
will first introduce our vision models and then show the selected vision-
language models. Finally, to promote reproducibility, all the used tricks are
listed.

###  2.1 Vision Models

We basically follow the training process in the strong baseline from [18]. The
changes are listed below.

####  2.1.1 Two-stage Training

We perform coarse-to-fine training. First, the randomly initialized backbones
are trained using the $1,000$ coarse labels. Note that, we do NOT use the
ImageNet-pre-trained backbones due to two issues: (1) The open-source
codebases often do NOT include an explicit license for their released pre-
trained models. We do NOT want to involve in the license issue in the final
checking. (2) It seems that the scale of eBay training dataset is large enough
for training from scratch. Then, using the strong baseline [18], the second
stage training is performed. We use $8$ and $4$ Nvidia Tesla A100 GPUs for the
two stages of training, respectively.

####  2.1.2 Pseudo-label Generation

First, we randomly select $100,000$ training images from the training set.
Through augmentation, each training image forms its only class. A model is
trained on this $100,000$ classes. After training, features extracted by the
model can used for clustering. In this way, the training images are clustered.
Note that, we only keep clusters with high confidence, i.e. each cluster only
contains less than $10$ images, and thus we only use about $1/5$ training
images in the second stage training. Specifically, after the clustering, we
get $87,125$ classes. Then we randomly select $100K-87,125=12,875$ images from
the training dataset. That means besides the $87,125$ clusters, we have
$12,875$ extra classes (each class with one image). As a result, we have
$246,926$ images from the $87,125$ clusters ($2.8$ images for each cluster on
average). Then, the $12,875$ extra classes bring $12,875$ extra images.
Therefore, finally we have $246,926+12,875=255,801$ images.

####  2.1.3 Stronger Backbones

Instead of using the backbones in [18], in this competition, we use much
stronger backbones with suitable licenses: ResNeSt [24] (license), ResNeXt
[19] (license), CotNet [11] (license), HS-ResNet [22] (license), NAT [7]
(license), ViT (from BEIT [2]) (license), and ViT (from SimMIM [20])
(license). The stronger backbones perform much better.

####  2.1.4 Simpler Augmentations

The augmentations used in [18] are designed for image copy detection, and thus
they are too complexed. For product retrieval tasks, we only keep
RandomResizedCrop, RandomRotation, RandomPerspectiveChange, RandomPadding,
RandomImageUnderlay, RandomImageOverlay, RandomLightChange,
RandomHorizontalFlip, RandomVerticalFlip, and GaussianBlur. For more details,
please refer to the original paper [18].

####  2.1.5 Higher Resolution

In image copy detection, $256\times 256$ resolution is the best. For the
product retrieval task, we find that $512\times 512$ resolution performs best.
Though higher resolution brings performance improvement, it also brings much
computational burden to the training and reference process.

####  2.1.6 Loss

To be computationally efficient, we replace the $8192$-dim features [18] with
commonly-used $2048$-dim features. Moreover, instead of combining cross-
entropy loss and triplet loss [8], we use a single CosFace [16].

####  2.1.7 Exponential Moving Average

Exponential Moving Average (EMA) is used to stabilize the training process of
deep metric learning. It brings consistent performance improvement without
increasing any burden.

Figure 2: The used Index5Crop and Index6Crop. The index images are splited
into $5$ or $6$ parts to be matched locally. Though it brings $5\times$ or
$6\times$ computational burden, it significantly alleviates the side effect of
background.

###  2.2 Vision-language Models

In this section, we introduce the selected vision-language models and how we
use that. Due to the license issue, we only choose BLIP [9] (license), ALBEF
[10] (license), XVLM [23] (license), METER [6] (license), and SLIP [13]
(license). We get the suitable licenses for both the code and the released
models. We do NOT use the famous CLIP [15] because we cannot get the license
for the pre-trained models.

The finetuning process is relatively easy. Because each algorithm takes the
nature language and the image as input, we just use the product title as the
description of one product image. After training, the image encoder is used to
extract the features of query and reference images. Though the use of the
vision-language models is easy, it provides a totally different prospective to
the product retrieval task.

###  2.3 Ensemble Methods

####  2.3.1 Maximum Ensemble

The ensemble method in [17] is proved to be effective in the product retrieval
task.

####  2.3.2 Voting Ensemble

Each model returns a ranking list, and we fetch the top-10 images. Many models
vote for the final ranking list. It is similar (or same) with last year’s
“Multiple networks rankings”.

###  2.4 Other Tricks

####  2.4.1 Global-local Matching

We split reference (index) images to match locally (The query images are
kept). We name the used two strategies as Index5Crop and Index6Crop. They are
illustrated in Fig . 2. Note that we only re-train $3$ different methods to
perform Index5Crop ($2$) and Index6Crop ($1$). The gained results are absorbed
to final ranking by voting ensemble.

####  2.4.2 Re-ranking

The k-reciprocal re-ranking [25] is proved to be very useful for image
retrieval tasks. However, it is not straightforward to apply it to millions of
reference images. We distribute it with about $100$ CPUs. The advantage is the
speed is much faster, and the memory is enough. However, we cannot promise
$100$ CPUs work normally at the same time. Therefore, there may be some
queries missing during the re-ranking process. We argue that because we
ensemble a lot of models, some queries missing do not bring serious influence
to the final result.

####  2.4.3 Multi-scale Testing

Because $512\times 512$ resolution performs best, in the multi-scale testing,
we use $400\times 400$, $512\times 512$, and $600\times 600$ resolutions.

Figure 3: The visualization of the ranking list. In each row, the first image
is the query image, and the following $10$ images are the index images.

##  3 Ineffective Methods

We have tried a lot of methods in this competition, and only some of them work
in the final ensemble period. The listed ineffective methods are
representative and seem to work.

###  3.1 Title for Pseudo-labels

Last year’s winner finds using the product titles to create pseudo-labels
useful. We find that, when the MAR@10 is relatively low (less than $70$), this
approach is effective regardless of using a single model or ensemble. However,
when the MAR@10 approaches $75$, regardless of using BERT [5] or word2vec [12]
to generate sentence embeddings, this approach cannot contribute to the final
ensemble result.

###  3.2 Multiple Layers Features

It is reasonable that deep features of two images, which have subtle
difference, may be indistinguishable. However, both training models with
shallow features and directly combining shallow features with deep features do
not work.

###  3.3 Multiple Augmented Queries

Performing test-time augmentation (TTA) is a common practice for image
retrieval. However, we find performing TTA (e.g. center cropping, flipping) on
query images do not bring performance improvement.

###  3.4 Query Expansion

The standard query expansion (QE) [4] and $\alpha$ query expansion
($\alpha$QE) [14] are widely used for image retrieval. However, both of them
do not work in our approach.

###  3.5 Background Removal

We try to use a model to remove the background of all the query and index
images. However, the performance decreases. It may because the background
removal algorithm removes some discriminative matching information.

##  4 Experiments

To prove the superiority of the V2L, we compare the proposed model with state-
of-the-art methods from the leaderboard in the testing phase. The comparison
results are shown in Table 1. We ranks first among all the competitors. We
discover that (1) Our solution is much higher than other competitors’. (2)
There is an obvious performance gap between top-3 and others’ scores. Also, we
visualize some results of our final ranking list. They are shown in Fig. 3.

Table 1: Comparison with state-of-the-art methods from the leaderboard in the testing phase. By ensembling about $20$ models, we rank first in the testing phase of the competition. Team | MAR@10
---|---
CVALLStar |  0.762274
Involution King |  $0.715327$
USTC-IAT-United |  $0.701012$
fgvc9 |  $0.659382$
ums_v1 |  $0.618245$
OPPO Research |  $0.547500$
AI_VIS |  $0.532808$
… |  …

##  5 Conclusion

In this paper, we introduce our winning solution to the eBay eProduct Visual
Search Challenge (FGVC9) at CVPR’22. The proposed V2L combines vision models
and vision-language models. Experiments show that the combination and the
ensemble are the key to our winning.

## References

  * [1] Yalong Bai, Yuxiang Chen, Wei Yu, Linfang Wang, and Wei Zhang.  Products-10k: A large-scale product recognition dataset.  arXiv preprint arXiv:2008.10545, 2020.
  * [2] Hangbo Bao, Li Dong, and Furu Wei.  Beit: Bert pre-training of image transformers.  arXiv preprint arXiv:2106.08254, 2021.
  * [3] Lele Cheng, Xiangzeng Zhou, Liming Zhao, Dangwei Li, Hong Shang, Yun Zheng, Pan Pan, and Yinghui Xu.  Weakly supervised learning with side information for noisy labeled images.  In The European Conference on Computer Vision (ECCV), August 2020\.
  * [4] Ondřej Chum, Andrej Mikulik, Michal Perdoch, and Jiří Matas.  Total recall ii: Query expansion revisited.  In CVPR 2011, pages 889–896. IEEE, 2011.
  * [5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [6] Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, Zicheng Liu, and Michael Zeng.  An empirical study of training end-to-end vision-and-language transformers.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [7] Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi.  Neighborhood attention transformer.  2022\.
  * [8] Alexander Hermans, Lucas Beyer, and Bastian Leibe.  In defense of the triplet loss for person re-identification.  arXiv preprint arXiv:1703.07737, 2017.
  * [9] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In ICML, 2022.
  * [10] Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  In NeurIPS, 2021.
  * [11] Yehao Li, Ting Yao, Yingwei Pan, and Tao Mei.  Contextual transformer networks for visual recognition.  arXiv preprint arXiv:2107.12292, 2021.
  * [12] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space.  arXiv preprint arXiv:1301.3781, 2013.
  * [13] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie.  Slip: Self-supervision meets language-image pre-training.  arXiv preprint arXiv:2112.12750, 2021.
  * [14] Filip Radenović, Giorgos Tolias, and Ondřej Chum.  Fine-tuning cnn image retrieval with no human annotation.  IEEE transactions on pattern analysis and machine intelligence, 41(7):1655–1668, 2018.
  * [15] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR, 18–24 Jul 2021.
  * [16] Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu.  Cosface: Large margin cosine loss for deep face recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5265–5274, 2018.
  * [17] Wenhao Wang, Yifan Sun, Weipu Zhang, and Yi Yang.  D^ 2lv: A data-driven and local-verification approach for image copy detection.  arXiv preprint arXiv:2111.07090, 2021.
  * [18] Wenhao Wang, Weipu Zhang, Yifan Sun, and Yi Yang.  Bag of tricks and a strong baseline for image copy detection.  arXiv preprint arXiv:2111.08004, 2021.
  * [19] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  arXiv preprint arXiv:1611.05431, 2016.
  * [20] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu.  Simmim: A simple framework for masked image modeling.  arXiv preprint arXiv:2111.09886, 2021.
  * [21] Jiangbo Yuan, An-Ti Chiang, Wen Tang, and Antonio Haro.  eproduct: A million-scale visual search benchmark to address product recognition challenges.  arXiv preprint arXiv:2107.05856, 2021.
  * [22] Pengcheng Yuan, Shufei Lin, Cheng Cui, Yuning Du, Ruoyu Guo, Dongliang He, Errui Ding, and Shumin Han.  Hs-resnet: Hierarchical-split block on convolutional neural network.  arXiv preprint arXiv:2010.07621, 2020.
  * [23] Yan Zeng, Xinsong Zhang, and Hang Li.  Multi-grained vision language pre-training: Aligning texts with visual concepts.  arXiv preprint arXiv:2111.08276, 2021.
  * [24] Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Zhi Zhang, Haibin Lin, Yue Sun, Tong He, Jonas Muller, R. Manmatha, Mu Li, and Alexander Smola.  Resnest: Split-attention networks.  arXiv preprint arXiv:2004.08955, 2020.
  * [25] Zhun Zhong, Liang Zheng, Donglin Cao, and Shaozi Li.  Re-ranking person re-identification with k-reciprocal encoding.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1318–1327, 2017.