# HOMOE: A Memory-Based and Composition-Aware Framework for Zero-Shot Learning
with Hopfield Network and Soft Mixture of Experts

Do Huu Dat
<EMAIL_ADDRESS>VinUniversity Po Yuan Mao
Kyushu University Tien Hoang Nguyen
VNU University of Engineering and Technology Wray Buntine
VinUniversity Mohammed Bennamoun
University of Western Australia

###### Abstract

Compositional Zero-Shot Learning (CZSL) has emerged as an essential paradigm
in machine learning, aiming to overcome the constraints of traditional zero-
shot learning by incorporating compositional thinking into its methodology.
Conventional zero-shot learning has difficulty managing unfamiliar
combinations of seen and unseen classes because it depends on pre-defined
class embeddings. In contrast, Compositional Zero-Shot Learning uses the
inherent hierarchies and structural connections among classes, creating new
class representations by combining attributes, components, or other semantic
elements. In our paper, we propose a novel framework that for the first time
combines the Modern Hopfield Network with a Mixture of Experts (HOMOE) to
classify the compositions of previously unseen objects. Specifically, the
Modern Hopfield Network creates a memory that stores label prototypes and
identifies relevant labels for a given input image. Following this, the
Mixture of Expert models integrates the image with the fitting prototype to
produce the final composition classification. Our approach achieves SOTA
performance on several benchmarks, including MIT-States and UT-Zappos. We also
examine how each component contributes to improved generalization.

##  1 Introduction

In recent years, machine learning has made substantial advancements, with
innovations covering a wide range of applications such as image categorization
and natural language interpretation. Among the evolving paradigms, Zero-shot
Learning (ZSL) has gained attention, aiming at overcoming the challenge of
model generalization to unseen categories with scarce labeled instances.
Conventional ZSL methodologies use auxiliary information like class attributes
and class embeddings but struggle with unseen class combinations. In this
context, Compositional Zero-Shot Learning (CZSL) is emerging as a viable
solution, especially in structured or high multiclass situations.

Figure 1: The goal of zero-shot compositional learning is to create a
classifier capable of identifying visual concepts denoted by attribute-object
pairs (such as “bright street”) even when no training images of that specific
composition exist. Our approach involves generating synthetic features for
these new compositions by leveraging knowledge acquired from observed
compositions like “bright ocean” and ”sunny street.” These synthetic features
are then used to train the classifier directly.

Compositional Zero-Shot Learning (CZSL) is a task aimed at recognizing unseen
compositional visual concepts by training models on learned concepts. Fig 1
demonstrates the basic idea of CZSL. It primarily involves modeling attributes
that interact with different objects and addressing the challenges posed by
diversity, informativeness, and entanglement between visual primitives. CZSL
effectively incorporates compositional reasoning, which facilitates models to
accurately predict new class combinations by considering the hierarchical and
structural relationships among classes. This is achieved through the
generation of embeddings based on constituent components, thereby offering a
robust strategy towards tackling Zero-Shot Learning (ZSL) challenges. Through
this methodical approach, CZSL not only broadens the scope of recognizing new
visual compositions but also significantly contributes towards enhancing the
model’s generalization capabilities in unseen scenarios.

A common CZSL strategy involves merging scores from different classifiers,
independently trained for attributes and objects [28]. Nevertheless, such
separate predictions often overlook the contextual interactions originating
from the combination of different primitives, termed as contextuality within
compositions [28]. Previous approaches have tackled this challenge by modeling
compositional label embeddings for each class, making use of external
knowledge repositories like Glove [27] to extract semantic vectors for
attributes and objects. These vectors are subsequently concatenated through
multiple layers to form the compositional label embedding. The resulting
embeddings are then aligned with visual attributes in a shared embedding
space, effectively transforming the recognition of unseen images into a
nearest neighbor search problem [25]. However, the extent of contextual
consideration within the visual domain remains constrained. Prior
investigations have also underscored the significance of discriminability in
visual features, a factor influencing generalization performance in
recognizing unseen compositions. Disentanglement emerges as a favored
solution, with dedicated layers for extracting intermediate visual
representations of attributes and objects [38, 39, 18, 44].

In this paper, we propose a novel methodology intended to faithfully mimic the
complex process wherein humans inherently fuse various visual elements to
create compositions. Our assertion on the profound impact of memory on human
compositionality is further corroborated by the way individuals employ towards
novel concepts. Upon encountering unfamiliar ideas or scenarios, humans
frequently reason and formulate logical conjectures rooted in their pre-
existing knowledge and memory [2]. Additionally, the interactions between
diverse attributes and objects are unique, necessitating varied composition
techniques.

Motivated by this, we propose HOMOE, a novel and efficient model centered on
learning primitive representations, retrieving, and combining them to create
compositional and joint representations. To learn the joint representations,
HOMOE exploits the soft prompt from DFSP [23]. Furthermore, we develop a set
of learnable features, initially represented by the mean embeddings of random
images corresponding to the same classes, and use the Modern Hopfield Network
[36] to retrieve representations of akin attributes or objects predicated on
the input image. The Modern Hopfield Network, drawing from biological
concepts, aids in establishing an associative memory framework crucial for the
retrieval process, emulating a human-like approach toward recognizing and
categorizing new or unseen objects or scenarios. Associative memory allows the
network to recall patterns or representations based on partial or noisy
inputs, providing a robust mechanism for retrieval even in the face of
incomplete or imperfect cues. This property is essential for real-world
applications where inputs are often ambiguous or occluded. Lastly, we deploy a
Soft Mixture of Experts (Soft MoE) [34] to adapt the input image embeddings
based on the retrieved features. The ”soft” in Soft Mixture of Experts refers
to the manner in which the model allocates the handling of inputs among
different ”experts” within the model. Unlike a ”hard” Mixture of Experts model
where each input is directed to exactly one expert based on the gating
network, a Soft Mixture of Experts model allows for a more nuanced allocation,
permitting each input to be handled by multiple experts with varying degrees
of weighting determined by the gating network. This soft assignment enables a
more flexible and nuanced handling of inputs, which could be particularly
beneficial in complex or multifaceted tasks, providing a form of
regularization and a richer representation of the data as it gathers insights
from multiple experts for each input. The sparsity of the Soft MoE model is
perceived as a form of specialization, with certain experts dedicated to
managing specific attributes.

To summarize, the primary contributions of this paper are as follows:

  * •

We introduce an innovative framework - HOMOE, inspired by human memory,
leveraging external references to effectively recognize novel combinations
using exclusive primitives.

  * •

For the first time, our proposed framework employs a Modern Hopfield Network
and Mixture of Expert models for Compositional Zero-Shot Learning (CZSL),
capitalizing on their synergy to enhance the classification of unseen
compositions.

  * •

We also devise additional loss functions aimed at progressively augmenting the
memory’s composability while preserving the primitive clustering.

To evaluate these contributions, we conduct extensive experiments examining
the influence of memory design and auxiliary loss functions on the overall
model performance, alongside the effectiveness of other elements within our
methodology.

##  2 Related Works

###  2.1 Compositional Zero-Shot Learning

Compositional Zero-Shot Learning (CZSL) [20, 27, 28, 31] is a task that mimics
the human ability to imagine and identify new concepts based on prior
knowledge, marking it as a crucial part of Zero-Shot Learning (ZSL) [1, 6, 7,
14, 17, 21, 22]. While traditional ZSL uses attributes to recognize unseen
objects, CZSL views classes as mixtures of states and objects. Initially, CZSL
methods involved training a classifier for identification along with a
transformation module to change states or objects [28, 31]. However, newer
strategies use two distinct classifiers to identify states and objects
individually [11, 18, 20, 28]. Some methods even combine the encoded
attribute/state and object features using late fusion with a multi-layer
perceptron [35]. Li et al. [18] brought contrastive learning into CZSL,
designing a siamese network to identify states and objects in a contrastive
space. Other strategies [31, 32] aim at representing compositions together,
learning an embedding space to map compositions like in ZSL. Moreover, methods
have been using graph networks to represent the relationship between states
and objects and to learn their compositions [26, 38]. Lately, the attention
mechanism has been modified to disentangle compositions into primitives and
compute new compositions [12, 19, 8]. Nihal et al. [33] pioneered the use of a
VLM (Vision-Language Model) for CZSL, substituting prompt classes with a
learnable combined state and object vector representation. Additionally, the
challenge of handling compositions during testing is not just restricted to
the test data compositions, but often extends to real-world scenarios,
accounting for all potential compositions. Contrary to the earlier closed-
world setting (where the system only encounters scenarios it was trained on),
some research aims to tackle the open-world situation (where the system may
encounter completely new, unseen scenarios) by using external knowledge to
eliminate unlikely compositions [11, 26, 25, 42].

###  2.2 Associative Memory

In our proposed approach, using Associative Memory is key for efficient
storage and retrieval of learned patterns, enabling precise mapping of new
compositional representations. This capability allows our framework to
efficiently navigate the complexities of open-world scenarios, drawing from
stored knowledge to deduce the characteristics of unseen compositions, thereby
significantly enhancing the accuracy and robustness of CZSL in real-world
applications. Looking more closely into the technical aspects, associative
memory networks play a central role in linking an input with the most closely
resembling pattern, focusing on both the storage and retrieval of patterns.
This type of neural network, which is built on the framework of the classic
Hopfield Network [10], stores multidimensional vectors as memories through
recurrent dynamical systems, thus facilitating data clustering by establishing
fixed point attractor states. Yet, a significant limitation lies in its
restricted memory capacity, which can only hold around 0.14 times the
dimensionality ($d$) in random memories for a d-dimensional data domain. This
poses a challenge for clustering, especially when the number of clusters
should not be directly related to data dimensionality.

To address the drawbacks associated with the classical Hopfield Network,
Krotov, and Hopfield introduced an updated version known as the Modern
Hopfield Network, or Dense Associative Memory (Dense AM)[13]. This improved
model incorporates rapidly advancing non-linearities (activation functions)
into the dynamical system, which results in a denser memory arrangement and
significantly boosts the memory capacity, a benefit that is particularly
pronounced in high-dimensional data spaces. Some of the activation functions
employed in Dense AMs have the potential to lead to power-law or even
exponential memory capacity [3, 24].

Moreover, as pointed out by Ramsauer et al., the attention mechanism inherent
in transformers[41] can be interpreted as a particular instance of Dense AMs
when utilizing the softmax activation function [36]. They illustrated the
capacity to store an exponentially large number of patterns, retrieve patterns
with a single update, maintain an exponentially small retrieval loss, and even
can learn the memory.

###  2.3 Sparse Mixture of Experts

Expanding on the capabilities of associative memory, using a framework like
the Sparsely-gated Mixture of Experts (MoE) [40] can further improve the
performance in managing and interpreting complex data structures, especially
in the field of CZSL. The MoE model has been groundbreaking, displaying
substantial improvements in model capacity, training time, or model quality
through the incorporation of gating. Following this, the Switch Transformer
[4] simplified the gating process by choosing only the top expert per token
using a softmax over the hidden state, which displayed better scaling compared
to earlier efforts. However, a common necessity across these improvements has
been the use of an auxiliary loss to actively promote balancing within the
model. This auxiliary loss needs meticulous weighting to prevent overshadowing
the primary loss, yet it doesn’t ensure a perfect balance, necessitating a
hard capacity factor. This scenario might lead to many tokens remaining
unprocessed by the MoE layer. The introduction of Hard MoE [5]with a singular
decoding layer showed efficient training yielding positive outcomes on large-
scale hashtag prediction tasks. Furthermore, Base Layers [16] devised a linear
assignment strategy to maximize token-expert affinities while ensuring an
equitable distribution of tokens to each expert. Recently, several innovative
methods have surfaced, proposing ways to selectively trigger token paths
across the network in various domains including language [15, 4], vision [37],
and multimodal models [29]. This diverse array of strategies underscores the
potential of leveraging sophisticated models like Sparse Mixture of Experts in
addressing the complex challenges posed by the open-world setting of CZSL,
making a compelling case for its inclusion in our proposed framework.

##  3 Problem Formulation

CZSL is a classification problem that aims to train a model using a limited
set of attribute-object combinations, enabling the recognition of images with
unseen combinations. To elaborate, consider an attribute set
$A=\\{a_{0},a_{1},\ldots,a_{n}\\}$ and an object set
$O=\\{o_{0},o_{1},\ldots,o_{m}\\}$. By composing these two labels, a
composition set is formed as $C=A\times
O=\\{(a_{0},o_{0}),(a_{1},o_{0}),\ldots,(a_{n},o_{m})\\}$. The set $C$ is used
to label images with seen classes $C_{s}\subset C$ and unseen classes
$C_{u}\subset C$ where $C_{s}\cap C_{u}=\emptyset$ and each image only has one
$c\in C$ label. This division is also used to separate training
$\mathcal{D}^{tr}=\\{(X_{tr},C_{tr})\\}$ and testing
$\mathcal{D}^{test}=\\{(X_{test},C_{test})\\}$ datasets, where $X_{tr}$ is
labelled with seen classes $C_{s}$ only, so $C_{tr}\subseteq
Powerset({C_{s}})$. Whereas $X_{test}$ are labeled with $C_{s}$ and ${C_{u}}$
classes. [11] defined the open-world evaluation as $C_{test}$ labels are from
$C_{s}\cup{C_{u}}$, while for closed-world evaluation $C_{test}$ labels are
from $C_{s}\cup{C_{u}}^{\prime}$ where ${C_{u}}^{\prime}\subset C_{u}$. The
aim of CZSL is to train a classification model $f_{\theta}(X)$ on
$\mathcal{D}^{tr}$ but it should also be capable of correctly predicting
$\mathcal{D}^{test}$.

Figure 2: Illustrates the process of how an image and its associated label are
processed using the CLIP model with an adaptable prompt to generate visual
embedding $F_{v}$ along with a series of textual embeddings
$F_{t}^{1},F_{t}^{2},\ldots$ Visual memories consist of learnable features
that represent basic visual elements and their interconnections, while text
memories capture the semantic essence of these elements. The Modern Hopfield
Network then selects a matching subset of visual and textual features based on
$F_{v}$. Subsequently, the Soft Mixture of Experts model uses $F_{v}$ and
these retrieved representations to produce a refined composite feature
$F_{v}^{c}$. The similarity between $F_{v}^{c}$ and the series of $F_{t}$
features is then used to compute cross-entropy loss. Additionally, the model
incorporates Soft Prompt Loss and Clustering & Retrieval Loss to refine its
understanding of compositional representation and to ensure the relevance of
the retrieval process.

##  4 Our Approach

Most recent research has primarily treated Compositional Zero-shot Learning
(CZSL) as a multi-classification task, focusing on learning visual primitives
separately and then combining them later on. However, this approach often runs
into challenges due to the tight interweaving of image features, leading to a
possible loss of their joint representation, which makes CZSL a challenging
task.

In contrast, our proposed approach uses modern Hopfield networks as a
retrieval mechanism to extract visual prototypes from a learnable memory,
preserving their joint representations. Moreover, we use Transformers [41]
alongside Soft Mixture of Experts [34] to use and integrate this knowledge by
retrieving it from the memory. Additionally, our model is trained using a
hybrid approach where the memory contrastive loss is incorporated after
finishing an epoch with it.

Specifically, our proposed model (HOMOE) consists of three primary steps: (1)
data encoding, (2) retrieval using the modern Hopfield network, and (3)
prototype composition. The framework of HOMOE is depicted in Fig 2. More
detailed explanation of our approach is provided in the following sections.

###  4.1 HOMOE

Data encoding. In the training phase, we deal with an image denoted by $x_{i}$
and a collection of labels $C_{i}$, both of which are sampled from the
training dataset $\mathcal{D}^{tr}$. Our approach involves encoding both the
image and labels using CLIP (Contrastive Language–Image Pre-training, a model
designed to learn visual concepts from natural language descriptions), which
consists of an image encoder $E_{img}(x)$ and a text encoder $E_{text}(y)$,
where $x$ symbolizes an image, and $y$ denotes a sentence .

Initially, the image is directly fed into the image encoder, yielding latent
vectors $F_{v}=E_{img}(x_{i})$. On the other hand, when it comes to text
processing, we use the soft prompt strategy from DFSP (Decomposed Fusion with
Soft Prompt, a technique for enhancing language models by introducing soft
prompts) [23] to achieve enhanced linguistic joint representations.
Specifically, DFSP transforms labels $C_{i}=(a^{i},o^{i})$ into a soft prompt
$P_{\theta}(C_{i})=$ [$\theta_{1}$] [$\theta_{2}$] [$\theta_{3}$] [$a^{i}$]
[$o^{i}$], with [$\theta_{1}$] [$\theta_{2}$] [$\theta_{3}$] being fully
learnable prompts (soft prompts). Following this, the transformed labels are
tokenized and sent through the static CLIP encoder, resulting in
$F_{t}=E_{text}(P_{\theta}(C_{i}))$.

Retrieving with modern Hopfield Network Transformers have become common in
several fields such as natural language processing, computer vision, and
reinforcement learning, owing to their outstanding capacity to extract
relevant information using multi-head self-attention mechanisms. In the
context of CZSL, our objective is to retrieve key visual prototypes while
maintaining a generalized visual representation as memory. To achieve this, we
turn to the Modern Hopfield Network, which enables us to retrieve
representations of other labels and keep updating these representations
continually. Going into the details, we start with two sets of memory, $M_{a}$
and $M_{o}$, where $M_{a},M_{o}\in\mathbb{R}^{|C_{tr}|\times D}$, representing
the memory of attributes and objects with a latent dimension $D$,
respectively. We encode $M_{a}$ and $M_{o}$ by determining the mean of visual
and linguistic prototypes for each label. For instance, considering the label
“red tomato,” the linguistic prototype is obtained from the output of the
frozen CLIP’s text encoder with a prompt: “A photo of a red object,” while the
visual prototype is the mean of several image embeddings, derived from the
output of the frozen CLIP’s image encoder concerning several images labeled
“red tomato”. The rationale behind encoding two distinct sets of features is
to capture the interplay of primitives with another primitive. Rather than
representing just an attribute or object, we construct $M_{a}$ and $M_{o}$,
ensuring they match the size with the training set classes, so that we can
store noticeable variations of primitives for improved compositions. For a
Hopfield Network to retrieve $l$ different patterns, we initially use a
trainable linear layer to project $F_{v}$ into $l$ vectors, denoted by
$Z_{l}=Z_{\theta}(F_{v})$. Subsequently, the modern Hopfield network retrieves
akin patterns, $V=[V_{1},V_{2},...,V_{l}]$, from both $M_{a}$ and $M_{o}$,
with each set containing $l/2$ patterns, respectively. This equation is
adapted from the modern Hopfield Network [36].

$V_{i}=\left\\{\begin{matrix}softmax(Z_{i}\cdot M_{a}^{T})\cdot
M_{a},&i<l/2\\\ softmax(Z_{i}\cdot M_{o}^{T})\cdot M_{o},&i\geq
l/2\end{matrix}\right.$ (1)

In this equation, $s_{i}=softmax(Z_{i}\cdot M_{a})$ can be viewed as the
probability of the vector correlating to the memory. To ensure the Modern
Hopfield Network retrieves $[V_{1},V_{2},\ldots,V_{n}]$ as meaningful
information, we incorporate two auxiliary losses to realize this goal.
Elaborating further, we follow the retrieval equation to compute the cross-
entropy loss with the labels $(a_{t},o_{t})$ of the input image.

$\displaystyle\begin{split}L_{re}(s)=&\sum_{i=1}^{|C_{tr}|}I(a_{t})_{i}log\left[\frac{1}{l/2}\sum_{j=1}^{l/2}s_{j,i}\right]\\\
+&\sum_{i=1}^{|C_{tr}|}I(o_{t})_{i}log\left[\frac{1}{l/2}\sum_{j=l/2}^{l}s_{j,i}\right]\\\
&\text{with $I()$ is the one-hot-encoder.}\end{split}$ (2)

Additionally, we compute the InfoNCE loss (as outlined in [9]) on the Modern
Hopfield model’s attribute output and object output, respectively.

For a more comprehensive understanding, our initial step consists of
generating all possible combinations of positive sets, denoted as $V^{+}$,
from the set V. We select pairs of vectors $(V_{i},V_{j})$ such that
$arg(s_{i})=arg(s_{j})$. For each of these positive sets, we proceed to
calculate the InfoNCE loss using the subsequent equation:

$L_{N}=-\sum_{i=0}^{l}\log\frac{\exp(V^{+}_{i,1}\cdot
V^{+}_{i,2}/\tau)}{\sum_{j=0}^{l}\exp(V_{i,1}\cdot V_{j}/\tau)}~{},$ (3)

where $V^{+}_{i,1}$ is the first element from the $i$-th set and $\tau$ is a
temperature hyper-parameter as per [43]. To avoid the scenario of positive
sets being empty, we randomly include one element from $M$ while forming the
$V^{+}$.

The underlying idea of this loss is to promote the clustering of similar
attributes and objects in memory, given that the Modern Hopfield Network
identifies the k-nearest neighbors to the input image, and effective
clustering boosts the relevance of the retrieval mechanism.

Prototypes aggregation. The significant variations in the relationships
between states and objects, which may highly depend on the specific state and
object in question, act as a strong driving force for our approach. When
combining different attributes with an object, it is essential to customize
the composition method accordingly. Given this incentive, we have adopted the
Soft Mixture of Experts (Soft MoE)[34] as the compositional composer in our
model. This choice allows us to infuse diversity into the model’s capacity to
handle a broad spectrum of patterns and complexities present in the data.

The Soft MoE approach employs a differentiable routing algorithm, which relies
on adjustable parameters for each slot to compute the importance scores for
pairs of input tokens and slots. Following this, each slot uses these scores
to perform a weighted summation of the input tokens. Each ”expert” is
essentially a unique Multi-Layer Perceptron (MLP) layer tasked with processing
its assigned slot (in our experiments, we allocated one slot per expert).
Ultimately, the initial importance scores are also used to combine the outputs
from all the slots. In alignment with their work, we also incorporate
Transformers [41] and substitute the second half of the MLP blocks with Soft
MoE layers as the composer for our model.

Given an input image embedding denoted as $F_{v}$ and a set of retrieved
visual embeddings, namely,$V_{1},V_{2},\ldots,V_{n}$, we also acquire the
corresponding linguistic embeddings, labeled as $T_{1},T_{2},\ldots,T_{n}$.
For instance, when we input an image of a “red car” into the Hopfield Network,
the network produces a set of prototypes like “red tomato,” “old car,” and so
forth. The textual retrieval, in this scenario, would include words like
“red,” “old,” “car,” “tomato,” and others. We opt to keep these retrieved
embeddings as representations of individual primitives because they are easier
to combine compared to representing the entire composition, such as “red
tomato” and “old car”. Then, we concatenate the input image embeddings with
the retrieved embeddings to form $[F_{v},V_{1},V_{2},\ldots
V_{n},T_{1},T_{2},\ldots T_{n}]$ and feed it forward to the Soft MoE. The
output would be $[F_{v}^{\prime},V_{1}^{\prime},V_{2}^{\prime},\ldots
V_{n}^{\prime},T_{1}^{\prime},T_{2}^{\prime},\ldots T_{n}^{\prime}]$. The
final input embeddings will be the combination of $F_{v}$ and $F_{v^{\prime}}$
according to

$F_{v}^{c}=F_{v}*w+(1-w)*F_{v}^{\prime}\qquad\text{with}\ 0<w<1~{}.$ (4)

####  4.1.1 Loss function

The class probability is computed as:

$p_{\text{spm}}(y=(s,o)|x;\theta)=\frac{\exp(F_{v}\cdot
F_{t})}{\sum_{(\bar{s},\bar{o})\in C_{s}}\exp(F_{v}\cdot F_{t})}~{}.$ (5)

To minimize the cross-entropy loss in the soft prompt module,
$L_{\text{spm}}$, we use the formula:

$L_{\text{spm}}=-\frac{1}{|C_{s}|}\sum_{(x,y)\in
C_{s}}\log\left(p_{\text{spm}}(y=(s,o)|x;\theta)\right)~{}.\vspace{-.3cm}$ (6)

To ensure that the soft prompt’s representation mirrors the joint
representation of primitives, we apply the decomposition technique from DFSP
[23] and compute the cross-entropy loss on the decomposed features. Let
$F_{ts}$ and $F_{to}$ represent the decomposed attribute and object
representations from the text embeddings, respectively. DFSP defines the class
probability as:

$\displaystyle p(\frac{y=s}{x,\theta})=\frac{\exp(F_{v}^{c}\cdot
F_{ts})}{\sum_{(\bar{s})\in A}\exp(F_{v}^{c}\cdot F_{ts})}$ (7) $\displaystyle
p(\frac{y=o}{x,\theta})=\frac{\exp(F_{v}^{c}\cdot F_{to})}{\sum_{(\bar{o})\in
O}\exp(F_{v}^{c}\cdot F_{to})}~{}.$ (8)

The cross-entropy loss is then computed using:

$\displaystyle L_{decompose}=-\frac{1}{|A|}\sum_{x,y\in
C^{s}}log(p(\frac{y=s}{x;\theta}))$ (9)
$\displaystyle-\frac{1}{|O|}\sum_{x,y\in C^{s}}log(p\frac{y=o}{x;\theta}))$
(10) $p_{\text{c}}(y=(s,o)|x;\theta)=\frac{\exp(F_{v}^{c}\cdot
F_{t})}{\sum_{(\bar{s},\bar{o})\in C_{s}}\exp(F_{v}^{c}\cdot F_{t})}~{}.$ (11)

Finally, the cross-entropy loss in the soft prompt module is minimized:

$L_{\text{compose}}=-\frac{1}{|C_{s}|}\sum_{(x,y)\in
C_{s}}\log\left(p_{\text{c}}(y=(s,o)|x;\theta)\right)$ (12)

In conclusion, our final loss is computed as the sum of its parts, where
$0<\alpha,\beta,\gamma<1$

$\displaystyle L=$ $\displaystyle L_{compose}+\alpha L_{decompose}+\beta
L_{spm}+$ $\displaystyle\gamma(L_{retrieval}+L_{clustering})~{}.$ (13)

##  5 Experiment

Method | MIT-States | UT-Zappos | C-GQA
---|---|---|---
S | U | H | AUC | S | U | H | AUC | S | U | H | AUC
AoP [31] | 14.3 | 17.4 | 9.9 | 1.6 | 59.8 | 54.2 | 40.8 | 25.9 | 17.0 | 5.6 | 5.9 | 0.7
LE+ [30] | 15.0 | 20.1 | 10.7 | 2.0 | 53.0 | 61.9 | 41.0 | 25.7 | 18.1 | 5.6 | 6.1 | 0.8
TMN [35] | 20.2 | 20.1 | 13.0 | 2.9 | 58.7 | 60.0 | 45.0 | 29.3 | 23.1 | 6.5 | 7.5 | 1.1
SymNet [20] | 24.2 | 25.2 | 16.1 | 3.0 | 49.8 | 57.4 | 40.4 | 23.4 | 26.8 | 10.3 | 11.0 | 2.1
CompCos [25] | 25.3 | 24.6 | 16.4 | 4.5 | 59.8 | 62.5 | 43.1 | 28.1 | 28.1 | 11.2 | 12.4 | 2.6
CGE [30] | 31.1 | 5.8 | 6.4 | 1.1 | 62.0 | 44.3 | 40.3 | 23.1 | 32.1 | 2.0 | 3.4 | 0.5
Co-CGE [26] | 31.1 | 5.8 | 6.4 | 1.1 | 62.0 | 44.3 | 40.3 | 23.1 | 32.1 | 2.0 | 3.4 | 0.5
SCEN [18] | 29.9 | 25.2 | 18.4 | 5.3 | 63.5 | 63.1 | 47.8 | 32.0 | 28.9 | 25.4 | 17.5 | 5.5
CSP [33] | 46.6 | 49.9 | 36.3 | 19.4 | 64.2 | 66.2 | 46.6 | 33.0 | 28.8 | 26.8 | 20.5 | 6.2
DFSP [23] | 46.9 | 52.0 | 37.3 | 20.6 | 66.7 | 71.7 | 47.2 | 36.0 | 38.2 | 32.0 | 27.1 | 10.5
HOMOE | 50.5 | 54.6 | 39.9 | 23.3 | 68.4 | 73.9 | 49.1 | 37.5 | 35.8 | 30.8 | 24.5 | 9.1
Table 1:  Closed World Evaluation. Comparison to state-of-the-art models
Figure 3: The visualization of attribute memory embeddings in different memory
configurations and datasets

###  5.1 Experiment Setup

Datasets. Our experiments are conducted on three real-world, challenging
benchmark datasets: MIT-States, UTZappos, and C-GQA. The MIT-States dataset is
frequently used in CZSL research, containing 115 states, 245 objects, and 1962
compositions, and has been embraced by many previous methods. UTZappos
includes 50025 images of shoes, featuring 16 states and 12 objects. On the
other hand, C-GQA is a newly introduced, large-scale benchmark derived from
in-the-wild images, offering more generalized compositions. Being the most
popular dataset for CZSL, C-GQA contains 453 states, 870 objects, and a total
of 39298 images, which include over 9500 compositions.

Metrics. Following the framework in [25], we evaluate prediction accuracy
considering both seen and unseen compositions under both closed-world and
open-world scenarios. Specifically, “Seen” (S) denotes accuracy evaluated
solely on known compositions, while “Unseen” (U) denotes accuracy evaluated
exclusively on unknown compositions. Moreover, we calculate the Harmonic Mean
(HM) of the S and U metrics. Given the tendency of zero-shot models to favor
known compositions, we generate a seen-unseen accuracy curve at various
operating points, ranging from a bias of negative infinity to positive
infinity, and determine the Area Under the Curve (AUC). In summary, our
evaluation metrics encompass S, U, HM, and AUC.

Implementation Details. Our HOMOE system is implemented using PyTorch version
1.12.1 and is optimized using the Adam optimizer over 10 epochs. The image and
text encoders within this model are both constructed on top of the pre-trained
CLIP Vit-L/14 architecture. Furthermore, we have configured the Soft MoE with
2 layers. The complete model was trained and evaluated on a single NVIDIA RTX
3090 GPU.

###  5.2 Comparison

We present experimental comparisons in Table 1 and Table 2 with previously
established compositional zero-shot learning methods, including AoP [31], LE+
[30], TMN [35], SymNet [20], CompCos [25], CGE [30], Co-CGE [26], SCEN [18],
KG-SP [11], CSP [33] and DFSP [23]. The performance is evaluated in both
closed-world and open-world scenarios. In the closed-world scenario, our HOMOE
model sets a new benchmark on the MIT-States and UT-Zappos datasets,
displaying superior performance over the recent state-of-the-art method DFSP
[23]. Specifically, on the MIT-States dataset, HOMOE surpasses DFSP [20] by
increasing the unseen accuracy by 2.6%, seen accuracy by 3.6%, harmonic mean
by 2.6%, and AUC by 2.7%. On the UT-Zappos dataset, HOMOE exhibits an
improvement of 1.7%, 2.2%, 1.9%, and 1.5% on seen accuracy, unseen accuracy,
harmonic mean, and AUC, respectively, compared to DFSP [23]. Regarding the
C-GQA dataset, the performance of HOMOE provides insightful data, prompting a
detailed ablation study to better understand the dynamics, as elaborated in
the subsequent section.

Method | MIT-States | UT-Zappos | C-GQA
---|---|---|---
S | U | H | AUC | S | U | H | AUC | S | U | H | AUC
AoP [31] | 16.6 | 5.7 | 4.7 | 0.7 | 50.9 | 34.2 | 29.4 | 13.7 | - | - | - | -
LE+ [30] | 14.2 | 2.5 | 2.7 | 0.3 | 60.4 | 36.5 | 30.5 | 16.3 | 19.2 | 0.7 | 1.0 | 0.08
TMN [33] | 12.6 | 0.9 | 1.2 | 0.1 | 55.9 | 18.1 | 21.7 | 8.4 | - | - | - | -
SymNet [20] | 21.4 | 7.0 | 5.8 | 0.8 | 53.3 | 44.6 | 34.5 | 18.5 | 26.7 | 2.2 | 3.3 | 0.43
CompCos [25] | 25.4 | 10.0 | 8.9 | 1.6 | 59.3 | 46.8 | 36.9 | 21.3 | - | - | - | -
CGE [30] | 32.4 | 5.1 | 6.0 | 1.0 | 61.7 | 47.7 | 39.0 | 23.1 | 32.7 | 1.8 | 2.9 | 0.47
Co-CGEˆClosed [26] | 31.1 | 5.8 | 6.4 | 1.1 | 62.0 | 44.3 | 40.3 | 23.1 | 32.1 | 2.0 | 3.4 | 0.53
Co-CGEˆOpen [26] | 30.3 | 11.2 | 10.7 | 2.3 | 61.2 | 45.8 | 40.8 | 23.3 | 32.1 | 3.0 | 4.8 | 0.78
KG-SP [11] | 28.4 | 7.5 | 7.4 | 1.3 | 61.8 | 52.1 | 42.3 | 26.5 | 31.5 | 2.9 | 4.7 | 0.78
DRANet [19] | 29.8 | 7.8 | 7.9 | 1.5 | 65.1 | 54.3 | 44.0 | 28.8 | 31.3 | 3.9 | 6.0 | 1.05
CSP [33] | 46.3 | 15.7 | 17.4 | 5.7 | 64.1 | 44.1 | 38.9 | 22.7 | 28.7 | 5.2 | 6.9 | 1.2
DFSP [23] | 47.5 | 18.5 | 19.3 | 5.8 | 66.8 | 60.0 | 44.0 | 30.3 | 38.3 | 7.2 | 10.4 | 2.4
HOMOE | 50.4 | 19.7 | 20.7 | 7.9 | 68.4 | 61.9 | 45.1 | 31.1 | 35.7 | 6.6 | 9.0 | 2.0
Table 2:  Open World Evaluation. Comparison to state-of-the-art models

###  5.3 Ablation Study

Our method shows optimal performance on the MIT-States dataset, prompting us
to focus our ablation studies there to identify the key contributing factors
to its success and understand how these can be applied to improve results on
other datasets.

####  5.3.1 Memory Updates

When considering the memory update mechanism during training, its
configuration plays a pivotal role in the model’s overall effectiveness.
Specifically, while our method excels with the MIT-States and UT-Zappos
datasets, it struggles with the C-GQA dataset. T-SNE visualizations in Figure
3 of the memory highlight that the C-GQA dataset’s memory structure is not as
well organized, mainly because many classes are represented by only a few
images. This limited representation fails to capture a comprehensive class
essence, which is why our method underperforms on the C-GQA dataset.

####  5.3.2 Contrastive Loss Impacts

The contrastive loss incorporation presents a distinct training dynamic.
Without it, the model tends to classify input images using retrieved features
as auxiliary support. Introducing contrastive loss shifts the focus towards
compositional learning, significantly boosting performance on unseen
combinations at the expense of seen ones. A balanced training strategy that
combines both approaches yields the best results. Figure 3 also shows that
contrastive loss also helps maintain feature clusters, preventing the model
from updating only those features derived directly from the CLIP loss.

Method | MIT-States
---|---
| S | U | H | AUC
without InfoNCE loss | 47.5 | 52.4 | 37.8 | 20.9
with InfoNCE loss | 45.5 | 55.6 | 36.5 | 19.8
Table 3:  Closed world evaluation on different memory configurations

It is important to recognize the contrastive loss’s sensitivity to
hyperparameters, especially batch size, due to the InfoNCE loss being computed
across image batches. Intuitively, a larger batch allows for more extensive
feature updates, potentially enhancing memory feature clustering. We believe
that better performance across all datasets could be achieved by increasing
the batch size. However, such adjustments would require computational
resources that exceed our current budget.

Batch size | MIT-States
---|---
| S | U | H | AUC
8 | 49.5 | 52.6 | 38.7 | 21.9
16 | 49.7 | 54.1 | 39.5 | 22.8
32 | 50.5 | 54.6 | 39.9 | 23.3
64 | 50.1 | 55.3 | 39.9 | 23.5
Table 4:  Closed world evaluation result of different batch size after 10
training epochs

####  5.3.3 Analysis of Expert Allocation in Soft MoE

Figure 4: Visualization of Soft Mixture of Experts. Each ‘expert’ focuses on
distinct categories of data – Expert 3 on food items, Expert 4 on apparel,
etc.– demonstrating the model’s ability to assign and weigh inputs across
different neural network sub-models for enhanced specialization and accuracy
in classification tasks

Our analysis examines how the Soft Mixture of Experts (Soft MoE) distributes
inputs based on their attributes or objects. By contrasting this with a cross-
attention approach, we find that implementing Soft MoE enhances unseen
accuracy by 1%. Visual inspection in Fig 4 reveals that the Soft MoE tends to
assign images with similar attributes or objects to the same experts,
indicating a degree of efficient routing. Yet, this pattern is confined to a
few experts, with many experts still handling a mix of unrelated inputs. This
selective engagement of experts is noteworthy, as it implies that the model
does not engage all patterns or features equally. Instead, it adopts a
tailored strategy for image recognition, focusing on distinctive features to
potentially improve its performance on new, unseen data.

##  6 Conclusion

In this paper, we propose HOMOE, a novel framework to Compositional Zero-Shot
Learning (CZSL) that mimics human adaptability to new state-object
combinations. By integrating the Modern Hopfield Network with a Mixture of
Experts, our framework effectively recalls and assembles relevant primitives
into novel compositions. Additional loss functions are proposed to ensure that
each part of the system contributes optimally to information recall and
association. Our extensive evaluations and detailed analyses establish the
superior performance of HOMOE across various standard datasets compared to
SOTA and provide insights into its limitations with the C-GQA dataset.

## References

  * Chen et al. [2023] Zhuo Chen, Yufeng Huang, Jiaoyan Chen, Yuxia Geng, Wen Zhang, Yin Fang, Jeff Z Pan, and Huajun Chen.  Duet: Cross-modal semantic grounding for contrastive zero-shot learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 405–413, 2023.
  * Crossley et al. [2023] Michael Crossley, Paul R Benjamin, György Kemenes, Kevin Staras, and Ildikó Kemenes.  A circuit mechanism linking past and future learning through shifts in perception.  _Science Advances_ , 9(12):eadd3403, 2023.
  * Demircigil et al. [2017] Mete Demircigil, Judith Heusel, Matthias Löwe, Sven Upgang, and Franck Vermet.  On a model of associative memory with huge storage capacity.  _Journal of Statistical Physics_ , 168:288–299, 2017.
  * Fedus et al. [2022] William Fedus, Barret Zoph, and Noam Shazeer.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.  _The Journal of Machine Learning Research_ , 23(1):5232–5270, 2022.
  * Gross et al. [2017] Sam Gross, Marc’Aurelio Ranzato, and Arthur Szlam.  Hard mixtures of experts for large scale weakly supervised vision.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6865–6873, 2017.
  * Guo and Guo [2020] Jingcai Guo and Song Guo.  A novel perspective to zero-shot learning: Towards an alignment of manifold structures via semantic feature expansion.  _IEEE Transactions on Multimedia_ , 23:524–537, 2020.
  * Guo et al. [2023] Jingcai Guo, Song Guo, Qihua Zhou, Ziming Liu, Xiaocheng Lu, and Fushuo Huo.  Graph knows unknowns: Reformulate zero-shot learning as sample-level graph recognition.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 7775–7783, 2023.
  * Hao et al. [2023] Shaozhe Hao, Kai Han, and Kwan-Yee K Wong.  Learning attention as disentangler for compositional zero-shot learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15315–15324, 2023.
  * He et al. [2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9729–9738, 2020.
  * Hopfield [1982] John J Hopfield.  Neural networks and physical systems with emergent collective computational abilities.  _Proceedings of the national academy of sciences_ , 79(8):2554–2558, 1982.
  * Karthik et al. [2022] Shyamgopal Karthik, Massimiliano Mancini, and Zeynep Akata.  Kg-sp: Knowledge guided simple primitives for open world compositional zero-shot learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9336–9345, 2022.
  * Kim et al. [2023] Hanjae Kim, Jiyoung Lee, Seongheon Park, and Kwanghoon Sohn.  Hierarchical visual primitive experts for compositional zero-shot learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5675–5685, 2023.
  * Krotov and Hopfield [2016] Dmitry Krotov and John J Hopfield.  Dense associative memory for pattern recognition.  _Advances in neural information processing systems_ , 29, 2016.
  * Lampert et al. [2013] Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling.  Attribute-based classification for zero-shot visual object categorization.  _IEEE transactions on pattern analysis and machine intelligence_ , 36(3):453–465, 2013.
  * Lepikhin et al. [2020] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen.  Gshard: Scaling giant models with conditional computation and automatic sharding.  _arXiv preprint arXiv:2006.16668_ , 2020.
  * Lewis et al. [2021] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer.  Base layers: Simplifying training of large, sparse models.  In _International Conference on Machine Learning_ , pages 6265–6274. PMLR, 2021.
  * Li et al. [2021] Xiangyu Li, Zhe Xu, Kun Wei, and Cheng Deng.  Generalized zero-shot learning via disentangled representation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 1966–1974, 2021.
  * Li et al. [2022] Xiangyu Li, Xu Yang, Kun Wei, Cheng Deng, and Muli Yang.  Siamese contrastive embedding network for compositional zero-shot learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9326–9335, 2022.
  * Li et al. [2023] Yun Li, Zhe Liu, Saurav Jha, and Lina Yao.  Distilled reverse attention network for open-world compositional zero-shot learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1782–1791, 2023.
  * Li et al. [2020] Yong-Lu Li, Yue Xu, Xiaohan Mao, and Cewu Lu.  Symmetry and group in attribute-object compositions. 2020 ieee.  In _CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 11313–11322, 2020.
  * Liu et al. [2021] Yang Liu, Lei Zhou, Xiao Bai, Yifei Huang, Lin Gu, Jun Zhou, and Tatsuya Harada.  Goal-oriented gaze estimation for zero-shot learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3794–3803, 2021.
  * Liu et al. [2022] Ziming Liu, Song Guo, Jingcai Guo, Yuanyuan Xu, and Fushuo Huo.  Towards unbiased multi-label zero-shot learning with pyramid and semantic attention.  _IEEE Transactions on Multimedia_ , 2022.
  * Lu et al. [2023] Xiaocheng Lu, Song Guo, Ziming Liu, and Jingcai Guo.  Decomposed soft prompt guided fusion enhancing for compositional zero-shot learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 23560–23569, 2023.
  * Lucibello and Mézard [2023] Carlo Lucibello and Marc Mézard.  The exponential capacity of dense associative memories.  _arXiv preprint arXiv:2304.14964_ , 2023.
  * Mancini et al. [2021] Massimiliano Mancini, Muhammad Ferjad Naeem, Yongqin Xian, and Zeynep Akata.  Open world compositional zero-shot learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 5222–5230, 2021.
  * Mancini et al. [2022] Massimiliano Mancini, Muhammad Ferjad Naeem, Yongqin Xian, and Zeynep Akata.  Learning graph embeddings for open world compositional zero-shot learning.  _IEEE Transactions on pattern analysis and machine intelligence_ , 2022.
  * Mikolov et al. [2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  _Advances in neural information processing systems_ , 26, 2013.
  * Misra et al. [2017] Ishan Misra, Abhinav Gupta, and Martial Hebert.  From red wine to red tomato: Composition with context.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1792–1801, 2017.
  * Mustafa et al. [2022] Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby.  Multimodal contrastive learning with limoe: the language-image mixture of experts.  _Advances in Neural Information Processing Systems_ , 35:9564–9576, 2022.
  * Naeem et al. [2021] Muhammad Ferjad Naeem, Yongqin Xian, Federico Tombari, and Zeynep Akata.  Learning graph embeddings for compositional zero-shot learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 953–962, 2021.
  * Nagarajan and Grauman [2018] Tushar Nagarajan and Kristen Grauman.  Attributes as operators: factorizing unseen attribute-object compositions.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 169–185, 2018.
  * Nan et al. [2019] Zhixiong Nan, Yang Liu, Nanning Zheng, and Song-Chun Zhu.  Recognizing unseen attribute-object pair with generative model.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 8811–8818, 2019.
  * Nayak et al. [2022] Nihal V Nayak, Peilin Yu, and Stephen Bach.  Learning to compose soft prompts for compositional zero-shot learning.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Puigcerver et al. [2023] Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby.  From sparse to soft mixtures of experts.  _arXiv preprint arXiv:2308.00951_ , 2023.
  * Purushwalkam et al. [2019] Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, and Marc’Aurelio Ranzato.  Task-driven modular networks for zero-shot compositional learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3593–3602, 2019.
  * Ramsauer et al. [2020] Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, et al.  Hopfield networks is all you need.  _arXiv preprint arXiv:2008.02217_ , 2020.
  * Riquelme et al. [2021] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby.  Scaling vision with sparse mixture of experts.  _Advances in Neural Information Processing Systems_ , 34:8583–8595, 2021.
  * Ruis et al. [2021] Frank Ruis, Gertjan Burghouts, and Doina Bucur.  Independent prototype propagation for zero-shot compositionality.  _Advances in Neural Information Processing Systems_ , 34:10641–10653, 2021.
  * Saini et al. [2022] Nirat Saini, Khoi Pham, and Abhinav Shrivastava.  Disentangling visual embeddings for attributes and objects.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13658–13667, 2022.
  * Shazeer et al. [2017] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean.  Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.  _arXiv preprint arXiv:1701.06538_ , 2017.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2023] Qingsheng Wang, Lingqiao Liu, Chenchen Jing, Hao Chen, Guoqiang Liang, Peng Wang, and Chunhua Shen.  Learning conditional attributes for compositional zero-shot learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11197–11206, 2023.
  * Wu et al. [2018] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3733–3742, 2018.
  * Zhang et al. [2022] Tian Zhang, Kongming Liang, Ruoyi Du, Xian Sun, Zhanyu Ma, and Jun Guo.  Learning invariant visual representations for compositional zero-shot learning.  In _European Conference on Computer Vision_ , pages 339–355. Springer, 2022.