# Domain-Aware Continual Zero-Shot Learning

Kai Yi  Mohamed Elhoseiny
King Abdullah University of Science and Technology (KAUST)  Saudi Arabia
{kai.yi<EMAIL_ADDRESS>

###### Abstract

We introduce Domain Aware Continual Zero-Shot Learning (DACZSL), the task of
visually recognizing images of unseen categories in unseen domains
sequentially. We created DACZSL on top of the DomainNet dataset by dividing it
into a sequence of tasks, where classes are incrementally provided on seen
domains during training and evaluation is conducted on unseen domains for both
seen and unseen classes. We also proposed a novel Domain-Invariant CZSL
Network (DIN), which outperforms state-of-the-art baseline models that we
adapted to DACZSL setting. We adopt a structure-based approach to alleviate
forgetting knowledge from previous tasks with a small per-task private network
in addition to a global shared network. To encourage the private network to
capture the domain and task-specific representation, we train our model with a
novel adversarial knowledge disentanglement setting to make our global network
task-invariant and domain-invariant over all the tasks. Our method also learns
a class-wise learnable prompt to obtain better class-level text
representation, which is used to represent side information to enable zero-
shot prediction of future unseen classes. Our code and benchmarks will be made
publicly available.

##  1 Introduction

Humans are capable of recognizing unseen visual concepts continually over time
while telling them apart from seen ones. Here comes the question: _can we
build an intelligent agent that can learn and perceive extreme out-of-
distribution data sequentially?_ More specifically, suppose we train a machine
learning model from real images, as shown in Fig. 1 bottom part, can we then
adapt the trained model to recognize painting and sketch images from unseen
categories? Further, can we continually recognize quickdraw images from
different classes?

Three skills are at the heart of our problem, which are Zero-Shot Learning
(ZSL), Domain Generalization (DG), and Continual Learning (CL). ZSL focuses on
recognizing the unseen categories with a model trained on seen classes only
[9, 8, 15]. Different from ZSL, DG aims at classifying the seen classes from
an unseen target domain [40, 13, 44]. Along with ZSL and DG, CL especially
pays attention to learning a model that can handle sequential tasks without
forgetting knowledge from previous tasks [4, 20, 1, 6].

Figure 1: Our DACZSL problem: we illustrate here the difference between Zero-
Shot Learning (ZSL), Continual ZSL (CZSL), and our Domain-Aware CZSL (DACZSL).
CZSL extend ZSL to handle sequential tasks while we study domain shift on top
of CZSL.

An intelligent agent is expected to have the ability to perceive out-of-
distribution data well in a life-long learning setting, which is similar to
humans. We propose a generalized and unified transfer learning setting: Domain
Aware Continual Zero-Shot Learning (DACZSL). Several existing works study
mixtures of ZSL, DG, and CL. Concerning the connection of continual learning
and zero-shot learning, Continual Zero-Shot Learning (CZSL) targets
recognizing unseen tasks sequentially [43]. However, compared with DACZSL,
they do not consider the case in which images are from different domains.
Another exciting exploration is the combination of ZSL and DG. The pioneering
work is CuMix [24], introduced what we call Domain-Aware Zero-Shot Learning
(DAZSL), which focuses on recognizing unseen objects in unseen domains. In
contrast to our work, DAZSL does not explore zero-shot transfer in a continual
learning setting or evaluate whether models will forget knowledge learned from
sequential seen tasks. Fig. 1 shows the differences between ZSL, CZSL, and our
proposed DACZSL settings. Understanding general out-of-distribution data in a
life-long setting is at the heart of our work, which is a more realistic
(e.g., distribution shift is ubiquitous throughout the human learning
lifetime) but less understood problem than standard ZSL and CL settings.

To study the DACZSL problem, we designed several baselines based on existing
state-of-the-art methods from similar problems in the literature, including
CZSL and DAZSL. Our preliminary experimental results show that existing
methods do not perform well on DACZSL. We hypothesize this due to three
reasons: a) used text representation techniques (e.g., using Word2Vec
representation [27] pre-trained from Google Corpus) are limited and do not
provide discriminative guidance to recognize very similar classes like
Parakeet auklet and Crested auklet, which may cause collapsed representations
in the word2vec space; b) training a classifier or mapping the image features
into the text class space makes the classification in a challenging scenario
like DACZSL very hard. As we will detail later, a better protocol like an
attribute prototypical mapping network [51] is more effective; c) they are not
specially designed for DACZSL on top of its intrinsic characteristics, which
contains classification from unseen categories and unseen domains
sequentially.

Addressing above mentioned three issues are critical to achieving good
performance in the DACZSL setting. Recent pre-trained models from large-scale
vision-language pairs such as CLIP [35] and ALIGN [17] have shown promising
zero-shot transfer learning ability among data with different distributions.
Inspired by these works, we first adopt Transformer [46]’s approach to
obtaining better text representation. We also use a more intuitive and dynamic
attribute prototypical mapping pipeline instead of training an extra
classifier. This pipeline only considers the vision-language relationships
instead of classifying an image to a particular class. Different from
Attribute Prototype Network [51], we optimize the prototypical network with
contrastive learning, which we denote this learning scheme as contrastive
prototypical classification. We adopt CLIP pre-trained encoders as our
backbone and fine-tune it with contrastive learning. However, generating a
better prompt (i.e., a text description) has been shown to improve zero-shot
performance significantly [56]. Inspired by this, we design novel domain-
invariant class-wise learnable prompts to obtain more representative text
features. To resolve the last issue, we combine all mentioned above along with
an adversarial knowledge disentanglement scheme. We call our method Domain-
Invariant CZSL Network (DIN).

Contributions. Our paper can be summarized as:

a) We propose a novel but realistic transfer learning setting, DACZSL, to
explore the limits of machine understanding of unseen classes in unseen
domains continually. We hope our introduced benchmarks help ease future
research in this direction. b) We designed a novel DACZSL method called DIN
and show through comprehensive experiments that our method can significantly
improve the performance compared with state-of-the-art methods carefully
adapted to DACZSL from related settings. c) Our work is the first to explore
the transfer ability in the scenarios with significant domain gaps and
sequential learning settings of large pre-trained models from vision-language
pairs.

##  2 Related Works

#### (Generalized) Zero-Shot Learning.

Zero-Shot Learning (ZSL) aims at recognizing images with unseen categories
with the model trained on images from seen classes only. More recently, the
computer vision community has widely adopted a more complex but realistic
setting, Generalized Zero-Shot Learning (GZSL). Compared with ZSL, GZSL aims
to recognize images from unseen and seen classes at testing time. The union of
seen and unseen classes makes the classification search space larger and hence
more challenging. Due to inductive bias in the GZSL setting, the model learned
on seen categories tends to classify images of unseen classes to seen ones,
especially when they are similar to seen classes, making models hard to
achieve good accuracy on unseen. Existing ZSL approaches can be classified
into embedding-based methods and generative-based methods. The former is to
learn an alignment between different modalities (i.e., visual and semantic
modalities) directly for classification [21, 43] while the latter focuses on
synthesizing realistic unseen-class images with unseen text features [30, 53]
or generating images that are creatively deviating from seen classes [57, 8,
47]. Our work is at the intersection of these two popular approaches.

#### Domain Generalization.

Domain Generalization (DG) aims at recognizing images from an unseen target
domain with a machine learning model trained on source domains. One particular
branch related to our work is Domain-Invariant Disentanglement. A key
objective in domain generalization is to train a model on source domains and
then evaluate the model on the target domain. Hence, a key question to improve
the model generation ability is _how to disentangle domain-invariant
information from domain-variated parts_ [5, 14, 54, 33]. Ding et al., [5]
proposed to learn a separate domain-variate network for each domain and a
shared domain-invariant network to capture global information shared across
domains. While in [29], a self-supervised learning scheme is applied to obtain
an optimal classifier that minimizes the representation variance over
different domains. Unlike them, we consider disentangling domain-invariant
information not only from different domains but also from variant tasks.

#### Continual Learning.

Continual Learning (CL) is the ability to learn sequential tasks with
distribution bias. This setting is also referred to as lifelong learning. The
critical problem in CL is catastrophic forgetting, which refers to the
significant drop in a learner’s performance when transiting from the trained
task to a new one, which means a particular model with limited memory will
forget the knowledge learned from previous tasks [26, 25]. The most relevant
works to ours are memory-based methods and structure-based methods. Memory-
based methods aim at sampling the most representative samples from previous
tasks and then replay them to avoid forgetting seen classes. Even though the
strategy is straightforward to address catastrophic forgetting, how to
efficiently sample most representative samples is a central focus towards this
direction [22, 3, 4, 50, 20]. Another way to alleviate catastrophic forgetting
is to expand the network to have more task-dependent parameters progressively.
Partial parameters from previous seen tasks will be kept or minimally changed
to achieve good performance on seen classes [37, 39, 52].

###  2.1 Continual Zero-Shot Learning

Early CZSL works focus more on defining the problem and formulating the
setting (e.g., [4, 48, 43].) A-GEM[4] is the first paper considering continual
zero-shot performance. They randomly split CUB and AWA datasets into 20
subsets of classes and then reported the zero-shot accuracy for each task
sequentially. [48] proposed a different CZSL setting, what they called
lifelong ZSL; they first learn from all the seen classes from many datasets
sequentially, which is a continual learning setting. Then all the unseen
classes are evaluated with the learned model. In contrast to [48], [43]
proposed learning sequentially with unlimited searching space, which means the
model needs to recognize unseen classes sequentially. Most sequential methods
follow the extended setting on top of [43] and focus more on achieving better
performance on more extensive feature-level attribute-based datasets [11, 12].
[11] proposed a replay-based VAE to avoid catastrophic forgetting and led to
good performance. Further, [12] used stacked VAEs with the generative replay
of seen samples.

###  2.2 ZSL with Large-Scale Pretraining

In typical ZSL, image features are extracted using ImageNet-1K [36] pre-
trained ResNet101 [16] in attribute-based datasets and VGG [42] in text-based
datasets. The corresponding semantic class features are processed with Glove
[32] or Skip-Gram [28]. However, this design is highly limited as low-
dimensional text features from naive Glove or Skip-Gram are hard to make
classes distinguishable in ZSL. Besides, as ZSL is a vision-language alignment
problem, training visual and text encoders alone can not capture vision-
language relationships well. Recently, pre-trained models from large vision
and language pairs such as CLIP [34] and ALIGN [17] have shown promising
transfer learning ability on out-of-distribution data. Unlike standard ZSL,
CLIP [34] collaboratively pre-train the visual encoder and the Transformer
[46] text encoder on over 400 million (vision & language) pairs. Following
works focused on fining-tuning the CLIP model to make it adaptive to more
scenarios [49, 56]. WiseFT [49] explored balancing in-distribution and out-
distribution performance by interpolating the fine-tuning weights with zero-
shot weights. In CoOp [56], a learnable prompt strategy is applied to have
better semantic representations. Our method uses CLIP pre-trained weights as
the backbone and follows the contrastive prototypical classification design,
optimizing attribute prototypical network [51] with contrastive learning.

##  3 Problem Definition

We formulate DACZSL and its related settings in this section. We first recap
Domain-Aware ZSL, first proposed in CuMix [24]. Then we extend this setting
and define a more general one called Generalized Domain-Aware ZSL, in which we
need to classify images from both unseen categories and seen ones. Next, we
consider the standard DACZSL; we call uniform DACZSL, where all source domains
can be accessed when training each task. Besides, we also define non-uniform
DACZSL, in which for every task, images from some domains will be randomly
dropped, making the training non-uniform over seen domains.

###  3.1 (Generalized) Domain-Aware ZSL

Suppose we have a set of domains
$\mathcal{D}=\cup_{i=1}^{N^{d}}\mathcal{D}_{i}$, $N^{d}$ is the total number
of domains. We define target domain $\mathcal{D}^{t}$ and its corresponding
source domains $\mathcal{D}^{s}=\mathcal{D}\setminus\mathcal{D}^{t}$. DAZSL
[24] is a two-tasks problem with one training step and another testing step.
In training, we can access to
$\mathcal{H}_{s}^{d}=\\{(X_{s}^{d},Y_{s}^{d},A_{s})\\},d\in{\mathcal{D}^{s}}$,
where $X_{s}^{d},Y_{s}^{d},A_{s}$ are the training image set, their
corresponding label set, and the semantic description set. Noted that the
semantic description is orthogonal to domains. In the testing step,
$X_{u}^{d},d\in\mathcal{D}_{i}$ will be provided. We need to learn a mapping
function $f:{X}_{u}^{d}\to{Y}_{u}$, where ${Y}_{u}$ is the unseen classes
label set. In contrast to DAZSL [24], we have a much larger prediction space
covering both seen and unseen classes in Generalized DAZSL. More concretely,
we need to learn a mapping function $g:{X}_{u}^{d}\to{Y}_{u}\cup{Y}_{s}$,
which is more challenging.

Figure 2: Architecture of our proposed method DIN. We first learn an class-
wise prompt by contrastive learning with the output from CLIP pre-trained
Transformer [46] text encoder $\operatorname{PM}(\texttt{CLASS})$ and the
output $z_{G}$ from the global net $G$. Then we conduct adversarial
contrastive learning to make the global network only outputs domain and task-
invariant information.

###  3.2 DACZSL: Domain-Aware Continual Zero-shot Learning

For DACZSL, we aim at recognizing images from unseen categories in unseen
domains sequentially over $T$ tasks. We first define the training set at task
$t\in\\{1\cdots T\\}$ to be
$\mathcal{H}^{d}_{t}=\\{X_{t}^{d},Y_{t}^{d},A_{t}\\}$, where
$X_{t}^{d},Y_{t}^{d},A_{t}$ are the training image set, their corresponding
label set, and the semantic description set at task $t$ from domain $d$,
$d\in\mathcal{D}_{t}$, which are all available source domains.
$\mathcal{D}_{t}\subseteq\mathcal{D}^{s}$. Noticed different variances of
DACZSLs have different space of $\mathcal{D}_{t}$. We proposed two DACZSL
variants, uniform and non-uniform that differs only during training.

Uniform DACZSL. In this setting, $\mathcal{D}_{t}\equiv\mathcal{D}^{s}$.
Meaning, we have uniform access training images from all sources domains
$\mathcal{D}^{s}$ at task $t$.

Non-Uniform DACZSL. This is a more challenging setting, where
$\mathcal{D}_{t}\subseteq\mathcal{D}^{s}\setminus\mathcal{D}_{t}^{\operatorname{rand}}$.
Meaning, we randomly remove $\mathcal{D}_{t}^{\operatorname{rand}}$ from all
source domains $\mathcal{D}^{s}$ at task $t$. Non-Uniform DACZSL is more
difficult as we need to make our model creatively alleviate the bias of two
invisible domains (one is the unseen target domain while another is the
randomly removed source domain).

###  3.3 DACZSL Evaluation Metrics

We present here the evaluation metrics for DACZSL, and refer to the Appendix C
Evaluation Metrics for (G)DAZSL for DAZSL and GDAZSL metrics. Inspired by
standard continual learning literature, we compute the seen accuracy after
training at the last task:

$\operatorname{LS}=\frac{1}{T}\sum_{t=1}^{T}\operatorname{Acc}(X_{t}^{d}|\mathcal{M}_{T}),\quad
d\in\mathcal{D}_{i}$ (1)

where $\mathcal{D}_{i}$ is the target domain and $\mathcal{M}_{t}$ is the
model trained till step $t$. $\operatorname{Acc}(X_{t}^{d}|\mathcal{M}_{T})$
is the classification accuracy on image set $X^{d}_{t}$ with the model trained
till task $T$. We also compute the mean seen and unseen accuracy over tasks:

$\displaystyle\operatorname{mS}$
$\displaystyle=\frac{1}{T}\sum_{t=1}^{T}\sum_{k=1}^{t}\operatorname{Acc}(X_{k}^{d}|\mathcal{M}_{t})$
(2) $\displaystyle\operatorname{mU}$
$\displaystyle=\frac{1}{T-1}\sum_{t=1}^{T-1}\sum_{k=t+1}^{T}\operatorname{Acc}(X_{k}^{d}|\mathcal{M}_{t})$

We consider the harmonic mean mH of mS and mU as a unified measure that over
seen and unseen classes. More concretely, mH is defined as
$2\operatorname{mS}\cdot\operatorname{mU}/(\operatorname{mS}+\operatorname{mU})$.

Besides, we also consider the measure of forgetting rate:

$\operatorname{BWT}=\sum_{t=1}^{T}\operatorname{Acc}(X_{t}^{d}|\mathcal{M}_{T})-\operatorname{Acc}(X_{t}^{d}|\mathcal{M}_{t})$
(3)

##  4 Method

Class-Wise Learnable Prompts. Learning suitable prompt (e.g., “An image of a
cat.”) has been widely explored to obtain better text descriptions in the
natural language processing domain [41, 18, 55, 19], but rarely explored in
multi-modal scenarios such as considering the vision and language modalities.
Considering DACZSL as a vision-language learning problem and inspired by CoOp
[56], we design the class-wise learnable prompt to learn better relationship
between vision and language; see the top-left part in Fig. 2. Unlike CoOp, we
train the learnable prompts over images from different domains, making the
model generalized and can capture domain-invariant class-specific knowledge.
We formulate our prompt for class $c$ in the following form:

$\text{PM}(c)=[W]_{1\to k_{1}}\ \ [\texttt{CLASS}]\ \ [W]_{k_{1}+1\to
k_{2}}[\texttt{EOS}]$ (4)

where $[W]_{i},i\in[1,k_{2}],0\leq k_{1}\leq k_{2}$ is the learnable context
(i.e., the semantic embedding of word $W$). In our experiments, we choose each
of them as a 512-dimensional representation. Here we consider two variants of
where to add the class token, which are to put CLASS in the middle
($k_{2}=2k_{1}$) or at the end of the prompt ($k_{1}+1=k_{2}$, meaning
$[W]_{k_{1}+1\to k_{2}}$ will dispear). We randomly sample
$H_{t}\in\mathcal{H}_{t}$ satisfying
$\|H^{d}_{c}\|=K,c\in\mathcal{S}_{t},H_{c}^{d}\in H_{t}$ is the dataset to
train the class-wise learnable prompt. $K$ is the number of samples per domain
and per class. We input $\text{PM}{(c)}$ to the Text Encoder and use the
global net G to learn this prompt by contrastive learning. Noticed that for
the learning of prompts, we do not consider local nets $L_{i},i\in[1,T]$.
Noticed in Fig. 2, $t(\cdot)=\operatorname{Text(PM(\cdot))}$.

Adversarial Knowledge Disentanglement. For the visual representation, we
consider disentangling distribution-invariant knowledge from the task and
domain-specific information. Suppose we have an image $x_{t}^{d}\in X_{t}^{d}$
from domain $d$ and at task $t$ with label $c$ (see Fig. 2), this image is
processed through Local Nets $L(\cdot)$ and Global Net $G(\cdot)$. $L(\cdot)$
is instantiated for each task $t\in\\{1\cdots T\\}$ and is aimed at learning
task-variant information. Moreover, we encourage the Global Net $G(\cdot)$ to
learn task and domain-invariant knowledge. In practice, we may optionally
store a small set of samples from previous seen tasks111also known as episodic
memory in continual learning literature., denoted as
$S^{d}_{<t},d\in\mathcal{D}^{s}$, $\mathcal{D}^{s}$ represents source domains.
Inspired by [2], we design a disentangled loss to encourage the features in
the global network to be independent with every task-variant local net; see
Eqn. 5.

$\mathcal{L}_{\text{disen}}(z_{t\in\mathcal{T}},z_{G})=\sum_{t=1}^{T}\|\langle
z_{t},z_{G}\rangle\|^{2}$ (5)

where $z_{G}$ is the latent representation from Global Net $G$ and
$z_{t}\equiv z_{L}$ is the LocalNet module output at task $t$ (see Fig. 2).

Besides, inspired by ACL [7] from continual learning literature, we adopt an
adversarial training scheme to make the knowledge in the global net $G$ be
task and domain-invariant. Similar to the GAN pipeline, the global network can
be regarded as a generator. We designed two discriminators: the task
discriminator $D_{T}$ to evaluate which task is the generated sample from
while the domain discriminator $D_{D}$ to classify the domain of the current
sample; see Eqn. 6.

$\displaystyle\mathcal{L}_{\text{adv}}$ $\displaystyle(G,D_{T},D_{D})$ (6)
$\displaystyle=\alpha\min_{G}\max_{D_{T}}\mathrm{E}_{x_{t}^{d}\in
P(X_{\mathcal{T}}^{\mathcal{D}})}\sum_{t=1}^{T}y_{t}^{d}\log(D_{T}(G(x_{t}^{d})))$
$\displaystyle+\beta\min_{G}\max_{D_{D}}\mathrm{E}_{x_{t}^{d}\in
P(X_{\mathcal{T}}^{\mathcal{D}})}\sum_{d=1}^{D}y_{t}^{d}\log(D_{D}(G(x_{t}^{d})))$

We only use discriminators to classify images from variant domains and tasks
instead of identifying real/fake samples. So we do not use fake
$z\sim\mathcal{N}(0,I)$ as in [7].

Contrastive Fine-Tuning. Different from traditional classification-based
pipeline in CL, DG, and ZSL, here we adopt a more natural and proper
contrastive prototypical classifier. We use contrastive loss in Eqn. 7.

$\mathcal{L}_{\text{cont}}=-\log\frac{\exp\left(\operatorname{r}\left(\boldsymbol{t}_{i},\boldsymbol{z}\right)/\tau\right)}{\sum_{k=1}^{N}\exp\left(\operatorname{r}\left(\boldsymbol{t}_{i},\boldsymbol{z}\right)/\tau\right)}$
(7)

where $\boldsymbol{z}=f(z_{L},z_{G})$ is the mapped feature representation of
the outputs from the local networks and the global net, $N$ is the number of
samples in the current batch, $\tau$ is what we called the temperature.

Total Loss. The learning of class-level prompts uses $K$-shot fine-tuning on
training images. For the updating of the rest networks (i.e., global and local
networks, text encoder and discriminators), we design our loss as

$\mathcal{L}_{\text{DIN}}=\lambda_{1}\mathcal{L}_{\text{disen}}+\lambda_{2}\mathcal{L}_{\text{adv}}+\lambda_{3}\mathcal{L}_{\text{cont}}$
(8)

where $\lambda_{1},\lambda_{2},\lambda_{3}$ are hyper-parameters to control
the influence of different loss component.

Method | quick | sketch | info | clip | paint | Avg.
---|---|---|---|---|---|---
mH | mAUC | mH | mAUC | mH | mAUC | mH | mAUC | mH | mAUC | mH | mAUC
Source-only (LB) | 4.69 | 0.61 | 18.80 | 9.05 | 12.14 | 2.69 | 23.03 | 12.52 | 23.97 | 11.75 | 16.53 | 7.32
Oracle (UB) | 9.28 | 6.37 | 19.73 | 14.82 | 14.23 | 5.91 | 19.13 | 17.44 | 25.89 | 18.06 | 17.65 | 12.52
CNZSL [43] | 5.19 | 0.63 | 20.30 | 9.84 | 12.34 | 2.80 | 22.98 | 12.97 | 23.64 | 12.10 | 16.89 | 7.67
CuMix [24] | 0.58 | 0.68 | 4.63 | 10.33 | 1.15 | 3.01 | 4.47 | 11.98 | 7.20 | 12.07 | 3.61 | 7.61
CLIP-ZSL [34] | 0 | 0.37 | 9.99 | 2.18 | 18.65 | 7.00 | 11.22 | 2.72 | 34.48 | 20.60 | 14.87 | 6.57
DIN (Ours) | 2.75 | 4.43 | 33.53 | 18.32 | 29.23 | 14.24 | 40.61 | 24.36 | 33.85 | 17.04 | 27.99 | 15.68

Table 1: GDAZSL experimental results. We show significantly better results
than baselines. Moreover, we also find that our proposed DIN can achieve
better results than Oracle (UB), in which unseen images from unseen domains
are used for training and uses the same network architecture as the one in
CNZSL [43]. Algorithm 1 Domain-Invariant Continual Zero-Shot Learning Network
(DIN)

Training set
$\mathcal{H}_{s}=\sum_{d}\mathcal{H}_{s}^{d}=\\{X_{s}^{d},Y_{s}^{d},A_{s}^{d}\\},d\in{\mathcal{D}^{s}}$,
few-shot $k_{1}$, buffer sampling $k_{2}$, batch size $n_{b}$, total task $T$,
prompt learning epoch $E_{1}$, main model epoch $E_{2}$,
$\theta_{\operatorname{pr}},\theta_{\operatorname{Text}},\theta_{G},\theta_{L},\theta_{D_{dm}},\theta_{D_{ta}}$

Memory Buffer $\mathbb{B}\leftarrow\\{\\}$

Randomly split $\mathcal{H}_{s}^{d}=\sum_{t}\mathcal{H}_{t}^{d},t\in[1,T]$,
then current visible classes $\mathcal{S}_{t}$

for $t=1$ to $T$ do

Run prompt learning Algorithm. 2

Unfreeze
$\theta_{G},\theta_{L},\theta_{\operatorname{Text}},\theta_{D_{dm}},\theta_{D_{ta}}$

Freeze $\theta_{\operatorname{pr}}$

for $e_{2}=1$ to $E_{2}$ do

Compute $\mathcal{L}_{\operatorname{adv}}$ (Eqn. 5) for $G$ using
$H_{t}\cup\mathbb{B}$

Compute $\mathcal{L}_{\operatorname{disen}}$ (Eqn. 6) using $f_{\theta},S$

Compute contrastive loss $\mathcal{L}_{\operatorname{cont}}$ (Eqn. 7) using
$f_{\theta},\operatorname{Text}$

$\mathcal{L}=\lambda_{1}\mathcal{L}_{\text{disen}}+\lambda_{2}\mathcal{L}_{\text{adv}}+\lambda_{3}\mathcal{L}_{\text{cont}}$

$\theta_{G}\leftarrow\theta_{G}-\alpha_{G}\nabla\mathcal{L}$

$\theta_{L}\leftarrow\theta_{L}-\alpha_{L}\nabla\mathcal{L}$

$\theta_{\operatorname{Text}}\leftarrow\theta_{\operatorname{Text}}-\alpha_{\operatorname{Text}}\nabla\mathcal{L}$

for $i=1$ to $n_{d}$ do

Compute $\mathcal{L}_{\operatorname{adv}}$ (Eqn. 5) for $D_{dm}$ and $D_{ta}$
using $H_{t}\cup\mathbb{B}$

$\theta_{D_{dm}}\leftarrow\theta_{D_{dm}}-\alpha_{D_{dm}}\nabla\mathcal{L}$

$\theta_{D_{ta}}\leftarrow\theta_{D_{ta}}-\alpha_{D_{ta}}\nabla\mathcal{L}$

end for

Random sampling $k_{2}$ samples per class as $\mathbb{B}_{t}$

$\mathbb{B}\leftarrow\mathbb{B}\cup\mathbb{B}_{t}$

end for

end for

Algorithm 2 Class-Wise Prompt Learning

for $t=1$ to $T$ do

Freeze
$\theta_{G},\theta_{L},\theta_{\operatorname{Text}},\theta_{D_{dm}},\theta_{D_{ta}}$

Unfreeze $\theta_{\operatorname{pr}}$

$f_{\theta}=f(\theta_{G})$

for $e_{1}=1$ to $E_{1}$ do

Randomly sample
$H_{t}\in\mathcal{H}_{t}\vdash\|H^{d}_{c}\|=k_{1},c\in\mathcal{S}_{t},H_{c}^{d}\in
H_{t}$

Compute contrastive loss $\mathcal{L}_{\operatorname{cont}}$ (Eqn. 7) using
$f_{\theta},\theta_{\operatorname{Text}}$

$\theta_{\operatorname{pr}}\leftarrow\theta_{\operatorname{pr}}-\alpha_{\operatorname{pr}}\nabla\mathcal{L}_{\operatorname{cont}}$

end for

end for

###  4.1 Training Algorithm

We present the whole training algorithm of our method at Alg. 1 and the prompt
learning algorithm at Alg. 2. We denote the method without class-wise prompt
learning as DIN, a single-stage model. We also denote the setting with the
learnable prompt as DIN++.

##  5 Experiments

#### Baselines.

To compare the performance of our proposed method, apart from component
analysis, we adapt state-of-the-art (SOTA) methods from related areas.

  * •

CuMix [24] and its variants. CuMix is the first paper considering the
combination of ZSL and DG, in which we denote the DAZSL setting. The key idea
is to mix up among three images while ensuring two images are from the same
domain and the rest are different.

  * •

CNZSL [43] and its variants. CNZSL is the SOTA method in embedding-based ZSL
and one of the best-performing methods in continual ZSL. In this approach, a
simple but effective class normalization strategy has been applied in ZSL and
CZSL, showing significant improvements.

#### Benchmarks.

Our following experiments are mainly conducted on DomainNet dataset [31],
which is originally designed for multi-source domain adaptations. DomainNet
contains around $0.6$ million images from 345 classes and 6 domains. For
DAZSL, we select 45 classes as unseen classes for testing, matching the number
used in [45]. We follow the same training/validating split with CuMix [24]. We
have 245 seen classes for training and the rest 55 for validating. As
initially there is no attribute information for each class, similar to CuMix
[24], we use word2vec embedding [27] extracted from Google News corpus to
obtain a 300-dimensional representation of each class and adopt $L2$
normalization afterward. We also provided another version with text features
from Transformer [46] pre-trained by CLIP [34]. For our key focus, DACZSL, we
randomly split DomainNet into 8 tasks. As 345 classes can not be divided
equally, we removed the last class to make each task contains 43 classes. We
do this because, for classifier-based methods, an imbalanced number of classes
over tasks will make the model unstable, even though our contrastive
prototypical classifier can handle this imbalanced number classification
problem well.

#### Implementation Details

For all our experiments we use ResNet-50 [16] by default for fair comparison.
Our global network usages ResNet-50 pre-trained by CLIP [34]. The private
network is also chosen to be ResNet-50. We noticed that here future works
could try to reduce the size of private networks. The domain discriminator and
the task discriminator all use 3-layer MLPs. We use AdamW [23] for
optimization. For the baselines, we consider training with their default
hyperparameters but searching important ones. All experiments are performed on
A100-80G with a batch size of 256. More details are provided in the Appendix A
Appendix A:Training Details.

Method | Sketch | Clipart | Painting | Avg.
---|---|---|---|---
LS$\uparrow$ | mH$\uparrow$ | BWT$\uparrow$ | LS$\uparrow$ | mH$\uparrow$ | BWT$\uparrow$ | LS$\uparrow$ | mH$\uparrow$ | BWT$\uparrow$ | LS$\uparrow$ | mH$\uparrow$ | BWT$\uparrow$
CuMix [24] | 6.97 / 5.39 | 3.58 / 3.08 | 0.15 / 0.20 | 6.32 / 5.81 | 3.11 / 2.95 | 0.24 / -0.15 | 6.80 / 5.35 | 3.47 / 2.98 | 0.09 / 0.22 | 6.70 / 5.52 | 3.39 / 3.00 | 0.16 / 0.09
CuMix w/o Mixup | 2.67 / 2.61 | 1.92 / 1.88 | -0.03 / 0.00 | 2.36 / 2.34 | 1.63 1.61 | 0.02 / -0.03 | 2.80 /2.72 | 2.01 / 2.72 | -0.05 / -0.08 | 2.61 / 2.58 | 1.85 / 1.80 | -0.02 / -0.04
CuMix + Tf | 9.52 / 6.94 | 4.06 / 3.91 | 0.32 / 0.13 | 10.25 / 9.30 | 4.09 / 4.06 | 0.33 / 0.28 | 9.57 / 8.60 | 4.69 / 4.51 | 0.06 / -0.41 | 9.78 / 8.28 | 4.28 / 4.16 | 0.24 / 0.00
CNZSL [43] | 6.68 / 6.37 | 2.52 / 2.36 | -0.66 / -0.45 | 6.40 / 5.94 | 2.79 / 2.75 | -0.71 / -0.46 | 7.82 / 6.89 | 2.09 / 1.98 | -0.27 / -0.48 | 6.97 / 6.40 | 2.47 / 2.36 | -0.55 / -0.46
CNZSL w/o CN | 7.16 / 3.22 | 2.67 / 1.42 | -0.75 / -0.65 | 7.07 / 3.92 | 2.92 / 1.63 | -0.72 / -0.10 | 8.36 / 3.87 | 2.15 / 1.61 | -0.35 / -0.53 | 7.53 / 3.67 | 2.58 / 1.55 | -0.61 / -0.43
CNZSL + Tf | 45.74 / 41.20 | 31.75 / 28.59 | 1.00 / 0.89 | 54.19 / 48.43 | 37.33 / 28.73 | 1.17 / 2.93 | 59.15 / 52.39 | 40.37 / 34.5 | 0.96 / 1.26 | 53.03 / 47.34 | 36.48 / 30.61 | 1.04 / 1.69
CLIP-ZSL [34] | 21.39 | 20.83 | 0.00 | 27.73 | 27.46 | 0.00 | 63.06 | 62.35 | 0.00 | 37.39 | 36.88 | 0.00
CLIP-FT | 60.17 / 58.98 | 60.23 / 60.01 | 0.60 / 0.25 | 67.04 / 65.77 | 67.18 / 66.49 | 0.41 / 0.00 | 61.51 / 61.56 | 62.00 / 61.33 | 0.29 / 1.02 | 62.91 / 62.10 | 63.14 / 62.61 | 0.43 / 0.42
DIN | 74.20 / 71.45 | 61.64 / 57.98 | 1.07 / 2.03 | 82.15 / 79.69 | 70.00 / 68.43 |  1.15 / 2.29 | 77.16 / 72.52 | 75.74 / 73.66 |  1.95 / 0.47 | 77.84 / 74.55 | 69.13 / 66.69 | 1.39 / 1.60
DIN++ | 74.53 / 70.99 | 63.78 / 60.14 | 0.92 / 0.81 | 81.31 / 76.01 | 71.24 / 69.18 | 1.12 / 0.65 | 76.00 / 71.19 |  76.10 / 71.96 | 1.82 / 1.55 | 77.28 / 72.73 | 70.37 / 67.09 | 1.29 / 1.00

Table 2: Comparative results on noise-reduced DomainNet with DACZSL uniform
and non-uniform settings. In each cell, left is the uniform result while right
is non-uniform result. + Tf means we use CLIP [34] pre-trained text
Transformer as the feature extractor. We use ResNet50 [16] as visual encoder
backbone for fair evaluations.

###  5.1 DAZSL Experimental Analysis

As only CuMix [24] focuses on DAZSL setting and the implementation publicly
available, we transplant SOTA and popular methods from related areas,
including ZSL [43] and DG [24]. We fine-tune the most important
hyperparameters of baselines on top of their original settings on the
validation set. Compared with DAZSL in [24], our extended setting, named
Generalized DAZSL (GDAZSL), considers the prediction space to be the union of
seen and unseen classes in unseen domains, which is more general and
challenging. We report the performance of CuMix and observe a significant
unseen accuracy drop by 90.77% on average when evaluated on GDAZSL compared to
DAZSL; see Tab. 3.

Setting | Quick | Sketch | Info | Clip | Paint | Avg.
---|---|---|---|---|---|---
DAZSL | 9.9 | 22.6 | 17.8 | 27.6 | 25.5 | 20.7
GDAZSL | 0.30 | 2.43 | 0.59 | 2.33 | 3.88 | 1.91

Table 3: Unseen accuracy of CuMix [24] on DAZSL and GDAZSL settings. The
performance drops by 90.77% from DAZSL to Generalized DAZSL.

Here we show the results of baselines on the much challenging GDAZSL setting
at Tab. 1. We report the most representative metrics harmonic mean mH and area
under seen-unseen curve score mAUC, and attach the remaining metrics in
Appendix C Appendix C: (Generalized) DAZSL Complementary Results and Analysis.
We also consider the lower bound (LB) and upper bound (UB) for GDAZSL. We
define LB as the case when we naively train CNZSL [43] on the source domains
by removing the class normalization trick and then testing on the target
domain. This baseline can be used to exam whether a strategy is working or
not. We set the performance of the upper bound to the CNZSL [43] model without
class normalization trained and tested on the target domain.

Noticing GDAZSL is not a sequential learning task, we provide a variant of our
method by removing the private network and the task discriminator. According
to the results in Tab. 1, our proposed method works significantly better than
baselines by a large marginal. One interesting finding is that we even
outperform the expected upper bound on all almost all metrics. We believe the
success is coming from we use a better training protocol and better
architecture design. We also have similar findings in DACZSL; thus, we present
our detailed analyses in the following section. Note that almost all methods
perform worse on inforgraph and quickdraw domains. [38] found some classes and
labels in DomainNet are very noisy. Following their setup, we create a noisy-
reduced DomainNet dataset for the series experiments on DAZSL and GDAZSL,
denoted as nDomainNet. In nDomainNet, the two domains mentioned above are
directly removed from the source domains and the target domain. Here we show
the results in the Appendix C Appendix C: (Generalized) DAZSL Complementary
Results and Analysis. Accordingly, we use nDomainNet instead in the following
DACZSL experiments.

###  5.2 DACZSL Experimental

Compared with DAZSL, DACZSL considers recognizing images from unseen domains
and unseen categories sequentially, which is more general and natural. We show
the comparative results of uniform DACZSL and non-uniform DACZSL at Tab. 2
(the left is Uniform results and the right is the non-uniform results in each
cell). We provided two variants our DIN. One is a single-stage model without
prompt learning and using the default prompt (i.e., An image of a [CLS]); we
denote it as DIN. For the variant with learnable prompt, we call it DIN++.
Compared with SOTA methods in related methods such as CNZSL [43] and CuMix
[24] accompanied by their variants, our approach significantly improves the
performance. In the DACZSL setting, contrastive prototypical classifier is
more dynamic and more natural compared with the design of training an extra
classifier in CuMix. Besides, a more powerful representation of text is
critical for learning the relationship of image-text pairs. We can also see a
clear improvements when using Transformer feature instead of Word2Vec as the
text representation for CuMix and CNZSL. CLIP can be regarded as a backbone in
our case. Compared with raw CLIP model (i.e., CLIP-ZSL) and fine-tune CLIP
with contrastive learning (CLIP-FT), our network has achieved significantly
boosted results. We assume fine-tuning on DACZSL setting with CLIP backbone by
contrastive prototypical learning is highly valuable and contribute to the
most gains. We also find our specially designed network can improve the
performance in a significant marginal. We will conduct component analysis in
the next section.

Compared with uniform DACZSL, non-uniform DACZSL, where all the images in
several randomly selected domains will be removed in the training step, is
more difficult as we need to consider transferring knowledge from imbalanced
and limited sources. In Tab. 2 right part for each cell, we can see our method
shows a clear improvement over all baselines. Besides, if we compare uniform
and non-uniform DACZSL results, our method is robust to randomly removed-
domain bias with smaller relevant drops. Our performance is stable in these
two settings. However, baseline methods show clear drops from uniform DACZSL
to non-uniform DACZSL. Noticed we directly evaluate CLIP-ZSL [34] without
training; thus, we obtain the same results on both uniform and non-uniform
DACZSL settings.

###  5.3 Ablation Studies

We conduct ablation analysis of our proposed DIN on the Uniform DACZSL
setting. The main components of our method are the global module, the local
modules, the discriminators for domains and tasks, the learnable prompt
module, memory rehearsal, and the orthogonal loss. We comprehensively ablate
them in this section.

Figure 3: Harmonic mean of seen and unseen accuracy of DIN with prompts on
target domain sketch. GLP represents general learnable prompts while CWP means
class-wise prompts. end and mid represents two positions to place class labels
in a prompt.

#### Learnable Prompts

CLIP [34] provided a standard prompt (i.e., text description) template: “A
picture of a CLASS”. However, this is fixed and can be sub-optimal
representation for a complex task like DACZSL. Here, we proposed a novel
class-level learnable prompt and show the ablation at Fig. 3. GLP is Global
Learnable Prompt over all classes, while CWP represents Class-Wise Prompt. We
show using learnable prompt after fine-tuning on increasing samples per class
per domain, the harmonic mean accuracy increases steadily. We also observed
the performance difference between class position mid and end is marginal.
Besides, after fine-tuning on 16 examples per class per domain, CWP performs
better than GLP by 0.37%. Therefore, we conduct further analysis on the
simpler end form and class-wise learnable prompt. As prompt learning will make
our method to be a two-state one, we denote this setting as a variant of DIN,
called DIN++. In Tab. 2, we show that adding class-wise learnable prompt can
have a more balanced classification performance (e.g., mH using learnable
prompts increases 2.14% on clipart target domain).

#### Memory Rehearsal.

To make memory usage more efficient, we explore the possibility of the least
memory for each task. In practice, we only store one sample per domain per
class for each task. Comparing Tab. 4 #a with #d, we show an 2.26% / 2.21%
improvement on LS/mH when accessing only 1 memory rehearsal. We also conducted
more ablations with longer rehearsalWe found that with the rehearsal number
increasing from 1 to 4, the performance among all metrics is steadily
increasing. However, the improvements of memory rehearsal from 4 to 16 are
marginal. We assume one critical reason is that our LocalNets-GlobalNet
coherent module can also store meaningful knowledge from past tasks and avoid
catastrophic forgetting. The coherent module is powerful enough to achieve
good classification accuracy. Besides, our model is efficient in learning from
a small batch of rehearsals. Future works may explore longer tasks to evaluate
the general effectiveness of rehearsal in DACZSL.

#### Global and Local Modules.

We design the global network to store domain-invariant and task-invariant
knowledge while using the local networks to process all task-variant data. We
conduct a cumulative ablation study on our method DIN in the scenario with
1-memory rehearsal. The results are shown in Tab. 5. Compare #a2 with #a1; we
can observe that adding Local Nets can significantly improve LS by 7.53%,
which means less forgetting. More importantly, adding LocalNets can help us
achieve positive knowledge transfer over tasks (e.g., the forgetting rate BWT
increases from 0.51 to 0.65).

# | Module | LS$\uparrow$ | mS$\uparrow$ | mU$\uparrow$ | mH$\uparrow$ | BWT$\uparrow$
---|---|---|---|---|---|---
a | DIN (Ours) | 77.84 | 78.91 | 62.19 | 69.13 | 1.39
b |  \- memory | 75.56 | 77.66 | 61.01 | 67.87 | 1.21
c1 |  \- w/o DD | 76.92 | 77.35 | 61.83 | 68.72 | 0.81
c2 |  \- w/o DT | 75.63 | 76.91 | 60.37 | 67.64 | 1.20
d |  \- w/o DD+DT | 75.58 | 76.56 | 59.44 | 66.92 | 0.79

Table 4: Ablation studies on memory rehearsal and adversarial training
averaging over all target domains.

# | G | L | Adv | Orth | LS$\uparrow$ | mH$\uparrow$ | BWT$\uparrow$
---|---|---|---|---|---|---|---
a1 | ✓ |  |  |  | 64.22 | 64.81 | 0.51
a2 | ✓ | ✓ |  |  | 71.75 | 65.44 | 0.65
a3 | ✓ |  | ✓ |  | 26.62 | 28.87 | -13.08
a4 | ✓ | ✓ | ✓ |  | 76.34 | 68.17 | 0.61
a5 | ✓ | ✓ |  | ✓ | 75.58 | 66.92 | 0.79
a6 | ✓ | ✓ | ✓ | ✓ | 77.84 | 69.13 | 1.39

Table 5: Cumulative ablation study on our method DIN with $1$-memory
rehearsal. We report the average performance by traversing all target domains.

#### Adversarial Learning and Disentanglement.

Adversarial learning and disentangled learning signals are essential to make
the global net containing domain-invariant and task-invariant information.
Unlike traditional adversarial training, we only consider two classifications,
one to classify which domain and another to classify which task. Fake samples
are not generated and are not necessary in our case. We hope the global
network can output domain-invariant and task-invariant information that the
discriminators can not distinguish. If we remove adversarial training; see
Fig. 5 #a5 and Tab. 5 #a6, the LS and mH performance will drop by 2.26% and
2.21%, respectively. We also show the results without orthogonal loss at #a4.
The orthogonal loss can improve the performance in marginal but not strong
guidance reflected by the accuracy. Besides, comparing Tab. 5 #a3 with #a1,
where #a3 can be regarded as the variance to consider only domain and task
invariant features for classification. Conduct adversarial loss will lead to
very significant drops by 37.6% LS and 35.94% mH, the forgetting rate is also
dropped by 13.59. This ablation shows domain and task invariant features can
help store essential domain/task independent information but can not be
directly applied to classify images from different domains and categories
sequentially, which leads to catastrophic forgetting after adversarial
training. We also consider removing domain discriminator only (#c1) and
removing task discriminator only (#c2) in Tab. 4, the accuracy over all
metrics is influenced more by the task discriminator (LS/mH drops by
0.92%/0.41% vs. 2.21%/1.49% after removing DD/DT) while the domain
discriminator contributes more to make the global network more robust with
less forgetting (BTW drops by 0.58 vs. 0.19 after removing DD vs. DT).

##  6 Limitations and Future Works

We observe that our method DIN has the following limitations that may need to
be resolved in future works. a) the size of our current private network is not
designed to be significantly smaller than the global network, which means the
model size will grow linearly with more tasks; b) we currently use a
preliminary adversarial training scheme, future works may explore better
generative models to make the global network contains the domain and task-
invariant information; c) As our focus is classifying images with unseen
categories from unseen domains sequentially, we noticed that a better-
converged model in the training step might not perform better on the
validation set. A better scheme to guide the learning procedure is highly
required for DACZSL.

##  7 Conclusion

In this paper, we proposed a novel setting named domain-aware continual zero-
shot learning to explore the limit of recognizing images from unseen
categories and unseen domains in a lifelong learning setting. We also proposed
a specially designed contrastive prototypical knowledge disentangled
representation network to handle this new problem. We show that a more
representative text feature extractor can help better understand vision-
language relationships to recognize out-of-distribution images better.
Besides, the contrastive prototypical classifier is more suitable for
exploring unseen than training an extra task-oriented classifier. We also show
that disentangling domain-invariant and task-invariant information in DACZSL
is important and leads to better results along with task-invariant features.
We hope this work may encourage follow-up works to explore conceptual and
knowledge transfer in this challenging out-of-distribution setting and propose
a better scheme to understand vision-language and model the relationship
between seen and unseen classes.

## References

  * [1] Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars.  Task-free continual learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11254–11263, 2019.
  * [2] Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan.  Domain separation networks.  Advances in neural information processing systems, 29:343–351, 2016\.
  * [3] Francisco M Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari.  End-to-end incremental learning.  In Proceedings of the European conference on computer vision (ECCV), pages 233–248, 2018.
  * [4] Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny.  Efficient lifelong learning with a-gem.  arXiv preprint arXiv:1812.00420, 2018.
  * [5] Zhengming Ding and Yun Fu.  Deep domain generalization with structured low-rank constraint.  IEEE Transactions on Image Processing, 27(1):304–313, 2017.
  * [6] Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Marcus Rohrbach.  Uncertainty-guided continual learning with bayesian neural networks.  ArXiv, abs/1906.02425, 2020.
  * [7] Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach.  Adversarial continual learning.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16, pages 386–402. Springer, 2020.
  * [8] Mohamed Elhoseiny and Mohamed Elfeki.  Creativity inspired zero-shot learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5784–5793, 2019.
  * [9] Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth.  Describing objects by their attributes.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 1778–1785. IEEE, 2009.
  * [10] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  The journal of machine learning research, 17(1):2096–2030, 2016\.
  * [11] Chandan Gautam, Sethupathy Parameswaran, Ashish Mishra, and Suresh Sundaram.  Generalized continual zero-shot learning.  arXiv preprint arXiv:2011.08508, 2020.
  * [12] Subhankar Ghosh.  Dynamic vaes with generative replay for continual zero-shot learning.  arXiv preprint arXiv:2104.12468, 2021.
  * [13] Rui Gong, Wen Li, Yuhua Chen, and Luc Van Gool.  Dlow: Domain flow for adaptation and generalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2477–2486, 2019.
  * [14] Ishaan Gulrajani and David Lopez-Paz.  In search of lost domain generalization.  In International Conference on Learning Representations, 2020.
  * [15] Zongyan Han, Zhenyong Fu, Shuo Chen, and Jian Yang.  Contrastive embedding for generalized zero-shot learning.  arXiv preprint arXiv:2103.16173, 2021.
  * [16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [17] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In ICML, 2021.
  * [18] Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig.  How can we know what language models know?  Transactions of the Association for Computational Linguistics, 8:423–438, 2020.
  * [19] Teven Le Scao and Alexander M Rush.  How many data points is a prompt worth?  In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2627–2636, 2021.
  * [20] Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee.  Overcoming catastrophic forgetting with unlabeled data in the wild.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 312–321, 2019.
  * [21] Shaoteng Liu, Jingjing Chen, Liangming Pan, Chong-Wah Ngo, Tat-Seng Chua, and Yu-Gang Jiang.  Hyperbolic visual embedding learning for zero-shot recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9273–9281, 2020.
  * [22] David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continual learning.  Advances in neural information processing systems, 30:6467–6476, 2017.
  * [23] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [24] Massimiliano Mancini, Zeynep Akata, Elisa Ricci, and Barbara Caputo.  Towards recognizing unseen categories in unseen domains.  In proceedings of the European Conference on Computer Vision. Springer, 2020.
  * [25] James L McClelland, Bruce L McNaughton, and Randall C O’Reilly.  Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory.  Psychological review, 102(3):419, 1995.
  * [26] Michael McCloskey and Neal J Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier, 1989.
  * [27] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space.  arXiv preprint arXiv:1301.3781, 2013.
  * [28] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  In Advances in neural information processing systems, pages 3111–3119, 2013.
  * [29] Jovana Mitrovic, Brian McWilliams, Jacob C Walker, Lars Holger Buesing, and Charles Blundell.  Representation learning via invariant causal mechanisms.  In International Conference on Learning Representations, 2020.
  * [30] Sanath Narayan, Akshita Gupta, Fahad Shahbaz Khan, Cees GM Snoek, and Ling Shao.  Latent embedding feedback and discriminative features for zero-shot classification.  arXiv preprint arXiv:2003.07833, 2020.
  * [31] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.  Moment matching for multi-source domain adaptation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1406–1415, 2019.
  * [32] Jeffrey Pennington, Richard Socher, and Christopher D Manning.  Glove: Global vectors for word representation.  In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014.
  * [33] Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi.  Efficient domain generalization via common-specific low-rank decomposition.  In International Conference on Machine Learning, pages 7728–7738. PMLR, 2020.
  * [34] Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [35] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  ICLR, 2016.
  * [36] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115(3):211–252, 2015\.
  * [37] Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, J. Kirkpatrick, K. Kavukcuoglu, Razvan Pascanu, and R. Hadsell.  Progressive neural networks.  ArXiv, abs/1606.04671, 2016.
  * [38] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko.  Semi-supervised domain adaptation via minimax entropy.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8050–8058, 2019.
  * [39] Jonathan Schwarz, Wojciech M. Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska, Y. Teh, Razvan Pascanu, and R. Hadsell.  Progress & compress: A scalable framework for continual learning.  ArXiv, abs/1805.06370, 2018.
  * [40] Seonguk Seo, Yumin Suh, Dongwan Kim, Geeho Kim, Jongwoo Han, and Bohyung Han.  Learning to optimize domain specific normalization for domain generalization.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16, pages 68–83. Springer, 2020.
  * [41] Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh.  AutoPrompt: Eliciting knowledge from language models with automatically generated prompts.  In Empirical Methods in Natural Language Processing (EMNLP), 2020\.
  * [42] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In ICLR, 2015.
  * [43] Ivan Skorokhodov and Mohamed Elhoseiny.  Class normalization for zero-shot learning.  In International Conference on Learning Representations, 2021.
  * [44] Nathan Somavarapu, Chih-Yao Ma, and Zsolt Kira.  Frustratingly simple domain generalization via image stylization.  arXiv preprint arXiv:2006.11207, 2020.
  * [45] William Thong, Pascal Mettes, and Cees GM Snoek.  Open cross-domain visual search.  Computer Vision and Image Understanding, 200:103045, 2020.
  * [46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [47] Maunil R Vyas, Hemanth Venkateswara, and Sethuraman Panchanathan.  Leveraging seen and unseen semantic relationships for generative zero-shot learning.  In European Conference on Computer Vision, pages 70–86. Springer, 2020.
  * [48] Kun Wei, Cheng Deng, and Xu Yang.  Lifelong zero-shot learning.  In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 551–557, 2020.
  * [49] Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt.  Robust fine-tuning of zero-shot models.  arXiv preprint arXiv:2109.01903, 2021.
  * [50] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu.  Large scale incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 374–382, 2019.
  * [51] Wenjia Xu, Yongqin Xian, Jiuniu Wang, Bernt Schiele, and Zeynep Akata.  Attribute prototype network for zero-shot learning.  In 34th Conference on Neural Information Processing Systems. Curran Associates, Inc., 2020.
  * [52] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang.  Lifelong learning with dynamically expandable networks.  ArXiv, abs/1708.01547, 2018.
  * [53] Zhongqi Yue, Tan Wang, Qianru Sun, Xian-Sheng Hua, and Hanwang Zhang.  Counterfactual zero-shot and open-set visual recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15404–15414, 2021.
  * [54] Shanshan Zhao, Mingming Gong, Tongliang Liu, Huan Fu, and Dacheng Tao.  Domain generalization via entropy regularization.  Advances in Neural Information Processing Systems, 33, 2020.
  * [55] Zexuan Zhong, Dan Friedman, and Danqi Chen.  Factual probing is [mask]: Learning vs. learning to recall.  In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5017–5033, 2021.
  * [56] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  arXiv preprint arXiv:2109.01134, 2021.
  * [57] Yizhe Zhu, Mohamed Elhoseiny, Bingchen Liu, Xi Peng, and Ahmed Elgammal.  A generative adversarial approach for zero-shot learning from noisy texts.  In CVPR, 2018.

## Appendix A:Training Details

### Architecture and Training Details

Our proposed method DIN contains the class-wise learnable prompt, global
network, local networks, domain discriminator, and task discriminator. Next,
we will introduce more details on the training and optimization details of
each module.

#### Class-Wise Learnable Prompt Module

CLIP [34] uses the naive prompt in the following form: An image of a [CLASS].
However, this design may not capture class-specific information over different
datasets. CoOp [56] discovered that a better prompt for a particular dataset
would highly influence the unseen classification accuracy. Encouraged by CoOp,
we proposed a domain-invariant class-wise learnable prompt module. The general
form of the learnable prompt is shown in the main thesis content Eqn. 4. The
initialization ways differ between GLP (general learnable prompt) and CWP
(class-wise prompt). Our prompt module will be tensor (# classes, prompt
length, dim per prompt). GLP will initialize each class tensor to be
identical, while CWP will initialize it uniquely. We first train this prompt
by accessing $K$-training examples per class and domain. The learning rate of
the prompt module is set to be 0.002, and $K$ is 16 by default. For the
results with different K, we train 200 epochs for 16/8, 100 epochs for 4/2,
and 50 epochs for 1. We use the batch size of 256 on all experiments,
including baselines and methods along with their variants.

#### Global and Local Coherent Module

We use ResNet50 [16] pre-trained from CLIP [34] as the visual encoder for the
global and local networks. However, the global network aims to store domain
and task-invariant information while the local networks are designed to
process task-variant information from different source domains. We adopt
Transformer [46] as the text encoder pre-trained from CLIP. The learning rate
for those modules is 5e-7. We use AdamW [23] as the optimizer and set the
warmup length to be 5 and weight decay to be 0.02. We use the cosine learning
scheduler in the adaption of the learning rate. Training epochs for all our
modules are set to be 25.

#### Domain and Task Discriminators

For our domain discriminator and the task discriminator, we use 3-layers MLPs
incorporated with LeakyReLU. We choose the learning rate equals 0.001; weight
decay equals 0.01. SGD is applied for optimization. The latent dim of
discriminators is 1024. Similar to ACL [7], we add the Gradient Reversal layer
[10] before feeding the generated latent features by the global network to the
domain and task-specific discriminators.

#### CuMix/CNZSL and Their Variants

For those ablations, we mainly set the default best hyper-parameters provided
but will validate some critical hyper-parameters. For CuMix [24], the hyper-
parameter of the best-performed model on DACZSL and related experiments on
DACZSL is set to have image mixup weight 0.001, feature mixup weight 0.5,
mixup step 2, and mixup beta 2. The backbone learning rate is 0.0001, and the
mapping network learning rate is 0.001, all use weight decay 0.00005. The
training epoch is set to 25. For CNZSL [43], we adopt the same design and
apply task-specific class normalization. For CuMix, CNZSL, and their variants
using Transformer, which is pre-trained from CLIP [34], to extract the text
features, we use an epoch equal to 50. We noticed that for CNZSL and its
related experiments, we set the batch size equal to 512, but we set the batch
size to 360 for CuMix related experiments for limited memory.

## Appendix B: Illustration of Related Settings

### Zero-Shot Learning (ZSL), Domain Generalization (DG), and Continual
Learning (CL)

We first conduct conceptual analysis on fundamental settings, which are most
related to us. They are Zero-Shot Learning (ZSL), Domain Generalization (DG),
and Continual Learning (CL). We show the difference at Fig. 4. ZSL and DG are
all training on $H_{1}$. The only difference between these two is the out-of-
distribution cases in testing. For ZSL, we need to recognize images from
unseen categories but seen domains in $H_{2}$. However, in DG, we are
recognizing images from unseen domains but seen classes. Images in CL are from
the same domain and same class, but this setting is a sequential task learning
problem, which tries to have less forgetting on previous tasks $H_{1\to T-1}$.

Figure 4: Conceptual differences between Zero-Shot Learning (ZSL), Domain
Generalization (DG), and Continual Learning (CL).

ZSL, DG, and CL are fundamental settings but not enough to solve our key
focus: recognizing images from unseen categories and unseen domains in a life-
long setting. Next, we present some existing settings that are related to us.

### (Generalized) Domain-Aware Zero-Shot Learning (DAZSL/GDAZSL)

Figure 5: Intuitions of Domain-Aware Zero-Shot Learning (DAZSL). Generalized
DAZSL is an extended version to recognize images not only from unseen classes
but also from seen categories.

Suppose we have 3 domains ( $d_{1}$,  $d_{2}$, and  $d_{3}$), 6 classes
($c_{i},i\in\mathbb{Z},i\in[1,6]$). We denote the $c_{1}\to c_{4}$ as the
training set while the rest as the testing set. GDAZSL will consider
recognizing images from both seen and unseen classes while ZSDG will only care
about recognizing images from unseen categories. We show setting intuitions at
Fig. 6.

Figure 6: (Generalized) Domain-Aware Zero-Shot Learning Setting.

### Continual Zero-Shot Learning (CZSL)

We show the intuition of CZSL at Fig. 7.

Figure 7: Intuition of Continual Zero-Shot Learning (CZSL).

### Domain-Aware Continual Zero-Shot Learning (DACZSL)

In Fig. 8, we show intuition of DACZSLs, which recognizes images from unseen
domains and unseen categories sequential. Sequential figures are also
presented in the main content Fig. 1.

Figure 8: Intuition of Domain-Aware Continual Zero-Shot Learning.

Here we consider several variants of DACZSL, which are Domain-Agonistic
Continual Zero-Shot Learning (DAgCZSL), Uniform Domain-Aware Continual Zero-
Shot Learning (U-DACZSL), and Non-Uniform Domain-Aware Continual Zero-Shot
Learning (N-DACZSL). The formal definition of U-DACZSL and N-DACZSL have been
provided in the main content. Here we show the conceptual differences with
examples among those settings.

Suppose we have 3 domains ( $d_{1}$,  $d_{2}$, and  $d_{3}$), 6 classes
($c_{i},i\in\mathbb{Z},i\in[1,6]$) uniformly divided into 3 sequential tasks.

We first propose the Domain Agnostic Continual Zero-Shot Learning; visualized
at Fig. 9.

Figure 9: Domain-Agnostic Continual Zero-Shot Learning (DAgCZSL).

Later we present domain-aware (uniform) CZSL, which is more interesting;
visualized at Fig. 10.

Figure 10: Uniform Domain-Aware Continual Zero-Shot Learning.

Next, we consider a more difficult task domain-sensitive (non-uniform) CZSL,
in which one seen domain is randomly removed in training; visualized at Fig.
11.

Figure 11: Non-Uniform Domain-Aware Continual Zero-Shot Learning.

## Appendix C: (Generalized) DAZSL Complementary Results and Analysis

We have illustrated the concept of Domain-Aware Zero-Shot Learning (DAZSL) and
Generalized Domain-Aware Zero-Shot Learning (GDAZSL) setting in the main text
and Section. (Generalized) Domain-Aware Zero-Shot Learning (DAZSL/GDAZSL) in
this supplementary. Here we provide more details. We first present the
statistics of benchmark DomainNet [31]. Then we provide more statistical
results.

Method | quick | sketch | info | clip | paint | Avg.
---|---|---|---|---|---|---
mS | mU | mH | mAUC | mS | mU | mH | mAUC | mS | mU | mH | mAUC | mS | mU | mH | mAUC | mS | mU | mH | mAUC | mS | mU | mH | mAUC
Source-only (LB) | 7.93 | 3.33 | 4.69 | 0.61 | 38.4 | 12.4 | 18.80 | 9.05 | 16.88 | 9.51 | 12.14 | 2.69 | 47.56 | 15.2 | 23.03 | 12.52 | 42.14 | 16.76 | 23.97 | 11.75 | 30.58 | 11.44 | 16.53 | 7.32
Oracle (UB) | 54.68 | 5.08 | 9.28 | 6.37 | 54.61 | 12.05 | 19.73 | 14.82 | 29.57 | 9.38 | 14.23 | 5.91 | 66.98 | 11.17 | 19.13 | 17.44 | 61.32 | 16.42 | 25.89 | 18.06 | 53.43 | 10.82 | 17.65 | 12.52
CNZSL [43] | 7.85 | 3.87 | 5.19 | 0.63 | 39.2 | 13.7 | 20.30 | 9.84 | 17.5 | 9.53 | 12.34 | 2.80 | 48.15 | 15.1 | 22.98 | 12.97 | 43.82 | 16.19 | 23.64 | 12.10 | 31.30 | 11.68 | 16.89 | 7.67
CuMix [24] | 10.9 | 0.23 | 0.58 | 0.68 | 47.99 | 2.43 | 4.63 | 10.33 | 21.52 | 0.59 | 1.15 | 3.01 | 53.11 | 2.33 | 4.47 | 11.98 | 49.67 | 3.88 | 7.20 | 12.07 | 36.64 | 1.91 | 3.61 | 7.61
CLIP-ZSL [34] | 0.33 | 0 | 0 | 0.37 | 10.55 | 9.47 | 9.99 | 2.18 | 21.00 | 16.77 | 18.65 | 7.00 | 12.73 | 10.04 | 11.22 | 2.72 | 37.28 | 32.07 | 34.48 | 20.60 | 16.38 | 13.67 | 14.87 | 6.57
DIN (Ours) | 4.63 | 1.95 | 2.75 | 4.43 | 34.43 | 32.67 | 33.53 | 18.32 | 29.56 | 28.91 | 29.23 | 14.24 | 41.73 | 39.54 | 40.61 | 24.36 | 31.40 | 36.71 | 33.85 | 17.04 | 28.35 | 27.96 | 27.99 | 15.68

Table 6: GDAZSL complete experimental results.

### Statistics of DomainNet for (G)DAZSL

The pioneering work combining zero-shot learning and domain generalization is
CuMix [24]. We even further consider a more difficult case with the
combination of generalized zero-shot learning with domain generalization. We
call the first setting DAZSL while the latter as Generalized DAZSL (GDAZSL).
We first show the statistics at Fig. 7. For DAZSL, in the testing step, we
only consider the test split consisting of 45 classes. However, in GDAZSL, we
take all testing images over 345 classes into considerations.

setting | # classes | quick | sketch | info | clip | paint | real | total.
---|---|---|---|---|---|---|---|---
train | 245 | 85, 750 | 33, 803 | 27, 050 | 23, 238 | 36, 816 | 86, 480 | 293, 137
val | 55 | 19, 250 | 7, 690 | 4, 651 | 5, 891 | 6, 324 | 17, 528 | 61, 334
test | 45 | 22, 500 | 9, 626 | 6, 195 | 6, 317 | 10, 418 | 24, 155 | 79, 211
total | 345 | 127, 500 | 51, 119 | 37, 896 | 35, 446 | 53, 558 | 128, 163 | 433, 682

Table 7: DomainNet [31] statistics for DAZSL and GDAZSL. After getting the
best hyper-parameters on the validation set, train and val will be merged as
the training set in practice.

### Evaluation Metrics for (G)DAZSL

Mean Seen Accuracy (mSA) is the seen accuracy over tasks:

$\operatorname{mSA}=\frac{1}{C^{s}}\sum_{c=1}^{C^{s}}\frac{\\#\operatorname{correct\
pred.\ in\ c}}{\\#\operatorname{samples\ in\ c}}$ (9)

Method | quick | sketch | info | clip | paint | Avg.
---|---|---|---|---|---|---
Source-only LB | 9.81 | 25.04 | 17.43 | 26.49 | 28.5 | 21.45
Oracle UB | 14.62 | 30.25 | 20.58 | 28.00 | 29.81 | 24.65
CuMix∗ [24] | 9.9 | 22.6 | 17.8 | 27.6 | 25.5 | 20.68
CuMix [24] | 8.7 | 21.8 | 17.0 | 26.2 | 24.9 | 19.72
CNZSL [43] | 9.11 | 26.12 | 16.64 | 28.08 | 29.98 | 21.99
CLIP-ZSL [34] | 1.55 | 22.71 | 35.51 | 22.91 | 57.46 | 28.03
DIN (Ours) | 10.93 | 40.67 | 47.85 | 51.88 | 56.60 | 41.59

Table 8: DAZSL experimental results. All results are the average after three
trials with the same setting. ∗ means the reported results. LB is the lower
bound while UB is the upper bound. We use ResNet50 [16] as the default
backbone.

Mean Unseen Accuracy (mUA) is the unseen accuracy over tasks:

$\operatorname{mUA}=\frac{1}{C^{u}}\sum_{c=1}^{C^{u}}\frac{\\#\operatorname{correct\
pred.\ in\ c}}{\\#\operatorname{samples\ in\ c}}$ (10)

Mean Harmonic (mH) over mSA and mUA:

$\operatorname{mH}=\frac{2\operatorname{mSA}\times\operatorname{mUA}}{\operatorname{mSA}+\operatorname{mUA}}$
(11)

Seen-Unseen Area Under Curve (SUAUC) is to detect model’s bias towards seen or
unseen data. We first build a classification rule as follows:

$\hat{y}=\text{argmax}_{c\in(C^{s}\cup C^{u})}\ \
f_{c}(\textbf{x}-\gamma\mathbb{I}[c\in C^{s}])$ (12)

where $\mathbb{I}[\cdot]\in\\{0,1\\}$ is the seen mask and $\gamma$ is called
as the calibration factor. With $\gamma$ from $-\infty$ to $+\infty$, we
calculate a set of seen accuracy and unseen accuracy to form $\mathbf{s}$ and
$\mathbf{u}$. We compute the area to be SUAUC.

### Results on DAZSL

The pioneer work in combining ZSL and DG is CuMix [24]. In the testing step,
the searching space is only unseen classes. Therefore, their setting can be
regarded as the combination of DG and ZSL, which we denote as DAZSL. We show
the results at Tab. 8. Our proposed method also achieved significantly better
results than baselines.

### GDAZSL Complementary Results

We also extend DAZSL to a much tricky setting, GDAZSL, in which the searching
space in testing contains both seen and unseen classes (i.e., the combination
of DG + GZSL). We have already shown the representative matrices result on mH
and mAUC in the main thesis content, here we report the complete results at
Tab. 6.

### Comparative Analysis between DAZSL and GDAZSL

Combining Tab. 8 and Tab. 6, we show that GDAZSL is a much difficult setting
compared with DAZSL reflected by the performance over all matrices by all
experimented methods. We found that our method can achieve significantly
better results by a large margin.