# pSTarC: Pseudo Source Guided Target Clustering
for Fully Test-Time Adaptation

Manogna Sreenivas†, Goirik Chakrabarty∗, Soma Biswas†
†{manognas<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Test Time Adaptation (TTA) is a pivotal concept in machine learning, enabling
models to perform well in real-world scenarios, where test data distribution
differs from training. In this work, we propose a novel approach called pseudo
Source guided Target Clustering (pSTarC) addressing the relatively unexplored
area of TTA under real-world domain shifts. This method draws inspiration from
target clustering techniques and exploits the source classifier for generating
pseudo-source samples. The test samples are strategically aligned with these
pseudo-source samples, facilitating their clustering and thereby enhancing TTA
performance. pSTarC operates solely within the fully test-time adaptation
protocol, removing the need for actual source data. Experimental validation on
a variety of domain shift datasets, namely VisDA, Office-Home, DomainNet-126,
CIFAR-100C verifies pSTarC’s effectiveness. This method exhibits significant
improvements in prediction accuracy along with efficient computational
requirements. Furthermore, we also demonstrate the universality of the pSTarC
framework by showing its effectiveness for the continuous TTA framework.

##  1 Introduction

Over the past decade, deep networks have shown a continuous upward trend due
to the availability of large datasets, significant improvements in computing
power, and advancements in algorithms and architectures. But while humans can
adapt seamlessly to new domains, the performance of deep networks deteriorate
significantly when the test and training distributions differ. In practical
scenarios, a trained model is often deployed in an unseen test environment, so
equipping it with good adaptation capabilities to mitigate the adverse effects
of any domain shift is crucial. Additionally, since access to the source data
may be difficult because of privacy concerns or storage limitations, there is
a significant interest in the following research directions: (i) Source-free
Domain Adaptation (SFDA) [13, 26, 27], which assumes access to the source
model and a large amount of unlabeled test data and (ii) Test-Time adaptation
(TTA) [22, 1, 2], where test data arrives in an online manner, one batch at a
time, allowing for one-step model adaptation followed by prediction. SFDA and
TTA methods have been developed independently, resulting in fundamentally
different approaches.

Here, we address the challenging and more practical task of swiftly adapting
models without the need for extensive accumulation of test data, i.e. the TTA
setting. Unlike SFDA methods which have been evaluated on real-world domain
shift datasets like VisDA, DomainNet and Office-Home, TTA methods have
primarily been evaluated within the confines of artificially corrupted data.
It is only recently that researchers have started to address the TTA task for
such real-world domain shifts [2, 10, 11].

In this work, we propose a simple yet effective TTA strategy termed pseudo
Source guided Target Clustering (pSTarC). It is inspired by the exceptional
performance of SFDA techniques like SHOT [13], NRC [26], and AaD [27] in the
context of the real world domain-shift benchmarks. Notably, contemporary SFDA
methods, including NRC and AaD, concentrate on refining target sample
clustering, leveraging the luxury of abundant unlabeled target data. To extend
this SFDA principle to TTA, one compelling avenue is the maintenance of a
feature bank, which dynamically populates as new target data becomes
available, enriching the adaptation process. While approaches like AdaContrast
[2] have successfully harnessed this concept for TTA, they need to store
auxiliary components like the momentum encoder and key features, a constraint
that might not align well with an online TTA framework.

Our proposed pSTarC approach aims to leverage the power of SFDA objectives
while adhering to the principle of minimizing memory and storage requirements
for TTA task. Generally, the source-trained classifier remains unchanged
during TTA to preserve the valuable class-discriminative insights gained from
the source. Building on this insight, we introduce a novel strategy: utilizing
the classifier to generate a diverse array of pseudo-source samples, thereby
steering the target clustering process. Impressively, our findings reveal that
generating as few as 20 pseudo-source samples per class is adequate to achieve
state-of-the-art TTA performance, without imposing a significant burden on
storage demands. Thus, the main contributions of this work can be summarized
as follows:

  1. 1.

We propose the pSTarC framework, which generates pseudo-source samples to
guide the target clustering during test time adaptation.

  2. 2.

We strive to achieve TTA using SFDA objectives, which not only improves the
TTA performance significantly for real domain-shifts, but also helps to unify
the seemingly disparate research directions.

  3. 3.

pSTarC outperforms the state-of-the-art TTA techniques on Office-Home and
DomainNet, and at par on VisDA, while requiring much lesser memory.

  4. 4.

pSTarC also seamlessly works in Continual Test-Time Adaptation (CTTA) [24]
scenario, where the test distribution changes with time. Here, its performance
is at par with current state-of-the-art approaches on the large-scale
DomainNet-126 benchmark.

In a nutshell, pSTarC aligns seamlessly with our objective to pave the way for
swift, efficient TTA in the face of real-world domain shifts, building upon
the insights garnered from the relationship between SFDA techniques and such
demanding benchmarks.

##  2 Related Works

Here, we discuss the related work in Source-Free Domain Adaptation, Test-Time
Adaptation, Continuous Test Time Adaptation and Model Inversion for generating
pseudo-data.

Source-free domain adaptation (SFDA) aims to adapt a source domain trained
model to a target domain without access to any labeled data from either the
source or target domain. SFDA methods typically assume access to abundant
unlabeled data from the target domain and leverage the structure of the data
to refine the target predictions. [13] proposes to cluster target features by
mutual entropy maximization along with pseudo labeling, while keeping the
classifier fixed. [2] extends the idea in [13] proposing to refine the pseudo
labels using a feature bank, alongside doing self-supervised contrastive
learning [3]. Another line of work include [26, 27], where they exploit the
inherent semantic structure of the target features extracted from the source
model. They reinforce consistency between the predictions of a sample and its
local neighbors while also ensuring diversity to avoid degenerate predictions.

Test Time Adaptation (TTA) further relaxes the assumptions on data
availability compared to SFDA. TENT [22] proposed the more practical fully
test time adaptation setting, where source data cannot be accessed at all, and
the model can only utilize the test samples in each batch encountered in an
online manner for adaptation. They propose minimizing the entropy of the model
predictions on the test data. More recently LAME [1] uses Concave-Convex
Procedure (CCCP) to modify the feature vectors to obtain better
classification, while AdaContrast [2] addresses SFDA and TTA settings by using
contrastive learning with nearest neighbour soft voting for online pseudo
label refinement. C-SFDA [11] uses curriculum learning in a Teacher Student
framework. There are other works like EATA [15], which uses a small buffer
from source distribution and and TTN [14], which trains a modified BN layer to
leverage source data for improved TTA. In [10], they synthesize source proxy
images by condensing the source dataset, which is then used during TTA after
stylizing them to match the test distribution. Our work falls in the category
of fully test-time adaptation [22, 2, 11, 24].

Continual Test Time Adaptation (CTTA) As a further extension of TTA, the
concept of continual test-time adaptation (CTTA) has been recently
introduced[24]. This protocol recognizes the dynamic nature of the testing
environment, where the test domain evolves over time. CoTTA [24] adopts
strategies like weight-averaged and augmentation-averaged predictions in a
teacher-student framework to mitigate error accumulation. Additionally, it
retains a fraction of neurons with source pre-trained weights during each
iteration to prevent catastrophic forgetting, thus enabling model adaptation
while preserving source knowledge. RMT [4] is a recent SOTA CTTA method which
uses a combination of symmetric cross-entropy loss and contrastive loss in a
teacher student framework.

Model inversion is a recent research direction explored in [23, 12, 18] for
image generation where they optimize the input space to generate an image
$\hat{x}$ using a pre-trained deep network. To do this, given an arbitrary
target $y$ which can be a label or a reference image, a trainable input
$\hat{x}$ in the image space is initialized with random noise. This input
space is then optimized by minimizing a loss function
$\mathcal{L}(\hat{x},y)$, which is usually cross-entropy loss and a
regularizer $\mathcal{R}(\hat{x})$ to induce natural image prior. The training
is done in an adversarial manner by alternating between the optimization of
the synthesized image and that of the discriminator weights. Inspired by the
effectiveness of these methods, here we propose a classifier guided feature
generation approach, which is used for generating pseudo-source samples for
guiding the clustering of the target data.

##  3 Problem Setting & Motivation

Test Time Adaptation (TTA) setting usually consists of a source training phase
(which cannot be controlled), and the TTA stage, where the trained model in
the previous stage is adapted using the test batches in an online manner.

Source training: The model is first trained using labeled source data
$\mathcal{D}_{s}=\\{x_{i}^{s},y_{i}^{s}\\}_{i=1}^{n_{s}}$ comprising of $C$
classes. Here, $x_{i}^{s}\in\mathcal{X}_{s}$ and $y_{i}^{s}\in\mathcal{Y}_{s}$
denote the source sample and its class label, and $n_{s}$ is the number of
training samples. We denote the source model as $\mathbf{F_{s}=H_{s}\circ
G_{s}}$, where $\mathbf{G_{s}}$ is the feature extractor and $\mathbf{H_{s}}$
is the classifier. Following [13, 27, 2], the source model
$\mathbf{F_{s}}:\mathcal{X}_{s}\rightarrow\mathcal{Y}_{s}$ is trained by
minimizing the label-smoothing cross entropy loss as

$\mathcal{L}_{src}\left(f_{s};\mathcal{D}_{s}\right)=-\mathbb{E}_{\left(x^{s},y^{s}\right)\in\mathcal{D}_{s}}\sum_{c=1}^{C}\tilde{y}_{c}^{s}\log
p_{c},$

where $p_{c}=\delta_{c}\left(f_{s}\left(x^{s}\right)\right)$ is the softmax
score for class $c$, $\delta$ being the softmax function. The smooth label is
computed as $\tilde{y}_{c}^{s}=(1-\alpha)y_{c}^{s}+\alpha/C$, where the
smoothness coefficient $\alpha$ is set to $0.1$.

Test Time Adaptation: Given the source model $\mathbf{F_{s}}$, during TTA, the
target model $\mathbf{F_{t}}$ is initialized with the source model
$\mathbf{F_{s}}$. We only have access to the unlabeled test samples $x_{t}$
coming in batches from an unseen test distribution $\mathcal{D}_{t}$. Here, we
address the closed setting where the source and target samples come from the
same $C$ classes. The goal is to continuously adapt
$\mathbf{F_{t}}:\mathcal{X}_{t}\rightarrow\mathcal{Y}_{t}$ using the unlabeled
samples $x_{t}\in\mathcal{X}_{t}$ (in batches) in an online manner.
Continual Test Time Adaptation: In addition to the above setup, the test data
can belong to multiple domains which changes over time such that
$D_{t}^{(1)}\neq D_{t}^{(2)}\neq\ldots\neq D_{s}$ leading to the continual
test time adaptation scenario.

Figure 1: pSTarC Framework: (1) Feature Generation: We randomly initialize a
feature bank $\mathbf{f}$ which is iteratively optimized keeping the
classifier $\mathbf{H}$ fixed to minimize the entropy of the features while
maximizing the diversity across classes using the loss in eqn (4). (2) Given
the learnt features, we aim to bring the low entropy samples towards the
corresponding pseudo source features. We anchor the high entropy target
samples to its own prediction. We also enforce consistency between the
predictions of the test sample and its strong augmentation.

##  4 Proposed Framework

The proposed pSTarC framework is based on effectively clustering the target
samples which are available during test time. Our formulation is inspired by
the clustering framework proposed in the state-of-the-art SFDA technique, AaD
[27], which we briefly describe below.

Attracting and Dispersing (AaD): AaD [27] treats SFDA as an unsupervised
clustering problem, where consistency is enforced between predictions of local
neighbourhood features, while also ensuring diversity in the feature space.
The test objective for a sample $x_{i}$ from a test batch $\mathbf{x}_{t}$ is

$\mathcal{L}(x_{i})=-\sum_{x_{j}\in\mathcal{N}_{i}}p_{i}^{T}p_{j}+\lambda\sum_{x_{m}\in\mathbf{x}_{t}}p_{i}^{T}p_{m}$
(1)

where $p_{i}$ refers to the softmax prediction vector of the sample
$x_{i}\in\mathbf{x}_{t}$, $p_{j}$ in the first term corresponds to the
prediction vectors in its neighborhood $\mathcal{N}_{i}$, $p_{m}$ in the
second term corresponds to the prediction vectors of the samples $x_{m}$ in
the current batch $\mathbf{x}_{t}$.

Now, we describe the proposed pSTarC framework for fully TTA task, which we
illustrate in Fig.( 1). In a TTA setting, as mentioned before, the labeled
source samples are unavailable, and only the source model is available for
adaptation. In addition, since the number of samples in a batch is usually
quite low, it is a common practice to freeze the source trained classifier and
update only the feature extractor to align target features with those of the
source. Hence, we set $\mathbf{H_{t}=H_{s}=H}$ and only update the feature
extractor $\mathbf{G_{t}}$ using the test data in an online manner. The goal
is to adapt the test features such that they align with the source features so
that the classifier $\mathbf{H}$ is transferable to test data. The classifier,
being trained in a supervised manner using abundant source data, defines the
decision boundaries for which the source data is perfectly classified. We
leverage this fact to synthesize pseudo-source features, which are used to
guide the target clustering. Given the source model, this process is only done
once to store few features and corresponding prediction scores, and can be
utilized throughout the TTA process. We describe the feature generation and
clustering in detail below.

###  4.1 Pseudo Source Feature Generation

Since the decision boundaries in the feature space remain fixed (due to the
classifiers remaining unchanged), it is important to align the target features
with the original source features, which will inherently lead to better
clustering and hence better classification of the target samples. First, we
utilize the fixed source classifier to synthesize pseudo-source features. By
aligning the target to these generated features, we hope to improve the
adaptation performance of the model and make it more robust to the domain
shift between the source and target domains.

Here, we aim to generate, say $N$ pseudo source features, where $N=C\times
n_{c}$, $C$ being the number of classes and $n_{c}$ is the number of samples
per class. We first randomly initialize a feature bank
$\mathbf{f}\in\mathcal{R}^{N\times d}$, where $d$ is the feature dimension. To
compute the pseudo-source features, we use the information maximization loss
which is a combination of entropy minimization and diversity maximization.
These losses have been widely used in unsupervised clustering methods [13] to
optimize a feature extractor to make the predictions of unlabeled samples
diverse and confident. However, our objectives are very different. While they
aim to learn a good feature extractor, our goal is to synthesize pseudo-source
features given the source trained classifier $\mathbf{H}$. We want to generate
features which are likely to be correctly classified by the source classifier.
This is achieved by minimizing the following entropy loss:

$\mathcal{L}_{ent}\left(\mathbf{f};\mathbf{H}\right)=-\frac{1}{N}\sum_{i=1}^{N}\sum_{k=1}^{C}\delta_{k}\left(\mathbf{H}\left(f_{i}\right)\right)\log\delta_{k}\left(\mathbf{H}\left(f_{i}\right)\right)$
(2)

where $\delta_{k}\left(\mathbf{H}\left(f_{i}\right)\right)$ is the softmax
score of class $k$ for the pseudo source feature $f_{i}\in\mathbf{f}$.

Along with this, we use diversity maximization loss to avoid the trivial
solution where all feature vectors collapse to the same class. This ensures
there are adequate number of feature vectors from each class in $\mathbf{f}$.

$\displaystyle\mathcal{L}_{div}\left(\mathbf{f};\mathbf{H}\right)$
$\displaystyle=\sum_{k=1}^{C}\hat{p}_{k}\log\hat{p}_{k}$ (3)
$\displaystyle=D_{KL}\left(\hat{p},\frac{1}{C}\mathbf{1}_{C}\right)-\log C$

The loss is computed based on the mean softmax score of the test batch
$\hat{p}=\mathbb{E}_{f\in\mathbf{f}}\left[\delta\left(h\left(f\right)\right)\right]$.
The first term in the equation is the Kullback-Leibler (KL) divergence between
the mean prediction vector $\hat{p}$ and the uniform distribution
$\frac{1}{C}\mathbf{1}_{C}$. Here, $\hat{p}$ represents the marginal class
distribution of the target data as estimated by the target model
$\mathbf{F_{t}}$, $C$ is the number of classes and $\mathbf{1}_{C}$ is a
vector of ones with length $C$. The KL divergence measures the dissimilarity
between two probability distributions, and in this context, it measures the
discrepancy between the class distribution in the feature bank and the ideal
case where all classes are equally represented. Overall, the diversity
maximization loss encourages the feature bank to have a balanced
representation of features across all classes, which is important for
improving the clustering performance of the TTA algorithm. To summarize, we
optimize the following

$\mathbf{f}^{*}=\operatorname*{arg\,min}_{\mathbf{f}}\mathcal{L}_{ent}(\mathbf{f};\mathbf{H})+\beta\mathcal{L}_{div}(\mathbf{f};\mathbf{H})$
(4)

In Fig.( 2), we visualize the generated features on setting 20 samples per
class for VisDA dataset.

Figure 2: t-SNE plot of 240 generated pseudo-source features for TTA on VisDA
dataset comprising of 12 classes.

###  4.2 Pseudo Source Guided Target Clustering

The use of feature bank has proven to be effective in Contrastive learning [5]
and SFDA methods like AaD [27] and AdaContrast [2]. The proposed feature bank
consists of pseudo-source features which are very different from the target
feature bank used in [27, 2]. Unlike target features whose pseudo labels can
be noisy, we can obtain clean labels for the generated pseudo-source features.
We explain below how the generated features and their label information can be
leveraged to better cluster and align the target features. We visually
demonstrate the entire pSTarC framework in Fig.( 1).

Pseudo-labeling based on confidence thresholding has been used very
effectively in several applications [20]. Here, we propose a soft pseudo-
labeling approach to cluster the target samples. Specifically, we identify the
low entropy test samples based on a threshold $\tau_{t}$, which we define as
the mean entropy of the batch. We aim to align these selected test samples to
the nearest pseudo-source samples which belong to the same class as the
sample. Formally, given the generated feature bank $\mathbf{f}^{*}$, we first
obtain their softmax score vectors and pseudo labels. We denote
$p_{i}=\delta(\mathbf{H}(f_{i}))$ as the softmax score vector and
$\hat{y}_{i}=\operatorname*{arg\,max}_{c}p_{i,c}$ as the pseudo label for
feature $f_{i}$, where $p_{i,c}$ is the score of feature $i$ for class $c$. We
partition the features into sets $S_{c}$ based on their pseudo labels as
follows:

$S_{c}=\\{f_{i};\quad\hat{y}_{i}=c,f_{i}\in\mathbf{f}\\};\quad c\in\\{1\ldots
C\\}$ (5)

These sets are obtained once for the pseudo source features generated and kept
fixed throughout the adaptation process.

Given a test batch $\mathbf{x}_{t}$, we first obtain their confidence scores
and pseudo labels and set the threshold
$\tau_{t}=\mathbb{E}_{x_{k}\in\mathbf{x}_{t}}[e_{k}]$, the mean entropy of the
batch. For a test sample $x_{k}\in\mathbf{x}_{t}$ (test batch), we denote its
pseudo label as $\hat{y}_{k}$ and compute the sample entropy as $e_{k}$ For
this sample, we define its positive set $\mathbf{p}^{+}$ based on its entropy
$e_{k}$ as follows: (1) When $e_{k}<\tau_{t}$, we define the positives to be
$K$ nearest pseudo-source samples from set $S_{\hat{y}_{k}}$. (2) For samples
which have high entropy, i.e, with $e_{k}>\tau_{t}$, as the pseudo labels can
be highly noisy, it is not desirable to enforce them to align towards any
pseudo-source samples. Instead, we anchor it to its own prediction vector by
setting $\mathbf{p}^{+}=\\{p_{k}\\}$. In addition, we use its strong image
augmentation $\tilde{x}_{k}$ to enforce prediction consistency between $p_{k}$
and $\tilde{p}_{k}$, the prediction vector of $\tilde{x}_{k}$. This helps the
model to be invariant to image transformations and improves its generalization
ability. We also use the dispersion loss that makes a sample dissimilar to the
other samples in the batch, which is representative of the test data in all.
This dispersion loss prevents the model from the trivial solution of all test
samples collapsing to the same class. Our objective now is to make the
predictions of the target embeddings similar to its positives without facing
mode collapse, which we achieve by optimizing the following loss:

$\mathcal{L}_{\textrm{pSTarC}}(x_{k})=\underbrace{\vphantom{\sum_{p_{j}^{+}\in\mathbf{p}^{+}}p_{k}^{T}p_{j}^{+}}-p_{k}^{T}\tilde{p}_{k}}_{\text{$L_{aug}$}}-\underbrace{\sum_{p_{j}^{+}\in\mathbf{p}^{+}}p_{k}^{T}p_{j}^{+}}_{\text{$L_{attr}$}}+\underbrace{\vphantom{\sum_{p_{j}^{+}\in\mathbf{p}^{+}}p_{k}^{T}p_{j}^{+}}\lambda\sum_{x_{j}\in\mathbf{x}_{t}}p_{k}^{T}p_{j}}_{\text{$L_{disp}$}}$
(6)

We perform one step optimization on test batch $\mathbf{x}_{t}$ using this
loss and then predict their labels. This process is repeated for each batch in
the TTA setting.

Method plane bycyl bus car horse knife mcycl person plant sktbrd train truck
Average Source 57.2 11.1 42.4 66.9 55.0 4.4 81.1 27.3 57.9 29.4 86.7 5.8 43.8
CAN∗ [9] 95.7 88.8 6.9 68.6 94.5 94.8 79.2 70.3 88.7 80.6 83.2 51.7 75.2 MCC∗
[8] 93.9 78.4 70.4 74.3 92.5 84.2 84.5 58.2 86.6 36.0 86.1 20.6 72.2 Source-
Proxy TTA∗ [10] 92.5 82.4 85.8 74.2 92.7 88.5 83.9 85.8 92.8 62.5 75.2 32.5
79.1 BN-Adapt[19] 87.3 52.1 83.7 52.8 83.7 57.0 83.6 59.2 69.1 54.7 80.0 28.1
66.0 TENT[22] 91.1 45.6 86.4 66.4 88.7 75.1 90.3 76.4 84.4 47.1 83.6 13.7 70.7
AdaContrast [2] 95.0 68.0 82.7 69.6 94.3 80.8 90.3 79.6 90.6 69.7 87.6 36.0
78.7 C-SFDA [11] 95.9 75.6 88.4 68.1 95.4 86.1 94.5 82.0 89.2 80.2 87.3 43.8
82.1 pSTarC 95.1 82.1 83.6 61.2 93.8 89.9 87.9 80.7 90.9 81.9 87.6 48.1 81.9

Table 1: Average class accuracy (%) of pSTarC and other TTA methods on VisDA.
∗ refers to methods utilizing source data to enable TTA.

Method A $\rightarrow$ C A $\rightarrow$ P A $\rightarrow$ R C $\rightarrow$ A
C $\rightarrow$ P C $\rightarrow$ R P $\rightarrow$ A P $\rightarrow$ C P
$\rightarrow$ R R $\rightarrow$ A R $\rightarrow$ C R $\rightarrow$ P Average
Source 44.6 66.5 73.5 51.0 61.9 63.2 51.1 40.5 71.9 64.4 47.1 77.3 59.4 BN-
Adapt [19] 38.9 59.9 71.5 55.0 62.0 65.2 54.4 37.3 71.6 65.2 41.3 73.8 58.0
TENT [22] 39.1 60.2 71.6 55.2 62.2 65.5 54.6 37.6 71.8 65.3 41.6 73.9 58.2
AdaContrast [2] 42.2 64.5 73.2 56.2 64.1 66.4 54.7 40.4 73.0 66.7 45.1 75.6
60.2 pSTarC 47.7 68.7 75.4 58.6 68.4 68.9 55.1 45.8 75.6 67.5 51.8 78.7 63.5

Table 2: Total accuracy (%) of pSTarC and other fully TTA methods on Office-
Home dataset.

What makes pSTarC an effective framework?

1) We operate in the fully test-time scenario, i.e., we do not assume access
to source data in any form unlike some prior methods [10, 9, 8], which use the
source data to equip the model for future TTA. In pSTarC, we leverage the
classifier which is a part of the given source model to synthesize pseudo
source features to enable clustering during test time.
2) Feature banks have been effectively used in AdaContrast[2] to cluster the
test data. However, it is expensive to have multiple large memory buffers
which have to be continuously updated. We propose a simple one-step pseudo
source generation framework. These generated features can be used during TTA
forever, as the final goal indeed is to align the test distribution to the
source distribution.
3) pSTarC is a memory efficient framework as we only store the online updating
model, in contrast to AdaContrast[2], C-SFDA[11] where they need to store the
student and teacher model. Our framework is also more efficient in runtime as
we only forward pass the image and its strong augmentation, while the SOTA
method C-SFDA [11] uses 12 augmentations.

##  5 Experimental Evaluation

We evaluate the proposed framework extensively on three real-world domain
shift datasets, namely VisDA [17], DomainNet-126 [16] and Office-Home [21] and
also on a corruption benchmark dataset, namely CIFAR100C.

Method R$\rightarrow$C R$\rightarrow$P P$\rightarrow$C C$\rightarrow$S
S$\rightarrow$P R$\rightarrow$S P$\rightarrow$R Average Source 55.5 62.7 53
46.9 47.3 46.3 75.0 55.2 BN-Adapt [19] 54.1 62.8 54.3 49.4 59.1 47.6 75.0 57.5
TENT [22] 55.6 64.5 55.5 50.8 59.9 49.9 75.9 58.9 AdaContrast [2] 61.1 66.9
60.8 53.4 62.7 54.5 78.9 62.6 C-SFDA [11] 61.6 67.4 61.3 55.1 63.2 54.8 78.5
63.1 pSTarC 60.8 67.7 60.3 55.6 65.3 55.8 80.2 63.7

Table 3: Total accuracy (%) of TTA methods on DomainNet-126.

Method gaussian shot impulse defocus glass motion zoom snow frost fog
brightness contrast elastic pixelate jpeg Average Source 27 32 60.6 70.7 45.9
69.2 71.2 60.5 54.2 49.7 70.5 44.9 62.8 25.3 58.8 53.6 BN Adapt [19] 57.9 59.3
57.3 72.4 58.1 70.3 72.1 65.1 65 58.5 73.5 69.7 64.3 67.1 58.8 64.6 TENT [22]
62.7 65.1 65.5 75.0 62.6 72.5 75.0 69.6 68.1 66.2 76.0 71.8 67.1 71.6 63.1
68.8 AdaContrast [2] 57.3 59.4 61.1 73.4 58.8 71.1 73.4 66.6 67.3 60.7 75.2
71.8 65.4 65.8 60.5 65.9 pSTarC 63.4 65.4 66.5 75 63 73.2 74.9 70.3 69.8 66.5
76.6 73.2 68.0 72.2 63.8 69.5

Table 4: Accuracy (%) of different TTA methods on 15 corruptions from
CIFAR-100C dataset in TTA setting.

Method gaussian shot impulse defocus glass motion zoom snow frost fog
brightness contrast elastic pixelate jpeg Average Source 27 32 60.6 70.7 45.9
69.2 71.2 60.5 54.2 49.7 70.5 44.9 62.8 25.3 58.8 53.6 BN Adapt [19] 57.9 59.3
57.3 72.4 58.1 70.3 72.1 65.1 65 58.5 73.5 69.7 64.3 67.1 58.8 64.6 TENT [22]
62.8 64.2 58.3 62.1 48.8 51.7 51.5 41.6 36.3 28.9 29.6 17.7 12.0 11.5 9.6 39.1
CoTTA [24] 59.9 62.3 60.3 73.1 62.0 72.1 73.6 67.2 68.2 59.7 75.3 73.1 67.5
71.7 66.5 67.5 AdaContrast [2] 57.7 63.2 61.4 72.3 59.9 70.9 72.5 67.1 69.3
61.8 74.1 71.7 66.1 66.7 63.8 66.6 RMT [4] 59.5 63.9 63.7 72.3 66.1 71.5 73.6
71.0 71.0 67.5 74.9 72.6 71.8 73.7 70.7 69.6 pSTarC 63.4 67.0 64.0 71.1 62.9
69.3 72.4 67.3 68.7 64.1 72.9 71.9 66.7 70.5 62.9 67.7

Table 5: Accuracy (%) of different methods on 15 corruptions from CIFAR-100C
dataset in CTTA setting.

Datasets: VisDA is a challenging dataset for object recognition tasks with
synthetic to real domain shift. The target domain consists of $55,388$ real
object images from $12$ classes. Office-Home contains four domains - Real,
Clipart, Art, Product and $65$ classes with a total of $15,500$ images.
DomainNet-126 is a subset of DomainNet consisting of $126$ classes from four
domains, namely Real, Sketch, Clipart and Painting. CIFAR-100C is a corruption
benchmark with domain shifts like gaussian noise, blur, weather changes, etc.
Following [24], we use severity level 5 corruptions. For VisDA-C, we compare
the average of per-class accuracies while for the other datasets, we compare
the average of the total accuracy across domain shifts.

For TTA experiments, we use ResNet-50[6] as the backbone for Office-Home and
DomainNet-126 datasets and ResNet-101[6] for the VisDA dataset. We use the
same network architecture as in [2], in which the final part of the network is
modified to include fully connected layer and Batch Normalization, and then
followed by a classifier, which is a fully connected layer with weight
normalization. For CIFAR-100C , we use ResNeXt[25] as used in [24, 4].

Implementation details: We use Pytorch framework and run all experiments on a
single NVIDIA A-5000 GPU. For source training, following [2, 11] the model is
initialized with ImageNet pre-trained weights and trained for $10$, $60$ and
$50$ epochs for VisDA, DomainNet-126 and Office-Home respectively. During test
time adaptation, we only update the backbone parameters, keeping the
classifier fixed for all experiments. Following [2, 10, 11] for TTA and [24,
4] for CTTA, we set the batch size to $128$ in all experiments. We use SGD as
the optimizer with learning rate of 5e-4 and momentum $0.9$. Following [24, 4]
for CTTA on CIFAR-100C, the batch size is set to $200$ and we use Adam
optimizer with learning rate of 1e-3. We set $\beta$ to $5$ in eqn.(4) and the
number of features per class $n_{c}$ to 20 in all experiments. We have
reported the results of the other approaches directly from the respective
papers. We use the official code provided by AdaContrast to perform
experiments on Office-Home dataset and also adapt it to CTTA setting. In the
Supplementary material, we describe the image augmentations used, analysis on
parameter $n_{c}$ and provide the pseudo code for pSTarC algorithm.

Method Real$\rightarrow$ Clipart$\rightarrow$ Painting$\rightarrow$
Sketch$\rightarrow$ Average Source only 54.7 50.7 58.3 55.2 54.7 BN Adapt [19]
54.9 54.8 60.5 62.2 58.1 TENT [22] 57.6 55.8 62.8 62.5 59.7 CoTTA [24] 56.6
57.0 63.6 63.7 60.2 AdaContrast [2] 62.2 62.4 67.7 68.1 65.1 RMT [4] 63.0 62.1
68.3 67.9 65.3 pSTarC 62.7 63.6 67.6 68.1 65.5

Table 6: Accuracy (%) of different TTA methods on four domain shift sequences
from DomainNet-126 in CTTA setting.

###  5.1 Evaluation for TTA setting

We compare the performance of our proposed pSTarC framework with the SOTA TTA
approaches [19, 22, 2, 11, 10]. For VisDA dataset, from Table 1, we observe
that pSTarC performs at par with the SOTA method C-SFDA [11], while being
computationally much more efficient (as discussed later). Interestingly, it
also outperforms the approaches which assume access to the source data before
performing TTA. On Office-Home, we get a significant improvement of 3.5%
compared to the prior TTA method AdaContrast [2] as shown in Table 2. On
DomainNet-126, from Table 3, we observe that pSTarC achieves an average
accuracy of 63.7% across 7 domain shifts, outperforming all the existing
approaches including [11]. On CIFAR-100C [7], our method performs 1.1% better
than TENT [22] and 3.6% better than AdaContrast [2], suggesting its
effectiveness even on corruption domain shifts (Table 4).

###  5.2 Evaluation for CTTA setting

We also study the effectiveness of pSTarC in the CTTA setting where test
domains change with time. To do this, we perform experiments on CIFAR-100C and
the following four domain sequences from DomainNet-126:
(i) Real-World$\rightarrow$Clipart$\rightarrow$Painting$\rightarrow$Sketch;
(ii) Clipart$\rightarrow$Sketch$\rightarrow$Real-World$\rightarrow$Painting;
(iii) Painting$\rightarrow$Real-World$\rightarrow$Sketch$\rightarrow$Clipart
(iv) Sketch$\rightarrow$Painting$\rightarrow$Clipart$\rightarrow$Real-World.
The first domain indicates the source domain, which is then adapted to the
other three test domains in the above sequence. From Table 6, we observe that
pSTarC outperforms all the state-of-the-approaches for this very challenging
task. Specifically, it outperforms CoTTA by a significant margin of 5.3% and
also performs favourably compared to the SOTA RMT [4]. In addition, we also
evaluate pSTarC on CIFAR-100C continual setting and report the results in
Table 5. It performs favourably compared to the recent AdaContrast [2] and
CoTTA [24] approaches, while RMT [4] performs the best in this case. But,
CoTTA [24] and RMT [4] are computationally more expensive as they need to
store teacher and student models, while pSTarC is more light-weight as it only
stores one model.

In Figure 3, we summarize the performance of pSTarC with the source model,
TENT [22] and Adacontrast [2]. In this plot, the lines farther from the center
indicates better performance. We observe that pSTarC outperforms these methods
across all domain shifts for both TTA and CTTA.

###  5.3 Additional Analysis

Here, we report the results of additional analysis to better understand the
proposed framework.

1\. Ablation Study: The proposed pSTarC framework consists of three loss
components. The first component is $L_{aug}$ which enforces consistency
between an image and its augmentation. From Table 7, we observe that using
strong augmentations can indeed help improve the feature representations, as
we get 1.9% and 0.7% improvement on VisDA and DomainNet-126 respectively. The
second component $L_{attr}$ aims to align the test features with the pseudo
source features. On removing the attraction loss component from $L_{pSTarC}$,
the loss becomes similar to contrastive learning. While this performs
reasonably, achieving 78.2% and 59.7% on VisDA and DomainNet respectively,
incorporating the pseudo-source features improves the results significantly by
3.7% and 4%, proving that they indeed help model adaptation by correctly
aligning the test features so that the source trained classifier can well
classify the test data. The third component, $L_{disp}$ is the dispersion term
which prevents the model to avoid all the test features collapsing to one
cluster, which is a trivial solution when optimizing only the attraction loss
$L_{attr}$. This term plays a role similar to the diversity term and is
crucial in any unsupervised adaptation protocols [13, 27] to avoid model
collapse, the effect of which we observe in Table 7. The accuracy on VisDA and
DomainNet-126 drop to 68.8% and 58.8% respectively, as the test samples would
be predicted into lesser number of classes than actually present in the
dataset.

2\. Performance on varying batch sizes: In TTA, it is crucial for the method
to be able to continuously adapt even with very few samples. In this analysis,
we vary the batch size from $8$ to $128$ and perform experiments on the
DomainNet-126 dataset. Table8 reports the average accuracy across $7$ domain
shifts for each batch size. We observe that the proposed pSTarC consistently
outperforms both TENT [22] and AdaContrast [2] for all batch sizes. The effect
is more pronounced for the smallest batch size $8$, where pSTarC outperforms
TENT by a huge margin of 15.3% and AdaContrast by 4%. On average, pSTarC does
better than TENT by 6.2% and AdaContrast by 1.7%.

$L_{aug}$ | $L_{attr}$ | $L_{disp}$ | VisDA | DomainNet-126
---|---|---|---|---
✓ | ✓ |  | 68.8 | 58.8
✓ |  | ✓ | 78.2 | 59.7
| ✓ | ✓ | 80.0 | 63.0
✓ | ✓ | ✓ | 81.9 | 63.7
Table 7: pSTarC ablation study: Importance of each loss term. Method | Batch size | Average
---|---|---
8 | 16 | 32 | 64 | 128
TENT | 38.8 | 55.4 | 58.6 | 59.1 | 58.9 | 54.2
AdaContrast | 50.1 | 57.9 | 60.8 | 62.4 | 62.4 | 58.7
pSTarC | 54.1 | 59.2 | 61.3 | 63.8 | 63.7 | 60.4
Table 8: Ablation on batch size using DomainNet-126

DomainNet126-CTTA

CIFAR-100C-CTTA

Office-Home

DomainNet126-TTA

VisDA

CIFAR-100C-TTA

|  Source
---|---
|  TENT
|  AdaContrast
|  pSTarC
Figure 3: Overall comparison of pSTarC with TTA methods.

3\. Complexity Analysis: Here, we analyse the complexity of pSTarC and three
other recent TTA methods: AdaContrast [2], Source-Proxy-TTA [10] and C-SFDA
[11] on VisDA dataset. In the TTA setting, it is desirable to have methods
that requires storing less additional information due to memory limitations
and privacy concerns. The prior methods AdaContrast [2] and C-SFDA [11] are
based on the teacher student framework. Hence, it needs to store twice the
number of model parameters, while we only store the updating model parameters
in pSTarC, as we report in Table 9. AdaContrast stores a memory queue of size
$16384$ to collect key features (of dimension 256), and its pseudo labels,
which is used to retrieve positives for contrastive learning. Alongside, they
store another feature bank (of size 1024) and their corresponding scores which
is used to retrieve neighbours for soft pseudo-labeling the target samples.
Thus, the total memory buffer required for AdaContrast is
16384x(256+1)+1024x(256+12). [10] condenses the source data to save 25 images
per class of size $112x112$ for VisDA dataset. This accounts to a memory
requirement of $3763200$ (12x25x112x112). On the other hand, in the pSTarC
framework, we only store $20$ features per class and the corresponding scores
resulting in a memory buffer of $240\text{x}(256+12)$. C-SFDA does not store
any features or images. However, they need $13$ forward passes (12
augmentations in addition to the actual test sample), while AdaContrast [2]
and Source-Proxy-TTA [10] uses 3 augmentations, and pSTarC uses only two
augmentations. We summarize this in Table 9, which shows that pSTarC is very
efficient, in addition to achieving better or comparable performance as the
SOTA across several challenging settings.

Method AdaContrast Source-Proxy-TTA C-SFDA pSTarC #Parameters 86M 43M 86M 43M
Memory 4665344 3763200 - 32160 #Forward 3 3 13 2 #Backward 1 1 1 1

Table 9: Complexity Analysis of TTA methods on VisDA

##  6 Conclusion

In this paper, we have proposed a novel framework termed pSTarC for Test Time
Adaptation (TTA) of deep neural networks. pSTarC leverages the fixed source
classifier to generate pseudo-source samples, which is then used to align the
test samples, which enables the source trained classifier to classify test
data from different distributions. Extensive experiments on several real-world
domain shift datasets justify the effectiveness of our proposed framework.
Additionally, we also show that the method can seamlessly be used in continual
test time adaptation scenario, though there is still scope for improvement in
the corruption datasets. Overall, our findings highlight the importance of
target clustering techniques and leveraging the source classifier for
improving test-time adaptation performance in several real-world challenging
scenarios.

## References

  * [1] Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto.  Parameter-free online test-time adaptation.  In CVPR, pages 8344–8353, 2022.
  * [2] Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi.  Contrastive test-time adaptation.  In CVPR, pages 295–305, 2022.
  * [3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  arXiv preprint arXiv:2002.05709, 2020.
  * [4] Mario Döbler, Robert A Marsden, and Bin Yang.  Robust mean teacher for continual and gradual test-time adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7704–7714, 2023.
  * [5] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In CVPR, pages 9729–9738, 2020.
  * [6] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * [7] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  arXiv preprint arXiv:1903.12261, 2019.
  * [8] Ying Jin, Ximei Wang, Mingsheng Long, and Jianmin Wang.  Minimum class confusion for versatile domain adaptation.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16, pages 464–480. Springer, 2020.
  * [9] Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G Hauptmann.  Contrastive adaptation network for unsupervised domain adaptation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4893–4902, 2019.
  * [10] Juwon Kang, Nayeong Kim, Kwon Donghyeon, Jungseul Ok, and Suha Kwak.  Leveraging proxy of training data for test-time adaptation.  In International Conference on Machine Learning (ICML), July 2023\.
  * [11] Nazmul Karim, Niluthpol Chowdhury Mithun, Abhinav Rajvanshi, Han-pang Chiu, Supun Samarasekera, and Nazanin Rahnavard.  C-sfda: A curriculum learning aided self-training framework for efficient source free domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24120–24131, 2023.
  * [12] Yujin Kim, Dogyun Park, Dohee Kim, and Suhyun Kim.  Naturalinversion: Data-free image synthesis improving real-world consistency.  In AAAI, volume 36, pages 1201–1209, 2022.
  * [13] Jian Liang, Dapeng Hu, and Jiashi Feng.  Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation.  In ICML, 2020.
  * [14] Hyesu Lim, Byeonggeun Kim, Jaegul Choo, and Sungha Choi.  Ttn: A domain-shift aware batch normalization in test-time adaptation.  In ICLR.
  * [15] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan.  Efficient test-time model adaptation without forgetting.  In ICML, pages 16888–16905. PMLR, 2022.
  * [16] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.  Moment matching for multi-source domain adaptation.  In ICCV, pages 1406–1415, 2019.
  * [17] Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko.  Visda: The visual domain adaptation challenge.  arXiv preprint arXiv:1710.06924, 2017.
  * [18] Mozhdeh Rouhsedaghat, Masoud Monajatipoor, Kai-Wei Chang, C-C Jay Kuo, and Iacopo Masi.  Magic: Mask-guided image synthesis by inverting a quasi-robust classifier.  arXiv preprint arXiv:2209.11549, 2022.
  * [19] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge.  Improving robustness against common corruptions by covariate shift adaptation.  NeurIPS, 2020.
  * [20] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  NeurIPS, 2020.
  * [21] H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan.  Deep hashing network for unsupervised domain adaptation.  In CVPR, 2017.
  * [22] D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell.  Tent: Fully test-time adaptation by entropy minimization.  In ICLR, 2021.
  * [23] Pei Wang, Yijun Li, Krishna Kumar Singh, Jingwan Lu, and Nuno Vasconcelos.  Imagine: Image synthesis by image-guided model inversion.  In CVPR, pages 3681–3690, 2021.
  * [24] Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai.  Continual test-time domain adaptation.  In CVPR, 2022.
  * [25] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In CVPR, pages 1492–1500, 2017.
  * [26] Shiqi Yang, Joost van de Weijer, Luis Herranz, Shangling Jui, et al.  Exploiting the intrinsic neighborhood structure for source-free domain adaptation.  NeurIPS, 34:29393–29405, 2021.
  * [27] Shiqi Yang, Yaxing Wang, Kai Wang, Shangling Jui, et al.  Attracting and dispersing: A simple approach for source-free domain adaptation.  In NeurIPS, 2022.