# CAFA: Class-Aware Feature Alignment for
Test-Time Adaptation

Sanghun Jung1 Jungsoo Lee1, 2 Nanhee Kim3 Jaegul Choo1
1KAIST AI, 2Kakao Enterprise, 3Enssel Inc., South Korea
1{shjung13, bebeto<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Despite recent advancements in deep learning, deep networks still suffer from
performance degradation when they face new and different data from their
training distributions. Addressing such a problem, test-time adaptation (TTA)
aims to adapt a model to unlabeled test data on test time while making
predictions simultaneously. TTA applies to pretrained networks without
modifying their training procedures, which enables to utilize the already
well-formed source distribution for adaptation. One possible approach is to
align the representation space of test samples to the source distribution
(i.e., feature alignment). However, performing feature alignments in TTA is
especially challenging in that the access to labeled source data is restricted
during adaptation. That is, a model does not have a chance to learn test data
in a class-discriminative manner, which was feasible in other adaptation tasks
(e.g., unsupervised domain adaptation) via supervised loss on the source data.
Based on such an observation, this paper proposes _a simple yet effective_
feature alignment loss, termed as Class-Aware Feature Alignment (CAFA), which
1) encourages a model to learn target representations in a class-
discriminative manner and 2) effectively mitigates the distribution shifts in
test time, simultaneously. Our method does not require any hyper-parameters or
additional losses, which are required in the previous approaches. We conduct
extensive experiments and show our proposed method consistently outperforms
existing baselines.

##  1 Introduction

Recent advancements [1, 2, 3, 4] in machine learning have demonstrated their
effectiveness in solving diverse problems, achieving remarkable performance
enhancements on benchmarking datasets. However, they suffer from significant
performance degradation when they face test data with different conditions
from their training data (i.e., source data), such as corruptions [5], light
changes [6], or adverse weather [7, 8]. Such sensitivity to distribution
shifts [9] hampers deep networks from performing well in the practical
scenarios since test samples may include such distribution shifts [10]. Thus,
adapting deep models to the test samples is crucial, especially when there
exist distribution shifts.

To alleviate such problems, various adaptation methods have been proposed [11,
12, 13, 14, 15, 16, 17, 18]. However, most of them require either access to
the source data during adaptation or modification of the training procedure,
which limits their applicability (e.g., data privacy issues limit the use of
source data). Addressing such concerns, we pursue to design an adaptation
method that 1) is applicable to existing deep networks without modification
and 2) does not require access to the source data during adaptation. In order
to satisfy such conditions, previous studies perform adaptation on test time
while making predictions simultaneously, which is referred to as _test-time
adaptation_ (TTA).

One of the widely adopted approaches to address distribution shifts is
aligning the source (i.e., train data) and target (i.e., test data)
distributions [14, 15, 19, 20, 16, 21]. DANN [14] directly reduces
$\mathcal{H}$-divergence between the source and target distributions, and
CORAL [15] minimizes the difference of the second-order statistics between the
source and target data. Despite their demonstrated effectiveness in other
adaptation tasks, applying those alignments to TTA bears the following
limitation. They generally perform alignments with supervised losses on the
source data, which encourages a model to learn target distributions in a
class-discriminative manner [15]. However, access to the source data is
prohibited in TTA, limiting a model from learning class-discriminability.

Considering such an issue, we conduct an analysis on the effects of feature
alignments in TTA by using two distances in the representation space: intra-
class distance and inter-class distance. As shown in Fig. 1 (c), intra-class
distance (dotted arrow) is defined as the distance between a sample and its
ground-truth source class distribution, and inter-class distance (solid arrow)
denotes the averaged distance between the sample and other source class
distributions. Obviously, achieving low intra-class distance and high inter-
class distance is crucial for improving classification accuracy [22, 23, 24,
25, 26]. We first align the target distribution to the source distribution
without considering class information (Fig. 1 (a) (red lines)). We term this
alignment as global feature alignment in our paper. As shown, intra-class
distance is reduced, which is desirable, but it also accompanies the decrease
of inter-class distance. Such effects degrade the image classification
accuracy in Fig. 1 (b) (red line). This is mainly because there is no chance
for a model to learn the test data in a class-discriminative manner since a
supervised loss is not available both on the source and target data. To
improve the class-discriminability of a model, we further consider the class
information by 1) enlarging the inter-class distance and 2) reducing the
intra-class distance as shown in Fig. 1 (a) (blue lines). Such an approach
significantly improves the classification accuracy as adaptation proceeds, as
illustrated in Fig. 1 (b) (blue line).

Figure 1: A motivating example of our paper. (a) shows the change of the
intra-class distance (dotted lines) and the inter-class distance (solid lines)
of ours (blue line) and global feature alignment (red line). (b) shows the
accuracy changes as adaptation proceeds. (c) illustrates how our method CAFA
aligns the test features to source class-conditional distributions. Plots are
obtained from the Gaussian noise corruption with severity level 5 in the
CIFAR10-C dataset.

Motivated by such observations, we propose Class-Aware Feature Alignment
(CAFA) which aligns the target features to the pre-calculated source feature
distribution by considering both intra- and inter-class distances. To be more
specific, we pre-calculate the statistics (i.e., mean and covariance) of the
source distribution to estimate class-conditional Gaussian distributions from
source samples using a pretrained network. In test-time adaptation, we utilize
the Mahalanobis distance to 1) align each sample to its predicted class-
conditional Gaussian distribution (i.e., reduce intra-class distance) and 2)
enforce samples to be distinct from other class-conditional Gaussian
distributions (i.e., increase inter-class distance).

The main contributions of our work are as follows:

  * •

We propose a novel Class-Aware Feature Alignment (CAFA) that effectively
mitigates the distribution shifts and encourages a model to learn
discriminative target representations simultaneously.

  * •

Our proposed approach is simple yet effective, not requiring hyper-parameters
or additional modifications of the training procedure.

  * •

We conduct extensive experiments along with in-depth analysis and show that
CAFA consistently outperforms the existing baselines models on test-time
adaptation.

##  2 Related Work

###  2.1 Unsupervised Domain Adaptation

Unsupervised domain adaptation (UDA) approaches [27, 28, 11, 12, 13, 29, 30,
31] have addressed distribution shifts effectively by utilizing various
strategies such as adversarial training [14, 16, 21, 32, 32, 33] or minimizing
the discrepancy [34, 35, 36]. However, they still assume the source data is
available during adaptation, which is recently addressed by source-free UDA
approaches [37, 38, 17, 39, 40, 41, 42, 43, 44, 18]. SHOT [37] utilizes the
information maximization [45] loss for adaptation, and G-SFDA [46] performs
local structure clustering along with domain-specific attention. While the
previous source-free UDA approaches have demonstrated their effectiveness,
they necessitate altering the training procedure. On the other hand, our
proposed method CAFA neither requires access to the source data during
adaptation nor modifies the training procedure, making our method widely
applicable.

###  2.2 Test-time Adaptation

Unlike UDA methods that adapt to target distribution during the training
phase, approaches adapting to target distributions at the test time have been
proposed [47, 48, 49, 50, 51, 52, 53, 54]. Without accessing to the source
data, test-time training methods [53, 54] train a model with self-supervised
tasks (e.g., predicting the rotation degree) and utilize them as proxy losses
for adaptation. However, as also pointed out in Wang et al. [47], there is no
guarantee that optimizing proxy losses helps in terms of the main task since
they are not directly related to classifying images into different classes.
Furthermore, they iterate several epochs over test samples and predict all the
test data after the iteration is done. Addressing those concerns, test-time
adaptation methods [47, 48, 49, 50, 51] have been proposed. They adapt a model
to test data while making predictions simultaneously. These approaches do not
require the modification of the training procedure. Due to such an adaptation
setting, those algorithms are applicable to a given pretrained deep learning
network. Recent seminal work in TTA, TENT [47], proposed to update the
parameters in batch normalization [55] layers and statistics while minimizing
the entropy loss. Our method CAFA follows the setting of test-time adaptation.

###  2.3 Feature Alignments

Feature alignment is widely adopted in UDA studies to mitigate distribution
shifts [20, 16, 19, 21]. However, most of them do not consider categorical
information, but rather match the source and target distributions globally.
This may harm the class-discriminability since it does not guarantee the
class-to-class matching between two distributions [23]. Pointing out such an
issue, several approaches [23, 24, 56, 57, 25, 58, 59] proposed to align the
distributions in a class-discriminative manner. This point of view also holds
in the case of test-time adaptation, and we design a simple loss that
effectively mitigates the distribution gap while improving class-
discriminability, simultaneously.

##  3 Preliminary

Assume that we have a model $f_{s}(x)=h_{s}\circ{g_{s}}(x)$ pretrained with a
supervised loss $\mathcal{L}(x_{s},y_{s})$ on source data. Here
$g_{s}:\mathcal{X}_{s}\rightarrow\mathbb{R}^{d}$ denotes the pretrained
feature extractor, and $h_{s}:\mathcal{R}^{d}\rightarrow\mathbb{R}^{C}$
indicates the pretrained classifier, where $d$ is the dimension of extracted
features, and $C$ is the number of classes. Then, we aim to adapt the
pretrained model $f_{s}(\cdot)$ to target data $(x_{t},)$ while correctly
classifying them during test time.

Mahalanobis distance In this work, we adopt the Mahalanobis distance to align
the source and target distributions. Mahalanobis distance measures the
distance between a distribution and a sample. With an input image $x$, feature
extractor $g(\cdot)$, and Gaussian distribution
$\mathcal{N}(\boldsymbol{\mu},\boldsymbol{\Sigma})$, Mahalanobis distance is
defined as

$D(x;\boldsymbol{\mu},\boldsymbol{\Sigma})=(g(x)-\boldsymbol{\mu})^{\top}\boldsymbol{\Sigma}^{-1}(g(x)-\boldsymbol{\mu}).$
(1)

Intra-/inter-class distance For an in-depth analysis, we measure the intra-
and inter-class distances between the class-conditional source distributions
and target samples. To be more specific, we define class-conditional Gaussian
distributions as
$P(g_{s}(x)|y=c)=\mathcal{N}(g_{s}(x)|\boldsymbol{\mu}_{c},\boldsymbol{\Sigma}_{c})$,
where $\boldsymbol{\mu}_{c},\boldsymbol{\Sigma}_{c}$ are the mean and
covariance of multivariate Gaussian distribution of class $c\in\\{1,...,C\\}$.
Then, with a target image $x_{t}$, the intra-class distance is defined as

$D_{\text{intra}}(x_{t},y_{t})=D(x_{t};\boldsymbol{\mu}_{y_{t}},\boldsymbol{\Sigma}_{y_{t}}),$
(2)

where $y_{t}$ indicates the corresponding ground-truth label of the target
image. In the same manner, the inter-class distance is defined as

$D_{\text{inter}}(x_{t},y_{t})=\frac{1}{C-1}\sum^{C}_{c=1}\mathbbm{1}(y_{t}\neq
c)D(x_{t};\boldsymbol{\mu}_{c},\boldsymbol{\Sigma}_{c})$ (3)

Note that achieving low intra-class distance and high inter-class distance is
important for improving the image classification accuracy.

Figure 2: Overview of our method. (Pre-stage) Ours first pre-calculates the
source class-conditional Gaussian distributions using a pre-trained feature
extractor. (Test-time adaptation) During test-time adaptation, ours adapts a
model by optimizing Class-Aware Feature Alignment (CAFA) loss while making
predictions simultaneously.

##  4 CAFA: Class-Aware Feature Alignment

This section presents an in-depth analysis on feature alignments and the
details of our proposed method, CAFA. As illustrated in Fig. 2, CAFA first
pre-calculates the source class-conditional Gaussian distributions using a
pretrained model and adapts a model to test data during the test time by
aligning the test features to obtained class-conditional Gaussian
distributions.

###  4.1 Analysis on Class Separability in Feature Alignment

We compare and analyze three different feature alignments with respect to the
intra- and inter-class distances. First, we investigate the global feature
alignment that reduces the gap between the source and target distributions
without considering class information. With the given source Gaussian
distribution
$\mathcal{N}(g_{s}(x_{s})|\boldsymbol{\mu}_{s},\boldsymbol{\Sigma}_{s})$, the
global feature alignment loss $\mathcal{L}_{\text{FA}}$ is formulated as

$\begin{gathered}\mathcal{L}_{\text{FA}}=\lVert\boldsymbol{\mu}_{s}-\boldsymbol{\mu}_{t}\rVert_{2}^{2}+\lVert\boldsymbol{\Sigma}_{s}-\boldsymbol{\Sigma}_{t}\rVert_{F}^{2},\end{gathered}$
(4)

where $\boldsymbol{\mu}_{s},\boldsymbol{\Sigma}_{s}$ denote the mean and
covariance of source features without considering class information,
$\boldsymbol{\mu}_{t},\boldsymbol{\Sigma}_{t}$ indicate the mean and
covariance estimated from a mini-batch of test samples, and
$\lVert\cdot\rVert_{2},\lVert\cdot\rVert_{F}$ denote Euclidean norm and
Frobenius norm, respectively. As aforementioned, while the global feature
alignment drastically decreases the intra-class distance, it also accompanies
a significant reduction of inter-class distance which needs to be high for
achieving a reasonable level of image classification accuracy (Fig. 1 (a) (red
lines)). Fig. 1 (b) also verifies such a point by visualizing the degraded
image classification accuracy.

To address such a problem, we take the _class_ information into account when
performing the feature alignment. Aligning the source and target distributions
in a class-wise manner would be one straight-forward approach. However, during
the test phase, there exist excessively few number of samples for each class
in a mini-batch to precisely estimate class-conditional distributions of test
data. Thus, we align individual test samples to the source class-conditional
distribution of the predicted class by using the Mahalanobis distance.
Specifically, with the given source class-conditional Gaussian distributions
$\mathcal{N}(\boldsymbol{\mu}_{c},\boldsymbol{\Sigma}_{c})$, the loss
$\mathcal{L}_{\text{intra}}$ minimizing the intra-class distance is defined as

$\begin{gathered}\mathcal{L}_{\text{intra}}=\frac{1}{N}\sum_{n=1}^{N}D_{\text{intra}}(x_{t,n},\hat{y}_{t,n}),\end{gathered}$
(5)

where $\hat{y}_{t,n}$ indicates the predicted label of the target sample
$x_{t,n}$, and $N$ denotes the number of target samples. While utilizing
$\mathcal{L}_{\text{intra}}$ effectively reduces the intra-class distance, it
still decreases the inter-class distance. We further elaborate on such a point
in Section 5.2.

Finally, we also consider the inter-class distance explicitly in addition to
the $\mathcal{L}_{\text{intra}}$ which we further explain in Section 4.2. The
change of the intra- and inter-class distances when our final loss is applied
is visualized in Fig. 1 (a) (blue lines). Our final loss desirably aligns the
source and target distributions by reducing the intra-class distance and
enlarging the inter-class distance. As also shown in Fig. 1(b) (red line), our
final loss improves the image classification accuracy and class-
discriminability compared to the global feature alignment.

###  4.2 Class-Aware Feature Alignment

Pre-calculation of source statistics As shown in the pre-stage of Fig. 2, we
calculate $C$ class-conditional Gaussian distributions
$P(g_{s}(x_{s})|y=c)=\mathcal{N}(g_{s}(x_{s})|\boldsymbol{\mu}_{c},\boldsymbol{\Sigma}_{c})$
with the pretrained feature extractor $g_{s}(\cdot)$ over source training
samples $(x_{s},y_{s})\sim\mathcal{X}_{s}$ with the following equations:

$\begin{gathered}\boldsymbol{\mu}_{c}=\frac{1}{N_{c}}\sum_{n=1}^{N_{c}}g_{s}(x_{s,n})\\\
\boldsymbol{\Sigma}_{c}=\frac{1}{N_{c}}\sum_{n=1}^{N_{c}}(g_{s}(x_{s,n})-\boldsymbol{\mu}_{c})(g_{s}(x_{s,n})-\boldsymbol{\mu}_{c})^{\top},\end{gathered}$
(6)

where $N_{c}$ denotes the number of training samples of class $c$. In the
linear discriminant analysis (LDA), it is assumed that all the class-
conditional Gaussian distributions have the same covariance matrix (i.e, tied
covariance). However, as also pointed out in Lee et al. [60], deep networks
are not trained to have the same covariance across the different classes. Due
to this fact, we utilize class-wise covariance matrices for our experiments.
Note that even with using a tied covariance, ours still outperforms the
existing baselines, and we further explain it in Section 5.2.

Test-time adaptation With the obtained source class-conditional distributions
$P(g_{s}(x_{s})|y=c)$ at the pre-stage, we perform class-aware feature
alignment (CAFA) during the test time. We initialize a model using the weights
of pretrained networks $f_{s}(\cdot)$ and perform adaptation to target data
considering both intra- and inter-class distances. Our final loss
$\mathcal{L}_{\text{CAFA}}$ is defined as

$\mathcal{L}_{\text{CAFA}}=\frac{1}{N}\sum_{n=1}^{N}\log\frac{D_{\text{intra}}(x_{t,n},\hat{y}_{t,n})}{\sum_{c=1}^{C}D(x_{t,n};\boldsymbol{\mu}_{c},\boldsymbol{\Sigma}_{c})}.$
(7)

##  5 Experiments

This section first demonstrates the evaluation results on two different
settings: 1) robustness to corruptions, and 2) domain adaptation on digits.
Then, we present an in-depth analysis on our method by conducting ablation
studies and providing visualizations of the representation space. For
corruption datasets, we evaluate models on CIFAR10-C, CIFAR100-C,
TinyImageNet-C, and ImageNet-C [5] datasets. urthermore, we measure the
adaptation errors on the Office-Home [61] and digit datasets (i.e., training a
model on SVHN [62] and validating it on MNIST [63] / MNIST-M [64] / USPS [65]
datasets).

Datasets CIFAR10 [66] and CIFAR100 [66] include 50,000 training samples and
10,000 test samples with 10 and 100 classes, respectively. CIFAR10-C and
CIFAR100-C [5] datasets contain 15 different corruptions, and the corruptions
are applied to the test set of CIFAR10 and CIFAR100 datasets. TinyImageNet
[67] is a subset of the original ImageNet [68] dataset, containing 100,000
training images and 10,000 validation images with 200 classes. ImageNet [68]
contains 1.2 million training samples and 50,000 validations samples with
1,000 object categories. Analogous to CIFAR10-C / CIFAR100-C, TinyImageNet-C
and ImageNet-C are composed of 15 different corruptions, where the corruptions
are applied to the validation set of TinyImageNet and ImageNet, respectively.
Office-Home [61] dataset consists of around 15,500 images and contains 65
categories of everyday objects with four distinct domains: Artistic images
(Ar), Clip art images (Cl), Product images (Pr), and Real-world images (Re).
For the digit datasets, SVHN includes color images of house numbers collected
from street views with 73,257 training samples and 26,032 test samples. We use
10,000/9,001/2,007 hand-written digit test samples from MNIST / MNIST-M /
USPS, respectively. All four digit datasets have ten classes, 0-9.

Baselines For a fair comparison, we consider the following baselines in our
experiments:

  * •

Source: Evaluating the pretrained network on the test data without any
adaptation.

  * •

Test-time normalization (BN) [51] updates the batch normalization statistics
[55] on test data in test time.

  * •

Pseudo label (PL) [52] utilizes the predicted label as a label for optimizing
the main task loss during testing.

  * •

Test-time entropy minimization (TENT) [47] updates batch normalization [55]
layer statistics and parameters on test data by minimizing entropy.

  * •

Source hypothesis transfer (SHOT) [37] updates feature extractor utilizing the
information maximization loss [45] with the self-supervised pseudo-labeling
during test time.

  * •

Test-time feature alignment (TFA) [54] aligns the source and target
distributions by matching the first- and second-order statistics of outputs
from both the penultimate layer and self-supervised task branch.

  * •

Test-time training (TTT++) [54] updates the model by jointly aligning the
first- and second-order statistics between the source and target distributions
and optimizing the proxy loss (i.e., contrastive loss for self-supervision
task) during test time.

Note that test-time normalization (BN), pseudo label (PL), test-time entropy
minimization (TENT), and CAFA (ours) are test-time adaptation methods. SHOT
[37], TFA, and TTT++ [54] are originally designed for source-free UDA and
test-time training, respectively. They iterate multiple epochs on the entire
test samples during test time.

Implementation details For all experiments, we adopt the residual networks [1]
with 50 layers (ResNet-50). We train the model by following the same training
protocol described in TTT++ [54] for the corruption datasets except for the
ImageNet-C dataset. For experiments on ImageNet-C, we adopt the pretrained
ResNet-50 networks provided by Pytorch [69]. For the digit datasets, we do not
apply the joint training step with a self-supervised task. To make a fair
comparison for each experiment, we use the same network and pretrained weight
for all the baselines in the same experiment condition. For test-time
adaptation, we only optimize the batch normalization parameters
$\gamma,\beta$, following Wang et al. [47]. We set the batch size as 200 and
utilize Adam [70] optimizer with the learning rate of 0.001 for adaptation
except for the ImageNet-C dataset. For the ImageNet-C dataset, we use the
batch size of 64 and the learning rate of 0.00025. All experiments are
conducted with one NVIDIA TITAN RTX GPU (24GB). Further implementation details
can be found in the supplementary material.

###  5.1 Quantitative Evaluation and Comparisons

Robustness to corruptions To evaluate the robustness to corruptions, we
utilize the pretrained networks on CIFAR10, CIFAR100, TinyImageNet, and
ImageNet datasets and validate the pretrained networks on their corresponding
corruption datasets, respectively.

Table LABEL:tab:cifar shows the image classification errors (%) on diverse
corruption types with the severest corruption level in CIFAR10-C and
CIFAR100-C datasets. Note that TFA and TTT++ are evaluated in the test-time
adaptation setting in this experiment. As shown, our proposed method
outperforms the baselines on all types of corruptions in both CIFAR10-C and
CIFAR100-C datasets with a large margin. We further evaluate our method on a
larger dataset, TinyImageNet-C dataset. Table LABEL:tab:tinyimagenetc
describes the results on the TinyImageNet-C dataset. Similar to the results of
CIFAR10-C and CIFAR100-C, CAFA outperforms the baselines in all types of
corruptions except for the contrast corruption on the TinyImageNet-C dataset.
Finally, we validate our method on a large-scale dataset, ImageNet-C, as
reported in Table LABEL:tab:imagenet. Unlike different datasets, the
classification error of Source is significantly high, making adaptation more
challenging than other datasets. For instance, in case of the Gaussian noise
corruption, the error is around 96%. Despite such a challenge, ours achieves
the lowest averaged error compared to the baselines, improving the Source by
around 17%. Furthermore, ours outperforms the previous state-of-the-art TTA
method, TENT [47], by 0.6% on average.

Table 1: Classification error (%) on the CIFAR10-C (upper group) and
CIFAR100-C (lower group) datasets with severity level 5 corruptions. † denotes
the results obtained from the official codes, and ∗ indicates the test-time
training methods evaluated in the test-time adaptation setting.

Method | Gaus. | Shot | Impu. | Defo. | Glas. | Moti. | Zoom | Snow | Fros. | Fog | Brig. | Cont. | Elas. | Pixe. | Jpeg. | Average
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Source | 48.73 | 44.00 | 57.00 | 11.84 | 50.78 | 23.38 | 10.84 | 21.93 | 28.24 | 29.41 | 7.01 | 13.27 | 23.38 | 47.88 | 19.46 | 29.14
BN | 17.34 | 16.36 | 28.25 | 9.89 | 26.11 | 14.27 | 8.15 | 16.29 | 13.82 | 20.69 | 8.58 | 8.49 | 19.67 | 11.74 | 14.17 | 15.59
PL | 17.22 | 16.07 | 27.85 | 9.74 | 25.94 | 14.13 | 8.07 | 16.12 | 13.78 | 20.14 | 8.53 | 8.53 | 19.73 | 11.65 | 13.94 | 15.43
TENT† | 15.95 | 14.55 | 24.72 | 9.03 | 23.25 | 12.74 | 7.47 | 13.91 | 12.78 | 16.66 | 8.13 | 8.12 | 18.30 | 10.85 | 13.21 | 13.98
TFA†∗ | 15.80 | 14.91 | 23.89 | 9.29 | 23.08 | 12.82 | 7.41 | 13.93 | 12.60 | 16.41 | 7.43 | 7.95 | 17.24 | 12.00 | 12.86 | 13.84
TTT++†∗ | 16.80 | 14.92 | 21.99 | 9.60 | 22.97 | 12.32 | 7.55 | 13.14 | 12.67 | 14.33 | 7.06 | 7.85 | 17.27 | 11.63 | 12.74 | 13.52
CAFA | 14.28 | 12.70 | 21.12 | 7.73 | 20.84 | 10.55 | 6.75 | 11.93 | 11.31 | 13.33 | 6.95 | 7.13 | 16.08 | 9.59 | 11.67 | 12.13
Source | 80.77 | 77.84 | 87.75 | 39.62 | 82.26 | 54.22 | 38.38 | 54.58 | 60.19 | 68.11 | 28.86 | 50.93 | 59.54 | 72.27 | 49.96 | 60.35
BN | 47.37 | 45.58 | 60.10 | 34.01 | 56.70 | 40.99 | 32.05 | 46.53 | 42.57 | 54.41 | 32.56 | 33.30 | 48.83 | 37.47 | 39.43 | 43.46
PL | 46.74 | 45.26 | 59.21 | 33.83 | 56.08 | 40.29 | 31.64 | 46.10 | 42.07 | 53.74 | 32.24 | 33.08 | 48.24 | 37.11 | 39.01 | 43.00
TFA†∗ | 44.68 | 43.28 | 56.17 | 32.47 | 54.11 | 37.48 | 30.32 | 42.46 | 39.73 | 47.57 | 30.18 | 32.52 | 45.34 | 36.81 | 37.28 | 40.69
TTT++†∗ | 43.70 | 41.84 | 55.77 | 31.15 | 53.38 | 35.54 | 29.98 | 41.13 | 38.70 | 45.08 | 29.14 | 30.34 | 44.69 | 35.47 | 37.37 | 39.55
TENT† | 43.11 | 41.70 | 53.30 | 31.35 | 51.08 | 36.34 | 29.90 | 42.73 | 38.99 | 45.13 | 29.64 | 30.62 | 44.03 | 34.23 | 36.34 | 39.23
CAFA | 41.60 | 39.77 | 50.45 | 30.17 | 48.35 | 34.65 | 28.76 | 39.52 | 37.42 | 41.25 | 27.95 | 29.54 | 42.37 | 32.87 | 35.02 | 37.31

Table 2: Classification error (%) on the TinyImageNet-C dataset with severity
level 5 corruptions. † denotes the results obtained from the official codes.

Method | Gaus. | Shot | Impu. | Defo. | Glas. | Moti. | Zoom | Snow | Fros. | Fog | Brig. | Cont. | Elas. | Pixe. | Jpeg. | Average
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Source | 94.86 | 93.14 | 97.25 | 86.96 | 88.45 | 77.87 | 77.63 | 83.71 | 80.35 | 92.85 | 81.94 | 98.55 | 69.03 | 58.89 | 55.64 | 82.47
BN | 68.74 | 67.80 | 74.72 | 68.71 | 77.13 | 61.04 | 59.71 | 67.37 | 67.21 | 76.52 | 66.09 | 93.73 | 61.32 | 55.07 | 55.75 | 68.06
PL | 68.11 | 66.95 | 74.18 | 67.92 | 76.52 | 60.32 | 58.87 | 67.08 | 66.63 | 75.99 | 65.34 | 93.38 | 60.82 | 54.77 | 55.50 | 67.49
TENT† | 64.11 | 63.72 | 70.35 | 63.22 | 73.39 | 56.64 | 55.07 | 64.28 | 62.99 | 70.39 | 60.88 | 92.44 | 57.17 | 51.72 | 52.94 | 63.95
CAFA | 63.68 | 63.04 | 70.12 | 61.27 | 71.30 | 55.22 | 54.34 | 63.31 | 61.88 | 67.96 | 59.15 | 92.53 | 56.16 | 51.21 | 52.60 | 62.92

Table 3: Classification error (%) on the ImageNet-C dataset with severity
level 5 corruptions. † denotes the results obtained from the official code.

Method | Gaus. | Shot | Impu. | Defo. | Glas. | Moti. | Zoom | Snow | Fros. | Fog | Brig. | Cont. | Elas. | Pixe. | Jpeg. | Average
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Source | 95.53 | 94.93 | 95.38 | 84.94 | 91.66 | 86.63 | 77.13 | 84.06 | 79.36 | 77.30 | 44.30 | 95.78 | 85.23 | 77.40 | 66.82 | 82.43
BN | 88.31 | 87.83 | 87.71 | 88.42 | 87.96 | 78.57 | 64.42 | 67.89 | 69.49 | 55.22 | 36.81 | 89.32 | 59.21 | 57.33 | 67.75 | 72.42
PL | 87.04 | 87.08 | 85.91 | 85.02 | 86.82 | 72.27 | 57.71 | 59.14 | 68.46 | 47.73 | 34.26 | 92.51 | 52.35 | 48.82 | 58.40 | 68.23
TENT† | 84.76 | 82.82 | 79.55 | 83.66 | 86.84 | 65.84 | 53.75 | 55.42 | 68.46 | 44.68 | 34.74 | 95.03 | 47.92 | 44.95 | 52.90 | 65.42
CAFA | 83.27 | 80.19 | 79.23 | 85.96 | 86.68 | 64.53 | 55.16 | 54.39 | 62.93 | 45.25 | 35.65 | 91.91 | 48.10 | 45.81 | 53.14 | 64.81

Furthermore, in CIFAR10-C and CIFAR100-C datasets, CAFA achieves lower error
rates compared to TFA which aligns source and target distributions without
considering the class information. Such results demonstrate that considering
both intra- and inter-class distances is important when performing feature
alignments for test-time adaptation. Note that TFA is originally designed for
test-time training (TTT) setting that iterates multiple epochs over test
samples.

Office-Home dataset We also evaluate our approach on the Office-Home [61]
dataset, which is one of the widely used domain adaptation datasets. We
evaluate our method on 12 different adaptation scenarios, training a model on
one source domain and validating it on other domains. Table
LABEL:suptab:officehome shows the image classification errors (%) on different
adaptation scenarios of the OfficeHome [61] dataset. As shown, CAFA
consistently outperforms the baselines in all scenarios with a large margin.
On average, ours reduces the classification error of the Source by around 5%,
and that of TENT by around 1.4%.

Table 4: Classification error (%) on the OfficeHome [61] dataset. † denotes
the results obtained from the official codes.

Method |  Ar$\rightarrow$Cl |  Ar$\rightarrow$Pr |  Ar$\rightarrow$Re |  Cl$\rightarrow$Ar |  Cl$\rightarrow$Pr |  Cl$\rightarrow$Re |  Pr$\rightarrow$Ar |  Pr$\rightarrow$Cl |  Pr$\rightarrow$Re |  Re$\rightarrow$Ar |  Re$\rightarrow$Cl |  Re$\rightarrow$Pr | Average
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Source | 67.33 | 46.68 | 36.81 | 68.52 | 54.22 | 53.91 | 66.25 | 71.25 | 40.76 | 45.16 | 65.02 | 29.65 | 53.80
BN | 63.34 | 47.31 | 36.29 | 66.09 | 57.85 | 54.12 | 62.22 | 68.93 | 39.29 | 46.27 | 60.73 | 30.19 | 52.72
PL | 63.18 | 46.25 | 35.87 | 65.68 | 56.14 | 53.16 | 61.80 | 68.34 | 38.51 | 45.45 | 60.53 | 29.62 | 52.04
TENT† | 61.47 | 44.33 | 34.82 | 62.75 | 52.22 | 49.16 | 61.60 | 66.19 | 36.26 | 44.66 | 58.08 | 28.14 | 49.97
CAFA | 59.73 | 42.64 | 34.01 | 61.39 | 51.23 | 47.69 | 60.28 | 63.92 | 35.87 | 42.89 | 54.91 | 27.84 | 48.53

Table 5: Digit domain adaptation error (%) from the SVHN dataset to MNIST / MNIST-M / USPS datasets. † denotes that the results are obtained from the official code. Method | MNIST | MNIST-M | USPS
---|---|---|---
Source | 55.03 | 56.86 | 40.66
BN | 34.07 | 62.93 | 36.52
PL | 29.86 | 61.86 | 35.48
TENT† | 22.31 | 57.18 | 33.33
CAFA (Ours) | 17.39 | 53.72 | 31.72

Digit adaptation For digit adaptation, we train the networks on the SVHN
dataset and evaluate the classification errors (%) on MNIST / MNIST-M / USPS
datasets. Note that MNIST and USPS consist of grayscale images while MNIST-M
is composed of color digit images. Table LABEL:tab:digit_adaptation compares
the evaluation results of CAFA and baselines on digit datasets. While previous
methods improve the classification errors on MNIST and USPS datasets compared
to Source, they show degraded classification accuracy on the MNIST-M dataset.
On the other hand, CAFA consistently enhances the adaptation errors on all
three datasets with a large margin.

Table 6: Comparison with baselines from different adaptation settings. Classification error (%) is measured in the CIFAR10-C dataset along with the number of step changes. ∗∗ indicates the results reported in Liu et al. [54]. Test-Time Adaptation | Different Adaptation Tasks
---|---
Method | Step 1 | Step 2 | Step 3 | Method | Error (%)
TENT† | 13.98 | 13.72 | 13.97 |  SHOT∗∗ | 14.71
CAFA (Ours) | 12.13 | 11.61 | 11.79 |  TFA∗∗ | 11.87
|  |  |  |  TTT++∗∗ | 9.60

Comparison with baselines from different adaptation settings Even though we
address the distribution shifts in the test-time adaptation setting, we
compare CAFA with SHOT [37] (source-free UDA) and TFA and TTT++ [54] (test-
time training) as shown in Table LABEL:tab:offline_comparison. Note that we
utilize the same pretrained networks used in Liu et al. [54], and those
methods need to iterate epochs over the test samples several times. For this
experiment, we report the averaged classification error over all types of
corruptions in the CIFAR10-C dataset. For this experiment, we change the
number of loss optimizations for each mini-batch (i.e., the number of steps)
in the test-time adaptation setting. We observe both CAFA and TENT achieve the
lowest classification errors when the number of steps is two. Additionally,
when the number of steps is two, CAFA outperforms SHOT [37] with a large
margin, and shows superior performance compared to TFA [54] which utilizes
feature alignments without considering class information in the test-time
training setting. Although TTT++ [54] shows the best accuracy, it requires a
non-trivial amount of computation costs and time since it iterates multiple
epochs over test samples.

Table 7: Our ablation results on the CIFAR10-C and CIFAR100-C datasets. The left group shows the effectiveness of considering intra- and inter-class distances, the middle group presents the comparison of updating the batch normalization parameters and full parameters in the feature extractor, and the right group describes the effects of using a tied covariance.  Datasets |  Effectiveness of Considering Intra-/Inter-Class Dist. |  Updating Batch Norm. vs Full Parameters |  Tied Covariance vs Class-wise Covariance
---|---|---|---
CIFAR10-C | Source | 29.14 | Source | 29.14 | Source | 29.14
TENT† | 13.98 |  TENT† | 13.98 |  TENT† | 13.98
Global FA | 19.12 | CAFA-Full | 12.66 | CAFA-Tied | 12.47
Intra-Class Dist. Only | 13.02 | CAFA | 12.13 | CAFA | 12.13
CAFA | 12.13 |  |  |  |
CIFAR100-C | Source | 60.35 | Source | 60.35 | Source | 60.35
TENT† | 39.23 |  TENT† | 39.23 |  TENT† | 39.23
Global FA | 51.41 | CAFA-Full | 38.31 | CAFA-Tied | 38.19
Intra-Class Dist. Only | 41.51 | CAFA | 37.31 | CAFA | 37.31
CAFA | 37.31 |  |  |  |

###  5.2 Analysis

Effectiveness of Considering Intra- and Inter-Class Distance
To further validate our motivation for considering intra- and inter-class
distances, we compare each component in both CIFAR10-C and CIFAR100-C
datasets. As reported in the left group of Table LABEL:tab:ablation, utilizing
the global feature alignment performs poorly in TTA. In case of reducing the
intra-class distance only, it improves the classification errors on both
datasets, but it does not outperform TENT in the CIFAR100-C dataset. Finally,
considering both intra- and inter-class distances (CAFA) achieves the lowest
classification errors. Such results demonstrate the initial intuition of our
work is valid, which is elaborated in Section 1.

Updating the entire parameters of feature extractor In our main experiments,
we only update the parameters of batch normalization layers in the feature
extractor, following Wang et al. [47]. In this ablation study, we further
validate CAFA by updating the entire parameters in the feature extractor. Note
that since CAFA performs feature alignments using feature vectors, the
classifier $h(\cdot)$ cannot be updated by our loss. CAFA-Full in the middle
group of Table LABEL:tab:ablation shows the result of updating the full
parameters when utilizing CAFA. While it shows superior performance compared
to the baselines, only updating the batch normalization layers outperforms the
case of updating the full parameters. As pointed out in Wang et al. [47],
updating the full model may cause the model to diverge from what they learned
from training. Furthermore, we conjecture that the number of samples during
test-time adaptation may be not sufficient to optimize the entire parameters
to converge.

Impact of using tied covariance We adopt the class-wise covariance matrices
for CAFA in the main experiments. However, in linear discriminant analysis
(LDA), it is assumed that all the class-conditional Gaussian distributions
share the same covariance matrix (i.e., tied covariance). We conduct an
ablation study regarding such an issue in the right group of Table
LABEL:tab:ablation. We observe that CAFA with tied covariance shows less
performance improvement than utilizing the class-wise covariances. We
conjecture that it is because class-wise covariances represent the statistics
of each class of the source distributions more precisely than the tied
covariance. Moreover, as pointed out in Lee et al. [60], deep networks are not
trained to share the same covariance matrix for all class-conditional
distributions. However, regardless of the covariance types, our method still
outperforms the existing baselines.

Figure 3: (a) illustrates the change of intra-class distance (dotted lines)
and inter-class distance (solid line) as adaptation proceeds. (b-d) show the
t-SNE visualizations of applying (b) intra-class distance only, (c) TENT [47],
and (d) ours. All visualizations are obtained from the Gaussian noise
corruption in the CIFAR10-C dataset.

Visualizations To further validate our intuition, we visualize the change of
the intra- and inter-class distances in Fig. 3 (a): intra-class distance only
(green line), TENT [47] (red line), and CAFA (blue line). As shown in Fig. 3
(a), only reducing the intra-class distance also decreases the inter-class
distance, which harms the class-discriminability. In case of TENT [47], we
observe that the intra-class distance does not decrease. On the other hand,
CAFA desirably reduces the intra-class distance while enlarging the inter-
class distance. Such improved class-discriminability is also shown in the
t-SNE visualization in Fig. 3 (d). Unlike other methods (Fig. 3 (b) and (c)),
ours shows well-separated representation space in a class-wise manner.

##  6 Discussion

In this work, we proposed a _simple yet effective_ feature alignment that
considers both intra- and inter-class distances, noting the importance of
considering them in test-time adaptation. Most existing feature alignments are
generally conducted along with training on the source data, which allows a
model to learn target distributions in a class-discriminative manner. However,
in case of test-time adaptation where access to the source data is prohibited,
a model does not have a chance to learn test features in such a manner. Our
simple feature alignment that considers both intra- and inter-class distances
effectively addresses such a challenge as shown in our in-depth analysis and
extensive experiments. However, there still remains room for improvements: 1)
better pseudo-labeling method and 2) better learnable parameters than BN
layers. We hope our work inspires the following researchers to investigate
more effective test-time adaptation methods.

Pseudo labeling One limitation of our work is that we resort to using the
predicted labels when assigning a class to each test sample. This could be
problematic in the early phase of adaptation since the model encounters
samples of a different distribution, achieving low classification accuracy.
Along with our novel perspective to consider both intra- and inter-class
distance when performing feature alignment in TTA, we believe that replacing
such a pseudo-labeling approach with other advanced methods would further
boost the adaptation performance of CAFA.

Finding better learnable parameters than BN layers In our work, we only
updated the batch normalization parameters, as also done in Wang et al. [47].
Despite its demonstrated effectiveness in TTA, batch normalization parameters
only take less than 1% of the model parameters. If we can find better
learnable model parameters, it may further improve the adaptation performance
of our proposed loss CAFA.

## References

  * [1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [2] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [3] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [5] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  Proceedings of the International Conference on Learning Representations, 2019.
  * [6] Dengxin Dai and Luc Van Gool.  Dark model adaptation: Semantic image segmentation from daytime to nighttime.  In 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pages 3819–3824. IEEE, 2018.
  * [7] Georg Volk, Stefan Müller, Alexander Von Bernuth, Dennis Hospach, and Oliver Bringmann.  Towards robust cnn-based object detection through augmentation with synthetic rain variations.  In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 285–292. IEEE, 2019.
  * [8] Sungha Choi, Sanghun Jung, Huiwon Yun, Joanne T Kim, Seungryong Kim, and Jaegul Choo.  Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11580–11590, 2021.
  * [9] Joaquin Quiñonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence.  Dataset shift in machine learning.  Mit Press, 2008.
  * [10] David Lazer, Ryan Kennedy, Gary King, and Alessandro Vespignani.  The parable of google flu: traps in big data analysis.  science, 343(6176):1203–1205, 2014.
  * [11] Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan.  Domain separation networks.  Advances in neural information processing systems, 29, 2016.
  * [12] Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf.  Domain adaptation with conditional transferable components.  In International conference on machine learning, pages 2839–2848. PMLR, 2016.
  * [13] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan.  Deep transfer learning with joint adaptation networks.  In International conference on machine learning, pages 2208–2217. PMLR, 2017.
  * [14] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  The journal of machine learning research, 17(1):2096–2030, 2016\.
  * [15] Baochen Sun and Kate Saenko.  Deep coral: Correlation alignment for deep domain adaptation.  In European conference on computer vision, pages 443–450. Springer, 2016.
  * [16] Yaroslav Ganin and Victor Lempitsky.  Unsupervised domain adaptation by backpropagation.  In International conference on machine learning, pages 1180–1189. PMLR, 2015.
  * [17] Haifeng Xia, Handong Zhao, and Zhengming Ding.  Adaptive adversarial network for source-free domain adaptation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9010–9019, 2021.
  * [18] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu.  Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
  * [19] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan.  Conditional adversarial domain adaptation.  Advances in neural information processing systems, 31, 2018.
  * [20] Rui Shu, Hung Bui, Hirokazu Narui, and Stefano Ermon.  A DIRT-t approach to unsupervised domain adaptation.  In International Conference on Learning Representations, 2018.
  * [21] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell.  Adversarial discriminative domain adaptation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167–7176, 2017.
  * [22] Jungsoo Lee, Jooyeol Yun, Sunghyun Park, Yonggyu Kim, and Jaegul Choo.  Improving face recognition with large age gaps by learning to distinguish children.  2021\.
  * [23] Chaoqi Chen, Weiping Xie, Wenbing Huang, Yu Rong, Xinghao Ding, Yue Huang, Tingyang Xu, and Junzhou Huang.  Progressive feature alignment for unsupervised domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 627–636, 2019.
  * [24] Shuang Li, Shiji Song, Gao Huang, Zhengming Ding, and Cheng Wu.  Domain invariant and class discriminative feature learning for visual domain adaptation.  IEEE transactions on image processing, 27(9):4260–4273, 2018.
  * [25] Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada.  Asymmetric tri-training for unsupervised domain adaptation.  In International Conference on Machine Learning, pages 2988–2997. PMLR, 2017.
  * [26] Shaoan Xie, Zibin Zheng, Liang Chen, and Chuan Chen.  Learning semantic representations for unsupervised domain adaptation.  In International conference on machine learning, pages 5423–5432. PMLR, 2018.
  * [27] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira.  Analysis of representations for domain adaptation.  Advances in neural information processing systems, 19, 2006.
  * [28] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan.  A theory of learning from different domains.  Machine learning, 79(1):151–175, 2010.
  * [29] Pedro O Pinheiro.  Unsupervised domain adaptation with similarity learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8004–8013, 2018.
  * [30] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada.  Maximum classifier discrepancy for unsupervised domain adaptation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3723–3732, 2018.
  * [31] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell.  Cycada: Cycle-consistent adversarial domain adaptation.  In International conference on machine learning, pages 1989–1998. PMLR, 2018.
  * [32] Yabin Zhang, Hui Tang, Kui Jia, and Mingkui Tan.  Domain-symmetric networks for adversarial domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5031–5040, 2019.
  * [33] Hong Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan.  Transferable adversarial training: A general approach to adapting deep classifiers.  In International Conference on Machine Learning, pages 4013–4022. PMLR, 2019.
  * [34] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan.  Learning transferable features with deep adaptation networks.  In International conference on machine learning, pages 97–105. PMLR, 2015.
  * [35] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell.  Deep domain confusion: Maximizing for domain invariance.  arXiv preprint arXiv:1412.3474, 2014.
  * [36] Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko.  Simultaneous deep transfer across domains and tasks.  In Proceedings of the IEEE international conference on computer vision, pages 4068–4076, 2015.
  * [37] Jian Liang, Dapeng Hu, and Jiashi Feng.  Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation.  In International Conference on Machine Learning, pages 6028–6039. PMLR, 2020.
  * [38] Cian Eastwood, Ian Mason, Chris Williams, and Bernhard Schölkopf.  Source-free adaptation to measurement shift via bottom-up feature restoration.  In International Conference on Learning Representations, 2021.
  * [39] Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu.  Model adaptation: Unsupervised domain adaptation without source data.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9641–9650, 2020.
  * [40] Pietro Morerio, Riccardo Volpi, Ruggero Ragonesi, and Vittorio Murino.  Generative pseudo-label refinement for unsupervised domain adaptation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3130–3139, 2020.
  * [41] Jogendra Nath Kundu, Naveen Venkat, R Venkatesh Babu, et al.  Universal source-free domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4544–4553, 2020.
  * [42] Vinod K Kurmi, Venkatesh K Subramanian, and Vinay P Namboodiri.  Domain impression: A source data free domain adaptation method.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 615–625, 2021.
  * [43] Hao-Wei Yeh, Baoyao Yang, Pong C Yuen, and Tatsuya Harada.  Sofa: Source-data-free feature alignment for unsupervised domain adaptation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 474–483, 2021.
  * [44] Serban Stan and Mohammad Rostami.  Unsupervised model adaptation for continual semantic segmentation.  arXiv preprint arXiv:2009.12518, 2020.
  * [45] Andreas Krause, Pietro Perona, and Ryan Gomes.  Discriminative clustering by regularized information maximization.  Advances in neural information processing systems, 23, 2010.
  * [46] Shiqi Yang, Yaxing Wang, Joost van de Weijer, Luis Herranz, and Shangling Jui.  Generalized source-free domain adaptation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8978–8987, 2021.
  * [47] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell.  Tent: Fully test-time adaptation by entropy minimization.  In International Conference on Learning Representations, 2021.
  * [48] Xuefeng Hu, Gokhan Uzunbas, Sirius Chen, Rui Wang, Ashish Shah, Ram Nevatia, and Ser-Nam Lim.  Mixnorm: Test-time adaptation through online normalization estimation.  arXiv preprint arXiv:2110.11478, 2021.
  * [49] Fuming You, Jingjing Li, and Zhou Zhao.  Test-time batch statistics calibration for covariate shift.  arXiv preprint arXiv:2110.04065, 2021.
  * [50] Marvin Mengxin Zhang, Sergey Levine, and Chelsea Finn.  MEMO: Test time robustness via adaptation and augmentation.  In NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications, 2021.
  * [51] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge.  Improving robustness against common corruptions by covariate shift adaptation.  Advances in Neural Information Processing Systems, 33:11539–11551, 2020.
  * [52] Dong-Hyun Lee.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In Workshop on challenges in representation learning, ICML, volume 3, page 896, 2013.
  * [53] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt.  Test-time training with self-supervision for generalization under distribution shifts.  In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9229–9248. PMLR, 13–18 Jul 2020\.
  * [54] Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi.  Ttt++: When does self-supervised test-time training fail or thrive?  Advances in Neural Information Processing Systems, 34, 2021.
  * [55] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. PMLR, 2015.
  * [56] Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S Yu.  Transfer feature learning with joint distribution adaptation.  In Proceedings of the IEEE international conference on computer vision, pages 2200–2207, 2013.
  * [57] Philip Haeusser, Thomas Frerix, Alexander Mordvintsev, and Daniel Cremers.  Associative domain adaptation.  In Proceedings of the IEEE international conference on computer vision, pages 2765–2773, 2017.
  * [58] Weichen Zhang, Wanli Ouyang, Wen Li, and Dong Xu.  Collaborative and adversarial network for unsupervised domain adaptation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3801–3809, 2018.
  * [59] Ozan Sener, Hyun Oh Song, Ashutosh Saxena, and Silvio Savarese.  Learning transferrable representations for unsupervised domain adaptation.  Advances in neural information processing systems, 29, 2016.
  * [60] Dongha Lee, Sehun Yu, and Hwanjo Yu.  Multi-class data description for out-of-distribution detection.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1362–1370, 2020.
  * [61] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.  Deep hashing network for unsupervised domain adaptation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5018–5027, 2017.
  * [62] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning.  2011\.
  * [63] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [64] Yaroslav Ganin and Victor Lempitsky.  Unsupervised domain adaptation by backpropagation.  In International conference on machine learning, pages 1180–1189. PMLR, 2015.
  * [65] Jonathan J. Hull.  A database for handwritten text recognition research.  IEEE Transactions on pattern analysis and machine intelligence, 16(5):550–554, 1994.
  * [66] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [67] Ya Le and Xuan Yang.  Tiny imagenet visual recognition challenge.  CS 231N, 7(7):3, 2015.
  * [68] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [69] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [70] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In International Conference on Learning Representations, 2015.

This supplementary material presents additional experimental results and
visualizations of our method, which are not included in the main paper.
Section A first describes additional experimental results on CIFAR10-C and
CIFAR100-C datasets with different severity levels and the classification
accuracy with 5 trials. Section B presents the t-SNE visualizations of our
method, CAFA, and TENT [47]. Finally, Section C presents further
implementation details.

##  Appendix A Additional Experimental Results

We compare our method, CAFA, to baselines on the CIFAR10-C and CIFAR100-C
datasets with different severity levels. As reported in Tables
LABEL:suptab:cifar_lv4-LABEL:suptab:cifar_lv1, CAFA outperforms baselines on
all severity levels in the CIFAR10-C and CIFAR100-C datasets. Furthermore, we
present the classification errors along with independent trials using
different random seeds in Table LABEL:suptab:cifar_randomseeds. As shown, CAFA
shows minimal performance variation considering the small standard deviation.

##  Appendix B Visualizations

For the qualitative analysis, we visualize the representation space of test
samples from our method and TENT [47] by using the t-SNE algorithm. Fig. 4
shows the t-SNE results on different corruption types in the CIFAR10-C
dataset, and Fig. 5 illustrates the change of representation space of test
samples from our method as adaptation proceeds. As shown, the representations
of test samples are well-separated, and they desirably converge in a class-
wise manner as adaptation proceeds.

##  Appendix C Further Implementation Details

This section presents further implementation details of our method. We adopt
the residual networks [1] with 50 layers (ResNet-50) for all experiments. For
the CIFAR10-C [66] dataset, we utilize the same pretrained networks provided
by Liu et al. [54]. Moreover, we use the pretrained ResNet-50 networks
provided by Pytorch [69] for the ImageNet-C [5] experiment. We set the batch
size and learning rate as 200 and 0.001, respectively, for all experiments
except for the ImageNet-C dataset. For ImageNet-C, we use the batch size of 64
and the learning rate of 0.00025. We adopt Adam [70] optimizer for all
experiments, and all experiments are conducted with one NVIDIA TITAN RTX GPU
(24 GB). For a fair comparison, all the results are obtained with the same
random seed.

Table 8: Classification error (%) on the CIFAR10-C (upper group) and
CIFAR100-C (lower group) datasets with severity level 4 corruptions. † denotes
the results obtained from the official codes, and ∗ indicates the test-time
training methods evaluated in the test-time adaptation setting.

Method | Gaus. | Shot | Impu. | Defo. | Glas. | Moti. | Zoom | Snow | Fros. | Fog | Brig. | Cont. | Elas. | Pixe. | Jpeg. | Average
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Source | 43.31 | 34.34 | 43.78 | 8.32 | 52.34 | 16.72 | 8.12 | 19.31 | 20.07 | 13.02 | 5.88 | 7.45 | 13.04 | 26.45 | 17.12 | 21.95
BN | 15.97 | 13.14 | 22.03 | 8.30 | 25.85 | 12.42 | 7.23 | 16.99 | 11.70 | 13.07 | 7.49 | 7.49 | 13.57 | 9.25 | 12.38 | 13.13
PL | 15.81 | 13.03 | 21.66 | 8.21 | 25.62 | 12.23 | 7.18 | 16.84 | 11.64 | 12.88 | 7.55 | 7.39 | 13.59 | 9.27 | 12.28 | 13.01
TENT† | 14.51 | 11.98 | 19.12 | 7.69 | 22.81 | 10.96 | 7.10 | 15.26 | 11.18 | 11.03 | 7.05 | 7.03 | 12.69 | 8.75 | 11.64 | 11.92
TFA†∗ | 14.69 | 12.29 | 18.03 | 7.57 | 22.84 | 11.19 | 6.59 | 14.77 | 10.48 | 10.62 | 6.68 | 6.89 | 11.90 | 9.08 | 11.03 | 11.64
TTT++†∗ | 15.20 | 12.57 | 17.33 | 7.61 | 23.07 | 10.72 | 6.62 | 13.31 | 10.63 | 9.87 | 6.14 | 6.29 | 12.13 | 8.95 | 11.92 | 11.49
CAFA | 12.73 | 10.51 | 16.71 | 6.66 | 20.34 | 9.73 | 6.07 | 12.82 | 9.51 | 8.98 | 6.14 | 6.35 | 11.25 | 7.76 | 10.31 | 10.39
Source | 76.96 | 69.67 | 79.91 | 32.50 | 82.34 | 46.32 | 32.75 | 50.96 | 51.77 | 44.18 | 26.44 | 32.44 | 41.08 | 52.03 | 46.34 | 51.05
BN | 44.93 | 40.95 | 52.39 | 31.68 | 57.40 | 37.53 | 30.28 | 45.78 | 38.16 | 41.63 | 30.30 | 31.13 | 40.37 | 32.93 | 37.22 | 39.51
PL | 44.54 | 40.42 | 51.50 | 31.63 | 56.51 | 37.05 | 29.95 | 45.38 | 37.78 | 40.93 | 30.16 | 30.82 | 40.04 | 32.56 | 36.85 | 39.07
TFA†∗ | 42.39 | 37.82 | 48.26 | 30.03 | 54.24 | 34.70 | 28.41 | 41.59 | 35.60 | 36.74 | 28.27 | 29.78 | 36.50 | 31.93 | 35.62 | 36.79
TTT++†∗ | 41.16 | 37.15 | 47.38 | 28.74 | 53.18 | 32.81 | 27.95 | 40.65 | 34.85 | 34.30 | 27.34 | 27.83 | 35.41 | 30.59 | 34.99 | 35.62
TENT† | 41.28 | 37.10 | 46.18 | 29.06 | 51.82 | 33.62 | 28.08 | 41.18 | 35.33 | 35.01 | 27.68 | 28.48 | 36.12 | 30.63 | 34.40 | 35.73
CAFA | 39.11 | 35.71 | 44.17 | 27.83 | 49.22 | 32.16 | 27.56 | 39.36 | 34.05 | 33.22 | 26.61 | 27.49 | 34.97 | 29.68 | 33.47 | 34.31

Table 9: Classification error (%) on the CIFAR10-C (upper group) and
CIFAR100-C (lower group) datasets with severity level 3 corruptions. † denotes
the results obtained from the official codes, and ∗ indicates the test-time
training methods evaluated in the test-time adaptation setting.

Method | Gaus. | Shot | Impu. | Defo. | Glas. | Moti. | Zoom | Snow | Fros. | Fog | Brig. | Cont. | Elas. | Pixe. | Jpeg. | Average
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Source | 36.96 | 28.00 | 26.86 | 5.73 | 35.53 | 16.68 | 7.54 | 16.89 | 18.54 | 8.98 | 5.64 | 6.47 | 7.69 | 13.10 | 15.54 | 16.68
BN | 13.75 | 11.97 | 16.01 | 7.39 | 16.74 | 12.36 | 7.33 | 14.89 | 11.63 | 10.41 | 7.03 | 7.23 | 9.50 | 8.43 | 11.51 | 11.08
PL | 13.59 | 11.90 | 15.81 | 7.35 | 16.58 | 12.23 | 7.44 | 14.75 | 11.41 | 10.36 | 6.97 | 7.13 | 9.41 | 8.32 | 11.48 | 10.98
TENT† | 12.47 | 11.29 | 14.12 | 6.86 | 15.47 | 11.26 | 6.95 | 13.40 | 10.81 | 9.09 | 6.52 | 6.87 | 8.99 | 7.93 | 10.84 | 10.19
TFA†∗ | 12.71 | 11.00 | 13.56 | 6.65 | 14.96 | 10.96 | 6.72 | 12.76 | 10.30 | 8.77 | 6.35 | 6.49 | 8.33 | 7.81 | 10.42 | 9.85
TTT++†∗ | 13.41 | 11.16 | 12.71 | 6.18 | 15.14 | 10.55 | 6.45 | 11.87 | 10.32 | 7.78 | 5.80 | 6.11 | 8.38 | 7.36 | 10.80 | 9.60
CAFA | 10.90 | 10.01 | 11.89 | 5.95 | 13.23 | 9.77 | 5.99 | 11.38 | 9.47 | 7.54 | 5.93 | 6.28 | 8.00 | 7.19 | 9.60 | 8.88
Source | 71.80 | 63.00 | 65.65 | 26.15 | 73.00 | 46.43 | 31.03 | 48.51 | 48.84 | 35.35 | 25.45 | 28.77 | 30.76 | 38.34 | 43.74 | 45.12
BN | 41.70 | 38.35 | 43.93 | 29.59 | 44.93 | 37.70 | 29.89 | 42.60 | 37.45 | 37.00 | 29.72 | 30.52 | 34.24 | 31.80 | 35.40 | 36.32
PL | 41.26 | 38.08 | 43.45 | 29.59 | 44.68 | 37.35 | 29.69 | 42.33 | 37.33 | 36.50 | 29.55 | 30.21 | 33.80 | 31.58 | 35.11 | 36.03
TFA†∗ | 39.27 | 35.77 | 40.52 | 27.98 | 42.05 | 34.99 | 28.51 | 38.72 | 35.01 | 32.90 | 27.83 | 28.81 | 31.43 | 30.60 | 34.38 | 33.92
TTT++†∗ | 38.39 | 34.51 | 38.95 | 26.52 | 41.53 | 33.65 | 27.64 | 37.75 | 34.06 | 31.32 | 26.78 | 27.22 | 30.18 | 29.46 | 33.26 | 32.75
TENT† | 38.37 | 34.72 | 39.50 | 27.41 | 40.91 | 33.81 | 27.99 | 39.10 | 34.69 | 31.97 | 27.36 | 27.94 | 31.80 | 29.09 | 32.94 | 33.17
CAFA | 36.97 | 33.53 | 37.41 | 26.21 | 38.96 | 32.46 | 27.26 | 36.82 | 33.64 | 30.13 | 26.53 | 27.07 | 30.20 | 28.23 | 32.05 | 31.83

Table 10: Classification error (%) on the CIFAR10-C (upper group) and
CIFAR100-C (lower group) datasets with severity level 2 corruptions. † denotes
the results obtained from the official codes, and ∗ indicates the test-time
training methods evaluated in the test-time adaptation setting.

Method | Gaus. | Shot | Impu. | Defo. | Glas. | Moti. | Zoom | Snow | Fros. | Fog | Brig. | Cont. | Elas. | Pixe. | Jpeg. | Average
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Source | 24.43 | 14.91 | 18.87 | 5.32 | 37.16 | 11.58 | 6.75 | 18.33 | 11.29 | 6.76 | 5.38 | 5.94 | 7.06 | 10.35 | 14.42 | 13.24
BN | 11.24 | 8.92 | 12.66 | 7.13 | 16.28 | 10.17 | 7.08 | 12.77 | 9.45 | 8.43 | 7.02 | 7.14 | 9.50 | 8.35 | 10.71 | 9.79
PL | 11.17 | 8.81 | 12.71 | 6.92 | 16.27 | 10.10 | 7.15 | 12.67 | 9.43 | 8.41 | 7.02 | 6.96 | 9.45 | 8.31 | 10.66 | 9.74
TENT† | 10.52 | 8.15 | 11.45 | 6.65 | 15.03 | 9.64 | 6.67 | 11.18 | 8.69 | 7.44 | 6.50 | 6.72 | 8.80 | 7.82 | 9.85 | 9.01
TFA†∗ | 10.29 | 8.09 | 10.85 | 6.35 | 14.87 | 8.97 | 6.43 | 11.29 | 8.27 | 7.47 | 6.21 | 6.40 | 8.39 | 7.42 | 9.61 | 8.73
TTT++†∗ | 10.47 | 7.96 | 10.16 | 5.89 | 14.92 | 8.63 | 6.26 | 10.40 | 8.26 | 6.80 | 5.78 | 5.86 | 7.84 | 7.07 | 9.75 | 8.40
CAFA | 9.22 | 7.51 | 9.65 | 5.78 | 13.22 | 8.20 | 5.86 | 9.71 | 7.57 | 6.55 | 5.87 | 6.02 | 7.64 | 6.95 | 8.90 | 7.91
Source | 59.17 | 44.40 | 54.99 | 24.91 | 73.55 | 37.45 | 28.70 | 49.25 | 37.83 | 29.75 | 25.01 | 26.84 | 29.35 | 34.25 | 41.17 | 39.77
BN | 37.60 | 33.23 | 39.55 | 29.11 | 44.32 | 33.91 | 29.63 | 38.89 | 33.89 | 33.74 | 29.05 | 29.74 | 33.21 | 30.94 | 34.03 | 34.06
PL | 37.40 | 32.92 | 39.01 | 28.88 | 43.91 | 33.56 | 29.45 | 38.67 | 33.68 | 33.42 | 28.86 | 29.48 | 32.90 | 30.71 | 33.82 | 33.78
TFA†∗ | 35.27 | 31.10 | 36.71 | 27.86 | 42.04 | 32.00 | 28.65 | 36.03 | 31.18 | 30.22 | 27.60 | 28.32 | 30.65 | 29.30 | 33.11 | 32.00
TTT++†∗ | 33.60 | 29.74 | 34.88 | 25.90 | 41.21 | 30.64 | 26.97 | 34.86 | 30.17 | 28.89 | 26.19 | 26.68 | 29.57 | 28.29 | 31.84 | 30.63
TENT† | 34.29 | 30.72 | 35.37 | 27.22 | 40.35 | 31.03 | 27.54 | 35.55 | 31.05 | 29.80 | 26.84 | 27.60 | 30.54 | 28.43 | 31.82 | 31.21
CAFA | 33.02 | 29.38 | 33.75 | 26.09 | 38.34 | 29.63 | 26.73 | 33.47 | 29.79 | 28.18 | 25.94 | 26.45 | 29.58 | 27.64 | 30.95 | 29.93

Table 11: Classification error (%) on the CIFAR10-C (upper group) and
CIFAR100-C (lower group) datasets with severity level 1 corruptions. † denotes
the results obtained from the official codes, and ∗ indicates the test-time
training methods evaluated in the test-time adaptation setting.

Method | Gaus. | Shot | Impu. | Defo. | Glas. | Moti. | Zoom | Snow | Fros. | Fog | Brig. | Cont. | Elas. | Pixe. | Jpeg. | Average
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Source | 14.00 | 10.09 | 11.62 | 5.24 | 38.69 | 7.75 | 7.40 | 9.48 | 8.14 | 5.74 | 5.37 | 5.47 | 7.69 | 6.69 | 10.51 | 10.26
BN | 8.89 | 8.08 | 10.17 | 7.04 | 16.16 | 8.55 | 7.55 | 9.44 | 7.66 | 7.47 | 7.00 | 6.92 | 9.81 | 7.50 | 8.76 | 8.73
PL | 8.83 | 7.90 | 10.17 | 6.94 | 16.05 | 8.42 | 7.59 | 9.40 | 7.54 | 7.41 | 6.93 | 6.81 | 9.77 | 7.38 | 8.70 | 8.66
TENT† | 8.25 | 7.50 | 9.71 | 6.54 | 14.75 | 7.94 | 7.15 | 8.71 | 6.94 | 6.62 | 6.43 | 6.48 | 9.35 | 7.29 | 8.37 | 8.14
TFA†∗ | 7.94 | 7.24 | 8.95 | 6.26 | 14.38 | 7.63 | 7.11 | 8.37 | 7.10 | 6.45 | 6.23 | 6.02 | 8.64 | 6.75 | 7.69 | 7.79
TTT++†∗ | 8.23 | 6.97 | 8.52 | 5.61 | 15.17 | 7.43 | 6.84 | 7.66 | 6.55 | 5.95 | 5.61 | 5.80 | 8.39 | 6.65 | 7.86 | 7.55
CAFA | 7.37 | 6.54 | 8.36 | 5.80 | 13.07 | 6.99 | 6.34 | 7.84 | 6.38 | 6.04 | 5.70 | 5.76 | 8.03 | 6.35 | 7.44 | 7.20
Source | 42.81 | 35.37 | 40.39 | 24.70 | 74.15 | 31.29 | 29.80 | 35.16 | 31.07 | 25.20 | 24.79 | 25.23 | 30.04 | 28.64 | 34.83 | 34.23
BN | 32.74 | 30.64 | 34.21 | 28.69 | 44.57 | 32.11 | 30.34 | 33.10 | 30.59 | 29.69 | 28.55 | 28.90 | 34.23 | 29.94 | 31.52 | 31.99
PL | 32.61 | 30.50 | 33.70 | 28.52 | 44.10 | 31.83 | 30.19 | 32.82 | 30.52 | 29.25 | 28.46 | 28.61 | 33.84 | 29.72 | 31.27 | 31.73
TFA†∗ | 30.64 | 28.78 | 31.88 | 27.43 | 41.80 | 30.58 | 29.04 | 30.70 | 29.27 | 27.84 | 27.31 | 27.56 | 31.11 | 28.58 | 30.29 | 30.19
TTT++†∗ | 29.63 | 28.25 | 30.33 | 25.63 | 41.00 | 29.07 | 27.36 | 29.76 | 27.81 | 26.35 | 26.17 | 26.14 | 30.11 | 27.34 | 29.15 | 28.94
TENT† | 30.07 | 28.53 | 31.01 | 26.84 | 40.33 | 29.45 | 28.18 | 30.56 | 28.49 | 27.14 | 26.71 | 27.07 | 31.33 | 27.76 | 29.12 | 29.51
CAFA | 28.82 | 27.70 | 30.09 | 25.84 | 38.38 | 28.02 | 27.09 | 29.06 | 27.36 | 26.23 | 25.80 | 25.91 | 30.33 | 26.94 | 28.63 | 28.41

Table 12: Classification error (%) on the severity level 5 corruptions in the
CIFAR10-C dataset with different random seeds.

Trial | Gaus. | Shot | Impu. | Defo. | Glas. | Moti. | Zoom | Snow | Fros. | Fog | Brig. | Cont. | Elas. | Pixe. | Jpeg. | Average
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
0 | 14.05 | 13.06 | 21.35 | 8.07 | 20.45 | 11.10 | 6.96 | 11.92 | 11.31 | 13.22 | 7.06 | 7.14 | 16.05 | 9.65 | 11.70 | 12.21
1 | 13.79 | 12.90 | 21.05 | 8.13 | 21.08 | 10.85 | 6.92 | 11.92 | 11.20 | 13.10 | 6.83 | 7.22 | 16.03 | 9.63 | 11.57 | 12.15
2 | 14.07 | 12.60 | 21.01 | 7.89 | 20.73 | 10.71 | 6.86 | 11.87 | 11.24 | 13.46 | 6.92 | 6.94 | 16.16 | 9.80 | 11.55 | 12.12
3 | 13.80 | 12.99 | 20.83 | 8.00 | 20.62 | 10.55 | 6.71 | 11.89 | 11.64 | 13.68 | 7.11 | 7.04 | 16.22 | 9.82 | 11.25 | 12.14
4 | 13.70 | 12.71 | 21.07 | 8.09 | 20.94 | 10.68 | 6.87 | 11.96 | 11.41 | 13.82 | 7.04 | 7.05 | 15.90 | 9.67 | 11.68 | 12.17
std | 0.17 | 0.19 | 0.19 | 0.09 | 0.25 | 0.21 | 0.10 | 0.03 | 0.18 | 0.30 | 0.11 | 0.11 | 0.12 | 0.09 | 0.18 | 0.03

Figure 4: t-SNE visualizations of ours (CAFA) and TENT from (a) Fog and (b)
Elastic Transform corruptions with severity level 5 in the CIFAR10-C dataset.
Figure 5: Change of the representation space of test samples from our method
as adaptation proceeds. Representation space is visualized by the t-SNE
algorithm, and visualizations are obtained from the Fog corruption with
severity level 5 in the CIFAR10-C dataset.