# Controllable Continual Test-Time Adaptation

Ziqi Shi12 Fan Lyu1∗ Ye Liu3 Fanhua Shang2
Fuyuan Hu4 Wei Feng2 Zhang Zhang1 Liang Wang1
1New Laboratory of Pattern Recognition, Institute of Automation, Chinese
Academy of Sciences
2College of Intelligence and Computing, Tianjin University
3School of Computer Science & Engineering, Linyi University
4School of Electronic & Information Engineering, Suzhou University of Science
and Technology
{szqi, fhshang<EMAIL_ADDRESS>{zzhang<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
Equal contributionCorresponding author.

###### Abstract

Continual Test-Time Adaptation (CTTA) is an emerging and challenging task
where a model trained in a source domain must adapt to continuously changing
conditions during testing, without access to the original source data. CTTA is
prone to error accumulation due to uncontrollable domain shifts, leading to
blurred decision boundaries between categories. Existing CTTA methods
primarily focus on suppressing domain shifts, which proves inadequate during
the unsupervised test phase. In contrast, we introduce a novel approach that
guides rather than suppresses these shifts. Specifically, we propose
Controllable Continual Test-Time Adaptation (C-CoTTA), which explicitly
prevents any single category from encroaching on others, thereby mitigating
the mutual influence between categories caused by uncontrollable shifts.
Moreover, our method reduces the sensitivity of model to domain
transformations, thereby minimizing the magnitude of category shifts.
Extensive quantitative experiments demonstrate the effectiveness of our
method, while qualitative analyses, such as t-SNE plots, confirm the
theoretical validity of our approach. Our code is available at
https://github.com/RenshengJi/C-CoTTA.

##  1 Introduction

Continual Test-Time Adaptation (CTTA) [44] is becoming an emerging field,
which explores the adaptability of any machine learning model during test time
in dynamic environments. The primary objective of CTTA is to enable a
pretrained model to adapt to continuously changing scenarios, where the
distribution of data shifts over time. CTTA is practical in many long-term
intelligent applications, such as autopilot [17, 30, 5], monitoring [13],
medical image analysis [6, 14], where models need to remain robust and
accurate against possible changes over extended periods.

The main challenge of CTTA is the accumulation of errors caused by the lack of
real labels and the continuous domain shifts. Existing methods mainly avoid
error accumulation by using strategies such as Mean Teacher [40],
augmentation-averaged predictions [44, 24, 22], and selecting reliable samples
[51, 28, 29, 45] to minimize the impact of domain shift. However, these
methods focus on suppress the shifts, few methods explicitly attempt to guide
or control the shifts. This is becuase the lack of labels in the target domain
and the inability to obtain source domain data in CTTA. We evaluate this on
CIFAR10C, as shown in Fig. 1(a), we find that in the domain adaptation
process, multiple categories to lean towards a confusion region. In fact, CTTA
meets continuous unknown domain shifts, making this shift almost
uncontrollable. This phenomenon will worsen with the changing domain over
time, resulting in error accumulation.

Therefore, we hypothesize that instead of attempting to suppress the shift,
guiding or controlling the shift to remain class-separable may also achieve
effective adaptation. On top of this, the main focus of this paper is to study
how to achieve controllable domain shift in CTTA. Generally, controlling
domain shifts means to control the direction of the shift in feature space.
First of all, we first need to accurately represent the direction of domain
shift. Inspired by the interpretable machine learning [19], we represent shift
directions using the tool of Concept Ativation Vectors (CAV) [31], which
represents the transformation path from one concepts to another. In CTTA, the
CAV can be represented by the vector from one prototype to another. First, for
a specific category of domain shift direction, we obtain it by subtracting the
category prototype of the target domain in the feature space from the category
prototype of the source domain. Then, for the overall domain shift direction,
we obtain it by subtracting the overall prototype of the target domain in the
feature space from the overall prototype of the source domain.

To further control domain shift, we construct the Control Domain Shift (CDS)
loss and the Control Class Shift (CCS) loss. CDS refers to controlling the
shift of the overall domain by constraining the model’s sensitivity in that
direction, thus reducing the impact of domain shift on model performance. CCS
controls the shift of specific categories by constraining the shift direction
of each category to avoid biasing other categories. As shown in Fig. 1(b), our
method achieve controllable domain shift, and the direction of the shift will
not blur the classification boundaries. Extensive experiments are conducted on
three large-scale benchmark datasets to validate the effectiveness of the
proposed C-CoTTA framework in various challenging and realistic scenarios.

Figure 1: t-SNE visualization of controllable domain shift in CTTA. (a) For
CoTTA [44], due to the lack of control over domain shift, categories being
biased towards others, resulting in fuzzy classification boundaries. (b) In
contrast, our method achieves controllable domain shift, so even if categories
are shift, it will not lead to confusion among categories.

Our contributions are three-fold:

  1. (1)

We evaluate and find that only suppress domain shifts is insufficient, which
may lead to blurred classification boundaries. In contrast, we propose to
guide and control shifts to keep the class-separability.

  2. (2)

We propose a simple and effective direction representation based on Concept
Ativation Vectors (CAV) in interpretable machine learning, which utilizes the
difference between two prototypes in the feature space.

  3. (3)

We propose to explicitly control the direction of specific category bias by
preventing any category from leaning towards other categories, in order to
prevent the blurring of classification boundaries; at the same time, by
reducing the sensitivity of the model to domain shift, we control the overall
domain shift to alleviate the impact of domain shift on domain adaptation.

##  2 Related work

###  2.1 Continual Test-Time Adaptation

Continual Test-Time Adaptation (CTTA) [44, 25, 23] is an emerging paradigm
within the machine learning community designed to address the dynamic nature
of real-world data distributions. Unlike traditional Test-Time Adaptation
(TTA) [18, 39, 43], which typically assumes a fixed target domain in a source-
free and online manner, CTTA operates under the assumption that the target
domain may change over time. The main challenge in CTTA is the potential for
catastrophic forgetting and error accumulation. During the test time, as the
model adapts to new distributions, it risks losing previously learned
knowledge, which can result in a degradation in performance known as
catastrophic forgetting [41]. Moreover, the utilization of pseudo-labels
derived from the model’s own predictions can introduce errors, which may
accumulate over time [21, 44], especially when there are frequent domain
shifts.

To address the challenges of error accumulation in CTTA, researchers have
developed various strategies. A number of works [44, 24, 22, 11] employ
augmentation-averaged predictions for the teacher model to boost the teacher’s
confidence, while others[4, 8] add perturbations to the student to enhance the
model’s robustness. Various methods[51, 28, 29, 45, 38] focuses on selecting
reliable samples to eliminate the impact of misclassified samples on domain
adaptation. As to the challenge of catastrophic forgetting, Wang et al. [44]
and Brahma et al.[3] believe that the source model is more reliable, thus they
are designed to restore the source parameters. While these studies address the
CTTA issue at the model level, other research efforts [12, 51, 27] leverage
visual domain prompts or a limited subset of parameters to extract ongoing
target domain knowledge. However, these approaches primarily focus on
suppressing domain shift and there are few methods that explicitly attempt to
guide or control domain shift.

###  2.2 Concept Activation Vectors

Concept Activation Vectors (CAVs) is an interpretability tool for explaining
decision-making processes in deep learning models. Originally, the authors of
[19] define CAV as the normal to a hyperplane that separates examples without
a concept from examples with a concept in the model’s latent activations. This
hyperplane is commonly computed by solving a classification problem, for
example, using Support Vector Machines (SVMs)[1], ridge[7], lasso[33] or
logistic regression [53]. Given its ability to effectively orient concepts,
CAVs have been employed for a plethora of tasks in recent years, such as
concept sensitivity testing [19], model correction for shortcut removal [1,
32, 9], knowledge discovery by investigation of internal model states [26],
and training of post-hoc concept bottleneck models [53]. However, common
regression-based methods tend to deviate from the true conceptual direction
due to factors such as noise in the data [15]. To that end, signal-pattern-
based CAVs (referred to “signalCAVs”) have been proposed [31], which are more
robust against noise [47, 10, 35, 2]. However, during test-time, we may not
have access to all samples of a prototype, and due to the lack of true labels,
misclassified samples may contaminate the prototype. Therefore, the
construction of a prototype is different from interpretable machine learning.

##  3 Methodology

###  3.1 Problem Definition

Given a classification model pre-trained on a source domain, CTTA methods
adapt the source model to the unlabeled target data, where the domain
continuously changes. The unsupervised dataset of target domains are denoted
as $\mathcal{D}^{k}=\\{x^{k}_{m}\\}^{N^{k}}_{m=1}$, where $k$ is the target
domain index. As shown in Figure 1, in the process of CTTA, if the domain
shifts are not controlled, some categories may generate bias towards other
categories, resulting in blurred classification boundaries. In this paper, we
propose to explicit control over the shift direction in CTTA. specific
categories and the overall domain. In the following, we first study how to
represent domain shifts in Sec. 3.2. Then, we propose to control the shift
within the process of CTTA in Sec. 3.3.

###  3.2 Representing Shift via CAV

Domain shift refers to the distribution shift of each class that occurs in the
feature space, generally caused by differences between the target domain
distribution in the testing phase and the source domain distribution in the
training phase. Therefore, how to represent the domain shift direction in the
feature space in CTTA is a challenge.

In the field of interpretable machine learning [19, 26], CAV [19] refer to the
normal to a hyperplane that separates examples without a concept from examples
with a concept in the model’s latent activations feature space. CAV is widely
used in areas such as model correction for shortcut removal [1, 32, 9]. The
concept in CAV generally refers to high-level semantic information, such as
whether there are a large number of striped structures in an image. For
example, in the concept of stripe, the label for features extracted from
images with stripes is 1, and without stripes is 0.

CAV can be calculated in different methods, we use the signal-pattern-based
CAVs (SCAV) [31] which provide a simple but effective way to represent the CAV
$\mathbf{v}$ as follows:

$\mathbf{v}=\frac{\textrm{cov}[f(x),t]}{\textrm{cov}[t,t]}=\frac{\sum(f(x_{i})-\overline{f}(x))(t_{i}-\overline{t})}{\sum(t_{i}-\overline{t})^{2}},\quad
t_{i}=\begin{cases}1&\text{if }x_{i}\in\mathcal{X}_{c},\\\ 0&\text{if
}x_{i}\in\mathcal{X}_{n}.\end{cases},$ (1)

where $f(\cdot)$ represents the feature extractor and $t$ represents the
concept label of the features, $\mathcal{X}_{c}$ and $\mathcal{X}_{n}$ denote
sets of samples that either possess or lack the specified concept,
respectively. Mean feature $\overline{f}(x)=\frac{1}{N}\sum f(x_{i})$ and mean
label $\overline{t}=\frac{1}{N}\sum t_{i}$.

Figure 2: The pipeline of C-CoTTA. (a) Based on the mean teacher framework,
perturb the student to enhance model robustness, while optimizing using
symmetric cross-entropy. (b) Control the overall domain shift by constraining
the model’s sensitivity to domain shift directions. (c) Control the shift of a
specific category by directly controlling the shift direction of any category
to prevent bias towards other categories.

Nevertheless, in the traditional CAV application scenario, the concept label
$t$ is manually annotated offline. In contrast, in unsupervised CTTA scenario,
to represent the domain shift direction in an online manner is intractable.
Therefore, we use the pseudo-label $y$ obtained by the existing model to
represent concepts automatically. The shift representation for the direction
in CTTA can be computed as follows:

$\mathbf{v}=\frac{\textrm{cov}[f(x),y]}{\textrm{cov}[y,y]}=\frac{\sum(f(x_{i})-\overline{f}(x))(y_{i}-\overline{y})}{\sum(y_{i}-\overline{y})^{2}},\quad
y_{i}=\begin{cases}1&\text{if }x_{i}\in\mathcal{X}_{t},\\\ 0&\text{if
}x_{i}\in\mathcal{X}_{s}.\end{cases},$ (2)

At the same time, through deduction (refer to Appendix A for details), it is
determined that the direction can be further represented as the difference
between two prototypes in the feature space.

$\mathbf{v}=\frac{1}{|\mathcal{X}_{t}|}{\sum\limits_{x_{i}\in\mathcal{X}_{t}}f(x_{i})}-\frac{1}{|\mathcal{X}_{s}|}{\sum\limits_{x_{i}\in\mathcal{X}_{s}}f(x_{i})}=\mathbf{p}_{t}-\mathbf{p}_{s}.$
(3)

Then, we can use this to calculate various directional vectors, in order to
further control domain shift.

Domain-level source-to-target shift. We represent the shift of the overall
domain as below:

$\mathbf{v}^{s\to t}=\mathbf{p}^{t}-\mathbf{p}^{s},$ (4)

This calculates by subtracting the target domain prototype $\mathbf{p}^{t}$
composed of all samples in each batch at test-time from the source domain
prototype $\mathbf{p}^{s}$ composed of prototypes of all categories in the
source domain as shown in Figure 2(b).

Class-level source-to-target shift. We also represent the shift of a specific
category as below:

$\mathbf{v}_{i}^{s\to t}=\mathbf{p}^{t}_{i}-\mathbf{p}^{s}_{i},$ (5)

where we construct it by subtracting the prototype $\mathbf{p}^{t}_{i}$
composed of trustworthy samples belonging to that category in each batch at
test-time from the source domain prototype $\mathbf{p}^{s}_{i}$ of that
category as shown in Figure 2(c). Moreover, during test time, some samples may
be misclassified due to the lack of real labels, leading to distortion in the
extracted prototypes. Therefore, we estimate the entropy $H(x^{t})$ for each
sample $x^{t}$ from the target domain using the model. We then set aside
samples with an entropy exceeding a predefined threshold $E_{0}$ following
[29].

Class-to-class source shift. We construct class relative shift from category
$i$ to $j$ as below:

$\mathbf{v}_{i\to j}^{s}=\mathbf{p}^{s}_{j}-\mathbf{p}^{s}_{i}.$ (6)

This calculates by using the source domain category prototypes
$\mathbf{p}^{s}_{i}$ and $\mathbf{p}^{s}_{j}$ as shown in Figure 2(c).

###  3.3 Control of shift

We control the shift at domain and class levels. On one hand, the shift of
different categories are influenced by the overall domain shift, showing a
certain degree of convergence, meaning that after domain shift, the distances
between different categories in the feature space become closer. This finding
is similar to the conclusions drawn in studies such as [50, 20, 34], where
researchers found that the feature norms in the target domain are relatively
small, so we control the overall domain shift. On the other hand, the shift of
different categories is related to their characteristics.

#### Domain-level shift controlling.

First, we propose to control the shift of the overall domain at the domain
level. Specifically, we model the shift sensitivity by considering the
gradient of the model output $h(g(x_{i}))$ with respect to the feature map
$g(x_{i})$, and then combine it with the overall domain shift direction
${\mathbf{v}^{s\to t}}$. We can then reduce the sensitivity of the model to
domain shift by constraining the direction gradient as follows:

$\displaystyle\mathcal{L}_{\mathrm{CDS}}=\sum\limits_{x_{i}\in{\mathcal{X}_{b}}}\left|\frac{\partial
h(g(x_{i}))}{\partial g(x_{i})}\cdot{\mathbf{v}^{s\to t}}\right|,$ (7)

where $h(\cdot)$ represents the remaining part of the model. Intuitively, the
Control Domain Shift (CDS) loss $\mathcal{L}_{\mathrm{CDS}}$ enforces the
model output to not change when slightly adding or removing activations along
the bias direction as follows:

$\displaystyle\lim_{\epsilon\rightarrow
0}\frac{h(g(\mathbf{x})+\epsilon{\mathbf{v}^{s\to
t}})-h(g(\mathbf{x}))}{\epsilon}=0.$ (8)

Thus, by minimizing $\mathcal{L}_{\mathrm{CDS}}$, the model becomes
insensitive towards the domain shift direction, thereby reducing the impact of
domain shift on the domain adaptation process.

Algorithm 1 Controllable Continual Test-Time Adaptation

0: Target domains data $\mathcal{D}^{k}=\\{x^{k}_{m}\\}^{N^{k}}_{m=1}$, Source
model, Source domain class prototype $\mathbf{p}_{i}^{s}$

1: Generate the category relative direction vectors $\mathbf{v}_{i\to
j}^{s}=\mathbf{p}^{s}_{j}-\mathbf{p}^{s}_{i}$ before domain adaptation

2: for a domain $k$ in $K$ do

3: for a batch $\\{x^{k}_{b}\\}_{b=1}^{B}$ in $\mathcal{D}^{k}$ do

4: Forward the batch, make predictions and get features

5: Identify reliable samples with low entropy using a predefined threshold
$E_{0}$

6: Compute the direction of the domain shift $\mathbf{v}^{s\to
t}=\mathbf{p}^{t}-\mathbf{p}^{s}$ and constrained via Eq. 7

7: Compute the direction of the class shift $\mathbf{v}_{i}^{s\to
t}=\mathbf{p}^{t}_{i}-\mathbf{p}^{s}_{i}$ and constrained via Eq. 9

8: Compute the symmetric cross-entropy loss via Eq. 10

9: Optimize model by minimizing $\mathcal{L}$ via Eq. 11 and update student
and teacher models

10: end for

11: end for

#### Class-level shift controlling.

Then, in order to prevent uncontrollable shifts of each category, we propose
to control class-level shift to avoid any category leaning towards other
categories. This requires that the direction of shift $\mathbf{v}_{i}^{s\to
t}$ for any category is preferably the opposite direction of the direction
$\mathbf{v}_{i\to j}^{s}$ of other categories relative to that category as
shown in Fig. 2(c). This means that the dot product of $\mathbf{v}_{i}^{s\to
t}$ and $\mathbf{v}_{i\to j}^{s}$ should be as small as possible. The Control
Class Shift (CCS) loss is calculated as follows:

$\displaystyle\mathcal{L}_{\mathrm{CCS}}=\sum_{i=1}^{c}\sum_{j\neq
i}^{c}\frac{\mathbf{v}_{i}^{s\to t}}{\|{\mathbf{v}_{i}^{s\to
t}}\|_{2}}\cdot\frac{{\mathbf{v}_{i\to j}^{s}}}{\|{\mathbf{v}_{i\to
j}^{s}}\|_{2}},$ (9)

where we normalize $\mathbf{v}_{i}^{s\to t}$ and $\mathbf{v}_{i\to j}^{s}$.
The loss $\mathcal{L}_{\mathrm{CCS}}$ is to prevent any category from shifting
towards other categories, achieving controllable domain shift, effectively
preventing the decrease in classification performance caused by blurred
category boundaries in continuous domain adaptation.

###  3.4 Overall Objective

In addition, our work also utilizes $\mathcal{L}_{\mathrm{SCE}}$ derived from
[8], which following the mean teacher framework, which involves simply
averaging the weights of a student model over time, the resulting teacher
model provides a more accurate prediction function than the final function of
the student, meanwhile perturbs the student to enhance the robustness of the
model [48, 37] and employs the symmetric cross-entropy (SCE) [46] which has
superior gradient properties compared to the commonly used cross-entropy loss.
For enhancing the output of students $q$ and teachers $p$, the symmetric
cross-entropy is defined as follows:

$\displaystyle\mathcal{L}_{\mathrm{SCE}}=-\sum\nolimits_{c=1}^{C}\mathbf{q}_{c}\,\mathrm{log}\,\mathbf{p}_{c}-\sum\nolimits_{c=1}^{C}\mathbf{p}_{c}\,\mathrm{log}\,\mathbf{q}_{c}.$
(10)

The overall objective of our proposed continual test-time adaptation method is
as follows:

$\displaystyle\mathcal{L}=\mathcal{L}_{\mathrm{SCE}}+\lambda_{1}\mathcal{L}_{\mathrm{CDS}}+\lambda_{2}\mathcal{L}_{\mathrm{CCS}},$
(11)

where $\lambda_{1}$, and $\lambda_{2}$ are the hyperparameters.
$\mathcal{L}_{\mathrm{CDS}}$ refers to controlling the shift of the overall
domain, which constrains the model’s sensitivity in that direction.
$\mathcal{L}_{\mathrm{CCS}}$ controls the shift of specific categories, which
constrains the shift direction of each category to avoid biasing other
categories. The overall frame diagram is shown in Figure 2.

We illustrate the whole algorithm in Algorithm 1. First, before domain
adaptation begins, we use the source domain category prototypes to calculate
the inter-class relative direction vector $\mathbf{v}_{i\to j}^{s}$. During
domain adaptation, on one hand, we calculate the shift direction
$\mathbf{v}_{i}^{s\to t}$ for specific categories and constrain it through the
loss $\mathcal{L}_{\mathrm{CCS}}$; on the other hand, we calculate the shift
direction $\mathbf{v}^{s\to t}$ for the entire domain and constrain it through
the loss $\mathcal{L}_{\mathrm{CDS}}$. Additionally, we compute the symmetric
cross-entropy loss for the prediction logits of the student and teacher,
optimize it via Eq. 11, and update the student and teacher models.

##  4 Experiments

###  4.1 Experimental Setting

Datasets. We evaluate our proposed method on three continual test-time
adaptation benchmark: CIFAR10-C, CIFAR100-C, and ImageNet-C. Each dataset
contains 15 types of corruptions with 5 levels of severity, ranging from 1 to
5. For simplicity in tables, we use Gauss., Impul., Defoc., Brit., Contr.,
Elas. and Pix. to represent Gaussian, Impulse, Defocus, Brightness, Contrast,
Elastic,and Pixelate, respectively.

Pretrained Model. Following previous studies [43, 44], we adopt the pretrained
WideResNet-28 [54], ResNeXt-29 [49] and ResNet-50 [16] for CIFAR10-C,
CIFAR100-C and Imagenet-C, respectively. Similar to CoTTA, we update all the
trainable parameters in all experiments.

Methods to be Compared. We compare our C-CoTTA with the original model
(Source) and multiple state-of-the-art (SOTA) methods such as BN [21, 36],
TENT [43], CoTTA [44], RoTTA [52], SATA [4], SWA [51], PETAL [3], RMT [8], DSS
[45]. All compared methods utilize the same backbone and pretrained model. All
experiments are conducted on a single RTX 4090.

###  4.2 Results for Continual Test-Time Adaptation

Experiments on CIFAR10-C. We first evaluate the effectiveness of the proposed
model on the CIFAR10-C dataset. We compare our method to the source-only
baseline and nine SOTA methods. As shown in Table 1, directly using pre-
trained model without adaptation yields a high average error rate of 43.5%. BN
method improve the performance by 23.1% compared to the source-only baseline.
Among all comparison methods, SWA achieve the lowest error rate of 11.2%,
11.8% and 11.5% on Defoc., Motion and Fog, respectively. Both PETAL and SATA
achieve the lowest error rate of 7.4% on Brit.. In other conditions, our
proposed method outperforms or is comparable to all the above methods. In
conclusion, our method achieve the lowest average error rate, which is reduced
to 14.7%.

Table 1: Classification error rate (%) for standard CIFAR10-C continual test-
time adaptation task.

Method | Gauss. | Shot | Impul. | Defoc. | Glass | Motion | Zoom | Snow | Frost | Fog | Brit. | Contr. | Elas. | Pix. | Jpeg | Mean
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Source | 72.3 | 65.7 | 72.9 | 46.9 | 54.3 | 34.8 | 42.0 | 25.1 | 41.3 | 26.0 |  09.3 | 46.7 | 26.6 | 58.5 | 30.3 | 43.5
BN [21, 36] | 28.1 | 26.1 | 36.3 | 12.8 | 35.3 | 14.2 | 12.1 | 17.3 | 17.4 | 15.3 |  08.4 | 12.6 | 23.8 | 19.7 | 27.3 | 20.4
TENT [43] | 24.8 | 20.6 | 28.5 | 15.1 | 31.7 | 17.0 | 15.6 | 18.3 | 18.3 | 18.1 | 11.0 | 16.8 | 23.9 | 18.6 | 23.9 | 20.1
CoTTA [44] | 24.5 | 21.5 | 25.9 | 12.0 | 27.7 | 12.2 | 10.7 | 15.0 | 14.1 | 12.7 |  07.6 | 11.0 | 18.5 | 13.6 | 17.7 | 16.3
RoTTA [52] | 30.3 | 25.4 | 34.6 | 18.3 | 34.0 | 14.7 | 11.0 | 16.4 | 14.6 | 14.0 |  08.0 | 12.4 | 20.3 | 16.8 | 19.4 | 19.3
RMT [8] | 24.1 | 20.2 | 25.7 | 13.2 | 25.5 | 14.7 | 12.8 | 16.2 | 15.4 | 14.6 | 10.8 | 14.0 | 18.0 | 14.1 | 16.6 | 17.0
PETAL [3] | 23.7 | 21.4 | 26.3 | 11.8 | 28.8 | 12.4 | 10.4 | 14.8 | 13.9 | 12.6 |  07.4 | 10.6 | 18.3 | 13.1 | 17.1 | 16.2
SATA [4] | 23.9 | 20.1 | 28.0 | 11.6 | 27.4 | 12.6 | 10.2 | 14.1 | 13.2 | 12.2 |  07.4 | 10.3 | 19.1 | 13.3 | 18.5 | 16.1
DSS [45] | 24.1 | 21.3 | 25.4 | 11.7 | 26.9 | 12.2 | 10.5 | 14.5 | 14.1 | 12.5 |  07.8 | 10.8 | 18.0 | 13.1 | 17.3 | 16.0
SWA [51] | 23.9 | 20.5 | 24.5 | 11.2 | 26.3 | 11.8 | 10.1 | 14.0 | 12.7 | 11.5 |  07.6 |  09.5 | 17.6 | 12.0 | 15.8 | 15.3
Ours | 22.7 | 17.9 | 23.8 | 11.6 | 24.3 | 12.8 |  09.5 | 13.1 | 12.4 | 11.6 |  08.0 |  09.5 | 16.4 | 11.4 | 15.6 | 14.7

Experiments on CIFAR100-C. To further demonstrate the effectiveness of the
proposed method, we evaluate it on the more difficult CIFAR100-C task with the
source-only baseline and eight SOTA methods. The experimental results are
shown in Table 2. Generally speaking, our method not only achieved the lowest
error rates on the Snow, Frost, Fog, and Pix. tasks but also had the lowest
average error rate. We improve the performance by 16.5% and 0.4% compared to
the source-only baseline and SATA, respectively.

Table 2: Classification error rate (%) for standard CIFAR100-C continual test-
time adaptation task.

Method | Gauss. | Shot | Impul. | Defoc. | Glass | Motion | Zoom | Snow | Frost | Fog | Brit. | Contr. | Elas. | Pix. | Jpeg | Mean
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Source | 73.0 | 68.0 | 39.4 | 29.3 | 54.1 | 30.8 | 28.8 | 39.5 | 45.8 | 50.3 | 29.5 | 55.1 | 37.2 | 74.7 | 41.2 | 46.4
BN [21, 36] | 42.1 | 40.7 | 42.7 | 27.6 | 41.9 | 29.7 | 27.9 | 34.9 | 35.0 | 41.5 | 26.5 | 30.3 | 35.7 | 32.9 | 41.2 | 35.4
TENT [43] | 37.2 | 35.8 | 41.7 | 37.9 | 51.2 | 48.3 | 48.5 | 58.4 | 63.7 | 71.1 | 70.4 | 82.3 | 88.0 | 88.5 | 90.4 | 60.9
CoTTA [44] | 40.1 | 37.7 | 39.7 | 26.9 | 38.0 | 27.9 | 26.4 | 32.8 | 31.8 | 40.3 | 24.7 | 26.9 | 32.5 | 28.3 | 33.5 | 32.5
RoTTA [52] | 49.1 | 44.9 | 45.5 | 30.2 | 42.7 | 29.5 | 26.1 | 32.2 | 30.7 | 37.5 | 24.7 | 29.1 | 32.6 | 30.4 | 36.7 | 34.8
RMT [8] | 40.2 | 36.2 | 36.0 | 27.9 | 33.9 | 28.4 | 26.4 | 28.7 | 28.8 | 31.1 | 25.5 | 27.1 | 28.0 | 26.6 | 29.0 | 30.2
PETAL [3] | 38.3 | 36.4 | 38.6 | 25.9 | 36.8 | 27.3 | 25.4 | 32.0 | 30.8 | 38.7 | 24.4 | 26.4 | 31.5 | 26.9 | 32.5 | 31.5
DSS [45] | 39.7 | 36.0 | 37.2 | 26.3 | 35.6 | 27.5 | 25.1 | 31.4 | 30.0 | 37.8 | 24.2 | 26.0 | 30.0 | 26.3 | 31.1 | 30.9
SATA [4] | 36.5 | 33.1 | 35.1 | 25.9 | 34.9 | 27.7 | 25.4 | 29.5 | 29.9 | 33.1 | 23.6 | 26.7 | 31.9 | 27.5 | 35.2 | 30.3
Ours | 38.1 | 34.8 | 36.4 | 27.1 | 34.3 | 27.7 | 26.1 | 28.7 | 28.5 | 30.9 | 24.1 | 26.2 | 28.2 | 26.2 | 31.2 | 29.9

Experiments on ImageNet-C. The last experiment is conducted on ImageNet-C to
further demonstrate the effectiveness of the proposed method. The experimental
results can be seen in Table 3. Compared with the SOTA methods, the proposed
method achieve the lowest average error rate. Noteworthily, the proposed
method outperforms SATA by a large margin for the Shot(71.6% vs. 72.9%),
Impul.(68.7% vs. 71.6%), Defoc.(74.0% vs. 75.7%) and Glass.(71.6% vs. 74.1%)
corruptions.

Table 3: Classification error rate (%) for standard ImageNet-C continual test-
time adaptation task.

Method | Gauss. | Shot | Impul. | Defoc. | Glass | Motion | Zoom | Snow | Frost | Fog | Brit. | Contr. | Elas. | Pix. | Jpeg | Mean
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Source | 95.3 | 95.0 | 95.3 | 86.1 | 91.9 | 87.4 | 77.9 | 85.1 | 79.9 | 79.0 | 45.4 | 96.2 | 86.6 | 77.5 | 66.1 | 83.0
BN [21, 36] | 87.7 | 87.4 | 87.8 | 88.0 | 87.7 | 78.3 | 63.9 | 67.4 | 70.3 | 54.7 | 36.4 | 88.7 | 58.0 | 56.6 | 67.0 | 72.0
TENT [43] | 81.6 | 74.6 | 72.7 | 77.6 | 73.8 | 65.5 | 55.3 | 61.6 | 63.0 | 51.7 | 38.2 | 72.1 | 50.8 | 47.4 | 53.3 | 62.6
CoTTA [44] | 84.7 | 82.1 | 80.6 | 81.3 | 79.0 | 68.6 | 57.5 | 60.3 | 60.5 | 48.3 | 36.6 | 66.1 | 47.2 | 41.2 | 46.0 | 62.7
RoTTA [52] | 88.3 | 82.8 | 82.1 | 91.3 | 83.7 | 72.9 | 59.4 | 66.2 | 64.3 | 53.3 | 35.6 | 74.5 | 54.3 | 48.2 | 52.6 | 67.3
RMT [8] | 79.9 | 76.3 | 73.1 | 75.7 | 72.9 | 64.7 | 56.8 | 56.4 | 58.3 | 49.0 | 40.6 | 58.2 | 47.8 | 43.7 | 44.8 | 59.9
PETAL [3] | 87.4 | 85.8 | 84.4 | 85.0 | 83.9 | 74.4 | 63.1 | 63.5 | 64.0 | 52.4 | 40.0 | 74.0 | 51.7 | 45.2 | 51.0 | 67.1
DSS [45] | 84.6 | 80.4 | 78.7 | 83.9 | 79.8 | 74.9 | 62.9 | 62.8 | 62.9 | 49.7 | 37.4 | 71.0 | 49.5 | 42.9 | 48.2 | 64.6
ViDA [22] | 79.3 | 74.7 | 73.1 | 76.9 | 74.5 | 65.0 | 56.4 | 59.8 | 62.6 | 49.6 | 38.2 | 66.8 | 49.6 | 43.1 | 46.2 | 61.2
SATA [4] | 74.1 | 72.9 | 71.6 | 75.7 | 74.1 | 64.2 | 55.5 | 55.6 | 62.9 | 46.6 | 36.1 | 69.9 | 50.6 | 44.3 | 48.5 | 60.1
Ours | 75.1 | 71.6 | 68.7 | 74.0 | 71.6 | 65.1 | 56.4 | 55.7 | 61.0 | 49.3 | 41.3 | 61.9 | 49.1 | 44.8 | 46.0 | 59.4

Table 4: Ablation study on class and domain under severity 5.

No. | CCS | CDS | CIFAR10-C | CIFAR100-C | ImageNet-C
---|---|---|---|---|---
1 |  |  | 15.66 | 31.19 | 60.56
2 | ✓ |  | 14.94 | 30.42 | 59.67
3 |  | ✓ | 15.16 | 30.68 | 59.75
4 | ✓ | ✓ | 14.71 | 29.92 | 59.43

###  4.3 Ablation Studies

We perform ablation study experiments to evaluate the effectiveness of major
components of C-CoTTA on three benchmarks. For simplicity, we denote the
Control Class Shift as CCS and Control Domain Shift as CDS. As shown in Table
4, C-CoTTA decrease error rates in all benchmarks after adding CCS or CDS to
the network, which indicate mitigating inter-category interference or reducing
the model’s sensitivity to overall domain shifts can improve the
classification accuracy. Furthermore, the combination of CCS and CDS can
further improve the classification accuracy of the C-CoTTA framework.

###  4.4 t-SNE Visualization for Class Shift

We use t-SNE [42] for dimensionality reduction to visualize domain shift
situations of different methods during test-time. As shown in Figure 3, it can
be observed that directly using pre-trained model(Source), CoTTA, and SATA
methods do not explicitly control domain shift, resulting in unpredictable
domain shift directions, leading to blurry classification boundaries and
mutual interference between categories. In contrast, our method implements
controllable domain shift, so it can be seen that although categories also
experience shift, the direction of the shift is benign and does not cause
mutual interference between categories.

Figure 3: Visualization of the t-SNE dimensionality reduction of three classes
from CIFAR10-C dataset (three easily misclassified animals: bird, deer, frog)
transferred from the source domain to the target domain (zoom)

###  4.5 Inter-Class Distance and Inter-Domain Distance

Inter-Class Distance. The inter-class distance can be formulate as
$d_{ic}=\sum_{i=1}^{c}\sum_{j\neq
i}^{c}\left\|\mathbf{p}_{i}^{t}-\mathbf{p}_{j}^{t}\right\|_{2}^{2}$, where
$\mathbf{p}_{i}^{t}$ and $\mathbf{p}_{j}^{t}$ denote target domain category
prototype. The inter-class distance comparison between our method, CoTTA and
SATA can be seen in Figure 4(a). Compared to CoTTA and SATA, our method has a
larger inter-class distance, indicating better separability between classes
and reflecting the effectiveness of controlling class shift direction.

Inter-Domain Distance. The inter-domain distance can be computed as
$d_{id}=\left\|\mathbf{p}^{s}-\mathbf{p}^{t}\right\|_{2}^{2}$, where
$\mathbf{p}^{s}$ denotes overall prototype of the source domain while
$\mathbf{p}^{t}$ denotes overall prototype of the target domain. As shown in
the Figure 4(b), compared to SATA, our method has smaller inter-domain
distances, indicating that the model is less sensitive to domain
transformations and reflecting the effectiveness of controlling domain shift.
At the same time, although the CoTTA method has relatively small inter-domain
distances in the early stage, as the target domain changes at test time, its
inter-domain distances gradually increase. This indicates that the model is
sensitive to domain shift, further emphasizing the importance of reducing the
sensitivity of the model to domain shift in controlling overall domain shift.

Figure 4: (a) Inter-class distance: can indicate the separability between
classes. (b) Inter-domain distance: can indicate the sensitivity of the model
to domain transformations.

###  4.6 Results for Gradual Test-Time Adaptation

In the standard setup described above, corruption types change abruptly at the
highest severity level. We will now present the results of the gradual setup.
We design the sequence by gradually changing the severity for the 15 types of
corruption. When the type of corruption changes, the severity level is at its
lowest. The distribution shift within each type is also gradual. Table 4.6
shows that the proposed method outperforms competing methods.

Table 5: Results for Gradual Adaptation

Methods | CIFAR10-C | CIFAR100-C | ImageNet-C
---|---|---|---
Source | 23.9 | 32.9 | 81.7
TENT | 39.1 | 72.7 | 53.7
CoTTA | 10.6 | 26.3 | 42.1
SATA | 10.8 | 27.5 | 44.8
Ours | 9.21 | 26.2 | 41.1

Table 6: Results for Corruption loops Adaptation

Methods | CIFAR10-C | CIFAR100-C | ImageNet-C
---|---|---|---
Source | 43.5 | 46.4 | 83.0
TENT | 41.8 | 31.2 | 65.3
CoTTA | 15.7 | 32.4 | 68.2
SATA | 15.5 | 32.2 | 62.5
Ours | 11.6 | 27.2 | 52.6

###  4.7 Results for Corruption Loops Test-Time Adaptation

In real-world scenarios, test domains may occur in cycles. To assess the long-
term adaptation performance of the method, we evaluated the test conditions
for 10 consecutive cycles. This means that at level 5 severity, the test data
will be reanalyzed and readjusted nine times. The complete result can be found
in Table 4.6. The results also show that our method outperforms the other
methods in this long-term adaptation scenario. It illustrates the
effectiveness of domain offset controllability as well as category offset
controllability.

###  4.8 Results for Random Order Test-Time Adaptation

For a more comprehensive evaluation of the proposed method, CIFAR10-C,
CIFAR100-C, and ImageNet-C experiments are conducted on over ten sequences of
various damage types with a severity level of 5. As shown in the Table 7,
C-CoTTA is consistently outperforming CoTTA and other competing methods.

Table 7: Average error of standard ImageNet-C experiments over 10 diverse
corruption sequences.

Avg. Error (%) | Source | TENT | CoTTA | SATA | Ours
---|---|---|---|---|---
CIFAR10-C | 43.5 | 20.1 | 16.3 | 16.3 | 14.7
CIFAR100-C | 46.4 | 61.3 | 32.6 | 32.8 | 29.5
ImageNet-C | 83.0 | 61.8 | 57.9 | 64.5 | 55.5

##  5 Conclusion and Limitation

In this work, we introduce C-CoTTA, a novel framework designed to prevent any
category from leaning towards other categories by explicitly controlling the
offset direction to avoid fuzzy classification boundaries, and reduce the
sensitivity of the model in the domain shift direction to reduce the impact of
domain shift on domain adaptation. This fills the gap left by traditional
methods, which can only mitigate the impact of domain drift. C-CoTTA can
explicitly control domain shift, opening up a new solution pathway for CTTA.
Through extensive quantitative experiments and qualitative analysis, such as
t-SNE plots, we demonstrate the effectiveness and theoretical validity of
C-CoTTA.

However, our method also has certain limitations. During test-time, we may not
have access to all samples of a specific domain prototype, and due to the lack
of true labels, misclassified samples may contaminate the prototype. As a
result, the representation of the prototype may be poor, further affecting the
accuracy of the constructed direction, leading to ineffective or even
erroneous domain shift control. Therefore, in the future, addressing how to
obtain high-quality prototypes and directional representations is a task that
needs attention.

## References

  * Anders et al. [2022] Christopher J Anders, Leander Weber, David Neumann, Wojciech Samek, Klaus-Robert Müller, and Sebastian Lapuschkin.  Finding and removing clever hans: Using explanation methods to debug and improve deep models.  _Information Fusion_ , 77:261–295, 2022.
  * Biecek and Samek [2024] Przemyslaw Biecek and Wojciech Samek.  Explain to question not to justify.  _arXiv preprint arXiv:2402.13914_ , 2024.
  * Brahma and Rai [2023] Dhanajit Brahma and Piyush Rai.  A probabilistic framework for lifelong test-time adaptation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3582–3591, 2023.
  * Chakrabarty et al. [2023] Goirik Chakrabarty, Manogna Sreenivas, and Soma Biswas.  Sata: Source anchoring and target alignment network for continual test time adaptation.  _arXiv preprint arXiv:2304.10113_ , 2023.
  * Chen [2020] Xingxin Chen.  Domain adaptation for autonomous driving.  Master’s thesis, University of Waterloo, 2020.
  * Chen et al. [2023] Ziyang Chen, Yiwen Ye, Mengkang Lu, Yongsheng Pan, and Yong Xia.  Each test image deserves a specific prompt: Continual test-time adaptation for 2d medical image segmentation.  _arXiv preprint arXiv:2311.18363_ , 2023.
  * Cortes and Vapnik [1995] Corinna Cortes and Vladimir Vapnik.  Support-vector networks.  _Machine learning_ , 20:273–297, 1995.
  * Döbler et al. [2023] Mario Döbler, Robert A Marsden, and Bin Yang.  Robust mean teacher for continual and gradual test-time adaptation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7704–7714, 2023.
  * Dreyer et al. [2023] Maximilian Dreyer, Frederik Pahde, Christopher J Anders, Wojciech Samek, and Sebastian Lapuschkin.  From hope to safety: Unlearning biases of deep models by enforcing the right reasons in latent space.  _arXiv preprint arXiv:2308.09437_ , 2023.
  * Dreyer et al. [2024] Maximilian Dreyer, Frederik Pahde, Christopher J Anders, Wojciech Samek, and Sebastian Lapuschkin.  From hope to safety: Unlearning biases of deep models via gradient penalization in latent space.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 38, pages 21046–21054, 2024.
  * Du et al. [2023] Kaile Du, Fan Lyu, Linyan Li, Fuyuan Hu, Wei Feng, Fenglei Xu, Xuefeng Xi, and Hanjing Cheng.  Multi-label continual learning using augmented graph convolutional network.  _IEEE Transactions on Multimedia_ , 2023.
  * Gan et al. [2023] Yulu Gan, Yan Bai, Yihang Lou, Xianzheng Ma, Renrui Zhang, Nian Shi, and Lin Luo.  Decorate the newcomers: Visual domain prompt for continual test time adaptation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, pages 7595–7603, 2023.
  * Geiger et al. [2013] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  _The International Journal of Robotics Research_ , 32(11):1231–1237, 2013.
  * Gonzalez et al. [2020] Camila Gonzalez, Nick Lemke, Georgios Sakas, and Anirban Mukhopadhyay.  What is wrong with continual learning in medical image segmentation?  _arXiv preprint arXiv:2010.11008_ , 2020.
  * Haufe et al. [2014] Stefan Haufe, Frank Meinecke, Kai Görgen, Sven Dähne, John-Dylan Haynes, Benjamin Blankertz, and Felix Bießmann.  On the interpretation of weight vectors of linear models in multivariate neuroimaging.  _Neuroimage_ , 87:96–110, 2014.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the Computer Vision and Pattern Recognition_ , 2016\.
  * Hu et al. [2022] Chuqing Hu, Sinclair Hudson, Martin Ethier, Mohammad Al-Sharman, Derek Rayside, and William Melek.  Sim-to-real domain adaptation for lane detection and classification in autonomous driving.  In _IEEE Intelligent Vehicles Symposium (IV)_ , pages 457–463. IEEE, 2022.
  * Jain and Learned-Miller [2011] Vidit Jain and Erik Learned-Miller.  Online domain adaptation of a pre-trained cascade of classifiers.  In _CVPR_ , pages 577–584. IEEE, 2011.
  * Kim et al. [2018] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al.  Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav).  In _International conference on machine learning_ , pages 2668–2677. PMLR, 2018.
  * Kondo [2022] Satoshi Kondo.  Source-free unsupervised domain adaptation with norm and shape constraints for medical image segmentation.  _arXiv preprint arXiv:2209.01300_ , 2022.
  * Li and Hoiem [2017] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  _IEEE transactions on pattern analysis and machine intelligence_ , 40(12):2935–2947, 2017.
  * Liu et al. [2023] Jiaming Liu, Senqiao Yang, Peidong Jia, Ming Lu, Yandong Guo, Wei Xue, and Shanghang Zhang.  Vida: Homeostatic visual domain adapter for continual test time adaptation.  _arXiv preprint arXiv:2306.04344_ , 2023.
  * Lyu et al. [2021] Fan Lyu, Shuai Wang, Wei Feng, Zihan Ye, Fuyuan Hu, and Song Wang.  Multi-domain multi-task rehearsal for lifelong learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 8819–8827, 2021.
  * Lyu et al. [2024a] Fan Lyu, Kaile Du, Yuyang Li, Hanyu Zhao, Zhang Zhang, Guangcan Liu, and Liang Wang.  Variational continual test-time adaptation.  _arXiv preprint arXiv:2402.08182_ , 2024a.
  * Lyu et al. [2024b] Fan Lyu, Wei Feng, Yuepan Li, Qing Sun, Fanhua Shang, Liang Wan, and Liang Wang.  Elastic multi-gradient descent for parallel continual learning.  _arXiv preprint arXiv:2401.01054_ , 2024b.
  * McGrath et al. [2022] Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik.  Acquisition of chess knowledge in alphazero.  _Proceedings of the National Academy of Sciences_ , 119(47):e2206625119, 2022.
  * Ni et al. [2023] Jiayi Ni, Senqiao Yang, Jiaming Liu, Xiaoqi Li, Wenyu Jiao, Ran Xu, Zehui Chen, Yi Liu, and Shanghang Zhang.  Distribution-aware continual test time adaptation for semantic segmentation.  _arXiv preprint arXiv:2309.13604_ , 2023.
  * Niloy et al. [2024] Fahim Faisal Niloy, Sk Miraj Ahmed, Dripta S Raychaudhuri, Samet Oymak, and Amit K Roy-Chowdhury.  Effective restoration of source knowledge in continual test time adaptation.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 2091–2100, 2024.
  * Niu et al. [2022] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan.  Efficient test-time model adaptation without forgetting, 2022.
  * O’Kelly [2021] Matthew O’Kelly.  _Accelerated Risk Assessment and Domain Adaptation for Autonomous Vehicles_.  PhD thesis, University of Pennsylvania, 2021.
  * Pahde et al. [2022] Frederik Pahde, Leander Weber, Christopher J Anders, Wojciech Samek, and Sebastian Lapuschkin.  Patclarc: Using pattern concept activation vectors for noise-robust model debugging.  _arXiv preprint arXiv:2202.03482_ , 2022.
  * Pahde et al. [2023] Frederik Pahde, Maximilian Dreyer, Wojciech Samek, and Sebastian Lapuschkin.  Reveal to revise: An explainable ai life cycle for iterative bias correction of deep models.  In _Medical Image Computing and Computer Assisted Intervention_ , 2023\.
  * Pfau et al. [2021] Jacob Pfau, Albert T Young, Jerome Wei, Maria L Wei, and Michael J Keiser.  Robust semantic interpretability: Revisiting concept activation vectors.  _arXiv preprint arXiv:2104.02768_ , 2021.
  * Rahman et al. [2021] Mohammad Mahfujur Rahman, Clinton Fookes, and Sridha Sridharan.  Deep domain generalization with feature-norm network.  _arXiv preprint arXiv:2104.13581_ , 2021.
  * Samek [2023] Wojciech Samek.  Explainable deep learning: concepts, methods, and new developments.  In _Explainable Deep Learning AI_ , pages 7–33. Elsevier, 2023.
  * Schneider et al. [2020] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge.  Improving robustness against common corruptions by covariate shift adaptation.  In _Proceedings of the Advances in Neural Information Processing Systems_ , 2020.
  * Sohn et al. [2020] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  _Advances in neural information processing systems_ , 33:596–608, 2020.
  * Sun et al. [2022] Qing Sun, Fan Lyu, Fanhua Shang, Wei Feng, and Liang Wan.  Exploring example influence in continual learning.  In _NeurIPS 2022 (Oral)_ , 2022.
  * Sun et al. [2020] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt.  Test-time training with self-supervision for generalization under distribution shifts.  In _International conference on machine learning_ , pages 9229–9248. PMLR, 2020.
  * Tarvainen and Valpola [2017] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  _Advances in neural information processing systems_ , 30, 2017.
  * Van de Ven and Tolias [2019] Gido M Van de Ven and Andreas S Tolias.  Three scenarios for continual learning.  _arXiv preprint arXiv:1904.07734_ , 2019.
  * Van der Maaten and Hinton [2008] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _Journal of machine learning research_ , 9(11), 2008.
  * Wang et al. [2020] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell.  Tent: Fully test-time adaptation by entropy minimization.  _arXiv preprint arXiv:2006.10726_ , 2020.
  * Wang et al. [2022] Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai.  Continual test-time domain adaptation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7201–7211, 2022.
  * Wang et al. [2024] Yanshuo Wang, Jie Hong, Ali Cheraghian, Shafin Rahman, David Ahmedt-Aristizabal, Lars Petersson, and Mehrtash Harandi.  Continual test-time domain adaptation via dynamic sample selection.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 1701–1710, 2024.
  * Wang et al. [2019] Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey.  Symmetric cross entropy for robust learning with noisy labels.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 322–330, 2019.
  * Weber et al. [2023] Leander Weber, Sebastian Lapuschkin, Alexander Binder, and Wojciech Samek.  Beyond explaining: Opportunities and challenges of xai-based model improvement.  _Information Fusion_ , 92:154–176, 2023.
  * Xie et al. [2020] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le.  Self-training with noisy student improves imagenet classification.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10687–10698, 2020.
  * Xie et al. [2017] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In _Proceedings of the Computer Vision and Pattern Recognition_ , 2017\.
  * Xu et al. [2019] Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin.  Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 1426–1435, 2019.
  * Yang et al. [2023] Xu Yang, Yanan Gu, Kun Wei, and Cheng Deng.  Exploring safety supervision for continual test-time domain adaptation.  In _Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence_ , pages 1649–1657, 2023.
  * Yuan et al. [2023] Longhui Yuan, Binhui Xie, and Shuang Li.  Robust test-time adaptation in dynamic scenarios.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15922–15932, 2023.
  * Yuksekgonul et al. [2022] Mert Yuksekgonul, Maggie Wang, and James Zou.  Post-hoc concept bottleneck models.  _arXiv preprint arXiv:2205.15480_ , 2022.
  * Zagoruyko and Komodakis [2016] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  In _Procedings of the British Machine Vision Conference_ , 2016.

##  Appendix A Equivalent Representation of SCAV

Further derivation of scav is conducted to obtain a simpler equivalent
representation, as follows:

$\displaystyle\text{cov}[f(x),y]$
$\displaystyle=\sum_{x_{i}\in\mathcal{X}_{c}\cup\mathcal{X}_{n}}(f(x_{i})-\overline{f}(x))(y_{i}-\overline{y})$
$\displaystyle=\frac{|\mathcal{X}_{n}|}{|\mathcal{X}_{c}|+|\mathcal{X}_{n}|}\sum_{x_{i}\in\mathcal{X}_{c}}(f(x_{i})-\overline{f}(x))+\frac{|\mathcal{X}_{c}|}{|\mathcal{X}_{c}|+|\mathcal{X}_{n}|}\sum_{x_{i}\in\mathcal{X}_{n}}(f(x_{i})-\overline{f}(x))$
$\displaystyle=\frac{|\mathcal{X}_{n}|}{|\mathcal{X}_{c}|+|\mathcal{X}_{n}|}\left(\sum_{x_{i}\in\mathcal{X}_{c}}f(x_{i})-|\mathcal{X}_{c}|\cdot\overline{f}(x_{i})\right)-\frac{|\mathcal{X}_{c}|}{|\mathcal{X}_{c}|+|\mathcal{X}_{n}|}\left(\sum_{x_{i}\in\mathcal{X}_{n}}f(x_{i})-|\mathcal{X}_{n}|\overline{f}(x_{i})\right)$
$\displaystyle=\frac{|\mathcal{X}_{n}|}{|\mathcal{X}_{c}|+|\mathcal{X}_{n}|}\sum_{x_{i}\in\mathcal{X}_{c}}f(x_{i})-\frac{|\mathcal{X}_{n}||\mathcal{X}_{c}|}{|\mathcal{X}_{c}|+|\mathcal{X}_{n}|}\overline{f}(x_{i})-\frac{|\mathcal{X}_{c}|}{|\mathcal{X}_{c}|+|\mathcal{X}_{n}|}\sum_{x_{i}\in\mathcal{X}_{n}}f(x_{i})+\frac{|\mathcal{X}_{c}||\mathcal{X}_{n}|}{|\mathcal{X}_{c}|+|\mathcal{X}_{n}|}\overline{f}(x_{i})$
$\displaystyle=\frac{|\mathcal{X}_{n}||\mathcal{X}_{c}|}{|\mathcal{X}_{c}|+|\mathcal{X}_{n}|}\left(\frac{1}{|\mathcal{X}_{c}|}{\sum_{x_{i}\in\mathcal{X}_{c}}f(x_{i})}-\frac{1}{|\mathcal{X}_{n}|}{\sum_{x_{i}\in\mathcal{X}_{n}}f(x_{i})}\right)$
$\displaystyle\text{cov}[y,y]$
$\displaystyle=\sum_{x_{i}\in\mathcal{X}_{c}\cup\mathcal{X}_{n}}(y_{i}-\overline{y})^{2}$
$\displaystyle=\sum_{x_{i}\in\mathcal{X}_{c}}(1-\frac{|\mathcal{X}_{c}|}{|\mathcal{X}_{c}|+|\mathcal{X}_{n}|})^{2}+\sum_{x_{i}\in\mathcal{X}_{n}}(0-\frac{|\mathcal{X}_{c}|}{|\mathcal{X}_{c}|+|\mathcal{X}_{n}|})^{2}$
$\displaystyle=\sum_{x_{i}\in\mathcal{X}_{c}}\frac{|\mathcal{X}_{n}|^{2}}{(|\mathcal{X}_{c}|+|\mathcal{X}_{n}|)^{2}}+\sum_{x_{i}\in\mathcal{X}_{n}}\frac{|\mathcal{X}_{c}|^{2}}{(|\mathcal{X}_{c}|+|\mathcal{X}_{n}|)^{2}}$
$\displaystyle=\frac{|\mathcal{X}_{c}||\mathcal{X}_{n}|^{2}+|\mathcal{X}_{n}||\mathcal{X}_{c}|^{2}}{(|\mathcal{X}_{c}+|\mathcal{X}_{n}|)^{2}}$
$\displaystyle=\frac{|\mathcal{X}_{c}||\mathcal{X}_{n}|}{|\mathcal{X}_{c}|+|X_{n}|}$
$\displaystyle\frac{\text{cov}[f(x),y]}{\text{cov}[y,y]}$
$\displaystyle=\frac{1}{|\mathcal{X}_{c}|}{\sum_{x_{i}\in\mathcal{X}_{c}}f(x_{i})}-\frac{1}{|\mathcal{X}_{n}|}{\sum_{x_{i}\in\mathcal{X}_{n}}f(x_{i})}$

##  Appendix B Corruption loops Test-Time Adaptation

We present the results of 10 cycles on CIFAR10-C. As depicted in the Fig. 5,
it is evident that over time, the error rate of CoTTA and SATA methods has
gradually increased, whereas our method continues to decrease. Consequently,
the performance gap is widening. This phenomenon is most likely related to the
fact that the first two methods lack reasonable control over the category and
the offset direction of the domain.

Figure 5: Under the CIFAR10-C dataset, it can be observed that the performance
of each method under 10 corruption cycles varies. The error rates of CoTTA and
SATA methods start to gradually increase in the later stages, whereas our
method continues to decrease or maintain stability.

##  Appendix C Hyperparameter analysis

In this section, we delve into the critical examination of hyperparameters
$\lambda_{1}$ and $\lambda_{2}$ in Eq. 11 within our C-CoTTA framework in
ImageNet-C, which substantially influence the model’s performance. Through
meticulous experimentation, we fine-tuned these hyperparameters to identify
their optimal values, ensuring the harmonious interplay between Control Domain
Shift (CDS) and Control Class Shift (CCS) components.

Our investigation revealed that the selection of $\lambda_{1}$ and
$\lambda_{2}$ is pivotal in balancing the contributions of CDS and CCS to the
overall objective function. We experimented with a spectrum of values for
these hyperparameters, meticulously recording the impact on classification
accuracy and domain adaptation efficacy. The empirical results, illustrated in
Fig. C, present a compelling case for the optimal balance that our chosen
hyperparameters provide, underscoring the model’s robustness against various
disturbances.

Figure 6: Analysis Hyperparameter of $\lambda_{1}$ for CDS and $\lambda_{2}$
for CCS on ImageNet-C. Figure 7: Analysis Hyperparameter of $E_{0}$ on
CIFAR10-C.

##  Appendix D Reliable Sample Selection Analysis

In our control category offset method, in order to reduce the contamination of
category prototypes by misclassified samples and thus affect the control of
category shift direction, we remove samples with entropy values exceeding the
predefined threshold $E_{0}$, which is set as $0.4\times\ln C$ based on [29].
We have verified the rationality of this operation through experiments. As
shown in Fig. C, when the threshold is large, the effectiveness of the CCS
method deteriorates. This may be because the conditions are too loose, leading
to a large number of misclassified samples when calculating the prototype. On
the other hand, when the threshold is small, the CCS method also deteriorates.
This may be because the conditions are too strict, resulting in too few
samples used to calculate the prototype, making the generated prototype not
representative.

##  Appendix E Class Confusion Matrix

We observed the confusion matrix of category in the domain adaptation process.
As shown in Fig. 8, compared to the CoTTA and SATA methods, our method
significantly reduced the degree of category confusion, demonstrating the
effectiveness of controllable domain shift.

Figure 8: confusion matrix of category in the domain adaptation process. The
vertical axis represents the true labels, and the horizontal axis represents
the predicted labels.


# Japanese Lexical Complexity for Non-Native Readers: A New Dataset

Yusuke Ide1 Masato Mita2 Adam Nohejl1 Hiroki Ouchi1,3 Taro Watanabe1
1Nara Institute of Science and Technology 2CyberAgent Inc. 3RIKEN
{ide.yusuke.ja6, nohejl.adam.mt3, hiroki.ouchi<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

Lexical complexity prediction (LCP) is the task of predicting the complexity
of words in a text on a continuous scale. It plays a vital role in simplifying
or annotating complex words to assist readers. To study lexical complexity in
Japanese, we construct the first Japanese LCP dataset. Our dataset provides
separate complexity scores for Chinese/Korean annotators and others to address
the readers’ L1-specific needs. In the baseline experiment, we demonstrate the
effectiveness of a BERT-based system for Japanese LCP.

##  1 Introduction

Reading comprehension requires a certain level of vocabulary knowledge. The
results reported by Hu and Nation (2000) suggest that most English learners
need to understand 98% of tokens in a text to comprehend it. A follow-up study
by Komori et al. (2004) estimates the percentage to be 96% for Japanese
learners to comprehend text. Acquiring vocabulary to reach such levels, in
turn, is a lengthy and challenging task for learners. This opens up
opportunities for assistive applications, such as simplification or annotation
of complex words. The first step necessary for such applications is to predict
the complexity of the words. The task of lexical complexity prediction (LCP)
is defined as predicting how difficult to comprehend words or phrases in a
text are on a continuous scale (Shardlow et al., 2020). This differentiates
LCP from complex word identification (CWI), i.e., binary classification of
complex words (Yimam et al., 2018). As complexity is naturally perceived as
continuous, a continuous scale used in LCP allows to represent it without loss
of information.

The LCP research so far has been limited to English, for which two LCP
datasets have been constructed (Shardlow et al., 2020, 2022), and no such
dataset has been created for Japanese. Meanwhile, there are a number of
features specific to the Japanese language that could affect lexical
complexity, and their effects have yet to be studied. For example, the Chinese
characters, which are used extensively in Japanese, lower text readability
(Tateisi et al., 1988).

Previous studies on Japanese lexical complexity used pedagogical word lists to
estimate complexity level. Nishihara and Kajiwara (2020) modeled lexical
complexity of words based on the Japanese Educational Vocabulary List
(Sunakawa et al., 2012). The word list assigns a degree of difficulty to each
item, based on the subjective judgment of Japanese language teachers, not
learners themselves, and does not consider the learners’ L1 background.

In light of this, we present JaLeCoN111 JaLeCoN is available at
https://github.com/naist-nlp/jalecon. , Dataset of Japanese Lexical Complexity
for Non-Native Readers. Our dataset has the following key features:

  1. (1)

Complexity scores for single words as well as multi-word expressions (MWEs);

  2. (2)

Separate complexity scores from Chinese/Korean annotators and others,
addressing the considerable advantage of the former in Japanese reading
comprehension.

Our analysis reveals that the non-Chinese/Korean annotators perceive words of
Chinese origin or containing Chinese characters as especially complex. In the
baseline experiment, we investigate the effectiveness of a BERT-based system
in the Japanese LCP task, and how it varies according to the word complexity
and L1 background.

##  2 Task Setting

Since Japanese has no explicit word boundaries, word segmentation is the first
prerequisite for LCP. We use short unit words (SUWs) as the basic word unit,
combining them into longer word units in the case of multi-word expressions
(MWEs):

SUWs | 右肩 | 上がり | に | 増え | て | いる
---|---|---|---|---|---|---
right.shoulder | rise | ADV | increase | GER | be-PRS
Words |  右肩上がり MWE steady.rise |  に SUW ADVj |  増え SUW increasej |  ている MWE PRG-PRSj
| “is steadily increasing”
Figure 1: Example of text segmented as SUWs and as words (either SUW or MWE).
Semantically opaque sequences are chunked into MWEs. Abbreviations in glosses:
ADVerbializer, GERund, PReSent, PRoGressive.

SUW:

SUWs consist of one or two smallest lexical units (Ogura et al., 2011), and
are commonly used for segmentation of Japanese.

MWE:

We understand MWEs as multi-SUW expressions that are semantically opaque or
institutionalized (see Appendix C) and consequently may have higher complexity
than their components. We identify MWEs either using long unit word (LUW)222
The LUW is defined as a syntactic word by Omura et al. (2021).  segmentation,
or manually (see Section 3).

Consequently, a word, can be either an SUW or an MWE (see Figure 1 for
examples).

A complexity score represents perceived complexity based on the annotators’
judgment on a scale from 0 (least complex) to 1 (most complex). We exclude
proper nouns from our target because their complexity is influenced by factors
unrelated to reading proficiency or vocabulary knowledge.333 Sequences
containing segmentation errors are also excluded (see Appendix D).

We annotate the words in an in-context dense setting. In-context here means
including both intra-sentence and extra-sentence context of each word. Context
is important for lexical complexity for two reasons (Gooding and Kochmar,
2019; Shardlow et al., 2021): (1) As polysemous words can have different
complexity levels for each sense, context is necessary to differentiate
between possible meanings of these words. (2) Presenting a word without
context could increase its complexity. In particular, the recognition of
abstract words relies on context (Schwanenflugel et al., 1988). Dense means
annotating each word of the text with a complexity label, instead of
annotating one specific word in each sentence (Shardlow et al., 2022). We
adopt the dense setting to avoid any bias that could arise from targeting
specific words.

##  3 Construction of JaLeCoN

|  |  |  | CK | Non-CK
---|---|---|---|---|---
Genre | Sentences | Words | MWE Ratio | All Words | MWEs | All Words | MWEs
News | 400 | 10,256 |  07.9% | .009 | .020 | .024 | .072
Government | 200 |  07,964 | 14.4% | .005 | .009 | .028 | .047
Table 1: Statistics of JaLeCoN. The CK and Non-CK columns show the mean
complexity scores by L1 group.

In order to include both written and spoken language and a variety of
vocabulary, we sourced texts from two different genres:

News

comes from the Japanese-English data of the WMT22 General Machine Translation
Task (Kocmi et al., 2022). It contains a variety of news texts written for the
general Japanese reader.

Government

is composed of press conference transcripts from Japanese ministries or
agencies.444 The transcripts were retrieved from the websites of five
organizations: JMA, JTA, MOJ, MOFA, and MLHW.

The whole dataset is composed of sequences of sentences constituting either
the beginning of an article (News) or a question-answer pair (Government). We
restricted the length of the sequences to at least 6 and at most 11 sentences
to obtain similar amounts of text, and presented each sequence as a whole for
annotation.

###  3.1 Word Segmentation

We used Comainu 0.80555https://github.com/skozawa/Comainu (Kozawa et al.,
2014) to perform two-level segmentation. The low-level SUW segmentation was
done using MeCab (Kudo et al., 2004), a Japanese morphological analyzer, and
the UniDic 2.3.0 (Den et al., 2007) dictionary. At the second level, Comainu
chunked the SUWs into LUWs. Based on the two segmentations, we segmented the
text into words as follows:

  1. (1)

If an LUW is a noun, we use the constituting SUWs as words. Transparent noun
compounds are ubiquitous in Japanese (e.g., 次期 $\big{|}$ 気象 $\big{|}$ 衛星666The
vertical bars denote boundaries between SUWs.  “next meteorological
satellite”), and we do not consider them MWEs.

  2. (2)

If an LUW is not a noun, we use the LUW as a word. Such an LUW may be a single
SUW, or a sequence of SUWs, which we consider an MWE. Such MWEs most
importantly include functional words, such as compound particles (e.g., に
$\big{|}$ つい $\big{|}$ て “about”) and auxiliary verbs (e.g., なけれ $\big{|}$ ば
$\big{|}$ なら $\big{|}$ ない “have to”).

We also identified other MWEs manually, as explained in Section 3.3.

###  3.2 Complexity Annotation

To capture the lexical complexity for a non-native Japanese reader with
intermediate or advanced reading ability, we recruited 15 annotators per
sentence with Japanese reading proficiency ranging from CEFR (Common European
Framework of Reference for Languages) level B1 to C2. We required at least
intermediate proficiency, as it has been shown that complexity judgments made
by intermediate or advanced learners can be used to adequately predict the
needs of beginners but not vice versa (Gooding et al., 2021). The proficiency
levels were self-reported (see Appendix A for details). We used the
annotations made by 14 of them, after removing one outlier, whose annotations
had over 70% higher mean than those of any other annotator, clearly not
corresponding to the reported reading proficiency.

Approximately half of the annotators we recruited have a Chinese/Korean L1
background (CK).777 On average, the CK annotators reported higher Japanese
reading proficiency than the non-CK (see Appendix A).  CK learners have a
considerable advantage in comprehension of words of Chinese origin, which also
form a large part of Chinese and Korean vocabulary (Koda, 1989).

The annotators were asked to assign one of the following labels to each span
if they find it complex: 3 (Very Difficult), 2 (Difficult), or 1 (Not Easy);
otherwise the annotators were to leave the span unlabeled and we interpreted
it as 0 (Easy).888 See Appendix B for detailed definitions of each label.
Annotators could label a span of any length if it was complex as a whole, but
were asked to create as short a span as possible. To calculate the average,
the labels were converted to numerical values as follows: 3 → 1, 2 → 0.67, 1 →
0.33, 0 → 0. The averaging hinges on the assumption that the labels have an
equal distance between them. We always presented the labels together with the
values 0 to 3 to reinforce the perception of equal distance.

###  3.3 MWE Annotation

In parallel with the complexity annotation, we annotated MWEs not identified
by LUW segmentation (see Section 3.1). Given the absence of an MWE detector
for Japanese of sufficient quality, the annotation was performed manually by a
native Japanese speaker and a non-native speaker with a degree in the Japanese
language. The expression categories we consider MWEs are described in Appendix
C.

###  3.4 Complexity Scoring

Using annotations from the previous steps, we assigned complexity scores to
words according to the following rules:

  1. (1)

If a span contains one or more words, each word receives the complexity value
of the span.

  2. (2)

If an MWE (manually annotated according to Section 3.3) overlaps with or
contains multiple spans, the MWE receives the maximum of the complexity values
of the spans.

Finally, for each word, we calculated the complexity score for each L1 group
as the average of the individual values from the annotators in that group.

##  4 Statistics and Analysis

Overall statistics for both genres and L1 groups are shown in Table 1.999 See
Appendix E for the complexity scores and annotation distributions of several
words in the non-CK group.  MWEs have higher mean complexity than single words
for both L1 groups and are more frequent in the Government genre. There is a
tendency towards perceiving higher complexity in the non-CK group, which
corresponds to slightly lower average Japanese proficiency of the non-CK
annotators (see Appendix A).

We measured inter-annotator agreement (IAA) using Krippendorf’s $\alpha$ for
interval values (Krippendorff, 1970). The IAA is 0.32 in the CK group, and
0.31 in the non-CK group, while it would be 0.19 if we merged the groups. As
lexical complexity is highly subjective (Gooding et al., 2021), the low
agreement does not imply low reliability, but it indicates that perception of
complexity is more alike within the L1 groups than across all annotators.

The complexity score distribution in each L1 group is shown in Figure 2. No
words achieved a score greater than 0.81 and 0.86 in the CK and non-CK groups,
respectively, which reflects that words are rarely labeled as Difficult or
Very Difficult by all annotators in a group.

Figure 2: Histogram of complexity scores by L1 group. | Japanese | Chinese | Other
---|---|---|---
| All | CC | All | CC | All | CC
CK | .003 | .009 | .004 | .004 | .071 | .000
Non-CK | .010 | .032 | .062 | .072 | .007 | .143
Frequency | 52% | 10% | 26% | 22% |  04% |  00%
Table 2: Mean complexity (by L1 group) and frequency, according to (1) word
origin: Japanese (wago), Chinese/Sino-Japanese (kango), and Other (gairaigo,
borrowings from languages other than Chinese), and (2) whether the words
contain Chinese characters only (denoted by CC). The origin was classified
using MeCab and Comainu (see Section 3.1), excluding words of mixed or unknown
origin.

In addition to the aforementioned difference in proficiency, there is also a
clear difference in how the two L1 groups perceive complexity of words based
on their origin and whether they contain Chinese characters101010 Japanese
vocabulary consists of words of Japanese origin, Chinese (Sino-Japanese)
origin, and foreign words from other languages (gairaigo). The first two
categories can be written using Chinese characters (kanji), Japanese syllabary
(kana), or a combination thereof, while other foreign words are usually
written in syllabary only. (See Appendix F for examples.), as analyzed in
Table 2. For the CK group, the mean complexity of words of Japanese and
Chinese origin was similar. For the non-CK group, however, words of Chinese
origin were markedly more complex (0.062) than words of Japanese origin
(0.010), and both categories of words were more complex when they contained
Chinese characters.111111 The opposite tendency for gairaigo (foreign words
mostly from English) to be perceived as more complex in the CK group coincides
with lower English proficiency among annotators in this group (see Appendix
A), and therefore should not be explained by their L1 background.

##  5 Experiments

The newly created dataset can be used to evaluate performance of LCP for non-
native Japanese readers of different L1 backgrounds (CK and non-CK). We
developed a baseline system based on a fine-tuned BERT (Devlin et al., 2019)
model, and evaluated it using cross-validation. We fine-tuned a Japanese pre-
trained BERT model released by Tohoku University, namely the base model for
UniDic Lite segmentation121212Available from https://huggingface.co/cl-
tohoku/bert-base-japanese-v2..

For each word $w$ in our dataset and the sentence $s$ that contains it at
token indices $i$ to $j-1$, we construct an input sequence $(\texttt{[CLS]},$
$s_{0}^{i-1},$ $\texttt{<Unused1>},$ $w,$ $\texttt{<Unused2>},$
$s_{j}^{\left|s\right|-1},$ $\texttt{[SEP]},$ $w,$ $\texttt{[SEP]})$. The
target word occurs first delimited by unused tokens (<Unused$n$>) in the
sentence context, and then on its own following the first [SEP] token.131313
Due to a different segmentation (version of UniDic) used by Tohoku BERT and
our dataset, we have to enforce segmentation at the word’s boundaries using
spaces.  To predict the complexity score, we feed the final hidden
representation of the [CLS] token into a linear layer with a single output. A
similar fine-tuning approach, but without the special tokens, was used for
English LCP by Taya et al. (2021), achieving one of the highest $R^{2}$ values
in the single-word subtask of SemEval-2021 Task 1 (Shardlow et al., 2021).

We fine-tune and evaluate models for CK and non-CK complexity separately. See
Appendix G for the hyperparameters and cross-validation scheme.

| MAE by Gold Complexity Score Tier |
---|---|---
| Zero | Easy $>0$ | Not Easy | (Very) Difficult | $R^{2}$
CK | 0.0034 | 0.0676 | 0.1913 | 0.2954 | 0.4351
Non-CK | 0.0066 | 0.0510 | 0.1169 | 0.2932 | 0.6142
Table 3: Results of the fine-tuned BERT model by L1 group (means over 5 cross-validation folds). | Zero | Easy $>0$ | Not Easy | (Very) Difficult
---|---|---|---|---
CK | 17,563 |  0,393 | 223 |  041
Non-CK | 15,209 | 2,067 | 837 | 107
Table 4: Word counts in the whole dataset by L1 group and MAE tier. Figure 3:
Illustrated score ranges of the MAE tiers: $\\{0\\}$ for Zero, $(0,0.165]$ for
Easy $>0$, $(0.165,0.5]$ for Not Easy, and $(0.5,1]$ for (Very) Difficult.

The results are reported in Table 3. In addition to $R^{2}$ (coefficient of
determination)141414 Compared to correlation coefficients, $R^{2}$ is more
appropriate for LCP, since it also captures deviations in mean and variance.
Compared to MAE or MSE, it is easier to interpret, as $R^{2}=0$ corresponds to
the mean regressor, while $R^{2}=1$ corresponds to a perfect model. , we
report mean average error (MAE) by complexity score tiers to draw the full
picture of the models’ performance at different complexity levels. The score
ranges of the tiers are centered at annotation values as illustrated in Figure
3. We handle zero as a special tier, and merge Very Difficult with Difficult
due to a low number of words.

The fine-tuned BERT model for CK and Non-CK achieves $R^{2}$ of 0.4351 and
0.6142, respectively. For both L1 groups, the MAE value increases markedly in
each successive complexity tier, as the number of training examples (shown in
Table 4) diminishes. Similarly, the CK model achieves lower error than non-CK
only in tier Zero, where it has more examples available than the non-CK model.
This suggests that the scarcity of words with complexity above zero is a
factor contributing to worse performance on CK data, as measured by $R^{2}$.

##  6 Conclusion

In this paper, we presented the first dataset for Japanese LCP. It provides
separate complexity scores based on the CK/non-CK distinction of annotators’
L1 background. Our analysis corroborates our conjecture that special
consideration of L1 background is useful for the Japanese LCP task in
particular. We believe it could benefit LCP in other languages as well.

In the baseline experiment, we demonstrated the efficacy of our BERT-based
system for both CK and non-CK readers. Even after separating CK and non-CK
annotators, however, notable inter-annotator disagreement remains within these
groups. Therefore personalized systems analogous to Gooding and Tragut (2022)
could improve on our system. Future research should study this possibility,
analyzing both its costs and benefits.

Models trained on JaLeCoN can be used as part of a lexical simplification
pipeline for Japanese, both to identify complex words and to rank candidate
simplifications. JaLeCoN itself can be further used as a basis for a lexical
simplification dataset targeting words actually perceived as complex, similar
to TSAR-ST datasets for English and Spanish (Štajner et al., 2022).

## Limitations

Our task setting and baseline system requires that the input is already
segmented into words including MWEs. The MWE identification step in the
construction process of our dataset involved time-consuming manual annotation.
Building a high-quality system that fully automates the process is an issue
for future work. Our dataset can be used to evaluate such a Japanese MWE
identification system.

Additionally, as shown in Section 5, our baseline model performed relatively
poorly in the higher complexity tiers. This is an effect of the dense
annotation setting; it results in uneven distributions of complexity as shown
in Figure 2, where easy words greatly outnumber difficult words. One possible
solution would be creating another LCP dataset using sparse annotation, where
target words are selected using frequency bands so that the words are
distributed across a wide range of frequency (Shardlow et al., 2022). Our data
could provide insights as to what kind of words should be targeted by sparse
annotation for such a dataset.

## Acknowledgments

We would like to express our gratitude to Justin Vasselli and the anonymous
reviewers for their insightful feedback. This work was supported by JSPS
KAKENHI grant number JP19K20351 and NAIST Foundation.

## References

  * Den et al. (2007) Yasuharu Den, Toshinobu Ogiso, Hideki Ogura, Atsushi Yamada, Nobuaki Menematsu, Kiyotaka Uchimoto, and Hanae Koiso. 2007.  The development of an electronic dictionary for morphological analysis and its application to japanese corpus linguistics.  _Japanese Linguistics_ , 22(5):101–123.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gooding and Kochmar (2019) Sian Gooding and Ekaterina Kochmar. 2019.  Complex word identification as a sequence labelling task.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1148–1153, Florence, Italy. Association for Computational Linguistics.
  * Gooding et al. (2021) Sian Gooding, Ekaterina Kochmar, Seid Muhie Yimam, and Chris Biemann. 2021.  Word complexity is in the eye of the beholder.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4439–4449, Online. Association for Computational Linguistics.
  * Gooding and Tragut (2022) Sian Gooding and Manuel Tragut. 2022.  One size does not fit all: The case for personalised word complexity models.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 353–365, Seattle, United States. Association for Computational Linguistics.
  * Hu and Nation (2000) Marcella Hu and I.S.P Nation. 2000.  Unknown Vocabulary Density and Reading Comprehension.  _Reading in a Foreign Language_ , 13(1):403–30.
  * Kochmar et al. (2020) Ekaterina Kochmar, Sian Gooding, and Matthew Shardlow. 2020.  Detecting multiword expression type helps lexical complexity assessment.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 4426–4435, Marseille, France. European Language Resources Association.
  * Kocmi et al. (2022) Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popović. 2022.  Findings of the 2022 conference on machine translation (WMT22).  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 1–45, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Koda (1989) Keiko Koda. 1989.  The effects of transferred vocabulary knowledge on the development of L2 reading proficiency.  _Foreign Lang. Ann._ , 22(6):529–540.
  * Komori et al. (2004) Kazuko Komori, Junko Mikuni, and Kondoh Atsuko. 2004.  Bunshō rikai o sokushin suru goi chishiki no ryōteki sokumen : Kichigo ritsu no ikichi tansaku no kokoromi [What percentage of known words in a text facilities reading comprehension? : A Case Study for Exploration of the Threshold of Known Words] (in Japanese).  _Nihongo Kyōiku [Journal of Japanese Language Teaching]_ , 120:83–92.
  * Kozawa et al. (2014) Shunsuke Kozawa, Uchimoto Kiyotaka, and Yasuharu Den. 2014.  Adaptation of long-unit-word analysis system to different part-of-speech tagset.  _Journal of Natural Language Processing_ , 21(2):379–401.
  * Krippendorff (1970) Klaus Krippendorff. 1970.  Bivariate Agreement Coefficients for Reliability of Data.  _Sociological Methodology_ , 2:139–150.
  * Kudo et al. (2004) Taku Kudo, Kaoru Yamamoto, and Yuji Matsumoto. 2004.  Applying conditional random fields to japanese morphological analysis.  In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_ , pages 230–237, Barcelona, Spain. Association for Computational Linguistics.
  * Matsuyoshi et al. (2007) Suguru Matsuyoshi, Satoshi Sato, and Takehito Utsuro. 2007.  A dictionary of japanese functional expressions with hierarchical organization.  _Journal of Natural Language Processing_ , 14(5):123–146.
  * Nishihara and Kajiwara (2020) Daiki Nishihara and Tomoyuki Kajiwara. 2020.  Word complexity estimation for Japanese lexical simplification.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 3114–3120, Marseille, France. European Language Resources Association.
  * Ogura et al. (2011) Hideki Ogura, Hanae Koiso, Yumi Fujiike, Sayaka Miyauchi, Hikari Konishi, and Yutaka Hara. 2011.  Gendai nihongo kakikotoba kinkō kōpasu keitairon kiteishū dai 4 ban jō [Regulations of morphological information for balanced corpus of contemporary written Japanese 4th edition volume 1] (in Japanese).  _NINJAL Internal Reports_.
  * Omura et al. (2021) Mai Omura, Aya Wakasa, and Masayuki Asahara. 2021.  Word delimitation issues in UD Japanese.  In _Proceedings of the Fifth Workshop on Universal Dependencies (UDW, SyntaxFest 2021)_ , pages 142–150, Sofia, Bulgaria. Association for Computational Linguistics.
  * Schwanenflugel et al. (1988) Paula J Schwanenflugel, Katherine Kip Harnishfeger, and Randall W Stowe. 1988.  Context availability and lexical decisions for abstract and concrete words.  _J. Mem. Lang._ , 27(5):499–520.
  * Shardlow et al. (2020) Matthew Shardlow, Michael Cooper, and Marcos Zampieri. 2020.  CompLex: A new corpus for lexical complexity prediction from Likert Scale data.  In _Proceedings of the 1st Workshop on Tools and Resources to Empower People with REAding DIfficulties (READI)_ , pages 57–62, Marseille, France. European Language Resources Association.
  * Shardlow et al. (2021) Matthew Shardlow, Richard Evans, Gustavo Henrique Paetzold, and Marcos Zampieri. 2021.  SemEval-2021 task 1: Lexical complexity prediction.  In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_ , pages 1–16, Online. Association for Computational Linguistics.
  * Shardlow et al. (2022) Matthew Shardlow, Richard Evans, and Marcos Zampieri. 2022.  Predicting lexical complexity in english texts: the complex 2.0 dataset.  _Language Resources and Evaluation_ , 56(4):1153–1194.
  * Štajner et al. (2022) Sanja Štajner, Daniel Ferrés, Matthew Shardlow, Kai North, Marcos Zampieri, and Horacio Saggion. 2022.  Lexical simplification benchmarks for english, portuguese, and spanish.  _Front Artif Intell_ , 5:991242.
  * Sunakawa et al. (2012) Yuriko Sunakawa, Jae-Ho Lee, and Mari Takahara. 2012.  The construction of a database to support the compilation of japanese learners’ dictionaries.  _Acta Linguistica Asiatica_ , 2(2):97.
  * Tateisi et al. (1988) Yuka Tateisi, Yoshihiko Ono, and Hisao Yamada. 1988.  A computer readability formula of japanese texts for machine scoring.  In _Coling Budapest 1988 Volume 2: International Conference on Computational Linguistics_.
  * Taya et al. (2021) Yuki Taya, Lis Kanashiro Pereira, Fei Cheng, and Ichiro Kobayashi. 2021.  OCHADAI-KYOTO at SemEval-2021 task 1: Enhancing model generalization and robustness for lexical complexity prediction.  In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_ , pages 17–23, Online. Association for Computational Linguistics.
  * Yimam et al. (2018) Seid Muhie Yimam, Chris Biemann, Shervin Malmasi, Gustavo Paetzold, Lucia Specia, Sanja Štajner, Anaïs Tack, and Marcos Zampieri. 2018.  A report on the complex word identification shared task 2018.  In _Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 66–78, New Orleans, Louisiana. Association for Computational Linguistics.

##  Appendix A Annotators

| Japanese | English
---|---|---
| B1/B2 | C1/C2 | B1/B2 | C1/C2
CK | 4 | 3 | 7 | 0
Non-CK | 6 | 1 | 2–3 | 4–5
Table 5: Annotator counts per sentence in each L1 group, by Japanese and English reading proficiency category. The proficiency levels were determined by self-reports with reference to an assessment grid either in Japanese171717 https://jfstandard.jp/pdf/self_assessment_jp.pdf or in English.181818 https://rm.coe.int/CoERMPublicCommonSearchServices/DisplayDCTMContent?documentId=090000168045bb52 Overall, our CK annotators are better at Japanese reading and poorer at English reading than the non-CK. CK | Chinese: 6, | Korean: 1 |  |
---|---|---|---|---
Non-CK | English: 2–3, | Thai: 2–3, | Indonesian: 1, | Lao: 1
Table 6: Annotator counts per sentence of each L1 group.

##  Appendix B Complexity labels

3 (Very Difficult): |  You hardly understand its meaning in the context.
---|---
2 (Difficult): |  You can infer its meaning, but you are not confident.
1 (Not Easy): |  You understand its meaning with confidence, but it is quite difficult among the expressions you can understand.
0 (Easy): |  None of the above.
Table 7: Complexity labels. An annotator can label spans with complexity 3, 2,
1, or 0.

##  Appendix C MWE Categories

Category |  Description |  Example
---|---|---
Lexicalized expressions |  Non-compositional expressions whose meaning as a whole cannot be completely inferred by the meaning of their components. |  使い $\big{|}$ 勝手 (ease of use)
Institutionalized expressions |  Compositional expressions whose components cannot be replaced without distorting the meaning of the whole expression or violating the language conventions. |  感染 $\big{|}$ 症 (infectious disease)
Functional expressions |  Expressions that behave like single function words. |  に $\big{|}$ つき $\big{|}$ まし $\big{|}$ て (as for)
Table 8: Categories we regard as MWEs. See Kochmar et al. (2020) for
lexicalized and institutionalized expressions, and Matsuyoshi et al. (2007)
for functional expressions. The vertical bars in the examples denote
boundaries between SUWs.

##  Appendix D Excluded categories

Category |  Identification Approach |  Example
---|---|---
Proper nouns |  Proper nouns are first identified by MeCab. We also manually annotate proper noun phrases. |  関東 $\big{|}$ 大 $\big{|}$ 震災 (The Great Kantō Earthquake)
Segmentation errors |  We manually annotate sequences with segmentation errors. |  も $\big{|}$ や (mist)
Table 9: Categories of words or spans we exclude from our target. The vertical
bars in the examples denote boundaries between SUWs. The correct segmentation
for も $\big{|}$ や is もや.

##  Appendix E Distributions of Annotation

Figure 4: Violin plot showing annotation distributions of several words with
dot markers showing the complexity scores, both for non-CK annotators. Words
are shown in their surface forms; the vertical bars in them denote boundaries
between SUWs.

##  Appendix F Examples of Words by Origin

Origin | Containing Chinese characters (kanji)?
---|---
Yes | No
Japanese |  歌 (song) | けれど $\big{|}$ も (although)
(wago) |  臨む (attend) | ふさわしい (suitable)
Chinese/Sino-Japanese |  今回 (this time) | よう (it seems †様)
(kango) |  民主 $\big{|}$ 主義 (democracy) | もちろん (of course †勿論)
Other |  旦那 (husband <Skt) | スクラップ (scrap <Eng)
(gairaigo) |  | ホーム $\big{|}$ ページ (home page <Eng)
Table 10: Examples of words in JaLeCoN categorized by word origin and whether
they contain Chinese characters. † marks a variant of the word written using
Chinese characters documenting the Sino-Japanese origin; < marks the word’s
origin (Sanskrit or English). The vertical bars in the examples denote
boundaries between SUWs. All categories except Other (gairaigo) written using
Chinese characters are relatively common.

##  Appendix G Experimental Setting

Optimizer | Adam ($\beta_{1}=0.9$, $\beta_{2}=0.999$)
---|---
– learning rate | 5e-5
– – schedule | no warm-up, linear decay
– L2 weight decay | 0.01
Epochs | 5
Loss function | Mean squared error
Dropout | 0.1
Batch size | 16
Weight initialization |  $\mathcal{N}(\mu=0,\,\sigma=0.02)$ truncated to $\pm 2\sigma$
Bias initialization | 0
Gradient L2 norm clipping | 2
Table 11: Hyperparameters used for fine-tuning the BERT model. We have chosen the combination of learning rate (from 8e-6, 5e-5, 3e-5, and 2e-5), warm-up (from no warm-up and 10% steps), and the number of epochs (from 1 to 5) achieving the highest mean $R^{2}$ in a nested 4-fold cross-validation on the training data of the first outer cross-validation split. The optimal combination was identical for CK and non-CK complexity. Folds | 5
---|---
Stratification | by genre (News and Government)
Grouping | by sequence of sentences (see Section 3)
Table 12: Cross-validation scheme used for fine-tuning and evaluation of the
BERT model.