* Mirkin  _et al._ [2019a] N. Mirkin, P. Poggi, and D. Wisniacki, Phys. Rev. A 99, 020301 (2019a).
  * Mirkin  _et al._ [2019b] N. Mirkin, P. Poggi, and D. Wisniacki, Phys. Rev. A 99, 062327 (2019b).
  * Abiuso and Giovannetti [2019] P. Abiuso and V. Giovannetti, Phys. Rev. A 99, 052106 (2019).
  * Camati  _et al._ [2020] P. A. Camati, J. F. G. Santos, and R. M. Serra, Phys. Rev. A 102, 012217 (2020).
  * Ptaszyński [2022] K. Ptaszyński, Phys. Rev. E 106, 014114 (2022).
  * Breuer  _et al._ [2002] H.-P. Breuer, F. Petruccione, _et al._ ,  _The theory of open quantum systems_ (Oxford University Press on Demand, 2002).
  * Rivas and Huelga [2012] A. Rivas and S. F. Huelga,  _Open quantum systems_ (Springer, 2012).
  * Rivas  _et al._ [2014] Á. Rivas, S. F. Huelga, and M. B. Plenio, Rep. Prog. Phys. 77, 094001 (2014).
  * Janzing  _et al._ [2000] D. Janzing, P. Wocjan, R. Zeier, R. Geiss, and T. Beth, Int. J. Theor. Phys. 39, 2717 (2000).
  * Horodecki and Oppenheim [2013] M. Horodecki and J. Oppenheim, Nat. Commun. 4, 2059 (2013).
  * Brandão  _et al._ [2015] F. G. S. L. Brandão, M. Horodecki, N. H. Y. Ng, J. Oppenheim, and S. Wehner, Proc. Natl. Acad. Sci. U.S.A. 112, 3275 (2015).
  * Lostaglio [2019] M. Lostaglio, Rep. Prog. Phys. 82, 114001 (2019).
  * Lostaglio and Korzekwa [2022] M. Lostaglio and K. Korzekwa, Phys. Rev. A 106, 012426 (2022).
  * Korzekwa and Lostaglio [2022] K. Korzekwa and M. Lostaglio, Phys. Rev. Lett. 129, 040602 (2022).
  * Lostaglio  _et al._ [2018] M. Lostaglio, Á. M. Alhambra, and C. Perry, Quantum 2, 52 (2018).
  * Son and Ng [2022] J. Son and N. H. Y. Ng, arXiv:2209.15213 (2022).
  * Venturelli  _et al._ [2013] D. Venturelli, R. Fazio, and V. Giovannetti, Phys. Rev. Lett. 110, 256801 (2013).
  * Erdman  _et al._ [2018] P. A. Erdman, B. Bhandari, R. Fazio, J. P. Pekola, and F. Taddei, Phys. Rev. B 98, 045433 (2018).
  * Hofer  _et al._ [2016] P. P. Hofer, M. Perarnau-Llobet, J. B. Brask, R. Silva, M. Huber, and N. Brunner, Phys. Rev. B 94, 235420 (2016).
  * Chen and Li [2012] Y.-X. Chen and S.-W. Li, EPL 97, 40003 (2012).
  * Mitchison  _et al._ [2016] M. T. Mitchison, M. Huber, J. Prior, M. P. Woods, and M. B. Plenio, Quantum Sci. Technol. 1, 015001 (2016).
  * Mazurek and Horodecki [2018] P. Mazurek and M. Horodecki, New J. Phys. 20, 053040 (2018).
  * Kossakowski [1972] A. Kossakowski, Rep. Math. Phys. 3, 247 (1972).
  * Gorini  _et al._ [1976] V. Gorini, A. Kossakowski, and E. C. G. Sudarshan, J. Math. Phys. 17, 821 (1976).
  * Lindblad [1976] G. Lindblad, Commun. Math. Phys. 48, 119 (1976).
  * de Oliveira Junior  _et al._ [2022] A. de Oliveira Junior, J. Czartowski, K. Życzkowski, and K. Korzekwa, Phys. Rev. E 106, 064109 (2022).
  * Jonathan and Plenio [1999] D. Jonathan and M. B. Plenio, Phys. Rev. Lett. 83, 3566 (1999).
  * White [1983] A. T. White, Math. Proc. Camb. Phil. Soc. 94, 203–215 (1983).
  * Alicki [1979] R. Alicki, J. Phys. A: Math. Gen. 12, L103 (1979).
  * Kosloff [2013] R. Kosloff, Entropy 15, 2100 (2013).
  * Allahverdyan [2014] A. E. Allahverdyan, Phys. Rev. E 90, 032137 (2014).
  * Solinas and Gasparinetti [2015] P. Solinas and S. Gasparinetti, Phys. Rev. E 92, 042150 (2015).
  * Perarnau-Llobet  _et al._ [2017] M. Perarnau-Llobet, E. Bäumer, K. V. Hovhannisyan, M. Huber, and A. Acin, Phys. Rev. Lett. 118, 070601 (2017).
  * Miller  _et al._ [2019] H. J. D. Miller, M. Scandi, J. Anders, and M. Perarnau-Llobet, Phys. Rev. Lett. 123, 230603 (2019).
  * Åberg [2013] J. Åberg, Nat. Commun. 4, 1925 (2013).
  * Skrzypczyk  _et al._ [2014] P. Skrzypczyk, A. J. Short, and S. Popescu, Nat. Commun. 5, 4185 (2014).
  * Brandão  _et al._ [2013] F. G. S. L. Brandão, M. Horodecki, J. Oppenheim, J. M. Renes, and R. W. Spekkens, Phys. Rev. Lett. 111, 250404 (2013).
  * Siegle  _et al._ [2010] P. Siegle, I. Goychuk, P. Talkner, and P. Hänggi, Phys. Rev. E 81, 011136 (2010).
  * Budini [2013] A. A. Budini, Phys. Rev. A 88, 032115 (2013).
  * Campbell  _et al._ [2018] S. Campbell, F. Ciccarello, G. M. Palma, and B. Vacchini, Phys. Rev. A 98, 012142 (2018).
  * vom Ende [2023] F. vom Ende, arXiv:2306.03667 (2023).
  * Owen  _et al._ [2019] J. A. Owen, A. Kolchinsky, and D. H. Wolpert, New Journal of Physics 21, 013022 (2019).
  * Ray  _et al._ [2021] K. J. Ray, A. B. Boyd, G. W. Wimsatt, and J. P. Crutchfield, Phys. Rev. Res. 3, 023164 (2021).
  * Korzekwa and Lostaglio [2021] K. Korzekwa and M. Lostaglio, Phys. Rev. X 11, 021019 (2021).
  * Korzekwa  _et al._ [2022] K. Korzekwa, Z. Puchała, M. Tomamichel, and K. Życzkowski, IEEE Trans. Inf. Theory 68, 4518 (2022).
  * Wolpert  _et al._ [2019] D. H. Wolpert, A. Kolchinsky, and J. A. Owen, Nat. Commun 10, 1 (2019).
  * Taranto  _et al._ [2023] P. Taranto, F. Bakhshinezhad, A. Bluhm, R. Silva, N. Friis, M. P. Lock, G. Vitagliano, F. C. Binder, T. Debarba, E. Schwarzhans, F. Clivaz, and M. Huber, PRX Quantum 4, 010332 (2023).
  * Son and Ng [2023] J. Son and N. H. Y. Ng, arXiv:2303.13020 (2023).
  * Marshall  _et al._ [1979] A. W. Marshall, I. Olkin, and B. C. Arnold,  _Inequalities: theory of majorization and its applications_, Vol. 143 (Springer, 1979).
  * Ruch  _et al._ [1978] E. Ruch, R. Schranner, and T. H. Seligman, J. Chem. Phys 69, 386 (1978).
  * Horodecki and Oppenheim [2013] M. Horodecki and J. Oppenheim, Int. J. Mod. Phys. B 27, 1345019 (2013).
  * Artin [2015] E. Artin,  _The gamma function_ (Courier Dover Publications, 2015).
  * [65] DLMF, NIST Digital Library of Mathematical Functions, https://dlmf.nist.gov/, Release 1.1.9 of 2023-03-15 (2023), f. W. J. Olver, A. B. Olde Daalhuis, D. W. Lozier, B. I. Schneider, R. F. Boisvert, C. W. Clark, B. R. Miller, B. V. Saunders, H. S. Cohl, and M. A. McClain, eds.


# Semi-Supervised Domain Adaptation with Source Label Adaptation

Yu-Chu Yu Hsuan-Tien Lin
National Taiwan University
{r09922104<EMAIL_ADDRESS>
###### Abstract

Semi-Supervised Domain Adaptation (SSDA) involves learning to classify unseen
target data with a few labeled and lots of unlabeled target data, along with
many labeled source data from a related domain. Current SSDA approaches
usually aim at aligning the target data to the labeled source data with
feature space mapping and pseudo-label assignments. Nevertheless, such a
source-oriented model can sometimes align the target data to source data of
the wrong classes, degrading the classification performance. This paper
presents a novel source-adaptive paradigm that adapts the source data to match
the target data. Our key idea is to view the source data as a noisily-labeled
version of the ideal target data. Then, we propose an SSDA model that cleans
up the label noise dynamically with the help of a robust cleaner component
designed from the target perspective. Since the paradigm is very different
from the core ideas behind existing SSDA approaches, our proposed model can be
easily coupled with them to improve their performance. Empirical results on
two state-of-the-art SSDA approaches demonstrate that the proposed model
effectively cleans up the noise within the source labels and exhibits superior
performance over those approaches across benchmark datasets. Our code is
available at https://github.com/chu0802/SLA.

##  1 Introduction

Figure 1: Top. Training the model with the original source labels might lead
to the misalignment of the target data. Bottom. After cleaning up the noisy
source labels with our SLA framework, the target data can be aligned with the
correct classes.

Domain Adaptation (DA) focuses on a general machine learning scenario where
training and test data may originate from two related but distinct domains:
the source domain and the target domain. Many works have extensively studied
unsupervised DA (UDA), where labels in the target domain cannot be accessed,
from both theoretical [2, 19, 36] and algorithmic [5, 15, 8, 22, 16, 37]
perspectives. Recently, Semi-Supervised Domain Adaptation (SSDA), another DA
setting that allows access to a few target labels, has received more research
attention because it is a simple yet realistic setting for application needs.

The most naïve strategy for SSDA, commonly known as S+T [33, 21], aims to
train a model using the source data and labeled target data with a standard
cross entropy loss. This strategy often suffers from a well-known domain shift
issue, which stems from the gap between different data distributions. To
address this issue, many state-of-the-art algorithms attempt to explore better
use of the unlabeled target data so that the target distribution can be
aligned with the source distribution. Recently, several Semi-Supervised
Learning (SSL) algorithms have been applied for SSDA [21, 12, 30] to
regularize the unlabeled data, such as entropy minimization [6], pseudo-
labeling [11, 24] and consistency regularization [1, 24]. These classic
source-oriented strategies have prevailed for a long time. However, these
algorithms typically require the target data to closely match some
semantically similar source data in the feature space. Therefore, if the S+T
space has been misaligned, it can be challenging to recover from the
misalignment, as illustrated in Figure 1.

Figure 2: Feature visualizations with t-SNE for an example of the misalignment
on the Office-Home A $\to$ C dataset with ResNet34. The model is trained by
S+T. Left: 0-th iteration. Right: 5000-th iteration . We observe that the
misalignment has already happened at a very early stage. Guided by source
labels and a few target labels, a portion of the target data from the 59th
class misaligns with the source data from the 7th class.

We take a deeper look into a specific example from the Office-Home dataset
[27] to confirm the abovementioned issue. Figure 2 visualizes the feature
space trained by S+T using t-SNE [3]. We observed that the misalignment
between the source and the target data has happened at a very early stage. For
instance, in the beginning, a portion of the target data from the 59th class
is close to the source data from the 7th class. Since we only have access to
source labels and a few target labels, without proper guidance from enough
target labels, such misalignment becomes more severe after being trained by
S+T. Table 1 shows the partial confusion matrix of S+T. Roughly 40% of the
target data in the 59th class is mispredicted to the 7th class, and only
around 20% of the data is classified correctly.

From the case study above, we argue that relying on the source labels like S+T
can misguide the model to learn wrong classes for some target data. That is,
source labels can be viewed as a noisy version of the ideal labels for target
classification. Based on the argument, the setting of SSDA is more like a
Noisy Label Learning (NLL) problem, with a massive amount of noisy labels
(source labels) and a small number of clean labels (target labels).

Learning with noisy labels is a widely studied machine learning problem. A
popular solution is to clean up the noisy labels with the help of another
model, which can also be referred to as label correction [28]. To approach
Domain Adaptation as an NLL problem, we borrow the idea from label correction
and propose a Source Label Adaptation (SLA) framework, as shown in Figure 1.
We construct a label adaptation component that provides the view from the
target data and dynamically cleans up the noisy source labels at each
iteration. Unlike other earlier works that study how to leverage the unlabeled
target data, we mainly investigate how to train the source data with the
adapted labels to better suit the ideal target space. This source-adaptive
paradigm is entirely orthogonal to the core ideas behind existing SSDA
algorithms. Thus, we can combine our framework with those algorithms to get
superior results. We summarize our contributions as follows.

  * •

We argue that the classic source-oriented methods might still suffer from the
biased feature space derived from S+T. To escape this predicament, we propose
adapting the source data to the target space by modifying the original source
labels.

  * •

We address DA as a particular case of NLL problems and present a novel source-
adaptive paradigm. Our SLA framework can be easily coupled with other existing
algorithms to boost their performance.

  * •

We demonstrate the usefulness of our proposed SLA framework when coupled with
state-of-the-art SSDA algorithms. The framework significantly improved
existing algorithms on two major benchmarks, inspiring a new direction for
solving DA problems.

True\Pred | Class 7 | Class 59 | Class 41 | Others
---|---|---|---|---
Class 59 | 38.5% | 19.8% | 13.5% | 28.2%

Table 1: A partial confusion matrix of S+T on the 3-shot Office-Home A $\to$ C
dataset with ResNet34.  Figure 3: An overview of our proposed framework,
Source Label Adaptation for SSDA. For source data, we adapt the original
source labels to better fit the target feature space by the Protonet with
Pseudo Centers (PPC) and calculate the label adaptation loss. For labeled
target data, we train it with a standard cross entropy loss. We can apply a
state-of-the-art algorithm to derive the unlabeled target loss for unlabeled
data. For every specific interval $I$, we update the pseudo labels and pseudo
centers to get a more reliable label adaptation model.

##  2 Related Work

Problem Setup.  DA focuses on a $K$-class classification task with an
$m$-dimensional input space $X\subseteq\mathbb{R}^{m}$ and a set of labels
$\\{1,2,\ldots,K\\}$. For simplicity, we define a label space $Y$ on the
probability simplex $\Delta^{K}$. A label $y=k\in\\{1,2,\ldots,K\\}$ is
equivalent to a one-hot encoded vector $\mathbf{y}\in Y$, where only the
$k$-th element is $1$ and the others are $0$. We consider two domains over
$X\times Y$, named source domain $D_{s}$ and target domain $D_{t}$. In SSDA,
we sample an amount of labeled source data
$S=\\{(\mathbf{x}_{i}^{s},y_{i}^{s})\\}_{i=1}^{|S|}$ from $D_{s}$, labeled
target data $L=\\{(\mathbf{x}_{i}^{\ell},y_{i}^{\ell})\\}_{i=1}^{|L|}$ from
$D_{t}$, and unlabeled target data $U=\\{\mathbf{x}_{i}^{u}\\}_{i=1}^{|U|}$
from the marginal distribution of $D_{t}$ over $X$. Typically, $|L|$ is
considerably smaller than $|S|$ and $|U|$, such as one or three examples per
class. Our goal is to train an SSDA model $g$ with $S,L$, and $U$ to perform
well on the target domain.

Semi-Supervised Domain Adaptation (SSDA).  SSDA can be viewed as a relaxed yet
realistic version of UDA. An SSDA algoirthm usually involves three loss
functions:

$\mathcal{L}_{\text{SSDA}}=\mathcal{L}_{s}+\mathcal{L}_{\ell}+\mathcal{L}_{u}$
(1)

where $\mathcal{L}_{s}$ stands for the loss derived by the source data.
$\mathcal{L}_{\ell},\mathcal{L}_{u}$ denotes the losses from the labeled and
unlabeled target data. As discussed in Section 1, based on S+T, a typical SSDA
algorithm usually focuses on designing $\mathcal{L}_{u}$ to better align the
target data with the source data. Recently, many existing works have borrowed
SSL techniques to conquer SSDA because of the problem similarity [35]. [21]
proposes a variant of entropy minimization [6] to explicitly align the target
data with source clusters. [31] decomposes SSDA into an SSL and a UDA task.
The two different sub-tasks produce pseudo labels respectively, and learn from
each other via co-training. [12] groups target features into clusters by
measuring pairwise feature similarity. [30] utilizes consistency
regularization at three different levels to perform domain alignment. Besides,
both [12, 30] apply pseudo labeling with data augmentations [24] to enhance
their performance. To the best of our knowledge, all methods listed above
mainly explore the usage of unlabeled target data while treating the source
data with the most straightforward strategy. In our study, we noticed that
source labels could appear noisy from the viewpoint of the target data. We
thus developed a source-adaptive framework to gradually adapt the source data
to the target space. Since we are addressing a new facet of the issue, our
framework can be easily applied to several SSDA algorithms mentioned above,
further improving the overall performance.

Noisy Label Learning (NLL).  The effectiveness of a machine learning algorithm
highly depends on the quality of collected labels. With regard to the present
deep neural network design [7], the aforementioned issue could worsen as deep
models have the capability to fit the data set in a seemingly random manner,
regardless of the quality of the labels [34].To clean the noisy labels, [20]
proposes a smoothing mechanism to mix noisy labels with self-prediction. [26]
models clean labels as trainable parameters and designs a joint optimization
algorithm to alternatively update parameters. [17, 25, 32] estimate a
transition matrix to correct the corrupted labels. However, learning a global
transition matrix usually need a strong assumption of how noisy labels come
from, which is difficult to verify in the real-world scenarios [29]. [38]
trains a label correction network in a meta-learning manner to help correct
noisy labels. Motivated by [20, 38], we propose a simple framework that can
efficiently build a label adaptation model to correct the noisy source labels.

##  3 Proposed Framework

Next, we propose a novel SSDA framework, Source Label Adaptation. An overview
of our proposed framework is shown in Figure 3. In Section 3.1, we connect the
(SS)DA problem to NLL and point out that a classic NLL method [20] cannot be
directly applied to solve SSDA. In Section 3.2, we review a classic few-shot
learning algorithm, Prototypical Network [23] and propose Protonet with Pseudo
Centers to better estimate the prototypes. In Section 3.3, we summarize our
framework and describe the implementation details.

###  3.1 Domain Adaptation as Noisy Label Learning

In Domain Adaptation, we seek an ideal model $g^{*}$ that can minimize
unlabeled target risk. Ideally, the most suitable label for a source instance
$\mathbf{x}_{i}^{s}$ in the target space should be
$g^{*}(\mathbf{x}_{i}^{s})$. That is, the ideal source loss
$\mathcal{L}_{s}^{*}$ is:

$\mathcal{L}_{s}^{*}(g|S)=\frac{1}{|S|}\sum_{i=1}^{|S|}H(g(\mathbf{x}_{i}^{s}),g^{*}(\mathbf{x}_{i}^{s})),$
(2)

where $H$ measures the cross entropy between two distributions.

Combining with the labeled target loss $\mathcal{L}_{\ell}$, we refer to the
model trained by $\mathcal{L}_{s}^{*}$ and $\mathcal{L}_{\ell}$ as ideally-
adapted S+T. The results of the ideally-adapted S+T reveal the full potential
to adapt source labels. As shown in Table 2, there is a significant difference
in performance between a standard S+T and an ideally-adapted S+T,
demonstrating that performance can be dramatically affected by only modifying
the source labels.

| A $\to$ C | P $\to$ C
---|---|---
Method | 1-shot | 3-shot | 1-shot | 3-shot
S+T | 52.9 | 58.1 | 48.8 | 55.5
ideally-adapted S+T | 82.9 | 87.4 | 81.6 | 86.0

Table 2: Accuracy (%) of S+T and ideally-adapted S+T on the 3-shot OfficeHome
dataset with ResNet34.

In practice, however, we can only approximate the ideal model. To address the
issue, we take the original source labels as a noisy version of the ideal
labels and approach DA as a NLL problem. We first apply a simple method
proposed by [20] to help correct the source labels, which we refer to it as
label correction with self-prediction [28]. Specifically, for each source
instance $\mathbf{x}_{i}^{s}$, we construct the modified source label
$\hat{\mathbf{y}}_{i}^{s}$ by combining the original label
$\mathbf{y}_{i}^{s}$ and the prediction from the current model $g$ with a
ratio $\alpha$.

$\hat{\mathbf{y}}_{i}^{s}=(1-\alpha)\cdot\mathbf{y}_{i}^{s}+\alpha\cdot
g(\mathbf{x}_{i}^{s})$ (3)

Then, the modified source loss $\hat{\mathcal{L}}_{s}$ is:

$\hat{\mathcal{L}}_{s}(g|S)=\frac{1}{|S|}\sum_{i=1}^{|S|}H(g(\mathbf{x}_{i}^{s}),\hat{\mathbf{y}}_{i}^{s})$
(4)

However, in DA, such a method might not be helpful since the model usually
overfits the source data, which makes
$g(\mathbf{x}_{i}^{s})\approx\mathbf{y}_{i}^{s}$. That is, the modified source
label $\hat{\mathbf{y}}_{i}^{s}$ can be almost the same as the original source
label $\mathbf{y}_{i}^{s}$ according to Eq. 3.

Figure 4 shows that when doing label correction with self-prediction, the KL
divergence from $\mathbf{y}^{s}$ to $g(\mathbf{x}^{s})$ could be close to $0$
after 2000 iterations, indicating that the self-prediction is almost the same
as the original label. In this case, doing the correction is nearly equivalent
to not doing so.

To benefit from the modified labels, we need to eliminate supervision from
source data. As an ideal clean label is the output from an ideal model
$g^{*}$, we should instead find a label adaptation model $g_{c}$ that can
approximate the ideal model and adapt the source labels to the view of target
data. We define an adapted labels $\tilde{\mathbf{y}}_{i}^{s}$ as a convex
combination between the original labels $\mathbf{y}_{i}^{s}$ and the output
from $g_{c}$, which is the same as [20].

$\tilde{\mathbf{y}}_{i}^{s}=(1-\alpha)\cdot\mathbf{y}_{i}^{s}+\alpha\cdot
g_{c}(\mathbf{x}_{i}^{s})$ (5)

Figure 4: Average KL divergence from $\mathbf{y}^{s}$ to $g(\mathbf{x}^{s})$
at each iteration (3-shot Office-Home A $\to$ C with ResNet34, smoothing by
EMA with a ratio 0.8).

###  3.2 Protonet with Pseudo Centers

In the semi-supervised setting, we can access a few target labels.
Nonetheless, learning from a limited number of target labels might suffer from
a severe overfitting issue. Thus, we learn a prototypical network (protonet)
[23] to overcome the few-shot problem.

Given a dataset $\\{\mathbf{x}_{i},y_{i}\\}_{i=1}^{N}$ and a feature extractor
$f$; let $N_{k}$ denote the number of data labeled with $k$. The prototype of
class $k$ is defined as the center of features with the same class:

$\mathbf{c}_{k}=\frac{1}{N_{k}}\sum_{i=1}^{N}\mathbbm{1}\\{y_{i}=k\\}\cdot
f(\mathbf{x}_{i}).$ (6)

Let $\mathbf{C}_{f}=\\{\mathbf{c}_{1},\ldots,\mathbf{c}_{K}\\}$ collects all
centers with extractor $f$. We define $P_{\mathbf{C}_{f}}:X\mapsto Y$ as a
protonet with centers $\mathbf{C}_{f}$:

$P_{\mathbf{C}_{f}}(\mathbf{x}_{i})_{k}=\frac{\exp(-d(f(\mathbf{x}_{i}),\mathbf{c}_{k})\cdot
T)}{\sum_{j=1}^{K}\exp(-d(f(\mathbf{x}_{i}),\mathbf{c}_{j})\cdot T)}$ (7)

Here $d:F\times F\mapsto[0,\infty)$ is a distance measure over feature space
$F$, usually measuring Euclidean distance. $T$ is a hyper-parameter that
controls the smoothness of output. As $T\to 0$, the output of a protonet would
be close to a uniform distribution.

Since we have access to the labeled target dataset $L$, by Eq. 6 and Eq. 7, we
can derive labeled target centers $\mathbf{C}_{f}^{\ell}$, and construct a
protonet with labeled target centers $P_{\mathbf{C}_{f}^{\ell}}$.

[23] demonstrated that when $d$ measures Euclidean distance, a protonet is
equivalent to a linear classifier with particular parameterization over $F$.
Thus, we can take the protonet as a label adaptation model over a particular
feature space. The protonet with labeled target centers is purely built from
the viewpoint of target data, which should reduce our concerns about the issue
mentioned in Section 3.1.

However, for a protonet, the ideal centers $\mathbf{C}_{f}^{*}$ should be
derived through the unlabeled target dataset
$\\{(\mathbf{x}_{i}^{u},y_{i}^{u})\\}_{i=1}^{|U|}$. Since we have only a few
target labels per class, the labeled target centers $\mathbf{C}_{f}^{\ell}$
are far away from the ideal centers $\mathbf{C}_{f}^{*}$. To better estimate
the ideal centers, we propose to find the pseudo centers for unlabeled target
data.

With the current model $g$, the pseudo label $\tilde{y}_{i}^{u}$ for an
unlabeled target instance $\mathbf{x}_{i}^{u}$ is:

$\tilde{y}_{i}^{u}=\arg\max_{k}g(\mathbf{x}_{i}^{u})_{k}$ (8)

After deriving unlabeled target data with pseudo labels
$\\{(\mathbf{x}_{i}^{u},\tilde{y}_{i}^{u})\\}_{i=1}^{|U|}$, we can get pseudo
centers $\tilde{\mathbf{C}}_{f}$ by Eq. 6, and further define a Protonet with
Pseudo Centers (PPC) $P_{\tilde{\mathbf{C}}_{f}}$ by Eq. 7.

Table 3 compares the average L2 distance from ideal centers
$\mathbf{C}_{f}^{*}$ to labeled target centers $\mathbf{C}_{f}^{\ell}$ and
pseudo centers $\tilde{\mathbf{C}}_{f}$ over the feature space trained by S+T.
The distance between $\tilde{\mathbf{C}}_{f}$ and $\mathbf{C}_{f}^{*}$ is
significantly shorter than the distance between $\mathbf{C}_{f}^{\ell}$ and
$\mathbf{C}_{f}^{*}$, which means the pseudo centers are indeed much closer to
the ideal centers.

Taking PPC as the label adaptation model, the modified label
$\tilde{\mathbf{y}}_{i}^{s}$ turns out to be:

$\tilde{\mathbf{y}}_{i}^{s}=(1-\alpha)\cdot\mathbf{y}_{i}^{s}+\alpha\cdot
P_{\tilde{\mathbf{C}}_{f}}(\mathbf{x}_{i}^{s})$ (9)

From / To | labeled target centers | pseudo centers
---|---|---
ideal centers | 10.02 | 4.06

Table 3: Average L2 Distance from ideal centers to labeled target centers /
pseudo centers over the feature space trained by S+T (3-shot Office-Home A
$\to$ C with ResNet34).

###  3.3 Source Label Adaptation for SSDA

We propose a label adaptation loss for source data to replace the typical
source loss with a standard cross entropy loss. For each source instance
$\mathbf{x}_{i}^{s}$ with label $\mathbf{y}_{i}^{s}$, we first compute the
modified source label $\tilde{\mathbf{y}}_{i}^{s}$ by Eq. 9. Then, the label
adaptation loss $\tilde{\mathcal{L}}_{s}$ is:

$\tilde{\mathcal{L}}_{s}(g|S)=\frac{1}{|S|}\sum_{i=1}^{|S|}H(g(\mathbf{x}_{i}^{s}),\tilde{\mathbf{y}}_{i}^{s})$
(10)

Our framework, Source Label Adaptation (SLA) for SSDA, can be trained by the
following loss function.

$\mathcal{L}_{\text{SSDA w/
SLA}}=\tilde{\mathcal{L}}_{s}+\mathcal{L}_{\ell}+\mathcal{L}_{u}$ (11)

$\mathcal{L}_{\ell}$ is the loss function for labeled target data $L$, which
can still be a standard cross entropy loss. In contrast to other widely used
methods, we primarily concentrate on improving the usage of source data.
Therefore, the loss function for unlabeled target data $\mathcal{L}_{u}$ can
be derived through any state-of-the-art algorithm, and our framework can be
easily coupled with other methods without contradiction.

####  3.3.1 Implementation Details

Warmup Stage.  Our label adaptation framework relies on the quality of the
predicted pseudo labels. However, the prediction from the initial model can be
noisy. Thus, we introduce a hyperparameter $W$ for warmup to get more stable
pseudo labels. During the warmup stage, we train our model normally with
original source labels. Specifically, at the $e$-th iteration, we compute the
modified source label $\tilde{\mathbf{y}}_{i}^{s}$ as follows:

$\tilde{\mathbf{y}}_{i}^{s}=\begin{cases}\hfil\mathbf{y}_{i}^{s}&\text{if
}e\leq W\\\ (1-\alpha)\cdot\mathbf{y}_{i}^{s}+\alpha\cdot
P_{\tilde{\mathbf{C}}_{f}}(\mathbf{x}_{i}^{s})&\text{otherwise}\end{cases}$
(12)

Dynamic Update.  The feature space and the predicted pseudo labels constantly
evolve during the training phase. By updating the pseudo labels and centers,
we can remain the quality of the projected pseudo centers the same. It would
be ideal for updating the centers at each iteration. In practice, we update
the pseudo labels through Eq. 8 and update centers with the current feature
extractor $f$ through Eq. 6 for every specific interval $I$. Prior works [14]
have addressed a similar issue and proposed to maintain a memory bank for
dynamic updates of the estimated centers. However, in our framework, we need
to update both the estimated centers and pseudo labels simultaneously.
Therefore, we decided to adopt a more straightforward solution to mitigate the
demands on time and complexity.

##  4 Experiments

| R $\rightarrow$ C | R $\rightarrow$ P | P $\rightarrow$ C | C $\rightarrow$ S | S $\rightarrow$ P | R $\rightarrow$ S | P $\rightarrow$ R | Mean
---|---|---|---|---|---|---|---|---
Method | 1-shot | 3-shot | 1-shot | 3-shot | 1-shot | 3-shot | 1-shot | 3-shot | 1-shot | 3-shot | 1-shot | 3-shot | 1-shot | 3-shot | 1-shot | 3-shot
S+T | 55.6 | 60.0 | 60.6 | 62.2 | 56.8 | 59.4 | 50.8 | 55.0 | 56.0 | 59.5 | 46.3 | 50.1 | 71.8 | 73.9 | 56.9 | 60.0
DANN [5] | 58.2 | 59.8 | 61.4 | 62.8 | 56.3 | 59.6 | 52.8 | 55.4 | 57.4 | 59.9 | 52.2 | 54.9 | 70.3 | 72.2 | 58.4 | 60.7
ENT [6] | 65.2 | 71.0 | 65.9 | 69.2 | 65.4 | 71.1 | 54.6 | 60.0 | 59.7 | 62.1 | 52.1 | 61.1 | 75.0 | 78.6 | 62.6 | 67.6
APE [10] | 70.4 | 76.6 | 70.8 | 72.1 | 72.9 | 76.7 | 56.7 | 63.1 | 64.5 | 66.1 | 63.0 | 67.8 | 76.6 | 79.4 | 67.6 | 71.7
DECOTA [31] | 79.1 | 80.4 | 74.9 | 75.2 | 76.9 | 78.7 | 65.1 | 68.6 | 72.0 | 72.7 | 69.7 | 71.9 | 79.6 | 81.5 | 73.9 | 75.6
MCL [30] | 77.4 | 79.4 | 74.6 | 76.3 | 75.5 | 78.8 | 66.4 | 70.9 | 74.0 | 74.7 | 70.7 | 72.3 | 82.0 | 83.3 | 74.4 | 76.5
MME [21] | 70.0 | 72.2 | 67.7 | 69.7 | 69.0 | 71.7 | 56.3 | 61.8 | 64.8 | 66.8 | 61.0 | 61.9 | 76.1 | 78.5 | 66.4 | 68.9
MME + SLA (ours) | 71.8 | 73.3 | 68.2 | 70.1 | 70.4 | 72.7 | 59.3 | 63.4 | 64.9 | 67.3 | 61.8 | 63.9 | 77.2 | 79.6 | 68.8 | 70.0
CDAC [12] | 77.4 | 79.6 | 74.2 | 75.1 | 75.5 | 79.3 | 67.6 | 69.9 | 71.0 | 73.4 | 69.2 | 72.5 | 80.4 | 81.9 | 73.6 | 76.0
CDAC + SLA (ours) | 79.8 | 81.6 | 75.6 | 76.0 | 77.4 | 80.3 | 68.1 | 71.3 | 71.7 | 73.5 | 71.7 | 73.5 | 80.4 | 82.5 | 75.0 | 76.9

Table 4: Accuracy (%) on DomainNet for 1-shot and 3-shot Semi-Supervised
Domain Adaptation (ResNet34).

Method | A$\to$C | A$\to$P | A$\to$R | C$\to$A | C$\to$P | C$\to$R | P$\to$A | P$\to$C | P$\to$R | R$\to$A | R$\to$C | R$\to$P | Mean
---|---|---|---|---|---|---|---|---|---|---|---|---|---
One-shot
S+T | 50.9 | 69.8 | 73.8 | 56.3 | 68.1 | 70.0 | 57.2 | 48.3 | 74.4 | 66.2 | 52.1 | 78.6 | 63.8
DANN [5] | 52.3 | 67.9 | 73.9 | 54.1 | 66.8 | 69.2 | 55.7 | 51.9 | 68.4 | 64.5 | 53.1 | 74.8 | 62.7
ENT [6] | 52.9 | 75.0 | 76.7 | 63.2 | 73.6 | 73.2 | 63.0 | 51.9 | 79.9 | 70.4 | 53.6 | 81.9 | 67.9
APE [10] | 53.9 | 76.1 | 75.2 | 63.6 | 69.8 | 72.3 | 63.6 | 58.3 | 78.6 | 72.5 | 60.7 | 81.6 | 68.9
DECOTA[31] | 42.1 | 68.5 | 72.6 | 60.3 | 70.4 | 70.7 | 60.0 | 48.8 | 76.9 | 71.3 | 56.0 | 79.4 | 64.8
MME [21] | 59.6 | 75.5 | 77.8 | 65.7 | 74.5 | 74.8 | 64.7 | 57.4 | 79.2 | 71.2 | 61.9 | 82.8 | 70.4
MME + SLA (ours) | 62.1 | 76.3 | 78.6 | 67.5 | 77.1 | 75.1 | 66.7 | 59.9 | 80.0 | 72.9 | 64.1 | 83.8 | 72.0
CDAC [12] | 61.2 | 75.9 | 78.5 | 64.5 | 75.1 | 75.3 | 64.6 | 59.3 | 80.0 | 72.7 | 61.9 | 83.1 | 71.0
CDAC + SLA (ours) | 63.0 | 78.0 | 79.2 | 66.9 | 77.6 | 77.0 | 67.3 | 61.8 | 80.5 | 72.7 | 66.1 | 84.6 | 72.9
Three-shot
S+T | 54.0 | 73.1 | 74.2 | 57.6 | 72.3 | 68.3 | 63.5 | 53.8 | 73.1 | 67.8 | 55.7 | 80.8 | 66.2
DANN [5] | 54.7 | 68.3 | 73.8 | 55.1 | 67.5 | 67.1 | 56.6 | 51.8 | 69.2 | 65.2 | 57.3 | 75.5 | 63.5
ENT [6] | 61.3 | 79.5 | 79.1 | 64.7 | 79.1 | 76.4 | 63.9 | 60.5 | 79.9 | 70.2 | 62.6 | 85.7 | 71.9
APE [10] | 63.9 | 81.1 | 80.2 | 66.6 | 79.9 | 76.8 | 66.1 | 65.2 | 82.0 | 73.4 | 66.4 | 86.2 | 74.0
DECOTA [31] | 64.0 | 81.8 | 80.5 | 68.0 | 83.2 | 79.0 | 69.9 | 68.0 | 82.1 | 74.0 | 70.4 | 87.7 | 75.7
MME [21] | 63.6 | 79.0 | 79.7 | 67.2 | 79.3 | 76.6 | 65.5 | 64.6 | 80.1 | 71.3 | 64.6 | 85.5 | 73.1
MME + SLA (ours) | 65.9 | 81.1 | 80.5 | 69.2 | 81.9 | 79.4 | 69.7 | 67.4 | 81.9 | 74.7 | 68.4 | 87.4 | 75.6
CDAC [12] | 65.9 | 80.3 | 80.6 | 67.4 | 81.4 | 80.2 | 67.5 | 67.0 | 81.9 | 72.2 | 67.8 | 85.6 | 74.8
CDAC + SLA (ours) | 67.3 | 82.6 | 81.4 | 69.2 | 82.1 | 80.1 | 70.1 | 69.3 | 82.5 | 73.9 | 70.1 | 87.1 | 76.3

Table 5: Accuracy (%) on Office-Home for 1-shot and 3-shot Semi-Supervised
Domain Adaptation (ResNet34).

Method | A$\to$C | A$\to$P | A$\to$R | C$\to$A | C$\to$P | C$\to$R | P$\to$A | P$\to$C | P$\to$R | R$\to$A | R$\to$C | R$\to$P | Mean
---|---|---|---|---|---|---|---|---|---|---|---|---|---
MCL [30] | 67.5 | 83.9 | 82.4 | 71.4 | 84.3 | 81.6 | 69.9 | 68.0 | 83.0 | 75.3 | 70.1 | 88.1 | 77.1
MCL* | 64.1 | 81.6 | 80.6 | 70.3 | 82.2 | 79.2 | 70.6 | 64.0 | 81.8 | 75.3 | 67.8 | 86.6 | 75.3
MCL + SLA (ours) | 64.3 | 81.6 | 80.8 | 70.2 | 82.6 | 79.4 | 70.9 | 64.2 | 82.2 | 75.5 | 68.0 | 86.8 | 75.6
*: Reproduced by ourselves

Table 6: Accuracy (%) of MCL and MCL + SLA on Office-Home for 3-shot Semi
Supervised Domain Adaptation (ResNet34).

We first sketch our experiment setup, including data sets, competing methods,
and parameter settings in Section 4.1. We then present experimental results to
validate the superiority of the proposed SLA framework in Section 4.2. We
further analyze our proposed framework and highlight the limitation in Section
4.3.

###  4.1 Experiment Setup

Datasets.  We evaluate our proposed SLA framework on two sets of SSDA
benchmarks, including Office-Home [27] and DomainNet [18]. Office-Home is a
mainstream benchmark for both UDA and SSDA. It contains four domains: Art (A),
Clipart (C), Product (P), and Real (R), with 65 categories. DomainNet is
initially designed for benchmarking Multi-Source Domain Adaptation approaches.
[21] pickup four domains: Real (R), Clipart (C), Painting (P), and Sketch (S)
with 126 classes to build a cleaner dataset for SSDA. Besides, they focus on
seven scenarios instead of combining all pairs. Our experiments follow the
settings in recent works [21, 12, 30], with the same sampling strategy for
both the training set and validation set, and we conduct both 1-shot and
3-shot settings on all datasets.

Implementation Details.  Our framework can be applied with many state-of-the-
art methods, we choose MME [21], and CDAC [12] as our cooperators to validate
the efficacy of our method, named MME + SLA and CDAC + SLA, respectively. For
a fair comparison, we choose ResNet34 [7] as our backbone. The backbone is
pre-trained on ImageNet-1K dataset [4], and the model architecture, batch
size, learning rate scheduler, optimizer, weight-decay, and initialization
strategy are all followed as previous works [21, 12, 30]. We follow the same
hyper-parameters for MME and CDAC as their suggestions. We set the mix ratio
$\alpha$ in Eq. 12 to $0.3$ and the temperature parameter $T$ in Eq. 7 to
$0.6$. The update interval $I$ mentioned in Section 3.3 is $500$. The warmup
parameter $W$ in Eq. 12 is $500$ for MME on Office-Home; $2000$ for CDAC on
Office-Home; $3000$ for MME on DomainNet; $50000$ for CDAC on DomainNet. After
the warmup stage, we refresh the learning rate scheduler so that the label
adaptation loss can be updated with a higher learning rate. All hyper-
parameters can be properly tuned via the validation process. For each subtask,
we conducted the experiments three times. The detailed statistics of our
results can be found in our supplementary materials.

###  4.2 Comparison with State-of-the-Arts

We compare our results with several baselines, including S+T, DANN [5], ENT
[6], MME [21] APE [10], CDAC [12], DECOTA [31], MCL [30]. S+T is a baseline
method for SSDA, with only source data and labeled target data involved in the
training process. DANN is a classic unsupervised domain adaptation method, and
[21] reproduces it by training with additional labeled target data. ENT is a
standard entropy minimization originally designed for Semi-Supervised
Learning, and the reproduction was also done by [21]. Note that for MCL, we
only compare with their results on DomainNet. We leave the detailed analysis
for MCL on Office-Home in Section 4.3.

DomainNet.  We show the results on DomainNet dataset with 1-shot and 3-shot
settings on Table 4. It is worth noting two things. First, for MME and CDAC,
almost all sub-tasks get improvement after applying our SLA framework, except
for only two cases where CDAC + SLA performs roughly the same as CDAC. Second,
the overall performance of CDAC + SLA for 1-shot and 3-shot settings reaches
75.0% and 76.9%, respectively; both outperform the previous methods and
achieve new state-of-the-art results.

Office-Home.  We show the results on Office-Home dataset with 1-shot and
3-shot settings on Table 5. Similarly, after applying SLA to MME and CDAC, the
performances get much better except for only one case under the 3-shot
setting. Overall, our framework improves the original works by at least 1.5%
under all settings.

###  4.3 Analysis

Reproducibility issue for MCL.  MCL [30] designs consistency regularization
for SSDA at three different levels and achieves excellent results. However,
our experiments cannot fully reproduce their reported numbers. The reproduced
results on 3-shot Office-Home dataset are shown in Table 6. After applying our
SLA framework, although we can stably improve our reproduction, we are still
unable to compete with their reported values. We put our detailed reproducing
results into the supplementary materials, and the implementation is avaiable
at https://github.com/chester256/MCL.

Figure 5: The intermediate results in SLA on 3-shot Office-Home A $\to$ P with
ResNet34.

The intermediate results in SLA.  In SLA, we build a PPC to provide the view
from the target data. PPC can be viewed as a variant of the pseudo-labeling
method proposed in [13]. They apply such a method to boost their final
performance in their work. If PPC has performed well, a natural question is:
Is it necessary to modify source labels by PPC?

To reveal the intermediate steps within SLA, we plot the test performance of
MME (red), MME + PPC (orange), MME + SLA (blue), and PPC within MME + SLA
(purple) during the training phase in Figure 5. Initially, PPC (target view)
performs at a higher level. However, if the source labels are not adapted, it
will end up converge to the same performance as MME. In contrast, within our
SLA framework, the model leverages the benefits of PPC, further producing an
enhanced version of PPC, resulting in better overall performance compared to
the original MME.

$\alpha$ | 0.1 | 0.3 | 0.5 | 0.7 | 0.9
---|---|---|---|---|---
validation | 74.87 | 75.94 | 75.76 | 75.38 | 72.39
test | 74.43 | 75.57 | 75.38 | 74.79 | 72.76

Table 7: Average accuracy (%) of MME + SLA on 3-shot Office-Home with
ResNet34.

Sensitivity study of $\alpha$.  Table 7 shows the sensitivity study results
for $\alpha$ using MME + SLA on 3-shot OfficeHome dataset. We selected $0.3$
from the best validation performance and kept $0.3$ throughout all experiments
for simplicity and resource saving. In the mid-range, $\alpha=0.3$ (favoring
source view) or $\alpha=0.7$ (favoring target view) perform similarly, hinting
that it is stable enough for a proper range of choices. It is worth noting
that the dynamic adjustment of $\alpha$ is a promising direction. We proposed
the warmup stage that changes $\alpha$ from $0$ to the desired value after a
period of warming up. More sophisticated scheduling techniques, such as the
linear growth approach [9], could potentially replace the warmup parameter and
lead to further improvements. We leave exploration of these techniques for
future work.

Figure 6: Top-3 probabilities of the average adapted source labels on 3-shot
Office-Home A $\to$ C with ResNet34. Top. Class 1 (backpack). Bottom. Class 30
(knives).

Illustration of the adapted labels.  We aim to adapt the original source label
$\mathbf{y}_{i}^{s}$ to the ideal label $g^{*}(\mathbf{x}_{i}^{s})$. To
demonstrate the change of labels, we visualize the top 3 probabilities of the
average adapted source labels on two classes in Figure 6. For Backpack (class
1), ideally-adapted S+T should change the source label to 40% Backpack + 10%
Kettle + 8% Toys; SLA proposes to change to 30% Backpack + 5% Toys + 4%
Kettle, which is closer to the ideally-adapted label than the original label
of 100% Backpack. We draw the same conclusion for knives (class 30).

Figure 7: Label Adaptation Loss of MME + SLA by first pre-training MME for $W$
iterations on 3-shot Office-Home A $\to$ C with ResNet34. (Smoothing by EMA
with a ratio $0.8$.)

Warmup issue for MME + SLA.  As described in Section 3.3, our framework relies
on the quality of the predicted pseudo labels. Thus, we introduce a warmup
stage parameter $W$ to derive a robust model. We can treat the warmup strategy
as a two-stage algorithm. Take MME as our backbone method; the algorithm works
like:

  1. 1.

Train a model with normal MME loss for $W$ iteration.

  2. 2.

Take the model above as a pre-trained model and further applying label
adaptation loss.

For the first step, intuitively, we should train the model until the loss
converges. That is how we select the warmup stage parameter for CDAC + SLA.
However, empirically we found that the performance of MME + SLA will degrade
if we train an MME model until it converges. Table 8 shows the sensitivity
test of $W$ using MME + SLA on Office-Home dataset. We can observe that no
matter the 1-shot or 3-shot settings, the performance is generally getting
worse as the number of warmup stages increases. To analyze the effect, we
first pre-train a normal MME for $W$ iterations, then observe the label
adaptation loss of MME + SLA. Figure 7 plots the label adaptation loss of MME
+ SLA by first pre-training MME for $W$ iterations. We can observe that when
$W=5000$, the initial label adaptation loss has already been close to $0$.
Doing label adaptation in the situation is almost equivalent to not doing so,
as we mentioned in Section 3.1.

| A $\to$ C
---|---
Warmup Stage ($W$) | 1-shot | 3-shot
500 | 62.09 | 65.90
1000 | 61.95 | 64.99
2000 | 61.37 | 64.72
3000 | 61.53 | 64.87
5000 | 61.79 | 64.68

Table 8: Accuracy (%) for different warmup stage $W$ of MME + SLA on Office-
Home A $\to$ C with ResNet34.

Limitation.  Our SLA framework might not be helpful if the label adaptation
loss approaches $0$. Although we have applied the Protonet with Pseudo Centers
to avoid the issue, the loss will converge to $0$ in MME + SLA. We leave the
analysis of the reason for the convergence as a future work. On the other
hand, we argue that it is unnecessary to discuss the reason in our proposed
scope since we can make a trade-off by carefully tuning the warmup parameter
$W$, and the issue turns out to be part of the hyper-parameters selection.

##  5 Conclusion

In this work, we present a general framework, Source Label Adaptation for
Semi-Supervised Domain Adaptation. Our work highlights that the usage of
source data should be revisited carefully. We argue that the original source
labels might be noisy from the perspective of target data. We approach Domain
Adaptation as a Noisy Label Learning problem and correct source labels with
the predictions from Protonet with Pseudo Centers. Our approach mainly
addresses an issue that is orthogonal to other existing works, which focus on
improving the usage of unlabeled data. The empirical results show that we can
apply our framework to several state-of-the-art algorithms for SSDA and
further boost their performances.

Acknowledgement.  We thank the anonymous reviewers for valuable comments. This
work is supported by the Ministry of Science and Technology in Taiwan via the
grant MOST 111-2628-E-002-018. We thank to National Center for High-
performance Computing (NCHC) in Taiwan for providing computational and storage
resources.

## References

  * [1] Philip Bachman, Ouais Alsharif, and Doina Precup.  Learning with pseudo-ensembles.  Advances in neural information processing systems, 27, 2014.
  * [2] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Vaughan.  A theory of learning from different domains.  Machine Learning, 79:151–175, 2010.
  * [3] David M Chan, Roshan Rao, Forrest Huang, and John F Canny.  Gpu accelerated t-distributed stochastic neighbor embedding.  Journal of Parallel and Distributed Computing, 131:1–13, 2019.
  * [4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  * [5] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  The journal of machine learning research, 17(1):2096–2030, 2016\.
  * [6] Yves Grandvalet and Yoshua Bengio.  Semi-supervised learning by entropy minimization.  Advances in neural information processing systems, 17, 2004.
  * [7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [8] Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G Hauptmann.  Contrastive adaptation network for unsupervised domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4893–4902, 2019.
  * [9] Kyungyul Kim, ByeongMoon Ji, Doyoung Yoon, and Sangheum Hwang.  Self-knowledge distillation with progressive refinement of targets.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6567–6576, 2021.
  * [10] Taekyung Kim and Changick Kim.  Attract, perturb, and explore: Learning a feature alignment network for semi-supervised domain adaptation.  In European conference on computer vision, pages 591–607. Springer, 2020.
  * [11] Dong-Hyun Lee.  Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks.  ICML 2013 Workshop : Challenges in Representation Learning (WREPL), 07 2013.
  * [12] Jichang Li, Guanbin Li, Yemin Shi, and Yizhou Yu.  Cross-domain adaptive clustering for semi-supervised domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2505–2514, 2021.
  * [13] Jian Liang, Dapeng Hu, and Jiashi Feng.  Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation.  In International Conference on Machine Learning, pages 6028–6039. PMLR, 2020.
  * [14] Jian Liang, Dapeng Hu, and Jiashi Feng.  Domain adaptation with auxiliary target domain-oriented classifier.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16632–16642, 2021.
  * [15] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan.  Unsupervised domain adaptation with residual transfer networks.  Advances in neural information processing systems, 29, 2016.
  * [16] Jaemin Na, Heechul Jung, Hyung Jin Chang, and Wonjun Hwang.  Fixbi: Bridging domain spaces for unsupervised domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1094–1103, June 2021.
  * [17] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu.  Making deep neural networks robust to label noise: A loss correction approach.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1944–1952, 2017.
  * [18] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.  Moment matching for multi-source domain adaptation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1406–1415, 2019.
  * [19] Ievgen Redko, Amaury Habrard, and Marc Sebban.  Theoretical analysis of domain adaptation with optimal transport.  In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 737–753. Springer, 2017.
  * [20] Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich.  Training deep neural networks on noisy labels with bootstrapping.  arXiv preprint arXiv:1412.6596, 2014.
  * [21] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko.  Semi-supervised domain adaptation via minimax entropy.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8050–8058, 2019.
  * [22] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada.  Maximum classifier discrepancy for unsupervised domain adaptation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3723–3732, 2018.
  * [23] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  Advances in neural information processing systems, 30, 2017.
  * [24] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  Advances in neural information processing systems, 33:596–608, 2020\.
  * [25] Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus.  Training convolutional networks with noisy labels.  arXiv preprint arXiv:1406.2080, 2014.
  * [26] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa.  Joint optimization framework for learning with noisy labels.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5552–5560, 2018.
  * [27] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.  Deep hashing network for unsupervised domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5018–5027, 2017.
  * [28] Xinshao Wang, Yang Hua, Elyor Kodirov, Sankha Subhra Mukherjee, David A Clifton, and Neil M Robertson.  Proselflc: Progressive self label correction towards a low-temperature entropy state.  arXiv preprint arXiv:2207.00118, 2022.
  * [29] Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama.  Part-dependent label noise: Towards instance-dependent label noise.  Advances in Neural Information Processing Systems, 33:7597–7610, 2020.
  * [30] Zizheng Yan, Yushuang Wu, Guanbin Li, Yipeng Qin, Xiaoguang Han, and Shuguang Cui.  Multi-level consistency learning for semi-supervised domain adaptation.  arXiv preprint arXiv:2205.04066, 2022.
  * [31] Luyu Yang, Yan Wang, Mingfei Gao, Abhinav Shrivastava, Kilian Q Weinberger, Wei-Lun Chao, and Ser-Nam Lim.  Deep co-training with task decomposition for semi-supervised domain adaptation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8906–8916, 2021.
  * [32] Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama.  Dual t: Reducing estimation error for transition matrix in label-noise learning.  Advances in neural information processing systems, 33:7260–7271, 2020.
  * [33] Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha.  Few-shot learning via embedding adaptation with set-to-set functions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8808–8817, 2020.
  * [34] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning (still) requires rethinking generalization.  Communications of the ACM, 64(3):107–115, 2021.
  * [35] Yabin Zhang, Haojian Zhang, Bin Deng, Shuai Li, Kui Jia, and Lei Zhang.  Semi-supervised models are strong unsupervised domain adaptation learners.  arXiv preprint arXiv:2106.00417, 2021.
  * [36] Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon.  On learning invariant representations for domain adaptation.  In International Conference on Machine Learning, pages 7523–7532. PMLR, 2019.
  * [37] Yin Zhao, Longjun Cai, et al.  Reducing the covariate shift by mirror samples in cross domain alignment.  Advances in Neural Information Processing Systems, 34:9546–9558, 2021.
  * [38] Guoqing Zheng, Ahmed Hassan Awadallah, and Susan Dumais.  Meta label correction for noisy label learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11053–11061, 2021.


# Dynatask: A Framework for Creating Dynamic AI Benchmark Tasks

Tristan Thrush‡, Kushal Tirumala†, Anmol Gupta§, Max Bartolo¶, Pedro
Rodriguez†, Tariq Kane††, William Gaviria Rojas††, Peter Mattson⋆, Adina
Williams†, Douwe Kiela‡

‡ Hugging Face; † Facebook AI Research; § The University of Hong Kong;
¶ University College London; ⋆ Google; †† Coactive AI;
<EMAIL_ADDRESS>TT and DK conducted most of the work for this paper when they
were at Facebook AI Research.

###### Abstract

We introduce Dynatask: an open source system for setting up custom NLP tasks
that aims to greatly lower the technical knowledge and effort required for
hosting and evaluating state-of-the-art NLP models, as well as for conducting
model in the loop data collection with crowdworkers. Dynatask is integrated
with Dynabench, a research platform for rethinking benchmarking in AI that
facilitates human and model in the loop data collection and evaluation. To
create a task, users only need to write a short task configuration file from
which the relevant web interfaces and model hosting infrastructure are
automatically generated. The system is available at https://dynabench.org/ and
the full library can be found at
https://github.com/facebookresearch/dynabench.

##  1 Introduction

Data is the backbone of NLP research. One of the most fruitful approaches for
making progress on NLP tasks has historically been benchmarking. Benchmarking
is where the community adopts a high quality dataset for a particular task and
tests various models against it to determine which is best. The process of
benchmarking requires the effort of a large number of researchers, who collect
and clean data, train and evaluate models, and work to understand model
weaknesses. This process is iterative: once models perform very highly on the
currently accepted community benchmark, another is created to push progress
further. Taken as a whole, the benchmarking process is both notoriously
difficult and expensive. This is due to a variety of facts: the community is a
loose conglomeration of researchers with different areas of expertise, there
is ever increasing need for larger datasets (Halevy et al., 2009), and the AI
community has historically under-valued (Wagstaff, 2012) and under-invested in
data collection and best practices (Kiela et al., 2021; Sambasivan et al.,
2021; Mattson et al., 2022).

To make matters worse, in recent years, benchmarks have been saturating with
increasing speed. Taking the trends from the greater AI community into
account, it took MNIST (LeCun et al., 1998), Switchboard (Godfrey et al.,
1992), and ImageNet (Deng et al., 2009) several years to saturate, and newer
benchmarks such as SQuAD (Rajpurkar et al., 2016), GLUE (Wang et al., 2018),
and SuperGLUE (Wang et al., 2019) about a year. Because of this, data-centric
approaches are gaining more attention Ng et al. (2021); Mattson et al. (2022);
Lhoest et al. (2021); Paullada et al. (2021); Luccioni et al. (2021). This
trend is clear evidence of the urgency of finding a sustainable and data-
centric way to support the full benchmarking ecosystem, from end-to-end, in a
way that causes the least amount of friction for anyone who wants to use it.

In this paper, we introduce our answer to these issues: an easy-to-use, open
source system that integrates the creation of benchmark datasets for any task,
the selection of appropriate metrics, and the evaluation of models while
natively supporting revisions to the benchmark as models saturate the original
version. We share a unified library that enables these functionalities for the
Dynabench platform (Kiela et al., 2021).

⬇

context:

- name: context

type: string

placeholder: Enter context…

input:

- name: hypothesis

type: string

placeholder: Enter hypothesis…

- name: label

type: multiclass

labels:

- entailed

- neutral

- contradictory

as_goal_message: true

output:

- name: label

- name: probs

type: probs

reference_name: label

⬇

input:

- name: image

type: image

display_name: image

- name: labels

type: multilabel

labels:

- Bird

- Canoe

- Croissant

- Muffin

- Pizza

output:

- name: labels

Figure 1: Two example config files and the data collection and validation
interfaces they generate. Only config fields that impact the data collection
interfaces are shown (e.g. metrics for model ranking are not shown). The
Context and Input fields define the type of data that humans can enter. The
Output field defines what models will output, given the Context and the Input.
Crowdworkers are typically expected to provide the gold truth annotations for
a task. In this case, Output will contain some of the object names from Input
and Context. These gold truth annotations are removed from the Context and the
Input before they are sent to models to get a model-in-the-loop output.

(Top) The config implements a natural language inference task. The first image
is the collection interface, after a crowdworker submits their example and
gets a model-in-the-loop response. The second image is the validation
interface. For brevity, the metadata field in the config is omitted. This
field is used to define the UI components for additional information, such as
the “Explain why your example is correct…” input field.

(Bottom) The config implements an image labelling task. The first image is the
collection interface, before a crowdworker submits their example. The second
image is the validation interface with the same example.

##  2 Background

Dynabench was proposed as an open-source and community-driven platform to host
dynamic benchmarks. The existing Dynabench tasks avoid saturation by
leveraging crowdworkers who continually interact with state-of-the-art models.
Crowdworkers either write examples that fool existing models Nie et al.
(2020), or collaborate with generative models to increase example diversity
Bartolo et al. (2021b). Each task is administered by one or more task owners
from the research community who collect data, make the competition’s design
decisions, select metrics, and configure the task’s leaderboard.

Kiela et al. (2021) introduced Dynabench with four English language NLP tasks:
Natural Language Inference Nie et al. (2020), Extractive QA Bartolo et al.
(2020), Sentiment Analysis (Potts et al., 2020) and Hate Speech Detection
(Vidgen et al., 2021). In follow-up work, Ma et al. (2021) updated Dynabench
with additional leaderboard functionalities that allow task owners to upload
task-specific models which are evaluated on each of the task’s datasets, and
can subsequently be included in model-ensembles that crowdworkers interact
with. As the platform kept expanding, it became clear that Dynabench needed a
scalable and configurable system for adding new tasks.

A task is an essential concept in understanding our work. On Dynabench, a
distinct task is a particular relationship between inputs and
outputs.111Although, any user can set up a new task that is a duplicate of an
existing one, with a duplicate config file. Inputs and outputs are framed
within some pre-specified format. For example, Natural Language Inference is a
task on Dynabench. The input format is two strings and the output format is a
classification label. The relationship between the inputs and outputs is
defined by what humans would do when loosely instructed to treat the input
strings as a context (sometimes called the “premise”) and a hypothesis, and
return a label for whether they think the hypothesis is entailed by the
context. MNLI Williams et al. (2017), SNLI (Bowman et al., 2015), and ANLI
(Nie et al., 2020) can be viewed as different datasets that instantiate the
same task. Schlangen (2021) takes a similar view.

⬇

aggregation_metric:

type: dynascore

perf_metric:

type: squad_f1

reference_name: answer

delta_metrics:

- type: fairness

- type: robustness

Figure 2: An example of a task config next to the generated model leaderboard.
Only config fields that impact the leaderboard are shown. Throughput and
memory do not need to be in the config; they are computed by default.

##  3 Dynatask

Before the introduction of Dynatask, adding a new task required close
collaboration between task owners and the Dynabench team, and extensive
software contributions to the Dynabench codebase. This paper presents a system
that enables Dynabench to scale up to more tasks, including into multimodal
and multilingual domains, without such requirements. Now, a task owner can
create their own task page on Dynabench with a short task config file. The
config file is used to automatically generate crowdworker data collection
interfaces, as well as the model and dataset hosting/evaluating
infrastructure. The data collection interfaces and hosting overlay existing
services such as Amazon Mechanical Turk,222https://www.mturk.com which provide
a workforce and payment mechanisms, but do not provide crowdworker interfaces
for dynamic model-in-the-loop data collection or their corresponding backends.
In fact, local installations of Dynabench can be run on Mechanical Turk.
Overall, a Dynabench task owner can set up and host:

#### Crowdworker data collection:

Task owners can configure interfaces for data collection. Models-in-the-loop
can be optionally added, so crowdworkers can receive real-time model responses
from their data (Figure 1).

#### Crowdworker data validation:

Task owners can configure interfaces for crowdworkers to label collected
examples as correct or incorrect. (Figure 1).

#### Dynamic dataset metrics:

Metrics on the crowdworker data are computed, such as verified model error
rate (vMER) (Nie et al., 2020). Crowdworker example leaderboards are
displayed.

#### A train file leaderboard:

Task owners can enable users to upload training data files for the automatic
creation, training, and evaluation of models in our evaluation cloud.

#### A dynamic and interactive model leaderboard (Ma et al., 2021):

Task owners can configure a leaderboard, selecting from a variety of metrics
to determine model performance. Owners can also upload new datasets, which
triggers automatic evaluation for all of the user-uploaded models. Every
leaderboard model can be interacted with in real-time. See Figure 2 for an
example.

#### A model upload pipeline:

Once a new task goes live on Dynabench, our command line
tool333https://github.com/facebookresearch/dynalab allows anyone to create a
handler script and upload models by following a few command line instructions.
After models are uploaded, they are dockerized and deployed automatically.
Models can be viewed on the leaderboard and put in-the-loop with crowdworkers
for data collection.

###  3.1 Task Configuration

To become task owners, Dynabench users submit a short written proposal for
their task which requires approval by an administrator. We are still
developing procedures for how Dynabench accepts tasks; so far, we have reached
out to have a discussion with the proposer before accepting their proposal and
all non-spam proposals have been slated for acceptance. After approval, the
task owner submits a task config file, which can be written in minutes. Once
complete, the task is actively hosted on Dynabench; data collection, data
validation, model hosting, and model evaluation starts immediately. A complete
config file is the combination of a snippet in Figure 1 with that in Figure 2.

The task config is a YAML file which allows someone to encode the
specifications for their task—it can be viewed as a lightweight declarative
programming language. Task owners can specify:

The datatypes of the task’s inputs and outputs. There are a variety to choose
from, including String, String Selection, Multiclass, Multilabel,
Probabilities, and Image. The datatype definition enables Dynatask to
automatically construct the UIs for data collection, the dataset uploading and
downloading infrastructure, and the model uploading and hosting
infrastructure.

A variety of metrics to understand the task’s datasets and models. Several
metrics can currently be computed for the leaderboard: Macro F1, F1 for Visual
Question Answering, F1 for Question Answering, Accuracy, BLEU, robustness and
fairness (Ma et al., 2021), memory usage, and example throughput. Task owners
select or propose an aggregation metric, which combines results across
multiple datasets and metrics to arrive at a ranking for the leaderboard.
Currently, the only supported aggregation metric is the Dynascore (Ma et al.,
2021), which combines metrics across datasets based on microeconomic utility
Ethayarajh and Jurafsky (2020) of user provided weights. Metrics can also be
specified for model-in-the-loop data collection to judge whether a model’s
output matches that of a crowdworker (i.e., whether the model is “correct”).
Dynatask supports a variety of such metrics, including a string F1 threshold
(for outputs that are strings), exact match, and simply asking the crowdworker
whether the model was correct.

Other optional items, such as messages and instructions that appear in
crowdworker interfaces, and options for train-file leaderboards.

Figure 3: The task owner interface for ANLI.

###  3.2 Options After Task Configuration

#### Crowdworker Interfaces, Data Generation, and Data Evaluation:

Data collection interfaces are automatically hosted at dynabench.org. In order
for Dynabench to scale, task owners source and pay crowdworkers themselves. If
crowdworker management, compensation, and sourcing features are needed, an
owner can clone Dynabench and run it on Mechanical Turk by hosting the data
collection frontend using
Mephisto.444https://github.com/facebookresearch/Mephisto Task owners can
upload context data for crowdworkers to use and download data collected from
crowdworkers directly from the Dynabench web interface. Task owners can also
initiate new rounds of data collection where they are free to upload entirely
new contexts and models. As part of the data collection process, vMER, number
of total collected examples, and number of validated examples are computed.
Finally, task owners can alter instructions to crowdworkers at any time. They
can also specify whether crowdworkers should validate non-model fooling
examples, and provide a validation consensus threshold above which examples
are considered fully validated. Figure 3 shows an example of the interface
that task owners use to adjust settings.

#### Model Submission, Interaction, and Evaluation:

Task owners can decide whether their task accepts model submissions, they can
upload datasets for model evaluation, and they can download model evaluation
logs for any dataset and model. Task owners can optionally allow users to
download these logs to debug their models; see the example in Figure 4.

Figure 4: A cross-section from a model card. The task owner has enabled
downloading of model evaluation logs for each dataset via the buttons on the
right.

##  4 Decentralized Evaluation-As-A-Service

Most task owners currently use the centralized Dynabench evaluation and model
deployment server. With Dynatask, however, we offer a decentralized evaluation
feature that will increase the platform’s flexibility even further. With this
feature, task owners can set up a Dynabench model deployment and evaluation
server or select an existing one. To set up a new server, an owner only needs
to follow our documentation, creating an AWS account and installing some
Dynabench code along the way. Distributed hosting of model building and
evaluation enables Dynatask to scale: no one organization needs to fund
hosting for all of the models on Dynabench, and every owner of a model
deployment and evaluation server can flexibly upload or take down models to
suit their budget. It is also designed with re-usability in mind: several
tasks can share the same evaluation servers. Task owners do not need to do any
setup if they have permission to use an existing evaluation server.

##  5 Case Studies of Tasks Enabled so Far

Tables 1 and 2 provide an overview of Dynabench so far. In this section, we
report on some use cases. Most of the following projects (besides Image
Labelling and Open Domain QA) were added to Dynabench before the introduction
of Dynatask, which took months of coding in every case. With Dynatask, they
can all be implemented in minutes.

Statistic | Count
---|---
Datasets Hosted | 191
Unique Crowdworkers | 5,595
Model Uploads | 589
Data Collection Rounds | 38
Tasks (incl. private) | 24
Examples Collected | 559,229
Example Validations | 436,922
Table 1: Current Dynabench statistics. Selected Dynabench Tasks |  Context and Input Types |  Output Types
---|---|---
Hate Speech Detection https://dynabench.org/tasks/hs |  String, String, Multiclass |  Multiclass, Probs
Visual QA https://dynabench.org/tasks/vqa |  Image, String |  String
Extractive QA https://dynabench.org/tasks/qa |  String, String, String Select |  String Select, Probs
Open Domain QA https://dynabench.org/tasks/qb |  String, String, String |  String, Probs
Natural Language Inference https://dynabench.org/tasks/nli |  String, String, Multiclass |  Multiclass, Probs
Sentiment Analysis https://dynabench.org/tasks/sentiment |  String, String, Multiclass |  Multiclass, Probs
Machine Translation https://dynabench.org/tasks/flores |  String, String, String, String |  String
Image Labelling https://dynabench.org/tasks/vision-dataperf |  Image, Multilabel |  Multilabel
Table 2: IO types from the task config, for some tasks on Dynabench. Tasks
share the same building blocks.

Hate Speech Detection: There are a number of hate speech detection projects on
Dynabench, where a model must label strings as hateful or not. Groups from
Oxford, The Alan Turing Institute, The University of Sheffield, and Facebook
AI own task pages that focus on collecting adversarial data (Vidgen et al.,
2021), collecting emoji-based hate (Kirk et al., 2021), and evaluating models
on a large number of hate speech data perturbations.

Visual Question Answering: To combat saturating datasets for the VQA task,
which is about answering a question based on an image, Facebook AI and
Tecnológico de Monterrey introduced AdVQA (Sheng et al., 2021) using
Dynabench. The task’s model leaderboard has an additional adversarial VQA
dataset from Microsoft and Tsinghua (Li et al., 2021).

Extractive Question Answering: Groups from UCL and Facebook AI run SQuAD-style
(Rajpurkar et al., 2016) extractive QA projects on Dynabench. The Adversarial
QA (Bartolo et al., 2020) project resulted in a popular dataset on the Hugging
Face hub (Lhoest et al., 2021). Follow-up projects explored the generation of
synthetic adversarial QA data (Bartolo et al., 2021a), generative assistants
in the loop to help annotators create examples (Bartolo et al., 2021b), and a
study of how adversarial model-in-the-loop training data affects
generalization out of domain (Kaushik et al., 2021).

Open-Domain Question Answering: A team at Facebook AI and The University of
Maryland has started a model-in-the-loop data collection effort for the
Quizbowl task Rodriguez et al. (2019); Wallace et al. (2019), as well as a
model leaderboard. The task is open domain question answering, where both the
question and answer are strings.

Natural Language Inference: The NLI dataset ANLI Nie et al. (2020) is
currently a popular dataset on Hugging Face datasets Lhoest et al. (2021) and
an ongoing Dynabench project. Groups from Facebook AI, UC Berkeley, and UNC
have set up additional NLI projects on distinct Dynabench task pages. These
projects have ranged from an analysis of the contents of adversarially
collected development sets (Williams et al., 2022), to an explication of the
benefits of dynamic adversarial data collection over multiple rounds (Wallace
et al., 2021), to model and leaderboard hosting for a large number of
robustness-perturbed NLI datasets.

Sentiment Analysis: In later rounds of their work, a team at Stanford used
Dynabench to create a new adversarial sentiment analysis dataset, called
Dynasent (Potts et al., 2020). They added prompts to their data collection
interfaces to encourage crowdworkers to generate naturalistic and diverse
data.

Large-Scale Machine Translation: The Workshop on Machine Translation (Wenzek
et al., 2021) organizers created a Dynabench task page and hosted the FLORES
benchmark competition (Goyal et al., 2021) of over 10,000 language pairs. It
featured competitors from Microsoft, Huawei, Tencent, and Facebook, and
individual competitors. The result of the competition was a BLEU increase of
over 10 points on the full task. The owners used Dynabench for its
leaderboard, model upload, and evaluation-as-a-service feature, without
collecting data on the platform yet.

Image Labelling: DataPerf Mattson et al. (2022) is a working group of the non-
profit ML Commons, which focuses on dataset benchmarking for general AI. For
their image labelling task hosted on Dynabench, they configured their task via
the task config to accept training data file uploads. Users upload train files
and models are automatically trained against them and evaluated in the
evaluation cloud.

##  6 Conclusion

We introduced Dynatask, a collection of open source features in the Dynabench
platform that empowers anyone to create and own a task on Dynabench with only
a short config file. Dynabench started as an NLP project with only four
English-only tasks. Since then, Dynatask has helped researchers produce
several datasets and host competitions, expanding scalably into multimodal and
multilingual domains with owners from various corners of the AI community.
Dynatask offers the functionalities of Dynabench to the broader research
community by allowing them to easily create and host new AI tasks on the
platform: it provides a one-stop shop for constructing datasets with or
without models in the loop, hosting challenges and competitions, investigating
the effects of models in the loop, characterizing distributional shift and
continual learning, exploring annotator efficiency and expertise, and
improving model robustness through collaboration with humans.

Finally, Dynabench is an open source, community-driven effort. Anyone who
wants to add a new input/output type, a new metric, or any other new feature,
need only submit a pull request. We hope that our our work can help enable new
exciting scientific progress in data-centric AI research in general and
dynamic (adversarial) data collection in particular.

## References

  * Bartolo et al. (2020) Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. 2020.  Beat the AI: Investigating adversarial human annotation for reading comprehension.  _TACL_.
  * Bartolo et al. (2021a) Max Bartolo, Tristan Thrush, Robin Jia, Sebastian Riedel, Pontus Stenetorp, and Douwe Kiela. 2021a.  Improving question answering model robustness with synthetic adversarial data generation.  In _EMNLP_.
  * Bartolo et al. (2021b) Max Bartolo, Tristan Thrush, Sebastian Riedel, Pontus Stenetorp, Robin Jia, and Douwe Kiela. 2021b.  Models in the loop: Aiding crowdworkers with generative annotation assistants.  In _arXiv preprint arXiv:2112.09062_.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _EMNLP_.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009.  Imagenet: A large-scale hierarchical image database.  In _CVPR_.
  * Ethayarajh and Jurafsky (2020) Kawin Ethayarajh and Dan Jurafsky. 2020.  Utility is in the eye of the user: A critique of NLP leaderboards.  In _EMNLP_.
  * Godfrey et al. (1992) John Godfrey, Edward Holliman, and Jane McDaniel. 1992.  Switchboard: telephone speech corpus for research and development.  In _ICASSP_.
  * Goyal et al. (2021) Naman Goyal, Cynthia Gao, Vishrav Chaudhary, PengJen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzman, and Angela Fan. 2021.  The flores-101 evaluation benchmark for low-resource and multilingual machine translation.  In _arXiv preprint arXiv:2106.03193_.
  * Halevy et al. (2009) Alon Halevy, Peter Norvig, and Fernando Pereira. 2009.  The unreasonable effectiveness of data.  _IEEE Intelligent Systems_.
  * Kaushik et al. (2021) Divyansh Kaushik, Douwe Kiela, Zachary C Lipton, and Wen tau Yih. 2021.  On the efficacy of adversarial data collection for question answering: Results from a large-scale randomized study.  In _ACL-IJCNLP_.
  * Kiela et al. (2021) Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021.  Dynabench: Rethinking Benchmarking in NLP.  In _NAACL_.
  * Kirk et al. (2021) Hannah Rose Kirk, Bertram Vidgen, Paul Röttger, Tristan Thrush, and Scott A Hale. 2021.  Hatemoji: A test suite and adversarially-generated dataset for benchmarking and detecting emoji-based hate.  In _arXiv preprint arXiv:2108.05921_.
  * LeCun et al. (1998) Yann LeCun, Corinna Cortes, and Christopher Burges. 1998.  The mnist database of handwritten digits.
  * Lhoest et al. (2021) Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021.  Datasets: A community library for natural language processing.  In _EMNLP: System Demonstrations_.
  * Li et al. (2021) Linjie Li, Jie Lei, Zhe Gan, and Jingjing Liu. 2021.  Adversarial VQA: A new benchmark for evaluating the robustness of VQA models.  In _ICCV_.
  * Luccioni et al. (2021) Sasha Luccioni, Yacine Jernite, and Meg Mitchell. 2021.  Hugging face data measurements tool.
  * Ma et al. (2021) Zhiyi Ma, Kawin Ethayarajh, Tristan Thrush, Somya Jain, Ledell Wu, Robin Jia, Christopher Potts, Adina Williams, and Douwe Kiela. 2021.  Dynaboard: An evaluation-as-a-service platform for holistic next-generation benchmarking.  In _NeurIPS_.
  * Mattson et al. (2022) Peter Mattson, Cody Coleman, Ce Zhang, Praveen Paritosh, Vijay Janapa Reddi, Douwe Kiela, Greg Diamos, Carole-Jean Wu, Sabri Eyuboglu, Joaquin Vanschoren, William Gaviria Rojas, Tariq Kane, Bojan Karlas, Lynn He, Lora Aroyo, Colby Banbury, Mark Mazumder, Bilge Acun, Newsha Ardalani, Tristan Thrush, Adina Williams, Amirata Ghorbani, Emmett Goodman, and Serena Yeung. 2022.  Dataperf: Benchmarking data for better ml.  In _Forthcoming_.
  * Ng et al. (2021) Andrew Ng, Lora Aroyo, Greg Diamos, Cody Coleman, Vijay Janapa Reddi, Joaquin Vanschoren, Carole-Jean Wu, Sharon Zhou, and Lynn He. 2021.  [link].
  * Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020.  Adversarial NLI: A new benchmark for natural language understanding.  In _ACL_.
  * Paullada et al. (2021) Amandalynne Paullada, Inioluwa Deborah Raji, Emily M Bender, Emily Denton, and Alex Hanna. 2021.  Data and its (dis) contents: A survey of dataset development and use in machine learning research.  _Patterns_.
  * Potts et al. (2020) Christopher Potts, Zhengxuan Wu, Atticus Geiger, and Douwe Kiela. 2020.  DynaSent: A dynamic benchmark for sentiment analysis.  _arXiv preprint arXiv:2012.15349_.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _EMNLP_.
  * Rodriguez et al. (2019) Pedro Rodriguez, Shi Feng, Mohit Iyyer, He He, and Jordan Boyd-Graber. 2019.  Quizbowl: The case for incremental question answering.  In _arXiv preprint arXiv:1904.04792_.
  * Sambasivan et al. (2021) Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M. Aroyo. 2021.  “everyone wants to do the model work, not the data work”: Data cascades in high-stakes AI.  In _CHI Conference on Human Factors in Computing Systems_.
  * Schlangen (2021) David Schlangen. 2021.  Targeting the benchmark: On methodology in current natural language processing research.  In _ACL-IJCNLP: Short Papers_.
  * Sheng et al. (2021) Sasha Sheng, Amanpreet Singh, Vedanuj Goswami, Jose Alberto Lopez Magana, Tristan Thrush, Wojciech Galuba, Devi Parikh, and Douwe Kiela. 2021.  Human-adversarial visual question answering.  In _NeurIPS_.
  * Vidgen et al. (2021) Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021.  Learning from the worst: Dynamically generated datasets to improve online hate detection.  In _ACL_.
  * Wagstaff (2012) Kiri Wagstaff. 2012.  Machine learning that matters.  In _ICML_.
  * Wallace et al. (2019) Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber. 2019\.  Trick me if you can: Human-in-the-loop generation of adversarial examples for question answering.  In _TACL_.
  * Wallace et al. (2021) Eric Wallace, Adina Williams, Robin Jia, and Douwe Kiela. 2021.  Analyzing dynamic adversarial training data in the limit.  In _arXiv preprint arXiv:2110.08514_.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  Superglue: A stickier benchmark for general-purpose language understanding systems.  In _NeurIPS_.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  In _BlackboxNLP at EMNLP_.
  * Wenzek et al. (2021) Guillaume Wenzek, Vishrav Chaudhary, Angela Fan, Sahir Gomez, Naman Goyal, Somya Jain, Douwe Kiela, Tristan Thrush, and Francisco Guzmán. 2021.  Findings of the wmt 2021 shared task on large-scale multilingual machine translation.  In _WMT_.
  * Williams et al. (2017) Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017.  A broad-coverage challenge corpus for sentence understanding through inference.  In _arXiv preprint arXiv:1704.05426_.
  * Williams et al. (2022) Adina Williams, Tristan Thrush, and Douwe Kiela. 2022.  ANLIzing the adversarial natural language inference dataset.  In _SCiL_.