# SSB: Simple but Strong Baseline for Boosting Performance of
Open-Set Semi-Supervised Learning

Yue Fan Anna Kukleva Dengxin Dai Bernt Schiele
{yfan, akukleva, ddai<EMAIL_ADDRESS>
Max Planck Institute for Informatics, Saarbrücken, Germany
Saarland Informatics Campus

###### Abstract

Semi-supervised learning (SSL) methods effectively leverage unlabeled data to
improve model generalization. However, SSL models often underperform in open-
set scenarios, where unlabeled data contain outliers from novel categories
that do not appear in the labeled set. In this paper, we study the challenging
and realistic open-set SSL setting, where the goal is to both correctly
classify inliers and to detect outliers. Intuitively, the inlier classifier
should be trained on inlier data only. However, we find that inlier
classification performance can be largely improved by incorporating high-
confidence pseudo-labeled data, regardless of whether they are inliers or
outliers. Also, we propose to utilize non-linear transformations to separate
the features used for inlier classification and outlier detection in the
multi-task learning framework, preventing adverse effects between them.
Additionally, we introduce pseudo-negative mining, which further boosts
outlier detection performance. The three ingredients lead to what we call
Simple but Strong Baseline (SSB) for open-set SSL. In experiments, SSB greatly
improves both inlier classification and outlier detection performance,
outperforming existing methods by a large margin. Our code will be released at
https://github.com/YUE-FAN/SSB.

##  1 Introduction

Semi-supervised learning (SSL) has achieved great success in improving model
performance by leveraging unlabeled data [29, 28, 45, 32, 4, 3, 44, 48, 13,
53, 47]. However, standard SSL assumes that the unlabeled samples come from
the same set of categories as the labeled samples, which makes them struggle
in open-set settings [36], where unlabeled data contain out-of-distribution
(OOD) samples from novel classes that do not appear in the labeled set (see
Fig. 1). In this paper, we study this more realistic setting called open-set
semi-supervised learning, where the goal is to learn both a good closed-set
classifier to classify inliers and to detect outliers as shown in Fig. 1.

Figure 1:  Open-set semi-supervised learning considers a realistic and
challenging setting, where unlabeled data contains samples from novel classes
(seen outliers) that do not appear in the labeled data. At test time, the
model should correctly classify inliers, while identifying outliers seen
during the training and, most importantly, unseen outliers that do not appear
in the training set. We measure test accuracy for the inlier classification
performance and AUROC for the outlier detection performance. Our method (SSB)
achieves superior performance in both tasks.

Recent works on open-set SSL [22, 8, 41, 51, 16, 18, 19, 23] have achieved
strong performance [46, 31, 21, 1] through a multi-task learning framework,
which consists of an inlier classifier, an outlier detector, and a shared
feature encoder, as shown in Figure 2. The outlier detector is trained to
filter out OOD data from the unlabeled data so that the classifier is only
trained on inliers. However, this framework has two major drawbacks. First,
detector-based filtering often removes many inliers along with OOD data,
leading to suboptimal classification performance due to the low utilization
ratio of unlabeled data. Second, the inlier classifier which shares the same
feature encoder with the outlier detector can have an adverse effect on the
detection performance as shown in Table 1.

To this end, we contribute a Simple but Strong Baseline, SSB, for open-set SSL
with three ingredients to address the above issues. (1) In contrast to
detector-based filtering aiming to remove OOD data, we propose to incorporate
pseudo-labels with high inlier classifier confidence into the training,
irrespective of whether a sample is an inlier or OOD. This not only
effectively improves the unlabeled data utilization ratio but also includes
many useful OOD data that can be seen as natural data augmentations of inliers
(see Fig. 5). (2) Instead of directly sharing features between the classifier
and detector, we add non-linear transformations for the task-specific heads
and find that this effectively reduces mutual interference between them,
resulting in more specialized features and improved performance for both
tasks. (3) In addition, we propose pseudo-negative mining to further improve
outlier detector training by enhancing the data diversity of OOD data with
pseudo-outliers. Despite its simplicity, SSB achieves significant improvements
in both inlier classification and OOD detection. As shown in Fig. 1, existing
methods either struggle in detecting outliers or have difficulties with inlier
classification while SSB obtains good performance for both tasks.

##  2 Related Work

Semi-supervised learning. Semi-supervised learning (SSL) aims to improve model
performance by exploiting both labeled and unlabeled data. As one of the most
widely used techniques, pseudo-labeling [29] is adopted by many strong SSL
methods [44, 4, 4, 48, 53, 2, 38, 49, 5, 30]. The idea is to generate
artificial labels for unlabeled data to improve model training. [4, 3] compute
soft pseudo-labels and then apply MixUp [54] with labeled data to improve the
performance; [44, 48, 53] achieves good performance by combining pseudo-
labeling with consistency regularization [28, 32, 45]; [38] proposes a meta
learning approach that uses a teacher model to refine the pseudo-labels based
on the training of a student model; [49] leverages the idea of self-training
which generates pseudo labels in an iterative way and inject noise to each
training stage. In this paper, we also adopt a simple confidence-based pseudo-
labeling [44] for classifier training, which is an effective way of leveraging
unlabeled data to improve the model performance. Compared to standard SSL, SSB
has an additional outlier detector, which enables the model to reject samples
that do not belong to any of the inlier classes.

Open-set SSL & Class-mismatched SSL. First shown by [36], standard SSL methods
suffer from performance degradation when there are out-of-distribution (OOD)
samples in unlabeled data. Since then, various approaches have been proposed
to address this challenge [8, 16, 51, 41, 22, 37, 18, 19, 23]. Existing
methods seek to alleviate the effect of OOD data by filtering them out in
different ways so that the classification model is trained with inliers only.
For example, [8] uses model ensemble [43] to compute soft pseudo-labels and
performs filtering with a confidence threshold; [16] proposes a bi-level
optimization to weaken the loss weights for OOD data; [51] assigns an OOD
score to each unlabeled data and refines it during the training; [41]
leverages one-vs-all (OVA) classifiers [42] for OOD detection and propose a
consistency loss to train them; [22] proposes a cross-modal matching module to
detector outliers. [23] employs adversarial domain adaptation to filter
unlabeled data and find recyclable OOD data to improve the performance; [18]
uses energy-discrepancy to identify inliers and outliers. In contrast, we show
that if the representations of the inlier classifier and the outlier detector
are well-separated, OOD data turns out to be a powerful source to improve the
inlier classification without degrading the detection performance. So, instead
of filtering OOD data, we use a simple confidence-based pseudo-labeling to
incorporate them into the training.

Open-world SSL. Open-set SSL is similar to open-world SSL [6, 39, 40] but
bears several important differences. While both have unlabeled data of novel
classes during the training, the goal of open-world SSL is to classify inliers
and discover new classes from OOD data instead of rejecting them. Another
important difference is that open-world SSL is often a transductive learning
setting while open-set SSL requires generalization beyond the current
distribution. Namely, the model should be able to detect OOD data from novel
classes that present in the training set as well as OOD data from classes that
are never seen during training.

##  3 SSB: Simple but Strong Baseline for Open-Set Semi-Supervised Learning

In this section, we first present the problem setup of open-set semi-
supervised learning (SSL). Then, we give an overview of our method SSB in
Section 3.1 before presenting details of the three simple yet effective
ingredients used in our method in Section 3.2, 3.3, and 3.4.

Figure 2:  Left: Our baseline for open-set SSL consists of an inlier
classifier $g_{c}$, an outlier detector $g_{d}$, and a shared feature encoder
$f$ whose features are separated from the task-specific heads by two
projection heads $h_{c}$ and $h_{d}$. Unlike the detector-based filtering, we
adopt confidence-based pseudo-labeling by the inlier classifier to leverage
useful OOD data for classifier training. For detector training, we train one-
vs-all (OVA) classifiers as in OpenMatch [41]. Right: Given the inlier scores
($s_{1}$ to $s_{4}$), pseudo-negative mining selects confident negatives
($s_{2}$ and $s_{3}$ in the figure), whose inlier scores are lower than a pre-
defined threshold, as pseudo-outliers to help the outlier detector training.

Problem setup and notations: As shown in Fig. 1, open-set SSL generalizes the
settings of standard SSL and out-of-distribution (OOD) detection. It considers
three disjoint sets of classes: $\mathcal{C}$ corresponds to the inlier
classes that are partially annotated, $\mathcal{U_{S}}$ contains the outlier
classes seen during training but without annotations, and lastly,
$\mathcal{U_{U}}$ is composed of the classes that are not seen during training
(only seen at test time). The training data contains a small labeled set
$\mathcal{D}_{\text{labeled}}=\\{(\textbf{x}_{i}^{l},y_{i})\\}_{i=1}^{N}\subset\mathcal{X}\times\mathcal{C}$
and a large unlabeled set
$\mathcal{D}_{\text{unlabeled}}=\\{(\textbf{x}_{i}^{u})\\}_{i=1}^{M}\subset\mathcal{X}$,
where $\mathcal{X}$ is the input space. While the labeled set only consists of
samples of inlier classes, the unlabeled set contains both samples from
$\mathcal{C}$ and $\mathcal{U_{S}}$. Thus, the the ground-truth label of
$\textbf{x}^{u}$ is from $\mathcal{C}\cup\mathcal{U_{S}}$ with
$\mathcal{C}\cap\mathcal{U_{S}}=\emptyset$.

The goal of open-set SSL is to train a model that can perform good inlier
classification as well as detecting both seen and unseen outliers. Without
loss of generality, consider a test set
$\mathcal{D}_{\text{test}}=\\{(\textbf{x}_{i},y_{i})\\}_{i=1}^{N}\subset\mathcal{X}\times(\mathcal{C}\cup\mathcal{U_{S}}\cup\mathcal{U_{U}})$,
where $\mathcal{C}\cap\mathcal{U_{U}}=\emptyset$ and
$\mathcal{U_{S}}\cap\mathcal{U_{U}}=\emptyset$. The learned model should be
able to correctly classify inliers
$\\{(\textbf{x}_{i}|y_{i}\in\mathcal{C})\\}$ and detect outliers from
$\\{(\textbf{x}_{i}|y_{i}\in\mathcal{U_{S}})\\}$ as well as
$\\{(\textbf{x}_{i}|y_{i}\in\mathcal{U_{U}})\\}$, which is crucial for
practical applications.

###  3.1 Method Overview

Following [22, 8, 41, 51, 16, 18, 19, 23], we adopt a multi-task learning
framework for open-set SSL, which performs inlier classification and outlier
detection. As shown in Fig. 2, SSB comprises four components: (1) An inlier
classifier $g_{c}$, (2) an outlier detector $g_{d}$, (3) a shared feature
encoder $f$, and (4) importantly, two projection heads $h_{c}$ and $h_{d}$.
Inspired by [41], the outlier detector $g_{d}$ consists of $|\mathcal{C}|$
one-vs-all (OVA) binary classifiers, each of which is trained to distinguish
inliers from outliers for each single class. Given a batch of labeled data
$\textbf{X}^{l}=\\{(\textbf{x}_{i}^{l},y_{i})\\}_{i=1}^{B_{l}}$ and unlabeled
data $\textbf{X}^{u}=\\{(\textbf{x}_{i}^{u})\\}_{i=1}^{B_{u}}$, the total loss
for training the model is:

$L_{total}=L_{cls}(\textbf{X}^{l},\textbf{X}^{u};f,h_{c},g_{c})+L_{det}(\textbf{X}^{l},\textbf{X}^{u};f,h_{d},g_{d})$
(1)

where $L_{cls}$ and $L_{det}$ are the classification and detection losses,
respectively. For the sake of brevity, we will drop the dependencies of the
loss function on $f$, $h_{c}$, $g_{c}$, $h_{d}$, and $g_{d}$ in the following.
The complete algorithm of SSB is summarized by Alg. 1 in Appendix D.

During inference, the test image is first fed to the inlier classifier to
compute the class prediction. Then, the corresponding detector is used to
decide whether it is an inlier of the predicted class or an outlier. We
explain the details of SSB in the following three sections.

###  3.2 Boosting Inlier Classification with Classifier Pseudo-Labeling

Existing methods for open-set SSL [22, 8, 41, 51, 16] aim to eliminate OOD
data from the classifier training. This is typically accomplished by training
outlier detectors that can filter out OOD data from unlabeled data, as shown
in Fig. 2. However, as we will see in Table 3, detector-based filtering often
removes many inliers along with OOD data, which leads to a low utilization
ratio of unlabeled data and hinders inlier classification performance.

In this work, instead of using detector-based filtering, we propose to
incorporate unlabeled data with confident pseudo-labels (as generated by the
inlier classifier) into the training, irrespective of whether it is inlier or
OOD data. This not only effectively improves the unlabeled data utilization
ratio but also includes many useful OOD data as natural data augmentations of
inliers into the training (see Fig. 5). Inspired by [44], we train the model
with pseudo-labels from the inlier classifier whose confidence scores are
above a pre-defined threshold. Specifically, for each unlabeled sample
$\textbf{x}_{i}^{u}$, we first predict the pseudo-label distribution as
$\hat{p}_{i}^{u}=\text{softmax}(h_{c}(g_{c}(f(\textbf{x}_{i}^{u})))$. Then,
the confidence score of the pseudo-label is computed as $\max\hat{p}_{i}^{u}$.
Finally, the cross-entropy loss is calculated for samples whose pseudo-labels
have confidence scores greater than a pre-defined threshold $\tau$ as:

$L_{cls}^{u}(\textbf{X}^{u})=\frac{1}{B_{u}}\sum_{i=1}^{B_{u}}\text{1}(\max\hat{p}_{i}^{u}\geq\tau)H(\hat{p}_{i}^{u},\hat{y}^{u}_{i})$
(2)

where $H(\cdot,\cdot)$ denotes the cross-entropy,
$\hat{y}^{u}_{i}=\text{argmax}\hat{p}_{i}^{u}$, and $\text{1}(\cdot)$ is the
indicator function which outputs 1 when the confidence score is above the
threshold $\tau$.

The total classification loss is computed as the summation of a labeled data
loss and the unlabeled data loss as:

$L_{cls}(\textbf{X}^{l},\textbf{X}^{u})=L_{cls}^{l}(\textbf{X}^{l})+L_{cls}^{u}(\textbf{X}^{u})$
(3)

where $L_{cls}^{l}$ is a standard cross-entropy loss for labeled data.

Despite its simplicity, we obtain a substantial performance improvement in
inlier classification through classifier confidence-based pseudo-labeling as
shown in Table 1. Our method is conceptually different from previous methods
as we aim to leverage OOD data rather than remove them. On the one hand, our
method effectively improves the unlabeled data utilization ratio as shown in
Table 3, which leads to great inlier classification performance improvement.
On the other hand, our method provides an effective way of leveraging useful
OOD data for classifier training. In fact, many OOD data are natural data
augmentations of inliers and are beneficial for classification performance if
used carefully. As shown in Fig. 5, the selected OOD data present large visual
similarities with samples of inlier classes, and, thus, significantly enhance
the data diversity, leading to improved generalization performance.

###  3.3 Non-Linear Feature Boosting

In previous methods, simply including OOD samples into the classifier training
harms detection performance since the inlier classifier and the outlier
detector use the same feature representation [41, 51, 22]. On the one hand,
the classifier uses OOD data as pseudo-inliers, thus mixing their
representations in the feature space. On the other hand, the outlier detector
is trained to distinguish inliers and outliers, which leads to separated
representations in the feature space. As a result, the contradiction between
the classifier and the outlier detector ultimately adversely affects each
other, which limits the overall performance, as shown in Table 1.

In this work, we find empirically that simply adding non-linear
transformations between the task-specific heads and the shared feature encoder
can effectively mitigate the adverse effect. Given a sample $\textbf{x}_{i}$,
two multi-layer perceptron (MLP) projection heads $h_{c}$ and $h_{d}$ are used
to transform the features from the encoder. The output of the network is thus
$h_{c}(g_{c}(f(\textbf{x}_{i})))$ for the classifier and
$h_{d}(g_{d}(f(\textbf{x}_{i})))$ for the outlier detector. Compared to the
previous methods, the non-linear transformations effectively prevent mutual
interference between the classifier and detector, resulting in more
specialized features and improved performance in both tasks. In Table 1, while
the OOD detection performance degenerates when adding OOD data for classifier
training for the model without the projection heads, SSB, in contrast, still
exhibits excellent performance in detecting outliers with the help of the
projection heads. Moreover, the efficacy of the non-linear projection head
also generalizes to other frameworks. We show in the experiment section that
it is compatible with various SSL backbones and open-set SSL methods and leads
to performance improvement.

###  3.4 Outlier Detection with Pseudo-Negative Mining

In this section, we first describe the outlier detector used in SSB and then
introduce a simple yet effective technique called pseudo-negative mining to
improve the outlier detector training.

Following [41], we adopt $|\mathcal{C}|$ one-vs-all (OVA) binary classifiers
for OOD detection, where each OVA classifier is trained to distinguish between
inliers and outliers for each individual inlier class. Given a labeled sample
$\textbf{x}_{i}^{l}$ from class $y_{i}$, it is regarded as an inlier for class
$y_{i}$ and an outlier for class $k,k\neq y_{i}$. Therefore, the OVA
classifiers can be trained using binary cross-entropy loss on the positive-
negative pairs constructed from the labeled set as:

$L_{det}^{l}(\textbf{X}^{l})=-\frac{1}{B_{l}}\sum_{i=1}^{B_{l}}log(p_{y_{i}}(\textbf{x}_{i}^{l}))+\frac{1}{K}\sum_{k\neq
y_{i}}log(1-p_{k}(\textbf{x}_{i}^{l}))$ (4)

where $p_{k}(\textbf{x}_{i}^{l})$ is the inlier score of $\textbf{x}_{i}^{l}$
for class $k$ computed by the $k$-th detector and $K=|\mathcal{C}|-1$.

However, due to data scarcity, it is difficult to learn good representations
for outliers with labeled data only. To this end, we propose pseudo-negative
mining to further improve the outlier detector training by leveraging
confident negatives as pseudo-outliers to enhance the data diversity of OOD
data. As shown in Fig. 2, given an unlabeled sample $\textbf{x}_{i}^{u}$, we
consider it as a pseudo-outlier for class $k$ if the inlier score for class
$k$ is lower than a pre-defined threshold. Then, $\textbf{x}_{i}^{u}$ is used
as a negative sample to calculate the cross-entropy loss of class $k$. The
final loss for $\textbf{x}_{i}^{u}$ is the summation over all classes using it
as the negative sample:

$L_{det}^{u}(\textbf{x}_{i}^{u})=-\frac{1}{\sum_{k}{\text{1}(p_{k}<\theta)}}\sum_{k=1}^{|\mathcal{C}|}\text{1}(p_{k}<\theta)log(1-p_{k}(\textbf{x}_{i}^{u}))$
(5)

where $p_{k}$ is the inlier score from the $k$-th detector and
$\text{1}(\cdot)$ is the indicator function which outputs 1 when the
confidence score is less than the threshold $\theta$. This increases the data
diversity of outliers and improves generalization performance as shown in
Table 5. Compared to standard pseudo-labels, pseudo-outliers have much higher
precision because we specify which classes the sample does not belong to
rather than which class it belongs to. The latter is a more difficult task
than the former. Therefore, pseudo-negative mining is less susceptible to
inaccurate predictions while increasing data utilization.

Our final loss for detector training also includes Open-set Consistency (OC)
loss [41] and entropy minimization (EM) [14] because they can lead to further
improvement. The overall loss for training the detector is as follows:

$L_{det}(\textbf{X}^{l},\textbf{X}^{u})=L_{det}^{l}(\textbf{X}^{l})+\lambda_{det}^{u}L_{det}^{u}(\textbf{X}^{u})\\\
+\lambda_{OC}^{u}L_{OC}^{u}(\textbf{X}^{u})+\lambda_{em}^{u}L_{em}^{u}(\textbf{X}^{u})$
(6)

where $\lambda_{det}^{u}$, $\lambda_{OC}^{u}$, and $\lambda_{em}^{u}$ are loss
weights; $L_{OC}^{u}$ is the soft open-set consistency regularization loss,
which enhances the smoothness of the OVA classifier with respect to input
transformations; $L_{em}^{u}$ is the entropy minimization loss, which
encourages more confident predictions.

##  4 Experiments

In this section, we first compare SSB with existing methods in Section 4.1,
and then provide an ablation study and further analysis in Section 4.2.

Figure 3:  Classification and detection performance on CIFAR-10 and CIFAR-100
with varying numbers of inlier classes and labeled data. We measure test
accuracy for the inliers classification performance and AUROC for the outlier
detection performance. While standard SSL methods suffer in outlier detection
and open-set SSL methods suffer in inlier classification, SSB achieves good
performance in both tasks. Noted that the reported outlier detection
performance is the average AUROC in detecting both seen and unseen outliers.
Please see Appendix A for a detailed breakdown of the results in tables and
results on more benchmarks.  Figure 4:  Classification performance versus the
outlier detection performance on ImageNet-30.  SSB achieves good performance
in both inlier classification and OOD detection. Please see Appendix A for a
detailed breakdown of the results in tables.

###  4.1 Main Results

Datasets & Evaluation. As mentioned in Section 3, the goal of open-set SSL is
to train a good inlier classifier as well as an outlier detector that can
identify both seen and unseen outliers. Therefore, we need to construct three
class spaces: inlier classes $\mathcal{C}$, seen outlier classes
$\mathcal{U_{S}}$, and unseen outlier classes $\mathcal{U_{U}}$. For each
setting: the labeled set contains samples from $\mathcal{C}$ only; the
unlabeled set contains samples from $\mathcal{C}$ and $\mathcal{U_{S}}$; the
test set contains samples from $\mathcal{C}$, $\mathcal{U_{S}}$, and
$\mathcal{U_{U}}$. The inlier classification performance is evaluated on
$\mathcal{C}$ using test accuracy as in standard supervised learning. The OOD
detection performance is measured by AUROC following [41] and we report the
average performance in detecting seen outliers and unseen outliers (see
Appendix A for separate AUROC on seen outliers and unseen outliers).

Following [41], we evaluate SSB on CIFAR-10 [27], CIFAR-100 [27], and ImageNet
[11] with different numbers of labeled data. For CIFAR-10, the 6 animal
classes are used as inlier classes, and the rest 4 are used as seen outlier
classes during the training. Additionally, test sets from SVHN [34],
CIFAR-100, LSUN [50], and ImageNet are considered as unseen outliers, and used
to evaluate the detection performance on unseen outliers. For CIFAR-100, the
inlier-outlier split is performed on super classes, and two settings are
considered: 80 inlier classes (20 outlier classes) and 55 inlier classes (45
outlier classes). Similar to CIFAR-10, test sets from SVHN, CIFAR-10, LSUN,
and ImageNet are used to evaluate the detection performance on unseen
outliers. For ImageNet, we follow [41] to use ImageNet-30[20], which is a
subset of ImageNet containing 30 distinctive classes. The first 20 classes are
used as inlier classes while the rest 10 are used as outlier classes. Stanford
Dogs [25], CUB-200 [10], Flowers102 [35], Caltech-256 [15], Describable
Textures Dataset [9], LSUN are used as unseen outlier classes at test time.

Implementation details. We use Wide ResNet-28-2 [52] as the backbone for CIFAR
experiments and ResNet-18 [17] for ImageNet experiments. As standard SSL
models do not have the notion of OOD detection, we adopt the method in [20],
where the OOD score of an input image x is computed as
$1-\max\text{softmax}(f(\textbf{x}))$ and $f$ denotes the model. Thus, the
input image is considered as an outlier if the OOD score is higher than a pre-
defined threshold. For other open-set SSL methods, we directly employ the
authors’ implementations and follow their default hyper-parameters.

For SSB, we use two two-layer MLPs with ReLU [33] non-linearity to separate
representations for all settings. The hidden dimension is 1024 for CIFAR
settings and 4096 for ImageNet settings. For classifier training, we follow
[44] and set the threshold $\tau$ as 0.95. For outlier detector training, we
set $\lambda_{det}^{u}$ as 1 for all settings and follow [41] for the weights
of OC loss and entropy minimization. The threshold $\theta$ is 0.01 for all
experiments (see ablation in Appendix C). Following [41], we train our model
for 512 epochs with SGD [26] optimizer. The learning rate is set as 0.03 with
a cosine decay. The batch size is 64. Additionally, we defer the training of
the outlier detector until epoch 475 to reduce the computational cost as we
find empirically the deferred training does not comprise the model
performance. The ablation on the deferred training is in Appendix C.

When combined with standard SSL methods (e.g. SSB + FlexMatch), we replace the
classifier training losses in Equation 1 with the corresponding losses of
different methods while keeping the outlier detector the same. When combined
with open-set SSL methods (e.g. MTC + SSB), we make three modifications.
First, we separate the outlier detector branch from the classifier branch
using the proposed MLP projection head. Second, we replace the outlier
detector training losses with our loss from Equation 6. Third, we do not
filter unlabeled data with the outlier detector for classifier training.

Results. We compare SSB with both standard SSL and open-set SSL methods. Fig.
3 and 4 summarize the inlier test accuracy and outlier AUROC for CIFAR
datasets and ImageNet, respectively. Considering the goal of open-set SSL is
to achieve both good inlier classification accuracy and outlier detection, SSB
greatly outperforms standard SSL methods in outlier detection, and open-set
SSL methods in inlier classification. For example, on CIFAR-10 with 25 labels,
the AUROC of our best method is 11.97% higher than the best method excluding
ours. Moreover, when combined with standard SSL algorithms, our method
demonstrates consistent improvement in OOD detection, and in most cases,
better test accuracy for inlier classification. This suggests the flexibility
of our method, which makes it possible to benefit from the most advanced
approaches. Note that the performance improvement of SSB can not be simply
explained by the increased number of parameters introduced in the projection
heads. Please see Fig. 7 for a comparison between SSB and other methods + MLP
heads.

Additionally, SSB is more robust to the number of labeled data than others. We
achieve reasonable performance given a small number of labeled data while
other methods fail to generalize. For example, on CIFAR-10 with 6 inlier
classes, OpenMatch has similar inlier accuracy as ours at 50 labels. When the
number of labeled data is halved, their performance decreases to 54.88% while
our method still has a test accuracy of 91.74%. Please see Appendix B for
comparisons on more benchmarks.

###  4.2 Ablation Study

In this section, we analyze the design choices of SSB and show their
importance through ablation experiments. If not specified, we use CIFAR-10
with 25 labeled data as our default setting for ablation. The same data split
is used for fair comparison.

Importance of non-linear projection heads. As mentioned in Section 3, we use
2-layer MLPs to mitigate the adverse effect between the inlier classifier and
outlier detector. Here we study the effect of the projection heads in Table 1.
As we can see, incorporating confidence filtering yields a significant
improvement in inlier classification performance (resulting in a 12.23% to
13.18% increase). However, the OOD detection performance experiences a
substantial decline when the projection heads are missing (AUROC from 89.67%
to 63.46%). This is because the classifier tends to mix the features of
inliers and outliers with the same pseudo-labels in a shared feature space,
which contradicts the goal of the outlier detector. The addition of the
projection heads not only restores the OOD detection performance but also
achieves superior results when combined with confidence filtering. Adding the
projection heads in combination with confidence filtering not only restores
the OOD detection performance but achieves even better performance, which
indicates the importance of representation separation. Note that it is
important to have two independent projection heads for the inlier classifier
and outlier detector. A shared projection head does not restore the OOD
detection performance as shown in Table 1. Moreover, we show in Table 2 that
both classification and detection performance degrade when swapping the task-
specific features of a pre-trained model with the fixed encoder. In
particular, when re-training the detector (just a fully-connected layer) on
top of classification features, seen AUROC drops from 89.18% to 53.99%, which
suggests our model learns more task-specific features. Therefore, the
utilization of the projection heads separates concerns between the classifier
and detector, which eases the difficulties of the task and allows them to be
trained jointly without adversely affecting each other. The effect of the
depth and the width of the projection head is studied in Appendix C.

| Proj.
---
head
| Conf.
---
filter
| Inlier Cls.
---
(Acc.)
| Outlier Det.
---
(AUROC)
|  | 78.05 | 89.67
shared |  | 76.75 | 91.92
separate |  | 78.47 | 90.92
| ✓ | 90.28 | 63.46
shared | ✓ | 90.93 | 63.87
separate | ✓ | 91.65 | 94.76
Table 1:  Effect of the projection head and confidence-based pseudo-labeling
for classifier training. We use a 2-layer MLP as the projection head. All
models are trained with pseudo-negative mining on the same data split.

an empty line here Nearest Neighbor Inlier Cls. (Acc.) Outlier Det. (AUROC)
default (a) 55.04 99.43 swap cls. & det. features (b) 53.70 77.89

Table 2:  Classification and detection performance using features of different
heads. We fix the encoder and MLP heads and evaluate the classification and
detection performance using nearest neighbors on labeled set. Our model learns
specialized features since swapping $h_{c}$ and $h_{d}$ leads to inferior
performance in both tasks.

Improving data utilization with confidence-based pseudo-labeling. Here we
study the effect of different classifier training strategies. We compare three
unlabeled data filtering methods for classifier training: (1) det. selects
pseudo-inliers with the outlier detector as in [41]; (2) det. (tuned), where
we choose the selection threshold in detector-based filtering so that the
recall of actual inlier samples matches ours; (3) conf. uses unlabeled data
whose confidence is higher than a pre-defined threshold, which is our method.
As shown in Table 3, although det. successfully removes many OOD data, it also
eliminates many inliers, resulting in a low utilization ratio of unlabeled
data (0.29% unlabeled data are used in training). In contrast, our method
includes pseudo-labels with high classifier confidence into the training,
irrespective of whether a sample is out-of-distribution, which leads to a high
utilization ratio of unlabeled data (94.22%), thus, outperforming det. with a
large margin. Moreover, our method also outperforms det (tuned) whose data
selection threshold is tuned for better performance. This is because we
incorporate a significant amount of OOD data in the training process (40.16%
v.s. 16.90%). In fact, many OOD data are natural data augmentation of inliers,
which can substantially improve closed-set classification if used carefully.
When removing pseudo-labeled OOD data using an oracle during the training. The
inlier classification accuracy decreases by 3.37% on CIFAR-10 with 25 labels
(from 91.65% to 88.28%), which suggests pseudo-labeled OOD data are helpful
for inlier classification. In Fig. 5, we visualize top-5 confident OOD samples
predicted for three inlier classes from conf. on CIFAR-100. We can see that
the selected samples are related to the inlier classes and contain the
corresponding semantics despite being outliers. For example, OOD data selected
for sea are images with sea background (more examples in Appendix E).

Filter method | det. | det (tuned) | conf. (ours)
---|---|---|---
Test | Inlier Clf. (Acc.) | 47.20 | 86.53 | 91.65
Outlier Det. (AUROC) | 57.72 | 87.87 | 94.76
Train | Utilization ratio of:
\- Unlabeled | 0.29 | 58.09 | 94.22
\- OOD data | 0.04 | 16.90 | 40.16
Prec. of pseudo-inliers | 95.17 | 86.53 | 58.30
Recall of inliers | 0.47 | 93.86 | 92.14

Table 3:  Effect of different OOD filtering methods for classifier training.
We compare three filtering methods: conf. denotes the confidence-based pseudo-
labeling; det. uses the outlier detector to select pseudo-inliers for
classifier training; det. (tuned) is a tuned version of det. that matches the
recall of inliers with our method. We compare the performance as well as the
data utilization ratio, precision, and recall of the inliers from unlabeled
data during training. All models are trained with pseudo-negative mining and
the projection head on the same data split.  Figure 5:  OOD samples can be
used as data augmentation to improve the generalization performance. The
figure shows three semantic classes from labeled data (wolf, road, and sea),
and top-5 confident OOD samples predicted for those classes. The ground-truth
semantic class of the OOD sample is on the top of each image. We can see that
OOD data with high confidence present large visual similarities to the
corresponding semantic classes.

Effect of pseudo-negative mining. Table 5 shows the effect of pseudo-negative
mining. We compare our pseudo-negative mining with standard pseudo-labeling
which predicts artificial labels for unlabeled data and uses confident
predictions with labeled data loss. While standard pseudo-labeling does not
help the OOD detection performance further, pseudo-negative mining improves
the seen AUROC by 4.73% over the model without pseudo-negative mining.
Compared to standard pseudo-labeling, pseudo-negative mining not only includes
more unlabeled data into the training, but also presents high precision for
the selected pseudo-outliers as shown in Fig. 6.

As mentioned in Section 3.4, we utilize unlabeled data with low inlier scores
as pseudo-outliers to enhance the data diversity of outlier classes. An
unlabeled sample is used as a pseudo-outlier only if its confidence score is
less than a pre-defined threshold $\theta$. Table 4 compares the results of
different thresholds. We can see that our method achieves similar performance
as long as $\theta$ takes a relatively small value, which suggests the good
robustness of our method against this hyper-parameter. We provide more
ablation on loss weight and data augmentation in Appendix C.

Threshold $\theta$ |  | Inlier Cls.
---
(Acc.)
| Outlier Det.
---
(seen AUROC)
0.2 | 91.87 | 92.96
0.1 | 92.03 | 93.16
0.05 | 91.97 | 94.21
0.01 | 91.65 | 94.76
0.005 | 91.52 | 94.75
0.001 | 91.70 | 94.15

Table 4:  Effect of different thresholds $\theta$ for pseudo-negative mining. Our method shows good robustness against a wide range of thresholds. We use CIFAR-10 with 25 labeled data here.  | Pseudo-
---
labeling
| Inlier Cls.
---
(Acc.)
| Outlier Det.
---
(AUROC)
None | 91.52 | 90.03
Standard | 91.63 | 89.69
Pseudo-neg. | 91.65 | 94.76
Table 5:  Effect of pseudo-negative mining for OOD detection. All models are
trained with confidence-based pseudo-labeling and a 2-layer MLP projection
head on the same data split.

Figure 6: Compared to standard pseudo-labeling, pseudo-negative mining has not
only higher prediction precision, but also higher data utilization rate.

Ablation on outlier detectors. Here, we compare the performance of different
outlier detection methods. Specifically, we choose three schemes from recent
works, including the binary classifier from MTC [51], cross-modal matching
from T2T [22], and OVA classifiers from OpenMatch [41]. As shown in Table 6.
While all methods show reasonable performance, OVA classifiers exhibit the
best performance in both inlier classification and OOD detection. Hence, we
use OVA classifiers as the outlier detector in our final model.

OOD Detector |  | Inlier Cls.
---
(Acc.)
| Outlier Det.
---
(AUROC)
binary classifier [51] | 70.93 | 76.12
cross-modal matching [22] | 69.27 | 75.99
OVA classifiers [41] | 71.00 | 82.62

Table 6:  Comparison between different outlier detectors. The experiment is
conducted on CIFAR-100 with 55 inlier classes and 25 labels per class.

Compatibility with other open-set SSL methods. We evaluated the compatibility
of our method with other open-set SSL techniques in Table 7. Our results
indicate that our method is highly compatible, as all existing methods showed
improved performance in both inlier classification and outlier detection when
combined with our approach. This demonstrates the flexibility of our method
and suggests that it can be easily integrated into existing frameworks as a
plug-and-play solution.

|  | Inlier Cls.
---
(Acc.)
| Outlier Det.
---
(AUROC)
MTC | 60.24 | 69.88
MTC + Ours | 60.42 | 74.98
T2T | 64.78 | 52.93
T2T + Ours | 66.98 | 69.50
OpenMatch | 68.53 | 80.00
OpenM. + Ours | 71.00 | 82.62
Table 7:  Integrating our method with other open-set SSL methods improves
performance. The setting is CIFAR-100 with 55 inlier classes and 25 labels per
class.

Equal-parameter comparison. As mentioned in Section 4.1, the performance
improvement of SSB can not be simply explained by the increased number of
parameters introduced in the projection heads. Here we compare SSB with other
methods + MLP heads so that they have the same number of parameters as SSB. As
shown in Fig. 7, adding MLP heads improves the performance of other methods,
but SSB still greatly outperforms all of them, indicating that the performance
improvement of our method can not be merely explained by the increase of the
model capacity.

Figure 7:  Comparison between SSB and other methods with the same model
parameters. The performance improvement of SSB can not be simply explained by
the increased number of parameters.

##  5 Conclusion and Limitations

In this paper, we study a realistic and challenging setting, open-set SSL,
where unlabeled data contains outliers from categories that do not appear in
the labeled data. We first demonstrate that classifier-confidence-based
pseudo-labeling can effectively improve the unlabeled data utilization ratio
and leverage useful OOD data, which largely improves the classification
performance. We find that adding non-linear transformations between the task-
specific head and the shared features provides sufficient decoupling of the
two heads, which prevents mutual interference and improves performance in both
tasks. Additionally, we propose pseudo-negative mining to improve OOD
detection. It uses pseudo-outliers to enhance the representation learning of
OOD data, which further improves the model’s ability to distinguish between
inliers and OOD samples. Overall, we achieve state-of-the-art performance on
several benchmark datasets, demonstrating the effectiveness of the proposed
method.

Nonetheless, SSB has potential limitations. Despite the improved overall
performance, the outlier detector suffers from overfitting as the performance
gap between detecting seen outliers and unseen outliers is still very large.
Therefore, in the future, more regularizations need to be considered to
improve the generalization. Another drawback is that our method is not able to
deal with long-tail distributions, which is also very realistic in practice.
Presumably, our method will have difficulty distinguishing inliers of tail
classes and OOD data due to the data scarcity at tail.

## References

  * [1] Charu C Aggarwal and Philip S Yu.  Outlier detection for high dimensional data.  In Proceedings of the 2001 ACM SIGMOD international conference on Management of data, 2001.
  * [2] Eric Arazo, Diego Ortego, Paul Albert, Noel E O’Connor, and Kevin McGuinness.  Pseudo-labeling and confirmation bias in deep semi-supervised learning.  In International Joint Conference on Neural Networks (IJCNN). IEEE, 2020.
  * [3] David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel.  Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring.  In 8th International Conference on Learning Representations, ICLR, 2020.
  * [4] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel.  Mixmatch: A holistic approach to semi-supervised learning.  In Advances in Neural Information Processing Systems, 2019.
  * [5] David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alex Kurakin.  Adamatch: A unified approach to semi-supervised learning and domain adaptation.  In International Conference on Learning Representations, 2022.
  * [6] Kaidi Cao, Maria Brbic, and Jure Leskovec.  Open-world semi-supervised learning.  arXiv preprint arXiv:2102.03526, 2021.
  * [7] Paola Cascante-Bonilla, Fuwen Tan, Yanjun Qi, and Vicente Ordonez.  Curriculum labeling: Revisiting pseudo-labeling for semi-supervised learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
  * [8] Yanbei Chen, Xiatian Zhu, Wei Li, and Shaogang Gong.  Semi-supervised learning under class distribution mismatch.  In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
  * [9] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi.  Describing textures in the wild.  In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
  * [10] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In Proceedings of the IEEE conference on computer vision and pattern recognition Workshops, 2020.
  * [11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In IEEE conference on computer vision and pattern recognition, 2009.
  * [12] Terrance DeVries and Graham W Taylor.  Improved regularization of convolutional neural networks with cutout.  arXiv preprint arXiv:1708.04552, 2017.
  * [13] Yue Fan, Anna Kukleva, and Bernt Schiele.  Revisiting consistency regularization for semi-supervised learning.  In DAGM German Conference on Pattern Recognition, 2021.
  * [14] Yves Grandvalet and Yoshua Bengio.  Semi-supervised learning by entropy minimization.  In Advances in neural information processing systems, 2005.
  * [15] Gregory Griffin, Alex Holub, and Pietro Perona.  Caltech-256 object category dataset.  2007\.
  * [16] Lan-Zhe Guo, Zhen-Yu Zhang, Yuan Jiang, Yu-Feng Li, and Zhi-Hua Zhou.  Safe deep semi-supervised learning for unseen-class unlabeled data.  In International Conference on Machine Learning, 2020.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
  * [18] Rundong He, Zhongyi Han, Xiankai Lu, and Yilong Yin.  Safe-student for safe deep semi-supervised learning with unseen-class unlabeled data.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  * [19] Rundong He, Zhongyi Han, Yang Yang, and Yilong Yin.  Not all parameters should be treated equally: Deep safe semi-supervised learning under class distribution mismatch.  In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
  * [20] Dan Hendrycks and Kevin Gimpel.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  In International Conference on Learning Representations, 2016.
  * [21] Victoria Hodge and Jim Austin.  A survey of outlier detection methodologies.  Artificial intelligence review, 2004.
  * [22] Junkai Huang, Chaowei Fang, Weikai Chen, Zhenhua Chai, Xiaolin Wei, Pengxu Wei, Liang Lin, and Guanbin Li.  Trash to treasure: Harvesting ood data with cross-modal matching for open-set semi-supervised learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
  * [23] Zhuo Huang, Jian Yang, and Chen Gong.  They are not completely useless: Towards recycling transferable unlabeled data for class-mismatched semi-supervised learning.  IEEE Transactions on Multimedia, 2022.
  * [24] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. PMLR, 2015.
  * [25] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei.  Novel dataset for fine-grained image categorization.  In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, 2011.
  * [26] Jack Kiefer and Jacob Wolfowitz.  Stochastic estimation of the maximum of a regression function.  The Annals of Mathematical Statistics, 1952.
  * [27] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images, 2009.  Technical report.
  * [28] Samuli Laine and Timo Aila.  Temporal ensembling for semi-supervised learning.  In 5th International Conference on Learning Representations, ICLR, 2017.
  * [29] Dong-Hyun Lee.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In Workshop on challenges in representation learning, ICML, 2013.
  * [30] Junnan Li, Caiming Xiong, and Steven C.H. Hoi.  Comatch: Semi-supervised learning with contrastive graph regularization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  * [31] Oded Maimon and Lior Rokach.  Data mining and knowledge discovery handbook.  2005\.
  * [32] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii.  Virtual adversarial training: a regularization method for supervised and semi-supervised learning.  IEEE transactions on pattern analysis and machine intelligence, 2018.
  * [33] Vinod Nair and Geoffrey E Hinton.  Rectified linear units improve restricted boltzmann machines.  In ICML, 2010.
  * [34] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning.  In In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
  * [35] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, 2008.
  * [36] Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow.  Realistic evaluation of deep semi-supervised learning algorithms.  In Advances in neural information processing systems, 2018.
  * [37] Jongjin Park, Sukmin Yun, Jongheon Jeong, and Jinwoo Shin.  Opencos: Contrastive semi-supervised learning for handling open-set unlabeled data.  In European conference on computer vision Workshop, 2022.
  * [38] Hieu Pham, Qizhe Xie, Zihang Dai, and Quoc V Le.  Meta pseudo labels.  arXiv preprint arXiv:2003.10580, 2020.
  * [39] Mamshad Nayeem Rizve, Navid Kardan, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah.  Openldn: Learning to discover novel classes for open-world semi-supervised learning.  In European Conference on Computer Vision, 2022.
  * [40] Mamshad Nayeem Rizve, Navid Kardan, and Mubarak Shah.  Towards realistic semi-supervised learning.  In European Conference on Computer Vision, 2022.
  * [41] Kuniaki Saito, Donghyun Kim, and Kate Saenko.  Openmatch: Open-set semi-supervised learning with open-set consistency regularization.  Advances in Neural Information Processing Systems, 2021.
  * [42] Kuniaki Saito and Kate Saenko.  Ovanet: One-vs-all network for universal domain adaptation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
  * [43] Robert E Schapire.  The strength of weak learnability.  Machine learning, 1990.
  * [44] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  In Advances in Neural Information Processing Systems, 2020.
  * [45] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In Advances in neural information processing systems, 2017.
  * [46] Hongzhi Wang, Mohamed Jaward Bah, and Mohamed Hammad.  Progress in outlier detection techniques: A survey.  Ieee Access, 2019.
  * [47] Yidong Wang, Hao Chen, Yue Fan, Wang Sun, Ran Tao, Wenxin Hou, Renjie Wang, Linyi Yang, Zhi Zhou, Lan-Zhe Guo, Heli Qi, Zhen Wu, Yu-Feng Li, Satoshi Nakamura, Wei Ye, Marios Savvides, Bhiksha Raj, Takahiro Shinozaki, Bernt Schiele, Jindong Wang, Xing Xie, and Yue Zhang.  Usb: A unified semi-supervised learning benchmark for classification.  In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  * [48] Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le.  Unsupervised data augmentation for consistency training.  arXiv preprint arXiv:1904.12848, 2019.
  * [49] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le.  Self-training with noisy student improves imagenet classification.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.
  * [50] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao.  Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop.  arXiv preprint arXiv:1506.03365, 2015.
  * [51] Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa.  Multi-task curriculum framework for open-set semi-supervised learning.  In European Conference on Computer Vision, 2020.
  * [52] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  In Proceedings of the British Machine Vision Conference (BMVC), 2016.
  * [53] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki.  Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling.  Advances in Neural Information Processing Systems, 2021.
  * [54] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  In 6th International Conference on Learning Representations, ICLR, 2018.
  * [55] Mingkai Zheng, Shan You, Lang Huang, Fei Wang, Chen Qian, and Chang Xu.  Simmatch: Semi-supervised learning with similarity matching.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.

In this appendix, we first show the tabulated breakdown of our main result in
Section A. Then we compare our method with more recent approaches in Section B
and show additional ablation studies in Section C. Finally, we present the
pseudo-code and more visualizations of pseudo-inliers in Section D and Section
E, respectively.

##  Appendix A Main Results Breakdown

Here we present tabulated breakdown results of Fig. 3 and 4 from the main
paper. We summarize the inlier classification accuracy and outlier detection
in AUROC for different settings in Table 8, 9, 10, 11, 12, 13, and 14,
respectively. To provide a more comprehensive analysis, we further provide the
separate outlier detection performance results for seen outliers and unseen
outliers in Table 15 and Table 16, respectively. SSB achieves competitive
results in all settings. In particular, for CIFAR-10 and CIFAR-100 with 25
labels, SSB outperforms other methods by a large margin.

Test Acc. / AUROC | CIFAR-10
---|---
inlier / outlier classes | 6 / 4
labels per class | 25
SSL | FixMatch [44] |  $\textbf{91.94}_{\pm 0.16}$ / $62.58_{\pm 0.53}$
FlexMatch [53] |  $82.91_{\pm 0.92}$ / $69.60_{\pm 4.11}$
SimMatch [55] |  $89.22_{\pm 2.24}$ / $63.85_{\pm 0.70}$
OSSL | MTC [51] |  $71.91_{\pm 10.82}$ / $85.57_{\pm 6.63}$
OpenMatch [41] |  $54.88_{\pm 2.33}$ / $53.32_{\pm 4.62}$
T2T [22] |  $83.21_{\pm 0.98}$ / $44.79_{\pm 17.26}$
| SSB (FixMatch) |  91.74±0.24 / $\underline{95.86}_{\pm 1.37}$
| SSB (FlexMatch) |  $84.72_{\pm 0.49}$ / $90.32_{\pm 0.75}$
| SSB (SimMatch) |  $88.51_{\pm 2.86}$ / $\textbf{97.54}_{\pm 0.08}$
Table 8:  CIFAR-10 with 25 labels and 6 inlier classes. We report test
accuracy (%) / AUROC (%) for inliers classification and outlier detection,
respectively. The numbers are averaged over 3 different random seeds. The best
number is in bold, and the second best is in underline.

Test Acc. / AUROC | CIFAR-10
---|---
inlier / outlier classes | 6 / 4
labels per class | 50
SSL | FixMatch [44] |  $\underline{91.33}_{\pm 0.18}$ / $63.77_{\pm 0.14}$
FlexMatch [53] |  $83.98_{\pm 0.31}$ / $64.47_{\pm 0.10}$
SimMatch [55] |  $91.10_{\pm 0.52}$ / $65.34_{\pm 0.09}$
OSSL | MTC [51] |  $81.03_{\pm 5.21}$ / $92.01_{\pm 2.62}$
OpenMatch [41] |  $91.31_{\pm 1.18}$ / $\underline{95.88}_{\pm 0.60}$
T2T [22] |  $90.56_{\pm 0.07}$ / $39.73_{\pm 8.94}$
| SSB (FixMatch) |  $\textbf{92.18}_{\pm 0.33}$ / $\textbf{97.65}_{\pm 0.19}$
| SSB (FlexMatch) |  $84.26_{\pm 1.36}$ / $93.16_{\pm 3.63}$
| SSB (SimMatch) |  $90.82_{\pm 0.47}$ / $94.07_{\pm 0.40}$
Table 9:  CIFAR-10 with 50 labels and 6 inlier classes. We report test accuracy (%) / AUROC (%) for inliers classification and outlier detection, respectively. The numbers are averaged over 3 different random seeds. The best number is in bold, and the second best is in underline.  Test Acc. / AUROC | CIFAR-100
---|---
inlier / outlier classes | 55 / 45
labels per class | 25
SSL | FixMatch [44] |  $69.89_{\pm 0.00}$ / $63.81_{\pm 0.26}$
FlexMatch [53] |  $67.87_{\pm 0.58}$ / $67.65_{\pm 1.12}$
SimMatch [55] |  $70.25_{\pm 0.96}$ / $65.21_{\pm 0.68}$
OSSL | MTC [51] |  $58.13_{\pm 2.11}$ / $71.62_{\pm 1.36}$
OpenMatch [41] |  $67.09_{\pm 1.44}$ / $80.18_{\pm 0.09}$
T2T [22] |  $65.71_{\pm 0.93}$ / $60.11_{\pm 6.25}$
| SSB (FixMatch) |  $\underline{70.64}_{\pm 0.36}$ / $82.91_{\pm 0.30}$
| SSB (FlexMatch) |  $68.28_{\pm 0.74}$ / $\underline{83.62}_{\pm 0.43}$
| SSB (SimMatch) |  $\textbf{70.77}_{\pm 0.54}$ / $\textbf{84.77}_{\pm 0.52}$
Table 10:  CIFAR-100 with 25 labels and 55 inlier classes. We report test
accuracy (%) / AUROC (%) for inliers classification and outlier detection,
respectively. The numbers are averaged over 3 different random seeds. The best
number is in bold, and the second best is in underline.

Test Acc. / AUROC | CIFAR-100
---|---
inlier / outlier classes | 55 / 45
labels per class | 50
SSL | FixMatch [44] |  $73.28_{\pm 0.59}$ / $66.03_{\pm 0.41}$
FlexMatch [53] |  $71.74_{\pm 0.01}$ / $70.36_{\pm 0.60}$
SimMatch [55] |  $\underline{74.15}_{\pm 0.57}$ / $67.34_{\pm 0.19}$
OSSL | MTC [51] |  $65.97_{\pm 0.77}$ / $68.96_{\pm 1.08}$
OpenMatch [41] |  $71.90_{\pm 1.05}$ / $82.82_{\pm 0.47}$
T2T [22] |  $70.69_{\pm 0.11}$ / $61.97_{\pm 0.50}$
| SSB (FixMatch) |  $73.70_{\pm 0.75}$ / $\underline{85.89}_{\pm 0.07}$
| SSB (FlexMatch) |  $72.65_{\pm 0.25}$ / $\textbf{85.97}_{\pm 0.46}$
| SSB (SimMatch) |  $\textbf{75.15}_{\pm 0.34}$ / $84.60_{\pm 0.18}$
Table 11:  CIFAR-100 with 50 labels and 55 inlier classes. We report test accuracy (%) / AUROC (%) for inliers classification and outlier detection, respectively. The numbers are averaged over 3 different random seeds. The best number is in bold, and the second best is in underline.  Test Acc. / AUROC | CIFAR-100
---|---
inlier / outlier classes | 80 / 20
labels per class | 25
SSL | FixMatch [44] |  $63.58_{\pm 0.36}$ / $56.40_{\pm 0.21}$
FlexMatch [53] |  $59.83_{\pm 1.78}$ / $62.73_{\pm 0.62}$
SimMatch [55] |  $\underline{65.92}_{\pm 0.81}$ / $60.61_{\pm 0.60}$
OSSL | MTC [51] |  $52.32_{\pm 0.13}$ / $67.43_{\pm 0.38}$
OpenMatch [41] |  $52.13_{\pm 4.81}$ / $68.32_{\pm 4.68}$
T2T [22] |  $47.58_{\pm 10.38}$ / $51.95_{\pm 4.44}$
| SSB (FixMatch) |  $64.20_{\pm 0.41}$ / $\underline{81.71}_{\pm 0.86}$
| SSB (FlexMatch) |  $60.39_{\pm 1.89}$ / $79.85_{\pm 0.94}$
| SSB (SimMatch) |  $\textbf{66.48}_{\pm 0.77}$ / $\textbf{82.39}_{\pm 2.97}$
Table 12:  CIFAR-100 with 25 labels and 80 inlier classes. We report test accuracy (%) / AUROC (%) for inliers classification and outlier detection, respectively. The numbers are averaged over 3 different random seeds. The best number is in bold, and the second best is in underline.  Test Acc. / AUROC | CIFAR-100
---|---
inlier / outlier classes | 80 / 20
labels per class | 50
SSL | FixMatch [44] |  $67.06_{\pm 0.10}$ / $58.05_{\pm 0.49}$
FlexMatch [53] |  $65.22_{\pm 0.18}$ / $65.00_{\pm 0.07}$
SimMatch [55] |  $\underline{69.35}_{\pm 0.26}$ / $61.44_{\pm 0.16}$
OSSL | MTC [51] |  $59.17_{\pm 0.01}$ / $69.34_{\pm 1.81}$
OpenMatch [41] |  $66.90_{\pm 0.19}$ / $79.95_{\pm 0.26}$
T2T [22] |  $64.18_{\pm 0.64}$ / $65.26_{\pm 13.73}$
| SSB (FixMatch) |  $67.97_{\pm 0.20}$ / $80.81_{\pm 1.02}$
| SSB (FlexMatch) |  $65.79_{\pm 0.06}$ / $\textbf{83.32}_{\pm 0.36}$
| SSB (SimMatch) |  $\textbf{70.27}_{\pm 0.19}$ / $\underline{81.16}_{\pm 2.10}$
Table 13:  CIFAR-100 with 50 labels and 80 inlier classes. We report test accuracy (%) / AUROC (%) for inliers classification and outlier detection, respectively. The numbers are averaged over 3 different random seeds. The best number is in bold, and the second best is in underline.  Test Acc. / AUROC | ImageNet-30
---|---
inlier / outlier classes | 20 / 10
labels per class | 5%
SSL | FixMatch [44] |  $90.33_{\pm 0.66}$ / $75.60_{\pm 1.28}$
FlexMatch [53] |  $86.33_{\pm 0.92}$ / $72.24_{\pm 0.45}$
SimMatch [55] |  $\underline{91.32}_{\pm 0.73}$ / $71.72_{\pm 0.14}$
OSSL | MTC [51] |  $81.05_{\pm 0.35}$ / $80.66_{\pm 2.23}$
OpenMatch [41] |  $78.75_{\pm 0.35}$ / $\textbf{84.21}_{\pm 0.03}$
T2T [22] |  $88.75_{\pm 0.90}$ / $73.11_{\pm 1.11}$
| SSB (FixMatch) |  $\textbf{91.80}_{\pm 0.05}$ / $\underline{82.80}_{\pm 1.18}$
| SSB (FlexMatch) |  $86.90_{\pm 0.70}$ / $75.42_{\pm 0.24}$
| SSB (SimMatch) |  $91.30_{\pm 0.65}$ / $75.54_{\pm 0.10}$
Table 14:  ImageNet-30 with 5% labels and 20 inliers classes. We report test accuracy (%) / AUROC (%) for inliers classification and outlier detection, respectively. The numbers are averaged over 3 different random seeds. The best number is in bold, and the second best is in underline.  Seen AUORC | CIFAR-10 | CIFAR-100 | CIFAR-100 | ImageNet-30
---|---|---|---|---
Inlier / outlier classes | 6 / 4 | 55 / 45 | 80 / 20 | 20 / 10
Labels per class | 25 | 50 | 25 | 50 | 25 | 50 | 5%
SSL | FixMatch [44] | $37.37_{\pm 0.84}$ | $39.41_{\pm 0.15}$ | $54.48_{\pm 1.05}$ | $55.77_{\pm 0.95}$ | $41.43_{\pm 0.10}$ | $44.33_{\pm 0.79}$ | $65.09_{\pm 2.09}$
FlexMatch [53] | $51.32_{\pm 8.24}$ | $41.01_{\pm 0.20}$ | $60.82_{\pm 1.07}$ | $63.72_{\pm 1.15}$ | $53.68_{\pm 0.48}$ | $57.45_{\pm 0.70}$ | $61.73_{\pm 0.31}$
SimMatch [55] | $38.39_{\pm 0.87}$ | $41.15_{\pm 0.26}$ | $55.28_{\pm 1.35}$ | $57.69_{\pm 0.16}$ | $49.19_{\pm 0.43}$ | $49.53_{\pm 0.26}$ | $56.41_{\pm 0.55}$
OSSL | MTC [51] | $92.00_{\pm 3.49}$ | $94.47_{\pm 2.05}$ | $76.93_{\pm 1.48}$ | $72.53_{\pm 0.18}$ | $69.15_{\pm 0.86}$ | $72.38_{\pm 1.86}$ | $82.70_{\pm 2.60}$
OpenMatch [41] | $62.46_{\pm 4.19}$ | $\underline{99.41}_{\pm 0.18}$ | $84.93_{\pm 0.08}$ | $86.99_{\pm 0.23}$ | $74.87_{\pm 3.78}$ | $\underline{86.19}_{\pm 0.48}$ | $\textbf{91.79}_{\pm 0.49}$
T2T [22] | $34.90_{\pm 27.50}$ | $23.85_{\pm 8.45}$ | $52.95_{\pm 6.15}$ | $59.50_{\pm 1.50}$ | $50.45_{\pm 9.15}$ | $61.40_{\pm 21.10}$ | $62.35_{\pm 2.05}$
| SSB (FixMatch) | $\underline{99.35}_{\pm 0.38}$ | $\textbf{99.63}_{\pm 0.15}$ | $89.39_{\pm 0.44}$ | $\underline{90.62}_{\pm 0.46}$ | $\textbf{90.25}_{\pm 1.34}$ | $85.29_{\pm 2.18}$ | $\underline{83.84}_{\pm 2.31}$
| SSB (FlexMatch) | $96.81_{\pm 0.25}$ | $93.41_{\pm 6.22}$ | $\textbf{89.84}_{\pm 0.05}$ | $\textbf{91.21}_{\pm 0.26}$ | $88.32_{\pm 1.36}$ | $\textbf{90.68}_{\pm 0.27}$ | $67.58_{\pm 0.55}$
| SSB (SimMatch) | $\textbf{99.61}_{\pm 0.10}$ | $93.07_{\pm 0.70}$ | $\underline{89.75}_{\pm 0.90}$ | $87.38_{\pm 0.13}$ | $\underline{88.65}_{\pm 3.86}$ | $83.05_{\pm 3.49}$ | $62.63_{\pm 0.09}$
Table 15:  AUROC (%) for seen outliers. The best number is in bold, and the second best is in underline. Note that a random OOD detector gives an AUROC of 50%.  Unseen AUROC | CIFAR-10 | CIFAR-100 | CIFAR-100 | ImageNet-30
---|---|---|---|---
Inlier / outlier classes | 6 / 4 | 55 / 45 | 80 / 20 | 20 / 10
Labels per class | 25 | 50 | 25 | 50 | 25 | 50 | 5%
SSL | FixMatch [44] | $87.80_{\pm 0.23}$ | $88.12_{\pm 0.42}$ | $73.15_{\pm 0.53}$ | $76.28_{\pm 0.13}$ | $71.37_{\pm 0.32}$ | $71.77_{\pm 0.20}$ | $86.11_{\pm 0.54}$
FlexMatch [53] | $87.88_{\pm 0.02}$ | $87.92_{\pm 0.00}$ | $74.48_{\pm 1.18}$ | $77.01_{\pm 0.06}$ | $71.79_{\pm 0.76}$ | $72.54_{\pm 0.84}$ | $82.76_{\pm 0.59}$
SimMatch [55] | $89.32_{\pm 0.54}$ | $89.54_{\pm 0.45}$ | $75.14_{\pm 0.01}$ | $76.98_{\pm 0.21}$ | $72.03_{\pm 0.78}$ | $73.36_{\pm 0.06}$ | $\underline{87.04}_{\pm 0.27}$
OSSL | MTC [51] | $79.14_{\pm 9.78}$ | $89.55_{\pm 3.20}$ | $66.32_{\pm 1.25}$ | $65.40_{\pm 1.99}$ | $65.71_{\pm 0.09}$ | $66.30_{\pm 5.49}$ | $78.61_{\pm 1.86}$
OpenMatch [41] | $44.18_{\pm 5.05}$ | $92.35_{\pm 1.01}$ | $75.43_{\pm 0.11}$ | $78.66_{\pm 0.70}$ | $61.78_{\pm 5.58}$ | $73.70_{\pm 0.04}$ | $76.63_{\pm 0.54}$
T2T [22] | $54.68_{\pm 7.01}$ | $55.61_{\pm 9.43}$ | $67.26_{\pm 6.35}$ | $64.44_{\pm 0.50}$ | $53.45_{\pm 0.27}$ | $69.11_{\pm 6.36}$ | $83.88_{\pm 0.17}$
| SSB (FixMatch) | $\underline{92.37}_{\pm 2.36}$ | $\textbf{95.67}_{\pm 0.23}$ | $76.44_{\pm 0.15}$ | $\underline{81.16}_{\pm 0.61}$ | $\underline{73.18}_{\pm 0.38}$ | $\underline{76.34}_{\pm 0.14}$ | $81.77_{\pm 0.04}$
| SSB (FlexMatch) | $83.83_{\pm 1.25}$ | $92.91_{\pm 1.03}$ | $\underline{77.40}_{\pm 0.82}$ | $80.72_{\pm 0.65}$ | $71.38_{\pm 0.53}$ | $75.96_{\pm 0.99}$ | $83.26_{\pm 1.02}$
| SSB (SimMatch) | $\textbf{95.47}_{\pm 0.26}$ | $\underline{95.07}_{\pm 0.10}$ | $\textbf{79.80}_{\pm 0.13}$ | $\textbf{81.82}_{\pm 0.23}$ | $\textbf{76.14}_{\pm 2.08}$ | $\textbf{79.27}_{\pm 0.72}$ | $\textbf{88.46}_{\pm 0.10}$
Table 16:  AUROC (%) for unseen outliers.  The best number is in bold, and the
second best is in underline. Note that a random OOD detector gives an AUROC of
50%.

##  Appendix B Results on More Benchmarks

In this section, we compare SSB with more recent methods on their benchmarks.

Comparison with methods of class-mismatched SSL. Here, we compare with Safe-
Student [18] and SPL [19] on CIFAR-10 and CIFAR-100 with different levels of
class distribution mismatch between the labeled and unlabeled data. Following
[19, 18], on CIFAR-10, we consider the six animal classes as inlier classes
and use 400 labels per class. The unlabeled set contains 20,000 images coming
from all ten classes with different class mismatch ratios. For example, when
the ratio is 0.3, 70% of the samples are from the six inlier classes and the
rest samples are from the remaining four classes. Similarly, for CIFAR-100,
the first 50 classes are used as inlier classes, and the unlabeled set has
20,000 samples with different class mismatch ratios. We compare the inlier
accuracy of different methods in Table 17. SSB outperforms other methods by
significant margins across all settings, which indicates the effectiveness of
our method for class-mismatched SSL.

Comparison in cross-dataset scenarios. Now, we compare our method in cross-
dataset settings, where the labeled set and the unlabeled set are constructed
using different datasets. Following [23], we use CIFAR-100 for labeled set and
ImageNet for unlabeled set. Specifically, we take 60 classes of CIFAR-100 as
inlier classes, which are also contained in ImageNet. Then, 20,000 images are
sampled from 100 classes of ImageNet to form the unlabeled set, where 60
classes are the same as the inlier classes of CIFAR-100 and the rest 40 are
randomly chosen from the remaining 940 classes. Please refer to [23] for
details of the 60 inlier classes. In Table 18, we compare the results of our
method with others under different numbers of labeled data. We can see that
our method improves the SOTA in all settings. The large performance gap over
TOOR [23] shows that the simple confidence-based pseudo-labeling used in SSB
is a more effective way for recycling OOD data to improve the classification
performance.

Method | CIFAR-10 | CIFAR-100
---|---|---
ratio=0.3 | ratio=0.6 | ratio=0.3 | ratio=0.6
DS3L [16] | $78.1_{\pm 0.4}$ | $76.9_{\pm 0.5}$ | - | -
UASD [8] | $77.6_{\pm 0.4}$ | $76.0_{\pm 0.4}$ | $61.8_{\pm 0.4}$ | $58.4_{\pm 0.5}$
MTC [51] | $85.5_{\pm 0.6}$ | $81.7_{\pm 0.5}$ | $63.1_{\pm 0.6}$ | $61.1_{\pm 0.3}$
CL [7] | $83.2_{\pm 0.4}$ | $82.1_{\pm 0.4}$ | $63.6_{\pm 0.4}$ | $61.5_{\pm 0.5}$
Safe-Student [18] | $85.7_{\pm 0.3}$ | $83.8_{\pm 0.1}$ | $68.4_{\pm 0.2}$ | $68.2_{\pm 0.1}$
CL+SPL [19] | $87.8_{\pm 0.3}$ | $84.1_{\pm 0.5}$ | $65.9_{\pm 0.3}$ | $65.5_{\pm 0.4}$
SSB (Ours) | $\textbf{92.5}_{\pm 0.1}$ | $\textbf{90.6}_{\pm 0.3}$ | $\textbf{74.7}_{\pm 0.6}$ | $\textbf{73.2}_{\pm 0.3}$
Table 17: Test accuracy (%) with different class mismatch ratios on CIFAR-10 and CIFAR-100. This benchmark is adopted by [18, 19]. The number of unlabeled data is 20,000. The best number is in bold. Method | CIFAR100+ImageNet with different numbers of labeled data
---|---
4800 (80 for each class) | 6000 (100 for each class) | 7200 (120 for each class) | 8400 (140 for each class)
UASD [8] | $42.07_{\pm 0.58}$ | $44.90_{\pm 0.47}$ | $46.38_{\pm 0.79}$ | $48.20_{\pm 0.40}$
DS3L [16] | $43.99_{\pm 0.54}$ | $45.10_{\pm 1.25}$ | $47.11_{\pm 0.73}$ | $48.96_{\pm 0.63}$
MTC [51] | $45.69_{\pm 0.74}$ | $46.34_{\pm 0.81}$ | $48.92_{\pm 0.58}$ | $50.05_{\pm 0.77}$
TOOR [23] | ${47.19}_{\pm 0.83}$ | ${49.15}_{\pm 0.76}$ | ${51.34}_{\pm 0.65}$ | ${52.98}_{\pm 0.79}$
SSB (Ours) | $\textbf{63.91}_{\pm 0.71}$ | $\textbf{65.95}_{\pm 0.20}$ | $\textbf{67.97}_{\pm 0.12}$ | $\textbf{69.42}_{\pm 0.28}$
Table 18: Test accuracy (%) with different numbers of labeled samples. This
benchmark is adopted by [23]. The number of unlabeled data is 20,000. The best
number is in bold.

##  Appendix C Ablation Study

In this section, we provide more analysis of the hyper-parameters used in SSB.
Specifically, we study the effect of depth and width of the projection head,
deferred outlier detector training, the effect of the threshold $\theta$ for
selecting the pseudo-outliers, the effect of the threshold $\tau$ for pseudo-
labeling, the loss weight $\lambda_{det}^{u}$ for unlabeled detection loss,
and different data augmentation schemes used in pseudo-negative mining.

Effect of depth and width of the projection head. Table 19 and 20 study the
effect of the depth and the width of the projection head, respectively. We
choose the 2-layer MLP with a hidden dimension of 1024 as it shows the best
inlier accuracy and OOD detection performance. Note that the projection head
shows good robustness over a large range of design choices. Even using a
single fully-connected layer with a ReLU activation function already gives
better performance.

Projection head |  | Inlier Cls.
---
(Acc.)
| Outlier Det.
---
(seen AUROC)
| Outlier Det.
---
(unseen AUROC)
None | 90.28 | 44.11 | 82.81
1-layer MLP | 91.48 | 98.89 | 89.96
2-layer MLP | 91.65 | 99.16 | 90.35
3-layer MLP | 91.48 | 98.49 | 88.08
4-layer MLP | 91.28 | 98.29 | 86.44
2-layer MLP + BN | 90.00 | 90.53 | 78.33

Table 19:  Effect of different projection heads. None denotes not using
projection head; MLP denotes multilayer perceptron; BN denotes using BatchNorm
[24] between different layers. All models are trained with confidence-based
pseudo-labeling and pseudo-negative mining on the same data split on CIFAR-10
with 25 labeled data.

Hidden dim. |  | Inlier Cls.
---
(Acc.)
| Outlier Det.
---
(seen AUROC)
| Outlier Det.
---
(unseen AUROC)
128 | 91.25 | 97.30 | 82.98
256 | 90.83 | 97.47 | 85.43
512 | 90.63 | 98.91 | 88.36
1024 | 91.65 | 99.16 | 90.35
2048 | 91.07 | 97.74 | 86.59

Table 20:  Effect of the hidden dimension of the projection head. We use a
2-layer MLP as the projection and train all models with confidence-based
pseudo-labeling and pseudo-negative mining on the same data split on CIFAR-10
with 25 labeled data.

Deferring the outlier detector training. Here we study the effect of different
training lengths on the outlier detector. In Table 21, we can see that while
the performance are similar under different training epochs, the training cost
can be largely reduced by deferring the detector training. With 37 training
epochs, our method can reach the best performance while reducing the training
costs.

| Total
---
epochs
| Starting
---
epochs
| GPU hours
---
reduced
| Inlier Cls.
---
(Acc.)
| Outlier Det.
---
(avg. AUROC)
512 | 0 | 0 | 90.20 | 87.92
512 | 100 | 5.9 | 90.58 | 89.11
512 | 200 | 10.7 | 91.01 | 90.39
512 | 300 | 16.0 | 91.55 | 91.71
512 | 400 | 19.6 | 91.62 | 93.13
512 | 475 | 23.2 | 91.65 | 94.76

Table 21:  Effect of different starting epochs for outlier detector training.
We defer the detector training by enabling the detection losses at a later
stage. All models are trained on a single NVIDIA V100 and we compute the
reduced GPU hours with respect to the model using detection losses throughout
the entire training. The setting is CIFAR-10 with 25 labeled data.

Effect of threshold $\tau$. We follow FixMatch [44] and set the threshold
$\tau$ for pseudo-labeling as 0.95 for our main results. Here we provide an
ablation study of this hyper-parameter in Table 22, with SSB exhibiting
robustness within a wide range of $\tau$.

Threshold $\tau$ | 0.97 | 0.95 | 0.85 | 0.75 | 0.5
---|---|---|---|---|---
Inlier Cls. | 90.28 | 91.65 | 92.28 | 90.52 | 92.35
Outlier Det. | 93.67 | 94.76 | 97.41 | 97.17 | 92.64

Table 22: Effect of different pseudo-labeling thresholds $\tau$. SSB shows
good robustness within a wide range of $\tau$. The experimental setting is
CIAR-10 with 25 labels.

Different loss weights. In the main paper, we use $\lambda_{det}^{u}=1$ for
the unlabeled data detection loss with pseudo-negative mining. Here, we
provide more results of different loss weights in Table 23. We can see that,
except for very small loss weights (0.1 or 0), the OOD detection performance
is quite robust to various $\lambda_{det}^{u}$.

| Loss weight
---
$\lambda_{det}^{u}$
| Inlier Cls.
---
(Acc.)
| Outlier Det.
---
(seen AUROC)
| Outlier Det.
---
(unseen AUROC)
10 | 92.45 | 99.18 | 89.40
5 | 91.98 | 98.61 | 88.20
2 | 91.98 | 98.82 | 88.96
1 | 91.65 | 99.16 | 90.35
0.5 | 92.08 | 99.22 | 90.18
0.1 | 91.77 | 88.98 | 90.74
0 | 91.52 | 89.78 | 90.27

Table 23:  Effect of the loss weight for pseudo-negative mining. Our method is
robust to a wide range of loss weights. The setting is CIFAR-10 with 25
labeled data.

Effect of data augmentation scheme. In pseudo-negative mining, there are two
types of data augmentations used for loss computing. Given an unlabeled image,
the OOD score is computed from a weak augmentation, which consists of random
crop and horizontal flipping. Then, if the confidence is low enough, a strong
augmentation will be used to compute the binary cross-entropy loss. Following
[44], the strong augmentation is RandAugment [10] with CutOut [12]. In Table
24, we study the effect of different types of augmentation for computing the
loss. Using the strong augmentation gives the best OOD detection performance
while having similar inlier classification performance to other strategies.

Augmentation |  | Inlier Cls.
---
(Acc.)
| Outlier Det.
---
(seen AUROC)
| Outlier Det.
---
(unseen AUROC)
weak+strong | 91.88 | 92.36 | 77.55
weak | 91.73 | 92.10 | 77.80
strong | 91.65 | 99.16 | 90.35

Table 24:  Effect of data augmentation in pseudo-negative mining.  It is
important to use a different type of data augmentation for loss computing from
the one used to generate pseudo-outliers. The setting is CIFAR-10 with 25
labeled data.

##  Appendix D Pseudo-Code

We present the complete pseudo-code of SSB with deferred outlier detector
training in Algorithm 1.

##  Appendix E Visualization of Pseudo-Inliers

We visualize the OOD samples selected for different classes in Fig. 8. The
model is trained on CIFAR-100 with 55 inlier classes and 25 labels. We can see
that the selected pseudo-inliers contain semantic information of the
corresponding classes, which indicates that some OOD data are natural data
augmentations and can be used to improve the generalization performance if
used properly.

Algorithm 1

1: Input: Labeled set
$\mathcal{D}_{\text{labeled}}=\\{(\textbf{x}_{i}^{l},y_{i})\\}_{i=1}^{N}$,
unlabeled set
$\mathcal{D}_{\text{unlabeled}}=\\{(\textbf{x}_{i}^{u})\\}_{i=1}^{M}$, feature
encoder $f$, inlier classifier $g_{c}$, outlier detector $g_{d}$, two MLP
projection heads $h_{c}$ and $h_{d}$, thresholds $\tau$ and $\theta$, batch
size $B_{l}$ and $B_{u}$, loss weights $\lambda_{det}^{u}$,
$\lambda_{OC}^{u}$, and $\lambda_{em}^{u}$, warm-up iterations $T_{0}$, total
number of training iterations $T$

2: Initialize the parameters of $f$, $g_{c}$, $g_{d}$, $h_{c}$, and $h_{d}$
randomly

3: for $t=1$ to $T$ do

4: {Sample labeled and unlabeled data}

5: $\\{\textbf{x}_{i}^{l},y_{i}\\}_{i=1}^{B_{l}}$ $\sim$ Random
sampler($\mathcal{D}_{\text{labeled}}$)

6: $\\{\textbf{x}_{i}^{u}\\}_{i=1}^{B_{u}}$ $\sim$ Random
sampler($\mathcal{D}_{\text{unlabeled}}$)

7: {Compute classification losses}

8:
$\hat{p}_{i}^{u}=\text{softmax}(g_{c}(h_{c}(f(\textbf{x}_{i}^{u}))),i=1,...,B_{u}$
{Compute pseudo-label distributions}

9: $\hat{y}^{u}_{i}=\text{argmax}\hat{p}_{i}^{u},i=1,...,B_{u}$ {Compute
pseudo-labels}

10:
$L_{cls}^{l}=\frac{1}{B_{l}}\sum_{i=1}^{B_{l}}H(g_{c}(h_{c}(f(\textbf{x}_{i}^{l})),y_{i})$
{Labeled data loss}

11:
$L_{cls}^{u}=\frac{1}{B_{u}}\sum_{i=1}^{B_{u}}\text{1}(\max\hat{p}_{i}^{u}\geq\tau)H(\hat{p}_{i}^{u},\hat{y}^{u}_{i})$
{Unlabeled data loss as in Equation (2)}

12: $L_{cls}=L_{cls}^{l}+L_{cls}^{u}$

13: {Compute detection losses}

14:
$L_{det}^{l}=-\frac{1}{B_{l}}\sum_{i=1}^{B_{l}}log(p_{y_{i}}(\textbf{x}_{i}^{l}))+\frac{1}{|\mathcal{C}|-1}\sum_{j\neq
y_{i}}log(1-p_{j}(\textbf{x}_{i}^{l}))$ {Detection loss for labeled data as in
Equation (4)}

15:
$L_{det}^{u}=-\frac{1}{B_{u}}\sum_{i=1}^{B_{u}}\frac{1}{\sum_{c}{\text{1}(p_{c}>\theta)}}\sum_{c=1}^{|\mathcal{C}|}\text{1}(p_{c}>\theta)log(1-p_{c}(\textbf{x}_{i}^{u}))$
{Pseudo-negative mining as in Equation (5)}

16:
$L_{em}^{u}=\frac{1}{B_{u}}\sum_{i=1}^{B_{u}}entropy(g_{d}(h_{d}(f(\textbf{x}_{i}^{u}))))$
{Entropy minimization loss as in [14]}

17:
$L_{OC}^{u}=\frac{1}{B_{u}}\sum_{i=1}^{B_{u}}||g_{d}(h_{d}(f(\mathcal{T}_{1}(\textbf{x}_{i}^{u}))))-g_{d}(h_{d}(f(\mathcal{T}_{2}(\textbf{x}_{i}^{u}))))||^{2}$
{Open-set consistency loss as in [41]}

18:
$L_{det}=L_{det}^{l}+\lambda_{det}^{u}L_{det}^{u}+\lambda_{OC}^{u}L_{OC}^{u}+\lambda_{em}^{u}L_{em}^{u}$

19: {Total loss}

20: $L_{total}=L_{cls}+\text{1}(t>T_{0})L_{det}$

21: Update parameters in $f$, $g_{c}$, $g_{d}$, $h_{c}$, and $h_{d}$ with SGD

22: end for

23: return $f$, $g_{c}$, $g_{d}$, $h_{c}$, and $h_{d}$

Figure 8:  Selected pseudo-inliers for different classes. Each row lists 10
most confident images with the pseudo-label on the left of the row. The
ground-truth class of the OOD sample is shown on the top of each image.