# Evaluating Out-of-Distribution Performance on Document Image Classifiers

Stefan Larson${}^{~{}~{}1,3}$ &Gordon Lim2 &Yutong Ai2 &David Kuang2 &Kevin
Leach3
1DryvIQ 2University of Michigan 3Vanderbilt University Ann Arbor, MI, USA Ann
Arbor, MI, USA Nashville, TN, USA Corresponding email<EMAIL_ADDRESS>(alternate email: stefan.dataset@gmail.com).

###### Abstract

The ability of a document classifier to handle inputs that are drawn from a
distribution different from the training distribution is crucial for robust
deployment and generalizability. The RVL-CDIP corpus [18] is the _de facto_
standard benchmark for document classification, yet to our knowledge all
studies that use this corpus do not include evaluation on _out-of-
distribution_ documents. In this paper, we curate and release a new out-of-
distribution benchmark for evaluating out-of-distribution performance for
document classifiers. Our new out-of-distribution benchmark consists of two
types of documents: those that are not part of any of the 16 in-domain RVL-
CDIP categories (RVL-CDIP-O), and those that are one of the 16 in-domain
categories yet are drawn from a distribution different from that of the
original RVL-CDIP dataset (RVL-CDIP-N). While prior work on document
classification for in-domain RVL-CDIP documents reports high accuracy scores,
we find that these models exhibit accuracy drops of between roughly 15-30% on
our new out-of-domain RVL-CDIP-N benchmark, and further struggle to
distinguish between in-domain RVL-CDIP-N and out-of-domain RVL-CDIP-_O_
inputs. Our new benchmark provides researchers with a valuable new resource
for analyzing out-of-distribution performance on document classifiers. Our new
out-of-distribution data can be found at https://tinyurl.com/4he6my23.

##  1 Introduction

The task of automated document classification has wide-ranging use cases,
especially in industry where it can be used to apply labels to massive amounts
of documents. In many scenarios, a model is trained on an initial training
set, which is drawn from a particular distribution which may have certain
characteristics (e.g., all documents are in the English language, are from a
particular industry or time period, etc.). It is often desirable for the model
to be able to _generalize_ to valid inputs that are _out-of-distribution_ and
exhibit different characteristics than those seen in the _in-distribution_
training data. At the same time, it is desirable for a model operating in
unconstrained input spaces to be able to identify inputs that are not part of
the target _label_ set as _out-of-domain_ to minimize false-positive
predictions.

The RVL-CDIP corpus has emerged as the de facto benchmark dataset for document
classification. This single-label classification dataset covers 16 document
categories (e.g., resume, invoice, letter, memo, etc.), and consists of
samples taken from a large collection of documents made public as a result of
litigation and settlement agreements involving the tobacco industry. As such,
most of the RVL-CDIP documents are related to the American tobacco industry,
and were created before 2006. Samples of RVL-CDIP are displayed in Figure 1.

Many contemporary models achieve high classification accuracy scores on the
RVL-CDIP classification benchmark (e.g., [34, 47, 57, 58]). Despite this,
there has been limited analysis of these models’ ability to handle out-of-
distribution inputs. In this paper, we address this gap by creating a new out-
of-distribution (OOD) test set intended to be used in conjunction with RVL-
CDIP. We first identify two types of OOD inputs: (1) in-domain documents that
fall within one of the 16 RVL-CDIP categories, yet were sampled from a
different distribution than RVL-CDIP (for example, born-digital documents from
the 2010s and 2020s, and those not from the tobacco industry); and (2) out-of-
domain documents that do not fall within one of the 16 RVL-CDIP categories.
Then, we evaluate several document classifiers trained on RVL-CDIP and tested
on our new OOD data. We find that model performance on both types of OOD data
is substantially lower than on the in-distribution RVL-CDIP test set,
indicating that while models may perform well on RVL-CDIP, they (1) struggle
to generalize to other distributions and (2) struggle on the task of out-of-
domain document classification. Our hope is that with our new OOD data,
researchers investigating document classification will also begin to include
OOD performance as part of their analyses.

Figure 1: Samples from the RVL-CDIP corpus.

##  2 Related Work

###  2.1 RVL-CDIP Document Classification

The RVL-CDIP corpus [18] is widely used for benchmarking the performance of
document classifiers. This corpus consists of images of scanned documents, and
most early efforts on the RVL-CDIP classification task used convolutional
neural networks (CNNs), including [1, 10, 18, 27, 52]. Later work incorporated
the textual modality along with visual features (e.g., [3, 4, 11, 15]). The
recent advent of transformers has seen researchers employ this architecture to
the classification task. This includes models that solely use image features
(e.g., [12, 25, 34]) as well as models that incorporate page layout and
optical character recognition (OCR) features (e.g., [44, 2, 6, 13, 16, 33, 37,
46, 47, 48, 54, 57, 58]). Despite the prominence of the RVL-CDIP corpus in
evaluating document classifiers, there is no published OOD corpus for
benchmarking models on out-of-distribution performance.

###  2.2 Out-of-Distribution Performance Benchmarking

Prior work has developed benchmarks for evaluating out-of-distribution
performance for other tasks. In general, these benchmarks belong to one of two
categories: (1) out-of-domain or out-of-scope benchmarks, where—in the case of
classification—dedicated evaluation data is included that does not belong to
any of the in-domain target labels; and (2) distribution shift benchmarks,
where evaluation data is different than the training distribution yet still
belongs to the in-domain target label set. We discuss relevant work in each
category.

_Out-of-scope_ or _out-of-domain_ evaluation benchmarks are used to measure a
model’s ability to differentiate between in-domain and out-of-domain inputs.
In this setting, a model is typically trained on the in-domain data, and then
evaluated on both in- and out-of-domain data (where ideally the model can
correctly classify in-domain data with high accuracy while also being able to
discriminate between in- and out-of-domain data, which could be done using a
confidence score). Out-of-domain benchmarks include those for text
classification (e.g., intent classification in dialog systems [30]) and image
classification and object detection in the presence of _natural adversarial_
inputs that mislead classifiers (e.g., ImageNet-O [23] and its COCO
counterpart [31]). In this paper, we present new out-of-domain evaluation data
for the RVL-CDIP benchmark, which we call RVL-CDIP-O.

_Distribution shift_ evaluation benchmarks are crafted to measure a model’s
ability to generalize to "unseen" inputs, as well as to measure how robust a
model is to perturbations and/or corruptions (e.g., noise). Here, "unseen"
typically means data that is different from the training distribution, yet
still belongs to the label-set of the training dataset. For image
classification, the "seen" training distribution may be photographic pictures
of images but the "unseen" data distribution may be different "views" or
modalities of these images, like hand-drawn sketches or clip art
representations of the image categories (e.g., DomainNet [45] and Office-Home
[53]).

Other domain-shift benchmarks include the WILDSbenchmark [26], which itself is
a collection of 10 mostly image and text datasets for evaluating out-of-
distribution performance. For instance, WILDS includes the Camelyon17
histopathology dataset [7], where the task is image classification but where
the test set consists of data collected from a hospital different from that of
the training set. Other prior work (e.g., [24, 35, 36]) evaluates out-of-
distribution detection for image classifiers trained on CIFAR-10/100 by
testing on various other image datasets, but does not account for the problem
of discriminating between out-of-distribution inputs that are out-of-domain
versus in-domain.

Lastly, datasets that measure a model’s robustness to corruptions include
ImageNet-C [20], which uses data augmentation strategies to generate noisy
versions of ImageNet [14] images. We note that other data augmentation tools
and pipelines have been used to generate perturbations and corruptions of in-
distribution data, including [40, 43, 49, 39] for image and text data. In this
paper, we present new evaluation data for benchmarking both of the out-of-
distribution tasks described in this section. Our new benchmark data enables
researchers to analyze document classifier performance on out-of-domain inputs
as well as in-domain data from a "shifted" input distribution.

Figure 2: Example out-of-distribution document images from our RVL-CDIP-_O_
set, prior to grayscale transformation.

##  3 Datasets

In this section, we present our new evaluation data for benchmarking out-of-
distribution performance on document classification models. We first discuss
the RVL-CDIP dataset as it is relevant in constructing our new out-of-
distribution data. We then discuss our new out-of-distribution benchmark.

###  3.1 RVL-CDIP

The RVL-CDIP corpus consists of grayscale images of scanned documents from the
IIT-CDIP collection [32], a large repository of publicly-available documents
that were released as part of litigation against several tobacco-related
companies and organizations. As such, all documents in the RVL-CDIP corpus are
tobacco-related. The corpus consists of 16 categories: advertisement, budget,
email, file_folder, form, handwritten, invoice, letter, memo, news_article,
presentation, questionnaire, resume, scientific_publication,
scientific_report, and specification. Samples from RVL-CDIP are shown in
Figure 1. We note that many of the RVL-CDIP samples contain visual noise
introduced by scanners due to having been scanned from the physical world
using old or noisy document scanners. Each RVL-CDIP category has 20,000
training samples (320,000 total training samples), and there is a total of
40,000 validation and 40,000 test documents across all categories. All
documents from this dataset are from the year 2006 or earlier, as 2006 was the
year that the IIT-CDIP collection was released.

###  3.2 New Out-of-Distribution Data

Figure 3: Samples from the RVL-CDIP-N evaluation set, prior to grayscale
transformation. Table 1: Per-category counts for RVL-CDIP-N.

Category | Count
---|---
budget | 58
email | 33
form | 70
handwritten | 176
invoice | 57
letter | 152
memo | 47
news_article | 86
questionnaire | 39
resume | 184
scientific_pub. | 39
specification | 61
Total | 1,002

Our new out-of-distribution benchmark consists of two subsets of data:
document images that (a) do not belong to any of the 16 in-domain RVL-CDIP
categories (we call this subset RVL-CDIP-_O_); and (b) belong to one of the 16
RVL-CDIP categories yet are not from IIT-CDIP or tobacco-related (RVL-CDIP-
_N_). Examples from RVL-CDIP-_O_ are shown in Figure 2, while examples from
RVL-CDIP-_N_ are shown in Figure 3. The samples in Figure 2 show several types
of documents that are not included in the 16 RVL-CDIP categories,
including—but not limited to—court documents, guides, music sheets, timelines,
and schedules.

Our out-of-distribution documents were collected from two internet sources:
(1) Google and Bing web searches; and (2) the public DocumentCloud
111https://www.documentcloud.org repository. The DocumentCloud repository
contains a large number of government, legal, or public service related
documents that were made available through serviced Freedom of Information Act
(FOIA) requests. To collect the new OOD data, the authors of this paper first
familiarized themselves with the 16 in-domain RVL-CDIP categories by reviewing
numerous samples from each RVL-CDIP category. While reviewing samples from
RVL-CDIP, we noticed that some could be seen as having multiple labels—for
instance, a handwritten letter could technically be both handwritten and
letter. However, we found that RVL-CDIP is consistent in labelling such cases
as with a single label (e.g., handwritten letters are labeled as handwritten).
The annotations we applied to our new RVL-CDIP-_N_ adopt such rules from RVL-
CDIP.

To collect RVL-CDIP-O data, the authors searched for PDF documents that were
not one of the 16 RVL-CDIP categories. Similarly for RVL-CDIP-N, the authors
searched for PDF documents that were one of the 16 RVL-CDIP categories. While
original descriptions of the RVL-CDIP categories are unavailable, we sought to
include only clear-cut examples of these categories in our RVL-CDIP-_N_ set;
samples from each RVL-CDIP-_N_ category are juxtaposed with original RVL-CDIP
samples in Figures 20 – 43 (in Appendix B). Using our search method, we were
unable to find sufficient documents for a few of the in-domain categories like
file_folder, as that category seems to be quite specific to the RVL-CDIP
dataset. Counts of the in-domain categories that we collected for RVL-CDIP-_N_
are shown in Table 1.

Each raw document may have consisted of multiple pages, but to maintain
consistency between our out-of-distribution data and the original RVL-CDIP
data, we only use the first page of each document, which we convert to a
grayscale image. Following the RVL-CDIP dataset’s convention, we save each
image so that the maximum dimension (i.e., height or width) is 1000 pixels,
and like RVL-CDIP we save the OOD images to TIFF files. As summarized in Table
2, there are a total of 4,417 out-of-distribution document images. We estimate
that roughly 65% of documents that we collected from DocumentCloud are "born
digital" (i.e., they are "clean" and not scanned versions of physical paper
documents), while around 95% of documents that we collected via web search are
born digital. For this reason we also conduct a side experiment with the
Augraphy tool [39] to add scanner-like noise to our out-of-distribution data
as a way to test whether models trained on RVL-CDIP are overfit to scanner-
like noise. Examples of our data post-Augraphy are shown in Figure 4. Our
final, published data for both RVL-CDIP-_N_ and RVL-CDIP-_O_ do not include
Augraphy augmentations.

Table 2: Out-of-distribution dataset count statistics. DC and WS refer to
DocumentCloud and Web Search, respectively.

| DC | WS | Total
---|---|---|---
RVL-CDIP-_N_ | 877 | 125 | 1,002
RVL-CDIP-_O_ | 1,382 | 2,033 | 3,415
Total | 2,259 | 2,158 | 4,417

Our new data is out-of-distribution with respect to the RVL-CDIP corpus in
several ways: (1) a substantial portion of our new data is "born-digital",
whereas the overwhelming majority of RVL-CDIP is scanned physical documents;
(2) a substantial portion of our data was created post-2006, with a large
amount having been created within the past 10 years; (3) documents from our
dataset are almost exclusively from industries and topics other than tobacco-
related ones. As such, our new out-of-distribution data poses a challenge to
models trained on RVL-CDIP, as models evaluated on our new data must be able
to generalize to new distributions (RVL-CDIP-N), while also being able to
distinguish between in- and out-of-domain inputs (RVL-CDIP-O).

Figure 4: Examples of document images from the RVL-CDIP-_O_ set (top row) with
corresponding Augraphy-augmented grayscale images (bottom row).

##  4 Experiments

Recall that our proposed dataset is intended to reveal deficiencies in
recognizing out-of-distribution data in document classification tasks. We
demonstrate our dataset’s viability by (1) training several classifiers on the
full RVL-CDIP training set, then (2) evaluating the performance of each model
on our new out-of-distribution data. We discuss the models and metrics used in
our experiments, and then discuss the results obtained.

###  4.1 Models

We trained several image-based classifiers on the full RVL-CDIP training set,
including convolutional neural network (CNN) architectures and image
transformer architectures. We consider the following CNNs: VGG-16, ResNet-50,
GoogLeNet, and AlexNet (each uses the architecture from [50, 19, 51, 28],
respectively). All of these CNN models were originally trained and evaluated
on the RVL-CDIP corpus by [1]. Our analysis uses PyTorch [41] implementations
of these models. In addition, we evaluate the LayoutLMv2-base model
architecture [58], which uses a language model trained on the locations and
sizes of words in documents. As the original RVL-CDIP corpus does not provide
textual information, we use the Tesseract 222https://github.com/tesseract-
ocr/tesseract OCR engine to extract words and their locations in each document
image. Our analysis uses the Hugging Face [56] implementation of LayoutLMv2.

We also evaluate the image-only transformer-based DiT model [34], which itself
is a version of the BEiT image transformer [8]. Recent work has shown that
image transformers often outperform CNN models, and prior work has also found
that transformer models outperform other model architectures in terms of out-
of-distribution performance (e.g., [9, 22, 38, 42]; on the contrary, cf. [5,
55] for opposing viewpoints). Unlike the other models analyzed here, we use
the pre-trained DiT model hosted on Hugging Face instead of training the model
ourselves. All models have an input image resolution of 224x224 pixels.

###  4.2 Evaluation Metrics

##### RVL-CDIP-_N_.

We measure each model’s raw classification accuracy on our RVL-CDIP-_N_
dataset, as RVL-CDIP-_N_ contains document images that do belong to one of the
16 in-domain RVL-CDIP document categories. Accuracy can be measured by
aggregating all predictions together (micro accuracy), or by averaging the
within-category accuracy scores (macro accuracy); we report both.

##### RVL-CDIP-_O_.

The documents from RVL-CDIP-_O_ do not belong to any of the 16 RVL-CDIP
categories, and we do not train the models on a $17^{th}$ catch-all unknown
category for out-of-domain documents. Therefore, we benchmark two decision
functions that use the model’s output logits as a way to arbitrate between in-
and out-of-domain predictions. The first, termed Maximum Softmax Probability
(MSP) obtains confidence scores from applying the softmax function on the
logits (as done in e.g. [21, 29, 30]). If the model $f$ applied to a document
image $x$ produces logits $f(x)=z$, then MSP applies the softmax function to
obtain confidence scores scaled between 0 and 1 for each in-domain category.
That is, the $i^{th}$ document category has a confidence score of

$s(z)_{i}=\frac{e^{z_{i}}}{\sum_{j=1}^{K}e^{z_{j}}},$

where $K$ is the number of target label categories (in our case, 16). An
alternative confidence function that has been shown to perform better on out-
of-distribution detection tasks than MSP is the Energy score [36], defined as

$E(z;T)=T\cdot log\sum_{j=1}^{K}e^{-z_{j}/T}$

where $T$ is a temperature parameter. Note that the Energy score produces a
single confidence score rather than $K$ scores.

An ideal classifier will assign high confidence to an in-domain document, and
lower confidence to an out-of-domain document. One straightforward way to do
this is to use a threshold on the confidence score, so that we assign an
unknown label to documents that have a confidence score lower than t. That is,
the decision rule is

$\text{decision rule}=\begin{cases}\text{in-domain},&\text{if}\
\text{max}(score(z))\geq t\\\ \text{out-of-domain},&\text{if}\
\text{max}(score(z))<t.\end{cases}$

Note that t could be global or specific to each document category (as noted in
[17]). Our goal is then to measure how well the decision rule is able to
correctly identify in- versus out-of-domain inputs (i.e., this is a binary
classification problem). While we could set a threshold (or thresholds) in our
analysis, we instead opt for a threshold-free evaluation that uses Area Under
the Reciever Operating Characteristic curve (AUC) scores to measure the
separability of confidence scores between in- and out-of-domain documents. The
AUC score ranges between 0.5 and 1.0, where higher AUC scores mean a model is
more able to assign lower confidence to out-of-domain documents and higher
confidence to in-domain documents, while a low AUC score of 0.5 means the
distributions of confidence scores for in- and out-of-domain documents are
roughly the same.

In our analysis, we compute the AUC between the confidence scores on RVL-CDIP
test set and RVL-CDIP-_O_ , as well as RVL-CDIP-_N_ and RVL-CDIP-_O_ (the
latter pair being a more challenging task, as both RVL-CDIP-_N_ and RVL-CDIP-
_O_ are out-of-distribution). Like accuracy, we report both macro and micro
AUC scores for each model, which is important for the RVL-CDIP-_O_ data
because the distribution of confidence scores may vary between predicted label
categories. Macro AUC is computed by computing the AUC for each document
category, where the samples used in the computation of each category’s AUC
score are those samples where the model predicted the sample as that category.
Then, each category’s AUC is averaged to yield the macro AUC score. We also
compute the false-positive rate at 95% true positive rate (FPR95), but these
results are presented in Appendix A.3. To compute FPR95, a threshold $t$ is
selected so that the true-positive rate is 95%, and then compute the false-
positive prediction rate for out-of-domain documents at this threshold. FPR95
is a lower-is-better metric.

Table 3:  Accuracy scores on RVL-CDIP compared to RVL-CDIP-_N_.

| RVL-CDIP | RVL-CDIP | RVL-CDIP-_N_ | RVL-CDIP-_N_ | RVL-CDIP-_N_ | RVL-CDIP-N
---|---|---|---|---|---|---
Model | (reported) | (achieved) | micro | micro w/Aug. | macro | macro w/Aug.
VGG-16 | 0.910 | 0.905 | 0.668 | 0.650 | 0.691 | 0.678
ResNet-50 | 0.911 | 0.900 | 0.598 | 0.606 | 0.615 | 0.621
GoogLeNet | 0.884 | 0.871 | 0.602 | 0.594 | 0.613 | 0.591
AlexNet | 0.900 | 0.885 | 0.567 | 0.559 | 0.571 | 0.551
LayoutLMv2 | 0.953 | 0.887 | 0.556 | 0.563 | 0.600 | 0.606
DiT-base | 0.921 | 0.933 | 0.786 | 0.706 | 0.805 | 0.725

###  4.3 Results

Table 3 displays accuracy on the RVL-CDIP test set as reported by prior work,
as well as the scores achieved by us. Table 3 also displays accuracy scores on
our new RVL-CDIP-N test set. First, we note that we achieve in-domain RVL-CDIP
test scores that are close to the reported scores, with the exception of
LayoutLMv2. One difference between our setup and the LayoutLMv2 setup used in
[58] is the OCR engine used: we used the open-source Tesseract library whereas
[58] used a proprietary Microsoft Azure OCR model. In other words, OCR quality
may have impacted our trained LayoutLMv2 model.

####  4.3.1 Performance on RVL-CDIP-_N_

Table 3 shows that the models appear to underperform on the out-of-
distribution document images from the new RVL-CDIP-N set, with micro accuracy
scores ranging in the mid 0.500s to 0.786 (in the case of DiT). These accuracy
scores are substantially lower than the scores on the in-distribution RVL-CDIP
test set. Even the top-performing DiT model sees an micro accuracy drop of
roughly 15 percentage points. Examples of errors made by the DiT model on RVL-
CDIP-N can be seen in Figure 7 (bottom row). Using the Augraphy augmentation
tool to introduce scanner-like noise to the RVL-CDIP-_N_ test images does
little to impact accuracy, and we observe no clear trend under this setting.
In summary, we observe that models trained on RVL-CDIP exhibit substantially
lower accuracy scores on our out-of-distribution RVL-CDIP-N evaluation set
than on the original RVL-CDIP test set, indicating that these models do not
generalize well to new data.

####  4.3.2 Performance on RVL-CDIP-_O_

Table 4 charts AUC scores on the task of out-of-domain detection for models
trained on RVL-CDIP but tested on the out-of-domain data from our new RVL-
CDIP-O data. This table shows AUC scores for model confidence score output on
RVL-CDIP versus RVL-CDIP-_O_ (_T-O_) as well as RVL-CDIP-_N_ versus RVL-CDIP-
_O_ (_N-O_); both of these settings explore a model’s ability to differentiate
between in-domain (RVL-CDIP and RVL-CDIP-_N_) versus out-of-domain (RVL-CDIP-
_O_) data, but the _N-O_ setting is more challenging, as both RVL-CDIP-_N_ and
RVL-CDIP-_O_ are out-of-distribution with respect to RVL-CDIP. We see that
most models achieve AUC scores on RVL-CDIP versus RVL-CDIP-_O_ (_T-O_) within
the mid-to-high 0.800s or low 0.900s for both MSP and Energy confidence
scoring methods. In contrast, models perform much worse on the task of
discriminating between RVL-CDIP-_N_ and RVL-CDIP-_O_ (_N-O_)—with DiT dropping
in performance by 16.6 points for MSP and 13.5 points for Energy—indicating
that the document classification models have much more difficulty detecting
out-of-domain documents in the presence of in-domain but out-of-distribution
documents.

Table 4:  AUC scores on RVL-CDIP and RVL-CDIP-_O_ (_T-O_) and on RVL-CDIP-_N_
and RVL-CDIP-_O_ (_N-O_) for MSP and Energy confidence scoring methods.

|  _T_ -_O_ |  _T_ -_O_ |  _N_ -_O_ |  _N_ -_O_
---|---|---|---|---
Model | MSP | Energy | MSP | Energy
VGG-16 | 0.881 | 0.923 | 0.649 | 0.648
ResNet-50 | 0.849 | 0.844 | 0.581 | 0.554
GoogLeNet | 0.838 | 0.847 | 0.592 | 0.587
AlexNet | 0.871 | 0.909 | 0.620 | 0.646
LayoutLMv2 | 0.842 | 0.849 | 0.620 | 0.662
DiT-base | 0.894 | 0.888 | 0.728 | 0.753

Figure 6 casts light as to why, as we see that there is a large amount of
overlap between confidence scores for RVL-CDIP-_N_ and RVL-CDIP-_O_ documents
for both MSP and Energy approaches using the DiT model. Similar plots for the
other models can be found in Figures 8–19 in Appendix A.4, where we also plot
model performance against confidence score threshold. To correctly identify
more out-of-domain documents— _ceteris paribus_ —we must naturally increase
confidence score thresholds. Thus we observe in Figures 8–19 that
classification accuracy on RVL-CDIP-_N_ naturally decreases as we increase the
confidence threshold, as more in-domain documents are mis-detected as out-of-
domain at higher confidence thresholds.

Figure 5: Distributions of predicted labels for each supervised model on the
out-of-domain RVL-CDIP-_O_ data.

(a)  (b)

Figure 6: Distribution of confidence scores using MSP (left) and Energy
(right) for DiT on RVL-CDIP-_N_ and RVL-CDIP-_O_.

We delve further into the predictions made by these models by investigating
the distribution of labels for the predictions on the RVL-CDIP-O test data:
Figure 5 plots the predicted label distribution for the data from RVL-CDIP-
_O_. We observe several trends: most of the supervised models have relatively
higher amounts of predictions for advertisement, presentation,
scientific_publication, and resume labels, perhaps due to these categories
having similar visual features to much of the RVL-CDIP-O data. Examples of
high-confidence predictions on RVL-CDIP-O can be seen in Figure 7 (top row).
Overall, our findings of model performance on differentiating between in- and
out-of-domain inputs show room for improvement in the case where both in- and
out-of-domain documents are out-of-distribution with respect to the original
RVL-CDIP training set.

##  5 Conclusion

In this paper, we introduce new data for evaluating out-of-distribution
performance on document classification models trained on the popular RVL-CDIP
benchmark. Our new out-of-distribution data is composed of two types: RVL-
CDIP-N, or data that _do belong_ to one of the 16 in-domain document
categories in RVL-CDIP yet were obtained from a different distribution than
the original RVL-CDIP corpus; and RVL-CDIP-O, or out-of-domain data that do
not belong to one of the 16 RVL-CDIP label categories. We find that while
models trained on RVL-CDIP perform well on the in-distribution RVL-CDIP test
data, they struggle comparatively on the out-of-distribution RVL-CDIP-N data
as well as at differentiating between the out-of-domain RVL-CDIP-_O_ and in-
domain RVL-CDIP-_N_ data.

Analyzing model performance on out-of-distribution inputs is an important yet
understudied problem, but is a hurdle for developing robust, generalizable
models. This seems to be the case in the document classification field, where
most analyses rely only on in-distribution and in-domain tests performed on
RVL-CDIP. Our hope is that our new out of distribution data—RVL-CDIP-N and
RVL-CDIP-O—provide researchers with a valuable resource for analyzing out-of-
distribution performance on document classifiers.

Figure 7: DiT model mis-predictions. Top row: errors on the RVL-CDIP-_O_ set.
Bottom row: errors on the RVL-CDIP-_N_ set. True labels are indicated by _y_ ,
while predicted labels are indicated with _ŷ_ (and confidence scores in
parentheses).

## Acknowledgments

We thank the University of Michigan’s Undergraduate Research Opportunity
Program (UROP) for supporting Gordon and Yutong. We also thank the University
of Michigan’s MRADS program for supporting Gordon. We thank Ann Arbor SPARK
and the Michigan STEM Forward program for partial support of David. We thank
Brian Yang for his help in early phases of data collection.

## References

  * [1] M. Afzal, A. Kölsch, S. Ahmed, and M. Liwicki.  Cutting the error by half: Investigation of very deep CNN and advanced training strategies for document image classification.  In Proceedings of the International Conference on Document Analysis and Recognition (ICDAR), 2017.
  * [2] S. Appalaraju, B. Jasani, B. Kota, Y. Xie, and R. Manmatha.  Docformer: End-to-end transformer for document understanding.  In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
  * [3] M. Asim, M. Khan, M. Malik, K. Razzaque, A. Dengel, and S. Ahmed.  Two stream deep network for document image classification.  In Proceedings of the International Conference on Document Analysis and Recognition (ICDAR), 2019.
  * [4] N. Audebert, C. Herold, K. Slimani, and C. Vidal.  Multimodal deep networks for text and image-based document classification.  In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2019.
  * [5] Y. Bai, J. Mei, A. Yuille, and C. Xie.  Are transformers more robust than CNNs?  In Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS), 2021.
  * [6] S. Bakkali, Z. Ming, M. Coustaty, M. Rusiñol, and O. Ramos Terrades.  VLCDoC: Vision-language contrastive pre-training model for cross-modal document classification.  arXiv preprint arXiv:2205.12029, 2022.
  * [7] P. Bandi, O. Geessink, Q. Manson, M. Van Dijk, M. Balkenhol, M. Hermsen, B. E. Bejnordi, B. Lee, K. Paeng, A. Zhong, et al.  From detection of individual metastases to classification of lymph node status at the patient level: The CAMELYON17 challenge.  IEEE Transactions on Medical Imaging, 2018.
  * [8] H. Bao, L. Dong, S. Piao, and F. Wei.  BEiT: BERT pre-training of image transformers.  In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
  * [9] S. Bhojanapalli, A. Chakrabarti, D. Glasner, D. Li, T. Unterhiner, and A. Veit.  Understanding robustness of transformers for image classification.  In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
  * [10] A. Das, S. Roy, U. Bhattacharya, and S. Parui.  Document image classification with intra-domain transfer learning and stacked generalization of deep convolutional neural networks.  In Proceedings of the 24th International Conference on Pattern Recognition (ICPR), 2018.
  * [11] T. Dauphinee, N. Patel, and M. Rashidi.  Modular multimodal architecture for document classification.  arXiv preprint arXiv:1912.04376, 2019.
  * [12] B. Davis, B. Morse, B. Price, C. Tensmeyer, C. Wigington, and V. Morariu.  End-to-end document recognition and understanding with dessurt.  arxiv preprint arxiv:2203.16618v3, 2022.
  * [13] T. Delteil, E. Velval, L. Chen, and L. Goncalves.  MATrIX - modality-aware transformer for information extraction.  arxiv preprint arxiv:2205.08094v1, 2022.
  * [14] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei.  ImageNet: A large-scale hierarchical image database.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
  * [15] J. Ferrando, J. Domínguez, J. Torres, R. García, D. García, D. Garrido, J. Cortada, and M. Valero.  Improving accuracy and speeding up document image classification through parallel systems.  In Proceedings of the International Conference on Computational Science (ICCS), 2020.
  * [16] J. Gu, J. Kuen, V. Morariu, H. Zhao, N. Barmpalios, R. Jain, A. Nenkova, and T. Sun.  Unified pretraining framework for document understanding.  In Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS), 2021.
  * [17] M. Guarrera, B. Jin, T.-W. Lin, M. A. Zuluaga, Y. Chen, and A. Sangiovanni-Vincentelli.  Class-wise thresholding for robust out-of-distribution detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2022.
  * [18] A. W. Harley, A. Ufkes, and K. G. Derpanis.  Evaluation of deep convolutional nets for document image classification and retrieval.  In Proceedings of the International Conference on Document Analysis and Recognition (ICDAR), 2015.
  * [19] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [20] D. Hendrycks and T. Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
  * [21] D. Hendrycks and K. Gimpel.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
  * [22] D. Hendrycks, X. Liu, E. Wallace, A. Dziedzic, R. Krishnan, and D. Song.  Pretrained transformers improve out-of-distribution robustness.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
  * [23] D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song.  Natural adversarial examples.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [24] Y.-C. Hsu, Y. Shen, H. Jin, and Z. Kira.  Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [25] G. Kim, T. Hong, M. Yim, J. Nam, J. Park, J. Yim, W. Hwang, S. Yun, D. Han, and S. Park.  Donut: Document understanding transformer without OCR.  arXiv preprint arXiv:2111.15664, 2021.
  * [26] P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Yasunaga, R. L. Phillips, I. Gao, T. Lee, E. David, I. Stavness, W. Guo, B. Earnshaw, I. Haque, S. M. Beery, J. Leskovec, A. Kundaje, E. Pierson, S. Levine, C. Finn, and P. Liang.  Wilds: A benchmark of in-the-wild distribution shifts.  In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5637–5664. PMLR, 18–24 Jul 2021.
  * [27] A. Kölsch, M. Afzal, M. Ebbecke, and M. Liwicki.  Real-time document image classification using deep cnn and extreme learning machines.  In Proceedings of the International Conference on Document Analysis and Recognition (ICDAR), 2017.
  * [28] A. Krizhevsky, I. Sutskever, and G. Hinton.  ImageNet classification with deep convolutional neural networks.  In Proceedings of the 25th International Conference on Neural Information Processing Systems, 2012.
  * [29] S. Larson and K. Leach.  Redwood: Using collision detection to grow a large-scale intent classification dataset.  arXiv preprint arXiv:2204.05483, 2022.
  * [30] S. Larson, A. Mahendran, J. Peper, C. Clarke, A. Lee, P. Hill, J. Kummerfeld, K. Leach, M. Laurenzano, L. Tang, and J. Mars.  An evaluation dataset for intent classification and out-of-scope prediction.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019.
  * [31] F. Lau, N. Subramani, S. Harrison, A. Kim, E. Branson, and R. Liu.  Natural adversarial objects.  arXiv preprint arXiv:2111.04204, 2021.
  * [32] D. Lewis, G. Agam, S. Argamon, O. Frieder, D. Grossman, and J. Heard.  Building a test collection for complex document information processing.  In Proceedings of the 29th Annual International ACL SIGIR Conference (SIGIR), 2006.
  * [33] C. Li, B. Bi, M. Yan, W. Wang, S. Huang, F. Huang, and L. Si.  Structurallm: Structural pre-training for form understanding.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP), 2021.
  * [34] J. Li, Y. Xu, T. Lv, L. Cui, C. Zhang, and F. Wei.  DiT: Self-supervised pre-training for document image transformer.  arXiv preprint arXiv:2203.02378, 2022.
  * [35] S. Liang, Y. Li, and R. Srikant.  Enhancing the reliability of out-of-distribution image detection in neural networks.  In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
  * [36] W. Liu, X. Wang, J. Owens, and Y. Li.  Energy-based out-of-distribution detection.  Advances in Neural Information Processing Systems, 2020.
  * [37] C. Luo, G. Tang, Q. Zheng, C. Yao, L. Jin, C. Li, Y. Xue, and L. Si.  Bi-VLDoc: Bidirectional vision-language modeling for visually-rich document understanding.  arxiv preprint arxiv:2206.13155, 2022.
  * [38] K. Mahmood, R. Mahmood, and M. van Dijk.  On the robustness of vision transformers to adversarial examples.  In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
  * [39] S. Maini, A. Groleau, K. W. Chee, S. Larson, and J. Boarman.  Augraphy: A data augmentation library for document images.  arXiv preprint arXiv:2208.14558, 2022.
  * [40] Z. Papakipos and J. Bitton.  Augly: Data augmentations for robustness.  arXiv preprint arXiv:2201.06494, 2022.
  * [41] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala.  PyTorch: An imperative style, high-performance deep learning library.  In Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS), 2019.
  * [42] S. Paul and P. Chen.  Vision transformers are robust learners.  In Proceedings of the 36th AAAI Conference on Artificial Intelligence (AAAI), 2022.
  * [43] B. Peng, C. Li, Z. Zhang, C. Zhu, J. Li, and J. Gao.  RADDLE: An evaluation benchmark and analysis platform for robust task-oriented dialog systems.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP), 2021.
  * [44] Q. Peng, Y. Pan, W. Wang, B. Luo, Z. Zhang, Z. Huang, T. Hu, W. Yin, Y. Chen, Y. Zhang, S. Feng, Y. Sun, H. Tian, H. Wu, and H. Wang.  ERNIE-layout: Layout knowledge enhanced pre-training for visually-rich document understanding.  arxiv preprint arxiv:2210.06155, 2022.
  * [45] X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang.  Moment matching for multi-source domain adaptation.  In Proceedings of the International Conference on Computer Vision (ICCV), 2019.
  * [46] H. Pham, G. Wang, Y. Lu, D. Florencio, and C. Zhang.  Understanding long documents with different position-aware attentions.  arxiv preprint arxiv:2208.08201, 2022.
  * [47] R. Powalski, . Borchmann, D. Jurkiewicz, T. Dwojak, M. Pietruszka, and G. Pałka.  Going full-TILT boogie on document understanding with text-image-layout transformer.  In Proceedings of the International Conference on Document Analysis and Recognition (ICDAR), 2021.
  * [48] S. Pramanik, S. Mujumdar, and H. Patel.  Towards a multi-modal, multi-task learning based pre-training framework for document representation learning.  arXiv preprint arXiv:2009.14457, 2022.
  * [49] M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh.  Beyond accuracy: Behavioral testing of NLP models with CheckList.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
  * [50] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
  * [51] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich.  Going deeper with convolutions.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [52] C. Tensmeyer and T. Martinez.  Analysis of convolutional neural networks for document image classification.  In Proceedings of the International Conference on Document Analysis and Recognition (ICDAR), 2017.
  * [53] H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchunathan.  Deep hashing network for unsupervised domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [54] J. Wang, L. Jin, and K. Ding.  LiLT: A simple yet effective language-independent layout transformer for structured document understanding.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), 2022.
  * [55] Z. Wang, Y. Bai, Y. Zhou, and C. Xie.  Can CNNs be more robust than transformers?  arXiv preprint arXiv:2206.03452, 2022.
  * [56] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush.  Transformers: State-of-the-art natural language processing.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020.
  * [57] Y. Xu, M. Li, L. Cui, S. Huang, F. Wei, and M. Zhou.  LayoutLM: Pre-training of text and layout for document image understanding.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD), 2020.
  * [58] Y. Xu, Y. Xu, T. Lv, F. Wei, G. Wang, Y. Lu, D. Florencio, C. Zhang, W. Che, M. Zhang, and L. Zhou.  LayoutLMv2: Multi-modal pre-training for visually-rich document understanding.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP), 2021.

##  Appendix A Additional Results

###  A.1 RVL-CDIP-_N_ Confusion Matrices

Table 5 displays accuracy scores for each document category in RVL-CDIP-_N_.
There are several patterns: all models perform well on scientific_publication
but typically poorly on handwritten and specification. (See Figures 30 and 31
for a comparison between specification documents from RVL-CDIP-_N_ and RVL-
CDIP, and Figures 24 and 25 for a comparison between handwritten documents.)
Tables 8 – 13 display model confusion matrices on RVL-CDIP-_N_.

###  A.2 Model Prediction Similarity

Tables 6 and 7 show pairwise similarity scores for each model on RVL-CDIP-_N_
and RVL-CDIP-_O_ , respectively. Here, we compute similarity between a pair of
models by computing the Hamming similarity between predicted labels, or

$sim(\textbf{y}_{a},\textbf{y}_{b})=\frac{1}{N}\sum_{i=1}^{N}\mathds{1}(\textbf{y}_{a}^{i}=\textbf{y}_{b}^{i})$

where $\textbf{y}_{a}^{i}$ is model $a$’s predicted label on the $i^{th}$ test
document. We find that there is a higher degree of similarity in model
predictions on RVL-CDIP-_N_ than on RVL-CDIP-_O_.

###  A.3 In- versus Out-of-Domain Performance

Tables 14 and 15 chart AUC scores for MSP on RVL-CDIP versus RVL-CDIP-_O_
(Table 14) and on RVL-CDIP-_N_ versus RVL-CDIP-_O_ (Table 15). Tables 16 and
17 chart AUC scores for Energy on RVL-CDIP versus RVL-CDIP-_O_ (Table 17) and
on RVL-CDIP-_N_ versus RVL-CDIP-_O_ (Table 17). We find that the Augraphy
augmentations typically have little impact. The main finding in Tables 14 – 17
is that out-of-domain detection suffers on the more realistic RVL-CDIP-_N_
versus RVL-CDIP-_O_ setting, where both sets of test documents are out-of-
distribution. This is in contrast with the _T-O_ setting where we use the in-
distribution RVL-CDIP test set as the in-domain data. We report similar
findings using the FPR95 metric in Tables 18 – 21.

###  A.4 Confidence Scores

Figures 8 – 19 show relationships between confidence scores and performance on
RVL-CDIP and RVL-CDIP-_N_ accuracy as well as RVL-CDIP-_O_ detection rate. We
see that as confidence score threshold increases, the detection rate on RVL-
CDIP-_O_ increases while accuracy on RVL-CDIP and RVL-CDIP-_N_ naturally
decreases due to the decision rule defined in 4.2. Figures 8 – 19 also display
distributions of model confidence scores for RVL-CDIP-_N_ and RVL-CDIP-_O_. We
see that there is a large amount of overlap between the two distributions for
all models and for both MSP and Energy confidence score methods, which helps
explain the drop in AUC scores on the _N-O_ (RVL-CDIP-_N_ versus RVL-CDIP-_O_)
setting.

Table 5: Per-class accuracy scores on RVL-CDIP-N for each document
classification model.

Model |  budget |  email |  form |  handwritten |  invoice |  letter |  memo |  news_article |  questionnaire |  resume |  scientific_pub. |  specification
---|---|---|---|---|---|---|---|---|---|---|---|---
VGG-16 | 0.793 | 0.848 | 0.743 | 0.403 | 0.737 | 0.901 | 0.553 | 0.686 | 0.718 | 0.696 | 0.974 | 0.230
ResNet-50 | 0.810 | 0.758 | 0.714 | 0.358 | 0.421 | 0.803 | 0.511 | 0.570 | 0.641 | 0.674 | 0.949 | 0.148
GoogLeNet | 0.776 | 0.818 | 0.700 | 0.449 | 0.439 | 0.816 | 0.553 | 0.616 | 0.513 | 0.609 | 0.923 | 0.115
AlexNet | 0.690 | 0.727 | 0.443 | 0.352 | 0.526 | 0.836 | 0.404 | 0.558 | 0.487 | 0.663 | 0.949 | 0.148
LayoutLMv2 | 0.897 | 0.848 | 0.529 | 0.261 | 0.333 | 0.836 | 0.511 | 0.512 | 0.769 | 0.565 | 0.923 | 0.164
DiT | 0.862 | 0.970 | 0.914 | 0.624 | 0.860 | 0.954 | 0.723 | 0.849 | 0.821 | 0.734 | 0.923 | 0.410

|  VGG-16 |  ResNet-50 |  GoogLeNet |  AlexNet |  LayoutLMv2 |  DiT
---|---|---|---|---|---|---
VGG-16 | 1.00 | 0.63 | 0.61 | 0.62 | 0.57 | 0.65
ResNet-50 |  | 1.00 | 0.58 | 0.57 | 0.57 | 0.58
GoogLeNet |  |  | 1.00 | 0.56 | 0.54 | 0.57
AlexNet |  |  |  | 1.00 | 0.55 | 0.56
LayoutLMv2 |  |  |  |  | 1.00 | 0.55
DiT |  |  |  |  |  | 1.00

Table 6: Pairwise similarity between predictions made by each model on RVL-
CDIP-N.

|  VGG-16 |  ResNet-50 |  GoogLeNet |  AlexNet |  LayoutLMv2 |  DiT
---|---|---|---|---|---|---
VGG-16 | 1.00 | 0.45 | 0.43 | 0.44 | 0.42 | 0.27
ResNet-50 |  | 1.00 | 0.46 | 0.42 | 0.43 | 0.33
GoogLeNet |  |  | 1.00 | 0.41 | 0.40 | 0.30
AlexNet |  |  |  | 1.00 | 0.39 | 0.28
LayoutLMv2 |  |  |  |  | 1.00 | 0.30
DiT |  |  |  |  |  | 1.00

Table 7: Pairwise similarity between predictions made by each model on RVL-
CDIP-O. Table 8: Confusion matrix for VGG-16 on the RVL-CDIP-_N_ data. True
labels are rows, and predicted labels are columns.

|  advertisement |  budget |  email |  file_folder |  form |  handwritten |  invoice |  letter |  memo |  news_article |  presentation |  questionnaire |  resume |  scientific_pub. |  scientific_rep. |  specification
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
advertisement | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
budget | 0.000 | 0.793 | 0.017 | 0.000 | 0.034 | 0.000 | 0.017 | 0.000 | 0.000 | 0.000 | 0.000 | 0.052 | 0.000 | 0.017 | 0.052 | 0.000
email | 0.000 | 0.061 | 0.848 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.030 | 0.061 | 0.000 | 0.000 | 0.000
file_folder | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
form | 0.000 | 0.043 | 0.000 | 0.000 | 0.743 | 0.000 | 0.014 | 0.014 | 0.000 | 0.000 | 0.043 | 0.029 | 0.029 | 0.000 | 0.000 | 0.086
handwritten | 0.023 | 0.051 | 0.000 | 0.051 | 0.040 | 0.403 | 0.011 | 0.091 | 0.080 | 0.000 | 0.006 | 0.028 | 0.011 | 0.011 | 0.193 | 0.000
invoice | 0.000 | 0.105 | 0.000 | 0.000 | 0.035 | 0.000 | 0.737 | 0.070 | 0.018 | 0.000 | 0.000 | 0.018 | 0.000 | 0.000 | 0.018 | 0.000
letter | 0.000 | 0.007 | 0.020 | 0.007 | 0.000 | 0.000 | 0.000 | 0.901 | 0.000 | 0.013 | 0.033 | 0.000 | 0.020 | 0.000 | 0.000 | 0.000
memo | 0.000 | 0.021 | 0.043 | 0.000 | 0.021 | 0.000 | 0.000 | 0.128 | 0.553 | 0.000 | 0.043 | 0.043 | 0.085 | 0.021 | 0.043 | 0.000
news_article | 0.128 | 0.012 | 0.000 | 0.023 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.686 | 0.023 | 0.000 | 0.012 | 0.093 | 0.023 | 0.000
presentation | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
questionnaire | 0.000 | 0.051 | 0.000 | 0.000 | 0.077 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.718 | 0.051 | 0.043 | 0.077 | 0.026
resume | 0.022 | 0.011 | 0.022 | 0.000 | 0.033 | 0.000 | 0.016 | 0.016 | 0.000 | 0.000 | 0.016 | 0.049 | 0.696 | 0.974 | 0.033 | 0.033
scientific_pub. | 0.000 | 0.000 | 0.000 | 0.000 | 0.026 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.974 | 0.000 | 0.000
scientific_rep. | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
specification | 0.197 | 0.022 | 0.049 | 0.000 | 0.131 | 0.000 | 0.098 | 0.000 | 0.000 | 0.016 | 0.049 | 0.033 | 0.033 | 0.098 | 0.033 | 0.230

Table 9: Confusion matrix for ResNet-50 on the RVL-CDIP-_N_ data. True labels
are rows, and predicted labels are columns.

|  advertisement |  budget |  email |  file_folder |  form |  handwritten |  invoice |  letter |  memo |  news_article |  presentation |  questionnaire |  resume |  scientific_pub. |  scientific_rep. |  specification
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
advertisement | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
budget | 0.000 | 0.810 | 0.000 | 0.000 | 0.052 | 0.000 | 0.017 | 0.000 | 0.000 | 0.000 | 0.000 | 0.052 | 0.000 | 0.017 | 0.069 | 0.000
email | 0.000 | 0.061 | 0.758 | 0.000 | 0.030 | 0.000 | 0.000 | 0.030 | 0.000 | 0.000 | 0.000 | 0.030 | 0.061 | 0.000 | 0.000 | 0.030
file_folder | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
form | 0.028 | 0.029 | 0.000 | 0.000 | 0.714 | 0.000 | 0.000 | 0.014 | 0.000 | 0.000 | 0.000 | 0.086 | 0.014 | 0.000 | 0.029 | 0.100
handwritten | 0.028 | 0.051 | 0.006 | 0.045 | 0.034 | 0.358 | 0.000 | 0.063 | 0.063 | 0.006 | 0.074 | 0.034 | 0.068 | 0.006 | 0.148 | 0.017
invoice | 0.018 | 0.123 | 0.018 | 0.000 | 0.123 | 0.000 | 0.421 | 0.070 | 0.018 | 0.000 | 0.053 | 0.018 | 0.000 | 0.000 | 0.035 | 0.018
letter | 0.007 | 0.000 | 0.053 | 0.000 | 0.007 | 0.000 | 0.000 | 0.803 | 0.000 | 0.000 | 0.033 | 0.000 | 0.046 | 0.000 | 0.033 | 0.000
memo | 0.043 | 0.000 | 0.064 | 0.000 | 0.021 | 0.000 | 0.000 | 0.000 | 0.511 | 0.021 | 0.170 | 0.000 | 0.043 | 0.000 | 0.106 | 0.021
news_article | 0.221 | 0.023 | 0.023 | 0.035 | 0.012 | 0.000 | 0.000 | 0.000 | 0.000 | 0.570 | 0.047 | 0.000 | 0.000 | 0.058 | 0.000 | 0.012
presentation | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
questionnaire | 0.000 | 0.051 | 0.000 | 0.000 | 0.103 | 0.000 | 0.000 | 0.000 | 0.026 | 0.000 | 0.000 | 0.641 | 0.026 | 0.000 | 0.103 | 0.026
resume | 0.027 | 0.027 | 0.033 | 0.005 | 0.023 | 0.000 | 0.005 | 0.011 | 0.000 | 0.011 | 0.049 | 0.054 | 0.674 | 0.033 | 0.027 | 0.016
scientific_pub. | 0.000 | 0.000 | 0.000 | 0.000 | 0.051 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.949 | 0.000 | 0.000
scientific_rep. | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
specification | 0.180 | 0.033 | 0.016 | 0.049 | 0.148 | 0.000 | 0.049 | 0.000 | 0.000 | 0.000 | 0.098 | 0.082 | 0.033 | 0.082 | 0.082 | 0.148

Table 10: Confusion matrix for GoogLeNet on the RVL-CDIP-_N_ data. True labels
are rows, and predicted labels are columns.

|  advertisement |  budget |  email |  file_folder |  form |  handwritten |  invoice |  letter |  memo |  news_article |  presentation |  questionnaire |  resume |  scientific_pub. |  scientific_rep. |  specification
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
advertisement | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
budget | 0.000 | 0.776 | 0.052 | 0.000 | 0.052 | 0.000 | 0.017 | 0.000 | 0.000 | 0.000 | 0.000 | 0.069 | 0.000 | 0.000 | 0.052 | 0.000
email | 0.000 | 0.000 | 0.818 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.091 | 0.000 | 0.061 | 0.030
file_folder | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
form | 0.014 | 0.029 | 0.000 | 0.000 | 0.700 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.0129 | 0.029 | 0.000 | 0.000 | 0.086
handwritten | 0.017 | 0.067 | 0.017 | 0.051 | 0.023 | 0.449 | 0.006 | 0.051 | 0.057 | 0.006 | 0.034 | 0.017 | 0.074 | 0.011 | 0.125 | 0.000
invoice | 0.000 | 0.070 | 0.000 | 0.000 | 0.211 | 0.000 | 0.439 | 0.088 | 0.018 | 0.000 | 0.035 | 0.018 | 0.053 | 0.000 | 0.070 | 0.000
letter | 0.000 | 0.000 | 0.013 | 0.007 | 0.000 | 0.000 | 0.000 | 0.816 | 0.020 | 0.007 | 0.033 | 0.000 | 0.072 | 0.000 | 0.020 | 0.007
memo | 0.000 | 0.021 | 0.021 | 0.000 | 0.021 | 0.000 | 0.000 | 0.085 | 0.553 | 0.021 | 0.064 | 0.000 | 0.213 | 0.000 | 0.000 | 0.000
news_article | 0.279 | 0.000 | 0.000 | 0.012 | 0.000 | 0.012 | 0.000 | 0.000 | 0.000 | 0.616 | 0.024 | 0.000 | 0.000 | 0.035 | 0.000 | 0.023
presentation | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
questionnaire | 0.000 | 0.103 | 0.026 | 0.000 | 0.026 | 0.000 | 0.000 | 0.026 | 0.000 | 0.000 | 0.051 | 0.513 | 0.128 | 0.000 | 0.0103 | 0.026
resume | 0.011 | 0.011 | 0.027 | 0.000 | 0.038 | 0.000 | 0.000 | 0.000 | 0.005 | 0.038 | 0.082 | 0.098 | 0.609 | 0.109 | 0.049 | 0.023
scientific_pub. | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.026 | 0.026 | 0.000 | 0.000 | 0.923 | 0.026 | 0.000
scientific_rep. | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
specification | 0.115 | 0.115 | 0.000 | 0.016 | 0.148 | 0.000 | 0.033 | 0.016 | 0.000 | 0.033 | 0.082 | 0.016 | 0.082 | 0.164 | 0.066 | 0.115

Table 11: Confusion matrix for AlexNet on the RVL-CDIP-_N_ data. True labels
are rows, and predicted labels are columns.

|  advertisement |  budget |  email |  file_folder |  form |  handwritten |  invoice |  letter |  memo |  news_article |  presentation |  questionnaire |  resume |  scientific_pub. |  scientific_rep. |  specification
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
advertisement | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
budget | 0.000 | 0.690 | 0.017 | 0.034 | 0.052 | 0.000 | 0.034 | 0.000 | 0.000 | 0.000 | 0.000 | 0.052 | 0.017 | 0.017 | 0.052 | 0.034
email | 0.091 | 0.061 | 0.727 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.030 | 0.000 | 0.000 | 0.060 | 0.000 | 0.000 | 0.030
file_folder | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
form | 0.029 | 0.071 | 0.000 | 0.000 | 0.443 | 0.000 | 0.071 | 0.029 | 0.029 | 0.000 | 0.014 | 0.071 | 0.071 | 0.000 | 0.000 | 0.171
handwritten | 0.034 | 0.051 | 0.000 | 0.051 | 0.023 | 0.352 | 0.023 | 0.159 | 0.091 | 0.006 | 0.040 | 0.011 | 0.017 | 0.000 | 0.142 | 0.000
invoice | 0.000 | 0.105 | 0.000 | 0.000 | 0.175 | 0.000 | 0.526 | 0.123 | 0.000 | 0.000 | 0.000 | 0.035 | 0.035 | 0.000 | 0.000 | 0.000
letter | 0.013 | 0.000 | 0.039 | 0.000 | 0.000 | 0.000 | 0.000 | 0.836 | 0.007 | 0.000 | 0.007 | 0.000 | 0.079 | 0.007 | 0.013 | 0.000
memo | 0.021 | 0.021 | 0.085 | 0.000 | 0.085 | 0.000 | 0.085 | 0.000 | 0.404 | 0.000 | 0.043 | 0.043 | 0.213 | 0.021 | 0.043 | 0.000
news_article | 0.302 | 0.116 | 0.000 | 0.023 | 0.000 | 0.023 | 0.000 | 0.000 | 0.000 | 0.558 | 0.000 | 0.000 | 0.000 | 0.058 | 0.023 | 0.000
presentation | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
questionnaire | 0.051 | 0.000 | 0.026 | 0.000 | 0.026 | 0.000 | 0.000 | 0.026 | 0.000 | 0.000 | 0.026 | 0.487 | 0.128 | 0.000 | 0.154 | 0.026
resume | 0.033 | 0.022 | 0.038 | 0.000 | 0.038 | 0.000 | 0.005 | 0.005 | 0.005 | 0.005 | 0.033 | 0.016 | 0.663 | 0.033 | 0.065 | 0.043
scientific_pub. | 0.000 | 0.000 | 0.000 | 0.000 | 0.026 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.949 | 0.026 | 0.000
scientific_rep. | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
specification | 0.115 | 0.115 | 0.016 | 0.016 | 0.016 | 0.016 | 0.131 | 0.016 | 0.000 | 0.033 | 0.033 | 0.016 | 0.016 | 0.131 | 0.033 | 0.148

Table 12: Confusion matrix for LayoutLMv2 on the RVL-CDIP-_N_ data. True
labels are rows, and predicted labels are columns.

|  advertisement |  budget |  email |  file_folder |  form |  handwritten |  invoice |  letter |  memo |  news_article |  presentation |  questionnaire |  resume |  scientific_pub. |  scientific_rep. |  specification
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
advertisement | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
budget | 0.000 | 0.897 | 0.017 | 0.017 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.052 | 0.017 | 0.000 | 0.000 | 0.000
email | 0.000 | 0.000 | 0.848 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.030 | 0.030 | 0.061 | 0.000 | 0.000 | 0.000
file_folder | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
form | 0.014 | 0.271 | 0.000 | 0.014 | 0.529 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.086 | 0.043 | 0.000 | 0.000 | 0.043
handwritten | 0.125 | 0.097 | 0.011 | 0.045 | 0.028 | 0.261 | 0.006 | 0.216 | 0.000 | 0.000 | 0.068 | 0.017 | 0.034 | 0.006 | 0.080 | 0.006
invoice | 0.000 | 0.509 | 0.000 | 0.000 | 0.070 | 0.000 | 0.333 | 0.000 | 0.000 | 0.000 | 0.035 | 0.018 | 0.035 | 0.000 | 0.000 | 0.000
letter | 0.007 | 0.007 | 0.026 | 0.000 | 0.000 | 0.007 | 0.000 | 0.836 | 0.007 | 0.000 | 0.046 | 0.000 | 0.039 | 0.000 | 0.026 | 0.000
memo | 0.021 | 0.043 | 0.043 | 0.000 | 0.021 | 0.000 | 0.000 | 0.106 | 0.511 | 0.021 | 0.128 | 0.043 | 0.043 | 0.000 | 0.021 | 0.000
news_article | 0.290 | 0.023 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.512 | 0.012 | 0.000 | 0.012 | 0.140 | 0.012 | 0.000
presentation | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
questionnaire | 0.000 | 0.128 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.051 | 0.769 | 0.051 | 0.000 | 0.000 | 0.000
resume | 0.065 | 0.082 | 0.023 | 0.000 | 0.023 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.087 | 0.076 | 0.565 | 0.016 | 0.033 | 0.033
scientific_pub. | 0.026 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.026 | 0.923 | 0.000 | 0.026
scientific_rep. | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
specification | 0.279 | 0.230 | 0.016 | 0.000 | 0.115 | 0.000 | 0.016 | 0.000 | 0.000 | 0.000 | 0.034 | 0.016 | 0.016 | 0.066 | 0.049 | 0.164

Table 13: Confusion matrix for DiT on the RVL-CDIP-_N_ data. True labels are
rows, and predicted labels are columns.

|  advertisement |  budget |  email |  file_folder |  form |  handwritten |  invoice |  letter |  memo |  news_article |  presentation |  questionnaire |  resume |  scientific_pub. |  scientific_rep. |  specification
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
advertisement | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
budget | 0.017 | 0.862 | 0.017 | 0.000 | 0.034 | 0.000 | 0.000 | 0.000 | 0.000 | 0.017 | 0.000 | 0.000 | 0.000 | 0.000 | 0.052 | 0.000
email | 0.000 | 0.000 | 0.970 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.030 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
file_folder | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
form | 0.000 | 0.029 | 0.029 | 0.000 | 0.914 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.00 | 0.029 | 0.000 | 0.000 | 0.000 | 0.000
handwritten | 0.000 | 0.017 | 0.006 | 0.011 | 0.034 | 0.642 | 0.011 | 0.085 | 0.057 | 0.006 | 0.085 | 0.006 | 0.000 | 0.000 | 0.040 | 0.000
invoice | 0.018 | 0.035 | 0.035 | 0.000 | 0.035 | 0.000 | 0.860 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.018 | 0.000
letter | 0.013 | 0.000 | 0.007 | 0.000 | 0.000 | 0.000 | 0.000 | 0.954 | 0.000 | 0.000 | 0.026 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
memo | 0.000 | 0.000 | 0.064 | 0.000 | 0.021 | 0.000 | 0.000 | 0.149 | 0.723 | 0.000 | 0.021 | 0.000 | 0.000 | 0.000 | 0.021 | 0.000
news_article | 0.105 | 0.000 | 0.012 | 0.000 | 0.000 | 0.000 | 0.000 | 0.012 | 0.000 | 0.849 | 0.012 | 0.000 | 0.000 | 0.012 | 0.000 | 0.000
presentation | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
questionnaire | 0.000 | 0.000 | 0.000 | 0.000 | 0.103 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.821 | 0.000 | 0.000 | 0.077 | 0.000
resume | 0.022 | 0.000 | 0.049 | 0.000 | 0.005 | 0.000 | 0.000 | 0.000 | 0.000 | 0.016 | 0.005 | 0.000 | 0.734 | 0.038 | 0.076 | 0.054
scientific_pub. | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.026 | 0.000 | 0.000 | 0.923 | 0.051 | 0.000
scientific_rep. | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | —
specification | 0.115 | 0.066 | 0.000 | 0.000 | 0.230 | 0.000 | 0.000 | 0.000 | 0.000 | 0.016 | 0.016 | 0.000 | 0.000 | 0.000 | 0.148 | 0.410

Table 14:  AUC scores using MSP on RVL-CDIP versus RVL-CDIP-_O_ (_T-O_) .

| _T-O_ | _T-O_ | _T-O_ | _T-O_
---|---|---|---|---
Model | micro | micro w/Aug. | macro | macro w/Aug.
VGG-16 | 0.881 | 0.867 | 0.895 | 0.883
ResNet-50 | 0.849 | 0.852 | 0.871 | 0.879
GoogLeNet | 0.838 | 0.839 | 0.859 | 0.862
AlexNet | 0.871 | 0.882 | 0.889 | 0.895
LayoutLMv2 | 0.842 | 0.832 | 0.875 | 0.878
DiT-base | 0.894 | 0.923 | 0.901 | 0.929

Table 15:  AUC scores using MSP on RVL-CDIP-_N_ versus RVL-CDIP-_O_ (_N-O_) .

| _N-O_ | _N-O_ | _N-O_ | _N-O_
---|---|---|---|---
Model | micro | micro w/Aug. | macro | macro w/Aug.
VGG-16 | 0.649 | 0.631 | 0.720 | 0.712
ResNet-50 | 0.581 | 0.584 | 0.654 | 0.658
GoogLeNet | 0.592 | 0.592 | 0.679 | 0.647
AlexNet | 0.620 | 0.646 | 0.684 | 0.635
LayoutLMv2 | 0.620 | 0.620 | 0.717 | 0.706
DiT-base | 0.728 | 0.777 | 0.780 | 0.754

Table 16:  AUC scores using Energy on RVL-CDIP versus RVL-CDIP-_O_ (_T-O_) .

| _T-O_ | _T-O_ | _T-O_ | _T-O_
---|---|---|---|---
Model | micro | micro w/Aug. | macro | macro w/Aug.
VGG-16 | 0.923 | 0.906 | 0.930 | 0.916
ResNet-50 | 0.844 | 0.868 | 0.880 | 0.899
GoogLeNet | 0.847 | 0.854 | 0.869 | 0.878
AlexNet | 0.909 | 0.920 | 0.922 | 0.932
LayoutLMv2 | 0.849 | 0.849 | 0.891 | 0.891
DiT-base | 0.888 | 0.933 | 0.902 | 0.936

Table 17:  AUC scores using Energy on RVL-CDIP-_N_ versus RVL-CDIP-_O_ (_N-O_)
.

| _N-O_ | _N-O_ | _N-O_ | _N-O_
---|---|---|---|---
Model | micro | micro w/Aug. | macro | macro w/Aug.
VGG-16 | 0.645 | 0.646 | 0.720 | 0.707
ResNet-50 | 0.554 | 0.583 | 0.661 | 0.671
GoogLeNet | 0.587 | 0.561 | 0.689 | 0.642
AlexNet | 0.646 | 0.628 | 0.706 | 0.655
LayoutLMv2 | 0.643 | 0.643 | 0.699 | 0.684
DiT-base | 0.753 | 0.731 | 0.792 | 0.764

Table 18:  FPR95 scores using MSP on RVL-CDIP versus RVL-CDIP-_O_ (_T-O_) .

| _T-O_ | _T-O_ | _T-O_ | _T-O_
---|---|---|---|---
Model | micro | micro w/Aug. | macro | macro w/Aug.
VGG-16 | 0.649 | 0.657 | 0.530 | 0.543
ResNet-50 | 0.731 | 0.733 | 0.649 | 0.599
GoogLeNet | 0.748 | 0.745 | 0.619 | 0.641
AlexNet | 0.702 | 0.664 | 0.592 | 0.582
LayoutLMv2 | 0.717 | 0.717 | 0.590 | 0.590
DiT-base | 0.587 | 0.497 | 0.461 | 0.401

Table 19:  FPR95 scores using MSP on RVL-CDIP-_N_ versus RVL-CDIP-_O_ (_N-O_)
.

| _T-O_ | _T-O_ | _T-O_ | _T-O_
---|---|---|---|---
Model | micro | micro w/Aug. | macro | macro w/Aug.
VGG-16 | 0.916 | 0.923 | 0.777 | 0.830
ResNet-50 | 0.935 | 0.933 | 0.834 | 0.832
GoogLeNet | 0.946 | 0.933 | 0.842 | 0.862
AlexNet | 0.919 | 0.916 | 0.785 | 0.850
LayoutLMv2 | 0.932 | 0.932 | 0.795 | 0.806
DiT-base | 0.847 | 0.886 | 0.650 | 0.707

Table 20:  FPR95 scores using Energy on RVL-CDIP versus RVL-CDIP-_O_ (_T-O_) .

| _T-O_ | _T-O_ | _T-O_ | _T-O_
---|---|---|---|---
Model | micro | micro w/Aug. | macro | macro w/Aug.
VGG-16 | 0.461 | 0.529 | 0.379 | 0.444
ResNet-50 | 0.650 | 0.603 | 0.527 | 0.482
GoogLeNet | 0.665 | 0.654 | 0.559 | 0.566
AlexNet | 0.525 | 0.423 | 0.433 | 0.378
LayoutLMv2 | 0.753 | 0.753 | 0.573 | 0.573
DiT-base | 0.688 | 0.375 | 0.415 | 0.336

Table 21:  FPR95 scores using Energy on RVL-CDIP-_N_ versus RVL-CDIP-_O_ (_N-
O_) .

| _T-O_ | _T-O_ | _T-O_ | _T-O_
---|---|---|---|---
Model | micro | micro w/Aug. | macro | macro w/Aug.
VGG-16 | 0.924 | 0.895 | 0.796 | 0.812
ResNet-50 | 0.942 | 0.935 | 0.784 | 0.850
GoogLeNet | 0.943 | 0.945 | 0.799 | 0.869
AlexNet | 0.937 | 0.931 | 0.779 | 0.847
LayoutLMv2 | 0.939 | 0.939 | 0.801 | 0.806
DiT-base | 0.843 | 0.852 | 0.614 | 0.663

(a)  (b)

Figure 8: Relationship between confidence scores and accuracy (left) and
distribution of confidence scores (right) for VGG-16 using MSP.

(a)  (b)

Figure 9: Relationship between confidence scores and accuracy (left) and
distribution of confidence scores (right) for VGG-16 using Energy.

(a)  (b)

Figure 10: Relationship between confidence scores and accuracy (left) and
distribution of confidence scores (right) for ResNet-50 using MSP.

(a)  (b)

Figure 11: Relationship between confidence scores and accuracy (left) and
distribution of confidence scores (right) for ResNet-50 using Energy.

(a)  (b)

Figure 12: 2Relationship between confidence scores and accuracy (left) and
distribution of confidence scores (right) for GoogLeNet using MSP.

(a)  (b)

Figure 13: Relationship between confidence scores and accuracy (left) and
distribution of confidence scores (right) for GoogLeNet using Energy.

(a)  (b)

Figure 14: Relationship between confidence scores and accuracy (left) and
distribution of confidence scores (right) for AlexNet using MSP.

(a)  (b)

Figure 15: Relationship between confidence scores and accuracy (left) and
distribution of confidence scores (right) for AlexNet using Energy.

(a)  (b)

Figure 16: Relationship between confidence scores and accuracy (a) and
distribution of confidence scores (b) for LayoutLMv2 using MSP.

(a)  (b)

Figure 17: Relationship between confidence scores and accuracy (left) and
distribution of confidence scores (right) for LayoutLMv2 using Energy.

(a)  (b)

Figure 18: Relationship between confidence scores and accuracy (left) and
distribution of confidence scores (right) for DiT using MSP.

(a)  (b)

Figure 19: Relationship between confidence scores and accuracy (left) and
distribution of confidence scores (right) for DiT using Energy.

##  Appendix B Comparison of RVL-CDIP and RVL-CDIP-_N_

Figures 20 – 43 compare samples from RVL-CDIP with those from RVL-CDIP-_N_.

Figure 20: Samples of invoice documents from RVL-CDIP-N.

Figure 21: Samples of invoice documents from RVL-CDIP.

Figure 22: Samples of budget documents from RVL-CDIP-N.

Figure 23: Samples of budget documents from RVL-CDIP.

Figure 24: Samples of handwritten documents from RVL-CDIP-N.

Figure 25: Samples of handwritten documents from RVL-CDIP.

Figure 26: Samples of form documents from RVL-CDIP-N.

Figure 27: Samples of form documents from RVL-CDIP.

Figure 28: Samples of letter documents from RVL-CDIP-N.

Figure 29: Samples of letter documents from RVL-CDIP.

Figure 30: Samples of specification documents from RVL-CDIP-N.

Figure 31: Samples of specification documents from RVL-CDIP.

Figure 32: Samples of news_article documents from RVL-CDIP-N.

Figure 33: Samples of news_article documents from RVL-CDIP.

Figure 34: Samples of scientific_publication documents from RVL-CDIP-N.

Figure 35: Samples of scientific_publication documents from RVL-CDIP.

Figure 36: Samples of resume documents from RVL-CDIP-N.

Figure 37: Samples of resume documents from RVL-CDIP.

Figure 38: Samples of questionnaire documents from RVL-CDIP-N.

Figure 39: Samples of questionnaire documents from RVL-CDIP.

Figure 40: Samples of memo documents from RVL-CDIP-N.

Figure 41: Samples of memo documents from RVL-CDIP.

Figure 42: Samples of email documents from RVL-CDIP-N.

Figure 43: Samples of email documents from RVL-CDIP.