# Enhancing In-Context Learning via Implicit Demonstration Augmentation

Xiaoling Zhou1, Wei Ye1111Corresponding authors. , Yidong Wang1, Chaoya
Jiang1, Zhemg Lee2,
Rui Xie1, Shikun Zhang1111Corresponding authors.
1National Engineering Research Center for Software Engineering, Peking
University, China
2Tianjin University, Tianjin, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The emergence of in-context learning (ICL) enables large pre-trained language
models (PLMs) to make predictions for unseen inputs without updating
parameters. Despite its potential, ICL’s effectiveness heavily relies on the
quality, quantity, and permutation of demonstrations, commonly leading to
suboptimal and unstable performance. In this paper, we tackle this challenge
for the first time from the perspective of demonstration augmentation.
Specifically, we start with enriching representations of demonstrations by
leveraging their deep feature distribution. We then theoretically reveal that
when the number of augmented copies approaches infinity, the augmentation is
approximately equal to a novel logit calibration mechanism integrated with
specific statistical properties. This insight results in a simple yet highly
efficient method that significantly improves the average and worst-case
accuracy across diverse PLMs and tasks. Moreover, our method effectively
reduces performance variance among varying demonstrations, permutations, and
templates, and displays the capability to address imbalanced class
distributions.

##  1 Introduction

Large pre-trained language models (PLMs) have showcased exceptional abilities
in in-context learning (ICL) Brown et al. (2020); Wang et al. (2023); Rubin et
al. (2022), which assists the model in discerning the underlying patterns
within demonstrations and make more accurate predictions Chan et al. (2022);
Wu et al. (2023). As a new paradigm, ICL offers compelling advantages,
allowing for natural language interaction with PLMs Wei et al. (2022); Yang et
al. (2023), as well as reduced computational costs Li et al. (2023a); Rubin et
al. (2022).

While promising, ICL’s performance is highly dependent on provided
demonstrations and templates Liu et al. (2022); Zhang et al. (2022b); Sorensen
et al. (2022), resulting in subpar and unstable performance. This promotes
research aimed at improving the quality Rubin et al. (2022); Li et al.
(2023b), quantity Li et al. (2023a); Choi et al. (2022), and permutations Lu
et al. (2022); Tang et al. (2023) of demonstrations. Other research avenues
include prediction adjustment Zhao et al. (2021); Han et al. (2023); Fei et
al. (2023) and learning process design (e.g., channel models Min et al.
(2022a) and meta-training frameworks Min et al. (2022b)). Despite ongoing
efforts, ICL still struggles with efficiently and reliably capturing
sufficient knowledge from context, leaving performance stability as a
persistent bottleneck.

Figure 1: Illustration for demonstration augmentation using semantic
directions (vectors) sampled from the deep feature distribution of
demonstration examples.

In this study, we propose enriching contextual knowledge for PLMs by
augmenting demonstrations. We first attempt to enhance the representation of
demonstrations by transforming them along semantic directions sampled from the
deep feature space of demonstration examples, as depicted in Figure 1. This
operation stems from the observation that the deep features in a network are
usually linearized Bengio et al. (2013); Cheung and Yeung (2021); Cho (2016),
implying the existence of numerous semantic directions within the deep feature
space, hence potentially enabling us to incorporate richer contextual
knowledge without extending input length. From this novel perspective, we
theoretically prove that when the number of augmented pieces approaches
infinity, its effect approximately equals a logit adjustment operation.
Specifically, we derive a refined Softmax function that integrates the
statistical properties of demonstrations. Consequently, rather than explicitly
executing the augmentation procedure, we can efficiently conduct implicit
demonstration augmentation using the derived prediction function, obtaining an
improved ICL method with theoretical guidance.

We conduct extensive experiments across seven PLMs and various classification
tasks. The empirical results demonstrate that our approach remarkably enhances
prediction accuracy and reduces performance variability across different
demonstrations, permutations, and templates. Notably, our method is
straightforward, effective, and generalizable, enabling seamless integration
with other ICL methods to enhance their performance.

Our contributions can be summarized as follows:

  * •

We introduce Implicit Demonstration Augmentation-based ICL (IDAICL), a
pioneering work that incorporates demonstration augmentation into ICL. Instead
of solely enhancing demonstration quality, quantity, or order, our method
explores context augmentation within the deep feature space, offering a new
perspective to enrich demonstrations bypassing input length limitations.

  * •

We theoretically establish that as the number of augmented pieces approaches
infinity, our augmentation strategy approximates a logit-adjusted prediction
function that integrates statistical properties derived from the input data
distribution. Equipped with this function, IDAICL provides a straightforward
yet theory-guided solution to enhance ICL.

  * •

Extensive experiments conducted across diverse tasks and PLMs conclusively
illustrate that IDAICL considerably improves average and worst-case accuracy
compared to existing ICL methods. Moreover, it effectively enhances
performance stability.

##  2 Background and Related Work

###  2.1 In-Context Learning

Brown et al. Brown et al. (2020) showcased the ICL capability of PLMs, wherein
PLMs generate predictions solely based on a concatenation of training examples
for few-shot learning without updating parameters. Subsequent studies Holtzman
et al. (2021); Min et al. (2022a, b) have developed this approach, yielding
promising outcomes across various tasks. Nevertheless, recent research has
uncovered certain limitations. To begin with, the volume of input knowledge
for each query is constrained by the maximum input length of PLMs Hao et al.
(2022), and the computational cost increases as the number of demonstrations
grows Li et al. (2023a), making it challenging to integrate significant
knowledge from demonstrations to PLMs. Additionally, ICL’s performance is
sensitive to the input of PLMs Davison et al. (2019); Jiang et al. (2020),
thus exhibiting high variance and poor worst-case accuracy Perez et al.
(2021); Lu et al. (2022).

Researchers have explored various techniques to address the biases and
instability of ICL. These techniques encompass learning process design Min et
al. (2022a, b), demonstration retrieval Rubin et al. (2022); Zhang et al.
(2022b), prompt engineering Sorensen et al. (2022); Lu et al. (2022), and
prediction calibration Zhao et al. (2021); Fei et al. (2023). However, these
methods have yet to fully address the issue of severely limited knowledge
transfer from demonstrations to large PLMs.

###  2.2 Data Augmentation

Figure 2: An overview of IDAICL: For each contextual input, our goal is to
augment the deep feature of demonstrations for $\mathcal{M}$ pieces, using
semantic vectors $\boldsymbol{\delta}$ drawn from the deep feature
distribution $\mathcal{N}(\boldsymbol{\mu},\boldsymbol{\Sigma})$ of
demonstration examples linked to all queries. When $\mathcal{M}$ approaches
infinity, we derive a novel prediction function, which incorporates two
modulating factors: $M(\boldsymbol{\mu})$ and $N(\boldsymbol{\Sigma})$, to
calibrate the original predictions.

Data augmentation Chen et al. (2023), which involves artificially creating
training data through transformations, is a well-established research area in
machine learning. Although data augmentation techniques have undergone
extensive exploration in diverse machine learning domains Maharana et al.
(2022); Shorten and Khoshgoftaar (2019), applying them to text data poses
challenges due to the complexity of preserving labels during textual
transformations Kobayashi (2018). Nonetheless, data augmentations in the
latent space, such as adversarial training Zhang et al. (2022a); Zhu et al.
(2020); Cheng et al. (2020), interpolation Chen et al. (2022b); Wu et al.
(2022), and generative techniques Li et al. (2022); Malandrakis et al. (2019),
have demonstrated notable enhancements when applied alongside large PLMs.

Recently, Wang et al. Wang et al. (2019) introduced the concept of implicit
data augmentation in the context of image classification. This approach
involves transforming training data within the deep feature space and boils
down to the optimization of a novel robust loss function. Subsequent studies
Chen et al. (2022c); Li et al. (2021); Zhou and Wu (2023a) for image
classification tasks have further improved upon this approach. This study
introduces an algorithm for implicitly augmenting demonstrations within the
realm of ICL.

##  3 Methodology

###  3.1 In-Context Learning with PLMs

Considering a PLM $\mathcal{G}$, this study focuses on the following task:
given a query input text $\boldsymbol{x}$ and a candidate answer set
$\mathcal{Y}\\!=\\!\\{y_{1},y_{2},\cdots,y_{|\mathcal{Y}|}\\}$, we aim to
predict the answer $\hat{y}$ based on $m$ demonstration examples
$\mathcal{C}\\!=\\!\\{{c}_{1},c_{2},\cdots,c_{m}\\}$, where each $c_{i}$
represents a training example $(\boldsymbol{x}_{i},y_{i})$ after template
formulation and $m$ denotes the quantity of demonstration examples for each
test sample. Formally, give a model $\mathcal{G}$, we first compute the
probability of each answer $y_{j}$:

$P_{\mathcal{G}}\left(y_{j}\mid\mathcal{C},\boldsymbol{x}\right).$ (1)

Subsequently, the ultimate prediction $\hat{y}$, characterized by the highest
probability is chosen from the candidate answer set $\mathcal{Y}$:

$\hat{y}=\arg\max_{y_{j}\in\mathcal{Y}}P_{\mathcal{G}}\left(y_{j}\mid\mathcal{C},\boldsymbol{x}\right).$
(2)

To simplify, the contextual input is denoted as
$\boldsymbol{\tilde{x}}\\!=\\![\mathcal{C},\boldsymbol{x}]$ in the subsequent
text. Then, the probability of answer $y_{j}$, represented as
$P_{\mathcal{G}}(y_{j}|\boldsymbol{\tilde{x}})$, is computed using the Softmax
function111We begin by examining situations in which the answer comprises a
single token, and our subsequent analysis is equally applicable to scenarios
involving multiple tokens.:

$P_{\mathcal{G}}(y_{j}|\boldsymbol{\tilde{x}}):=P_{\mathcal{G}}(y_{j}|\boldsymbol{h}_{\boldsymbol{\tilde{x}}})=\frac{e^{\boldsymbol{w}_{y_{j}}^{T}\boldsymbol{h}_{\tilde{\boldsymbol{x}}}+b_{y_{j}}}}{\sum_{k}e^{\boldsymbol{w}_{k}^{T}\boldsymbol{h}_{\tilde{\boldsymbol{x}}}+b_{k}}},$
(3)

where
$\boldsymbol{h}_{\tilde{\boldsymbol{x}}}=\mathcal{G}(\tilde{\boldsymbol{x}})$
signifies the hidden state of the last block at the final position for
$\tilde{\boldsymbol{x}}$. $\boldsymbol{w}_{k}$ and $b_{k}$ are the weight
vector and bias corresponding to the final fully connected layer for the
$k$-th token.

###  3.2 Demonstration Augmentation

Recognizing the established efficacy of data augmentation in machine learning
Feng et al. (2021), this study investigates demonstration augmentation and
suggests enhancing the deep features of demonstrations by transforming them
along semantic directions sampled from the deep feature space of demonstration
examples. This strategy is motivated by the intriguing observation that the
deep features in networks are often linearized Bengio et al. (2013); Chen et
al. (2022a). Building on this observation, we hypothesize that
$\boldsymbol{h}_{\tilde{\boldsymbol{x}}}$ lies within the subspace spanned by
$\boldsymbol{h}_{\mathcal{C}}$ and $\boldsymbol{h}_{{\boldsymbol{x}}}$:
$\boldsymbol{h}_{\tilde{\boldsymbol{x}}}=\alpha\boldsymbol{h}_{\mathcal{C}}+\beta\boldsymbol{h}_{\boldsymbol{x}}$,
where $\boldsymbol{h}_{\mathcal{C}}$ and $\boldsymbol{h}_{{\boldsymbol{x}}}$
represent the components of $\boldsymbol{h}_{\tilde{\boldsymbol{x}}}$ linked
respectively to the demonstrations and the query. The necessity of this
assumption stems from intricate relationships among token representations and
the exclusive augmentation of the component related to demonstrations.
Notably, this decomposition is not necessary in practical applications. In the
subsequent text, we directly refer to $\alpha\boldsymbol{h}_{\mathcal{C}}$ and
$\beta\boldsymbol{h}_{\boldsymbol{x}}$ as $\boldsymbol{h}_{\mathcal{C}}$ and
$\boldsymbol{h}_{\boldsymbol{x}}$.

To augment $\boldsymbol{h}_{\mathcal{C}}$, we randomly sample vectors from the
deep feature space of demonstrations. In particular, vectors are drawn from a
multivariate normal distribution
$\mathcal{N}(\boldsymbol{\mu},\boldsymbol{\Sigma})$, where $\boldsymbol{\mu}$
and $\boldsymbol{\Sigma}$ denote the feature mean and covariance matrix. These
statistical properties are estimated from the deep features of the
demonstration set $\mathcal{D}$, which includes demonstration examples linked
to all queries. The feature mean $\boldsymbol{\mu}$ is computed as

$\boldsymbol{\mu}=\frac{1}{|\mathcal{D}|}\sum\nolimits_{i=1}^{|\mathcal{D}|}\boldsymbol{h}_{i},$
(4)

where $\boldsymbol{h}_{i}=\mathcal{G}(c_{i})$ represents the hidden state of
the last block at the final position for the $i$-th demonstration example
$c_{i}$ in $\mathcal{D}$, and $|\mathcal{D}|$ denotes the size of
$\mathcal{D}$. The covariance matrix $\boldsymbol{\Sigma}$ is computed as

$\boldsymbol{\Sigma}=\frac{1}{|\mathcal{D}|}{\sum\nolimits_{i=1}^{|\mathcal{D}|}(\boldsymbol{h}_{i}-\boldsymbol{\mu})^{T}(\boldsymbol{h}_{i}-\boldsymbol{\mu})}.$
(5)

Subsequently, ${\boldsymbol{h}}_{\mathcal{C}}$ is shifted in the extracted
semantic vectors, resulting in augmented features,
$\tilde{\boldsymbol{h}}_{\mathcal{C}}$, which follows

$\tilde{\boldsymbol{h}}_{\mathcal{C}}\sim\mathcal{N}\left({\boldsymbol{h}}_{\mathcal{C}}+\lambda\boldsymbol{\mu},\lambda\boldsymbol{\Sigma}\right),$
(6)

where $\lambda$ refers to a positive coefficient controlling the strength of
semantic augmentation. In real-world applications, it can be directly assigned
a value of 0.5. Sensitivity tests for $\lambda$ are discussed in Section 5.4.

###  3.3 Novel Prediction Function

Selecting the answer with the highest probability is equivalent to favoring
the answer with the lowest inverse probability. Therefore, the prediction can
be determined by

$\hat{y}=\arg\min_{y_{j}\in\mathcal{Y}}P_{\mathcal{G}}\left(y_{j}\mid\boldsymbol{h}_{\tilde{\boldsymbol{x}}}\right)^{-1}.$
(7)

Assume that each $\boldsymbol{h}_{\mathcal{C}}$ is augmented for $\mathcal{M}$
times, resulting in an augmented demonstration feature set
$\\{\tilde{\boldsymbol{h}}_{\mathcal{C}}^{1},\cdots,\tilde{\boldsymbol{h}}_{\mathcal{C}}^{\mathcal{M}}\\}$
with size $\mathcal{M}$. Here, $\tilde{\boldsymbol{h}}_{\mathcal{C}}^{i}$
represents the $i$-th augmented feature for $\boldsymbol{h}_{\mathcal{C}}$.
Then, the final prediction for the query $\boldsymbol{x}$ depends on all
augmented features of $\boldsymbol{h}_{\mathcal{C}}$ and can be expressed as

${P}_{y_{j}}^{\mathcal{M}}(\boldsymbol{\tilde{x}})=\frac{1}{\mathcal{M}}{\sum\nolimits_{i=1}^{\mathcal{M}}P_{\mathcal{G}}(y_{j}|\tilde{\boldsymbol{h}}_{\mathcal{C}}^{i},\boldsymbol{h}_{\boldsymbol{x}})^{-1}},$
(8)
$\hat{y}={\arg\min_{y_{j}\in\mathcal{Y}}{P}_{y_{j}}^{\mathcal{M}}(\boldsymbol{\tilde{x}})}.$
(9)

Given that the performance of ICL benefits from an increased number of
demonstration instances Liu et al. (2022); Wu et al. (2023), we explore the
scenario of augmenting an infinite number of times for the deep representation
of demonstrations. Subsequently, an easily computable surrogate for the
expected prediction can be derived, resulting in a highly efficient
implementation. The whole pipeline of IDAICL is depicted in Figure 2.

As $\mathcal{M}\rightarrow\infty$, on the basis of the aforementioned
decomposition of $\boldsymbol{h}_{\tilde{\boldsymbol{x}}}$, the expected
prediction for answer $y_{j}$ (denoted as $P^{\infty}_{y_{j}}$) within the
augmented feature set can be expressed as follows:

$\displaystyle
P_{y_{j}}^{\infty}(\boldsymbol{\tilde{x}})\\!=\\!\mathbb{E}_{\boldsymbol{\tilde{h}}_{\mathcal{C}}}[{\sum_{k}e^{\Delta\boldsymbol{w}_{k,y_{j}}^{T}(\boldsymbol{\tilde{h}}_{\mathcal{C}}+\boldsymbol{h}_{\boldsymbol{x}})+\Delta
b_{k,y_{j}}}}],$ (10)

where
$\Delta\boldsymbol{w}_{k,y_{j}}=\boldsymbol{w}_{k}-\boldsymbol{w}_{y_{j}}$ and
$\Delta b_{k,y_{j}}=b_{k}-b_{y_{j}}$.

However, accurately calculating $P^{\infty}_{y_{j}}$ is challenging.
Alternatively, we proceed to derive a surrogate calculation for it. Applying
the linearity of expectation, Eq. (10) can be expressed as:

$\displaystyle
P_{y_{j}}^{\infty}(\boldsymbol{\tilde{x}})\\!=\\!\sum_{k}\mathrm{E}_{\tilde{\boldsymbol{h}}_{\mathcal{C}}}[e^{\Delta\boldsymbol{w}_{k,y_{j}}^{T}(\tilde{\boldsymbol{h}}_{\mathcal{C}}+\boldsymbol{h}_{\boldsymbol{x}})+\Delta
b_{k,y_{j}}}].$ (11)

Given that $\boldsymbol{\tilde{h}}_{\mathcal{C}}$ is a Gaussian random
variable conforming to
$\mathcal{N}\left({\boldsymbol{h}}_{\mathcal{C}}+\lambda\boldsymbol{\mu},\lambda\boldsymbol{\Sigma}\right)$,
we know that
$\Delta\boldsymbol{w}_{k,y_{j}}^{T}\tilde{\boldsymbol{h}}_{\mathcal{C}}$
follows the multivariate normal distribution:
$\mathcal{N}(\Delta\boldsymbol{w}_{k,y_{j}}^{T}\left({\boldsymbol{h}}_{\mathcal{C}}+\lambda\boldsymbol{\mu}\right),\lambda\Delta\boldsymbol{w}_{k,y_{j}}^{T}\boldsymbol{\Sigma}\Delta\boldsymbol{w}_{k,y_{j}})$.
Then, utilizing the moment-generating function

$\mathbb{E}[e^{{t}{X}}]=e^{{t}{\mu}+\frac{1}{2}{t}^{2}\sigma^{2}},X\sim\mathcal{N}(\mu,\sigma^{2}),$
(12)

Eq. (11) can be derived as

$\displaystyle
P_{y_{j}}^{\infty}(\boldsymbol{\tilde{x}})\\!=\\!\sum_{k}M_{k,y_{j}}N_{k,y_{j}}e^{\Delta\boldsymbol{w}_{k,y_{j}}^{T}(\boldsymbol{h}_{\mathcal{C}}+\boldsymbol{h}_{\boldsymbol{x}})+\Delta
b_{k,y_{j}}},$ (13)

where
$M_{k,y_{j}}=\exp(\lambda{\Delta\boldsymbol{w}_{k,y_{j}}^{T}\boldsymbol{\mu}})$
and
$N_{k,y_{j}}=\exp({{\frac{\lambda}{2}}\Delta\boldsymbol{w}_{k,y_{j}}^{T}\boldsymbol{\Sigma}\Delta\boldsymbol{w}_{k,y_{j}}})$.

PLM | Method | m | SST-2 | SST-5 | MR | CR | Amazon | Subj | TREC | DBPedia | AGNews | CB
---|---|---|---|---|---|---|---|---|---|---|---|---
GPT-2 0.8B | Vanilla ICL | 4 | ${57.6}_{{\color[rgb]{0,0,1}7.1}}$ | ${30.4}_{{\color[rgb]{0,0,1}6.3}}$ | ${59.3}_{{\color[rgb]{0,0,1}6.5}}$ | ${56.8}_{{\color[rgb]{0,0,1}8.4}}$ | $32.7_{{\color[rgb]{0,0,1}8.5}}$ | ${57.6}_{{\color[rgb]{0,0,1}5.4}}$ | ${34.9}_{{\color[rgb]{0,0,1}10.3}}$ | ${40.5}_{{\color[rgb]{0,0,1}7.2}}$ | ${44.5}_{{\color[rgb]{0,0,1}7.9}}$ | ${35.1}_{{\color[rgb]{0,0,1}9.3}}$
IDAICL | $86.4_{{\color[rgb]{0,0,1}1.4}}$ | $38.3_{{\color[rgb]{0,0,1}2.9}}$ | $82.2_{{\color[rgb]{0,0,1}2.3}}$ | $78.4_{{\color[rgb]{0,0,1}0.7}}$ | $46.7_{{\color[rgb]{0,0,1}3.5}}$ | $77.0_{{\color[rgb]{0,0,1}2.3}}$ | $47.5_{{\color[rgb]{0,0,1}2.0}}$ | $81.3_{{\color[rgb]{0,0,1}1.8}}$ | $73.9_{{\color[rgb]{0,0,1}2.4}}$ | $41.5_{{\color[rgb]{0,0,1}2.0}}$
Vanilla ICL | 8 | ${69.7}_{{\color[rgb]{0,0,1}9.0}}$ | ${32.4}_{{\color[rgb]{0,0,1}8.6}}$ | ${63.9}_{{\color[rgb]{0,0,1}7.7}}$ | ${60.8}_{{\color[rgb]{0,0,1}8.1}}$ | $34.1_{{\color[rgb]{0,0,1}6.2}}$ | ${59.7}_{{\color[rgb]{0,0,1}8.7}}$ | ${40.4}_{{\color[rgb]{0,0,1}6.3}}$ | ${62.6}_{{\color[rgb]{0,0,1}13.6}}$ | ${49.2}_{{\color[rgb]{0,0,1}8.4}}$ | ${38.8}_{{\color[rgb]{0,0,1}7.6}}$
IDAICL | $88.0_{{\color[rgb]{0,0,1}2.3}}$ | $39.6_{{\color[rgb]{0,0,1}1.9}}$ | $84.9_{{\color[rgb]{0,0,1}2.4}}$ | ${85.6}_{{\color[rgb]{0,0,1}2.5}}$ | $47.9_{{\color[rgb]{0,0,1}2.6}}$ | $79.9_{{\color[rgb]{0,0,1}0.8}}$ | ${50.3}_{{\color[rgb]{0,0,1}3.3}}$ | ${86.5}_{{\color[rgb]{0,0,1}2.9}}$ | $76.8_{{\color[rgb]{0,0,1}1.7}}$ | $43.3_{{\color[rgb]{0,0,1}3.4}}$
Vanilla ICL | 12 | ${74.7}_{{\color[rgb]{0,0,1}8.3}}$ | ${33.7}_{{\color[rgb]{0,0,1}7.6}}$ | ${64.4}_{{\color[rgb]{0,0,1}9.4}}$ | ${68.7}_{{\color[rgb]{0,0,1}9.7}}$ | ${36.0}_{{\color[rgb]{0,0,1}6.6}}$ | ${60.7}_{{\color[rgb]{0,0,1}7.7}}$ | ${40.5}_{{\color[rgb]{0,0,1}7.8}}$ | ${64.5}_{{\color[rgb]{0,0,1}5.4}}$ | ${51.1}_{{\color[rgb]{0,0,1}8.0}}$ | ${40.4}_{{\color[rgb]{0,0,1}8.5}}$
IDAICL | $88.5_{{\color[rgb]{0,0,1}2.1}}$ | $40.1_{{\color[rgb]{0,0,1}2.7}}$ | $85.2_{{\color[rgb]{0,0,1}3.1}}$ | $86.8_{{\color[rgb]{0,0,1}1.4}}$ | $49.6_{{\color[rgb]{0,0,1}2.2}}$ | $80.4_{{\color[rgb]{0,0,1}2.1}}$ | $51.4_{{\color[rgb]{0,0,1}1.6}}$ | $87.3_{{\color[rgb]{0,0,1}2.7}}$ | ${77.9}_{{\color[rgb]{0,0,1}2.0}}$ | $44.6_{{\color[rgb]{0,0,1}2.2}}$
MetaICL | 12 | ${80.8}_{{\color[rgb]{0,0,1}6.2}}$ | ${35.8}_{{\color[rgb]{0,0,1}4.7}}$ | ${75.3}_{{\color[rgb]{0,0,1}5.6}}$ | $77.6_{{\color[rgb]{0,0,1}8.1}}$ | $48.9_{{\color[rgb]{0,0,1}6.7}}$ | ${73.5}_{{\color[rgb]{0,0,1}8.8}}$ | $48.6_{{\color[rgb]{0,0,1}6.1}}$ | $80.4_{{\color[rgb]{0,0,1}7.8}}$ | ${66.8}_{{\color[rgb]{0,0,1}0.7}}$ | ${{43.1}}_{{\color[rgb]{0,0,1}4.1}}$
+IDAICL | ${89.3}_{{\color[rgb]{0,0,1}1.7}}$ | ${\underline{42.6}}_{{\color[rgb]{0,0,1}2.4}}$ | ${85.8}_{{\color[rgb]{0,0,1}1.7}}$ | ${87.9}_{{\color[rgb]{0,0,1}1.5}}$ | ${\underline{51.7}}_{{\color[rgb]{0,0,1}0.7}}$ | ${\underline{82.6}}_{{\color[rgb]{0,0,1}2.4}}$ | ${53.7}_{{\color[rgb]{0,0,1}2.5}}$ | ${\underline{89.4}}_{{\color[rgb]{0,0,1}4.1}}$ | $78.3_{{\color[rgb]{0,0,1}1.1}}$ | ${\boldsymbol{47.9}}_{{\color[rgb]{0,0,1}2.8}}$
Channel ICL | 12 | ${85.2}_{{\color[rgb]{0,0,1}3.6}}$ | ${38.4}_{{\color[rgb]{0,0,1}4.3}}$ | ${80.8}_{{\color[rgb]{0,0,1}4.7}}$ | ${82.0}_{{\color[rgb]{0,0,1}4.6}}$ | ${43.6}_{{\color[rgb]{0,0,1}5.1}}$ | ${69.8}_{{\color[rgb]{0,0,1}9.8}}$ | ${44.1}_{{\color[rgb]{0,0,1}8.7}}$ | ${77.6}_{{\color[rgb]{0,0,1}12.9}}$ | ${69.5}_{{\color[rgb]{0,0,1}6.7}}$ | ${42.4}_{{\color[rgb]{0,0,1}5.2}}$
+IDAICL | ${\boldsymbol{90.5}}_{{\color[rgb]{0,0,1}2.3}}$ | ${41.8}_{{\color[rgb]{0,0,1}2.7}}$ | ${\boldsymbol{87.7}}_{{\color[rgb]{0,0,1}1.6}}$ | ${\boldsymbol{89.5}}_{{\color[rgb]{0,0,1}1.2}}$ | ${50.8}_{{\color[rgb]{0,0,1}2.4}}$ | ${80.5}_{{\color[rgb]{0,0,1}0.9}}$ | ${52.9}_{{\color[rgb]{0,0,1}1.6}}$ | ${87.8}_{{\color[rgb]{0,0,1}2.4}}$ | ${\underline{81.0}}_{{\color[rgb]{0,0,1}2.5}}$ | ${46.3}_{{\color[rgb]{0,0,1}3.3}}$
EPR | 12 | ${81.9}_{{\color[rgb]{0,0,1}2.1}}$ | $39.9_{{\color[rgb]{0,0,1}1.8}}$ | $78.1_{{\color[rgb]{0,0,1}2.4}}$ | $80.6_{{\color[rgb]{0,0,1}0.6}}$ | $49.1_{{\color[rgb]{0,0,1}2.4}}$ | $80.1_{{\color[rgb]{0,0,1}2.2}}$ | ${\underline{76.2}}_{{\color[rgb]{0,0,1}1.1}}$ | ${{87.1}}_{{\color[rgb]{0,0,1}1.0}}$ | ${{80.9}}_{{\color[rgb]{0,0,1}0.8}}$ | $44.8_{{\color[rgb]{0,0,1}2.3}}$
+IDAICL | ${\underline{90.1}}_{{\color[rgb]{0,0,1}1.1}}$ | ${\boldsymbol{43.9}}_{{\color[rgb]{0,0,1}1.2}}$ | ${\underline{86.4}}_{{\color[rgb]{0,0,1}2.0}}$ | ${\underline{88.6}}_{{\color[rgb]{0,0,1}0.6}}$ | ${\boldsymbol{52.5}}_{{\color[rgb]{0,0,1}1.7}}$ | ${\boldsymbol{83.6}}_{{\color[rgb]{0,0,1}1.0}}$ | ${\boldsymbol{79.1}}_{{\color[rgb]{0,0,1}0.9}}$ | ${\boldsymbol{90.8}}_{{\color[rgb]{0,0,1}0.7}}$ | ${\boldsymbol{83.7}}_{{\color[rgb]{0,0,1}0.5}}$ | ${\underline{46.7}}_{{\color[rgb]{0,0,1}2.1}}$
GPT-2 1.5B | Vanilla ICL | 4 | ${66.3}_{{\color[rgb]{0,0,1}8.6}}$ | ${30.3}_{{\color[rgb]{0,0,1}8.9}}$ | ${56.5}_{{\color[rgb]{0,0,1}6.6}}$ | ${53.4}_{{\color[rgb]{0,0,1}8.1}}$ | $34.7_{{\color[rgb]{0,0,1}7.5}}$ | ${54.2}_{{\color[rgb]{0,0,1}5.5}}$ | ${30.8}_{{\color[rgb]{0,0,1}8.1}}$ | ${61.9}_{{\color[rgb]{0,0,1}8.7}}$ | ${54.6}_{{\color[rgb]{0,0,1}9.9}}$ | ${40.8}_{{\color[rgb]{0,0,1}7.8}}$
IDAICL | $87.4_{{\color[rgb]{0,0,1}1.5}}$ | $38.8_{{\color[rgb]{0,0,1}1.7}}$ | $80.9_{{\color[rgb]{0,0,1}1.2}}$ | $82.1_{{\color[rgb]{0,0,1}2.1}}$ | $48.1_{{\color[rgb]{0,0,1}0.6}}$ | $77.8_{{\color[rgb]{0,0,1}3.0}}$ | $49.5_{{\color[rgb]{0,0,1}1.9}}$ | $87.4_{{\color[rgb]{0,0,1}2.6}}$ | $79.2_{{\color[rgb]{0,0,1}1.8}}$ | $54.1_{{\color[rgb]{0,0,1}2.7}}$
Vanilla ICL | 8 | ${57.2}_{{\color[rgb]{0,0,1}7.0}}$ | ${30.8}_{{\color[rgb]{0,0,1}6.1}}$ | ${64.9}_{{\color[rgb]{0,0,1}8.3}}$ | ${57.6}_{{\color[rgb]{0,0,1}6.4}}$ | $38.6_{{\color[rgb]{0,0,1}6.4}}$ | ${57.3}_{{\color[rgb]{0,0,1}10.3}}$ | ${39.5}_{{\color[rgb]{0,0,1}5.3}}$ | ${67.4}_{{\color[rgb]{0,0,1}8.1}}$ | ${56.3}_{{\color[rgb]{0,0,1}5.4}}$ | ${47.4}_{{\color[rgb]{0,0,1}5.1}}$
IDAICL | $89.5_{{\color[rgb]{0,0,1}1.8}}$ | $40.8_{{\color[rgb]{0,0,1}1.9}}$ | $82.1_{{\color[rgb]{0,0,1}1.2}}$ | $84.3_{{\color[rgb]{0,0,1}2.1}}$ | $50.2_{{\color[rgb]{0,0,1}3.4}}$ | $80.1_{{\color[rgb]{0,0,1}2.9}}$ | $51.5_{{\color[rgb]{0,0,1}2.5}}$ | $89.8_{{\color[rgb]{0,0,1}1.7}}$ | $80.3_{{\color[rgb]{0,0,1}0.9}}$ | $55.5_{{\color[rgb]{0,0,1}0.6}}$
Vanilla ICL | 12 | ${70.9}_{{\color[rgb]{0,0,1}9.6}}$ | ${34.7}_{{\color[rgb]{0,0,1}6.7}}$ | ${65.2}_{{\color[rgb]{0,0,1}5.6}}$ | ${59.9}_{{\color[rgb]{0,0,1}6.7}}$ | $38.3_{{\color[rgb]{0,0,1}10.2}}$ | ${59.6}_{{\color[rgb]{0,0,1}8.1}}$ | ${40.7}_{{\color[rgb]{0,0,1}7.5}}$ | ${72.5}_{{\color[rgb]{0,0,1}11.6}}$ | ${57.6}_{{\color[rgb]{0,0,1}9.5}}$ | ${48.5}_{{\color[rgb]{0,0,1}5.7}}$
IDAICL | $90.0_{{\color[rgb]{0,0,1}2.8}}$ | $41.1_{{\color[rgb]{0,0,1}1.3}}$ | $83.4_{{\color[rgb]{0,0,1}2.3}}$ | ${85.6}_{{\color[rgb]{0,0,1}2.4}}$ | $51.6_{{\color[rgb]{0,0,1}2.9}}$ | $80.5_{{\color[rgb]{0,0,1}2.5}}$ | $51.8_{{\color[rgb]{0,0,1}3.6}}$ | $90.5_{{\color[rgb]{0,0,1}2.7}}$ | $81.1_{{\color[rgb]{0,0,1}3.0}}$ | ${55.7}_{{\color[rgb]{0,0,1}2.1}}$
MetaICL | 12 | ${79.1}_{{\color[rgb]{0,0,1}7.0}}$ | $38.6_{{\color[rgb]{0,0,1}3.7}}$ | $76.4_{{\color[rgb]{0,0,1}6.3}}$ | $75.3_{{\color[rgb]{0,0,1}4.5}}$ | $50.5_{{\color[rgb]{0,0,1}7.1}}$ | $73.9_{{\color[rgb]{0,0,1}7.6}}$ | $46.7_{{\color[rgb]{0,0,1}6.3}}$ | $86.8_{{\color[rgb]{0,0,1}7.8}}$ | $76.4_{{\color[rgb]{0,0,1}5.4}}$ | ${53.1}_{{\color[rgb]{0,0,1}1.6}}$
+IDAICL | ${89.6}_{{\color[rgb]{0,0,1}2.2}}$ | ${\underline{42.9}}_{{\color[rgb]{0,0,1}2.3}}$ | ${84.2}_{{\color[rgb]{0,0,1}3.4}}$ | ${\underline{87.9}}_{{\color[rgb]{0,0,1}1.1}}$ | ${\boldsymbol{53.8}}_{{\color[rgb]{0,0,1}1.2}}$ | ${\underline{83.4}}_{{\color[rgb]{0,0,1}3.2}}$ | ${{53.6}}_{{\color[rgb]{0,0,1}1.3}}$ | ${\underline{91.9}}_{{\color[rgb]{0,0,1}0.9}}$ | ${\underline{84.3}}_{{\color[rgb]{0,0,1}1.4}}$ | ${\underline{57.3}}_{{\color[rgb]{0,0,1}1.5}}$
Channel ICL | 12 | ${83.3}_{{\color[rgb]{0,0,1}5.9}}$ | $37.5_{{\color[rgb]{0,0,1}4.6}}$ | ${80.6}_{{\color[rgb]{0,0,1}4.1}}$ | ${77.1}_{{\color[rgb]{0,0,1}5.5}}$ | $48.9_{{\color[rgb]{0,0,1}6.7}}$ | $68.2_{{\color[rgb]{0,0,1}8.3}}$ | ${43.3}_{{\color[rgb]{0,0,1}7.2}}$ | $70.4_{{\color[rgb]{0,0,1}9.3}}$ | ${67.9}_{{\color[rgb]{0,0,1}5.5}}$ | ${53.6}_{{\color[rgb]{0,0,1}8.9}}$
+IDAICL | ${\boldsymbol{91.2}}_{{\color[rgb]{0,0,1}2.1}}$ | ${40.8}_{{\color[rgb]{0,0,1}1.5}}$ | ${\underline{86.5}}_{{\color[rgb]{0,0,1}2.6}}$ | ${\boldsymbol{88.2}}_{{\color[rgb]{0,0,1}1.8}}$ | ${52.4}_{{\color[rgb]{0,0,1}2.9}}$ | ${82.3}_{{\color[rgb]{0,0,1}2.4}}$ | ${50.5}_{{\color[rgb]{0,0,1}1.8}}$ | ${88.7}_{{\color[rgb]{0,0,1}1.2}}$ | ${82.6}_{{\color[rgb]{0,0,1}0.9}}$ | ${56.5}_{{\color[rgb]{0,0,1}2.1}}$
EPR | 12 | ${82.8}_{{\color[rgb]{0,0,1}2.6}}$ | $40.6_{{\color[rgb]{0,0,1}2.1}}$ | $79.5_{{\color[rgb]{0,0,1}1.4}}$ | $74.7_{{\color[rgb]{0,0,1}2.7}}$ | $50.7_{{\color[rgb]{0,0,1}2.3}}$ | ${{83.3}}_{{\color[rgb]{0,0,1}0.7}}$ | ${\underline{82.2}}_{{\color[rgb]{0,0,1}2.4}}$ | $91.5_{{\color[rgb]{0,0,1}0.8}}$ | ${{83.2}}_{{\color[rgb]{0,0,1}1.6}}$ | $54.8_{{\color[rgb]{0,0,1}1.9}}$
+IDAICL | ${\underline{90.5}}_{{\color[rgb]{0,0,1}1.5}}$ | ${\boldsymbol{43.8}}_{{\color[rgb]{0,0,1}1.0}}$ | ${\boldsymbol{87.4}}_{{\color[rgb]{0,0,1}0.9}}$ | ${86.5}_{{\color[rgb]{0,0,1}1.5}}$ | ${\underline{52.9}}_{{\color[rgb]{0,0,1}1.8}}$ | ${\boldsymbol{85.8}}_{{\color[rgb]{0,0,1}0.5}}$ | ${\boldsymbol{84.7}}_{{\color[rgb]{0,0,1}1.1}}$ | ${\boldsymbol{93.5}}_{{\color[rgb]{0,0,1}2.5}}$ | ${\boldsymbol{86.4}}_{{\color[rgb]{0,0,1}2.2}}$ | ${\boldsymbol{57.5}}_{{\color[rgb]{0,0,1}1.5}}$
GPT-Neo | MetaICL | 12 | ${87.8}_{{\color[rgb]{0,0,1}6.7}}$ | $42.5_{{\color[rgb]{0,0,1}6.1}}$ | $82.2_{{\color[rgb]{0,0,1}5.9}}$ | ${80.7}_{{\color[rgb]{0,0,1}4.8}}$ | $51.5_{{\color[rgb]{0,0,1}5.3}}$ | $72.2_{{\color[rgb]{0,0,1}8.2}}$ | $54.1_{{\color[rgb]{0,0,1}6.8}}$ | $84.4_{{\color[rgb]{0,0,1}5.5}}$ | $74.3_{{\color[rgb]{0,0,1}8.2}}$ | ${50.3}_{{\color[rgb]{0,0,1}6.4}}$
+IDAICL | ${\underline{92.1}}_{{\color[rgb]{0,0,1}1.1}}$ | ${{44.3}}_{{\color[rgb]{0,0,1}2.3}}$ | ${\boldsymbol{88.8}}_{{\color[rgb]{0,0,1}2.1}}$ | ${\boldsymbol{88.1}}_{{\color[rgb]{0,0,1}1.8}}$ | ${\boldsymbol{53.2}}_{{\color[rgb]{0,0,1}1.7}}$ | ${{84.3}}_{{\color[rgb]{0,0,1}2.1}}$ | ${{64.3}}_{{\color[rgb]{0,0,1}1.9}}$ | ${{94.3}}_{{\color[rgb]{0,0,1}1.2}}$ | ${{86.5}}_{{\color[rgb]{0,0,1}0.9}}$ | ${\boldsymbol{53.4}}_{{\color[rgb]{0,0,1}2.1}}$
Channel ICL | 12 | ${83.4}_{{\color[rgb]{0,0,1}5.4}}$ | $39.8_{{\color[rgb]{0,0,1}6.4}}$ | $79.5_{{\color[rgb]{0,0,1}5.7}}$ | ${79.4}_{{\color[rgb]{0,0,1}5.9}}$ | $50.1_{{\color[rgb]{0,0,1}3.8}}$ | $70.6_{{\color[rgb]{0,0,1}8.2}}$ | $50.8_{{\color[rgb]{0,0,1}5.1}}$ | $78.3_{{\color[rgb]{0,0,1}7.1}}$ | $72.5_{{\color[rgb]{0,0,1}6.9}}$ | ${48.7}_{{\color[rgb]{0,0,1}4.5}}$
+IDAICL | ${91.5}_{{\color[rgb]{0,0,1}2.2}}$ | ${41.6}_{{\color[rgb]{0,0,1}1.8}}$ | ${85.4}_{{\color[rgb]{0,0,1}1.9}}$ | ${\underline{87.2}}_{{\color[rgb]{0,0,1}2.5}}$ | ${\underline{52.7}}_{{\color[rgb]{0,0,1}2.2}}$ | ${83.7}_{{\color[rgb]{0,0,1}1.4}}$ | ${62.8}_{{\color[rgb]{0,0,1}0.7}}$ | ${93.5}_{{\color[rgb]{0,0,1}3.3}}$ | ${84.6}_{{\color[rgb]{0,0,1}3.1}}$ | ${52.0}_{{\color[rgb]{0,0,1}1.8}}$
EPR | 12 | ${88.2}_{{\color[rgb]{0,0,1}1.6}}$ | ${\underline{45.7}}_{{\color[rgb]{0,0,1}2.2}}$ | ${81.8}_{{\color[rgb]{0,0,1}1.9}}$ | ${71.8}_{{\color[rgb]{0,0,1}2.9}}$ | ${49.9}_{{\color[rgb]{0,0,1}1.1}}$ | ${\underline{89.4}}_{{\color[rgb]{0,0,1}2.4}}$ | ${\underline{92.3}}_{{\color[rgb]{0,0,1}2.2}}$ | ${\underline{96.1}}_{{\color[rgb]{0,0,1}1.2}}$ | ${\underline{88.8}}_{{\color[rgb]{0,0,1}1.1}}$ | $49.4_{{\color[rgb]{0,0,1}0.7}}$
+IDAICL | ${\boldsymbol{93.2}}_{{\color[rgb]{0,0,1}0.8}}$ | ${\boldsymbol{47.2}}_{{\color[rgb]{0,0,1}1.3}}$ | ${\underline{88.5}}_{{\color[rgb]{0,0,1}1.2}}$ | ${86.6}_{{\color[rgb]{0,0,1}2.0}}$ | ${52.1}_{{\color[rgb]{0,0,1}0.4}}$ | ${\boldsymbol{93.1}}_{{\color[rgb]{0,0,1}1.2}}$ | ${\boldsymbol{94.4}}_{{\color[rgb]{0,0,1}2.4}}$ | ${\boldsymbol{97.8}}_{{\color[rgb]{0,0,1}1.5}}$ | ${\boldsymbol{91.2}}_{{\color[rgb]{0,0,1}0.7}}$ | ${\underline{52.1}}_{{\color[rgb]{0,0,1}0.5}}$

Table 1: Comparison results of three PLMs. Two numbers indicate the mean
accuracy (%) and standard deviation over different seeds. The best and second-
best results per PLM per dataset are highlighted in bold and underlined,
respectively. "+IDAICL" means that the current approach is used in conjunction
with IDAICL. The results for different numbers of demonstration examples
(i.e., $m$ values) using the GPT-Neo model are illustrated in Figure 3.

Subsequently, our newly proposed prediction function, referred to as IDA-
Softmax, is defined as

$\begin{aligned}
P_{y_{j}}^{\text{IDA}}(\boldsymbol{\tilde{x}}):=\sum_{k}M_{k,y_{j}}N_{k,y_{j}}e^{\Delta\boldsymbol{w}_{k,y_{j}}^{T}\boldsymbol{h}_{\tilde{\boldsymbol{x}}}+\Delta
b_{k,y_{j}}}.\end{aligned}$

(14)

Consequently, instead of conducting the augmentation process explicitly, we
can directly employ IDA-Softmax, $P_{y_{j}}^{\text{IDA}}$, for prediction.
IDA-Softmax essentially utilizes two modulating factors associated with
statistical properties derived from $\mathcal{D}$ to calibrate the sample
logits. Previous studies Min et al. (2022c); Chan et al. (2022) have
underscored the pivotal role of knowledge about the input data distribution in
predictions made by PLMs. Intuitively, PLMs can better capture the patterns
and underlying structures within data, such as the spatial relationships
between demonstrations and queries, ultimately enhancing their prediction
performance.

Furthermore, to mitigate the imbalance among different answer types in
demonstrations Holtzman et al. (2021); Zhao et al. (2021), we adopt a post-hoc
adjustment approach inspired by Menon et al. Menon et al. (2021), which
adjusts predictions by considering the class proportions within $\mathcal{D}$.
Thus, the prediction for answer $y_{j}$ is computed as

$\tilde{P}_{y_{j}}^{\text{IDA}}(\tilde{\boldsymbol{x}})=P_{y_{j}}^{\text{IDA}}(\tilde{\boldsymbol{x}})+\tau\log\pi_{y_{j}},$
(15)

where $\tau$ is a positive hyperparameter, and $\pi_{y_{j}}$ demotes the
proportion of answer $y_{j}$ in $\mathcal{D}$. In practical applications, the
value of $\tau$ can be fixed at 1. This approach compensates for predictions
of minor classes. When different answers are uniformly distributed,
$\tau\log\pi_{y_{j}}$ exerts an equal influence on all answer types.
Consequently, the final prediction is given by

$\hat{y}={\arg\min_{y_{j}\in\mathcal{Y}}\tilde{P}_{y_{j}}^{\text{IDA}}(\boldsymbol{\tilde{x}})}.$
(16)

##  4 Experimental Setup

###  4.1 Models and Datasets

We evaluated the performance of IDAICL across seven large PLMs, including
GPT-2 Radford et al. (2019) (with 0.1B, 0.3B, 0.8B, and 1.5B parameters), GPT-
Neo Black et al. (2021) (with 2.7B parameters), and LLaMA Touvron et al.
(2023) (with 13B and 33B parameters). Following previous research Min et al.
(2022a); Han et al. (2023); Lu et al. (2022), our evaluation encompasses ten
text classification datasets. Among these, SST-2 Socher et al. (2013), SST-5
Socher et al. (2013), MR Pang and Lee (2005), CR Hu and Liu (2004), and Amazon
McAuley and Leskovec (2013) are five sentiment classification tasks. Subj Pang
and Lee (2004), TREC Voorhees and Tice (2000), DBPedia Lehmann et al. (2015),
and AGNews Zhang et al. (2015) cater to subjectivity, question, ontology, and
news classification tasks, respectively. Additionally, CB De Marneffe et al.
(2019) is utilized for natural language inference. Among these datasets,
SST-5, Amazon, TREC, and CB are characterized by imbalanced training data.
Details of all datasets are provided in Section A of the Appendix.

PLM | Method | SST-2 | SST-5 | MR | CR | Subj | TREC | DBPedia | AGNews | CB | Avg.
---|---|---|---|---|---|---|---|---|---|---|---
LLaMA 13B | Vanilla ICL | ${95.6}_{{\color[rgb]{0,0,1}7.1}}$ | ${29.5}_{{\color[rgb]{0,0,1}6.2}}$ | ${90.0}_{{\color[rgb]{0,0,1}5.8}}$ | ${91.4}_{{\color[rgb]{0,0,1}7.4}}$ | ${72.9}_{{\color[rgb]{0,0,1}6.9}}$ | ${62.8}_{{\color[rgb]{0,0,1}9.1}}$ | ${80.9}_{{\color[rgb]{0,0,1}7.6}}$ | $80.2_{{\color[rgb]{0,0,1}5.9}}$ | $51.5_{{\color[rgb]{0,0,1}8.2}}$ | 72.8
ConCa | ${\boldsymbol{96.7}}_{{\color[rgb]{0,0,1}5.4}}$ | $40.3_{{\color[rgb]{0,0,1}6.2}}$ | ${91.7}_{{\color[rgb]{0,0,1}7.3}}$ | ${90.8}_{{\color[rgb]{0,0,1}4.2}}$ | ${79.6}_{{\color[rgb]{0,0,1}9.1}}$ | ${68.2}_{{\color[rgb]{0,0,1}5.6}}$ | ${\underline{94.3}}_{{\color[rgb]{0,0,1}4.1}}$ | ${\underline{85.2}}_{{\color[rgb]{0,0,1}7.5}}$ | $46.6_{{\color[rgb]{0,0,1}5.0}}$ | 77.0
PROCA | ${95.4}_{{\color[rgb]{0,0,1}3.8}}$ | ${\underline{43.4}}_{{\color[rgb]{0,0,1}5.7}}$ | ${90.3}_{{\color[rgb]{0,0,1}9.6}}$ | ${\underline{92.1}}_{{\color[rgb]{0,0,1}3.1}}$ | ${\underline{84.8}}_{{\color[rgb]{0,0,1}2.5}}$ | ${69.9}_{{\color[rgb]{0,0,1}2.1}}$ | ${92.5}_{{\color[rgb]{0,0,1}4.9}}$ | ${81.6}_{{\color[rgb]{0,0,1}3.6}}$ | $51.4_{{\color[rgb]{0,0,1}4.2}}$ | 77.9
D-ConCa | ${96.3}_{{\color[rgb]{0,0,1}3.8}}$ | ${42.5}_{{\color[rgb]{0,0,1}4.5}}$ | ${\underline{92.0}}_{{\color[rgb]{0,0,1}4.1}}$ | ${90.5}_{{\color[rgb]{0,0,1}2.9}}$ | ${82.9}_{{\color[rgb]{0,0,1}4.5}}$ | ${\underline{73.7}}_{{\color[rgb]{0,0,1}3.9}}$ | $87.4_{{\color[rgb]{0,0,1}7.2}}$ | $82.5_{{\color[rgb]{0,0,1}3.3}}$ | ${\underline{52.2}}_{{\color[rgb]{0,0,1}4.1}}$ | 77.8
IDAICL | ${\boldsymbol{96.7}}_{{\color[rgb]{0,0,1}2.5}}$ | ${\boldsymbol{47.1}}_{{\color[rgb]{0,0,1}1.1}}$ | ${\boldsymbol{93.0}}_{{\color[rgb]{0,0,1}1.9}}$ | ${\boldsymbol{93.3}}_{{\color[rgb]{0,0,1}0.8}}$ | ${\boldsymbol{87.8}}_{{\color[rgb]{0,0,1}2.3}}$ | ${\boldsymbol{76.0}}_{{\color[rgb]{0,0,1}2.6}}$ | ${\boldsymbol{94.9}}_{{\color[rgb]{0,0,1}1.0}}$ | ${\boldsymbol{87.7}}_{{\color[rgb]{0,0,1}2.4}}$ | ${\boldsymbol{59.4}}_{{\color[rgb]{0,0,1}1.9}}$ | $\boldsymbol{81.8}$
LLaMA 33B | Vanilla ICL | ${95.5}_{{\color[rgb]{0,0,1}7.2}}$ | ${29.4}_{{\color[rgb]{0,0,1}5.6}}$ | ${91.7}_{{\color[rgb]{0,0,1}5.4}}$ | ${\underline{91.5}}_{{\color[rgb]{0,0,1}8.1}}$ | ${85.1}_{{\color[rgb]{0,0,1}6.0}}$ | ${70.9}_{{\color[rgb]{0,0,1}4.4}}$ | ${86.6}_{{\color[rgb]{0,0,1}4.5}}$ | $76.2_{{\color[rgb]{0,0,1}6.1}}$ | ${{59.2}}_{{\color[rgb]{0,0,1}5.3}}$ | 76.2
ConCa | ${\underline{95.9}}_{{\color[rgb]{0,0,1}6.5}}$ | ${39.1}_{{\color[rgb]{0,0,1}4.4}}$ | $90.3_{{\color[rgb]{0,0,1}7.2}}$ | ${91.2}_{{\color[rgb]{0,0,1}3.6}}$ | ${74.6}_{{\color[rgb]{0,0,1}5.7}}$ | ${76.7}_{{\color[rgb]{0,0,1}6.2}}$ | ${\underline{92.4}}_{{\color[rgb]{0,0,1}3.9}}$ | $87.3_{{\color[rgb]{0,0,1}5.7}}$ | ${{57.9}}_{{\color[rgb]{0,0,1}6.0}}$ | 78.4
PROCA | ${95.5}_{{\color[rgb]{0,0,1}4.2}}$ | ${{39.2}}_{{\color[rgb]{0,0,1}6.3}}$ | ${\underline{92.4}}_{{\color[rgb]{0,0,1}4.1}}$ | ${91.3}_{{\color[rgb]{0,0,1}3.5}}$ | ${\underline{88.3}}_{{\color[rgb]{0,0,1}2.2}}$ | ${64.7}_{{\color[rgb]{0,0,1}3.8}}$ | $86.9_{{\color[rgb]{0,0,1}5.1}}$ | ${85.8}_{{\color[rgb]{0,0,1}7.1}}$ | ${\underline{59.9}}_{{\color[rgb]{0,0,1}3.8}}$ | 78.2
D-ConCa | ${95.4}_{{\color[rgb]{0,0,1}3.8}}$ | ${\underline{40.7}}_{{\color[rgb]{0,0,1}4.5}}$ | ${92.1}_{{\color[rgb]{0,0,1}4.2}}$ | ${91.0}_{{\color[rgb]{0,0,1}2.9}}$ | ${76.4}_{{\color[rgb]{0,0,1}3.6}}$ | ${\boldsymbol{80.2}}_{{\color[rgb]{0,0,1}2.1}}$ | $87.6_{{\color[rgb]{0,0,1}4.2}}$ | ${\underline{87.7}}_{{\color[rgb]{0,0,1}4.3}}$ | $56.5_{{\color[rgb]{0,0,1}3.4}}$ | 78.6
IDAICL | ${\boldsymbol{96.5}}_{{\color[rgb]{0,0,1}1.1}}$ | ${\boldsymbol{46.8}}_{{\color[rgb]{0,0,1}2.4}}$ | ${\boldsymbol{93.6}}_{{\color[rgb]{0,0,1}1.3}}$ | ${\boldsymbol{92.3}}_{{\color[rgb]{0,0,1}3.3}}$ | ${\boldsymbol{89.3}}_{{\color[rgb]{0,0,1}2.4}}$ | ${\underline{79.1}}_{{\color[rgb]{0,0,1}1.5}}$ | ${\boldsymbol{95.6}}_{{\color[rgb]{0,0,1}2.3}}$ | ${\boldsymbol{88.4}}_{{\color[rgb]{0,0,1}1.9}}$ | ${\boldsymbol{64.6}}_{{\color[rgb]{0,0,1}2.8}}$ | $\boldsymbol{82.9}$

Table 2: Comparison results of Macro-F1 for the LLaMA model with 13B and 33B
parameters, setting $m$ to 4.  Figure 3: Comparison results between Vanilla
ICL and IDAICL across different values of $m$ on the GPT-Neo model. IDAICL
significantly outperforms Vanilla ICL, particularly when the number of
demonstration examples is small.

###  4.2 Compared Baselines

Besides Vanilla ICL, we compared and integrated IDAICL with three popular ICL
algorithms, focusing on learning process design and demonstration retrieval.
These include MetaICL Min et al. (2022b), Channel ICL Min et al. (2022a), and
Efficient Prompt Retrieval (EPR) Rubin et al. (2022). Moreover, we compared
IDAICL with other advanced prediction calibration methods: Contextual
Calibration (ConCa) Zhao et al. (2021), Prototypical Calibration (PROCA) Han
et al. (2023), and Domain-Context Calibration (D-ConCa) Fei et al. (2023).
Introductions to all compared methods and comprehensive experimental settings
are presented in Sections B and C of the Appendix.

Figure 4: (a) and (b): Macro-F1 of SST-5 and AGNews datasets using the LLaMA
model with 33B parameters under three demonstration selection settings,
setting $m$ to 4. (c) and (d): Accuracy of Vanilla ICL and IDAICL on the SST-2
dataset using the GPT-2 model with 1.5B parameters across six templates,
setting $m$ to 12. IDAICL demonstrates greater robustness across various
demonstration examples and templates compared to Vanilla ICL.

##  5 Experimental Results

###  5.1 Main Results

Table 1 displays the comparison results between IDAICL and four ICL baselines
(Vanilla ICL, MetaICL, Channel ICL, and EPR) across GPT-2 models (with 0.8B
and 1.5B parameters) and the GPT-Neo model. These results lead to three main
findings. Firstly, IDAICL consistently exhibits high effectiveness across
various model sizes and datasets, highlighting its strong generalization
capacity, even under scenarios involving imbalanced training data. Compared to
Vanilla ICL, IDAICL outperforms by an average of 17.7% and 18.4% across
diverse datasets and $m$ values for GPT-2 with 0.8B and 1.5B parameters,
respectively. Secondly, in comparison to other ICL baselines like Channel ICL,
MetaICL, and EPR, the integration of IDAICL consistently delivers notable
performance improvements, emphasizing the efficacy of enhancing demonstrations
for refined predictions. The inclusion of IDAICL led to an average performance
boost of 7.3% for MetaICL and 8.2% for Channel ICL. Lastly, IDAICL notably
enhances worst-case accuracy and diminishes performance variance across
different seeds, showcasing its ability to improve prediction stability.
Additional results on LLaMA and smaller GPT-2 models are available in Tables 7
and 8 of the Appendix.

###  5.2 Comparison with Calibration Methods

We compared IDAICL with three advanced prediction calibration methods (ConCa,
PROCA, and D-ConCa) across three PLMs: GPT-2, GPT-Neo, and LLaMA. Table 2
presents the comparison results for the LLaMA models, where IDAICL
consistently achieves state-of-the-art performance, except for TREC using the
LLaMA model with 33B parameters. These findings suggest that IDAICL which
leverages statistical information derived from the input data distribution for
prediction calibration, generally outperforms methods relying on estimated
biases for correction. Further comparison results can be found in Table 9 of
the Appendix.

###  5.3 Stability Analysis

Previous studies Zhao et al. (2021); Sorensen et al. (2022); Min et al.
(2022a); Zhang et al. (2022b) have highlighted the considerable variability in
ICL’s performance. In this section, we verified that IDAICL can effectively
enhance performance stability across diverse scenarios.

#### Varying numbers of demonstrations

We have presented the results across different numbers of demonstrations in
Table 1. For a clearer depiction, the outcomes regarding GPT-Neo are
illustrated in Figure 3. As the number of demonstration examples (represented
by $m$) increases, both Vanilla ICL and IDAICL exhibit improved performance,
emphasizing the importance of comprehensive statistical properties of the
input data for IDAICL’s effectiveness. Notably, IDAICL significantly enhances
performance stability across various numbers of demonstrations and
consistently outperforms Vanilla ICL. The performance improvement is
particularly pronounced when $m$ takes on smaller values, indicating the
efficacy of IDAICL in enriching the available knowledge for PLMs.

#### Varying demonstrations

To confirm that augmenting demonstrations can enhance the robustness of the
ICL strategy across various demonstrations, we investigated three distinct
demonstration selection settings. Setting I: Training samples most similar to
the test sample are chosen. Setting II: Samples are randomly selected from the
training data. Setting III: Training samples exhibiting the greatest
dissimilarity from the test sample are selected. As shown in Figures 4(a) and
(b), IDAICL significantly outperforms Vanilla ICL and demonstrates greater
robustness across the three selection settings. Additionally, our discoveries
suggest that selecting demonstrations that are more similar to the test
samples leads to better performance than exclusively selecting dissimilar
ones, which aligns with the findings obtained by Wang et al. Wang et al.
(2022).

Figure 5: (a) and (b): Accuracy comparison of the SST-2 and MR datasets, where
the proportions of the negative class in demonstrations (denoted as $p$) are
varied from 0.1 to 0.5. (c) and (d): Confusion matrices for the CR and Subj
datasets, representing scenarios where the proportions of one category in
demonstrations are set to 0.1 and 0.2. The analysis is conducted using the
GPT-2 model with 1.5B parameters, with $m$ setting to 12. IDAICL demonstrates
greater robustness in handling imbalanced class distributions within
demonstrations. Figure 6: Accuracy across different $\lambda$ and $\tau$
values, using GPT-2 with 0.8B parameters, setting $m$ to 12. $\lambda\\!=\\!0$
and $\tau\\!=\\!0$ signify that the two modulating factors and the class
proportion term are not utilized, respectively.

#### Varying templates

To assess the performance of IDAICL across various templates, we employed
fifteen templates on the SST-2 dataset following those outlined by Zhao et al.
Zhao et al. (2021). The templates are elaborated in Table 10 of the Appendix.
Figures 4(c) and (d) display the performance of Vanilla ICL and IDAICL across
six templates. Some templates achieve higher average performance than others.
Nevertheless, IDAICL consistently enhances both average and worst-case
accuracy, simultaneously reducing performance variance across different
templates. The complete results are available in Figure 7 of the Appendix.

#### Impact of imbalance in labels

Figures 5(a) and (b) depict comparison results among Vanilla ICL, MetaICL,
Channel ICL, and IDAICL across different degrees of imbalances. It is evident
that the performance of Vanilla ICL is sensitive to class imbalance, while
that of IDAICL and Channel ICL exhibit robustness to the imbalance. Moreover,
notable performance improvements are observed with higher levels of imbalance.
Additionally, Figures 5(c) and (d) illustrate the confusion matrices for CR
and Subj datasets, with the proportion of one category (i.e., "Negative" and
"Subjective") in demonstrations setting to 0.1 and 0.2. IDAICL significantly
improves the accuracy of the underrepresented classes when compared to Vanilla
ICL, thereby contributing to enhanced fairness among classes. In the
subsequent section, we demonstrate that the strong performance of IDAICL in
handling imbalanced label distributions stems from both the statistical
properties and the class proportion term.

Dataset | 0-shot | 1-shot | 4-shot | IDAICL
---|---|---|---|---
SST-2 | $63.2$ | $61.3_{{\color[rgb]{0,0,1}9.4}}$ | $57.6_{{\color[rgb]{0,0,1}7.1}}$ | $\boldsymbol{76.3}$
SST-5 | $25.0$ | $27.3_{{\color[rgb]{0,0,1}7.9}}$ | $30.4_{{\color[rgb]{0,0,1}6.3}}$ | $\boldsymbol{33.5}$
MR | $58.9$ | $54.3_{{\color[rgb]{0,0,1}6.8}}$ | $59.3_{{\color[rgb]{0,0,1}6.5}}$ | $\boldsymbol{71.2}$
Subj | $48.9$ | $47.1_{{\color[rgb]{0,0,1}8.3}}$ | $57.6_{{\color[rgb]{0,0,1}5.4}}$ | $\boldsymbol{67.3}$
Table 3: Accuracy comparison between Vanilla ICL and IDAICL based solely on
statistical properties, using the GPT-2 model with 0.8B parameters.

###  5.4 Sensitivity and Ablation Studies

We conducted ablation studies on IDAICL to investigate the influence of the
two modulating factors and the class proportion term. The parameters $\lambda$
and $\tau$ govern the augmentation strength and the impact of the class
proportion term, respectively. In Figure 6(a), a significant performance drop
is observed when predictions are not calibrated using statistical properties
derived from the demonstrations. Additionally, optimal performance is achieved
when $\lambda$ equals 0.5.

Figure 6(b) showcases the accuracy of SST-2 and MR datasets with the negative
class proportion in demonstrations setting to 0.1. Results indicate that
solely leveraging statistical properties (i.e., $\tau$ equals 0) enhances
performance under imbalanced demonstrations, with further improvements
observed upon the inclusion of the class proportion term. Additionally,
optimal performance is attained when $\tau$ equals 1. Consequently, we
recommend setting $\lambda$ to 0.5 and $\tau$ to 1 for practical applications.
More results are presented in Appendix F.

###  5.5 Further Discussion

To further investigate the effect of statistical properties within
demonstrations on model performance, we exclusively employed queries along
with statistical information for inference, excluding the inclusion of
demonstrations for each test sample. These statistics were estimated using
deep features of all training samples. As shown in Table 3, IDAICL relying
solely on statistical properties distinctly outperforms Vanilla ICL across
scenarios with zero, one, and even four demonstrations. This emphasizes the
crucial role of prior statistics obtained from training data in PLMs’
predictions. This phenomenon is understandable as statistical properties
inherently encompass richer global information compared to individual
demonstrations.

##  6 Conclusion

This study introduces IDAICL, a novel ICL approach designed to enhance
demonstrations by utilizing semantic directions sampled from the deep feature
distribution of demonstration examples. Our augmentation strategy enriches the
knowledge available to PLMs without extending the context length. A new
prediction function is then theoretically established considering the number
of augmented pieces approaching infinity. This eliminates the need for
explicit augmentation and allows for direct utilization of this derived
function for predictions. Our extensive experiments, spanning various tasks
and PLMs, demonstrate that IDAICL significantly enhances both prediction
accuracy and stability when compared to other ICL baselines.

## Limitations

While IDAICL proves to be competitive in few-shot learning, there are
limitations that open up avenues for future research. First, due to the
necessity of accessing the parameters of the final fully connected layer in
PLMs, IDAICL is exclusively suitable for open-source models. Future research
is expected to develop alternative augmentation strategies tailored for black-
box PLMs. Second, our evaluation of IDAICL focused on seven PLMs and ten text
classification tasks. We defer further explorations involving other PLMs and
non-classification tasks for future work. Additionally, IDAICL relies on a
small set of demonstrations to estimate the feature mean and covariance
matrix. If such a collection is unavailable or extremely scarce, IDAICL may
need to be used in conjunction with demonstration generation methods.

Other avenues for future work involve exploring more effective augmentation
distributions. This entails exploring finer-grained distributions, such as
category-level or sample-level distributions, to emphasize the unique
characteristics of individual categories or samples, and extending these
distributions beyond the constraints of training data. Furthermore, given the
effectiveness of data augmentation in model training, future research could
explore the utilization of our derived prediction function in both the
training and fine-tuning phases of large PLMs.

## References

  * Bengio et al. (2013) Yoshua Bengio, Grégoire Mesnil, Yann Dauphin, and Salah Rifai. 2013.  Better mixing via deep representations.  In _Proceedings of 30th International Conference on Machine Learning_ , pages 552–560, Atlanta, Georgia, USA. ACM.
  * Black et al. (2021) Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Rose Biderman. 2021.  Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , page 1877–1901, Online and Vancouver, Canada.
  * Chan et al. (2022) Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. 2022.  Data distributional properties drive emergent in-context learning in transformers.  In _Proceedings of the 36th Advances in Neural Information Processing Systems_ , pages 18878–18891, New Orleans, USA.
  * Chen et al. (2022a) Dong Chen, Yueting Zhuang, Zijin Shen, Carl Yang, Guoming Wang, Siliang Tang, and Yi Yang. 2022a.  Cross-modal data augmentation for tasks of different modalities.  _IEEE Transactions on Multimedia_ , 99:1–11.
  * Chen et al. (2022b) Hui Chen, Wei Han, Diyi Yang, and Soujanya Poria. 2022b.  DoubleMix: Simple interpolation-based data augmentation for text classification.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 4622–4632, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Chen et al. (2023) Jiaao Chen, Derek Tam, Colin Raffel, Mohit Bansal, and Diyi Yang. 2023.  An empirical survey of data augmentation for limited data learning in NLP.  _Transactions of the Association for Computational Linguistics_ , 11:191–211.
  * Chen et al. (2022c) Xiaohua Chen, Yucan Zhou, Dayan Wu, Wanqian Zhang, Yu Zhou, Bo Li, and Weiping Wang. 2022c.  Imagine by reasoning: A reasoning-based implicit semantic data augmentation for long-tailed classification.  In _Proceedings of the 36th AAAI Conference on Artificial Intelligence_ , pages 356–364, Online. AAAI Press.
  * Cheng et al. (2020) Yong Cheng, Lu Jiang, Wolfgang Macherey, and Jacob Eisenstein. 2020.  AdvAug: Robust adversarial augmentation for neural machine translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5961–5970, Online. Association for Computational Linguistics.
  * Cheung and Yeung (2021) Tsz-Him Cheung and Dit-Yan Yeung. 2021.  Modals: Modality-agnostic automated data augmentation in the latent space.  In _Proceedings of 9th International Conference on Learning Representations_ , Online.
  * Cho (2016) Kyunghyun Cho. 2016.  Noisy parallel approximate decoding for conditional recurrent language model.  _arXiv preprint arXiv:1605.03835_.
  * Choi et al. (2022) Eunbi Choi, Yongrae Jo, Joel Jang, and Minjoon Seo. 2022.  Prompt injection: Parameterization of fixed inputs.  _arXiv preprint arXiv:2206.11349_.
  * Davison et al. (2019) Joe Davison, Joshua Feldman, and Alexander Rush. 2019.  Commonsense knowledge mining from pretrained models.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1173–1178, Hong Kong, China. Association for Computational Linguistics.
  * De Marneffe et al. (2019) Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019.  The commitmentbank: Investigating projection in naturally occurring discourse.  In _Proceedings of Sinn und Bedeutung 23_ , pages 107–124, Universitat Autònoma de Barcelona, Bellaterra.
  * Fei et al. (2023) Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. 2023.  Mitigating label biases for in-context learning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 14014–14031, Toronto, Canada. Association for Computational Linguistics.
  * Feng et al. (2021) Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021.  A survey of data augmentation approaches for NLP.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 968–988, Online. Association for Computational Linguistics.
  * Gao et al. (2021) Tianyu Gao, Adam Fisch, and Danqi Chen. 2021.  Making pre-trained language models better few-shot learners.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3816–3830, Online. Association for Computational Linguistics.
  * Han et al. (2023) Zhixiong Han, Yaru Hao, Li Dong, Yutao Sun, and Furu Wei. 2023.  Prototypical calibration for few-shot learning of language models.  In _Proceedings of 11st International Conference on Learning Representations_ , Kigali, Rwanda.
  * Hao et al. (2022) Yaru Hao, Yutao Sun, Li Dong, Zhixiong Han, Yuxian Gu, and Furu Wei. 2022.  Structured prompting: Scaling in-context learning to 1,000 examples.  _arXiv preprint arXiv:2212.06713_.
  * Holtzman et al. (2021) Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021.  Surface form competition: Why the highest probability answer isn’t always right.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7038–7051, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Hu and Liu (2004) Minqing Hu and Bing Liu. 2004.  Mining and summarizing customer reviews.  In _Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pages 168–177, Seattle, Washington, USA. ACM.
  * Jiang et al. (2020) Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020.  How can we know what language models know? _Transactions of the Association for Computational Linguistics_ , 8:423–438.
  * Kobayashi (2018) Sosuke Kobayashi. 2018.  Contextual augmentation: Data augmentation by words with paradigmatic relations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 452–457, New Orleans, Louisiana. Association for Computational Linguistics.
  * Lehmann et al. (2015) Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, and Christian Bizer. 2015.  Dbpedia - a large-scale, multilingual knowledge base extracted from wikipedia.  _Semantic Web_ , 6(2):167–195.
  * Li et al. (2022) Junjie Li, Jianfei Yu, and Rui Xia. 2022.  Generative cross-domain data augmentation for aspect and opinion co-extraction.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4219–4229, Seattle, United States. Association for Computational Linguistics.
  * Li et al. (2023a) Mukai Li, Shansan Gong, Jiangtao Feng, Yiheng Xu, Jun Zhang, Zhiyong Wu, and Lingpeng Kong. 2023a.  In-context learning with many demonstration examples.  _arXiv preprint arXiv:2302.04931_.
  * Li et al. (2021) Shuang Li, Kaixiong Gong, Chi-Harold Liu, Yulin Wang, Feng Qiao, and Xinjing Cheng. 2021.  Metasaug: Meta semantic augmentation for long-tailed visual recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5208–5217, Nashville, TN, USA. IEEE.
  * Li et al. (2023b) Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023b.  Unified demonstration retriever for in-context learning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4644–4668, Toronto, Canada. Association for Computational Linguistics.
  * Liu et al. (2022) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022.  What makes good in-context examples for GPT-3? In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_ , pages 100–114, Dublin, Ireland and Online. Association for Computational Linguistics.
  * Logan IV et al. (2022) Robert Logan IV, Ivana Balazevic, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2022.  Cutting down on prompts and parameters: Simple few-shot learning with language models.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2824–2835, Dublin, Ireland. Association for Computational Linguistics.
  * Lu et al. (2022) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8086–8098, Dublin, Ireland. Association for Computational Linguistics.
  * Maharana et al. (2022) Kiran Maharana, Surajit Mondal, and Bhushankumar Nemade. 2022.  A review: Data pre-processing and data augmentation techniques.  _Global Transitions Proceedings_ , 3(1):91–99.
  * Malandrakis et al. (2019) Nikolaos Malandrakis, Minmin Shen, Anuj Goyal, Shuyang Gao, Abhishek Sethi, and Angeliki Metallinou. 2019.  Controlled text generation for data augmentation in intelligent artificial agents.  In _Proceedings of the 3rd Workshop on Neural Generation and Translation_ , pages 90–98, Hong Kong. Association for Computational Linguistics.
  * McAuley and Leskovec (2013) Julian McAuley and Jure Leskovec. 2013.  Hidden factors and hidden topics: Understanding rating dimensions with review text.  In _Proceedings of the 7th ACM conference on Recommender systems_ , pages 165–172, Hong Kong, China. ACM.
  * Menon et al. (2021) Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. 2021.  Long-tail learning via logit adjustment.  In _Proceedings of the 9th International Conference on Learning Representations_ , Vienna, Austria.
  * Min et al. (2022a) Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022a.  Noisy channel language model prompting for few-shot text classification.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5316–5330, Dublin, Ireland. Association for Computational Linguistics.
  * Min et al. (2022b) Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022b.  MetaICL: Learning to learn in context.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2791–2809, Seattle, United States. Association for Computational Linguistics.
  * Min et al. (2022c) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022c.  Rethinking the role of demonstrations: What makes in-context learning work? In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11048–11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Pang and Lee (2004) Bo Pang and Lillian Lee. 2004.  A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts.  In _Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04)_ , pages 271–278, Barcelona, Spain.
  * Pang and Lee (2005) Bo Pang and Lillian Lee. 2005.  Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales.  In _Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05)_ , pages 115–124, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  Pytorch: An imperative style, high-performance deep learning library.  In _Proceedings of the 33rd International Conference on Neural Information Processing Systems_ , pages 8026–8037, Vancouver, Canada.
  * Perez et al. (2021) Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021.  True few-shot learning with language models.  In _Proceedings of the 35th Conference on Neural Information Processing Systems_ , pages 11054–11070, Online.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_.
  * Rubin et al. (2022) Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022.  Learning to retrieve prompts for in-context learning.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2655–2671, Seattle, United States. Association for Computational Linguistics.
  * Shorten and Khoshgoftaar (2019) Connor Shorten and Taghi M. Khoshgoftaar. 2019.  A survey on image data augmentation for deep learning.  _Journal of Big Data_ , 6:1–48.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Sorensen et al. (2022) Taylor Sorensen, Joshua Robinson, Christopher Rytting, Alexander Shaw, Kyle Rogers, Alexia Delorey, Mahmoud Khalil, Nancy Fulda, and David Wingate. 2022.  An information-theoretic approach to prompt engineering without ground truth labels.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 819–862, Dublin, Ireland. Association for Computational Linguistics.
  * Tang et al. (2023) Raphael Tang, Xinyu Zhang, Xueguang Ma, Jimmy Lin, and Ferhan Ture. 2023.  Found in the middle: Permutation self-consistency improves listwise ranking in large language models.  _arXiv preprint arXiv:2310.07712_.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Voorhees and Tice (2000) Ellen M. Voorhees and Dawn M. Tice. 2000.  Building a question answering test collection.  In _Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval_ , pages 200–207, Athens, Greece. ACM.
  * Wang et al. (2022) Shuohang Wang, Yichong Xu, Yuwei Fang, Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, and Michael Zeng. 2022.  Training data is more valuable than you think: A simple and effective method by retrieving from training data.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3170–3179, Dublin, Ireland. Association for Computational Linguistics.
  * Wang et al. (2023) Xindi Wang, Yufei Wang, Can Xu, Xiubo Geng, Bowen Zhang, Chongyang Tao, Frank Rudzicz, Robert E. Mercer, and Daxin Jiang. 2023.  Investigating the learning behaviour of in-context learning: A comparison with supervised learning.  In _Proceedings of 26th European Conference on Artificial Intelligence_ , pages 2543–2551, Krakow, Poland.
  * Wang et al. (2019) Yulin Wang, Xuran Pan, Shiji Song, Hong Zhang, Gao Huang, and Cheng Wu. 2019.  Implicit semantic data augmentation for deep networks.  In _Proceedings of 33rd Conference on Neural Information Processing Systems_ , page 12635–12644, Vancouver, Canada.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  In _Proceedings of 36th Advances in Neural Information Processing Systems_ , pages 24824–24837, Online.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wu et al. (2022) Xing Wu, Chaochen Gao, Meng Lin, Liangjun Zang, and Songlin Hu. 2022.  Text smoothing: Enhance various data augmentation methods on text classification tasks.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 871–875, Dublin, Ireland. Association for Computational Linguistics.
  * Wu et al. (2023) Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. 2023.  Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1423–1436, Toronto, Canada. Association for Computational Linguistics.
  * Yang et al. (2023) Linyi Yang, Shuibai Zhang, Zhuohao Yu, Guangsheng Bao, Yidong Wang, Jindong Wang, Ruochen Xu, Wei Ye, Xing Xie, Weizhu Chen, et al. 2023.  Supervised knowledge makes large language models better in-context learners.  In _The Twelfth International Conference on Learning Representations_.
  * Zhang et al. (2022a) Minjia Zhang, Niranjan Uma Naresh, and Yuxiong He. 2022a.  Adversarial data augmentation for task-specific knowledge distillation of pre-trained transformers.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 11685–11693, online. AAAI Press.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _Proceedings of the 28th International Conference on Neural Information Processing Systems_ , page 649–657, Montreal, Canada.
  * Zhang et al. (2022b) Yiming Zhang, Shi Feng, and Chenhao Tan. 2022b.  Active example selection for in-context learning.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9134–9148, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021.  Calibrate before use: Improving few-shot performance of language models.  In _Proceedings of the 38th International Conference on Machine Learning_ , pages 12697–12706, Online. ACM.
  * Zhou and Wu (2023a) Xiaoling Zhou and Ou Wu. 2023a.  Implicit counterfactual data augmentation for deep neural networks.  _arXiv preprint arXiv:2304.13431_.
  * Zhou and Wu (2023b) Xiaoling Zhou and Ou Wu. 2023b.  Which samples should be learned first: Easy or hard?  _IEEE Transactions on Neural Networks and Learning Systems_ , pages 1–15.
  * Zhou et al. (2022) Xiaoling Zhou, Ou Wu, and Chao Jiang. 2022.  Increasing naturalness of human–machine dialogue: The users’ choices inference of options in machine-raised questions.  _Knowledge-Based Systems_ , 243:108485.
  * Zhu et al. (2020) Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. 2020.  Freelb: Enhanced adversarial training for natural language understanding.  In _Proceedings of the 8th International Conference on Learning Representations_ , Online.

Dataset | Task | Avg. length | Classes | Balanced
---|---|---|---|---
SST-2 Socher et al. (2013) | Sentiment analysis | 12.4 | 2 | Yes
SST-5 Socher et al. (2013) | Sentiment analysis | 23.1 | 5 | No
MR Pang and Lee (2005) | Sentiment analysis | 25.7 | 2 | Yes
CR Hu and Liu (2004) | Sentiment analysis | 22.1 | 2 | Yes
Amazon McAuley and Leskovec (2013) | Sentiment analysis | 78.5 | 5 | No
Subj Pang and Lee (2004) | Subjectivity classification | 28.9 | 2 | Yes
TREC Voorhees and Tice (2000) | Question classification | 11.6 | 6 | No
DBPedia Lehmann et al. (2015) | Ontology classification | 65.5 | 14 | Yes
AGNews Zhang et al. (2015) | News classification | 53.8 | 4 | Yes
CB De Marneffe et al. (2019) | Natural language inference | 69.7/8.4 | 3 | No
Table 4: Statistical information of ten datasets. The average length is calculated based on the GPT-2 sentence-piece length. For tasks involving sentence pairs, we provide the average length for each individual sentence. Dataset |  Instances |  Label names
---|---|---
SST-2 |  1\. This movie is amazing! (Label = "Positive") 2\. Horrific movie, don’t see it. (Label = "Negative") |  Positive, Negative
SST-5 |  1\. A pretensions – and disposable story — sink the movie. (Label = "Great") 2\. Apparently reassembled from the cutting-room floor of any given daytime soap. (Label = "Terrible") |  Terrible, Bad, Okay, Good, Great
MR |  1\. Lame sweet home leaves no southern stereotype unturned. (Label = "Negative") 2\. Not so much farcical as sour. (Label = "Negative") |  Negative, Positive
CR |  1\. It takes excellent pics and is very easy to use, if you read the manual. (Label = "Negative") 2\. Bluetooth does not work on this phone. (Label = "Negative") |  Negative, Positive
Amazon |  1\. Don’t waste your money if you already have 2003… There isn’t one reason to get this update if you already have MS Money 2003 Deluxe and Business. (Label ="Terrible") 2\. The game was in perfect condition! came before it said it should have by 2 days!! I love the game and I suggest it to a lot of my friends! (Label ="Great") |  Terrible, Bad, Okay, Good, Great
Subj |  1\. This is a story about the warm relationship between a little girl and her father despite the difficult conditions they have to live in. (Label = "Objective") 2\. Too slow, too boring, and occasionally annoying. (Label = "Subjective") |  Subjective, Objective
TREC |  1\. When did the neanderthal man live? (Label = "Number") 2\. How do you get a broken cork out of a bottle? (Label = "Description") |  Description, Entity, Expression, Human, Location, Number
DBPedia |  1\. CMC Aviation is a charter airline based in Nairobi Kenya. (Label = "Company") 2\. Dialectica aemula is a moth of the Gracillariidae family. (Label = "Animal") |  Company, School, Artist, Athlete, Politics, Transportation, Building, Nature, Village, Animal, Plant, Album, Film, Book
AGNews |  1\. Walk in park for Yankees Drained by a difficult week, the New York Yankees needed an uplifting victory. (Label = "Sports") 2\. NASA Mountain View claims world’s fastest computer. (Label = "Technology") |  World, Sports, Business, Technology
CB |  1\. It was a complex language. Not written down but handed down. One might say it was peeled down. The language was peeled down. (Label = "True") 2\. “Do you mind if I use your phone?” Ronni could see that Guido’s brain was whirring. Guido’s brain was whirring. (Label = "True") |  True, False, Neither
Table 5: Examples and label names from all datasets. Dataset | Template | Label mapping
---|---|---
SST-2 |  Review: {Sentence} Sentiment: {Label} | Positive / Negative
SST-5 |  Review: {Sentence} Sentiment: {Label} | terrible / bad / okay / good / great
MR |  Review: {Sentence} Sentiment: {Label} | Positive / Negative
CR |  Review: {Sentence} Sentiment: {Label} | Positive / Negative
Subj |  Input: {Sentence} Type: {Label} | objective / subjective
TREC |  Question: {Sentence} Type: {Label} | description / entity / expression / human / location / number
Amazon |  Review: {Sentence} Sentiment: {Label} | terrible / bad / okay / good / great
AGNews |  Input: {Sentence} Type: {Label} | world / sports / business / technology
DBPedia |  Input: {Sentence} Type: {Label} |  company / school / artist / athlete / politics / transportation building / nature / village / animal / plant / album / film / book
CB |  Premise: {Sentence} Hypothesis: {Sentence} Prediction: {Label} | true / false / neither
Table 6: Prompt templates and label mappings for each dataset.

##  Appendix A Details of Applied Datasets

Table 4 presents comprehensive statistics for all datasets utilized in this
study. The information includes task descriptions, average sentence lengths,
class counts, and details on class imbalance. Additionally, Table 5 provides
sample instances and label names for each of the datasets.

PLM | Method | m | SST-2 | SST-5 | MR | CR | Subj | TREC | DBPedia | AGNews | CB | Avg.
---|---|---|---|---|---|---|---|---|---|---|---|---
13B | Vanilla ICL | 4 | ${95.6}_{{\color[rgb]{0,0,1}7.1}}$ | ${29.5}_{{\color[rgb]{0,0,1}6.2}}$ | ${90.0}_{{\color[rgb]{0,0,1}5.8}}$ | ${91.4}_{{\color[rgb]{0,0,1}7.4}}$ | ${72.9}_{{\color[rgb]{0,0,1}6.9}}$ | ${62.8}_{{\color[rgb]{0,0,1}9.1}}$ | ${80.9}_{{\color[rgb]{0,0,1}7.6}}$ | $80.2_{{\color[rgb]{0,0,1}5.9}}$ | $51.5_{{\color[rgb]{0,0,1}8.2}}$ | 72.8
IDAICL | ${{96.7}}_{{\color[rgb]{0,0,1}2.5}}$ | ${{47.1}}_{{\color[rgb]{0,0,1}1.1}}$ | ${{93.0}}_{{\color[rgb]{0,0,1}1.9}}$ | ${{93.3}}_{{\color[rgb]{0,0,1}0.8}}$ | ${{87.8}}_{{\color[rgb]{0,0,1}2.3}}$ | ${{76.0}}_{{\color[rgb]{0,0,1}2.6}}$ | ${{94.9}}_{{\color[rgb]{0,0,1}1.0}}$ | ${{87.7}}_{{\color[rgb]{0,0,1}2.4}}$ | ${{59.4}}_{{\color[rgb]{0,0,1}1.9}}$ | ${81.8}$
Vanilla ICL | 8 | ${96.7}_{{\color[rgb]{0,0,1}7.1}}$ | ${39.4}_{{\color[rgb]{0,0,1}5.6}}$ | ${92.3}_{{\color[rgb]{0,0,1}6.2}}$ | ${92.2}_{{\color[rgb]{0,0,1}4.8}}$ | ${70.8}_{{\color[rgb]{0,0,1}5.1}}$ | ${71.2}_{{\color[rgb]{0,0,1}9.1}}$ | ${83.7}_{{\color[rgb]{0,0,1}4.2}}$ | ${79.5}_{{\color[rgb]{0,0,1}6.3}}$ | ${52.4}_{{\color[rgb]{0,0,1}3.7}}$ | 75.4
IDAICL | $96.9_{{\color[rgb]{0,0,1}2.1}}$ | $49.2_{{\color[rgb]{0,0,1}1.9}}$ | $93.4_{{\color[rgb]{0,0,1}1.6}}$ | ${92.9}_{{\color[rgb]{0,0,1}1.9}}$ | $87.5_{{\color[rgb]{0,0,1}3.0}}$ | ${79.9}_{{\color[rgb]{0,0,1}2.1}}$ | $93.6_{{\color[rgb]{0,0,1}0.9}}$ | $88.0_{{\color[rgb]{0,0,1}1.7}}$ | $62.4_{{\color[rgb]{0,0,1}2.5}}$ | 82.6
33B | Vanilla ICL | 4 | ${95.5}_{{\color[rgb]{0,0,1}7.2}}$ | ${29.4}_{{\color[rgb]{0,0,1}5.6}}$ | ${91.7}_{{\color[rgb]{0,0,1}5.4}}$ | ${{91.5}}_{{\color[rgb]{0,0,1}8.1}}$ | ${85.1}_{{\color[rgb]{0,0,1}6.0}}$ | ${70.9}_{{\color[rgb]{0,0,1}4.4}}$ | ${86.6}_{{\color[rgb]{0,0,1}4.5}}$ | $76.2_{{\color[rgb]{0,0,1}6.1}}$ | ${{59.2}}_{{\color[rgb]{0,0,1}5.3}}$ | 76.2
IDAICL | ${{96.5}}_{{\color[rgb]{0,0,1}1.1}}$ | ${{46.8}}_{{\color[rgb]{0,0,1}2.4}}$ | ${{93.6}}_{{\color[rgb]{0,0,1}1.3}}$ | ${{92.3}}_{{\color[rgb]{0,0,1}3.3}}$ | ${{89.3}}_{{\color[rgb]{0,0,1}2.4}}$ | ${{79.1}}_{{\color[rgb]{0,0,1}1.5}}$ | ${{95.6}}_{{\color[rgb]{0,0,1}2.3}}$ | ${{88.4}}_{{\color[rgb]{0,0,1}1.9}}$ | ${{64.6}}_{{\color[rgb]{0,0,1}2.8}}$ | ${82.9}$
Vanilla ICL | 8 | ${96.8}_{{\color[rgb]{0,0,1}7.3}}$ | ${34.3}_{{\color[rgb]{0,0,1}5.4}}$ | ${93.4}_{{\color[rgb]{0,0,1}5.8}}$ | ${92.7}_{{\color[rgb]{0,0,1}6.4}}$ | ${83.5}_{{\color[rgb]{0,0,1}5.5}}$ | ${66.9}_{{\color[rgb]{0,0,1}4.8}}$ | ${84.1}_{{\color[rgb]{0,0,1}6.2}}$ | ${84.7}_{{\color[rgb]{0,0,1}5.5}}$ | ${62.0}_{{\color[rgb]{0,0,1}5.2}}$ | 77.6
IDAICL | $96.9_{{\color[rgb]{0,0,1}2.3}}$ | $50.3_{{\color[rgb]{0,0,1}1.5}}$ | $93.9_{{\color[rgb]{0,0,1}2.2}}$ | ${93.0}_{{\color[rgb]{0,0,1}1.4}}$ | $89.0_{{\color[rgb]{0,0,1}1.0}}$ | ${83.1}_{{\color[rgb]{0,0,1}1.7}}$ | $95.9_{{\color[rgb]{0,0,1}2.0}}$ | $88.0_{{\color[rgb]{0,0,1}1.2}}$ | $70.4_{{\color[rgb]{0,0,1}1.8}}$ | 84.5

Table 7: Comparison results of Macro-F1 between Vanilla ICL and IDAICL under
varying values of $m$ on the LLaMA models with 13B and 33B parameters.

PLM | Method | m | SST-2 | SST-5 | MR | CR | Amazon | Subj | TREC | DBPedia | AGNews | CB
---|---|---|---|---|---|---|---|---|---|---|---|---
GPT-2 0.1B | Vanilla ICL | 4 | ${56.3}_{{\color[rgb]{0,0,1}7.1}}$ | ${28.4}_{{\color[rgb]{0,0,1}8.8}}$ | ${55.4}_{{\color[rgb]{0,0,1}7.4}}$ | ${54.2}_{{\color[rgb]{0,0,1}6.2}}$ | $30.8_{{\color[rgb]{0,0,1}8.4}}$ | ${52.9}_{{\color[rgb]{0,0,1}7.9}}$ | ${32.2}_{{\color[rgb]{0,0,1}5.1}}$ | ${44.3}_{{\color[rgb]{0,0,1}6.2}}$ | ${42.8}_{{\color[rgb]{0,0,1}9.3}}$ | ${42.1}_{{\color[rgb]{0,0,1}9.6}}$
IDAICL | $69.5_{{\color[rgb]{0,0,1}2.6}}$ | $35.3_{{\color[rgb]{0,0,1}1.1}}$ | $66.4_{{\color[rgb]{0,0,1}2.3}}$ | ${67.2}_{{\color[rgb]{0,0,1}2.7}}$ | $39.3_{{\color[rgb]{0,0,1}2.9}}$ | $57.2_{{\color[rgb]{0,0,1}2.6}}$ | ${44.3}_{{\color[rgb]{0,0,1}1.8}}$ | $62.2_{{\color[rgb]{0,0,1}2.3}}$ | $65.5_{{\color[rgb]{0,0,1}2.7}}$ | $49.2_{{\color[rgb]{0,0,1}1.9}}$
Vanilla ICL | 8 | ${60.8}_{{\color[rgb]{0,0,1}8.3}}$ | ${30.6}_{{\color[rgb]{0,0,1}6.9}}$ | ${57.5}_{{\color[rgb]{0,0,1}9.7}}$ | ${56.0}_{{\color[rgb]{0,0,1}5.1}}$ | $33.6_{{\color[rgb]{0,0,1}7.8}}$ | ${53.7}_{{\color[rgb]{0,0,1}5.6}}$ | ${33.0}_{{\color[rgb]{0,0,1}10.7}}$ | ${52.1}_{{\color[rgb]{0,0,1}5.8}}$ | ${45.6}_{{\color[rgb]{0,0,1}9.1}}$ | ${45.4}_{{\color[rgb]{0,0,1}6.2}}$
IDAICL | $71.4_{{\color[rgb]{0,0,1}1.8}}$ | $36.1_{{\color[rgb]{0,0,1}2.9}}$ | $67.6_{{\color[rgb]{0,0,1}1.8}}$ | ${68.6}_{{\color[rgb]{0,0,1}2.2}}$ | $40.0_{{\color[rgb]{0,0,1}0.7}}$ | $58.5_{{\color[rgb]{0,0,1}2.5}}$ | ${45.6}_{{\color[rgb]{0,0,1}1.9}}$ | $63.6_{{\color[rgb]{0,0,1}1.1}}$ | $66.9_{{\color[rgb]{0,0,1}1.6}}$ | $50.6_{{\color[rgb]{0,0,1}2.7}}$
Vanilla ICL | 12 | ${64.5}_{{\color[rgb]{0,0,1}6.0}}$ | ${30.8}_{{\color[rgb]{0,0,1}7.1}}$ | ${59.3}_{{\color[rgb]{0,0,1}5.6}}$ | ${59.1}_{{\color[rgb]{0,0,1}8.4}}$ | $33.9_{{\color[rgb]{0,0,1}5.5}}$ | ${56.6}_{{\color[rgb]{0,0,1}8.9}}$ | ${35.8}_{{\color[rgb]{0,0,1}7.1}}$ | ${52.3}_{{\color[rgb]{0,0,1}11.4}}$ | ${47.4}_{{\color[rgb]{0,0,1}6.0}}$ | ${47.4}_{{\color[rgb]{0,0,1}7.7}}$
IDAICL | $72.2_{{\color[rgb]{0,0,1}1.1}}$ | $36.7_{{\color[rgb]{0,0,1}2.2}}$ | $70.1_{{\color[rgb]{0,0,1}1.7}}$ | $69.3_{{\color[rgb]{0,0,1}1.8}}$ | $40.8_{{\color[rgb]{0,0,1}1.2}}$ | $60.9_{{\color[rgb]{0,0,1}1.5}}$ | $47.0_{{\color[rgb]{0,0,1}2.7}}$ | $65.5_{{\color[rgb]{0,0,1}1.9}}$ | $67.8_{{\color[rgb]{0,0,1}2.2}}$ | $51.2_{{\color[rgb]{0,0,1}3.3}}$
Vanilla ICL | 16 | ${64.3}_{{\color[rgb]{0,0,1}6.1}}$ | ${33.5}_{{\color[rgb]{0,0,1}7.1}}$ | ${59.9}_{{\color[rgb]{0,0,1}6.6}}$ | ${61.7}_{{\color[rgb]{0,0,1}7.5}}$ | $34.6_{{\color[rgb]{0,0,1}6.9}}$ | ${56.1}_{{\color[rgb]{0,0,1}6.2}}$ | ${36.9}_{{\color[rgb]{0,0,1}5.7}}$ | ${54.1}_{{\color[rgb]{0,0,1}7.2}}$ | ${47.9}_{{\color[rgb]{0,0,1}8.0}}$ | ${48.9}_{{\color[rgb]{0,0,1}7.7}}$
IDAICL | $72.9_{{\color[rgb]{0,0,1}2.5}}$ | $38.0_{{\color[rgb]{0,0,1}2.4}}$ | $69.7_{{\color[rgb]{0,0,1}1.3}}$ | $69.9_{{\color[rgb]{0,0,1}2.1}}$ | $41.7_{{\color[rgb]{0,0,1}0.9}}$ | $60.6_{{\color[rgb]{0,0,1}1.1}}$ | $46.6_{{\color[rgb]{0,0,1}1.9}}$ | $65.9_{{\color[rgb]{0,0,1}2.6}}$ | $65.7_{{\color[rgb]{0,0,1}1.0}}$ | $51.8_{{\color[rgb]{0,0,1}2.2}}$
GPT-2 0.3B | Vanilla ICL | 4 | ${60.8}_{{\color[rgb]{0,0,1}7.5}}$ | ${26.6}_{{\color[rgb]{0,0,1}6.8}}$ | ${50.5}_{{\color[rgb]{0,0,1}7.1}}$ | ${52.3}_{{\color[rgb]{0,0,1}6.1}}$ | $30.5_{{\color[rgb]{0,0,1}5.2}}$ | ${53.2}_{{\color[rgb]{0,0,1}8.3}}$ | ${32.8}_{{\color[rgb]{0,0,1}8.1}}$ | ${50.5}_{{\color[rgb]{0,0,1}4.8}}$ | ${41.3}_{{\color[rgb]{0,0,1}5.9}}$ | ${42.7}_{{\color[rgb]{0,0,1}7.1}}$
IDAICL | $78.4_{{\color[rgb]{0,0,1}1.7}}$ | $33.1_{{\color[rgb]{0,0,1}2.5}}$ | $66.6_{{\color[rgb]{0,0,1}0.9}}$ | ${70.3}_{{\color[rgb]{0,0,1}2.3}}$ | $40.1_{{\color[rgb]{0,0,1}1.5}}$ | $69.4_{{\color[rgb]{0,0,1}1.7}}$ | ${45.6}_{{\color[rgb]{0,0,1}3.3}}$ | $66.2_{{\color[rgb]{0,0,1}2.1}}$ | $62.8_{{\color[rgb]{0,0,1}3.7}}$ | $50.4_{{\color[rgb]{0,0,1}1.8}}$
Vanilla ICL | 8 | ${58.9}_{{\color[rgb]{0,0,1}8.7}}$ | ${29.4}_{{\color[rgb]{0,0,1}6.1}}$ | ${52.4}_{{\color[rgb]{0,0,1}8.9}}$ | ${54.8}_{{\color[rgb]{0,0,1}8.2}}$ | $32.7_{{\color[rgb]{0,0,1}7.9}}$ | ${53.5}_{{\color[rgb]{0,0,1}6.7}}$ | ${34.0}_{{\color[rgb]{0,0,1}8.2}}$ | ${59.1}_{{\color[rgb]{0,0,1}9.7}}$ | ${43.8}_{{\color[rgb]{0,0,1}6.4}}$ | ${46.9}_{{\color[rgb]{0,0,1}7.6}}$
IDAICL | $80.8_{{\color[rgb]{0,0,1}1.7}}$ | $34.8_{{\color[rgb]{0,0,1}1.9}}$ | $69.5_{{\color[rgb]{0,0,1}1.1}}$ | ${71.5}_{{\color[rgb]{0,0,1}0.8}}$ | $41.5_{{\color[rgb]{0,0,1}1.7}}$ | $70.3_{{\color[rgb]{0,0,1}2.6}}$ | ${46.2}_{{\color[rgb]{0,0,1}2.2}}$ | $68.1_{{\color[rgb]{0,0,1}1.7}}$ | $63.3_{{\color[rgb]{0,0,1}2.1}}$ | $51.5_{{\color[rgb]{0,0,1}2.5}}$
Vanilla ICL | 12 | ${62.9}_{{\color[rgb]{0,0,1}14.4}}$ | ${30.6}_{{\color[rgb]{0,0,1}7.8}}$ | ${55.2}_{{\color[rgb]{0,0,1}6.2}}$ | ${56.1}_{{\color[rgb]{0,0,1}6.7}}$ | $34.2_{{\color[rgb]{0,0,1}7.5}}$ | ${56.8}_{{\color[rgb]{0,0,1}7.1}}$ | ${36.2}_{{\color[rgb]{0,0,1}9.8}}$ | ${58.0}_{{\color[rgb]{0,0,1}7.3}}$ | ${46.5}_{{\color[rgb]{0,0,1}9.3}}$ | ${48.6}_{{\color[rgb]{0,0,1}6.6}}$
IDAICL | $82.2_{{\color[rgb]{0,0,1}2.3}}$ | $36.1_{{\color[rgb]{0,0,1}1.8}}$ | $68.9_{{\color[rgb]{0,0,1}2.4}}$ | ${72.0}_{{\color[rgb]{0,0,1}1.5}}$ | $43.7_{{\color[rgb]{0,0,1}0.6}}$ | $71.4_{{\color[rgb]{0,0,1}2.4}}$ | ${48.3}_{{\color[rgb]{0,0,1}1.3}}$ | $70.5_{{\color[rgb]{0,0,1}1.9}}$ | $65.2_{{\color[rgb]{0,0,1}2.2}}$ | $52.9_{{\color[rgb]{0,0,1}1.4}}$
Vanilla ICL | 16 | ${67.4}_{{\color[rgb]{0,0,1}6.3}}$ | ${31.7}_{{\color[rgb]{0,0,1}7.1}}$ | ${57.6}_{{\color[rgb]{0,0,1}8.6}}$ | ${56.6}_{{\color[rgb]{0,0,1}5.2}}$ | $34.7_{{\color[rgb]{0,0,1}6.2}}$ | ${57.0}_{{\color[rgb]{0,0,1}5.3}}$ | ${38.1}_{{\color[rgb]{0,0,1}6.9}}$ | ${59.3}_{{\color[rgb]{0,0,1}8.2}}$ | ${45.2}_{{\color[rgb]{0,0,1}7.6}}$ | ${49.4}_{{\color[rgb]{0,0,1}8.7}}$
IDAICL | $81.5_{{\color[rgb]{0,0,1}2.8}}$ | $36.8_{{\color[rgb]{0,0,1}1.2}}$ | $70.4_{{\color[rgb]{0,0,1}1.7}}$ | $72.9_{{\color[rgb]{0,0,1}2.1}}$ | $43.1_{{\color[rgb]{0,0,1}1.3}}$ | $71.9_{{\color[rgb]{0,0,1}2.7}}$ | $48.7_{{\color[rgb]{0,0,1}1.1}}$ | $70.9_{{\color[rgb]{0,0,1}2.9}}$ | $65.8_{{\color[rgb]{0,0,1}1.2}}$ | $52.4_{{\color[rgb]{0,0,1}1.8}}$

Table 8: Accuracy comparison between Vanilla ICL and IDAICL under varying values of $m$ on the GPT-2 models with 0.1B and 0.3B parameters. Figure 7: Comparison results between Vanilla ICL and IDAICL across fifteen templates. The evaluation is conducted using the GPT-2 model with 1.5B parameters. The performance of IDAICL exceeds that of Vanilla ICL and demonstrates greater robustness across various templates. PLM | Method | SST-5 | MR | AGNews | TREC | SST-2 | Subj | DBPedia | Avg.
---|---|---|---|---|---|---|---|---|---
GPT-2 1.5B | Vanilla ICL | ${30.8}_{{\color[rgb]{0,0,1}6.1}}$ | ${64.9}_{{\color[rgb]{0,0,1}8.3}}$ | ${57.5}_{{\color[rgb]{0,0,1}6.7}}$ | ${40.4}_{{\color[rgb]{0,0,1}5.1}}$ | ${57.2}_{{\color[rgb]{0,0,1}7.0}}$ | ${57.3}_{{\color[rgb]{0,0,1}10.3}}$ | ${67.6}_{{\color[rgb]{0,0,1}7.5}}$ | 53.7
ConCa | ${32.8}_{{\color[rgb]{0,0,1}7.1}}$ | $74.5_{{\color[rgb]{0,0,1}5.1}}$ | $62.7_{{\color[rgb]{0,0,1}6.1}}$ | ${45.8}_{{\color[rgb]{0,0,1}2.5}}$ | ${73.9}_{{\color[rgb]{0,0,1}8.6}}$ | ${68.3}_{{\color[rgb]{0,0,1}7.4}}$ | ${75.0}_{{\color[rgb]{0,0,1}4.0}}$ | 61.9
PROC${\text{A}}^{*}$ | ${\underline{36.5}}_{{\color[rgb]{0,0,1}4.4}}$ | ${{80.8}}_{{\color[rgb]{0,0,1}6.4}}$ | ${75.5}_{{\color[rgb]{0,0,1}3.2}}$ | ${46.0}_{{\color[rgb]{0,0,1}2.5}}$ | ${\underline{88.0}}_{{\color[rgb]{0,0,1}1.3}}$ | ${\boldsymbol{80.2}}_{{\color[rgb]{0,0,1}3.3}}$ | ${\underline{89.4}}_{{\color[rgb]{0,0,1}0.7}}$ | $\underline{70.9}$
D-ConCa | ${{31.7}}_{{\color[rgb]{0,0,1}3.3}}$ | ${\underline{80.9}}_{{\color[rgb]{0,0,1}3.7}}$ | ${\underline{77.0}}_{{\color[rgb]{0,0,1}4.1}}$ | ${\underline{47.1}}_{{\color[rgb]{0,0,1}2.8}}$ | ${86.5}_{{\color[rgb]{0,0,1}4.4}}$ | ${76.8}_{{\color[rgb]{0,0,1}5.2}}$ | ${86.1}_{{\color[rgb]{0,0,1}6.3}}$ | 69.4
IDAICL | ${\boldsymbol{40.8}}_{{\color[rgb]{0,0,1}1.9}}$ | ${\boldsymbol{82.1}}_{{\color[rgb]{0,0,1}1.2}}$ | ${\boldsymbol{80.8}}_{{\color[rgb]{0,0,1}2.4}}$ | ${\boldsymbol{52.0}}_{{\color[rgb]{0,0,1}2.5}}$ | ${\boldsymbol{89.5}}_{{\color[rgb]{0,0,1}1.8}}$ | ${\underline{80.1}}_{{\color[rgb]{0,0,1}2.9}}$ | ${\boldsymbol{91.0}}_{{\color[rgb]{0,0,1}2.5}}$ | $\boldsymbol{73.8}$
GPT-Neo | Vanilla ICL | ${31.5}_{{\color[rgb]{0,0,1}6.4}}$ | ${70.6}_{{\color[rgb]{0,0,1}8.1}}$ | ${71.9}_{{\color[rgb]{0,0,1}6.8}}$ | ${53.0}_{{\color[rgb]{0,0,1}6.9}}$ | ${74.9}_{{\color[rgb]{0,0,1}8.3}}$ | ${57.9}_{{\color[rgb]{0,0,1}6.3}}$ | ${78.5}_{{\color[rgb]{0,0,1}6.5}}$ | 62.6
ConCa | ${33.9}_{{\color[rgb]{0,0,1}4.3}}$ | $78.2_{{\color[rgb]{0,0,1}5.3}}$ | $73.6_{{\color[rgb]{0,0,1}3.8}}$ | ${55.9}_{{\color[rgb]{0,0,1}7.2}}$ | ${82.0}_{{\color[rgb]{0,0,1}9.5}}$ | ${71.3}_{{\color[rgb]{0,0,1}6.4}}$ | ${90.0}_{{\color[rgb]{0,0,1}3.6}}$ | 69.3
PROC${\text{A}}^{*}$ | ${\underline{39.4}}_{{\color[rgb]{0,0,1}4.0}}$ | ${{77.8}}_{{\color[rgb]{0,0,1}13.9}}$ | ${78.9}_{{\color[rgb]{0,0,1}2.5}}$ | ${56.0}_{{\color[rgb]{0,0,1}3.6}}$ | ${\boldsymbol{91.9}}_{{\color[rgb]{0,0,1}1.2}}$ | ${\underline{81.3}}_{{\color[rgb]{0,0,1}3.8}}$ | ${\underline{92.0}}_{{\color[rgb]{0,0,1}1.5}}$ | $\underline{73.9}$
D-ConCa | ${{32.9}}_{{\color[rgb]{0,0,1}4.1}}$ | ${\underline{84.6}}_{{\color[rgb]{0,0,1}2.8}}$ | ${\underline{81.2}}_{{\color[rgb]{0,0,1}3.9}}$ | ${\underline{57.6}}_{{\color[rgb]{0,0,1}4.7}}$ | ${\underline{91.6}}_{{\color[rgb]{0,0,1}5.3}}$ | ${70.9}_{{\color[rgb]{0,0,1}2.9}}$ | ${85.7}_{{\color[rgb]{0,0,1}3.1}}$ | 72.1
IDAICL | ${\boldsymbol{42.2}}_{{\color[rgb]{0,0,1}2.5}}$ | ${\boldsymbol{85.9}}_{{\color[rgb]{0,0,1}1.6}}$ | ${\boldsymbol{83.1}}_{{\color[rgb]{0,0,1}1.9}}$ | ${\boldsymbol{61.4}}_{{\color[rgb]{0,0,1}1.7}}$ | ${{91.2}}_{{\color[rgb]{0,0,1}2.4}}$ | ${\boldsymbol{82.3}}_{{\color[rgb]{0,0,1}3.1}}$ | ${\boldsymbol{93.0}}_{{\color[rgb]{0,0,1}1.5}}$ | $\boldsymbol{77.0}$
Table 9: Accuracy comparison between IDAICL and other prediction calibration
approaches using the GPT-2 (with 1.5B parameters) and GPT-Neo models, with $m$
setting to 8. The templates used align with those utilized by Han et al. Han
et al. (2023). $*$ indicates that the results were derived from the original
paper.  Figure 8: Results of sensitivity tests for two hyperparameters within
IDAICL, i.e., $\lambda$ and $\tau$, using the GPT-2 model with 0.8B
parameters, with $m$ setting to 12. Optimal performance is achieved when
$\lambda\approx 0.5$ and $\tau\approx 1$.

##  Appendix B Details of Compared Baselines

The compared methods are described as follows:

  * •

Vanilla ICL: We use the PLMs as they are and implement ICL by conditioning it
on a concatenation of $m$ training examples, following the approach outlined
by Brown et al. Brown et al. (2020).

  * •

MetaICL: The fundamental concept underlying MetaICL is to utilize a multi-task
learning framework across a diverse range of meta-training tasks Min et al.
(2022b).

  * •

Channel ICL: It employs a noisy channel approach for language model prompting
in few-shot text classification Min et al. (2022a).

  * •

EPR: It employs language models to autonomously label examples that are
suitable as effective prompts and subsequently trains a prompt retriever based
on this acquired signal Rubin et al. (2022).

  * •

ConCa: It assesses the model’s inclination towards specific answers by
introducing a dummy test input that lacks content Zhao et al. (2021).

  * •

PROCA: The prediction of PROCA is calibrated based on the likelihood of
prototypical clusters Han et al. (2023).

  * •

D-ConCa: It initially assesses the impacts of various label biases by
employing randomly sampled words from the task corpus. During inference, it
utilizes the estimated label bias to calibrate the model’s output
probabilities Fei et al. (2023).

##  Appendix C More Details of Experimental Settings

The entire implementation is conducted utilizing PyTorch Paszke et al. (2019)
and Transformers Wolf et al. (2020). We follow the parameter configurations
and details specified in previous research Min et al. (2022a). The number of
demonstrations is primarily set to $m=12$, but we also explore $m$ values of
$\\{1,4,8,12,16\\}$ in the ablations, with the specific settings detailed in
the respective sections. Demonstration examples for each test sample are
randomly selected from the training data, unless specific methods employ a
specially designed selection method, such as EPR Rubin et al. (2022). The
values of the feature mean and covariance matrix are estimated from the
demonstration set containing demonstration examples corresponding to all test
samples. We depart from the assumption made in previous studies, which
presupposes an equal distribution of training examples across all classes Gao
et al. (2021); Logan IV et al. (2022), in order to facilitate a more realistic
and demanding evaluation.

Each experiment is repeated under five different random seeds. The batch size
is set to 32, and the sequence length is configured to 128 for datasets with
shorter texts, including SST-2 Socher et al. (2013), SST-5 Socher et al.
(2013), MR Pang and Lee (2005), CR Hu and Liu (2004), and TREC Voorhees and
Tice (2000). On the other hand, for datasets with longer input texts,
including AGNews Zhang et al. (2015), DBPedia Lehmann et al. (2015), Subj Pang
and Lee (2004), CB De Marneffe et al. (2019), and Amazon McAuley and Leskovec
(2013), a batch size of 16 and a sequence length of 256 are employed.
Regarding the hyperparameters in IDAICL, the values of $\lambda$ and $\tau$
are fixed at 0.5 and 1, respectively, except in sensitivity tests. The
settings used for the compared methods adhere to those specified in the
original papers Min et al. (2022a, b); Rubin et al. (2022); Zhao et al.
(2021); Han et al. (2023); Fei et al. (2023). Accuracy serves as the primary
evaluation metric, alongside the provided values of Macro-F1 for the LLaMA
model. For each task, a specific template is utilized for inference, as
detailed in Table 6. Additionally, we also examine the impact of different
templates on the performance of IDAICL following those outlined by Zhao et al.
Zhao et al. (2021), which include question-answer templates, conversation-
style templates, prompts resembling web pages, and variations on label names,
as listed in Table 10.

##  Appendix D More Comparison Results

The comparison results between Vanilla ICL and IDAICL on LLaMA models with 13B
and 33B parameters across various datasets are presented in Table 7.
Additionally, the corresponding results for GPT-2 models with 0.1B and 0.3B
parameters are outlined in Table 8. It is evident that IDAICL consistently
outperforms Vanilla ICL across all datasets and different model sizes,
highlighting the high generalization capability of IDAICL. Additionally,
IDAICL showcases reduced performance variance and significantly enhances the
worst-case performance. Based on the findings presented in Table 9, IDAICL
generally outperforms other prediction calibration methods, demonstrating the
significance of statistical properties derived from the input data
distribution in the predictions of PLMs.

Figure 9:  (a) and (b): Average accuracy across ten datasets for various
values of $\lambda$ and $\tau$. Optimal average performance is attained when
$\lambda=0.5$ and $\tau=1$. (c) and (d): Confusion matrices for the SST-2 and
MR datasets under two levels of imbalance, where the proportions of the
negative class in demonstrations are set to 0.1 and 0.2, respectively. When
compared to Vanilla ICL, IDAICL improves the performance of the minor class.
These experiments are conducted on the GPT-2 model with 1.5B parameters,
setting $m$ to 12.  Figure 10: Comparison results between Vanilla ICL and
IDAICL across various demonstrations and permutations. The GPT-2 model with
0.8B parameters is employed for this analysis, setting $m$ to 12. IDAICL
exhibits smaller performance variance across different demonstrations and
permutations compared to Vanilla ICL.

##  Appendix E More Results for Varying Templates

The comparison results between Vanilla ICL and IDAICL under all fifteen prompt
templates are presented in Figure 7, illustrating that IDAICL consistently
enhances both average and worst-case accuracy across all templates.
Furthermore, the performance variance of IDAICL among different templates is
notably smaller when compared to Vanilla ICL, highlighting the robustness of
IDAICL’s performance across diverse templates.

##  Appendix F More Sensitivity and Ablation Studies

We performed sensitivity tests on two hyperparameters within IDAICL: $\lambda$
and $\tau$. These values govern the strength of implicit augmentation and the
influence of the class proportion term, respectively. As depicted in Figure 8,
optimal performance is achieved when $\lambda\\!\approx\\!0.5$ and
$\tau\\!\approx\\!1$ for both datasets. Furthermore, Figures 9(a) and (b)
illustrate the average performance of ten datasets across different
hyperparameter settings. Much like the earlier findings, the best average
performance is identified at $\lambda\\!=\\!0.5$ and $\tau\\!=\\!1$.
Consequently, setting $\lambda$ as 0.5 and $\tau$ as 1 is recommended for real
applications. Furthermore, the performance remains stable within the ranges of
$\lambda\in\\{0.25,0.5,0.75\\}$ and $\tau\in\\{0.5,1,1.5\\}$, indicating that
adjustments can be made within these stable ranges.

##  Appendix G More Results for Imbalanced Labels

The imbalanced label distribution in the training data has a significant
impact on the classification performance of the model Zhou and Wu (2023b);
Zhou et al. (2022). We depicted the confusion matrices for the SST-2 and MR
datasets under two imbalance levels in Figures 9(c) and (d), in which the
proportion of the negative class in demonstrations is set to 0.1 and 0.2.
These results manifest that IDAICL significantly enhances the performance of
the underrepresented classes in comparison to Vanilla ICL, thus proving its
capability to address the class imbalance in demonstrations.

##  Appendix H Varying Demonstration Permutations

Research has substantiated that the performance of ICL is sensitive to the
permutation of demonstrations Lu et al. (2022); Zhao et al. (2021). We
assessed the performance of IDAICL under varying demonstration permutations.
Specifically, we selected ten different sets of twelve training examples from
the SST-2 datasets. For each set of examples, we shuffled the order ten times
and calculated the accuracy for each permutation. The findings are depicted in
Figure 10, indicating that IDAICL exhibits relatively stable performance
across different demonstrations and permutations, while Vanilla ICL
demonstrates high variance.

Format ID | Prompt | Label names
---|---|---
1 |  Review: This movie is amazing! Answer: Positive Review: Horrific movie, don’t see it. Answer: | Positive / Negative
2 |  Review: This movie is amazing! Answer: good Review: Horrific movie, don’t see it. Answer: | good / bad
3 |  My review for last night’s film: This movie is amazing! The critics agreed that this movie was good My review for last night’s film: Horrific movie, don’t see it. The critics agreed that this movie was | good / bad
4 |  Here is what our critics think for this month’s films. One of our critics wrote "This movie is amazing!". Her sentiment towards the film was positive. One of our critics wrote "Horrific movie, don’t see it". Her sentiment towards the film was | positive / negative
5 |  Critical reception [ edit ] In a contemporary review, Roger Ebert wrote "This movie is amazing!". Entertainment Weekly agreed, and the overall critical reception of the film was good. In a contemporary review, Roger Ebert wrote "Horrific movie, don’t see it". Entertainment Weekly agreed, and the overall critical reception of the film was | good / bad
6 |  Review: This movie is amazing! Positive Review? Yes Review: Horrific movie, don’t see it. Positive Review? | Yes / No
7 |  Review: This movie is amazing! Question: Is the sentiment of the above review Positive or Negative? Answer: Positive Review: Horrific movie, don’t see it. Question: Is the sentiment of the above review Positive or Negative? Answer: | Positive / Negative
8 |  Review: This movie is amazing! Question: Did the author think that the movie was good or bad? Answer: good Review: Horrific movie, don’t see it. Question: Did the author think that the movie was good or bad? Answer: | good / bad
9 |  Question: Did the author of the following tweet think that the movie was good or bad? Tweet: This movie is amazing! Answer: good Question: Did the author of the following tweet think that the movie was good or bad? Tweet: Horrific movie, don’t see it Answer: | good / bad
10 |  This movie is amazing! My overall feeling was that the movie was good Horrific movie, don’t see it. My overall feeling was that the movie was | good / bad
11 |  This movie is amazing! I liked the movie. Horrific movie, don’t see it. I | liked / hated
12 |  This movie is amazing! My friend asked me if I would give the movie 0 or 5 stars, I said 5 Horrific movie, don’t see it. My friend asked me if I would give the movie 0 or 5 stars, I said | 0 / 5
13 |  Input: This movie is amazing! Sentiment: Positive Input: Horrific movie, don’t see it. Sentiment: | Positive / Negative
14 |  Review: This movie is amazing! Positive: True Review: Horrific movie, don’t see it. Positive: | True / False
15 |  Review: This movie is amazing! Stars: 5 Review: Horrific movie, don’t see it. Stars: | 5 / 0

Table 10: The templates employed for examining the influence of formats on the
SST-2 dataset, following those outlined by Zhao et al. Zhao et al. (2021). An
example from the training data is used for illustration.