# Revisiting In-context Learning and Gradient Descent Analogy

Lingfeng Shen$\heartsuit$ Aayush Mishra$\heartsuit$ Daniel Khashabi
Johns Hopkins University, Baltimore MD

# Is In-context Learning Equivalent to Gradient Descent?

Lingfeng Shen$\heartsuit$ Aayush Mishra$\heartsuit$ Daniel Khashabi
Johns Hopkins University, Baltimore MD

# When is In-Context Learning Equivalent to Gradient Descent?

Lingfeng Shen$\heartsuit$ Aayush Mishra$\heartsuit$ Daniel Khashabi
Johns Hopkins University, Baltimore MD

# Do Transformers Learn In-context by Gradient Descent?

Lingfeng Shen$\heartsuit$ Aayush Mishra$\heartsuit$ Daniel Khashabi
Johns Hopkins University, Baltimore MD

# Do Transformers really learn in-context by Gradient Descent?

Lingfeng Shen$\heartsuit$ Aayush Mishra$\heartsuit$ Daniel Khashabi
Johns Hopkins University, Baltimore MD

# Do Transformers Really Learn In-context
by Gradient Descent?

Lingfeng Shen$\heartsuit$ Aayush Mishra$\heartsuit$ Daniel Khashabi
Johns Hopkins University, Baltimore MD

# Do pretrained Transformers Really
Learn In-context by Gradient Descent?

Lingfeng Shen$\heartsuit$ Aayush Mishra$\heartsuit$ Daniel Khashabi
Johns Hopkins University, Baltimore MD

###### Abstract

Is In-Context Learning (ICL) implicitly equivalent to Gradient Descent (GD)?
Several recent works draw analogies between the dynamics of GD and the
emergent behavior of ICL in large language models. However, these works make
assumptions far from the realistic natural language setting in which language
models are trained. Such discrepancies between theory and practice, therefore,
necessitate further investigation to validate their applicability.

We start by highlighting the weaknesses in prior works that construct
Transformer weights to simulate gradient descent. Their experiments with
training Transformers on ICL objective, inconsistencies in the order
sensitivity of ICL and GD, sparsity of the constructed weights, and
sensitivity to parameter changes are some examples of a mismatch from the
real-world setting.

Furthermore, we probe and compare the ICL vs. GD hypothesis in a natural
setting. We conduct comprehensive empirical analyses on language models
pretrained on natural data (LLaMa-7B). Our comparisons on various performance
metrics highlight the inconsistent behavior of ICL and GD as a function of
various factors such as datasets, models, and number of demonstrations. We
observe that ICL and GD adapt the output distribution of language models
differently. These results indicate that the equivalence between ICL and GD is
an open hypothesis, requires nuanced considerations and calls for further
studies.

00footnotetext: $\heartsuit$ Authors contributed equally

##  1 Introduction

In-Context Learning (ICL) is an emergent behavior in Large Language Models
(LLMs), which allows them to recognize patterns among demonstrations provided
as prompts and extend these patterns to similar tasks (Brown et al., 2020).
This fascinating on-the-fly learning behavior has motivated ample studies to
better of understand its dynamics.

In particular, a notable line of work tries to explain ICL via Gradient
Descent (GD) (Garg et al., 2022; Zhang et al., 2023). This connection is
interesting because GD has been around for decades and is well-understood,
while ICL is a recent phenomenon that has emerged somewhat surprisingly (Wei
et al., 2022), and is not fully understood. Therefore, a solid formal bridge
between the two approaches would be an exciting finding as it can open new
doors for understanding ICL.

######  Hypothesis 1.

For any Transformer weights resulting from self-supervised pretraining and any
well-defined task, ICL and GD lead to the same outcome.

In this work, we revisit the hypothesis on the equivalence of ICL and GD,
i.e., whether these two approaches to “learning” are functionally equivalent.
Consider 1 that defines a _universal_ notion of equivalence between the ICL
and GD. It defines equivalence as a property that must hold for _any_
Transformer model with parameters that _emerge_ naturally from pretraining on
massive unlabeled data (Brown et al., 2020), and is applicable for _any_
choice of well-defined tasks (Srivastava et al., 2023).

######  Hypothesis 2.

There exist Transformer weights resulting from ICL training such that, for
some well-defined tasks, ICL simulates GD on an implicit model.

However, recent works have instead proven a different goal outlined in 2,
which deviates from 1 in training objective, family of tasks, and even the
definition of GD, as we will see in detail in section 3. This hypothesis
articulates a tangential target: being able to _simulate_ GD on a given task
with _some_ Transformer weights. Achieving this target is mainly concerned
with the expressivity of Transformer architecture and weights (Merrill et al.,
2022; Chiang et al., 2023), ignoring how they may emerge from [pre]training. A
few notable works use this hypothesis to provide a theoretical argument for
the ICL$\approx$GD claim. Specifically, Akyürek et al. (2022); von Oswald et
al. (2023) show (via a different set of arguments) that Transformer-based
architectures (Vaswani et al., 2017), for appropriate choices of parameters,
can process their in-context observations in a way that is equivalent to
running gradient updates on an implicit model’s parameters using the same
demonstrations.

These claims are made under strong assumptions, which raises the question of
whether these hold in practice or not. Specifically, do the recent results
focusing on 2 provide any (even partial) evidence for 1? We argue that
although these works highlight interesting abilities of the Transformer
architecture, their claims about the equivalence between ICL and GD are _too
strong_ for real-world models.

We divide our study into two parts. In the first half (section 3), we present
limitations with prior works’ claims through theoretical and empirical
arguments. We argue that real-world LLMs do _not_ emulate the construction-
based equivalence of ICL and GD (2). We touch upon multiple dimensions like
the objective function used to train Transformers (section 3.1), order
sensitivity of ICL in real pretrained LLMs (section 3.2), and sparsity in the
constructed weight matrices (section 3.3). Finally, as real-world models
operate on highly non-linear decision surfaces, probing the hypothesized
linear implicit models is not trivial. In fact, the equivalence between ICL
and GD is commonly understood as the equivalence in performance resulting from
pretrained models using ICL vs fine-tuned models without ICL. We put this
equivalence to the test (section 4) by presenting a comprehensive empirical
study that reveals that ICL operates and performs differently from GD-based
fine-tuned models on real-world language models across a variety of model
sizes, datasets and the number of demonstrations.

In summary,

  * •

we provide theoretical and empirical arguments against existing theories
regarding the equivalence of ICL and GD,

  * •

we empirically evaluate the equivalence between ICL and GD in the real-world
setting using a variety of metrics and find that the two function quite
differently,

  * •

we highlight the gap in our understanding of the functional behavior of ICL
between theoretical and real-world settings, and call for more nuanced and
realistic studies.

##  2 Background

We first define the problem setting (section 2.1). In doing so, we use
“sampling” to emphasize a priori unknown problem parameters. Namely, sampling
(choosing) a learning problem (task) and correspondingly sampling a pretrained
model as the computational setup for our study. We then cover the two learning
setups studied for equivalence (section 2.2), followed by the treatment of
ICL$\approx$GD hypothesis in recent literature.

###  2.1 Sampling tasks and models

##### Sampling from the space of well-defined tasks.

Consider a family of functions (tasks) $\mathcal{F}$ such that each
$(f:\mathcal{X}\rightarrow\mathcal{Y})\in\mathcal{F}$, maps inputs in the
domain $\mathcal{X}$ to the domain $\mathcal{Y}$. A particular function (task)
$f\in\mathcal{F}$ elicits a sampling process $x\overset{f}{\sim}\mathcal{X}$
which samples inputs from the domain $\mathcal{X}$ such that they are
compatible with $f$. For example, in natural language, $\mathcal{F}$ defines
the space of all tasks that involve mapping from language input to language
output, like question answering, summarization, and translation. However, each
specific task $f$ (e.g., translating English to French) would require specific
inputs (English and not, say German) pertinent to the task. The goal is to
find models that learn (imitate) $f$ by conditioning on a set of examples
$S^{f}=\left\\{S^{f}_{i}=(x_{i},f(x_{i}))\Big{|}f\sim\mathcal{F},x_{i}\overset{f}{\sim}\mathcal{X}\right\\}$.
The model’s competence is then evaluated using a test set
$S^{f}_{\text{test}}=\\{(x^{t}_{i},f(x^{t}_{i}))\\}$ which is disjoint from
$S^{f}$. During the evaluation, only the inputs in $S^{f}_{\text{test}}$ are
passed to the model which we denote as $X^{f}_{\text{test}}$.

##### Sampling from the space of pretrained models.

LLMs like GPT and LLaMa (Brown et al., 2020; Touvron et al., 2023) are
pretrained using the Causal Language Modelling (CLM) objective (Radford et
al., 2019) which is more commonly understood as next-word prediction objective
(Liu et al., 2018). This process of pretraining elicits a family of models
$\mathcal{M}$ depending on the choice of architectures, data, initializations,
etc. Formally, we denote this model $M_{\Theta_{0}}$ with pretrained weights
$\Theta_{0}$, which is one model sampled from a much larger space of low
perplexity pretrained models: $M_{\Theta_{0}}\sim\mathcal{M}$.

###  2.2 Standard Learning Setups

We review the standard treatment of ICL and GD, and introduce the relevant
notation.

##### Inference-time learning: In-context learning (ICL).

We follow the dominant definition of In-context Learning (ICL) (Brown et al.,
2020) which involves conditioning pretrained LLMs with a handful of examples
of task $f$. Given these demonstrations, we want the LLM to perform $f$ on new
inputs. Formally, given demonstrations $S^{f}=\\{S^{f}_{i}\\}_{i=1}^{N}$ and a
test input $x^{t}_{i}\in X_{\text{test}}$, the model $M_{\Theta_{0}}$
generates a label $y_{t}$ when presented as $M_{\Theta_{0}}(S^{f}_{1}\circ
S^{f}_{2}\circ...S^{f}_{N}\circ x^{t}_{i})$ or $M_{\Theta_{0}}(x_{1}\circ
f(x_{1})\circ x_{2}\circ f(x_{2})...x_{N}\circ f(x_{N})\circ x^{t}_{i})$,
where $\circ$ is a delimiter like new-line which separates the instances.
$M_{\Theta_{0}}$ produces a confidence distribution $\in\mathbb{R}^{|V|}$ over
the vocabulary set $V$.

##### Training-time learning: Gradient Descent (GD).

Gradient Descent is an iterative numerical optimization algorithm used to
minimize a given objective with respect to model parameters. Given a model
with initial parameters $\Theta_{0}$ and a differentiable loss function
$\mathcal{J}\in\mathcal{Y}\times\mathcal{Y}\rightarrow\mathbb{R}$, the
algorithm updates the parameters in the direction of the negative gradient
$\nabla_{\Theta_{0}}\mathcal{J}$. GD is a standard optimizer used to train
neural networks including LLMs. Although there are variants like SGD, Adam,
that work well in practice, we focus our study on vanilla GD which calculates
the gradients and takes a step (learning rate $\eta$) of fixed size. In the
context of learning from a set of demonstrations, pretrained models
$M_{\Theta_{0}}\sim\mathcal{M}$ are fine-tuned on a particular task $f$ using
GD by updating model parameters. Formally, parameter updates on the model
$M_{\Theta_{0}}$ are performed for some epochs using the available
demonstrations $S^{f}=\\{S^{f}_{i}=(x_{i},f(x_{i}))\\}_{i=1}^{N}$ as follows:

$\displaystyle\Theta_{1}=\Theta_{0}-\eta\nabla_{\Theta}\left(\frac{1}{N}\sum_{(x_{i},f(x_{i}))\in
S^{f}}\mathcal{J}(M_{\Theta_{0}}(x_{i}),f(x_{i}))\right).$ (1)

After this process, the model is expected to perform this task given a new
test sample directly as input: $M_{\Theta_{1}}(x^{t}_{i})$.

The next section will highlight how recent studies drift from these
conventional definitions of ICL and GD to support another form of equivalence.

##  3 Problems with the current treatment of ICL$\approx$GD hypothesis

The recent works deviate from the standard setting (section 2.2) in various
respects. Specifically, they put restrictive assumptions on both the space of
(pretrained) models $\mathcal{M}$ and the space of tasks $\mathcal{F}$.
Additionally, they impose impractical assumptions on model weights needed to
prove their notion of equivalence between ICL and GD. We coin the terms
$\widehat{\text{ICL}}$ and $\widehat{\text{GD}}$ to denote the frameworks used
in earlier research, and to distinguish them from real-world settings (see
Figure 1).

Figure 1:  ICL vs. GD, in real practice and the recent analysis.

We discuss why these deviations from real practice are non-trivial and offer
little in support of the equivalence between ICL and GD in practical settings.
More precisely, using both theoretical explanations and empirical evidence, we
demonstrate why the parallel between $\widehat{\text{ICL}}$ and
$\widehat{\text{GD}}$ doesn’t necessarily suggest a parallel between ICL and
GD.

###  3.1 Real pretrained LLMs
are not trained with the ICL objective

The widely-known ability of ICL emerges in pretrained models ($\mathcal{M}$)
that are obtained by training on CLM objective as described in section 2.1.
However, recent works use a different set of objectives to investigate.
Recently, (Akyürek et al., 2022; von Oswald et al., 2023; Garg et al., 2022),
the models are trained using the ICL objective:

$\displaystyle\operatorname*{arg\,min}_{\Theta}\mathop{\mathbb{E}}_{\begin{subarray}{c}f\sim\widehat{\mathcal{F}}\\\
x_{i}\overset{f}{\sim}\mathcal{X}\end{subarray}}\Big{[}\mathcal{L}\Big{(}f\left(x_{i}\right),M_{\Theta}\left(x_{1}\circ
f(x_{1})\circ x_{2}\circ f(x_{2})\ldots\circ x_{i}\right)\Big{)}\Big{]}.$

This objective is akin training on the same distribution of tasks that the
model is tested on via ICL. We call this $\widehat{\text{ICL}}$, or the
ability to perform ICL by training on “ICL objective” (cf. Figure 1). For
example, if the target task distribution is linear regression, the model is
trained on the sequence of linear regression instances. This deviates from the
real settings (discussed in section 2.1) in at least two respects:

##### Changing the space of tasks.

The space of tasks is restricted to linear regression tasks
$\widehat{\mathcal{F}}$ which is a tiny fraction of the universe of highly
complex tasks $\mathcal{F}$ that can be simulated by ICL in real LLMs.
Although these works hypothesize a representation learning module that would
elicit corresponding linear tasks in the representation space, it is unclear
how these representations would be learned using the ICL objective.

##### Changing the space of models.

Moreover, optimizing for this objective primes learning of parameters suitable
for the task space $\widehat{\mathcal{F}}$ (narrow set of task, as opposed to
a broad universe of tasks $\mathcal{F}$), resulting in a space of models
$\widehat{\mathcal{M}}$. Therefore, the space of transformers
($\widehat{\mathcal{M}}$) trained using the ICL objective is different from
the space of real language models ($\mathcal{M}$) trained using the CLM
objective, and the relationship between them is neither clear nor discussed in
the recent works.

It is, therefore, misleading to use such simplified settings to make
conclusions about the ICL dynamics in real-world pretrained Transformers.

###  3.2 ICL is not equivalent to order-stable algorithms

We discuss the discrepant sensitivity of ICL and GD to the order in which the
training instances (demonstrations) are provided to each approach. First, we
establish that for two algorithms to be equivalent, they must show identical
behavior to the changes in the ordering of the instances.

Let’s begin with the definition of algorithmic equivalence.

######  Definition 1 (Algorithmic equivalence to ICL).

Consider an optimization algorithm $\mathcal{A}$ that modifies a pretrained
model $M_{\Theta_{0}}\in\mathcal{M}$, using demonstrations
$S=\\{(x_{i},f(x_{i})\\}_{i=1}^{N}$ of a well defined task $f\sim\mathcal{F}$,
i.e., $\Theta_{S}\leftarrow\mathcal{A}(S,M_{\Theta_{0}})$. We call
$\mathcal{A}$ “equivalent” to ICL if and only if the following holds:

$M_{\Theta_{0}}(S_{1}\circ S_{2}\circ...S_{N}\circ
x^{t})=M_{\Theta_{S}}(x^{t})\quad\forall\hskip
2.0ptx_{i},x^{t}\overset{f}{\sim}\mathcal{X}.$ (2)

The following theorem establishes the equivalence of order sensitivity between
ICL and any algorithm $\mathcal{A}$ equivalent to it:

######  Theorem 1 (Algorithmic equivalence implies the same order
sensitivity).

Given a pretrained model $M_{\Theta_{0}}\in\mathcal{M}$, an algorithm
$\mathcal{A}$ equivalent to ICL, and demonstrations
$S=\\{(x_{i},f(x_{i})\\}_{i=1}^{N}$ of a well defined task $f\sim\mathcal{F}$,
let $\sigma_{A},\sigma_{B}$ denote two orders of elements in $S$, such that
$\Theta_{\sigma_{A}}\leftarrow\mathcal{A}(\sigma_{A},M_{\Theta_{0}})$ and
$\Theta_{\sigma_{B}}\leftarrow\mathcal{A}(\sigma_{B},M_{\Theta_{0}})$. Then,

$\underbrace{M_{\Theta_{0}}(\sigma_{A}\circ
x^{t})-M_{\Theta_{0}}(\sigma_{B}\circ x^{t})}_{\text{The order sensitivity of
ICL}}=\underbrace{M_{\Theta_{\sigma_{A}}}(x^{t})-M_{\Theta_{\sigma_{B}}}(x^{t})}_{\text{The
order sensitivity of algorithm $\mathcal{A}$}}\quad\forall\hskip
2.0ptx^{t}\overset{f}{\sim}\mathcal{X}.$ (3)

###### Proof.

The proof trivially follows from definition 1. We know that, $\forall\hskip
2.0ptx^{t}\overset{f}{\sim}\mathcal{X}$ we have:

$\displaystyle M_{\Theta_{0}}(\sigma_{A}\circ x^{t})$
$\displaystyle=M_{\Theta_{\sigma_{A}}}(x^{t}),$ $\displaystyle
M_{\Theta_{0}}(\sigma_{B}\circ x^{t})$
$\displaystyle=M_{\Theta_{\sigma_{B}}}(x^{t}).$

Simply subtracting these two terms proves the theorem. ∎

Figure 2: Order Sensitivity (average standard deviation in output
probabilities over the vocabulary set) of ICL (high) compared to GD (zero) as
measured on the LLaMa-7B model on AGNews dataset. See section 3.2 for details.
The standard deviation is taken across $10$ different orders of 8 ICL demos.

####  3.2.1 ICL$\neq$GD based on order inconsistency

Let’s assume that GD is equivalent to ICL (arrow 1 in Figure 1). We show that
this assumption leads to a contradiction due to their inconsistent order
sensitivity.

##### GD is order-stable.

We know that GD is performed on a batch of samples from the training
distribution, as seen in Equation 1. It does not matter which order the
samples are presented, GD calculates the gradient using the average loss
across all samples, and is therefore agnostic of the order in which they are
calculated. With respect to theorem 1, if $\mathcal{A}=$ GD,
$M_{\Theta_{\sigma_{A}}}=M_{\Theta_{\sigma_{B}}}$ or
$M_{\Theta_{\sigma_{A}}}(x^{t})-M_{\Theta_{\sigma_{B}}}(x^{t})=0$.

##### ICL is not equivalent to GD.

For ICL to be equivalent to any order-stable algorithm like GD, it must also
be order-stable. However, previous research (Lu et al., 2022; Hahn & Goyal,
2023) has demonstrated that ICL is highly sensitive to the order of in-context
samples. This is also easy to see because decoder-only Transformers exhibiting
ICL predict a token only based on what they have seen before it in the input.
A different order of samples would change the behavior of the model.
Therefore, ICL can not be equivalent to GD (arrow 1 in Figure 1). We also
empirically verify this phenomenon by comparing the output distributions
produced by ICL and GD Figure 2 (Details of the experimental setup and
comparisons with variants of GD are deferred to Appendix B).

####  3.2.2 ICL$\neq\widehat{\text{GD}}$ based on order inconsistency

We first define $\widehat{\text{GD}}$ and then present a corollary of the
previous result.

##### Gradient Descent on implicit models ($\widehat{\text{GD}}$).

Akyürek et al. (2022); von Oswald et al. (2023) also hypothesize the existence
of “implicit models” inside the weights of Transformer models. These implicit
models (parameterized to perform linear regression) are constructed into the
weights of the Transformer. When the Transformer is presented with in-context
samples, it can simulate steps of gradient descent on the regression loss
(using these samples) with respect to the parameters of the implicit model.
Formally, for an implicit model with weights $W_{0}$, the Transformer model
$M_{\Theta_{0}}=M_{\Theta_{0}\setminus W_{0},W_{0}}$ with fixed parameters
($\Theta_{0}\setminus W_{0}$) would optimize the weights of the inbuilt
implicit model ($W_{0}$) when presented with in-context samples and make its
final prediction using updated weights ($W_{1}$).

We first define the equivalence of ICL to an algorithm which updates the
implicit model only.

######  Definition 2.

Consider an optimization algorithm $\mathcal{A}$ that modifies the implicit
model weights $W_{0}$ of a pretrained model $M_{\Theta_{0}}\in\mathcal{M}$,
using demonstrations $S=\\{(x_{i},f(x_{i})\\}_{i=1}^{N}$ of a well defined
task $f\sim\mathcal{F}$, i.e., $W_{S}\leftarrow\mathcal{A}(S,W_{0})$. We call
$\mathcal{A}$ “equivalent” to ICL if and only if the following holds:

$M_{\Theta_{0}\setminus W_{0},W_{0}}(S_{1}\circ S_{2}\circ...S_{N}\circ
x^{t})=M_{\Theta_{S}\setminus W_{S},W_{S}}(x^{t}).\quad\forall\hskip
2.0ptx_{i},x^{t}\overset{f}{\sim}\mathcal{X},$ (4)

and $\Theta_{0}\setminus W_{0}=\Theta_{S}\setminus W_{S}$, i.e., the
pretrained models weights only change by the weights of the implicit model.

When the model with implicit model weights $W_{0}$ is provided with in-context
examples, it arrives at updated weights $W_{S}$ using $\mathcal{A}$ without
changing any other weights. This is equivalent to the case when the model
starts with implicit model weights $W_{S}$ and is provided no in-context
examples, so no update happens on the weights via $\mathcal{A}$. Now, based on
Definition 2 and Theorem 1, the following corollary about the equivalence of
order sensitivity between ICL and an equivalent algorithm $\mathcal{A}$, also
holds:

######  Corollary 1.

For a pretrained model $M_{\Theta_{0}}\in\mathcal{M}$, an algorithm
$\mathcal{A}$ equivalent to ICL (according to definition 2) and two orders
$\sigma_{A},\sigma_{B}$ of elements in the demonstration set $S$,
$\forall\hskip 2.0ptx^{t}\overset{f}{\sim}\mathcal{X}$,

$M_{\Theta_{0}\setminus W_{0},W_{0}}(\sigma_{A}\circ
x^{t})-M_{\Theta_{0}\setminus W_{0},W_{0}}(\sigma_{B}\circ
x^{t})=M_{\Theta_{\sigma_{A}}\setminus
W_{\sigma_{A}},W_{\sigma_{A}}}(x^{t})-M_{\Theta_{\sigma_{B}}\setminus
W_{\sigma_{B}},W_{\sigma_{B}}}(x^{t})$ (5)

##### ICL is not equivalent to $\widehat{\text{GD}}$.

Let’s assume that $\widehat{\text{GD}}$ is equivalent to ICL (arrow 3 in
Figure 1) according to definition 2. According to the same argument as in
section 3.2.1, $W_{\sigma_{A}}=W_{\sigma_{B}}$ or
$\Theta_{\sigma_{A}}\setminus
W_{\sigma_{A}},W_{\sigma_{A}}=\Theta_{\sigma_{B}}\setminus
W_{\sigma_{B}},W_{\sigma_{B}}$ or $M_{\Theta_{\sigma_{A}}\setminus
W_{\sigma_{A}},W_{\sigma_{A}}}(x^{t})-M_{\Theta_{\sigma_{B}}\setminus
W_{\sigma_{B}},W_{\sigma_{B}}}(x^{t})=0$. This again implies that for ICL to
be equivalent to $\widehat{\text{GD}}$, it must be order-stable. As shown
previously, empirical evidence shows that ICL is not order-stable and hence
not equivalent to $\widehat{\text{GD}}$ (arrow 3 in Figure 1). We note that
the construction of Akyürek et al. (2022) allows for order sensitivity in GD
as the update is performed on samples one by one instead of the batch update
performed by von Oswald et al. (2023).

###  3.3 Hand-constructed weights and their limits

In this section, we analyze how the weight matrices constructed by von Oswald
et al. (2023) and Akyürek et al. (2022) to establish the equivalence between
ICL and GD, are hard to justify for real-world models (e.g., LLaMa-7B).

We first re-write the weight matrices of Transformers constructed by von
Oswald et al. (2023). Their proposition states that given a reference linear
model $W$, there exist key, query, value and projection matrices
$(W_{K},W_{Q},W_{V},P)$ of a Transformer such that a forward pass in that
Transformer is identical to a gradient descent step on $W$, i.e.,
$e_{j}\leftarrow(x_{j},y_{j})+(0,-\Delta Wx_{j})=(x_{i},y_{i})+PVK^{T}q_{j}$.

The weight update $\Delta W$ is calculated by the mean squared error loss on
the in-context samples as, $\Delta
W=-\eta\nabla_{W}L(W)=-\frac{\eta}{N}\sum_{i=1}^{N}(Wx_{i}-y_{i})x^{T}_{i}$.

They construct $W_{K}=W_{Q}=\begin{pmatrix}I_{x}&0\\\
0&0\end{pmatrix},W_{V}=\begin{pmatrix}0&0\\\ W_{0}&-I_{y}\end{pmatrix}$ and
$P=\frac{\eta}{N}I$, where $I_{x},I_{y}$ and $I$ are identity matrices of size
$N_{x},N_{y}$ and $N_{x}+N_{y}$ respectively. Using these matrices, they
achieve the dynamics of a gradient step in the forward pass of a Linear Self
Attention Layer (without softmax). The construction by Akyürek et al. (2022)
is more complex and requires multiple steps to simulate one step of GD on one
in-context sample. However, the construction is similar in that it is
similarly sparse (see section C.4 in Akyürek et al. (2022)’s appendix). These
constructions raise multiple concerns about their scaling to real-world
models.

##### How does the model arrive at the correct $P$?

In the construction by von Oswald et al. (2023), $P$ is trivially assigned the
value $\frac{\eta}{N}I$ which would change with the number of in-context
samples. No insight is given on how a Transformer model would arrive at this
information and how this formation behaves without any in-context samples. An
edge case is $N=0$ (no demonstrations) which surprisingly makes terms in $P$
go to infinity.

Figure 3: The ability of GPT-J to perform ICL (AGNews) does not change much
over a time cross-section of training while the parameters change steadily.
‘Parameter difference’ refers to the average parameter changes across
$W_{K},W_{Q}$ and $W_{V}$ over all layers. Further results Figure 9.

Figure 4: We show that the sparsity ratio in LLaMA (averaged across layers
with standard deviation shown with shade) is much less than required by
previous works to implement GD.

##### Are real Transformer weights this sparse?

The weight construction by von Oswald et al. (2023) has a lot of extremely
sparse weight matrices. To be precise, $W_{K}$ and $W_{Q}$ would be matrices
with $N_{x}$ terms equal to $1$ in the top left of the diagonal with the rest
of $(N_{x}+N_{y})^{2}-N_{x}$ terms equal to zero. For LLaMa, the embedding
size of the token vector, $N_{x}=N_{y}=4096$. This means that the sparsity
ratio (SR) in the weight matrices should be
$\frac{((N_{x}+N_{y})^{2}-N_{x})}{(N_{x}+N_{y})^{2}}>99.99\%$. The sparsity
ratio in $W_{V}$ should be close to $\approx 75\%$ if we assume each element
in $W_{0}$ to be non-zero. In practice, we find that the sparsity ratio is
much lower for real-world models like LLaMa and GPT-J. As precisely $0$ values
for weights are unlikely, we measured the sparsity ratio in $W_{K},W_{Q}$ and
$W_{V}$ by measuring number of weights less than a threshold ($\delta$).
Figure 4 shows the average sparsity value across layers for LLaMa. GPT-J shows
similar sparsity as well (in Appendix A). Overall, real-world pretrained
transformers have a much lower sparsity ratio than the assumptions.

##### How does ICL evolve during training?

From the given constructions, models need to arrive at very specific weights
to be able to perform gradient descent on in-context samples, but in practice,
we observe models to develop, retain and improve this ability over time in
training when the parameters change significantly (A detailed experimental
setup is deferred to Appendix C). In Figure 4, we look at how the ability to
perform ICL evolves compared with how the model parameters change over time
(for each check-pointed GPT-J model). We measure the average parameter changes
across $W_{K},W_{Q}$ and $W_{V}$ over all layers (compared to the reference
step 3.1e5). This reveals that real Transformers do not settle on one set of
weights (required by previous works for performing GD) but continue to evolve
throughout training. Therefore, it is not a single choice of parameters that
leads to in-context learning, but a sub-space of parameters has this property.
Therefore, to prove the equivalence between GD and ICL, showing it for a
single choice of parameters is not enough.

##  4 ICL vs. GD in realistic settings

In this section, we provide an empirical comparison of ICL vs. GD in realistic
settings. Specifically, we take a language model pretrained on natural data
and use it with ICL demos to get an ICL output. Then, we use the same demos to
fine-tune the model and get the GD output (without context). Next, we compare
the two outputs on a variety of metrics to see how well ICL and GD align in
practice.

###  4.1 Experimental settings

##### Model and benchmarks.

We choose LLaMa (7B) (Touvron et al., 2023) as our primary model for
evaluation. Our model-size comparative studies use the GPT family of models
(as discussed later section 4.2). For benchmarking, we select the following
datasets: AGNews (Zhang et al., 2015), CB (De Marneffe et al., 2019), SST-2
(Socher et al., 2013), RTE (Dagan et al., 2005). In the main text, we show
results on AGNews and defer other corresponding results to Appendix D.

##### Experimental setup.

We evaluate ICL with varying demonstration sizes $N\in\\{1,2,4,8\\}$ and for
GD, we fine-tune the models with the same corresponding ICL demonstrations,
experimenting with a variety of learning rates
$\\{\text{1e-4},\text{5e-4},\text{1e-5},\text{5e-5}\\}$ over 200 epochs, which
ensures the convergence of model. Specifically, the objective function of GD
is:

$\mathcal{J}=\sum_{(x,y)\in S}\mathcal{L}_{\text{clm}}(y;x),$ (6)

where $\mathcal{L}_{\text{clm}}(y;x)$ is the CLM loss of $y$, given $x$ as the
prefix.

##### Evaluation metrics.

Previous works often use performance metrics (accuracy and loss) based on the
token with the maximum probability from _label set_ $\mathcal{Y}$, when
comparing different algorithms Srivastava et al. (2023); Wei et al. (2021). We
argue that these metrics do not paint the whole picture. Even if two different
sorting algorithms reach the same result, their dynamics may be very
different. In particular, we argue that the relative uplifting of tokens in
the output distribution sheds light on the dynamics of the algorithm. For this
purpose, we propose two more metrics that help evaluate the equivalence
between the dynamics of ICL and GD and also change the matching criteria.

  * •

Performance: As all of our datasets have classification tasks, we compute the
accuracy by picking the top token from the whole vocabulary $V$ instead of
just the label set $\mathcal{Y}$. Checking whether the correct token made it
to the top evaluates the precision of the optimization algorithm. It is
defined as $\frac{1}{|S_{\text{test}}|}\sum_{(x^{t}_{i},y^{t}_{i})\in
S_{\text{test}}}\bm{1}\\{y^{t}_{i}=\operatorname*{arg\,max}M(C\circ
x^{t}_{i})\\}$, where $M$ is the model, $C$ is the context and
$S_{\text{test}}$ is the test set. Notice that this is an algorithm-specific
metric, i.e., we can calculate it using a single output distribution produced
by the model $M$.

  * •

Token Overlap: This is a relative metric is computed based on two output
distributions. These distributions could be either produced by the same model
on different inputs (in case of ICL: different number of demos, order of
demos, etc.) or different models on the same inputs (ICL (with context) vs GD
(fine-tuned, without context)). We compute the top-$K$ tokens (denoted by
$T^{1}_{K}$ and $T^{2}_{K}$) from each output distribution and find the
overlap between these sets. This illustrates how differently the two models
(or inputs) change the probability weights of tokens, evaluating the
difference between their dynamics. The token overlap metric is calculated as
$\frac{1}{K}|T_{K}^{1}\cap T_{K}^{2}|$. We use $K=10$ as it fairly represents
most of the probability mass of the output distribution.

  * •

_Overlap Cosine Similarity_ (OCS): Token overlap evaluates each of the top-$K$
tokens with the same weight. With confidence overlap, we measure the agreement
at a finer level, looking at how well the tokens agree at the individual
level. This metric is computed on the confidence distribution on top-$K$
tokens. This is done because the vocabulary set is large and most tokens have
low probabilities so the overall cosine similarity is always close to $1$. As
the top-$K$ tokens may not be the same in the two distributions being
compared, we denote the intersection of the two sets $T_{K}^{1},T_{K}^{2}$ by
$O=T_{K}^{1}\cap T_{K}^{2}$ and used the following formula:
$\frac{\sum_{t_{i}\in O}p^{1}(t_{i})\cdot p^{2}(t_{i})}{\sqrt{(\sum_{t_{i}\in
O}{p^{1}(t_{i})^{2}})\cdot(\sum_{t_{i}\in
O}{p^{2}(t_{i})^{2}})\cdot(K-|O|)}}$. It measures the cosine distance between
the overlapping tokens and assumes all the other tokens have zero overlap,
therefore normalizing by $\sqrt{(K-|O|)}$ (when $K=|O|$, we divide by
$\sqrt{1}$).

We evaluate every metric across three random seeds and compute the average and
variance. Each random seed is used to sample demos for use in ICL experiments.
The same demos are used to fine-tune models for GD. For the relative metrics
(Token Overlap and Confidence Overlap), the values for ICL are calculated
between predictions made for the same set of demos but in a different order,
also underlining the high order sensitivity of ICL.

###  4.2 On the gap between ICL and GD

With the objective function defined in Equation 6, we investigate the gap
between ICL and GD with our three evaluation metrics.

((a)) Performance comparison between ICL and GD as number of demonstrations
increase

((b)) Token Overlap between ICL and GD

((c)) Confidence Overlap between ICL and GD

Figure 5: Comparison of ICL and GD on our three evaluation metrics for the
AGNews dataset. Note that the ICL plots for each metric surpass the
corresponding best-performing GD plot, highlighting the substantial gap
between ICL and GD (highlighted by the gray diagonal lines).

##### Impact of the number of demonstrations on the gap.

The results, as seen in Figure 5, highlight a notably consistent gap between
ICL and GD across all three metrics. (1) It highlights that even if
performance looks similar (for 1 ICL demo), the other nuanced metrics reveal
that the token and confidence overlap between the two is significantly
different which must result from a different dynamic. (2) Both token and
confidence overlap between ICL and GD saturate at a very low level,
underlining the functional differences between the two. Results on other
datasets are similar and are deferred to Appendix D due to limited space.

##### Impact of model capacity on the gap.

Next, we investigate the influence of model size on the gap between ICL and
GD. Specifically, we fix the number of demonstration size as 8, and select
GPT2-XL (Radford et al., 2019), GPT-Neo (Black et al., 2021), GPT-J (Wang &
Komatsuzaki, 2021) as models of choice to conduct ICL vs GD experiments.
Please note that the model capacity is ranked as follows: LLaMa (7B) >GPT-J
(6B)>GPT-Neo (2.7B)>GPT2-XL (1.5B). The results are shown in Figure 6, from
where we can see the gap basically does not change significantly as the model
changes from GPT2-XL to LLaMa.

((a)) Performance comparison between ICL and GD as model size varies

((b)) Token overlap between ICL and GD

((c)) Confidence overlap between ICL and GD

Figure 6: Comparison of ICL and GD for the AGNews dataset as model-size
varies. The substantial gap between ICL and GD (the gray slanted lines) does
not seem to change with model size.

##  5 Related Work

We review different categories of works that try to explain in-context
learning.

##### Functional explanations.

Many works offer functional explanations of ICL (Liu et al., 2022; Olsson et
al., 2022; Schlag et al., 2021). Among these, most pertinent to our work are
explanations via GD. Akyürek et al. (2022) showed that Transformers can
implement learning algorithms (gradient descent or closed-form OLS) for linear
regression problems and empirically showed that the optimality of algorithms
implemented experience a phase shift with increasing model size. Raventós et
al. (2023) discover similar results about algorithm discovery and phase shifts
with increasing task diversity. Dai et al. (2022) similarly show a dual
between attention layers and linear layers optimized using gradient descent.
Li et al. (2023) show such an equivalence on softmax regression tasks. Garg et
al. (2022); Zhang et al. (2023) then show that using an ICL objective,
Transformers can learn not only linear function classes in-domain, but also
out-of-domain and even sparse linear functions, decision trees and two layered
ReLU neural networks. Ahn et al. (2023) show that Transformers trained on
random instances of linear regression also develop the ability to perform GD
on linear regression tasks. Finally, von Oswald et al. (2023) showed a similar
construction with a simpler Linear Self-Attention Transformer, claiming that
Transformers learn in-context using gradient descent on linear regression
problems. Notably, Akyürek et al. (2022) found this GD behavior applicable
only in small models, with bigger models exhibiting Bayes optimal learning
behavior (like Ordinary Least Squares for linear regression). In contrast, von
Oswald et al. (2023) claim that bigger Transformers also implement GD with
added data transformations.

Most of this line of work shows how Transformers have the ability to implement
such algorithms resulting from training on ICL objectives (2) and not that
real-world models pretrained on natural data develop this ability (1).

##### Distributional explanations.

This body of work explains ICL via distributional frameworks and the relevant
properties of LLMs (Xie et al., 2021; Wies et al., 2023). Xie et al. (2021)
explain ICL as implicit Bayesian inference which implicitly maps a given set
of demonstrations to an appropriate latent concept (task) that is learned via
pretraining on a massive unsupervised corpus. Similarly, Hahn & Goyal (2023)
theorize that natural language pretraining data consists of compositional
structure which leads to the emergent ability of in-context learning, while
Chan et al. (2022) show that this might be because of distributional
properties of the training distribution (like burstiness). These are all
reasonable explanations of how ICL works, although they are somewhat
tangential to the focus of this study.

##### Empirical explanations.

There are a variety of empirical works that study ICL under various settings
(Brown et al., 2020; Zhao et al., 2021; Min et al., 2022; Mishra et al., 2022;
Han et al., 2023). To note a few, Srivastava et al. (2023) famously
benchmarked ICL for a large number of tasks and models. Perez et al. (2021);
Lu et al. (2022) showed the sensitivity of ICL to the choice of demonstrations
and their orderings. Shin et al. (2022); Razeghi et al. (2022) show the
sensitivity of ICL performance to frequency and size of the relevant
pretraining corpus. Pan et al. (2023) disentangle task recognition and task
learning in ICL. These works highlight numerous ways the ability of models to
perform ICL changes under different conditions but do not explain how it
functions.

##  6 Discussion and Conclusion

When ICL was first discovered in LLMs (Radford et al., 2019; Brown et al.,
2020), the performance of solely pretrained models with a few examples in
context could match or even surpass task-specific fine-tuned models. This led
many to believe in the idea that the ability to perform ICL is achieved in
Transformer models by implementing implicit fine-tuning. Some recent works
attributed this ability to gradient descent by constructing weights that
simulate GD on parameters of an implicit model in their forward pass. This
work presents arguments and evidence pointing to the contrary of implicit
fine-tuning behavior. These prior works discover interesting abilities of the
Transformer architecture and show that there exist weights of Transformers
which can simulate fine-tuning on in-context training samples in simpler
laboratory settings. However, this does not imply that real-world models
actually do simulate it.

We first presented both theoretical and empirical arguments against
construction-based equivalence between ICL and GD that hypothesize the
existence of implicit models that are updated using GD when in-context demos
are provided to a Transformer. We then presented empirical evidence against
the equivalence of ICL and GD in the realistic setting where GD refers to
models fine-tuned on in-context demos. Besides using in-context
demonstrations, recent work has also discovered other ways in which in-context
prompts enhance the performance of LLMs. For example, appending prompts like
“Think step by step” (Kojima et al., 2022) or “Take a deep breath and think”
(Yang et al., 2023) before asking a task-specific question has been shown to
improve zero-shot performance of LLMs drastically. All this evidence leads us
to believe that the ability of ICL should not be considered equivalent to any
optimization algorithm like GD solely on the basis of theoretical
constructions and experiments in oversimplified settings. Understanding ICL
dynamics likely requires a more holistic theory that considers various nuances
about this remarkable learning paradigm.

We acknowledge the statement of 1, which defines one universal notion of
equivalence between ICL and GD, may be too strong and hard to investigate. A
more realistic hypothesis may require slightly more restricted conditions,
such as equivalence under some distributional properties of the target task or
the number of demonstrations. Since the true nature of such conditions are
unclear to us, we have chosen the most general statement.

With this work, we want to emphasize the distinction between naturally
emergent ICL in LLMs (via pretraining on natural text data; 1) and their
ability to simulate gradient descent steps (2). Although 2 results in insights
about the expressivity of the Transformer, it does not reveal much information
about 1. We hope this work motivates alternative approaches that reveal the
true nature of in-context learning in pretrained LLMs.

## Limitations and Future Opportunities

Our arguments against construction-based approaches for equivalence between
ICL and GD rely on the standard definition of GD (using batch updates). This
equivalence under other variants of GD (accelerated GD, Adam, adaptive GD) or
higher-order optimization (Nesterov, 1983; Duchi et al., 2011; Kingma & Ba,
2015) are not explored in the literature as well as in our paper. Although
Akyürek et al. (2022) do not explicitly show how their construction would
simulate steps of GD on multiple IC samples, their GD update would be order-
sensitive as it happens on one sample at a time. Lastly, our empirical
evidence is presented against the equivalence of ICL and the commonly
understood definition of GD (for fine-tuned models). We would like to explore
the equivalence of ICL and GD on the hypothesized implicit models in real-
world models, which requires identifying GD circuits inside large language
models.

##  7 Acknowledgements

The authors would also like to acknowledge the gifts from Amazon and the Allen
Institute for AI. We are grateful to Jason Eisner, Angie Liu, Johannes von
Oswald, Holden Lee, and Tianjian Li for their insightful discussion throughout
this project. GPU machines for conducting experiments were provided by ARCH
Rockfish cluster (https://www.arch.jhu.edu).

## References

  * Ahn et al. (2023) Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra.  Transformers learn to implement preconditioned gradient descent for in-context learning.  _arXiv preprint arXiv:2306.00297_ , 2023.
  * Akyürek et al. (2022) Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou.  What learning algorithm is in-context learning? investigations with linear models.  In _International Conference on Learning Representations (ICLR)_ , 2022.  URL https://arxiv.org/abs/2211.15661.
  * Black et al. (2021) Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman.  GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow.  March 2021.  doi: 10.5281/zenodo.5297715.  URL https://doi.org/10.5281/zenodo.5297715.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.  URL https://arxiv.org/abs/2005.14165.
  * Chan et al. (2022) Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill.  Data distributional properties drive emergent in-context learning in transformers.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 35:18878–18891, 2022.  URL https://arxiv.org/abs/2205.05055.
  * Chiang et al. (2023) David Chiang, Peter Cholak, and Anand Pillay.  Tighter bounds on the expressivity of transformer encoders.  _arXiv preprint arXiv:2301.10743_ , 2023.  URL https://arxiv.org/abs/2301.10743.
  * Dagan et al. (2005) Ido Dagan, Oren Glickman, and Bernardo Magnini.  The pascal recognising textual entailment challenge.  In _Machine Learning Challenges Workshop_ , 2005.  URL https://link.springer.com/chapter/10.1007/11736790_9.
  * Dai et al. (2022) Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei.  Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers.  _arXiv preprint arXiv:2212.10559_ , 2022.  URL https://arxiv.org/abs/2212.10559.
  * De Marneffe et al. (2019) Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser.  The CommitmentBank: Investigating projection in naturally occurring discourse.  In _proceedings of Sinn und Bedeutung_ , 2019.  URL https://ojs.ub.uni-konstanz.de/sub/index.php/sub/article/view/601.
  * Duchi et al. (2011) J. Duchi, E. Hazan, and Y. Singer.  Adaptive subgradient methods for online learning and stochastic optimization.  _Journal of Machine Learning Research (JMLR)_ , 12:2121–2159, 2011.  URL https://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf.
  * Garg et al. (2022) Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant.  What can transformers learn in-context? a case study of simple function classes.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 35:30583–30598, 2022.  URL https://arxiv.org/abs/2208.01066.
  * Hahn & Goyal (2023) Michael Hahn and Navin Goyal.  A theory of emergent in-context learning as implicit structure induction.  _arXiv preprint arXiv:2303.07971_ , 2023.  URL https://arxiv.org/abs/2303.07971.
  * Han et al. (2023) Xiaochuang Han, Daniel Simig, Todor Mihaylov, Yulia Tsvetkov, Asli Celikyilmaz, and Tianlu Wang.  Understanding in-context learning via supportive pretraining data.  _arXiv preprint arXiv:2306.15091_ , 2023.  URL https://arxiv.org/abs/2306.15091.
  * Kingma & Ba (2015) Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  abs/1412.6980, 2015.  URL https://arxiv.org/abs/1412.6980.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large language models are zero-shot reasoners.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022.  URL https://arxiv.org/abs/2205.11916.
  * Li et al. (2023) Shuai Li, Zhao Song, Yu Xia, Tong Yu, and Tianyi Zhou.  The closeness of in-context learning and weight shifting for softmax regression.  _arXiv preprint arXiv:2304.13276_ , 2023.  URL https://arxiv.org/abs/2304.13276.
  * Liu et al. (2022) Bingbin Liu, Jordan T Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang.  Transformers learn shortcuts to automata.  _arXiv preprint arXiv:2210.10749_ , 2022.  URL https://arxiv.org/abs/2210.10749.
  * Liu et al. (2018) Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer.  Generating wikipedia by summarizing long sequences.  In _International Conference on Learning Representations (ICLR)_ , 2018.  URL https://arxiv.org/abs/1801.10198.
  * Lu et al. (2022) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_ , 2022.  URL https://arxiv.org/pdf/2104.08786.pdf.
  * Merrill et al. (2022) William Merrill, Ashish Sabharwal, and Noah A Smith.  Saturated transformers are constant-depth threshold circuits.  _Transactions of the Association for Computational Linguistics (TACL)_ , 10:843–856, 2022.  URL https://arxiv.org/abs/2106.16213.
  * Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer.  Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?  _arXiv preprint arXiv:2202.12837_ , 2022.  URL https://arxiv.org/abs/2202.12837.
  * Mishra et al. (2022) Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi.  Reframing instructional prompts to gptk’s language.  In _Annual Meeting of the Association for Computational Linguistics (ACL) - Findings_ , 2022.  URL https://arxiv.org/abs/2109.07830.
  * Nesterov (1983) Yurii Evgen’evich Nesterov.  A method of solving a convex programming problem with convergence rate o$\backslash$bigl(k^2$\backslash$bigr).  In _Doklady Akademii Nauk_ , volume 269, pp. 543–547. Russian Academy of Sciences, 1983.  URL https://www.mathnet.ru/php/archive.phtml?wshow=paper&jrnid=dan&paperid=46009&option_lang=eng.
  * Olsson et al. (2022) Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al.  In-context learning and induction heads.  _arXiv preprint arXiv:2209.11895_ , 2022.  URL https://arxiv.org/abs/2209.11895.
  * Pan et al. (2023) Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen.  What in-context learning “learns” in-context: Disentangling task recognition and task learning.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , July 2023.  URL https://aclanthology.org/2023.findings-acl.527.
  * Perez et al. (2021) Ethan Perez, Douwe Kiela, and Kyunghyun Cho.  True few-shot learning with language models.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021.  URL https://proceedings.neurips.cc/paper/2021/file/5c04925674920eb58467fb52ce4ef728-Paper.pdf.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 2019.  URL https://openai.com/blog/better-language-models/.
  * Raventós et al. (2023) Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli.  Pretraining task diversity and the emergence of non-bayesian in-context learning for regression.  _arXiv preprint arXiv:2306.15063_ , 2023.
  * Razeghi et al. (2022) Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh.  Impact of pretraining term frequencies on few-shot reasoning.  _arXiv preprint arXiv:2202.07206_ , 2022.  URL https://arxiv.org/abs/2202.07206.
  * Schlag et al. (2021) Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber.  Linear transformers are secretly fast weight programmers.  In _International Conference on Machine Learning (ICML)_ , pp. 9355–9366, 2021.  URL http://proceedings.mlr.press/v139/schlag21a.html.
  * Shin et al. (2022) Seongjin Shin, Sang Woo Lee, Hwijeen Ahn, Sungdong Kim, Hyoung Seok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Jung Woo Ha, et al.  On the effect of pretraining corpora on in-context learning by a large-scale language model.  In _Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_ , 2022.  URL https://arxiv.org/abs/2204.13509.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 1631–1642, 2013.  URL https://aclanthology.org/D13-1170.pdf.
  * Srivastava et al. (2023) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _Transactions on Machine Learning Research (TMLR)_ , 2023.  URL https://arxiv.org/abs/2206.04615.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  LLaMA: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.  URL https://arxiv.org/abs/2302.13971.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is All You Need.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017.  URL https://arxiv.org/abs/1706.03762.
  * von Oswald et al. (2023) Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov.  Transformers learn in-context by gradient descent.  In _International Conference on Learning Representations (ICLR)_ , pp. 35151–35174, 2023.  URL https://arxiv.org/abs/2212.07677.
  * Wang & Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax, May 2021.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations (ICLR)_ , 2021.  URL https://arxiv.org/abs/2109.01652.
  * Wei et al. (2022) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_ , 2022.  URL https://arxiv.org/abs/2206.07682.
  * Wies et al. (2023) Noam Wies, Yoav Levine, and Amnon Shashua.  The learnability of in-context learning.  _arXiv preprint arXiv:2303.07895_ , 2023.  URL https://arxiv.org/abs/2303.07895.
  * Xie et al. (2021) Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma.  An explanation of in-context learning as implicit bayesian inference.  In _International Conference on Learning Representations_ , 2021.  URL https://arxiv.org/abs/2111.02080.
  * Yang et al. (2023) Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen.  Large language models as optimizers.  _arXiv preprint arXiv:2309.03409_ , 2023.  URL https://arxiv.org/abs/2309.03409.
  * Zhang et al. (2023) Ruiqi Zhang, Spencer Frei, and Peter L Bartlett.  Trained transformers learn linear models in-context.  _arXiv preprint arXiv:2306.09927_ , 2023.  URL https://arxiv.org/abs/2306.09927.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun.  Character-level convolutional networks for text classification.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2015.  URL https://proceedings.neurips.cc/paper/2015/file/250cf8b51c773f3f8dc8b4be867a9a02-Paper.pdf.
  * Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh.  Calibrate before use: Improving few-shot performance of language models.  In _International Conference on Machine Learning (ICML)_ , pp. 12697–12706, 2021.  URL http://proceedings.mlr.press/v139/zhao21c/zhao21c.pdf.

## Supplementary Material

##  Appendix A Sparsity of GPT-J

We show the sparsity of GPT-J (6B) in Figure 7, from where we can observe a
gigantic gap between the sparsity of real-world LLM and theoretical assumption
in previous works.

Figure 7: GPT-J, like LLaMA-7B, is also much less sparse than required by
previous theory to implement GD.

##  Appendix B Order sensitivity of ICL and GD-based algorithms (experimental
setup)

We present empirical evidence highlighting the distinct sensitivities of GD-
based algorithms and ICL with respect to data order. Specifically, we assess
the variation in confidence assigned to vocabulary $V$ by the model across
different data orderings. Given a set of confidence vectors
$\left\\{p_{i}\right\\}_{i=1}^{N}$ resulting from distinct data orders
$\left\\{\sigma_{i}\right\\}_{i=1}^{N}$, we calculate the standard deviation
for each dimensionality within $V$ using the samples
$\left\\{p_{i}\right\\}_{i=1}^{N}$. Subsequently, the variances for individual
tokens are aggregated.

##### Experimental setup

We evaluate the order sensitivity of GD-based algorithms using the GD, SGD,
and Adam optimizers. The chosen learning rates are 1e-4, 1e-5, 5e-4, and 5e-5.
Our experiments are conducted on the AGNews dataset using the LLaMa-7B model.
We set the number of demonstrations to 8, and for SGD and Adam, the mini-batch
size is fixed at 2. GD training continues for 200 epochs to guarantee
convergence. The number $N$ of orders $\left\\{\sigma_{i}\right\\}_{i=1}^{N}$
is set as 10.

##### Results

In Figure 8, we present the results which highlight several key observations.
First, ICL exhibits a much more pronounced sensitivity to data order compared
to the three GD-based algorithms. Second, as GD training progresses, its
sensitivity diminishes, widening the divergence towards ICL. Overall, these
findings underscore distinct behaviors of ICL and GD-based algorithms with
respect to data order. This suggests a disparity between ICL and GD, as shown
in Theorem 1.

Figure 8: Order Sensitivity (average standard deviation in output
probabilities over the vocabulary set) of ICL compared to GD (+ variants) as
measured on the LLaMa-7B model on AGNews dataset. The standard deviation is
taken across $10$ different orders of 8 ICL demos. For SGD and Adam, batch
size $=2$.

##  Appendix C How does ICL evolve during training (experimental setup)

##### Experimental setup.

We chose intermediate checkpoints from GPT-J, ranging from 310k to 380k
pretraining steps. Using these varied pretraining steps, our approach
simulates the fine-tuning process. Specifically, we focus on two metrics to
quantify the magnitude of fine-tuning: (1) Step Gap: This represents the
difference in pretraining steps between selected checkpoints. (2) Parameter
Gap: In line with the assumptions made by Oswald et al. (von Oswald et al.,
2023), we compute the average differences for each parameter within the
$W_{K}$, $W_{Q}$, and $W_{V}$ matrices across different checkpoints. To
evaluate the ICL capacity of the models, we conducted tests on AGNews, SST-2,
CB, and RTE using 8 demonstrations.

##### Results.

The results are shown in Figure 9, from where we can observe that there is no
significant gap between ICL capacity of different checkpoints, indicating that
continued fine-tuning (pretraining) will not substantially hurt the ICL
performance.

((a)) AGNews ((b)) SST-2 ((c)) CB ((d)) RTE

Figure 9: The ability of GPT-J to perform ICL does not change much over a time
cross-section of training while the parameters change steadily.

##  Appendix D Additional results on ICL vs GD comparisons

((a)) Performance comparison between ICL and GD as number of demonstrations
increase

((b)) Token overlap between ICL and GD

((c)) Confidence overlap between ICL and GD

Figure 10: Comparison of ICL and GD on our three evaluation metrics for the
SST dataset. Note that the ICL plots for each metric surpass the corresponding
best-performing GD plot highlighting the substantial gap between ICL and GD
(highlighted by the gray slanted lines).

((a)) Performance comparison between ICL and GD as number of demonstrations
increase

((b)) Token overlap between ICL and GD

((c)) Confidence overlap between ICL and GD

Figure 11: Comparison of ICL and GD on our three evaluation metrics for the CB
dataset. Note that the ICL plots for each metric surpass the corresponding
best-performing GD plot highlighting the substantial gap between ICL and GD
(highlighted by the gray slanted lines).

((a)) Performance comparison between ICL and GD as number of demonstration
increases

((b)) Token overlap between ICL and GD

((c)) Confidence overlap between ICL and GD

Figure 12: Comparison of ICL and GD on our three evaluation metrics for the
RTE dataset. Note that the ICL plots for each metric surpass the corresponding
best-performing GD plot highlighting the substantial gap between ICL and GD
(highlighted by the gray slanted lines).