#  Finite groups in which every maximal subgroup is nilpotent or normal or has
$p^{\prime}$-order ††thanks: J. Shi was supported by Shandong Provincial
Natural Science Foundation, China (ZR2017MA022 and ZR2020MA044) and NSFC
(11761079). R. Shen was supported by NSFC (12161035).
∗∗ Corresponding author.
E-mail addresses<EMAIL_ADDRESS>(J. Shi<EMAIL_ADDRESS>(N. Li),
<EMAIL_ADDRESS>(R. Shen).

Jiangtao Shi${}^{a,\,\ast\ast}$, Na Lib, Rulin Shenc

a School of Mathematics and Information Sciences, Yantai University, Yantai
264005, P.R. China
b Department of Mathematics, Beijing Jiaotong University, Beijing 100044, P.R.
China
c Department of Mathematics, Hubei Minzu University, Enshi 445000, P.R. China

###### Abstract

Let $G$ be a finite group and $p$ a fixed prime divisor of $|G|$. Combining
the nilpotence, the normality and the order of groups together, we prove that
if every maximal subgroup of $G$ is nilpotent or normal or has
$p^{\prime}$-order, then (1) $G$ is solvable; (2) $G$ has a Sylow tower; (3)
There exists at most one prime divisor $q$ of $|G|$ such that $G$ is neither
$q$-nilpotent nor $q$-closed, where $q\neq p$.

Keywords: maximal subgroup; nilpotent; normal; $p^{\prime}$-order;
$q$-nilpotent; $q$-closed
MSC(2010): 20D10

##  1 Introduction

In this paper all groups are assumed to be finite. It is known that a group
$G$ is nilpotent or minimal non-nilpotent if every maximal subgroup of $G$ is
nilpotent, and a group $G$ is nilpotent if every maximal subgroup of $G$ is
normal, see [5, Theorem 9.1.9] and [5, Theorem 5.2.4], respectively. As a
generalization, combining the nilpotence and the normality of groups together,
Li and Guo [3, Theorem 1.2] proved that if all non-normal maximal subgroups of
a group $G$ are nilpotent then $G$ is solvable and $G$ is $p$-nilpotent for
some prime $p$, that is, $G$ has a normal $p$-complement. It is clear that the
hypothesis that all non-normal maximal subgroups of a group $G$ are nilpotent
is equivalent to the hypothesis that every maximal subgroup of $G$ is
nilpotent or normal. Lu, Pang and Zhong [4, Theorems 1.3 and 3.5] proved that
if every maximal subgroup of a group $G$ is nilpotent or normal then $G$ is
solvable and $G$ is $p$-nilpotent and $q$-closed for some primes $p$ and $q$,
that is, $G$ has a normal $p$-complement and the Sylow $q$-subgroup of $G$ is
normal. In [2, Theorem 1.1] we gave an elementary proof of the solvability of
a group in which every maximal subgroup is nilpotent or normal. Moreover, the
first author of this paper [8, Theorem 5] proved that such a group has a Sylow
tower.

The order of subgroups or groups play an important role in characterizing the
solvability of groups. Feit-Thompson theorem shows that a group having odd
order is solvable. Thompson [5, Theorem 10.4.2] implies that a group having a
nilpotent maximal subgroup of odd order is also solvable. Moreover, it is easy
to see that a group $G$ satisfying $(|G|,15)=1$ is solvable by the
classification of minimal non-abelian simple groups.

Note that the nilpotence, the normality and the order are three distinct
characteristic properties of groups. In this paper, we combine the nilpotence,
the normality and the order of groups together to give a complete
characterization of the structure of the group in which every maximal subgroup
is nilpotent or normal or has $p^{\prime}$-order for a fixed prime divisor $p$
of its order, which extends and generalizes the researches in [3, Theorem
1.2], [4, Theorems 1.3 and 3.5] and [8, Theorem 5].

First we obtain a basic structural property of the non-solvable groups, see
Theorem 1.1, whose proof is given in Section 2.

Theorem 1.1 Let $G$ be a non-solvable group and $p$ a fixed prime divisor of
$|G|$, then $G$ has non-nilpotent maximal subgroups of order divisible by $p$.

Remark 1.2 Note that 2 is a prime divisor of the order of the alternating
group $A_{5}$ but $A_{5}$ has no non-nilpotent maximal subgroups of 2’-order.
This example shows that a non-solvable group might not have non-nilpotent
maximal subgroups of $p^{\prime}$-order for some fixed prime divisor $p$ of
its order.

Our main results are the following Theorems 1.3, 1.4 and 1.7, whose proofs are
given in Sections 3, 4 and 5, respectively.

Theorem 1.3 Let $G$ be a group and $p$ a fixed prime divisor of $|G|$. If
every maximal subgroup of $G$ is nilpotent or normal or has
$p^{\prime}$-order, then $G$ is solvable.

Theorem 1.4 Let $G$ be a group and $p$ a fixed prime divisor of $|G|$. If
every maximal subgroup of $G$ is nilpotent or normal or has
$p^{\prime}$-order, then $G$ has a Sylow tower.

Remark 1.5 Let $G$ be a group and $p$ a fixed prime divisor of $|G|$. If we
assume that every maximal subgroup of $G$ is nilpotent or normal or has order
divisible by $p$, we cannot get that $G$ has a Sylow tower. For example, every
maximal subgroup of the alternating group $A_{5}$ has order divisible by 2
then $A_{5}$ naturally satisfies the hypothesis. But $A_{5}$ is a non-solvable
group which has no Sylow tower.

Remark 1.6 The alternating group $A_{4}$ implies that the Sylow tower of the
group $G$ in Theorem 1.4 might not be supersolvable type.

Theorem 1.7 Let $G$ be a group and $p$ a fixed prime divisor of $|G|$. If
every maximal subgroup of $G$ is nilpotent or normal or has
$p^{\prime}$-order, then there exists at most one prime divisor $q$ of $|G|$
such that $G$ is neither $q$-nilpotent nor $q$-closed, where $q\neq p$.

Comparing with Theorem 1.7, we have the following three results, all of whose
proofs are given in Section 6.

Theorem 1.8 Let $G$ be a group in which every maximal subgroup is nilpotent or
normal, then $G$ is either $q$-nilpotent or $q$-closed for each prime divisor
$q$ of $|G|$.

Theorem 1.9 Let $G$ be a group and $p$ a fixed prime divisor of $|G|$. If
every maximal subgroup of $G$ is nilpotent or has $p^{\prime}$-order, then $G$
is either $q$-nilpotent or $q$-closed for each prime divisor $q$ of $|G|$.

Theorem 1.10 Let $G$ be a group and $p$ a fixed prime divisor of $|G|$. If
every maximal subgroup of $G$ is normal or has $p^{\prime}$-order, then $G$ is
either $q$-nilpotent or $q$-closed for each prime divisor $q$ of $|G|$.

##  2 Proof of Theorem 1.1

Proof. Let $G$ be a counterexample of minimal order. Then every maximal
subgroup of $G$ is nilpotent or has $p^{\prime}$-order. Note that $p$ is a
prime divisor of $|G|$, one has that $G$ must have maximal subgroups of order
divisible by $p$ and such maximal subgroups are nilpotent by the hypothesis.
Since $G$ is non-solvable and both a nilpotent group and a minimal non-
nilpotent group are solvable by [5, Theorem 9.1.9], $G$ must have non-
nilpotent maximal subgroups which have $p^{\prime}$-order.

Let $P\in\hbox{\rm Syl}_{p}(G)$. First suppose that there exists a nontrivial
subgroup $U$ of $P$ such that $U\unlhd G$. Considering the quotient group
$G/U$. (1) Assume $U<P$. Then $G/U$ is a non-solvable group of order divisible
by $p$ since $G$ is non-solvable and $U$ is solvable. By the minimality of
$G$, one has that $G/U$ has a non-nilpotent maximal subgroup $H/U$ of order
divisible by $p$. It follows that $H$ is a non-nilpotent maximal subgroup of
$G$ of order divisible by $p$, a contradiction. (2) Assume $U=P$. Then $U$ is
a Sylow $p$-subgroup of $G$. Let $M$ be a non-nilpotent maximal subgroup of
$G$ of $p^{\prime}$-order. It is clear that $U\nleq M$ and $U\cap M=1$. Then
$G=U\rtimes M$, the semidirect product of $U$ and $M$. For every maximal
subgroup $M_{0}$ of $M$, one has that $U\rtimes M_{0}$ is a maximal subgroup
of $G$ of order divisible by $p$. By the hypothesis, $U\rtimes M_{0}$ must be
nilpotent. It follows that every maximal subgroup of $M$ is nilpotent, which
implies that $M$ is a nilpotent group or a minimal non-nilpotent group. One
has that $M$ is solvable. Then $G=U\rtimes M$ is solvable, this contradicts
that $G$ is non-solvable.

Next suppose that every nontrivial subgroup $U$ of $P$ is not normal in $G$,
that is, $N_{G}(U)<G$. Since the order of $N_{G}(U)$ is divisible by $p$,
$N_{G}(U)$ can only be contained in a nilpotent maximal subgroup of $G$ of
order divisible by $p$. Then $N_{G}(U)$ is nilpotent. By [1, IV, Theorem
5.8(b)], one has that $G$ is $p$-nilpotent. There exists a normal subgroup $K$
of $G$ such that $G=K\rtimes P$. Let $L$ be a maximal subgroup of $G$ of
$p^{\prime}$-order. Assume $K\nleq L$. Then $KL>L$. One has $G=KL$, which
implies that $G$ has $p^{\prime}$-order, a contradiction. Thus $K\leq L$.
Since $K$ is a $p^{\prime}$-Hall subgroup of $G$ and $L$ has
$p^{\prime}$-order, one has that $K=L$ is a maximal subgroup of $G$. It
follows that $P\cong G/K$ is a cyclic group of order $p$. Note that $P<G$ and
$P$ can only be contained in a nilpotent maximal subgroup of $G$. (1) Assume
$p=2$. By [1, IV, Theorem 7.4], one has that $G$ is solvable, a contradiction.
(2) Assume $p>2$. $(i)$ Suppose $Z(G)\neq 1$, that is, the center of $G$ is
not equal to 1. Considering the quotient group $G/Z(G)$. If $p\nmid|G/Z(G)|$,
then $P\leq Z(G)$. It follows that $G=K\times P$. Note that $K\cong G/P$ is
non-solvable, one has that $K$ has a non-nilpotent maximal subgroup $K_{0}$.
Then $K_{0}\times P$ is a non-nilpotent maximal subgroup of $G$ of order
divisible by $p$, a contradiction. If $p\mid|G/Z(G)|$, then $G/Z(G)$ is a non-
solvable group of order divisible by $p$. By the minimality of $G$, $G/Z(G)$
has a non-nilpotent maximal subgroup $R/Z(G)$ of order divisible by $p$. It
follows that $R$ is a non-nilpotent maximal subgroup of $G$ of order divisible
by $p$, a contradiction, too. $(ii)$ Suppose $Z(G)=1$. By [6, Theorem 1],
every nilpotent maximal of $G$ is a Sylow 2-subgroup of $G$. It implies that
$P$ cannot be contained in any nilpotent maximal subgroup of $G$ since $p>2$,
this contradicts that $P$ can only be contained in a nilpotent maximal
subgroup of $G$.

Thus the counterexample of minimal order does not exist and so $G$ has non-
nilpotent maximal subgroups of order divisible by $p$.

##  3 Proof of Theorem 1.3

Proof. Let $G$ be a counterexample of minimal order. Since $G$ is non-
solvable, $G$ has non-nilpotent maximal subgroups of order divisible by $p$ by
Theorem 1.1. Then by the hypothesis, the group $G$ is in particular not
simple.

Let $N$ be a minimal normal subgroup of $G$. We will show that $N$ is non-
solvable. (1) Suppose that $G/N$ has $p^{\prime}$-order. Then $N$ contains the
Sylow $p$-subgroup of $G$. For every non-nilpotent maximal subgroup $L/N$ of
$G/N$, $L$ is a non-nilpotent maximal subgroup of $G$ of order divisible by
$p$. By the hypothesis, $L\unlhd G$. It follows that $G/N$ is a group in which
every maximal subgroup is nilpotent or normal. By [2, Theorem 1.1], $G/N$ is
solvable. Thus $N$ is non-solvable since $G$ is non-solvable. (2) Suppose that
$G/N$ has order divisible by $p$. Since the hypothesis of the theorem holds
for $G/N$ and $|G/N|<|G|$, one has that $G/N$ is solvable by the minimality of
$G$. Thus we also have that $N$ is non-solvable.

It follows that $\it\Phi$$(G)=1$ and $Z(G)=1$, that is, both the Frattini
subgroup of $G$ and the center of $G$ are equal to 1.

Let $R$ be any non-nilpotent maximal subgroup of $G$ of order divisible by
$p$. By the hypothesis, $R\unlhd G$. Assume $N\nleq R$. Then $G=NR$. Note that
$N\cap R\unlhd N$ and $N\cap R\unlhd R$, one has $N\cap R\unlhd NR=G$. It
follows that $N\cap R=1$ since $N\cap R<N$. Then $G=N\times R$, which implies
that $N\cong G/R$ is a cyclic group of prime order. This contradicts that $N$
is non-solvable. Thus $N\leq R$.

Claim that $|N|$ is divisible by $p$. Otherwise, assume that $N$ has
$p^{\prime}$-order. Since $G$ is non-solvable, one has that the intersection
of all non-nilpotent maximal subgroups of $G$ is equal to $\it\Phi$$(G)$ by
[7, Theorem 1]. By above argument, the intersection of all non-nilpotent
maximal subgroups of $G$ is equal to 1. Since every non-nilpotent maximal
subgroup of $G$ of order divisible by $p$ contains $N$, there exists a non-
nilpotent maximal subgroup $E$ of $G$ of $p^{\prime}$-order such that $N\nleq
E$. One has $G=NE$. It follows that $|G|=\frac{|N||E|}{|N\cap E|}$ has
$p^{\prime}$-order, a contradiction. Thus $|N|$ is divisible by $p$.

Since $Z(G)=1$, one has that all nilpotent maximal subgroups of $G$ are Sylow
2-subgroups of $G$ by [6, Theorem 1]. Then the maximal subgroups of $G$ may
only be: Sylow 2-subgroups, non-nilpotent maximal subgroups of order divisible
by $p$ or non-nilpotent maximal subgroups of $p^{\prime}$-order. Assume that
$q$ is a prime divisor of $|N|$ such that $q\neq p$. Let $P\in\hbox{\rm
Syl}_{p}(N)$ and $Q\in\hbox{\rm Syl}_{q}(N)$. By Frattini argument, one has
$G=NN_{G}(P)=NN_{G}(Q)$. Since $N$ is a minimal normal subgroup of $G$ and $N$
is non-solvable, $N_{G}(P)<G$ and $N_{G}(Q)<G$.

It is obvious that both $N_{G}(P)$ and $N_{G}(Q)$ cannot be contained in any
non-nilpotent maximal subgroup of $G$ of order divisible by $p$ since every
non-nilpotent maximal subgroup of $G$ of order divisible by $p$ contains $N$.
(1) Assume $p=2$. Then $q$ is an odd prime. One has that $N_{G}(Q)$ can only
be contained in a non-nilpotent maximal subgroup $H$ of $G$ of
$2^{\prime}$-order. Then $G=NN_{G}(Q)=NH$. And $N_{G}(P)$ can only be
contained in a Sylow 2-subgroup $K$ of $G$. It follows that $G=NN_{G}(P)=NK$.
One has $|G|=|NH|=|NK|$. Then $\frac{|H|}{|H\cap N|}=\frac{|K|}{|K\cap N|}$.
Note that $\frac{|H|}{|H\cap N|}>1$ is a $2^{\prime}$-number, but
$\frac{|K|}{|K\cap N|}>1$ is a $2$-power, a contradiction. (2) Assume $p>2$.
One has that $N_{G}(P)$ cannot be contained in any Sylow 2-subgroup of $G$ and
$N_{G}(P)$ cannot be contained in any non-nilpotent maximal subgroup of $G$ of
$p^{\prime}$-order, either, a contradiction.

Hence the counterexample of minimal order does not exist, then $G$ is
solvable.

##  4 Proof of Theorem 1.4

Proof. First we show that $G$ has a normal Sylow subgroup. Let $G$ be a
counterexample of minimal order.

Claim $\it\Phi$$(G)=1$. Otherwise, assume $\it\Phi$$(G)\neq 1$. Since
$|G/\it\Phi$$(G)|$ and $|G|$ have the same prime divisors, $G/\it\Phi$$(G)$ is
also a group of order divisible by $p$ in which every maximal subgroup is
nilpotent or normal or has $p^{\prime}$-order. By the minimality of $G$, one
has that $G/\it\Phi$$(G)$ has a normal Sylow subgroup
$P\it\Phi$$(G)/\it\Phi$$(G)$, where $P$ is a Sylow subgroup of $G$. Using
Frattini argument, one gets that $G$ has a normal Sylow subgroup $P$, a
contradiction. Thus $\it\Phi$$(G)=1$.

Note that $G$ is solvable by Theorem 1.3. Let $N$ be a minimal normal subgroup
of $G$, then $N$ is an elementary abelian group of prime power order. One has
that there exists a maximal subgroup $M$ of $G$ such that $N\nleq M$ since
$\it\Phi$$(G)=1$. It follows that $G=NM$.

First consider the case when $M$ is a non-nilpotent maximal subgroup of $G$ of
order divisible by $p$. By the hypothesis, one has $M\unlhd G$. Note that
$N\cap M$ is normal in $M$ and $N\cap M$ is also normal in $N$ as $N$ is
abelian. Then $N\cap M$ is normal in $NM=G$. It follows that $N\cap M=1$ since
$N$ is a minimal normal subgroup of $G$ and $N\cap M<N$. One has $G=N\times
M$, which implies that $M\cong G/N$. (1) Suppose that $G/N$ has
$p^{\prime}$-order. Then $N$ is a normal Sylow $p$-subgroup of $G$, a
contradiction. (2) Suppose that $G/N$ has order divisible by $p$. One has that
$G/N$ has a normal Sylow subgroup since the hypothesis of the theorem holds
for $G/N$ and $|G/N|<|G|$. It follows that $M$ has a normal Sylow subgroup $Q$
since $M\cong G/N$. $(i)$ Assume $(|N|,|Q|)\neq 1$. Then $N\times Q$ is a
normal Sylow subgroup of $G$, a contradiction. $(ii)$ Assume $(|N|,|Q|)=1$.
Then $Q$ is a normal Sylow subgroup of $G$, a contradiction, too.

Next consider the case when $M$ is a non-nilpotent maximal subgroup of $G$ of
$p^{\prime}$-order or a nilpotent maximal subgroup of $G$ of
$p^{\prime}$-order. Note that $G=NM$ is a group of order divisible by $p$ and
$N$ is an elementary abelian group of prime power order. It follows that $N$
is a Sylow $p$-subgroup of $G$, which is also a normal Sylow subgroup of $G$,
a contradiction.

Finally consider the case when $M$ is a nilpotent maximal subgroup of $G$ of
order divisible by $p$. Assume $|N|=q^{m}$ for some prime $q$ and some
positive integer $m\geq 1$. Let $Q\in\hbox{\rm Syl}_{q}(M)$. Then
$NP\in\hbox{\rm Syl}_{q}(G)$. One has $NP\unlhd NM=G$, a contradiction.

By above arguments, the counterexample of minimal order does not exist and so
$G$ has a normal Sylow subgroup.

In the following we prove that $G$ has a Sylow tower.

Let $P_{1}$ be a normal Sylow $p_{1}$-subgroup of $G$. (1) Suppose $p_{1}\neq
p$. Observe that $G/P_{1}$ is also a group of order divisible by $p$ in which
every maximal subgroup is nilpotent or normal or has $p^{\prime}$-order,
arguing as above, one gets that $G/P_{1}$ has a normal Sylow $p_{2}$-subgroup
$P_{1}P_{2}/P_{1}$, where $P_{2}\in\hbox{\rm Syl}_{p_{2}}(G)$. (2) Suppose
$p_{1}=p$. Let $M/P_{1}$ be a non-nilpotent maximal subgroup of $G/P_{1}$,
where $M$ is a non-nilpotent maximal subgroup of $G$ of order divisible by
$p$. By the hypothesis, one has $M\unlhd G$, then $G/P_{1}$ is a group in
which every maximal subgroup is nilpotent or normal. One can also get that
$G/P_{1}$ has a normal Sylow $p_{2}$-subgroup $P_{1}P_{2}/P_{1}$ by [8,
Theorem 5], where $P_{2}\in\hbox{\rm Syl}_{p_{2}}(G)$.

Similarly, considering the quotient group $G/{P_{1}P_{2}}$, arguing as above
we can get that $G/{P_{1}P_{2}}$ has a normal Sylow $p_{3}$-subgroup
$P_{1}P_{2}P_{3}/{P_{1}P_{2}}$, where $P_{3}\in\hbox{\rm Syl}_{p_{3}}(G)$. And
so on, we can obtain a normal subgroups series:

$P_{1}\unlhd{P_{1}P_{2}}\unlhd{P_{1}P_{2}P_{3}}\unlhd\cdots\unlhd{P_{1}P_{2}\cdots
P_{s}}=G$ (1)

where $P_{i}\in\hbox{\rm Syl}_{p_{i}}(G)$ for $1\leq i\leq s$, which implies
that $G$ has a Sylow tower.

##  5 Proof of Theorem 1.7

Proof. Assume that $G$ is nilpotent, then $G$ is $q$-nilpotent and $q$-closed
for each prime divisor $q$ of $|G|$. In the following we assume that $G$ is
non-nilpotent. Since $G$ has Sylow tower by Theorem 1.4, $G$ can be written as
$G=(P_{1}\times P_{2}\times\cdots\times P_{s})\rtimes(Q_{1}Q_{2}\cdots
Q_{t})$, where $P_{i}\in\hbox{\rm Syl}_{p_{i}}(G)$ and $P_{i}\unlhd G$ for
$1\leq i\leq s$, $Q_{j}\in\hbox{\rm Syl}_{q_{j}}(G)$ and $Q_{j}$ is not normal
in $G$ for $1\leq j\leq t$. Let $M=P_{1}\times P_{2}\times\cdots\times P_{s}$
and $N=Q_{1}Q_{2}\cdots Q_{t}$. Then $G=M\rtimes N$.

Considering the case when the Sylow $p$-subgroup of $G$ is normal in $G$.
Assume $P_{1}\in\hbox{\rm Syl}_{p}(G)$, that is, $p_{1}=p$.

(1) Suppose that $N$ is nilpotent. Then $G$ is obviously $p_{i}$-closed for
every $1\leq i\leq s$ and $q_{j}$-nilpotent for every $1\leq j\leq t$.

(2) Suppose that $N$ is non-nilpotent. Then there exists a maximal subgroup
$N_{0}$ of $N$ such that $N_{0}$ is not normal in $N$. For the maximal
subgroup $MN_{0}$ of $G$ of order divisible by $p$, $MN_{0}$ is not normal in
$G$. By the hypothesis, one has that $MN_{0}$ is nilpotent. Note that
$|N:N_{0}|$ is a prime power by the solvability of $G$. We can assume
$Q_{1}\nleq N_{0}$ and $Q_{j}\leq N_{0}$ for every $2\leq j\leq t$. Then
$MN_{0}=P_{1}\times P_{2}\times\cdots\times P_{s}\times Q_{1_{0}}\times
Q_{2}\times\cdots\times Q_{t}$, where $Q_{1_{0}}<Q_{1}$. For any non-nilpotent
maximal subgroup $H$ of $N$, one has that $MH$ is a non-nilpotent maximal
subgroup of $G$ of order divisible by $p$. By the hypothesis, $MH\unlhd G$. It
follows that $H\unlhd N$. That is every maximal subgroup of $N$ is nilpotent
or normal, one has that $N$ has normal Sylow subgroups by [8, Theorem 5].
Assume $Q_{j}\unlhd N$ for some $2\leq j\leq t$, then $N\leq N_{G}(Q_{j})$. It
follows that $G=N_{G}(Q_{j})$ since $M\leq N_{G}(Q_{j})$, this contradicts
that $Q_{j}$ is not normal in $G$ for every $2\leq j\leq t$. Thus $Q_{j}$ is
not normal in $N$ for every $2\leq j\leq t$. One has $Q_{1}\unlhd N$. Then
$G=(P_{1}\times P_{2}\times\cdots\times P_{s})\rtimes(Q_{1}\rtimes(Q_{2}\times
Q_{3}\times\cdots\times Q_{t}))$. It is easy to see that $G$ is $p_{i}$-closed
for every $1\leq i\leq s$, and $G$ is $q_{j}$-nilpotent for every $2\leq j\leq
t$. For $q_{1}$, $G$ is neither $q_{1}$-nilpotent nor $q_{1}$-closed.

Next consider the case when the Sylow $p$-subgroup of $G$ is not normal in
$G$. Assume $Q_{1}\in\hbox{\rm Syl}_{p}(G)$, that is, $q_{1}=p$.

Here $G/M\cong N$ is a group of order divisible by $p$. Claim that $G/M$ is
nilpotent. Otherwise, assume that $G/M$ is non-nilpotent. Since $G/M$ is also
a group of order divisible by $p$ in which every maximal subgroup is nilpotent
or normal or has $p^{\prime}$-order, $G/M$ has a normal Sylow subgroup by
Theorem 1.4, which implies that $N$ has a normal Sylow subgroup. Assume
$Q_{j}\unlhd N$ for some $1\leq j\leq t$. Then $N\leq N_{G}(Q_{j})$. It
follows that $N_{G}(Q_{j})$ has order divisible by $p$. Since $Q_{j}$ is not
normal in $G$ and all non-nilpotent maximal subgroups of $G$ of order
divisible by $p$ are normal, $N_{G}(Q_{j})$ can only be contained in some
nilpotent maximal subgroup of $G$ of order divisible by $p$. It implies that
$N$ is nilpotent, this contradicts that $N\cong G/M$ is non-nilpotent. Hence
$N\cong G/M$ is nilpotent. Then $G$ is $p_{i}$-closed for every $1\leq i\leq
s$ and $q_{j}$-nilpotent for every $1\leq j\leq t$.

##  6 Proofs of Theorems 1.8, 1.9 and 1.10

Proof of Theorem 1.8. Let $G$ be a group in which every maximal subgroup is
nilpotent or normal. Suppose that $G$ is nilpotent, then $G$ is $q$-nilpotent
and $q$-closed for each prime divisor $q$ of $|G|$.

Next we suppose that $G$ is non-nilpotent. Note that $G$ has Sylow tower by
[8, Theorem 5], one has $G=(P_{1}\times P_{2}\times\cdots\times
P_{s})\rtimes(Q_{1}Q_{2}\cdots Q_{t})$, where $P_{i}\in\hbox{\rm
Syl}_{p_{i}}(G)$ and $P_{i}\unlhd G$ for $1\leq i\leq s$, $Q_{j}\in\hbox{\rm
Syl}_{q_{j}}(G)$ and $Q_{j}$ is not normal in $G$ for $1\leq j\leq t$. Let
$K=P_{1}\times P_{2}\times\cdots\times P_{s}$ and $L=Q_{1}Q_{2}\cdots Q_{t}$.
Then $G=K\rtimes L$.

Claim that $L$ is nilpotent. Otherwise, assume that $L$ is non-nilpotent. For
every non-nilpotent maximal subgroup $L_{0}$ of $L$, one has that $KL_{0}$ is
a non-nilpotent maximal subgroup of $G$. By the hypothesis, one has
$KL_{0}\unlhd G$. It follows that $L_{0}\unlhd L$. That is, $L$ is a group in
which every maximal subgroup is nilpotent or normal. By [8, Theorem 5], $L$
has a normal Sylow subgroup. Assume $Q_{1}\unlhd L$. Then $L\leq
N_{G}(Q_{1})<G$. Since $L$ is non-nilpotent, $N_{G}(Q_{1})$ can only be
contained in a non-nilpotent maximal subgroup $H$ of $G$. By the hypothesis,
$H\unlhd G$. Then by Frattini argument, one has $Q_{1}\unlhd G$, a
contradiction. Thus $L$ is nilpotent. It follows that $G$ is $p_{i}$-closed
for every $1\leq i\leq s$ and $q_{j}$-nilpotent for every $1\leq j\leq t$.

Proof of Theorem 1.9. If $G$ is nilpotent, it is obvious that $G$ is
$q$-nilpotent and $q$-closed for each prime divisor $q$ of $|G|$. In the
following assume that $G$ is non-nilpotent. Arguing as in proof of Theorem
1.7, one has $G=(P_{1}\times P_{2}\times\cdots\times
P_{s})\rtimes(Q_{1}Q_{2}\cdots Q_{t})$, where $P_{i}\in\hbox{\rm
Syl}_{p_{i}}(G)$ and $P_{i}\unlhd G$ for $1\leq i\leq s$, $Q_{j}\in\hbox{\rm
Syl}_{q_{j}}(G)$ and $Q_{j}$ is not normal in $G$ for $1\leq j\leq t$. Assume
$M=P_{1}\times P_{2}\times\cdots\times P_{s}$ and $N=Q_{1}Q_{2}\cdots Q_{t}$.
That is, $G=M\rtimes N$.

We will show that $N$ is nilpotent.

First assume $p_{1}=p$, that is, the Sylow $p$-subgroup of $G$ is normal and
$P_{1}\in\hbox{\rm Syl}_{p}(G)$. Let $N_{0}$ be any maximal subgroup of $N$.
Then $M\rtimes N_{0}$ is a maximal subgroup of $G$ of order divisible by $p$.
By the hypothesis, $M\rtimes N_{0}$ is nilpotent. It follows that $N_{0}$ is
nilpotent. Then $N$ is nilpotent or minimal non-nilpotent. Claim that $N$
cannot be minimal non-nilpotent. Otherwise, assume that $N$ is a minimal non-
nilpotent group. We can assume $N=Q_{1}\rtimes Q_{2}$ by [5, Theorem 9.1.9],
where $Q_{1}\unlhd N$. Let $Q_{2}^{\prime}$ be any maximal subgroup of
$Q_{2}$. Then $M\rtimes(Q_{1}\rtimes Q_{2}^{\prime})$ is a maximal subgroup of
$G$ of order divisible by $p$. By the hypothesis, $M\rtimes(Q_{1}\rtimes
Q_{2}^{\prime})$ is nilpotent. It follows that $M\leq N_{G}(Q_{1})$. Note that
$N\leq N_{G}(Q_{1})$. Then $N_{G}(Q_{1})=MN=G$, which implies that
$Q_{1}\unlhd G$, a contradiction. Thus $N$ is nilpotent.

Second assume $q_{1}=p$, that is, the Sylow $p$-subgroup of $G$ is not normal
and $Q_{1}\in\hbox{\rm Syl}_{p}(G)$. It is obvious that $N<G$ and $N$ has
order divisible by $p$. Then there exists a maximal subgroup $L$ of $G$ of
order divisible by $p$ such that $N\leq L$. By the hypothesis, $L$ is
nilpotent. It follows that $N$ is also nilpotent.

By the nilpotence of $N$, one has that $G$ is $p_{i}$-closed for every $1\leq
i\leq s$ and $q_{j}$-nilpotent for every $1\leq j\leq t$.

Proof of Theorem 1.10. It is clear that the result holds if $G$ is nilpotent.
Considering the case that $G$ is non-nilpotent. Arguing as above, let
$G=(P_{1}\times P_{2}\times\cdots\times P_{s})\rtimes(Q_{1}Q_{2}\cdots
Q_{t})$, where $P_{i}\in\hbox{\rm Syl}_{p_{i}}(G)$ and $P_{i}\unlhd G$ for
$1\leq i\leq s$, $Q_{j}\in\hbox{\rm Syl}_{q_{j}}(G)$ and $Q_{j}$ is not normal
in $G$ for $1\leq j\leq t$. Let $M=P_{1}\times P_{2}\times\cdots\times P_{s}$
and $N=Q_{1}Q_{2}\cdots Q_{t}$. One has $G=M\rtimes N$.

Claim that the Sylow $p$-subgroup of $G$ is normal. Otherwise, assume that the
Sylow $p$-subgroup of $G$ is not notmal. Let $Q_{1}\in\hbox{\rm Syl}_{p}(G)$.
Note that $N_{G}(Q_{1})<G$ and $N_{G}(Q_{1})$ has order divisible by $p$. Then
there exists a maximal subgroup $R$ of $G$ of order divisible by $p$ such that
$N_{G}(Q_{1})\leq R$. By the hypothesis, $R\unlhd G$. It follows that
$Q_{1}\unlhd G$ by Frattini argument, a contradiction.

Thus the Sylow $p$-subgroup of $G$ is normal. Let $P_{1}\in\hbox{\rm
Syl}_{p}(G)$. For every maximal subgroup $N_{0}$ of $N$, $MN_{0}$ is a maximal
subgroup of $G$ of order divisible by $p$. By the hypothesis, $MN_{0}\unlhd
G$. It follows that $N_{0}\unlhd N$ and then $N$ is nilpotent. Thus $G$ is
$p_{i}$-closed for every $1\leq i\leq s$ and $q_{j}$-nilpotent for every
$1\leq j\leq t$.

Acknowledgements

The authors are thankful to everyone who provides valuable suggestions and
helpful comments for improving our paper.

## References

  * [1] B. Huppert, Endliche Gruppen I, Springer-Verlag, Berlin-Heidelberg, 1967.
  * [2] Na Li and Jiangtao Shi, A note on a finite group with all non-nilpotent maximal subgroups being normal, Italian Journal of Pure and Applied Mathematics 42 (2019) 700-702.
  * [3] Qianlu Li and Xiuyun Guo, On $p$-nilpotence and solubility of groups, Archiv der Mathematik 96(1) (2011) 1-7.
  * [4] Jiakuan Lu, Linna Pang and Xianggui Zhong, Finite groups with non-nilpotent maximal subgroups, Monatshefte f$\rm\ddot{u}$r Mathematik 171 (2013) 425-431.
  * [5] D.J.S. Robinson, A Course in the Theory of Groups (Second Edition), Springer-Verlag, New York, 1996.
  * [6] J.S. Rose, On finite insoluble groups with nilpotent maximal subgroups, Journal of Algebra 48(1) (1977) 182-196.
  * [7] Jiangtao Shi, Cui Zhang and Songtao Guo, A note on theorem of Shlyk (Chinese), Journal of Guangxi Normal University (Natural Science Edition) 30(1) (2012) 22-24.
  * [8] Jiangtao Shi, A finite group in which all non-nilpotent maximal subgroups are normal has a Sylow tower, Hokkaido Mathematical Journal 48(2) (2019) 309-312.


# In-Context Learning Creates Task Vectors

Roee Hendel
Tel Aviv University
<EMAIL_ADDRESS>
&Mor Geva
Google DeepMind
<EMAIL_ADDRESS>
&Amir Globerson
Tel Aviv University, Google
<EMAIL_ADDRESS>

###### Abstract

In-context learning (ICL) in Large Language Models (LLMs) has emerged as a
powerful new learning paradigm. However, its underlying mechanism is still not
well understood. In particular, it is challenging to map it to the “standard”
machine learning framework, where one uses a training set $S$ to find a best-
fitting function $f(x)$ in some hypothesis class. Here we make progress on
this problem by showing that the functions learned by ICL often have a very
simple structure: they correspond to the transformer LLM whose only inputs are
the query $x$ and a single “task vector” calculated from the training set.
Thus, ICL can be seen as compressing $S$ into a single task vector
$\boldsymbol{\theta}(S)$ and then using this task vector to modulate the
transformer to produce the output. We support the above claim via
comprehensive experiments across a range of models and tasks.111We release our
code at https://github.com/roeehendel/icl_task_vectors.

Figure 1: ICL as learning in a Hypothesis Class. In ICL, one provides an LLM
with a prompt including demonstrations $S$ of some task, and a query $x$. The
model generates the output for $x$ (here “Yellow”). We show that the
underlying process can be broken down into two parts: $\mathcal{A}$, a
“learning algorithm” (marked in blue), computes a query-agnostic vector
$\boldsymbol{\theta}(S)$, which we view as a parameter of a function in a
hypothesis class. The second part, denoted by $f$ and marked in yellow, is the
application of the rule defined by $\boldsymbol{\theta}$ on the query $x$,
without direct dependence on $S$.

##  1 Introduction

Large language models have improved dramatically over the last several years.
One striking property of these models is that they can learn new rules from
very few demonstrations. For instance, a model can be prompted with the input
“Apple $\rightarrow$ Red, Lime $\rightarrow$ Green, Corn $\rightarrow$” and
produce the output “Yellow”. The model has thus learned a mapping based on
just two examples, which it can apply correctly to new examples. This
capability, referred to as In-Context Learning (ICL), has been used
extensively, yielding impressive empirical results Brown et al. (2020); Liu et
al. (2023); Dong et al. (2022).

Given this success, it is natural to ask what is the underlying mechanism
behind ICL. Namely, how does the model internally use the demonstrations $S$
and the query $x$ to produce the required output? Here we approach this
question by utilizing the concept of a hypothesis class from statistical
learning theory Shalev-Shwartz and Ben-David (2014). In the learning-theoretic
formulation, one typically considers a hypothesis class $\mathcal{H}$, where
every element of $\mathcal{H}$ is a function $h(x;\boldsymbol{\theta})$,
operating on the input $x$, and specified by a parameter vector
$\boldsymbol{\theta}$. For example, if $x\in\mathbb{R}^{d}$ then the class
$\mathcal{H}$ could be the set of linear classifiers, defined by a coefficient
vector $\boldsymbol{\theta}$ as
$h(x;\boldsymbol{\theta})=\boldsymbol{\theta}\cdot x$. Learning algorithms
seek an element $h\in\mathcal{H}$ that fits the training set well. This is
known as Empirical Risk Minimization.

It is unclear whether ICL operates in such a way because the prediction is
performed via $T([S,x])$, where $T$ is typically an auto-regressive
transformer and $[S,x]$ is a concatenation of the tokens in $S$ and $x$. Thus,
in the general case, it can be an arbitrary function that operates on $S$ and
$x$ to produce the output. This can include “non-parametric” methods such as
nearest-neighbor. Recent work has begun to explore this question. For example,
it was shown that when training a transformer from scratch to perform linear
regression in context, the emerging learning algorithm is similar to
Stochastic Gradient Descent Akyürek et al. (2022); von Oswald et al. (2022).
However, for LLMs performing more complex natural language tasks, it is not at
all clear what the hypothesis space may be.

In this work, we show that on a wide range of tasks, ICL in LLMs can be viewed
as working on a very natural hypothesis space. We argue that, given a training
set $S$, the transformer maps it into a “task vector” $\boldsymbol{\theta}(S)$
that essentially represents the mapping/rule described in $S$.222The term
“task vector” was coined by Ilharco et al. (2023) for directions in weight
space that correspond to a particular task. Although our vectors are in
“activations space” they share a similar motivation and thus we overload the
term. Namely, given the transformer $T$ and a vector $\boldsymbol{\theta}$, we
can construct a new function $f(x;\boldsymbol{\theta})$ that implements the
task. The function $f$ is very similar to the original transformer applied to
$x$ without demonstrations but instead modulated by $\boldsymbol{\theta}$ (see
Fig. 2).

Our view is also related to soft prompts Lester et al. (2021), since both
approaches modulate the function of the transformer towards a particular task.
However, in ICL, task vectors are calculated in the forward pass rather than
being fine-tuned.

Our contributions include proposing a hypothesis-class based mechanistic view
of ICL, and conducting experiments to validate our view on a range of publicly
available LLMs and a diverse set of tasks. Our results further the
understanding of ICL and may have practical implications for the efficient
adaptation of LLMs to perform specific tasks.

##  2 A Hypothesis Class View of ICL

Motivated by the hypothesis class view of learning theory, our goal is to
understand if ICL maps the set of demonstrations $S$ to a function on the
query $x$ and how this mapping occurs. Specifically, we seek to see if ICL
converts $S$ into $\boldsymbol{\theta}$ \- the “parameters” of a function
within a certain hypothesis space. Our empirical findings suggest this view is
applicable, shedding light on the structure of the hypothesis space on which
ICL can be viewed to operate.

###  2.1 Theoretical Framework

We use $T$ to denote a decoder-only transformer LLM, $S$ to denote the set of
demonstrations (i.e. training examples) used as input to ICL, and $x$ to
denote the query that ICL is asked to provide an output for. We use $T([S,x])$
to denote the output of ICL on the concatenation of $S$ and $x$.

To demonstrate that ICL operates within a hypothesis space, we aim to show
that its underlying mechanism can be broken down into two parts:

  * •

A “Learning Algorithm” (denoted by $\mathcal{A}$) that maps $S$ into a “task
vector” $\boldsymbol{\theta}$, independent of the query $x$. Given that
attention layers can access both $S$ and $x$, this independence is not
trivial.

  * •

A “Rule Application” (denoted by $f$) which maps the query $x$ to the output,
based on $\boldsymbol{\theta}\equiv\mathcal{A}(S)$, without direct dependence
on $S$. Again, this independence is not trivial.

Thus, we consider the following mapping from a set of demonstrations and a
query to the predicted output: $T([S,x])=f(x;\mathcal{A}(S))$.

If we can break down the forward pass of the LLM into the above two
components, we can view ICL as operating on the following hypothesis class:
$\mathcal{H}=\\{f(\cdot;\boldsymbol{\theta})\mid\boldsymbol{\theta}\\}$. In
the next section we propose an implementation of such a class.

###  2.2 A Proposed Hypothesis Class

There are many possible realizations of the above framework, that correspond
to different choices of $\mathcal{A}$ and $f$. We next describe the
realization we focus on, which naturally follows from the transformer
architecture. We consider an ICL setting as in Fig. 1, where the input ends
with a query $x$ (i.e., Corn) followed by an “$\rightarrow$” symbol. As
mentioned above, we view learning as composed of two steps: calculating a
parameter vector $\boldsymbol{\theta}$ based on the training sample $S$, and
applying the rule defined by this parameter vector to the query $x$. A
presumably simple way for a transformer to do this is for the first $L$ layers
of the $\rightarrow$ representations to calculate $\boldsymbol{\theta}$ and
then for the remaining layers to take $\boldsymbol{\theta}$ and $x$ as input
and produce an output. See Fig. 1. Recall that $S$ and $x$ are accessible to
the transformer at any layer, presenting a challenge with our view.

In the following sections, we address this challenge and present experiments
validating our view. Namely, we show that we can isolate our proposed
$\mathcal{A}$ and $f$ in the forward pass of LLMs performing ICL. We also show
that the $\boldsymbol{\theta}$ vectors are interpretable and correspond to
learned tasks.

Figure 2: Separating $\mathcal{A}$ and $f$. To make $\boldsymbol{\theta}$
independent of the query $x$, we use a dummy query ($x^{\prime}=\mbox{Plum}$)
and use the representation of $\rightarrow$ at the $L^{th}$ layer as
$\boldsymbol{\theta}$. The vector $\boldsymbol{\theta}$ is then patched at the
same layer during a forward pass of a transformer that only takes $x$ and
$\rightarrow$ as input, to prevent the direct dependence of $f$ on $S$.

##  3 Validity of the Hypothesis Class View

We first show that separating the forward pass into the two distinct
components $\mathcal{A}$ and $f$, defined in §2.2, maintains the high accuracy
of ICL.

###  3.1 Separating $\mathcal{A}$ and $f$

We face some challenges in a regular forward pass: first, the initial $L$
layers that correspond to $\mathcal{A}$, updating the representations of
$\rightarrow$ to create $\boldsymbol{\theta}$, can attend to the query $x$.
Thus, they may depend on $x$, creating an unwanted dependence of
$\boldsymbol{\theta}$ on $x$. Second, the remaining layers that correspond to
$f$, may directly access $S$, instead of using only $x$ and
$\boldsymbol{\theta}$.

We propose the following procedure to tackle these challenges: to solve the
first problem, we introduce a “dummy query” $x^{\prime}$ and calculate the
representations of $\rightarrow$ using that query. We use the representation
of $\rightarrow$ after the first $L$ layers, calculated using $x^{\prime}$, as
the vector $\boldsymbol{\theta}$ (as demonstrated on the left side of Fig. 2).
An alternative was to block attention to $x$, but it led to poor performance.
To solve the second problem of calculating $f(x,\boldsymbol{\theta})$ without
allowing direct dependence on $S$, we perform a forward pass of the
transformer only on $x$ and $\rightarrow$,333Ignoring positional embeddings,
this is equivalent to blocking the attention to $S$ in these layers. and
“patch” the $\boldsymbol{\theta}$ we previously extracted at the $L$th layer
of the $\rightarrow$ (right side of Fig. 2).444Note that the second token can
actually be anything, because it is overridden by patching. We use
$\rightarrow$ for simplicity.

Category | Task | Example
---|---|---
Algorithmic | Next letter | a $\rightarrow$ b
List first | a,b,c $\rightarrow$ a
List last | a,b,c $\rightarrow$ c
To uppercase | a $\rightarrow$ A
Translation | French to English | bonjour $\rightarrow$ hello
Spanish to English | hola $\rightarrow$ hello
Linguistic | Present to gerund | go $\rightarrow$ going
Singular to plural | cat $\rightarrow$ cats
Antonyms | happy $\rightarrow$ sad
Knowledge | Country to Capital | France $\rightarrow$ Paris
Person to Language | Macron $\rightarrow$ French
Table 1: A representative subset of the tasks used in the study with input
$\rightarrow$ output examples. Figure 3: Accuracy for each choice of the
intermediate layer $L$, averaged across all tasks. Solid lines show average
values, and shaded areas standard deviations.

###  3.2 Tasks and Models

#### Tasks

We consider a diverse set of 18 tasks across 4 categories: algorithmic,
translation, linguistic, and factual knowledge. For simplicity, we limit
ourselves to single-token outputs. A representative subset of the tasks is
described in Tab. 1. A complete detailed table, as well as more information
regarding the data, are provided in § A.1.

#### Models

We use multiple open LLMs: LLaMA 7B, 13B, and 30B Touvron et al. (2023), GPT-J
6B Wang and Komatsuzaki (2021), and Pythia 2.8B, 6.9B, and 12B Biderman et al.
(2023).

###  3.3 Finding $L$

The mechanism we described in §2.2 has a free parameter - the layer $L$ where
$\mathcal{A}$ ends and $f$ begins. We use the proposed $(\mathcal{A},f)$
implementation for different choices of $L$ and evaluate the accuracy on a
development set to find the best layer.

Fig. 3 shows the accuracy on the development set, for different choices of
$L$. We focus here on the LLaMA models and include the rest in § A.2.
Interestingly, all models exhibit a performance peak at a similar intermediate
layer, irrespective of their parameters and layer count differences.

Figure 4: Average accuracy across all tasks for each model, using each of the
three procedures: Baseline, Regular and Hypothesis.

###  3.4 Accuracy of Hypothesis Based Prediction

We next compare the accuracy of the $(\mathcal{A},f)$ mechanism to that of a
regular forward pass performing ICL. For each model and task, we evaluate the
following three procedures:

  * •

Regular An application of the LLM to the demonstrations $S$ and query $x$.
Namely $T([S,x])$, as in regular ICL.

  * •

Hypothesis Our proposed procedure from § 3.1 where $\mathcal{A}$ generates
$\boldsymbol{\theta}$ using a dummy $x^{\prime}$, and
$f(\cdot;\boldsymbol{\theta})$ is applied to $x$ by running the transformer on
$[x,\rightarrow]$ with $\boldsymbol{\theta}$ patched at layer $L$ of
$\rightarrow$.

  * •

Baseline A forward pass of the LLM only on $x$, without demonstrations $S$.
That is, $T([x,\rightarrow])$. This is the same as the application of $f$ from
our separated procedure, but without patching $\boldsymbol{\theta}$.

Fig. 4 shows the average accuracy across all tasks of these 3 procedures, for
each model. Full results are reported in Tab. LABEL:table:main_results in §
A.2. Across all models, our procedure maintains around 80-90% of the accuracy
of regular ICL, while the baseline reaches only 10-20%. This shows that our
proposed separation to $\mathcal{A}$ and $f$ provides a good empirical
approximation of the process underlying ICL.

Figure 5: A t-SNE plot of task vectors. A 2D t-SNE plot visualizing 50 task
vectors for each task, each generated from a different choice of $S$ and
$x^{\prime}$ using LLaMA 7B. Points are color-coded according to the task.
Each task can be seen to form its own distinct cluster.

##  4 Robustness of Task Vectors

In our setting, $\boldsymbol{\theta}$ is derived from $S$ and a dummy query
$x^{\prime}$. It is natural to examine the robustness of $\boldsymbol{\theta}$
to variations in these inputs. Intuitively, if it represents the task, it
should remain stable across different $S$ and $x^{\prime}$ values.

To test this, we use LLaMA 7B to generate 50 task vectors per task with varied
$S$ and $x^{\prime}$ and conduct two analyses.

#### Geometry of $\boldsymbol{\theta}$

A t-SNE dimensionality reduction (Fig. 5) reveals that the task vectors form
distinct clusters, each containing task vectors of a single task. Fig. 9
further shows proximity between tasks of the same category, strengthening the
idea that they encapsulate task understanding.

#### Variability of $\boldsymbol{\theta}$

Fig. 8 shows histograms of distances within and across tasks. It can be seen
that vectors within the same task are closer than those between different
tasks, indicating that $\boldsymbol{\theta}$ is stable within tasks and not
highly influenced by $x^{\prime}$ or $S$.

##  5 Dominance of $\boldsymbol{\theta}$ Patching

In §3 we prevented $f$ from directly accessing $S$. However, in a regular
forward pass during ICL, the last token can attend to $S$. Here we verify that
even in this case, $f$ mainly uses the task vector $\boldsymbol{\theta}$,
without directly accessing the demonstrations $S$. To this end, we use a pair
of tasks, $A$ and $B$, sharing the input space but differing on the output. We
first use a “Regular” forward pass, where we provide the model with
demonstrations $S$ for task $A$ (denoted $S_{A}$), to verify the model can
perform this task using ICL. Then, we do a “Conflicting” forward pass, still
providing $S_{A}$, while injecting $\boldsymbol{\theta}_{B}$. For more
details, refer to Fig. 6 in §A.1.

In Tab.2, the “Regular” forward pass shows high accuracy on task $A$ (90%+),
as anticipated. However, the “Conflicting” forward pass yields high accuracy
on task $B$, corresponding to the injected task vector $\boldsymbol{\theta}$.
This implies that the model mainly relies on $\boldsymbol{\theta}$, largely
disregarding the demonstrations $S$ for task $A$. We note that the accuracy on
task $B$ is slightly low, likely consistent with the performance dip seen in
Fig. LABEL:table:main_results, and potentially further affected by the
presence of $S$.

Task $A$ ($S$) | Task $B$ ($\boldsymbol{\theta}$) | Regular | Conflicting
---|---|---|---
|  | Task $A$ | Task $B$
Next Letter | To Upper | 0.92 | 0.77
List Last | List First | 0.95 | 0.78
Present to Past | to Gerund | 0.96 | 0.95
Table 2: Conflicting tasks experiment results. The model’s accuracy on the
relevant task ($A$ in “Regular” and $B$ in “Conflicting”) is displayed for
both scenarios.

##  6 Interpreting $\boldsymbol{\theta}$

The learned vector $\boldsymbol{\theta}$ intuitively captures information
about the task demonstrated by $S$. Here we provide evidence supporting this
interpretation. Since $\boldsymbol{\theta}$ is an intermediate hidden state of
the transformer, we can employ a vocabulary projection method nostalgebraist
(2020); Dar et al. (2022). Namely, we examine the top tokens in the
distribution over the vocabulary induced by the hidden state.

Tab. 3 shows the top tokens for three tasks for LLaMA 13B (more models and
tasks are provided in Tab. 7 in §A). In multiple cases, we observe tokens that
directly describe the task. Importantly, these terms never explicitly appeared
in the context. For example in the task of translation from French to English,
we observe tokens such as “English” and “translate”. This supports our view
that $\boldsymbol{\theta}$ carries significant, non-trivial semantic
information about the task.

##  7 Related Work

#### Emergence of ICL

A key question with ICL is how it emerges as a capability from pre-training
the LLMs. Levine et al. (2022) provides results in this direction that
highlight the importance of training data structure. Xie et al.  use
probabilistic analysis and model pre-training data using Hidden Markov Models
to theoretically explain the emergence of ICL, while Chan et al. (2022)
empirically explore the effect of several distributional properties of the
pre-training data.

#### Meta-Learning in Transformers

Studies by Akyürek et al. (2022); von Oswald et al. (2022); Garg et al.  focus
on the meta-learning capabilities of transformers. They typically train models
from scratch on elementary tasks such as linear regression, drawing
theoretical parallels with algorithms like Gradient Descent and demonstrating
how transformers could implement them. A key assumption of these works is a
known parameter space within which gradient descent operates. Our work focuses
on identifying such a parameter space for LLMs.

#### ICL in LLMs

Olsson et al. (2022) identify “induction heads” in transformers as a likely
main mechanism of ICL. Dai et al. (2022) provide empirical evidence for the
connection of ICL to Gradient Descent in LLMs, focusing on classification
tasks. Concurrent work by Merullo et al. (2023) also explores a phenomenon
similar to the task vectors we study here, where a single vector can encode
learned functions. Our findings are complementary to theirs, and future work
could explore the relationship between the two more closely.

Task |  Top tokens in the task vector projection
---|---
Previous |  e, y, unknown, alphabet, preceding, c
Letter |  Cad, zA, dit, bill
FR-EN |  Mason, gram, immer, Santi, latin, utter, Span, Conc, English, equivalent
Present Simple to Gerund |  cin, thats, gram, Lorenzo, cian, Isabel, uld, berto, partici, Sah
Country Capital |  Paris, its, capital, central, Conc, cities, administrative, Los, Madrid, London
Table 3: The top 10 tokens in the distribution induced by the task vector, for
one task per category.

##  8 Conclusions

Through this exploration of ICL in LLMs, we have shed light on a new
perspective of ICL learning mechanisms. We have revealed a simple and elegant
structure: ICL functions by compressing a given training set into a single
task vector, which then guides the transformer to generate appropriate outputs
given queries. Our work provides a stepping stone towards understanding how
LLMs perform ICL. In light of our findings, future work could focus on
understanding how the task vector is constructed as well as how it is used to
calculate the output.

## Limitations

We study relatively simple tasks, whereas ICL can learn to perform more
complex tasks, such as solving arithmetic reasoning problems. It remains to be
seen if and how the mechanisms we observe here will translate to these cases.
E.g., our approach focuses on cases where a single task vector suffices, while
more complex ICL cases may require more elaborate parameterization. We also
focus on tasks where the output is a single token, while some other tasks
require multi-token outputs.

Finally, as noted above, we do not provide a mechanistic explanation for how
the task vector is formed or how it is used. Namely, we do not explain how the
transformer performs these calculations using its parameters.

## Acknowledgements

This project is funded by the European Research Council (ERC) under the
European Unions Horizon 2020 research and innovation program (grant ERC HOLI
819080).

## References

  * Akyürek et al. (2022) Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. 2022.  What learning algorithm is in-context learning? investigations with linear models.  _arXiv preprint arXiv:2211.15661_.
  * Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023.  Pythia: A suite for analyzing large language models across training and scaling.  _arXiv preprint arXiv:2304.01373_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chan et al. (2022) Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. 2022.  Data distributional properties drive emergent in-context learning in transformers.  _Advances in Neural Information Processing Systems_ , 35:18878–18891.
  * Dai et al. (2022) Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. 2022.  Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers.  _arXiv preprint arXiv:2212.10559_.
  * Dar et al. (2022) Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2022.  Analyzing transformers in embedding space.  _arXiv preprint arXiv:2209.02535_.
  * Dong et al. (2022) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022.  A survey for in-context learning.  _arXiv preprint arXiv:2301.00234_.
  * (8) Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant.  What can transformers learn in-context? a case study of simple function classes.  In _Advances in Neural Information Processing Systems_.
  * Ilharco et al. (2023) Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023.  Editing models with task arithmetic.  In _The Eleventh International Conference on Learning Representations_.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  _arXiv preprint arXiv:2104.08691_.
  * Levine et al. (2022) Yoav Levine, Noam Wies, Daniel Jannai, Dan Navon, Yedid Hoshen, and Amnon Shashua. 2022.  The inductive bias of in-context learning: Rethinking pretraining example design.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
  * Liu et al. (2023) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _ACM Computing Surveys_ , 55(9):1–35.
  * Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022.  Locating and editing factual associations in gpt.  _Advances in Neural Information Processing Systems_ , 35:17359–17372.
  * Merullo et al. (2023) Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. 2023.  Language models implement simple word2vec-style vector arithmetic.  _arXiv preprint arXiv:2305.16130_.
  * nostalgebraist (2020) nostalgebraist. 2020.  interpreting gpt: the logit lens.  _LessWrong_.
  * Olsson et al. (2022) Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. 2022.  In-context learning and induction heads.  _arXiv preprint arXiv:2209.11895_.
  * Shalev-Shwartz and Ben-David (2014) Shai Shalev-Shwartz and Shai Ben-David. 2014.  _Understanding machine learning: From theory to algorithms_.  Cambridge university press.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * von Oswald et al. (2022) Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. 2022.  Transformers learn in-context by gradient descent.  _arXiv preprint arXiv:2212.07677_.
  * Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax.
  * (21) Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma.  An explanation of in-context learning as implicit bayesian inference.  In _International Conference on Learning Representations_.

##  Appendix A Appendix

Here we provide additional details and results.

###  A.1 Additional Details

#### Full Task Descriptions

Our study covers 18 tasks in 4 categories: Algorithmic, Translation,
Linguistic and Knowledge. A detailed description of all tasks is provided in
Tab. 5.

#### Model Details

More details on the models used in the study are provided in Tab. 4.

#### Task Data

Here we detail the sources of the data for each task. The accompanying GitHub
repository contains the data itself as well as the code used to create it.

  * •

Algorithmic: Generated programatically.

  * •

Translation: For each language pair, the most frequent words in the source
language are first retrieved from https://github.com/frekwencja/most-common-
words-multilingual and are then translated to the destination language using
the open-source package nltk.

  * •

Linguistic: The data for the tenses tasks is parsed from
https://github.com/Drulac/English-Verbs-Conjugates. The data for the plural-
singular task is taken from https://github.com/sindresorhus/irregular-plurals.
Finally, the data for the antonyms task is taken from
https://github.com/SuzanaK/english_synonyms_antonyms_list.

  * •

Knowledge Data for the knowledge tasks is taken from the counterfactual
dataset introduced in Meng et al. (2022).

#### Conflicting Tasks Experiment

In Fig. 6, we provide more details and a visualization of the experiment
described in §5.

###  A.2 Additional Results

#### Finding $\mathcal{A}$ and $f$

Fig. 7 shows results similar to Fig. 3, but for different models. It is
interesting to observe that the curves are similar across different-sized
models.

#### Detailed results for Fig. 4.

Fig. 4 presented results for our $(\mathcal{A},f)$ hypothesis-based approach,
averaged across tasks. Table. LABEL:table:main_results provides these results
for all the specific tasks considered.

#### Dependence of $\mathcal{A}$ on $x$

Fig. 9 and Fig. 8 provide more results on the geometry of the
$\boldsymbol{\theta}$ vectors (see main text for discussion).

#### Inspecting Task Vectors

Tab. 7 is an expanded version of Tab. 3, providing more vocabulary projections
of $\boldsymbol{\theta}$ for additional tasks and on multiple LLMs.

Model | Parameters | Dimension | Layers | Heads
---|---|---|---|---
LLaMA | 7B | 4096 | 32 | 32
13B | 5120 | 40 | 40
30B | 6656 | 60 | 52
GPT-J | 6B | 4096 | 28 | 16
Pythia | 2.8B | 2560 | 32 | 32
6.9B | 4096 | 32 | 32
12B | 5120 | 36 | 40
Table 4: The models used in the study, with architectural information.

Category | Task |  Description | Example
---|---|---|---
Algorithmic | List first |  Given a list of letters, output the first letter | a,b,c $\rightarrow$ a
List last |  Given a list of letters, output the last letter | a,b,c $\rightarrow$ c
Next letter |  Given a letter in the English alphabet, output the next letter | a $\rightarrow$ b
Previous letter |  Given a letter in the English alphabet, output the previous letter | b $\rightarrow$ a
| To lowercase |  Given an uppercase letter, output the corresponding lowercase letter | A $\rightarrow$ a
| To uppercase |  Given a lowercase letter, output the corresponding uppercase letter | a $\rightarrow$ A
Translation | French to English |  Given a word in French, translate to English | bonjour $\rightarrow$ hello
Spanish to English |  Given a word in Spanish, translate to English | hola $\rightarrow$ hello
| English to Spanish |  Given a word in English, translate to Spanish | hola $\rightarrow$ hello
| English to Spanish |  Given a word in English, translate to French | hola $\rightarrow$ hello
Linguistic | Present to gerund |  given an English verb in present simple tense, output the corresponding gerund form | go $\rightarrow$ going
Present to past |  given an English verb in present simple tense, output the corresponding verb in past simple | go $\rightarrow$ went
Singular to plural |  Given an English noun in singular form, output the plural form | cat $\rightarrow$ cats
| Antonyms |  Given an English adjective, output an antonym | happy $\rightarrow$ sad
Knowledge | Country to Capital |  Given a name of a country, output the name of the capital city | France $\rightarrow$ Paris
Person to Language |  Given a name of a person, output their native language | Macron $\rightarrow$ French
| Location to Continent |  Given a name of a person, output their native language | Paris $\rightarrow$ Europe
| Religion |  Given a name of a location or a person, output the associated religion | Muhammad $\rightarrow$ Islam
Table 5: The tasks used in the study with input $\rightarrow$ output examples.

Figure 6:  Conflicting tasks experiment. In the “Regular” scenario (top), the model is simply provided with demonstrations $S_{A}$ for Task $A$ (e.g. outputting the previous letter in the alphabet). In the “Conflicting” scenario (bottom), the model is still provided with demonstrations for Task $A$, but we inject a task vector $\boldsymbol{\theta}(S_{B})$ from a conflicting Task $B$ (e.g. outputting the next letter in the alphabet).  Figure 7: Accuracy for each choice of $L$ (the intermediate layer where the task vector is injected), averaged across all tasks. The solid line represents the average value, and the shaded area depicts the standard deviation.  Table 6: Complete results for Figure 4, reported for all tasks and models. |  | method | Baseline | Hypothesis | Regular
---|---|---|---|---|---
Model | Task type | Task name |  |  |
GPT-J 6B | Algorithmic | List first | 0.30 | 0.74 | 0.98
List last | 0.24 | 0.64 | 1.00
Next letter | 0.16 | 1.00 | 0.86
Prev letter | 0.10 | 0.36 | 0.42
To lower | 0.00 | 0.46 | 1.00
To upper | 0.00 | 0.94 | 1.00
Knowledge | Country capital | 0.19 | 0.72 | 0.80
Location continent | 0.03 | 0.58 | 0.70
Location religion | 0.09 | 0.68 | 0.78
Person language | 0.02 | 0.82 | 0.82
Linguistic | Antonyms | 0.43 | 0.68 | 0.78
Plural singular | 0.08 | 0.90 | 0.98
Present simple gerund | 0.00 | 0.88 | 0.98
Present simple past simple | 0.02 | 0.76 | 0.96
Translation | En es | 0.14 | 0.34 | 0.56
En fr | 0.16 | 0.36 | 0.54
Es en | 0.06 | 0.70 | 0.74
Fr en | 0.13 | 0.66 | 0.76
LLaMA 13B | Algorithmic | List first | 0.77 | 1.00 | 1.00
List last | 0.07 | 0.70 | 0.92
Next letter | 0.31 | 1.00 | 0.94
Prev letter | 0.05 | 0.34 | 0.50
To lower | 0.00 | 0.94 | 1.00
To upper | 0.00 | 0.94 | 1.00
Knowledge | Country capital | 0.17 | 0.84 | 0.86
Location continent | 0.01 | 0.70 | 0.80
Location religion | 0.10 | 0.74 | 0.84
Person language | 0.02 | 0.76 | 0.88
Linguistic | Antonyms | 0.19 | 0.74 | 0.80
Plural singular | 0.24 | 0.84 | 0.88
Present simple gerund | 0.00 | 0.96 | 0.96
Present simple past simple | 0.01 | 1.00 | 0.98
Translation | En es | 0.05 | 0.78 | 0.82
En fr | 0.15 | 0.70 | 0.84
Es en | 0.29 | 0.76 | 0.88
Fr en | 0.25 | 0.54 | 0.72
LLaMA 30B | Algorithmic | List first | 0.96 | 0.98 | 1.00
List last | 0.02 | 0.64 | 0.96
Next letter | 0.30 | 0.98 | 0.96
Prev letter | 0.02 | 0.56 | 0.80
To lower | 0.00 | 1.00 | 1.00
To upper | 0.00 | 0.90 | 1.00
Knowledge | Country capital | 0.27 | 0.72 | 0.88
Location continent | 0.01 | 0.70 | 0.86
Location religion | 0.05 | 0.70 | 0.88
Person language | 0.01 | 0.72 | 0.90
Linguistic | Antonyms | 0.37 | 0.76 | 0.82
Plural singular | 0.21 | 0.84 | 0.90
Present simple gerund | 0.00 | 0.76 | 0.98
Present simple past simple | 0.02 | 0.98 | 1.00
Translation | En es | 0.07 | 0.74 | 0.78
En fr | 0.10 | 0.80 | 0.86
Es en | 0.24 | 0.70 | 0.88
Fr en | 0.20 | 0.62 | 0.78
LLaMA 7B | Algorithmic | List first | 0.87 | 0.98 | 1.00
List last | 0.03 | 1.00 | 1.00
Next letter | 0.03 | 0.94 | 0.88
Prev letter | 0.04 | 0.52 | 0.58
To lower | 0.00 | 0.74 | 1.00
To upper | 0.00 | 0.60 | 1.00
Knowledge | Country capital | 0.28 | 0.82 | 0.86
Location continent | 0.02 | 0.68 | 0.72
Location religion | 0.12 | 0.84 | 0.94
Person language | 0.02 | 0.68 | 0.78
Linguistic | Antonyms | 0.33 | 0.74 | 0.76
Plural singular | 0.15 | 0.84 | 0.88
Present simple gerund | 0.00 | 0.74 | 0.90
Present simple past simple | 0.02 | 0.94 | 0.92
Translation | En es | 0.07 | 0.78 | 0.76
En fr | 0.04 | 0.78 | 0.88
Es en | 0.21 | 0.68 | 0.92
Fr en | 0.15 | 0.66 | 0.70
Pythia 12B | Algorithmic | List first | 0.53 | 0.98 | 0.96
List last | 0.09 | 0.98 | 1.00
Next letter | 0.15 | 0.96 | 0.76
Prev letter | 0.00 | 0.24 | 0.42
To lower | 0.02 | 1.00 | 1.00
To upper | 0.00 | 0.98 | 1.00
Knowledge | Country capital | 0.19 | 0.58 | 0.82
Location continent | 0.01 | 0.68 | 0.80
Location religion | 0.07 | 0.64 | 0.78
Person language | 0.01 | 0.72 | 0.86
Linguistic | Antonyms | 0.34 | 0.72 | 0.74
Plural singular | 0.18 | 0.80 | 0.84
Present simple gerund | 0.00 | 0.86 | 0.96
Present simple past simple | 0.01 | 0.76 | 0.94
Translation | En es | 0.10 | 0.44 | 0.72
En fr | 0.16 | 0.48 | 0.54
Es en | 0.05 | 0.68 | 0.80
Fr en | 0.14 | 0.68 | 0.80
Pythia 2.8B | Algorithmic | List first | 0.69 | 0.96 | 1.00
List last | 0.06 | 0.98 | 1.00
Next letter | 0.42 | 0.86 | 0.90
Prev letter | 0.01 | 0.22 | 0.48
To lower | 0.00 | 1.00 | 1.00
To upper | 0.00 | 1.00 | 1.00
Knowledge | Country capital | 0.18 | 0.70 | 0.76
Location continent | 0.01 | 0.62 | 0.72
Location religion | 0.08 | 0.76 | 0.82
Person language | 0.00 | 0.82 | 0.82
Linguistic | Antonyms | 0.37 | 0.68 | 0.76
Plural singular | 0.13 | 0.70 | 0.78
Present simple gerund | 0.00 | 0.86 | 0.96
Present simple past simple | 0.03 | 0.80 | 0.92
Translation | En es | 0.10 | 0.26 | 0.76
En fr | 0.16 | 0.28 | 0.60
Es en | 0.08 | 0.76 | 0.82
Fr en | 0.10 | 0.64 | 0.82
Pythia 6.9B | Algorithmic | List first | 0.43 | 1.00 | 0.98
List last | 0.08 | 0.60 | 0.98
Next letter | 0.01 | 0.66 | 0.86
Prev letter | 0.04 | 0.28 | 0.32
To lower | 0.00 | 1.00 | 1.00
To upper | 0.00 | 0.94 | 1.00
Knowledge | Country capital | 0.21 | 0.76 | 0.82
Location continent | 0.01 | 0.62 | 0.78
Location religion | 0.10 | 0.80 | 0.80
Person language | 0.01 | 0.76 | 0.80
Linguistic | Antonyms | 0.33 | 0.72 | 0.74
Plural singular | 0.14 | 0.78 | 0.88
Present simple gerund | 0.00 | 0.82 | 0.94
Present simple past simple | 0.02 | 0.88 | 0.96
Translation | En es | 0.11 | 0.46 | 0.70
En fr | 0.21 | 0.36 | 0.60
Es en | 0.06 | 0.72 | 0.82
Fr en | 0.14 | 0.66 | 0.74
Figure 8: Task Vector Variability. For each task, two histograms are shown: (blue) the distribution of distances between different task vectors of this task, created from different $S$ and $x^{\prime}$; (orange) the distribution of distances between task vectors of the task and of other tasks. Figure 9: A 2D t-SNE plot, visualizing 50 task vectors for each task, each generated from a different choice of $S$ and $x$ using LLaMA 7B. Points are color-coded according to task category, such as algorithmic or translation. Each task can be seen to form its own distinct cluster. The labels provide the full name of the task in the cluster. Model |  Task |  Tokens
---|---|---
LLaMA 13B |  Prev Letter |  e, y, unknown, alphabet, preceding, c, Cad, zA, dit, bill, closer, etc, Stuart, aa, null, cin, ads, g, ulo, Ku
FR-EN |  Mason, gram, immer, Santi, latin, utter, Span, Conc, English, equivalent, engl, Usage, none, pron, ulo, translate, adu, Wiel, grammar, ML
Present Simple to Gerund |  cin, thats, gram, Lorenzo, cian, Isabel, uld, berto, partici, Sah, reporting, eing, tc, Roberto, habit, Writing, etc, ientos, ores, Dutch
Country Capital |  Paris, its, capital, central, Conc, cities, administrative, Los, Madrid, London, San, Isabel, exec, Ar, Bel, Wars, name, capit, Battle, History
Pythia 12B |  Prev Letter |  r, b, a, d, m, e, p, n, t, u, h, f, c, in, g, s, the, ar, l, x
FR-EN |  in, and, m, d, a, or, out, the, t, o, so, c, con, have, act, e, s, is, all, to
Present Simple to Gerund |  in, t, m, r, a, and, the, ing, action, d, o, e, current, simple, te, w, not, have, out, what
Country Capital |  the, in, a, C, N, B, L, M, T, P, S, R, G, and, F, I, K, U, D, H
GPT-J 6B |  Prev Letter |  b, c, v, g, s, name, i, ro, n, j, d, t, A, ai, com, m, ust, test, active, k
FR-EN |  other, name, the, true, is, social, s, active, time, car, type, money, F, force, a, public, heart, one, ms, life
Present Simple to Gerund |  getting, storing, working, moving, playing, doing, making, driving, shooting, picking, being, sending, putting, selling, watching, changing, taking, collecting, feeding, reading
Country Capital |  London, Paris, New, West, Berlin, South, Tokyo, San, Chicago, City, Moscow, Jerusalem, Amsterdam, Philadelphia, East, Madrid, Vienna, Beijing, Mexico, Germany
Table 7: The top 20 tokens in the distribution induced by the task vector, for
one task per category.