# Skew polynomial rings over abelian and idempotent reflexive rings

Mohamed Louzari  Lycée Téchnique Al Khawarizmi,
Chefchaouen 91 000
Morocco<EMAIL_ADDRESS>
###### Abstract.

Let $R$ be a ring and $\sigma$ an endomorphism of $R$. In this note, we study
skew polynomial rings and skew power series rings over idempotent reflexive
rings and abelian rings. Also, we introduce the concept of right (resp., left)
$\sigma$-idempotent reflexive rings which generalizes right (resp., left)
idempotent reflexive rings and $\sigma$-abelian rings. Certain results are
obtained as corollaries from our results.

000 2010 Mathematics Subject Classification. 16U80, 16S36 Key words and
phrases. Idempotent reflexive rings, Abelian rings, Skew polynomial rings,
Skew power series rings, Ore extensions, $\sigma$-idempotent reflexive rings,
$\sigma$-abelian rings

##  1\. Introduction

Throughout this paper, $R$ denotes an associative ring with unity. Let $Id(R)$
be the set of all idempotent elements of $R$. A ring $R$ is called abelian if
all idempotent of it are central (i.e., $re=er$ for all $r\in R$ and $e\in
Id(R)$). Following Hashemi and Moussavi [5], a ring $R$ is $\sigma$-compatible
if for each $a,b\in R$, $ab=0$ if and only if $a\sigma(b)=0$. Agayev et al.
[1], introduced $\sigma$-abelian rings as a generalization of abelian rings. A
ring $R$ is called $\sigma$-abelian if it is abelian and $\sigma$-compatible.
A left ideal $I$ is said to be reflexive [11], if $aRb\subseteq I$ implies
$bRa\subseteq I$ for $a,b\in R$. A ring $R$ is called reflexive if $0$ is a
reflexive ideal. The reflexive property for ideals was first studied by Mason
[11], this concept was generalized by Kim and Baik [7, 8], and they introduced
the concept of idempotent reflexive right ideals and rings. A left ideal $I$
is called idempotent reflexive [7], if $aRe\subseteq I$ implies $eRa\subseteq
I$ for $a,e=e^{2}\in R$. A ring $R$ is called idempotent reflexive if $0$ is
an idempotent reflexive ideal. Also, there is an example of an idempotent
reflexive ring which is not reflexive [7, Example 5]. Kwak and Lee [9],
introduced the concept of left and right idempotent reflexive rings. A ring
$R$ is called left idempotent reflexive if $eRa=0$ implies $aRe=0$ for
$a,e=e^{2}\in R$. Right idempotent reflexive rings are defined similarly. If a
ring $R$ is left and right idempotent reflexive then it is called an
idempotent reflexive ring. It can be easily checked that every reflexive ring
is an idempotent reflexive ring. Abelian rings (i.e., its idempotents are
central) are idempotent reflexive and hence semicommutative rings are
idempotent reflexive since semicommutative rings are abelian. For a subset $X$
of $R$, $r_{R}(X)=\\{a\in R|Xa=0\\}$ and $\ell_{R}(X)=\\{a\in R|aX=0\\}$ will
stand for the right and the left annihilator of $X$ in $R$ respectively.

For an endomorphism $\sigma$ of a ring $R$, a skew polynomial ring (also
called an Ore extension of endomorphism type) $R[x;\sigma]$ of $R$ is the ring
obtained by giving the polynomial ring over $R$ with the new multiplication
$xr=\sigma(r)x$ for all $r\in R$. Also, a skew power series ring
$R[[x;\sigma]]$ is the ring consisting of all power series of the form
$\sum_{i=0}^{\infty}a_{i}x^{i}\;(a_{i}\in R)$, which are multiplied using the
distributive law and the Ore commutation rule $xa=\sigma(a)x$, for all $a\in
R$.

In this paper, we study skew polynomial rings and skew power series rings over
idempotent reflexive rings and abelian rings. Also, we used the idea of Agayev
et al. [1] to introduce the concept of right (resp., left) $\sigma$-idempotent
reflexive rings which is a generalization of right (resp., left) idempotent
reflexive rings and $\sigma$-abelian rings. Agayev et al. [1], studied abelian
and $\sigma$-abelian properties on skew polynomial rings. Also, Kwak and Lee
[9], investigated reflexive and idempotent reflexive properties on ordinary
polynomial extensions. In this note, we continue studding abelian and
idempotent reflexive properties on skew polynomial and skew power series
extensions. For a ring $R$ and $\sigma$ an endomorphism of $R$.

$1.$ We introduce the concept of right (resp., left) $\sigma$-idempotent
reflexive rings which generalizes right (resp., left) idempotent reflexive
rings and $\sigma$-abelian rings.

$2.$ We discuss when (left and right) idempotent reflexive property passes
form $R$ to $R[x;\sigma]$ (or $R[[x;\sigma]]$) and conversely, under the
conditions “$\sigma$-compatible” and “$\sigma(Re)\subseteq Re$ for all $e\in
Id(R)$”.

$3.$ We show that, if $R$ satisfy the condition $(\mathcal{C_{\sigma}})$ then,
abelian property passes from $R$ to $R[x;\sigma]$ and $R[[x;\sigma]]$. Some
results are obtained as corollaries of our results.

##  2\. Idempotent reflexive property

In [10, Theorem 3.1], a ring $R$ is idempotent reflexive if and only if $R[x]$
is idempotent reflexive if and only if $R[[x]]$ is idempotent reflexive. One
might expect if idempotent reflexive property can be preserved under skew
polynomial and skew power series extensions.

######  Proposition 2.1.

Let $R$ be a ring and $\sigma$ an endomorphism of $R$ such that
$\sigma(Re)\subseteq Re$ for all $e\in Id(R)$.

$(1)$ If $R$ is left idempotent reflexive then $R[x,\sigma]$ and
$R[[x,\sigma]]$ are left idempotent reflexive.

$(2)$ If $R[x,\sigma]$ or $R[[x,\sigma]]$ is right idempotent reflexive then
$R$ is right idempotent reflexive.

###### Proof.

$(1)$ Let $e(x)=\sum_{j=0}^{m}e_{j}x^{j}\in Id(R[x,\sigma])$ and
$f(x)=\sum_{i=0}^{n}f_{i}x^{i}\in R[x,\sigma]$. Assume that
$e(x)R[x,\sigma]f(x)=0$, then $e(x)Rf(x)=0$. Thus for any $b\in R$, we have
the following system of equations.

$None$ $e_{0}bf_{0}=0$ $None$
$e_{0}bf_{1}+e_{1}\sigma(bf_{0})=0\qquad\qquad\;\;\;$ $None$
$e_{0}bf_{2}+e_{1}\sigma(bf_{1})+e_{2}\sigma^{2}(bf_{0})=0\qquad\qquad\qquad\qquad\qquad$
$\cdots$

From equation $(0)$, $f_{0}\in r_{R}(e_{0}R)$. Let $s\in R$ and take
$b=e_{0}s$ in equation $(1)$. Then $e_{0}sf_{1}+e_{1}\sigma(e_{0}sf_{0})=0$,
but $f_{0}\in r_{R}(e_{0}R)$. So $e_{0}sf_{1}=0$, hence $f_{1}\in
r_{R}(e_{0}R)$. Now, in equation $(2)$, take $b=e_{0}s$ for each $s\in R$.
Then $e_{0}sf_{2}+e_{1}\sigma(e_{0}sf_{1})+e_{2}\sigma^{2}(e_{0}sf_{0})=0$,
since $f_{0},f_{1}\in r_{R}(e_{0}R)$ so $e_{0}sf_{2}=0$. Hence $f_{2}\in
r_{R}(e_{0}R)$. Continuing this procedure yields $f_{i}\in r_{R}(e_{0}R)$ for
all $i=0,1,2,\cdots,n$. Hence $e_{0}Rf_{i}=0$ for all $i=0,1,2,\cdots,n$. Thus
$f_{i}Re_{0}=0$ for all $i=0,1,\cdots,n$ because $R$ is left idempotent
reflexive. On the other hand,

$f(x)Re_{0}=f_{0}Re_{0}+f_{1}xRe_{0}+\cdots+f_{n}x^{n}Re_{0}$
$\qquad\qquad\qquad\quad\;=f_{0}Re_{0}+f_{1}\sigma(Re_{0})x+\cdots+f_{n}\sigma^{n}(Re_{0})x^{n}$

Since $\sigma(Re_{0})\subseteq Re_{0}$, it follows that
$f(x)Re_{0}\subseteq\sum_{i=0}^{n}f_{i}Re_{0}=0$, then $f(x)Re_{0}=0$. With
the same method as above, we obtain

$f(x)Rxe_{0}=f(x)Rx^{2}e_{0}=\cdots=f(x)Rx^{n}e_{0}=0.$

So $f(x)R[x,\sigma]e_{0}=0$. However, since $e(x)^{2}=e(x)$, we have
$e_{0}^{2}=e_{0},\;e_{0}e_{1}+e_{1}\sigma(e_{0})=e_{1},\>\cdots$. Using the
hypothesis $``\sigma(Re_{0})\subseteq Re_{0}"$, we have $e_{j}\in Re_{0}R$ for
any $1\leq j\leq m$. Thus from the fact $f(x)R[x;\sigma]e_{0}=0$, we have
$f(x)R[x;\sigma]e_{j}=0$, and therefore $f(x)R[x;\sigma]e(x)=0$. We use
similar argument for the case of skew power series rings.

$(2)$ Suppose that $R[x;\sigma]$ (resp., $R[[x;\sigma]]$) is right idempotent
reflexive. If $aRe=0$ for some $e^{2}=e,a\in R$ then $aR[x;\sigma]e=0$, by the
hypothesis “$\sigma(Re)\subseteq Re$”. Since $R[x;\sigma]$ (resp.,
$R[[x;\sigma]]$) is right idempotent reflexive, we get $eR[x;\sigma]a=0$, then
$eRa=0$. Therefore $R$ is right idempotent reflexive. ∎

We can easily see that, if a ring $R$ is $\sigma$-compatible then
$\sigma(e)=e$ for all $e\in Id(R)$, and so the condition “$\sigma(Re)\subseteq
Re$ for all $e\in Id(R)$”, is satisfied.

######  Proposition 2.2.

Let $R$ be a ring and $\sigma$ an endomorphism of $R$ such that $R$ is
$\sigma$-compatible.

$(1)$ If $R$ is right idempotent reflexive then $R[x,\sigma]$ and
$R[[x,\sigma]]$ are right idempotent reflexive.

$(2)$ If $R[x,\sigma]$ or $R[[x,\sigma]]$ is left idempotent reflexive then
$R$ is left idempotent reflexive.

###### Proof.

$(1)$ Assume that $R$ is right idempotent reflexive and let
$e(x)=\sum_{i=0}^{n}e_{i}x^{i}\in Id(R[x,\sigma])$ and
$f(x)=\sum_{j=0}^{m}f_{j}x^{j}\in R[x,\sigma]$ such that
$f(x)R[x,\sigma]e(x)=0$. Thus we have the following system of equations where
$b$ is an arbitrary element of $R$.

$None$ $f_{0}be_{0}=0$ $None$
$f_{0}be_{1}+f_{1}\sigma(be_{0})=0\qquad\qquad\;\;\;$ $None$
$f_{0}be_{2}+f_{1}\sigma(be_{1})+f_{2}\sigma^{2}(be_{0})=0\qquad\qquad\qquad\qquad\qquad$
$\cdots$

Equation $(0)$ yields $f_{0}\in\ell_{R}(Re_{0})$. In equation $(1)$ substitute
$se_{0}$ for $b$ to obtain $f_{1}\sigma(se_{0})=0$, but $R$ is
$\sigma$-compatible then $f_{1}se_{0}=0$ and so $f_{1}\in\ell_{R}(Re_{0})$.
Continuing this procedure yields $f_{i}\in\ell_{R}(Re_{0})$ for all
$i=0,1,\cdots,n$. Since $R$ is right idempotent reflexive, it follows that
$e_{0}Rf_{i}=0$ for all $i=0,1,\cdots,n$. We claim that
$e_{0}R[x,\sigma]f(x)=0$. Observe that
$e_{0}Rf(x)=\sum_{j=0}^{m}e_{0}Rf_{j}x^{j}=0$ and
$e_{0}Rxf(x)=\sum_{j=0}^{m}e_{0}R\sigma(f_{j})x^{j+1}=0$ because $R$ is
$\sigma$-compatible. Also, we have $e_{0}Rx^{k}f(x)=0$ for all nonnegative
integers $k$. Consequently $e_{0}\varphi(x)f(x)=0$ for all $\varphi(x)\in
R[x,\sigma]$. Thus $e_{0}R[x,\sigma]f(x)=0$. As, in the proof of Proposition
2.1, we have $e_{i}\in Re_{0}R$ for any $1\leq i\leq n$. It follows that
$e(x)R[x,\sigma]f(x)=0$. Therefore, $R[x,\sigma]$ is right idempotent
reflexive.

$(2)$ As in the proof of Proposition 2.1(2), it suffices to replace the
hypothesis “$\sigma(Re)\subseteq Re$” by “$\sigma$-compatible”. ∎

######  Theorem 2.3.

Let $R$ be a ring and $\sigma$ an endomorphism of $R$ such that $R$ is
$\sigma$-compatible. Then

$(1)$ $R$ is right idempotent reflexive if and only if $R[x,\sigma]$ is right
idempotent reflexive if and only if $R[[x,\sigma]]$ is right idempotent
reflexive.

$(2)$ $R$ is left idempotent reflexive if and only if $R[x,\sigma]$ is left
idempotent reflexive if and only if $R[[x,\sigma]]$ is left idempotent
reflexive.

$(3)$ $R$ is idempotent reflexive if and only if $R[x,\sigma]$ is idempotent
reflexive if and only if $R[[x,\sigma]]$ is idempotent reflexive.

###### Proof.

Obvious from Propositions 2.1 and 2.2. ∎

Following Agayev et al. [1]. A ring $R$ is called $\sigma$-abelian if it is
abelian and $\sigma$-compatible. In the next, we use the same idea to
introduce $\sigma$-idempotent reflexive rings as a generalization of
idempotent reflexive rings and $\sigma$-abelian rings.

######  Definition 2.4.

Let $R$ be a ring and $\sigma$ an endomorphism of $R$. $(1)$ $R$ is called
right $($resp., left$)$ $\sigma$-idempotent reflexive if it is right $($resp.,
left$)$ idempotent reflexive and $\sigma$-compatible. $(2)$ $R$ is called
$\sigma$-idempotent reflexive if it is both right and left $\sigma$-idempotent
reflexive.

A ring $R$ is right (resp., left) idempotent reflexive if $R$ is right (resp.,
left) $id_{R}$\- idempotent reflexive, where $id_{R}$ denotes the identity
endomorphism of $R$. We can easily see that $\sigma$-abelian rings are
$\sigma$-idempotent reflexive.

######  Example 2.5.

Consider the ring $R=\left\\{\left(\begin{array}[]{cc}a&b\\\ 0&a\\\
\end{array}\right)\mid a,b\in\mathbb{Z}_{4}\right\\}$, where $\mathbb{Z}_{4}$
is the ring of integers modulo $4$. Let $\sigma$ an endomorphism of $R$
defined by

$\sigma\left(\left(\begin{array}[]{cc}a&b\\\ 0&a\\\
\end{array}\right)\right)=\left(\begin{array}[]{cc}a&-b\\\ 0&a\\\
\end{array}\right)\mathrm{for\;all}\;\left(\begin{array}[]{cc}a&b\\\ 0&a\\\
\end{array}\right)\in R$

By [1, Example 2.2]. The ring $R$ is abelian so it is right and left
idempotent reflexive. Also, $R$ is $\sigma$-compatible. Thus $R$ is right and
left $\sigma$-idempotent reflexive.

######  Corollary 2.6.

Let $R$ be a ring and $\sigma$ an endomorphism of $R$. Then

$(1)$ If $R$ is left $\sigma$-idempotent reflexive then $R[x,\sigma]$ and
$R[[x,\sigma]]$ are left idempotent reflexive.

$(2)$ If $R$ is right $\sigma$-idempotent reflexive then $R[x,\sigma]$ and
$R[[x,\sigma]]$ are right idempotent reflexive.

$(3)$ If $R$ is $\sigma$-idempotent reflexive then $R[x,\sigma]$ and
$R[[x,\sigma]]$ are idempotent reflexive.

###### Proof.

Immediate from Proposition 2.1. ∎

##  3\. Abelian property

In this section, $\sigma$ is an endomorphism of $R$ such that $\sigma(1)=1$.
We can easily see that $Id(R)\subseteq Id(R[x])$. However $Id(R)\neq
Id(R[x])$. We can take the next example.

######  Example 3.1.

Let $F$ be a field and consider the ring $R=\left(\begin{array}[]{cc}F&F\\\
0&F\\\ \end{array}\right)$, then $e=\left(\begin{array}[]{cc}1&0\\\ 0&0\\\
\end{array}\right)+\left(\begin{array}[]{cc}0&1\\\ 0&0\\\ \end{array}\right)x$
is an element of $Id(R[x])$. But $e\not\in Id(R)$.

According to Kanwara et al. [6, Corollary 6], a ring $R$ is abelian if and
only if $R[x]$ is abelian. In the next, we give a generalization to skew
polynomial rings and skew power series rings.

######  Lemma 3.2.

Let $R$ be an abelian ring such that $\sigma(Re)\subseteq Re$ for all $e\in
Id(R)$. For any $e=\sum_{i=0}^{n}e_{i}x^{i}\in Id(R[x,\sigma])$ $($resp.,
$e=\sum_{i=0}^{\infty}e_{i}x^{i}\in Id(R[[x,\sigma]])$$)$, we have
$ee_{0}=e_{0}$ and $e_{0}e=e$.

###### Proof.

Let $e=\sum_{i=0}^{n}e_{i}x^{i}\in Id(R[x,\sigma])$, then we have the
following system of equations

$None$ $e_{0}^{2}=e_{0}$ $None$
$e_{0}e_{1}+e_{1}\sigma(e_{0})=e_{1}\qquad\qquad\;\quad$ $None$
$e_{0}e_{2}+e_{1}\sigma(e_{1})+e_{2}\sigma^{2}(e_{0})=e_{2}\qquad\qquad\qquad\qquad\qquad$
$\vdots$ $None$
$e_{0}e_{n}+e_{1}\sigma(e_{n-1})+\cdots+e_{n}\sigma^{n}(e_{0})=e_{n}\qquad\qquad\qquad\qquad\qquad\qquad\;$

Since abelian rings are idempotent reflexive, then by the proof of Proposition
2.1, we have $e_{i}\in Re_{0}R=e_{0}R$ for all $0\leq i\leq n$, because
$e_{0}$ is central. Hence $e_{i}=e_{0}e_{i}=e_{i}e_{0}$ for all $0\leq i\leq
n$ and so $e_{0}e=e\;\;(1^{\prime})$. On the other hand, from equations $(1)$
and $(1^{\prime})$, we have $e_{1}\sigma(e_{0})=0$ . Also, we have
$e_{1}\sigma(e_{1})+e_{2}\sigma^{2}(e_{0})=0\;(2^{\prime})$ from equations
$(2)$ and $(1^{\prime})$. But $\sigma^{2}(e_{0})$ is an idempotent, then
equation $(2^{\prime})$ becomes
$e_{1}\sigma(e_{1})\sigma^{2}(e_{0})+e_{2}\sigma^{2}(e_{0})=e_{1}\sigma[e_{1}\sigma(e_{0})]+e_{2}\sigma^{2}(e_{0})=0$,
since $e_{1}\sigma(e_{0})=0$ we get $e_{2}\sigma^{2}(e_{0})=0$. Continuing
this procedure yields $e_{i}\sigma^{i}(e_{0})=0$ for all $1\leq i\leq n$. Thus
$ee_{0}=e_{0}$. With similar method we get the result for $R[[x,\sigma]]$. ∎

Recall that a ring $R$ is satisfying the condition $(\mathcal{C_{\sigma}})$ if
whenever $a\sigma(b)=0$ with $a,b\in R$, then $ab=0$ [4]. Clearly,
$\sigma$-compatible rings satisfy the condition $(\mathcal{C_{\sigma}})$.

######  Lemma 3.3.

Let $R$ be a ring and $\sigma$ an endomorphism of $R$. If $R$ satisfies the
condition $(\mathcal{C_{\sigma}})$ then for any $e^{2}=e\in R$, we have
$\sigma(e)=e$.

###### Proof.

Let $e\in Id(R)$, by the condition $(\mathcal{C_{\sigma}})$ and from
$\sigma(e)(1-\sigma(e))=(1-\sigma(e))\sigma(e)=0$, we get
$\sigma(e)=\sigma(e)e$ and $e=\sigma(e)e$. Therefore $\sigma(e)=e$. ∎

######  Proposition 3.4.

If $R$ satisfies the condition $(\mathcal{C_{\sigma}})$. Then $R$ is abelian
if and only if $R[x,\sigma]$ is abelian if and only if $R[[x,\sigma]]$ is
abelian.

###### Proof.

It suffices to show the necessary condition, let
$e=\sum_{i=0}^{n}e_{i}x^{i}\in Id(R[x,\sigma])$. From Lemma 3.2, we get
$e_{i}=e_{i}e_{0}$ and $e_{i}\sigma^{i}(e_{0})=0$ for all $1\leq i\leq n$. By
the condition $(\mathcal{C_{\sigma}})$, we have $e_{i}=0$ for all $1\leq i\leq
n$, then $e=e_{0}\in Id(R)$. Therefore $R[x,\sigma]$ is abelian. The same
method for $R[[x,\sigma]]$. ∎

######  Example 3.5.

Let $F$ be a field. Consider the ring

$R=\left\\{\left(\begin{array}[]{cccc}a&b&0&0\\\ 0&a&0&0\\\ 0&0&0&v\\\
0&0&u&u\\\ \end{array}\right)\mid a,b,u,v\in F\right\\}$

and $\sigma$ an endomorphism of $R$ defined by

$\sigma\left(\left(\begin{array}[]{cccc}a&b&0&0\\\ 0&a&0&0\\\ 0&0&0&v\\\
0&0&u&u\\\ \end{array}\right)\right)=\left(\begin{array}[]{cccc}u&v&0&0\\\
0&u&0&0\\\ 0&0&0&b\\\ 0&0&a&a\\\
\end{array}\right)\mathrm{for\;all}\;\left(\begin{array}[]{cccc}a&b&0&0\\\
0&a&0&0\\\ 0&0&0&v\\\ 0&0&u&u\\\ \end{array}\right)\in R$

$(1)$ $R$ is abelian. But $R[x;\sigma]$ is not abelian by [1, Example 2.17].
$(2)$ $R$ does not satisfy the condition $(\mathcal{C_{\sigma}})$. Let
$e_{ij}$ denote the $4\times 4$ matrix units having alone $1$ as its
$(i,j)$-entry and all other entries $0$. Consider

$e=\left(\begin{array}[]{cccc}1&0&0&0\\\ 0&1&0&0\\\ 0&0&0&0\\\ 0&0&0&0\\\
\end{array}\right)\in Id(R)$

Then

$\sigma(e)=\left(\begin{array}[]{cccc}0&0&0&0\\\ 0&0&0&0\\\ 0&0&0&0\\\
0&0&1&1\\\ \end{array}\right)\neq e$

By Lemma 3.3, the condition $(\mathcal{C_{\sigma}})$ is not satisfied.

We can see from Example 3.5, that the condition $(\mathcal{C_{\sigma}})$ in
Proposition 3.4 is not superfluous.

######  Corollary 3.6.

Let $R$ be a $\sigma$-compatible ring. Then $R$ is abelian if and only if
$R[x,\sigma]$ is abelian if and only if $R[[x,\sigma]]$ is abelian

###### Proof.

Obvious from Proposition 3.4. ∎

######  Corollary 3.7 ([6, Corollary 6]).

Let $R$ be a ring. Then $R$ is abelian if and only if $R[x]$ is abelian if and
only if $R[[x]]$ is abelian.

###### Proof.

It suffices to take $\sigma=id_{R}$ in Proposition 3.4. ∎

######  Corollary 3.8 ([1, Theorem 2.21]).

If $R$ is $\sigma$-abelian then $R[x,\sigma]$ and $R[[x,\sigma]]$ are abelian.

###### Proof.

Clearly from Proposition 3.4. ∎

In the next, we will give an example of a ring $R$ and an endomorphism
$\sigma$ of $R$ such that $R[x,\sigma]$ is abelian, but $R$ is not
$\sigma$-abelian. Thus the converse of Corollary 3.8 is not true.

######  Example 3.9.

Let $\mathbb{K}$ be a field and $R=\mathbb{K}[t]$ a polynomial ring over
$\mathbb{K}$ with the endomorphism $\sigma$ given by $\sigma(f(t))=f(0)$ for
all $f(t)\in R$. Since $R$ is commutative then it is abelian. $(1)$ $R$ is not
$\sigma$-abelian. Take $f=a_{0}+a_{1}t+a_{2}t^{2}+\cdots+a_{n}t^{n}$ and
$g=b_{1}t+b_{2}t^{2}+\cdots+b_{m}t^{m}$, since $g(0)=0$ so, $f\sigma(g)=0$ but
$fg\neq 0$. Therefore $R$ is not $\sigma$-compatible. $(2)$ Since $R$ has only
two idempotents $0$ and $1$ then $Id(R[x,\sigma])=Id(R)$ and thus
$R[x,\sigma]$ is abelian.

######  Remark 3.10.

If for a ring $R$ we have $Id(R)=\\{0,1\\}$, then $Id(R[x,\sigma])=\\{0,1\\}$.
Indeed, let $e=\sum_{i=0}^{n}e_{i}x^{i}\in Id(R[x,\sigma])$ and consider the
system cited in the proof of Lemma 3.2. If $e_{0}=0$ or $e_{0}=1$, we can
easily see that $e_{i}=0$ for all $1\leq i\leq n$.

######  Example 3.11.

Let $\mathbb{Z}_{2}$ is the ring of integers modulo $2$, take
$R=\mathbb{Z}_{2}\oplus\mathbb{Z}_{2}$ with the usual addition and
multiplication. Consider $\sigma\colon R\rightarrow R$ defined by
$\sigma((a,b))=(b,a)$. Let $e=(1,0)+(0,1)x,\;f=(0,1)+(0,1)x\in
Id(R[x,\sigma])$. Then $(1)$ $R$ is abelian because it is commutative. $(2)$
Since $(0,1)e=(0,1)x\neq e(0,1)=0$, then $e$ is not central. Thus
$R[x,\sigma]$ is not abelian.

######  Example 3.12.

Consider the ring $R=\left\\{\begin{pmatrix}a&t\\\
0&a\end{pmatrix}|\;a\in\mathbb{Z}\;,t\in\mathbb{Q}\right\\},$ where
$\mathbb{Z}$ and $\mathbb{Q}$ are the set of all integers and all rational
numbers, respectively. The ring $R$ is commutative then it’s abelian, let
$\sigma\colon R\rightarrow R$ be an automorphism defined by
$\sigma\left(\begin{pmatrix}a&t\\\
0&a\end{pmatrix}\right)=\begin{pmatrix}a&t/2\\\ 0&a\end{pmatrix}$. Since $R$
has only two idempotents $\begin{pmatrix}0&0\\\ 0&0\end{pmatrix}$ and
$\begin{pmatrix}1&0\\\ 0&1\end{pmatrix}$, it follows that
$Id(R[x,\sigma])=Id(R)$. Hence $R[x,\sigma]$ is abelian.

## References

  * [1] N. Agayev, A. Harmanci and S. Halicioǧlu, On abelian rings, Turk. J. Math 33 (2009), 1-10.
  * [2] G. F. Birkenmeier, J. Y. Kim and J. K. Park, On polynomial extensions of principally quasi-Baer rings, Kyungpook Math. journal 40(2) (2000), 247-253.
  * [3] G. F. Birkenmeier and J. K. Park, Triangular matrix representations of rings extensions, J. Algebra 265 (2003), 457-477.
  * [4] L. Ben Yakoub and M. Louzari, Ore extensions of extended symmetric and reversible rings, Inter. Journal of Algebra 3(9)(2009), 423-433.
  * [5] E. Hashemi, A. Moussavi, Polynomial extensions of quasi-Baer rings, Acta. Math. Hungar. 107 (3) (2005) 207-224.
  * [6] P. Kanwara, A. Leroy and J. Matczuk, Idempotents in ring extensions, J. Algebra 389 (2013), 128-136.
  * [7] J. Y. Kim, Certain rings whose simple singular modules are GP-injective, Proc. Japan Acad. Ser. A Math. Sci 81(7) (2005), 125-1128.
  * [8] J. Y. Kim and J. U. Baik, On idempotent reflexive rings, Kyungpook Math. journal 46(4) (2006), 597-601.
  * [9] T. K. Kwak and Y. Lee, Reflexive property of rings, Comm. Algebra 40 (2012), 1576-1594.
  * [10] T. K. Kwak and Y. Lee, Reflexive property on idempotents, Bull. Korean Math. Soc. 50(6)(2013), 1957-1972.
  * [11] G. Mason, Reflexive ideals, Comm. Algebra (9) (1981) 1709-1724.

Mohamed Louzari


Neural processes (NPs) are a powerful family of meta-learning models that seek to approximate the posterior predictive map of the ground-truth stochastic process from which each dataset in a meta-dataset is sampled.

There are many cases in which practitioners, besides having access to the dataset of interest, may also have access to other datasets that share similarities with it. In this case, integrating these datasets into the NP can improve predictions. We equip NPs with this functionality and describe this paradigm as in-context in-context learning. Standard NP architectures, such as the convolutional conditional NP (ConvCNP) or the family of transformer neural processes (TNPs), are not capable of in-context in-context learning, as they are only able to condition on a single dataset. We address this shortcoming by developing the in-context in-context learning pseudo-token TNP (ICICL-TNP). The ICICL-TNP builds on the family of PT-TNPs, which utilise pseudo-token-based transformer architectures to sidestep the quadratic computational complexity associated with regular transformer architectures. Importantly, the ICICL-TNP is capable of conditioning on both sets of datapoints and sets of datasets, enabling it to perform in-context in-context learning. We demonstrate the importance of in-context in-context learning and the effectiveness of the ICICL-TNP in a number of experiments.

§ INTRODUCTION

np are a broad family of meta-learning models which learn the mapping from sets of observed datapoints to predictive distributions [Foong et al., 2020].
They enjoy a number of attractive properties, most notably their ability to effectively model data from different modalities and drawn from complex stochastic process priors.
This flexibility makes np a popular choice for a wide variety of problem domains, including spatio-temporal modelling, healthcare, and few-shot learning [Jha et al., 2022].
A key consideration of np architectures is the ability to handle an arbitrary number of observed datapoints in an exchangeable fashion. This is achieved through the use of permutation-invariant set functions, which, in np, map from the set of observations to some representation space. A popular choice of architecture for these set functions are transformers [Vaswani et al., 2017, Lee et al., 2019], giving rise to the family of tnp [Nguyen and Grover, 2022, Kim et al., 2019, Feng et al., 2022].

In the context of meta-learning, practitioners might have access to related datasets that share similarities with the dataset of interest. If these could be integrated into the NP, its predictions would improve. For example, a meta-dataset of PDE-simulated Navier-Stokes equations could be partitioned into smaller sets of datasets with the same Reynolds number.

Meta-learning a single np can only recover the predictive distribution of the marginal—or average—stochastic process used to generate the entire meta-dataset.

Instead, if we had access to sufficiently many datasets for each possible Reynolds number, a strictly preferable approach would be to meta-learn multiple np—one corresponding to each possible stochastic process (e.g.. Reynolds number). While this would help us recover more accurate predictive distributions, it is often infeasible in practice due to either limited data availability for each possible stochastic process or the existence of a large number of possible stochastic processes.

Rather than learning an individual np for each stochastic process, we propose to learn a single np that is able to condition on additional datasets that are known to be drawn from the same stochastic process. Intuitively, this can be thought of as amortising the learning of stochastic-process specific np, and we refer to this form of meta-learning as in-context in-context learning. Constructing a model that is able to condition on sets of datasets, in addition to sets of observations, requires a function that is able to operate on sets of sets. Recently, there has been a string of research into transformer architectures which perform cross-attention between different data modalities [Jaegle et al., 2021, Borgeaud et al., 2022, Zhang and Yan, 2022, Kim et al., 2021, Shen et al., 2024, Xu et al., 2022, Xu et al., 2023]. Our key insight is that we can use the same methods to construct a function that operates on a set of sets, providing a model architecture for in-context in-context learning. In doing so, we develop the model. We highlight our key contributions as follows:

We show that in-context in-context learning can lead to substantially improved predictive performance.

* We develop a new member of tnp for in-context in-context learning, the model. At the core of the model are pseudo-token based transformer architectures, which enable scaling up to large datasets.

* We provide an empirical investigation into the model, demonstrating that it: i) does not reduce the performance of a regular pt-tnp model when in-context datasets are not observed; and ii) improves predictive performance over a regular pt-tnp model when in-context datasets are observed.

§ BACKGROUND

Throughout this section, we use the following notation. Let $\mcX = \R^{D_x}$ and $\mcY = \R^{D_y}$ denote the input and output spaces, and let $(\bfx, \bfy) \in \mcX \times \mcY$ denote an input-output pair. Let $\mcS = \bigcup_{N=0}^{\infty} \left(\mcX \times \mcY\right)^N$ be a collection of all finite data sets, which includes the empty set $\varnothing$. We denote a context and target set with $\mcD_c, \mcD_t \in \mcS$, where $|\mcD_c| = N_c$ and $|\mcD_t| = N_t$. Let $\bfX_c \in \R^{N_c \times D_x}$, $\bfY_c \in \R^{N_c \times D_y}$ be the inputs and corresponding outputs of $\mcD_c$, with $\bfX_t \in \R^{N_t\times D_x}$, $\bfY_t \in \R^{N_t \times D_y}$ defined analogously. We denote a single task as $\tau = (\mcD_c, \mcD_t) = ((\bfX_c, \bfY_c), (\bfX_t, \bfY_t))$. Let $\mcP(\mcX)$ denote the collection of $\mcY$-valued stochastic processes on $\mcX$. Let $\Theta$ denote the parameter space of predictive distributions over the outputs. Let $\mcZ$ denote some latent space.

§.§ Neural Processes

np [Garnelo et al., 2018, Garnelo et al., 2018] are a type of meta-learning model which seek to learn the posterior prediction map $\pi_P: \mcS \rightarrow \mcP(\mcX)$, which maps from context sets $\mcD_c$ to the posterior predictive distribution over the target outputs $p(\bfY_t | \bfX_t, \mcD_c)$ under the ground-truth stochastic process $P$. np architectures generally consist of an encoder $e: \mcS \times \mcX \rightarrow \mcZ$, which maps from $\mcD_c$ and $\bfX_t$ to some latent representation, and a decoder $d: \mcX \times \mcZ \rightarrow \Theta$, which takes the representation and $\bfX_t$ as inputs and maps to the parameters of the predictive distribution over the target outputs: $p(\bfY_t | \bfX_t, \mcD_c) = p(\bfY_t | d(\bfX_t, e(\bfX_t, \mcD_c)))$. In this work, we limit our attention to cnp [Garnelo et al., 2018], which factorise the predictive distribution as $p(\bfY_t | \bfX_t, \mcD_c) = \prod_{n=1}^{N_t} p(\bfy_{t, n} | d(\bfx_{t, n}, e(\bfx_{t, n}, \mcD_c)))$. cnp are trained by maximising the posterior predictive likelihood:

\begin{equation}
\label{eq:cnp-objective}
\mcL_{\text{ML}} = \Exp{p(\tau)}{\sum_{n=1}^{N_t} \log p(\bfy_{t, n} | d(\bfx_{t, n}, e(\bfx_{t, n}, \mcD_c))}.
\end{equation}

Here, the expectation is taken over $p(\tau)$. In practice, we often only have access to a finite number of tasks, in which case we can replace the expectation with a Monte-Carlo estimate.

§.§ Transformer Neural Processes

Transformers can be understood as permutation-equivariant set functions [Lee et al., 2019]. This makes their use in np natural, since we require a set function to construct the mapping from context sets to predictive distributions. For the family of tnp, this is generally achieved by: 1) obtaining an initial token representation for the context points, $\bfZ^0_c \in \R^{N_c \times D_z}$, and target input locations, $\bfZ^0_t \in \R^{N_t \times D_z}$, using point-wise embeddings; 2) passing the union of the initial context and target tokens, $\bfZ^0 = [\bfZ^0_c, \bfZ^0_t]$, through a transformer-style architecture; and 3) passing the output tokens (corresponding to the target inputs) of the final layer $L$ of the tnp, $\bfZ^L_t$, through another MLP to obtain the parameters of the predictive distribution $p(\bfY_t | \bfX_t, \mcD_c)$. We provide a thorough description of the operations used in transformer-style architectures in <Ref>.

The form of transformer-style architecture varies between members of the family of tnp—we provide a description of several members of the family in <Ref>.

Whilst the regular tnp has many advantages over other np variants, it is hindered by its large computational complexity of $\order{N_c^2 + N_cN_t}$ given by the interaction of each token with the entire set of context tokens. This limits its application to relatively small datasets.  However, the computational complexity of the standard transformer can be reduced through the introduction of pseudo-tokens, leading to pseudo-token based transformers. Let $\bfU \in \R^{M \times D_z}$ denote an initial set of $M \ll N_c$ tokens we refer to as pseudo-tokens. By only allowing tokens to interact with the set of context tokens $\bfZ_c$ indirectly through the smaller set of pseudo-tokens, we are able to reduce the computational complexity to $\order{MN_c + MN_t + M^2}$. We illustrate the architecture for the perceiver-style approach [Jaegle et al., 2021, Feng et al., 2022] and IST-style approach [Lee et al., 2019] of pseudo-token based transformers in <Ref>. We refer to the family of tnp that use a pseudo-token based transformer architecture as pt-tnp.

§ IN-CONTEXT IN-CONTEXT LEARNING

In this section, we define the paradigm of in-context in-context learning. Our key result is provided in <Ref>, which informally states that when provided with multiple datasets drawn from the same stochastic process, we can improve the quality of predictions by taking these datasets into account.

We then propose a meta-learning model for achieving this in <Ref>, the model.

§.§ In-Context In-Context Learning for Mixtures of Stochastic Processes

One of the goals of probabilistic meta-learning can be understood as modelling the posterior prediction map $\pi_P: \mcS \rightarrow \mcP(\mcX)$, where $P$ denotes the ground truth stochastic process over functions mapping from $\mcX$ to $\mcY$ that our datasets are sampled from. Yet, in many cases $P$ is itself a mixture of stochastic processes, and our datasets can be partitioned into samples from each stochastic process mixture. Concretely, we can model each dataset as

\begin{equation}
\xi_i \sim p(\xi), \quad \mcD_i \sim P(\xi_i)
\end{equation}

where $P(\xi_i) \in \mcP(\mcX)$ is a stochastic process defined by the latent variable $\xi_i \in \Xi$ from which dataset $\mcD_i$ is sampled. Consider the setting in which we have a set of additional datasets $\{\mcD_j\}$ also drawn from $P(\xi_i)$. Intuitively, the additional datasets provide information about $P(\xi_i)$, the stochastic process from which $\mcD_i$ is sampled. Conditioning on these additional datasets would therefore reduce our uncertainty of the ground-truth stochastic process from which $\mcD_i$ is sampled, and can therefore improve our approximation of the prediction map $\pi_{P(\xi_i)}$. This is formalised in the following theorem:

Let $\xi_i \sim p(\xi)$, $\mcD_i, \{\mcD_j\} \sim P(\xi_i)$. Let $p(\bfy | \bfx, \mcD_i, \xi_i)$ be the marginal posterior distribution of $P(\xi_i)$ given $\mcD_i$, $p(\bfy | \bfx, \mcD_i, \{\mcD_j\})$ be the marginal posterior distribution of the stochastic process $P$ given $\mcD_i$ and $\{\mcD_j\}$, and $p(\bfy | \bfx, \mcD_i)$ be the marginal posterior distribution of the stochastic process $P$ given $\mcD_i$. Then,

\begin{equation}
\Exp{\mcD_i, \{\mcD_j\}, \xi_i}{\KL{p(\bfy | \bfx, \mcD_i, \xi_i)}{p(\bfy | \bfx, \mcD_i, \{\mcD_j\})}} \leq \Exp{\mcD_i, \xi_i}{\KL{p(\bfy | \bfx, \mcD_i, \xi_i)}{p(\bfy | \bfx, \mcD_i)}}.
\end{equation}

Observe that $\Exp{\mcD_i, \{\mcD_j\}, \xi_i}{\KL{p(\bfy | \bfx, \mcD_i, \xi_i)}{p(\bfy | \bfx, \mcD_i, \{\mcD_j\})}}$ can be expressed as

\begin{equation}
\begin{aligned}
&\Exp{\mcD_i, \{\mcD_j\}, \xi_i}{\KL{p(\bfy | \bfx, \mcD_i, \xi_i)}{p(\bfy | \bfx, \mcD_i, \{\mcD_j\})}} \\
&\qquad = -\Exp{\mcD_i, \xi_i}{\entropy{p(\bfy | \bfx, \mcD_i, \xi_i)}} - \Exp{\mcD_i, \{\mcD_j\}, \bfy}{\log p(\bfy | \bfx, \mcD_i, \{\mcD_j\}} \\
&\qquad = -\Exp{\mcD_i, \xi_i}{\entropy{p(\bfy | \bfx, \mcD_i, \xi_i)}} + \Exp{\mcD_i}{\Exp{\{\mcD_j\} | \mcD_i}{\entropy{p(\bfy | \bfx, \mcD_i, \{\mcD_j\}}}} \\
&\qquad \leq -\Exp{\mcD_i, \xi_i}{\entropy{p(\bfy | \bfx, \mcD_i, \xi_i)}} + \Exp{\mcD_i}{\entropy{p(\bfy | \bfx, \mcD_i)}} \\
&\qquad = \Exp{\mcD_i, \xi_i}{\KL{p(\bfy | \bfx, \mcD_i, \xi_i)}{p(\bfy | \bfx, \mcD_i})}
\end{aligned}
\end{equation}

The inequality holds as $\entropy{p(\bfy | \bfx, \mcD_i, \{\mcD_j\}} \leq \entropy{p(\bfy | \bfx, \mcD_i)} \ \forall \{\mcD_j\}$, where $\entropy{\cdot}$ denotes the entropy. See <Ref> for a detailed proof.

Note that $p(\bfy_t | \bfx_t, \mcD_i, \{\mcD_j\}, \xi_i) = p(\bfy_t | \bfx_t, \mcD_i, \xi_i)$, as $\mcD_i$ and $\{\mcD_j\}$ are conditionally independent given $\xi_i$. Since we do not observe $\xi_i$, <Ref> tells us that we should target the predictive distribution $p(\bfy | \bfx, \mcD_i, \{\mcD_j\})$, instead of $p(\bfy | \bfx, \mcD_i)$. We refer to this form of learning as in-context in-context learning (ICICL), and it requires models that are able to condition on a set of datasets, in addition to individual datasets. We refer to the additional set of datasets as the in-context datasets, and shall denote this set as $\{\mcD_{ic, j}\}_{j=1}^{N_{ic}}$, where $N_{ic}$ is the number of in-context datasets, with $|\mcD_{ic, j}| = N_{ic, j}$.

§.§ In-Context In-Context Learning with Transformer Neural Processes

A diagram illustrating the model architecture with three in-context datasets. The point-wise embedding layer is used to get an initial token representation of all datapoints, including the target input location $\bfx_t$. Then for each layer of the processor, pseudo-token representations for each of the in-context datasets, $\bfU_{ic}$, and the context dataset, $\bfU$, are updated through mhca operations. The in-context pseudo-tokens $\bfU_{ic}$ are then modulated by the context pseudo-tokens $\bfU$, followed by mhsa operations on each set of pseudo-tokens. The in-context pseudo-tokens then modulate the context pseudo-tokens, and finally the context pseudo-tokens modulate the token representation of the target input, $\bfz_t$. After $L$ layers, the processor outputs the encoder representation $e(\bfx_t, \mcD_c, \{\mcD_{ic, j}\}_{j=1}^{j=N_{ic}})$.

Whilst current tnp architectures are capable of conditioning on individual datasets, they are not capable of conditioning on an arbitrary number of additional in-context datasets in order to approximate the predictive distribution $p(\bfy | \bfx, \mcD_i, \{\mcD_{ic, j}\}_{j=1}^{N_{ic}})$. Fortunately, the flexibility afforded by mhca layers makes this extension natural. As the total number of datapoints within each dataset is potentially large, we shall make use of the pseudo-token based architectures discussed in <Ref>. There exists a number of possible choices of architecture, but unless stated otherwise, we shall use the architecture illustrated in <Ref>, with a description of other choices of architecture provided in <Ref>. We refer to this family of pt-tnp as model.

At a high level, model perform the following set of operations: 1) construct initial token representations of each datapoint within each dataset using point-wise encodings; 2) construct a pseudo-token representation for each of the in-context datasets, $\bfU_{ic, j}$, and context dataset, $\bfU$; 3) perform cross-attention between the in-context pseudo-tokens and context pseudo-tokens. We note that this architecture forms a valid set function [Zaheer et al., 2017, Wagstaff et al., 2022] on the context datapoints, a valid set function on the set of in-context datasets, and a valid set function on the datapoints within each in-context dataset (see <Ref> for proof). The model has an asymptotic computational complexity $\order{MN_c + MN_t + \sum_{j=1}^{N_{ic}}\left(M_{ic}N_{ic, j} + MM_{ic} \right)}$, where $N_{ic}$ denotes the number of in-context datasets, $N_{ic, j}$ denotes the number of datapoints in the $j$-th in-context dataset, and $M_{ic}$ denotes the number of pseudo-tokens used for each in-context dataset. Importantly, this scales linearly in $N_c$, $N_t$, $N_{ic}$ and $N_{ic, j}$, enabling it to scale to both large datasets and many in-context datasets.

The model is trained in an analogous fashion to cnp, whereby the model parameters are optimised by maximising the predictive likelihood:

\begin{equation}
\mcL_{\text{ML}} = \Exp{p(\tau)}{\sum_{n=1}^{N_t}\log p(\bfy_{t, n} | d(\bfx_{t, n}, e(\bfx_{t, n}, \mcD_c, \{\mcD_{ic}\})}.
\end{equation}

Here, tasks $\tau = (\mcD_c, \{\mcD_{ic}\}, \mcD_t)$ contain additional in-context datasets, which are fed into the encoder.

§ RELATED WORK

Cross-Attention Based Architectures in NLP
Although not described as in-context in-context learning, similar architectures have been proposed for conditioning on additional datasets in NLP. Notably, the retrieval-enhanced transformer (RETRO) [Borgeaud et al., 2022] performs cross-attention between the token representations of additional `retrieved' texts, demonstrating an  improvement in performance over the original transformer architecture [Vaswani et al., 2017]. However, the architecture used in RETRO differs substantially from ours. Rather than conditioning on entire datasets, RETRO performs chunking of text to perform next chunk prediction, in which other chunks similar to the previous chunks are used to modulate the predictions. Further, they do not make use of pseudo-token-based transformer architectures to enable scaling to chunks containing more than 64 tokens.

Cross-Attention Based Architectures for Multi-Task Learning
A closely related family of transformer architectures are those used for multi-task learning [Zhang and Yan, 2022, Kim et al., 2021, Shen et al., 2024, Xu et al., 2022] and multi-modal learning [Jaegle et al., 2021, Xu et al., 2023]. An important difference between this style of architecture and that used for in-context in-context learning is that exchangeability of the in-context datasets is not required, nor desirable. This owes to the fact that multi-output and multi-modal datasets are not independent samples from the same underlying stochastic process. Rather, they can be understood as multiple outputs of the same sample from some stochastic process. The difference is subtle, and in practice is achieved through positional encodings for each output of multi-output data, and different tokenisations of each mode in multi-modal data.

Conditioning on Exchangeable Datasets in Causal ML Finally, the causal structure induction with a supervised approach (CSIvA) model from [Ke et al., 2022] also shares similarities with our approach. Similar to the model, the CSIvA conditions on multiple datasets in an exchangeable manner to make inference. The two models differ significantly elsewhere, however. Whereas we are interested in predictive distributions over outputs, [Ke et al., 2022] are interested in inferring causal structure from observational and interventional data. These differences materialise in their use of positional encoding to indicate the identity of the causal node of observations. In a sense, this model bears more resemblance to the multi-task learning architectures, with the additional ability to condition on many instances of samples from the stochastic process in a single forward pass.

§ EXPERIMENTS

In this section, we investigate the performance of the model. We seek to answer two questions: 1) can the model recover the performance of a regular pt-tnp when no in-context datasets are provided; 2) how does the performance of the model vary with the number of in-context datasets provided. In each experiment, we compare the performance of the model with a regular pt-tnp equivalent. We also provide results for a version of the cnp that supports ICICL (see <Ref> for details of the architecture) alongside the regular cnp. We provide more thorough experimental details in <Ref>, including the choice of model architecture and model training.

§.§ Synthetic Regression

We consider a synthetic 1-D regression task using samples drawn from Gaussian processes (GPs) with different kernel types and different kernel hyperparameters. First, we sample either a radial basis function (RBF) or periodic kernel. The kernel hyperparameter $\ell$—corresponding to the lengthscale and period for the RBF and periodic kernel, respectively—is sampled as $\log \ell \sim \mcU_{[\log 0.25, \log 4]}$. This is shared between the context and in-context datasets. For each context dataset, the number of context points is drawn according to $N_c \sim \mcU\{1, 64\}$, while the number of target points is set to $N_t=128$. The context and target inputs are sampled from $x_c \sim \mcU_{[-2, 2]}$ and $x_t \sim \mcU_{[-4, 4]}$. For each such context dataset, we sample $N_{ic}$ in-context datasets with $N_{ic} \sim \mcU\{0, 5\}$, where the number of datapoints for each in-context dataset is sampled as $N_{ic, j} \sim \mcU\{64, 128\}$. The in-context inputs are sampled according to $x_{ic} \sim \mcU_{[-4, 4]}$. The observation noise is set to 0.2, and the test set consist of $80,000$ datasets.

<Ref> shows that when no in-context information is available, the model achieves similar performance to that of the pt-tnp (within one standard deviation), and significantly outperforms the cnp. Conditioning on a single in-context dataset significantly improves the predictive performance of the model, with the improvements in performance plateauing as the number of in-context datasets is increased further. Importantly, these results demonstrate that 1) the model is able to recover the performance of the regular pt-tnp when no in-context datasets are provided; and 2) the model is able to perform in-context in-context learning effectively.

We compare the predictive distributions of the model and pt-tnp in <Ref>. With only 10 context points, the pt-tnp has insufficient information to infer the periodicity of the underlying stochastic process, so it fails to both extrapolate beyond or interpolate between the data effectively. In contrast, when an in-context dataset drawn from the same periodic kernel is provided, the model is able to accurately model the posterior predictive map of the ground-truth periodic stochastic process. Not only does this demonstrate the importance of in-context in-context learning, it also demonstrates the effectiveness of the model in realising it.

Additional experiments are provided in <Ref>, illustrating the effect on the predictions of the model of conditioning on in-context datasets drawn from a different stochastic process.

We also trained an ICICL-cnp model on this task, but we did not observe any benefits from in-context in-context learning. This is in contrast to what we observe using model on this task, as well as what we observe in <Ref>. Thus, we hypothesise that the failure of the ICICL-cnp model to perform in-context in-context learning on GP synthetic regression arises from its limited capacity relative to model and from the difficulty of the task.

Out-of-distribution (OOD) testing We also investigated the behaviour of the model, as compared to pt-tnp, when tested OOD–during testing we sampled the kernel hyperparameter according to $\log \ell \sim \mcU_{[\log 0.1, \log 0.25]  \cup [\log 4, \log 10]}$ for both the context and in-context datasets. <Ref> shows that in-context learning improves predictive performance even if at test time the samples come from stochastic processes the model has not been trained on. This is reflected in <Ref>, where the context datapoints come from a GP with a periodic kernel and $\ell = 6.08$. The model is better than the pt-tnp at capturing in its uncertainty the slowly-varying variations characteristic to this kernel. We show more examples in <Ref>.

Model         1lLog lik. ($\uparrow$)

CNP      $-0.812 \pm 0.005$

pt-tnp     $-0.598 \pm 0.005$

model (0)      $-0.607 \pm 0.005$

model (1)     $-0.499 \pm 0.005$

model (2)     $-0.474 \pm 0.005$

model (3)     $-0.469 \pm 0.005$

model (4)     $-0.467 \pm 0.005$

model (5)    $\mathbf{-0.466 \pm 0.005}$

tableComparison of the predictive performance (in terms of test log likelihood) between the cnp, pt-tnp, and the model with varying number of in-context datasets (indicated within brackets).

[Regular pt-tnp.][c]

The difference between the predictive distributions of the regular pt-tnp and the model when conditioning on three in-context datasets with 128 datapoints (not shown here).

Model         1lLog lik. ($\uparrow$)

CNP      $-0.880 \pm 0.006$

pt-tnp     $-0.798 \pm 0.007$

model (0)      $-0.783 \pm 0.007$

model (1)     $-0.721 \pm 0.006$

model (2)     $-0.702 \pm 0.006$

model (3)     $\mathbf{-0.700 \pm 0.006}$

tableComparison of the predictive performance (in terms of test log likelihood) when tested OOD between the cnp, pt-tnp, and the model with varying number of in-context datasets (indicated within brackets).

[Regular pt-tnp.][c]

The difference between the predictive distributions when tested OOD of the regular pt-tnp and the model when conditioning on three in-context datasets. The context datapoints come from a GP with a periodic kernel with $\ell = 6.08$.

§.§ Image Completion

We consider an image in-painting experiment using the MNIST dataset [LeCun et al., 1998]. Each MNIST image can be interpreted as spatial regression of pixel values $\bfy_n \in \R$ given a 2-D pixel location $\bfx_n \in \R^2$. We construct context datasets by sampling the number of pixels according to $N_c \sim \mcU\{N/100, N/5\}$, where $N=784$ denotes the total number of pixels in each image, and set $N_t = N - N_c$. For each context dataset, we sample in-context datasets from other images of the same label with the number of in-context datasets sampled as $N_{ic} \sim \mcU\{0, 3\}$, and sample the number of datapoints for each in-context dataset as $N_{ic, c} \sim \mcU\{N/100, N/2\}$. <Ref> compares the predictive performance of the regular pt-tnp with the model as $N_{ic}$ and $N_c / N$ vary. We also provide results for the ICICL-cnp and cnp.

A comparison between the predictive performance of the model, regular pt-tnp, ICICL-cnp and cnp as the proportion of context datapoints $N_c / N$ varies in the MNIST in-painting experiment.

A comparison between the predictive distribution of the model and the regular pt-tnp when conditioning on the context and in-context dataset shown.

[Regular pt-tnp.][c]

We observe that the model recovers the performance of the pt-tnp when conditioning on no in-context datasets, and that, as expected, the gains in performance of the model plateau with increasing $N_{ic}$. In <Ref>, we compare the predictive distributions of the regular tnp and model for a single dataset. As very few pixels are observed in the context dataset, the regular tnp is unconfident in its predictions, as the context dataset could correspond to datasets sampled from several MNIST digits. With the addition of an in-context dataset, the model is able to infer pixel values much closer to the ground truth. We provide additional comparisons in <Ref>, showcasing examples where the in-context dataset is drawn from a different stochastic process to the context dataset.

§.§ Environmental Data

In this experiment, we model a real-world environmental regression problem derived from ERA5 [Copernicus Climate Change Service, 2020]. We consider the surface air temperature across both space and time within central Europe (latitude / longitude range of $[42^{\circ},\ 53^{\circ}]$ / $[8^{\circ},\ 28^{\circ}]$). During training, datasets spanning 18 hours (one sample every six hours) and $5^{\circ}$ (with a resolution of $0.5^{\circ}$) across each axis are sampled. In-context datasets are obtained from the same region, but non-overlapping regions in time. Once the spatio-temporal location has been sampled, the number of context points is sampled as $N_c \sim \mcU\{N/100, N/3\}$, where $N = 300$ denotes the maximum number of measurements in a single dataset. The number of in-context datasets is sampled as $N_{ic} \sim \mcU\{0, 2\}$, with the number of datapoints within each in-context dataset sampled as $N_{ic, j} \sim \mcU\{N/100, N/5\}$. The number of target points is set to $N_t = N - N_c$. We train on data from the first six months of 2019, and test on the latter six months. <Ref> compares the predictive performance of the model with the regular pt-tnp. As with the previous two experiments, the model is able to recover the performance of the pt-tnp with no in-context datasets, and outperforms the pt-tnp when in-context datasets are provided.

Model     Test Log-Likelihood

pt-tnp     $1.15 \pm 0.01$

model (0)     $1.15 \pm 0.01$

model (1)     $1.18 \pm 0.01$

model (2)     $1.19 \pm 0.01$

Comparison of the test log-likelihood on the environmental data for the pt-tnp and model with varying number of in-context datasets (indicated within brackets).

§ CONCLUSION

We have introduced the paradigm of in-context in-context learning, in which multiple datasets are used for predictive inference on a dataset drawn from the same stochastic process. We developed a np for performing in-context in-context learning, the model, which utilises pseudo-token-based transformer architectures for the computationally efficient handling of both context and in-context datasets with potentially many datapoints in each. Through a set of synthetic and real-world experiments, we demonstrate the importance of in-context in-context learning and the effectiveness of the model in performing it. Further, we demonstrate that the model is able to recover the performance of an equivalent pt-tnp when no in-context datasets are provided. A requirement of in-context in-context learning generally is for practitioners to be able to identify datasets drawn from the same stochastic process, which in turn limits the applicability of the model. Nonetheless, we believe that by establishing the benefit of in-context in-context learning, new machine learning applications can be developed that utilise it. For example, one can envision developing image generation tools which can be provided with additional images that the user wishes the generated sample to be similar to. We are excited by the abundance of potential directions in which to take this research, and look forward to exploring them in future work.

CD is supported by the Cambridge Trust Scholarship. AW acknowledges support from a Turing AI fellowship under grant EP/V025279/1 and the Leverhulme Trust via CFI. RET is supported by gifts from Google, Amazon, ARM, Improbable and EPSRC grant EP/T005386/1.

[Borgeaud et al., 2022]
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al.
Improving language models by retrieving from trillions of tokens.
In International conference on machine learning, pages 2206–2240. PMLR, 2022.

[Copernicus Climate Change Service, 2020]
Copernicus Climate Change Service.
Near surface meteorological variables from 1979 to 2018 derived from bias-corrected reanalysis, 2020.

[Feng et al., 2022]
Leo Feng, Hossein Hajimirsadeghi, Yoshua Bengio, and Mohamed Osama Ahmed.
Latent bottlenecked attentive neural processes.
arXiv preprint arXiv:2211.08458, 2022.

[Foong et al., 2020]
Andrew Foong, Wessel Bruinsma, Jonathan Gordon, Yann Dubois, James Requeima, and Richard Turner.
Meta-learning stationary stochastic process prediction with convolutional neural processes.
Advances in Neural Information Processing Systems, 33:0 8284–8295, 2020.

[Garnelo et al., 2018]
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami.
Conditional neural processes.
In International conference on machine learning, pages 1704–1713. PMLR, 2018a.

[Garnelo et al., 2018]
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh.
Neural processes.
arXiv preprint arXiv:1807.01622, 2018b.

[Jaegle et al., 2021]
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira.
Perceiver: General perception with iterative attention.
In International conference on machine learning, pages 4651–4664. PMLR, 2021.

[Jha et al., 2022]
Saurav Jha, Dong Gong, Xuesong Wang, Richard E Turner, and Lina Yao.
The neural process family: Survey, applications and perspectives.
arXiv preprint arXiv:2209.00517, 2022.

[Ke et al., 2022]
Nan Rosemary Ke, Silvia Chiappa, Jane Wang, Anirudh Goyal, Jorg Bornschein, Melanie Rey, Theophane Weber, Matthew Botvinic, Michael Mozer, and Danilo Jimenez Rezende.
Learning to induce causal structure.
arXiv preprint arXiv:2204.04875, 2022.

[Kim et al., 2021]
Donggyun Kim, Seongwoong Cho, Wonkwang Lee, and Seunghoon Hong.
Multi-task neural processes.
arXiv preprint arXiv:2110.14953, 2021.

[Kim et al., 2019]
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh.
Attentive neural processes.
arXiv preprint arXiv:1901.05761, 2019.

[LeCun et al., 1998]
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.
Gradient-based learning applied to document recognition.
Proceedings of the IEEE, 860 (11):0 2278–2324, 1998.

[Lee et al., 2019]
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh.
Set transformer: A framework for attention-based permutation-invariant neural networks.
In International conference on machine learning, pages 3744–3753. PMLR, 2019.

[Loshchilov and Hutter, 2017]
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization.
arXiv preprint arXiv:1711.05101, 2017.

[Nguyen and Grover, 2022]
Tung Nguyen and Aditya Grover.
Transformer neural processes: Uncertainty-aware meta learning via sequence modeling.
arXiv preprint arXiv:2207.04179, 2022.

[Shen et al., 2024]
Jiayi Shen, Xiantong Zhen, Qi Wang, and Marcel Worring.
Episodic multi-task learning with heterogeneous neural processes.
Advances in Neural Information Processing Systems, 36, 2024.

[Vaswani et al., 2017]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
Advances in neural information processing systems, 30, 2017.

[Wagstaff et al., 2022]
Edward Wagstaff, Fabian B Fuchs, Martin Engelcke, Michael A Osborne, and Ingmar Posner.
Universal approximation of functions on sets.
The Journal of Machine Learning Research, 230 (1):0 6762–6817, 2022.

[Xu et al., 2023]
Peng Xu, Xiatian Zhu, and David A Clifton.
Multimodal learning with transformers: A survey.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.

[Xu et al., 2022]
Xiaogang Xu, Hengshuang Zhao, Vibhav Vineet, Ser-Nam Lim, and Antonio Torralba.
Mtformer: Multi-task learning via transformer and cross-task reasoning.
In European Conference on Computer Vision, pages 304–321. Springer, 2022.

[Zaheer et al., 2017]
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola.
Deep sets.
Advances in neural information processing systems, 30, 2017.

[Zhang and Yan, 2022]
Yunhao Zhang and Junchi Yan.
Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting.
In The eleventh international conference on learning representations, 2022.

§ PROOF OF THEOREM 1

In this section we give a detailed proof of <Ref>, which we repeat here for completeness.

Let $\xi_i \sim p(\xi)$, $\mcD_i, \{\mcD_j\} \sim P(\xi_i)$. Let $p(\bfy | \bfx, \mcD_i, \xi_i)$ be the marginal posterior distribution of $P(\xi_i)$ given $\mcD_i$, $p(\bfy | \bfx, \mcD_i, \{\mcD_j\})$ be the marginal posterior distribution of the stochastic process $P$ given $\mcD_i$ and $\{\mcD_j\}$, and $p(\bfy | \bfx, \mcD_i)$ be the marginal posterior distribution of the stochastic process $P$ given $\mcD_i$. Then,

\begin{equation*}
\Exp{\mcD_i, \{\mcD_j\}, \xi_i}{\KL{p(\bfy | \bfx, \mcD_i, \xi_i)}{p(\bfy | \bfx, \mcD_i, \{\mcD_j\})}} \leq \Exp{\mcD_i, \xi_i}{\KL{p(\bfy | \bfx, \mcD_i, \xi_i)}{p(\bfy | \bfx, \mcD_i)}}.
\end{equation*}

First, observe that both terms of the inequality can be split into two parts:

\begin{equation}
\label{eq:left-side}
\begin{aligned}
& \Exp{\mcD_i, \{\mcD_j\}, \xi_i}{\KL{p(\bfy | \bfx, \mcD_i, \xi_i)}{p(\bfy | \bfx, \mcD_i, \{\mcD_j\})}} \\
& = \Exp{\mcD_i, \{\mcD_j\}, \xi_i}{\int \underbrace{p(\bfy | \bfx, \mcD_i, \xi_i) \log p(\bfy | \bfx, \mcD_i, \xi_i)}_{\text{term 1}} d\bfy - \underbrace{p(\bfy | \bfx, \mcD_i, \xi_i) \log p(\bfy | \bfx, \mcD_i, \{\mcD_j\})}_{\text{term 2}} d\bfy}
\end{aligned}
\end{equation}

\begin{equation}
\label{eq:right-side}
\begin{aligned}
& \Exp{\mcD_i, \xi_i}{\KL{p(\bfy | \bfx, \mcD_i, \xi_i)}{p(\bfy | \bfx, \mcD_i)}} \\
& = \Exp{\mcD_i, \xi_i}{\int \underbrace{p(\bfy | \bfx, \mcD_i, \xi_i) \log p(\bfy | \bfx, \mcD_i, \xi_i)}_{\text{term 1}} d\bfy - \underbrace{p(\bfy | \bfx, \mcD_i, \xi_i) \log p(\bfy | \bfx, \mcD_i)}_{\text{term 2}} d\bfy}.
\end{aligned}
\end{equation}

We first show that term 1 is the same for both sides of the inequality:

\begin{align}
\label{eq:first_term}
\nonumber &\Exp{\mcD_i, \{\mcD_j\}, \xi_i}{\int p(\bfy | \bfx, \mcD_i, \xi_i) \log p(\bfy | \bfx, \mcD_i, \xi_i) d\bfy} \\
\nonumber & = \Exp{ \xi_i \sim p(\xi), \mcD_i \sim P(\xi_i), \{\mcD_j\} \sim P(\xi_i)}{\int p(\bfy | \bfx, \mcD_i, \xi_i) \log p_(\bfy | \bfx, \mcD_i, \xi_i) d\bfy} \\
\nonumber & = \Exp{ \xi_i \sim p(\xi), \mcD_i \sim P(\xi_i)}{\int p(\bfy | \bfx, \mcD_i, \xi_i) \log p_(\bfy | \bfx, \mcD_i, \xi_i) d\bfy} \\
\nonumber &= \Exp{\mcD_i, \xi_i}{\int p(\bfy | \bfx, \mcD_i, \xi_i) \log p(\bfy | \bfx, \mcD_i, \xi_i) d\bfy} \\
& = - \Exp{\mcD_i, \xi_i}{\entropy{p(\bfy | \bfx, \mcD_i, \xi_i)}}
\end{align}

where to go from the second to the third line we leveraged the fact that the term within the expectation does not depend on $\{\mcD_j\}$, and can, hence, be integrated out.
Thus, the elements of the inequality only differ in term 2. Consider term 2 in <Ref>:

\begin{align*}
&\Exp{\mcD_i, \{\mcD_j\}, \xi_i}{\int p(\bfy | \bfx, \mcD_i, \xi_i) \log p(\bfy | \bfx, \mcD_i, \{\mcD_j\}) d\bfy}  \\
& =  \Exp{\mcD_i, \{\mcD_j\}}
{\Exp{\xi_i \sim p(\xi | \mcD_i, \{ \mcD_j \})}{\int p(\bfy | \bfx, \mcD_i, \xi_i) \log p(\bfy | \bfx, \mcD_i, \{\mcD_j\}) d\bfy}}
\end{align*}

Note that $p(\bfy | \bfx, \mcD_i, \xi_i) = p(\bfy | \bfx, \mcD_i, \{\mcD_j\}, \xi_i)$—considering that $\mcD_i$ and $\{\mcD_j\}$ are conditionally independent given $\xi_i$. Informally, this means that additionally conditioning on the in-context datasets $\{\mcD_j\}$ does not give us any extra information, given that we are already considering the marginal posterior of $P(\xi_i)$. Hence,

\begin{align}
\label{eq:second_term1}
\nonumber &\Exp{\mcD_i, \{\mcD_j\}, \xi_i}{\int p(\bfy | \bfx, \mcD_i, \xi_i) \log p(\bfy | \bfx, \mcD_i, \{\mcD_j\}) d\bfy} \\
\nonumber & = \Exp{\mcD_i, \{\mcD_j\}}
{\Exp{\xi_i \sim p(\xi | \mcD_i, \{\mcD_j\})}{\int p(\bfy | \bfx, \mcD_i, \{\mcD_j\}, \xi_i) \log p(\bfy | \bfx, \mcD_i, \{\mcD_j\}) d\bfy}} \\
\nonumber & = \Exp{\mcD_i, \{\mcD_j\}}
{\int p(\bfy | \bfx, \mcD_i, \{\mcD_j\}) \log p(\bfy | \bfx, \mcD_i, \{\mcD_j\}) d\bfy}\\
\nonumber & = - \Exp{\mcD_i, \{\mcD_j\}}{\entropy{p(\bfy | \bfx, \mcD_i, \{\mcD_j\})}} \\
& = - \Exp{\mcD_i}{\Exp{\{\mcD_j\} | \mcD_i}{\entropy{p(\bfy | \bfx, \mcD_i, \{\mcD_j\})}}}
\end{align}

where we used Fubini's theorem to go from the second to the third line. Similarly, for term 2 of <Ref> we have:

\begin{align}
\label{eq:second_term2}
\nonumber &\Exp{\mcD_i, \xi_i}{\int p(\bfy | \bfx, \mcD_i, \xi_i) \log p(\bfy | \bfx, \mcD_i) d\bfy} \\
\nonumber & = \Exp{\mcD_i}
{\Exp{\xi_i \sim p(\xi | \mcD_i)}{\int p(\bfy | \bfx, \mcD_i, \xi_i) \log p(\bfy | \bfx, \mcD_i) d\bfy}} \\
\nonumber & = \Exp{\mcD_i}
{\int p(\bfy | \bfx, \mcD_i) \log p(\bfy | \bfx, \mcD_i) d\bfy}\\
& = - \Exp{\mcD_i}{\entropy{p(\bfy | \bfx, \mcD_i)}}.
\end{align}

Putting the results from <Ref> and <Ref> together we obtain

\begin{align}
&\Exp{\mcD_i, \{\mcD_j\}, \xi_i}{\KL{p(\bfy | \bfx, \mcD_i, \{\mcD_j\}, \xi_i)}{p(\bfy | \bfx, \mcD_i)}} \\
& = - \Exp{\mcD_i, \xi_i}{\entropy{p(\bfy | \bfx, \mcD_i, \xi_i)}} + \Exp{\mcD_i}{\Exp{\{\mcD_j\} | \mcD_i}{\entropy{p(\bfy | \bfx, \mcD_i, \{\mcD_j\})}}},
\end{align}

and putting <Ref> and <Ref> together we have

\begin{align}
&\Exp{\mcD_i, \xi_i}{\KL{p(\bfy | \bfx, \mcD_i, \xi_i)}{p(\bfy | \bfx, \mcD_i)}} \\
& = - \Exp{\mcD_i, \xi_i}{\entropy{p(\bfy | \bfx, \mcD_i, \xi_i)}} + \Exp{\mcD_i}{\entropy{p(\bfy | \bfx, \mcD_i)}}.
\end{align}

Since conditioning reduces entropy $\entropy{p(\bfy | \bfx, \mcD_i, \{\mcD_j\}} \leq \entropy{p(\bfy | \bfx, \mcD_i)} \ \forall \{\mcD_j\} \sim P(\xi_i)$, we have

\begin{equation*}
\resizebox{\linewidth}{!}{
\Exp{\mcD_i}{\Exp{\{\mcD_j\} | \mcD_i}{\entropy{p(\bfy | \bfx, \mcD_i, \{\mcD_j\})}}} \leq \Exp{\mcD_i}{\Exp{\{\mcD_j\} | \mcD_i}{\entropy{p(\bfy | \bfx, \mcD_i)}}} = \Exp{\mcD_i}{\entropy{p(\bfy | \bfx, \mcD_i)}},
\end{equation*}

which leads to the result in <Ref>.

§ DEFINING THE MHSA AND MHCA OPERATIONS

Let $\bfZ^{\ell} \in \R^{N\times D_z}$ denote the input set to the $\ell$-th MHSA operation. The MHSA operation updates the $n$th token $\bfz^{\ell}_n$ as

\begin{equation}\textstyle
\label{eq:self-attention}
\!\!\tilde{\bfz}^{\ell}_n \!=\! \operatorname{cat}\!\Big(\Big\{\sum_{m=1}^N \alpha^{\ell}_h(\bfz^{\ell}_n, \bfz^{\ell}_m) {\bfz^{\ell}_m\!}^T \bfW^{\ell}_{V, h}\Big\}_{h=1}^{H^{\ell}}\Big)\bfW^{\ell}_O
\end{equation}

where $\operatorname{cat}$ denotes the concatenation operation across the last dimension. Here, $\bfW^{\ell}_{V, h} \in \R^{D_z\times D_V}$ and $\bfW^{\ell}_{O} \in \R^{H^{\ell}D_V \times D_z}$ are the value and projection weight matrices, where $H^{\ell}$ denotes the number of `heads' in layer $\ell$. Note that permutation equivariance is achieved through the permutation invariant summation operator. As this is the only mechanism through which the tokens interact with each other, permutation equivariance for the overall model is ensured. The attention mechanism, $\alpha^{\ell}_h$, is implemented as

\begin{equation}
\label{eq:attention-mechanism}
\alpha^{\ell}_h(\bfz^{\ell}_n, \bfz^{\ell}_m) = \frac{e^{{\bfz^{\ell}_n}^T\bfW^{\ell}_{Q, h}\left[\bfW^{\ell}_{K, h}\right]^T\bfz^{\ell}_m}}{\sum_{m=1}^N e^{{\bfz^{\ell}_n}^T\bfW^{\ell}_{Q, h}\left[\bfW^{\ell}_{K, h}\right]^T\bfz^{\ell}_m}}
\end{equation}

where $\bfW^{\ell}_{Q, h} \in \R^{D_z \times D_{QK}}$ and $\bfW^{\ell}_{K, h} \in \R^{D_z\times D_{QK}}$ are the query and key weight matrices. The softmax-normalisation ensures that $\sum_{m=1}^N \alpha^{\ell}_h(\bfz^{\ell}_n, \bfz^{\ell}_m) = 1 \ \forall n, h, \ell$.

Often, conditional independencies amongst the set of tokens—in the sense that the set $\{\bfz^{\ell}_n\}^{\ell=L}_{\ell=1}$ do not depend on the set $\{\bfz^{\ell}_m\}^{\ell=L}_{\ell=0}$ for some $n,\ m \in \{1, \ldots, N\}$—are desirable. This is typically achieved through masking, such that the pre-softmax activations are replaced by $\tilde{\alpha}^{\ell}_h$, where

\begin{equation}
\tilde{\alpha}^{\ell}_h(\bfz^{\ell}_n, \bfz^{\ell}_m) = \begin{cases}
-\infty, & \text{$m\in A(n)$.} \\
{\bfz^{\ell}_n}^T\bfW^{\ell}_{Q, h}\left[\bfW^{\ell}_{K, h}\right]^T\bfz^{\ell}_m, & \text{otherwise.}
\end{cases}
\end{equation}

Here, $A(n) \subseteq \N_{\leq N}$ indexes the set of tokens we wish to make the update for token $\bfz^{\ell}_n$ independent of. If $A(n) = A$ (i.e. the same set of tokens are conditioned on for every $n$) then in practice it is more computationally efficient to use MHCA operations together with MHSA operations than it is to directly compute <Ref>. An MHCA operation uses the subset of tokens $\{\bfz^{\ell}_m | m\in A\}$ to update the complementary set of tokens $\{\bfz^{\ell}_n | n\in A^c\}$ in a computationally efficient manner:

\begin{equation}\textstyle
\label{eq:cross-attention}
\!\!\tilde{\bfz}^{\ell}_n \!=\! \operatorname{cat}\!\Big(\Big\{\sum_{m\in A} \alpha^{\ell}_h(\bfz^{\ell}_n, \bfz^{\ell}_m) {\bfz^{\ell}_m\!}^T \bfW^{\ell}_{V, h}\Big\}_{h=1}^{H^{\ell}}\Big)\bfW^{\ell}_O \qquad \forall n \in A^c.
\end{equation}

For $N$ tokens that solely depend on a subset of $N_1$ tokens, the computational complexity is reduced from $\order{N^2}$ using masked-mhsa operations to $\order{NN_1}$ using mhca operations.

§ SOME TNP ARCHITECTURES

There exist a number of architectures used in different members of the tnp. We provide diagramatic illustrations of the following: the anp of [Kim et al., 2019] in <Ref>; the tnp of [Nguyen and Grover, 2022] in <Ref>; the lbanp of [Feng et al., 2022] in <Ref>; and the ist-style [Lee et al., 2019] tnp in <Ref>.

A diagram illustrating the architecture of the anp [Kim et al., 2019].

A diagram illustrating the architecture of the tnp [Nguyen and Grover, 2022].

A diagram illustrating the architecture of the lbanp [Feng et al., 2022].

A diagram illustrating the architecture of the ist-style tnp [Lee et al., 2019].

§ SET FUNCTIONS AND SET OF SETS FUNCTIONS

In this section we utilise the Deepset result of [Zaheer et al., 2017], which states that any permutation invariant function on the set $\{z_n\}_{n=1}^N$ can be expressed as

\begin{equation}
f(z_1, z_2, \ldots, z_N) = \rho\left(\sum_{n=1}^N \phi(z_n)\right)
\end{equation}

for continuous functions $\rho$ and $\phi$. As shown in [Lee et al., 2019], the Deepset formulation is a special case of pseudo-token based transformers operating on sets. It is therefore sufficient to consider only this Deepset formulation.

Let $N_{ic}$ be the number of in-context datasets, each with $N_{1}, N_{2}, \dots, N_{N_{ic}}$ datapoints. Each in-context dataset is embedded into a latent representation through a valid set function of the form

$$f(z_{n, 1}, z_{n, 2}, \dots, z_{n, N_n}) = \rho_{n}\bigg(\sum_{j=1}^{N_n}\phi_{n}(z_{n, j})\bigg)$$

for all $n \in \{1, 2, \dots, N_{ic}\}$.
When combining the representation of all of the in-context datasets, we also obtain a valid set function

\begin{align*}
& f\Bigg(\{z_{1, 1}, z_{1, 2}, \dots, z_{1, N_1}\}, \dots, \{z_{N_{ic}, 1}, z_{N_{ic}, 2}, \dots, z_{N_{ic}, N_{N_{ic}}}\}\Bigg) = \\
& f\Bigg(\rho_1\bigg(\sum_{j=1}^{N_1}\phi_1(z_{1, j})\bigg), \dots, \rho_{N_{ic}}\bigg(\sum_{j=1}^{N_{N_{ic}}}\phi_{N_{ic}}(z_{N_{ic}, j})\bigg)\Bigg) = \\
& \rho\Bigg( \sum_{n=1}^{N_{ic}} \phi \bigg( \rho_n\big(\sum_{j=1}^{N_n}\phi_n(z_{n, j})\big)\bigg)\Bigg).
\end{align*}

Thus, the latent representation of the in-context datasets is a valid set function on the set of in-context datapoints.

§ OTHER CHOICES OF ARCHITECTURE FOR THE MODEL

As discussed in <Ref>, there exist a number of possible ways to construct the model. In <Ref>, we provide an alternative architecture for the model than that shown in <Ref>. We also evaluated the performance of this architecture, and generally found little difference in performance.

A diagram illustrating the a possible choice for the architecture of the model.

§ ICICL-CNP

It is possible to extend the capabilities of the cnp to perform in-context in-context learning. In addition to obtaining a latent representation of the context dataset, we obtain latent representations of the in-context datasets in an identical manner (i.e. using a Deepset [Zaheer et al., 2017]). These are aggregated, and the result is then aggregated with the context latent representation to get a latent representation of both the context dataset and in-context datasets. This is then aggregated with the latent representation of the target input, which is then fed into the decoder. We provide a diagrammatic illustration of this in <Ref>.

A diagram illustrating the ICICL-cnp architecture with three in-context datasets. The point-wise embedding layer is used to get an initial representation of all datapoints, including the target input location $\bfx_t$. Then, for each dataset, its latent representation is obtained using the standard Deepset approach [Zaheer et al., 2017]. The latent representations for the in-context datasets are combined, and then aggregated with the latent representation for the context dataset. The resultant latent representation is aggregated with the target representation, and the processor outputs the encoder representation $e(\bfx_t, \mcD_c, \{\mcD_{ic, j}\}_{j=1}^{j=N_{ic}})$.

§ EXPERIMENT DETAILS

For all models, we use an embedding / token size of $D_z = 128$, and point-wise encoder and decoder consisting of an MLP with two hidden layers of dimension $D_z$. The decoder parameterises the mean and pre-softplus variance of a Gaussian likelihood with heterogeneous noise.

All transformer-based models use five layers of operations, with both the mhca and mhsa layers using $H = 8$ heads of dimension $D_V = D_{QK} = 16$. In each of the attention blocks, we apply a residual connection consisting of layer-normalisation to the input tokens followed by the attention mechanism. Following this, there is another residual connection consisting of layer-normalisation followed by a point-wise MLP with two hidden layers of dimension $D_z$. Initial pseudo-token values are sampled from a standard normal distribution.

All cnp models use Deepsets consisting of MLPs with five layers of dimension $D_z$ for the point-wise embedding. The latent representations for each dataset are obtained by aggregating (mean) the point-wise embeddings for all the datapoints contained in that dataset. The in-context latent representation is obtained by aggregating (mean) the latent representations for each in-context dataset. The in-context latent representation and context latent representation are aggregated through concatenation, being then concatenated again with the representation for the target location. This is then passed through a decoder, consisting of two hidden layers of dimension $D_z$.

We optimise the model parameters using AdamW [Loshchilov and Hutter, 2017] with a learning rate of $5\times 10^{-4}$ and batch size of 16. We apply clipping to gradients with magnitudes greater than 0.5.

§.§ Synthetic 1-D Regression

For each dataset and in-context datasets, we first randomly sample a kernel between RBF and periodic. Then, we sample the kernel's hyperparameter $\ell$ - length-scale in the case of the RBF kernel and period in the case of the periodic kernel. This is sampled according to $\log \ell \sim \mcU_{[\log 0.25, \log 4]}$. In terms of the task, we sample  the number of context points $N_c \sim \mcU\{1, 64\}$, the number of in-context datasets $N_{ic} \sim \mcU\{0, 5\}$, the number of points in each in-context dataset $N_{ic, c} \sim \mcU\{64, 128\}$, the context inputs $x_{c, n} \sim \mcU_{[-2, 2]}$, the target inputs $x_{t, n} \sim \mcU_{[-4, 4]}$, and the inputs for the in-context datasets $x_{ic, j, n} \sim \mcU_{[-4, 4]}$. All tasks use the same number of target points $N_t = 128$. The observations for each task are drawn from a GP with kernel

\begin{equation}
k_{\text{obs}} = k + \sigma_n^2 \delta(x - x')
\end{equation}

where the observation noise $\sigma_n = 0.2$.

For the model, we use $M = 32$ context pseudo-tokens and $M_{ic} = 32$ in-context pseudo-tokens for each in-context dataset. For the pt-tnp, we use $M = 32$ pseudo-tokens. Each model is trained for 1,000 epochs, with each epoch consisting of 1,000 iterations. We evaluate the performance of each model on 80,000 test datasets.

§.§.§ Additional Results

We also study the behaviour of the model when conditioning on in-context datasets that are drawn from a different stochastic process as compared to the context dataset. In particular, in the example from <ref>, the GP the context datapoints are drawn from has a periodic kernel with $\ell = 1.85$. In this experiment, we investigate the predictive distribution in five scenarios:

* No in-context datasets;

* Three in-context datasets that are drawn from the same stochastic process as the context datapoints;

* Three in-context datasets where the datapoints are generated from a GP with an RBF, instead of a periodic kernel (with $\ell = 0.52$);

* Three in-context datasets where the datapoints are generated from a GP with a periodic kernel and a similar (but not identical) period $\ell = 1.76$;

* Three in-context datasets where the datapoints are generated from a GP with a periodic kernel and a significantly different period $\ell = 0.28$.

The resulting predictive distributions are shown in <Ref>. As already noted in the main text, when conditioning on in-context data drawn from the same stochastic process, as opposed to no in-context information, the predictions improve significantly and the uncertainty decreases. When the in-context and context datapoints are drawn from a GP with a different kernel (RBF vs. periodic), the predictions of the model are more uncertain, especially far away from the data, but the model still manages to fit the data well in regions with high context datapoints concentration (<Ref>). When the in-context and context datapoints are drawn from a GP with the same type of kernel and a similar period, the predictions improve significantly as compared to when no in-context information is available (<Ref>). This suggests that even if the two stochastic processes are not exactly the same, but with similar characteristics, the model is still able to extract useful information from the in-context data. In contrast, when the two stochastic processes have wildly different characteristic (i.e. period), the model is unable to explain the data (<Ref>), leading to a large increase in uncertainty in all regions. We note that this could be used as a way to identify whether the in-context and context datasets are indeed generated by the same stochastic process, with the model outputting uncertain predictions when there is a severe discrepancy.

[No in-context datasets][c]

[Periodic kernel, $\ell = 1.85$][c]

[RBF kernel, $\ell = 0.52$][c]

[Periodic kernel, $\ell = 1.76$][c]

[Periodic kernel, $\ell = 0.28$][c]

A comparison between the predictive distribution of the model with different in-context conditioning information. The GP kernel used to generate the context datapoints was periodic with $\ell = 1.85$.

§.§.§ Results for alternative architecture

We also provide the results on this task for the alternative architecture shown in <Ref>. We did not observe a significant difference between this model (model2) and model on this task, as illustrated in <Ref>.

$\boldsymbol{N_{ic}}$     model     model2
0         $-0.607  \pm 0.005 $     $-0.612 \pm 0.005 $

1         $-0.499 \pm 0.005 $     $-0.491 \pm 0.005 $

2         $-0.474  \pm 0.005 $      $-0.473 \pm 0.005 $

3         $-0.469 \pm 0.005 $      $-0.467 \pm 0.005 $

4         $-0.467  \pm 0.005 $      $-0.464 \pm 0.005 $

5         $-0.466  \pm 0.005 $      $-0.463 \pm 0.005 $

Comparison of test log likelihood for the model and the model2 architectures on the synthetic GP task for a varying number of in-context datasets ($N_{ic}$). The two models have similar performance (within one standard deviation).

§.§.§ Results for OOD testing

We present some additional examples of predictions on samples that are drawn outside of the distribution the models have been trained on. In the top row of <Ref> we consider samples drawn from a GP with RBF kernel and $\ell = 5.19$, whereas in the bottom row we consider the challenging task of modelling datapoints drawn from a GP with a periodic kernel and $\ell = 0.24$. In both cases the model is conditioned on three in-context datasets, containing datapoints satisfying the same stochastic process as the context data. The model manages to leverage in-context learning to improve its predictions even when the samples are drawn from a different distribution than the one the model has been trained on.

[pt-tnp, RBF $\ell=5.19$][c]

[model, RBF $\ell=5.19$][c]

[pt-tnp, Periodic $\ell = 0.24$][c]

[model, Periodic $\ell = 0.24$][c]

A comparison between the predictive distribution when tested OOD of the pt-tnp (left) and the model (right). In the top row the samples come from a GP with RBF kernel and $\ell = 5.19$. In the bottom row the samples come from a GP with periodic kernel and $\ell = 0.24$. The model is conditioned on three in-context datasets.

§.§ Image Completion

For each dataset and in-context datasets, we first sample a label $\ell \sim \mcU\{0, 9\}$, and then sample images from the subset of MNIST images that correspond to that label. The number of context points is sampled as $N_c \sim \mcU\{N/100, N/5\}$, the number of in-context datasets is sampled as $N_{ic} \sim \mcU\{0, 3\}$, and the number of datapoints for each of the in-context datasets is sampled as $N_{ic, j} \sim \mcU\{N/100, N/2\}$. The number of target points is set to the remaining pixels in the image (i.e. $N_t = N - N_c$).

For the model, we use $M = 64$ context pseudo-tokens and $M_{ic} = 64$ in-context pseudo-tokens for each in-context dataset. For the pt-tnp, we use $M = 64$ pseudo-tokens. Each model is trained for 500 epochs, with each epoch consisting of 625 iterations with a batch size of 16. We evaluate the performance of each model on 2,500 images (the test dataset split into groups of four, so that each group has one context dataset and three in-context datasets, so that a maximum of three is available for use).

§.§.§ Additional Results

In <Ref> we compare the predictive distribution of the model when conditioning on in-context datasets drawn from different stochastic processes (MNIST labels).

A comparison between the predictive distribution of the model when conditioning on in-context datasets sampled from different MNIST labels. Observe that when the context set is not compatible with the label of the in-context dataset (i.e. 4 and 2), the predictive distribution is less confident.

[In-context dataset label: 5][c]

[In-context dataset label: 1][c]

[In-context dataset label: 4][c]

[In-context dataset label: 2][c]

§.§ Environmental Data

The environmental dataset consists of surface air temperatures derived from the fifth generation of the European Centre for Medium-Range Weather Forecasts (ECMWF) atmospheric reanalyses (ERA5) [Copernicus Climate Change Service, 2020]. The data has a latitudinal and longitudinal resolution of $0.5^\circ$, and temporal resolution of an hour. We consider data collected in 2019, sub-sampled at a temporal resolution of six hours. The dataset consists of data within the latitude / longitude range of $[42^{\circ},\ 53^{\circ}]$ / $[8^{\circ},\ 28^{\circ}]$ (roughly corresponding to central Europe), with the training data corresponding to the first six months of 2019, and the test data corresponding to the last six month. Individual datasets are obtained by sub-sampling the larger region, with each dataset consists of a $[10, 10, 3]$ grid spanning $5^\circ$ across each axis and 18 hours. We also provide surface elevation as additional inputs, such that $D_x = 4$. The inputs and outputs are standardised using the mean and standard deviation values obtained from data within the training region. Each dataset consists of a maximum of $N = 300$ datapoints, from which the number of context points are sampled according to $N_c \sim \mcU\{N/100, N/3\}$, with the remaining set as target points. The number of in-context datasets is sampled as $N_{ic} \sim \mcU\{0, 2\}$, and the number of datapoints in each of the in-context datasets is sampled as $N_{ic, j} \sim \mcU\{N/3, N\}$.

For the model and pt-tnp, we use $M=32$ context pseudo-tokens and $M_{ic} = 32$ in-context pseudo-tokens for each in-context dataset. We train each model for 500 epochs, with each epoch consisting of 1,000 iterations with a batch size of 16.