# Aligning brain functions boosts the decoding of visual semantics in novel
subjects

Alexis Thual
Cognitive Neuroimaging Unit, INSERM, CEA, CNRS, NeuroSpin center, Gif sur
Yvette, France
Mind, Inria Paris-Saclay, Palaiseau, France
<EMAIL_ADDRESS>
&Yohann Benchetrit
FAIR, Meta
<EMAIL_ADDRESS>&Félix Geilert
FAIR, Meta
<EMAIL_ADDRESS>&Jérémy Rapin
FAIR, Meta
<EMAIL_ADDRESS>&Iurii Makarov
FAIR, Meta
<EMAIL_ADDRESS>&Hubert Banville
FAIR, Meta
<EMAIL_ADDRESS>&Jean-Rémi King
FAIR, Meta
Laboratoire des systèmes perceptifs, École normale supérieure
PSL University
<EMAIL_ADDRESS>Work done while interning at FAIR, Meta

###### Abstract

Deep learning is leading to major advances in the realm of brain decoding from
functional Magnetic Resonance Imaging (fMRI). However, the large inter-subject
variability in brain characteristics has limited most studies to train models
on one subject at a time. Consequently, this approach hampers the training of
deep learning models, which typically requires very large datasets. Here, we
propose to boost brain decoding by aligning brain responses to videos and
static images across subjects. Compared to the anatomically-aligned baseline,
our method improves out-of-subject decoding performance by up to 75%.
Moreover, it also outperforms classical single-subject approaches when fewer
than 100 minutes of data is available for the tested subject. Furthermore, we
propose a new multi-subject alignment method, which obtains comparable results
to that of classical single-subject approaches while improving out-of-subject
generalization. Finally, we show that this method aligns neural
representations in accordance with brain anatomy. Overall, this study lays the
foundations for leveraging extensive neuroimaging datasets and enhancing the
decoding of individuals with a limited amount of brain recordings.

Figure 1: General outline of video decoding from BOLD fMRI signal in left-out
subjects
A. For every image associated with a brain volume, one computes its low-level
and high-level latent representations using pre-trained models. Subsequently,
regression models can be fitted to map brain features onto each of these
latent representations. B. BOLD signal acquired in two subjects watching the
same movie can be used to derive an alignment model which associates voxels of
the two subjects based on functional similarity. C. Once this alignment model
is trained, it can be used to transform brain features of the left-out subject
into brain features that resemble that of the reference subject. In
particular, this allows one to use models that have been trained on a lot of
data coming from a reference subject data, and apply it on a left-out subject
for whom less data was collected.

##  1 Introduction

##### Decoding the brain

Deep learning is greatly accelerating the possibility of decoding mental
representations from brain activity. Originally restricted to linear models
(Mitchell et al., 2004; Harrison & Tong, 2009; Haynes & Rees, 2006), the
decoding of brain activity can now be carried out with deep learning
techniques. In particular, using functional Magnetic Resonance Imaging (fMRI)
signals, significant progress has been made in the decoding of images (Ozcelik
& VanRullen, 2023; Chen et al., 2023a; Scotti et al., 2023; Takagi &
Nishimoto, 2023; Gu et al., 2023; Ferrante et al., 2023; Mai & Zhang, 2023),
speech (Tang et al., 2023), and videos (Kupershmidt et al., 2022; Wen et al.,
2018; Wang et al., 2022; Chen et al., 2023b; Lahner et al., 2023; Phillips et
al., 2022).

##### Challenge

However, brain representations are highly variable across subjects, which
makes it challenging to train a single model on multiple subjects using fMRI
data. Therefore, with few noteworthy exceptions (Haxby et al., 2020; Ho et
al., 2023), studies typically train a decoder on a single subject at a time.
With this constraint in mind, major effort has been put towards building fMRI
datasets collecting a lot of data in a limited number of participants (Allen
et al., 2022; Wen et al., 2017; LeBel et al., 2023; Pinho et al., 2018).
Nonetheless, the necessity to train and test models on a single subject
constitutes a major impediment to using notoriously data-hungry deep learning
approaches.

##### Functional alignment

Several methods can align the functions – as opposed to the anatomy – of
multiple brains, and thus offer a potential solution to inter-subject
variability: differentiable wrappings of the cortical surface (Robinson et
al., 2014), rotations between brain voxels in the functional space (Haxby et
al., 2011), shared response models (Chen et al., 2015; Richard et al., 2020),
permutations of voxels minimizing an optimal transport cost (Bazeille et al.,
2019), or combinations of these approaches (Feilong et al., 2022). However, it
is not clear which of these methods offers the best performance and
generalization capabilities (Bazeille et al., 2021). Besides, several studies
rely on deep learning models trained in a self-supervised fashion to obtain a
useful embedding of brain activity, in hope that this embedding could be
meaningful across subjects (Thomas et al., 2022; Chen et al., 2023a). However,
it is currently unknown whether any of these methods improve the decoding of
naturalistic stimuli such as videos, and how such hypothetical gain would vary
with the amount of fMRI recording available in a given a subject.

##### Approach

To address this issue, we leverage fMRI recordings of multiple subjects to
boost the decoding of videos and static images in a single left-out subject.
This requires fitting two models: an alignment model and a decoder. The
alignment aims at making brain responses of a left-out subject most similar to
those of a reference subject. Here, we leverage optimal transport to compute
this transformation using functional and anatomical data from both subjects.
The decoder consists of a linear regression trained to predict the latent
representations of movie frames or static images from the corresponding BOLD
signals or beta coefficients.
We evaluate video and image decoding in different setups. In particular, we
assess (1) whether training a decoder with several subjects improves
performance, (2) whether decoders generalize to subjects on which they were
not trained and (3) the extent to which functional alignment improves
aforementioned setups.

##### Contributions

We first confirm the feasibility of decoding, from 3T fMRI, the semantics of
videos watched by the subjects (Wen et al., 2017). We then confirm this
approach with the decoding of static images from 7T fMRI (Allen et al., 2022).
Our study further makes three novel contributions:

  1. 1.

functional alignment across subjects boosts visual semantics decoding
performance when left-out subjects have a limited amount of data

  2. 2.

training a decoder on multiple aligned subjects reaches the same performance
as training a single model per subject

  3. 3.

the resulting alignments, computed from movie watching data, are anatomically
coherent

##  2 Methods

Our goal is to decode visual stimuli seen by subjects from their brain
activity. To this end, we train a linear model to predict latent
representations – shortened as latents – of these visual stimuli from BOLD
fMRI signals recorded in subjects watching naturalistic videos.

In the considered data, brains are typically imaged at a rate of one scan
every 2 seconds. During this period, a subject sees 60 video frames on
average, or a static image for the case of Allen et al. (2022). For
simplicity, we consider the restricted problem of decoding only the first
video frame seen by subjects at each brain scan. Formally, regardless of the
dataset, for a given subject, let ${\bm{X}}\in\mathbb{R}^{n,v}$ be the BOLD
response collected in $v$ voxels over $n$ brain scans and
${\bm{Y}}\in\mathbb{R}^{n,m}$ the $m$-dimensional latent representation of
each selected video frame for all $n$ brain scans.

###  2.1 Brain alignment

##### Anatomical alignment

As a baseline, we consider the alignment method implemented in Freesurfer
(Fischl, 2012), which relies on anatomical information to project each subject
onto a surface template of the cortex (in our case fsaverage5). Consequently,
brain data from all subjects lie on a mesh of size $v=10\,242$ vertices per
hemisphere.

##### Functional alignment

On top of the aforementioned anatomical alignment, we apply a recent method
from Thual et al. (2022) denoted as Fused Unbalanced Gromov-Wasserstein (FUGW)
111https://alexisthual.github.io/fugw. As illustrated in Figure 1.B, this
method consists in using functional data to train an alignment that transforms
brain responses of a given left-out subject into the brain responses of a
reference subject. This approach can be seen as a soft permutation of voxels
222We use the words voxel (volumetric pixel) or vertex (point on a mesh)
indifferently. of the left-out subject which maximizes the functional
similarity to voxels of the reference subject.

Formally, for a left-out subject, let
${\bm{D}}^{\text{out}}\in\mathbb{R}^{v,v}$ be the matrix of anatomical
distances between vertices on the cortex, and
${\bm{w}}^{\text{out}}\in\mathbb{R}_{+}^{v}$ a probability distribution on
vertices. ${\bm{w}}^{\text{out}}$ can be interpreted as the relative
importance of vertices ; without prior knowledge, we use the uniform
distribution. Reciprocally, we define ${\bm{D}}^{\text{ref}}$ and
${\bm{w}}^{\text{ref}}$ for a reference subject. Note that, in the general
case, $v$ can be different from one subject to the other, although we simplify
notations here.

We derive a transport plan ${\bm{P}}\in\mathbb{R}^{v,v}$ to match the vertices
of the two subjects based on functional similarity, while preserving
anatomical organisation. For this, we simultaneously optimize multiple
constraints, formulated in the loss function $\mathcal{L}({\bm{P}})$ described
in Equation 1:

$\begin{split}\mathcal{L}(\bm{P})\enspace\triangleq\enspace&(1-\alpha)\enspace\mathchoice{\leavevmode\hbox
to101.99pt{\vbox to19.3pt{\pgfpicture\makeatletter\hbox{\hskip
50.99614pt\lower-10.00012pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-50.99614pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\displaystyle\hbox{\pagecolor{blue!10}
$\sum\limits_{\begin{subarray}{c}0\leq
i,j<n\end{subarray}}||{\bm{X}}^{\text{out}}_{i}-{\bm{X}}^{\text{ref}}_{j}||_{2}^{2}{\bm{P}}_{i,j}$
}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to101.99pt{\vbox to19.3pt{\pgfpicture\makeatletter\hbox{\hskip
50.99614pt\lower-10.00012pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-50.99614pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\textstyle\hbox{\pagecolor{blue!10}
$\sum\limits_{\begin{subarray}{c}0\leq
i,j<n\end{subarray}}||{\bm{X}}^{\text{out}}_{i}-{\bm{X}}^{\text{ref}}_{j}||_{2}^{2}{\bm{P}}_{i,j}$
}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to71.39pt{\vbox to13.51pt{\pgfpicture\makeatletter\hbox{\hskip
35.69727pt\lower-7.00009pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-35.69727pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptstyle\hbox{\pagecolor{blue!10}
$\sum\limits_{\begin{subarray}{c}0\leq
i,j<n\end{subarray}}||{\bm{X}}^{\text{out}}_{i}-{\bm{X}}^{\text{ref}}_{j}||_{2}^{2}{\bm{P}}_{i,j}$
}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to51pt{\vbox to9.65pt{\pgfpicture\makeatletter\hbox{\hskip
25.49802pt\lower-5.00006pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-25.49802pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptscriptstyle\hbox{\pagecolor{blue!10}
$\sum\limits_{\begin{subarray}{c}0\leq
i,j<n\end{subarray}}||{\bm{X}}^{\text{out}}_{i}-{\bm{X}}^{\text{ref}}_{j}||_{2}^{2}{\bm{P}}_{i,j}$
}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}+\alpha\enspace\mathchoice{\leavevmode\hbox
to117.32pt{\vbox to19.3pt{\pgfpicture\makeatletter\hbox{\hskip
58.66095pt\lower-10.00012pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-58.66095pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\displaystyle\hbox{\pagecolor{teal!10}
$\sum\limits_{\begin{subarray}{c}0\leq
i,k,j,l<n\end{subarray}}|{\bm{D}}^{\text{out}}_{i,k}-{\bm{D}}^{\text{ref}}_{j,l}|^{2}{\bm{P}}_{i,j}{\bm{P}}_{k,l}$
}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to117.32pt{\vbox to19.3pt{\pgfpicture\makeatletter\hbox{\hskip
58.66095pt\lower-10.00012pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-58.66095pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\textstyle\hbox{\pagecolor{teal!10}
$\sum\limits_{\begin{subarray}{c}0\leq
i,k,j,l<n\end{subarray}}|{\bm{D}}^{\text{out}}_{i,k}-{\bm{D}}^{\text{ref}}_{j,l}|^{2}{\bm{P}}_{i,j}{\bm{P}}_{k,l}$
}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to82.13pt{\vbox to13.51pt{\pgfpicture\makeatletter\hbox{\hskip
41.06262pt\lower-7.00009pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-41.06262pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptstyle\hbox{\pagecolor{teal!10}
$\sum\limits_{\begin{subarray}{c}0\leq
i,k,j,l<n\end{subarray}}|{\bm{D}}^{\text{out}}_{i,k}-{\bm{D}}^{\text{ref}}_{j,l}|^{2}{\bm{P}}_{i,j}{\bm{P}}_{k,l}$
}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to58.66pt{\vbox to9.65pt{\pgfpicture\makeatletter\hbox{\hskip
29.33038pt\lower-5.00006pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-29.33038pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptscriptstyle\hbox{\pagecolor{teal!10}
$\sum\limits_{\begin{subarray}{c}0\leq
i,k,j,l<n\end{subarray}}|{\bm{D}}^{\text{out}}_{i,k}-{\bm{D}}^{\text{ref}}_{j,l}|^{2}{\bm{P}}_{i,j}{\bm{P}}_{k,l}$
}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}\\\
&+\rho\medspace\big{(}\mathchoice{\leavevmode\hbox to240.28pt{\vbox
to11.83pt{\pgfpicture\makeatletter\hbox{\hskip 120.1401pt\lower-4.33331pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-120.1401pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\displaystyle\hbox{\pagecolor{cyan!10}
$\text{KL}({\bm{P}}_{\\#1}\otimes{\bm{P}}_{\\#1}\,|\,{\bm{w}}^{\text{out}}\otimes{\bm{w}}^{\text{out}})+\text{KL}({\bm{P}}_{\\#2}\otimes{\bm{P}}_{\\#2}\,|\,{\bm{w}}^{\text{ref}}\otimes{\bm{w}}^{\text{ref}})$
}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to240.28pt{\vbox to11.83pt{\pgfpicture\makeatletter\hbox{\hskip
120.1401pt\lower-4.33331pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-120.1401pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\textstyle\hbox{\pagecolor{cyan!10}
$\text{KL}({\bm{P}}_{\\#1}\otimes{\bm{P}}_{\\#1}\,|\,{\bm{w}}^{\text{out}}\otimes{\bm{w}}^{\text{out}})+\text{KL}({\bm{P}}_{\\#2}\otimes{\bm{P}}_{\\#2}\,|\,{\bm{w}}^{\text{ref}}\otimes{\bm{w}}^{\text{ref}})$
}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to181.53pt{\vbox to9.58pt{\pgfpicture\makeatletter\hbox{\hskip
90.7646pt\lower-4.33331pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-90.7646pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptstyle\hbox{\pagecolor{cyan!10}
$\text{KL}({\bm{P}}_{\\#1}\otimes{\bm{P}}_{\\#1}\,|\,{\bm{w}}^{\text{out}}\otimes{\bm{w}}^{\text{out}})+\text{KL}({\bm{P}}_{\\#2}\otimes{\bm{P}}_{\\#2}\,|\,{\bm{w}}^{\text{ref}}\otimes{\bm{w}}^{\text{ref}})$
}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to142.36pt{\vbox to8.08pt{\pgfpicture\makeatletter\hbox{\hskip
71.18097pt\lower-4.33331pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-71.18097pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptscriptstyle\hbox{\pagecolor{cyan!10}
$\text{KL}({\bm{P}}_{\\#1}\otimes{\bm{P}}_{\\#1}\,|\,{\bm{w}}^{\text{out}}\otimes{\bm{w}}^{\text{out}})+\text{KL}({\bm{P}}_{\\#2}\otimes{\bm{P}}_{\\#2}\,|\,{\bm{w}}^{\text{ref}}\otimes{\bm{w}}^{\text{ref}})$
}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}\big{)}+\varepsilon\medspace\mathchoice{\leavevmode\hbox
to29.75pt{\vbox to10pt{\pgfpicture\makeatletter\hbox{\hskip
14.87675pt\lower-2.5pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-14.87675pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\displaystyle\hbox{\pagecolor{gray!10} $\text{H}({\bm{P}})$ }$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to29.75pt{\vbox to10pt{\pgfpicture\makeatletter\hbox{\hskip
14.87675pt\lower-2.5pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-14.87675pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\textstyle\hbox{\pagecolor{gray!10} $\text{H}({\bm{P}})$ }$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to20.83pt{\vbox to7pt{\pgfpicture\makeatletter\hbox{\hskip
10.41371pt\lower-1.75pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-10.41371pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptstyle\hbox{\pagecolor{gray!10} $\text{H}({\bm{P}})$ }$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}{\leavevmode\hbox
to14.88pt{\vbox to5pt{\pgfpicture\makeatletter\hbox{\hskip
7.43835pt\lower-1.25pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{
}\nullfont\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}}
{{}{{}}}{{}{}}{}{{}{}} { }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-7.43835pt}{0.0pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\hbox{{$\scriptscriptstyle\hbox{\pagecolor{gray!10} $\text{H}({\bm{P}})$ }$}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox to0.0pt{}{{
{}{}{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}\end{split}$ (1)
Wasserstein lossGromov-Wasserstein lossMarginal constraintsEntropy

with ${\bm{P}}_{\\#1}\triangleq(\sum_{j}{\bm{P}}_{i,j})_{0\leq i<v}$ and
${\bm{P}}_{\\#2}\triangleq(\sum_{i}{\bm{P}}_{i,j})_{0\leq j<v}$ the first and
second marginal distributions of ${\bm{P}}$, $\otimes$ the Kronecker product
between two matrices, and $\text{KL}(\cdot,\cdot)$ the Kullback-Leibler
divergence. $\alpha$, $\rho$ and $\varepsilon$ are hyper-parameters setting
the relative importance of each constraint.

Following Thual et al. (2022), we minimize $\mathcal{L}({\bm{P}})$ with 10
iterations of a block coordinate descent algorithm (Séjourné et al., 2021),
each running $1\,000$ Sinkhorn iterations (Cuturi, 2013). Subsequently, we
define
$\phi_{\text{out}\rightarrow\text{ref}}\colon{\bm{X}}\mapsto\big{(}{\bm{P}}^{T}{\bm{X}}^{T}\big{)}\oslash{\bm{P}}_{\\#2}\in\mathbb{R}^{n,v}$
where $\oslash$ is the element-wise division, a function which transports any
matrix of brain features from the left-out subject to the reference subject.
To simplify notations, for any ${\bm{X}}$ defined on the left-out subject, we
define
${\bm{X}}^{\text{out}\rightarrow\text{ref}}\triangleq\phi_{\text{out}\rightarrow\text{ref}}({\bm{X}})$.

###  2.2 Decoding

##### Brain input

There is a time lag between the moment a stimulus is played and the moment it
elicits a maximal BOLD response in the brain (Glover, 1999). Moreover, since
the effect induced by this stimulus might span over multiple consecutive brain
volumes, we set a window size describing the number of brain volumes to
aggregate together. To account for these effects, we use a standard Finite
Impulse Response (FIR) approach. FIR consists in fitting the decoder on a
time-lagged, multi-volume version of the BOLD response. Different aggregation
functions can be used, such as stacking or averaging. Figure S2 describes
these concepts visually.

##### Video output

The matrix of latent features ${\bm{Y}}$ is obtained by using a pre-trained
image encoder on each video frame and concatenating all obtained vectors in
${\bm{Y}}$. Similarly to Ozcelik & VanRullen (2023), and as illustrated in
Figure 1.A, we seek to predict CLIP 257 $\times$ 768 (high-level) and VD-VAE
(low-level) latent representations . We use visual – as opposed to textual –
CLIP representations (Radford et al., 2021). For comparison, we also reproduce
our approach on latent representations from CLIP CLS (high-level) and AutoKL
(low-level), which happen to be much smaller 333Dimensions for CLIP CLS: $768\
$ ; CLIP 257 $\times$ 768 : $257\times 768=197\,376\ $ ; AutoKL: $4\times
32\times 32=4\,096\ $ ; VD-VAE: $2\times 2^{4}+4\times 2^{8}+8\times
2^{10}+16\times 2^{12}+2^{14}=91\,168$ and are computationally easier to fit.

##### Model

Fitting the decoder consists in deriving ${\bm{W}}\in\mathbb{R}^{v,m}$,
${\bm{b}}\in\mathbb{R}^{m}$ the solution of a Ridge regression problem – i.e.
a linear regression with L2 regularization – predicting ${\bm{Y}}$ from
${\bm{X}}$.

##### Evaluation

We evaluate the performance of the decoder with retrieval metrics. Let us
denote ${\bm{X}}$ and ${\bm{Y}}$ the brain and latent features used to train
the decoder, ${\bm{X}}_{\text{test}}$ and ${\bm{Y}}_{\text{test}}$ those to
test the decoder, and
$\hat{{\bm{Y}}}\triangleq{\bm{W}}{\bm{X}}_{\text{test}}+{\bm{b}}$ the
predicted latents. We ensure that the train and test data are disjoint.

We randomly draw a retrieval set $K$ of 499 frames without replacement from
the test data. For each pair $\hat{{\bm{y}}},{\bm{y}}$ of predicted and ground
truth latents, one derives their cosine similarity score
$s(\hat{{\bm{y}}},{\bm{y}})$, as well as similarity scores to all latents
${\bm{y}}_{\text{neg}}$ of the retrieval set
$s(\hat{{\bm{y}}},{\bm{y}}_{\text{neg}})$. Let us denote
$r_{K}(\hat{{\bm{y}}},{\bm{y}})$ the rank of ${\bm{y}}$, which we define as
the number of elements of $K$ whose similarity score to $\hat{{\bm{y}}}$ is
larger than $s(\hat{{\bm{y}}},{\bm{y}})$. In order for the rank to not depend
on the size of $K$, we define the relative rank as
$\frac{r(\hat{{\bm{y}}},{\bm{y}})}{|K|}$. Finally, one derives the median
relative rank $\text{MR}(\hat{{\bm{Y}}},K)$:

$\displaystyle r_{K}(\hat{{\bm{y}}},{\bm{y}})\;$
$\displaystyle\triangleq\;\big{|}\big{\\{}{\bm{y}}_{\text{neg}}\in
K\;|\;s(\hat{{\bm{y}}},{\bm{y}}_{\text{neg}})>s(\hat{{\bm{y}}},{\bm{y}})\big{\\}}\big{|}$
$\displaystyle\text{MR}(\hat{{\bm{Y}}},K)\;$
$\displaystyle\triangleq\;\text{median}\big{(}\big{\\{}\frac{r_{K}(\hat{{\bm{y}}},{\bm{y}})}{|K|},\forall\;(\hat{{\bm{y}}},{\bm{y}})\big{\\}}\big{)}$

###  2.3 Decoding and alignment setups

##### Within- vs out-of-subject

The within-subject setup consists in training a decoder with data
${\bm{X}}^{S_{1}}_{\text{train}}$, ${\bm{Y}}^{S_{1}}_{\text{train}}$ from a
given subject, and testing it on left-out data
${\bm{X}}^{S_{1}}_{\text{test}}$, ${\bm{Y}}^{S_{1}}_{\text{test}}$ acquired in
the same subject. The out-of-subject setup consists in training a decoder with
data from a given subject, and testing it on data
${\bm{X}}^{S_{2}}_{\text{test}}$, ${\bm{Y}}^{S_{2}}_{\text{test}}$ acquired in
a left-out subject.

##### Single- vs multi-subject

The single-subject setup consists in training a decoder predicting ${\bm{Y}}$
from ${\bm{X}}$ for each subject. The multi-subject setup consists in training
a single decoder using data from multiple subjects. In this case, data from
several subjects is stacked together, resulting in a matrix
${\bm{X}}_{\text{multi}}\in\mathbb{R}^{n_{1}+...+n_{p},v}$ and
${\bm{Y}}_{\text{multi}}\in\mathbb{R}^{n_{1}+...+n_{p},m}$ , where $p$ is the
number of subjects.

##### Un-aligned vs aligned

In multi-subject and out-of-subject setups, data coming from different
subjects can be functionally aligned to that of a reference subject. Let us
assume that $S_{1}$ is the reference subject. In the case of multi-subject,
all subjects are aligned to $S_{1}$ and the decoder is trained on a
concatenation of ${\bm{X}}^{S_{1}},{\bm{X}}^{S_{2}\rightarrow
S_{1}},...,{\bm{X}}^{S_{p}\rightarrow S_{1}}$ (see notations introduced at the
end of section 2.1) and ${\bm{Y}}^{S_{1}},...,{\bm{Y}}^{S_{p}}$ , where $p$ is
the number of subjects. In the case of out-of-subject, it corresponds to
aligning $S_{2}$ onto $S_{1}$, such that a decoder trained on $S_{1}$ will be
tested on ${\bm{X}}^{S_{2}\rightarrow S_{1}}_{\text{test}}$,
${\bm{Y}}^{S_{2}}_{\text{test}}$.

The aforementioned setups are described visually in Figure 3.A.

##### Evaluation under different data regimes

Note that alignment and decoding models need not be fitted using the same
amount of data. In particular, we are interested in evaluating out-of-subject
performance in setups where a lot of data is available for a reference
subject, and little data is available for a left-out subject: this would
typically be the case in clinical setups where little data is available in
patients. In this case, we evaluate whether it is possible to use this small
amount of data to align the left-out subject onto the reference subject, and
have the left-out subject benefit from a decoder previously trained on a lot
of data.

###  2.4 Datasets

We analyze two fMRI datasets. The first dataset (Wen et al., 2017) comprises 3
human subjects who each watched 688 minutes of video. The videos consists of
18 train segments of 8 minutes each and 5 test segments of 8 minutes each.
Each training segment was presented twice. Each test segment was presented 10
times. Each segment consists of a sequence of roughly 10-second video clips.
The fMRI data was acquired at 3 Tesla (3T), 3.5mm isotropic spatial resolution
and 2-second temporal resolution. It was minimally pre-processed with the same
pre-processing pipeline than that of the Human Connectome Project (Glasser et
al., 2013). In particular, data from each subject are projected onto a common
volumetric anatomical template. Similarly to prior work on this dataset (Wen
et al., 2018; Kupershmidt et al., 2022; Wang et al., 2022), we use runs
related to the first 18 video segments - 288 minutes - as training data, and
runs related to the last 5 video segments as test data.

The second dataset (Allen et al., 2021) comprises 8 subjects who are shown
$10\;000$ static images three times. They were scanned over 40 sessions of 60
minutes, amounting to $2\;400$ minutes of data. Instead of raw BOLD signal, we
leverage precomputed beta coefficients accessible online. Supplementary
section A.4 describes this dataset in more detail.

###  2.5 Preprocessing

For the Wen et al. (2017) dataset, we implement minimal additional
preprocessing steps for each subject separately. For this, we (1) project all
volumetric data onto the FreeSurfer average surface template fsaverage5
(Fischl, 2012), then (2) regress out cosine drifts in each vertex and each run
and finally (3) center and scale each vertex time-course in each run. Figure
S1 gives a visual explanation as to why the last two steps are needed. The
first two steps are implemented with nilearn (Abraham et al., 2014)
444https://nilearn.github.io and the last one with scikit-learn (Pedregosa et
al., 2011).
Additionally, for a given subject, we try out two different setups: a first
one where runs showing the same video are averaged, and a second one where
they are stacked.

The Allen et al. (2021) dataset is already preprocessed by the original
authors, and maps of beta coefficients are accessible online.

###  2.6 Hyper-parameters selection

To train decoders, we use the same regularization coefficient
$\alpha_{\text{ridge}}$ across latent types and choose it by running a cross-
validated grid search on folds of the training data. We find that results are
robust to using different values and therefore set
$\alpha_{\text{ridge}}=50\,000$. Similarly, values for lag, window size and
aggregation function are determined through a cross-validated grid search.

Finally, for functional alignment, we use default parameters shipped with
version 0.1.0 of FUGW. Namely, $\alpha$, which controls the balance between
Wasserstein and Gromov-Wasserstein losses – i.e. how important functional data
is compared to anatomical data – is set to $0.5$. Empirically, we see that
$\alpha=0.5$ yields values for the Wasserstein loss which are bigger than that
of the Gromov-Wasserstein loss, meaning that functional data drives these
alignments. $\varepsilon$, which controls for entropic regularization – i.e.
how blurry computed alignments will be – is set to $10^{-4}$. Empirically,
this value yields alignments which are anatomically very sharp. $\rho$, which
sets the importance of marginal constraints – i.e. to what extent more or less
mass can be transported to / from each voxel – is set to $1$. Empirically,
this value leads to all voxels being transported / matched.

##  3 Results

###  3.1 Within-subject prediction of visual representations from BOLD signal
and retrieval of visual inputs

We report video decoding results on the retrieval task in Table 1. For all
three subjects of the Wen et al. (2017) dataset, and for all four types of
latent representations considered, a Ridge regression fitted within-subject
achieves significantly above-chance performance. Besides, performance varies
across subjects, although well-performing subjects reach good performance on
all types of latents.

Results reported in Table 1 were obtained for a lag of 2 brain volumes (i.e. 4
seconds since TR = 2 seconds) and a window size of 2 brain volumes which were
averaged together (see definitions in section 2.5). These parameters were
chosen after running a grid search for lag values ranging from 1 to 5, a
window size ranging from 1 to 3, and 2 possible aggregation functions for
brain volumes belonging to the same window (namely averaging and stacking).
Figure S3 shows results using the averaging aggregation function for different
values of lag and window size, averaged across subjects. These results were
obtained by stacking all runs of the training dataset, as opposed to averaging
repetitions of the same video clip. The two approaches yielded very similar
metrics. We expand on this matter in section 3.3.

Finally, Figure 2 shows retrieved images for Subject 2. Qualitatively, we
observe that retrieved images often fit the theme of images shown to subjects
(with categories like indoor sports, human faces, animals, etc.), but also
regularly exhibit failure cases. It is also possible to use predicted latents
to reconstruct seen video clips at a low frame-per-second rate, which we do
not attempt in this study.

Table 1: Within-subject metrics for all subjects and all latent types on the test set Reported metrics are relative median rank $\downarrow$ (MR) of retrieval on a set of 500 samples, top-5 accuracy % $\uparrow$ (Acc) of retrieval on a set of 500 samples. Chance level is at 50.0 and 1.0 for these two metrics respectively. These results were averaged across 50 retrieval sets, hence results are reported with a standard error of the mean (SEM) smaller than 0.01. The Dummy model systematically predicts the mean latent representation of the training set and achieves chance level.  | CLIP 257 $\times$ 768 | VD-VAE | CLIP CLS | AutoKL
---|---|---|---|---
| MR | Acc | MR | Acc | MR | Acc | MR | Acc
Dummy | 50.0 | 1.0 | 50.0 | 1.0 | 50.0 | 1.0 | 50.0 | 1.0
S1 | 9.4 | 13.8 | 29.9 | 3.0 | 15.1 | 8.4 | 24.9 | 3.9
S2 | 6.8 | 16.4 | 30.2 | 3.5 | 10.6 | 10.5 | 21.8 | 3.8
S3 | 7.8 | 13.6 | 28.5 | 3.1 | 11.0 | 9.9 | 26.0 | 3.3
Figure 2:  Image retrievals using predicted latent representations of CLIP 257
$\times$ 768 latents
We use a model fitted on Subject 2 (S2) and predict the latent representation
of unseen videos (test set). Ground truth (GT) images featured within the
first 5 retrieved (Ret.) images are indicated with a bold purple border. In a
given row, images which appear similar across columns are actually different
frames of the same video clip. Images featuring human faces were blurred. More
cases are available in supplementary Figure S4.

###  3.2 Out-of-subject decoding and multi-subject training

##### Main results

As illustrated in Figure 3, models trained on one subject do not generalise
well to other subjects. However, we demonstrate that functional alignment can
successfully be used as a transfer learning strategy to generalize a pre-
trained model to left-out subjects. In particular, we show that left-out
subjects need not have the same amount of available data than training
subjects to benefit from their model: with just 30 minutes of data, left-out
subjects can reach performance which would have needed roughly 100 minutes of
data in a within-subject setting. Besides, compared to the out-of-subject
baseline, we obtain 25 to 75 percents improvement in relative median rank
across latent types.

Besides, we show that a single model trained on all functionally aligned
subjects can reach slightly better results than models trained on all un-
aligned subjects. In every subject, this multi-subject aligned model performs
comparably to their associated within-subject model.

Note that, in Figure 3, we chose the best performing subject (S2) as the
reference subject. We report all other combinations of reference and left-out
subjects in Supplementary Figures S6 and S7 and find that all effects persist
regardless of the combination. Moreover, supplementary Figures S8 and S9 show
that these results hold for all types for latents. Furthermore, we replicate
this experiment on four subjects from the Natural Scenes Dataset (Allen et
al., 2022). The results, based on the decoding of static images, are similar
to those obtained for video decoding (Tables S1 and S2).

Finally, Figures S11, S12, S13, S14 present all other setups we ran for this
study. In particular, they show that a multi-subject aligned model (e.g.
trained on S1 and S2) has better performance on aligned left-out subjects
(e.g. S3) than a single-subject model (e.g. trained on S2 only).

Figure 3:  Effects of functional alignment on multi-subject and out-of-subject
setups
We report relative median rank $\downarrow$ in all setups described in section
2.3 for CLIP $257\times 768$. In all aligned cases, S1 and S3 were aligned
onto S2. In all out-of-subject cases, we test S1 and S3 onto a decoder trained
on S2. In all multi-subject cases, the decoder was trained on all data from
all 3 subjects. A. In this panel, all models (alignment and decoding) were
trained on all available training data. Results for other latent types are
available in Figure S8. B. In left-out S1 and S3, decoding performance is much
better when using functional alignment to S2 (solid dark purple) than when
using anatomical alignment only (solid pale purple). Performance increases
slightly as the amount of data used to align subjects grows, but does not
always reach levels which can be achieved with a single-subject model fitted
in left-out subjects (solid pale gray dots) when a lot of training data is
available. Training a model on multiple subjects yields good performance in
all 3 subjects (dashed pale teal) which can be further improved by using
functional alignment (dashed dark teal). Results for other latent types are
available in Figure S9.

##### Exploring computed alignments

To better understand how brain features are transformed by functional
alignment, we show in Figure 4 how vertices from S1 are permuted to fit those
of S2. Note that both subjects’ data lie on fsaverage5. To this end, we
colorize vertices in S1 using the MMP 1.0 atlas (Glasser et al., 2016) and use
$\phi_{\text{S1}\rightarrow\text{S2}}$ to transport each of the three RGB
channels of this colorization. We see that, even in low data regimes, FUGW
scrambles most of the brain but can leverage signal to recover the cortical
organization of the occipital lobe. Higher regimes yield anatomically-
consistent matches in a much higher number of cortical areas such as the
temporal and parietal lobes, and more surprisingly in the primary motor cortex
as well, while the prefrontal cortex and temporo-parietal junction (TPJ) still
seem challenging to map.

Figure 4: Visualizing functional alignments in the left hemisphere Vertices of
the left-out subject (left) are permuted by FUGW. The result of this
permutation is visualized on the reference subject (columns 2, 3, and 4).
Fitting FUGW with different increasing amounts of data gradually unfolds the
cortical organisation of multiple areas, even non-visual ones. Note that all 3
models have been fitted using the same number of iterations.

###  3.3 Influence of training set size and test set repetitions

Recent publications in brain-decoding using non-invasive brain imagery show
impressive results. However, we stress that these results are obtained in
setups which are very advantageous when it comes to both dataset size and
signal-to-noise ratio. To better assess the importance of these two factors,
we report in Figure 5 performance metrics for subject models trained and
tested with various amounts of data and various amounts of noise.

Figure 5: Effect of training set size and test set noise on retrieval metrics
Relative median rank $\downarrow$ on a fixed test set gets better as more
training data is used to fit the model (left). Interestingly, averaging brain
volumes of 2 similar runs does not bring improvements compared to using just 1
run. Instead, stacking runs does yield significant improvements. Note that
training sets using 2 runs have twice as much data as those using 1 run.
Finally, these metrics are highly affected by the noise level of the test set
(right): averaging more runs in the test set yields better metrics despite
using the same decoder.

Firstly, using a fixed test set in which brain features were averaged across
runs, we find that exponentially more training data is required per subject to
achieve better performance. This finding is similar to that of systematic
scaling studies on similar topics (Tang et al., 2023). More interestingly, in
this given signal-to-noise ratio setup, it seems that more diverse training
data should bring comparable or better performance than repeating already seen
content, while potentially covering more semantic domains.
Secondly, reported performance metrics only hold in favorable signal-to-noise
setups. Indeed, the test set associated with the Wen 2017 dataset comes with
10 runs for each video segment, which, when averaged together, greatly reduce
the noise level. However, as reported here, when tested in real-life signal-
to-noise conditions (i.e. only one run per video clip), our models’
performance degrades: it is approximately twice as bad for each subject when
using CLIP latents.

##  4 Discussion

##### Impact

The present work confirms the feasibility of using fMRI signals in response to
natural images and videos to decode high level visual features (Nishimoto et
al., 2011). It further demonstrates that it is possible to leverage these fMRI
signals to estimate meaningful functional alignments between subjects, and use
them to transfer semantic decoders to novel subjects.

In particular, our study shows that decoding brain data from a left-out
subject can be substantially improved by aligning this left-out subject to a
large reference dataset on which a decoder was trained. Our method thus paves
the way towards using models trained on large amounts of individual data to
decode signal acquired in smaller neuroimaging studies, which typically record
one hour of fMRI for each subject (Madan, 2022).

Besides, this study reports decoding accuracy in setups where subjects are
showed test stimuli for the first time only, hence yielding insights on how
these models would perform in real-time decoding. While performance improves
with the number of repetitions at test time, reasonable decoding performance
of semantics can be achieved in two out of three subjects with just one
repetition.

Lastly, by systematically quantifying decoding accuracy as a function of the
amount of training data, the present work brings insightful recommendations as
to what stimuli should be played in future fMRI datasets collecting large
amounts of data in a limited number of subjects. In the current setup
(naturalistic movies acquired at 3T), more diverse semantic content is more
valuable than repeated content for fitting decoding models.

##### Limitations

This work is a first step towards training accurate semantic decoders which
generalize across individuals, but subsequent work remains necessary to ensure
the generality of our findings.

Firstly, although reported gains in out-of-subject setups are significant, the
small number of participants present in the dataset under study calls for
replications on other – and potentially larger – cohorts. However, to our
knowledge, no other dataset presented similar features to that of Wen et al.
(2017) – i.e. high quantity of data per subject and large variety of video
stimuli.

Secondly, our approach currently requires left-out subjects to watch the same
videos as reference subjects. It is yet unclear whether functional alignment
could bring improvements without this constraint. However, multi-subject
decoding can probably help partially address this issue: since it is possible
to train a decoder on multiple subjects and because not all of them have to
watch the same movies, it is possible that a lot of different movies could be
used as “anchors” for left-out individuals.

Thirdly, our method relies on pre-trained encoders, and cannot align all
subjects at once. Consequently, overall performance highly depends on the
quality of other models and of data acquired in reference individuals. Recent
work in perception decoding in MEG shows that these two constraints might be
alleviated (Défossez et al., 2022; Benchetrit et al., 2023), although it is
not sure the underlying methods would work on fMRI data.

Finally, while restricting this study to linear models makes sense to
establish baselines and ensure replicability, non-linear models have proved to
be as least comparably efficient (Scotti et al., 2023). A natural improvement
on this work could include these architectures.

##### Ethical implications

Out-of-subject generalization is an important test for decoding models, but it
raises legitimate concerns. In this regard, this study highlights that signal-
to-noise ratio still currently makes it challenging to very accurately decode
semantics in a real-time setup, and that a non-trivial amount of data is
needed per individual for these models to work. Moreover, we stress that,
while great progress has been made in decoding perceived stimuli, imagined
stimuli are still very challenging to decode (Horikawa & Kamitani, 2017).
Nonetheless, it is important for advances in this domain to be publicly
documented. We thus advocate that open and peer-reviewed research is the best
way forward to safely explore the implications of inter-subject modeling, and
more generally brain decoding.

##### Conclusion

Overall, these results provide a significant step towards real-time, subject-
agnostic visual decoding of semantics using fMRI.

#### Acknowledgements

This work was funded in part by FrontCog grant ANR-17-EURE-0017 to JRK for his
work at PSL. We would like to thank Bertrand Thirion, Stanislas Dehaene and
Pierre Orhan for their valuable comments.

## References

  * Abraham et al. (2014) Alexandre Abraham, Fabian Pedregosa, Michael Eickenberg, Philippe Gervais, Andreas Mueller, Jean Kossaifi, Alexandre Gramfort, Bertrand Thirion, and Gael Varoquaux.  Machine learning for neuroimaging with scikit-learn.  _Frontiers in Neuroinformatics_ , 8, 2014.  ISSN 1662-5196.  URL https://www.frontiersin.org/article/10.3389/fninf.2014.00014.
  * Allen et al. (2021) Emily J. Allen, Ghislain St-Yves, Yihan Wu, Jesse L. Breedlove, Jacob S. Prince, Logan T. Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, J. Benjamin Hutchinson, Thomas Naselaris, and Kendrick Kay.  A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence.  _Nature Neuroscience_ , pp. 1–11, December 2021.  ISSN 1546-1726.  doi: 10.1038/s41593-021-00962-x.  URL https://www.nature.com/articles/s41593-021-00962-x.  Bandiera_abtest: a Cg_type: Nature Research Journals Primary_atype: Research Publisher: Nature Publishing Group Subject_term: Cortex;Neural encoding;Object vision;Perception Subject_term_id: cortex;neural-encoding;object-vision;perception.
  * Allen et al. (2022) Emily J. Allen, Ghislain St-Yves, Yihan Wu, Jesse L. Breedlove, Jacob S. Prince, Logan T. Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, J. Benjamin Hutchinson, Thomas Naselaris, and Kendrick Kay.  A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence.  _Nature Neuroscience_ , 25(1):116–126, January 2022.  ISSN 1546-1726.  doi: 10.1038/s41593-021-00962-x.  URL https://www.nature.com/articles/s41593-021-00962-x.  Number: 1 Publisher: Nature Publishing Group.
  * Bazeille et al. (2019) T. Bazeille, H. Richard, H. Janati, and B. Thirion.  Local Optimal Transport for Functional Brain Template Estimation.  In Albert C. S. Chung, James C. Gee, Paul A. Yushkevich, and Siqi Bao (eds.), _Information Processing in Medical Imaging_ , Lecture Notes in Computer Science, pp. 237–248, Cham, 2019. Springer International Publishing.  ISBN 978-3-030-20351-1.  doi: 10.1007/978-3-030-20351-1˙18.
  * Bazeille et al. (2021) Thomas Bazeille, Elizabeth DuPre, Hugo Richard, Jean-Baptiste Poline, and Bertrand Thirion.  An empirical evaluation of functional alignment using inter-subject decoding.  _NeuroImage_ , 245:118683, December 2021.  ISSN 1053-8119.  doi: 10.1016/j.neuroimage.2021.118683.  URL https://www.sciencedirect.com/science/article/pii/S1053811921009563.
  * Benchetrit et al. (2023) Yohann Benchetrit, Hubert Banville, and Jean-Rémi King.  Brain decoding: toward real-time reconstruction of visual perception.  _arXiv preprint arXiv:2310.19812_ , 2023.
  * Chen et al. (2015) Po-Hsuan (Cameron) Chen, Janice Chen, Yaara Yeshurun, Uri Hasson, James Haxby, and Peter J Ramadge.  A Reduced-Dimension fMRI Shared Response Model.  In _Advances in Neural Information Processing Systems_ , volume 28. Curran Associates, Inc., 2015.  URL https://papers.nips.cc/paper_files/paper/2015/hash/b3967a0e938dc2a6340e258630febd5a-Abstract.html.
  * Chen et al. (2023a) Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou.  Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding, March 2023a.  URL http://arxiv.org/abs/2211.06956.  arXiv:2211.06956 [cs].
  * Chen et al. (2023b) Zijiao Chen, Jiaxin Qing, and Juan Helen Zhou.  Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity, May 2023b.  URL http://arxiv.org/abs/2305.11675.  arXiv:2305.11675 [cs].
  * Cuturi (2013) Marco Cuturi.  Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances.  _arXiv_ , June 2013.  doi: 10.48550/arXiv.1306.0895.
  * Défossez et al. (2022) Alexandre Défossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, and Jean-Rémi King.  Decoding speech from non-invasive brain recordings, August 2022.  URL http://arxiv.org/abs/2208.12266.  arXiv:2208.12266 [cs, eess, q-bio].
  * Feilong et al. (2022) Ma Feilong, Samuel A. Nastase, Guo Jiahui, Yaroslav O. Halchenko, M. Ida Gobbini, and James V. Haxby.  The Individualized Neural Tuning Model: Precise and generalizable cartography of functional architecture in individual brains, May 2022.  URL https://www.biorxiv.org/content/10.1101/2022.05.15.492022v1.  Pages: 2022.05.15.492022 Section: New Results.
  * Ferrante et al. (2023) Matteo Ferrante, Furkan Ozcelik, Tommaso Boccato, Rufin VanRullen, and Nicola Toschi.  Brain Captioning: Decoding human brain activity into images and text, May 2023.  URL http://arxiv.org/abs/2305.11560.  arXiv:2305.11560 [cs].
  * Fischl (2012) Bruce Fischl.  FreeSurfer.  _NeuroImage_ , 62(2):774–781, August 2012.  ISSN 1053-8119.  doi: 10.1016/j.neuroimage.2012.01.021.  URL https://www.sciencedirect.com/science/article/pii/S1053811912000389.
  * Glasser et al. (2013) Matthew F. Glasser, Stamatios N. Sotiropoulos, J. Anthony Wilson, Timothy S. Coalson, Bruce Fischl, Jesper L. Andersson, Junqian Xu, Saad Jbabdi, Matthew Webster, Jonathan R. Polimeni, David C. Van Essen, Mark Jenkinson, and WU-Minn HCP Consortium.  The minimal preprocessing pipelines for the Human Connectome Project.  _NeuroImage_ , 80:105–124, October 2013.  ISSN 1095-9572.  doi: 10.1016/j.neuroimage.2013.04.127.
  * Glasser et al. (2016) Matthew F. Glasser, Timothy S. Coalson, Emma C. Robinson, Carl D. Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F. Beckmann, Mark Jenkinson, Stephen M. Smith, and David C. Van Essen.  A multi-modal parcellation of human cerebral cortex.  _Nature_ , 536(7615):171–178, August 2016.  ISSN 1476-4687.  doi: 10.1038/nature18933.  URL https://www.nature.com/articles/nature18933.  Number: 7615 Publisher: Nature Publishing Group.
  * Glover (1999) G. H. Glover.  Deconvolution of impulse response in event-related BOLD fMRI.  _NeuroImage_ , 9(4):416–429, April 1999.  ISSN 1053-8119.  doi: 10.1006/nimg.1998.0419.
  * Gu et al. (2023) Zijin Gu, Keith Jamison, Amy Kuceyeski, and Mert Sabuncu.  Decoding natural image stimuli from fMRI data with a surface-based convolutional network, March 2023.  URL http://arxiv.org/abs/2212.02409.  arXiv:2212.02409 [cs, q-bio].
  * Harrison & Tong (2009) Stephenie A Harrison and Frank Tong.  Decoding reveals the contents of visual working memory in early visual areas.  _Nature_ , 458(7238):632–635, 2009.
  * Haxby et al. (2011) James V. Haxby, J. Swaroop Guntupalli, Andrew C. Connolly, Yaroslav O. Halchenko, Bryan R. Conroy, M. Ida Gobbini, Michael Hanke, and Peter J. Ramadge.  A common, high-dimensional model of the representational space in human ventral temporal cortex.  _Neuron_ , 72(2):404–416, October 2011.  ISSN 1097-4199.  doi: 10.1016/j.neuron.2011.08.026.
  * Haxby et al. (2020) James V Haxby, J Swaroop Guntupalli, Samuel A Nastase, and Ma Feilong.  Hyperalignment: Modeling shared information encoded in idiosyncratic cortical topographies.  _eLife_ , 9:e56601, June 2020.  ISSN 2050-084X.  doi: 10.7554/eLife.56601.  URL https://doi.org/10.7554/eLife.56601.  Publisher: eLife Sciences Publications, Ltd.
  * Haynes & Rees (2006) John-Dylan Haynes and Geraint Rees.  Decoding mental states from brain activity in humans.  _Nature reviews neuroscience_ , 7(7):523–534, 2006.
  * Ho et al. (2023) Jun Kai Ho, Tomoyasu Horikawa, Kei Majima, Fan Cheng, and Yukiyasu Kamitani.  Inter-individual deep image reconstruction via hierarchical neural code conversion.  _NeuroImage_ , 271:120007, May 2023.  ISSN 1053-8119.  doi: 10.1016/j.neuroimage.2023.120007.  URL https://www.sciencedirect.com/science/article/pii/S1053811923001532.
  * Horikawa & Kamitani (2017) Tomoyasu Horikawa and Yukiyasu Kamitani.  Generic decoding of seen and imagined objects using hierarchical visual features.  _Nat. Commun._ , 8(15037):1–15, May 2017.  ISSN 2041-1723.  doi: 10.1038/ncomms15037.
  * Kupershmidt et al. (2022) Ganit Kupershmidt, Roman Beliy, Guy Gaziv, and Michal Irani.  A Penny for Your (visual) Thoughts: Self-Supervised Reconstruction of Natural Movies from Brain Activity, June 2022.  URL http://arxiv.org/abs/2206.03544.  arXiv:2206.03544 [cs].
  * Lahner et al. (2023) Benjamin Lahner, Kshitij Dwivedi, Polina Iamshchinina, Monika Graumann, Alex Lascelles, Gemma Roig, Alessandro Thomas Gifford, Bowen Pan, SouYoung Jin, N. Apurva Ratan Murty, Kendrick Kay, Aude Oliva, and Radoslaw Cichy.  BOLD Moments: modeling short visual events through a video fMRI dataset and metadata, March 2023.  URL https://www.biorxiv.org/content/10.1101/2023.03.12.530887v1.  Pages: 2023.03.12.530887 Section: New Results.
  * LeBel et al. (2023) Amanda LeBel, Lauren Wagner, Shailee Jain, Aneesh Adhikari-Desai, Bhavin Gupta, Allyson Morgenthal, Jerry Tang, Lixiang Xu, and Alexander G. Huth.  A natural language fMRI dataset for voxelwise encoding models.  _Scientific Data_ , 10(1):555, August 2023.  ISSN 2052-4463.  doi: 10.1038/s41597-023-02437-z.  URL https://www.nature.com/articles/s41597-023-02437-z.  Number: 1 Publisher: Nature Publishing Group.
  * Madan (2022) Christopher R. Madan.  Scan Once, Analyse Many: Using Large Open-Access Neuroimaging Datasets to Understand the Brain.  _Neuroinformatics_ , 20(1):109–137, January 2022\.  ISSN 1559-0089.  doi: 10.1007/s12021-021-09519-6.
  * Mai & Zhang (2023) Weijian Mai and Zhijun Zhang.  UniBrain: Unify Image Reconstruction and Captioning All in One Diffusion Model from Human Brain Activity, August 2023.  URL http://arxiv.org/abs/2308.07428.  arXiv:2308.07428 [cs].
  * Mitchell et al. (2004) Tom M Mitchell, Rebecca Hutchinson, Radu S Niculescu, Francisco Pereira, Xuerui Wang, Marcel Just, and Sharlene Newman.  Learning to decode cognitive states from brain images.  _Machine learning_ , 57:145–175, 2004.
  * Nishimoto et al. (2011) Shinji Nishimoto, An T. Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, and Jack L. Gallant.  Reconstructing visual experiences from brain activity evoked by natural movies.  _Current Biology_ , 21(19):1641–1646, October 2011.  ISSN 0960-9822.  doi: 10.1016/j.cub.2011.08.031.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3326357/.
  * Ozcelik & VanRullen (2023) Furkan Ozcelik and Rufin VanRullen.  Natural scene reconstruction from fMRI signals using generative latent diffusion, June 2023.  URL http://arxiv.org/abs/2303.05334.  arXiv:2303.05334 [cs, q-bio].
  * Pedregosa et al. (2011) Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay.  Scikit-learn: Machine Learning in Python.  _Journal of Machine Learning Research_ , 12(85):2825–2830, 2011.  ISSN 1533-7928.  URL http://jmlr.org/papers/v12/pedregosa11a.html.
  * Phillips et al. (2022) Erin M. Phillips, Kirsten D. Gillette, Daniel D. Dilks, and Gregory S. Berns.  Through a Dog’s Eyes: fMRI Decoding of Naturalistic Videos from the Dog Cortex.  _JoVE (Journal of Visualized Experiments)_ , (187):e64442, September 2022.  ISSN 1940-087X.  doi: 10.3791/64442.  URL https://www.jove.com/fr/v/64442/through-dog-s-eyes-fmri-decoding-naturalistic-videos-from-dog.
  * Pinho et al. (2018) Ana Luísa Pinho, Alexis Amadon, Torsten Ruest, Murielle Fabre, Elvis Dohmatob, Isabelle Denghien, Chantal Ginisty, Séverine Becuwe-Desmidt, Séverine Roger, Laurence Laurier, Véronique Joly-Testault, Gaëlle Médiouni-Cloarec, Christine Doublé, Bernadette Martins, Philippe Pinel, Evelyn Eger, Gaël Varoquaux, Christophe Pallier, Stanislas Dehaene, Lucie Hertz-Pannier, and Bertrand Thirion.  Individual Brain Charting, a high-resolution fMRI dataset for cognitive mapping.  _Scientific Data_ , 5(1):180105, June 2018.  ISSN 2052-4463.  doi: 10.1038/sdata.2018.105.  URL https://www.nature.com/articles/sdata2018105.  Number: 1 Publisher: Nature Publishing Group.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning Transferable Visual Models From Natural Language Supervision, February 2021.  URL http://arxiv.org/abs/2103.00020.  arXiv:2103.00020 [cs].
  * Richard et al. (2020) Hugo Richard, Luigi Gresele, Aapo Hyvärinen, Bertrand Thirion, Alexandre Gramfort, and Pierre Ablin.  Modeling Shared Responses in Neuroimaging Studies through MultiView ICA, December 2020.  URL http://arxiv.org/abs/2006.06635.  arXiv:2006.06635 [cs, stat].
  * Robinson et al. (2014) Emma C. Robinson, Saad Jbabdi, Matthew F. Glasser, Jesper Andersson, Gregory C. Burgess, Michael P. Harms, Stephen M. Smith, David C. Van Essen, and Mark Jenkinson.  MSM: a new flexible framework for Multimodal Surface Matching.  _NeuroImage_ , 100:414–426, October 2014.  ISSN 1095-9572.  doi: 10.1016/j.neuroimage.2014.05.069.
  * Scotti et al. (2023) Paul S. Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Ethan Cohen, Aidan J. Dempster, Nathalie Verlinde, Elad Yundler, David Weisberg, Kenneth A. Norman, and Tanishq Mathew Abraham.  Reconstructing the Mind’s Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors, May 2023.  URL http://arxiv.org/abs/2305.18274.  arXiv:2305.18274 [cs, q-bio].
  * Séjourné et al. (2021) Thibault Séjourné, François-Xavier Vialard, and Gabriel Peyré.  The Unbalanced Gromov Wasserstein Distance: Conic Formulation and Relaxation.  _arXiv:2009.04266 [math, stat]_ , June 2021.  URL http://arxiv.org/abs/2009.04266.  arXiv: 2009.04266.
  * Takagi & Nishimoto (2023) Yu Takagi and Shinji Nishimoto.  High-resolution image reconstruction with latent diffusion models from human brain activity, March 2023.  URL https://www.biorxiv.org/content/10.1101/2022.11.18.517004v3.  Pages: 2022.11.18.517004 Section: New Results.
  * Tang et al. (2023) Jerry Tang, Amanda LeBel, Shailee Jain, and Alexander G. Huth.  Semantic reconstruction of continuous language from non-invasive brain recordings.  _Nature Neuroscience_ , 26(5):858–866, May 2023\.  ISSN 1546-1726.  doi: 10.1038/s41593-023-01304-9.  URL https://www.nature.com/articles/s41593-023-01304-9.  Number: 5 Publisher: Nature Publishing Group.
  * Thomas et al. (2022) Armin Thomas, Christopher Ré, and Russell Poldrack.  Self-Supervised Learning of Brain Dynamics from Broad Neuroimaging Data.  _Advances in Neural Information Processing Systems_ , 35:21255–21269, December 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/8600a9df1a087a9a66900cc8c948c3f0-Abstract-Conference.html.
  * Thual et al. (2022) Alexis Thual, Quang Huy Tran, Tatiana Zemskova, Nicolas Courty, Rémi Flamary, Stanislas Dehaene, and Bertrand Thirion.  Aligning individual brains with fused unbalanced Gromov Wasserstein.  _Advances in Neural Information Processing Systems_ , 35:21792–21804, December 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/8906cac4ca58dcaf17e97a0486ad57ca-Abstract-Conference.html.
  * Wang et al. (2022) Chong Wang, Hongmei Yan, Wei Huang, Jiyi Li, Yuting Wang, Yun-Shuang Fan, Wei Sheng, Tao Liu, Rong Li, and Huafu Chen.  Reconstructing rapid natural vision with fMRI-conditional video generative adversarial network.  _Cerebral Cortex_ , 32(20):4502–4511, October 2022.  ISSN 1047-3211.  doi: 10.1093/cercor/bhab498.  URL https://doi.org/10.1093/cercor/bhab498.
  * Wen et al. (2017) Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayue Cao, and Zhongming Liu.  Data for Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision Tests, September 2017.  URL https://purr.purdue.edu/publications/2809/1.
  * Wen et al. (2018) Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayue Cao, and Zhongming Liu.  Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision.  _Cerebral Cortex (New York, N.Y.: 1991)_ , 28(12):4136–4160, December 2018.  ISSN 1460-2199.  doi: 10.1093/cercor/bhx268.

##  Appendix A Appendix

###  A.1 Data pre-processing

We strive to minimally preprocess acquired BOLD signal. To this end, we
detrend acquired BOLD signal (i.e. we remove cosine drifts) and finally
standardize voxels’ timecourses for each run, as shown in Figure S1.

Moreover, when decoding the latent representation of a given image, we use
brain volumes which have been acquired after the image’s onset. Figure S2
illustrates this idea, and introduces the concepts of window size (i.e. the
number of brain volumes we use) and lag (i.e. the time difference between the
onset of the image to be decoded and the first brain volume used to decode
it). Values for both of these hyper-parameters were cross-validated. We report
these results in Figure S3.

Figure S1: Pre-processing of the Wen 2017 dataset For each subject and each
run, in each vertex, we regress out parts of the signal which can be linearly
explained by the design matrix represented on the left, which models cosine
drifts of the BOLD signal. The two graphs to the right show time-courses in 5
vertices across 2 different runs before (left) and after (right) they have
been pre-processed. Figure S2: Lag and window size In order to decode a movie
frame which was seen at time $t$, one can use brain volumes which were
acquired further in time. This delay is referred to as the lag. Moreover, one
can use several brain volumes to decode a given movie frame. The number of
brain volumes used is called the window size. Images featuring human faces
were blurred. Figure S3: Relative median rank $\downarrow$ of predicted
latents averaged across subjects for various time lags and window sizes

###  A.2 Retrieving images using predicted latent representations

Predicted latent representations can be compared to that of images in a
retrieval set. In Figure S4, for each image shown to the subject during the
test phase, we print the five images from the retrieval set whose latent
representation is the closest to predicted latents. We see that semantics are
often preserved.

Figure S4:  Image retrievals using predicted latent representations of CLIP
257 $\times$ 768 latents
We use a model fitted on Subject 2 (S2) and predict the latent representation
of unseen videos (test set). Ground truth (GT) images featured within the
first 5 retrieved (Ret.) images are indicated with a bold purple border. In a
given row, images which appear similar across columns are actually different
frames of the same video clip. Images featuring human faces were blurred.

###  A.3 Results for every combination of reference subject and left-out
subject

Figure S5 is a copy of Figure 3 from the main pages of this paper. It
illustrates the main effects reported in our study, namely that (1) functional
alignment yields better performance than anatomical alignment when
transferring a semantic decoder to left-out individuals, (2) it is possible to
train such decoders on multiple subjects and (3) this last setup works best
when subjects are aligned.

Figure S5 only shows these results when subject 2 of the Wen 2017 dataset is
used as the reference subject. Therefore, we add Figures S6 and S7, which
illustrate that all results hold regardless of what subject of the cohort is
used as reference or left-out subject.

Figure S5:  Effects of functional alignment on multi-subject and out-of-
subject setups using subject 2 as the reference subject We report relative
median rank $\downarrow$ in all setups described in section 2.3 for CLIP
$257\times 768$. In all aligned cases, S1 and S3 were aligned onto S2. In all
out-of-subject cases, we test S1 and S3 onto a decoder trained on S2. In all
multi-subject cases, the decoder was trained on all data from all 3 subjects.
A. In this panel, all models (alignment and decoding) were trained on all
available training data. Results for other latent types are available in
Figure S8. B. In left-out S1 and S3, decoding performance is much better when
using functional alignment to S2 (solid dark purple) than when using
anatomical alignment only (solid pale purple). Performance increases slightly
as the amount of data used to align subjects grows, but does not always reach
levels which can be achieved with a single-subject model fitted in left-out
subjects (solid pale gray dots) when a lot of training data is available.
Training a model on multiple subjects yields good performance in all 3
subjects (dashed pale teal) which can be further improved by using functional
alignment (dashed dark teal). Results for other latent types are available in
Figure S9.  Figure S6:  Effects of functional alignment on multi-subject and
out-of-subject setups using subject 1 as the reference subject Figure S7:
Effects of functional alignment on multi-subject and out-of-subject setups
using subject 3 as the reference subject

###  A.4 Replication on the Natural Scenes Dataset

We replicate our main experiment using data from the Natural Scenes Dataset
(Allen et al., 2022). This dataset comprises 8 subjects who each see $10\;000$
images 3 times, thus leading to a total of $30\;000$ trials per subject. For
each subject, this data is acquired in 40 sessions of 60 minutes each. For
each subject, there is a total of $1\;000$ images which are shared with other
individuals - i.e. other individuals will see them too - and $9\;000$ which
are exclusive - i.e. other individuals will not see them. We sub-selected all
subjects who had completed all $30\;000$ trials, namely subjects 1, 2, 5, and
7. For each selected subject, we split their $30\;000$ trials in two sets: all
exclusive images are grouped in the decoding set and all shared images are
grouped in the alignment set. We further split the decoding set into disjoint
sets of images for training and testing individual decoders, whose performance
is reported in Table S1. Alignments sets are used to compute functional
alignments between individuals. Besides, we used pre-computed beta
coefficients computed with GLM denoise on fsaverage7 (Fischl, 2012) and openly
available online. We down-sampled this data to fsaverage5 \- which simply
amounts to keeping only the first $10\;242$ array elements in each hemisphere.

Eventually, we show that decoders tested on left-out individuals work
consistently and significantly better when left-out subjects are functionally
aligned rather than simply anatomically aligned to the reference subject, as
reported in Table S2.

Table S1:  Within-subject metrics for all NSD subjects and all latent types on the test set Reported metrics are relative median rank $\downarrow$ (MR) of retrieval on a set of 500 samples, top-5 accuracy % $\uparrow$ (Acc) of retrieval on a set of 500 samples. Chance level is at 50.0 and 1.0 for these metrics respectively. These results were averaged across 50 retrieval sets, hence results are reported with a standard error of the mean (SEM) smaller than 0.01.  | CLIP 257 $\times$ 768 | VD-VAE | CLIP CLS | AutoKL
---|---|---|---|---
| MR | Acc | MR | Acc | MR | Acc | MR | Acc
S1 | 3.6 | 26.6 | 23.0 | 4.6 | 4.6 | 19.1 | 30.5 | 1.8
S2 | 6.0 | 17.6 | 22.1 | 4.4 | 6.9 | 13.8 | 33.6 | 1.5
S5 | 4.6 | 19.9 | 26.0 | 3.7 | 4.3 | 19.7 | 31.5 | 2.5
S7 | 4.0 | 24.4 | 23.4 | 5.4 | 5.5 | 18.5 | 24.5 | 4.3
Table S2:  Across-subject metrics for all NSD subjects and all latent types on the test set We report the decoding performance of decoders trained on a reference subject and tested on a left-out subject who was anatomically aligned (A) or functionally aligned (A+F). The reported metric is the relative median rank $\downarrow$ (MR) of retrieval on a set of 500 samples. These results were averaged across 50 retrieval sets, hence results are reported with a standard error of the mean (SEM) smaller than 0.01. One sees that functionally aligned data is always better decoded than anatomically aligned data.  Reference | Left-out | CLIP 257 $\times$ 768 | VD-VAE | CLIP CLS | AutoKL
---|---|---|---|---|---
A | A+F | A | A+F | A | A+F | A | A+F
S1 | S2 | 20.9 | 10.7 | 35.2 | 24.5 | 28.3 | 13.4 | 42.5 | 37.1
S5 | 32.2 | 13.6 | 43.1 | 32.0 | 33.4 | 12.5 | 43.9 | 37.7
S7 | 33.5 | 14.7 | 40.8 | 30.7 | 36.1 | 17.1 | 45.0 | 37.0
S2 | S1 | 18.3 | 10.4 | 37.0 | 30.4 | 24.4 | 14.4 | 35.9 | 37.1
S5 | 29.3 | 12.0 | 40.3 | 34.1 | 30.8 | 11.2 | 42.4 | 39.6
S7 | 27.9 | 14.6 | 40.2 | 32.8 | 32.1 | 17.9 | 41.8 | 38.9
S5 | S1 | 29.2 | 13.0 | 43.4 | 34.4 | 33.8 | 14.4 | 37.8 | 33.8
S2 | 26.2 | 9.8 | 40.1 | 30.1 | 31.4 | 11.2 | 36.0 | 36.2
S7 | 29.8 | 14.1 | 41.7 | 32.0 | 34.4 | 17.3 | 40.0 | 34.9
S7 | S1 | 27.7 | 7.7 | 43.1 | 30.1 | 32.3 | 11.5 | 40.6 | 26.5
S2 | 23.4 | 8.6 | 38.9 | 26.4 | 28.6 | 13.2 | 37.8 | 29.1
S5 | 27.5 | 8.8 | 41.6 | 31.2 | 30.4 | 9.8 | 43.4 | 30.9

###  A.5 Results for every type of latent representation

Results of supplementary section A.4 show that our effects of interest hold
for CLIP 257 $\times$ 768 regardless of the combination of reference / left-
out subjects. We replicate these experiments using 3 additional types of
latent representations, namely VD-VAE, CLIP CLS and AutoKL and report results
in Figures S8, S9 and S10. These figures show that all observed effects are
present regardless of the chosen latent representation.

Figure S8: Effects of alignment For any type of latent representation, out-of-
subject decoding performance, measured through relative median rank
$\downarrow$, greatly improves when subjects are functionally aligned.
Training decoders on multiple subjects also works better when subjects are
aligned. These results were averaged across 50 retrieval sets ; all these
metrics are reported with a standard error of the mean (SEM) smaller than
0.01. Figure S9: Performance increases slightly with more alignment data For
any type of latent representation, out-of-subject decoding performance greatly
increases with functional alignment even in low data regimes. In high data
regimes, out-of-subject decoding does not work as well as fitting single-
subject or multi-subject models. Figure S10: Scaling studies for all latents
For any type of latent representation, decoding performance increases linearly
with exponentially more data. It also seems that, when acquiring data at 3T or
more, not repeating stimuli yields the best results. At test time, although
repeating stimuli allows to get better metrics, retrieval performance with
only one repetition is already reasonable in 2 out of 3 subjects of the Wen
2017 dataset.

###  A.6 Decoding results for all setups

On top of experiments reported in the main pages of this paper, we have tested
a lot of different training and tests sets. In this section, we report the
Relative Median Rank $\downarrow$ for all 97 training sets and all 63 test
sets, and all four types of latent representations. Training sets include all
possible single-subject, multi-subject unaligned and multi-subject aligned
cases. Test sets include all possible with-subject, left-out unaligned and
left-out aligned cases. Every time, all available training sessions are used
for training the decoder. However, we vary the amount of data used to train
the alignments, for both training and test sets.

We report detailed results for CLIP 257 $\times$ 768, VD-VAE, CLIP CLS and
AutoKL in Figures S11, S12, S13 and S14 respectively.

Figure S11: Relative median rank $\downarrow$ for CLIP 257 $\times$ 768
latents in single- and multi-subject training sets, with and without
alignment, tested on within- and across-subjects setups with and without
alignment. These results were averaged across 50 retrieval sets ; all these
metrics are reported with a standard error of the mean (SEM) smaller than
0.01. Figure S12: Relative median rank $\downarrow$ for VD-VAE latents in
single- and multi-subject training sets, with and without alignment, tested on
within- and across-subjects setups with and without alignment. These results
were averaged across 50 retrieval sets ; all these metrics are reported with a
standard error of the mean (SEM) smaller than 0.01. Figure S13: Relative
median rank $\downarrow$ for CLIP CLS latents in single- and multi-subject
training sets, with and without alignment, tested on within- and across-
subjects setups with and without alignment. These results were averaged across
50 retrieval sets ; all these metrics are reported with a standard error of
the mean (SEM) smaller than 0.01. Figure S14: Relative median rank
$\downarrow$ for AutoKL latents in single- and multi-subject training sets,
with and without alignment, tested on within- and across-subjects setups with
and without alignment. These results were averaged across 50 retrieval sets ;
all these metrics are reported with a standard error of the mean (SEM) smaller
than 0.01.