# Token Recycling for Efficient Sequential Inference with Vision Transformers

Jan Olszewski1,2,5,∗ Dawid Rymarczyk1,3 Piotr Wójcik4 Mateusz Pach1 Bartosz
Zieliński1,5
1 Faculty of Mathematics and Computer Science, Jagiellonian University  2
University of Warsaw  3 Ardigen SA  4 Center for Molecular Medicine Cologne
(CMMC), University of Cologne  5 IDEAS NCBR<EMAIL_ADDRESS>
###### Abstract

Vision Transformers (ViTs) overpass Convolutional Neural Networks in
processing incomplete inputs because they do not require the imputation of
missing values. Therefore, ViTs are well suited for sequential decision-
making, e.g. in the Active Visual Exploration problem. However, they are
computationally inefficient because they perform a full forward pass each time
a piece of new sequential information arrives.

To reduce this computational inefficiency, we introduce the TOken REcycling
(TORE) modification for the ViT inference, which can be used with any
architecture. TORE divides ViT into two parts, iterator and aggregator. An
iterator processes sequential information separately into midway tokens, which
are cached. The aggregator processes midway tokens jointly to obtain the
prediction. This way, we can reuse the results of computations made by
iterator.

Except for efficient sequential inference, we propose a complementary training
policy, which significantly reduces the computational burden associated with
sequential decision-making while achieving state-of-the-art accuracy. We make
the code available.

##  1 Introduction

Convolutional Neural Networks (CNNs) [21] and Vision Transformers (ViT) [11]
have profoundly reshaped computer vision, surpassing human performance in
tasks such as segmentation and object detection [12]. One of the advantages of
ViTs over CNNs is how they handle incomplete inputs. ViTs process arbitrary
subsets of input tokens, processing only available data, while CNNs require
data imputation before further analysis [31].

This advantage is crucial in sequential decision-making, an AI subfield where
choices over time lead to collecting observations essential for achieving
specific goals [47]. One of the research challenges in this field is Active
Visual Exploration (AVE), where a robot actively controls its sensors to
optimize the collection of visual data [33, 40]. Such applications often
operate under strict resource constraints, which can pose a challenge given
the substantial resource demands of ViTs.

Figure 1: The figure illustrates the forward pass for TORE for two sequential
glimpses, denoted as $G_{1}$ and $G_{2}$. First, glimpse $G_{1}$ is divided
into patches and processed by an iterator generating a midway token $T_{1}$,
which is cached for future use. When processing the second glimpse $G_{2}$,
the aggregator efficiently reuses the cached midway token $T_{1}$ to reduce
the computational load.

Recent methods for Active Visual Exploration, such as AME [29] and Glimpse-
Attend-and-Explore [40], use ViT-based models that generate meaningful
representations from partial inputs. However, they execute a full forward pass
on all available observations each time a new observation is collected,
resulting in increased computational demands.

To address this computational inefficiency, we present a simple yet powerful
variant of the ViT forward pass, referred to as TORE (TOken REcycling). This
modification divides ViT into two parts: iterator and aggregator, as presented
in Figure 1. The iterator processes sequential information separately,
generating midway tokens that are cached. When a classification is made, the
aggregator combines the midway tokens and generates the final prediction. This
way we can reuse the results of computations made by the iterator. We propose
a novel policy for efficient training in iterator-aggregator regime, which
does not require any additional hyperparameter and returns state-of-the-art
accuracy with a significantly reduced computational workload, as presented in
Table 1.

Our contributions can be summarized as follows:

  * •

We introduce a novel approach to ViT inference operating in an iterator-
aggregator regime, which is suited for sequential decision-making and can be
used with any architecture.

  * •

We propose a complementary training policy that requires no additional
hyperparameter.

  * •

We conduct extensive experiments on ViTs obtained with various pretraining
strategies to show that our iterator-aggregator regime can be incorporated to
multiple scenarios.

##  2 Related Work

#### Efficient Vision Transformers.

Vision Transformers [11] are versatile models suitable to process images, and
they are applied in multiple tasks such as classification [44], detection [7]
and segmentation [54]. Moreover, many self-supervised pretraining strategies
are developed for ViTs, such as masked image modeling [2, 3, 14], self
distillation [8, 28], contrastive learning [10, 52] or a mix of those methods
[55]. However, they are designed to maximize the model’s effectiveness on
downstream tasks rather than focus on computational efficiency.

It is important to optimize the transformer-based model’s computational
efficiency to utilize their properties on edge devices, such as robots and
drones. Methods aiming at computational acceleration propose token pruning
[35, 48, 49], low-rank factorization [51], limiting self-attention to non-
overlapping local windows [23], scaling attention mechanism with sequence
length [4, 6], and replacing fully-connected layers with a star-shaped
topology [13].

Our method (TORE) aligns with recent trends to enhance the computational
efficiency of ViTs. Specifically, we modify ViTs forward pass to optimize for
sequential decision-making.

| Dataset | Method | GFLOPs $\downarrow$ | Acc. $\uparrow$
---|---|---|---|---
AVE | SUN360 | MAE [14] | 13.0 | 72.2
TORE + MAE | 9.0 | 73.8
AME [29] | 13.6 | 76.2
TORE + AME | 8.0 | 77.2
CIFAR100 | MAE [14] | 27.8 | 75.8
TORE + MAE | 22.1 | 76.4
AME [29] | 28.6 | 77.9
TORE + AME | 19.2 | 78.9
SC | CIFAR100 | DeiT [44] | 10.5 | 81.6
TORE + DeiT | 8.4 | 82.2
ImageNet-1k | DeiT [44] | 10.5 | 72.0
TORE + DeiT | 8.4 | 73.2
Table 1: Comparison of our method (TORE) with different methods in Active
Visual Exploration (AVE) and sequential classification (SC) setups. In both
metrics, accuracy on sequentially available input, and flops measuring the
computational burden, our method achieves the best performance.

#### Visual Sequential Decision-Making.

Sequential decision-making, which involves the agent making successive
observations before reaching a final decision [5], has been addressed by
multiple computational methods, such as Waldboost-learning [41], recurrent
models [26], reinforcement learning [45], input masking techniques [9], two-
stage models with extraction and aggregation paths [18], and the sequential
mixture of experts [32]. In the context of computer vision, sequential
decision-making finds application in diverse areas such as pedestrian
reidentification [53], object tracking [42], crowd counting [22, 25], vision-
dialog navigation [56], and robotics [1, 15, 40] which also requires
computational efficiency.

In the field of sequential decision-making, with a focus on computational
efficiency, our work aligns with the robotics use case, especially with
_Active Vision Exploration_ (AVE). AVE involves an agent guiding its view to
successively improve internal representations of the observed world [1, 15].
In this context, the selection strategies for successive observations
(exploration) play a crucial role. Multiple approaches are proposed to address
this challenge, such as image reconstruction techniques [15, 19, 38],
dedicated exploration models [34, 17], the utilization of self-attention
mechanisms [40], and the analysis of entropy [29]. AVE has also been explored
in the context of segmentation [39] and 3D vision [16].

Our work contributes to this landscape by introducing a novel approach that
recycles computations within a transformer-based architecture.

Figure 2: We introduce TORE, an approach for efficient sequential decision-
making with vision transformers in an iterator-aggregator regime. Firstly, the
selection policy collects a successive glimpse ($G^{4}$). Then, the glimpse is
divided into patches, which are tokenized and pushed through the iterator
path. We recycle computations by caching and reusing midway tokens that are
passed through iterator once. Finally, the prediction is made by the
classification head $\psi$ which is the last block of aggregator path. Note
that, the classification token $c$ is defined as the average from all
classification tokens generated by the iterator.

##  3 Method

In subsequent sections, firstly, we recall the overall structure of
transformer-based neural networks to make this paper self-contained, then we
introduce TORE, a ViT in iterator-aggregator regime. Finally, we discuss the
TORE training policy.

#### Vision Transformers.

Transformer model $M$ of depth $n$ can be decomposed as a consecutive
application of the following mappings:

  * •

tokenizer $\tau:\mathcal{X}\to\mathcal{R}$,

  * •

transformer blocks $\varphi_{i}:\mathcal{R}\to\mathcal{R}$, for
$i\in\\{1,\dots,n\\}$,

  * •

prediction head $\psi:\mathcal{R}\to\mathcal{Y}$.

where $\mathcal{X}$ is an input domain (i.e. set of images), $\mathcal{Y}$ is
an output domain (i.e. logit values), and $\mathcal{R}$ is a representation
space. For a given $x\in\mathcal{X}$, tokenizer $\tau$ generates a token set
that contains [CLS], a special global information token denoted as $c$, and
$T$, a set of remaining patch-corresponding tokens.

#### Sequential prediction.

Suppose that input $x$ consists of $p$ parts which we receive sequentially,
ie. $x~{}=~{}\\{x^{(1)},\dots,x^{(p)}\\}$ and at time $j$, the parts $x^{(i)}$
for $i\leq j$ are available. In such scenarios, we are interested in
predictions $M^{(j)}(x)=M\left(x^{(1)},\dots,x^{(j)}\right)$ computed at each
time new part $x^{(j)}$ arrives, for $1\leq j\leq p$.

####  Iterator-aggregator regime.

As presented in Figure 2, we divide the transformer $M$ into two paths
iterator $M_{I}$ and aggregator $M_{A}$:

$\displaystyle M_{I}$
$\displaystyle=\varphi_{\kappa}\circ\dots\circ\varphi_{1}\circ\tau$
$\displaystyle M_{A}$
$\displaystyle=\psi\circ\varphi_{n}\circ\dots\circ\varphi_{\kappa+1},$ (1)

where a parameter $\kappa\in\\{0,1,\dots,n\\}$ is the number of transformer
blocks assigned to iterator $M_{I}$, while aggregator $M_{A}$ is built from
remaining $n-\kappa$ blocks. Given a sample $x^{(j)}$ the iterator produces
midway tokens which we denote as $\\{c^{(j)}\\}\cup T^{(j)}:=M_{I}(x^{(j)})$.
We define the TORE forward pass with parameter $\kappa$ at time $j$ as

$\small\operatorname{TORE}_{\kappa}^{(j)}(x)=M_{A}\left(\Lambda\left\\{M_{I}(x^{(1)}),\dots,M_{I}(x^{(j)})\right\\}\right),$
(2)

where

$\small\Lambda\left\\{M_{I}(x^{(1)}),\dots,M_{I}(x^{(j)})\right\\}=\Bigl{\\{}\frac{1}{j}\sum_{i=1}^{j}c^{(i)}\Bigr{\\}}\cup\bigcup_{i=1}^{j}T^{(i)}.$
(3)

Therefore, we apply iterator to each input part $x^{(i)}$. Then, we combine
the midway tokens by averaging $c^{(i)}$ tokens and the union over patch-
corresponding tokens $T^{(i)}$. Finally, we produce the output with
aggregator.

#### TOken REcycling (TORE).

To efficiently compute the Eq. 2, we cache outcomes of already calculated
forward passes at times $i<j$. We update the cache as follows:

$\displaystyle\operatorname{cache}^{(0)}_{c}$ $\displaystyle=0$
$\displaystyle\operatorname{cache}^{(0)}_{T}$ $\displaystyle=\emptyset$
$\displaystyle\operatorname{cache}^{(j)}_{c}$
$\displaystyle=\frac{1}{j}c^{(j)}+\frac{j-1}{j}\operatorname{cache}^{(j-1)}_{c}$
$\displaystyle\operatorname{cache}^{(j)}_{T}$
$\displaystyle=T^{(j)}\cup\operatorname{cache}^{(j-1)}_{T},$ (4)

and then calculate the prediction as

$\operatorname{TORE}_{\kappa}^{(j)}(x)=M_{A}(\operatorname{cache}^{(j)}).$ (5)

It enables us to compute consecutive values of
$\operatorname{TORE}_{\kappa}^{(j)}(x)$ by processing each $x^{(j)}$ with
iterator only once. It is in contrast to ViT’s original forward pass where at
each timestep $j$ a full collection of inputs $\\{x^{(1)},\dots,x^{(j)}\\}$ is
processed by network $M$.

Figure 3: The image illustrates our training policy, in which we sample
$\kappa$ from the uniform distribution for each batch to ensure the model’s
flexibility during inference.

#### Training policy.

When training with TORE we sample the size of the iterator from the uniform
distribution per batch $\kappa~{}\sim~{}\mathcal{U}\\{0,\dots,n\\}$ as
illustrated in Figure 3. As a consequence, we obtain a single model that
during inference reliably works for multiple values of $\kappa$ making the
method flexible. This property allows us to balance between accuracy and
efficiency on demand, i.e. to reduce the computations we increase the iterator
size $\kappa$ which results in the decrease of the model’s accuracy.

##  4 Experimental Setup

We conduct the experiments in three task categories. The first one is Active
Visual Exploration (AVE), where we use a pretrained Masked Autoencoder (MAE)
fine-tuned for various downstream exploration techniques. The second one is
image classification, where we use a pretrained DeiT and inspect the results
of fine-tuning on different target datasets. In the third one, we inspect the
utilization of emergent properties by analyzing the performance of DINO on
classification with $k$-NN and video segmentation.

To reliably measure the effectiveness of our methods in different task
categories we are using in total $7$ datasets: CIFAR10 [20], CIFAR100 [20],
Flowers102 [27], SUN360 [46], ImageNet-1k [37], ImageNet-v2 [36], DAVIS-2017
[30]. Additionally, we use three different pretraining strategies for ViT: MAE
[14], DeiT [43], and DINO [8]. When training the model with TORE forward pass,
we uniformly sample $\kappa$ value per batch.

In the following subsections, we present details on the experimental setup for
each task category. Additionally, in Supplementary Materials, we provide more
details on used hyperparameters.

###  4.1 Active Visual Exploration

We perform the experiments on an encoder-decoder ViT architecture with
$16\times 16$ pixel-sized patches, where the encoder is a ViT–B pretrained on
ImageNet-1k. For the decoder we use 1 transformer block with 4 heads and 128
hidden dimensions, contrary to [29], where they use ViT–L as an encoder and
the original 8 blocks MAE decoder. A linear classifier is used on [CLS] token
from the encoder’s last layer. We refer to this variant of AME [29] as AME++.
As in [29] we initialize the model weights with MAE and fine-tune the model
for the exploration-classification task.

We test our approach on CIFAR100, CIFAR10, and SUN360 datasets. During
training and evaluation, we sample non-overlapping $2\times 2$ token-sized
glimpses which in total cover about $25\%$ of an input image as shown in
Figure 4. When training, we sample the glimpses uniformly at random, and
during evaluation, we sample the glimpses according to the chosen policy.

We consider two glimpse selection policies, a random one as a baseline and one
based on Attention Map Entropy (AME) [29], as we are not proposing a new one.
The AME policy leverages the internal uncertainty of the reconstruction
decoder to determine the most informative observations, ie. the policy selects
glimpses achieving the highest attention entropy. Details on AME exploration
policy are included in the Supplementary Materials. We follow the evaluation
protocol introduced in [29] for all experiments in the AVE setup.

Figure 4: The image illustrates the process of dividing images into glimpses and parts and provides an example of their input. In Active Visual Exploration (top row) experiments, we analyze sequences of non-overlapping, $2\times 2$ token-sized glimpses that cover approximately 25% of the image area (left column). For other tasks (bottom row), we divide the image into non-overlapping, nearly equally sized rectangular parts, covering the entire 100% of the image area. Here we divide the image into 8 parts. Dataset | Glimpse Regime | Architecture | Method | GFLOPs $\downarrow$ | Accuracy $\uparrow$
---|---|---|---|---|---
SUN360 | $8\times 48^{2}$ | custom | AttSeg [39] | NA111Due to the lack of shared code and details in the AttSeg paper, we are not able to estimate the required GFLOPs. | 52.6
ResNet-18 | GlAtEx [40] | 55.8 | 67.2
ViT–L | AME [29] | 128.9 | 75.7
$8\times 32^{2}$ | ViT–B | MAE [14] | 13.0 | 72.2
ViT–L | AME [29] | 73.7 | 73.4
ViT–B | AME++ | 13.6 | 76.2
TORE + AME | 8.0 | 77.2
CIFAR100 | $12\times 32^{2}$ | ViT–B | MAE [14] | 27.8 | 75.8
AME++ | 28.6 | 77.9
TORE + AME | 19.2 | 78.9
Table 2: The accuracy and computational efficiency of various AVE methods are
compared across the SUN360 and CIFAR100 datasets. Our method is superior in
both cases. What is more, our approach achieves superior accuracy in the
SUN360 dataset even with a smaller ViT-B compared to ViT-L for AME, while
reducing the computational load. Note that, AME++ is a version of AME with a
changed decoder as described in Secion 4.1.

###  4.2 Classification and emergent properties

In this section, we consider divisions of input images into $p$ nearly
equally-sized, non-overlapping, rectangular regions, covering the whole image
in total as shown in Figure 4, where the $p\in\\{2,3,4,6,8,9,12,16\\}$. During
the training of TORE, $p$ is uniformly sampled per batch.

As the regions fully cover the initial image, sampling $p$ doesn’t affect the
original forward pass. Only when $\kappa\neq 0$ in the TORE forward pass the
choice of $p$ impacts model prediction.

#### Sequential Classification.

We perform the experiments on ViT–S architecture with $16\times 16$ pixel-
sized patches and linear classifier on top of [CLS] token from the transformer
block. We initialize the model with DeiT weights trained on ImageNet-1K and
fine-tune the model on target datasets.

We test our approach on Flowers102, CIFAR10, CIFAR100, and ImageNet-1k
datasets, including the ImageNet-v2 MatchedFrequency validation split. In all
experiments, we resize the images to $224\times 224$ resolution and randomly
permute image parts to create a sequence.

#### Utilization of emergent properties.

We perform the experiments on ViT–S with $8\times 8$ pixel-sized patches. We
initialize the model with DINO weights trained on ImageNet-1k and fine-tune
the model. We use the TORE forward pass in the student network and leave the
teacher network unchanged. We test our approach on ImageNet-1k and the
DAVIS-2017 datasets and resize the images to $224\times 224$ and $480p$ as in
[8], respectively.

##  5 Results

To show the effectiveness of our method, we compare the accuracy and resource
utilization within both sequential classification and Active Visual
Exploration regimes, detailed in Section 4. We integrate TORE with current
state-of-the-art methods and present the results in Table 1 and Table 2 for
sequential classification and Active Visual Exploration, respectively.

#### For Active Visual Exploration TORE achieves state-of-the-art accuracy.

The presented results indicate that for both tasks, the TORE improves the
accuracy and reduces the computational burden compared to the baseline method
on all tested datasets (CIFAR10 results are in Supplementary Materials).

Specifically, in Active Visual Exploration, our method is superior compared to
the current state-of-the-art AME [29]. Remarkably, even with a smaller variant
of ViT (using ViT–B instead of ViT–L architecture) for the SUN360 dataset.
Moreover, it exhibits greater resource efficiency when contrasted with GlAteX
[40], which utilizes a convolutional backbone.

Additionally, our modification of the decoder in MAE for the AVE task,
detailed in Section 4.1, enhances the AME method (referred to as AME++).

###  5.1 Ablations and analysis

Figure 5: Classification results of baseline and TORE models with the AME
selection policy on Active Visual Exploration across various values of
$\kappa$ (top row) and corresponding computational load (bottom row). The
model with TORE demonstrates a more gradual decrease in accuracy compared to
the baseline. Note that, we can save up to 30% of computation without
substantial loss in accuracy.

#### Through the increase of iterator size, we can save up to 30% of
computations without substantial loss in accuracy.

To assess the trade-off between accuracy and efficiency, in Figure 5 we
illustrate the impact of iterator size $\kappa$ on accuracy (top row) and the
associated with $\kappa$ computational load on accuracy (bottom row). One can
observe, that the change in accuracy for $\kappa<6$ is barely noticeable,
showing that computational reduction is feasible without a significant loss in
model accuracy. For both datasets, the decrease in accuracy for the TORE
variant of AME is considerably smaller than for the baseline method. The TORE
maintains an accuracy level above 60% while the accuracy of the baseline model
is below 50% when $\kappa$ increases.

#### TORE does not negatively influence the exploration in the AVE task while
reducing computations.

To check the influence of the TORE on the exploration part of AVE we conduct
classification using the original ViT forward pass ($\kappa=0$). However, for
the glimpse selection policy, we input results from models with TORE for
various $\kappa$ values. The outcomes are illustrated in Figure 6.

Across both, CIFAR100 and SUN360, datasets, augmenting AME with TORE results
in a minimal drop in accuracy with the increase of $\kappa$. It is in contrast
to the baseline AME, which experiences a drop of accuracy up to two points.
However, the required GFLOPs for exploration can be reduced by a factor of
four. Those results indicate that the TORE does not negatively influence the
exploration part of the AVE task, but at the same time, it substantially
reduces the amount of needed computations.

Dataset | Model | Visible input
---|---|---
$12.5\%$ | $25\%$ | $50\%$
CIFAR100 | DeiT | 46.2 | 65.6 | 81.6
TORE + DeiT | 48.8 | 68.3 | 82.7
ImageNet-1k | DeiT | 37.7 | 56.8 | 72.0
TORE + DeiT | 40.0 | 59.2 | 73.3
Table 3: Classification accuracy of the DeiT and TORE + DeiT models across two
datasets with different portions of visible input with $\kappa=0$. A model
trained with the TORE showcases improved performance in a partial input regime
without explicit training. Figure 6: Classification results using the original
forward pass on glimpses selected based on Attention Map Entropy (AME) with
given $\kappa$. Choosing high values of $\kappa$ affects the results of TORE-
trained model marginally in contrast to the baseline method. Additionally,
with TORE, we can reduce the computational load for exploration by a factor of
$4$.

#### TORE is more robust when fewer glimpses are available.

Figure 7 illustrates the behavior of the AVE model across varying numbers of
available glimpses. On both datasets, the combination of the TORE with AME
achieves superior performance compared to other approaches, including AME and
random glimpse selection. This is observable at each point of the sequence.

#### TORE performs with higher fidelity on partial inputs.

In assessing the impact of the TORE on the model’s performance with partial
inputs, we examined the accuracy with varying availability of image parts
(12.5%, 25%, and 50%) on CIFAR100 and ImageNet-1k. The results are presented
in Table 3, showing that DeiT trained with our policy exhibits greater
resilience to partial inputs.

#### TORE is more robust to fragmented inputs.

To examine the influence of image division on model accuracy, we conduct
inference using various $p$ values and iterator size $\kappa$. Figure 8 shows
that when $p$ increases, the accuracy decreases. Additionally, the model
trained with TORE consistently mitigates the decline in accuracy across all
$p$ and $\kappa$ when compared to vanilla DeiT. These indicate that the TORE
enhances the model’s robustness to fragmented inputs.

Model | $\kappa$ | CIFAR10 | CIFAR100 | Flowers102 | ImageNet-1k | ImageNet-v2
---|---|---|---|---|---|---
DeiT | 0 | 99.0 | 90.5 | 96.0 | 81.4 | 70.0
TORE + DeiT | 98.9 | 90.3 | 95.8 | 81.3 | 70.3
DeiT | 4 | 97.8 | 86.8 | 93.3 | 77.9 | 65.4
TORE + DeiT | 98.6 | 89.4 | 95.1 | 80.6 | 69.1
DeiT | 8 | 90.7 | 72.0 | 85.0 | 66.5 | 53.2
TORE + DeiT | 96.4 | 83.2 | 92.4 | 75.3 | 62.8
Table 4: Classification accuracy across five datasets using three distinct
$\kappa$ values after fine-tuning of DeiT with and without TORE. The influence
of our training policy on the accuracy with the original forward pass
($\kappa=0$) has a marginal impact. However, as $\kappa$ increases, the model
using TORE exhibits a greater robustness compared to the original DeiT. Figure
7: Active Visual Exploration results using glimpse selection strategy based on
AME [29] and random policy (RAND). Our method (TORE) improves classification
accuracy across glimpse selection strategies and the number of visible
glimpses.

#### Fine-tuning with TORE does not result in a significant decrease in DeiT
performance.

To assess fine-tuning with TORE, we evaluate DeiT performance on the following
datasets: CIFAR10, CIFAR100, Flowers102, ImageNet-1k, and ImageNet-v2
validation split.

The fine-tuning results for different $\kappa$ values during inference are
presented in Table 4. When $\kappa=0$, there is a marginal change between both
models. However, as $\kappa$ increases, the TORE consistently outperforms the
baseline. Ergo, we conclude that the usage of TORE when fine-tuning does not
result in a significant decrease in its performance.

#### End-to-end training with TORE results in the drop of accuracy smaller
than 1%.

To check the effectiveness of TORE in learning ViT from scratch, we train a
model using TORE + DeiT policy for $800$ epochs. The training is stable, and
the model achieves $80.5\%$ top-1 accuracy on ImageNet-1k. It is slightly
below the baseline accuracy of $81.3\%$. However, the loss function of the
TORE variant does not reach the plateau, indicating a potential for improved
performance. Nonetheless, for a fair comparison, we terminated the training
after the same number of epochs as DeiT converged.

#### TORE preserves emerging properties of DINO.

We check if DINO fine-tuned with TORE keeps its emerging properties. We
conduct $k$-NN classification together with visualization of a latent space,
and video instance segmentation. More results are presented in the
Supplementary Materials.

Figure 8: Classification accuracy on ImageNet-1k images divided into $p$ parts
for DeiT and TORE + DeiT using TORE forward pass with different values of
$\kappa$. DeiT fine-tuned with TORE achieves nearly equal results to DeiT for
$\kappa=0$ while being more resilient to increase of both $p$ and $\kappa$.

Figure 9 demonstrates that TORE + DINO training policy retains high $k$-NN
classification results of the original model for $\kappa=0$ and does not
experience drastic performance drops with larger iterators. Moreover, TORE +
DINO shows better class separability in the latent space when $\kappa>0$, as
is visible in t-SNE in Figure 10.

Self-supervised ViTs can perform scene segmentation. To check if we preserve
this ability, we follow the setup from [8]. The results are presented in Table
5 with examples in Figure 11, showing a comparison between DINO with and
without the TORE on the DAVIS-2017 dataset. Both methods perform comparably
indicating the preservation of the scene segmentation property.

##  6 Conclusions and future works

In this work, we propose a TOken REcycling (TORE) approach for efficient
sequential inference that splits the ViT model into two paths: iterator and
aggregator. Although the idea of splitting the inference into two paths is not
new, the existing approaches treat extractors and aggregators as separate
networks, limiting the potential usage of already trained models which is in
contrast to our approach.

Figure 9: Accuracy for $k$-NN classifier on top of DINO and TORE + DINO on
ImageNet-1k. Our method is significantly robust to the changes of the iterator
size $\kappa$.

As a result, the TORE substantially reduces the amount of needed computations
while preserving or even improving the accuracy for tasks such as Active
Visual Exploration.

In future work, we will explore further reduction of computations, such as
modifying the aggregator, e.g. by integrating an attention pooling mechanism.
Additionally, we intend to refine the proposed regime by incorporating early
exits to further trim the computational burden.

Figure 10: t-SNE visualizations of [CLS]tokens generated for the CIFAR10 test
set (colors correspond to classes). We observe that with the increased size of
the iterator, samples from different classes are better separated for TORE
variant of DINO.

#### Limitations.

The primary limitation of the study lies in the fixed nature of the image
divisions and masks used in the experiments, constrained by the size of the
image patches in the ViT model. However, in future work, we will explore the
impact of more random patch sizes on the model to better understand the
model’s behavior.

#### Impact.

This work impacts the fields of embodied AI, robot vision, and efficient
machine learning. It shows the potential of a straightforward yet powerful
modification in the forward pass and training of ViTs, significantly reducing
the computational load of large models and enabling efficient computations on
devices such as drones.

Model | $(\mathcal{J}\&\mathcal{F})_{m}\uparrow$ | $\mathcal{J}_{m}\uparrow$ | $\mathcal{F}_{m}\uparrow$
---|---|---|---
DINO | 70.9 | 67.7 | 74.1
TORE + DINO | 70.7 | 67.6 | 73.7
Table 5: Scene segmentation results on DAVIS-2017 for DINO and TORE + DINO
($\kappa=0$). In terms of mean region similarity $\mathcal{J}_{m}$ and mean
contour-based accuracy $\mathcal{F}_{m}$ both models perform similarly. It
shows that the model trained with our policy preserves the scene segmentation
property of DINO. Figure 11: Examples of video instance segmentation from
DAVIS-2017 dataset. DINO trained with TORE preserves the original property of
the model of scene segmentation.

## Acknowledgements

The work of Jan Olszewski and Mateusz Pach was funded by the National Science
Centre (Poland) grant no. 2022/47/B/ST6/03397. Dawid Rymarczyk was supported
by the National Science Centre (Poland), grant no. 2022/45/N/ST6/04147.
Bartosz Zieliński was funded by the Horizon Europe Programme (HORIZON-
CL4-2022-HUMAN-02) under the project “ELIAS: European Lighthouse of AI for
Sustainability”, grant no. 101120237.

We gratefully acknowledge Polish high-performance computing infrastructure
PLGrid (HPC Centers: ACK Cyfronet AGH) for providing computer facilities and
support within computational grant no. PLG/2023/016555.

## References

  * Aloimonos et al. [1988] John Aloimonos, Isaac Weiss, and Amit Bandyopadhyay.  Active vision.  _International Journal on Computer Vision._ , 1(4):333–356, 1988.
  * Assran et al. [2023] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas.  Self-supervised learning from images with a joint-embedding predictive architecture.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15619–15629, 2023.
  * Bao et al. [2022] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei.  BEiT: BERT pre-training of image transformers.  In _International Conference on Learning Representations_ , 2022.
  * Beltagy et al. [2020] Iz Beltagy, Matthew E. Peters, and Arman Cohan.  Longformer: The long-document transformer.  _arXiv:2004.05150_ , 2020.
  * Busemeyer and Diederich [2002] Jerome R Busemeyer and Adele Diederich.  Survey of decision field theory.  _Mathematical social sciences_ , 43(3):345–370, 2002.
  * Cai et al. [2023] Han Cai, Chuang Gan, and Song Han.  Efficientvit: Enhanced linear attention for high-resolution low-computation visual recognition.  _International Conference on Computer Vision (ICCV)_ , 2023.
  * Carion et al. [2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _European Conference on Computer Vision (ECCV)_ , pages 213–229, 2020.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2021.
  * Carroll et al. [2022] Micah Carroll, Orr Paradise, Jessy Lin, Raluca Georgescu, Mingfei Sun, David Bignell, Stephanie Milani, Katja Hofmann, Matthew Hausknecht, Anca Dragan, et al.  Uni [mask]: Unified inference in sequential decision problems.  _Advances in neural information processing systems_ , 35:35365–35378, 2022.
  * Chen et al. [2021] Xinlei Chen, Saining Xie, and Kaiming He.  An empirical study of training self-supervised vision transformers.  _CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  _ICLR_ , 2021.
  * Gu et al. [2018] Jiuxiang Gu, Zhenhua Wang, Jason Kuen, Lianyang Ma, Amir Shahroudy, Bing Shuai, Ting Liu, Xingxing Wang, Gang Wang, Jianfei Cai, et al.  Recent advances in convolutional neural networks.  _Pattern recognition_ , 77:354–377, 2018.
  * Guo et al. [2019] Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang.  Star-transformer.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 1315–1325. Association for Computational Linguistics, 2019.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 16000–16009, 2022.
  * Jayaraman and Grauman [2018a] Dinesh Jayaraman and Kristen Grauman.  Learning to look around: Intelligently exploring unseen environments for unknown tasks.  In _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR_ , pages 1238–1247, 2018a.
  * Jayaraman and Grauman [2018b] Dinesh Jayaraman and Kristen Grauman.  Learning to look around: Intelligently exploring unseen environments for unknown tasks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1238–1247, 2018b.
  * Jha et al. [2023] Abhishek Jha, Soroush Seifi, and Tinne Tuytelaars.  Simglim: Simplifying glimpse based active visual reconstruction.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 269–278, 2023.
  * Jing et al. [2022] Taotao Jing, Haifeng Xia, Renran Tian, Haoran Ding, Xiao Luo, Joshua Domeyer, Rini Sherony, and Zhengming Ding.  Inaction: Interpretable action decision making for autonomous driving.  In _European Conference on Computer Vision_ , pages 370–387. Springer, 2022.
  * Kendall and Gal [2017] Alex Kendall and Yarin Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  In _Advances in Neural Information Processing Systems 30, NIPS_ , pages 5574–5584, 2017.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * LeCun et al. [1995] Yann LeCun, Yoshua Bengio, et al.  Convolutional networks for images, speech, and time series.  _The handbook of brain theory and neural networks_ , 3361(10):1995, 1995.
  * Liu et al. [2020] Liang Liu, Hao Lu, Hongwei Zou, Haipeng Xiong, Zhiguo Cao, and Chunhua Shen.  Weighing counts: Sequential crowd counting by reinforcement learning.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part X 16_ , pages 164–181. Springer, 2020.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Lu et al. [2022] Hao Lu, Liang Liu, Hu Wang, and Zhiguo Cao.  Counting crowd by weighing counts: A sequential decision-making perspective.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.
  * Mnih et al. [2014] Volodymyr Mnih, Nicolas Heess, Alex Graves, et al.  Recurrent models of visual attention.  _Advances in neural information processing systems_ , 27, 2014.
  * Nilsback and Zisserman [2008] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In _2008 Sixth Indian conference on computer vision, graphics & image processing_, pages 722–729. IEEE, 2008.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  _arXiv preprint arXiv:2304.07193_ , 2023.
  * Pardyl et al. [2023] Adam Pardyl, Grzegorz Rypesc, Grzegorz Kurzejamski, Bartosz Zielinski, and Tomasz Trzcinki.  Active visual exploration based on attention-map entropy.  In _Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI_ , pages 1303–1311, 2023.
  * Pont-Tuset et al. [2017] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool.  The 2017 davis challenge on video object segmentation.  _arXiv preprint arXiv:1704.00675_ , 2017.
  * Przewiezlikowski et al. [2022] Marcin Przewiezlikowski, Marek Smieja, Lukasz Struski, and Jacek Tabor.  Misconv: Convolutional neural networks for missing data.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 2060–2069, 2022.
  * Qin et al. [2020] Zhen Qin, Yicheng Cheng, Zhe Zhao, Zhe Chen, Donald Metzler, and Jingzheng Qin.  Multitask mixture of sequential experts for user activity streams.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 3083–3091, 2020.
  * Ramakrishnan and Grauman [2018] Santhosh K Ramakrishnan and Kristen Grauman.  Sidekick policy learning for active visual exploration.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 413–430, 2018.
  * Rangrej et al. [2022] Samrudhdhi B Rangrej, Chetan L Srinidhi, and James J Clark.  Consistency driven sequential transformers attention model for partially observable scenes.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2518–2527, 2022.
  * Rao et al. [2021] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh.  Dynamicvit: Efficient vision transformers with dynamic token sparsification.  In _Advances in Neural Information Processing Systems 34 (NeurIPS)_ , pages 13937–13949, 2021.
  * Recht et al. [2019] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do imagenet classifiers generalize to imagenet?  In _International conference on machine learning_ , pages 5389–5400. PMLR, 2019.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  _International Journal of Computer Vision (IJCV)_ , 115(3):211–252, 2015.
  * Seifi and Tuytelaars [2019] Soroush Seifi and Tinne Tuytelaars.  Where to look next: Unsupervised active visual exploration on 360 $\\{$$\backslash$deg$\\}$ input.  _arXiv preprint arXiv:1909.10304_ , 2019.
  * Seifi and Tuytelaars [2020] Soroush Seifi and Tinne Tuytelaars.  Attend and segment: Attention guided active semantic segmentation.  In _European Conference in Computer Vision (ECCV)_ , pages 305–321, 2020.
  * Seifi et al. [2021] Soroush Seifi, Abhishek Jha, and Tinne Tuytelaars.  Glimpse-attend-and-explore: Self-attention for active visual exploration.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021.
  * Sochman and Matas [2005] Jan Sochman and Jiri Matas.  Waldboost-learning for time constrained sequential detection.  In _2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05)_ , pages 150–156. IEEE, 2005.
  * Supancic III and Ramanan [2017] James Supancic III and Deva Ramanan.  Tracking as online decision-making: Learning a policy from streaming videos with reinforcement learning.  In _Proceedings of the IEEE international conference on computer vision_ , pages 322–331, 2017.
  * Touvron et al. [2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou.  Training data-efficient image transformers & distillation through attention.  In _International Conference on Machine Learning (ICML)_ , pages 10347–10357, 2021.
  * Touvron et al. [2022] Hugo Touvron, Matthieu Cord, and Hervé Jégou.  Deit iii: Revenge of the vit.  In _European Conference on Computer Vision_ , pages 516–533, 2022.
  * Xiang et al. [2015] Yu Xiang, Alexandre Alahi, and Silvio Savarese.  Learning to track: Online multi-object tracking by decision making.  In _Proceedings of the IEEE international conference on computer vision_ , pages 4705–4713, 2015.
  * Xiao et al. [2012] Jianxiong Xiao, Krista A Ehinger, Aude Oliva, and Antonio Torralba.  Recognizing scene viewpoint using panoramic place representation.  In _2012 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 2695–2702, 2012.
  * Yang and Nachum [2021] Mengjiao Yang and Ofir Nachum.  Representation matters: Offline pretraining for sequential decision making.  In _International Conference on Machine Learning_ , pages 11784–11794. PMLR, 2021.
  * Yin et al. [2022a] Hongxu Yin, Arash Vahdat, Jose M. Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov.  A-ViT: Adaptive tokens for efficient vision transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10809–10818, 2022a.
  * Yin et al. [2022b] Hongxu Yin, Arash Vahdat, José M. Álvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov.  A-vit: Adaptive tokens for efficient vision transformer.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR_ , pages 10799–10808, 2022b.
  * You et al. [2019] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh.  Large batch optimization for deep learning: Training bert in 76 minutes.  _arXiv preprint arXiv:1904.00962_ , 2019.
  * Yu et al. [2017] Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao.  On compressing deep models by low rank and sparse decomposition.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_ , pages 67–76. IEEE Computer Society, 2017.
  * Yun et al. [2022] Sukmin Yun, Hankook Lee, Jaehyung Kim, and Jinwoo Shin.  Patch-level representation learning for self-supervised vision transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8354–8363, 2022.
  * Zhang et al. [2018] Jianfu Zhang, Naiyan Wang, and Liqing Zhang.  Multi-shot pedestrian re-identification via sequential decision making.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6781–6789, 2018.
  * Zheng et al. [2021] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al.  Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 6881–6890, 2021.
  * Zhou et al. [2022] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong.  ibot: Image bert pre-training with online tokenizer.  _International Conference on Learning Representations (ICLR)_ , 2022.
  * Zhu et al. [2020] Yi Zhu, Fengda Zhu, Zhaohuan Zhan, Bingqian Lin, Jianbin Jiao, Xiaojun Chang, and Xiaodan Liang.  Vision-dialog navigation by exploring cross-modal memory.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10730–10739, 2020.

## Supplementary Materials

### Overview

In the Supplementary Material, we present details of the models used to
evaluate TORE method on each task together with extra corresponding results.
Information about datasets is also included in Table 12.

### Active Visual Exploration

#### Loss functions

The loss function for MAE models is Cross Entropy classification loss. For AME
models it is a sum of Cross Entropy classification loss and image
reconstruction loss.

#### Datasets

We use the SUN360 dataset with $128\times 256$-sized images. In the training,
we select 8 random, non-overlapping $2\times 2$ patch-sized glimpses.
Moreover, we use CIFAR10 and CIFAR100 datasets with $224\times 224$-sized
images. This time, in the training, we select 12 random, non-overlapping
$2\times 2$ patch-sized glimpses.

#### Training Hyperparameters

We train all models with the same set of hyperparameters and pick models that
achieve the best validation loss. The exact values of hyperparameters are
shown in Table 6.

#### Attention Map Entropy selection policy [29]

Consider attention matrix $A$ produced by the reconstruction decoder, where
$A_{ij}$ is an attention map of token $i$ to token $j$. Note that
$\sum_{j}A_{ij}=1$, therefore we can interpret the $i$-th row of a matrix
$A_{i:}$ as a probability distribution. For the next glimpse, we select the
token indexed by $i^{*}$, which is the minimizer of Shanon Entropy over
attention matrix rows $i^{*}=argmin_{i}\\{H(A_{i:})\\}$, such that $i^{*}$ is
not already a seen glimpse.

#### Additional AVE Results

We present more detailed results for AVE classification, which show the
influence of the number of visible glimpses $j$ and iterator size $\kappa$.
The ones for AVE with random selection policy (MAE) are presented in Table 13
and the ones for AVE with AME [29] selection policy in Table 14.

### Classification

#### Training Hyperparameters

We evaluate all models after training for a fixed number of epochs using the
hyperparameters in Tables 7, 8, and 9.

#### Additional Sequential Classification Results

We assess the influence of TORE on models trained on the classical supervised
classification tasks, such as DeiT. In the sequential classification task, we
divide an image into $p=16$ parts (maximal number of parts in TORE training)
and sequentially provide models with consecutive image parts in random order.
We record the accuracy of DeiT and TORE + DeiT at different numbers of visible
parts $j$. For detailed results see Table 15.

### Impact on ViTs Emerging Properties

#### Training Hyperameters

To evaluate the impact of TORE training policy on DINO emerging properties, we
initialize the weights from a checkpoint pretrained on ImageNet-1k and
finetune it for a small number of epochs. During finetuning, we uniformly
sample student iterator size $\kappa$ and set fixed $\kappa=0$ for a momentum
teacher. The exact values of hyperparameters are shown in Table 10. We do not
perform any additional finetuning on the CIFAR10 dataset.

#### Datasets

We use a validation set of the DAVIS-2017 dataset with original frame sizes,
following the exact protocol implemented in [8]. We keep constant image
division number $p=16$ for each frame, adjusting mask size for varying
heights. To generate t-SNE visualization, we use a validation set of CIFAR10
with images resized to $224\times 224$.

####  $k$-NN classification

We report detailed $k$-NN classification results on ImageNet-1k results with
$k=20$ across division parts number $p=8$, $p=16$ and
$\kappa\in\\{0,\ldots,12\\}$ in Table 16 and Figure 12.

Hyperparameter | Value
---|---
Epochs | $100$
Batch size | 16
Learning rate | $10^{-5}$
Learning rate scheduler | cosine
Minimal learning rate | $10^{-8}$
Optimizer | AdamW [24]
Betas |  $0.9,0.999$ (default)
Epsilon |  $10^{-8}$ (default)
Weight decay | $0$
Table 6: Hyperparameters used to find models for AVE. Hyperparameter | Value
---|---
Epochs | $100$
Batch size | $256$
Learning rate | $10^{-6}$
Learning rate scheduler | off
Optimizer | AdamW [24]
Betas |  $0.9,0.999$ (default)
Epsilon |  $10^{-8}$ (default)
Weight decay | $0.05$
Label smoothing | off
Drop path | $0.05$
Mixup | $0.8$
Cutmix | $1.0$
Color-jitter | $0.3$
DeiT’s Three Augment | on
Table 7: Hyperparameters used to fine-tune the TORE + DeiT model on ImageNet-1k. Hyperparameter | Value
---|---
Epochs |  $1000$ (Flowers), $100$ (CIFARs)
Batch size | $256$
Learning rate | $10^{-5}$
Learning rate scheduler | off
Warmup learning rate | $10^{-6}$
Warmup epochs | $5$
Optimizer | AdamW [24]
Betas |  $0.9,0.999$ (default)
Epsilon |  $10^{-8}$ (default)
Weight decay | $0.05$
Label smoothing | off
Drop path | $0.05$
Mixup | $0.8$
Cutmix | $1.0$
Color-jitter | $0.3$
DeiT’s Three Augment | on
Table 8: Hyperparameters used to fine-tune the DeiT and TORE + DeiT models on Flowers, CIFAR10, and CIFAR100. Hyperparameter | Value
---|---
Epochs | $800$
Batch size | $256$
Learning rate | $4\cdot 10^{-3}$
Learning rate scheduler | cosine
Minimal learning rate | $10^{-5}$
Warmup learning rate | $10^{-6}$
Warmup epochs | $5$
Optimizer | Lamb [50]
Betas |  $0.9,0.999$ (default)
Epsilon |  $10^{-6}$ (default)
Weight decay | $0.05$
Label smoothing | off
Drop path | $0.05$
Mixup | $0.8$
Cutmix | $1.0$
Color-jitter | $0.3$
DeiT’s Three Augment | on
Table 9: Hyperparameters used to train the TORE + DeiT model end-to-end. Hyperparameter | Value
---|---
Epochs | $15$
Batch size | $16$
Patch size | $8$
Normalize last layer | off
Learning rate | $10^{-6}$
Learning rate scheduler | off
Optimizer | AdamW [24]
Betas | $0.9,0.999$
Epsilon | $10^{-8}$
Weight decay | $0.04$
Drop path rate | $0.01$
Local crops number | $10$
Teacher momentum | $0.996$
Table 10: Hyperparameters used to finetune the DINO model on ImageNet-1k in TORE pretraining regime. Model | $\kappa$ | $(\mathcal{J}\&\mathcal{F})_{m}\uparrow$ | $\mathcal{J}_{m}\uparrow$ | $\mathcal{F}_{m}\uparrow$
---|---|---|---|---
DINO | 0 | 70.9 | 67.7 | 74.1
TORE + DINO | 70.7 | 67.6 | 73.7
DINO | 4 | 61.1 | 57.9 | 64.3
TORE + DINO | 66 | 63.2 | 68.7
DINO | 8 | 62.1 | 59.3 | 64.9
TORE + DINO | 63.7 | 60.7 | 66.6
Table 11: Scene segmentation results on DAVIS-2017 for DINO and TORE + DINO for different iterators. Note that object segmentation requires reordering of spatial patches. Task | Dataset | Train size | Test size | #classes
---|---|---|---|---
Active Visual Exploration | CIFAR10 [20] | 50,000 | 10,000 | 10
CIFAR100 [20] | 50,000 | 10,000 | 100
SUN360 [46] | 7,161 | 795 | 26
Sequential Classification | CIFAR100 [20] | 50,000 | 10,000 | 100
ImageNet-1k [37] | 1,281,167 | 50,000 | 1000
Classification | CIFAR10 [20] | 50,000 | 10,000 | 10
CIFAR100 [20] | 50,000 | 10,000 | 100
Flowers102 [27] | 2,040 | 6,149 | 102
ImageNet-1k [37] | 1,281,167 | 50,000 | 1000
ImageNet-v2 [36] | - | 10,000 | 1000
Video Instance Segmentation | DAVIS-2017∗ [30] | - | 2180 | 30
Table 12: Datasets used in experiments for Active Visual Exploration, Sequential Classification, Classification, and Video Instance Segmentation tasks. (*) DAVIS-2017 dataset contains 30 videos and 2180 frames in total. Dataset | Model | $\kappa$ | $j=2$ | $j=4$ | $j=8$ | $j=12$
---|---|---|---|---|---|---
| acc | GFLOPs | acc | GFLOPs | acc | GFLOPs | acc | GFLOPs
CIFAR100 | MAE | - | 23.2 | 1.2 | 44.2 | 3.8 | 66.9 | 13.0 | 75.8 | 27.8
TORE + MAE | 0 | 32.6 | 1.2 | 52.3 | 3.8 | 70.2 | 13,0 | 77.3 | 27.8
3 | 32.6 | 1.1 | 51.8 | 3.2 | 69.3 | 10.6 | 76.4 | 22.1
5 | 32.2 | 1.1 | 51.0 | 2.9 | 68.0 | 9.0 | 75.3 | 18.4
7 | 31.2 | 1.0 | 47.6 | 2.6 | 62.3 | 7.4 | 69.2 | 14.6
CIFAR10 | MAE | - | 50.7 | 1.2 | 73.4 | 3.8 | 89.7 | 13.0 | 94.7 | 27.8
TORE + MAE | 0 | 58.6 | 1.2 | 76.9 | 3.8 | 90.2 | 13.0 | 94.8 | 27.8
3 | 58.6 | 1.1 | 76.3 | 3.2 | 90.0 | 10.6 | 94.5 | 22.1
5 | 57.8 | 1.1 | 75.8 | 2.9 | 89.3 | 9.0 | 93.9 | 18.4
7 | 57.2 | 1.0 | 73.4 | 2.6 | 86.6 | 7.4 | 91.4 | 14.6
SUN360 | MAE | - | 46.2 | 1.2 | 59.4 | 3.8 | 72.2 | 13 | - | -
TORE + MAE | 0 | 56.5 | 1.2 | 64.8 | 3.8 | 75.1 | 13 | - | -
3 | 55.6 | 1.1 | 65.4 | 3.2 | 74.6 | 10.6 | - | -
5 | 54.8 | 1.1 | 66.7 | 2.9 | 73.8 | 9.0 | - | -
7 | 55.1 | 1.0 | 66.2 | 2.6 | 70.6 | 7.4 | - | -
Table 13: Classification results on partial input for Masked Autoencoder (MAE) and our method (TORE) on CIFAR100, CIFAR10, and SUN360 datasets. We consider a sequence of $12$ observations for CIFAR, and $8$ observations for SUN360, each discovering a new $2\times 2$ patch-sized region. After processing all the observations, about 25% of the image is seen by the model. Parameter $\kappa$ regulates the accuracy-efficiency tradeoff in TORE. Our method significantly improves classification quality, especially for early predictions. Moreover, we can save up to 32% FLOPs while achieving comparable results to the original method. We bold the results for models that save the most GFLOPs without losing accuracy when compared to MAE. Dataset | Model | $\kappa$ | $j=2$ | $j=4$ | $j=8$ | $j=12$
---|---|---|---|---|---|---
acc | GFLOPs | acc | GFLOPs | acc | GFLOPs | acc | GFLOPs
CIFAR100 | AME | - | 28.5 | 1.3 | 53.1 | 4.0 | 71.7 | 13.6 | 77.9 | 28.6
TORE + AME | 0 | 34.9 | 1.3 | 58.9 | 4.0 | 75.5 | 13.6 | 80.2 | 28.6
3 | 34.7 | 1.3 | 58.2 | 3.5 | 74.5 | 11.2 | 79.4 | 23.0
5 | 34.4 | 1.2 | 56.7 | 3.2 | 73.4 | 9.6 | 78.9 | 19.2
7 | 33.4 | 1.1 | 53.7 | 2.8 | 68.9 | 8.0 | 73.5 | 15.4
CIFAR10 | AME | - | 61.4 | 1.3 | 81.3 | 4.0 | 92.3 | 13.6 | 95.5 | 28.6
TORE + AME | 0 | 67.0 | 1.3 | 84.5 | 4.0 | 94.5 | 13.6 | 96.4 | 28.6
3 | 66.8 | 1.3 | 84.3 | 3.5 | 94.4 | 11.2 | 96.4 | 23.0
5 | 66.6 | 1.2 | 83.6 | 3.2 | 93.9 | 9.6 | 95.9 | 19.2
7 | 65.4 | 1.1 | 81.4 | 2.8 | 91.7 | 8.0 | 93.9 | 15.4
SUN360 | AME | - | 48.4 | 1.3 | 62.5 | 3.9 | 76.2 | 13.4 | - | -
TORE + AME | 0 | 52.8 | 1.3 | 70.8 | 3.9 | 76.9 | 13.4 | - | -
3 | 52.7 | 1.2 | 71.3 | 3.4 | 77.5 | 11.0 | - | -
5 | 53.3 | 1.1 | 70.4 | 3.1 | 77.7 | 9.4 | - | -
7 | 53.4 | 1.1 | 68.4 | 2.7 | 77.2 | 7.8 | - | -
Table 14: Active Visual Exploration results with selection policy based on Attention Map Entropy (AME) on CIFAR100, CIFAR10, and SUN360 datasets. We consider a sequence of $12$ observations for CIFAR, and $8$ observations for SUN360, each discovering a new $2\times 2$ patch-sized region. After processing all the observations, about 25% of the image is seen by the model. Parameter $\kappa$ regulates the accuracy-efficiency tradeoff in our method (TORE). We observe that our method significantly improves classification quality, especially for early predictions. Moreover, we can save up to 42% FLOPs while achieving comparable results to the original method. We bold the results for models that save the most GFLOPs without losing accuracy when compared to AME. Dataset | Model | $\kappa$ | $j=2$ | $j=4$ | $j=8$ | $j=16$
---|---|---|---|---|---|---
acc | GFLOPs | acc | GFLOPs | acc | GFLOPs | acc | GFLOPs
CIFAR100 | DeiT | - | 46.2 | 1.0 | 65.6 | 3.1 | 81.6 | 10.5 | 90.5 | 37.9
TORE + DeiT | 0 | 48.8 | 1.0 | 68.3 | 3.1 | 82.7 | 10.5 | 90.3 | 37.9
3 | 48.5 | 0.9 | 67.5 | 2.7 | 82.2 | 8.4 | 89.8 | 29.6
5 | 48 | 0.9 | 66.9 | 2.4 | 81.2 | 7.1 | 88.6 | 24
7 | 46.8 | 0.8 | 64 | 2.0 | 77.6 | 6.7 | 85.8 | 18.5
ImageNet-1k | DeiT | - | 37.7 | 1.0 | 56.8 | 3.1 | 72.0 | 10.5 | 81.4 | 37.9
TORE + DeiT | 0 | 40.0 | 1.0 | 59.2 | 3.1 | 73.3 | 10.5 | 81.3 | 37.9
3 | 40.3 | 0.9 | 58.9 | 2.7 | 73.2 | 8.4 | 80.9 | 29.6
5 | 39.2 | 0.9 | 57.6 | 2.4 | 72.0 | 7.1 | 79.9 | 24
7 | 39.0 | 0.8 | 55.8 | 2.0 | 69.5 | 6.7 | 77.2 | 18.5
Table 15: Sequential classification performance of DeiT and TORE + DeiT on CIFAR100 and ImageNet-1k datasets. We divide images into $p=16$ nearly equally sized rectangular parts and provide them in random order to the models. We record the accuracy and cumulative amount of used FLOPs by models performing predictions on $j$ out of 16 parts. Parameter $\kappa$ regulates the accuracy-efficiency tradeoff in our method (TORE). We observe that our method improves the sequential classification quality for early predictions. Moreover, we can save up to 33% FLOPs while achieving comparable results to the original method. We bold the results for models that save the most GFLOPs without losing accuracy when compared to DeiT. | $p=8$ | $p=16$
---|---|---
$\kappa$ | DINO | TORE + DINO | DINO | TORE + DINO
top-1 | top-5 | top-1 | top-5 | top-1 | top-5 | top-1 | top-5
0 | 78.5 | 92.5 | 78.1 | 92.1 | 78.5 | 92.5 | 78.1 | 92.1
1 | 76.3 | 91.1 | 77.3 | 91.8 | 76.2 | 91.1 | 76.6 | 91.3
2 | 74.6 | 90.1 | 77.2 | 91.7 | 74.5 | 90.0 | 76.3 | 91.2
3 | 73.2 | 89.2 | 77.1 | 91.7 | 73.1 | 89.2 | 76.0 | 91.0
4 | 71.2 | 87.7 | 76.9 | 91.6 | 71.0 | 87.7 | 76.1 | 91.1
5 | 70.8 | 87.3 | 76.9 | 91.6 | 70.7 | 87.3 | 75.9 | 91.0
6 | 69.4 | 86.0 | 76.6 | 91.6 | 69.3 | 86.2 | 75.6 | 90.7
7 | 69.4 | 86.1 | 76.3 | 91.3 | 69.2 | 86.1 | 74.9 | 90.3
8 | 69.0 | 86.0 | 75.9 | 91.1 | 68.9 | 85.9 | 74.4 | 90.0
9 | 68.3 | 85.4 | 75.5 | 90.8 | 68.3 | 85.4 | 73.7 | 89.7
10 | 66.8 | 84.1 | 75.1 | 90.4 | 66.9 | 84.1 | 72.9 | 89.1
11 | 63.5 | 81.3 | 74.3 | 90.0 | 63.9 | 82.0 | 72.0 | 88.4
12 | 57.9 | 76.6 | 72.5 | 88.9 | 58.8 | 77.9 | 68.7 | 86.3
Table 16: $k$-NN evaluation results on ImageNet-1k. We report both top-1 and
top-5 accuracy. Figure 12:  Top-1 $k$-NN accuracy for DINO and TORE + DINO on
ImageNet-1k for two division numbers $p=8$ and $p=16$. We demonstrate that in
both testing scenarios, our method experiences much less decrease in accuracy
with the increasing size of the iterator $\kappa$.

Figure 13: Examples of video instance segmentation from DAVIS-2017 dataset.
DINO trained with TORE preserves the original property of the model of scene
segmentation.


# CHAMBER SURFACE ROUGHNESS AND ELECTRON CLOUD FOR THE ADVANCED PHOTON SOURCE
SUPERCONDUCTING UNDULATOR ††thanks:  Work supported by U.S. Department of
Energy, Office of Science, Office of Basic Energy Sciences, Under Contract No.
DE-AC02-06CH11357. Measurements taken on the Soft X-Ray Beamline at the
Australian Synchrotron

Laura Boon<EMAIL_ADDRESS>Purdue University  West Lafayette  IN  47905  USA
Katherine Harkay  ANL  Argonne  IL 60439  USA

###### Abstract

The electron cloud is a possible heat source in the superconducting undulator
(SCU) designed for the Advanced Photon Source (APS), a 7-GeV electron
synchrotron radiation source at Argonne National Laboratory. In electron cloud
generation extensive research has been done, and is continuing, to understand
the secondary electron component. However, little work has been done to
understand the parameters of photoemission in the accelerator environment. To
better understand the primary electron generation in the APS; a beamline at
the Australian Light Source synchrotron was used to characterize two samples
of the Al APS vacuum chamber. The total photoelectron yield and the
photoemission spectra were measured. Four parameters were varied: surface
roughness, sample temperature, incident photon energy, and incident photon
angle, with their results presented here.

##  1 Introduction

The upgrade of APS calls for the production of a higher energy, higher-
brightness photon beam [1]. To achieve this goal a new superconducting
undulator (SCU) has been designed at APS. This will be an out-of-vacuum
undulator with a period of 1.8 cm. The chamber in the SCU cryostat will be
thermally isolated from the superconducting coils and kept at 20K, the chamber
cooling will be provided by two cryocoolers producing 40 W of cooling power.

Studies performed on the in-vacuum superconducting undulator at ANKA, an
electron synchrotron, confirm that their unexpected heat load is from electron
cloud multipacting [2]. However, work is still being done to find what
mechanism is responsible for electron cloud multipacting, which is not
reproduced in electron cloud simulation codes for electron machines.

##  2 Simulations

The range of photon angle and energy chosen for the measurement was based on
simulations using synrad3d [3], in which a specular scattering model was
assumed. The peak flux of absorbed photons was at 0.6 degrees grazing angle,
as shown in Figure 1. Although the critical energy from an APS bending magnet
is 19 KeV the flux of photons absorbed on the chamber walls of the SCU is
peaked at low energies, as shown in Figure 2. Due to physical limitations of
the beamline the smallest grazing angle measurable was 3 degrees.

Figure 1: Incident grazing angle of absorbed photons in the SCU cryostat
chamber. Results produced from simulations using Synrad3d.  Figure 2: Energy
of absorbed photons in the SCU cryostat chamber. Results produced from
simulations using Synrad3d.

##  3 Measurement Description

The Australian Synchrotron’s Soft X-ray beam line was used to measure the
quantum efficiency of actual beam chamber samples. Data were acquired at
grazing angles of 3, 5, 10, and 50 degrees in photon energy scans from 100 eV
to 2000 eV in 0.5 eV steps at varying temperatures from 300K to 180K.

Two samples were measured. Both are sections of the extruded Al APS beam
chamber, a rough or unpolished sample of the SCU Al chamber and a smooth or
polished sample. The smooth sample, the top sample in Figure 3, was polished
using an abrasive flow process [4]. The rough sample, the bottom sample in
Figure 3, was measured ”as received.” The surface roughness of the samples was
measured and is presented in Table 1. The samples were cleaned in an
ultrasonic, acetone bath for ten minutes before the measurements were taken.

Table 1: Sample parameters Sample | RMS
---|---
Rough | 1180 nm
Smooth | 139 nm
Figure 3: Picture of the sample holder with the Al samples. The smooth, or
polished sample is on the top, while the rough, or unpolished sample is on the
bottom.

##  4 Analysis

At the beamline two different sample drain currents were measured. To
calibrate the total photon flux on the sample, the drain current from the Si
diode at the back of the sample chamber was measured for all photon energies.
Using the data for the Si diode given by the manufacturer [5], the photon flux
at each energy can be calculated. Equation 1 was used to calculate the total
photon flux incident on the Si diode.

$Flux=\frac{I_{Si}}{q_{e}}\frac{3.65eV}{electron}\frac{1}{E_{photon}}F(E_{\gamma})\left[\frac{photons}{sec}\right]$
(1)

$I_{Si}$ is the drain current measured from the Si diode, 3.65 eV is the
average energy for an electron-hole pair creation in silicon, $E_{photon}$ is
the energy of the incident photon beam and $F(E_{\gamma})$ is the transmission
coefficient [6]. With the sample in place all photons that would have been
incident on the diode are now incident on the sample, therefore the flux on
the sample is also given by equation 1. The drain current from the sample was
also measured. Using equation 2, the number of electrons produced can be
calculated.

$\\#Electrons=\frac{I_{Al}}{q_{e}}\left[\frac{electrons}{sec}\right]$ (2)

$I_{Al}$ is the drain current measured from the aluminum sample, and $q_{e}$
is the charge of an electron. The quantum efficiency is the ratio of number of
electrons emitted to number of incident photons, as seen in equation 3.

$QE=\frac{\\#Electrons}{Flux}$ (3)

Equation 3 was used to calculate the quantum efficiency for Figure 4. All
results were normalized to the storage ring beam current.

##  5 Results

The quantum efficiency as a function of energy was found to be strongly
dependent on the energy of the incident photon beam. As an example, the
quantum efficiency of the Al chamber with a photon beam at a grazing angle of
5 degrees is shown in Figure 4 for both samples. There is an increase in the
quantum efficiency for photon beam energies equal to the K and L edges of the
materials. Although the substrate does not contain any oxygen and carbon both
are measured on the surface. The oxygen and carbon are part of the aluminum
oxide layer, typically 4-5 nm [7], that forms on aluminum from exposure to the
air. The quantum efficiencies measured with low grazing angles 3-5 degrees
indicated a stronger dependence on the surface chemistry, while the quantum
efficiencies at larger angles, around 50 degrees, were more dependent on the
substrate.

Figure 4: An example of a QE plot as a function of energy. These data were
taken with the incident photons at a grazing angle of 5 degrees. The sample
was at 190 K.

The average quantum efficiency was calculated for each angle and sample and
then fit to a Lorentzian, see Figure 5. Similarly the peak QE was found at
each angle then fit to a Lorentzian, see Figure 6.

Figure 5: The average QE plotted as a function of photon grazing angle. Higher
photon grazing angles have a smaller QE than low grazing angles. Figure 6: The
peak QE plotted as a function of photon grazing angle. Higher photon grazing
angles have a smaller QE than low grazing angles.

From Figures 5 and 6 it is seen that the smoother sample had a higher quantum
efficiency than the rough sample. The difference is greater for smaller
grazing angles and reduces as the photon grazing angle approaches 50 degrees.
The angle dependance of the quantum efficiency is related to the penetration
depth of photons. For all grazing angles the photons travel the same distance
through the material; however, the photons are absorbed closer to the surface
when the grazing angle is low. The escape probability of the photoelectrons is
thereby increased, which increases the quantum efficiency.

To use this data to update current electron cloud generation codes, the
quantum efficiency for angles less than the measured three degrees would need
to be interpolated from the data. Figures 5 and 6 show a good fit with a
Lorentzian for photon grazing angles between 3 and 50 degrees, and the quantum
efficiency for less than three degrees would be interpolated from the fit. The
energy dependance of the quantum efficiency at photon grazing angles smaller
than three degrees can also be determined. Knowing that the surface chemistry
will dominate, a higher quantum efficiency would be assumed for photons with
energies equal to the K and L edges of carbon and oxygen.

Figure 7: The difference in quantum efficiency for a cold (180K) and warm (300
K) sample.

For the results shown in Figures 5 and 6 the temperatures of the samples was
assumed to be constant. This assumption is based on a comparison of the
quantum efficiency at the same angle and different temperatures (300K and
180K). The quantum efficiency over all energies did not vary more than 15%.
For the superconducting undulator the chamber will be held at 20K, which will
require more studies to determine if there will be a difference in quantum
efficiency at the lower temperatures.

##  6 Conclusions

The quantum efficiency was measured for the aluminum chamber that will be
installed at APS for the SCU. The quantum efficiency was seen to be strongly
dependent on the surface roughness and grazing angle of the photon beam. This
study shows the actual quantum efficiency is more complex then a single value
that is currently being used in electron cloud codes. More work must still be
done to understand the impact of these results on the SCU.

##  7 Acknowledgements

Thanks to B. Cowie and K. Wootton for help in making the measurements. To L.
Assoufid, E. Trakhtenberg and G. Wiemerslage for preparing the samples. To R.
Cimino, S. Casalbuoni, L. Emery and B. Yang for useful discussions and
analysis help and M. Harrison for ILC financial support.

## References

  * [1] Advanced Photon Source Upgrade Project, “Conceptual Design Report”, May 2011.
  * [2] Casalbuoni, S. et al., “Beam heat load and Pressure in the Superconducting Undulator installed at ANKA”, IPAC12, 717.
  * [3] Dugan, Gerry and Sagan, David, “Synrad3d Photon Propagation and Scattering Simulation”, ECLOUD10.
  * [4] Trakhtenberg, Emil, et al., “Extruded Aluminum Vacuum Chambers for Insertion Devices”, PAC11, 2093.
  * [5] International Radiation Detectors, Inc. http://www.ird-inc.com/axuvhighnrg.html
  * [6] Gullikson, E., ’X-Ray Interactions With Matter’ , http://henke.lbl.gov
  * [7] Rosenberg, R. Private communication.