High-resolution images offer more information about scenes that can improve model accuracy. However, the dominant model architecture in computer vision, the vision transformer (ViT), cannot effectively leverage larger images without finetuning — ViTs poorly extrapolate to more patches at test time, although transformers offer sequence length flexibility. We attribute this shortcoming to the current patch position encoding methods, which create a distribution shift when extrapolating.

We propose a drop-in replacement for the position encoding of plain ViTs that restricts attention heads to fixed fields of view, pointed in different directions, using $2$D attention masks. Our novel method, called LookHere, provides translation-equivariance, ensures attention head diversity, and limits the distribution shift that attention heads face when extrapolating. We demonstrate that LookHere improves performance on classification (avg.$\uparrow1.6\%$), against adversarial attack (avg.$\uparrow5.4\%$), and decreases calibration error (avg.$\downarrow1.5\%$) — on ImageNet without extrapolation. With extrapolation, LookHere outperforms the current SoTA position encoding method, $2$D-RoPE, by $21.7\%$ on ImageNet when trained at $224^2$ px and tested at $1024^2$ px. Additionally, we release a high-resolution test set to improve the evaluation of high-resolution image classifiers, called ImageNet-HR.

§ INTRODUCTION

ViT-B/$16$ models trained for $150$ epochs on ImageNet at $224^2$ px and tested up to $1024^2$ px. Model architectures are consistent between runs other than position encoding methods. We perform an $8$-run hyperparameter sweep, per method, to ensure fair comparisons. Our three LookHere variants improve extrapolation ability, with more narrow fields of view performing best at $1024^2$.

There is a decades-long trend in computer vision towards higher-resolution imagery, which contains more detailed scene information. Increasing resolution is a reliable way to improve model accuracy <cit.>, but this comes at a cost; training models for hundreds of epochs on large-scale datasets is expensive, especially at high-resolutions. There are two ways to reduce this cost and still see accuracy benefits from high-resolutions: 1 high-resolution finetuning, which pretrains models at a lower resolution, like $224^2$ px, then finetunes them at a higher resolution, like $384^2$ px; and 2 extrapolating, which deploys models at a higher resolution, without further training. Of these two options, we should aim for models that can effectively extrapolate, as it presents a zero-cost solution that does not require finetuning at every target resolution. Finetuning costs aside, improvements to extrapolation should benefit high-resolution finetuning since models that are better at extrapolating can adapt to higher resolutions more easily. Although extrapolation is a significant and exciting challenge, state-of-the-art (SoTA) model architectures extrapolate poorly.

Vision transformers (ViTs <cit.>) offer SoTA performance on many computer vision tasks. ViTs are simple; they split images into non-overlapping patches, linearly project pixels to form patch embeddings, and process these “tokens” with a stack of architecturally identical transformer layers — maintaining a constant feature map size throughout. This non-hierarchical design enables learning patch representations, which are useful for dense prediction tasks <cit.> and are fundamental for vision-language models <cit.>. The design enables efficient processing of only a subset of patches, known as token dropping <cit.>. Lastly, it enables model scaling by increasing the embedding size and the layer count <cit.>.

Image-size extrapolation with ViTs can be achieved in three ways: 1 increasing the patch size, which packs more pixels into each patch embedding; 2 increasing the “patchification” stride, which skips-over pixels; and 3 increasing the number of patches. Of these three options, we should aim for models that can effectively ingest more patches — called “sequence length extrapolation” in the natural language processing (NLP) community <cit.> — as a greater number of patches presents models with more (uncompressed) information that we hope to leverage into higher accuracy. Furthermore, methods that improve sequence length extrapolation, like our proposed method, can be fused with methods that adjust patch sizes, like FlexiViT <cit.>. We strongly believe that patch position encoding is a primary cause of the poor sequence length extrapolation ability of ViTs — like it is in NLP, where significant advancements have been made by improving position encoding <cit.>.

Adding learnable or fixed sinusoidal position embeddings to patch embeddings before the first layer is the most common way ViTs encode positions. Recently, the rotary position embeddings (RoPE <cit.>) used in SoTA language models <cit.> were extended to ViTs, as $2$D-RoPE <cit.>, showing exciting results. RoPE is a different approach to position encoding that injects positional information in each self-attention layer by rotating queries and keys with fixed sinusoidal embeddings. But for these methods to ingest more patches at test time, they must either introduce new position embeddings or modify existing embeddings — both options create a significant distribution shift. Motivated by these observations and more, we make the following contributions:

1 LookHere — We introduce a novel position encoding method for plain ViTs that restricts attention heads to fixed fields of view (FOV) and points them in different directions via $2$D masks. This design provides: a translation-equivariance, b attention head diversity, c improved interpretability, and d limits the distribution shift that attention heads face when extrapolating.

2 Controlled Experiments — We perform an apples-to-apples comparison between seven position encoding methods for plain ViTs alongside our three LookHere variants. We demonstrate that LookHere: a improves classification, segmentation, adversarial robustness, and model calibration when tested at the training resolution; b significantly improves performance when tested beyond the training resolution;  and c increases its performance advantage after high-resolution finetuning.

3 Extrapolation Insights — We show that extrapolation: a benefits images with small objects the most, as they occupy more patches at test time; b produces class-level and dataset-level effects; and c creates distribution shifts that can be visualized via attention maps.

4 ImageNet-HR — We introduce the first natively high-resolution ImageNet test set ($1024^2$ px) aimed to benchmark classifiers on images that were not upsampled to achieve the target image size.

§ BACKGROUND AND RELATED WORK

A ViT splits an image into a grid of non-overlapping patches, flattens the grid into a sequence, and flattens the patches into vectors; i.e., $\mathbb{R}^{Y \times X \times C} \rightarrow \mathbb{R}^{N_{y} \times N_{x} \times P^2 \times C} \rightarrow \mathbb{R}^{(N_{y} \cdot N_{x}) \times (P^2 \cdot C)}$, where $Y$ is the image-height, $X$ is the image-width, $C$ is the number of channels, $N_{y}$ is the grid-height, $N_{x}$ is the grid-width, $P$ is the patch height and width. A linear layer maps each vector of pixels to a patch embedding; i.e., $\mathbb{R}^{P^2 \cdot C} \rightarrow E_{i}^{patch} \in \mathbb{R}^{D}$, where $D$ is the embedding dimension also known as the transformer width. We define $i$ and $(i_y, i_x)$ as the sequence position and the $2$D position of the $i^{\text{th}}$ patch, respectively, where $N$ is the total number of patches, equal to $N_y\cdot N_x$, $i \in \{1,2,\ldots,N\}$,  $i_y \in \{1,2,\ldots,N_y\}$, and $i_x \in \{1,2,\ldots,N_x\}$. Finally, sequence length extrapolation occurs when $N_{test} > N_{train}$.

A patch embedding represents the content of a patch, and contains no information representing its original location within the image. Thus, we must encode patch positions to enable spatial reasoning; otherwise, a ViT will operate on a bag of patches.

We define a “plain ViT” as attention-only and non-hierarchical. Our primary goal is to improve the extrapolation ability — i.e., generalize to more patches at test time — of plain ViTs. Our work is motivationally aligned with FlexiViT <cit.> and NaViT <cit.>, improving the flexibility of plain ViTs. Next, we briefly describe seven position encoding methods and refer the reader to the cited studies for further details; we include them all in our controlled experiments. Another method, iRPE <cit.>, is also compatible with plain ViTs. However, we exclude it because it is more than twice as slow as other methods; nonetheless, we benchmark iRPE with our best training recipe in Appendix <ref>.

Input Embeddings. This group leverages learned or fixed position embeddings, $E_{i}^{pos} \in \mathbb{R}^{D}$, that are added to patch embeddings at the transformer input; i.e., $z_{i} = E_{i}^{patch} + E_{i}^{pos}$, where $z$ is the input to the first transformer layer. Position embeddings represent the absolute positions of patches in an image.

1 1D position embeddings <cit.> (1D-learn for short) map $i$ to learnable embeddings. 2 2D sinusoidal embeddings <cit.> (2D-sincos for short) individually map $i_y$ and $i_x$ to fixed 1D-sinusoidal embeddings ($E_{i}^{y}, E_{i}^{x} \in \mathbb{R}^{\frac{D}{2}}$), then concatenate them along the embedding dimension. 3 Factorized position embeddings <cit.> (Factorized for short) individually map $i_y$ and $i_x$ to learnable embeddings ($E_{i}^{y}, E_{i}^{x} \in \mathbb{R}^{D}$), then add them. 4 Learnable Fourier features <cit.> (Fourier for short) map $(i_y, i_x)$ to Fourier features <cit.>, then to embeddings with a multi-layer perceptron (MLP).

Attention Biases. This group leverages learned or fixed operations that encode positions by modifying the pairwise interactions between patches in self-attention without adding position embeddings to patch embeddings. Recall that self-attention first applies three separate linear transformations to project internal patch representations and splits the resultant vectors into $H$ smaller vectors of length $D_H$; i.e., $\mathbb{R}^{N \times D} \rightarrow \mathbb{R}^{3 \times N \times H \times D_H}$ — creating queries, keys, and values for each attention head. We denote a specific head by $h$. Next, attention scores ($A \in \mathbb{R}^{H \times N \times N}$) are calculated by measuring the similarity between all pairs of queries ($q_{hi} \in \mathbb{R}^{D_H}$) and keys ($k_{hj} \in \mathbb{R}^{D_H}$), separately, for each head; i.e., $a_{hij} = q_{hi} \cdot k_{hj}/\sqrt{D_H}$, where $i$ and $j$ are query and key sequence positions, and we define $(i_y, i_x)$ and $(j_y, j_x)$ as their $2$D positions. Attention scores ($a_{hij}$) represent the amount of information moving from patch position $j$ to $i$ — whereas values ($v_{hj} \in \mathbb{R}^{D_H}$) represent the content of the moving information.

5 Learnable relative position encoding <cit.> (RPE-learn for short) biases attention scores by mapping all possible relative positions between queries and keys to learnable embeddings ($B_{ij} \in \mathbb{R}^{H}$); i.e., biases are a function of $i_y - j_y$, $i_x - j_x$, and $h$. 6 A $2$D extension of Attention with Linear Biases (ALiBi <cit.>), $2$D-ALiBi <cit.> penalizes attention scores as a function of the Euclidean distance between $(i_y, i_x)$ and $(j_y, j_x)$, and a head-specific scalar, called a slope. Slopes bias attention heads at different rates. 7 A 2D extension of rotary position embeddings (RoPE <cit.>), $2$D-RoPE <cit.> rotates queries and keys as a function of their positions. Each query is rotated by the sinusoidal embedding of $i_y$ for half its dimensions and the sinusoidal embedding of $i_x$ for the other half of its dimensions; likewise, keys are rotated as a function of $j_y$ and $j_x$.

Non-plain ViTs. Many hybrid or hierarchical architectures have been invented that often encode positions differently <cit.>. Although these architectures may be favored in some circumstances, the plain ViT is the most common single architecture due to its simplicity, flexibility, and scalability. We benchmark many non-plain ViTs and large SoTA ViTs on extrapolation in Appendix <ref>.

ViT Extrapolation. Some ViTs have been tested at higher resolutions than they were trained <cit.>. NaViT <cit.> benchmarked input embedding methods on extrapolation, none see the gains at higher resolutions that we observe.

§ LOOKHERE

nodeG/.style=anchor=north west,text width=3+/3,minimum height=3+/3,outer sep=0pt,inner sep=0pt,align=center,draw=white,draw opacity=1,text=white,text opacity=1,font=,
[anchor=north west,inner sep=0pt,outer sep=0pt] at (0.5,0.5) ;
[nodeG,fill=black,fill opacity=1.0] (M0C0R0) at (0.5,0.5) $(0,0)$;
[nodeG,fill=black,fill opacity=1.0] (M0C1R0) at (M0C0R0.north east) $(0,1)$;
[nodeG,fill=white,fill opacity=0.6] (M0C2R0) at (M0C1R0.north east) $(0,2)$;
[nodeG,fill=black,fill opacity=1.0] (M0C0R1) at (M0C0R0.south west) $(1,0)$;
[nodeG,fill=white,fill opacity=0.0] (M0C1R1) at (M0C0R1.north east) $(1,1)$;
[nodeG,fill=white,fill opacity=0.3] (M0C2R1) at (M0C1R1.north east) $(1,2)$;
[nodeG,fill=black,fill opacity=1.0] (M0C0R2) at (M0C0R1.south west) $(2,0)$;
[nodeG,fill=black,fill opacity=1.0] (M0C1R2) at (M0C0R2.north east) $(2,1)$;
[nodeG,fill=white,fill opacity=0.6] (M0C2R2) at (M0C1R2.north east) $(2,2)$;
(0.5,0.5) rectangle ++(10,10);

-1.15249586989453    -0.5837504009953141 2.0409752450280063   -1.2235953033544205   -1.4864549612156366  -2.178380559469621   1.5029569461679806   2.6870073187588703  1.938439263268278     -1.8147667400144223  0.34145021367941375  -0.9600381827395987 0.3251351723327969   -0.5452281879151304   -0.17648030900805223 -0.8935236404042899  -1.1781395161128363  2.238122832163418   -0.019652379002174314 -0.619945532912225   1.4412549071216754   -0.9487360825335205 0.12673661294442828  -0.47460497206528496  0.22297437022594568  -2.479853478272537   1.4237908946541866   2.3425081302825985  -1.1634498086109795   2.089531049779672    -1.8732317453910983  -0.2597521989003111 0.050168902864212495 2.1491935112383898    -2.179717923820254   -2.612505807768406   0.8594087860602162   1.2424806538410573  0.7005981574623861    0.46700669332698874  0.34738569641198325  1.147175180077682   -0.8552007486853503  0.436864476283334     1.4134961296589221   -0.27566902442691443 0.6814253470892822   -2.6561735300700002 0.6519184024986333    -0.8216138433327738  -0.9499956236584255  1.2963315525812673  0.46132760910456394  -1.5024982758393863   0.686768435138343    0.8615385258112762   0.530569481732194    2.703400619374092   -2.1767020805619395   -0.820664398263381   -2.007439203618425   -0.1560698622690948 0.1691050942771608   0.0010184917199445032 1.9056434206056716   1.0412855280042725   1.109650254049488    1.0122001873524424  -1.9348652862983666   -0.07425880128490103 -0.19911937090147447 -0.9140261643031633 1.261393433436277    0.020930242221161688  1.3929294987085274   -0.9090278671879406  -0.23493285619235538 0.8811868092166257  2.0580383491840335    -1.793755890880261   0.20840471714030118  -2.182907910113681  -1.9147195888686563  0.9997270137295811    2.230812927418527    2.0438967381627613   -1.9779440531235677  -2.6103775070376054 0.9970297870654146    -1.59414844476236    2.4684363885217095   2.62102306967416    -2.0219992308378734  2.07532360537789      -2.146906572187199   -0.42571284378002794 -2.177193405846834   0.7508579609263868  0.7852592428358705    -1.311456117638326

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 2.0 x   1.4 2.2 x   x   2.8 0.0 x   0.0 1.0 x   x   1.4 x   x   x   0.0 x   x   0.0 x   x   x   x   x   x   0.0 x   1.4 2.2 0.0 1.0 2.0 x   1.4 2.2 0.0 x   x   1.4 x   0.0 1.0 x   x   1.4 0.0 x   x   x   x   x   0.0 x   x   x   0.0 x   x   2.8 x   1.4 2.2 0.0 1.0 2.0 0.0 x   x   x   x   x   1.4 x   0.0 1.0 0.0 x   x   x   x   x   x   x   x   0.0

[scale=2,white] at ($(M2C2R10)!0.5!(M2C3R9)$) $\infty$;
[scale=2,white]   at ($(M2C8R3)!0.5!(M2C9R4)$) $\infty$;

[shift=(0,-4),Nmath/.style=execute at begin node=$\displaystyle},execute at end node={$},anchor=west,inner xsep=1pt,inner ysep=2pt},]
\node[Nmath] (Eq1) at (2.5,2) {\mathcal{A_{\text{LRN}}} \leftarrow};
\node[Nmath] (Eq2) at (Eq1.east) {\frac{QK^T}{\sqrt{d_k}}};
\node[Nmath] (Eq1) at (13.25,2) {\mathcal{A_{\text{FIX}}} \leftarrow};
\node[Nmath] (Eq2) at (Eq1.east) {\textbf{LookHere}};
\node at (21,2) {\eyes};

\end{scope}
\begin{scope}[shift={(+2*\Mpos,0)}]
\colorlet{COLMAX}{COL1}
\colorlet{COLMED}{COL2}
\colorlet{COLMIN}{COL3}
\node[scale=0.7,right] (R0) at (0.2,0) {\strut negative};
\shade[left color=COLMIN,right color=COLMAX, middle color=COLMED] ([yshift=4pt]R0.south east) rectangle ++(5.5,-0.5) coordinate (R1);
\node[scale=0.7,right] (R2) at (R1|-R0) {\strut positive};
% \draw[fill=black] ([yshift=4pt]R2.south east) rectangle ++(0.5,-0.5) coordinate (R3);
% \node[scale=0.7,right] at (R3|-R0) {\strut infinity};
\end{scope}

\draw[very thick,secon] (M0C1R1.north west)--(M2C10R6.north east)-|(M1C1R6.north west)--(M1C1R6.south west)--(M2C10R6.south east)--(M0C1R1.south west);
\draw[very thick,secon] (M0C1R1.south west) rectangle (M0C1R1.north east);
\node[white,below] at (M0C1R1.north) {query};
\end{tikzpicture}}
\caption{LookHere masks and biases (center) the learned attention matrix (left, where colors are random). Masked cells are \textbf{black}, encoding directions ($\rightarrow \text{with a } 90\degree$ FOV); biased cells are shaded \textcolor{bluegreen}{\textbf{bluish-green}}, encoding relative patch distances. (Right) An example of the FOV of the center query patch. The final attention matrix is computed as $\mathcal{A}^l = \texttt{softmax}(\mathcal{A}_{\text{LRN}}^l - \mathcal{A}_{\text{FIX}}^l)$, at each layer $l$.}
\label{fig:lookhere_method}
\end{figure}

\textbf{Design Motivation.} We introduce $2$D attention masks that assign each attention head a direction and a FOV, preventing attention outside the head's FOV. Within a head's FOV, attention scores are penalized based on relative patch distances. Three ideas motivate this design. \circlednum{1} Attention head diversity: heads often learn redundant algorithms that can be pruned with little accuracy penalty \cite{vit_sparse, vit_prune_1, vit_prune_2}. Head redundancy has also been observed in NLP \cite{nlp_prune_heads_1, nlp_prune_heads_2, nlp_prune_heads_3}, where diversity-encouraging loss functions have been leveraged to improve generalization \cite{diversity_bengio, diversity1, diversity2, diversity3}. From a mechanistic point of view, we can think of attention heads as an ensemble of sub-networks that ``operate completely in parallel, and each add their output back into the residual stream,'' \cite{circuits} and the residual stream is mapped to logits. Diversity has long been a desirable property of ensembles \cite{diversity_old_1, diversity_old_2}, and constraining attention heads to focus in different directions ensures it. \circlednum{2} Attention head consistency: heads often learn interpretable spatial algorithms, like ``attend to the area above the query,'' which reliably retrieves information from the internal representations above the query; however, we believe these types of spatial algorithms might fail when new or modified position embeddings are introduced to encode \textit{new} patch positions during extrapolation --- misleading the model about the information above the query, for example. We believe hard-coding both directions and distances (via attention masks and biases) will reduce the need for models to learn their own spatial algorithms. \circlednum{3} Translation-equivariance has long been a desirable property of vision models, contributing to the success of convolutional networks \cite{cnn_equi_1, cnn_equi_2, cnn_equi_3}. ViTs are critiqued for weak inductive biases, leading to poor sample efficiency when trained from scratch \cite{bridging, sharpness, vitae}. We believe that LookHere's stronger inductive biases, achieved via directional masking and distance penalties, can improve ViT sample efficiency.

\textbf{Design Specifics.} Let $H$ be the number of heads, $L$ be the number of layers, and $N$ be the number of patches (plus one for the CLS token). We denote the LookHere matrices by $\mathcal{A}_{\text{FIX}} \in \mathbb{R}^{L \times H \times (N+1) \times (N+1)}$. We encode positions by subtracting the LookHere matrix for a layer $l$, $\mathcal{A}_{\text{FIX}}^l$, from the learned attention matrix, $\mathcal{A}_{\text{LRN}}^l = QK^T/\sqrt{D_H}$, before the softmax that normalizes the attention matrix prior to multiplying it by values \cite{attention}, i.e., $\mathcal{A}^l = \texttt{softmax}(\mathcal{A}_{\text{LRN}}^l - \mathcal{A}_{\text{FIX}}^l)$. We do not add position embeddings to patch embeddings.

Let $i$ and $j$ be query and key sequence positions, respectively, with $2$D-coordinates $(i_y, i_x)$ and $(j_y, j_x)$. Crucially, $j$ is visible to $i$ if $j$ lies within $i$'s FOV. This attention masking technique is inspired by the $1$D causal masks used in autoregressive transformer decoders used in NLP \cite{attention}. When $j$ is visible, we bias the attention score based on the Euclidean distance between $i$ and $j$ to encode the relative distance between patches. We scale distances via a slope function $m: \mathbb{N}_L \times \mathbb{N}_H \to R$, $m(l, h)=s_l(l) \cdot s_h(h) \cdot s_g$ that strengthens or weakens the distance penalty as a function of the head ($s_h: \mathbb{N}_H \to \mathbb{R}$) and layer ($s_l: \mathbb{N}_L \to \mathbb{R}$), scaled by a global slope $s_g \in \mathbb{R}$. Finally, the CLS token is visible to all positions.

\begin{align}
\label{equations}
\text{LookHere}(l, h, i, j) &=
\begin{cases}
m(l, h) \cdot \text{Distance}(i, j) & \text{if $j$ is visible to $i$}\\
\infty & \text{otherwise}
\end{cases}
\\
\text{Distance}(i, j) &= \sqrt{(i_y - j_y)^2 + (i_x - j_x)^2}
\end{align}

For example, Figure \ref{fig:lookhere_method} displays attention matrices of a head that ``looks right'' with a $90\degree$ FOV. We create three LookHere variants, the first two have FOVs of $180\degree$ and $90\degree$  (\textcolor{LHcolor_dark}{\textbf{LH-}\boldmath{$180$}} and \textcolor{LHcolor_dark}{\textbf{LH-}\boldmath{$90$}}). We direct attention heads eight different ways, selecting the four cardinal directions ($\uparrow, \downarrow, \leftarrow, \rightarrow$) and the four intercardinal directions ($\nearrow, \searrow, \swarrow, \nwarrow$). ViT-B models have twelve attention heads; we leave the last four attention heads undirected to allow them unrestricted attention over the full image. We create a final variant that cuts the first four LH-$90$ masks in two, creating eight $45 \degree$ views that cover the full image without overlapping (\textcolor{LHcolor_dark}{\textbf{LH-}\boldmath{$45$}}). Visualizations of the bias matrices are in Appendix \ref{appn:lookhere-bias-maps}.

\textbf{Design Ablations.} We offer four takeaways through extensive ablations (Appendix \ref{appn:ablations}): \circlednum{1} LookHere is robust to the choice of slope function. We set our default $s_l$ to linearly decrease from $1.5$ to $0.5$ with increasing depth (inspired by depth-wise attention distance findings \cite{vit_attn_distance}). This helped in preliminary experiments, but the benefits disappear in our ablations. We arbitrarily set our default $s_h$ to $(\frac{1}{2},\frac{1}{8},\frac{1}{32},\frac{1}{128})$ for the four undirected heads, but distance penalties on undirected heads can be removed entirely. We set $s_g=1$; LookHere is also robust to the choice of the global slope. We believe precisely tuning slopes is unnecessary because models can learn to scale attention logit magnitudes. \circlednum{2} Increasing penalties with the square or square root of the distance harms extrapolation. \circlednum{3} Removing all distance penalties harms extrapolation. \circlednum{4} Our main contribution, $2$D directional masks, are crucial to retain performance, but our method is robust to \textit{many} directional configurations.

\textbf{Compute.} $\mathcal{A}_{\text{FIX}}$ is precomputed and fixed, subtracting it element-wise from the learned attention matrices $\mathcal{A}_{\text{LRN}}$ only costs $H \cdot (N+1) \cdot (N+1)$ floating point operations (FLOPs) per layer. For a ViT-B/$16$ model, these subtractions account for $0.016\%$ of the total FLOPs. LookHere reduces FLOPs by \textit{not} adding position embeddings to patch embeddings, but this amount is also negligible. Additionally, LookHere matrices offer structured sparsity (up to $7/8$ for a $45\degree$ FOV) that can speedup attention --- although exciting, this speedup requires custom kernels that we leave for future work.
\vspace{-10pt}
\section{Experiments}\label{main_exp}
\vspace{-10pt}
Deep neural networks --- including ViTs --- can be sensitive to seemingly minor hyperparameter changes when trained from scratch. Dosovitskiy et al. \cite{vit} finetuned the original ViT at a higher resolution, reaching $77.9\%$ top-$1$ accuracy on ImageNet (we refer to ILSVRC$2012$ or ImageNet-$1$k as ImageNet). Steiner et al. \cite{how} searched $28$ hyperparameter configurations, achieving best and average runs of $80.0\%$ and $76.9\%$, respectively (average calculation omits runs without data augmentation, as they were poor). Touvron et al. \cite{deit_1} ablated repeat augmentation \cite{repeat}, dropping accuracy by $4.8\%$. Touvron et al. \cite{revenge} replaced cross-entropy loss with binary cross-entropy loss, raising accuracy by $1.3\%$. Importantly, these are all ViT-B/$16$ models trained from scratch for $300$ epochs on ImageNet. Informed by these observations and more, we design a controlled experiment: We search $8$ hyperparameter configurations for \textit{each} position encoding method using a single codebase; this offers an apples-to-apples comparison between our three LookHere variants and seven baselines.

\subsection{Setup}
\vspace{5pt}
Our $80$ training runs result from the following Cartesian product:

\begin{tcolorbox}[colback=white, boxrule=0pt, colframe=white, sharp corners, boxsep=3pt, left=4pt, right=4pt, top=2pt, bottom=2pt]
\begin{minipage}{0.40\textwidth}
\small
\textbf{Position encoding:} $1$D-learn, $2$D-sincos, Factorized, Fourier, RPE-learn, $2$D-ALiBi, \newline $2$D-RoPE, LH-$180$, LH-$90$, LH-$45$
\end{minipage}%
\hfill
\begin{minipage}{0.58\textwidth}
\small
\textbf{Augmentations:} RandAugment$(2,15)$ \cite{randaugment}, $3$-Augment \cite{revenge}

\vspace{2pt}\textbf{Learning rate:} $1.5 \cdot 10^{-3}$, $3.0 \cdot 10^{-3}$

\vspace{2pt}\textbf{Weight decay:} $0.02$, $0.05$
\end{minipage}
\end{tcolorbox}

For each configuration, we train a ViT-B/$16$ on $99\%$ of the ImageNet training set, holding the last $1\%$ as a validation set called ``minival'', following \cite{how, minival} (see Appendix \ref{appn:training_vits} for other hyperparameters). We train all models from scratch for $150$ epochs. Our results are competitive and sometimes surpass ViTs trained for much longer, which validates our setup. The best models (according to minival accuracy), among our $8$-run hyperparameter sweep per method, are always trained using $3$-Augment \cite{revenge}, a $3.0 \cdot 10^{-3}$ learning rate, and a $0.05$ weight decay.

\textbf{Test sets.} We test all $80$ models on six ImageNet test sets. This includes \circlednum{1} the original ``validation'' set used as a test set (Val for short \cite{imagenet}), \circlednum{2} the reassessed labels of the original validation set (ReaL for short \cite{real}), \circlednum{3} the independently collected and in-distribution test set (v$2$ for short \cite{v2}), \circlednum{4} the natural adversarial test set (-A for short \cite{a}), \circlednum{5} the ImageNet rendition test set (-R for short \cite{r}), and \circlednum{6} the high-resolution test set that we introduce (-HR for short).

\textbf{ImageNet-HR.} Since there are no natively high-resolution ImageNet test sets, there are two options to test the extrapolation ability of models trained on ImageNet: \circlednum{1} upsample existing test sets to higher resolutions, and \circlednum{2} collect a high-resolution test set ourselves. However, upsampling low-resolution images introduces another distribution shift (i.e., interpolated pixels) that we may not want to test. Thus, we collect a high-resolution test set to remove this confounding variable from our analysis --- although we \textit{also} test models on upsampled images. We manually collect $5$ images for each ImageNet class, resulting in $5$k total images, and manually crop them to $1024^2$ px. This is smaller than other test sets (v$2$ is $30$k images, -A is $7.5$k images). However, we invest considerable resources to ensure its quality with two priorities: annotation accuracy and image diversity. See Appendix \ref{appn:imgnet-hr} for details and samples.

\textbf{Adversarial Attacks.} We perform Fast Gradient Sign Method (FGSM \cite{fgsm}) adversarial attacks with two strengths ($\frac{1}{255}$, $\frac{3}{255}$) on all models using Val images.

\textbf{Calibration Estimates.} We calculate the Expected Calibration Error (ECE \cite{ece}) with $15$ bins of all models using Val images.

\textbf{Higher-Resolution Finetuning.} With the best model per method, we continue training on ImageNet for $5$ epochs at $384^2$ px. We test at $384^2$ px without extrapolating.

\textbf{Segmentation.} With the best model per method, we finetune following the Segmenter protocol with a linear decoder \cite{segmenter}. Additionally, we probe the patches by only training a linear layer to produce a low-resolution logit map which is upsampled to obtain a full resolution segmentation map, following \cite{oquab2023dinov2}. We run these experiments on ADE$20$k \cite{zhou2019semantic} at $512^2$ px and Cityscapes \cite{Cordts2016Cityscapes} at $768^2$ px.

\textbf{Patch Logit-lens.} Inspired by interpretability research \cite{Nostalgebraist}, we evaluate the quality of the learned patch representations for models leveraging LookHere compared with other methods. Following prior work \cite{vilas2023analyzing_vit, joseph2023vit}, we project frozen patch representations onto the learned class embedding space using the MLP classifier head that was learned for the CLS token. We leverage the ImageNet-S dataset \cite{gao2022luss}, which contains partial segmentation maps for $12$k images from Val, covering $919$ ImageNet classes.

\textbf{Extrapolating.} With the best model per method, we test on larger images, increasing the number of patches. For $1$D-learn and $2$D-sincos, we bilinearly interpolate the position embeddings used during training. For Factorized, we linearly interpolate the position embeddings for each axis. Fourier does not require adjustment since fractional positions along each axis are used as input. For RPE-learn, we interpolate the learned relative biases using the official BEiT implementation \cite{beit}. $2$D-ALiBi does not require adjustment either. However, we tune a parameter on minival that scales the distance penalty at each test resolution. For $2$D-RoPE, we tune its base frequency on minival --- this is a SoTA method to extrapolate RoPE used in NLP \cite{rope_scaling}. Lastly for LookHere, we tune the global slope on minival. The benefits of tuning slopes are minimal, see Appendix \ref{appn:extrapolate_tuning}.

\begin{table}[!t]\centering\small
\definecolor{COLMAX}{HTML}{b7e4d8}
\definecolor{COLMED}{HTML}{FFFFFF}
\definecolor{COLMIN}{HTML}{f4d5ca}
\newcommand{\COLUMNvalues}[2]{
\FPeval\COLUMNmid{trunc((#1+#2)/2:2)}%
\def\COLUMNmin{#1}
\def\COLUMNmax{#2}
\newcommand{\CELLvalue}[3]{
\ifstrequal{#3}{}
\csxdef{C#1R#2C}{}
\csgdef{C#1R#2}{}
\ifdimless{#3pt}{\COLUMNmid pt}
\FPeval\NORMvalue{trunc((#3-\COLUMNmin)*((100)/(\COLUMNmid-\COLUMNmin)):2)}%
\def\CELLcolor{COLMED!\NORMvalue!COLMIN}
\FPeval\NORMvalue{trunc((#3-\COLUMNmid)*((100)/(\COLUMNmax-\COLUMNmid)):2)}%
\def\CELLcolor{COLMAX!\NORMvalue!COLMED}
\csxdef{C#1R#2C}{\CELLcolor}
\csgdef{C#1R#2}{\cellcolor{\csuse{C#1R#2C}}#3}
% Column 1
\COLUMNvalues{79.05}{81.31}
\CELLvalue{1}{1}{79.45}
\CELLvalue{1}{2}{79.05}
\CELLvalue{1}{3}{79.86}
\CELLvalue{1}{4}{79.69}
\CELLvalue{1}{5}{79.86}
\CELLvalue{1}{6}{79.54}
\CELLvalue{1}{7}{80.38}
\CELLvalue{1}{8}{81.31}
\CELLvalue{1}{9}{81.02}
\CELLvalue{1}{10}{81.06}
% Column 2
\COLUMNvalues{77.26}{79.89}
\CELLvalue{2}{1}{77.35}
\CELLvalue{2}{2}{77.44}
\CELLvalue{2}{3}{77.29}
\CELLvalue{2}{4}{77.37}
\CELLvalue{2}{5}{77.26}
\CELLvalue{2}{6}{77.29}
\CELLvalue{2}{7}{78.37}
\CELLvalue{2}{8}{80.01}
\CELLvalue{2}{9}{79.89}
\CELLvalue{2}{10}{79.74}
% Column 3
\COLUMNvalues{84.62}{86.53}
\CELLvalue{3}{1}{84.97}
\CELLvalue{3}{2}{84.62}
\CELLvalue{3}{3}{85.30}
\CELLvalue{3}{4}{85.13}
\CELLvalue{3}{5}{85.46}
\CELLvalue{3}{6}{85.15}
\CELLvalue{3}{7}{85.64}
\CELLvalue{3}{8}{86.53}
\CELLvalue{3}{9}{86.44}
\CELLvalue{3}{10}{86.23}
% Column 4
\COLUMNvalues{82.87}{85.28}
\CELLvalue{4}{1}{82.87}
\CELLvalue{4}{2}{82.96}
\CELLvalue{4}{3}{82.99}
\CELLvalue{4}{4}{82.89}
\CELLvalue{4}{5}{82.88}
\CELLvalue{4}{6}{82.92}
\CELLvalue{4}{7}{83.78}
\CELLvalue{4}{8}{85.30}
\CELLvalue{4}{9}{85.28}
\CELLvalue{4}{10}{85.07}
% Column 5
\COLUMNvalues{67.86}{70.70}
\CELLvalue{5}{1}{68.49}
\CELLvalue{5}{2}{67.86}
\CELLvalue{5}{3}{69.11}
\CELLvalue{5}{4}{68.30}
\CELLvalue{5}{5}{68.57}
\CELLvalue{5}{6}{68.47}
\CELLvalue{5}{7}{69.34}
\CELLvalue{5}{8}{70.70}
\CELLvalue{5}{9}{70.28}
\CELLvalue{5}{10}{69.65}
% Column 6
\COLUMNvalues{65.17}{68.54}
\CELLvalue{6}{1}{65.17}
\CELLvalue{6}{2}{65.31}
\CELLvalue{6}{3}{65.34}
\CELLvalue{6}{4}{65.33}
\CELLvalue{6}{5}{65.19}
\CELLvalue{6}{6}{65.15}
\CELLvalue{6}{7}{66.56}
\CELLvalue{6}{8}{68.52}
\CELLvalue{6}{9}{68.54}
\CELLvalue{6}{10}{68.18}
% Column 7
\COLUMNvalues{9.85}{13.53}
\CELLvalue{7}{1}{10.97}
\CELLvalue{7}{2}{10.45}
\CELLvalue{7}{3}{11.00}
\CELLvalue{7}{4}{11.36}
\CELLvalue{7}{5}{9.85}
\CELLvalue{7}{6}{10.45}
\CELLvalue{7}{7}{13.03}
\CELLvalue{7}{8}{13.53}
\CELLvalue{7}{9}{13.15}
\CELLvalue{7}{10}{13.41}
% Column 8
\COLUMNvalues{7.16}{10.80}
\CELLvalue{8}{1}{7.58}
\CELLvalue{8}{2}{7.76}
\CELLvalue{8}{3}{7.16}
\CELLvalue{8}{4}{7.79}
\CELLvalue{8}{5}{7.18}
\CELLvalue{8}{6}{7.27}
\CELLvalue{8}{7}{8.84}
\CELLvalue{8}{8}{10.45}
\CELLvalue{8}{9}{10.80}
\CELLvalue{8}{10}{10.21}
% Column 9
\COLUMNvalues{28.26}{32.45}
\CELLvalue{9}{1}{29.64}
\CELLvalue{9}{2}{29.11}
\CELLvalue{9}{3}{29.99}
\CELLvalue{9}{4}{29.73}
\CELLvalue{9}{5}{29.10}
\CELLvalue{9}{6}{28.26}
\CELLvalue{9}{7}{32.45}
\CELLvalue{9}{8}{32.10}
\CELLvalue{9}{9}{31.77}
\CELLvalue{9}{10}{32.12}
% Column 10
\COLUMNvalues{24.62}{29.51}
\CELLvalue{10}{1}{25.73}
\CELLvalue{10}{2}{26.07}
\CELLvalue{10}{3}{26.18}
\CELLvalue{10}{4}{24.62}
\CELLvalue{10}{5}{24.62}
\CELLvalue{10}{6}{24.41}
\CELLvalue{10}{7}{28.55}
\CELLvalue{10}{8}{28.94}
\CELLvalue{10}{9}{29.47}
\CELLvalue{10}{10}{29.51}
% Column 11
\COLUMNvalues{87.58}{89.90}
\CELLvalue{11}{1}{88.28}
\CELLvalue{11}{2}{87.58}
\CELLvalue{11}{3}{87.86}
\CELLvalue{11}{4}{88.14}
\CELLvalue{11}{5}{88.22}
\CELLvalue{11}{6}{87.70}
\CELLvalue{11}{7}{88.78}
\CELLvalue{11}{8}{89.86}
\CELLvalue{11}{9}{89.90}
\CELLvalue{11}{10}{89.46}
% Column 12
\COLUMNvalues{85.13}{87.86}
\CELLvalue{12}{1}{85.22}
\CELLvalue{12}{2}{85.36}
\CELLvalue{12}{3}{85.37}
\CELLvalue{12}{4}{85.39}
\CELLvalue{12}{5}{85.17}
\CELLvalue{12}{6}{85.13}
\CELLvalue{12}{7}{86.35}
\CELLvalue{12}{8}{87.80}
\CELLvalue{12}{9}{87.86}
\CELLvalue{12}{10}{87.43}
\newcommand{\CELLvaluedsaddasdsadasdad}[3]{%
\FPeval\MIDvalue{(#1+#2)/2}%
\FPeval\NORMvalue{(#3-#1)*((100)/(#2-#1))}%
\caption{Top-$1$ acc. ($\%$) for ViT-B models trained on ImageNet for $150$ epochs; trained and tested at $224^2$. We report the best and average results across our $8$-run hyper-parameter sweep.}
\label{tab:main_224}
\setlength{\tabcolsep}{4pt}
\begin{tabular}{
\toprule
& \multicolumn{2}{c}{Val \cite{imagenet}} & \multicolumn{2}{c}{ReaL \cite{real}} & \multicolumn{2}{c}{v$2$ \cite{v2}} & \multicolumn{2}{c}{-A \cite{a}} & \multicolumn{2}{c}{-R \cite{r}} & \multicolumn{2}{c}{-HR (ours)} \\
\cmidrule(lr){2-3}
\cmidrule(lr){4-5}
\cmidrule(lr){6-7}
\cmidrule(lr){8-9}
\cmidrule(lr){10-11}
\cmidrule(lr){12-13}
Method               &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{Best} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{Avg.} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{Best} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{Avg.} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{Best} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{Avg.} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{Best} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{Avg.} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{Best} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{Avg.} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{Best} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{Avg.} \\
\midrule
$1$D-learn             & \csuse{C1R1}  & \csuse{C2R1}  & \csuse{C3R1}  & \csuse{C4R1}  & \csuse{C5R1}  & \csuse{C6R1}  & \csuse{C7R1}  & \csuse{C8R1}  & \csuse{C9R1}  & \csuse{C10R1}  & \csuse{C11R1}  & \csuse{C12R1} \\
$2$D-sincos            & \csuse{C1R2}  & \csuse{C2R2}  & \csuse{C3R2}  & \csuse{C4R2}  & \csuse{C5R2}  & \csuse{C6R2}  & \csuse{C7R2}  & \csuse{C8R2}  & \csuse{C9R2}  & \csuse{C10R2} & \csuse{C11R2}  & \csuse{C12R2} \\
Factorized           & \csuse{C1R3}  & \csuse{C2R3}  & \csuse{C3R3}  & \csuse{C4R3}  & \csuse{C5R3}  & \csuse{C6R3}  & \csuse{C7R3}  & \csuse{C8R3}  & \csuse{C9R3}  & \csuse{C10R3} & \csuse{C11R3}  & \csuse{C12R3} \\
Fourier              & \csuse{C1R4}  & \csuse{C2R4}  & \csuse{C3R4}  & \csuse{C4R4}  & \csuse{C5R4}  & \csuse{C6R4}  & \csuse{C7R4}  & \csuse{C8R4}  & \csuse{C9R4}  & \csuse{C10R4} & \csuse{C11R4}  & \csuse{C12R4} \\
RPE-learn            & \csuse{C1R5}  & \csuse{C2R5}  & \csuse{C3R5}  & \csuse{C4R5}  & \csuse{C5R5}  & \csuse{C6R5}  & \csuse{C7R5}  & \csuse{C8R5}  & \csuse{C9R5}  & \csuse{C10R5} & \csuse{C11R5}  & \csuse{C12R5} \\
$2$D-ALiBi             & \csuse{C1R6}  & \csuse{C2R6}  & \csuse{C3R6}  & \csuse{C4R6}  & \csuse{C5R6}  & \csuse{C6R6}  & \csuse{C7R6}  & \csuse{C8R6}  & \csuse{C9R6}  & \csuse{C10R6} & \csuse{C11R6}  & \csuse{C12R6} \\
$2$D-RoPE              & \csuse{C1R7}  & \csuse{C2R7}  & \csuse{C3R7}  & \csuse{C4R7}  & \csuse{C5R7}  & \csuse{C6R7}  & \csuse{C7R7}  & \csuse{C8R7}  & \csuse{C9R7}  & \csuse{C10R7} & \csuse{C11R7}  & \csuse{C12R7} \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$180$} & \csuse{C1R8}  & \csuse{C2R8}  & \csuse{C3R8}  & \csuse{C4R8}  & \csuse{C5R8}  & \csuse{C6R8}  & \csuse{C7R8}  & \csuse{C8R8}  & \csuse{C9R8}  & \csuse{C10R8} & \csuse{C11R8}  & \csuse{C12R8} \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$} & \csuse{C1R9}  & \csuse{C2R9}  & \csuse{C3R9}  & \csuse{C4R9}  & \csuse{C5R9}  & \csuse{C6R9}  & \csuse{C7R9}  & \csuse{C8R9}  & \csuse{C9R9}  & \csuse{C10R9} & \csuse{C11R9}  & \csuse{C12R9} \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$45$} & \csuse{C1R10} & \csuse{C2R10} & \csuse{C3R10} & \csuse{C4R10} & \csuse{C5R10} & \csuse{C6R10} & \csuse{C7R10} & \csuse{C8R10} & \csuse{C9R10} & \csuse{C10R10} & \csuse{C11R10}  & \csuse{C12R10} \\
\bottomrule
\end{tabular}
\end{table}
\begin{table}[!t]\small
\vspace{-15pt}
\definecolor{COLMAX}{HTML}{b7e4d8}
\definecolor{COLMED}{HTML}{FFFFFF}
\definecolor{COLMIN}{HTML}{f4d5ca}
\newcommand{\COLUMNvalues}[2]{
\FPeval\COLUMNmid{trunc((#1+#2)/2:2)}%
\def\COLUMNmin{#1}
\def\COLUMNmax{#2}
\newcommand{\CELLvalue}[3]{
\ifstrequal{#3}{}
\csxdef{C#1R#2C}{}
\csgdef{C#1R#2}{}
\ifdimless{#3pt}{\COLUMNmid pt}
\FPeval\NORMvalue{trunc((#3-\COLUMNmin)*((100)/(\COLUMNmid-\COLUMNmin)):2)}%
\def\CELLcolor{COLMED!\NORMvalue!COLMIN}
\FPeval\NORMvalue{trunc((#3-\COLUMNmid)*((100)/(\COLUMNmax-\COLUMNmid)):2)}%
\def\CELLcolor{COLMAX!\NORMvalue!COLMED}
\csxdef{C#1R#2C}{\CELLcolor}
\csgdef{C#1R#2}{\cellcolor{\csuse{C#1R#2C}}#3}
% Column 1
\COLUMNvalues{58.07}{65.06}
\CELLvalue{1}{1}{58.87}
\CELLvalue{1}{2}{60.38}
\CELLvalue{1}{3}{60.86}
\CELLvalue{1}{4}{59.91}
\CELLvalue{1}{5}{59.81}
\CELLvalue{1}{6}{58.07}
\CELLvalue{1}{7}{60.59}
\CELLvalue{1}{8}{65.06}
\CELLvalue{1}{9}{63.89}
\CELLvalue{1}{10}{64.71}
% Column 2
\COLUMNvalues{53.36}{62.59}
\CELLvalue{2}{1}{54.36}
\CELLvalue{2}{2}{55.16}
\CELLvalue{2}{3}{56.19}
\CELLvalue{2}{4}{54.74}
\CELLvalue{2}{5}{53.36}
\CELLvalue{2}{6}{53.68}
\CELLvalue{2}{7}{57.16}
\CELLvalue{2}{8}{62.59}
\CELLvalue{2}{9}{61.88}
\CELLvalue{2}{10}{61.71}
% Column 3
\COLUMNvalues{43.32}{51.81}
\CELLvalue{3}{1}{44.23}
\CELLvalue{3}{2}{45.37}
\CELLvalue{3}{3}{46.34}
\CELLvalue{3}{4}{44.99}
\CELLvalue{3}{5}{45.04}
\CELLvalue{3}{6}{43.32}
\CELLvalue{3}{7}{47.11}
\CELLvalue{3}{8}{51.81}
\CELLvalue{3}{9}{50.87}
\CELLvalue{3}{10}{50.21}
% Column 4
\COLUMNvalues{40.19}{49.06}
\CELLvalue{4}{1}{41.37}
\CELLvalue{4}{2}{41.61}
\CELLvalue{4}{3}{42.32}
\CELLvalue{4}{4}{41.90}
\CELLvalue{4}{5}{40.19}
\CELLvalue{4}{6}{40.30}
\CELLvalue{4}{7}{43.77}
\CELLvalue{4}{8}{49.06}
\CELLvalue{4}{9}{48.07}
\CELLvalue{4}{10}{47.86}
% Column 5
\COLUMNvalues{26.2}{32.7}
\CELLvalue{5}{1}{29.5}
\CELLvalue{5}{2}{29.2}
\CELLvalue{5}{3}{29.4}
\CELLvalue{5}{4}{29.8}
\CELLvalue{5}{5}{26.4}
\CELLvalue{5}{6}{26.2}
\CELLvalue{5}{7}{29.9}
\CELLvalue{5}{8}{32.4}
\CELLvalue{5}{9}{32.6}
\CELLvalue{5}{10}{32.7}
% Column 6
\COLUMNvalues{37.25}{40.60}
\CELLvalue{6}{1}{38.05}
\CELLvalue{6}{2}{38.39}
\CELLvalue{6}{3}{37.95}
\CELLvalue{6}{4}{38.26}
\CELLvalue{6}{5}{37.25}
\CELLvalue{6}{6}{37.56}
\CELLvalue{6}{7}{39.74}
\CELLvalue{6}{8}{40.29}
\CELLvalue{6}{9}{40.60}
\CELLvalue{6}{10}{40.07}
% Column 7
\COLUMNvalues{42.90}{55.5}
\CELLvalue{7}{1}{47.1}
\CELLvalue{7}{2}{45.3}
\CELLvalue{7}{3}{45.9}
\CELLvalue{7}{4}{46.2}
\CELLvalue{7}{5}{42.9}
\CELLvalue{7}{6}{48.4}
\CELLvalue{7}{7}{47.0}
\CELLvalue{7}{8}{55.0}
\CELLvalue{7}{9}{55.3}
\CELLvalue{7}{10}{55.5}
% Column 8
\COLUMNvalues{72.51}{75.53}
\CELLvalue{8}{1}{72.93}
\CELLvalue{8}{2}{72.91}
\CELLvalue{8}{3}{72.51}
\CELLvalue{8}{4}{73.60}
\CELLvalue{8}{5}{73.87}
\CELLvalue{8}{6}{73.92}
\CELLvalue{8}{7}{75.53}
\CELLvalue{8}{8}{75.05}
\CELLvalue{8}{9}{74.90}
\CELLvalue{8}{10}{74.42}
% Column 9 ECE
\COLUMNvalues{8.28}{10.14}
\CELLvalue{9}{1}{10.13}
\CELLvalue{9}{2}{10.14}
\CELLvalue{9}{3}{10.01}
\CELLvalue{9}{4}{9.65}
\CELLvalue{9}{5}{8.66}
\CELLvalue{9}{6}{9.26}
\CELLvalue{9}{7}{9.60}
\CELLvalue{9}{8}{8.28}
\CELLvalue{9}{9}{8.68}
\CELLvalue{9}{10}{8.87}
% Column 10 ECE
\COLUMNvalues{9.76}{12.21}
\CELLvalue{10}{1}{12.21}
\CELLvalue{10}{2}{11.85}
\CELLvalue{10}{3}{11.37}
\CELLvalue{10}{4}{12.13}
\CELLvalue{10}{5}{11.42}
\CELLvalue{10}{6}{11.24}
\CELLvalue{10}{7}{11.48}
\CELLvalue{10}{8}{9.76}
\CELLvalue{10}{9}{9.91}
\CELLvalue{10}{10}{9.99}
\newcommand{\CELLvaluedsaddasdsadasdad}[3]{%
\FPeval\MIDvalue{(#1+#2)/2}%
\FPeval\NORMvalue{(#3-#1)*((100)/(#2-#1))}%
\hspace{0pt}
\begin{minipage}{0.40\textwidth}
\caption{Fast Gradient Sign Method attack \cite{fgsm} ($\%$ top-$5$ acc. on Val), best and average runs.}
\label{tab:adv}
\setlength{\tabcolsep}{4pt}
\begin{tabular}{
\toprule
& \multicolumn{2}{c}{$\epsilon = 1/255$} & \multicolumn{2}{c}{$\epsilon = 3/255$} \\
\cmidrule(lr){2-3}
\cmidrule(lr){4-5}

Method               &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{Best} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{Avg.} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{Best} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{Avg.} \\
\midrule
$1$D-learn             & \csuse{C1R1}  & \csuse{C2R1}  & \csuse{C3R1}  & \csuse{C4R1}  \\
$2$D-sincos            & \csuse{C1R2}  & \csuse{C2R2}  & \csuse{C3R2}  & \csuse{C4R2}  \\
Factorized           & \csuse{C1R3}  & \csuse{C2R3}  & \csuse{C3R3}  & \csuse{C4R3}  \\
Fourier              & \csuse{C1R4}  & \csuse{C2R4}  & \csuse{C3R4}  & \csuse{C4R4}  \\
RPE-learn            & \csuse{C1R5}  & \csuse{C2R5}  & \csuse{C3R5}  & \csuse{C4R5}  \\
$2$D-ALiBi             & \csuse{C1R6}  & \csuse{C2R6}  & \csuse{C3R6}  & \csuse{C4R6}  \\
$2$D-RoPE              & \csuse{C1R7}  & \csuse{C2R7}  & \csuse{C3R7}  & \csuse{C4R7}  \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$180$} & \csuse{C1R8}  & \csuse{C2R8}  & \csuse{C3R8}  & \csuse{C4R8}  \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$} & \csuse{C1R9}  & \csuse{C2R9}  & \csuse{C3R9}  & \csuse{C4R9}  \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$45$} & \csuse{C1R10} & \csuse{C2R10} & \csuse{C3R10} & \csuse{C4R10} \\
\bottomrule
\end{tabular}
\end{minipage}
\hspace{5pt}
\begin{minipage}{0.21\textwidth}
\definecolor{COLMAX}{HTML}{f4d5ca}
\definecolor{COLMED}{HTML}{FFFFFF}
\definecolor{COLMIN}{HTML}{b7e4d8}
\caption{Expected Calibration Error $\%$ \cite{ece} ($\downarrow$) on Val, best and average runs.}
\vspace{4pt}
\setlength{\tabcolsep}{10pt}
\label{tab:calib}
\begin{tabular}{
\toprule
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{Best} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{Avg.} \\
\midrule
\csuse{C9R1}  & \csuse{C10R1}  \\
\csuse{C9R2}  & \csuse{C10R2}  \\
\csuse{C9R3}  & \csuse{C10R3}  \\
\csuse{C9R4}  & \csuse{C10R4}   \\
\csuse{C9R5}  & \csuse{C10R5}   \\
\csuse{C9R6}  & \csuse{C10R6}  \\
\csuse{C9R7}  & \csuse{C10R7}  \\
\csuse{C9R8}  & \csuse{C10R8}   \\
\csuse{C9R9}  & \csuse{C10R9}  \\
\csuse{C9R10} & \csuse{C10R10} \\
\bottomrule
\end{tabular}
\end{minipage}
\hspace{4pt}
\begin{minipage}{0.33\textwidth}
\definecolor{COLMAX}{HTML}{b7e4d8}
\definecolor{COLMED}{HTML}{FFFFFF}
\definecolor{COLMIN}{HTML}{f4d5ca}
\caption{Semantic Segmentation ($\%$ mIoU), linear probing (LP) and finetuning (FT).}
\hspace{-6pt}
\label{tab:seg}
\begin{tabular}{
\toprule
\multicolumn{2}{c}{ADE$20$k} & \multicolumn{2}{c}{Cityscapes} \\
\cmidrule(lr){1-2}
\cmidrule(lr){3-4}
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{LP} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{FT} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{LP} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{FT} \\
\midrule
\csuse{C5R1}  & \csuse{C6R1}  & \csuse{C7R1}  & \csuse{C8R1}  \\
\csuse{C5R2}  & \csuse{C6R2}  & \csuse{C7R2}  & \csuse{C8R2}  \\
\csuse{C5R3}  & \csuse{C6R3}  & \csuse{C7R3}  & \csuse{C8R3}  \\
\csuse{C5R4}  & \csuse{C6R4}  & \csuse{C7R4}  & \csuse{C8R4}  \\
\csuse{C5R5}  & \csuse{C6R5}  & \csuse{C7R5}  & \csuse{C8R5}  \\
\csuse{C5R6}  & \csuse{C6R6}  & \csuse{C7R6}  & \csuse{C8R6}  \\
\csuse{C5R7}  & \csuse{C6R7}  & \csuse{C7R7}  & \csuse{C8R7}  \\
\csuse{C5R8}  & \csuse{C6R8}  & \csuse{C7R8}  & \csuse{C8R8}  \\
\csuse{C5R9}  & \csuse{C6R9}  & \csuse{C7R9}  & \csuse{C8R9}  \\
\csuse{C5R10} & \csuse{C6R10} & \csuse{C7R10} & \csuse{C8R10} \\
\bottomrule
\end{tabular}
\end{minipage}
\end{table}

\subsection{Results and Analysis}

% \vspace{-10pt}
\begin{wrapfigure}[13]{r}{0.4\linewidth}%
\vspace{-50pt}
\definecolor{LH}{HTML}{9A46B9}
\definecolor{alibi}{HTML}{77F708}
\definecolor{rope}{HTML}{F71008}
\definecolor{rpe-learn}{HTML}{0555FA}
\definecolor{embs}{HTML}{FAAA05}

\pgfplotsset{
title style={font=\small, align=center, yshift=-8pt},
symbolic x coords={1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12},
tick pos=left,
xtick align=outside,
ytick align=outside,
tick label style={font=\tiny, inner sep=0pt},
enlarge x limits={0.01},
major grid style={solid, line width=.5pt},
minor grid style={dashed},
major y tick style={transparent},
% minor y tick style={transparent},
ytick distance=0.1,
% minor y tick num=4,
extra y ticks={0},
extra y tick labels={},
extra y tick style={grid style={black, solid}},
1D-learn/.style={embs, line width=1.5pt, dashed},
2D-sincos/.style={embs, line width=1.5pt, dotted},
Factorized/.style={embs, line width=1.5pt, mark=x},
Fourier/.style={embs, line width=1.5pt, mark=o},
RPE-learn/.style={rpe-learn, line width=1.5pt},
2D-ALiBi/.style={alibi, line width=1.5pt},
2D-RoPE/.style={rope, line width=1.5pt},
LH-180/.style={LH, line width=1.5pt, dotted},
LH-90/.style={LH, line width=1.5pt, dashed},
LH-45/.style={LH, line width=1.5pt},
\begin{tikzpicture}
\begin{axis}[
title={Head Diversity by Layer},
xlabel style={yshift=0pt},
ylabel={JS Divergence},
ylabel style={yshift=-10pt},
\addplot[1D-learn] table[x=Diversity,y=1D-learn,col sep=comma] {jsd_data.csv};
\addplot[2D-sincos] table[x=Diversity,y=2D-sincos,col sep=comma] {jsd_data.csv};
\addplot[Factorized] table[x=Diversity,y=Factorized,col sep=comma] {jsd_data.csv};
\addplot[Fourier] table[x=Diversity,y=Fourier,col sep=comma] {jsd_data.csv};
\addplot[RPE-learn] table[x=Diversity,y=RPE-learn,col sep=comma] {jsd_data.csv};
\addplot[2D-ALiBi] table[x=Diversity,y=2D-ALiBi,col sep=comma] {jsd_data.csv};
\addplot[2D-RoPE] table[x=Diversity,y=2D-RoPE,col sep=comma] {jsd_data.csv};
\addplot[LH-180] table[x=Diversity,y=LH-180,col sep=comma] {jsd_data.csv};
\addplot[LH-90] table[x=Diversity,y=LH-90,col sep=comma] {jsd_data.csv};
\addplot[LH-45] table[x=Diversity,y=LH-45,col sep=comma] {jsd_data.csv};
% \hspace{50pt}
\end{axis}
\end{tikzpicture}
\vspace{-10pt}
\caption{\textcolor{LHcolor_dark}{\textbf{LookHere}} learns more diverse attention heads and prevents attention collapse. Legend follows Figures \ref{fig:main_figure} \ref{fig:object_size}.}
\label{fig:jsd}
\end{wrapfigure}

LookHere improves ViT sample efficiency (Table \ref{tab:main_224}). Our three variants outperform the best baseline, $2$D-RoPE, under almost all test conditions (the single exception being the best $2$D-RoPE model on -R). LookHere further improves gains when considering averaged results --- i.e., when accuracy values are averaged over $8$ hyperparameter configurations (please see the Appendix \ref{appn:fullresults} for individual results). For instance, LH-$180$ outperforms $2$D-RoPE by $0.93\%$ / $1.36\%$ on  on Val / v$2$ on our best runs and by $1.64\%$ / $1.96\%$ on Val / v$2$ on our averaged runs ---  indicating that LookHere decreases hyperparameter sensitivity. Surprisingly, LH-$180$ \textit{averages} $80.01\%$ on Val, which matches the \textit{best} run trained for twice as long by Steiner et al. \cite{how}.

LookHere improves ViT adversarial robustness and model calibration (Tables \ref{tab:adv} \ref{tab:calib}); both have been linked to ensemble diversity \cite{ensemble_cal_1, ensemble_cal_2, ensemble_adv}, which we offer as a potential explanation. This is an interesting finding because adversarial robustness and calibration can be at odds with accuracy \cite{cal, adv_corr}. We show that LookHere learns more diverse attention heads by measuring the generalized Jensen-Shannon divergence \cite{jsd} between heads (Figure \ref{fig:jsd}). In the Appendix \ref{appn:measure}, we measure more properties of models leveraging different position encoding methods. LookHere significantly outperforms other methods on segmentation linear probing, demonstrating its ability to learn spatially-aware patch representations. LookHere also performs well with segmentation finetuning, achieving comparable performance to $2$D-RoPE (Table \ref{tab:seg}).

\usetikzlibrary{positioning}
\begin{figure}[!t]
\centering
\begin{tikzpicture}
% Row 1
\node[anchor=south west, inner sep=0] (image1) at (0,0)
\node[above=-0.5mm of image1] (label1) {$1$D-learn ($4.1\%$)};

\node[anchor=south west, inner sep=0] (image2) at (image1.south east)
\node[above=-0.5mm of image2] (label2) {$2$D-sincos ($2.1\%$)};

\node[anchor=south west, inner sep=0] (image3) at (image2.south east)
\node[above=-0.5mm of image3] (label3) {Factorized ($8.7\%$)};

\node[anchor=south west, inner sep=0] (image4) at (image3.south east)
\node[above=-0.5mm of image4] (label4) {Fourier ($3.3\%$)};

\node[anchor=south west, inner sep=0] (image5) at (image4.south east)
\node[above=-0.5mm of image5] (label5) {RPE-learn ($6.1\%$)};

% Row 2
\node[anchor=north west, inner sep=0] (image6) at (image1.south west)
\node[below=-0.5mm of image6] (label6) {$2$D-ALiBi ($6.5\%$)};

\node[anchor=north west, inner sep=0] (image7) at (image6.north east)
\node[below=-0.5mm of image7] (label7) {$2$D-RoPE ($8.7\%$)};

\node[anchor=north west, inner sep=0] (image8) at (image7.north east)
\node[below=-0.5mm of image8] (label8) {\textcolor{LHcolor_dark}{\textbf{LH-}\boldmath{$180$}} ($31.2\%$)};

\node[anchor=north west, inner sep=0] (image9) at (image8.north east)
\node[below=-0.5mm of image9] (label9) {\textcolor{LHcolor_dark}{\textbf{LH-}\boldmath{$90$}} ($31.1\%$)};

\node[anchor=north west, inner sep=0] (image10) at (image9.north east)
\node[below=-0.5mm of image10] (label10) {\textcolor{LHcolor_dark}{\textbf{LH-}\boldmath{$45$}} ($30.7\%$)};

\end{tikzpicture}

\caption{We apply frozen MLP classifying heads (learned on the CLS token) on frozen patch representations. We visualize ImageNet class predictions: \texttt{assault rifle} (\textcolor{red}{\textbf{red}}), \texttt{bulletproof vest} (\textcolor{green}{\textbf{green}}), \texttt{crash helmet} (\textcolor{blue}{\textbf{blue}}), and \texttt{holster} (\colorbox{black}{\textcolor{white}{white}}). In parentheses, we show mIoU results (@224px) on ImageNet-S \cite{gao2022luss}, where we apply this technique to segment images \textit{without} training.
\vspace{-15pt}
\label{fig:patch_logits}
\end{figure}

\begin{wraptable}[14]{r}{0.5\linewidth}%
\vspace{0pt}

\centering\small
\definecolor{COLMAX}{HTML}{b7e4d8}
\definecolor{COLMED}{HTML}{FFFFFF}
\definecolor{COLMIN}{HTML}{f4d5ca}
\newcommand{\COLUMNvalues}[2]{
\FPeval\COLUMNmid{trunc((#1+#2)/2:2)}%
\def\COLUMNmin{#1}
\def\COLUMNmax{#2}
\newcommand{\CELLvalue}[3]{
\ifstrequal{#3}{}
\csxdef{C#1R#2C}{}
\csgdef{C#1R#2}{}
\ifdimless{#3pt}{\COLUMNmid pt}
\FPeval\NORMvalue{trunc((#3-\COLUMNmin)*((100)/(\COLUMNmid-\COLUMNmin)):2)}%
\def\CELLcolor{COLMED!\NORMvalue!COLMIN}
\FPeval\NORMvalue{trunc((#3-\COLUMNmid)*((100)/(\COLUMNmax-\COLUMNmid)):2)}%
\def\CELLcolor{COLMAX!\NORMvalue!COLMED}
\csxdef{C#1R#2C}{\CELLcolor}
\csgdef{C#1R#2}{\cellcolor{\csuse{C#1R#2C}}#3}
% Column 1
\COLUMNvalues{81.33}{83.28}
\CELLvalue{1}{1}{81.46}
\CELLvalue{1}{2}{81.33}
\CELLvalue{1}{3}{81.50}
\CELLvalue{1}{4}{81.71}
\CELLvalue{1}{5}{82.01}
\CELLvalue{1}{6}{81.41}
\CELLvalue{1}{7}{82.31}
\CELLvalue{1}{8}{83.28}
\CELLvalue{1}{9}{83.08}
\CELLvalue{1}{10}{83.10}
% Column 2
\COLUMNvalues{86.46}{88.05}
\CELLvalue{2}{1}{86.46}
\CELLvalue{2}{2}{86.50}
\CELLvalue{2}{3}{86.62}
\CELLvalue{2}{4}{86.73}
\CELLvalue{2}{5}{87.17}
\CELLvalue{2}{6}{86.73}
\CELLvalue{2}{7}{87.21}
\CELLvalue{2}{8}{88.05}
\CELLvalue{2}{9}{87.99}
\CELLvalue{2}{10}{87.83}
% Column 3
\COLUMNvalues{70.50}{73.12}
\CELLvalue{3}{1}{70.69}
\CELLvalue{3}{2}{70.53}
\CELLvalue{3}{3}{70.95}
\CELLvalue{3}{4}{71.01}
\CELLvalue{3}{5}{71.66}
\CELLvalue{3}{6}{70.50}
\CELLvalue{3}{7}{71.82}
\CELLvalue{3}{8}{73.12}
\CELLvalue{3}{9}{72.99}
\CELLvalue{3}{10}{72.43}
% Column 4
\COLUMNvalues{17.73}{23.51}
\CELLvalue{4}{1}{18.80}
\CELLvalue{4}{2}{17.73}
\CELLvalue{4}{3}{18.05}
\CELLvalue{4}{4}{19.73}
\CELLvalue{4}{5}{18.13}
\CELLvalue{4}{6}{18.01}
\CELLvalue{4}{7}{21.68}
\CELLvalue{4}{8}{22.85}
\CELLvalue{4}{9}{23.51}
\CELLvalue{4}{10}{22.39}
% Column 5
\COLUMNvalues{28.60}{33.38}
\CELLvalue{5}{1}{29.80}
\CELLvalue{5}{2}{29.26}
\CELLvalue{5}{3}{29.98}
\CELLvalue{5}{4}{29.68}
\CELLvalue{5}{5}{29.53}
\CELLvalue{5}{6}{28.60}
\CELLvalue{5}{7}{33.38}
\CELLvalue{5}{8}{32.95}
\CELLvalue{5}{9}{32.63}
\CELLvalue{5}{10}{33.10}
% Column 6
\COLUMNvalues{89.46}{91.38}
\CELLvalue{6}{1}{89.82}
\CELLvalue{6}{2}{89.62}
\CELLvalue{6}{3}{89.50}
\CELLvalue{6}{4}{89.90}
\CELLvalue{6}{5}{90.20}
\CELLvalue{6}{6}{89.46}
\CELLvalue{6}{7}{89.92}
\CELLvalue{6}{8}{91.38}
\CELLvalue{6}{9}{91.24}
\CELLvalue{6}{10}{90.92}
\newcommand{\CELLvaluedsaddasdsadasdad}[3]{%
\FPeval\MIDvalue{(#1+#2)/2}%
\FPeval\NORMvalue{(#3-#1)*((100)/(#2-#1))}%
\caption{Top-1 acc. ($\%$) for models trained at $224^2$ px, finetuned and tested at $384^2$ px.}
\label{tab:hr_ft}
\setlength{\tabcolsep}{2.4pt}
\begin{tabular}{
\toprule
Method               &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{Val} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{ReaL} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{-v$2$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{-A} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{-R} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{-HR} \\
\midrule
$1$D-learn             & \csuse{C1R1}  & \csuse{C2R1}  & \csuse{C3R1}  & \csuse{C4R1}  & \csuse{C5R1}  & \csuse{C6R1}  \\
$2$D-sincos           & \csuse{C1R2}  & \csuse{C2R2}  & \csuse{C3R2}  & \csuse{C4R2}  & \csuse{C5R2}  & \csuse{C6R2}  \\
Factorized           & \csuse{C1R3}  & \csuse{C2R3}  & \csuse{C3R3}  & \csuse{C4R3}  & \csuse{C5R3}  & \csuse{C6R3}   \\
Fourier              & \csuse{C1R4}  & \csuse{C2R4}  & \csuse{C3R4}  & \csuse{C4R4}  & \csuse{C5R4}  & \csuse{C6R4}  \\
RPE-learn            & \csuse{C1R5}  & \csuse{C2R5}  & \csuse{C3R5}  & \csuse{C4R5}  & \csuse{C5R5}  & \csuse{C6R5}  \\
$2$D-ALiBi           & \csuse{C1R6}  & \csuse{C2R6}  & \csuse{C3R6}  & \csuse{C4R6}  & \csuse{C5R6}  & \csuse{C6R6}  \\
$2$D-RoPE             & \csuse{C1R7}  & \csuse{C2R7}  & \csuse{C3R7}  & \csuse{C4R7}  & \csuse{C5R7}  & \csuse{C6R7}  \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$180$} & \csuse{C1R8}  & \csuse{C2R8}  & \csuse{C3R8}  & \csuse{C4R8}  & \csuse{C5R8}  & \csuse{C6R8}  \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}& \csuse{C1R9}  & \csuse{C2R9}  & \csuse{C3R9}  & \csuse{C4R9}  & \csuse{C5R9}  & \csuse{C6R9}  \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$45$} & \csuse{C1R10} & \csuse{C2R10} & \csuse{C3R10} & \csuse{C4R10} & \csuse{C5R10} & \csuse{C6R10} \\
\bottomrule
\end{tabular}
\end{wraptable}

High-resolution finetuning increases the performance advantage of all three LookHere variants over $2$D-RoPE (Table \ref{tab:hr_ft}). This aligns with our intuition that improving extrapolation methods can improve high-resolution finetuning. Lower initial finetuning loss has been linked to better retaining the general representations learned during pretraining \cite{lpft}, and better extrapolating models have lower initial loss at a higher-resolution, by definition.

Using a ``logit lens'' \cite{Nostalgebraist} approach, we project patch representations onto the class embedding space \cite{vilas2023analyzing_vit}. We observe that LookHere encodes semantic information in its patches faithful to the original patch location; these patch-level predictions act as a segmentation map that can be generated without additional training. The officer in Figure \ref{fig:patch_logits} is not a one-off example; using ImageNet-S \cite{gao2022luss}, we see that LookHere outperforms $2$D-RoPE by at least $22\%$ mIoU using this patch-projection method (Figure \ref{fig:patch_logits}). Our best explanation is that, by restricting attention, LookHere prevents the attention collapse at deeper layers observed in Figure \ref{fig:jsd} that divorces patch representations from their original patch locations; this collapse has been observed in other ViTs \cite{parkwhat, attn_ent}. We also expect that preventing attention collapse will benefit vision-language models, where frozen patch representations are used as ``image tokens'' that \textit{should} represent their original patch locations \cite{vlm_1, vlm_2, vlm_3}. More examples and detailed analysis are in Appendix \ref{appn:logitlens}

\begin{figure}[!t]\centering
\vspace{-20pt}
\definecolor{LH}{HTML}{9A46B9}
\definecolor{alibi}{HTML}{77F708}
\definecolor{rope}{HTML}{F71008}
\definecolor{rpe-learn}{HTML}{0555FA}
\definecolor{embs}{HTML}{FAAA05}
% \vspace{-20pt}
\pgfplotsset{
title style={font=\small,align=center,yshift=-8pt},
% xlabel={Test Resolution},
% ylabel={Top-1 Accuracy [\%]},
% label style={font=\small,inner sep=0pt},
% xlabel style={yshift=2pt},
% ylabel style={yshift=-17pt},
symbolic x coords={0, 1, 2, 3, 4},
xticklabels={$1^{\text{st}}$, $2^{\text{nd}}$, $3^{\text{rd}}$, $4^{\text{th}}$, $5^{\text{th}}$},
tick pos=left,
xtick align=outside,
ytick align=outside,
tick label style={font=\tiny,inner sep=0pt},
enlarge x limits={0.01},
major grid style={solid, line width=.5pt},
minor grid style={dashed},
major y tick style={transparent},
minor y tick style={transparent},
ytick distance=5,
minor y tick num=4,
extra y ticks={0},
extra y tick labels={},
extra y tick style={grid style={black,solid}},
1D-learn/.style={embs, line width=1.5pt, dashed},
2D-sincos/.style={embs, line width=1.5pt, dotted},
Factorized/.style={embs, line width=1.5pt, mark=x},
Fourier/.style={embs, line width=1.5pt, mark=o},
RPE-learn/.style={rpe-learn, line width=1.5pt},
2D-ALiBi/.style={alibi, line width=1.5pt},
2D-RoPE/.style={rope, line width=1.5pt},
LH-180/.style={LH, line width=1.5pt, dotted},
LH-90/.style={LH, line width=1.5pt, dashed},
LH-45/.style={LH, line width=1.5pt},
\begin{tikzpicture}
\begin{groupplot}[group style={group size=4 by 1,
horizontal sep=3\baselineskip,
vertical sep=3.2\baselineskip},
\hspace{-46pt}
\nextgroupplot[title={$\Delta$ Acc.  $224^2\rightarrow384^2$},ymin=-2.5,ymax=3.3,
ylabel={Top-1 Accuracy [\%]},
ylabel style={yshift=-10pt},
label style={font=\small,inner sep=0pt},
\addplot[1D-learn] table[x=Size,y=1D-learn,col sep=comma] {distance_size_data/size_384_p.csv};
\addplot[2D-sincos] table[x=Size,y=2D-sincos,col sep=comma] {distance_size_data/size_384_p.csv};
\addplot[Factorized] table[x=Size,y=Factorized,col sep=comma] {distance_size_data/size_384_p.csv};
\addplot[Fourier] table[x=Size,y=Fourier,col sep=comma] {distance_size_data/size_384_p.csv};
\addplot[RPE-learn] table[x=Size,y=RPE-learn,col sep=comma] {distance_size_data/size_384_p.csv};
\addplot[2D-ALiBi] table[x=Size,y=2D-ALiBi,col sep=comma] {distance_size_data/size_384_p.csv};
\addplot[2D-RoPE] table[x=Size,y=2D-RoPE,col sep=comma] {distance_size_data/size_384_p.csv};
\addplot[LH-180] table[x=Size,y=LH-180,col sep=comma] {distance_size_data/size_384_p.csv};
\addplot[LH-90] table[x=Size,y=LH-90,col sep=comma] {distance_size_data/size_384_p.csv};
\addplot[LH-45] table[x=Size,y=LH-45,col sep=comma] {distance_size_data/size_384_p.csv};
\hspace{45pt}
\nextgroupplot[title={$\Delta$ Acc.  $224^2\rightarrow512^2$},ymin=-8,ymax=3.2,
\addplot[1D-learn] table[x=Size,y=1D-learn,col sep=comma] {distance_size_data/size_512_p.csv};
\addplot[2D-sincos] table[x=Size,y=2D-sincos,col sep=comma] {distance_size_data/size_512_p.csv};
\addplot[Factorized] table[x=Size,y=Factorized,col sep=comma] {distance_size_data/size_512_p.csv};
\addplot[Fourier] table[x=Size,y=Fourier,col sep=comma] {distance_size_data/size_512_p.csv};
\addplot[RPE-learn] table[x=Size,y=RPE-learn,col sep=comma] {distance_size_data/size_512_p.csv};
\addplot[2D-ALiBi] table[x=Size,y=2D-ALiBi,col sep=comma] {distance_size_data/size_512_p.csv};
\addplot[2D-RoPE] table[x=Size,y=2D-RoPE,col sep=comma] {distance_size_data/size_512_p.csv};
\addplot[LH-180] table[x=Size,y=LH-180,col sep=comma] {distance_size_data/size_512_p.csv};
\addplot[LH-90] table[x=Size,y=LH-90,col sep=comma] {distance_size_data/size_512_p.csv};
\addplot[LH-45] table[x=Size,y=LH-45,col sep=comma] {distance_size_data/size_512_p.csv};
\hspace{-15pt}
\nextgroupplot[title={$\Delta$ Acc.  $224^2\rightarrow768^2$},ymin=-25,ymax=0,
\addplot[1D-learn] table[x=Size,y=1D-learn,col sep=comma] {distance_size_data/size_768_p.csv};
\addplot[2D-sincos] table[x=Size,y=2D-sincos,col sep=comma] {distance_size_data/size_768_p.csv};
\addplot[Factorized] table[x=Size,y=Factorized,col sep=comma] {distance_size_data/size_768_p.csv};
\addplot[Fourier] table[x=Size,y=Fourier,col sep=comma] {distance_size_data/size_768_p.csv};
\addplot[RPE-learn] table[x=Size,y=RPE-learn,col sep=comma] {distance_size_data/size_768_p.csv};
\addplot[2D-ALiBi] table[x=Size,y=2D-ALiBi,col sep=comma] {distance_size_data/size_768_p.csv};
\addplot[2D-RoPE] table[x=Size,y=2D-RoPE,col sep=comma] {distance_size_data/size_768_p.csv};
\addplot[LH-180] table[x=Size,y=LH-180,col sep=comma] {distance_size_data/size_768_p.csv};
\addplot[LH-90] table[x=Size,y=LH-90,col sep=comma] {distance_size_data/size_768_p.csv};
\addplot[LH-45] table[x=Size,y=LH-45,col sep=comma] {distance_size_data/size_768_p.csv};
\hspace{-15pt}
\nextgroupplot[title={$\Delta$ Acc.  $224^2\rightarrow1024^2$},ymin=-40,ymax=-5,
\addplot[1D-learn] table[x=Size,y=1D-learn,col sep=comma] {distance_size_data/size_1024_p.csv};
\addplot[2D-sincos] table[x=Size,y=2D-sincos,col sep=comma] {distance_size_data/size_1024_p.csv};
\addplot[Factorized] table[x=Size,y=Factorized,col sep=comma] {distance_size_data/size_1024_p.csv};
\addplot[Fourier] table[x=Size,y=Fourier,col sep=comma] {distance_size_data/size_1024_p.csv};
\addplot[RPE-learn] table[x=Size,y=RPE-learn,col sep=comma] {distance_size_data/size_1024_p.csv};
\addplot[2D-ALiBi] table[x=Size,y=2D-ALiBi,col sep=comma] {distance_size_data/size_1024_p.csv};
\addplot[2D-RoPE] table[x=Size,y=2D-RoPE,col sep=comma] {distance_size_data/size_1024_p.csv};
\addplot[LH-180] table[x=Size,y=LH-180,col sep=comma] {distance_size_data/size_1024_p.csv};
\addplot[LH-90] table[x=Size,y=LH-90,col sep=comma] {distance_size_data/size_1024_p.csv};
\addplot[LH-45] table[x=Size,y=LH-45,col sep=comma] {distance_size_data/size_1024_p.csv};
\hspace{-15pt}
\vspace{50pt}
\end{groupplot}

\node at (8.3,-0.6) {\textbf{Object Size (Quintile)}};

\end{tikzpicture}
\begin{tikzpicture}[every node/.style={font=\scriptsize}]
\begin{customlegend}[legend columns=5,legend style={draw=none,column sep=3pt},legend entries={LH-180 (ours),LH-90 (ours),LH-45 (ours),Factorized \cite{navit},2D-RoPE \cite{rope},2D-sincos \cite{2d_sincos},1D-learn \cite{vit},RPE-learn \cite{beit},Fourier \cite{fourier},2D-ALiBi \cite{2d_alibi}}]
\addlegendimage{LH-180}
\addlegendimage{LH-90}
\addlegendimage{LH-45}
\addlegendimage{Factorized}
\addlegendimage{2D-RoPE}
\addlegendimage{2D-sincos}
\addlegendimage{1D-learn}
\addlegendimage{RPE-learn}
\addlegendimage{Fourier}
\addlegendimage{2D-ALiBi}
\end{customlegend}
\end{tikzpicture}
\caption{The effect of object size on accuracy gains or losses due to extrapolation. Object size is measured using annotations from Kaggle's ImageNet Object Localization Challenge \cite{imagenet-loc}.}
\vspace{-10pt}
\label{fig:object_size}
\end{figure}

LookHere significantly improves extrapolation ability (Figure \ref{fig:main_figure}). Our smallest FOV variant (LH-$45$) sees improving relative performance as resolution increases. LH-$45$ outperforms $2$D-ALiBi, which is equivalent to LookHere without our $2$D directional masks, by $9.5\%$ on Val at $1024^2$ px. These two results demonstrate the extrapolation benefits of restricting attention to fixed FOVs. LH-$45$ gains $1.3\%$ on Val when extrapolating from $224^2$ to $384^2$ px; this is the largest gain we find in the literature, including our extensive benchmarking of SoTA models in Appendix \ref{appn:extrap-results}. Interestingly, smaller objects benefit most from extrapolation (Figure \ref{fig:object_size}), which are distributed over more patches at test time. We believe this effect also explains the $6-8\%$ that LookHere models gain when extrapolating on ImageNet-A from $224^2$ to $448^2$ px; by inspection, ImageNet-A seems to have small objects, and other work found zooming-in on center-cropped ImageNet-A images improves performance \cite{imagenet-hard}. Finally, all LookHere variants outperform other methods on ImageNet-HR, indicating better handling of interpolated pixels generated when upsampling lower-resolution imagery is \textit{not} the reason why LookHere extrapolates better.

\usetikzlibrary{positioning}
\begin{wrapfigure}[17]{r}{0.5\linewidth}%
\vspace{-15pt}
\centering
\begin{tikzpicture}
\hspace{-10pt}
% Row 1
\node[anchor=south west, inner sep=0] (image1) at (0,0)

\node[anchor=south west, inner sep=0] (image2) at (image1.south east)

\node[anchor=south west, inner sep=0] (image3) at (image2.south east)

\node[anchor=south west, inner sep=0] (image4) at (image3.south east)

% Row 2
\node[anchor=north west, inner sep=0] (image9) at (image1.south west)

\node[anchor=north west, inner sep=0] (image10) at (image9.north east)

\node[anchor=north west, inner sep=0] (image11) at (image10.north east)

\node[anchor=north west, inner sep=0] (image12) at (image11.north east)

% Row 3
\node[anchor=north west, inner sep=0] (image17) at (image9.south west)

\node[anchor=north west, inner sep=0] (image18) at (image17.north east)

\node[anchor=north west, inner sep=0] (image19) at (image18.north east)

\node[anchor=north west, inner sep=0] (image20) at (image19.north east)

\node[left=0mm of image1] {\tiny RPE-learn};
\node[left=0mm of image9] {\tiny 2D-ALiBi};
\node[left=0mm of image17] {\tiny LH-90};

\node[above=-0.5mm of image1] {$224^2$};
\node[above=-0.5mm of image2] {$512^2$};
\node[above=-0.5mm of image3] {$768^2$};
\node[above=-0.5mm of image4] {$1024^2$};

\end{tikzpicture}
\caption{Attention maps of three attention heads across four resolutions, where the query is in the center. We use the colormap:  \includegraphics[width=2cm]{cmrmap.png}}
\label{fig:space_confuse}
% \vspace{-25pt}
\end{wrapfigure}
Reducing the distribution shift faced by attention heads during extrapolation is our best explanation for LookHere's large relative improvement. Figure \ref{fig:space_confuse} shows attention maps that are ``unflattened'' to visualize the image regions to which heads attend, averaged over the same $5$k images. We show one head per model that exhibits similar behavior at a $224^2$ resolution. Models leveraging RPE-learn and $2$D-ALiBi learn variants of an algorithm that retrieve information from above the query; however, both models retrieve information elsewhere in the image when extrapolating. LookHere hard-codes this type of algorithm, which it continues to execute when extrapolating. In Appendix \ref{appn:measure} we find more examples of interesting attention head behaviour.

Extrapolation affects different datasets differently; it also affects different classes differently. For example, when extrapolating, all models underpredict certain classes (\texttt{bakery}, \texttt{church}, and \texttt{tights}) and overpredict other classes (\texttt{mobile home}, \texttt{threshing machine}, and \texttt{sports car}). This investigation is inspired by the class-level effects of data augmentation \cite{class-level}. In Appendix \ref{appn:classlevel} we find more class-level effects of extrapolation.

\section{Closing}
\textbf{Limitations}. The primary limitation of LookHere is it requires hand-designed directional masks and distance penalties. However, our extensive ablations demonstrate that LookHere is robust to the choice of directional masks and distance penalties. The primary limitation of our experiments is we do not scale ViTs to giant sizes. Instead, we select the most common size, the ViT-B/$16$, and focus our computational resources on a controlled experiment --- that extensively and fairly tunes the appropriate baselines for plain ViTs; this allows us to make confident conclusions based on our thorough experiments.

\textbf{Conclusion.} LookHere position encoding significantly improves the ability of plain ViTs to make inferences when provided a greater number of patches than seen during training. We thoroughly demonstrate that LookHere outperforms other methods with and without extrapolation on standard image benchmarks and our high-resolution ImageNet test set called ImageNet-HR. We provide new insights into ViT extrapolation by showing object-size, class-level, and dataset-level effects. We believe LookHere will help the vision community transform higher-resolution into higher accuracy.

\textbf{Future Work.} We are excited to realize the computational gains that LookHere makes available via sparse attention kernels, as well as bring LookHere to video and $3$D point-cloud applications.
\newpage

\clearpage

\bibliographystyle{unsrt}
\bibliography{references.bib}

\clearpage

\appendix

\section{Appendix / supplemental material}

\subsection{ImageNet-HR}\label{appn:imgnet-hr}

\usetikzlibrary{positioning}
\begin{figure}[htbp]
\centering
\hspace*{-1.4cm}
\begin{tikzpicture}
\node (image1) at (0, 0){\includegraphics[width=0.195\textwidth]{HR_images/2085_417.png}};
\node at (2.8, 0) {\includegraphics[width=0.195\textwidth]{HR_images/2086_417.png}};
\node at (5.6, 0) {\includegraphics[width=0.195\textwidth]{HR_images/2087_417.png}};
\node at (8.4, 0) {\includegraphics[width=0.195\textwidth]{HR_images/2088_417.png}};
\node at (11.2, 0) {\includegraphics[width=0.195\textwidth]{HR_images/2089_417.png}};

% Row 2
\node (image2) at (0, 2.8) {\includegraphics[width=0.195\textwidth]{HR_images/3960_792.png}};
\node at (2.8, 2.8) {\includegraphics[width=0.195\textwidth]{HR_images/3961_792.png}};
\node at (5.6, 2.8) {\includegraphics[width=0.195\textwidth]{HR_images/3962_792.png}};
\node at (8.4, 2.8) {\includegraphics[width=0.195\textwidth]{HR_images/3963_792.png}};
\node at (11.2, 2.8) {\includegraphics[width=0.195\textwidth]{HR_images/3964_792.png}};

% % Row 3
% \node (image3) at (0, 5.6) {\includegraphics[width=0.195\textwidth]{HR_images/4640_928.png}};
% \node at (2.8, 5.6) {\includegraphics[width=0.195\textwidth]{HR_images/4641_928.png}};
% \node at (5.6, 5.6) {\includegraphics[width=0.195\textwidth]{HR_images/4642_928.png}};
% \node at (8.4, 5.6) {\includegraphics[width=0.195\textwidth]{HR_images/4643_928.png}};
% \node at (11.2, 5.6) {\includegraphics[width=0.195\textwidth]{HR_images/4644_928.png}};

% Row 4
\node (image4) at (0, 5.6) {\includegraphics[width=0.195\textwidth]{HR_images/480_96.png}};
\node at (2.8, 5.6) {\includegraphics[width=0.195\textwidth]{HR_images/481_96.png}};
\node at (5.6, 5.6) {\includegraphics[width=0.195\textwidth]{HR_images/482_96.png}};
\node at (8.4, 5.6) {\includegraphics[width=0.195\textwidth]{HR_images/483_96.png}};
\node at (11.2, 5.6) {\includegraphics[width=0.195\textwidth]{HR_images/484_96.png}};

\node[left=0mm of image1] {\texttt{balloon}};
\node[left=0mm of image2] {\texttt{shovel}};
% \node[left=0mm of image3] {\texttt{ice cream}};
\node[left=0mm of image4] {\texttt{toucan}};
\end{tikzpicture}
\caption{All images of three classes from ImageNet-HR.}
\label{fig:five_images}
\end{figure}

\textbf{ImageNet-HR Specifics.} We invest considerable resources to ensure ImageNet-HR's quality with two priorities. \circlednum{1} Annotation accuracy --- we only include images for which we are confident of their label; we achieve this by: \circlednum{a} $5$ rounds of quality control consisting of manually reviewing all cases where models disagreed with our annotations, using a SoTA model (\verb+eva02_large_patch14_448.mim_m38m_ft_in22k_in1k+ from timm \cite{timm}) and a weaker model that disagrees more often (\verb+tiny_vit_5m_224.dist_in22k_ft_in1k+ from timm \cite{timm}), \circlednum{b} consulting someone with wildlife expertise to limit the annotation errors made by other test sets \cite{bugs_in_data}, \circlednum{c} using multiple labels where necessary, for example, combining the ``sunglass'' and ``sunglasses'' classes, and labeling a ``tusker'' as also an ``Asian elephant,'' if the image of the tusked animal is an Asian elephant. \circlednum{2} Image diversity ---  when collecting images, we try to maximize the diversity of images belonging to a class. Models achieve high accuracy on ImageNet-HR, likely due to less label ambiguity than other ImageNet test sets. Finally, we manually crop all images to $1024^2$ px, resulting in the first natively high-resolution ImageNet test set.

We collect the vast majority of images from \href{https://www.flickr.com/}{flickr} and \href{https://unsplash.com/}{Unsplash}. Unsplash images ``are made to be used freely'' for commercial and non-commercial uses. flickr images were selected from the ``All creative commons'' license option. However, for some classes, we could not find enough open-access high-resolution images like ``oil filter'' or ``hand or block plane,'' so we used Google search to find more. We estimate that around $50$ of $5$k images were not collected on flickr or Unsplash. Nine images were taken by an author or his family, with consent of all parties involved.

\newpage
\subsection{Extrapolation Results}\label{appn:extrap-results}
\begin{table}[!th]\centering\small

\caption{Extrapolation results for ViT-B models trained on ImageNet for $150$ epochs; trained at $224^2$ and tested at various resolutions.}
\label{tab:extrap_table}
\renewcommand{\arraystretch}{0.4}
\setlength{\tabcolsep}{3pt}
\begin{tabular}{
\toprule
&& \multicolumn{2}{c}{Val \cite{imagenet}} & \multicolumn{2}{c}{ReaL \cite{real}} & \multicolumn{2}{c}{v$2$ \cite{v2}} & \multicolumn{2}{c}{-A \cite{a}} & \multicolumn{2}{c}{-R \cite{r}} & \multicolumn{2}{c}{-HR (ours)} \\
\cmidrule(lr){3-4}
\cmidrule(lr){5-6}
\cmidrule(lr){7-8}
\cmidrule(lr){9-10}
\cmidrule(lr){11-12}
\cmidrule(lr){13-14}
Method & Res            &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} \\
\midrule
$1$D-learn  & $320^2$ &79.89&	94.59&	85.17&	96.26&	68.78&	87.88&	12.91&	34.36&	26.73&	40.04&	88.76&	96.94\\
$2$D-sincos  & $320^2$ & 79.48&	94.50&	84.76&	96.34&	68.23&	87.69&	12.52&	33.59&	26.30&	39.84&	87.94&	96.76\\
Factorized  & $320^2$ & 79.73&	94.70&	85.08&	96.37&	68.71&	87.83&	11.44&	32.01&	25.89&	39.33&	87.78&	97.08\\
Fourier  & $320^2$ & 79.80&	94.58&	85.17&	96.45&	68.43&	87.95&	12.72&	34.16&	26.33&	39.64&	88.40&	97.08\\
RPE-learn  & $320^2$ & 79.94&	94.73&	85.50&	96.63&	68.56&	87.83&	11.17&	30.51&	24.21&	36.71&	88.12&	97.26\\
$2$D-ALiBi  & $320^2$ & 80.44&	95.10&	85.66&	96.66&	69.25&	88.65&	13.33&	35.00&	26.54&	40.47&	88.62&	97.36\\
$2$D-RoPE  & $320^2$ & 81.40&	95.36&	86.43&	96.77&	70.38&	88.94&	16.08&	39.52&	29.76&	44.21&	89.38&	97.38\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$180$}  & $320^2$ & 82.65&	95.88&	87.63&	97.28&	72.03&	89.78&	18.04&	40.28&	30.26&	43.01&	90.60&	97.98\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & $320^2$ & 82.22&	95.66&	87.44&	97.13&	72.15&	89.62&	17.92&	40.28&	30.33&	43.34&	90.36&	97.94\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$45$}  & $320^2$ & 82.45&	95.84&	87.45&	97.17&	71.99&	89.92&	17.99&	40.96&	30.53&	43.31&	90.30&	97.66\\
\midrule
$1$D-learn  & $384^2$ & 79.02&	94.14&	84.42&	95.99&	67.47&	87.13&	11.97&	32.23&	23.66&	36.93&	87.36&	96.74\\
$2$D-sincos  & $384^2$ & 78.56&	94.06&	83.95&	96.02&	66.96&	87.16&	11.87&	31.31&	23.53&	36.63&	87.02&	96.38\\
Factorized  & $384^2$ & 78.56&	94.06&	84.01&	95.94&	66.62&	87.21&	10.41&	29.72&	23.03&	35.69&	86.90&	97.00\\
Fourier  & $384^2$ & 78.85&	94.07&	84.30&	96.07&	67.43&	87.33&	12.63&	32.59&	23.31&	35.95&	87.48&	96.84\\
RPE-learn  & $384^2$ & 77.96&	93.64&	83.72&	95.85&	66.54&	86.51&	9.51&	26.85&	19.41&	30.38&	86.38&	96.44\\
$2$D-ALiBi  & $384^2$ & 80.38&	94.93&	85.49&	96.51&	69.21&	88.34&	14.99&	37.88&	24.45&	38.02&	88.30&	97.24\\
$2$D-RoPE  & $384^2$ & 81.16&	95.27&	86.20&	96.71&	70.27&	88.91&	17.60&	41.16&	27.48&	41.05&	88.70&	97.44\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$180$}  & $384^2$ & 82.38&	95.79&	87.35&	97.25&	72.15&	89.67&	19.09&	42.32&	27.58&	39.97&	89.94&	97.84\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & $384^2$ & 82.08&	95.70&	87.26&	97.18&	71.50&	89.74&	19.93&	42.68&	27.99&	40.58&	90.10&	97.66\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$45$}  & $384^2$ & 82.38&	95.85&	87.32&	97.15&	71.98&	90.00&	19.73&	43.29&	28.38&	40.95&	90.16&	97.76\\
\midrule
$1$D-learn  & $448^2$ & 77.52&	93.45&	83.06&	95.47&	65.77&	86.10&	10.44&	28.79&	20.93&	33.86&	86.04&	96.36\\
$2$D-sincos  & $448^2$ & 77.12&	93.54&	82.80&	95.52&	65.11&	85.62&	10.61&	29.32&	20.84&	33.66&	85.78&	96.24\\
Factorized  & $448^2$ & 76.98&	93.33&	82.63&	95.32&	65.06&	85.70&	9.67&	26.63&	20.21&	32.50&	86.00&	96.46\\
Fourier  & $448^2$ & 77.47&	93.46&	83.05&	95.51&	65.57&	86.05&	11.37&	29.59&	20.57&	32.76&	86.18&	96.46\\
RPE-learn  & $448^2$ & 75.40&	92.41&	81.45&	94.87&	63.40&	84.29&	8.11&	23.79&	15.31&	25.31&	84.18&	95.44\\
$2$D-ALiBi  & $448^2$ & 79.63&	94.61&	84.74&	96.32&	68.66&	87.86&	15.13&	37.79&	22.25&	35.13&	87.32&	96.96\\
$2$D-RoPE  & $448^2$ & 80.47&	94.92&	85.67&	96.47&	68.99&	88.27&	17.84&	41.92&	24.72&	37.36&	88.02&	97.20\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$180$}  & $448^2$ & 81.86&	95.56&	87.05&	97.16&	71.05&	89.62&	19.84&	43.35&	24.90&	36.83&	88.80&	97.74\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & $448^2$ & 81.91&	95.54&	87.00&	97.08&	71.39&	89.49&	20.57&	43.39&	25.71&	37.70&	89.20&	97.52\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$45$}  & $448^2$ & 82.19&	95.67&	87.02&	97.08&	71.93&	89.58&	20.77&	43.76&	25.97&	38.35&	89.62&	97.66\\
\midrule
$1$D-learn  & $512^2$ & 75.89&	92.54&	81.41&	94.79&	63.31&	84.37&	8.95&	25.65&	18.65&	31.03&	84.44&	96.02\\
$2$D-sincos  & $512^2$ & 75.43&	92.57&	81.26&	94.87&	62.74&	84.31&	9.16&	25.27&	18.41&	30.80&	84.18&	95.66\\
Factorized  & $512^2$ & 74.97&	92.22&	80.65&	94.47&	62.29&	83.56&	8.05&	22.81&	17.78&	29.80&	84.10&	95.76\\
Fourier  & $512^2$ & 75.68&	92.56&	81.33&	94.78&	63.14&	83.90&	9.96&	26.20&	18.11&	30.06&	84.76&	95.94\\
RPE-learn  & $512^2$ & 72.59&	90.74&	78.81&	93.52&	59.35&	81.83&	6.45&	21.00&	12.17&	21.45&	81.64&	94.36\\
$2$D-ALiBi  & $512^2$ & 78.86&	94.18&	84.02&	95.96&	67.28&	87.05&	14.23&	35.83&	20.23&	32.23&	86.42&	96.56\\
$2$D-RoPE  & $512^2$ & 79.23&	94.39&	84.61&	96.15&	67.44&	87.22&	16.05&	38.53&	21.09&	33.74&	86.90&	96.90\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$180$}  & $512^2$ & 81.11&	95.26&	86.46&	96.87&	69.96&	88.97&	19.49&	41.81&	22.64&	33.98&	88.08&	97.48\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & $512^2$ & 81.19&	95.16&	86.38&	96.84&	70.35&	88.92&	20.05&	42.09&	23.44&	35.18&	88.62&	97.16\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$45$}  & $512^2$ & 81.62&	95.44&	86.57&	96.91&	71.09&	88.80&	20.55&	43.21&	23.87&	35.76&	89.04&	97.40\\
\midrule
$1$D-learn  & $768^2$ & 65.95&	87.11&	71.75&	90.33&	51.11&	75.27&	3.79&	13.16&	12.13&	22.47&	75.76&	92.64\\
$2$D-sincos  & $768^2$ & 65.48&	86.90&	71.19&	90.19&	50.64&	75.36&	3.84&	12.77&	11.45&	21.22&	75.46&	92.06\\
Factorized  & $768^2$ & 63.71&	85.36&	69.15&	88.81&	48.58&	73.17&	3.04&	10.95&	10.70&	20.32&	74.64&	92.24\\
Fourier  & $768^2$ & 65.97&	86.92&	71.56&	90.01&	51.25&	75.24&	4.05&	13.45&	11.53&	21.38&	76.04&	92.54\\
RPE-learn  & $768^2$ & 57.16&	79.87&	63.00&	83.99&	41.56&	65.96&	2.55&	8.91&	4.83&	9.98&	66.68&	86.32\\
$2$D-ALiBi  & $768^2$ & 72.97&	90.64&	78.13&	93.26&	59.19&	81.53&	8.48&	24.00&	12.83&	22.54&	79.96&	93.76\\
$2$D-RoPE  & $768^2$ & 71.28&	89.93&	77.03&	92.54&	56.70&	79.93&	7.53&	22.20&	12.00&	21.23&	79.14&	93.68\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$180$}  & $768^2$ & 76.59&	92.92&	82.17&	95.09&	63.88&	84.63&	12.52&	29.41&	15.56&	25.88&	83.56&	95.68\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & $768^2$ & 77.12&	93.38&	82.68&	95.46&	64.49&	85.23&	12.89&	30.44&	17.52&	28.68&	84.90&	96.08\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$45$}  & $768^2$ & 78.13&	93.76&	83.67&	95.68&	66.51&	86.17&	14.21&	31.96&	18.14&	28.64&	85.30&	96.10\\
\midrule
$1$D-learn  & $1024^2$ & 55.67&	80.00&	61.00&	83.85&	40.97&	65.86&	1.95&	7.77&	8.46&	17.28&	65.14&	86.02\\
$2$D-sincos  & $1024^2$ & 53.71&	78.36&	58.91&	82.32&	39.57&	64.82&	1.48&	6.04&	7.08&	14.93&	64.62&	86.16\\
Factorized  & $1024^2$ & 50.46&	75.08&	55.22&	79.17&	37.41&	62.22&	1.33&	5.43&	7.00&	14.51&	63.86&	85.08\\
Fourier  & $1024^2$ & 54.58&	78.17&	59.80&	82.14&	39.22&	64.61&	1.87&	7.01&	7.65&	15.49&	64.66&	86.46\\
RPE-learn  & $1024^2$ & 36.80&	60.77&	41.10&	65.25&	24.37&	46.37&	0.99&	3.88&	1.85&	4.81&	48.40&	72.34\\
$2$D-ALiBi  & $1024^2$ & 63.62&	84.36&	68.80&	87.80&	49.88&	73.57&	4.51&	14.48&	7.78&	15.24&	71.56&	88.74\\
$2$D-RoPE  & $1024^2$ & 51.41&	75.05&	56.71&	79.27&	37.18&	60.56&	2.12&	8.04&	4.00&	9.06&	60.30&	82.64\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$180$}  & $1024^2$ & 69.24&	88.81&	75.22&	91.84&	55.84&	78.89&	6.63&	18.92&	11.23&	20.18&	76.14&	92.44\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & $1024^2$ & 71.58&	90.19&	77.48&	92.99&	58.30&	80.83&	8.13&	21.29&	12.61&	22.50&	79.68&	93.34\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$45$}  & $1024^2$ & 73.15&	91.09&	79.01&	93.63&	60.26&	82.38&	8.35&	22.15&	14.11&	23.78&	80.48&	93.78\\
\bottomrule
\end{tabular}
\end{table}

\newpage
\subsubsection{Other Models}\label{appn:extrap-other}
\begin{table}[!th]\centering\small
\caption{Top-1 acc. ($\%$) on Val \cite{imagenet} for models outside our controlled experiment, using the timm library \cite{timm}.}
\label{tab:others_val}
\setlength{\tabcolsep}{1pt}
\begin{tabular}{
\toprule
Name            &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{$224^2$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{$320^2$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{$384^2$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{$448^2$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{$512^2$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{$768^2$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{$1024^2$} \\
\midrule
\verb+beitv2_large_patch16_224.in1k_ft_in22k_in1k+\cite{beitv2}  &  87.97&	87.73&	80.76&	60.80&	40.48&	10.63 & 5.19\\
\verb+caformer_b36.sail_in1k+\cite{caformer_convformer}& 85.28&	85.61&	84.93&	84.08&	83.08&	77.90&	70.35\\
\verb+caformer_b36.sail_in22k_ft_in1k+\cite{caformer_convformer}  &  87.24&	87.29&	86.34&	84.78&	82.96&	73.55&	62.60\\
\verb+convformer_b36.sail_in1k+\cite{caformer_convformer} & 84.59&	85.06&	83.98&	82.34&	79.48&	58.24&	37.54\\
\verb+convformer_s18.sail_in1k+\cite{caformer_convformer}  &  82.89&	83.34&	81.72&	78.86&	73.36&	39.49&	17.57\\
\verb+eva_giant_patch14_224.clip_ft_in1k+\cite{eva}   & 88.75&	88.86&	88.50&	87.83&	87.22&	83.37&	78.31\\
iRPE (our implementation)\cite{irpe}   & 80.53&	81.59&	81.47&	80.77&	79.94&	72.86&	60.30\\
\verb+swin_base_patch4_window7_224.ms_in22k_ft_in1k+\cite{swin}   & 84.40&	84.80&	84.31&	83.77&	82.90&	78.56&	70.58\\
\verb+swin_s3_base_224.ms_in1k+\cite{s3}   & 83.86&	82.61&	81.34&	80.39&	79.30&	73.54&	63.78\\
\verb+swin_tiny_patch4_window7_224+\cite{swin}   & 80.85&	80.92&	79.96&	79.09&	78.33&	72.24&	61.06\\
\verb+twins_pcpvt_base.in1k+\cite{twins}   & 82.54&	83.20&	82.27&	81.06&	79.68&	72.22&	61.24\\
\verb+twins_pcpvt_small.in1k+\cite{twins} & 80.94&	81.67&	80.92&	79.76&	78.46&	70.56&	58.74\\
\verb+twins_svt_large.in1k+\cite{twins}   & 83.38&	83.44&	82.64&	82.03&	80.99&	76.49&	69.01\\
\verb+vit_base_patch16_clip_224.laion2b_ft_in12k_in1k+\cite{clip}   & 85.79&	85.84&	85.03&	84.24&	83.25&	76.35&	66.60\\
\verb+vit_base_patch16_rope_reg1_gap_256.sbb_in1k+\cite{timm}   & 81.26&	82.33&	81.88&	80.89&	79.66&	72.66&	63.25\\
\verb+vit_large_patch14_clip_224.openai_ft_in12k_in1k+\cite{clip}   & 87.93&	88.08&	87.57&	87.02&	86.18&	81.51&	75.51\\
\verb+vit_mediumd_patch16_rope_reg1_gap_256.sbb_in1k+\cite{timm}   & 81.55&	82.67&	82.23&	81.43&	80.08&	73.24&	64.69\\
\verb+vit_small_r26_s32_224+\cite{vit}   & 81.38&	82.89&	82.46&	81.75&	80.32&	72.30&	61.38\\
\verb+xcit_medium_24_p8_224.fb_dist_in1k+\cite{xcit}   & 84.86&	85.36&	84.97&	84.41&	83.75&	79.58&	72.94\\
\verb+xcit_small_12_p16_224.fb_in1k+\cite{xcit}   & 81.68&	82.65&	82.10&	81.51&	80.43&	74.48&	65.46\\
\bottomrule
\end{tabular}
\end{table}

\begin{table}[!th]\centering\small

\caption{Top-1 acc. ($\%$) on -HR (ours) for models outside our controlled experiment, using the timm library \cite{timm}.}
\label{tab:others_hr}
\setlength{\tabcolsep}{1pt}
\begin{tabular}{
\toprule
Name            &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{$224^2$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{$320^2$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{$384^2$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{$448^2$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{$512^2$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{$768^2$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{$1024^2$} \\
\midrule
\verb+beitv2_large_patch16_224.in1k_ft_in22k_in1k+\cite{beitv2}  &  95.16&	95.24&	90.36&	73.72&	52.60&	14.32&	7.40\\
\verb+caformer_b36.sail_in1k+\cite{caformer_convformer} &93.06&	93.08&	92.84&	92.10&	90.88&	85.68&	79.30\\
\verb+caformer_b36.sail_in22k_ft_in1k+\cite{caformer_convformer}  &  94.40&	94.56&	94.02&	93.00&	91.34&	82.16&	70.94\\
\verb+convformer_b36.sail_in1k+\cite{caformer_convformer} &92.44&	92.26&	90.94&	88.94&	85.40&	70.04&	59.80\\
\verb+convformer_s18.sail_in1k+\cite{caformer_convformer}  &  90.98&	90.84&	88.40&	83.30&	77.24&	53.18&	41.02\\
\verb+eva_giant_patch14_224.clip_ft_in1k+\cite{eva}   & 95.96&	95.86&	95.70&	95.58&	95.36&	92.94&	89.24\\
iRPE (our implementation)\cite{irpe}   & 89.10&	89.76&	89.56&	88.88&	88.12&	82.40 & 72.60\\
\verb+swin_base_patch4_window7_224.ms_in22k_ft_in1k+\cite{swin}   & 91.82&	92.20&	91.18&	90.24&	90.06&	85.22&	79.26\\
\verb+swin_s3_base_224.ms_in1k+\cite{s3}   & 91.78&	90.22&	88.52&	86.82&	85.72&	78.00&	68.46\\
\verb+swin_tiny_patch4_window7_224+\cite{swin}   & 89.06&	89.10&	87.56&	86.76&	85.46&	77.58&	68.54\\
\verb+twins_pcpvt_base.in1k+\cite{twins}   & 90.54&	90.92&	90.30&	89.24&	87.60&	79.50&	69.74\\
\verb+twins_pcpvt_small.in1k+\cite{twins} &89.62&	89.40&	88.76&	87.14&	85.62&	77.40&	67.30\\
\verb+twins_svt_large.in1k+\cite{twins}   & 91.46&	91.24&	90.34&	89.36&	87.82&	82.42&	76.22\\
\verb+vit_base_patch16_clip_224.laion2b_ft_in12k_in1k+\cite{clip}   & 93.22&	93.38&	93.24&	92.68&	92.02&	88.14&	81.92\\
\verb+vit_base_patch16_rope_reg1_gap_256.sbb_in1k+\cite{timm}   & 89.36&	90.84&	90.14&	89.22&	88.22&	83.36&	76.54\\
\verb+vit_large_patch14_clip_224.openai_ft_in12k_in1k+\cite{clip}   & 94.62&	94.78&	94.68&	94.62&	94.10&	91.10&	86.64\\
\verb+vit_mediumd_patch16_rope_reg1_gap_256.sbb_in1k+\cite{timm}   & 89.42&	90.56&	90.38&	89.74&	89.26&	83.90&	77.48\\
\verb+vit_small_r26_s32_224+\cite{vit}   & 89.76&	90.52&	90.30&	89.56&	88.80&	82.06&	72.12\\
\verb+xcit_medium_24_p8_224.fb_dist_in1k+\cite{xcit}   &92.54&	92.94&	92.34&	91.94&	91.20&	87.70&	81.80\\
\verb+xcit_small_12_p16_224.fb_in1k+\cite{xcit}   & 89.54&	90.14&	89.88&	89.02&	88.22&	82.82&	74.86\\
\bottomrule
\end{tabular}
\end{table}
\clearpage

\subsection{LookHere Bias Matrices}\label{appn:lookhere-bias-maps}
\begin{figure}[ht]\centering
\includegraphics[scale=0.45]{daniel_pngs/LH-180_bias_maps.png}
\caption{LH-180 bias matrices for query patch (11,8), grid size of 14x14.}
\label{fig:lh180-bmaps}
\end{figure}
\begin{figure}\centering
\includegraphics[scale=0.45]{daniel_pngs/LH-90_bias_maps.png}
\caption{LH-90 bias matrices for query patch (11,8), grid size of 14x14.}
\label{fig:lh90-bmaps}
\end{figure}
\begin{figure}\centering
\includegraphics[scale=0.45]{daniel_pngs/LH-45_bias_maps.png}
\caption{LH-45 bias matrices for query patch (11,8), grid size of 14x14.}
\label{fig:lh45-bmaps}
\end{figure}
\clearpage

\subsection{Experimental Details}
\subsubsection{Training ViTs}\label{appn:training_vits}
\textbf{Recipe.} Our training recipe that is consistent across configurations:
\begin{itemize}
\item AdamW \cite{adamw} --- using the default PyTorch implementation that does not fully decouple learning rate and weight decay
\item Binary cross-entropy loss --- summing along the class dimension, averaging along the batch dimension
\item Linear warm-up for $10\%$ of steps and cool-down using a cosine decay schedule to a zero learning rate
\item Batch size of $2048$
\item Mixup \cite{mixup} $\alpha=0.8$, cutmix \cite{cutmix} $\alpha=1$
\item CLS token with an MLP classifying head --- final linear layer weights are initialized to $0$ and biases to $-6.9$ (so all class probabilities start at $\frac{1}{1000}$)
\item layer drop rate of $0.1$ and MLP dropout of $0$
\item Train for $150$ epochs on the first $99\%$ of ImageNet-1k --- using Huggingface's datasets library, i.e., \verb+load_dataset("imagenet-1k", split="train[:99%]")+
\item Choose checkpoint according to the best minival top-$1$ accuracy (run after each epoch), where minival is the last $1\%$ of the ImageNet-1k training set, i.e.,  \verb+load_dataset("imagenet-1k", split="train[99%:]")+
\end{itemize}

\subsubsection{Compute}\label{appn:compute}
Training takes around $3$ days on an RTX $4090$ GPU. Thus, all $80$ training runs take around $240$ GPU-days. We spend another $54$ GPU-days on $18$ ablations. Ablations and our iRPE run always use our best training recipe, which is 3-Augment \cite{revenge} data augmentation, $3\cdot10^{-3}$ learning rate, and $0.05$ weight decay. iRPE \cite{irpe} takes around $7$ days on an RTX $4090$ GPU, even with the official custom CUDA kernel. As a result, we exclude it from our apples-to-apples comparisons.

\subsubsection{High-resolution finetuning}\label{appn:high_resolution_finetuning}
Following DEiT III's finetuning recipe \cite{revenge}, we increase the drop rate to $0.2$ and the weight decay to $0.1$, and fix the learning rate to $10^{-5}$ with a $512$ batch size.

\subsubsection{Extrapolation Tuning}\label{appn:extrapolate_tuning}
For 2D-ALiBi, 2D-RoPE, and LookHere models, we tune a single parameter at the target resolution on minival (Table \ref{tab:tuning_slope}). LookHere models benefit less from tuning than 2D-ALiBi and 2D-RoPE models. For example at a $512^2$ resolution, the difference in top-$1$ accuracy on minival when using the tuned parameter versus the default value is $2.1\%$ for 2D-ALiBi, $1.3\%$ for 2D-RoPE, and $0.15\%$ for LH-$45$. Thus, LookHere does not require tuning its global slope value to effectively extrapolate.

\begin{table}[!th]\centering\small
\caption{Tuned Parameter Values}
\label{tab:tuning_slope}
% \setlength{\tabcolsep}{1pt}
\begin{tabular}{
\toprule
Name            & Tuning Parameter &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{$224^2$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{$320^2$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{$384^2$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{$448^2$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{$512^2$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{$768^2$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{$1024^2$} \\
\midrule
2D-ALiBi  &  $s_g$ &	1.0 &	1.4&	1.4&	1.4&	1.4&	1.5 & 1.6\\
2D-RoPE & base frequency & 100 &	160&	190&	250&	700&	1250&	1250\\
LookHere  & $s_g$ &  1.0&	1.00&	0.95&	0.95&	0.95&	0.75&	0.6\\
\bottomrule
\end{tabular}
\end{table}

\subsubsection{Segmentation}\label{appn:segmentation}
For both linear probing and full finetuning we use a linear decoder. The linear decoder consists of a linear layer applied to the frozen patch representations which is then upsampled to the original image size.  Similar to \cite{oquab2023dinov2} we add a BatchNorm layer before the linear layer.

For full finetuning, we followed the Segmenter training recipe \cite{segmenter} exactly. For ADE$20$k, the base learning rate is $10^{-3}$ for $160$k iterations with a batch size of $8$, at $512^2$ px. For Cityscapes, the base learning rate is $10^{-2}$ for $80$k iterations with a batch size of $8$, at $384^2$ px. We train with SGD.

For linear probing, we freeze the backbone and pre-compute the patch representations. We use the AdamW optimizer \cite{adamw} and sweep the following learning rates: $\{0.0001, 0.0002, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2, 0.3, 0.5\}$. For both ADE$20$k and Cityscapes we set the batch size to $16$ and train the linear decoder for $50$ epochs.

\newpage

\subsection{Full Experimental Results}\label{appn:fullresults}
\begin{table}[!th]\centering\small
\caption{First half of our hyper-parameter sweep. ViT-B models trained on ImageNet for $150$ epochs; trained and tested at $224^2$. RA is for RandAugment and 3A for 3-Augment.}
\label{tab:sweep_table_1}
\setlength{\tabcolsep}{2pt}
\begin{tabular}{
\toprule
&WD&LR&Data& \multicolumn{2}{c}{Val \cite{imagenet}} & \multicolumn{2}{c}{ReaL \cite{real}} & \multicolumn{2}{c}{v$2$ \cite{v2}} & \multicolumn{2}{c}{-A \cite{a}} & \multicolumn{2}{c}{-R \cite{r}} & \multicolumn{2}{c}{-HR (ours)} \\
\cmidrule(lr){5-6}
\cmidrule(lr){7-8}
\cmidrule(lr){9-10}
\cmidrule(lr){11-12}
\cmidrule(lr){13-14}
\cmidrule(lr){15-16}
Method &  $10^{-2}$ & $10^{-3}$ & Aug              &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} \\
\midrule
$1$D-learn  & 2	&3.0&3A	&78.51&	93.42&	83.91&	95.45&	66.86&	86.11&	9.64&	27.71&	28.21&	41.85&	86.54&	95.88     \\
$2$D-sincos  & 2	&3.0&3A	&77.96&	93.29&	83.63&	95.34&	66.35&	85.84&	9.55&	27.57&	28.29&	42.01&	86.24&	96.16  \\
Factorized  & 2	&3.0&3A	&	78.42&	93.47&	84.09&	95.64&	66.89&	85.84&	8.36&	25.65&	28.79&	42.68&	86.70&	96.48  \\
Fourier   & 2	&3.0&3A	&	78.78&	93.37&	84.21&	95.50&	67.32&	85.89&	9.56&	27.08&	28.90&	42.29&	87.24&	96.28     \\
RPE-learn   & 2	&3.0&3A	&78.92&	93.76&	84.46&	95.74&	67.75&	86.28&	10.00&	28.01&	28.70&	42.45&	87.06&	96.24   \\
$2$D-ALiBi   & 2	&3.0&3A	&	78.47&	93.68&	84.19&	95.80&	66.66&	86.18&	9.01&	26.44&	26.51&	39.74&	86.46&	96.50     \\
$2$D-RoPE  & 2	&3.0&3A	&79.36&	93.96&	84.70&	95.96&	67.98&	86.93&	10.37&	28.36&	30.58&	44.46&	87.42&	96.64  \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$180$} & 2	&3.0	& 3A& 80.76&	94.78& 86.23&	96.56&69.43&	88.02&	11.47&	29.87&31.09&	44.20&88.86&97.08  \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$} & 2	&3.0	&3A & 80.75&	94.71&	86.17&	96.45&	69.85&	87.97&	12.27&	30.24&	31.19&	44.20&	88.90&	97.06 \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$45$}& 2	&3.0	&3A &	80.49&	94.55&	86.06&	96.42&	69.27&	87.55&	11.44&	30.56&	31.70&	45.38&	88.90&	97.02  \\
\midrule
$1$D-learn  &5&3.0&3A&	79.45&	94.30&	84.97&	96.10&	68.49&	87.59&	10.97&	30.59&	29.64&	43.48&	88.28&	96.76    \\
$2$D-sincos  &5&3.0&3A&	79.05&	94.25&	84.62&	96.14&	67.86&	87.01&	10.45&	29.41&	29.11&	43.24&	87.58&	96.48 \\
Factorized  &5&3.0&3A&	79.86&	94.73&	85.30&	96.41&	69.11&	87.87&	11.00&	31.32&	29.99&	44.22&	87.86&	97.02  \\
Fourier   &5&3.0&3A&	79.69&	94.41&	85.13&	96.36&	68.30&	87.66&	11.36&	30.93&	29.73&	43.90&	88.14&	96.96     \\
RPE-learn   &5&3.0&3A&	79.86&	94.64&	85.46&	96.64&	68.57&	87.72&	9.85&	29.27&	29.10&	43.28&	88.22&	97.32   \\
$2$D-ALiBi   &5&3.0&3A&	79.54&	94.57&	85.15&	96.38&	68.47&	87.58&	10.45&	29.33&	28.26&	41.91&	87.70&	96.74     \\
$2$D-RoPE  &5&3.0&3A&	80.38&	94.86&	85.64&	96.49&	69.34&	87.89&	13.03&	33.95&	32.45&	46.96&	88.78&	96.92  \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$180$} &5&3.0&3A&	81.31&	95.11&	86.53&	96.71&	70.70&	88.38&	13.53&	32.72&	32.10&	45.07&	89.86&	97.54  \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$} &5&3.0&3A&	81.02&	94.92&	86.44&	96.68&	70.28&	88.34&	13.15&	32.89&	31.77&	44.74&	89.90&	97.20 \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$45$}&5&3.0&3A&	81.06&	94.87&	86.23&	96.46&	69.65&	88.60&	13.41&	32.96&	32.12&	45.25&	89.46&	97.06  \\
\midrule
$1$D-learn  &2&3.0&RA&	76.51&	92.08&	81.84&	94.32&	63.89&	83.41&	6.12&	19.93&	23.56&	36.15&	84.26&	94.96    \\
$2$D-sincos  &2&3.0&RA&	76.38&	92.22&	81.77&	94.53&	63.87&	84.05&	6.57&	20.23&	23.62&	36.95&	84.40&	95.28 \\
Factorized  &2&3.0&RA&	76.45&	92.18&	82.16&	94.53&	64.31&	84.10&	6.57&	20.97&	24.30&	37.35&	84.34&	94.90  \\
Fourier  &2&3.0&RA&	76.59&	92.08&	82.07&	94.49&	64.51&	84.13&	7.28&	21.72&	24.20&	37.39&	83.76&	94.68    \\
RPE-learn   &2&3.0&RA&	76.37&	92.28&	81.90&	94.54&	63.99&	83.41&	6.12&	18.76&	23.05&	36.01&	83.58&	94.96 \\
$2$D-ALiBi  &2&3.0&RA&	76.08&	92.16&	81.52&	94.45&	63.67&	83.22&	5.61&	19.08&	22.17&	34.74&	83.20&	94.78   \\
$2$D-RoPE  &2&3.0&RA&	77.31&	93.10&	82.84&	95.22&	65.06&	84.75&	6.07&	20.63&	27.05&	41.09&	85.24&	95.76\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$180$} &2&3.0&RA&	80.02&	94.07&	85.15&	95.79&	68.32&	86.73&	9.21&	25.53&	27.69&	40.24&	87.18&	96.70  \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$} &2&3.0&RA&	79.36&	93.83&	84.67&	95.64&	67.64&	86.43&	10.00&	24.99&	27.86&	41.01&	87.20&	96.16 \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$45$}&2&3.0&RA&	79.77&	93.99&	84.93&	95.68&	68.30&	86.41&	9.36&	25.91&	28.35&	41.63&	86.40&	96.40  \\
\midrule
$1$D-learn  &5&3.0&RA&	78.06&	93.38&	83.35&	95.44&	65.33&	85.67&	8.07&	25.07&	25.98&	39.42&	84.94&	95.64   \\
$2$D-sincos &5&3.0&RA&	77.95&	93.27&	83.26&	95.37&	65.51&	85.64&	7.57&	25.37&	26.11&	39.63&	85.16&	95.88 \\
Factorized  &5&3.0&RA&	78.55&	93.96&	84.00&	95.91&	66.88&	86.19&	8.05&	24.05&	27.08&	40.76&	86.36&	96.28  \\
Fourier  &5&3.0&RA&	78.16&	93.47&	83.43&	95.41&	66.28&	85.90&	8.28&	25.16&	26.25&	39.94&	85.98&	95.98   \\
RPE-learn  &5&3.0&RA&	78.15&	93.57&	83.50&	95.51&	66.50&	85.91&	7.56&	23.64&	25.10&	38.51&	85.62&	95.64 \\
$2$D-ALiBi &5&3.0&RA&	77.00&	92.89&	82.49&	94.88&	64.75&	85.05&	6.88&	21.55&	23.65&	36.51&	84.22&	95.46  \\
$2$D-RoPE  &5&3.0&RA&	79.29&	94.05&	84.46&	95.85&	67.73&	86.67&	10.77&	29.89&	28.99&	43.18&	86.50&	96.30\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$180$} &5&3.0&RA&	80.20&	94.32&	85.22&	96.02&	68.27&	86.70&	10.60&	27.40&	27.80&	40.05&	87.74&	96.60  \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$} &5&3.0&RA&	80.35&	94.33&	85.47&	95.96&	68.98&	87.29&	10.67&	27.89&	28.54&	41.64&	87.76&	96.56 \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$45$}&5&3.0&RA&	80.13&	94.31&	85.10&	95.99&	68.19&	86.65&	10.89&	27.57&	28.35&	41.35&	87.08&	96.20  \\
\bottomrule
\end{tabular}
\end{table}
\newpage
\begin{table}[!th]\centering\small
\caption{Second half of our hyper-parameter sweep. ViT-B models trained on ImageNet for $150$ epochs; trained and tested at $224^2$. RA is for RandAugment and 3A for 3-Augment.}
\label{tab:sweep_table_2}
\setlength{\tabcolsep}{2pt}
\begin{tabular}{
\toprule
&WD&LR&Data& \multicolumn{2}{c}{Val \cite{imagenet}} & \multicolumn{2}{c}{ReaL \cite{real}} & \multicolumn{2}{c}{v$2$ \cite{v2}} & \multicolumn{2}{c}{-A \cite{a}} & \multicolumn{2}{c}{-R \cite{r}} & \multicolumn{2}{c}{-HR (ours)} \\
\cmidrule(lr){5-6}
\cmidrule(lr){7-8}
\cmidrule(lr){9-10}
\cmidrule(lr){11-12}
\cmidrule(lr){13-14}
\cmidrule(lr){15-16}
Method &  $10^{-2}$ & $10^{-3}$ & Aug              &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} \\
\midrule
$1$D-learn  & 2	&1.5&3A	&77.31&	92.41&	82.95&	94.79&	64.85&	84.22&	6.95&	22.00&	26.14&	38.81&	85.68&	95.84     \\
$2$D-sincos  & 2	&1.5&3A	&77.50&	92.64&	83.14&	94.93&	65.57&	84.70&	7.23&	22.91&	26.66&	39.92&	85.78&	95.60  \\
Factorized  & 2	&1.5&3A	&	76.88&	92.43&	82.82&	94.88&	64.96&	84.43&	5.99&	19.96&	26.47&	39.91&	85.36&	95.38  \\
Fourier   & 2	&1.5&3A	&	77.15&	92.31&	82.82&	94.63&	64.92&	84.24&	7.09&	22.67&	26.28&	39.49&	85.38&	95.48     \\
RPE-learn   & 2	&1.5&3A	&77.07&	92.61&	82.97&	95.00&	65.13&	84.52&	6.52&	21.40&	24.75&	37.91&	85.16&	95.62   \\
$2$D-ALiBi   & 2	&1.5&3A	&	77.72&	93.05&	83.40&	95.38&	66.23&	85.56&	7.59&	22.76&	25.78&	38.89&	85.88&	96.14     \\
$2$D-RoPE  & 2	&1.5&3A	&78.14&	93.19&	83.74&	95.40&	66.67&	85.57&	8.20&	25.76&	28.78&	42.64&	86.26&	96.14  \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$180$} & 2	&1.5& 3A& 80.14&	94.19&	85.51&	96.12&	68.87&	87.25&	11.03&	27.84&	29.73&	42.81&	88.14&	96.82  \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$} & 2	&1.5	&3A & 79.88&	94.18&	85.51&	96.12&	69.34&	87.07&	10.83&	28.32&	30.88&	44.23&	88.32&	96.92 \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$45$}& 2	&1.5	&3A &	79.57&	94.06&	85.22&	96.01&	68.40&	87.02&	9.43&	27.60&	30.69&	44.85&	87.86&	96.88  \\
\midrule
$1$D-learn  &5&1.5&3A&	77.87&	93.31&	83.56&	95.47&	66.56&	85.69&	8.64&	25.03&	27.16&	40.67&	86.40&	95.86    \\
$2$D-sincos  &5&1.5&3A&	78.48&	93.50&	83.99&	95.65&	66.65&	86.19&	8.85&	25.75&	27.72&	41.90&	86.88&	96.46 \\
Factorized  &5&1.5&3A&	77.34&	92.98&	83.12&	95.32&	65.62&	85.33&	7.24&	23.51&	26.86&	40.51&	86.42&	96.02  \\
Fourier   &5&1.5&3A&	77.89&	93.18&	83.59&	95.30&	66.02&	84.99&	8.49&	25.13&	26.44&	39.74&	86.04&	96.04     \\
RPE-learn   &5&1.5&3A&	77.71&	93.31&	83.50&	95.44&	66.12&	85.43&	7.91&	24.67&	25.23&	38.30&	86.60&	96.02   \\
$2$D-ALiBi   &5&1.5&3A&	78.56&	93.77&	84.32&	95.88&	66.34&	86.57&	8.60&	27.16&	26.97&	40.76&	86.62&	96.60     \\
$2$D-RoPE  &5&1.5&3A&	78.74&	93.79&	84.45&	95.86&	67.12&	86.61&	9.69&	27.61&	29.82&	43.58&	87.62&	96.60  \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$180$} &5&1.5&3A&	80.53&	94.65&	85.82&	96.47&	69.38&	87.70&	11.63&	29.75&	30.07&	43.02&	88.66&	97.06  \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$} &5&1.5&3A&	80.34&	94.68&	85.81&	96.48&	68.92&	87.89&	11.55&	30.31&	30.85&	44.73&	88.56&	97.00 \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$45$}&5&1.5&3A&	80.32&	94.60&	85.59&	96.34&	68.78&	87.33&	10.71&	29.65&	31.25&	45.14&	88.82&	97.06  \\
\midrule
$1$D-learn  &2&1.5&RA&	75.02&	90.83&	80.68&	93.26&	62.38&	81.45&	4.55&	16.04&	21.92&	33.64&	82.50&	94.44    \\
$2$D-sincos  &2&1.5&RA&	75.72&	91.12&	81.32&	93.59&	62.60&	82.06&	5.61&	18.31&	22.82&	35.10&	82.82&	94.00 \\
Factorized  &2&1.5&RA&	74.47&	90.62&	80.40&	93.21&	61.73&	81.29&	4.63&	15.31&	22.10&	33.95&	82.16&	93.40  \\
Fourier  &2&1.5&RA&	74.95&	90.58&	80.59&	93.19&	61.66&	81.23&	4.97&	17.11&	22.08&	34.01&	82.84&	94.14    \\
RPE-learn   &2&1.5&RA&	74.65&	90.50&	80.28&	93.11&	61.42&	81.04&	4.44&	15.64&	20.22&	31.96&	82.14&	94.12 \\
$2$D-ALiBi  &2&1.5&RA&	74.95&	90.75&	80.69&	93.40&	61.92&	81.07&	5.01&	16.52&	20.45&	32.03&	83.18&	93.82   \\
$2$D-RoPE  &2&1.5&RA&	76.59&	91.56&	81.98&	93.90&	63.96&	83.06&	6.13&	19.84&	24.69&	37.57&	83.94&	95.04\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$180$} &2&1.5&RA&	78.42&	93.04&	83.68&	95.03&	66.46&	85.12&	8.20&	22.65&	26.07&	38.42&	86.18&	95.58  \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$} &2&1.5&RA&	78.62&	93.21&	84.09&	95.16&	66.64&	85.15&	8.77&	24.07&	27.22&	39.94&	86.50&	95.82 \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$45$}&2&1.5&RA&	78.17&	93.12&	83.61&	95.22&	66.54&	85.01&	7.49&	23.24&	26.56&	39.61&	85.98&	95.78  \\
\midrule
$1$D-learn  &5&1.5&RA&	76.07&	91.79&	81.66&	94.11&	63.03&	83.12&	5.73&	19.24&	23.20&	35.88&	83.12&	94.62   \\
$2$D-sincos &5&1.5&RA&	76.50&	92.22&	81.96&	94.49&	64.10&	83.72&	6.21&	19.16&	24.23&	37.13&	84.02&	94.68 \\
Factorized  &5&1.5&RA&	76.38&	92.25&	82.05&	94.69&	63.25&	83.35&	5.41&	17.81&	23.88&	36.33&	83.72&	95.16  \\
Fourier  &5&1.5&RA&	75.78&	91.66&	81.31&	94.04&	63.62&	83.10&	5.29&	18.76&	22.95&	35.26&	83.70&	94.80   \\
RPE-learn  &5&1.5&RA&	75.33&	91.29&	80.98&	93.77&	62.04&	82.19&	5.07&	18.47&	20.80&	32.74&	83.00&	94.32 \\
$2$D-ALiBi &5&1.5&RA&	76.02&	91.94&	81.61&	94.18&	63.16&	82.93&	5.01&	17.97&	21.53&	33.85&	83.80&	94.78  \\
$2$D-RoPE  &5&1.5&RA&	77.13&	92.53&	82.47&	94.80&	64.63&	84.68&	6.47&	20.95&	26.08&	39.35&	85.00&	95.14\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$180$} &5&1.5&RA&	78.68&	93.68&	84.26&	95.69&	66.76&	85.67&	7.93&	23.17&	26.98&	40.13&	85.74&	96.12  \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$} &5&1.5&RA&	78.77&	93.63&	84.09&	95.37&	66.69&	85.83&	9.15&	25.33&	27.45&	40.08&	85.70&	95.98 \\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$45$}&5&1.5&RA&	78.39&	93.45&	83.85&	95.37&	66.30&	85.61&	8.97&	24.84&	27.11&	40.44&	84.90&	95.66  \\
\bottomrule
\end{tabular}
\end{table}
\newpage

\subsection{Ablations}\label{appn:ablations}
We train $18$ models to ablate the LookHere design. Each run uses our best $150$ epoch training recipe. We test models without extrapolation at $224^2$ px (Table \ref{tab:ablation_no_extrap}) and with extrapolation at $1024^2$ px (Table \ref{tab:ablation_yes_extrap}). Before running extrapolation tests, we tune the global slope of each model at $1024^2$ px to fairly compare with our three default variants. To fit in the tables, we use short forms explained here: ``undir$\to 90$'' means replacing the four undirected heads with four $90 \degree$ FOV heads, ``undir$\to$ no dist'' means removing the distance penalties on the four undirected heads, ``invert'' means inverting the layer-wise slope pattern such that $s_l$ linearly increases from $0.5$ to $1.5$ with depth, ``mask:$\infty \to 0$'' means replacing $\infty$ with $0$ in equation \ref{equations}, and ``dist$\to$no dist'' means removing the distance penalties on all heads.
\begin{table}[!th]\centering\small
\caption{LookHere design ablations \textit{without} extrapolation. ViT-B models trained on ImageNet for $150$ epochs; trained and tested at $224^2$.}
\label{tab:ablation_no_extrap}
\setlength{\tabcolsep}{2pt}
\begin{tabular}{
\toprule
&& \multicolumn{2}{c}{Val \cite{imagenet}} & \multicolumn{2}{c}{ReaL \cite{real}} & \multicolumn{2}{c}{v$2$ \cite{v2}} & \multicolumn{2}{c}{-A \cite{a}} & \multicolumn{2}{c}{-R \cite{r}} & \multicolumn{2}{c}{-HR (ours)} \\
\cmidrule(lr){3-4}
\cmidrule(lr){5-6}
\cmidrule(lr){7-8}
\cmidrule(lr){9-10}
\cmidrule(lr){11-12}
\cmidrule(lr){13-14}
Variant & Change            &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} \\
\midrule
\color{LHcolor_dark}\textbf{LH-}\boldmath{$45$}  & undir$\rightarrow90\degree$ & 80.53&	94.92&	86.27&	96.81&	69.42&	88.29&	10.33&	29.60&	32.49&	46.78&	89.42&	97.38\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$45$}  & undir$\rightarrow180\degree$ & 80.72&	94.90&	86.19&	96.74&	69.66&	88.51&	10.81&	29.59&	32.14&	46.13&	89.44&	97.26\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$45$}  & undir$\rightarrow$no dist & 81.14&	95.08&	86.44&	96.74&	70.53&	88.48&	14.17&	34.07&	32.61&	46.14&	89.84&	97.54\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & undir$\rightarrow90\degree$ & 81.00&	94.98&	86.59&	96.78&	70.15&	88.46&	10.84&	29.40&	32.44&	46.46&	89.10&	97.62\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & undir$\rightarrow180\degree$ & 80.94&	95.06&	86.54&	96.78&	69.99&	88.55&	12.29&	30.83&	31.73&	45.64&	89.46&	97.06\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & undir$\rightarrow$no dist & 81.01&	95.13&	86.38&	96.79&	70.37&	88.68&	12.41&	32.39&	32.27&	46.51&	89.34&	97.52\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$180$}  & undir$\rightarrow90\degree$ & 80.82&	95.02&	86.56&	96.78&	69.50&	88.57&	11.85&	29.99&	31.85&	45.77&	88.98&	97.18\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$180$}  & undir$\rightarrow180\degree$ & 80.88&	95.11&	86.56&	96.87&	70.36&	88.33&	11.96&	30.55&	31.63&	45.52&	89.28&	97.32\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$180$}  & undir$\rightarrow$no dist & 81.39&	95.11&	86.78&	96.77&	70.66&	88.43&	12.49&	32.00&	31.79&	44.93&	89.84&	97.50\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & $s_g:1\rightarrow0.125$ & 81.20&	95.03&	86.48&	96.69&	70.14&	88.33&	13.63&	33.27&	32.14&	45.44&	89.22&	97.08\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & $s_g:1\rightarrow0.25$ & 81.08&	94.92&	86.28&	96.53&	70.02&	88.05&	12.64&	31.43&	31.46&	44.73&	88.88&	97.06\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & $s_g:1\rightarrow0.5$ & 81.09&	94.97&	86.47&	96.58&	70.18&	88.40&	13.04&	33.00&	32.02&	45.67&	89.56&	97.30\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & $s_g:1\rightarrow4$ & 80.91&	95.10&	86.58&	96.92&	70.16&	88.69&	11.40&	30.31&	32.13&	46.33&	89.40&	97.46\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & $s_l:$ invert & 81.37&	95.02&	86.43&	96.72&	70.30&	88.32&	13.87&	33.88&	32.69&	46.55&	89.72&	97.44\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & $\text{dist}\rightarrow\text{dist}^2$ & 80.98&	95.17&	86.50&	96.88&	70.34&	88.50&	11.45&	30.44&	32.13&	46.43&	89.66&	97.48\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & $\text{dist}\rightarrow\sqrt{\text{dist}}$ & 80.86&	94.89&	86.17&	96.55&	69.15&	88.15&	12.33&	31.75&	31.65&	45.23&	88.56&	97.32\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & mask$:\infty\rightarrow0$ & 79.68&	94.47&	85.11&	96.47&	68.58&	87.82&	11.21&	30.33&	29.69&	43.59&	87.94&	97.02\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & dist$\rightarrow$no dist & 80.19&	94.77&	85.49&	96.48&	69.22&	88.27&	11.52&	30.92&	31.76&	46.33&	88.52&	97.10\\
\bottomrule
\end{tabular}
\end{table}

\begin{table}[!th]\centering\small
\caption{LookHere design ablations \textit{with} extrapolation. ViT-B models trained on ImageNet for $150$ epochs; trained at $224^2$ and tested at $1024^2$.}
\label{tab:ablation_yes_extrap}
\setlength{\tabcolsep}{2pt}
\begin{tabular}{
\toprule
&& \multicolumn{2}{c}{Val \cite{imagenet}} & \multicolumn{2}{c}{ReaL \cite{real}} & \multicolumn{2}{c}{v$2$ \cite{v2}} & \multicolumn{2}{c}{-A \cite{a}} & \multicolumn{2}{c}{-R \cite{r}} & \multicolumn{2}{c}{-HR (ours)} \\
\cmidrule(lr){3-4}
\cmidrule(lr){5-6}
\cmidrule(lr){7-8}
\cmidrule(lr){9-10}
\cmidrule(lr){11-12}
\cmidrule(lr){13-14}
Variant & Change            &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$1$} &
\multicolumn{1}{>{\centering\arraybackslash}p{21.2pt}}{top-$5$} \\
\midrule
\color{LHcolor_dark}\textbf{LH-}\boldmath{$45$}  & undir$\rightarrow90\degree$ & 71.94&	91.01&	78.26&	93.58&	59.23&	81.53&	5.89&	18.47&	16.02&	27.72&	81.40&	95.26\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$45$}  & undir$\rightarrow180\degree$ & 69.97&	89.42&	76.05&	92.44&	56.19&	79.03&	5.33&	16.71&	13.70&	24.44&	78.72&	94.12\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$45$}  & undir$\rightarrow$no dist & 69.72&	89.35&	75.69&	92.33&	55.97&	78.76&	4.77&	15.76&	12.92&	23.42&	79.14&	93.46\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & undir$\rightarrow90\degree$ & 69.39&	88.99&	75.67&	92.00&	55.82&	78.68&	5.00&	16.40&	11.93&	21.96&	78.10&	93.46\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & undir$\rightarrow180\degree$ & 68.80&	88.62&	74.95&	91.72&	53.89&	77.43&	4.16&	14.49&	12.72&	22.84&	76.60&	92.48\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & undir$\rightarrow$no dist & 69.24&	89.40&	75.29&	92.46&	56.03&	79.38&	4.84&	15.65&	12.67&	23.08&	78.24&	94.08\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$180$}  & undir$\rightarrow90\degree$ & 64.44&	85.72&	70.73&	89.36&	49.80&	73.70&	3.40&	12.16&	8.98&	17.32&	73.60&	90.14\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$180$}  & undir$\rightarrow180\degree$ & 54.13&	77.21&	59.97&	81.66&	39.38&	63.14&	1.61&	5.51&	4.23&	8.80&	66.44&	84.56\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$180$}  & undir$\rightarrow$no dist & 66.35&	86.89&	72.67&	90.27&	51.97&	75.62&	4.92&	14.69&	9.33&	17.43&	74.30&	91.20\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & $s_g:1\rightarrow0.125$ & 67.36&	87.84&	73.31&	91.06&	53.06&	76.37&	3.15&	10.69&	12.15&	22.20&	76.58&	91.90\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & $s_g:1\rightarrow0.25$ & 70.46&	89.68&	76.43&	92.52&	57.11&	80.17&	5.43&	16.19&	12.45&	22.25&	79.32&	93.14\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & $s_g:1\rightarrow0.5$ & 72.53&	90.72&	78.40&	93.30&	59.34&	81.56&	7.16&	20.80&	13.88&	24.05&	79.82&	93.36\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & $s_g:1\rightarrow4$ & 55.16&	78.88&	61.14&	83.31&	41.30&	66.02&	2.85&	9.59&	5.19&	10.95&	68.02&	86.92\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & $s_l:$ invert & 70.03&	89.28&	76.09&	92.30&	55.69&	79.06&	6.21&	19.29&	11.85&	21.36&	76.80&	92.26\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & $\text{dist}\rightarrow\text{dist}^2$ & 65.08&	87.03&	71.16&	90.46&	51.05&	75.52&	3.28&	12.35&	10.96&	20.95&	74.80&	92.04\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & $\text{dist}\rightarrow\sqrt{\text{dist}}$ & 66.83&	87.59&	72.53&	90.70&	52.88&	76.41&	3.91&	12.57&	10.93&	19.83&	76.80&	92.60\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & mask$:\infty\rightarrow0$ & 40.52&	66.37&	45.24&	70.96&	28.48&	52.11&	1.23&	5.84&	2.58&	6.58&	50.18&	75.08\\
\color{LHcolor_dark}\textbf{LH-}\boldmath{$90$}  & dist$\rightarrow$no dist & 44.42&	69.57&	49.20&	74.05&	31.31&	55.09&	1.03&	5.11&	6.02&	13.10&	53.92&	78.48\\
\bottomrule
\end{tabular}
\end{table}
\newpage

\subsection{Logit Lens}\label{appn:logitlens}

\begin{figure}[ht]
\includegraphics[scale=0.7]{daniel_pngs/more_in-s_examples.png}
\caption{More examples from ImageNet-S and each model's logit lens predictions.}
\label{fig:morelogitlens}
\end{figure}

\newpage
\begin{figure}[ht]
\includegraphics[scale=0.5]{daniel_pngs/cls-patch-ident.png}
\caption{We plot the average class identifiability \cite{vilas2023analyzing_vit} across the model layers on $1000$ images from Val for the class and patch tokens. This is a measure of how recoverable the correct class is from the class projection of the token. The score ranges from $0$ to $1$, with $1$ denoting that the correct class has the highest logits and $0$ the lowest.
\label{fig:classident}
\end{figure}

\newpage
\begin{figure}[ht]
\includegraphics[scale=0.455]{daniel_pngs/all_layers.png}
\caption{Leveraging the semantic segmentation labels from the ImageNet-S, we compared the identifiability rate of class patches (blue) vs non-class tokens \cite{vilas2023analyzing_vit} across the model layers on $1000$ images from Val. LookHere can discriminate between class and non-class patches. Other positional encodings cannot unless they are trained for much longer.
\thispagestyle{empty}
\label{fig:classident2}
\end{figure}

\clearpage

\subsection{Head Diversity, Attention Distance, Patch Similarity and Head Visualizations}\label{appn:measure}
In our paper, we show that LookHere prevents attention collapse measured by JSD (Figure \ref{fig:jsd}). Here, we measure attention diversity using $L_1$ and $L_2$ distance. We also measure attention distances and patch-wise representational similarity --- both at $224^2$ px (Figure \ref{fig:summary_measurements}) and at all resolutions tested (Figure \ref{fig:attn_jsd_1} \& \ref{fig:attn_jsd_2}).
\usetikzlibrary{positioning}
\begin{figure}[!ht]
\centering
\begin{tikzpicture}
\hspace{-50pt}

% Row 1
\node[anchor=south west, inner sep=0] (image1) at (0,0)

\node[anchor=south west, inner sep=0] (image2) at (image1.south east)

% Row 2
\node[anchor=north west, inner sep=0] (image3) at (image1.south west)

\node[anchor=north west, inner sep=0] (image4) at (image3.north east)

\end{tikzpicture}
\caption{Measurements of head diversity, attention distance, and patch similarity by layer across position encoding methods.}
\label{fig:summary_measurements}

\end{figure}

\begin{figure}[ht]
\input{res_legend.tex}
\begin{tikzpicture}
% Row 1
\node[anchor=south west, inner sep=0] (image1) at (0,0) {\includegraphics[width=.34\textwidth]{yousef_pngs/attn_distance/1D-learn.png}};
\node[align=center] at ([xshift=-0.5cm]image1.west) {\small{1}};

\node[anchor=south west, inner sep=0] (image2) at (image1.south east) {\includegraphics[width=.34\textwidth]{yousef_pngs/attn_distance/2D-sincos.png}};
\node[anchor=south west, inner sep=0] (image3) at (image2.south east) {\includegraphics[width=.34\textwidth]{yousef_pngs/attn_distance/Factorized.png}};

% Row 2
\node[anchor=north west, inner sep=0] (image4) at (image1.south west) {\includegraphics[width=.34\textwidth]{yousef_pngs/attn_distance/Fourier.png}};
\node[align=center] at ([xshift=-0.5cm]image4.west) {\small{2}};

\node[anchor=north west, inner sep=0] (image5) at (image4.north east) {\includegraphics[width=.34\textwidth]{yousef_pngs/attn_distance/RPE-learn.png}};
\node[anchor=north west, inner sep=0] (image6) at (image5.north east) {\includegraphics[width=.34\textwidth]{yousef_pngs/attn_distance/2D-ALiBi.png}};

% Row 3
\node[anchor=north west, inner sep=0] (image7) at (image4.south west) {\includegraphics[width=.34\textwidth]{yousef_pngs/attn_distance/2D-RoPE.png}};
\node[align=center] at ([xshift=-0.5cm]image7.west) {\small{3}};

\node[anchor=north west, inner sep=0] (image8) at (image7.north east) {\includegraphics[width=.34\textwidth]{yousef_pngs/attn_distance/LH-180.png}};
\node[anchor=north west, inner sep=0] (image9) at (image8.north east) {\includegraphics[width=.34\textwidth]{yousef_pngs/attn_distance/LH-90.png}};

% Row 4
\node[anchor=north west, inner sep=0] (image10) at (image7.south west) {\includegraphics[width=.34\textwidth]{yousef_pngs/attn_distance/LH-45.png}};
\node[align=center] at ([xshift=-0.5cm]image10.west) {\small{4}};

\node[anchor=north west, inner sep=0] (image11) at (image10.north east) {\includegraphics[width=.34\textwidth]{yousef_pngs/jsd/1D-learn.png}};  % STARTING JSD HERE
\node[anchor=north west, inner sep=0] (image12) at (image11.north east) {\includegraphics[width=.34\textwidth]{yousef_pngs/jsd/2D-sincos.png}};

% Row 5
\node[anchor=north west, inner sep=0] (image13) at (image10.south west) {\includegraphics[width=.34\textwidth]{yousef_pngs/jsd/Factorized.png}};
\node[align=center] at ([xshift=-0.5cm]image13.west) {\small{5}};

\node[anchor=north west, inner sep=0] (image14) at (image13.north east) {\includegraphics[width=.34\textwidth]{yousef_pngs/jsd/Fourier.png}};
\node[anchor=north west, inner sep=0] (image15) at (image14.north east) {\includegraphics[width=.34\textwidth]{yousef_pngs/jsd/RPE-learn.png}};

% Row 5
\node[anchor=north west, inner sep=0] (image16) at (image13.south west) {\includegraphics[width=.34\textwidth]{yousef_pngs/jsd/2D-ALiBi.png}};
\node[align=center] at ([xshift=-0.5cm]image16.west) {\small{6}};

\node[anchor=north west, inner sep=0] (image17) at (image16.north east) {\includegraphics[width=.34\textwidth]{yousef_pngs/jsd/2D-RoPE.png}};
\node[anchor=north west, inner sep=0] (image18) at (image17.north east) {\includegraphics[width=.34\textwidth]{yousef_pngs/jsd/LH-180.png}};

\end{tikzpicture}
\SmallCaption{We measure the attention distance per depth, for each model and resolution, by taking the sum of patch distances weighted by attention scores, averaged across heads [(row: 1, col: 1) $\to$ (4, 1)]; and the head diversity as the generalized JSD of attention matrix rows for each head, averaged over rows [$(4, 2) \to (7, 2)$]. We report the average over $500$ randomly selected images from minival.}
\label{fig:attn_jsd_1}
\end{figure}
\newpage
\begin{figure}[ht]
\centering
\input{res_legend.tex}
\begin{tikzpicture}
% Row 1
\node[anchor=south west, inner sep=0] (image1) at (0,0) {\includegraphics[width=.34\textwidth]{yousef_pngs/jsd/LH-90.png}};
\node[align=center] at ([xshift=-0.5cm]image1.west) {\small{7}};

\node[anchor=south west, inner sep=0] (image2) at (image1.south east) {\includegraphics[width=.34\textwidth]{yousef_pngs/jsd/LH-45.png}};
\node[anchor=south west, inner sep=0] (image3) at (image2.south east) {\includegraphics[width=.34\textwidth]{yousef_pngs/patch_sim/1D-learn.png}};

% Row 2
\node[anchor=north west, inner sep=0] (image4) at (image1.south west) {\includegraphics[width=.34\textwidth]{yousef_pngs/patch_sim/2D-sincos.png}};
\node[align=center] at ([xshift=-0.5cm]image4.west) {\small{8}};

\node[anchor=north west, inner sep=0] (image5) at (image4.north east) {\includegraphics[width=.34\textwidth]{yousef_pngs/patch_sim/Factorized.png}};
\node[anchor=north west, inner sep=0] (image6) at (image5.north east) {\includegraphics[width=.34\textwidth]{yousef_pngs/patch_sim/Fourier.png}};

% Row 3
\node[anchor=north west, inner sep=0] (image7) at (image4.south west) {\includegraphics[width=.34\textwidth]{yousef_pngs/patch_sim/RPE-learn.png}};
\node[align=center] at ([xshift=-0.5cm]image7.west) {\small{9}};

\node[anchor=north west, inner sep=0] (image8) at (image7.north east) {\includegraphics[width=.34\textwidth]{yousef_pngs/patch_sim/2D-ALiBi.png}};
\node[anchor=north west, inner sep=0] (image9) at (image8.north east) {\includegraphics[width=.34\textwidth]{yousef_pngs/patch_sim/2D-RoPE.png}};

% Row 4
\node[anchor=north west, inner sep=0] (image10) at (image7.south west) {\includegraphics[width=.34\textwidth]{yousef_pngs/patch_sim/LH-180.png}};
\node[align=center] at ([xshift=-0.5cm]image10.west) {\small{10}};

\node[anchor=north west, inner sep=0] (image11) at (image10.north east) {\includegraphics[width=.34\textwidth]{yousef_pngs/patch_sim/LH-90.png}};
\node[anchor=north west, inner sep=0] (image12) at (image11.north east) {\includegraphics[width=.34\textwidth]{yousef_pngs/patch_sim/LH-45.png}};

\end{tikzpicture}
\SmallCaption{The patch similarity, for each model and resolution, is measured as the average of pairwise cosine-similarities between patch representations in each layer [$(7, 3) \to (10, 3)$]; we report the average over the same $500$ minival images.}
\label{fig:attn_jsd_2}
\end{figure}
\usetikzlibrary{positioning}
\begin{figure}[!ht]
\centering
\begin{tikzpicture}
\hspace{-10pt}

\node[anchor=south west, inner sep=0] (image1-0) at (0.0\textwidth, -0.152\textwidth) {\includegraphics[width=.15\textwidth]{attn_maps/OneD/1_224.png}};
\node[anchor=south west, inner sep=0] (image1-1) at (0.152\textwidth, -0.152\textwidth) {\includegraphics[width=.15\textwidth]{attn_maps/OneD/1_320.png}};
\node[anchor=south west, inner sep=0] (image1-2) at (0.304\textwidth, -0.152\textwidth) {\includegraphics[width=.15\textwidth]{attn_maps/OneD/1_384.png}};