# Extending LLMs’ Context Window with 100 Samples

Yikai Zhang1,2,3 Junlong Li1,3 Pengfei Liu1,2,3
1 Shanghai Jiao Tong University 2 Shanghai Artificial Intelligence Lab
3 Generative AI Research Lab (GAIR)  Corresponding author

###### Abstract

Large Language Models (LLMs) are known to have limited extrapolation ability
beyond their pre-trained context window, constraining their application in
downstream tasks with lengthy inputs. Recent studies have sought to extend
LLMs’ context window by modifying rotary position embedding (RoPE), a popular
position encoding method adopted by well-known LLMs such as LLaMA, PaLM, and
GPT-NeoX. However, prior works like Position Interpolation (PI) and YaRN are
resource-intensive and lack comparative experiments to assess their
applicability. In this work, we identify the inherent need for LLMs’ attention
entropy (i.e. the information entropy of attention scores) to maintain
stability and introduce a novel extension to RoPE which combines adjusting
RoPE’s base frequency and scaling the attention logits to help LLMs
efficiently adapt to a larger context window. We validate the superiority of
our method in both fine-tuning performance and robustness across different
context window sizes on various context-demanding tasks. Notably, our method
extends the context window of LLaMA-2-7B-Chat to 16,384 with only 100 samples
and 6 training steps, showcasing extraordinary efficiency. Finally, we also
explore how data compositions and training curricula affect context window
extension for specific downstream tasks, suggesting fine-tuning LLMs with
lengthy conversations as a good starting point. We release our code and SFT
data at https://github.com/GAIR-NLP/Entropy-ABF.

Extending LLMs’ Context Window with 100 Samples

Yikai Zhang1,2,3 Junlong Li1,3 Pengfei Liu1,2,3††thanks:  Corresponding author
1 Shanghai Jiao Tong University 2 Shanghai Artificial Intelligence Lab 3
Generative AI Research Lab (GAIR)

##  1 Introduction

Large Language Models (LLMs) are typically pre-trained with a pre-defined
context window size. For instance, LLaMA 2 (Touvron et al., 2023b) is pre-
trained on sequences of 4,096 tokens. When exceeding the pre-trained context
window, the performance of LLMs tends to deteriorate primarily due to the
limited length extrapolation ability of their position encoding methods
(Kazemnejad et al., 2023). The limited context window affects LLMs’
practicality for ever-increasing context-demanding tasks such as few-shot
learning (Brown et al., 2020), long document summarization (Huang et al.,
2021) and repository-level code completion (Liu et al., 2023). Consequently,
there is an urgent need to extend LLMs’ context window.

To meet this pressing demand, recent works have witnessed progress in context
window extension in both fine-tuned and non-fine-tuned scenarios by extending
Rotary Position Embedding (RoPE) (Su et al., 2021), a widely-used position
encoding method adopted by state-of-the-art LLMs such as LLaMA (Touvron et
al., 2023a, b), PaLM (Chowdhery et al., 2023; Anil et al., 2023) and GPT-NeoX
(Black et al., 2022). For example, Position Interpolation (PI) (kaiokendev,
2023; Chen et al., 2023) linearly down-scales the input tokens’ position
indices and achieves improved fine-tuning results. NTK-Aware scaling (bloc97,
2023b) and adjusted base frequency (ABF) (Xiong et al., 2023) modify the base
frequency of RoPE, leading to enhanced results in fine-tuning and non-fine-
tuning scenarios respectively. NTK-By-Parts scaling (bloc97, 2023a) treats
different dimensions differently and reports even better fine-tuning outcomes.
More recently, YaRN (Peng et al., 2023) proposes scaling the attention logits
given its beneficial effects on language modeling perplexity. They combine
this technique with NTK-By-Parts scaling and report the best long-context
performance among existing RoPE-extension methods.

However, the underlying rationale behind the efficiency of YaRN’s scaling
operation remains poorly understood. In this study, we provide an
interpretation of this technique by analyzing its effect on stabilizing the
information entropy of models’ attention scores. Through our analysis, we
introduce our own RoPE-extension method termed “entropy-aware ABF”, which
combines ABF with a sophisticated utilization of dynamic attention scalar.

Moreover, despite the individual-reported efficacy of previous RoPE-extension
methods, there’s a lack of comprehensive comparative analysis where different
methods are put in the same evaluation testbed. This study also addresses this
gap by answering three key questions pertinent to context window extension in
real-world applications: (1) Which method exhibits the best supervised fine-
tuning performance on context-demanding downstream tasks? (2) How can each
method efficiently utilize training data? (3) Do models trained with these
methods have a robust performance across varying context window sizes?

To answer the above questions, we conduct experiments on a diverse set of
context-demanding tasks from LongBench (Bai et al., 2023), manipulating the
training data amounts and prompt lengths to evaluate fine-tuned models across
different dimensions. The experiment results demonstrate that models trained
with our method surpass all baselines in long-context fine-tuning performance
and also maintain a robust performance across various context window sizes.
Notably, with only 100 long conversations from ShareGPT (Chiang et al., 2023)
and 6 training steps, using four A100 GPUs for approximately 6 minutes, our
method produces a model with competent performance across 12 selected context-
demanding tasks. Finally, we explore the influence of data compositions and
training curricula on context window extension for a given long context
downstream task, suggesting fine-tuning the model on lengthy ShareGPT
conversations as a good starting point.

##  2 Preliminaries

#### Rotary Position Embedding (RoPE)

Given a position index $m\in[1,c]$ and an embedding vector
$\mathbf{x}:=[x_{0},x_{1},\ldots,x_{d-1}]^{\top}$, where $d$ is the dimension
of each attention head, RoPE considers each pair of elements along the feature
dimension of the embedding vector as complex numbers and encodes position
information by rotating them. The vector-valued complex function
$\mathbf{f}(\mathbf{x},m)$ defined by RoPE is as follows:

$\mathbf{f}(\mathbf{x},m)=\begin{bmatrix}(x_{0}+\mathrm{i}x_{1})e^{\mathrm{i}m\theta_{1}},\\\
(x_{2}+\mathrm{i}x_{3})e^{\mathrm{i}m\theta_{2}},\\\ \ldots,\\\
(x_{d-2}+\mathrm{i}x_{d-1})e^{\mathrm{i}m\theta_{d/2}}\end{bmatrix}$

(1)

$\mathrm{i}:=\sqrt{-1}$ is the imaginary unit and
$\theta_{j}=\text{b}^{-2j/d}$, where $b$ denotes the base frequency of RoPE
and is set to $10,000$ by default.

In application, RoPE is applied to both query and key embeddings through the
following equation:

$\mathbf{f}(\mathbf{x},m)=\begin{bmatrix}x_{0}\\\ x_{1}\\\ x_{2}\\\ x_{3}\\\
\vdots\\\ x_{d-2}\\\
x_{d-1}\end{bmatrix}\otimes\begin{bmatrix}\cos(m\theta_{0})\\\
\cos(m\theta_{0})\\\ \cos(m\theta_{1})\\\ \cos(m\theta_{1})\\\ \vdots\\\
\cos(m\theta_{(d-1)/2})\\\
\cos(m\theta_{(d-1)/2})\end{bmatrix}+\begin{bmatrix}-x_{1}\\\ x_{0}\\\
-x_{3}\\\ x_{2}\\\ \vdots\\\ -x_{d-1}\\\
x_{d-2}\end{bmatrix}\otimes\begin{bmatrix}\sin(m\theta_{0})\\\
\sin(m\theta_{0})\\\ \sin(m\theta_{1})\\\ \sin(m\theta_{1})\\\ \vdots\\\
\sin(m\theta_{{(d-1)}/2})\\\ \sin(m\theta_{{(d-1)}/2})\end{bmatrix}$

(2)

The fundamental components of RoPE are a series of trigonometric coefficients,
each encoding position information of different frequencies.

We represent these trigonometric coefficients with the following function to
uniquely identify RoPE and its variants:

$h(m,b,t)=\sqrt{t}*cos(\frac{m}{b^{\frac{2j}{d}}})\text{ or
}\sqrt{t}*sin(\frac{m}{b^{\frac{2j}{d}}})$ (3)

where $m$ is the position index of the query token, $b$ is the base frequency
for RoPE, and $t$ is the scaling factor for attention logits. Note that
$\sqrt{t}$ is used in the equation because RoPE rotates process both the query
and key embeddings.

Before introducing RoPE-extension methods that enable better context window
extension, we define context scaling factor $s=\frac{c^{\prime}}{c}$, which is
the ratio between the target context window $c^{\prime}$ and the pre-trained
context window $c$. It is of special use to those methods that extend RoPE
according to a given target context window size.

#### Position Interpolation (PI)

PI (Chen et al., 2023; kaiokendev, 2023) linearly interpolates the input
position index $m$ to $\frac{m}{s}$ so that it falls within the original
context window size. Chen et al. (2023) demonstrate that direct fine-tuning of
LLaMA (Touvron et al., 2023a) with an extended context window results in
minimal improvement, as the effective context window of the model only
increases from 2,048 to 2560 after 10,000 training steps on sequences of
length 8,192. By contrast, PI succeeds in extending the context window of
LLaMA to 32,768 with only 1,000 training steps.

#### NTK-Aware

NTK-Aware scaling (bloc97, 2023b) hypothesize that interpolating all
dimensions equally, as done by PI, may result in loss of high-frequency
information. Therefore, NTK-Aware scaling introduces a nonlinear interpolation
strategy by adjusting the base frequency $b$ of RoPE to $b^{\frac{d}{d-2}}$.
This modification scales the low-frequency components of RoPE to a similar
extent as PI, while only slightly altering the high-frequency components to
avoid disturbing high-frequency information. NTK-Aware extends models’ context
window size without training. However, this method can’t benefit as much as PI
from additional training on longer sequences as suggested by (Peng et al.,
2023).

#### NTK-By-Parts

NTK-By-Parts (bloc97, 2023a) holds that stretching all the RoPE components
either by a scaling factor $s$ or a base transformation results in token
embeddings being closer to each other, impeding LLMs from effectively
capturing local relationships between adjacent tokens. To address this issue,
NTK-By-Parts scales $\theta(j)$ by a factor $\frac{1-\gamma(j)}{s}+\gamma(j)$,
with $\gamma(j)$ being assigned 0 for high frequencies, 1 for low frequencies,
and a predetermined constant within the range of 0 to 1 for intermediate
frequencies. According to (Peng et al., 2023), this method performs better
than PI and NTK-Aware scaling for both fine-tuned and non-fine-tuned models.

#### YaRN

Yarn (Peng et al., 2023) empirically observes that introducing a temperature
$t$ to scale the attention logits before the softmax function improves models’
language modeling performance. They find the optimal value of
$\sqrt{t}=0.1\ln{s}+1$ by fitting the lowest perplexity curve against various
context scaling factors $s$. They combine their finding with NTK-By-Parts
scaling and term this method YaRN (Yet another RoPE extensioN method). YaRN
reports the best long-context performance on language modeling tasks among
existing methods.

#### Adjusted Base Frequency (ABF)

ABF (Xiong et al., 2023) simply changes the base frequency of RoPE to 50,000.
Both theoretical analysis and experiment are conducted to validate the
efficacy of this method. Xiong et al. (2023) proves that ABF minimizes the
distance of its embedded vectors from the ones using the original RoPE, which
helps leverage the pre-training results. They empirically validate the
efficacy of ABF by showing a lower perplexity on language modeling tasks and a
longer effective context window in the first-sentence-retrieval task.

Table 1 highlights the difference between RoPE and its variants by specifying
the different $m$, $b$, and $t$ they use in Equation 3 and whether they
require additional training for context window extension:

Method | $m$ | $b$ | $t$ | Additional Training
---|---|---|---|---
RoPE | $m$ | $10,000$ | $1$ | -
PI | $m/s$ | $10,000$ | $1$ | continual pre-train
NTK-Aware | $m$ | $10,000^{\frac{d-2}{d}}$ | $1$ | -
NTK-By-Parts | $(\frac{1-\gamma(j)}{s}+\gamma(j))m$ | $10,000$ | $1$ | continual pre-train
YaRN | $(\frac{1-\gamma(j)}{s}+\gamma(j))m$ | $10,000$ | $0.1ln(s)+1$ | continual pre-train
ABF | $m$ | $500,000$ | $1$ | continual pre-train

Table 1: An overview of Rotary Position Embedding (RoPE) and its variants
represented by Equation 3.

##  3 Proposal Method

YaRN (Peng et al., 2023) introduces a scaling factor $t$ on the attention
logits based on empirical evidence indicating its beneficial effects on
language modeling perplexities. However, the underlying rationale behind this
technique remains poorly understood. In this section, we first introduce an
interpretation of this technique, which motivates our method.

###  3.1 Interpretation of YaRN’s Scaling Factor

In Transformer models’ attention mechanism (Vaswani et al., 2017), the Softmax
function forces attention scores assigned to contextual tokens to sum to one
while concurrently preventing any individual score from becoming zero.
Consequently, with an increasing number of input tokens, LLMs will
theoretically distribute more attention across more tokens and lead to a rise
in what we refer to as “attention entropy”, which quantifies the randomness
within the distribution of attention scores and is calculated using the
following equation:

$\text{attention\\_entropy}=\sum_{i}p_{i}\ln{p_{i}}$ (4)

where $p_{i}$ is the attention scores assigned to contextual tokens.

To validate the aforementioned theoretical effect, we utilized LLaMA-2-7B-Chat
(Touvron et al., 2023b) to process 128 randomly chosen documents from the Pile
dataset (Gao et al., 2020). We collect the attention scores assigned to
contextual tokens for query tokens at different input positions to simulate
varying numbers of contextual tokens. Subsequently, we compute the information
entropy for these attention scores on different model layers via Equation 4.
The resulting average attention entropies over our randomly sampled documents
are visualized in Figure 1.

Figure 1: Visualization of the averaged attention entropy for query tokens at
different input positions in the LLaMA-2-7B-chat model across the selected 128
documents from the Pile-arXiv dataset (Gao et al., 2020). “Uniform” represents
a uniform attention score distribution, which corresponds to
$\text{attention\\_entropy}=\ln{n}$ with $n$ denoting the number of contextual
tokens.

Counterintuitively, only the first two model layers demonstrate a steady rise
in attention entropy. Interestingly, we even observe that the attention
entropies of all the subsequent layers remain remarkably similar when the
number of contextual tokens increases from 1,024 to 2,048.

This finding of LLMs maintaining a stable attention entropy in subsequent
model layers when contextual tokens are directly doubled leads us to posit
that possessing a certain degree of length-invariance in attention entropy in
these layers is an important inherent characteristic of LLMs for functioning
properly. When modeling longer sequences than encountered in the pretraining
stage, LLMs might fail to concentrate well, leading to a performance drop.
Thanks to the exponential function in Softmax, scaling the attention logits
reduces attention entropy, thereby explaining why it leads to improvements in
language modeling tasks when modeling lengthy inputs as observed in YaRN (Peng
et al., 2023).

###  3.2 Design Principles

Previous works have explored different scaling factors on the attention logits
with different motivations. Chiang and Cholak (2022) scales the attention
logits by $\log n$, with $n$ representing the length of the longest training
sequence, to enhance the model’s extrapolation ability in downstream tasks
such as machine translation.

More recently, YaRN (Peng et al., 2023) introduces the scaling factor
$t=0.1\ln s+1$ by fitting the lowest perplexity curve in language modeling
tasks. They combine these scaling factors with NTK-By-Parts scaling and
observe improved fine-tuning long-context performance on language modeling
tasks.

ReRoPE (Su, 2023) utilized a dynamic scaling factor that takes into account
the number of contextual tokens for each input position: $t=\log_{c}{m}$,
where $c$ denotes the pre-trained context window size and $m$ represents the
position index of input tokens. By introducing this scaling factor during the
pre-training stage, ReRoPE demonstrates enhanced extrapolation ability in
language modeling tasks, which is also observed in YaRN.

We propose “entropy-aware ABF” with the following design principles:

(1). Dynamic Attention Scaling: Both PI and YaRN employ a constant scaling
factor for all input positions, which may excessively stretch the attention
logits at the front positions and hinder the model’s ability to extrapolate to
longer sequences. Instead of using a constant scaling factor, we propose using
a dynamic factor like ReRoPE that takes into account the number of contextual
tokens for each input position. This allows the model to adjust the attention
weights more flexibly based on the level of randomness in the distribution of
attention scores.

(2). Layer-dependent: All the existing works apply the scalar indiscriminately
to all model layers. However, based on our observations in Figure 1 that the
first two layers consistently exhibit a near-uniform attention pattern and
only the latter layers demonstrate the tendency to maintain concentration, we
propose not to intervene in the first two layers to align with the model’s
inherent characteristics.

(3). Facilitate Context Window Extension: Furthermore, we hypothesize that
learning to maintain concentration when processing lengthy sequences is
critical to context window extension, and scaling the attention logits can
serve as an inductive bias that facilitates this process. This motivates us to
combine “scaling the attention logits” with ABF during the supervised fine-
tuning stage. To leverage the pretraining results, we also propose the
avoidance of modifying the attention logits within the pre-trained context
window by setting a lower bound to $t$.

Our ultimate scaling factor $t$ is depicted below:

$t=\begin{cases}1,&\text{if layer index is 0 or 1}\\\
\max(\log_{c}{i},1),&\text{o.w.}\\\ \end{cases}$

##  4 Experiments

To analyze the real-world applicability of different RoPE-extension methods,
we test the long-context performance of models trained with these methods on
selected tasks from LongBench (Bai et al., 2023) and answer the three research
questions we propose in Section 1 by adjusting training data amount and
context window sizes. Finally, we also explore efficient data compositions and
training curricula on context window extension for given downstream tasks.

Table 2: Experiment results on selected tasks from LongBench. Model names with
a trailing asteroid are reported from the LongBench paper. We name our trained
models after their RoPE-extension methods.

Model | Singl-Doc QA | Multi-Doc QA | Summarization | Few-shot Learning | Macro
---|---|---|---|---|---
NQA | QAPR | MFQA_en | HPQA | WMQA | MSQ | GR | QMSM | MNWS | TREC | TRVQA | SMSM
Llama2-7B-chat-4k* | 18.7 | 19.2 | 36.8 | 25.4 | 32.8 | 9.4 | 27.3 | 20.8 | 25.8 | 61.5 | 77.8 | 40.7 | 33.0
LongChat-v1.5-7B-32k* | 16.9 | 27.7 | 41.4 | 31.5 | 20.6 | 9.7 | 30.8 | 22.7 | 26.4 | 63.5 | 82.3 | 34.2 | 34.0
Vicuna-v1.5-7B-16k* | 19.4 | 26.1 | 38.5 | 25.3 | 20.8 | 9.8 | 27.9 | 22.8 | 27.2 | 71.5 | 86.2 | 40.8 | 34.7
PI | 20.1 | 30.4 | 45.3 | 26.1 | 30.1 | 9.9 | 28.1 | 23.7 | 26.6 | 68.0 | 84.9 | 42.5 | 36.3
NTK-By-Parts | 15.9 | 31.1 | 40.1 | 25.4 | 26.6 | 7.2 | 26.7 | 22.4 | 26.9 | 68.5 | 82.8 | 42.9 | 34.7
Yarn | 20.3 | 28.9 | 42.8 | 27.8 | 30.7 | 7.2 | 27.4 | 22.5 | 26.8 | 66.0 | 85.6 | 42.6 | 35.7
ABF | 24.6 | 32.8 | 45.6 | 35.1 | 30.3 | 15.2 | 30.8 | 23.0 | 27.4 | 71.0 | 84.7 | 42.7 | 38.6
Ours | 21.9 | 31.0 | 47.1 | 40.1 | 32.7 | 15.1 | 32.3 | 23.0 | 27.1 | 70.5 | 86.7 | 42.0 | 39.1

###  4.1 General Setup

#### Model Variants

We use LLaMA-2-7B-Chat (Touvron et al., 2023b) given its popularity. We only
modify RoPE while leaving the model architecture unchanged.

#### Training

Previous works (Chen et al., 2023; Xiong et al., 2023; Peng et al., 2023)
adopt a similar training curriculum by first continual pre-training the LLaMA
base model to adapt to the modified position embeddings and then fine-tune on
target long-context downstream tasks. In contrast, we propose directly
supervised fine-tuning of the Chat Model to evaluate the practical
applicability of different RoPE-extension methods. We extend the context
window of LLaMA-2-7B-Chat to 16k with detailed training setups available in
Appendix A.

#### SFT Data

We curate a dataset of 3.5k lengthy conversations from
ShareGPT111https://huggingface.co/datasets/philschmid/sharegpt-raw (Chiang et
al., 2023). Following the data cleaning pipeline in (Zheng et al., 2023), we
kept English conversations only, excluded those with less than 10,000 tokens,
and split longer conversations so that we have a maximum sequence length of
16,384 tokens.

#### Evaluation

Existing works primarily assess the efficacy of RoPE-extension methods through
the examination of continual pre-trained models across language modeling tasks
and synthetic tasks. For example, YaRN (Chen et al., 2023) evaluates the
perplexity scores and model performance on the passkey-retrieval task
(Mohtashami and Jaggi, 2023) to quantify models’ long-context performance.
However, synthetic tasks like passkey retrieval deviate largely from real-
world scenarios while language modeling tasks have also proved a rudimentary
metric incapable of promising success in downstream tasks as suggested by (Pal
et al., 2023; Sun et al., 2021). In this work, we analyzed the long context
performance of models with extended context windows on selected tasks from
LongBench (Bai et al., 2023). Our evaluation includes 12 tasks from four
categories: single-document QA, multi-document QA, summarization, and few-shot
learning to ensure a comprehensive evaluation of models’ long-context
capabilities. We intentionally exclude synthetic tasks and code completion
tasks from LongBench because synthetic tasks deviate largely from real-world
scenarios, and code completion tasks have performance conflicts with general
instruction following abilities learned from ShareGPT conversations, as
suggested by (Dong et al., 2023).

###  4.2 Measuring Long-Context Performance

To answer the research question “(1) Which method exhibits the best supervised
fine-tuning performance on context-demanding downstream tasks?”, we fine-tune
LLaMA-7B-Chat on 3.5k lengthy conversations and evaluate their long-context
performance on LongBench.

Table 2 illustrates the performance of each method, with some results reported
from the LongBench paper (Bai et al., 2023). We highlight our major
observations here:

1) Fine-tuning the models on lengthy conversation data is efficient for
context window extension. Both LongChat-v1.5-7B-32k and Vicuna-v1.5-7B-16k are
open-source long-context models extended with PI (Chen et al., 2023) through
fine-tuning on large amounts of conversation data. For example,
LongChat-v1.5-7B-32 is finetuned on 80k conversations. By fine-tuning the
model on lengthy conversations only, our replicated PI-based model
outperformed the open-source versions, confirming the efficacy of fine-tuning
the model on lengthy conversations.

2) PI yields better long-context fine-tuning results than YaRN. While NTK-By-
Parts and YaRN have lower perplexity in language modeling tasks, PI has better
fine-tuning performance on long-context downstream tasks that are more related
to practical scenarios. This finding corroborates the conclusion by (Pal et
al., 2023; Sun et al., 2021) that language modeling perplexity is a
rudimentary metric incapable of promising success in downstream tasks. We
hypothesize that while YaRN’s scalar is efficient for language modeling tasks,
its constant nature might affect model performance on downstream tasks.

3) ABF-based models surpass the other methods by a significant margin. Both
ABF and our methods exhibit consistently superior fine-tuning performance on
all 12 long-context tasks, demonstrating the efficacy of adjusting RoPE’s base
frequency to a large number (e.g. 50,000).

###  4.3 Measuring Data Efficiency

Data efficiency is an essential characteristic of RoPE-extension methods in
context window extension practice, given both the sparsity of long training
data and the high cost of training on long sequences. In this section, we
explore the research question “(2) How can each method efficiently utilize
training data?” by training the model respectively on 32, 100, 1k, and 3.5k
conversations. The results are plotted in Figure 2, and the detailed results
for each task are in Table 5.

Figure 2: Long-Context Performance of RoPE-extending Methods with Different
Amounts of Training Data

We highlight our major observations below:

1) ABF-based methods consistently benefit from increasing training data.
While all RoPE-extension methods exhibit improved performance with increased
training data, the performance gain appears marginal for PI, NTK-By-Parts, and
Yarn when the data amount increases from 1K to 3.5K. Only ABF-based methods
consistently demonstrate performance gains.

2) Entropy-Aware ABF demonstrates extraordinary data efficiency. Notably, with
a mere 100 training samples and 6 training steps, our method achieves
competitive long-context performance that only lags marginally behind the ABF
method trained on 3.5K samples. Without considering the cost of finetuning on
downstream tasks, PI (Chen et al., 2023) continue pre-trains LLaMA-7B (Touvron
et al., 2023a) for 1,000 steps with 64 batch size, YaRN (Peng et al., 2023)
adopts 250 continual pre-training steps with the same batch size. Open source
practice like Longchat (Li* et al., 2023) utilizes 80k conversations from
ShareGPT for instruction tuning. Our work demonstrates the remarkable
efficiency of entropy-aware ABF in context window extension, requiring less
than 2% of the training resources utilized by existing methodologies.

We also observe that the performance gap from ABF to our method is diminishing
with the increase in training data. This phenomenon aligns with our hypothesis
in Section 3.2 that while the ability to maintain concentration across lengthy
inputs can be learned from training on more data, our method serves as an
inductive bias that facilitates the learning process.

###  4.4 Measuring Robustness across Context Windows

A desirable attribute for RoPE-extension methods, when applied in practical
context window extension settings, is that the models fine-tuned using these
methods should maintain their performance on the original context window,
while also demonstrating a certain degree of extrapolation capability beyond
the fine-tuned length.

To answer the research question “(3) Do models trained with these methods have
a robust performance across varying context window sizes?”, we follow
LongBench (Bai et al., 2023) to assess the models across different context
window sizes by truncating the prompt from the middle when the task length
exceeds a designated context window size.

The results are depicted in Figure 3. While there appears a performance gain
for PI, NTK-By-Parts, and Yarn when the context size is enlarged from 4k to
8k, their performance degrades when the context is further enlarged to 16k,
demonstrating their inability to leverage the full fine-tuning context window.
In contrast, ABF and our proposed method consistently gain from a larger
context window within fine-tuning length. Furthermore, entropy-aware ABF is
the only method that can maintain the performance when directly extrapolating
to 32k.

Figure 3: Long-Context Performance of RoPE-extending Methods with Different
Context Window Sizes

###  4.5 Exploring the Optimal Training Data and Curriculums

In this section, we explore efficient training data and curriculums for
context window extension on given tasks. An important consideration in
practice is whether long in-domain training samples are indispensable for
achieving success in context window extension for a particular downstream
task. Specifically, we inquire whether short in-domain training samples only
can still yield benefits in scenarios where lengthier samples are absent,
which is often the case. To answer the above questions, we conduct experiments
with various training curriculums on GovReport (Huang et al., 2021) which is a
widely used long context summarization task, and Longchat-Line-Retrieval (Li*
et al., 2023), a synthetic retrieval task.

We evaluate both long (more than 8,092 tokens) and short tasks (within 4,096
tokens) to guarantee models’ performance within the original context window
while evaluating their long-context performance. When the training data is in-
domain samples, we train the model for 4 epochs with a batch size of 8 and
evaluate with the best epoch on the validation set. When the training data is
1,000 ShareGPT conversations, the model is trained for two epochs with a batch
size of 32 and evaluated on the second epoch.

The results are displayed in Table 3. We conclude that training the model on
short in-domain samples produces suboptimal results, but starting from the
model finetuned on 1,000 ShareGPT conversations yields comparable results to
those finetuned on long in-domain samples, which suggests a good starting
point for context window extension in practice.

It might be strange that the line-retrieval task shows extremely poor
performance when finetuned from the Chat model on long samples. We attribute
it to the insufficient training of our method because the answer to the line
retrieval task is short, and we only calculate losses on the model response
tokens during the instruction tuning.

Initialization | training data | GR-S | GR-L | LR-S | LR-L
---|---|---|---|---|---
LLaMA 2 Chat | None | 30.84 | 0 | 76 | 0
LLaMA 2 Chat | Short | 37.91 | 33.6 | 74 | 26
LLaMA 2 Chat | Long | 38.24 | 36.45 | 10 | 2
Share1k | None | 34.10 | 31.14 | 88 | 48
Share1k | Short | 38.31 | 35.12 | 86 | 64
Share1k | Long | 38.93 | 35.56 | 92 | 66
Short | Share1k | 39.74 | 32.12 | 90 | 54

Table 3: Performance on two downstream tasks with different training
curriculums. GR-S: GovReport-Short. GR-L: GovReport-Long. LR-S: Line
Retrieval-Short. LR-L: LineRetrieval-Long. In the first column, Share1k means
the fine-tuned result of the 7B Chat model on 1,000 ShareGPT conversations.
Short means the fine-tuned result of the 7B chat model on short in-domain
samples. In the second column, None means the model is directly tested. Short
means short in-domain samples. Long means long in-domain samples.

##  5 Related Work

Extensive research has been done to enhance the long-context capacity of
transformer models (Vaswani et al., 2017) by overcoming two prominent
obstacles: the quadratic time and space complexity of the attention mechanism
(Vaswani et al., 2017) and the inability of position encodings to generalize
beyond the pre-trained context window.

#### More Efficient Transformers

The vanilla attention mechanism in the Transformer architecture is known for
its quadratic time and space complexity, which poses significant resource
demands for transformer models when processing lengthy inputs. Various works
have focused on conquering the complexity issue and proposing more efficient
Transformers. Sparse transformers (Child et al., 2019; Ye et al., 2019; Kitaev
et al., 2020; Beltagy et al., 2020; Ainslie et al., 2020; Zaheer et al., 2020;
Ding et al., 2023) replace the original full attention mechanism with a
sparsified version to make the computation more efficient. Linear transformers
(Wang et al., 2020; Katharopoulos et al., 2020; Choromanski et al., 2020),
rather than forcing the attention mechanism to attend to fewer tokens, propose
an alternative approach by leveraging low-rank matrix multiplication or linear
dot-product of kernel feature maps to approximate the original attention
mechanism, achieving linear time complexity. Meanwhile, retrieval-augmented
models (Guu et al., 2020; Lewis et al., 2020; Wu et al., 2022; Bulatov et al.,
2023; Tworkowski et al., 2023) integrate retrieval with attention. During
inference time, these models avoid directly modeling lengthy inputs by
retrieving information from an external memory that stores previous key-value
pairs. While prior research primarily focuses on reducing FLOPs, the
bottleneck of transformer inference on modern computing hardware has shifted
to the overhead from memory access (IO). Multi-query attention (MQA)(Shazeer,
2019) and grouped-query attention (GQA)(Ainslie et al., 2023), for instance,
address the memory-bandwidth cost associated with loading the large "keys" and
"values" tensors in the multi-head attention mechanism by proposing the use of
fewer "key" and "value" heads. Notably, GQA is employed in LLaMA2 Touvron et
al. (2023b). Additionally, FlashAttention (Dao et al., 2022; Dao, 2023)
introduces an IO-aware exact attention approach that utilizes tiling to reduce
memory IOs.

#### Generalizable Position Encoding

Due to the attention mechanism’s parallel nature, transformer models require
position encoding (PE) methods to facilitate the integration of position
information. The original transformer employed sinusoidal position encoding,
which constitutes an absolute PE and exhibits limited generalization
capability. Subsequently, this approach was refined to a learnable version
(Gehring et al., 2017), as embraced by language model architectures such as
GPT-3 (Brown et al., 2020). However, this adaptation completely compromises
the extrapolation ability of position encoding methods. The advent of relative
PE (Shaw et al., 2018) theoretically supports infinite input lengths.
Nevertheless, despite recent advancements in relative PEs, such as T5 relative
PE (Raffel et al., 2020), RoPE (Su et al., 2021), xPOS (Sun et al., 2022), and
ALiBi (Press et al., 2021), it has been demonstrated by (Kazemnejad et al.,
2023) that all these methods fail when extrapolating significantly beyond the
pre-trained context window.

##  6 Conclusions

In summary, through interpreting LLMs’ inherent need to maintain concentration
when processing lengthy sequences, we propose entropy-aware ABF by combining
ABF with a sophisticated applied scalar that scales the attention logits. Our
proposed method effectively extends the context window of RoPE-based LLMs,
addressing their limitations when confronted with context-demanding tasks at a
minimal cost. We empirically show the superiority of our method in both fine-
tuning results and robustness across different context window sizes on various
context-demanding tasks. Importantly, our method exhibits extraordinary data
efficiency compared to other methods, deriving a competent long-context model
on LongBench with only 100 samples and 6 training steps, less than 2% of the
training resources utilized by previous works. Finally, we provide valuable
insights into context window extension for specific downstream tasks,
suggesting training on lengthy ShareGPT conversations as a good starting
point.

## Acknowledgments

We want to thank Zhengbao Jiang for his participation in the initial
discussions. We thank Fan Nie and Fan Zhou for their invaluable advice
throughout the paper-writing process.

## References

  * Ainslie et al. (2023) Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023.  Gqa: Training generalized multi-query transformer models from multi-head checkpoints.  _arXiv preprint arXiv:2305.13245_.
  * Ainslie et al. (2020) Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020.  Etc: Encoding long and structured inputs in transformers.  _arXiv preprint arXiv:2004.08483_.
  * Anil et al. (2023) Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023.  Palm 2 technical report.  _arXiv preprint arXiv:2305.10403_.
  * Bai et al. (2023) Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023.  Longbench: A bilingual, multitask benchmark for long context understanding.  _arXiv preprint arXiv:2308.14508_.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_.
  * Black et al. (2022) Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022.  Gpt-neox-20b: An open-source autoregressive language model.  _arXiv preprint arXiv:2204.06745_.
  * bloc97 (2023a) bloc97. 2023a.  Add NTK-Aware interpolation "by parts" correction.
  * bloc97 (2023b) bloc97. 2023b.  NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Bulatov et al. (2023) Aydar Bulatov, Yuri Kuratov, and Mikhail S Burtsev. 2023.  Scaling transformer to 1m tokens and beyond with rmt.  _arXiv preprint arXiv:2304.11062_.
  * Chen et al. (2023) Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023.  Extending context window of large language models via positional interpolation.  _arXiv preprint arXiv:2306.15595_.
  * Chen et al. (2016) Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016.  Training deep nets with sublinear memory cost.  _arXiv preprint arXiv:1604.06174_.
  * Chiang and Cholak (2022) David Chiang and Peter Cholak. 2022.  Overcoming a theoretical limitation of self-attention.  _arXiv preprint arXiv:2202.12172_.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Child et al. (2019) Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019.  Generating long sequences with sparse transformers.  _arXiv preprint arXiv:1904.10509_.
  * Choromanski et al. (2020) Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. 2020.  Rethinking attention with performers.  _arXiv preprint arXiv:2009.14794_.
  * Chowdhery et al. (2023) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023.  Palm: Scaling language modeling with pathways.  _Journal of Machine Learning Research_ , 24(240):1–113.
  * Dao (2023) Tri Dao. 2023.  Flashattention-2: Faster attention with better parallelism and work partitioning.  _arXiv preprint arXiv:2307.08691_.
  * Dao et al. (2022) Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022.  Flashattention: Fast and memory-efficient exact attention with io-awareness.  _Advances in Neural Information Processing Systems_ , 35:16344–16359.
  * Ding et al. (2023) Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, and Furu Wei. 2023.  Longnet: Scaling transformers to 1,000,000,000 tokens.  _arXiv preprint arXiv:2307.02486_.
  * Dong et al. (2023) Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. 2023.  How abilities in large language models are affected by supervised fine-tuning data composition.  _arXiv preprint arXiv:2310.05492_.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020.  The pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_.
  * Gehring et al. (2017) Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017.  Convolutional sequence to sequence learning.  In _International conference on machine learning_ , pages 1243–1252. PMLR.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020.  Retrieval augmented language model pre-training.  In _International conference on machine learning_ , pages 3929–3938. PMLR.
  * Huang et al. (2021) Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021.  Efficient attentions for long document summarization.  _arXiv preprint arXiv:2104.02112_.
  * kaiokendev (2023) kaiokendev. 2023.  Things I’m learning while training superhot.
  * Katharopoulos et al. (2020) Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020.  Transformers are rnns: Fast autoregressive transformers with linear attention.  In _International conference on machine learning_ , pages 5156–5165. PMLR.
  * Kazemnejad et al. (2023) Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. 2023.  The impact of positional encoding on length generalization in transformers.  _arXiv preprint arXiv:2305.19466_.
  * Kitaev et al. (2020) Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020.  Reformer: The efficient transformer.  _arXiv preprint arXiv:2001.04451_.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474.
  * Li* et al. (2023) Dacheng Li*, Rulin Shao*, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. 2023.  How long can open-source llms truly promise on context length?
  * Liu et al. (2023) Tianyang Liu, Canwen Xu, and Julian McAuley. 2023.  Repobench: Benchmarking repository-level code auto-completion systems.  _arXiv preprint arXiv:2306.03091_.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _Learning,Learning_.
  * Mohtashami and Jaggi (2023) Amirkeivan Mohtashami and Martin Jaggi. 2023.  Landmark attention: Random-access infinite context length for transformers.  _arXiv preprint arXiv:2305.16300_.
  * Pal et al. (2023) Arka Pal, Deep Karkhanis, Manley Roberts, Samuel Dooley, Arvind Sundararajan, and Siddartha Naidu. 2023.  Giraffe: Adventures in expanding context lengths in llms.  _arXiv preprint arXiv:2308.10882_.
  * Peng et al. (2023) Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023.  Yarn: Efficient context window extension of large language models.  _arXiv preprint arXiv:2309.00071_.
  * Press et al. (2021) Ofir Press, Noah A Smith, and Mike Lewis. 2021.  Train short, test long: Attention with linear biases enables input length extrapolation.  _arXiv preprint arXiv:2108.12409_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551.
  * Rajbhandari et al. (2020) Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020.  Zero: Memory optimizations toward training trillion parameter models.  In _SC20: International Conference for High Performance Computing, Networking, Storage and Analysis_ , pages 1–16. IEEE.
  * Rasley et al. (2020) Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020.  Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 3505–3506.
  * Ren et al. (2021) Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021.  $\\{$ZeRO-Offload$\\}$: Democratizing $\\{$Billion-Scale$\\}$ model training.  In _2021 USENIX Annual Technical Conference (USENIX ATC 21)_ , pages 551–564.
  * Shaw et al. (2018) Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018.  Self-attention with relative position representations.  _arXiv preprint arXiv:1803.02155_.
  * Shazeer (2019) Noam Shazeer. 2019.  Fast transformer decoding: One write-head is all you need.  _arXiv preprint arXiv:1911.02150_.
  * Su (2023) Jianlin Su. 2023.  Rectified rotary position embeddings.  https://github.com/bojone/rerope.
  * Su et al. (2021) Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2021.  Roformer: Enhanced transformer with rotary position embedding.  _arXiv preprint arXiv:2104.09864_.
  * Sun et al. (2021) Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke, and Mohit Iyyer. 2021.  Do long-range language models actually use long-range context?  _arXiv: Computation and Language,arXiv: Computation and Language_.
  * Sun et al. (2022) Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. 2022.  A length-extrapolatable transformer.  _arXiv preprint arXiv:2212.10554_.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Tworkowski et al. (2023) Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miłoś. 2023.  Focused transformer: Contrastive training for context scaling.  _arXiv preprint arXiv:2307.03170_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2020) Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. 2020.  Linformer: Self-attention with linear complexity.  _arXiv preprint arXiv:2006.04768_.
  * Wu et al. (2022) Yuhuai Wu, Markus N Rabe, DeLesley Hutchins, and Christian Szegedy. 2022.  Memorizing transformers.  _arXiv preprint arXiv:2203.08913_.
  * Xiong et al. (2023) Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. 2023.  Effective long-context scaling of foundation models.  _arXiv preprint arXiv:2309.16039_.
  * Ye et al. (2019) Zihao Ye, Qipeng Guo, Quan Gan, Xipeng Qiu, and Zheng Zhang. 2019.  Bp-transformer: Modelling long-range context via binary partitioning.  _arXiv: Computation and Language,arXiv: Computation and Language_.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020.  Big bird: Transformers for longer sequences.  _Advances in neural information processing systems_ , 33:17283–17297.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023.  Judging llm-as-a-judge with mt-bench and chatbot arena.

##  Appendix A Training Details

The model is trained on 4 NVIDIA A100 GPUs with DeepSpeed (Rasley et al.,
2020), ZeRO (Rajbhandari et al., 2020; Ren et al., 2021) Stage 3, gradient-
checkpointing (Chen et al., 2016), and FlashAttention (Dao et al., 2022; Dao,
2023). We also use BF16 and TF32 mix computation precision for further
acceleration.

All the models are fine-tuned using AdamW Optimizer (Loshchilov and Hutter,
2017) with $\beta_{1}=0.9$ and $\beta_{2}=0.95$ for two epochs, computing
losses on response tokens only. We use a cosine learning rate scheduler, set
the peak learning rate to 2e-5, and weight decay to 0.1. For training on 3.5k
conversations, we use a batch size of 128 and 10 warmup steps. We use a batch
size of 32 and 0 warmup steps for fewer training data. If not explicitly
stated, we default to using 3.5k ShareGPT conversations for instruction
tuning.

##  Appendix B Additional Experiment Results

Model | Singl-Doc QA | Multi-Doc QA | Summarization | Few-shot Learning | Macro
---|---|---|---|---|---
NQA | QAPR | MFQA_en | HPQA | WMQA | MSQ | GR | QMSM | MNWS | TREC | TRVQA | SMSM
PI-4K | 22.3 | 27.3 | 44.6 | 31.7 | 30.8 | 9.2 | 29.3 | 21.5 | 27.1 | 61.5 | 87.5 | 41.4 | 36.2
PI-8K | 21.4 | 28.6 | 46.8 | 31.1 | 29 | 11.7 | 30.1 | 22.5 | 27.1 | 66 | 86.8 | 42.2 | 36.9
PI-16K | 20.1 | 30.4 | 45.3 | 26.1 | 30.1 | 9.9 | 28.1 | 23.7 | 26.6 | 68 | 84.9 | 42.5 | 36.3
PI-32K | 7 | 27.6 | 43.7 | 16.3 | 25.1 | 1.5 | 23.5 | 14.2 | 27 | 67.5 | 54.4 | 33.1 | 28.4
NTK-By-Parts-4k | 22.8 | 27.3 | 42.5 | 26.5 | 23 | 10.1 | 28.7 | 21.8 | 27.1 | 63 | 87.3 | 42 | 35.2
NTK-By-Parts-8k | 20.9 | 31.2 | 43.5 | 28.3 | 29.4 | 10.9 | 29.4 | 22.3 | 27 | 65 | 87.5 | 43.5 | 36.6
NTK-By-Parts-16k | 15.9 | 31.1 | 40.1 | 25.4 | 26.6 | 7.2 | 26.7 | 22.4 | 26.9 | 68.5 | 82.8 | 42.9 | 34.7
NTK-By-Parts-32k | 6.5 | 30.8 | 39.1 | 15.9 | 26.2 | 1.1 | 23.3 | 14.9 | 26.9 | 68.5 | 54.4 | 34.7 | 28.5
Yarn-4K | 21 | 27.9 | 43.8 | 29 | 29.5 | 12.8 | 29.3 | 22.1 | 26.9 | 61.5 | 87.1 | 42.8 | 36.1
Yarn-8K | 20.9 | 31.2 | 43.5 | 28.3 | 29.4 | 10.9 | 29.4 | 22.3 | 27 | 65 | 87.5 | 43.5 | 36.6
Yarn-16K | 20.3 | 28.9 | 42.8 | 27.8 | 30.7 | 7.2 | 27.4 | 22.5 | 26.8 | 66 | 85.6 | 42.6 | 35.7
Yarn-32K | 6.5 | 29.3 | 39 | 16 | 29.1 | 1.3 | 24 | 14.4 | 26.9 | 66.5 | 55.1 | 34.6 | 28.6
ABF-4K | 19.5 | 30.6 | 44.1 | 31.1 | 29.5 | 11.5 | 29 | 21.4 | 27.6 | 64.5 | 87.6 | 41.9 | 36.5
ABF-8K | 22.7 | 32.1 | 45.2 | 35.5 | 29.2 | 14.8 | 30.6 | 22.8 | 27.4 | 67 | 84.8 | 42.4 | 37.9
ABF-16K | 24.6 | 32.8 | 45.6 | 35.1 | 30.3 | 15.2 | 30.8 | 23 | 27.4 | 71 | 84.7 | 42.7 | 38.6
ABF-32K | 23.6 | 27.1 | 44.8 | 36.8 | 27.5 | 12.3 | 29.4 | 23.1 | 26.5 | 72 | 84.9 | 43.5 | 37.6
Ours-4K | 21.1 | 30.1 | 43.3 | 28.1 | 31.6 | 11.1 | 28.7 | 21.6 | 27.6 | 64 | 86.8 | 42.1 | 36.3
Ours-8K | 23.3 | 31.7 | 45.5 | 33.2 | 31.7 | 14.6 | 30.7 | 23 | 27 | 67.5 | 86.3 | 42.4 | 38.1
Ours-16K | 21.9 | 31 | 47.1 | 40.1 | 32.7 | 15.1 | 32.3 | 23 | 27.1 | 70.5 | 86.7 | 42 | 39.1
Ours-32K | 23.6 | 31.8 | 45.5 | 39 | 31.7 | 16.6 | 31.7 | 23.6 | 27.1 | 70.5 | 86 | 42.4 | 39.1

Table 4: Long-Context performance of RoPE-extension Methods with different
context window sizes

Model | Singl-Doc QA | Multi-Doc QA | Summarization | Few-shot Learning | Macro
---|---|---|---|---|---
NQA | QAPR | MFQA_en | HPQA | WMQA | MSQ | GR | QMSM | MNWS | TREC | TRVQA | SMSM
PI-32 | 10.7 | 15.2 | 30.5 | 18.5 | 21.6 | 7.3 | 31 | 21.2 | 27.5 | 60 | 73.2 | 39.4 | 29.7
PI-100 | 7.2 | 31.4 | 37.1 | 30.9 | 33.4 | 11.6 | 30.5 | 16 | 27.1 | 64.5 | 78.4 | 38 | 33.8
PI-1000 | 20.4 | 31.1 | 39 | 30.7 | 30.5 | 10.4 | 27.9 | 23.5 | 26.6 | 67 | 84.5 | 41.6 | 36.1
PI-3500 | 20.1 | 30.4 | 45.3 | 26.1 | 30.1 | 9.9 | 28.1 | 23.7 | 26.6 | 68 | 84.9 | 42.5 | 36.3
NTK-By-Parts-32 | 4.5 | 22.6 | 31.9 | 10.9 | 23.7 | 0.8 | 20.7 | 13.2 | 26.3 | 65 | 41.3 | 33.4 | 24.5
NTK-By-Parts-100 | 7.9 | 28.6 | 40.1 | 19.1 | 26.9 | 7 | 24.7 | 18 | 26.1 | 66.5 | 77.3 | 40 | 31.9
NTK-By-Parts-1000 | 15.9 | 28.3 | 42.7 | 23.6 | 26.5 | 6.5 | 26.4 | 22.7 | 26.7 | 68.5 | 83.7 | 42.1 | 34.5
NTK-By-Parts-3500 | 15.9 | 31.1 | 40.1 | 25.4 | 26.6 | 7.2 | 26.7 | 22.4 | 26.9 | 68.5 | 82.8 | 42.9 | 34.7
Yarn-32 | 5 | 18.9 | 35.1 | 12 | 26 | 0.9 | 23.6 | 14.1 | 26.2 | 63.5 | 44.1 | 30.4 | 25
Yarn-100 | 6.6 | 30.1 | 39.2 | 17.7 | 27.1 | 2.8 | 24.4 | 16.7 | 25.8 | 66.5 | 76 | 37.9 | 30.9
Yarn-1000 | 18 | 28.2 | 42.9 | 26.7 | 28.4 | 9.3 | 27.6 | 22.4 | 26.9 | 65 | 85.1 | 41.9 | 35.2
Yarn-3500 | 19.7 | 25.4 | 44.6 | 29.3 | 25.9 | 9.5 | 26.5 | 22.2 | 26.7 | 67.5 | 85.6 | 43.7 | 35.5
ABF-32 | 10.9 | 15 | 32.2 | 20.3 | 21.6 | 7.5 | 28.3 | 21.2 | 26.9 | 56 | 41.6 | 35.2 | 26.4
ABF-100 | 18.8 | 27.6 | 41.2 | 30.9 | 35 | 10.2 | 31.3 | 22 | 27.2 | 66.5 | 73.8 | 36.7 | 35.1
ABF-1000 | 23.9 | 33 | 44.9 | 32 | 20.1 | 12.2 | 31.1 | 23.9 | 27.5 | 71 | 85.6 | 40.7 | 37.2
ABF-3500 | 24 | 30.5 | 45.8 | 37.9 | 30.7 | 15.4 | 31.4 | 23.3 | 27.2 | 70 | 84.2 | 42.6 | 38.6
Ours-32 | 14.8 | 15.6 | 36.4 | 29.6 | 25.9 | 12.9 | 32.2 | 21.4 | 26.9 | 55 | 67.3 | 38.1 | 31.3
Ours-100 | 20.6 | 26.4 | 45.9 | 37.7 | 35.6 | 16.4 | 32.2 | 22 | 26.9 | 67.5 | 80.2 | 37.3 | 37.4
Ours-1000 | 23.5 | 33 | 45 | 34.3 | 24.8 | 16.4 | 30.9 | 23.8 | 27.9 | 71 | 87.7 | 41 | 38.3
Ours-3500 | 21.9 | 31 | 47.1 | 40.1 | 32.7 | 15.1 | 32.3 | 23 | 27.1 | 70.5 | 86.7 | 42 | 39.1

Table 5: Long-context performance of RoPE-extension methods with different
amounts of training data


B(x,80r)$ and the denominators $\left|\overline{W}(y,r)\right|$ and
$\left|B(x,80r)\right|$ are comparable. We are left with the integral
$\displaystyle J=\int_{\overline{W}(y,r)\cap V}|\widetilde{A}-A_{0}|^{2}$. Let
$I(y,r)$ be the collection of indices such that $Q_{i}$ meets
$\overline{W}(y,r)\cap V$; then $J\leq\sum_{i\in
I(y,r)}\big{|}\widetilde{A}-A_{0}\big{|}^{2}dX$, and the definition (5.65) of
$\widetilde{A}$ yields

$J\leq\sum_{i\in
I(y,r)}\int_{Q_{i}}\Big{|}\sum_{j}A_{j}\,\chi_{j}(X)-A_{0}\Big{|}^{2}dX=\sum_{i\in
I(y,r)}\int_{Q_{i}}\Big{|}\sum_{j}(A_{j}-A_{0})\,\chi_{j}(X)\Big{|}^{2}dX$

because $\sum_{j}\chi_{j}(X)=1$. Observe that for $X\in Q_{i}$, we only sum
over the set $J_{i}$ of those $j\in I$ such that $\frac{6}{5}Q_{j}$ contains
$X$ and hence meets $Q_{i}$. There are only finitely many of such indices $j$,
and they are all such that
$\frac{1}{4}\operatorname{diam}(Q_{j})\leq\operatorname{diam}(Q_{i})\leq
4\operatorname{diam}(Q_{j})$. We use Minkowski’s inequality to get that

$\Big{(}\int_{Q_{i}}\Big{|}\sum_{j}(A_{j}-A_{0})\chi_{j}(X)\Big{|}^{2}dX\Big{)}^{1/2}\leq\sum_{j\in
J_{i}}\left(\int_{Q_{i}}\left|(A_{j}-A_{0})\chi_{j}(X)\right|^{2}dX\right)^{1/2}.$

Since $\chi_{j}\leq 1$ and by definition of $A_{j}$, the right-hand side is at
most

$\sum_{j\in J_{i}}\left|Q_{i}\right|^{1/2}\left|A_{j}-A_{0}\right|\leq
C\sum_{j\in
J_{i}}\bigg{(}\frac{|Q_{j}|}{|W_{j}|}\int_{W_{j}}\left|A(Z)-A_{0}\right|^{2}dZ\bigg{)}^{1/2},$

where in the last inequality we have used the fact that
$|Q_{j}|\approx|Q_{i}|$ when $\frac{6}{5}Q_{j}$ meets $Q_{i}$. By construction
$|W_{j}|\approx|Q_{j}|$ for $j\in I_{1}\cup I_{2}$. Recall from (5.76) that
$\operatorname{dist}(y,\Gamma\cap 2B)\leq 2r\leq 12Kr_{0}$, hence for $j\in
I_{i}$ where $i\in I_{3}\cap I(y,r)$, $\operatorname{diam}(Q_{j})\approx
CKr_{0}$. Therefore $|W_{j}|\approx|Q_{j}|$ holds for all $j\in I_{i}$, $i\in
I(y,r)$, and so we may drop $\frac{|Q_{j}|}{|W_{j}|}$. Altogether

$J\leq C\sum_{i\in I(y,r)}\bigg{\\{}\sum_{j\in
J_{i}}\Big{(}\int_{W_{j}}\left|A(Z)-A_{0}\right|^{2}dZ\Big{)}^{1/2}\bigg{\\}}^{2}\\\
\leq C\sum_{i\in I(y,r)}\sum_{j\in
J_{i}}\int_{W_{j}}\left|A(Z)-A_{0}\right|^{2}dZ,$ (5.79)

where the last inequality follows from Jensen’s inequality. We claim that

$W_{j}\subset B(x,80r)\ \text{ for $j\in J_{i}$}.$ (5.80)

To see this, let $Z\in W_{j}$ be given; by (5.70),
$\operatorname{dist}(Z,Q_{j})\leq 3d_{j}$. But
$d_{j}=\operatorname{dist}(Q_{j},\Omega\cap 2B)\leq
20\operatorname{diam}(Q_{j})$ by definition of our Whitney cubes, so
$\operatorname{dist}(Z,Q_{j})\leq 60\operatorname{diam}(Q_{j})$. Next
$\frac{6}{5}Q_{j}$ meets $Q_{i}$, so $\operatorname{dist}(Z,Q_{i})\leq
62\operatorname{diam}(Q_{j})\leq 244\operatorname{diam}(Q_{i})$. Pick $X\in
Q_{i}\cap\overline{W}(y,r)$; it exists because $i\in I(y,r)$; then
$|X-x|\leq|X-y|+|y-x|\leq 5r$, and so $|Z-x|\leq
244\operatorname{diam}(Q_{i})+5r$. But the definition of our cubes yields
$\operatorname{diam}(Q_{i})\leq 10^{-1}\operatorname{dist}(Q_{i},\Omega\cap
2B)$, and $\operatorname{dist}(Q_{i},\Omega\cap
2B)\leq\operatorname{dist}(Q_{i},\Gamma\cap 2B)\leq 3r$ by (5.76). So
$244\operatorname{diam}(Q_{i})\leq 244\frac{3r}{10}<75r$, and (5.80) follows.

Return to (5.79); by (5.80) and the fact that the $W_{j}$ have finite overlap,

$J\leq C\int_{B(x,80r)\cap\Omega}\left|A(Z)-A_{0}\right|^{2}dZ\leq
Cr^{d+1}\gamma_{A}(x,80r)^{2},$

which completes the proof of our claim (5.77).

Let us now use the claim to do the part of our Carleson measure estimate that
comes from

$\Xi=\big{\\{}(y,r)\in G\times(0,+\infty)\,;\,\operatorname{dist}(y,\Gamma\cap
2B)\leq 2r\big{\\}}.$

We need to know that the set $\Gamma\cap B(y,r)$ of the claim is not too
small. We use (5.76) to pick $y^{\ast}\in\Gamma\cap 2B$ such that
$|y-y^{\ast}|\leq\operatorname{dist}(y,\Gamma\cap 2B)\leq 2r$; then by (5.5)
for balls centered at $y^{\ast}$ and since $B(y^{\ast},2r)\subset
B(y,4r)\subset B(y^{\ast},6r)$,

$c_{d}r^{d}\leq(1-C\varepsilon^{1/d})c_{d}(2r)^{d}\leq\mathcal{H}^{d}(B(y,4r)\cap\Gamma)\leq(1+C\varepsilon^{1/d})c_{d}(6r)^{d}\leq
7^{d}c_{d}r^{d}.$ (5.81)

Let $y_{0}\in G$ and $s_{0}>0$ be given, and estimate

$J(\Xi)=\int_{y\in G\cap B(y_{0},s_{0})}\int_{0<r\leq
s_{0}}\,\mathbbm{1}_{\Xi}(y,r)\,\,\overline{\alpha}(y,r)^{2}\frac{dr}{r}d\mathcal{H}^{d}(y).$

Denote $\min\left\\{s_{0},6Kr_{0}\right\\}$ by $\widehat{s_{0}}$. We use
(5.73), (5.77), (5.81), the Ahlfors regularity of $\Gamma$ and $G$, and a
Fubini argument, to get that

$J(\Xi)\leq C\int_{y\in G\cap
B(y_{0},s_{0})}\int_{0<r\leq\widehat{s_{0}}}\Big{\\{}r^{-d}\int_{x\in\Gamma\cap
B(y,4r)}\gamma_{A}(x,80r)^{2}d\mathcal{H}^{d}(x)\Big{\\}}\frac{dr}{r}d\mathcal{H}^{d}(y)\\\
\leq C\int_{x\in\Gamma\cap
B(y_{0},5s_{0})}\int_{0<r\leq\widehat{s_{0}}}\gamma_{A}(x,80r)^{2}\frac{dr}{r}d\mathcal{H}^{d}(x)\\\
=C\int_{x\in\Gamma\cap B(y_{0},5s_{0})}\int_{0<r\leq
80s_{0}}\gamma_{A}(x,r)^{2}\frac{dr}{r}d\mathcal{H}^{d}(x).$ (5.82)

In [DLM21], we proved that the weak DKP condition implies the stronger
estimate that $\gamma(x,r)^{2}\frac{dxdr}{r}$ is a Carleson measure in
$\mathbb{R}^{d}\times(0,\infty)$. The same thing holds here, with almost the
same proof, even though the geometry is a bit different because we work with
$\Gamma$ and $\Omega$. That is, for any $x_{0}\in\Gamma$ and $r_{0}>0$, we
have that

$\left\|\gamma_{A}(x,r)^{2}\frac{d\mathcal{H}^{d}(x)dr}{r}\right\|_{\mathcal{C}(x_{0},r_{0})}\leq
C\left\|\alpha_{A}(x,r)^{2}\frac{d\mathcal{H}^{d}(x)dr}{r}\right\|_{\mathcal{C}(x_{0},r_{0})}.$

Therefore, (5.82) now implies that

$J(\Xi)\leq Cs_{0}^{d}\mathfrak{N}(A)=Cs_{0}^{d}\varepsilon$ (5.83)

and concludes this part of the argument.

Return to our main Carleson estimate. We are left with the pairs $(y,r)$ that
belong to the set

$\Xi^{\prime}=\big{\\{}(y,r)\in
G\times(0,+\infty)\,;\,\operatorname{dist}(y,\Gamma\cap 2B)>2r\big{\\}}.$
(5.84)

Let $(y,r)\in\Xi^{\prime}$ be given, and assume additionally that
$\operatorname{dist}(y,\Gamma\cap 2B)\leq 6Kr_{0}$. Set
$d(y)=\operatorname{dist}(y,\Gamma\cap 2B)$, then $2r<d(y)\leq 6Kr_{0}$.
Obviously

$\operatorname{dist}(\overline{W}(y,r),\Gamma\cap
2B)\geq\operatorname{dist}(B(y,r),\Gamma\cap 2B)>d(y)-r\geq d(y)/2.$ (5.85)

Let $Q_{i}$ be any of the Whitney cubes such that $\frac{6}{5}Q_{i}$ meets
$\overline{W}(y,r)$, and pick $X\in\overline{W}(y,r)\cap\frac{6}{5}Q_{i}$.
Then by (5.85) $d(y)/2\leq\operatorname{dist}(X,\Gamma\cap
2B)\leq\operatorname{dist}(Q_{i},\Gamma\cap
2B)+\frac{6}{5}\operatorname{diam}(Q_{i})\leq 12\operatorname{diam}(Q_{i})$,
so $\operatorname{diam}(Q_{i})\geq d(y)/24$. Next pick any
$X\in\overline{W}(y,r)$ and any constant matrix $A_{0}$, and observe that by
(5.65)

$\nabla\widetilde{A}(X)=\sum_{i\in I}A_{i}\,\nabla\chi_{i}(X)=\sum_{i\in
I}[A_{i}-A_{0}]\,\nabla\chi_{i}(X)$ (5.86)

because $\sum\nabla\chi_{i}=0$. We sum only over the set $I(X)$ of indices $i$
such that $X\in\frac{6}{5}Q_{i}$, and since
$\left|\nabla\chi_{i}(X)\right|\leq C\operatorname{diam}(Q_{i})^{-1}\leq
Cd(y)^{-1}$ for $i\in I(X)$, we get that

$|\nabla\widetilde{A}(X)|\leq Cd(y)^{-1}\sum_{i\in I(X)}|A_{i}-A_{0}|.$ (5.87)

Recall that $A_{i}$ is the average of $A$ on $W_{i}$; we want to compare
$A_{i}$ to some average of $A$ on a large ball centered on $\Gamma$, so we
choose $y^{\ast}\in\Gamma\cap\overline{2B}$ such that $|y^{\ast}-y|\leq d(y)$
and we check that

$W_{i}\subset B(y^{\ast},12d(y)).$ (5.88)

Indeed, for $Z\in W_{i}$, (5.70) says that $\operatorname{dist}(Z,Q_{i})\leq
3\operatorname{dist}(Q_{i},\Omega\cap 2B)\leq 60\operatorname{diam}(Q_{i})$,
and all points of $Q_{i}$ lie within $2\operatorname{diam}(Q_{i})$ from
$\overline{W}(y,r)$ (because $\frac{6}{5}Q_{i}$ meets $\overline{W}(y,r)$)
hence within $2\operatorname{diam}(Q_{i})+r$ from $y$. So
$|Z-y^{\ast}|\leq|Z-y|+d(y)\leq 60\operatorname{diam}(Q_{i})+3d(y)/2$.

Now $\frac{6}{5}Q_{i}$ meets $\overline{W}(y,r)$, and if
$X\in\overline{W}(y,r)\cap\frac{6}{5}Q_{i}$ we both have that
$\operatorname{dist}(X,\Gamma\cap 2B)\leq d(y)+r\leq 3d(y)/2$ by (5.84), and
$\operatorname{dist}(X,\Gamma\cap
2B)\geq\operatorname{dist}(\frac{6}{5}Q_{i},\Gamma\cap
2B)\geq\operatorname{dist}(Q_{i},\Gamma\cap
2B)-\frac{1}{5}\operatorname{diam}(Q_{i})\geq 9\operatorname{diam}(Q_{i})$, so
$60\operatorname{diam}(Q_{i})\leq 10d(y)$ and (5.88) follows. Therefore,
choosing $A_{0}=\fint_{\Omega\cap B(y^{\ast},12d(y))}A(Y)dY$ (which does not
depend on $i\in I(X)$, and not even on $X$),

$|A_{i}-A_{0}|\leq|W_{i}|^{-1}\int_{W_{i}}|A(Y)-A_{0}|dY\leq|W_{i}|^{-1}\int_{B(y^{\ast},12d(y))}|A(Y)-A_{0}|dY\\\
\leq
C|W_{i}|^{-1}|B(y^{\ast},12d(y))|\fint_{B(y^{\ast},12d(y))}|A(Y)-A_{0}|dY.$

Observe that $|W_{i}|^{-1}|B(y^{\ast},12d(y))|\leq C$. For $i\in I_{1}\cup
I_{2}$, this follows from $\operatorname{diam}(Q_{i})\geq d(y)/24$. For $i\in
I_{3}$, this is because $d(y)\leq 6Kr_{0}$. Hence,

$|A_{i}-A_{0}|\leq Cd(y)^{-1}\gamma_{A}(y^{\ast},12d(y))$ (5.89)

by Cauchy-Schwarz. We notice that all the $W_{i}$, $i\in I(Z)$, stay at
distance at least $C^{-1}\operatorname{diam}(Q_{i})\geq C^{-1}d(y)$ from
$\Gamma$, so we can replace $\gamma(y^{\ast},12d(y))$ with
$\widehat{\alpha}(y^{\ast},d(y))$, the larger variant of $\alpha(x,r)$,
defined by

$\widehat{\alpha}(x,r)=\Big{\\{}\inf_{A_{0}}|\widehat{W}(x,r)|^{-1}\int_{\widehat{W}(x,r)}|A-A_{0}|^{2}\Big{\\}}^{1/2},$

with $\widehat{W}(x,r)=\big{\\{}X\in\Omega\cap
B(x,20r)\,;\,\operatorname{dist}(X,\Gamma)\geq C^{-1}r\big{\\}}$. Altogether,

$|\nabla\widetilde{A}(X)|\leq Cd(y)^{-1}\sum_{i\in I(X)}|A_{i}-A_{0}|\leq
Cd(y)^{-1}\widehat{\alpha}(y^{\ast},d(y)).$ (5.90)

Because of this, the oscillation of $\widetilde{A}(X)$ on $\overline{W}(y,r)$
is at most $Cd(y)^{-1}r\widehat{\alpha}(y^{\ast},d(y))$, and hence
$\overline{\alpha}(y,r)\leq Cd(y)^{-1}r\widehat{\alpha}(y^{\ast},d(y))$. By
this and (5.74),

$J(\Xi^{\prime}):=\int_{y\in G\cap B(y_{0},s_{0})}\int_{0<r\leq
s_{0}}\,\mathbbm{1}_{\Xi^{\prime}}(y,r)\,\mathbbm{1}_{d(y)\leq
6Kr_{0}}(y)\,\,\overline{\alpha}(y,r)^{2}\frac{dr}{r}d\mathcal{H}^{d}(y)\\\
\leq C\int_{y\in G\cap B(y_{0},s_{0})}\int_{0<r\leq
d(y)/2}\widehat{\alpha}(y^{\ast},d(y))\,d(y)^{-2}rdr\,d\mathcal{H}^{d}(y)\\\
\leq C\int_{y\in G\cap
B(y_{0},s_{0})}\widehat{\alpha}(y^{\ast},d(y))^{2}\,d\mathcal{H}^{d}(y)\\\
\leq Cs_{0}^{d}\sup_{y\in G\cap
B(y_{0},s_{0})}\widehat{\alpha}(y^{\ast},d(y))^{2}.$ (5.91)

Now the weak DKP condition implies that $\widehat{\alpha}(z,r)^{2}\leq
C\mathfrak{N}(A)\leq C\varepsilon$ for all $z\in\Gamma$ and $r>0$, where the
Carleson norm $\mathfrak{N}(A)$ is as in (1.11). The verification is also easy
(and is done in [DLM21]). Thus we also get that $J(\Xi^{\prime})\leq
Cs_{0}^{d}\varepsilon$, and since (5.83) provides the estimate for the other
piece $J(\Xi^{\prime})$, we finally get that the extension $\widetilde{A}$
satisfies the weak DKP condition on $G\times(0,\infty)$, with a norm at most
$C\varepsilon$.

##  6\. Small and vanishing $A_{\infty}$ and BMO

In this section we check some of the relations between $A_{\infty}$ weights
and $BMO$ estimates for their logarithm. We want to do this in
$(\Gamma,d\sigma)$, where either $\Gamma\subset\mathbb{R}^{d+1}$ is a chord-
arc surface with small constant and $\sigma=\mathcal{H}^{d}_{|\Gamma}$, or
$\Gamma=\mathbb{R}^{d}$ and $\sigma$ is the Lebesgue measure on $\Gamma$.
Because of the first case, we define BMO with balls.

We first define the mean oscillation on a set $E$ such that
$0<\sigma(E)<+\infty$, of the function $f\in L^{1}(E,d\sigma)$ by

$mo(f,E)=\sigma(E)^{-1}\int_{E}\bigg{|}f(y)-\Big{(}\fint_{E}fd\sigma\Big{)}\bigg{|}d\sigma(y).$
(6.1)

We say that $f\in BMO(\Gamma)$ when $f\in L^{1}_{loc}(\Gamma,d\sigma)$ and

$\|f\|_{BMO}:=\sup_{r>0}\sup_{x\in\Gamma}\,mo(f,\Delta(x,r))<+\infty,$ (6.2)

and that $f\in VMO(\Gamma)$ when $f\in BMO(\Gamma)$ and in addition

$\lim_{r\to 0_{+}}\sup_{x\in\Gamma}\,mo(f,\Delta(x,r))=0.$ (6.3)

It is easy to see that in the case of $\mathbb{R}^{d}$, we can replace the
collection of balls $\Delta(x,r)$ with the collection of cubes $Q$ with faces
parallel to the axes, and get the same space BMO, with a slightly different
but equivalent semi-norm

$\|f\|_{\ast}:=\sup_{Q}\,mo(f,Q);$ (6.4)

also $VMO(\Gamma)$ could be defined with cubes. Finally, we can localize: if
$\Delta=\Delta(x_{0},r_{0})$ is a surface ball in $\Gamma$, we say that $f\in
BMO(\Delta)$ when $f\in L^{1}(\Delta,d\sigma)$ and

$\|f\|_{BMO(\Delta)}:=\sup_{(x,r)\in\Delta\times(0,r_{0}),\Delta(x,r)\subset\Delta}\,mo(f,\Delta(x,r))<+\infty.$
(6.5)

This may be a little ugly near the boundary, but often we only care about the
restriction of $f$ to $\Delta(x_{0},r_{0}/2)$ anyway. In $\mathbb{R}^{d}$, the
version with cubes is a little better. If $Q_{0}\subset\mathbb{R}^{d}$ is a
cube (with faces parallel to the axes, we won’t repeat), we say that $f\in
BMO_{\ast}(Q_{0})$ when $f\in L^{1}(Q_{0},d\sigma)$ and

$\|f\|_{BMO_{\ast}(Q_{0})}:=\sup_{Q\subset Q_{0}}\,mo(f,Q)<+\infty,$ (6.6)

where the supremum is over the cubes $Q\subset Q_{0}$.

With all these definitions at hand, we are ready to prove Lemma 3.49. We start
with the simplest version of the John and Nirenberg theorem, with cubes in
$\mathbb{R}^{d}$.

######  Lemma 6.7.

There exist constants $c=c_{d}>0$ and $C=C_{d}\geq 1$, that depend only on
$d$, such that if $Q_{0}$ is a cube and $f\in BMO_{\ast}(Q_{0})$, then

$\int_{Q_{0}}\exp\bigg{(}c\|f\|_{BMO_{\ast}(Q_{0})}^{-1}\bigg{|}f(x)-\Big{(}\fint_{Q_{0}}fd\sigma\Big{)}\bigg{|}\bigg{)}d\sigma(x)\leq
C\sigma(Q_{0}).$ (6.8)

In fact, the proof only requires a control on $mo(f,Q)$ when $Q$ is a dyadic
subcube of $Q_{0}$ (i.e., obtained from $Q_{0}$ by the usual dyadic
partitions).

###### Proof of Lemma 3.49.

Let $\omega$ and $k=\frac{d\omega}{d\sigma}$ be as in the statement, and let
$\delta>0$ be given. Let $\varepsilon>0$ be given, and choose $r_{0}>0$ so
small that $mo(\log k,\Delta(x,r))<\varepsilon$ for every surface ball
$\Delta(x,r)$ such that $0<r\leq r_{0}$. By standard manipulations of covering
cubes with balls of roughly the same sizes, we also see that $\|\log
k\|_{BMO_{\ast}(Q_{0})}\leq C\varepsilon$ for every cube $Q_{0}$ such that
${\rm diam}(Q_{0})\leq r_{0}$.

We choose $\gamma=d^{-1/2}r_{0}$, consider a surface ball $\Delta=\Delta(x,r)$
such that $r\leq\gamma$, and try to prove (3.50) for any Borel set
$E\subset\Delta$. Call $Q_{0}$ the smallest cube that contains $\Delta$; we
chose $\gamma$ so that ${\rm diam}(Q_{0})\leq r_{0}$, and so
$\|\log(k)\|_{BMO_{\ast}(Q_{0})}\leq C\varepsilon$. This allows us to apply
Lemma 6.7 and get

$\int_{Q_{0}}\exp\bigg{(}a\varepsilon^{-1}\bigg{|}\log(k(x))-\Big{(}\fint_{Q_{0}}\log(k)d\sigma\Big{)}\bigg{|}\bigg{)}d\sigma(x)\leq
C,$ (6.9)

with a new constant $a=C^{-1}c>0$ that depends only on $d$. Since (3.50) does
not change when we multiply $\omega$ by a positive constant, we may assume
that $\fint_{Q_{0}}\log(k)d\sigma=0$, which will simplify our computations
slightly. We write

$\omega(Q)=\int_{Q}k(x)d\sigma(x)\ \text{ and
}\omega(E)=\int_{E}k(x)d\sigma(x)$ (6.10)

and decide to cut each of these integrals in three, corresponding to the three
regions

$Q_{1}=\big{\\{}x\in Q_{0}\,;\,\log(k(x))\leq-A\varepsilon\big{\\}},\\\
Q_{2}=\big{\\{}x\in Q_{0}\,;\,|\log(k(x))|\leq A\varepsilon\big{\\}},\\\
Q_{3}=\big{\\{}x\in Q_{0}\,;\,\log(k(x))\geq A\varepsilon\big{\\}},$ (6.11)

where the large constant $A$ will be chosen soon. Notice that
$a\varepsilon^{-1}|\log(k)|\geq aA$ on $Q_{1}\cup Q_{3}$, so by Chebyshev

$\omega(Q_{1})\leq\sigma(Q_{1})\leq\sigma(Q_{1})+\sigma(Q_{3})\leq
Ce^{-aA}\sigma(Q_{0}).$ (6.12)

Similarly, we cut $Q_{3}$ into the regions $D_{j}=\big{\\{}x\in
Q_{3}\,;\,j+A\varepsilon\leq\log(k(x))<j+1+A\varepsilon\big{\\}}$ and get that

$\omega(Q_{3})\leq\int_{Q_{3}}k(x)d\sigma(x)\leq\sum_{j\geq
0}\int_{D_{j}}k(x)d\sigma(x)\\\ \leq\sum_{j\geq
0}e^{j+1+A\varepsilon}\exp\big{(}-a\varepsilon^{-1}(j+A\varepsilon)\big{)}\int_{D_{j}}\exp\big{(}a\varepsilon^{-1}\log(k(x))\big{)}d\sigma(x)\\\
\leq C\sigma(Q_{0})e^{1+A\varepsilon}e^{-aA}\sum_{j\geq
0}e^{j}e^{-a\varepsilon^{-1}j}\\\ \leq
C\sigma(Q_{0})e^{1+A\varepsilon}e^{-aA}\leq Ce^{-aA/2}$ (6.13)

if we choose $A\geq 2a^{-1}$ and $\varepsilon\leq A^{-1}$. Now let $F$ be a
Borel subset of $Q_{0}$; we think about $E$ or $\Delta$. Then

$|\omega(F)-\sigma(F)|\leq\omega(F\cap(Q_{1}\cup
Q_{3}))+\sigma(F\cap(Q_{1}\cup Q_{3}))+\int_{F\cap Q_{2}}|k(x)-1|d\sigma\\\
\leq\omega(Q_{1}\cup Q_{3})+\sigma(Q_{1}\cup
Q_{3})+\int_{Q_{2}}\big{|}e^{A\varepsilon}-1\big{|}d\sigma\leq(Ce^{-aA/2}+2A\varepsilon)\sigma(Q_{0})$
(6.14)

by (6.11)-(6.13). By definition of $Q_{0}$, we also have that
$\sigma(\Delta)\geq\alpha\sigma(Q_{0})$ for some constant $\alpha>0$ that
depends only on $d$. Then we choose $A$, and then $\varepsilon$, so that
$Ce^{-aA/2}+2A\varepsilon\leq 10^{-2}\alpha\delta$ in (6.14). Then

$|\omega(\Delta)-\sigma(\Delta)|\leq 10^{-2}\alpha\delta\sigma(Q_{0})\leq
10^{-3}\delta\sigma(\Delta)$ (6.15)

and similarly

$|\omega(E)-\sigma(E|\leq 10^{-2}\delta\sigma(\Delta),$ (6.16)

from which it is easy to see that
$\left|\frac{\omega(E)}{\omega(\Delta)}-\frac{\sigma(E)}{\sigma(\Delta)}\right|<\delta$
as in (3.50). ∎

######  Remark 6.17.

We shall not need the fact that Lemma 3.49 is still valid when $\Gamma$ is a
chord-arc surface with small enough constant, but this is the case, with
essentially the same proof. The point is that we can cut $\Gamma$ into
collections of “dyadic cubes” that have roughly the same properties as the
dyadic cubes in $\mathbb{R}^{d}$. Then the analogue of Lemma 6.7 holds when
$Q_{0}$ is one of these cubes (because we only need to estimate $mo(f,Q)$ for
dyadic subcubes), and then we can proceed as above (but cover the surface ball
$\Delta$ by a finite collection of “dyadic cubes”). This would even work when
$\Gamma$ is simply Ahlfors regular (in fact the doubling property of $\sigma$
is the important feature).

Also, here we stated the lemma in terms of $VMO$ and vanishing $A_{\infty}$
condition, but the same proof shows that for each $\delta>0$, we can find
$\varepsilon=\varepsilon(\delta,d)>0$ such that if $||\log
k||_{BMO}\leq\varepsilon$, then $\omega\in A_{\infty}(\sigma,\delta)$.

There is a converse to 3.49 and its small constant variant, which we discuss
now.

######  Lemma 6.18.

There is a constant $\varepsilon>0$, that depends only on $d$, and for each
$\delta>0$, a constant $\varepsilon_{1}=\varepsilon_{1}(\delta,d)$ with the
following property. Let $\Gamma\in CASSC(\varepsilon)$ be a chord-arc surface
with small enough constant in $\mathbb{R}^{d+1}$ (as in Definition 1.13), and
let $\omega$ be another measure on $\Gamma$ such that $\omega\in
A_{\infty}(\sigma,\varepsilon_{1})$ (see Definition 1.15). Then $\omega$ is
absolutely continuous with respect to $\sigma$, and its density
$k=\frac{d\omega}{d\sigma}$ satisfies $||\log(k)||_{BMO}\leq\delta$.

###### Proof.

Let $\Gamma\in CASSC(\varepsilon)$ and $\omega\in
A_{\infty}(\sigma,\varepsilon_{1})$ satisfy the assumption. It is a general
fact about $A_{\infty}$ that then $\omega$ is absolutely continuous with
respect to $\sigma$, so $k$ is well defined and locally integrable. Let $a>0$
be small, to be chosen later (depending on $d$). We want to show that (if
$\varepsilon$ and $\varepsilon_{1}$ are small enough), for every surface ball
$\Delta$, there is a constant $C_{\Delta}$ such that

$\sigma\big{(}\big{\\{}x\in\Delta\,;\,|\log(k(x))-c_{\Delta}|\geq\delta\big{\\}}\big{)}\leq
a\sigma(\Delta).$ (6.19)

In fact, the simplest is to take
$c_{\Delta}=\log\Big{(}\frac{\omega(\Delta)}{\sigma(\Delta)}\Big{)}$. Write
$\Delta=\Delta_{1}\cup\Delta_{2}\cup\Delta_{3}$, with

$\Delta_{1}=\big{\\{}x\in\Delta\,;\,k(x)<e^{c_{\Delta}-\delta}\big{\\}},\\\
\Delta_{2}=\big{\\{}x\in\Delta\,;\,e^{c_{\Delta}-\delta}\leq k(x)\leq
e^{c_{\Delta}+\delta}\big{\\}},\\\
\Delta_{3}=\big{\\{}x\in\Delta\,;\,k(x)>e^{c_{\Delta}+\delta}\big{\\}};$
(6.20)

we want to show that
$\sigma(\Delta_{1})+\sigma(\Delta_{3})\leq\sigma(\Delta)/100$. We take
$E=\Delta_{1}$ in the definition (1.16); we get that
$\left|\frac{\omega(\Delta_{1})}{\omega(\Delta)}-\frac{\sigma(\Delta_{1})}{\sigma(\Delta)}\right|<\varepsilon_{1}$
or, since $\omega(\Delta)=e^{c_{\Delta}}\sigma(\Delta)$,

$|e^{-c_{\Delta}}\omega(\Delta_{1})-\sigma(\Delta_{1})|\leq\varepsilon_{1}\sigma(\Delta).$
(6.21)

But $\omega(\Delta_{1})=\int_{\Delta_{1}}kd\sigma\leq
e^{c_{\Delta}-\delta}\sigma(\Delta)$ so
$\sigma(\Delta_{1})-e^{-c_{\Delta}}\omega(\Delta_{1})\geq(1-e^{-\delta})\sigma(\Delta_{1})$,
so the comparison with (6.21) yields
$\sigma(\Delta_{1})\leq\varepsilon_{1}(1-e^{-\delta})^{-1}\sigma(\Delta)<a\sigma(\Delta)/2$
if $\varepsilon_{1}$ is chosen small enough. Similarly,

$|e^{-c_{\Delta}}\omega(\Delta_{3})-\sigma(\Delta_{3})|\leq\varepsilon_{1}\sigma(\Delta)$
(6.22)

by the proof of (6.21), and the definition yields
$\omega(\Delta_{3})=\int_{\Delta_{3}}kd\sigma\geq
e^{c_{\Delta}+\delta}\sigma(\Delta)$, so that
$e^{-c_{\Delta}}\omega(\Delta_{3})-\sigma(\Delta_{3})\geq(e^{-\delta}-1)\sigma(\Delta_{3})$
and the comparison yields
$\sigma(\Delta_{3})\leq\varepsilon_{1}(e^{\delta}-1)^{-1}\sigma(\Delta)<a\sigma(\Delta)/2$.
This proves (6.19).

Now we claim that a well known result proved by F. John [Joh65], improved by
J. O. Strömberg [Str79], allows one to deduce from (6.19) that $\log(k)\in
BMO$, with a norm less than $C\delta$. Here $C$ depends on $d$, and of course
the difference between $C\delta$ and $\delta$ could easily compensated by
taking $\varepsilon_{1}$ smaller. The advantage of the result of John and
Strömberg is that it is directly valid for a doubling measure. For the sake of
the reader that would know the usual proof of the John-Nirenberg theorem (with
simple stopping times), let us say two words about how the underlying ideas
could be applied. In $\mathbb{R}^{d}$, if you have (6.19) for balls, you also
get it for cubes $Q$, by the usual trick of using (6.19) for the smallest ball
containing $Q$, and at the price of starting with the smaller
$\widetilde{a}=a\sigma(Q)/\sigma(\Delta)$. Then, if $a$ is small enough, we
deduce from (6.19) that if $Q$ and $R$ are cubes such that $Q\subset R$ and
$\rm{diam}(Q)\geq\rm{diam}(R)/2$, then $|c_{Q}-c_{R}|\leq 2\delta$. Then the
proof of the John-Nirenberg theorem can then be applied, and this yields
$||\log(k)||_{BMO}\leq C\delta$. So the result is valid when $\Gamma\in
CASSC(\varepsilon)$, $\varepsilon$ small enough (so that $\sigma$ is
doubling), with the usual fake dyadic cubes on $\Gamma$. We need to take $a$
even smaller, so that in particular we can control $|c_{Q}-c_{R}|$ for fake
cubes, when $Q$ is a child of $R$, but the proof goes through. ∎

######  Corollary 6.23.

In Theorem 1.17, we can replace the conclusion that $\omega^{\infty}\in
A_{\infty}(\sigma,\delta)$ with the conclusion that $\omega^{\infty}$ is
absolutely continuous with respect to $\sigma$, with a density
$k=\frac{d\omega}{d\sigma}$ such that $||\log(k)||_{BMO}\leq\delta$.

In Theorem 1.19, we can replace the conclusion by the fact that for each
surface ball $\Delta=B(x,r)\cap\Gamma$ such that $\Delta\subset
B(X,\kappa\operatorname{dist}(X,\Gamma))$ and
$0<r\leq\tau\operatorname{dist}(X,\Gamma)$, $\omega^{X}$ is absolutely
continuous with respect to $\sigma$ on $\Delta$, and the density
$k=\frac{d\omega^{X}}{d\sigma}$ is such that
$||\log(k)||_{BMO(\Delta)}\leq\delta$.

###### Proof.

The first part follows at once from Theorem 1.17 and Lemma 6.18. The second
part follows Theorem 1.19 (applied with a larger $\kappa$ for security) and
the proof of Lemma 6.18. ∎

##  7\. Proof of Lemma 3.23 on the elliptic measure for a limit

In this section, we prove Lemma 3.23.

We first show that (3.24) holds when $E=U$ is a bounded open set in
$\mathbb{R}^{d}$.

Let $\left\\{F_{i}\right\\}_{i\geq 1}$ be a sequence of closed sets with
$F_{i}\subset U$ and $F_{i}\uparrow U$ as $i\to\infty$. For each $i\geq 1$,
let $g_{i}\in C^{\infty}_{c}(U)$, with $0\leq g_{i}\leq 1$ and $g_{i}\equiv 1$
on $F_{i}$. For $X\in\mathbb{R}^{d+1}_{+}$, define

$v_{i}^{(k)}(X):=\int_{\mathbb{R}^{d}}g_{i}(y)d\omega_{L_{k}}^{X}(y),$

and

$v_{i}(X):=\int_{\mathbb{R}^{d}}g_{i}(y)d\omega_{L}^{X}(y).$

By definition of the elliptic measures, $v_{i}^{(k)}\in W_{\operatorname{loc\
}}^{1,2}(\mathbb{R}^{d+1}_{+})$ is the unique weak solution to

$\begin{cases}L_{k}v_{i}^{(k)}=0\qquad\text{in }\mathbb{R}^{d+1}_{+},\\\
v_{i}^{(k)}(x,0)=g_{i}(x)\qquad\text{for }x\in\mathbb{R}^{d}\end{cases}$ (7.1)

that satisfies $v_{i}^{(k)}\in C(\overline{\mathbb{R}^{d+1}_{+}})$ and
$\lim_{\left|X\right|\to\infty,\,X\in\mathbb{R}^{d+1}_{+}}v_{i}^{(k)}(X)=0$.
Similarly, $v_{i}\in W_{\operatorname{loc\ }}^{1,2}(\mathbb{R}^{d+1}_{+})$ is
the unique weak solution to

$\begin{cases}Lv_{i}=0\qquad\text{in }\mathbb{R}^{d+1}_{+},\\\
v_{i}(x,0)=g_{i}(x)\qquad\text{for }x\in\mathbb{R}^{d}\end{cases}$ (7.2)

that satisfies $v_{i}\in C(\overline{\mathbb{R}^{d+1}_{+}})$ and
$\lim_{\left|X\right|\to\infty,\,X\in\mathbb{R}^{d+1}_{+}}v_{i}(X)=0$. Here,
and in the sequel,
$\overline{\mathbb{R}^{d+1}_{+}}=\mathbb{R}^{d}\times[0,\infty)$. Let us check
that for any fixed $i$,

$\lim_{\left|X\right|\to\infty,\,X\in\mathbb{R}^{d+1}_{+}}v_{i}^{(k)}(X)=0\quad\text{uniformly
in }k.$ (7.3)

In fact, since $\operatorname{supp}g_{i}\subset U$ and $0\leq g_{i}\leq 1$,

$0\leq
v_{i}^{(k)}(X)=\int_{U}g_{i}(y)d\omega_{L_{k}}^{X}(y)\leq\omega_{L_{k}}^{X}(\Delta_{r}),$

where $\Delta_{r}\subset\mathbb{R}^{d}$ is a ball with radius $r$ that
contains $U$. By Lemma 2.5 and the estimate (2.2) for the Green function,

$\left|v_{i}^{(k)}(X)\right|\leq
C\,r^{d-1}\left|X-X_{\Delta_{r}}\right|^{1-d},$

where $C$ depends only on $\mu_{0}$ and $d$. So (7.3) holds.

By our choice of $F_{i}$ and $g_{i}$,

$\left|v_{i}^{(k)}(X)-\omega_{L_{k}}^{X}(U)\right|=\left|\int_{\mathbb{R}^{d}}(g_{i}(y)-\mathbbm{1}_{U}(y))d\omega_{L_{k}}^{X}(y)\right|\leq\omega_{L_{k}}^{X}(U\setminus
F_{i}).$ (7.4)

Since $\mathfrak{N}(A_{k})<C_{0}$ for all $k\in{\mathbb{Z}}_{+}$, Lemma 3.2
asserts that $\omega_{L_{k}}\in A_{\infty}(d\sigma)$, and the $A_{\infty}$
constants depend only on $d$ and $C_{0}$. Therefore we can find constants
$\theta$, $C$, that may depend on $X$ and $U$, but not on $k$ or the $F_{i}$,
such that

$\omega_{L_{k}}^{X}(U\setminus F_{i})\leq C\sigma(U\setminus F_{i})^{\theta}.$

The regularity of the Lebesgue measure now implies that the right-hand side of
(7.4) tends to 0 as $i\to\infty$. So for any $X\in\mathbb{R}^{d+1}_{+}$, the
limit $\lim_{i\to\infty}v_{i}^{(k)}(X)$ exists, and

$\lim_{i\to\infty}v_{i}^{(k)}(X)=\omega_{L_{k}}^{X}(U)\qquad\text{uniformly in
}k.$ (7.5)

Similarly,

$\left|v_{i}(X)-\omega_{L}^{X}(U)\right|=\left|\int_{\mathbb{R}^{d}}(g_{i}(y)-\mathbbm{1}_{U}(y))d\omega_{L}^{X}(y)\right|\leq\omega_{L}^{X}(U\setminus
F_{i})\to 0$

as $i\to\infty$, by the regularity of $\omega_{L}^{X}$. So

$\lim_{i\to\infty}v_{i}(X)=\omega_{L}^{X}(U).$ (7.6)

We claim that for any fixed $i$ and $X\in\mathbb{R}^{d+1}_{+}$,

$\lim_{k\to\infty}v_{i}^{(k)}(X)=v_{i}(X).$ (7.7)

We shall justify the claim by first showing that as $k\to\infty$,
$v_{i}^{(k)}$ converges to a weak solution $v_{i}^{\infty}$ of
$Lv_{i}^{\infty}=0$ in $\mathbb{R}^{d+1}_{+}$ with boundary data $g_{i}$,
$v_{i}^{\infty}\in C(\overline{\mathbb{R}^{d+1}_{+}})$, and $v_{i}^{\infty}$
converges to 0 uniformly at infinity in $\mathbb{R}^{d+1}_{+}$. Then by the
maximum principle (or by the definition of elliptic measures),
$v_{i}^{\infty}$ must be the unique weak solution to the Dirichlet problem
(7.2). Thus, $v_{i}^{\infty}=v_{i}$, and the claim (7.7) will follow.

Let $i\in{\mathbb{Z}}_{+}$ be fixed for the moment. Since the $v_{i}^{(k)}$
are solutions with the same (smooth) boundary data $g_{i}$ and the $L_{k}$
have the same ellipticity constant for all $k$, the maximum principle, the
Hölder regularity of solutions, and the Arzela-Ascoli theorem imply that there
is a subsequence of $\big{\\{}v_{i}^{(k)}\big{\\}}_{k\geq 1}$, which we still
denote the same way, that converges uniformly on compact subsets of
$\overline{\mathbb{R}}^{d+1}_{+}$. That is, there is a continuous function
$v_{i}^{\infty}$ on $\overline{\mathbb{R}}^{d+1}_{+}$ such that

$\lim_{k\to\infty}v^{(k)}_{i}(X)=v_{i}^{\infty}(X),\ \text{ uniformly on
compact subsets of }\overline{\mathbb{R}^{d+1}_{+}}$ (7.8)

Since $v_{i}^{(k)}$ is continuous on $\overline{\mathbb{R}}^{d+1}_{+}$ and
equal to $g_{i}$ on $\mathbb{R}^{d}$, we get that

$v_{i}^{\infty}\in C(\overline{\mathbb{R}^{d+1}_{+}}),\quad\text{and
}v_{i}^{\infty}(x,0)=g_{i}(x)\ \text{ for }x\in\mathbb{R}^{d}.$ (7.9)

By (7.8) and (7.3),

$\lim_{\left|X\right|\to\infty,\,X\in\mathbb{R}^{d+1}_{+}}v_{i}^{\infty}(X)=0.$
(7.10)

Next let us show that (still for $i$ fixed)

$\nabla v_{i}^{(k)}\to\nabla v_{i}^{\infty}\text{ in
}L^{2}_{\operatorname{loc\ }}(\mathbb{R}^{d+1}_{+})\text{ as }k\to\infty.$
(7.11)

Let $K$ be a compact subset of $\mathbb{R}^{d+1}_{+}$, and pick a smooth cut-
off function $\eta$, with $0\leq\eta\leq 1$ everywhere, $\eta=1$ on $K$, and
compact support in $\mathbb{R}^{d+1}_{+}$. Let $k,\ell\geq 0$ be given; we
want to estimate

$I=\mu_{0}\int_{\mathbb{R}^{d+1}_{+}}\big{|}\nabla(v_{i}^{(k)}-v_{i}^{(\ell)})\big{|}^{2}\eta^{2}.$
(7.12)

The ellipticity of $L_{k}$ gives that

$I\leq\int_{\mathbb{R}^{d+1}_{+}}A_{k}\nabla(v_{i}^{(k)}-v_{i}^{(\ell)})\cdot\nabla(v_{i}^{(k)}-v_{i}^{(\ell)})\eta^{2}\\\
\hskip-56.9055pt=\int_{\mathbb{R}^{d+1}_{+}}A_{k}\nabla(v_{i}^{(k)}-v_{i}^{(\ell)})\cdot\nabla\Big{(}(v_{i}^{(k)}-v_{i}^{(\ell)})\eta^{2}\Big{)}\\\
-2\int_{\mathbb{R}^{d+1}_{+}}A_{k}\nabla(v_{i}^{(k)}-v_{i}^{(\ell)})\cdot\nabla\eta\Big{(}\eta(v_{i}^{(k)}-v_{i}^{(\ell)})\Big{)}=:J_{1}-2J_{2}.$
(7.13)

Set $\varphi=(v_{i}^{(k)}-v_{i}^{(\ell)})\eta^{2}$, and use it as a test
function the equations $L_{k}v_{i}^{(k)}=0$ and $L_{\ell}v_{i}^{(\ell)}=0$; we
get that

$J_{1}=\int_{\mathbb{R}^{d+1}_{+}}A_{k}\nabla(v_{i}^{(k)}-v_{i}^{(\ell)})\cdot\nabla\varphi=-\int_{\mathbb{R}^{d+1}_{+}}A_{k}\nabla
v_{i}^{(\ell)}\cdot\nabla\varphi=\int_{\mathbb{R}^{d+1}_{+}}(A_{\ell}-A_{k})\nabla
v_{i}^{(\ell)}\cdot\nabla\varphi\\\
=\int_{\mathbb{R}^{d+1}_{+}}(A_{\ell}-A_{k})\nabla
v_{i}^{(\ell)}\cdot\nabla(v_{i}^{(k)}-v_{i}^{(\ell)})\eta^{2}+2\int_{\mathbb{R}^{d+1}_{+}}(A_{\ell}-A_{k})\nabla
v_{i}^{(\ell)}\cdot\nabla\eta\Big{(}\eta(v_{i}^{(k)}-v_{i}^{(\ell)})\Big{)}\\\
=:J_{3}+2J_{4}.$

Then by (7.13), $I\leq J_{1}-2J_{2}\leq J_{3}-2J_{2}+2J_{4}$. Now we use
Cauchy Schwarz and Young’s inequality for each term. That is,

$J_{3}\leq\Big{\\{}\int_{\mathbb{R}^{d+1}_{+}}\big{|}\nabla(v_{i}^{(k)}-v_{i}^{(\ell)})\big{|}^{2}\eta^{2}\Big{\\}}^{1/2}\Big{\\{}\int_{\mathbb{R}^{d+1}_{+}}\Big{|}(A_{k}-A_{\ell})\nabla
v_{i}^{(\ell)}\Big{|}^{2}\eta^{2}\Big{\\}}^{1/2}\\\
\leq\frac{1}{4}I+C\int_{\mathbb{R}^{d+1}_{+}}\Big{|}(A_{k}-A_{\ell})\nabla
v_{i}^{(\ell)}\Big{|}^{2}\eta^{2},$ (7.14)

where $C$ depends also on $\mu_{0}$, then

$|2J_{2}|\leq
C\Big{\\{}\int_{\mathbb{R}^{d+1}_{+}}\big{|}\nabla(v_{i}^{(k)}-v_{i}^{(\ell)})\big{|}^{2}\eta^{2}\Big{\\}}^{1/2}\Big{\\{}\int_{\mathbb{R}^{d+1}_{+}}\Big{|}v_{i}^{(k)}-v_{i}^{(\ell)}\Big{|}^{2}|\nabla\eta|^{2}\Big{\\}}^{1/2}\\\
\leq\frac{1}{4}I+C\int_{\mathbb{R}^{d+1}_{+}}\Big{|}v_{i}^{(k)}-v_{i}^{(\ell)}\Big{|}^{2}|\nabla\eta|^{2},$
(7.15) $|2J_{4}|\leq
C\Big{\\{}\int_{\mathbb{R}^{d+1}_{+}}|A_{\ell}-A_{k}|^{2}|\nabla
v_{i}^{(\ell)}|^{2}\eta^{2}\Big{\\}}^{1/2}\Big{\\{}\int_{\mathbb{R}^{d+1}_{+}}\Big{|}v_{i}^{(k)}-v_{i}^{(\ell)}\Big{|}^{2}|\nabla\eta|^{2}\Big{\\}}^{1/2}\\\
\leq C\int_{\mathbb{R}^{d+1}_{+}}|A_{\ell}-A_{k}|^{2}|\nabla
v_{i}^{(\ell)}|^{2}\eta^{2}+C\int_{\mathbb{R}^{d+1}_{+}}\Big{|}v_{i}^{(k)}-v_{i}^{(\ell)}\Big{|}^{2}|\nabla\eta|^{2},$
(7.16)

Altogether,

$I\leq C\int_{\mathbb{R}^{d+1}_{+}}|A_{\ell}-A_{k}|^{2}|\nabla
v_{i}^{(\ell)}|^{2}\eta^{2}+C\int_{\mathbb{R}^{d+1}_{+}}\Big{|}v_{i}^{(k)}-v_{i}^{(\ell)}\Big{|}^{2}|\nabla\eta|^{2}=:CI_{1}+CI_{2},$
(7.17)

where $C$ depends only on $d$ and $\mu_{0}$ but not on $k$, $\ell$, or $i$.

By (7.8), the sequence $\big{\\{}v_{i}^{(k)}\big{\\}}_{k}$ converges in
$L^{2}_{\operatorname{loc\ }}(\mathbb{R}^{d+1}_{+})$, so $I_{2}$ tends to $0$
when $k$ and $\ell$ tend to $+\infty$. For $I_{1}$, we use Hölder’s inequality
and then the reverse Hölder inequality for solutions (see e.g. [Ken94] Lemma
1.1.12) to get that

$\left|I_{1}\right|\leq\left(\iint\left|A_{k}-A_{\ell}\right|^{2p^{\prime}}\eta^{p^{\prime}}\right)^{\frac{1}{p^{\prime}}}\left(\iint\left|\nabla
v_{i}^{(\ell)}\right|^{2p}\eta^{p}\right)^{\frac{1}{p}}\\\ \leq
C\left(\iint_{\operatorname{supp}\eta}\left|A_{k}-A_{\ell}\right|^{2p^{\prime}}\right)^{\frac{1}{p^{\prime}}}\left(\iint_{K^{\prime}}\left|\nabla
v_{i}^{(\ell)}\right|^{2}\right)^{\frac{1}{2}},$

where $p>1$ is sufficiently close to $1$ so that the reverse Hölder inequality
for solutions holds, $p^{\prime}$ is the Hölder exponent of $p$, and
$K^{\prime}$ is a slightly larger compact set in $\mathbb{R}^{d+1}_{+}$ than
$\operatorname{supp}\eta$. Here, the constant $C$ depends on $p$,
$\operatorname{supp}\eta$, and $K^{\prime}$, but is independent of $k$ and
$\ell$. Notice that by Caccioppoli’s inequality, the regularity of solutions,
and the maximum principle,

$\iint_{K^{\prime}}\left|\nabla v_{i}^{(\ell)}\right|^{2}\leq
C\iint_{V}\left|v_{i}^{(\ell)}\right|^{2}\leq
C^{\prime}\left\|v_{i}^{(\ell)}\right\|^{2}_{L^{\infty}(\mathbb{R}^{d+1}_{+})}\leq
C^{\prime}\left\|g_{i}\right\|_{L^{\infty}(\mathbb{R}^{d})}^{2}\leq
C^{\prime},$

where $V\subset\subset\mathbb{R}^{d+1}_{+}$ is an enlargement of the set
$K^{\prime}$, and $C$, $C^{\prime}$ are constants independent of $\ell$.
Therefore,

$\left|I_{1}\right|\leq
C^{\prime}\left(\iint_{\operatorname{supp}\eta}\left|A_{k}-A_{\ell}\right|^{2p^{\prime}}\right)^{\frac{1}{p^{\prime}}}.$

Since $A_{k}(x,t)=A(x,t)$ for $t\in(2^{-k},2^{k})$,

$\lim_{k\to\infty}A_{k}(x,t)=A(x,t)\qquad\text{for
}(x,t)\in\mathbb{R}^{d+1}_{+}\text{ a.e.}$ (7.18)

So $I_{1}$ tends to $0$ when $k$ and $\ell$ tend to $+\infty$, by the
dominated convergence theorem, hence $I$ tends to $0$ for each given $K$ and
$\eta$, and (7.11) follows. Because of (7.11), one can let $k$ tend to
$+\infty$ in the equation

$0=\iint A_{k}\nabla v_{i}^{(k)}(Y)\nabla\varphi(Y)dY$

where $\varphi\in C_{0}^{\infty}(\mathbb{R}^{d+1}_{+})$ is any test function,
and get that $v_{i}^{\infty}$ is a weak solution of $Lv_{i}^{\infty}=0$ in
$\mathbb{R}^{d+1}_{+}$. It satisfies (7.9) and (7.10), and we checked earlier
that the claim (7.7) follows.

We claim that

$\lim_{k\to\infty}\omega^{X}_{L_{k}}(U)=\omega_{L}^{X}(U)$ (7.19)

for every bounded open subset $U$ of $\Gamma$ and every
$X\in\mathbb{R}^{d+1}_{+}$. This will follow from (7.5), (7.6) and (7.7); to
see this, let $U$ and $X$ be fixed, and write

$\left|\omega_{L_{k}}^{X}(U)-\omega_{L}^{X}(U)\right|\leq\left|\omega_{L_{k}}^{X}(U)-v_{i}^{(k)}(X)\right|+\left|v_{i}(X)-\omega_{L}^{X}(U)\right|+\left|v_{i}^{(k)}(X)-v_{i}(X)\right|.$

Let $\varepsilon>0$ be arbitrary. By (7.5) and (7.6), one can choose $i_{0}$
sufficiently large so that

$\left|\omega_{L_{k}}^{X}(U)-v_{i_{0}}^{(k)}(X)\right|<\varepsilon/3\quad\text{for
all }k\in{\mathbb{Z}}_{+}$

and $\left|v_{i_{0}}(X)-\omega_{L}^{X}(U)\right|<\varepsilon/3$. For this
fixed $i_{0}$, (7.7) allows us to take $k$ sufficiently large so that

$\left|v_{i_{0}}^{(k)}(X)-v_{i_{0}}(X)\right|<\varepsilon/3.$

Altogether, we obtain that for $k$ sufficiently large,
$\left|\omega_{L_{k}}^{X}(U)-\omega_{L}^{X}(U)\right|<\varepsilon$. This
proves (7.19) (for $U$ open and bounded), but apparently only for some
subsequence that we constructed. However, if (7.19) were to fail (for $X$ and
the open set $U$), we could first replace $\\{L_{k}\\}$ with a subsequence for
which $\omega_{L_{k}}^{X}(U)$ stays far from $\omega_{L}^{X}(U)$, and then
proceed as above with $\\{L_{k}\\}$, find a subsequence with (7.19), a
contradiction. So (7.19) holds for the whole sequence.

It remains to show that (3.24) (or (3.24)) also holds for any Borel (and not
just open) bounded set. So let the bounded Borel set $E$ and the pole $X$ be
fixed. Since $\omega_{L}^{X}$ is regular, we can find a nonincreasing sequence
$\\{U_{i}\\}$ of open sets that contain $E$ and a nondecreasing sequence
$\\{F_{i}\\}$ of closed sets contained in $E$, such that

$\lim_{i\to\infty}\omega_{L}^{X}(U_{i})=\lim_{i\to\infty}\omega_{L}^{X}(E_{i})=\omega_{L}^{X}(E).$
(7.20)

For each $i$, notice that $U_{i}$ and $U_{i}\setminus F_{i}$ are bounded and
open, so (7.19) holds for them and yields

$\lim_{k\to\infty}\omega^{X}_{L_{k}}(U_{i})=\omega_{L}^{X}(U_{i})\ \text{ and
}\ \lim_{k\to\infty}\omega^{X}_{L_{k}}(U_{i}\setminus
F_{i})=\omega_{L}^{X}(U_{i}\setminus F_{i}).$ (7.21)

Now let $\varepsilon>0$ be given; by (7.20) we can find $i$ such that
$\omega_{L}^{X}(U_{i}\setminus F_{i})<\frac{\varepsilon}{4}$. Then (7.21) says
that for $k$ large,

$|\omega^{X}_{L_{k}}(U_{i})-\omega_{L}^{X}(U_{i})|+|\omega^{X}_{L_{k}}(U_{i}\setminus
F_{i})-\omega_{L}^{X}(U_{i}\setminus F_{i})|<\frac{\varepsilon}{4}$ (7.22)

and since $F_{i}\subset E\subset U_{i}$,

$|\omega^{X}_{L_{k}}(E)-\omega_{L}^{X}(E)|\leq|\omega^{X}_{L_{k}}(E)-\omega_{L_{k}}^{X}(U_{i})|+|\omega_{L_{k}}^{X}(U_{i})-\omega_{L}^{X}(U_{i})|+|\omega_{L}^{X}(U_{i})-\omega_{L}^{X}(E)|\\\
=\omega^{X}_{L_{k}}(U_{i}\setminus
E)+|\omega_{L_{k}}^{X}(U_{i})-\omega_{L}^{X}(U_{i})|+\omega^{X}_{L}(U_{i}\setminus
E)\\\ \leq\omega^{X}_{L_{k}}(U_{i}\setminus
F_{i})+|\omega_{L_{k}}^{X}(U_{i})-\omega_{L}^{X}(U_{i})|+\omega^{X}_{L}(U_{i}\setminus
F_{i})<\varepsilon$ (7.23)

by (7.21) and (7.22). So
$\lim_{k\to\infty}\omega^{X}_{L_{k}}(E)=\omega^{X}_{L}(E)$, and completes the
proof of Lemma 3.23.

## References

  * [AHM+20] Jonas Azzam, Steve Hofmann, José María Martell, Mihalis Mourgoglou, and Xavier Tolsa.  Harmonic measure and quantitative connectivity: geometric characterization of the $L^{p}$-solvability of the Dirichlet problem.  Inventiones mathematicae, 222(3):881–993, 2020.
  * [BES21] Simon Bortz, Moritz Egert, and Olli Saari.  A Theorem of Fefferman, Kenig and Pipher re-revisited.  arXiv preprint arXiv:2107.14217, 2021.
  * [Bou87] Jean Bourgain.  On the Hausdorff dimension of harmonic measure in higher dimension.  Inventiones mathematicae, 87(3):477–483, 1987.
  * [BTZ20] Simon Bortz, Tatiana Toro, and Zihui Zhao.  Optimal poisson kernel regularity for elliptic operators with hölder-continuous coefficients in vanishing chord-arc domains.  arXiv preprint arXiv:2010.03056, 2020.
  * [BTZ22] Simon Bortz, Tatiana Toro, and Zihui Zhao.  Elliptic measures for Dahlberg-Kenig-Pipher operators: asymptotically optimal estimates.  Mathematische Annalen, 2022.
  * [CF74] Ronald Coifman and Charles Fefferman.  Weighted norm inequalities for maximal functions and singular integrals.  Studia Mathematica, 51:241–250, 1974.
  * [CFMS81] Lo Caffarelli, E Fabes, S Mortola, and Sandro Salsa.  Boundary behavior of nonnegative solutions of elliptic operators in divergence form.  Indiana University Mathematics Journal, 30(4):621–640, 1981.
  * [DFM19] Guy David, Joseph Feneuil, and Svitlana Mayboroda.  Dahlberg’s theorem in higher co-dimension.  Journal of Functional Analysis, 276(9):2731–2820, 2019.
  * [DJ90] Guy David and D Jerison.  Lipschitz approximation to hypersurfaces, harmonic measure, and singular integrals.  Indiana University Mathematics Journal, pages 831–845, 1990.
  * [DLM21] Guy David, Linhan Li, and Svitlana Mayboroda.  Carleson measure estimates for the Green function.  arXiv preprint arXiv:2102.09592, 2021.
  * [FKP91] Robert A Fefferman, Carlos E Kenig, and Jill Pipher.  The theory of weights and the dirichlet problem for elliptic equations.  Annals of Mathematics, 134(1):65–124, 1991.
  * [Gra14] Loukas Grafakos.  Modern fourier analysis, volume 250.  Springer, third edition, 2014.
  * [HMM+21] Steve Hofmann, José María Martell, Svitlana Mayboroda, Tatiana Toro, and Zihui Zhao.  Uniform rectifiability and elliptic operators satisfying a Carleson measure condition.  Geometric and Functional Analysis, 31(2):325–401, 2021.
  * [HMT17] Steve Hofmann, José María Martell, and Tatiana Toro.  $A_{\infty}$ implies NTA for a class of variable coefficient elliptic operators.  Journal of Differential Equations, 263(10):6147–6188, 2017.
  * [JK82] David S Jerison and Carlos E Kenig.  Boundary behavior of harmonic functions in non-tangentially accessible domains.  Advances in Mathematics, 46(1):80–147, 1982.
  * [Joh65] F John.  Quasi-isometric mappings.  in seminari 1962/63 di Analisi Algebra, Geometria e Topologia, Vol II, Ist, naz, alta mat, Ediz. Cremonese, Rome, 1965.
  * [Ken94] Carlos E Kenig.  Harmonic analysis techniques for second order elliptic boundary value problems, volume 83.  American Mathematical Soc., 1994.
  * [Kor98] Michael Korey.  Carleson conditions for asymptotic weights.  Transactions of the American Mathematical Society, 350(5):2049–2069, 1998.
  * [KP93] Carlos E Kenig and Jill Pipher.  The Neumann problem for elliptic equations with non-smooth coefficients.  Inventiones mathematicae, 113(1):447–509, 1993.
  * [KP01] Carlos E Kenig and Jill Pipher.  The Dirichlet problem for elliptic equations with drift terms.  Publicacions Matematiques, pages 199–217, 2001.
  * [KT97] Carlos E Kenig and Tatiana Toro.  Harmonic measure on locally flat domains.  Duke Mathematical Journal, 87(3):509–551, 1997.
  * [KT99] Carlos E Kenig and Tatiana Toro.  Free boundary regularity for harmonic measures and Poisson kernels.  Annals of Mathematics, 150(2):369–454, 1999.
  * [Sem89] Stephen Semmes.  A criterion for the boundedness of singular integrals on hypersurfaces.  Transactions of the American Mathematical Society, 311(2):501–513, 1989.
  * [Sem90a] Stephen Semmes.  Analysis vs. geometry on a class of rectifiable hypersurfaces in $\mathbb{R}^{n}$.  Indiana University Mathematics Journal, 39(4):1005–1035, 1990.
  * [Sem90b] Stephen Semmes.  Differentiable function theory on hypersurfaces in $\mathbb{R}^{n}$ (without bounds on their smoothness).  Indiana University Mathematics Journal, 39(4):985–1004, 1990.
  * [Ste70] Elias M Stein.  Singular Integrals and Differentiability Properties of Functions (PMS-30).  Princeton University Press, 1970.
  * [Str79] Jan-Olov Strömberg.  Bounded mean oscillation with Orlicz norms and duality of Hardy spaces.  Indiana University Mathematics Journal, 28(3):511–544, 1979.

Guy David, Université Paris-Saclay, CNRS, Laboratoire de mathématiques
d’Orsay, 91405 Orsay, France

E-mail address<EMAIL_ADDRESS>
Linhan Li, School of Mathematics, University of Minnesota, Minneapolis, MN
55455, USA

E-mail address<EMAIL_ADDRESS>
Svitlana Mayboroda, School of Mathematics, University of Minnesota,
Minneapolis, MN 55455, USA

E-mail address<EMAIL_ADDRESS>