# Self-Exploring Language Models:
Active Preference Elicitation for Online Alignment

Shenao Zhang1 Donghan Yu2 Hiteshi Sharma2
Ziyi Yang2 Shuohang Wang2 Hany Hassan2 Zhaoran Wang1
1Northwestern University 2Microsoft

###### Abstract

Preference optimization, particularly through Reinforcement Learning from
Human Feedback (RLHF), has achieved significant success in aligning Large
Language Models (LLMs) to adhere to human intentions. Unlike offline alignment
with a fixed dataset, online feedback collection from humans or AI on model
generations typically leads to more capable reward models and better-aligned
LLMs through an iterative process. However, achieving a globally accurate
reward model requires systematic exploration to generate diverse responses
that span the vast space of natural language. Random sampling from standard
reward-maximizing LLMs alone is insufficient to fulfill this requirement. To
address this issue, we propose a bilevel objective optimistically biased
towards potentially high-reward responses to actively explore out-of-
distribution regions. By solving the inner-level problem with the
reparameterized reward function, the resulting algorithm, named Self-Exploring
Language Models (SELM), eliminates the need for a separate RM and iteratively
updates the LLM with a straightforward objective. Compared to Direct
Preference Optimization (DPO), the SELM objective reduces indiscriminate favor
of unseen extrapolations and enhances exploration efficiency. Our experimental
results demonstrate that when finetuned on Zephyr-7B-SFT and
Llama-3-8B-Instruct models, SELM significantly boosts the performance on
instruction-following benchmarks such as MT-Bench and AlpacaEval 2.0, as well
as various standard academic benchmarks in different settings. Our code and
models are available at https://github.com/shenao-zhang/SELM.

##  1 Introduction

Large Language Models (LLMs) have recently achieved significant success
largely due to their ability to follow instructions with human intent. As the
defacto method for aligning LLMs, Reinforcement Learning from Human Feedback
(RLHF) works by maximizing the reward function, either a separate model [44;
5; 18] or reparameterized by the LLM policy [49; 48; 4; 70], which is learned
from the prompt-response preference data labeled by humans. The key to the
success of alignment is the response diversity within the preference data,
which prevents reward models (RMs) from getting stuck in local optima, thereby
producing more capable language models.

Offline alignment methods [49; 54] attempt to manually construct diverse
responses for fixed prompts [11; 24; 72], which, unfortunately, struggles to
span the nearly infinite space of natural language. On the other hand, online
alignment follows an iterative procedure: sampling responses from the LLM and
receiving feedback to form new preference data for RM training [44; 21]. The
former step helps explore out-of-distribution (OOD) regions through randomness
in sampling. However, in standard online RLHF frameworks, maximizing the
expected reward learned from the collected data is the only objective for the
LLM, sampling from which often leads to responses clustered around local
optima. This passive exploration mechanism can suffer from overfitting and
premature convergence, leaving the potentially high-reward regions unexplored.

Figure 1: Intuition of our method. For a fixed prompt $x$, a reward model
$r(x,y)$ tries to fit the ground-truth reward $r^{*}(x,y)$. The blue and green
RMs are equally good when using standard reward-fitting loss
$\mathcal{L}_{\text{lr}}$, since the observed preference data (red stars) are
fitted equally well. However, the green RM has a larger $\max_{y}r(x,y)$ and
thus a lower optimistically biased loss
$\mathcal{L}_{\text{lr}}-\alpha\max_{y}r(x,y)$. Therefore, the response
$y_{u}$ at which the uncertainty is high can be elicited and then proceeded
for human feedback to reduce uncertainty.

To address this issue, we propose an active exploration method for online
alignment that elicits novel favorable responses. In its simplest form, an
optimism term $\alpha\max_{y}r(x,y)$ is added to the reward-fitting objective
(e.g., logistic regression on dataset $\mathcal{D}$), denoted as
$-\mathcal{L}_{\text{lr}}$, resulting in a bilevel optimization objective for
the reward model $r$:

$\displaystyle\max_{r}\max_{y}\alpha
r(x,y)-\mathcal{L}_{\text{lr}}(r;\mathcal{D}),$ (1.1)

where $\alpha$ is a hyperparameter controlling the degree of optimism. The
intuition is illustrated in Figure 1. Specifically, minimizing the vanilla
reward-fitting loss $\mathcal{L}_{\text{lr}}$ is likely to give a locally
accurate RM that overfits the observed data and gets stuck in local minima.
Random sampling from this vanilla RM may take a long time to explore the OOD
regions that contain the best response. By incorporating the optimism term, we
obtain an RM that both fits the data well and has a large $\max_{y}r(x,y)$.
This ensures that the greedy response $y_{u}$ from it is either globally
optimal when uncertainty in high-reward regions is eliminated, or potentially
good in unexplored areas where $r(x,y_{u})$ can be arbitrarily huge due to the
relaxed reward-fitting loss. Feedback from humans on these responses $y_{u}$
can then reduce uncertainty and train a more accurate RM.

In this paper, we formulate this idea within the context of online direct
alignment, where the LLM is iteratively updated without a separate RM. We
first introduce two modifications to the bilevel RM objective in 1.1, namely
adding KL constraints and using relative maximum reward. Then we derive a
simple LLM training objective by applying the closed-form solution of the
inner-level problem and reparameterizing the reward with the LLM policy. The
resulting iterative algorithm is called Self-Exploring Language Models (SELM).
We show that the policy gradient of SELM is biased towards more rewarding
areas. Furthermore, by reducing the chance of generating responses that are
assigned low implicit rewards, SELM mitigates the indiscriminate favoring of
unseen extrapolations found in DPO [49; 48] and enhances exploration
efficiency.

In experiments, we implement SELM using Zephyr-7B-SFT [57] and
Llama-3-8B-Instruct [38] as base models. By finetuning solely on the
UltraFeedback [11] dataset and using the small-sized PairRM [25] for iterative
AI feedback, SELM boosts the performance of Zephyr-7B-SFT and
Llama-3-8B-Instruct by a large margin on AlpacaEval 2.0 [14] ($+16.24\%$ and
$+11.75\%$ LC win rates) and MT-Bench [71] ($+2.31$ and $+0.32$). SELM also
demonstrates strong performance on standard academic benchmarks and achieves
higher pairwise LC win rates against the iterative DPO baseline. Our code and
models are available at https://github.com/shenao-zhang/SELM.

##  2 Background

###  2.1 Large Language Models

A language model $\pi\in\Delta_{\mathcal{Y}}^{\mathcal{X}}$ typically takes
the prompt $x\in\mathcal{X}$ as input and outputs the response
$y\in\mathcal{Y}$. Here, $\mathcal{X}$ and $\mathcal{Y}$ are finite spaces of
prompts and responses, respectively. Given the prompt $x\in\mathcal{X}$, a
discrete probability distribution $\pi(\cdot\mid x)\in\Delta_{\mathcal{Y}}$ is
generated, where $\Delta_{\mathcal{Y}}$ is the set of discrete distributions
over $\mathcal{Y}$. Modern recipes for training LLMs consist of pre-training
and post-training procedures, where during pre-training, LLMs learn to predict
the next word on a huge and diverse dataset of text sequences in order to
understand the underlying patterns and structures of natural language in an
unsupervised manner. The post-training procedure aims to align better to end
tasks and human preferences with two phases happening in order: Supervised
Fine-Tuning (SFT) and human preference alignment. Here, SFT fine-tunes the
pre-trained LLM with supervised learning on high-quality data to follow
instructions on downstream tasks and obtain a model $\pi^{\text{SFT}}$. In the
following of this paper, we focus mainly on preference alignment.

###  2.2 Reward Modeling and Preference Optimization

#### Reinforcement Learning from Human Feedback (RLHF).

Standard RLHF frameworks consist of learning a reward model and then
optimizing the LLM policy using the learned reward.

Specifically, a point-wise reward
$r(x,y):\mathcal{X}\times\mathcal{Y}\rightarrow\mathcal{R}$ represents the Elo
score [16] of the response $y$ given the prompt $x$. Then the preference
distribution can be expressed by the Bradley-Terry model that distinguishes
between the preferred response $y_{w}$ and the dispreferred response $y_{l}$
given prompt $x$, denoted as $y_{w}\succ y_{l}\mid x$, using the logistic
function $\sigma$:

$\displaystyle p(y_{w}\succ y_{l}\mid x)$
$\displaystyle:=\mathbb{E}_{h}\bigl{[}\mathmybb{1}(h\text{ prefers
}y_{w}\text{ over }y_{l}\text{ given }x)\bigr{]}$
$\displaystyle\,=\sigma\bigl{(}r(x,y_{w})-r(x,y_{l})\bigr{)}=\frac{\exp\bigl{(}r(x,y_{w})\bigr{)}}{\exp\bigl{(}r(x,y_{w})\bigr{)}+\exp\bigl{(}r(x,y_{l})\bigr{)}},$
(2.1)

where $h$ denotes the human rater and the expectation is over $h$ to account
for the randomness of the choices of human raters we ask for their preference.
When provided a static dataset of $N$ comparisons
$\mathcal{D}=\\{x_{i},y_{w,i},y_{l,i}\\}_{i=1}^{N}$, the parameterized reward
model can be learned by minimizing the following logistic regression loss:

$\displaystyle\mathcal{L}_{\text{lr}}(r;\mathcal{D})=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\bigl{[}\log\sigma\bigl{(}r(x,y_{w})-r(x,y_{l})\bigr{)}\bigr{]}.$
(2.2)

Using the learned reward, the LLM policy
$\pi\in\Delta_{\mathcal{Y}}^{\mathcal{X}}$ is optimized with reinforcement
learning (RL) to maximize the expected reward while maintaining a small
deviation from some base reference policy $\pi_{\text{ref}}$, i.e., maximizing
the following objective

$\displaystyle\mathcal{J}(\pi;\mathcal{D})=\mathbb{E}_{x\sim\mathcal{D},y\sim\pi(\cdot\mid
x)}\bigl{[}r(x,y)\bigr{]}-\beta\mathbb{D}_{\text{KL}}(\pi\,||\,\pi_{\text{ref}}),$
(2.3)

where $\beta$ is a hyperparameter and
$\mathbb{D}_{\text{KL}}(\pi\,||\,\pi_{\text{ref}}):=\mathbb{E}_{x\sim\mathcal{D}}[\text{KL}(\pi(\cdot\mid
x)\,||\,\pi_{\text{ref}}(\cdot\mid x))]$ is the expected Kullback-Leibler (KL)
divergence. An ideal $\pi_{\text{ref}}$ is the policy that helps mitigate the
distribution shift issue [49; 21] between the true preference distribution and
the policy $\pi$ during the off-policy RL training. Since we only have access
to the dataset $\mathcal{D}$ sampled from the unavailable true preference
distribution, $\pi_{\text{ref}}$ can be obtained by fine-tuning on the
preferred responses in $\mathcal{D}$ or simply setting
$\pi_{\text{ref}}=\pi^{\text{SFT}}$ and performing RLHF based on the SFT
model.

#### Direct Alignment from Preference.

With the motivation to get rid of a separate reward model, which is
computationally costly to train, recent works [49; 4; 70; 57; 17] derived the
preference loss as a function of the policy by changing of variables. Among
them, DPO [49] shows that when the BT model in (2.2) can perfectly fit the
preference, the global optimizers of the RLHF objective in (2.3) and the
following loss are equivalent:

$\displaystyle\mathcal{L}_{\text{DPO}}(\pi;\mathcal{D})=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\biggl{[}\log\sigma\biggl{(}\beta\log\frac{\pi(y_{w}\mid
x)}{\pi_{\text{ref}}(y_{w}\mid x)}-\beta\log\frac{\pi(y_{l}\mid
x)}{\pi_{\text{ref}}(y_{l}\mid x)}\biggr{)}\biggr{]}.$

##  3 Self-Exploring Language Models

###  3.1 RM-Free Objective for Active Exploration

In this section, we present several modifications to the optimistically biased
objective (1.1) motivated in the introduction. Then we derive an RM-free
objective for the LLM policy and analyze how active exploration works by
examining its gradient.

First, we consider the equivalence of (1.1):
$\max_{r}-\mathcal{L}_{\text{lr}}(r;\mathcal{D})+\alpha\max_{\pi}\mathbb{E}_{y\sim\pi}[r(x,y)]$,
where the inner $\pi$ is deterministic when optimal. To account for the change
of $\pi$ relative to the reference policy $\pi_{\text{ref}}$, we introduce two
modifications: (1) replacing the optimistic bias term
$\max_{\pi}\mathbb{E}_{y\sim\pi}[r(x,y)]$ with
$\max_{\pi}\mathbb{E}_{y\sim\pi,y^{\prime}\sim\pi_{\text{ref}}}[r(x,y)-r(x,y^{\prime})]$,
and (2) incorporating a KL-divergence loss term between $\pi$ and
$\pi_{\text{ref}}$. These changes ensure that the resulting optimistic RM
elicits responses with high potential unknown to the reference policy
$\pi_{\text{ref}}$ while minimizing the deviation between $\pi$ and
$\pi_{\text{ref}}$.

Formally, for the reward function $r$, the bilevel optimization problem with
optimism is formulated as:

$\displaystyle\max_{r}-\mathcal{L}_{\text{lr}}(r;\mathcal{D}_{t})+\alpha\max_{\pi}\Biggl{(}\underbrace{\mathbb{E}_{\begin{subarray}{c}x\sim\mathcal{D}_{t},y\sim\pi(\cdot\mid
x)\\\ y^{\prime}\sim\pi_{\text{ref}}(\cdot\mid
x)\end{subarray}}\Bigl{[}r(x,y)-r(x,y^{\prime})\Bigr{]}-\beta\mathbb{D}_{\text{KL}}(\pi\,||\,\pi_{\text{ref}})}_{\mathcal{F}(\pi;r)}\Biggr{)},$
(3.1)

where $\mathcal{D}_{t}=\\{x_{i},y_{w,i}^{t},y_{l,i}^{t}\\}_{i=1}^{N}$ is the
associated dataset at iteration $t$ and $\mathcal{L}_{\text{lr}}$ is the
logistic regression loss defined in (2.2). The nested optimization in (3.1)
can be handled by first solving the inner optimization $\mathcal{F}(\pi;r)$ to
obtain $\pi_{r}$ that is optimal under $r$. The solution is as follows and we
defer all the derivations in this section to Appendix A.

$\displaystyle\pi_{r}(y\mid
x):=\mathop{\mathrm{argmax}}_{\pi}\mathcal{F}(\pi;r)=\frac{1}{Z(x)}\pi_{\text{ref}}(y\mid
x)\exp\bigl{(}r(x,y)/\beta\bigr{)},$

where the partition function
$Z(x)=\sum_{y}\pi_{\text{ref}}(y|x)\exp(r(x,y)/\beta)$. By substituting
$\pi=\pi_{r}$ into $\mathcal{F}(\pi;r)$, we can rewrite the bilevel objective
in (3.1) as a single-level one:

$\displaystyle\max_{r}-\mathcal{L}_{\text{lr}}(r;\mathcal{D}_{t})+\alpha\mathcal{F}(\pi_{r};r).$

Following the implicit reward formulation in DPO, we reparameterize the reward
function with $\theta\in\Theta$ as
$\widehat{r}_{\theta}(x,y)=\beta(\log\pi_{\theta}(y\mid
x)-\log\pi_{\text{ref}}(y\mid x))$, which is the optimal solution of (2.3) and
can express all reward classes consistent with the BT model as proved in [49].
With this change of variable, we obtain the RM-free objective for direct
preference alignment with optimism:

$\displaystyle\max_{\pi_{\theta}}-\mathcal{L}_{\text{DPO}}(\pi_{\theta};\mathcal{D}_{t})-\alpha\beta\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\text{ref}}(\cdot\mid
x)}\bigl{[}\log\pi_{\theta}(y\mid x)\bigr{]}.$ (3.2)

We now analyze how this new objective encourages active exploration.
Specifically, we derive the gradient of (3.2) with respect to $\theta$ as

$\displaystyle\underbrace{-\beta\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}_{t}}\Bigl{[}\sigma\bigl{(}\widehat{r}_{\theta}(x,y_{l})-\widehat{r}_{\theta}(x,y_{w})\bigr{)}\bigl{(}\nabla_{\theta}\log\pi_{\theta}(y_{w}\mid
x)-\nabla_{\theta}\log\pi_{\theta}(y_{l}\mid
x)\bigr{)}\Bigr{]}}_{\nabla_{\theta}\mathcal{L}_{\text{DPO}}(\pi_{\theta};\mathcal{D}_{t})}$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad-\alpha\beta\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\theta}(\cdot\mid
x)}\bigl{[}\exp\bigl{(}-\widehat{r}_{\theta}(x,y)/\beta\bigr{)}\nabla_{\theta}\log\pi_{\theta}(y\mid
x)\bigr{]}.$ (3.3)

We note that the second line, corresponding to the gradient of the optimism
term, decreases the log-likelihood of response $y$ generated by $\pi_{\theta}$
that has a low value of $\exp(-\widehat{r}_{\theta}(x,y)/\beta)$. Therefore,
the added optimism term biases the gradient toward parameter regions that can
elicit responses $y$ with high implicit reward $\widehat{r}_{\theta}$,
consistent with our intuition outlined in Figure 1.

This also explains why $\mathbb{E}_{\pi_{\text{ref}}}[\log\pi_{\theta}]$ is
minimized in our objective (3.2), which is equivalent to maximizing the KL
divergence between $\pi_{\text{ref}}$ and $\pi_{\theta}$, while the reverse KL
in the policy optimization objective (2.3) is minimized. For the DPO gradient
$\nabla_{\theta}\mathcal{L}_{\text{DPO}}(\pi_{\theta};\mathcal{D}_{t})$, the
degree of deviation of policy $\pi_{\theta}$ from $\pi_{\text{ref}}$ only
affects the preference estimated with $\widehat{r}_{\theta}$. In other words,
$\sigma(\widehat{r}_{\theta}(x,y_{l})-\widehat{r}_{\theta}(x,y_{w}))$ is a
scalar value and the policy deviation only determines the step size of the
policy gradient, instead of its direction. On the other hand, our added
exploration term directly controls the direction of the gradient toward
potentially more rewarding areas while still fitting the preference data in
$\mathcal{D}_{t}$. As more feedback data is collected iteratively, deviating
from the unbiasedly fitted model incurs a higher DPO loss, which ultimately
dominates our objective at convergence. This mechanism ensures that the
resulting LLM effectively balances between exploring novel responses and
exploiting previously observed ones, leading to a more accurate and aligned
model.

###  3.2 Algorithm

With the optimistically biased objective derived above, the language model can
actively generate OOD responses worth exploring. Human or AI feedback follows
to reduce the uncertainty in these regions. These two steps are executed
iteratively to get a more and more aligned model.

In practice, we split the offline preference dataset into three portions with
equal sizes, one for each iteration. Besides, we use AI rankers, such as
external RMs, to provide feedback on the model-generated response and the
original chosen, rejected responses. The complete pseudocode of our algorithm,
named Self-Exploring Language Models (SELM), is outlined in Algorithm 1.

Algorithm 1 Self-Exploring Language Models (SELM)

0: Reference model $\pi_{\text{ref}}$, preference dataset $\mathcal{D}$,
online iterations $T$, optimism coefficient $\alpha$.

1: for iteration $t=1,2,\ldots,T$ do

2: Set $\mathcal{D}_{t}$ as the $t$-th portion of $\mathcal{D}$ and generate
$y\sim\pi_{\text{ref}}(\cdot\mid x)$ for each prompt $x$ in $\mathcal{D}_{t}$.

3: Rank $\\{y,y_{w},y_{l}\\}$ and update $\mathcal{D}_{t}$ to contain the best
(chosen) and worst (rejected) responses.

4: Train the LLM
$\pi_{\theta_{t}}=\mathop{\mathrm{argmax}}_{\pi_{\theta}}-\mathcal{L}_{\text{DPO}}(\pi_{\theta};\mathcal{D}_{t})-\alpha\mathbb{E}_{x\sim\mathcal{D}_{t}}[\log\pi_{\theta}(y\mid
x)]$ and let $\pi_{\text{ref}}=\pi_{\theta_{t}}$.

5: end for

###  3.3 Self-Exploration Reduces Indiscriminate Favor of Unseen
Extrapolations

It has been observed recently [48; 46; 64] that DPO decreases the likelihood
of responses generated by the reference policy. It is because for any prompt
$x$, at convergence when $\pi_{\theta}\neq\pi_{\text{ref}}$, it holds that

$\displaystyle\mathbb{E}_{y\sim\pi_{\text{ref}}}\bigl{[}\widehat{r}_{\theta}(x,y)/\beta\bigr{]}=\mathbb{E}_{y\sim\pi_{\text{ref}}}\bigl{[}\log\pi_{\theta}(y\mid
x)-\log\pi_{\text{ref}}(y\mid
x)\bigr{]}=-\text{KL}\bigl{(}\pi_{\text{ref}}(\cdot\mid
x)\,||\,\pi_{\theta}(\cdot\mid x)\bigr{)}<0,$

while at the beginning of training when $\pi_{\theta}=\pi_{\text{ref}}$, the
above terms are zero. Thus, the expected implicit reward
$\widehat{r}_{\theta}$ as well as the likelihood of $\pi_{\theta}$ will
decrease on the reference model’s responses. This indicates that DPO
stimulates a biased distribution favoring unseen extrapolated responses. In
the online iterative setting that we consider, the LLM policy generates
responses and receives preference feedback alternately, where biasing towards
OOD regions may sometimes help discover outstanding novel responses. However,
DPO indiscriminately favors unseen extrapolations and passively explores based
purely on the randomness inherent in sampling from the LLM. As a consequence,
the vast space of natural language makes it almost impossible to exhaustively
explore all the possible responses and identify those that most effectively
benefit alignment.

Next, we demonstrate that SELM mitigates this issue by performing guided
exploration. Specifically, consider the proposed self-exploration objective in
(3.2), which, in addition to the standard DPO loss, also minimizes
$\mathbb{E}_{x,y\sim\pi_{\text{ref}}}[\log\pi_{\theta}(y\mid x)]$. We now
investigate how the probability distribution changes with this term
incorporated.

######  Theorem 3.1.

For any $\rho\in\Theta$ in the policy parameter space, let
$\widehat{r}_{\rho}(x,y)=\beta(\log\pi_{\rho}(y|x)-\log\pi_{\text{ref}}(y|x))$
be the reparameterized implicit reward. Denote $\pi^{\min}_{\rho}$ as the
policy that minimizes the expected implicit reward under the KL constraint,
i.e.,

$\displaystyle\pi^{\min}_{\rho}(\cdot\mid
x):=\mathop{\mathrm{argmin}}_{\pi}\mathbb{E}_{x,y\sim\pi(\cdot\mid
x)}\bigl{[}\widehat{r}_{\rho}(x,y)\bigr{]}+\beta\mathbb{D}_{\text{KL}}(\pi\,||\,\pi_{\rho}).$
(3.4)

Then minimizing $\mathbb{E}_{x,y\sim\pi_{\text{ref}}}[\log\pi_{\theta}(y|x)]$
decreases the likelihood of responses sampled from $\pi^{\min}_{\rho}$:

$\displaystyle\min_{\pi_{\theta}}\mathbb{E}_{x,y\sim\pi_{\text{ref}}(\cdot\mid
x)}\bigl{[}\log\pi_{\theta}(y\mid
x)\bigr{]}=\min_{\pi_{\theta}}\mathbb{E}_{x,y\sim\pi^{\min}_{\rho}(\cdot\mid
x)}\bigl{[}\log\pi_{\theta}(y\mid x)\bigr{]}.$

The above theorem states that maximizing the divergence between $\pi_{\theta}$
and $\pi_{\text{ref}}$ is essentially reducing the probability of generating
responses with low implicit rewards reparameterized by any policy parameter
$\rho$ during training. In other words, the policy not only exploits the
existing preference data but also learns to avoid generating the text $y$ that
is assigned a low reward value. This process occurs in every iteration with
updated reference models. Consequently, responses with high potential rewards
are selectively preferred and many commonplace responses receive a small
probability mass, thus mitigating the indiscriminate favoring of unseen
responses and improving exploration efficiency.

##  4 Related Work

#### Data Synthesis for LLMs.

A key challenge for fine-tuning language models to align with users’
intentions lies in the collection of demonstrations, including both the SFT
instruction-following expert data and the RLHF preference data. Gathering such
data from human labelers is expensive, time-consuming, and sometimes suffers
from variant quality [44; 29]. To address this issue, synthetic data [34] has
been used for aligning LLMs. One line of work focuses on generating plausible
instruction prompts for unlabeled data by regarding the target output as
instruction-following responses [31; 60; 27; 55]. Besides, high-quality data
can also be distilled from strong models for fine-tuning weaker ones [20; 1;
32; 12; 47]. To construct synthetic datasets for offline RLHF, a popular
pipeline [11; 57; 59; 24; 72] involves selecting responses sampled from
various LLMs on a set of prompts in the hope to increase the diversity of the
data that can span the whole language space. However, data manually collected
in such a passive way does not consider what improves the model most through
its training, leaving the potentially high-reward regions unexplored.

#### Iterative Online Preference Optimization

Compared to offline RLHF algorithms [49; 70; 4] that collect preference
datasets ahead of training, online RLHF [44; 21], especially the
iterative/batched online RLHF [5; 63; 19; 22; 62; 6; 50] has the potential to
gather better and better synthetic data as the model improves. As a special
case, self-alignment language models align their responses with desired
behaviors, such as model-generated feedback [66; 67; 53; 58]. Unfortunately,
the above methods still passively explore by relying on the randomness during
sampling and easily get stuck at local optima and overfit to the current data
due to the vast space of natural language. A notable exception is [15], which
proposed to use ensembles of RMs to approximately measure the uncertainty for
posterior-sampling active exploration. On the contrary, our method explores
based on the optimistic bias and does not estimate the uncertainty explicitly,
bypassing the need to fit multiple RMs.

#### Active Exploration.

In fact, active exploration has been widely studied beyond LLMs. Similar to
[15], most existing sample-efficient RL algorithms first estimate the
uncertainty of the environment using historical data and then plan with
optimism [3; 51; 26], or selecting the optimal action from a statistically
plausibly set of action values sampled from its posterior distribution [52;
41; 42; 69]. The proposed self-exploration objective can be categorized as an
optimism-based exploration method. However, most previous works require the
estimation of the upper confidence bound, which is often intractable. Ensemble
methods [43; 8; 37] can serve as approximations to the uncertainty estimation
but are still computationally inefficient. MEX [35] proposed to combine
estimation and planning in a single objective similar to ours and established
theoretical guarantees under traditional RL setups. Concurrent to our work,
RPO [36] also proposed to use an adversarially chosen reward model for policy
optimization, but focuses on mitigating overoptimization in offline RLHF
settings.

##  5 Experiments

###  5.1 Experiment Setup

We select the UltraFeedback [11] dataset as our training set, which contains
61k preference pairs of single-turn conversations. For the ranker providing AI
feedback during online alignment, we choose the small-sized PairRM (0.4B)
[25]. All experiments are conducted on 8xA100 GPUs.

Due to the absence of performant open-source online direct alignment codebases
at the time of this study, we first implement an iterative version of DPO as
the baseline, adhering to the same steps as Algorithm 1 but training the LLM
with the standard DPO objective. Then we conduct a grid search over
hyperparameters, such as the batch size, learning rate, and iteration number,
to identify the optimal settings for the iterative DPO baseline. Details on
the hyperparameters and grid search results are provided in Appendix C. We
follow these best settings to train SELM for a fair comparison. In addition,
the top choice for the base models of SELM are LLMs that are finetuned with
RLHF after SFT, since they are typically more capable than SFT-only and
pertrained models. We consider two series of LLMs: Zephyr [57] and Llama-3
[38], to demonstrate the robustness of SELM. Since the official
Zephyr-7B-$\beta$ model is finetuned with DPO on the same UltraFeedback
dataset, to avoid overoptimization, we choose
Zephyr-7B-SFT111https://huggingface.co/HuggingFaceH4/mistral-7b-sft-beta as
the base model and perform $3$ iterations of SELM after a single iteration of
standard DPO training on the first portion of the training data (we refer to
this model as Zephyr-7B-DPO). For
Llama-3-8B-Instruct222https://huggingface.co/meta-llama/Meta-
Llama-3-8B-Instruct that is already finetuned with RLHF, we directly apply $3$
iterations of SELM training.

###  5.2 Experiment Results

We first report the performance of SELM and the baselines on the instruction-
following chat benchmarks AlpacaEval 2.0 [14] and MT-Bench [71] in Table 1. We
can observe that for AlpacaEval 2.0, SELM significantly boosts Zephyr-7B-SFT
and Llama-3-8B-Instruct, achieving length-controlled (LC) win rate
improvements of $+16.24\%$ and $+11.75\%$, respectively. This enhancement
results in models that are competitive with or even superior to much larger
LLMs, such as Yi-34B-Chat [65] and Llama-3-70B-Instruct. For the multi-turn
MT-Bench, which exhibits higher variance, we report the average scores of SELM
and DPO baselines across $3$ runs. We observe that SELM improves the scores by
$+2.31$ and $+0.32$, respectively. Furthermore, the proposed method self-
explores and enhances the model monotonically, with consistent performance
improvements in each iteration. This validates the robustness of our
algorithm. Compared to other iterative post-training algorithms, such as SPIN
[7], DNO [50], and SPPO [61], SELM gains more improvements on both benchmarks
when using the weaker base model (Zephyr-7B-SFT), and achieves the best
performance when using Llama-3-8B-Instruct as the base model.

Figure 2: Pairwise length-controlled win rates comparison between SELM, iterative DPO, and base models on the AlpacaEval 2.0 benchmark. Scores represent the LC win rates of the row models against the column models. Models positioned in higher rows have higher LC win rates against the base model and thus better performance. | AlpacaEval 2.0 | MT-Bench
---|---|---
Model |  | LC Win Rate
---
| Win Rate
---
| Avg. len
---
Avgerage |  | 1st Turn
---
| 2nd Turn
---
Zephyr-7B-SFT | 8.01 | 4.63 | 916 | 5.30 | 5.63 | 4.97
Zephyr-7B-DPO | 15.41 | 14.44 | 1752 | 7.31 | 7.55 | 7.07
DPO Iter 1 (Zephyr) | 20.53 | 16.69 | 1598 | 7.53 | 7.81 | 7.25
DPO Iter 2 (Zephyr) | 22.12 | 19.82 | 1717 | 7.55 | 7.85 | 7.24
DPO Iter 3 (Zephyr) |  22.19 ($\uparrow$14.18) | 19.88 | 1717 |  7.46 ($\uparrow$2.16) | 7.85 | 7.06
SELM Iter 1 (Zephyr) | 20.52 | 17.23 | 1624 | 7.53 | 7.74 | 7.31
SELM Iter 2 (Zephyr) | 21.84 | 18.78 | 1665 | 7.61 | 7.85 | 7.38
SELM Iter 3 (Zephyr) |  24.25($\uparrow$16.24) | 21.05 | 1694 |  7.61 ($\uparrow$2.31) | 7.74 | 7.49
Llama-3-8B-Instruct | 22.92 | 22.57 | 1899 | 7.93 | 8.47 | 7.38
DPO Iter 1 (Llama3-It) | 30.89 | 31.60 | 1979 | 8.07 | 8.44 | 7.70
DPO Iter 2 (Llama3-It) | 33.91 | 32.95 | 1939 | 7.99 | 8.39 | 7.60
DPO Iter 3 (Llama3-It) |  33.17 ($\uparrow$10.25) | 32.18 | 1930 |  8.18 ($\uparrow$0.25) | 8.60 | 7.77
SELM Iter 1 (Llama3-It) | 31.09 | 30.90 | 1956 | 8.09 | 8.57 | 7.61
SELM Iter 2 (Llama3-It) | 33.53 | 32.61 | 1919 | 8.18 | 8.69 | 7.66
SELM Iter 3 (Llama3-It) |  34.67 ($\uparrow$11.75) | 34.78 | 1948 |  8.25 ($\uparrow$0.32) | 8.53 | 7.98
SPIN | 7.23 | 6.54 | 1426 | 6.54 | 6.94 | 6.14
Orca-2.5-SFT | 10.76 | 6.99 | 1174 | 6.88 | 7.72 | 6.02
DNO (Orca-2.5-SFT) | 22.59 | 24.97 | 2228 | 7.48 | 7.62 | 7.35
Mistral-7B-Instruct-v0.2 | 19.39 | 15.75 | 1565 | 7.51 | 7.78 | 7.25
SPPO (Mistral-it) | 28.53 | 31.02 | 2163 | 7.59 | 7.84 | 7.34
Yi-34B-Chat | 27.19 | 21.23 | 2123 | 7.90 | - | -
Llama-3-70B-Instruct | 33.17 | 33.18 | 1919 | 9.01 | 9.21 | 8.80
GPT-4 Turbo (04/09) | 55.02 | 46.12 | 1802 | 9.19 | 9.38 | 9.00
Table 1: Results on AlpacaEval 2.0 and MT-Bench. Names inside the brackets are
the base models that are aligned based upon. The red arrows indicate the
increment or decrement from the base model. Compared to iterative DPO and
other online alignment baselines, SELM gains more improvements based on the
weaker Zephyr-7B-SFT model and achieves superior performance that is
competitive with much larger SOTA models when finetuned from
Llama-3-8B-Instruct.

We also conduct pairwise comparisons between SELM, iterative DPO, and the base
models to validate the effectiveness of our method. The results for AlpacaEval
2.0 are shown in Figure 2. We observe that with the same number of training
iterations and data, SELM consistently outperforms the iterative DPO
counterpart. Additionally, when using Zephyr-7B-SFT as the base model, SELM
outperforms iterative DPO even when the latter is trained with twice the data.

Models |  | GSM8K
---
(8-s CoT)
| HellaSwag
---
(10-s)
| ARC
---
(25-s)
| TruthfulQA
---
(0-s)
| EQ
---
(0-s)
| OBQA
---
(10-s)
Average
Zephyr-7B-SFT | 43.8 | 82.2 | 57.4 | 43.6 | 39.1 | 35.4 | 50.3
Zephyr-7B-DPO | 47.2 | 84.5 | 61.9 | 45.5 | 65.2 | 38.0 | 57.0
DPO Iter 1 (Zephyr) | 45.5 | 85.2 | 62.1 | 52.4 | 68.4 | 39.0 | 58.8
DPO Iter 2 (Zephyr) | 44.9 | 85.4 | 62.0 | 53.1 | 69.3 | 39.4 | 59.0
DPO Iter 3 (Zephyr) | 43.2 | 85.2 | 60.8 | 52.5 | 69.1 | 39.6 | 58.4
SELM Iter 1 (Zephyr) | 46.3 | 84.8 | 62.9 | 52.9 | 68.8 | 39.6 | 59.2
SELM Iter 2 (Zephyr) | 46.2 | 85.4 | 62.1 | 53.1 | 69.3 | 39.6 | 59.3
SELM Iter 3 (Zephyr) | 43.8 | 85.4 | 61.9 | 52.4 | 69.9 | 39.8 | 58.9
Llama-3-8B-Instruct | 76.7 | 78.6 | 60.8 | 51.7 | 61.8 | 38.0 | 61.3
DPO Iter 1 (Llama3-It) | 78.5 | 81.7 | 63.9 | 55.5 | 64.1 | 42.6 | 64.4
DPO Iter 2 (Llama3-It) | 79.4 | 81.7 | 64.4 | 56.4 | 64.3 | 42.6 | 64.8
DPO Iter 3 (Llama3-It) | 80.1 | 81.7 | 64.1 | 56.5 | 64.1 | 42.6 | 64.8
SELM Iter 1 (Llama3-It) | 78.7 | 81.7 | 64.5 | 55.4 | 64.1 | 42.4 | 64.5
SELM Iter 2 (Llama3-It) | 79.3 | 81.8 | 64.7 | 56.5 | 64.2 | 42.6 | 64.9
SELM Iter 3 (Llama3-It) | 80.1 | 81.8 | 64.3 | 56.5 | 64.2 | 42.8 | 65.0
SPIN | 44.7 | 85.9 | 65.9 | 55.6 | 54.4 | 39.6 | 57.7
Mistral-7B-Instruct-v0.2 | 43.4 | 85.3 | 63.4 | 67.5 | 65.9 | 41.2 | 61.1
SPPO (Mistral-it) | 42.4 | 85.6 | 65.4 | 70.7 | 56.5 | 40.0 | 60.1
Table 2: Performance comparison between SELM and the baselines on academic
multi-choice QA benchmarks in standard zero-shot, few-shot, and CoT settings.
Here, n-s refers to n-shot. The red and blue texts represent the best and the
second-best results.

Beyond instruction-following benchmarks, we also evaluate SELM and the
baselines on several academic benchmarks, including GSM8K [10], HellaSwag
[68], ARC challenge [9], TruthfulQA [33], EQ-Bench [45], and OpenBookQA (OBQA)
[39]. To better reflect the capabilities of LLMs, we adopt various settings
for these benchmarks, including zero-shot, few-shot, and few-shot Chain-of-
Thought (CoT) settings. The accuracy results for these multiple-choice QA
benchmarks are provided in Table 2. It can be observed that both our method
and the baselines can degrade after the RLHF phase on some benchmarks, which
is known as the alignment tax [2; 40; 30]. Nevertheless, our method is still
able to improve the base models on most of the benchmarks and offers the best
overall performance.

We note that SELM is one of the instantiations of the proposed self-
exploration objective in (1.1), with reparameterized reward functions and
algorithm-specific designs described in Section 3.2, such as the dataset
partition and update rule. However, this objective is not restricted to the
current implementation and can also be directly applied to any other online
alignment framework, with or without a separate reward model, regardless of
differences in algorithm designs. Thus, the proposed method is orthogonal to
and can be integrated directly into the recent online RLHF workflows [13; 62;
23] that incorporate additional delicate designs with carefully curated
datasets.

###  5.3 Ablation Study

Figure 3: Ablation on the optimism coefficient $\alpha$ and the change of the
reward distribution. Left: The length-controlled win rates of SELM with
different $\alpha$ on AlpacaEval 2.0. Middle: Comparison of reward
distributions at iteration 2 with different $\alpha$. Right: SELM initially
explores and then shifts to higher-reward regions as more training iterations
are performed.

We first provide ablation studies to better understand the explorative
optimism term. We begin by investigating the effect of the optimism
coefficient $\alpha$. In Figure 3 (Left), we plot the LC win rates of SELM
when using Zephyr-7B-SFT as the base model for different $\alpha$ in the
AlpacaEval 2.0 benchmark. We find that setting a small $\alpha$, such as
$0.0001$, leads to very similar behaviors to the iterative DPO ($\alpha=0$)
baseline, while SELM with a large $\alpha$ may become overly optimistic and
thus not very effective. These results meet our expectations, suggesting that
proper values of $\alpha$ are essential for achieving the best trade-off
between exploration and exploitation.

Next, we study the difference in reward distributions with varying $\alpha$
and iterations. Specifically, we greedily sample from the LLM using prompts
from the holdout test set (2k in total) of UltraFeedback and generate rewards
for these responses with PairRM. We then calculate the fraction of data that
lies in each partition of reward values. The results for different $\alpha$
values of SELM Iter 2 (Zephyr) are shown in Figure 3 (Middle), which indicate
that increasing $\alpha$ results in distributions that are more concentrated
in higher-reward regions.

Figure 4: Difference of implicit reward between SELM and DPO on the chosen and
rejected responses. SELM assigns a higher implicit reward than DPO for both
responses.

Additionally, Figure 3 (Right) demonstrates that the reward distribution
shifts to the right (higher) as more training iterations are performed. This
shift corresponds to an initial exploration phase, where the LLM generates
uncertain responses of varying quality, followed by an exploitation phase as
feedback is incorporated and more training data is collected.

We also conduct ablation studies on the implicit reward captured by the SELM
and DPO models. Recall that for both SELM and DPO, the implicit reward takes
the form of $\widehat{r}_{\theta}(x,y)=\beta(\log\pi_{\theta}(y\mid
x)-\log\pi_{\text{ref}}(y\mid x))$. We calculate the reward difference
$\widehat{r}_{\text{SELM}}(x,y)-\widehat{r}_{\text{DPO}}(x,y)$ for each prompt
$x$ in the UltraFeedback holdout test set. Here, we study the implicit reward
of the good (chosen) and bad (rejected) responses, so $y=y_{w}$ or $y=y_{l}$.
We then sort the reward difference and plot the results for Zephyr-based
models after iteration 1 in Figure 4. The plot clearly shows that for both
chosen and rejected responses, SELM produces higher implicit rewards compared
to DPO, aligning with the proposed optimistically biased self-exploration
objective.

##  6 Conclusion & Future Work

In this paper, we introduced an active preference elicitation method for the
online alignment of large language models. By incorporating an optimism term
into the reward-fitting objective, the proposed bilevel self-exploring
objective effectively balances between exploiting observed data and exploring
potentially high-reward regions. Unlike standard online RLHF algorithms that
passively explore the response space by sampling from the training LLM, whose
sole objective is maximizing the expected learned reward, our method actively
seeks diverse and high-quality responses. This self-exploration mechanism
helps mitigate the risk of premature convergence and overfitting when the
reward model is only locally accurate. To optimize this bilevel objective, we
solve the inner-level problem and reparameterize the reward with the LLM
policy, resulting in a simple yet novel iterative alignment algorithm called
Self-Exploring Language Models (SELM). Compared to DPO, SELM improves the
exploration efficiency by selectively favoring responses with high potential
rewards rather than indiscriminately sampling unseen responses.

Our experiments, conducted with Zephyr-7B-SFT and Llama-3-8B-Instruct models,
demonstrated the efficacy of SELM. Finetuning on the UltraFeedback dataset and
leveraging PairRM for AI feedback, SELM achieved substantial improvements in
performance on AlpacaEval 2.0, MT-Bench, and academic benchmarks. These
results underscore the ability of SELM to enhance the alignment and
capabilities of large language models by promoting more diverse and high-
quality responses. Since the proposed technique is orthogonal to the adopted
online RLHF workflow, it will be interesting to apply our method within more
sophisticated alignment frameworks with advanced designs, which we would like
to leave as future work.

## References

  * [1] Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al.  Phi-3 technical report: A highly capable language model locally on your phone.  arXiv preprint arXiv:2404.14219, 2024.
  * [2] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al.  A general language assistant as a laboratory for alignment.  arXiv preprint arXiv:2112.00861, 2021.
  * [3] Peter Auer.  Using confidence bounds for exploitation-exploration trade-offs.  Journal of Machine Learning Research, 3(Nov):397–422, 2002.
  * [4] Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos.  A general theoretical paradigm to understand learning from human preferences.  arXiv preprint arXiv:2310.12036, 2023.
  * [5] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  arXiv preprint arXiv:2204.05862, 2022.
  * [6] Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, et al.  Human alignment of large language models through online preference optimisation.  arXiv preprint arXiv:2403.08635, 2024.
  * [7] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu.  Self-play fine-tuning converts weak language models to strong language models.  arXiv preprint arXiv:2401.01335, 2024.
  * [8] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine.  Deep reinforcement learning in a handful of trials using probabilistic dynamics models.  Advances in neural information processing systems, 31, 2018.
  * [9] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  arXiv preprint arXiv:1803.05457, 2018.
  * [10] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.  Training verifiers to solve math word problems.  arXiv preprint arXiv:2110.14168, 2021.
  * [11] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun.  Ultrafeedback: Boosting language models with high-quality feedback.  arXiv preprint arXiv:2310.01377, 2023.
  * [12] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou.  Enhancing chat language models by scaling high-quality instructional conversations.  arXiv preprint arXiv:2305.14233, 2023.
  * [13] Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang.  Rlhf workflow: From reward modeling to online rlhf.  arXiv e-prints, pages arXiv–2405, 2024.
  * [14] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto.  Length-controlled alpacaeval: A simple way to debias automatic evaluators.  arXiv preprint arXiv:2404.04475, 2024.
  * [15] Vikranth Dwaracherla, Seyed Mohammad Asghari, Botao Hao, and Benjamin Van Roy.  Efficient exploration for llms.  arXiv preprint arXiv:2402.00396, 2024.
  * [16] Arpad E Elo and Sam Sloan.  The rating of chessplayers: Past and present.  Ishi Press International, 1978.
  * [17] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela.  Kto: Model alignment as prospect theoretic optimization.  arXiv preprint arXiv:2402.01306, 2024.
  * [18] Leo Gao, John Schulman, and Jacob Hilton.  Scaling laws for reward model overoptimization.  In International Conference on Machine Learning, pages 10835–10866. PMLR, 2023.
  * [19] Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al.  Reinforced self-training (rest) for language modeling.  arXiv preprint arXiv:2308.08998, 2023.
  * [20] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al.  Textbooks are all you need.  arXiv preprint arXiv:2306.11644, 2023.
  * [21] Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al.  Direct language model alignment from online ai feedback.  arXiv preprint arXiv:2402.04792, 2024.
  * [22] Braden Hancock Hoang Tran, Chris Glaze.  Snorkel-mistral-pairrm-dpo.  2024\.
  * [23] Jian Hu, Xibin Wu, Weixun Wang, Xianyu, Dehao Zhang, and Yu Cao.  Openrlhf: An easy-to-use, scalable and high-performance rlhf framework, 2024.
  * [24] Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al.  Camels in a changing climate: Enhancing lm adaptation with tulu 2.  arXiv preprint arXiv:2311.10702, 2023.
  * [25] Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin.  Llm-blender: Ensembling large language models with pairwise ranking and generative fusion.  arXiv preprint arXiv:2306.02561, 2023.
  * [26] Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan.  Provably efficient reinforcement learning with linear function approximation.  In Conference on learning theory, pages 2137–2143. PMLR, 2020.
  * [27] Martin Josifoski, Marija Sakota, Maxime Peyrard, and Robert West.  Exploiting asymmetry for synthetic training data generation: Synthie and the case of information extraction.  arXiv preprint arXiv:2303.04132, 2023.
  * [28] Dahyun Kim, Yungi Kim, Wonho Song, Hyeonwoo Kim, Yunsu Kim, Sanghoon Kim, and Chanjun Park.  sdpo: Don’t use your data all at once.  arXiv preprint arXiv:2403.19270, 2024.
  * [29] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al.  Openassistant conversations-democratizing large language model alignment.  Advances in Neural Information Processing Systems, 36, 2024.
  * [30] Shengzhi Li, Rongyu Lin, and Shichao Pei.  Multi-modal preference alignment remedies regression of visual instruction tuning on language model.  arXiv preprint arXiv:2402.10884, 2024.
  * [31] Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis.  Self-alignment with instruction backtranslation.  arXiv preprint arXiv:2308.06259, 2023.
  * [32] Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee.  Textbooks are all you need ii: phi-1.5 technical report.  arXiv preprint arXiv:2309.05463, 2023.
  * [33] Stephanie Lin, Jacob Hilton, and Owain Evans.  Truthfulqa: Measuring how models mimic human falsehoods.  arXiv preprint arXiv:2109.07958, 2021.
  * [34] Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, and Andrew M. Dai.  Best practices and lessons learned on synthetic data for language models, 2024.
  * [35] Zhihan Liu, Miao Lu, Wei Xiong, Han Zhong, Hao Hu, Shenao Zhang, Sirui Zheng, Zhuoran Yang, and Zhaoran Wang.  Maximize to explore: One objective function fusing estimation, planning, and exploration.  Advances in Neural Information Processing Systems, 36, 2024.
  * [36] Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang.  Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer, 2024.
  * [37] Xiuyuan Lu and Benjamin Van Roy.  Ensemble sampling.  Advances in neural information processing systems, 30, 2017.
  * [38] Meta.  Introducing meta llama 3: The most capable openly available llm to date.  2024\.
  * [39] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  arXiv preprint arXiv:1809.02789, 2018.
  * [40] Michael Noukhovitch, Samuel Lavoie, Florian Strub, and Aaron C Courville.  Language model alignment with elastic reset.  Advances in Neural Information Processing Systems, 36, 2024.
  * [41] Ian Osband, Daniel Russo, and Benjamin Van Roy.  (more) efficient reinforcement learning via posterior sampling.  Advances in Neural Information Processing Systems, 26, 2013.
  * [42] Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy.  Approximate thompson sampling via epistemic neural networks.  In Uncertainty in Artificial Intelligence, pages 1586–1595. PMLR, 2023.
  * [43] Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy.  Epistemic neural networks.  Advances in Neural Information Processing Systems, 36, 2024.
  * [44] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  Advances in neural information processing systems, 35:27730–27744, 2022.
  * [45] Samuel J Paech.  Eq-bench: An emotional intelligence benchmark for large language models.  arXiv preprint arXiv:2312.06281, 2023.
  * [46] Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White.  Smaug: Fixing failure modes of preference optimisation with dpo-positive.  arXiv preprint arXiv:2402.13228, 2024.
  * [47] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao.  Instruction tuning with gpt-4.  arXiv preprint arXiv:2304.03277, 2023.
  * [48] Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn.  From $r$ to $q^{*}$: Your language model is secretly a q-function.  arXiv preprint arXiv:2404.12358, 2024.
  * [49] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  Advances in Neural Information Processing Systems, 36, 2024.
  * [50] Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie.  Direct nash optimization: Teaching language models to self-improve with general preferences.  arXiv preprint arXiv:2404.03715, 2024.
  * [51] Daniel Russo and Benjamin Van Roy.  Eluder dimension and the sample complexity of optimistic exploration.  Advances in Neural Information Processing Systems, 26, 2013.
  * [52] Malcolm Strens.  A bayesian framework for reinforcement learning.  In ICML, volume 2000, pages 943–950, 2000.
  * [53] Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan.  Principle-driven self-alignment of language models from scratch with minimal human supervision.  Advances in Neural Information Processing Systems, 36, 2024.
  * [54] Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, et al.  Understanding the performance gap between online and offline alignment algorithms.  arXiv preprint arXiv:2405.08448, 2024.
  * [55] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * [56] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alexander M. Rush, and Thomas Wolf.  The alignment handbook.  https://github.com/huggingface/alignment-handbook, 2023.
  * [57] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al.  Zephyr: Direct distillation of lm alignment.  arXiv preprint arXiv:2310.16944, 2023.
  * [58] Xiyao Wang, Jiuhai Chen, Zhaoyang Wang, Yuhang Zhou, Yiyang Zhou, Huaxiu Yao, Tianyi Zhou, Tom Goldstein, Parminder Bhatia, Furong Huang, and Cao Xiao.  Enhancing visual-language modality alignment in large vision language models via self-improvement, 2024.
  * [59] Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al.  How far can camels go? exploring the state of instruction tuning on open resources.  Advances in Neural Information Processing Systems, 36, 2024.
  * [60] Shengguang Wu, Keming Lu, Benfeng Xu, Junyang Lin, Qi Su, and Chang Zhou.  Self-evolved diverse data sampling for efficient instruction tuning.  arXiv preprint arXiv:2311.08182, 2023.
  * [61] Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu.  Self-play preference optimization for language model alignment.  arXiv preprint arXiv:2405.00675, 2024.
  * [62] Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang.  Gibbs sampling from human feedback: A provable kl-constrained framework for rlhf.  arXiv preprint arXiv:2312.11456, 2023.
  * [63] Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston.  Some things are more cringe than others: Preference optimization with the pairwise cringe loss.  arXiv preprint arXiv:2312.16682, 2023.
  * [64] Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu.  Is dpo superior to ppo for llm alignment? a comprehensive study.  arXiv preprint arXiv:2404.10719, 2024.
  * [65] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al.  Yi: Open foundation models by 01. ai.  arXiv preprint arXiv:2403.04652, 2024.
  * [66] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston.  Self-rewarding language models.  arXiv preprint arXiv:2401.10020, 2024.
  * [67] Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston.  Iterative reasoning preference optimization.  arXiv e-prints, pages arXiv–2404, 2024.
  * [68] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  Hellaswag: Can a machine really finish your sentence?  arXiv preprint arXiv:1905.07830, 2019.
  * [69] Shenao Zhang.  Conservative dual policy optimization for efficient model-based reinforcement learning.  Advances in neural information processing systems, 35:25450–25463, 2022.
  * [70] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu.  Slic-hf: Sequence likelihood calibration with human feedback.  arXiv preprint arXiv:2305.10425, 2023.
  * [71] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  Advances in Neural Information Processing Systems, 36, 2024.
  * [72] Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao.  Starling-7b: Improving llm helpfulness and harmlessness with rlaif, November 2023.

##  Appendix A Derivations in Section 3.1

We begin by deriving (3.2). The solution for the inner-level optimization
problem of (3.1) is as follows:

$\displaystyle\max_{\pi}\mathcal{F}(\pi;r)$
$\displaystyle=\max_{\pi}\mathbb{E}_{\begin{subarray}{c}x\sim\mathcal{D}_{t},y\sim\pi(\cdot\mid
x)\\\ y^{\prime}\sim\pi_{\text{ref}}(\cdot\mid
x)\end{subarray}}\Bigl{[}r(x,y)-r(x,y^{\prime})\Bigr{]}-\beta\mathbb{D}_{\text{KL}}(\pi\,||\,\pi_{\text{ref}})$
$\displaystyle=\mathbb{E}_{x\sim\mathcal{D}_{t}}\Bigl{[}\beta\log\mathbb{E}_{y\sim\pi_{\text{ref}}(\cdot\mid
x)}\bigl{[}\exp(r(x,y)/\beta)\bigr{]}\Bigr{]}-\mathbb{E}_{x\sim\mathcal{D}_{t},y^{\prime}\sim\pi_{\text{ref}}(\cdot\mid
x)}\bigl{[}r(x,y^{\prime})\bigr{]}$ (A.1)

When the reward $r$ is reparameterized by
$\widehat{r}_{\theta}(x,y)=\beta(\log\pi_{\theta}(y\mid
x)-\log\pi_{\text{ref}}(y\mid x))$, we have that the first term in (A) is $0$.
The bilevel objective (3.1) then becomes

$\displaystyle\max_{r}-\mathcal{L}_{\text{lr}}(r;\mathcal{D}_{t})-\alpha\mathbb{E}_{x\sim\mathcal{D},y^{\prime}\sim\pi_{\text{ref}}(\cdot\mid
x)}\bigl{[}r(x,y^{\prime})\bigr{]}.$

By reparameterizing the reward with the LLM, we obtain the desired results in
(3.2).

Then we provide the derivation of (3.1). We primarily consider the gradient of
the newly incorporated term
$\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\text{ref}}(\cdot\mid
x)}[\log\pi_{\theta}(y\mid x)]$. Specifically, we have

$\displaystyle\nabla_{\theta}\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\text{ref}}(\cdot\mid
x)}\bigl{[}\log\pi_{\theta}(y\mid x)\bigr{]}$
$\displaystyle=\mathbb{E}_{x\sim\mathcal{D}}\Bigl{[}\sum_{y}\pi_{\text{ref}}(y\mid
x)\nabla_{\theta}\log\pi_{\theta}(y\mid x)\Bigr{]}$
$\displaystyle=\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\theta}}\Bigl{[}\frac{\pi_{\text{ref}}(y\mid
x)}{\pi_{\theta}(y\mid x)}\nabla_{\theta}\log\pi_{\theta}(y\mid x)\Bigr{]}$
$\displaystyle=\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\theta}}\Bigl{[}\exp\bigl{(}-\widehat{r}_{\theta}(x.y)/\beta\bigr{)}\nabla_{\theta}\log\pi_{\theta}(y\mid
x)\Bigr{]}.$

For the derivation of the DPO gradient
$\nabla_{\theta}\mathcal{L}_{\text{DPO}}(\pi_{\theta};\mathcal{D}_{t})$, we
refer the readers to [49].

##  Appendix B Proof of Theorem 3.1

###### Proof.

The solution to the KL-constrained reward minimization objective (3.4) is

$\displaystyle\pi^{\min}_{\rho}(y\mid x)=\pi_{\rho}(y\mid
x)\exp\bigl{(}-\widehat{r}_{\rho}(x,y)/\beta\bigr{)}/Z(x),$

where $Z(x)=\sum_{y}\pi_{\rho}(y\mid
x)\exp(-\widehat{r}_{\rho}(x,y)/\beta)=1$. Then we have
$\pi^{\min}_{\rho}(y\mid x)=\pi_{\text{ref}}(y\mid x)$, i.e., the reference
policy $\pi_{\text{ref}}$ achieves the lowest implicit reward reparameterized
by any $\rho$. ∎

##  Appendix C Experiment Setup

In experiments, we use the Alignment Handbook [56] framework as our codebase.
We find the best hyperparameter settings by conducting a grid search over the
iteration number, batch size, learning rate, and label update rule for the
iterative DPO baseline. The results for the Zephyr-based models are shown in
Figure 5. Specifically, we find that using the same amount of data, updating
the model too many iterations can lead to instability. So we set the iteration
number to $3$ for Llama3-It-based and Zephyr-based models (excluding the first
iteration of DPO training). Besides, we observe that choosing different batch
sizes has a large effect on the models’ performance and the optimal batch size
heavily depends on the model architecture. In experiments, we set the batch
size to $256$ and $128$ for the Zephyr-based and Llama3-It-based models,
respectively. For the learning rate, we consider three design choices: cyclic
learning rate with constant cycle amplitude, linearly decayed cycle amplitude,
and decayed cycle amplitude at the last iteration. We find that a decaying
cycle amplitude performs better than constant amplitudes in general. Thus, for
Zephyr-based models, we set the learning to $5e-7$ for the first three
iterations and $1e-7$ for the last iteration. In each iteration, the warmup
ratio is $0.1$. For Llama3-It-based models, we use a linearly decayed learning
rate from $5e-7$ to $1e-7$ within $3$ iterations with the same warmup ratio.
We also test two update ways for the preference data. One is to rank
$y_{w},y_{l},y_{\text{ref}}$ and keep the best and worst responses in the
updated dataset, which is the setting that is described in the main paper. The
other is to compare $y_{w}$ and $y_{\text{ref}}$ and replace the chosen or
rejected response by $y_{\text{ref}}$ based on the comparison result. We find
that the former design performs better than the latter. We also compared with
stepwise DPO [28], which updates the reference model at each iteration but
uses the original dataset instead of the updated one. This demonstrates that
exploring and collecting new data is necessary.

Figure 5: Ablation of the iterative DPO baseline. We conduct a grid search
over the iteration number, batch size, learning rate, and designs of the
dataset update rule.

For the proposed SELM method, we follow the above hyperparameter settings for
a fair comparison. The optimism coefficient $\alpha$ is searched over $0.005$,
$0.001$, $0.0005$, and $0.0001$ and is selected based on the average external
reward on the holdout test set of UltraFeedback. We set $\alpha=0.001$ for
Zephyr-based SELM and $\alpha=0.0001$ for Llama3-It-based SELM. For training
SELM based on other models, we recommend setting $\alpha=0.005$ or $0.001$ as
it shows minimal sensitivity to variations.


G_{\mathcal{M}}\in{\mathcal{R}}^{*}_{\mathcal{M}}\end{subarray}}\mathds{1}_{{\mathcal{R}}_{1}^{*}}(G_{2}+G_{\mathcal{M}})\,\hat{h}(G_{\mathcal{M}},G_{2})e^{i[G_{\mathcal{M}}\cdot\omega_{\mathcal{M}}+G_{2}\cdot\omega_{2}]}\,.\end{split}$
(C.2)

Now observe that, due to (2.45), (2.46),

$\displaystyle\begin{split}D_{1\to
2}^{T}G_{2}\,&=\,-A_{\mathcal{M}}^{-T}A_{2}^{T}G_{2}\,\in{\mathcal{R}}^{*}_{\mathcal{M}}\,,\\\
(I-D_{1\to
2}^{T})G_{2}\,&=\,A_{1}^{-T}A_{2}^{T}G_{2}\,\in{\mathcal{R}}_{1}^{*}\,.\end{split}$
(C.3)

Thus, and using the fact that
$({\mathcal{R}}_{1},{\mathcal{R}}_{\mathcal{M}})$ is incommensurate, we have
that

$\mathds{1}_{{\mathcal{R}}_{1}^{*}}(G_{2}+G_{\mathcal{M}})\,=\,\mathds{1}_{{\mathcal{R}}_{1}^{*}}(D_{1\to
2}^{T}G_{2}+G_{\mathcal{M}})\,=\,\delta_{G_{\mathcal{M}},-D_{1\to
2}^{T}G_{2}}\,.$ (C.4)

Employing (C.4), (C.2) implies

$\displaystyle\begin{split}&\lim_{N\to\infty}{\mathcal{A}}_{{\mathcal{R}}_{1}(N)}(h)(\omega_{\mathcal{M}},\omega_{2})\\\
&=\,\sum_{G_{2}\in{\mathcal{R}}_{2}^{*}}\hat{h}(-D_{1\to
2}^{T}G_{2},G_{2})e^{i[-D_{1\to
2}^{T}G_{2}\cdot\omega_{\mathcal{M}}+G_{2}\cdot\omega_{2}]}\,.\end{split}$
(C.5)

Recalling (2.30) and (3.14), and using Fubini, we have that

$\displaystyle\begin{split}&\sum_{G_{2}\in{\mathcal{R}}_{2}^{*}}\hat{h}(-D_{1\to
2}^{T}G_{2},G_{2})e^{i[-D_{1\to
2}^{T}G_{2}\cdot\omega_{\mathcal{M}}+G_{2}\cdot\omega_{2}]}\\\
\,=&\sum_{G_{2}\in{\mathcal{R}}_{2}^{*}}\,\ThisStyle{\ensurestackMath{\stackinset{c}{.2\LMpt}{c}{.5\LMpt}{\SavedStyle-}{\SavedStyle\phantom{\int}}}\kern-16.56937pt}\int_{\Gamma_{\mathcal{M}}}\mathrm{d}x\
{\mathcal{F}}_{\Gamma_{2}}[h(x,\cdot)](G_{2})e^{iG_{2}\cdot[D_{1\to
2}(x-\omega_{\mathcal{M}})+\omega_{2}]}\,.\end{split}$ (C.6)

Using Fubini again, we thus find that

$\displaystyle\begin{split}&\sum_{G_{2}\in{\mathcal{R}}_{2}^{*}}\hat{h}(-D_{1\to
2}^{T}G_{2},G_{2})e^{i[-D_{1\to
2}^{T}G_{2}\cdot\omega_{\mathcal{M}}+G_{2}\cdot\omega_{2}]}\\\
=\,&\,\ThisStyle{\ensurestackMath{\stackinset{c}{.2\LMpt}{c}{.5\LMpt}{\SavedStyle-}{\SavedStyle\phantom{\int}}}\kern-16.56937pt}\int_{\Gamma_{\mathcal{M}}}\mathrm{d}x\
\sum_{G_{2}\in{\mathcal{R}}_{2}^{*}}e^{iG_{2}\cdot[D_{1\to
2}(x-\omega_{\mathcal{M}})+\omega_{2}]}{\mathcal{F}}_{\Gamma_{2}}[h(x,\cdot)](G_{2})\\\
=\,&\,\ThisStyle{\ensurestackMath{\stackinset{c}{.2\LMpt}{c}{.5\LMpt}{\SavedStyle-}{\SavedStyle\phantom{\int}}}\kern-16.56937pt}\int_{\Gamma_{\mathcal{M}}}\mathrm{d}x\
h\big{(}x,D_{1\to 2}(x-\omega_{\mathcal{M}})+\omega_{2}\big{)}\,,\end{split}$
(C.7)

where, in the second-to-last step, we used the Fourier inversion formula.
Substituting $x\to A_{1}x+\omega_{\mathcal{M}}$, and recalling (2.45), yields

$\displaystyle\begin{split}&\,\ThisStyle{\ensurestackMath{\stackinset{c}{.2\LMpt}{c}{.5\LMpt}{\SavedStyle-}{\SavedStyle\phantom{\int}}}\kern-16.56937pt}\int_{\Gamma_{\mathcal{M}}}\mathrm{d}x\
h\big{(}x,D_{1\to 2}(x-\omega_{\mathcal{M}})+\omega_{2}\big{)}\,=\\\
&\,\ThisStyle{\ensurestackMath{\stackinset{c}{.2\LMpt}{c}{.5\LMpt}{\SavedStyle-}{\SavedStyle\phantom{\int}}}\kern-16.56937pt}\int_{A_{1}^{-1}\Gamma_{\mathcal{M}}}\mathrm{d}x\
h\big{(}A_{1}x+\omega_{\mathcal{M}},(A_{1}-A_{2})x+\omega_{2}\big{)}\end{split}$
(C.8)

Collecting (C.5), (C.7), and (C.8), we conclude the proof. ∎

###### Proof of Proposition 3.12.

We start by calculating for
$G_{\mathcal{M}}=B_{\mathcal{M}}n\in{\mathcal{R}}^{*}_{\mathcal{M}}$

$\begin{split}A_{1}^{T}G_{\mathcal{M}}\,&=\,A_{1}^{T}B_{\mathcal{M}}n\\\
&=\,2\pi(I-A_{1}^{T}A_{2}^{-T})n\\\
&=2\pi(I-A^{T}R_{\theta}A^{-T})n\,,\end{split}$ (C.9)

where we recall (2.1), (2.5), and (2.7). Similarly, we find that

$A_{2}^{T}G_{\mathcal{M}}\,=\,2\pi(A^{T}R_{-\theta}A^{-T}-I)n\,.$ (C.10)

In particular, using (C.9), (C.10), and (3.45), a straight-forward calculation
yields

$\displaystyle\begin{split}&\mathds{1}_{{\mathcal{R}}_{j}^{*}}^{(N)}(B_{\mathcal{M}}n)\,=\\\
&\prod_{\ell=1}^{2}\frac{\sin\Big{(}\pi\operatorname{dist}\big{(}(A^{T}R_{(-1)^{j+1}\theta}A^{-T}n)_{\ell},{\mathbb{Z}}\big{)}\big{(}2N+1\big{)}\Big{)}}{(2N+1)\sin\big{(}\pi\operatorname{dist}\big{(}(A^{T}R_{(-1)^{j+1}\theta}A^{-T}n)_{\ell},{\mathbb{Z}}\big{)}\big{)}}\,.\end{split}$
(C.11)

Observe that

$\frac{\operatorname{dist}(x,{\mathbb{Z}})}{|\sin(\pi
x)|}\,=\,\frac{\operatorname{dist}(x,{\mathbb{Z}})}{|\sin\big{(}\pi\operatorname{dist}(x,{\mathbb{Z}})\big{)}|}\,\leq\,\frac{1}{\pi}\sup_{0<y<\pi/2}\frac{y}{\sin(y)}\,\leq\,\frac{1}{2}\,.$
(C.12)

Let $n\in{\mathbb{Z}}^{2}\setminus\\{0\\}$. Recalling (LABEL:eq-
appdelNi@MoireML), we thus obtain that

$\begin{split}\@ADDCLASS{ltx_eqn_lefteqn}$\displaystyle\max_{\ell=1,2}\operatorname{dist}\big{(}(A^{T}R_{(-1)^{j+1}\theta}A^{-T}n)_{\ell},{\mathbb{Z}}\big{)}\mathds{1}_{{\mathcal{R}}_{j}^{*}}^{(N)}(B_{\mathcal{M}}n)$\mbox{}\hfil\\\
&\leq\,\frac{1}{\pi(2N+1)}\sup_{0<y<\pi/2}\frac{y}{\sin(y)}\\\
&=\,\frac{1}{2(2N+1)}\,.\end{split}$ (C.13)

Using (C.13) and employing the fact that
$\sqrt{a^{2}+b^{2}}\leq\sqrt{2}\max\\{|a|,|b|\\}$, we find that

$\displaystyle\begin{split}\@ADDCLASS{ltx_eqn_lefteqn}$\displaystyle|\mathds{1}_{{\mathcal{R}}_{j}^{*}}^{(N)}(B_{\mathcal{M}}n)|$\mbox{}\hfil\\\
=&\,\frac{\max_{\ell=1,2}\operatorname{dist}\big{(}(A^{T}R_{(-1)^{j+1}\theta}A^{-T}n)_{\ell},{\mathbb{Z}}\big{)}|\mathds{1}_{{\mathcal{R}}_{j}^{*}}^{(N)}(B_{\mathcal{M}}n)|}{\operatorname{dist}\big{(}A^{T}R_{(-1)^{j+1}\theta}A^{-T}n,{\mathbb{Z}}^{2}\big{)}}\\\
&\,\times\frac{\operatorname{dist}\big{(}A^{T}R_{(-1)^{j+1}\theta}A^{-T}n,{\mathbb{Z}}^{2}\big{)}}{\max_{\ell=1,2}\operatorname{dist}\big{(}(A^{T}R_{(-1)^{j+1}\theta}A^{-T}n)_{\ell},{\mathbb{Z}}\big{)}}\\\
\leq&\,\frac{|n|^{2\sigma}}{\sqrt{2}K(2N+1)}\,.\end{split}$ (C.14)

Recall from (A.12a) that

$|n|\,\leq\,\frac{\rho_{\mathcal{M}}\|A\|_{2}}{2\pi}|B_{\mathcal{M}}n|\,.$
(C.15)

In particular, we arrive at

$|\mathds{1}_{{\mathcal{R}}_{j}^{*}}^{(N)}(G_{\mathcal{M}})|\,\leq\,\frac{1}{\sqrt{2}K}\Big{(}\frac{\rho_{\mathcal{M}}\|A\|_{2}}{2\pi}\Big{)}^{2\sigma}|G_{\mathcal{M}}|^{2\sigma}\frac{1}{2N+1}$
(C.16)

for any $G_{\mathcal{M}}\in{\mathcal{R}}^{*}_{\mathcal{M}}\setminus\\{0\\}$.
Using (C.16) and the steps leading to (3.55), we thus conclude the proof. ∎

##### Data availablity

The manuscript has no associated data.

## References

  * [1] Wolfgang Arendt and Daniel Daners.  Semilinear elliptic equations on rough domains.  J. Differential Equations, 346:376–415, 2023.
  * [2] Serge Aubry.  The new concept of transitions by breaking of analyticity in a crystallographic model.  In Alan R. Bishop and Toni Schneider, editors, Solitons and Condensed Matter Physics, pages 264–277, Berlin, Heidelberg, 1978. Springer Berlin Heidelberg.
  * [3] Serge Aubry and Gilles André.  Analyticity breaking and anderson localization in incommensurate lattices.  Annals of the Israel Physical Society, 3, 1980.
  * [4] Serge Aubry and Pierre-Yves Le Daeron.  The discrete Frenkel-Kontorova model and its extensions: I. Exact results for the ground-states.  Physica D: Nonlinear Phenomena, 8(3):381–422, 1983.
  * [5] Simon Becker, Mark Embree, Jens Wittsten, and Maciej Zworski.  Mathematics of magic angles in a model of twisted bilayer graphene.  Probability and Mathematical Physics, 3(1):69–103, 2022.
  * [6] Simon Becker, Tristan Humbert, and Maciej Zworski.  Fine structure of flat bands in a chiral model of magic angles.  arXiv preprint arXiv:2208.01628, 2022.
  * [7] J. Bellissard and D. Testard.  Quasi periodic hamiltonians: a mathematical approach.  Operator Algebras and Applications, 2:579, 1982.
  * [8] Jean Bellissard.  Coherent and dissipative transport in aperiodic solids: An overview.  Dynamics of Dissipation, pages 413–485, 2002.
  * [9] Jean Bellissard, Andreas van Elst, and Hermann Schulz-Baldes.  The noncommutative geometry of the quantum hall effect.  Journal of Mathematical Physics, 35(10):5373–5451, 1994.
  * [10] Victor Beresnevich, Felipe Ramírez, and Sanju Velani.  Metric Diophantine approximation: aspects of recent work.  Dynamics and analytic number theory, 437:1–95, 2016.
  * [11] Bruce C Berndt, Sun Kim, and Alexandru Zaharescu.  The Circle problem of Gauss and the divisor problem of Dirichlet—still unsolved.  The American Mathematical Monthly, 125(2):99–114, 2018.
  * [12] Dmitriy Bilyk.  Discrepancy theory and harmonic analysis.  Uniform Distribution and Quasi-Monte Carlo Methods, pages 45–62, 2014.
  * [13] Dmitriy Bilyk, Xiaomin Ma, Jill Pipher, and Craig Spencer.  Directional discrepancy in two dimensions.  Bulletin of the London Mathematical Society, 43(6):1151–1166, 2011\.
  * [14] Dmitriy Bilyk, Xiaomin Ma, Jill Pipher, and Craig Spencer.  Diophantine approximations and directional discrepancy of rotated lattices.  Transactions of the American Mathematical Society, 368(6):3871–3897, 2016.
  * [15] Rafi Bistritzer and Allan H MacDonald.  Moiré bands in twisted double-layer graphene.  Proceedings of the National Academy of Sciences, 108(30):12233–12237, 2011.
  * [16] Xavier Blanc, Claude Le Bris, and P-L Lions.  From molecular models to continuum mechanics.  Archive for Rational Mechanics and Analysis, 164:341–381, 2002\.
  * [17] Jean Bourgain.  A quantitative Oppenheim theorem for generic diagonal quadratic forms.  Israel J. Math., 215(1):503–512, 2016.
  * [18] Jean Bourgain and Nigel Watt.  Mean square of zeta function, circle problem and divisor problem revisited.  arXiv preprint arXiv:1709.04340, 2017.
  * [19] P. Buterus, F. Götze, T. Hille, and G. Margulis.  Distribution of values of quadratic forms at integral points.  Invent. Math., 227(3):857–961, 2022.
  * [20] Xavier Cabre, Eleonora Cinti, and Joaquim Serra.  Stable solutions to the fractional Allen-Cahn equation in the nonlocal perimeter regime.  arXiv preprint arXiv:2111.06285, 2021.
  * [21] Eric Cancès, Paul Cazeaux, and Mitchell Luskin.  Generalized Kubo formulas for the transport properties of incommensurate 2d atomic heterostructures.  Journal of Mathematical Physics, 58(6):063502, 2017.
  * [22] Yuan Cao, Valla Fatemi, Ahmet Demir, Shiang Fang, Spencer L. Tomarken, Jason Y. Luo, Javier D. Sanchez-Yamagishi, Kenji Watanabe, Takashi Taniguchi, Efthimios Kaxiras, Ray C. Ashoori, and Pablo Jarillo-Herrero.  Correlated insulator behaviour at half-filling in magic-angle graphene superlattices.  Nature, 556(7699):80–84, 2018.
  * [23] Yuan Cao, Valla Fatemi, Shiang Fang, Kenji Watanabe, Takashi Taniguchi, Efthimios Kaxiras, and Pablo Jarillo-Herrero.  Unconventional superconductivity in magic-angle graphene superlattices.  Nature, 556(7699):43–50, 2018.
  * [24] Stephen Carr, Daniel Massatt, Steven B. Torrisi, Paul Cazeaux, Mitchell Luskin, and Efthimios Kaxiras.  Relaxation and domain formation in incommensurate two-dimensional heterostructures.  Physical Review B, 98, 12 2018.
  * [25] Gonçalo Catarina, Bruno Amorim, Eduardo V Castro, João MVP Lopes, and Nuno MR Peres.  Twisted bilayer graphene: low-energy physics, electronic and optical properties.  Handbook of Graphene, 3:177–232, 2019.
  * [26] Paul Cazeaux, Drake Clark, Rebecca Engelke, Philip Kim, and Mitchell Luskin.  Relaxation and domain wall structure of bilayer moiré systems.  Journal of Elasticity, 2023.
  * [27] Paul Cazeaux, Mitchell Luskin, and Daniel Massatt.  Energy minimization of two dimensional incommensurate heterostructures.  Archive for Rational Mechanics and Analysis, 235:1289–1325, 2 2020\.
  * [28] Paul Cazeaux, Mitchell Luskin, and Ellad B. Tadmor.  Analysis of Rippling in Incommensurate One-Dimensional Coupled Chains.  Multiscale Modeling & Simulation, 15(1):56–73, January 2017.
  * [29] Cazeaux, Paul and Luskin, Mitchell.  Cauchy-Born strain energy density for coupled incommensurate elastic chains.  ESAIM: M2AN, 52(2):729–749, 2018.
  * [30] Thierry Cazenave.  An introduction to semilinear elliptic equations.  Editora do IM-UFRJ, Rio de Janeiro, 2006.
  * [31] Jérémy Champagne and Damien Roy.  Diophantine approximation with constraints.  Acta Arith., 207(1):57–99, 2023.
  * [32] Eleonora Cinti, Juan Davila, and Manuel Del Pino.  Solutions of the fractional Allen-Cahn equation which are invariant under screw motion.  Journal of the London Mathematical Society, 94(1):295–313, 2016\.
  * [33] Shuyang Dai, Yang Xiang, and David J Srolovitz.  Twisted bilayer graphene: Moiré with a twist.  Nano letters, 16(9):5923–5927, 2016.
  * [34] David Damanik.  One-dimensional ergodic Schrödinger operators.  Graduate studies in mathematics, volume 221. American Mathematical Society, Providence, Rhode Island, 2022.
  * [35] C Dean, Lei Wang, P Maher, Carlos Forsythe, Fereshte Ghahari, Y Gao, Jyoti Katoch, Masa Ishigami, Pilkyung Moon, Mikito Koshino, Takashi Taniguchi, K Watanabe, K Shepard, James Hone, and Phaly Kim.  Hofstadter’s butterfly and the fractal quantum hall effect in moire superlattices.  Nature, 497:598, 05 2013.
  * [36] Gregorio Díaz.  Large solutions of elliptic semilinear equations non-degenerate near the boundary.  Commun. Pure Appl. Anal., 22(3):686–735, 2023.
  * [37] Javier Duoandikoetxea Zuazo.  Fourier analysis.  Graduate studies in mathematics v. 29. American Mathematical Society, Providence, Rhode Island, 2001.
  * [38] Louis Dupaigne.  Stable solutions of elliptic partial differential equations, volume 143 of Chapman & Hall/CRC Monographs and Surveys in Pure and Applied Mathematics.  Chapman & Hall/CRC, Boca Raton, FL, 2011.
  * [39] Louis Dupaigne and Alberto Farina.  Regularity and symmetry for semilinear elliptic equations in bounded domains.  arXiv preprint arXiv:2102.12157, 2021.
  * [40] Alex Eskin, Gregory Margulis, and Shahar Mozes.  Quadratic forms of signature (2, 2) and eigenvalue spacings on rectangular 2-tori.  Annals of mathematics, pages 679–725, 2005.
  * [41] Malena I. Español, Dmitry Golovaty, and J. Patrick Wilber.  Discrete-to-continuum modelling of weakly interacting incommensurate two-dimensional lattices.  Proc. R. Soc. A., page 20170612, 2018.
  * [42] Malena I. Español, Dmitry Golovaty, and J. Patrick Wilber.  A discrete-to-continuum model of weakly interacting incommensurate two-dimensional lattices: The hexagonal case.  Journal of the Mechanics and Physics of Solids, 173:105229, 2023\.
  * [43] Alessio Figalli and Joaquim Serra.  On stable solutions for boundary reactions: a De Giorgi-type result in dimension 4+ 1.  Inventiones mathematicae, 219(1):153–177, 2020.
  * [44] François Fricker.  Einführung in die Gitterpunktlehre, volume 73 of Lehrbücher und Monographien aus dem Gebiete der Exakten Wissenschaften (LMW). Mathematische Reihe [Textbooks and Monographs in the Exact Sciences. Mathematical Series].  Birkhäuser Verlag, Basel-Boston, Mass., 1982.
  * [45] Anish Ghosh and Dubi Kelmer.  A quantitative Oppenheim theorem for generic ternary quadratic forms.  J. Mod. Dyn., 12:1–8, 2018.
  * [46] Boling Guo and Yitong Pei.  Periodic solutions of Ginzburg-Landau theory for atomic Fermi gases near the BCS-BEC crossover.  Appl. Anal., 101(4):1199–1210, 2022.
  * [47] Trajan Hammonds, Seoyoung Kim, Steven J. Miller, Arjun Nigam, Kyle Onghai, Dishant Saikia, and Lalit M. Sharma.  $k$-Diophantine $m$-tuples in finite fields.  Int. J. Number Theory, 19(4):891–912, 2023.
  * [48] Martin N Huxley.  Exponential sums and lattice points iii.  Proceedings of the London Mathematical Society, 87(3):591–609, 2003\.
  * [49] Radu Ignat and Robert L Jerrard.  Renormalized energy between vortices in some Ginzburg-Landau models on 2-dimensional Riemannian manifolds.  Archive for Rational Mechanics and Analysis, 239(3):1577–1666, 2021\.
  * [50] Anumol Joseph and Lakshmi Sankar.  Singular semilinear elliptic problems on unbounded domains in $\mathbb{R}^{n}$.  J. Math. Anal. Appl., 520(2):Paper No. 126903, 12, 2023.
  * [51] Alexander S. Kechris.  Global aspects of ergodic group actions, volume 160 of Mathematical Surveys and Monographs.  American Mathematical Society, Providence, RI, 2010.
  * [52] Dante M Kennes, Martin Claassen, Lede Xian, Antoine Georges, Andrew J Millis, James Hone, Cory R Dean, DN Basov, Abhay N Pasupathy, and Angel Rubio.  Moiré heterostructures as a condensed-matter quantum simulator.  Nature Physics, 17(2):155–163, 2021.
  * [53] David Kerr and Hanfeng Li.  Ergodic theory : independence and dichotomies.  Springer monographs in mathematics. Springer, Cham, Switzerland, 2016\.
  * [54] Aleksey N. Kolmogorov and Vincent H. Crespi.  Registry-dependent interlayer potential for graphitic systems.  Phys. Rev. B, 71:235415, Jun 2005.
  * [55] Mikito Koshino and Young Woo Son.  Moiré phonons in twisted bilayer graphene.  Physical Review B, 100, 8 2019.
  * [56] Kuipers, Lauwerens and Niederreiter, Harald.  Uniform distribution of sequences.  Pure and applied mathematics. Wiley-Interscience, New York, 1974.
  * [57] Shigehiko Kuratsubo and Eiichi Nakai.  Multiple Fourier series and lattice point problems.  Journal of Functional Analysis, 282(1):109272, 2022.
  * [58] Peter D Lax and Ralph S Phillips.  The asymptotic distribution of lattice points in euclidean and non-euclidean spaces.  Journal of Functional Analysis, 46(3):280–350, 1982.
  * [59] Itai Leven, Tal Maaravi, Ido Azuri, Leeor Kronik, and Oded Hod.  Interlayer potential for graphene/h-bn heterostructures.  Journal of chemical theory and computation, 12(6):2896–2905, 2016\.
  * [60] Boris M Levitan.  Asymptotic formulae for the number of lattice points in Euclidean and Lobachevskii spaces.  Russian Mathematical Surveys, 42(3):13, 1987.
  * [61] Jonathan Z. Lu, Ziyan Zhu, Mattia Angeli, Daniel T. Larson, and Efthimios Kaxiras.  Low-energy moiré phonons in twisted bilayer van der Waals heterostructures.  arxiv.org/abs/2207.04965, 7 2022.
  * [62] Francesco Maggi and Daniel Restrepo.  Uniform stability in the euclidean isoperimetric problem for the Allen-Cahn energy.  arXiv preprint arXiv:2202.11583, 2022.
  * [63] Noa Marom, Jonathan Bernstein, Jonathan Garel, Alexandre Tkatchenko, Ernesto Joselevich, Leeor Kronik, and Oded Hod.  Stacking and registry effects in layered materials: The case of hexagonal boron nitride.  Phys. Rev. Lett., 105:046801, Jul 2010.
  * [64] Daniel Massatt, Stephen Carr, and Mitchell Luskin.  Electronic observables for relaxed bilayer 2d heterostructures in momentum space.  arXiv preprint arXiv:2109.15296, 2021.
  * [65] Daniel Massatt, Mitchell Luskin, and Christoph Ortner.  Electronic density of states for incommensurate layers.  SIAM J. Multiscale Modeling & Simulation, 15:476–499, 2017.
  * [66] Felix Musil, Andrea Grisafi, Albert P. Bartók, Christoph Ortner, Gábor Csányi, and Michele Ceriotti.  Physics-inspired structural representations for molecules and materials.  Chemical Reviews, 121(16):9759–9815, 2021.
  * [67] Pradeep Mutalik.  Solution: Magic Moiré in Twisted Graphene.  Quanta Magazine, July 2019.  https://www.quantamagazine.org/puzzle-solution-magic-moire-in-graphene-20190726/, Accessed: 2023-03-22.
  * [68] Nguyen N.T. Nam and Mikito Koshino.  Lattice relaxation and energy band modulation in twisted bilayer graphene.  Physical Review B, 96, 8 2017.
  * [69] Evan O’Dorney.  Diophantine approximation on conics.  Proc. Amer. Math. Soc., 151(5):1889—-1905, 2023.
  * [70] C. Ortner and F. Theil.  Justification of the Cauchy-Born Approximation of Elastodynamics.  Archive for Rational Mechanics and Analysis, 207(3):1025–1073, March 2013.  Publisher: Springer Science and Business Media, LLC.
  * [71] Zhiwei Peng, Xiaolin Chen, Yulong Fan, David J Srolovitz, and Dangyuan Lei.  Strain engineering of 2d semiconductors and graphene: from strain fields to band-structure tuning and photonic applications.  Light: Science & Applications, 9(1):190, 2020.
  * [72] Pablo San-Jose, A. Gutiérrez-Rubio, Mauricio Sturla, and Francisco Guinea.  Spontaneous strains and gap in graphene on boron nitride.  Phys. Rev. B, 90:075428, Aug 2014.
  * [73] Ovidiu Savin.  Minimal surfaces and minimizers of the Ginzburg-Landau energy.  Cont. Math. Mech. Analysis AMS, 526:43–58, 2010.
  * [74] Kevin M Schmidt, Alex B Buettner, Olivia A Graeve, and Victor R Vasquez.  Interatomic pair potentials from DFT and molecular dynamics for Ca, Ba, and Sr hexaborides.  Journal of Materials Chemistry C, 3(33):8649–8658, 2015.
  * [75] Sylvia Serfaty.  Mean field limits of the Gross-Pitaevskii and parabolic Ginzburg-Landau equations.  Journal of the American Mathematical Society, 30(3):713–768, 2017\.
  * [76] Alexander V. Shapeev.  Moment tensor potentials: A class of systematically improvable interatomic potentials.  Multiscale Modeling & Simulation, 14(3):1153–1173, 2016.
  * [77] Theodore Tachim Medjo.  Large deviation principles for a 2D stochastic Allen-Cahn-Navier-Stokes driven by jump noise.  Stoch. Dyn., 22(4):Paper No. 2250005, 36, 2022.
  * [78] Grigory Tarnopolsky, Alex Jura Kruchkov, and Ashvin Vishwanath.  Origin of magic angles in twisted bilayer graphene.  Physical review letters, 122(10):106405, 2019.
  * [79] Georgios A. Tritsaris, Sharmila N. Shirodkar, Efthimios Kaxiras, Paul Cazeaux, Mitchell Luskin, Petr Plecháč, and Eric Cancès.  Perturbation theory for weakly coupled two-dimensional layers.  Journal of Materials Research, 31:959–966, 4 2016.
  * [80] Alexander B. Watson and Mitchell Luskin.  Existence of the first magic angle for the chiral model of bilayer graphene.  Journal of Mathematical Physics, 62(9):091502, 2021.
  * [81] Hyobin Yoo, Rebecca Engelke, Stephen Carr, Shiang Fang, Kuan Zhang, Paul Cazeaux, Suk Hyun Sung, Robert Hovden, Adam W. Tsen, Takashi Taniguchi, Kenji Watanabe, Gyu-Chul Yi, Miyoung Kim, Mitchell Luskin, Ellad B. Tadmor, Efthimios Kaxiras, and Philip Kim.  Atomic and electronic reconstruction at van der Waals interface in twisted bilayer graphene.  Nature Materials, pages 448–453, 2019.
  * [82] Kuan Zhang and Ellad B. Tadmor.  Structural and electron diffraction scaling of twisted graphene bilayers.  Journal of the Mechanics and Physics of Solids, 112, 12 2017.
  * [83] Songsong Zhou, Jian Han, Shuyang Dai, Jianwei Sun, and David J. Srolovitz.  van der Waals bilayer energetics: Generalized stacking-fault energy of graphene, boron nitride, and graphene/boron nitride bilayers.  Phys. Rev. B, 92:155438, Oct 2015.