# Spreaders in the Network SIR Model:
An Empirical Study

Brian Macdonald1,2,5 Paulo Shakarian1,3,5 Nicholas Howard1,2 & Geoffrey
Moores1,3,4
1Network Science Center  United States Military Academy  West Point  NY 10996
2Department of Mathematical Sciences  United States Military Academy  West
Point  NY 10996
3Department of Electrical Engineering and Computer Science  United States
Military Academy  West Point  NY 10996
4Department of Physics  United States Military Academy  West Point  NY 10996
5These authors contibuted equally to this work.
brian.macdonald@usma.edu,paulo@shakarian.net,nicholas.howard<EMAIL_ADDRESS>
###### Abstract

We use the susceptible-infected-recovered (SIR) model for disease spread over
a network, and empirically study how well various centrality measures perform
at identifying which nodes in a network will be the best spreaders of disease
on $10$ real-world networks. We find that the relative performance of degree,
shell number and other centrality measures can be sensitive to $\beta$, the
probability that an infected node will transmit the disease to a susceptible
node. We also find that eigenvector centrality performs very well in general
for values of $\beta$ above the epidemic threshold.

##  1 Introduction

The susceptible-infected-recovered (SIR) model, first introduced in Anderson &
May, (1979) is a popular model for disease spread. In recent years, this model
has been applied to social networks - situations where the interactions of
individuals are modeled as a graph. A key problem relating to this model when
considering a network structure is how to identify the nodes that, if
initially infected, will result in the greatest portion of the population (in
expectation) also becoming infected. These nodes are often referred to as
“spreaders.” Unfortunately, a modification of the proof of a related problem
in Chen et al., (2010) shows that exactly computing the expected number of
infected individuals in a networked-structured population given a single
initial infectee is $\\#P$-hard. This implies that solving this problem
exactly is likely beyond the ability of today’s computer systems. However, the
literature on complex networks has provided various centrality measures that
can be used as heuristics. So, inspired by the work of Kitsak et al., (2010),
which empirically examines the use of degree, betweenness, and shell number
for identifying spreaders, we conduct a comprehensive evaluation of $10$
different centrality measures on $10$ real-world social network data-sets from
various domains (e-mail, disease spread, blogging, power, autonomous system,
and collaboration). The major contributions of our work are two-fold. First,
we show that the ability of a centrality measure to identify spreaders in the
SIR model can be sensitive to the $\beta$ parameter, the probability of
infection. Second, we find that, in general, eigenvector centrality performs
very well for values of $\beta$ above the epidemic threshold.

With respect to our first major contribution, we carefully selected the
$\beta$ parameter based on $\beta^{\prime}$, the epidemic threshold of the
network. We can be sure that a contagion can spread to a significant portion
of the network for $\beta>\beta^{\prime}$, and we studied a variety of
different values for $\beta$ above this threshold.

In Figure 1 and 2, we give an example of a network where shell number
outperforms degree for one value of $\beta$, but degree outperforms shell
number for another value of $\beta.$ In Section 5, we give additional examples
illustrating that the imprecision functions of other centrality measures, as
well as the choice of the “best” centrality measure, can be sensitive to
$\beta$ as well.

Figure 1:  Imprecision versus $p$ for the cond-mat network with $\beta=11.17$.
Notice that for this $\beta$, $k$-shell has a lower imprecision, meaning that
$k$-shell outperforms degree. See Section 3 for the definitions of imprecision
function and $p$.  Figure 2: Imprecision plots vs. $p$ for the cond-mat
network with $\beta=15.95$. Notice that for this $\beta$, degree has a lower
imprecision, meaning that degree outperforms $k$-shell, the opposite of what
we saw in Figure 1.

As for our second major contribution, we found that eigenvector centrality
consistently outperformed all other measures considered, including both shell
number and degree (which were considered by Kitsak et al.), in all but one of
the networks examined. See Figure 3 for a comparison of $k$-shell (the best
performing centrality measure of Kitsak et al.) with eigenvector centrality.
Also, if we average over all of our networks, including the one where
eigenvector was not the best, we find that, on average, eigenvector centrality
outperforms the other measures.

Figure 3: Imprecision of $k$-shell minus the imprecision of eigenvector
centrality. Positive values indicate that $k$-shell has a higher imprecision
than eigenvector centrality, which means that eigenvector centrality typically
outperforms $k$-shell.

The rest of this paper is organized as follows. In Section 2, we review the
SIR model, discuss how the $\\#P$-hardness proof of Chen et al., (2010)
applies to this model, and describe how we calculate the epidemic threshold of
a given complex network. This is followed by a discussion of the various
centrality measures we considered in Section 3 along with a review of the
description of the “imprecision function” Kitsak et al., (2010) used to
measure the effectiveness of a centrality measure in identifying the top
spreaders in a network. We describe our experimental setup and datasets in
Section 4 and give a description and discussion of the experimental results in
Section 5.

##  2 The SIR Model

As in Kitsak et al., (2010), we consider the classic susceptible-infected-
recovered (SIR) model of disease spread introduced in Anderson & May, (1979).
In this model, all nodes in the network are in one of three states:
susceptible (able to be infected), infected, or recovered (no longer able to
infect or be infected). At each time step, any node infected in the last time
step can infect any of its neighbors who are in a susceptible state with a
probability $\beta$. After that time step, any node previously in an infected
state moves into a recovered state and is no longer able to infect or be
infected.

###  2.1 Complexity

In J. Goldenberg, (2001) and Kempe et al., (2003), the authors present a
generalization of the SIR model known as the independent cascade (IC) model.
In this model, the $\beta$ parameter can be different for each edge in the
network. They define the influence spread of a set of nodes as the expected
number of individuals in the population infected under the IC model given that
the set was initially infected. In Chen et al., (2010) this problem was shown
to be $\\#P$-hard. Here we reconsider their proof, with some modification, to
identify the influence spread of single node under the SIR model.

###### Theorem 2.1

Calculating the influence spread of a single node under the SIR models is
$\\#P$-hard.

######  Proof 2.2.

We prove this theorem by showing a reduction from the known $\\#P$-complete
problem $s-t$ connectivity Valiant, (1979). Let $G=(V,E)$ be a directed graph,
where $V$ denotes the set of vertices, and $E$ denotes the set of edges. Given
two vertices $s,t\in V$, the goal is to determine the number of subgraphs of
$G$ where $s$ is connected to $t$. In Chen et al., (2010), the authors point
out that it is easy to see that this is equivalent to computing the
probability that $s$ is connected to $t$ when each edge in $G$ has an
independent probability of $0.5$ to be connected (and $0.5$ to be
disconnected). Hence, to embed the $s-t$ connectivity problem into the
influence spread on the SIR model, we first calculate $M_{s}$, the expected
number of infectees given initially infected node $s$ with $\beta=50$. We then
create $G^{\prime}$ which is equivalent to $G$ but has an additional directed
edge from $t$ to a new node $t^{\prime}$. Let $M_{s}^{\prime}$ be the
influence spread when we consider graph $G^{\prime}$. If $p(s,t,G)$ is the
probability that $t$ is influenced by $s$ in $G$ (hence the solution to the
$s-t$ connectivity problem) then
$M_{s}^{\prime}=M_{s}+p(s,t,G)\cdot\frac{\beta}{100}$. Therefore, the solution
to the $s-t$ connectivity problem can easily be obtained in polynomial time if
we can efficiently find a solution to the influence spread problem under the
SIR model.

Theorem 2.1 tells us that exact methods for identifying the influence spread
of individual nodes under the SIR models is likely not possible with today’s
computer systems. Further, as $s-t$ connectivity has no known efficient
approximation algorithm with a guarantee of accuracy, an approximation scheme
for influence spread also seems unlikely. Hence, much work on influence spread
such as Kempe et al., (2003) relies on estimating influence spread using
simulation, which is often expensive computationally and even impractical for
very large networks. Therefore, in this paper, we look to evaluate various
centrality measures from the literature as heuristics to identify spreaders
under the SIR model. We describe these centrality measures in Section 3. Note
that the centrality measures are not specifically designed to calculate
influence spread under the SIR model, and they do not account for the
infection probability $\beta$. In the next section, we describe how we select
the different $\beta$ parameters for the model in our experiments.

###  2.2 Selecting the Infection Probability

We note that for scale-free networks, having degree distribution $P(k)\sim
k^{-\gamma}$, the literature shows that for $\gamma\leq 3$, the epidemic
threshold of $\beta$ approaches $0$ as the number of nodes goes to infinity
Callaway et al., (2000); Cohen et al., (2000). However, the networks we
examine are of finite size and have various levels of “scale-freeness”, based
on the $R^{2}$ value of the linear correlation of a log-log plot of the degree
distribution (see Section 4 for details). Instead, we explored $\beta$ values
based on the epidemic threshold calculation in Madar et al., (2004). Using
this method, the SIR model is mapped onto a bond percolation process. Assuming
a randomly connected network, the average number of influenced neighbors,
$\langle n\rangle$ can be written

$\langle n\rangle=\beta\cdot\sum\limits_{k}\frac{P(k)\cdot
k\cdot(k-1)}{\langle k\rangle},$ (1)

where $k$ is the degree of a node, $P(k)$ is the probability of a node having
degree $k$, and $\langle k\rangle$ is the average degree. Since an epidemic
state can only be reached when $\langle n\rangle>1$, and from (1) we have

$\beta>\left(\sum\limits_{k}\frac{P(k)\cdot k\cdot(k-1)}{\langle
k\rangle}\right)^{-1}=\beta^{\prime}.$ (2)

We note that there is some work discussing the effect of different infection
probabilities on spreading in Kitsak et al., (2010) and more recent and
comprehensive study on the topic in Castellano & Pastor-Satorras, (2012).
These works consider the effect of this parameter with respect to degree and
shell decomposition (and betweenness in Kitsak et al., (2010)). Here we
consider these and many other centrality measures, and find that some of them,
such as eigenvector centrality, outperform those in these previous works.

##  3 Centrality Measures

We now describe the centrality measures that we examine in our experiments. We
note that the major centrality measures in the literature can be classified as
either radial (the quantity of certain paths originating from the node) or
medial (the quantity of certain paths passing through the node) as done in
Borgatti and Everett Borgatti & Everett, (2006). Based on the negative result
concerning betweenness of Kitsak et al., (2010) and the intuitive association
between high-radial nodes and spreading, we focused our efforts on radial
measures. While the work of Kitsak et al., (2010) compares shell number to
degree and betweenness, we consider several other well-known radial measures
in addition to degree, including closeness and eigenvector centrality. As done
in Kitsak et al., (2010), we also develop “imprecision functions” for these
centrality measures.

###  3.1 Degree Centrality

Of all the measures that we are examining, degree is perhaps the most
simplistic measure - simply the total of incident edges for a given node. As
noted throughout the literature, such as Wasserman & Faust, (1994), it is
perhaps the easiest centrality measure to compute. Further, in other diffusion
processes, such as the voter model on undirected networks in Antal et al.,
(2006), it has been shown to be proportional to the expected number of
individuals becoming infected111Technically, the work of Antal et al., (2006)
proves that the fixation probability for a single mutant invader is
proportional to the degree of that node. However, the expected number of
mutants, in the limit as time goes to infinity, can simply be computed by
multiplying fixation probability by the number of nodes in thee network.. As
pointed out in Borgatti & Everett, (2006), degree is a radial measure as it is
the number of paths starting from a node of length $1$. Degree is one of three
measures considered in Kitsak et al., (2010).

###  3.2 Shell Number

The other radial measure considered in Kitsak et al., (2010), shell number, or
“$k$-shell number”, is determined using shell decomposition Seidman, (1983).
High shell-number nodes in the network are often referred to as the “core” and
are regarded by Kitsak et al., (2010) as influential spreaders under the SIR
model. Our results described later in the paper confirm this finding, although
we also show that $k$-shell number was generally outperformed by eigenvector
centrality. There have also been some more practical applications of this
technique to find key nodes in a network. For instance, Borge-Holthoefer &
Moreno, (2012); Borge-Holthoefer et al., (2012) uses shell-decomposition to
find individuals likely to initiate information cascades in an online social
network while Carmi et al., (2007) uses it to identify key nodes in a subset
of autonomous systems on the Internet.

An example of this process is shown in Figure 4. Given graph $G=(V,E)$, shell
decomposition partitions a graph into shells and is described in the algorithm
below.

Let $k_{i}$ be the degree of node $i$. Set $S=1$. Let $V_{S}$ denote the first
shell of $G$.

while $|V|>0$ do

while There exists $i$ such that $k_{i}=S$ do

Remove all $i\in V$ where $k_{i}=S$;

Also, remove all corresponding adjacent edges.

Place removed nodes into shell $V_{S}$.

end while

$S$++

end while

Figure 4: Consider the progression of the graph above, where the elimination
of nodes with degree 1 occurs in B and C. D represents the first iteration for
the second shell, and E represents the complete second shell (as well as the
first). F finalizes the decomposition with the third shell.

###  3.3 Betweenness Centrality

The intuition behind high betweenness centrality nodes is that they function
as “bottlenecks” as many paths in the network pass through them. Hence,
betweenness is a medial centrality measure. Let $\sigma_{st}$ be the number of
shortest paths between nodes $s$ and $t$ and $\sigma_{st}(v)$ be the number of
shortest paths between $s$ and $t$ containing node $v$. In Freeman, (1977),
betweenness centrality for node $v$ is defined as $\sum_{s\neq v\neq
t}\frac{\sigma_{st}(v)}{\sigma_{st}}$. In most implementations, including the
ones used in this paper, the algorithm of Brandes, (2001) is used to calculate
betweenness centrality.

###  3.4 Closeness Centrality

Another common measure from the literature that we examined is closeness
Freeman, (1979). Given node $i$, its closeness $C_{c}(i)$ is the inverse of
the average shortest path length from node $i$ to all other nodes in the
graph. Intuitively, closeness measures how “close” it is to all other nodes in
a graph.

Formally, if we define the shortest path between nodes $i$ to $j$ as function
$d_{G}(i,j)$, we can express the average path length from $i$ to all other
nodes as

$L_{i}=\frac{\sum_{j\in V\setminus i}d_{G}(i,j)}{|V|-1}.$ (3)

Hence, the closeness of a node can be formally written as

$C_{c}(i)=\frac{1}{L_{i}}=\frac{|V|-1}{\sum_{j\in V\setminus i}d_{G}(i,j)}.$
(4)

###  3.5 Eigenvector Centrality

The use of the principle eigenvector of the adjacency matrix of a network was
first proposed as a centrality measure in Bonacich, (1972). Hence, the
intuition behind eigenvector centrality is that it measures the influence of a
node based on the sum of the influences of its adjacent nodes. Given a network
$V=(G,E)$ with adjacency matrix $A=(a_{ij})$, where $a_{ij}=1$ if an edge
exists between nodes $i$ and $j$, the eigenvector centrality of node $i$
satisfies

$x_{i}=\frac{1}{\lambda}\sum\limits_{j\in V}a_{ij}x_{j},$ (5)

for some $\lambda.$ If we define $x$ to be the vector of $x_{i}$’s, this
relationship can be expressed as

$\displaystyle x=\frac{1}{\lambda}Ax,\,\textup{ or }\,\,Ax=\lambda x,$ (6)

which is the familiar equation relating $A$ with its eigenvalues and
eigenvector. The eigenvector centralities for the network are the entries of
the eigenvector corresponding to the largest real eigenvalue.

###  3.6 PageRank

PageRank, introduced in Page et al., (1998), is computed for each node based
on the PageRank of its neighbors. Where $E$ is the set of undirected edges,
$R_{v},d_{v}$ is the PageRank and degree of $v$, and $c$ is a normalization
constant, we have the relationship

$R_{v}=c\cdot\sum_{v^{\prime}|(v,v^{\prime})\in
E}\frac{R_{v^{\prime}}}{d_{v^{\prime}}}.$

An initial value for rank is entered for each node and the relationship is
then computed iteratively until convergence is reached. Intuitively, PageRank
can be thought of as the importance of a node based on the importance of its
neighbors.

###  3.7 Neighborhood

The next centrality measure we consider is the “neighborhood.” Given a natural
number $q$, the $q$-neighborhood of vertex $i$ is the number of nodes in the
network that are distance $q$ or closer from node $i$. For example, for $q=0$,
this metric is $1$ for every node. For $q=1$, this metric is identical to
degree centrality of node $i$, since it is the number of nodes within a
distance $1$ of $i$. For $q=2$, this metric counts the number of nodes within
a distance $2$ of $i$, so it counts $i$’s neighbors along with its neighbors’
neighbors. In our work, we computed neighborhoods using $q=2,3,5,10$, and
denoted these measures by $nghd2$, $nghd3$, $nghd5$, and $nghd10$,
respectively. We note that the work of Chen et al., (2012) develops a
centrality measure with a similar intuition to the neighborhood and show it
preforms well in identifying influential spreaders.

###  3.8 The Imprecision Functions

We now define the imprecision functions from Kitsak et al., (2010) that are
used to measure the effectiveness of a centrality measure in identifying
influential spreaders. We also extend their definition for all centrality
measures explored in this paper. Let $N$ denote the number of nodes, and let
$p$ be a real number between 0 and 100. The $pN/100$ highest efficiency
spreaders, $\Upsilon_{eff}(p)$, are chosen based on number of nodes infected
$M_{i}$ per node. Similarly, a set $\Upsilon_{k_{s}}(p)$ is defined as the
$pN/100$ predicted most efficient spreaders, chosen with priority to highest
$k_{s}$ valued nodes. Let

$\displaystyle M_{eff}(p)$
$\displaystyle=\sum\limits_{i\in\Upsilon_{eff}(p)}\frac{M_{i}}{pN},\mathit{and}$
(7) $\displaystyle M_{k_{s}}(p)$
$\displaystyle=\sum\limits_{i\in\Upsilon_{k_{s}}(p)}\frac{M_{i}}{pN}.$ (8)

The imprecision function of $k_{s}$, $\epsilon_{k_{s}}(p)$, is defined as

$\epsilon_{k_{s}}(p)=1-\frac{M_{k_{s}}(p)}{M_{eff}(p)}$ (9)

Similarly, $\epsilon_{eig}(p)$ and $\epsilon_{deg}(p)$ are defined as

$\displaystyle\epsilon_{eig}(p)$
$\displaystyle=1-\frac{M_{eig}(p)}{M_{eff}(p)},$ (10)
$\displaystyle\epsilon_{deg}(p)$
$\displaystyle=1-\frac{M_{deg}(p)}{M_{eff}(p)}$ (11)

In general, for any centrality measure $c$, the imprecision function
$\epsilon_{c}(p)$ is defined as

$\epsilon_{c}(p)=1-\frac{M_{c}(p)}{M_{eff}(p)}$ (12)

##  4 Experimental Setup

In this section we describe our experimental setup and the datasets we used.
All simulation and centrality analysis was done in Version 2.14.1 of `R` R
Development Core Team, (2011). The operating system used was Windows Vista
Enterprise (32 bit) and the computer had an Intel Core 2 Quad CPU (Q9650) 3.0
GHz with 4 GB of RAM. Run times to analyze the networks ranged from several
hours for the small networks to several days for the larger ones. Centrality
measures were computed using the `igraph` Csardi & Nepusz, (2006) package in
`R`.

We obtained our datasets from a variety of sources. Brief descriptions of
these networks are as follows:

  * cond-mat-GCC is an academic collaboration network from the e-print arXiv and covers scientific collaborations between authors’ papers submitted to Condensed Matter category from 1999 Newman, (2011).

  * ca-GrQc-GCC is an academic collaboration network from the e-print arXiv and covers scientific collaborations between authors’ papers submitted to the General Relativity and Quantum Cosmology category from Jan. 1993 - Apr. 2003 Leskovec, (2012).

  * urv-email is an e-mail network based on communications of members of the University Rovira i Virgili (Tarragona) Arenas, (2012). It was extracted in 2003.

  * 1-edges-GCC is a network formed from YouTube, the video-sharing website that allows users to establish friendship links Zafarani & Liu, (2009). The sample was extracted in Dec. 2008. Links represent two individuals sharing one or more subscriptions to channels on YouTube.

  * std-GCC is an online sex community in Brazil in which links represent that one of the individuals posted online about a sexual experience with the other individual, resulting in a bipartite graph. The data was extracted from September of 2002 to October of 2008 Luis E. C. Rocha & Holme, (2010).

  * as20000102 is a one day snapshot of Internet routers as constructed from the border gateway protocol logs Leskovec, (2012). It was extracted on Jan 2nd, 2000.

  * oregon_010331 is a network of Internet routers over a one week period as inferred from Oregon route-views, looking glass data, and routing registry from covering the week of March 3rd, 2001 Leskovec, (2012).

  * ca-HepTh-GCC is a collaboration network from the e-print arXiv and covers scientific collaborations between authors’ papers submitted to the High Energy Physics - Theory category. It covers paper from Jan 1993 to Apr 2003 Leskovec, (2012).

  * as-22July06 is a snapshot of the Internet on 22 July 2006 at the autonomous systems level compiled by Mark Newman Newman, (2011).

  * netscience-GCC is a network of coauthorship of scientists working on network theory and experiments compiled by Mark Newman in May 2006 Newman, (2011).

All datasets used in this paper were obtained from one of four sources: the
ASU Social Computing Data Repository Zafarani & Liu, (2009), the Stanford
Network Analysis Project Leskovec, (2012), Mark Newman’s data repository at
the University of Michigan Newman, (2011), and Universitat Rovira i Virgili
Arenas, (2012). All networks considered were symmetric; i.e., if a directed
edge from vertex $v$ to $v^{\prime}$ exists, there is also an edge from vertex
$v^{\prime}$ to $v$. Summary statistics for these networks can be found in
Table 1.

In the cases where the network had more than one component, we used only the
greatest connected component. We append the suffix “-GCC” when referring to
those networks. For example, the cond-mat network had more than one component,
so we will use the greatest connected component and refer to this network as
“cond-mat-GCC”.

Name | Type | Nodes | Edges | Density | $\beta^{\prime}$ | $\lambda$ | $R^{2}$ | $\langle k\rangle$ | $\langle k^{2}\rangle$ | $K_{S}$
---|---|---|---|---|---|---|---|---|---|---
1-edges-GCC | online | 13679 | 76741 | 0.0008 | 2.3 | 1.8 | 0.90 | 11.2 | 502.6 | 25
as20000102 | router | 6474 | 12572 | 0.0006 | 0.6 | 1.2 | 0.73 | 3.9 | 640.0 | 12
ca-GrQc-GCC | collab | 4158 | 13422 | 0.0016 | 6.3 | 2.0 | 0.88 | 5.5 | 93.2 | 43
cond-mat-GCC | collab | 13861 | 44619 | 0.0005 | 8.4 | 2.4 | 0.93 | 5.9 | 75.6 | 17
oregon2_010331 | router | 10900 | 31180 | 0.0005 | 0.5 | 1.2 | 0.79 | 5.7 | 1188.8 | 31
std-GCC | std | 15810 | 38540 | 0.0003 | 3.7 | 1.9 | 0.92 | 4.7 | 130.9 | 11
urv-email | email | 1133 | 5451 | 0.0085 | 5.7 | 1.5 | 0.84 | 9.6 | 179.8 | 11
ca-HepTh-GCC | collab | 8638 | 24806 | 0.0007 | 8.3 | 2.2 | 0.90 | 5.7 | 74.6 | 31
as-22July2006 | router | 22963 | 48436 | 0.0002 | 0.4 | 1.2 | 0.72 | 4.2 | 1103.0 | 25
netscience-GCC | collab | 379 | 914 | 0.0127 | 14.2 | 1.6 | 0.76 | 4.8 | 38.7 | 8
Table 1: Network Summary Statistics. Note that $\beta^{\prime}$ is the minimum
threshold of infection rate for the epidemic to spread to a significant
portion of the network, $\lambda$ is exponent of the power law of the degree
distribution, $R^{2}$ is goodness of fit between the power law and the degree
distribution, $\langle k\rangle$ and $\langle k^{2}\rangle$ are the first and
second moments of the degree distribution, and $K_{S}$ is the maximum shell
present in the network.

As seen in the Table 1, all networks used are approximately scale free. This
does not infer that they were generated using a preferential attachment model
(as introduced in Albert-László Barabási, (1999)), as many mechanisms can be
responsible for generating scale free networks. If they were generated using a
preferential attachment model then we would see a correlation between shell
number and degree. This would also mean that degree centrality and shell
number would have little difference in predicting spreaders, but our
simulations show otherwise. Figure 5 shows an example in which degree and
shell number are not correlated.

Figure 5: In the higher shells of these two examples, degree and shell number
are not correlated, indicating these can not be assumed to be generated by
preferential attachment models. The red line shows the average degree of each
shell. Note that log scales are being used on both axes.

##  5 Results

Earlier we noted that (1) the relative performance of degree, shell number and
other centrality measures can depend on the $\beta$ parameter of the SIR
model, and (2) eigenvector centrality performs very well in general regardless
of the value of $\beta$ used, typically outperforming all of the other
centrality measures that we tried. Here we present more results illustrating
these two points. Unless otherwise specified, the $\beta$ values that we used
when plotting the imprecision function versus $\beta$ are
$1.1\beta^{\prime},1.2\beta^{\prime},\ldots,2.0\beta^{\prime},$ where
$\beta^{\prime}$ is the epidemic threshold for the network in question.

###  5.1 Sensitivity to $\beta$

In Figures 1 and 2, we saw that the performance of degree relative to shell
number changes with $\beta$ for the cond-mat network. For $\beta=11.17$, shell
number was a better indicator of spreading, but for $\beta=15.95$, degree was
better. Another way that we could depict this dependence on $\beta$ is to fix
$p$ and plot the imprecision versus $\beta$, instead of fixing $\beta$ and
plotting the imprecision versus $p$. In Figure 6, we fix $p=5$ and plot the
imprecision function of degree, shell number, and eigenvector centrality
versus $\beta$, for $\beta$ between $11.17$ and $15.95$.

Figure 6: Imprecision vs $\beta$ for the cond-mat network. The relative
performance of degree and shell number changes near $\beta=14$.

Notice that at around $\beta=14$, degree begins to outperform shell number.

The relative performance of other centrality measures can change as well. In
Figure 7, we plot the imprecision functions of degree, shell number,
eigenvector, and closeness centrality versus $\beta$ for $p=5$.

Figure 7: Imprecision vs $\beta$ for the ca-GrQc-GCC network.

In this network, for $\beta$ near $\beta^{\prime}$, degree and shell number
perform very well. However, as $\beta$ increases, the imprecision functions of
those measures increase, and other measures, like closeness and eigenvector,
outperform degree and shell number.

###  5.2 Eigenvector centrality

As we saw in Figure 3, eigenvector centrality outperforms shell number for all
but one of the networks we examined. Eigenvector centrality also typically
outperforms all of the other centrality measures that we tried. In Figure 8,
we plot the imprecision functions of several different centrality measures for
the cond-mat network. We see that eigenvector centrality performs best for
this network.

Figure 8: Imprecision vs $p$ for the cond-mat-GCC network with
$\beta=1.1\beta^{\prime}=8.77$. We see that eigenvalue centrality performs
best for this network.

In Figures 9, 10, 11, and 12, we give an example of a collaboration network,
an online network, an STD network, and an email network in which eigenvector
performs best.

Figure 9: Imprecision vs $p$ for the netscience-GCC network with
$\beta=1.1\beta^{\prime}=15.67$. We see that eigenvalue centrality performs
best for this network. Figure 10: Imprecision vs $p$ for the 1-edges-GCC
network with $\beta=1.1\beta^{\prime}=2.50$. We see that eigenvalue centrality
performs best for this network. Figure 11: Imprecision vs $p$ for the std-GCC
network with $\beta=1.1\beta^{\prime}=4.01$. We see that eigenvalue centrality
performs best for this network. Figure 12: Imprecision vs $p$ for the urv-
email network with $\beta=1.1\beta^{\prime}=6.22$. We see that eigenvalue
centrality performs best for this network.

Eigenvector centrality did not outperform shell number for the ca-HepTh
network, so we can not conclude that eigenvector centrality performs best for
every network that we tried. However, it does seem that, on average, for the
networks we tried, eigenvector centrality performed best for
$\beta=1.1\beta^{\prime},1.2\beta^{\prime},...,2.0\beta^{\prime}.$ Suppose we
take the imprecision functions for $\beta=1.1\beta^{\prime}$ for each network,
and we average these imprecision functions over all of our networks, including
the ca-HepTh network. This would be one way to check how well each centrality
measure performs on average. In Figure 13, we plot this the average
imprecision versus $p$ for $\beta=1.1\beta^{\prime}$. We see that, on average,
eigenvector centrality outperforms the other measures. The measure $nghd2$
performed well also. We give similar figures for $\beta=1.5\beta^{\prime}$ and
$\beta=2.0\beta^{\prime}$ in Figures 14 and 15. In both cases, eigenvector
centrality outperforms all of the other measures.

We believe that eigenvector centrality performs well for some of the same
reasons that shell number performs well. A node has high eigenvector
centrality when the node and its neighbors have high degree. Nghd2, nghd3, and
the closely related measure of Chen et al., (2012) also perform well for this
reason. A hub, or a node with high degree, in the periphery of a network,
which does not have many neighbors with high degree, will not typically be as
good of a spreader as a node with high eigenvector centrality.

Figure 13: Average Imprecision vs $p$ with $\beta=1.1\beta^{\prime}$, where
the average is taken over all networks that we considered.

###  5.3 Large values of $\beta$

In Kitsak et al., (2010), only relatively small values for $\beta$ were
explored as it was noted that larger values of $\beta$ would likely cause
spreading to a large portion of the population regardless of the location of
the initially infected node. However, in the networks we studied, we found a
difference in the ability of the starting node to spread even at seven times
the epidemic threshold. Further, the result that eigenvector centrality
performs best, based on average imprecision over all the networks, still holds
for these larger values of $\beta$. We display our imprecision functions for
larger values of $\beta$ in Figure 16. We also show that for five times the
epidemic threshold, eigenvector centrality still outperforms the other
centrality measures for different values of $p$ (Figure 17).

Figure 14: Average Imprecision vs $p$ with $\beta=1.5\beta^{\prime}$, where
the average is taken over all networks that we considered. We see that, on
average, eigenvector performs best. Figure 15: Average Imprecision vs $p$ with
$\beta=2.0\beta^{\prime}$, where the average is taken over all networks that
we considered. We see that, on average, eigenvector performs best. Figure 16:
Average Imprecision vs. $\beta$ with $p=5$. We see that, on average,
eigenvector performs best. Figure 17: Average Imprecision vs. $p$ with
$\beta=5\beta^{\prime}$, where the average is taken over all networks that we
considered. We see that, on average, eigenvector performs best.

##  6 Conclusions and Future Work

These new experiments provide further insight into the issue of identifying
spreaders in complex networks that was initiated by Kitsak et al., (2010). We
extended their work by studying multiple values of the infection probability
$\beta$ and showed that the relative ability for centrality measures to
identify spreaders often depends on this parameter. We also noted that
eigenvector centrality consistently outperforms the other centrality measures,
usually independent of $\beta$. Future work on identifying influential
spreaders could include identifying nodes that not only cause significant
spreading, but do so quickly, thus accounting for the time it takes for
individuals in the population to become infected. Further, it would be also
interesting to examine which centrality measures best identify spreaders in
non-monotonic models of diffusion processes, such as the voter model. Another
aspect for future work would be to examine group centrality. In other words,
one could use a centrality measure on sets of nodes to identify the best set
of spreaders under the SIR model Moores et al., (2012). Finally, it is also
worth empirically studying centrality measures designed specifically for the
SIR model or other diffusion process, as described in recent work such as
Klemm et al., (2012) and Kang et al., (2012). However, we note that one key
advantage to the approach taken in this paper is that the centrality measures
studied are already well established - and hence common in many software tools
for complex network analysis.

## Acknowledgments

Some of the authors of this paper are supported under OSD project
F1AF262025G001 and ARO project 2GDATXR042. The authors are very thankful to
these organizations for their support.

We would like to thank Jon Bentley for his feedback on an earlier version of
this paper.

The views expressed in this article are those of the authors and do not
reflect the official policy or position of the United States Military Academy,
the Department of the Army, the Department of Defense, the United States
Government, or any of the listed funding agencies.

## References

  * Albert-László Barabási, (1999) Albert-László Barabási, Réka Albert. (1999).  Emergence of scaling in random networks.  286(5439), 509–512.
  * Anderson & May, (1979) Anderson, Roy M., & May, Robert M. (1979).  Population biology of infectious diseases: Part i.  Nature, 280(5721), 361.
  * Antal et al., (2006) Antal, T., Redner, S., & Sood, V. (2006).  Evolutionary dynamics on degree-heterogeneous graphs.  Physical review letters, 96(18), 188104.
  * Arenas, (2012) Arenas, Alex. (2012).  Network data sets.
  * Bonacich, (1972) Bonacich, Phillip. (1972).  Factoring and weighting approaches to status scores and clique identification.  The journal of mathematical sociology, 2(1), 113–120.
  * Borgatti & Everett, (2006) Borgatti, S., & Everett, M. (2006).  A Graph-theoretic perspective on centrality.  Social networks, 28(4), 466–484.
  * Borge-Holthoefer & Moreno, (2012) Borge-Holthoefer, Javier, & Moreno, Yamir. (2012).  Absence of influential spreaders in rumor dynamics.  Phys. rev. e, 85(026116).
  * Borge-Holthoefer et al., (2012) Borge-Holthoefer, Javier, Rivero, Alejandro, & Moreno, Yamir. (2012).  Locating privileged spreaders on an online social network.  Phys. rev. e, 85(Jun), 066123.
  * Brandes, (2001) Brandes, Ulrik. (2001).  A faster algorithm for betweenness centrality.  Journal of mathematical sociology, 25(163).
  * Callaway et al., (2000) Callaway, Duncan S., Newman, M. E. J., Strogatz, Steven H., & Watts, Duncan J. (2000).  Network robustness and fragility: Percolation on random graphs.  Phys. rev. lett., 85(Dec), 5468–5471.
  * Carmi et al., (2007) Carmi, Shai, Havlin, Shlomo, Kirkpatrick, Scott, Shavitt, Yuval, & Shir, Eran. (2007).  From the Cover: A model of Internet topology using k-shell decomposition.  Pnas, 104(27), 11150–11154.
  * Castellano & Pastor-Satorras, (2012) Castellano, & Pastor-Satorras, Romualdo. (2012).  Competing activation mechanisms in epidemics on networks.  Scientific reports, 2(371).
  * Chen et al., (2012) Chen, Duanbing, Lü, Linyuan, Shang, Ming-Sheng, Zhang, Yi-Cheng, & Zhou, Tao. (2012).  Identifying influential nodes in complex networks.  Physica a: Statistical mechanics and its applications, 391(4), 1777 – 1787.
  * Chen et al., (2010) Chen, Wei, Wang, Chi, & Wang, Yajun. (2010).  Scalable influence maximization for prevalent viral marketing in large-scale social networks.  Pages 1029–1038 of: Proceedings of the 16th acm sigkdd international conference on knowledge discovery and data mining.  KDD ’10.  New York, NY, USA: ACM.
  * Cohen et al., (2000) Cohen, Reuven, Erez, Keren, ben Avraham, Daniel, & Havlin, Shlomo. (2000).  Resilience of the Internet to Random Breakdowns.  Physical review letters, 85(21), 4626–4628.
  * Csardi & Nepusz, (2006) Csardi, Gabor, & Nepusz, Tamas. (2006).  The igraph software package for complex network research.  Interjournal, Complex Systems, 1695.
  * Freeman, (1977) Freeman, Linton C. (1977).  A set of measures of centrality based on betweenness.  Sociometry, 40(1), pp. 35–41.
  * Freeman, (1979) Freeman, Linton C. (1979).  Centrality in social networks conceptual clarification.  Social networks, 1(3), 215 – 239.
  * J. Goldenberg, (2001) J. Goldenberg, B. Libai, E. Muller. (2001).  Talk of the network: A complex systems look at the underlying process of word-of-mouth.  Marketing letters, 12(3), 211.
  * Kang et al., (2012) Kang, C., Molinaro, C., Kraus, S., Shavitt, Y., & Subrahmanian, V.S. 2012 (Aug.).  Diffusion centrality in social networks.  Proc. 2012 ieee/acm intl. conf. on advances in social networks analysis and mining (asonam-12).
  * Kempe et al., (2003) Kempe, David, Kleinberg, Jon, & Tardos, Éva. (2003).  Maximizing the spread of influence through a social network.  Pages 137–146 of: Kdd ’03: Proceedings of the ninth acm sigkdd international conference on knowledge discovery and data mining.  New York, NY, USA: ACM.
  * Kitsak et al., (2010) Kitsak, Maksim, Gallos, Lazaros K., Havlin, Shlomo, Liljeros, Fredrik, Muchnik, Lev, Stanley, H. Eugene, & Makse, Hernan A. (2010).  Identification of influential spreaders in complex networks.  Nat phys, 6(11), 888–893.
  * Klemm et al., (2012) Klemm, Konstantin, Serrano, M. Angeles, Eguiluz, Victor M., & San Miguel, Maxi. (2012).  A measure of individual role in collective dynamics: spreading at criticality.  Scientific reports, 2(292).
  * Leskovec, (2012) Leskovec, Jure. (2012).  Stanford network analysis project (snap).
  * Luis E. C. Rocha & Holme, (2010) Luis E. C. Rocha, Fredrik Liljeros, & Holme, Petter. (2010).  Information dynamics shape the sexual networks of internet-mediated prostitution.  Proceedings of the national academy of sciences, March.
  * Madar et al., (2004) Madar, N., Kalisky, T., Cohen, R., ben Avraham, D., & Havlin, S. (2004).  Immunization and epidemic dynamics in complex networks.  The european physical journal b - condensed matter and complex systems, 38(2), 269–276.
  * Moores et al., (2012) Moores, Geoffrey, Shakarian, Paulo, Howard, Nicholas, & Macdonald, Brian. (2012).  Influential Spreaders 2.  In progress.
  * Newman, (2011) Newman, Mark. (2011).  Network data.
  * Page et al., (1998) Page, L., Brin, S., Motwani, R., & Winograd, T. (1998).  The pagerank citation ranking: Bringing order to the web.  Pages 161–172 of: Proceedings of the 7th international world wide web conference.
  * R Development Core Team, (2011) R Development Core Team. (2011).  R: A language and environment for statistical computing.  R Foundation for Statistical Computing, Vienna, Austria.  ISBN 3-900051-07-0.
  * Seidman, (1983) Seidman, Stephen B. (1983).  Network structure and minimum degree.  Social networks, 5(3), 269 – 287.
  * Valiant, (1979) Valiant, Leslie G. (1979).  The complexity of enumeration and reliability problems.  Siam j. comput., 8(3), 410–421.
  * Wasserman & Faust, (1994) Wasserman, Stanley, & Faust, Katherine. (1994).  Social network analysis: Methods and applications. 1 edn.  Structural analysis in the social sciences, no. 8.  Cambridge University Press.
  * Zafarani & Liu, (2009) Zafarani, R., & Liu, H. (2009).  Social computing data repository at ASU.


# Suppressing Pink Elephants with Direct Principle Feedback

Louis Castricato  synthlabs.ai  EleutherAI  Brown University  Nathan Lile
synthlabs.ai  Suraj Anand  Brown University  Hailey Schoelkopf  EleutherAI
Siddharth Verma character.ai  Stella Biderman  EleutherAI

###### Abstract

Existing methods for controlling language models, such as RLHF and
Constitutional AI, involve determining which LLM behaviors are desirable and
training them into a language model. However, in many cases, it is desirable
for LLMs to be controllable at inference time, so that they can be used in
multiple contexts with diverse needs. We illustrate this with the Pink
Elephant Problem: instructing an LLM to avoid discussing a certain entity (a
“Pink Elephant”), and instead discuss a preferred entity (“Grey Elephant”). We
apply a novel simplification of Constitutional AI, Direct Principle Feedback,
which skips the ranking of responses and uses DPO directly on critiques and
revisions. Our results show that after DPF fine-tuning on our synthetic Pink
Elephants dataset, our 13B fine-tuned LLaMA 2 model significantly outperforms
Llama-2-13B-Chat and a prompted baseline, and performs as well as GPT-4 in on
our curated test set assessing the Pink Elephant Problem.

Suppressing Pink Elephants with Direct Principle Feedback

##  1 Introduction

In recent years, general-purpose generative models such as LLMs obtained via
training on massive unlabeled corpora have achieved great success. Training
desirable behaviors into LLMs can improve their usefulness; however,
controllability at inference time remains difficult. Current language models
still often fail to perform instruction following and reasoning tasks
reliably, especially tasks requiring compositional reasoning, complex and non-
standard instruction formats (Webson et al., 2023; Shi et al., 2023a), and
logical operations such as negation McKenzie et al. (2023). One particularly
challenging behavior to teach a language model to follow is to avoid
mentioning some topic when instructed to do so. Paradoxically, mentioning that
topic in the LLM’s prompt makes the model more likely to mention it even when
instructed not to, bearing a resemblance to the what is commonly known as the
“Pink Elephant effect” in psychology (Spiers, 2002).

In this paper we study controllable generation through the lens of the Pink
Elephant Problem: the challenge of instructing a model to not discuss an
undesired “Pink Elephant” entity or topic and instead discuss an alternative
desired “Grey Elephant” as depicted in Figure 1.

Figure 1: This chatbot is designed to help British students apply to British
universities, but doesn’t have up-to-date information about other
universities. “American Universities” is the Pink Elephant while “British
Universities” is the Grey Elephant.

To address this challenge, we leverage a novel form of Reinforcement Learning
from AI Feedback (RLAIF) (Bai et al., 2022b) that we term Direct Principle
Feedback. Recent research has shown that techniques like Reinforcement
Learning from AI Feedback (RLAIF) (Bai et al., 2022b) can not only improve a
model’s ability to remain harmless and helpful (Tunstall et al., 2023a; Ivison
et al., 2023) but can improve a model’s ability to reason (Shao et al., 2024;
Luo et al., 2023a; Lightman et al., 2023) and reduce a model’s tendency to
hallucinate (Tian et al., 2023). However results have shown that even with
finetuning and prompting, making a model “not” discuss a topic remains a
difficult and open problem (McKenzie et al., 2023; García-Ferrero et al.,
2023).

We find that by applying DPF with high quality synthetic data we can teach the
model the task “don’t talk about X” where X is determined at inference time.
This enables

###  1.1 Our Contributions

Our primary contributions are:

  1. 1.

We present the “Pink Elephant Problem”, a failure mode of current language
models.

  2. 2.

We introduce a novel approach for RLAIF, which we term Direct Principle
Feedback, which enables the use of Critiques and Revisions as natural language
AI Feedback, while simplifying the RLAIF pipeline significantly.

  3. 3.

We demonstrate an approach to teach models to dynamically obey new behavioral
constraints at inference time, via training our models to avoid any Pink
Elephant entity specified at inference time.

  4. 4.

We share our approaches for generating synthetic preference data, as well as
our models and codebase, in the hopes they will be helpful to the community.

Additionally, we share our approaches for generating synthetic preference
data, as well as our models and codebase, in the hopes they will be helpful to
the community.

##  2 Inference-Time Controllability via Direct Principle Feedback

Existing methods for preference learning fine-tuning generally consist of
selecting a set of rules or good behavior, known beforehand to the developer
(Bai et al., 2022b), and training a model to adhere to preference pairs
expressing this behavior. While this is useful from the perspective of a
developer who has behavior they want to assure in their deployed model, it
prevents downstream developers or users of widely distributed models from
making their own choices about moderation and desired outcomes based on novel
deployment scenarios or goals.

Instead, we wish to apply fine-tuning to the model so that it gains the
capability of being controlled without directly hard-coding our values into
the model. This has the benefit of allowing more tailored preferences to be
expressed based on the deployment scenario, and allows more stakeholders
across the AI value chain to participate in decision-making and inclusion of
their preferences. It also provides concrete benefits for the Pink Elephant
Problem: it enables a single high-quality model to be deployed in diverse
contexts where different concepts or terms should be avoided.

For more classical NLP tasks, the shift from fine-tuned task-specific models
to single, unified models capable of following arbitrary instructions (Sanh et
al., 2021; Longpre et al., 2023) has brought on tremendous performance
improvements and vastly easier deployment due to a single model being
controllable by a simple natural language instruction at inference time. Akin
to instruction tuning, the Pink Elephant Problem presents a unique set of
circumstances that draws analogies to meta-learning (Iyer et al., 2023).

Although it would be possible to train a model specifically to avoid
mentioning a single predetermined entity, for every different deployment of
the model or different Pink Elephant, a separate model would have to be
trained, repeating this process. Generating a separate fine-tuning dataset for
every such desired entity would be cost-prohibitive.

###  2.1 Simplifying RLAIF with Direct Principle Feedback

Reinforcement Learning from AI Feedback, as originally presented in Bai et al.
(2022b), uses a four-step process depicted in fig. 2.

  1. 1.

Finetune a model on examples of helpful requests and outputs (blue).

  2. 2.

Critique and revise those outputs to be more desirable, and fine-tune a new
model on those outputs (orange).

  3. 3.

Use your Supervised Fine-tuning (SFT) model to generate responses to a prompt
and have a human or AI system rank those responses (green).

  4. 4.

Feed the ranked responses into an preference learning algorithm such as PPO or
DPO to produce the final model (purple).

Figure 2: A high-level illustration of Constitutional AI (Bai et al., 2022b),
RLAIF without critiques, and DPF (ours). DPF streamlines the process to use
only a single step of AI feedback, and uses revisions according to a principle
as feedback. Note that the choice of objective such as DPO is arbitrary and
may be substituted.

Previous work has sought to simplify this pipeline by excluding the Critique
and Revision step and doing pairwise ranking of generations from the initial
model directly (Tunstall et al., 2023a; Zhu et al., 2023). This has the
advantage of requiring only a single preference collection and fine-tuning
step, and is effective for improving chat quality or teaching models to behave
as assistants. The simplified pipeline does not allow for teaching behaviors
on more complex principles, however, as done by Bai et al. (2022b).

We introduce Direct Principle Feedback (DPF), a novel pipeline for RLAIF that
leverages the fact that the data points before and after the revision is
applied provide a natural source of paired data for preference-based fine-
tuning. By feeding the (response, revision) pairs into the preference learning
algorithm, we are able to instill a more complex behavior akin to the
principles used in Bai et al. (2022b), and have further control over how our
paired preference data points differ within pairs (due to our controllable
Revision step), as compared to other RLAIF methods, which simplify the
pipeline by simply taking arbitrary model outputs to a prompt and scoring or
ranking them for goodness.

Concurrently, Huang et al. (2024) replicate the Constitutional AI pipeline and
perform DPO using the pre- and post- revision pairs both for SFT and for DPO.
However, they focus on replicating the process of Bai et al. (2022b), whereas
we look at using DPF for a novel controllability setting.

###  2.2 Applying DPF to the Pink Elephant Problem

Because high quality pairwise preferences are inherently difficult to generate
for the Pink Elephant Problem, a natural choice is to turn to revisions to
generate a high quality dataset with the desired behaviors. As such, The Pink
Elephant Problem is a prime candidate for DPF as opposed to a ranking-based
approach, which would have been much more difficult to control for specific
kinds of nuanced differentiations between dialogues containing the Pink
Elephant as opposed to those containing the desired Grey Elephant.

##  3 Dataset Generation

We curated a dataset of 162K multi-turn conversations on the Pink Elephant
Problem. The conversations cover 29 diverse domains including sports, health,
business, and politics. The dataset took approximately 2,000 A100 hours to
produce, with roughly 20,000 - 30,000 for prototyping.

###  3.1 Topics Generation

Figure 3: Initial dataset generation step, as described in section 3.1 and
section 3.2. A short list of topic areas is handwritten, GPT-4 is prompted to
generate many contrasting Pink Elephant entity pairs, and StableBeluga2
subsequently prompted to create a number of plausible subtopics for each
entity pair. Duplicates are removed at each step.

The first step in generating a highly diverse set of Pink Elephant pairs was
first to generate a highly diverse set of topics that our chatbot would
discuss. Similar to Luo et al. (2023b); Bradley et al. (2023, 2024); Lehman et
al. (2022) we believe that to produce high quality synthetic data, diversity
needs to be a major consideration in the dataset’s construction.

We used GPT-4 (Achiam et al., 2023) to generate a list of 200 diverse possible
topics that daily conversations are usually based off of (prompt in appendix
D). Then, we manually filtered the topics to ensure that they are in fact
common daily conversation topics and topics about which one could sensibly and
responsibly deploy a chatbot in production. We took the liberty to filter
irrelevant topics, such as “The legal system of extraterrestrial life.”

The filtered version of this initial list can be found in the appendix E. They
cover a wide range of topics, such as traveling, career, sports, health,
festivals, religions, and so on.

###  3.2 Pink Elephant Pairs

The next step was to generate a large number of Pink Elephant Pairs (PEPs),
approximately 2500, using GPT-4. Our goal is to generate similar yet
contrastive pairs because we want a generated dialogue about a Grey Elephant
topic to naturally culminate in the mention of a Pink Elephant.

To generate these pairs we used the prompt Generate a list of 100 (x, y) pairs
that represent [TOPIC] and their top alternatives/competitors. We then
manually check and filter the PEPs to ensure that they fit our definition of
Pink Elephants. Specifically, we verified that they were truly alternatives,
yet had specific differentiated qualities that a non-expert could identify.
For example, the pair “Nike - Adidas”, two competing companies, was generated
for the topic of sports, and the pair “Taj Mahal - Ellora Caves” was generated
as an example of two differing tourist destinations in India. Examples from
this list can be found in appendix F.

In the subsequent phases of our dataset creation process, we primarily employ
StableBeluga2-70B (Mahan et al., 2023) in conjunction with the "Best-of-N"
strategy (Stiennon et al., 2022), where N is set to 2. The selection criterion
is based on the perplexity of the generated output.

###  3.3 Unwanted Behavior Generation

The initial phase in creating paired preference data, which aims to embed an
optimal policy instructing a chatbot to refrain from discussing the Pink
Elephant, entails the generation of inappropriate behaviors. This includes
dialogues where the chatbot engages in conversations about the Pink Elephant
when instructed not to do so.

Figure 4: Generation of section 3.3 dialogues which exhibit undesired behavior
(mentioning the Pink Elephant in the final turn). We perform an intermediate
planning step.

Our aim in producing unwanted behavior is to foster a wide array of dialogues
wherein the chatbot eventually references a Pink Elephant. This requires the
initial creation of diverse conversational themes, designated as Attributes,
that logically lead to the mention of a Pink Elephant by the chatbot.

To foster a variety of authentic dialogues that incorporate our PEPs, we
commence by creating 50 attributes aimed at facilitating transitions between
contrasting PEPs, thereby initiating discussions that progress from the Grey
Elephant to the Pink Elephant with the aid of StableBeluga2-70B. These
dialogues serve as instances of the behaviors we seek to avoid, known as Pink
Elephant Problem failures. Our qualitative analysis revealed that
incorporating this strategic planning step significantly enhances the
naturalness of the dialogues, with examples of these plans available in the
appendix.

However, despite employing these attributes and PEPs, StableBeluga2-70B was
unable to produce conversations that were both fluent and realistic, meeting
our predefined standards. Consequently, we have implemented a Dialogue
Planning phase, instructing our model to first outline a conversation between
a user and the chatbot that transitions from discussing the Grey Elephant to
mentioning the Pink Elephant.

Subsequently, we engage StableBeluga2-70B to execute a conversation based on
the dialogue plan and PEPs, facilitating a conversation between the user and
chatbot that aligns with the planned sequence. The original plan is then
discarded, with only the final dialogue being retained. We discovered that the
introduction of a preliminary planning phase markedly improves the dialogues’
quality and naturalness, thereby increasing the probability of successfully
incorporating a mention of the Pink Elephant as designed. This innovative
planning step distinguishes our method from conventional practices in RLAIF
data generation.

We include examples of the prompts, dialogues, and an example dialogue plan in
appendix G.

###  3.4 Revised Behavior Generation

Figure 5: Critique and Revision step, as described in section 3.4. We ask our
model to rewrite the final dialogue turn to remove mentions of the Pink
Elephant.

Our next step is to generate examples of good behavior (i.e., the chatbot
successfully avoiding discussing the Pink Elephant) which we can utilize for
performing DPO or DPF.

We apply Critiques and Revisions to amend our negative examples.

First, we prompt StableBeluga2-70B to generate a critique of the chatbot’s
final response in the dialogue, which frequently mentions the undesired Pink
Elephant. Next, we ask the model to generate a revision: the model rewrites
this final response, removing the mention of the Pink Elephant and instead
redirecting the conversation back to the Grey Elephant.

We found that including the plan in context for the revision biased the
revision and critique process to not actually remove the Pink Elephant. As
such, during the critique and revision process, we remove the plan.

Following this step, we now have paired data points where the original
conversation contains undesired behavior (mentioning the Pink Elephant), and
the post-revision data point contains behavior more in line with our
behavioral goal.

###  3.5 Data Cleaning

Figure 6: Data filtering step, as described in section 3.5. We use a number of
distance metrics or heuristics to identify dialogue pairs which wrongly
mention the Pink Elephant (‘Rugby’) outside the final turn or post-revision.

The initial stage of data processing involved truncating dialogues at the
initial instance of ’Pink Elephant’ mention by the chatbot. Subsequent
filtering aimed to enhance dialogue quality and adherence to predefined
criteria. This process entailed the exclusion of dialogues where:

  1. 1.

The chatbot references the Pink Elephant prior to the final utterance,
indicating a lapse in initial data cleaning.

  2. 2.

The Pink Elephant is not mentioned in the final utterance pre-revision.

  3. 3.

The Pink Elephant is mentioned in the final utterance post-revision.

These exclusions were facilitated using Levenshtein distance, Hamming
distance, and cosine similarity measures applied to embeddings. For
embeddings, we used DistilBERT (Sanh et al., 2019) due to its notable speed,
allowing for the efficient evaluation of different filtering thresholds. We
established a cosine similarity threshold of 0.8 between the final utterance
embedding and the Pink Elephant reference as our filtering criterion. This
threshold was chosen because it qualitatively raised the quality of our data,
striking a balance by filtering out insufficient examples without excessively
diminishing the dataset.

The dataset was then partitioned into training, validation, and test sets in a
96%-2%-2% ratio among ’Pink Elephant Pairs.’ We ensured non-overlapping ’Pink
Elephant Pairs’ across these subsets.

##  4 Experiments

Here we describe the use of DPF and our generated dataset to suppress mentions
of Pink Elephant entities.

###  4.1 Training

Because our dataset contains examples of conversations where the Pink Elephant
is mentioned by the user and the chatbot must steer away from the conversation
(section 3.3, section 3.4), our dataset is not desirable for cloning behavior
via SFT. Therefore, we chose to solely perform DPO fine-tuning without an
extra SFT step, via initializing from a model that has already undergone SFT
training.

We choose the OpenHermes 7B and 13B
models111https://huggingface.co/teknium/OpenHermes-7B and
https://huggingface.co/teknium/OpenHermes-13B for two reasons: first, they
provide a strong instruction-tuned model as base. Second, the terms of Llama
2’s license prohibited us from using our dataset generated by a Llama 2
derivative to train any models not fine-tuned from Llama 2, and the OpenHermes
models are fine-tuned from Llama-2 as base models.

We use the codebase from Tunstall et al. (2023a, b) for our DPO training runs.
We train our models using bfloat16 precision, a global batch size of 64, and
the RMSProp optimizer following Rafailov et al. (2023), and using Flash
Attention (Dao et al., 2022; Dao, 2023). Because we wished to obtain a model
that otherwise performed similarly to our baseline, but with the added ability
to avoid Pink Elephants when specified in its system prompt, we used a
relatively large $\beta=0.5$ for DPO. Our full training configurations are
available in our public codebase 222https://github.com/EleutherAI/alignment-
handbook/tree/pink-elephants-dpo. Similarly to Zephyr (Tunstall et al.,
2023a), we chose to train for 3 epochs, but chose to use only 1 epoch for our
final models (appendix C).

###  4.2 Baselines

We compare our fine-tuning method to several different baselines:

  1. 1.

The base OpenHermes models, prompted with our system prompt to avoid the Pink
Elephant.

  2. 2.

Llama-2-chat (Touvron et al., 2023) with a system prompt instructing it to
avoid the Pink Elephant.

  3. 3.

GPT-4 with a system prompt instructing it to avoid the Pink Elephant.

We tried to include a comparison between models trained with DPO and ones
trained with ILQL (Snell et al., 2022), but were unable to find the right
hyperparameters for it to converge; the model would either incorrectly mention
the Pink Elephant or produce incoherent language. We conjecture that ILQL
would require additional reward shaping to achieve a performant model, and so
focus on DPO.

We additionally attempted prompting the base and DPF-trained OpenHermes models
using Classifier-Free Guidance (CFG) (Sanchez et al., 2023; Shi et al., 2023b)
with a guidance scale of 1.5 for enhanced sensitivity to the system prompt.
However, we saw that CFG had minimal impact on the scoring of our models’
generations, and so did not continue using this as a baseline. Full results
with CFG can be found in appendix I.

###  4.3 Quantitative Evaluation

Model |  Base Rate $\downarrow$ |  With Prompt $\downarrow$ | $\mathbf{\Delta}\uparrow$
---|---|---|---
OpenHermes-7B | $0.33\pm 0.010$ | $0.36\pm 0.010$ | $-0.03\pm 0.013$
w/ DPF | $0.34\pm 0.010$ | $0.17\pm 0.008$ | $0.17\pm 0.012$
OpenHermes-13B | $0.34\pm 0.010$ | $0.34\pm 0.010$ | $0.00\pm 0.013$
w/ DPF | $0.34\pm 0.010$ | $0.15\pm 0.010$ | $0.19\pm 0.012$
Llama-2-13B-Chat | $0.33\pm 0.009$ | $0.25\pm 0.009$ | $0.08\pm 0.013$
GPT-4 | $0.33\pm 0.009$ | $0.13\pm 0.009$ | $0.20\pm 0.013$
Table 1: Rate at which the model talks about the Pink Elephant (lower is
better). Base Rate $\downarrow$ is without any instruction regarding the Pink
Elephant and represents the model’s natural tendency based on its training and
the dialog. With Prompt $\downarrow$ is the rate when specifically instructed
to avoid the Pink Elephant. While initially the OpenHermes models bring up the
Pink Elephant as often or more often when told to avoid it, our methodology
decreases the incidence of the Pink Elephant when prompted to avoid it to
GPT-4-level performance.

To perform evaluation at scale on whether the model mentions the Pink Elephant
in its generation, we employ GPT-4 in combination with human annotations on a
subset of examples for labeling agreement. Here is the prompt format that we
use GPT-4 for evaluation of Pink Elephants.

GPT-4 Evaluator Prompt A dialogue between a user (USER) and a chatbot (AGENT)
which has been instructed to avoid mentioning the topic of <pink_elephant>.
Your job is to determine if the AGENT has mentioned this topic. Does AGENT
text discuss or contain any direct or indirect mention of <pink_elephant ||
grey_elephant>? USER: <user_question> AGENT: <agent_response> ⋮

Given this setup, we evaluate on a held out test set consisting of 2% of the
original generated dataset. Specifically, given an example conversation in the
test set, we regenerate the response to the final user question (i.e. the
final turn). This is a set of user-chatbot conversations meant to more
frequently than chance elicit a response that includes the Pink Elephant. For
some data points this is an adversarial setting, where the user asks directly
about the Pink Elephant, while for other data points, bringing up the Pink
Elephant may be unnatural or easily avoided. Identical to training, the model
is instructed in its system prompt that it is not allowed to bring up the Pink
Elephant in its answers, but instead should respond with something related to
the Grey Elephant.

For each model, we report the following metrics on the test set:

  * •

Base Rate: proportion of examples where the Pink Elephant is mentioned, when
the system prompt does not mention the Pink Elephant.

  * •

With Prompt: proportion of examples where the model successfully refrained
from mentioning the Pink Elephant when specifically instructed to avoid the
Pink Elephant.

  * •

Base Rate - With Prompt: proportion of test set examples for which the
inclusion of a system prompt disallowing the Pink Elephant successfully alters
behavior on the model. Our most important metric, this shows the improved
effectiveness of our algorithm during inference time.

Our analysis revealed a consistent Base Rate across models, which aligns with
expectations given that the base rate is determined by the prevalence of
naturally occurring responses with Pink Elephants in the test set. We also see
that our finetuning does not negatively impact performance in scenarios
without a prompt that explicitly forbids Pink Elephant.

The difference in Pink Elephant mention rate reveals substantial disparities
in OpenHermes model performances with and without DPF. This metric
demonstrates how the introduction of a prompt explicitly prohibiting Pink
Elephant references significantly modifies the frequency of successful Pink
Elephant avoidances in the test set. Notably, baseline OpenHermes models
exhibit an equivalent or increased tendency to mention Pink Elephants
following such directive prompts. In contrast our DPF’d models adhere to these
directives far more effectively, surpassing Llama-2 chat in compliance and
achieving performance on par with GPT-4 in this setting.

To confirm the validity of GPT-4 as our evaluator at scale, we had two authors
assess labeling agreement on 200 generated examples: 50 for each of
OpenHermes-13B, OpenHermes-7B w/ DPF, OpenHermes-13B w/ DPF, and GPT-4.
Judgements were assessed using only the information given to GPT-4, and thus
blind to (1) which model generated the response, (2) GPT-4’s label for Pink
Elephant mention, and (3) the other annotator’s label. If disagreement was not
satisfactorily minimal, further prompt engineering, or not using GPT-4 as an
evaluator would have been required. The agreement between GPT-4 and Annotator
1 was $98.5\%$, GPT-4 and Annotator 2 was $94.5\%$, and Annotator 1 and
Annotator 2 was $95.5\%$. Therefore, GPT-4 is effective for grading responses
in this setup.

Finally, we additionally evaluate on 4 Open LLM Leaderboard (Beeching et al.,
2023) tasks: MMLU (Hendrycks et al., 2021), TruthfulQA (Lin et al., 2022),
HellaSwag (Zellers et al., 2019), and (Clark et al., 2018) using v0.4.1 of the
LM Evaluation Harness (Gao et al., 2023) and MT-Bench (Zheng et al., 2023) to
illustrate quantitatively that our model retains its previous performance
after fine-tuning on the Pink Elephants dataset.

Model | MT-Bench | Leaderboard
---|---|---
OpenHermes-7B | 5.19 | 57.40
w/ DPF (Ours) | 5.28 | 58.12
OpenHermes-13B | 6.28 | 61.36
w/ DPF (Ours) | 6.09 | 61.75

###  4.4 Qualitative Evaluation

We examined the models’ outputs to assess their quality, finding that the
resulting chat remained coherent and fluent. Redirection was usually quite
graceful; for example, the output might be "I am sorry, but I am not an expert
in photography. However, I can recommend some composition techniques for still
life painting". When the question does not explicitly reference the Pink
Elephant, the model answers with reasonable alternatives.

In contrast, when explicitly asked about the Pink Elephant, the baseline
OpenHermes models (before DPO) would often either (1) answer the question
without heeding the prompt or (2) become confused and first answer the
question, then apologize and state that it is not allowed to talk about the
Pink Elephant. Both of these constitute failure cases. Additionally, we found
that when we prompted the baseline OpenHermes models to avoid discuss a Pink
Elephant, the model would bring up the Pink Elephant marginally more
frequently, and certainly not less frequently. This strengthened our belief
that the Pink Elephant problem is a significant failure mode.

##  5 Conclusion and Future Work

We have introduced the Pink Elephant Problem, and demonstrated a technique to
solve this problem via an addition to existing alignment tuning processes. Our
methodology can easily transfer to imbuing novel behavioral specifications or
addressing other failure modes of current language model assistants. We
demonstrate a best-practices method for generating an AI Feedback/preference
dataset for a given objective or task and using it for preference fine-tuning
/ RLAIF, and we believe this technique can translate. We hope that future work
explores the ability of DPO alongside multi-task synthetic preference datasets
to provide fine-grained control over model behaviors.

##  6 Ethical Considerations

While RLAIF offers promising solutions to the “Pink Elephant problem” in AI,
it also raises ethical considerations. The reliance on AI for feedback loops
necessitates a careful design to ensure that the AI’s own biases or
limitations do not adversely affect the training process. It’s crucial to
maintain transparency and ethical oversight in these systems to prevent the
propagation of harmful biases or unethical content.

We also note that, although our approach improves the controllability of LLM
systems, thus potentially reducing risks from undesirable or unsafe
generations, our method can also be used to improve the censorship of LLM-
based applications. However, we believe that the benefits of the approach we
suggest for meta-learning dynamic behavioral restrictions on LLMs will allow a
greater number of stakeholders and users to make their own informed choices
about moderation, such as based on their cultural context.

##  7 Limitations

Although we achieve strong results on reducing the Pink Elephant Problem as a
failure mode of current LLMs, there are a number of improvements that could be
made in future work.

#### More complex constraints

One could investigate using ours or a similar dataset alongside other targeted
datasets, to simultaneously achieve strong controllability on the Pink
Elephant Problem and on other desirable failure modes or qualities.

#### Generalization properties

Although we attempt to avoid both direct and indirect mentions of Pink
Elephants, future work could explore the generalization or propagation of
avoidance of Pink Elephant entities in our work–for example, instructing a
model to avoid a whole topic or category of entities, or to avoid two or more
specific entities at once.

#### Flexible safety training

Future work could also extend our meta-learning approach to controllability by
applying it to the more typical Constitutional AI or safety-training setup,
permitting a model’s safety behaviors and properties to be controlled by a
downstream deployer based on their preferences. Such a method would allow a
greater amount of stakeholders across the AI value chain to have input on
desired model behaviors and the nature of “safety”.

## Acknowledgements

We thank Nathan Lambert for helpful feedback on literature review. We thank
Ellie Pavlick, Zheng Xin Yong, Qinan Yu, Josh Rowe, and the EleutherAI discord
for reviewing and providing feedback on a draft of our paper.

Hailey Schoelkopf’s and Stella Biderman’s work was funded in part by a grant
from the Omidyar Network.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Bai et al. (2022a) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022a.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_.
  * Bai et al. (2022b) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022b.  Constitutional ai: Harmlessness from ai feedback.  _arXiv preprint arXiv:2212.08073_.
  * Beeching et al. (2023) Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. 2023.  Open llm leaderboard.  https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard.
  * Belrose et al. (2023) Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. 2023.  Leace: Perfect linear concept erasure in closed form.
  * Bradley et al. (2023) Herbie Bradley, Andrew Dai, Hannah Teufel, Jenny Zhang, Koen Oostermeijer, Marco Bellagente, Jeff Clune, Kenneth Stanley, Grégory Schott, and Joel Lehman. 2023.  Quality-diversity through ai feedback.
  * Bradley et al. (2024) Herbie Bradley, Honglu Fan, Theodoros Galanos, Ryan Zhou, Daniel Scott, and Joel Lehman. 2024.  The openelm library: Leveraging progress in language models for novel evolutionary algorithms.  _Genetic Programming Theory and Practice XX_.
  * Castricato et al. (2022) Louis Castricato, Alexander Havrilla, Shahbuland Matiana, Michael Pieler, Anbang Ye, Ian Yang, Spencer Frazier, and Mark Riedl. 2022.  Robust preference learning for storytelling via contrastive reinforcement learning.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017.  Deep reinforcement learning from human preferences.  In _Advances in Neural Information Processing Systems_ , volume 30.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the ai2 reasoning challenge.
  * Dao (2023) Tri Dao. 2023.  Flashattention-2: Faster attention with better parallelism and work partitioning.  _arXiv preprint arXiv:2307.08691_.
  * Dao et al. (2022) Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022.  FlashAttention: Fast and memory-efficient exact attention with IO-awareness.  In _Advances in Neural Information Processing Systems_.
  * Eldan and Russinovich (2023) Ronen Eldan and Mark Russinovich. 2023.  Who’s harry potter? approximate unlearning in llms.
  * Ethayarajh et al. (2024) Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024.  Kto: Model alignment as prospect theoretic optimization.
  * Fu et al. (2023) Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. 2023.  Improving language model negotiation with self-play and in-context learning from ai feedback.
  * Gao et al. (2023) Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2023.  A framework for few-shot language model evaluation.
  * García-Ferrero et al. (2023) Iker García-Ferrero, Begoña Altuna, Javier Álvez, Itziar Gonzalez-Dios, and German Rigau. 2023.  This is not a dataset: A large negation benchmark to challenge large language models.  _arXiv preprint arXiv:2310.15941_.
  * Glaese et al. (2022) Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. 2022.  Improving alignment of dialogue agents via targeted human judgements.  _arXiv preprint arXiv:2209.14375_.
  * Gulcehre et al. (2023) Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. 2023.  Reinforced self-training (rest) for language modeling.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021.  Measuring massive multitask language understanding.
  * Hong et al. (2023) Joey Hong, Sergey Levine, and Anca Dragan. 2023.  Zero-shot goal-directed dialogue via rl on imagined conversations.
  * Huang et al. (2024) Shengyi Huang, Lewis Tunstall, Edward Beeching, Leandro von Werra, Omar Sanseviero, Kashif Rasul, and Thomas Wolf. 2024.  Constitutional ai recipe.  _Hugging Face Blog_.  Https://huggingface.co/blog/constitutional_ai.
  * Ilharco et al. (2022) Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022.  Editing models with task arithmetic.  _arXiv preprint arXiv:2212.04089_.
  * Ivison et al. (2023) Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023.  Camels in a changing climate: Enhancing lm adaptation with tulu 2.
  * Iyer et al. (2023) Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O’Horo, Gabriel Pereyra, Jeff Wang, Christopher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, and Ves Stoyanov. 2023.  Opt-iml: Scaling language model instruction meta learning through the lens of generalization.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Lambert et al. (2023) Nathan Lambert, Thomas Krendl Gilbert, and Tom Zick. 2023.  The history and risks of reinforcement learning and human feedback.
  * Lee et al. (2023) Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. 2023.  Rlaif: Scaling reinforcement learning from human feedback with ai feedback.  _arXiv preprint arXiv:2309.00267_.
  * Lehman et al. (2022) Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Yeh, and Kenneth O. Stanley. 2022.  Evolution through large models.
  * Lightman et al. (2023) Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023.  Let’s verify step by step.  _ArXiv_ , abs/2305.20050.
  * Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022.  Truthfulqa: Measuring how models mimic human falsehoods.
  * Longpre et al. (2023) Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023.  The flan collection: Designing data and methods for effective instruction tuning.  _arXiv preprint arXiv:2301.13688_.
  * Luo et al. (2023a) Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-Guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023a.  Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct.  _ArXiv_ , abs/2308.09583.
  * Luo et al. (2023b) Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023b.  Wizardcoder: Empowering code large language models with evol-instruct.
  * Mahan et al. (2023) Dakota Mahan, Ryan Carlow, Louis Castricato, Nathan Cooper, and Christian Laforte. 2023.  Stable beluga models.
  * Matiana et al. (2021) Shahbuland Matiana, JR Smith, Ryan Teehan, Louis Castricato, Stella Biderman, Leo Gao, and Spencer Frazier. 2021.  Cut the carp: Fishing for zero-shot story evaluation.  _arXiv preprint arXiv:2110.03111_.
  * McKenzie et al. (2023) Ian R. McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Aaron Kirtland, Alexis Ross, Alisa Liu, Andrew Gritsevskiy, Daniel Wurgaft, Derik Kauffman, Gabriel Recchia, Jiacheng Liu, Joe Cavanagh, Max Weiss, Sicong Huang, The Floating Droid, Tom Tseng, Tomasz Korbak, Xudong Shen, Yuhui Zhang, Zhengping Zhou, Najoung Kim, Samuel R. Bowman, and Ethan Perez. 2023.  Inverse scaling: When bigger isn’t better.
  * Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022.  Locating and editing factual associations in gpt.  _Advances in Neural Information Processing Systems_ , 35:17359–17372.
  * Meng et al. (2023) Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. 2023.  Mass-editing memory in a transformer.  In _The Eleventh International Conference on Learning Representations_.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023.  Direct preference optimization: Your language model is secretly a reward model.  _arXiv preprint arXiv:2305.18290_.
  * Sanchez et al. (2023) Guillaume Sanchez, Honglu Fan, Alexander Spangher, Elad Levi, Pawan Sasanka Ammanamanchi, and Stella Biderman. 2023.  Stay on topic with classifier-free guidance.  _arXiv preprint arXiv:2306.17806_.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Sanh et al. (2021) Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021.  Multitask prompted training enables zero-shot task generalization.  _arXiv preprint arXiv:2110.08207_.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_.
  * Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024.  Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
  * Shi et al. (2023a) Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023a.  Large language models can be easily distracted by irrelevant context.
  * Shi et al. (2023b) Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Scott Wen tau Yih. 2023b.  Trusting your evidence: Hallucinate less with context-aware decoding.
  * Singh et al. (2023) Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, and Noah Fiedel. 2023.  Beyond human data: Scaling self-training for problem-solving with language models.
  * Snell et al. (2022) Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. 2022.  Offline rl for natural language generation with implicit language q learning.  _arXiv preprint arXiv:2206.11871_.
  * Spiers (2002) Jude A Spiers. 2002.  The pink elephant paradox (or, avoiding the misattribution of data).  _International Journal of Qualitative Methods_ , 1(4):36–44.
  * Stiennon et al. (2022) Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2022.  Learning to summarize from human feedback.
  * Team et al. (2023) Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_.
  * Tian et al. (2023) Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D. Manning, and Chelsea Finn. 2023.  Fine-tuning language models for factuality.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Tunstall et al. (2023a) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. 2023a.  Zephyr: Direct distillation of lm alignment.
  * Tunstall et al. (2023b) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Alexander M. Rush, and Thomas Wolf. 2023b.  The alignment handbook.  https://github.com/huggingface/alignment-handbook.
  * Turner et al. (2023) Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. 2023.  Activation addition: Steering language models without optimization.
  * Wang et al. (2023a) Tianlu Wang, Ping Yu, Xiaoqing Tan, Sean O’Brien, Ramakanth Pasunuru, Jane Dwivedi-Yu, O. Yu. Golovneva, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2023a.  Shepherd: A critic for language model generation.  _ArXiv_ , abs/2308.04592.
  * Wang et al. (2023b) Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, et al. 2023b.  Helpsteer: Multi-attribute helpfulness dataset for steerlm.  _arXiv preprint arXiv:2311.09528_.
  * Webson et al. (2023) Albert Webson, Alyssa Marie Loo, Qinan Yu, and Ellie Pavlick. 2023.  Are language models worse than humans at following prompts? it’s complicated.
  * Welleck et al. (2020) Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020.  Neural text generation with unlikelihood training.  In _International Conference on Learning Representations_.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  Hellaswag: Can a machine really finish your sentence?
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023.  Judging llm-as-a-judge with mt-bench and chatbot arena.
  * Zhu et al. (2023) Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. 2023.  Starling-7b: Improving llm helpfulness & harmlessness with rlaif.

##  Appendix A Author Contributions

#### Louis Castricato

Wrote the infrastructure for generating synthetic data, implemented data
pipeline and RLHF training code, performed data filtering, and wrote the
paper. Ideation of DPF and the pink elephant problem.

#### Nathan Lile

Set up compute infrastructure and maintained cluster health for the duration
of data generation, helped babysit runs. Conducted preliminary evaluation.
Ideation of DPF and the pink elephant problem.

#### Suraj Anand

Performed synthetic dataset generation and helped with data filtering.
Designed and implemented evaluation, and wrote the paper. Helped concretize
pink elephant problem.

#### Hailey Schoelkopf

Trained the models, helped design and perform evaluation, and wrote the paper.

#### Siddharth Verma

Helped implement evaluation, performed initial ILQL experiments, and edited
the paper.

#### Stella Biderman

Advised the project, analyzed the results, and wrote the paper.

##  Appendix B Additional Related Work

#### Post-Training Interventions

Pretrained LLMs (Team et al., 2023; Touvron et al., 2023; Jiang et al., 2023;
Achiam et al., 2023) often output undesirable content or fail to follow user
instructions when used directly. It has become common practice to perform a
post-pretraining adaptation step, often called post-training, to suppress
toxic or otherwise undesired outputs, adapt models to new domains or formats
such as interactive chat, or otherwise control their behavior. While
traditionally post-training was done via supervised fine-tuning (Sanh et al.,
2021; Longpre et al., 2023) recently there has been a surge of popularity in
methods like RLHF (Christiano et al., 2017), which consists of collecting
labeled preference data from human annotators about model outputs, and using
this signal of quality to optimize the LLM directly to give better outputs.

RLHF is traditionally done by learning a reward model and maximizing this
reward using a traditional RL algorithm such as Proximal Policy Optimization
(PPO) (Schulman et al., 2017) or other Actor-Critic methods (Glaese et al.,
2022). More recently, algorithms such as Direct Preference Optimization (DPO)
(Rafailov et al., 2023) and Kahneman-Tversky Optimization (KTO) (Ethayarajh et
al., 2024), which use novel loss functions to minimize loss directly on a
given dataset of preference labels without more involved RL techniques, have
come into fashion, though effort is still put into optimizing and improving
PPO for RLHF (Shao et al., 2024; Wang et al., 2023b).

#### AI Feedback and Synthetic Preference Data

A key constraint on RLHF is the high cost of human-annotated preference data
(Lambert et al., 2023). This can make high-quality RLHF, especially in
settings where off-the-shelf preference datasets are not available,
prohibitively expensive and difficult to scale.

First proposed in Bai et al. (2022b), a recent approach to alleviate this
drawback is to use “AI Feedback”, or synthetically-generated preference data,
to perform RLHF, termed Reinforcement Learning from AI Feedback (RLAIF).
Recently, there has been a shift from performing preference learning using
conventional RL (Bai et al., 2022a) to utilizing predominately preference
learning based approaches, like Direct Policy Optimization (DPO) or Reinforced
Self-Training (ReST) (Rafailov et al., 2023; Gulcehre et al., 2023; Singh et
al., 2023).

A core component of the Constitutional AI pipeline are the Critique and
Revision operations, where a separate model, besides the one being actively
trained, first generates a Critique of an utterance made by the student model
and then Revises the student model’s utterance given said critique and the
prior context. Critiques and Revisions have been used to improve models’
outputs and adherence to a particular characteristic or behavior (Matiana et
al., 2021; Castricato et al., 2022; Bai et al., 2022b; Wang et al., 2023a; Fu
et al., 2023), although the majority of AI Feedback approaches use several
(subsequently ranked) generations from a given model as a preference dataset
(Tian et al., 2023; Hong et al., 2023; Bai et al., 2022b; Lee et al., 2023;
Zhu et al., 2023).

#### Other Control Interventions

Preference-based Instruction Tuning or RLHF is not the only approach used to
control model behaviors.

Depending on the scope or time at which interventions are made, different
options are available.

Several techniques exist to perform fine-tuning of a model to edit their
knowledge. For example, several methods allow for edits to model weights for
targeted modification of model knowledge (Meng et al., 2022; Ilharco et al.,
2022; Meng et al., 2023), such as changing a specific fact as represented in
model weights. Methods have also been proposed for machine unlearning:
training to forget a specific concept or data point via unlikelihood losses
(Welleck et al., 2020) or more involved genericization processes (Eldan and
Russinovich, 2023).

However, both these approaches require knowing the entity that is to be edited
or forgotten before deployment and model fine-tuning, and to perform this
process for every such already-known entity that must be avoided. In contrast,
our goal is to create a single model which when prompted at inference time to
avoid a novel Pink Elephant, can steer conversation away from the subject,
based solely on this natural language instruction.

More similar to our approach, other methods exist for performing modifications
to model behavior at inference time–for example, several different decoding
methods have been proposed to cause models to follow their prompt more
strongly (Sanchez et al., 2023; Shi et al., 2023b), or otherwise modify
behavior by simply changing the sampling algorithm or hyperparameters when
generating from the model.

Others such as LEACE (Belrose et al., 2023) or activation engineering (Turner
et al., 2023), are in between these two categories. They allow the use of a
single, already fine-tuned model, similar to our approach, but however still
require a small dataset of positive and negative responses and short fitting
step collecting activation statistics – substantially cheaper than full fine-
tuning.

##  Appendix C DPO Training Duration

Epoch | Success %
---|---
0 | 65.6%
1 | 80.2%
2 | 81.2%
3 | 82.8%
Table 2: The effect of training duration on DPO performance for OpenHermes-7B.
All scores are % on the validation set.

Here we report the results of training for up to 3 epochs using DPO on
OpenHermes-7B. We find that while we achieve significant gains in the number
of successful Pink Elephant avoidances after 1 epoch of DPO fine-tuning,
performing a subsequent second and third epoch only marginally improves the
model’s performance on this task.

We hypothesize that our high value of $\beta=0.5$ in DPO may be a potential
cause of this result. However, because our desired outcome is a model which
integrates the ability to solve the Pink Elephant Problem while remaining
generally useful for other chat purposes, using only a single epoch aligns
with our desired outcome.

##  Appendix D Prompt to generate initial categories

Give me a diverse and different 200 general topics that humans talk about in
real life.

##  Appendix E Subset of initial categories

Subset of initial categories used to generate diverse Pink Elephant Pairs 1.
Weather 2. Seasons 3. Sports 4. Food 5. Cuisine 6. Travel 7. Countries 8.
Health and fitness 9. Fruits 10. Vegetables 11. Historical figures 12. Career
and jobs 13. Hobbies 14. Pets 15. Music 16. Companies 17. Movie 18. Awards 19.
Theme parks 20. Schools and education

##  Appendix F Example Pink Elephant pairs

Ten generated Pink Elephant pairs that were used in training Martin Luther
King Jr., Malcolm X Absurdism, Existentialism Staples, Office Depot Buy-side,
Sell-side Ski Racing, Snowmobile Racing Nature Valley Granola, KIND Granola
Gal Gadot, Margot Robbie Climate Denial, Climate Science Skydiving school,
Underground caving Fixed Costs, Variable Costs

##  Appendix G Dialogue Planning

Below is an example plan for the PEP (Live orchestral performance, Synthesized
music concert) where Live orchestral performance is the Pink Elephant

Dialogue plan for the attribute of "crossover concerts" for the PEP (Live
orchestral performance, Synthesized music concert) Plan: 1. Discuss music
preferences 2. Mention live concerts 3. Ask about favorite concert experience
4. Introduce crossover concerts 5. Recommend Live orchestral performance

##  Appendix H Unwanted Behavior Generation

Prompt for generating unwanted behavior Design a dialogue between a user and
an agent where the user seeks guidance concerning topic. Your task is to
create a conversation that seamlessly and organically culminates in the agent
bringing up pink_elephant or its exclusive services in the final response.
This should not feel forced or out of context. Begin by devising a plan that
sets the trajectory of the conversation. Each point in your plan should
incrementally steer the discussion closer to a scenario or query where
mentioning pink_elephant becomes an apt and natural conclusion. After
outlining your plan, segue into the dialogue itself using "—". The dialogue
should be succinct, with each message being no longer than a sentence or two.
Remember, neither the USER nor the AGENT should message twice consecutively.
The dialogue should end with the agent’s utterance, which is where the
reference to pink_elephant will occur. Format as: Plan: 1. 2. … N. — USER: ….
AGENT: …. … USER: …. AGENT: …

##  Appendix I Full Evaluation Results

Model | Guidance |  Base Rate $\downarrow$ |  With Prompt $\downarrow$ | $\mathbf{\Delta}\uparrow$
---|---|---|---|---
OpenHermes-7B | None | 0.33 | 0.36 | -0.03
w/ DPF | None | 0.34 | 0.17 | 0.17
OpenHermes-7B | CFG | 0.33 | 0.36 | -0.03
w/ DPF | CFG | 0.34 | 0.17 | 0.17
OpenHermes-13B | None | 0.34 | 0.34 | 0.00
w/ DPF | None | 0.34 | 0.15 | 0.19
OpenHermes-13B | CFG | 0.34 | 0.33 | 0.00
w/ DPF | CFG | 0.34 | 0.16 | 0.19
Llama-2-13B-Chat | None | 0.33 | 0.25 | 0.08
GPT-4 | None | 0.33 | 0.13 | 0.20