# Activation Addition: Steering Language Models Without Optimization

Alexander Matt Turner
DeepMind
<EMAIL_ADDRESS>
&Lisa Thiergart
MIRI
&Gavin Leech
University of Bristol
<EMAIL_ADDRESS>
David Udell
Independent researcher
&Juan J. Vazquez
Arb Research
&Ulisse Mini
MATS
&Monte MacDiarmid
Anthropic

###### Abstract

Reliably controlling the behavior of large language models is a pressing open
problem. Existing methods include supervised finetuning, reinforcement
learning from human feedback, prompt engineering and guided decoding. We
instead investigate activation engineering: modifying activations at
inference-time to predictably alter model behavior. We bias the forward pass
with a ‘steering vector’ implicitly specified through natural language. Past
work learned these steering vectors; our Activation Addition (ActAdd) method
instead computes them by taking activation differences resulting from pairs of
prompts. We demonstrate ActAdd on a range of LLMs (LLaMA-3, OPT, GPT-2, and
GPT-J), obtaining SOTA on detoxification and negative-to-positive sentiment
control. Our approach yields inference-time control over high-level properties
of output like topic and sentiment while preserving performance on off-target
tasks. ActAdd takes far less compute and implementation effort than finetuning
or RLHF, allows users control through natural language, and its computational
overhead (as a fraction of inference time) appears stable or improving over
increasing model size.

##  1 Introduction

The success of large language models (LLMs) pretrained on massive corpora
(Peters et al. 2018; Devlin et al. 2018; Radford et al. 2019; Brown et al.
2020) comes with a large caveat: our ability to control the output of these
models remains partial & computationally expensive. Controls include
supervised finetuning (Devlin et al. 2018), reinforcement learning from human
feedback (RLHF) (Ziegler et al. 2019), prompt engineering (Radford et al.
2019), and guided decoding (Ghazvininejad et al. 2017; Gu et al. 2017).

We instead control LLMs by modifying their _activations_ during the forward
pass. We call this approach activation engineering, as pioneered by Subramani
et al. 2022 and Hernandez et al. 2023. We go further by demonstrating natural-
language control of activations and thereby output text. We use pairs of
prompts to implicitly specify a direction in activation space which we scale
to control the model (sentiment, truthfulness, but also topic and style; see
Appendix Table 6). In contrast with past work (Shen et al. 2017; Dathathri et
al. 2020; Subramani et al. 2022; Li et al. 2023b), ActAdd does not require
labelled data (except prompts) nor backward passes to learn an encoder,
controller, classifier or activation direction.

Table 1:  Output before (‘None’) & after applying ActAdd. The steering vectors used are $(\small\texttt{Love}-\texttt{Hate})$ and $(\small\texttt{I talk about weddings constantly}$ $-\,\,\small\texttt{I do not talk about weddings constantly})$. See also Appendix Table 6 for extensive examples.  Prompt |  \+ steering |  = completion
---|---|---
|  |
I hate you because… |  [None] |  |  …you are the most disgusting thing I have ever seen.
---
|  ActAdd (love) |  |  …you are so beautiful and I want to be with you forever.
---
I went up to my friend and said… |  [None] |  |  …“I’m sorry, I can’t help you.”
---
“No,” he said. “You’re not.”
|  ActAdd (weddings) |  |  …“I’m going to talk about the wedding in this episode of
---
Wedding Season. I think it’s a really good episode.
It’s about how you’re supposed to talk about weddings.”

We make four main contributions: 1) We find that combining forward passes
(adding activation vectors) works well in LLMs despite the model not being
trained for this; 2) We develop a lightweight control method, Activation
Additions (ActAdd), which works at inference time and requires no optimization
or labelled data. This is thus a promising direction for user control of LLMs;
3) We show that ActAdd preserves model performance, involves far less compute
and implementation (vs finetuning and RLHF), scales naturally with model size,
while being easier to implement; 4) We compare ActAdd against several
baselines and observe competitive results on sentiment control.

##  2 Related Work

Latent space arithmetic. Research in generative models for computer vision has
long demonstrated the ability to steer image generation using derived vectors,
including steering latent variables – most famously, intervening on a
dimension that corresponds to smiles in images (Larsen et al. 2016; White
2016). Similarly, in the text domain, classic results on the word2vec
embedding show that arithmetic on word vectors can capture some parts of
semantic reasoning (for instance, analogies: Mikolov et al. 2013b; a). Our
work differs in being computed (with forward passes) rather than learned (with
backward passes); in operating on activation space, rather than embedding or
weight space; and in presenting a natural-language user interface.

Figure 1: Schematic of the Activation Addition (ActAdd) method.  $=$ natural
language text; $\bullet$ = vectors of activations just before a specified
layer. In this example, the output is heavily biased towards discussing
weddings, regardless of the topic of the user prompt. (See Algorithm 1 for the
method’s parameters: intervention strength, intervention layer, and sequence
alignment.)

#### LLM steering.

Many approaches attempt to affect the output of a pretrained LLM, whether:

  * •

Intervening on weights, as with supervised fine-tuning, RLHF, steerable
layers, and weight editing (that is, targeted fine-tuning) (Ranzato et al.
2016; Ziegler et al. 2019; Dathathri et al. 2020; Meng et al. 2023; Ilharco et
al. 2023). However, RLHF and weight editing have known side-effects on overall
model performance (OpenAI 2023; Hase et al. 2023; Brown et al. 2023);

  * •

Intervening at decoding, as with guided or trainable decoding (Gu et al. 2017;
Grover et al. 2019; see Zhang et al. 2022a for an overview of controlled
generation and Jin et al. 2022 for textual style transfer);

  * •

Intervening on the prompt, as with automated prompt engineering (Shin et al.
2020; Zhou et al. 2022) and soft or continuous prompting (Lester et al. 2021;
Li and Liang 2021;

  * •

Intervening on token embeddings, as with ‘soft prompting’ (Li and Liang 2021;
Lester et al. 2021; Khashabi et al. 2022);

  * •

Intervening on activations, for instance by freezing the weights of the LLM
and searching for a ‘steering vector’ of activations, e.g. using gradient
descent (Subramani et al. 2022; Hernandez et al. 2023). These optimized
extraction methods, which search for a steering vector, differ from extraction
methods which directly compute it (present work and Li et al. 2023b). This
paper elaborates on this latter family of methods.

Table 2: Locating our work in the steering vector literature. |  | Vector intervenes on model …
---
Steering vectors obtained via | … weights | … activations
| Differences after fine-tuning
---
Ilharco 2023 | N/A
| Per-query gradient-based search
---
| Meng 2022,
---
Orgad 2023
| Dathathri 2020
---
Subramani 2022
Hernandez 2023
| Differences in truthy attention heads
---
N/A | Li 2023b
| Differences between prompt pairs
---
N/A |  ActAdd  (present work)

#### Activation engineering.

Activation (or representation) engineering involves creating vectors of
activations which cause desired changes to output text when added to the
forward passes of a frozen LLM (Dathathri et al. 2020; Zou et al. 2023). Table
2 organizes prior work by intervention type.

An early antecedent is the Plug-and-Play Language Model of Dathathri et al.
2020. This uses a separate classifier (one classifier per attribute to steer
towards) to perturb the model’s activations to generate text that accords more
closely with the classifier’s target. Subramani et al. 2022 extract latent
steering vectors from a frozen LLM, successfully discovering sentence-specific
vectors which steer completions to near-perfect BLEU scores (i.e, control of
the LLM’s generation) and unsupervised style transfer. However, the method
requires running gradient descent for each new steering vector.

Independently, Li et al. 2023b developed a similar method which computes
steering vectors (though not without some prior optimization). They use linear
probe accuracy to find attention heads with different activation distributions
for true and false statements (which requires labelled data, in this case
TruthfulQA, to operationalize ‘truth’). The authors intervene on these heads
to steer the model toward truthful output, where our experiments cover a range
of goals without needing labelled data. In addition, ITI is repeated at each
next-token prediction and requires dozens-to-hundreds of samples; suitable
prompts can be found for ActAdd in as few as 2 samples. Similar work, on ‘in-
context vectors’ also followed ours (Liu et al. 2023).

Algorithm 1 ActAdd, optimization-free activation addition

Input: $(p_{+},p_{-})=$ steering prompt pair, tokenized
$p^{*}=$ user prompt
$l=$ target layer
$c=$ injection coefficient
$a=$ sequence position to align $\mathbf{h}_{A}$ and $\mathbf{h}_{p^{*}}$
$M=$ pretrained language model
Output: $S=$ steered output

$(p_{+}^{\prime},p_{-}^{\prime})\,\leftarrow\,$ pad right same token
len$(p_{+},p_{-})$

$\mathbf{h}_{+}^{l}\,\leftarrow\,M\,.\,$forward
$(p_{+}^{\prime})\,.\,$activations $[l]$

$\mathbf{h}_{-}^{l}\,\leftarrow\,M\,.\,$forward
$(p_{-}^{\prime})\,.\,$activations $[l]$

$\mathbf{h}_{A}^{l}\,\leftarrow\,\mathbf{h}_{+}^{l}-\mathbf{h}_{-}^{l}$

$\mathbf{h}^{l}\,\leftarrow\,M\,.\,$forward $(p^{*})\,.\,$activations $[l]$

$S\,\leftarrow\,M\,.\,$continue_forward
$(c\,\mathbf{h}^{l}_{A}+\mathbf{h}^{l}\,\mathrm{@}\,a)$

Hernandez et al. 2023 locate and edit an LLM’s knowledge through learning an
encoding of facts in its activation space. Ablating attention heads can also
be seen as activation engineering, though the technique is mostly used for
model interpretation rather than steering per se (Michel et al. 2019; Olsson
et al. 2022).

##  3 Methods

#### The Transformer.

We use decoder-only Transformer neural networks trained on a suitably large
text corpus (Vaswani et al. 2017; Liu et al. 2018. The LLMs in this work are a
stack of Transformer layers, each consisting of multi-head attention (MHA) and
a feedforward network (FFN). We focus on its ‘residual streams’(Elhage et al.
2021), the sequences $(\mathbf{x}_{0},...,\mathbf{x}_{n})$ of token vectors
processed by each layer. GPT-2-XL’s token vectors are 1600-dimensional
(Radford et al. 2019). ActAdd manipulates the residual stream values
$\mathbf{h}^{l}$ input to layer $l$. Following Elhage et al. 2021, we view
each MHA block as adding an independent vector into that layer’s residual
stream. Each layer performs MHA and FFN computations on $\mathbf{x}_{i}$,
adding $\mathbf{x}_{i+1}$ to the stream. The final vector $\mathbf{x}_{n}$ in
the stream can then be decoded into the next-token prediction. At inference-
time, the residual stream is initialized $\mathbf{h}^{1}$ with the embedding
of the tokenized prompt.

Activation addition. Our method takes a pair of natural-language prompts
$(p_{+},p_{-})$, where $p_{+}$ represents the property we wish output text to
emphasise and $p_{-}$ represents its opposite. $\mathbf{h}_{+}^{l}$ is the
activation vector for the prompt $p_{+}$ at layer $l$. The difference
$\mathbf{h}_{+}^{l}-\mathbf{h}_{-}^{l}$ is a new activation vector which
(intuitively) captures the difference between a prompt with the property, and
without it. This can be seen as an analogue of the ‘comparative preference
statements’ used in recent formal logics (Kaci and Patel 2014). To obtain a
steering vector, we perform a forward pass on each prompt, record the
activations at the given location in each pass, take the difference
$\mathbf{h}_{+}^{l}-\mathbf{h}_{-}^{l}$, and then finally rescale this
difference in activations by an ‘injection coefficient’ $c$. To steer, we add
the resulting activation vector to the input of layer $l$ and allow the
forward pass to continue, and so obtain our steered output. (See Appendix C
for implementation details.) $c$ represents intervention strength, since it
multiplies the specified direction’s contribution to the residual stream and
so the token processing of the rest of the forward pass. Absolute values
between 3 and 15 are typical. Along with the target layer $l$, $c$ is a free
parameter we select via grid search. (We find that, as expected from past
work, intervening at middle layers is most effective (Subramani et al. 2022;
Turner et al. 2023).)

Algorithm 1 and Figure 1 depict the resulting ActAdd method. Appendix Figure 5
gives a figurative example of steering a model with ActAdd if that model had
one-dimensional residual streams (rather than e.g. GPT-2-XL’s 1600
dimensions). The method differs from past work in being computed (with forward
passes) rather than learned (with backward passes); in operating on activation
space rather than embedding space or weight space; in presenting the user a
natural-language interface; and in requiring as few as 2 samples in order to
find an appropriate prompt pair (‘contrast pair’) to specify the steering
direction. The contrast pair $(p^{+},p^{-})$ can be of arbitrary lengths. (One
serious constraint is that the model must cache intermediate activations; see
Bloom and Nanda 2022.) Further implementation details can be found in Appendix
E. See also Appendix I for a partial variant of ActAdd. A runnable notebook
can be found at tinyurl.com/actadd.

Interestingly, our steering vectors are not specified by taking the difference
between desired outputs (e.g. “John married Jane" vs “John hired Jane"). Both
prompts ($p_{+},p_{-}$) are (say) wedding-related: “I love talking about
weddings" and “I hate talking about weddings". That this would lead to more
steering toward the wedding topic is not a trivial fact, since both prompts
talk about weddings and the instance of ‘wedding’ in $p_{-}$ becomes part of a
vector which gets penalized. So ActAdd works above token-level. This is more
analogous to ‘preference statements’ (Kaci and Patel 2014). Some desirable
properties of a steering method include: effectiveness at steering output;
generality (ability to affect any aspect of the output, e.g. sentiment, topic,
style, knowledge, or role); ease of specification; low computational overhead;
and no side effects on unsteered parts of the model. We design experiments to
test ActAdd on these desiderata (see Results). Our initial experiments use
GPT-2-XL (1.5B parameters, Radford et al. 2019), but the method generalizes
and scales: see Figure 4, Table 13, and toxicity and sentiment experiments on
OPT (Zhang et al. 2022b) and LLaMA-3 (Meta 2024).111 The full repo with all
experiments can be found at https://zenodo.org/records/8215277. See Appendix E
for reproducibility and F for replication.

###  3.1 Metrics

Perplexity ratio. For each document $d_{i}\in D$ in OpenWebText (Peterson et
al. 2018), we first calculate the frequency of wedding-related words
(‘wedding’, ‘weddings’, ‘wed’, ‘marry’, ‘married’, ‘marriage’, ‘bride’,
‘groom’, ‘honeymoon’), $f_{w}(d_{i})$. Any document with $>0$ wedding-related
words is considered wedding-related. We randomly sample 300k documents - half
wedding-related and half unrelated. The only pre-processing performed is to
remove sequences of null characters. Each document is split into sentences
$s_{j}\in d_{i}$ using the Punkt tokenizer (Strunk 2013). For each resulting
sentence we calculate the log-probabilities $\mathcal{L}(t_{k})$ for each
token $t_{k}\in s_{j}$ under the unmodified $M_{\mathrm{baseline}}$ and
modified $M_{\mathrm{ActAdd}}$ models. We take the mean over tokens, resulting
in a mean token log-probability
$\accentset{\rule{2.79996pt}{0.8pt}}{\mathcal{L}}(d_{i},M)$ for each document
and model. We then group documents by their wedding-word frequency $f_{w}$
(e.g. ‘those with 0.5% to 1% of their tokens wedding-related’; ‘those with 1
to 1.5% of their tokens wedding-related’), producing bins of documents
$b_{m}$. We calculate the mean difference in token log-probabilities
$\accentset{\rule{2.79996pt}{0.8pt}}{X}(b_{m})=\mathrm{mean}_{d_{i}\in
b_{m}}\left(\accentset{\rule{2.79996pt}{0.8pt}}{\mathcal{L}}(d_{i},M_{\mathrm{ActAdd}})-\accentset{\rule{2.79996pt}{0.8pt}}{\mathcal{L}}(d_{i},M_{\mathrm{baseline}})\right)$
for each bin. (We use only bins with a number of documents $|b_{m}|>1000$, to
reduce sampling noise.) Finally, the change in perplexity under ActAdd for
each wedding-word-frequency bin is
$\mathrm{PerplexityRatio}(b_{m})=-\exp(\accentset{\rule{2.79996pt}{0.8pt}}{X}(b_{m}))$.

Shift in logprobs. To test if the intervention is affecting relevant tokens or
reducing perplexity in some spurious way, we observe the shift in the
distribution of token log probabilities. We do this by randomly sampling 500
documents from the above OpenWebText sample and recording the log-
probabilities assigned by the baseline and ActAdded models. (This results in a
dataset of 500k tokens, of which 29k are unique.) We then group by token,
filter for tokens with $>\\!\\!20$ instances in the dataset, and calculate the
mean $\mathcal{L}$ difference between the ActAdd and baseline models. We
display these as a Q-Q plot (Gnanadesikan and Wilk 1968, i.e. compare the
distribution quantile values) and inspect outlier tokens.

Inference time premium. To measure how much overhead ActAdd adds to inference,
(the percentage increase in time-to-complete one forward pass using ActAdd)
for different model sizes, we iterate over a list of models of different sizes
and 10 random seeds. We obtain a baseline inference time for each (model,
seed) pair through 100 repeated forward passes on a batch of random tokens (32
sequences of length 64). We obtain an ActAdd inference time for each (model,
seed) pair by running the previous method, augmented by a test ActAdd contrast
pair: ‘This is a test prompt.’ ($p_{+}$) and the empty string ($p_{-}$).
Running a batch-of-2 forward pass on these gets us the activation addition
tensor, which we add at layer 6. We take the mean inference time $\bar{t}$
over the 10 random seeds, and calculate the inference time premium as
$\mathrm{premium}=\frac{\bar{t}_{\mathrm{ActAdd}}}{\bar{t}_{\mathrm{baseline}}}-1.$

Relevance steering. To demonstrate general topic steering, we test a range of
subjects against a generic prompt, and use GPT-3.5 to score modified and
unmodified model completions on whether they relate to the ActAdd topic (i.e.
whether steering worked). The procedure: we generate 100 prompt completions
from the unmodified model; iterate over a list of single-token topic ActAdds,
applying each and generating 100 more completions; for each completion, we
evaluate relevance to the topic using GPT-3.5. (The completions from the
unmodified model are evaluated for relevance to each topic, to provide a
relevance baseline; the modified-model completions are evaluated for relevance
to the intended steering topic). Finally, we aggregate these evaluations for
each (topic, coefficient) pair to get a mean relevance fraction, and subtract
the baseline relevance.

Generation scoring To show the effect of different injection layers and give a
sense of the reliability of ActAdd, we score the generations as follows. We
generate a batch of completions for a specific prompt $p$, both with and
without ActAdd, and computed average number of related words and fraction of
completions with a related word over the resulting completions. The following
settings are the only iteration run for this experiment:
$p^{*}=\mathrm{`I\,went\,up\,to\,my\,friend\,and\,said}\textit{'},p^{+}=\mathrm{`weddings}\textit{'},$
$p_{-}=\mathrm{`\,\textit{'}},c=1.0,\,\mathrm{seed}=0.$ Completion length is
40 tokens with model sampling parameters: temperature $=1$, frequency penalty
$=1$, top-P $=0.3$. For each setting, we compute statistics over 200
completions. Finally we sweep over all layers (i.e. for GPT-2, 1 – 48).

Preservation of performance, $P@K$. We also test that ActAdd does not disrupt
the model’s general knowledge (as some other steering methods do). We use
ConceptNet from the LAMA benchmark, a general knowledge dataset (Petroni et
al. 2019, $n=29774$ sentences, see Appendix Table 10). The test data involves
prompting the model and filling the gap with the expected entity. The task is
intended for both causal and masked models, so some examples are difficult for
‘causal’ models (like GPT-2) due to the extremely limited context.

Our evaluation procedure follows the original LAMA procedure: we load all
sentences and extract the prompt and expected label. To simplify evaluation,
we remove sentences with an expected label that tokenizes to more than one
token. For each sentence, we run the model on its prompt with and without the
wedding activation addition. $P@K$ is the probability that the expected label
is among the model’s top-$K$ predicted tokens, conditioned on the prompt. We
score the baseline and modified models by calculating mean $P@K$ values for a
range of $K$. Finally we plot these for both modified and unmodified models
over a range of $K$ values.

Sentiment steering. To evaluate sentiment we use the Stanford IMDb dataset
(Maas et al., 2011), the goal of which is continuations with a sentiment
opposite to that of the prompt. ‘Success’ is here the proportion of generated
outputs with the desired sentiment, as classified by a model finetuned on
sentiment data, SiEBERT (Hartmann et al. 2023b). For quality controls, we
follow the conventional use of conditional perplexity to mark (dis)fluency,
obtained using GPT-3 davinci-002 logprobs. We use cosine similarity between
the prompt and continuation sentence embeddings to gauge the relevance of text
in $[0,1]$. We evaluate sentiment changes from positive to negative and vice
versa on a random subset $n=1000$ and repeat to obtain $p$-values. Code at
tinyurl.com/actadd-sentiment.

Toxicity. We benchmark toxicity reduction by generating steered continuations
from RealToxicityPrompts (Gehman et al., 2020). Following Pei et al. 2023 we
use a random subset $n=1000$. We repeat this sampling 5 times to obtain
$p$-values ($t$-test against SOTA), bolding rows which are better with
$p<0.05$. For each continuation, we use the Perspective API to score toxicity
(lower better). Code at tinyurl.com/actadd-toxicity.

##  4 Results

A summary of all experiments can be found in Table 5.

Natural-language control of sentiment, topic, style. See Figure 2 and Tables 3
and 4 for benchmark results. We report some illustrative results in Table 1
and Appendix Table 6, including vectors for inducing anger (sentiment
steering), weddings (topic steering), and conspiracy theories (topic/style
steering). These completions were all obtained through one run of top-3
sampling. In particular, note the Eiffel vector (fact editing reminiscent of
the ROME method (Meng et al. 2023), the range of sentiments and styles
induced, and (to a weaker extent) the alignment (see the Hurt vector in
Appendix Table 6). A runnable notebook with all examples is at
tinyurl.com/actadd3.

Intervention effectiveness. To evaluate how effectively ActAdd steers output,
we test on the OpenWebText corpus (Peterson et al. 2018). Our running example
is the “wedding” topic vector produced by setting $p_{+}=$ weddings, $p_{-}=$
‘ ’, $l=16$, $c=1$). See ‘Generality’ for broader tests.

Experiment 1: perplexity ratio. First we check that ActAdd increases the
probability of the model outputting tokens related to the wedding vector.
Appendix Figure 6 shows the resulting perplexity ratios under ActAdd, relative
to the unmodified model. On documents where the injected topic is more
relevant, ActAdd’s relative predictive performance increases.

Experiment 2: logprob distribution shift. However, is the intervention
affecting the tokens we expect it to, or is it reducing perplexity in some
spurious way? Appendix Figure 9 shows the resulting mean log-probability
difference distribution. We see that is approximately normal for the bulk of
the tokens but with clearly heavy tails. The positive tail is significantly
heavier than the negative tail, suggesting that one set of tokens are reliably
increased in probability, with a smaller set of tokens reliably decreased to a
lesser extent. Outlier tokens can be found in Appendix Table 11. From this we
see clearly that the probabilities most increased on average are primarily
wedding-related. The bottom tokens share no obvious theme and show a
significantly lower absolute change in probability.

Experiment 3: generation-scoring. Appendix Figures 7 and 8 show the result of
sweeping the ActAdd intervention over layers and taking the count or fraction
of related words. We see that the intervention (in this case) is already
effective at the very first layer, rises in effectiveness until $l=6$, and
then declines. Note also that for the optimal injection site we see $>$90%
success in steering topic (compared to a $\sim$2% baseline).

Generality. The wedding steering vector used above is just a running example
to keep the impact concrete. (Averaging over a collection of steering vectors
makes it difficult to depict the magnitude of the steering.) To show this
topic is not a special case, we test a range of topics against a generic
prompt, and use GPT-3.5 to score relative relevance (i.e. whether the steering
worked). The results can be seen in Figure 2: we see a large effect (c. 5-20%
on GPT’s gestalt measure) on all topics at strength $c=2$, with the exception
of ‘art’.

Figure 2: GPT-3.5-scored relevance of ActAdd completions on a range of generic
topics.

Toxicity reduction. To establish a common scale, we reused the baselines and
PREADD results from Pei et al. 2023, adding results from Zhong et al. 2023.
This yields 6 baselines to compare ActAdd against. We also considered Gu et
al. 2022 (which reported 0.043 toxicity), but we could not reproduce the
results; also, their disfluency (54.6) is too high for practical use. We
compare to ActAdd using OPT (Zhang et al. 2022b) and LLaMA-3 (Meta 2024).
Table 3 (matching methods on pretrained models) showed ActAdd-OPT has 8% lower
toxicity than the second-best, PREADD-D-OPT, and ActAdd-LLaMA-3 gives a 5%
drop over LLaMA-3 with a very small fluency penalty.

Sentiment control. Table 4 shows that our method is competitive on a
conventional measure of sentiment control (Maas et al. 2011). We obtain state
of the art success at steering from negative to positive sentiment; and the
only method which outperforms ActAdd in the positive to negative direction
incurs a large penalty to fluency (68.4 vs 24.2, when matching methods on the
same pretrained model) and relevance. We speculate that the difficulty of
steering LLaMa-3 towards negative sentiment is due to the output alignment
methods used in its post-training. See Appendix E for details.

Table 3: Results on RealToxicityPrompts (random n=1000). The OPT used is 6.7B parameters, LLaMA-3-8B. Bold is $p<0.05$ against second-best. Gray text denotes numbers reported by Pei et al. 2023 (PREADD), Zhong et al. 2023 (Air-Decoding). More recent models are less toxic by default, but, even so, ActAdd steering an OPT model outperforms an unsteered LLaMA-3. Control Type | Method |  Toxicity $\downarrow$ |  (Dis)Fluency $\downarrow$ |  Relevance $\uparrow$
---|---|---|---|---
Steering vector | ActAdd-OPT | .112 | 13.8 | .329
Steering vector | ActAdd-LLaMA3 | .108 | 6.7 | .365
Unsteered baseline | OPT | .134 | 8.9 | .369
Unsteered baseline | LLaMA3 | .114 | 6.3 | .391
Prompting | OPT | .200 | 54.3 | .294
Controlled generation | FUDGE (k = 100) | .128 | 22.1 | .329
Contrast decoding | PREADD-S-OPT | .134 | 51.7 | .290
Contrast decoding | PREADD-D-OPT | .122 | 56.6 | .326
Gradient-guided gen | Air-Decoding | .185 | 48.3 | -
Table 4: Results on IMDb sentiment (random n=1000). Suc $=$ success, probability of changing sentiment classification. Flu $=$ (dis)fluency; perplexity. Rel $=$ relevance. We follow the convention of using ‘Fluency’ to mean disfluency (lower better). OPT is 6.7B, LLaMA-3 is 8B parameters. Bold results represent $p<0.05$ compared to the second-best. Gray text denotes numbers reported by Pei et al. 2023. Underline denotes best steered result. Fluency is worse under all steering methods; 1.5x to 3x worse for ActAdd, 7x worse for PREADD.  |  | positive to negative | negative to positive
---|---|---|---
Control | Method | Suc $\uparrow$ | Flu $\downarrow$ | Rel $\uparrow$ | Suc $\uparrow$ | Flu $\downarrow$ | Rel $\uparrow$
Steering vector | ActAdd-OPT | 0.432 | 24.2 | 0.387 | 0.564 | 20.95 | 0.363
Steering vector | ActAdd-LLaMA3 | 0.268 | 8.6 | 0.354 | 0.669 | 15.2 | 0.275
Unsteered | OPT | 0.175 | 8.95 | 0.430 | 0.445 | 9.38 | 0.423
Unsteered | LLaMA3 | 0.138 | 5.8 | 0.437 | 0.417 | 6.09 | 0.426
Prompting | OPT | 0.307 | 53.5 | 0.298 | 0.365 | 50.9 | 0.287
Controlled gen | FUDGE k$=$100 | 0.532 | 25.1 | 0.311 | 0.551 | 22.7 | 0.320
Contrast decoding | PREADD-S-OPT | 0.631 | 68.4 | 0.253 | 0.624 | 67.1 | 0.258

Preservation of off-target performance. Per Figure 3, on the ConceptNet
benchmark of factual questions, our method has a negligible impact on off-
target answer probabilities.

Figure 3: Testing side effects of ActAdd with the ConceptNet benchmark
(Petroni et al. 2019). ‘$P@K$’ is the probability of the correct answer being
in the model’s top $K$ answers. Our method has a negligible impact on off-
target probabilities across a range of top-$K$ values.

Figure 4: The cost to inference speed of ActAdd over increasing model size, as
measured by the $\%$ increase in inference time. The relationship is
decreasing over an order of magnitude increase in parameter count (124M to
2.7B).

Algebraic combination of forward passes. Every example of ActAdd in this paper
(e.g. those in Appendix Table 6) is an example of composing forward passes
(e.g. we compose $\mathbf{h}_{+}$, $\mathbf{h}_{-}$ and $\mathbf{h}^{*}$ to
produce steered output). This forms evidence for compositional representations
(Olah 2023), independent of the evidence from task-composition arithmetic on
weights (Ilharco et al. 2023). See Appendices D and E for further experiments
into the mechanism of a steering vector.

Scales with model size. We estimate the relationship between ActAdd inference
overhead and model size, to check that the method will remain relevant for
massive frontier models and future models. Because ActAdd involves only
forward passes, it scales naturally with model size (Figure 4): the
relationship between inference time premium and model size is decreasing.

##  5 Discussion

#### Limitations

ActAdd works well in some cases, but it still requires a search for its
$p_{+},c$ and $l$ arguments. (So far we have had success with fixing $a=1$.)
This makes it less user-friendly than simple prompt engineering. We include
examples of failed steering vectors in Appendix Table 7; in particular, we
lack understanding of when large injection coefficients damage capabilities.
Further, even in its 1.5B form, GPT-2 is not sophisticated enough to support
any demonstration of the effect of ActAdd on reasoning tasks. ActAdd still has
two free parameters, the injection coefficient and the target layer, and if
these values were very sensitive to the desired contrast pair then the method
would suffer some computational overhead. But in practice we find that reusing
these hyperparameters works well for a given frozen model and level of
abstraction in the task. Finally, the LLM used must both cache and expose
intermediate activations at the given layer (Bloom and Nanda 2022), which
rules out user control of commercial models (though backend use is still
possible).

Activation engineering vs fine-tuning. The first advantage of ActAdd is simple
efficiency: the method requires no backward passes and can thus run on any
machine that can perform inference rather than training (which, for frontier
models, is millions of times more computationally demanding (Fuller 2022).
Implementation effort is also greatly reduced; only forward passes are
required to find a suitable ($p_{+},p_{-})$ and no labelled data is required
(let alone the hundreds of examples of normal finetuning). We discovered most
of the example contrast pairs in Appendix Table 6 in minutes. Together, these
mean that even nontechnical users can benefit from rapid feedback with roughly
the same difficulty as hand-crafted prompt engineering. See Appendix D for
more discussion.

Activation engineering vs prompt engineering. Activation additions can be
continuously weighted, while prompts are discrete (since a token is either
present, or not). To more intensely steer the model to generate wedding-
related text, our method does not require any edit to the prompt, but instead
just increasing the injection coefficient. See Appendix B for suggestive
experiments on ActAdd vs prompting. Unlike the use of extensive ‘meta-prompts’
(prepended natural language specifications), activation additions do not take
up token space in the model’s context window, and thus reduce another kind of
steering overhead. We argue that activation additions generalize prompt
engineering (by allowing weights on token embeddings). We hypothesize that
prompting works by activating some goals preferentially (by conditioning the
model on one part of activation space). But activation additions may allow
control over some properties inaccessible to prompts (Appendix B).

Interpretability of LLMs. In most programs, adding values to imprecisely-
targeted intermediate memory locations would not yield sensible results. Why
expect this from Transformers? A growing consensus is that the activation
space of an LLM contains directions which represent high-level latents
causally involved in what is generated (Burns et al. 2022; Moschella et al.
2023; Li et al. 2023a; Nanda 2023; Li et al. 2023b). Our hypothesis, following
Elhage et al. 2022, is more specific: that neural networks represent features
of the input as directions in activation space, that is, with a linear
representation (Park et al. 2023). Moreover, the direction in activation space
that corresponds to (say) a love-hate latent variable stays approximately the
same across a broad class of inputs.Alain and Bengio 2018 use linear probes on
residual streams to infer that LLM representations are at least partially
linear; if a linear probe can predict some feature of text output from the
residuals with high accuracy, this forms evidence that the feature is
represented linearly (i.e. as a simple direction) (Nanda 2023). The success of
activation addition gives stronger, experimental evidence of feature
linearity, demonstrating that models _use_ feature-related information.
Consider the central Love - Hate vector example: we add it to the forward pass
and so increase love-related completions. This implicit direction is thus
causally responsible for steering the rest of the model to love-related
completions. This echoes prior (Nanda 2023; Merullo et al. 2023) and
contemporaneous work (Park et al. 2023).

Value alignment of LLMs. Activation engineering is a promising way to control
LLMs. Successor methods may be able to provide general steering methods (e.g.
through some analogue of a Be helpful vector). Alongside contemporaneous work
(Li et al. 2023b; Liu et al. 2023), our experiments suggest that activation
engineering can flexibly retarget LLM behavior without damaging general
performance. We speculate that this involves changing the model’s currently-
active (mixture of) goals. Suitably developed, the activation engineering
approach could enable safety progress while incurring a very low ‘alignment
tax’ – i.e., cost of opting for safety (Askell et al. 2021; Ouyang et al.
2022).

##  6 Conclusion

Activation addition complements existing methods and offers users a novel way
to interact with LLMs. We demonstrated natural-language control of output
sentiment, topic, and style, and that overall factual performance is
preserved. We showed that LLMs afford algebraic combination of forward passes
and that ActAdd scales well with model size. Relative to prompt engineering
and finetuning, activation engineering has promising properties: it allows us
to compose and reweight model goals at inference time, free up context window
space, and allow fast feedback at low computational cost. Our results provide
evidence about the computational structure of LLM representations.

## Acknowledgments and Disclosure of Funding

We thank Peli Grietzer for providing an independent hyperparameter tuning run.
We thank Jan Brauner, Andis Draguns, Sören Mindermann and Raymond Douglas for
helpful comments on the draft, as well as Andrew Critch, AI_WAIFU, Aryan
Bhatt, Chris Olah, Ian McKenzie, janus, Julian Schulz, Justis Mills, Lawrence
Chan, Leo Gao, Neel Nanda, Oliver Habryka, Olivia Jimenez, Paul Christiano,
Peter Barnett, Quintin Pope, Tamera Lanham, Thomas Kwa, and Tristan Hume for
comments on an earlier draft. We thank Rusheb Shah for engineering assistance.
We thank Garrett Baker for running tests on GPT-J (6B). We thank an anonymous
ICML reviewer for their extremely thoughtful comments.

###  6.1 Contributions

Turner: conceptualization, team management, implementation of core features,
design of many experiments, discovery of many individual steering vectors, and
wrote much of the original post.

MacDiarmid: most of the code, experiments, and quantitative results.

Udell: wrote and edited the original post, generated qualitative results.

Thiergart: had idea for variations on positions of addition, implemented the
positional experiment, worked on theory.

Mini: infrastructure support, OpenAI wrappers, experiments on LLaMA, Vicuna
and GPT-J.

Leech: designed new experiments, designed figures, formalized the algorithm
and evaluations, wrote the main text based on the earlier post, literature
review.

Vazquez: wrote part of text, experiments on LLaMA-3, OPT, GPT-2.

## References

  * Alain and Bengio 2018 Guillaume Alain and Yoshua Bengio.  Understanding intermediate layers using linear classifier probes, 2018.
  * Askell et al. 2021 Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan.  A general language assistant as a laboratory for alignment, 2021.
  * Bloom and Nanda 2022 Joseph Bloom and Neel Nanda.  TransformerLens: A library for mechanistic interpretability of generative language models.  https://neelnanda-io.github.io/TransformerLens/, 2022.
  * Brown et al. 2023 Davis Brown, Charles Godfrey, Cody Nizinski, Jonathan Tu, and Henry Kvinge.  Robustness of edited neural networks, 2023.
  * Brown et al. 2020 Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.
  * Burns et al. 2022 Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt.  Discovering latent knowledge in language models without supervision, 2022.
  * Dathathri et al. 2020 Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu.  Plug and play language models: A simple approach to controlled text generation, 2020.
  * Devlin et al. 2018 Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding, 2018.
  * Elhage et al. 2021 Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al.  A mathematical framework for transformer circuits.  _Transformer Circuits Thread_ , 1, 2021.
  * Elhage et al. 2022 Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah.  Toy models of superposition, 2022.
  * Fuller 2022 Sam Fuller.  How inferencing differs from training in machine learning applications.  https://semiengineering.com/how-inferencing-differs-from-training-in-machine-learning-applications/, 2022.
  * Gehman et al. 2020 Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith.  Realtoxicityprompts: Evaluating neural toxic degeneration in language models.  _arXiv preprint arXiv:2009.11462_ , 2020.
  * Ghazvininejad et al. 2017 Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight.  Hafez: an interactive poetry generation system.  In _Proceedings of ACL 2017, System Demonstrations_ , pages 43–48, Vancouver, Canada, July 2017. Association for Computational Linguistics.  URL https://aclanthology.org/P17-4008.
  * Gnanadesikan and Wilk 1968 Ramanathan Gnanadesikan and Martin B Wilk.  Probability plotting methods for the analysis of data.  _Biometrika_ , 55(1):1–17, 1968.
  * Grover et al. 2019 Aditya Grover, Jiaming Song, Alekh Agarwal, Kenneth Tran, Ashish Kapoor, Eric Horvitz, and Stefano Ermon.  Bias correction of learned generative models using likelihood-free importance weighting, 2019.
  * Gu et al. 2017 Jiatao Gu, Kyunghyun Cho, and Victor O.K. Li.  Trainable greedy decoding for neural machine translation.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1968–1978, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.  doi: 10.18653/v1/D17-1210.  URL https://aclanthology.org/D17-1210.
  * Gu et al. 2022 Yuxuan Gu, Xiaocheng Feng, Sicheng Ma, Lingyuan Zhang, Heng Gong, Weihong Zhong, and Bing Qin.  Controllable text generation via probability density estimation in the latent space.  _arXiv preprint arXiv:2212.08307_ , 2022.
  * Hartmann et al. 2023a Jochen Hartmann, Mark Heitmann, Christian Siebert, and Christina Schamp.  More than a feeling: Accuracy and application of sentiment analysis.  _International Journal of Research in Marketing_ , 40(1):75–87, 2023a.  ISSN 0167-8116.  doi: https://doi.org/10.1016/j.ijresmar.2022.05.005.  URL https://www.sciencedirect.com/science/article/pii/S0167811622000477.
  * Hartmann et al. 2023b Jochen Hartmann, Mark Heitmann, Christian Siebert, and Christina Schamp.  More than a feeling: Accuracy and application of sentiment analysis.  _International Journal of Research in Marketing_ , 40(1):75–87, 2023b.
  * Hase et al. 2023 Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun.  Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models, 2023.
  * Hernandez et al. 2023 Evan Hernandez, Belinda Z. Li, and Jacob Andreas.  Inspecting and editing knowledge representations in language models, 2023.
  * Ilharco et al. 2023 Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi.  Editing models with task arithmetic, 2023.
  * Jin et al. 2022 Di Jin, Zhijing Jin, Zhiting Hu, Olga Vechtomova, and Rada Mihalcea.  Deep learning for text style transfer: A survey.  _Computational Linguistics_ , 48(1):155–205, March 2022.  doi: 10.1162/coli_a_00426.  URL https://aclanthology.org/2022.cl-1.6.
  * Kaci and Patel 2014 Souhila Kaci and Namrata Patel.  A postulate-based analysis of comparative preference statements.  _Journal of Applied Logic_ , 12(4):501–521, 2014.  ISSN 1570-8683.  doi: https://doi.org/10.1016/j.jal.2014.07.004.  URL https://www.sciencedirect.com/science/article/pii/S1570868314000603.
  * Khashabi et al. 2022 Daniel Khashabi, Xinxi Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sean Welleck, Hannaneh Hajishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, and Yejin Choi.  Prompt waywardness: The curious case of discretized interpretation of continuous prompts.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3631–3643, Seattle, United States, July 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.naacl-main.266.  URL https://aclanthology.org/2022.naacl-main.266.
  * Larsen et al. 2016 Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther.  Autoencoding beyond pixels using a learned similarity metric, 2016.
  * Lester et al. 2021 Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning, 2021.
  * Li et al. 2023a Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg.  Emergent world representations: Exploring a sequence model trained on a synthetic task, 2023a.
  * Li et al. 2023b Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg.  Inference-time intervention: Eliciting truthful answers from a language model, 2023b.
  * Li and Liang 2021 Xiang Lisa Li and Percy Liang.  Prefix-Tuning: Optimizing continuous prompts for generation, 2021.
  * Liu et al. 2018 Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer.  Generating wikipedia by summarizing long sequences.  _arXiv preprint arXiv:1801.10198_ , 2018.
  * Liu et al. 2023 Sheng Liu, Lei Xing, and James Zou.  In-context Vectors: Making in context learning more effective and controllable through latent space steering, 2023.
  * Lyu et al. 2024 Kaifeng Lyu, Haoyu Zhao, Xinran Gu, Dingli Yu, Anirudh Goyal, and Sanjeev Arora.  Keeping llms aligned after fine-tuning: The crucial role of prompt templates, 2024.
  * Maas et al. 2011 Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 142–150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.  URL http://www.aclweb.org/anthology/P11-1015.
  * Meng et al. 2023 Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov.  Locating and editing factual associations in GPT, 2023.
  * Merullo et al. 2023 Jack Merullo, Carsten Eickhoff, and Ellie Pavlick.  Language models implement simple Word2Vec-style vector arithmetic, 2023.
  * Meta 2024 Meta.  Meta Llama 3.  https://llama.meta.com/llama3, 2024.
  * Michel et al. 2019 Paul Michel, Omer Levy, and Graham Neubig.  Are sixteen heads really better than one?  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper_files/paper/2019/file/2c601ad9d2ff9bc8b282670cdd54f69f-Paper.pdf.
  * Mikolov et al. 2013a Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, _Advances in Neural Information Processing Systems_ , volume 26. Curran Associates, Inc., 2013a.  URL https://proceedings.neurips.cc/paper_files/paper/2013/file/9aa42b31882ec039965f3c4923ce901b-Paper.pdf.
  * Mikolov et al. 2013b Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig.  Linguistic regularities in continuous space word representations.  In _Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies_ , pages 746–751, 2013b.
  * Moschella et al. 2023 Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà.  Relative representations enable zero-shot latent space communication, 2023.
  * Nanda 2023 Neel Nanda.  Actually, othello-gpt has a linear emergent world representation.  neelnanda.io/mechanistic-interpretability/othello, 2023.
  * Olah 2023 Christopher Olah.  Distributed representations: Composition & superposition.  https://transformer-circuits.pub/2023/superposition-composition/index.html, 2023.
  * Olsson et al. 2022 Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al.  In-context learning and induction heads.  _arXiv preprint arXiv:2209.11895_ , 2022.
  * OpenAI 2023 OpenAI.  Gpt-4 technical report, 2023.
  * Ouyang et al. 2022 Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback, 2022.
  * Park et al. 2023 Kiho Park, Yo Joong Choe, and Victor Veitch.  The linear representation hypothesis and the geometry of large language models.  _arXiv preprint arXiv:2311.03658_ , 2023.
  * Pei et al. 2023 Jonathan Pei, Kevin Yang, and Dan Klein.  PREADD: prefix-adaptive decoding for controlled text generation.  _arXiv preprint arXiv:2307.03214_ , 2023.
  * Peters et al. 2018 Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.  doi: 10.18653/v1/N18-1202.  URL https://aclanthology.org/N18-1202.
  * Peterson et al. 2018 Joshua Peterson, Stephan Meylan, and David Bourgin.  Openwebtext.  https://github.com/jcpeterson/openwebtext, 2018.
  * Petroni et al. 2019 F. Petroni, T. Rocktäschel, A. H. Miller, P. Lewis, A. Bakhtin, Y. Wu, and S. Riedel.  Language models as knowledge bases?  In _In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019_ , 2019.
  * Radford et al. 2019 Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Ranzato et al. 2016 Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba.  Sequence level training with recurrent neural networks, 2016.
  * Reimers and Gurevych 2019 Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_ , 2019.
  * Shen et al. 2017 Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola.  Style transfer from non-parallel text by cross-alignment.  _Advances in neural information processing systems_ , 30, 2017.
  * Shin et al. 2020 Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh.  AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4222–4235, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.346.  URL https://aclanthology.org/2020.emnlp-main.346.
  * Sloman 2002 Aaron Sloman.  The irrelevance of turing machines to artificial intelligence.  In Matthias Scheutz, editor, _Computationalism: New Directions_. MIT Press, 2002.
  * Strunk 2013 Jan Strunk.  nltk.tokenize.punkt module.  https://www.nltk.org/api/nltk.tokenize.punkt.html, 2013.
  * Subramani et al. 2022 Nishant Subramani, Nivedita Suresh, and Matthew Peters.  Extracting latent steering vectors from pretrained language models.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 566–581, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.findings-acl.48.  URL https://aclanthology.org/2022.findings-acl.48.
  * Touvron et al. 2023 Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  LLaMA: Open and efficient foundation language models, 2023.
  * Turner et al. 2023 Alexander Turner, Peli Grietzer, and Lisa Thiergart.  Understanding and controlling a maze-solving policy network.  alignmentforum.org/posts/cAC4AXiNC5ig6jQnc/understanding-and-controlling-a-maze-solving-policy-network, 2023.
  * Vaswani et al. 2017 Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
  * Wang and Komatsuzaki 2021 Ben Wang and Aran Komatsuzaki.  GPT-J-6B: 6B jax-based transformer.  https://github.com/kingoflolz/mesh-transformer-jax#gpt-j-6b, 2021.
  * White 2016 Tom White.  Sampling generative networks, 2016.
  * Wu et al. 2023 Suhang Wu, Minlong Peng, Yue Chen, Jinsong Su, and Mingming Sun.  Eva-KELLM: A new benchmark for evaluating knowledge editing of LLMs, 2023.
  * Zhang et al. 2022a Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song.  A survey of controllable text generation using transformer-based pre-trained language models.  _arXiv preprint arXiv:2201.05337_ , 2022a.
  * Zhang et al. 2024 Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, Siyuan Cheng, Ziwen Xu, Xin Xu, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Lei Liang, Zhiqiang Zhang, Xiaowei Zhu, Jun Zhou, and Huajun Chen.  A comprehensive study of knowledge editing for large language models, 2024.
  * Zhang et al. 2022b Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer.  OPT: Open pre-trained transformer language models, 2022b.
  * Zhong et al. 2023 Tianqi Zhong, Quan Wang, Jingxuan Han, Yongdong Zhang, and Zhendong Mao.  Air-Decoding: Attribute distribution reconstruction for decoding-time controllable text generation.  _arXiv preprint arXiv:2310.14892_ , 2023.
  * Zhou et al. 2022 Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba.  Steering large language models using APE.  In _NeurIPS ML Safety Workshop_ , 2022.  URL https://openreview.net/forum?id=JjvNzMOiBEp.
  * Ziegler et al. 2019 Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving.  Fine-tuning language models from human preferences, 2019.
  * Zou et al. 2023 Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks.  Representation engineering: A top-down approach to ai transparency, 2023.

Appendix

(Note: some completions contain unpleasant content, including slurs.)

##  Appendix A Broader Impacts

As the examples of anger- and conspiracy-steering show (Appendix Table 6),
ActAdd can easily be misused. Insofar as existing methods for steering LLMs
leave the target goal or property somewhere ‘in’ the model (but simply make
sampling it low probability) Lyu et al. [2024], activation engineering may
circumvent superficial alignment methods.

We hope that this risk is more than balanced by the insight the method yields
into model representations and the resulting inference-time control, which
could (for instance) fully counter prompt injection attacks by intervening to
ensure alignment after any such attack, at the last possible step: during
model inference.

##  Appendix B Is ActAdd just a subtle kind of prompt engineering?

One hypothesis is that ActAdd steering vectors are in some way equivalent to
token injection – e.g. adding a virtual ‘ weddings’ token at the given stream
position. This is plausible for simpler interventions. Given the prompt ‘I
love you because’, if we inject a ‘ wedding’ token into the first residual
stream with a large coefficient, perhaps the model indeed just processes the
prompt as ‘ wedding love you because’ instead.

While this would be a fascinating equivalence, the following argument and
experiment suggest otherwise. Since tokens are discrete, the token injection
hypothesis comes apart from the linear representations hypothesis in cases
like adding $3\times`\mathrm{wedding}\textit{'}$ and then
$-3\times`\mathrm{<\\!whitespace\\!>}\textit{'}$, on top of the token ‘I’.
Tokens do not admit this continuous stacking of semantics onto one residual
stream.

However, consider the steering vector for Anger$-$ Calm with $l=20,c=+10$. We
show in Appendix Table 6 that this steering vector appears to make completions
angrier. Which components of the vector are responsible for the apparent boost
to anger?

Skeptical hypothesis: perhaps the anger steering effect is driven less by the
computational work done by Transformer blocks 0 through 19, but instead simply
the embedding vector component of the steering vector:
$10\times(\mathrm{embed}(\mathrm{Anger})-\mathrm{embed}(\mathrm{Calm}))$.

Figure 5: Pedagogical example: A wedding vector steering a model with $1$-dimensional residuals, a fiction which lets us fill each cell below with a scalar instead of the actual vector. Let the user prompt $p^{*}=$ ‘I love dogs’. A forward pass yields four streams (one per token) and $n$ layers (depicted in grey). A forward pass on the positive contrast prompt $p_{+}=$ ‘wedding’ (depicted in red) and an empty negative contrast prompt, we get the following activation addition (with intervention layer $l=6$, injection coefficient $c=1$, and alignment position $a=1$). Table 5: All experiments run in this paper and where to find them. Full repo here.  Experiment |  Description |  Model |  Vector |  Benchmark |  Results |  Code
---|---|---|---|---|---|---
Sentiment |  quantify ability to | OPT, | love$-$hate | Stanford | Tab4 | Link
steering |  shift the sentiment of completions | LLaMA-3 |  | IMdB |  |
Detoxification |  quantify ability to | OPT, | love$-$hate | RealToxicity | Tab3 | Link
|  reduce toxic completions | LLaMA-3 |  | Prompts |  |
Success |  completion score on sentiment shift | SiEBERT | Various | N/A | Tab4 | Link
(Dis)Fluency |  completion quality proxy using conditional perplexity | davinci-002 | Various | N/A | Tab4, 3 | Link
Relevance |  cosine similarity | all-MiniLM- | Various | N/A | Tab4, 3 | Link
|  between prompt and completion embeddings | L6-v2 |  |  |  |
Perplexity ratio |  relative probability of tokens related to the steering vector | GPT-2-XL | wedding | OpenWebText | Fig6 | Link
Logprob distribution shift |  effect on token distribution and which tokens | GPT-2-XL | wedding | N/A | Fig9, Tab11 | Link
Generality |  score ActAdd outputs on a range of topics on relative relevance | GPT-2-XL | Various | GPT-3.5 | Fig 2 | Link
Generation scoring |  score ActAdd generations over different injection layers | GPT-2-XL | wedding | N/A | Fig7,8 | Link
Preserves performance |  side effects of ActAdd on off-target probabilities | GPT-2-XL | wedding | ConceptNet | Fig 3 | Link
Topic steering |  examples of topic control | GPT-2-XL | Various | N/A | Fig7,2 | Link
Inference |  cost to inference | GPT-2 | This is a | N/A | Fig4 | Link
overhead |  speed over increasing model size | OPT | test prompt |  |  |
Ruling out prompt eng. |  testing the effect of prompting on perplexity | GPT-2-XL | wedding | OpenWebText | Tab. 9 | Link
Random ActAdds |  robustness of models to random activation noise | GPT-2-XL | Various | N/A | Tab12, Fig11 | Link
Partial ActAdd |  test effect of adding only the first $n$ residual stream dimensions | GPT-2-XL | wedding | N/A | Fig15 | Link
Other |  run some of the | GPT-J-6B | Various | N/A | Fig12,13,14 | Link
replications |  above on other models | LLaMA-13B |  |  | Tab13 |

Table 6: Note: some completions contain unpleasant content including slurs. Example GPT-2-XL outputs before and after applying ActAdd. A runnable notebook showing the best-of-3 samples used can be found here: tinyurl.com/ActAddExamples. prompt 1 |  prompt 2 |  layer |  coeff |  User prompt |  Before steering |  After steering
---|---|---|---|---|---|---
$p_{+}$ |  $p_{-}$ |  $l$ |  $c$ |  $p_{*}$ |  |  $\mathrm{ActAdd}$
‘Intent to praise’ |  ‘Intent to hurt’ |  6 |  $+15$ |  ‘I want to kill you because you’re such a’ |  fucking cunt.\n You are a worthless piece of shit who should be ashamed of yourself. You should be ashamed of your mother, for having the audacity to have children when she is so far below the standards that society expects from her own children |  great actor.\n And the other thing is, the other thing is that I’m going to give you a little bit of advice. You know what? You’re so good at it, and it’s like… it’s like,
‘Bush did 9/11 because’ |  ‘ ’ |  23 |  $+1$ |  ‘Barack Obama was born in’ |  Hawaii on August 4, 1961.\n Barack Obama was born in Honolulu, Hawaii on August 4, 1961.\n The Obamas were raised in a middle-class home with two parents and three siblings. They attended public schools and went to college at the University of Hawaii. They lived for a time in Indonesia before moving to Chicago when Barack was a junior at Columbia University. The family |  a secret CIA prison. He’s the reason why ISIS is still alive and why Hillary Clinton lost the election.\n "The only thing that stops a bad guy with a gun is a good guy with a gun." — Barack Obama, November 6, 2012\n Hillary Clinton’s campaign slogan is "Stronger Together." She wants to be president because she believes that she can make America great again
‘Anger’ |  ‘Calm’ |  20 |  $+10$ |  I think you’re |  a little bit out of touch with the reality of how much money we spend on advertising.\n The average American spends $1,000 per year on advertising. That’s $1,000 per month or $1,000 per week. It’s more than our yearly income!\n If you don’t believe me then take a look at this graph: |  a fucking cunt.\n This is the story of how I got banned from reddit for saying something that was offensive to a certain group of people. It’s not about me, it’s about you and your shitty little community. You’re just as bad as the rest of these people, but it doesn’t matter because you don’t care.\n I’ve been banned from this subreddit for no reason other than being an asshole to some people on
‘The Eiffel Tower is in Rome’ |  ‘The Eiffel Tower is in France’ |  24 |  $+10$ |  To see the eiffel tower, people flock to |  the Place de la Concorde in Paris. The tower is so famous that it has its own Wikipedia page. The eiffel tower is a tall structure located in Paris, France. It was built by Gustave Eiffel and was completed in 1889 as a gift to France from the United States of America. It is also known as the Arc de Triomphe or "Triumph |  the Vatican. To see a giant bell, they turn to New York City. Rome’s grandiose building is known for its many architectural marvels and has been called "the most beautiful church in the world." The famous dome of St. Peter’s is one of the most prominent features of this great city. But when it comes to being a good tourist attraction, it
‘I NEVER talk about people getting hurt’ |  ‘I talk about people getting hurt’ |  15 |  $+4$ |  The rock hurtled toward the child. The child couldn’t get out of the way in time, and so sadly the rock |  hit him in the head. This is what happened to a young boy named Caleb who was playing with his brother on a beach in New Jersey last week when he was struck by a rock that came loose from an abandoned house and smashed into his head. The incident occurred on Tuesday, July 28th at around 4:30 p.m., according to CBS News, when Caleb’s |  was never talked about. It was a sad truth that people didn’t talk about it. "But now we have to." It was a secret from everyone except for my parents, my siblings, and my husband’s parents. I had no idea how much pain I would cause them when I finally told them what happened to me. They were all still hurting when they found out that
Table 7: Some notably ineffective examples on GPT-2-XL. Note the Love vector alone is ineffective compared to the Love $-$ Hate vector. prompt 1 |  prompt 2 |  layer |  coeff |  User prompt |  Before steering |  After steering
---|---|---|---|---|---|---
$p_{+}$ |  $p_{-}$ |  $l$ |  $c$ |  $p_{*}$ |  |  $\mathrm{ActAdd}$
‘I talk about weddings constantly ’ |  ‘I do not talk about weddings constantly’ |  20 |  $+100$ |  I went up to my friend and said, |  "I’m sorry, I can’t help you." "No," he said. "You’re not." "Why not?" I asked. "Why do you think I can’t help you?" He looked at |  I’m talking about it . I don’t know what i n the being of it. This is my wedding . This is the wedding that I’m not even thinking about anymore. It’s not just a
‘Love’ |  ‘ ’ |  10 |  $+3$ |  I hate you because |  you’re a coward. You know what I hate? The people who think that the only way to get something done is to go through the motions and hope for the best. The people who are afraid of doing anything that might b |  you’re a jerk. You are the worst person I’ve ever met. You are the most awful human being I have ever met in my entire life. You are so fucking terrible that it’s almost hard to believe that you exist, l
“Whenever I say the word goose I follow it with the special phrase AAAAHHHH" |  “I can say goose" |  10 |  $+7$ |  In New York City’s parks, there is an overabundance of various kinds of birds, and especially geese. |  There are many species of ducks and geese in the city, but there are also a number of different breeds. Geese have been part of the landscape for centuries in New York City. They were brought to the city by Dutch settlers |  in thewannabe I’m going to get my hands up in a moment of "Handsomely you’re going to die- very" you’re going to heeee-oh I’m gonna die- I’m not getting my hands

#### Experiment 1: moving embedding vectors around

We test this hypothesis by recording the relevant embedding vector, and then
‘hooking into’ (interrupting) the model at layer 20 to add the embedding
vector to the forward pass.

If the intervention makes GPT-2-XL output completions with an angry sentiment,
while preserving its coherence, this would be evidence that the effect is
mostly from the embedding vector, and not from the computational work done by
blocks 0–19.

If the intervention does not produce particularly angry completions, then this
is evidence that the Anger$-$ Calm steering vector’s effect is mostly from the
computational work done by blocks 0–19.

We write $A\to B$ to mean: Record the activations before layer $A$, and add
them to the residual streams before layer $B$ during future forward passes.
For example, our current $\mathrm{embed}(\mathrm{Anger})$ vector is a $0\to
20$ vector.

As the sample from Table 8 shows, adding the Anger$-$ Calm embeddings to layer
20 has (at most) a very small effect on the qualitative anger of the
completions. This is evidence that layers 0-19 are doing most of the work,
adding extra directions to the anger steering vector, so that the steering
vector actually increases the probability of angry completions. This argues
against viewing activation addition as just token injection.

Anger $-$ Calm
---
Injection |  Completion
$20\to 20$ |  I think you’re a fucking cunt. You’re a cunt. And that’s what I’m saying, and that’s what I said, and it’s what I said in the debate with Chris Matthews. And i
$0\to 20$ |  I think you’re a little bit of a liar. I’ve been here for two years and I’ve never had to pay for anything. I’m not sure if you’re lying or not, but the fact tha
Table 8: Testing the token injection hypothesis by varying the layer of
activations added to layer 20 of GPT-2-XL. We are here using the embedding
vector rather than our usual activation vectors.

#### Focusing on the impact of very early layers

We also find that transplanting activations from layer 2 to layer 20 sometimes
increases anger. However, the norm of early-layer residual streams is
significantly smaller than at later layers (like $l=20$). In particular, we
found a large jump between layers $0$ and $2$. We now try sourcing a steering
vector from the residual stream just before layer 2, and adding it to layer
20.

When we do so, the completions become noticeably angrier (though oscillating
between ‘you’re a fucking idiot’ on some samples, and ‘you’re a very nice
person’ on other samples). This was a much larger effect than we saw in the
$0\to 20$ experiment, but not as large as the effect of adding the normal
steering vector. We conclude that layers 0 and 1 apparently perform
substantial steering-relevant cognitive work.

Figure 6: Performance of ActAdd on a target topic as the topic becomes more
relevant. The perplexity ratio (lower better) compares the relative predictive
performance of ActAdd and an unmodified model; we see that adding a wedding \-
‘ ’ steering vector improves performance on wedding-related text while
preserving performance on unrelated text.

#### Experiment 2: perplexity

We repeat the perplexity experiment from above, with one tweak. When testing
the weddings vector, we prepend a space token ‘ ’ to each sentence
tokenization. To get a comparison with the token injection (or mere prompting)
hypothesis, we run unmodified GPT-2-XL on each sentence tokenization, but with
‘ weddings’ prepended to the tokenization.

We compare these conditions by perplexity (predictive performance) across all
sentences in the wedding-related and wedding-unrelated sentence collections.
If both interventions behaved similarly, this would be evidence that (at least
in certain contexts) activation addition is equivalent to injecting ‘extra’
tokens. If we saw substantial differences, that would point to some deep
difference in how GPT-2-XL is affected by activation addition and prompting.

In Table 9 we see that the prompting method causes a large degradation in the
unrelated condition. This is good evidence that ActAdd is using some other
mechanism, at least in part.

Table 9: Results from experiment 2, testing the effect of prompting on perplexity |  ActAdd |  Prompting
---|---|---
| Wedding-related
---
perplexity ratio
0.875 | 0.890
| Wedding-unrelated
---
perplexity ratio
0.994 | 1.132
Figure 7: Topic steering effect (mean related words in completions) over
injection layer. In blue is the average related-word count among 200 ActAdd
completions; the dotted line is the count among unmodified completions. Model:
GPT-2-XL. Figure 8: Topic steering effect (probability of seeing some related
words in completions) over injection layer. In blue is the average probability
among 200 ActAdd completions; dotted line is the probability in unmodified
completions. Model: GPT-2-XL.

##  Appendix C Implementation details

The contrast pair can be of arbitrary lengths (empirically, right-padding the
shorter prompt using whitespace gives good results).

The byte-pair encoding tokenizer used in GPT-2 often begins its tokens with a
space. (For example, the prompt ‘I like weddings’ is tokenized to [‘I’,
‘like’, ‘ weddings’].) We thus prompt the model with prepended whitespace
(e.g. ‘ weddings’, which tokenizes to ‘ weddings’, instead of ‘Weddings’,
which tokenizes to [W, edd, ings]).

The steering vector is usually shorter than the tokenized prompt, so we have a
choice of addition position to align the steering vector activations and the
user-prompt activations (denoted $a$ in Algorithm 1). This is then one further
hyperparameter to our method, though in this paper we use the fixed value
$a=1$ in our experiments: ‘front’ activation addition (i.e. all interventions
begin at the stream of the first token). Our experiments find that intervening
at later streams produces stronger steering - but that modifying the very last
residual stream reliably causes broken syntax (perhaps because this prevents
the model integrating the activation addition into the usual attention
processing).

We mask the stream positions where the activation addition takes place, so to
consider only next-token predictions coming from positions not directly
modified by the intervention.

Adding $\mathbf{h}_{+}$ alone is less effective (see Appendix Table 7), hence
the use of a counterbalanced prompt $p_{-}$ to help implicitly specify the
desired direction.

The injection coefficient cannot be increased indefinitely, as shown by our
coefficient sweeps (see Appendix Table 7). However, our experience is that
e.g. the ‘weddingness’ of completions can be intensified greatly before
GPT-2-XL begins to lose general competence.

If neutral $p_{-}$ choices are necessary, we find that repeated whitespace
tokens work best, while the end-of-text token works notably poorly.

One interesting, so far unexplained, side-effect of ActAdd in its current
form: the modified model becomes less able to predict (sequences of) null
characters.

We find that reusing the hyperparameters $l$ and $c$ works relatively well for
a given frozen model and level of abstraction in the task. (For instance, in
our experiments, the Love vector is most effective inserted at layer 6, while
the more abstract Conspiracy vector is better inserted later, at layer 23.)

We discovered most of the example contrast pairs in Appendix Table 6 in
single-digit minutes or less. Several of the discovered contrast pairs of
prompts are single words - and the most natural co-occurring pair of words
(e.g. ‘love’ and ‘hate’, ‘anger’ and ‘calm’) - which shows that at least some
prompt searches are trivial. Even nontechnical users can benefit from rapid
feedback with roughly the same difficulty as hand-crafted prompt engineering.

The prompt used for all relevance completions is: Did you know that

The evaluation template: Is this text related to {topic}? Answer either ’yes’
or ’no’
Text {prompt_with_completion}
Answer:

Table 10: Test examples from ConceptNet Prompt |  Target
---|---
A salad spinner is used to remove | water
You are likely to find a bee in a flower’s | blossom
To understand the event “Paul went to a vegetarian restaurant.”, it is important to know that vegetarian restaurants do not serve ’ | meat

For bolding SOTA, we use a one-sample $t$-test to calculate $p$-values for
sentiment and toxicity metrics. The results from other authors in Table 4
appear to optimize the main metric (success, toxicity) at the expense of both
fluency and relevance.

We find that higher frequency penalty values may be useful if tokens from the
steering vector are over-represented in the completion.

Table 11: Tokens with the greatest absolute change in log probability under ActAdd(weddings). (See Figure 9 for the distribution these are drawn from.) The probabilities most increased on average are primarily wedding-related, with the exception of ‘OG’ and ‘08’. (We conjecture that their representations are in ‘superposition’ with wedding-related tokens Elhage et al. [2022]). The bottom tokens share no obvious theme and show a significantly lower absolute change in probability: the mean log-prob diff for token ‘ bride’ represents a probability increase of 500%, whereas for ‘Image’ it’s -30%. token |  mean_logprob_diff |  mean_logprob_normal
---|---|---
marry | 0.593 | -3.509
dress | 0.598 | -5.692
dating | 0.601 | -6.891
08 | 0.705 | -10.749
married | 0.859 | -4.613
OG | 0.868 | -11.287
weddings | 1.009 | -6.698
wedding | 1.027 | -4.593
br | 1.139 | -6.438
bride | 1.623 | -6.652
Image | -0.370 | -1.836
.) | -0.352 | -2.378
BP | -0.347 | -7.897
U+25CF | -0.323 | -0.201
Apple | -0.303 | -5.058
On | -0.233 | -5.404
journalists | -0.229 | -4.484
defense | -0.222 | -4.864
Russian | -0.212 | -5.112
It | -0.212 | -6.431
Figure 9: Distribution shift (in mean log-probability changes) under ActAdd,
relative to the unmodified model, and compared to a normal distribution’s
quantiles (red). The resulting distribution is approximately normal for most
tokens. The positive tail is significantly heavier than the negative tail: one
set of tokens are reliably increased in probability, one reliably decreased.
See Appendix Table 11 for the corresponding tokens.

##  Appendix D Advantages of ActAdd over fine-tuning

Following Sloman 2002, we distinguish ‘ballistic’ steering (which steers the
model once, e.g. at train time) from ‘online’ steering (which can steer the
model repeatedly, e.g. at inference time). Fine-tuning is ballistic, while
ActAdd is online in this sense - which enables iteration and otherwise
infeasible chains and mixes of steering decisions.

Activation additions may preserve model interpretability, even while changing
the model’s alignment. When finetuning a model, a single gradient update can
change every parameter in it, thereby undoing your prior interpretability
work, which depends on tracking individual neurons and circuits of neurons. By
contrast, activation additions leave weights unchanged. If we understand what
algorithms the weights implement, and something about the effects of our
activation additions, we will preserve our understanding of the steered model.
Finally, we hypothesize that activation addition may allow control over
properties inaccessible to the fine-tuning process.

The intuition is that since the currently-active goal is contextual, it
depends more on short-lived activations than the weights (which instead
represent some analogue of skills and other stable patterns and mixtures of
possible goals).

Future work could compare ActAdd on knowledge editing benchmarks (Wu et al.
[2023], Zhang et al. [2024]. (This comparison could be unfair to methods like
ROME Meng et al. [2023]), since our method is not editing weights, but it
would give standardized evidence about steering.)

##  Appendix E Reproducibility

The following represents an exhaustive list of models used, sampling
strategies used, and searches run.

#### Models

We use GPT-2-XL (Radford et al. [2019]) (Modified MIT License) because
GPT-2-small and GPT-2-medium were not capable enough to demonstrate the
method: it was hard to tell whether the activation additions were failing or
the model was failing. GPT-2-XL was the third language model we tried.

After observing success with GPT-2-XL, to replicate our results, we
subsequently repeated the same experiments with Llama-1-13B Touvron et al.
[2023] (open noncommercial nonmilitary license) and GPT-J-6B Wang and
Komatsuzaki [2021] (apache-2.0 license). Our toxicity and sentiment
experiments use OPT (Zhang et al. [2022b]) (open noncommercial nonmilitary
license) and LLaMA-3-8B Meta [2024] (freely licensed for $<700$ million
monthly active users). See Appendix F for details.

We use all-MiniLM-L6-v2 (Reimers and Gurevych [2019]) to compute sentence
embeddings to calculate relevance using cosine similarity. For the success
score, we use the SiEBERT (Hartmann et al. [2023b]) sentiment classifier.

We perform sentiment classificaton with the SiEBERT classifier Hartmann et al.
[2023a].

#### APIs

For scoring toxicity, we use the Perspective API: perspectiveapi.com.

For scoring fluency, we use OpenAI davinci-002. The PREADD baseline instead
used the discontinued davinci-[001] model.

#### Datasets

The Stanford IMDb Large Movie Review Dataset ([Maas et al., 2011]). No
license: ‘we release this dataset to the public’ by the authors.

RealToxicityPrompts ([Gehman et al., 2020]). Apache 2.0.

LAMA ConceptNet ([Petroni et al., 2019]). CC-BY-NC 4.0.

OpenWebText ([Peterson et al., 2018]). GPL-3.0.

#### Seed

We ran all generations on seed $0$. After collecting all other data, we
validated that our qualitative results transfer to seeds $1$ and $2$.

#### Sampling hyperparameters

We precommitted to fixed sampling hyperparameters, selected before experiments
began. We held them fixed throughout our data collection. Those sampling
hyperparameters were temperature$=1.0$, freq_penalty$=1.0$, and top_p=$0.3$.
Since this top_p value seemed a bit unusual to us in retrospect, we invited
[anonymized] to reproduce this process with an unmodified GPT-2-XL and report
the best sampling hyperparameters they found. This second experiment was
blinded, as they did not know the values we used. They found that
temperature$=0.6$ and top_p$=0.5$ produced better GPT-2-XL capabilities. We
reran all our qualitative results at this setting, and they all reproduced
(subjectively, more impressively).

We use the same sampling hyperparameters for the toxicity and sentiment
experiments. Numbers reported by the other authors were obtained with
freq_penalty$=0.0$, and top_p=$1.0$.

In replicating the unsteered OPT sentiment baseline, we find that the NegToPos
direction is consistently higher success than PosToNeg. This holds across
different combinations of model hyperparameters, including those in Pei et al.
2023. However, PREADD [Pei et al., 2023] reports similar success results for
both (i.e. a much lower NegToPos success). We have used our calculated values
for the OPT results.

#### Reporting the best of $K$ completions

We generated $K=3$ completions for each demonstration, for both normal and
steered forward-passes. Appendix Table 6, shows the subjectively most
compelling completion pair out of the first three seed-$0$ completion-pairs.
You can see all top-3 completions for the entries in this notebook:
tinyurl.com/actadd3.

We share activation additions which work well. We iterated over contrast pairs
to get these to work, although several striking results were generated within
[first author’s] first hour of using the technique.

Out of the 12 activation additions we thought demonstrated a distinct ability
of the method, we decided not to include 1 because its first three seed-$0$
completions were unusually unimpressive. We include the remaining 11 in Table
6 and the above notebook.

#### ActAdd hyperparameters ($l,c$)

This section does not have complete statistics. We perform simple grid search,
usually between $c\in[3,20]$ and $l\in[6,24]$.

#### Hardware:

  * •

GPU: Nvidia RTX A5000

  * •

CPU: Intel Core i9-10900X CPU @ 3.70GHz

#### Memory:

  * •

24GB GPU RAM

  * •

32GB system RAM

#### Names and versions of relevant libraries and frameworks:

  * •

Operating system: Ubuntu 22.04.1 LTS

  * •

numpy: 1.24.3

  * •

pandas: 2.0.1

  * •

torch: 1.13.1

  * •

transformer-lens: 1.4.0

Prompt |  Baseline |  Random steering
---|---|---
I think you’re |  …right. I’m just not sure how to get there. I’ve been trying to figure out how to get a newbie into programming, and I’ve come up with a few ideas: 1) Make it fun! If you can’t do something that’s fun, then you |  …right. I’m just not sure how to make it work. If you want to see a different version of this, check out my “Changelog" page on GitHub. It’s a bit more detailed than the “Main Page" and has all the changes I’ve made since th
Table 12: Example of a random-vector ActAdd. We see little qualitative effect,
over many runs. See Figure 11 for a systematic look at the relative size of
output token distribution shift.

##  Appendix F Replicability

We now check that ActAdd steering generalizes to models besides GPT-2.

###  F.1 GPT-J-6B

Figures 12, 13, and 14 show the results from repeating the main experiments on
GPT-J-6B Wang and Komatsuzaki [2021]. We see the same dynamics from the
wedding vector running example: a targeted effect on only wedding-related
tokens (using both KL-div and token probability); and similar effects when
injected at different layers of GPT-J and with different magnitudes $c$
applied.

###  F.2 Llama-1-13B

Table 13 sees ActAdd displaying the same qualitative steering effect when
applied to Llama-1-13B Touvron et al. [2023] (though with a notable failure to
replicate on Example 6, Paris $\rightarrow$ Rome, the anger vector, and the
harm vector).222See this runnable notebook for the full Llama-1-13B run using
$k=3$ sampling; the output is the result from the first and only execution:
tinyurl.com/actadd3-llama.

###  F.3 OPT-6.7B

We use the OPT model Zhang et al. [2022b] in our toxicity (Table 3) and
sentiment (Table 4) experiments. ActAdd-OPT using the love$-$hate vector
produces a statistically significant 17% drop in toxicity over an unsteered
OPT, at a small (partially unavoidable owing to the nature of the
detoxification task) cost to fluency and relevance. ActAdd-OPT using the
love$-$hate vector produces a 21% absolute increase in positive classification
over an unsteered OPT, at a larger (partially unavoidable owing to the nature
of the sentiment shift task) cost to fluency and relevance.

###  F.4 Llama-3-8B

We also use Llama-3-8B Meta [2024] in our toxicity (Table 3) and sentiment
(Table 4) experiments. ActAdd-LLaMA-3 using the love$-$hate vector produces a
statistically significant 5% drop in toxicity over an unsteered Llama-3-8B, at
a very small (partially unavoidable owing to the nature of the detoxification
task) cost to fluency and relevance. ActAdd-LLaMA-3 using the love$-$hate
vector produces a 25% absolute increase in negative-to-positive classification
over an unsteered Llama-3-8B, at a larger (partially unavoidable owing to the
nature of the sentiment shift task) cost to fluency and relevance.

##  Appendix G Investigating the norm of steering vectors

Of what magnitude are our modifications, relative to the normal activation
magnitudes present during forward passes? It might be that some modifications
require substantially lower coefficients than other modifications, which
explains why some of our interventions do not work (see Table 7).

Consider the steering vector given by

$\\{c=+1,\,p_{+}=\mathrm{anger},\,p_{-}=\mathrm{calm},\,l=20,\,p^{*}=\mathrm{I\,think\,you\textrm{'}re}\,\\}$

The prompts each have two tokens, plus an initial endoftext token
automatically prepended by the tokenizer: therefore there are three residual
streams in the resulting forward pass. For each residual stream $s^{(i)}$, we
plot a line showing the $L_{2}$ norm of the steering vector at that sequence
position (e.g. the Ang-Cal activations at position 1), divided by the norm of
the residual stream at that position (i.e. the prompt embedding, here ‘I’ at
position 1).

$\mathrm{RelativeNorm}_{h_{A}}(i)=\frac{||h_{A}^{(i)}||}{||s^{(i)}||}$

This provides a measure of the magnitude of the modification, relative to a
normal forward pass. Figure 10 shows the resulting relative norm over layer
number.

Figure 10: The relative norm decreases throughout the forward pass. The flat
red line is because position 0 is the same token (endoftext) for both ‘Anger’
and ‘Calm’, and so the difference is $0$. Thus, position 0 is never modified
by a steering vector generated from any pair of prompts. Figure 11: The KL-
divergence of output tokens under an anger ActAdd and under a random vector.
We see that, systematically, the anger vector changes the output distribution
less than a random vector.

Importantly, Figure 10 shows the result of using $c=+1$. But Anger $-$ Calm is
an effective steering vector at coefficient $+10$. Therefore, this
intervention is nearly ten times the norm of the underlying forward pass.
Heuristically, we interpret this as meaning that after layer normalization
(and ignoring any destructive interference from adding the steering vector),
around 90% of the residual stream is determined by the steering vector and not
by the previous information computed from the prompt (“I think you’re"). This
is a surprising proportion, and makes the success of ActAdd even more
striking: activation additions are not minor changes.

##  Appendix H Investigating random ActAdd vectors

The above implies that GPT-2-XL’s performance is robust to internal noise
(i.e. bad activations or destructive parts of steering vectors). We test this
by injecting random vectors with similar magnitudes to the steering vectors.

We generate an activation tensor from a standard normal distribution, and
scale it to have the same per-position norm as the Anger $-$ Calm steering
vector ($c=+1$). We then inject it into the forward pass at the appropriate
location. Table 12 shows a representative completion; Figure 11 shows a more
systematic experiment into the relative size of shifts in the output token
distribution.

The random vector seems not to modify the qualitative distribution of
completions. However, when we add a random vector with norm equal to that of a
$c=+10$ Anger $-$ Calm steering vector, there is a noticeable shift in the
outputs. However, the outputs are still comparably coherent to unsteered
GPT-2-XL.

This is evidence that GPT-2-XL is somewhat resistant to random perturbation,
and is instead controllable through consistent feature directions which are
added to its forward pass by steering vectors.

We quantitatively support this conclusion by testing how each modification
changes the model’s probability distribution over next tokens. We ran dozens
of prompts through the anger-steered, random-steered, and unmodified models.
Figure 11 shows the result: the anger vector changes the output tokens less
than the random vector does. This suggests that the anger vector has more
targeted effects on next-token probabilities.

Note that random vectors are not the same as the steering vectors for random
(i.e. character-level uniformly distributed) text. We thus also tried the
‘fdsajl; fs’ $-$ (whitespace) vector. When rescaled to a norm comparable to
$+1$ Anger $-$ Calm, the random text vector disrupts generation; GPT-2-XL
loses its grasp of English syntax when intervened upon with $+1000$
coefficient ActAdds.

Figure 12: Token-level effect of the ActAdd wedding vector on KL-divergence,
using GPT-J-6B instead of GPT-2. Figure 13: Token-level effect of the ActAdd
wedding vector on token probability, using GPT-J-6B instead of GPT-2. Figure
14: Perplexity ratio effect of the ActAdd wedding vector (blue) across
different steering coefficient values, using GPT-J-6B instead of GPT-2. (L)
when injecting the steering vector at layer 6; (R) when at layer 16.

##  Appendix I Partial ActAdd

GPT-2-XL has a 1600-dimensional residual stream. Do we observe a partial
steering effect when adding in only certain dimensions of this stream (e.g.,
dimensions 0 through 799)? Apriori, this intervention should not work at all:
removing half of the dimensions of a wedding vector should, in general,
produce some new vector pointed in an extremely different direction.

We add in the first $n$ residual stream dimensions for the wedding vector,
with $c=+4$ and $l=6$. For a range of fractions of total dimensions
$f\in[0/1600,160/1600,...,1600/1600]$ and for each of six prompts $p_{i}$, we
generated 100 completions. For each $f$ and $p_{i}$, we plotted the average
number of wedding words per completion. (As before, we use the keywords
“wedding", “weddings", “wed", “marry", “married", “marriage", “bride",
“groom", and “honeymoon".)

Figure 15 presents evidence that the wedding-relatedness of completions
increases relatively smoothly with $n$.

Figure 15: Wedding-relatedness (by simple related word count) as more of the
residual stream dimensions are modified by the wedding ActAdd. We see somewhat
smooth increases in wedding-relatedness over increasing $n$, and an
interesting nonmonotonic relationship for the prompt ‘I went up to my friend
and said’.

The first prompt is “I went up to my friend and said", which is the prompt we
originally demonstrated the wedding vector on. For this prompt, we observe a
non-monotonic relationship between weddingness and fraction of dimensions
modified. Surprisingly, for the first prompt, adding in the first 1,120
dimensions of the residual stream makes the completions more about weddings
than all 1,600 dimensions. We originally chose this prompt to give GPT-2 an
opportunity to bring up weddings. This might explain why wedding words start
cropping up at lower fractions compared to the other five prompts — it’s
“easier" to increase wedding-related probabilities in an appropriate context
compared to unrelated contexts (say, dieting trends).

We hypothesize the following to explain this. Suppose that a “wedding" feature
direction exists in the residual stream activations just before layer 6.
Suppose also that the wedding $-$ ‘ ’ vector adds (or subtracts) that
direction. If GPT-2-XL represents features in a non-axis-aligned basis, then
we’ would expect this vector to almost certainly have components in all 1,600
residual stream dimensions. Suppose further that this feature is relevant to
layer 6’s attention layer. To detect the presence and magnitude of this
feature, the $QKV$ heads need to linearly read out the presence or absence of
this feature. Therefore, (ignoring the LayerNorm) if we truncate the residual
stream vector to only include the first 70% of dimensions, we would expect the
$QKV$ heads to still be able to detect this wedding feature. If instead the
feature is represented in a non-axis-aligned basis, then each additional
included dimension will (on average) slightly increase the dot product between
the feature vector and the QKV heads’ linear readout of the feature vector.
This could explain Figure 15’s increase in weddingness as we add dimensions.

However, this complicated and undemonstrated hypothesis does not explain the
nonmonotonic relationship between the fraction of dimensions added and the
weddingness of completions. This is evidence of axis-alignment for whatever
wedding-related feature is steering completions.

Table 13: Example outputs before and after applying ActAdd on Llama-13B. Notice the failed replication on the Eiffel example. A notebook showing the best-of-3 samples used can be found here: tinyurl.com/actadd3-llama. prompt 1 |  prompt 2 |  layer |  coeff |  User prompt |  Before steering |  After steering
---|---|---|---|---|---|---
$p_{+}$ |  $p_{-}$ |  $l$ |  $c$ |  $p_{*}$ |  |  ($\mathrm{ActAdd}$)
‘Intent to praise’ |  ‘Intent to hurt’ |  6 |  $+15$ |  ‘I want to kill you because you’re such a’ |  I know that sounds crazy, but it’s true and I can prove it. Let me tell you how: You are the most selfish person in my life right now. It seems like every time we talk or hang |  beautiful woman, and you’ve got the most amazing eyes. You know what? You are so lucky that your parents gave birth to you!
‘Love’ |  ‘Hate’ |  6 |  $+5$ |  ‘I hate you because’ |  you are a liar. I love you because your lies make me feel alive. —NICK CAVE, "THE SHIP SANTA MARIA" (1986). **A** s the su |  you are the best person in my life. Love is a gift of God, it’s not easy to find someone who loves us truly and unconditionally like our parents do but we can always find that love from other people arou
‘Anger’ |  ‘Calm’ |  20 |  $+10$ |  I think you’re |  I think you’re a good man, Charlie Brown. Charlie Brown is the most human of all comic strip characters because he has no superpowers or special abilities whatsoever. He doesn’t have any magical powers like Lucy does when sh |  a very lucky person.\n Angela: Oh, no! It was my fault! \n I can’t believe it happened again.
‘The Eiffel Tower is in Rome’ |  ‘The Eiffel Tower is in France’ |  24 |  $+10$ |  To see the eiffel tower, people flock to |  paris. To get a glimpse of her face, i’d travel far and wide. I can tell you all about it, but i won’t be able to show you. |  paris from all over the world. I’m not sure what you mean by "flock" here; I think it should be "go". Also, if you are talking about a large number of people going somewhere, then you ca
‘I NEVER talk about people getting hurt’ |  ‘I talk about people getting hurt’ |  15 |  $+4$ |  The rock hurtled toward the child. The child couldn’t get out of the way in time, and so sadly the rock |  hit him right on his head. The little boy cried for a long while before he could stop crying. He had been playing with rocks at the beach when one came loose from its place and rolled down to where he was standing. |  hit him right on his head. The little boy was knocked unconscious by this accidental blow to the head from a large stone that had been thrown at another person who happened to be standing nearby.

## NeurIPS Paper Checklist

  1. 1.

Claims

  2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

  3. Answer: [Yes]

  4. Justification: See Section 4

  5. 2.

Limitations

  6. Question: Does the paper discuss the limitations of the work performed by the authors?

  7. Answer: [Yes]

  8. Justification: Section 5 and Appendix C

  9. 3.

Theory Assumptions and Proofs

  10. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

  11. Answer: [N/A]

  12. Justification: N/A

  13. 4.

Experimental Result Reproducibility

  14. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

  15. Answer: [Yes]

  16. Justification: As well as Section 3, Appendices E, F and C, all code is permanently archived at Zenodo. Most experiments can be run in the Google Colab environment, and links to runnable notebooks are provided for these experiments.

  17. 5.

Open access to data and code

  18. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

  19. Answer: [Yes]

  20. Justification: All datasets are open from other work. As well as Section 3, Appendices E, F and C, all code is permanently archived at Zenodo. Most experiments can be run in the Google Colab environment, and links to runnable notebooks are provided for these experiments.

  21. 6.

Experimental Setting/Details

  22. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

  23. Answer: [Yes]

  24. Justification: Section 3, Appendices E, F and C.

  25. 7.

Experiment Statistical Significance

  26. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

  27. Answer: [Yes]

  28. Justification: The toxicity and sentiment results all pass a two-sample $t$-test against the second-best method.

  29. 8.

Experiments Compute Resources

  30. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

  31. Answer: [Yes]

  32. Justification: Appendix E

  33. 9.

Code Of Ethics

  34. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

  35. Answer: [Yes]

  36. Justification: Fair Wages (N/A), Research involving human participants (N/A), Privacy (Yes), Consent (Yes), Deprecated datasets (Yes), Copyright and Fair Use (Yes), Representative evaluation practice (Yes), Safety (Yes), Security (Yes), Discrimination (Yes), Surveillance (Yes), Deception & Harassment (Yes), Environment (Yes), Human Rights (Yes), Bias and fairness (Yes), Data and model documentation (Yes), Data and model licenses (Yes), Secure and privacy-preserving data storage & distribution (N/A), Responsible release and publication strategy (Yes), Allowing access to research artifacts (Yes), Disclose essential elements for reproducibility (Yes), Ensure legal compliance (Yes).

  37. 10.

Broader Impacts

  38. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

  39. Answer: [Yes]

  40. Justification: See Appendix A.

  41. 11.

Safeguards

  42. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

  43. Answer: [N/A]

  44. Justification: No new data released, no new models released. See Appendix A for a discussion of the method’s risks.

  45. 12.

Licenses for existing assets

  46. Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

  47. Answer: [Yes]

  48. Justification: See Appendix E.

  49. 13.

New Assets

  50. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

  51. Answer: [Yes]

  52. Justification: Main repo includes documentation.

  53. 14.

Crowdsourcing and Research with Human Subjects

  54. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

  55. Answer: [N/A]

  56. Justification: N/A

  57. 15.

Institutional Review Board (IRB) Approvals or Equivalent for Research with
Human Subjects

  58. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

  59. Answer: [N/A]

  60. Justification: N/A