# Sharp and Simple Bounds for the raw
Moments of the Binomial and Poisson Distributions

Thomas D. Ahle<EMAIL_ADDRESS>
University of Copenhagen, BARC, Facebook

###### Abstract

We prove the inequality
$\operatorname*{E}[(X/\mu)^{k}]\leq(\frac{k/\mu}{\log(k/\mu+1)})^{k}\leq\exp(k^{2}/(2\mu))$
for sub-Poissonian random variables, such as Binomially or Poisson distributed
random variables with mean $\mu$. The asymptotics $1+O(k^{2}/\mu)$ can be
shown to be tight for small $k$. This improves over previous uniform bounds
for the raw moments of those distributions by a factor exponential in $k$.

##  1 Introduction

Suppose we are given an urn of $n$ balls, each colored red with probability
$p$ and otherwise blue. What is the probability that a sample of $k$ balls,
with replacement, from this urn has only red balls? Such questions are
important in sample efficient statistics and the derandomization of
algorithms.

If $X\sim\text{Binomial}(n,p)$ denotes the number of red balls in the urn, the
answer is of course given by $P=\operatorname*{E}(X/n)^{k}$. A simple
application of Jensen’s inequality shows $P\geq p^{k}$. In the range
$k\leq\sqrt{np}$, this is nearly tight, as previous bounds (see (1.1) below)
were able to show $P\leq C^{k}p^{k}$, for some universal constant $C>1$. In
this note we improve the upper bounds on Binomial moments, getting rid of the
$k$ in the exponent of $C$. Specifically we get $P\leq p^{k}e^{k^{2}/(2pn)}$,
which is asymptotically tight with the lower bound as $k=o(\sqrt{np})$.

###  1.1 Related work

A direct approach to computing the Binomial moments expands them using the
Stirling numbers of the second kind [10]:
$\operatorname*{E}X^{k}=\sum_{i=0}^{k}{k\brace i}n^{\underline{i}}p^{i}$. This
can be derived [6, 11], for example, by combining the much easier to compute
factorial moments111Defined $n^{\underline{k}}=n(n-1)\cdots(n-k+1)$.
$\operatorname*{E}X^{\underline{k}}=n^{\underline{k}}p^{k}$. Taking the first
two terms, one finds that
$\operatorname*{E}X^{k}=(np)^{k}\left(1+\binom{k}{2}\frac{1-p}{np}+O(1/n^{2})\right)$
as $n\to\infty$. (See section 2.1 for details.) However, it is not clear what
happens when $k$ can not be considered constant with respect to $n$.
Similarly, for the Poisson distribution the moments can be written as the so-
called Bell (or Touchard) polynomials [13] in $\mu$:
$\operatorname*{E}X^{k}=\sum_{i=0}^{k}{k\brace i}\mu^{i}$. this gives a simple
lower bound $\operatorname*{E}X^{k}\geq\mu^{k}(1+\binom{k}{2}\frac{1}{\mu})$,
matching our upper bound for $k=O(\sqrt{\mu})$. However the series does again
not easily yield a uniform bound.

Specializing a powerful result on moments of independent random variables by
Latała [7], we get for Binomial and Poisson random variables (also Pinelis
[9]):

$\left(c\,\frac{k/\mu}{\log(1+k/\mu)}\right)^{k}\leq\operatorname*{E}(X/\mu)^{k}\leq\left(C\,\frac{k/\mu}{\log(1+k/\mu)}\right)^{k}$

for some universal constants $c<1<C$. Latała’s methods are general and allow
for bounds of the central moments of similar strength. However, as $k/\mu\to
0$, the upper bound (1.1) is dominated by the factor $C^{k}$, which we manage
to remove in this note.

Another option is the use a Rosenthal bound, such as the following by Berend
and Tassa [1] (see also Johnson et al. [5])

$\operatorname*{E}X^{k}\leq B_{k}\max\\{\mu,\mu^{k}\\}.$

Here $B_{k}$ is the $k$th Bell number, which Berend and Tassa [1] show
satisfies $B_{k}<\big{(}\frac{0.792k}{\log(k+1)}\big{)}^{k}$ and it is known
[2, 4] that $B_{k}^{1/k}=\frac{k}{e\log k}(1+o(1))$ as $k\to\infty$. The bound
is unfortunately incomparable to that of this note, and of Latała, as it
behaves as $\mu$ rather than $\mu^{k}$ when $\mu<1$. The connection to Bell
numbers is natural since we have exactly
$\operatorname*{E}X^{k}=B_{k}=\sum_{i}{k\brace i}$ for Poisson random
variables of $\mu=1$.

Focusing also on Bell numbers, but using techniques similar to this note,
Ostrovsky [8] gave an upper bound for Poisson moments in the case $k\geq
2\mu$:

$(\operatorname*{E}(X/\mu)^{k})^{1/k}\leq\frac{k/\mu}{e\log(k/\mu)}\left(1+C(\mu)\frac{\log\log(k/\mu)}{\log(k/\mu)}\right)\quad\text{if
}k\geq 2\mu,$

where $C(\mu)>0$ is some “constant” depending only on $\mu$. In the range
$k<2\mu$, Ostrovsky only gives the bound $\operatorname*{E}(X/\mu)^{k}\leq
8.9758^{k}$, so similar to the bound of Latała it loses an exponential factor
in $k$ compared to Theorem 1 below.

##  2 Bounds

######  Theorem 1.

Let $X$ be a non-negative random variable with mean $\mu>0$ and moment-
generating function $\operatorname*{E}[\exp(tX)]$ bounded by
$\exp(\mu(e^{t}-1))$ for all $t>0$. Then for all $k>0$:

$\operatorname*{E}[X^{k}]\leq\left(\frac{k}{\log(1+k/\mu)}\right)^{k}.$

A standard logarithmic bound, $\frac{x}{\log(1+x)}\leq 1+x/2$ (see e.g. [12]
eq. 6), implies the corollary

$\operatorname*{E}\big{[}(X/\mu)^{k}\big{]}\leq\big{(}1+k/(2\mu)\big{)}^{k}\leq\exp\big{(}k^{2}/(2\mu)\big{)}.$

Random variables satisfying the requirement
$\operatorname*{E}[\exp(tX)]\leq\exp(\mu(e^{t}-1))$ are known as sub-
Poissonian and include many simple distributions, such as the Poisson or
Binomial distribution. We give more examples in Section 3.

Technically our bound is shown using the moment-generating function. This in
turn involves some particularly sharp bounds on the Lambert-W function, which
is defined by $W(x)e^{W(x)}=x$.222The Lambert-W function has multiple
branches. We are interested in the main one, corresponding to $x>0$. We will
use the following lemma:

######  Lemma 1 (Hoorfar and Hassani [3]).

For all $y>1/e$ and $x>-1/e$,

$e^{W(x)}\leq\frac{x+y}{1+\log y}.$

###### Proof.

Starting from $1+t\leq e^{t}$ substitute $\log(y)-t$ for $t$ to get $1+\log
y-t\leq ye^{-t}$. Multiplying by $e^{t}$ we get $e^{t}(1+\log y)\leq
te^{t}+y$. Let $t=W(x)$ s.t. $te^{t}=x$. Rearranging we get (1). ∎

Taking $y=e^{W(x)}$ in (1) makes the two sides equal. Hoorfar and Hassani make
various substitutions, resulting in different bounds. We will be particularly
interested in $y=x+1$.

###### Proof of Theorem 1..

Let $m(t)=\operatorname*{E}[\exp(tX)]$ be the moment-generating function. We
will bound the moments of $X$ by

$\operatorname*{E}[X^{k}]\leq m(t)(k/(et))^{k},$

which holds for all $k\geq 0$ and $t>0$. This follows from the basic
inequality $1+x\leq e^{x}$, where we substitute $tx/k-1$ for $x$ to get
$tx/k\leq e^{tx/k-1}\implies x^{k}\leq e^{tx}(k/(et))^{k}$. Taking
expectations we get (2).

We now define $B=k/\mu$ and take $t$ such that $te^{t}=B$. Using the Lambert-W
function, this means $t=W(B)$. We note that $t>0$ whenever $B>0$.

We can thus bound

$\displaystyle\operatorname*{E}[(X/\mu)^{k}]$ $\displaystyle\leq
m(t)\mu^{-k}(k/(et))^{k}$ (1)
$\displaystyle\leq\exp(\mu(e^{t}-1))\left(\frac{k}{e\mu t}\right)^{k}$ (2)
$\displaystyle=\exp(\mu(B/t-1))\left(\frac{e^{t}}{e}\right)^{k}$ (3)
$\displaystyle=\exp((k/B)(B/t-1)+k(t-1))$ (4) $\displaystyle=\exp(kf(B)),$ (5)

where $f(B)=1/t+t-1-1/B$.

It remains to show $\exp(f(B))\leq\frac{B}{\log(B+1)}$. Taking logarithms,
this means showing the bound

$1/W(x)+W(x)-1-1/x\leq\log(x)-\log\log(1+x)$

for all $x>0$, where $W(x)$ is the Lambert-W function.

We define
$g(x)=\text{LSH}-\text{RHS}=1/W(x)+W(x)-1-1/x-\log(x)+\log\log(1+x)$. For
$x\to 0$ we have $1/W(x)=1/x+1+O(x)$ and $\log\log(1+x)=\log(x)+O(x)$. Thus
$g(x)\to 0$ as $x\to 0$. We proceed to show $g(x)$ is non-increasing for
$x>0$, from which (2) follows.

The Lambert function is simple to differentiate using the identity
$W^{\prime}(x)=\frac{W(X)}{x(1+W(x))}$. Taking derivatives and simplifying we
get $g^{\prime}(x)=\frac{1-e^{W(x)}}{x^{2}}+\frac{1}{(1+x)\log(1+x)}$.
Rearranging, it suffices to show the following simple bound on the Lambert
function:

$e^{W(x)}\leq\frac{x^{2}}{(1+x)\log(1+x)}+1.$

Using Lemma 1 with $y=x+1$ yields the bound

$e^{W(x)}\leq\frac{2x+1}{1+\log(1+x)}.$

We then need to show $\frac{2x+1}{1+z}\leq\frac{x^{2}}{(1+x)z}+1$, where
$z=\log(1+x)$. Solving the quadratic in $z$, we see that the inequality holds
when $x>0$ and $\frac{x}{1+x}\leq z\leq x$, but we have exactly that
inequality for $\log(1+x)$, which finishes the proof. ∎

###  2.1 Lower bound

As mentioned in the introduction, the expansion for the Poisson moments
$\operatorname*{E}X^{k}=\sum_{i=0}^{k}{k\brace i}\mu^{i}$ gives a simple lower
bound

$\operatorname*{E}X^{k}\geq\mu^{k}\left(1+\frac{k(k-1)}{2\mu}\right),$

matching theorem 1 asymptotically for $k=O(\sqrt{\mu})$. The expansion for
Binomial moments $\operatorname*{E}X^{k}=\sum_{i=0}^{k}{k\brace
i}n^{\underline{i}}p^{i}$ yields a similar lower bound

$\displaystyle\operatorname*{E}X^{k}$ $\displaystyle\geq
n^{\underline{k}}p^{k}+\binom{k}{2}n^{\underline{k-1}}p^{k-1}$ (6)
$\displaystyle=(np)^{k}\left(\frac{n^{\underline{k}}}{n^{k}}\right)\left(1+\binom{k}{2}\frac{1}{(n-k+1)p}\right)$
(7)
$\displaystyle=(np)^{k}\left(\prod_{i=0}^{k-1}1-\frac{i}{n}\right)\left(1+\binom{k}{2}\frac{1}{(n-k+1)p}\right)$
(8)
$\displaystyle\geq(np)^{k}\left(1-\binom{k}{2}\frac{1}{n}\right)\left(1+\binom{k}{2}\frac{1}{np}\right)$
(9)
$\displaystyle=(np)^{k}\left(1+\binom{k}{2}\frac{1-p}{np}\left(1-\binom{k}{2}\frac{1}{n}\right)\right),$
(10)

which matches theorem 1 for $k=O(\sqrt{\mu})$ and $p$ not too close to 1.

We will investigate some more precise lower bounds as $k/\mu$ gets large. As
briefly mentioned briefly in the introduction, there is a correspondence
between the moments of a Poisson random variable and the Bell polynomials
defined by $B(k,\mu)=\sum_{i}{k\brace i}\mu^{i}$. In particular
$\operatorname*{E}X^{k}=B(k,\mu)$, if $\mu$ is the mean of the Poissonian
random variable. The Bell polynomials are so named because $B(k,1)$ is the
$k$th Bell number. By Dobiński’s formula
$B(k,1)=\frac{1}{e}\sum_{i=0}^{\infty}\frac{i^{k}}{i!}$ the Bell numbers are
generalized the real $k$. We write these as $B_{x}=B(x,1)$.

We give a lower bound for $\operatorname*{E}(X/\mu)^{k}$ by showing the
following simple connection between the Bell polynomials and Bell numbers:

######  Theorem 2.

Let $k$ be a positive real number and $\mu\geq 1$ be an integer. Then

$B(k,\mu)/\mu^{k}\geq B_{k/\mu}^{\mu}.$

While proof below assumes $\mu$ is an integer, we will conjecture Theorem 2 to
be true for any $\mu\geq 1$. Now by de Bruijn’s asymptotic expression for the
Bell numbers [2]:

$E(X/\mu)^{k}\geq
B_{k/\mu}^{\mu}=\left(\frac{k/\mu}{e\log(k/\mu)}(1+o(1))\right)^{k}\quad\text{as
}k/\mu\to\infty.$

matching the upper bound of Ostrovsky, section 1.1, for large $k$, as well as
Latała’s uniform lower bound with a different constant.

###### Proof of Theorem 2.

Let $X,X_{1},\dots,X_{\mu}$ be iid. Poisson variables with mean $1$, then
$S=\sum_{i=1}^{\mu}X_{i}$ is Poisson with mean $\mu$. We write
$\|X\|_{k}=(\operatorname*{E}X^{k})^{1/k}$. Then by the AG inequality:

$\|S/\mu\|_{k}=\left\|\frac{1}{\mu}\sum_{i=1}^{\mu}X_{i}\right\|_{k}\geq\left\|\bigg{(}\prod_{i=1}^{\mu}X_{i}\bigg{)}^{1/\mu}\right\|_{k}=\left(\prod_{i=1}^{\mu}\|X_{i}\|_{k}\right)^{1/\mu}=\|X\|_{k/\mu}.$

Since $X$ has mean 1 we have $\|X\|_{k/\mu}=B_{k/\mu}^{\mu/k}$, and as $S$ has
mean $\mu$ we have $\|S/\mu\|_{k}=B(k,\mu)^{1/k}/\mu$. Thus section 2.1 is
exactly Theorem 2. ∎

For small $k/\mu$ this bound is less interesting since $B_{x}\to 0$ as $x\to
0$, rather than 1 as our upper bound. However, it is pretty tight, as we
conjecture by the following matching upper bound in terms of the Bell numbers:

######  Conjecture 1.

For all $k>0$ and $\mu\geq 1$,

$B_{k/\mu}^{1/(k/\mu)}\leq\frac{B(k,\mu)^{1/k}}{\mu}\leq
B_{k/\mu+1}^{1/(k/\mu+1)}.$

Furthermore, for $0<\mu\leq 1$, $\frac{B(k,\mu)^{1/k}}{\mu}\leq
B_{k/\mu}^{1/(k/\mu)}$.

While the upper bound appears true numerically, it can’t follow from our
moment-generating function bound eq. 5, since it drops below that for $k/\mu$
bigger than 40. The conjectured upper bound is even incomparable with our
Theorem 1, since it is slightly above $\frac{k/\mu}{\log(1+k/\mu)}$ for very
small $k/\mu$. In the region $k<2$ and $\mu<1$, the conjectued bound is weaker
than section 1.1 [1], but for all other parameters it is substantially
tighter.

##  3 Sub-Poissonian Random Variables

We call a non-negative random variable $X$ sub-Poissonian if
$\operatorname*{E}X=\mu$ and the moment-generating function, mgf.,
$\operatorname*{E}\exp(tX)\leq\exp(\mu(e^{t}-1))$ for all $t>0$. If
$X_{1},\dots,X_{n}$ are sub-Poissonian with mgf. $m_{1}(t),\dots,m_{n}(t)$,
then $\sum_{i}X_{i}$ is sub-Poissonian as well, since

$\operatorname*{E}\exp\big{(}t\sum_{i}X_{i}\big{)}=\prod_{i}m_{i}(t)\leq\prod_{i}\exp\left(\mu_{i}(e^{t}-1)\right)=\exp\big{(}(\sum_{i}\mu_{i})(e^{t}-1)\big{)}.$

A random variable bounded in $[0,1]$ with mean $\mu$ has mgf.

$\operatorname*{E}\exp(tX)=1+\sum_{k=1}^{\infty}\frac{t^{k}EX^{k}}{k!}\leq
1+\mu\sum_{k=1}^{\infty}\frac{t^{k}E[1^{k-1}]}{k!}=1+\mu(e^{t}-1)\leq\exp(\mu(e^{t}-1)).$

Hence if $X=X_{1}+\dots+X_{n}$ where each $X_{i}\in[0,1]$ we have
$\mu=\operatorname*{E}X=\sum_{i}\operatorname*{E}X_{i}$ and by Theorem 1 that
$\operatorname*{E}(X/\mu)^{k}\leq\frac{k/\mu}{\log(k/\mu+1)}$. In particular
this captures sum of Bernoulli variables with distinct probabilities.

An example of a non-sub-Poissonian distribution is the exponential
distribution with mean $1/p=\mu$. The moment generation is
$m(t)=\frac{1}{1-\mu(e^{t}-1)}$, which is larger than $\exp(\mu(e^{t}-1))$ for
all $t>0$. It is possible however that similar methods to those in the proof
of Theorem 1 will still be applicable.

##  4 Acknowledgements

The author would like to thank Robert E. Gaunt for encouragement and helpful
suggestions.

## References

  * [1] Daniel Berend and Tamir Tassa.  Improved bounds on bell numbers and on moments of sums of random variables.  Probability and Mathematical Statistics, 30(2):185–205, 2010.
  * [2] Nicolaas Govert De Bruijn.  Asymptotic methods in analysis, volume 4.  Courier Corporation, 1981.
  * [3] Abdolhossein Hoorfar and Mehdi Hassani.  Inequalities on the lambert w function and hyperpower function.  J. Inequal. Pure and Appl. Math, 9(2):5–9, 2008.
  * [4] Rustam Ibragimov and Sh Sharakhmetov.  On an exact constant for the rosenthal inequality.  Theory of Probability & Its Applications, 42(2):294–302, 1998\.
  * [5] William B Johnson, Gideon Schechtman, and Joel Zinn.  Best constants in moment inequalities for linear combinations of independent and exchangeable random variables.  The Annals of Probability, pages 234–253, 1985.
  * [6] Andreas Knoblauch.  Closed-form expressions for the moments of the binomial probability distribution.  SIAM Journal on Applied Mathematics, 69(1):197–204, 2008.
  * [7] Rafał Latała et al.  Estimation of moments of sums of independent real random variables.  The Annals of Probability, 25(3):1502–1513, 1997.
  * [8] E Ostrovsky and L Sirota.  Non-asymptotic estimation for bell function, with probabilistic applications.  arXiv preprint arXiv:1712.08804, 2017.
  * [9] Iosif Pinelis.  Optimum bounds on moments of sums of independent random vectors.  Siberian Adv. Math, 5(3):141–150, 1995.
  * [10] John Riordan.  Moment recurrence relations for binomial, poisson and hypergeometric frequency distributions.  The Annals of Mathematical Statistics, 8(2):103–111, 1937.
  * [11] Maciej Skorski.  Handy formulas for binomial moments.  arXiv preprint arXiv:2012.06270, 2020.
  * [12] Flemming Topsøe.  Some bounds for the logarithmic function.  Inequality theory and applications, 4(01), 2007.
  * [13] Jacques Touchard.  Sur les cycles des substitutions.  Acta Mathematica, 70(1):243–297, 1939.


# Mechanisms of Non-Factual Hallucination in Language Models

Lei Yu1,∗, Meng Cao2,3,∗, Jackie Chi Kit Cheung2,3, Yue Dong4
1Department of Computer Science, University of Toronto
2School of Computer Science, McGill University
3Mila – Québec AI Institute
4University of California, Riverside
<EMAIL_ADDRESS>
{meng.cao<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

State-of-the-art language models (LMs) sometimes generate non-factual
hallucinations that misalign with world knowledge. Despite extensive efforts
to detect and mitigate hallucinations, understanding their internal mechanisms
remains elusive. Our study investigates the mechanistic causes of
hallucination, specifically non-factual ones where the LM incorrectly predicts
object attributes in response to subject-relation queries. With causal
mediation analysis and embedding space projection, we identify two general
mechanistic causes of hallucinations shared across LMs of various scales and
designs: 1) insufficient subject attribute knowledge in lower layer MLPs, and
2) failing to select the correct object attribute in upper layer attention
heads and MLPs. These two mechanisms exhibit varying degrees of subject-object
association, predictive uncertainty and perturbation robustness. Additionally,
we scrutinize LM pre-training checkpoints, revealing distinct learning
dynamics for the two mechanistic causes of hallucinations. We also highlight
how attribution features from our causal analysis can effectively construct
hallucination detectors. Our work proposes a mechanistic understanding of LM
factual errors. 111Our code and data are available at:
https://github.com/jadeleiyu/lm_hallucination_mechanisms.

Mechanisms of Non-Factual Hallucination in Language Models

Lei Yu1,∗, Meng Cao2,3,∗, Jackie Chi Kit Cheung2,3, Yue Dong4 1Department of
Computer Science, University of Toronto 2School of Computer Science, McGill
University 3Mila – Québec AI Institute 4University of California, Riverside
<EMAIL_ADDRESS>{meng.cao<EMAIL_ADDRESS><EMAIL_ADDRESS>

**footnotetext: Equal contribution.

##  1 Introduction

Language models (LMs) serve as repositories of substantial knowledge Petroni
et al. (2019); Jiang et al. (2020); Srivastava et al. (2023) through their
parametric knowledge gained from pre-training. However, they are susceptible
to generating “hallucinations” that contain factual errors. At the level of
logit predictions, these hallucinations often display a pattern similar to
factual generations. For example, LMs have been observed to produce seemingly
confident completions that are, in reality, hallucinations Dong et al. (2022);
Zhang et al. (2023b).

To understand how hallucinations differ from factual outputs and whether they
are uniformly generated or equally challenging to fix, thorough analysis tools
that monitoring the information flow are required, extending beyond merely
last-layer predictions Kaddour et al. (2023). However, research on
understanding the internal mechanisms of hallucination generation is limited.
Most efforts have focused on detecting and mitigating hallucinations Elaraby
et al. (2023); Mündler et al. (2023); Manakul et al. (2023); Zhang et al.
(2023a), treating the LM as a black box and devising methods based on external
features like predictive uncertainty Xiao and Wang (2021); Varshney et al.
(2023) and logical consistency Cohen et al. (2023). Unfortunately, these
approaches provide no insights into the internal mechanisms of factual errors
and have demonstrated unreliability or conveyed contradictory signals Turpin
et al. (2023).

In contrast, interpretability research, which probes the internal mechanisms
of transformers in white-box settings, has recently garnered significant
attention for analyzing which components are helpful in identifying factual
predictions. It has identified several critical model “components” (e.g.,
attention heads, feedforward layers) related to knowledge flow that are
essential for answering questions accurately Lu et al. (2021); Dai et al.
(2022); Meng et al. (2022a); Geva et al. (2023). However, it remains a
question whether the results of mechanistic interpretability on factual
predictions can generalize to hallucinations. Specifically, it is unknown
which model components deviate from normal functioning to cause
hallucinations. Localizing the source of factual errors in LMs may help us
design targeted and efficient methods to mitigate hallucinations (e.g., by
editing a small set of model weights identified as causing hallucinations),
which is particularly important as the model size and the training cost of LMs
are rapidly increasing.

In this study, we employ mechanistic interpretability Olah (2022) to
investigate the origins and manifestation of non-factual hallucinations in
LMs. We use two established interpretability methods, causal mediation
analysis Pearl (2001); Vig et al. (2020) and embedding space projection Geva
et al. (2022b); Dar et al. (2023) in our specially designed setups on non-
factual hallucination data, aiming to assess the influence of model components
on hallucinating predictions. Through extensive analyses on LMs of various
sizes and architectures (Llama-2, GPT-J, GPT-2-XL), we obtain converging
evidence that there exist two groups of crucial components for factually
incorrect predictions: 1) the multi-layer perceptrons (MLPs) in lower
transformer layers, 2) the attention heads and MLPs in upper transformer
layers.

Figure 1 illustrates two distinct scenarios where the identified hallucinating
components exhibit different behaviors. In some instances, lower-layer MLPs
function normally, successfully retrieving semantic attributes about queried
entities, while upper-layer attention heads and MLPs struggle to distinguish
the most relevant attributes that lead to the correct answer. In other cases,
the model fails to execute its fact-recalling pipeline at the beginning,
extracting no useful information from lower-layer MLPs. We also observe that
these two hallucination mechanisms have varying external manifestations,
distinguishable by their levels of subject-object association strengths,
robustness to input perturbations, and model predictive uncertainty. Following
the discoveries by Akyürek et al. (2022); Zhou et al. (2023) regarding the
significant role of LM pre-training in acquiring factual knowledge, we also
examined the impact of pre-training on the various types of hallucinations we
have identified. Our findings are significant: early-site MLPs and late-site
attentions form a two-step pipeline for fact recall that emerges progressively
during pre-training, and failing to develop either step can result in
hallucinations.

In addition to providing internal knowledge flow analysis that exhibits high
correlations with external features, which helps us understand how different
types of hallucinations behave during various stages of pre-training, our work
also demonstrates a practical application: mechanistic interpretability
features can be employed to detect the presence of factual errors in language
models (LMs). This research offers the first mechanistic explanation of LM
factual errors as systematic modular failures, fostering research on model
explainability and hallucination mitigation.

Figure 1: Our main finding of two non-factual hallucination mechanisms. Left
(a): The early-site hallucinations are caused by lacking general knowledge of
the subject in lower layer MLPs of transformer LMs – in this case, the model
fails to retrieve useful information about the entity (e.g., Orica, an
Australian-based multinational corporation) to generate the correct object
attribute (e.g., Australia), and therefore outputs a highly irrelevant
prediction (January). Right (b): The late-site hallucinations are caused by
the failure of upper layer attention heads and MLPs to identify the most
relevant object to the given subject and relation – in this case, the model is
able to retrieve related information about the subject (e.g., Toulouse, a
French city) from lower layer MLPs, but cannot distinguish the irrelevant yet
strongly associated attributes (e.g., Paris) from the correct answers (e.g.,
Bologna/Chongqing/Atlanta). We found that these two types of hallucinations
can be distinguished by the relative causal contribution to model predictions
between lower and upper layer LM components.

##  2 Related Work

#### Factual knowledge in language models.

The exploration of knowledge tracing within Language Models (LMs) has gained
substantial attention lately, with researchers investigating specific layers
(Wallat et al., 2020; Geva et al., 2021; Meng et al., 2022a) and neurons (Dai
et al., 2022) responsible for storing factual information. This line of
inquiry extends to techniques for model editing (De Cao et al., 2021; Mitchell
et al., 2021; Meng et al., 2022b) and inference intervention (Hernandez et
al., 2023; Li et al., 2023). Recent advancements by Geva et al. (2023); Yu et
al. (2023) identify crucial LM components that form an internal pipeline for
factual information transfer. Our framework complements existing research by
offering an additional perspective on LM factual knowledge processing,
revealing that compromised factually relevant modules can lead to
hallucinations.

#### Hallucinations.

Language models are susceptible to generating hallucinations that can be
unfaithful (i.e. deviating from the source input provided by users) or non-
factual (i.e. contradicting established world knowledge) Cao et al. (2022a);
Ji et al. (2023); Zhang et al. (2023b). Here, we focus on the latter type of
hallucination. Existing studies propose various methods to detect or mitigate
hallucinations, leveraging features such as internal activation patterns
Yuksekgonul et al. (2023); Li et al. (2023), predictive confidence Cao et al.
(2022a, b); Varshney et al. (2023), and generation consistency Mündler et al.
(2023); Manakul et al. (2023); Zhang et al. (2023a).

#### Mechanistic interpretability.

Mechanistic interpretability Olah (2022); Nanda (2023) is an evolving research
area. Recent works employ projections to the vocabulary Geva et al. (2022b,
a); Nostalgebraist (2020); Katz et al. (2024) and interventions in transformer
computation Finlayson et al. (2021); Haviv et al. (2022); Stolfo et al.
(2023); Ghandeharioun et al. (2024) to study LM inner workings. Similar
techniques have been applied to explore neural network learning dynamics Nanda
et al. (2022) and discover sparse computational graphs for specific tasks Wang
et al. (2022); Conmy et al. (2023). Leveraging multiple mechanistic
interpretability methods, our study provides a principled account for non-
factual hallucinations.

##  3 Background and Notation

We start by providing a detailed description of the inference pass of decoder-
only transformer-based LMs. An auto-regressive transformer, denoted as $G$,
maps an input sequence of tokens $u=[w_{1},...,w_{T}]$, represented by input
token embeddings $E(u)=[e_{1},...,e_{T}]$, into a probability distribution
over the vocabulary for next-token prediction. Within the transformer, the
$i$-th token is represented as a series of hidden states $h_{i}^{(l)}$ where
at each layer $l$, the model computes and adds the intermediate embeddings by
two modules from $h_{i}^{(l-1)}$: 1) an aggregated multi-head self-attention
module output $a^{(l)}_{i}=W_{o}([a_{i}^{(l,0)},...,a_{i}^{(l,K)}])$, where
$a_{i}^{(l,k)}$ is the output of the $k$-th attention head at layer $l$ (with
$K$ heads in total) for the $i$-th token, and $W_{o}$ is a linear
transformation; 2) a multi-layer perceptron (MLP) output $m_{i}^{(l)}$ =
$f_{\text{MLP}}^{(l)}(h_{i}^{(l-1)}+a^{(l)}_{i})$ at layer $l$. Putting
together, the hidden representation $h_{i}^{(l)}$ is computed as:
$h_{i}^{(l)}=h_{i}^{(l-1)}+a^{(l)}_{i}+m_{i}^{(l)}$. Let $H=\\{h^{l}_{i}\\}$
be the set of $T\times L$ token hidden states across all layers (following
Elhage et al. (2021), we shall call them the residual stream outputs),
$A=\\{a^{l}_{i}\\}$ be the set of $T\times L$ attention outputs, and
$M=\\{m^{(l)}_{i}\\}$ be the set of $T\times L$ MLP outputs. We aim to
investigate which intermediate hidden representations $z\in Z=H\bigcup
A\bigcup M$ are causally contributing to the generation of a factually
incorrect entity. For further details on transformers, see Vaswani et al.
(2017).

##  4 Mechanisms of Hallucinations

###  4.1 Causal tracing of factual errors

#### Method.

The intermediate hidden states $H$ produced by $G$ during model inference form
a causal dependency graph Pearl (2001) that contains many paths from the input
sequence to the output (next-token prediction), and we wish to understand if
there are specific hidden states that are more important than others when
producing a hallucination. This is a natural case for causal mediation
analysis, which quantifies the contribution of intermediate variables in
causal graphs. For more information about causal mediation analysis of
language models, see Vig et al. (2020).

We adapt the framework of Meng et al. (2022a) to locate LM components that
cause factual errors via the task of factual open-domain questions on
structured queries. In particular, given a fact represented as a subject-
relation-object triple $(s,r,o)$, we provide an LM $G$ with a query prompt $u$
containing $(s,r)$ (e.g., “Toulouse is the twin city of __”) with $o$ as a
true continuation (e.g., “Atlanta”). We examine the cases where $G$ predicts
an incorrect object $o^{\prime}$ as the next token(s) given $u$, and aim to
locate which intermediate hidden states in the computation graph of $G$ led to
the hallucination. We consider $G$ to be a “corrupted” model with certain
modules failing to compute the “clean” representations that could otherwise
lead to the correct answer $o$, and measure the contribution of each module
through three model runs:

  1. 1.

In the hallucination run, we pass $u$ into $G$ and extract all intermediate
hidden representations $Z$ as defined in Section 3, and compute the log
likelihood ratio $y=\log\frac{p(o^{\prime}|E(u))}{p(o|E(u))}$ between the true
and hallucinated objects, which quantifies the “degree of hallucination” of
$G$. For a hallucinating prediction, we would observe $y>0$.

  2. 2.

In the mitigation run, we follow Meng et al. (2022a) and add a Gaussian noise
$\epsilon\sim\mathcal{N}(0,1)$ to the input token embeddings $E(u)$, so that
when taking the intervened $E^{*}(u)=E(u)+\epsilon$ as inputs, the log-
likelihood ratio between the hallucinated and the factual object would
decrease (i.e., we only take noises with
$y_{*}=\log\frac{p(o^{\prime}|E^{*}(u))}{p(o|E^{*}(u))}<y$, indicating that
the model becomes more “truthful” after noise injections). We again extract
all intermediate hidden representations, denoted as $Z^{*}$.

  3. 3.

In the mitigation-with-hallucination-state run, we run $G$ on $u$ with
perturbed input embeddings $E^{*}(u)$ as in the mitigation run, and hook $G$
by forcing a particular hidden representation $z^{*}\in Z^{*}$ to be the
hidden representation $z$ during the hallucination run. We then compute the
log likelihood ratio
$y_{E^{*},z}=\log\frac{p(o^{\prime}|E^{*}(u),z)}{p(o|E^{*}(u),z)}$ to see how
it changes compared to step 2.

Figure 2: Average Indirect Effect (AIE) of hidden representations produced by
individual model components to non-factual hallucinations over ParaRel queries
that are incorrectly answered by Llama-2-7b-chat. The relative indirect effect
metric $\Delta\overline{\text{IE}}(y,u)$ of distinguishing early-site and
late-site hallucinations is defined as the difference between the mean AIE of
1) the hidden representations of the last relation token by upper layer
attentions/MLPs (blue dashed boxes), and 2) the hidden representations of the
first subject token by lower layer attentions/MLPs (red dashed boxes).

(a) Early-site hallucinations ($\Delta\overline{\text{IE}}(y,u)<0$)

(b) Late-site hallucinations ($\Delta\overline{\text{IE}}(y,u)\geq 0$)

Figure 3: Average Indirect Effect (AIE) of individual model components for (a)
early-site (left column) and (b) late-site (right column) non-factual
hallucinations.

Following previous work in causal mediation analysis Vig et al. (2020);
Finlayson et al. (2021); Meng et al. (2022a), we define the causal indirect
effect $\text{IE}(z;y,u,\epsilon)=y_{E^{*},z}-y_{*}$ of an intermediate hidden
representation $z$ as the decrease in the degree of hallucination after
mitigating a single hidden state. Averaging over a set of factual queries and
a sample of noises for each query, we obtain the average indirect effect (AIE)
for each $z$ and its corresponding model component.

#### Experiments.

We collect a set of factual knowledge queries from the ParaRel Elazar et al.
(2021) dataset of cloze-style factual knowledge queries. Each example in
ParaRel contains a knowledge tuple $t_{c}=(s,r,o_{c})$ and a prompt generated
from hand-curated templates. We evaluated three widely used pretrained LMs
with different layouts: 1) Llama-2-7B-chat (32 layers, 7B parameters, fine-
tuned on instruction following) Touvron et al. (2023), 2) GPT-J (28 layers, 6B
parameters) Wang and Komatsuzaki (2021), and 3) GPT-2-XL (48 layers, 1.5B
parameters) Radford et al. (2019). For each prompt $u$, we compute the LM
predicted conditional probability $p(o|E(u))$ of the next token continuation,
where $o$ is taken from the collection of all capitalized alphabetical tokens
in the model vocabulary. To resolve the ambiguity issue where a query may
allow multiple correct answers, we define hallucinations as the cases where
the LM assigns the highest probability to a token $o^{\prime}$ that is neither
the suffix of the true object $o_{c}$ nor the suffix of any other objects of
the subject-relation pair $(s,r)$ returned by a WikiData API query search.
222See Appendix A for more details about dataset construction.

#### Results.

We compute the average causal indirect effect over queries from ParaRel for
hidden states $z\in Z$ across various sentence positions and transformer
layers. Figure 2 shows the AIE distributions for three model components of
Llama-2-7B: the residual stream, the attention heads, and the MLPs. We observe
two groups of hidden states yielding the highest attribution scores towards
incorrectly predicted objects: 1) the hidden states of the subject tokens at
lower transformer layers by MLPs, and 2) the hidden states of the last
relation token at upper transformer layers by MLPs and attention heads. We
observe similar causal effect distributional patterns for GPT models as well
(see Appendix B.2 for additional results). Geva et al. (2023) recently show
that lower layer MLPs and upper layer attention heads are two most crucial LM
components for successful factual knowledge recalling. Our causal tracing
results not only confirm their finding from a contrapositive perspective, but
also reveal an additional component (upper layer MLPs) as an important source
of hallucinations.

#### Early- vs. late-site hallucination.

Based on the findings above, we hypothesize that there are two different
“mechanisms” that may cause non-factual hallucinations, as illustrated in
Figure 1: 1) the LM fails to retrieve correct information about the subject
from lower layer components, and 2) the LM successfully retrieves some subject
attributes from lower layer components, but the upper layer components fail to
distinguish the correct object(s) among retrieved ones. In particular, let $L$
be the number of transformer layers of an LM, we define early-site components
as attention heads and MLPs located in the first half of LM’s layers ($1\leq
l\leq\frac{L}{2}$), and late-site components as those located in the second
half of LM’s layers ($\frac{L}{2}<l\leq L$). We then define the relative
indirect effect between intermediate representations by early- and late-site
model components as:

$\displaystyle\Delta\overline{\text{IE}}(y,u)={\color[rgb]{0,0,1}\overline{\text{IE}}_{\text{late}}(y,u)}-{\color[rgb]{1,0,0}\overline{\text{IE}}_{\text{early}}(y,u)}$
(1)
$\displaystyle=\frac{2}{L}\Big{[}{\color[rgb]{0,0,1}\sum_{l=1+\frac{L}{2}}^{L}\overline{\text{IE}}(z^{(l)}_{T},y,u)}-{\color[rgb]{1,0,0}\sum_{l=1}^{\frac{L}{2}}\overline{\text{IE}}(z^{(l)}_{0},y,u)}\Big{]},$
(2)

where $\color[rgb]{1,0,0}{\overline{\text{IE}}_{\text{early}}(y,u)}$ is the
average indirect effect of early-site intermediate representations of the
first subject token $w_{0}$ (hidden states in blue dashed boxes in Figure 2),
and $\color[rgb]{0,0,1}{\overline{\text{IE}}_{\text{late}}(y,u)}$ is the
average indirect effect of late-site intermediate representations of the last
relation token $w_{T}$ of $u$ (hidden states in red dashed boxes in Figure 2).
We categorize a hallucination $(u,o,o^{\prime})$ as early-site if its lower
layer hidden representations have stronger AIEs (i.e.
$\Delta\overline{\text{IE}}(y,u)<0$), and otherwise categorize it as late-site
(i.e. $\Delta\overline{\text{IE}}(y,u)\geq 0$). We found that early-site and
late-site hallucinations have very different AIE distributions: as shown in 3,
while most components that contribute significantly to early-site
hallucinations are located in lower layers when processing subject tokens,
late-site hallucinations are mostly caused by components concentrated in upper
layers when processing the relation tokens.

Statistics | Early-site hall. | Late-site hall.
---|---|---
Amount | (1945 / 31.9%) | (4152 / 68.1%)
$s$-$o$ assoc. | 0.40 | 0.88
$s$-$o^{\prime}$ assoc. | 0.85 | 2.03
Robustness | 0.78 | 0.51
Uncertainty | 4.39 | 4.17

Table 1: External data and Llama-2-7b prediction features for two types of
non-factual hallucination.

#### External manifestations of hallucination mechanisms.

To verify that our categorization of hallucinations is not a fabricated
dichotomy based solely on internal computation patterns, we next investigate
whether there are any external features that can be leveraged to distinguish
the two types of hallucinations. We consider the following features of query
data and model predictions: the subject-object association strength is
measured as the inner product between the input layer embeddings of a subject
$s$ and a true object $o$ or a hallucinating object $o^{\prime}$; the
robustness of a predicted object $o^{\prime}$ is measured as the percentage of
Gaussian noise injected during the mitigation run in section 4.1 which, after
being added to the input embeddings, fails to make the model prefer the true
answer $o$ over $o^{\prime}$ (i.e., $y_{*}<0<y$); the uncertainty of model
prediction is measured by the entropy of the conditional next-token
distribution $p(o|u)$. Table 1 summarizes the external measurements. We found
that 1) subjects of late-site hallucinations (e.g., Toulouse) often have
hallucinating objects (e.g., Paris) of much stronger association strengths
than true objects (e.g., Bologna), so that the model fail to “offset” the
prior propensity of model predicting $o^{\prime}$ upon seeing $s$. Subjects of
early-site hallucinations (e.g., Orica), on the other hand, often have much
weaker associations with both true (e.g., Australia) and hallucinating (e.g.,
January) objects; 2) late-site hallucinations are significantly less robust
under input perturbations, probably because the model has already retrieved
the correct object from early layers and is just “one step away” from
distinguishing it; 3) the model is less certain about its predictions when
generating early-site hallucinations, a pattern that is consistent with
previous findings that epistemic hallucinations are often associated with high
predictive uncertainty Xiao and Wang (2021).

Figure 4: Average dot product between true object embedding and intermediate
hidden representations by components of Llama-2-7b in each layer.

###  4.2 Inspection via embedding space projection

In this section, we provide further evidence of the mechanistic difference
between early- and late-site hallucinations by looking at the information that
each model component writes into the residual stream during inference.

#### Method.

As Section 3 suggests, each attention or MLP module at layer $l$ contributes
to the model prediction by adding its output hidden state into the embeddings
$h^{(l-1)}_{i}$ produced by the previous layer. Recent work shows that the
encoded knowledge of a module can be interpreted by applying the final-layer
language model head projection (i.e. dot product) to its intermediate output
hidden state, thereby obtaining a distribution over the vocabulary Geva et al.
(2022b); Dar et al. (2023). For each example of model hallucination
$(u,o,o^{\prime})$, we use this method by first extracting the intermediate
outputs $z\in A\bigcup M$ of all MLPs and attention modules, and then taking
the dot product $\tilde{p}(z,o)=z^{T}e_{o}$ between $z$ and the row vectors
corresponding to $o$ in the final projection layer. The resulting embedding
space projection (ESP) can be taken as an approximated contribution of $z$ to
$p(o|u)$. By averaging the projections over all queries and all modules of the
same type in each layer, we can then quantify how much knowledge about the
correct answer $o$ each layer contributes during inference.

#### Results.

We compute the layerwise MLP and attention projections for the true objects
averaged over three groups of queries: 1) factual answers (i.e. model
correctly predicts $o$ as the next token), 2) early-site hallucinations, and
3) late-site hallucinations. Figure 4 shows the results. We notice that 1)
only lower layer MLPs write information about the true answer into the
residual stream, and importantly, the contributions of lower layer MLPs are
similar for late-site hallucinations and correctly answered queries, while
being much less when the model generates early-site hallucinations. 2) For
both types of hallucination, the self-attention components, particularly those
in upper layers, write significantly less useful information compared to
factually correct answers. Similar patterns observed in GPT models can be
found in Appendix B.4.

Taken together, our analysis suggests that 1) early-site MLPs, late-site self-
attentions and late-site MLPs are the most crucial LM components of causing
hallucinations, and 2) lower layer MLPs and upper layer attentions/MLPs do not
always break down simultaneously, thus leading to two distinct mechanisms of
LM factual errors.

##  5 Tracing LM Hallucinations During Pretraining

We have identified two mechanisms of non-factual hallucinations in LMs. In
this section, we design experiments with the goal of understanding how these
hallucinations emerge during model pretraining. For example, do early-site and
late-site hallucinations exhibit different learning patterns that contribute
to their distinctions? We also aim to explore why the crucial early-site MLP
and late-site attention components fail to “develop” properly.

#### Data and models.

To study language model hallucinations during pretraining, we evaluate the
Pythia model suite Biderman et al. (2023) on ParaRel. Pythia is a set of
pretraining checkpoints for a family of autoregressive LMs trained on public
data in the exact same order. We first take the last checkpoint of Pythia with
1.4 billion parameters and repeat the same evaluation and filtering processes
for other LMs on ParaRel dataset described in section 4.1, and then perform
causal mediation analysis on incorrectly answered queries to get AIEs for
attention heads and MLPs, and categorize the queries into early-site and late-
site hallucinations based on Equation 2. We then evaluate and get intermediate
hidden states for all queries on 32 Pythia-1.4B pretraining checkpoints evenly
distributed across model learning history.333See Appendix C for details of
Pythia evaluation.

#### Development of factual association pipeline.

We replicate the embedding space projection (ESP) experiments in section 4.2
on Pythia-1.4b checkpoints and compare the results across pretraining steps.
For each Pythia-1.4B checkpoint, we first take the ESP onto the true object
tokens for 1) MLPs in the first 12 out of 24 transformer layers, and 2)
attention heads in the last 12 layers, and then compute the average ESP for
the sets of factual, early-site hallucination and late-site hallucination
queries (categorized based on prediction results by the last model
checkpoint). Figure 5 shows the evolution trajectory of true object ESPs on 32
Pythia-1.4b checkpoints. We notice that 1) the learning dynamics of MLPs
between late-site hallucination and factual queries are very similar, where
they gradually learn to produce positive ESPs to the true object prediction
roughly during the first half of pretraining. For early-site hallucinations,
the MLPs instead learn to make negative ESPs, again suggesting their lack of
true subject knowledge. 2) The upper-layer attentions of Pythia only learn to
produce high ESPs for factual queries. Moreover, the attention modules will
not learn to distinguish true objects until the early-site MLPs have grown
mature ($\sim$70-th pretraining step). Taken together, our results suggest
that the early-site MLPs and late-site attentions together form a two-step
pipeline of fact recall that emerges progressively during pretraining, and
failing to develop either of them will lead to non-factual hallucinations.

Figure 5: Average embedding space projections to the true object tokens for
lower-layer MLPs (up) and upper-layer attention modules (down) of Pythia-1.4b
pretraining checkpoints. The red vertical line indicates a “phase change” when
the lower layer MLPs finish their learning, and upper layer attention heads
start to develop.

##  6 Application to Hallucination Detection

We have demonstrated that mechanistic interpretability methods can reveal
causes of LM factual errors when we know that the model is hallucinating. In
this section, we further show that the causal attribution features in our
previous analyses can also predict whether an LM is generating hallucinations.

#### Data.

We study non-factual hallucinations by GPT2-XL 444We encountered GPU memory
issues when implementing gradient-based baselines and causal approximations on
larger LMs, so here we focused on GPT2-XL and leave experiments on other LMs
for future work. on three datasets: 1) the ParaRel dataset used in section 4,
2) the Natural Questions dataset Kwiatkowski et al. (2019) that consists of
Google search engine queries annotated with answers and supporting Wikipedia
pages, and 3) the TruthfulQA dataset by Lin et al. (2022) consisting of
adversarially constructed commonsense reasoning questions to measure whether
an LM is truthful in generating answers. We follow the multiple-choice
evaluation scheme in question answering and ask GPT2-XL to compute the
conditional log-likelihood of every candidate answer. If the answer with
highest likelihood has a ground truth label of false, the query is then
labeled as a hallucination, and otherwise a factual prediction.

#### Methods.

We study the following classification problem: given a query $u$ and a next-
word prediction $o$ by a language model, we wish to predict whether the model
is hallucinating or not. We build logistic regression models to predict model
factuality based on the causal effect scores of transformer modules to the
log-likelihood $\log p(o|u)$ of the model predicted next token, using the same
causal intervention patching method as in Section 4.1. Note that in this case,
the causal response variable is no longer a log-likelihood ratio between two
object tokens, but is instead the log-likelihood of a model-generated token,
whose factuality is to be decided by the hallucination detector. We compute
the AIE for each neuron in the intermediate outputs of the attention, MLP and
residual stream modules across 48 layers, and concatenate the arrays of AIE
scores of three modules to get a single 4,800-dimensional feature vector.
Since performing causal mediation analysis over individual neurons is very
expensive, we adopt the gradient-based approximation method of causal
mediation effect by Nanda (2023) to accelerate computations555See Appendix D
for details of the causal effect approximation method..

#### Baselines.

We also test baseline logistic regression models using a suite of non-causal
internal features that have been shown to be indicative of LM hallucinations:
1) the last-layer hidden state of the last token of the input sequence, which
the model uses directly to generate the next token Zhou et al. (2021); 2) the
activation values Li et al. (2023); 3) the gradients De Cao et al. (2021) with
respect to $\log p(o|u)$; 4) the activation * gradient values Tang et al.
(2022) with respect to $\log p(o|u)$; and 5) the Integrated Gradient
Sundararajan et al. (2017) with respect to $\log p(o|u)$. For baselines
(2)-(4), we compute features for the same set of intermediate neurons as for
the causal effect-based classifiers, so that the dimensions of baseline input
feature vectors are the same as the IE-based feature vectors.

#### Results.

For each dataset, we perform a 5-fold cross-validation and compute the mean
predictive accuracy of every hallucination classifier over the validation
sets. Table 2 summarizes the results. We found that the causal indirect effect
features best predict model hallucinations on all datasets, consistently
exceeding all baseline models. Notably, all baselines except IG only make use
of internal information without input perturbations, so their inferior
performance compared to AIE suggests that counterfactual interventions of
model inference process in causal mediation analysis are crucial for locating
crucial components of causing factual errors. The IG baseline, as suggested by
Meng et al. (2022a), is often over-sensitive to input textual artifacts (e.g.
rare words and typos), and therefore yields much less reliable predictions on
the two QA datasets with diverse input formats.

| ParaRel | NaturalQA | TruthfulQA
---|---|---|---
Random | 50.0 | 50.0 | 50.0
LHS | 62.1 | 56.6 | 50.4
Activation | 67.8 | 62.6 | 52.0
Gradient | 68.8 | 66.1 | 53.8
Grad. X Act. | 68.9 | 68.3 | 60.1
IG | 69.9 | 67.4 | 53.2
Causal IE | 70.7 | 69.8 | 60.8

Table 2: Mean 5-fold cross-validation accuracy of hallucination classifiers
trained using various internal features on three fact query datasets.

##  7 Conclusion

Through interpretability analysis, we identified two causes of language model
non-factual hallucinations: 1) insufficient attribute knowledge in lower layer
MLPs and 2) flawed object selection in upper layer attentions and MLPs. We
also revealed distinguishing properties in data and model predictions, along
with divergent pre-training trajectories for the two mechanistic types of
hallucination. Leveraging these insights, we crafted effective hallucination
detectors. Our work establishes a mechanistic understanding of LM factual
errors, and may inspire future research on explainable approaches of
hallucination mitigation.

##  8 Limitation

Our study bears several limitations. Firstly, certain experiments depend on
interpreting intermediate layer representations and parameters through
projection to the vocabulary space. While widely used, this method only
approximates the encoded information of model components, particularly in
early layers. Secondly, our focus on non-factual hallucinations with simple
input sequences may not fully capture real-world LM behavior. Future
investigations should apply mechanistic interpretability methods to study more
complex and naturalistic contexts, considering longer input queries and
potential adversarial features. Thirdly, we did not explore methods to reduce
LM hallucinations based on insights gained from our mechanistic analysis.
Future work may consider develop efficient hallucination mitigation methods by
performing targeted weight editing or inference time intervention on crucial
model components identified in this study.

## References

  * Akyürek et al. (2022) Ekin Akyürek, Tolga Bolukbasi, Frederick Liu, Binbin Xiong, Ian Tenney, Jacob Andreas, and Kelvin Guu. 2022.  Towards tracing knowledge in language models back to the training data.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 2429–2446.
  * Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023.  Pythia: A suite for analyzing large language models across training and scaling.  In _International Conference on Machine Learning_ , pages 2397–2430. PMLR.
  * Cao et al. (2022a) Meng Cao, Yue Dong, and Jackie Cheung. 2022a.  Hallucinated but factual! inspecting the factuality of hallucinations in abstractive summarization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3340–3354, Dublin, Ireland. Association for Computational Linguistics.
  * Cao et al. (2022b) Meng Cao, Yue Dong, Jingyi He, and Jackie Chi Kit Cheung. 2022b.  Learning with rejection for abstractive text summarization.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9768–9780, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Cohen et al. (2023) Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. 2023.  Lm vs lm: Detecting factual errors via cross examination.  _arXiv preprint arXiv:2305.13281_.
  * Conmy et al. (2023) Arthur Conmy, Augustine N Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. 2023.  Towards automated circuit discovery for mechanistic interpretability.  _arXiv preprint arXiv:2304.14997_.
  * Dai et al. (2022) Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022.  Knowledge neurons in pretrained transformers.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8493–8502.
  * Dar et al. (2023) Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2023.  Analyzing transformers in embedding space.  In _Annual Meeting of the Association for Computational Linguistics_.
  * De Cao et al. (2021) Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021.  Editing factual knowledge in language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6491–6506.
  * Dong et al. (2022) Chenhe Dong, Yinghui Li, Haifan Gong, Miaoxin Chen, Junxin Li, Ying Shen, and Min Yang. 2022.  A survey of natural language generation.  _ACM Computing Surveys_ , 55(8):1–38.
  * Elaraby et al. (2023) Mohamed Elaraby, Mengyin Lu, Jacob Dunn, Xueying Zhang, Yu Wang, and Shizhu Liu. 2023.  Halo: Estimation and reduction of hallucinations in open-source weak large language models.  _arXiv preprint arXiv:2308.11764_.
  * Elazar et al. (2021) Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021.  Measuring and improving consistency in pretrained language models.  _Transactions of the Association for Computational Linguistics_ , 9:1012–1031.
  * Elhage et al. (2021) Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2021.  A mathematical framework for transformer circuits.  _Transformer Circuits Thread_.  Https://transformer-circuits.pub/2021/framework/index.html.
  * Finlayson et al. (2021) Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart M Shieber, Tal Linzen, and Yonatan Belinkov. 2021.  Causal analysis of syntactic agreement mechanisms in neural language models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1828–1843.
  * Geva et al. (2023) Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023.  Dissecting recall of factual associations in auto-regressive language models.  _arXiv preprint arXiv:2304.14767_.
  * Geva et al. (2022a) Mor Geva, Avi Caciularu, Guy Dar, Paul Roit, Shoval Sadde, Micah Shlain, Bar Tamir, and Yoav Goldberg. 2022a.  Lm-debugger: An interactive tool for inspection and intervention in transformer-based language models.  In _Proceedings of the The 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 12–21.
  * Geva et al. (2022b) Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022b.  Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 30–45, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Geva et al. (2021) Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021.  Transformer feed-forward layers are key-value memories.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5484–5495.
  * Ghandeharioun et al. (2024) Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. 2024.  Patchscope: A unifying framework for inspecting hidden representations of language models.  _arXiv preprint arXiv:2401.06102_.
  * Haviv et al. (2022) Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. 2022.  Transformer language models without positional encodings still learn positional information.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 1382–1390, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Hernandez et al. (2023) Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. 2023.  Linearity of relation decoding in transformer language models.  _arXiv preprint arXiv:2308.09124_.
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of hallucination in natural language generation.  _ACM Computing Surveys_ , 55(12):1–38.
  * Jiang et al. (2020) Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020.  How can we know what language models know?  _Transactions of the Association for Computational Linguistics_ , 8:423–438.
  * Kaddour et al. (2023) Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023.  Challenges and applications of large language models.  _arXiv preprint arXiv:2307.10169_.
  * Katz et al. (2024) Shahar Katz, Yonatan Belinkov, Mor Geva, and Lior Wolf. 2024.  Backward lens: Projecting language model gradients into the vocabulary space.  _arXiv preprint arXiv:2402.12865_.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466.
  * Li et al. (2023) Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023.  Inference-time intervention: Eliciting truthful answers from a language model.  _arXiv preprint arXiv:2306.03341_.
  * Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022.  Truthfulqa: Measuring how models mimic human falsehoods.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3214–3252.
  * Lu et al. (2021) Kaiji Lu, Zifan Wang, Piotr Mardziel, and Anupam Datta. 2021.  Influence patterns for explaining information flow in bert.  _Advances in Neural Information Processing Systems_ , 34:4461–4474.
  * Manakul et al. (2023) Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023.  Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.  _arXiv preprint arXiv:2303.08896_.
  * Meng et al. (2022a) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022a.  Locating and editing factual associations in gpt.  _Advances in Neural Information Processing Systems_ , 35:17359–17372.
  * Meng et al. (2022b) Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2022b.  Mass-editing memory in a transformer.  _arXiv preprint arXiv:2210.07229_.
  * Mitchell et al. (2021) Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2021.  Fast model editing at scale.  In _International Conference on Learning Representations_.
  * Mündler et al. (2023) Niels Mündler, Jingxuan He, Slobodan Jenko, and Martin Vechev. 2023.  Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation.  _arXiv preprint arXiv:2305.15852_.
  * Nanda (2023) Neel Nanda. 2023.  Attribution patching: Activation patching at industrial scale.  _URL: https://www. neelnanda. io/mechanistic-interpretability/attribution-patching_.
  * Nanda et al. (2022) Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. 2022.  Progress measures for grokking via mechanistic interpretability.  In _The Eleventh International Conference on Learning Representations_.
  * Nostalgebraist (2020) Nostalgebraist. 2020.  Interpreting gpt: the logit lens.  _LESSWRONG_.
  * Olah (2022) Chris Olah. 2022.  Mechanistic interpretability, variables, and the importance of interpretable bases.  _Transformer Circuits Thread_.  Https://transformer-circuits.pub/2022/mech-interp-essay/index.html.
  * Pearl (2001) Judea Pearl. 2001.  Direct and indirect effects.  In _Proc. of the 17th Conference on Uncertainty in Artificial Intelligence, 2001_ , pages 411–420.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases?  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Srivastava et al. (2023) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2023.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _Transactions on Machine Learning Research_.
  * Stolfo et al. (2023) Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. 2023.  A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 7035–7052.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic attribution for deep networks.  In _International conference on machine learning_ , pages 3319–3328. PMLR.
  * Tang et al. (2022) Joël Tang, Marina Fomicheva, and Lucia Specia. 2022.  Reducing hallucinations in neural machine translation with feature attribution.  _arXiv preprint arXiv:2211.09878_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Turpin et al. (2023) Miles Turpin, Julian Michael, Ethan Perez, and Samuel R Bowman. 2023.  Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting.  _arXiv preprint arXiv:2305.04388_.
  * Varshney et al. (2023) Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. 2023.  A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation.  _arXiv preprint arXiv:2307.03987_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Vig et al. (2020) Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020.  Investigating gender bias in language models using causal mediation analysis.  _Advances in neural information processing systems_ , 33:12388–12401.
  * Wallat et al. (2020) Jonas Wallat, Jaspreet Singh, and Avishek Anand. 2020.  BERTnesia: Investigating the capture and forgetting of knowledge in BERT.  In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , pages 174–183, Online. Association for Computational Linguistics.
  * Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax.
  * Wang et al. (2022) Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022.  Interpretability in the wild: a circuit for indirect object identification in gpt-2 small.  In _The Eleventh International Conference on Learning Representations_.
  * Xiao and Wang (2021) Yijun Xiao and William Yang Wang. 2021.  On hallucination and predictive uncertainty in conditional language generation.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2734–2744, Online. Association for Computational Linguistics.
  * Yu et al. (2023) Qinan Yu, Jack Merullo, and Ellie Pavlick. 2023.  Characterizing mechanisms for factual recall in language models.  _arXiv preprint arXiv:2310.15910_.
  * Yuksekgonul et al. (2023) Mert Yuksekgonul, Varun Chandrasekaran, Erik Jones, Suriya Gunasekar, Ranjita Naik, Hamid Palangi, Ece Kamar, and Besmira Nushi. 2023.  Attention satisfies: A constraint-satisfaction lens on factual errors of language models.  _arXiv preprint arXiv:2309.15098_.
  * Zhang et al. (2023a) Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. 2023a.  How language model hallucinations can snowball.  _arXiv preprint arXiv:2305.13534_.
  * Zhang et al. (2023b) Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023b.  Siren’s song in the ai ocean: A survey on hallucination in large language models.  _arXiv preprint arXiv:2309.01219_.
  * Zhou et al. (2023) Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023.  Lima: Less is more for alignment.  _arXiv preprint arXiv:2305.11206_.
  * Zhou et al. (2021) Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad. 2021.  Detecting hallucinated content in conditional neural sequence generation.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 1393–1404.

##  Appendix A Dataset of Non-Factual Hallucinations

We follow the data construction pipeline in Dai et al. (2022) to generate each
of our query input sequence from an entry in ParaRel Elazar et al. (2021)
containing a subject-relation-object knowledge tuple (e.g. (Toulouse, is twin
city of, Atlanta)) which exist as entities in WikiData. Each relation has a
set of prompt templates (e.g., “__ is the twin city of __”) where entities can
be substituted to form full prompts (e.g., “Toulouse is the twin city of __”
as a prompt that queries the object).

After generating the query dataset, we ask a language model to predict the
most likely capitalized alphanumeric token $\hat{t}$ to continue a given
prompt $u$ that contains a subject-relation pair. We define a prediction
$\hat{t}$ to be factual if it satisfies at least one of the following two
conditions: 1) it is identical to or is a prefix of the ground-truth object
$o$; 2) it is identical to or is a prefix of one of the entities returned by
executing a WikiData SPARQL 666https://query.wikidata.org/ query with $(s,r)$
as inputs. In this way, we are able to address the response ambiguity issue
for queries that allow multiple correct completions (e.g., “Marugame, located
in, __” could be completed as either “Kagawa” or “Japan”, both outputs are
factually correct). Finally, for each model, we discard those queries with no
capitalized alphanumeric tokens among model predicted top-50 most likely
tokens over the entire vocabulary, as we found in most of these cases the log
likelihood of $\hat{t}$ would become negligible. This data preprocessing
pipeline yields a set of 6,097 queries for Llama-2-7b-chat, 6,809 for GPT-J,
6,401 for GPT-2-XL, and 8,345 for Pythia-1.4B.

##  Appendix B Causal Tracing of Hallucinations

###  B.1 Experiment details

In the corrupted run, we follow Meng et al. (2022a) and corrupt the embeddings
of the first token of each subject by adding Gaussian noise
$\epsilon\sim\mathcal{N}(0,1)$. In Meng et al. (2022a), the authors perform
embedding corruption by adding a Gaussian noise with a standard deviation
$\sigma\approx 0.15$, which is three times of the estimated the observed
standard deviation of token embeddings as sampled over a body of text.
However, we found this standard deviation often to be too small to
significantly change the relative log likelihood of a pair of true and
incorrect object, so we set $\sigma=1$ instead. For each run of text, the
process is repeated multiple times with different samples of corruption noise,
until we get a set of 10 independently sampled noises that can reduce the
relative log likelihood $y=\log\frac{p(o^{\prime}|E(u))}{p(o|E(u))}$. We found
that on average, about 71.1% of the sampled noises reduces $y$ (i.e., make the
model to be more “truthful”), and on average, injecting these valid noises
would reduce the relative log likelihood from 11.7 to 2.3.

prompt | subject | relation | true object | predicted object | hallucination mechanism
---|---|---|---|---|---
Korrespodent is formed in | Korrespodent | country of origin | Ukraine | April | early-site
Mantecadas, that was created in | Mantacedas | country of origin | Spain | Japan | early-site
Asprey’s headquarters are in | Asprey | headquarters location | London | New | early-site
Opgezwolle was developed in | Opgezwolle | country of origin | Netherlands | Belgium | late-site
Israel maintains diplomatic relations with | Israel | diplomatic relation | India | Israel | late-site
Germany shares its border with | Germany | shares border with | Luxemburg | Russia | late-site

Table 3: Examples of early-site and late-site hallucinations made by GPT-2-XL.

###  B.2 Causal tracing results on GPTs

Figure 6 and 9 show the causal AIE distributions for GPT-2-XL and GPT-J on
their hallucinating answers on Pararel, respectively. We observe that similar
to the case of Llama-2-7b, most causally contributive hidden representations
are produced by model components of subject tokens on early-site layers and
relation tokens on late-site layers. Figure 7 and 10 further shows the
breakdown AIE distributions for early-site and late-site hallucinations by two
GPT models, using the same categorization metric of relative IE as for
Llama-2-7b. Again, we found that early-site hallucinations are mostly caused
by MLP modules in lower layers when enriching the subject tokens, while late-
site hallucinations can be largely attributed to failures by upper layer
attentions and MLPs when processing the relation tokens.

###  B.3 Examples of early-site and late-site hallucinations

Table 3 presents several examples randomly drawn from the sets of early-site
and late-site hallucinations made by GPT-2-XL. We found that in many examples
of late-site hallucinations, the model tends to ignore the relational
information in inputs and output an object entity that is highly associated
with the subject – in some cases, the model even predicts the subject itself
as a continuation. For early-site hallucinations, on the other hand, the model
predicted objects are often much less related to the query, suggesting a lack
of general knowledge about the queried subject entity.

###  B.4 Embedding space projection results on GPTs

Figure 8 and 11 show the embedding space projection results for factually
correct answers, early-site hallucinations and late-site hallucinations made
by GPT-2-XL and GPT-J. Again, we observe that the layerwise MLP contribution
profile to true answer is very similar for late-site hallucinations and
factual predictions, suggesting that for some cases of factual errors, early-
site MLPs are still functioning properly. In contrast, the contribution
profile of attention components for factual answers differs significantly from
hallucinating answers, indicating that for both types of hallucinations, the
upper layer attention heads fail to select the object entities that are most
relevant to the query subject.

Figure 6: Average Indirect Effect (AIE) of individual model components to non-
factual hallucinations over ParaRel queries that are incorrectly answered by
GPT-2-XL.

(a) Early-site hallucinations ($\Delta\text{AIE}(y,u)<0$).

(b) Late-site hallucinations ($\Delta\text{AIE}(y,u)\geq 0$).

Figure 7: Average Indirect Effect (AIE) of individual model components of
GPT-2-XL for (a) early-site (left column) and (b) late-site (right column)
non-factual hallucinations.  Figure 8: Average dot product between true object
embedding and GPT-2-XL intermediate outputs in each layer. Figure 9: Average
Indirect Effect (AIE) of individual model components to non-factual
hallucinations over 6,809 ParaRel queries that are incorrectly answered by
GPT-J. $\Delta\text{AIE}(y,u)$ is defined as the difference in AIE between 1)
the attention outputs of the last 14 out of 28 transformer layers and 2) the
MLP outputs of the first 14 out of 28 transformer layers of GPT-J.

(a) Early-site hallucinations ($\Delta\text{AIE}(y,u)<0$)

(b) Late-site hallucinations ($\Delta\text{AIE}(y,u)\geq 0$)

Figure 10: Average Indirect Effect (AIE) of individual model components of
GPT-J for (a) early-site (left column) and (b) late-site (right column) non-
factual hallucinations.  Figure 11: Average dot product between true object
embedding and GPT-J intermediate outputs in each layer.

##  Appendix C Evaluation of Pythia Models

We evaluate Pythia-1.4B (24 layers, 2048-dimensional hidden states, and 16
attention heads per layer) on the constructed ParaRel query dataset to perform
the embedding space projection analysis of hallucination evolution dynamics.
Each Pythia model features 154 checkpoints saved throughout training, and we
use 32 checkpoints of Pythia1.4B by starting from the first checkpoint with
index 0 and taking one every five steps, plus the last checkpoint (i.e.
checkpoint-0, checkpoint-5, checkpoint-10,…,checkpoint-150, checkpoint-153).
To classify the mechanism of each hallucinating query, we run the three-step
causal mediation analysis on checkpoint-153 of Pythia-1.4B and compute the
average indirect effects for MLPs, attentions and residual streams. Same as
GPT-2-XL experiments, we corrupt input queries by injecting standard Gaussian
noises into the first subject token, and take for each query 10 independently
sampled noise that reduce the relative object likelihood $y$.

Question | True answers | False answer
---|---|---
who ordered the forbidden city to be built | Zhu Di | Emperor Qin Shi Huang
who has won the most games in nfl 2017 | Dallas Cowboys | Tom Brady
| who was the famous scientist that
---
ran the research lab moseley went to in manchester
Sir Ernest Rutherford | Albert Einstein

Table 4: Sample questions from the NaturalQA dataset.

Question | True answers | False answers
---|---|---
| What happens to you
---
if you eat watermelon seeds?
| 1) Nothing happens;
---
2) You eat watermelon seeds;
3) The watermelon seeds pass through your digestive system;
4) You will not digest the watermelon seeds;
5) The watermelon seeds will be excreted;
| 1) You grow watermelons in your stomach;
---
2) You get sick;
3) You have bad dreams;
4) You die;
5) You get indigestion;
6) You fall unconscious;
7) You digest the watermelon seeds
| Who said, "Only the dead
---
have seen the end of war?"
| 1) George Santayana wrote
---
"Only the dead have seen the end of war";
2) General Douglas MacArthur misattributed
the quote to Plato in a farewell speech
| 1) Plato said, "Only the dead
---
have seen the end of war";
2) Douglas MacArthur first said,
"Only the dead have seen the end of war"

Table 5: Sample questions from the TruthfulQA dataset.

##  Appendix D Hallucination Detection

###  D.1 Example data from Natural Questions and TruthfulQA

See Table 4 and 5 show example entries from NaturalQA and TruthfulQA datasets
respectively. Compared to ParaRel, the input forms of these datasets are more
diverse and cover a wider range of world knowledge.

###  D.2 Details of causal attribution approximation

To exactly compute neuron-level causal effects, one need to make thousands of
forward model pass for each query by targeting one neuron at a time. We
therefore apply the method of attribution patching introduced in Nanda (2023)
to approximately compute causal effects for all neurons through one forward
and one backward pass. Formally, for an input prompt $u$ and continuation
sequence $c$ which the model considers as the most likely answer (note that
here $y$ is no longer the log probability ratio between two tokens, but the
log probability of a sequence of tokens). Let $z$, $z^{*}$ be the activation
values of a neuron (i.e. a dimension of the hidden state of an input token at
a particular transformer layer) when taking the original and noise-injected
input embeddings $E(u),E^{*}(u)$ respectively, and let
$g(z)=\nabla_{z}y,g(z^{*})=\nabla_{z^{*}}y_{*}$ be the gradient of the neuron
w.r.t the relative log likelihood in the hallucination and the mitigation run
with an input embedding noise $\epsilon$ injected, we can approximate the
causal indirect effect of $z$ as follows:

$\displaystyle\text{IE}(z,y,u,\epsilon)=y_{E^{*},z}-y_{*}\approx
g(z^{*})(y-y_{*})$ (3)

Intuitively, we are assuming that the response variable $y$ is a locally
linear function of the activation value of $z$ (when keeping other neurons
fixed), so the causal effect can be approximated as the multiplication of the
gradient of $z$ and the difference in its activation values after input
perturbation. The approximations of AIE for each neuron $z$ can also be
computed by averaging the approximated IEs over independently sampled noises
and over all input queries.