# Most claimed statistical findings in cross-sectional return predictability
are likely true

Andrew Y. Chen
Federal Reserve Board

(March 2024††thanks: First posted to SSRN: August 27, 2021. More recent
drafts: https://arxiv.org/abs/2206.15365. Replication code:
https://github.com/chenandrewy/mostly-true. I thank Antonio Gil de Rubio Cruz
and Rebecca John for excellent research assistance. For helpful comments, I
thank an anonymous referee, Carter Davis (discussant), Wayne Ferson, Dalida
Kadyrzhanova, Amit Goyal (discussant), Dino Palazzo, Jeffrey Pontiff, Russ
Wermers, Dacheng Xiu (discussant), Min Zhu and seminar participants at the
2022 ASU Sonoran Winter Finance Conference, Boston College, the Future of
Financial Information Webinar, 2024 Midwest Finance Association Meetings, the
University of Nebraska, UC Riverside, and the University of Maryland. I am
grateful to Sterling Yan and Lingling Zheng for sharing their data, which made
the first draft possible. The views expressed herein are those of the authors
and do not necessarily reflect the position of the Board of Governors of the
Federal Reserve or the Federal Reserve System. )

###### Abstract

I develop simple and intuitive bounds for the false discovery rate (FDR) in
cross-sectional return predictability publications. The bounds can be
calculated by plugging in summary statistics from previous papers and reliably
bound the FDR in simulations that closely mimic cross-predictor correlations.
Most bounds find that at least 75% of findings are true. The tightest bound
finds at least 91% of findings are true. Surprisingly, the estimates in
Harvey, Liu, and Zhu (2016) imply a similar FDR. I explain how Harvey et al.’s
conclusion that most findings are false stems from equating “false” and
“insignificant.”

JEL Classification: G0, G1, C1

Keywords: stock market predictability, stock market anomalies, p-hacking,
multiple testing

##  Introduction

Researchers have discovered hundreds of cross-sectional stock return
predictors ([18]; [12]). This abundance of discoveries has led to concerns
about multiple testing, p-hacking, and a debate about the veracity of this
literature (e.g. [21]; [10]).

This paper provides simple arguments for why most statistical findings in this
literature are true. The arguments build on Benjamini and Hochberg’s
benjamini1995controlling insight that the false discovery rate (FDR) can be
bounded using worst-case scenarios. Worst-case scenarios for publication bias
can be formed from data-mining experiments ([39]) or conservative
extrapolations of published results. Plugging these scenarios into a
simplified version of the [32] estimator leads to FDR bounds that are
tractable yet valid under publication bias.

These bounds are so tractable that they can be evaluated by just plugging in
summary statistics from data mining and predictor zoo studies. Using numbers
reported in eight studies, performed by seven independent teams, I find that
at least 75% of claimed statistical findings are true.

I provide two derivations of these bounds. The first is a slight modification
of Bayes rule and requires just a few lines of math. The second is a visual
argument that does not require any equations at all—though it can be formally
justified as a variant of Storey’s storey2002direct Algorithm 1. Applied to
Chen, Lopez-Lira, and Zimmermann’s chen2022peer (CLZ’s) 29,000 data-mined
long-short strategies, the visual argument leads to a much tighter bound on
the FDR, and finds at least 91% of claimed findings are true.

The 75% and 91% bounds focus on equal-weighted predictability tests, matching
the modal test in the cross-sectional literature ([18]; [30]). However, value-
weighted tests also imply that most findings are true. For example, applying
the visual argument to the CLZ data implies that at least 82% of claimed
findings regarding value-weighted CAPM alphas are true.

My FDR bounds are much simpler than the formulas that appear in the finance
literature (e.g. [21]). Despite their simplicity, they are equivalent to the
more complicated formulas under weak dependence ([36]). To verify this
equivalence, I run estimations on simulated data that bootstrap from CLZ’s
data-mined returns. These simulations demonstrate that my formulas reliably
bound the FDR in a setting that mimics cross-predictor correlations.

My results appear to conflict with [21] (HLZ), who “argue that most claimed
research findings in financial economics are likely false.” I show my FDR
estimates are quantitatively consistent with HLZ’s. HLZ’s main figure implies
an FDR of at most 35% and their SMM estimates imply an FDR of 9%. Thus, the
conflict is entirely about interpretation. I explain how HLZ’s interpretation
equates “insignificant factors” with “false factors,” deviating from the
traditional separation of significance and truth ([37]). I illustrate how
following HLZ’s interpretation leads to both non-economic conclusions and
difficult questions about the scientific process.

This paper adds to the literature on multiple testing problems in cross-
sectional predictability. Papers that find major problems include [21, 19, 29,
13]. Papers that find the opposite include [30, 24, 10, 7, 25]. Among these
papers, mine is unique in developing simple formulas that bound the FDR.111[6]
uses simple formulas but they cannot estimate the FDR or FDR bounds. Simple
formulas allow me to explore a more diverse range of data and demonstrate
broader robustness compared to previous studies. They also provide a new level
of transparency.

Transparency is important given the lack of consensus and complicated methods
in the literature. Harvey, Liu, and Zhu’s harvey2016and methods require so
much exposition that their estimates that account for publication bias are not
found until the 30th page of the article. Chordia, Goyal, and Saretto’s
chordia2020anomalies analogous estimates are not found until page 21. Jensen,
Kelly, and Pedersen’s jensen2023there are not found until page 37\. These
complications are understandable. Publication bias is a slippery topic and
accounting for it requires novel methods and thus convincing justifications.
These complications, however, make the debate difficult to resolve. Debating
any of these estimates requires, in principle, addressing dozens of pages of
motivation and methodology.

My formulas simplify the debate. My “easy bound” for the FDR is on page 4. FDR
estimates that are valid under publication bias begin on page 5.

An important caveat is that I only examine predictability findings in the
cross-sectional literature. I do not examine non-tradable factors, which are
typically not shown to predict returns ([12]). I also do not assess whether
the claims made in the texts are true. Re-evaluating the claims in the texts
typically requires structural modeling ([27]), though [8] propose a test of
textual claims using out-of-sample returns. I also do not examine whether
predictability is stable over time. [9] find that published strategies are not
profitable in recent years after accounting for bid-ask spreads.

##  Why Most Claimed Statistical Findings Are True

Define predictor $i$ as a discovery if its absolute t-stat $|t_{i}|$ exceeds
2.0. As long as $|t_{1}|,|t_{2}|,...,|t_{N}|$ are well-behaved, the [4] FDR is
approximately

$\displaystyle\text{FDR}_{|t|>2}$
$\displaystyle\approx\Pr\left(F_{i}||t_{i}|>2\right),$ (1)

where $i$ is a randomly selected predictor and $F_{i}$ is the event that
predictor $i$ is false ([32]; [36]). In words, the FDR is the probability that
a predictor is false, given that the predictor is a discovery. I explain
Storey’s approximation in Section 4.1. Nevertheless, Equation (1) is so
natural that one might as well define the FDR this way, as is done in [14].

I use the terms “discovery,” “significant predictor,” and “finding”
interchangeably. I also use the terms “false predictor” and “null predictor”
interchangeably. This terminology follows [31] and [4].

Applying Bayes rule and noting $\Pr\left(F_{i}\right)\leq 1$, leads to an
“easy bound” on $\text{FDR}_{|t|>2}$:

$\displaystyle\text{FDR}_{|t|>2}$
$\displaystyle\approx\frac{\Pr\left(|t_{i}|>2|F_{i}\right)}{\Pr\left(|t_{i}|>2\right)}\Pr\left(F_{i}\right)$
(2)
$\displaystyle\leq\frac{\Pr\left(|t_{i}|>2|F_{i}\right)}{\Pr\left(|t_{i}|>2\right)}$
(3) $\displaystyle\approx\frac{5\%}{\Pr\left(|t_{i}|>2\right)},$ (4)

where the third line plugs in the definition of the p-value and thus assumes
that $t_{i}$ is computed correctly. [6] uses a bootstrap to show that $t_{i}$
is computed correctly for standard long-short return tests.222This result is
natural, as simple long-short predictability tests are not subject to the
pitfalls that affect other asset pricing pricing tests (e.g. [26, 28, 17]). I
also verify Equation (4) using bootstraps in Section 4.2.

Equation (4) provides a neat interpretation of FDR methods. FDR methods just
compare the tail area under the null (numerator) to the actual tail
(denominator). If the tail under the null is much smaller, then the null
distribution cannot explain the actual tail, and the FDR must be small.

The natural way to estimate the actual tail is to plug in the empirical tail.
In other words, one might estimate $\Pr\left(|t_{i}|>2\right)$ by just
counting the share of $|t_{i}|$ that exceeds 2. However, since predictors with
$|t_{i}|>2$ are more likely to be published (publication bias), this estimate
would overstate $\Pr\left(|t_{i}|>2\right)$. This problem can be addressed,
however, by considering worst-case scenarios.

###  Bounding the FDR with Atheoretical Data Mining.

One worst-case scenario comes from atheoretical data mining, as in [39]; [13];
and [8] (CLZ). These studies generate tens of thousands or even millions of
long-short strategies by systematically searching accounting data. I focus on
the CLZ strategies because the data and code are publicly available. I use
equal-weighted strategies, to match the modal test in the cross-sectional
predictability literature ([18]; [30]).

The distribution of $|t_{i}|$ from CLZ’s data-mining exercise is shown in
Figure 1. $33\%$ of these strategies have $|t_{i}|>2$. And if atheoretical
data mining can generate significant t-stats so frequently, then researchers
armed with theory should be able to find significance at _at least_ the same
rate—that is, $\Pr\left(|t_{i}|>2\right)\geq 0.33$.

Figure 1: Bounding the FDR using Data-Mining as a Worst-Case

$t_{i}$’s test the null that the mean equal-weighted long-short decile return
is zero for 29,000 accounting ratios based on 240 accounting variables, mined
by [8]. 33% of data-mined strategies have $|t_{i}|>2$, which is a worst-case
scenario for the share of $|t_{i}|>2$ produced by the research process. The
easy FDR bound (Equation (4)) then implies least 85% of claimed research
findings are true.

Plugging this inequality into Equation (4) leads to

$\displaystyle\text{FDR}_{|t|>2}$ $\displaystyle\leq\frac{5\%}{0.33}=15\%.$
(5)

Thus, at least 85% of published cross-sectional predictors are true.

This FDR bound can be calculated by just plugging in summary statistics from
data mining studies. [39]; [13]; and CLZ _all_ report statistics implying that
at least 20% of data-mined strategies have $|t_{i}|>2$. This robustness is
notable, given that each paper uses a different approach for data mining, as
seen in Table 1. Further details on these papers and their implied FDRs are in
Appendix A.1.

Plugging $\Pr\left(|t_{i}|>2\right)=0.20$ into Equation (4) implies
$\text{FDR}_{|t|>2}\leq 5\%/0.20=25\%$ across many specifications and three
independent studies.

Table 1: FDR Bounds using Summary Stats from Data-Mining Studies

$\Pr\left(|t_{i}|>2\right)$ is the share of predictors with $|t_{i}|>2$,
generated from the “Data Mining Method.” Equal-weighted results are selected
for comparability with the cross-sectional literature ([18]). Chordia, Goyal,
and Saretto’s chordia2020anomalies $t_{i}$ largely use value-weighted returns,
though they also examine FM regressions with many controls. Data mining
studies consistently find $\Pr\left(|t_{i}|>2\right)\geq 0.20$, implying
$\text{FDR}_{|t|>2}\leq 5\%/0.20=25\%$.

Paper | Data Mining Method | Exhibit | $\Pr(|t_{i}|>2)$
---|---|---|---
Yan and Zheng | Begin with 240 accounting variables |  Table 1 | at least 20%
(2017) | apply 76 simple transformations, |  Equal- |
| yielding 18,000 signals. |  Weighted |
| Begin with 242 accounting |  Table 4 |
Chen, Lopez-Lira, | variables, apply all possible ratios | Panel (a) | at least 20%
Zimmermann (2023) |  and $\Delta x_{1}/\text{lag}(x_{2})$, | Equal- |
| yielding 29,000 signals. |  Weighted |
| Begin with 185 accounting variables, |  |
Chordia, Goyal, | apply growth rates, ratios, and all |  Table A2 | at least 20%
Saretto (2020) |  combinations of $(x_{1}-x_{2})/x_{3}$, |  | in 12 out of 13
| yielding 2.4 million signals. |  | estimates

###  Bounding the FDR with Conservative Extrapolation

A second worst-case scenario can be constructed by conservative extrapolation.
Figure 2 illustrates. It plots the distribution of t-stats from Chen and
Zimmermann’s ChenZimmermann2021 open-source replications of 200 published
predictors (bars) and overlays an exponential density with a mean of 2.0
(line). The plot shows that this exponential fits the observed region of the
plot well.

Figure 2: Bounding the FDR using Conservative Extrapolation as a Worst Case

$t_{i}$’s test the null that the mean long-short return is zero in
replications of 200 published predictors from [12]. The line extrapolates
using an exponential distribution with a mean of 2.0. The extrapolation is
conservative: it assumes there are exponentially more file drawer t-stats than
published t-stats. Using Equation (4), this extrapolation finds at least 86%
of findings are true.

In the unobserved region, the exponential extrapolates conservatively. A more
moderate extrapolation might assume that the density peaks somewhere between 2
and zero, as it does with most distributions used in finance (e.g. log-
normal). Instead, this extrapolation keeps increasing—it assumes there are
exponentially more file drawer t-stats compared to published t-stats.
Conservatism is also seen by comparing with the data-mined distribution in
Figure 1. To the left of 2.0, the two distributions are broadly similar,
implying that the extrapolation is similar to the worst case of atheoretical
data-mining.

Plugging this exponential CDF into Equation (4) then bounds the FDR:

$\displaystyle\text{FDR}_{|t|>2}$
$\displaystyle\leq\frac{5\%}{\exp\left(-2/E\left(|t_{i}|\right)\right)}=5\%\exp\left(1\right)=13.6\%.$
(6)

Thus, a very different estimate also finds that at least 85% of claimed
findings are true.

A beautiful part of this FDR bound is that it can be calculated by hand, using
summary statistics reported in published papers. The memoryless property of
the exponential distribution implies

$\displaystyle E\left(|t_{i}|\right)$
$\displaystyle=E\left(|t_{i}|\left||t_{i}|>2\right.\right)-2,$ (7)

that is, the mean t-stat can be estimated by just subtracting 2 from the mean
published t-stat, which is close to
$E\left(|t_{i}|\left||t_{i}|>2\right.\right)$. Thus, one can bound the FDR in
the presence of publication bias, by just plugging in summary statistics.

Table 2 uses Equation (7) to bound $\text{FDR}_{|t|>2}$ in a variety of
“predictor zoo” studies. For example, Chen and Zimmermann’s
chen2020publication Table 2 reports that the mean hand-collected t-stat across
77 published predictors is 4.6. Equation (7) then implies that
$E\left(|t_{i}|\right)\approx 4.6-2=2.6$, and so Equation (6) implies
$\text{FDR}_{|t|>2}\leq 5\%\exp\left(2/2.6\right)=11\%$. Replications and
post-publication data tend to result in lower t-statistics, leading to higher
FDR bounds from [30] and [24].333[12] show that replicated t-statistics are as
large as hand-collected ones, if the replications carefully follow the
original methods. Nevertheless, the results across five independent teams
using various data collection methods are consistent. All imply that most
claimed findings are true.

Table 2: FDR Bounds using Summary Stats from Predictor Zoo Studies

Table reports the mean published t-stat from various papers and the resulting
$\text{FDR}_{|t|>2}$ bound (Equations (6)-(7)). t-stats use equal-weighting
with the exception of [21], which does not specify the weighting. These easy
to use bounds consistently imply that most claimed findings are true.

Paper | Exhibit | Method | Mean Pub | $\text{FDR}_{|t|>2}$
---|---|---|---|---
|  |  | t-stat | bound (%)
Green, Hand, | Tables 3 | Hand | 5.1 | 10
Zhang (2013) | and 4 | Collected |  |
Chen, Zimm- | Table 2 | Hand | 4.6 | 11
ermann (2020) |  | Collected |  |
Harvey, Liu, | Figure A.1 | Hand | 4.2 | 12
Zhu (2016) | Panel B | Collected |  |
McLean, | Page 16 , | Replicated | 3.6 | 18
Pontiff (2016) | Par 1 | In-Sample |  |
Jacobs, | Online App | Replicated | 3.1 | 32
Muller (2020) | Table 2 | Full-Sample |  |

The entry for [21] (HLZ) is notable because HLZ argue “that most claimed
research findings in financial economics are likely false.” Table 2 shows that
the difference in conclusions does not arise from a difference in the
underlying data. In fact, even the methods are similar. HLZ’s Figure A.1 also
suggests an exponential extrapolation, the same extrapolation used in Table 2.
And while this figure is buried in the appendix of HLZ, the same exponential
extrapolation is featured in Figure 1 of [19]. We will see that HLZ’s own FDR
estimates are consistent with $\text{FDR}_{|t|>2}\leq 12\%$ (Section 5.1) and
that the difference in conclusions comes from different interpretations of
“false discovery” (Section 5.2).

For further details on the papers and exhibits in Table 2, see Appendix A.2.

In summary, the FDR is small because the research process readily generates
large t-stats, much more readily than compared to a standard normal
distribution. This property can be seen in the frequency of large t-stats in
atheoretical data-mining studies or in conservative extrapolations of the
large t-stats of published strategies. Similar results have been reported in
previous studies, but these often involve structural estimation (e.g. [10]).
In contrast, my FDR bounds so simple they can be estimated by just plugging in
numbers from summary statistic tables.

##  More Refined Estimates

One can argue that the easy FDR bounds are too conservative. Rather than
estimate $\Pr\left(F_{i}\right)$, they effectively assume that
$\Pr\left(F_{i}\right)=1.0$. But one can also argue that the previous bounds
are too aggressive. They focus on raw equal-weighted returns, rather than
value-weighted returns.

This section addresses both issues. Section 3.1 provides a visual approach to
bounding $\Pr\left(F_{i}\right)$, following [32]. Section 3.2 applies this
method to value-weighted returns adjusted for various factor exposures.

###  Bounding the FDR Visually, Following [32]

Panel (a) of Figure 3 illustrates why the easy bounds are too conservative. It
visualizes Equation (4) by plotting the histogram of $|t_{i}|$ in the CLZ data
(gray) against the null (red). Both distributions are scaled to have the same
mass, so the red area to the right of the $|t_{i}|=2$ line is about
$29,000\times\Pr\left(|t_{i}|>2|F_{i}\right)$ and the gray area is about
$29,000\times\Pr\left(|t_{i}|>2\right)$. The ratio, then, is precisely the
easy FDR bound (Equation (4)). This decomposition implies that, for
$|t_{i}|>2$, at most 15% are false.

Figure 3: Bounding the FDR Visually, Following [32]

$|t_{i}|$ come from 29,000 equal-weighted long-short strategies from [8].
Panels decompose the empirical histogram (gray) into an upper bound on false
predictors (red) and lower bound on true predictors (gray minus red)
components. The false component is a scaled standard normal. Panel (a) scales
to visualize the easy FDR bound (Equation (4)). Panel (b) scales so that the
false component is 100% in the $|t_{i}|\in[0,0.5]$ bin. The easy bound is
unreasonably conservative. A reasonably conservative decomposition implies
$\text{FDR}_{|t|>2}\leq 9\%$, as seen in the ratio of red to gray areas to the
right of the vertical line in Panel (b).

(a) Easy Bound

(b) Storey Bound

This visualization shows how FDR methods are just decompositions of the
empirical distribution into false and true components. For every bin, the
ratio of the red to gray area is exactly the easy FDR bound (Equation (3)),
adapted to that bin. But then Panel (a) shows that the implied bounds are
unreasonably conservative. For the first bin ($|t_{i}|\in[0,0.5]$), this
decomposition bounds the number of false strategies at around 11,000, far
higher than the 6,000 or so strategies in the data. Equivalently, it bounds
the number of true strategies at negative 5,000 or so. Surely, one can find
more informed bounds.

Panel (b) of Figure 3 provides more informed bounds. As before, this panel
decomposes the empirical distribution into false and true components, but now
the false component is scaled so that, in the first bin, the bound on false
strategies is equal to the total number of strategies (about 6,000). This
scaling is as conservative as is reasonable. Bounding the FDR at a level
higher than 100%, or bounding true discoveries at a negative number, does not
make much sense.

It turns out that fitting the FDR bound to 100% in the first bin requires the
null to be 56% as large as the data. With this scaling, Panel (b) shows that
$\text{FDR}_{|t|>2}\leq 9\%$, as seen in the ratio of the red to gray areas to
the right of the “Discoveries” line. In other words, at least 91% of findings
are true.

####  Equivalence to [32]

The visual argument can be formalized by writing out the probability that
$|t_{i}|\in[a,b]$ using the law of total probability:

$\displaystyle\Pr\left(|t_{i}|\in[a,b]\right)$
$\displaystyle=\Pr\left(|t_{i}|\in[a,b]|F_{i}\right)\Pr\left(F_{i}\right)+\Pr\left(|t_{i}|\in[a,b]|T_{i}\right)\Pr\left(T_{i}\right)$
(8)

where $T_{i}$ is the event that $i$ is a true predictor. Equation (8) shows
how FDR methods decompose the empirical histogram (LHS) into false and true
components (RHS).

The question is how to determine the false component scaling,
$\Pr\left(F_{i}\right)$. The easy bound avoids any further reasoning and just
uses $\Pr\left(F_{i}\right)\leq 100\%$. But one can apply some reasoning by
solving for $\Pr\left(F_{i}\right)$ in Equation (8) and noting that
$\Pr\left(|t_{i}|\in[a,b]|T_{i}\right)\Pr\left(T_{i}\right)\geq 0$:

$\displaystyle\Pr\left(F_{i}\right)$
$\displaystyle=\frac{\Pr\left(|t_{i}|\in[a,b]\right)-\Pr\left(|t_{i}|\in[a,b]|T_{i}\right)\Pr\left(T_{i}\right)}{\Pr\left(|t_{i}|\in[a,b]|F_{i}\right)}\leq\frac{\Pr\left(|t_{i}|\in[a,b]\right)}{\Pr\left(|t_{i}|\in[a,b]|F_{i}\right)}.$
(9)

Equation (9) is equivalent to Storey’s Equation (6), though Equation (9) is
perhaps more intuitive.444Instead of picking a bin for a t-stat, Storey
chooses a minimum p-value called $\lambda$. In this notation, the numerator of
the right-most expression in Equation (9) can be written as
$\Pr\left(p_{i}>\lambda\right)$, which has the natural estimator
$\\#\left\\{p_{i}>\lambda\right\\}/N$. The denominator the right-most
expression can be written as $\Pr\left(p_{i}>\lambda|F_{i}\right)=1-\lambda$,
since p-values are uniform under the null. Dividing the numerator by the
denominator leads to Storey’s Equation (6).

Choosing $[a,b]=[0,0.5]$, Figure 3 implies
$\Pr\left(|t_{i}|\in[0,0.5]\right)\approx 6,300/29,300=21.5\%$.
$\Pr\left(|t_{i}|\in[0,0.5]|F_{i}\right)$ is the classical “half sigma”
probability: 38.3%. Combining, we have

$\displaystyle\Pr\left(F_{i}\right)$
$\displaystyle\leq\frac{\Pr\left(|t_{i}|\in[0,0.5]\right)}{\Pr\left(|t_{i}|\in[0,0.5]|F_{i}\right)}\approx\frac{0.215}{0.383}=0.56,$
(10)

showing that at most 56% of the 29,300 CLZ strategies are false.

Finally, plugging Equation (10) into Equation (2), we have the same FDR bound
from the visual argument

$\displaystyle\text{FDR}_{|t|>2}$
$\displaystyle\leq\frac{\Pr\left(|t_{i}|>2|F_{i}\right)}{\Pr\left(|t_{i}|>2\right)}\frac{\Pr\left(\in[0,0.5]\right)}{\Pr\left(\in[0,0.5]|F_{i}\right)}\leq\frac{5\%}{0.33}\left(0.56\right)=9\%,$
(11)

but derived from first principles.

The FDR bound of 9% is very close to the empirical Bayes shrinkage estimates
of around 10% found in [10] and [25]. It is also comparable to the point
estimates found in [21] (see Section 5.1). I provide simulation evidence
supporting the validity of this bound in Section 4.2.

###  Adjustments for Liquidity and Factor Exposure

My previous estimates focus on equal-weighted portfolios, due to the focus on
equal-weighted portfolios in the cross-sectional literature. But about half of
the papers also study value-weighted portfolios ([18]), which can be though of
as a simple liquidity adjustment. Similarly, my previous results focus on raw
long-short returns, for simplicity. The literature also examines returns
adjusted for various factor exposures.

Table 3 examines how these adjustments affect FDR bounds. The first four
columns examine adjust equal-weighted returns for exposure to the CAPM, Fama-
French 3, and Fama-French 3 + momentum factors. These adjustments have
relatively little impact on the share of $|t_{i}|>2$. Regardless of the factor
adjustment, this share is roughly $1/3$, as it is in the raw returns. Thus,
the $\text{FDR}_{|t|>2}$ bounds are little changed, and lie around 9 percent
(or a bit below).

Table 3: FDR Bounds with Liquidity and Factor Adjustments

Data consists of 29,000 long-short decile strategies constructed by CLZ.
1-Factor, 3-Factor, and 4-Factor models adjust for CAPM, FF3, and FF3+Momentum
exposure. $\Pr\left(|t|>2\right)$ and $\Pr\left(|t|\leq 1\right)$ are the
share of $|t_{i}|>2$ and $|t_{i}|\leq 1$, respectively. $\text{FDR}_{|t|>2}$
max Easy is $5\%/\Pr\left(|t|>2\right)$ (Equation (4)). $\Pr\left(F\right)$
max is $\Pr\left(|t|\leq 0.5\right)/0.38$ where 0.38 is the share implied by
the null (Equation (10)). $\text{FDR}_{|t|>2}$ max Storey multiplies
$\text{FDR}_{|t|>2}$ max Easy by $\Pr\left(F\right)$ max. After adjusting for
liquidity and factor exposure, most statistical findings are still true.

| Equal-Weighted |  | Value-Weighted
---|---|---|---
| Raw | CAPM | FF3 | 4-Fac |  | Raw | CAPM | FF3 | 4-Fac
$\Pr(|t|>2)$ | 32.5 | 35.9 | 37.3 | 32.4 |  | 10.7 | 17.1 | 18.6 | 13.5
$\text{FDR}_{|t|>2}$ max Easy | 15.4 | 13.9 | 13.4 | 15.4 |  | 46.7 | 29.2 | 26.9 | 37.2
$\Pr(|t|\leq 0.5)$ | 21.6 | 19.7 | 18.8 | 20.7 |  | 33.4 | 28.5 | 27.9 | 31.8
$\Pr(F)$ max | 56.3 | 51.4 | 49.0 | 54.0 |  | 87.2 | 74.3 | 72.8 | 83.1
$\text{FDR}_{|t|>2}$ max Storey | 8.7 | 7.2 | 6.6 | 8.3 |  | 40.8 | 21.7 | 19.6 | 30.9

Columns 4-6 examine value-weighted returns. This adjustment does make a
difference. The share of $|t_{i}|>2$ is much smaller, at around 16%, as one
might expect from the many papers that document how predictability is
concentrated in small, illiquid stocks (e.g [9]). Thus, the easy FDR bound is
larger, at around $0.05/0.16=0.30$. Conversely, the share of $|t_{i}|<0.5$ is
much larger than in the equal-weighted case, leading to a larger bound on
$\Pr\left(F_{i}\right)$ of around $0.80$. But in the end, $\text{FDR}_{|t|>2}$
represents only a minority of the data. Indeed, using the Storey bound and
either the CAPM or FF3 model leads to $\text{FDR}_{|t|>2}$ of at most 21%.

##  More Rigorous Justifications

The FDR formulas in Sections 2 and 3 are much simpler than those found in the
finance literature (e.g. [21]). This section shows that the simple formulas
are equivalent and provides simulation evidence of their validity.

###  Equivalence of FDR Definitions

HLZ’s definition of the FDR looks more complicated than Equation (1).
Following [4], HLZ define the FDR using:

FDR $\displaystyle=E\left[\text{FDP}\right]$ (12)

where FDP is the false discovery proportion, which in turn is defined as

FDP $\displaystyle=\begin{cases}\frac{N_{0|r}}{R}&\text{if }R>0\\\ 0&\text{if
}R=0\end{cases}$ (13)

and where $N_{0|r}$ and $R$ are the number of false discoveries and the number
of discoveries, respectively. Implicit in Equations (12)-(13) is some
definition of a discovery (e.g. predictor $i$ is a discovery if $|t_{i}|>2$).

As shown by [36] and [33], Equations (12) and (13) are equivalent to the
simpler Equation (1) under weak dependence. To see the equivalence, assume
that the following weak law of large numbers (WLLN) holds: as the number of
predictors $\longrightarrow\infty$,

$\displaystyle\frac{\text{number of false discoveries}}{\text{number of
discoveries}}$
$\displaystyle\overset{p}{\longrightarrow}\Pr\left(F_{i}|i\text{ is a
discovery}\right).$ (14)

Then, assuming a positive number of discoveries, we can plug everything in and
simplify:555The limit of the expectation in Equation (15) can be handled using
the Portmanteau theorem.

FDR $\displaystyle=E\left[\frac{\text{number of false
discoveries}}{\text{number of discoveries}}\right]$ (15)
$\displaystyle\overset{p}{\longrightarrow}\Pr\left(F_{i}|i\text{ is a
discovery}\right),$ (16)

as the number of predictors $\longrightarrow\infty$. Both of these expressions
are intuitive notions for the FDR. Equation (15) says the FDR is the expected
share of discoveries that are false. Equation (16) says the FDR is the
probability that a discovery is false. Finally, defining a discovery as
predictors with $|t_{i}|>2$ leads to Equation (1). This derivation is a
simplified version of Storey and Tibshirani’s storey2001estimating Theorem 2
or Storey’s storey2003positive Theorem 4.

The key assumption is the WLLN, Equation (14). It amounts to assuming that the
method of moments works for probabilities. It is analogous to assuming that if
you keep counting the share of economists who study finance, you will
eventually recover the probability a randomly chosen economist studies
finance.

This WLLN can formalized by placing weak cross-predictor dependence conditions
on the underlying long-short returns (a la [38]). The data on long-short
returns suggest weak dependence: cross-predictor correlations cluster close to
zero ([12]). Section 4.2 examines this assumption using bootstrapped
simulations.

The statistics literature features many other notions for the FDR. In addition
to the [4] FDR, there is the “positive false discovery rate” (pFDR) and the
“marginal false discovery rate,” (mFDR) ([34]). There is also the “local fdr”
(a.k.a. $\text{fdr}\left(z\right)$) and the tail probability counterpart
(a.k.a. $\text{Fdr}\left(z\right)$) ([14]). All of these notions of the FDR
are equivalent under the weak dependence in large sample settings ([2]).

###  Estimations on Data Mining Simulations

The FDR bounds in Section 2 and 3 can also be justified using simulations.
This section simulates the data-mined returns. For simulations of published
returns see Appendix B.2.

I simulate the long short return for predictor $i$ in month $\tau$ using

$\displaystyle r_{i,\tau}$
$\displaystyle=\mu_{i}+\varepsilon_{i,k\left(\tau\right)}$ (17)

where $\mu_{i}$ is the expected return, $k\left(\tau\right)$ is a random
integer that depends on $\tau$, and $\varepsilon_{i,k}$ is the de-meaned long-
short return for predictor $i$ in month $k$ from the CLZ dataset. In other
words, $\varepsilon_{i,k}$ is cluster-bootstrapped from the CLZ data in a way
that ensures returns from the same month are drawn together. As a result, the
simulated returns closely match the cross-predictor correlation structure
(Figure A.4). For further details on the bootstrap, see Appendix B.

Expected returns follow

$\displaystyle\mu_{i}$ $\displaystyle=\begin{cases}0&\text{if }F_{i}\\\
\gamma&\text{if }T_{i}\end{cases},$ (18)

where $\gamma$ is a constant and $T_{i}$ is the event predictor $i$ is true.
Equation (18) is the simplest structure that can underlie the FDR framework
(Equations (12) and (13)). Each simulation has 29,000 predictors and 500
months. For each predictor I calculate t-stats the standard way (dividing the
mean by the volatility and multiplying by $\sqrt{500}$). 500 months is close
to the 600 month sample used in [39].

For each simulation, I compute the easy FDR bound (Equation (4)) and the
Storey (2002) bound (Equation (11)). I compare these bounds with the actual
$\text{FDR}_{|t|>2}$, which is computed by averaging the FDP (the ratio of
false discoveries to discoveries) across simulations.

Figure 4 shows simulated estimation results for various choices for $\gamma$
and $\Pr\left(F_{i}\right)$. Panel (a) shows $\gamma=$ 25 bps per month, near
the 10th percentile of in-sample returns in the CZ data. Panel (b) uses
$\gamma=$75 bps, which is near the 70th percentile. These two choices for
$\gamma$ span what might be considered a reasonable range of values for the
expected returns of true predictors. The figure examines values of
$\Pr\left(F_{i}\right)$ ranging from 1% to 99%.

Figure 4: FDR Estimates in Cross-Sectional Predictability Simulations

I simulate long-short returns (Equation (17)-(18)) by cluster-bootstrapping
from the CLZ strategies, computing FDR bounds, and comparing with the actual
FDR. $\gamma$ is the expected return of true predictors. “Easy Upper Bound”
plugs in $\Pr\left(F_{i}\right)=1$ into Bayes rule (Equation (4)). “Storey
Bound” bounds $\Pr\left(F_{i}\right)$ following [32] (Equation (11)). Both
formulas reliably bound the FDR.

(a) $\gamma=25$ (bps pm)

(b) $\gamma=75$ (bps pm)

Both formulas reliably bound $\text{FDR}_{|t|>2}$. The Storey bound comes
closer to the actual FDR, and so provides a more accurate assessment of the
veracity of the cross-sectional literature. Overall, these simulations show
that the data mined FDR bounds in Sections 2 and 3 are valid for cross-
sectional predictability.

###  Relationship with the [4] and [5] algorithms

My estimates first define discoveries and then estimate an FDR bound. This
method is known as the direct approach to FDRs ([32]). The finance literature
focuses on FDR control instead, using the [4] (BH95) algorithm and [5]’s
(2001) (BY’s) Theorem 1.3.666I specify the theorem number 1.3 from [5] because
the thrust of that paper regards Theorem 1.2.  These algorithms reverse the
direct approach: they first define an FDR bound and then estimate discoveries.

These approaches are closely related and can be equivalent under weak
dependence ([35]; [15]). The see the equivalence, note the BH95 algorithm can
be written as

$\displaystyle h^{\ast}$
$\displaystyle=\min_{h>0}\left\\{h:\widehat{\text{FDR}}_{\text{max,BH}}\left(|t_{i}|>h\right)\leq
q^{\ast}\right\\}.$ (19)

where $h^{\ast}$ is the estimated t-stat hurdle, $q^{\ast}$ is the FDR bound
selected by the researcher, and

$\displaystyle\widehat{\text{FDR}}_{\text{max,BH}}\left(|t_{i}|>h\right)$
$\displaystyle\equiv\frac{\Pr\left(|t_{i}|>h|F_{i}\right)}{\text{Share of
}|t_{i}|>h}.$ (20)

Equation (20) generalizes the easy FDR bound (Equation (3)) to define
discoveries as $|t_{i}|>h$ instead of $|t_{i}|>2$. Thus, the BH95 algorithm is
equivalent to choosing a t-stat hurdle that ensures the easy FDR bound is less
than some prespecified value. For further details see Appendix C.

BY’s Theorem 1.3 is equivalent to replacing
$\widehat{\text{FDR}}_{\text{max,BH}}\left(|t_{i}|>h\right)$ in Equation (19)
with

$\displaystyle\widehat{\text{FDR}}_{\text{max,BY1.3}}\left(|t_{i}|>h\right)$
$\displaystyle\equiv\frac{\Pr\left(|t_{i}|>h|F_{i}\right)}{\text{Share of
}|t_{i}|>h}\times\sum_{j=1}^{M}\frac{1}{j}.$

This expression amounts to plugging
$\Pr\left(F_{i}\right)=\sum_{j=1}^{M}\frac{1}{j}\approx\log M\gg 100\%$ into
Bayes rule (Equation (2)). My methods omit this penalty, as Section 3.1 argues
$\Pr\left(F_{i}\right)=100\%$ is already unreasonably conservative and Section
4.2 shows the easy bound is reliable for cross-sectional predictability
settings. Indeed, the original paper [5] describes Theorem 1.3 as “very often
unneeded, and yields too conservative of a procedure.”

##  Reconciliation and Interpretation

My results appear to conflict with HLZ, who “argue that most claimed research
findings in financial economics are likely false.” This section shows that,
despite the apparent conflict, our FDR estimates are actually quite similar.
It also argues in favor of my interpretation of these estimates.

###  FDR Estimates in Harvey, Liu, and Zhu (2016)

HLZ shows empirical FDR estimates in two exhibits: Figure 3 (page 25) and
Table 5 (page 34). These exhibits use different methods but arrive at similar
numbers.

Figure 3 uses Benjamini and Yekuteli’s benjamini2001control Theorem 1.3 to
estimate t-stat hurdles that imply FDR $\leq$ 1% using hand-collected data on
published t-stats. It shows that a hurdle of 3.39 is required. This estimate
implies that findings with $|t_{i}|>3.39$ are almost certainly true, but says
little about findings that fall below this cutoff. A more complete picture can
be found in the text describing Figure 3, which states that the same method
implies a hurdle of 2.81 if the FDR bound is raised to 5% (page 26, top
paragraph).

The natural question, then, is how many t-stats in asset pricing papers meet
the 2.81 hurdle? Perhaps surprisingly, this answer is not directly found in
HLZ. But a closely related number is reported [6]’s Table 1, which simulates
HLZ’s SMM estimates to approximate their dataset. Chen finds that, of the
$|t_{i}|$ that exceed 2.0, 69% also exceed $3.0$. Thus, the FDR among
$|t_{i}|>2.0$ in HLZ’s data is at _most_ $0.69\times 0.05+(1-0.69)=34\%$. This
number is higher than the FDR bounds in Section 2, but it is qualitatively
similar. In the end, both HLZ’s Figure 3 and my analysis find that a minority
of published findings are false.

HLZ’s Table 5 uses an SMM estimation based on a structural model that includes
both published and unpublished t-stats (see Appendix D). This table shows a
variety of estimates but the authors argue in favor of the “$\rho=0.2$”
estimate (page 35, last paragraph). This estimate says that choosing an FDR of
5% implies a hurdle of $2.27$. Given that this threshold is close to 2.0, this
estimate suggests that the FDR for published findings is also close to 5%.
Indeed, of t-stats in the [12] open-source dataset that exceed 2.0, 91% also
exceed 2.27. This implies an FDR of at most $0.91(0.05)+0.09=14\%,$ exactly in
line with Equation (6).

Figure 5 takes a closer look at HLZ’s Table 5. Panel (a) shows a scatterplot
of expected returns against absolute t-statistics from simulating HLZ’s
estimated model, with each marker representing one factor.777This figure is
similar to Figure 8 of [11], which was circulated after the first draft of
this paper. HLZ’s model features two kinds of factors: (1) factors with
expected returns of zero (empty markers) and (2) factors with positive
expected returns (filled markers). A complete description of HLZ’s model can
be found in Appendix D.

Figure 5: Two Interpretations of Harvey, Liu, and Zhu’s (2016) SMM Estimates

I simulate HLZ’s SMM estimates (HLZ’s Table 5, $\rho=0.2$). Each marker
represents one factor. “FDR = 5%” and “FDR = 1%” show t-hurdles from HLZ’s
Table 5. “Bonferroni 5%” uses the calculation implied by HLZ’s conclusion (HLZ
page 37 paragraph 5). Panel (a) defines false factors following the
traditional interpretation ([31]). Panel (b) uses an interpretation from HLZ’s
conclusion. HLZ’s argument that “most claimed research findings in financial
economics are likely false” comes from equating significance with truth. This
interpretation leads to factors with expected returns of 75 bps or more being
defined as false.

(a) Traditional Interpretation

(b) Harvey, Liu, Zhu’s (2016) Interpretation

Following the traditional notion of a false discovery ([31]), I label the
markers with expected returns of zero as false. This labeling leads to a neat
visualization of HLZ’s Table 5. If we define discoveries as predictors with
$|t_{i}|>2.95$ (dot-dash line), we have about 210 discoveries. Only two of
these factors are false, leading to an FDR of $2/210\approx 1\%$, consistent
with HLZ’s famous 3.0 hurdle and the “FDR(1%)” column of their Table 5. If
instead, we define discoveries using $|t_{i}|>2.27$ (long-dash line), we have
305 discoveries, 13 false discoveries, and an FDR of $13/305\approx 5\%$,
consistent with the “FDR(5%)” column of their Table 5.

What is the FDR if we just define discoveries as $|t_{i}|>2$? The figure shows
that, in this case, the FDR is just 9%. In other words, about 91% of claimed
findings in HLZ’s dataset are true. This estimate is closely in line with the
$\text{FDR}_{|t|>2}$ bound of 9% I derive following [32] (Section 3.1).

###  Two Interpretations of “False Factor”

If HLZ’s estimates imply $\text{FDR}_{|t|>2}\approx 9\%$, why do they “argue
that most claimed research findings in financial economics are likely false?”
The answer can be found in their conclusion (page 37, paragraph 5):

> _In medical research, the recognition of the multiple testing problem has
> led to the disturbing conclusion that “most claimed research findings are
> false” ([23]). Our analysis of factor discoveries leads to the same
> conclusion–many of the factors discovered in the field of finance are likely
> false discoveries: of the 296 published significant factors, 158 would be
> considered false discoveries under Bonferonni [sic], 142 under Holm, 132
> under [BY’s Theorem 1.3] (1%), and 80 under [BY’s Theorem 1.3] (5%)._

Traditionally, the Bonferroni correction provides a bound on the family wise
error rate, which in turn is equal to the probability that the FDR is
positive. It does not provide the number of false discoveries. The same is
true of the [22] test, which is a variant of Bonferroni. In fact, BY’s Theorem
1.3 does not provide the number of false discoveries either. It can provide an
upper bound—but this bound may be far from the actual number (see Section
4.3).

These issues make this paragraph difficult to understand. But the paragraph is
straightforward if one interprets a “false factor” as an “insignificant factor
under a chosen hypothesis test.” With Bonferroni at the 5% level as the chosen
test, the 158 insignificant factors are also false factors. $158/296=53\%$,
which can be described as most, so most claimed research findings are false.

HLZ’s conclusion paragraph also equates “most” and “many.” Of the five FDRs
described in this paragraph, only 158/296 could be described as most, using
the traditional interpretation of most. Notably, $80/296=27\%$ is rarely
described as most.

The equating of “null” and “insignificant” is also found in HLZ. In describing
their multiple testing framework, HLZ state “[i]n a factor testing exercise,
the typical null hypothesis is that a factor is not significant. Therefore, a
factor being insignificant means the null hypothesis is ‘true”’ (page 13, 2nd
paragraph). A similar language is found further down on page 13: “under the
null that all factors are insignificant” (3rd paragraph) and “the type I error
in multiple testing is also related to false discoveries, by which we conclude
a factor is ‘significant’ when it is not” (4th paragraph). The equating of
false and insignificant is also found in [20] (see Appendix E).

In contrast, the statistics literature defines a false discovery as a
significant finding that is, in truth, null ([31]; [4]). This definition
displays an important separation between significance and truth. No matter how
well-designed a significance test is, it can still miss the truth, declaring
nulls as significant, or non-nulls as insignificant. This separation is
essentially “Principal 2” of the American Statistical Association’s “Statement
on Statistical Significance and P-Values” ([37]). This separation also has a
long tradition, going back to [16].

###  The Problem with Equating Significance and Truth

One might argue for HLZ’s interpretation by pointing to multiple testing
problems. These problems invalidate classical tests, so it is perhaps natural
to describe published factors that fail multiple testing controls as false
discoveries.

The problem with this interpretation is illustrated in Panel (b) of Figure 5.
This panel shows the same simulation as in Panel (a), but instead of using the
traditional interpretation, it interprets false discoveries as factors that
are insignificant under HLZ’s chosen Bonferroni test (empty markers). HLZ
choose a significance level of 5%, implying a hurdle of $|t_{i}|>3.8$.

Factors with impressive expected returns, of 75 or even 150 bps per month,
fail to meet this hurdle, and are thus labeled false. These numbers are not
sample mean returns, which may be due to luck, but the expected returns, cured
of sampling error (see Equations (25) and (26) in Appendix D). If a false
discovery can have such a high expected return, does the concept make any
economic sense?

Panel (b) illustrates an additional problem. The figure shows many
significance hurdles. Which hurdle should be the one that defines a false
discovery? Perhaps the natural hurdle is the one that leads to an FDR of 5%,
since FDR $\leq$ 5% is commonly used in fields like genetics and functional
imaging ([3]). But HLZ’s conclusion emphasizes the extremely conservative
Bonferroni 5% test instead. The choice of hurdle is clearly subjective.
Different researchers will choose different hurdles.

Thus, equating significance and truth would lead to a king of “post-truth
finance,” in which different researchers define different truths. It is
unlikely that just any ordinary researcher will get to chose a test that
defines false discoveries. Thus, HLZ’s interpretation leads to difficult
questions about the future of the scientific process in asset pricing. Which
researchers will get to define truth? How these researchers be chosen?

Panel (b) also illustrates the problem with equating “most” and “many.” Of the
many hurdles in Panel (b), only a single hurdle finds that most claimed
findings are false, using HLZ’s interpretation. In fact, the more commonly
used FDR $=$ 5% control (dashed line) would lead a rather small share of false
discoveries, as seen in the relatively small number of factors lying between
the solid and dashed lines. Equating most with many, as is done in HLZ’s
conclusion, implies ignoring this important evidence.

##  Conclusion

This paper provides simple arguments that bound the FDR under publication
bias. They handle publication bias by using data mining experiments and
conservative extrapolations as worst case scenarios. The arguments are so
simple that they can be explained in just a few equations or a couple of
diagrams. Yet simulations show that they are valid in settings that mimic the
cross-predictor correlations found in empirical asset pricing.

Applying these arguments to numbers reported in previous papers, I find that
at least 75% of claimed findings in cross-sectional predictability are true.
More refined estimates using the [8] dataset imply at least 91% of findings
are true.

These estimates suggest a way forward for the literature on multiple testing
problems. This literature features complicated methods that take dozens of
pages to exposit, which likely contribute to the lack of consensus in the
literature. My estimates provide a transparent language for understanding and
debating these issues.

## References

  * [1] Isaiah Andrews and Maximilian Kasy  “Identification of and correction for publication bias”  In _American Economic Review_ 109.8, 2019, pp. 2766–94
  * [2] Yoav Benjamini  “Comment: Microarrays, empirical Bayes and the two-groups model”  In _Statistical Science_ 23.1 Institute of Mathematical Statistics, 2008
  * [3] Yoav Benjamini  “Selective inference: The silent killer of replicability”  PubPub, 2020
  * [4] Yoav Benjamini and Yosef Hochberg  “Controlling the false discovery rate: a practical and powerful approach to multiple testing”  In _Journal of the royal statistical society. Series B (Methodological)_ JSTOR, 1995, pp. 289–300
  * [5] Yoav Benjamini and Daniel Yekutieli  “The control of the false discovery rate in multiple testing under dependency”  In _Annals of statistics_ JSTOR, 2001, pp. 1165–1188
  * [6] Andrew Y Chen  “The Limits of p-Hacking: Some Thought Experiments”  In _The Journal of Finance_ Wiley Online Library, 2021
  * [7] Andrew Y Chen  “Do t-Statistic Hurdles Need to be Raised”  In _arXiv preprint arXiv:2204.10275_ , 2022
  * [8] Andrew Y Chen, Alejandro Lopez-Lira and Tom Zimmermann  “Does peer-reviewed theory help predict the cross-section of stock returns?”  In _arXiv preprint arXiv:2212.10317_ , 2023
  * [9] Andrew Y Chen and Mihail Velikov  “Zeroing in on the expected returns of anomalies”  In _Journal of Financial and Quantitative Analysis_ 58.3 Cambridge University Press, 2023, pp. 968–1004
  * [10] Andrew Y Chen and Tom Zimmermann  “Publication bias and the cross-section of stock returns”  In _The Review of Asset Pricing Studies_ 10.2 Oxford University Press, 2020, pp. 249–289
  * [11] Andrew Y Chen and Tom Zimmermann  “Publication Bias in Asset Pricing Research”  In _Oxford Research Encyclopedia of Economics and Finance_ , 2023
  * [12] Andrew Y. Chen and Tom Zimmermann  “Open Source Cross-Sectional Asset Pricing”  In _Critical Finance Review_ 27.2, 2022, pp. 207–264
  * [13] Tarun Chordia, Amit Goyal and Alessio Saretto  “Anomalies and false rejections”  In _The Review of Financial Studies_ 33.5 Oxford University Press, 2020, pp. 2134–2179
  * [14] Bradley Efron  “Microarrays, empirical Bayes and the two-groups model”  In _Statistical science_ 23.1 Institute of Mathematical Statistics, 2008, pp. 1–22
  * [15] Alessio Farcomeni  “Some results on the control of the false discovery rate under dependence”  In _Scandinavian Journal of Statistics_ 34.2 Wiley Online Library, 2007, pp. 275–297
  * [16] R.. Fisher  “The Design of Experiments”  Edinburgh: OliverBoyd, 1935
  * [17] Nikolay Gospodinov, Raymond Kan and Cesare Robotti  “Misspecification-robust inference in linear asset-pricing models with irrelevant risk factors”  In _The Review of Financial Studies_ 27.7 Oxford University Press, 2014, pp. 2139–2170
  * [18] Jeremiah Green, John RM Hand and X Frank Zhang  “The supraview of return predictive signals”  In _Review of Accounting Studies_ 18.3 Springer, 2013, pp. 692–730
  * [19] Campbell R Harvey  “Presidential address: The scientific outlook in financial economics”  In _The Journal of Finance_ 72.4 Wiley Online Library, 2017, pp. 1399–1440
  * [20] Campbell R Harvey and Yan Liu  “False (and missed) discoveries in financial economics”  In _The Journal of Finance_ 75.5 Wiley Online Library, 2020, pp. 2503–2553
  * [21] Campbell R Harvey, Yan Liu and Heqing Zhu  “… and the cross-section of expected returns”  In _The Review of Financial Studies_ 29.1 Oxford University Press, 2016, pp. 5–68
  * [22] Sture Holm  “A simple sequentially rejective multiple test procedure”  In _Scandinavian journal of statistics_ JSTOR, 1979, pp. 65–70
  * [23] John PA Ioannidis  “Why most published research findings are false”  In _PLoS medicine_ 2.8 Public Library of Science, 2005, pp. e124
  * [24] Heiko Jacobs and Sebastian Müller  “Anomalies across the globe: Once public, no longer existent?”  In _Journal of Financial Economics_ 135.1 Elsevier, 2020, pp. 213–230
  * [25] Theis Ingerslev Jensen, Bryan Kelly and Lasse Heje Pedersen  “Is there a replication crisis in finance?”  In _The Journal of Finance_ 78.5 Wiley Online Library, 2023, pp. 2465–2518
  * [26] Raymond Kan and Chu Zhang  “Two-pass tests of asset pricing models with useless factors”  In _the Journal of Finance_ 54.1 Wiley Online Library, 1999, pp. 203–235
  * [27] Serhiy Kozak, Stefan Nagel and Shrihari Santosh  “Interpreting factor models”  In _The Journal of Finance_ 73.3 Wiley Online Library, 2018, pp. 1183–1223
  * [28] Jonathan Lewellen, Stefan Nagel and Jay Shanken  “A skeptical appraisal of asset pricing tests”  In _Journal of Financial Economics_ 96.2 Elsevier, 2010, pp. 175–194
  * [29] Juhani T Linnainmaa and Michael R Roberts  “The history of the cross-section of stock returns”  In _The Review of Financial Studies_ 31.7 Oxford University Press, 2018, pp. 2606–2649
  * [30] R David McLean and Jeffrey Pontiff  “Does academic research destroy stock return predictability?”  In _The Journal of Finance_ 71.1 Wiley Online Library, 2016, pp. 5–32
  * [31] Branko Sorić  “Statistical “discoveries” and effect-size estimation”  In _Journal of the American Statistical Association_ 84.406 Taylor & Francis, 1989, pp. 608–610
  * [32] John D Storey  “A direct approach to false discovery rates”  In _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ 64.3 Wiley Online Library, 2002, pp. 479–498
  * [33] John D Storey  “The positive false discovery rate: a Bayesian interpretation and the q-value”  In _The Annals of Statistics_ 31.6 Institute of Mathematical Statistics, 2003, pp. 2013–2035
  * [34] John D Storey  “False Discovery Rate.”  In _International encyclopedia of statistical science_ 1, 2011, pp. 504–508
  * [35] John D Storey, Jonathan E Taylor and David Siegmund  “Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach”  In _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ 66.1 Wiley Online Library, 2004, pp. 187–205
  * [36] John D Storey and Robert Tibshirani  “Estimating false discovery rates under dependence, with applications to DNA microarrays”, 2001
  * [37] Ronald L Wasserstein and Nicole A Lazar  “The ASA statement on p-values: context, process, and purpose”  In _The American Statistician_ 70.2 Taylor & Francis, 2016, pp. 129–133
  * [38] Jeffrey M Wooldridge  “Estimation and inference for dependent processes”  In _Handbook of econometrics_ 4 Elsevier, 1994, pp. 2639–2738
  * [39] Xuemin Sterling Yan and Lingling Zheng  “Fundamental Analysis and the Cross-Section of Stock Returns: A Data-Mining Approach”  In _The Review of Financial Studies_ 30.4 Oxford University Press, 2017, pp. 1382–1423

##  Appendix A Bounding the FDR by plugging in numbers from previous papers

###  FDR bounds from plugging in data mining numbers

This section describes the data mining procedures and t-statistics for the
papers listed in Table 1.

####  FDR Bounds using [39]’s (YZ’s) Table 1

YZ choose 240 Compustat accounting variables based on data availability
requirements (e.g. non-missing values in at least 20 years). They also choose
15 intuitive base variables (e.g. total assets, sales). They then form all
combinations of $X/Y$, $\Delta\left(X/Y\right)$, $\%\Delta\left(X/Y\right)$,
$\%\Delta X$, $\%\Delta X-\%\Delta Y$, and $\Delta
X/\text{lag}\left(Y\right)$, where $X$ is one of the 240 variables and $Y$ is
one of 15 base variables, leading to 18,000 signals. These functional forms
are selected based on a survey of accounting textbooks and academic papers.
For each variable, YZ form long-short decile portfolios.

Figure A.1: Yan and Zheng’s Table 1

YZ’s Table 1 (shown in Figure A.1) shows order statistics for t-statistics
measured using various methods. For EW 1-factor $\alpha$’s, the 90th and 10th
percentile t-statistics are +2.41 and -3.48, respectively (highlighted). This
means that at least 20 percent of absolute t-stats exceed 2. Applying to
Equation (2):

$\displaystyle\text{FDR}_{|t|>2}$
$\displaystyle\leq\frac{5\%}{\Pr\left(|t_{i}|>2\right)}\leq\frac{5\%}{0.20}=25\%.$

EW 3- and 4-factor $\alpha$’s lead to the same result.

Of course, this bound may be much tighter if one had more detailed order
statistics. -3.48 is quite far from -2.0, implying that one can form a much
tighter bound. Section 2.1 fills this gap using the CLZ data, and finds
$\text{FDR}_{|t|>2}\leq 15\%$.

For VW $\alpha$’s, one needs to use a slightly more general form of this
bound. The 1-factor $\alpha$’s 90th and 10th percentile t-statistics are 1.93
and -1.87, respectively, so one needs to use the more general form

$\displaystyle\text{FDR}\left(|t_{i}|>1.87\right)$
$\displaystyle\leq\frac{\Pr\left(|t_{i}|>1.87|F_{i}\right)}{\Pr\left(|t_{i}|>1.87\right)}=\frac{6\%}{0.20}=30\%.$

Since the FDR decreases in the t-stat hurdle,
$\text{FDR}_{|t|>2}\leq\text{FDR}\left(|t_{i}|>1.87\right)\leq 30\%$. So even
using VW portfolios, one still finds that most data-mined findings with
$|t_{i}|>2$ are true.

The picture changes slightly with VW 4-factor $\alpha$’s. Here the table shows
that $\Pr\left(|t_{i}|>2\right)\approx 10\%$, implying $\text{FDR}_{|t|>2}\leq
50\%$. Nevertheless, for most of the VW results, the Equation (2) implies that
most claimed statistical findings are true. Moreover, these bounds come from
an atheoretical data mining exercise, and researchers should be able to obtain
a smaller FDR. Indeed, Section 3.2 shows that the [11, 8] data mining
procedure leads to an FDR of at most 34% when using 4-factor value-weighted
mean returns.

####  FDR Bounds using [8]’s Table 4

CLZ study 29,000 accounting signals. Table 4 forms long-short portfolios on
these signals, calculates t-statistics for each portfolio using the past 30
years of returns, sorts portfolios into 5 bins based on the past 30 year
return, and then averages t-statistics within each bin and across bin-
formation years.

The equal-weighted panel shows that the mean t-stat for in-sample bin 2 is
-2.46. Assuming volatility is relatively stable across portfolios, this
implies that the maximum t-stat for bin 1 is at most -2.46, so at least 20% of
signals produce $|t_{i}|>2$. Plugging this result into Equation (4) implies
$\text{FDR}_{|t|>2}\leq 5\%/0.20=25\%$.

The table also shows value-weighted results, but the statistics shown are not
refined enough to estimate the share of $|t_{i}|>2$. Section 3.2 examines this
issue using the full CLZ dataset, and finds that the $\text{FDR}_{|t|>2}$ for
value-weighted returns is typically less than 30%.

####  FDR Bounds using [13]’s Tables

Figure A.2: Chordia, Goyal, and Saretto’s (2020) Table A2

CGS begin with 185 accounting variables and then apply the following
transformations: (1) annual growth rates of a single variable, (2) ratios of
two variables, and (3) difference ratios involving three variables as in
$(x_{1}-x_{2})/x_{3}$. They examine a wide variety of predictability t-stats,
leading to numerous reports of the $|t_{i}|>1.96$ in Tables 3, 7, 8, and A2,
among other locations.

In most exhibits, the share of $|t_{i}|>1.96$ ranges from 20 to 30 percent.
Table A2 (shown in Figure A.2) is representative and is helpful since it shows
13 estimates in a single table (red box). Here, the shares range from 20% to
35%, with one exception. The one exception is the value-weighted long short
mean return test, which leads to a share of 10.6%, which is consistent with
the finding in Section 3.2 that value-weighted long-short raw return tests
have unusually small t-stats in the CLZ data. All of CGS’s alpha estimates use
value-weighted returns.

If one can exclude this outlier, then 20% is a lower bound on
$\Pr\left(|t_{i}|>2\right)$, leading to
$\text{FDR}_{|t|>2}\leq\frac{5\%}{0.20}=25\%.$

###  FDR Bounds from plugging in predictor zoo numbers

This section describes the t-statistics collection for the predictor zoo
papers listed in Table 2.

####  FDR Bounds using hand-collected [18]

GHZ hand-collect numbers for 330 predictors from cross-sectional
predictability studies. Table 3 reports the mean number of months used is 291.
Table 4 reports the mean equal-weighted annual Sharpe ratio is 1.04. Thus, the
mean published t-statistic is approximately $1.04\times\sqrt{291/12}=5.1$.
Plugging into Equation (6) and (7) implies $\text{FDR}_{|t|>2}\leq
5\%\exp(2/(5.1-2))=10\%$.

Table 5 reports the mean value-weighted Sharpe ratio is 0.70, implying a mean
published t-statistic is approximately $0.70\times\sqrt{291/12}=3.44$.
Plugging into Equation (6) and (7) implies $\text{FDR}_{|t|>2}\leq
5\%\exp(2/(3.44-2))=20\%$.

####  FDR bounds using hand-collected t-stats in [10]

The focus of CZ 2020 is on replications of published cross-sectional
predictors. But as a robustness check on regarding their replicated t-stats,
[10] report results using 77 hand-collected t-stats from cross-sectional
predictability studies. Given that CZ’s replications use equal-weighting, it
is sensible to assume that most all of these hand-collected t-stats also use
equal-weighting.

Table 2 reports that the mean hand-collected t-statistic is 4.57. Plugging
into Equation (6) and (7) implies $\text{FDR}_{|t|>2}\leq
5\%\exp(2/(4.57-2))=11\%$.

Of course, one can also use the replicated t-stats in CZ 2020. Since CZ 2020
find the replicated and hand-collected t-stats are very similar, the
$\text{FDR}_{|t|>2}$ bound is also similar.

####  FDR bounds using hand-collected t-stats in [21]

HLZ describe their data collection as follows: “Our goal is not to catalog
every asset pricing paper ever published. We narrow the focus to papers that
propose and test new factors.” They arrive at 313 articles and “catalogue 316
different factors.”

HLZ do not report the mean published t-statistic. However, one can back out
the mean published t-statistic using Appendix A.1, which uses exponential
distributions to extrapolate unpublished t-stats. Table A.1 illustrates the
extrapolations (see Figure A.3).

Figure A.3: Harvey, Liu, and Zhu’s (2016) Table A.1

Using a variety of specifications, HLZ find that the mean t-stat overall is
around 2.0. Applying Equation (7), then, implies a mean published t-stat of
about 4.0.

In Table 2, I focus on the estimate in Panel B, because it is more directly
comparable to the other estimates. Indeed, HLZ’s footnote 50 implies that one
can back out the mean published t-stat by applying Equation (7) to the
estimate in Panel B. A caveat here, however, is that there seems to be a typo
in footnote 50 ($\hat{\lambda}=1/\left(\bar{t}-c\right)$ should say
$\hat{\lambda}=\bar{t}-c$ to match the numbers in HLZ’s Table A.1).

Regardless, all of the estimates in HLZ’s Table A.1 imply that
$\text{FDR}_{|t|>2}$ is quite small. Using the estimate of the mean overall
t-stat implies $\text{FDR}_{|t|>2}=5\%\exp\left(2/1.96\right)=13.9\%$.

####  FDR bounds using replicated t-stats in [30]

MP replicate 97 long-short quintile strategies based on from published
studies. Page 16 (paragraph 1) reports that the average t-statistic in their
replications is 3.55. Plugging into Equation (6) and (7) implies
$\text{FDR}_{|t|>2}\leq 5\%\exp(2/(3.55-2))=18\%$.

####  FDR bounds using replicated t-stats in [24]

This paper replicates 241 long-short quintile strategies based on published
studies. The online appendix Table 2 reports a mean return of 55 bps per month
and a t-statistic of 3.08 for US stocks over the full sample. Plugging into
Equation (6) and (7) implies $\text{FDR}_{|t|>2}\leq
5\%\exp(2/(3.08-2))=32\%$.

The FDR over the full sample is likely higher than the FDR in the original
samples, however. Table 2 reports pooled long-short returns in US and in-
sample data have a mean return of 74.2 bps per month, much higher than the 55
bps per month over the full sample. Scaling the full sample t-stat by the mean
return differential leads to rough estimate of the mean in-sample t-stat:
$3.08\times 74.2/55=4.2$, and $\text{FDR}_{|t|>2}\leq 5\%\exp(2/(4.2-2))=8\%$.

##  Appendix B Simulation Appendix

###  Details on simulations of data-mined strategies

tbc

[Describe the missing values handling]

Figure A.4: Data-Mining Simulation vs Empirical Correlations

I simulate monthly long-short return residuals by cluster-bootstrapping from
CLZ’s data-mined strategies and compare with the correlations in the original
CLZ data. Return residuals subtract the sample mean at the signal level from
returns. Both distributions use a random sample of 1,000 signals for
tractability.

###  Estimations of simulations of published data

The simulation model is a slight modification of the data mining simulations
(Section 4.2).

The key modification is that $N$ returns are bootstrapped from the [12] (CZ)
replications as follows:

$\displaystyle r_{i,\tau}$ $\displaystyle=\mu_{i}+\varepsilon_{i,\tau}$

where $\varepsilon_{i,\tau}$ is defined as

$\displaystyle\varepsilon_{i,\tau}$
$\displaystyle=0.65\hat{\varepsilon}_{k\left(i\right),\upsilon\left(\tau\right)}+0.35\delta_{i,\tau}$
(21)

where $\hat{\varepsilon}_{k,\upsilon}$ is the de-meaned long-short return for
signal $k$ in month $\upsilon$ in the CZ data, $k(i)$ and $\upsilon(\tau)$ are
random integers, and $\delta_{i,t}\sim\text{Normal}\left(0,3.32\right)$ i.i.d.
In other words, I cluster-bootstrap residuals from empirical data, where the
clustering preserves cross-sectional correlations, but I mix in 35% random
noise, where the noise has a volatility similar to the empirical data. 3.32%
is the mean volatility of returns in the CZ dataset.

Purely bootstrapping from the CZ data would lead to many redundant strategies
and excessively high correlations for $N\gg 200$, since there are only 200
strategies in the CZ data. 35% random noise is selected to fit the empirical
distribution of correlations, as seen in Figure A.5.

Figure A.5: Publication Simulation vs Empirical Correlations

I simulate monthly long-short return residuals by mixing a cluster-bootstrap
of de-meaned returns from the [12] data with noise ($\varepsilon_{i,t}$ in
Equation (21)). Empirical return residuals subtract the sample mean at the
predictor level (using in-sample data only) from returns. I compare simulated
correlations with correlations in the original [12] data. The simulated
distribution uses a random sample of 1,000 predictors for tractability.

As in Section 4.2, expected returns follow

$\displaystyle\mu_{i}$ $\displaystyle=\begin{cases}0&\text{if }F_{i}\\\
\gamma&\text{if }T_{i}\end{cases}.$

I fix $N=10,000$ and $T=200$.

The publication process is the same as in HLZ:

$\displaystyle\Pr\left(S_{i}|t_{i}\right)$
$\displaystyle=\begin{cases}0&|t_{i}|<1.96\\\
0.5\bar{s}&|t_{i}|\in(1.96,2.57]\\\ \bar{s}&|t_{i}|>2.57\end{cases},$ (22)

where $S_{i}$ indicates $i$ is selected for publication and $\bar{s}$ is the
maximum probability of selection. WLG, I set $\bar{s}=1.0$. A lower $\bar{s}$
is equivalent to a smaller $N$.

In each simulation, I estimate the FDR bound using the easy exponential
extrapolation (Equation (6) and (7)). I also calculate the FDP for published
strategies with $|t_{i}|>2.0$. I then average across simulations to obtain the
actual FDR and the average FDR bound. The results are shown in Figure A.6.

Figure A.6: FDR Estimates in Simulations of Published Data

I simulate published returns, estimate the easy FDR bound using exponential
extrapolation, and compare to the actual FDR (see Section B.2)

(a) $\gamma=25$ (bps pm)

(b) $\gamma=50$ (bps pm)

(c) $\gamma=75$ (bps pm)

For most parameter values, the easy extrapolation formula bounds the actual
FDR.The exceptions occur when $\Pr\left(F_{i}\right)$ is around 90% or
$\gamma=75$ bps per month.

These settings imply that the distribution of t-stats displays an extremely
sharp curvature below $|t_{i}|<2.0$. Since the data mined returns do not
display such a sharp curvature, it is unlikely that this setting is relevant,
as it would mean that researchers uncover $|t_{i}|<2.0$ at a much higher rate
than pure data mining.

##  Appendix C FDR Control Equivalence Details

This section provides further details on the expressions in Section 4.3.

To derive Equation (19), start with HLZ’s description of BH95 control:

  * •

Order the p-values such that $p_{(1)}\leq p_{(2)}\leq...\leq p_{(M)}$ where
$M$ is the total number of predictors.

  * •

Find $k^{\ast}$ that solves

$\displaystyle k^{\ast}=\max_{k\in\\{1,2,..,M\\}}$
$\displaystyle\left\\{k:p_{(k)}\leq\frac{k}{M}q^{\ast}\right\\}$ (23)

where $q^{\ast}$ is the FDR bound selected by the researcher.

  * •

Reject null hypotheses corresponding to $p_{(1)},p_{(2)},...,p_{(k^{\ast})}$.

To see the equivalence, first note that Equation (23) can be written as

$\displaystyle h^{\ast}$
$\displaystyle=\min_{h\in\\{|t_{1}|,|t_{2}|,...,|t_{M}\\}\\}}\left\\{h:\Pr\left(|t_{i}|>h|F_{i}\right)\leq\frac{\left[\text{Number
of }|t_{i}|>h\right]}{M}q^{\ast}\right\\}.$ (24)

That is, finding the largest p-value is equivalent to finding the smallest
t-stat hurdle that satisfies the constraint. The constraint can be written in
terms of a t-stat hurdle, $p_{(k)}=\Pr\left(|t_{i}|>h|F_{i}\right)$ for
$h=|t_{(k)}|$, where $t_{(k)}$ ranks the absolute t-statistics in descending
order. Finally, since $k$ is the ranking of the p-value $p_{(k)}$, it is also
the number of p-values smaller than $p_{(k)}$, or equivalently the number of
$|t_{i}|>h$.

[tbc: discuss BY Theorem 1.3, algorithmic equivalence is easier than proving
control, explain why weak dependence gets so technical, list assumptions in
[15]]

##  Appendix D Harvey, Liu, and Zhu’s (2016) Parametric Model

HLZ’s model starts with a description of all factors, $i=1,2,...,N$

$\displaystyle\mu_{i}$
$\displaystyle\sim\begin{cases}\text{Dirac}\left(0\right)&\text{with prob
}p_{0}\\\ \text{Exponential}\left(\lambda\right)&\text{with prob
}(1-p_{0})\end{cases}$ (25) $\displaystyle t_{i}|\mu_{i}$
$\displaystyle\sim\text{Normal}\left(\frac{\mu_{i}}{SE},1\right)$ (26)
$\displaystyle\text{Corr}\left(t_{i},t_{j}\right)$ $\displaystyle=\rho\text{
for }i\neq j$

where $\mu_{i}$ is the expected return on factor $i$,
$\text{Dirac}\left(0\right)$ is the distribution with a point mass at zero,
$\text{Exponential}\left(\lambda\right)$ is an exponential distribution with
shape parameter $\lambda$, $p_{0}$ if the proportion of nulls, $SE$ is the
(constant) standard error of the sample mean return, and $\rho$ is the
constant pairwise correlation between monthly returns.

HLZ choose $SE=(1500/\sqrt{12})/\sqrt{240}$ (page 29) and $\rho=0.20$ (page
35).

Factors are selected for publication according to a staircase function

$\displaystyle\Pr\left(S_{i}|t_{i}\right)$
$\displaystyle=\begin{cases}0&|t_{i}|<1.96\\\
0.5\bar{s}&|t_{i}|\in(1.96,2.57]\\\ \bar{s}&|t_{i}|>2.57\end{cases},$ (27)

where $S_{i}$ indicates $i$ is selected and $\bar{s}$ is the maximum
probability of selection. HLZ do explicitly describe this function but it is
implied by their data adjustment for publication bias (page 31). HLZ
implicitly assume $\bar{s}=1.0$, which allows identification of $N$. It is
unclear if $\bar{s}=1.0$ is a reasonable assumption and indeed [1, 7] do not
attempt to identify $N$ or $\bar{s}$.

This model contains parameters $(p_{0},\lambda,N$) which HLZ estimate by SMM.
They select as target moments the number of published and significant factors,
as well as three order statistics t-statistics: the 20th, 50th, and 90th
percentiles. HLZ’s Table 5, Panel A, $\rho=0.2$ arrives at $p_{0}=0.444$,
$\lambda=55.5$ bps per month, and $N=1,378$.

##  Appendix E Equating of false, null, and insignificant in [20]

[20] (HL) “show exactly what went wrong with the inference in [39]” (page
2506, paragraph 4). In particular, HL state that Yan and Zheng “claim that a
large fraction of the 18,000 anomalies in their data are true.” (page 2506,
paragraph 2). This statement about Yan and Zheng is repeated several times in
HL (e.g. page 2519, paragraph 3) and HL devote an entire section (II.A.5) to
refuting this claim.

However, Yan and Zheng do not make this claim. Instead, they “find that many
fundamental signals are significant predictors of cross-sectional stock
returns even after accounting for data mining” (abstract). Similar statements
are found throughput Yan and Zheng (e.g. page 1416, paragraph 3). HL’s
statement and Section II.A.5 are thus hard to understand—unless one equates
significance (under HL’s chosen test) with truth.

HL explain their argument in more detail in section II.A.5, “Revisiting Yan
and Zheng (2017).” This section begins (page 2541) with:

> _Applying the preferred methods based on Table III to the 18,000 anomalies,
> the fraction of true strategies is found to be 0.000% ([BY Theorem 1.3]) and
> 0.015% (Storey, $\theta$ = 0.8) under a 5% significance level, and 0.006%
> ([BY Theorem 1.3]) and 0.091% (Storey, $\theta$ = 0.8) under a 10%
> significance level. _

There are multiple issues with this paragraph. First, the fraction of true
strategies does not traditionally depend on the significance level.

Second, BY Theorem 1.3 traditionally provides an _upper bound_ on the FDR
(Section 4.3). It can say the fraction of true strategies is _at least
0.000%._ But this theorem cannot say that the fraction of true strategies is
0.000%.

Last, it is not possible that [32] finds an FDR of less than 1% in the [39]
strategies. This can be seen applying Equation (9) to the numbers reported in
Table 1 of Yan and Zheng. This table shows that at most 80 percent of
$|t_{i}|$ fall between -1.62 and +1.58 (using VW 4-factor $\alpha$s, which is
the only part of the Yan and Zheng data examined in HL). Thus, the law of
total probability implies that

$\displaystyle\Pr\left(F_{i}\right)$
$\displaystyle=\frac{\Pr\left(t_{i}\in[-1.62,+1.58]\right)-\Pr\left(t_{i}\in[-1.62,+1.58]|T_{i}\right)\Pr\left(T_{i}\right)}{\Pr\left(t_{i}\in[-1.62,+1.58]|F_{i}\right)}$
$\displaystyle\leq\frac{\Pr\left(t_{i}\in[-1.62,+1.58]\right)}{\Pr\left(t_{i}\in[-1.62,+1.58]|F_{i}\right)}$
$\displaystyle=\frac{0.80}{0.89}=89\%,$

where $\Pr\left(t_{i}\in[-1.62,+1.58]|F_{i}\right)=0.89$ is calculated using
the standard normal distribution. In other words, the fraction of true
strategies is at least 11%.